Dragon-s-Lair-X68k/docs/STATUS.md

# Status & next-session handoff — end of session 5 (2026-08-23)

## Start here: is the tree still green?

```
./tools/bench/check.sh
```
~40 s, needs the Blu-ray mounted. Re-runs both display regression tests from
source media and prints `ALL GREEN`. Verified green at end of session 4.
If it fails, fix that before doing anything else — everything downstream assumes
the display path is pixel-exact.

## Decisions locked

| decision | value | why |
|---|---|---|
| Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint |
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
| **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 |
| **Quality modes** | **two: `sasi` and `scsi`** (USER DECISION, session 2) | one codec, one decoder, one bitstream; only `lam` differs |
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
| SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution |

### The SASI/SCSI question is RESOLVED
Session 1 left "which machine do we target" open. The user's answer: **ship both**,
as two quality profiles. This is now implemented rather than hypothetical — the
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:

| profile | target | lam | quality (00020 / 00146) | machine |
|---|---|---|---|---|
| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |

Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on
SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**,
not an average over seek latency.

**Both profiles fit with room.** Ring-buffer simulation on the real per-frame
sizes gives **zero required prefill** for every scene at both profiles: the fill
delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds
that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance,
far more than an SD-backed seek needs. FINDINGS 21.

An earlier warning here said `scsi` did not fit because a frame peaked at 96.4%
of the pipe. That compared instantaneous demand to a sustained rate as if they
had to match frame-by-frame; with a buffer the test is cumulative, and it passes.

`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
RD curve rather than derived from the hardware. See FINDINGS 17.

Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured
and rejected — see FINDINGS 14, it was a false-good result from a rate model
that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff.

Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised
frame (measured 0.00 dB loss). The profiles are two points on one continuous
rate-distortion curve, not two codecs.

---

## What session 3 settled

1. **The display path works and is verified end to end.** First real frame on an
   emulated X68000 screen: `docs/images/x68k_first_frame_compare.png`. Full
   write-up in **FINDINGS 22**. Everything before this session was Python-side
   or a headless `-video none` run, which cannot snapshot at all.
2. **The render is pixel-exact, not merely close.** With monitor contrast at 15,
   all 256 palette entries render exactly as `GGGGGRRRRRBBBBBI` + `pal6bit`
   predicts. That exactness is the regression test — see
   `tools/bench/verify_frame.py`, which exits non-zero if it ever drifts.
3. **Three hardware facts that were previously assumed are now confirmed from
   MAME 0.277 source**, not folklore: the palette word format, the 1024-byte
   GVRAM line stride, and the 256-colour page aliasing in `HARDWARE.md`. All
   three were already written down correctly; they are now cited.
4. ~~**A new quality ceiling was measured** — the 15-bit+I palette alone costs
   38.88 dB.~~ **Superseded by session 4:** that figure assumed the shared LSB
   `I` is always 1. Chosen per entry, the ceiling is **40.81 dB**. FINDINGS 23.3.
5. **Two shell traps that wedged session 2's background jobs** are documented in
   the working-setup section below. They cost ~1.5 h of wall clock and a wedged
   CPU core, and one of them was hit again this session.

## What session 5 settled

1. **68000 code drew a frame, and the blit was measured.** `tools/bench/blit.s`
   + `blit.lua`. The snapshot passes `verify_frame256.py` unchanged — pixel-exact
   in the real 256x256 mode. **FINDINGS 23.5 is closed**: no longer "proven from
   Lua only".
2. **The 38% full-frame blit estimate is dead. It is 53.6%.** And that is a
   zero-wait-state floor — MAME models no GVRAM wait states, so real hardware is
   worse. FINDINGS 24. Every variant was hand-derived from the MC68000 timing
   tables before being measured and the two agree to 0.006-0.43%, so this is not
   another MAME artefact.
3. **Reading the source frame is exactly half the blit cost** (V1 53.6% vs a
   write-only floor V3 of 27.1%). That is what makes the architecture question
   below live.
4. **The decoder architecture now hinges on one unmeasured number.** Writing
   codewords straight into GVRAM costs 76.6% of the frame budget for a *full*
   frame (V4 — the 1024-byte stride kills the `movem.l` burst), but scales with
   the non-SKIP block fraction and needs **no RAM reference frame at all**,
   because the previous frame is already in GVRAM. Compose-then-blit is a flat
   53.6%. **They cross at 70% of blocks changed.** FINDINGS 24.5.

---

## What session 4 settled

1. **A real 256x256 CRTC mode exists and is verified.** `crtc_mode.lua`, derived
   from `x68k_crtc.cpp`'s divisor ladder rather than recalled — the derivation is
   self-checking (368 = 1104/3 exactly, so the horizontal registers divide by
   three with no remainder). Snapshot is native 256x512, active area pixel-exact,
   letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone.
2. **The palette ceiling was wrong by 2 dB, in our favour.** The shared LSB `I`
   must be chosen **per palette entry**, not hardcoded to 1. Doing so lifts the
   display ceiling from 38.85 to **40.81 dB** and is the only way to get true
   black at all (`pal6bit(1) = 4`). 102 of 256 entries want `I = 0`. This
   supersedes FINDINGS 22.4 and gives `scsi` ~2 dB more headroom than believed.
   **The encoder does not do this yet** — see the encoder-gaps list.
3. **Letterboxing costs one palette entry.** 255 colours + a reserved black at
   index 0, with `I = 0` on it. `prep_frame.py --reserve-black`. FINDINGS 23.4.
4. **MAME's graphics double-scan is phase-shifted one raster line** — pairs are
   (1,2),(3,4),..., not (0,1), because `get_gfx_pixel` halves the *absolute*
   scanline and `vbegin = 41` is odd. Cost a false failure. FINDINGS 23.2.

---

## What session 2 settled

1. **The critical-path question is answered.** "Does VQ soften Bluth's linework
   unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.**
   Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two
   images in `docs/images/`. Both profiles use **k=256**; see item 2b.
2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured).
   Halving the framerate does not halve the bitrate. FINDINGS 8.
2b. **A fourth false-good result was produced and caught this session** — k=1024
   codebooks looked like a +2.4 dB free win because the rate model charged 1 byte
   for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
3. **The 256-colour palettised frame is the real quality ceiling** and it looks
   excellent. Judge the codec against that, not against 1080p.
4. Encoder exists and produces a real bitstream: `tools/encoder/`.

---

## Encoder — working

```
python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
python3 tools/encoder/encode.py  /tmp/fr_00020 out.dlx --profile sasi --preview p.png
```

| file | role |
|---|---|
| `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise |
| `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR |
| `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision |
| `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control |
| `encode.py` | CLI + `DLX1` container writer |

`DLX1` container layout is documented in the `encode.py` docstring. All
multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`.

### Known encoder gaps
- **Rate control is written but not yet wired into `encode.py`** — the CLI uses a
  fixed `lam` from the profile. `ratectl.encode_rate_controlled()` exists and
  builds a lam-ladder per frame; it needs hooking up and validating.
- **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in
  the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
- **Palette packing is not implemented in the encoder.** It still emits 24-bit
  palettes; the X68000 word packing happens Lua-side. Whatever writes real
  palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3,
  worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4).
- Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
- `_paint` is a Python per-block loop — fine for prototyping, slow for a full
  disc encode. Vectorise before the 224-stream run.

---

## Working setup (unchanged from session 1, re-verified)

**MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**.
```
mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3
```
**Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`

**Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM`
(still mounted as of end of session 2).

**MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the
notifier subscription in a global; the stack register is `SP` not `A7`;
`autoboot_script` fires at PC=0 before boot) are documented in FINDINGS.

**Two shell traps, both hit again this session:**
- piping MAME (or any long job) through `grep` block-buffers — write to a file.
- `pkill -f <pattern>` matches your own shell and kills it (exit 144).
  Use `pkill -x` or kill by PID.
- **`until ! pgrep -f foo.py; do sleep; done` watcher loops never exit.** The
  watching shell's own command line contains the string `foo.py`, so `pgrep -f`
  matches the watcher itself and the loop spins forever. Session 2 left 11 of
  these wedged for over an hour. Wait on the PID (`while kill -0 $PID`) or on a
  sentinel file the job touches when it finishes -- never on a `-f` name match.
- **`timeout N mame ...` does not kill MAME.** MAME catches SIGTERM and, with an
  autoboot script blocked waiting on a flag that never arrives, never reaches
  its shutdown path. `timeout` without `-k` then waits forever while MAME burns
  a full core at `-nothrottle`. Always `timeout -k 5 N`.

---

## Disk throughput benchmark — still blocked, no longer gating

`IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested
hypotheses are in session 1's notes (git history of this file, commit 65112b9);
the ordered plan for retrying is in **`docs/BENCHMARK.md`**.

**Status changed twice this session — read this rather than the git history.**
It was briefly promoted to critical-path while the working bandwidth figure was
misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the
ring-buffer simulation showing **zero required prefill** for both profiles
(FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not**
available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a
"measure it and maybe ship transparent" decision waiting.

What the benchmark is still worth doing for:
- **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is
  not recorded. Every profile hangs off it.
- **Confirming DMA is actually used.** If transfers fall back to PIO the CPU
  cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the
  binding constraint. This is the worst plausible outcome and the cheapest to
  check — do it first.

**Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are
functional models, not timing-accurate; a KB/s figure from MAME measures the
emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach
(MAME validates the path, derivation bounds it, real hardware settles it).

## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5).

The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`.

**Session 5 closed the gap this paragraph used to describe.** GVRAM is now
filled by 68000 instructions and the result is still pixel-exact, and the blit
cost is measured rather than estimated: **53.6% of a 12fps frame**, not 38%
(FINDINGS 24). The remaining caveat is different and narrower: MAME models
**no GVRAM wait states**, so 53.6% is a floor and real hardware is worse.
Full write-up in **FINDINGS 22**. Harness: `tools/bench/show_frame.lua` +
`tools/bench/prep_frame.py`.

Three facts the player MUST honour, none of which were guessable:

| what | where | value |
|---|---|---|
| **Un-hide the graphics layer** | CRTC R20 `$E80028` | clear bit 11 ("G-VRAM set to buffer"); IPL leaves `0x0B16` |
| Colour setup (256c) | CRTC R20 bits 9-8 | `0x0100` |
| **Monitor contrast** | `$E8E001` bits 3-0 | IPL leaves **14**; write **15** or everything renders 7% dark |

The `R20 = 0x0116` value quoted here in session 3 is the **768-wide IPL timing**
with the gate cleared. The shipping value is **`R20 = 0x0110`** — see the mode
table in `tools/bench/crtc_mode.lua`, which is now the single source of truth
for all of R00-R08 and R20.

Bit 11 is the one that cost the most time: GVRAM writes land and read back
correctly while the layer is invisible, so the video controller looks guilty and
is not. Contrast `0` blanks the screen — free fade-to-black for transitions.

Palette format is now **confirmed from MAME source**, not assumed:
`GGGGGRRRRRBBBBBI` (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as
`pal6bit((field<<1)|I)`. With contrast at 15 the render is **pixel-exact**.

Ceiling: the 15-bit+I palette costs **40.81 dB** against the 24-bit palettised
source, once `I` is chosen per entry (FINDINGS 23.3 — session 3's 38.88 dB
assumed `I = 1`). Still the same order as the `scsi` profile's own codec error
(39.4 dB), so `scsi` remains near display-transparent, with ~2 dB more headroom
than session 3 thought.

Snapshot recipe that works (`-video none` CANNOT snapshot):
```
SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
  -sound none -nothrottle -plugins -autoboot_script <script>.lua \
  -snapshot_directory ./snap -snapview native -seconds_to_run 6
```
`-snapview native` drops MAME's LED artwork and gives a clean 768x512 screen.

## Next steps, in priority order

1. **Measure the non-SKIP block fraction.** *(new top priority, session 5)*
   FINDINGS 24.5: compose-in-RAM-then-blit costs a flat 53.6% of the frame
   budget; decode-direct-to-GVRAM costs 76.6% x (fraction of blocks that are not
   SKIP) and needs no RAM reference frame. **They cross at 70%.** Which side of
   70% the content sits on decides which decoder inner loop to write, so this
   must come before writing one.
   **It needs no new machinery** — the mode decision in `vq_hybrid.py` already
   computes it per frame and simply never reports it. Add the histogram
   (SKIP / V1 / V4 / RAW counts per frame) to `encode.py` output and run it over
   the clips already extracted. Report the *distribution*, not the mean: a
   scene-cut frame is ~100% non-SKIP and a held frame near 0%, and the mean of
   those two is a number describing no actual frame.

2. **68000 decoder skeleton**, with the inner loop chosen by (1). Parse `DLX1`,
   expand codebooks, blit per block mode. The display path is verified *by 68000
   code* now (FINDINGS 24) and the harness pattern is `tools/bench/blit.s` +
   `blit.lua`, which already loads code, masks interrupts, times a loop against
   a flag, and snapshots the result for `verify_frame256.py`. Copy that.
   Assembler: `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`.

2a. **Re-budget everything against 53.6%, not 38%.** Several downstream figures
   were derived from the old estimate. The blit alone now eats over half the
   frame at 12fps in the compose-then-blit design, before any decode, and MAME
   models no GVRAM wait states so that is a floor. This may reopen questions
   that were closed against the 38% number — check FINDINGS 17.2's entropy-coding
   rejection, which was argued as "54% LZ4 with no room beside a 38% blit". The
   conclusion gets *stronger*, not weaker, but the arithmetic should be restated.

3. **Full-disc survey.** Only 4 clips of 1.2-1.7 s out of 224 streams have been
   measured, and 00146 already runs 23% hotter than 00020. A *sustained* action
   sequence is the one thing that could still break the bitrate. Classify menu
   vs content first (FINDINGS 13) or the averages are diluted by static menus.
   **Vectorise `_paint` before this run** — it is a Python per-block loop.
   Pairs naturally with (1): the same run produces both numbers.

4. **Wire rate control into `encode.py`.** No longer a blocker (FINDINGS 21), but
   it is what gives a deterministic ceiling over content not yet measured, which
   was the original reason for choosing VQ. Insurance, not a fix. Pairs with (1).
5. **Confirm DMA vs PIO in MAME** (see the benchmark section above) — cheap, and
   the only thing that could still move CPU into the binding position.
6. **Resolve the framing question** (FINDINGS 12: crop vs squash vs wide).
   Needs an eyeball against arcade reference, not a measurement.
7. **Import the scene graph.** SNES project `data/events/` (MIT, cleared),
   cross-checked against DirkSimple (zlib) which transcribed the same data
   independently — diff them to catch transcription errors before committing
   any of it to 68000 tables.
8. **ADPCM audio.** MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in
   `ratectl.py`, not yet extracted or encoded.

### Explicitly abandoned — do not re-propose
- ~~Entropy-code the payload.~~ Deflate decode is ~216% of the frame budget on a
  68000; LZ4 is ~54% with no room beside a 38% blit (FINDINGS 17.2). All bitrates
  are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8
  to a compression upper bound, not a shippable design.
- ~~k=1024 codebooks.~~ False-good result from a rate model that charged 1 byte
  for a 10-bit index (FINDINGS 14). k=256 wins at every matched bitrate.
- ~~Flat 4x4 VQ.~~ Rejected by eye (FINDINGS 9).

## Not yet started
- **Any 68000 player code.** `src/player/` is still empty. 68000 code has now
  drawn a frame, but it lives in `tools/bench/blit.s` as a benchmark, not in a
  player: it does no bitstream parsing, no mode dispatch, no codebook expansion.
- ADPCM audio extraction/encoding
- Disk image packaging
- Game logic (scene branching, input windows, death clips)

## Reproducing the 256x256 mode result (session 4)

```
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
mkdir -p tmp/snap256 && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 90 mame x68000 \
  -bios ipl10 -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/show_frame256.lua \
  -snapshot_directory ./snap256 -snapview native -seconds_to_run 6
cd .. && python3 tools/bench/verify_frame256.py
```
Exits non-zero on any drift. Expected: `256x512 native, double-scan exact,
active 256x192 pixel-exact, letterbox true black`, ceiling 40.81 dB.

## Reproducing the display result

```
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame.bin 0
mkdir -p tmp/snap_verify && cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 \
  -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/show_frame.lua \
  -snapshot_directory ./snap_verify -snapview native -seconds_to_run 6
cd .. && python3 tools/bench/verify_frame.py
```
Verified cold from the Blu-ray at end of session 3: exact match, 38.88 dB.
(That 38.88 is correct *for this test*: `show_frame.lua` still packs `I = 1`.
The 40.81 dB ceiling comes from `show_frame256.lua`, which picks `I` per entry.)

`tmp/` is gitignored scratch. The frames are NOT in the repo — regenerate them
with `extract.py`; the earlier ones lived in `/tmp` and do not survive a reboot.

## Reference material on this box (not in the repo)

- **MAME 0.277 source: `~/src/mame-mame0277/`** (tarball `~/src/mame0277.tar.gz`).
  Downloaded this session to settle the graphics-layer question. The files that
  matter are `src/mame/sharp/x68k_v.cpp`, `x68k_crtc.cpp`, `x68k_crtc.h`,
  `x68k.cpp`. **Read these before theorising about X68000 video behaviour** —
  six register-poking attempts failed against a gate that one grep found.
- Blu-ray mounted at `/media/reala-misaki/BDROM` via
  `udisksctl loop-setup -r -f DRAGONS_LAIR.iso`.

---

## Parked ideas (not scheduled, not abandoned)

- **Cliff Hanger, retitled as Lupin III** (user, session 4). Stern's 1983
  laserdisc game was cut from *Castle of Cagliostro* and *Mystery of Mamo* with
  the Lupin branding stripped; a port would restore it. Technically **cheaper
  than this project**: same content class (cel animation, flat colour, hard
  cuts), ~13 min of footage vs Dragon's Lair's ~22, and flatter linework than
  Bluth's, so fewer blocks should escape to V4/RAW. The codec, the display path,
  and `crtc_mode.lua` would all drop straight in.
  **The real cost is media prep, not code:** there is no clean master cut to
  Stern's scene boundaries the way `DRAGONS_LAIR.iso` is, so the footage would
  have to be sourced and cut to match. Not to be started until the CPU path is
  proven — it changes nothing about whether this design works.

## Reproducing the blit measurement (session 5)

```
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
tools/vasm/vasmm68k_mot -Fbin -o tmp/blit.bin tools/bench/blit.s
mkdir -p tmp/snap_blit && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
  -bios ipl10 -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/blit.lua \
  -snapshot_directory ./snap_blit -snapview native -seconds_to_run 120
```
~25 s wall. Prints cycles/frame and % of a 12fps budget for V1-V4, and snapshots
V1's output. To check that snapshot is still pixel-exact:
`sed 's|snap256|snap_blit|' tools/bench/verify_frame256.py | python3 -`

Not added to `check.sh`: `check.sh` asserts pixel-exactness, and asserting wall
timings there would make the green-light check sensitive to host load.