Dragon-s-Lair-X68k/docs/STATUS.md

# Status & next-session handoff — session 4 in progress (2026-08-23)

## Decisions locked

| decision | value | why |
|---|---|---|
| Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint |
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
| **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 |
| **Quality modes** | **two: `sasi` and `scsi`** (USER DECISION, session 2) | one codec, one decoder, one bitstream; only `lam` differs |
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
| SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution |

### The SASI/SCSI question is RESOLVED
Session 1 left "which machine do we target" open. The user's answer: **ship both**,
as two quality profiles. This is now implemented rather than hypothetical — the
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:

| profile | target | lam | quality (00020 / 00146) | machine |
|---|---|---|---|---|
| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |

Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on
SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**,
not an average over seek latency.

**Both profiles fit with room.** Ring-buffer simulation on the real per-frame
sizes gives **zero required prefill** for every scene at both profiles: the fill
delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds
that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance,
far more than an SD-backed seek needs. FINDINGS 21.

An earlier warning here said `scsi` did not fit because a frame peaked at 96.4%
of the pipe. That compared instantaneous demand to a sustained rate as if they
had to match frame-by-frame; with a buffer the test is cumulative, and it passes.

`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
RD curve rather than derived from the hardware. See FINDINGS 17.

Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured
and rejected — see FINDINGS 14, it was a false-good result from a rate model
that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff.

Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised
frame (measured 0.00 dB loss). The profiles are two points on one continuous
rate-distortion curve, not two codecs.

---

## What session 3 settled

1. **The display path works and is verified end to end.** First real frame on an
   emulated X68000 screen: `docs/images/x68k_first_frame_compare.png`. Full
   write-up in **FINDINGS 22**. Everything before this session was Python-side
   or a headless `-video none` run, which cannot snapshot at all.
2. **The render is pixel-exact, not merely close.** With monitor contrast at 15,
   all 256 palette entries render exactly as `GGGGGRRRRRBBBBBI` + `pal6bit`
   predicts. That exactness is the regression test — see
   `tools/bench/verify_frame.py`, which exits non-zero if it ever drifts.
3. **Three hardware facts that were previously assumed are now confirmed from
   MAME 0.277 source**, not folklore: the palette word format, the 1024-byte
   GVRAM line stride, and the 256-colour page aliasing in `HARDWARE.md`. All
   three were already written down correctly; they are now cited.
4. **A new quality ceiling was measured** — the 15-bit+I palette alone costs
   38.88 dB, the same order as the `scsi` profile's own codec error. FINDINGS
   22.4. This bounds how much further `scsi` is worth raising.
5. **Two shell traps that wedged session 2's background jobs** are documented in
   the working-setup section below. They cost ~1.5 h of wall clock and a wedged
   CPU core, and one of them was hit again this session.

## What session 4 settled (in progress)

1. **A real 256x256 CRTC mode exists and is verified.** `crtc_mode.lua`, derived
   from `x68k_crtc.cpp`'s divisor ladder rather than recalled — the derivation is
   self-checking (368 = 1104/3 exactly, so the horizontal registers divide by
   three with no remainder). Snapshot is native 256x512, active area pixel-exact,
   letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone.
2. **The palette ceiling was wrong by 2 dB, in our favour.** The shared LSB `I`
   must be chosen **per palette entry**, not hardcoded to 1. Doing so lifts the
   display ceiling from 38.85 to **40.81 dB** and is the only way to get true
   black at all (`pal6bit(1) = 4`). 102 of 256 entries want `I = 0`. This
   supersedes FINDINGS 22.4 and gives `scsi` ~2 dB more headroom than believed.
   **The encoder does not do this yet** — see the encoder-gaps list.
3. **Letterboxing costs one palette entry.** 255 colours + a reserved black at
   index 0, with `I = 0` on it. `prep_frame.py --reserve-black`. FINDINGS 23.4.
4. **MAME's graphics double-scan is phase-shifted one raster line** — pairs are
   (1,2),(3,4),..., not (0,1), because `get_gfx_pixel` halves the *absolute*
   scanline and `vbegin = 41` is odd. Cost a false failure. FINDINGS 23.2.

---

## What session 2 settled

1. **The critical-path question is answered.** "Does VQ soften Bluth's linework
   unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.**
   Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two
   images in `docs/images/`. Both profiles use **k=256**; see item 2b.
2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured).
   Halving the framerate does not halve the bitrate. FINDINGS 8.
2b. **A fourth false-good result was produced and caught this session** — k=1024
   codebooks looked like a +2.4 dB free win because the rate model charged 1 byte
   for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
3. **The 256-colour palettised frame is the real quality ceiling** and it looks
   excellent. Judge the codec against that, not against 1080p.
4. Encoder exists and produces a real bitstream: `tools/encoder/`.

---

## Encoder — working

```
python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
python3 tools/encoder/encode.py  /tmp/fr_00020 out.dlx --profile sasi --preview p.png
```

| file | role |
|---|---|
| `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise |
| `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR |
| `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision |
| `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control |
| `encode.py` | CLI + `DLX1` container writer |

`DLX1` container layout is documented in the `encode.py` docstring. All
multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`.

### Known encoder gaps
- **Rate control is written but not yet wired into `encode.py`** — the CLI uses a
  fixed `lam` from the profile. `ratectl.encode_rate_controlled()` exists and
  builds a lam-ladder per frame; it needs hooking up and validating.
- **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in
  the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
- **Palette packing is not implemented in the encoder.** It still emits 24-bit
  palettes; the X68000 word packing happens Lua-side. Whatever writes real
  palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3,
  worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4).
- Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
- `_paint` is a Python per-block loop — fine for prototyping, slow for a full
  disc encode. Vectorise before the 224-stream run.

---

## Working setup (unchanged from session 1, re-verified)

**MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**.
```
mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3
```
**Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`

**Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM`
(still mounted as of end of session 2).

**MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the
notifier subscription in a global; the stack register is `SP` not `A7`;
`autoboot_script` fires at PC=0 before boot) are documented in FINDINGS.

**Two shell traps, both hit again this session:**
- piping MAME (or any long job) through `grep` block-buffers — write to a file.
- `pkill -f <pattern>` matches your own shell and kills it (exit 144).
  Use `pkill -x` or kill by PID.
- **`until ! pgrep -f foo.py; do sleep; done` watcher loops never exit.** The
  watching shell's own command line contains the string `foo.py`, so `pgrep -f`
  matches the watcher itself and the loop spins forever. Session 2 left 11 of
  these wedged for over an hour. Wait on the PID (`while kill -0 $PID`) or on a
  sentinel file the job touches when it finishes -- never on a `-f` name match.
- **`timeout N mame ...` does not kill MAME.** MAME catches SIGTERM and, with an
  autoboot script blocked waiting on a flag that never arrives, never reaches
  its shutdown path. `timeout` without `-k` then waits forever while MAME burns
  a full core at `-nothrottle`. Always `timeout -k 5 N`.

---

## Disk throughput benchmark — still blocked, no longer gating

`IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested
hypotheses are in session 1's notes (git history of this file, commit 65112b9);
the ordered plan for retrying is in **`docs/BENCHMARK.md`**.

**Status changed twice this session — read this rather than the git history.**
It was briefly promoted to critical-path while the working bandwidth figure was
misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the
ring-buffer simulation showing **zero required prefill** for both profiles
(FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not**
available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a
"measure it and maybe ship transparent" decision waiting.

What the benchmark is still worth doing for:
- **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is
  not recorded. Every profile hangs off it.
- **Confirming DMA is actually used.** If transfers fall back to PIO the CPU
  cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the
  binding constraint. This is the worst plausible outcome and the cheapest to
  check — do it first.

**Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are
functional models, not timing-accurate; a KB/s figure from MAME measures the
emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach
(MAME validates the path, derivation bounds it, real hardware settles it).

## Display path — VERIFIED (session 3), in a real mode (session 4). CPU path — still unproven.

The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`.

**What this does and does not mean.** The video hardware is genuinely emulated
and the render is bit-exact. But GVRAM was filled by a MAME Lua script, not by
68000 code — no 68000 instruction has drawn a pixel yet. Lua writes cost zero
68000 cycles, so the 38% full-frame blit estimate underpinning the whole CPU
budget is still unvalidated. "Verified end to end" applies to the *display*
path only. See FINDINGS 22 scope note.
Full write-up in **FINDINGS 22**. Harness: `tools/bench/show_frame.lua` +
`tools/bench/prep_frame.py`.

Three facts the player MUST honour, none of which were guessable:

| what | where | value |
|---|---|---|
| **Un-hide the graphics layer** | CRTC R20 `$E80028` | clear bit 11 ("G-VRAM set to buffer"); IPL leaves `0x0B16` |
| Colour setup (256c) | CRTC R20 bits 9-8 | `0x0100` -> `R20 = 0x0116` |
| **Monitor contrast** | `$E8E001` bits 3-0 | IPL leaves **14**; write **15** or everything renders 7% dark |

Bit 11 is the one that cost the most time: GVRAM writes land and read back
correctly while the layer is invisible, so the video controller looks guilty and
is not. Contrast `0` blanks the screen — free fade-to-black for transitions.

Palette format is now **confirmed from MAME source**, not assumed:
`GGGGGRRRRRBBBBBI` (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as
`pal6bit((field<<1)|I)`. With contrast at 15 the render is **pixel-exact**.

New ceiling: the 15-bit+I palette alone costs **38.88 dB** against the 24-bit
palettised source — the same order as the `scsi` profile's own codec error
(39.4 dB). `scsi` is close to display-transparent on real hardware. See
FINDINGS 22.4 before considering raising quality further.

Snapshot recipe that works (`-video none` CANNOT snapshot):
```
SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
  -sound none -nothrottle -plugins -autoboot_script <script>.lua \
  -snapshot_directory ./snap -snapview native -seconds_to_run 6
```
`-snapview native` drops MAME's LED artwork and gives a clean 768x512 screen.

## Next steps, in priority order

1. **Full-disc survey.** Only 4 clips of 1.2-1.7 s out of 224 streams have been
   measured, and 00146 already runs 23% hotter than 00020. A *sustained* action
   sequence is the one thing that could still break the bitrate. Classify menu
   vs content first (FINDINGS 13) or the averages are diluted by static menus.
   **Vectorise `_paint` before this run** — it is a Python per-block loop.
2. **68000 decoder skeleton.** Parse `DLX1`, expand codebooks to word-per-pixel,
   blit SKIP/V1/V4/RAW. Measure real cycles with the existing MAME Lua harness.
   **Fully unblocked** — the display path is verified (FINDINGS 22) AND the
   target CRTC mode is now real (FINDINGS 23), so 68000 code has a defined
   geometry to write into: 256 words per row, 1024-byte line stride, picture in
   rows 32..223 of a 256-row page. `tools/bench/show_frame256.lua` gives a
   known-good reference image to diff the 68000's output against. This is what
   validates the 38% full-frame blit estimate the whole CPU budget rests on.
   **This is now the top priority** — it is the only remaining unknown that can
   still invalidate the design.
2a. ~~CRTC mode table for 256x192-in-256x256.~~ **DONE, session 4.** Derived from
   the CRTC divisor ladder (not recalled), verified by snapshot, pixel-exact.
   `tools/bench/crtc_mode.lua`; write-up in FINDINGS 23; regression test
   `tools/bench/verify_frame256.py`. Untested on real hardware, but the blanking
   timing is identical to the IPL's 768 mode, which is what a monitor cares about.

3. **Wire rate control into `encode.py`.** No longer a blocker (FINDINGS 21), but
   it is what gives a deterministic ceiling over content not yet measured, which
   was the original reason for choosing VQ. Insurance, not a fix. Pairs with (1).
4. **Confirm DMA vs PIO in MAME** (see the benchmark section above) — cheap, and
   the only thing that could still move CPU into the binding position.
5. **Resolve the framing question** (FINDINGS 12: crop vs squash vs wide).
   Needs an eyeball against arcade reference, not a measurement.
6. **Import the scene graph.** SNES project `data/events/` (MIT, cleared),
   cross-checked against DirkSimple (zlib) which transcribed the same data
   independently — diff them to catch transcription errors before committing
   any of it to 68000 tables.
7. **ADPCM audio.** MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in
   `ratectl.py`, not yet extracted or encoded.

### Explicitly abandoned — do not re-propose
- ~~Entropy-code the payload.~~ Deflate decode is ~216% of the frame budget on a
  68000; LZ4 is ~54% with no room beside a 38% blit (FINDINGS 17.2). All bitrates
  are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8
  to a compression upper bound, not a shippable design.
- ~~k=1024 codebooks.~~ False-good result from a rate model that charged 1 byte
  for a 10-bit index (FINDINGS 14). k=256 wins at every matched bitrate.
- ~~Flat 4x4 VQ.~~ Rejected by eye (FINDINGS 9).

## Not yet started
- **Any 68000 player code.** `src/player/` is still empty. The display path is
  proven, but proven *from Lua* — no 68000 instruction has yet drawn a pixel.
- ADPCM audio extraction/encoding
- Disk image packaging
- Game logic (scene branching, input windows, death clips)

## Reproducing the 256x256 mode result (session 4)

```
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
mkdir -p tmp/snap256 && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 90 mame x68000 \
  -bios ipl10 -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/show_frame256.lua \
  -snapshot_directory ./snap256 -snapview native -seconds_to_run 6
cd .. && python3 tools/bench/verify_frame256.py
```
Exits non-zero on any drift. Expected: `256x512 native, double-scan exact,
active 256x192 pixel-exact, letterbox true black`, ceiling 40.81 dB.

## Reproducing the display result

```
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame.bin 0
mkdir -p tmp/snap_verify && cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 \
  -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/show_frame.lua \
  -snapshot_directory ./snap_verify -snapview native -seconds_to_run 6
cd .. && python3 tools/bench/verify_frame.py
```
Verified cold from the Blu-ray at end of session 3: exact match, 38.88 dB.

`tmp/` is gitignored scratch. The frames are NOT in the repo — regenerate them
with `extract.py`; the earlier ones lived in `/tmp` and do not survive a reboot.

## Reference material on this box (not in the repo)

- **MAME 0.277 source: `~/src/mame-mame0277/`** (tarball `~/src/mame0277.tar.gz`).
  Downloaded this session to settle the graphics-layer question. The files that
  matter are `src/mame/sharp/x68k_v.cpp`, `x68k_crtc.cpp`, `x68k_crtc.h`,
  `x68k.cpp`. **Read these before theorising about X68000 video behaviour** —
  six register-poking attempts failed against a gate that one grep found.
- Blu-ray mounted at `/media/reala-misaki/BDROM` via
  `udisksctl loop-setup -r -f DRAGONS_LAIR.iso`.