Dragon-s-Lair-X68k/docs/STATUS.md

Status & next-session handoff — end of session 7 (2026-08-23)

NEXT SESSION: measure a span, then make the mode decision cost-aware

The decoder exists, it is pixel-exact, and it does not fit. On the worst sustained window at sasi it costs a mean of 81.7% of a 12fps frame and 31% of frames exceed 100% (scsi: 94.9% median, 42% miss). FINDINGS 28. CPU is the binding constraint now — the first time in this project.

Two levers, and the cheap one has to be measured first.

Lever A — spend bandwidth to buy cycles. The bus sits 4x idle: sasi uses 110 KB/s of 488. Every codec decision was made when bytes were scarce, so each one trades cycles to save them, and the cheapest thing a 68000 can be handed is the most expensive thing to store — word-expanded pixels in row-linear runs. Adding one mode, a per-row span of literal words movem.l-ed straight from the stream buffer into GVRAM, prices out at (FINDINGS 29, 12_span_tradeoff.py):

	today	+ literal spans
median frame	74.4%	43.0%
worst frame	136.2%	106.2%
frames missing	37/120	8/120
bitrate	101.7 KB/s	453.2 KB/s (bus 488)

This is DERIVED, not measured, and it is load-bearing — so measure it first. Extend tools/bench/blit.s with a span variant and time it against run length. The 9.08 cycles/pixel it rests on is real (FINDINGS 24 V1) but was measured at full row width with 12-register bursts; short and oddly-aligned spans cannot burst as well and are flattered by the model. If spans come in near the derived figure, the whole mode set changes and lever B optimises over different modes — which is exactly why this goes first. FINDINGS 29.5 lists the other three things that have to hold, of which confirming DMA vs PIO is the cheapest and now the most consequential: at 453 KB/s a PIO fallback puts the transfer back on the CPU this is trying to relieve.

Lever B — stop buying modes the CPU cannot afford. vq_hybrid.decide() minimises D + lam*R — distortion against BYTES — on a machine whose binding budget is CYCLES, and the two are not proportional:

mode	payload bytes	measured cycles	cycles per byte
SKIP	0	13 (clustered)	—
V1	1	300	300
V4	4	448	112
RAW	16	400	25
word-expanded literal block	32	~240 (derived)	7.5

V4 is 25% of blocks and 50% of the cycles. The lagrangian charges it 4x a V1 block; the CPU charges it 1.49x. Note the last row: a literal block is cheaper than every codebook mode, and pixel-exact — the codebook is a byte optimisation that now costs cycles (FINDINGS 29.2).

The work, in order:

0. Measure the span cost on the 68000 (lever A above). Cheap, and everything below optimises over whatever mode set it leaves.

1. Add a cycle term to the mode decision. decide() already builds a cost matrix of error + lam * bytes per mode per block; add + mu * cycles, with the per-mode cycles measured in FINDINGS 28.2. SKIP is not a constant and this is the one trap here. A SKIP block costs 13.25 cycles when all four blocks in its header byte are SKIP (one tst.b clears the group) and ~45 when it sits in a mixed byte — so SKIP's price depends on its neighbours, which a per-block lagrangian cannot see. Do not pick one number and move on: 45 overcharges clustered SKIPs and pushes the encoder away from the mode that saves cycles, 13.25 undercharges isolated ones and lets frames overrun. The way out is that the budget check does not have to use the same cost function as the mode decision — score frames with the exact clustered cost (cycles() in tools/analysis/11_cpu_budget.py, validated to 1 point against the 68000) and let the bisection converge on that, while the per-block term uses a constant purely to rank modes.

2. Then bisect mu per frame against the 833,333-cycle budget, exactly as session 6 bisects lam against the byte budget. The machinery is already there and already gated: ratectl.encode_rate_controlled is frame-driven and feeds back the frame it emitted. But cycles have NO bucket. Bytes can be banked in the ring buffer; a frame that misses its decode deadline is just late, because there is no double buffer to decode ahead into. So this is a hard per-frame ceiling, not a leaky bucket — simpler than rate control, and the two controllers have to run together (raising mu moves blocks to SKIP and V1, which also lowers the bitrate, so the byte controller must see it).

3. Measure the quality cost. Everything session 6 did for bytes: what does fitting 100% of frames in the CPU budget cost in dB, and does any frame hit a cliff? tools/analysis/11_cpu_budget.py scores a container without needing MAME, so the search loop is cheap; confirm the winner on the 68000 with tools/bench/decode.lua.

3b. Know which misses are yours to fix before starting. Re-coding every non-SKIP block as V1 is the floor any mode assignment can reach, and it still misses 11 frames at sasi and 12 at scsi — every frame above ~90% non-SKIP. So the cost-aware decision can reach about three quarters of the misses (26 of 37 at sasi) and the rest are item 4. FINDINGS 28.7.

3c. Buy RAW, not V4, wherever the bytes allow. RAW is 400 cycles against V4's 448 and is pixel-exact, so on the CPU axis V4 is strictly dominated — the byte lagrangian's preference inverts. scsi can take that escape and sasi cannot afford it, so expect the cycle ceiling to cost sasi more quality even though it costs sasi fewer cycles. FINDINGS 28.8.

4. Scene cuts: 28.5 said impossible, 29.4 reopened it. An all-V1 frame — the cheapest full redraw the current mode set allows — is 110.5% of budget, so no mode assignment fits a 100%-changed frame. With literal spans the arithmetic changes: CPU needs at least 19% of the frame sent as spans, the bus allows up to 39%, and that interval is not empty. So 28.5 was a ceiling of the bitstream, not of the machine — if lever A measures out. If it does not, this is still a design decision that needs the user: one late frame at each cut (the outgoing content is unrelated, so it may be invisible), a cut spread over two frame times, or 10fps.

Do not start by hand-optimising decode.s. The hand-derived timings agree with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), so the inner loop is close to what the instruction set allows; the plausible wins are single-digit percentages against a 36-point gap. The V4 write pattern is the one place worth a look afterwards — pairing sub-block rows into movem.l d0/d2,(a4) saves ~16 of 448 cycles.

---

What session 7 settled

1. 68000 code parses a bitstream and draws frames, pixel-exact. src/player/decode.s + tools/bench/decode.lua. 120 frames of the Singe window decoded in sequence, all four block modes, verified against the new reference decoder tools/encoder/dlx.py. Because SKIP blocks are claims about the previous frame, the last frame is only right if all 120 were. In check.sh now. FINDINGS 28.
2. It does not fit. Mean 81.7% of a 12fps frame, p90 116.4%, worst 135.8%; 31% of frames miss at sasi, 42% at scsi. Zero-wait-state floor, as ever.
3. The dual-display-path plan (FINDINGS 24.5/25.6) is withdrawn as incoherent — the sixth false premise this project has caught. The compose path needs a RAM copy of the previous reconstruction; the direct path's whole selling point is that it keeps none. Mixing them displays stale pixels on 70 of 120 frames, worst frame 18.8% of the screen. Every coherent repair is worse than not mixing. tools/analysis/10_pathmix_drift.py, kept runnable as a counterexample and gated in check.sh. FINDINGS 28.1.
4. 24.5 also compared a copy against a copy. Its 53.6% and 76.6% both come from blit.s and neither includes decoding. Compose = decode-into-RAM plus the 53.6% blit, so it is strictly dearer than decoding into GVRAM. There was never a crossover. The player has one path and no reference frame, which also gives back 96 KB.
5. The four block modes cost 300 / 448 / 400 cycles, not one number. V4 is 1.49x a V1 block while the mode decision charges it 4x the bytes. The 24.5 model is 2.03x optimistic at the median. tools/analysis/11_cpu_budget.py reproduces all four frames timed on the 68000 to within 1 point. FINDINGS 28.2.
6. The container is big-endian but not aligned, and on a 68000 that is an address error, not a slow read. Frame records are variable-length and laid end to end, so their boundaries land on odd addresses. Frame 0 decoded perfectly, then the length read for frame 1 vectored into the IPL and sat there for 59 emulated seconds looking like an infinite loop. Found by dumping PC and the address registers — the code was right, the data layout was not. FINDINGS 28.3. Encoder gap: encode.py should pad records to 4 bytes. Measured cost 1.66 B/frame = 20 B/s against 110 KB/s.
7. A full frame does not fit at 12fps in any mode. All-V1 is 110.5%, all-V4 165.2%, all-RAW 147.6%. At most ~88% of the screen can change in one frame however cheaply it is coded, and scene cuts change 100%. FINDINGS 28.5.

---

What session 6 settled

1. Rate control works, is wired in, and is ON by default. encode.py bisects lam per frame under a leaky bucket; --fixed-lam restores session 5 behaviour. FINDINGS 27.
2. Both overshoots are closed for under 1 dB. On the Singe window, totals including audio: sasi 137.4 -> 109.5 KB/s (target 110) for -0.60 dB, scsi 381.6 -> 280.0 KB/s (target 280) for -0.91 dB. Zero frames hit the lam=800 cliff at either profile. FINDINGS 27.2.
3. The FINDINGS 26 desync is gone by construction, not by tuning. The encoder is frame-drivable (vq_hybrid.frame_ctx / decide / paint) and rate control feeds back the frame it actually emitted. The regression test tools/analysis/09_ratectl_drift.py goes 111/120 drifting frames -> 0, and it is now part of ./tools/bench/check.sh. FINDINGS 27.1.
4. Rate control makes the display path cheaper. Raising lam moves blocks to SKIP and V1, so there is less to write: scsi's median display cost drops 53.6% -> 47.1%. The decoder conclusion of 25.6 is unaffected. FINDINGS 27.3.
5. FINDINGS 26.5 was wrong in both halves, and this is the fifth false premise this project has caught. _paint was not the bottleneck (14% of a frame) and the ladder was never "minutes" (~18 s; the minutes were k-means in build). Vectorising it was still right — 17.1x — but what actually makes per-frame rate control affordable is that VQ.assign depends on neither lam nor prev, so it is cached: a 12-step search over 120 frames costs 0.31 s against 49.1 s. FINDINGS 27.6.
6. --prefill is a trap and defaults to 0. It buys a permission to overshoot of exactly bucket/nframes; at prefill=1.0 the Singe window goes to 116.3 KB/s against a 110 ceiling, and on a 14-frame clip it disables the controller outright. FINDINGS 27.4.
7. Fixed-lam sasi was already 5% over target on 00020, the clip everyone called easy — nothing noticed because the profile table quotes PSNR, not bitrate. FINDINGS 27.5.
8. 1.2-second clips cannot be used to judge rate control. The bucket's startup transient is bucket/nframes: 6% on a 10 s window, 20% on 00020. Same lesson as FINDINGS 25.3, different costume.

---

Start here: is the tree still green?

./tools/bench/check.sh

~3 min, needs the Blu-ray mounted. From source media it re-runs both display regression tests, the rate-control drift test (session 6), the display-path coherency counterexample and a 120-frame 68000 decode (session 7), then prints ALL GREEN. Verified green at end of session 7. If it fails, fix that before doing anything else — everything downstream assumes the display path is pixel-exact.

The two session-7 stages are worth knowing the shape of before they fail on you:

• 10_pathmix_drift.py is expected to exit non-zero; check.sh fails if it ever starts passing, because that would mean the counterexample behind the one-path decoder had stopped reproducing.
• the decode stage needs tmp/rc_fr_singe_sasi_rcprofile.dlx and will spend ~55 s encoding it if it is missing, nearly all of that k-means in H.build.

Decisions locked

decision	value	why
Target CPU	68000 @ 10MHz (stock)	hardest honest constraint
Display mode	256 colors, 256x192 in 256x256 CRTC mode	every mode is 1 word-access/pixel, so 256c is free vs 16c
Double buffer	none — page 1 sacrificed	enables movem.l 24px bursts; delta coding needs a RAM reference frame anyway
Codec	hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion	flat 4x4 VQ was measured and rejected — see FINDINGS 9-10
Quality modes	two: sasi and scsi (USER DECISION, session 2)	one codec, one decoder, one bitstream; only lam differs
Profile axis	I/O bandwidth only	the profiles say nothing about CPU; both target the same stock 10MHz 68000, and the Super has SCSI at 10MHz. FINDINGS 28.7
Framerate	12 fps, explicit decimation	source has zero duplicate frames; no free "twos" win
Emulator	MAME 0.277 x68000	accurate enough that measured cycles mean something
SNES project reuse	MIT — cleared	data/events/ scene graph is reusable with attribution

The SASI/SCSI question is RESOLVED

Session 1 left "which machine do we target" open. The user's answer: ship both, as two quality profiles. This is now implemented rather than hypothetical — the bitrate ceiling is a build parameter in tools/encoder/ratectl.py:

profile	target	lam	quality (00020 / 00146)	machine
sasi	110 KB/s	60 (floor)	36.9 / 29.6 dB	stock 10MHz ACE/EXPERT
scsi	280 KB/s	10 (floor)	39.4 / 32.3 dB	Super/XVI, or CZ-6BS1 board

That "machine" column is about the BUS, not the CPU. The profiles are an I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has built-in SCSI at 10 MHz (x68k.cpp:1194, 40_MHz_XTAL/4, same as the base machine), and only the XVI is faster. Both profiles target the same stock 10 MHz 68000, so both must fit the same 833,333-cycle frame budget — and as of session 7 neither does. FINDINGS 28.7.

As of session 6 lam is a floor, not a setting. The target is a ceiling and the encoder bisects lam per frame to stay under it; the profile's lam is the best quality it is allowed to spend on a quiet frame. On the worst sustained window that takes sasi from 137.4 to 109.5 KB/s and scsi from 381.6 to 280.0 KB/s, for -0.60 and -0.91 dB. FINDINGS 27.2.

Sized against the user's working figure of 4 Mbps = 488 KB/s sustained, on SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited constant, not an average over seek latency.

Both profiles fit with room. Ring-buffer simulation on the real per-frame sizes gives zero required prefill for every scene at both profiles: the fill delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance, far more than an SD-backed seek needs. FINDINGS 21.

An earlier warning here said scsi did not fit because a frame peaked at 96.4% of the pipe. That compared instantaneous demand to a sustained rate as if they had to match frame-by-frame; with a buffer the test is cumulative, and it passes.

scsi is now within 0.5 dB of the palette ceiling on 00020. These were initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the RD curve rather than derived from the hardware. See FINDINGS 17.

Codebooks are k=256 with 1-byte indices in both profiles. k=1024 was measured and rejected — see FINDINGS 14, it was a false-good result from a rate model that undercharged the index. Do not ship past lam~800; FINDINGS 15 has the cliff.

Because of the RAW escape mode, lam=0 is pixel-exact against the palettised frame (measured 0.00 dB loss). The profiles are two points on one continuous rate-distortion curve, not two codecs.

---

What session 3 settled

1. The display path works and is verified end to end. First real frame on an emulated X68000 screen: docs/images/x68k_first_frame_compare.png. Full write-up in FINDINGS 22. Everything before this session was Python-side or a headless -video none run, which cannot snapshot at all.
2. The render is pixel-exact, not merely close. With monitor contrast at 15, all 256 palette entries render exactly as GGGGGRRRRRBBBBBI + pal6bit predicts. That exactness is the regression test — see tools/bench/verify_frame.py, which exits non-zero if it ever drifts.
3. Three hardware facts that were previously assumed are now confirmed from MAME 0.277 source, not folklore: the palette word format, the 1024-byte GVRAM line stride, and the 256-colour page aliasing in HARDWARE.md. All three were already written down correctly; they are now cited.
4. A new quality ceiling was measured — the 15-bit+I palette alone costs 38.88 dB. Superseded by session 4: that figure assumed the shared LSB I is always 1. Chosen per entry, the ceiling is 40.81 dB. FINDINGS 23.3.
5. Two shell traps that wedged session 2's background jobs are documented in the working-setup section below. They cost ~1.5 h of wall clock and a wedged CPU core, and one of them was hit again this session.

What session 5 settled

1. 68000 code drew a frame, and the blit was measured. tools/bench/blit.s
   • blit.lua. The snapshot passes verify_frame256.py unchanged — pixel-exact in the real 256x256 mode. FINDINGS 23.5 is closed: no longer "proven from Lua only".
2. The 38% full-frame blit estimate is dead. It is 53.6%. And that is a zero-wait-state floor — MAME models no GVRAM wait states, so real hardware is worse. FINDINGS 24. Every variant was hand-derived from the MC68000 timing tables before being measured and the two agree to 0.006-0.43%, so this is not another MAME artefact.
3. Reading the source frame is exactly half the blit cost (V1 53.6% vs a write-only floor V3 of 27.1%). That is what makes the architecture question below live.
4. That number is now measured, and the answer is "implement both paths". On the worst sustained window found on the disc, 30% of frames (sasi) to 53% (scsi) sit above the 70% crossover and want the flat blit; the rest want direct-to-GVRAM. A player that picks per frame — the mode headers are parsed before any pixel is written, so the count is free — pays a median 37.0% and is capped at 53.6%. FINDINGS 25.6.
5. The sustained action sequence exists, was found by measurement, and breaks both profiles. tools/analysis/07_motion_survey.py scans a whole stream for the hottest sliding window; on 00223 it is t=539.4s, the Singe endgame, at 2.01x the stream mean. There, fixed-lam sasi overshoots 110 -> 129.6 KB/s (+18%) and scsi 280 -> 373.8 KB/s (+34%). Rate control is no longer insurance — it is required. FINDINGS 25.3.
6. The two largest streams on the disc are bonus material, not game footage. 00216 is the feature with a burned-in commentary PiP; 00215 is the commentary itself. 00223 (9.4 min) is the clean one. A size-ranked survey would have encoded live action. FINDINGS 25.1.
7. Rate control is unsound as written, caught before wiring it up. The lam-ladder in ratectl.py picks frames from independent temporal chains, so SKIP blocks reference reconstructions the decoder never saw: 111 of 120 frames drift, worst frame 43.4%, reported PSNR overstated 0.36 dB. Regression test tools/analysis/09_ratectl_drift.py. FINDINGS 26.
8. On hard content the scene palette, not the display, is the binding ceiling — 31.33 dB on the Singe window against 39.90 dB on 00020 and 40.81 dB for the X68000 display. scsi is already within 0.51 dB of it. FINDINGS 25.4.

Superseded within session 5

4a. The decoder architecture hinged on one unmeasured number. Writing codewords straight into GVRAM costs 76.6% of the frame budget for a full frame (V4 — the 1024-byte stride kills the movem.l burst), but scales with the non-SKIP block fraction and needs no RAM reference frame at all, because the previous frame is already in GVRAM. Compose-then-blit is a flat 53.6%. They cross at 70% of blocks changed. FINDINGS 24.5.

---

What session 4 settled

1. A real 256x256 CRTC mode exists and is verified. crtc_mode.lua, derived from x68k_crtc.cpp's divisor ladder rather than recalled — the derivation is self-checking (368 = 1104/3 exactly, so the horizontal registers divide by three with no remainder). Snapshot is native 256x512, active area pixel-exact, letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone.
2. The palette ceiling was wrong by 2 dB, in our favour. The shared LSB I must be chosen per palette entry, not hardcoded to 1. Doing so lifts the display ceiling from 38.85 to 40.81 dB and is the only way to get true black at all (pal6bit(1) = 4). 102 of 256 entries want I = 0. This supersedes FINDINGS 22.4 and gives scsi ~2 dB more headroom than believed. The encoder does not do this yet — see the encoder-gaps list.
3. Letterboxing costs one palette entry. 255 colours + a reserved black at index 0, with I = 0 on it. prep_frame.py --reserve-black. FINDINGS 23.4.
4. MAME's graphics double-scan is phase-shifted one raster line — pairs are (1,2),(3,4),..., not (0,1), because get_gfx_pixel halves the absolute scanline and vbegin = 41 is odd. Cost a false failure. FINDINGS 23.2.

---

What session 2 settled

1. The critical-path question is answered. "Does VQ soften Bluth's linework unacceptably?" — flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no. Verified by eye, not just PSNR. See docs/FINDINGS.md 9-11 and the two images in docs/images/. Both profiles use k=256; see item 2b.
2. Session 1's 12fps bitrate was wrong (183 KB/s claimed, 340 KB/s measured). Halving the framerate does not halve the bitrate. FINDINGS 8. 2b. A fourth false-good result was produced and caught this session — k=1024 codebooks looked like a +2.4 dB free win because the rate model charged 1 byte for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
3. The 256-colour palettised frame is the real quality ceiling and it looks excellent. Judge the codec against that, not against 1080p.
4. Encoder exists and produces a real bitstream: tools/encoder/.

---

Encoder — working

python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
python3 tools/encoder/encode.py  /tmp/fr_00020 out.dlx --profile sasi --preview p.png

file	role
extract.py	.m2ts -> 256x192 PNGs, 12fps, spatial-only denoise
vq.py	palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR
vq_hybrid.py	the codec: 4 block modes + lagrangian mode decision
ratectl.py	SASI/SCSI profiles, leaky-bucket rate control
encode.py	CLI + DLX1 container writer

DLX1 container layout is documented in the encode.py docstring. All multi-byte fields are big-endian so the 68000 reads them with a plain move.

Known encoder gaps

• Rate control is written but not yet wired into encode.py. DONE, session 6. It is on by default; --fixed-lam restores the old behaviour. Gated by tools/analysis/09_ratectl_drift.py, which is now in check.sh.
• Payload is deliberately NOT entropy-coded — deflate decode does not fit in the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
• Frame records are not aligned. They must be padded to a 4-byte boundary: unaligned is an ADDRESS ERROR on a 68000, not a slow read (FINDINGS 28.3). prep_dlx.py repairs it at load time, which a player streaming from disc cannot do. The pad is real bytes on disc, so it belongs inside the rate controller's accounting. 1.66 B/frame, 20 B/s.
• The mode decision is blind to CPU cost. It charges V4 four payload bytes and ignores that it costs 1.49x a V1 block to draw. This is the top item at the head of this file. FINDINGS 28.2.
• Palette packing is not implemented in the encoder. It still emits 24-bit palettes; the X68000 word packing happens Lua-side. Whatever writes real palette words must pick I per entry by minimum squared error (FINDINGS 23.3, worth 1.96 dB) and reserve index 0 as black with I = 0 (FINDINGS 23.4).
• Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
• _paint is a Python per-block loop. DONE, session 6 — vectorised, 17.1x. It was never the bottleneck, though: VQ.assign is 78% of a frame and H.build's k-means is 51 s of a 55 s run. That k-means is now the thing to attack before the full-disc survey, not anything in the per-frame path. FINDINGS 27.6.

---

Working setup (unchanged from session 1, re-verified)

MAME ROMs — ~/mame/roms/x68000.zip. Must pass -bios ipl10.

mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3

Assembler — tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s

Blu-ray — udisksctl loop-setup -r -f DRAGONS_LAIR.iso -> /media/reala-misaki/BDROM (still mounted as of end of session 2).

MAME Lua harness — tools/bench/*.lua, working. Three gotchas (retain the notifier subscription in a global; the stack register is SP not A7; autoboot_script fires at PC=0 before boot) are documented in FINDINGS.

Two shell traps, both hit again this session:

• piping MAME (or any long job) through grep block-buffers — write to a file.
• pkill -f <pattern> matches your own shell and kills it (exit 144). Use pkill -x or kill by PID.
• pgrep -f <name> | xargs kill kills your own shell too — exit 144. Same root cause as the pkill -f trap above: the shell's own command line contains the pattern. Hit again in session 5, which makes it four times across three sessions. Kill by PID captured at launch ($!), or use pkill -x.
• until ! pgrep -f foo.py; do sleep; done watcher loops never exit. The watching shell's own command line contains the string foo.py, so pgrep -f matches the watcher itself and the loop spins forever. Session 2 left 11 of these wedged for over an hour. Wait on the PID (while kill -0 $PID) or on a sentinel file the job touches when it finishes -- never on a -f name match.
• timeout N mame ... does not kill MAME. MAME catches SIGTERM and, with an autoboot script blocked waiting on a flag that never arrives, never reaches its shutdown path. timeout without -k then waits forever while MAME burns a full core at -nothrottle. Always timeout -k 5 N.

---

Disk throughput benchmark — still blocked, no longer gating

IOCS _B_READ returns -1 uniformly. Full diagnosis and the four untested hypotheses are in session 1's notes (git history of this file, commit 65112b9); the ordered plan for retrying is in docs/BENCHMARK.md.

Status changed twice this session — read this rather than the git history. It was briefly promoted to critical-path while the working bandwidth figure was misread as 4 MB/s. With the correct figure (4 Mbps = 488 KB/s) and the ring-buffer simulation showing zero required prefill for both profiles (FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is not available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a "measure it and maybe ship transparent" decision waiting.

What the benchmark is still worth doing for:

• Confirming the 4 Mbps figure. It is user-supplied and its provenance is not recorded. Every profile hangs off it.
• Confirming DMA is actually used. If transfers fall back to PIO the CPU cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the binding constraint. This is the worst plausible outcome and the cheapest to check — do it first.

Do not try to get the bandwidth number out of MAME. Its SCSI/SASI devices are functional models, not timing-accurate; a KB/s figure from MAME measures the emulator's scheduler. docs/BENCHMARK.md covers the three-tier approach (MAME validates the path, derivation bounds it, real hardware settles it).

Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5), by a 68000 DECODER (session 7).

The first real frame is on screen: docs/images/x68k_first_frame_compare.png.

Session 7 went from copying a frame to parsing one. src/player/decode.s reads DLX1, dispatches all four block modes and writes straight into GVRAM; 120 frames decoded in sequence are pixel-exact against tools/encoder/dlx.py (tools/bench/verify_decode.py, in check.sh). The blit numbers below are still correct for what they measured — a copy — but they are no longer the display-path budget: the decoder costs 300/448/400 cycles per V1/V4/RAW block and misses the 12fps budget on 31% of frames. FINDINGS 28.

Session 5 closed the gap this paragraph used to describe. GVRAM is now filled by 68000 instructions and the result is still pixel-exact, and the blit cost is measured rather than estimated: 53.6% of a 12fps frame, not 38% (FINDINGS 24). The remaining caveat is different and narrower: MAME models no GVRAM wait states, so 53.6% is a floor and real hardware is worse. Full write-up in FINDINGS 22. Harness: tools/bench/show_frame.lua + tools/bench/prep_frame.py.

Three facts the player MUST honour, none of which were guessable:

what	where	value
Un-hide the graphics layer	CRTC R20 $E80028	clear bit 11 ("G-VRAM set to buffer"); IPL leaves 0x0B16
Colour setup (256c)	CRTC R20 bits 9-8	0x0100
Monitor contrast	$E8E001 bits 3-0	IPL leaves 14; write 15 or everything renders 7% dark

The R20 = 0x0116 value quoted here in session 3 is the 768-wide IPL timing with the gate cleared. The shipping value is R20 = 0x0110 — see the mode table in tools/bench/crtc_mode.lua, which is now the single source of truth for all of R00-R08 and R20.

Bit 11 is the one that cost the most time: GVRAM writes land and read back correctly while the layer is invisible, so the video controller looks guilty and is not. Contrast 0 blanks the screen — free fade-to-black for transitions.

Palette format is now confirmed from MAME source, not assumed: GGGGGRRRRRBBBBBI (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as pal6bit((field<<1)|I). With contrast at 15 the render is pixel-exact.

Ceiling: the 15-bit+I palette costs 40.81 dB against the 24-bit palettised source, once I is chosen per entry (FINDINGS 23.3 — session 3's 38.88 dB assumed I = 1). Still the same order as the scsi profile's own codec error (39.4 dB), so scsi remains near display-transparent, with ~2 dB more headroom than session 3 thought.

Snapshot recipe that works (-video none CANNOT snapshot):

SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
  -sound none -nothrottle -plugins -autoboot_script <script>.lua \
  -snapshot_directory ./snap -snapview native -seconds_to_run 6

-snapview native drops MAME's LED artwork and gives a clean 768x512 screen.

Next steps, in priority order

1. Measure the non-SKIP block fraction. DONE, session 5, and its answer WITHDRAWN in session 7 — FINDINGS 28.1/28.2. It concluded "implement both display paths and pick per frame, median 37.0%, capped at 53.6%". Mixing the paths is incoherent (the compose path needs a RAM reference the direct path never writes) and the two costs it compared were both copies with no decode in either. The shipping decoder has one path. The non-SKIP fraction is still reported by encode.py and is still the right thing to look at — it is just no longer a switch. Original framing kept below, because its instruction to report the distribution rather than the mean is the part that held up: FINDINGS 24.5: compose-in-RAM-then-blit costs a flat 53.6% of the frame budget; decode-direct-to-GVRAM costs 76.6% x (fraction of blocks that are not SKIP) and needs no RAM reference frame. They cross at 70%. Which side of 70% the content sits on decides which decoder inner loop to write, so this must come before writing one. It needs no new machinery — the mode decision in vq_hybrid.py already computes it per frame and simply never reports it. Add the histogram (SKIP / V1 / V4 / RAW counts per frame) to encode.py output and run it over the clips already extracted. Report the distribution, not the mean: a scene-cut frame is ~100% non-SKIP and a held frame near 0%, and the mean of those two is a number describing no actual frame.

1b. Wire rate control into encode.py. DONE, session 6. FINDINGS 27. Both overshoots closed for under 1 dB, drift test at zero, check.sh gates it. The remaining rate-control question is not a defect: whether --rc-floor open is worth taking on quiet content. It measured as worth 0.00 dB on the Singe window (no frame there is quiet enough to saturate the bucket), so it needs a genuinely quiet scene to decide, and it is a quality-per-byte judgement rather than a correctness one.

2. 68000 decoder skeleton. DONE, session 7. src/player/decode.s, pixel-exact over 120 frames, gated in check.sh. It answered the question it was written to answer, and the answer is no: it does not fit — mean 81.7% of a 12fps frame, 31% of frames over 100%. FINDINGS 28. The follow-on is priority 0 at the top of this file.

2a. Re-budget everything against the MEASURED per-mode costs, not 53.6% and not 38%. Session 7 replaced the model twice over (FINDINGS 28.2): the display path is not one number times a block fraction, and the median frame is 74.4% rather than 36.6%. The original note is kept below because its warning about downstream figures derived from a dead estimate is exactly what happened again. Re-budget everything against 53.6%, not 38%. Several downstream figures were derived from the old estimate. The blit alone now eats over half the frame at 12fps in the compose-then-blit design, before any decode, and MAME models no GVRAM wait states so that is a floor. This may reopen questions that were closed against the 38% number — check FINDINGS 17.2's entropy-coding rejection, which was argued as "54% LZ4 with no room beside a 38% blit". The conclusion gets stronger, not weaker, but the arithmetic should be restated.

2b. Pad frame records to 4 bytes in encode.py. Not optional: unaligned records are an address error on a 68000 (FINDINGS 28.3), and prep_dlx.py currently repairs it at load time, which the shipping player streaming from disc cannot do. The padding is real bytes on disc, so it has to be inside the rate controller's accounting, not added after it. 20 B/s at 12fps.

3. Full-disc survey. Now scoped by session 5 rather than open-ended: the worst sustained window is measured (FINDINGS 25), so what remains is the distribution over content, not the worst case.

   • Classify content / menu / bonus — not just menu vs content. FINDINGS 25.1: the two largest streams are bonus material and look like content by size, duration and bitrate alike.
   • Run tools/analysis/07_motion_survey.py per stream first; it is cheap (96x72 greyscale) and gives a hot-window shortlist so the expensive encode only runs where it matters.
   • Vectorise _paint before this run. Done. The cost to attack now is H.build's k-means: 51 s of a 55 s run, and it runs once per scene.
   • Do it after rate control (1b), or it measures an encoder nobody ships. Rate control is in, so the survey now measures the shipping encoder.

4. Confirm DMA vs PIO in MAME (see the benchmark section above) — cheap, and the only thing that could still move CPU into the binding position.

5. Resolve the framing question (FINDINGS 12: crop vs squash vs wide). Needs an eyeball against arcade reference, not a measurement.

6. Import the scene graph. SNES project data/events/ (MIT, cleared), cross-checked against DirkSimple (zlib) which transcribed the same data independently — diff them to catch transcription errors before committing any of it to 68000 tables.

7. ADPCM audio. MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in ratectl.py, not yet extracted or encoded.

Explicitly abandoned — do not re-propose

• Entropy-code the payload. Deflate decode is ~216% of the frame budget on a 68000; LZ4 is ~54% with no room beside a 38% blit (FINDINGS 17.2). All bitrates are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8 to a compression upper bound, not a shippable design.
• k=1024 codebooks. False-good result from a rate model that charged 1 byte for a 10-bit index (FINDINGS 14). k=256 wins at every matched bitrate.
• Flat 4x4 VQ. Rejected by eye (FINDINGS 9).

Not yet started

• A player, as opposed to a decoder. src/player/decode.s parses DLX1, dispatches all four block modes and draws pixel-exact frames, but it decodes from RAM that Lua pre-loaded. There is no disc streaming, no ring buffer, no audio, no timing against the VBL, and no scene branching.
• Codebook expansion on the 68000. prep_dlx.py does it host-side because it is a load-time cost and including it would flatter or damn the inner loop. The player must do it: 8 KB + 2 KB per scene.
• ADPCM audio extraction/encoding
• Disk image packaging
• Game logic (scene branching, input windows, death clips)

Reproducing the 256x256 mode result (session 4)

python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
mkdir -p tmp/snap256 && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 90 mame x68000 \
  -bios ipl10 -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/show_frame256.lua \
  -snapshot_directory ./snap256 -snapview native -seconds_to_run 6
cd .. && python3 tools/bench/verify_frame256.py

Exits non-zero on any drift. Expected: 256x512 native, double-scan exact, active 256x192 pixel-exact, letterbox true black, ceiling 40.81 dB.

Reproducing the display result

python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame.bin 0
mkdir -p tmp/snap_verify && cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 \
  -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/show_frame.lua \
  -snapshot_directory ./snap_verify -snapview native -seconds_to_run 6
cd .. && python3 tools/bench/verify_frame.py

Verified cold from the Blu-ray at end of session 3: exact match, 38.88 dB. (That 38.88 is correct for this test: show_frame.lua still packs I = 1. The 40.81 dB ceiling comes from show_frame256.lua, which picks I per entry.)

tmp/ is gitignored scratch. The frames are NOT in the repo — regenerate them with extract.py; the earlier ones lived in /tmp and do not survive a reboot.

Reference material on this box (not in the repo)

• MAME 0.277 source: ~/src/mame-mame0277/ (tarball ~/src/mame0277.tar.gz). Downloaded this session to settle the graphics-layer question. The files that matter are src/mame/sharp/x68k_v.cpp, x68k_crtc.cpp, x68k_crtc.h, x68k.cpp. Read these before theorising about X68000 video behaviour — six register-poking attempts failed against a gate that one grep found.
• Blu-ray mounted at /media/reala-misaki/BDROM via udisksctl loop-setup -r -f DRAGONS_LAIR.iso.

---

Parked ideas (not scheduled, not abandoned)

• Cliff Hanger, retitled as Lupin III (user, session 4). Stern's 1983 laserdisc game was cut from Castle of Cagliostro and Mystery of Mamo with the Lupin branding stripped; a port would restore it. Technically cheaper than this project: same content class (cel animation, flat colour, hard cuts), ~13 min of footage vs Dragon's Lair's ~22, and flatter linework than Bluth's, so fewer blocks should escape to V4/RAW. The codec, the display path, and crtc_mode.lua would all drop straight in. The real cost is media prep, not code: there is no clean master cut to Stern's scene boundaries the way DRAGONS_LAIR.iso is, so the footage would have to be sourced and cut to match. Not to be started until the CPU path is proven — it changes nothing about whether this design works.

Reproducing the blit measurement (session 5)

python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
tools/vasm/vasmm68k_mot -Fbin -o tmp/blit.bin tools/bench/blit.s
mkdir -p tmp/snap_blit && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
  -bios ipl10 -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/blit.lua \
  -snapshot_directory ./snap_blit -snapview native -seconds_to_run 120

~25 s wall. Prints cycles/frame and % of a 12fps budget for V1-V4, and snapshots V1's output. To check that snapshot is still pixel-exact: sed 's|snap256|snap_blit|' tools/bench/verify_frame256.py | python3 -

Not added to check.sh: check.sh asserts pixel-exactness, and asserting wall timings there would make the green-light check sensitive to host load.

Reproducing the decoder result (session 7)

python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_fr_singe_sasi_rcprofile.dlx --profile sasi
python3 tools/bench/prep_dlx.py tmp/rc_fr_singe_sasi_rcprofile.dlx
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s
mkdir -p tmp/snap_decode && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
  -bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
  -autoboot_script ../tools/bench/decode.lua \
  -snapshot_directory ./snap_decode -snapview native -seconds_to_run 150
cd .. && python3 tools/bench/verify_decode.py tmp/rc_fr_singe_sasi_rcprofile.dlx

~90 s wall. Prints cycles/frame and % of a 12fps budget for four real frames spanning the non-SKIP distribution, four synthetic single-mode frames, and one full 120-frame pass; then verifies the last frame is pixel-exact. Expected: median 73.8%, p90 116.4%, max 135.8%, mean 81.7%; V1 299.9 / V4 448.2 / RAW 400.4 cycles per block.

-ramsize 2M matters — MAME defaults to 4M and the locked target is a stock 2MB machine. DLX_VERIFY_ONLY=1 drops the timing anchors, which is how check.sh runs it.

Score a container against the measured costs without touching MAME:

python3 tools/analysis/11_cpu_budget.py tmp/rc_fr_singe_scsi_rcprofile.dlx

And re-demonstrate why there is only one display path (exits non-zero by design — it is the counterexample):

python3 tools/analysis/10_pathmix_drift.py                 # 70/120 frames corrupt
python3 tools/analysis/10_pathmix_drift.py --fix direct    # clean, and cheapest

Reproducing the rate-control result (session 6)

python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
for prof in sasi scsi; do
  python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof --fixed-lam
  python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof
done
python3 tools/analysis/09_ratectl_drift.py       # must exit 0, zero drifting frames

Expected, totals including the 7.8 KB/s audio allowance: sasi 137.4 -> 109.5 KB/s and 27.82 -> 27.22 dB; scsi 381.6 -> 280.0 KB/s and 30.81 -> 29.90 dB; zero frames at the lam=800 cliff in either. ~55 s per encode, nearly all of it k-means in H.build.

The block-mode map now renders the rate-controlled encoder by default:

python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes_rc.webm \
        --profile sasi --scale 2                 # add --fixed-lam to compare

Do not judge rate control on tmp/fr_00020. It is 14 frames; the leaky bucket's startup transient is bucket/nframes, so it lands 18% under target there for reasons that have nothing to do with the content. FINDINGS 27.5.

Reproducing the sustained-action result (session 5)

python3 tools/analysis/07_motion_survey.py 00223 10        # -> hottest window t=539.4s
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_sasi.dlx --profile sasi
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_scsi.dlx --profile scsi
python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes.webm \
        --profile sasi --scale 2

extract.py now takes optional [start_s] [dur_s] — needed because 00223 is 9.4 min and the windows that stress the codec are seconds long.

08_mode_map.py renders palettised source | decoded | block-mode map at 12fps. Output format follows the extension; prefer .webm — GIF re-quantises to 256 colours, which is a poor fit for output whose subject is colour fidelity, and runs larger. It uses yuv444p because the mode map is flat saturated colour on a 4-pixel grid and chroma subsampling smears exactly those edges.