# Status & next-session handoff — end of session 8 (2026-08-23) ## Where this stands Session 7 left the decoder pixel-exact and **31% of frames over the CPU budget** at `sasi`, 42% at `scsi` (FINDINGS 28), with two levers proposed and neither measured. Session 8 did both. **Lever A, spans: measured.** A row-linear span of word-expanded literals costs **43.7 cycles per span + 9.152 per pixel** — but only in an encoder-assisted format, `{u32 absolute GVRAM address, u16 jump displacement}` into an unrolled copy chain. The obvious decoder, handed `(x, npix)`, is 97.9 + 10.46. FINDINGS 30, `tools/bench/span.sh` (~25 s). **Lever B, the cost-aware mode decision: implemented, measured, and ON by default.** `decide()` minimises `D + lam*bytes + mu*cycles`; `mu` is bisected per frame against 833,333 cycles with the `lam` bisection nested inside it. FINDINGS 31, `tools/analysis/13_cpu_ratectl.py`. | `sasi`, worst sustained window | PSNR | KB/s | CPU median | CPU max | missing | |---|---:|---:|---:|---:|---:| | bytes only (session 7) | 27.22 dB | 109.5 | 74.4% | 136.2% | **37/120** | | + cycle ceiling (now the default) | 26.95 dB | 109.4 | 81.5% | 110.6% | **1/120** | | + ceiling + spans (MODEL, nothing implements it) | — | 449.3 | 56.8% | 91.5% | **0/120** | `scsi`: 51/120 -> 1/120 for 0.62 dB. The one remaining miss at either profile is **frame 0**, which has no previous reconstruction and so is 100% changed by definition — the same case as a scene cut. It is emitted late on purpose. The cost-aware container is verified pixel-exact on the 68000 (120 frames, `tools/bench/verify_decode.py`). ## NEXT SESSION, in order 0. **Green light first.** `./tools/bench/check.sh` (~3 min, Blu-ray mounted). The drift stage now runs BOTH controllers; both must report 0/120. 1. **Time the cost-aware container on the 68000.** Everything in FINDINGS 31 is the validated cost MODEL (`vq_hybrid.cycles`, within 1 point of the machine on four frames of the session-7 container), not a measurement of this one. The full timing pass was started and killed for time: ``` python3 tools/bench/prep_dlx.py tmp/rc_fr_singe_sasi_cpufit.dlx tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s ( cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -ramsize 2M \ -video soft -window -sound none -nothrottle -plugins \ -autoboot_script ../tools/bench/decode.lua -snapshot_directory ./snap_decode \ -snapview native -seconds_to_run 300 > decode_cpufit.log 2>&1 ) ``` **Budget real time for it: the run was still going at 12 minutes of CPU.** MAME's stdout is block-buffered to a file, so there is no progress to watch — wait on the PID, never on a `pgrep -f` match (see the shell traps below). Confirm the four anchors against `11_cpu_budget.py` on the same container, and update FINDINGS 31 with measured-vs-model errors. 2. **Put spans in the bitstream.** This is the big one and it is now fully specified by measurement: format in FINDINGS 30.2, costs in 30.5, and the scene-cut arithmetic in 30.6. It touches `encode.py` (a fifth mode and a run-aware decision), `dlx.py` (the reference decoder), and `decode.s`. The 24-pixel quantisation and the free row overrun are part of the format, not optimisations to add later. Order it AFTER item 1 so the model that prices it has been checked against the machine once more. 3. **The three open items of FINDINGS 29.5/30.7**, now load-bearing because a span design runs at ~449 KB/s of a 488 KB/s pipe: re-run the ring-buffer simulation at that rate (FINDINGS 21 was established at 110 and 280), confirm the provenance of the user's 4 Mbps figure, and **confirm DMA rather than PIO** — a PIO fallback puts a 449 KB/s transfer back on the CPU the whole lever exists to relieve. The DMA check is the cheapest of the three and the most consequential. 4. **Encoder gap, still open from session 7:** `encode.py` should pad frame records to 4 bytes. Frame boundaries land on odd addresses and a 68000 takes an address error, not a slow read (FINDINGS 28.3). `prep_dlx.py` pads at load time, which is why the decoder works; the container itself does not. Measured cost of fixing it: 1.5 B/frame = 18 B/s. 5. **A quality-vs-framerate question that is the user's, not the encoder's.** Every miss is now one frame per cut. The options remain: one late frame at each cut (the outgoing content is unrelated, so it may be invisible), a cut spread over two frame times, or 10fps. Spans (item 2) make the question go away if they land as modelled. **Do not start by hand-optimising `decode.s`.** The hand-derived timings agree with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), so the inner loop is close to what the instruction set allows. --- ## What session 8 settled 0. **The mode decision can see cycles, it is on by default, and it costs 0.26 dB.** `decide(ctx, lam, mu)` minimises `D + lam*bytes + mu*cycles`; `ratectl` bisects mu per frame against a HARD 833,333-cycle ceiling (bytes bank in the ring buffer, cycles cannot — there is no double buffer to decode ahead into). `sasi` 37/120 misses -> 1, `scsi` 51 -> 1. Bitrate does not move: mu changes which modes are bought, not how many bytes. FINDINGS 31, `tools/analysis/13_cpu_ratectl.py`. 0b. **28.7's "11 frames are impossible" was too pessimistic — it is 1.** That floor held the SKIP set fixed; the real decision can also move a block to SKIP, which above ~90% non-SKIP is the only lever left. FINDINGS 31.3. 0c. **V4 collapses when cycles are priced**, as 28.8 predicted: 25.2 -> 20.3% of blocks at `sasi` and **15.0 -> 5.3%** at `scsi`, where RAW takes it. RAW is dearer in bytes and cheaper in cycles, so the byte lagrangian's preference inverts and only the byte-rich profile can take the escape. 0d. **SKIP's price depends on its neighbours, and the way out is two cost functions**: a ranking constant inside the per-block lagrangian, the exact clustered rule (`vq_hybrid.cycles`, validated to 1 point against the 68000) for the frame-level bisection. That function is now defined once and imported by `11_cpu_budget.py`. FINDINGS 31.4. 0e. **Both controllers are gated against decoder drift.** `09_ratectl_drift.py` runs bytes-only AND bytes+cycles; both 0/120. 1. **The span is measured: 43.7 cycles/span + 9.152/pixel, fitted to 0.3% over eleven span lengths.** `tools/bench/blit.s` v5/v6, `prep_spans.py`, `span.lua`, driven by `tools/bench/span.sh` (~25 s, not in `check.sh` because it is a wall timing). FINDINGS 30. 2. **Only in an encoder-assisted format.** `{u32 absolute GVRAM address, u16 jump displacement}` into an unrolled chain, versus `(x, npix)` and a decoder that works it out: 43.7 + 9.152 against 97.9 + 10.46. All the arithmetic a span decoder would do per frame is known at encode time. FINDINGS 30.2. 3. **The per-pixel cost is a function of register pressure**, which FINDINGS 24 could not have shown: 9.08 was a fixed blit with 12 registers free, v5 can spare 8 and pays 10.46, v6 gets 12 back by making the encoder hold the state. 4. **Short spans die in the remainder path, and the fix is padding.** A 12-pixel span costs more than a 16-pixel one in v5. v6 has no remainder path: lengths are multiples of 24 pixels, padding is free of everything but bytes, and an overrun past the visible 256 lands in the invisible half of the 1024-byte line stride. FINDINGS 30.3. 5. **Odd-`x` alignment is free** (259.0 vs 261.8 cycles/span) — expected on a 16-bit bus, now measured rather than assumed. 6. **The trade is smaller than 29 derived but the conclusion holds**, including 29.4's reopening of the scene cut. All 23 timing configs also drew a pixel-exact frame, so nothing here was timed against a decoder that skipped work. FINDINGS 30.5/30.6. --- ## What session 7 settled 1. **68000 code parses a bitstream and draws frames, pixel-exact.** `src/player/decode.s` + `tools/bench/decode.lua`. 120 frames of the Singe window decoded in sequence, all four block modes, verified against the new reference decoder `tools/encoder/dlx.py`. Because SKIP blocks are claims about the previous frame, the last frame is only right if all 120 were. In `check.sh` now. **FINDINGS 28.** 2. **It does not fit.** Mean 81.7% of a 12fps frame, p90 116.4%, worst 135.8%; 31% of frames miss at `sasi`, 42% at `scsi`. Zero-wait-state floor, as ever. 3. **The dual-display-path plan (FINDINGS 24.5/25.6) is withdrawn as incoherent — the sixth false premise this project has caught.** The compose path needs a RAM copy of the previous reconstruction; the direct path's whole selling point is that it keeps none. Mixing them displays stale pixels on **70 of 120 frames**, worst frame 18.8% of the screen. Every coherent repair is worse than not mixing. `tools/analysis/10_pathmix_drift.py`, kept runnable as a counterexample and gated in `check.sh`. FINDINGS 28.1. 4. **24.5 also compared a copy against a copy.** Its 53.6% and 76.6% both come from `blit.s` and neither includes decoding. Compose = decode-into-RAM *plus* the 53.6% blit, so it is strictly dearer than decoding into GVRAM. There was never a crossover. The player has **one path and no reference frame**, which also gives back 96 KB. 5. **The four block modes cost 300 / 448 / 400 cycles, not one number.** V4 is 1.49x a V1 block while the mode decision charges it 4x the bytes. The 24.5 model is 2.03x optimistic at the median. `tools/analysis/11_cpu_budget.py` reproduces all four frames timed on the 68000 to within 1 point. FINDINGS 28.2. 6. **The container is big-endian but not aligned, and on a 68000 that is an address error, not a slow read.** Frame records are variable-length and laid end to end, so their boundaries land on odd addresses. Frame 0 decoded perfectly, then the length read for frame 1 vectored into the IPL and sat there for 59 emulated seconds looking like an infinite loop. Found by dumping PC and the address registers — the code was right, the data layout was not. FINDINGS 28.3. **Encoder gap: `encode.py` should pad records to 4 bytes.** Measured cost 1.66 B/frame = 20 B/s against 110 KB/s. 7. **A full frame does not fit at 12fps in any mode.** All-V1 is 110.5%, all-V4 165.2%, all-RAW 147.6%. At most ~88% of the screen can change in one frame however cheaply it is coded, and scene cuts change 100%. FINDINGS 28.5. --- ## What session 6 settled 1. **Rate control works, is wired in, and is ON by default.** `encode.py` bisects lam per frame under a leaky bucket; `--fixed-lam` restores session 5 behaviour. FINDINGS 27. 2. **Both overshoots are closed for under 1 dB.** On the Singe window, totals including audio: `sasi` 137.4 -> **109.5 KB/s** (target 110) for -0.60 dB, `scsi` 381.6 -> **280.0 KB/s** (target 280) for -0.91 dB. Zero frames hit the lam=800 cliff at either profile. FINDINGS 27.2. 3. **The FINDINGS 26 desync is gone by construction, not by tuning.** The encoder is frame-drivable (`vq_hybrid.frame_ctx` / `decide` / `paint`) and rate control feeds back the frame it actually emitted. The regression test `tools/analysis/09_ratectl_drift.py` goes 111/120 drifting frames -> **0**, and it is now part of `./tools/bench/check.sh`. FINDINGS 27.1. 4. **Rate control makes the display path cheaper.** Raising lam moves blocks to SKIP and V1, so there is less to write: `scsi`'s median display cost drops 53.6% -> 47.1%. The decoder conclusion of 25.6 is unaffected. FINDINGS 27.3. 5. **FINDINGS 26.5 was wrong in both halves, and this is the fifth false premise this project has caught.** `_paint` was not the bottleneck (14% of a frame) and the ladder was never "minutes" (~18 s; the minutes were k-means in `build`). Vectorising it was still right — 17.1x — but what actually makes per-frame rate control affordable is that `VQ.assign` depends on neither `lam` nor `prev`, so it is cached: a 12-step search over 120 frames costs **0.31 s** against 49.1 s. FINDINGS 27.6. 6. **`--prefill` is a trap and defaults to 0.** It buys a permission to overshoot of exactly bucket/nframes; at prefill=1.0 the Singe window goes to 116.3 KB/s against a 110 ceiling, and on a 14-frame clip it disables the controller outright. FINDINGS 27.4. 7. **Fixed-lam `sasi` was already 5% over target on 00020**, the clip everyone called easy — nothing noticed because the profile table quotes PSNR, not bitrate. FINDINGS 27.5. 8. **1.2-second clips cannot be used to judge rate control.** The bucket's startup transient is bucket/nframes: 6% on a 10 s window, 20% on 00020. Same lesson as FINDINGS 25.3, different costume. --- ## Start here: is the tree still green? ``` ./tools/bench/check.sh ``` ~4 min, needs the Blu-ray mounted. From source media it re-runs both display regression tests, the rate-control drift test (session 6, now covering BOTH controllers -- bytes, and bytes+cycles), the display-path coherency counterexample and a **120-frame 68000 decode** (session 7), then prints `ALL GREEN`. Verified green at end of session 8. Do not run two of these at once, and do not run one alongside a MAME timing job: they share `tmp/` snapshot directories and log files, and the second run silently truncates the first one's output. If it fails, fix that before doing anything else — everything downstream assumes the display path is pixel-exact. The two session-7 stages are worth knowing the shape of before they fail on you: - `10_pathmix_drift.py` is expected to exit **non-zero**; `check.sh` fails if it ever starts passing, because that would mean the counterexample behind the one-path decoder had stopped reproducing. - the decode stage needs `tmp/rc_fr_singe_sasi_rcprofile.dlx` and will spend ~55 s encoding it if it is missing, nearly all of that k-means in `H.build`. ## Decisions locked | decision | value | why | |---|---|---| | Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint | | Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c | | Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway | | **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 | | **Quality modes** | **two: `sasi` and `scsi`** (USER DECISION, session 2) | one codec, one decoder, one bitstream; only `lam` differs | | Profile axis | **I/O bandwidth only** | the profiles say nothing about CPU; **both target the same stock 10MHz 68000**, and the Super has SCSI at 10MHz. FINDINGS 28.7 | | Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win | | Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something | | SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution | ### The SASI/SCSI question is RESOLVED Session 1 left "which machine do we target" open. The user's answer: **ship both**, as two quality profiles. This is now implemented rather than hypothetical — the bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`: | profile | target | lam | quality (00020 / 00146) | machine | |---|---|---|---|---| | `sasi` | 110 KB/s | 60 (floor) | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT | | `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board | **That "machine" column is about the BUS, not the CPU.** The profiles are an I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has built-in SCSI at 10 MHz (`x68k.cpp:1194`, `40_MHz_XTAL/4`, same as the base machine), and only the XVI is faster. **Both profiles target the same stock 10 MHz 68000**, so both must fit the same 833,333-cycle frame budget — and as of session 7 neither does. FINDINGS 28.7. **As of session 6 `lam` is a floor, not a setting.** The target is a ceiling and the encoder bisects lam per frame to stay under it; the profile's lam is the best quality it is allowed to spend on a quiet frame. On the worst sustained window that takes `sasi` from 137.4 to 109.5 KB/s and `scsi` from 381.6 to 280.0 KB/s, for -0.60 and -0.91 dB. FINDINGS 27.2. Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**, not an average over seek latency. **Both profiles fit with room.** Ring-buffer simulation on the real per-frame sizes gives **zero required prefill** for every scene at both profiles: the fill delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance, far more than an SD-backed seek needs. FINDINGS 21. An earlier warning here said `scsi` did not fit because a frame peaked at 96.4% of the pipe. That compared instantaneous demand to a sustained rate as if they had to match frame-by-frame; with a buffer the test is cumulative, and it passes. `scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the RD curve rather than derived from the hardware. See FINDINGS 17. Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured and rejected — see FINDINGS 14, it was a false-good result from a rate model that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff. Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised frame (measured 0.00 dB loss). The profiles are two points on one continuous rate-distortion curve, not two codecs. --- ## What session 3 settled 1. **The display path works and is verified end to end.** First real frame on an emulated X68000 screen: `docs/images/x68k_first_frame_compare.png`. Full write-up in **FINDINGS 22**. Everything before this session was Python-side or a headless `-video none` run, which cannot snapshot at all. 2. **The render is pixel-exact, not merely close.** With monitor contrast at 15, all 256 palette entries render exactly as `GGGGGRRRRRBBBBBI` + `pal6bit` predicts. That exactness is the regression test — see `tools/bench/verify_frame.py`, which exits non-zero if it ever drifts. 3. **Three hardware facts that were previously assumed are now confirmed from MAME 0.277 source**, not folklore: the palette word format, the 1024-byte GVRAM line stride, and the 256-colour page aliasing in `HARDWARE.md`. All three were already written down correctly; they are now cited. 4. ~~**A new quality ceiling was measured** — the 15-bit+I palette alone costs 38.88 dB.~~ **Superseded by session 4:** that figure assumed the shared LSB `I` is always 1. Chosen per entry, the ceiling is **40.81 dB**. FINDINGS 23.3. 5. **Two shell traps that wedged session 2's background jobs** are documented in the working-setup section below. They cost ~1.5 h of wall clock and a wedged CPU core, and one of them was hit again this session. ## What session 5 settled 1. **68000 code drew a frame, and the blit was measured.** `tools/bench/blit.s` + `blit.lua`. The snapshot passes `verify_frame256.py` unchanged — pixel-exact in the real 256x256 mode. **FINDINGS 23.5 is closed**: no longer "proven from Lua only". 2. **The 38% full-frame blit estimate is dead. It is 53.6%.** And that is a zero-wait-state floor — MAME models no GVRAM wait states, so real hardware is worse. FINDINGS 24. Every variant was hand-derived from the MC68000 timing tables before being measured and the two agree to 0.006-0.43%, so this is not another MAME artefact. 3. **Reading the source frame is exactly half the blit cost** (V1 53.6% vs a write-only floor V3 of 27.1%). That is what makes the architecture question below live. 4. **That number is now measured, and the answer is "implement both paths".** On the worst sustained window found on the disc, 30% of frames (`sasi`) to 53% (`scsi`) sit above the 70% crossover and want the flat blit; the rest want direct-to-GVRAM. A player that picks per frame — the mode headers are parsed before any pixel is written, so the count is free — pays a **median 37.0%** and is **capped at 53.6%**. FINDINGS 25.6. 5. **The sustained action sequence exists, was found by measurement, and breaks both profiles.** `tools/analysis/07_motion_survey.py` scans a whole stream for the hottest sliding window; on 00223 it is t=539.4s, the Singe endgame, at 2.01x the stream mean. There, fixed-lam `sasi` overshoots 110 -> 129.6 KB/s (+18%) and `scsi` 280 -> 373.8 KB/s (+34%). **Rate control is no longer insurance — it is required.** FINDINGS 25.3. 6. **The two largest streams on the disc are bonus material, not game footage.** 00216 is the feature with a burned-in commentary PiP; 00215 is the commentary itself. **00223 (9.4 min) is the clean one.** A size-ranked survey would have encoded live action. FINDINGS 25.1. 7. **Rate control is unsound as written, caught before wiring it up.** The lam-ladder in `ratectl.py` picks frames from independent temporal chains, so SKIP blocks reference reconstructions the decoder never saw: 111 of 120 frames drift, worst frame 43.4%, reported PSNR overstated 0.36 dB. Regression test `tools/analysis/09_ratectl_drift.py`. FINDINGS 26. 8. **On hard content the scene palette, not the display, is the binding ceiling** — 31.33 dB on the Singe window against 39.90 dB on 00020 and 40.81 dB for the X68000 display. `scsi` is already within 0.51 dB of it. FINDINGS 25.4. ### Superseded within session 5 4a. **The decoder architecture hinged on one unmeasured number.** Writing codewords straight into GVRAM costs 76.6% of the frame budget for a *full* frame (V4 — the 1024-byte stride kills the `movem.l` burst), but scales with the non-SKIP block fraction and needs **no RAM reference frame at all**, because the previous frame is already in GVRAM. Compose-then-blit is a flat 53.6%. **They cross at 70% of blocks changed.** FINDINGS 24.5. --- ## What session 4 settled 1. **A real 256x256 CRTC mode exists and is verified.** `crtc_mode.lua`, derived from `x68k_crtc.cpp`'s divisor ladder rather than recalled — the derivation is self-checking (368 = 1104/3 exactly, so the horizontal registers divide by three with no remainder). Snapshot is native 256x512, active area pixel-exact, letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone. 2. **The palette ceiling was wrong by 2 dB, in our favour.** The shared LSB `I` must be chosen **per palette entry**, not hardcoded to 1. Doing so lifts the display ceiling from 38.85 to **40.81 dB** and is the only way to get true black at all (`pal6bit(1) = 4`). 102 of 256 entries want `I = 0`. This supersedes FINDINGS 22.4 and gives `scsi` ~2 dB more headroom than believed. **The encoder does not do this yet** — see the encoder-gaps list. 3. **Letterboxing costs one palette entry.** 255 colours + a reserved black at index 0, with `I = 0` on it. `prep_frame.py --reserve-black`. FINDINGS 23.4. 4. **MAME's graphics double-scan is phase-shifted one raster line** — pairs are (1,2),(3,4),..., not (0,1), because `get_gfx_pixel` halves the *absolute* scanline and `vbegin = 41` is odd. Cost a false failure. FINDINGS 23.2. --- ## What session 2 settled 1. **The critical-path question is answered.** "Does VQ soften Bluth's linework unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.** Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two images in `docs/images/`. Both profiles use **k=256**; see item 2b. 2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured). Halving the framerate does not halve the bitrate. FINDINGS 8. 2b. **A fourth false-good result was produced and caught this session** — k=1024 codebooks looked like a +2.4 dB free win because the rate model charged 1 byte for a 10-bit index. FINDINGS 14. The k=256 configuration ships. 3. **The 256-colour palettised frame is the real quality ceiling** and it looks excellent. Judge the codec against that, not against 1080p. 4. Encoder exists and produces a real bitstream: `tools/encoder/`. --- ## Encoder — working ``` python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop python3 tools/encoder/encode.py /tmp/fr_00020 out.dlx --profile sasi --preview p.png ``` | file | role | |---|---| | `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise | | `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR | | `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision | | `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control | | `encode.py` | CLI + `DLX1` container writer | `DLX1` container layout is documented in the `encode.py` docstring. All multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`. ### Known encoder gaps - ~~Rate control is written but not yet wired into `encode.py`.~~ **DONE, session 6.** It is on by default; `--fixed-lam` restores the old behaviour. Gated by `tools/analysis/09_ratectl_drift.py`, which is now in `check.sh`. - **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later. - **Frame records are not aligned.** They must be padded to a 4-byte boundary: unaligned is an ADDRESS ERROR on a 68000, not a slow read (FINDINGS 28.3). `prep_dlx.py` repairs it at load time, which a player streaming from disc cannot do. The pad is real bytes on disc, so it belongs inside the rate controller's accounting. 1.66 B/frame, 20 B/s. - **The mode decision is blind to CPU cost.** It charges V4 four payload bytes and ignores that it costs 1.49x a V1 block to draw. This is the top item at the head of this file. FINDINGS 28.2. - **Palette packing is not implemented in the encoder.** It still emits 24-bit palettes; the X68000 word packing happens Lua-side. Whatever writes real palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3, worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4). - Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse. - ~~`_paint` is a Python per-block loop.~~ **DONE, session 6** — vectorised, 17.1x. It was never the bottleneck, though: `VQ.assign` is 78% of a frame and `H.build`'s k-means is 51 s of a 55 s run. **That k-means is now the thing to attack before the full-disc survey**, not anything in the per-frame path. FINDINGS 27.6. --- ## Working setup (unchanged from session 1, re-verified) **MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**. ``` mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3 ``` **Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s` **Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM` (still mounted as of end of session 2). **MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the notifier subscription in a global; the stack register is `SP` not `A7`; `autoboot_script` fires at PC=0 before boot) are documented in FINDINGS. **Two shell traps, both hit again this session:** - piping MAME (or any long job) through `grep` block-buffers — write to a file. - `pkill -f ` matches your own shell and kills it (exit 144). Use `pkill -x` or kill by PID. - **`pgrep -f | xargs kill` kills your own shell too — exit 144.** Same root cause as the `pkill -f` trap above: the shell's own command line contains the pattern. **Hit again in session 5**, which makes it four times across three sessions. Kill by PID captured at launch (`$!`), or use `pkill -x`. - **`until ! pgrep -f foo.py; do sleep; done` watcher loops never exit.** The watching shell's own command line contains the string `foo.py`, so `pgrep -f` matches the watcher itself and the loop spins forever. Session 2 left 11 of these wedged for over an hour. Wait on the PID (`while kill -0 $PID`) or on a sentinel file the job touches when it finishes -- never on a `-f` name match. - **`timeout N mame ...` does not kill MAME.** MAME catches SIGTERM and, with an autoboot script blocked waiting on a flag that never arrives, never reaches its shutdown path. `timeout` without `-k` then waits forever while MAME burns a full core at `-nothrottle`. Always `timeout -k 5 N`. --- ## Disk throughput benchmark — still blocked, no longer gating `IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested hypotheses are in session 1's notes (git history of this file, commit 65112b9); the ordered plan for retrying is in **`docs/BENCHMARK.md`**. **Status changed twice this session — read this rather than the git history.** It was briefly promoted to critical-path while the working bandwidth figure was misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the ring-buffer simulation showing **zero required prefill** for both profiles (FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not** available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a "measure it and maybe ship transparent" decision waiting. What the benchmark is still worth doing for: - **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is not recorded. Every profile hangs off it. - **Confirming DMA is actually used.** If transfers fall back to PIO the CPU cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the binding constraint. This is the worst plausible outcome and the cheapest to check — do it first. **Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are functional models, not timing-accurate; a KB/s figure from MAME measures the emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach (MAME validates the path, derivation bounds it, real hardware settles it). ## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5), by a 68000 DECODER (session 7). The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`. **Session 7 went from copying a frame to parsing one.** `src/player/decode.s` reads DLX1, dispatches all four block modes and writes straight into GVRAM; 120 frames decoded in sequence are pixel-exact against `tools/encoder/dlx.py` (`tools/bench/verify_decode.py`, in `check.sh`). The blit numbers below are still correct for what they measured — a *copy* — but they are no longer the display-path budget: the decoder costs 300/448/400 cycles per V1/V4/RAW block and misses the 12fps budget on 31% of frames. FINDINGS 28. **Session 5 closed the gap this paragraph used to describe.** GVRAM is now filled by 68000 instructions and the result is still pixel-exact, and the blit cost is measured rather than estimated: **53.6% of a 12fps frame**, not 38% (FINDINGS 24). The remaining caveat is different and narrower: MAME models **no GVRAM wait states**, so 53.6% is a floor and real hardware is worse. Full write-up in **FINDINGS 22**. Harness: `tools/bench/show_frame.lua` + `tools/bench/prep_frame.py`. Three facts the player MUST honour, none of which were guessable: | what | where | value | |---|---|---| | **Un-hide the graphics layer** | CRTC R20 `$E80028` | clear bit 11 ("G-VRAM set to buffer"); IPL leaves `0x0B16` | | Colour setup (256c) | CRTC R20 bits 9-8 | `0x0100` | | **Monitor contrast** | `$E8E001` bits 3-0 | IPL leaves **14**; write **15** or everything renders 7% dark | The `R20 = 0x0116` value quoted here in session 3 is the **768-wide IPL timing** with the gate cleared. The shipping value is **`R20 = 0x0110`** — see the mode table in `tools/bench/crtc_mode.lua`, which is now the single source of truth for all of R00-R08 and R20. Bit 11 is the one that cost the most time: GVRAM writes land and read back correctly while the layer is invisible, so the video controller looks guilty and is not. Contrast `0` blanks the screen — free fade-to-black for transitions. Palette format is now **confirmed from MAME source**, not assumed: `GGGGGRRRRRBBBBBI` (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as `pal6bit((field<<1)|I)`. With contrast at 15 the render is **pixel-exact**. Ceiling: the 15-bit+I palette costs **40.81 dB** against the 24-bit palettised source, once `I` is chosen per entry (FINDINGS 23.3 — session 3's 38.88 dB assumed `I = 1`). Still the same order as the `scsi` profile's own codec error (39.4 dB), so `scsi` remains near display-transparent, with ~2 dB more headroom than session 3 thought. Snapshot recipe that works (`-video none` CANNOT snapshot): ``` SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \ -sound none -nothrottle -plugins -autoboot_script