# Status & next-session handoff — end of session 12 (2026-08-23) ## Where this stands Session 12 built v7 into the player. **`src/player/decode.s` paints v7 literal spans, and it is pixel-exact under both CPU cores** over a container where every frame carries 128-216 spans covering up to 38% of the picture. FINDINGS 41. The container is **DLX3**: a span section between the mode header and the block payload, `{u32 GVRAM address, u16 coarse disp}` per span with the fine displacement mid-stream. `tools/analysis/16_span_roundtrip.py` gates it and is in `check.sh`. **The measured cost transfers.** Two synthetic all-SPAN anchors price v7 inside `decode.s` at **151.2 and 225.6 clocks per 4x4 block**, against FINDINGS 40's table of 151 and 226 — 0.2% on both emulators. ### The two things that were not on the list **1. There are TWO byte budgets, and FINDINGS 40's 18/120 was scored at the wrong one.** The `scsi` profile is 280 KB/s; `14_dmac_chain.py` scores spans against the 488 KB/s PIPE, which is 40,977 B/frame against 23,228. At the profile rate the lam search has already spent the allowance and spans fire on 5 frames of 120. The profile is a chosen quality rate point; the pipe is hardware. `--kbps` and `--span-kbps` are now separate, and spans run before `mu` because a span pays in bytes and `mu` pays in picture. FINDINGS 41.2. | 120-frame `scsi` window | KB/s | over budget | PSNR | |---|---:|---:|---:| | no spans | 278.3 | 86/120 | 29.27 dB | | spans, profile budget only | 280.0 | 77/120 | 29.23 dB | | **spans on the 488 KB/s pipe** | 487.7 | **34/120** | **29.63 dB** | **2. `C_SKIP_MIXED` was never measured, and it was 18% low — 45.0, now 55.0.** It is the one constant in the cost table that came from a derivation, because the synthetic frame that would measure it cannot exist (a byte needs a coded block for its SKIP to be mixed). Four new bracketing anchors measure it on both emulators, and with it corrected the model predicts a real spanned decode to **-0.06% mean / 0.09% worst**, against -2.99% / 4.30% before. It matters here because **a span marks its run SKIP**, so mixed SKIPs are the dominant population in exactly the frames spans are judged on. FINDINGS 41.5. **And the metric everything has been quoted in is unstable.** 34/120 delivered against 14's simulated 18/120 is a 1.4% difference in mean frame cost. 55 of 120 frames sit within 5% of the deadline because the rate controller aims there, so a 1% cost shift moves 22 frames. Quote the distribution, not the count. FINDINGS 41.6. Green light: `./tools/bench/check.sh` **ALL GREEN**, now gating on a span-heavy DLX3 container. ## NEXT SESSION, in order 0. **Green light first.** `./tools/bench/check.sh` (~6 min, Blu-ray mounted). Verified green at end of session 12. The gate container is now `tmp/rc_fr_singe_scsi_span.dlx` (scsi modes, spans on the full pipe) and the rig fits 37 of 120 frames in a 2 MB machine. 1. **Decide the rate point, because the span result now depends on it.** This is the user's call and it is the first real fork since the profile was set: spans only pay if the stream is allowed to run near the pipe (487.7 KB/s delivered, 34/120 over budget) rather than at the 280 KB/s profile (77/120). That is a delivery-medium question — FINDINGS 32 dropped SASI on capacity and parked the 110 KB/s point for CD-ROM, and 488 KB/s is 93% of a 4 Mbps figure whose provenance is still unconfirmed (FINDINGS 29.5 item 3). **Do not spend another session optimising against a budget nobody has chosen.** 2. **Re-derive the span selection jointly with lam, not after it.** The encoder picks modes at one budget and then spans what is left, which FINDINGS 39.3 already called a lower bound. A frame that misses its deadline would often do better raising lam to free room for spans than lowering it — spans are pixel-exact, so the quality trade is not what it looks like. Bisecting a span reserve fraction inside the existing search is the tractable version. 3. **Re-run `13_cpu_ratectl.py` against a DLX3 container.** 14 and 15 are done (15 now counts span bus traffic and still reproduces the C68K measurement to 0.04%); 13 has not been re-run since the constant changed. 4. **Make sure the player actually gets DMA** — unchanged from sessions 10-11, and still not an optimisation. Benchmark `x68000 -exp1 cz6bs1`, **never `x68ksupr`**; MAME's internal SCSI has no DMA glue and would measure a PIO fallback the real machine does not have. This is now more load-bearing, not less: the delivered stream is 487.7 KB/s and the disk debit is 163,798 clocks a frame, 20% of the budget. 5. **Re-run the ring-buffer simulation at the surviving rate** (FINDINGS 29.5/30.7, still open) and confirm the 488 KB/s figure's provenance. **Do not start by hand-optimising `decode.s`.** Unchanged and still true. The cycles to be won are in the budget, not the loop — and session 12 is the second demonstration that the *model* of the budget is where the errors live. **A new trap, worth reading before quoting any figure:** the rig had been writing its synthetic timing frames 26 KB past the top of a 2 MB machine, and got away with it because the modes it overran are data-independent. A span is not — its jump displacements come out of the stream. FINDINGS 41.4. --- ## What session 11 settled Session 11 measured v7 in `blit.s` and left it there; session 12 built it into the player. Items 0 and 1 of session 11's list are done (FINDINGS 40, 41) and the rest are carried forward in the list above. ## What session 10 settled Session 10 cross-checked the whole cycle model against a second emulator, then found that the model was denominated in the wrong currency. **FINDINGS 38 is the headline: the project is BUS-bound, not CPU-bound.** Nothing since FINDINGS 24 had counted the 68000's local memory bus — one 4-clock cycle at a time, carrying instruction prefetch as well as data. Measured, the decoder occupies **86.7%** of it, and **prefetch is 62% of that traffic**. Scoring the `scsi` window with the bus as the shared resource, **52 of 53 missed frames are bus-limited and one is CPU-limited**. Every optimisation since 24 has been aimed at the budget that is not binding. The measurement is two sources checking each other: `c68k_bench` counts every bus callback exactly, and a static walk of `decode.lst` adds the prefetch no emulator here can report. The walk reproduces the measured data half to **0.04%**, which is what licenses its prefetch half. **The DMAC array-chain LOSES, on the datasheet.** My first pass derived its cost from bus arithmetic — a read cycle plus a write cycle, 8 clocks a pixel — and scored it at 1/120 frames over budget against v6's 10/120. Then I read the MC68450 manual (Motorola Jul 1989, bitsavers). **Fig 4-25 sheet 4: a dual-address word between two 16-bit ports is 9 clocks, because note 2 gives the DMAC 4-clock reads and 5-clock WRITES.** The 68000 writes in 4. | per pixel | clocks | source | |---|---:|---| | DMAC dual-address word | **9.000** | MC68450 Fig 4-25 sheet 4 | | v6 `movem` chain | **9.152** | MEASURED, FINDINGS 30 | Scored additively (see below), against the same mode maps: | | frames over budget | |---|---:| | today | 84/120 | | v6 span as built | 55/120 | | **v6 with a finer chain tail — software only** | **18/120** | | DMAC chain | 12/120 | **86% of the DMAC's advantage is v6's 24-pixel padding quantum**, which is a property of v6's unrolled `movem` chain and fixable in software: add a second chain of 2-register units for the tail and the padding drops from 23 pixels to 3, at no per-span cost. **Recommendation (mine): fix the quantum, drop the DMAC.** Six frames of 120 does not buy a reserved channel, a two-region container layout, and a timing dependency neither emulator here can verify. FINDINGS 39. The container work is not wasted either way: v6's record and an HD63450 chaining entry are both 6 bytes, so **the chain array is the span table** and nothing has to change if a hardware measurement later moves the DMAC's number. FINDINGS 39.1. **I also got FINDINGS 35 wrong before the datasheet corrected me.** I argued its flat CPU debit was too pessimistic and rescored the window at 53/120 using `max(CPU, bus)`. A 68000 has no cache and a two-word prefetch queue, so it stalls the moment another master takes the bus, and the MC68450 hands the bus over in *slabs* under limited-rate auto-request rather than interleaving per operand. **DMA is additive. 84/120 stands.** What 86.7% bus occupancy really says is that there is almost no room to overlap anything on this machine. FINDINGS 38.3. **And the datasheet settled STATUS's own "most load-bearing unmeasured number".** FINDINGS 5's 8 clocks/word for the SCSI DMA: Fig 4-25 sheet 3 gives single-address write 5 clocks, so it is **5 clk/word with the bus held and ~12 if the DMAC arbitrates per word**. 8 is the midpoint of a bracket the datasheet supports. Which end applies is a player design decision — how the MB89352 drives REQ, and whether cycle-steal-with-hold is used — worth 7 clocks a word on a 480 KB/s stream. FINDINGS 39.7. **The decoder is now pixel-exact under two independent CPU cores.** `tools/bench/c68k/` links px68k's C68K core into a headless harness and decodes the same container to the same pixels. Cycle-table error against MAME is bounded at **3.3%**, and it runs against us. Two incidental results worth keeping: MAME 0.277's `M68000` is the **microcode core**, not Musashi (`m68000.lst` + `m68000gen.py`), so this is two structurally different timing models agreeing rather than two tables; and FINDINGS 28.8's "V4 costs more than RAW" reproduces independently. FINDINGS 37. --- ## What session 10 put on the list Kept for the reasoning, not the priorities: items 0 and 1 are done (FINDINGS 40) and the rest are carried forward, reordered, in the session-11 list at the top. Everything below this point is from session 9 or earlier and still stands unless a later section above says otherwise. ### session 10's list 0. **Green light first.** `./tools/bench/check.sh` (~5 min, Blu-ray mounted). Verified green at end of session 9. The gate container is now `tmp/rc_fr_singe_scsi_cpufit.dlx`, and it decodes **80 of 120 frames**, not 120: the rig loads the whole stream into a 2 MB machine and a `scsi` window is 2.84 MB, which overran the top of RAM by 940 KB and produced a failure that looked exactly like a decoder bug. `prep_dlx.py` now truncates to what fits and says so; `verify_decode.py --nframes` replays the same prefix; and the stage fails loudly if the sequential pass did not complete. FINDINGS 36. At 278.6 KB/s a 2 MB machine holds ~6.7 s of stream, so **the strongest test in the tree can only ever audit a prefix of a window.** Getting the coverage back means gating on more than one window, not one longer pass. 1. **Measure the finer chain tail with `span.sh`.** It is the largest single win on the table — 55/120 to 18/120 — it is software only, and the tool that measures it already exists: `blit.s` gains a v7 with a 2-register tail chain, `prep_spans.py` generates the streams, `span.sh` times all of it in ~25 s and asserts every config drew a pixel-exact frame. The 18/120 figure is DERIVED from a conservative 56-clocks-per-4-pixel-unit model and should not be believed until it is measured. FINDINGS 39.4. 2. **Make sure the player actually gets DMA.** DMA-vs-PIO is a property of our code, not the board: the CZ-6BS1's DMA path is real and modelled (FINDINGS 32.4), but if the player reads through IOCS and IOCS does PIO we get the 120/120 row. `docs/BENCHMARK.md` item 4 (drive the MB89352 directly) is no longer an optimisation. Benchmark `x68000 -exp1 cz6bs1`, **never `x68ksupr`** — MAME's internal SCSI has no DMA glue at all (`// TODO: duplicate DMA glue from CZ-6BS1`) and would measure a PIO fallback the real machine does not have. 3. **Re-decide the framerate, now that it is the lever that pays for the disk.** 10 fps absorbs the DMA steal on current estimates. This was item 5 and a quality question; it is now arithmetic. It is still the user's call. 4. **Then spans, on the CPU.** The format is fully specified by measurement (FINDINGS 30.2, costs 30.5, scene-cut arithmetic 30.6) and `decode.s` does not implement them yet. Build the v7 tail from item 1 into it. The DMAC variant is costed and loses (39.5); the container is identical either way (39.1), so nothing is foreclosed. 5. **Re-run the ring-buffer simulation at the surviving rate** and confirm the 488 KB/s figure's provenance (FINDINGS 29.5/30.7, still open). **Do not start by hand-optimising `decode.s`.** The hand-derived timings agree with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), and FINDINGS 34 confirms the model on a second container. The inner loop is close to what the instruction set allows; the cycles to be won are in the budget, not the loop. **Three of this session's near-misses had the same shape: an unobservable run almost produced a false finding.** The 23-minute "hang" (FINDINGS 34.1), the RAM overrun that looked like a 4x-slow decoder (FINDINGS 36), and the truncated gate that reported 49,005 differing pixels. In each case the instrument was broken, not the thing being measured. **Always `stdbuf -oL` a MAME job that prints progress.** A file is block-buffered too, so a long run is unobservable until it exits — and an unobservable run that is merely finishing looks exactly like one that is wedged. Session 8 lost this measurement to that; session 9 lost 23 minutes to it before spending 25 seconds getting the answer with `stdbuf`. FINDINGS 34.1. --- ## What session 8 settled 0. **The mode decision can see cycles, it is on by default, and it costs 0.26 dB.** `decide(ctx, lam, mu)` minimises `D + lam*bytes + mu*cycles`; `ratectl` bisects mu per frame against a HARD 833,333-cycle ceiling (bytes bank in the ring buffer, cycles cannot — there is no double buffer to decode ahead into). `sasi` 37/120 misses -> 1, `scsi` 51 -> 1. Bitrate does not move: mu changes which modes are bought, not how many bytes. FINDINGS 31, `tools/analysis/13_cpu_ratectl.py`. 0b. **28.7's "11 frames are impossible" was too pessimistic — it is 1.** That floor held the SKIP set fixed; the real decision can also move a block to SKIP, which above ~90% non-SKIP is the only lever left. FINDINGS 31.3. 0c. **V4 collapses when cycles are priced**, as 28.8 predicted: 25.2 -> 20.3% of blocks at `sasi` and **15.0 -> 5.3%** at `scsi`, where RAW takes it. RAW is dearer in bytes and cheaper in cycles, so the byte lagrangian's preference inverts and only the byte-rich profile can take the escape. 0d. **SKIP's price depends on its neighbours, and the way out is two cost functions**: a ranking constant inside the per-block lagrangian, the exact clustered rule (`vq_hybrid.cycles`, validated to 1 point against the 68000) for the frame-level bisection. That function is now defined once and imported by `11_cpu_budget.py`. FINDINGS 31.4. 0e. **Both controllers are gated against decoder drift.** `09_ratectl_drift.py` runs bytes-only AND bytes+cycles; both 0/120. 1. **The span is measured: 43.7 cycles/span + 9.152/pixel, fitted to 0.3% over eleven span lengths.** `tools/bench/blit.s` v5/v6, `prep_spans.py`, `span.lua`, driven by `tools/bench/span.sh` (~25 s, not in `check.sh` because it is a wall timing). FINDINGS 30. 2. **Only in an encoder-assisted format.** `{u32 absolute GVRAM address, u16 jump displacement}` into an unrolled chain, versus `(x, npix)` and a decoder that works it out: 43.7 + 9.152 against 97.9 + 10.46. All the arithmetic a span decoder would do per frame is known at encode time. FINDINGS 30.2. 3. **The per-pixel cost is a function of register pressure**, which FINDINGS 24 could not have shown: 9.08 was a fixed blit with 12 registers free, v5 can spare 8 and pays 10.46, v6 gets 12 back by making the encoder hold the state. 4. **Short spans die in the remainder path, and the fix is padding.** A 12-pixel span costs more than a 16-pixel one in v5. v6 has no remainder path: lengths are multiples of 24 pixels, padding is free of everything but bytes, and an overrun past the visible 256 lands in the invisible half of the 1024-byte line stride. FINDINGS 30.3. 5. **Odd-`x` alignment is free** (259.0 vs 261.8 cycles/span) — expected on a 16-bit bus, now measured rather than assumed. 6. **The trade is smaller than 29 derived but the conclusion holds**, including 29.4's reopening of the scene cut. All 23 timing configs also drew a pixel-exact frame, so nothing here was timed against a decoder that skipped work. FINDINGS 30.5/30.6. --- ## What session 7 settled 1. **68000 code parses a bitstream and draws frames, pixel-exact.** `src/player/decode.s` + `tools/bench/decode.lua`. 120 frames of the Singe window decoded in sequence, all four block modes, verified against the new reference decoder `tools/encoder/dlx.py`. Because SKIP blocks are claims about the previous frame, the last frame is only right if all 120 were. In `check.sh` now. **FINDINGS 28.** 2. **It does not fit.** Mean 81.7% of a 12fps frame, p90 116.4%, worst 135.8%; 31% of frames miss at `sasi`, 42% at `scsi`. Zero-wait-state floor, as ever. 3. **The dual-display-path plan (FINDINGS 24.5/25.6) is withdrawn as incoherent — the sixth false premise this project has caught.** The compose path needs a RAM copy of the previous reconstruction; the direct path's whole selling point is that it keeps none. Mixing them displays stale pixels on **70 of 120 frames**, worst frame 18.8% of the screen. Every coherent repair is worse than not mixing. `tools/analysis/10_pathmix_drift.py`, kept runnable as a counterexample and gated in `check.sh`. FINDINGS 28.1. 4. **24.5 also compared a copy against a copy.** Its 53.6% and 76.6% both come from `blit.s` and neither includes decoding. Compose = decode-into-RAM *plus* the 53.6% blit, so it is strictly dearer than decoding into GVRAM. There was never a crossover. The player has **one path and no reference frame**, which also gives back 96 KB. 5. **The four block modes cost 300 / 448 / 400 cycles, not one number.** V4 is 1.49x a V1 block while the mode decision charges it 4x the bytes. The 24.5 model is 2.03x optimistic at the median. `tools/analysis/11_cpu_budget.py` reproduces all four frames timed on the 68000 to within 1 point. FINDINGS 28.2. 6. **The container is big-endian but not aligned, and on a 68000 that is an address error, not a slow read.** Frame records are variable-length and laid end to end, so their boundaries land on odd addresses. Frame 0 decoded perfectly, then the length read for frame 1 vectored into the IPL and sat there for 59 emulated seconds looking like an infinite loop. Found by dumping PC and the address registers — the code was right, the data layout was not. FINDINGS 28.3. **Encoder gap: `encode.py` should pad records to 4 bytes.** Measured cost 1.66 B/frame = 20 B/s against 110 KB/s. 7. **A full frame does not fit at 12fps in any mode.** All-V1 is 110.5%, all-V4 165.2%, all-RAW 147.6%. At most ~88% of the screen can change in one frame however cheaply it is coded, and scene cuts change 100%. FINDINGS 28.5. --- ## What session 6 settled 1. **Rate control works, is wired in, and is ON by default.** `encode.py` bisects lam per frame under a leaky bucket; `--fixed-lam` restores session 5 behaviour. FINDINGS 27. 2. **Both overshoots are closed for under 1 dB.** On the Singe window, totals including audio: `sasi` 137.4 -> **109.5 KB/s** (target 110) for -0.60 dB, `scsi` 381.6 -> **280.0 KB/s** (target 280) for -0.91 dB. Zero frames hit the lam=800 cliff at either profile. FINDINGS 27.2. 3. **The FINDINGS 26 desync is gone by construction, not by tuning.** The encoder is frame-drivable (`vq_hybrid.frame_ctx` / `decide` / `paint`) and rate control feeds back the frame it actually emitted. The regression test `tools/analysis/09_ratectl_drift.py` goes 111/120 drifting frames -> **0**, and it is now part of `./tools/bench/check.sh`. FINDINGS 27.1. 4. **Rate control makes the display path cheaper.** Raising lam moves blocks to SKIP and V1, so there is less to write: `scsi`'s median display cost drops 53.6% -> 47.1%. The decoder conclusion of 25.6 is unaffected. FINDINGS 27.3. 5. **FINDINGS 26.5 was wrong in both halves, and this is the fifth false premise this project has caught.** `_paint` was not the bottleneck (14% of a frame) and the ladder was never "minutes" (~18 s; the minutes were k-means in `build`). Vectorising it was still right — 17.1x — but what actually makes per-frame rate control affordable is that `VQ.assign` depends on neither `lam` nor `prev`, so it is cached: a 12-step search over 120 frames costs **0.31 s** against 49.1 s. FINDINGS 27.6. 6. **`--prefill` is a trap and defaults to 0.** It buys a permission to overshoot of exactly bucket/nframes; at prefill=1.0 the Singe window goes to 116.3 KB/s against a 110 ceiling, and on a 14-frame clip it disables the controller outright. FINDINGS 27.4. 7. **Fixed-lam `sasi` was already 5% over target on 00020**, the clip everyone called easy — nothing noticed because the profile table quotes PSNR, not bitrate. FINDINGS 27.5. 8. **1.2-second clips cannot be used to judge rate control.** The bucket's startup transient is bucket/nframes: 6% on a 10 s window, 20% on 00020. Same lesson as FINDINGS 25.3, different costume. --- ## Start here: is the tree still green? ``` ./tools/bench/check.sh ``` ~4 min, needs the Blu-ray mounted. From source media it re-runs both display regression tests, the rate-control drift test (session 6, now covering BOTH controllers -- bytes, and bytes+cycles), the display-path coherency counterexample and a **120-frame 68000 decode** (session 7), then prints `ALL GREEN`. Verified green at end of session 8. Do not run two of these at once, and do not run one alongside a MAME timing job: they share `tmp/` snapshot directories and log files, and the second run silently truncates the first one's output. If it fails, fix that before doing anything else — everything downstream assumes the display path is pixel-exact. The two session-7 stages are worth knowing the shape of before they fail on you: - `10_pathmix_drift.py` is expected to exit **non-zero**; `check.sh` fails if it ever starts passing, because that would mean the counterexample behind the one-path decoder had stopped reproducing. - the decode stage needs `tmp/rc_fr_singe_sasi_rcprofile.dlx` and will spend ~55 s encoding it if it is missing, nearly all of that k-means in `H.build`. ## Decisions locked | decision | value | why | |---|---|---| | Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint | | Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c | | Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway | | **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 | | **Quality modes** | **one: `scsi`** (USER DECISION, session 9 — `sasi` dropped) | a SASI volume is 40 MB and the game is 146 MiB at the LOWEST rate this codec makes. FINDINGS 32 | | Delivery medium | **SD-backed SCSI** (BlueSCSI / SCSI2SD), as locked in session 2 | capacity does not bind on SD at any rate this codec makes; it is what killed SASI, and it is what rules CD-ROM OUT (650 MiB with spans, and 1x CD is ~150 KB/s). FINDINGS 32.3 | | Profile axis | **I/O bandwidth only** | the profiles say nothing about CPU; **both target the same stock 10MHz 68000**, and the Super has SCSI at 10MHz. FINDINGS 28.7 | | Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win | | Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something | | SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution | ### The SASI/SCSI question is RESOLVED — and in session 9 SASI was DROPPED Session 1 left "which machine do we target" open. Session 2's answer was **ship both**, as two quality profiles. **Session 9 retired `sasi`** (USER DECISION) on CAPACITY, not bandwidth: a SASI volume is limited to 40 MB, and the 22.8 minutes of unique scene footage on the source Blu-ray is **146 MiB even at 110 KB/s** — more than the machine's whole 4-unit SASI address space. FINDINGS 32. Everything below this line about *two* profiles is the session-2..8 record, kept because every measurement in FINDINGS 27-31 was taken against it. One profile ships: | profile | target | lam | quality (00020 / 00146) | machine | |---|---|---|---|---| | ~~`sasi`~~ | ~~110 KB/s~~ | ~~60~~ | ~~36.9 / 29.6 dB~~ | **RETIRED session 9** | | `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board | The 110 KB/s **rate point** is not necessarily gone with the interface: a 1x CD-ROM sustains ~150 KB/s and CD-ROM is the only period medium with room for the whole game (374 MiB at `scsi`, 600 MiB with spans). The user's call was to ship one profile now and settle the medium when the pipe is measured. FINDINGS 32.3. **That "machine" column is about the BUS, not the CPU.** The profiles are an I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has built-in SCSI at 10 MHz (`x68k.cpp:1194`, `40_MHz_XTAL/4`, same as the base machine), and only the XVI is faster. **Both profiles target the same stock 10 MHz 68000**, so both must fit the same 833,333-cycle frame budget — and as of session 7 neither does. FINDINGS 28.7. **As of session 6 `lam` is a floor, not a setting.** The target is a ceiling and the encoder bisects lam per frame to stay under it; the profile's lam is the best quality it is allowed to spend on a quiet frame. On the worst sustained window that takes `sasi` from 137.4 to 109.5 KB/s and `scsi` from 381.6 to 280.0 KB/s, for -0.60 and -0.91 dB. FINDINGS 27.2. Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**, not an average over seek latency. **Both profiles fit with room.** Ring-buffer simulation on the real per-frame sizes gives **zero required prefill** for every scene at both profiles: the fill delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance, far more than an SD-backed seek needs. FINDINGS 21. An earlier warning here said `scsi` did not fit because a frame peaked at 96.4% of the pipe. That compared instantaneous demand to a sustained rate as if they had to match frame-by-frame; with a buffer the test is cumulative, and it passes. `scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the RD curve rather than derived from the hardware. See FINDINGS 17. Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured and rejected — see FINDINGS 14, it was a false-good result from a rate model that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff. Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised frame (measured 0.00 dB loss). The profiles are two points on one continuous rate-distortion curve, not two codecs. --- ## What session 3 settled 1. **The display path works and is verified end to end.** First real frame on an emulated X68000 screen: `docs/images/x68k_first_frame_compare.png`. Full write-up in **FINDINGS 22**. Everything before this session was Python-side or a headless `-video none` run, which cannot snapshot at all. 2. **The render is pixel-exact, not merely close.** With monitor contrast at 15, all 256 palette entries render exactly as `GGGGGRRRRRBBBBBI` + `pal6bit` predicts. That exactness is the regression test — see `tools/bench/verify_frame.py`, which exits non-zero if it ever drifts. 3. **Three hardware facts that were previously assumed are now confirmed from MAME 0.277 source**, not folklore: the palette word format, the 1024-byte GVRAM line stride, and the 256-colour page aliasing in `HARDWARE.md`. All three were already written down correctly; they are now cited. 4. ~~**A new quality ceiling was measured** — the 15-bit+I palette alone costs 38.88 dB.~~ **Superseded by session 4:** that figure assumed the shared LSB `I` is always 1. Chosen per entry, the ceiling is **40.81 dB**. FINDINGS 23.3. 5. **Two shell traps that wedged session 2's background jobs** are documented in the working-setup section below. They cost ~1.5 h of wall clock and a wedged CPU core, and one of them was hit again this session. ## What session 5 settled 1. **68000 code drew a frame, and the blit was measured.** `tools/bench/blit.s` + `blit.lua`. The snapshot passes `verify_frame256.py` unchanged — pixel-exact in the real 256x256 mode. **FINDINGS 23.5 is closed**: no longer "proven from Lua only". 2. **The 38% full-frame blit estimate is dead. It is 53.6%.** And that is a zero-wait-state floor — MAME models no GVRAM wait states, so real hardware is worse. FINDINGS 24. Every variant was hand-derived from the MC68000 timing tables before being measured and the two agree to 0.006-0.43%, so this is not another MAME artefact. 3. **Reading the source frame is exactly half the blit cost** (V1 53.6% vs a write-only floor V3 of 27.1%). That is what makes the architecture question below live. 4. **That number is now measured, and the answer is "implement both paths".** On the worst sustained window found on the disc, 30% of frames (`sasi`) to 53% (`scsi`) sit above the 70% crossover and want the flat blit; the rest want direct-to-GVRAM. A player that picks per frame — the mode headers are parsed before any pixel is written, so the count is free — pays a **median 37.0%** and is **capped at 53.6%**. FINDINGS 25.6. 5. **The sustained action sequence exists, was found by measurement, and breaks both profiles.** `tools/analysis/07_motion_survey.py` scans a whole stream for the hottest sliding window; on 00223 it is t=539.4s, the Singe endgame, at 2.01x the stream mean. There, fixed-lam `sasi` overshoots 110 -> 129.6 KB/s (+18%) and `scsi` 280 -> 373.8 KB/s (+34%). **Rate control is no longer insurance — it is required.** FINDINGS 25.3. 6. **The two largest streams on the disc are bonus material, not game footage.** 00216 is the feature with a burned-in commentary PiP; 00215 is the commentary itself. **00223 (9.4 min) is the clean one.** A size-ranked survey would have encoded live action. FINDINGS 25.1. 7. **Rate control is unsound as written, caught before wiring it up.** The lam-ladder in `ratectl.py` picks frames from independent temporal chains, so SKIP blocks reference reconstructions the decoder never saw: 111 of 120 frames drift, worst frame 43.4%, reported PSNR overstated 0.36 dB. Regression test `tools/analysis/09_ratectl_drift.py`. FINDINGS 26. 8. **On hard content the scene palette, not the display, is the binding ceiling** — 31.33 dB on the Singe window against 39.90 dB on 00020 and 40.81 dB for the X68000 display. `scsi` is already within 0.51 dB of it. FINDINGS 25.4. ### Superseded within session 5 4a. **The decoder architecture hinged on one unmeasured number.** Writing codewords straight into GVRAM costs 76.6% of the frame budget for a *full* frame (V4 — the 1024-byte stride kills the `movem.l` burst), but scales with the non-SKIP block fraction and needs **no RAM reference frame at all**, because the previous frame is already in GVRAM. Compose-then-blit is a flat 53.6%. **They cross at 70% of blocks changed.** FINDINGS 24.5. --- ## What session 4 settled 1. **A real 256x256 CRTC mode exists and is verified.** `crtc_mode.lua`, derived from `x68k_crtc.cpp`'s divisor ladder rather than recalled — the derivation is self-checking (368 = 1104/3 exactly, so the horizontal registers divide by three with no remainder). Snapshot is native 256x512, active area pixel-exact, letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone. 2. **The palette ceiling was wrong by 2 dB, in our favour.** The shared LSB `I` must be chosen **per palette entry**, not hardcoded to 1. Doing so lifts the display ceiling from 38.85 to **40.81 dB** and is the only way to get true black at all (`pal6bit(1) = 4`). 102 of 256 entries want `I = 0`. This supersedes FINDINGS 22.4 and gives `scsi` ~2 dB more headroom than believed. **The encoder does not do this yet** — see the encoder-gaps list. 3. **Letterboxing costs one palette entry.** 255 colours + a reserved black at index 0, with `I = 0` on it. `prep_frame.py --reserve-black`. FINDINGS 23.4. 4. **MAME's graphics double-scan is phase-shifted one raster line** — pairs are (1,2),(3,4),..., not (0,1), because `get_gfx_pixel` halves the *absolute* scanline and `vbegin = 41` is odd. Cost a false failure. FINDINGS 23.2. --- ## What session 2 settled 1. **The critical-path question is answered.** "Does VQ soften Bluth's linework unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.** Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two images in `docs/images/`. Both profiles use **k=256**; see item 2b. 2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured). Halving the framerate does not halve the bitrate. FINDINGS 8. 2b. **A fourth false-good result was produced and caught this session** — k=1024 codebooks looked like a +2.4 dB free win because the rate model charged 1 byte for a 10-bit index. FINDINGS 14. The k=256 configuration ships. 3. **The 256-colour palettised frame is the real quality ceiling** and it looks excellent. Judge the codec against that, not against 1080p. 4. Encoder exists and produces a real bitstream: `tools/encoder/`. --- ## Encoder — working ``` python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop python3 tools/encoder/encode.py /tmp/fr_00020 out.dlx --profile sasi --preview p.png ``` | file | role | |---|---| | `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise | | `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR | | `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision | | `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control | | `encode.py` | CLI + `DLX1` container writer | `DLX1` container layout is documented in the `encode.py` docstring. All multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`. ### Known encoder gaps - ~~Rate control is written but not yet wired into `encode.py`.~~ **DONE, session 6.** It is on by default; `--fixed-lam` restores the old behaviour. Gated by `tools/analysis/09_ratectl_drift.py`, which is now in `check.sh`. - **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later. - **Frame records are not aligned.** They must be padded to a 4-byte boundary: unaligned is an ADDRESS ERROR on a 68000, not a slow read (FINDINGS 28.3). `prep_dlx.py` repairs it at load time, which a player streaming from disc cannot do. The pad is real bytes on disc, so it belongs inside the rate controller's accounting. 1.66 B/frame, 20 B/s. - **The mode decision is blind to CPU cost.** It charges V4 four payload bytes and ignores that it costs 1.49x a V1 block to draw. This is the top item at the head of this file. FINDINGS 28.2. - **Palette packing is not implemented in the encoder.** It still emits 24-bit palettes; the X68000 word packing happens Lua-side. Whatever writes real palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3, worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4). - Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse. - ~~`_paint` is a Python per-block loop.~~ **DONE, session 6** — vectorised, 17.1x. It was never the bottleneck, though: `VQ.assign` is 78% of a frame and `H.build`'s k-means is 51 s of a 55 s run. **That k-means is now the thing to attack before the full-disc survey**, not anything in the per-frame path. FINDINGS 27.6. --- ## Working setup (unchanged from session 1, re-verified) **MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**. ``` mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3 ``` **Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s` **Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM` (still mounted as of end of session 2). **MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the notifier subscription in a global; the stack register is `SP` not `A7`; `autoboot_script` fires at PC=0 before boot) are documented in FINDINGS. **Two shell traps, both hit again this session:** - piping MAME (or any long job) through `grep` block-buffers — write to a file. - `pkill -f ` matches your own shell and kills it (exit 144). Use `pkill -x` or kill by PID. - **`pgrep -f | xargs kill` kills your own shell too — exit 144.** Same root cause as the `pkill -f` trap above: the shell's own command line contains the pattern. **Hit again in session 5**, which makes it four times across three sessions. Kill by PID captured at launch (`$!`), or use `pkill -x`. - **`until ! pgrep -f foo.py; do sleep; done` watcher loops never exit.** The watching shell's own command line contains the string `foo.py`, so `pgrep -f` matches the watcher itself and the loop spins forever. Session 2 left 11 of these wedged for over an hour. Wait on the PID (`while kill -0 $PID`) or on a sentinel file the job touches when it finishes -- never on a `-f` name match. - **`timeout N mame ...` does not kill MAME.** MAME catches SIGTERM and, with an autoboot script blocked waiting on a flag that never arrives, never reaches its shutdown path. `timeout` without `-k` then waits forever while MAME burns a full core at `-nothrottle`. Always `timeout -k 5 N`. --- ## Disk throughput benchmark — still blocked, no longer gating `IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested hypotheses are in session 1's notes (git history of this file, commit 65112b9); the ordered plan for retrying is in **`docs/BENCHMARK.md`**. **Status changed twice this session — read this rather than the git history.** It was briefly promoted to critical-path while the working bandwidth figure was misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the ring-buffer simulation showing **zero required prefill** for both profiles (FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not** available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a "measure it and maybe ship transparent" decision waiting. What the benchmark is still worth doing for: - **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is not recorded. Every profile hangs off it. - **Confirming DMA is actually used.** If transfers fall back to PIO the CPU cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the binding constraint. This is the worst plausible outcome and the cheapest to check — do it first. **Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are functional models, not timing-accurate; a KB/s figure from MAME measures the emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach (MAME validates the path, derivation bounds it, real hardware settles it). ## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5), by a 68000 DECODER (session 7). The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`. **Session 7 went from copying a frame to parsing one.** `src/player/decode.s` reads DLX1, dispatches all four block modes and writes straight into GVRAM; 120 frames decoded in sequence are pixel-exact against `tools/encoder/dlx.py` (`tools/bench/verify_decode.py`, in `check.sh`). The blit numbers below are still correct for what they measured — a *copy* — but they are no longer the display-path budget: the decoder costs 300/448/400 cycles per V1/V4/RAW block and misses the 12fps budget on 31% of frames. FINDINGS 28. **Session 5 closed the gap this paragraph used to describe.** GVRAM is now filled by 68000 instructions and the result is still pixel-exact, and the blit cost is measured rather than estimated: **53.6% of a 12fps frame**, not 38% (FINDINGS 24). The remaining caveat is different and narrower: MAME models **no GVRAM wait states**, so 53.6% is a floor and real hardware is worse. Full write-up in **FINDINGS 22**. Harness: `tools/bench/show_frame.lua` + `tools/bench/prep_frame.py`. Three facts the player MUST honour, none of which were guessable: | what | where | value | |---|---|---| | **Un-hide the graphics layer** | CRTC R20 `$E80028` | clear bit 11 ("G-VRAM set to buffer"); IPL leaves `0x0B16` | | Colour setup (256c) | CRTC R20 bits 9-8 | `0x0100` | | **Monitor contrast** | `$E8E001` bits 3-0 | IPL leaves **14**; write **15** or everything renders 7% dark | The `R20 = 0x0116` value quoted here in session 3 is the **768-wide IPL timing** with the gate cleared. The shipping value is **`R20 = 0x0110`** — see the mode table in `tools/bench/crtc_mode.lua`, which is now the single source of truth for all of R00-R08 and R20. Bit 11 is the one that cost the most time: GVRAM writes land and read back correctly while the layer is invisible, so the video controller looks guilty and is not. Contrast `0` blanks the screen — free fade-to-black for transitions. Palette format is now **confirmed from MAME source**, not assumed: `GGGGGRRRRRBBBBBI` (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as `pal6bit((field<<1)|I)`. With contrast at 15 the render is **pixel-exact**. Ceiling: the 15-bit+I palette costs **40.81 dB** against the 24-bit palettised source, once `I` is chosen per entry (FINDINGS 23.3 — session 3's 38.88 dB assumed `I = 1`). Still the same order as the `scsi` profile's own codec error (39.4 dB), so `scsi` remains near display-transparent, with ~2 dB more headroom than session 3 thought. Snapshot recipe that works (`-video none` CANNOT snapshot): ``` SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \ -sound none -nothrottle -plugins -autoboot_script