drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Drop SASI on capacity, then find the budget never had the disk in it

Browse Source 
USER DECISION: drop the `sasi` profile. Not on bandwidth -- on capacity. A SASI
volume is 40 MB, and the 22.8 min of unique scene footage on the source Blu-ray
(streams 00000-00201, measured, not recalled) is 146 MiB at the LOWEST rate this
codec makes -- more than the machine's whole 4-unit SASI space. `scsi` is the
only profile now. FINDINGS 32.

Then the user asked whether we were drawing the wrong conclusions about PIO vs
DMA, and we were, more broadly than the question implied. Every CPU figure in
FINDINGS 24-34 is scored against the full 833,333 cycles/frame with nothing
subtracted for moving the bitstream off disk. Debiting the HD63450 cycle-steal
at the long-standing 8 clk/word ESTIMATE, "1 frame of 120 misses" becomes 84 of
120, median 112.4%. PIO at the span rate is 99.8% of the machine. Spans buy
cycles by spending bandwidth and the bandwidth returns as steal, so 31.6's "fits
completely" becomes a worst frame of 114.3%. 10 fps absorbs it: median 93.7%,
1/120. FINDINGS 35. `11_cpu_budget.py` takes --io dma|pio|none, defaults to dma,
and warns if asked for none.

Also landed:
- item 1 done: the cost model checked against the 68000 on a cost-aware
  container, -3.07% to +0.01%, whole-window mean -1.22%. FINDINGS 34.
- item 4 done: the container carries its own 4-byte record alignment (DLX2).
  94/120 record starts were on odd addresses -- an address error, not a slow
  read -- now 0/120 for 16 B/s. Re-encoding reproduces 31.1 exactly. FINDINGS 33.
- a `scsi` window does not fit the 2 MB machine the rig emulates (2.84 MB of
  stream past a 0x200000 ceiling). The gate now verifies 80 of 120 frames and
  SAYS so, and fails loudly when the pass does not complete, instead of
  reporting a phantom 49,005-pixel diff. FINDINGS 36.

Three near-misses this session had one shape: an unobservable run nearly
produced a false finding. stdbuf -oL on any MAME job that prints progress -- a
file is block-buffered too, and a run that is merely finishing looks exactly
like one that is wedged.

check.sh ALL GREEN.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 17:09:47 -07:00
parent 06b98d4b47
commit 7d365b3ff5
12 changed files with 664 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/STATUS.md
+106 -75
View File
@@ -1,93 +1,112 @@
# Status & next-session handoff — end of session 8 (2026-08-23)
# Status & next-session handoff — end of session 9 (2026-08-23)
## Where this stands
Session 7 left the decoder pixel-exact and **31% of frames over the CPU budget**
at `sasi`, 42% at `scsi` (FINDINGS 28), with two levers proposed and neither
measured. Session 8 did both.
Session 9 did three things: dropped a profile on the user's instruction, closed
the last encoder gap, validated the cost model against the machine — and then
found that **the CPU budget every one of those numbers was scored against has
never had the disk in it**.
**Lever A, spans: measured.** A row-linear span of word-expanded literals costs
**43.7 cycles per span + 9.152 per pixel** — but only in an encoder-assisted
format, `{u32 absolute GVRAM address, u16 jump displacement}` into an unrolled
copy chain. The obvious decoder, handed `(x, npix)`, is 97.9 + 10.46.
FINDINGS 30, `tools/bench/span.sh` (~25 s).
**FINDINGS 35 is the headline, and it is bad news.** Raised by the user ("PIO is
such a CPU killer. DMA is not. I'm concerned about us drawing the wrong
conclusions"). Every CPU figure in FINDINGS 24-34 is measured against the full
833,333 cycles/frame, with nothing subtracted for moving the bitstream off SCSI.
Debiting the HD63450 cycle-steal at the long-standing 8 clocks/word estimate:
**Lever B, the cost-aware mode decision: implemented, measured, and ON by
default.** `decide()` minimises `D + lam*bytes + mu*cycles`; `mu` is bisected
per frame against 833,333 cycles with the `lam` bisection nested inside it.
FINDINGS 31, `tools/analysis/13_cpu_ratectl.py`.
| `scsi` container | budget left | median frame | worst | frames missing |
|---|---:|---:|---:|---:|
| no I/O — the pre-session-9 premise | 833,333 | 99.6% | 110.6% | **1/120** |
| **DMA** (8 clk/word, ESTIMATED) | 738,234 | **112.4%** | 124.8% | **84/120** |
| PIO (12 clk/B, hand-derived floor) | 548,036 | 151.4% | 168.1% | **120/120** |
| `sasi`, worst sustained window | PSNR | KB/s | CPU median | CPU max | missing |
|---|---:|---:|---:|---:|---:|
| bytes only (session 7) | 27.22 dB | 109.5 | 74.4% | 136.2% | **37/120** |
| + cycle ceiling (now the default) | 26.95 dB | 109.4 | 81.5% | 110.6% | **1/120** |
| + ceiling + spans (MODEL, nothing implements it) | — | 449.3 | 56.8% | 91.5% | **0/120** |
So "1 frame of 120 misses" was against the wrong budget. `11_cpu_budget.py` now
takes `--io dma|pio|none`, defaults to `dma`, and warns if asked for `none`.
`scsi`: 51/120 -> 1/120 for 0.62 dB. The one remaining miss at either profile is
**frame 0**, which has no previous reconstruction and so is 100% changed by
definition — the same case as a scene cut. It is emitted late on purpose.
**The way out is 10 fps, and it works on paper**: same container, DMA debited,
10 fps -> median 93.7%, worst 104.0%, **1/120** — and that is conservative,
because it holds the 12 fps byte rate. FINDINGS 35.5.
The cost-aware container is verified pixel-exact on the 68000 (120 frames,
`tools/bench/verify_decode.py`).
**SASI is dropped (USER DECISION).** A SASI volume is 40 MB and the game's 22.8
minutes of unique footage is 146 MiB at the *lowest* rate this codec makes.
`scsi` is the only profile. Delivery is SD-backed SCSI, as locked in session 2 —
capacity does not bind there, and it is what rules CD-ROM out. FINDINGS 32.
**The encoder gap is closed.** `encode.py` emits **DLX2**, 4-byte-aligned frame
records: 94/120 record starts were on odd addresses, now 0/120, for 16 B/s.
Re-encoding reproduces FINDINGS 31.1 exactly. FINDINGS 33.
**The cost model is validated on a cost-aware container** (STATUS item 1, done):
errors -3.07% to +0.01% against the 68000, whole-window mean -1.22%. It
under-predicts light frames and is exact on heavy ones. FINDINGS 34.
**A `scsi` window does not fit the 2 MB machine the rig emulates** — 2.84 MB of
stream against a 0x200000 ceiling. The decoder gate now verifies 80 of 120
frames and says so, rather than failing as a phantom pixel diff. FINDINGS 36.
## NEXT SESSION, in order
0. **Green light first.** `./tools/bench/check.sh` (~3 min, Blu-ray mounted).
The drift stage now runs BOTH controllers; both must report 0/120.
0. **Green light first.** `./tools/bench/check.sh` (~5 min, Blu-ray mounted).
Verified green at end of session 9. The gate container is now
`tmp/rc_fr_singe_scsi_cpufit.dlx`, and it decodes **80 of 120 frames**, not
120: the rig loads the whole stream into a 2 MB machine and a `scsi` window
is 2.84 MB, which overran the top of RAM by 940 KB and produced a failure
that looked exactly like a decoder bug. `prep_dlx.py` now truncates to what
fits and says so; `verify_decode.py --nframes` replays the same prefix; and
the stage fails loudly if the sequential pass did not complete. FINDINGS 36.
1. **Time the cost-aware container on the 68000.** Everything in FINDINGS 31 is
the validated cost MODEL (`vq_hybrid.cycles`, within 1 point of the machine
on four frames of the session-7 container), not a measurement of this one.
The full timing pass was started and killed for time:
```
python3 tools/bench/prep_dlx.py tmp/rc_fr_singe_sasi_cpufit.dlx
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s
( cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -ramsize 2M \
-video soft -window -sound none -nothrottle -plugins \
-autoboot_script ../tools/bench/decode.lua -snapshot_directory ./snap_decode \
-snapview native -seconds_to_run 300 > decode_cpufit.log 2>&1 )
```
**Budget real time for it: the run was still going at 12 minutes of CPU.**
MAME's stdout is block-buffered to a file, so there is no progress to watch
— wait on the PID, never on a `pgrep -f` match (see the shell traps below).
Confirm the four anchors against `11_cpu_budget.py` on the same container,
and update FINDINGS 31 with measured-vs-model errors.
At 278.6 KB/s a 2 MB machine holds ~6.7 s of stream, so **the strongest test
in the tree can only ever audit a prefix of a window.** Getting the coverage
back means gating on more than one window, not one longer pass.
2. **Put spans in the bitstream.** This is the big one and it is now fully
specified by measurement: format in FINDINGS 30.2, costs in 30.5, and the
scene-cut arithmetic in 30.6. It touches `encode.py` (a fifth mode and a
run-aware decision), `dlx.py` (the reference decoder), and `decode.s`. The
24-pixel quantisation and the free row overrun are part of the format, not
optimisations to add later. Order it AFTER item 1 so the model that prices it
has been checked against the machine once more.
1. **Measure the DMA cycle-steal, or derive it from primary sources.** This is
now the most load-bearing unmeasured number in the project: at 8 clocks/word
the port is marginal, at 12 it is dead, at 4 it is comfortable. It has been
an estimate from a session-2 datasheet reading since FINDINGS 5. MAME cannot
answer it (functional models, not transfer-timing accurate) — this needs the
HD63450 and MB89352 datasheets or real hardware. **Everything below is
contingent on it.**
3. **The three open items of FINDINGS 29.5/30.7**, now load-bearing because a
span design runs at ~449 KB/s of a 488 KB/s pipe: re-run the ring-buffer
simulation at that rate (FINDINGS 21 was established at 110 and 280), confirm
the provenance of the user's 4 Mbps figure, and **confirm DMA rather than
PIO** — a PIO fallback puts a 449 KB/s transfer back on the CPU the whole
lever exists to relieve. The DMA check is the cheapest of the three and the
most consequential.
2. **Make sure the player actually gets DMA.** DMA-vs-PIO is a property of our
code, not the board: the CZ-6BS1's DMA path is real and modelled
(FINDINGS 32.4), but if the player reads through IOCS and IOCS does PIO we
get the 120/120 row. `docs/BENCHMARK.md` item 4 (drive the MB89352 directly)
is no longer an optimisation. Benchmark `x68000 -exp1 cz6bs1`, **never
`x68ksupr`** — MAME's internal SCSI has no DMA glue at all (`// TODO:
duplicate DMA glue from CZ-6BS1`) and would measure a PIO fallback the real
machine does not have.
4. **Encoder gap, still open from session 7:** `encode.py` should pad frame
records to 4 bytes. Frame boundaries land on odd addresses and a 68000 takes
an address error, not a slow read (FINDINGS 28.3). `prep_dlx.py` pads at load
time, which is why the decoder works; the container itself does not.
Measured cost of fixing it: 1.5 B/frame = 18 B/s.
3. **Re-decide the framerate, now that it is the lever that pays for the disk.**
10 fps absorbs the DMA steal on current estimates. This was item 5 and a
quality question; it is now arithmetic. It is still the user's call.
5. **A quality-vs-framerate question that is the user's, not the encoder's.**
Every miss is now one frame per cut. The options remain: one late frame at
each cut (the outgoing content is unrelated, so it may be invisible), a cut
spread over two frame times, or 10fps. Spans (item 2) make the question go
away if they land as modelled.
4. **Then spans.** Fully specified by measurement (format FINDINGS 30.2, costs
30.5, scene-cut arithmetic 30.6). Note what 35.2 does to the case for them:
spans buy cycles by spending bandwidth, and the bandwidth comes back as
steal, so 31.6's "fits completely" becomes a worst frame of 114.3%. They are
still worth 14 points at the median — they are just no longer sufficient
alone.
5. **Re-run the ring-buffer simulation at the surviving rate** and confirm the
488 KB/s figure's provenance (FINDINGS 29.5/30.7, still open).
**Do not start by hand-optimising `decode.s`.** The hand-derived timings agree
with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), so the inner
loop is close to what the instruction set allows.
with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), and FINDINGS
34 confirms the model on a second container. The inner loop is close to what the
instruction set allows; the cycles to be won are in the budget, not the loop.
**Three of this session's near-misses had the same shape: an unobservable run
almost produced a false finding.** The 23-minute "hang" (FINDINGS 34.1), the
RAM overrun that looked like a 4x-slow decoder (FINDINGS 36), and the truncated
gate that reported 49,005 differing pixels. In each case the instrument was
broken, not the thing being measured. **Always `stdbuf -oL` a MAME job that
prints progress.** A file is
block-buffered too, so a long run is unobservable until it exits — and an
unobservable run that is merely finishing looks exactly like one that is wedged.
Session 8 lost this measurement to that; session 9 lost 23 minutes to it before
spending 25 seconds getting the answer with `stdbuf`. FINDINGS 34.1.
---
## What session 8 settled
0. **The mode decision can see cycles, it is on by default, and it costs
@@ -245,22 +264,34 @@ The two session-7 stages are worth knowing the shape of before they fail on you:
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
| **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 |
| **Quality modes** | **two: `sasi` and `scsi`** (USER DECISION, session 2) | one codec, one decoder, one bitstream; only `lam` differs |
| **Quality modes** | **one: `scsi`** (USER DECISION, session 9 — `sasi` dropped) | a SASI volume is 40 MB and the game is 146 MiB at the LOWEST rate this codec makes. FINDINGS 32 |
| Delivery medium | **SD-backed SCSI** (BlueSCSI / SCSI2SD), as locked in session 2 | capacity does not bind on SD at any rate this codec makes; it is what killed SASI, and it is what rules CD-ROM OUT (650 MiB with spans, and 1x CD is ~150 KB/s). FINDINGS 32.3 |
| Profile axis | **I/O bandwidth only** | the profiles say nothing about CPU; **both target the same stock 10MHz 68000**, and the Super has SCSI at 10MHz. FINDINGS 28.7 |
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
| SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution |
### The SASI/SCSI question is RESOLVED
Session 1 left "which machine do we target" open. The user's answer: **ship both**,
as two quality profiles. This is now implemented rather than hypothetical — the
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
### The SASI/SCSI question is RESOLVED — and in session 9 SASI was DROPPED
Session 1 left "which machine do we target" open. Session 2's answer was **ship
both**, as two quality profiles. **Session 9 retired `sasi`** (USER DECISION) on
CAPACITY, not bandwidth: a SASI volume is limited to 40 MB, and the 22.8 minutes
of unique scene footage on the source Blu-ray is **146 MiB even at 110 KB/s** —
more than the machine's whole 4-unit SASI address space. FINDINGS 32.
Everything below this line about *two* profiles is the session-2..8 record, kept
because every measurement in FINDINGS 27-31 was taken against it. One profile
ships:
| profile | target | lam | quality (00020 / 00146) | machine |
|---|---|---|---|---|
| `sasi` | 110 KB/s | 60 (floor) | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
| ~~`sasi`~~ | ~~110 KB/s~~ | ~~60~~ | ~~36.9 / 29.6 dB~~ | **RETIRED session 9** |
| `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
The 110 KB/s **rate point** is not necessarily gone with the interface: a 1x
CD-ROM sustains ~150 KB/s and CD-ROM is the only period medium with room for the
whole game (374 MiB at `scsi`, 600 MiB with spans). The user's call was to ship
one profile now and settle the medium when the pipe is measured. FINDINGS 32.3.
**That "machine" column is about the BUS, not the CPU.** The profiles are an
I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has
built-in SCSI at 10 MHz (`x68k.cpp:1194`, `40_MHz_XTAL/4`, same as the base