drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Drop SASI on capacity, then find the budget never had the disk in it

Browse Source 
USER DECISION: drop the `sasi` profile. Not on bandwidth -- on capacity. A SASI
volume is 40 MB, and the 22.8 min of unique scene footage on the source Blu-ray
(streams 00000-00201, measured, not recalled) is 146 MiB at the LOWEST rate this
codec makes -- more than the machine's whole 4-unit SASI space. `scsi` is the
only profile now. FINDINGS 32.

Then the user asked whether we were drawing the wrong conclusions about PIO vs
DMA, and we were, more broadly than the question implied. Every CPU figure in
FINDINGS 24-34 is scored against the full 833,333 cycles/frame with nothing
subtracted for moving the bitstream off disk. Debiting the HD63450 cycle-steal
at the long-standing 8 clk/word ESTIMATE, "1 frame of 120 misses" becomes 84 of
120, median 112.4%. PIO at the span rate is 99.8% of the machine. Spans buy
cycles by spending bandwidth and the bandwidth returns as steal, so 31.6's "fits
completely" becomes a worst frame of 114.3%. 10 fps absorbs it: median 93.7%,
1/120. FINDINGS 35. `11_cpu_budget.py` takes --io dma|pio|none, defaults to dma,
and warns if asked for none.

Also landed:
- item 1 done: the cost model checked against the 68000 on a cost-aware
  container, -3.07% to +0.01%, whole-window mean -1.22%. FINDINGS 34.
- item 4 done: the container carries its own 4-byte record alignment (DLX2).
  94/120 record starts were on odd addresses -- an address error, not a slow
  read -- now 0/120 for 16 B/s. Re-encoding reproduces 31.1 exactly. FINDINGS 33.
- a `scsi` window does not fit the 2 MB machine the rig emulates (2.84 MB of
  stream past a 0x200000 ceiling). The gate now verifies 80 of 120 frames and
  SAYS so, and fails loudly when the pass does not complete, instead of
  reporting a phantom 49,005-pixel diff. FINDINGS 36.

Three near-misses this session had one shape: an unobservable run nearly
produced a false finding. stdbuf -oL on any MAME job that prints progress -- a
file is block-buffered too, and a run that is merely finishing looks exactly
like one that is wedged.

check.sh ALL GREEN.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 17:09:47 -07:00
parent 06b98d4b47
commit 7d365b3ff5
12 changed files with 664 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/FINDINGS.md
+346
View File
@@ -1783,3 +1783,349 @@ The intra frame lands at 91.5% — spans are what make a full redraw fit, which
30.6's arithmetic arriving in a real container. That row is still a **model** of
a bitstream nothing implements; the two levers have never run on the 68000
together, and the ring-buffer question of 30.7 gets sharper at 449 KB/s.
## 32. SASI is dropped, and the reason is capacity, not bandwidth (session 9)
**USER DECISION**: drop the `sasi` profile. A SASI volume on this machine is
limited to 40 MB, and the game does not fit in one.
That ends the two-quality-mode decision of session 2. `scsi` is now the only
profile, and `encode.py --profile` has one choice. The retired 110 KB/s rate
point is not deleted from the record, for the reason in 32.3.
### 32.1 How much video there actually is
Measured off the source Blu-ray rather than recalled: the unique scene footage
is streams `00000`-`00201`, **1366.6 s = 22.8 min**. The longer streams
(`00215` 1376 s, `00216` 1152 s, `00223` 566 s) are compilations of the same
material and are not additional content — 00223 is the window every codec
measurement in this project has been taken on. Total across all 224 streams is
88.3 min, which is the figure to *not* quote.
22.8 min agrees with the ~22 min of laserdisc footage the arcade original is
usually credited with, which is the cross-check that the compilations really
are duplicates.
At the rates this codec has actually produced, including the 7.8 KB/s audio
allowance:
| stream | rate | whole game |
|---|---:|---:|
| retired 110 KB/s profile | 109.4 KB/s | **146.0 MiB** |
| `scsi`, measured (FINDINGS 31) | 278.6 KB/s | **371.8 MiB** |
| `scsi` + spans (MODEL, 31.6) | 449.3 KB/s | **599.6 MiB** |
### 32.2 Where the 40 MB actually comes from
It is not a bus-addressing limit. MAME 0.277's `src/mame/sharp/x68k_hdc.cpp`
builds the SASI LBA from a 6-byte Group-0 CDB as
`(cmd[1] & 0x1f) << 16 | cmd[2] << 8 | cmd[3]` — **21 bits of 256-byte blocks,
so 512 MiB is addressable per unit**. `call_create` makes a 20 MB image
(`0x13c98` blocks) because that is what a period drive was.
So the 40 MB ceiling is a **Human68k / IPL volume-format and period-drive
limit**, not something the SASI command set imposes. That distinction does not
rescue the profile: four units at 40 MB is 160 MiB, and 146.0 MiB of video
would consume essentially the entire SASI address space of the machine at the
*lowest* rate this codec has ever produced, leaving nothing for Human68k, the
player, or the game's own data.
*Scope: the 21-bit CDB and the 256-byte block are read out of MAME's
implementation. The 40 MB volume figure is the user's, and is consistent with
Human68k's SASI partitioning; it has not been measured here.*
### 32.3 The rate point may come back, under a different name
Dropping SASI removes an interface, not a bitrate, and the two are on different
axes — the profile axis has been I/O bandwidth only since FINDINGS 28.7. The
awkward part is that **capacity and bandwidth now pull in opposite directions**:
- the only period medium with room for 371.8 MiB (let alone 599.6) is
**CD-ROM** at 540-650 MB, and
- a **1x CD-ROM sustains ~150 KB/s**, which is *below* the surviving 280 KB/s
profile and much nearer the rate that was just retired.
A SCSI hard disk has the bandwidth but has to be large for the era at 372 MiB.
**The user's call was to ship `scsi` as the only profile now and settle the
medium when the pipe is measured** — the blocked disk benchmark
(`docs/BENCHMARK.md`) and the DMA-vs-PIO check of FINDINGS 29.5.
**Correction to the framing above, found after that call was made.** The
medium is less open than this section first presented it. FINDINGS 21.2 already
committed the deployment target to **SD-backed SCSI (BlueSCSI / SCSI2SD)**, in
session 2, and that is the premise the whole 488 KB/s constant rests on. On SD
there is no capacity problem at any rate this codec produces — an SD card is
gigabytes — and no seek tail either. So:
- **Capacity does not choose between the survivors.** It killed SASI, whose
40 MB ceiling is a Human68k volume-format limit that SD emulation does not
lift, and it does not bind on SD-backed SCSI at all.
- **CD-ROM is the one that capacity rules out**, not in. With spans the stream
is 487.1 KB/s = **650.1 MiB** for the whole game, past a CD's ~620 MiB
usable — and 487 KB/s is more than 3x a 1x CD-ROM's ~150 KB/s. A CD-ROM
delivery would mean giving up the span lever *and* re-deriving a profile
around 150 KB/s.
So the open question is not "which medium" but the one FINDINGS 29.5/30.7
already had: **confirm the 488 KB/s figure's provenance, and confirm DMA**.
`profile_gen.py` exists precisely to re-derive a profile from a measured
bandwidth once there is one.
### 32.4 What MAME says about the SCSI path that survives
Read out of MAME 0.277 while settling 32.2, and directly relevant because the
medium decision is now the thing gating the profile:
**The CZ-6BS1's DMA is real and fully modelled.** `x68k_scsiext.cpp` wires the
MB89352's DREQ to the expansion slot and replaces the data register at
`$EA0015` with DMA-aware glue: on a DMA cycle (`m_slot->exown()`, driven by
`m_hd63450->own()`) a read goes to `spc->dma_r()` and #DTACK is negated until
DRQ asserts. `x68k.cpp:1114` closes the loop the other way
(`out_dtack_callback -> hd63450_device::dtack_w`). That is a genuine
DMAC-driven transfer with hardware flow control, on the **stock `x68000`
driver** — the one MAME marks working. This is the configuration FINDINGS 29.5
asked about, and the answer for this board is **DMA, not PIO**.
**The internal SCSI of the Super/XVI/030 is NOT modelled that way**, and it is
a trap. `x68k.cpp:1176` reads, verbatim, `// TODO: duplicate DMA glue from
CZ-6BS1`. So MAME's internal SCSI is PIO-only. A benchmark run on `x68ksupr`
would measure a PIO fallback the real machine does not have — on top of those
drivers already being MACHINE_NOT_WORKING (FINDINGS 28.7). **Benchmark
`x68000 -exp1 cz6bs1`, not `x68ksupr`.**
**CD-ROM is a first-class SCSI device on the internal bus** — `x68k.cpp:1168`
puts an `NSCSI_CDROM` at ID 6 by default — but the CZ-6BS1 card's own device
list offers `harddisk` only. So the CD-ROM delivery route of 32.3 is
emulatable, but not on the board whose DMA is modelled, without a source
change.
None of this is a transfer RATE. `docs/BENCHMARK.md`'s split still holds and is
worth restating because 32.3 defers a decision to a measurement: MAME can
settle whether the path works and whether it is DMA, and **cannot** settle
KB/s, because its device models are functional rather than transfer-timing
accurate. The rate half of the medium question needs derivation or real
hardware, not a longer MAME run.
## 33. The container carries its own alignment: DLX1 -> DLX2 (session 9)
The encoder gap left open since session 7 (FINDINGS 28.3, STATUS item 4) is
closed. `encode.py` now emits **DLX2**, which pads every frame record up to a
4-byte boundary — the first one included, by padding the codebook tables so
`off_frm` is aligned. `dlx.py` reads both versions; DLX1 containers stay
readable because every measurement in FINDINGS 28-31 was taken on one.
Measured on the same 120-frame window:
| | record starts not 4-aligned | padding cost |
|---|---:|---:|
| DLX1 (through session 8) | **94/120** | 0 (the loader added 180 B) |
| DLX2 (now) | **0/120** | 160 B = 1.33 B/frame = **16 B/s** |
16 B/s against 278.6 KB/s is 0.006% of the stream. The thing it buys is not
speed: an odd `move.l (a0)+` on a 68000 is an **address error**, which vectors
into the IPL and presents as an infinite loop, not as a slow read. That is the
bug that cost session 7 an afternoon.
`tools/bench/prep_dlx.py` still realigns at load time and now says whether it
had to — `0/120 record starts unaligned -- the container carries its own
padding` on a DLX2 input. It is kept rather than deleted because it is what
makes the session 7-8 containers decodable, and those are the containers the
published timings belong to.
**Cross-check that this changed nothing else:** re-encoding the `scsi` window
with the DLX2 writer reproduces FINDINGS 31.1 exactly — 29.27 dB, 278.6 KB/s,
median 99.6% / max 110.6% of a 12fps frame, 1/120 frames missing. The padding
is additive; it does not touch the mode decision.
## 34. The cost model, checked against the machine on a cost-aware container (session 9)
STATUS item 1. Everything in FINDINGS 31 was the validated cost MODEL
(`vq_hybrid.cycles`) applied to a container it had never been checked against —
the 1-point validation of 28.2 belongs to the *session 7* stream. This is the
cost-aware container timed on the emulated 68000, same harness, same scope
(instruction cycles, zero-wait-state GVRAM, interrupts masked; a LOWER BOUND).
| anchor | non-SKIP | model | measured | error |
|---|---:|---:|---:|---:|
| min non-SKIP | 15.4% | 254,683 cyc / 30.6% | 262,751 / 31.5% | **-3.07%** |
| median | 53.2% | 681,199 / 81.7% | 690,251 / 82.8% | **-1.31%** |
| p90 | 75.7% | 832,116 / 99.9% | 834,213 / 100.1% | **-0.25%** |
| max non-SKIP | 100.0% | 921,293 / 110.6% | 921,187 / 110.5% | **+0.01%** |
| whole 120-frame mean | — | 649,089 / 77.9% | 657,081 / 78.8% | **-1.22%** |
The model holds, and its error is **signed**: it under-predicts by 1-3% on light
frames and converges to exact on heavy ones. That is the right direction to be
wrong in for a ceiling controller — the bisection is tightest where the model is
most accurate — but it means the median frame is ~1 point dearer than FINDINGS
31 reports, not cheaper.
The four synthetic single-mode frames reproduce session 7 exactly: all-V1
110.5%, all-V4 165.2%, all-RAW 147.6%, all-SKIP 4.9%. Those are properties of
`decode.s`, not of the container, so agreeing across two different streams is
the cross-check that the harness is measuring what it claims.
### 34.1 The 23-minute "hang" was the buffering trap again
The session-8 note said this run "was still going at 12 minutes of CPU". It was
re-run here and sat at 99.9% CPU for **23 minutes** with a 0-byte log, then was
killed. Re-launched under **`stdbuf -oL`** with `-seconds_to_run 60`, the
identical plan completed in about **25 seconds of wall time** and printed every
line as it went — MAME reports `Average speed: 528.72% (52 seconds)`, so the
whole plan needs ~52 emulated seconds and the machine runs it at 5x realtime.
The lesson is the one already in STATUS, one level deeper: it is not enough to
write MAME's output to a file instead of a pipe. **A file is block-buffered too**,
so a long MAME run is unobservable until it exits, and an unobservable run that
is merely finishing looks exactly like one that is wedged. Session 8 lost the
measurement to that, and session 9 lost 23 minutes to it before spending 25
seconds getting the answer. **`stdbuf -oL` on every MAME job that prints
progress.**
## 35. The CPU budget has never had the disk in it (session 9)
**Raised by the user: "PIO is such a CPU killer. DMA is not. I'm concerned about
us drawing the wrong conclusions."** The concern is correct, and it is larger
than the labelling question of 32.4. This is the seventh false premise this
project has caught, and the most expensive one.
Every CPU figure in FINDINGS 24 through 34 is measured against **833,333 cycles
per frame**, the full 10 MHz clock divided by 12 fps. Nothing has ever been
subtracted from it for moving the bitstream off the disk. The decoder has been
scored as though the data arrives for free.
### 35.1 What the transfer actually costs
`profile_gen.py` has carried `DMA_CLOCKS_PER_WORD = 8` since session 2 (FINDINGS
5, an **ESTIMATE** from HD63450 timing, never measured) and prints a "DMA steal"
line — but that line was only ever compared against the 38.3% *blit* figure of
FINDINGS 17, which FINDINGS 24 superseded and which was never the decoder cost.
It was never debited from the decoder budget.
At the rates that matter, on a 10 MHz 68000:
| stream | DMA @ 8 clk/word | PIO, unrolled (~12 clk/B) | PIO, byte loop (~20 clk/B) |
|---|---:|---:|---:|
| `scsi`, 278.6 KB/s | **11.4%** | 34.2% | 57.1% |
| `scsi` + spans, 487.1 KB/s | **20.0%** | 59.9% | **99.8%** |
The PIO columns are hand-derived floors, not measurements: a byte from an I/O
register plus a store is 16 cycles on a 68000 before any loop overhead. They are
here to size the risk, and the size of the risk is that **PIO at the span rate
consumes the entire machine**.
### 35.2 What that does to the conclusions of FINDINGS 31
Debiting the DMA steal — the *cheap* case, the one we are hoping for:
| | KB/s | steal | budget left | median | p90 | worst | fits? |
|---|---:|---:|---:|---:|---:|---:|:--|
| `scsi` today | 278.6 | 11.4% | 738,238 | **112.4%** | 112.9% | 124.8% | **no** |
| `scsi` + spans | 487.1 | 20.0% | 667,070 | **98.3%** | 102.8% | 114.3% | **no** |
FINDINGS 31's headline — "1 frame of 120 misses" — is measured against a budget
with no I/O in it. With DMA debited the surviving profile does not fit at all:
the *median* frame is over. And 31.6's "with spans the window fits completely"
becomes a worst frame of 114.3%, because **the span lever buys cycles by
spending bandwidth, and the bandwidth comes back out of the CPU as steal.**
Spans still help — 112.4% -> 98.3% at the median, 14 points — but they no longer
close the gap on their own.
### 35.3 Why this is not settled by the DMA finding of 32.4
32.4 established that the CZ-6BS1's DMA *path* exists and is modelled. Three
things it does not establish, and all three are load-bearing:
1. **DMA vs PIO is a property of OUR player, not of the board.** The hardware
supports DMA; if the player reads through IOCS and IOCS does PIO, we get PIO
and the table above. `docs/BENCHMARK.md` item 4 already proposed driving the
MB89352 registers directly for exactly this reason — that is now not an
optimisation but the difference between fitting and not.
2. **8 clocks per word has never been measured.** It is now the single most
load-bearing unmeasured number in the project: at 8 the port is marginal, at
12 it is dead, at 4 it is comfortable. It comes from a datasheet reading in
session 2 and nothing has checked it since.
3. **MAME cannot settle it.** Its device models are functional, not
transfer-timing accurate (BENCHMARK.md), and it models no GVRAM wait states
either — so a MAME run can confirm the transfer is a DMA cycle and cannot
price it. This needs derivation from the HD63450 and MB89352 datasheets, or
real hardware.
### 35.4 What this does and does not overturn
It does **not** overturn the decoder measurements: 300/448/400 cycles per block
and the model validation of FINDINGS 34 are properties of `decode.s` and stand
unchanged. What it overturns is every statement of the form "N frames of 120
miss the budget", because the budget was wrong. Those all need re-running
against `833,333 * (1 - steal)` once `steal` is a measurement rather than a
datasheet estimate.
It also sharpens the framerate question of STATUS item 5 considerably. At 10 fps
the budget is 1,000,000 cycles and the same DMA steal is proportionally smaller
per frame, which is now a much stronger argument for 10 fps than "one late frame
per cut" ever was.
### 35.5 `11_cpu_budget.py` now debits it, and 10 fps absorbs it
The tool takes `--io dma|pio|none` (default **dma**) and prints the budget it is
actually scoring against. On `tmp/rc_fr_singe_scsi_cpufit.dlx`:
| `--io` | budget left | median | worst | frames missing |
|---|---:|---:|---:|---:|
| `none` — the pre-session-9 premise | 833,333 | 99.6% | 110.6% | **1/120** |
| `dma` (8 clk/word, estimated) | 738,234 | 112.4% | 124.8% | **84/120** |
| `pio` (12 clk/B, floor) | 548,036 | 151.4% | 168.1% | **120/120** |
`--io none` prints a warning naming FINDINGS 35, so the old number cannot be
produced by accident.
**At 10 fps and DMA the same container goes back to 1/120** — median 93.7%,
worst 104.0%. That is conservative, because it holds the 12 fps byte rate: a
real 10 fps encode carries ~17% fewer bytes per second, so the steal falls too.
This changes what the framerate decision (STATUS item 5) is *for*. It was a
quality question about one late frame per scene cut. It is now the lever that
pays for the disk, and on current estimates it is the difference between a
stream that fits and one that misses 70% of its frames.
## 36. A `scsi` window does not fit in the machine the test rig emulates (session 9)
Swapping the decoder gate onto the surviving profile's container made it fail —
`frame 119 not pixel-exact: 49,005 px differ`. That is not a decoder bug and not
the DLX2 change. **The container does not fit in RAM.**
`tools/bench/decode.lua` loads the entire stream into emulated memory at
`STREAM = 0x30000`, and the locked target is a stock **2 MB** machine:
| container | stream | ends at | verdict |
|---|---:|---:|---|
| session 7-8 `sasi` | 1,108,888 B | 0x13EB98 = 1.25 MB | fits |
| `scsi` cost-aware | 2,840,860 B | 0x2E591C = 2.90 MB | **overruns 0x200000 by 940 KB** |
The loader wrote 940 KB past the top of memory, the decoder then parsed whatever
that reads back as, and the run neither completed its sequential pass nor drew
the right picture. Every 68000 decode verification before session 9 was done on
a container small enough to fit by accident — the `sasi` profile was a third the
bitrate, so nobody met this.
**This is a property of the test rig, not of the player.** The shipping player
streams from disk into a ring buffer and holds seconds of video, not minutes.
But it does bound what the rig can prove: at 278.6 KB/s, a 2 MB machine holds
about **6.7 seconds** of stream, so the strongest test in the tree can only ever
audit a prefix of a window.
The fix keeps the test honest rather than making it pass:
- `prep_dlx.py` truncates the frame list to what fits, **prints that it did**,
and takes `--ram` / `--all-frames`. On the `scsi` window it keeps **80 of
120** frames.
- `verify_decode.py` takes `--nframes` so the reference decoder replays exactly
the prefix the 68000 decoded, instead of running 40 frames ahead of it.
- `check.sh` reads the count back out of `decode_meta.lua` and passes it through,
and now **fails loudly if the sequential pass did not complete** — the missing
`snapshot taken` marker — instead of reporting a pixel diff against a
half-drawn screen. That guard is what turned this from a mystery into a
five-minute diagnosis.
Verifying a prefix is still a real test: SKIP blocks make every frame a claim
about the one before it, so frame 79 is only correct if all 80 were. What is
lost is coverage of the last 40 frames, and the honest way to get it back is to
gate on more than one window rather than to pretend one pass covers everything.
**The timing confirms the diagnosis.** Truncated to 80 frames the pass completes
in **8 emulated seconds** and the frame is pixel-exact; the model predicts
~6.6 s for 80 frames at this container's cost, so that is the expected number.
The 120-frame run that overran RAM could not finish the same work in **44**.
A decoder reading garbage does not run slowly for an interesting reason — it was
parsing lengths out of unmapped memory and walking wherever they pointed. Any
"the decoder is 4x slower than the model on RAW-heavy streams" conclusion drawn
from that run would have been entirely false, which is the third time in this
session that an unobservable run nearly produced a wrong finding.