drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Handoff: reconcile docs and tooling with the corrections made this session

Browse Source 
Session 2 reversed several of its own conclusions. The docs are append-only, so
a reader could land on a superseded section and act on it. This pass makes the
repo internally consistent.

Defects found and fixed in STATUS.md:
- claimed "Hybrid VQ with k=1024: no" as the answer to the linework question,
  directly contradicting FINDINGS 14, which rejected k=1024. Both profiles are
  k=256.
- malformed profile table (six column separators, five columns).
- next-steps list had two items numbered 3 and listed the full-disc survey
  twice.
- the disk-benchmark section still read CRITICAL-PATH with "if SCSI sustains
  >=800 KB/s, ship pixel-exact". That was written while the bandwidth figure
  was misread as 4 MB/s. At 4 Mbps pixel-exact needs 92-97% of the pipe and is
  not available, and the ring-buffer result means the design no longer hangs on
  the benchmark at all. Rewritten with what it IS still worth doing: confirming
  the 4 Mbps provenance, and confirming DMA is used rather than PIO.

FINDINGS now carries supersession blockquotes on 5, 8, 11, 17 and 18 pointing
at the sections that correct them. 18 is the dangerous one -- its peak-vs-
sustained test is reversed by 21 -- so it is marked DO NOT ACT ON THIS SECTION
while noting the per-frame data itself remains valid.

profile_gen.py had the same problem in code: it defaulted to the superseded
peak sizing and returned lam=25 where the docs say lam=10. The buffered test is
now the default and peak sizing is behind --size-for-peak as a bound only. A
tool that contradicts the findings is worse than no tool.

Also preserves the five measurement scripts that produced this session's
numbers as tools/analysis/05-09, following the session 1 precedent, and adds an
"explicitly abandoned -- do not re-propose" list to STATUS covering entropy
coding, k=1024 codebooks and flat 4x4 VQ.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 12:28:41 -07:00
parent fb8a1462b0
commit 64cd1ffd72
10 changed files with 345 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/BENCHMARK.md
+20 -12
View File
@@ -106,20 +106,24 @@ FINDINGS 17.4:
python3 tools/encoder/profile_gen.py --bw-mbps 4 --name scsi
```
It accounts for the three things that eat the pipe before video sees any of it:
audio (7.8 KB/s), peak-to-mean burstiness (measured 1.4-1.9x), and it reports
the DMA cycle-steal so the CPU coupling stays visible.
It accounts for what eats the pipe before video sees any of it: audio
(7.8 KB/s), the buffering condition, and it reports the DMA cycle-steal so the
CPU coupling stays visible.
At 4 Mbps it currently returns:
At 4 Mbps it returns:
| | lam | mean | peak | 00020 | 00146 | CPU |
|---|---|---|---|---|---|---|
| today (no rate control) | 25 | 194 KB/s | 368 KB/s | -1.22 dB | -4.21 dB | 53% |
| with rate control wired | 10 | 305 KB/s | 305 KB/s | -0.52 dB | -2.98 dB | 51% |
| sizing rule | lam | mean | 00020 | 00146 | CPU |
|---|---|---|---|---|---|
| **buffered (default, FINDINGS 21)** | **10** | 305 KB/s | -0.52 dB | -2.98 dB | 51% |
| `--size-for-peak` (FINDINGS 18, superseded) | 25 | 194 KB/s | -1.22 dB | -4.21 dB | 46% |
**Rate control is worth a full step on the quality ladder** — it is not a
tidiness feature, it is the difference between sizing for the peak and sizing
for the mean. That is the strongest argument yet for wiring up `ratectl.py`.
The default is the buffered test: cumulative demand vs cumulative supply.
Ring-buffer simulation gives **zero required prefill** for every measured scene,
so `lam=10` ships without rate control. `--size-for-peak` reproduces the earlier
pessimistic sizing and is kept only as a bound.
**Rate control is therefore insurance, not a fix.** Its value is a deterministic
ceiling over the 220 streams not yet measured — see the survey caveat below.
## What would change the design
@@ -127,7 +131,11 @@ for the mean. That is the strongest argument yet for wiring up `ratectl.py`.
collapses toward today's `sasi`, and the two profiles stop being meaningfully
different. At that point reconsider 10 fps, or a narrower active area.
- **If sustained is much above** (>=8 Mbps / 976 KB/s): `lam=0` fits with
margin and the port ships **pixel-exact** video on SCSI.
margin and the port ships **pixel-exact** video on SCSI. At the current
4 Mbps figure this is NOT available — `lam=0` needs 92-97% of the pipe.
- **If the full-disc survey finds a sustained action sequence hotter than
00146** (313 KB/s mean, the worst of 4 clips sampled): that is the scenario
rate control exists for, and the reason to wire it up before the survey run.
- **If DMA cannot be used** and transfers fall back to PIO, the CPU cost rises
from ~15% to something far larger and CPU becomes the binding constraint.
This is the single worst outcome and is worth checking early in Tier 1.
docs/FINDINGS.md
+24
View File
@@ -103,6 +103,10 @@ regression references; 03 and 04 are the correct ones.
## 5. Storage interface — the SASI/SCSI split
> **SUPERSEDED IN PART.** The claim below that DMA means streaming "costs
> essentially no CPU" is **wrong** — see 19. The bandwidth figures here are
> folklore; the working figure is now **4 Mbps = 488 KB/s** (21).
[Yasuma, X68030 internal SCSI controller]
- Interface: **SCSI-1**, 50-pin, 5 MB/s bus spec
@@ -196,6 +200,11 @@ Caveat: all of the above is from README/repo-tree summaries, not their source.
## 8. CORRECTION to session 1: halving the framerate does NOT halve the bitrate
> **PARTLY SUPERSEDED.** The framerate correction stands. The
> "changed-spans + deflate = 247 KB/s" figure is a **compression upper bound,
> not a shippable design** — deflate decode does not fit the 68000's frame
> budget. See 17.2.
Session 1 measured 365 KB/s for naive delta+RLE at 24 fps and wrote
"(~183 KB/s at 12fps)". **That extrapolation is wrong.** Decimating to 12 fps
roughly doubles the per-frame delta, so the *rate* stays nearly flat.
@@ -267,6 +276,9 @@ should: under a hard ceiling, detail blocks are the first thing sacrificed.
## 11. Codebook size sweep (flat 4x4, for reference)
> **SUPERSEDED.** The k=1024 result below is an artifact of a rate model that
> charged 1 byte for a 10-bit index. k=256 ships. See 14.
| block | k | PSNR | loss | key B | changed% | KB/s @12 | codebook RAM |
|---|---|---|---|---|---|---|---|
| 4x4 | 256 | 30.46 | 8.39 | 3072 | 52.7 | 28.5 | 8K |
@@ -373,6 +385,12 @@ transcription errors before any of it is committed to 68000 tables.
## 17. The profiles were set far too low — and entropy coding is a CPU trap
> **PARTLY SUPERSEDED.** 17.1's diagnosis (the profiles were not derived from
> hardware) and 17.2's CPU analysis both stand. But 17 reasoned against a
> misread bandwidth of 4 MB/s; the correct figure is **4 Mbps = 488 KB/s**, so
> the "ship pixel-exact if SCSI sustains >=800 KB/s" conclusion in 17.5 is
> **not available**. See 18 and 21.
Prompted by the user asking why the SCSI profile was only 75 KB/s. It should not
have been. Two separate errors, one of them serious.
@@ -453,6 +471,12 @@ SCSI is now waiting on one measurement.
## 18. Peak-to-mean burstiness — the mean was hiding the problem
> **SUPERSEDED — DO NOT ACT ON THIS SECTION.** The peak-vs-sustained comparison
> below is the **wrong test**. With a ring buffer the correct test is cumulative
> demand vs cumulative supply, and both profiles pass it with **zero required
> prefill**. `scsi` at lam=10 ships. See 21. The per-frame peak numbers
> themselves are still valid data; only the conclusion drawn from them is wrong.
Prompted by the user clarifying that the bandwidth figure is **4 Mbps = 488 KB/s**,
not 4 MB/s. That is ~8x tighter than what 17 was reasoning against, and it
changes the answer.
docs/STATUS.md
+56 -38
View File
@@ -19,7 +19,7 @@ as two quality profiles. This is now implemented rather than hypothetical — th
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
| profile | target | lam | quality (00020 / 00146) | machine |
|---|---|---|---|---|---|
|---|---|---|---|---|
| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
@@ -54,8 +54,9 @@ rate-distortion curve, not two codecs.
## What session 2 settled
1. **The critical-path question is answered.** "Does VQ soften Bluth's linework
unacceptably?" — **flat 4x4 k=256 VQ: yes, badly. Hybrid VQ with k=1024: no.**
Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11.
unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.**
Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two
images in `docs/images/`. Both profiles use **k=256**; see item 2b.
2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured).
Halving the framerate does not halve the bitrate. FINDINGS 8.
2b. **A fourth false-good result was produced and caught this session** — k=1024
@@ -119,49 +120,66 @@ notifier subscription in a global; the stack register is `SP` not `A7`;
---
## STILL BLOCKED: disk throughput benchmark
## Disk throughput benchmark — still blocked, no longer gating
Unchanged from session 1 — `IOCS _B_READ` returns -1 uniformly. Full diagnosis
and the four untested hypotheses are in session 1's notes (git history of this
file, commit 65112b9).
`IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested
hypotheses are in session 1's notes (git history of this file, commit 65112b9);
the ordered plan for retrying is in **`docs/BENCHMARK.md`**.
**This is now CRITICAL-PATH, not optional.** Session 1 dismissed it because
"VQ at 30 KB/s is correct whether SASI does 300 or 600 KB/s". That reasoning is
dead: the profiles now sit at 110 and 280 KB/s, close enough to the folklore
ceilings that the error bars change the product. Specifically —
**Status changed twice this session — read this rather than the git history.**
It was briefly promoted to critical-path while the working bandwidth figure was
misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the
ring-buffer simulation showing **zero required prefill** for both profiles
(FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not**
available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a
"measure it and maybe ship transparent" decision waiting.
**If SCSI sustains >=800 KB/s, the correct `scsi` profile is `lam=0`: pixel-exact
video, ~450 KB/s, and only 38% of the CPU budget.** Whether this port ships
transparent or lossy on SCSI is waiting on one measurement.
What the benchmark is still worth doing for:
- **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is
not recorded. Every profile hangs off it.
- **Confirming DMA is actually used.** If transfers fall back to PIO the CPU
cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the
binding constraint. This is the worst plausible outcome and the cheapest to
check — do it first.
Next move is the untried SCSI path: `-exp1 cz6bs1 -hard disk.chd`.
---
**Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are
functional models, not timing-accurate; a KB/s figure from MAME measures the
emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach
(MAME validates the path, derivation bounds it, real hardware settles it).
## Next steps, in priority order
1. **Full-disc survey** — we have measured 4 clips of 1.2-1.7 s out of 224
streams, and 00146 already runs 23% hotter than 00020. A sustained action
sequence is the one thing that could still break the bitrate. This is now
the highest-value measurement, ahead of rate control.
2. **Wire rate control into `encode.py`** — no longer a blocker (FINDINGS 21),
but it is what gives a deterministic ceiling over content we have not
measured, which was the original reason for choosing VQ. Insurance, not a fix.
3. ~~Entropy-code the payload~~ — **ABANDONED, see FINDINGS 17.2.** Deflate
decode is ~216% of the frame budget on a 68000 and LZ4 is ~54%; there is no
room beside a 38% blit. All bitrates are raw payload. This also demotes the
"247 KB/s lossless" figure in FINDINGS 8 to a compression bound, not a design.
3. **68000 decoder skeleton**: parse `DLX1`, expand codebooks to word-per-pixel,
blit V1/V4/RAW/SKIP. Measure real cycles with the existing MAME Lua harness —
this is the first time the harness gets used for its actual purpose.
4. **Full-disc survey** — classify menu vs content first (FINDINGS 13), then
measure bitrate across all 224 streams per profile.
1. **Full-disc survey.** Only 4 clips of 1.2-1.7 s out of 224 streams have been
measured, and 00146 already runs 23% hotter than 00020. A *sustained* action
sequence is the one thing that could still break the bitrate. Classify menu
vs content first (FINDINGS 13) or the averages are diluted by static menus.
**Vectorise `_paint` before this run** — it is a Python per-block loop.
2. **68000 decoder skeleton.** Parse `DLX1`, expand codebooks to word-per-pixel,
blit SKIP/V1/V4/RAW. Measure real cycles with the existing MAME Lua harness —
the first time that harness gets used for its actual purpose. Validates the
38% full-frame blit estimate that the whole CPU budget rests on.
3. **Wire rate control into `encode.py`.** No longer a blocker (FINDINGS 21), but
it is what gives a deterministic ceiling over content not yet measured, which
was the original reason for choosing VQ. Insurance, not a fix. Pairs with (1).
4. **Confirm DMA vs PIO in MAME** (see the benchmark section above) — cheap, and
the only thing that could still move CPU into the binding position.
5. **Resolve the framing question** (FINDINGS 12: crop vs squash vs wide).
6. Unblock the disk benchmark via the SCSI path, then re-set profile bitrates.
7. Import the SNES project's `data/events/` (MIT, cleared) as the scene graph.
Cross-check against DirkSimple (zlib) which has the same data independently.
8. ADPCM audio: MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in `ratectl`,
not yet extracted or encoded.
Needs an eyeball against arcade reference, not a measurement.
6. **Import the scene graph.** SNES project `data/events/` (MIT, cleared),
cross-checked against DirkSimple (zlib) which transcribed the same data
independently — diff them to catch transcription errors before committing
any of it to 68000 tables.
7. **ADPCM audio.** MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in
`ratectl.py`, not yet extracted or encoded.
### Explicitly abandoned — do not re-propose
- ~~Entropy-code the payload.~~ Deflate decode is ~216% of the frame budget on a
68000; LZ4 is ~54% with no room beside a 38% blit (FINDINGS 17.2). All bitrates
are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8
to a compression upper bound, not a shippable design.
- ~~k=1024 codebooks.~~ False-good result from a rate model that charged 1 byte
for a 10-bit index (FINDINGS 14). k=256 wins at every matched bitrate.
- ~~Flat 4x4 VQ.~~ Rejected by eye (FINDINGS 9).
## Not yet started
- Any 68000 player code