Raise both quality profiles; rule out entropy coding on CPU grounds

The profiles shipped in e4062ed were set far too low. 45 KB/s (sasi) and
75 KB/s (scsi) are 12% and 7% of the respective folklore bus figures. They had
been read off the knee of the rate-distortion curve and then presented as
though bandwidth-derived, which they were not.

Raised to sasi 110 KB/s (lam=60) and scsi 280 KB/s (lam=10) -- 35% and 28%
utilisation. scsi is now within 0.52 dB of the palette ceiling on scene 00020.

Checking the CPU side, which nobody had done for the decode path, produces a
second and more important result. Against the 833k cycle/frame budget at 12fps:

  full-frame blit, every frame     319k   38%   affordable
  LZ4/LZSS decode ~30KB/frame      450k   54%
  deflate decode  ~30KB/frame     1800k  216%   infeasible

So raising the VQ bitrate is nearly free -- RAW, the mode that dominates at
high rate, is the cheapest mode to blit -- but entropy coding is not viable at
all. That demotes the "247 KB/s lossless changed-spans+deflate" figure from
FINDINGS 8 to a compression upper bound rather than a shippable design, and
removes entropy coding from the roadmap. VQ is the right architecture precisely
because its decode is a table copy.

Also confirms the architecture unifies: the hybrid at lam=0 lands within 3% of
the purpose-built lossless coder, so there is no separate lossless path.

Consequence for planning: the blocked disk benchmark is now critical-path, not
optional. If SCSI sustains >=800 KB/s the correct scsi profile is lam=0 --
pixel-exact video at ~450 KB/s and 38% CPU. Whether this port ships transparent
or lossy on SCSI is waiting on one measurement.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 12:06:52 -07:00
parent e4062ed294
commit f0f2f807a4
3 changed files with 129 additions and 25 deletions
+24 -13
View File
@@ -18,10 +18,14 @@ Session 1 left "which machine do we target" open. The user's answer: **ship both
as two quality profiles. This is now implemented rather than hypothetical — the
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
| profile | target | lam | quality (00020 / 00146) | machine |
|---|---|---|---|---|
| `sasi` | 45 KB/s | 300 | 34.8 / 28.3 dB | stock 10MHz ACE/EXPERT |
| `scsi` | 75 KB/s | 100 | 35.9 / 29.0 dB | Super/XVI, or CZ-6BS1 board |
| profile | target | lam | quality (00020 / 00146) | bus utilisation | machine |
|---|---|---|---|---|---|
| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | 35% of 300 KB/s | stock 10MHz ACE/EXPERT |
| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | 28% of 1 MB/s | Super/XVI, or CZ-6BS1 board |
`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
RD curve rather than derived from the hardware. See FINDINGS 17.
Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured
and rejected — see FINDINGS 14, it was a false-good result from a rate model
@@ -71,8 +75,8 @@ multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`
- **Rate control is written but not yet wired into `encode.py`** — the CLI uses a
fixed `lam` from the profile. `ratectl.encode_rate_controlled()` exists and
builds a lam-ladder per frame; it needs hooking up and validating.
- **Payload is not entropy-coded.** Deflate on the payload should buy ~1.4x
(measured on the lossless path, FINDINGS 8). LZ decode is cheap on a 68000.
- **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in
the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
- Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
- `_paint` is a Python per-block loop — fine for prototyping, slow for a full
disc encode. Vectorise before the 224-stream run.
@@ -107,12 +111,16 @@ Unchanged from session 1 — `IOCS _B_READ` returns -1 uniformly. Full diagnosis
and the four untested hypotheses are in session 1's notes (git history of this
file, commit 65112b9).
**This now matters more than session 1 thought.** Session 1 dismissed it because
"VQ at 30 KB/s is correct whether SASI does 300 or 600 KB/s". But we now ship
*two profiles*, and the profile bitrates (45 / 120 KB/s) are set against
**folklore** bandwidth figures. A real measurement would let us set them
honestly instead of conservatively. Next move is the untried SCSI path:
`-exp1 cz6bs1 -hard disk.chd`.
**This is now CRITICAL-PATH, not optional.** Session 1 dismissed it because
"VQ at 30 KB/s is correct whether SASI does 300 or 600 KB/s". That reasoning is
dead: the profiles now sit at 110 and 280 KB/s, close enough to the folklore
ceilings that the error bars change the product. Specifically —
**If SCSI sustains >=800 KB/s, the correct `scsi` profile is `lam=0`: pixel-exact
video, ~450 KB/s, and only 38% of the CPU budget.** Whether this port ships
transparent or lossy on SCSI is waiting on one measurement.
Next move is the untried SCSI path: `-exp1 cz6bs1 -hard disk.chd`.
---
@@ -120,7 +128,10 @@ honestly instead of conservatively. Next move is the untried SCSI path:
1. **Wire rate control into `encode.py`** and validate that the hard ceiling
actually holds on an action scene (the whole point of choosing VQ).
2. **Entropy-code the payload** (deflate) — ~1.4x for cheap 68000 decode cost.
2. ~~Entropy-code the payload~~**ABANDONED, see FINDINGS 17.2.** Deflate
decode is ~216% of the frame budget on a 68000 and LZ4 is ~54%; there is no
room beside a 38% blit. All bitrates are raw payload. This also demotes the
"247 KB/s lossless" figure in FINDINGS 8 to a compression bound, not a design.
3. **68000 decoder skeleton**: parse `DLX1`, expand codebooks to word-per-pixel,
blit V1/V4/RAW/SKIP. Measure real cycles with the existing MAME Lua harness —
this is the first time the harness gets used for its actual purpose.