Files

T

prosolis fb8a1462b0 Correct the peak analysis: both profiles fit 4 Mbps with zero prefill

The previous commit warned that scsi "does not fit 4 Mbps" because a frame
peaked at 96.4% of the sustained rate. That was the wrong test, and the user
was right to push back on it. It compared instantaneous frame demand against a
sustained rate as if they had to match frame-by-frame; the disk keeps filling
during the frame, and a ring buffer absorbs any shortfall.

Correct test is cumulative demand vs cumulative supply. Simulated on the real
per-frame sizes at a constant 488 KB/s fill:

  required prefill = 0.0 KB for EVERY scene at both profiles

Fill delivers 40.69 KB per frame time; only one measured frame (00146,
42.10 KB) exceeds it and the next frame recovers it. A 256 KB buffer -- 12.5%
of RAM -- carries ~1 second of stall tolerance, orders of magnitude more than
an SD-backed seek needs. scsi at lam=10 stands; the hardest sampled scene runs
313 KB/s mean, 64% of the pipe.

Also carries through a consequence of SD-backed deployment that session 1 noted
as a caveat but never applied: with BlueSCSI/SCSI2SD the sustained rate is a
bus-limited CONSTANT, not an average over variable seek latency, so there is no
long tail to leave margin for and we can size much closer to the ceiling.
Conservative margins here were cargo-culted from a constraint this deployment
does not have. The SASI/SCSI split remains correct because it is about bus
protocol, not media.

Rate control drops from load-bearing to insurance: intra-scene peaks are a
non-problem, but only 4 clips of 1.2-1.7s out of 224 streams have been measured
and 00146 already runs 23% hotter than 00020. The full-disc survey is now the
highest-value measurement, ahead of wiring up ratectl.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6

2026-08-23 12:23:08 -07:00

8.5 KiB

Raw Blame History

Status & next-session handoff — end of session 2 (2026-08-23)

Decisions locked

decision	value	why
Target CPU	68000 @ 10MHz (stock)	hardest honest constraint
Display mode	256 colors, 256x192 in 256x256 CRTC mode	every mode is 1 word-access/pixel, so 256c is free vs 16c
Double buffer	none — page 1 sacrificed	enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway
Codec	hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion	flat 4x4 VQ was measured and rejected — see FINDINGS 9-10
Quality modes	two: `sasi` and `scsi` (USER DECISION, session 2)	one codec, one decoder, one bitstream; only `lam` differs
Framerate	12 fps, explicit decimation	source has zero duplicate frames; no free "twos" win
Emulator	MAME 0.277 x68000	accurate enough that measured cycles mean something
SNES project reuse	MIT — cleared	`data/events/` scene graph is reusable with attribution

The SASI/SCSI question is RESOLVED

Session 1 left "which machine do we target" open. The user's answer: ship both, as two quality profiles. This is now implemented rather than hypothetical — the bitrate ceiling is a build parameter in tools/encoder/ratectl.py:

profile	target	lam	quality (00020 / 00146)	machine
`sasi`	110 KB/s	60	36.9 / 29.6 dB	stock 10MHz ACE/EXPERT
`scsi`	280 KB/s	10	39.4 / 32.3 dB	Super/XVI, or CZ-6BS1 board

Sized against the user's working figure of 4 Mbps = 488 KB/s sustained, on SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited constant, not an average over seek latency.

Both profiles fit with room. Ring-buffer simulation on the real per-frame sizes gives zero required prefill for every scene at both profiles: the fill delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance, far more than an SD-backed seek needs. FINDINGS 21.

An earlier warning here said scsi did not fit because a frame peaked at 96.4% of the pipe. That compared instantaneous demand to a sustained rate as if they had to match frame-by-frame; with a buffer the test is cumulative, and it passes.

scsi is now within 0.5 dB of the palette ceiling on 00020. These were initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the RD curve rather than derived from the hardware. See FINDINGS 17.

Codebooks are k=256 with 1-byte indices in both profiles. k=1024 was measured and rejected — see FINDINGS 14, it was a false-good result from a rate model that undercharged the index. Do not ship past lam~800; FINDINGS 15 has the cliff.

Because of the RAW escape mode, lam=0 is pixel-exact against the palettised frame (measured 0.00 dB loss). The profiles are two points on one continuous rate-distortion curve, not two codecs.

What session 2 settled

The critical-path question is answered. "Does VQ soften Bluth's linework unacceptably?" — flat 4x4 k=256 VQ: yes, badly. Hybrid VQ with k=1024: no. Verified by eye, not just PSNR. See docs/FINDINGS.md 9-11.
Session 1's 12fps bitrate was wrong (183 KB/s claimed, 340 KB/s measured). Halving the framerate does not halve the bitrate. FINDINGS 8. 2b. A fourth false-good result was produced and caught this session — k=1024 codebooks looked like a +2.4 dB free win because the rate model charged 1 byte for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
The 256-colour palettised frame is the real quality ceiling and it looks excellent. Judge the codec against that, not against 1080p.
Encoder exists and produces a real bitstream: tools/encoder/.

Encoder — working

python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
python3 tools/encoder/encode.py  /tmp/fr_00020 out.dlx --profile sasi --preview p.png

file	role
`extract.py`	.m2ts -> 256x192 PNGs, 12fps, spatial-only denoise
`vq.py`	palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR
`vq_hybrid.py`	the codec: 4 block modes + lagrangian mode decision
`ratectl.py`	SASI/SCSI profiles, leaky-bucket rate control
`encode.py`	CLI + `DLX1` container writer

DLX1 container layout is documented in the encode.py docstring. All multi-byte fields are big-endian so the 68000 reads them with a plain move.

Known encoder gaps

Rate control is written but not yet wired into encode.py — the CLI uses a fixed lam from the profile. ratectl.encode_rate_controlled() exists and builds a lam-ladder per frame; it needs hooking up and validating.
Payload is deliberately NOT entropy-coded — deflate decode does not fit in the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
_paint is a Python per-block loop — fine for prototyping, slow for a full disc encode. Vectorise before the 224-stream run.

Working setup (unchanged from session 1, re-verified)

MAME ROMs — ~/mame/roms/x68000.zip. Must pass -bios ipl10.

mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3

Assembler — tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s

Blu-ray — udisksctl loop-setup -r -f DRAGONS_LAIR.iso -> /media/reala-misaki/BDROM (still mounted as of end of session 2).

MAME Lua harness — tools/bench/*.lua, working. Three gotchas (retain the notifier subscription in a global; the stack register is SP not A7; autoboot_script fires at PC=0 before boot) are documented in FINDINGS.

Two shell traps, both hit again this session:

piping MAME (or any long job) through grep block-buffers — write to a file.
pkill -f <pattern> matches your own shell and kills it (exit 144). Use pkill -x or kill by PID.

STILL BLOCKED: disk throughput benchmark

Unchanged from session 1 — IOCS _B_READ returns -1 uniformly. Full diagnosis and the four untested hypotheses are in session 1's notes (git history of this file, commit 65112b9).

This is now CRITICAL-PATH, not optional. Session 1 dismissed it because "VQ at 30 KB/s is correct whether SASI does 300 or 600 KB/s". That reasoning is dead: the profiles now sit at 110 and 280 KB/s, close enough to the folklore ceilings that the error bars change the product. Specifically —

If SCSI sustains >=800 KB/s, the correct scsi profile is lam=0: pixel-exact video, ~450 KB/s, and only 38% of the CPU budget. Whether this port ships transparent or lossy on SCSI is waiting on one measurement.

Next move is the untried SCSI path: -exp1 cz6bs1 -hard disk.chd.

Next steps, in priority order

Full-disc survey — we have measured 4 clips of 1.2-1.7 s out of 224 streams, and 00146 already runs 23% hotter than 00020. A sustained action sequence is the one thing that could still break the bitrate. This is now the highest-value measurement, ahead of rate control.
Wire rate control into encode.py — no longer a blocker (FINDINGS 21), but it is what gives a deterministic ceiling over content we have not measured, which was the original reason for choosing VQ. Insurance, not a fix.
~~Entropy-code the payload~~ — ABANDONED, see FINDINGS 17.2. Deflate decode is ~216% of the frame budget on a 68000 and LZ4 is ~54%; there is no room beside a 38% blit. All bitrates are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8 to a compression bound, not a design.
68000 decoder skeleton: parse DLX1, expand codebooks to word-per-pixel, blit V1/V4/RAW/SKIP. Measure real cycles with the existing MAME Lua harness — this is the first time the harness gets used for its actual purpose.
Full-disc survey — classify menu vs content first (FINDINGS 13), then measure bitrate across all 224 streams per profile.
Resolve the framing question (FINDINGS 12: crop vs squash vs wide).
Unblock the disk benchmark via the SCSI path, then re-set profile bitrates.
Import the SNES project's data/events/ (MIT, cleared) as the scene graph. Cross-check against DirkSimple (zlib) which has the same data independently.
ADPCM audio: MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in ratectl, not yet extracted or encoded.

Not yet started

Any 68000 player code
ADPCM audio extraction/encoding
Disk image packaging
Game logic (scene branching, input windows, death clips)

8.5 KiB Raw Blame History