prosolis
fb8a1462b0
The previous commit warned that scsi "does not fit 4 Mbps" because a frame peaked at 96.4% of the sustained rate. That was the wrong test, and the user was right to push back on it. It compared instantaneous frame demand against a sustained rate as if they had to match frame-by-frame; the disk keeps filling during the frame, and a ring buffer absorbs any shortfall. Correct test is cumulative demand vs cumulative supply. Simulated on the real per-frame sizes at a constant 488 KB/s fill: required prefill = 0.0 KB for EVERY scene at both profiles Fill delivers 40.69 KB per frame time; only one measured frame (00146, 42.10 KB) exceeds it and the next frame recovers it. A 256 KB buffer -- 12.5% of RAM -- carries ~1 second of stall tolerance, orders of magnitude more than an SD-backed seek needs. scsi at lam=10 stands; the hardest sampled scene runs 313 KB/s mean, 64% of the pipe. Also carries through a consequence of SD-backed deployment that session 1 noted as a caveat but never applied: with BlueSCSI/SCSI2SD the sustained rate is a bus-limited CONSTANT, not an average over variable seek latency, so there is no long tail to leave margin for and we can size much closer to the ceiling. Conservative margins here were cargo-culted from a constraint this deployment does not have. The SASI/SCSI split remains correct because it is about bus protocol, not media. Rate control drops from load-bearing to insurance: intra-scene peaks are a non-problem, but only 4 clips of 1.2-1.7s out of 224 streams have been measured and 00146 already runs 23% hotter than 00020. The full-disc survey is now the highest-value measurement, ahead of wiring up ratectl. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
171 lines
8.5 KiB
Markdown
171 lines
8.5 KiB
Markdown
# Status & next-session handoff — end of session 2 (2026-08-23)
|
|
|
|
## Decisions locked
|
|
|
|
| decision | value | why |
|
|
|---|---|---|
|
|
| Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint |
|
|
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
|
|
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
|
|
| **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 |
|
|
| **Quality modes** | **two: `sasi` and `scsi`** (USER DECISION, session 2) | one codec, one decoder, one bitstream; only `lam` differs |
|
|
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
|
|
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
|
|
| SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution |
|
|
|
|
### The SASI/SCSI question is RESOLVED
|
|
Session 1 left "which machine do we target" open. The user's answer: **ship both**,
|
|
as two quality profiles. This is now implemented rather than hypothetical — the
|
|
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
|
|
|
|
| profile | target | lam | quality (00020 / 00146) | machine |
|
|
|---|---|---|---|---|---|
|
|
| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
|
|
| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
|
|
|
|
Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on
|
|
SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**,
|
|
not an average over seek latency.
|
|
|
|
**Both profiles fit with room.** Ring-buffer simulation on the real per-frame
|
|
sizes gives **zero required prefill** for every scene at both profiles: the fill
|
|
delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds
|
|
that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance,
|
|
far more than an SD-backed seek needs. FINDINGS 21.
|
|
|
|
An earlier warning here said `scsi` did not fit because a frame peaked at 96.4%
|
|
of the pipe. That compared instantaneous demand to a sustained rate as if they
|
|
had to match frame-by-frame; with a buffer the test is cumulative, and it passes.
|
|
|
|
`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
|
|
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
|
|
RD curve rather than derived from the hardware. See FINDINGS 17.
|
|
|
|
Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured
|
|
and rejected — see FINDINGS 14, it was a false-good result from a rate model
|
|
that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff.
|
|
|
|
Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised
|
|
frame (measured 0.00 dB loss). The profiles are two points on one continuous
|
|
rate-distortion curve, not two codecs.
|
|
|
|
---
|
|
|
|
## What session 2 settled
|
|
|
|
1. **The critical-path question is answered.** "Does VQ soften Bluth's linework
|
|
unacceptably?" — **flat 4x4 k=256 VQ: yes, badly. Hybrid VQ with k=1024: no.**
|
|
Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11.
|
|
2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured).
|
|
Halving the framerate does not halve the bitrate. FINDINGS 8.
|
|
2b. **A fourth false-good result was produced and caught this session** — k=1024
|
|
codebooks looked like a +2.4 dB free win because the rate model charged 1 byte
|
|
for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
|
|
3. **The 256-colour palettised frame is the real quality ceiling** and it looks
|
|
excellent. Judge the codec against that, not against 1080p.
|
|
4. Encoder exists and produces a real bitstream: `tools/encoder/`.
|
|
|
|
---
|
|
|
|
## Encoder — working
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
|
|
python3 tools/encoder/encode.py /tmp/fr_00020 out.dlx --profile sasi --preview p.png
|
|
```
|
|
|
|
| file | role |
|
|
|---|---|
|
|
| `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise |
|
|
| `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR |
|
|
| `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision |
|
|
| `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control |
|
|
| `encode.py` | CLI + `DLX1` container writer |
|
|
|
|
`DLX1` container layout is documented in the `encode.py` docstring. All
|
|
multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`.
|
|
|
|
### Known encoder gaps
|
|
- **Rate control is written but not yet wired into `encode.py`** — the CLI uses a
|
|
fixed `lam` from the profile. `ratectl.encode_rate_controlled()` exists and
|
|
builds a lam-ladder per frame; it needs hooking up and validating.
|
|
- **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in
|
|
the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
|
|
- Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
|
|
- `_paint` is a Python per-block loop — fine for prototyping, slow for a full
|
|
disc encode. Vectorise before the 224-stream run.
|
|
|
|
---
|
|
|
|
## Working setup (unchanged from session 1, re-verified)
|
|
|
|
**MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**.
|
|
```
|
|
mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3
|
|
```
|
|
**Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`
|
|
|
|
**Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM`
|
|
(still mounted as of end of session 2).
|
|
|
|
**MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the
|
|
notifier subscription in a global; the stack register is `SP` not `A7`;
|
|
`autoboot_script` fires at PC=0 before boot) are documented in FINDINGS.
|
|
|
|
**Two shell traps, both hit again this session:**
|
|
- piping MAME (or any long job) through `grep` block-buffers — write to a file.
|
|
- `pkill -f <pattern>` matches your own shell and kills it (exit 144).
|
|
Use `pkill -x` or kill by PID.
|
|
|
|
---
|
|
|
|
## STILL BLOCKED: disk throughput benchmark
|
|
|
|
Unchanged from session 1 — `IOCS _B_READ` returns -1 uniformly. Full diagnosis
|
|
and the four untested hypotheses are in session 1's notes (git history of this
|
|
file, commit 65112b9).
|
|
|
|
**This is now CRITICAL-PATH, not optional.** Session 1 dismissed it because
|
|
"VQ at 30 KB/s is correct whether SASI does 300 or 600 KB/s". That reasoning is
|
|
dead: the profiles now sit at 110 and 280 KB/s, close enough to the folklore
|
|
ceilings that the error bars change the product. Specifically —
|
|
|
|
**If SCSI sustains >=800 KB/s, the correct `scsi` profile is `lam=0`: pixel-exact
|
|
video, ~450 KB/s, and only 38% of the CPU budget.** Whether this port ships
|
|
transparent or lossy on SCSI is waiting on one measurement.
|
|
|
|
Next move is the untried SCSI path: `-exp1 cz6bs1 -hard disk.chd`.
|
|
|
|
---
|
|
|
|
## Next steps, in priority order
|
|
|
|
1. **Full-disc survey** — we have measured 4 clips of 1.2-1.7 s out of 224
|
|
streams, and 00146 already runs 23% hotter than 00020. A sustained action
|
|
sequence is the one thing that could still break the bitrate. This is now
|
|
the highest-value measurement, ahead of rate control.
|
|
2. **Wire rate control into `encode.py`** — no longer a blocker (FINDINGS 21),
|
|
but it is what gives a deterministic ceiling over content we have not
|
|
measured, which was the original reason for choosing VQ. Insurance, not a fix.
|
|
3. ~~Entropy-code the payload~~ — **ABANDONED, see FINDINGS 17.2.** Deflate
|
|
decode is ~216% of the frame budget on a 68000 and LZ4 is ~54%; there is no
|
|
room beside a 38% blit. All bitrates are raw payload. This also demotes the
|
|
"247 KB/s lossless" figure in FINDINGS 8 to a compression bound, not a design.
|
|
3. **68000 decoder skeleton**: parse `DLX1`, expand codebooks to word-per-pixel,
|
|
blit V1/V4/RAW/SKIP. Measure real cycles with the existing MAME Lua harness —
|
|
this is the first time the harness gets used for its actual purpose.
|
|
4. **Full-disc survey** — classify menu vs content first (FINDINGS 13), then
|
|
measure bitrate across all 224 streams per profile.
|
|
5. **Resolve the framing question** (FINDINGS 12: crop vs squash vs wide).
|
|
6. Unblock the disk benchmark via the SCSI path, then re-set profile bitrates.
|
|
7. Import the SNES project's `data/events/` (MIT, cleared) as the scene graph.
|
|
Cross-check against DirkSimple (zlib) which has the same data independently.
|
|
8. ADPCM audio: MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in `ratectl`,
|
|
not yet extracted or encoded.
|
|
|
|
## Not yet started
|
|
- Any 68000 player code
|
|
- ADPCM audio extraction/encoding
|
|
- Disk image packaging
|
|
- Game logic (scene branching, input windows, death clips)
|