prosolis
3641f37e28
The codec was designed when bytes were scarce, so every decision in it trades
cycles to save bytes. That is now backwards: sasi spends 110 KB/s of a 488 KB/s
pipe while missing 31% of frames on CPU.
The cheapest thing a 68000 can be handed is the most expensive thing to store.
Measured, per pixel: row-linear copy from word-expanded memory 9.08 cycles,
block-order 12.98, V1 codebook 18.74, RAW byte literals 25.03. So the 1024-byte
stride costs 43% and unpacking bytes to words costs more than the write itself.
Pricing one new mode -- a per-row span of word-expanded literals movem.l'd
straight from the stream buffer -- against the UNCHANGED mode maps:
sasi median 74.4% -> 43.0%, worst 136.2% -> 106.2%, misses 37 -> 8/120,
101.7 -> 453.2 KB/s
scsi median 94.9% -> 69.4%, misses 51 -> 18/120, 272 -> 479.7 KB/s
scsi gains less precisely because it has less idle bandwidth left to trade.
Two consequences worth flagging. A word-expanded literal block derives to ~240
cycles, cheaper than V1's measured 299.9 and pixel-exact -- so every codebook
mode is CPU-dominated by a literal, and the codebook is a byte optimisation
that now costs cycles. And 28.5's "a scene cut cannot fit at 12fps" reopens:
CPU needs >=19% of the frame as spans, the bus allows <=39%, and that interval
is not empty.
DERIVED, NOT MEASURED, and labelled as such everywhere. The 9.08 cycles/pixel
is real but was measured at full row width with 12-register bursts, so short
spans are flattered. Measuring one span on the 68000 is now step 0 of the next
session, ahead of the cost-aware mode decision, because it changes the mode set
that decision optimises over.
FINDINGS 29. tools/analysis/12_span_tradeoff.py.
Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
807 lines
45 KiB
Markdown
807 lines
45 KiB
Markdown
# Status & next-session handoff — end of session 7 (2026-08-23)
|
|
|
|
## NEXT SESSION: measure a span, then make the mode decision cost-aware
|
|
|
|
The decoder exists, it is pixel-exact, and **it does not fit**. On the worst
|
|
sustained window at `sasi` it costs a mean of **81.7% of a 12fps frame** and
|
|
**31% of frames exceed 100%** (`scsi`: 94.9% median, 42% miss). FINDINGS 28.
|
|
CPU is the binding constraint now — the first time in this project.
|
|
|
|
**Two levers, and the cheap one has to be measured first.**
|
|
|
|
*Lever A — spend bandwidth to buy cycles.* The bus sits 4x idle: `sasi` uses 110
|
|
KB/s of 488. Every codec decision was made when bytes were scarce, so each one
|
|
trades cycles to save them, and the cheapest thing a 68000 can be handed is the
|
|
most expensive thing to store — **word-expanded pixels in row-linear runs**.
|
|
Adding one mode, a per-row span of literal words `movem.l`-ed straight from the
|
|
stream buffer into GVRAM, prices out at (FINDINGS 29, `12_span_tradeoff.py`):
|
|
|
|
| | today | + literal spans |
|
|
|---|---:|---:|
|
|
| median frame | 74.4% | **43.0%** |
|
|
| worst frame | 136.2% | **106.2%** |
|
|
| frames missing | **37/120** | **8/120** |
|
|
| bitrate | 101.7 KB/s | 453.2 KB/s (bus 488) |
|
|
|
|
**This is DERIVED, not measured, and it is load-bearing — so measure it first.**
|
|
Extend `tools/bench/blit.s` with a span variant and time it against run length.
|
|
The 9.08 cycles/pixel it rests on is real (FINDINGS 24 V1) but was measured at
|
|
full row width with 12-register bursts; short and oddly-aligned spans cannot
|
|
burst as well and are flattered by the model. If spans come in near the derived
|
|
figure, the whole mode set changes and lever B optimises over different modes —
|
|
which is exactly why this goes first. FINDINGS 29.5 lists the other three things
|
|
that have to hold, of which **confirming DMA vs PIO is the cheapest and now the
|
|
most consequential**: at 453 KB/s a PIO fallback puts the transfer back on the
|
|
CPU this is trying to relieve.
|
|
|
|
*Lever B — stop buying modes the CPU cannot afford.* `vq_hybrid.decide()`
|
|
minimises `D + lam*R` — distortion against BYTES — on a machine whose binding
|
|
budget is CYCLES, and the two are not proportional:
|
|
|
|
| mode | payload bytes | measured cycles | cycles per byte |
|
|
|---|---:|---:|---:|
|
|
| SKIP | 0 | 13 (clustered) | — |
|
|
| V1 | 1 | 300 | 300 |
|
|
| V4 | 4 | 448 | 112 |
|
|
| RAW | 16 | 400 | 25 |
|
|
| *word-expanded literal block* | *32* | *~240 (derived)* | *7.5* |
|
|
|
|
V4 is **25% of blocks and 50% of the cycles**. The lagrangian charges it 4x a V1
|
|
block; the CPU charges it 1.49x. Note the last row: a literal block is cheaper
|
|
than **every** codebook mode, and pixel-exact — the codebook is a byte
|
|
optimisation that now costs cycles (FINDINGS 29.2).
|
|
|
|
**The work, in order:**
|
|
|
|
0. **Measure the span cost on the 68000** (lever A above). Cheap, and everything
|
|
below optimises over whatever mode set it leaves.
|
|
|
|
1. **Add a cycle term to the mode decision.** `decide()` already builds a cost
|
|
matrix of `error + lam * bytes` per mode per block; add `+ mu * cycles`,
|
|
with the per-mode cycles measured in FINDINGS 28.2.
|
|
**SKIP is not a constant and this is the one trap here.** A SKIP block costs
|
|
13.25 cycles when all four blocks in its header byte are SKIP (one `tst.b`
|
|
clears the group) and ~45 when it sits in a mixed byte — so SKIP's price
|
|
depends on its *neighbours*, which a per-block lagrangian cannot see. Do not
|
|
pick one number and move on: 45 overcharges clustered SKIPs and pushes the
|
|
encoder away from the mode that saves cycles, 13.25 undercharges isolated
|
|
ones and lets frames overrun. The way out is that the **budget check does not
|
|
have to use the same cost function as the mode decision** — score frames with
|
|
the exact clustered cost (`cycles()` in `tools/analysis/11_cpu_budget.py`,
|
|
validated to 1 point against the 68000) and let the bisection converge on
|
|
that, while the per-block term uses a constant purely to *rank* modes.
|
|
2. **Then bisect `mu` per frame against the 833,333-cycle budget**, exactly as
|
|
session 6 bisects `lam` against the byte budget. The machinery is already
|
|
there and already gated: `ratectl.encode_rate_controlled` is frame-driven and
|
|
feeds back the frame it emitted. **But cycles have NO bucket.** Bytes can be
|
|
banked in the ring buffer; a frame that misses its decode deadline is just
|
|
late, because there is no double buffer to decode ahead into. So this is a
|
|
hard per-frame ceiling, not a leaky bucket — simpler than rate control, and
|
|
the two controllers have to run together (raising `mu` moves blocks to SKIP
|
|
and V1, which also *lowers* the bitrate, so the byte controller must see it).
|
|
3. **Measure the quality cost.** Everything session 6 did for bytes: what does
|
|
fitting 100% of frames in the CPU budget cost in dB, and does any frame hit a
|
|
cliff? `tools/analysis/11_cpu_budget.py` scores a container without needing
|
|
MAME, so the search loop is cheap; confirm the winner on the 68000 with
|
|
`tools/bench/decode.lua`.
|
|
|
|
3b. **Know which misses are yours to fix before starting.** Re-coding every
|
|
non-SKIP block as V1 is the floor any mode assignment can reach, and it
|
|
still misses 11 frames at `sasi` and 12 at `scsi` — every frame above ~90%
|
|
non-SKIP. So the cost-aware decision can reach about three quarters of the
|
|
misses (26 of 37 at `sasi`) and the rest are item 4. FINDINGS 28.7.
|
|
|
|
3c. **Buy RAW, not V4, wherever the bytes allow.** RAW is 400 cycles against
|
|
V4's 448 *and* is pixel-exact, so on the CPU axis V4 is strictly dominated —
|
|
the byte lagrangian's preference inverts. `scsi` can take that escape and
|
|
`sasi` cannot afford it, so expect the cycle ceiling to cost `sasi` more
|
|
quality even though it costs `sasi` fewer cycles. FINDINGS 28.8.
|
|
|
|
4. **Scene cuts: 28.5 said impossible, 29.4 reopened it.** An all-V1 frame —
|
|
the cheapest full redraw the *current* mode set allows — is 110.5% of budget,
|
|
so no mode assignment fits a 100%-changed frame. With literal spans the
|
|
arithmetic changes: CPU needs at least 19% of the frame sent as spans, the
|
|
bus allows up to 39%, **and that interval is not empty**. So 28.5 was a
|
|
ceiling of the bitstream, not of the machine — *if* lever A measures out.
|
|
If it does not, this is still a design decision that needs the user: one late
|
|
frame at each cut (the outgoing content is unrelated, so it may be
|
|
invisible), a cut spread over two frame times, or 10fps.
|
|
|
|
**Do not start by hand-optimising `decode.s`.** The hand-derived timings agree
|
|
with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), so the
|
|
inner loop is close to what the instruction set allows; the plausible wins are
|
|
single-digit percentages against a 36-point gap. The V4 write pattern is the one
|
|
place worth a look afterwards — pairing sub-block rows into `movem.l d0/d2,(a4)`
|
|
saves ~16 of 448 cycles.
|
|
|
|
---
|
|
|
|
## What session 7 settled
|
|
|
|
1. **68000 code parses a bitstream and draws frames, pixel-exact.**
|
|
`src/player/decode.s` + `tools/bench/decode.lua`. 120 frames of the Singe
|
|
window decoded in sequence, all four block modes, verified against the new
|
|
reference decoder `tools/encoder/dlx.py`. Because SKIP blocks are claims
|
|
about the previous frame, the last frame is only right if all 120 were.
|
|
In `check.sh` now. **FINDINGS 28.**
|
|
2. **It does not fit.** Mean 81.7% of a 12fps frame, p90 116.4%, worst 135.8%;
|
|
31% of frames miss at `sasi`, 42% at `scsi`. Zero-wait-state floor, as ever.
|
|
3. **The dual-display-path plan (FINDINGS 24.5/25.6) is withdrawn as incoherent
|
|
— the sixth false premise this project has caught.** The compose path needs a
|
|
RAM copy of the previous reconstruction; the direct path's whole selling
|
|
point is that it keeps none. Mixing them displays stale pixels on **70 of 120
|
|
frames**, worst frame 18.8% of the screen. Every coherent repair is worse
|
|
than not mixing. `tools/analysis/10_pathmix_drift.py`, kept runnable as a
|
|
counterexample and gated in `check.sh`. FINDINGS 28.1.
|
|
4. **24.5 also compared a copy against a copy.** Its 53.6% and 76.6% both come
|
|
from `blit.s` and neither includes decoding. Compose = decode-into-RAM *plus*
|
|
the 53.6% blit, so it is strictly dearer than decoding into GVRAM. There was
|
|
never a crossover. The player has **one path and no reference frame**, which
|
|
also gives back 96 KB.
|
|
5. **The four block modes cost 300 / 448 / 400 cycles, not one number.** V4 is
|
|
1.49x a V1 block while the mode decision charges it 4x the bytes. The 24.5
|
|
model is 2.03x optimistic at the median. `tools/analysis/11_cpu_budget.py`
|
|
reproduces all four frames timed on the 68000 to within 1 point. FINDINGS 28.2.
|
|
6. **The container is big-endian but not aligned, and on a 68000 that is an
|
|
address error, not a slow read.** Frame records are variable-length and laid
|
|
end to end, so their boundaries land on odd addresses. Frame 0 decoded
|
|
perfectly, then the length read for frame 1 vectored into the IPL and sat
|
|
there for 59 emulated seconds looking like an infinite loop. Found by dumping
|
|
PC and the address registers — the code was right, the data layout was not.
|
|
FINDINGS 28.3. **Encoder gap: `encode.py` should pad records to 4 bytes.**
|
|
Measured cost 1.66 B/frame = 20 B/s against 110 KB/s.
|
|
7. **A full frame does not fit at 12fps in any mode.** All-V1 is 110.5%, all-V4
|
|
165.2%, all-RAW 147.6%. At most ~88% of the screen can change in one frame
|
|
however cheaply it is coded, and scene cuts change 100%. FINDINGS 28.5.
|
|
|
|
---
|
|
|
|
## What session 6 settled
|
|
|
|
1. **Rate control works, is wired in, and is ON by default.** `encode.py`
|
|
bisects lam per frame under a leaky bucket; `--fixed-lam` restores session 5
|
|
behaviour. FINDINGS 27.
|
|
2. **Both overshoots are closed for under 1 dB.** On the Singe window, totals
|
|
including audio: `sasi` 137.4 -> **109.5 KB/s** (target 110) for -0.60 dB,
|
|
`scsi` 381.6 -> **280.0 KB/s** (target 280) for -0.91 dB. Zero frames hit the
|
|
lam=800 cliff at either profile. FINDINGS 27.2.
|
|
3. **The FINDINGS 26 desync is gone by construction, not by tuning.** The
|
|
encoder is frame-drivable (`vq_hybrid.frame_ctx` / `decide` / `paint`) and
|
|
rate control feeds back the frame it actually emitted. The regression test
|
|
`tools/analysis/09_ratectl_drift.py` goes 111/120 drifting frames -> **0**,
|
|
and it is now part of `./tools/bench/check.sh`. FINDINGS 27.1.
|
|
4. **Rate control makes the display path cheaper.** Raising lam moves blocks to
|
|
SKIP and V1, so there is less to write: `scsi`'s median display cost drops
|
|
53.6% -> 47.1%. The decoder conclusion of 25.6 is unaffected. FINDINGS 27.3.
|
|
5. **FINDINGS 26.5 was wrong in both halves, and this is the fifth false premise
|
|
this project has caught.** `_paint` was not the bottleneck (14% of a frame)
|
|
and the ladder was never "minutes" (~18 s; the minutes were k-means in
|
|
`build`). Vectorising it was still right — 17.1x — but what actually makes
|
|
per-frame rate control affordable is that `VQ.assign` depends on neither
|
|
`lam` nor `prev`, so it is cached: a 12-step search over 120 frames costs
|
|
**0.31 s** against 49.1 s. FINDINGS 27.6.
|
|
6. **`--prefill` is a trap and defaults to 0.** It buys a permission to overshoot
|
|
of exactly bucket/nframes; at prefill=1.0 the Singe window goes to 116.3 KB/s
|
|
against a 110 ceiling, and on a 14-frame clip it disables the controller
|
|
outright. FINDINGS 27.4.
|
|
7. **Fixed-lam `sasi` was already 5% over target on 00020**, the clip everyone
|
|
called easy — nothing noticed because the profile table quotes PSNR, not
|
|
bitrate. FINDINGS 27.5.
|
|
8. **1.2-second clips cannot be used to judge rate control.** The bucket's
|
|
startup transient is bucket/nframes: 6% on a 10 s window, 20% on 00020. Same
|
|
lesson as FINDINGS 25.3, different costume.
|
|
|
|
---
|
|
|
|
## Start here: is the tree still green?
|
|
|
|
```
|
|
./tools/bench/check.sh
|
|
```
|
|
~3 min, needs the Blu-ray mounted. From source media it re-runs both display
|
|
regression tests, the rate-control drift test (session 6), the display-path
|
|
coherency counterexample and a **120-frame 68000 decode** (session 7), then
|
|
prints `ALL GREEN`. Verified green at end of session 7.
|
|
If it fails, fix that before doing anything else — everything downstream assumes
|
|
the display path is pixel-exact.
|
|
|
|
The two session-7 stages are worth knowing the shape of before they fail on you:
|
|
- `10_pathmix_drift.py` is expected to exit **non-zero**; `check.sh` fails if it
|
|
ever starts passing, because that would mean the counterexample behind the
|
|
one-path decoder had stopped reproducing.
|
|
- the decode stage needs `tmp/rc_fr_singe_sasi_rcprofile.dlx` and will spend
|
|
~55 s encoding it if it is missing, nearly all of that k-means in `H.build`.
|
|
|
|
## Decisions locked
|
|
|
|
| decision | value | why |
|
|
|---|---|---|
|
|
| Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint |
|
|
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
|
|
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
|
|
| **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 |
|
|
| **Quality modes** | **two: `sasi` and `scsi`** (USER DECISION, session 2) | one codec, one decoder, one bitstream; only `lam` differs |
|
|
| Profile axis | **I/O bandwidth only** | the profiles say nothing about CPU; **both target the same stock 10MHz 68000**, and the Super has SCSI at 10MHz. FINDINGS 28.7 |
|
|
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
|
|
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
|
|
| SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution |
|
|
|
|
### The SASI/SCSI question is RESOLVED
|
|
Session 1 left "which machine do we target" open. The user's answer: **ship both**,
|
|
as two quality profiles. This is now implemented rather than hypothetical — the
|
|
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
|
|
|
|
| profile | target | lam | quality (00020 / 00146) | machine |
|
|
|---|---|---|---|---|
|
|
| `sasi` | 110 KB/s | 60 (floor) | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
|
|
| `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
|
|
|
|
**That "machine" column is about the BUS, not the CPU.** The profiles are an
|
|
I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has
|
|
built-in SCSI at 10 MHz (`x68k.cpp:1194`, `40_MHz_XTAL/4`, same as the base
|
|
machine), and only the XVI is faster. **Both profiles target the same stock
|
|
10 MHz 68000**, so both must fit the same 833,333-cycle frame budget — and as of
|
|
session 7 neither does. FINDINGS 28.7.
|
|
|
|
**As of session 6 `lam` is a floor, not a setting.** The target is a ceiling and
|
|
the encoder bisects lam per frame to stay under it; the profile's lam is the
|
|
best quality it is allowed to spend on a quiet frame. On the worst sustained
|
|
window that takes `sasi` from 137.4 to 109.5 KB/s and `scsi` from 381.6 to
|
|
280.0 KB/s, for -0.60 and -0.91 dB. FINDINGS 27.2.
|
|
|
|
Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on
|
|
SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**,
|
|
not an average over seek latency.
|
|
|
|
**Both profiles fit with room.** Ring-buffer simulation on the real per-frame
|
|
sizes gives **zero required prefill** for every scene at both profiles: the fill
|
|
delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds
|
|
that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance,
|
|
far more than an SD-backed seek needs. FINDINGS 21.
|
|
|
|
An earlier warning here said `scsi` did not fit because a frame peaked at 96.4%
|
|
of the pipe. That compared instantaneous demand to a sustained rate as if they
|
|
had to match frame-by-frame; with a buffer the test is cumulative, and it passes.
|
|
|
|
`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
|
|
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
|
|
RD curve rather than derived from the hardware. See FINDINGS 17.
|
|
|
|
Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured
|
|
and rejected — see FINDINGS 14, it was a false-good result from a rate model
|
|
that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff.
|
|
|
|
Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised
|
|
frame (measured 0.00 dB loss). The profiles are two points on one continuous
|
|
rate-distortion curve, not two codecs.
|
|
|
|
---
|
|
|
|
## What session 3 settled
|
|
|
|
1. **The display path works and is verified end to end.** First real frame on an
|
|
emulated X68000 screen: `docs/images/x68k_first_frame_compare.png`. Full
|
|
write-up in **FINDINGS 22**. Everything before this session was Python-side
|
|
or a headless `-video none` run, which cannot snapshot at all.
|
|
2. **The render is pixel-exact, not merely close.** With monitor contrast at 15,
|
|
all 256 palette entries render exactly as `GGGGGRRRRRBBBBBI` + `pal6bit`
|
|
predicts. That exactness is the regression test — see
|
|
`tools/bench/verify_frame.py`, which exits non-zero if it ever drifts.
|
|
3. **Three hardware facts that were previously assumed are now confirmed from
|
|
MAME 0.277 source**, not folklore: the palette word format, the 1024-byte
|
|
GVRAM line stride, and the 256-colour page aliasing in `HARDWARE.md`. All
|
|
three were already written down correctly; they are now cited.
|
|
4. ~~**A new quality ceiling was measured** — the 15-bit+I palette alone costs
|
|
38.88 dB.~~ **Superseded by session 4:** that figure assumed the shared LSB
|
|
`I` is always 1. Chosen per entry, the ceiling is **40.81 dB**. FINDINGS 23.3.
|
|
5. **Two shell traps that wedged session 2's background jobs** are documented in
|
|
the working-setup section below. They cost ~1.5 h of wall clock and a wedged
|
|
CPU core, and one of them was hit again this session.
|
|
|
|
## What session 5 settled
|
|
|
|
1. **68000 code drew a frame, and the blit was measured.** `tools/bench/blit.s`
|
|
+ `blit.lua`. The snapshot passes `verify_frame256.py` unchanged — pixel-exact
|
|
in the real 256x256 mode. **FINDINGS 23.5 is closed**: no longer "proven from
|
|
Lua only".
|
|
2. **The 38% full-frame blit estimate is dead. It is 53.6%.** And that is a
|
|
zero-wait-state floor — MAME models no GVRAM wait states, so real hardware is
|
|
worse. FINDINGS 24. Every variant was hand-derived from the MC68000 timing
|
|
tables before being measured and the two agree to 0.006-0.43%, so this is not
|
|
another MAME artefact.
|
|
3. **Reading the source frame is exactly half the blit cost** (V1 53.6% vs a
|
|
write-only floor V3 of 27.1%). That is what makes the architecture question
|
|
below live.
|
|
4. **That number is now measured, and the answer is "implement both paths".**
|
|
On the worst sustained window found on the disc, 30% of frames (`sasi`) to
|
|
53% (`scsi`) sit above the 70% crossover and want the flat blit; the rest
|
|
want direct-to-GVRAM. A player that picks per frame — the mode headers are
|
|
parsed before any pixel is written, so the count is free — pays a **median
|
|
37.0%** and is **capped at 53.6%**. FINDINGS 25.6.
|
|
5. **The sustained action sequence exists, was found by measurement, and breaks
|
|
both profiles.** `tools/analysis/07_motion_survey.py` scans a whole stream
|
|
for the hottest sliding window; on 00223 it is t=539.4s, the Singe endgame,
|
|
at 2.01x the stream mean. There, fixed-lam `sasi` overshoots 110 -> 129.6
|
|
KB/s (+18%) and `scsi` 280 -> 373.8 KB/s (+34%). **Rate control is no longer
|
|
insurance — it is required.** FINDINGS 25.3.
|
|
6. **The two largest streams on the disc are bonus material, not game footage.**
|
|
00216 is the feature with a burned-in commentary PiP; 00215 is the commentary
|
|
itself. **00223 (9.4 min) is the clean one.** A size-ranked survey would have
|
|
encoded live action. FINDINGS 25.1.
|
|
7. **Rate control is unsound as written, caught before wiring it up.** The
|
|
lam-ladder in `ratectl.py` picks frames from independent temporal chains,
|
|
so SKIP blocks reference reconstructions the decoder never saw: 111 of 120
|
|
frames drift, worst frame 43.4%, reported PSNR overstated 0.36 dB. Regression
|
|
test `tools/analysis/09_ratectl_drift.py`. FINDINGS 26.
|
|
8. **On hard content the scene palette, not the display, is the binding
|
|
ceiling** — 31.33 dB on the Singe window against 39.90 dB on 00020 and 40.81
|
|
dB for the X68000 display. `scsi` is already within 0.51 dB of it.
|
|
FINDINGS 25.4.
|
|
|
|
### Superseded within session 5
|
|
4a. **The decoder architecture hinged on one unmeasured number.** Writing
|
|
codewords straight into GVRAM costs 76.6% of the frame budget for a *full*
|
|
frame (V4 — the 1024-byte stride kills the `movem.l` burst), but scales with
|
|
the non-SKIP block fraction and needs **no RAM reference frame at all**,
|
|
because the previous frame is already in GVRAM. Compose-then-blit is a flat
|
|
53.6%. **They cross at 70% of blocks changed.** FINDINGS 24.5.
|
|
|
|
---
|
|
|
|
## What session 4 settled
|
|
|
|
1. **A real 256x256 CRTC mode exists and is verified.** `crtc_mode.lua`, derived
|
|
from `x68k_crtc.cpp`'s divisor ladder rather than recalled — the derivation is
|
|
self-checking (368 = 1104/3 exactly, so the horizontal registers divide by
|
|
three with no remainder). Snapshot is native 256x512, active area pixel-exact,
|
|
letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone.
|
|
2. **The palette ceiling was wrong by 2 dB, in our favour.** The shared LSB `I`
|
|
must be chosen **per palette entry**, not hardcoded to 1. Doing so lifts the
|
|
display ceiling from 38.85 to **40.81 dB** and is the only way to get true
|
|
black at all (`pal6bit(1) = 4`). 102 of 256 entries want `I = 0`. This
|
|
supersedes FINDINGS 22.4 and gives `scsi` ~2 dB more headroom than believed.
|
|
**The encoder does not do this yet** — see the encoder-gaps list.
|
|
3. **Letterboxing costs one palette entry.** 255 colours + a reserved black at
|
|
index 0, with `I = 0` on it. `prep_frame.py --reserve-black`. FINDINGS 23.4.
|
|
4. **MAME's graphics double-scan is phase-shifted one raster line** — pairs are
|
|
(1,2),(3,4),..., not (0,1), because `get_gfx_pixel` halves the *absolute*
|
|
scanline and `vbegin = 41` is odd. Cost a false failure. FINDINGS 23.2.
|
|
|
|
---
|
|
|
|
## What session 2 settled
|
|
|
|
1. **The critical-path question is answered.** "Does VQ soften Bluth's linework
|
|
unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.**
|
|
Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two
|
|
images in `docs/images/`. Both profiles use **k=256**; see item 2b.
|
|
2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured).
|
|
Halving the framerate does not halve the bitrate. FINDINGS 8.
|
|
2b. **A fourth false-good result was produced and caught this session** — k=1024
|
|
codebooks looked like a +2.4 dB free win because the rate model charged 1 byte
|
|
for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
|
|
3. **The 256-colour palettised frame is the real quality ceiling** and it looks
|
|
excellent. Judge the codec against that, not against 1080p.
|
|
4. Encoder exists and produces a real bitstream: `tools/encoder/`.
|
|
|
|
---
|
|
|
|
## Encoder — working
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
|
|
python3 tools/encoder/encode.py /tmp/fr_00020 out.dlx --profile sasi --preview p.png
|
|
```
|
|
|
|
| file | role |
|
|
|---|---|
|
|
| `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise |
|
|
| `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR |
|
|
| `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision |
|
|
| `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control |
|
|
| `encode.py` | CLI + `DLX1` container writer |
|
|
|
|
`DLX1` container layout is documented in the `encode.py` docstring. All
|
|
multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`.
|
|
|
|
### Known encoder gaps
|
|
- ~~Rate control is written but not yet wired into `encode.py`.~~ **DONE,
|
|
session 6.** It is on by default; `--fixed-lam` restores the old behaviour.
|
|
Gated by `tools/analysis/09_ratectl_drift.py`, which is now in `check.sh`.
|
|
- **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in
|
|
the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
|
|
- **Frame records are not aligned.** They must be padded to a 4-byte boundary:
|
|
unaligned is an ADDRESS ERROR on a 68000, not a slow read (FINDINGS 28.3).
|
|
`prep_dlx.py` repairs it at load time, which a player streaming from disc
|
|
cannot do. The pad is real bytes on disc, so it belongs inside the rate
|
|
controller's accounting. 1.66 B/frame, 20 B/s.
|
|
- **The mode decision is blind to CPU cost.** It charges V4 four payload bytes
|
|
and ignores that it costs 1.49x a V1 block to draw. This is the top item at
|
|
the head of this file. FINDINGS 28.2.
|
|
- **Palette packing is not implemented in the encoder.** It still emits 24-bit
|
|
palettes; the X68000 word packing happens Lua-side. Whatever writes real
|
|
palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3,
|
|
worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4).
|
|
- Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
|
|
- ~~`_paint` is a Python per-block loop.~~ **DONE, session 6** — vectorised,
|
|
17.1x. It was never the bottleneck, though: `VQ.assign` is 78% of a frame and
|
|
`H.build`'s k-means is 51 s of a 55 s run. **That k-means is now the thing to
|
|
attack before the full-disc survey**, not anything in the per-frame path.
|
|
FINDINGS 27.6.
|
|
|
|
---
|
|
|
|
## Working setup (unchanged from session 1, re-verified)
|
|
|
|
**MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**.
|
|
```
|
|
mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3
|
|
```
|
|
**Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`
|
|
|
|
**Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM`
|
|
(still mounted as of end of session 2).
|
|
|
|
**MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the
|
|
notifier subscription in a global; the stack register is `SP` not `A7`;
|
|
`autoboot_script` fires at PC=0 before boot) are documented in FINDINGS.
|
|
|
|
**Two shell traps, both hit again this session:**
|
|
- piping MAME (or any long job) through `grep` block-buffers — write to a file.
|
|
- `pkill -f <pattern>` matches your own shell and kills it (exit 144).
|
|
Use `pkill -x` or kill by PID.
|
|
- **`pgrep -f <name> | xargs kill` kills your own shell too — exit 144.** Same
|
|
root cause as the `pkill -f` trap above: the shell's own command line contains
|
|
the pattern. **Hit again in session 5**, which makes it four times across three
|
|
sessions. Kill by PID captured at launch (`$!`), or use `pkill -x`.
|
|
- **`until ! pgrep -f foo.py; do sleep; done` watcher loops never exit.** The
|
|
watching shell's own command line contains the string `foo.py`, so `pgrep -f`
|
|
matches the watcher itself and the loop spins forever. Session 2 left 11 of
|
|
these wedged for over an hour. Wait on the PID (`while kill -0 $PID`) or on a
|
|
sentinel file the job touches when it finishes -- never on a `-f` name match.
|
|
- **`timeout N mame ...` does not kill MAME.** MAME catches SIGTERM and, with an
|
|
autoboot script blocked waiting on a flag that never arrives, never reaches
|
|
its shutdown path. `timeout` without `-k` then waits forever while MAME burns
|
|
a full core at `-nothrottle`. Always `timeout -k 5 N`.
|
|
|
|
---
|
|
|
|
## Disk throughput benchmark — still blocked, no longer gating
|
|
|
|
`IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested
|
|
hypotheses are in session 1's notes (git history of this file, commit 65112b9);
|
|
the ordered plan for retrying is in **`docs/BENCHMARK.md`**.
|
|
|
|
**Status changed twice this session — read this rather than the git history.**
|
|
It was briefly promoted to critical-path while the working bandwidth figure was
|
|
misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the
|
|
ring-buffer simulation showing **zero required prefill** for both profiles
|
|
(FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not**
|
|
available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a
|
|
"measure it and maybe ship transparent" decision waiting.
|
|
|
|
What the benchmark is still worth doing for:
|
|
- **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is
|
|
not recorded. Every profile hangs off it.
|
|
- **Confirming DMA is actually used.** If transfers fall back to PIO the CPU
|
|
cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the
|
|
binding constraint. This is the worst plausible outcome and the cheapest to
|
|
check — do it first.
|
|
|
|
**Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are
|
|
functional models, not timing-accurate; a KB/s figure from MAME measures the
|
|
emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach
|
|
(MAME validates the path, derivation bounds it, real hardware settles it).
|
|
|
|
## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5), by a 68000 DECODER (session 7).
|
|
|
|
The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`.
|
|
|
|
**Session 7 went from copying a frame to parsing one.** `src/player/decode.s`
|
|
reads DLX1, dispatches all four block modes and writes straight into GVRAM;
|
|
120 frames decoded in sequence are pixel-exact against `tools/encoder/dlx.py`
|
|
(`tools/bench/verify_decode.py`, in `check.sh`). The blit numbers below are
|
|
still correct for what they measured — a *copy* — but they are no longer the
|
|
display-path budget: the decoder costs 300/448/400 cycles per V1/V4/RAW block
|
|
and misses the 12fps budget on 31% of frames. FINDINGS 28.
|
|
|
|
**Session 5 closed the gap this paragraph used to describe.** GVRAM is now
|
|
filled by 68000 instructions and the result is still pixel-exact, and the blit
|
|
cost is measured rather than estimated: **53.6% of a 12fps frame**, not 38%
|
|
(FINDINGS 24). The remaining caveat is different and narrower: MAME models
|
|
**no GVRAM wait states**, so 53.6% is a floor and real hardware is worse.
|
|
Full write-up in **FINDINGS 22**. Harness: `tools/bench/show_frame.lua` +
|
|
`tools/bench/prep_frame.py`.
|
|
|
|
Three facts the player MUST honour, none of which were guessable:
|
|
|
|
| what | where | value |
|
|
|---|---|---|
|
|
| **Un-hide the graphics layer** | CRTC R20 `$E80028` | clear bit 11 ("G-VRAM set to buffer"); IPL leaves `0x0B16` |
|
|
| Colour setup (256c) | CRTC R20 bits 9-8 | `0x0100` |
|
|
| **Monitor contrast** | `$E8E001` bits 3-0 | IPL leaves **14**; write **15** or everything renders 7% dark |
|
|
|
|
The `R20 = 0x0116` value quoted here in session 3 is the **768-wide IPL timing**
|
|
with the gate cleared. The shipping value is **`R20 = 0x0110`** — see the mode
|
|
table in `tools/bench/crtc_mode.lua`, which is now the single source of truth
|
|
for all of R00-R08 and R20.
|
|
|
|
Bit 11 is the one that cost the most time: GVRAM writes land and read back
|
|
correctly while the layer is invisible, so the video controller looks guilty and
|
|
is not. Contrast `0` blanks the screen — free fade-to-black for transitions.
|
|
|
|
Palette format is now **confirmed from MAME source**, not assumed:
|
|
`GGGGGRRRRRBBBBBI` (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as
|
|
`pal6bit((field<<1)|I)`. With contrast at 15 the render is **pixel-exact**.
|
|
|
|
Ceiling: the 15-bit+I palette costs **40.81 dB** against the 24-bit palettised
|
|
source, once `I` is chosen per entry (FINDINGS 23.3 — session 3's 38.88 dB
|
|
assumed `I = 1`). Still the same order as the `scsi` profile's own codec error
|
|
(39.4 dB), so `scsi` remains near display-transparent, with ~2 dB more headroom
|
|
than session 3 thought.
|
|
|
|
Snapshot recipe that works (`-video none` CANNOT snapshot):
|
|
```
|
|
SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
|
|
-sound none -nothrottle -plugins -autoboot_script <script>.lua \
|
|
-snapshot_directory ./snap -snapview native -seconds_to_run 6
|
|
```
|
|
`-snapview native` drops MAME's LED artwork and gives a clean 768x512 screen.
|
|
|
|
## Next steps, in priority order
|
|
|
|
1. ~~**Measure the non-SKIP block fraction.**~~ **DONE, session 5**, and its
|
|
answer **WITHDRAWN in session 7** — FINDINGS 28.1/28.2. It concluded
|
|
"implement both display paths and pick per frame, median 37.0%, capped at
|
|
53.6%". Mixing the paths is incoherent (the compose path needs a RAM
|
|
reference the direct path never writes) and the two costs it compared were
|
|
both copies with no decode in either. **The shipping decoder has one path.**
|
|
The non-SKIP fraction is still reported by `encode.py` and is still the right
|
|
thing to look at — it is just no longer a switch. Original framing kept
|
|
below, because its instruction to report the distribution rather than the
|
|
mean is the part that held up:
|
|
FINDINGS 24.5: compose-in-RAM-then-blit costs a flat 53.6% of the frame
|
|
budget; decode-direct-to-GVRAM costs 76.6% x (fraction of blocks that are not
|
|
SKIP) and needs no RAM reference frame. **They cross at 70%.** Which side of
|
|
70% the content sits on decides which decoder inner loop to write, so this
|
|
must come before writing one.
|
|
**It needs no new machinery** — the mode decision in `vq_hybrid.py` already
|
|
computes it per frame and simply never reports it. Add the histogram
|
|
(SKIP / V1 / V4 / RAW counts per frame) to `encode.py` output and run it over
|
|
the clips already extracted. Report the *distribution*, not the mean: a
|
|
scene-cut frame is ~100% non-SKIP and a held frame near 0%, and the mean of
|
|
those two is a number describing no actual frame.
|
|
|
|
1b. ~~**Wire rate control into `encode.py`.**~~ **DONE, session 6.** FINDINGS 27.
|
|
Both overshoots closed for under 1 dB, drift test at zero, `check.sh` gates
|
|
it. The remaining rate-control question is not a defect: whether `--rc-floor
|
|
open` is worth taking on quiet content. It measured as worth **0.00 dB** on
|
|
the Singe window (no frame there is quiet enough to saturate the bucket), so
|
|
it needs a genuinely quiet scene to decide, and it is a quality-per-byte
|
|
judgement rather than a correctness one.
|
|
|
|
2. ~~**68000 decoder skeleton.**~~ **DONE, session 7.** `src/player/decode.s`,
|
|
pixel-exact over 120 frames, gated in `check.sh`. It answered the question it
|
|
was written to answer, and the answer is no: **it does not fit** — mean 81.7%
|
|
of a 12fps frame, 31% of frames over 100%. FINDINGS 28. The follow-on is
|
|
priority 0 at the top of this file.
|
|
|
|
2a. **Re-budget everything against the MEASURED per-mode costs**, not 53.6% and
|
|
not 38%. Session 7 replaced the model twice over (FINDINGS 28.2): the display
|
|
path is not one number times a block fraction, and the median frame is 74.4%
|
|
rather than 36.6%. The original note is kept below because its warning about
|
|
downstream figures derived from a dead estimate is exactly what happened
|
|
again.
|
|
~~Re-budget everything against 53.6%, not 38%.~~ Several downstream figures
|
|
were derived from the old estimate. The blit alone now eats over half the
|
|
frame at 12fps in the compose-then-blit design, before any decode, and MAME
|
|
models no GVRAM wait states so that is a floor. This may reopen questions
|
|
that were closed against the 38% number — check FINDINGS 17.2's entropy-coding
|
|
rejection, which was argued as "54% LZ4 with no room beside a 38% blit". The
|
|
conclusion gets *stronger*, not weaker, but the arithmetic should be restated.
|
|
|
|
2b. **Pad frame records to 4 bytes in `encode.py`.** Not optional: unaligned
|
|
records are an address error on a 68000 (FINDINGS 28.3), and `prep_dlx.py`
|
|
currently repairs it at load time, which the shipping player streaming from
|
|
disc cannot do. The padding is real bytes on disc, so it has to be inside
|
|
the rate controller's accounting, not added after it. 20 B/s at 12fps.
|
|
|
|
3. **Full-disc survey.** Now scoped by session 5 rather than open-ended: the
|
|
worst *sustained* window is measured (FINDINGS 25), so what remains is the
|
|
distribution over content, not the worst case.
|
|
- Classify **content / menu / bonus** — not just menu vs content. FINDINGS
|
|
25.1: the two largest streams are bonus material and look like content by
|
|
size, duration and bitrate alike.
|
|
- Run `tools/analysis/07_motion_survey.py` per stream first; it is cheap
|
|
(96x72 greyscale) and gives a hot-window shortlist so the expensive encode
|
|
only runs where it matters.
|
|
- ~~Vectorise `_paint` before this run.~~ Done. The cost to attack now is
|
|
`H.build`'s k-means: 51 s of a 55 s run, and it runs once per scene.
|
|
- ~~Do it after rate control (1b), or it measures an encoder nobody ships.~~
|
|
Rate control is in, so the survey now measures the shipping encoder.
|
|
|
|
5. **Confirm DMA vs PIO in MAME** (see the benchmark section above) — cheap, and
|
|
the only thing that could still move CPU into the binding position.
|
|
6. **Resolve the framing question** (FINDINGS 12: crop vs squash vs wide).
|
|
Needs an eyeball against arcade reference, not a measurement.
|
|
7. **Import the scene graph.** SNES project `data/events/` (MIT, cleared),
|
|
cross-checked against DirkSimple (zlib) which transcribed the same data
|
|
independently — diff them to catch transcription errors before committing
|
|
any of it to 68000 tables.
|
|
8. **ADPCM audio.** MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in
|
|
`ratectl.py`, not yet extracted or encoded.
|
|
|
|
### Explicitly abandoned — do not re-propose
|
|
- ~~Entropy-code the payload.~~ Deflate decode is ~216% of the frame budget on a
|
|
68000; LZ4 is ~54% with no room beside a 38% blit (FINDINGS 17.2). All bitrates
|
|
are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8
|
|
to a compression upper bound, not a shippable design.
|
|
- ~~k=1024 codebooks.~~ False-good result from a rate model that charged 1 byte
|
|
for a 10-bit index (FINDINGS 14). k=256 wins at every matched bitrate.
|
|
- ~~Flat 4x4 VQ.~~ Rejected by eye (FINDINGS 9).
|
|
|
|
## Not yet started
|
|
- **A player, as opposed to a decoder.** `src/player/decode.s` parses DLX1,
|
|
dispatches all four block modes and draws pixel-exact frames, but it decodes
|
|
from RAM that Lua pre-loaded. There is no disc streaming, no ring buffer, no
|
|
audio, no timing against the VBL, and no scene branching.
|
|
- **Codebook expansion on the 68000.** `prep_dlx.py` does it host-side because
|
|
it is a load-time cost and including it would flatter or damn the inner loop.
|
|
The player must do it: 8 KB + 2 KB per scene.
|
|
- ADPCM audio extraction/encoding
|
|
- Disk image packaging
|
|
- Game logic (scene branching, input windows, death clips)
|
|
|
|
## Reproducing the 256x256 mode result (session 4)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
|
|
mkdir -p tmp/snap256 && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 90 mame x68000 \
|
|
-bios ipl10 -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/show_frame256.lua \
|
|
-snapshot_directory ./snap256 -snapview native -seconds_to_run 6
|
|
cd .. && python3 tools/bench/verify_frame256.py
|
|
```
|
|
Exits non-zero on any drift. Expected: `256x512 native, double-scan exact,
|
|
active 256x192 pixel-exact, letterbox true black`, ceiling 40.81 dB.
|
|
|
|
## Reproducing the display result
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame.bin 0
|
|
mkdir -p tmp/snap_verify && cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 \
|
|
-video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/show_frame.lua \
|
|
-snapshot_directory ./snap_verify -snapview native -seconds_to_run 6
|
|
cd .. && python3 tools/bench/verify_frame.py
|
|
```
|
|
Verified cold from the Blu-ray at end of session 3: exact match, 38.88 dB.
|
|
(That 38.88 is correct *for this test*: `show_frame.lua` still packs `I = 1`.
|
|
The 40.81 dB ceiling comes from `show_frame256.lua`, which picks `I` per entry.)
|
|
|
|
`tmp/` is gitignored scratch. The frames are NOT in the repo — regenerate them
|
|
with `extract.py`; the earlier ones lived in `/tmp` and do not survive a reboot.
|
|
|
|
## Reference material on this box (not in the repo)
|
|
|
|
- **MAME 0.277 source: `~/src/mame-mame0277/`** (tarball `~/src/mame0277.tar.gz`).
|
|
Downloaded this session to settle the graphics-layer question. The files that
|
|
matter are `src/mame/sharp/x68k_v.cpp`, `x68k_crtc.cpp`, `x68k_crtc.h`,
|
|
`x68k.cpp`. **Read these before theorising about X68000 video behaviour** —
|
|
six register-poking attempts failed against a gate that one grep found.
|
|
- Blu-ray mounted at `/media/reala-misaki/BDROM` via
|
|
`udisksctl loop-setup -r -f DRAGONS_LAIR.iso`.
|
|
|
|
---
|
|
|
|
## Parked ideas (not scheduled, not abandoned)
|
|
|
|
- **Cliff Hanger, retitled as Lupin III** (user, session 4). Stern's 1983
|
|
laserdisc game was cut from *Castle of Cagliostro* and *Mystery of Mamo* with
|
|
the Lupin branding stripped; a port would restore it. Technically **cheaper
|
|
than this project**: same content class (cel animation, flat colour, hard
|
|
cuts), ~13 min of footage vs Dragon's Lair's ~22, and flatter linework than
|
|
Bluth's, so fewer blocks should escape to V4/RAW. The codec, the display path,
|
|
and `crtc_mode.lua` would all drop straight in.
|
|
**The real cost is media prep, not code:** there is no clean master cut to
|
|
Stern's scene boundaries the way `DRAGONS_LAIR.iso` is, so the footage would
|
|
have to be sourced and cut to match. Not to be started until the CPU path is
|
|
proven — it changes nothing about whether this design works.
|
|
|
|
## Reproducing the blit measurement (session 5)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
|
|
tools/vasm/vasmm68k_mot -Fbin -o tmp/blit.bin tools/bench/blit.s
|
|
mkdir -p tmp/snap_blit && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
|
|
-bios ipl10 -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/blit.lua \
|
|
-snapshot_directory ./snap_blit -snapview native -seconds_to_run 120
|
|
```
|
|
~25 s wall. Prints cycles/frame and % of a 12fps budget for V1-V4, and snapshots
|
|
V1's output. To check that snapshot is still pixel-exact:
|
|
`sed 's|snap256|snap_blit|' tools/bench/verify_frame256.py | python3 -`
|
|
|
|
Not added to `check.sh`: `check.sh` asserts pixel-exactness, and asserting wall
|
|
timings there would make the green-light check sensitive to host load.
|
|
|
|
## Reproducing the decoder result (session 7)
|
|
|
|
```
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_fr_singe_sasi_rcprofile.dlx --profile sasi
|
|
python3 tools/bench/prep_dlx.py tmp/rc_fr_singe_sasi_rcprofile.dlx
|
|
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s
|
|
mkdir -p tmp/snap_decode && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
|
|
-bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/decode.lua \
|
|
-snapshot_directory ./snap_decode -snapview native -seconds_to_run 150
|
|
cd .. && python3 tools/bench/verify_decode.py tmp/rc_fr_singe_sasi_rcprofile.dlx
|
|
```
|
|
~90 s wall. Prints cycles/frame and % of a 12fps budget for four real frames
|
|
spanning the non-SKIP distribution, four synthetic single-mode frames, and one
|
|
full 120-frame pass; then verifies the last frame is pixel-exact. Expected:
|
|
median 73.8%, p90 116.4%, max 135.8%, mean 81.7%; V1 299.9 / V4 448.2 / RAW
|
|
400.4 cycles per block.
|
|
|
|
`-ramsize 2M` matters — MAME defaults to 4M and the locked target is a stock 2MB
|
|
machine. `DLX_VERIFY_ONLY=1` drops the timing anchors, which is how `check.sh`
|
|
runs it.
|
|
|
|
Score a container against the measured costs without touching MAME:
|
|
```
|
|
python3 tools/analysis/11_cpu_budget.py tmp/rc_fr_singe_scsi_rcprofile.dlx
|
|
```
|
|
And re-demonstrate why there is only one display path (exits non-zero **by
|
|
design** — it is the counterexample):
|
|
```
|
|
python3 tools/analysis/10_pathmix_drift.py # 70/120 frames corrupt
|
|
python3 tools/analysis/10_pathmix_drift.py --fix direct # clean, and cheapest
|
|
```
|
|
|
|
## Reproducing the rate-control result (session 6)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
|
|
for prof in sasi scsi; do
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof --fixed-lam
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof
|
|
done
|
|
python3 tools/analysis/09_ratectl_drift.py # must exit 0, zero drifting frames
|
|
```
|
|
Expected, totals including the 7.8 KB/s audio allowance: `sasi` 137.4 -> 109.5
|
|
KB/s and 27.82 -> 27.22 dB; `scsi` 381.6 -> 280.0 KB/s and 30.81 -> 29.90 dB;
|
|
zero frames at the lam=800 cliff in either. ~55 s per encode, nearly all of it
|
|
k-means in `H.build`.
|
|
|
|
The block-mode map now renders the rate-controlled encoder by default:
|
|
```
|
|
python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes_rc.webm \
|
|
--profile sasi --scale 2 # add --fixed-lam to compare
|
|
```
|
|
|
|
**Do not judge rate control on `tmp/fr_00020`.** It is 14 frames; the leaky
|
|
bucket's startup transient is bucket/nframes, so it lands 18% under target there
|
|
for reasons that have nothing to do with the content. FINDINGS 27.5.
|
|
|
|
## Reproducing the sustained-action result (session 5)
|
|
|
|
```
|
|
python3 tools/analysis/07_motion_survey.py 00223 10 # -> hottest window t=539.4s
|
|
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_sasi.dlx --profile sasi
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_scsi.dlx --profile scsi
|
|
python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes.webm \
|
|
--profile sasi --scale 2
|
|
```
|
|
`extract.py` now takes optional `[start_s] [dur_s]` — needed because 00223 is
|
|
9.4 min and the windows that stress the codec are seconds long.
|
|
|
|
`08_mode_map.py` renders palettised source | decoded | block-mode map at 12fps.
|
|
Output format follows the extension; **prefer `.webm`** — GIF re-quantises to
|
|
256 colours, which is a poor fit for output whose subject is colour fidelity,
|
|
and runs larger. It uses `yuv444p` because the mode map is flat saturated colour
|
|
on a 4-pixel grid and chroma subsampling smears exactly those edges.
|