prosolis
06b98d4b47
The decoder has been CPU-bound since FINDINGS 28 while the mode decision
minimised D + lam*R -- distortion against BYTES. decide() now minimises
D + lam*bytes + mu*cycles, and ratectl bisects mu per frame against the
833,333-cycle budget with the lam bisection nested inside it. On the worst
sustained window:
sasi 27.22 -> 26.95 dB, 109.5 -> 109.4 KB/s, 37/120 misses -> 1
scsi 29.90 -> 29.27 dB, 280.0 -> 278.6 KB/s, 51/120 misses -> 1
Bitrate does not move: the byte controller still binds, and mu changes WHICH
modes are bought. V4 is what it stops buying -- 25.2 -> 20.3% of blocks at sasi
and 15.0 -> 5.3% at scsi, where RAW takes it. That is 28.8's inversion in
practice: RAW is dearer in bytes and cheaper in cycles, so only the byte-rich
profile can buy its way out of V4.
Three things worth knowing beyond the headline:
- The one frame that still misses, at both profiles, is FRAME 0 -- no previous
reconstruction, so 100% changed by definition, which is also what a scene
cut is. It comes out at the all-V1 floor of 110.6% and is emitted late on
purpose. Freezing a cut to make a deadline is the worse failure.
- 28.7's "11 frames are impossible" was too pessimistic. That floor held the
SKIP set fixed and asked how cheaply the drawn blocks could be drawn; the
real decision can also MOVE a block to SKIP, which above ~90% non-SKIP is
the only lever left.
- SKIP's price depends on its neighbours (13.25 cycles clustered, 45 mixed),
which a per-block lagrangian cannot see. The way out is that the two uses
need not share a cost function: a ranking constant inside decide(), the
exact clustered rule for the frame-level bisection. vq_hybrid.cycles() is
now the one definition of that rule and 11_cpu_budget.py imports it.
Gated: 09_ratectl_drift.py runs both controllers, both 0/120 drifting frames.
The cost-aware container decodes pixel-exact on the 68000 (120 frames). ON by
default in encode.py; --no-cpu-fit restores session 7. check.sh ALL GREEN.
Still a model, not a measurement, for THIS container: FINDINGS 31's cycle
figures come from vq_hybrid.cycles (within 1 point of the 68000 on four frames
of the session-7 container). Timing this one on the machine is step 1 of the
next session -- it was started and killed for time, and it is slow.
FINDINGS 31. tools/analysis/13_cpu_ratectl.py.
Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
832 lines
47 KiB
Markdown
832 lines
47 KiB
Markdown
# Status & next-session handoff — end of session 8 (2026-08-23)
|
|
|
|
## Where this stands
|
|
|
|
Session 7 left the decoder pixel-exact and **31% of frames over the CPU budget**
|
|
at `sasi`, 42% at `scsi` (FINDINGS 28), with two levers proposed and neither
|
|
measured. Session 8 did both.
|
|
|
|
**Lever A, spans: measured.** A row-linear span of word-expanded literals costs
|
|
**43.7 cycles per span + 9.152 per pixel** — but only in an encoder-assisted
|
|
format, `{u32 absolute GVRAM address, u16 jump displacement}` into an unrolled
|
|
copy chain. The obvious decoder, handed `(x, npix)`, is 97.9 + 10.46.
|
|
FINDINGS 30, `tools/bench/span.sh` (~25 s).
|
|
|
|
**Lever B, the cost-aware mode decision: implemented, measured, and ON by
|
|
default.** `decide()` minimises `D + lam*bytes + mu*cycles`; `mu` is bisected
|
|
per frame against 833,333 cycles with the `lam` bisection nested inside it.
|
|
FINDINGS 31, `tools/analysis/13_cpu_ratectl.py`.
|
|
|
|
| `sasi`, worst sustained window | PSNR | KB/s | CPU median | CPU max | missing |
|
|
|---|---:|---:|---:|---:|---:|
|
|
| bytes only (session 7) | 27.22 dB | 109.5 | 74.4% | 136.2% | **37/120** |
|
|
| + cycle ceiling (now the default) | 26.95 dB | 109.4 | 81.5% | 110.6% | **1/120** |
|
|
| + ceiling + spans (MODEL, nothing implements it) | — | 449.3 | 56.8% | 91.5% | **0/120** |
|
|
|
|
`scsi`: 51/120 -> 1/120 for 0.62 dB. The one remaining miss at either profile is
|
|
**frame 0**, which has no previous reconstruction and so is 100% changed by
|
|
definition — the same case as a scene cut. It is emitted late on purpose.
|
|
|
|
The cost-aware container is verified pixel-exact on the 68000 (120 frames,
|
|
`tools/bench/verify_decode.py`).
|
|
|
|
## NEXT SESSION, in order
|
|
|
|
0. **Green light first.** `./tools/bench/check.sh` (~3 min, Blu-ray mounted).
|
|
The drift stage now runs BOTH controllers; both must report 0/120.
|
|
|
|
1. **Time the cost-aware container on the 68000.** Everything in FINDINGS 31 is
|
|
the validated cost MODEL (`vq_hybrid.cycles`, within 1 point of the machine
|
|
on four frames of the session-7 container), not a measurement of this one.
|
|
The full timing pass was started and killed for time:
|
|
```
|
|
python3 tools/bench/prep_dlx.py tmp/rc_fr_singe_sasi_cpufit.dlx
|
|
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s
|
|
( cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -ramsize 2M \
|
|
-video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/decode.lua -snapshot_directory ./snap_decode \
|
|
-snapview native -seconds_to_run 300 > decode_cpufit.log 2>&1 )
|
|
```
|
|
**Budget real time for it: the run was still going at 12 minutes of CPU.**
|
|
MAME's stdout is block-buffered to a file, so there is no progress to watch
|
|
— wait on the PID, never on a `pgrep -f` match (see the shell traps below).
|
|
Confirm the four anchors against `11_cpu_budget.py` on the same container,
|
|
and update FINDINGS 31 with measured-vs-model errors.
|
|
|
|
2. **Put spans in the bitstream.** This is the big one and it is now fully
|
|
specified by measurement: format in FINDINGS 30.2, costs in 30.5, and the
|
|
scene-cut arithmetic in 30.6. It touches `encode.py` (a fifth mode and a
|
|
run-aware decision), `dlx.py` (the reference decoder), and `decode.s`. The
|
|
24-pixel quantisation and the free row overrun are part of the format, not
|
|
optimisations to add later. Order it AFTER item 1 so the model that prices it
|
|
has been checked against the machine once more.
|
|
|
|
3. **The three open items of FINDINGS 29.5/30.7**, now load-bearing because a
|
|
span design runs at ~449 KB/s of a 488 KB/s pipe: re-run the ring-buffer
|
|
simulation at that rate (FINDINGS 21 was established at 110 and 280), confirm
|
|
the provenance of the user's 4 Mbps figure, and **confirm DMA rather than
|
|
PIO** — a PIO fallback puts a 449 KB/s transfer back on the CPU the whole
|
|
lever exists to relieve. The DMA check is the cheapest of the three and the
|
|
most consequential.
|
|
|
|
4. **Encoder gap, still open from session 7:** `encode.py` should pad frame
|
|
records to 4 bytes. Frame boundaries land on odd addresses and a 68000 takes
|
|
an address error, not a slow read (FINDINGS 28.3). `prep_dlx.py` pads at load
|
|
time, which is why the decoder works; the container itself does not.
|
|
Measured cost of fixing it: 1.5 B/frame = 18 B/s.
|
|
|
|
5. **A quality-vs-framerate question that is the user's, not the encoder's.**
|
|
Every miss is now one frame per cut. The options remain: one late frame at
|
|
each cut (the outgoing content is unrelated, so it may be invisible), a cut
|
|
spread over two frame times, or 10fps. Spans (item 2) make the question go
|
|
away if they land as modelled.
|
|
|
|
**Do not start by hand-optimising `decode.s`.** The hand-derived timings agree
|
|
with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), so the inner
|
|
loop is close to what the instruction set allows.
|
|
|
|
---
|
|
|
|
|
|
## What session 8 settled
|
|
|
|
0. **The mode decision can see cycles, it is on by default, and it costs
|
|
0.26 dB.** `decide(ctx, lam, mu)` minimises `D + lam*bytes + mu*cycles`;
|
|
`ratectl` bisects mu per frame against a HARD 833,333-cycle ceiling (bytes
|
|
bank in the ring buffer, cycles cannot — there is no double buffer to decode
|
|
ahead into). `sasi` 37/120 misses -> 1, `scsi` 51 -> 1. Bitrate does not
|
|
move: mu changes which modes are bought, not how many bytes. FINDINGS 31,
|
|
`tools/analysis/13_cpu_ratectl.py`.
|
|
0b. **28.7's "11 frames are impossible" was too pessimistic — it is 1.** That
|
|
floor held the SKIP set fixed; the real decision can also move a block to
|
|
SKIP, which above ~90% non-SKIP is the only lever left. FINDINGS 31.3.
|
|
0c. **V4 collapses when cycles are priced**, as 28.8 predicted: 25.2 -> 20.3%
|
|
of blocks at `sasi` and **15.0 -> 5.3%** at `scsi`, where RAW takes it. RAW
|
|
is dearer in bytes and cheaper in cycles, so the byte lagrangian's preference
|
|
inverts and only the byte-rich profile can take the escape.
|
|
0d. **SKIP's price depends on its neighbours, and the way out is two cost
|
|
functions**: a ranking constant inside the per-block lagrangian, the exact
|
|
clustered rule (`vq_hybrid.cycles`, validated to 1 point against the 68000)
|
|
for the frame-level bisection. That function is now defined once and imported
|
|
by `11_cpu_budget.py`. FINDINGS 31.4.
|
|
0e. **Both controllers are gated against decoder drift.**
|
|
`09_ratectl_drift.py` runs bytes-only AND bytes+cycles; both 0/120.
|
|
1. **The span is measured: 43.7 cycles/span + 9.152/pixel, fitted to 0.3% over
|
|
eleven span lengths.** `tools/bench/blit.s` v5/v6, `prep_spans.py`,
|
|
`span.lua`, driven by `tools/bench/span.sh` (~25 s, not in `check.sh`
|
|
because it is a wall timing). FINDINGS 30.
|
|
2. **Only in an encoder-assisted format.** `{u32 absolute GVRAM address, u16
|
|
jump displacement}` into an unrolled chain, versus `(x, npix)` and a decoder
|
|
that works it out: 43.7 + 9.152 against 97.9 + 10.46. All the arithmetic a
|
|
span decoder would do per frame is known at encode time. FINDINGS 30.2.
|
|
3. **The per-pixel cost is a function of register pressure**, which FINDINGS 24
|
|
could not have shown: 9.08 was a fixed blit with 12 registers free, v5 can
|
|
spare 8 and pays 10.46, v6 gets 12 back by making the encoder hold the state.
|
|
4. **Short spans die in the remainder path, and the fix is padding.** A 12-pixel
|
|
span costs more than a 16-pixel one in v5. v6 has no remainder path: lengths
|
|
are multiples of 24 pixels, padding is free of everything but bytes, and an
|
|
overrun past the visible 256 lands in the invisible half of the 1024-byte
|
|
line stride. FINDINGS 30.3.
|
|
5. **Odd-`x` alignment is free** (259.0 vs 261.8 cycles/span) — expected on a
|
|
16-bit bus, now measured rather than assumed.
|
|
6. **The trade is smaller than 29 derived but the conclusion holds**, including
|
|
29.4's reopening of the scene cut. All 23 timing configs also drew a
|
|
pixel-exact frame, so nothing here was timed against a decoder that skipped
|
|
work. FINDINGS 30.5/30.6.
|
|
|
|
---
|
|
|
|
## What session 7 settled
|
|
|
|
1. **68000 code parses a bitstream and draws frames, pixel-exact.**
|
|
`src/player/decode.s` + `tools/bench/decode.lua`. 120 frames of the Singe
|
|
window decoded in sequence, all four block modes, verified against the new
|
|
reference decoder `tools/encoder/dlx.py`. Because SKIP blocks are claims
|
|
about the previous frame, the last frame is only right if all 120 were.
|
|
In `check.sh` now. **FINDINGS 28.**
|
|
2. **It does not fit.** Mean 81.7% of a 12fps frame, p90 116.4%, worst 135.8%;
|
|
31% of frames miss at `sasi`, 42% at `scsi`. Zero-wait-state floor, as ever.
|
|
3. **The dual-display-path plan (FINDINGS 24.5/25.6) is withdrawn as incoherent
|
|
— the sixth false premise this project has caught.** The compose path needs a
|
|
RAM copy of the previous reconstruction; the direct path's whole selling
|
|
point is that it keeps none. Mixing them displays stale pixels on **70 of 120
|
|
frames**, worst frame 18.8% of the screen. Every coherent repair is worse
|
|
than not mixing. `tools/analysis/10_pathmix_drift.py`, kept runnable as a
|
|
counterexample and gated in `check.sh`. FINDINGS 28.1.
|
|
4. **24.5 also compared a copy against a copy.** Its 53.6% and 76.6% both come
|
|
from `blit.s` and neither includes decoding. Compose = decode-into-RAM *plus*
|
|
the 53.6% blit, so it is strictly dearer than decoding into GVRAM. There was
|
|
never a crossover. The player has **one path and no reference frame**, which
|
|
also gives back 96 KB.
|
|
5. **The four block modes cost 300 / 448 / 400 cycles, not one number.** V4 is
|
|
1.49x a V1 block while the mode decision charges it 4x the bytes. The 24.5
|
|
model is 2.03x optimistic at the median. `tools/analysis/11_cpu_budget.py`
|
|
reproduces all four frames timed on the 68000 to within 1 point. FINDINGS 28.2.
|
|
6. **The container is big-endian but not aligned, and on a 68000 that is an
|
|
address error, not a slow read.** Frame records are variable-length and laid
|
|
end to end, so their boundaries land on odd addresses. Frame 0 decoded
|
|
perfectly, then the length read for frame 1 vectored into the IPL and sat
|
|
there for 59 emulated seconds looking like an infinite loop. Found by dumping
|
|
PC and the address registers — the code was right, the data layout was not.
|
|
FINDINGS 28.3. **Encoder gap: `encode.py` should pad records to 4 bytes.**
|
|
Measured cost 1.66 B/frame = 20 B/s against 110 KB/s.
|
|
7. **A full frame does not fit at 12fps in any mode.** All-V1 is 110.5%, all-V4
|
|
165.2%, all-RAW 147.6%. At most ~88% of the screen can change in one frame
|
|
however cheaply it is coded, and scene cuts change 100%. FINDINGS 28.5.
|
|
|
|
---
|
|
|
|
## What session 6 settled
|
|
|
|
1. **Rate control works, is wired in, and is ON by default.** `encode.py`
|
|
bisects lam per frame under a leaky bucket; `--fixed-lam` restores session 5
|
|
behaviour. FINDINGS 27.
|
|
2. **Both overshoots are closed for under 1 dB.** On the Singe window, totals
|
|
including audio: `sasi` 137.4 -> **109.5 KB/s** (target 110) for -0.60 dB,
|
|
`scsi` 381.6 -> **280.0 KB/s** (target 280) for -0.91 dB. Zero frames hit the
|
|
lam=800 cliff at either profile. FINDINGS 27.2.
|
|
3. **The FINDINGS 26 desync is gone by construction, not by tuning.** The
|
|
encoder is frame-drivable (`vq_hybrid.frame_ctx` / `decide` / `paint`) and
|
|
rate control feeds back the frame it actually emitted. The regression test
|
|
`tools/analysis/09_ratectl_drift.py` goes 111/120 drifting frames -> **0**,
|
|
and it is now part of `./tools/bench/check.sh`. FINDINGS 27.1.
|
|
4. **Rate control makes the display path cheaper.** Raising lam moves blocks to
|
|
SKIP and V1, so there is less to write: `scsi`'s median display cost drops
|
|
53.6% -> 47.1%. The decoder conclusion of 25.6 is unaffected. FINDINGS 27.3.
|
|
5. **FINDINGS 26.5 was wrong in both halves, and this is the fifth false premise
|
|
this project has caught.** `_paint` was not the bottleneck (14% of a frame)
|
|
and the ladder was never "minutes" (~18 s; the minutes were k-means in
|
|
`build`). Vectorising it was still right — 17.1x — but what actually makes
|
|
per-frame rate control affordable is that `VQ.assign` depends on neither
|
|
`lam` nor `prev`, so it is cached: a 12-step search over 120 frames costs
|
|
**0.31 s** against 49.1 s. FINDINGS 27.6.
|
|
6. **`--prefill` is a trap and defaults to 0.** It buys a permission to overshoot
|
|
of exactly bucket/nframes; at prefill=1.0 the Singe window goes to 116.3 KB/s
|
|
against a 110 ceiling, and on a 14-frame clip it disables the controller
|
|
outright. FINDINGS 27.4.
|
|
7. **Fixed-lam `sasi` was already 5% over target on 00020**, the clip everyone
|
|
called easy — nothing noticed because the profile table quotes PSNR, not
|
|
bitrate. FINDINGS 27.5.
|
|
8. **1.2-second clips cannot be used to judge rate control.** The bucket's
|
|
startup transient is bucket/nframes: 6% on a 10 s window, 20% on 00020. Same
|
|
lesson as FINDINGS 25.3, different costume.
|
|
|
|
---
|
|
|
|
## Start here: is the tree still green?
|
|
|
|
```
|
|
./tools/bench/check.sh
|
|
```
|
|
~4 min, needs the Blu-ray mounted. From source media it re-runs both display
|
|
regression tests, the rate-control drift test (session 6, now covering BOTH
|
|
controllers -- bytes, and bytes+cycles), the display-path coherency
|
|
counterexample and a **120-frame 68000 decode** (session 7), then prints
|
|
`ALL GREEN`. Verified green at end of session 8.
|
|
|
|
Do not run two of these at once, and do not run one alongside a MAME timing
|
|
job: they share `tmp/` snapshot directories and log files, and the second run
|
|
silently truncates the first one's output.
|
|
If it fails, fix that before doing anything else — everything downstream assumes
|
|
the display path is pixel-exact.
|
|
|
|
The two session-7 stages are worth knowing the shape of before they fail on you:
|
|
- `10_pathmix_drift.py` is expected to exit **non-zero**; `check.sh` fails if it
|
|
ever starts passing, because that would mean the counterexample behind the
|
|
one-path decoder had stopped reproducing.
|
|
- the decode stage needs `tmp/rc_fr_singe_sasi_rcprofile.dlx` and will spend
|
|
~55 s encoding it if it is missing, nearly all of that k-means in `H.build`.
|
|
|
|
## Decisions locked
|
|
|
|
| decision | value | why |
|
|
|---|---|---|
|
|
| Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint |
|
|
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
|
|
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
|
|
| **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 |
|
|
| **Quality modes** | **two: `sasi` and `scsi`** (USER DECISION, session 2) | one codec, one decoder, one bitstream; only `lam` differs |
|
|
| Profile axis | **I/O bandwidth only** | the profiles say nothing about CPU; **both target the same stock 10MHz 68000**, and the Super has SCSI at 10MHz. FINDINGS 28.7 |
|
|
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
|
|
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
|
|
| SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution |
|
|
|
|
### The SASI/SCSI question is RESOLVED
|
|
Session 1 left "which machine do we target" open. The user's answer: **ship both**,
|
|
as two quality profiles. This is now implemented rather than hypothetical — the
|
|
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
|
|
|
|
| profile | target | lam | quality (00020 / 00146) | machine |
|
|
|---|---|---|---|---|
|
|
| `sasi` | 110 KB/s | 60 (floor) | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
|
|
| `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
|
|
|
|
**That "machine" column is about the BUS, not the CPU.** The profiles are an
|
|
I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has
|
|
built-in SCSI at 10 MHz (`x68k.cpp:1194`, `40_MHz_XTAL/4`, same as the base
|
|
machine), and only the XVI is faster. **Both profiles target the same stock
|
|
10 MHz 68000**, so both must fit the same 833,333-cycle frame budget — and as of
|
|
session 7 neither does. FINDINGS 28.7.
|
|
|
|
**As of session 6 `lam` is a floor, not a setting.** The target is a ceiling and
|
|
the encoder bisects lam per frame to stay under it; the profile's lam is the
|
|
best quality it is allowed to spend on a quiet frame. On the worst sustained
|
|
window that takes `sasi` from 137.4 to 109.5 KB/s and `scsi` from 381.6 to
|
|
280.0 KB/s, for -0.60 and -0.91 dB. FINDINGS 27.2.
|
|
|
|
Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on
|
|
SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**,
|
|
not an average over seek latency.
|
|
|
|
**Both profiles fit with room.** Ring-buffer simulation on the real per-frame
|
|
sizes gives **zero required prefill** for every scene at both profiles: the fill
|
|
delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds
|
|
that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance,
|
|
far more than an SD-backed seek needs. FINDINGS 21.
|
|
|
|
An earlier warning here said `scsi` did not fit because a frame peaked at 96.4%
|
|
of the pipe. That compared instantaneous demand to a sustained rate as if they
|
|
had to match frame-by-frame; with a buffer the test is cumulative, and it passes.
|
|
|
|
`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
|
|
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
|
|
RD curve rather than derived from the hardware. See FINDINGS 17.
|
|
|
|
Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured
|
|
and rejected — see FINDINGS 14, it was a false-good result from a rate model
|
|
that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff.
|
|
|
|
Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised
|
|
frame (measured 0.00 dB loss). The profiles are two points on one continuous
|
|
rate-distortion curve, not two codecs.
|
|
|
|
---
|
|
|
|
## What session 3 settled
|
|
|
|
1. **The display path works and is verified end to end.** First real frame on an
|
|
emulated X68000 screen: `docs/images/x68k_first_frame_compare.png`. Full
|
|
write-up in **FINDINGS 22**. Everything before this session was Python-side
|
|
or a headless `-video none` run, which cannot snapshot at all.
|
|
2. **The render is pixel-exact, not merely close.** With monitor contrast at 15,
|
|
all 256 palette entries render exactly as `GGGGGRRRRRBBBBBI` + `pal6bit`
|
|
predicts. That exactness is the regression test — see
|
|
`tools/bench/verify_frame.py`, which exits non-zero if it ever drifts.
|
|
3. **Three hardware facts that were previously assumed are now confirmed from
|
|
MAME 0.277 source**, not folklore: the palette word format, the 1024-byte
|
|
GVRAM line stride, and the 256-colour page aliasing in `HARDWARE.md`. All
|
|
three were already written down correctly; they are now cited.
|
|
4. ~~**A new quality ceiling was measured** — the 15-bit+I palette alone costs
|
|
38.88 dB.~~ **Superseded by session 4:** that figure assumed the shared LSB
|
|
`I` is always 1. Chosen per entry, the ceiling is **40.81 dB**. FINDINGS 23.3.
|
|
5. **Two shell traps that wedged session 2's background jobs** are documented in
|
|
the working-setup section below. They cost ~1.5 h of wall clock and a wedged
|
|
CPU core, and one of them was hit again this session.
|
|
|
|
## What session 5 settled
|
|
|
|
1. **68000 code drew a frame, and the blit was measured.** `tools/bench/blit.s`
|
|
+ `blit.lua`. The snapshot passes `verify_frame256.py` unchanged — pixel-exact
|
|
in the real 256x256 mode. **FINDINGS 23.5 is closed**: no longer "proven from
|
|
Lua only".
|
|
2. **The 38% full-frame blit estimate is dead. It is 53.6%.** And that is a
|
|
zero-wait-state floor — MAME models no GVRAM wait states, so real hardware is
|
|
worse. FINDINGS 24. Every variant was hand-derived from the MC68000 timing
|
|
tables before being measured and the two agree to 0.006-0.43%, so this is not
|
|
another MAME artefact.
|
|
3. **Reading the source frame is exactly half the blit cost** (V1 53.6% vs a
|
|
write-only floor V3 of 27.1%). That is what makes the architecture question
|
|
below live.
|
|
4. **That number is now measured, and the answer is "implement both paths".**
|
|
On the worst sustained window found on the disc, 30% of frames (`sasi`) to
|
|
53% (`scsi`) sit above the 70% crossover and want the flat blit; the rest
|
|
want direct-to-GVRAM. A player that picks per frame — the mode headers are
|
|
parsed before any pixel is written, so the count is free — pays a **median
|
|
37.0%** and is **capped at 53.6%**. FINDINGS 25.6.
|
|
5. **The sustained action sequence exists, was found by measurement, and breaks
|
|
both profiles.** `tools/analysis/07_motion_survey.py` scans a whole stream
|
|
for the hottest sliding window; on 00223 it is t=539.4s, the Singe endgame,
|
|
at 2.01x the stream mean. There, fixed-lam `sasi` overshoots 110 -> 129.6
|
|
KB/s (+18%) and `scsi` 280 -> 373.8 KB/s (+34%). **Rate control is no longer
|
|
insurance — it is required.** FINDINGS 25.3.
|
|
6. **The two largest streams on the disc are bonus material, not game footage.**
|
|
00216 is the feature with a burned-in commentary PiP; 00215 is the commentary
|
|
itself. **00223 (9.4 min) is the clean one.** A size-ranked survey would have
|
|
encoded live action. FINDINGS 25.1.
|
|
7. **Rate control is unsound as written, caught before wiring it up.** The
|
|
lam-ladder in `ratectl.py` picks frames from independent temporal chains,
|
|
so SKIP blocks reference reconstructions the decoder never saw: 111 of 120
|
|
frames drift, worst frame 43.4%, reported PSNR overstated 0.36 dB. Regression
|
|
test `tools/analysis/09_ratectl_drift.py`. FINDINGS 26.
|
|
8. **On hard content the scene palette, not the display, is the binding
|
|
ceiling** — 31.33 dB on the Singe window against 39.90 dB on 00020 and 40.81
|
|
dB for the X68000 display. `scsi` is already within 0.51 dB of it.
|
|
FINDINGS 25.4.
|
|
|
|
### Superseded within session 5
|
|
4a. **The decoder architecture hinged on one unmeasured number.** Writing
|
|
codewords straight into GVRAM costs 76.6% of the frame budget for a *full*
|
|
frame (V4 — the 1024-byte stride kills the `movem.l` burst), but scales with
|
|
the non-SKIP block fraction and needs **no RAM reference frame at all**,
|
|
because the previous frame is already in GVRAM. Compose-then-blit is a flat
|
|
53.6%. **They cross at 70% of blocks changed.** FINDINGS 24.5.
|
|
|
|
---
|
|
|
|
## What session 4 settled
|
|
|
|
1. **A real 256x256 CRTC mode exists and is verified.** `crtc_mode.lua`, derived
|
|
from `x68k_crtc.cpp`'s divisor ladder rather than recalled — the derivation is
|
|
self-checking (368 = 1104/3 exactly, so the horizontal registers divide by
|
|
three with no remainder). Snapshot is native 256x512, active area pixel-exact,
|
|
letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone.
|
|
2. **The palette ceiling was wrong by 2 dB, in our favour.** The shared LSB `I`
|
|
must be chosen **per palette entry**, not hardcoded to 1. Doing so lifts the
|
|
display ceiling from 38.85 to **40.81 dB** and is the only way to get true
|
|
black at all (`pal6bit(1) = 4`). 102 of 256 entries want `I = 0`. This
|
|
supersedes FINDINGS 22.4 and gives `scsi` ~2 dB more headroom than believed.
|
|
**The encoder does not do this yet** — see the encoder-gaps list.
|
|
3. **Letterboxing costs one palette entry.** 255 colours + a reserved black at
|
|
index 0, with `I = 0` on it. `prep_frame.py --reserve-black`. FINDINGS 23.4.
|
|
4. **MAME's graphics double-scan is phase-shifted one raster line** — pairs are
|
|
(1,2),(3,4),..., not (0,1), because `get_gfx_pixel` halves the *absolute*
|
|
scanline and `vbegin = 41` is odd. Cost a false failure. FINDINGS 23.2.
|
|
|
|
---
|
|
|
|
## What session 2 settled
|
|
|
|
1. **The critical-path question is answered.** "Does VQ soften Bluth's linework
|
|
unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.**
|
|
Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two
|
|
images in `docs/images/`. Both profiles use **k=256**; see item 2b.
|
|
2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured).
|
|
Halving the framerate does not halve the bitrate. FINDINGS 8.
|
|
2b. **A fourth false-good result was produced and caught this session** — k=1024
|
|
codebooks looked like a +2.4 dB free win because the rate model charged 1 byte
|
|
for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
|
|
3. **The 256-colour palettised frame is the real quality ceiling** and it looks
|
|
excellent. Judge the codec against that, not against 1080p.
|
|
4. Encoder exists and produces a real bitstream: `tools/encoder/`.
|
|
|
|
---
|
|
|
|
## Encoder — working
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
|
|
python3 tools/encoder/encode.py /tmp/fr_00020 out.dlx --profile sasi --preview p.png
|
|
```
|
|
|
|
| file | role |
|
|
|---|---|
|
|
| `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise |
|
|
| `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR |
|
|
| `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision |
|
|
| `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control |
|
|
| `encode.py` | CLI + `DLX1` container writer |
|
|
|
|
`DLX1` container layout is documented in the `encode.py` docstring. All
|
|
multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`.
|
|
|
|
### Known encoder gaps
|
|
- ~~Rate control is written but not yet wired into `encode.py`.~~ **DONE,
|
|
session 6.** It is on by default; `--fixed-lam` restores the old behaviour.
|
|
Gated by `tools/analysis/09_ratectl_drift.py`, which is now in `check.sh`.
|
|
- **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in
|
|
the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
|
|
- **Frame records are not aligned.** They must be padded to a 4-byte boundary:
|
|
unaligned is an ADDRESS ERROR on a 68000, not a slow read (FINDINGS 28.3).
|
|
`prep_dlx.py` repairs it at load time, which a player streaming from disc
|
|
cannot do. The pad is real bytes on disc, so it belongs inside the rate
|
|
controller's accounting. 1.66 B/frame, 20 B/s.
|
|
- **The mode decision is blind to CPU cost.** It charges V4 four payload bytes
|
|
and ignores that it costs 1.49x a V1 block to draw. This is the top item at
|
|
the head of this file. FINDINGS 28.2.
|
|
- **Palette packing is not implemented in the encoder.** It still emits 24-bit
|
|
palettes; the X68000 word packing happens Lua-side. Whatever writes real
|
|
palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3,
|
|
worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4).
|
|
- Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
|
|
- ~~`_paint` is a Python per-block loop.~~ **DONE, session 6** — vectorised,
|
|
17.1x. It was never the bottleneck, though: `VQ.assign` is 78% of a frame and
|
|
`H.build`'s k-means is 51 s of a 55 s run. **That k-means is now the thing to
|
|
attack before the full-disc survey**, not anything in the per-frame path.
|
|
FINDINGS 27.6.
|
|
|
|
---
|
|
|
|
## Working setup (unchanged from session 1, re-verified)
|
|
|
|
**MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**.
|
|
```
|
|
mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3
|
|
```
|
|
**Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`
|
|
|
|
**Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM`
|
|
(still mounted as of end of session 2).
|
|
|
|
**MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the
|
|
notifier subscription in a global; the stack register is `SP` not `A7`;
|
|
`autoboot_script` fires at PC=0 before boot) are documented in FINDINGS.
|
|
|
|
**Two shell traps, both hit again this session:**
|
|
- piping MAME (or any long job) through `grep` block-buffers — write to a file.
|
|
- `pkill -f <pattern>` matches your own shell and kills it (exit 144).
|
|
Use `pkill -x` or kill by PID.
|
|
- **`pgrep -f <name> | xargs kill` kills your own shell too — exit 144.** Same
|
|
root cause as the `pkill -f` trap above: the shell's own command line contains
|
|
the pattern. **Hit again in session 5**, which makes it four times across three
|
|
sessions. Kill by PID captured at launch (`$!`), or use `pkill -x`.
|
|
- **`until ! pgrep -f foo.py; do sleep; done` watcher loops never exit.** The
|
|
watching shell's own command line contains the string `foo.py`, so `pgrep -f`
|
|
matches the watcher itself and the loop spins forever. Session 2 left 11 of
|
|
these wedged for over an hour. Wait on the PID (`while kill -0 $PID`) or on a
|
|
sentinel file the job touches when it finishes -- never on a `-f` name match.
|
|
- **`timeout N mame ...` does not kill MAME.** MAME catches SIGTERM and, with an
|
|
autoboot script blocked waiting on a flag that never arrives, never reaches
|
|
its shutdown path. `timeout` without `-k` then waits forever while MAME burns
|
|
a full core at `-nothrottle`. Always `timeout -k 5 N`.
|
|
|
|
---
|
|
|
|
## Disk throughput benchmark — still blocked, no longer gating
|
|
|
|
`IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested
|
|
hypotheses are in session 1's notes (git history of this file, commit 65112b9);
|
|
the ordered plan for retrying is in **`docs/BENCHMARK.md`**.
|
|
|
|
**Status changed twice this session — read this rather than the git history.**
|
|
It was briefly promoted to critical-path while the working bandwidth figure was
|
|
misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the
|
|
ring-buffer simulation showing **zero required prefill** for both profiles
|
|
(FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not**
|
|
available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a
|
|
"measure it and maybe ship transparent" decision waiting.
|
|
|
|
What the benchmark is still worth doing for:
|
|
- **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is
|
|
not recorded. Every profile hangs off it.
|
|
- **Confirming DMA is actually used.** If transfers fall back to PIO the CPU
|
|
cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the
|
|
binding constraint. This is the worst plausible outcome and the cheapest to
|
|
check — do it first.
|
|
|
|
**Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are
|
|
functional models, not timing-accurate; a KB/s figure from MAME measures the
|
|
emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach
|
|
(MAME validates the path, derivation bounds it, real hardware settles it).
|
|
|
|
## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5), by a 68000 DECODER (session 7).
|
|
|
|
The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`.
|
|
|
|
**Session 7 went from copying a frame to parsing one.** `src/player/decode.s`
|
|
reads DLX1, dispatches all four block modes and writes straight into GVRAM;
|
|
120 frames decoded in sequence are pixel-exact against `tools/encoder/dlx.py`
|
|
(`tools/bench/verify_decode.py`, in `check.sh`). The blit numbers below are
|
|
still correct for what they measured — a *copy* — but they are no longer the
|
|
display-path budget: the decoder costs 300/448/400 cycles per V1/V4/RAW block
|
|
and misses the 12fps budget on 31% of frames. FINDINGS 28.
|
|
|
|
**Session 5 closed the gap this paragraph used to describe.** GVRAM is now
|
|
filled by 68000 instructions and the result is still pixel-exact, and the blit
|
|
cost is measured rather than estimated: **53.6% of a 12fps frame**, not 38%
|
|
(FINDINGS 24). The remaining caveat is different and narrower: MAME models
|
|
**no GVRAM wait states**, so 53.6% is a floor and real hardware is worse.
|
|
Full write-up in **FINDINGS 22**. Harness: `tools/bench/show_frame.lua` +
|
|
`tools/bench/prep_frame.py`.
|
|
|
|
Three facts the player MUST honour, none of which were guessable:
|
|
|
|
| what | where | value |
|
|
|---|---|---|
|
|
| **Un-hide the graphics layer** | CRTC R20 `$E80028` | clear bit 11 ("G-VRAM set to buffer"); IPL leaves `0x0B16` |
|
|
| Colour setup (256c) | CRTC R20 bits 9-8 | `0x0100` |
|
|
| **Monitor contrast** | `$E8E001` bits 3-0 | IPL leaves **14**; write **15** or everything renders 7% dark |
|
|
|
|
The `R20 = 0x0116` value quoted here in session 3 is the **768-wide IPL timing**
|
|
with the gate cleared. The shipping value is **`R20 = 0x0110`** — see the mode
|
|
table in `tools/bench/crtc_mode.lua`, which is now the single source of truth
|
|
for all of R00-R08 and R20.
|
|
|
|
Bit 11 is the one that cost the most time: GVRAM writes land and read back
|
|
correctly while the layer is invisible, so the video controller looks guilty and
|
|
is not. Contrast `0` blanks the screen — free fade-to-black for transitions.
|
|
|
|
Palette format is now **confirmed from MAME source**, not assumed:
|
|
`GGGGGRRRRRBBBBBI` (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as
|
|
`pal6bit((field<<1)|I)`. With contrast at 15 the render is **pixel-exact**.
|
|
|
|
Ceiling: the 15-bit+I palette costs **40.81 dB** against the 24-bit palettised
|
|
source, once `I` is chosen per entry (FINDINGS 23.3 — session 3's 38.88 dB
|
|
assumed `I = 1`). Still the same order as the `scsi` profile's own codec error
|
|
(39.4 dB), so `scsi` remains near display-transparent, with ~2 dB more headroom
|
|
than session 3 thought.
|
|
|
|
Snapshot recipe that works (`-video none` CANNOT snapshot):
|
|
```
|
|
SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
|
|
-sound none -nothrottle -plugins -autoboot_script <script>.lua \
|
|
-snapshot_directory ./snap -snapview native -seconds_to_run 6
|
|
```
|
|
`-snapview native` drops MAME's LED artwork and gives a clean 768x512 screen.
|
|
|
|
## Next steps, in priority order
|
|
|
|
1. ~~**Measure the non-SKIP block fraction.**~~ **DONE, session 5**, and its
|
|
answer **WITHDRAWN in session 7** — FINDINGS 28.1/28.2. It concluded
|
|
"implement both display paths and pick per frame, median 37.0%, capped at
|
|
53.6%". Mixing the paths is incoherent (the compose path needs a RAM
|
|
reference the direct path never writes) and the two costs it compared were
|
|
both copies with no decode in either. **The shipping decoder has one path.**
|
|
The non-SKIP fraction is still reported by `encode.py` and is still the right
|
|
thing to look at — it is just no longer a switch. Original framing kept
|
|
below, because its instruction to report the distribution rather than the
|
|
mean is the part that held up:
|
|
FINDINGS 24.5: compose-in-RAM-then-blit costs a flat 53.6% of the frame
|
|
budget; decode-direct-to-GVRAM costs 76.6% x (fraction of blocks that are not
|
|
SKIP) and needs no RAM reference frame. **They cross at 70%.** Which side of
|
|
70% the content sits on decides which decoder inner loop to write, so this
|
|
must come before writing one.
|
|
**It needs no new machinery** — the mode decision in `vq_hybrid.py` already
|
|
computes it per frame and simply never reports it. Add the histogram
|
|
(SKIP / V1 / V4 / RAW counts per frame) to `encode.py` output and run it over
|
|
the clips already extracted. Report the *distribution*, not the mean: a
|
|
scene-cut frame is ~100% non-SKIP and a held frame near 0%, and the mean of
|
|
those two is a number describing no actual frame.
|
|
|
|
1b. ~~**Wire rate control into `encode.py`.**~~ **DONE, session 6.** FINDINGS 27.
|
|
Both overshoots closed for under 1 dB, drift test at zero, `check.sh` gates
|
|
it. The remaining rate-control question is not a defect: whether `--rc-floor
|
|
open` is worth taking on quiet content. It measured as worth **0.00 dB** on
|
|
the Singe window (no frame there is quiet enough to saturate the bucket), so
|
|
it needs a genuinely quiet scene to decide, and it is a quality-per-byte
|
|
judgement rather than a correctness one.
|
|
|
|
2. ~~**68000 decoder skeleton.**~~ **DONE, session 7.** `src/player/decode.s`,
|
|
pixel-exact over 120 frames, gated in `check.sh`. It answered the question it
|
|
was written to answer, and the answer is no: **it does not fit** — mean 81.7%
|
|
of a 12fps frame, 31% of frames over 100%. FINDINGS 28. The follow-on is
|
|
priority 0 at the top of this file.
|
|
|
|
2a. **Re-budget everything against the MEASURED per-mode costs**, not 53.6% and
|
|
not 38%. Session 7 replaced the model twice over (FINDINGS 28.2): the display
|
|
path is not one number times a block fraction, and the median frame is 74.4%
|
|
rather than 36.6%. The original note is kept below because its warning about
|
|
downstream figures derived from a dead estimate is exactly what happened
|
|
again.
|
|
~~Re-budget everything against 53.6%, not 38%.~~ Several downstream figures
|
|
were derived from the old estimate. The blit alone now eats over half the
|
|
frame at 12fps in the compose-then-blit design, before any decode, and MAME
|
|
models no GVRAM wait states so that is a floor. This may reopen questions
|
|
that were closed against the 38% number — check FINDINGS 17.2's entropy-coding
|
|
rejection, which was argued as "54% LZ4 with no room beside a 38% blit". The
|
|
conclusion gets *stronger*, not weaker, but the arithmetic should be restated.
|
|
|
|
2b. **Pad frame records to 4 bytes in `encode.py`.** Not optional: unaligned
|
|
records are an address error on a 68000 (FINDINGS 28.3), and `prep_dlx.py`
|
|
currently repairs it at load time, which the shipping player streaming from
|
|
disc cannot do. The padding is real bytes on disc, so it has to be inside
|
|
the rate controller's accounting, not added after it. 20 B/s at 12fps.
|
|
|
|
3. **Full-disc survey.** Now scoped by session 5 rather than open-ended: the
|
|
worst *sustained* window is measured (FINDINGS 25), so what remains is the
|
|
distribution over content, not the worst case.
|
|
- Classify **content / menu / bonus** — not just menu vs content. FINDINGS
|
|
25.1: the two largest streams are bonus material and look like content by
|
|
size, duration and bitrate alike.
|
|
- Run `tools/analysis/07_motion_survey.py` per stream first; it is cheap
|
|
(96x72 greyscale) and gives a hot-window shortlist so the expensive encode
|
|
only runs where it matters.
|
|
- ~~Vectorise `_paint` before this run.~~ Done. The cost to attack now is
|
|
`H.build`'s k-means: 51 s of a 55 s run, and it runs once per scene.
|
|
- ~~Do it after rate control (1b), or it measures an encoder nobody ships.~~
|
|
Rate control is in, so the survey now measures the shipping encoder.
|
|
|
|
5. **Confirm DMA vs PIO in MAME** (see the benchmark section above) — cheap, and
|
|
the only thing that could still move CPU into the binding position.
|
|
6. **Resolve the framing question** (FINDINGS 12: crop vs squash vs wide).
|
|
Needs an eyeball against arcade reference, not a measurement.
|
|
7. **Import the scene graph.** SNES project `data/events/` (MIT, cleared),
|
|
cross-checked against DirkSimple (zlib) which transcribed the same data
|
|
independently — diff them to catch transcription errors before committing
|
|
any of it to 68000 tables.
|
|
8. **ADPCM audio.** MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in
|
|
`ratectl.py`, not yet extracted or encoded.
|
|
|
|
### Explicitly abandoned — do not re-propose
|
|
- ~~Entropy-code the payload.~~ Deflate decode is ~216% of the frame budget on a
|
|
68000; LZ4 is ~54% with no room beside a 38% blit (FINDINGS 17.2). All bitrates
|
|
are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8
|
|
to a compression upper bound, not a shippable design.
|
|
- ~~k=1024 codebooks.~~ False-good result from a rate model that charged 1 byte
|
|
for a 10-bit index (FINDINGS 14). k=256 wins at every matched bitrate.
|
|
- ~~Flat 4x4 VQ.~~ Rejected by eye (FINDINGS 9).
|
|
|
|
## Not yet started
|
|
- **A player, as opposed to a decoder.** `src/player/decode.s` parses DLX1,
|
|
dispatches all four block modes and draws pixel-exact frames, but it decodes
|
|
from RAM that Lua pre-loaded. There is no disc streaming, no ring buffer, no
|
|
audio, no timing against the VBL, and no scene branching.
|
|
- **Codebook expansion on the 68000.** `prep_dlx.py` does it host-side because
|
|
it is a load-time cost and including it would flatter or damn the inner loop.
|
|
The player must do it: 8 KB + 2 KB per scene.
|
|
- ADPCM audio extraction/encoding
|
|
- Disk image packaging
|
|
- Game logic (scene branching, input windows, death clips)
|
|
|
|
## Reproducing the 256x256 mode result (session 4)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
|
|
mkdir -p tmp/snap256 && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 90 mame x68000 \
|
|
-bios ipl10 -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/show_frame256.lua \
|
|
-snapshot_directory ./snap256 -snapview native -seconds_to_run 6
|
|
cd .. && python3 tools/bench/verify_frame256.py
|
|
```
|
|
Exits non-zero on any drift. Expected: `256x512 native, double-scan exact,
|
|
active 256x192 pixel-exact, letterbox true black`, ceiling 40.81 dB.
|
|
|
|
## Reproducing the display result
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame.bin 0
|
|
mkdir -p tmp/snap_verify && cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 \
|
|
-video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/show_frame.lua \
|
|
-snapshot_directory ./snap_verify -snapview native -seconds_to_run 6
|
|
cd .. && python3 tools/bench/verify_frame.py
|
|
```
|
|
Verified cold from the Blu-ray at end of session 3: exact match, 38.88 dB.
|
|
(That 38.88 is correct *for this test*: `show_frame.lua` still packs `I = 1`.
|
|
The 40.81 dB ceiling comes from `show_frame256.lua`, which picks `I` per entry.)
|
|
|
|
`tmp/` is gitignored scratch. The frames are NOT in the repo — regenerate them
|
|
with `extract.py`; the earlier ones lived in `/tmp` and do not survive a reboot.
|
|
|
|
## Reference material on this box (not in the repo)
|
|
|
|
- **MAME 0.277 source: `~/src/mame-mame0277/`** (tarball `~/src/mame0277.tar.gz`).
|
|
Downloaded this session to settle the graphics-layer question. The files that
|
|
matter are `src/mame/sharp/x68k_v.cpp`, `x68k_crtc.cpp`, `x68k_crtc.h`,
|
|
`x68k.cpp`. **Read these before theorising about X68000 video behaviour** —
|
|
six register-poking attempts failed against a gate that one grep found.
|
|
- Blu-ray mounted at `/media/reala-misaki/BDROM` via
|
|
`udisksctl loop-setup -r -f DRAGONS_LAIR.iso`.
|
|
|
|
---
|
|
|
|
## Parked ideas (not scheduled, not abandoned)
|
|
|
|
- **Cliff Hanger, retitled as Lupin III** (user, session 4). Stern's 1983
|
|
laserdisc game was cut from *Castle of Cagliostro* and *Mystery of Mamo* with
|
|
the Lupin branding stripped; a port would restore it. Technically **cheaper
|
|
than this project**: same content class (cel animation, flat colour, hard
|
|
cuts), ~13 min of footage vs Dragon's Lair's ~22, and flatter linework than
|
|
Bluth's, so fewer blocks should escape to V4/RAW. The codec, the display path,
|
|
and `crtc_mode.lua` would all drop straight in.
|
|
**The real cost is media prep, not code:** there is no clean master cut to
|
|
Stern's scene boundaries the way `DRAGONS_LAIR.iso` is, so the footage would
|
|
have to be sourced and cut to match. Not to be started until the CPU path is
|
|
proven — it changes nothing about whether this design works.
|
|
|
|
## Reproducing the blit measurement (session 5)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
|
|
tools/vasm/vasmm68k_mot -Fbin -o tmp/blit.bin tools/bench/blit.s
|
|
mkdir -p tmp/snap_blit && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
|
|
-bios ipl10 -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/blit.lua \
|
|
-snapshot_directory ./snap_blit -snapview native -seconds_to_run 120
|
|
```
|
|
~25 s wall. Prints cycles/frame and % of a 12fps budget for V1-V4, and snapshots
|
|
V1's output. To check that snapshot is still pixel-exact:
|
|
`sed 's|snap256|snap_blit|' tools/bench/verify_frame256.py | python3 -`
|
|
|
|
Not added to `check.sh`: `check.sh` asserts pixel-exactness, and asserting wall
|
|
timings there would make the green-light check sensitive to host load.
|
|
|
|
## Reproducing the decoder result (session 7)
|
|
|
|
```
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_fr_singe_sasi_rcprofile.dlx --profile sasi
|
|
python3 tools/bench/prep_dlx.py tmp/rc_fr_singe_sasi_rcprofile.dlx
|
|
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s
|
|
mkdir -p tmp/snap_decode && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
|
|
-bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/decode.lua \
|
|
-snapshot_directory ./snap_decode -snapview native -seconds_to_run 150
|
|
cd .. && python3 tools/bench/verify_decode.py tmp/rc_fr_singe_sasi_rcprofile.dlx
|
|
```
|
|
~90 s wall. Prints cycles/frame and % of a 12fps budget for four real frames
|
|
spanning the non-SKIP distribution, four synthetic single-mode frames, and one
|
|
full 120-frame pass; then verifies the last frame is pixel-exact. Expected:
|
|
median 73.8%, p90 116.4%, max 135.8%, mean 81.7%; V1 299.9 / V4 448.2 / RAW
|
|
400.4 cycles per block.
|
|
|
|
`-ramsize 2M` matters — MAME defaults to 4M and the locked target is a stock 2MB
|
|
machine. `DLX_VERIFY_ONLY=1` drops the timing anchors, which is how `check.sh`
|
|
runs it.
|
|
|
|
Score a container against the measured costs without touching MAME:
|
|
```
|
|
python3 tools/analysis/11_cpu_budget.py tmp/rc_fr_singe_scsi_rcprofile.dlx
|
|
```
|
|
And re-demonstrate why there is only one display path (exits non-zero **by
|
|
design** — it is the counterexample):
|
|
```
|
|
python3 tools/analysis/10_pathmix_drift.py # 70/120 frames corrupt
|
|
python3 tools/analysis/10_pathmix_drift.py --fix direct # clean, and cheapest
|
|
```
|
|
|
|
## Reproducing the rate-control result (session 6)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
|
|
for prof in sasi scsi; do
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof --fixed-lam
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof
|
|
done
|
|
python3 tools/analysis/09_ratectl_drift.py # must exit 0, zero drifting frames
|
|
```
|
|
Expected, totals including the 7.8 KB/s audio allowance: `sasi` 137.4 -> 109.5
|
|
KB/s and 27.82 -> 27.22 dB; `scsi` 381.6 -> 280.0 KB/s and 30.81 -> 29.90 dB;
|
|
zero frames at the lam=800 cliff in either. ~55 s per encode, nearly all of it
|
|
k-means in `H.build`.
|
|
|
|
The block-mode map now renders the rate-controlled encoder by default:
|
|
```
|
|
python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes_rc.webm \
|
|
--profile sasi --scale 2 # add --fixed-lam to compare
|
|
```
|
|
|
|
**Do not judge rate control on `tmp/fr_00020`.** It is 14 frames; the leaky
|
|
bucket's startup transient is bucket/nframes, so it lands 18% under target there
|
|
for reasons that have nothing to do with the content. FINDINGS 27.5.
|
|
|
|
## Reproducing the sustained-action result (session 5)
|
|
|
|
```
|
|
python3 tools/analysis/07_motion_survey.py 00223 10 # -> hottest window t=539.4s
|
|
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_sasi.dlx --profile sasi
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_scsi.dlx --profile scsi
|
|
python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes.webm \
|
|
--profile sasi --scale 2
|
|
```
|
|
`extract.py` now takes optional `[start_s] [dur_s]` — needed because 00223 is
|
|
9.4 min and the windows that stress the codec are seconds long.
|
|
|
|
`08_mode_map.py` renders palettised source | decoded | block-mode map at 12fps.
|
|
Output format follows the extension; **prefer `.webm`** — GIF re-quantises to
|
|
256 colours, which is a poor fit for output whose subject is colour fidelity,
|
|
and runs larger. It uses `yuv444p` because the mode map is flat saturated colour
|
|
on a 4-pixel grid and chroma subsampling smears exactly those edges.
|