prosolis
c5ca56330e
Three things, and the last one reversed itself when the datasheet arrived.
A SECOND EMULATOR. tools/bench/c68k/ links px68k's C68K core into a headless
harness -- no SDL, no ROMs, no emulated machine, because the decoder touches
nothing but RAM, the control block and GVRAM. decode.s is now pixel-exact under
two independent CPU cores, and cycle-table error against MAME is bounded at
3.3%, running against us. MAME 0.277's M68000 turns out to be the MICROCODE
core, not Musashi (m68000.lst + m68000gen.py), so this is two structurally
different timing models agreeing rather than two tables. FINDINGS 28.8's "V4
costs more than RAW" reproduces independently. FINDINGS 37.
THE BUS. Nothing since FINDINGS 24 had counted the 68000's local memory bus --
one 4-clock cycle at a time, carrying instruction prefetch as well as data. The
decoder occupies 86.7% of it and PREFETCH IS 62% OF THAT TRAFFIC, so a data-only
count understates occupancy by 2x. Two sources check each other: c68k_bench
counts every bus callback exactly, and a static walk of decode.lst supplies the
prefetch no emulator here can report. The walk reproduces the measured data half
to 0.04%, which is what licenses its prefetch half, and 15_bus_occupancy.py is a
gate rather than a report because every bus figure depends on that check.
FINDINGS 38.
THE DMAC CHAIN LOSES. FINDINGS 29.6 named it the one uncosted lever. Costed from
bus arithmetic -- a read cycle plus a write cycle, 8 clocks a pixel -- it scored
1/120 frames over budget against the v6 span's 10/120 and looked decisive. Then
the MC68450 manual (Motorola Jul 1989, now at ~/src/mc68450.pdf): Fig 4-25 sheet
4 puts a dual-address word between two 16-bit ports at 9 CLOCKS, because note 2
gives the DMAC 4-clock reads and 5-clock WRITES. The 68000 writes in 4.
DMAC 9.000 clocks/pixel datasheet
v6 9.152 clocks/pixel measured, FINDINGS 30
1.7%. Scored additively, 86% of what remains of the DMAC's advantage is v6's
24-pixel padding quantum -- a property of its unrolled movem chain, fixable in
software with a finer tail chain, worth 55/120 -> 18/120 against the DMAC's
12/120. Recommendation: fix the quantum, drop the DMAC. Six frames does not buy
a reserved channel, a two-region container layout and a timing dependency
neither emulator here can verify. The container is identical either way -- v6's
record and an HD63450 chaining entry are both 6 bytes, so the chain array IS the
span table -- so nothing is foreclosed. FINDINGS 39.
TWO CORRECTIONS TO MY OWN WORK IN THE SAME SESSION:
- I argued FINDINGS 35's flat CPU debit for the disk was too pessimistic and
rescored the window at 53/120 with max(CPU, bus). Wrong. A 68000 has no cache
and a two-word prefetch queue, so it stalls the moment another master takes
the bus, and the MC68450 hands the bus over in SLABS under limited-rate
auto-request rather than interleaving per operand. DMA is additive. 84/120
stands and 14_dmac_chain.py reproduces it exactly. What 86.7% occupancy really
says is that there is almost no room to overlap anything. FINDINGS 38.3.
- The first DMAC costing was derived where a primary source existed. Both wrong
answers were confident and both were caught by reading the manual.
Also landed:
- FINDINGS 5's 8 clocks/word for the SCSI DMA, STATUS's own "most load-bearing
unmeasured number", is now bracketed by the datasheet: 5 clk/word with the bus
held, ~12 if the DMAC arbitrates per word. 8 is a supported midpoint, and
which end applies is a player design decision worth 7 clocks a word on a
480 KB/s stream. FINDINGS 39.7.
- check.sh gains two gates: the C68K pixel-exact decode (seconds, no MAME) and
the bus-model self-check. Both skip cleanly without a px68k checkout.
- spanned blocks are now charged their mode-map dispatch, which FINDINGS 30.7
flagged as uncounted in 12_span_tradeoff.py.
- MAME timed runs must be budgeted by WALL CLOCK, not -seconds_to_run: this box
runs x68000 at ~0.033x realtime and two runs were killed by their own timeout.
That is why the all-RAW cell in 37.3 is empty. The C68K harness does the same
work in seconds because it emulates a CPU and not a machine.
Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
908 lines
51 KiB
Markdown
908 lines
51 KiB
Markdown
# Status & next-session handoff — end of session 10 (2026-08-23)
|
|
|
|
## Where this stands
|
|
|
|
Session 10 cross-checked the whole cycle model against a second emulator, then
|
|
found that the model was denominated in the wrong currency.
|
|
|
|
**FINDINGS 38 is the headline: the project is BUS-bound, not CPU-bound.** Nothing
|
|
since FINDINGS 24 had counted the 68000's local memory bus — one 4-clock cycle at
|
|
a time, carrying instruction prefetch as well as data. Measured, the decoder
|
|
occupies **86.7%** of it, and **prefetch is 62% of that traffic**. Scoring the
|
|
`scsi` window with the bus as the shared resource, **52 of 53 missed frames are
|
|
bus-limited and one is CPU-limited**. Every optimisation since 24 has been aimed
|
|
at the budget that is not binding.
|
|
|
|
The measurement is two sources checking each other: `c68k_bench` counts every bus
|
|
callback exactly, and a static walk of `decode.lst` adds the prefetch no emulator
|
|
here can report. The walk reproduces the measured data half to **0.04%**, which
|
|
is what licenses its prefetch half.
|
|
|
|
**The DMAC array-chain LOSES, on the datasheet.** My first pass derived its cost
|
|
from bus arithmetic — a read cycle plus a write cycle, 8 clocks a pixel — and
|
|
scored it at 1/120 frames over budget against v6's 10/120. Then I read the
|
|
MC68450 manual (Motorola Jul 1989, bitsavers). **Fig 4-25 sheet 4: a
|
|
dual-address word between two 16-bit ports is 9 clocks, because note 2 gives the
|
|
DMAC 4-clock reads and 5-clock WRITES.** The 68000 writes in 4.
|
|
|
|
| per pixel | clocks | source |
|
|
|---|---:|---|
|
|
| DMAC dual-address word | **9.000** | MC68450 Fig 4-25 sheet 4 |
|
|
| v6 `movem` chain | **9.152** | MEASURED, FINDINGS 30 |
|
|
|
|
Scored additively (see below), against the same mode maps:
|
|
|
|
| | frames over budget |
|
|
|---|---:|
|
|
| today | 84/120 |
|
|
| v6 span as built | 55/120 |
|
|
| **v6 with a finer chain tail — software only** | **18/120** |
|
|
| DMAC chain | 12/120 |
|
|
|
|
**86% of the DMAC's advantage is v6's 24-pixel padding quantum**, which is a
|
|
property of v6's unrolled `movem` chain and fixable in software: add a second
|
|
chain of 2-register units for the tail and the padding drops from 23 pixels to
|
|
3, at no per-span cost. **Recommendation (mine): fix the quantum, drop the
|
|
DMAC.** Six frames of 120 does not buy a reserved channel, a two-region
|
|
container layout, and a timing dependency neither emulator here can verify.
|
|
FINDINGS 39.
|
|
|
|
The container work is not wasted either way: v6's record and an HD63450 chaining
|
|
entry are both 6 bytes, so **the chain array is the span table** and nothing has
|
|
to change if a hardware measurement later moves the DMAC's number. FINDINGS 39.1.
|
|
|
|
**I also got FINDINGS 35 wrong before the datasheet corrected me.** I argued its
|
|
flat CPU debit was too pessimistic and rescored the window at 53/120 using
|
|
`max(CPU, bus)`. A 68000 has no cache and a two-word prefetch queue, so it
|
|
stalls the moment another master takes the bus, and the MC68450 hands the bus
|
|
over in *slabs* under limited-rate auto-request rather than interleaving per
|
|
operand. **DMA is additive. 84/120 stands.** What 86.7% bus occupancy really
|
|
says is that there is almost no room to overlap anything on this machine.
|
|
FINDINGS 38.3.
|
|
|
|
**And the datasheet settled STATUS's own "most load-bearing unmeasured number".**
|
|
FINDINGS 5's 8 clocks/word for the SCSI DMA: Fig 4-25 sheet 3 gives
|
|
single-address write 5 clocks, so it is **5 clk/word with the bus held and ~12
|
|
if the DMAC arbitrates per word**. 8 is the midpoint of a bracket the datasheet
|
|
supports. Which end applies is a player design decision — how the MB89352 drives
|
|
REQ, and whether cycle-steal-with-hold is used — worth 7 clocks a word on a
|
|
480 KB/s stream. FINDINGS 39.7.
|
|
|
|
**The decoder is now pixel-exact under two independent CPU cores.**
|
|
`tools/bench/c68k/` links px68k's C68K core into a headless harness and decodes
|
|
the same container to the same pixels. Cycle-table error against MAME is bounded
|
|
at **3.3%**, and it runs against us. Two incidental results worth keeping: MAME
|
|
0.277's `M68000` is the **microcode core**, not Musashi (`m68000.lst` +
|
|
`m68000gen.py`), so this is two structurally different timing models agreeing
|
|
rather than two tables; and FINDINGS 28.8's "V4 costs more than RAW" reproduces
|
|
independently. FINDINGS 37.
|
|
|
|
Everything below this line is from session 9 and still stands unless a session-10
|
|
section above says otherwise.
|
|
|
|
## NEXT SESSION, in order
|
|
|
|
0. **Green light first.** `./tools/bench/check.sh` (~5 min, Blu-ray mounted).
|
|
Verified green at end of session 9. The gate container is now
|
|
`tmp/rc_fr_singe_scsi_cpufit.dlx`, and it decodes **80 of 120 frames**, not
|
|
120: the rig loads the whole stream into a 2 MB machine and a `scsi` window
|
|
is 2.84 MB, which overran the top of RAM by 940 KB and produced a failure
|
|
that looked exactly like a decoder bug. `prep_dlx.py` now truncates to what
|
|
fits and says so; `verify_decode.py --nframes` replays the same prefix; and
|
|
the stage fails loudly if the sequential pass did not complete. FINDINGS 36.
|
|
|
|
At 278.6 KB/s a 2 MB machine holds ~6.7 s of stream, so **the strongest test
|
|
in the tree can only ever audit a prefix of a window.** Getting the coverage
|
|
back means gating on more than one window, not one longer pass.
|
|
|
|
1. **Measure the finer chain tail with `span.sh`.** It is the largest single win
|
|
on the table — 55/120 to 18/120 — it is software only, and the tool that
|
|
measures it already exists: `blit.s` gains a v7 with a 2-register tail chain,
|
|
`prep_spans.py` generates the streams, `span.sh` times all of it in ~25 s and
|
|
asserts every config drew a pixel-exact frame. The 18/120 figure is DERIVED
|
|
from a conservative 56-clocks-per-4-pixel-unit model and should not be
|
|
believed until it is measured. FINDINGS 39.4.
|
|
|
|
2. **Make sure the player actually gets DMA.** DMA-vs-PIO is a property of our
|
|
code, not the board: the CZ-6BS1's DMA path is real and modelled
|
|
(FINDINGS 32.4), but if the player reads through IOCS and IOCS does PIO we
|
|
get the 120/120 row. `docs/BENCHMARK.md` item 4 (drive the MB89352 directly)
|
|
is no longer an optimisation. Benchmark `x68000 -exp1 cz6bs1`, **never
|
|
`x68ksupr`** — MAME's internal SCSI has no DMA glue at all (`// TODO:
|
|
duplicate DMA glue from CZ-6BS1`) and would measure a PIO fallback the real
|
|
machine does not have.
|
|
|
|
3. **Re-decide the framerate, now that it is the lever that pays for the disk.**
|
|
10 fps absorbs the DMA steal on current estimates. This was item 5 and a
|
|
quality question; it is now arithmetic. It is still the user's call.
|
|
|
|
4. **Then spans, on the CPU.** The format is fully specified by measurement
|
|
(FINDINGS 30.2, costs 30.5, scene-cut arithmetic 30.6) and `decode.s` does
|
|
not implement them yet. Build the v7 tail from item 1 into it. The DMAC
|
|
variant is costed and loses (39.5); the container is identical either way
|
|
(39.1), so nothing is foreclosed.
|
|
|
|
5. **Re-run the ring-buffer simulation at the surviving rate** and confirm the
|
|
488 KB/s figure's provenance (FINDINGS 29.5/30.7, still open).
|
|
|
|
**Do not start by hand-optimising `decode.s`.** The hand-derived timings agree
|
|
with the measurements to 0.5% on V1 and 1% on RAW (FINDINGS 28.4), and FINDINGS
|
|
34 confirms the model on a second container. The inner loop is close to what the
|
|
instruction set allows; the cycles to be won are in the budget, not the loop.
|
|
|
|
**Three of this session's near-misses had the same shape: an unobservable run
|
|
almost produced a false finding.** The 23-minute "hang" (FINDINGS 34.1), the
|
|
RAM overrun that looked like a 4x-slow decoder (FINDINGS 36), and the truncated
|
|
gate that reported 49,005 differing pixels. In each case the instrument was
|
|
broken, not the thing being measured. **Always `stdbuf -oL` a MAME job that
|
|
prints progress.** A file is
|
|
block-buffered too, so a long run is unobservable until it exits — and an
|
|
unobservable run that is merely finishing looks exactly like one that is wedged.
|
|
Session 8 lost this measurement to that; session 9 lost 23 minutes to it before
|
|
spending 25 seconds getting the answer with `stdbuf`. FINDINGS 34.1.
|
|
|
|
---
|
|
|
|
## What session 8 settled
|
|
|
|
0. **The mode decision can see cycles, it is on by default, and it costs
|
|
0.26 dB.** `decide(ctx, lam, mu)` minimises `D + lam*bytes + mu*cycles`;
|
|
`ratectl` bisects mu per frame against a HARD 833,333-cycle ceiling (bytes
|
|
bank in the ring buffer, cycles cannot — there is no double buffer to decode
|
|
ahead into). `sasi` 37/120 misses -> 1, `scsi` 51 -> 1. Bitrate does not
|
|
move: mu changes which modes are bought, not how many bytes. FINDINGS 31,
|
|
`tools/analysis/13_cpu_ratectl.py`.
|
|
0b. **28.7's "11 frames are impossible" was too pessimistic — it is 1.** That
|
|
floor held the SKIP set fixed; the real decision can also move a block to
|
|
SKIP, which above ~90% non-SKIP is the only lever left. FINDINGS 31.3.
|
|
0c. **V4 collapses when cycles are priced**, as 28.8 predicted: 25.2 -> 20.3%
|
|
of blocks at `sasi` and **15.0 -> 5.3%** at `scsi`, where RAW takes it. RAW
|
|
is dearer in bytes and cheaper in cycles, so the byte lagrangian's preference
|
|
inverts and only the byte-rich profile can take the escape.
|
|
0d. **SKIP's price depends on its neighbours, and the way out is two cost
|
|
functions**: a ranking constant inside the per-block lagrangian, the exact
|
|
clustered rule (`vq_hybrid.cycles`, validated to 1 point against the 68000)
|
|
for the frame-level bisection. That function is now defined once and imported
|
|
by `11_cpu_budget.py`. FINDINGS 31.4.
|
|
0e. **Both controllers are gated against decoder drift.**
|
|
`09_ratectl_drift.py` runs bytes-only AND bytes+cycles; both 0/120.
|
|
1. **The span is measured: 43.7 cycles/span + 9.152/pixel, fitted to 0.3% over
|
|
eleven span lengths.** `tools/bench/blit.s` v5/v6, `prep_spans.py`,
|
|
`span.lua`, driven by `tools/bench/span.sh` (~25 s, not in `check.sh`
|
|
because it is a wall timing). FINDINGS 30.
|
|
2. **Only in an encoder-assisted format.** `{u32 absolute GVRAM address, u16
|
|
jump displacement}` into an unrolled chain, versus `(x, npix)` and a decoder
|
|
that works it out: 43.7 + 9.152 against 97.9 + 10.46. All the arithmetic a
|
|
span decoder would do per frame is known at encode time. FINDINGS 30.2.
|
|
3. **The per-pixel cost is a function of register pressure**, which FINDINGS 24
|
|
could not have shown: 9.08 was a fixed blit with 12 registers free, v5 can
|
|
spare 8 and pays 10.46, v6 gets 12 back by making the encoder hold the state.
|
|
4. **Short spans die in the remainder path, and the fix is padding.** A 12-pixel
|
|
span costs more than a 16-pixel one in v5. v6 has no remainder path: lengths
|
|
are multiples of 24 pixels, padding is free of everything but bytes, and an
|
|
overrun past the visible 256 lands in the invisible half of the 1024-byte
|
|
line stride. FINDINGS 30.3.
|
|
5. **Odd-`x` alignment is free** (259.0 vs 261.8 cycles/span) — expected on a
|
|
16-bit bus, now measured rather than assumed.
|
|
6. **The trade is smaller than 29 derived but the conclusion holds**, including
|
|
29.4's reopening of the scene cut. All 23 timing configs also drew a
|
|
pixel-exact frame, so nothing here was timed against a decoder that skipped
|
|
work. FINDINGS 30.5/30.6.
|
|
|
|
---
|
|
|
|
## What session 7 settled
|
|
|
|
1. **68000 code parses a bitstream and draws frames, pixel-exact.**
|
|
`src/player/decode.s` + `tools/bench/decode.lua`. 120 frames of the Singe
|
|
window decoded in sequence, all four block modes, verified against the new
|
|
reference decoder `tools/encoder/dlx.py`. Because SKIP blocks are claims
|
|
about the previous frame, the last frame is only right if all 120 were.
|
|
In `check.sh` now. **FINDINGS 28.**
|
|
2. **It does not fit.** Mean 81.7% of a 12fps frame, p90 116.4%, worst 135.8%;
|
|
31% of frames miss at `sasi`, 42% at `scsi`. Zero-wait-state floor, as ever.
|
|
3. **The dual-display-path plan (FINDINGS 24.5/25.6) is withdrawn as incoherent
|
|
— the sixth false premise this project has caught.** The compose path needs a
|
|
RAM copy of the previous reconstruction; the direct path's whole selling
|
|
point is that it keeps none. Mixing them displays stale pixels on **70 of 120
|
|
frames**, worst frame 18.8% of the screen. Every coherent repair is worse
|
|
than not mixing. `tools/analysis/10_pathmix_drift.py`, kept runnable as a
|
|
counterexample and gated in `check.sh`. FINDINGS 28.1.
|
|
4. **24.5 also compared a copy against a copy.** Its 53.6% and 76.6% both come
|
|
from `blit.s` and neither includes decoding. Compose = decode-into-RAM *plus*
|
|
the 53.6% blit, so it is strictly dearer than decoding into GVRAM. There was
|
|
never a crossover. The player has **one path and no reference frame**, which
|
|
also gives back 96 KB.
|
|
5. **The four block modes cost 300 / 448 / 400 cycles, not one number.** V4 is
|
|
1.49x a V1 block while the mode decision charges it 4x the bytes. The 24.5
|
|
model is 2.03x optimistic at the median. `tools/analysis/11_cpu_budget.py`
|
|
reproduces all four frames timed on the 68000 to within 1 point. FINDINGS 28.2.
|
|
6. **The container is big-endian but not aligned, and on a 68000 that is an
|
|
address error, not a slow read.** Frame records are variable-length and laid
|
|
end to end, so their boundaries land on odd addresses. Frame 0 decoded
|
|
perfectly, then the length read for frame 1 vectored into the IPL and sat
|
|
there for 59 emulated seconds looking like an infinite loop. Found by dumping
|
|
PC and the address registers — the code was right, the data layout was not.
|
|
FINDINGS 28.3. **Encoder gap: `encode.py` should pad records to 4 bytes.**
|
|
Measured cost 1.66 B/frame = 20 B/s against 110 KB/s.
|
|
7. **A full frame does not fit at 12fps in any mode.** All-V1 is 110.5%, all-V4
|
|
165.2%, all-RAW 147.6%. At most ~88% of the screen can change in one frame
|
|
however cheaply it is coded, and scene cuts change 100%. FINDINGS 28.5.
|
|
|
|
---
|
|
|
|
## What session 6 settled
|
|
|
|
1. **Rate control works, is wired in, and is ON by default.** `encode.py`
|
|
bisects lam per frame under a leaky bucket; `--fixed-lam` restores session 5
|
|
behaviour. FINDINGS 27.
|
|
2. **Both overshoots are closed for under 1 dB.** On the Singe window, totals
|
|
including audio: `sasi` 137.4 -> **109.5 KB/s** (target 110) for -0.60 dB,
|
|
`scsi` 381.6 -> **280.0 KB/s** (target 280) for -0.91 dB. Zero frames hit the
|
|
lam=800 cliff at either profile. FINDINGS 27.2.
|
|
3. **The FINDINGS 26 desync is gone by construction, not by tuning.** The
|
|
encoder is frame-drivable (`vq_hybrid.frame_ctx` / `decide` / `paint`) and
|
|
rate control feeds back the frame it actually emitted. The regression test
|
|
`tools/analysis/09_ratectl_drift.py` goes 111/120 drifting frames -> **0**,
|
|
and it is now part of `./tools/bench/check.sh`. FINDINGS 27.1.
|
|
4. **Rate control makes the display path cheaper.** Raising lam moves blocks to
|
|
SKIP and V1, so there is less to write: `scsi`'s median display cost drops
|
|
53.6% -> 47.1%. The decoder conclusion of 25.6 is unaffected. FINDINGS 27.3.
|
|
5. **FINDINGS 26.5 was wrong in both halves, and this is the fifth false premise
|
|
this project has caught.** `_paint` was not the bottleneck (14% of a frame)
|
|
and the ladder was never "minutes" (~18 s; the minutes were k-means in
|
|
`build`). Vectorising it was still right — 17.1x — but what actually makes
|
|
per-frame rate control affordable is that `VQ.assign` depends on neither
|
|
`lam` nor `prev`, so it is cached: a 12-step search over 120 frames costs
|
|
**0.31 s** against 49.1 s. FINDINGS 27.6.
|
|
6. **`--prefill` is a trap and defaults to 0.** It buys a permission to overshoot
|
|
of exactly bucket/nframes; at prefill=1.0 the Singe window goes to 116.3 KB/s
|
|
against a 110 ceiling, and on a 14-frame clip it disables the controller
|
|
outright. FINDINGS 27.4.
|
|
7. **Fixed-lam `sasi` was already 5% over target on 00020**, the clip everyone
|
|
called easy — nothing noticed because the profile table quotes PSNR, not
|
|
bitrate. FINDINGS 27.5.
|
|
8. **1.2-second clips cannot be used to judge rate control.** The bucket's
|
|
startup transient is bucket/nframes: 6% on a 10 s window, 20% on 00020. Same
|
|
lesson as FINDINGS 25.3, different costume.
|
|
|
|
---
|
|
|
|
## Start here: is the tree still green?
|
|
|
|
```
|
|
./tools/bench/check.sh
|
|
```
|
|
~4 min, needs the Blu-ray mounted. From source media it re-runs both display
|
|
regression tests, the rate-control drift test (session 6, now covering BOTH
|
|
controllers -- bytes, and bytes+cycles), the display-path coherency
|
|
counterexample and a **120-frame 68000 decode** (session 7), then prints
|
|
`ALL GREEN`. Verified green at end of session 8.
|
|
|
|
Do not run two of these at once, and do not run one alongside a MAME timing
|
|
job: they share `tmp/` snapshot directories and log files, and the second run
|
|
silently truncates the first one's output.
|
|
If it fails, fix that before doing anything else — everything downstream assumes
|
|
the display path is pixel-exact.
|
|
|
|
The two session-7 stages are worth knowing the shape of before they fail on you:
|
|
- `10_pathmix_drift.py` is expected to exit **non-zero**; `check.sh` fails if it
|
|
ever starts passing, because that would mean the counterexample behind the
|
|
one-path decoder had stopped reproducing.
|
|
- the decode stage needs `tmp/rc_fr_singe_sasi_rcprofile.dlx` and will spend
|
|
~55 s encoding it if it is missing, nearly all of that k-means in `H.build`.
|
|
|
|
## Decisions locked
|
|
|
|
| decision | value | why |
|
|
|---|---|---|
|
|
| Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint |
|
|
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
|
|
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
|
|
| **Codec** | **hybrid VQ: SKIP / V1 4x4 / V4 four-2x2 / RAW, per-block rate-distortion** | flat 4x4 VQ was measured and rejected — see FINDINGS 9-10 |
|
|
| **Quality modes** | **one: `scsi`** (USER DECISION, session 9 — `sasi` dropped) | a SASI volume is 40 MB and the game is 146 MiB at the LOWEST rate this codec makes. FINDINGS 32 |
|
|
| Delivery medium | **SD-backed SCSI** (BlueSCSI / SCSI2SD), as locked in session 2 | capacity does not bind on SD at any rate this codec makes; it is what killed SASI, and it is what rules CD-ROM OUT (650 MiB with spans, and 1x CD is ~150 KB/s). FINDINGS 32.3 |
|
|
| Profile axis | **I/O bandwidth only** | the profiles say nothing about CPU; **both target the same stock 10MHz 68000**, and the Super has SCSI at 10MHz. FINDINGS 28.7 |
|
|
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
|
|
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
|
|
| SNES project reuse | **MIT — cleared** | `data/events/` scene graph is reusable with attribution |
|
|
|
|
### The SASI/SCSI question is RESOLVED — and in session 9 SASI was DROPPED
|
|
Session 1 left "which machine do we target" open. Session 2's answer was **ship
|
|
both**, as two quality profiles. **Session 9 retired `sasi`** (USER DECISION) on
|
|
CAPACITY, not bandwidth: a SASI volume is limited to 40 MB, and the 22.8 minutes
|
|
of unique scene footage on the source Blu-ray is **146 MiB even at 110 KB/s** —
|
|
more than the machine's whole 4-unit SASI address space. FINDINGS 32.
|
|
|
|
Everything below this line about *two* profiles is the session-2..8 record, kept
|
|
because every measurement in FINDINGS 27-31 was taken against it. One profile
|
|
ships:
|
|
|
|
| profile | target | lam | quality (00020 / 00146) | machine |
|
|
|---|---|---|---|---|
|
|
| ~~`sasi`~~ | ~~110 KB/s~~ | ~~60~~ | ~~36.9 / 29.6 dB~~ | **RETIRED session 9** |
|
|
| `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
|
|
|
|
The 110 KB/s **rate point** is not necessarily gone with the interface: a 1x
|
|
CD-ROM sustains ~150 KB/s and CD-ROM is the only period medium with room for the
|
|
whole game (374 MiB at `scsi`, 600 MiB with spans). The user's call was to ship
|
|
one profile now and settle the medium when the pipe is measured. FINDINGS 32.3.
|
|
|
|
**That "machine" column is about the BUS, not the CPU.** The profiles are an
|
|
I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has
|
|
built-in SCSI at 10 MHz (`x68k.cpp:1194`, `40_MHz_XTAL/4`, same as the base
|
|
machine), and only the XVI is faster. **Both profiles target the same stock
|
|
10 MHz 68000**, so both must fit the same 833,333-cycle frame budget — and as of
|
|
session 7 neither does. FINDINGS 28.7.
|
|
|
|
**As of session 6 `lam` is a floor, not a setting.** The target is a ceiling and
|
|
the encoder bisects lam per frame to stay under it; the profile's lam is the
|
|
best quality it is allowed to spend on a quiet frame. On the worst sustained
|
|
window that takes `sasi` from 137.4 to 109.5 KB/s and `scsi` from 381.6 to
|
|
280.0 KB/s, for -0.60 and -0.91 dB. FINDINGS 27.2.
|
|
|
|
Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on
|
|
SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**,
|
|
not an average over seek latency.
|
|
|
|
**Both profiles fit with room.** Ring-buffer simulation on the real per-frame
|
|
sizes gives **zero required prefill** for every scene at both profiles: the fill
|
|
delivers 40.69 KB per frame time and only one measured frame (42.10 KB) exceeds
|
|
that, recovered by the next. A 256 KB buffer carries ~1 s of stall tolerance,
|
|
far more than an SD-backed seek needs. FINDINGS 21.
|
|
|
|
An earlier warning here said `scsi` did not fit because a frame peaked at 96.4%
|
|
of the pipe. That compared instantaneous demand to a sustained rate as if they
|
|
had to match frame-by-frame; with a buffer the test is cumulative, and it passes.
|
|
|
|
`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
|
|
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
|
|
RD curve rather than derived from the hardware. See FINDINGS 17.
|
|
|
|
Codebooks are **k=256 with 1-byte indices** in both profiles. k=1024 was measured
|
|
and rejected — see FINDINGS 14, it was a false-good result from a rate model
|
|
that undercharged the index. Do not ship past `lam~800`; FINDINGS 15 has the cliff.
|
|
|
|
Because of the RAW escape mode, `lam=0` is **pixel-exact** against the palettised
|
|
frame (measured 0.00 dB loss). The profiles are two points on one continuous
|
|
rate-distortion curve, not two codecs.
|
|
|
|
---
|
|
|
|
## What session 3 settled
|
|
|
|
1. **The display path works and is verified end to end.** First real frame on an
|
|
emulated X68000 screen: `docs/images/x68k_first_frame_compare.png`. Full
|
|
write-up in **FINDINGS 22**. Everything before this session was Python-side
|
|
or a headless `-video none` run, which cannot snapshot at all.
|
|
2. **The render is pixel-exact, not merely close.** With monitor contrast at 15,
|
|
all 256 palette entries render exactly as `GGGGGRRRRRBBBBBI` + `pal6bit`
|
|
predicts. That exactness is the regression test — see
|
|
`tools/bench/verify_frame.py`, which exits non-zero if it ever drifts.
|
|
3. **Three hardware facts that were previously assumed are now confirmed from
|
|
MAME 0.277 source**, not folklore: the palette word format, the 1024-byte
|
|
GVRAM line stride, and the 256-colour page aliasing in `HARDWARE.md`. All
|
|
three were already written down correctly; they are now cited.
|
|
4. ~~**A new quality ceiling was measured** — the 15-bit+I palette alone costs
|
|
38.88 dB.~~ **Superseded by session 4:** that figure assumed the shared LSB
|
|
`I` is always 1. Chosen per entry, the ceiling is **40.81 dB**. FINDINGS 23.3.
|
|
5. **Two shell traps that wedged session 2's background jobs** are documented in
|
|
the working-setup section below. They cost ~1.5 h of wall clock and a wedged
|
|
CPU core, and one of them was hit again this session.
|
|
|
|
## What session 5 settled
|
|
|
|
1. **68000 code drew a frame, and the blit was measured.** `tools/bench/blit.s`
|
|
+ `blit.lua`. The snapshot passes `verify_frame256.py` unchanged — pixel-exact
|
|
in the real 256x256 mode. **FINDINGS 23.5 is closed**: no longer "proven from
|
|
Lua only".
|
|
2. **The 38% full-frame blit estimate is dead. It is 53.6%.** And that is a
|
|
zero-wait-state floor — MAME models no GVRAM wait states, so real hardware is
|
|
worse. FINDINGS 24. Every variant was hand-derived from the MC68000 timing
|
|
tables before being measured and the two agree to 0.006-0.43%, so this is not
|
|
another MAME artefact.
|
|
3. **Reading the source frame is exactly half the blit cost** (V1 53.6% vs a
|
|
write-only floor V3 of 27.1%). That is what makes the architecture question
|
|
below live.
|
|
4. **That number is now measured, and the answer is "implement both paths".**
|
|
On the worst sustained window found on the disc, 30% of frames (`sasi`) to
|
|
53% (`scsi`) sit above the 70% crossover and want the flat blit; the rest
|
|
want direct-to-GVRAM. A player that picks per frame — the mode headers are
|
|
parsed before any pixel is written, so the count is free — pays a **median
|
|
37.0%** and is **capped at 53.6%**. FINDINGS 25.6.
|
|
5. **The sustained action sequence exists, was found by measurement, and breaks
|
|
both profiles.** `tools/analysis/07_motion_survey.py` scans a whole stream
|
|
for the hottest sliding window; on 00223 it is t=539.4s, the Singe endgame,
|
|
at 2.01x the stream mean. There, fixed-lam `sasi` overshoots 110 -> 129.6
|
|
KB/s (+18%) and `scsi` 280 -> 373.8 KB/s (+34%). **Rate control is no longer
|
|
insurance — it is required.** FINDINGS 25.3.
|
|
6. **The two largest streams on the disc are bonus material, not game footage.**
|
|
00216 is the feature with a burned-in commentary PiP; 00215 is the commentary
|
|
itself. **00223 (9.4 min) is the clean one.** A size-ranked survey would have
|
|
encoded live action. FINDINGS 25.1.
|
|
7. **Rate control is unsound as written, caught before wiring it up.** The
|
|
lam-ladder in `ratectl.py` picks frames from independent temporal chains,
|
|
so SKIP blocks reference reconstructions the decoder never saw: 111 of 120
|
|
frames drift, worst frame 43.4%, reported PSNR overstated 0.36 dB. Regression
|
|
test `tools/analysis/09_ratectl_drift.py`. FINDINGS 26.
|
|
8. **On hard content the scene palette, not the display, is the binding
|
|
ceiling** — 31.33 dB on the Singe window against 39.90 dB on 00020 and 40.81
|
|
dB for the X68000 display. `scsi` is already within 0.51 dB of it.
|
|
FINDINGS 25.4.
|
|
|
|
### Superseded within session 5
|
|
4a. **The decoder architecture hinged on one unmeasured number.** Writing
|
|
codewords straight into GVRAM costs 76.6% of the frame budget for a *full*
|
|
frame (V4 — the 1024-byte stride kills the `movem.l` burst), but scales with
|
|
the non-SKIP block fraction and needs **no RAM reference frame at all**,
|
|
because the previous frame is already in GVRAM. Compose-then-blit is a flat
|
|
53.6%. **They cross at 70% of blocks changed.** FINDINGS 24.5.
|
|
|
|
---
|
|
|
|
## What session 4 settled
|
|
|
|
1. **A real 256x256 CRTC mode exists and is verified.** `crtc_mode.lua`, derived
|
|
from `x68k_crtc.cpp`'s divisor ladder rather than recalled — the derivation is
|
|
self-checking (368 = 1104/3 exactly, so the horizontal registers divide by
|
|
three with no remainder). Snapshot is native 256x512, active area pixel-exact,
|
|
letterbox true black. FINDINGS 23. The x=512 wrap of FINDINGS 22.5 is gone.
|
|
2. **The palette ceiling was wrong by 2 dB, in our favour.** The shared LSB `I`
|
|
must be chosen **per palette entry**, not hardcoded to 1. Doing so lifts the
|
|
display ceiling from 38.85 to **40.81 dB** and is the only way to get true
|
|
black at all (`pal6bit(1) = 4`). 102 of 256 entries want `I = 0`. This
|
|
supersedes FINDINGS 22.4 and gives `scsi` ~2 dB more headroom than believed.
|
|
**The encoder does not do this yet** — see the encoder-gaps list.
|
|
3. **Letterboxing costs one palette entry.** 255 colours + a reserved black at
|
|
index 0, with `I = 0` on it. `prep_frame.py --reserve-black`. FINDINGS 23.4.
|
|
4. **MAME's graphics double-scan is phase-shifted one raster line** — pairs are
|
|
(1,2),(3,4),..., not (0,1), because `get_gfx_pixel` halves the *absolute*
|
|
scanline and `vbegin = 41` is odd. Cost a false failure. FINDINGS 23.2.
|
|
|
|
---
|
|
|
|
## What session 2 settled
|
|
|
|
1. **The critical-path question is answered.** "Does VQ soften Bluth's linework
|
|
unacceptably?" — **flat 4x4 VQ: yes, badly. The hybrid (SKIP/V1/V4/RAW): no.**
|
|
Verified by eye, not just PSNR. See `docs/FINDINGS.md` 9-11 and the two
|
|
images in `docs/images/`. Both profiles use **k=256**; see item 2b.
|
|
2. **Session 1's 12fps bitrate was wrong** (183 KB/s claimed, 340 KB/s measured).
|
|
Halving the framerate does not halve the bitrate. FINDINGS 8.
|
|
2b. **A fourth false-good result was produced and caught this session** — k=1024
|
|
codebooks looked like a +2.4 dB free win because the rate model charged 1 byte
|
|
for a 10-bit index. FINDINGS 14. The k=256 configuration ships.
|
|
3. **The 256-colour palettised frame is the real quality ceiling** and it looks
|
|
excellent. Judge the codec against that, not against 1080p.
|
|
4. Encoder exists and produces a real bitstream: `tools/encoder/`.
|
|
|
|
---
|
|
|
|
## Encoder — working
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 /tmp/fr_00020 12 crop
|
|
python3 tools/encoder/encode.py /tmp/fr_00020 out.dlx --profile sasi --preview p.png
|
|
```
|
|
|
|
| file | role |
|
|
|---|---|
|
|
| `extract.py` | .m2ts -> 256x192 PNGs, 12fps, spatial-only denoise |
|
|
| `vq.py` | palette, blockify, hand-rolled k-means (no sklearn on this box), PSNR |
|
|
| `vq_hybrid.py` | the codec: 4 block modes + lagrangian mode decision |
|
|
| `ratectl.py` | SASI/SCSI profiles, leaky-bucket rate control |
|
|
| `encode.py` | CLI + `DLX1` container writer |
|
|
|
|
`DLX1` container layout is documented in the `encode.py` docstring. All
|
|
multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`.
|
|
|
|
### Known encoder gaps
|
|
- ~~Rate control is written but not yet wired into `encode.py`.~~ **DONE,
|
|
session 6.** It is on by default; `--fixed-lam` restores the old behaviour.
|
|
Gated by `tools/analysis/09_ratectl_drift.py`, which is now in `check.sh`.
|
|
- **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in
|
|
the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later.
|
|
- **Frame records are not aligned.** They must be padded to a 4-byte boundary:
|
|
unaligned is an ADDRESS ERROR on a 68000, not a slow read (FINDINGS 28.3).
|
|
`prep_dlx.py` repairs it at load time, which a player streaming from disc
|
|
cannot do. The pad is real bytes on disc, so it belongs inside the rate
|
|
controller's accounting. 1.66 B/frame, 20 B/s.
|
|
- **The mode decision is blind to CPU cost.** It charges V4 four payload bytes
|
|
and ignores that it costs 1.49x a V1 block to draw. This is the top item at
|
|
the head of this file. FINDINGS 28.2.
|
|
- **Palette packing is not implemented in the encoder.** It still emits 24-bit
|
|
palettes; the X68000 word packing happens Lua-side. Whatever writes real
|
|
palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3,
|
|
worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4).
|
|
- Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse.
|
|
- ~~`_paint` is a Python per-block loop.~~ **DONE, session 6** — vectorised,
|
|
17.1x. It was never the bottleneck, though: `VQ.assign` is 78% of a frame and
|
|
`H.build`'s k-means is 51 s of a 55 s run. **That k-means is now the thing to
|
|
attack before the full-disc survey**, not anything in the per-frame path.
|
|
FINDINGS 27.6.
|
|
|
|
---
|
|
|
|
## Working setup (unchanged from session 1, re-verified)
|
|
|
|
**MAME ROMs** — `~/mame/roms/x68000.zip`. Must pass **`-bios ipl10`**.
|
|
```
|
|
mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3
|
|
```
|
|
**Assembler** — `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`
|
|
|
|
**Blu-ray** — `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` -> `/media/reala-misaki/BDROM`
|
|
(still mounted as of end of session 2).
|
|
|
|
**MAME Lua harness** — `tools/bench/*.lua`, working. Three gotchas (retain the
|
|
notifier subscription in a global; the stack register is `SP` not `A7`;
|
|
`autoboot_script` fires at PC=0 before boot) are documented in FINDINGS.
|
|
|
|
**Two shell traps, both hit again this session:**
|
|
- piping MAME (or any long job) through `grep` block-buffers — write to a file.
|
|
- `pkill -f <pattern>` matches your own shell and kills it (exit 144).
|
|
Use `pkill -x` or kill by PID.
|
|
- **`pgrep -f <name> | xargs kill` kills your own shell too — exit 144.** Same
|
|
root cause as the `pkill -f` trap above: the shell's own command line contains
|
|
the pattern. **Hit again in session 5**, which makes it four times across three
|
|
sessions. Kill by PID captured at launch (`$!`), or use `pkill -x`.
|
|
- **`until ! pgrep -f foo.py; do sleep; done` watcher loops never exit.** The
|
|
watching shell's own command line contains the string `foo.py`, so `pgrep -f`
|
|
matches the watcher itself and the loop spins forever. Session 2 left 11 of
|
|
these wedged for over an hour. Wait on the PID (`while kill -0 $PID`) or on a
|
|
sentinel file the job touches when it finishes -- never on a `-f` name match.
|
|
- **`timeout N mame ...` does not kill MAME.** MAME catches SIGTERM and, with an
|
|
autoboot script blocked waiting on a flag that never arrives, never reaches
|
|
its shutdown path. `timeout` without `-k` then waits forever while MAME burns
|
|
a full core at `-nothrottle`. Always `timeout -k 5 N`.
|
|
|
|
---
|
|
|
|
## Disk throughput benchmark — still blocked, no longer gating
|
|
|
|
`IOCS _B_READ` returns -1 uniformly. Full diagnosis and the four untested
|
|
hypotheses are in session 1's notes (git history of this file, commit 65112b9);
|
|
the ordered plan for retrying is in **`docs/BENCHMARK.md`**.
|
|
|
|
**Status changed twice this session — read this rather than the git history.**
|
|
It was briefly promoted to critical-path while the working bandwidth figure was
|
|
misread as 4 MB/s. With the correct figure (**4 Mbps = 488 KB/s**) and the
|
|
ring-buffer simulation showing **zero required prefill** for both profiles
|
|
(FINDINGS 21), the design no longer hangs on it. Pixel-exact on SCSI is **not**
|
|
available at 4 Mbps — it needs 92-97% of the pipe — so there is no longer a
|
|
"measure it and maybe ship transparent" decision waiting.
|
|
|
|
What the benchmark is still worth doing for:
|
|
- **Confirming the 4 Mbps figure.** It is user-supplied and its provenance is
|
|
not recorded. Every profile hangs off it.
|
|
- **Confirming DMA is actually used.** If transfers fall back to PIO the CPU
|
|
cost rises far above the ~12-15% cycle-steal estimate and CPU becomes the
|
|
binding constraint. This is the worst plausible outcome and the cheapest to
|
|
check — do it first.
|
|
|
|
**Do not try to get the bandwidth number out of MAME.** Its SCSI/SASI devices are
|
|
functional models, not timing-accurate; a KB/s figure from MAME measures the
|
|
emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach
|
|
(MAME validates the path, derivation bounds it, real hardware settles it).
|
|
|
|
## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5), by a 68000 DECODER (session 7).
|
|
|
|
The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`.
|
|
|
|
**Session 7 went from copying a frame to parsing one.** `src/player/decode.s`
|
|
reads DLX1, dispatches all four block modes and writes straight into GVRAM;
|
|
120 frames decoded in sequence are pixel-exact against `tools/encoder/dlx.py`
|
|
(`tools/bench/verify_decode.py`, in `check.sh`). The blit numbers below are
|
|
still correct for what they measured — a *copy* — but they are no longer the
|
|
display-path budget: the decoder costs 300/448/400 cycles per V1/V4/RAW block
|
|
and misses the 12fps budget on 31% of frames. FINDINGS 28.
|
|
|
|
**Session 5 closed the gap this paragraph used to describe.** GVRAM is now
|
|
filled by 68000 instructions and the result is still pixel-exact, and the blit
|
|
cost is measured rather than estimated: **53.6% of a 12fps frame**, not 38%
|
|
(FINDINGS 24). The remaining caveat is different and narrower: MAME models
|
|
**no GVRAM wait states**, so 53.6% is a floor and real hardware is worse.
|
|
Full write-up in **FINDINGS 22**. Harness: `tools/bench/show_frame.lua` +
|
|
`tools/bench/prep_frame.py`.
|
|
|
|
Three facts the player MUST honour, none of which were guessable:
|
|
|
|
| what | where | value |
|
|
|---|---|---|
|
|
| **Un-hide the graphics layer** | CRTC R20 `$E80028` | clear bit 11 ("G-VRAM set to buffer"); IPL leaves `0x0B16` |
|
|
| Colour setup (256c) | CRTC R20 bits 9-8 | `0x0100` |
|
|
| **Monitor contrast** | `$E8E001` bits 3-0 | IPL leaves **14**; write **15** or everything renders 7% dark |
|
|
|
|
The `R20 = 0x0116` value quoted here in session 3 is the **768-wide IPL timing**
|
|
with the gate cleared. The shipping value is **`R20 = 0x0110`** — see the mode
|
|
table in `tools/bench/crtc_mode.lua`, which is now the single source of truth
|
|
for all of R00-R08 and R20.
|
|
|
|
Bit 11 is the one that cost the most time: GVRAM writes land and read back
|
|
correctly while the layer is invisible, so the video controller looks guilty and
|
|
is not. Contrast `0` blanks the screen — free fade-to-black for transitions.
|
|
|
|
Palette format is now **confirmed from MAME source**, not assumed:
|
|
`GGGGGRRRRRBBBBBI` (G 15:11, R 10:6, B 5:1, shared LSB I), expanded as
|
|
`pal6bit((field<<1)|I)`. With contrast at 15 the render is **pixel-exact**.
|
|
|
|
Ceiling: the 15-bit+I palette costs **40.81 dB** against the 24-bit palettised
|
|
source, once `I` is chosen per entry (FINDINGS 23.3 — session 3's 38.88 dB
|
|
assumed `I = 1`). Still the same order as the `scsi` profile's own codec error
|
|
(39.4 dB), so `scsi` remains near display-transparent, with ~2 dB more headroom
|
|
than session 3 thought.
|
|
|
|
Snapshot recipe that works (`-video none` CANNOT snapshot):
|
|
```
|
|
SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
|
|
-sound none -nothrottle -plugins -autoboot_script <script>.lua \
|
|
-snapshot_directory ./snap -snapview native -seconds_to_run 6
|
|
```
|
|
`-snapview native` drops MAME's LED artwork and gives a clean 768x512 screen.
|
|
|
|
## Next steps, in priority order
|
|
|
|
1. ~~**Measure the non-SKIP block fraction.**~~ **DONE, session 5**, and its
|
|
answer **WITHDRAWN in session 7** — FINDINGS 28.1/28.2. It concluded
|
|
"implement both display paths and pick per frame, median 37.0%, capped at
|
|
53.6%". Mixing the paths is incoherent (the compose path needs a RAM
|
|
reference the direct path never writes) and the two costs it compared were
|
|
both copies with no decode in either. **The shipping decoder has one path.**
|
|
The non-SKIP fraction is still reported by `encode.py` and is still the right
|
|
thing to look at — it is just no longer a switch. Original framing kept
|
|
below, because its instruction to report the distribution rather than the
|
|
mean is the part that held up:
|
|
FINDINGS 24.5: compose-in-RAM-then-blit costs a flat 53.6% of the frame
|
|
budget; decode-direct-to-GVRAM costs 76.6% x (fraction of blocks that are not
|
|
SKIP) and needs no RAM reference frame. **They cross at 70%.** Which side of
|
|
70% the content sits on decides which decoder inner loop to write, so this
|
|
must come before writing one.
|
|
**It needs no new machinery** — the mode decision in `vq_hybrid.py` already
|
|
computes it per frame and simply never reports it. Add the histogram
|
|
(SKIP / V1 / V4 / RAW counts per frame) to `encode.py` output and run it over
|
|
the clips already extracted. Report the *distribution*, not the mean: a
|
|
scene-cut frame is ~100% non-SKIP and a held frame near 0%, and the mean of
|
|
those two is a number describing no actual frame.
|
|
|
|
1b. ~~**Wire rate control into `encode.py`.**~~ **DONE, session 6.** FINDINGS 27.
|
|
Both overshoots closed for under 1 dB, drift test at zero, `check.sh` gates
|
|
it. The remaining rate-control question is not a defect: whether `--rc-floor
|
|
open` is worth taking on quiet content. It measured as worth **0.00 dB** on
|
|
the Singe window (no frame there is quiet enough to saturate the bucket), so
|
|
it needs a genuinely quiet scene to decide, and it is a quality-per-byte
|
|
judgement rather than a correctness one.
|
|
|
|
2. ~~**68000 decoder skeleton.**~~ **DONE, session 7.** `src/player/decode.s`,
|
|
pixel-exact over 120 frames, gated in `check.sh`. It answered the question it
|
|
was written to answer, and the answer is no: **it does not fit** — mean 81.7%
|
|
of a 12fps frame, 31% of frames over 100%. FINDINGS 28. The follow-on is
|
|
priority 0 at the top of this file.
|
|
|
|
2a. **Re-budget everything against the MEASURED per-mode costs**, not 53.6% and
|
|
not 38%. Session 7 replaced the model twice over (FINDINGS 28.2): the display
|
|
path is not one number times a block fraction, and the median frame is 74.4%
|
|
rather than 36.6%. The original note is kept below because its warning about
|
|
downstream figures derived from a dead estimate is exactly what happened
|
|
again.
|
|
~~Re-budget everything against 53.6%, not 38%.~~ Several downstream figures
|
|
were derived from the old estimate. The blit alone now eats over half the
|
|
frame at 12fps in the compose-then-blit design, before any decode, and MAME
|
|
models no GVRAM wait states so that is a floor. This may reopen questions
|
|
that were closed against the 38% number — check FINDINGS 17.2's entropy-coding
|
|
rejection, which was argued as "54% LZ4 with no room beside a 38% blit". The
|
|
conclusion gets *stronger*, not weaker, but the arithmetic should be restated.
|
|
|
|
2b. **Pad frame records to 4 bytes in `encode.py`.** Not optional: unaligned
|
|
records are an address error on a 68000 (FINDINGS 28.3), and `prep_dlx.py`
|
|
currently repairs it at load time, which the shipping player streaming from
|
|
disc cannot do. The padding is real bytes on disc, so it has to be inside
|
|
the rate controller's accounting, not added after it. 20 B/s at 12fps.
|
|
|
|
3. **Full-disc survey.** Now scoped by session 5 rather than open-ended: the
|
|
worst *sustained* window is measured (FINDINGS 25), so what remains is the
|
|
distribution over content, not the worst case.
|
|
- Classify **content / menu / bonus** — not just menu vs content. FINDINGS
|
|
25.1: the two largest streams are bonus material and look like content by
|
|
size, duration and bitrate alike.
|
|
- Run `tools/analysis/07_motion_survey.py` per stream first; it is cheap
|
|
(96x72 greyscale) and gives a hot-window shortlist so the expensive encode
|
|
only runs where it matters.
|
|
- ~~Vectorise `_paint` before this run.~~ Done. The cost to attack now is
|
|
`H.build`'s k-means: 51 s of a 55 s run, and it runs once per scene.
|
|
- ~~Do it after rate control (1b), or it measures an encoder nobody ships.~~
|
|
Rate control is in, so the survey now measures the shipping encoder.
|
|
|
|
5. **Confirm DMA vs PIO in MAME** (see the benchmark section above) — cheap, and
|
|
the only thing that could still move CPU into the binding position.
|
|
6. **Resolve the framing question** (FINDINGS 12: crop vs squash vs wide).
|
|
Needs an eyeball against arcade reference, not a measurement.
|
|
7. **Import the scene graph.** SNES project `data/events/` (MIT, cleared),
|
|
cross-checked against DirkSimple (zlib) which transcribed the same data
|
|
independently — diff them to catch transcription errors before committing
|
|
any of it to 68000 tables.
|
|
8. **ADPCM audio.** MSM6258, 15.6kHz mono, 7.8 KB/s — already budgeted in
|
|
`ratectl.py`, not yet extracted or encoded.
|
|
|
|
### Explicitly abandoned — do not re-propose
|
|
- ~~Entropy-code the payload.~~ Deflate decode is ~216% of the frame budget on a
|
|
68000; LZ4 is ~54% with no room beside a 38% blit (FINDINGS 17.2). All bitrates
|
|
are raw payload. This also demotes the "247 KB/s lossless" figure in FINDINGS 8
|
|
to a compression upper bound, not a shippable design.
|
|
- ~~k=1024 codebooks.~~ False-good result from a rate model that charged 1 byte
|
|
for a 10-bit index (FINDINGS 14). k=256 wins at every matched bitrate.
|
|
- ~~Flat 4x4 VQ.~~ Rejected by eye (FINDINGS 9).
|
|
|
|
## Not yet started
|
|
- **A player, as opposed to a decoder.** `src/player/decode.s` parses DLX1,
|
|
dispatches all four block modes and draws pixel-exact frames, but it decodes
|
|
from RAM that Lua pre-loaded. There is no disc streaming, no ring buffer, no
|
|
audio, no timing against the VBL, and no scene branching.
|
|
- **Codebook expansion on the 68000.** `prep_dlx.py` does it host-side because
|
|
it is a load-time cost and including it would flatter or damn the inner loop.
|
|
The player must do it: 8 KB + 2 KB per scene.
|
|
- ADPCM audio extraction/encoding
|
|
- Disk image packaging
|
|
- Game logic (scene branching, input windows, death clips)
|
|
|
|
## Reproducing the 256x256 mode result (session 4)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
|
|
mkdir -p tmp/snap256 && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 90 mame x68000 \
|
|
-bios ipl10 -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/show_frame256.lua \
|
|
-snapshot_directory ./snap256 -snapview native -seconds_to_run 6
|
|
cd .. && python3 tools/bench/verify_frame256.py
|
|
```
|
|
Exits non-zero on any drift. Expected: `256x512 native, double-scan exact,
|
|
active 256x192 pixel-exact, letterbox true black`, ceiling 40.81 dB.
|
|
|
|
## Reproducing the display result
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame.bin 0
|
|
mkdir -p tmp/snap_verify && cd tmp && SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 \
|
|
-video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/show_frame.lua \
|
|
-snapshot_directory ./snap_verify -snapview native -seconds_to_run 6
|
|
cd .. && python3 tools/bench/verify_frame.py
|
|
```
|
|
Verified cold from the Blu-ray at end of session 3: exact match, 38.88 dB.
|
|
(That 38.88 is correct *for this test*: `show_frame.lua` still packs `I = 1`.
|
|
The 40.81 dB ceiling comes from `show_frame256.lua`, which picks `I` per entry.)
|
|
|
|
`tmp/` is gitignored scratch. The frames are NOT in the repo — regenerate them
|
|
with `extract.py`; the earlier ones lived in `/tmp` and do not survive a reboot.
|
|
|
|
## Reference material on this box (not in the repo)
|
|
|
|
- **MC68450 DMAC manual: `~/src/mc68450.pdf`** (Motorola, Jul 1989, from
|
|
bitsavers; `curl` it with a browser User-Agent or you get a 403). This is the
|
|
primary source for FINDINGS 39 and it overturned two derived answers in one
|
|
session. Fig 4-25 is the timing table — sheet 1 chaining, sheet 3
|
|
single-address, sheet 4 dual-address; note 2 (4-clock reads, 5-clock writes)
|
|
is the one that mattered. Sects 4.5.2.1-3 are the arbitration overheads,
|
|
5.2.3.2 the limited-rate auto-request slabs. `pdftotext` handles it.
|
|
- **px68k source: `~/src/px68k`** — only `m68000/c68k.c` is used, by
|
|
`tools/bench/c68k/`.
|
|
- **MAME 0.277 source: `~/src/mame-mame0277/`** (tarball `~/src/mame0277.tar.gz`).
|
|
Downloaded this session to settle the graphics-layer question. The files that
|
|
matter are `src/mame/sharp/x68k_v.cpp`, `x68k_crtc.cpp`, `x68k_crtc.h`,
|
|
`x68k.cpp`. **Read these before theorising about X68000 video behaviour** —
|
|
six register-poking attempts failed against a gate that one grep found.
|
|
- Blu-ray mounted at `/media/reala-misaki/BDROM` via
|
|
`udisksctl loop-setup -r -f DRAGONS_LAIR.iso`.
|
|
|
|
---
|
|
|
|
## Parked ideas (not scheduled, not abandoned)
|
|
|
|
- **Cliff Hanger, retitled as Lupin III** (user, session 4). Stern's 1983
|
|
laserdisc game was cut from *Castle of Cagliostro* and *Mystery of Mamo* with
|
|
the Lupin branding stripped; a port would restore it. Technically **cheaper
|
|
than this project**: same content class (cel animation, flat colour, hard
|
|
cuts), ~13 min of footage vs Dragon's Lair's ~22, and flatter linework than
|
|
Bluth's, so fewer blocks should escape to V4/RAW. The codec, the display path,
|
|
and `crtc_mode.lua` would all drop straight in.
|
|
**The real cost is media prep, not code:** there is no clean master cut to
|
|
Stern's scene boundaries the way `DRAGONS_LAIR.iso` is, so the footage would
|
|
have to be sourced and cut to match. Not to be started until the CPU path is
|
|
proven — it changes nothing about whether this design works.
|
|
|
|
## Reproducing the blit measurement (session 5)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
|
|
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
|
|
tools/vasm/vasmm68k_mot -Fbin -o tmp/blit.bin tools/bench/blit.s
|
|
mkdir -p tmp/snap_blit && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
|
|
-bios ipl10 -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/blit.lua \
|
|
-snapshot_directory ./snap_blit -snapview native -seconds_to_run 120
|
|
```
|
|
~25 s wall. Prints cycles/frame and % of a 12fps budget for V1-V4, and snapshots
|
|
V1's output. To check that snapshot is still pixel-exact:
|
|
`sed 's|snap256|snap_blit|' tools/bench/verify_frame256.py | python3 -`
|
|
|
|
Not added to `check.sh`: `check.sh` asserts pixel-exactness, and asserting wall
|
|
timings there would make the green-light check sensitive to host load.
|
|
|
|
## Reproducing the decoder result (session 7)
|
|
|
|
```
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_fr_singe_sasi_rcprofile.dlx --profile sasi
|
|
python3 tools/bench/prep_dlx.py tmp/rc_fr_singe_sasi_rcprofile.dlx
|
|
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s
|
|
mkdir -p tmp/snap_decode && cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 900 mame x68000 \
|
|
-bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
|
|
-autoboot_script ../tools/bench/decode.lua \
|
|
-snapshot_directory ./snap_decode -snapview native -seconds_to_run 150
|
|
cd .. && python3 tools/bench/verify_decode.py tmp/rc_fr_singe_sasi_rcprofile.dlx
|
|
```
|
|
~90 s wall. Prints cycles/frame and % of a 12fps budget for four real frames
|
|
spanning the non-SKIP distribution, four synthetic single-mode frames, and one
|
|
full 120-frame pass; then verifies the last frame is pixel-exact. Expected:
|
|
median 73.8%, p90 116.4%, max 135.8%, mean 81.7%; V1 299.9 / V4 448.2 / RAW
|
|
400.4 cycles per block.
|
|
|
|
`-ramsize 2M` matters — MAME defaults to 4M and the locked target is a stock 2MB
|
|
machine. `DLX_VERIFY_ONLY=1` drops the timing anchors, which is how `check.sh`
|
|
runs it.
|
|
|
|
Score a container against the measured costs without touching MAME:
|
|
```
|
|
python3 tools/analysis/11_cpu_budget.py tmp/rc_fr_singe_scsi_rcprofile.dlx
|
|
```
|
|
And re-demonstrate why there is only one display path (exits non-zero **by
|
|
design** — it is the counterexample):
|
|
```
|
|
python3 tools/analysis/10_pathmix_drift.py # 70/120 frames corrupt
|
|
python3 tools/analysis/10_pathmix_drift.py --fix direct # clean, and cheapest
|
|
```
|
|
|
|
## Reproducing the rate-control result (session 6)
|
|
|
|
```
|
|
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
|
|
for prof in sasi scsi; do
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof --fixed-lam
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof
|
|
done
|
|
python3 tools/analysis/09_ratectl_drift.py # must exit 0, zero drifting frames
|
|
```
|
|
Expected, totals including the 7.8 KB/s audio allowance: `sasi` 137.4 -> 109.5
|
|
KB/s and 27.82 -> 27.22 dB; `scsi` 381.6 -> 280.0 KB/s and 30.81 -> 29.90 dB;
|
|
zero frames at the lam=800 cliff in either. ~55 s per encode, nearly all of it
|
|
k-means in `H.build`.
|
|
|
|
The block-mode map now renders the rate-controlled encoder by default:
|
|
```
|
|
python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes_rc.webm \
|
|
--profile sasi --scale 2 # add --fixed-lam to compare
|
|
```
|
|
|
|
**Do not judge rate control on `tmp/fr_00020`.** It is 14 frames; the leaky
|
|
bucket's startup transient is bucket/nframes, so it lands 18% under target there
|
|
for reasons that have nothing to do with the content. FINDINGS 27.5.
|
|
|
|
## Reproducing the sustained-action result (session 5)
|
|
|
|
```
|
|
python3 tools/analysis/07_motion_survey.py 00223 10 # -> hottest window t=539.4s
|
|
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_sasi.dlx --profile sasi
|
|
python3 tools/encoder/encode.py tmp/fr_singe tmp/singe_scsi.dlx --profile scsi
|
|
python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes.webm \
|
|
--profile sasi --scale 2
|
|
```
|
|
`extract.py` now takes optional `[start_s] [dur_s]` — needed because 00223 is
|
|
9.4 min and the windows that stress the codec are seconds long.
|
|
|
|
`08_mode_map.py` renders palettised source | decoded | block-mode map at 12fps.
|
|
Output format follows the extension; **prefer `.webm`** — GIF re-quantises to
|
|
256 colours, which is a poor fit for output whose subject is colour fidelity,
|
|
and runs larger. It uses `yuv444p` because the mode map is flat saturated colour
|
|
on a 4-pixel grid and chroma subsampling smears exactly those edges.
|