Handoff: reconcile the docs with a decoder that works and does not fit

Session 7 handoff. check.sh green end to end, including both new stages.

STATUS now leads with the cost-aware mode decision, and the stale statements
that session 7 overturned are struck rather than left to be re-read as current:

- next-steps item 1 still announced "implement both display paths and pick per
  frame" as a settled answer. Withdrawn, with the reason, and the part that did
  hold up (report the distribution, not the mean) kept.
- the profile table's "machine" column reads as a CPU claim and is not one. It
  is the bus. Both profiles target the same stock 10 MHz 68000 and neither
  fits; the Super has SCSI at 10 MHz.
- the display-path section stopped at "68000 code copied a frame". It parses
  one now, and the blit figures are no longer the display-path budget.
- the green-light section still described a 2-minute, three-stage check, and
  did not warn that 10_pathmix_drift.py is SUPPOSED to exit non-zero.

Sharpened the one trap in the next session's first step: SKIP is not a
per-block constant. It costs 13.25 cycles inside an all-SKIP header byte and
~45 in a mixed one, so its price depends on its neighbours and a per-block
lagrangian cannot see that. Picking either number is wrong in a different
direction; the budget check can use the exact clustered cost function while
the per-block term only has to rank modes.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 15:21:12 -07:00
parent 31c4c1aba1
commit cb05e77a42
+57 -18
View File
@@ -26,9 +26,19 @@ worth 4 bytes, with no idea what it costs to draw.
**The work, in order:**
1. **Add a cycle term to the mode decision.** `decide()` already builds a cost
matrix of `error + lam * bytes` per mode per block; add `+ mu * cycles`, with
the cycles vector `[13, 300, 448, 400]` measured in FINDINGS 28.2. One extra
row of arithmetic in a function that is already vectorised.
matrix of `error + lam * bytes` per mode per block; add `+ mu * cycles`,
with the per-mode cycles measured in FINDINGS 28.2.
**SKIP is not a constant and this is the one trap here.** A SKIP block costs
13.25 cycles when all four blocks in its header byte are SKIP (one `tst.b`
clears the group) and ~45 when it sits in a mixed byte — so SKIP's price
depends on its *neighbours*, which a per-block lagrangian cannot see. Do not
pick one number and move on: 45 overcharges clustered SKIPs and pushes the
encoder away from the mode that saves cycles, 13.25 undercharges isolated
ones and lets frames overrun. The way out is that the **budget check does not
have to use the same cost function as the mode decision** — score frames with
the exact clustered cost (`cycles()` in `tools/analysis/11_cpu_budget.py`,
validated to 1 point against the 68000) and let the bisection converge on
that, while the per-block term uses a constant purely to *rank* modes.
2. **Then bisect `mu` per frame against the 833,333-cycle budget**, exactly as
session 6 bisects `lam` against the byte budget. The machinery is already
there and already gated: `ratectl.encode_rate_controlled` is frame-driven and
@@ -43,6 +53,7 @@ worth 4 bytes, with no idea what it costs to draw.
cliff? `tools/analysis/11_cpu_budget.py` scores a container without needing
MAME, so the search loop is cheap; confirm the winner on the 68000 with
`tools/bench/decode.lua`.
3b. **Know which misses are yours to fix before starting.** Re-coding every
non-SKIP block as V1 is the floor any mode assignment can reach, and it
still misses 11 frames at `sasi` and 12 at `scsi` — every frame above ~90%
@@ -154,12 +165,20 @@ saves ~16 of 448 cycles.
```
./tools/bench/check.sh
```
~2 min, needs the Blu-ray mounted. Re-runs both display regression tests from
source media **and the rate-control drift test** (session 6), then prints
`ALL GREEN`. Verified green at end of session 6.
~3 min, needs the Blu-ray mounted. From source media it re-runs both display
regression tests, the rate-control drift test (session 6), the display-path
coherency counterexample and a **120-frame 68000 decode** (session 7), then
prints `ALL GREEN`. Verified green at end of session 7.
If it fails, fix that before doing anything else — everything downstream assumes
the display path is pixel-exact.
The two session-7 stages are worth knowing the shape of before they fail on you:
- `10_pathmix_drift.py` is expected to exit **non-zero**; `check.sh` fails if it
ever starts passing, because that would mean the counterexample behind the
one-path decoder had stopped reproducing.
- the decode stage needs `tmp/rc_fr_singe_sasi_rcprofile.dlx` and will spend
~55 s encoding it if it is missing, nearly all of that k-means in `H.build`.
## Decisions locked
| decision | value | why |
@@ -184,6 +203,13 @@ bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
| `sasi` | 110 KB/s | 60 (floor) | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
| `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
**That "machine" column is about the BUS, not the CPU.** The profiles are an
I/O-bandwidth axis and say nothing about clock speed: the X68000 Super has
built-in SCSI at 10 MHz (`x68k.cpp:1194`, `40_MHz_XTAL/4`, same as the base
machine), and only the XVI is faster. **Both profiles target the same stock
10 MHz 68000**, so both must fit the same 833,333-cycle frame budget — and as of
session 7 neither does. FINDINGS 28.7.
**As of session 6 `lam` is a floor, not a setting.** The target is a ceiling and
the encoder bisects lam per frame to stay under it; the profile's lam is the
best quality it is allowed to spend on a quiet frame. On the worst sustained
@@ -434,10 +460,18 @@ functional models, not timing-accurate; a KB/s figure from MAME measures the
emulator's scheduler. `docs/BENCHMARK.md` covers the three-tier approach
(MAME validates the path, derivation bounds it, real hardware settles it).
## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5).
## Display path — VERIFIED (session 3), in a real mode (session 4), by 68000 code (session 5), by a 68000 DECODER (session 7).
The first real frame is on screen: `docs/images/x68k_first_frame_compare.png`.
**Session 7 went from copying a frame to parsing one.** `src/player/decode.s`
reads DLX1, dispatches all four block modes and writes straight into GVRAM;
120 frames decoded in sequence are pixel-exact against `tools/encoder/dlx.py`
(`tools/bench/verify_decode.py`, in `check.sh`). The blit numbers below are
still correct for what they measured — a *copy* — but they are no longer the
display-path budget: the decoder costs 300/448/400 cycles per V1/V4/RAW block
and misses the 12fps budget on 31% of frames. FINDINGS 28.
**Session 5 closed the gap this paragraph used to describe.** GVRAM is now
filled by 68000 instructions and the result is still pixel-exact, and the blit
cost is measured rather than estimated: **53.6% of a 12fps frame**, not 38%
@@ -483,11 +517,16 @@ SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
## Next steps, in priority order
1. ~~**Measure the non-SKIP block fraction.**~~ **DONE, session 5** — FINDINGS
25.6. Answer: implement **both** display paths and pick per frame; median
37.0% of the frame budget, capped at 53.6%. Reporting is wired into
`encode.py`. Original framing kept below because the reasoning still governs
the decoder's inner loop:
1. ~~**Measure the non-SKIP block fraction.**~~ **DONE, session 5**, and its
answer **WITHDRAWN in session 7** — FINDINGS 28.1/28.2. It concluded
"implement both display paths and pick per frame, median 37.0%, capped at
53.6%". Mixing the paths is incoherent (the compose path needs a RAM
reference the direct path never writes) and the two costs it compared were
both copies with no decode in either. **The shipping decoder has one path.**
The non-SKIP fraction is still reported by `encode.py` and is still the right
thing to look at — it is just no longer a switch. Original framing kept
below, because its instruction to report the distribution rather than the
mean is the part that held up:
FINDINGS 24.5: compose-in-RAM-then-blit costs a flat 53.6% of the frame
budget; decode-direct-to-GVRAM costs 76.6% x (fraction of blocks that are not
SKIP) and needs no RAM reference frame. **They cross at 70%.** Which side of
@@ -514,12 +553,6 @@ SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
of a 12fps frame, 31% of frames over 100%. FINDINGS 28. The follow-on is
priority 0 at the top of this file.
2b. **Pad frame records to 4 bytes in `encode.py`.** Not optional: unaligned
records are an address error on a 68000 (FINDINGS 28.3), and `prep_dlx.py`
currently repairs it at load time, which the shipping player streaming from
disc cannot do. The padding is real bytes on disc, so it has to be inside
the rate controller's accounting, not added after it. 20 B/s at 12fps.
2a. **Re-budget everything against the MEASURED per-mode costs**, not 53.6% and
not 38%. Session 7 replaced the model twice over (FINDINGS 28.2): the display
path is not one number times a block fraction, and the median frame is 74.4%
@@ -534,6 +567,12 @@ SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \
rejection, which was argued as "54% LZ4 with no room beside a 38% blit". The
conclusion gets *stronger*, not weaker, but the arithmetic should be restated.
2b. **Pad frame records to 4 bytes in `encode.py`.** Not optional: unaligned
records are an address error on a 68000 (FINDINGS 28.3), and `prep_dlx.py`
currently repairs it at load time, which the shipping player streaming from
disc cannot do. The padding is real bytes on disc, so it has to be inside
the rate controller's accounting, not added after it. 20 B/s at 12fps.
3. **Full-disc survey.** Now scoped by session 5 rather than open-ended: the
worst *sustained* window is measured (FINDINGS 25), so what remains is the
distribution over content, not the worst case.