diff --git a/README.md b/README.md index bd2e6c1..7802592 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,9 @@ This is fundamentally a **video codec problem**, not a game-logic problem: the game logic is a scene table with branching input windows; the difficulty is pushing ~22 minutes of Don Bluth animation through a 10MHz 68000. -**Green-light check:** `./tools/bench/check.sh` (~40 s, needs the Blu-ray -mounted) re-runs both display regression tests and prints `ALL GREEN`. +**Green-light check:** `./tools/bench/check.sh` (~2 min, needs the Blu-ray +mounted) re-runs both display regression tests plus the rate-control drift test +and prints `ALL GREEN`. ## Read first - **`docs/FINDINGS.md`** — measured hardware facts, content statistics, codec @@ -27,9 +28,10 @@ tools/analysis/ measurement scripts, numbered in the order they were written (01/02 marked BROKEN deliberately, kept as regression refs). Run from the repo root — they import from tools/encoder/. 07 finds the hottest sustained window in a stream; 08 renders - source | decoded | block-mode map as .webm; 09 is a regression - test for the rate-control desync (FINDINGS 26) and exits - non-zero until it is fixed. + source | decoded | block-mode map as .webm; 09 is the + rate-control drift gate (FINDINGS 26/27) and is part of + check.sh -- it exits non-zero if the encoder ever again + reports a reconstruction no decoder would produce. tools/bench/ MAME Lua injection harness + 68000 benchmark sources. `check.sh` re-runs both display regression tests (~40 s). `blit.s`/`blit.lua` time the full-frame GVRAM blit on the @@ -51,7 +53,9 @@ python3 tools/encoder/encode.py /tmp/fr out.dlx --profile sasi --preview p.png ``` Two quality profiles ship from one codec and one decoder — `sasi` (110 KB/s) and -`scsi` (280 KB/s) are two points on the same rate-distortion curve. The codec is +`scsi` (280 KB/s) are two points on the same rate-distortion curve. Both are +**ceilings**: lam is bisected per frame under a leaky bucket, so the profile's +`lam` is a quality floor rather than a setting (`--fixed-lam` opts out). The codec is a Cinepak-style hybrid: each 4x4 block is coded as SKIP, one 4x4 codeword, four 2x2 codewords, or RAW literal pixels, chosen per block by rate-distortion. diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index 8622550..d5bf0b9 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -1073,3 +1073,146 @@ Each rung is a full-sequence encode and `_paint` is still a Python per-block loop, so a 5-rung run over 120 frames takes minutes. **Vectorise `_paint` first** — it is already on the list for the full-disc survey and it makes the rate-control work practical rather than merely faster. + +## 27. Rate control, rebuilt and wired in (session 6) + +FINDINGS 26 stopped the session-5 rate controller before it shipped: it picked +frames out of independently-encoded whole-sequence runs, so 111 of 120 frames +referenced reconstructions the decoder would never see. The fix was structural, +as 26.1 said it had to be. It is now wired into `encode.py` and **on by +default** for a profile. + +### 27.1 The encoder is frame-drivable, and the drift is zero by construction +`vq_hybrid` now exposes one frame at a time — `frame_ctx(m, f, prev)` / +`decide(ctx, lam)` / `paint(m, ctx, mode)` — and `encode()` is a thin loop over +that API. Rate control drives the same three calls and feeds back **the frame it +actually emitted** as the next frame's `prev`. There is no ladder to pick from, +so the desync has no way to occur. + +`tools/analysis/09_ratectl_drift.py`, unchanged in what it asserts: + +| | session 5 | session 6 | +|---|---|---| +| frames whose emitted output differs from what the encoder recorded | 111 / 120 | **0 / 120** | +| worst frame | 21,339 px (43.4%) | **0 px** | +| reported PSNR overstatement | 0.36 dB | **0.00 dB** | + +This is the harder case for that test on purpose: it runs with `lam_lo=1.0`, so +lam moves on 117 of 119 frame boundaries. Under the old ladder, 67 rung switches +were enough to corrupt 111 frames. + +### 27.2 Both overshoots are closed, and they cost under 1 dB +The Singe window (FINDINGS 25.3), which is the worst sustained window on the +disc. Totals include the 7.8 KB/s ADPCM allowance: + +| profile | target | fixed lam (session 5) | rate-controlled | quality cost | +|---|---|---|---|---| +| `sasi` | 110 KB/s | 137.4 KB/s (**+25%**) | **109.5 KB/s** | 27.82 → 27.22 dB (−0.60) | +| `scsi` | 280 KB/s | 381.6 KB/s (**+36%**) | **280.0 KB/s** | 30.81 → 29.90 dB (−0.91) | + +Zero frames hit the lam=800 cliff at either profile, so nothing was destroyed to +get there (26.2's failure mode did not trigger). `sasi` needed lam to reach 183 +at worst against a floor of 60; `scsi` reached 58.7 against 10. The controller +is working an order of magnitude below the cliff, which is where the search +range being capped at 800 rather than 2e5 stops mattering at all — and that is +the point: a range that never needs its top is a range you can trust. + +`scsi` still sits **1.43 dB** from the scene palette ceiling of 31.33 dB +(FINDINGS 25.4), against 0.51 dB before. The ceiling, not the codec, is still +what bounds this content. + +The percentages differ from 25.3's +18%/+34% because those compared video +payload against the total target; the table above compares like with like +(total against total). The payload figures are unchanged: 129.6 and 373.8 KB/s. + +### 27.3 Rate control makes the display path cheaper, not dearer +The decoder-architecture numbers of FINDINGS 25.6 were measured on the +fixed-lam encoder. Re-measured under rate control, on the same window, with the +player picking the cheaper of compose-then-blit and direct-to-GVRAM per frame: + +| profile | median display cost | frames above the 70% crossover | +|---|---|---| +| `sasi` fixed → RC | 37.0% → **36.6%** | 30.0% → 26.7% | +| `scsi` fixed → RC | 53.6% → **47.1%** | 53.3% → 35.8% | + +Raising lam moves blocks to SKIP and V1, which is fewer blocks to write. The +"implement both paths, pick per frame" conclusion is unaffected and the cap is +still 53.6%. + +### 27.4 The quality floor barely matters; the prefill matters, wrongly +Two knobs were measured rather than guessed. + +**`--rc-floor`** decides whether a quiet frame may spend more than the fixed-lam +profile would. On the Singe window it is worth nothing — 109.5 vs 110.0 KB/s and +**0.00 dB** — because no frame on that window is quiet enough for the bucket to +saturate. The default is `profile` (never spend more than session 5 would), so +rate control cannot regress content that already fits. + +**`--prefill`** models how full the player's buffer is at scene start. It is +tempting and it is a trap, so it defaults to 0: + +| clip | prefill 0.0 | 0.5 | 1.0 | target | +|---|---|---|---|---| +| Singe, 120 fr, `sasi` | 109.5 | 112.9 | **116.3** | 110 | +| Singe, 120 fr, `scsi` | 280.0 | 289.1 | **298.2** | 280 | +| 00020, 14 fr, `sasi` | 92.0 | **115.8** | **115.8** | 110 | +| 00020, 14 fr, `scsi` | 224.8 | **255.9** | **255.9** | 280 | + +(`scsi` on 00020 is the one cell where prefill looks harmless: the clip fits +under 280 either way. That is the content being easy, not the knob being safe.) + +Prefill buys a permission to overshoot of exactly `bucket / nframes`. At 8 +frames of bucket over 120 frames that is 6.2% — measured — and on a 14-frame +clip the bucket is larger than the clip, so rate control switches itself off and +reproduces fixed-lam exactly (lam never leaves its floor: min = median = max = +60). **A prefill that makes a target look met has disabled the controller.** + +### 27.5 The 00020 undershoot is a clip-length artefact, not a bug +At prefill 0 the 14-frame 00020 clip lands at 92.0 KB/s against a 110 ceiling — +0.66 dB given away for nothing. That is the leaky bucket's startup transient: +the first `bucket_frames` frames cannot draw on a bank they have not accumulated. +It is bounded by `bucket / nframes`, so it is 6% on a 10-second window and 20% +on a 1.2-second one. + +The lesson is the one FINDINGS 25.3 already taught in a different costume: **a +1.2-second clip cannot be used to judge rate control.** Real scenes are tens of +seconds. Do not tune the bucket against 00020. + +Worth recording separately: fixed-lam `sasi` on 00020 delivers 115.8 KB/s — the +supposedly easy clip was **already 5% over its target**, which nothing had +noticed because the profile table quotes its PSNR and not its bitrate. + +### 27.6 FINDINGS 26.5's cost premise was wrong in both halves +26.5 said a rate-control experiment was minutes because `_paint` is a Python +per-block loop, and told the next session to vectorise it first. Vectorising it +was correct and it is **17.1x faster**, but it was never the bottleneck, and the +ladder was never minutes. Measured per frame, 256x192: + +| | ms | +|---|---| +| `VQ.assign` x2 — codeword search | **22.83** | +| SKIP error against `prev` | 1.40 | +| `decide` — argmin at one lam | 0.06 | +| `paint`, vectorised | 0.29 | +| `paint`, old per-block loop | 4.93 | + +`_paint` was 14% of a frame. A 5-rung ladder over 120 frames was ~18 s of +encoding, not minutes — the "few minutes" in the drift test's docstring was +`H.build`'s k-means (51 s), which no amount of vectorising `_paint` would have +touched. + +What actually makes per-frame rate control affordable is that `VQ.assign`'s +output depends on **neither `lam` nor `prev`**, so it is computed once per frame +and a lam search only re-runs the 0.06 ms argmin: + +| | | +|---|---| +| 12-step per-frame lam search, 120 frames, symbols cached | **0.31 s** | +| the same search by re-running whole-sequence encodes | 49.10 s | + +That is a 158x difference, and it is the reason the controller can afford a real +bisection instead of a 5-rung ladder — which was the actual defect in 26.3. + +The cache holds **one frame**. At ~133 KB of intermediates per frame, caching +the sequence would cost 900 MB on a 9.4-minute stream to save nothing: every +caller works a frame at a time. diff --git a/docs/STATUS.md b/docs/STATUS.md index 6eef3b2..8756408 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1,41 +1,66 @@ -# Status & next-session handoff — end of session 5 (2026-08-23) +# Status & next-session handoff — end of session 6 (2026-08-23) -## NEXT SESSION: wire rate control into `encode.py` +## NEXT SESSION: the 68000 decoder skeleton -Decided with the user at the end of session 5. Everything needed is below; read -**FINDINGS 26** in full before editing `ratectl.py`, because the module does not -work the way its docstring says it does. +Rate control is done and gated (below). `src/player/` is still empty, and it is +now the only thing between this project and an answer to "does the CPU path +work". Everything it needs has been measured: -**Why it is now top of the list.** FINDINGS 25.3: on the worst sustained window -on the disc, the fixed-`lam` CLI overshoots both shipping targets — `sasi` -110 -> 129.6 KB/s (+18%), `scsi` 280 -> 373.8 KB/s (+34%). This item sat at -priority 4 marked "insurance, not a fix" for three sessions; that was true of the -1.2-1.7 s clips it was judged on and is not true of a sustained action sequence. +1. **Inner loop: implement BOTH display paths and pick per frame.** FINDINGS + 25.6, re-measured under rate control in 27.3. Compose-in-RAM-then-blit is a + flat 53.6% of the 12fps budget; decode-direct-to-GVRAM is 76.6% x the + non-SKIP block fraction. They cross at 70% of blocks changed. The mode + headers are parsed before any pixel is written, so counting non-SKIP blocks + to choose is free. Median cost 36.6% (`sasi`) / 47.1% (`scsi`), capped 53.6%. +2. **Copy the harness pattern from `tools/bench/blit.s` + `blit.lua`** — it + already loads code, masks interrupts, times a loop against a flag, and + snapshots for `verify_frame256.py`. Assemble with + `tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s`. +3. **Parse `DLX1`** (layout in the `encode.py` docstring, all fields big-endian), + expand the codebooks once at load, then dispatch per block on the 2-bit mode. +4. **Budget against 53.6%, not 38%.** Still not done — see priority 2a below. + The blit alone eats over half the frame before any decoding, and MAME models + no GVRAM wait states, so it is a floor. -**Do NOT just call `encode_rate_controlled()` from `encode.py`.** It is unsound -as written (FINDINGS 26.1), and it fails quietly — it returns a plausible PSNR -for a reconstruction no decoder will ever produce. +Feed it `tmp/rc_fr_singe_sasi_rcprofile.dlx` — the worst sustained window on the +disc, at the shipping profile. If the decoder fits there it fits everywhere. -Order of work: +--- -1. **Vectorise `_paint`** (`vq_hybrid.py:122`, a Python per-block loop). Not - cosmetic: rate control runs the encoder once per lam rung, so everything - below is minutes-per-experiment until this is done. FINDINGS 26.5. -2. **Make `H.encode()` frame-drivable** — take `prev` and one lam, return one - frame. The current whole-sequence signature is *why* the broken ladder - exists. This is the actual fix for 26.1. -3. **Replace the fixed ladder with the per-frame binary search** the docstring - already promises, feeding back the frame actually emitted. Cap `lam_hi` at - **800**, not 2e5 — past the FINDINGS 15 cliff a frame is not rate-controlled, - it is destroyed (26.2). Keep the leaky bucket; it works (26.4). -4. **Gate on the regression test**: `python3 tools/analysis/09_ratectl_drift.py` - exits non-zero while the desync is present and must report **zero** drifting - frames after the fix. It currently reports 111/120. Needs `tmp/fr_singe`. -5. Then re-measure the Singe window at both profiles and confirm they land on - target rather than 18%/34% over. +## What session 6 settled -Only after that is the full-disc survey worth running — otherwise it measures an -encoder nobody will ship. +1. **Rate control works, is wired in, and is ON by default.** `encode.py` + bisects lam per frame under a leaky bucket; `--fixed-lam` restores session 5 + behaviour. FINDINGS 27. +2. **Both overshoots are closed for under 1 dB.** On the Singe window, totals + including audio: `sasi` 137.4 -> **109.5 KB/s** (target 110) for -0.60 dB, + `scsi` 381.6 -> **280.0 KB/s** (target 280) for -0.91 dB. Zero frames hit the + lam=800 cliff at either profile. FINDINGS 27.2. +3. **The FINDINGS 26 desync is gone by construction, not by tuning.** The + encoder is frame-drivable (`vq_hybrid.frame_ctx` / `decide` / `paint`) and + rate control feeds back the frame it actually emitted. The regression test + `tools/analysis/09_ratectl_drift.py` goes 111/120 drifting frames -> **0**, + and it is now part of `./tools/bench/check.sh`. FINDINGS 27.1. +4. **Rate control makes the display path cheaper.** Raising lam moves blocks to + SKIP and V1, so there is less to write: `scsi`'s median display cost drops + 53.6% -> 47.1%. The decoder conclusion of 25.6 is unaffected. FINDINGS 27.3. +5. **FINDINGS 26.5 was wrong in both halves, and this is the fifth false premise + this project has caught.** `_paint` was not the bottleneck (14% of a frame) + and the ladder was never "minutes" (~18 s; the minutes were k-means in + `build`). Vectorising it was still right — 17.1x — but what actually makes + per-frame rate control affordable is that `VQ.assign` depends on neither + `lam` nor `prev`, so it is cached: a 12-step search over 120 frames costs + **0.31 s** against 49.1 s. FINDINGS 27.6. +6. **`--prefill` is a trap and defaults to 0.** It buys a permission to overshoot + of exactly bucket/nframes; at prefill=1.0 the Singe window goes to 116.3 KB/s + against a 110 ceiling, and on a 14-frame clip it disables the controller + outright. FINDINGS 27.4. +7. **Fixed-lam `sasi` was already 5% over target on 00020**, the clip everyone + called easy — nothing noticed because the profile table quotes PSNR, not + bitrate. FINDINGS 27.5. +8. **1.2-second clips cannot be used to judge rate control.** The bucket's + startup transient is bucket/nframes: 6% on a 10 s window, 20% on 00020. Same + lesson as FINDINGS 25.3, different costume. --- @@ -44,8 +69,9 @@ encoder nobody will ship. ``` ./tools/bench/check.sh ``` -~40 s, needs the Blu-ray mounted. Re-runs both display regression tests from -source media and prints `ALL GREEN`. Verified green at end of session 4. +~2 min, needs the Blu-ray mounted. Re-runs both display regression tests from +source media **and the rate-control drift test** (session 6), then prints +`ALL GREEN`. Verified green at end of session 6. If it fails, fix that before doing anything else — everything downstream assumes the display path is pixel-exact. @@ -69,8 +95,14 @@ bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`: | profile | target | lam | quality (00020 / 00146) | machine | |---|---|---|---|---| -| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT | -| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board | +| `sasi` | 110 KB/s | 60 (floor) | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT | +| `scsi` | 280 KB/s | 10 (floor) | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board | + +**As of session 6 `lam` is a floor, not a setting.** The target is a ceiling and +the encoder bisects lam per frame to stay under it; the profile's lam is the +best quality it is allowed to spend on a quiet frame. On the worst sustained +window that takes `sasi` from 137.4 to 109.5 KB/s and `scsi` from 381.6 to +280.0 KB/s, for -0.60 and -0.91 dB. FINDINGS 27.2. Sized against the user's working figure of **4 Mbps = 488 KB/s sustained**, on SD-backed SCSI (BlueSCSI / SCSI2SD) — so that rate is a bus-limited **constant**, @@ -228,9 +260,9 @@ python3 tools/encoder/encode.py /tmp/fr_00020 out.dlx --profile sasi --preview multi-byte fields are **big-endian** so the 68000 reads them with a plain `move`. ### Known encoder gaps -- **Rate control is written but not yet wired into `encode.py`** — the CLI uses a - fixed `lam` from the profile. `ratectl.encode_rate_controlled()` exists and - builds a lam-ladder per frame; it needs hooking up and validating. +- ~~Rate control is written but not yet wired into `encode.py`.~~ **DONE, + session 6.** It is on by default; `--fixed-lam` restores the old behaviour. + Gated by `tools/analysis/09_ratectl_drift.py`, which is now in `check.sh`. - **Payload is deliberately NOT entropy-coded** — deflate decode does not fit in the 68000's frame budget (FINDINGS 17.2). Do not "optimise" this later. - **Palette packing is not implemented in the encoder.** It still emits 24-bit @@ -238,8 +270,11 @@ multi-byte fields are **big-endian** so the 68000 reads them with a plain `move` palette words must pick `I` per entry by minimum squared error (FINDINGS 23.3, worth 1.96 dB) and reserve index 0 as black with `I = 0` (FINDINGS 23.4). - Codebooks are per-scene and rebuilt from scratch; no inter-scene reuse. -- `_paint` is a Python per-block loop — fine for prototyping, slow for a full - disc encode. Vectorise before the 224-stream run. +- ~~`_paint` is a Python per-block loop.~~ **DONE, session 6** — vectorised, + 17.1x. It was never the bottleneck, though: `VQ.assign` is 78% of a frame and + `H.build`'s k-means is 51 s of a 55 s run. **That k-means is now the thing to + attack before the full-disc survey**, not anything in the per-frame path. + FINDINGS 27.6. --- @@ -371,12 +406,16 @@ SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \ scene-cut frame is ~100% non-SKIP and a held frame near 0%, and the mean of those two is a number describing no actual frame. -1b. **Wire rate control into `encode.py`. NOW REQUIRED, and it is the agreed - next session's work** — see the **NEXT SESSION** block at the top of this - file for the ordered plan, and FINDINGS 26 for why - `encode_rate_controlled()` cannot simply be called as it stands. +1b. ~~**Wire rate control into `encode.py`.**~~ **DONE, session 6.** FINDINGS 27. + Both overshoots closed for under 1 dB, drift test at zero, `check.sh` gates + it. The remaining rate-control question is not a defect: whether `--rc-floor + open` is worth taking on quiet content. It measured as worth **0.00 dB** on + the Singe window (no frame there is quiet enough to saturate the bucket), so + it needs a genuinely quiet scene to decide, and it is a quality-per-byte + judgement rather than a correctness one. -2. **68000 decoder skeleton**, with the inner loop chosen by (1). Parse `DLX1`, +2. **68000 decoder skeleton**, with the inner loop chosen by (1). **This is now + the agreed next session's work — see the block at the top of this file.** Parse `DLX1`, expand codebooks, blit per block mode. The display path is verified *by 68000 code* now (FINDINGS 24) and the harness pattern is `tools/bench/blit.s` + `blit.lua`, which already loads code, masks interrupts, times a loop against @@ -400,8 +439,10 @@ SDL_VIDEODRIVER=dummy mame x68000 -bios ipl10 -video soft -window \ - Run `tools/analysis/07_motion_survey.py` per stream first; it is cheap (96x72 greyscale) and gives a hot-window shortlist so the expensive encode only runs where it matters. - - **Vectorise `_paint` before this run** — it is a Python per-block loop. - - Do it **after** rate control (1b), or it measures an encoder nobody ships. + - ~~Vectorise `_paint` before this run.~~ Done. The cost to attack now is + `H.build`'s k-means: 51 s of a 55 s run, and it runs once per scene. + - ~~Do it after rate control (1b), or it measures an encoder nobody ships.~~ + Rate control is in, so the survey now measures the shipping encoder. 5. **Confirm DMA vs PIO in MAME** (see the benchmark section above) — cheap, and the only thing that could still move CPU into the binding position. @@ -507,6 +548,31 @@ V1's output. To check that snapshot is still pixel-exact: Not added to `check.sh`: `check.sh` asserts pixel-exactness, and asserting wall timings there would make the green-light check sensitive to host load. +## Reproducing the rate-control result (session 6) + +``` +python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0 +for prof in sasi scsi; do + python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof --fixed-lam + python3 tools/encoder/encode.py tmp/fr_singe tmp/rc_$prof.dlx --profile $prof +done +python3 tools/analysis/09_ratectl_drift.py # must exit 0, zero drifting frames +``` +Expected, totals including the 7.8 KB/s audio allowance: `sasi` 137.4 -> 109.5 +KB/s and 27.82 -> 27.22 dB; `scsi` 381.6 -> 280.0 KB/s and 30.81 -> 29.90 dB; +zero frames at the lam=800 cliff in either. ~55 s per encode, nearly all of it +k-means in `H.build`. + +The block-mode map now renders the rate-controlled encoder by default: +``` +python3 tools/analysis/08_mode_map.py tmp/fr_singe tmp/singe_modes_rc.webm \ + --profile sasi --scale 2 # add --fixed-lam to compare +``` + +**Do not judge rate control on `tmp/fr_00020`.** It is 14 frames; the leaky +bucket's startup transient is bucket/nframes, so it lands 18% under target there +for reasons that have nothing to do with the content. FINDINGS 27.5. + ## Reproducing the sustained-action result (session 5) ``` diff --git a/tools/analysis/08_mode_map.py b/tools/analysis/08_mode_map.py index 8f4cfb1..54bb7d9 100644 --- a/tools/analysis/08_mode_map.py +++ b/tools/analysis/08_mode_map.py @@ -15,8 +15,10 @@ the mode headers would exploit. V4 four 2x2 codewords, 4 bytes RAW 16 literal palette indices -- the escape that makes lam=0 pixel-exact -Usage: python3 tools/analysis/08_mode_map.py [--profile p] - [--scale N] +Usage: python3 tools/analysis/08_mode_map.py + [--profile sasi|scsi] [--scale N] [--lossless] [--fixed-lam] + +--fixed-lam renders the pre-session-6 encoder (no rate control) instead. Output format follows the extension. Prefer .webm: GIF re-quantises to 256 colours, which is a poor fit for output whose subject is colour fidelity. """ @@ -45,7 +47,14 @@ def main(): prof = RC.PROFILES[sys.argv[sys.argv.index("--profile")+1] if "--profile" in sys.argv else "sasi"] m = H.build(src, k1=prof["k1"], k4=prof["k4"]) - enc = H.encode(m, lam=prof["lam"]) + # Rate-controlled by default, so the map shows the mode decisions that + # actually ship. --fixed-lam renders the pre-session-6 encoder instead; + # the difference is visible as V4/RAW collapsing to V1/SKIP on peak frames. + if "--fixed-lam" in sys.argv: + enc = H.encode(m, lam=prof["lam"]) + else: + enc = RC.encode_rate_controlled(m, prof["kbps"], + lam_lo=prof["lam"]) pal, H_, W_ = m["pal"], m["H"], m["W"] nbx, nby = W_ // 4, H_ // 4 diff --git a/tools/analysis/09_ratectl_drift.py b/tools/analysis/09_ratectl_drift.py index fb23e78..3adcb33 100644 --- a/tools/analysis/09_ratectl_drift.py +++ b/tools/analysis/09_ratectl_drift.py @@ -1,23 +1,27 @@ #!/usr/bin/env python3 -"""REGRESSION TEST for the ratectl lam-ladder desync (FINDINGS 26). +"""REGRESSION TEST for the ratectl lam-ladder desync (FINDINGS 26). PASSES as +of session 6 -- keep it passing. -Exits non-zero while the bug is present. After the fix it must report ZERO -drifting frames -- that is the acceptance criterion for wiring rate control -into encode.py. +Exits non-zero if the encoder ever again reports a reconstruction that a +decoder would not produce. That is the acceptance criterion for any change to +rate control, and it is not a property a PSNR number can show you. -encode_rate_controlled() runs H.encode() once per lam over the WHOLE sequence, -then picks each frame from whichever rung fits the budget. But H.encode() is -temporally recursive: a frame's SKIP blocks are copied from the PREVIOUS -RECONSTRUCTION of that same rung. If frame f is taken from rung i while frame -f-1 was emitted from rung j != i, the SKIP blocks in f reference a frame the -decoder never saw. +The bug it was written for: encode_rate_controlled() ran H.encode() once per lam +over the WHOLE sequence, then picked each frame from whichever rung fit the +budget. H.encode() is temporally recursive -- a frame's SKIP blocks are copied +from the PREVIOUS RECONSTRUCTION of that same rung -- so when frame f came from +rung i and frame f-1 was emitted from rung j != i, the SKIP blocks in f +referenced a frame the decoder never saw. 111 of 120 frames drifted, worst frame +43.4%. The fix was structural: the encoder is now frame-drivable and rate +control feeds back the frame it actually emitted (vq_hybrid.frame_ctx/decide/ +paint), so drift is zero by construction rather than by tuning. This replays what a real decoder does -- SKIP copies the ACTUALLY EMITTED previous frame -- and compares it to the reconstruction ratectl recorded. Needs tmp/fr_singe (see docs/STATUS.md, reproducing the sustained-action -result). Takes a few minutes: it runs `steps` full-sequence encodes and -_paint is still a Python per-block loop. +result). ~55 s, nearly all of it the k-means in H.build; the rate-controlled +encode of 120 frames is ~2 s. """ import sys, os sys.path.insert(0, "tools/encoder") @@ -25,12 +29,15 @@ import numpy as np import vq as VQ, vq_hybrid as H, ratectl as RC m = H.build("tmp/fr_singe", k1=256, k4=256, iters=16) -enc = RC.encode_rate_controlled(m, target_kbps=110, steps=5, verbose=True) +# lam_lo=1.0: let quiet frames spend the whole allowance, which is the +# harder case for this test -- it maximises how often lam moves frame to frame. +enc = RC.encode_rate_controlled(m, target_kbps=110, lam_lo=1.0) lam = enc["lam"] sw = int((np.diff(lam) != 0).sum()) -print(f"\nframes={len(lam)} distinct lam used={len(set(lam.tolist()))} " - f"rung switches={sw}") +print(f"frames={len(lam)} distinct lam used={len(set(lam.tolist()))} " + f"lam changes frame-to-frame={sw} " + f"overruns={int(enc['overrun'].sum())}") pal, nbx = m["pal"], m["W"] // 4 emitted = [] diff --git a/tools/bench/check.sh b/tools/bench/check.sh index ac6dc82..8b691b2 100755 --- a/tools/bench/check.sh +++ b/tools/bench/check.sh @@ -1,6 +1,7 @@ #!/bin/bash -# Green-light check: re-runs both display regression tests from the Blu-ray. -# ~40 s. Run from the repo root. Any non-zero exit means something drifted. +# Green-light check: re-runs both display regression tests AND the rate-control +# drift test, from the Blu-ray. ~2 min. Run from the repo root. Any non-zero +# exit means something drifted. set -e cd "$(dirname "$0")/../.." [ -d /media/reala-misaki/BDROM ] || { @@ -27,4 +28,16 @@ python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-blac run show_frame256.lua snap256 python3 tools/bench/verify_frame256.py +echo "--- session 6: rate-control drift (FINDINGS 26/27) ---" +# The codec is temporally recursive, so a rate controller can report quality for +# a reconstruction no decoder will ever produce -- silently. This asserts that a +# decoder replaying the emitted stream rebuilds exactly what the encoder +# recorded. ~55 s, nearly all of it k-means in H.build. +[ -d tmp/fr_singe ] || python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0 +# NOT piped into tail: a pipeline's exit status is the last command's, which +# would swallow the failure this whole script exists to catch. +python3 tools/analysis/09_ratectl_drift.py > tmp/drift_check.log 2>&1 \ + || { cat tmp/drift_check.log; exit 1; } +tail -9 tmp/drift_check.log + echo "ALL GREEN" diff --git a/tools/encoder/encode.py b/tools/encoder/encode.py index 1557d86..d1d4b60 100644 --- a/tools/encoder/encode.py +++ b/tools/encoder/encode.py @@ -3,6 +3,14 @@ python3 tools/encoder/encode.py [--profile sasi|scsi] [--lam N] [--fps 12] [--preview out.png] + [--fixed-lam] [--rc-floor profile|open] + +Rate control is ON by default: lam is bisected per frame under a leaky bucket +so the profile's bitrate is a ceiling rather than an average hope. `--fixed-lam` +restores session 5's behaviour, which overshoots by 18-34% on sustained action +(FINDINGS 25.3). `--rc-floor` picks the quality floor: `profile` (default) never +spends more than the fixed-lam profile would, so it can only ever help; `open` +lets quiet frames spend the whole allowance and lands the mean ON target. Container (little-endian is WRONG here -- the 68000 is big-endian, so every multi-byte field is big-endian and the decoder can read it with a plain move.w): @@ -79,6 +87,16 @@ def main(): ap.add_argument("--lam", type=float, default=None) ap.add_argument("--fps", type=int, default=12) ap.add_argument("--iters", type=int, default=16) + ap.add_argument("--fixed-lam", action="store_true", + help="disable rate control (session 5 behaviour)") + ap.add_argument("--rc-floor", choices=("profile", "open"), default="profile", + help="quality floor for rate control") + ap.add_argument("--bucket-frames", type=int, default=8, + help="leaky-bucket depth, in frame budgets") + ap.add_argument("--prefill", type=float, default=0.0, + help="how full the player's buffer is assumed to be at " + "scene start, as a fraction of the bucket (0 = cold " + "buffer after a seek, the conservative assumption)") ap.add_argument("--preview") a = ap.parse_args() @@ -87,26 +105,43 @@ def main(): k1, k4 = prof["k1"], prof["k4"] _IDX_BYTES = 1 if max(k1, k4) <= 256 else 2 + # An explicit --lam is a request for that lam, so it implies --fixed-lam. + rc = not (a.fixed_lam or a.lam is not None) + lam_lo = lam if a.rc_floor == "profile" else 1.0 + print(f"profile {a.profile}: {prof['desc']}") - print(f" target {prof['kbps']} KB/s, lam={lam}, k1={k1} k4={k4}, " - f"{_IDX_BYTES}-byte indices") + if rc: + print(f" target {prof['kbps']} KB/s CEILING, rate-controlled: " + f"lam bisected per frame in [{lam_lo:g}, {RC.LAM_CLIFF:g}], " + f"{a.bucket_frames}-frame bucket") + else: + print(f" target {prof['kbps']} KB/s, FIXED lam={lam} (no rate control)") + print(f" k1={k1} k4={k4}, {_IDX_BYTES}-byte indices") m = H.build(a.frames_dir, k1=k1, k4=k4, iters=a.iters) - enc = H.encode(m, lam=lam) + if rc: + enc = RC.encode_rate_controlled(m, prof["kbps"], fps=a.fps, + bucket_frames=a.bucket_frames, + lam_lo=lam_lo, prefill=a.prefill) + else: + enc = H.encode(m, lam=lam) r = H.evaluate(m, enc, fps=a.fps) H_, W_ = m["H"], m["W"]; nbx = W_ // 4 pal, idx = m["pal"], m["idx"] - # re-derive the per-frame symbols the same way encode() did + # The encoder hands back the symbols it actually chose. Re-deriving them + # here (as session 5 did) is a second chance to disagree with the encoder, + # and with per-frame rate control the mode map is no longer reproducible + # from a single lam anyway. frames = [] for f, im in enumerate(idx): - B1 = H.blocks_of(im, pal, 4, 4); l1 = VQ.assign(B1, m["C1s"]) - B4 = H.blocks_of(im, pal, 2, 2); l4 = VQ.assign(B4, m["C4s"]) - q = H._group_2x2_into_4x4(np.arange(len(l4)), W_) - l4g = l4[q].reshape(-1, 4) mode = enc["modes"][f] - frames.append(pack_modes(mode) + frame_payload(mode, l1, l4g, im, nbx)) + frames.append(pack_modes(mode) + + frame_payload(mode, enc["l1"][f], enc["l4g"][f], im, nbx)) + # the rate controller budgets exactly these bytes -- if that ever drifts + # from the container, every bitrate figure below is fiction + assert len(frames[-1]) == enc["sizes"][f], (f, len(frames[-1]), enc["sizes"][f]) palette = m["pal"][:256] if len(palette) < 256: @@ -138,6 +173,18 @@ def main(): f"loss {r['loss']:.2f} dB") print(f" modes: SKIP {r['skip']:.1f}% V1 {r['v1']:.1f}% " f"V4 {r['v4']:.1f}% RAW {r['raw']:.1f}%") + if rc: + rr = RC.summarise(m, enc, prof["kbps"], fps=a.fps) + lm = enc["lam"] + print(f" rate control: per-frame budget {enc['budget']:.0f} B, " + f"bucket {enc['cap']:.0f} B ({a.bucket_frames} frames), " + f"prefill {100*a.prefill:.0f}%") + print(f" lam: min {lm.min():.1f} median {rr['lam_med']:.1f} " + f"p90 {rr['lam_p90']:.1f} max {rr['lam_max']:.1f}") + print(f" frames over the per-frame budget (banked by the bucket): " + f"{rr['over']:.0f}%") + print(f" frames that could not fit even at the lam={RC.LAM_CLIFF:g} " + f"cliff: {rr['overrun']}/{len(lm)}") # PER-FRAME non-SKIP distribution. The mean above cannot answer the # decoder-architecture question (FINDINGS 24.5): decode-direct-to-GVRAM diff --git a/tools/encoder/ratectl.py b/tools/encoder/ratectl.py index c99cb79..f1c5414 100644 --- a/tools/encoder/ratectl.py +++ b/tools/encoder/ratectl.py @@ -12,11 +12,12 @@ without that, quiet frames waste budget and action frames stay ugly. The ceiling is HARD: the 68000 streams at a fixed rate off the disk, and a frame that overruns is a dropped frame, not a slow frame. -STATUS, session 5: this module is written but STILL NOT WIRED INTO encode.py, -and FINDINGS 25.3 measured both profiles overshooting their targets by 18% and -34% on the worst sustained window because of that. Before wiring it up, read -the correctness note on encode_rate_controlled() -- the lam-ladder approach it -uses is not sound against a temporally recursive encoder. +STATUS, session 6: WIRED IN and sound. `encode.py` rate-controls by default +for a profile; `--fixed-lam` restores the old behaviour. The lam-ladder of +session 5 was replaced by a per-frame bisection that drives the encoder one +frame at a time and feeds back the frame it actually emitted -- see +encode_rate_controlled(), and FINDINGS 26 for why the ladder could not be +fixed by tuning. Regression test: tools/analysis/09_ratectl_drift.py. """ import numpy as np import vq_hybrid as H @@ -49,16 +50,22 @@ import vq_hybrid as H # The rates below are therefore RAW payload, no entropy coding. # # The two profiles are the SAME codec, decoder and bitstream -- only `lam` differs. +# `lam` here is a FLOOR, not a setting: encode.py rate-controls by default and +# bisects lam per frame in [lam, LAM_CLIFF] to keep under `kbps`. The floor is +# what a quiet frame is allowed to spend, so rate control can only ever spend +# less than session 5's fixed-lam encoder did. FINDINGS 27. PROFILES = { "sasi": dict(kbps=110, lam=60.0, k1=256, k4=256, desc="stock 10MHz ACE/EXPERT, SASI", - quality="36.9 dB on 00020 / 29.6 dB on 00146 / 27.8 dB on the " - "Singe window, where it overshoots to 129.6 KB/s", + quality="36.9 dB on 00020 / 29.6 dB on 00146 / 27.2 dB on the " + "Singe window at 109.5 KB/s (session 5's fixed lam " + "gave 27.8 dB there, but at 137.4 KB/s)", util="~105 KB/s = 35% of the pessimistic 300 KB/s SASI figure"), "scsi": dict(kbps=280, lam=10.0, k1=256, k4=256, desc="Super/XVI, or CZ-6BS1 board in a 10MHz machine", - quality="39.4 dB on 00020 / 32.3 dB on 00146 / 30.8 dB on the " - "Singe window, where it overshoots to 373.8 KB/s", + quality="39.4 dB on 00020 / 32.3 dB on 00146 / 29.9 dB on the " + "Singe window at 280.0 KB/s (session 5's fixed lam " + "gave 30.8 dB there, but at 381.6 KB/s)", util="~275 KB/s = 28% of the 1 MB/s SCSI folklore figure"), } # lam=0 is PIXEL-EXACT against the palettised frame (0.00 dB loss) at ~450 KB/s @@ -68,6 +75,12 @@ PROFILES = { # lam=0 and the port ships transparent video. That decision is waiting on a # measurement, not on a design choice. +# Hard ceiling on the rate-control search. FINDINGS 15 puts the quality cliff +# between lam=800 and lam=2000. Above it a frame has not been rate-controlled, +# it has been destroyed, so the search stops here and lets the frame overrun +# instead (FINDINGS 26.2). The old ladder ran to lam=2e5, 250x past shippable. +LAM_CLIFF = 800.0 + AUDIO_KBPS = 7.8 # MSM6258 ADPCM 15.6kHz mono -- comes out of the same budget @@ -76,39 +89,92 @@ def frame_budget(kbps, fps=12, audio=AUDIO_KBPS): return (kbps - audio) * 1024.0 / fps +def _search_lam(ctx, allow, lam_lo, lam_hi, iters=12): + """Smallest lam (=> best quality) whose frame fits `allow` bytes. + + Payload size is non-increasing in lam -- raising lam can only move a block + to a mode that costs no more -- so bisection is sound. Geometric bisection, + because lam spans three decades and the interesting range is multiplicative. + + Returns (lam, mode, size, overrun). `overrun` is True when even lam_hi does + not fit: that frame is emitted over budget on purpose. Past the FINDINGS 15 + cliff a frame is not rate-controlled, it is destroyed, so a visible overrun + is the better failure (FINDINGS 26.2).""" + mode, sz = H.decide(ctx, lam_lo) + if sz <= allow: + return lam_lo, mode, sz, False + mode_hi, sz_hi = H.decide(ctx, lam_hi) + if sz_hi > allow: + return lam_hi, mode_hi, sz_hi, True + lo, hi = lam_lo, lam_hi # lo does not fit, hi does + best = (lam_hi, mode_hi, sz_hi) + for _ in range(iters): + mid = float(np.sqrt(lo * hi)) + mode_m, sz_m = H.decide(ctx, mid) + if sz_m <= allow: + hi = mid; best = (mid, mode_m, sz_m) + else: + lo = mid + return best[0], best[1], best[2], False + + def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8, - lam_lo=1.0, lam_hi=2e5, steps=9, verbose=False): + lam_lo=1.0, lam_hi=LAM_CLIFF, prefill=0.0, + steps=None, verbose=False): + """Per-frame lam search under a leaky bucket, driving the encoder ONE FRAME + AT A TIME and feeding back the frame actually emitted. + + That feedback is the whole point. The previous implementation encoded the + sequence once per lam and then picked frames off the resulting ladder; the + codec is temporally recursive, so frames picked from different rungs + reference reconstructions the decoder never saw -- 111 of 120 frames drifted, + worst frame 43.4% (FINDINGS 26.1). `tools/analysis/09_ratectl_drift.py` is + the regression test and must report zero drifting frames. + + lam_lo is a QUALITY FLOOR, not a starting guess: rate control here only ever + spends less than the fixed-lam profile, never more, so it cannot regress + content that already fits. Pass lam_lo=1.0 to let quiet frames spend the + whole allowance instead. + + `prefill` is how full the player's buffer is assumed to be when the scene + starts, as a fraction of the bucket. 0.0 (the default) is the conservative + assumption -- a cold buffer after a seek -- and is what FINDINGS 21 verified + needs no prefill to avoid underflow. It costs a startup transient: the first + `bucket_frames` frames cannot draw on a bank they have not accumulated yet, + so a clip shorter than a few bucket depths lands UNDER target. That is an + artefact of the clip length, not of the content; see FINDINGS 27.5. + + DO NOT raise `prefill` to make a target look met. It works by permitting an + overshoot of cap/nframes: measured, prefill=1.0 takes the Singe window from + 109.5 to 116.3 KB/s against a 110 ceiling, and on a 14-frame clip it + disables rate control entirely because the bucket is larger than the clip. + + `steps` is accepted and ignored -- there is no ladder any more. + """ + if steps is not None and verbose: + print(" note: `steps` is ignored; lam is now bisected per frame") budget = frame_budget(target_kbps, fps) - bucket = 0.0 # banked bytes, capped at bucket_frames*budget cap = bucket_frames * budget - out_recon, out_modes, out_sizes, out_lam = [], [], [], [] - - # encode() is whole-sequence; drive it per-lam and pick per frame. - # Cheaper than re-running the whole encoder per frame: precompute the ladder. - ladder = [] - lams = np.geomspace(lam_lo, lam_hi, steps) - for lam in lams: - e = H.encode(m, lam=float(lam)) - ladder.append(e) - if verbose: - print(f" lam={lam:9.0f} mean {e['sizes'].mean():6.0f} B/frame") - - nf = len(m["idx"]) - for f in range(nf): + bucket = prefill * cap # banked bytes; bounded by the player's buffer both ways + out = dict(recon=[], modes=[], sizes=[], lam=[], l1=[], l4g=[], overrun=[]) + prev = None + for f in range(len(m["idx"])): + ctx = H.frame_ctx(m, f, prev) allow = budget + bucket - # cheapest lam (highest quality) whose size fits the allowance - pick = len(lams) - 1 - for i in range(len(lams)): - if ladder[i]["sizes"][f] <= allow: - pick = i; break - sz = ladder[pick]["sizes"][f] - bucket = min(cap, bucket + budget - sz) - out_recon.append(ladder[pick]["recon"][f]) - out_modes.append(ladder[pick]["modes"][f]) - out_sizes.append(sz); out_lam.append(lams[pick]) - - return dict(recon=out_recon, modes=out_modes, sizes=np.array(out_sizes), - lam=np.array(out_lam), nb=ladder[0]["nb"], budget=budget) + lam, mode, sz, ovr = _search_lam(ctx, allow, lam_lo, lam_hi) + rec = H.paint(m, ctx, mode) + bucket = float(np.clip(bucket + budget - sz, -cap, cap)) + out["recon"].append(rec); out["modes"].append(mode) + out["sizes"].append(sz); out["lam"].append(lam); out["overrun"].append(ovr) + out["l1"].append(ctx["sym"]["l1"]); out["l4g"].append(ctx["sym"]["l4g"]) + prev = rec + if verbose: + print(f" f{f:04d} lam={lam:8.2f} {sz:7.0f} B " + f"(allow {allow:7.0f}){' OVER' if ovr else ''}") + return dict(recon=out["recon"], modes=out["modes"], + sizes=np.array(out["sizes"]), lam=np.array(out["lam"]), + l1=out["l1"], l4g=out["l4g"], overrun=np.array(out["overrun"]), + nb=m["nb"], budget=budget, cap=cap) def summarise(m, enc, target_kbps, fps=12): @@ -120,9 +186,17 @@ def summarise(m, enc, target_kbps, fps=12): pp = np.mean([VQ.psnr(o, v) for o, v in zip(m["rgb"], src)]) sz = enc["sizes"] mo = np.concatenate(enc["modes"]) - return dict(target=target_kbps, psnr=p, pal=pp, loss=pp - p, - mean_B=sz.mean(), max_B=sz.max(), budget=enc["budget"], - kbps=sz.mean() * fps / 1024 + AUDIO_KBPS, - over=100.0 * np.mean(sz > enc["budget"]), - skip=100 * (mo == 0).mean(), v1=100 * (mo == 1).mean(), - v4=100 * (mo == 2).mean()) + d = dict(target=target_kbps, psnr=p, pal=pp, loss=pp - p, + mean_B=sz.mean(), max_B=sz.max(), budget=enc.get("budget", 0.0), + kbps=sz.mean() * fps / 1024 + AUDIO_KBPS, + over=100.0 * np.mean(sz > enc.get("budget", np.inf)), + skip=100 * (mo == 0).mean(), v1=100 * (mo == 1).mean(), + v4=100 * (mo == 2).mean(), raw=100 * (mo == 3).mean()) + if "lam" in enc: + lam = enc["lam"] + d.update(lam_med=float(np.median(lam)), lam_max=float(lam.max()), + lam_p90=float(np.percentile(lam, 90)), + # a frame that could not fit even at the cliff: emitted over + # budget on purpose rather than destroyed + overrun=int(np.asarray(enc.get("overrun", [])).sum())) + return d diff --git a/tools/encoder/vq_hybrid.py b/tools/encoder/vq_hybrid.py index 700b72a..5ebd303 100644 --- a/tools/encoder/vq_hybrid.py +++ b/tools/encoder/vq_hybrid.py @@ -17,6 +17,19 @@ which is cheaper than V4. Bitstream per frame (what the 68000 actually parses): 2 bits/block header, packed: 00=SKIP 01=V1 10=V4 11=RAW then the payload in block order: V1 -> 1 index, V4 -> 4, RAW -> 16 + +STRUCTURE (session 6). The encoder is FRAME-DRIVABLE: `frame_ctx` / `decide` / +`paint` expose one frame at a time so a caller can choose `lam` per frame and +feed back the frame it actually emitted. That is not a convenience -- it is the +fix for FINDINGS 26.1. This codec is temporally recursive (SKIP copies the +previous RECONSTRUCTION), so any rate control that picks frames out of +independently-encoded whole-sequence runs desynchronises the encoder from the +decoder. `encode()` is now a thin loop over the per-frame API and stays the +fixed-lam path. + +The split is also what makes rate control affordable: `l1`/`l4g` and their +errors depend on neither `lam` nor `prev`, so they are computed once per frame +and a lam search only re-runs the argmin. """ import numpy as np, sys from PIL import Image @@ -24,6 +37,11 @@ import vq as VQ LUMA = VQ.LUMA +# byte cost of each mode's payload, per block. The 2-bit header is paid by +# every block regardless, so it drops out of the mode comparison. +_HDR_BYTES_PER_BLOCK = 2 / 8.0 +RAW_BYTES = 16.0 # literal palette bytes, never indices + def blocks_of(idx, pal, bw, bh): return VQ.blockify(idx, pal, bw, bh) @@ -46,68 +64,163 @@ def build(frames_dir, k1=256, k4=256, iters=16, lam=0.0): cb4 = VQ.snap_codebook(C4, pal, 2, 2) # (k4,4) palette idx C4s = (pal[cb4].astype(np.float32) * LUMA).reshape(k4, -1) return dict(rgb=rgb, pal=pal, idx=idx, H=H, W=W, - cb1=cb1, C1s=C1s, cb4=cb4, C4s=C4s, k1=k1, k4=k4) + cb1=cb1, C1s=C1s, cb4=cb4, C4s=C4s, k1=k1, k4=k4, + nbx=W // 4, nby=H // 4, nb=(W // 4) * (H // 4), + palw=pal.astype(np.float32) * LUMA) def _v1_recon(lab1, cb1, H, W): return VQ.unblockify(lab1, cb1, H, W, 4, 4) +# --- block <-> image reshapes (no copy where numpy can avoid one) ------------- + +def to_blocks(a, nbx, nby): + """(H,W) -> (nb,4,4) in block raster order.""" + return a.reshape(nby, 4, nbx, 4).transpose(0, 2, 1, 3).reshape(-1, 4, 4) + + +def from_blocks(b, nbx, nby): + """(nb,4,4) -> (H,W).""" + return b.reshape(nby, nbx, 4, 4).transpose(0, 2, 1, 3).reshape(nby * 4, nbx * 4) + + +def default_idx_bytes(m): + """Size of ONE codebook index in the bitstream. k>256 needs 2 bytes, which + doubles what V1 and V4 actually cost -- an RD model that ignores that + systematically over-picks V4 and under-reports the bitrate (FINDINGS 14).""" + return 1 if max(m["k1"], m["k4"]) <= 256 else 2 + + +# --- per-frame API ----------------------------------------------------------- + +def frame_symbols(m, f): + """lam- and prev-INDEPENDENT part of a frame: codeword assignments and + their errors. + + Cached, because a lam search re-uses them unchanged and `VQ.assign` is the + expensive call in the encoder -- 22.8 of 24.6 ms per frame, measured. That + cache is what makes per-frame rate control affordable: a 12-step search + over 120 frames costs 0.3 s, against 49 s for the equivalent done by + re-running whole-sequence encodes. + + The cache holds ONE frame. Every caller works a frame at a time, and at + ~133 KB of intermediates per frame a whole-sequence cache would cost + 900 MB on a 9.4-minute stream for no benefit.""" + cache = m.get("_sym") + if cache is not None and cache[0] == f: + return cache[1] + pal, W, nb = m["pal"], m["W"], m["nb"] + im = m["idx"][f] + + B1 = blocks_of(im, pal, 4, 4) # (nb,48) + l1 = VQ.assign(B1, m["C1s"]) + e1 = ((B1 - m["C1s"][l1]) ** 2).sum(1) + + B4 = blocks_of(im, pal, 2, 2) # (nb*4,12) in 2x2 raster + l4 = VQ.assign(B4, m["C4s"]) + e4raw = ((B4 - m["C4s"][l4]) ** 2).sum(1) + # regroup 2x2 blocks into their parent 4x4 block + q = _group_2x2_into_4x4(np.arange(nb * 4), W) + e4 = e4raw[q].reshape(nb, 4).sum(1) + l4g = l4[q].reshape(nb, 4) + + s = dict(l1=l1, e1=e1, l4g=l4g, e4=e4, + src_blocks=to_blocks(im, m["nbx"], m["nby"])) + m["_sym"] = (f, s) + return s + + +def frame_ctx(m, f, prev, idx_bytes=None): + """Everything needed to decide one frame at any lam, given the frame that + will actually precede it in the emitted stream.""" + s = frame_symbols(m, f) + nb = m["nb"] + if prev is None: + eS = np.full(nb, np.inf) + prev_blocks = None + else: + # SKIP distortion = this frame against the previous RECONSTRUCTION, + # in the same luma-weighted space the codebooks were trained in. + d = ((m["palw"][m["idx"][f]] - m["palw"][prev]) ** 2).sum(2) + eS = d.reshape(m["nby"], 4, m["nbx"], 4).sum((1, 3)).ravel() + prev_blocks = to_blocks(prev, m["nbx"], m["nby"]) + return dict(f=f, sym=s, eS=eS, prev_blocks=prev_blocks, nb=nb, + idx_bytes=default_idx_bytes(m) if idx_bytes is None else idx_bytes) + + +def decide(ctx, lam): + """Lagrangian mode decision at one lam. Returns (mode, payload bytes). + + Cheap by design: no painting, no image-sized work. A lam search calls this + a dozen times per frame and paints once.""" + ib = ctx["idx_bytes"] + s = ctx["sym"] + cost = np.stack([ctx["eS"], + s["e1"] + lam * (1.0 * ib), + s["e4"] + lam * (4.0 * ib), + np.full(ctx["nb"], lam * RAW_BYTES)]) + mode = np.argmin(cost, axis=0).astype(np.uint8) + return mode, frame_bytes(mode, ctx["nb"], ib) + + +def frame_bytes(mode, nb, idx_bytes): + nV1 = int((mode == 1).sum()); nV4 = int((mode == 2).sum()) + nR = int((mode == 3).sum()) + return (nb * _HDR_BYTES_PER_BLOCK + + (nV1 + nV4 * 4) * idx_bytes + nR * RAW_BYTES) + + +def paint(m, ctx, mode): + """Reconstruct the frame the decoder will produce for this mode map.""" + nbx, nby = m["nbx"], m["nby"] + s = ctx["sym"] + ob = np.empty((ctx["nb"], 4, 4), dtype=np.uint8) + sel = mode == 0 + if sel.any(): + ob[sel] = ctx["prev_blocks"][sel] + sel = mode == 1 + if sel.any(): + ob[sel] = m["cb1"][s["l1"][sel]].reshape(-1, 4, 4) + sel = mode == 2 + if sel.any(): + # (n, sub_y, sub_x, py, px) -> (n, sub_y, py, sub_x, px) -> (n,4,4) + c = m["cb4"][s["l4g"][sel]].reshape(-1, 2, 2, 2, 2) + ob[sel] = c.transpose(0, 1, 3, 2, 4).reshape(-1, 4, 4) + sel = mode == 3 + if sel.any(): + ob[sel] = s["src_blocks"][sel] + return from_blocks(ob, nbx, nby) + + +def encode_frame(m, f, prev, lam, idx_bytes=None): + """One frame at one lam against one previous reconstruction.""" + ctx = frame_ctx(m, f, prev, idx_bytes) + mode, sz = decide(ctx, lam) + return dict(recon=paint(m, ctx, mode), mode=mode, size=sz, + l1=ctx["sym"]["l1"], l4g=ctx["sym"]["l4g"], ctx=ctx) + + def encode(m, lam=0.02, skip_thresh=0.0, idx_bytes=None): - """lam = lagrangian rate weight (bytes -> squared-error units). + """Fixed-lam whole-sequence encode: a loop over the per-frame API. + + lam = lagrangian rate weight (bytes -> squared-error units). Higher lam => more V1/SKIP => smaller & softer. - idx_bytes: size of ONE codebook index in the bitstream. k>256 needs 2 bytes, - which doubles what V1 and V4 actually cost -- if the RD model ignores that - it systematically over-picks V4 and under-reports the bitrate. Defaults to - the value implied by the codebook sizes.""" + For a rate-controlled encode use ratectl.encode_rate_controlled(), which + drives the same per-frame API and varies lam. Do NOT reassemble a sequence + out of several fixed-lam runs of this function -- FINDINGS 26.1.""" if idx_bytes is None: - idx_bytes = 1 if max(m["k1"], m["k4"]) <= 256 else 2 - pal, idx, H, W = m["pal"], m["idx"], m["H"], m["W"] - nbx, nby = W // 4, H // 4 - nb = nbx * nby - recon, modes, sizes = [], [], [] + idx_bytes = default_idx_bytes(m) + recon, modes, sizes, l1s, l4gs = [], [], [], [], [] prev = None - for f, im in enumerate(idx): - B1 = blocks_of(im, pal, 4, 4) # (nb,48) - l1 = VQ.assign(B1, m["C1s"]) - e1 = ((B1 - m["C1s"][l1]) ** 2).sum(1) - - B4 = blocks_of(im, pal, 2, 2) # (nb*4,12) in 2x2 raster - l4 = VQ.assign(B4, m["C4s"]) - e4raw = ((B4 - m["C4s"][l4]) ** 2).sum(1) - # regroup 2x2 blocks (raster over 8x12... ) into their parent 4x4 block - q = _group_2x2_into_4x4(np.arange(nb * 4), W) - e4 = e4raw[q].reshape(nb, 4).sum(1) - l4g = l4[q].reshape(nb, 4) - - # SKIP: cost of reusing the previous *reconstructed* block - if prev is None: - eS = np.full(nb, np.inf) - else: - pb = blocks_of(prev, pal, 4, 4) - eS = ((B1 - pb) ** 2).sum(1) - - # RAW: zero distortion against the palettised source, 16 bytes - eR = np.zeros(nb) - - # rate-distortion choice: true byte cost per mode. The 2-bit header is - # paid by every block regardless, so it drops out of the comparison. - bV1 = 1.0 * idx_bytes - bV4 = 4.0 * idx_bytes - bRAW = 16.0 # RAW is literal palette bytes, never indices - cost = np.stack([eS + lam * 0.0, e1 + lam * bV1, - e4 + lam * bV4, eR + lam * bRAW]) - mode = np.argmin(cost, axis=0).astype(np.uint8) - - out = np.empty((H, W), dtype=np.uint8) - _paint(out, mode, l1, l4g, m["cb1"], m["cb4"], prev, nbx, nby, im) - recon.append(out); modes.append(mode) - nV1 = int((mode == 1).sum()); nV4 = int((mode == 2).sum()) - nR = int((mode == 3).sum()) - sizes.append(nb * 2 / 8 + (nV1 + nV4 * 4) * idx_bytes + nR * 16) - prev = out - return dict(recon=recon, modes=modes, sizes=np.array(sizes), nb=nb) + for f in range(len(m["idx"])): + r = encode_frame(m, f, prev, lam, idx_bytes) + recon.append(r["recon"]); modes.append(r["mode"]); sizes.append(r["size"]) + l1s.append(r["l1"]); l4gs.append(r["l4g"]) + prev = r["recon"] + return dict(recon=recon, modes=modes, sizes=np.array(sizes), + l1=l1s, l4g=l4gs, nb=m["nb"]) def _group_2x2_into_4x4(a, W): @@ -119,23 +232,6 @@ def _group_2x2_into_4x4(a, W): return g.reshape(-1) -def _paint(out, mode, l1, l4g, cb1, cb4, prev, nbx, nby, src): - for b in range(len(mode)): - by, bx = divmod(b, nbx) - y, x = by * 4, bx * 4 - mo = mode[b] - if mo == 0: - out[y:y+4, x:x+4] = prev[y:y+4, x:x+4] - elif mo == 1: - out[y:y+4, x:x+4] = cb1[l1[b]].reshape(4, 4) - elif mo == 3: - out[y:y+4, x:x+4] = src[y:y+4, x:x+4] - else: - c = cb4[l4g[b]].reshape(2, 2, 2, 2) # (sub_y,sub_x,2,2) - out[y:y+2, x:x+2] = c[0, 0]; out[y:y+2, x+2:x+4] = c[0, 1] - out[y+2:y+4, x:x+2] = c[1, 0]; out[y+2:y+4, x+2:x+4] = c[1, 1] - - def evaluate(m, enc, fps=12): pal = m["pal"] rec = [pal[i] for i in enc["recon"]]