Rate control: rebuilt per-frame, wired in, and gated at zero drift

FINDINGS 26 stopped the session-5 rate controller before it shipped: it built a
lam-ladder of independent whole-sequence encodes and picked frames off it, so
SKIP blocks referenced reconstructions the decoder never saw -- 111 of 120
frames drifted. The fix is the structural one 26.1 said it had to be.

vq_hybrid is now frame-drivable -- frame_ctx / decide / paint -- and encode() is
a thin loop over it. Rate control drives the same three calls, bisects lam per
frame under the leaky bucket, and feeds back the frame it actually emitted. The
desync has no way to occur, and 09_ratectl_drift.py goes 111/120 -> 0/120. That
test is now part of check.sh, which is ~2 min rather than ~40 s.

Both overshoots on the worst sustained window are closed for under 1 dB, totals
including audio: sasi 137.4 -> 109.5 KB/s (-0.60 dB), scsi 381.6 -> 280.0 KB/s
(-0.91 dB). Zero frames hit the lam=800 cliff, so nothing was destroyed to get
there. Rate control also makes the display path cheaper -- scsi's median drops
53.6% -> 47.1% -- because raising lam moves blocks to SKIP and V1.

Two knobs measured rather than guessed. --rc-floor is worth 0.00 dB on that
window and defaults to the profile lam, so rate control cannot regress content
that already fits. --prefill defaults to 0 and is documented as a trap: it buys
a permission to overshoot of exactly bucket/nframes, and on a 14-frame clip it
disables the controller outright.

FINDINGS 26.5 was wrong in both halves and 27.6 records it. _paint was not the
bottleneck (14% of a frame, though vectorising it was still right at 17.1x) and
the ladder was never "minutes" -- those were k-means in build(). What makes
per-frame rate control affordable is that VQ.assign depends on neither lam nor
prev, so it is cached one frame deep: a 12-step search over 120 frames costs
0.31 s against 49.1 s.

Also caught: fixed-lam sasi was already 5% over target on 00020, the clip
everyone called easy. Nothing noticed because the profile table quotes PSNR and
not bitrate.

check.sh: ALL GREEN.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6

tools/analysis/08_mode_map.py

@@ -15,8 +15,10 @@ the mode headers would exploit.
   V4    four 2x2 codewords, 4 bytes
   RAW   16 literal palette indices -- the escape that makes lam=0 pixel-exact
 
-Usage: python3 tools/analysis/08_mode_map.py <frames_dir> <out.webm> [--profile p]
-                                            [--scale N]
+Usage: python3 tools/analysis/08_mode_map.py <frames_dir> <out.webm>
+           [--profile sasi|scsi] [--scale N] [--lossless] [--fixed-lam]
+
+--fixed-lam renders the pre-session-6 encoder (no rate control) instead.
 Output format follows the extension. Prefer .webm: GIF re-quantises to 256
 colours, which is a poor fit for output whose subject is colour fidelity.
 """
@@ -45,7 +47,14 @@ def main():
     prof = RC.PROFILES[sys.argv[sys.argv.index("--profile")+1]
                        if "--profile" in sys.argv else "sasi"]
     m   = H.build(src, k1=prof["k1"], k4=prof["k4"])
-    enc = H.encode(m, lam=prof["lam"])
+    # Rate-controlled by default, so the map shows the mode decisions that
+    # actually ship. --fixed-lam renders the pre-session-6 encoder instead;
+    # the difference is visible as V4/RAW collapsing to V1/SKIP on peak frames.
+    if "--fixed-lam" in sys.argv:
+        enc = H.encode(m, lam=prof["lam"])
+    else:
+        enc = RC.encode_rate_controlled(m, prof["kbps"],
+                                        lam_lo=prof["lam"])
     pal, H_, W_ = m["pal"], m["H"], m["W"]
     nbx, nby = W_ // 4, H_ // 4
 

tools/analysis/09_ratectl_drift.py

@@ -1,23 +1,27 @@
 #!/usr/bin/env python3
-"""REGRESSION TEST for the ratectl lam-ladder desync (FINDINGS 26).
+"""REGRESSION TEST for the ratectl lam-ladder desync (FINDINGS 26). PASSES as
+of session 6 -- keep it passing.
 
-Exits non-zero while the bug is present. After the fix it must report ZERO
-drifting frames -- that is the acceptance criterion for wiring rate control
-into encode.py.
+Exits non-zero if the encoder ever again reports a reconstruction that a
+decoder would not produce. That is the acceptance criterion for any change to
+rate control, and it is not a property a PSNR number can show you.
 
-encode_rate_controlled() runs H.encode() once per lam over the WHOLE sequence,
-then picks each frame from whichever rung fits the budget. But H.encode() is
-temporally recursive: a frame's SKIP blocks are copied from the PREVIOUS
-RECONSTRUCTION of that same rung. If frame f is taken from rung i while frame
-f-1 was emitted from rung j != i, the SKIP blocks in f reference a frame the
-decoder never saw.
+The bug it was written for: encode_rate_controlled() ran H.encode() once per lam
+over the WHOLE sequence, then picked each frame from whichever rung fit the
+budget. H.encode() is temporally recursive -- a frame's SKIP blocks are copied
+from the PREVIOUS RECONSTRUCTION of that same rung -- so when frame f came from
+rung i and frame f-1 was emitted from rung j != i, the SKIP blocks in f
+referenced a frame the decoder never saw. 111 of 120 frames drifted, worst frame
+43.4%. The fix was structural: the encoder is now frame-drivable and rate
+control feeds back the frame it actually emitted (vq_hybrid.frame_ctx/decide/
+paint), so drift is zero by construction rather than by tuning.
 
 This replays what a real decoder does -- SKIP copies the ACTUALLY EMITTED
 previous frame -- and compares it to the reconstruction ratectl recorded.
 
 Needs tmp/fr_singe (see docs/STATUS.md, reproducing the sustained-action
-result). Takes a few minutes: it runs `steps` full-sequence encodes and
-_paint is still a Python per-block loop.
+result). ~55 s, nearly all of it the k-means in H.build; the rate-controlled
+encode of 120 frames is ~2 s.
 """
 import sys, os
 sys.path.insert(0, "tools/encoder")
@@ -25,12 +29,15 @@ import numpy as np
 import vq as VQ, vq_hybrid as H, ratectl as RC
 
 m = H.build("tmp/fr_singe", k1=256, k4=256, iters=16)
-enc = RC.encode_rate_controlled(m, target_kbps=110, steps=5, verbose=True)
+# lam_lo=1.0: let quiet frames spend the whole allowance, which is the
+# harder case for this test -- it maximises how often lam moves frame to frame.
+enc = RC.encode_rate_controlled(m, target_kbps=110, lam_lo=1.0)
 
 lam = enc["lam"]
 sw = int((np.diff(lam) != 0).sum())
-print(f"\nframes={len(lam)}  distinct lam used={len(set(lam.tolist()))}  "
-      f"rung switches={sw}")
+print(f"frames={len(lam)}  distinct lam used={len(set(lam.tolist()))}  "
+      f"lam changes frame-to-frame={sw}  "
+      f"overruns={int(enc['overrun'].sum())}")
 
 pal, nbx = m["pal"], m["W"] // 4
 emitted = []