ab15c0749aa574b59170ba1ea6bc6a8426713655
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 prosolis
|
1be428c270 |
Align the container to the disc, and find the decoder-free packed player fits
Two sessions, unrecorded until now, committed together because their edits share files and cannot be split cleanly after the fact. Session 28 (FINDINGS 60): the container is DLX5 -- every record sector-aligned, 120/120 starting on a boundary where 3/120 did, +0.48% on the wire and zero clocks -- and the ring's release rounds to RECALN so no pad is stranded. Two encoder levers measured and refused: `--spans all` buys +0.19 dB for +67% of the wire, and joint span/lam selection emits byte-identical containers because `lam` never leaves its floor on any of 120 frames. Session 29 (FINDINGS 61): the packed full-frame blit is 27.3% of a 12 fps frame, a channel fills GVRAM in buffer mode off the disc with the CPU halted, and it walks the 1,024 B line stride itself through array chaining. At the 9 clk/B dual-address floor the codec is 110.4% of a frame and a decoder-free packed literal player is 55.2%, at +4.89 dB -- 2.75 dB past a ceiling the codec's scene-wide palette cannot cross. Encoder work is parked; the codec is kept and not built on. check.sh is ALL GREEN before and after, plus one new stage that gates the ORDER of the measured paint costs rather than their values. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |
||
|
prosolis
|
2676f3b835 |
Put the ring on the 68000, and find the disc stops whenever the player is not asking
ROADMAP P5. The loader moved in session 21 and the frame clock in 22; the ring producer was the last policy living outside the machine. src/player/ring.i does `aligned` placement, the descriptor ring, a prefill, 51.2's slack rule and a seek, and the host keeps only the transport. It needed a container change. `aligned` asks whether the next record fits before the end of the ring -- a length asked BEFORE the record is fetched -- and every reader in this tree answered that by walking the frame stream, which is exactly what a player streaming off a disc cannot do. DLX4 carries nframes u16 record lengths in the scene header. Frame payloads are byte-identical to the DLX3 encode, so no fitted constant moves; the scene header goes 5,920 to 6,164 B. The producer reproduces the host's tiling exactly: 18 wraps, 14.7 KB mean hole, pixel-exact, a third independent implementation of the same policy. What it exposed is bigger than the item. A channel only moves bytes while it has a request and only the CPU can issue one, so the disc stands still between records by an amount the PLAYER sets, not the medium -- and no host-filled run could see it. At 488 KB/s in a 256 KB ring a one-deep request queue gives away 6.8% of the pipe and underruns 59 of 120 frames; two-deep gives away 3.4% and underruns none. The container's whole surplus over the wire is 8.7%, so the player's own loop was spending most of the slack a branch point saves up. Prefill is the weaker lever: six records of it still leaves 24 underruns. Three silent bugs are recorded in FINDINGS 55.7 -- all produced wrong pixels or a desync rather than a fault -- plus a rig one: MAME renders a screen line by line, so snapshotting the frame the decoder finished in captures a tear that reads exactly like a decoder bug. check.sh gains the machine-owned ring and a seek with the decode after it. decode.bin is unchanged at 1,296 B and a host-filled run executes none of the new code, so every FINDINGS 49/51 figure stands. ALL GREEN before and after. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |
||
|
prosolis
|
2f9f5cc995 |
Pace the ring, then read the DMAC config out of the IPL ROM: audio is cheap and the disk is not
Two sessions that were never separated in the working tree, so they land as one commit. check.sh ALL GREEN before and after both. SESSION 19 -- the ring rig gets a frame clock (FINDINGS 51). src/player/stream.s had no frame clock: it asked for record i the instant it finished i-1, outran any finite pipe, and never let the ring back up. The 49.1 sweep passing at 48 KB was therefore a wrap-correctness result and nothing else. PACE/PACEON ($18034/$18038) hold the decoder to 12 fps, so FR_HEAD-FR_TAIL finally means what it reads as: whole frames the decoder could still draw with delivery stopped dead. PACEON=0 free-runs and is what the wrap gate still uses, so every figure in 49 is unmoved. Paced, on the gate container: 64 KB holds 2 frames, 256 KB holds 7-8, 512 KB holds 14-15, all pixel-exact. Tolerance is ceiling-1, measured by cutting the pipe: 256 KB buys 500 ms of dead pipe, not 583. SLACK IS ACCUMULATED, NOT OWNED. It is built out of pipe-wire and a seek spends all of it. At 488 KB/s a 256 KB ring needs 4.83 s of play to reach its ceiling from empty; 512 KB needs 8.42 s to reach 14. A bigger ring raises the ceiling AND lengthens the climb, so a branch point does not ask "is the buffer big enough" but "has there been enough play since the last one" -- and Dragon's Lair's decision points are seconds apart. The rig now also says WHICH resource is binding: at 460 KB/s every ring from 192 KB to 512 KB is rate-bound at ceiling 4 and never fills, so larger rings are dead RAM in that scene. 20_seek_slack.py is the same model rewritten in Python from record sizes, sharing no code with the Lua producer: 35/35 ceilings inside its bracket. SESSION 20 -- the DMAC configuration was in the IPL ROM the whole time (FINDINGS 52). ROADMAP's "do this first" was to put the ADPCM stream on the bus. That needs a clocks-per-byte figure for the audio channel, and 11_cpu_budget.py was charging audio the DISK's rate -- 5 clk/B, its own help text calling it "single-address, bus held". Audio was being charged the favourable end of B3, a 242 KB/s open question. It never had to be a guess. The IPL ROM programs all four HD63450 channels itself and MAME boots the rig with it, so 21_iplrom_dmac.py reads the configuration out of the image and decodes the MC68450 fields. Eight (address, expected bytes, meaning) sites; a mismatch or an unknown revision exits non-zero. In check.sh, no emulator, milliseconds. ch3 DCR=$80, OCR=$32: dual address, 8-bit port, cycle steal WITHOUT hold, REQG=10 external request. The DMAC arbitrates once per byte with no burst to amortise the 5..8 + 2 over, so an audio byte is 16..19 clocks, not 5 -- the old debit was 3.2x..3.8x small. And on the bus it is still nothing: 651 B/frame is 1.25%..1.48% of a frame, about 4% of what the decoder leaves. P6's bus risk does not materialise. The unit worry was worth checking and nearly right: 15.6 kHz is 8 MHz/512 = 15,625 samples/s, two 4-bit samples to a byte = 7,812.5 B/s exactly, and AUDIO_KBPS=7.8 is that in decimal kB while the tool multiplied by 1024. THE DISK CHANNEL IS PROGRAMMED IDENTICALLY. ch1 (SASI) is DCR=$80 too, and so is ch0. That is 16..19 clocks per delivered byte, where 42.4 brackets W at 5..12 and 42.5 has W=8 already missing 47/120 frames. The only worked example of a disk DMA configuration on this machine sits above the entire bracket, and at that price nothing fits at any container size. It is not scsiexrom.bin so B3 stays open -- what changed is that a cheap configuration is now the thing that has to be SHOWN. W <= 12 is a requirement on the player's DMAC programming, not a range the hardware hands us, and it is now the largest open number in the project, ahead of the rate. An unforced cross-check fell out: 15_bus_occupancy.py's new W sweep puts W=8 at 105.7% of the frame, agreeing with 42.5's 47/120, from mode histograms and bus clocks respectively, two models sharing no code. Also: ADPCM outranks the disk at the arbiter (CPR 1 against 2), so an audio byte never waits and a video byte does -- relevant to 51's smooth-rate delivery model. README MEDIA. stream.lua gains DLX_SNAP_EVERY=1 (needs DLX_PACE, off by default, on no path check.sh takes) and tools/media/make_readme_media.py turns the PNGs into docs/img/. The stills and both clips are MAME's own screen pixels. Building it turned up something worth recording. 116 of 119 captured frames are pixel-exact against dlx.py; three are TORN -- frame n on top, frame n-1 below the tear line -- because MAME captured the screen while the block loop was partway down it. decode.s writes straight to the displayed page (one display path, 28.1), so a real player tears the same way, and this is the first time that consequence has been visible rather than argued. The script ASSERTS the tear and refuses to build otherwise, rather than trimming three frames and reporting "every frame I kept is exact". Second correction the capture forced: the snapshot fires before frame n is decoded, so the obvious reading is that it holds frame n-1 -- it does not, because MAME renders the screen at the end of the machine frame, by which time the 68000 has finished frame n. 11_cpu_budget.py's "validated to within 1 pt" line is also corrected: the model reads 2..10 pt HIGH and by more as the frame gets harder, which was already true before either session. src/player/decode.s is unchanged; decode.bin is still 1,296 B at the same MD5. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |
||
|
prosolis
|
b49bbdc939 |
Build v7 into the player, and find the cost model 18% wrong on the block it made commonest
src/player/decode.s now paints v7 literal spans, pixel-exact under MAME and px68k's C68K core over a container where every frame carries 128-216 spans covering up to 38% of the picture. The span pass is blit.s v7 verbatim: the 66.0/9.143/9.978 fit was measured on that instruction sequence. The container is DLX3 -- a span section between the mode header and the block payload, since that is the only place the 68000 can reach without first parsing something of variable length. 16_span_roundtrip.py gates it in check.sh, and asserts it emitted enough spans to have tested anything. Two synthetic all-SPAN anchors price v7 inside decode.s at 151.2 and 225.6 clocks per 4x4 block, against FINDINGS 40's table of 151 and 226 -- 0.2% on both emulators. The measured mode costs what it was said to cost. Two things that were not on the list: TWO BYTE BUDGETS. FINDINGS 40's 18/120 was scored against the 488 KB/s PIPE, not the 280 KB/s profile, and at the profile rate the lam search has already spent the allowance -- spans fired on 5 frames of 120 and looked like a regression. The profile is a chosen quality rate point; the pipe is hardware. --kbps and --span-kbps are now separate and spans run before mu, because a span pays in bytes and mu pays in picture. Delivered: 86/120 over budget without spans, 77/120 at the profile budget, 34/120 on the pipe for +0.36 dB. C_SKIP_MIXED WAS NEVER MEASURED, and it was 18% low -- 45.0, now 55.0. It is the one constant in the table that came from a derivation, because the synthetic frame that would measure it cannot exist: a byte needs a coded block for its SKIP to be mixed. Four bracketing anchors measure it on both emulators with the header byte rotated through all four positions, and the partner mode solves back to its own anchored value to 0.2%. With it corrected the model predicts a real spanned decode to -0.06% mean / 0.09% worst, against -2.99% / 4.30%. It matters because a span marks its run SKIP, so mixed SKIPs dominate exactly the frames spans are judged on. Also: the rig had been writing its synthetic timing frames 26 KB past the top of a 2 MB machine, and got away with it because the modes it overran are data-independent. A span's jump displacements come out of the stream, so it is not. And frames-over-budget is no longer a safe headline -- the controller aims at the deadline, so 55 of 120 frames sit within 5% of it and a 1% cost shift moves 22 frames. FINDINGS 41. check.sh ALL GREEN, now gating on a span-heavy DLX3 container. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |
||
|
prosolis
|
7d365b3ff5 |
Drop SASI on capacity, then find the budget never had the disk in it
USER DECISION: drop the `sasi` profile. Not on bandwidth -- on capacity. A SASI volume is 40 MB, and the 22.8 min of unique scene footage on the source Blu-ray (streams 00000-00201, measured, not recalled) is 146 MiB at the LOWEST rate this codec makes -- more than the machine's whole 4-unit SASI space. `scsi` is the only profile now. FINDINGS 32. Then the user asked whether we were drawing the wrong conclusions about PIO vs DMA, and we were, more broadly than the question implied. Every CPU figure in FINDINGS 24-34 is scored against the full 833,333 cycles/frame with nothing subtracted for moving the bitstream off disk. Debiting the HD63450 cycle-steal at the long-standing 8 clk/word ESTIMATE, "1 frame of 120 misses" becomes 84 of 120, median 112.4%. PIO at the span rate is 99.8% of the machine. Spans buy cycles by spending bandwidth and the bandwidth returns as steal, so 31.6's "fits completely" becomes a worst frame of 114.3%. 10 fps absorbs it: median 93.7%, 1/120. FINDINGS 35. `11_cpu_budget.py` takes --io dma|pio|none, defaults to dma, and warns if asked for none. Also landed: - item 1 done: the cost model checked against the 68000 on a cost-aware container, -3.07% to +0.01%, whole-window mean -1.22%. FINDINGS 34. - item 4 done: the container carries its own 4-byte record alignment (DLX2). 94/120 record starts were on odd addresses -- an address error, not a slow read -- now 0/120 for 16 B/s. Re-encoding reproduces 31.1 exactly. FINDINGS 33. - a `scsi` window does not fit the 2 MB machine the rig emulates (2.84 MB of stream past a 0x200000 ceiling). The gate now verifies 80 of 120 frames and SAYS so, and fails loudly when the pass does not complete, instead of reporting a phantom 49,005-pixel diff. FINDINGS 36. Three near-misses this session had one shape: an unobservable run nearly produced a false finding. stdbuf -oL on any MAME job that prints progress -- a file is block-buffered too, and a run that is merely finishing looks exactly like one that is wedged. check.sh ALL GREEN. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |
||
|
prosolis
|
06b98d4b47 |
Price cycles in the mode decision: 37 misses become 1, for 0.26 dB
The decoder has been CPU-bound since FINDINGS 28 while the mode decision
minimised D + lam*R -- distortion against BYTES. decide() now minimises
D + lam*bytes + mu*cycles, and ratectl bisects mu per frame against the
833,333-cycle budget with the lam bisection nested inside it. On the worst
sustained window:
sasi 27.22 -> 26.95 dB, 109.5 -> 109.4 KB/s, 37/120 misses -> 1
scsi 29.90 -> 29.27 dB, 280.0 -> 278.6 KB/s, 51/120 misses -> 1
Bitrate does not move: the byte controller still binds, and mu changes WHICH
modes are bought. V4 is what it stops buying -- 25.2 -> 20.3% of blocks at sasi
and 15.0 -> 5.3% at scsi, where RAW takes it. That is 28.8's inversion in
practice: RAW is dearer in bytes and cheaper in cycles, so only the byte-rich
profile can buy its way out of V4.
Three things worth knowing beyond the headline:
- The one frame that still misses, at both profiles, is FRAME 0 -- no previous
reconstruction, so 100% changed by definition, which is also what a scene
cut is. It comes out at the all-V1 floor of 110.6% and is emitted late on
purpose. Freezing a cut to make a deadline is the worse failure.
- 28.7's "11 frames are impossible" was too pessimistic. That floor held the
SKIP set fixed and asked how cheaply the drawn blocks could be drawn; the
real decision can also MOVE a block to SKIP, which above ~90% non-SKIP is
the only lever left.
- SKIP's price depends on its neighbours (13.25 cycles clustered, 45 mixed),
which a per-block lagrangian cannot see. The way out is that the two uses
need not share a cost function: a ranking constant inside decide(), the
exact clustered rule for the frame-level bisection. vq_hybrid.cycles() is
now the one definition of that rule and 11_cpu_budget.py imports it.
Gated: 09_ratectl_drift.py runs both controllers, both 0/120 drifting frames.
The cost-aware container decodes pixel-exact on the 68000 (120 frames). ON by
default in encode.py; --no-cpu-fit restores session 7. check.sh ALL GREEN.
Still a model, not a measurement, for THIS container: FINDINGS 31's cycle
figures come from vq_hybrid.cycles (within 1 point of the 68000 on four frames
of the session-7 container). Timing this one on the machine is step 1 of the
next session -- it was started and killed for time, and it is slow.
FINDINGS 31. tools/analysis/13_cpu_ratectl.py.
Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
|
||
|
prosolis
|
497f88b945 |
Rate control: rebuilt per-frame, wired in, and gated at zero drift
FINDINGS 26 stopped the session-5 rate controller before it shipped: it built a lam-ladder of independent whole-sequence encodes and picked frames off it, so SKIP blocks referenced reconstructions the decoder never saw -- 111 of 120 frames drifted. The fix is the structural one 26.1 said it had to be. vq_hybrid is now frame-drivable -- frame_ctx / decide / paint -- and encode() is a thin loop over it. Rate control drives the same three calls, bisects lam per frame under the leaky bucket, and feeds back the frame it actually emitted. The desync has no way to occur, and 09_ratectl_drift.py goes 111/120 -> 0/120. That test is now part of check.sh, which is ~2 min rather than ~40 s. Both overshoots on the worst sustained window are closed for under 1 dB, totals including audio: sasi 137.4 -> 109.5 KB/s (-0.60 dB), scsi 381.6 -> 280.0 KB/s (-0.91 dB). Zero frames hit the lam=800 cliff, so nothing was destroyed to get there. Rate control also makes the display path cheaper -- scsi's median drops 53.6% -> 47.1% -- because raising lam moves blocks to SKIP and V1. Two knobs measured rather than guessed. --rc-floor is worth 0.00 dB on that window and defaults to the profile lam, so rate control cannot regress content that already fits. --prefill defaults to 0 and is documented as a trap: it buys a permission to overshoot of exactly bucket/nframes, and on a 14-frame clip it disables the controller outright. FINDINGS 26.5 was wrong in both halves and 27.6 records it. _paint was not the bottleneck (14% of a frame, though vectorising it was still right at 17.1x) and the ladder was never "minutes" -- those were k-means in build(). What makes per-frame rate control affordable is that VQ.assign depends on neither lam nor prev, so it is cached one frame deep: a 12-step search over 120 frames costs 0.31 s against 49.1 s. Also caught: fixed-lam sasi was already 5% over target on 00020, the clip everyone called easy. Nothing noticed because the profile table quotes PSNR and not bitrate. check.sh: ALL GREEN. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |
||
|
prosolis
|
e00264a058 |
Find the sustained action sequence: it breaks both profiles
The open risk since session 2 was "a sustained action sequence could still break the bitrate", with every clip measured so far being 1.2-1.7 s. Closed by measurement rather than by sampling clips by hand. 07_motion_survey.py scans a whole stream at 96x72 for the hottest sliding window of inter-frame difference. On 00223 the spread between the quietest and hottest sustained 10 s windows is 10.6x, which is the argument for not eyeballing it. Hottest is t=539.4s, the Singe endgame. There, with the fixed lam the CLI uses, sasi overshoots 110 -> 129.6 KB/s (+18%) and scsi 280 -> 373.8 KB/s (+34%). Rate control moves from "insurance, not a fix" to required, and is promoted above the full-disc survey. The bus is not broken -- 381.6 KB/s still fits the 488 KB/s figure -- so FINDINGS 21 survives, at 78% of the pipe instead of a comfortable margin. Three further corrections fall out: - The two largest streams on the disc are bonus material. 00216 is the feature with a burned-in commentary PiP; 00215 is the commentary. 00223 is the clean 9.4 min. A size-ranked survey would have encoded live action. - On hard content the 256-colour scene palette (31.33 dB) binds well before the X68000 display (40.81 dB); scsi is already within 0.51 dB of it. - FINDINGS 24.5's architecture question resolves to "both paths, chosen per frame": 30-53% of frames sit above the 70% crossover. Picking per frame costs a median 37.0% of the frame budget and caps at 53.6%. Reporting for this is wired into encode.py, which previously only printed a mean over all frames -- the one statistic that cannot answer a per-frame question. extract.py takes optional start/dur; 08_mode_map.py renders source | decoded | block-mode map to .webm. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |
||
|
prosolis
|
e4062ed294 |
Session 2: hybrid VQ codec, two quality profiles, three corrections
Answers session 1's critical-path question. Flat 4x4 VQ at k=256 was prototyped and REJECTED by eye: Dirk's face disintegrates and ink outlines break into 4-pixel stair-steps. The 256-colour palettised frame is excellent, so the palette was never the problem -- block VQ was. Replaced it with a Cinepak-style hybrid: each 4x4 block is SKIP, one 4x4 codeword, four 2x2 codewords, or RAW literal pixels, chosen per block by rate-distortion. The RAW escape makes lam=0 pixel-exact (measured 0.00 dB loss), so the quality knob spans lossless to heavily-compressed in one bitstream. Per the user's decision, ships TWO quality profiles from that one codec, one decoder and one bitstream -- only the rate knob differs: sasi 45 KB/s lam=300 34.8 dB stock 10MHz ACE/EXPERT scsi 75 KB/s lam=100 35.9 dB Super/XVI or CZ-6BS1 Three corrections to earlier numbers: 1. Session 1's "183 KB/s at 12fps" was a bad extrapolation. Halving the framerate does not halve the bitrate -- decimation roughly doubles the per-frame delta. Re-measured directly: 340 KB/s for session 1's own RLE, 247 KB/s for changed-spans+deflate. The lossless floor is 319 MB. 2. A FOURTH false-good result, same family as the three in FINDINGS 4: k=1024 codebooks appeared to buy +2.4 dB free, because the rate model charged 1 byte for a 10-bit index. Charging the true cost reverses the verdict -- k=256 wins at every matched bitrate, and by 5 dB at the low end where the SASI profile lives. k=256 ships. 3. Stream inventory: the ~3-5MB clips are 1.2-1.7s, not ~60s, and some 60s streams are menus, not content. Any survey must classify before averaging. Also cleared both candidate sources for the game-logic layer: the SNES project is MIT and DirkSimple is zlib, so the arcade scene graph can be imported and the two transcriptions diffed against each other. Encoder is working end-to-end: extract.py -> vq/vq_hybrid/ratectl -> encode.py, emitting a big-endian DLX1 container the 68000 can parse with plain moves. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6 |