Put the frame in a container with no decoder, and find the palette is not free

ROADMAP K2. DLXP1: a 49,664 B record that is 97 sectors exactly, no index and
no length word, because a packed record's length is geometry rather than
content. 582.0 KB/s, which is what FINDINGS 61.9 predicted to the tenth, and it
encodes in 3.3 s because there is no k-means in it.

px68k's own x68k/gvram.c renders the container's bytes index-exact with the
harness computing no interleave -- the only test that can catch an encoder whose
byte order is wrong, since a container round-trips against its own inverse
either way. Both negative controls fail as they must.

The picture is re-derived against this project's builder rather than PIL's
(34.05 dB against 61.9's 34.08) and the GGGGGRRRRRBBBBBI word is charged for the
first time in this tree: 0.53 dB, on every row, so it moves no comparison.

What the control found is the finding. A packed container on a SCENE palette
lands exactly on the codec's ceiling, so the whole +2.31 dB is the per-frame
palette and nothing else -- and 231 of 256 entries change every frame, which
makes a mismatched paint 12.8 dB worse than the correct pairing, on screen for
roughly half of every frame slot if buffer mode does not blank. So B2 now
decides which packed CONTAINER ships, not only which player. The fallback is
already a flag: --scene-palette --no-palette is 30.79 dB, zero churn, 576.0 KB/s
and still +2.07 dB on the shipping codec.

62.5 is priced and is a wash: palette first 20.32 dB, palette last 20.33.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-25 07:31:05 -07:00
parent 07f36c2af9
commit f1007a0dbc
15 changed files with 1189 additions and 16 deletions
+158
View File
@@ -6432,3 +6432,161 @@ encoder K2 has not been written yet, and the whole question is **moot if buffer
mode blanks the layer** (47.4/B2) because nothing is displayed during the paint
at all. **Filed, not answered.** It is named here so that the choice in the
final player is a decision rather than an accident of which run happened first.
## 63. The packed container, and the palette that buys 2.31 dB has a price nobody had counted (session 31)
**Emulated, and one stage of it is not emulated at all.** The quality numbers
are host arithmetic over the Blu-ray's own frames. The rendering check is
**px68k's real `x68k/gvram.c`**, linked headless the way `tools/bench/c68k`
links its CPU core. No MAME run was needed for any of this and no real hardware
ran. 47.4 — does buffer mode blank the layer — is still the board question, and
this session made it **bigger**.
ROADMAP **K2 is done**. `tools/encoder/dlxp.py` is the format,
`tools/encoder/pack.py` is the encoder, `tools/analysis/30_packed_container.py`
is the gate and the re-derivation, `tools/bench/gvpack/verify_dlxp.py` renders
the container's own bytes on the second emulator, and `check.sh` runs all four.
### 63.1 DLXP1: the container whose correctness is that nothing parses it
`dlx.py` exists because a 68000 has to **parse** the codec's container and can
get it wrong. `dlxp.py` exists for the opposite reason: a DMA channel must not
have to parse anything, and the format is what makes that true.
| | |
|---|---|
| record | **49,664 B = 97 sectors EXACTLY** — 512 B of palette, 49,152 B of picture |
| picture | 192 rows x 128 words, word `i` = `(pix[y][i+128] << 8) \| pix[y][i]` — the layout 47.2 rendered pixel-exactly on both emulators |
| index | **none, and none is possible to need**: a record's length is geometry, so record `i` is at `off_frm + i * rec_bytes` and a seek is arithmetic |
| wire | **582.0 KB/s, fixed** — 61.9 predicted 582.0 and the container is 582.0 |
| encode | **3.3 s for 120 frames**, against ~55 s for the codec, 95% of which is k-means |
DLX4's record index was invented (49.3) because a codec record's length is
content-dependent and a producer streaming off a disc cannot learn it by walking
bytes it has not fetched. **A packed record has no such problem to solve**, and
that is the same reason the packed player has no ring: there is nothing
variable-length to keep contiguous.
**Sector alignment, which cost session 28 a whole re-encode (60.1), is free
here.** 49,664 is 97 sectors because a 256x192 picture and a 256-entry palette
happen to be. It is still *checked* rather than assumed, because the thing being
protected is not tidiness: the channel copies bytes and has no opinion about
them, so a container whose geometry is a byte wrong does not fail, it paints.
### 63.2 The container's OWN BYTES, through px68k's GVRAM code, with no interleave computed
`verify_gvpack.py` (47.2) checks the **layout**: it hands the harness a picture
and the harness computes the interleave. That cannot catch an encoder whose byte
order is wrong, because a container round-trips against its own inverse either
way. `verify_dlxp.py` writes a **DLXP1 record's bytes into GVRAM verbatim** and
asks px68k what they display as.
**256x192 index-exact, letterbox on the reserved black, index 0 never on
screen** — frames 0, 60 and 119. Two negative controls, both mechanisms this
container depends on rather than decoration: **R20 bit 11 clear loses 24,576 px**
(exactly the 128 columns page 1 carries) and **page 1 unscrolled loses 48,958**.
### 63.3 The picture, RE-DERIVED against this project's own builder, and charged the hardware word
This is what the session 30 handoff asked for. 61.9's 34.08 dB was PIL's free
256-colour MEDIANCUT and was filed as "a direction, not the player's number". It
had **two** debts, not one, and the second had gone unnamed: every PSNR this
project has ever quoted — 29.19, 31.33, 34.08 — is measured in the **RGB888
palette domain**, upstream of the X68000's `GGGGGRRRRRBBBBBI` word (23.3). A
packed record carries that word and nothing else, so a player's number has to
come from the other side of it.
PSNR vs the 24-bit source, mean over the same 120-frame window, on the gate
container's own bytes:
| | RGB888 | **GRB555** |
|---|---:|---:|
| CODEC, the gate container (440.4 KB/s) | 29.07 | **28.72** |
| 256c scene palette — the codec's CEILING | 31.32 | **30.79** |
| PACKED, 254c SCENE palette *(the control)* | 31.32 | **30.79** |
| **PACKED CONTAINER, 254c PER-FRAME** | **34.05** | **33.10** |
**61.9's direction survives the real builder: 34.05 against its 34.08.** The two
entries the packed layout reserves — index 0 for the transparency key, 255 for
black — cost **+0.0003 dB**, which is to say nothing at all, and marginally the
right way. 60.3 measured one reserved entry at 0.04 dB; two is not twice that,
it is noise.
**The GRB555 word costs 0.53 dB** and it costs every row of the table, so no
comparison in this project moves because of it. It is stated because a player's
number should be a player's number.
**And the CONTROL is the finding under the headline.** A packed container with
one scene palette lands **exactly on the codec's ceiling, 30.79 dB** — as it
must, because at that point the only difference left between them is VQ. So the
whole of the packed branch's picture advantage over the codec's *ceiling*, +2.31
dB, is **the per-frame palette and nothing else**. Not the packing, not the
literal frames, not 254 colours. One mechanism, and 61.9 named it correctly.
### 63.4 62.5 is priced, and the answer is that the ORDER does not matter and the MISMATCH does
62.5 filed palette-first-or-193rd as a free choice whose severity "depends on how
much the palette moves between consecutive frames, which is a property of the
encoder K2 has not been written yet". It is written now.
| | |
|---|---:|
| palette entries that CHANGE frame to frame | **231.1 of 256 (90%)** |
| palette FIRST — old rows under the new palette | **20.32 dB** (12.79) |
| palette LAST — new rows under the old palette | **20.33 dB** (12.78) |
**The two orders are indistinguishable — 0.01 dB apart — so 62.5's choice is a
wash and can be made on other grounds.** The container makes it a flag
(`--palette-last`, flags bit 1) rather than an assumption, so K3 can run both.
**What the same measurement found is not a wash.** A per-frame palette is not a
small delta: 90% of the entries move every frame, and a picture under the
neighbouring frame's palette is **12.8 dB worse than the correct pairing**. The
mismatch is a wipe rather than a flash — rows arrive top to bottom, so part of
the screen is always right, and the transfer is 55.2% of a frame slot (61) —
but at 12 fps that is a colour-scrambled region present for roughly half of
every frame slot, forever.
**And it was LOOKED AT, not only scored.**
`30_packed_container.py --mismatch-png` writes the frame whose mismatch is
closest to the mean — chosen that way so the picture is not an outlier picked to
flatter the number — as correct render | same frame under the next frame's
palette | 24-bit source:
![The palette mismatch: correct, mismatched, source](img/palette-mismatch.png)
**It is chroma speckle and a shifted ground, not a scramble.** Two median-cut
palettes of adjacent frames occupy a similar gamut, so nothing goes
psychedelic — the lava turns red where it should be ochre, the dragon's scales
break into noise, and the torch survives. That is milder than 12.8 dB sounds and
worse than it looks in a still, because the still does not show it arriving and
leaving twelve times a second.
**So B2 stopped being a headroom question and became a picture question, again
and worse.** 61.6 said whether buffer mode blanks decides *which player exists*.
63.4 says that if it does **not** blank, it also decides **which packed
container** exists, because the per-frame palette is what the artefact is made
of. The codec never had this exposure: its palette is scene-constant, so its
tear is old picture against new picture and never old colours against new ones.
**And the fallback is already in the encoder, which is the useful half.**
`pack.py --scene-palette` emits the control row: **30.79 dB, zero palette churn,
no mismatch to have**, still **+2.07 dB on the shipping codec as the display
renders both**, and with `--no-palette` alongside it — a scene palette is loaded
once at scene setup, which is what the codec has always done — the record loses
its 512 B and the wire is **576.0 KB/s against 582.0**, cheaper on the one
resource the packed branch is short of. The per-frame palette is worth +2.31 dB and it is now a **priced**
+2.31 dB rather than a free one.
### 63.5 What this does not settle
- **Nothing about clocks.** 29_packed_player.py owns those, off the measured
blit, and nothing here moves them: 55.2% of a frame at the 9 clk/B
dual-address floor, unchanged.
- **Nothing about the medium.** 582.0 KB/s is geometry. Whether anything
sustains it is **B1**, and this container cannot negotiate — a codec's
bitrate is a lever and a literal frame's is arithmetic.
- **Nothing on a machine.** No packed frame has been put on screen from a
container by a 68000 or by a channel. That is **K3**, and 63.2 is the strongest
statement available without it: the bytes are right, on a second emulator's
own GVRAM model, with the harness computing nothing.
+48 -10
View File
@@ -42,6 +42,19 @@ the run bounds the model and not the board, and what a real palette register
does with a byte write is UNMEASURED. B4 is the cheapest hardware item in the
project and a negative costs 0.28% of a frame. **K2, the packed container, is
next.**
Amended end of session 31: **K2 IS DONE, AND IT COST THE BRANCH SOMETHING
(FINDINGS 63).** DLXP1 is a 49,664 B record = 97 sectors exactly, no index, no
decoder, 582.0 KB/s — exactly 61.9's prediction — and px68k's own `gvram.c`
renders the container's bytes index-exact with the harness computing no
interleave. 61.9's picture claim survives the real builder: **34.05 dB against
its 34.08**, and the hardware GRB555 word is charged on top for the first time
in this project (0.53 dB, on every row, so it moves nothing). **But the control
row landed exactly on the codec's ceiling**, so the whole +2.31 dB is the
PER-FRAME PALETTE and nothing else — and 90% of that palette changes every
frame, which makes a mismatched paint **12.8 dB worse** than the correct
pairing. **B2 now decides which packed CONTAINER exists, not only which player**
(63.4). The fallback is a flag: `--scene-palette --no-palette` is 30.79 dB, zero
churn, 576.0 KB/s and still +2.07 dB on the shipping codec. **K3 is next.**
**THE COMPLETION TARGET IS M3, THE VERTICAL SLICE** (USER DECISION): one scene
tree — a decision point, two outcomes, a death clip — with audio, streaming from
@@ -128,6 +141,17 @@ the priority register `0xE82500` at `0x0000` (47.3).
> is dark for **30..113%**. Under MAME's reading the cheap architecture is the
> dark one. **B2 and B1 are now the same decision from two sides, and B2 is the
> five-minute half.**
>
> **Session 31 raised it again, from the other direction (FINDINGS 63.4).** If
> buffer mode does NOT blank, the packed player's per-frame palette is not
> merely visible during the paint — **231 of 256 entries change every frame**,
> and a picture under the neighbouring frame's palette is **12.8 dB worse** than
> the correct pairing, on screen for roughly half of every frame slot at 12 fps.
> So a non-blanking board does not just cost the packed player a blank interval,
> it may cost it the per-frame palette — which is **the whole +2.31 dB** the
> branch has over the codec's ceiling. The fallback is already an encoder flag
> (`pack.py --scene-palette --no-palette`: 30.79 dB, no churn, 576.0 KB/s), so
> what B2 decides is now **which packed container ships**, not whether one can.
**B4. Does a real palette register take a BYTE write?** (62.4, new session 30.)
`$E82000` is 256 16-bit registers. The decoder-free packed player's per-frame
@@ -243,14 +267,25 @@ frame is not tied to a scene palette the codec's codewords index into. It costs
over `memory_array`, whose `write16` is a plain `COMBINE_DATA`, so it has no
handler to be wrong about and cannot discriminate. And it filed one open
design choice, 62.5: palette FIRST or 193rd is visible on screen and is not
decided.
- **K2. A packed container.** Quantise to **254** colours (index 0 is the
transparency key, black at 255 — 47.2), interleave column *i* with *i+128*,
emit a per-frame palette, sector-align the record. Simpler than a coded one:
no VQ, no rate control, no mode map, no `lam`. `encode.py` already aligns
(DLX5) and `prep_frame.py --pack-transparent` already emits the layout.
**Re-derive 61.9's 34.08 dB against `vq.scene_palette` here** — the measured
figure is PIL's MEDIANCUT and is a direction, not the player's number.
decided. **Session 31 PRICED it and it is a wash — 20.32 dB against 20.33
(63.4) — so it is a container flag (`--palette-last`) and K3 runs both.**
~~**K2. A packed container.**~~
**DONE, session 31 — FINDINGS 63.** `tools/encoder/dlxp.py` is DLXP1 and
`pack.py` writes it: 254 colours with index 0 held free and black at 255,
column *i* interleaved with *i+128*, a per-frame palette, and a **49,664 B
record that is 97 sectors EXACTLY** — the alignment that cost session 28 a
re-encode is free here because a packed record's length is geometry. **No
index and no length word**, for the same reason: nothing has to be walked.
**582.0 KB/s, which is 61.9's prediction to the tenth.** Encodes in 3.3 s
because there is no k-means in it.
**The re-derivation is done and 61.9 survives it: 34.05 dB against 34.08**,
and the GRB555 word is charged for the first time (63.3) — 0.53 dB, on every
row of the table, so it moves no comparison. **The two reserved entries cost
0.0003 dB.**
**What it also found** (63.4): the SCENE-palette control lands exactly on the
codec's ceiling, so the whole +2.31 dB is the per-frame palette; 90% of that
palette changes every frame; and a mismatched paint is 12.8 dB worse, for
roughly half of every frame slot, if buffer mode does not blank.
- **K3. End to end, off the disc.** Palette, page-1 X-scroll 384, priority
`vc1 = 0x0002`, R20 bit 11; one chained DMA a frame. Gated pixel-exact over the
same 120-frame window the decoder is gated on.
@@ -261,8 +296,10 @@ DMAC-direct packed player has **no ring** — `ring.i`, `xfer.i` and most of
simplification that large usually hides something, and 61.7.2 names the specific
untested thing: a chained transfer has never run back to back at 12 fps.
**K1 and K2 survive a bad answer to B2. K3 does not.** ~~Do K1 first.~~ **K1 is
done (session 30, FINDINGS 62); K2 is next.**
**K1 and K2 survive a bad answer to B2. K3 does not.** ~~Do K1 first.~~ ~~K1 is
done (session 30, FINDINGS 62); K2 is next.~~ **Both are done. K3 is next — and
63.4 added a second thing for it to run: BOTH palette orders, which is a flag
(`--palette-last`) and not a re-encode.**
---
@@ -683,6 +720,7 @@ P2 re-encode bundle DONE (60): DLX5, records ARE sectors ───────
K1 palette-register DMA? ─> K2 packed container ─> K3 end to end ──┤
(61.9; K1/K2 survive a bad B2, K3 does not) │
K1 DONE s30 (62), K2 DONE s31 (63) ── K3 is the open one │
P4a WIRING (the channel behind ring.i's mailbox) <- THE LAST ITEM ─┤
P1 P2 P3 P4b P5 P7, P6 bus cost (52), G1 scene graph (56) ────────┼─> M2 ─>
B1 seek+rate (sets HEADROOM, not fit) ─────────────────────────────┘ M3 ─> M4
+112
View File
@@ -1,3 +1,115 @@
# Status & next-session handoff — end of session 31 (2026-08-25)
## Session 31: the packed container, and the palette that buys 2.31 dB has a price
**Green light first and last: `./tools/bench/check.sh` was ALL GREEN before any
of this (`tmp/check_s31_start.log`) and ALL GREEN after** — the same stages, plus
one new one.
**FINDINGS 63.** Mostly **host arithmetic** over the Blu-ray's own frames; the
one rendering check is **px68k's real `x68k/gvram.c`**, linked headless the way
`tools/bench/c68k` links its CPU core. No MAME run was needed and no real
hardware ran. 47.4 — does buffer mode blank the layer — is still the board
question, and this session made it **bigger**.
**THE HEADLINE. ROADMAP K2 is done.** `tools/encoder/dlxp.py` is the format,
`pack.py` the encoder, `tools/analysis/30_packed_container.py` the gate and the
re-derivation, `tools/bench/gvpack/verify_dlxp.py` the second-emulator render.
| | |
|---|---|
| record | **49,664 B = 97 sectors EXACTLY**, 512 B palette + 49,152 B picture |
| index | **none, and none can be needed** — a packed record's length is geometry |
| wire | **582.0 KB/s**, exactly what 61.9 predicted |
| encode | **3.3 s** for 120 frames, against ~55 s for the codec |
| render | **index-exact on px68k, with the harness computing no interleave** |
**1. THE RE-DERIVATION SESSION 30 ASKED FOR, and it had TWO debts not one**
(63.3). 61.9's 34.08 dB was PIL's free 256-colour MEDIANCUT; the unnamed second
debt was that **every PSNR this project has quoted — 29.19, 31.33, 34.08 — is
RGB888, upstream of the `GGGGGRRRRRBBBBBI` word (23.3)**, and a packed record
carries that word and nothing else.
| | RGB888 | **GRB555** |
|---|---:|---:|
| CODEC, the gate container (440.4 KB/s) | 29.07 | **28.72** |
| 256c scene palette — the codec's CEILING | 31.32 | **30.79** |
| PACKED, 254c SCENE palette *(control)* | 31.32 | **30.79** |
| **PACKED CONTAINER, 254c PER-FRAME** | **34.05** | **33.10** |
61.9 survives the real builder: **34.05 against 34.08**. The layout's two
reserved entries cost **+0.0003 dB**. The GRB555 word costs 0.53 dB and costs
every row, so it moves no comparison.
**2. THE CONTROL IS THE FINDING UNDER THE HEADLINE.** A packed container with a
SCENE palette lands **exactly on the codec's ceiling**, as it must. So the whole
+2.31 dB the packed branch has over that ceiling is **the per-frame palette and
nothing else** — not the packing, not literal frames, not 254 colours.
**3. 62.5 IS PRICED, AND THE ORDER IS A WASH** (63.4). Palette first 20.32 dB,
palette last 20.33 — 0.01 apart, so the choice can be made on other grounds, and
`dlxp.py` makes it a container flag rather than an assumption.
**4. AND THE MISMATCH IS NOT A WASH.** **231 of 256 palette entries change every
frame (90%)**, and a picture under the neighbouring frame's palette is **12.8 dB
worse**. It is a wipe, not a flash, and the transfer is 55.2% of a frame slot —
so at 12 fps a colour-scrambled region is on screen for roughly half of every
slot, forever. **B2 stopped being a headroom question and became a picture
question again**: if buffer mode does not blank, it decides not just which
player exists (61.6) but **which packed container** does. The codec never had
this exposure — its palette is scene-constant, so its tear is old picture
against new picture, never old colours against new ones.
**5. THE FALLBACK IS ALREADY IN THE ENCODER.** `pack.py --scene-palette
--no-palette`: **30.79 dB, zero churn, nothing to mismatch, +2.07 dB on the
shipping codec as the display renders both, and 576.0 KB/s instead of 582.0.**
The per-frame palette is now a **priced** +2.31 dB rather than a free one.
## HANDOFF — start here
**THE TREE IS ALL GREEN.** Session 31's work is a commit of its own.
### The work, in the order it should be done
**1. K3 — END TO END, OFF THE DISC.** Scene setup — palette, page-1 X-scroll
384, priority `vc1 = 0x0002`, R20 bit 11, the 384 zeroed words a row and the
letterbox rows — then **one chained DMA a frame** out of a DLXP1 record, with
the palette as the 193rd or 1st array entry (63.4 says either). Gate it
pixel-exact over the same 120-frame window the decoder is gated on. The
container is `tmp/packed_singe.dlxp` and `check.sh` rebuilds it every run.
**2. RUN BOTH PALETTE ORDERS while K3 is being built.** It is a flag
(`--palette-last`), the containers already exist, and 63.4 bounds the cost but
cannot decide it — dB over a whole frame is not what an eye sees in a wipe.
### What is PARKED, so it is not re-derived
**E7, E4 and C1** (61.8), and **P4a's wiring** for the video path. Do not delete
any of it — B2 is unanswered and 48.1's prior leans against packing.
### Risks that are OURS, not hardware
1. **A chained transfer has never run back to back at 12 fps.** Unchanged since
session 30: run 9 was seven entries, once. A frame is 193 and a second is
twelve frames.
2. ~~34.08 dB is PIL's MEDIANCUT~~ **CLOSED, 63.3: 34.05 dB against this
project's own builder, and the hardware word is charged on top.**
3. **The packed player deletes `ring.i`, `xfer.i` and most of `stream.s` from
the video path.** A simplification that large usually hides something.
4. **B2 sits over all of it, and 63.4 raised what it decides.** B4 still sits
over the palette entry specifically, with a small blast radius.
### Reproducing this session
./tools/bench/check.sh # ALL GREEN
python3 tools/encoder/pack.py tmp/fr_singe tmp/packed_singe.dlxp --nframes 120
python3 tools/analysis/30_packed_container.py tmp/packed_singe.dlxp
python3 tools/bench/gvpack/verify_dlxp.py tmp/packed_singe.dlxp 0 --controls
**WHAT IS NEXT.** K3: end to end, off the disc.
---
# Status & next-session handoff — end of session 30 (2026-08-25)
## Session 30: one channel start paints a whole frame, and the palette has no handler to be wrong about
Binary file not shown.

After

Width:  |  Height:  |  Size: 224 KiB