drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Put the frame in a container with no decoder, and find the palette is not free

Browse Source 
ROADMAP K2. DLXP1: a 49,664 B record that is 97 sectors exactly, no index and
no length word, because a packed record's length is geometry rather than
content. 582.0 KB/s, which is what FINDINGS 61.9 predicted to the tenth, and it
encodes in 3.3 s because there is no k-means in it.

px68k's own x68k/gvram.c renders the container's bytes index-exact with the
harness computing no interleave -- the only test that can catch an encoder whose
byte order is wrong, since a container round-trips against its own inverse
either way. Both negative controls fail as they must.

The picture is re-derived against this project's builder rather than PIL's
(34.05 dB against 61.9's 34.08) and the GGGGGRRRRRBBBBBI word is charged for the
first time in this tree: 0.53 dB, on every row, so it moves no comparison.

What the control found is the finding. A packed container on a SCENE palette
lands exactly on the codec's ceiling, so the whole +2.31 dB is the per-frame
palette and nothing else -- and 231 of 256 entries change every frame, which
makes a mismatched paint 12.8 dB worse than the correct pairing, on screen for
roughly half of every frame slot if buffer mode does not blank. So B2 now
decides which packed CONTAINER ships, not only which player. The fallback is
already a flag: --scene-palette --no-palette is 30.79 dB, zero churn, 576.0 KB/s
and still +2.07 dB on the shipping codec.

62.5 is priced and is a wash: palette first 20.32 dB, palette last 20.33.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-25 07:31:05 -07:00
parent 07f36c2af9
commit f1007a0dbc
15 changed files with 1189 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/encoder/dlxp.py
+218
View File
@@ -0,0 +1,218 @@
#!/usr/bin/env python3
"""DLXP1 -- the PACKED container, and the one place its layout rule is written.
from dlxp import DLXP, pack_picture, unpack_picture, write
THIS CONTAINER HAS NO DECODER. That is the point of it (FINDINGS 61): a record
is the bytes a DMA channel puts straight into the palette registers and GVRAM,
so the "reference decoder" here is not a decoder at all -- it is an assertion
about where each byte lands. `dlx.py` exists because a 68000 has to PARSE the
codec's container and can get it wrong; this file exists because a DMA channel
must NOT have to parse anything, and the format is what makes that true.
The layout, all of it fixed, all of it verified as a picture in FINDINGS 47.2:
* 256-colour GVRAM normally throws away the high byte of every word a CPU
writes, so a picture byte costs two disc bytes. CRTC R20 bit 11 turns the
masking off (46.5/47.1), and with the two 256-colour pages scrolled apart by
384 one word carries TWO pixels: word `i` of a row is
(pix[y][i+128] << 8) | pix[y][i]
-- page 1 (X-scrolled 384, transparent top) shows columns 128..255, page 0
(unscrolled, opaque bottom) shows columns 0..127.
* so a row is 128 words = 256 BYTES for 256 pixels, and big-endian storage
makes the byte order `pix[128], pix[0], pix[129], pix[1], ...`. That byte
order is not a serialisation choice: it is what the 68000's bus puts on the
high half of the word, and the channel copies bytes.
* 192 rows -> 49,152 B of picture, 1.0 B/pixel against the unpacked 2.0.
* index 0 is the TRANSPARENCY KEY and never appears; black is 255, which is
what the letterbox rows display (`vq.frame_palette`).
* words 128..511 of each row, and the letterbox rows themselves, are STATIC
SETUP -- written once at scene setup, never per frame -- so they are not in
the container. FINDINGS 47.2 lists them; keeping them out is what makes the
per-frame payload exactly the picture.
* the GVRAM line stride is 1,024 B and a row is 256 B, so the container's rows
are CONTIGUOUS and the 1,024 B step is the channel's, walked by array
chaining from one start (FINDINGS 62). A container that carried the stride
would be 4x the size and would say nothing extra.
header, 32 bytes, big-endian, then zero pad to the first sector:
0 'DLXP'
4 u16 version (1)
6 u16 flags bit 0: a per-frame palette is present
bit 1: the palette is at the END of the record
8 u16 width, u16 height
12 u16 fps, u16 nframes
16 u32 record bytes fixed, and a whole number of 512 B sectors
20 u32 palette bytes 512 (256 GRB555+I words), or 0
24 u32 picture bytes 49,152
28 u32 frames offset 512 B, i.e. sector 1
then nframes FIXED-SIZE records, each 49,664 B = 97 sectors EXACTLY.
THERE IS NO RECORD INDEX AND NO LENGTH WORD, and that is the difference DLX4's
index was invented for (49.3): a codec record's length is content-dependent, so
a producer cannot know where record i+1 starts without being told. A packed
record's length is GEOMETRY -- 192 rows of 256 B plus a palette -- so record `i`
is at `off_frm + i * rec_bytes` and a seek is arithmetic. Nothing in this
format has to be walked, which is also why the packed player has no ring
(ROADMAP K3): there is no variable-length thing to keep contiguous.
THE PALETTE ORDER IS A DECISION, NOT AN ACCIDENT (FINDINGS 62.5). Palette first
or 193rd is visible on screen for one paint -- old rows under the new palette, or
new rows under the old one -- and it is moot if buffer mode blanks the layer
(47.4/B2). It is a CONTAINER property here, chosen at encode time by
`--palette-last` and recorded in flags bit 1, so K3 can measure both without a
re-encode being an argument about which one the format assumed.
"""
import struct
import numpy as np
MAGIC = b"DLXP"
VERSION = 1
SECTOR = 512 # same rule and the same reason as dlx.SECTOR
PAL_BYTES = 512 # 256 entries, one GRB555+I word each
HDR_BYTES = 32
FLAG_PALETTE = 1 << 0
FLAG_PALETTE_LAST = 1 << 1
def pack_picture(idx):
"""(H,W) palette indices -> the bytes GVRAM wants, in GVRAM order.
The ONE place the interleave rule is applied, for the same reason
`dlx.record_lengths` is the one place the alignment rule is: every caller
that carries its own copy of a layout rule is a place the layout can drift.
"""
H, W = idx.shape
if W % 2:
raise ValueError(f"packed layout needs an even width, got {W}")
half = W // 2
left, right = idx[:, :half], idx[:, half:]
out = np.empty((H, half, 2), np.uint8)
out[:, :, 0] = right # high byte of the word -> page 1 -> col i+128
out[:, :, 1] = left # low byte -> page 0 -> col i
return out.reshape(H, half * 2)
def unpack_picture(buf, W, H):
"""The inverse, and the assertion that `pack_picture` is reversible."""
b = np.frombuffer(buf, np.uint8, W * H).reshape(H, W // 2, 2)
idx = np.empty((H, W), np.uint8)
idx[:, W // 2:] = b[:, :, 0]
idx[:, :W // 2] = b[:, :, 1]
return idx
def record_bytes(W, H, palette=True):
n = W * H + (PAL_BYTES if palette else 0)
if n % SECTOR:
raise ValueError(f"a {W}x{H} packed record is {n} B, which is not a "
f"whole number of {SECTOR} B sectors")
return n
def write(path, W, H, fps, frames, palette_last=False):
"""`frames` is a sequence of (palette_words_bytes | None, picture_bytes)."""
frames = list(frames)
pal_b = PAL_BYTES if frames and frames[0][0] is not None else 0
pic_b = W * H
rec_b = record_bytes(W, H, palette=bool(pal_b))
flags = ((FLAG_PALETTE if pal_b else 0)
| (FLAG_PALETTE_LAST if palette_last and pal_b else 0))
hdr = (MAGIC + struct.pack(">HHHHHH", VERSION, flags, W, H, fps, len(frames))
+ struct.pack(">III", rec_b, pal_b, pic_b)
+ struct.pack(">I", SECTOR))
assert len(hdr) == HDR_BYTES, len(hdr)
with open(path, "wb") as fh:
fh.write(hdr + b"\0" * (SECTOR - HDR_BYTES))
for i, (pw, pic) in enumerate(frames):
if len(pic) != pic_b or (pal_b and len(pw) != pal_b):
raise ValueError(f"frame {i}: record parts are the wrong size")
rec = pic if not pal_b else (pic + pw if palette_last else pw + pic)
fh.write(rec)
return rec_b
class DLXP:
"""Reader, and every invariant the format claims, CHECKED rather than read.
The checks are not defensive coding. A packed record is written into GVRAM
and the palette registers with no bounds test anywhere -- the channel has no
opinion about what it is copying -- so a container whose geometry is a byte
wrong does not fail, it paints.
"""
def __init__(self, path):
b = self.raw = open(path, "rb").read()
if b[:4] != MAGIC:
raise ValueError(f"{path}: not a DLXP container")
(self.version, self.flags, self.W, self.H, self.fps,
self.nframes) = struct.unpack(">HHHHHH", b[4:16])
(self.rec_bytes, self.pal_bytes,
self.pic_bytes, self.off_frm) = struct.unpack(">IIII", b[16:32])
if self.version != VERSION:
raise ValueError(f"{path}: DLXP version {self.version}")
self.has_palette = bool(self.flags & FLAG_PALETTE)
self.palette_last = bool(self.flags & FLAG_PALETTE_LAST)
if self.pic_bytes != self.W * self.H:
raise ValueError(f"{path}: picture is {self.pic_bytes} B for "
f"{self.W}x{self.H} -- the packed layout is 1.0 B/px")
if self.pal_bytes != (PAL_BYTES if self.has_palette else 0):
raise ValueError(f"{path}: palette section is {self.pal_bytes} B")
if self.rec_bytes != self.pal_bytes + self.pic_bytes:
raise ValueError(f"{path}: record is {self.rec_bytes} B, parts are "
f"{self.pal_bytes} + {self.pic_bytes}")
# The whole reason for DLX5 (58.3/59.4), and here it is free rather than
# a re-encode: the record is a fixed multiple of a sector by geometry.
if self.off_frm % SECTOR or self.rec_bytes % SECTOR:
raise ValueError(f"{path}: not sector-aligned -- stream at "
f"{self.off_frm}, record {self.rec_bytes}")
want = self.off_frm + self.nframes * self.rec_bytes
if len(b) != want:
raise ValueError(f"{path}: {len(b)} bytes, geometry says {want}")
def record(self, i):
o = self.off_frm + i * self.rec_bytes
return self.raw[o:o + self.rec_bytes]
def _split(self, i):
r = self.record(i)
if not self.has_palette:
return None, r
if self.palette_last:
return r[self.pic_bytes:], r[:self.pic_bytes]
return r[:self.pal_bytes], r[self.pal_bytes:]
def palette_words(self, i):
pw, _ = self._split(i)
if pw is None:
raise ValueError("this container carries no palette -- it was made "
"with --no-palette, and the palette a player would "
"display is not in the file to be read back")
return np.frombuffer(pw, ">u2").astype(np.uint16)
def indices(self, i):
_, pic = self._split(i)
return unpack_picture(pic, self.W, self.H)
def palette_rgb(self, i):
"""(256,3) uint8 -- what the DISPLAY produces, not what the encoder meant.
The palette in a record is already a GRB555+I word, so this is where the
5-bit hardware quantisation gets charged. Everything upstream of the
container is in RGB888 and 61.9's +4.89 dB was quoted there; a player's
number has to come from here. Same maths as `dlxload.pack_palette` and
`tools/bench/verify_frame256.py` -- the shared LSB `I` is a bit in the
word, so unpacking it needs no choice made.
"""
w = self.palette_words(i).astype(int)
f = np.stack([(w >> 6) & 31, (w >> 11) & 31, (w >> 1) & 31], 1) # R,G,B
p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
return p6((f << 1) | (w & 1)[:, None]).astype(np.uint8)
def render(self, i):
"""(H,W,3) uint8 -- the frame as the display produces it."""
return self.palette_rgb(i)[self.indices(i)]
def kbps(self):
return self.nframes * self.rec_bytes / (self.nframes / self.fps) / 1024