Dragon-s-Lair-X68k/tools/encoder/dlxp.py

#!/usr/bin/env python3
"""DLXP1 -- the PACKED container, and the one place its layout rule is written.

    from dlxp import DLXP, pack_picture, unpack_picture, write

THIS CONTAINER HAS NO DECODER.  That is the point of it (FINDINGS 61): a record
is the bytes a DMA channel puts straight into the palette registers and GVRAM,
so the "reference decoder" here is not a decoder at all -- it is an assertion
about where each byte lands.  `dlx.py` exists because a 68000 has to PARSE the
codec's container and can get it wrong; this file exists because a DMA channel
must NOT have to parse anything, and the format is what makes that true.

The layout, all of it fixed, all of it verified as a picture in FINDINGS 47.2:

  * 256-colour GVRAM normally throws away the high byte of every word a CPU
    writes, so a picture byte costs two disc bytes.  CRTC R20 bit 11 turns the
    masking off (46.5/47.1), and with the two 256-colour pages scrolled apart by
    384 one word carries TWO pixels: word `i` of a row is
        (pix[y][i+128] << 8) | pix[y][i]
    -- page 1 (X-scrolled 384, transparent top) shows columns 128..255, page 0
    (unscrolled, opaque bottom) shows columns 0..127.
  * so a row is 128 words = 256 BYTES for 256 pixels, and big-endian storage
    makes the byte order `pix[128], pix[0], pix[129], pix[1], ...`.  That byte
    order is not a serialisation choice: it is what the 68000's bus puts on the
    high half of the word, and the channel copies bytes.
  * 192 rows -> 49,152 B of picture, 1.0 B/pixel against the unpacked 2.0.
  * index 0 is the TRANSPARENCY KEY and never appears; black is 255, which is
    what the letterbox rows display (`vq.frame_palette`).
  * words 128..511 of each row, and the letterbox rows themselves, are STATIC
    SETUP -- written once at scene setup, never per frame -- so they are not in
    the container.  FINDINGS 47.2 lists them; keeping them out is what makes the
    per-frame payload exactly the picture.
  * the GVRAM line stride is 1,024 B and a row is 256 B, so the container's rows
    are CONTIGUOUS and the 1,024 B step is the channel's, walked by array
    chaining from one start (FINDINGS 62).  A container that carried the stride
    would be 4x the size and would say nothing extra.

  header, 32 bytes, big-endian, then zero pad to the first sector:
    0   'DLXP'
    4   u16 version (1)
    6   u16 flags        bit 0: a per-frame palette is present
                         bit 1: the palette is at the END of the record
    8   u16 width, u16 height
    12  u16 fps,   u16 nframes
    16  u32 record bytes         fixed, and a whole number of 512 B sectors
    20  u32 palette bytes        512 (256 GRB555+I words), or 0
    24  u32 picture bytes        49,152
    28  u32 frames offset        512 B, i.e. sector 1

  then nframes FIXED-SIZE records, each 49,664 B = 97 sectors EXACTLY.

THERE IS NO RECORD INDEX AND NO LENGTH WORD, and that is the difference DLX4's
index was invented for (49.3): a codec record's length is content-dependent, so
a producer cannot know where record i+1 starts without being told.  A packed
record's length is GEOMETRY -- 192 rows of 256 B plus a palette -- so record `i`
is at `off_frm + i * rec_bytes` and a seek is arithmetic.  Nothing in this
format has to be walked, which is also why the packed player has no ring
(ROADMAP K3): there is no variable-length thing to keep contiguous.

THE PALETTE ORDER IS A DECISION, NOT AN ACCIDENT (FINDINGS 62.5).  Palette first
or 193rd is visible on screen for one paint -- old rows under the new palette, or
new rows under the old one -- and it is moot if buffer mode blanks the layer
(47.4/B2).  It is a CONTAINER property here, chosen at encode time by
`--palette-last` and recorded in flags bit 1, so K3 can measure both without a
re-encode being an argument about which one the format assumed.
"""
import struct
import numpy as np

MAGIC = b"DLXP"
VERSION = 1
SECTOR = 512                 # same rule and the same reason as dlx.SECTOR
PAL_BYTES = 512              # 256 entries, one GRB555+I word each
HDR_BYTES = 32
FLAG_PALETTE = 1 << 0
FLAG_PALETTE_LAST = 1 << 1


def pack_picture(idx):
    """(H,W) palette indices -> the bytes GVRAM wants, in GVRAM order.

    The ONE place the interleave rule is applied, for the same reason
    `dlx.record_lengths` is the one place the alignment rule is: every caller
    that carries its own copy of a layout rule is a place the layout can drift.
    """
    H, W = idx.shape
    if W % 2:
        raise ValueError(f"packed layout needs an even width, got {W}")
    half = W // 2
    left, right = idx[:, :half], idx[:, half:]
    out = np.empty((H, half, 2), np.uint8)
    out[:, :, 0] = right          # high byte of the word -> page 1 -> col i+128
    out[:, :, 1] = left           # low  byte             -> page 0 -> col i
    return out.reshape(H, half * 2)


def unpack_picture(buf, W, H):
    """The inverse, and the assertion that `pack_picture` is reversible."""
    b = np.frombuffer(buf, np.uint8, W * H).reshape(H, W // 2, 2)
    idx = np.empty((H, W), np.uint8)
    idx[:, W // 2:] = b[:, :, 0]
    idx[:, :W // 2] = b[:, :, 1]
    return idx


def record_bytes(W, H, palette=True):
    n = W * H + (PAL_BYTES if palette else 0)
    if n % SECTOR:
        raise ValueError(f"a {W}x{H} packed record is {n} B, which is not a "
                         f"whole number of {SECTOR} B sectors")
    return n


def write(path, W, H, fps, frames, palette_last=False):
    """`frames` is a sequence of (palette_words_bytes | None, picture_bytes)."""
    frames = list(frames)
    pal_b = PAL_BYTES if frames and frames[0][0] is not None else 0
    pic_b = W * H
    rec_b = record_bytes(W, H, palette=bool(pal_b))
    flags = ((FLAG_PALETTE if pal_b else 0)
             | (FLAG_PALETTE_LAST if palette_last and pal_b else 0))
    hdr = (MAGIC + struct.pack(">HHHHHH", VERSION, flags, W, H, fps, len(frames))
           + struct.pack(">III", rec_b, pal_b, pic_b)
           + struct.pack(">I", SECTOR))
    assert len(hdr) == HDR_BYTES, len(hdr)
    with open(path, "wb") as fh:
        fh.write(hdr + b"\0" * (SECTOR - HDR_BYTES))
        for i, (pw, pic) in enumerate(frames):
            if len(pic) != pic_b or (pal_b and len(pw) != pal_b):
                raise ValueError(f"frame {i}: record parts are the wrong size")
            rec = pic if not pal_b else (pic + pw if palette_last else pw + pic)
            fh.write(rec)
    return rec_b


class DLXP:
    """Reader, and every invariant the format claims, CHECKED rather than read.

    The checks are not defensive coding.  A packed record is written into GVRAM
    and the palette registers with no bounds test anywhere -- the channel has no
    opinion about what it is copying -- so a container whose geometry is a byte
    wrong does not fail, it paints.
    """

    def __init__(self, path):
        b = self.raw = open(path, "rb").read()
        if b[:4] != MAGIC:
            raise ValueError(f"{path}: not a DLXP container")
        (self.version, self.flags, self.W, self.H, self.fps,
         self.nframes) = struct.unpack(">HHHHHH", b[4:16])
        (self.rec_bytes, self.pal_bytes,
         self.pic_bytes, self.off_frm) = struct.unpack(">IIII", b[16:32])
        if self.version != VERSION:
            raise ValueError(f"{path}: DLXP version {self.version}")
        self.has_palette = bool(self.flags & FLAG_PALETTE)
        self.palette_last = bool(self.flags & FLAG_PALETTE_LAST)
        if self.pic_bytes != self.W * self.H:
            raise ValueError(f"{path}: picture is {self.pic_bytes} B for "
                             f"{self.W}x{self.H} -- the packed layout is 1.0 B/px")
        if self.pal_bytes != (PAL_BYTES if self.has_palette else 0):
            raise ValueError(f"{path}: palette section is {self.pal_bytes} B")
        if self.rec_bytes != self.pal_bytes + self.pic_bytes:
            raise ValueError(f"{path}: record is {self.rec_bytes} B, parts are "
                             f"{self.pal_bytes} + {self.pic_bytes}")
        # The whole reason for DLX5 (58.3/59.4), and here it is free rather than
        # a re-encode: the record is a fixed multiple of a sector by geometry.
        if self.off_frm % SECTOR or self.rec_bytes % SECTOR:
            raise ValueError(f"{path}: not sector-aligned -- stream at "
                             f"{self.off_frm}, record {self.rec_bytes}")
        want = self.off_frm + self.nframes * self.rec_bytes
        if len(b) != want:
            raise ValueError(f"{path}: {len(b)} bytes, geometry says {want}")

    def record(self, i):
        o = self.off_frm + i * self.rec_bytes
        return self.raw[o:o + self.rec_bytes]

    def _split(self, i):
        r = self.record(i)
        if not self.has_palette:
            return None, r
        if self.palette_last:
            return r[self.pic_bytes:], r[:self.pic_bytes]
        return r[:self.pal_bytes], r[self.pal_bytes:]

    def palette_words(self, i):
        pw, _ = self._split(i)
        if pw is None:
            raise ValueError("this container carries no palette -- it was made "
                             "with --no-palette, and the palette a player would "
                             "display is not in the file to be read back")
        return np.frombuffer(pw, ">u2").astype(np.uint16)

    def indices(self, i):
        _, pic = self._split(i)
        return unpack_picture(pic, self.W, self.H)

    def palette_rgb(self, i):
        """(256,3) uint8 -- what the DISPLAY produces, not what the encoder meant.

        The palette in a record is already a GRB555+I word, so this is where the
        5-bit hardware quantisation gets charged.  Everything upstream of the
        container is in RGB888 and 61.9's +4.89 dB was quoted there; a player's
        number has to come from here.  Same maths as `dlxload.pack_palette` and
        `tools/bench/verify_frame256.py` -- the shared LSB `I` is a bit in the
        word, so unpacking it needs no choice made.
        """
        w = self.palette_words(i).astype(int)
        f = np.stack([(w >> 6) & 31, (w >> 11) & 31, (w >> 1) & 31], 1)   # R,G,B
        p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
        return p6((f << 1) | (w & 1)[:, None]).astype(np.uint8)

    def render(self, i):
        """(H,W,3) uint8 -- the frame as the display produces it."""
        return self.palette_rgb(i)[self.indices(i)]

    def kbps(self):
        return self.nframes * self.rec_bytes / (self.nframes / self.fps) / 1024