drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Drop SASI on capacity, then find the budget never had the disk in it

Browse Source 
USER DECISION: drop the `sasi` profile. Not on bandwidth -- on capacity. A SASI
volume is 40 MB, and the 22.8 min of unique scene footage on the source Blu-ray
(streams 00000-00201, measured, not recalled) is 146 MiB at the LOWEST rate this
codec makes -- more than the machine's whole 4-unit SASI space. `scsi` is the
only profile now. FINDINGS 32.

Then the user asked whether we were drawing the wrong conclusions about PIO vs
DMA, and we were, more broadly than the question implied. Every CPU figure in
FINDINGS 24-34 is scored against the full 833,333 cycles/frame with nothing
subtracted for moving the bitstream off disk. Debiting the HD63450 cycle-steal
at the long-standing 8 clk/word ESTIMATE, "1 frame of 120 misses" becomes 84 of
120, median 112.4%. PIO at the span rate is 99.8% of the machine. Spans buy
cycles by spending bandwidth and the bandwidth returns as steal, so 31.6's "fits
completely" becomes a worst frame of 114.3%. 10 fps absorbs it: median 93.7%,
1/120. FINDINGS 35. `11_cpu_budget.py` takes --io dma|pio|none, defaults to dma,
and warns if asked for none.

Also landed:
- item 1 done: the cost model checked against the 68000 on a cost-aware
  container, -3.07% to +0.01%, whole-window mean -1.22%. FINDINGS 34.
- item 4 done: the container carries its own 4-byte record alignment (DLX2).
  94/120 record starts were on odd addresses -- an address error, not a slow
  read -- now 0/120 for 16 B/s. Re-encoding reproduces 31.1 exactly. FINDINGS 33.
- a `scsi` window does not fit the 2 MB machine the rig emulates (2.84 MB of
  stream past a 0x200000 ceiling). The gate now verifies 80 of 120 frames and
  SAYS so, and fails loudly when the pass does not complete, instead of
  reporting a phantom 49,005-pixel diff. FINDINGS 36.

Three near-misses this session had one shape: an unobservable run nearly
produced a false finding. stdbuf -oL on any MAME job that prints progress -- a
file is block-buffered too, and a run that is merely finishing looks exactly
like one that is wedged.

check.sh ALL GREEN.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 17:09:47 -07:00
parent 06b98d4b47
commit 7d365b3ff5
12 changed files with 664 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/encoder/dlx.py
+19 -4
View File
@@ -23,8 +23,15 @@ class DLX:
def __init__(self, path):
self.raw = open(path, "rb").read()
b = self.raw
if b[:4] != b"DLX1":
raise ValueError(f"{path}: not a DLX1 container")
# DLX2 pads every frame record up to a 4-byte boundary; DLX1 lays them
# end to end. On a 68000 that is not a slow read but an ADDRESS ERROR
# (FINDINGS 28.3), so the padding is part of the format, not a loader
# convenience -- but DLX1 containers stay readable, because every
# measurement in FINDINGS 28-31 was taken on one.
if b[:4] not in (b"DLX1", b"DLX2"):
raise ValueError(f"{path}: not a DLX container")
self.version = int(b[3:4])
self.aligned = self.version >= 2
(self.W, self.H, self.fps, self.nframes,
self.k1, self.k4) = struct.unpack(">HHHHHH", b[4:16])
off_pal, off_cb1, off_cb4, off_frm = struct.unpack(">IIII", b[16:32])
@@ -41,14 +48,22 @@ class DLX:
self.mode_bytes = (self.nb * 2 + 7) // 8
# frame directory: (offset of the mode header, payload length)
if self.aligned and off_frm % 4:
raise ValueError(f"{path}: DLX2 frame stream starts at {off_frm}, "
f"which is not 4-byte aligned")
self.frames = []
p = off_frm
for _ in range(self.nframes):
(n,) = struct.unpack(">I", b[p:p + 4])
self.frames.append((p + 4, n))
p += 4 + n
if p != len(b):
raise ValueError(f"{path}: {len(b) - p} trailing bytes after "
if self.aligned:
p += -p % 4 # skip the pad to the next record
# The writer does not pad after the LAST record -- nothing follows it --
# so `p` may have advanced past the end by up to 3 bytes there.
slack = len(b) - p
if not (slack == 0 or (self.aligned and -3 <= slack < 0)):
raise ValueError(f"{path}: {slack} trailing bytes after "
f"{self.nframes} frames")
def modes(self, f):
tools/encoder/encode.py
+24 -7
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Encode one scene to the DLX bitstream, at a chosen quality profile.
python3 tools/encoder/encode.py <frames_dir> <out.dlx> [--profile sasi|scsi]
python3 tools/encoder/encode.py <frames_dir> <out.dlx> [--profile scsi]
[--lam N] [--fps 12] [--preview out.png]
[--fixed-lam] [--rc-floor profile|open]
@@ -16,7 +16,7 @@ Container (little-endian is WRONG here -- the 68000 is big-endian, so every
multi-byte field is big-endian and the decoder can read it with a plain move.w):
header, 32 bytes
0 'DLX1' magic
0 'DLX2' magic ('DLX1' = the same, unaligned; still read)
4 u16 width, u16 height
8 u16 fps, u16 nframes
12 u16 k1, u16 k4 codebook sizes
@@ -25,7 +25,9 @@ multi-byte field is big-endian and the decoder can read it with a plain move.w):
20 u32 cb1 offset (k1 * 16 bytes of palette indices)
24 u32 cb4 offset (k4 * 4 bytes)
28 u32 frames offset
then, per frame:
then, per frame, each record starting on a 4-BYTE BOUNDARY (0-3 zero pad
bytes before it; a 68000 takes an address error, not a slow read, on an odd
`move.l` -- FINDINGS 28.3):
u32 payload length, then
ceil(nblocks*2/8) bytes of 2-bit mode headers, MSB-first, block raster order
then payloads in block order: V1 -> 1 byte, V4 -> 4 bytes, RAW -> 16 bytes
@@ -83,7 +85,7 @@ def main():
global _IDX_BYTES
ap = argparse.ArgumentParser()
ap.add_argument("frames_dir"); ap.add_argument("out")
ap.add_argument("--profile", choices=list(RC.PROFILES), default="sasi")
ap.add_argument("--profile", choices=list(RC.PROFILES), default="scsi")
ap.add_argument("--lam", type=float, default=None)
ap.add_argument("--fps", type=int, default=12)
ap.add_argument("--iters", type=int, default=16)
@@ -165,19 +167,34 @@ def main():
off_cb1 = off_pal + len(pal_b)
off_cb4 = off_cb1 + len(cb1_b)
off_frm = off_cb4 + len(cb4_b)
hdr = (b"DLX1" + struct.pack(">HHHHHH", W_, H_, a.fps, len(idx), k1, k4)
# DLX2: every frame record starts on a 4-byte boundary, including the
# first. Payload lengths are arbitrary, so end-to-end records land on odd
# addresses -- and `move.l (a0)+` at an odd address is an ADDRESS ERROR on
# a 68000, not a slow read. It vectors into the IPL and looks exactly like
# an infinite loop (FINDINGS 28.3). tools/bench/prep_dlx.py has been
# realigning at load time; the container now carries it.
tbl_pad = -off_frm % 4
off_frm += tbl_pad
hdr = (b"DLX2" + struct.pack(">HHHHHH", W_, H_, a.fps, len(idx), k1, k4)
+ struct.pack(">IIII", off_pal, off_cb1, off_cb4, off_frm))
assert len(hdr) == 32, len(hdr)
frm_pad = 0
with open(a.out, "wb") as fh:
fh.write(hdr); fh.write(pal_b); fh.write(cb1_b); fh.write(cb4_b)
for p in frames:
fh.write(b"\0" * tbl_pad)
for i, p in enumerate(frames):
fh.write(struct.pack(">I", len(p))); fh.write(p)
if i + 1 < len(frames): # nothing follows the last record
n = -(4 + len(p)) % 4
fh.write(b"\0" * n); frm_pad += n
total = os.path.getsize(a.out)
vid = sum(len(p) + 4 for p in frames)
vid = sum(len(p) + 4 for p in frames) + frm_pad
print(f" wrote {a.out}: {total} B "
f"(header+tables {total-vid} B, video {vid} B)")
print(f" DLX2 4-byte record alignment: {frm_pad} B over {len(frames)} frames "
f"({frm_pad/len(frames):.2f} B/frame = {frm_pad/len(frames)*a.fps:.0f} B/s)")
print(f" {vid/len(idx):.0f} B/frame -> {vid/len(idx)*a.fps/1024:.1f} KB/s video"
f" + {RC.AUDIO_KBPS} KB/s audio = {vid/len(idx)*a.fps/1024+RC.AUDIO_KBPS:.1f} KB/s")
print(f" PSNR {r['psnr']:.2f} dB palette ceiling {r['pal']:.2f} dB "
tools/encoder/ratectl.py
+17 -10
View File
@@ -49,18 +49,25 @@ import vq_hybrid as H
# ~216% of the frame budget on a 68000 and even LZ4 is ~54%. See FINDINGS 17.
# The rates below are therefore RAW payload, no entropy coding.
#
# The two profiles are the SAME codec, decoder and bitstream -- only `lam` differs.
# `lam` here is a FLOOR, not a setting: encode.py rate-controls by default and
# bisects lam per frame in [lam, LAM_CLIFF] to keep under `kbps`. The floor is
# what a quiet frame is allowed to spend, so rate control can only ever spend
# less than session 5's fixed-lam encoder did. FINDINGS 27.
#
# THE `sasi` PROFILE IS GONE (session 9, USER DECISION). It was dropped on
# CAPACITY, not bandwidth: a SASI volume on this machine tops out at 40 MB, and
# the 22.8 minutes of unique scene footage on the source Blu-ray is 147 MB even
# at the 110 KB/s the profile targeted -- more than the whole 4-unit SASI
# address space, with nothing left for Human68k or the game. FINDINGS 32.
#
# That leaves ONE profile, which is also the end of the two-quality-mode
# decision of session 2. The 110 KB/s RATE POINT may still return under another
# name: a 1x SCSI CD-ROM sustains ~150 KB/s, below this profile, and CD-ROM is
# the only period medium with the capacity for the span-heavy stream. That is
# deferred to the blocked disk benchmark and the DMA-vs-PIO check (docs/
# BENCHMARK.md, FINDINGS 29.5), because every bandwidth figure here is folklore
# until one of them lands.
PROFILES = {
"sasi": dict(kbps=110, lam=60.0, k1=256, k4=256,
desc="stock 10MHz ACE/EXPERT, SASI",
quality="36.9 dB on 00020 / 29.6 dB on 00146 / 27.2 dB on the "
"Singe window at 109.5 KB/s (session 5's fixed lam "
"gave 27.8 dB there, but at 137.4 KB/s)",
util="~105 KB/s = 35% of the pessimistic 300 KB/s SASI figure"),
"scsi": dict(kbps=280, lam=10.0, k1=256, k4=256,
desc="Super/XVI, or CZ-6BS1 board in a 10MHz machine",
quality="39.4 dB on 00020 / 32.3 dB on 00146 / 29.9 dB on the "
@@ -82,9 +89,9 @@ PROFILES = {
LAM_CLIFF = 800.0
# Ceiling on the CYCLE search. mu prices a cycle in the same units lam prices a
# byte, so the scale that matters is set by their ratio: at the `sasi` floor of
# lam=60, mu=0.2 makes a V1 block's 300 cycles cost what its 1 payload byte
# costs. MU_CLIFF=100 is three decades past that: a V1 block priced at 30,000
# byte, so the scale that matters is set by their ratio: at a lam floor of 60
# (the retired `sasi` profile's, and the highest this codec has shipped),
# mu=0.2 makes a V1 block's 300 cycles cost what its 1 payload byte costs. MU_CLIFF=100 is three decades past that: a V1 block priced at 30,000
# distortion units.
#
# It does NOT freeze the picture, and that is the point. At MU_CLIFF a block