Drop SASI on capacity, then find the budget never had the disk in it
USER DECISION: drop the `sasi` profile. Not on bandwidth -- on capacity. A SASI volume is 40 MB, and the 22.8 min of unique scene footage on the source Blu-ray (streams 00000-00201, measured, not recalled) is 146 MiB at the LOWEST rate this codec makes -- more than the machine's whole 4-unit SASI space. `scsi` is the only profile now. FINDINGS 32. Then the user asked whether we were drawing the wrong conclusions about PIO vs DMA, and we were, more broadly than the question implied. Every CPU figure in FINDINGS 24-34 is scored against the full 833,333 cycles/frame with nothing subtracted for moving the bitstream off disk. Debiting the HD63450 cycle-steal at the long-standing 8 clk/word ESTIMATE, "1 frame of 120 misses" becomes 84 of 120, median 112.4%. PIO at the span rate is 99.8% of the machine. Spans buy cycles by spending bandwidth and the bandwidth returns as steal, so 31.6's "fits completely" becomes a worst frame of 114.3%. 10 fps absorbs it: median 93.7%, 1/120. FINDINGS 35. `11_cpu_budget.py` takes --io dma|pio|none, defaults to dma, and warns if asked for none. Also landed: - item 1 done: the cost model checked against the 68000 on a cost-aware container, -3.07% to +0.01%, whole-window mean -1.22%. FINDINGS 34. - item 4 done: the container carries its own 4-byte record alignment (DLX2). 94/120 record starts were on odd addresses -- an address error, not a slow read -- now 0/120 for 16 B/s. Re-encoding reproduces 31.1 exactly. FINDINGS 33. - a `scsi` window does not fit the 2 MB machine the rig emulates (2.84 MB of stream past a 0x200000 ceiling). The gate now verifies 80 of 120 frames and SAYS so, and fails loudly when the pass does not complete, instead of reporting a phantom 49,005-pixel diff. FINDINGS 36. Three near-misses this session had one shape: an unobservable run nearly produced a false finding. stdbuf -oL on any MAME job that prints progress -- a file is block-buffered too, and a run that is merely finishing looks exactly like one that is wedged. check.sh ALL GREEN. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
@@ -16,7 +16,7 @@ the mode headers would exploit.
|
||||
RAW 16 literal palette indices -- the escape that makes lam=0 pixel-exact
|
||||
|
||||
Usage: python3 tools/analysis/08_mode_map.py <frames_dir> <out.webm>
|
||||
[--profile sasi|scsi] [--scale N] [--lossless] [--fixed-lam]
|
||||
[--profile scsi] [--scale N] [--lossless] [--fixed-lam]
|
||||
|
||||
--fixed-lam renders the pre-session-6 encoder (no rate control) instead.
|
||||
Output format follows the extension. Prefer .webm: GIF re-quantises to 256
|
||||
@@ -45,7 +45,7 @@ def main():
|
||||
if "--scale" in sys.argv:
|
||||
SCALE = int(sys.argv[sys.argv.index("--scale")+1])
|
||||
prof = RC.PROFILES[sys.argv[sys.argv.index("--profile")+1]
|
||||
if "--profile" in sys.argv else "sasi"]
|
||||
if "--profile" in sys.argv else "scsi"]
|
||||
m = H.build(src, k1=prof["k1"], k4=prof["k4"])
|
||||
# Rate-controlled by default, so the map shows the mode decisions that
|
||||
# actually ship. --fixed-lam renders the pre-session-6 encoder instead;
|
||||
|
||||
@@ -23,6 +23,8 @@ sys.path.insert(0, "tools/encoder")
|
||||
import numpy as np
|
||||
from dlx import DLX
|
||||
import vq_hybrid as H
|
||||
import ratectl as RC
|
||||
RC_AUDIO_BPS = RC.AUDIO_KBPS * 1024
|
||||
|
||||
# Machine clocks, confirmed from MAME 0.277 src/mame/sharp/x68k.cpp:1133/1194/
|
||||
# 1200 -- not recalled. x68000 and x68ksupr are BOTH 40_MHz_XTAL/4 = 10 MHz;
|
||||
@@ -45,6 +47,19 @@ ap.add_argument("container", nargs="?",
|
||||
ap.add_argument("--machine", default="stock", choices=list(CLOCKS),
|
||||
help="which X68000's clock to budget against (default stock)")
|
||||
ap.add_argument("--fps", type=float, default=FPS)
|
||||
# FINDINGS 35: the frame budget has never had the disk in it. The bitstream has
|
||||
# to be moved off SCSI into the ring buffer, and on this machine that costs CPU
|
||||
# whether it is DMA (the HD63450 cycle-steals) or PIO (the 68000 moves every
|
||||
# byte). Default ON, because scoring a decoder against a budget that assumes the
|
||||
# data arrives for free is exactly the mistake 35 was raised to stop.
|
||||
ap.add_argument("--io", default="dma", choices=["dma", "pio", "none"],
|
||||
help="how the bitstream reaches RAM (default dma)")
|
||||
ap.add_argument("--dma-clocks-per-word", type=float, default=8.0,
|
||||
help="HD63450 cycle-steal. ESTIMATE from FINDINGS 5, NEVER "
|
||||
"MEASURED, and the most load-bearing unmeasured number "
|
||||
"in the project (FINDINGS 35.3)")
|
||||
ap.add_argument("--pio-clocks-per-byte", type=float, default=12.0,
|
||||
help="hand-derived floor for a 68000 register-to-RAM copy")
|
||||
a = ap.parse_args()
|
||||
CPUHZ = CLOCKS[a.machine] * 1e6
|
||||
FPS = a.fps
|
||||
@@ -54,14 +69,35 @@ if not os.path.exists(a.container):
|
||||
|
||||
d = DLX(a.container)
|
||||
|
||||
# --- what the transfer costs, from the container's own byte rate
|
||||
vid_bps = sum(n + 4 for (_, n) in d.frames) / d.nframes * d.fps
|
||||
io_bps = vid_bps + RC_AUDIO_BPS
|
||||
if a.io == "dma":
|
||||
io_cycles_per_s = (io_bps / 2) * a.dma_clocks_per_word
|
||||
elif a.io == "pio":
|
||||
io_cycles_per_s = io_bps * a.pio_clocks_per_byte
|
||||
else:
|
||||
io_cycles_per_s = 0.0
|
||||
io_pct = 100 * io_cycles_per_s / CPUHZ
|
||||
FRAME_NET = FRAME * (1 - io_pct / 100)
|
||||
|
||||
modes = [d.modes(f) for f in range(d.nframes)]
|
||||
cyc = np.array([cycles(m) for m in modes])
|
||||
pct = 100 * cyc / FRAME
|
||||
pct = 100 * cyc / FRAME_NET
|
||||
ns = np.array([100 * (m != 0).mean() for m in modes])
|
||||
|
||||
print(f"{a.container}: {d.nframes} frames, {d.nb} blocks/frame")
|
||||
print(f"budget: {a.machine} @ {CLOCKS[a.machine]:.2f} MHz, {FPS:g} fps "
|
||||
f"-> {FRAME:,.0f} cycles/frame")
|
||||
print(f" I/O ({a.io}): {io_bps/1024:.1f} KB/s costs {io_pct:.1f}% of the CPU "
|
||||
f"-> {FRAME_NET:,.0f} cycles/frame left for decoding")
|
||||
if a.io == "dma":
|
||||
print(f" {a.dma_clocks_per_word:g} clocks/word is an ESTIMATE (FINDINGS 5), "
|
||||
f"never measured -- see FINDINGS 35.3")
|
||||
elif a.io == "none":
|
||||
print(" WARNING: --io none scores the decoder as if the disk were free. "
|
||||
"That is the\n premise FINDINGS 35 overturned; every 'N frames miss' "
|
||||
"figure before session 9\n was computed this way.")
|
||||
if a.machine != "stock":
|
||||
print(" (derived: scaled by clock from cycles measured on the 10 MHz core.\n"
|
||||
" MAME 0.277 marks x68ksupr/x68kxvi/x68030 MACHINE_NOT_WORKING, so\n"
|
||||
@@ -100,10 +136,11 @@ if a.machine == "stock" and a.fps == 12:
|
||||
f"(optimistic by {np.median(pct)/np.median(old):.2f}x at the median)")
|
||||
|
||||
miss = pct > 100
|
||||
print(f"\nframes that do NOT fit {FRAME:,.0f} cycles: {miss.sum()}/{d.nframes} "
|
||||
print(f"\nframes that do NOT fit {FRAME_NET:,.0f} cycles: {miss.sum()}/{d.nframes} "
|
||||
f"({100*miss.mean():.0f}%)")
|
||||
print(f" sustainable framerate if EVERY frame must fit: "
|
||||
f"{CPUHZ/cyc.max():.1f} fps; at the mean frame {CPUHZ/cyc.mean():.1f} fps")
|
||||
f"{CPUHZ*(1-io_pct/100)/cyc.max():.1f} fps; at the mean frame "
|
||||
f"{CPUHZ*(1-io_pct/100)/cyc.mean():.1f} fps")
|
||||
if miss.any():
|
||||
print(f" worst {pct.max():.1f}% -- {(pct.max()-100)/100*1000/FPS:.0f} ms late "
|
||||
f"on an {1000/FPS:.0f} ms frame")
|
||||
@@ -116,5 +153,7 @@ print(f"\nwhere the cycles go, over the whole window:")
|
||||
for k, n in enumerate(("SKIP", "V1", "V4", "RAW")):
|
||||
print(f" {n:<5} {100*tot[k]/tot.sum():5.1f}% of blocks "
|
||||
f"{100*spend[k]/spend.sum():5.1f}% of the cycles")
|
||||
print(f"\nV4 is {C_V4/C_V1:.2f}x a V1 block for {4}x the payload bytes -- the mode "
|
||||
f"decision\nin vq_hybrid.py charges it the bytes but not the cycles.")
|
||||
print(f"\nV4 is {C_V4/C_V1:.2f}x a V1 block for {4}x the payload bytes. Since "
|
||||
f"session 8 the mode\ndecision charges it BOTH (decide(ctx, lam, mu), "
|
||||
f"FINDINGS 31), which is why V4 is now\nthe rarest non-SKIP mode here -- "
|
||||
f"a byte-rich profile buys its way out to RAW instead.")
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""What does fitting the CPU budget cost in quality? (session 8, lever B)
|
||||
|
||||
python3 tools/analysis/13_cpu_ratectl.py [frames_dir] [--profiles sasi,scsi]
|
||||
python3 tools/analysis/13_cpu_ratectl.py [frames_dir] [--profiles scsi]
|
||||
|
||||
Session 6 made the BYTE budget a ceiling by bisecting `lam` per frame. FINDINGS
|
||||
28 then showed the binding budget is CYCLES, not bytes, and that the mode
|
||||
@@ -28,7 +28,7 @@ import vq as VQ, vq_hybrid as H, ratectl as RC
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("frames_dir", nargs="?", default="tmp/fr_singe")
|
||||
ap.add_argument("--profiles", default="sasi,scsi")
|
||||
ap.add_argument("--profiles", default="scsi")
|
||||
ap.add_argument("--fps", type=int, default=12)
|
||||
ap.add_argument("--cache", default=None, help="pickle of H.build (auto by dir)")
|
||||
a = ap.parse_args()
|
||||
@@ -97,5 +97,7 @@ for name in a.profiles.split(","):
|
||||
print()
|
||||
|
||||
print("FINDINGS 28.7: re-coding every non-SKIP block as V1 is the floor the "
|
||||
"CURRENT mode set\nallows, and it still misses 11 frames at sasi / 12 at "
|
||||
"scsi. Misses above that floor\nare item 4 (spans), not item 1.")
|
||||
"CURRENT mode set\nallows, and it still missed 11 frames at the retired "
|
||||
"110 KB/s profile / 12 at scsi.\nMisses above that floor are spans, not "
|
||||
"the mode decision -- and 31.3 showed the\nfloor itself was too "
|
||||
"pessimistic, because the real decision can move a block to SKIP.")
|
||||
|
||||
+28
-5
@@ -57,17 +57,40 @@ echo "--- session 7: 68000 decoder is pixel-exact (FINDINGS 28) ---"
|
||||
# 68000 code, every block mode, full temporal recursion. A SKIP block is a claim
|
||||
# about the previous frame still being on screen, so the last frame is only
|
||||
# right if all 120 were.
|
||||
DLX=tmp/rc_fr_singe_sasi_rcprofile.dlx
|
||||
[ -f "$DLX" ] || python3 tools/encoder/encode.py tmp/fr_singe "$DLX" --profile sasi
|
||||
# The gate container is the CURRENT default encode: scsi (the only profile left
|
||||
# after session 9 dropped sasi on capacity, FINDINGS 32), cost-aware mode
|
||||
# decision on, DLX2 4-byte-aligned records. It is also the heavier stream --
|
||||
# 43% RAW against sasi's 10% -- so it exercises the decoder harder than the
|
||||
# session-7 container this gate used to run on.
|
||||
DLX=tmp/rc_fr_singe_scsi_cpufit.dlx
|
||||
[ -f "$DLX" ] || python3 tools/encoder/encode.py tmp/fr_singe "$DLX" --profile scsi
|
||||
python3 tools/bench/prep_dlx.py "$DLX" > tmp/prep_dlx.log
|
||||
# The rig loads the whole stream into a 2 MB machine, so a scsi window does not
|
||||
# fit and prep_dlx truncates it. Verify against exactly the prefix it emitted.
|
||||
NF=$(sed -n 's/.*nframes=\([0-9]*\),.*/\1/p' tmp/decode_meta.lua)
|
||||
grep -a "TRUNCATED" tmp/prep_dlx.log || true
|
||||
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s > /dev/null
|
||||
mkdir -p tmp/snap_decode
|
||||
rm -f tmp/snap_decode/x68000/*.png
|
||||
( cd tmp && DLX_VERIFY_ONLY=1 SDL_VIDEODRIVER=dummy timeout -k 5 300 mame x68000 \
|
||||
# stdbuf -oL: a FILE is block-buffered too, so without it a long MAME run is
|
||||
# unobservable until it exits and a run that is merely finishing looks exactly
|
||||
# like one that is wedged (FINDINGS 34.1).
|
||||
# -seconds_to_run must cover the WHOLE sequential pass. The scsi container is
|
||||
# 2.7x the payload of the session-7 one this gate used to run on, and at 20 s
|
||||
# the pass was truncated -- MAME exited mid-decode and verify_decode.py then
|
||||
# compared a partially drawn screen and reported 49,005 differing pixels, which
|
||||
# reads as a decoder bug and is not one.
|
||||
( cd tmp && DLX_VERIFY_ONLY=1 SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 300 mame x68000 \
|
||||
-bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
|
||||
-autoboot_script ../tools/bench/decode.lua \
|
||||
-snapshot_directory ./snap_decode -snapview native -seconds_to_run 20 \
|
||||
-snapshot_directory ./snap_decode -snapview native -seconds_to_run 45 \
|
||||
> decode_check.log 2>&1 )
|
||||
python3 tools/bench/verify_decode.py "$DLX"
|
||||
# A truncated run must fail as a truncated run. Without this the only symptom is
|
||||
# a pixel diff against a half-drawn frame.
|
||||
grep -q "snapshot taken" tmp/decode_check.log || {
|
||||
echo "FAIL: the 68000 sequential pass did not complete -- no snapshot marker."
|
||||
echo " Raise -seconds_to_run; the pass needs the whole container decoded."
|
||||
tail -5 tmp/decode_check.log; exit 1; }
|
||||
python3 tools/bench/verify_decode.py "$DLX" --nframes "$NF"
|
||||
|
||||
echo "ALL GREEN"
|
||||
|
||||
+41
-4
@@ -29,9 +29,26 @@ sys.path.insert(0, "tools/encoder")
|
||||
import numpy as np
|
||||
from dlx import DLX
|
||||
|
||||
# The harness loads the WHOLE container into emulated RAM at STREAM=0x30000 and
|
||||
# the target is a stock 2 MB machine, so there is a hard ceiling on how much of
|
||||
# a stream can be verified in one pass. The shipping player streams from disk
|
||||
# into a ring buffer and has no such limit; this is a property of the test rig.
|
||||
# A `scsi` window overruns it -- 2.84 MB of stream ends at 0x2E591C, 940 KB past
|
||||
# the 0x200000 top of RAM -- so the frame list is truncated to what fits and the
|
||||
# truncation is announced. Verifying a prefix is still a real test: SKIP blocks
|
||||
# make every frame a claim about the one before it.
|
||||
STREAM_BASE = 0x30000
|
||||
RAM_TOP = 0x200000
|
||||
MARGIN = 0x8000 # stack, flags, codebooks live below STREAM_BASE
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("container")
|
||||
ap.add_argument("--out", default="tmp/decode")
|
||||
ap.add_argument("--ram", type=lambda v: int(v, 0), default=RAM_TOP,
|
||||
help="top of emulated RAM (default 0x200000, a stock 2 MB machine)")
|
||||
ap.add_argument("--all-frames", action="store_true",
|
||||
help="do NOT truncate to what fits in RAM (the loader will write "
|
||||
"past the top of memory and the decoder will read garbage)")
|
||||
a = ap.parse_args()
|
||||
|
||||
d = DLX(a.container)
|
||||
@@ -65,12 +82,24 @@ dark = int(((render(I).astype(int)) ** 2).sum(1).argmin())
|
||||
# so this loader realigns it; the encoder should carry the padding itself
|
||||
# (FINDINGS 28.3). It costs at most 3 bytes per frame -- 36 B/s at 12fps,
|
||||
# against a 110 KB/s budget.
|
||||
budget = a.ram - STREAM_BASE - MARGIN
|
||||
stream, rec_off, pad = bytearray(), [], 0
|
||||
dropped = 0
|
||||
for (o, n) in d.frames:
|
||||
while len(stream) % 4:
|
||||
stream += b"\0"; pad += 1
|
||||
if not a.all_frames and len(stream) + 4 + n > budget:
|
||||
dropped = d.nframes - len(rec_off)
|
||||
break
|
||||
rec_off.append(len(stream))
|
||||
stream += n.to_bytes(4, "big") + d.raw[o:o + n]
|
||||
NFRAMES = len(rec_off)
|
||||
if dropped:
|
||||
print(f" TRUNCATED: {NFRAMES}/{d.nframes} frames fit in RAM "
|
||||
f"(stream budget {budget:,} B at 0x{STREAM_BASE:X} under a "
|
||||
f"{a.ram/1024/1024:.0f} MB machine); {dropped} frames dropped.\n"
|
||||
f" This is the TEST RIG's limit, not the player's -- the player "
|
||||
f"streams into a ring buffer.")
|
||||
|
||||
# Synthetic single-mode frames. No real frame is all one mode, but the mix is
|
||||
# exactly what the "76.6% x non-SKIP fraction" model of FINDINGS 24.5 assumes
|
||||
@@ -86,7 +115,7 @@ for name, mo, per in (("all-SKIP", 0, 0), ("all-V1", 1, 1),
|
||||
stream += (d.mode_bytes + d.nb * per).to_bytes(4, "big") + hdr + bytes(d.nb * per)
|
||||
|
||||
# --- timing anchors: the distribution, not its mean (FINDINGS 25.6's lesson)
|
||||
ns = np.array([100 * (d.modes(i) != 0).mean() for i in range(d.nframes)])
|
||||
ns = np.array([100 * (d.modes(i) != 0).mean() for i in range(NFRAMES)])
|
||||
order = np.argsort(ns)
|
||||
pick = {
|
||||
"min non-SKIP %.1f%%" % ns[order[0]]: int(order[0]),
|
||||
@@ -104,7 +133,7 @@ open(a.out + "_data.bin", "wb").write(blob)
|
||||
|
||||
with open(a.out + "_meta.lua", "w") as fh:
|
||||
fh.write("-- generated by tools/bench/prep_dlx.py -- do not edit\nreturn {\n")
|
||||
fh.write(f" W={d.W}, H={d.H}, fps={d.fps}, nframes={d.nframes},\n")
|
||||
fh.write(f" W={d.W}, H={d.H}, fps={d.fps}, nframes={NFRAMES},\n")
|
||||
fh.write(f" k1={d.k1}, k4={d.k4}, dark={dark},\n")
|
||||
fh.write(f" cb1_len={cb1.nbytes}, cb4_len={cb4.nbytes}, pal_len={palb.nbytes},\n")
|
||||
fh.write(f" stream_len={len(stream)},\n")
|
||||
@@ -119,5 +148,13 @@ print(f" cb1 {cb1.nbytes} B + cb4 {cb4.nbytes} B expanded, palette {palb.nbytes
|
||||
print(f" non-SKIP blocks/frame: median {np.median(ns):.1f}% "
|
||||
f"p90 {np.percentile(ns,90):.1f}% max {ns.max():.1f}%")
|
||||
print(f" darkest palette entry: index {dark} -> {tuple(render(I)[dark])}")
|
||||
print(f" 4-byte record alignment cost {pad} B over {d.nframes} frames "
|
||||
f"({pad / d.nframes:.2f} B/frame = {pad / d.nframes * d.fps:.0f} B/s)")
|
||||
# A DLX2 container already carries this padding (FINDINGS 28.3 closed, session
|
||||
# 9), so the realignment above re-derives bytes that were already there and the
|
||||
# loader is doing no work. On a DLX1 container it is load-bearing: 94 of 120
|
||||
# record starts land on odd addresses, and each one is an address error.
|
||||
src_bad = sum(1 for (o, _) in d.frames[:NFRAMES] if (o - 4) % 4)
|
||||
print(f" 4-byte record alignment cost {pad} B over {NFRAMES} frames "
|
||||
f"({pad / NFRAMES:.2f} B/frame = {pad / NFRAMES * d.fps:.0f} B/s)")
|
||||
print(f" source container is DLX{d.version}: {src_bad}/{NFRAMES} record starts "
|
||||
f"unaligned" + (" -- this loader is what makes it decodable"
|
||||
if src_bad else " -- the container carries its own padding"))
|
||||
|
||||
@@ -22,11 +22,20 @@ from dlx import DLX
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("container")
|
||||
ap.add_argument("--snap", default="tmp/snap_decode")
|
||||
# The harness can only load as much of a container as fits in the emulated
|
||||
# machine's RAM, so it may have decoded a PREFIX (tools/bench/prep_dlx.py
|
||||
# --ram). Compare against the same prefix, or the reference runs ahead of the
|
||||
# 68000 and reports a mismatch that is an artefact of the rig.
|
||||
ap.add_argument("--nframes", type=int, default=None,
|
||||
help="frames the 68000 actually decoded (default: all)")
|
||||
a = ap.parse_args()
|
||||
|
||||
d = DLX(a.container)
|
||||
NF = a.nframes if a.nframes is not None else d.nframes
|
||||
if NF > d.nframes:
|
||||
sys.exit(f"--nframes {NF} exceeds the container's {d.nframes}")
|
||||
canvas = np.zeros((d.H, d.W), np.uint8)
|
||||
for f in range(d.nframes):
|
||||
for f in range(NF):
|
||||
d.paint(canvas, f)
|
||||
|
||||
pal = d.pal.astype(int)
|
||||
@@ -52,7 +61,7 @@ else:
|
||||
bad = diff.any(2)
|
||||
by, bx = np.where(bad)
|
||||
blocks = sorted(set(zip((by//4).tolist(), (bx//4).tolist())))
|
||||
fail.append(f"3. frame {d.nframes-1} not pixel-exact: {bad.sum()} px in "
|
||||
fail.append(f"3. frame {NF-1} not pixel-exact: {bad.sum()} px in "
|
||||
f"{len(blocks)} blocks differ, maxdiff {diff.max()}; "
|
||||
f"first block (by={blocks[0][0]}, bx={blocks[0][1]})")
|
||||
|
||||
@@ -60,7 +69,7 @@ for x in fail:
|
||||
print("FAIL " + x)
|
||||
if fail:
|
||||
sys.exit(1)
|
||||
print(f"OK {d.nframes} frames decoded on the 68000, final frame pixel-exact "
|
||||
print(f"OK {NF} frames decoded on the 68000, final frame pixel-exact "
|
||||
f"against tools/encoder/dlx.py")
|
||||
print(f" {d.W}x{d.H}, {d.nb} blocks/frame, k1={d.k1} k4={d.k4}, "
|
||||
f"all four block modes exercised")
|
||||
|
||||
+19
-4
@@ -23,8 +23,15 @@ class DLX:
|
||||
def __init__(self, path):
|
||||
self.raw = open(path, "rb").read()
|
||||
b = self.raw
|
||||
if b[:4] != b"DLX1":
|
||||
raise ValueError(f"{path}: not a DLX1 container")
|
||||
# DLX2 pads every frame record up to a 4-byte boundary; DLX1 lays them
|
||||
# end to end. On a 68000 that is not a slow read but an ADDRESS ERROR
|
||||
# (FINDINGS 28.3), so the padding is part of the format, not a loader
|
||||
# convenience -- but DLX1 containers stay readable, because every
|
||||
# measurement in FINDINGS 28-31 was taken on one.
|
||||
if b[:4] not in (b"DLX1", b"DLX2"):
|
||||
raise ValueError(f"{path}: not a DLX container")
|
||||
self.version = int(b[3:4])
|
||||
self.aligned = self.version >= 2
|
||||
(self.W, self.H, self.fps, self.nframes,
|
||||
self.k1, self.k4) = struct.unpack(">HHHHHH", b[4:16])
|
||||
off_pal, off_cb1, off_cb4, off_frm = struct.unpack(">IIII", b[16:32])
|
||||
@@ -41,14 +48,22 @@ class DLX:
|
||||
self.mode_bytes = (self.nb * 2 + 7) // 8
|
||||
|
||||
# frame directory: (offset of the mode header, payload length)
|
||||
if self.aligned and off_frm % 4:
|
||||
raise ValueError(f"{path}: DLX2 frame stream starts at {off_frm}, "
|
||||
f"which is not 4-byte aligned")
|
||||
self.frames = []
|
||||
p = off_frm
|
||||
for _ in range(self.nframes):
|
||||
(n,) = struct.unpack(">I", b[p:p + 4])
|
||||
self.frames.append((p + 4, n))
|
||||
p += 4 + n
|
||||
if p != len(b):
|
||||
raise ValueError(f"{path}: {len(b) - p} trailing bytes after "
|
||||
if self.aligned:
|
||||
p += -p % 4 # skip the pad to the next record
|
||||
# The writer does not pad after the LAST record -- nothing follows it --
|
||||
# so `p` may have advanced past the end by up to 3 bytes there.
|
||||
slack = len(b) - p
|
||||
if not (slack == 0 or (self.aligned and -3 <= slack < 0)):
|
||||
raise ValueError(f"{path}: {slack} trailing bytes after "
|
||||
f"{self.nframes} frames")
|
||||
|
||||
def modes(self, f):
|
||||
|
||||
+24
-7
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Encode one scene to the DLX bitstream, at a chosen quality profile.
|
||||
|
||||
python3 tools/encoder/encode.py <frames_dir> <out.dlx> [--profile sasi|scsi]
|
||||
python3 tools/encoder/encode.py <frames_dir> <out.dlx> [--profile scsi]
|
||||
[--lam N] [--fps 12] [--preview out.png]
|
||||
[--fixed-lam] [--rc-floor profile|open]
|
||||
|
||||
@@ -16,7 +16,7 @@ Container (little-endian is WRONG here -- the 68000 is big-endian, so every
|
||||
multi-byte field is big-endian and the decoder can read it with a plain move.w):
|
||||
|
||||
header, 32 bytes
|
||||
0 'DLX1' magic
|
||||
0 'DLX2' magic ('DLX1' = the same, unaligned; still read)
|
||||
4 u16 width, u16 height
|
||||
8 u16 fps, u16 nframes
|
||||
12 u16 k1, u16 k4 codebook sizes
|
||||
@@ -25,7 +25,9 @@ multi-byte field is big-endian and the decoder can read it with a plain move.w):
|
||||
20 u32 cb1 offset (k1 * 16 bytes of palette indices)
|
||||
24 u32 cb4 offset (k4 * 4 bytes)
|
||||
28 u32 frames offset
|
||||
then, per frame:
|
||||
then, per frame, each record starting on a 4-BYTE BOUNDARY (0-3 zero pad
|
||||
bytes before it; a 68000 takes an address error, not a slow read, on an odd
|
||||
`move.l` -- FINDINGS 28.3):
|
||||
u32 payload length, then
|
||||
ceil(nblocks*2/8) bytes of 2-bit mode headers, MSB-first, block raster order
|
||||
then payloads in block order: V1 -> 1 byte, V4 -> 4 bytes, RAW -> 16 bytes
|
||||
@@ -83,7 +85,7 @@ def main():
|
||||
global _IDX_BYTES
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("frames_dir"); ap.add_argument("out")
|
||||
ap.add_argument("--profile", choices=list(RC.PROFILES), default="sasi")
|
||||
ap.add_argument("--profile", choices=list(RC.PROFILES), default="scsi")
|
||||
ap.add_argument("--lam", type=float, default=None)
|
||||
ap.add_argument("--fps", type=int, default=12)
|
||||
ap.add_argument("--iters", type=int, default=16)
|
||||
@@ -165,19 +167,34 @@ def main():
|
||||
off_cb1 = off_pal + len(pal_b)
|
||||
off_cb4 = off_cb1 + len(cb1_b)
|
||||
off_frm = off_cb4 + len(cb4_b)
|
||||
hdr = (b"DLX1" + struct.pack(">HHHHHH", W_, H_, a.fps, len(idx), k1, k4)
|
||||
# DLX2: every frame record starts on a 4-byte boundary, including the
|
||||
# first. Payload lengths are arbitrary, so end-to-end records land on odd
|
||||
# addresses -- and `move.l (a0)+` at an odd address is an ADDRESS ERROR on
|
||||
# a 68000, not a slow read. It vectors into the IPL and looks exactly like
|
||||
# an infinite loop (FINDINGS 28.3). tools/bench/prep_dlx.py has been
|
||||
# realigning at load time; the container now carries it.
|
||||
tbl_pad = -off_frm % 4
|
||||
off_frm += tbl_pad
|
||||
hdr = (b"DLX2" + struct.pack(">HHHHHH", W_, H_, a.fps, len(idx), k1, k4)
|
||||
+ struct.pack(">IIII", off_pal, off_cb1, off_cb4, off_frm))
|
||||
assert len(hdr) == 32, len(hdr)
|
||||
|
||||
frm_pad = 0
|
||||
with open(a.out, "wb") as fh:
|
||||
fh.write(hdr); fh.write(pal_b); fh.write(cb1_b); fh.write(cb4_b)
|
||||
for p in frames:
|
||||
fh.write(b"\0" * tbl_pad)
|
||||
for i, p in enumerate(frames):
|
||||
fh.write(struct.pack(">I", len(p))); fh.write(p)
|
||||
if i + 1 < len(frames): # nothing follows the last record
|
||||
n = -(4 + len(p)) % 4
|
||||
fh.write(b"\0" * n); frm_pad += n
|
||||
|
||||
total = os.path.getsize(a.out)
|
||||
vid = sum(len(p) + 4 for p in frames)
|
||||
vid = sum(len(p) + 4 for p in frames) + frm_pad
|
||||
print(f" wrote {a.out}: {total} B "
|
||||
f"(header+tables {total-vid} B, video {vid} B)")
|
||||
print(f" DLX2 4-byte record alignment: {frm_pad} B over {len(frames)} frames "
|
||||
f"({frm_pad/len(frames):.2f} B/frame = {frm_pad/len(frames)*a.fps:.0f} B/s)")
|
||||
print(f" {vid/len(idx):.0f} B/frame -> {vid/len(idx)*a.fps/1024:.1f} KB/s video"
|
||||
f" + {RC.AUDIO_KBPS} KB/s audio = {vid/len(idx)*a.fps/1024+RC.AUDIO_KBPS:.1f} KB/s")
|
||||
print(f" PSNR {r['psnr']:.2f} dB palette ceiling {r['pal']:.2f} dB "
|
||||
|
||||
+17
-10
@@ -49,18 +49,25 @@ import vq_hybrid as H
|
||||
# ~216% of the frame budget on a 68000 and even LZ4 is ~54%. See FINDINGS 17.
|
||||
# The rates below are therefore RAW payload, no entropy coding.
|
||||
#
|
||||
# The two profiles are the SAME codec, decoder and bitstream -- only `lam` differs.
|
||||
# `lam` here is a FLOOR, not a setting: encode.py rate-controls by default and
|
||||
# bisects lam per frame in [lam, LAM_CLIFF] to keep under `kbps`. The floor is
|
||||
# what a quiet frame is allowed to spend, so rate control can only ever spend
|
||||
# less than session 5's fixed-lam encoder did. FINDINGS 27.
|
||||
#
|
||||
# THE `sasi` PROFILE IS GONE (session 9, USER DECISION). It was dropped on
|
||||
# CAPACITY, not bandwidth: a SASI volume on this machine tops out at 40 MB, and
|
||||
# the 22.8 minutes of unique scene footage on the source Blu-ray is 147 MB even
|
||||
# at the 110 KB/s the profile targeted -- more than the whole 4-unit SASI
|
||||
# address space, with nothing left for Human68k or the game. FINDINGS 32.
|
||||
#
|
||||
# That leaves ONE profile, which is also the end of the two-quality-mode
|
||||
# decision of session 2. The 110 KB/s RATE POINT may still return under another
|
||||
# name: a 1x SCSI CD-ROM sustains ~150 KB/s, below this profile, and CD-ROM is
|
||||
# the only period medium with the capacity for the span-heavy stream. That is
|
||||
# deferred to the blocked disk benchmark and the DMA-vs-PIO check (docs/
|
||||
# BENCHMARK.md, FINDINGS 29.5), because every bandwidth figure here is folklore
|
||||
# until one of them lands.
|
||||
PROFILES = {
|
||||
"sasi": dict(kbps=110, lam=60.0, k1=256, k4=256,
|
||||
desc="stock 10MHz ACE/EXPERT, SASI",
|
||||
quality="36.9 dB on 00020 / 29.6 dB on 00146 / 27.2 dB on the "
|
||||
"Singe window at 109.5 KB/s (session 5's fixed lam "
|
||||
"gave 27.8 dB there, but at 137.4 KB/s)",
|
||||
util="~105 KB/s = 35% of the pessimistic 300 KB/s SASI figure"),
|
||||
"scsi": dict(kbps=280, lam=10.0, k1=256, k4=256,
|
||||
desc="Super/XVI, or CZ-6BS1 board in a 10MHz machine",
|
||||
quality="39.4 dB on 00020 / 32.3 dB on 00146 / 29.9 dB on the "
|
||||
@@ -82,9 +89,9 @@ PROFILES = {
|
||||
LAM_CLIFF = 800.0
|
||||
|
||||
# Ceiling on the CYCLE search. mu prices a cycle in the same units lam prices a
|
||||
# byte, so the scale that matters is set by their ratio: at the `sasi` floor of
|
||||
# lam=60, mu=0.2 makes a V1 block's 300 cycles cost what its 1 payload byte
|
||||
# costs. MU_CLIFF=100 is three decades past that: a V1 block priced at 30,000
|
||||
# byte, so the scale that matters is set by their ratio: at a lam floor of 60
|
||||
# (the retired `sasi` profile's, and the highest this codec has shipped),
|
||||
# mu=0.2 makes a V1 block's 300 cycles cost what its 1 payload byte costs. MU_CLIFF=100 is three decades past that: a V1 block priced at 30,000
|
||||
# distortion units.
|
||||
#
|
||||
# It does NOT freeze the picture, and that is the point. At MU_CLIFF a block
|
||||
|
||||
Reference in New Issue
Block a user