Drop SASI on capacity, then find the budget never had the disk in it
USER DECISION: drop the `sasi` profile. Not on bandwidth -- on capacity. A SASI volume is 40 MB, and the 22.8 min of unique scene footage on the source Blu-ray (streams 00000-00201, measured, not recalled) is 146 MiB at the LOWEST rate this codec makes -- more than the machine's whole 4-unit SASI space. `scsi` is the only profile now. FINDINGS 32. Then the user asked whether we were drawing the wrong conclusions about PIO vs DMA, and we were, more broadly than the question implied. Every CPU figure in FINDINGS 24-34 is scored against the full 833,333 cycles/frame with nothing subtracted for moving the bitstream off disk. Debiting the HD63450 cycle-steal at the long-standing 8 clk/word ESTIMATE, "1 frame of 120 misses" becomes 84 of 120, median 112.4%. PIO at the span rate is 99.8% of the machine. Spans buy cycles by spending bandwidth and the bandwidth returns as steal, so 31.6's "fits completely" becomes a worst frame of 114.3%. 10 fps absorbs it: median 93.7%, 1/120. FINDINGS 35. `11_cpu_budget.py` takes --io dma|pio|none, defaults to dma, and warns if asked for none. Also landed: - item 1 done: the cost model checked against the 68000 on a cost-aware container, -3.07% to +0.01%, whole-window mean -1.22%. FINDINGS 34. - item 4 done: the container carries its own 4-byte record alignment (DLX2). 94/120 record starts were on odd addresses -- an address error, not a slow read -- now 0/120 for 16 B/s. Re-encoding reproduces 31.1 exactly. FINDINGS 33. - a `scsi` window does not fit the 2 MB machine the rig emulates (2.84 MB of stream past a 0x200000 ceiling). The gate now verifies 80 of 120 frames and SAYS so, and fails loudly when the pass does not complete, instead of reporting a phantom 49,005-pixel diff. FINDINGS 36. Three near-misses this session had one shape: an unobservable run nearly produced a false finding. stdbuf -oL on any MAME job that prints progress -- a file is block-buffered too, and a run that is merely finishing looks exactly like one that is wedged. check.sh ALL GREEN. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
@@ -16,7 +16,7 @@ the mode headers would exploit.
|
||||
RAW 16 literal palette indices -- the escape that makes lam=0 pixel-exact
|
||||
|
||||
Usage: python3 tools/analysis/08_mode_map.py <frames_dir> <out.webm>
|
||||
[--profile sasi|scsi] [--scale N] [--lossless] [--fixed-lam]
|
||||
[--profile scsi] [--scale N] [--lossless] [--fixed-lam]
|
||||
|
||||
--fixed-lam renders the pre-session-6 encoder (no rate control) instead.
|
||||
Output format follows the extension. Prefer .webm: GIF re-quantises to 256
|
||||
@@ -45,7 +45,7 @@ def main():
|
||||
if "--scale" in sys.argv:
|
||||
SCALE = int(sys.argv[sys.argv.index("--scale")+1])
|
||||
prof = RC.PROFILES[sys.argv[sys.argv.index("--profile")+1]
|
||||
if "--profile" in sys.argv else "sasi"]
|
||||
if "--profile" in sys.argv else "scsi"]
|
||||
m = H.build(src, k1=prof["k1"], k4=prof["k4"])
|
||||
# Rate-controlled by default, so the map shows the mode decisions that
|
||||
# actually ship. --fixed-lam renders the pre-session-6 encoder instead;
|
||||
|
||||
@@ -23,6 +23,8 @@ sys.path.insert(0, "tools/encoder")
|
||||
import numpy as np
|
||||
from dlx import DLX
|
||||
import vq_hybrid as H
|
||||
import ratectl as RC
|
||||
RC_AUDIO_BPS = RC.AUDIO_KBPS * 1024
|
||||
|
||||
# Machine clocks, confirmed from MAME 0.277 src/mame/sharp/x68k.cpp:1133/1194/
|
||||
# 1200 -- not recalled. x68000 and x68ksupr are BOTH 40_MHz_XTAL/4 = 10 MHz;
|
||||
@@ -45,6 +47,19 @@ ap.add_argument("container", nargs="?",
|
||||
ap.add_argument("--machine", default="stock", choices=list(CLOCKS),
|
||||
help="which X68000's clock to budget against (default stock)")
|
||||
ap.add_argument("--fps", type=float, default=FPS)
|
||||
# FINDINGS 35: the frame budget has never had the disk in it. The bitstream has
|
||||
# to be moved off SCSI into the ring buffer, and on this machine that costs CPU
|
||||
# whether it is DMA (the HD63450 cycle-steals) or PIO (the 68000 moves every
|
||||
# byte). Default ON, because scoring a decoder against a budget that assumes the
|
||||
# data arrives for free is exactly the mistake 35 was raised to stop.
|
||||
ap.add_argument("--io", default="dma", choices=["dma", "pio", "none"],
|
||||
help="how the bitstream reaches RAM (default dma)")
|
||||
ap.add_argument("--dma-clocks-per-word", type=float, default=8.0,
|
||||
help="HD63450 cycle-steal. ESTIMATE from FINDINGS 5, NEVER "
|
||||
"MEASURED, and the most load-bearing unmeasured number "
|
||||
"in the project (FINDINGS 35.3)")
|
||||
ap.add_argument("--pio-clocks-per-byte", type=float, default=12.0,
|
||||
help="hand-derived floor for a 68000 register-to-RAM copy")
|
||||
a = ap.parse_args()
|
||||
CPUHZ = CLOCKS[a.machine] * 1e6
|
||||
FPS = a.fps
|
||||
@@ -54,14 +69,35 @@ if not os.path.exists(a.container):
|
||||
|
||||
d = DLX(a.container)
|
||||
|
||||
# --- what the transfer costs, from the container's own byte rate
|
||||
vid_bps = sum(n + 4 for (_, n) in d.frames) / d.nframes * d.fps
|
||||
io_bps = vid_bps + RC_AUDIO_BPS
|
||||
if a.io == "dma":
|
||||
io_cycles_per_s = (io_bps / 2) * a.dma_clocks_per_word
|
||||
elif a.io == "pio":
|
||||
io_cycles_per_s = io_bps * a.pio_clocks_per_byte
|
||||
else:
|
||||
io_cycles_per_s = 0.0
|
||||
io_pct = 100 * io_cycles_per_s / CPUHZ
|
||||
FRAME_NET = FRAME * (1 - io_pct / 100)
|
||||
|
||||
modes = [d.modes(f) for f in range(d.nframes)]
|
||||
cyc = np.array([cycles(m) for m in modes])
|
||||
pct = 100 * cyc / FRAME
|
||||
pct = 100 * cyc / FRAME_NET
|
||||
ns = np.array([100 * (m != 0).mean() for m in modes])
|
||||
|
||||
print(f"{a.container}: {d.nframes} frames, {d.nb} blocks/frame")
|
||||
print(f"budget: {a.machine} @ {CLOCKS[a.machine]:.2f} MHz, {FPS:g} fps "
|
||||
f"-> {FRAME:,.0f} cycles/frame")
|
||||
print(f" I/O ({a.io}): {io_bps/1024:.1f} KB/s costs {io_pct:.1f}% of the CPU "
|
||||
f"-> {FRAME_NET:,.0f} cycles/frame left for decoding")
|
||||
if a.io == "dma":
|
||||
print(f" {a.dma_clocks_per_word:g} clocks/word is an ESTIMATE (FINDINGS 5), "
|
||||
f"never measured -- see FINDINGS 35.3")
|
||||
elif a.io == "none":
|
||||
print(" WARNING: --io none scores the decoder as if the disk were free. "
|
||||
"That is the\n premise FINDINGS 35 overturned; every 'N frames miss' "
|
||||
"figure before session 9\n was computed this way.")
|
||||
if a.machine != "stock":
|
||||
print(" (derived: scaled by clock from cycles measured on the 10 MHz core.\n"
|
||||
" MAME 0.277 marks x68ksupr/x68kxvi/x68030 MACHINE_NOT_WORKING, so\n"
|
||||
@@ -100,10 +136,11 @@ if a.machine == "stock" and a.fps == 12:
|
||||
f"(optimistic by {np.median(pct)/np.median(old):.2f}x at the median)")
|
||||
|
||||
miss = pct > 100
|
||||
print(f"\nframes that do NOT fit {FRAME:,.0f} cycles: {miss.sum()}/{d.nframes} "
|
||||
print(f"\nframes that do NOT fit {FRAME_NET:,.0f} cycles: {miss.sum()}/{d.nframes} "
|
||||
f"({100*miss.mean():.0f}%)")
|
||||
print(f" sustainable framerate if EVERY frame must fit: "
|
||||
f"{CPUHZ/cyc.max():.1f} fps; at the mean frame {CPUHZ/cyc.mean():.1f} fps")
|
||||
f"{CPUHZ*(1-io_pct/100)/cyc.max():.1f} fps; at the mean frame "
|
||||
f"{CPUHZ*(1-io_pct/100)/cyc.mean():.1f} fps")
|
||||
if miss.any():
|
||||
print(f" worst {pct.max():.1f}% -- {(pct.max()-100)/100*1000/FPS:.0f} ms late "
|
||||
f"on an {1000/FPS:.0f} ms frame")
|
||||
@@ -116,5 +153,7 @@ print(f"\nwhere the cycles go, over the whole window:")
|
||||
for k, n in enumerate(("SKIP", "V1", "V4", "RAW")):
|
||||
print(f" {n:<5} {100*tot[k]/tot.sum():5.1f}% of blocks "
|
||||
f"{100*spend[k]/spend.sum():5.1f}% of the cycles")
|
||||
print(f"\nV4 is {C_V4/C_V1:.2f}x a V1 block for {4}x the payload bytes -- the mode "
|
||||
f"decision\nin vq_hybrid.py charges it the bytes but not the cycles.")
|
||||
print(f"\nV4 is {C_V4/C_V1:.2f}x a V1 block for {4}x the payload bytes. Since "
|
||||
f"session 8 the mode\ndecision charges it BOTH (decide(ctx, lam, mu), "
|
||||
f"FINDINGS 31), which is why V4 is now\nthe rarest non-SKIP mode here -- "
|
||||
f"a byte-rich profile buys its way out to RAW instead.")
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""What does fitting the CPU budget cost in quality? (session 8, lever B)
|
||||
|
||||
python3 tools/analysis/13_cpu_ratectl.py [frames_dir] [--profiles sasi,scsi]
|
||||
python3 tools/analysis/13_cpu_ratectl.py [frames_dir] [--profiles scsi]
|
||||
|
||||
Session 6 made the BYTE budget a ceiling by bisecting `lam` per frame. FINDINGS
|
||||
28 then showed the binding budget is CYCLES, not bytes, and that the mode
|
||||
@@ -28,7 +28,7 @@ import vq as VQ, vq_hybrid as H, ratectl as RC
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("frames_dir", nargs="?", default="tmp/fr_singe")
|
||||
ap.add_argument("--profiles", default="sasi,scsi")
|
||||
ap.add_argument("--profiles", default="scsi")
|
||||
ap.add_argument("--fps", type=int, default=12)
|
||||
ap.add_argument("--cache", default=None, help="pickle of H.build (auto by dir)")
|
||||
a = ap.parse_args()
|
||||
@@ -97,5 +97,7 @@ for name in a.profiles.split(","):
|
||||
print()
|
||||
|
||||
print("FINDINGS 28.7: re-coding every non-SKIP block as V1 is the floor the "
|
||||
"CURRENT mode set\nallows, and it still misses 11 frames at sasi / 12 at "
|
||||
"scsi. Misses above that floor\nare item 4 (spans), not item 1.")
|
||||
"CURRENT mode set\nallows, and it still missed 11 frames at the retired "
|
||||
"110 KB/s profile / 12 at scsi.\nMisses above that floor are spans, not "
|
||||
"the mode decision -- and 31.3 showed the\nfloor itself was too "
|
||||
"pessimistic, because the real decision can move a block to SKIP.")
|
||||
|
||||
Reference in New Issue
Block a user