drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Handoff: reconcile docs and tooling with the corrections made this session

Browse Source 
Session 2 reversed several of its own conclusions. The docs are append-only, so
a reader could land on a superseded section and act on it. This pass makes the
repo internally consistent.

Defects found and fixed in STATUS.md:
- claimed "Hybrid VQ with k=1024: no" as the answer to the linework question,
  directly contradicting FINDINGS 14, which rejected k=1024. Both profiles are
  k=256.
- malformed profile table (six column separators, five columns).
- next-steps list had two items numbered 3 and listed the full-disc survey
  twice.
- the disk-benchmark section still read CRITICAL-PATH with "if SCSI sustains
  >=800 KB/s, ship pixel-exact". That was written while the bandwidth figure
  was misread as 4 MB/s. At 4 Mbps pixel-exact needs 92-97% of the pipe and is
  not available, and the ring-buffer result means the design no longer hangs on
  the benchmark at all. Rewritten with what it IS still worth doing: confirming
  the 4 Mbps provenance, and confirming DMA is used rather than PIO.

FINDINGS now carries supersession blockquotes on 5, 8, 11, 17 and 18 pointing
at the sections that correct them. 18 is the dangerous one -- its peak-vs-
sustained test is reversed by 21 -- so it is marked DO NOT ACT ON THIS SECTION
while noting the per-frame data itself remains valid.

profile_gen.py had the same problem in code: it defaulted to the superseded
peak sizing and returned lam=25 where the docs say lam=10. The buffered test is
now the default and peak sizing is behind --size-for-peak as a bound only. A
tool that contradicts the findings is worse than no tool.

Also preserves the five measurement scripts that produced this session's
numbers as tools/analysis/05-09, following the session 1 precedent, and adds an
"explicitly abandoned -- do not re-propose" list to STATUS covering entropy
coding, k=1024 codebooks and flat 4x4 VQ.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 12:28:41 -07:00
parent fb8a1462b0
commit 64cd1ffd72
10 changed files with 345 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/analysis/05_lossless_floor.py
+63
View File
@@ -0,0 +1,63 @@
"""What does LOSSLESS (palettised-quality) delta coding actually cost at 12fps?
Session 1 measured only a hand-rolled row-span+RLE (3.2:1). The 68000 has ~12x
CPU headroom, so a real LZ decoder is affordable -- byte copies are what the
68000 is good at. This measures the achievable floor at zero extra quality loss.
"""
import sys, zlib, lzma; sys.path.insert(0,'tools/encoder')
import vq, numpy as np
S=sys.argv[1]
RAW=256*192
def rle_delta(a,b,gap=4):
"""session 1's row-span + RLE, reimplemented for comparison"""
t=0
for y in range(a.shape[0]):
d=np.nonzero(a[y]!=b[y])[0]
if not len(d): continue
sp=[];s=d[0];p=d[0]
for x in d[1:]:
if x-p>gap: sp.append((s,p)); s=x
p=x
sp.append((s,p)); t+=2
for s0,e0 in sp:
seg=b[y][s0:e0+1]; t+=2; i=0
while i<len(seg):
r=1
while i+r<len(seg) and seg[i+r]==seg[i] and r<127: r+=1
t += 2 if r>=3 else r; i+=r
return t
print(f'{"scene":8}{"frames":>7}{"raw KB/s":>9}{"rle":>8}{"xor+zl":>8}{"delta+zl":>9}{"lzma":>8}',flush=True)
tot={}
for s in ['00010','00020','00146','00181']:
rgb=vq.load_frames(f'{S}/fr_{s}')
ref,pal=vq.scene_palette(rgb)
idx=vq.palettise(rgb,ref)
n=len(idx)-1
r_rle=sum(rle_delta(idx[i-1],idx[i]) for i in range(1,len(idx)))/n
# XOR against previous frame then deflate -- unchanged pixels become 0 runs
r_xor=sum(len(zlib.compress((idx[i]^idx[i-1]).tobytes(),9)) for i in range(1,len(idx)))/n
# RLE-of-changed-spans payload, then deflate on top
def spans(a,b,gap=8):
out=bytearray()
for y in range(a.shape[0]):
d=np.nonzero(a[y]!=b[y])[0]
if not len(d): continue
sp=[];st=d[0];p=d[0]
for x in d[1:]:
if x-p>gap: sp.append((st,p)); st=x
p=x
sp.append((st,p))
for s0,e0 in sp:
out += bytes([y,s0,e0-s0]) + b[y][s0:e0+1].tobytes()
return bytes(out)
r_dz=sum(len(zlib.compress(spans(idx[i-1],idx[i]),9)) for i in range(1,len(idx)))/n
r_lz=sum(len(lzma.compress(spans(idx[i-1],idx[i]),preset=9))for i in range(1,len(idx)))/n
f=12/1024.0
print(f'{s:8}{len(idx):7}{RAW*f:9.0f}{r_rle*f:8.0f}{r_xor*f:8.0f}{r_dz*f:9.0f}{r_lz*f:8.0f}',flush=True)
for k,v in [('rle',r_rle),('xor',r_xor),('dz',r_dz),('lz',r_lz)]: tot.setdefault(k,[]).append(v)
print()
for k,v in tot.items():
m=np.mean(v)
print(f'{k:8} mean {m:7.0f} B/frame {m*12/1024:6.0f} KB/s {m*12*22*60/1048576:5.0f} MB/22min ratio {RAW/m:.1f}:1')
tools/analysis/06_codebook_fair.py
+15
View File
@@ -0,0 +1,15 @@
"""Fair k=256 vs k=1024 comparison: k=1024 pays 2-byte indices, so the +2.4 dB
it showed earlier may be entirely eaten by the doubled payload. Sweep lam and
report the rate-distortion CURVE, then compare at matched KB/s."""
import sys; sys.path.insert(0,'tools/encoder')
import vq_hybrid as H, numpy as np
S=sys.argv[1]
for s in ['00020','00146']:
print(f'=== scene {s} ===',flush=True)
for k in [256,1024]:
m=H.build(f'{S}/fr_{s}',k1=k,k4=k,iters=16)
ib=1 if k<=256 else 2
print(f' k={k} ({ib}-byte idx) {"lam":>7}{"PSNR":>8}{"KB/s":>8}{"SKIP":>7}{"V1":>6}{"V4":>6}{"RAW":>6}',flush=True)
for lam in [25.,100.,300.,800.,2000.]:
e=H.encode(m,lam=lam); r=H.evaluate(m,e)
print(f' {"":16}{lam:7.0f}{r["psnr"]:8.2f}{r["kbps"]:8.1f}{r["skip"]:7.1f}{r["v1"]:6.1f}{r["v4"]:6.1f}{r["raw"]:6.1f}',flush=True)
tools/analysis/07_highrate_deflate.py
+62
View File
@@ -0,0 +1,62 @@
"""How high should the SCSI profile go?
Two questions the current profiles never answered:
1. With the payload deflated, where does the hybrid's lam->0 end actually land?
(Un-deflated it is 439 KB/s, which is NOT comparable to the 247 KB/s
lossless changed-spans+deflate path.)
2. Is there any point shipping lossy VQ on SCSI at all, or does the lossless
path dominate once both are entropy-coded?
"""
import sys, zlib; sys.path.insert(0,'tools/encoder')
import vq_hybrid as H, vq as VQ, numpy as np, struct
S=sys.argv[1]
def pack_modes(mode):
n=len(mode); out=bytearray((n*2+7)//8)
for i,m in enumerate(mode): out[i//4] |= (int(m)&3)<<(6-2*(i%4))
return bytes(out)
def payload(mode,l1,l4g,src,nbx):
b=bytearray()
for i,mo in enumerate(mode):
if mo==1: b.append(int(l1[i]))
elif mo==2:
for j in range(4): b.append(int(l4g[i][j]))
elif mo==3:
by,bx=divmod(i,nbx); b+=src[by*4:by*4+4,bx*4:bx*4+4].tobytes()
return bytes(b)
def spans(a,b,gap=8):
out=bytearray()
for y in range(a.shape[0]):
d=np.nonzero(a[y]!=b[y])[0]
if not len(d): continue
sp=[];st=d[0];p=d[0]
for x in d[1:]:
if x-p>gap: sp.append((st,p)); st=x
p=x
sp.append((st,p))
for s0,e0 in sp: out+=bytes([y,s0,e0-s0])+b[y][s0:e0+1].tobytes()
return bytes(out)
for s in ['00020','00146']:
m=H.build(f'{S}/fr_{s}',k1=256,k4=256,iters=16)
pal,idx,W=m['pal'],m['idx'],m['W']; nbx=W//4
ceil_db=np.mean([VQ.psnr(o,pal[i]) for o,i in zip(m['rgb'],idx)])
# lossless reference, same frames
lz=np.mean([len(zlib.compress(spans(idx[i-1],idx[i]),9)) for i in range(1,len(idx))])
print(f'=== {s} palette ceiling {ceil_db:.2f} dB ===',flush=True)
print(f' LOSSLESS changed-spans+deflate : {lz*12/1024:6.1f} KB/s (pixel-exact)',flush=True)
print(f' {"lam":>6}{"PSNR":>8}{"raw KB/s":>10}{"defl KB/s":>11}{"gain":>7}{"RAW%":>7}',flush=True)
for lam in [0.,10.,25.,60.,150.,300.,800.]:
e=H.encode(m,lam=lam); r=H.evaluate(m,e)
tot_r=tot_d=0
for f,im in enumerate(idx):
B1=H.blocks_of(im,pal,4,4); l1=VQ.assign(B1,m['C1s'])
B4=H.blocks_of(im,pal,2,2); l4=VQ.assign(B4,m['C4s'])
q=H._group_2x2_into_4x4(np.arange(len(l4)),W); l4g=l4[q].reshape(-1,4)
blob=pack_modes(e['modes'][f])+payload(e['modes'][f],l1,l4g,im,nbx)
tot_r+=len(blob); tot_d+=len(zlib.compress(blob,9))
n=len(idx)
rk=tot_r/n*12/1024; dk=tot_d/n*12/1024
print(f' {lam:6.0f}{r["psnr"]:8.2f}{rk:10.1f}{dk:11.1f}{rk/dk:7.2f}{r["raw"]:7.1f}',flush=True)
tools/analysis/08_peaks.py
+18
View File
@@ -0,0 +1,18 @@
"""At a 488 KB/s (4 Mbps) ceiling and ~52% mean utilisation, the mean is not the
risk -- the peaks are. Measure per-frame peak-to-mean, then check whether the
leaky-bucket rate controller actually holds the ceiling."""
import sys; sys.path.insert(0,'tools/encoder')
import vq_hybrid as H, ratectl as RC, numpy as np
S=sys.argv[1]
BW=4_000_000/8/1024
print(f"ceiling {BW:.0f} KB/s (4 Mbps), audio {RC.AUDIO_KBPS} KB/s\n")
print(f'{"scene":8}{"lam":>5}{"mean":>8}{"p90":>8}{"MAX":>8}{"pk/mean":>9}{"MAX % of pipe":>15}',flush=True)
for s in ['00010','00020','00146','00181']:
m=H.build(f'{S}/fr_{s}',k1=256,k4=256,iters=16)
for lam in [60.,10.]:
e=H.encode(m,lam=lam)
kb=e['sizes']*12/1024 # per-frame instantaneous KB/s
tot=kb+RC.AUDIO_KBPS
print(f'{s:8}{lam:5.0f}{tot.mean():8.1f}{np.percentile(tot,90):8.1f}'
f'{tot.max():8.1f}{tot.max()/tot.mean():9.2f}{tot.max()/BW*100:14.1f}%'
+ (' OVER' if tot.max()>BW else ''),flush=True)
tools/analysis/09_buffer_sim.py
+34
View File
@@ -0,0 +1,34 @@
"""Ring-buffer simulation. The peak-vs-sustained comparison in FINDINGS 18 was
the wrong test: with SD-backed SCSI the fill rate is a CONSTANT, and a burst
frame is absorbed by the buffer rather than having to arrive within one frame.
What actually matters:
1. required PREFILL so the buffer never underruns mid-scene
2. STALL TOLERANCE at a branch point -- Dragon's Lair seeks between streams,
and the buffer drains while the seek completes
"""
import sys; sys.path.insert(0,'tools/encoder')
import vq_hybrid as H, ratectl as RC, numpy as np
S=sys.argv[1]; BW=4_000_000/8/1024; FPS=12
fill=BW/FPS
print(f"fill {BW:.0f} KB/s = {fill:.2f} KB per frame time\n")
print(f'{"scene":8}{"lam":>5}{"mean":>8}{"max f":>8}{"prefill":>9}{"stall @0KB":>12}{"stall @256KB":>14}')
for s in ['00010','00020','00146','00181']:
m=H.build(f'{S}/fr_{s}',k1=256,k4=256,iters=16)
for lam in [10.,60.]:
e=H.encode(m,lam=lam)
kb=e['sizes']/1024 + RC.AUDIO_KBPS/FPS # KB demanded per frame
# cumulative deficit: worst shortfall of supply vs demand
deficit=np.maximum.accumulate(np.cumsum(kb-fill))
prefill=max(0.0,deficit.max())
# stall tolerance: with buffer B prefilled, how many frame times can the
# fill be zero (seeking) before the buffer empties, at mean drain
mean_kb=kb.mean()
stall0 = prefill/mean_kb if mean_kb>0 else 0
stall256 = (256.0+0.0)/mean_kb
print(f'{s:8}{lam:5.0f}{kb.mean()*FPS:8.1f}{kb.max():8.2f}{prefill:9.1f}'
f'{stall0:9.1f} fr{stall256:11.1f} fr')
print()
print("prefill = KB the buffer must hold before playback starts")
print("stall = frame times the buffer survives with NO fill (seek/branch)")
print(" at 12fps, 1 frame = 83.3 ms")
tools/encoder/profile_gen.py
+34 -20
View File
@@ -11,12 +11,12 @@ bandwidth and it returns the lam that fits, with the headroom accounted for.
Three things eat the pipe before video gets any:
1. AUDIO -- 7.8 KB/s of MSM6258 ADPCM, constant.
2. PEAK/MEAN -- measured 1.4-1.9x (FINDINGS 18). The disk delivers a
SUSTAINED rate; a frame that overruns is a DROPPED frame.
Either size for the peak, or rate-control to the mean and
carry a bucket. We do the latter, so we need bucket depth
rather than peak headroom -- but until rate control is
actually wired in (it is not), size for the peak.
2. BUFFERING -- NOT peak/mean. FINDINGS 18 sized against the per-frame
peak; 21 showed that is the wrong test. The disk keeps
filling DURING a frame, so the condition is cumulative
demand vs cumulative supply, which every measured scene
passes with ZERO required prefill. Peak sizing is kept
behind --size-for-peak only as a pessimistic bound.
3. DMA CYCLE-STEAL -- the HD63450 steals ~8 clocks per 16-bit word from the
68000. At 488 KB/s that is 20% of the CPU, on top of the
blit. Bandwidth and CPU are NOT independent budgets.
@@ -49,16 +49,23 @@ CURVE = [
]
CEILING = {"00020": 39.90, "00146": 35.25}
PEAK_OVER_MEAN = 1.9 # measured worst case, FINDINGS 18
# Per-frame fill at 12fps must cover the worst single frame, else the buffer
# has to carry the difference. Measured worst frame is 42.10 KB (00146 lam=10).
WORST_FRAME_KB = 42.10
def dma_steal_pct(kbps):
return (kbps * 1024 / 2) * DMA_CLOCKS_PER_WORD / CLK * 100
def pick(bw_kbps, peak_factor=PEAK_OVER_MEAN, margin=0.85, rate_controlled=False):
"""Largest-quality lam whose worst-case demand fits inside bw_kbps."""
def pick(bw_kbps, peak_factor=PEAK_OVER_MEAN, margin=0.85, size_for_peak=False):
"""Largest-quality lam whose sustained demand fits inside bw_kbps.
Default is the BUFFERED test (FINDINGS 21): compare mean demand against the
sustained fill. size_for_peak=True restores the pessimistic FINDINGS 18
sizing, which is retained only as a bound -- it is not the shipping rule."""
usable = bw_kbps * margin - AUDIO_KBPS
factor = 1.0 if rate_controlled else peak_factor
factor = peak_factor if size_for_peak else 1.0
allow_mean = usable / factor
for lam, k20, d20, k146, d146 in CURVE:
worst = max(k20, k146)
@@ -81,9 +88,9 @@ def main():
help="fraction of the pipe we allow ourselves (seeks, "
"container overhead, and the fact that the bandwidth "
"figure itself is folklore)")
ap.add_argument("--rate-controlled", action="store_true",
help="assume the leaky bucket absorbs peaks (NOT YET TRUE "
"-- ratectl.py is written but not wired into encode.py)")
ap.add_argument("--size-for-peak", action="store_true",
help="pessimistic FINDINGS 18 sizing (mean * 1.9). Superseded "
"by 21 -- kept only as a bound, not the shipping rule.")
a = ap.parse_args()
bw = a.bw_kbps if a.bw_kbps else a.bw_mbps * 1_000_000 / 8 / 1024
@@ -91,26 +98,33 @@ def main():
print(f"bandwidth {src} = {bw:.0f} KB/s sustained")
print(f" usable at {a.margin:.0%} margin : {bw*a.margin:.0f} KB/s")
print(f" less audio ({AUDIO_KBPS}) : {bw*a.margin-AUDIO_KBPS:.0f} KB/s for video")
if not a.rate_controlled:
print(f" less peak/mean {PEAK_OVER_MEAN}x : "
f"{(bw*a.margin-AUDIO_KBPS)/PEAK_OVER_MEAN:.0f} KB/s mean allowance")
fill_per_frame = bw / FPS
if a.size_for_peak:
print(f" less peak/mean {PEAK_OVER_MEAN}x : "
f"{(bw*a.margin-AUDIO_KBPS)/PEAK_OVER_MEAN:.0f} KB/s mean allowance"
f" [pessimistic, FINDINGS 18 -- superseded]")
else:
print(" peaks absorbed by rate control (bucket depth must be validated)")
print(f" fill per frame time : {fill_per_frame:.2f} KB "
f"(worst measured frame {WORST_FRAME_KB:.2f} KB"
f"{' -- COVERED' if fill_per_frame >= WORST_FRAME_KB else ' -- needs buffer'})")
r = pick(bw, margin=a.margin, rate_controlled=a.rate_controlled)
r = pick(bw, margin=a.margin, size_for_peak=a.size_for_peak)
if r is None:
print("\n NO PROFILE FITS -- even lam=800 overruns. Lower the framerate,")
print(" the resolution, or get more bandwidth.")
return
steal = dma_steal_pct(r["peak_kbps"])
print(f"\n -> {a.name}: lam={r['lam']}, {r['mean_kbps']:.0f} KB/s mean, "
f"{r['peak_kbps']:.0f} KB/s peak")
steal = dma_steal_pct(r["mean_kbps"])
print(f"\n -> {a.name}: lam={r['lam']}, {r['mean_kbps']:.0f} KB/s mean")
print(f" quality 00020 {r['psnr20']:.2f} dB (-{r['loss20']:.2f} from ceiling)")
print(f" 00146 {r['psnr146']:.2f} dB (-{r['loss146']:.2f} from ceiling)")
print(f" CPU blit {BLIT_FULL_FRAME_PCT:.0f}% + DMA steal {steal:.1f}% "
f"= {BLIT_FULL_FRAME_PCT+steal:.0f}% of the frame budget")
if BLIT_FULL_FRAME_PCT + steal > 85:
print(" WARNING: CPU is now the binding constraint, not the bus.")
if not a.size_for_peak and fill_per_frame < WORST_FRAME_KB:
short = WORST_FRAME_KB - fill_per_frame
print(f" note: worst frame exceeds one frame-time of fill by "
f"{short:.2f} KB -- buffer must carry it (2MB RAM, non-issue).")
if __name__ == "__main__":