Pace the ring, then read the DMAC config out of the IPL ROM: audio is cheap and the disk is not
Two sessions that were never separated in the working tree, so they land as one commit. check.sh ALL GREEN before and after both. SESSION 19 -- the ring rig gets a frame clock (FINDINGS 51). src/player/stream.s had no frame clock: it asked for record i the instant it finished i-1, outran any finite pipe, and never let the ring back up. The 49.1 sweep passing at 48 KB was therefore a wrap-correctness result and nothing else. PACE/PACEON ($18034/$18038) hold the decoder to 12 fps, so FR_HEAD-FR_TAIL finally means what it reads as: whole frames the decoder could still draw with delivery stopped dead. PACEON=0 free-runs and is what the wrap gate still uses, so every figure in 49 is unmoved. Paced, on the gate container: 64 KB holds 2 frames, 256 KB holds 7-8, 512 KB holds 14-15, all pixel-exact. Tolerance is ceiling-1, measured by cutting the pipe: 256 KB buys 500 ms of dead pipe, not 583. SLACK IS ACCUMULATED, NOT OWNED. It is built out of pipe-wire and a seek spends all of it. At 488 KB/s a 256 KB ring needs 4.83 s of play to reach its ceiling from empty; 512 KB needs 8.42 s to reach 14. A bigger ring raises the ceiling AND lengthens the climb, so a branch point does not ask "is the buffer big enough" but "has there been enough play since the last one" -- and Dragon's Lair's decision points are seconds apart. The rig now also says WHICH resource is binding: at 460 KB/s every ring from 192 KB to 512 KB is rate-bound at ceiling 4 and never fills, so larger rings are dead RAM in that scene. 20_seek_slack.py is the same model rewritten in Python from record sizes, sharing no code with the Lua producer: 35/35 ceilings inside its bracket. SESSION 20 -- the DMAC configuration was in the IPL ROM the whole time (FINDINGS 52). ROADMAP's "do this first" was to put the ADPCM stream on the bus. That needs a clocks-per-byte figure for the audio channel, and 11_cpu_budget.py was charging audio the DISK's rate -- 5 clk/B, its own help text calling it "single-address, bus held". Audio was being charged the favourable end of B3, a 242 KB/s open question. It never had to be a guess. The IPL ROM programs all four HD63450 channels itself and MAME boots the rig with it, so 21_iplrom_dmac.py reads the configuration out of the image and decodes the MC68450 fields. Eight (address, expected bytes, meaning) sites; a mismatch or an unknown revision exits non-zero. In check.sh, no emulator, milliseconds. ch3 DCR=$80, OCR=$32: dual address, 8-bit port, cycle steal WITHOUT hold, REQG=10 external request. The DMAC arbitrates once per byte with no burst to amortise the 5..8 + 2 over, so an audio byte is 16..19 clocks, not 5 -- the old debit was 3.2x..3.8x small. And on the bus it is still nothing: 651 B/frame is 1.25%..1.48% of a frame, about 4% of what the decoder leaves. P6's bus risk does not materialise. The unit worry was worth checking and nearly right: 15.6 kHz is 8 MHz/512 = 15,625 samples/s, two 4-bit samples to a byte = 7,812.5 B/s exactly, and AUDIO_KBPS=7.8 is that in decimal kB while the tool multiplied by 1024. THE DISK CHANNEL IS PROGRAMMED IDENTICALLY. ch1 (SASI) is DCR=$80 too, and so is ch0. That is 16..19 clocks per delivered byte, where 42.4 brackets W at 5..12 and 42.5 has W=8 already missing 47/120 frames. The only worked example of a disk DMA configuration on this machine sits above the entire bracket, and at that price nothing fits at any container size. It is not scsiexrom.bin so B3 stays open -- what changed is that a cheap configuration is now the thing that has to be SHOWN. W <= 12 is a requirement on the player's DMAC programming, not a range the hardware hands us, and it is now the largest open number in the project, ahead of the rate. An unforced cross-check fell out: 15_bus_occupancy.py's new W sweep puts W=8 at 105.7% of the frame, agreeing with 42.5's 47/120, from mode histograms and bus clocks respectively, two models sharing no code. Also: ADPCM outranks the disk at the arbiter (CPR 1 against 2), so an audio byte never waits and a video byte does -- relevant to 51's smooth-rate delivery model. README MEDIA. stream.lua gains DLX_SNAP_EVERY=1 (needs DLX_PACE, off by default, on no path check.sh takes) and tools/media/make_readme_media.py turns the PNGs into docs/img/. The stills and both clips are MAME's own screen pixels. Building it turned up something worth recording. 116 of 119 captured frames are pixel-exact against dlx.py; three are TORN -- frame n on top, frame n-1 below the tear line -- because MAME captured the screen while the block loop was partway down it. decode.s writes straight to the displayed page (one display path, 28.1), so a real player tears the same way, and this is the first time that consequence has been visible rather than argued. The script ASSERTS the tear and refuses to build otherwise, rather than trimming three frames and reporting "every frame I kept is exact". Second correction the capture forced: the snapshot fires before frame n is decoded, so the obvious reading is that it holds frame n-1 -- it does not, because MAME renders the screen at the end of the machine frame, by which time the 68000 has finished frame n. 11_cpu_budget.py's "validated to within 1 pt" line is also corrected: the model reads 2..10 pt HIGH and by more as the frame gets harder, which was already true before either session. src/player/decode.s is unchanged; decode.bin is still 1,296 B at the same MD5. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
+47
-6
@@ -174,22 +174,44 @@ def main():
|
||||
help="override the profile's bitrate CEILING. The profile "
|
||||
"is a rate point on a delivery medium; this is for "
|
||||
"asking what the codec does at another one -- e.g. "
|
||||
"the 488 KB/s bus figure the span analyses of "
|
||||
"FINDINGS 30/40 are scored against.")
|
||||
"a measured delivery rate. Do not reach for the "
|
||||
"retired 4 Mbps figure; FINDINGS 42.1.")
|
||||
ap.add_argument("--span-kbps", type=float, default=None,
|
||||
help="byte ceiling the SPAN pass may draw on, if it differs "
|
||||
"from the profile's. The profile is a quality rate "
|
||||
"point; the pipe is hardware. Bytes between the two "
|
||||
"buy a better picture if spent on lam and the 68000's "
|
||||
"deadline if spent on spans -- and nothing at all if "
|
||||
"left unspent (FINDINGS 41.2). 488 is the bus figure "
|
||||
"tools/analysis/14_dmac_chain.py scores against.")
|
||||
"left unspent (FINDINGS 41.2). Pass the pipe rate "
|
||||
"tools/analysis/14_dmac_chain.py is scored against.")
|
||||
ap.add_argument("--spans", choices=("off", "need", "all"), default="need",
|
||||
help="v7 literal spans (FINDINGS 40). `need` (default) "
|
||||
"spends container bytes on spans only where a frame "
|
||||
"misses the 68000's decode deadline; `all` spends "
|
||||
"every profitable byte, which is the model "
|
||||
"14_dmac_chain.py scores; `off` emits DLX2.")
|
||||
ap.add_argument("--disk-clk-byte", type=float, default=None,
|
||||
help="clocks the SCSI DMA steals per DELIVERED BYTE, "
|
||||
"charged against the same frame budget the decoder "
|
||||
"spends (FINDINGS 43). Default 5.0, the "
|
||||
"single-address floor; 9.0 is dual-address; 0 "
|
||||
"restores the pre-43 encoder, which priced a byte at "
|
||||
"nothing and reported deadlines it could not meet.")
|
||||
ap.add_argument("--joint-decide", action="store_true",
|
||||
help="let the per-block lagrangian see the disk debit too, "
|
||||
"pricing a payload byte at lam+mu*c instead of lam. "
|
||||
"The default is OFF because it MEASURES as a wash: "
|
||||
"same 1/120, 0.02 dB worse, and it trades 6,058 "
|
||||
"clocks of disk for 17,207 of block decode "
|
||||
"(FINDINGS 44)")
|
||||
ap.add_argument("--joint-bucket", action="store_true",
|
||||
help="cap what the leaky bucket may lend a frame at what "
|
||||
"its clock budget can still absorb, since a borrowed "
|
||||
"byte is DISK_CLK_BYTE borrowed clocks and there is "
|
||||
"no double buffer to repay them from (FINDINGS 43.5). "
|
||||
"Default OFF: measured, it is worth one frame of 120 "
|
||||
"at --spans need and a 2%% regression at --spans all "
|
||||
"(FINDINGS 44)")
|
||||
ap.add_argument("--no-cpu-fit", action="store_true",
|
||||
help="drop the per-frame 68000 decode ceiling (session 7 "
|
||||
"behaviour: 31%% of frames on hard content do not fit)")
|
||||
@@ -214,6 +236,10 @@ def main():
|
||||
# The CPU ceiling is hardware, not taste: without it 31%% of frames on the
|
||||
# worst sustained window do not decode in time on a stock 68000, and with
|
||||
# it that is one frame -- the intra frame -- for 0.26 dB. FINDINGS 31.
|
||||
if a.disk_clk_byte is not None:
|
||||
RC.DISK_CLK_BYTE = a.disk_clk_byte
|
||||
RC.JOINT_DECIDE = a.joint_decide
|
||||
RC.JOINT_BUCKET = a.joint_bucket
|
||||
cyc_budget = None if a.no_cpu_fit else RC.FRAME_CYCLES
|
||||
span_mode = None if (a.spans == "off" or not rc) else a.spans
|
||||
|
||||
@@ -225,6 +251,13 @@ def main():
|
||||
print(f" CPU ceiling: " + (f"mu bisected per frame against "
|
||||
f"{RC.FRAME_CYCLES:,.0f} cycles (12fps, stock 68000)"
|
||||
if cyc_budget else "OFF (--no-cpu-fit)"))
|
||||
print(f" disk debit: {RC.DISK_CLK_BYTE:g} clocks per delivered byte, "
|
||||
f"charged INSIDE that ceiling (FINDINGS 43), and "
|
||||
+ ("SEEN by the mode decision at lam+mu*c (--joint-decide)"
|
||||
if RC.JOINT_DECIDE else "not seen by the mode decision, "
|
||||
"which measures as the better container (FINDINGS 44)")
|
||||
if RC.DISK_CLK_BYTE else
|
||||
" disk debit: 0 -- bytes priced at nothing (pre-FINDINGS-43)")
|
||||
else:
|
||||
print(f" target {prof['kbps']} KB/s, FIXED lam={lam} (no rate control)")
|
||||
print(f" k1={k1} k4={k4}, {_IDX_BYTES}-byte indices")
|
||||
@@ -305,11 +338,19 @@ def main():
|
||||
# moved into the span section rather than removed.
|
||||
cyc = (np.asarray(enc["cycles"]) if "cycles" in enc
|
||||
else np.array([H.cycles(mm) for mm in enc["modes"]]))
|
||||
pct = 100 * cyc / RC.FRAME_CYCLES
|
||||
# The frame's real cost is decode PLUS the bus the SCSI DMA steals to
|
||||
# deliver it. Reporting only `cyc` is what let session 13 print 0/120 for
|
||||
# a container no machine could have played (FINDINGS 43).
|
||||
disk = np.asarray(enc["sizes"], float) * RC.DISK_CLK_BYTE
|
||||
pct = 100 * (cyc + disk) / RC.FRAME_CYCLES
|
||||
miss = int((pct > 100).sum())
|
||||
print(f" non-SKIP blocks/frame: median {np.median(ns):.1f}% "
|
||||
f"p90 {np.percentile(ns, 90):.1f}% max {ns.max():.1f}%")
|
||||
print(f" decode cost: median {np.median(pct):.1f}% "
|
||||
dpct = 100 * disk / RC.FRAME_CYCLES
|
||||
print(f" disk debit: median {np.median(dpct):.1f}% "
|
||||
f"p90 {np.percentile(dpct, 90):.1f}% max {dpct.max():.1f}% "
|
||||
f"of the frame budget, at {RC.DISK_CLK_BYTE:g} clocks/byte")
|
||||
print(f" decode+disk: median {np.median(pct):.1f}% "
|
||||
f"p90 {np.percentile(pct, 90):.1f}% max {pct.max():.1f}% "
|
||||
f"of a {a.fps}fps frame")
|
||||
print(f" frames that do NOT decode in time: {miss}/{len(pct)} "
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Derive quality profiles FROM a measured bandwidth, instead of guessing lam.
|
||||
|
||||
python3 tools/encoder/profile_gen.py --bw-kbps 488 --name scsi
|
||||
python3 tools/encoder/profile_gen.py --bw-kbps <KB/s> --name scsi
|
||||
python3 tools/encoder/profile_gen.py --bw-mbps 4 # same thing
|
||||
|
||||
Session 2 set the profile bitrates by eye off the rate-distortion knee, which
|
||||
@@ -18,7 +18,7 @@ Three things eat the pipe before video gets any:
|
||||
passes with ZERO required prefill. Peak sizing is kept
|
||||
behind --size-for-peak only as a pessimistic bound.
|
||||
3. DMA CYCLE-STEAL -- the HD63450 steals ~8 clocks per 16-bit word from the
|
||||
68000. At 488 KB/s that is 20% of the CPU, on top of the
|
||||
68000. At ~500 KB/s that is ~20% of the CPU, on top of the
|
||||
blit. Bandwidth and CPU are NOT independent budgets.
|
||||
FINDINGS 5 said streaming "costs essentially no CPU";
|
||||
that is wrong -- cycle-stealing DMA is not free DMA.
|
||||
|
||||
+73
-10
@@ -110,6 +110,58 @@ MU_FLOOR = 1e-4 # bisection is geometric, so lo must be > 0
|
||||
# a frame that misses its deadline is simply late. FINDINGS 28.
|
||||
FRAME_CYCLES = 10_000_000 / 12.0
|
||||
|
||||
# What one delivered byte costs the 68000, in clocks of that same budget. The
|
||||
# SCSI DMA does not overlap with the CPU -- it steals the bus (FINDINGS 38.3) --
|
||||
# and the MB89352 is an 8-bit port, so the DMAC pays PER BYTE and not per word
|
||||
# (FINDINGS 43). 5.0 is the floor: single-address device-to-memory, bus held,
|
||||
# zero drive wait (MC68450 Fig 4-25 sheet 2). Dual-address costs 9.
|
||||
# 0.0 restores the pre-43 encoder, which priced a byte at nothing.
|
||||
DISK_CLK_BYTE = 5.0
|
||||
|
||||
# Does the MODE DECISION see that debit, or only the fit test above it?
|
||||
# FINDINGS 43.6 charged the disk inside the rate controller's ceiling but left
|
||||
# vq_hybrid.decide() ranking modes with a free byte, so `mu` still bought
|
||||
# cycles by moving V4 -> RAW: 47.8 cycles saved for 12 extra payload bytes,
|
||||
# which at any c >= 4 clocks/byte is a net LOSS on the very budget mu is
|
||||
# enforcing. True makes `decide` price a byte at `lam + mu*DISK_CLK_BYTE`.
|
||||
#
|
||||
# IT IS OFF, AND THAT IS A MEASUREMENT, NOT AN OVERSIGHT (FINDINGS 44). The
|
||||
# inconsistency is real and the fix is a wash: on the 120-frame Singe window at
|
||||
# c=5 it delivers 482.5 KB/s / 29.17 dB against 496.7 / 29.19, the same 1/120
|
||||
# frames over budget, the same 112.9% worst frame, and 5,389 MORE clocks in the
|
||||
# mean frame -- it buys 6,058 clocks of disk with 17,207 clocks of block
|
||||
# decode. Scored at c=4 and c=9 as well, it is marginally the worse container
|
||||
# at every price. `--joint-decide` turns it on.
|
||||
JOINT_DECIDE = False
|
||||
|
||||
# The leaky bucket lets a quiet frame bank bytes for a busy one, because the
|
||||
# player's ring buffer can hold them. True when a byte was only a byte. A byte
|
||||
# is now also DISK_CLK_BYTE clocks of the frame's decode budget, and FINDINGS 28
|
||||
# established there is no double buffer to decode ahead into -- so a frame that
|
||||
# borrows bytes from the bucket borrows clocks it cannot repay (FINDINGS 43.5).
|
||||
# True caps the borrow at what the clock budget can still absorb; the bucket
|
||||
# then smooths only what is left after the disk is paid. It never caps BELOW
|
||||
# the frame's own un-banked byte budget: that is rate control's job, not the
|
||||
# clock budget's.
|
||||
#
|
||||
# IT IS OFF, AND THAT IS A MEASUREMENT (FINDINGS 44). At `--spans all`, the
|
||||
# mode this project now recommends, the byte-side controller is INERT: a
|
||||
# 32-frame bucket and an 8-frame bucket emit the same container byte for byte,
|
||||
# and lam never leaves its floor on any of 120 frames. The cap is worth one
|
||||
# frame of 120 at `--spans need` (2/120 -> 1/120, for +26 KB/s) and is a 2%
|
||||
# regression at `all` (506.4 KB/s and 89.6% median frame against 496.7 and
|
||||
# 87.9%), because capping the block payload only moves those bytes into the
|
||||
# span section, which draws on its own flat pipe. Extending the cap to the
|
||||
# span section too was measured and is much worse: 273.7 KB/s, 28.88 dB, and
|
||||
# still 1/120 -- it starves the pass that was buying the deadline.
|
||||
# `--joint-bucket` turns it on.
|
||||
JOINT_BUCKET = False
|
||||
|
||||
|
||||
def _byte_clk():
|
||||
"""The debit the mode decision is allowed to see (0 = the old decision)."""
|
||||
return DISK_CLK_BYTE if JOINT_DECIDE else 0.0
|
||||
|
||||
AUDIO_KBPS = 7.8 # MSM6258 ADPCM 15.6kHz mono -- comes out of the same budget
|
||||
|
||||
|
||||
@@ -129,17 +181,18 @@ def _search_lam(ctx, allow, lam_lo, lam_hi, iters=12, mu=0.0):
|
||||
not fit: that frame is emitted over budget on purpose. Past the FINDINGS 15
|
||||
cliff a frame is not rate-controlled, it is destroyed, so a visible overrun
|
||||
is the better failure (FINDINGS 26.2)."""
|
||||
mode, sz = H.decide(ctx, lam_lo, mu)
|
||||
bc = _byte_clk()
|
||||
mode, sz = H.decide(ctx, lam_lo, mu, bc)
|
||||
if sz <= allow:
|
||||
return lam_lo, mode, sz, False
|
||||
mode_hi, sz_hi = H.decide(ctx, lam_hi, mu)
|
||||
mode_hi, sz_hi = H.decide(ctx, lam_hi, mu, bc)
|
||||
if sz_hi > allow:
|
||||
return lam_hi, mode_hi, sz_hi, True
|
||||
lo, hi = lam_lo, lam_hi # lo does not fit, hi does
|
||||
best = (lam_hi, mode_hi, sz_hi)
|
||||
for _ in range(iters):
|
||||
mid = float(np.sqrt(lo * hi))
|
||||
mode_m, sz_m = H.decide(ctx, mid, mu)
|
||||
mode_m, sz_m = H.decide(ctx, mid, mu, bc)
|
||||
if sz_m <= allow:
|
||||
hi = mid; best = (mid, mode_m, sz_m)
|
||||
else:
|
||||
@@ -172,12 +225,12 @@ def _search_mu(ctx, allow, lam_lo, lam_hi, cyc_budget, iters=10):
|
||||
Returns (mu, lam, mode, size, cyc, over_bytes, over_cycles)."""
|
||||
lam, mode, sz, ovr = _search_lam(ctx, allow, lam_lo, lam_hi, mu=0.0)
|
||||
cyc = H.cycles(mode)
|
||||
if cyc <= cyc_budget:
|
||||
if cyc + DISK_CLK_BYTE * sz <= cyc_budget:
|
||||
return 0.0, lam, mode, sz, cyc, ovr, False
|
||||
|
||||
lam_h, mode_h, sz_h, ovr_h = _search_lam(ctx, allow, lam_lo, lam_hi, mu=MU_CLIFF)
|
||||
cyc_h = H.cycles(mode_h)
|
||||
if cyc_h > cyc_budget: # cannot fit even frozen: emit late
|
||||
if cyc_h + DISK_CLK_BYTE * sz_h > cyc_budget: # cannot fit even frozen
|
||||
return MU_CLIFF, lam_h, mode_h, sz_h, cyc_h, ovr_h, True
|
||||
|
||||
lo, hi = MU_FLOOR, MU_CLIFF # lo overruns, hi fits
|
||||
@@ -186,7 +239,7 @@ def _search_mu(ctx, allow, lam_lo, lam_hi, cyc_budget, iters=10):
|
||||
mid = float(np.sqrt(lo * hi))
|
||||
lam_m, mode_m, sz_m, ovr_m = _search_lam(ctx, allow, lam_lo, lam_hi, mu=mid)
|
||||
cyc_m = H.cycles(mode_m)
|
||||
if cyc_m <= cyc_budget:
|
||||
if cyc_m + DISK_CLK_BYTE * sz_m <= cyc_budget:
|
||||
hi = mid; best = (mid, lam_m, mode_m, sz_m, cyc_m, ovr_m)
|
||||
else:
|
||||
lo = mid
|
||||
@@ -226,7 +279,7 @@ def _fit_spans(m, ctx, mode, sz, room, cyc_budget, span_mode, ib):
|
||||
return mode, sz, H.cycles(mode), None
|
||||
sel = SP.select(mode, src, m["nbx"], m["nby"], room,
|
||||
need_clocks=(None if span_mode == "all" else cyc_budget),
|
||||
idx_bytes=ib)
|
||||
idx_bytes=ib, disk_clk_byte=DISK_CLK_BYTE, base_bytes=sz)
|
||||
if not sel["spans"]:
|
||||
return mode, sz, H.cycles(mode), None
|
||||
nmode = sel["mode"]
|
||||
@@ -292,16 +345,26 @@ def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
|
||||
# and a pipe cannot be saved up. None means "the same allowance the lam
|
||||
# search had", which is what leaves spans nothing to buy with at a rate
|
||||
# point the block coder has already spent (FINDINGS 41.2).
|
||||
hard = None
|
||||
if (JOINT_BUCKET and cycle_budget is not None and DISK_CLK_BYTE > 0):
|
||||
# What can this frame's clock budget still absorb, once its own
|
||||
# block decode is paid? Priced at the mode map the UN-banked budget
|
||||
# buys, so the cap is a property of the frame's content rather than
|
||||
# of how full the bucket happens to be.
|
||||
_, mode0, _, _ = _search_lam(ctx, budget, lam_lo, lam_hi)
|
||||
hard = (cycle_budget - H.cycles(mode0)) / DISK_CLK_BYTE
|
||||
allow = min(allow, max(budget, hard))
|
||||
span_allow = allow if span_budget is None else span_budget
|
||||
sel = None
|
||||
lam, mode, sz, ovr = _search_lam(ctx, allow, lam_lo, lam_hi)
|
||||
mu, cyc, late = 0.0, H.cycles(mode), False
|
||||
if span_mode and (span_mode == "all"
|
||||
or (cycle_budget is not None and cyc > cycle_budget)):
|
||||
or (cycle_budget is not None
|
||||
and cyc + DISK_CLK_BYTE * sz > cycle_budget)):
|
||||
mode_pre = mode
|
||||
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
|
||||
cycle_budget, span_mode, ib)
|
||||
if cycle_budget is not None and cyc > cycle_budget:
|
||||
if cycle_budget is not None and cyc + DISK_CLK_BYTE * sz > cycle_budget:
|
||||
# The byte allowance could not buy the frame's deadline, so fall
|
||||
# back to the controller that pays in picture -- and then offer
|
||||
# spans the bytes the smaller mode map just freed.
|
||||
@@ -311,7 +374,7 @@ def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
|
||||
mode_pre = mode
|
||||
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
|
||||
cycle_budget, span_mode, ib)
|
||||
late = cyc > cycle_budget
|
||||
late = cyc + DISK_CLK_BYTE * sz > cycle_budget
|
||||
# Paint from the mode map as it was BEFORE spanning. A spanned run's
|
||||
# blocks read SKIP in the emitted header, but SKIP means "hold the
|
||||
# previous reconstruction" and on the first frame there is none -- and
|
||||
|
||||
+25
-9
@@ -117,20 +117,31 @@ BLK_BYT = {1: 1, 2: 4, 3: 16}
|
||||
|
||||
|
||||
def select(mode, src_idx, nbx, nby, byte_room, need_clocks=None,
|
||||
idx_bytes=1):
|
||||
idx_bytes=1, disk_clk_byte=0.0, base_bytes=0):
|
||||
"""Choose which runs to paint as spans.
|
||||
|
||||
`mode` 1-D mode map, modified nowhere (a new one is returned)
|
||||
`src_idx` (H, W) palettised source -- what the spans will carry
|
||||
`byte_room` container bytes the frame may still spend
|
||||
`need_clocks` stop as soon as the frame's decode cost is at or below this;
|
||||
`need_clocks` stop as soon as the frame's cost is at or below this;
|
||||
None spends every profitable byte instead (the model
|
||||
tools/analysis/14_dmac_chain.py scores).
|
||||
`disk_clk_byte` what a delivered byte costs the 68000 in clocks, because
|
||||
the SCSI DMA steals the bus from it (FINDINGS 43). 0.0 is
|
||||
the pre-43 behaviour: bytes are free and a span is judged on
|
||||
decode clocks alone. At the real value a span's two wire
|
||||
bytes per pixel are the dominant term and the ranking
|
||||
inverts on short runs.
|
||||
`base_bytes` the frame's byte count before any span is added, so the
|
||||
running total this loop compares against `need_clocks` can
|
||||
include the disk debit the frame is already carrying.
|
||||
|
||||
Ranked by clocks saved per byte spent, which is the same greedy 12 and 14
|
||||
use. Selection is deliberately conservative in two ways and the reported
|
||||
figures are exact rather than greedy: a run is only offered if the span
|
||||
beats the blocks it replaces on cycles ALONE, and the saving credited here
|
||||
use. A run is only offered if the span beats the blocks it replaces on
|
||||
TOTAL clocks -- decode plus the disk debit of the bytes it adds -- which at
|
||||
`disk_clk_byte`=0 reduces to the cycles-alone test this used before
|
||||
FINDINGS 43. Selection is otherwise deliberately conservative and the
|
||||
reported figures are exact rather than greedy: the saving credited here
|
||||
ignores the extra all-SKIP header bytes spanning tends to create. The
|
||||
caller recomputes the frame's real cost from the returned mode map.
|
||||
|
||||
@@ -146,10 +157,13 @@ def select(mode, src_idx, nbx, nby, byte_room, need_clocks=None,
|
||||
cur_b = sum(BLK_BYT[int(b)] * (idx_bytes if int(b) != 3 else 1)
|
||||
for b in m2[by][i:j])
|
||||
sc = run_clocks(L) + L * C_SKIP_MIXED # the dispatch still happens
|
||||
if sc >= cur_c:
|
||||
continue
|
||||
db = run_bytes(L) - cur_b
|
||||
cand.append(((cur_c - sc) / max(db, 1), cur_c - sc, db, by, i, j))
|
||||
# TOTAL saving: decode clocks won, less the clocks the extra bytes cost
|
||||
# on the way in. With disk_clk_byte=0 this is (cur_c - sc) exactly.
|
||||
net = (cur_c - sc) - disk_clk_byte * db
|
||||
if net <= 0:
|
||||
continue
|
||||
cand.append((net / max(db, 1), cur_c - sc, db, by, i, j))
|
||||
cand.sort(key=lambda s: -s[0])
|
||||
|
||||
# `need_clocks` is measured against the frame as it stands, so the loop
|
||||
@@ -161,7 +175,9 @@ def select(mode, src_idx, nbx, nby, byte_room, need_clocks=None,
|
||||
total_b, total_c = 0.0, 0.0
|
||||
chosen = []
|
||||
for _, dc, db, by, i, j in cand:
|
||||
if need_clocks is not None and H.cycles(cur) + total_c <= need_clocks:
|
||||
if (need_clocks is not None
|
||||
and H.cycles(cur) + total_c
|
||||
+ disk_clk_byte * (base_bytes + total_b) <= need_clocks):
|
||||
break
|
||||
if total_b + db > byte_room:
|
||||
continue
|
||||
|
||||
+45
-6
@@ -15,6 +15,7 @@ entries MUST be legal palette indices -- both quantisation losses compose.
|
||||
No sklearn on this box; k-means is hand-rolled (chunked, numpy).
|
||||
"""
|
||||
import numpy as np, glob, os, sys
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from PIL import Image
|
||||
|
||||
BW = BH = 4 # block size
|
||||
@@ -83,14 +84,52 @@ def kmeans(X, k, iters=24, seed=0):
|
||||
return C, assign(X, C)
|
||||
|
||||
|
||||
def assign(X, C, chunk=8192):
|
||||
"""Nearest centroid, chunked to bound memory."""
|
||||
# k-means assignment is ~95% of an encode's wall clock (the rest of the encoder,
|
||||
# rate control included, is about a second for a 120-frame window), so it is
|
||||
# worth the three details below. All three are EXACT: the labels are unchanged
|
||||
# bit for bit, which is what lets the containers this encoder emits stay
|
||||
# byte-identical across the change.
|
||||
_ASSIGN_CHUNK = 2048 # measured: see the note in assign()
|
||||
_ASSIGN_THREADS = min(8, (os.cpu_count() or 1))
|
||||
|
||||
|
||||
def _assign_range(X, CT, Cn, out, lo, hi, chunk):
|
||||
for i in range(lo, hi, chunk):
|
||||
j = min(i + chunk, hi)
|
||||
d = Cn[None, :] - 2.0 * (X[i:j] @ CT) # + |x|^2, constant per row
|
||||
out[i:j] = np.argmin(d, axis=1)
|
||||
|
||||
|
||||
def assign(X, C, chunk=_ASSIGN_CHUNK, threads=None):
|
||||
"""Nearest centroid, chunked to bound memory.
|
||||
|
||||
Three things make this 5.9x faster than the obvious version, measured on
|
||||
the 120-frame Singe window (1,474,560 2x2 blocks against k=256), and none
|
||||
of them changes a label:
|
||||
|
||||
* `C.T` is a VIEW, and a non-contiguous right-hand operand makes BLAS
|
||||
copy it per chunk: 1.83s -> 1.02s just from materialising it once.
|
||||
* chunk=2048, not 8192. The temporary is (chunk, k) float32 and the win
|
||||
is cache residency, not memory: 8192 is 1.01s, 32768 is 2.80s.
|
||||
* the chunk loop is embarrassingly parallel and numpy releases the GIL in
|
||||
both the matmul and the argmin, so a plain thread pool scales it:
|
||||
1.02 -> 0.31s on 8 threads. Partitioning by row cannot change an
|
||||
argmin, so the labels are identical to the serial ones -- asserted in
|
||||
tools/analysis/09_ratectl_drift.py by the fact that every container
|
||||
this encoder emits still hashes the same.
|
||||
"""
|
||||
Cn = (C ** 2).sum(1)
|
||||
CT = np.ascontiguousarray(C.T)
|
||||
out = np.empty(X.shape[0], dtype=np.int32)
|
||||
for i in range(0, X.shape[0], chunk):
|
||||
x = X[i:i + chunk]
|
||||
d = Cn[None, :] - 2.0 * (x @ C.T) # + |x|^2, constant per row
|
||||
out[i:i + chunk] = np.argmin(d, axis=1)
|
||||
n = X.shape[0]
|
||||
nt = _ASSIGN_THREADS if threads is None else threads
|
||||
if nt <= 1 or n < 4 * chunk:
|
||||
_assign_range(X, CT, Cn, out, 0, n, chunk)
|
||||
return out
|
||||
bnd = [(n * i) // nt for i in range(nt + 1)]
|
||||
with ThreadPoolExecutor(nt) as ex:
|
||||
list(ex.map(lambda ab: _assign_range(X, CT, Cn, out, ab[0], ab[1], chunk),
|
||||
zip(bnd[:-1], bnd[1:])))
|
||||
return out
|
||||
|
||||
|
||||
|
||||
@@ -215,24 +215,38 @@ def frame_ctx(m, f, prev, idx_bytes=None):
|
||||
idx_bytes=default_idx_bytes(m) if idx_bytes is None else idx_bytes)
|
||||
|
||||
|
||||
def decide(ctx, lam, mu=0.0):
|
||||
def decide(ctx, lam, mu=0.0, byte_clk=0.0):
|
||||
"""Lagrangian mode decision at one lam and one mu. Returns (mode, bytes).
|
||||
|
||||
Minimises `distortion + lam*bytes + mu*cycles` per block. `mu=0` is the
|
||||
byte-only decision every session before 8 made; the machine's binding
|
||||
budget is cycles, and bytes and cycles do not rank the modes the same way
|
||||
(V4 is 4x V1 in bytes, 1.49x in cycles; RAW is dearer than V4 in bytes and
|
||||
CHEAPER in cycles, so mu inverts that preference -- FINDINGS 28.8).
|
||||
Minimises `distortion + lam*bytes + mu*(decode cycles + byte_clk*bytes)`
|
||||
per block. `mu=0` is the byte-only decision every session before 8 made.
|
||||
|
||||
`byte_clk` is what a DELIVERED byte costs the 68000 in clocks of the same
|
||||
budget mu is bisected against -- ratectl.DISK_CLK_BYTE, 5.0 on the
|
||||
single-address row of FINDINGS 43.2. It defaults to 0, which reproduces
|
||||
sessions 8-14 exactly, and it is the last place in the encoder where a byte
|
||||
was still free: FINDINGS 43.6 put the disk debit in the rate controller's
|
||||
FIT TEST, but the decision underneath it still ranked modes as though the
|
||||
16 bytes of a RAW block cost nothing to deliver.
|
||||
|
||||
THAT INVERTS FINDINGS 28.8. RAW is 400.4 cycles against V4's 448.2, so
|
||||
with a free byte, raising mu buys cycles by moving V4 -> RAW. Priced, a RAW
|
||||
block costs `400.4 + 16c` and a V4 block `448.2 + 4c`, which cross at
|
||||
**c = 3.98 clocks/byte** -- and 43.1's floor argument (a 68000 bus cycle is
|
||||
four clocks and the SPC hands over one byte per cycle) says c >= 4 on any
|
||||
real machine. So on hardware mu's escape hatch was never there: it was
|
||||
spending 12 clocks of bus to save 47.8 of CPU.
|
||||
|
||||
Cheap by design: no painting, no image-sized work. A search calls this a
|
||||
dozen times per lam step and paints once."""
|
||||
ib = ctx["idx_bytes"]
|
||||
s = ctx["sym"]
|
||||
mc = mu * MODE_CYCLES
|
||||
lb = lam + mu * byte_clk # what one payload byte costs, both budgets
|
||||
cost = np.stack([ctx["eS"] + mc[0],
|
||||
s["e1"] + lam * (1.0 * ib) + mc[1],
|
||||
s["e4"] + lam * (4.0 * ib) + mc[2],
|
||||
np.full(ctx["nb"], lam * RAW_BYTES + mc[3])])
|
||||
s["e1"] + lb * (1.0 * ib) + mc[1],
|
||||
s["e4"] + lb * (4.0 * ib) + mc[2],
|
||||
np.full(ctx["nb"], lb * RAW_BYTES + mc[3])])
|
||||
mode = np.argmin(cost, axis=0).astype(np.uint8)
|
||||
return mode, frame_bytes(mode, ctx["nb"], ib)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user