drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: drwily/Dragon-s-Lair-X68k:31c4c1aba12e46dc68a15048b4b87f7272c70c64
Branches Tags
drwily/Dragon-s-Lair-X68k:main
...
compare: drwily/Dragon-s-Lair-X68k:e565dfbbabc78ffbc95e6d2c1a86bd3c0035249b
Branches Tags
drwily/Dragon-s-Lair-X68k:main
10 Commits
31c4c1aba1 ... e565dfbbab
Author SHA1 Message Date
prosolis e565dfbbab Take the disc mount out of the source, so someone else's copy of the media works 
Three files hardcoded /media/reala-misaki/BDROM -- extract.py, 07_motion_survey.py
and check.sh -- which made the pipeline unrunnable for anyone whose disc mounts
anywhere else. DLX_BDROM now overrides it everywhere, defaulting to
/media/$USER/BDROM, so this box is unchanged and another one works. Verified by
running extract.py against a symlinked mount at a different path.

check.sh's failure message names the path it looked at and the variable to set,
instead of assuming udisks put it where this machine puts it.

README gains a "Reproducing this" section: no media ships here and none of it is
redistributable, so it says what you have to bring (the disc) and what is
already packaged (vasm is vendored as a binary with its source tarball; the
k-means is hand-rolled, so numpy and Pillow are the whole Python dependency).
It also names the two gates that SKIP rather than fail -- the px68k second-core
pass and the IPL ROM DMAC gate -- because both live outside this repo and a
silent skip is worth reading as a skip.

One trap called out rather than left to bite: scene selection is a hard-coded
stream number, not a search. A different pressing that numbers its .m2ts files
differently will extract the wrong footage and the green light will PASS on it.

check.sh ALL GREEN.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 18:19:54 -07:00
prosolis 2f9f5cc995 Pace the ring, then read the DMAC config out of the IPL ROM: audio is cheap and the disk is not 
Two sessions that were never separated in the working tree, so they land as one
commit. check.sh ALL GREEN before and after both.

SESSION 19 -- the ring rig gets a frame clock (FINDINGS 51).

src/player/stream.s had no frame clock: it asked for record i the instant it
finished i-1, outran any finite pipe, and never let the ring back up. The 49.1
sweep passing at 48 KB was therefore a wrap-correctness result and nothing else.
PACE/PACEON ($18034/$18038) hold the decoder to 12 fps, so FR_HEAD-FR_TAIL
finally means what it reads as: whole frames the decoder could still draw with
delivery stopped dead. PACEON=0 free-runs and is what the wrap gate still uses,
so every figure in 49 is unmoved.

Paced, on the gate container: 64 KB holds 2 frames, 256 KB holds 7-8, 512 KB
holds 14-15, all pixel-exact. Tolerance is ceiling-1, measured by cutting the
pipe: 256 KB buys 500 ms of dead pipe, not 583.

SLACK IS ACCUMULATED, NOT OWNED. It is built out of pipe-wire and a seek spends
all of it. At 488 KB/s a 256 KB ring needs 4.83 s of play to reach its ceiling
from empty; 512 KB needs 8.42 s to reach 14. A bigger ring raises the ceiling
AND lengthens the climb, so a branch point does not ask "is the buffer big
enough" but "has there been enough play since the last one" -- and Dragon's
Lair's decision points are seconds apart. The rig now also says WHICH resource
is binding: at 460 KB/s every ring from 192 KB to 512 KB is rate-bound at
ceiling 4 and never fills, so larger rings are dead RAM in that scene.
20_seek_slack.py is the same model rewritten in Python from record sizes,
sharing no code with the Lua producer: 35/35 ceilings inside its bracket.

SESSION 20 -- the DMAC configuration was in the IPL ROM the whole time
(FINDINGS 52).

ROADMAP's "do this first" was to put the ADPCM stream on the bus. That needs a
clocks-per-byte figure for the audio channel, and 11_cpu_budget.py was charging
audio the DISK's rate -- 5 clk/B, its own help text calling it "single-address,
bus held". Audio was being charged the favourable end of B3, a 242 KB/s open
question.

It never had to be a guess. The IPL ROM programs all four HD63450 channels
itself and MAME boots the rig with it, so 21_iplrom_dmac.py reads the
configuration out of the image and decodes the MC68450 fields. Eight
(address, expected bytes, meaning) sites; a mismatch or an unknown revision
exits non-zero. In check.sh, no emulator, milliseconds.

ch3 DCR=$80, OCR=$32: dual address, 8-bit port, cycle steal WITHOUT hold,
REQG=10 external request. The DMAC arbitrates once per byte with no burst to
amortise the 5..8 + 2 over, so an audio byte is 16..19 clocks, not 5 -- the old
debit was 3.2x..3.8x small. And on the bus it is still nothing: 651 B/frame is
1.25%..1.48% of a frame, about 4% of what the decoder leaves. P6's bus risk
does not materialise. The unit worry was worth checking and nearly right: 15.6
kHz is 8 MHz/512 = 15,625 samples/s, two 4-bit samples to a byte = 7,812.5 B/s
exactly, and AUDIO_KBPS=7.8 is that in decimal kB while the tool multiplied by
1024.

THE DISK CHANNEL IS PROGRAMMED IDENTICALLY. ch1 (SASI) is DCR=$80 too, and so
is ch0. That is 16..19 clocks per delivered byte, where 42.4 brackets W at 5..12
and 42.5 has W=8 already missing 47/120 frames. The only worked example of a
disk DMA configuration on this machine sits above the entire bracket, and at
that price nothing fits at any container size. It is not scsiexrom.bin so B3
stays open -- what changed is that a cheap configuration is now the thing that
has to be SHOWN. W <= 12 is a requirement on the player's DMAC programming, not
a range the hardware hands us, and it is now the largest open number in the
project, ahead of the rate.

An unforced cross-check fell out: 15_bus_occupancy.py's new W sweep puts W=8 at
105.7% of the frame, agreeing with 42.5's 47/120, from mode histograms and bus
clocks respectively, two models sharing no code.

Also: ADPCM outranks the disk at the arbiter (CPR 1 against 2), so an audio byte
never waits and a video byte does -- relevant to 51's smooth-rate delivery model.

README MEDIA.

stream.lua gains DLX_SNAP_EVERY=1 (needs DLX_PACE, off by default, on no path
check.sh takes) and tools/media/make_readme_media.py turns the PNGs into
docs/img/. The stills and both clips are MAME's own screen pixels.

Building it turned up something worth recording. 116 of 119 captured frames are
pixel-exact against dlx.py; three are TORN -- frame n on top, frame n-1 below
the tear line -- because MAME captured the screen while the block loop was
partway down it. decode.s writes straight to the displayed page (one display
path, 28.1), so a real player tears the same way, and this is the first time
that consequence has been visible rather than argued. The script ASSERTS the
tear and refuses to build otherwise, rather than trimming three frames and
reporting "every frame I kept is exact". Second correction the capture forced:
the snapshot fires before frame n is decoded, so the obvious reading is that it
holds frame n-1 -- it does not, because MAME renders the screen at the end of
the machine frame, by which time the 68000 has finished frame n.

11_cpu_budget.py's "validated to within 1 pt" line is also corrected: the model
reads 2..10 pt HIGH and by more as the frame gets harder, which was already true
before either session.

src/player/decode.s is unchanged; decode.bin is still 1,296 B at the same MD5.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 18:14:01 -07:00
prosolis b49bbdc939 Build v7 into the player, and find the cost model 18% wrong on the block it made commonest 
src/player/decode.s now paints v7 literal spans, pixel-exact under MAME and
px68k's C68K core over a container where every frame carries 128-216 spans
covering up to 38% of the picture. The span pass is blit.s v7 verbatim: the
66.0/9.143/9.978 fit was measured on that instruction sequence.

The container is DLX3 -- a span section between the mode header and the block
payload, since that is the only place the 68000 can reach without first parsing
something of variable length. 16_span_roundtrip.py gates it in check.sh, and
asserts it emitted enough spans to have tested anything.

Two synthetic all-SPAN anchors price v7 inside decode.s at 151.2 and 225.6
clocks per 4x4 block, against FINDINGS 40's table of 151 and 226 -- 0.2% on
both emulators. The measured mode costs what it was said to cost.

Two things that were not on the list:

TWO BYTE BUDGETS. FINDINGS 40's 18/120 was scored against the 488 KB/s PIPE,
not the 280 KB/s profile, and at the profile rate the lam search has already
spent the allowance -- spans fired on 5 frames of 120 and looked like a
regression. The profile is a chosen quality rate point; the pipe is hardware.
--kbps and --span-kbps are now separate and spans run before mu, because a span
pays in bytes and mu pays in picture. Delivered: 86/120 over budget without
spans, 77/120 at the profile budget, 34/120 on the pipe for +0.36 dB.

C_SKIP_MIXED WAS NEVER MEASURED, and it was 18% low -- 45.0, now 55.0. It is
the one constant in the table that came from a derivation, because the
synthetic frame that would measure it cannot exist: a byte needs a coded block
for its SKIP to be mixed. Four bracketing anchors measure it on both emulators
with the header byte rotated through all four positions, and the partner mode
solves back to its own anchored value to 0.2%. With it corrected the model
predicts a real spanned decode to -0.06% mean / 0.09% worst, against -2.99% /
4.30%. It matters because a span marks its run SKIP, so mixed SKIPs dominate
exactly the frames spans are judged on.

Also: the rig had been writing its synthetic timing frames 26 KB past the top
of a 2 MB machine, and got away with it because the modes it overran are
data-independent. A span's jump displacements come out of the stream, so it is
not. And frames-over-budget is no longer a safe headline -- the controller aims
at the deadline, so 55 of 120 frames sit within 5% of it and a 1% cost shift
moves 22 frames.

FINDINGS 41. check.sh ALL GREEN, now gating on a span-heavy DLX3 container.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 20:02:03 -07:00
prosolis c520a89e14 Measure the finer chain tail: 84/120 becomes 18/120, and the derivation was right by cancellation 
blit.s gains v7 -- v6's 24-pixel movem chain plus a second chain whose unit is
one `move.l (a0)+,(a2)+`. Measured over 13 span lengths by span.sh, every config
pixel-exact:

    cycles = 66.0 per span + 9.143 per COARSE pixel + 9.978 per FINE pixel

fitting all 13 to within 0.2%. v5 and v6 re-measure to FINDINGS 30 exactly, so
the harness has not drifted underneath the new variant.

Rescored against the same scsi window and the same additive model, v7 takes
84/120 frames over budget to 18/120 -- exactly what FINDINGS 39.4 derived, and
that agreement is two cancelling errors: the derivation's 2-register movem tail
is 29% too dear per pixel, and its "nothing per span" for the second chain entry
is 22.3 clocks too cheap. The plain post-incrementing move.l is the right tail
instruction, and it makes the padding quantum 2 pixels, which a run of 4x4
blocks pads to exactly zero.

The DMAC stays dropped on a measurement now rather than an argument: v7 takes
back 37 of the 43 frames the array chain would, with no reserved channel and no
timing neither emulator here can verify. Break-even against all-V1 moves from
L=4 blocks to L=2.

The fine displacement is carried mid-stream rather than in the span record, so
the decoder holds nothing across the copy and keeps all 12 payload registers --
which is the whole reason the coarse unit is 24 pixels.

span.sh is now -seconds_to_run 200 (30 s wall, 36 configs) and takes its
expected snapshot count from the generated metadata instead of a literal 23.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 19:02:23 -07:00
prosolis c5ca56330e A second emulator agrees, the bus was never counted, and the DMAC loses by one clock 
Three things, and the last one reversed itself when the datasheet arrived.

A SECOND EMULATOR. tools/bench/c68k/ links px68k's C68K core into a headless
harness -- no SDL, no ROMs, no emulated machine, because the decoder touches
nothing but RAM, the control block and GVRAM. decode.s is now pixel-exact under
two independent CPU cores, and cycle-table error against MAME is bounded at
3.3%, running against us. MAME 0.277's M68000 turns out to be the MICROCODE
core, not Musashi (m68000.lst + m68000gen.py), so this is two structurally
different timing models agreeing rather than two tables. FINDINGS 28.8's "V4
costs more than RAW" reproduces independently. FINDINGS 37.

THE BUS. Nothing since FINDINGS 24 had counted the 68000's local memory bus --
one 4-clock cycle at a time, carrying instruction prefetch as well as data. The
decoder occupies 86.7% of it and PREFETCH IS 62% OF THAT TRAFFIC, so a data-only
count understates occupancy by 2x. Two sources check each other: c68k_bench
counts every bus callback exactly, and a static walk of decode.lst supplies the
prefetch no emulator here can report. The walk reproduces the measured data half
to 0.04%, which is what licenses its prefetch half, and 15_bus_occupancy.py is a
gate rather than a report because every bus figure depends on that check.
FINDINGS 38.

THE DMAC CHAIN LOSES. FINDINGS 29.6 named it the one uncosted lever. Costed from
bus arithmetic -- a read cycle plus a write cycle, 8 clocks a pixel -- it scored
1/120 frames over budget against the v6 span's 10/120 and looked decisive. Then
the MC68450 manual (Motorola Jul 1989, now at ~/src/mc68450.pdf): Fig 4-25 sheet
4 puts a dual-address word between two 16-bit ports at 9 CLOCKS, because note 2
gives the DMAC 4-clock reads and 5-clock WRITES. The 68000 writes in 4.

    DMAC   9.000 clocks/pixel   datasheet
    v6     9.152 clocks/pixel   measured, FINDINGS 30

1.7%. Scored additively, 86% of what remains of the DMAC's advantage is v6's
24-pixel padding quantum -- a property of its unrolled movem chain, fixable in
software with a finer tail chain, worth 55/120 -> 18/120 against the DMAC's
12/120. Recommendation: fix the quantum, drop the DMAC. Six frames does not buy
a reserved channel, a two-region container layout and a timing dependency
neither emulator here can verify. The container is identical either way -- v6's
record and an HD63450 chaining entry are both 6 bytes, so the chain array IS the
span table -- so nothing is foreclosed. FINDINGS 39.

TWO CORRECTIONS TO MY OWN WORK IN THE SAME SESSION:

- I argued FINDINGS 35's flat CPU debit for the disk was too pessimistic and
  rescored the window at 53/120 with max(CPU, bus). Wrong. A 68000 has no cache
  and a two-word prefetch queue, so it stalls the moment another master takes
  the bus, and the MC68450 hands the bus over in SLABS under limited-rate
  auto-request rather than interleaving per operand. DMA is additive. 84/120
  stands and 14_dmac_chain.py reproduces it exactly. What 86.7% occupancy really
  says is that there is almost no room to overlap anything. FINDINGS 38.3.
- The first DMAC costing was derived where a primary source existed. Both wrong
  answers were confident and both were caught by reading the manual.

Also landed:
- FINDINGS 5's 8 clocks/word for the SCSI DMA, STATUS's own "most load-bearing
  unmeasured number", is now bracketed by the datasheet: 5 clk/word with the bus
  held, ~12 if the DMAC arbitrates per word. 8 is a supported midpoint, and
  which end applies is a player design decision worth 7 clocks a word on a
  480 KB/s stream. FINDINGS 39.7.
- check.sh gains two gates: the C68K pixel-exact decode (seconds, no MAME) and
  the bus-model self-check. Both skip cleanly without a px68k checkout.
- spanned blocks are now charged their mode-map dispatch, which FINDINGS 30.7
  flagged as uncounted in 12_span_tradeoff.py.
- MAME timed runs must be budgeted by WALL CLOCK, not -seconds_to_run: this box
  runs x68000 at ~0.033x realtime and two runs were killed by their own timeout.
  That is why the all-RAW cell in 37.3 is empty. The C68K harness does the same
  work in seconds because it emulates a CPU and not a machine.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 18:30:23 -07:00
prosolis 7d365b3ff5 Drop SASI on capacity, then find the budget never had the disk in it 
USER DECISION: drop the `sasi` profile. Not on bandwidth -- on capacity. A SASI
volume is 40 MB, and the 22.8 min of unique scene footage on the source Blu-ray
(streams 00000-00201, measured, not recalled) is 146 MiB at the LOWEST rate this
codec makes -- more than the machine's whole 4-unit SASI space. `scsi` is the
only profile now. FINDINGS 32.

Then the user asked whether we were drawing the wrong conclusions about PIO vs
DMA, and we were, more broadly than the question implied. Every CPU figure in
FINDINGS 24-34 is scored against the full 833,333 cycles/frame with nothing
subtracted for moving the bitstream off disk. Debiting the HD63450 cycle-steal
at the long-standing 8 clk/word ESTIMATE, "1 frame of 120 misses" becomes 84 of
120, median 112.4%. PIO at the span rate is 99.8% of the machine. Spans buy
cycles by spending bandwidth and the bandwidth returns as steal, so 31.6's "fits
completely" becomes a worst frame of 114.3%. 10 fps absorbs it: median 93.7%,
1/120. FINDINGS 35. `11_cpu_budget.py` takes --io dma|pio|none, defaults to dma,
and warns if asked for none.

Also landed:
- item 1 done: the cost model checked against the 68000 on a cost-aware
  container, -3.07% to +0.01%, whole-window mean -1.22%. FINDINGS 34.
- item 4 done: the container carries its own 4-byte record alignment (DLX2).
  94/120 record starts were on odd addresses -- an address error, not a slow
  read -- now 0/120 for 16 B/s. Re-encoding reproduces 31.1 exactly. FINDINGS 33.
- a `scsi` window does not fit the 2 MB machine the rig emulates (2.84 MB of
  stream past a 0x200000 ceiling). The gate now verifies 80 of 120 frames and
  SAYS so, and fails loudly when the pass does not complete, instead of
  reporting a phantom 49,005-pixel diff. FINDINGS 36.

Three near-misses this session had one shape: an unobservable run nearly
produced a false finding. stdbuf -oL on any MAME job that prints progress -- a
file is block-buffered too, and a run that is merely finishing looks exactly
like one that is wedged.

check.sh ALL GREEN.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 17:09:47 -07:00
prosolis 06b98d4b47 Price cycles in the mode decision: 37 misses become 1, for 0.26 dB 
The decoder has been CPU-bound since FINDINGS 28 while the mode decision
minimised D + lam*R -- distortion against BYTES. decide() now minimises
D + lam*bytes + mu*cycles, and ratectl bisects mu per frame against the
833,333-cycle budget with the lam bisection nested inside it. On the worst
sustained window:

  sasi  27.22 -> 26.95 dB, 109.5 -> 109.4 KB/s, 37/120 misses -> 1
  scsi  29.90 -> 29.27 dB, 280.0 -> 278.6 KB/s, 51/120 misses -> 1

Bitrate does not move: the byte controller still binds, and mu changes WHICH
modes are bought. V4 is what it stops buying -- 25.2 -> 20.3% of blocks at sasi
and 15.0 -> 5.3% at scsi, where RAW takes it. That is 28.8's inversion in
practice: RAW is dearer in bytes and cheaper in cycles, so only the byte-rich
profile can buy its way out of V4.

Three things worth knowing beyond the headline:

  - The one frame that still misses, at both profiles, is FRAME 0 -- no previous
    reconstruction, so 100% changed by definition, which is also what a scene
    cut is. It comes out at the all-V1 floor of 110.6% and is emitted late on
    purpose. Freezing a cut to make a deadline is the worse failure.
  - 28.7's "11 frames are impossible" was too pessimistic. That floor held the
    SKIP set fixed and asked how cheaply the drawn blocks could be drawn; the
    real decision can also MOVE a block to SKIP, which above ~90% non-SKIP is
    the only lever left.
  - SKIP's price depends on its neighbours (13.25 cycles clustered, 45 mixed),
    which a per-block lagrangian cannot see. The way out is that the two uses
    need not share a cost function: a ranking constant inside decide(), the
    exact clustered rule for the frame-level bisection. vq_hybrid.cycles() is
    now the one definition of that rule and 11_cpu_budget.py imports it.

Gated: 09_ratectl_drift.py runs both controllers, both 0/120 drifting frames.
The cost-aware container decodes pixel-exact on the 68000 (120 frames). ON by
default in encode.py; --no-cpu-fit restores session 7. check.sh ALL GREEN.

Still a model, not a measurement, for THIS container: FINDINGS 31's cycle
figures come from vq_hybrid.cycles (within 1 point of the 68000 on four frames
of the session-7 container). Timing this one on the machine is step 1 of the
next session -- it was started and killed for time, and it is slow.

FINDINGS 31. tools/analysis/13_cpu_ratectl.py.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 16:24:22 -07:00
prosolis 29eb78a599 Measure the span: the mode survives, and it is an encoder format 
FINDINGS 29 priced a literal-span mode at 4*(50 + 4L*9.08) cycles and labelled
the whole section DERIVED. Session 8 step 0 was to measure it before optimising
over the mode set it implies. Two variants in blit.s, one stream per span length
from prep_spans.py, timed by span.lua, driven by span.sh in ~25 s:

  v5, handed (x, npix) and left to work the copy out:  97.9/span + 10.459/px
  v6, handed an address and a jump displacement:       43.7/span +  9.152/px
  29 assumed                                           50.0/span +  9.080/px

So 29's arithmetic was right about a format nobody had written. The difference
is not tuning: v5 spends ~122 cycles a span computing a destination, dividing
npix into bursts and handling a 0..15 remainder, all of which the encoder knows
at build time. v6's record is {u32 absolute GVRAM address, u16 jump
displacement} into an unrolled chain of 24-pixel copy units -- no loop, no
remainder, no arithmetic -- and it fits 11 span lengths to 0.3%.

Three things that measurement showed and derivation could not:

  - The per-pixel cost is a function of REGISTER PRESSURE. FINDINGS 24's 9.08
    was a fixed blit with 12 registers free; v5 can spare 8 and pays 10.46; v6
    gets 12 back only because the encoder holds the state.
  - Short spans die in the remainder path -- a 12-pixel span costs MORE than a
    16-pixel one -- and the fix is padding, not avoidance.
  - Odd-x alignment is free (259.0 vs 261.8 cycles/span), as a 16-bit bus
    implies but nobody had checked.

Re-priced against the unchanged mode maps, sasi: median 74.4% -> 52.0% (29 said
43.0), misses 37 -> 10/120 (29 said 8), 448.0 KB/s. Break-even moved from runs
of 2 blocks to runs of 4. 29.4 survives: a scene cut needs x >= 0.196 of the
frame as spans and the bus allows x <= 0.373, so it fits at 12fps.

All 23 timing configs are also checked pixel-exact, so none of this was timed
against a decoder that quietly skipped work.

FINDINGS 30. Next: lever B, the cost-aware mode decision.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 15:45:51 -07:00
prosolis 3641f37e28 The bus is 4x idle while the CPU is pinned: price the trade 
The codec was designed when bytes were scarce, so every decision in it trades
cycles to save bytes. That is now backwards: sasi spends 110 KB/s of a 488 KB/s
pipe while missing 31% of frames on CPU.

The cheapest thing a 68000 can be handed is the most expensive thing to store.
Measured, per pixel: row-linear copy from word-expanded memory 9.08 cycles,
block-order 12.98, V1 codebook 18.74, RAW byte literals 25.03. So the 1024-byte
stride costs 43% and unpacking bytes to words costs more than the write itself.

Pricing one new mode -- a per-row span of word-expanded literals movem.l'd
straight from the stream buffer -- against the UNCHANGED mode maps:

  sasi   median 74.4% -> 43.0%, worst 136.2% -> 106.2%, misses 37 -> 8/120,
         101.7 -> 453.2 KB/s
  scsi   median 94.9% -> 69.4%, misses 51 -> 18/120, 272 -> 479.7 KB/s

scsi gains less precisely because it has less idle bandwidth left to trade.

Two consequences worth flagging. A word-expanded literal block derives to ~240
cycles, cheaper than V1's measured 299.9 and pixel-exact -- so every codebook
mode is CPU-dominated by a literal, and the codebook is a byte optimisation
that now costs cycles. And 28.5's "a scene cut cannot fit at 12fps" reopens:
CPU needs >=19% of the frame as spans, the bus allows <=39%, and that interval
is not empty.

DERIVED, NOT MEASURED, and labelled as such everywhere. The 9.08 cycles/pixel
is real but was measured at full row width with 12-register bursts, so short
spans are flattered. Measuring one span on the 68000 is now step 0 of the next
session, ahead of the cost-aware mode decision, because it changes the mode set
that decision optimises over.

FINDINGS 29. tools/analysis/12_span_tradeoff.py.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 15:28:29 -07:00
prosolis cb05e77a42 Handoff: reconcile the docs with a decoder that works and does not fit 
Session 7 handoff. check.sh green end to end, including both new stages.

STATUS now leads with the cost-aware mode decision, and the stale statements
that session 7 overturned are struck rather than left to be re-read as current:

- next-steps item 1 still announced "implement both display paths and pick per
  frame" as a settled answer. Withdrawn, with the reason, and the part that did
  hold up (report the distribution, not the mean) kept.
- the profile table's "machine" column reads as a CPU claim and is not one. It
  is the bus. Both profiles target the same stock 10 MHz 68000 and neither
  fits; the Super has SCSI at 10 MHz.
- the display-path section stopped at "68000 code copied a frame". It parses
  one now, and the blit figures are no longer the display-path budget.
- the green-light section still described a 2-minute, three-stage check, and
  did not warn that 10_pathmix_drift.py is SUPPOSED to exit non-zero.

Sharpened the one trap in the next session's first step: SKIP is not a
per-block constant. It costs 13.25 cycles inside an all-SKIP header byte and
~45 in a mixed one, so its price depends on its neighbours and a per-block
lagrangian cannot see that. Picking either number is wrong in a different
direction; the budget check can use the exact clustered cost function while
the per-block term only has to rank modes.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-23 15:21:12 -07:00
67 changed files with 11318 additions and 472 deletions
Expand all files Collapse all files
.gitignore
+1
View File
@@ -13,3 +13,4 @@ __pycache__/
*.dlx
a.out
tmp/
tools/bench/c68k/c68k_bench
README.md
+270 -10
View File
@@ -1,16 +1,173 @@
# Dragon's Lair — Sharp X68000 port
Porting Dragon's Lair to a stock X68000 (68000 @ 10MHz, 2MB, SASI/SCSI).
Porting Dragon's Lair to a stock X68000 (68000 @ 10MHz, 2MB, SCSI).
This is fundamentally a **video codec problem**, not a game-logic problem: the
game logic is a scene table with branching input windows; the difficulty is
pushing ~22 minutes of Don Bluth animation through a 10MHz 68000.
---
## What it looks like
![Blu-ray source next to the 68000's output](docs/img/source-vs-decoded.png)
Left, the Blu-ray frame cropped to 256x192. Right, **the same frame as the
emulated 68000 actually drew it** — 256 colours out of the X68000's 65536, one
16-colour-per-4x4-block codebook, decoded by `src/player/decode.s` from the
container. Not a re-render: these are the pixels MAME had on screen, extracted
from its own snapshot. 2x nearest-neighbour, no filtering.
**The player, running.** 119 frames out of a **256 KB ring buffer on an emulated
stock 2 MB X68000**, paced to a 12 fps frame clock, streamed from a host file at
488 KB/s — `src/player/stream.s`, no Lua in the decode path. Source on the left,
the machine's screen on the right.
<video src="docs/img/player.webm" controls muted loop width="100%"></video>
[`docs/img/player.webm`](docs/img/player.webm) — 119 frames, 12 fps, VP9
116 of those 119 frames are **pixel-exact** against `tools/encoder/dlx.py`'s
reference reconstruction. The other three are **torn** — the top of the picture
is frame *n* and the bottom still holds frame *n-1*, because MAME captured the
screen while the block loop was partway down it. That is not a rig artefact:
`decode.s` writes straight to the displayed page (one display path, FINDINGS
28.1 — the dual-path plan is kept runnable as a counterexample precisely because
it corrupts frames), so a real player tears the same way.
`tools/media/make_readme_media.py` asserts the tear rather than trimming it —
every differing pixel has to come from the previous frame, or it refuses to
build.
**What the decoder is actually doing.** The same window with the block-mode map
beside it: **black = SKIP** (costs nothing, draws nothing — the previous frame
stands), **blue = V1** (one codebook index for a whole 4x4 block), **amber = V4**
(four indices), **red = RAW** (sixteen bytes verbatim). The mode mix is what
every cost table in `docs/FINDINGS.md` is really about — V4 costs 1.5x V1, and
since session 8 the mode decision is charged both bytes *and* cycles, which is
why a byte-rich profile buys its way out to RAW instead of V4.
<video src="docs/img/modes.webm" controls muted loop width="100%"></video>
[`docs/img/modes.webm`](docs/img/modes.webm) — the same 119 frames with the mode map
**Name the layer:** everything above is **emulated** (MAME 0.277 `x68000`,
`-bios ipl10`, stock 10 MHz / 2 MB), cross-checked frame-for-frame on a second
CPU core (px68k's C68K). Nothing in this project has run on real hardware yet.
---
**And the binding resource is the 68000's local BUS, not its clock.** The
decoder occupies 86.7% of it once instruction prefetch is counted, and 52 of the
53 frames that miss the 12fps budget miss it on the bus, not the CPU
(FINDINGS 38). Read that before optimising anything for cycles.
The **literal span with a fine tail** (v7) is now IN the player: `decode.s`
paints it, pixel-exact under both CPU cores, and it costs inside the decoder
what `blit.s` said it would to 0.2% (FINDINGS 41).
**The delivery path is built and tested too** (FINDINGS 49). `src/player/stream.s`
decodes the whole 120-frame window **out of a 256 KB ring on a stock 2 MB
machine**, final frame pixel-exact, with the container in a host file rather than
preloaded into RAM. The constraint turned out to be **contiguity, not byte
count** — the block loop reads with a monotonically increasing `a0` and no bounds
check, so the ring needs the whole next record resident *and contiguous*, which
is a condition no byte-counting buffer simulation can see.
**And building it caught a live defect**, then cost the project a constant.
The shipping candidate is 496.7 KB/s; the pipe figure the design had been
simulated against since session 2 was smaller, and nothing in the tree was
comparing the two — the rate controller binds on clocks and has no pipe term at
all, while the buffer sizing kept standing on a constant the design had stopped
enforcing.
**So the pipe constant is retired (session 18, USER DECISION).** It was never a
bus measurement — user-supplied, no provenance, 10% of SCSI-1's asynchronous
rating (FINDINGS 42.1). It is gone as a default from every analysis tool and
from `stream.lua`; `--bus` / `--kbps` / `DLX_STREAM_KBPS` are now **required
arguments**, so no table can be scored against a rate its own output does not
state. **There is no working delivery figure, and that is the honest state.**
What replaces it is a requirement rather than a constant:
`tools/analysis/19_ring_stream.py` reports the **zero-prefill pipe**, the rate a
medium must clear for a container to need no prefill. For the candidate that is
**513.2 KB/s** — a hardware acceptance test to measure a BlueSCSI against.
**Bytes are not free, and the number that said they were was in the wrong
unit.** Session 13 found the pipe figure the design was built against was never
a bus figure (SCSI-1 is 1.5 MB/s asynchronous) and concluded the span pass
saturates at ~837 KB/s, 0/120 frames over budget. Session 14 found the disk
debit behind that was charged **per word of stream to a byte-wide port** — the
MB89352 is an 8-bit SPC, so the DMAC pays per BYTE, and the debit is 2x every
table since FINDINGS 5. No 68000 bus cycle is shorter than four clocks, so the
old figure was below a physical floor.
**What survives, re-encoded honestly: 496.7 KB/s at 29.19 dB, 1 frame of 120
over the 12fps budget** — and that one is frame 0, the intra frame, late on
purpose. The remaining lever is not ours: **whether the CZ-6BS1 wires the SPC's
DACK to the bus's `#EXACK`**, which decides 5 clocks/byte against 9, and with
it 242 KB/s and 0.69 dB. Read FINDINGS 43 before quoting any rate figure.
**And session 20 read the answer the machine already had.** The X68000's IPL ROM
programs all four HD63450 channels itself, and MAME boots the rig with it, so
`tools/analysis/21_iplrom_dmac.py` decodes the configuration straight out of the
image and gates on the bytes still being there. The audio channel is
dual-address, 8-bit port, cycle steal **without hold**, one external request per
byte: **16..19 clocks a byte, not 5** — which prices the ADPCM stream at
1.25%..1.48% of a frame and closes ROADMAP's "do this first" item. The disk
channel is programmed **identically**. That is 16..19 clocks per delivered byte,
above the whole 5..12 bracket the project costs the transport in, and at that
price nothing fits. It is SASI and not the MB89352, so it does not settle the
question — but **a cheap configuration is now the thing that has to be shown,
not the thing assumed.** FINDINGS 52.
**Green-light check:** `./tools/bench/check.sh` (~3 min, needs the Blu-ray
mounted) re-runs both display regression tests, the rate-control drift test, the
display-path coherency counterexample and a 120-frame 68000 decode, then prints
`ALL GREEN`.
## Reproducing this
**No media ships in this repo, and none of it is redistributable.** Bring your
own Dragon's Lair Blu-ray. Everything else needed to rebuild every number and
every picture above is here or is packaged.
You need:
| | |
|---|---|
| the disc | loop-mounted read-only. `udisksctl loop-setup -r -f DRAGONS_LAIR.iso` — the tree was built against a decrypted UDF 2.x image (7-Zip cannot read UDF 2.x; use the loop mount) |
| `python3` | plus **numpy** and **Pillow**. Nothing else — the k-means is hand-rolled rather than pulling in sklearn |
| `ffmpeg` / `ffprobe` | frame extraction, and the README clips |
| **MAME** | tested on 0.277, with the `x68000` ROM set. The rigs drive it headless via `-autoboot_script` |
| vasm (m68k, Motorola syntax) | **vendored**: `tools/vasm/vasmm68k_mot` is a Linux x86-64 binary, with the source tarball beside it to rebuild elsewhere |
Then:
```sh
export DLX_BDROM=/path/to/your/mounted/bluray # if not /media/$USER/BDROM
./tools/bench/check.sh # ~3 min, prints ALL GREEN
```
`DLX_BDROM` is honoured by every tool that reads the disc. Two stages are
optional and **skip rather than fail** when their input is absent, because both
live outside this repo:
- `PX68K=/path/to/px68k` — a px68k checkout, for the second-CPU-core gate. This
is the cheapest strong test in the tree (seconds, no MAME, no ROMs), and it is
what licenses the bus and cycle figures.
- `IPLROM=/path/to/iplrom.dat` — the X68000 IPL ROM, for the DMAC-configuration
gate (FINDINGS 52). Defaults to `~/mame/roms/iplrom.dat`.
To rebuild the stills and clips in `docs/img/` you also need a paced recording
run — see the header of `tools/media/make_readme_media.py`.
**Scene selection is a hard-coded stream number**, not a search: the gates use
stream `00020` and `00223` of the disc's 224 `.m2ts` files, which are the ones
FINDINGS §1 and §25 characterise. A different pressing may number them
differently, and if so the green light will extract the wrong footage rather
than fail — check that `tmp/fr_singe/` looks like the Singe encounter (which is
what the directory is named for) before trusting any figure.
## Read first
- **`docs/FINDINGS.md`** — measured hardware facts, content statistics, codec
decision, and a section on measurement traps that produced three separate
@@ -18,6 +175,9 @@ display-path coherency counterexample and a 120-frame 68000 decode, then prints
- **`docs/STATUS.md`** — current state, working setup, blockers, next steps.
**Start here.** It also lists what has been explicitly abandoned, so old ideas
do not get re-proposed.
- **`docs/ROADMAP.md`** — the remaining work to a completion target, and which
milestone that target is. Read it with STATUS, not instead of it: STATUS holds
the measurements, ROADMAP holds the shape and goes stale first.
- **`docs/BENCHMARK.md`** — how to measure the storage subsystem, and why a
bandwidth figure out of MAME would be meaningless.
- **`docs/HARDWARE.md`** — X68000 GVRAM/CRTC reference.
@@ -37,21 +197,89 @@ tools/analysis/ measurement scripts, numbered in the order they were written
demonstrates that the two-display-path plan of FINDINGS
24.5/25.6 corrupts 70 of 120 frames (FINDINGS 28.1).
11 scores a container against the MEASURED per-mode block
costs without needing MAME.
costs without needing MAME; 12 prices the literal-span mode of
FINDINGS 30 against those same mode maps, and prints whether a
scene cut still fits at 12fps; 13 measures what fitting the
CPU budget costs in dB (FINDINGS 31) and caches H.build so the
search loop is seconds, not minutes.
14 prices the HD63450 array-chain against the v6 and v7
spans (FINDINGS 39/40) and prints the sensitivity that decides
it -- v7 is measured, and takes 37 of the 43 frames the DMAC
would, so the DMAC stays dropped;
15 measures how much of the 68000's LOCAL bus the decoder
occupies (FINDINGS 38) and exits non-zero if its derived
model stops matching the harness's measurement.
16 is the DLX3 span container ROUND-TRIP gate (part of
check.sh): it encodes, writes the container, reads it back with
the reference decoder and fails if a pixel differs -- or if it
emitted too few spans to have tested anything. 17 prices the
spans the encoder ACTUALLY emitted, with no selection model,
which is what 12 and 14 could only simulate.
18 measures what a 16-colour text-plane literal would cost in
dB, and closes that direction (FINDINGS 46.3).
19 is the RING-BUFFER simulation, and it supersedes 09_buffer_
sim.py's question rather than repeating it: it models the ring's
ADDRESSES, because src/player/ needs each record contiguous and
not merely resident. It reports the ZERO-PREFILL PIPE -- the
rate a medium must clear for a container to need no prefill --
and warns explicitly when demand exceeds supply on the MEAN,
where a "required prefill" figure would flatter a sustained
overrun. Its wrap count and hole size match tools/bench/
stream.lua's, measured on a real 68000, to the digit.
buscost.py is the shared bus-cycle table both import; the
per-BLOCK constants live in tools/encoder/vq_hybrid.py and are
imported, never copied (session 12 corrected one of them).
tools/bench/ MAME Lua injection harness + 68000 benchmark sources.
`check.sh` re-runs both display regression tests (~40 s).
`blit.s`/`blit.lua` time the full-frame GVRAM blit on the
68000 itself (FINDINGS 24) — not part of check.sh, because
wall timings would make the green-light check host-sensitive.
`span.sh` (prep_spans.py + span.lua + blit.s v5/v6/v7)
measures the literal-span mode the same way (FINDINGS 30 and
40, ~30 s); it also asserts that every one of its 36 timing
configs drew a pixel-exact frame, the count taken from the
generated metadata so a new config cannot weaken the gate.
v7 is v6 with a second, 2-pixel chain for the span tail:
66.0 cycles/span + 9.143 per coarse pixel + 9.978 per fine
pixel, MEASURED, which is the win FINDINGS 39.4 predicted.
`crtc_mode.lua` is the single source of truth for CRTC R00-R08
and R20 — do not write CRTC values anywhere else.
`prep_dlx.py`/`decode.lua`/`verify_decode.py` load, time and
verify `src/player/decode.s`; the verify pass is in check.sh.
`prep_stream.py`/`stream.lua` do the same for `stream.s`, but
lay the container out as a DISK in a host file and feed it
through a bounded ring at a modelled pipe rate -- so the rig is
no longer bounded by the emulated machine's RAM, and a stock
2 MB machine runs the whole window. `dlxload.py` holds the
codebook/palette load-time maths both preps share.
tools/bench/c68k/ headless px68k C68K harness -- a SECOND emulator for every
68000 cycle figure (FINDINGS 37). Links only px68k's CPU core:
no SDL, no ROMs, no emulated machine. `make PX68K=~/src/px68k`
then `run.sh`; `verify_c68k.py` checks the decode is
pixel-exact, which is what licenses the cycle numbers. It also
counts BUS cycles, which MAME cannot report.
The Makefile's -no-pie and the harness's MAP_32BIT arena are
load-bearing: C68K truncates host pointers to 32 bits.
tools/vasm/ vasm m68k assembler (built from source)
tools/encoder/ hybrid VQ encoder + DLX1 container writer (working).
tools/encoder/ hybrid VQ encoder + DLX3 container writer (working).
spans.py is the v7 span geometry, selection and serialiser, and
the single place the chain layout is stated on the encoder side
-- it must match blit.s/decode.s (11 coarse units of 24 px, 11
fine of 2).
DLX2 4-byte-aligns every frame record: an odd `move.l` is an
ADDRESS ERROR on a 68000, not a slow read (FINDINGS 28.3).
dlx.py is the reference DECODER -- ground truth for the 68000.
src/player/ decode.s: the 68000 DLX1 decoder. Pixel-exact, and 31% of
frames over the 12fps CPU budget. See FINDINGS 28.
src/player/ decode.s: the 68000 DLX3 decoder, PRELOADED-stream front-end.
stream.s: the same decoder behind a bounded RING (FINDINGS 49).
Both include frame.i (the block loop and span chain) and geom.i
(the constants) so there is exactly ONE copy of the bytes every
cycle constant in FINDINGS 24/30/40/41 is fitted to. check.sh
asserts decode.s still assembles to the same 1,296 bytes.
decode.s: the 68000 DLX3 decoder. Pixel-exact under MAME and
px68k's C68K core, blocks and v7 literal spans both. The span
pass is blit.s v7 verbatim -- the same instruction sequence the
66.0/9.143/9.978 fit was measured on, so do not tidy it.
See FINDINGS 28, 31, 40 and 41.
assets/ extracted frames/audio (gitignored)
```
@@ -59,13 +287,45 @@ assets/ extracted frames/audio (gitignored)
```
python3 tools/encoder/extract.py 00020 /tmp/fr 12 crop
python3 tools/encoder/encode.py /tmp/fr out.dlx --profile sasi --preview p.png
python3 tools/encoder/encode.py /tmp/fr out.dlx --profile scsi --preview p.png
```
Two quality profiles ship from one codec and one decoder — `sasi` (110 KB/s) and
`scsi` (280 KB/s) are two points on the same rate-distortion curve. Both are
**ceilings**: lam is bisected per frame under a leaky bucket, so the profile's
`lam` is a quality floor rather than a setting (`--fixed-lam` opts out). The codec is
**Two budgets, not one.** `--kbps` is the quality rate point and `--span-kbps`
is the ceiling the span pass may draw on. They are different things: the profile
is chosen, the pipe is hardware, and bytes between them buy a better picture if
spent on `lam`, the 68000's deadline if spent on spans, and nothing if left
unspent. Spans run before `mu` because a span pays in bytes and `mu` pays in
picture (FINDINGS 41.2).
**One profile: `scsi`, 280 KB/s.** The 110 KB/s `sasi` profile was dropped in
session 9 on capacity, not bandwidth — a SASI volume is limited to 40 MB, and
the game's 22.8 minutes of footage is 146 MiB even at that rate (FINDINGS 32).
The rate point may return under another name once the delivery medium is
settled, because a 1x CD-ROM sustains ~150 KB/s and CD-ROM is the only period
medium with the capacity.
The profile bitrate is a **ceiling**: lam is bisected per frame under a leaky
bucket, so the profile's `lam` is a quality floor rather than a setting
(`--fixed-lam` opts out).
**But at `--spans all` none of that binds.** A 32-frame bucket emits the same
container byte for byte as an 8-frame one, and `lam` never leaves its floor on
any frame of the reference window: the rate is set by the span pass and by `mu`,
not by `--kbps` or the bucket (FINDINGS 44.3). Two known unit inconsistencies on
that side are implemented and default OFF because they measure as a wash --
`--joint-decide` (the per-block lagrangian prices a byte at `lam + mu*c` rather
than `lam`) and `--joint-bucket` (the bucket may not lend clocks it cannot
repay). FINDINGS 44.
An encode is ~95% k-means; a 120-frame window is ~29 s, of which ~22 s is
training the two codebooks (FINDINGS 44.5).
There are **two** ceilings, on two different axes. The second is the 68000's
decode budget: `mu` is bisected per frame against 833,333 cycles so the frame
also *decodes* in time, which takes the worst sustained window from 37 frames
over budget to 1 for 0.62 dB at `scsi` (FINDINGS 31). It is on by default; `--no-cpu-fit`
restores session 7 behaviour. Unlike bytes, cycles have no bucket — there is no
double buffer to decode ahead into, so it is a hard per-frame ceiling. The codec is
a Cinepak-style hybrid: each 4x4 block is coded as SKIP, one 4x4 codeword, four
2x2 codewords, or RAW literal pixels, chosen per block by rate-distortion.
docs/BENCHMARK.md
+23 -5
View File
@@ -53,13 +53,31 @@ was used. **Do not record KB/s and treat it as a hardware figure.**
Already partly in FINDINGS 5. Bounds worth tightening from datasheets:
- 68000 bus cycle: 4 clocks @ 10MHz, 16-bit => **5 MB/s** absolute ceiling
- HD63450 single-address DMA, ~8 clocks/word => **~2.5 MB/s** practical ceiling
- HD63450 single-address DMA, **5 clocks/BYTE** => **2.0 MB/s** practical ceiling
(dual-address is 9 clocks/byte => 1.11 MB/s). CORRECTED session 14: this line
read "~8 clocks/word => ~2.5 MB/s", which charged a byte-wide SPC per word.
FINDINGS 43.
- SCSI-1 asynchronous REQ/ACK handshake per byte, plus MB89352 FIFO depth
=> the real limiter, and the number we do not have from a primary source
The user's working figure is **4 Mbps = 488 KB/s**, which sits sensibly between
the derived DMA ceiling and observed period-drive rates. **Provenance not yet
recorded — worth pinning down, because every profile now hangs off it.**
**RETIRED, session 18 (USER DECISION).** This document used to name a working
figure of "4 Mbps" here and note that every profile hung off it. It was never a
bus measurement — user-supplied, no provenance, and 10% of SCSI-1's asynchronous
rating (FINDINGS 42.1). It has been removed as a default from every analysis
tool and from `tools/bench/stream.lua`; the tools now REQUIRE an explicit rate,
so nothing can be scored against a figure the scorer never restates.
**There is no working delivery figure. That is the honest state, and it is the
point:** the rate is a property of the medium, the medium is a BlueSCSI, and it
has not been measured. `tools/analysis/19_ring_stream.py` reports the
**zero-prefill pipe** — the rate a medium must clear for a given container to
need no prefill at all — which is the threshold a measurement should be taken
against. For the session-14 candidate that is **513.2 KB/s** (FINDINGS 49.5).
One place still carries the old number: `GATE_SPAN_KBPS` in
`tools/bench/check.sh`, because the gate container was *encoded* with it and
every per-block and span constant in FINDINGS 41/43/45/49 is fitted to that
container. It is a container recipe, not a claim about any medium.
### The coupling nobody had counted
Cycle-stealing DMA is not free DMA. At ~8 clocks per 16-bit word:
@@ -69,7 +87,7 @@ Cycle-stealing DMA is not free DMA. At ~8 clocks per 16-bit word:
| 110 KB/s | 4.5% | 42.8% |
| 250 KB/s | 10.2% | 48.5% |
| 450 KB/s | 18.4% | 56.7% |
| 488 KB/s | 20.0% | 58.3% |
| ~490 KB/s | 20.0% | 58.3% |
FINDINGS 5 concluded that because transfers are DMA, "streaming costs
essentially no CPU". **That is wrong.** It costs up to a fifth of the machine at
docs/FINDINGS.md
+3039 -4
View File
File diff suppressed because it is too large Load Diff
docs/ROADMAP.md
+244
View File
@@ -0,0 +1,244 @@
# Roadmap — remaining work to a completion target
Written end of session 19 (2026-08-24), against a tree that is ALL GREEN.
**THE COMPLETION TARGET IS M3, THE VERTICAL SLICE** (USER DECISION): one scene
tree — a decision point, two outcomes, a death clip — with audio, streaming from
a real SCSI volume on a stock 2 MB machine, playable. That is the point at which
every layer of this design has been shown to work at once. M4 is listed because
it is real work, but past M3 it is content grinding rather than open questions.
`docs/STATUS.md` remains the session-by-session record and the handoff. This file
is the shape of what is left; where the two disagree about what is done, STATUS
is the one with the measurements and this one is the one that goes stale. Both
were wrong about two encoder gaps until this file was written — see "What was
already done" below.
---
## Status of the four resources
The project's own framing, restated because every item below is priced in one of
these units:
| resource | state |
|---|---|
| **68000 local bus** | the binding one. Decoder occupies 86.7%; 52 of 53 missed frames miss on the bus, not the clock (FINDINGS 38). |
| **68000 clocks** | measured, and the rate controller binds on them. |
| **Delivery rate** | **no working figure, deliberately** (FINDINGS 50, USER DECISION). Every tool REQUIRES an explicit rate. |
| **Seek time** | **no figure at all, and never had one.** 51.3/51.4 made it matter. |
| **W, clocks stolen per delivered byte** | bracketed 5..12 (42.4); the IPL ROM's own disk channel is **16..19** (52.5). **The largest open number in the project.** |
---
## What was already done, and was still on the list
Found while inventorying for this file. Both had been closed in code for several
sessions and were still listed as open gaps in `docs/STATUS.md`:
- **4-byte record padding.** `DLX2`, `encode.py:139-156`, inside rate-control
accounting, reported per frame and per second.
- **CPU cost in the mode decision.** `vq_hybrid.py:218`, priced against measured
per-mode cycles with the exact clustered SKIP rule.
Both entries are now struck in STATUS. **The lesson is procedural: a gap list
that is only ever appended to manufactures phantom work.** Anything crossed off
below should be crossed off in STATUS in the same sitting.
---
## Blocked on hardware this tree does not have
None of these block M2 or M3 software work, because session 18 forced every rate
to be an explicit argument. They set constants, and two of them decide how much
headroom the finished player has.
**B1. Measure the BlueSCSI — throughput AND seek time.**
Throughput has an acceptance test already derived from real record sizes:
**513.2 KB/s** for the session-14 candidate, **451.4 KB/s** for the gate
container (`19_ring_stream.py`, FINDINGS 49.5). Seek time has nothing.
51.3/51.4 is why the second half matters: slack is *accumulated* out of
`pipe - wire`, so what a branch point costs is set by the rate and the time since
the last branch, not by the ring size. At 460 KB/s every ring from 192 KB to
512 KB is rate-bound and never fills. **Do not substitute a guess** — run at
several explicit rates and report the sensitivity. That is exactly how the
retired pipe constant survived five sessions after 42.1 called it folklore.
**B2. Does buffer mode blank the display?** `probe_bit11_blank.lua` is written
and settles it in minutes on a real board. FINDINGS 48 shifted the prior toward
MAME and toward "unusable" — **do not pre-build on 1.0 B/pixel**. Same sitting:
the priority register `0xE82500` at `0x0000` (47.3).
**B3. Single-address vs dual-address DMA.** 242 KB/s and 0.69 dB. Needs
`scsiexrom.bin` (8 KB, CRC `7be488de`) sourced, then its DMAC init disassembled
for DCR's DTYP: `10`/`11` = single (5.0 clk/B), `00`/`01` = dual (9.0).
FINDINGS 48.4. Not on this machine (checked, session 18).
**This is also P4's input** — the handshake the player drives is the same
question from the software side.
> **Session 20 moved the prior hard, and it moved the wrong way (FINDINGS 52.5).**
> The IPL ROM *is* on this machine, and `tools/analysis/21_iplrom_dmac.py` reads
> its HD63450 setup: the on-board disk channel (ch1, SASI) is `DCR = $80` —
> **dual address, 8-bit port, cycle steal WITHOUT hold**, with `REQG = 10`
> external request, i.e. a full bus arbitration per byte. That is **16..19
> clocks per delivered byte**, above the whole 5..12 bracket 42.4 costs P4 in.
> Same vendor, same DMAC, same class of 8-bit port — but it is *not*
> `scsiexrom.bin`, so B3 stays open. What it changes is that a cheap
> configuration is now the thing that has to be **shown**, not assumed.
---
## M2 — a player, as opposed to a decoder
`decode.s` draws pixel-exact frames from RAM Lua pre-loaded; `stream.s` decodes
out of a bounded ring fed by a host file on a paced clock. Neither is a player.
**Exit criterion: boots from a real SCSI volume on a stock 2 MB X68000, plays
one scene at 12 fps from disc, no host-file pipe, no Lua in the loop. Silent.**
**P1. Codebook expansion on the 68000.** `dlxload.py:19` expands CB1 to 32 B per
entry and CB4 to 8 B, host-side, because at the time it was a load-time cost that
would have flattered or damned the inner loop. The player must do it: **8 KB +
2 KB per scene**. Note where that lands — *at a scene change, when the ring is
empty because of the seek*. It compounds with 51.3 and should be priced against
the refill climb, not treated as free setup.
**P2. Palette packing on the 68000.** The encoder still emits RGB888; the X68000
word packing is Lua-side. Whatever writes real palette words must pick `I` per
entry by minimum squared error (**1.96 dB**, FINDINGS 23.3) and reserve index 0
as black with `I = 0` (23.4).
**P3. A real frame clock.** `stream.s` has `PACE`/`PACEON` (`$18034`/`$18038`)
but the 12 fps tick comes from the Lua producer. Needs MFP timer or VBL. Keep
`PACEON=0` free-run working — the wrap gate uses it and every FINDINGS 49 figure
depends on it.
**P4. Real transport.** Drive the MB89352 instead of a host file. The `W`
handshake — clocks stolen per delivered byte, bracketed 5..12 by MC68450 Fig
4-25 — is listed in "Decisions locked" as UNDECIDED and as the thing that
decides the project: `W<=6` fits 0/120 frames, `W=8` misses 47/120. It is a
property of how the player drives the SPC, **so it is ours to choose, not to
receive** (FINDINGS 42.4-42.6). B3 informs it.
**Session 20 promoted this to the project's biggest open number.** FINDINGS 52.5
found the only worked example of a disk DMA configuration on this machine — the
IPL ROM's own — sitting at 16..19 clk/B, outside the bracket entirely, where the
whole design fails at any container size (`15_bus_occupancy.py` sweeps it).
`W <= 12` is now a **requirement on the player's DMAC programming**, not a range
the hardware hands us, and demonstrating a configuration that meets it is P4's
first job rather than its last.
**P5. Seek and branch.** Per-record index (the `aligned` producer needs one
anyway, 49.3), prefill policy, and the accumulated-slack rule from 51.3 made
explicit in the player rather than implied by the rig.
**P7. Boot.** The player as an executable loading from the SCSI volume.
---
## M3 — the vertical slice, and the completion target
**Exit criterion: one decision point, two outcomes, a death clip, with audio,
playing from disc on stock hardware.**
**P6. Audio — and it is the largest unpriced risk left in the project.**
MSM6258 ADPCM, 15.6 kHz mono, **7.8 KB/s**. That figure is in `ratectl.py`'s
budget and nowhere else: not extracted, not encoded, not interleaved into the
container, and **never priced on the bus**. Two reasons to treat it as a risk
rather than a task:
1. A second DMA consumer attacks **the bus** — the resource this project already
established is the binding one, at 86.7% occupied. Clock headroom says
nothing about whether it fits.
2. 7.8 KB/s is a *byte* figure. The last time a byte/word unit error went
unexamined in a delivery budget it cost the project a 2x error in every table
since FINDINGS 5 (session 14, the MB89352 being an 8-bit SPC).
~~**Price it before writing it**: add the ADPCM DMA stream to `15_bus.py` and see
what it does to the 86.7%.~~ **DONE, session 20 — FINDINGS 52.** It is in
`15_bus_occupancy.py` and the answer is **1.25%..1.48% of the frame**, about 4%
of what the decoder leaves. The per-byte cost is no longer a guess borrowed from
the disk: `tools/analysis/21_iplrom_dmac.py` reads the IPL ROM's own HD63450
configuration and finds ch3 dual-address, 8-bit port, cycle steal without hold,
external request — **16..19 clocks per byte**, where `11_cpu_budget.py` had been
charging audio the disk's 5. Both worries above resolve:
1. **The bus concern does not materialise.** A second DMA consumer at 7.8 kB/s
is not what a bus at 88% occupancy is short of.
2. **The unit was checked and is nearly right.** 15.6 kHz = 8 MHz ÷ 512 =
15,625 samples/s, 4 bits each, two to a byte = **7,812.5 B/s exactly**. The
7.8 was decimal kB being multiplied by 1024; 2.4% high, now derived from the
sample rate in `buscost.ADPCM_BYTES_PER_S`.
**What is still open in P6 is everything except the bus:** extraction, encode,
container interleave, and what a second stream does to `wire` — and therefore to
`pipe - wire`, and therefore to 51.3's refill climb. That last one is the
interaction to price next, and it is E2's question with a second consumer in it.
**E6. Container v2** — audio interleave, per-record index, scene table. Depends
on P6's answer and on P5's index.
**G1. Import the scene graph — early, because it is a measurement input.**
SNES project `data/events/` (MIT, cleared) diffed against DirkSimple (zlib),
which transcribed the same data independently, to catch transcription errors
before anything reaches 68000 tables. **Neither is on this box** — both need
fetching.
The reason to pull this ahead of the game logic that consumes it: 51.3 says
4.83 s of play to refill a 256 KB ring at 488 KB/s, and Dragon's Lair's decision
points are seconds apart. **Nothing in this tree can currently say what the worst
gap between consecutive decision points is** — only the scene table knows, and
until it is imported, whether this design survives a back-to-back branch is an
open question nobody is able to ask.
---
## M4 — the whole game
Listed for completeness; past M3 these are scope, not risk.
- **C1. Full-disc survey**, 22.8 minutes. Classify **content / menu / bonus** —
not menu vs content: the two largest streams are bonus material and look like
content by size, duration and bitrate alike (25.1). Run
`07_motion_survey.py` per stream first for a hot-window shortlist.
**Gated by E4.**
- **E4. `H.build` k-means**, 51 s of a 55 s run, once per scene. The thing to
attack before C1, and not anything in the per-frame path (27.6).
- **E2. `--spans all` as default.** Still a recommendation, not a measurement
(43.6.1), and the only loaded lever on the encoder's byte side (44.3). **It
spends every profitable byte, which raises `wire`, which shrinks `pipe - wire`,
which lengthens the refill climb after every branch.** That interaction is not
priced, and M3 is where it becomes measurable.
- **E3. Re-derive span selection jointly with `lam`** (39.3).
- **C2. Framing** — crop vs squash vs wide (FINDINGS 12). Needs an eyeball
against arcade reference, not a measurement. Cheap; blocks only final encodes.
- **C3. Disk image packaging**, ~1.09 GiB at the candidate rate.
- **G2/G3.** Branching, input windows, death clips, attract mode; playtest.
---
## Dependency summary
```
B1 seek+rate ─┐
B3 DTYP ──────┴─> P4 transport ─┐
├─> M2 ─> M3 (COMPLETION TARGET) ─> M4
P1 P2 P3 P5 P7 ─────────────────┘ ^
│
P6 (bus cost DONE, 52) ──────────────────┤
G1 scene graph (fetch, do early) ─────────┘
B2 blanking ─> (page 1; do not pre-build on it)
```
## Standing rules that apply to all of it
- **Green light first and last.** `./tools/bench/check.sh`, ALL GREEN, before and
after. **Never two MAME jobs at once** — session 18 did it, two `decode.lua`
runs shared a log file, and it produced a 0-byte log and 15 wasted minutes.
- **Name the layer.** Emulated, or real hardware. Every progress claim.
- **Label measured / estimated / folklore.** A rate with no provenance is
folklore even when it is plausible, and this project has already paid for that
twice.
- **No new default constants.** Rates stay explicit arguments. If a measurement
is not available, report the sensitivity across several rates rather than
picking one.
docs/STATUS.md
+1220 -91
View File
File diff suppressed because it is too large Load Diff
docs/img/decoded-frame.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

docs/img/modes.webm
BIN
View File
Binary file not shown.
docs/img/player.webm
BIN
View File
Binary file not shown.
docs/img/source-vs-decoded.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 182 KiB

src/player/decode.s
+37 -116
View File
@@ -38,6 +38,39 @@
; paid only by blocks that are NOT all-SKIP: a header byte of zero clears four
; blocks with one tst.b, and SKIP is the median block.
;
; LITERAL SPANS (v7, FINDINGS 40). A run of horizontally adjacent dirty blocks
; is cheaper to paint as four ROW-LINEAR runs of word-expanded literal pixels
; than as blocks: 226 clocks per 4x4 block at a run of 4, against V1's 299.9,
; and the break-even is a run of 2. The run's blocks read SKIP in the mode
; header and the span section paints them instead, so the block loop below is
; unchanged -- it sees a SKIP and advances, exactly as it does for a genuinely
; held block.
;
; The section sits BETWEEN the mode header and the block payload because that is
; the only place the 68000 can reach without first parsing something of variable
; length: the header is a fixed 768 bytes. Per span the record is {u32 absolute
; GVRAM address, u16 coarse displacement}, then the coarse pixels, then a u16
; FINE displacement, then the fine pixels.
;
; The two displacements are jumps into two unrolled copy chains -- 24 pixels per
; coarse unit (a 12-register movem pair) and 2 per fine unit (one
; `move.l (a0)+,(a2)+`) -- so a span of any length is straight-line code with no
; loop, no remainder and no address arithmetic. A run of 4x4 blocks is always a
; multiple of 4 pixels long, and 4 is a multiple of the 2-pixel fine quantum, so
; NOTHING is padded (FINDINGS 40.3).
;
; The fine displacement is in the STREAM rather than in the span record because
; that is what pays for the second dispatch: when the coarse chain falls out
; into `move.w (a0)+,d0 / jmp`, d0 is dead payload and a0 is already pointing at
; it, so the decoder holds nothing extra across the copy and keeps all twelve
; payload registers (FINDINGS 40.4). Twelve is why the coarse unit is 24 pixels
; and not V5's 16, and it is the whole reason the per-pixel cost is 9.143 rather
; than 10.459 (FINDINGS 30.4).
;
; a1 (the mode header cursor) is one of those twelve, so it goes on the stack
; across the span pass. Two long accesses per frame, against the 24 pixels a
; register buys per chain unit.
;
; ALIGNMENT. Frame records are [u32 length][768-byte mode header][payload] laid
; end to end, and payload lengths are arbitrary -- so record boundaries land on
; odd addresses, and `move.l (a0)+,d0` on an odd address is an ADDRESS ERROR on
@@ -55,14 +88,8 @@ FPTR = $18010 ; -> first frame record
SCR_N = $18014 ; frames remaining this pass
SCR_END = $18018 ; expected end of the current payload
CB1 = $20000 ; expanded 4x4 codebook
CB4 = $22000 ; expanded 2x2 codebook
include "src/player/geom.i"
DST0 = $C08000 ; GVRAM + 32*1024 (first picture row)
DSTE = $C38000 ; GVRAM + 224*1024 (one past last)
BROW = 4096 ; bytes per block row (4 picture rows)
ROWLEN = 512 ; bytes per block row of blocks (64 * 8)
MODEB = 768 ; packed mode header, 3072 blocks * 2 bits
org $10000
start:
@@ -75,7 +102,8 @@ frameloop:
lea 0(a0,d0.l),a1
move.l a1,SCR_END.l ; where the payload must end
move.l a0,a1 ; a1 = packed mode header
lea MODEB(a0),a0 ; a0 = payload
lea MODEB(a0),a0 ; a0 = span section
bsr paint_spans ; -> a0 = block payload, a1 preserved
bsr decode_frame
cmpa.l SCR_END.l,a0 ; bitstream desync is silent otherwise
bne desync
@@ -92,111 +120,4 @@ hold: bra.s hold
desync: move.l #$EE,FLAG.l
bra.s hold
; ---------------------------------------------------------------- one block
; \1 = right-shift needed to bring this block's 2 mode bits to bits 1-0.
BLOCK macro
move.b (a1),d0
ifne \1
lsr.b #\1,d0
endc
and.w #3,d0
beq .sk\@ ; 00 SKIP -- the median block
subq.w #1,d0
beq .v1\@ ; 01 V1
subq.w #1,d0
bne .rw\@ ; 11 RAW, else 10 V4
; -- V4: four 2x2 codewords, sub-block order TL TR BL BR (vq_hybrid.paint)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,(a4)
move.l d1,1024(a4)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,4(a4)
move.l d1,1028(a4)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,2048(a4)
move.l d1,3072(a4)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,2052(a4)
move.l d1,3076(a4)
bra .sk\@
; -- V1: one 4x4 codeword, 32 bytes, straight out of the expanded codebook
.v1\@:
moveq #0,d0
move.b (a0)+,d0
lsl.w #5,d0
movem.l (a2,d0.w),d0-d7 ; EA is resolved before the load
movem.l d0-d1,(a4)
movem.l d2-d3,1024(a4)
movem.l d4-d5,2048(a4)
movem.l d6-d7,3072(a4)
bra .sk\@
; -- RAW: 16 literal palette indices. Two indices are assembled into one long
; via swap, so each pair of pixels costs one write instead of two; the high
; byte of each word is left as zero because the hardware discards it anyway.
.rw\@:
RAWPAIR 0
RAWPAIR 4
RAWPAIR 1024
RAWPAIR 1028
RAWPAIR 2048
RAWPAIR 2052
RAWPAIR 3072
RAWPAIR 3076
.sk\@:
addq.l #8,a4
endm
RAWPAIR macro
moveq #0,d0
move.b (a0)+,d0
swap d0
move.b (a0)+,d0
move.l d0,\1(a4)
endm
; ------------------------------------------------------------- one frame
; in: a0 = payload, a1 = packed mode header
; out: a0 = one past the last payload byte consumed
decode_frame:
lea CB1,a2
lea CB4,a3
lea DST0,a6
rowloop:
move.l a6,a4
lea ROWLEN(a6),a5
byteloop:
tst.b (a1) ; four SKIPs in one test
beq allskip
BLOCK 6
BLOCK 4
BLOCK 2
BLOCK 0
addq.l #1,a1
cmpa.l a5,a4
bne byteloop
bra rowdone
allskip:
addq.l #1,a1
lea 32(a4),a4
cmpa.l a5,a4
bne byteloop
rowdone:
lea BROW(a6),a6
cmpa.l #DSTE,a6
bne rowloop
rts
include "src/player/frame.i"
src/player/frame.i
+183
View File
@@ -0,0 +1,183 @@
; ---------------------------------------------------------------- one block
; \1 = right-shift needed to bring this block's 2 mode bits to bits 1-0.
BLOCK macro
move.b (a1),d0
ifne \1
lsr.b #\1,d0
endc
and.w #3,d0
beq .sk\@ ; 00 SKIP -- the median block
subq.w #1,d0
beq .v1\@ ; 01 V1
subq.w #1,d0
bne .rw\@ ; 11 RAW, else 10 V4
; -- V4: four 2x2 codewords, sub-block order TL TR BL BR (vq_hybrid.paint)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,(a4)
move.l d1,1024(a4)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,4(a4)
move.l d1,1028(a4)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,2048(a4)
move.l d1,3072(a4)
moveq #0,d0
move.b (a0)+,d0
lsl.w #3,d0
movem.l (a3,d0.w),d0-d1
move.l d0,2052(a4)
move.l d1,3076(a4)
bra .sk\@
; -- V1: one 4x4 codeword, 32 bytes, straight out of the expanded codebook
.v1\@:
moveq #0,d0
move.b (a0)+,d0
lsl.w #5,d0
movem.l (a2,d0.w),d0-d7 ; EA is resolved before the load
movem.l d0-d1,(a4)
movem.l d2-d3,1024(a4)
movem.l d4-d5,2048(a4)
movem.l d6-d7,3072(a4)
bra .sk\@
; -- RAW: 16 literal palette indices. Two indices are assembled into one long
; via swap, so each pair of pixels costs one write instead of two; the high
; byte of each word is left as zero because the hardware discards it anyway.
.rw\@:
RAWPAIR 0
RAWPAIR 4
RAWPAIR 1024
RAWPAIR 1028
RAWPAIR 2048
RAWPAIR 2052
RAWPAIR 3072
RAWPAIR 3076
.sk\@:
addq.l #8,a4
endm
RAWPAIR macro
moveq #0,d0
move.b (a0)+,d0
swap d0
move.b (a0)+,d0
move.l d0,\1(a4)
endm
; ------------------------------------------------------- the span section
; in: a0 = span section, a1 = mode header (preserved across the call)
; out: a0 = one past the section, i.e. the block payload
;
; This is tools/bench/blit.s v7 verbatim, and deliberately so: the 66.0 clocks
; per span + 9.143 per coarse pixel + 9.978 per fine pixel of FINDINGS 40 were
; measured on exactly this instruction sequence, over thirteen span lengths, and
; a "tidier" rewrite here would silently invalidate every span figure in
; FINDINGS 39/40 and in tools/analysis/14_dmac_chain.py.
;
; The fine chain is entered by FALLING OUT of the coarse one, so a span with no
; coarse units enters at v7cx with d0 already reloaded -- which is why the
; coarse displacement for c=0 is SPCN*SPCU, one past the last coarse unit,
; rather than a special case.
paint_spans:
move.w (a0)+,d7 ; spans in this frame
subq.w #1,d7
bmi spnone ; a frame may legitimately have none (the
; chain is far past a short branch)
move.l a1,-(sp) ; a1 is a payload register below
spspan: move.l (a0)+,a2 ; absolute GVRAM destination
move.w (a0)+,d0 ; (SPCN - coarse) * SPCU
jmp spch(pc,d0.w)
spch:
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
spcx: move.w (a0)+,d0 ; (SPFN - fine) * SPFU, from mid-stream
jmp spfh(pc,d0.w)
spfh:
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
dbra d7,spspan
move.l (sp)+,a1
spnone: rts
; ------------------------------------------------------------- one frame
; in: a0 = payload, a1 = packed mode header
; out: a0 = one past the last payload byte consumed
decode_frame:
lea CB1,a2
lea CB4,a3
lea DST0,a6
rowloop:
move.l a6,a4
lea ROWLEN(a6),a5
byteloop:
tst.b (a1) ; four SKIPs in one test
beq allskip
BLOCK 6
BLOCK 4
BLOCK 2
BLOCK 0
addq.l #1,a1
cmpa.l a5,a4
bne byteloop
bra rowdone
allskip:
addq.l #1,a1
lea 32(a4),a4
cmpa.l a5,a4
bne byteloop
rowdone:
lea BROW(a6),a6
cmpa.l #DSTE,a6
bne rowloop
rts
src/player/geom.i
+27
View File
@@ -0,0 +1,27 @@
; Geometry and codebook constants shared by every front-end in src/player/.
;
; Split out of decode.s in session 18 so that decode.s (the preloaded-stream
; rig, gated by tools/bench/check.sh) and stream.s (the ring-buffer streaming
; rig, FINDINGS 49) assemble from LITERALLY THE SAME BYTES for the block loop
; and the span chain. Those bytes are not incidental: the 66.0 clocks/span,
; 9.143 clocks/coarse pixel and 9.978 clocks/fine pixel of FINDINGS 40, and
; every per-block constant in FINDINGS 24/30/41, are fitted to this exact
; instruction sequence. Two hand-maintained copies of it would drift, and the
; drift would be invisible -- both would still decode correctly, and only the
; cost model would be wrong.
;
; The split is a no-op by construction: tools/bench/check.sh asserts that
; decode.s still assembles to the same 1,296 bytes it did before it.
CB1 = $20000 ; expanded 4x4 codebook
CB4 = $22000 ; expanded 2x2 codebook
DST0 = $C08000 ; GVRAM + 32*1024 (first picture row)
DSTE = $C38000 ; GVRAM + 224*1024 (one past last)
BROW = 4096 ; bytes per block row (4 picture rows)
ROWLEN = 512 ; bytes per block row of blocks (64 * 8)
MODEB = 768 ; packed mode header, 3072 blocks * 2 bits
SPCU = 12 ; bytes of code per COARSE span unit (24 px)
SPCN = 11 ; coarse units: 11*24 = 264 px >= one row
SPFU = 2 ; bytes of code per FINE span unit (2 px)
SPFN = 11 ; fine units: 11*2 = 22 px > one coarse unit
src/player/stream.s
+184
View File
@@ -0,0 +1,184 @@
; DLX3 frame decoder, RING-BUFFER front-end -- STATUS item 3, FINDINGS 49.
;
; src/player/decode.s decodes a stream that is ALREADY WHOLLY IN RAM: the rig
; preloads 5,261,814 bytes at $30000 and walks a0 forward through all of it.
; That gate proves the decoder is pixel-exact over a 120-frame window
; (FINDINGS 45) and says NOTHING about how the bytes got there. The shipping
; player never holds a window at once; it streams from a SCSI disk into a ring
; a fraction of the size. Nothing in this tree has ever tested that path.
;
; WHAT IS ACTUALLY HARD ABOUT IT. The block loop and the span chain read the
; stream with a monotonically increasing a0 and no bounds check anywhere --
; `move.l (a0)+,d0`, `lea MODEB(a0),a0`, eleven unrolled `movem.l (a0)+`, a
; `move.b (a0)+` per block index. None of it can survive an address that wraps
; mid-record. So the ring does not merely need ENOUGH BYTES resident by the
; deadline -- 09_buffer_sim.py's question, and FINDINGS 21's answer -- it needs
; the WHOLE NEXT RECORD resident and CONTIGUOUS.
;
; THE WRAP POLICY IS `aligned`, and it was chosen on measurement, not taste
; (tools/analysis/19_ring_stream.py). The producer refuses to start a record it
; cannot finish before the end of the ring: it leaves a hole and restarts at 0.
;
; aligned costs RAM -- a mean hole of 23.4 KB in a 256 KB ring, 9.1% of it --
; and ZERO CPU.
; split lets records wrap and mirrors the ring's first MAXREC bytes into a
; shadow past its end, so any record start reads linearly. Costs
; zero RAM and 46,394 clocks/frame of memcpy -- 5.57% of the frame
; budget, forever.
;
; Both figures are for s14_d5_all1500, the shipping candidate, in a 256 KB ring;
; they scale with record size, so they are per container, not universal. The
; lighter gate container makes it 5.7% of the ring against 3.64% of the budget --
; same direction, same verdict.
;
; The decoder already spends 77.0% of the budget on the mean frame and 91.1% at
; p90 (FINDINGS 45). 5.57% more puts p90 at 96.7%. RAM is the resource this
; machine has 2 MB of and clocks are the one it has none of, so the trade is not
; close. `aligned` also needs a per-record INDEX on the fill side, which a
; BRANCHING laserdisc game needs anyway to seek to a branch point -- so the
; policy that costs no clocks also reuses a structure the player cannot avoid.
;
; The third option -- teach the block loop to wrap its own reads -- is the
; expensive one and not because of the branch. A bounds test lands INSIDE the
; instruction sequences FINDINGS 30.4 and 40 fitted their constants to, so it
; does not cost a compare, it costs every span and per-block figure in the tree
; being re-measured.
;
; THE PRODUCER IS OUTSIDE THIS FILE. Here it is tools/bench/stream.lua playing
; a SCSI disk at a modelled byte rate; in the player it is the MB89352 and a
; DMAC channel. The handshake is deliberately the same either way:
;
; producer -> FR_HEAD count of records made wholly resident (monotonic)
; DESC[] ring of record base addresses, DESCN entries
; decoder -> FR_TAIL count of records consumed (monotonic)
; RD_PTR one past the last byte read; everything below is free
;
; Two monotonic counters and a released-to pointer -- no lock, no shared cursor,
; single reader and single writer, so it is correct on a 68000 with no atomics
; provided each side only ever writes its own words. That is why FR_TAIL is
; the decoder's and FR_HEAD is the producer's rather than one shared index.
;
; STALLS ARE COUNTED, NOT HIDDEN. A frame whose record is not resident when the
; decoder wants it spins in `waitrec`, and STALLS counts the FRAMES that had to
; wait at all (not the polls). A rig that silently absorbed an underrun would
; report a pixel-exact decode of a stream that arrived late, which is precisely
; the failure this front-end exists to make visible. The spin is bounded:
; SPINMAX polls without progress sets FLAG=$E1, so a wedged producer fails as a
; wedged producer instead of as a MAME timeout with no diagnosis (FINDINGS 34.1).
FLAG = $18000 ; 0 idle / 1 running / $FF done / $EE desync
; / $E1 producer stalled out
ITER = $18008 ; outer repeat count, written by Lua
NFR = $1800C ; frames per pass
SCR_N = $18014 ; frames remaining this pass
SCR_END = $18018 ; expected end of the current payload
RD_PTR = $18020 ; decoder -> producer: released up to here
FR_HEAD = $18024 ; producer -> decoder: records resident
FR_TAIL = $18028 ; decoder -> producer: records consumed
STALLS = $1802C ; frames that had to wait for their record
SPINS = $18030 ; total poll iterations spent waiting
PACE = $18034 ; producer -> decoder: frame ticks elapsed since
; release. Frame i may not START before tick i.
PACEON = $18038 ; 1 = obey PACE. 0 leaves the loop free-running,
; byte for byte the loop FINDINGS 49 measured.
DESC = $18100 ; DESCN x u32, record base addresses
DESCN = 64 ; power of two; the index is masked, not compared
DESCM = (DESCN-1)*4 ; mask for a BYTE offset into DESC
SPINMAX = 2000000 ; polls with no progress before giving up
include "src/player/geom.i"
org $10000
start:
move.l #1,FLAG.l ; timer starts here
outer:
move.l NFR.l,SCR_N.l
clr.l FR_TAIL.l
clr.l STALLS.l
clr.l SPINS.l
frameloop:
; ---- PACE GATE (FINDINGS 49.7.2, and it is the whole point of this session).
; Free-running, this loop asks for record i the instant it finishes record i-1,
; so it outruns any finite pipe, the ring NEVER backs up, and the producer's
; overlap test never refuses a placement. A ring-size sweep under those
; conditions tests WRAP CORRECTNESS at each size and nothing about BUFFERING:
; 48 KB passes while holding one record. A shipping player does not do this --
; it draws frame i, waits for its slot, and spends the rest of the frame time
; idle while the disk fills the ring behind it.
;
; So the rig gets a frame clock. PACE is bumped by the producer (in the player,
; vblank or an MFP timer) and frame i is forbidden to start before tick i. With
; the decoder held to 12 fps the ring fills, the producer starts hitting its own
; overlap test, and FR_HEAD-FR_TAIL becomes what it claims to be: the number of
; whole frames the decoder could run on if delivery stopped dead -- which is the
; branch-point seek question stated in frames.
;
; It also makes STALLS mean something. Free-running, a stall is EARLINESS
; (49.6); paced, a frame that has to wait for its record is a real underrun.
tst.l PACEON.l
beq.s nopace
pacewait:
move.l PACE.l,d0
cmp.l FR_TAIL.l,d0 ; d0 - FR_TAIL; carry = tick not reached
bcs.s pacewait
nopace:
; ---- wait until the producer has made this record wholly resident.
; d1 counts polls for this frame; a nonzero d1 on exit means the frame stalled.
moveq #0,d1
move.l FR_TAIL.l,d2
waitrec:
move.l FR_HEAD.l,d0
cmp.l d2,d0
bhi.s gotrec ; HEAD > TAIL: at least one record ready
addq.l #1,d1
cmp.l #SPINMAX,d1
bcs.s waitrec
move.l #$E1,FLAG.l ; producer never delivered
bra hold
gotrec:
tst.l d1
beq.s nostall
addq.l #1,STALLS.l
add.l d1,SPINS.l
nostall:
; ---- pop the descriptor. DESCN is a power of two, so the wrap is an and.
move.l d2,d0
lsl.l #2,d0
and.w #DESCM,d0
lea DESC,a1 ; DESC is absolute; (d0.w) needs a base
move.l (a1,d0.w),a0 ; a1 is reloaded from a0 two lines below
; ---- from here to the release, byte for byte what decode.s does. a0 is
; inside the ring rather than inside a preloaded blob, and the block loop
; cannot tell the difference -- which is the whole claim being tested.
move.l (a0)+,d0 ; u32 payload length, big-endian
lea 0(a0,d0.l),a1
move.l a1,SCR_END.l ; where the payload must end
move.l a0,a1 ; a1 = packed mode header
lea MODEB(a0),a0 ; a0 = span section
bsr paint_spans ; -> a0 = block payload, a1 preserved
bsr decode_frame
cmpa.l SCR_END.l,a0 ; bitstream desync is silent otherwise
bne desync
; ---- release. Round up to 4 the same way decode.s does: the producer lays
; records on 4-byte boundaries, so the byte one past this record's padded
; end is the first byte the producer may reuse.
move.l a0,d0
addq.l #3,d0
and.b #$FC,d0
move.l d0,RD_PTR.l
addq.l #1,FR_TAIL.l
subq.l #1,SCR_N.l
bne frameloop
subq.l #1,ITER.l
bne outer
move.l #$FF,FLAG.l ; timer stops here
hold: bra.s hold
desync: move.l #$EE,FLAG.l
bra.s hold
include "src/player/frame.i"
tools/analysis/07_motion_survey.py
+4 -2
View File
@@ -13,10 +13,12 @@ window it picks then gets encoded for real.
Usage: python3 tools/analysis/07_motion_survey.py 00223 [window_seconds]
"""
import subprocess, sys
import subprocess, sys, os, getpass
import numpy as np
STREAM_DIR = "/media/reala-misaki/BDROM/BDMV/STREAM"
# See tools/encoder/extract.py: DLX_BDROM overrides where the disc is mounted.
BDROM = os.environ.get("DLX_BDROM") or f"/media/{getpass.getuser()}/BDROM"
STREAM_DIR = f"{BDROM}/BDMV/STREAM"
W, H, FPS = 96, 72, 12
def frames(stream):
tools/analysis/08_mode_map.py
+2 -2
View File
@@ -16,7 +16,7 @@ the mode headers would exploit.
RAW 16 literal palette indices -- the escape that makes lam=0 pixel-exact
Usage: python3 tools/analysis/08_mode_map.py <frames_dir> <out.webm>
[--profile sasi|scsi] [--scale N] [--lossless] [--fixed-lam]
[--profile scsi] [--scale N] [--lossless] [--fixed-lam]
--fixed-lam renders the pre-session-6 encoder (no rate control) instead.
Output format follows the extension. Prefer .webm: GIF re-quantises to 256
@@ -45,7 +45,7 @@ def main():
if "--scale" in sys.argv:
SCALE = int(sys.argv[sys.argv.index("--scale")+1])
prof = RC.PROFILES[sys.argv[sys.argv.index("--profile")+1]
if "--profile" in sys.argv else "sasi"]
if "--profile" in sys.argv else "scsi"]
m = H.build(src, k1=prof["k1"], k4=prof["k4"])
# Rate-controlled by default, so the map shows the mode decisions that
# actually ship. --fixed-lam renders the pre-session-6 encoder instead;
tools/analysis/08_peaks.py
+1 -1
View File
@@ -1,4 +1,4 @@
"""At a 488 KB/s (4 Mbps) ceiling and ~52% mean utilisation, the mean is not the
"""At any fixed delivery ceiling and ~52% mean utilisation, the mean is not the
risk -- the peaks are. Measure per-frame peak-to-mean, then check whether the
leaky-bucket rate controller actually holds the ceiling."""
import sys; sys.path.insert(0,'tools/encoder')
tools/analysis/09_ratectl_drift.py
+41 -23
View File
@@ -19,6 +19,10 @@ paint), so drift is zero by construction rather than by tuning.
This replays what a real decoder does -- SKIP copies the ACTUALLY EMITTED
previous frame -- and compares it to the reconstruction ratectl recorded.
Session 8 added a SECOND controller (mu, the per-frame 68000 decode ceiling)
that also varies the mode map frame to frame, so it is exposed to exactly the
same failure and is tested here too. Both configurations must show zero drift.
Needs tmp/fr_singe (see docs/STATUS.md, reproducing the sustained-action
result). ~55 s, nearly all of it the k-means in H.build; the rate-controlled
encode of 120 frames is ~2 s.
@@ -29,20 +33,27 @@ import numpy as np
import vq as VQ, vq_hybrid as H, ratectl as RC
m = H.build("tmp/fr_singe", k1=256, k4=256, iters=16)
# lam_lo=1.0: let quiet frames spend the whole allowance, which is the
# harder case for this test -- it maximises how often lam moves frame to frame.
enc = RC.encode_rate_controlled(m, target_kbps=110, lam_lo=1.0)
lam = enc["lam"]
sw = int((np.diff(lam) != 0).sum())
print(f"frames={len(lam)} distinct lam used={len(set(lam.tolist()))} "
def check(label, cycle_budget):
"""Encode, replay as a decoder would, and return the drift in pixels."""
print(f"\n=== {label} ===")
m.pop("_sym", None)
# lam_lo=1.0: let quiet frames spend the whole allowance, which is the
# harder case for this test -- it maximises how often lam moves frame to
# frame.
enc = RC.encode_rate_controlled(m, target_kbps=110, lam_lo=1.0,
cycle_budget=cycle_budget)
lam = enc["lam"]
sw = int((np.diff(lam) != 0).sum())
print(f"frames={len(lam)} distinct lam used={len(set(lam.tolist()))} "
f"lam changes frame-to-frame={sw} "
f"overruns={int(enc['overrun'].sum())}")
pal, nbx = m["pal"], m["W"] // 4
emitted = []
drift_px, drift_db = [], []
for f, (rec, mode) in enumerate(zip(enc["recon"], enc["modes"])):
pal, nbx = m["pal"], m["W"] // 4
emitted = []
drift_px, drift_db = [], []
for f, (rec, mode) in enumerate(zip(enc["recon"], enc["modes"])):
out = rec.copy()
if f > 0:
prev_true = emitted[-1]
@@ -55,23 +66,30 @@ for f, (rec, mode) in enumerate(zip(enc["recon"], enc["modes"])):
drift_px.append(d)
drift_db.append(VQ.psnr(pal[rec], pal[out]))
drift_px = np.array(drift_px)
print(f"pixels differing from what the encoder recorded:")
print(f" frames with ANY drift: {int((drift_px>0).sum())}/{len(drift_px)}")
print(f" max {drift_px.max()} px ({100*drift_px.max()/(m['H']*m['W']):.1f}% of frame)")
print(f" mean {drift_px.mean():.0f} px")
fin = [d for d in drift_db if np.isfinite(d)]
if fin:
drift_px = np.array(drift_px)
print(f"pixels differing from what the encoder recorded:")
print(f" frames with ANY drift: {int((drift_px>0).sum())}/{len(drift_px)}")
print(f" max {drift_px.max()} px ({100*drift_px.max()/(m['H']*m['W']):.1f}% of frame)")
print(f" mean {drift_px.mean():.0f} px")
fin = [d for d in drift_db if np.isfinite(d)]
if fin:
print(f" encoder-vs-decoder agreement: min {min(fin):.1f} dB "
f"(inf = identical on {len(drift_db)-len(fin)} frames)")
r = RC.summarise(m, enc, 110)
print(f"\nratectl reports PSNR {r['psnr']:.2f} dB, {r['kbps']:.1f} KB/s "
r = RC.summarise(m, enc, 110)
print(f"\nratectl reports PSNR {r['psnr']:.2f} dB, {r['kbps']:.1f} KB/s "
f"(target 110), {r['over']:.0f}% of frames over budget")
tp = np.mean([VQ.psnr(o, pal[e]) for o, e in zip(m["rgb"], emitted)])
print(f"what a decoder actually reconstructs: {tp:.2f} dB "
tp = np.mean([VQ.psnr(o, pal[e]) for o, e in zip(m["rgb"], emitted)])
print(f"what a decoder actually reconstructs: {tp:.2f} dB "
f"-> overstated by {r['psnr']-tp:.2f} dB")
return drift_px
# Acceptance criterion for the fix: a decoder replaying the emitted stream must
# reconstruct exactly what the encoder recorded.
sys.exit(1 if (drift_px > 0).any() else 0)
# reconstruct exactly what the encoder recorded -- under either controller.
bad = 0
for label, cb in (("bytes only (session 6)", None),
("bytes + CPU ceiling (session 8)", RC.FRAME_CYCLES)):
d = check(label, cb)
bad += int((d > 0).any())
sys.exit(1 if bad else 0)
tools/analysis/11_cpu_budget.py
+97 -23
View File
@@ -22,6 +22,15 @@ import sys, os, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
import vq_hybrid as H
import ratectl as RC
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import buscost as B
# The audio byte rate is now DERIVED, not restated: 15.6 kHz mono MSM6258V is
# 15,625 4-bit samples/s, two to a byte. RC.AUDIO_KBPS's 7.8 is that figure in
# DECIMAL kB, and was being multiplied by 1024 here -- a 2.4% overstatement,
# harmless, but it hid which unit the constant was in.
RC_AUDIO_BPS = B.ADPCM_BYTES_PER_S
# Machine clocks, confirmed from MAME 0.277 src/mame/sharp/x68k.cpp:1133/1194/
# 1200 -- not recalled. x68000 and x68ksupr are BOTH 40_MHz_XTAL/4 = 10 MHz;
@@ -30,10 +39,13 @@ from dlx import DLX
CLOCKS = {"stock": 10.0, "super": 10.0, "xvi": 33.33 / 2, "x68030": 25.0}
FPS = 12
# cycles per block, measured on the emulated 68000 (synthetic single-mode frames)
C_V1, C_V4, C_RAW = 299.9, 448.2, 400.4
C_SKIP_FAST = 53.0 / 4 # all-SKIP header byte: one tst.b for 4
C_SKIP_MIXED = 45.0 # a SKIP block inside a mixed byte
# Cycles per block, measured on the emulated 68000 (synthetic single-mode
# frames). Defined in tools/encoder/vq_hybrid.py, which is where the mode
# decision needs them too -- one copy, not two, so a re-measurement cannot
# leave the encoder and the scorer disagreeing.
C_V1, C_V4, C_RAW = H.C_V1, H.C_V4, H.C_RAW
C_SKIP_FAST, C_SKIP_MIXED = H.C_SKIP_CLUSTERED, H.C_SKIP_MIXED
cycles = H.cycles
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?",
@@ -41,6 +53,37 @@ ap.add_argument("container", nargs="?",
ap.add_argument("--machine", default="stock", choices=list(CLOCKS),
help="which X68000's clock to budget against (default stock)")
ap.add_argument("--fps", type=float, default=FPS)
# FINDINGS 35: the frame budget has never had the disk in it. The bitstream has
# to be moved off SCSI into the ring buffer, and on this machine that costs CPU
# whether it is DMA (the HD63450 cycle-steals) or PIO (the 68000 moves every
# byte). Default ON, because scoring a decoder against a budget that assumes the
# data arrives for free is exactly the mistake 35 was raised to stop.
ap.add_argument("--io", default="dma", choices=["dma", "pio", "none"],
help="how the bitstream reaches RAM (default dma)")
ap.add_argument("--dma-clocks-per-word", type=float, default=8.0,
help="HD63450 cycle-steal. ESTIMATE from FINDINGS 5, NEVER "
"MEASURED, and the most load-bearing unmeasured number "
"in the project (FINDINGS 35.3)")
ap.add_argument("--dma-clocks-per-byte", type=float, default=5.0,
help="what the SCSI DMA costs per DELIVERED BYTE. The MB89352 "
"is an 8-bit port, so the DMAC pays per byte and the "
"per-word denominator of FINDINGS 5/39.7 was half the "
"real debit (FINDINGS 43). 5 = single-address, bus held, "
"no drive wait; 9 = dual-address")
ap.add_argument("--pio-clocks-per-byte", type=float, default=12.0,
help="hand-derived floor for a 68000 register-to-RAM copy")
# Audio is NOT the disk, and charging it the disk's rate was charging it the
# favourable side of an open question. tools/analysis/21_iplrom_dmac.py reads
# the IPL ROM's own HD63450 setup: channel 3 is dual address, 8-bit port, cycle
# steal WITHOUT hold, external request -- one full arbitration per byte, no
# burst to amortise it over. 16 is the datasheet best case, 19 the worst.
ap.add_argument("--adpcm-clocks-per-byte", type=float,
default=B.ADPCM_CLK_BYTE_BEST,
help="what an ADPCM byte costs. READ OUT OF THE IPL ROM's DMAC "
"configuration (21_iplrom_dmac.py), not assumed: dual "
"address + per-byte arbitration = 16 best, 19 worst. The "
"audio stream always DMAs, whatever --io says about the "
"disk")
a = ap.parse_args()
CPUHZ = CLOCKS[a.machine] * 1e6
FPS = a.fps
@@ -50,25 +93,48 @@ if not os.path.exists(a.container):
d = DLX(a.container)
def cycles(mode):
g = mode.reshape(-1, 4) # one header byte = four blocks
allskip = (g == 0).all(1)
c = allskip.sum() * 4 * C_SKIP_FAST
m = g[~allskip]
c += (m == 0).sum() * C_SKIP_MIXED
c += (m == 1).sum() * C_V1
c += (m == 2).sum() * C_V4
c += (m == 3).sum() * C_RAW
return c
# --- what the transfer costs, from the container's own byte rate
vid_bps = sum(n + 4 for (_, n) in d.frames) / d.nframes * d.fps
io_bps = vid_bps + RC_AUDIO_BPS
aud_cycles_per_s = RC_AUDIO_BPS * a.adpcm_clocks_per_byte
if a.io == "dma":
io_cycles_per_s = vid_bps * a.dma_clocks_per_byte + aud_cycles_per_s
elif a.io == "pio":
io_cycles_per_s = vid_bps * a.pio_clocks_per_byte + aud_cycles_per_s
else:
io_cycles_per_s = 0.0
io_pct = 100 * io_cycles_per_s / CPUHZ
aud_pct = 100 * aud_cycles_per_s / CPUHZ
FRAME_NET = FRAME * (1 - io_pct / 100)
modes = [d.modes(f) for f in range(d.nframes)]
cyc = np.array([cycles(m) for m in modes])
pct = 100 * cyc / FRAME
pct = 100 * cyc / FRAME_NET
ns = np.array([100 * (m != 0).mean() for m in modes])
print(f"{a.container}: {d.nframes} frames, {d.nb} blocks/frame")
print(f"budget: {a.machine} @ {CLOCKS[a.machine]:.2f} MHz, {FPS:g} fps "
f"-> {FRAME:,.0f} cycles/frame")
print(f" I/O ({a.io}): {io_bps/1024:.1f} KB/s costs {io_pct:.1f}% of the CPU "
f"-> {FRAME_NET:,.0f} cycles/frame left for decoding")
if a.io != "none":
print(f" video {vid_bps/1024:6.1f} KB/s x "
f"{(a.dma_clocks_per_byte if a.io=='dma' else a.pio_clocks_per_byte):g}"
f" clk/B = {io_pct-aud_pct:5.2f}% "
f"(W: still open, ROADMAP B3 / FINDINGS 42.4)\n"
f" audio {RC_AUDIO_BPS/1024:6.2f} KB/s x {a.adpcm_clocks_per_byte:g}"
f" clk/B = {aud_pct:5.2f}% "
f"(SETTLED: read out of the IPL ROM, FINDINGS 52)")
if a.io == "dma":
print(f" {a.dma_clocks_per_byte:g} clocks/BYTE, the MC68450 datasheet "
f"floor for an 8-bit port (FINDINGS 43).\n It is not measured on "
f"hardware; what IS settled is that the per-word denominator this\n"
f" used before session 14 was physically impossible -- 2.5 "
f"clocks/byte is below\n the 68000's 4-clock minimum bus cycle.")
elif a.io == "none":
print(" WARNING: --io none scores the decoder as if the disk were free. "
"That is the\n premise FINDINGS 35 overturned; every 'N frames miss' "
"figure before session 9\n was computed this way.")
if a.machine != "stock":
print(" (derived: scaled by clock from cycles measured on the 10 MHz core.\n"
" MAME 0.277 marks x68ksupr/x68kxvi/x68030 MACHINE_NOT_WORKING, so\n"
@@ -85,15 +151,20 @@ TIMED_FRAMES = (("min non-SKIP", 15.4, 31.5), ("median", 48.1, 73.8),
("p90", 82.5, 116.4), ("max non-SKIP", 100.0, 135.8))
if (os.path.abspath(a.container) == os.path.abspath(TIMED)
and a.machine == "stock" and a.fps == 12):
print("model vs the frames actually timed on the 68000:")
print("model vs the frames actually timed on the 68000 "
"(the model reads HIGH, and by more\n as the frame gets harder -- "
"so a 'does not fit' from it is the safe direction):")
for label, frac, meas in TIMED_FRAMES:
i = int(np.argmin(abs(ns - frac)))
print(f" {label:<14} non-SKIP {ns[i]:5.1f}% model {pct[i]:6.1f}% "
f"measured {meas:5.1f}% error {pct[i]-meas:+.1f} pt")
else:
print(f"(no 68000 timings for this container/machine -- the model was "
f"validated to\n within 1 pt on {TIMED} at stock/12fps;\n"
f" run tools/bench/decode.lua to time another container)")
print(f"(no 68000 timings for this container/machine. The model is "
f"validated against four\n frames timed on the 68000, and only on "
f"{TIMED}\n at stock/12fps -- run it on that container to see the "
f"errors, which are a few points\n CONSERVATIVE and grow with the "
f"non-SKIP fraction. Run tools/bench/decode.lua to\n time another "
f"container.)")
print(f"\nper-frame cost, % of a {FPS:g}fps frame budget:")
print(f" measured-cost model: median {np.median(pct):5.1f} "
@@ -107,10 +178,11 @@ if a.machine == "stock" and a.fps == 12:
f"(optimistic by {np.median(pct)/np.median(old):.2f}x at the median)")
miss = pct > 100
print(f"\nframes that do NOT fit {FRAME:,.0f} cycles: {miss.sum()}/{d.nframes} "
print(f"\nframes that do NOT fit {FRAME_NET:,.0f} cycles: {miss.sum()}/{d.nframes} "
f"({100*miss.mean():.0f}%)")
print(f" sustainable framerate if EVERY frame must fit: "
f"{CPUHZ/cyc.max():.1f} fps; at the mean frame {CPUHZ/cyc.mean():.1f} fps")
f"{CPUHZ*(1-io_pct/100)/cyc.max():.1f} fps; at the mean frame "
f"{CPUHZ*(1-io_pct/100)/cyc.mean():.1f} fps")
if miss.any():
print(f" worst {pct.max():.1f}% -- {(pct.max()-100)/100*1000/FPS:.0f} ms late "
f"on an {1000/FPS:.0f} ms frame")
@@ -123,5 +195,7 @@ print(f"\nwhere the cycles go, over the whole window:")
for k, n in enumerate(("SKIP", "V1", "V4", "RAW")):
print(f" {n:<5} {100*tot[k]/tot.sum():5.1f}% of blocks "
f"{100*spend[k]/spend.sum():5.1f}% of the cycles")
print(f"\nV4 is {C_V4/C_V1:.2f}x a V1 block for {4}x the payload bytes -- the mode "
f"decision\nin vq_hybrid.py charges it the bytes but not the cycles.")
print(f"\nV4 is {C_V4/C_V1:.2f}x a V1 block for {4}x the payload bytes. Since "
f"session 8 the mode\ndecision charges it BOTH (decide(ctx, lam, mu), "
f"FINDINGS 31), which is why V4 is now\nthe rarest non-SKIP mode here -- "
f"a byte-rich profile buys its way out to RAW instead.")
tools/analysis/12_span_tradeoff.py
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""What does spending the idle bus bandwidth buy back in CPU cycles?
python3 tools/analysis/12_span_tradeoff.py [container.dlx] --bus <KB/s>
FINDINGS 28 leaves the decoder CPU-bound at 110 KB/s on a much wider pipe. Every
codec decision was made when bytes were scarce, so each one trades cycles to
save them -- and the cheapest thing a 68000 can be handed is the most expensive
thing to store: word-expanded pixels in row-linear runs.
This prices ONE new mode against the real mode maps: a per-row SPAN of
word-expanded literals, `movem.l`-ed straight from the stream buffer into GVRAM.
A run of L horizontally adjacent dirty blocks becomes 4 spans of 4L pixels.
MEASURED as of session 8 (FINDINGS 30), on the 68000, with the span decoder in
tools/bench/blit.s v6 and the streams in tools/bench/prep_spans.py:
43.7 cycles per span + 9.152 per pixel, fitting eleven span lengths to within
0.3%. That is the ENCODER-ASSISTED format: the record is an absolute GVRAM
address and a jump displacement into an unrolled copy chain, so the decoder does
no arithmetic per span. The obvious decoder -- handed (x, npix) and left to work
the copy out -- measures 97.9 + 10.46 and is 2.2x dearer on a 24-pixel span (v5).
Span length is therefore a multiple of 24 pixels, and a run pads up to it; the
padding is free of cycles beyond its pixels and correct on screen, because a
literal span carries true pixels of the current frame.
The mode maps are NOT re-optimised: this only re-codes regions the encoder
already chose to redraw, so it is a lower bound on what a cost-aware encoder
would find.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
FRAME_CYC = 833333.0 # 12fps at 10 MHz
AUDIO_KBPS = 7.8
C_V1, C_V4, C_RAW = 299.9, 448.2, 400.4 # FINDINGS 28.2 (measured)
C_SKIP_CLUSTERED, C_SKIP_MIXED = 13.25, 45.0
SPAN_OVERHEAD = 43.7 # per span, MEASURED, FINDINGS 30
CYC_PX_ROWLIN = 9.152 # per pixel, MEASURED, FINDINGS 30
SPAN_UNIT_PX = 24 # 12 registers of movem.l, one chain unit
SPAN_BYTES_PX = 2 # word-expanded: 1 pixel = 1 word
SPAN_HDR = 6 # u32 GVRAM address + u16 jump displacement
def span_px(npix): # a span is a whole number of units
return -(-npix // SPAN_UNIT_PX) * SPAN_UNIT_PX
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?",
default="tmp/rc_fr_singe_sasi_rcprofile.dlx")
ap.add_argument("--bus", type=float, required=True,
help="REQUIRED. There is no default: the delivery rate is a property of the medium and this project has never measured it. FINDINGS 42.1 -- the figure this tool used to default to was a user-supplied '4 Mbps' with no provenance, was a tenth of SCSI-1's asynchronous rating, and was never a bus measurement at all. A default let every table in FINDINGS 30-49 be scored against it without anyone restating it. Pass one explicitly.")
ap.add_argument("--fps", type=float, default=12.0)
a = ap.parse_args()
if not os.path.exists(a.container):
sys.exit(f"missing {a.container}")
BYTE_BUD = (a.bus - AUDIO_KBPS) * 1024 / a.fps
d = DLX(a.container)
BLK_C = {1: C_V1, 2: C_V4, 3: C_RAW}
BLK_B = {1: 1, 2: 4, 3: 16}
rows = []
for f in range(d.nframes):
mode = d.modes(f)
g = mode.reshape(-1, 4)
allskip = (g == 0).all(1)
base = allskip.sum() * 4 * C_SKIP_CLUSTERED
mm = g[~allskip]
base += (mm == 0).sum() * C_SKIP_MIXED
for k, c in BLK_C.items():
base += (mm == k).sum() * c
base_b = d.mode_bytes + sum(BLK_B.get(int(x), 0) for x in mode)
m = mode.reshape(d.nby, d.nbx)
cand = []
for by in range(d.nby):
dirty = m[by] != 0
i = 0
while i < d.nbx:
if not dirty[i]:
i += 1
continue
j = i
while j < d.nbx and dirty[j]:
j += 1
L = j - i
cur_c = sum(BLK_C[int(b)] for b in m[by][i:j])
cur_b = sum(BLK_B[int(b)] for b in m[by][i:j])
sp = span_px(4 * L) # padded to the chain's 24-pixel unit
span_c = 4 * (SPAN_OVERHEAD + sp * CYC_PX_ROWLIN)
span_b = 4 * (SPAN_HDR + sp * SPAN_BYTES_PX)
if span_c < cur_c:
cand.append((cur_c - span_c, span_b - cur_b, L))
i = j
cand.sort(key=lambda s: -(s[0] / max(s[1], 1))) # best cycles per byte
cyc, byt, taken = base, base_b, 0
for dc, db, L in cand:
if byt + db <= BYTE_BUD:
cyc -= dc; byt += db; taken += 1
rows.append((base, cyc, base_b, byt, len(cand), taken))
base, new, bb, nb, ncand, ntaken = map(np.array, list(zip(*rows)))
pc = lambda v: 100 * v / FRAME_CYC
print(f"{a.container}: {d.nframes} frames")
print(f"bus {a.bus:.0f} KB/s - {AUDIO_KBPS} audio -> {BYTE_BUD:,.0f} B/frame "
f"at {a.fps:g}fps\n")
print(f"{'':<26}{'today':>12}{'+ literal spans':>18}")
for label, fn in (("median frame", np.median),
("p90 frame", lambda v: np.percentile(v, 90)),
("worst frame", np.max)):
print(f" {label:<24}{pc(fn(base)):>11.1f}%{pc(fn(new)):>17.1f}%")
print(f" {'frames missing budget':<24}{int((base>FRAME_CYC).sum()):>8}/{d.nframes}"
f"{int((new>FRAME_CYC).sum()):>14}/{d.nframes}")
print(f" {'bitrate':<24}{bb.mean()*a.fps/1024:>10.1f} KB/s"
f"{nb.mean()*a.fps/1024:>13.1f} KB/s")
print(f"\nspans taken: {ntaken.sum()} of {ncand.sum()} candidate runs "
f"({100*ntaken.sum()/max(ncand.sum(),1):.0f}%) -- the rest priced out by the bus")
brk = next(L for L in range(1, 65)
if 4*(SPAN_OVERHEAD + span_px(4*L)*CYC_PX_ROWLIN) < L*C_V1)
print(f"\nspan cost MEASURED (FINDINGS 30): {SPAN_OVERHEAD:.1f}/span + "
f"{CYC_PX_ROWLIN:.3f}/pixel, {SPAN_UNIT_PX}-pixel units.")
print(f"a run of L blocks beats all-V1 from L={brk} blocks up "
f"({4*(SPAN_OVERHEAD + span_px(4*brk)*CYC_PX_ROWLIN)/brk:.0f} vs {C_V1:.0f} "
f"cycles/block); the floor at a full row is "
f"{4*(SPAN_OVERHEAD + span_px(256)*CYC_PX_ROWLIN)/64:.0f}.")
print("The mode maps are NOT re-optimised, so this is a lower bound on a "
"cost-aware encoder.")
# FINDINGS 28.5 said a scene cut cannot fit at 12fps: the cheapest full redraw
# the codec's mode set allows is all-V1 at 110.5% of budget. 29.4 reopened that
# on derived span costs; this is the same arithmetic on measured ones. Mix a
# fraction x of a 100%-changed frame as full-row spans, V1 for the rest.
NB = d.nb
row_c = 4 * (SPAN_OVERHEAD + span_px(4 * d.nbx) * CYC_PX_ROWLIN) / d.nbx
row_b = 4 * (SPAN_HDR + span_px(4 * d.nbx) * SPAN_BYTES_PX) / d.nbx
x_cpu = (NB * C_V1 - FRAME_CYC) / (NB * (C_V1 - row_c))
x_bus = (BYTE_BUD - d.mode_bytes - NB * BLK_B[1]) / (NB * (row_b - BLK_B[1]))
print(f"\nscene cut (100% of blocks change), spans at full row width "
f"({row_c:.0f} cyc, {row_b:.1f} B per block):")
print(f" all-V1 costs {100*NB*C_V1/FRAME_CYC:.1f}% of the frame -- FINDINGS 28.5")
print(f" CPU needs x >= {x_cpu:.3f} of the frame as spans; "
f"the bus allows x <= {x_bus:.3f}")
print(" " + ("the interval is NOT empty: a cut fits at 12fps (FINDINGS 29.4 holds)"
if x_cpu <= x_bus else
"the interval IS empty: a cut does not fit (FINDINGS 28.5 stands)"))
tools/analysis/13_cpu_ratectl.py
+103
View File
@@ -0,0 +1,103 @@
#!/usr/bin/env python3
"""What does fitting the CPU budget cost in quality? (session 8, lever B)
python3 tools/analysis/13_cpu_ratectl.py [frames_dir] [--profiles scsi]
Session 6 made the BYTE budget a ceiling by bisecting `lam` per frame. FINDINGS
28 then showed the binding budget is CYCLES, not bytes, and that the mode
decision cannot see them: it minimises `D + lam*R` on a machine that charges V4
1.49x a V1 block while the lagrangian charges it 4x.
`ratectl.encode_rate_controlled(cycle_budget=...)` adds the second controller --
`mu` bisected per frame against 833,333 cycles, with the lam bisection nested
inside it. This measures what that costs: PSNR, bitrate, and how many frames
still miss, against the same encode with the ceiling off.
The cycle budget is HARD, not a bucket. Bytes bank in the player's ring buffer;
there is no double buffer to decode ahead into, so a frame that misses its
decode deadline is simply late (FINDINGS 28).
Both controllers score frames with the exact clustered cost `vq_hybrid.cycles`,
validated to 1 point against the 68000 (FINDINGS 28.2) -- not with the per-block
ranking constant the mode decision uses. See vq_hybrid's note on SKIP.
"""
import argparse, os, pickle, sys, time
sys.path.insert(0, "tools/encoder")
import numpy as np
import vq as VQ, vq_hybrid as H, ratectl as RC
ap = argparse.ArgumentParser()
ap.add_argument("frames_dir", nargs="?", default="tmp/fr_singe")
ap.add_argument("--profiles", default="scsi")
ap.add_argument("--fps", type=int, default=12)
ap.add_argument("--cache", default=None, help="pickle of H.build (auto by dir)")
a = ap.parse_args()
if not os.path.isdir(a.frames_dir):
sys.exit(f"missing {a.frames_dir} -- see tools/bench/check.sh for extraction")
BUDGET = RC.FRAME_CYCLES
# H.build is ~55 s, nearly all k-means, and it does not depend on the profile:
# both ship k1=k4=256. One build, cached, serves every row of the table.
cache = a.cache or f"tmp/model_{os.path.basename(a.frames_dir.rstrip('/'))}.pkl"
if os.path.exists(cache):
m = pickle.load(open(cache, "rb"))
print(f"model from {cache}")
else:
t = time.time()
m = H.build(a.frames_dir, k1=256, k4=256, iters=16)
pickle.dump(m, open(cache, "wb"))
print(f"built model in {time.time()-t:.0f} s -> {cache}")
print(f"{a.frames_dir}: {len(m['idx'])} frames, {m['nb']} blocks, "
f"budget {BUDGET:,.0f} cycles/frame at {a.fps}fps\n")
def run(prof_name, cycle_budget):
p = RC.PROFILES[prof_name]
m.pop("_sym", None) # the frame-symbol cache holds one frame
t = time.time()
enc = RC.encode_rate_controlled(m, p["kbps"], fps=a.fps, lam_lo=p["lam"],
cycle_budget=cycle_budget)
s = RC.summarise(m, enc, p["kbps"], fps=a.fps)
s["secs"] = time.time() - t
s["ns"] = float(np.mean([100*(mm != 0).mean() for mm in enc["modes"]]))
return s, enc
rows = []
for name in a.profiles.split(","):
for label, cb in (("bytes only", None), ("bytes + cycles", BUDGET)):
s, enc = run(name, cb)
rows.append((name, label, s))
print(f"{name:5s} {label:<15s} {s['secs']:5.1f} s "
f"PSNR {s['psnr']:.2f} dB {s['kbps']:6.1f} KB/s "
f"CPU med {100*s['cyc_med']/BUDGET:5.1f}% p90 "
f"{100*s['cyc_p90']/BUDGET:5.1f}% max {100*s['cyc_max']/BUDGET:5.1f}% "
f"miss {s['cpu_miss']:3d} late {s['late']:2d} "
f"mu med {s['mu_med']:.4f} max {s['mu_max']:.3f}")
print()
hdr = f"{'':<22}{'PSNR':>8}{'KB/s':>9}{'CPU med':>10}{'CPU max':>10}{'miss':>7}"
for name in a.profiles.split(","):
r = {lab: s for n, lab, s in rows if n == name}
b, c = r["bytes only"], r["bytes + cycles"]
print(f"--- {name} (target {RC.PROFILES[name]['kbps']} KB/s) ---")
print(hdr)
for lab, s in (("bytes only", b), ("bytes + cycles", c)):
print(f" {lab:<20}{s['psnr']:>7.2f} {s['kbps']:>8.1f} "
f"{100*s['cyc_med']/BUDGET:>9.1f}%{100*s['cyc_max']/BUDGET:>9.1f}%"
f"{s['cpu_miss']:>6d}")
print(f" {'cost of fitting':<20}{c['psnr']-b['psnr']:>+7.2f} dB, "
f"{c['kbps']-b['kbps']:+.1f} KB/s, "
f"{b['cpu_miss']-c['cpu_miss']} fewer misses, "
f"{c['late']} frames unfixable at mu={RC.MU_CLIFF:g}")
print(f" {'modes % (b/c)':<20}SKIP {b['skip']:.1f}/{c['skip']:.1f} "
f"V1 {b['v1']:.1f}/{c['v1']:.1f} V4 {b['v4']:.1f}/{c['v4']:.1f} "
f"RAW {b['raw']:.1f}/{c['raw']:.1f}")
print()
print("FINDINGS 28.7: re-coding every non-SKIP block as V1 is the floor the "
"CURRENT mode set\nallows, and it still missed 11 frames at the retired "
"110 KB/s profile / 12 at scsi.\nMisses above that floor are spans, not "
"the mode decision -- and 31.3 showed the\nfloor itself was too "
"pessimistic, because the real decision can move a block to SKIP.")
tools/analysis/14_dmac_chain.py
+233
View File
@@ -0,0 +1,233 @@
#!/usr/bin/env python3
"""Would letting the HD63450 paint the spans beat letting the 68000 do it?
python3 tools/analysis/14_dmac_chain.py [container.dlx] --bus <KB/s>
[--dma-px-bus 2] [--disk-bus-byte 1]
FINDINGS 29.6 called this the one lever that could move the CPU budget without
spending a byte, and left it uncosted. FINDINGS 30 measured the alternative --
the 68000 painting spans itself, 43.7 cycles per span + 9.152 per pixel. This
prices the two against each other, and the answer turns on a resource neither
section costed: the 68000's own LOCAL BUS.
FINDINGS 29's "the bus has 4x the headroom the CPU has" is about the SCSI pipe,
110 KB/s of the delivery pipe. That is a different bus. The 68000's memory bus runs one 4-clock
cycle at a time and carries instruction prefetch as well as data, and
tools/analysis/15_bus_occupancy.py measures the decoder using 86.7% of it.
THE TWO DESIGNS ARE THE SAME CONTAINER. v6's record is {u32 absolute GVRAM
address, u16 jump displacement} = 6 bytes; an MC68450/HD63450 array-chaining
entry is {u32 memory address, u16 transfer count} = 6 bytes. Set the channel to
dual-address, direction device->memory, Sequence Control counting both addresses
up: MAR reloads per entry (the GVRAM destination), DAR walks the stream buffer,
MTC is the span's word count. The chain array IS the span table.
THE DMAC CONSTANTS ARE NOW SOURCED, and they killed the first answer. From the
MC68450 manual (Motorola, Jul 1989, bitsavers), Fig 4-25 sheet 4: a dual-address
WORD operand between two 16-bit ports is **9 clocks**, because note 2 gives the
DMAC 4-clock reads and **5-clock writes**. The 68000 writes in 4. So:
DMAC 9.000 clocks/pixel (datasheet)
v6 9.152 clocks/pixel (measured, FINDINGS 30)
A 1.7% difference. Session 10's first pass guessed 2 bus cycles = 8 clocks from
bus arithmetic and was 12% optimistic; the extra clock on every DMAC write is
the whole story. Per span, sequential array chaining costs 36 clocks (Fig 4-25
sheet 1) against v6's measured 43.7 -- the DMAC's one real edge, and it is small.
AND DMA DOES NOT OVERLAP. The 68000 has no cache and a two-word prefetch queue,
so it stalls as soon as another master takes the bus. Frame time is therefore
CPU + DMA, additive. Session 10's first pass used max(CPU, bus) and got 53/120
where the additive model gives 84/120; FINDINGS 35's flat debit was right.
So the only material difference left is v6's 24-pixel padding quantum -- and
that is a property of v6's unrolled chain, not of the CPU. The `v7 fine tail`
column prices fixing it in software instead, and as of session 11 that column
is MEASURED on the 68000 (blit.s v7, tools/bench/span.sh, FINDINGS 40) rather
than derived: 66.0 clocks per span + 9.143 per coarse pixel + 9.978 per fine
pixel, with a 2-pixel quantum that a run of 4x4 blocks pads to exactly.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/analysis")
import numpy as np
from dlx import DLX
import buscost as B
FRAME_CYC = 833333.0
AUDIO_KBPS = 7.8
import vq_hybrid as _H
C_V1, C_V4, C_RAW = _H.C_V1, _H.C_V4, _H.C_RAW # FINDINGS 28.2 (MEASURED)
# 45.0 until session 12 measured it at 55.0 (FINDINGS 41.5) -- imported now, so
# the correction cannot be undone by a stale copy.
C_SKIP_CLUSTERED, C_SKIP_MIXED = _H.C_SKIP_CLUSTERED, _H.C_SKIP_MIXED
SPAN_BYTES_PX, SPAN_HDR = 2, 6
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/rc_fr_singe_scsi_cpufit.dlx")
ap.add_argument("--bus", type=float, required=True,
help="REQUIRED. There is no default: the delivery rate is a property of the medium and this project has never measured it. FINDINGS 42.1 -- the figure this tool used to default to was a user-supplied '4 Mbps' with no provenance, was a tenth of SCSI-1's asynchronous rating, and was never a bus measurement at all. A default let every table in FINDINGS 30-49 be scored against it without anyone restating it. Pass one explicitly.")
ap.add_argument("--fps", type=float, default=12.0)
ap.add_argument("--dma-px-clk", type=float, default=B.DMA_PX_CLK,
help="clocks the DMAC spends per pixel, dual-address word "
"between two 16-bit ports. 9 is the DATASHEET figure "
"(MC68450 Fig 4-25 sheet 4).")
ap.add_argument("--disk-clk-byte", type=float, default=5.0,
help="clocks the SCSI DMA steals per BYTE delivered. The SPC is "
"an 8-bit port, so the DMAC pays per byte, not per word "
"(FINDINGS 43). 5, the default, is the OPTIMISTIC end and "
"what ratectl encodes against: single-address, bus held, no "
"drive wait (Fig 4-25 sheet 2). 9 is dual-address, which is "
"what MAME models and what applies if the board does not "
"drive DACK. Score both.")
ap.add_argument("--disk-clk-word", type=float, default=None,
help="DEPRECATED denominator of FINDINGS 39.7/42, kept so the "
"old tables reproduce: sets --disk-clk-byte to half this")
a = ap.parse_args()
if a.disk_clk_word is not None:
a.disk_clk_byte = a.disk_clk_word / 2.0
if not os.path.exists(a.container):
sys.exit(f"missing {a.container}")
BYTE_BUD = (a.bus - AUDIO_KBPS) * 1024 / a.fps
BUS_SLOTS = FRAME_CYC / B.BUS_CLK
d = DLX(a.container)
BLK_C = {1: C_V1, 2: C_V4, 3: C_RAW}
BLK_B = {1: 1, 2: 4, 3: 16}
def runs(m, by):
dirty = m[by] != 0
i = 0
while i < d.nbx:
if not dirty[i]:
i += 1; continue
j = i
while j < d.nbx and dirty[j]:
j += 1
yield i, j
i = j
def span_cost(design, L):
"""(pixels carried, clocks charged to the frame) for a run of L blocks,
as 4 rows of 4L pixels. Every design is charged additively: the 68000
cannot execute while the DMAC owns the bus."""
if design == "v6":
px = B.pad24(4 * L)
return 4 * px, 4 * (B.V6_SPAN_CYC + px * B.V6_PX_CYC)
if design == "v7":
px, c = B.v7_span(4 * L)
return 4 * px, 4 * c
px = 4 * L
return 4 * px, 4 * (B.DMA_CHAIN_CLK + px * a.dma_px_clk)
def score(design):
"""Greedy, as 12_span_tradeoff.py: buy the best clocks-saved per byte spent
until the frame's byte budget is gone. Unlike 12, a spanned block still pays
its mode-map dispatch, which FINDINGS 30.7 flagged as uncounted."""
out = []
for f in range(d.nframes):
m = d.modes(f).reshape(d.nby, d.nbx)
byt = d.mode_bytes + sum(BLK_B.get(int(x), 0) for x in m.ravel())
spanned = np.zeros_like(m, bool)
span_clk = 0.0
cand = []
if design != "none":
for by in range(d.nby):
for i, j in runs(m, by):
L = j - i
cur_c = sum(BLK_C[int(b)] for b in m[by][i:j])
cur_b = sum(BLK_B[int(b)] for b in m[by][i:j])
px, sc = span_cost(design, L)
sc += L * C_SKIP_MIXED # the dispatch still happens
# v7 carries a second u16 (the fine displacement) per span.
hdr = B.V7_SPAN_HDR if design == "v7" else SPAN_HDR
span_b = 4 * hdr + px * SPAN_BYTES_PX
if sc < cur_c:
cand.append((cur_c - sc, span_b - cur_b, by, i, j, sc, L))
cand.sort(key=lambda s: -(s[0] / max(s[1], 1)))
for dc, db, by, i, j, sc, L in cand:
if byt + db <= BYTE_BUD:
byt += db
spanned[by][i:j] = True
span_clk += sc - L * C_SKIP_MIXED
g = m.copy()
g[spanned] = 0
gg = g.reshape(-1, 4)
allskip = (gg == 0).all(1)
cpu = allskip.sum() * 4 * C_SKIP_CLUSTERED
mm = gg[~allskip]
cpu += (mm == 0).sum() * C_SKIP_MIXED
for k, c in BLK_C.items():
cpu += (mm == k).sum() * c
pref, data = B.block_bus(m, spanned)
disk = byt * a.disk_clk_byte
# additive: CPU work, then span painting, then the disk stealing the bus
out.append((cpu + span_clk + disk, (pref + data) * B.BUS_CLK, byt,
spanned.sum()))
return np.array(out).T
DESIGNS = [("today", "none"), ("v6 span", "v6"),
("v7 fine tail", "v7"), ("DMAC chain", "dmac")]
res = {n: score(k) for n, k in DESIGNS}
print(f"{a.container}: {d.nframes} frames, {d.nb} blocks, {a.fps:g} fps")
print(f"SCSI pipe {a.bus:.0f} KB/s -> {BYTE_BUD:,.0f} B/frame; "
f"68000 bus {BUS_SLOTS:,.0f} cycles/frame; CPU {FRAME_CYC:,.0f} clocks\n")
print("PER PIXEL AND PER SPAN -- datasheet against measurement")
print(f" DMAC dual-address word, two 16-bit ports {B.DMA_PX_CLK:.3f} clocks "
f"MC68450 Fig 4-25 sheet 4")
print(f" v6 movem chain {B.V6_PX_CYC:.3f} clocks "
f"MEASURED, FINDINGS 30")
print(f" -> the DMAC is {100*(B.V6_PX_CYC-B.DMA_PX_CLK)/B.V6_PX_CYC:+.1f}% per pixel. "
f"The 68000 writes in 4 clocks; the DMAC takes 5.")
print(f" per span: DMAC array chaining {B.DMA_CHAIN_CLK} clocks against v6's "
f"{B.V6_SPAN_CYC:.1f}\n")
w = 15
print(f"{'':<26}" + "".join(f"{n:>{w}}" for n, _ in DESIGNS))
def row(label, fmt, get):
print(f" {label:<24}" + "".join(f"{fmt(get(res[n])):>{w}}" for n, _ in DESIGNS))
row("bitrate KB/s", lambda v: f"{v:.1f}", lambda r: r[2].mean() * a.fps / 1024)
row("frame, median", lambda v: f"{v:.1f}%", lambda r: 100*np.median(r[0])/FRAME_CYC)
row("frame, worst", lambda v: f"{v:.1f}%", lambda r: 100*r[0].max()/FRAME_CYC)
row("frames missing", lambda v: f"{v}/{d.nframes}",
lambda r: int((r[0] > FRAME_CYC).sum()))
row("blocks spanned/frame", lambda v: f"{v:,.0f}", lambda r: r[3].mean())
print(f"\n ADDITIVE: frame = CPU + span painting + disk DMA. The 68000 has no"
f"\n cache and a two-word prefetch queue, so it stalls the moment another"
f"\n master takes the bus. Disk debited at {a.disk_clk_byte:g} clocks/byte.")
# What is left of the case, isolated.
v6m = int((res["v6 span"][0] > FRAME_CYC).sum())
finem = int((res["v7 fine tail"][0] > FRAME_CYC).sum())
dmam = int((res["DMAC chain"][0] > FRAME_CYC).sum())
print(f"\nWHAT THE DMAC ACTUALLY BUYS, decomposed")
print(f" v6 as built {v6m}/{d.nframes} frames over")
print(f" v7, a finer chain tail (MEASURED) {finem}/{d.nframes}")
print(f" DMAC chain {dmam}/{d.nframes}")
print(f" -> of the gap between v6 and the DMAC, "
f"{100*(v6m-finem)/max(v6m-dmam,1):.0f}% is the 24-pixel padding")
print(f" quantum, which is a property of v6's unrolled chain and fixable")
print(f" in software. The rest is 1.7% a pixel and 7.7 clocks a span.")
# The additive model here IS FINDINGS 35's flat debit, and reproduces its
# 84/120 exactly in the "today" column. Session 10's first pass replaced it with
# max(CPU, bus) and got 53/120; that was wrong, because a 68000 cannot execute
# while the DMAC holds the bus.
print(f"\nbreak-even against all-V1 ({C_V1:.1f} cycles/block), clocks per block")
print(f" {'L':<16}" + "".join(f"{L:>8}" for L in (1, 2, 3, 4, 8, 16, 64)))
for nm, dz in (("v6 as built", "v6"), ("v7 fine tail", "v7"), ("DMAC chain", "dmac")):
print(f" {nm:<16}" + "".join(f"{span_cost(dz, L)[1]/L:>8.0f}"
for L in (1, 2, 3, 4, 8, 16, 64)))
for nm, dz in (("v6 as built", "v6"), ("v7 fine tail", "v7"), ("DMAC chain", "dmac")):
brk = next((L for L in range(1, 65) if span_cost(dz, L)[1] < L * C_V1), None)
print(f" {nm:<16} beats all-V1 from L={brk} blocks up")
tools/analysis/15_bus_occupancy.py
+204
View File
@@ -0,0 +1,204 @@
#!/usr/bin/env python3
"""How much of the bus does the 68000 decoder actually leave for a DMAC?
python3 tools/analysis/15_bus_occupancy.py [container.dlx] [--nframes N]
FINDINGS 29.6's DMAC idea only pays if the DMAC can find bus slots the CPU is
not using. That is not a cycle count, it is a BUS count, and nothing in the tree
had one.
Two sources, and the point is that they check each other:
DATA accesses MEASURED by tools/bench/c68k/c68k_bench, which counts every
Read/Write callback the C68K core makes. Exact.
INSTRUCTION DERIVED here by walking src/player/decode.s's straight-line
prefetch paths in tools/bench/decode.lst and multiplying by the mode
histogram. Not measurable from either emulator: MAME's core
does not expose a fetch count and C68K reads opcodes straight
through a host pointer with no callback.
If the derived DATA figure matches the measured one, the derived PREFETCH figure
from the same walk is trustworthy too. That check is the first thing printed,
and this script exits non-zero if it fails.
A 68000 bus cycle is 4 clocks, so a frame of C clocks holds C/4 bus slots.
"""
import sys, os, argparse, csv
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
from dlx import DLX
import buscost as B
from buscost import V7_FRAME_PREF, V7_FRAME_DATA
BUS_CLK = 4
# --- straight-line path costs, read off tools/bench/decode.lst -------------
# (instruction words, data bus cycles). A long access is two bus cycles on the
# 68000's 16-bit bus; movem.l of N registers is 2N.
#
# dispatch move.b (a1),d0 / lsr.b / and.w #3 / beq .sk 6w, 1 read
# + subq / beq .v1 -> 8w
# + subq / bne .rw -> 10w
# V4 body $10090..$100E2 = 82 B = 41w; 4 x (1 byte read
# + movem.l 2 regs = 4 reads + 2 move.l = 4 writes) = 36
# V1 body $100E2..$10106 = 36 B = 18w; 1 byte read
# + movem.l 8 regs = 16 reads + 4 x movem.l 2 = 16 w = 33
# RAW body $10106..$10164 = 94 B = 47w; 8 x (2 byte reads
# + 1 move.l = 2 writes) = 32
# .sk tail addq.l #8,a4 1w
# BLOCK 0 has no lsr.b, so one of the four dispatches in a group is 1w cheaper.
DISPATCH_SK, DISPATCH_V1, DISPATCH_V4 = 6, 8, 10
BODY = {0: (0, 0), 1: (18, 33), 2: (41, 36), 3: (47, 32)}
DISPATCH = {0: DISPATCH_SK, 1: DISPATCH_V1, 2: DISPATCH_V4, 3: DISPATCH_V4}
SK_TAIL = 1
GROUP_HEAD = 3 # tst.b (a1) 1w + beq allskip 2w
GROUP_TAIL = 4 # addq.l #1,a1 / cmpa.l a5,a4 / bne byteloop
ALLSKIP = 9 # the whole four-block fast path, tst.b included
ROW_HEAD, ROW_TAIL = 3, 7
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/rc_fr_singe_scsi_cpufit.dlx")
ap.add_argument("--csv", default="tmp/c68k_frames.csv",
help="per-frame output of tools/bench/c68k/run.sh")
ap.add_argument("--nframes", type=int, default=None)
a = ap.parse_args()
if not os.path.exists(a.container):
sys.exit(f"missing {a.container}")
d = DLX(a.container)
meas = {}
if os.path.exists(a.csv):
for r in csv.DictReader(open(a.csv)):
meas[int(r["frame"])] = (int(r["cycles"]),
int(r["bus_reads"]) + int(r["bus_writes"]))
NF = a.nframes or (max(meas) + 1 if meas else d.nframes)
pref_t, data_t, cyc_t = [], [], []
for f in range(NF):
m = d.modes(f).reshape(d.nby, d.nbx)
pref = d.nby * (ROW_HEAD + ROW_TAIL)
data = 0
for by in range(d.nby):
row = m[by]
for gi in range(0, d.nbx, 4):
g = row[gi:gi+4]
if (g == 0).all():
pref += ALLSKIP; data += 1
continue
pref += GROUP_HEAD + GROUP_TAIL - 1 # BLOCK 0 has no lsr.b
data += 1
for b in g:
b = int(b)
pw, pd = BODY[b]
pref += DISPATCH[b] + pw + SK_TAIL
data += 1 + pd
# The span section is bus traffic too, and it is most of the frame's data
# accesses in a span-heavy container: 48 per 24-pixel chain unit. Leaving it
# out would not merely understate the total -- it would break the CHECK
# below, which is the whole licence for the prefetch figure.
sp, _ = d.spans(f)
if sp:
pref += V7_FRAME_PREF; data += V7_FRAME_DATA
for _, _, px in sp:
sp_p, sp_d = B.v7_span_split(len(px))
pref += sp_p; data += sp_d
pref_t.append(pref); data_t.append(data)
cyc_t.append(meas.get(f, (0, 0))[0])
pref_t, data_t, cyc_t = map(np.array, (pref_t, data_t, cyc_t))
print(f"{a.container}: {NF} frames, {d.nb} blocks/frame\n")
if meas:
md = np.array([meas[f][1] for f in range(NF)])
err = 100 * (data_t - md) / md
print("CHECK -- derived DATA bus cycles against the C68K harness's measurement")
print(f" measured mean {md.mean():>10,.0f} /frame")
print(f" derived mean {data_t.mean():>10,.0f} /frame "
f"error {err.mean():+.2f}% mean, {np.abs(err).max():.2f}% worst")
if np.abs(err).max() > 2.0:
sys.exit("\nFAIL: the path walk does not reproduce the measured data "
"accesses, so its prefetch figure cannot be trusted either.")
print(" the walk reproduces the measurement, so its prefetch count stands\n")
slots = cyc_t / BUS_CLK
tot = pref_t + data_t
print(f"{'':<22}{'mean':>12}{'median':>12}{'worst frame':>14}")
for label, v in (("bus slots in a frame", slots),
(" data accesses", data_t),
(" instruction prefetch", pref_t),
(" total bus cycles", tot)):
print(f"{label:<22}{v.mean():>12,.0f}{np.median(v):>12,.0f}{v.max():>14,.0f}")
occ = 100 * tot / slots
print(f"{'bus OCCUPANCY':<22}{occ.mean():>11.1f}%{np.median(occ):>11.1f}%"
f"{occ.max():>13.1f}%")
free = slots - tot
print(f"{'slots left for a DMAC':<22}{free.mean():>12,.0f}{np.median(free):>12,.0f}"
f"{free.min():>14,.0f} (worst = fewest)")
print(f"\nprefetch is {100*pref_t.sum()/tot.sum():.0f}% of the decoder's bus traffic: "
f"the data-only\nfigure the harness prints understates occupancy by about 2x.")
print(f"A DMAC painting spans at 8 clocks (2 bus cycles) per pixel could use at\n"
f"most {free.mean()/2:,.0f} pixels' worth of the mean frame's spare slots "
f"-- against {d.nb*16:,} pixels\nin a whole screen.")
# ---------------------------------------------------------------------------
# THE OTHER TWO MASTERS. Everything above is the 68000's own traffic, and it
# was the whole of this tool until session 20. The frame also has to carry the
# bitstream in off the disk and a byte of ADPCM out to $E92003 every 128 us,
# and neither has ever appeared in a bus figure -- FINDINGS 35's lesson, which
# was about the CLOCK budget, had never been applied to the BUS one.
#
# The DMAC does not overlap with the CPU (buscost.DMA_OVERLAPS = False): the
# 68000 has no cache and a two-word prefetch queue that empties at once, so a
# stolen bus cycle is a stopped CPU. The three demands therefore ADD.
#
# Audio's per-byte figure is SETTLED, not bracketed by taste:
# tools/analysis/21_iplrom_dmac.py reads the IPL ROM's own HD63450 setup and
# finds channel 3 dual-address, 8-bit port, cycle steal without hold, external
# request -- one arbitration per byte, no burst. Video's is NOT settled: it is
# ROADMAP B3 / FINDINGS 42.4-42.6's W, so it is swept rather than picked.
print("\n" + "=" * 72)
print("THE OTHER TWO MASTERS -- what the DMAC takes out of the same frame\n")
FPS = d.fps
CPUHZ = 10e6 # stock X68000, MAME 0.277 x68k.cpp:1133
FRAME_CLK = CPUHZ / FPS
vid_bpf = sum(n + 4 for (_, n) in d.frames[:NF]) / NF # DLX2 record padding
aud_bpf = B.ADPCM_BYTES_PER_S / FPS
a_lo = aud_bpf * B.ADPCM_CLK_BYTE_BEST
a_hi = aud_bpf * B.ADPCM_CLK_BYTE_WORST
cpu_clk = cyc_t.mean() if cyc_t.any() else float("nan")
print(f"frame period at {FPS:g} fps on a 10 MHz 68000: {FRAME_CLK:,.0f} clocks")
if cyc_t.any():
print(f" decoder, MEASURED (C68K) {cpu_clk:>10,.0f} clk "
f"{100*cpu_clk/FRAME_CLK:5.1f}% worst frame "
f"{100*cyc_t.max()/FRAME_CLK:.1f}%")
print(f" audio DMA, {aud_bpf:,.1f} B/frame {a_lo:>10,.0f} clk "
f"{100*a_lo/FRAME_CLK:5.2f}% .. {a_hi:,.0f} clk "
f"({100*a_hi/FRAME_CLK:.2f}%)")
print(f" {B.ADPCM_CLK_BYTE_BEST}..{B.ADPCM_CLK_BYTE_WORST} clk/byte, "
f"from the ROM's own DCR/OCR (21_iplrom_dmac.py). NOT a guess, and\n"
f" not the disk's rate: audio arbitrates for the bus once per byte "
f"and cannot burst.")
print(f"\n video DMA, {vid_bpf:,.0f} B/frame, swept over W -- ROADMAP B3 is "
f"still open:")
print(f" {'W (clk/byte)':<16}{'clk/frame':>12}{'% of frame':>12} "
f"{'CPU+audio+video':>18}")
for W, note in ((5.0, "single address, bus held (11_cpu_budget.py default)"),
(8.0, "FINDINGS 5's long-standing per-word ESTIMATE"),
(12.0, "single address, arbitrated per byte"),
(16.0, "what the ROM programs for SASI (best case)"),
(19.0, "what the ROM programs for SASI (worst case)")):
v = vid_bpf * W
tot_clk = (cpu_clk if cyc_t.any() else 0) + a_lo + v
print(f" {W:<16.0f}{v:>12,.0f}{100*v/FRAME_CLK:>11.1f}% "
f"{100*tot_clk/FRAME_CLK:>17.1f}% {note}")
print(f"\n (the last column adds the MEASURED mean decode and the BEST-CASE "
f"audio, so it is\n the optimistic end of every row. 100% is the frame "
f"deadline at {FPS:g} fps.)")
print(f"""
Audio is {100*a_lo/FRAME_CLK:.2f}%..{100*a_hi/FRAME_CLK:.2f}% of the frame and video is {vid_bpf*5/FRAME_CLK*100:.0f}%..{vid_bpf*19/FRAME_CLK*100:.0f}%. The unpriced audio
stream was never the risk P6 called it -- ON THE BUS. What the same reading of
the ROM found is that the DISK's per-byte cost has a worked example on this
machine, it is 16..19 clocks, and at that price this design does not fit at any
container size. W is the number to attack, and it is a PLAYER decision.""")
tools/analysis/16_span_roundtrip.py
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env python3
"""GATE for the DLX3 span container: does the reference decoder reproduce the
encoder's own reconstruction, from the emitted bytes?
python3 tools/analysis/16_span_roundtrip.py [frames_dir] --kbps <KB/s>
Exits non-zero if any frame differs by a single pixel.
WHY THIS EXISTS SEPARATELY FROM 09. `09_ratectl_drift.py` replays SKIP
semantics in Python against the mode maps the encoder returned; it never reads
a container. A span breaks exactly that shortcut: a spanned block reads SKIP
in the mode header and is painted by the span section instead, so a replay that
knows only about mode maps reports drift where there is none, and -- far worse
-- a container whose span section is malformed would still pass, because 09
never parses one. This gate closes that: encode, WRITE THE CONTAINER, read it
back with tools/encoder/dlx.py (the byte-for-byte reference decoder the 68000
is checked against), and compare to what ratectl recorded.
It also has to prove it tested something. A round-trip over a container with
no spans in it is green by vacuity, which is the failure mode FINDINGS 40.6
named for the snapshot count: a gate must take its expected work from the
generated artefact, not from an assumption. So the thresholds below are
asserted, not printed.
The `--kbps` default is the BUS rate, not the `scsi` profile's 280: spans are
bought with bytes, and 14_dmac_chain.py scores them against the delivery pipe.
At the profile rate the lam search has already spent the allowance and there is
nothing left to buy a span with -- which is a real finding about the encoder
(FINDINGS 41.2), not a reason for the gate to test nothing.
"""
import argparse, os, pickle, sys, time
sys.path.insert(0, "tools/encoder")
import numpy as np
import vq_hybrid as H, ratectl as RC, encode as E
from dlx import DLX
ap = argparse.ArgumentParser()
ap.add_argument("frames_dir", nargs="?", default="tmp/fr_singe")
ap.add_argument("--kbps", type=float, required=True,
help="REQUIRED. There is no default: the delivery rate is a property of the medium and this project has never measured it. FINDINGS 42.1 -- the figure this tool used to default to was a user-supplied '4 Mbps' with no provenance, was a tenth of SCSI-1's asynchronous rating, and was never a bus measurement at all. A default let every table in FINDINGS 30-49 be scored against it without anyone restating it. Pass one explicitly.")
ap.add_argument("--out", default="tmp/s12_roundtrip")
ap.add_argument("--cache", default=None)
a = ap.parse_args()
cache = a.cache or f"tmp/model_{os.path.basename(a.frames_dir.rstrip('/'))}.pkl"
if os.path.exists(cache):
m = pickle.load(open(cache, "rb"))
print(f"model from {cache}")
else:
t = time.time()
m = H.build(a.frames_dir, k1=256, k4=256, iters=16)
pickle.dump(m, open(cache, "wb"))
print(f"built model in {time.time()-t:.0f} s -> {cache}")
bad = 0
for span_mode in ("need", "all"):
print(f"\n=== spans={span_mode}, {a.kbps:g} KB/s ===")
m.pop("_sym", None)
enc = RC.encode_rate_controlled(m, target_kbps=a.kbps, lam_lo=1.0,
cycle_budget=RC.FRAME_CYCLES,
span_mode=span_mode)
recs = E.build_records(m, enc, span_mode)
path = f"{a.out}_{span_mode}.dlx"
total, vid, _ = E.write_container(path, m, recs, 12, m["k1"], m["k4"],
span_mode)
nsp = sum(len(x) for x in enc["spans"])
nfr = sum(1 for x in enc["spans"] if x)
px = sum(len(p) for x in enc["spans"] for _, _, p in x)
print(f"{path}: {total:,} B, {len(recs)} frames, "
f"{nsp:,} spans on {nfr} frames, {px:,} pixels painted by one "
f"({100*px/(len(recs)*m['H']*m['W']):.1f}% of all pixels)")
d = DLX(path)
if d.version != 3:
print(f"FAIL: container is DLX{d.version}, not DLX3"); bad += 1; continue
# The decoder's own walk of the span section must land exactly where the
# block payload starts, and blocks() already raises if the payload does not
# consume the record -- so this reads the spans back through the same code
# path the 68000 is modelled on rather than trusting the writer.
got = d.decode_all()
diff = np.array([(g != r).sum() for g, r in zip(got, enc["recon"])])
print(f"pixels differing from the encoder's reconstruction: "
f"{diff.sum()} total, worst frame {diff.max()}, "
f"frames with any: {int((diff>0).sum())}/{len(diff)}")
if diff.sum():
f = int(np.argmax(diff))
ys, xs = np.where(got[f] != enc["recon"][f])
print(f"FAIL: frame {f} differs at {diff[f]} px, first (x={xs[0]}, "
f"y={ys[0]}), block (bx={xs[0]//4}, by={ys[0]//4}), "
f"mode there = {d.modes(f)[(ys[0]//4)*d.nbx + xs[0]//4]}")
bad += 1
# A green round-trip over a container with no spans in it proves nothing.
if span_mode == "all":
if nsp < 1000:
print(f"FAIL: only {nsp} spans emitted -- this gate did not "
f"exercise the span path"); bad += 1
if not (px and max(len(x) for x in enc["spans"]) > 50):
print(f"FAIL: no frame carries a substantial span table"); bad += 1
print()
if bad:
print(f"FAILED: {bad} check(s)")
sys.exit(1)
print("OK the DLX3 span container round-trips: the reference decoder rebuilds "
"the\n encoder's reconstruction exactly, from the emitted bytes.")
tools/analysis/17_span_delivered.py
+112
View File
@@ -0,0 +1,112 @@
#!/usr/bin/env python3
"""What do the spans the ENCODER actually emitted cost, and what do they buy?
python3 tools/analysis/17_span_delivered.py a.dlx [b.dlx ...] --bus <KB/s>
Every span figure before this one -- FINDINGS 29 through 40, and
tools/analysis/12 and 14 -- was scored by SIMULATING span selection over mode
maps that were chosen without spans available. FINDINGS 39.3 flagged that as a
lower bound on what a span-aware encoder would find, and docs/STATUS.md's item 2
asks for the figures to be re-run "against a container the encoder actually
emits with spans in it". This is that script: it reads the span section out of
a DLX3 container and prices exactly those spans, with no selection model at all.
THE MODEL IS 14_dmac_chain.py's, deliberately unchanged, so the columns are
comparable:
frame clocks = block decode + span painting + disk DMA
additive, because a 68000 has no cache and a two-word prefetch queue and stalls
the moment another master takes the bus (FINDINGS 38.3). Block cost is
vq_hybrid.cycles(), which reads a spanned block as SKIP -- correct, because the
span section is what paints it, and its cost is the second term.
The span term is the MEASURED v7 fit (FINDINGS 40), and as of session 12 that
fit is confirmed inside src/player/decode.s itself rather than only in
tools/bench/blit.s: the synthetic all-SPAN anchors of tools/bench/prep_dlx.py
reproduce it to 0.23% on both emulators (FINDINGS 41.3).
"""
import argparse, os, sys
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/analysis")
import numpy as np
import vq_hybrid as H
import spans as SP
import buscost as B
from dlx import DLX
FRAME_CYC = 833333.0
AUDIO_KBPS = 7.8
ap = argparse.ArgumentParser()
ap.add_argument("containers", nargs="+")
ap.add_argument("--bus", type=float, required=True,
help="REQUIRED. There is no default: the delivery rate is a property of the medium and this project has never measured it. FINDINGS 42.1 -- the figure this tool used to default to was a user-supplied '4 Mbps' with no provenance, was a tenth of SCSI-1's asynchronous rating, and was never a bus measurement at all. A default let every table in FINDINGS 30-49 be scored against it without anyone restating it. Pass one explicitly.")
ap.add_argument("--fps", type=float, default=12.0)
ap.add_argument("--disk-clk-byte", type=float, default=5.0,
help="clocks the SCSI DMA steals per BYTE delivered. The SPC is "
"an 8-bit port, so the DMAC pays per byte, not per word "
"(FINDINGS 43). 5, the default, is the OPTIMISTIC end and "
"what ratectl encodes against: single-address, bus held, no "
"drive wait (Fig 4-25 sheet 2). 9 is dual-address, which is "
"what MAME models and what applies if the board does not "
"drive DACK. Score both.")
ap.add_argument("--disk-clk-word", type=float, default=None,
help="DEPRECATED denominator of FINDINGS 39.7/42, kept so the "
"old tables reproduce: sets --disk-clk-byte to half this")
a = ap.parse_args()
if a.disk_clk_word is not None:
a.disk_clk_byte = a.disk_clk_word / 2.0
def score(path):
d = DLX(path)
rows = []
for f in range(d.nframes):
mode = d.modes(f)
sp, _ = d.spans(f)
_, n = d.frames[f]
blk = H.cycles(mode)
spc = sum(SP.clocks(len(p)) for _, _, p in sp)
disk = n * a.disk_clk_byte
rows.append((blk, spc, disk, n, len(sp),
sum(len(p) for _, _, p in sp)))
return d, np.array(rows).T
print(f"{'container':<34}{'KB/s':>8}{'spans':>9}{'span px':>9}"
f"{'median':>9}{'worst':>9}{'over':>9}")
print(f"{'':<34}{'':>8}{'/frame':>9}{'%':>9}"
f"{'% frame':>9}{'% frame':>9}{'budget':>9}")
for path in a.containers:
if not os.path.exists(path):
print(f"{path:<34} missing"); continue
d, r = score(path)
blk, spc, disk, byt, nsp, spx = r
tot = blk + spc + disk
kbps = byt.mean() * a.fps / 1024 + AUDIO_KBPS
print(f"{os.path.basename(path):<34}{kbps:>8.1f}{nsp.mean():>9.0f}"
f"{100*spx.mean()/(d.W*d.H):>9.1f}"
f"{100*np.median(tot)/FRAME_CYC:>9.1f}"
f"{100*tot.max()/FRAME_CYC:>9.1f}"
f"{int((tot > FRAME_CYC).sum()):>6}/{d.nframes:<3}")
print(f"\n ADDITIVE: frame = block decode + span painting + disk DMA, the model"
f"\n of 14_dmac_chain.py. Disk debited at {a.disk_clk_byte:g} clocks/byte "
f"over the\n container's own byte count; CPU budget {FRAME_CYC:,.0f} "
f"clocks at {a.fps:g} fps.")
# The decomposition is the point: a span moves work out of the block loop and
# into the span section, and it pays for it in bytes -- which the disk term
# then charges back. A design that only counted the CPU would show a win that
# the I/O it created takes away again (docs/FINDINGS.md 33).
print(f"\nWHERE EACH FRAME'S CLOCKS GO, mean over the container")
print(f" {'container':<34}{'blocks':>12}{'spans':>12}{'disk':>12}{'total':>12}")
for path in a.containers:
if not os.path.exists(path):
continue
d, r = score(path)
blk, spc, disk = r[0], r[1], r[2]
print(f" {os.path.basename(path):<34}{blk.mean():>12,.0f}{spc.mean():>12,.0f}"
f"{disk.mean():>12,.0f}{(blk+spc+disk).mean():>12,.0f}")
tools/analysis/18_text_plane_16col.py
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""What 256 -> 16 colours actually costs, on real frames.
python3 tools/analysis/18_text_plane_16col.py [frames_dir]
FINDINGS 46.3 opened a lead and could not price it: the X68000 text plane is
4bpp planar -- 0.5 bytes/pixel against the graphics planes' 2.0 -- so a LITERAL
uncompressed 16-colour frame is 288.0 KB/s against the shipping compressed
256-colour container's 496.7 KB/s. 42% cheaper on the wire, with no decoder.
The whole lead turns on one number nobody had computed: the quality cost of 16
colours. This computes it, and it is deliberately generous to the 16-colour
side on every axis where the hardware allows it:
* PER-FRAME palettes are legitimate here. The text palette is 16 entries and
reloading it is 16 words a frame -- nothing, against a 833,333-clock budget.
The 256-colour path cannot do this: its palette is shared scene-wide
(vq.scene_palette) because the codec's codebooks are indices INTO it.
* DITHERING is free here, and only here. The tree does not dither (vq.py:32,
"cel art is flat") because dither destroys the inter-frame coherence SKIP
blocks and v7 spans are built on. A literal frame has no codec to wreck, so
Floyd-Steinberg is available to this path at zero runtime cost.
Both are measured, so the comparison cannot be accused of hobbling the option it
is testing. Reported against the 256-colour scene-palette ceiling (the tree's
existing "palette ceiling" figure) and against the shipping container's PSNR.
"""
import sys, os
sys.path.insert(0, "tools/encoder")
import numpy as np
from PIL import Image
import vq as VQ
FRAMES = sys.argv[1] if len(sys.argv) > 1 else "tmp/fr_singe"
SHIPPED_PSNR = 29.19 # docs/STATUS.md, --spans all, c=5, 496.7 KB/s
rgb = VQ.load_frames(FRAMES)
H, W = rgb[0].shape[:2]
n = len(rgb)
print(f"{FRAMES}: {n} frames, {W}x{H}")
print()
def recon_scene(colors, dither):
"""One palette for the whole scene -- what the 256 path is forced to do."""
d = Image.FLOYDSTEINBERG if dither else Image.NONE
samp = np.concatenate([r.reshape(-1, 3) for r in rgb[::3]])
ref = Image.fromarray(samp.reshape(-1, 1, 3)).quantize(
colors=colors, method=Image.MEDIANCUT, dither=Image.NONE)
pal = np.array(ref.getpalette()[:colors * 3], np.uint8).reshape(-1, 3)
return [pal[np.asarray(Image.fromarray(r).quantize(palette=ref, dither=d),
np.uint8)] for r in rgb]
def recon_perframe(colors, dither):
"""A fresh palette every frame -- what the text plane can afford."""
d = Image.FLOYDSTEINBERG if dither else Image.NONE
out = []
for r in rgb:
q = Image.fromarray(r).quantize(colors=colors, method=Image.MEDIANCUT,
dither=d)
pal = np.array(q.getpalette()[:colors * 3], np.uint8).reshape(-1, 3)
out.append(pal[np.asarray(q, np.uint8)])
return out
def report(name, recon):
per = np.array([VQ.psnr(a, b) for a, b in zip(rgb, recon)])
print(f" {name:<42s} {per.mean():6.2f} dB "
f"(min {per.min():5.2f} max {per.max():5.2f})")
return per.mean()
print("PSNR vs the 24-bit source, mean over frames:")
c256 = report("256 colours, scene palette [the tree's]", recon_scene(256, False))
report("256 colours, per-frame palette", recon_perframe(256, False))
print()
s16 = report("16 colours, scene palette", recon_scene(16, False))
p16 = report("16 colours, per-frame palette", recon_perframe(16, False))
p16d = report("16 colours, per-frame + FS dither", recon_perframe(16, True))
print()
print(f" the 16-colour ceiling is the best of those: {max(s16, p16, p16d):.2f} dB")
print(f" cost of 256 -> 16, at each side's best: "
f"{c256 - max(s16, p16, p16d):.2f} dB")
print()
print(f" for scale, the shipping container delivers {SHIPPED_PSNR:.2f} dB "
f"at 496.7 KB/s")
print(f" a 16-colour literal would deliver "
f"{max(s16, p16, p16d):.2f} dB at 288.0 KB/s")
delta = max(s16, p16, p16d) - SHIPPED_PSNR
print(f" so the text-plane path is {abs(delta):.2f} dB "
f"{'BETTER' if delta > 0 else 'WORSE'} at 58% of the bitrate")
tools/analysis/19_ring_stream.py
+275
View File
@@ -0,0 +1,275 @@
"""Ring-buffer streaming simulation, against the CONTIGUITY constraint (STATUS 3/4).
09_buffer_sim.py asked one question -- does cumulative supply ever fall behind
cumulative demand -- and answered it in BYTES. FINDINGS 21 got "zero required
prefill" out of it at 110 and 280 KB/s. That test is necessary and not
sufficient, and the missing half is the whole of STATUS item 3:
src/player/decode.s reads a frame record with a MONOTONICALLY INCREASING a0
and no bounds check anywhere. `move.l (a0)+,d0` for the length, `lea
MODEB(a0),a0` for the span section, eleven unrolled `movem.l (a0)+` chains,
`move.b (a0)+` per block index. Nothing in it can survive an address that
wraps mid-record. So the buffer does not merely need ENOUGH BYTES resident
by the deadline -- it needs the WHOLE NEXT RECORD resident and CONTIGUOUS.
Having enough bytes and having them contiguous are different conditions, and a
byte-counting simulation cannot tell them apart. This one models the ring's
addresses, not just its occupancy.
THREE WRAP POLICIES, and the point of the tool is that they are not equivalent:
split the writer wraps mid-record; the reader cannot. Requires a SHADOW of
the ring's first MAXREC bytes mirrored past its end, so any record
start can be read linearly for MAXREC bytes. Every byte landing in
that first MAXREC is written twice. Costs 68000 CLOCKS, forever, at a
rate set by MAXREC/ring -- and those clocks come out of the same
budget the decoder is already spending 77.0% of (FINDINGS 45).
aligned the writer refuses to start a record it cannot finish before the end
of the ring; it leaves a hole and restarts at 0. Costs RAM (the mean
hole) and nothing else -- no copy, no per-byte work. Needs a frame
INDEX so the fill side knows record boundaries, which a branching
laserdisc game needs anyway to seek to a branch point.
none the decoder handles the wrap itself. Priced here only to show what it
would cost: a bounds test in the block loop is inside the sequence
FINDINGS 30.4/40 fitted, so it does not cost a branch -- it costs
every span and per-block constant in the tree being re-measured.
Not simulated; see the note printed at the end.
DEADLINE MODEL, and it is the conservative one: record i must be wholly
resident when frame i's decode BEGINS. The decoder in fact reads a record
progressively over ~77% of a frame time, so a byte arriving mid-frame would in
practice be in time -- but that is a race between the DMAC's fill address and
a0, and this tool refuses to certify a design on a race it cannot see.
Fill is quantised to 512-byte SCSI blocks: a partial sector is not resident.
python3 tools/analysis/19_ring_stream.py [container ...] --kbps R [--ring KB]
`--kbps` is REQUIRED and has no default -- see the argument's help text.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
import ratectl as RC
SECTOR = 512
# 5 clocks/byte for a 68000 `move.l (a0)+,(a1)+` copy: 20 clocks moves 4 bytes
# on a 16-bit bus (2 read + 2 write bus cycles at 4 clocks, plus the fetch it
# shares with the loop). Deliberately the OPTIMISTIC figure -- a movem-shaped
# copy is what the shadow would really use, and it is the same 5.0.
COPY_CLK_PER_BYTE = 5.0
CPUHZ = 10_000_000
def records(path):
"""Padded record sizes, exactly as the 68000 walks them.
prep_dlx.py rounds each record START up to 4 (FINDINGS 28.3), so the bytes
the ring must hold per frame are the padded ones, not the payload.
"""
d = DLX(path)
rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
return d, rec
def simulate(rec, fill_per_frame, ring, policy, maxrec):
"""Address-level ring simulation. Returns a dict of results.
The ring is modelled as a write cursor and a read cursor over `ring` bytes.
Supply arrives at `fill_per_frame` bytes per frame time, sector-quantised.
Record i is due at the start of frame i.
"""
n = len(rec)
resident = 0.0 # bytes fully arrived and not yet consumed
carry = 0.0 # sub-sector remainder of the fill
wcur = 0 # write cursor within the ring
holes = [] # bytes wasted per wrap, `aligned` policy
shadow_bytes = 0 # bytes double-written, `split` policy
occ = []
prefill = 0.0
late = []
free = ring
# Required prefill is solved rather than searched: run once with an infinite
# head start to find the worst deficit, exactly as 09_buffer_sim does, then
# assert the ring can hold it.
deficit = np.maximum.accumulate(np.cumsum(rec - fill_per_frame))
prefill = float(max(0.0, deficit.max()))
for i, r in enumerate(rec):
# --- supply for this frame time, sector-quantised
avail = carry + fill_per_frame
sectors = int(avail // SECTOR)
got = sectors * SECTOR
carry = avail - got
# --- placement: does this frame's arriving data cross the ring end?
if policy == "aligned":
# The writer will not start a record it cannot finish. Charge the
# hole when the NEXT record would not fit in the tail.
if wcur + r > ring:
holes.append(ring - wcur)
wcur = 0
wcur += r
else: # split
end = wcur + r
if end > ring:
wcur = end - ring
# every byte that landed in the first MAXREC of the ring is
# mirrored into the shadow
shadow_bytes += min(wcur, maxrec)
else:
wcur = end
if wcur <= maxrec:
shadow_bytes += r
elif wcur - r < maxrec:
shadow_bytes += maxrec - (wcur - r)
resident += got
if resident + 1e-9 < r:
late.append((i, float(r - resident)))
resident -= r
occ.append(resident)
hole_mean = float(np.mean(holes)) if holes else 0.0
usable = ring - hole_mean if policy == "aligned" else ring
copy_clk = shadow_bytes * COPY_CLK_PER_BYTE / max(1, n)
return dict(prefill=prefill, late=late, occ=np.array(occ),
holes=holes, hole_mean=hole_mean, usable=usable,
shadow_bytes=shadow_bytes, copy_clk_per_frame=copy_clk,
wraps=len(holes) if policy == "aligned" else None)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("containers", nargs="*",
default=["tmp/s14_d5_all1500.dlx",
"tmp/rc_fr_singe_scsi_span.dlx"])
ap.add_argument("--kbps", type=float, required=True,
help="delivered pipe, KB/s. REQUIRED, and deliberately has "
"no default: the delivery rate is a property of the "
"medium and this project has never measured it. The "
"figure that used to sit here was a user-supplied "
"'4 Mbps' with no provenance and was never a bus "
"measurement (FINDINGS 42.1); leaving it as a default "
"let table after table be scored against it without "
"anyone restating what it was.")
ap.add_argument("--ring", type=float, default=256.0,
help="ring size in KB (default 256, FINDINGS 21's sizing)")
a = ap.parse_args()
FPS = 12
print(f"ring {a.ring:.0f} KB sector {SECTOR} B "
f"audio {RC.AUDIO_KBPS} KB/s debited from the pipe\n")
for path in a.containers:
if not os.path.exists(path):
print(f"{path}: MISSING -- skipped\n"); continue
d, rec = records(path)
maxrec = int(rec.max())
ring = int(a.ring * 1024)
wire = rec.mean() * FPS / 1024 + RC.AUDIO_KBPS
print(f"=== {path}")
print(f" {d.nframes} frames @ {d.fps}fps, record bytes "
f"min {rec.min():,} median {int(np.median(rec)):,} max {maxrec:,}")
print(f" wire demand {wire:.1f} KB/s "
f"(video {rec.mean()*FPS/1024:.1f} + audio {RC.AUDIO_KBPS}), "
f"including the u32 length and the 4-byte record pad")
# A required prefill is only a startup cost if the window's MEAN demand
# is under the pipe. If the mean is over, the deficit grows for as long
# as the scene runs and the prefill this window reports is just how far
# it got in 120 frames -- no ring size fixes that, and quoting a KB
# figure for it would be the most flattering possible way to state a
# sustained overrun. FINDINGS 21's "zero prefill" never had to make
# this distinction because it ran far under the pipe it assumed.
if wire > a.kbps:
over = wire - a.kbps
print(f" !! SUSTAINED OVERRUN at the {a.kbps:.0f} KB/s pipe: "
f"demand exceeds supply by {over:.1f} KB/s on the MEAN, not "
f"on a burst.")
print(f" The deficit grows {over*1024/FPS:,.0f} B per frame "
f"for as long as the scene runs -- {over*1024*120/FPS/1024:.0f} "
f"KB over this 120-frame window, {over*60:.0f} KB per minute "
f"of play. Prefill below is where it got in 120 frames, NOT a "
f"startup cost that fixes it.")
if maxrec > ring:
print(f" !! MAXREC {maxrec:,} > ring {ring:,}: no policy works. "
f"decode.s needs one whole record contiguous.\n")
continue
# --- the requirement on the medium, which is the useful output, and
# the reason this tool takes no default rate. There is no measured
# pipe figure to score against (42.1), and the intent is to measure
# a BlueSCSI directly -- so the tool reports the THRESHOLD to
# measure against. The sweep is anchored to the container's own
# wire demand rather than to a list of fixed rates, so it stays
# meaningful for any container and privileges no constant.
print(f" {'pipe KB/s':>10} {'vs wire':>8} {'prefill KB':>11} "
f"{'records':>8} {'seek slack':>11}")
for mult in (0.90, 0.95, 1.00, 1.02, 1.05, 1.10, 1.25, 1.50, 2.00):
kbps = wire * mult
fill = (kbps - RC.AUDIO_KBPS) * 1024 / FPS
r = simulate(rec, fill, ring, "aligned", maxrec)
pf = r["prefill"]
# Branch-point seek slack, STATICALLY: with the ring FULL, how many
# frame times can the fill be zero before the next record is not
# resident? It is an upper bound and it assumes the premise that
# FINDINGS 51.3 took apart -- the ring is NOT full at a branch
# point, it is empty, and refilling it takes seconds of play. For
# the measured figure use tools/analysis/20_seek_slack.py, or the
# rig itself (tools/bench/pace_run.sh). Kept here as the ceiling
# this container's record sizes allow, which is what the rest of
# this row is about.
slack = (r["usable"] - maxrec) / rec.mean()
flag = ""
if pf + maxrec > r["usable"]:
flag = " <- does not fit the ring"
print(f" {kbps:>10.1f} {mult:>7.2f}x {pf/1024:>11.1f} "
f"{pf/rec.mean():>8.2f} {slack:>8.1f} fr{flag}")
# smallest pipe needing zero prefill, to 0.1 KB/s
lo, hi = wire, wire + 400
for _ in range(40):
mid = (lo + hi) / 2
f = (mid - RC.AUDIO_KBPS) * 1024 / FPS
if simulate(rec, f, ring, "aligned", maxrec)["prefill"] > 0:
lo = mid
else:
hi = mid
print(f" ZERO-PREFILL PIPE: {hi:.1f} KB/s "
f"({hi - wire:+.1f} KB/s over the wire demand, "
f"{100*hi/wire - 100:+.1f}%)")
print(f" ^ this is the number to measure a medium against. It is a "
f"REQUIREMENT, not a verdict.")
# --- the policy trade, at the default pipe
fill = (a.kbps - RC.AUDIO_KBPS) * 1024 / FPS
print(f" wrap policy, at pipe {a.kbps:.0f} KB/s:")
for policy in ("aligned", "split"):
r = simulate(rec, fill, ring, policy, maxrec)
if policy == "aligned":
print(f" aligned wraps {r['wraps']:3} mean hole "
f"{r['hole_mean']/1024:6.1f} KB usable ring "
f"{r['usable']/1024:6.1f} KB "
f"({100*r['usable']/ring:.1f}%) CPU cost 0")
else:
pct = 100 * r["copy_clk_per_frame"] / (CPUHZ / FPS)
print(f" split shadow {r['shadow_bytes']/1024:8.1f} KB "
f"= {r['copy_clk_per_frame']:8.0f} clk/frame = "
f"{pct:.2f}% of the frame budget, forever RAM cost 0")
print()
print("The `none` policy -- decoder wraps its own reads -- is not simulated.")
print("It has no RAM or copy cost and it is still the expensive one: the")
print("bounds test lands inside the exact instruction sequences FINDINGS")
print("30.4 and 40 fitted, so it does not cost a branch, it costs every span")
print("and per-block constant in the tree being re-measured. FINDINGS 28.3.")
if __name__ == "__main__":
main()
tools/analysis/20_seek_slack.py
+146
View File
@@ -0,0 +1,146 @@
"""Seek slack: how long a branch point can stop delivery (STATUS 4, FINDINGS 51).
19_ring_stream.py asks whether a container ARRIVES in time, and prints one
"seek slack" column derived statically as (usable ring - maxrec)/mean record.
That is a capacity estimate and it quietly assumes the ring is full when the
seek happens. It is not, and the difference is the whole finding:
A ring's slack is ACCUMULATED, not owned. It is built out of the surplus
between the pipe and the wire demand, at (pipe - wire) bytes per second, and
a seek spends all of it. How long a branch point can stall is a property of
the ring; how soon the NEXT branch point can be afforded is a property of the
surplus, and a bigger ring makes that one WORSE.
This is the paced-rig model (tools/bench/stream.lua with DLX_PACE=1) written
independently, and it exists to be compared against it, not to replace it. The
rig drives a real 68000 through a real ring and is the measurement; this is the
cheap sweep that says where to point it. Where they disagree, the rig wins.
python3 tools/analysis/20_seek_slack.py [container ...] --kbps R [R ...]
[--ring KB [KB ...]]
`--kbps` is REQUIRED and takes no default, for the reason FINDINGS 50 gives.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
import ratectl as RC
SECTOR = 512
def records(path):
d = DLX(path)
rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
return d, rec
def paced_sim(rec, ring, fill_per_frame, ticks_per_frame=8):
"""Paced-decoder ring sim. Returns TWO per-tick lookahead series.
THE ANSWER IS BRACKETED TO ONE RECORD AND IS NOT SHARPER THAN THAT. At
these rates the pipe delivers almost exactly one record per frame slot, so
"how many records are resident at slot i" depends on whether you look before
or after that slot's delivery -- and the two answers differ by one, every
time. Sampled after, this agreed with the rig's ceiling in 33 of 35 cells;
sampled before, it was exactly one record lower in 33 of 35. Neither is
wrong. Picking the one that matched would have been fitting the model to
the measurement and then reporting the agreement as a cross-check, so both
are returned and the caller prints the range. The rig sits at the top of it.
The producer is `aligned` (19_ring_stream.py): it will not start a record it
cannot finish before the end of the ring, and it will not place one over
bytes the decoder still owns. The decoder consumes exactly one record per
frame time and releases it whole.
Sub-stepping matters. Delivery and consumption interleave inside a frame
time on the rig -- the producer runs on MAME's machine-frame notifier, ~5x
per 12fps slot -- and a model that delivers a whole frame's bytes at once
can place a record into space the decoder has not released yet, or refuse
one it has. Eight sub-steps is well past the point the answer stops moving.
"""
n = len(rec)
live = [] # [idx, off, len] still owned by the decoder
wcur, nsent, credit = 0, 0, 0.0
lo, hi, ring_ref, rate_ref = [], [], 0, 0
def overlaps(off, ln):
return any(off < r[1] + r[2] and r[1] < off + ln for r in live)
for i in range(n):
if nsent < n:
lo.append(sum(1 for r in live if r[0] >= i))
for _ in range(ticks_per_frame):
credit += fill_per_frame / ticks_per_frame
while nsent < n:
r = int(rec[nsent])
if credit < r:
rate_ref += 1
break
w, hole = wcur, 0
if w + r > ring:
w, hole = 0, ring - wcur
if overlaps(w, r):
ring_ref += 1
break
# sector quantisation: a partial sector is not resident
credit -= r
live.append([nsent, w, r])
wcur, nsent = w + r, nsent + 1
if nsent < n:
hi.append(sum(1 for r in live if r[0] >= i))
# the decoder consumed record i during the slot and releases it whole
live = [r for r in live if r[0] > i]
return np.array(lo), np.array(hi), ring_ref, rate_ref
def main():
ap = argparse.ArgumentParser()
ap.add_argument("containers", nargs="*",
default=["tmp/rc_fr_singe_scsi_span.dlx"])
ap.add_argument("--kbps", type=float, nargs="+", required=True,
help="delivered pipe rates, KB/s. REQUIRED, no default "
"(FINDINGS 50): this project has never measured the "
"delivery pipe and a default is how the last unmeasured "
"one stayed load-bearing for five sessions.")
ap.add_argument("--ring", type=float, nargs="+",
default=[64, 96, 128, 192, 256, 384, 512])
a = ap.parse_args()
FPS = 12
for path in a.containers:
if not os.path.exists(path):
print(f"{path}: MISSING -- skipped\n"); continue
d, rec = records(path)
wire = rec.mean() * FPS / 1024 + RC.AUDIO_KBPS
print(f"=== {path}: {d.nframes} frames @ {d.fps}fps, mean record "
f"{rec.mean()/1024:.1f} KB, wire {wire:.1f} KB/s")
print(f"{'ring KB':>8} {'pipe':>8} {'ceiling':>9} {'build s':>8} "
f"{'mean':>11} bound")
for ring_kb in a.ring:
ring = int(ring_kb * 1024)
if rec.max() > ring:
print(f"{ring_kb:>8.0f} maxrec {rec.max():,} does not fit")
continue
for kbps in a.kbps:
fill = ((kbps - RC.AUDIO_KBPS) * 1024 / FPS) if kbps > 0 else 1e12
lo, hi, ring_ref, rate_ref = paced_sim(rec, ring, fill)
c_lo, c_hi = int(lo.max()), int(hi.max())
build = int(np.argmax(hi >= c_hi)) if len(hi) else -1
print(f"{ring_kb:>8.0f} {kbps:>8.0f} "
f"{f'{c_lo}-{c_hi}':>9} {build/FPS:>8.2f} "
f"{f'{lo.mean():.1f}-{hi.mean():.1f}':>11} "
f"{'ring' if ring_ref else 'rate'}")
# The surplus model, stated so it can be checked against the sweep
# above rather than believed: slack accrues at (pipe - wire) and a
# full ring holds `ceiling` records, so a branch point costs about
# ceiling*mean_record/(pipe - wire) seconds of play to earn back.
print()
print("Slack is accumulated, not owned. A bigger ring raises the ceiling AND")
print("lengthens the climb to it: the surplus (pipe - wire) is what fills it,")
print("and that is set by the encoder and the medium, not by the buffer.")
if __name__ == "__main__":
main()
tools/analysis/21_iplrom_dmac.py
+255
View File
@@ -0,0 +1,255 @@
#!/usr/bin/env python3
"""What the X68000's own ROM programs into the DMAC -- read out of the bytes.
python3 tools/analysis/21_iplrom_dmac.py [iplrom.dat]
FINDINGS 48.4 / ROADMAP B3 left the single-address vs dual-address question
open for the disk, priced it at 242 KB/s and 0.69 dB, and blocked it on
sourcing `scsiexrom.bin` so its DMAC init could be disassembled. The same
question was open for AUDIO and nobody had asked it: ROADMAP P6 budgets ADPCM
at 7.8 KB/s and `11_cpu_budget.py` charges those bytes the DISK's per-byte
rate, which is a guess about a channel whose configuration was never read.
It does not have to be a guess. **The IPL ROM is on this machine** -- MAME runs
the player rig with `-bios ipl10` -- and it programs all four HD63450 channels
itself. This script reads the configuration straight out of the ROM image and
decodes the MC68450 register fields, so every claim below is a byte at a named
address rather than a recollection about a chip.
It is a GATE, not a report: each piece of evidence is (address, expected bytes,
what it means), and a mismatch exits non-zero. If a different ROM revision is
pointed at it, it says so instead of quietly decoding something else.
SOURCED for the field layouts: MC68450 Direct Memory Access Controller,
Motorola, Jul 1989 (bitsavers) -- the same document FINDINGS 39 already cites
for the transfer timings in tools/analysis/buscost.py.
NOTE THE LAYER: this is the ROM's own choice of configuration, read from the
shipping image. It is not a measurement of a running machine, and it is not
proof that a different configuration is impossible -- our player programs these
registers itself. It is evidence about what Sharp's engineers could get the
board to do, from the vendor, for these exact devices.
"""
import sys, os, argparse, hashlib
BASE = 0xFE0000 # where the IPL ROM is mapped (and its 0xFF0000 alias)
# The image this was decoded against. A different revision is a different
# machine's answer, so it is named rather than assumed.
KNOWN = {
"7fd4caabac1d9169e289f0f7bbf71d8e":
"IPL 1.0 (MAME x68000 -bios ipl10), 131,072 B",
}
# --- MC68450 register map, by offset inside a channel's 0x40 block ----------
REG = {0x00: "CSR", 0x01: "CER", 0x04: "DCR", 0x05: "OCR", 0x06: "SCR",
0x07: "CCR", 0x0A: "MTC", 0x0C: "MAR", 0x14: "DAR", 0x1A: "BTC",
0x1C: "BAR", 0x25: "NIV", 0x27: "EIV", 0x29: "MFC", 0x2D: "CPR",
0x31: "DFC", 0x39: "BFC"}
XRM = {0: "burst",
1: "UNDEFINED",
2: "cycle steal WITHOUT hold (bus released between operands)",
3: "cycle steal with hold"}
DTYP = {0: "68000-compatible, EXPLICITLY addressed -> DUAL ADDRESS",
1: "6800-compatible, EXPLICITLY addressed -> DUAL ADDRESS",
2: "device with ACK, implicitly addressed -> SINGLE ADDRESS",
3: "device with ACK and RDY, implicit -> SINGLE ADDRESS"}
DPS = {0: "8-bit port", 1: "16-bit port"}
PCL = {0: "status input", 1: "status input with interrupt",
2: "start pulse", 3: "abort input"}
SIZE = {0: "byte", 1: "word", 2: "long word", 3: "byte, unpacked"}
CHAIN= {0: "none", 1: "UNDEFINED", 2: "array", 3: "linked array"}
REQG = {0: "auto-request at limited rate", 1: "auto-request at max rate",
2: "EXTERNAL request (one operand per device request)",
3: "auto-request first operand, external thereafter"}
def dcr(v):
return [f"XRM = {v>>6&3:02b} {XRM[v>>6&3]}",
f"DTYP = {v>>4&3:02b} {DTYP[v>>4&3]}",
f"DPS = {v>>3&1:b} {DPS[v>>3&1]}",
f"PCL = {v&3:02b} {PCL[v&3]}"]
def ocr(v):
return [f"DIR = {v>>7&1:b} " +
("device -> memory (read)" if v & 0x80 else "memory -> device (write)"),
f"SIZE = {v>>4&3:02b} {SIZE[v>>4&3]}",
f"CHAIN= {v>>2&3:02b} {CHAIN[v>>2&3]}",
f"REQG = {v&3:02b} {REQG[v&3]}"]
def scr(v):
m = {0: "no count", 1: "increment", 2: "decrement", 3: "UNDEFINED"}
return [f"MAC = {v>>2&3:02b} memory address {m[v>>2&3]}",
f"DAC = {v&3:02b} device address {m[v&3]}"]
# --- the evidence ----------------------------------------------------------
# (address, expected bytes, one-line description). Every register value quoted
# anywhere below comes out of one of these; nothing is typed in twice.
EV = [
(0xFF0BEA, "49f900e84080197c00080004197c0005",
"boot: lea $E84080,a4 (ch2) ; DCR=$08 ; SCR=$05..."),
(0xFF0C2E, "49f900e840c0197c00800004197c00040006197c00050029197c0001002d"
"197c00050031197c00050039297c00e92003",
"boot: lea $E840C0,a4 (ch3, ADPCM) ; DCR=$80 SCR=$04 MFC=$05 CPR=$01 "
"DFC=$05 BFC=$05 DAR=$E92003"),
(0xFF0D8E, "0480060429052d0031054480460469056d027105",
"boot: the ch0/ch1 init TABLE, ten (offset,value) pairs, written by the "
"loop at $FF0CD8"),
(0xFF0CE4, "217c00e940030014217c00e960010054",
"boot: DAR ch0 = $E94003 (FDC data) ; DAR ch1 = $E96001 (SASI data)"),
(0xFF9A82, "13fc003200e840c5610a13fc000200e920014e75",
"IOCS ADPCM PLAY: OCR(ch3) = $32 ; then command $02 to $E92001"),
(0xFF9A5E, "13fc00b200e840c5612e13fc000400e920014e75",
"IOCS ADPCM RECORD: OCR(ch3) = $B2 ; then command $04 to $E92001"),
(0xFF9A96, "13fc00ff00e840c023c900e840cc33c200e840ca",
"IOCS ADPCM arm: CSR=$FF ; MAR = a1 ; MTC = d2 (DCR/SCR untouched)"),
(0xFF9944, "13fc00ff00e8404013fc00b200e84045601013fc00ff00e8404013fc003200"
"e8404523c900e8404c33c300e8404a13fc008000e840474e75",
"IOCS SASI: OCR(ch1) = $B2 read / $32 write ; MAR ; MTC ; CCR = $80"),
]
ap = argparse.ArgumentParser()
ap.add_argument("rom", nargs="?",
default=os.path.expanduser("~/mame/roms/iplrom.dat"))
a = ap.parse_args()
if not os.path.exists(a.rom):
sys.exit(f"missing {a.rom} -- point this at the IPL ROM MAME boots the rig "
f"with (-bios ipl10).")
d = open(a.rom, "rb").read()
md5 = hashlib.md5(d).hexdigest()
print(f"{a.rom}: {len(d):,} B, md5 {md5}")
if md5 in KNOWN:
print(f" {KNOWN[md5]}\n")
else:
sys.exit(f"\nUNKNOWN ROM. Every field decoded below was read out of\n"
f" {list(KNOWN.values())[0]}\n"
f"and a different revision is a different machine's answer, not a "
f"detail. Add its\nmd5 to KNOWN only after re-reading the sites -- "
f"the addresses are revision-specific.")
print("EVIDENCE -- each line is bytes at an address, not a recollection")
bad = 0
for addr, hx, what in EV:
want = bytes.fromhex(hx)
got = d[addr - BASE: addr - BASE + len(want)]
ok = got == want
bad += not ok
print(f" {'OK ' if ok else 'FAIL'} ${addr:06X} {what}")
if not ok:
print(f" expected {want.hex()}\n got {got.hex()}")
if bad:
sys.exit(f"\nFAIL: {bad} evidence site(s) do not hold. The decode below "
"would be about\nsome other code, so it is not printed.")
# The ch0/ch1 table, decoded from the bytes rather than restated.
tbl = d[0xFF0D8E - BASE: 0xFF0D8E - BASE + 20]
init = {}
for i in range(0, len(tbl), 2):
off, val = tbl[i], tbl[i + 1]
init[(off >> 6, off & 0x3F)] = val
init[(2, 0x04)] = 0x08 # from the inline moves at $FF0BEA
init[(2, 0x06)] = 0x05
init[(2, 0x2D)] = 0x03
init[(3, 0x04)] = 0x80 # ...and at $FF0C2E
init[(3, 0x06)] = 0x04
init[(3, 0x2D)] = 0x01
DEV = {0: ("FDC", "$E94003"), 1: ("SASI", "$E96001"),
2: ("IOCS _DMAMOVE (general purpose)", "set per call"),
3: ("ADPCM MSM6258V", "$E92003")}
print("\nWHAT THE ROM PROGRAMS, per channel")
for ch in range(4):
name, dar = DEV[ch]
print(f"\n ch{ch} base $E840{ch*0x40:02X} {name} DAR = {dar}")
v = init[(ch, 0x04)]
print(f" DCR = ${v:02X}")
for line in dcr(v):
print(f" {line}")
v = init[(ch, 0x06)]
print(f" SCR = ${v:02X} " + " ; ".join(scr(v)))
print(f" CPR = ${init[(ch,0x2D)]:02X} channel priority "
f"({init[(ch,0x2D)]}, 0 = highest)")
print("\nAND THE PER-TRANSFER OCR, written every time a transfer is armed")
for label, ch, v in (("ADPCM playback", 3, 0x32), ("ADPCM record", 3, 0xB2),
("SASI write", 1, 0x32), ("SASI read", 1, 0xB2)):
print(f"\n {label:<15} ch{ch} OCR = ${v:02X}")
for line in ocr(v):
print(f" {line}")
print(f"""
WHAT THIS SETTLES
1. AUDIO IS DUAL ADDRESS, AND IT CANNOT HOLD THE BUS. ch3 DCR = $80: DTYP =
00, explicitly addressed, so every ADPCM byte is a MEMORY READ FOLLOWED BY A
DEVICE WRITE -- not the single-address 5 clocks the disk debit is written in.
XRM = 10 is cycle steal WITHOUT hold and OCR REQG = 10 is external request,
so the DMAC arbitrates for the bus ONCE PER BYTE and gives it straight back.
There is no burst to amortise the arbitration over.
2. THE PORT IS 8 BITS AND THE OPERAND IS A BYTE. DCR DPS = 0, OCR SIZE = 11.
One MSM6258V byte is two 4-bit samples, so 15.6 kHz is 7,812.5 BYTES/s and
7,812.5 DMA REQUESTS/s -- the request count does not halve the way a 16-bit
port's would. That is the FINDINGS 43 unit trap, in the other stream.
3. THE DISK CHANNEL IS PROGRAMMED IDENTICALLY, AND THAT IS THE BIGGER NEWS.
ch1 (SASI, DAR = $E96001) gets DCR = $80 and OCR = $B2 -- dual address,
8-bit port, cycle steal WITHOUT hold, external request. Byte by byte, with a
full arbitration each time, exactly like the audio. ch0 (FDC) too. Sharp
programs every explicitly-addressed 8-bit device on this board the same way.
This is not scsiexrom.bin and it does not close ROADMAP B3 -- a different
ROM drives a different SPC. But it is the same vendor, the same DMAC and the
same class of device, and it lands on the EXPENSIVE side of B3's 242 KB/s.
4. AND IT IS OUTSIDE THE BRACKET THE PROJECT HAS BEEN COSTING P4 IN.
FINDINGS 42.4-42.6 brackets W, the clocks stolen per delivered byte, at
5..12, and reports that W <= 6 fits 0/120 frames while W = 8 misses 47/120.
The ROM's own disk configuration costs 16..19. It is still true that the
player programs these registers itself and the choice is ours (42.6) -- but
the only worked example on the machine sits ABOVE the whole bracket, and
nothing in this tree has yet shown that a cheaper configuration is reachable
for an explicitly-addressed port. Treat W <= 12 as a REQUIREMENT ON THE
PLAYER'S DMAC PROGRAMMING, not as a range the hardware hands us.
5. AUDIO OUTRANKS THE DISK AT THE ARBITER. CPR: FDC 0, ADPCM 1, SASI 2,
_DMAMOVE 3, lower being higher priority. When both channels want the bus in
the same slot, the ROM's arrangement serves ADPCM first. An audio byte is
never the thing that waits; a video byte is.
""")
# --- what it costs ---------------------------------------------------------
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import buscost as B
ADPCM_HZ = 15625.0 # 8 MHz MSM6258V clock / 512
ADPCM_BPS = ADPCM_HZ / 2 # 4-bit samples, two to a byte
FPS, CPUHZ = 12.0, 10e6
lo = B.DMA_DUAL_BYTE_CLK + B.DMA_FRONT_CLK + B.DMA_BACK_CLK
hi = B.DMA_DUAL_BYTE_CLK + B.DMA_FRONT_CLK_WORST + B.DMA_BACK_CLK
bpf = ADPCM_BPS / FPS
print(f"WHAT IT COSTS, at the configuration above\n"
f" 15.6 kHz mono = {ADPCM_HZ:,.0f} samples/s = {ADPCM_BPS:,.1f} B/s "
f"= {ADPCM_BPS/1024:.2f} KiB/s\n"
f" (ratectl.AUDIO_KBPS is 7.8, which is this figure in DECIMAL kB; "
f"as KiB it is {ADPCM_BPS/1024:.2f})\n"
f" dual-address byte transfer {B.DMA_DUAL_BYTE_CLK} clk "
f"(read {B.DMA_READ_CLK} + write {B.DMA_WRITE_CLK}, Fig 4-25 sheet 4 note 2)\n"
f" + arbitration, EVERY byte {B.DMA_FRONT_CLK}..{B.DMA_FRONT_CLK_WORST}"
f" front + {B.DMA_BACK_CLK} back (sect 4.5.2.1/4.5.2.2)\n"
f" = {lo}..{hi} clocks per audio byte\n\n"
f" per frame at {FPS:g} fps: {bpf:,.1f} B costs {bpf*lo:,.0f}..{bpf*hi:,.0f} "
f"clocks of {CPUHZ/FPS:,.0f}\n"
f" = {100*bpf*lo/(CPUHZ/FPS):.2f}%..{100*bpf*hi/(CPUHZ/FPS):.2f}% of the "
f"frame, stolen from the 68000\n\n"
f" 11_cpu_budget.py charges audio --dma-clocks-per-byte, default 5, "
f"described\n as 'single-address, bus held, no drive wait'. The ROM says "
f"audio is neither\n single-address nor able to hold the bus, so that "
f"debit is {lo/5:.1f}x..{hi/5:.1f}x too small.\n"
f" In absolute terms it is small -- but it is small IN THE RESOURCE THE "
f"PROJECT IS\n SHORT OF, and it was being taken from the wrong side of "
f"an open question.")
tools/analysis/buscost.py
+215
View File
@@ -0,0 +1,215 @@
"""Bus-cycle cost of src/player/decode.s and of tools/bench/blit.s's v6 spans.
A 68000 bus cycle is 4 clocks (S0-S7) with no wait states, and the 68000
prefetches every instruction word over the same bus. So a block's bus cost is
`instruction words + data accesses`, a long access counting twice on the 16-bit
bus and `movem.l` of N registers counting 2N.
The per-path word counts are read off tools/bench/decode.lst and
tools/bench/blit.s. tools/analysis/15_bus_occupancy.py checks the DATA half of
this table against tools/bench/c68k/c68k_bench, which counts every bus callback
the C68K core makes: they agree to 0.04%. The prefetch half cannot be measured
from either emulator -- MAME does not expose a fetch count and C68K reads
opcodes through a host pointer with no callback -- so it rests on that check.
"""
BUS_CLK = 4
# --- decode.s, per block ---------------------------------------------------
# dispatch move.b (a1),d0 / lsr.b / and.w #3 / beq .sk 6w, 1 read
# + subq / beq .v1 -> 8w
# + subq / bne .rw -> 10w
# V4 body $10090..$100E2 = 82 B = 41w; 4 x (1 byte read
# + movem.l 2 = 4 reads + 2 move.l = 4 writes) = 36
# V1 body $100E2..$10106 = 36 B = 18w; 1 byte read
# + movem.l 8 = 16 reads + 4 x movem.l 2 = 16 wr = 33
# RAW body $10106..$10164 = 94 B = 47w; 8 x (2 byte reads
# + 1 move.l = 2 writes) = 32
BODY = {0: (0, 0), 1: (18, 33), 2: (41, 36), 3: (47, 32)}
DISPATCH = {0: 6, 1: 8, 2: 10, 3: 10}
SK_TAIL = 1 # addq.l #8,a4
GROUP_HEAD = 3 # tst.b (a1) + beq allskip
GROUP_TAIL = 4 # addq.l #1,a1 / cmpa.l a5,a4 / bne byteloop
ALLSKIP = 9 # the whole four-block fast path, tst.b included
ROW_HEAD, ROW_TAIL = 3, 7
# --- blit.s v6 spans -------------------------------------------------------
# One chain unit moves 12 registers = 48 B = 24 pixels:
# movem.l (a0)+,12 = 2w instr + 24 word reads = 26
# movem.l 12,(a2) = 2w instr + 24 word writes = 26
# lea 48(a2),a2 = 2w instr = 2
# Per span: move.l (a0)+,a2 (1w + 2 reads) + move.w (a0)+,d0 (1w + 1 read)
# + jmp v6ch(pc,d0.w) (2w) + dbra (2w) = 9
V6_UNIT_PX = 24
V6_UNIT_BUS = 54
V6_SPAN_BUS = 9
V6_SPAN_CYC = 43.7 # MEASURED, FINDINGS 30
V6_PX_CYC = 9.152 # MEASURED, FINDINGS 30
# --- a DMAC array-chaining span -------------------------------------------
# SOURCED, MC68450 Direct Memory Access Controller, Motorola, Jul 1989
# (bitsavers). These replace session-10's first pass, which guessed 2 bus
# cycles a pixel from bus arithmetic and was 12% optimistic.
#
# Fig 4-25 sheet 4, DUAL ADDRESS / OPERAND SIZE IS WORD / DEVICE SIZE IS
# 16-BITS, D->M or M->D: {WORD READ, WORD WRITE} = 9 CLOCKS.
# Confirmed by the long-operand row: two of each = 18 clocks.
# Fig 4-25 note 2: reads are 4 clocks and WRITES ARE 5. That extra clock on
# every write is the whole story -- it is why the DMAC does not beat a 68000
# movem chain, which writes in 4.
DMA_PX_CLK = 9
# Fig 4-25 sheet 1, SEQUENTIAL ARRAY CHAINING: 36 CLOCKS per entry (three
# word reads to fetch the 6-byte entry, plus reload).
DMA_CHAIN_CLK = 36
# Sect 4.5.2.1 front-end overhead 5 clocks best case, 8 worst; 4.5.2.2
# back-end 2 clocks best. Once per period of bus ownership, not per span.
DMA_FRONT_CLK, DMA_BACK_CLK = 5, 2
DMA_FRONT_CLK_WORST = 8
# Fig 4-25 note 2 again, split out because the ADPCM channel needs the halves
# apart: a DMAC READ is 4 clocks and a WRITE is 5, on either bus width. A
# dual-address BYTE transfer is therefore one 4 and one 5.
DMA_READ_CLK, DMA_WRITE_CLK = 4, 5
DMA_DUAL_BYTE_CLK = DMA_READ_CLK + DMA_WRITE_CLK
# Fig 4-25 sheet 3, SINGLE ADDRESS: W/B READ 4 clocks, W/B WRITE 5 clocks.
# A device->memory disk transfer is one memory WRITE = 5 clocks if the DMAC
# holds the bus, or 5 + front + back = 12 if it arbitrates per word.
# FINDINGS 5's long-standing 8 clk/word ESTIMATE sits inside that range.
DMA_DISK_CLK_WORD_HELD, DMA_DISK_CLK_WORD_ARB = 5, 12
# --- the ADPCM stream, as the IPL ROM actually programs it -----------------
# READ OUT OF THE ROM, not recalled: tools/analysis/21_iplrom_dmac.py decodes
# the HD63450 registers Sharp's own IPL 1.0 writes, and gates on the bytes still
# being there. Channel 3, DCR = $80, OCR = $32 for playback:
#
# DTYP = 00 explicitly addressed -> DUAL ADDRESS (memory read, device write)
# DPS = 0 8-bit port -> one byte per operand
# XRM = 10 cycle steal WITHOUT hold, and REQG = 10 external request
# -> the DMAC arbitrates ONCE PER BYTE. No burst to amortise over.
#
# So an audio byte costs the dual-address transfer PLUS a full arbitration,
# every time -- unlike a disk record, which can at least be argued to hold the
# bus for a run of bytes. This is the number the audio side of the I/O debit
# should be denominated in; DISK_CLK_BYTE is not it.
ADPCM_SAMPLE_HZ = 15625.0 # MSM6258V, 8 MHz clock / 512 (the 15.6 kHz mode)
ADPCM_BYTES_PER_S = ADPCM_SAMPLE_HZ / 2 # 4-bit samples, two to a byte
ADPCM_CLK_BYTE_BEST = DMA_DUAL_BYTE_CLK + DMA_FRONT_CLK + DMA_BACK_CLK # 16
ADPCM_CLK_BYTE_WORST = DMA_DUAL_BYTE_CLK + DMA_FRONT_CLK_WORST + DMA_BACK_CLK # 19
# The 68000 cannot execute while another master owns the bus: no cache, and a
# two-word prefetch queue that empties immediately. So DMA time is ADDITIVE to
# CPU time, not overlapped -- which is what FINDINGS 35's flat debit assumed
# and session 10's first pass wrongly "refined".
DMA_OVERLAPS = False
def pad24(npix):
return -(-npix // V6_UNIT_PX) * V6_UNIT_PX
def block_bus(mode_map, spanned=None):
"""(instruction words, data accesses) for one frame's CPU block decode.
`spanned` is a boolean array the same shape as mode_map marking blocks a
span will paint instead; those blocks still cost their dispatch, because
the mode map is walked either way, but not their body."""
nby, nbx = mode_map.shape
pref = nby * (ROW_HEAD + ROW_TAIL)
data = 0
for by in range(nby):
row = mode_map[by]
sp = spanned[by] if spanned is not None else None
for gi in range(0, nbx, 4):
g = row[gi:gi + 4]
if (g == 0).all():
pref += ALLSKIP
data += 1
continue
pref += GROUP_HEAD + GROUP_TAIL - 1 # BLOCK 0 has no lsr.b
data += 1
for k, b in enumerate(g):
b = int(b)
if sp is not None and sp[gi + k]:
b = 0 # the span paints it
pw, pd = BODY[b]
pref += DISPATCH[b] + pw + SK_TAIL
data += 1 + pd
return pref, data
# --- v7: v6 with a finer tail (MEASURED, session 11, FINDINGS 40) ----------
# v6 pads every span up to 24 pixels because its unrolled chain is built from
# 12-register movem units, and FINDINGS 39.3 attributed 86% of the DMAC array
# chain's advantage over v6 to exactly that padding. v7 keeps the coarse chain
# and appends a second chain whose unit is one `move.l (a0)+,(a2)+` -- 2 pixels,
# so the quantum is 2 and a run of 4x4 blocks pads to NOTHING.
#
# Session 10 proposed a 2-REGISTER MOVEM tail (4 pixels, derived at 56 clocks)
# and that would have been the wrong instruction: movem.l (a0)+,d0-d1 plus
# movem.l d0-d1,(a2) plus the lea is 14 bus cycles for 4 pixels, where two plain
# move.l are 10. The plainest instruction on the machine wins the tail.
#
# The second entry point needs a second dispatch, and the fine displacement is
# carried MID-STREAM (after the coarse pixels, before the fine ones) rather than
# in the span record, so the decoder holds nothing extra across the copy and
# keeps all 12 payload registers. Costed as 2 more bytes per span.
#
# MEASURED by tools/bench/span.sh (blit.s v7, 13 span lengths, every config
# pixel-exact): cycles = 66.0/span + 9.143/coarse pixel + 9.978/fine pixel,
# fitting all 13 to within 0.2%.
V7_SPAN_CYC = 66.0 # MEASURED, FINDINGS 40
V7_CPX_CYC = 9.143 # MEASURED, FINDINGS 40 (24-pixel coarse unit)
V7_FPX_CYC = 9.978 # MEASURED, FINDINGS 40 (2-pixel fine unit)
V7_FINE_PX = 2
# Bus: per span v6's 9 plus a second {move.w (a0)+,d0 ; jmp} = 2 + 2.
# Per fine unit: move.l (a0)+,(a2)+ = 1 instruction word + 2 reads + 2 writes.
V7_SPAN_BUS = 13
V7_FINE_BUS = 5
V7_SPAN_HDR = 8 # {u32 address, u16 coarse disp} + u16 fine disp
def pad2(npix):
return -(-npix // V7_FINE_PX) * V7_FINE_PX
def v7_span(npix):
"""(pixels carried, CPU clocks) for a v7 span of npix pixels."""
k, r = divmod(pad2(npix), V6_UNIT_PX)
return (k * V6_UNIT_PX + r,
V7_SPAN_CYC + k * V6_UNIT_PX * V7_CPX_CYC + r * V7_FPX_CYC)
def v7_span_bus(npix):
"""Bus CYCLES a v7 span occupies -- instruction words plus data accesses."""
p, d = v7_span_split(npix)
return p + d
def v7_span_split(npix):
"""(instruction words, data accesses) for one v7 span, separately.
15_bus_occupancy.py needs the two apart, because the DATA half is what the
C68K harness can check and the PREFETCH half is what rides on that check.
per span move.l (a0)+,a2 1 word + 2 reads
move.w (a0)+,d0 1 word + 1 read (coarse displacement)
jmp (pc,d0.w) 2 words
move.w (a0)+,d0 1 word + 1 read (fine, from mid-stream)
jmp (pc,d0.w) 2 words
dbra 2 words -> 9 words, 4 accesses
per coarse 2 movem.l of 12 + lea = 6 words, 24 reads + 24 writes
per fine move.l (a0)+,(a2)+ = 1 word, 2 reads + 2 writes
"""
k, r = divmod(pad2(npix), V6_UNIT_PX)
f = r // V7_FINE_PX
return (9 + k * 6 + f * 1,
4 + k * 48 + f * 4)
# Per FRAME, decode.s's paint_spans entry and exit: the span count read, the
# guard branch, and the push/pop of a1 that buys back a twelfth payload
# register. Two long accesses a frame against 24 pixels a chain unit.
V7_FRAME_PREF, V7_FRAME_DATA = 7, 7
def v6_span_bus(npix):
k = pad24(npix) // V6_UNIT_PX
return V6_SPAN_BUS + k * V6_UNIT_BUS
tools/bench/blit.s
+220
View File
@@ -31,6 +31,64 @@
; the block needs only one base pointer. V4 deliberately scrambles the
; picture (it reads a row-linear source in block order); it is a timing
; probe, which is why the correctness snapshot is taken after V1.
; V5 ROW-LINEAR LITERAL SPANS, the mode priced in FINDINGS 29 and never
; measured. Walks a stream of per-row span records
; row: u16 nspans, then nspans * { u16 x, u16 npix, npix*u16 pixels }
; for 192 rows, copying each span's word-expanded pixels straight from
; the stream buffer into GVRAM. Unlike V1-V4 the work per call is set by
; the STREAM, not by the code, so one variant measures every span length:
; tools/bench/prep_spans.py generates a stream per span length and
; tools/bench/span.lua times them and fits cycles = A*spans + B*pixels.
; The point of the measurement is A -- the per-span overhead FINDINGS 29
; guessed at 50 cycles -- and how much B degrades from V1's 9.08 when a
; span is too short to burst. Every config covers the whole frame, so
; V5 draws the SAME picture V1 does and can be verified, not just timed.
;
; Bursts are 8 registers (d0-d3/a3-a6 = 32 bytes = 16 pixels), not V1's
; 12: a0/a1/a2 and d4-d7 are all live across a span (stream, row base,
; destination, and three counters). The remainder is copied move.l at a
; time with a leading move.w when it is odd, so a 4-pixel span never
; reaches a movem at all -- which is exactly the case FINDINGS 29's
; full-row-width extrapolation flatters.
;
; V6 the SAME spans with the arithmetic moved into the encoder. V5 measures
; a decoder that is handed (x, npix) and has to work out how to copy it;
; most of its per-span cost is that working-out, and an encoder can do it
; once at build time instead of 12 times a second. V6's record is
; { u32 absolute GVRAM address, u16 jump displacement } -- no row
; structure, no counters, no remainder logic -- and the displacement
; jumps into an unrolled chain of 24-pixel copy units, so a span of any
; supported length is straight-line code with no loop at all.
; GVRAM sits at a fixed $C00000 on every X68000, so absolute destinations
; are a legitimate thing for an encoder to bake in.
;
; Two consequences of the format. Span lengths are multiples of 24
; pixels, and a span may overrun the 256 visible pixels of its row by up
; to 23 -- harmless, because the line stride is 1024 bytes and only the
; first 512 are displayed, so the overrun lands in the invisible half.
; And with row and remainder handling gone, 12 registers are free again
; (d0-d6/a1/a3-a6), which is why the unit is 24 pixels and not V5's 16.
;
; V7 v6 with a SECOND, finer chain for the tail (FINDINGS 39.4). v6 pays for
; its 24-pixel quantum in padding: an average span wastes ~11 pixels, and
; FINDINGS 39.3 attributes 86% of the DMAC array-chain's advantage over v6
; to exactly that. V7 keeps the 24-pixel coarse chain and appends a chain
; of 2-pixel units, so a span is 24*c + 2*f pixels and the padding is at
; most one pixel -- ZERO for the real case, where a span is a run of 4x4
; blocks and its length is a multiple of 4.
;
; The fine unit is `move.l (a0)+,(a2)+` (20 cycles, 2 pixels), NOT a
; 2-register movem: movem.l (a0)+,d0-d1 plus movem.l d0-d1,(a2) plus the
; lea is 52+8 cycles for 4 pixels, so the obvious "smaller movem" tail is
; 50% dearer per pixel than the plainest instruction on the machine.
;
; The second entry point costs a second dispatch, and the trick that pays
; for it is that the fine displacement is NOT in the span record: it sits
; in the STREAM, after the coarse pixels and before the fine ones. The
; coarse chain falls out into `move.w (a0)+,d0 / jmp`, by which point d0
; is dead payload and a0 is pointing exactly at it. So v7 holds nothing
; extra across the copy and keeps all 12 payload registers -- a record is
; still {u32 address, u16 displacement}, with one more u16 mid-span.
;
; 12 registers per movem burst (d0-d7/a2-a5 = 48 bytes) is the maximum
; available: a0=src, a1=dst, a6=end sentinel. The row counter lives in the
@@ -43,10 +101,18 @@
FLAG = $18000 ; 0 idle / 1 running / $FF done
VAR = $18004 ; variant selector, written by Lua
ITER = $18008 ; iteration count, written by Lua
SPTR = $1800C ; V5 span stream pointer, written by Lua
SRCW = $60000 ; word-expanded frame 192*512 = 96KB
SRCB = $80000 ; byte-per-pixel frame 192*256 = 48KB
DST0 = $C08000 ; GVRAM + 32*1024 (first picture row)
DSTE = $C38000 ; GVRAM + 224*1024 (one past last)
ROWS = 192 ; picture rows a V5 stream describes
V6UNIT = 12 ; bytes of code per V6 chain unit
V6MAX = 11 ; chain units = 11*24 = 264 pixels >= one row
V7CU = 12 ; bytes of code per V7 COARSE unit (24 px)
V7CN = 11 ; coarse units: 11*24 = 264 px >= one row
V7FU = 2 ; bytes of code per V7 FINE unit (2 px)
V7FN = 11 ; fine units: 11*2 = 22 px > one coarse unit
org $10000
start:
@@ -58,6 +124,12 @@ start:
beq v2
cmp.l #4,d0
beq v4
cmp.l #5,d0
beq v5
cmp.l #6,d0
beq v6
cmp.l #7,d0
beq v7
bra v3
; ---------------------------------------------------------------- V1
@@ -152,5 +224,153 @@ v4blk: movem.l (a0)+,d0-d7 ; 32 bytes = one 4x4 block, expanded
bne v4
bra done
; ---------------------------------------------------------------- V5
; a0 stream, a1 row base, a2 span destination, d7 rows, d6 spans, d5 pixels,
; d4 burst/tail counter. Everything else (d0-d3/a3-a6) is burst payload.
v5: move.l SPTR.l,a0
lea DST0,a1
move.w #ROWS-1,d7
v5row: move.w (a0)+,d6 ; spans in this row
subq.w #1,d6
bmi.s v5eor ; a row may legitimately have none
v5span: move.w (a0)+,d0 ; x, in pixels
add.w d0,d0 ; one pixel = one word
lea 0(a1,d0.w),a2
move.w (a0)+,d5 ; pixels in this span
move.w d5,d4
lsr.w #4,d4 ; 16-pixel bursts
beq.s v5tail
subq.w #1,d4
v5burst: movem.l (a0)+,d0-d3/a3-a6 ; 32 bytes straight out of the stream
movem.l d0-d3/a3-a6,(a2)
lea 32(a2),a2
dbra d4,v5burst
v5tail: moveq #15,d4
and.w d5,d4 ; 0..15 pixels left
beq.s v5eos
lsr.w #1,d4 ; C = odd pixel count
bcc.s v5t2
move.w (a0)+,(a2)+
v5t2: subq.w #1,d4
bmi.s v5eos
v5tl: move.l (a0)+,(a2)+
dbra d4,v5tl
v5eos: dbra d6,v5span
v5eor: lea 1024(a1),a1
dbra d7,v5row
subq.l #1,ITER.l
bne v5
bra done
; ---------------------------------------------------------------- V6
; a0 stream, a2 destination, d7 spans remaining; everything else is payload.
v6: move.l SPTR.l,a0
move.w (a0)+,d7 ; total spans in the frame
subq.w #1,d7
v6span: move.l (a0)+,a2 ; absolute GVRAM destination
move.w (a0)+,d0 ; (V6MAX - units) * V6UNIT, from the encoder
jmp v6ch(pc,d0.w)
v6ch:
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
dbra d7,v6span
subq.l #1,ITER.l
bne v6
bra done
; ---------------------------------------------------------------- V7
; a0 stream, a2 destination, d7 spans remaining; everything else is payload.
; Stream per span: u32 dest, u16 coarse disp, c*48 B pixels,
; u16 fine disp, f*4 B pixels.
v7: move.l SPTR.l,a0
move.w (a0)+,d7 ; total spans in the frame
subq.w #1,d7
v7span: move.l (a0)+,a2 ; absolute GVRAM destination
move.w (a0)+,d0 ; (V7CN - coarse) * V7CU
jmp v7ch(pc,d0.w)
v7ch:
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
movem.l (a0)+,d0-d6/a1/a3-a6
movem.l d0-d6/a1/a3-a6,(a2)
lea 48(a2),a2
v7cx: move.w (a0)+,d0 ; (V7FN - fine) * V7FU, from mid-stream
jmp v7fh(pc,d0.w)
v7fh:
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
move.l (a0)+,(a2)+
dbra d7,v7span
subq.l #1,ITER.l
bne v7
bra done
done: move.l #$FF,FLAG.l ; timer stops here
halt: bra.s halt
tools/bench/c68k/Makefile
+19
View File
@@ -0,0 +1,19 @@
# Build the headless C68K cycle harness. PX68K points at a px68k checkout;
# only m68000/c68k.c and the two header dirs are used -- no SDL, no ROMs.
PX68K ?= $(HOME)/src/px68k
# -no-pie is LOAD-BEARING, not a tidy-up. C68K is 64-bit-unsafe on purpose:
# its MOVEM macros do `src = (UINT32)(&D0)` -- they truncate the host address of
# the CPU register file to 32 bits and dereference it -- and C68k_Set_Fetch
# stores the opcode-fetch base in a UINT32 too. Under the default PIE the
# binary loads near 0x555555550000 and the first movem segfaults. -no-pie puts
# the image at 0x400000, and the harness mmaps its arena with MAP_32BIT, so
# every pointer C68K truncates still round-trips.
CFLAGS = -O2 -fno-strict-aliasing -no-pie -Wall -Wno-unused-result \
-Wno-int-to-pointer-cast -Wno-pointer-to-int-cast \
-I$(PX68K)/m68000 -I$(PX68K)/x11 -I$(PX68K)/win32api
c68k_bench: harness.c $(PX68K)/m68000/c68k.c
$(CC) $(CFLAGS) -no-pie -o $@ harness.c $(PX68K)/m68000/c68k.c
clean:
rm -f c68k_bench
tools/bench/c68k/compare.py
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""Two emulators, one decoder: MAME's cycles against px68k's C68K core.
python3 tools/bench/c68k/compare.py [--mame tmp/mame_timed.log]
[--c68k tmp/c68k.log]
WHY THIS EXISTS. Every 68000 cycle figure in FINDINGS 24-35 comes from one
instrument. This puts a second, structurally different one next to it:
MAME 0.277 M68000 is the microcode core (src/devices/cpu/m68000/m68000.lst
+ m68000gen.py), NOT Musashi -- timing emerges from the 68000's
modelled micro-sequence and 4-clock bus cycles.
C68K a static per-instruction cycle table hand-transcribed from the
Motorola manual (ORI_CLOCKS_* / EA_CLOCKS_* in c68kmacro.h).
Those are two different ways of being right, so agreement is evidence and
disagreement localises to whichever instruction the anchors separate. NEITHER
charges GVRAM wait states, so both are the same lower bound on real hardware.
"""
import argparse, re, sys
ap = argparse.ArgumentParser()
ap.add_argument("--mame", default="tmp/mame_timed.log")
ap.add_argument("--c68k", default="tmp/c68k.log")
ap.add_argument("--meta", default="tmp/decode_meta.lua")
a = ap.parse_args()
meta = open(a.meta).read()
fps = int(re.search(r"fps=(\d+)", meta).group(1))
budget = 10_000_000 / fps
# anchor name -> stream offset, so the two logs can be joined: decode.lua
# reports by name, the C68K harness by offset.
names = {int(o): n for n, o in re.findall(r'name="([^"]+)", off=(\d+)', meta)}
mame = {}
txt = open(a.mame, errors="replace").read()
for nm, cyc in re.findall(r"\[DEC\] frame @ (.+?)\n.*?->\s+(\d+) cycles/frame", txt):
mame[nm.strip()] = int(cyc)
m_seq = re.search(r"full \d+-frame pass.*?\n.*?->\s+(\d+) cycles/frame", txt)
c68k, c_seq = {}, None
for line in open(a.c68k, errors="replace"):
m = re.search(r"anchor off=(\d+)\s+(\d+) cyc", line)
if m and int(m.group(1)) in names:
c68k[names[int(m.group(1))]] = int(m.group(2))
m = re.search(r"sequential pass = (\d+) cyc, mean (\d+)", line)
if m:
c_seq = int(m.group(2))
if not mame:
sys.exit(f"no MAME anchor timings in {a.mame} -- run decode.lua WITHOUT "
f"DLX_VERIFY_ONLY=1 and give -seconds_to_run enough to finish")
w = max(len(n) for n in c68k) + 2
print(f"{'anchor':<{w}}{'MAME':>10}{'C68K':>10}{'delta':>9} {'MAME':>7}{'C68K':>7} of a {fps}fps frame")
rows = []
for nm, c in c68k.items():
m = mame.get(nm)
if m is None:
print(f"{nm:<{w}}{'--':>10}{c:>10}{'':>9} {'--':>7}{100*c/budget:>6.1f}% (MAME run did not reach it)")
continue
d = 100 * (c - m) / m
rows.append(d)
print(f"{nm:<{w}}{m:>10}{c:>10}{d:>+8.2f}% {100*m/budget:>6.1f}%{100*c/budget:>6.1f}%")
if m_seq and c_seq:
m, c = int(m_seq.group(1)), c_seq
d = 100 * (c - m) / m
print(f"{'MEAN over the window':<{w}}{m:>10}{c:>10}{d:>+8.2f}% "
f"{100*m/budget:>6.1f}%{100*c/budget:>6.1f}%")
if rows:
print(f"\nspread over {len(rows)} anchors: {min(rows):+.2f}% .. {max(rows):+.2f}%")
print("C68K reads HIGH throughout." if min(rows) > 0 else
"C68K reads high on some anchors and low on others.")
print("Neither instrument charges GVRAM wait states, so both are the same\n"
"LOWER BOUND: this bounds cycle-table error, not the distance to a\n"
"real X68000 (docs/BENCHMARK.md Tier 3).")
tools/bench/c68k/harness.c
+329
View File
@@ -0,0 +1,329 @@
/* Headless C68K cycle harness -- an independent second opinion on every
* 68000 cycle figure in FINDINGS 24-35.
*
* WHY. Every one of those numbers comes from ONE instrument: MAME 0.277's
* Musashi core, timed host-side from manager.machine.time. A cycle table is a
* hand-transcribed artefact; if Musashi's is wrong for our instruction mix, the
* 833,333-cycle budget is wrong by the same amount and nothing in the tree
* would show it. This runs the SAME decode.bin against the SAME
* decode_data.bin under px68k's C68K core, which has a completely separate
* cycle table (ORI_CLOCKS_* + EA_CLOCKS_* in c68kmacro.h) written by a
* different author from the same Motorola manual.
*
* WHAT IT DOES AND DOES NOT SETTLE. C68K, like MAMEs x68000, charges NO
* GVRAM wait states -- grep the px68k tree, there is no bus-timing model
* anywhere in x68k/*.c. So this is the same LOWER BOUND, measured twice. It
* cross-checks the cycle table. It says nothing about real-hardware wait
* states; that needs XM6 TypeG or an actual X68000 (docs/BENCHMARK.md Tier 3).
*
* WHY NOT JUST RUN px68k. The decoder touches nothing but RAM, the control
* block and GVRAM: no IPL, no CRTC, no MFP, no interrupts (the MAME rig masks
* them with SR=$2700). Booting a whole emulated machine would add SDL, ROMs
* and a 55Hz sampling clock to a measurement that wants none of them. Linking
* the core alone also buys EXACTNESS: the stop cycle is captured inside the
* write callback, so a frame's cost is known to within one instruction rather
* than MAME's 1/55.46 s. That is why the anchors here run iter=1 -- decode.lua
* only iterates to beat its own timing granularity.
*
* MEMORY MODEL mirrors px68k exactly, because the core requires it: RAM is
* stored BYTE-SWAPPED (MEM[addr ^ 1], mem_wrap.c:420) so C68K's
* READ_IMM_16() = *(UINT16 *)PC works with no swap on a little-endian host.
* GVRAM word writes discard the high byte, as the hardware and MAME's
* gvram_w case 0x0100 both do.
*
* The harness is self-validating: --dump writes the decoded screen and
* verify_c68k.py checks it pixel-for-pixel against tools/encoder/dlx.py. If
* the byte-swap or the memory map were wrong the decode could not come out
* exact, so a green verify is what licenses the cycle numbers next to it.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include "c68k.h"
/* c68k.c declares these extern and tests BusErrHandling every instruction. */
unsigned int BusErrHandling = 0;
unsigned int BusErrAdr = 0;
void Error(const char *s) { fprintf(stderr, "c68k: %s\n", s); exit(3); }
void p6logd(const char *fmt, ...) { (void)fmt; }
#define ADRMASK 0xFFFFFFu
#define ARENA (16u << 20)
#define RAMTOP 0x200000u
#define GV_LO 0xC00000u
#define GV_HI 0xC80000u
#define FLAG 0x18000u
#define ITER 0x18008u
#define NFR 0x1800Cu
#define FPTR 0x18010u
#define CB1 0x20000u
#define CB4 0x22000u
#define STREAM 0x30000u
#define CODE 0x10000u
#define STACK 0x8000u
#define GVBASE 0xC00000u
#define ROWBYTES 1024u
#define CPUHZ 10000000.0
static unsigned char *buf; /* byte-swapped, px68k convention */
/* Data bus cycles the 68000 issues. Every callback below is exactly one
* 68000 bus cycle -- C68K splits a long access into two word calls, which is
* what the 16-bit bus does too -- so counting calls counts bus cycles. This
* does NOT include instruction prefetch, which C68K reads straight through the
* fetch pointer with no callback; the count is therefore a LOWER BOUND on the
* CPU's bus occupancy, and the headroom it implies is an UPPER BOUND.
* It is still the measurement that matters for FINDINGS 29.6: if the decoder's
* data accesses alone left no room, a DMAC could not overlap with it at all. */
static long long bus_r, bus_w;
static int in_exec = 0;
/* Cycle capture. A single C68k_Exec slice runs the whole pass; the FLAG
* writes inside it record where the timed region starts and ends, so the
* count excludes nothing and includes no spin-loop tail. */
static long long slice;
static long long cyc_start = -1, cyc_stop = -1;
static int desync = 0;
static unsigned char rd8 (unsigned int a){ if (in_exec) bus_r++; return buf[(a & ADRMASK) ^ 1]; }
static unsigned short rd16(unsigned int a){ if (in_exec) bus_r++; a &= ADRMASK; return (unsigned short)(buf[a] | (buf[a+1] << 8)); }
static unsigned short peek16(unsigned int a){ a &= ADRMASK; return (unsigned short)(buf[a] | (buf[a+1] << 8)); }
static unsigned int rd32(unsigned int a){ return ((unsigned int)peek16(a) << 16) | peek16(a+2); }
static void wr8(unsigned int a, unsigned char d)
{
if (in_exec) bus_w++;
a &= ADRMASK;
if (a >= GV_LO && a < GV_HI) { if (a & 1) buf[a ^ 1] = d; return; } /* high byte discarded */
buf[a ^ 1] = d;
}
/* Only writes made BY the 68000 mean anything here. The harness sets FLAG
* itself during setup, and a `move.l` to FLAG arrives as two word writes, so
* the hook sees a half-updated long in between -- clearing FLAG from $FF to 0
* momentarily reads back as $FF again. Without in_exec that transient
* recorded a run's stop cycle before the run had started, and every frame
* after the first came out as the whole slice. */
static void note_flag(void)
{
unsigned int v = rd32(FLAG);
long long now = slice - C68K.ICount;
if (!in_exec) return;
if (v == 1 && cyc_start < 0) cyc_start = now;
else if (v == 0xFF || v == 0xEE) {
if (cyc_stop < 0) { cyc_stop = now; desync = (v == 0xEE); }
C68K.ICount = 0; /* stop the slice; we keep our own count */
}
}
static void wr16(unsigned int a, unsigned short d)
{
if (in_exec) bus_w++;
a &= ADRMASK;
if (a >= GV_LO && a < GV_HI) { buf[a] = (unsigned char)d; buf[a+1] = 0; return; }
buf[a] = (unsigned char)d; buf[a+1] = (unsigned char)(d >> 8);
if (a >= FLAG && a < FLAG + 4) note_flag();
}
static void wr32(unsigned int a, unsigned int d){ wr16(a, (unsigned short)(d >> 16)); wr16(a+2, (unsigned short)d); }
static void push(unsigned int a, const unsigned char *s, size_t n)
{
for (size_t i = 0; i < n; i++) wr8((unsigned int)(a + i), s[i]);
}
/* Prime the screen exactly as decode.lua's setup() does: active area at index
* 0, letterbox at the darkest palette entry. A SKIP block in frame 0 is a
* claim about THIS, so it is part of the decode contract. Pass 2 re-primes,
* because pass 1 left one frame's worth of residue on the screen and frame 0's
* SKIP blocks would otherwise inherit it. */
static void prime(unsigned int W, unsigned int H, unsigned int yoff, unsigned int dark)
{
for (unsigned int y = 0; y < 256; y++) {
unsigned short v = (y < yoff || y >= yoff + H) ? (unsigned short)dark : 0;
for (unsigned int x = 0; x < W; x++) wr16(GVBASE + y*ROWBYTES + x*2, v);
}
}
static unsigned char *slurp(const char *p, size_t *n)
{
FILE *f = fopen(p, "rb");
if (!f) { fprintf(stderr, "cannot open %s\n", p); exit(2); }
fseek(f, 0, SEEK_END); long L = ftell(f); fseek(f, 0, SEEK_SET);
unsigned char *b = malloc((size_t)L);
if (fread(b, 1, (size_t)L, f) != (size_t)L) { fprintf(stderr, "short read %s\n", p); exit(2); }
fclose(f); *n = (size_t)L; return b;
}
/* Run one pass and return its exact cycle count. */
static long long run(unsigned int off, unsigned int nfr, unsigned int iter)
{
cyc_start = cyc_stop = -1; desync = 0; bus_r = bus_w = 0;
wr32(FLAG, 0); wr32(ITER, iter); wr32(NFR, nfr); wr32(FPTR, STREAM + off);
C68k_Reset(&C68K);
C68k_Set_Reg(&C68K, C68K_SR, 0x2700); /* supervisor, all IRQs masked */
C68k_Set_Reg(&C68K, C68K_A7, STACK);
C68k_Set_Reg(&C68K, C68K_PC, CODE);
slice = 2000000000LL;
in_exec = 1;
C68k_Exec(&C68K, (INT32)slice);
in_exec = 0;
if (cyc_stop < 0) { fprintf(stderr, "TIMEOUT off=%u nfr=%u -- decoder never set FLAG\n", off, nfr); exit(4); }
if (desync) { fprintf(stderr, "BITSTREAM DESYNC off=%u nfr=%u\n", off, nfr); exit(5); }
/* A runaway is not a slow frame. Without this a bad record walk reports a
* two-billion-cycle "frame" as if it were a measurement. */
if (cyc_stop - cyc_start > 40LL * nfr * iter * 833333LL) {
fprintf(stderr, "RUNAWAY off=%u nfr=%u: %lld cyc (start=%lld stop=%lld) "
"PC=%06X FLAG=%08X SCR_N=%08X SCR_END=%08X len=%u\n",
off, nfr, cyc_stop - cyc_start, cyc_start, cyc_stop,
C68k_Get_Reg(&C68K, C68K_PC) & 0xFFFFFF, rd32(FLAG),
rd32(0x18014), rd32(0x18018), rd32(STREAM + off));
exit(6);
}
return cyc_stop - cyc_start;
}
int main(int argc, char **argv)
{
const char *fcode = "tmp/decode.bin", *fdata = "tmp/decode_data.bin", *dump = NULL;
unsigned int cb1_len=0, cb4_len=0, pal_len=0, stream_len=0, nframes=0, H=192, W=256, fps=12;
unsigned int dark = 255;
unsigned int anch[32]; int nanch = 0;
for (int i = 1; i < argc; i++) {
if (!strcmp(argv[i], "--code")) fcode = argv[++i];
else if (!strcmp(argv[i], "--data")) fdata = argv[++i];
else if (!strcmp(argv[i], "--dump")) dump = argv[++i];
else if (!strcmp(argv[i], "--cb1")) cb1_len = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--cb4")) cb4_len = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--pal")) pal_len = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--stream")) stream_len = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--nframes"))nframes = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--W")) W = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--H")) H = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--fps")) fps = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--dark")) dark = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--anchor")) { if (nanch < 32) anch[nanch++] = (unsigned)strtoul(argv[++i], NULL, 10); }
else { fprintf(stderr, "unknown arg %s\n", argv[i]); return 2; }
}
if (!nframes || !stream_len) { fprintf(stderr, "need --nframes and --stream (from decode_meta.lua)\n"); return 2; }
/* MAP_32BIT: C68K keeps its fetch base in a UINT32, so the arena must live
* below 4 GB or every opcode fetch reads a truncated pointer. */
buf = mmap(NULL, ARENA, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0);
if (buf == MAP_FAILED) { perror("mmap MAP_32BIT"); return 2; }
fprintf(stderr, "[C68K] arena at %p\n", (void *)buf);
size_t nc, nd;
unsigned char *code = slurp(fcode, &nc), *data = slurp(fdata, &nd);
size_t need = (size_t)cb1_len + cb4_len + pal_len + stream_len;
if (nd < need) { fprintf(stderr, "data blob %zu B < meta's %zu B\n", nd, need); return 2; }
size_t o = 0;
push(CB1, data + o, cb1_len); o += cb1_len;
push(CB4, data + o, cb4_len); o += cb4_len;
o += pal_len; /* palette: display only */
push(STREAM, data + o, stream_len);
push(CODE, code, nc);
/* Prime the screen exactly as decode.lua's setup() does: the active area
* starts at index 0 and the letterbox gets the darkest palette entry.
* A SKIP block in frame 0 is a claim about THIS, so it is part of the
* decode contract, not decoration. */
unsigned int yoff = (256u - H) / 2;
prime(W, H, yoff, dark);
C68k_Init(&C68K);
C68k_Set_ReadB (&C68K, rd8);
C68k_Set_ReadW (&C68K, rd16);
C68k_Set_WriteB(&C68K, wr8);
C68k_Set_WriteW(&C68K, wr16);
C68k_Set_Fetch (&C68K, 0x000000, 0xFFFFFF, (UINT32)(unsigned long)buf);
double frame_budget = CPUHZ / fps;
fprintf(stderr, "[C68K] %u frames, stream %u B, budget %.0f cyc/frame @ %u fps\n",
nframes, stream_len, frame_budget, fps);
/* Pass 1 -- every frame timed on its own. MAME could only afford eight
* anchor frames because its clock is 1/55.46 s; here each frame is exact,
* so the whole distribution comes out, which is what FINDINGS 31/35 score
* against. Record layout: [u32 len][768 mode][payload], next record start
* rounded up to 4 (FINDINGS 28.3). `len` counts the mode header TOO --
* decode.s sets SCR_END from the address AFTER the length word, so the
* record is 4 + len bytes, not 4 + 768 + len. */
printf("frame,offset,cycles,pct_of_frame,bus_reads,bus_writes,bus_pct\n");
unsigned int off = 0;
long long sum = 0, busr_tot = 0, busw_tot = 0;
for (unsigned int f = 0; f < nframes; f++) {
long long c = run(off, 1, 1);
sum += c;
long long br = bus_r, bw = bus_w;
busr_tot += br; busw_tot += bw;
printf("%u,%u,%lld,%.2f,%lld,%lld,%.2f\n", f, off, c,
100.0 * c / frame_budget, br, bw, 100.0 * 4.0 * (br + bw) / c);
unsigned int len = rd32(STREAM + off);
off = (off + 4 + len + 3) & ~3u;
}
fprintf(stderr, "[C68K] per-frame sum = %lld cyc, mean %.0f (%.1f%% of a %u fps frame)\n",
sum, (double)sum / nframes, 100.0 * sum / nframes / frame_budget, fps);
/* The number FINDINGS 29.6 needs. A 68000 bus cycle is 4 clocks, so a
* frame of `sum/nframes` clocks has room for a quarter that many bus
* cycles. What the decoder's DATA accesses do not use is the headroom a
* DMAC could paint spans in -- minus instruction prefetch, which is not
* counted here, so this OVERSTATES the headroom. */
{
double mean_cyc = (double)sum / nframes;
double slots = mean_cyc / 4.0;
double used = (double)(busr_tot + busw_tot) / nframes;
fprintf(stderr, "[C68K] data bus: %.0f reads + %.0f writes = %.0f cycles/frame "
"of %.0f slots = %.1f%% occupied\n",
(double)busr_tot / nframes, (double)busw_tot / nframes, used, slots,
100.0 * used / slots);
fprintf(stderr, "[C68K] headroom >= %.0f bus cycles/frame "
"(%.1f%%), MINUS instruction prefetch, which is not counted\n",
slots - used, 100.0 * (slots - used) / slots);
}
/* Pass 2 -- one sequential run of the whole window. Two jobs: it is the
* only honest correctness test (SKIP makes every frame a claim about the
* one before it), and its total against pass 1's sum prices the outer
* frame-loop overhead the per-frame runs each pay once. */
prime(W, H, yoff, dark);
long long seq = run(0, nframes, 1);
fprintf(stderr, "[C68K] sequential pass = %lld cyc, mean %.0f (%.1f%%); "
"per-frame sum is %+.3f%% of it\n",
seq, (double)seq / nframes, 100.0 * seq / nframes / frame_budget,
100.0 * (sum - seq) / seq);
/* Dump BEFORE the anchors run. They decode single frames onto this same
* screen, so anything after them is not the sequential reconstruction and
* verify_c68k.py would report every pixel wrong. */
if (dump) {
/* Active area only, one byte per pixel -- the low byte of each GVRAM
* word, which is all the hardware keeps. */
FILE *g = fopen(dump, "wb");
if (!g) { perror(dump); return 2; }
for (unsigned int y = 0; y < H; y++)
for (unsigned int x = 0; x < W; x++) {
unsigned char p = (unsigned char)rd16(GVBASE + (yoff + y)*ROWBYTES + x*2);
fwrite(&p, 1, 1, g);
}
fclose(g);
fprintf(stderr, "[C68K] screen dumped to %s (%ux%u indices)\n", dump, W, H);
}
/* Pass 3 -- decode.lua's timing anchors, at the same stream offsets, so the
* two instruments are quoted on the same eight frames. The four synthetic
* single-mode frames live past the end of the real stream and so are not
* reachable by the record walk in pass 1; they are the ones that price the
* modes separately (prep_dlx.py), which is where two cycle tables are most
* likely to disagree. */
for (int i = 0; i < nanch; i++) {
long long c = run(anch[i], 1, 1);
fprintf(stderr, "[C68K] anchor off=%-8u %8lld cyc %5.1f%% of a %u fps frame\n",
anch[i], c, 100.0 * c / frame_budget, fps);
}
return 0;
}
tools/bench/c68k/run.sh
Executable
+21
View File
@@ -0,0 +1,21 @@
#!/bin/bash
# Run the C68K harness against whatever tools/bench/prep_dlx.py last laid out,
# so it measures byte-for-byte the same code and container MAME did.
# tools/bench/c68k/run.sh [out.csv]
set -e
cd "$(dirname "$0")/../../.."
M=tmp/decode_meta.lua
[ -f "$M" ] || { echo "no $M -- run tools/bench/prep_dlx.py first"; exit 2; }
g() { sed -n "s/.*[ ,{]$1=\([0-9]*\).*/\1/p" "$M" | head -1; }
# Same anchor offsets decode.lua times, so the two instruments are quoted on the
# same frames -- including the four synthetic single-mode ones, which sit past
# the end of the real stream and price each block mode on its own.
ANCH=()
while read -r o; do ANCH+=(--anchor "$o"); done < <(sed -n 's/.*off=\([0-9]*\).*/\1/p' "$M")
tools/bench/c68k/c68k_bench \
--code tmp/decode.bin --data tmp/decode_data.bin \
--cb1 "$(g cb1_len)" --cb4 "$(g cb4_len)" --pal "$(g pal_len)" \
--stream "$(g stream_len)" --nframes "$(g nframes)" \
--W "$(g W)" --H "$(g H)" --fps "$(g fps)" --dark "$(g dark)" \
"${ANCH[@]}" \
--dump tmp/c68k_screen.bin > "${1:-tmp/c68k_frames.csv}" 2> >(tee tmp/c68k.log >&2)
tools/bench/c68k/verify_c68k.py
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""Is the C68K harness's decode pixel-exact against the reference decoder?
python3 tools/bench/c68k/verify_c68k.py <in.dlx> --nframes N
This is the licence for every cycle number the harness prints. The harness
rebuilds px68k's memory model from scratch -- byte-swapped RAM, GVRAM word
writes that discard the high byte, a hand-rolled 24-bit map -- and any of that
being subtly wrong would still produce plausible-looking cycle counts. It could
not produce a pixel-exact 80-frame temporal recursion.
Unlike tools/bench/verify_decode.py this compares palette INDICES, not rendered
RGB: the harness dumps the low byte of each GVRAM word directly, so there is no
palette round-trip to model and no snapshot geometry to unpick.
"""
import argparse, sys
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--dump", default="tmp/c68k_screen.bin")
ap.add_argument("--nframes", type=int, default=None)
a = ap.parse_args()
d = DLX(a.container)
NF = a.nframes if a.nframes is not None else d.nframes
if NF > d.nframes:
sys.exit(f"--nframes {NF} exceeds the container's {d.nframes}")
canvas = np.zeros((d.H, d.W), np.uint8)
for f in range(NF):
d.paint(canvas, f)
got = np.fromfile(a.dump, np.uint8)
if got.size != d.H * d.W:
sys.exit(f"FAIL 1. dump is {got.size} B, expected {d.H*d.W}")
got = got.reshape(d.H, d.W)
if not np.array_equal(got, canvas):
bad = got != canvas
by, bx = np.where(bad)
blocks = sorted(set(zip((by // 4).tolist(), (bx // 4).tolist())))
sys.exit(f"FAIL 2. frame {NF-1} not pixel-exact under C68K: {bad.sum()} px in "
f"{len(blocks)} blocks differ; first block "
f"(by={blocks[0][0]}, bx={blocks[0][1]})")
print(f"OK {NF} frames decoded on px68k's C68K core, final frame pixel-exact "
f"against tools/encoder/dlx.py")
print(f" {d.W}x{d.H}, {d.nb} blocks/frame, k1={d.k1} k4={d.k4}; the memory "
f"model (byte-swapped RAM, high-byte-discarding GVRAM) is therefore right")
tools/bench/check.sh
+209 -9
View File
@@ -4,12 +4,37 @@
# exit means something drifted.
set -e
cd "$(dirname "$0")/../.."
[ -d /media/reala-misaki/BDROM ] || {
echo "Blu-ray not mounted. udisksctl loop-setup -r -f DRAGONS_LAIR.iso"; exit 2; }
# No media ships with this repo. Bring your own disc; DLX_BDROM overrides the
# mount point, and every tool that reads the disc honours the same variable.
DLX_BDROM=${DLX_BDROM:-/media/${USER:-$(id -un)}/BDROM}
export DLX_BDROM
[ -d "$DLX_BDROM" ] || {
echo "Blu-ray not mounted at $DLX_BDROM."
echo " udisksctl loop-setup -r -f DRAGONS_LAIR.iso"
echo " or set DLX_BDROM to where yours is mounted."; exit 2; }
python3 tools/encoder/extract.py 00020 tmp/fr_00020 12 crop
mkdir -p tmp/snap_verify tmp/snap256
# ---------------------------------------------------------------------------
# THE ONE PLACE THE RETIRED PIPE FIGURE STILL LIVES. Session 18 removed it as
# a default from every analysis tool and from tools/bench/stream.lua, because it
# was never a bus measurement -- a user-supplied "4 Mbps" with no provenance,
# 10% of SCSI-1's asynchronous rating (FINDINGS 42.1) -- and a default let table
# after table be scored against it without anyone restating what it was.
#
# It survives HERE and only here because the gate container was ENCODED with it,
# and every per-block and span constant in FINDINGS 41/43/45/49 is fitted to that
# container. Changing this number is not an edit, it is a re-encode plus a
# re-measurement of all of them.
#
# It is a CONTAINER RECIPE, not a claim about any medium. Do not read a delivery
# rate out of it, do not copy it into a tool, and do not add a default anywhere
# that would resurrect it. When the pipe is finally measured, this becomes an
# ordinary encoder setting and the comment goes.
GATE_SPAN_KBPS=488
# ---------------------------------------------------------------------------
run() { # run <script> <snapdir>
rm -f "tmp/$2/x68000"/*.png
( cd tmp && SDL_VIDEODRIVER=dummy timeout -k 5 120 mame x68000 -bios ipl10 \
@@ -40,6 +65,19 @@ python3 tools/analysis/09_ratectl_drift.py > tmp/drift_check.log 2>&1 \
|| { cat tmp/drift_check.log; exit 1; }
tail -9 tmp/drift_check.log
echo "--- session 12: the DLX3 span container round-trips (FINDINGS 41) ---"
# 09 above replays SKIP semantics in Python and never reads a container. A v7
# span breaks exactly that shortcut -- a spanned block reads SKIP in the mode
# header and is painted by the span section instead -- so this encodes, WRITES
# the container, reads it back with the reference decoder and compares. It also
# asserts that it emitted enough spans to have tested anything.
# --kbps is required now (session 18): the tool has no default rate, so the gate
# has to say which one it is testing at. Same recipe constant as the container.
python3 tools/analysis/16_span_roundtrip.py --kbps $GATE_SPAN_KBPS \
> tmp/span_roundtrip.log 2>&1 \
|| { cat tmp/span_roundtrip.log; exit 1; }
tail -4 tmp/span_roundtrip.log
echo "--- session 7: display-path coherency (FINDINGS 28.1) ---"
# 10_pathmix_drift.py is a COUNTEREXAMPLE, kept runnable: the dual-path plan of
# FINDINGS 24.5/25.6 must still be shown to corrupt frames, and the strategy the
@@ -57,17 +95,179 @@ echo "--- session 7: 68000 decoder is pixel-exact (FINDINGS 28) ---"
# 68000 code, every block mode, full temporal recursion. A SKIP block is a claim
# about the previous frame still being on screen, so the last frame is only
# right if all 120 were.
DLX=tmp/rc_fr_singe_sasi_rcprofile.dlx
[ -f "$DLX" ] || python3 tools/encoder/encode.py tmp/fr_singe "$DLX" --profile sasi
python3 tools/bench/prep_dlx.py "$DLX" > tmp/prep_dlx.log
# The gate container is the HEAVIEST stream the encoder emits: the scsi mode
# decision (the only profile left after session 9 dropped sasi on capacity,
# FINDINGS 32) with the span pass drawing on a byte ceiling wide enough that
# every frame carries a span table and all four block modes are still exercised.
# That ceiling is GATE_SPAN_KBPS above -- a recipe, not a delivery rate.
# Spans are the newest and least-proven path in decode.s; gating on a container
# where they are rare would be gating on the old decoder. FINDINGS 41.
DLX=tmp/rc_fr_singe_scsi_span.dlx
[ -f "$DLX" ] || python3 tools/encoder/encode.py tmp/fr_singe "$DLX" --profile scsi \
--kbps 280 --span-kbps $GATE_SPAN_KBPS --spans all
# RIG_RAM is the EMULATED MACHINE's memory, and it is not a claim about the
# target. The rig preloads the whole container into RAM at 0x30000; the shipping
# player streams from disk into a ring buffer and never holds a window at once,
# so preloading is unlike the player at ANY size. At the 2 MB of a stock machine
# this gate covered 37 of 120 frames (FINDINGS 44.6.4) -- the span-heavy
# container is 5,261,814 B of stream, ending at 0x534BF6. 6 MB covers all 120.
#
# Raising it is licensed by measurement, not by convenience: at 2M and 6M the
# five synthetic anchors come out BIT-IDENTICAL (40,729 / 921,187 / 1,376,881 /
# 1,229,883 / 506,533 cycles) despite sitting at different addresses in the two
# layouts, so MAME's cycle model does not depend on ramsize over this range.
# FINDINGS 45. What is still NOT tested, at either size, is the streaming path.
RIG_RAM=${RIG_RAM:-6}
python3 tools/bench/prep_dlx.py "$DLX" --ram $((RIG_RAM * 0x100000)) > tmp/prep_dlx.log
# Verify against exactly the frame list prep_dlx emitted. It no longer truncates
# at the default RIG_RAM, but the guard stays: lower RIG_RAM, or a heavier
# container, brings truncation straight back and it must stay announced.
NF=$(sed -n 's/.*nframes=\([0-9]*\),.*/\1/p' tmp/decode_meta.lua)
grep -a "TRUNCATED" tmp/prep_dlx.log || true
tools/vasm/vasmm68k_mot -Fbin -o tmp/decode.bin src/player/decode.s > /dev/null
mkdir -p tmp/snap_decode
rm -f tmp/snap_decode/x68000/*.png
( cd tmp && DLX_VERIFY_ONLY=1 SDL_VIDEODRIVER=dummy timeout -k 5 300 mame x68000 \
-bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
# stdbuf -oL: a FILE is block-buffered too, so without it a long MAME run is
# unobservable until it exits and a run that is merely finishing looks exactly
# like one that is wedged (FINDINGS 34.1).
# -seconds_to_run must cover the WHOLE sequential pass. The scsi container is
# 2.7x the payload of the session-7 one this gate used to run on, and at 20 s
# the pass was truncated -- MAME exited mid-decode and verify_decode.py then
# compared a partially drawn screen and reported 49,005 differing pixels, which
# reads as a decoder bug and is not one.
( cd tmp && DLX_VERIFY_ONLY=1 SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 300 mame x68000 \
-bios ipl10 -ramsize ${RIG_RAM}M -video soft -window -sound none -nothrottle -plugins \
-autoboot_script ../tools/bench/decode.lua \
-snapshot_directory ./snap_decode -snapview native -seconds_to_run 20 \
-snapshot_directory ./snap_decode -snapview native -seconds_to_run 60 \
> decode_check.log 2>&1 )
python3 tools/bench/verify_decode.py "$DLX"
# A truncated run must fail as a truncated run. Without this the only symptom is
# a pixel diff against a half-drawn frame.
grep -q "snapshot taken" tmp/decode_check.log || {
echo "FAIL: the 68000 sequential pass did not complete -- no snapshot marker."
echo " Raise -seconds_to_run; the pass needs the whole container decoded."
tail -5 tmp/decode_check.log; exit 1; }
python3 tools/bench/verify_decode.py "$DLX" --nframes "$NF"
echo "--- session 10: the same decode on a second CPU core (FINDINGS 37) ---"
# A SECOND emulator, and the cheapest strong test in the tree: seconds, no MAME,
# no ROMs. px68k's C68K core has its own cycle table and its own memory model,
# so a pass here says decode.s is pixel-exact under two independent cores and
# that the harness's byte-swapped RAM / high-byte-discarding GVRAM is right --
# which is what licenses its cycle and bus numbers.
# Skipped rather than failed when px68k is not checked out: it is an external
# tree, not part of this repo.
PX68K=${PX68K:-$HOME/src/px68k}
if [ -f "$PX68K/m68000/c68k.c" ]; then
make -s -C tools/bench/c68k PX68K="$PX68K"
bash tools/bench/c68k/run.sh tmp/c68k_frames.csv 2>tmp/c68k.log
grep -a "sequential pass" tmp/c68k.log
python3 tools/bench/c68k/verify_c68k.py "$DLX" --nframes "$NF"
echo "--- session 10: the bus model still matches the machine (FINDINGS 38) ---"
# 15_bus_occupancy.py derives instruction prefetch, which no emulator here can
# report, and validates itself against the DATA accesses the harness counts.
# If that check ever stops holding, every bus figure in FINDINGS 38/39 is
# unfounded -- so it is a gate, not a report.
python3 tools/analysis/15_bus_occupancy.py "$DLX" | sed -n '3,7p'
else
echo " SKIPPED: no px68k at $PX68K (set PX68K= to point at a checkout)"
fi
echo "--- session 20: the DMAC config, read out of the IPL ROM (FINDINGS 52) ---"
# The audio and disk per-byte debits are no longer a recollection about the
# HD63450: they are bytes at named addresses in the ROM MAME boots this rig
# with. This gate re-reads them. It is cheap, it needs no emulator, and if a
# different ROM revision is ever pointed at it, it says so rather than decoding
# some other code and reporting a number.
# Skipped rather than failed when the ROM is not where MAME keeps it: that is a
# path outside this repo.
IPLROM=${IPLROM:-$HOME/mame/roms/iplrom.dat}
if [ -f "$IPLROM" ]; then
python3 tools/analysis/21_iplrom_dmac.py "$IPLROM" > tmp/iplrom_dmac.log 2>&1 \
|| { cat tmp/iplrom_dmac.log; exit 1; }
grep -ac "^ OK " tmp/iplrom_dmac.log | xargs printf " %s evidence sites hold; "
sed -n 's/^ = \(.*clocks per audio byte\)/audio is \1/p' tmp/iplrom_dmac.log
else
echo " SKIPPED: no IPL ROM at $IPLROM (set IPLROM= to point at it)"
fi
echo "--- session 18: the shared-body split is a no-op (FINDINGS 49.7.5) ---"
# src/player/decode.s and src/player/stream.s assemble from ONE copy of the block
# loop and the span chain (src/player/frame.i) so that the two front-ends cannot
# drift apart. The drift would be silent -- both would still decode correctly,
# and only the cost model would be wrong, because the 66.0 clocks/span, 9.143
# clocks/coarse pixel and every per-block constant in FINDINGS 24/30/40/41 are
# fitted to those exact bytes. So the split is asserted to be a no-op rather than
# assumed to be one.
DECODE_MD5=7a7a06f8c6d097ee0041bca4aefa3eb2 # decode.bin before the split, 1296 B
GOT=$(md5sum tmp/decode.bin | cut -d" " -f1)
[ "$GOT" = "$DECODE_MD5" ] || {
echo "FAIL: decode.bin is $GOT, expected $DECODE_MD5 ($(stat -c%s tmp/decode.bin) B)."
echo " The block loop or the span chain changed. That is allowed -- but"
echo " every cycle constant in FINDINGS 24/30/40/41 is fitted to the old"
echo " bytes, so re-measure them and move this hash, do not just move it."
exit 1; }
echo " decode.bin unchanged at $(stat -c%s tmp/decode.bin) B ($DECODE_MD5)"
# Same argument for the loader maths, which prep_dlx.py and prep_stream.py now
# share via tools/bench/dlxload.py: a second copy of the palette packing would
# drift and the symptom would be wrong colours in one rig only.
python3 tools/bench/prep_dlx.py "$DLX" --ram $((RIG_RAM * 0x100000)) --out tmp/_pdchk > /dev/null
cmp -s tmp/_pdchk_data.bin tmp/decode_data.bin || {
echo "FAIL: prep_dlx.py is not reproducible"; exit 1; }
echo " prep_dlx.py blob reproducible ($(stat -c%s tmp/decode_data.bin) B)"
rm -f tmp/_pdchk_data.bin tmp/_pdchk_meta.lua
echo "--- session 18: 120 frames through a bounded RING (FINDINGS 49) ---"
# The gate above preloads the whole container into RAM and proves the DECODER.
# This proves the DELIVERY path: the same 120 frames decoded out of a 256 KB
# ring on a STOCK 2 MB machine, with the container in a host file. The block
# loop reads with a monotonically increasing a0 and no bounds check, so a record
# placed wrongly by the wrap policy corrupts pixels rather than faulting -- which
# is why this is gated on the same pixel-exact comparison and not on a checksum.
tools/vasm/vasmm68k_mot -Fbin -o tmp/stream.bin src/player/stream.s > /dev/null
python3 tools/bench/prep_stream.py "$DLX" > tmp/prep_stream.log
mkdir -p tmp/snap_stream
rm -f tmp/snap_stream/x68000/*.png
( cd tmp && DLX_STREAM_KBPS=0 SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 600 \
mame x68000 -bios ipl10 -ramsize 2M -video soft -window -sound none \
-nothrottle -plugins -autoboot_script ../tools/bench/stream.lua \
-snapshot_directory ./snap_stream -snapview native -seconds_to_run 90 \
> stream_check.log 2>&1 )
# Same truncation trap as the decode stage: without this, a run that exited
# mid-decode is compared against a half-drawn screen and reads as a wrap bug.
grep -q "snapshot taken" tmp/stream_check.log || {
echo "FAIL: the ring-buffer pass did not complete -- no snapshot marker."
tail -5 tmp/stream_check.log; exit 1; }
grep -a "ring: \|DEADLINE" tmp/stream_check.log | sed "s/\[STR\] / /"
python3 tools/bench/verify_decode.py "$DLX" --snap tmp/snap_stream
echo "--- session 19: the PACED ring, and what a branch point costs (FINDINGS 51) ---"
# The stage above runs the ring FREE-RUNNING, which is right for what it gates:
# an unlimited pipe removes delivery as a variable and leaves the wrap policy
# alone under test. It cannot see buffering, because a decoder that never waits
# never lets the ring back up -- 49.7.2, and it is why 48 KB passed while
# holding one record. This runs the same 120 frames with the decoder held to
# 12 fps, which is the only configuration in which FR_HEAD-FR_TAIL means what
# it is read to mean.
#
# Gated on: pixel-exact, zero UNDERRUNS, and a ceiling that has not moved. The
# ceiling is a property of THIS container in a 256 KB ring; it is asserted
# rather than printed because a change in it is a change in how much a branch
# point can afford, and that should not slip through as a line in a log.
bash tools/bench/pace_run.sh 256 0 > tmp/pace_check.log 2>&1 || {
echo "FAIL: the paced ring pass did not complete."; tail -8 tmp/pace_check.log
exit 1; }
grep -aE "SEEK SLACK|UNDERRUNS" tmp/pace_check.log
grep -q "UNDERRUNS: 0/120" tmp/pace_check.log || {
echo "FAIL: the paced decoder underran -- a frame's slot arrived before its"
echo " record did. Free-running this is earliness (49.6); paced it is not."
exit 1; }
grep -q "ceiling 8 frames" tmp/pace_check.log || {
echo "FAIL: the 256 KB seek-slack ceiling is no longer 8 frames (FINDINGS 51)."
echo " Re-run tools/bench/pace_sweep.sh and re-derive 51 before editing"
echo " this number -- it is what a branch point can spend."
exit 1; }
grep -q "^OK" tmp/pace_check.log || { echo "FAIL: paced pass not pixel-exact";
tail -4 tmp/pace_check.log; exit 1; }
echo "ALL GREEN"
tools/bench/decode.lua
+5
View File
@@ -157,6 +157,11 @@ SUB = emu.add_machine_frame_notifier(function()
M.video:snapshot()
P("snapshot taken after the sequential pass -- last frame, 68000-decoded")
step = step + 1
-- DLX_VERIFY_ONLY leaves nothing after the correctness pass, and this
-- used to walk off the end of PLAN and raise a Lua error AFTER the
-- snapshot was already on disk -- harmless to check.sh, and exactly the
-- kind of thing that gets mistaken for a decoder failure later.
if not PLAN[step] then st = "finish"; return end
launch(PLAN[step].off, PLAN[step].nfr, PLAN[step].iter)
st, t0 = "running", nil; return
end
tools/bench/dlxload.py
+53
View File
@@ -0,0 +1,53 @@
"""Load-time transforms every src/player/ front-end's loader has to do.
Split out of prep_dlx.py in session 18 so that prep_dlx.py (the preloaded-stream
rig) and prep_stream.py (the ring-buffer streaming rig, FINDINGS 49) share ONE
copy of them. Two copies would drift, and the drift would be silent: both rigs
would still decode, and only the colours or the codebook scaling would be
subtly wrong in one of them.
The split is a no-op by construction -- tools/bench/check.sh asserts prep_dlx.py
still emits a byte-identical blob for the gate container.
Neither transform is part of the per-frame cost being measured. The 68000 would
do both once at load time; charging them to the inner loop would flatter or damn
it for no reason.
"""
import numpy as np
def expand_codebooks(d):
"""CB1/CB4 to one WORD per pixel, so the inner loop movems them straight out.
The high byte of every GVRAM word write is discarded by the hardware, so it
is left zero and never has to be cleared. Word-per-pixel form is also what
makes index scaling a shift rather than a multiply: lsl.w #5 and lsl.w #3.
"""
cb1 = np.zeros((d.k1, 16, 2), np.uint8); cb1[:, :, 1] = d.cb1.reshape(d.k1, 16)
cb4 = np.zeros((d.k4, 4, 2), np.uint8); cb4[:, :, 1] = d.cb4.reshape(d.k4, 4)
return cb1, cb4
def pack_palette(d):
"""24-bit palette -> GGGGGRRRRRBBBBBI, shared LSB chosen PER ENTRY.
Choosing I per entry by minimum squared error rather than fixing it is worth
1.96 dB (FINDINGS 23.3). Identical maths to tools/bench/verify_frame256.py,
which is the point: the verifier and the loader must agree or a colour bug
reads as a decoder bug.
Returns (palette bytes 256x2 big-endian, index of the darkest entry). The
encoder does not yet reserve a black entry (docs/STATUS.md, encoder gaps),
so the letterbox gets the closest thing to black the palette has.
"""
pal = d.pal.astype(int)
p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
f = pal >> 3
render = lambda I: p6((f << 1) | I[:, None])
I = (((render(np.ones(256, int)) - pal) ** 2).sum(1)
< ((render(np.zeros(256, int)) - pal) ** 2).sum(1)).astype(int)
words = (f[:, 1] << 11) | (f[:, 0] << 6) | (f[:, 2] << 1) | I
palb = np.zeros((256, 2), np.uint8)
palb[:, 0], palb[:, 1] = words >> 8, words & 0xFF
dark = int(((render(I).astype(int)) ** 2).sum(1).argmin())
return palb, dark, render(I)
tools/bench/gvpack/Makefile
+9
View File
@@ -0,0 +1,9 @@
PX68K ?= $(HOME)/src/px68k
CFLAGS = -O2 -fno-strict-aliasing -Wall -Wno-unused-result \
-I$(PX68K)/m68000 -I$(PX68K)/x11 -I$(PX68K)/win32api -I$(PX68K)/x68k
gvpack: harness.c $(PX68K)/x68k/gvram.c
$(CC) $(CFLAGS) -o $@ harness.c $(PX68K)/x68k/gvram.c
clean:
rm -f gvpack
tools/bench/gvpack/gvpack
Executable
BIN
View File
Binary file not shown.
tools/bench/gvpack/harness.c
+152
View File
@@ -0,0 +1,152 @@
/* Headless harness for px68k's GVRAM write and display model.
*
* Tests FINDINGS 46.6 -- the packed 1.0 byte/pixel layout -- on a SECOND
* emulator, the way tools/bench/c68k does for the CPU core. It links px68k's
* real x68k/gvram.c: the address decode, the CRTC R20 bit-11 buffer-mode write
* path, the page-byte selection, the scroll wrap and the index-0 transparency
* test are all px68k's own code, not a reimplementation.
*
* What IS glue here, and is declared as such: the ~12 lines of page-ordering
* from x11/windraw.c's 256-colour case (which page is drawn opaque and which
* transparent, as a function of the video controller's priority register).
* windraw.c is SDL-bound and cannot be linked headless, so that dispatch is
* mirrored. It is quoted verbatim in pick_order() so the mirroring is
* auditable.
*
* GrphPal is set to the IDENTITY, so what lands in Grp_LineBuf is the 8-bit
* palette INDEX rather than a host pixel. That keeps the harness out of
* px68k's host-format colour conversion, and it is faithful: px68k's
* transparency test is on the index (`if (v != 0x00)`), before the lookup.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "common.h"
#include "gvram.h"
/* --- the globals gvram.c expects from the rest of px68k -------------------- */
BYTE CRTC_Regs[48];
WORD CRTC_FastClrMask;
DWORD GrphScrollX[4], GrphScrollY[4];
WORD GrphPal[256];
BYTE TextDirtyLine[1024];
DWORD TextDotX, TextDotY;
DWORD VLINE;
BYTE Pal_Regs[1024];
WORD Pal16[65536];
WORD Ibit, Pal_HalfMask, Pal_Ix2;
extern BYTE GVRAM[0x80000];
extern WORD Grp_LineBuf[1024];
#define W 256
#define H 256
/* 68000 word write: two byte writes, high byte first, as the bus does. */
static void wr16(DWORD adr, WORD v)
{
GVRAM_Write(adr, (BYTE)(v >> 8));
GVRAM_Write(adr + 1, (BYTE)(v & 0xff));
}
static void set_r20(WORD r20) /* CRTC R20 = byte pair 0x28/0x29 */
{
CRTC_Regs[0x28] = (BYTE)(r20 >> 8);
CRTC_Regs[0x29] = (BYTE)(r20 & 0xff);
}
/* Mirrors x11/windraw.c, 256-colour case:
*
* if ( (VCReg1[1]&3) <= ((VCReg1[1]>>4)&3) ) {
* ... Grp_DrawLine8(1, 1); opaq = 0;
* ... Grp_DrawLine8(0, opaq);
* } else {
* ... Grp_DrawLine8(0, 1); opaq = 0;
* ... Grp_DrawLine8(1, opaq);
* }
*
* i.e. the first page drawn is OPAQUE (the bottom) and the second is drawn
* with opaq=0 (the transparent top).
*/
static void draw_line(BYTE vcreg1_lo)
{
int bottom = ((vcreg1_lo & 3) <= ((vcreg1_lo >> 4) & 3)) ? 1 : 0;
Grp_DrawLine8(bottom, 1);
Grp_DrawLine8(bottom ^ 1, 0);
}
int main(int argc, char **argv)
{
const char *blob = argc > 1 ? argv[1] : "tmp/frame256p.bin";
const char *out = argc > 2 ? argv[2] : "tmp/gvpack_px68k.raw";
int packed = !(argc > 3 && !strcmp(argv[3], "--unpacked"));
BYTE vc1 = (BYTE)(argc > 4 ? strtol(argv[4], NULL, 0) : 0x02);
/* --nobuffer: run the packed layout WITHOUT CRTC R20 bit 11, to show the
* bit is load-bearing here and not decoration. */
int buffer = !(argc > 5 && !strcmp(argv[5], "--nobuffer"));
int scroll = !(argc > 5 && !strcmp(argv[5], "--noscroll"));
/* --keepbuffer: leave R20 bit 11 SET while drawing. MAME blanks the
* graphics layer in that state; does px68k? */
int keepbuf = (argc > 5 && !strcmp(argv[5], "--keepbuffer"));
FILE *f = fopen(blob, "rb");
if (!f) { perror(blob); return 2; }
static BYTE d[8 + 768 + 256 * 256];
size_t n = fread(d, 1, sizeof d, f);
fclose(f);
int iw = (d[4] << 8) | d[5], ih = (d[6] << 8) | d[7];
if (n < (size_t)(8 + 768 + iw * ih)) { fprintf(stderr, "short blob\n"); return 2; }
const BYTE *pix = d + 8 + 768;
int yoff = (H - ih) / 2;
const BYTE BLACK = 255;
#define PIX(y, x) ((y) < yoff || (y) >= yoff + ih ? BLACK : pix[((y) - yoff) * iw + (x)])
memset(GVRAM, 0, sizeof GVRAM);
for (int i = 0; i < 256; i++) GrphPal[i] = (WORD)i; /* identity */
TextDotX = W; TextDotY = H;
/* 256x256, 256 colours -- the same R20 tools/bench/crtc_mode.lua applies */
const WORD R20_DISPLAY = 0x0110;
set_r20(R20_DISPLAY);
/* page 0 -> scroll sets 0,1; page 1 -> scroll sets 2,3 */
GrphScrollX[0] = GrphScrollX[1] = 0;
GrphScrollY[0] = GrphScrollY[1] = 0;
GrphScrollX[2] = GrphScrollX[3] = (packed && scroll) ? 384 : 0;
GrphScrollY[2] = GrphScrollY[3] = 0;
if (packed) {
if (buffer) set_r20(R20_DISPLAY | 0x0800); /* buffer mode: unmasked */
for (int y = 0; y < H; y++) {
DWORD base = 0xC00000 + y * 1024;
for (int i = 128; i < 512; i++) wr16(base + i * 2, 0);
for (int i = 0; i < 128; i++)
wr16(base + i * 2, (WORD)((PIX(y, i + 128) << 8) | PIX(y, i)));
}
if (!keepbuf) set_r20(R20_DISPLAY); /* back to display */
} else {
/* the ordinary 2.0 B/pixel path, for a control */
for (int y = 0; y < H; y++) {
DWORD base = 0xC00000 + y * 1024;
for (int x = 0; x < W; x++) wr16(base + x * 2, PIX(y, x));
}
}
FILE *o = fopen(out, "wb");
if (!o) { perror(out); return 2; }
for (int y = 0; y < H; y++) {
VLINE = (DWORD)y;
memset(Grp_LineBuf, 0, sizeof Grp_LineBuf);
draw_line(vc1);
static BYTE row[W];
for (int x = 0; x < W; x++) row[x] = (BYTE)(Grp_LineBuf[x] & 0xff);
fwrite(row, 1, W, o);
}
fclose(o);
fprintf(stderr, "[GVPACK] px68k model: %s, vcreg1=%02X, bottom page=%d -> %s\n",
packed ? (buffer ? "PACKED 1.0 B/px" : "PACKED but bit11 OFF")
: "unpacked 2.0 B/px", vc1,
((vc1 & 3) <= ((vc1 >> 4) & 3)) ? 1 : 0, out);
return 0;
}
tools/bench/gvpack/verify_gvpack.py
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""Check px68k's render of the packed layout against the same reference MAME is
judged on (tools/bench/verify_frame256.py, criterion 3 and 4).
python3 tools/bench/gvpack/verify_gvpack.py <raw> [blob]
<raw> is 256x256 palette INDICES straight out of px68k's Grp_DrawLine8.
"""
import struct, sys
import numpy as np
raw = sys.argv[1] if len(sys.argv) > 1 else "tmp/gvpack_px68k.raw"
blob = sys.argv[2] if len(sys.argv) > 2 else "tmp/frame256p.bin"
g = np.frombuffer(open(raw, "rb").read(), np.uint8).reshape(256, 256)
d = open(blob, "rb").read()
W, H = struct.unpack(">HH", d[4:8])
pal = np.frombuffer(d[8:8+768], np.uint8).reshape(256, 3).astype(int)
idx = np.frombuffer(d[8+768:8+768+W*H], np.uint8).reshape(H, W)
yoff = (256 - H) // 2
act = g[yoff:yoff+H]
fail = []
if not np.array_equal(act, idx):
bad = act != idx
fail.append(f"active area index-exact: {bad.sum()} px differ "
f"(left half {bad[:, :128].sum()}, right half {bad[:, 128:].sum()})")
bars = np.concatenate([g[:yoff], g[yoff+H:]])
if bars.size and (bars != 255).any():
fail.append(f"letterbox not the reserved black index 255: "
f"{(bars != 255).sum()} px")
if (act == 0).any():
fail.append(f"index 0 appeared in the picture: {(act == 0).sum()} px "
f"-- it is the transparency key and must stay unused")
for x in fail:
print("FAIL " + x)
if fail:
sys.exit(1)
p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
f = pal >> 3
render = lambda I: p6((f << 1) | I[:, None])
I = (((render(np.ones(256, int)) - pal) ** 2).sum(1)
< ((render(np.zeros(256, int)) - pal) ** 2).sum(1)).astype(int)
mse = ((render(I)[act].astype(int) - pal[idx]) ** 2).mean()
print(f"OK {raw}: px68k renders the packed layout index-exact over {W}x{H}, "
f"letterbox on the reserved black")
print(f" palette ceiling vs 24-bit palettised source: "
f"{10*np.log10(255**2/mse):.2f} dB")
tools/bench/pace_run.sh
Executable
+47
View File
@@ -0,0 +1,47 @@
#!/bin/bash
# One paced ring-buffer run (STATUS item 4, FINDINGS 49.7.2).
#
# tools/bench/check.sh runs the ring pass FREE-RUNNING, which is right for what
# it gates -- wrap correctness at a fixed ring size, delivery removed as a
# variable by an unlimited pipe. It cannot answer the buffering question,
# because a free-running decoder never lets the ring back up.
#
# This runs the same rig with the decoder held to the container's frame rate,
# so the ring fills and FR_HEAD-FR_TAIL means "frames the decoder could still
# draw with the pipe dead". Every run is verified PIXEL-EXACT: a paced decode
# that drops a pixel is not a slack measurement, it is a bug.
#
# tools/bench/pace_run.sh <ring_kb> <kbps> [cut_at_tick] [cut_frames]
#
# kbps 0 = unlimited pipe. There is no default rate anywhere in this tree
# (FINDINGS 50) and there is none here either.
set -e
cd "$(dirname "$0")/../.."
RING=${1:?ring KB}; KBPS=${2:?pipe KB/s, or 0 for unlimited}
CUT_AT=$3; CUT_FR=${4:-1}
DLX=${DLX:-tmp/rc_fr_singe_scsi_span.dlx}
TAG="r${RING}_k${KBPS}${CUT_AT:+_cut${CUT_AT}x${CUT_FR}}"
tools/vasm/vasmm68k_mot -Fbin -o tmp/stream.bin src/player/stream.s > /dev/null
[ -f tmp/stream_disk.bin ] || python3 tools/bench/prep_stream.py "$DLX" > tmp/prep_stream.log
mkdir -p "tmp/snap_pace_$TAG"; rm -f "tmp/snap_pace_$TAG/x68000"/*.png
# `env` rather than an assignment prefix: an empty ${CUT_AT:+...} in the middle
# of a prefix is not an assignment token, so bash takes the next word as the
# command and the run dies with "SDL_VIDEODRIVER=dummy: command not found".
CUTENV=(); [ -n "$CUT_AT" ] && CUTENV=(DLX_CUT_AT="$CUT_AT" DLX_CUT_FR="$CUT_FR")
( cd tmp && env DLX_PACE=1 DLX_RING_KB=$RING DLX_STREAM_KBPS=$KBPS \
"${CUTENV[@]}" DLX_SLACK_CSV="slack_$TAG.csv" \
SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 900 \
mame x68000 -bios ipl10 -ramsize 2M -video soft -window -sound none \
-nothrottle -plugins -autoboot_script ../tools/bench/stream.lua \
-snapshot_directory "./snap_pace_$TAG" -snapview native -seconds_to_run 90 \
> "pace_$TAG.log" 2>&1 )
# The completion marker is not optional: a run killed mid-decode compares a
# half-drawn screen and reads as a wrap bug rather than as a truncated run.
grep -q "snapshot taken" "tmp/pace_$TAG.log" || {
echo "FAIL($TAG): no snapshot marker -- the pass did not complete."
tail -6 "tmp/pace_$TAG.log"; exit 1; }
echo "=== $TAG"
grep -aE "decoder (PACED|FREE)|ring: |UNDERRUNS|SEEK SLACK|RING-BOUND|RATE-BOUND|BUILD TIME|PIPE CUT|DEADLINE|REQUIRED" \
"tmp/pace_$TAG.log" | sed "s/\[STR\] / /"
python3 tools/bench/verify_decode.py "$DLX" --snap "tmp/snap_pace_$TAG" | tail -2
tools/bench/pace_sweep.sh
Executable
+36
View File
@@ -0,0 +1,36 @@
#!/bin/bash
# Ring x pipe grid for the PACED rig (STATUS item 4, FINDINGS 51).
#
# tools/bench/pace_sweep.sh "<ring KB list>" "<KB/s list>"
#
# Every cell is a full 120-frame decode on the emulated 68000, pixel-verified.
# There is no default rate list: FINDINGS 50 removed the delivery constant from
# this tree and a sweep that invented one back would be the same mistake with
# more rows. `0` means an unlimited pipe, which measures the RING's ceiling with
# delivery removed as a variable -- an upper bound, not a prediction.
set -e
cd "$(dirname "$0")/../.."
RINGS=${1:?ring KB list, quoted}
RATES=${2:?pipe KB/s list, quoted, 0 = unlimited}
printf "%6s %9s %9s %9s %8s %9s %s\n" ring kbps ceiling build_s mean underruns bound
for R in $RINGS; do for K in $RATES; do
L=tmp/pace_r${R}_k${K}.log
bash tools/bench/pace_run.sh "$R" "$K" > /dev/null 2>&1 || { \
printf "%6s %9s FAILED (see %s)\n" "$R" "$K" "$L"; continue; }
python3 - "$L" "$R" "$K" <<'PY'
import re, sys
log, ring, kbps = sys.argv[1], sys.argv[2], sys.argv[3]
t = open(log, errors="replace").read()
def g(p, d="?"):
m = re.search(p, t)
return m.group(1) if m else d
ceil_ = g(r"SEEK SLACK: ceiling (\d+) frames")
build = g(r"BUILD TIME: (\d+) ticks")
mean = g(r"mean ([\d.]+) over the window")
under = g(r"UNDERRUNS: (\d+)/")
bound = "ring" if "RING-BOUND" in t else ("rate" if "RATE-BOUND" in t else "?")
fps = 12.0
print("%6s %9s %9s %9.2f %8s %9s %s" % (
ring, kbps, ceil_, (int(build)/fps if build.isdigit() else -1), mean, under, bound))
PY
done; done
tools/bench/prep_dlx.py
+152 -30
View File
@@ -28,32 +28,112 @@ import sys, os, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
sys.path.insert(0, "tools/bench")
import dlxload as DL
import spans as SP
# The harness loads the WHOLE container into emulated RAM at STREAM=0x30000 and
# the target is a stock 2 MB machine, so there is a hard ceiling on how much of
# a stream can be verified in one pass. The shipping player streams from disk
# into a ring buffer and has no such limit; this is a property of the test rig.
# A `scsi` window overruns it -- 2.84 MB of stream ends at 0x2E591C, 940 KB past
# the 0x200000 top of RAM -- so the frame list is truncated to what fits and the
# truncation is announced. Verifying a prefix is still a real test: SKIP blocks
# make every frame a claim about the one before it.
STREAM_BASE = 0x30000
RAM_TOP = 0x200000
MARGIN = 0x8000 # stack, flags, codebooks live below STREAM_BASE
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--out", default="tmp/decode")
ap.add_argument("--ram", type=lambda v: int(v, 0), default=RAM_TOP,
help="top of emulated RAM (default 0x200000, a stock 2 MB machine)")
ap.add_argument("--all-frames", action="store_true",
help="do NOT truncate to what fits in RAM (the loader will write "
"past the top of memory and the decoder will read garbage)")
a = ap.parse_args()
d = DLX(a.container)
if d.idx_bytes != 1:
sys.exit("2-byte codebook indices: decode.s assumes 1 (k<=256)")
# decode.s reads a u16 span count out of every frame record (FINDINGS 41), so a
# DLX2 container is not merely span-less to it -- the first two bytes of the
# block payload would be read as a count and the frame would decode as garbage.
# Fail here rather than there.
if not d.has_spans:
sys.exit(f"{a.container} is DLX{d.version}: src/player/decode.s expects the "
f"DLX3 span section. Re-encode (tools/encoder/encode.py emits DLX3 "
f"by default) or pass --spans off and use an older decoder.")
# --- codebooks, expanded to one WORD per pixel (high byte is discarded by
# gvram_w, so it is left zero and never has to be cleared)
cb1 = np.zeros((d.k1, 16, 2), np.uint8); cb1[:, :, 1] = d.cb1.reshape(d.k1, 16)
cb4 = np.zeros((d.k4, 4, 2), np.uint8); cb4[:, :, 1] = d.cb4.reshape(d.k4, 4)
# --- codebooks and palette. Both transforms live in tools/bench/dlxload.py so
# that prep_stream.py's ring-buffer rig shares one copy of them rather than
# keeping a second that could drift silently (FINDINGS 49).
cb1, cb4 = DL.expand_codebooks(d)
palb, dark, rendered = DL.pack_palette(d)
def build_synth(d):
"""The synthetic timing frames, as record bodies.
No real frame is all one mode, but the mix is exactly what the "76.6% x
non-SKIP fraction" model of FINDINGS 24.5 assumes away: it prices every
non-SKIP block as one V1-style burst. These price the modes separately,
which is the only way to see which one is expensive.
Every record carries the DLX3 span section, empty or not -- decode.s reads a
u16 count out of all of them, and a synthetic frame that omitted it would
desync the bitstream exactly where the harness is least likely to look.
The last two are the mode the block loop cannot express: a frame that is ALL
SPAN, its mode header entirely SKIP. Two run lengths, because a span costs
per-span plus per-pixel and one length cannot separate them --
all-SPAN-64 full-row runs, the floor of the mode (154 clocks/block,
FINDINGS 30.4)
all-SPAN-4 4-block runs, the break-even against V1 (FINDINGS 40.1)
They price v7 INSIDE decode.s against the constants tools/bench/span.sh
fitted in blit.s. Agreement cross-checks both; disagreement means the
decoder's span pass is not the sequence that was measured.
"""
out = {}
empty = SP.serialise([])
for name, mo, per in (("all-SKIP", 0, 0), ("all-V1", 1, 1),
("all-V4", 2, 4), ("all-RAW", 3, 16)):
out[name] = (bytes([mo * 0x55] * d.mode_bytes) + empty
+ bytes(d.nb * per))
# MIXED-SKIP frames. Every other synthetic frame here is a pure population,
# which is exactly why none of them prices the commonest block in a real
# container: a SKIP that shares its header byte with a coded block, and so
# cannot take the all-SKIP fast path. vq_hybrid's C_SKIP_MIXED has never
# been measured -- it was derived -- and a spanned container is made mostly
# of them, because a spanned block reads SKIP. FINDINGS 41.5.
#
# Two mixes per coded mode, because one equation cannot separate the SKIP
# cost from the cost of the block it shares a group with.
#
# THE HEADER BYTES ROTATE, and that is not decoration. decode.s reaches a
# block's 2 mode bits with `lsr.b #6/#4/#2` and no shift at all for the last
# one, so a block costs 52/48/44/34 clocks of dispatch depending on WHERE in
# its header byte it sits. A fixed byte like 0x01 puts every SKIP at the
# three expensive positions and every V1 at the free one, and solving two
# such equations returns a number that describes no real frame. Cycling the
# byte through the four rotations puts each mode at each position equally,
# which is what a real mode map does.
for nm, bys, per in (("mix-3SKIP-V1", (0x01, 0x04, 0x10, 0x40), 1),
("mix-1SKIP-3V1", (0x54, 0x51, 0x45, 0x15), 1),
("mix-3SKIP-RAW", (0x03, 0x0C, 0x30, 0xC0), 16),
("mix-1SKIP-3RAW", (0xFC, 0xF3, 0xCF, 0x3F), 16)):
hdr = bytes(bys[i % 4] for i in range(d.mode_bytes))
ncoded = sum(bin(b).count("1") and
sum(1 for k in range(4) if (b >> (2 * k)) & 3) for b in hdr[:1])
ncoded = sum(sum(1 for k in range(4) if (b >> (2 * k)) & 3) for b in hdr)
out[nm] = hdr + empty + bytes(ncoded * per)
pat = np.tile(np.arange(d.W, dtype=np.uint8), (d.H, 1))
for name, blocks in (("all-SPAN-64", d.W // 4), ("all-SPAN-4", 4)):
sp = [(y, x, pat[y, x:x + blocks * 4])
for y in range(d.H) for x in range(0, d.W, blocks * 4)]
out[name] = bytes(d.mode_bytes) + SP.serialise(sp)
return out
# --- palette words, I chosen per entry (identical maths to verify_frame256.py)
pal = d.pal.astype(int)
p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
f = pal >> 3
render = lambda I: p6((f << 1) | I[:, None])
I = (((render(np.ones(256, int)) - pal) ** 2).sum(1)
< ((render(np.zeros(256, int)) - pal) ** 2).sum(1)).astype(int)
words = (f[:, 1] << 11) | (f[:, 0] << 6) | (f[:, 2] << 1) | I
palb = np.zeros((256, 2), np.uint8)
palb[:, 0], palb[:, 1] = words >> 8, words & 0xFF
dark = int(((render(I).astype(int)) ** 2).sum(1).argmin())
# --- frame stream: [u32 len][modes][payload] per frame, each record start
# rounded up to a 4-byte boundary.
@@ -65,28 +145,56 @@ dark = int(((render(I).astype(int)) ** 2).sum(1).argmin())
# so this loader realigns it; the encoder should carry the padding itself
# (FINDINGS 28.3). It costs at most 3 bytes per frame -- 36 B/s at 12fps,
# against a 110 KB/s budget.
budget = a.ram - STREAM_BASE - MARGIN
stream, rec_off, pad = bytearray(), [], 0
dropped = 0
# The synthetic timing frames are built FIRST, so their size comes out of the
# RAM budget rather than being appended past it. It used to be appended: the
# stream ran 26 KB beyond the top of a 2 MB machine, which was survivable only
# because the modes it overran are data-independent -- their cost is in the
# mode header, and reading junk payload costs the same as reading pixels. A
# span is not: its two jump DISPLACEMENTS come out of the stream, so an
# out-of-RAM span record jumps into open bus. FINDINGS 41.4.
SYNTH = build_synth(d)
budget -= sum(4 + len(b) + 3 for b in SYNTH.values())
for (o, n) in d.frames:
while len(stream) % 4:
stream += b"\0"; pad += 1
if not a.all_frames and len(stream) + 4 + n > budget:
dropped = d.nframes - len(rec_off)
break
rec_off.append(len(stream))
stream += n.to_bytes(4, "big") + d.raw[o:o + n]
NFRAMES = len(rec_off)
if dropped:
print(f" TRUNCATED: {NFRAMES}/{d.nframes} frames fit in RAM "
f"(stream budget {budget:,} B at 0x{STREAM_BASE:X} under a "
f"{a.ram/1024/1024:.0f} MB machine); {dropped} frames dropped.\n"
f" This is the TEST RIG's limit, not the player's -- the player "
f"streams into a ring buffer.")
# Synthetic single-mode frames. No real frame is all one mode, but the mix is
# exactly what the "76.6% x non-SKIP fraction" model of FINDINGS 24.5 assumes
# away: it prices every non-SKIP block as one V1-style burst. These four price
# the modes separately, which is the only way to see which one is expensive.
# Append the synthetic frames the budget above already reserved.
synth = {}
for name, mo, per in (("all-SKIP", 0, 0), ("all-V1", 1, 1),
("all-V4", 2, 4), ("all-RAW", 3, 16)):
for name, body in SYNTH.items():
while len(stream) % 4:
stream += b"\0"; pad += 1
synth[name] = len(stream)
hdr = bytes([mo * 0x55] * d.mode_bytes)
stream += (d.mode_bytes + d.nb * per).to_bytes(4, "big") + hdr + bytes(d.nb * per)
stream += len(body).to_bytes(4, "big") + body
assert STREAM_BASE + len(stream) <= a.ram, (
f"stream ends at 0x{STREAM_BASE+len(stream):X}, past the 0x{a.ram:X} top "
f"of RAM -- the budget arithmetic above is wrong")
# --- timing anchors: the distribution, not its mean (FINDINGS 25.6's lesson)
ns = np.array([100 * (d.modes(i) != 0).mean() for i in range(d.nframes)])
#
# A spanned block reads SKIP here, so this fraction is the BLOCK-LOOP workload
# and no longer the frame's whole cost: the span section is the rest of it. The
# anchors still pick out the extremes of the block loop, which is what they are
# for, but a frame's total decode time now has two terms.
ns = np.array([100 * (d.modes(i) != 0).mean() for i in range(NFRAMES)])
nsp = np.array([len(d.spans(i)[0]) for i in range(NFRAMES)])
spx = np.array([sum(len(p) for _, _, p in d.spans(i)[0]) for i in range(NFRAMES)])
order = np.argsort(ns)
pick = {
"min non-SKIP %.1f%%" % ns[order[0]]: int(order[0]),
@@ -95,16 +203,18 @@ pick = {
"max non-SKIP %.1f%%" % ns[order[-1]]: int(order[-1]),
}
anchors = [(n, rec_off[i], float(ns[i])) for n, i in pick.items()]
for name in ("all-SKIP", "all-V1", "all-V4", "all-RAW"):
for name in ("all-SKIP", "all-V1", "all-V4", "all-RAW",
"all-SPAN-64", "all-SPAN-4", "mix-3SKIP-V1", "mix-1SKIP-3V1",
"mix-3SKIP-RAW", "mix-1SKIP-3RAW"):
anchors.append((f"synthetic {name}", synth[name],
0.0 if name == "all-SKIP" else 100.0))
0.0 if name.startswith(("all-SKIP", "all-SPAN")) else 100.0))
blob = cb1.tobytes() + cb4.tobytes() + palb.tobytes() + bytes(stream)
open(a.out + "_data.bin", "wb").write(blob)
with open(a.out + "_meta.lua", "w") as fh:
fh.write("-- generated by tools/bench/prep_dlx.py -- do not edit\nreturn {\n")
fh.write(f" W={d.W}, H={d.H}, fps={d.fps}, nframes={d.nframes},\n")
fh.write(f" W={d.W}, H={d.H}, fps={d.fps}, nframes={NFRAMES},\n")
fh.write(f" k1={d.k1}, k4={d.k4}, dark={dark},\n")
fh.write(f" cb1_len={cb1.nbytes}, cb4_len={cb4.nbytes}, pal_len={palb.nbytes},\n")
fh.write(f" stream_len={len(stream)},\n")
@@ -118,6 +228,18 @@ print(f" cb1 {cb1.nbytes} B + cb4 {cb4.nbytes} B expanded, palette {palb.nbytes
f"stream {len(stream)} B -> {a.out}_data.bin ({len(blob)} B)")
print(f" non-SKIP blocks/frame: median {np.median(ns):.1f}% "
f"p90 {np.percentile(ns,90):.1f}% max {ns.max():.1f}%")
print(f" darkest palette entry: index {dark} -> {tuple(render(I)[dark])}")
print(f" 4-byte record alignment cost {pad} B over {d.nframes} frames "
f"({pad / d.nframes:.2f} B/frame = {pad / d.nframes * d.fps:.0f} B/s)")
print(f" v7 spans/frame: median {np.median(nsp):.0f} max {nsp.max()} "
f"({int((nsp>0).sum())}/{NFRAMES} frames); pixels painted by one: "
f"median {100*np.median(spx)/(d.W*d.H):.1f}% "
f"max {100*spx.max()/(d.W*d.H):.1f}% of the picture")
print(f" darkest palette entry: index {dark} -> {tuple(rendered[dark])}")
# A DLX2 container already carries this padding (FINDINGS 28.3 closed, session
# 9), so the realignment above re-derives bytes that were already there and the
# loader is doing no work. On a DLX1 container it is load-bearing: 94 of 120
# record starts land on odd addresses, and each one is an address error.
src_bad = sum(1 for (o, _) in d.frames[:NFRAMES] if (o - 4) % 4)
print(f" 4-byte record alignment cost {pad} B over {NFRAMES} frames "
f"({pad / NFRAMES:.2f} B/frame = {pad / NFRAMES * d.fps:.0f} B/s)")
print(f" source container is DLX{d.version}: {src_bad}/{NFRAMES} record starts "
f"unaligned" + (" -- this loader is what makes it decodable"
if src_bad else " -- the container carries its own padding"))
tools/bench/prep_frame.py
+14 -3
View File
@@ -15,16 +15,27 @@ argv = [a for a in sys.argv[1:] if not a.startswith("--")]
# to 0 displays palette entry 0, and a free mediancut palette puts a real image
# colour there. Costs one of 256 entries; measured quality cost is negligible.
RESERVE = "--reserve-black" in sys.argv
# --pack-transparent: the layout FINDINGS 46.6 needs. The packed scheme puts
# the TOP graphics page's index 0 to work as a transparency key, so index 0 must
# never appear in the picture -- and black therefore cannot live there. So:
# quantise to 254, place them at 1..254, put black at 255, leave 0 UNUSED.
# Costs two of 256 entries against --reserve-black's one.
PACKT = "--pack-transparent" in sys.argv
src, out = argv[0], argv[1]
f = sorted(glob.glob(f"{src}/*.png"))[int(argv[2]) if len(argv) > 2 else 0]
im = Image.open(f).convert("RGB")
W, H = im.size
n = 255 if RESERVE else 256
n = 254 if PACKT else (255 if RESERVE else 256)
q = im.quantize(colors=n, method=Image.MEDIANCUT, dither=Image.NONE)
pal = np.array(q.getpalette()[:n*3], dtype=np.uint8).reshape(n, 3)
idx = np.asarray(q, dtype=np.uint8)
if RESERVE:
if PACKT:
# 0 unused (transparency key), 1..254 picture, 255 black
pal = np.vstack([np.zeros((1, 3), np.uint8), pal, np.zeros((1, 3), np.uint8)])
idx = idx + 1
assert idx.min() >= 1 and idx.max() <= 254, "index 0/255 must stay free"
elif RESERVE:
pal = np.vstack([np.zeros((1, 3), np.uint8), pal]) # index 0 = black
idx = idx + 1
@@ -37,4 +48,4 @@ with open(out, "wb") as fh:
# reference PNG of exactly what the X68000 should display
Image.fromarray(pal[idx]).save(out.replace(".bin", "_ref.png"))
print(f"src={f} {W}x{H} colors={len(np.unique(idx))}"
f"{' (idx 0 reserved black)' if RESERVE else ''} -> {out}")
f"{' (idx 0 unused/transparent, 255 black)' if PACKT else (' (idx 0 reserved black)' if RESERVE else '')} -> {out}")
tools/bench/prep_spans.py
+170
View File
@@ -0,0 +1,170 @@
#!/usr/bin/env python3
"""Generate V5 span streams for tools/bench/span.lua (FINDINGS 29.5 item 1).
FINDINGS 29 prices a new decoder mode -- a row-linear run of word-expanded
literal pixels, movem.l'd straight from the stream buffer into GVRAM -- at
`4 * (50 + 4L * 9.08)` cycles for a run of L blocks. Both halves of that are
extrapolations: the 50-cycle per-span overhead is hand-derived, and the 9.08
cycles/pixel was measured (FINDINGS 24 V1) at FULL ROW WIDTH with 12-register
bursts, which a short span cannot match. This script builds the stimulus that
replaces both numbers with measured ones.
One stream per span length. Every stream covers the SAME 192x256 picture
completely, so all of them draw an identical, verifiable frame and differ only
in how many spans it is cut into -- which is what lets span.lua regress
cycles = A * spans + B * pixels
across the set and read the per-span overhead off directly.
Two stream formats, both big-endian, both drawing the same frame.
v5 -- a decoder handed (x, npix) that works out the copy itself:
per row, 192 rows in order:
u16 nspans
nspans * { u16 x, u16 npix, npix * u16 pixel }
v6 -- the same spans with that arithmetic moved here, where it is free:
u16 nspans (whole frame; there is no row structure)
nspans * { u32 absolute GVRAM address, u16 jump displacement,
units * 48 bytes of pixels }
Span lengths are multiples of 24 pixels (one chain unit) and the last span
in a row may overrun the visible 256 by up to 23 pixels, which is free: the
line stride is 1024 bytes and only the first 512 are displayed. The jump
displacement selects an entry point into the decoder's unrolled copy chain.
v7 -- v6 plus a second, FINER chain for the tail (FINDINGS 39.4). Padding to
v6's 24-pixel quantum wastes ~11 pixels on an average span, and FINDINGS
39.3 attributes 86% of the DMAC array-chain's advantage over v6 to it. A
v7 span is 24*c + 2*f pixels, so the quantum is 2 and a run of 4x4 blocks
(always a multiple of 4 pixels) pads to NOTHING:
u16 nspans
nspans * { u32 absolute GVRAM address, u16 coarse displacement,
c * 48 bytes of pixels,
u16 fine displacement, f * 4 bytes of pixels }
The fine displacement is in the STREAM rather than the record because that
is what lets the decoder keep all 12 payload registers: the coarse chain
falls out into a `move.w (a0)+,d0 / jmp` with d0 dead and a0 pointing at
it. Costed here as an 8-byte record, since it is 2 more bytes a span.
Pixels are word-expanded with the palette index in the low byte; the high byte
is whatever we put there because gvram_w masks it off (x68k_crtc.cpp:501).
"""
import struct, sys
import numpy as np
SRC = sys.argv[1] if len(sys.argv) > 1 else "tmp/frame256.bin"
OUT = sys.argv[2] if len(sys.argv) > 2 else "tmp/spans.bin"
META = OUT.replace(".bin", "_meta.lua")
d = open(SRC, "rb").read()
assert d[:4] == b"DLXR", SRC
W, H = struct.unpack(">HH", d[4:8])
idx = np.frombuffer(d[8+768:8+768+W*H], np.uint8).reshape(H, W)
assert (W, H) == (256, 192), f"{W}x{H}: span bench assumes the 256x192 picture"
# (span length in pixels, x of the first span). 4 px = one 4x4 block wide, the
# case the whole FINDINGS 29 argument turns on; 256 = one span per row, the
# case closest to the V1 measurement it extrapolates from. 16u starts at an
# odd x so its bursts run at addr mod 4 == 2: a claim about the 68000's 16-bit
# bus that costs nothing to test and would be embarrassing to assume.
CONFIGS = [(4, 0), (8, 0), (12, 0), (16, 0), (16, 1), (20, 0), (24, 0),
(32, 0), (48, 0), (64, 0), (128, 0), (256, 0)]
# v6 geometry, and it must match blit.s: 12 registers per movem = 48 bytes =
# 24 pixels per chain unit, 11 units in the chain.
UNITPX, UNITSZ, UNITS = 24, 12, 11
# v7 geometry, and it must match blit.s: coarse unit as v6, fine unit is one
# `move.l (a0)+,(a2)+` = 2 bytes of code = 2 pixels, 11 of them (22 px > 24).
FINEPX, FINESZ, FINES = 2, 2, 11
GVRAM, YOFF, STRIDE = 0xC00000, 32, 1024
# v7 span lengths, in pixels. Multiples of 4 (a real span is a run of 4x4
# blocks), chosen so the fine remainder P mod 24 takes every value a real span
# can: 0, 4, 8, 12, 16, 20. 4/8/12/16/20 are pure-fine, 24/48/72/120/240 are
# pure-coarse, the rest mix -- which is what makes the three-term fit
# cycles = A*spans + Bc*coarse_px + Bf*fine_px identifiable.
V7CONFIGS = [4, 8, 12, 16, 20, 24, 28, 44, 48, 72, 100, 120, 256]
blob, metas = bytearray(), []
for P, x0 in CONFIGS:
off = len(blob)
nspans = npix = 0
for y in range(H):
cuts = []
x = 0
if x0: # a short leading span to shift the phase
cuts.append((0, x0)); x = x0
while x < W:
n = min(P, W - x)
cuts.append((x, n)); x += n
blob += struct.pack(">H", len(cuts))
for x, n in cuts:
blob += struct.pack(">HH", x, n)
blob += idx[y, x:x+n].astype(">u2").tobytes()
nspans += 1; npix += n
metas.append(dict(name=f"{P}{'u' if x0 else ''}", p=P, x0=x0, off=off,
len=len(blob)-off, nspans=nspans, npix=npix,
cpx=npix, fpx=0, var=5))
# v6: one config per chain depth, so the fit sees spans from 24 to 264 pixels.
for units in range(1, UNITS+1):
P = units * UNITPX
off = len(blob)
nspans = npix = 0
rows = []
for y in range(H):
x = 0
while x < W:
rows.append((y, x)); x += P
blob += struct.pack(">H", len(rows))
for y, x in rows:
blob += struct.pack(">IH", GVRAM + (YOFF+y)*STRIDE + x*2,
(UNITS-units)*UNITSZ)
# Pad the last span of a row past the visible width; the overrun lands
# in the undisplayed half of the line.
px = np.concatenate([idx[y, x:x+P], np.zeros(max(0, x+P-W), np.uint8)])
blob += px.astype(">u2").tobytes()
nspans += 1; npix += P
metas.append(dict(name=f"{P}", p=P, x0=0, off=off, len=len(blob)-off,
nspans=nspans, npix=npix, cpx=npix, fpx=0, var=6))
# v7: same tiling, but the span is cut at a 2-pixel quantum instead of 24.
for P in V7CONFIGS:
units, fine = divmod(P, UNITPX)
assert fine % FINEPX == 0 and fine // FINEPX <= FINES, P
assert units <= UNITS, P
off = len(blob)
nspans = npix = 0
rows = []
for y in range(H):
x = 0
while x < W:
rows.append((y, x)); x += P
blob += struct.pack(">H", len(rows))
for y, x in rows:
assert x + P <= STRIDE // 2, (P, x) # the overrun must stay on the line
blob += struct.pack(">IH", GVRAM + (YOFF+y)*STRIDE + x*2,
(UNITS-units)*UNITSZ)
px = np.concatenate([idx[y, x:x+P], np.zeros(max(0, x+P-W), np.uint8)])
blob += px[:units*UNITPX].astype(">u2").tobytes()
blob += struct.pack(">H", (FINES - fine//FINEPX)*FINESZ)
blob += px[units*UNITPX:].astype(">u2").tobytes()
nspans += 1; npix += P
metas.append(dict(name=f"{P}", p=P, x0=0, off=off, len=len(blob)-off,
nspans=nspans, npix=npix,
cpx=nspans*units*UNITPX, fpx=nspans*fine, var=7))
open(OUT, "wb").write(blob)
with open(META, "w") as f:
f.write("-- generated by tools/bench/prep_spans.py -- do not edit\nreturn {\n")
f.write(f" W={W}, H={H}, total={len(blob)},\n configs = {{\n")
for m in metas:
f.write(" {{var={var}, name=\"{name}\", p={p}, x0={x0}, off={off},"
" len={len}, nspans={nspans}, npix={npix}, cpx={cpx},"
" fpx={fpx}}},\n".format(**m))
f.write(" },\n}\n")
print(f"{SRC} {W}x{H} -> {OUT} {len(blob)} B, {len(metas)} configs")
for m in metas:
print(f" v{m['var']} span {m['name']:>4} px: {m['nspans']:6d} spans, "
f"{m['npix']:6d} px, {m['len']:7d} B "
f"(+{100*m['len']/(2*W*H)-100:.1f}% over bare pixels)")
tools/bench/prep_stream.py
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env python3
"""Lay a DLX3 container out as a DISK for the ring-buffer rig (FINDINGS 49).
python3 tools/bench/prep_stream.py <in.dlx> [--out tmp/stream]
prep_dlx.py's output is one blob that tools/bench/decode.lua pushes into
emulated RAM in its entirety. That is what makes its rig RAM-bound -- a `scsi`
window is 5,261,814 B of stream and needs a 6 MB machine to hold it (FINDINGS
45) -- and, much more importantly, it is nothing like the shipping player, which
never holds a window at once.
This writes three files instead:
<out>_cb.bin codebooks + palette. ~10 KB, loaded into RAM once, exactly as
before: these are LOAD-TIME costs and not per-frame ones.
<out>_disk.bin the frame records, `[u32 len][body]` each padded up to 4, laid
end to end. tools/bench/stream.lua reads this from the HOST
filesystem and feeds it into a bounded ring, so the emulated
machine's RAM stops bounding how much of a window can be
tested. A stock 2 MB machine can now run all 120 frames.
<out>_meta.lua geometry, and the record index.
THE RECORD INDEX IS NOT A CONVENIENCE. src/player/stream.s takes each frame's
base address from a descriptor the producer wrote, rather than deriving it from
where the last frame ended, because under the `aligned` wrap policy the next
record may be at the ring's base instead of just after its predecessor. The
producer therefore has to know record boundaries before it places them -- which
is what an index is. A branching laserdisc game needs one anyway to seek to a
branch point, so the policy that costs no clocks (tools/analysis/19_ring_stream.py)
reuses a structure the player cannot avoid.
The 4-byte record padding is the same one decode.s needs and DLX3 already
carries: `move.l (a0)+,d0` on an odd address is an ADDRESS ERROR on a 68000, not
a slow read. FINDINGS 28.3.
NO SYNTHETIC TIMING FRAMES. prep_dlx.py appends ten of them to price the block
modes separately; this rig measures delivery, not decode, and its per-frame cost
anchors are prep_dlx.py's job. Mixing them in would put frames on the wire that
no encoder emits and no rate controller sized.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/bench")
import numpy as np
from dlx import DLX
import dlxload as DL
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--out", default="tmp/stream")
a = ap.parse_args()
d = DLX(a.container)
if d.idx_bytes != 1:
sys.exit("2-byte codebook indices: src/player/ assumes 1 (k<=256)")
if not d.has_spans:
sys.exit(f"{a.container} is DLX{d.version}: src/player/stream.s expects the "
f"DLX3 span section (see prep_dlx.py for why a DLX2 container "
f"decodes as garbage rather than merely losing its spans).")
cb1, cb4 = DL.expand_codebooks(d)
palb, dark, rendered = DL.pack_palette(d)
open(a.out + "_cb.bin", "wb").write(cb1.tobytes() + cb4.tobytes() + palb.tobytes())
disk, index = bytearray(), []
for (o, n) in d.frames:
start = len(disk)
disk += n.to_bytes(4, "big") + d.raw[o:o + n]
while len(disk) % 4:
disk += b"\0"
index.append((start, len(disk) - start))
open(a.out + "_disk.bin", "wb").write(bytes(disk))
rec = np.array([n for _, n in index])
with open(a.out + "_meta.lua", "w") as fh:
fh.write("-- generated by tools/bench/prep_stream.py -- do not edit\nreturn {\n")
fh.write(f" W={d.W}, H={d.H}, fps={d.fps}, nframes={d.nframes}, dark={dark},\n")
fh.write(f" cb1_len={cb1.nbytes}, cb4_len={cb4.nbytes}, pal_len={palb.nbytes},\n")
fh.write(f" disk_len={len(disk)}, maxrec={int(rec.max())},\n")
fh.write(" index={\n")
for off, ln in index:
fh.write(f" {{off={off}, len={ln}}},\n")
fh.write(" },\n}\n")
print(f"{a.container}: {d.nframes} frames, {d.W}x{d.H}")
print(f" codebooks+palette {cb1.nbytes + cb4.nbytes + palb.nbytes:,} B -> "
f"{a.out}_cb.bin")
print(f" disk image {len(disk):,} B -> {a.out}_disk.bin "
f"(records: min {rec.min():,} median {int(np.median(rec)):,} "
f"max {rec.max():,})")
print(f" wire rate {rec.mean()*d.fps/1024:.1f} KB/s video at {d.fps} fps")
print(f" A ring must hold one whole record contiguously: >= {rec.max():,} B "
f"({rec.max()/1024:.1f} KB) before any policy or prefill.")
tools/bench/probe_bit11_blank.lua
+85
View File
@@ -0,0 +1,85 @@
-- FINDINGS 47: does CRTC R20 bit 11 ("G-VRAM set to buffer") blank the display?
--
-- Byte-for-byte tools/bench/show_frame256.lua -- the KNOWN-GOOD 256-colour test
-- that check.sh gates on -- with exactly one line added: R20 bit 11 is set after
-- MODE.apply. Everything else, including the ordinary one-word-per-pixel
-- picture, is unchanged, so anything but a correct frame is caused by that bit.
--
-- MEASURED on MAME: the screen goes FULLY BLACK (max channel 0). Buffer mode is
-- a write window, not a display mode.
-- MEASURED on px68k (tools/bench/gvpack --keepbuffer): it does NOT blank.
-- The two emulators disagree, and that disagreement is what FINDINGS 47.4 is
-- about -- it decides whether the packed path is usable for continuous video.
-- Same as show_frame.lua, but sets a REAL 256x256 CRTC mode instead of
-- borrowing the IPL's 768x512 text timing. Proves the mode table in
-- crtc_mode.lua and removes the x=512 wrap of FINDINGS 22.5.
M=manager.machine; SP=M.devices[":maincpu"].spaces["program"]; SUB=nil
local function load_mode()
for _,p in ipairs{"../tools/bench/crtc_mode.lua","tools/bench/crtc_mode.lua","crtc_mode.lua"} do
local f=loadfile(p); if f then return f() end
end
error("crtc_mode.lua not found")
end
local MODE = load_mode()
local GVRAM, GPAL = 0xC00000, 0xE82000
local f=io.open("frame256.bin","rb"); local d=f:read("a"); f:close()
local function B(i) return string.byte(d,i) end
local W,H = B(5)*256+B(6), B(7)*256+B(8)
local PAL0, PIX0 = 9, 9+256*3
local YOFF = (MODE.height - H) // 2 -- letterbox 192 rows inside 256
-- GGGGGRRRRRBBBBBI, confirmed from x68k_v.cpp. The LSB "I" is SHARED by all
-- three channels: each renders as pal6bit((field<<1)|I). Hardcoding I=1 (as
-- show_frame.lua does) makes true black unreachable -- pal6bit(1) = 4 -- so I
-- is chosen per entry to minimise summed squared error over R,G,B.
local function pal6(v) return ((v<<2)|(v>>4)) & 0xff end
local function pack(r,g,b)
local f = {r>>3, g>>3, b>>3}
local best, bestI = nil, 1
for I=0,1 do
local e=0
for c=1,3 do
local want = ({r,g,b})[c]
local d = pal6((f[c]<<1)|I) - want
e = e + d*d
end
if best==nil or e<best then best,bestI = e,I end
end
return (f[2]<<11)|(f[1]<<6)|(f[3]<<1)|bestI
end
local function T() local t=M.time; return t.seconds+t.attoseconds/1e18 end
local st,tp="wait",nil
SUB = emu.add_machine_frame_notifier(function()
local t=T()
if st=="wait" then
if t<3.0 then return end
MODE.apply(SP)
SP:write_u16(0xE80000+20*2, MODE.r20 | 0x0800) -- BUFFER MODE
-- clear the letterbox rows: GVRAM holds IPL leftovers, not zeros
for y=0,MODE.height-1 do
if y<YOFF or y>=YOFF+H then
local base=GVRAM+y*1024
for x=0,MODE.width-1 do SP:write_u16(base+x*2,0) end
end
end
for c=0,255 do
local o=PAL0+c*3
SP:write_u16(GPAL+c*2, pack(B(o),B(o+1),B(o+2)))
end
for y=0,H-1 do
local row,base = PIX0+y*W, GVRAM+(y+YOFF)*1024
for x=0,W-1 do SP:write_u16(base+x*2, B(row+x)) end
end
print(string.format("[256] R00-R08 %d %d %d %d %d %d %d %d %d R20=%04X yoff=%d t=%.3f",
SP:read_u16(0xE80000),SP:read_u16(0xE80002),SP:read_u16(0xE80004),SP:read_u16(0xE80006),
SP:read_u16(0xE80008),SP:read_u16(0xE8000A),SP:read_u16(0xE8000C),SP:read_u16(0xE8000E),
SP:read_u16(0xE80010),SP:read_u16(0xE80028), YOFF, t))
st,tp="painted",t
elseif st=="painted" and t>tp+0.30 then
M.video:snapshot(); print("[256] snapshot"); st="done"; M:exit()
end
end)
tools/bench/probe_packed.lua
+36
View File
@@ -0,0 +1,36 @@
-- Diagnostic for FINDINGS 46.6: separate the WRITE path from the DISPLAY path.
M=manager.machine; SP=M.devices[":maincpu"].spaces["program"]; SUB=nil
local function load_mode()
for _,p in ipairs{"../tools/bench/crtc_mode.lua","tools/bench/crtc_mode.lua"} do
local f=loadfile(p); if f then return f() end end
error("no crtc_mode.lua") end
local MODE = load_mode()
local CRTC, GV = 0xE80000, 0xC00000
local function T() local t=M.time; return t.seconds+t.attoseconds/1e18 end
local st="wait"
SUB = emu.add_machine_frame_notifier(function()
if st~="wait" then return end
if T()<3.0 then return end
MODE.apply(SP)
local function trial(name, r20)
SP:write_u16(CRTC+20*2, r20)
-- clear both bytes of word 0 via the two aliases, masked mode
SP:write_u16(CRTC+20*2, MODE.r20)
SP:write_u16(GV, 0); SP:write_u16(GV+0x80000, 0)
SP:write_u16(CRTC+20*2, r20)
SP:write_u16(GV, 0xAB5C) -- the write under test
local raw = SP:read_u16(GV)
SP:write_u16(CRTC+20*2, MODE.r20) -- back to masked to read pages
local p0 = SP:read_u16(GV) -- page 0 alias -> low byte
local p1 = SP:read_u16(GV+0x80000) -- page 1 alias -> high byte
SP:write_u16(CRTC+20*2, r20)
print(string.format("[PROBE] %-22s R20=%04X wrote AB5C raw=%04X page0=%02X page1=%02X",
name, r20, raw, p0 & 0xff, p1 & 0xff))
end
trial("masked (bit11=0)", MODE.r20)
trial("buffer (bit11=1)", MODE.r20 | 0x0800)
-- what does the video controller look like after MODE.apply?
print(string.format("[PROBE] VC R0=%04X R1=%04X R2=%04X",
SP:read_u16(0xE82400), SP:read_u16(0xE82500), SP:read_u16(0xE82600)))
st="done"; M:exit()
end)
tools/bench/probe_page1.lua
+52
View File
@@ -0,0 +1,52 @@
-- Does graphics PAGE 1 display at all in 256-colour mode, and under what
-- priority? page0 <- 0 everywhere, page1 <- 0xC0 (white) everywhere.
-- If page 0 composites on top TRANSPARENTLY, the screen should be white.
M=manager.machine; SP=M.devices[":maincpu"].spaces["program"]; SUB=nil
local function load_mode()
for _,p in ipairs{"../tools/bench/crtc_mode.lua","tools/bench/crtc_mode.lua"} do
local f=loadfile(p); if f then return f() end end
error("no crtc_mode.lua") end
local MODE = load_mode()
local CRTC, GV, GPAL = 0xE80000, 0xC00000, 0xE82000
local function T() local t=M.time; return t.seconds+t.attoseconds/1e18 end
local st,tp,n = "wait",nil,0
-- (label, VCReg1, VCReg2, page1 scrollX)
local TRIALS = {
{"vc1=0000 scroll384", 0x0000, 0x001F, 384},
{"vc1=0002 scroll384", 0x0002, 0x001F, 384},
{"vc1=06E4 scroll384", 0x06E4, 0x001F, 384},
{"vc1=0000 scroll0", 0x0000, 0x001F, 0},
{"vc1=0002 scroll0", 0x0002, 0x001F, 0},
{"vc1=0000 vc2=00FF", 0x0000, 0x00FF, 0},
}
SUB = emu.add_machine_frame_notifier(function()
local t=T()
if st=="wait" then
if t<3.0 then return end
MODE.apply(SP)
SP:write_u16(GPAL+0*2, 0x0000) -- index 0 = black
SP:write_u16(GPAL+0xC0*2, 0xFFFF) -- index C0 = white
-- buffer mode: one pass sets page0=0x00 and page1=0xC0 for every word
SP:write_u16(CRTC+20*2, MODE.r20 | 0x0800)
for y=0,255 do
local base=GV+y*1024
for x=0,255 do SP:write_u16(base+x*2, 0xC000) end
end
SP:write_u16(CRTC+20*2, MODE.r20)
st,tp="run",t
elseif st=="run" and t>tp+0.30 then
n = n + 1
if n > #TRIALS then print("[P1] done"); M:exit(); st="done"; return end
local tr = TRIALS[n]
SP:write_u16(0xE82500, tr[2])
SP:write_u16(0xE82600, tr[3])
SP:write_u16(CRTC+16*2, tr[4]); SP:write_u16(CRTC+18*2, tr[4])
print(string.format("[P1] trial %d: %s", n, tr[1]))
tp = t
st = "snap"
elseif st=="snap" and t>tp+0.20 then
M.video:snapshot(); st="run"; tp=t
end
end)
tools/bench/show_frame256_packed.lua
+141
View File
@@ -0,0 +1,141 @@
-- FINDINGS 46.6: does the PACKED layout display correctly?
--
-- The claim under test is that a 256-colour frame can be delivered at 1.0 byte
-- per pixel instead of 2.0, by writing FULL 16-bit words into GVRAM and letting
-- the two 256-colour pages show different halves of the screen:
--
-- * CRTC R20 bit 11 ("G-VRAM set to buffer") stops the write path masking the
-- CPU's high byte away, so one word write lands TWO picture bytes.
-- MAME x68k_crtc.cpp gvram_w; px68k GVRAM_Write's CRTC_Regs[0x28]&8.
-- * word value = (page1 << 8) | page0 -- page 0 is the LOW byte, page 1 the
-- HIGH byte (gvram_w writes `data & 0x00ff` for page 0 and
-- `(data & 0x00ff) << 8` for page 1).
-- * page 0 is the OPAQUE bottom layer, unscrolled: it carries screen columns
-- 0..127 from the low bytes of words 0..127.
-- * page 1 is the TRANSPARENT top layer, X-scrolled by 384 (= -128 mod 512).
-- Column c fetches page1[(c + 384) & 511]:
-- c = 128..255 -> storage 0..127 -> the high bytes of words 0..127,
-- which carry the right half.
-- c = 0..127 -> storage 384..511 -> zeroed, so transparent, so the
-- opaque page 0 shows through.
-- * so words 0..127 of each row hold the WHOLE row: 128 words = 256 bytes for
-- 256 pixels. 1.0 byte/pixel against 2.0.
--
-- Which page is on top is set by video controller R1 (0xE82500). MEASURED:
-- 0x0000 puts page 0 on top (its zeros then cover page 1 and the right half is
-- black -- this was the first failure); 0x0002 puts page 1 on top, which is
-- what this needs.
--
-- The transparency is why the blob is built with --pack-transparent: the top
-- page's index 0 is the key, so index 0 must never appear in the picture --
-- black lives at 255 instead.
--
-- PASS = the snapshot is pixel-identical to what the ordinary unpacked
-- 256-colour path produces, which tools/bench/verify_frame256.py already checks.
M=manager.machine; SP=M.devices[":maincpu"].spaces["program"]; SUB=nil
local function load_mode()
for _,p in ipairs{"../tools/bench/crtc_mode.lua","tools/bench/crtc_mode.lua","crtc_mode.lua"} do
local f=loadfile(p); if f then return f() end
end
error("crtc_mode.lua not found")
end
local MODE = load_mode()
local GVRAM, GPAL = 0xC00000, 0xE82000
local CRTC = 0xE80000
local f=io.open("frame256p.bin","rb"); local d=f:read("a"); f:close()
local function B(i) return string.byte(d,i) end
local W,H = B(5)*256+B(6), B(7)*256+B(8)
local PAL0, PIX0 = 9, 9+256*3
local YOFF = (MODE.height - H) // 2
local BLACK = 255 -- letterbox index, NOT 0
local function pal6(v) return ((v<<2)|(v>>4)) & 0xff end
local function pack(r,g,b)
local fl = {r>>3, g>>3, b>>3}
local best, bestI = nil, 1
for I=0,1 do
local e=0
for c=1,3 do
local want = ({r,g,b})[c]
local dd = pal6((fl[c]<<1)|I) - want
e = e + dd*dd
end
if best==nil or e<best then best,bestI = e,I end
end
return (fl[2]<<11)|(fl[1]<<6)|(fl[3]<<1)|bestI
end
-- screen index at (y,x) over the full 256x256, letterbox included
local function pix(y,x)
if y < YOFF or y >= YOFF+H then return BLACK end
return B(PIX0 + (y-YOFF)*W + x)
end
local function T() local t=M.time; return t.seconds+t.attoseconds/1e18 end
local st,tp="wait",nil
SUB = emu.add_machine_frame_notifier(function()
local t=T()
if st=="wait" then
if t<3.0 then return end
MODE.apply(SP)
-- R20 with bit 11 SET: unmasked full-word writes into GVRAM.
local R20 = MODE.r20 | 0x0800
SP:write_u16(CRTC + 20*2, R20)
-- Graphic scroll. A 256-colour page is assembled from TWO nibble planes
-- with independent scroll registers (px68k Grp_DrawLine8 reads scroll sets
-- page*2 and page*2+1), so BOTH of a page's registers must agree or the
-- page tears between its low and high nibble.
-- R12/R13 = set 0 X/Y, R14/R15 = set 1 -> page 0
-- R16/R17 = set 2 X/Y, R18/R19 = set 3 -> page 1
SP:write_u16(CRTC + 12*2, 0); SP:write_u16(CRTC + 13*2, 0)
SP:write_u16(CRTC + 14*2, 0); SP:write_u16(CRTC + 15*2, 0)
SP:write_u16(CRTC + 16*2, 384); SP:write_u16(CRTC + 17*2, 0)
SP:write_u16(CRTC + 18*2, 384); SP:write_u16(CRTC + 19*2, 0)
-- Priority: page 1 ON TOP of page 0, index 0 transparent. Measured on
-- MAME (tools/bench/probe_page1.lua): 0x0000 -> page 0 on top, screen right
-- half black; 0x0002 -> page 1 on top, right half correct.
SP:write_u16(0xE82500, 0x0002)
for c=0,255 do
local o=PAL0+c*3
SP:write_u16(GPAL+c*2, pack(B(o),B(o+1),B(o+2)))
end
-- The whole 256x256 screen, packed. Words 0..127 carry columns i (page 0,
-- low byte) and i+128 (page 1, high byte).
--
-- Words 128..511 are zeroed ONCE and never touched per frame: what matters
-- there is page 1's storage at 384..511, which the +384 scroll puts under
-- screen columns 0..127 and which must read 0 so the opaque page 0 shows
-- through. This is static setup, not part of the 1.0 B/pixel payload.
for y=0,MODE.height-1 do
local base = GVRAM + y*1024
for i=128,511 do
SP:write_u16(base + i*2, 0)
end
for i=0,127 do
SP:write_u16(base + i*2, (pix(y, i+128) << 8) | pix(y, i))
end
end
-- Buffer mode BLANKS the graphics layer (measured: the screen is black
-- while bit 11 is set), so it is a write window, not a display mode.
-- Clear it now that the packed words are in and let the display read them.
SP:write_u16(CRTC + 20*2, MODE.r20)
print(string.format("[PACK] R20=%04X (bit11=%d) scrollX p0=%d p1=%d "
.."wrote %d words/row for %d px/row yoff=%d",
SP:read_u16(CRTC+20*2), (SP:read_u16(CRTC+20*2)>>11)&1,
SP:read_u16(CRTC+12*2), SP:read_u16(CRTC+16*2), 128, 256, YOFF))
st,tp="painted",t
elseif st=="painted" and t>tp+0.30 then
M.video:snapshot(); print("[PACK] snapshot"); st="done"; M:exit()
end
end)
tools/bench/span.lua
+264
View File
@@ -0,0 +1,264 @@
-- Measure the cost of a row-linear literal SPAN on the 68000 (FINDINGS 29.5.1).
--
-- FINDINGS 29 proposes one new decoder mode and prices it at
-- 4 * (50 + 4L*9.08) cycles for a run of L blocks
-- then labels the whole section DERIVED, NOT MEASURED, because both terms are
-- extrapolations: the 50-cycle per-span overhead is hand-derived, and the 9.08
-- cycles/pixel is a FINDINGS 24 measurement taken at FULL ROW WIDTH with
-- 12-register bursts. A 4-pixel span cannot burst at all. Everything session
-- 8 wants to do downstream optimises over the mode set this number decides, so
-- it goes first.
--
-- Method: v5 in tools/bench/blit.s walks a stream of per-row span records and
-- copies each span into GVRAM. tools/bench/prep_spans.py emits one stream per
-- span length, every one covering the same whole frame, so the work differs
-- only in how finely it is cut. Regressing
-- cycles = A*spans + B*pixels
-- over the set reads A (the per-span overhead) and B (the per-pixel cost)
-- straight off, and every config also draws a verifiable picture: the frame is
-- cleared before each run and snapshotted after, so a config that timed fast
-- by not writing pixels fails tools/bench/verify_frame256.py.
--
-- MEASUREMENT SCOPE, unchanged from blit.lua: MAME's gvram_w carries no timing,
-- so these are 68000 instruction cycles against zero-wait-state memory -- a
-- LOWER BOUND on real hardware. Interrupts are masked (SR=$2700).
M = manager.machine
SP = M.devices[":maincpu"].spaces["program"]
local function findfile(n)
for _,p in ipairs{"../tools/bench/"..n, "tools/bench/"..n, n} do
local f = io.open(p,"rb"); if f then f:close(); return p end
end
error(n.." not found")
end
local MODE = loadfile(findfile("crtc_mode.lua"))()
local SPEC = loadfile("spans_meta.lua")()
local FLAG, VAR, ITER, SPTR = 0x18000, 0x18004, 0x18008, 0x1800C
local STREAM = 0x90000
local GVRAM, GPAL = 0xC00000, 0xE82000
local CPUHZ = 10000000 -- x68k.cpp:1133, 40_MHz_XTAL/4
local FRAME12 = CPUHZ / 12
local code do local f=io.open("blit.bin","rb"); code=f:read("a"); f:close() end
local blob do local f=io.open("spans.bin","rb"); blob=f:read("a"); f:close() end
local frame do local f=io.open("frame256.bin","rb"); frame=f:read("a"); f:close() end
local function B(i) return string.byte(frame,i) end
local W, H = B(5)*256+B(6), B(7)*256+B(8)
local PAL0 = 9
local YOFF = (MODE.height - H) // 2
-- Identical packing to blit.lua / show_frame256.lua: shared LSB I per entry.
local function pal6(v) return ((v<<2)|(v>>4)) & 0xff end
local function pack(r,g,b)
local f = {r>>3, g>>3, b>>3}
local best, bestI = nil, 1
for I = 0,1 do
local e = 0
for c = 1,3 do
local want = ({r,g,b})[c]
local d = pal6((f[c]<<1)|I) - want
e = e + d*d
end
if best == nil or e < best then best, bestI = e, I end
end
return (f[2]<<11)|(f[1]<<6)|(f[3]<<1)|bestI
end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local function P(s) print("[SPAN] "..s) end
-- 148 KB one byte at a time is 148k Lua->C calls; longwords cut that by four.
local function push(addr, s, from, len)
local i, n = from, len
while n >= 4 do
SP:write_u32(addr, (string.unpack(">I4", s, i)))
addr, i, n = addr+4, i+4, n-4
end
while n > 0 do
SP:write_u8(addr, string.byte(s,i)); addr, i, n = addr+1, i+1, n-1
end
end
local function clear_picture() -- so a config that writes nothing is caught
for y = YOFF, YOFF+H-1 do
local base = GVRAM + y*1024
for x = 0, MODE.width-1, 2 do SP:write_u32(base + x*2, 0) end
end
end
local function setup()
MODE.apply(SP)
for y = 0, MODE.height-1 do
local base = GVRAM + y*1024
for x = 0, MODE.width-1, 2 do SP:write_u32(base + x*2, 0) end
end
for c = 0, 255 do
local o = PAL0 + c*3
SP:write_u16(GPAL + c*2, pack(B(o), B(o+1), B(o+2)))
end
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
P(string.format("loaded blit.bin=%d B, %d span configs, picture %dx%d at yoff=%d",
#code, #SPEC.configs, W, H, YOFF))
end
local function launch(cfg)
push(STREAM, blob, cfg.off+1, cfg.len)
clear_picture()
-- ~4 emulated seconds per config: 1/55.46 s granularity costs under 0.5%.
local est = cfg.nspans*(cfg.var == 5 and 60 or (cfg.var == 6 and 50 or 70))
+ cfg.npix*10
cfg.iter = math.max(4, math.floor(4*CPUHZ/est))
SP:write_u32(FLAG, 0)
SP:write_u32(VAR, cfg.var)
SP:write_u32(ITER, cfg.iter)
SP:write_u32(SPTR, STREAM)
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
end
local results = {}
local function report(cfg, dt)
local cyc = dt * CPUHZ / cfg.iter
results[#results+1] = {cfg=cfg, cyc=cyc}
P(string.format("v%d span %4s px: %5d spans %6d px %d iter in %.4f s -> %8.0f cyc/frame"
.." %5.2f cyc/px %5.1f%% of a 12fps frame",
cfg.var, cfg.name, cfg.nspans, cfg.npix, cfg.iter, dt, cyc,
cyc/cfg.npix, 100*cyc/FRAME12))
end
-- Ordinary least squares on cycles = A*spans + B*pixels, no intercept: the
-- 192 row headers and the outer loop are the only work not attributable to a
-- span or a pixel, and at ~10 cycles a row they are 0.2% of the smallest run.
local function fit(rs)
local ss,sp,pp,sy,py = 0,0,0,0,0
for _,r in ipairs(rs) do
local s,p,y = r.cfg.nspans, r.cfg.npix, r.cyc
ss=ss+s*s; sp=sp+s*p; pp=pp+p*p; sy=sy+s*y; py=py+p*y
end
local det = ss*pp - sp*sp
return (sy*pp - py*sp)/det, (ss*py - sp*sy)/det
end
-- v7 has two per-pixel costs -- the 24-pixel coarse chain and the 2-pixel fine
-- chain -- so its fit is cycles = A*spans + Bc*coarse_px + Bf*fine_px, solved
-- by plain Gaussian elimination on the 3x3 normal equations. prep_spans.py
-- picks span lengths so every fine remainder a real span can have (0,4,..,20)
-- appears, which is what makes the three terms separable.
local function fit3(rs)
local M3 = {{0,0,0,0},{0,0,0,0},{0,0,0,0}}
for _,r in ipairs(rs) do
local x = {r.cfg.nspans, r.cfg.cpx, r.cfg.fpx}
for i=1,3 do
for j=1,3 do M3[i][j] = M3[i][j] + x[i]*x[j] end
M3[i][4] = M3[i][4] + x[i]*r.cyc
end
end
for c=1,3 do
local piv = c
for r=c+1,3 do if math.abs(M3[r][c]) > math.abs(M3[piv][c]) then piv=r end end
M3[c], M3[piv] = M3[piv], M3[c]
for r=1,3 do
if r ~= c then
local f = M3[r][c]/M3[c][c]
for k=c,4 do M3[r][k] = M3[r][k] - f*M3[c][k] end
end
end
end
return M3[1][4]/M3[1][1], M3[2][4]/M3[2][2], M3[3][4]/M3[3][3]
end
local step, st, t0 = 0, "boot", nil
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
local t = T()
if st == "boot" then
if t < 3.0 then return end
setup(); step = 1; launch(SPEC.configs[1]); st, t0 = "running", nil; return
end
if st == "running" then
local fl = SP:read_u32(FLAG)
if fl == 1 and not t0 then t0 = t; return end
if fl == 0xFF then
report(SPEC.configs[step], t - (t0 or t))
st = "snap"; return
end
if t > 900 then P("TIMEOUT flag="..string.format("%08X",fl)); M:exit() end
return
end
if st == "snap" then
M.video:snapshot() -- verified by tools/bench/span.sh
step = step + 1
if SPEC.configs[step] then
launch(SPEC.configs[step]); st, t0 = "running", nil
else
st = "finish"
end
return
end
if st == "finish" then
P("---- measured (instruction cycles only; real GVRAM adds wait states) ----")
for _,v in ipairs{5,6,7} do
local sub = {}
for _,r in ipairs(results) do if r.cfg.var == v then sub[#sub+1] = r end end
-- v5's fit is over its BURSTING configs only (span length a multiple of
-- the 16-pixel burst). Mixing the remainder-path configs in would hide
-- the two costs behind one bad line; they are reported against the fit
-- instead, which is where the remainder shows up as error.
local fitset = {}
for _,r in ipairs(sub) do
if v ~= 5 or r.cfg.p % 16 == 0 then fitset[#fitset+1] = r end
end
local A, Bp, Bf
if v == 7 then
A, Bp, Bf = fit3(fitset)
P(string.format("-- v7: cycles = %.1f per span + %.3f per COARSE pixel"
.." + %.3f per FINE pixel (fitted on %d of %d configs)",
A, Bp, Bf, #fitset, #sub))
else
A, Bp = fit(fitset)
Bf = Bp
P(string.format("-- v%d: cycles = %.1f per span + %.3f per pixel"
.." (fitted on %d of %d configs)", v, A, Bp, #fitset, #sub))
end
for _,r in ipairs(sub) do
local model = A*r.cfg.nspans + Bp*r.cfg.cpx + Bf*r.cfg.fpx
P(string.format(" span %4s px %8.0f cyc %5.2f cyc/px %6.1f cyc/span"
.." vs fit %+6.1f%%", r.cfg.name, r.cyc,
r.cyc/r.cfg.npix, r.cyc/r.cfg.nspans, 100*(model/r.cyc-1)))
end
-- What the mode decision actually needs: a run of L horizontally
-- adjacent 4x4 blocks is 4 spans of 4L pixels, one per pixel row, and
-- v6 pads each to a whole 24-pixel chain unit.
local line = " -> cycles per 4x4 block in a run of L blocks: "
for _,L in ipairs{1,2,4,8,16,64} do
local px, cyc = 4*L, nil
if v == 6 then
px = math.ceil(px/24)*24
cyc = A + px*Bp
elseif v == 7 then
local c = math.floor(px/24)*24
cyc = A + c*Bp + (px-c)*Bf -- a multiple of 4 pads to nothing
else
cyc = A + px*Bp
end
line = line..string.format("L=%d %.0f ", L, 4*cyc/L)
end
P(line.."(V1 is 299.9)")
if v == 5 then
P(" v5's fit only holds where 4L is a whole number of 16-pixel bursts.")
P(" L=1 and L=2 are extrapolations its own measured spans"
.." contradict: 721 and 482.")
end
end
P(" FINDINGS 29 assumed 50.0 per span + 9.080 per pixel, 4 spans per run")
M:exit()
end
end)
if not ok then print("[SPAN] LUA ERROR: "..tostring(err)); M:exit() end
end)
tools/bench/span.sh
Executable
+35
View File
@@ -0,0 +1,35 @@
#!/bin/bash
# Measure the cost of a row-linear literal span on the 68000 (FINDINGS 30).
# ~45 s. Run from the repo root. Needs tmp/frame256.bin (check.sh makes it).
#
# NOT part of check.sh, for the same reason blit.s is not: the output is a wall
# timing, so gating on it would make the green light host-sensitive. What IS
# gated here is correctness -- all 23 configs must draw a pixel-exact frame,
# which is what stops a config timing fast by quietly writing nothing.
set -e
cd "$(dirname "$0")/../.."
[ -f tmp/frame256.bin ] || { echo "need tmp/frame256.bin -- run tools/bench/check.sh"; exit 2; }
python3 tools/bench/prep_spans.py
tools/vasm/vasmm68k_mot -Fbin -o tmp/blit.bin tools/bench/blit.s > /dev/null
mkdir -p tmp/snap_span
rm -f tmp/snap_span/x68000/*.png
( cd tmp && SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 1800 mame x68000 -bios ipl10 \
-ramsize 2M -video soft -window -sound none -nothrottle -plugins \
-autoboot_script ../tools/bench/span.lua \
-snapshot_directory ./snap_span -snapview native -seconds_to_run 200 \
> span.log 2>&1 )
grep -a "^\[SPAN\]" tmp/span.log
# One snapshot per config, and the expected count comes from the generated
# metadata rather than a literal: adding a config must not silently weaken the
# assertion that every one of them drew the picture.
want=$(grep -c '{var=' tmp/spans_meta.lua)
n=0
for f in tmp/snap_span/x68000/*.png; do
python3 tools/bench/verify_frame256.py "$f" > /dev/null || {
echo "FAIL: $f is not pixel-exact"; python3 tools/bench/verify_frame256.py "$f"; exit 1; }
n=$((n+1))
done
[ "$n" -eq "$want" ] || { echo "FAIL: $n snapshots, expected $want"; exit 1; }
echo "OK $n/$want span configs drew a pixel-exact frame"
tools/bench/stream.lua
+487
View File
@@ -0,0 +1,487 @@
-- Drive src/player/stream.s: decode a whole window through a BOUNDED RING,
-- with a modelled SCSI pipe as the producer. STATUS item 3, FINDINGS 49.
--
-- tools/bench/decode.lua preloads the entire container into emulated RAM and
-- lets the 68000 walk a0 through all of it. That gate is pixel-exact over 120
-- frames (FINDINGS 45) and tests nothing about delivery -- 45.4.1 says so in as
-- many words. It is also RAM-bound for a reason that has nothing to do with the
-- player: 5,261,814 B of stream needs a 6 MB machine.
--
-- Here the container lives in a HOST file (tools/bench/prep_stream.py's disk
-- image) and this script plays the part of the MB89352 plus a DMAC channel:
-- it delivers bytes at a modelled rate into a ring of DLX_RING_KB, and the
-- 68000 decodes out of that ring and nothing else. The emulated machine holds
-- ~256 KB of stream instead of 5 MB, so a STOCK 2 MB machine runs the whole
-- window -- the RAM ceiling of FINDINGS 44.6.4/45 is a property of the old rig
-- and this one does not have it.
--
-- WHAT IS BEING TESTED, precisely: that the block loop and the span chain --
-- which read with a monotonically increasing a0 and no bounds check anywhere --
-- stay pixel-exact when a0 is inside a ring one twentieth the size of the
-- stream, and when the address it is handed jumps backwards to the ring base
-- roughly every sixth frame. A green run is not "the decoder still works"; it
-- is "the wrap policy in src/player/stream.s does not corrupt a single pixel of
-- a temporally recursive 120-frame decode".
--
-- THE WRAP POLICY IS `aligned` (tools/analysis/19_ring_stream.py): never start a
-- record that will not fit before the end of the ring; leave the hole and
-- restart at the base. The alternative, letting records wrap and mirroring the
-- ring head into a shadow, costs 5.57% of the frame budget forever against this
-- one's 9.1% of a buffer, and the decoder is already at 91.1% of budget at p90.
-- (Those two are s14_d5_all1500's; the gate container this rig usually runs
-- makes it 3.64% of the budget against 5.7% of the ring. Per container.)
--
-- MEASUREMENT SCOPE, unchanged from decode.lua: MAME's gvram_w/gvram_r carry no
-- timing, so decode times here are pure 68000 instruction cycles against
-- zero-wait-state memory -- a LOWER BOUND. The pipe, likewise, is a MODEL: a
-- constant byte rate on the emulated clock, not a simulation of the MB89352.
-- What it is honest about is ARRIVAL ORDER and RESIDENCY, which is what the
-- ring exists to manage; it says nothing about the clocks the DMAC steals from
-- the 68000 while it does it. That debit is FINDINGS 43.2's and is not modelled
-- here -- so a zero-stall result from this rig means "the bytes were in time",
-- NOT "the frame fits".
--
-- Env:
-- DLX_RING_KB ring size in KB (default 256, FINDINGS 21's)
-- DLX_STREAM_KBPS modelled pipe, KB/s REQUIRED -- no default.
-- 0 = unlimited, which isolates the WRAP question from the
-- DELIVERY one and is what the green light uses.
-- There is deliberately no default rate: this project has
-- never measured the delivery pipe, and the figure that used
-- to be defaulted to here was a user-supplied "4 Mbps" with
-- no provenance that was never a bus measurement at all
-- (FINDINGS 42.1). A default is how a folklore number ends up
-- silently underneath a table nobody restates it in.
-- DLX_PREFILL_KB bytes to deliver before releasing the CPU (default 0)
-- DLX_PACE 1 = hold the decoder to META.fps (default 0 = free-run)
-- DLX_CUT_AT frame tick at which the pipe stops dead (a seek). Needs
-- DLX_PACE; unset = no cut.
-- DLX_CUT_FR how many frame times the cut lasts (default 1)
-- DLX_SLACK_CSV write the per-tick lookahead series to this path
-- DLX_SNAP_EVERY 1 = snapshot every frame tick (needs DLX_PACE). For
-- recording the player; not used by check.sh.
--
-- PACING, AND WHY THE UNPACED RIG COULD NOT ANSWER THE BUFFERING QUESTION
-- (FINDINGS 49.7.2). Free-running, src/player/stream.s asks for record i the
-- instant it finishes record i-1. It therefore outruns any finite pipe, the
-- ring never backs up, `overlaps` never refuses a placement, and every ring size
-- down to 48 KB passes while holding ONE record. That sweep tests wrap
-- correctness, which is real, and says nothing about buffering, which is what a
-- branch point needs. With DLX_PACE=1 the producer supplies a frame clock and
-- the decoder may not start frame i before tick i, so the ring fills to its own
-- capacity and FR_HEAD-FR_TAIL becomes the honest number: whole frames the
-- decoder could run on with delivery stopped dead. DLX_CUT_AT/DLX_CUT_FR then
-- stop it dead and check the answer against a real underrun.
M = manager.machine
SP = M.devices[":maincpu"].spaces["program"]
local function findfile(n)
for _,p in ipairs{"../tools/bench/"..n, "tools/bench/"..n, n} do
local f = io.open(p,"rb"); if f then f:close(); return p end
end
error(n.." not found")
end
local MODE = loadfile(findfile("crtc_mode.lua"))()
local META = loadfile("stream_meta.lua")()
local FLAG, ITER, NFR = 0x18000, 0x18008, 0x1800C
local RD_PTR, FR_HEAD, FR_TAIL = 0x18020, 0x18024, 0x18028
local STALLS, SPINS, DESC = 0x1802C, 0x18030, 0x18100
local PACE, PACEON = 0x18034, 0x18038
local DESCN = 64
local CB1, CB4 = 0x20000, 0x22000
local RING = 0x40000
local GVRAM, GPAL = 0xC00000, 0xE82000
local CPUHZ = 10000000
local FRAME12 = CPUHZ / META.fps
local AUDIO_KBPS = 7.8 -- ratectl.AUDIO_KBPS; the pipe carries it too
local RING_KB = tonumber(os.getenv("DLX_RING_KB") or "") or 256
local KBPS = tonumber(os.getenv("DLX_STREAM_KBPS") or "")
if KBPS == nil then
print("[STR] DLX_STREAM_KBPS is not set and has no default. Set it to the "
.."delivery rate you want to model, or to 0 for an unlimited pipe "
.."(which is what tools/bench/check.sh uses -- it gates the WRAP "
.."policy, and an unlimited pipe removes delivery as a variable).")
manager.machine:exit()
return
end
local PREFILL = (tonumber(os.getenv("DLX_PREFILL_KB") or "") or 0) * 1024
local PACED = (os.getenv("DLX_PACE") == "1")
local CUT_AT = tonumber(os.getenv("DLX_CUT_AT") or "")
local CUT_FR = tonumber(os.getenv("DLX_CUT_FR") or "") or 1
local SLACK_CSV = os.getenv("DLX_SLACK_CSV")
-- One snapshot per frame tick instead of one at the end of the run. This is a
-- DOCUMENTATION artefact -- 120 PNGs of a paced player, for a recording -- and
-- it is deliberately not on any path tools/bench/check.sh takes. Needs
-- DLX_PACE: snapshotting a free-running decoder would sample the screen at
-- whatever rate the 68000 happened to finish frames at, which is not a frame
-- rate and would misrepresent the player as faster than it is.
local SNAP_EVERY = (os.getenv("DLX_SNAP_EVERY") == "1")
local RINGSZ = RING_KB * 1024
-- Video's share of the pipe. Debiting audio is not optional bookkeeping: the
-- ADPCM stream comes off the same disk and out of the same budget (FINDINGS 33).
local BPS = (KBPS > 0) and (KBPS - AUDIO_KBPS) * 1024 or math.huge
local code do local f=io.open("stream.bin","rb"); code=f:read("a"); f:close() end
local cb do local f=io.open("stream_cb.bin","rb"); cb=f:read("a"); f:close() end
local DISK = assert(io.open("stream_disk.bin","rb"))
local YOFF = (MODE.height - META.H) // 2
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local function P(s) print("[STR] "..s) end
local function push(addr, s, from, len)
local i, n = from, len
while n >= 4 do
SP:write_u32(addr, (string.unpack(">I4", s, i)))
addr, i, n = addr+4, i+4, n-4
end
while n > 0 do
SP:write_u8(addr, string.byte(s,i)); addr, i, n = addr+1, i+1, n-1
end
end
-- ------------------------------------------------------------ the producer
-- Ring occupancy is tracked as an explicit list of records still owned by the
-- decoder, rather than as a modular write-minus-read distance. Under `aligned`
-- the ring is not a simple modulus -- a wrap leaves a HOLE of arbitrary size --
-- so a distance would have to carry the holes as a correction term and would be
-- the easiest thing in this file to get quietly wrong. Six-ish live records is
-- a short list; an O(n) overlap test on it is exact and obviously exact.
-- arrival[i] = emulated time at which record i became RESIDENT. This, not the
-- decoder's spin counter, is what answers the delivery question. stream.s has
-- no frame clock: it asks for the next record the instant it finishes the last
-- one, so it outruns any finite pipe and "stalled" is what a decoder that is
-- merely EARLY looks like. A shipping player waits for vblank at 12 fps and
-- spends that same time idle. So the honest test is not "did the decoder ever
-- wait" but "was record i resident by its 12 fps deadline", which is a question
-- about arrival times alone and does not need the decoder paced.
local arrival = {}
local live, wcur, nsent = {}, 0, 0
local n_rate, n_ring = 0, 0
local holes, hole_bytes, credit = 0, 0, 0
local last_t, pf_t0, delivered = nil, nil, 0
local function overlaps(off, len)
for _,r in ipairs(live) do
if off < r.off + r.len and r.off < off + len then return true end
end
return false
end
local function reap()
local tail = SP:read_u32(FR_TAIL)
local i = 1
while i <= #live do
if live[i].idx < tail then
-- RD_PTR is the decoder's byte-granular release, and nothing here needs
-- it -- this producer has the index and works in whole records. Checking
-- it makes it a cross-check instead of a field nothing reads: a real
-- producer without an index (a DMAC chasing the CPU) has only this.
--
-- It holds for the LAST record retired in a pass and not for the others.
-- RD_PTR is a single released-to pointer, so it names the end of record
-- tail-1; if several records were consumed since the previous reap -- and
-- under a paced decoder with a stopped pipe, several is normal -- the
-- earlier ones are long overwritten by it. Asserting per record made the
-- check a test of how often reap happened to run.
if live[i].idx == tail - 1 then
local rp = SP:read_u32(RD_PTR)
local want = RING + live[i].off + live[i].len
if rp ~= want then
P(string.format("RD_PTR MISMATCH after frame %d: decoder released "
.."%08X, record ends %08X", live[i].idx, rp, want))
M:exit()
end
end
table.remove(live, i)
else
i = i + 1
end
end
end
-- Cut window: the pipe stops dead, as it does across a seek. Credit is FROZEN
-- rather than left to accumulate -- a drive that is repositioning is not
-- banking bytes it will burst on arrival, and letting credit build would hand
-- the ring back everything the cut took the moment it ended, which is the one
-- way to make a seek look free.
local cut_t0, cut_t1, cut_done = nil, nil, false
local function produce(now)
local dt = now - (last_t or now); last_t = now
local cut = (cut_t0 and now >= cut_t0 and now < cut_t1)
-- A seek stops DELIVERY. It does not stop the decoder, which goes on draining
-- the ring and releasing bytes behind itself, so reap() runs either way --
-- skipping it would have the ring look full for the whole cut and hide the
-- one thing the cut is for.
if not cut then credit = credit + dt * BPS end
reap()
if cut then return end
while nsent < META.nframes do
local rec = META.index[nsent + 1]
-- WHICH RESOURCE REFUSED, counted separately. A producer that stops
-- because it has no credit is RATE-bound and a bigger ring buys nothing; one
-- that stops because the decoder still owns the bytes is RING-bound and a
-- faster pipe buys nothing. The two look identical from the decoder's side
-- -- both are simply "no new record" -- and they have opposite fixes.
if credit < rec.len then n_rate = n_rate + 1; break end
-- `aligned`: refuse to start a record that will not finish inside the ring.
-- The hole is charged only once the record is actually PLACED. Charging it
-- at the point the wrap is decided counts one hole per retry while the
-- decoder still owns the ring base -- which is every tick of a fast pipe --
-- and reported 105 wraps over 120 records where there are 18.
local w, hole = wcur, 0
if w + rec.len > RINGSZ then w, hole = 0, RINGSZ - wcur end
if overlaps(w, rec.len) then n_ring = n_ring + 1; break end -- decoder owns them
if hole > 0 then holes = holes + 1; hole_bytes = hole_bytes + hole end
DISK:seek("set", rec.off)
push(RING + w, DISK:read(rec.len), 1, rec.len)
-- The descriptor MUST be visible before the count that advertises it. Here
-- the CPU is stopped while this runs so the order is academic; on hardware
-- it is not, and stream.s reads them in the opposite order for that reason.
SP:write_u32(DESC + (nsent % DESCN) * 4, RING + w)
live[#live+1] = {idx=nsent, off=w, len=rec.len}
wcur, nsent = w + rec.len, nsent + 1
credit, delivered = credit - rec.len, delivered + rec.len
SP:write_u32(FR_HEAD, nsent)
arrival[nsent] = now -- record nsent-1 is now resident
end
end
-- ------------------------------------------------------------------ setup
local function setup()
MODE.apply(SP)
push(CB1, cb, 1, META.cb1_len)
push(CB4, cb, 1 + META.cb1_len, META.cb4_len)
local palo = 1 + META.cb1_len + META.cb4_len
for c = 0, 255 do
SP:write_u16(GPAL + c*2, (string.unpack(">I2", cb, palo + c*2)))
end
for y = 0, MODE.height-1 do
local base, v = GVRAM + y*1024, 0
if y < YOFF or y >= YOFF+META.H then v = META.dark end
for x = 0, MODE.width-1 do SP:write_u16(base + x*2, v) end
end
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
SP:write_u32(FLAG, 0); SP:write_u32(FR_HEAD, 0); SP:write_u32(FR_TAIL, 0)
SP:write_u32(ITER, 1); SP:write_u32(NFR, META.nframes)
SP:write_u32(PACE, 0); SP:write_u32(PACEON, PACED and 1 or 0)
P(string.format("stream.bin=%d B, codebooks %d+%d B, disk %d B, %d frames",
#code, META.cb1_len, META.cb4_len, META.disk_len, META.nframes))
P(string.format("decoder %s%s", PACED and ("PACED at "..META.fps.." fps")
or "FREE-RUNNING (tests wrap, not buffering -- 49.7.2)",
CUT_AT and string.format(", pipe cut at tick %d for %.2f fr",
CUT_AT, CUT_FR) or ""))
P(string.format("ring %d KB at %06X, pipe %s, prefill %d KB, maxrec %d B",
RING_KB, RING, (KBPS > 0) and (KBPS.." KB/s") or "unlimited",
PREFILL // 1024, META.maxrec))
if META.maxrec > RINGSZ then
P("RING TOO SMALL: one record does not fit. stream.s needs a whole record "
.."contiguous."); M:exit()
end
end
local function launch()
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700 -- supervisor, ALL interrupts masked
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
end
local st, t0, t_rel = "boot", nil, 0
local pace, min_ahead, min_at = -1, math.huge, -1
local sum_ahead, n_ahead = 0, 0
local slack_series = {}
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
local t = T()
if st == "boot" then
if t < 3.0 then return end
setup(); last_t, pf_t0 = t, t; st = "prefill"; return
end
if st == "prefill" then
produce(t)
if delivered >= PREFILL then
P(string.format("prefill done: %.1f KB in %.3f s, releasing the CPU",
delivered/1024, t - pf_t0))
launch(); t_rel = t; st = "running"
end
return
end
if st == "running" then
if PACED then
local tick = math.floor((t - t_rel) * META.fps)
if tick > pace then
pace = tick
SP:write_u32(PACE, pace)
-- Taken BEFORE this tick's frame is decoded, so snapshot n is the
-- finished picture of frame n-1. tick 0 is skipped: nothing has been
-- drawn yet and it would record a black screen as a decoded frame.
if SNAP_EVERY and tick > 0 then M.video:snapshot() end
-- Sampled AT the tick, before this frame is decoded: FR_TAIL is the
-- count of frames already done, FR_HEAD the count resident, so the
-- difference is exactly how many further frames the decoder could
-- draw if the pipe went silent at this instant. Whole records, not
-- bytes -- the contiguity constraint of 49.2 means a partial record
-- buys nothing.
local ahead = SP:read_u32(FR_HEAD) - SP:read_u32(FR_TAIL)
-- Only sampled while the producer still HAS records to place. Once
-- it has sent the last one the lookahead drains to zero for reasons
-- that are about the window ending, not about the ring's capacity or
-- the pipe's rate -- and the minimum over the whole run would then
-- always be the drain tail, which is the one part of it that tells
-- you nothing.
if nsent < META.nframes then
if ahead < min_ahead then min_ahead, min_at = ahead, tick end
sum_ahead, n_ahead = sum_ahead + ahead, n_ahead + 1
slack_series[#slack_series+1] = {tick, ahead, n_ring, n_rate}
end
if CUT_AT and tick >= CUT_AT and not cut_t0 then
cut_t0, cut_t1 = t, t + CUT_FR / META.fps
P(string.format("PIPE CUT at tick %d for %.2f frame times "
.."(%.1f ms), with %d frames resident ahead",
tick, CUT_FR, 1000*CUT_FR/META.fps, ahead))
end
end
end
produce(t)
if cut_t0 and not cut_done and t >= cut_t1 then cut_done = true end
local fl = SP:read_u32(FLAG)
if fl == 1 and not t0 then t0 = t; return end
if fl == 0xEE then
P("BITSTREAM DESYNC -- the decoder consumed the wrong number of bytes.")
P(" Under a ring that is the whole point: it means a record was placed "
.."or described wrongly, not that the codec changed.")
M:exit(); return
end
if fl == 0xE1 then
P("PRODUCER STALLED OUT -- stream.s spun SPINMAX times with no new "
.."record. Delivered "..nsent.."/"..META.nframes..".")
M:exit(); return
end
if fl == 0xFF then
local dt = t - (t0 or t)
local stalls = SP:read_u32(STALLS)
local spins = SP:read_u32(SPINS)
if PACED then
-- Paced, the wall clock measures the PACE, not the decode: the loop
-- spends whatever is left of each slot spinning in `pacewait`. The
-- decode cost of this container is decode.lua's and FINDINGS 45's;
-- printing a cycles/frame here would just report 1/fps back.
P(string.format("decoded %d frames in %.4f s emulated at a %d fps "
.."pace (%.2f s nominal) -- the clock here measures "
.."the PACE, not the decode",
META.nframes, dt, META.fps, META.nframes/META.fps))
else
P(string.format("decoded %d frames in %.4f s emulated -> %.0f cycles/"
.."frame = %.1f%% of a %dfps frame",
META.nframes, dt, dt*CPUHZ/META.nframes,
100*(dt*CPUHZ/META.nframes)/FRAME12, META.fps))
end
P(string.format("ring: %d wraps, %d B of hole (mean %.1f KB, %.1f%% of "
.."the ring)", holes, hole_bytes,
holes > 0 and hole_bytes/holes/1024 or 0,
100*(holes > 0 and hole_bytes/holes or 0)/RINGSZ))
-- The decoder's own spin counter, kept for what it is: evidence that
-- stream.s outran the pipe, NOT evidence of an underrun. See the note
-- on `arrival` above.
if PACED then
-- Paced, a stall is an UNDERRUN: the frame's slot arrived and its
-- record had not. Free-running it is earliness (49.6) and means the
-- opposite thing, so the two are never printed in the same words.
P(string.format("UNDERRUNS: %d/%d frames waited past their %d fps slot "
.."(%d polls)", stalls, META.nframes, META.fps, spins))
-- WHAT THE MINIMUM OVER A RUN IS, AND IS NOT. At release the ring
-- holds only what the prefill put there, so the early ticks report a
-- buffer that has not been built yet, not a ring or a pipe that
-- cannot build it. A seek empties the ring the same way, so that
-- transient is the branch-point case rather than an artefact to be
-- trimmed -- but it has to be told apart from the CEILING, which is
-- what the ring is worth once it is full.
--
-- Which resource stopped the producer says which is which, and only
-- RING refusals count: a rate refusal fires on nearly every tick
-- (credit accrues continuously and records are placed whole), so it
-- marks nothing. A ring refusal means the ring actually filled.
local ceiling, ceil_at = 0, -1
for _,e in ipairs(slack_series) do
if e[2] > ceiling then ceiling, ceil_at = e[2], e[1] end
end
local ss_min, ss_at = math.huge, -1
for _,e in ipairs(slack_series) do
if e[3] > 0 and e[2] < ss_min then ss_min, ss_at = e[2], e[1] end
end
P(string.format("SEEK SLACK: ceiling %d frames (%.0f ms), first "
.."reached at tick %d; mean %.1f over the window",
ceiling, 1000*ceiling/META.fps, ceil_at,
sum_ahead/math.max(1,n_ahead)))
if n_ring > 0 then
P(string.format(" RING-BOUND: the ring filled (%d refusals). Once "
.."full it survives delivery stopped dead for %d "
.."frames = %.0f ms; min after first fill %d "
.."(tick %d)", n_ring, ceiling,
1000*ceiling/META.fps, ss_min, ss_at))
else
P(string.format(" RATE-BOUND: the ring NEVER filled in %d frames. "
.."A bigger ring buys nothing at this pipe; the "
.."buffer is still accumulating when the window "
.."ends.", META.nframes))
end
P(string.format(" BUILD TIME: %d ticks = %.2f s of play to reach the "
.."ceiling from empty -- which is what a branch point "
.."costs before the NEXT seek is affordable",
ceil_at, ceil_at/META.fps))
if SLACK_CSV then
local fh = io.open(SLACK_CSV, "w")
fh:write("tick,ahead,ring_refusals,rate_refusals\n")
for _,e in ipairs(slack_series) do
fh:write(string.format("%d,%d,%d,%d\n", e[1], e[2], e[3], e[4]))
end
fh:close()
P("slack series -> "..SLACK_CSV)
end
else
P(string.format("decoder waited on %d/%d frames (%d polls) -- it is "
.."free-running, so this is earliness, not underrun",
stalls, META.nframes, spins))
end
-- The delivery result. Deadline for record i is release + i/fps.
local misses, worst, prefill_s = 0, 0.0, 0.0
for i = 0, META.nframes-1 do
local a = arrival[i+1]
if a then
local late = a - (t_rel + i / META.fps)
if late > 0 then
misses = misses + 1
if late > worst then worst = late end
end
if late > prefill_s then prefill_s = late end
end
end
P(string.format("DEADLINE at %d fps: %d/%d records late, worst by "
.."%.1f ms (%.2f frame times)", META.fps, misses,
META.nframes, worst*1000, worst*META.fps))
-- The smallest start delay that makes every deadline: FINDINGS 21's
-- "required prefill", measured on the emulated clock through the real
-- ring rather than simulated from record sizes.
P(string.format("REQUIRED PREFILL: %.1f ms = %.2f frame times = %.1f KB "
.."at %s", prefill_s*1000, prefill_s*META.fps,
(KBPS > 0) and (prefill_s * BPS / 1024) or 0.0,
(KBPS > 0) and (KBPS.." KB/s") or "an unlimited pipe"))
M.video:snapshot()
P("snapshot taken after the sequential pass -- last frame, decoded "
.."entirely out of a "..RING_KB.." KB ring")
st = "snapped"; return
end
if t > 900 then P("TIMEOUT flag="..string.format("%08X",fl)); M:exit() end
return
end
if st == "snapped" then M:exit(); return end
end)
if not ok then print("[STR] LUA ERROR: "..tostring(err)); M:exit() end
end)
tools/bench/verify_decode.py
+12 -3
View File
@@ -22,11 +22,20 @@ from dlx import DLX
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--snap", default="tmp/snap_decode")
# The harness can only load as much of a container as fits in the emulated
# machine's RAM, so it may have decoded a PREFIX (tools/bench/prep_dlx.py
# --ram). Compare against the same prefix, or the reference runs ahead of the
# 68000 and reports a mismatch that is an artefact of the rig.
ap.add_argument("--nframes", type=int, default=None,
help="frames the 68000 actually decoded (default: all)")
a = ap.parse_args()
d = DLX(a.container)
NF = a.nframes if a.nframes is not None else d.nframes
if NF > d.nframes:
sys.exit(f"--nframes {NF} exceeds the container's {d.nframes}")
canvas = np.zeros((d.H, d.W), np.uint8)
for f in range(d.nframes):
for f in range(NF):
d.paint(canvas, f)
pal = d.pal.astype(int)
@@ -52,7 +61,7 @@ else:
bad = diff.any(2)
by, bx = np.where(bad)
blocks = sorted(set(zip((by//4).tolist(), (bx//4).tolist())))
fail.append(f"3. frame {d.nframes-1} not pixel-exact: {bad.sum()} px in "
fail.append(f"3. frame {NF-1} not pixel-exact: {bad.sum()} px in "
f"{len(blocks)} blocks differ, maxdiff {diff.max()}; "
f"first block (by={blocks[0][0]}, bx={blocks[0][1]})")
@@ -60,7 +69,7 @@ for x in fail:
print("FAIL " + x)
if fail:
sys.exit(1)
print(f"OK {d.nframes} frames decoded on the 68000, final frame pixel-exact "
print(f"OK {NF} frames decoded on the 68000, final frame pixel-exact "
f"against tools/encoder/dlx.py")
print(f" {d.W}x{d.H}, {d.nb} blocks/frame, k1={d.k1} k4={d.k4}, "
f"all four block modes exercised")
tools/bench/verify_frame256.py
+8 -4
View File
@@ -1,7 +1,9 @@
#!/usr/bin/env python3
"""Regression test for the 256x256 CRTC mode (docs/FINDINGS 23).
Checks tmp/snap256/x68000/0000.png against tmp/frame256.bin:
Checks a native snapshot (default tmp/snap256/x68000/0000.png, override with
argv[1] -- tools/bench/span.lua verifies twelve of them) against
tmp/frame256.bin:
1. native snapshot is 256x512 -- 256 dots, and 512 active scanlines of a
568-line 31.5kHz raster carrying 256 double-scanned graphics rows
2. double-scan pairing is (1,2),(3,4),... -- MAME halves the ABSOLUTE
@@ -16,8 +18,10 @@ import struct, sys
import numpy as np
from PIL import Image
s = np.asarray(Image.open("tmp/snap256/x68000/0000.png").convert("RGB")).astype(int)
d = open("tmp/frame256.bin", "rb").read()
snap = sys.argv[1] if len(sys.argv) > 1 else "tmp/snap256/x68000/0000.png"
blob = sys.argv[2] if len(sys.argv) > 2 else "tmp/frame256.bin"
s = np.asarray(Image.open(snap).convert("RGB")).astype(int)
d = open(blob, "rb").read()
W, H = struct.unpack(">HH", d[4:8])
pal = np.frombuffer(d[8:8+768], np.uint8).reshape(256, 3).astype(int)
idx = np.frombuffer(d[8+768:8+768+W*H], np.uint8).reshape(H, W)
@@ -53,7 +57,7 @@ if fail:
sys.exit(1)
mse = ((act - pal[idx]) ** 2).mean()
print(f"OK 256x512 native, double-scan exact, active {W}x{H} pixel-exact, "
print(f"OK {snap}: 256x512 native, double-scan exact, active {W}x{H} pixel-exact, "
f"letterbox true black")
print(f" palette ceiling vs 24-bit palettised source: "
f"{10*np.log10(255**2/mse):.2f} dB ({(I==0).sum()}/256 entries use I=0)")
tools/encoder/dlx.py
+101 -7
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""Reference DLX1 reader/decoder -- the ground truth the 68000 player is checked against.
"""Reference DLX reader/decoder -- the ground truth the 68000 player is checked against.
This is deliberately a *decoder*, not a re-run of the encoder: it parses the
container byte-for-byte the way `src/player/` must, so that any disagreement
@@ -12,9 +12,19 @@ Everything is big-endian (see the `encode.py` docstring). Block raster order,
V4 sub-block order is (sub_y, sub_x) row-major -- TL, TR, BL, BR -- matching
`vq_hybrid.paint`'s reshape(-1,2,2,2,2).transpose(0,1,3,2,4).
DLX3 adds the v7 LITERAL SPAN section between the mode header and the block
payload (FINDINGS 40, tools/encoder/spans.py). A spanned block reads SKIP in
the mode header and is painted by a span instead, so a reader that ignores the
section does not merely lose the spans -- it displays stale pixels wherever one
was. The section is at a KNOWN offset (header end) rather than behind the
block payload precisely so that the 68000 can paint it before it has parsed
anything of variable length.
"""
import struct
import os, sys, struct
import numpy as np
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import spans as SP
MODE_SKIP, MODE_V1, MODE_V4, MODE_RAW = 0, 1, 2, 3
@@ -23,8 +33,16 @@ class DLX:
def __init__(self, path):
self.raw = open(path, "rb").read()
b = self.raw
if b[:4] != b"DLX1":
raise ValueError(f"{path}: not a DLX1 container")
# DLX2 pads every frame record up to a 4-byte boundary; DLX1 lays them
# end to end. On a 68000 that is not a slow read but an ADDRESS ERROR
# (FINDINGS 28.3), so the padding is part of the format, not a loader
# convenience -- but DLX1 containers stay readable, because every
# measurement in FINDINGS 28-31 was taken on one.
if b[:4] not in (b"DLX1", b"DLX2", b"DLX3"):
raise ValueError(f"{path}: not a DLX container")
self.version = int(b[3:4])
self.aligned = self.version >= 2
self.has_spans = self.version >= 3
(self.W, self.H, self.fps, self.nframes,
self.k1, self.k4) = struct.unpack(">HHHHHH", b[4:16])
off_pal, off_cb1, off_cb4, off_frm = struct.unpack(">IIII", b[16:32])
@@ -41,14 +59,22 @@ class DLX:
self.mode_bytes = (self.nb * 2 + 7) // 8
# frame directory: (offset of the mode header, payload length)
if self.aligned and off_frm % 4:
raise ValueError(f"{path}: DLX2 frame stream starts at {off_frm}, "
f"which is not 4-byte aligned")
self.frames = []
p = off_frm
for _ in range(self.nframes):
(n,) = struct.unpack(">I", b[p:p + 4])
self.frames.append((p + 4, n))
p += 4 + n
if p != len(b):
raise ValueError(f"{path}: {len(b) - p} trailing bytes after "
if self.aligned:
p += -p % 4 # skip the pad to the next record
# The writer does not pad after the LAST record -- nothing follows it --
# so `p` may have advanced past the end by up to 3 bytes there.
slack = len(b) - p
if not (slack == 0 or (self.aligned and -3 <= slack < 0)):
raise ValueError(f"{path}: {slack} trailing bytes after "
f"{self.nframes} frames")
def modes(self, f):
@@ -57,6 +83,60 @@ class DLX:
m = np.stack([(h >> 6) & 3, (h >> 4) & 3, (h >> 2) & 3, h & 3], axis=1)
return m.reshape(-1)[:self.nb].copy()
def spans(self, f):
"""The frame's literal spans: (list of (y, x, pixels), payload offset).
Layout, big-endian, at the end of the mode header (DLX3 only):
u16 nspans
nspans * { u32 GVRAM address, u16 coarse disp, c*48 B pixels,
u16 fine disp, f*4 B pixels }
The displacements are JUMP offsets into the decoder's two unrolled copy
chains, so the pixel counts are read back out of them -- which is the
strongest available check that the encoder and `blit.s` agree about the
chain geometry, because a wrong displacement lands mid-chain and paints
the wrong number of pixels rather than failing loudly.
"""
o, n = self.frames[f]
p = o + self.mode_bytes
if not self.has_spans:
return [], p
b = self.raw
(ns,) = struct.unpack(">H", b[p:p + 2])
p += 2
out = []
for _ in range(ns):
addr, cd = struct.unpack(">IH", b[p:p + 6])
p += 6
c = SP.COARSE_N - cd // SP.COARSE_CODE
if cd % SP.COARSE_CODE or not 0 <= c <= SP.COARSE_N:
raise ValueError(f"frame {f}: coarse displacement {cd} is not "
f"an entry point in an {SP.COARSE_N}-unit chain")
px = list(np.frombuffer(b, ">u2", c * SP.COARSE_PX, p) & 0xFF)
p += c * SP.COARSE_PX * 2
(fd,) = struct.unpack(">H", b[p:p + 2])
p += 2
fu = SP.FINE_N - fd // SP.FINE_CODE
if fd % SP.FINE_CODE or not 0 <= fu <= SP.FINE_N:
raise ValueError(f"frame {f}: fine displacement {fd} is not "
f"an entry point in a {SP.FINE_N}-unit chain")
px += list(np.frombuffer(b, ">u2", fu * SP.FINE_PX, p) & 0xFF)
p += fu * SP.FINE_PX * 2
a = addr - SP.GVRAM
y, x = divmod(a, SP.STRIDE)
y -= SP.YOFF
if x % 2 or not (0 <= y < self.H) or not (0 <= x // 2 < self.W):
raise ValueError(f"frame {f}: span destination {addr:#x} is "
f"not a pixel of the {self.W}x{self.H} picture")
# blit.s tolerates a span running past the visible 256 pixels (the
# line stride is 1024 bytes and only the first 512 are displayed),
# but nothing an encoder emits should need to: a span is a run of
# whole blocks. numpy would truncate it here in silence.
if x // 2 + len(px) > self.W:
raise ValueError(f"frame {f}: span at ({x//2},{y}) of "
f"{len(px)} px overruns the picture width")
out.append((y, x // 2, np.array(px, np.uint8)))
return out, p
def blocks(self, f):
"""Decoded 4x4 palette-index blocks for the non-SKIP blocks of frame f.
@@ -67,7 +147,8 @@ class DLX:
"""
mode = self.modes(f)
o, n = self.frames[f]
p, end = o + self.mode_bytes, o + n
_, p = self.spans(f)
end = o + n
ib, out = self.idx_bytes, {}
b = self.raw
for i, mo in enumerate(mode):
@@ -100,6 +181,19 @@ class DLX:
for i, blk in blks.items():
by, bx = divmod(i, self.nbx)
canvas[by * 4:by * 4 + 4, bx * 4:bx * 4 + 4] = blk
sp, _ = self.spans(f)
for y, x, pix in sp:
# A span paints blocks the mode header calls SKIP. If it ever
# overlaps a coded block the two disagree about the same pixels and
# the 68000's answer depends on which it does last -- so this is a
# format invariant, not a courtesy check.
b0, b1 = x // 4, -(-(x + len(pix)) // 4)
bad = [b for b in range(b0, b1)
if mode[(y // 4) * self.nbx + b] != MODE_SKIP]
if bad:
raise ValueError(f"frame {f}: span at ({x},{y}) covers "
f"non-SKIP block(s) {bad} of block row {y//4}")
canvas[y, x:x + len(pix)] = pix
return mode
def decode_all(self):
tools/encoder/encode.py
+241 -82
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Encode one scene to the DLX bitstream, at a chosen quality profile.
python3 tools/encoder/encode.py <frames_dir> <out.dlx> [--profile sasi|scsi]
python3 tools/encoder/encode.py <frames_dir> <out.dlx> [--profile scsi]
[--lam N] [--fps 12] [--preview out.png]
[--fixed-lam] [--rc-floor profile|open]
@@ -16,7 +16,7 @@ Container (little-endian is WRONG here -- the 68000 is big-endian, so every
multi-byte field is big-endian and the decoder can read it with a plain move.w):
header, 32 bytes
0 'DLX1' magic
0 'DLX2' magic ('DLX1' = the same, unaligned; still read)
4 u16 width, u16 height
8 u16 fps, u16 nframes
12 u16 k1, u16 k4 codebook sizes
@@ -25,11 +25,22 @@ multi-byte field is big-endian and the decoder can read it with a plain move.w):
20 u32 cb1 offset (k1 * 16 bytes of palette indices)
24 u32 cb4 offset (k4 * 4 bytes)
28 u32 frames offset
then, per frame:
then, per frame, each record starting on a 4-BYTE BOUNDARY (0-3 zero pad
bytes before it; a 68000 takes an address error, not a slow read, on an odd
`move.l` -- FINDINGS 28.3):
u32 payload length, then
ceil(nblocks*2/8) bytes of 2-bit mode headers, MSB-first, block raster order
DLX3 only: the v7 LITERAL SPAN section (tools/encoder/spans.py) --
u16 nspans, then per span { u32 GVRAM address, u16 coarse displacement,
c*48 B pixels, u16 fine displacement, f*4 B pixels }
then payloads in block order: V1 -> 1 byte, V4 -> 4 bytes, RAW -> 16 bytes
The span section is between the header and the block payload, not after it,
because the 68000 has to reach it without first parsing something of variable
length: the mode header is a fixed 768 bytes, so the section starts at a known
offset and the block payload starts wherever the span walk finishes. Every
span record is a multiple of 4 bytes long, so nothing inside needs padding.
Codebooks are emitted as palette INDICES, not pixels. The player expands them
once at load time into word-per-pixel form so the blitter can movem them
straight into GVRAM -- k1=1024 costs 1024*16*2 = 32 KB of the 2 MB.
@@ -37,7 +48,7 @@ straight into GVRAM -- k1=1024 costs 1024*16*2 = 32 KB of the 2 MB.
import argparse, struct, sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import vq as VQ, vq_hybrid as H, ratectl as RC
import vq as VQ, vq_hybrid as H, ratectl as RC, spans as SP
# Measured on the emulated 68000, FINDINGS 24. Instruction cycles against
# zero-wait-state memory, so these are floors, not hardware predictions.
@@ -79,70 +90,35 @@ def _idx(v):
_IDX_BYTES = 1
def main():
global _IDX_BYTES
ap = argparse.ArgumentParser()
ap.add_argument("frames_dir"); ap.add_argument("out")
ap.add_argument("--profile", choices=list(RC.PROFILES), default="sasi")
ap.add_argument("--lam", type=float, default=None)
ap.add_argument("--fps", type=int, default=12)
ap.add_argument("--iters", type=int, default=16)
ap.add_argument("--fixed-lam", action="store_true",
help="disable rate control (session 5 behaviour)")
ap.add_argument("--rc-floor", choices=("profile", "open"), default="profile",
help="quality floor for rate control")
ap.add_argument("--bucket-frames", type=int, default=8,
help="leaky-bucket depth, in frame budgets")
ap.add_argument("--prefill", type=float, default=0.0,
help="how full the player's buffer is assumed to be at "
"scene start, as a fraction of the bucket (0 = cold "
"buffer after a seek, the conservative assumption)")
ap.add_argument("--preview")
a = ap.parse_args()
def build_records(m, enc, span_mode):
"""The per-frame records of the container, in order.
prof = RC.PROFILES[a.profile]
lam = a.lam if a.lam is not None else prof["lam"]
k1, k4 = prof["k1"], prof["k4"]
_IDX_BYTES = 1 if max(k1, k4) <= 256 else 2
The encoder hands back the symbols it actually chose. Re-deriving them here
(as session 5 did) is a second chance to disagree with the encoder, and with
per-frame rate control the mode map is no longer reproducible from a single
lam anyway.
# An explicit --lam is a request for that lam, so it implies --fixed-lam.
rc = not (a.fixed_lam or a.lam is not None)
lam_lo = lam if a.rc_floor == "profile" else 1.0
print(f"profile {a.profile}: {prof['desc']}")
if rc:
print(f" target {prof['kbps']} KB/s CEILING, rate-controlled: "
f"lam bisected per frame in [{lam_lo:g}, {RC.LAM_CLIFF:g}], "
f"{a.bucket_frames}-frame bucket")
else:
print(f" target {prof['kbps']} KB/s, FIXED lam={lam} (no rate control)")
print(f" k1={k1} k4={k4}, {_IDX_BYTES}-byte indices")
m = H.build(a.frames_dir, k1=k1, k4=k4, iters=a.iters)
if rc:
enc = RC.encode_rate_controlled(m, prof["kbps"], fps=a.fps,
bucket_frames=a.bucket_frames,
lam_lo=lam_lo, prefill=a.prefill)
else:
enc = H.encode(m, lam=lam)
r = H.evaluate(m, enc, fps=a.fps)
H_, W_ = m["H"], m["W"]; nbx = W_ // 4
pal, idx = m["pal"], m["idx"]
# The encoder hands back the symbols it actually chose. Re-deriving them
# here (as session 5 did) is a second chance to disagree with the encoder,
# and with per-frame rate control the mode map is no longer reproducible
# from a single lam anyway.
frames = []
for f, im in enumerate(idx):
Factored out of main() so tools/analysis/16_span_roundtrip.py can build the
same bytes the shipping encoder does -- a round-trip gate that rebuilt the
records itself would be testing its own copy of the format.
"""
nbx = m["W"] // 4
out = []
for f, im in enumerate(m["idx"]):
mode = enc["modes"][f]
frames.append(pack_modes(mode)
sp = enc.get("spans", [[]] * len(m["idx"]))[f]
rec = (pack_modes(mode)
+ (SP.serialise(sp) if span_mode else b"")
+ frame_payload(mode, enc["l1"][f], enc["l4g"][f], im, nbx))
# the rate controller budgets exactly these bytes -- if that ever drifts
# from the container, every bitrate figure below is fiction
assert len(frames[-1]) == enc["sizes"][f], (f, len(frames[-1]), enc["sizes"][f])
# from the container, every bitrate figure reported is fiction
assert len(rec) == enc["sizes"][f], (f, len(rec), enc["sizes"][f])
out.append(rec)
return out
def write_container(path, m, frames, fps, k1, k4, span_mode):
"""Write the whole container. Returns (total bytes, video bytes, pad)."""
palette = m["pal"][:256]
if len(palette) < 256:
palette = np.vstack([palette, np.zeros((256 - len(palette), 3), np.uint8)])
@@ -154,25 +130,184 @@ def main():
off_cb1 = off_pal + len(pal_b)
off_cb4 = off_cb1 + len(cb1_b)
off_frm = off_cb4 + len(cb4_b)
hdr = (b"DLX1" + struct.pack(">HHHHHH", W_, H_, a.fps, len(idx), k1, k4)
# DLX2: every frame record starts on a 4-byte boundary, including the
# first. Payload lengths are arbitrary, so end-to-end records land on odd
# addresses -- and `move.l (a0)+` at an odd address is an ADDRESS ERROR on
# a 68000, not a slow read. It vectors into the IPL and looks exactly like
# an infinite loop (FINDINGS 28.3). tools/bench/prep_dlx.py has been
# realigning at load time; the container now carries it.
tbl_pad = -off_frm % 4
off_frm += tbl_pad
hdr = ((b"DLX3" if span_mode else b"DLX2")
+ struct.pack(">HHHHHH", m["W"], m["H"], fps, len(frames), k1, k4)
+ struct.pack(">IIII", off_pal, off_cb1, off_cb4, off_frm))
assert len(hdr) == 32, len(hdr)
with open(a.out, "wb") as fh:
frm_pad = 0
with open(path, "wb") as fh:
fh.write(hdr); fh.write(pal_b); fh.write(cb1_b); fh.write(cb4_b)
for p in frames:
fh.write(struct.pack(">I", len(p))); fh.write(p)
fh.write(b"\0" * tbl_pad)
for i, rec in enumerate(frames):
fh.write(struct.pack(">I", len(rec))); fh.write(rec)
if i + 1 < len(frames): # nothing follows the last record
n = -(4 + len(rec)) % 4
fh.write(b"\0" * n); frm_pad += n
total = os.path.getsize(path)
return total, sum(len(r) + 4 for r in frames) + frm_pad, frm_pad
total = os.path.getsize(a.out)
vid = sum(len(p) + 4 for p in frames)
def main():
global _IDX_BYTES
ap = argparse.ArgumentParser()
ap.add_argument("frames_dir"); ap.add_argument("out")
ap.add_argument("--profile", choices=list(RC.PROFILES), default="scsi")
ap.add_argument("--lam", type=float, default=None)
ap.add_argument("--fps", type=int, default=12)
ap.add_argument("--iters", type=int, default=16)
ap.add_argument("--fixed-lam", action="store_true",
help="disable rate control (session 5 behaviour)")
ap.add_argument("--rc-floor", choices=("profile", "open"), default="profile",
help="quality floor for rate control")
ap.add_argument("--bucket-frames", type=int, default=8,
help="leaky-bucket depth, in frame budgets")
ap.add_argument("--kbps", type=float, default=None,
help="override the profile's bitrate CEILING. The profile "
"is a rate point on a delivery medium; this is for "
"asking what the codec does at another one -- e.g. "
"a measured delivery rate. Do not reach for the "
"retired 4 Mbps figure; FINDINGS 42.1.")
ap.add_argument("--span-kbps", type=float, default=None,
help="byte ceiling the SPAN pass may draw on, if it differs "
"from the profile's. The profile is a quality rate "
"point; the pipe is hardware. Bytes between the two "
"buy a better picture if spent on lam and the 68000's "
"deadline if spent on spans -- and nothing at all if "
"left unspent (FINDINGS 41.2). Pass the pipe rate "
"tools/analysis/14_dmac_chain.py is scored against.")
ap.add_argument("--spans", choices=("off", "need", "all"), default="need",
help="v7 literal spans (FINDINGS 40). `need` (default) "
"spends container bytes on spans only where a frame "
"misses the 68000's decode deadline; `all` spends "
"every profitable byte, which is the model "
"14_dmac_chain.py scores; `off` emits DLX2.")
ap.add_argument("--disk-clk-byte", type=float, default=None,
help="clocks the SCSI DMA steals per DELIVERED BYTE, "
"charged against the same frame budget the decoder "
"spends (FINDINGS 43). Default 5.0, the "
"single-address floor; 9.0 is dual-address; 0 "
"restores the pre-43 encoder, which priced a byte at "
"nothing and reported deadlines it could not meet.")
ap.add_argument("--joint-decide", action="store_true",
help="let the per-block lagrangian see the disk debit too, "
"pricing a payload byte at lam+mu*c instead of lam. "
"The default is OFF because it MEASURES as a wash: "
"same 1/120, 0.02 dB worse, and it trades 6,058 "
"clocks of disk for 17,207 of block decode "
"(FINDINGS 44)")
ap.add_argument("--joint-bucket", action="store_true",
help="cap what the leaky bucket may lend a frame at what "
"its clock budget can still absorb, since a borrowed "
"byte is DISK_CLK_BYTE borrowed clocks and there is "
"no double buffer to repay them from (FINDINGS 43.5). "
"Default OFF: measured, it is worth one frame of 120 "
"at --spans need and a 2%% regression at --spans all "
"(FINDINGS 44)")
ap.add_argument("--no-cpu-fit", action="store_true",
help="drop the per-frame 68000 decode ceiling (session 7 "
"behaviour: 31%% of frames on hard content do not fit)")
ap.add_argument("--prefill", type=float, default=0.0,
help="how full the player's buffer is assumed to be at "
"scene start, as a fraction of the bucket (0 = cold "
"buffer after a seek, the conservative assumption)")
ap.add_argument("--preview")
a = ap.parse_args()
prof = dict(RC.PROFILES[a.profile])
if a.kbps is not None:
prof["kbps"] = a.kbps
prof["desc"] = f"{prof['desc']} -- bitrate overridden to {a.kbps:g} KB/s"
lam = a.lam if a.lam is not None else prof["lam"]
k1, k4 = prof["k1"], prof["k4"]
_IDX_BYTES = 1 if max(k1, k4) <= 256 else 2
# An explicit --lam is a request for that lam, so it implies --fixed-lam.
rc = not (a.fixed_lam or a.lam is not None)
lam_lo = lam if a.rc_floor == "profile" else 1.0
# The CPU ceiling is hardware, not taste: without it 31%% of frames on the
# worst sustained window do not decode in time on a stock 68000, and with
# it that is one frame -- the intra frame -- for 0.26 dB. FINDINGS 31.
if a.disk_clk_byte is not None:
RC.DISK_CLK_BYTE = a.disk_clk_byte
RC.JOINT_DECIDE = a.joint_decide
RC.JOINT_BUCKET = a.joint_bucket
cyc_budget = None if a.no_cpu_fit else RC.FRAME_CYCLES
span_mode = None if (a.spans == "off" or not rc) else a.spans
print(f"profile {a.profile}: {prof['desc']}")
if rc:
print(f" target {prof['kbps']} KB/s CEILING, rate-controlled: "
f"lam bisected per frame in [{lam_lo:g}, {RC.LAM_CLIFF:g}], "
f"{a.bucket_frames}-frame bucket")
print(f" CPU ceiling: " + (f"mu bisected per frame against "
f"{RC.FRAME_CYCLES:,.0f} cycles (12fps, stock 68000)"
if cyc_budget else "OFF (--no-cpu-fit)"))
print(f" disk debit: {RC.DISK_CLK_BYTE:g} clocks per delivered byte, "
f"charged INSIDE that ceiling (FINDINGS 43), and "
+ ("SEEN by the mode decision at lam+mu*c (--joint-decide)"
if RC.JOINT_DECIDE else "not seen by the mode decision, "
"which measures as the better container (FINDINGS 44)")
if RC.DISK_CLK_BYTE else
" disk debit: 0 -- bytes priced at nothing (pre-FINDINGS-43)")
else:
print(f" target {prof['kbps']} KB/s, FIXED lam={lam} (no rate control)")
print(f" k1={k1} k4={k4}, {_IDX_BYTES}-byte indices")
m = H.build(a.frames_dir, k1=k1, k4=k4, iters=a.iters)
if rc:
enc = RC.encode_rate_controlled(m, prof["kbps"], fps=a.fps,
bucket_frames=a.bucket_frames,
lam_lo=lam_lo, prefill=a.prefill,
cycle_budget=cyc_budget,
span_mode=span_mode,
span_kbps=a.span_kbps)
else:
# Spans are a rate-control-era mode: `need` has no meaning without a
# per-frame byte allowance to spend, so --fixed-lam emits DLX2.
enc = H.encode(m, lam=lam)
r = H.evaluate(m, enc, fps=a.fps)
H_, W_ = m["H"], m["W"]; nbx = W_ // 4
pal, idx = m["pal"], m["idx"]
frames = build_records(m, enc, span_mode)
nspans = sum(len(x) for x in enc.get("spans", []))
total, vid, frm_pad = write_container(a.out, m, frames, a.fps, k1, k4,
span_mode)
print(f" wrote {a.out}: {total} B "
f"(header+tables {total-vid} B, video {vid} B)")
print(f" DLX2 4-byte record alignment: {frm_pad} B over {len(frames)} frames "
f"({frm_pad/len(frames):.2f} B/frame = {frm_pad/len(frames)*a.fps:.0f} B/s)")
print(f" {vid/len(idx):.0f} B/frame -> {vid/len(idx)*a.fps/1024:.1f} KB/s video"
f" + {RC.AUDIO_KBPS} KB/s audio = {vid/len(idx)*a.fps/1024+RC.AUDIO_KBPS:.1f} KB/s")
print(f" PSNR {r['psnr']:.2f} dB palette ceiling {r['pal']:.2f} dB "
f"loss {r['loss']:.2f} dB")
print(f" modes: SKIP {r['skip']:.1f}% V1 {r['v1']:.1f}% "
f"V4 {r['v4']:.1f}% RAW {r['raw']:.1f}%")
if span_mode:
spf = np.array([len(x) for x in enc["spans"]])
spb = np.array([SP.section_bytes(x) for x in enc["spans"]])
# a run of L blocks is four spans of 4L pixels, so a block is 16 span
# pixels -- not 4, which would count each block four times over
blk = np.array([sum(len(p) for _, _, p in x) // 16 for x in enc["spans"]])
print(f" v7 spans ({span_mode}): {nspans:,} over {len(idx)} frames, "
f"median {np.median(spf):.0f}/frame, max {spf.max()}/frame; "
f"{100*np.mean(spb)/np.mean([len(p) for p in frames]):.1f}% of the "
f"container")
print(f" frames with any span: {int((spf>0).sum())}/{len(idx)}; "
f"blocks painted by one: median {np.median(blk):.0f}, "
f"max {blk.max()} of {m['nb']} "
f"({100*blk.max()/m['nb']:.1f}%)")
if rc:
rr = RC.summarise(m, enc, prof["kbps"], fps=a.fps)
lm = enc["lam"]
@@ -186,23 +321,47 @@ def main():
print(f" frames that could not fit even at the lam={RC.LAM_CLIFF:g} "
f"cliff: {rr['overrun']}/{len(lm)}")
# PER-FRAME non-SKIP distribution. The mean above cannot answer the
# decoder-architecture question (FINDINGS 24.5): decode-direct-to-GVRAM
# costs 76.6% of a 12fps frame budget x (non-SKIP fraction), while
# compose-in-RAM-then-blit is a flat 53.6% regardless. They cross at 70%,
# and that is a decision taken FRAME BY FRAME -- a scene cut is ~100%
# non-SKIP and a held frame near 0%, so their mean describes no real frame.
# PER-FRAME DECODE COST, from the measured per-mode block costs
# (FINDINGS 28.2, vq_hybrid.cycles). The mean cannot answer this: a scene
# cut is ~100% non-SKIP and a held frame near 0%, so their mean describes
# no real frame. What matters is how many frames MISS, and by how much.
#
# This replaces the per-frame blit-vs-direct path choice that used to be
# printed here. That plan is withdrawn -- mixing the two paths displays
# stale pixels on 70 of 120 frames, and there was never a crossover to
# begin with, because the compose path pays the blit ON TOP of decoding.
# FINDINGS 28.1/28.4. The player has one path and no reference frame.
ns = np.array([100 * (mm != 0).mean() for mm in enc["modes"]])
over = int((ns > CROSSOVER_PCT).sum())
# enc["cycles"] already carries the span PAINTING clocks; H.cycles() sees
# only the mode map, in which a spanned block reads SKIP, so re-deriving
# here would report a frame as fitting on the strength of work the encoder
# moved into the span section rather than removed.
cyc = (np.asarray(enc["cycles"]) if "cycles" in enc
else np.array([H.cycles(mm) for mm in enc["modes"]]))
# The frame's real cost is decode PLUS the bus the SCSI DMA steals to
# deliver it. Reporting only `cyc` is what let session 13 print 0/120 for
# a container no machine could have played (FINDINGS 43).
disk = np.asarray(enc["sizes"], float) * RC.DISK_CLK_BYTE
pct = 100 * (cyc + disk) / RC.FRAME_CYCLES
miss = int((pct > 100).sum())
print(f" non-SKIP blocks/frame: median {np.median(ns):.1f}% "
f"p90 {np.percentile(ns, 90):.1f}% max {ns.max():.1f}%")
print(f" frames above the {CROSSOVER_PCT:.0f}% blit crossover: "
f"{over}/{len(ns)} ({100*over/len(ns):.1f}%) -> "
f"{'compose+blit wins on those' if over else 'direct-to-GVRAM wins throughout'}")
cost = np.minimum(BLIT_PCT, DIRECT_PCT * ns / 100)
print(f" display cost if the player picks the cheaper path per frame: "
f"median {np.median(cost):.1f}% p90 {np.percentile(cost, 90):.1f}% "
f"max {cost.max():.1f}% of a 12fps frame")
dpct = 100 * disk / RC.FRAME_CYCLES
print(f" disk debit: median {np.median(dpct):.1f}% "
f"p90 {np.percentile(dpct, 90):.1f}% max {dpct.max():.1f}% "
f"of the frame budget, at {RC.DISK_CLK_BYTE:g} clocks/byte")
print(f" decode+disk: median {np.median(pct):.1f}% "
f"p90 {np.percentile(pct, 90):.1f}% max {pct.max():.1f}% "
f"of a {a.fps}fps frame")
print(f" frames that do NOT decode in time: {miss}/{len(pct)} "
f"({100*miss/len(pct):.0f}%)"
+ (f" -- worst {pct.max():.1f}%" if miss else ""))
if rc and cyc_budget:
rr2 = RC.summarise(m, enc, prof["kbps"], fps=a.fps)
print(f" mu: median {rr2['mu_med']:.4f} max {rr2['mu_max']:.3f} "
f"frames needing any mu at all: {int((enc['mu'] > 0).sum())}/{len(pct)}")
print(f" frames that cannot fit even at mu={RC.MU_CLIFF:g} "
f"(emitted late on purpose): {rr2['late']}")
if a.preview:
from PIL import Image
tools/encoder/extract.py
+7 -2
View File
@@ -4,9 +4,14 @@
Source is 1920x1080 (16:9). The arcade original is 4:3, so we CENTER-CROP to
1440x1080 by default -- see docs/STATUS.md open question on framing.
"""
import subprocess, sys, os, shutil
import subprocess, sys, os, shutil, getpass
STREAM_DIR = "/media/reala-misaki/BDROM/BDMV/STREAM"
# Where the Dragon's Lair Blu-ray is mounted. Nothing in this repo ships the
# media -- bring your own disc, loop-mount it read-only, and point DLX_BDROM at
# the mount if it is not where udisks puts it:
# udisksctl loop-setup -r -f DRAGONS_LAIR.iso
BDROM = os.environ.get("DLX_BDROM") or f"/media/{getpass.getuser()}/BDROM"
STREAM_DIR = f"{BDROM}/BDMV/STREAM"
W, H = 256, 192
def duration(path):
tools/encoder/profile_gen.py
+2 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Derive quality profiles FROM a measured bandwidth, instead of guessing lam.
python3 tools/encoder/profile_gen.py --bw-kbps 488 --name scsi
python3 tools/encoder/profile_gen.py --bw-kbps <KB/s> --name scsi
python3 tools/encoder/profile_gen.py --bw-mbps 4 # same thing
Session 2 set the profile bitrates by eye off the rate-distortion knee, which
@@ -18,7 +18,7 @@ Three things eat the pipe before video gets any:
passes with ZERO required prefill. Peak sizing is kept
behind --size-for-peak only as a pessimistic bound.
3. DMA CYCLE-STEAL -- the HD63450 steals ~8 clocks per 16-bit word from the
68000. At 488 KB/s that is 20% of the CPU, on top of the
68000. At ~500 KB/s that is ~20% of the CPU, on top of the
blit. Bandwidth and CPU are NOT independent budgets.
FINDINGS 5 said streaming "costs essentially no CPU";
that is wrong -- cycle-stealing DMA is not free DMA.
tools/encoder/ratectl.py
+262 -19
View File
@@ -21,6 +21,7 @@ fixed by tuning. Regression test: tools/analysis/09_ratectl_drift.py.
"""
import numpy as np
import vq_hybrid as H
import spans as SP
# Profiles. Bandwidths are the sustained-read figures the player can rely on;
# see docs/FINDINGS.md 5 -- these are FOLKLORE-grade until the disk benchmark
@@ -49,18 +50,25 @@ import vq_hybrid as H
# ~216% of the frame budget on a 68000 and even LZ4 is ~54%. See FINDINGS 17.
# The rates below are therefore RAW payload, no entropy coding.
#
# The two profiles are the SAME codec, decoder and bitstream -- only `lam` differs.
# `lam` here is a FLOOR, not a setting: encode.py rate-controls by default and
# bisects lam per frame in [lam, LAM_CLIFF] to keep under `kbps`. The floor is
# what a quiet frame is allowed to spend, so rate control can only ever spend
# less than session 5's fixed-lam encoder did. FINDINGS 27.
#
# THE `sasi` PROFILE IS GONE (session 9, USER DECISION). It was dropped on
# CAPACITY, not bandwidth: a SASI volume on this machine tops out at 40 MB, and
# the 22.8 minutes of unique scene footage on the source Blu-ray is 147 MB even
# at the 110 KB/s the profile targeted -- more than the whole 4-unit SASI
# address space, with nothing left for Human68k or the game. FINDINGS 32.
#
# That leaves ONE profile, which is also the end of the two-quality-mode
# decision of session 2. The 110 KB/s RATE POINT may still return under another
# name: a 1x SCSI CD-ROM sustains ~150 KB/s, below this profile, and CD-ROM is
# the only period medium with the capacity for the span-heavy stream. That is
# deferred to the blocked disk benchmark and the DMA-vs-PIO check (docs/
# BENCHMARK.md, FINDINGS 29.5), because every bandwidth figure here is folklore
# until one of them lands.
PROFILES = {
"sasi": dict(kbps=110, lam=60.0, k1=256, k4=256,
desc="stock 10MHz ACE/EXPERT, SASI",
quality="36.9 dB on 00020 / 29.6 dB on 00146 / 27.2 dB on the "
"Singe window at 109.5 KB/s (session 5's fixed lam "
"gave 27.8 dB there, but at 137.4 KB/s)",
util="~105 KB/s = 35% of the pessimistic 300 KB/s SASI figure"),
"scsi": dict(kbps=280, lam=10.0, k1=256, k4=256,
desc="Super/XVI, or CZ-6BS1 board in a 10MHz machine",
quality="39.4 dB on 00020 / 32.3 dB on 00146 / 29.9 dB on the "
@@ -81,6 +89,79 @@ PROFILES = {
# instead (FINDINGS 26.2). The old ladder ran to lam=2e5, 250x past shippable.
LAM_CLIFF = 800.0
# Ceiling on the CYCLE search. mu prices a cycle in the same units lam prices a
# byte, so the scale that matters is set by their ratio: at a lam floor of 60
# (the retired `sasi` profile's, and the highest this codec has shipped),
# mu=0.2 makes a V1 block's 300 cycles cost what its 1 payload byte costs. MU_CLIFF=100 is three decades past that: a V1 block priced at 30,000
# distortion units.
#
# It does NOT freeze the picture, and that is the point. At MU_CLIFF a block
# only becomes SKIP if holding the previous reconstruction costs less than
# 28,665 units of distortion, so a frame with nothing on screen to hold -- the
# first frame of a stream, or a scene cut -- stays fully coded and comes out at
# the all-V1 floor of 110.6% (FINDINGS 28.5). Such a frame is emitted LATE on
# purpose, exactly as a frame that will not fit at LAM_CLIFF is emitted over
# budget. Freezing a cut to make the deadline would be the worse failure.
MU_CLIFF = 100.0
MU_FLOOR = 1e-4 # bisection is geometric, so lo must be > 0
# The hard per-frame decode budget. NOT a bucket: bytes can be banked in the
# player's ring buffer, but there is no double buffer to decode ahead into, so
# a frame that misses its deadline is simply late. FINDINGS 28.
FRAME_CYCLES = 10_000_000 / 12.0
# What one delivered byte costs the 68000, in clocks of that same budget. The
# SCSI DMA does not overlap with the CPU -- it steals the bus (FINDINGS 38.3) --
# and the MB89352 is an 8-bit port, so the DMAC pays PER BYTE and not per word
# (FINDINGS 43). 5.0 is the floor: single-address device-to-memory, bus held,
# zero drive wait (MC68450 Fig 4-25 sheet 2). Dual-address costs 9.
# 0.0 restores the pre-43 encoder, which priced a byte at nothing.
DISK_CLK_BYTE = 5.0
# Does the MODE DECISION see that debit, or only the fit test above it?
# FINDINGS 43.6 charged the disk inside the rate controller's ceiling but left
# vq_hybrid.decide() ranking modes with a free byte, so `mu` still bought
# cycles by moving V4 -> RAW: 47.8 cycles saved for 12 extra payload bytes,
# which at any c >= 4 clocks/byte is a net LOSS on the very budget mu is
# enforcing. True makes `decide` price a byte at `lam + mu*DISK_CLK_BYTE`.
#
# IT IS OFF, AND THAT IS A MEASUREMENT, NOT AN OVERSIGHT (FINDINGS 44). The
# inconsistency is real and the fix is a wash: on the 120-frame Singe window at
# c=5 it delivers 482.5 KB/s / 29.17 dB against 496.7 / 29.19, the same 1/120
# frames over budget, the same 112.9% worst frame, and 5,389 MORE clocks in the
# mean frame -- it buys 6,058 clocks of disk with 17,207 clocks of block
# decode. Scored at c=4 and c=9 as well, it is marginally the worse container
# at every price. `--joint-decide` turns it on.
JOINT_DECIDE = False
# The leaky bucket lets a quiet frame bank bytes for a busy one, because the
# player's ring buffer can hold them. True when a byte was only a byte. A byte
# is now also DISK_CLK_BYTE clocks of the frame's decode budget, and FINDINGS 28
# established there is no double buffer to decode ahead into -- so a frame that
# borrows bytes from the bucket borrows clocks it cannot repay (FINDINGS 43.5).
# True caps the borrow at what the clock budget can still absorb; the bucket
# then smooths only what is left after the disk is paid. It never caps BELOW
# the frame's own un-banked byte budget: that is rate control's job, not the
# clock budget's.
#
# IT IS OFF, AND THAT IS A MEASUREMENT (FINDINGS 44). At `--spans all`, the
# mode this project now recommends, the byte-side controller is INERT: a
# 32-frame bucket and an 8-frame bucket emit the same container byte for byte,
# and lam never leaves its floor on any of 120 frames. The cap is worth one
# frame of 120 at `--spans need` (2/120 -> 1/120, for +26 KB/s) and is a 2%
# regression at `all` (506.4 KB/s and 89.6% median frame against 496.7 and
# 87.9%), because capping the block payload only moves those bytes into the
# span section, which draws on its own flat pipe. Extending the cap to the
# span section too was measured and is much worse: 273.7 KB/s, 28.88 dB, and
# still 1/120 -- it starves the pass that was buying the deadline.
# `--joint-bucket` turns it on.
JOINT_BUCKET = False
def _byte_clk():
"""The debit the mode decision is allowed to see (0 = the old decision)."""
return DISK_CLK_BYTE if JOINT_DECIDE else 0.0
AUDIO_KBPS = 7.8 # MSM6258 ADPCM 15.6kHz mono -- comes out of the same budget
@@ -89,7 +170,7 @@ def frame_budget(kbps, fps=12, audio=AUDIO_KBPS):
return (kbps - audio) * 1024.0 / fps
def _search_lam(ctx, allow, lam_lo, lam_hi, iters=12):
def _search_lam(ctx, allow, lam_lo, lam_hi, iters=12, mu=0.0):
"""Smallest lam (=> best quality) whose frame fits `allow` bytes.
Payload size is non-increasing in lam -- raising lam can only move a block
@@ -100,17 +181,18 @@ def _search_lam(ctx, allow, lam_lo, lam_hi, iters=12):
not fit: that frame is emitted over budget on purpose. Past the FINDINGS 15
cliff a frame is not rate-controlled, it is destroyed, so a visible overrun
is the better failure (FINDINGS 26.2)."""
mode, sz = H.decide(ctx, lam_lo)
bc = _byte_clk()
mode, sz = H.decide(ctx, lam_lo, mu, bc)
if sz <= allow:
return lam_lo, mode, sz, False
mode_hi, sz_hi = H.decide(ctx, lam_hi)
mode_hi, sz_hi = H.decide(ctx, lam_hi, mu, bc)
if sz_hi > allow:
return lam_hi, mode_hi, sz_hi, True
lo, hi = lam_lo, lam_hi # lo does not fit, hi does
best = (lam_hi, mode_hi, sz_hi)
for _ in range(iters):
mid = float(np.sqrt(lo * hi))
mode_m, sz_m = H.decide(ctx, mid)
mode_m, sz_m = H.decide(ctx, mid, mu, bc)
if sz_m <= allow:
hi = mid; best = (mid, mode_m, sz_m)
else:
@@ -118,9 +200,97 @@ def _search_lam(ctx, allow, lam_lo, lam_hi, iters=12):
return best[0], best[1], best[2], False
def _search_mu(ctx, allow, lam_lo, lam_hi, cyc_budget, iters=10):
"""Smallest mu whose frame fits BOTH budgets: `allow` bytes and
`cyc_budget` 68000 cycles.
Two controllers, one nested inside the other, because the constraints are
not separable. Raising mu moves blocks to cheaper-to-DECODE modes, which
usually also shrinks the frame -- but not always: RAW is 400 cycles against
V4's 448 and 16 bytes against 4, so mu can buy cycles by SPENDING bytes
(FINDINGS 28.8). So every mu step re-runs the lam bisection and the byte
budget is enforced at the mu that is actually chosen.
Cost is scored with H.cycles(), the exact clustered rule, NOT with the
per-block ranking constant the decision uses -- see vq_hybrid's note on
SKIP. The controller therefore converges on what the 68000 will really do.
Monotonicity: at a fixed lam, raising mu can only move a block to a mode
that costs no more cycles, and it can only ADD to a SKIP cluster, so frame
cycles are non-increasing in mu. The nested lam re-search can perturb that
at the margin (a smaller frame permits a smaller lam, which buys quality
back and can cost a few cycles), so the bisection keeps the best FEASIBLE
point it has actually seen rather than trusting the invariant.
Returns (mu, lam, mode, size, cyc, over_bytes, over_cycles)."""
lam, mode, sz, ovr = _search_lam(ctx, allow, lam_lo, lam_hi, mu=0.0)
cyc = H.cycles(mode)
if cyc + DISK_CLK_BYTE * sz <= cyc_budget:
return 0.0, lam, mode, sz, cyc, ovr, False
lam_h, mode_h, sz_h, ovr_h = _search_lam(ctx, allow, lam_lo, lam_hi, mu=MU_CLIFF)
cyc_h = H.cycles(mode_h)
if cyc_h + DISK_CLK_BYTE * sz_h > cyc_budget: # cannot fit even frozen
return MU_CLIFF, lam_h, mode_h, sz_h, cyc_h, ovr_h, True
lo, hi = MU_FLOOR, MU_CLIFF # lo overruns, hi fits
best = (MU_CLIFF, lam_h, mode_h, sz_h, cyc_h, ovr_h)
for _ in range(iters):
mid = float(np.sqrt(lo * hi))
lam_m, mode_m, sz_m, ovr_m = _search_lam(ctx, allow, lam_lo, lam_hi, mu=mid)
cyc_m = H.cycles(mode_m)
if cyc_m + DISK_CLK_BYTE * sz_m <= cyc_budget:
hi = mid; best = (mid, lam_m, mode_m, sz_m, cyc_m, ovr_m)
else:
lo = mid
return (*best, False)
def _fit_spans(m, ctx, mode, sz, room, cyc_budget, span_mode, ib):
"""Buy 68000 cycles with container bytes, by painting runs as v7 spans.
Returns (mode, size, cycles, sel) where `sel` is spans.select()'s result.
ORDER MATTERS, and it is the reason this runs before the mu search rather
than inside it. Both controllers make a frame decode in time, but they pay
for it differently: mu buys cycles with QUALITY (it pushes blocks down to
cheaper modes and ultimately to SKIP), and a span buys them with BYTES --
and it carries literal source pixels, so it *removes* that run's
quantisation error. Spending bytes we already have is strictly better than
spending picture, so spans go first and mu is what is left when the byte
allowance runs out.
`span_mode` is "need" (stop as soon as the frame fits its cycle budget --
the default, and the cheapest way to make the deadline) or "all" (spend
every profitable byte, which is the model tools/analysis/14_dmac_chain.py
scores and costs several times the bitrate for a little more headroom).
`room` is a byte ceiling for the WHOLE frame, and it is not necessarily the
same one the lam search ran under. Those are two different budgets and
conflating them is what made the first measured span encode look like a
regression (FINDINGS 41.2): the profile's bitrate is a chosen quality rate
point, while the pipe is a hardware ceiling, and bytes left between them
buy nothing if they are not spent. Spending them on lam gets a better
picture; spending them on spans gets the deadline. `--span-kbps` picks.
"""
src = m["idx"][ctx["f"]]
room = room - sz - 2 # the u16 span count is always emitted
if room <= 0:
return mode, sz, H.cycles(mode), None
sel = SP.select(mode, src, m["nbx"], m["nby"], room,
need_clocks=(None if span_mode == "all" else cyc_budget),
idx_bytes=ib, disk_clk_byte=DISK_CLK_BYTE, base_bytes=sz)
if not sel["spans"]:
return mode, sz, H.cycles(mode), None
nmode = sel["mode"]
nsz = (H.frame_bytes(nmode, ctx["nb"], ib) + SP.section_bytes(sel["spans"]))
return nmode, nsz, H.cycles(nmode) + sel["clocks"], sel
def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
lam_lo=1.0, lam_hi=LAM_CLIFF, prefill=0.0,
steps=None, verbose=False):
steps=None, verbose=False, cycle_budget=None,
span_mode=None, span_kbps=None):
"""Per-frame lam search under a leaky bucket, driving the encoder ONE FRAME
AT A TIME and feeding back the frame actually emitted.
@@ -149,32 +319,97 @@ def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
109.5 to 116.3 KB/s against a 110 ceiling, and on a 14-frame clip it
disables rate control entirely because the bucket is larger than the clip.
`cycle_budget` adds the SECOND controller (session 8): a hard per-frame
68000 decode ceiling, bisected on `mu` inside the lam search. None (the
default) leaves it off and reproduces session 6 exactly, which is what
keeps tools/analysis/09_ratectl_drift.py comparable. Pass
FRAME_CYCLES for the 12fps stock-68000 budget.
`steps` is accepted and ignored -- there is no ladder any more.
"""
if steps is not None and verbose:
print(" note: `steps` is ignored; lam is now bisected per frame")
budget = frame_budget(target_kbps, fps)
span_budget = None if span_kbps is None else frame_budget(span_kbps, fps)
cap = bucket_frames * budget
bucket = prefill * cap # banked bytes; bounded by the player's buffer both ways
out = dict(recon=[], modes=[], sizes=[], lam=[], l1=[], l4g=[], overrun=[])
out = dict(recon=[], modes=[], sizes=[], lam=[], l1=[], l4g=[], overrun=[],
mu=[], cycles=[], late=[], spans=[])
ib = H.default_idx_bytes(m)
prev = None
for f in range(len(m["idx"])):
ctx = H.frame_ctx(m, f, prev)
allow = budget + bucket
# The span pass may draw on a DIFFERENT ceiling: flat per frame, not
# banked, because it is the delivery pipe rather than a quality target
# and a pipe cannot be saved up. None means "the same allowance the lam
# search had", which is what leaves spans nothing to buy with at a rate
# point the block coder has already spent (FINDINGS 41.2).
hard = None
if (JOINT_BUCKET and cycle_budget is not None and DISK_CLK_BYTE > 0):
# What can this frame's clock budget still absorb, once its own
# block decode is paid? Priced at the mode map the UN-banked budget
# buys, so the cap is a property of the frame's content rather than
# of how full the bucket happens to be.
_, mode0, _, _ = _search_lam(ctx, budget, lam_lo, lam_hi)
hard = (cycle_budget - H.cycles(mode0)) / DISK_CLK_BYTE
allow = min(allow, max(budget, hard))
span_allow = allow if span_budget is None else span_budget
sel = None
lam, mode, sz, ovr = _search_lam(ctx, allow, lam_lo, lam_hi)
rec = H.paint(m, ctx, mode)
bucket = float(np.clip(bucket + budget - sz, -cap, cap))
mu, cyc, late = 0.0, H.cycles(mode), False
if span_mode and (span_mode == "all"
or (cycle_budget is not None
and cyc + DISK_CLK_BYTE * sz > cycle_budget)):
mode_pre = mode
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
cycle_budget, span_mode, ib)
if cycle_budget is not None and cyc + DISK_CLK_BYTE * sz > cycle_budget:
# The byte allowance could not buy the frame's deadline, so fall
# back to the controller that pays in picture -- and then offer
# spans the bytes the smaller mode map just freed.
mu, lam, mode, sz, cyc, ovr, late = _search_mu(
ctx, allow, lam_lo, lam_hi, cycle_budget)
if span_mode:
mode_pre = mode
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
cycle_budget, span_mode, ib)
late = cyc + DISK_CLK_BYTE * sz > cycle_budget
# Paint from the mode map as it was BEFORE spanning. A spanned run's
# blocks read SKIP in the emitted header, but SKIP means "hold the
# previous reconstruction" and on the first frame there is none -- and
# more generally the held pixels would be wrong. The span overwrites
# exactly the run it covers (4 rows x 4L pixels = the blocks), so
# painting the pre-span modes and then laying the spans over them is
# what the 68000 produces, and it is defined on frame 0.
if span_mode and sel is None:
sz += 2 # the u16 span count is in every DLX3 frame record
# What the quality bucket banks is the BLOCK payload. Charging it the
# span bytes too would drive it to its floor on the first spanned frame
# and starve every later frame of quality for a budget the spans were
# never drawing on.
sz_quality = sz if (sel is None or span_budget is None) else sz - sel["bytes"]
rec = H.paint(m, ctx, mode if sel is None else mode_pre)
if sel is not None:
for y, x, pix in sel["spans"]:
rec[y, x:x + len(pix)] = pix
bucket = float(np.clip(bucket + budget - sz_quality, -cap, cap))
out["recon"].append(rec); out["modes"].append(mode)
out["sizes"].append(sz); out["lam"].append(lam); out["overrun"].append(ovr)
out["mu"].append(mu); out["cycles"].append(cyc); out["late"].append(late)
out["l1"].append(ctx["sym"]["l1"]); out["l4g"].append(ctx["sym"]["l4g"])
out["spans"].append([] if sel is None else sel["spans"])
prev = rec
if verbose:
print(f" f{f:04d} lam={lam:8.2f} {sz:7.0f} B "
f"(allow {allow:7.0f}){' OVER' if ovr else ''}")
return dict(recon=out["recon"], modes=out["modes"],
print(f" f{f:04d} lam={lam:8.2f} mu={mu:8.4f} {sz:7.0f} B "
f"(allow {allow:7.0f}) {100*cyc/FRAME_CYCLES:5.1f}% cpu"
f"{' OVER' if ovr else ''}{' LATE' if late else ''}")
return dict(recon=out["recon"], modes=out["modes"], spans=out["spans"],
sizes=np.array(out["sizes"]), lam=np.array(out["lam"]),
l1=out["l1"], l4g=out["l4g"], overrun=np.array(out["overrun"]),
nb=m["nb"], budget=budget, cap=cap)
mu=np.array(out["mu"]), cycles=np.array(out["cycles"]),
late=np.array(out["late"]),
nb=m["nb"], budget=budget, cap=cap, cycle_budget=cycle_budget)
def summarise(m, enc, target_kbps, fps=12):
@@ -192,6 +427,14 @@ def summarise(m, enc, target_kbps, fps=12):
over=100.0 * np.mean(sz > enc.get("budget", np.inf)),
skip=100 * (mo == 0).mean(), v1=100 * (mo == 1).mean(),
v4=100 * (mo == 2).mean(), raw=100 * (mo == 3).mean())
if "cycles" in enc:
cy = np.asarray(enc["cycles"])
d.update(cyc_med=float(np.median(cy)), cyc_max=float(cy.max()),
cyc_p90=float(np.percentile(cy, 90)),
cpu_miss=int((cy > FRAME_CYCLES).sum()),
mu_med=float(np.median(enc["mu"])),
mu_max=float(np.asarray(enc["mu"]).max()),
late=int(np.asarray(enc.get("late", [])).sum()))
if "lam" in enc:
lam = enc["lam"]
d.update(lam_med=float(np.median(lam)), lam_max=float(lam.max()),
tools/encoder/spans.py
+232
View File
@@ -0,0 +1,232 @@
#!/usr/bin/env python3
"""v7 literal spans: geometry, selection, and the bytes that go in the container.
A span is a ROW-LINEAR run of word-expanded literal pixels that the 68000
copies straight from the stream buffer into GVRAM through an unrolled chain of
`movem.l` units, with no address arithmetic, no loop and no remainder logic.
It is the mode FINDINGS 29 derived, FINDINGS 30 measured as v6, and FINDINGS 40
re-measured as v7 -- v6's 24-pixel coarse chain with a 2-pixel fine chain
appended, at
66.0 clocks/span + 9.143/coarse pixel + 9.978/fine pixel (MEASURED)
The span constants live in tools/analysis/buscost.py and the per-block ones in
tools/encoder/vq_hybrid.py; both are imported rather than copied, which is what
kept session 12's correction to C_SKIP_MIXED from having to be made twice.
WHAT A SPAN COVERS. A run of L horizontally adjacent 4x4 blocks inside one
block row, coded as FOUR spans of 4L pixels -- one per picture row. The run's
blocks are marked SKIP in the mode header and the span paints them instead, so
a span costs the mode-map dispatch but not the block body. That is exactly the
accounting tools/analysis/14_dmac_chain.py scores.
WHY THE PADDING IS ZERO. v7's fine unit is one `move.l (a0)+,(a2)+` = 2
pixels, and a span is a run of 4x4 blocks, so its length is always a multiple
of 4 and splits into 24*c + 2*f with nothing left over (FINDINGS 40.3). v6's
24-pixel quantum wasted ~11 pixels a span and was 86% of the DMAC's advantage
over it.
SPANS ARE LITERAL, SO THEY ARE PIXEL-EXACT. A span carries palette indices
straight out of the palettised source, exactly as a RAW block does. Spanning a
run therefore does not just buy cycles, it removes that run's quantisation
error -- which is why the selection below can only improve PSNR, and why the
reconstruction the encoder feeds back to the next frame has to include spans
(a temporally recursive codec drifts otherwise -- FINDINGS 26.1).
"""
import os, sys
import numpy as np
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)),
"..", "analysis"))
import buscost as B
# Display geometry, and it must match tools/bench/crtc_mode.lua: 256-colour
# page, one pixel per WORD of CPU address space, 1024-byte line stride, picture
# in rows 32..223 of a 256-row page. GVRAM is at a fixed $C00000 on every
# X68000, which is what makes an absolute destination address a legitimate
# thing for an encoder to bake into a stream (FINDINGS 30.2).
GVRAM, YOFF, STRIDE = 0xC00000, 32, 1024
# The two chains, and these must match tools/bench/blit.s v7 exactly.
COARSE_PX, COARSE_CODE, COARSE_N = 24, 12, 11
FINE_PX, FINE_CODE, FINE_N = 2, 2, 11
SPAN_HDR = B.V7_SPAN_HDR # {u32 address, u16 coarse disp} + u16 fine
BYTES_PX = 2 # word-expanded, high byte discarded by gvram_w
def split(npix):
"""(coarse units, fine units) for a span of npix pixels. Exact: npix is a
multiple of 4 for any real span, and 4 is a multiple of the 2-pixel fine
quantum, so nothing is padded."""
if npix % FINE_PX:
raise ValueError(f"span of {npix} px is not a multiple of {FINE_PX}")
c, r = divmod(npix, COARSE_PX)
f = r // FINE_PX
if c > COARSE_N or f > FINE_N:
raise ValueError(f"span of {npix} px exceeds the chain "
f"({c} coarse > {COARSE_N} or {f} fine > {FINE_N})")
return c, f
def clocks(npix):
"""68000 clocks to paint one span of npix pixels (MEASURED, FINDINGS 40)."""
c, f = split(npix)
return (B.V7_SPAN_CYC + c * COARSE_PX * B.V7_CPX_CYC
+ f * FINE_PX * B.V7_FPX_CYC)
def run_clocks(L):
"""Clocks for a run of L blocks: four spans of 4L pixels."""
return 4.0 * clocks(4 * L)
def run_bytes(L):
"""Container bytes for a run of L blocks."""
return 4 * (SPAN_HDR + 4 * L * BYTES_PX)
def dest(y, x):
"""Absolute GVRAM address of picture pixel (x, y)."""
return GVRAM + (YOFF + y) * STRIDE + x * 2
def dirty_runs(mode2d, nbx):
"""Maximal runs of horizontally adjacent non-SKIP blocks, per block row."""
for by in range(mode2d.shape[0]):
d = mode2d[by] != 0
i = 0
while i < nbx:
if not d[i]:
i += 1
continue
j = i
while j < nbx and d[j]:
j += 1
yield by, i, j
i = j
# Per-block decode cost, the same measured table vq_hybrid.cycles() uses --
# imported rather than copied, because session 12 corrected one of them and a
# second copy is how a corrected constant stops being corrected everywhere.
import vq_hybrid as _H
C_SKIP_MIXED = _H.C_SKIP_MIXED # a spanned block still pays its dispatch
BLK_CLK = {1: _H.C_V1, 2: _H.C_V4, 3: _H.C_RAW}
BLK_BYT = {1: 1, 2: 4, 3: 16}
def select(mode, src_idx, nbx, nby, byte_room, need_clocks=None,
idx_bytes=1, disk_clk_byte=0.0, base_bytes=0):
"""Choose which runs to paint as spans.
`mode` 1-D mode map, modified nowhere (a new one is returned)
`src_idx` (H, W) palettised source -- what the spans will carry
`byte_room` container bytes the frame may still spend
`need_clocks` stop as soon as the frame's cost is at or below this;
None spends every profitable byte instead (the model
tools/analysis/14_dmac_chain.py scores).
`disk_clk_byte` what a delivered byte costs the 68000 in clocks, because
the SCSI DMA steals the bus from it (FINDINGS 43). 0.0 is
the pre-43 behaviour: bytes are free and a span is judged on
decode clocks alone. At the real value a span's two wire
bytes per pixel are the dominant term and the ranking
inverts on short runs.
`base_bytes` the frame's byte count before any span is added, so the
running total this loop compares against `need_clocks` can
include the disk debit the frame is already carrying.
Ranked by clocks saved per byte spent, which is the same greedy 12 and 14
use. A run is only offered if the span beats the blocks it replaces on
TOTAL clocks -- decode plus the disk debit of the bytes it adds -- which at
`disk_clk_byte`=0 reduces to the cycles-alone test this used before
FINDINGS 43. Selection is otherwise deliberately conservative and the
reported figures are exact rather than greedy: the saving credited here
ignores the extra all-SKIP header bytes spanning tends to create. The
caller recomputes the frame's real cost from the returned mode map.
Returns dict(mode, spanned, spans, bytes, clocks).
"""
m2 = np.asarray(mode).reshape(nby, nbx)
spanned = np.zeros((nby, nbx), bool)
cand = []
for by, i, j in dirty_runs(m2, nbx):
L = j - i
cur_c = sum(BLK_CLK[int(b)] for b in m2[by][i:j])
cur_b = sum(BLK_BYT[int(b)] * (idx_bytes if int(b) != 3 else 1)
for b in m2[by][i:j])
sc = run_clocks(L) + L * C_SKIP_MIXED # the dispatch still happens
db = run_bytes(L) - cur_b
# TOTAL saving: decode clocks won, less the clocks the extra bytes cost
# on the way in. With disk_clk_byte=0 this is (cur_c - sc) exactly.
net = (cur_c - sc) - disk_clk_byte * db
if net <= 0:
continue
cand.append((net / max(db, 1), cur_c - sc, db, by, i, j))
cand.sort(key=lambda s: -s[0])
# `need_clocks` is measured against the frame as it stands, so the loop
# tracks the real running total rather than a delta: a spanned run's blocks
# become SKIP, and four SKIPs sharing a header byte cost 53 cycles instead
# of 4x55, which the greedy's per-run delta does not see.
import vq_hybrid as H
cur = m2.copy()
total_b, total_c = 0.0, 0.0
chosen = []
for _, dc, db, by, i, j in cand:
if (need_clocks is not None
and H.cycles(cur) + total_c
+ disk_clk_byte * (base_bytes + total_b) <= need_clocks):
break
if total_b + db > byte_room:
continue
total_b += db
total_c += run_clocks(j - i)
cur[by][i:j] = 0
spanned[by][i:j] = True
chosen.append((by, i, j))
spans = []
for by, i, j in sorted(chosen):
x, npix = i * 4, (j - i) * 4
for k in range(4):
y = by * 4 + k
spans.append((y, x, src_idx[y, x:x + npix].astype(np.uint8)))
spans.sort()
return dict(mode=cur.reshape(-1), spanned=spanned, spans=spans,
bytes=int(total_b), clocks=float(total_c))
def serialise(spans):
"""The span section of a frame record, exactly as blit.s v7 reads it.
u16 nspans
nspans * { u32 GVRAM address, u16 coarse disp, c*48 B pixels,
u16 fine disp, f*4 B pixels }
The fine displacement sits MID-STREAM rather than in the record because
that is what lets the decoder keep all 12 payload registers: the coarse
chain falls out into `move.w (a0)+,d0 / jmp` with d0 dead payload and a0
already pointing at it (FINDINGS 40.4).
Every field is big-endian and every span record is a multiple of 4 bytes
long (4 + 2 + 48c + 2 + 4f), so the section needs no internal padding.
"""
out = bytearray()
out += len(spans).to_bytes(2, "big")
for y, x, pix in spans:
c, f = split(len(pix))
w = np.zeros((len(pix), 2), np.uint8)
w[:, 1] = pix # high byte discarded by gvram_w
w = w.tobytes()
out += dest(y, x).to_bytes(4, "big")
out += ((COARSE_N - c) * COARSE_CODE).to_bytes(2, "big")
out += w[:c * COARSE_PX * 2]
out += ((FINE_N - f) * FINE_CODE).to_bytes(2, "big")
out += w[c * COARSE_PX * 2:]
return bytes(out)
def section_bytes(spans):
return 2 + sum(SPAN_HDR + len(p) * BYTES_PX for _, _, p in spans)
tools/encoder/vq.py
+45 -6
View File
@@ -15,6 +15,7 @@ entries MUST be legal palette indices -- both quantisation losses compose.
No sklearn on this box; k-means is hand-rolled (chunked, numpy).
"""
import numpy as np, glob, os, sys
from concurrent.futures import ThreadPoolExecutor
from PIL import Image
BW = BH = 4 # block size
@@ -83,14 +84,52 @@ def kmeans(X, k, iters=24, seed=0):
return C, assign(X, C)
def assign(X, C, chunk=8192):
"""Nearest centroid, chunked to bound memory."""
# k-means assignment is ~95% of an encode's wall clock (the rest of the encoder,
# rate control included, is about a second for a 120-frame window), so it is
# worth the three details below. All three are EXACT: the labels are unchanged
# bit for bit, which is what lets the containers this encoder emits stay
# byte-identical across the change.
_ASSIGN_CHUNK = 2048 # measured: see the note in assign()
_ASSIGN_THREADS = min(8, (os.cpu_count() or 1))
def _assign_range(X, CT, Cn, out, lo, hi, chunk):
for i in range(lo, hi, chunk):
j = min(i + chunk, hi)
d = Cn[None, :] - 2.0 * (X[i:j] @ CT) # + |x|^2, constant per row
out[i:j] = np.argmin(d, axis=1)
def assign(X, C, chunk=_ASSIGN_CHUNK, threads=None):
"""Nearest centroid, chunked to bound memory.
Three things make this 5.9x faster than the obvious version, measured on
the 120-frame Singe window (1,474,560 2x2 blocks against k=256), and none
of them changes a label:
* `C.T` is a VIEW, and a non-contiguous right-hand operand makes BLAS
copy it per chunk: 1.83s -> 1.02s just from materialising it once.
* chunk=2048, not 8192. The temporary is (chunk, k) float32 and the win
is cache residency, not memory: 8192 is 1.01s, 32768 is 2.80s.
* the chunk loop is embarrassingly parallel and numpy releases the GIL in
both the matmul and the argmin, so a plain thread pool scales it:
1.02 -> 0.31s on 8 threads. Partitioning by row cannot change an
argmin, so the labels are identical to the serial ones -- asserted in
tools/analysis/09_ratectl_drift.py by the fact that every container
this encoder emits still hashes the same.
"""
Cn = (C ** 2).sum(1)
CT = np.ascontiguousarray(C.T)
out = np.empty(X.shape[0], dtype=np.int32)
for i in range(0, X.shape[0], chunk):
x = X[i:i + chunk]
d = Cn[None, :] - 2.0 * (x @ C.T) # + |x|^2, constant per row
out[i:i + chunk] = np.argmin(d, axis=1)
n = X.shape[0]
nt = _ASSIGN_THREADS if threads is None else threads
if nt <= 1 or n < 4 * chunk:
_assign_range(X, CT, Cn, out, 0, n, chunk)
return out
bnd = [(n * i) // nt for i in range(nt + 1)]
with ThreadPoolExecutor(nt) as ex:
list(ex.map(lambda ab: _assign_range(X, CT, Cn, out, ab[0], ab[1], chunk),
zip(bnd[:-1], bnd[1:])))
return out
tools/encoder/vq_hybrid.py
+97 -10
View File
@@ -42,6 +42,72 @@ LUMA = VQ.LUMA
_HDR_BYTES_PER_BLOCK = 2 / 8.0
RAW_BYTES = 16.0 # literal palette bytes, never indices
# ---------------------------------------------------------------------------
# CYCLE cost of each mode, per block, MEASURED on the 68000 (FINDINGS 28.2,
# tools/bench/decode.lua). This is the other axis: `lam` prices bytes, `mu`
# prices cycles, and the two are not proportional -- V4 is 4x a V1 block in
# bytes and 1.49x in cycles.
#
# SKIP IS NOT A CONSTANT, and it is the one trap in here. A SKIP block costs
# 13.25 cycles when all four blocks sharing its header byte are SKIP (one
# `tst.b` clears the group) and ~45 when it sits in a mixed byte -- so its
# price depends on its NEIGHBOURS, which a per-block lagrangian cannot see.
# The way out is that the two uses do not need the same number:
# * `decide` uses C_SKIP_RANK purely to RANK modes within a block. SKIP is
# the cheapest mode either way, so the choice only scales the incentive:
# the V1-SKIP gap moves 12% between the two candidates.
# * `cycles()` scores a WHOLE frame with the exact clustered rule, and that
# is what the rate controller bisects against. Nothing downstream of the
# mode decision uses the ranking constant.
C_V1, C_V4, C_RAW = 299.9, 448.2, 400.4
C_SKIP_CLUSTERED = 53.0 / 4 # all-SKIP header byte: one tst.b for four
# C_SKIP_MIXED WAS THE ONE CONSTANT HERE THAT HAD NEVER BEEN MEASURED. It was
# 45.0, hand-derived, from session 7 until session 12 measured it -- and it was
# 18% low. Every other figure in this table comes from a synthetic frame of a
# single mode, and there was no such frame for a SKIP in a MIXED byte, because
# a frame of nothing but mixed SKIPs cannot exist: the byte has to hold a coded
# block for the SKIP to be mixed at all.
#
# tools/bench/prep_dlx.py now emits four that bracket it -- (3 SKIP + 1 V1),
# (1 SKIP + 3 V1), (3 SKIP + 1 RAW), (1 SKIP + 3 RAW), each with the header byte
# ROTATED through all four positions so no mode is pinned to the free `lsr`
# slot -- and each pair solves for the SKIP cost and its partner's together:
#
# MAME C68K (the partner solves back to its own anchored
# V1 pair 55.03 56.50 value to 0.2%, which is what says the pair
# RAW pair 55.83 56.50 is measuring the SKIP and not absorbing it)
#
# 55.0 is taken because every other constant here is MAME's; C68K reads V4 and
# RAW 3.2-3.5% higher on pure frames too, which is FINDINGS 37's known table
# spread and not a property of mixed bytes.
#
# It matters more than 10 clocks a block sounds, because a v7 SPAN marks its
# run SKIP: a spanned container is made largely of mixed SKIPs, so this is the
# dominant population in exactly the frames spans are judged on. FINDINGS 41.5.
C_SKIP_MIXED = 55.0 # a SKIP block inside a mixed byte, MEASURED
C_SKIP_RANK = C_SKIP_CLUSTERED # ranking only -- see above
MODE_CYCLES = np.array([C_SKIP_RANK, C_V1, C_V4, C_RAW], dtype=np.float64)
FRAME_CYCLES_12FPS = 10_000_000 / 12.0 # 833,333, x68k.cpp:1133
def cycles(mode):
"""Exact decode cost of one frame's mode map, in 68000 cycles.
Single source of truth: tools/analysis/11_cpu_budget.py imports this, and
it reproduces the four frames timed on the 68000 to within 1 point
(FINDINGS 28.2). Instruction cycles against zero-wait-state memory, so a
LOWER BOUND like every 68000 figure since FINDINGS 24."""
g = np.asarray(mode).reshape(-1, 4) # one header byte = four blocks
allskip = (g == 0).all(1)
c = allskip.sum() * 4 * C_SKIP_CLUSTERED
mm = g[~allskip]
c += (mm == 0).sum() * C_SKIP_MIXED
c += (mm == 1).sum() * C_V1
c += (mm == 2).sum() * C_V4
c += (mm == 3).sum() * C_RAW
return float(c)
def blocks_of(idx, pal, bw, bh):
return VQ.blockify(idx, pal, bw, bh)
@@ -149,17 +215,38 @@ def frame_ctx(m, f, prev, idx_bytes=None):
idx_bytes=default_idx_bytes(m) if idx_bytes is None else idx_bytes)
def decide(ctx, lam):
"""Lagrangian mode decision at one lam. Returns (mode, payload bytes).
def decide(ctx, lam, mu=0.0, byte_clk=0.0):
"""Lagrangian mode decision at one lam and one mu. Returns (mode, bytes).
Cheap by design: no painting, no image-sized work. A lam search calls this
a dozen times per frame and paints once."""
Minimises `distortion + lam*bytes + mu*(decode cycles + byte_clk*bytes)`
per block. `mu=0` is the byte-only decision every session before 8 made.
`byte_clk` is what a DELIVERED byte costs the 68000 in clocks of the same
budget mu is bisected against -- ratectl.DISK_CLK_BYTE, 5.0 on the
single-address row of FINDINGS 43.2. It defaults to 0, which reproduces
sessions 8-14 exactly, and it is the last place in the encoder where a byte
was still free: FINDINGS 43.6 put the disk debit in the rate controller's
FIT TEST, but the decision underneath it still ranked modes as though the
16 bytes of a RAW block cost nothing to deliver.
THAT INVERTS FINDINGS 28.8. RAW is 400.4 cycles against V4's 448.2, so
with a free byte, raising mu buys cycles by moving V4 -> RAW. Priced, a RAW
block costs `400.4 + 16c` and a V4 block `448.2 + 4c`, which cross at
**c = 3.98 clocks/byte** -- and 43.1's floor argument (a 68000 bus cycle is
four clocks and the SPC hands over one byte per cycle) says c >= 4 on any
real machine. So on hardware mu's escape hatch was never there: it was
spending 12 clocks of bus to save 47.8 of CPU.
Cheap by design: no painting, no image-sized work. A search calls this a
dozen times per lam step and paints once."""
ib = ctx["idx_bytes"]
s = ctx["sym"]
cost = np.stack([ctx["eS"],
s["e1"] + lam * (1.0 * ib),
s["e4"] + lam * (4.0 * ib),
np.full(ctx["nb"], lam * RAW_BYTES)])
mc = mu * MODE_CYCLES
lb = lam + mu * byte_clk # what one payload byte costs, both budgets
cost = np.stack([ctx["eS"] + mc[0],
s["e1"] + lb * (1.0 * ib) + mc[1],
s["e4"] + lb * (4.0 * ib) + mc[2],
np.full(ctx["nb"], lb * RAW_BYTES + mc[3])])
mode = np.argmin(cost, axis=0).astype(np.uint8)
return mode, frame_bytes(mode, ctx["nb"], ib)
@@ -193,10 +280,10 @@ def paint(m, ctx, mode):
return from_blocks(ob, nbx, nby)
def encode_frame(m, f, prev, lam, idx_bytes=None):
def encode_frame(m, f, prev, lam, idx_bytes=None, mu=0.0):
"""One frame at one lam against one previous reconstruction."""
ctx = frame_ctx(m, f, prev, idx_bytes)
mode, sz = decide(ctx, lam)
mode, sz = decide(ctx, lam, mu)
return dict(recon=paint(m, ctx, mode), mode=mode, size=sz,
l1=ctx["sym"]["l1"], l4g=ctx["sym"]["l4g"], ctx=ctx)
tools/media/make_readme_media.py
+219
View File
@@ -0,0 +1,219 @@
#!/usr/bin/env python3
"""Build the README's stills and clips out of a real emulated run.
tools/bench/pace_run.sh ... # or the DLX_SNAP_EVERY=1 run below
python3 tools/media/make_readme_media.py <container.dlx> [--snap tmp/snap_rec]
Everything this writes comes from PNGs MAME wrote while `src/player/stream.s`
decoded out of a 256 KB ring on an emulated stock X68000. Nothing is redrawn by
the reference decoder and nothing is upscaled with interpolation -- the pixels
in `docs/img/` are the pixels that were on the emulated screen.
THE MAPPING IS ASSERTED, NOT ASSUMED. `stream.lua` snapshots at the frame tick,
BEFORE that tick's frame is decoded, so snapshot n holds frame n-1 -- and that
is an off-by-one waiting to put the wrong caption under a picture. This script
finds the offset by comparing against `tools/encoder/dlx.py`'s reconstruction
and refuses to write anything unless every frame matches exactly at one offset.
A README that illustrates a pixel-exact decoder with an approximate picture
would be a small lie about the one property the project keeps testing.
Writes:
docs/img/decoded-frame.png one frame, 3x nearest
docs/img/source-vs-decoded.png Blu-ray source | 68000 output, same frame
docs/img/player.webm the 120-frame window, side by side, 12 fps
docs/img/modes.webm the same window with the block-mode map
"""
import argparse, os, subprocess, sys, shutil
sys.path.insert(0, "tools/encoder")
import numpy as np
from PIL import Image, ImageDraw
from dlx import DLX
SNAP_H, SNAP_W = 512, 256 # MAME -snapview native, double-scanned
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/rc_fr_singe_scsi_span.dlx")
ap.add_argument("--snap", default="tmp/snap_rec")
ap.add_argument("--src", default="tmp/fr_singe", help="the extracted frames")
ap.add_argument("--out", default="docs/img")
ap.add_argument("--still", type=int, default=96, help="which frame for the stills")
ap.add_argument("--fps", type=float, default=12.0)
a = ap.parse_args()
if not shutil.which("ffmpeg"):
sys.exit("ffmpeg not found -- needed for the webm")
os.makedirs(a.out, exist_ok=True)
d = DLX(a.container)
# --- the reference reconstruction, in the palette the machine actually shows.
# Same derivation verify_decode.py uses: the X68000 word is GGGGGRRRRRBBBBBI, so
# a channel is 5 bits plus a shared intensity, and I is chosen per entry by
# minimum squared error against the RGB888 the encoder emitted (FINDINGS 23.3).
pal = d.pal.astype(int)
p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
fl = pal >> 3
render = lambda I: p6((fl << 1) | I[:, None])
I = (((render(np.ones(256, int)) - pal) ** 2).sum(1)
< ((render(np.zeros(256, int)) - pal) ** 2).sum(1)).astype(int)
LUT = render(I).astype(np.uint8)
canvas = np.zeros((d.H, d.W), np.uint8)
ref, modes = [], []
for f in range(d.nframes):
d.paint(canvas, f)
ref.append(LUT[canvas].copy())
modes.append(d.modes(f).reshape(d.nby, d.nbx).copy())
def picture(png):
"""The 256x192 picture out of one MAME native snapshot."""
s = np.asarray(Image.open(png).convert("RGB"))
if s.shape[:2] != (SNAP_H, SNAP_W):
sys.exit(f"{png}: expected {SNAP_W}x{SNAP_H}, got {s.shape[1]}x{s.shape[0]}")
g = s[0::2] # undo the double scan
y = (g.shape[0] - d.H) // 2 # the picture is centred
return g[y:y + d.H]
snaps = sorted(f"{a.snap}/x68000/{n}" for n in os.listdir(f"{a.snap}/x68000")
if n.endswith(".png"))
if not snaps:
sys.exit(f"no snapshots in {a.snap}/x68000 -- run stream.lua with "
f"DLX_PACE=1 DLX_SNAP_EVERY=1")
shots = [picture(p) for p in snaps]
# --- match every snapshot to a frame, and classify what does not match -----
# The naive expectation is wrong in a way worth recording: stream.lua fires the
# snapshot at the tick, BEFORE frame n is decoded, but MAME renders the screen
# at the END of the machine frame -- by which time the 68000 has finished frame
# n (it needs 69% of a 12 fps slot). So snapshot n IS frame n. That is a
# statement about when a screen bitmap is captured, not about the decoder, and
# it is the kind of thing to check rather than reason about.
#
# A handful of snapshots are TORN: the top of the picture is frame n and the
# bottom still holds frame n-1, because the capture landed while the block loop
# was partway down the screen. That is not a decoder fault and it is not an
# artefact of the rig either -- decode.s writes straight to the displayed page
# (FINDINGS 28.1, one display path, no flip), so a real player tears the same
# way. They are KEPT, with the tear asserted: every differing pixel must equal
# the previous frame, or this is something else and the script stops.
frames, torn = {}, []
snap0 = shots[0]
for n in range(1, len(shots)):
j = n
if j >= d.nframes: # pace_run.sh's own end-of-run snapshot
continue
sh = shots[n]
if np.array_equal(sh, ref[j]):
frames[j] = sh
continue
diff = (sh != ref[j]).any(2)
if not np.array_equal(sh[diff], ref[j - 1][diff]):
sys.exit(f"snapshot {n} is neither frame {j} nor a tear against frame "
f"{j-1}:\n {diff.sum():,} pixels differ and they do not come "
f"from the previous frame.\nThat is a decoder fault or a "
f"changed snapshot geometry, not something to align around.")
rows = np.where(diff.any(1))[0]
torn.append((j, int(rows.min()), int(rows.max())))
frames[j] = sh
exact = len(frames) - len(torn)
print(f"{len(snaps)} snapshots -> frames 1..{max(frames)} of {d.nframes}.")
print(f" {exact} PIXEL-EXACT against tools/encoder/dlx.py")
if torn:
print(f" {len(torn)} torn by the capture, each verified to be frame n on "
f"top of frame n-1:")
for j, r0, r1 in torn:
print(f" frame {j}: rows {r0}..{r1} of {d.H} still hold frame {j-1}")
print(f" dropped snapshot 0 (frame 0 caught part-drawn, nothing behind it) "
f"and the\n end-of-run duplicate.")
src = {}
for j in frames:
p = f"{a.src}/f{j+1:04d}.png"
if os.path.exists(p):
src[j] = np.asarray(Image.open(p).convert("RGB"))
Z = 2 # nearest-neighbour zoom
BAR = 22 # caption strip height
def up(img, z=Z):
return np.repeat(np.repeat(img, z, 0), z, 1)
def captioned(img, text, z=Z):
w = img.shape[1] * z
out = Image.new("RGB", (w, img.shape[0] * z + BAR), (16, 16, 18))
out.paste(Image.fromarray(up(img, z)), (0, BAR))
ImageDraw.Draw(out).text((6, 6), text, fill=(190, 190, 196))
return out
def side_by_side(j):
left = captioned(src[j], "Blu-ray source, cropped 256x192")
right = captioned(frames[j], "68000 output (emulated X68000, 256 colours)")
out = Image.new("RGB", (left.width + right.width + 8, left.height), (16, 16, 18))
out.paste(left, (0, 0)); out.paste(right, (left.width + 8, 0))
return out
# --- stills ----------------------------------------------------------------
still = a.still if a.still in frames else sorted(frames)[len(frames) // 2]
Image.fromarray(up(frames[still], 3)).save(f"{a.out}/decoded-frame.png")
print(f" {a.out}/decoded-frame.png frame {still}, 3x nearest")
if still in src:
side_by_side(still).save(f"{a.out}/source-vs-decoded.png")
print(f" {a.out}/source-vs-decoded.png frame {still}")
# --- the block-mode map ----------------------------------------------------
# The decoder's whole cost model is per mode, so the map is the picture the
# FINDINGS tables are really about. Colours are the four modes, not a heat map.
MODE_RGB = np.array([[16, 16, 18], # SKIP -- costs nothing, draws nothing
[60, 130, 220], # V1 -- one index for a 4x4 block
[235, 175, 60], # V4 -- four indices
[225, 70, 70]], # RAW -- sixteen bytes verbatim
np.uint8)
NAMES = ("SKIP", "V1", "V4", "RAW")
def mode_panel(j):
m = MODE_RGB[modes[j]]
m = np.repeat(np.repeat(m, 4, 0), 4, 1) # a block is 4x4 pixels
return m[:d.H, :d.W]
def mode_pair(j):
left = captioned(frames[j], "68000 output")
counts = np.bincount(modes[j].ravel(), minlength=4)
lab = " ".join(f"{n} {100*c/counts.sum():.0f}%"
for n, c in zip(NAMES, counts))
right = captioned(mode_panel(j), "block modes: " + lab)
out = Image.new("RGB", (left.width + right.width + 8, left.height), (16, 16, 18))
out.paste(left, (0, 0)); out.paste(right, (left.width + 8, 0))
return out
# --- clips -----------------------------------------------------------------
def webm(name, maker, keys):
tmpd = f"tmp/_media_{name}"
os.makedirs(tmpd, exist_ok=True)
for n, j in enumerate(keys):
maker(j).save(f"{tmpd}/{n:04d}.png")
outp = f"{a.out}/{name}.webm"
# VP9, near-lossless: this is 256x192 pixel art scaled by integers, and a
# codec that smooths a 4x4 block boundary would be editorialising about the
# one thing the picture is evidence of.
cmd = ["ffmpeg", "-y", "-loglevel", "error", "-framerate", str(a.fps),
"-i", f"{tmpd}/%04d.png", "-c:v", "libvpx-vp9", "-crf", "12",
"-b:v", "0", "-pix_fmt", "yuv444p", "-row-mt", "1", outp]
subprocess.run(cmd, check=True)
shutil.rmtree(tmpd)
print(f" {outp} {len(keys)} frames @ {a.fps:g} fps, "
f"{os.path.getsize(outp)/1024:.0f} KB")
keys = sorted(k for k in frames if k in src)
webm("player", side_by_side, keys)
webm("modes", mode_pair, sorted(frames))