drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: drwily/Dragon-s-Lair-X68k:ed172c2da297c7f6931364373e16c45b5d13bb7d
Branches Tags
drwily/Dragon-s-Lair-X68k:main
..
compare: drwily/Dragon-s-Lair-X68k:main
Branches Tags
drwily/Dragon-s-Lair-X68k:main
21 Commits
ed172c2da2 .. main
Author SHA1 Message Date
prosolis 91e2e9803d Take the project's own story out of the README, starting with a claim about elapsed time 
USER CORRECTION: the README is a description, not a chronicle, and the intro
had it narrating an arc -- "for most of this project's life that was read as a
video codec problem" -- which is also a claim about elapsed time that nothing
measured supports. The tree is days old.

The intro now states the design: it is a delivery problem, the live design has
no decoder in it, and the comparison that settles it is a measurement (37,585
decoded costs more than 49,152 not decoded; 110.4% against 54.9% at the 9 clk/B
floor). The codec is "in the tree and parked (USER DECISION)" -- no session
number attached, because which sitting it happened in is STATUS's business.

Four more of the same, found by grepping rather than by eye:
  - B2 "is not a headroom question any more" -> it decides which player exists
    rather than how much headroom one has.
  - "Other things the machine has said" -> "What else a player has to carry".
  - ring.i/xfer.i/stream.s "left the video path when a DMAC-direct player
    deleted the ring" -> a DMAC-direct player has no use for them, because it
    has no ring. Same edit in the parked list and in Layout.
  - "Why it is parked" kept the standing reason and dropped the retelling.

And one where the story was carrying a fact that had to stay: the sasi profile
was "dropped on capacity rather than bandwidth" with the numbers left in
FINDINGS. They are in the sentence now -- a 40 MB volume against 146 MiB of
game -- so the clause survives without the reader needing the episode.

One "session" reference stays, at the STATUS.md bullet: that file IS ordered
newest session first, and saying so is navigation for another document rather
than this one telling a story.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 16:59:04 -07:00
prosolis 3b7f2af27e Lead the README with the player that has no decoder, and give it a clip with sound 
The README still opened by calling this a video codec problem and led with the
codec player's stills and clips, which have been PARKED since session 29
(FINDINGS 61.8). Everything measured since says the opposite: at the 9 clk/B
dual-address floor the codec is 110.4% of a 12 fps frame and the decoder-free
packed player is 54.9%, so decoding 37,585 bytes costs more than not decoding
49,152. The document now says that in its first paragraph.

871 -> 473 lines. "Where it stands" was an append-only pile of session
headlines -- which is what docs/STATUS.md is for -- and is now a STATE: what
runs end to end, the four resources every item is priced in, the one open
question that decides which player gets built (A vs B, blocked on B2), and the
machine facts a player has to carry. The codec keeps a section as the parked
branch, with its media and the reason it is parked.

Drift struck while restructuring:
  - "Current encode: 496.7 KB/s at 29.19 dB" was session 14's candidate, not
    the DLX5 gate container, and was presented as the project's current state.
  - The Encoder section documented only the parked encoder. It now leads with
    pack.py/dlxp.py/adpcm.py and uses the gate window the tree actually
    encodes (00223, 539.4 s, 10.0 s) rather than 00020.
  - Analysis scripts 25, 30 and 34 were described under tools/bench/gvpack/
    and tools/import/. They are tools/analysis/.
  - make_packed_media.py was never mentioned.
  - check.sh was "~6 min" in prose and "~3 min" in the code block, and its own
    header says ~2 min. No figure is quoted now rather than a fourth one.

AND THE HERO IS A SEQUENCE NOW, BECAUSE A PACKED FRAME IS A LITERAL. A still
cannot show the property that separates this branch from the codec's: there is
no recursion here, so frame 119 says nothing about frame 60 and 120 frames are
120 separate claims. make_packed_media.py grew a clip path that gates all of
them before writing any -- and refuses to write if one differs, because a clip
of the player is a claim that the player drew it.

The clip is a COMPOSITE of two runs of packed_run.sh and it says so on the
page: the picture is run 1, paced at half rate so the snapshot lands inside
the write window; the sound is run 5, MAME's own -wavwrite capture off the
speaker at 12 fps. They cannot be one run -- audio cut at 12 fps played at 6
is not this scene. What it is NOT is a real-time capture of the shipping
configuration, which at this container's own burst rate would show a blank
layer for 99.5% of every slot (FINDINGS 64.2).

The audio cut is GATED, not eyeballed. The capture opens with the machine
booting, so the stream starts at the first non-zero sample -- exactly the kind
of alignment that is off by one byte forever with every counter agreeing
(71.5). Lump 0 is decoded with the four ADPCM axes out of the container's own
header and required to be sample-exact from the cut: 50,603 samples in, 3.24 s
of boot dropped, terms/low, 10-bit, init -2.

check.sh is ALL GREEN with this in the tree (tmp/check_s40.log, exit 0, 101 OK
lines). It does not gate the README or the media builder, so that run confirms
the tree the README describes and not the description.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 15:48:00 -07:00
prosolis ba966efe7e Take the player through a branch with sound, and find the predictor does not seek 
FINDINGS 71, ROADMAP P6d.  70.3 named exactly what was missing -- packed.s starts
PG_AK/PG_AKF at lump 0 and has no audio seek path -- and priced its absence at a
mean 416.5 ms of silence over the arcade's 409 within-container seek targets.
pg_aseek is that path: the lump index, the stream position, the remainder
accumulator and the byte offset into the group, then the second READ(10) at the
lump's own LBA and a re-arm part way into the buffer.  Measured off a real volume:
132,162 B of spliced stream accounted for byte by byte in MAME's own capture,
across a branch at frame 37 -- four frames into lump 3, deliberately NOT on a
group boundary -- in both chip configurations.  Skip computed 2,604 B, cadence
says 2,604.

THE PREDICTOR DOES NOT SEEK, AND THE ERROR IS DC.  The MSM6258's accumulator is a
pure integrator with no leakage term, so a branch that hands the chip bytes chosen
for a state it is not in produces an offset that does not decay.  Playing through:
DC -355 of 511 with AC 0.00 -- the right shape from the wrong ground -- still -108
four seconds later.  STOP and re-PLAY: all 62,500 post-seek samples are EXACTLY a
decode from the container's own init, and the whole error is the single constant
-65.  A re-PLAY is 5.5x better and neither is zero, so PG_ARST is a mailbox with a
number under it.  The host computes -65 out of the container's bytes and the gate
asserts the equality rather than printing both.

AND THE ONLY FIX THAT REACHES ZERO IS THE ENCODER'S.  A player cannot set the
chip's accumulator, only reset it.  Resetting the encoder's predictor every frame
makes all 119 of the container's branch points exact for 0.33 dB (21.99 -> 21.66),
because the step table's floor is a constant 16.  That is a DLXP3 and it is
deliberately not in tools/encoder.

TWO SILENT BUGS, BOTH CAUGHT BY THE CAPTURE.  pg_udiv32 trashes d4 and pg_aseek
held hz there, so the offset came out 1 byte instead of 2,604 -- 166 ms of the
wrong part of the scene at exactly the right rate, every counter agreeing.  And
one already in the tree that had passed this gate three times: pg_ainit waited on
a READ-BACK MTC before PLAY, which is the same test as "a byte has left RAM" only
if no byte leaves in between.  One does, and the chip then plays the scene one
byte in, forever.  Found by locating the capture's opening samples in the
container image: sector 1 + 1.  The witness is now the count that was written.

ALL GREEN, two new stages included.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 15:10:20 -07:00
prosolis 8b5f51704c Take the climb to a real branch point, and find the cadence has a third column 
FINDINGS 70, and it closes ROADMAP P6.  65.6 recorded that "the slack table is
here, but 51.3's refill climb with a second consumer through a real branch point
is not", and 67.6 repeated it verbatim two sessions later.
tools/analysis/36_branch_audio.py is that run: 51.3's surplus model, 56.3's 612
real branch points, 32's cadence arithmetic, both containers' headers and the
buffer constants of the player 68 measured, in one place.  No machine -- all
five were already numbers.

AUDIO IS 1.7% OF THE CODEC WIRE AND UP TO 3.30x OF THE CLIMB.  Slack accrues out
of pipe - wire, which is a small difference of two large numbers, so a second
consumer is not priced by its share of the wire -- which is how every budget in
this tree since FINDINGS 8 has quoted it.  At 451.4 KB/s (the container's own
zero-prefill pipe, 49.5) in a 256 KB ring the climb goes 6.70 s -> 22.11 s and
the branch points arriving under it go 482/612 (79%) -> 603/612 (99%).  At 488
it is 1.19x, at 600 it is 1.05x.  Correction to 56.4: it charged audio at
ratectl.AUDIO_KBPS = 7.8, which is 7,812.5 B/s in decimal kB; in binary KB the
figure is 7.6294, so the placeholder was 2.24% high and every column moved in
the flattering direction by less than one part in six hundred of the wire.

AND ON THE PACKED BRANCH THERE IS NO CLIMB AT ALL.  A record goes straight into
GVRAM, so the video lookahead is zero records and 56.4's alarming column is
vacuous when every frame arrives with less lookahead than the one before it.
Acceptance there is a per-frame deadline, not an average, which is why B1 has to
name a burst rate.  The only consumer on that branch with any lookahead is the
audio one: 1.833 s of sound held against 0.000 s of picture.  And at 589.6 KB/s
-- the acceptance figure this project quotes -- the sounded container's surplus
is exactly zero: it plays and never banks a byte, while the silent one climbs
its record in 6.38 s.

THE COST NOBODY HAD COUNTED.  A DLXP2 group is `lump k, then F records`, so lump
k is at a LOWER address than every record of its group but the first, and a seek
to record i finds its audio behind it.  Measured on the 409 within-container
seek targets of the arcade's own graph at the shipped F=11: mean 416.5 ms of
silence entering a branch, median 416.7, p90 750.0, worst 833.3, and only 36 of
409 land on a group boundary.  The other 203 -- the scene changes -- are free by
construction, because lump 0 sits between the header and record 0.  The
container's own start is the one branch point the cadence costs nothing at, and
it was the only one anybody had looked at.

SO THE CADENCE PICK IS REOPENED, on a third column.  32_audio_wire chose F=11 on
padding and RAM; F=1, the cadence it called the worst one, has no group to enter
off-boundary at all, and gives back 12,288 B of RAM for +4.36 KB/s of wire
(0.74% of the acceptance figure).  The alternative is a second read: lump k is
up to 503,808 B behind, so it is a separate command of 7,168 B -- 11.7 ms
against 416, 36x cheaper in time -- and src/player/packed.s starts PG_AK/PG_AKF
at lump 0 and has no audio seek path at all.  Nothing here decides it; the
deciding number is what one more SCSI command costs, and that is B1's.  The
content is not uniform mod F either: the measured mean runs 0.67x..1.23x of a
uniform assumption across the sweep, and the 1.00x at F=11 is a coincidence.

New check.sh stage, ~10 s, and it gates the STRUCTURE and deliberately not the
milliseconds: audio never shortens a climb, the shipped cadence strands most
within-container branches, F=1 strands none, and the lump read is an order of
magnitude cheaper than the silence it removes.

The one thing 70.3 and 70.4 rest on is a DESIGN ASSUMPTION and is printed in the
tool's own output: one container per SCENE (53, 55.1, 56.3).  One container per
sequence makes every seek land on frame 0 and empties both sections.

No encoder, player or container byte moved.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 14:21:32 -07:00
prosolis ab15c0749a Measure the level off the whole disc, and find the headroom is not worth buying 
FINDINGS 69, ROADMAP P6 -- the item 66.3 reopened in session 34 and sessions 35
and 36 both deferred.  The chip clamps its accumulator at 10 bits INSIDE the
recursion, and the ten seconds every audio figure in this tree is quoted on peak
at 435 of 511: it fits, and it fits by accident, because that window is a
-13.4 dBFS passage.  Nothing knew what the loudest passage of the game was.

tools/analysis/35_audio_level.py reads every stream of the unique scene footage
(00000-00201) through extract_audio.py's own chain -- 1,291.6 s, 201 of 202
streams -- and encodes windows of it with adpcm.CHIP.  The disc peaks at 946 of
2048 = -6.71 dBFS (00200 @ 2.11 s), which is 5.35 dB over the clamp, and the
census behind that peak is 687 samples of 20,182,000 (0.0034%) in 402 events,
44.0 ms, longest 0.90 ms.

THE HEADLINE IS A NEGATIVE: THE LEVEL DOES NOT CHANGE.  Forty 2 s windows drawn
over the game's timeline at six gains -- the disc's own level (gain 1.0) has the
best mean SNR (22.03 dB) and the best median, and loses the worst-window column
to -3 dB by 0.04 dB.  The gain that guarantees zero clamping disc-wide (0.5402)
costs 0.85 dB of mean SNR across the whole game to buy back 1.90 dB on the
2.11 s that clamp, because the OKI step table's floor is a constant 16 and does
not scale with the signal.

AND 66.3's MECHANISM DOES NOT SURVIVE A CONTROL.  Error after a clamp run is
elevated ~5x -- and so is the same window at a gain that never clamps, read at
the same indices, because those samples are simply loud.  Worst ratio 1.28 over
64 offsets, and the clamped encode's whole-window mean |error| is the LOWER of
the two (4.71 vs 5.05).  adpcm.encode runs the chip's clamp inside its own
sixteen-way search, so it never loses the chip's state.  The worry was right
about the mechanism and aimed one layer too late: an encoder clamping at 12 bits
while the chip clamps at 10 is exactly that divergence, and 66 closed it.

pack.py gains --audio-gain (default 1.0) so the level is a named parameter with
a measurement behind it instead of a shift buried in a list comprehension, and
prints the encoded window's peak against the clamp.  tmp/packed_singe.dlxp
rebuilds byte-identical, all 6,039,040 B.  New check.sh stage, ~18 s.

Three rig facts in 69.4, because a shipping encoder meets all three: 00176 has
no audio track at all; 00199 is 61.31 s of video with 1.25 s of audio; and 18
stream pairs share duration, peak and RMS, 7 of them byte-identical.

The 10-bit clamp is a DRIVER SETTING, not a chip constant -- x68k.cpp:1089 sets
OUTPUT_10BITS -- so it is MAME's reading of the board, and hardware item 5 is
what settles it.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 14:20:55 -07:00
prosolis 191f2b47bb Put the container on the chip, and find the held bus costs 463 times the seam 
ROADMAP P6c, FINDINGS 68. 78,125 B of a DLXP2's audio out of channel 3,
sample-exact, while the video channel fetches records off the same disc.

The two pieces 67.6 said were missing: the lump buffer (pg_afill/pg_afetch,
three slots and the minimum is unmeasured) and 67.2's remainder accumulator
(pg_apay). The capture prices what the accumulator avoided at 1.26 s of
lip-sync over the game, against 67.2's predicted 1.25.

The finding is the third piece, which nothing had named: the MSM6258 has no
starvation state, so the gap between a channel counting out and the next arm
is a held nibble pair driving the predictor. Stealing, the seam is 0.51 ms
over ten seconds because dma.i's new DM_HOOK services the chip from inside
the transfer wait -- 250,000 of 250,240 looks. Held, the 68000 is halted and
gets 369: every one of the ten lump boundaries has a seam, worst 72.8 ms,
2.31% of the audio. Identical bytes, different sound. 64.3 reaching the audio.

Two bugs, and no counter in the player could see either. Clearing DM_BARV does
not unchain a channel -- OCR bits 3-2 are what it obeys -- and the symptom is
POLL TIMEOUT on the lump and every record after it. And the refill ran one lump
ahead of its ring and overwrote the buffer the channel was reading: 11 of 11
armed, 11 fetched, no starve, and the sound wrong from 0.2 s in. Which is why
the gate is a WAV: verify_packed_audio.py walks the stream one delivered byte
at a time, because MAME's okim6258 resets the nibble select on every write and
a byte is two nibbles only 99.994% of the time.

check.sh ALL GREEN before (tmp/check_s36_start.log) and after
(tmp/check_s36_end.log), with the new stage.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 13:11:37 -07:00
prosolis e3778f62b0 Put sound in the packed container, and find the padding is a rate error 
ROADMAP P6b, FINDINGS 67.  DLXP2: a 64-byte header and then groups -- one
audio lump of A sectors, then F records -- so record i is at
off_frm + i*rec + (i//F)*A*512 and lump k at off_aud + k*(F*rec + A*512).
Still no index and still none needed, which is the packed branch's whole
claim surviving the one change that could have ended it.  The player carries
the third term in six instructions once a frame and zero parsing, and 120 of
120 records are still pixel-exact off a real MB89352 volume with the
interleave in, against a silent control that says no picture byte moved.

The finding is what 65.3 called padding.  A lump is 7,168 B of SPACE; eleven
frames of audio is 7,161.4583... B, so the payload alternates 7,161 and
7,162 and the rest is zero.  A player that fed the chip the whole lump --
which is what "14 sectors every 11 frames" invites -- runs 0.09% fast, and
that is not waste, it is drift: 0.84 ms a group, 1.25 s of lip-sync over the
game's 22.8 minutes.  What a player carries is one accumulator,
acc += 11*15625; n = acc//24; acc %= 24, which is clock.i's shape for
clock.i's reason and the third time this tree has met the pattern.

The four ADPCM axes ride in the header as fields rather than a version
number, and the gate flips each one to prove they earn it: nibble order
-31.99 dB, delta formula -24.86, clamp 0.00, accumulator -0.49.  Nothing
parses a packed container, so the gate partitions the whole file -- 131
spans, no overlap, no gap -- and asserts what a cadence-blind player would
read: exactly records 11..119 wrong, and frames 0..10 identical either way,
which is how an off-by-one like that survives a rig that checks frame 0.

Wire 582.0 + 7.64 = 589.6 KB/s, 65.3's prediction to the tenth.

Green light ALL GREEN before (tmp/check_s35_start.log) and after
(tmp/check_s35_end.log), with the new stage in it.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 11:22:38 -07:00
prosolis 6dd3fb3597 Ask the chip which decoder it is, and find four wrong axes where one was expected 
ROADMAP P6a, on the machine. 68000 code programs HD63450 channel 3 with the
IPL ROM's own ADPCM bytes -- dual address, 8-bit port, cycle steal, external
request -- and feeds the MSM6258 a designed 1,678-nibble stream at the chip's
own pace: 839 B in 0.1074 s = 7,811.4 B/s against the format's 7,812.5, CER=$00.
That transport is P6b's, not scaffolding.

Sixteen candidate decoder models, three capture decimations and a searched
prologue are fitted to MAME's capture. Exactly one reproduces it sample-exact
over all 1,678 samples, and every axis carries a negative control: flip it
alone and the closest survivor disagrees on 826, 1,504, 156 and 1,522 samples.

The chip runs 'terms', takes the LOW nibble of a byte first, clamps the
accumulator at 10 bits and starts it at -2. tools/encoder/adpcm.py defaulted to
the opposite of all four, and 65.2 named the wrong axis as the risk: the delta
formula is worth -2.88 dB and the NIBBLE ORDER is worth -25.74 dB. 65.1's "high
first, measured" was a measurement of ffmpeg, i.e. of the VOX file convention,
which is a different question from what a chip does with a byte in its data
register.

The 10-bit clamp is free on the Singe window and only because that window peaks
at 435 of 511 -- 1.4 dB of headroom on a -13.4 dBFS passage, 12.1 dB below where
the encoder was clamping, and inside the recursion. So the audio level is an
open choice again, downward, and the loudest passage on the disc is unmeasured.

Session 33's silence had two ordinary causes: the PPI's port C is an input until
control word $92 says otherwise, and $01 is COMMAND_STOP. And a rig fact worth
the space: the 8 MHz ADPCM clock is CT1 in the YM2151's $1B, delivered on the
sound system's schedule rather than at the store, so a transfer started in the
same breath as the setup plays its first ~17 ms at the old clock and no model
fits a stream that changed rate part way through.

Name the layer: this is MAME 0.277's okim6258 device model measured end to end
through the machine's real transport. It settles the rig and not the silicon.
Also struck: 64.4's "no MAME source tree is on this machine" -- there is none on
disk, but the machine has network and the upstream tag fetches.

check.sh ALL GREEN before (tmp/check_s34_start.log) and after
(tmp/check_s34_end.log), with one new stage.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 09:50:03 -07:00
prosolis f925a1dd9a Put sound on the wire, and find three LSBs are worth 25 dB 
ROADMAP P6, everything in the item except the bus half session 20 closed.
tools/encoder/adpcm.py is an MSM6258 codec, tools/encoder/extract_audio.py
takes the same seconds of the same stream the frames come from,
tools/bench/verify_adpcm.py is the gate, tools/analysis/32_audio_wire.py the
container arithmetic.

There is no reference encoder -- ffmpeg has a decoder for this format and none
the other way -- so what is gated is the decoder the encoder runs INSIDE its
own nibble search, sample-exact against ffmpeg's over 4,268 nibbles. An
encoder that agrees with its own wrong decoder is what that catches. The Singe
window: 156,250 samples -> 78,125 B at 21.97 dB, which is 7,812.5 B/s to the
byte. Normalising the disc's -13.4 dBFS level moves the SNR 21.97 -> 21.97, so
the level is not a lever.

And the two published delta formulas are not the same codec. They differ by at
most 3 in 12-bit units; encode for one and decode on the other and the SNR
goes 21.97 -> -2.88 dB, the noise louder than the signal, because ADPCM is
recursive the way the video codec is temporally recursive. Which one the chip
runs is now P6a and it is a precondition on shipping any audio.

And audio is the first thing the packed branch's simplification has cost
anything for. A record has no index BY DESIGN, so audio cannot be per-record
without making records variable; it rides a fixed cadence (F, A), the obvious
F=1 wastes 57.3% of every audio sector, and the pick is F=11 A=14 -- 0.09%
padding, 14,336 B held, wire 582.0 -> 589.6 KB/s. The codec container, which
kept its index, pays zero.

The MAME experiment did not work and 65.5 says so: :okim6258 is there at
$E92001/$E92003, read out of the machine's own program map, and feeding it
from Lua recorded silence across control 0..3 x port C 0..15. The register
semantics were not guessed at further.

FINDINGS 65. check.sh ALL GREEN before and after, with a new stage.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 09:11:03 -07:00
prosolis 6f698ca226 Put the player on a real volume, and find the write window is the frame 
ROADMAP K3. src/player/packed.s (2,898 B) brings up its own display, builds
its own 193-entry DMA chain, keeps its own frame clock off V-DISP and fetches
every record itself with READ(10) off a CZ-6BS1. The rig writes no picture
byte, no palette entry and no CRTC register.

120 of 120 frames pixel-exact, every one compared, in both palette orders --
the gate had to grow to do it, because a packed frame is a LITERAL and the
codec's recursion was what made one comparison audit 120.

And the write window turns out to be the frame. A packed write requires R20
bit 11, buffer mode blanks the layer, and a DMAC-direct player holds the
window open for the whole data phase, so

    dark fraction of a slot = record bytes / (DATA-PHASE rate x slot)

which is 1.0 at the container's own 582.0 KB/s: every frame delivered, on
time, pixel-exact, and none of them displayed. The rate in that expression is
the BURST rate, a third hardware number B1 has no test for. It reverses 61.5's
ranking -- a packed player that DMAs to RAM and paints with the measured 27.3%
blit is on screen 72.7% of every slot at any rate, and the two are equal only
at 2,131 KB/s = 3.7x the wire.

And a held channel costs the frame clock half its ticks without the clock
being able to tell: 487 of 1,038 V-DISP edges lost, zero late frames reported,
the player believing 12 fps while the screen ran at 6.37.

FINDINGS 64. ROADMAP K4 opened and fenced behind B2.
check.sh ALL GREEN before and after.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 09:10:48 -07:00
prosolis f1007a0dbc Put the frame in a container with no decoder, and find the palette is not free 
ROADMAP K2. DLXP1: a 49,664 B record that is 97 sectors exactly, no index and
no length word, because a packed record's length is geometry rather than
content. 582.0 KB/s, which is what FINDINGS 61.9 predicted to the tenth, and it
encodes in 3.3 s because there is no k-means in it.

px68k's own x68k/gvram.c renders the container's bytes index-exact with the
harness computing no interleave -- the only test that can catch an encoder whose
byte order is wrong, since a container round-trips against its own inverse
either way. Both negative controls fail as they must.

The picture is re-derived against this project's builder rather than PIL's
(34.05 dB against 61.9's 34.08) and the GGGGGRRRRRBBBBBI word is charged for the
first time in this tree: 0.53 dB, on every row, so it moves no comparison.

What the control found is the finding. A packed container on a SCENE palette
lands exactly on the codec's ceiling, so the whole +2.31 dB is the per-frame
palette and nothing else -- and 231 of 256 entries change every frame, which
makes a mismatched paint 12.8 dB worse than the correct pairing, on screen for
roughly half of every frame slot if buffer mode does not blank. So B2 now
decides which packed CONTAINER ships, not only which player. The fallback is
already a flag: --scene-palette --no-palette is 30.79 dB, zero churn, 576.0 KB/s
and still +2.07 dB on the shipping codec.

62.5 is priced and is a wash: palette first 20.32 dB, palette last 20.33.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 07:31:05 -07:00
prosolis 07f36c2af9 Put the palette on the channel, and find one start paints a whole frame 
ROADMAP K1, the packed player's one open structural item.  A frame is a picture
AND a palette, and no run in this tree had pointed a DMA channel at the palette
registers.  dmagate.s runs 7-9, gated by dma_run.sh and check.sh:

  7. 512 B off the disc into $E82000, bus held -- byte-exact in 256 register
     words, read back OUT OF the registers by the 68000;
  8. the SAME transfer aimed at RAM -- byte-exact at $2C000, and 256 of 256
     palette words still read the poison the CPU wrote, which is what attributes
     run 7 to the channel's MAR rather than to the readback path;
  9. ONE array-chained start across two kinds of destination -- the palette and
     six picture rows at the 1,024 B line stride, 2,048 B byte-exact.

So a packed frame is one channel start: a 193-entry array, palette first, CPU
halted from the first byte to the last.  The array is scene-constant, because
the packed layout spends both 256-colour pages and there is no page to flip.
What is left on the CPU per frame in the video path is the channel start and the
READ(10) -- no per-frame PAINT, which is not the same claim as no per-frame CPU.

The destination is POISONED first (62.1).  Runs 4-6 wrote into RAM that was zero
and GVRAM that was stale against a record that is mostly pad; "it matches the
disc" was weaker than it read as.  The host counts whether the poison actually
discriminates instead of assuming it: 511 of 512, and the gate refuses under 500.

And it opened a hardware item (62.4, ROADMAP B4).  MAME maps the palette to
palette_device over memory_array, whose write16 is a plain COMBINE_DATA -- RAM
that honours mem_mask, with no handler that could refuse a byte write.  Unlike
GVRAM's 256-colour arm there is nothing here to be wrong about, so the run
bounds the model and not the board.  What a real palette register does with a
byte write is unmeasured.  A negative costs 0.28% of a frame and nothing else.

29_packed_player.py now also prints the two rows with the per-frame palette
charged -- 55.7% of a frame on the chain, 582 KB/s -- alongside the picture-only
figures the codec comparison is quoted against.

check.sh ALL GREEN before (tmp/check_s30_start.log) and after
(tmp/check_s30_end.log).

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 07:06:44 -07:00
prosolis 1be428c270 Align the container to the disc, and find the decoder-free packed player fits 
Two sessions, unrecorded until now, committed together because their edits
share files and cannot be split cleanly after the fact.

Session 28 (FINDINGS 60): the container is DLX5 -- every record sector-aligned,
120/120 starting on a boundary where 3/120 did, +0.48% on the wire and zero
clocks -- and the ring's release rounds to RECALN so no pad is stranded.  Two
encoder levers measured and refused: `--spans all` buys +0.19 dB for +67% of
the wire, and joint span/lam selection emits byte-identical containers because
`lam` never leaves its floor on any of 120 frames.

Session 29 (FINDINGS 61): the packed full-frame blit is 27.3% of a 12 fps
frame, a channel fills GVRAM in buffer mode off the disc with the CPU halted,
and it walks the 1,024 B line stride itself through array chaining.  At the
9 clk/B dual-address floor the codec is 110.4% of a frame and a decoder-free
packed literal player is 55.2%, at +4.89 dB -- 2.75 dB past a ceiling the
codec's scene-wide palette cannot cross.  Encoder work is parked; the codec is
kept and not built on.

check.sh is ALL GREEN before and after, plus one new stage that gates the ORDER
of the measured paint costs rather than their values.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 06:54:27 -07:00
prosolis 8800d8f8c0 Price the transport against the frame, and find dual address cannot fit 
FINDINGS 59.7. tools/analysis/15_bus_occupancy.py has always answered "what does
each W cost" and never "what can the frame afford", and after 59.2 those are not
the same question. It now answers both, and takes an optional --kbps for the
auto-request rows -- the only rows whose cost depends on how long the record
takes to arrive.

On the gate container at 12 fps, decode term MEASURED: decode 68.5%, audio 1.25%,
HEADROOM 30.2% = 6.74 clocks per byte at a 37,403 B record. Against that, P4a
cut the transport from 391.7% of a frame to 40..95% -- four to ten times, the
largest movement in this project's cost model since the decoder was written --
and it still does not fit. A dual-address byte is a 4-clock read of the device
plus a 5-clock write to memory, so 9 clk/B is a FLOOR and the frame affords 6.74.
No GCR share goes under it and no delivery rate goes under it: a share decides
whether the channel sits at the floor or above it. At 460 KB/s max-rate totals
165.1% and LRAR at 50% totals 117.4%, and a 50% share tops out at 543 KB/s, above
which the channel is the bottleneck and the rate falls back to exactly that floor.

So 59.2's three bounds arrive in the budget as one sentence: the configurations
this machine can run are the ones the frame cannot afford, and the one it can
afford -- single address, 5 clk/B, 92.2% total, 7.8% spare -- needs the device to
ACK the DMAC, which needs a request line MAME does not connect and the slot
pinout does have at B36/B37.

ROADMAP re-ranks accordingly. B3 stops being a constant to look up and becomes
DOES THE CARD DRIVE #EXREQ, ahead of B1: B1 sets how much headroom the player
has, B3 decides whether there is any. New E7 carries the other branch -- if the
answer is no, the container must reach 27,995 B a frame, 328 KB/s of payload,
against 438 now. The dependency diagram is redrawn around that fork.

The scope is stated rather than buried: this is the GATE container, deliberately
the heaviest thing the encoder emits, and the lighter cpufit family was NOT
priced -- 15_bus_occupancy.py refuses it, correctly, because the C68K measurement
it cross-checks against belongs to the gate container. E7 therefore begins with a
harness re-run, and until then "34% too big" is a statement about the fixture and
not about the project.

check.sh ALL GREEN before and after.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-25 00:17:29 -07:00
prosolis 621a5bb457 Put the data phase on the DMAC, and find auto-request is charged by time 
ROADMAP P4a. src/player/dma.i programs HD63450 channel 1 and takes the SCSI DATA
IN phase off the CPU; src/player/dmagate.s reads the same 2,048 B at LBA 1000
three ways -- PIO, the channel with the bus held, the channel stealing cycles --
and all three are byte-exact against the host's copy of the volume.

The evidence never reads $EA0015, because 57.3 established that it cannot: with
the DMAC's OWN asserted MAME cannot tell a CPU-driven byte there from a
DMAC-driven one. The discriminator is the CPU's own progress. MTC is sampled by
the INSTRUCTION AFTER the one that starts the channel, and held it reads 0 of
2,048 -- the whole transfer happened between two instructions, because the 68000
did not execute in between -- against the full count and 426 loop trips for the
stealing configuration. Put the stealing registers in the held slot and every
byte still arrives and tools/bench/dma_run.sh goes RED, which is what says the
counter can come out different; 58.3's vacuous "UNDERRUNS: 0/120" is the trap
being avoided. tools/analysis/27_dmac_config.py decodes the four register bytes
out of the player's own source, with the MC68450 field tables now in one copy
(tools/analysis/mc68450.py) shared with 21_iplrom_dmac.py, so the player's
configuration and the IPL ROM's 16..19 clk/B one are the same decoding.

Three bounds on the apparatus, read out of MAME 0.277 rather than inferred: the
CZ-6BS1 has NO request line to the DMAC (its flow control is DTACK), so external
request cannot be run; single address cannot be run either, because only channel
0 has device callbacks; and only burst is modelled as held. Of the four rows of
the W ladder exactly one -- dual address held -- has a code path here, and it is
the one demonstrated. W did not move by one clock, for the third session running.

What outlives the emulator is the currency. Every W in this project is clocks per
DELIVERED byte, which presumes the device asks; an auto-requested channel spends
its share of the bus whether or not a byte is there, so a record costs what it
costs to ARRIVE -- halve the delivery rate and the CPU cost of the same record
doubles. tools/analysis/28_autorequest_cost.py prices it from MC68450 3.8 and
5.2.3.3.2, gating its formulas against Table 5-3's sixteen rows first. At 37,405
B and an explicit 460 KB/s: max rate costs the whole 95.3% of a frame the record
takes to land, and of the GCR's four bus shares only BR=00, 50%, carries the
rate -- 10.61 clk/B, 47.6% of a frame, against 40.4% for the W=9 row and 391.8%
measured for PIO. The GCR is a design lever nothing in this tree had named.

59.4 changes what is left. sc_in_data now REFUSES a windowed read when the data
phase is the channel's (SCE_WINDOW), because a channel writes a contiguous run
and cannot drop the 300 B in front of a record. 117 of 120 records need one, so
sector-aligned records have gone from a preference in ROADMAP's re-encode bundle
to the precondition the transport enforces -- and that bundle is now the only
thing between this tree and M2.

One collision, recorded because the procedure is the finding: DM_USE first sat at
$18300, which is ring.i's XF_SLOT mailbox, and the P4b stage -- untouched by this
work -- went red on a run that never reached its snapshot. check.sh was ALL GREEN
before any of this, which is what made that red unambiguous. ALL GREEN after too,
with one new stage. decode.bin is unchanged at 1,296 B and the same MD5.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 23:55:07 -07:00
prosolis 5921fab118 Put the transport on the 68000, and find PIO costs 87 clocks a byte 
ROADMAP P4b. src/player/xfer.i answers src/player/ring.i's XF_* mailbox with a
real READ(10) to a real MB89352 in place of tools/bench/stream.lua's modelled
transport: 120 records, 4,488,588 B, pixel-exact out of a 256 KB ring, with a
real mid-stream seek in a second pass. The tiling is the SAME 18 wraps and
14.7 KB mean hole that 49.4's host producer and 55.4's modelled transport
produced -- a third transport, same placement, which is the assertion that
ring.i could not tell which side of the seam answered it.

What it costs is the finding. tools/bench/xfer_cost.sh subtracts the same 120
frames run twice and gets 87.28 clocks per delivered byte, against the 68000's
own cycle table for the loop, which says 87.15 -- 0.2% apart, so the cost is the
instruction stream and not MAME's device model, and it is the first number this
rig has produced that survives leaving the emulator. That is 391.8% of a 12 fps
frame; the machine's own V-DISP clock agrees from the other end at 2.57 fps.
Against the ladder, W=5 held is 22.4% of a frame and W=19 is 85.3%, so P4a is
worth 4.6x the worst DMA configuration in this tree and 17.5x the best -- where
before this session it was worth 9 against 19. W itself did not move by a clock.

"UNDERRUNS: 0/120" is vacuous with a synchronous transport, and stream.lua now
prints that argument next to the zero: a frame cannot start before its record
has landed because the decoder IS the transport. The counter that means
something is NO IDLE, 119/120 with a worst overrun of 441 whole ticks. Same
class of error as 49.7.2's free-running ring passing at 48 KB.

58.3: a record is not a sector -- 117 of 120 start part way into one, and
reading whole sectors into the ring corrupts the neighbours rather than wasting
bytes (49.2, no bounds check). scsi.i reads the covering sectors and stores only
the window, which is free in PIO and stops being free the moment P4a succeeds.
tools/analysis/26_sector_align.py prices the three ways out and sector-aligned
records win on both axes: +0.43% wire and zero clocks, against +1.34% and a
bounce copy at +5 clk/B. ROADMAP now carries a four-item re-encode bundle and
P4a should be attempted against a sector-aligned container.

check.sh gains two stages and was ALL GREEN before and after. decode.bin is
unchanged at 1,296 B and the same MD5.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 23:15:53 -07:00
prosolis e935d8661c Put the SPC on the 68000, and find P4 was blocked on a file nobody needed 
ROADMAP P4, first half. Every byte the player has ever consumed was placed in
emulated RAM by a host: decode.lua preloaded a container, stream.lua answered a
mailbox at a modelled rate. src/player/scsi.i selects a SCSI target on a real
MB89352 and issues READ(10) itself -- 4,096 B from LBA 0 and 2,048 B from LBA
1000, both byte-for-byte against the host's copy of the same volume, with no
IOCS and no host in the transfer path. The non-zero LBA is the half that
matters: a driver that emits a malformed LBA field still passes block 0,
because zero is what a malformed field usually is.

P4 was recorded as blocked in this tree and was not. Session 21's handoff said
MAME's x68000 has no MB89352 path; -exp1 cz6bs1 instantiates one next to the
HD63450, and FINDINGS 32.4 had read that card's DMA glue in session 9. The
session-21 note is a regression in the record, not a discovery. What is
genuinely absent is the 8 KB scsiexrom.bin MAME requires to INSTANTIATE the
card and the player never executes -- driving the SPC registers directly has
been the plan since BENCHMARK item 4 in session 2 -- so scsi_run.sh supplies a
zero-filled placeholder on its own rompath, leaves the user's romset alone, and
lets MAME print WRONG CHECKSUMS as it should. B3 is untouched: it wants that
ROM's bytes disassembled and a blank one has none.

The register map is measured, not inferred, and it corrects MAME's own
documentation. The probe walks $EA0000..$EA003F one address at a time with a
bus-error handler that records the fault and steps the index, because a
sequential dump reports the first hole as the answer -- the earlier version
took a bus error at $EA0006 and knew nothing about the other 57. 60 of 64
answer; the two holes are exactly the TMOD and EXBF the MB89352 omits and the
MB87030 has. MAME leaves HOLES and does not shift the later indices down, which
its own device summary claims it does, and that is what keeps DREG at $EA0015.

The data register is DMA-only here and a PIO write vanishes. x68k_scsiext.cpp
glues $EA0015 and nothing else, and with exown() asserted and DRQ low the byte
is discarded: no error bit, no status change, no interrupt. Quieting all four
DMAC channels does not change it. Measured rather than reasoned about -- write
$5A, read back $00 with the FIFO still empty -- because ten command bytes
vanishing without trace looks exactly like a target refusing a command, which
is how it first presented. So every transfer runs the SPC in DMA mode and the
CPU moves the bytes through the DMAC's own door.

That costs the argument something, and it is easy to overclaim here: with exown
asserted at idle MAME cannot distinguish a CPU-driven byte at $EA0015 from a
DMAC-driven one. This shows the DATA PATH and cannot by itself show that the
HD63450 is driving it, which is precisely what ROADMAP calls P4's first job.
Whether a real CZ-6BS1 also refuses PIO there is not settled; it is a property
of MAME's model and it wants a board.

W did not move by one clock, and could not have. MAME's device models are
functional rather than transfer-timing accurate and 42.5 reads its DMAC
configured in wall-clock attotimes, so this is BENCHMARK Tier 1 -- does the
read path work -- and never Tier 2. W is still the largest open number here.

Five bugs, four of them silent, recorded in 57.5 because the pattern is the
finding: a chain of rol.l #8 that loaded a transfer counter of ZERO from a
count of 10; a byte handed to a FIFO mistaken for a byte on the bus; a fixed
phase sequence where the bus decides the order; the discarded PIO write; and an
initiator that must drop ACK and only then release the bus. The last appeared
only once there were TWO reads -- one passed byte-exact and every conclusion
from it was sound, and the second could not select. A player issues one command
per record, so that failure would have been universal in the ring and invisible
in a one-read demonstration.

No decoder code changed; decode.bin is still 1,296 B at the same MD5. check.sh
gains a SCSI stage that builds the volume out of the same stream_disk.bin the
ring rig reads, gates the register window at 60 of 64 and both reads
byte-exact, and skips when chdman is absent. ALL GREEN before and after.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 22:53:58 -07:00
prosolis 00232bb22b Import the scene graph, and find the worst gap between two branch points is zero 
ROADMAP G1, scheduled early because it is a measurement input, and it paid for
that immediately. 51.3 established that a ring's lookahead is accumulated out of
pipe - wire and that a seek spends all of it, so what a branch point costs is
set by the time since the last one. 55.5 rehearsed a seek on the machine and
said out loud that it could not ask the question, because nothing here knew
where the branch points are.

They are everywhere, and 5.4% of them are free of charge to the player and not
to us. Over 612 distinct transitions into a seek, taking the earliest instant
each input window opens: worst 0.000 s, p10 0.950, median 3.473, best 82.497.
33 open on the first frame of a clip the disc SEEKED to -- press right as
flaming_ropes.enter_room appears and you are in fall_to_death, two seeks with no
play between them. 51.2's slack rule can therefore be answered NO by the
content rather than by the buffer, and no amount of ring is a defence.

It does not break the design. A branch on an empty ring costs the 2-record
prefill, 149.7 ms at 488 KB/s, not the climb. What it removes is margin: at that
rate in a 256 KB ring, 76% of this game's branch points arrive before the ring
has refilled, and a 512 KB ring makes that 90%, because doubling the ceiling
does not touch the surplus. The ring is not the lever; the surplus is.

CORRECTION to FINDINGS 16: there is only one transcription. The SNES chapter
set says in its own README that it is derived from DirkSimple, so the planned
diff of two independent sources catches conversion errors only. Run anyway:
durations agree 388/505 within a frame, branch structure 470/505, and of the 35
differences 16 are renames and 18 of the other 19 are that port dropping the
arcade's diagonals. Zero transcription discrepancies, and none were findable.

Two constraints on the input layer come free: the arcade needs eight directions,
and the shortest input window is 98 ms against 54.4's 72.13/90.16 ms frame slot,
so input cannot be polled on the frame tick.

The coupling to outside source is contained to one file (USER DECISION).
tools/import/scenegraph.py is the only code here that knows those projects
exist -- their paths, table names, timing formulas, constants -- and it writes
DLXSCENE1, this project's own schema, into gitignored tmp/ with the sources'
licences inside it. tools/analysis/25_scene_graph.py reads only that. Nothing is
vendored and nothing outside-derived is committed. The split was made after the
measurement and the whole output was re-run byte for byte to show it moved no
number.

Both import gates are negative-tested: deleting one sequence upstream fails the
516/906 count, and closing the table early fails the constructor-end check,
which replaced one that was vacuous.

No 68000 code ran or changed; decode.bin is still 1,296 B at the same MD5.
check.sh gains an import stage that skips when there is no checkout. ALL GREEN
before and after.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 22:11:04 -07:00
prosolis 2676f3b835 Put the ring on the 68000, and find the disc stops whenever the player is not asking 
ROADMAP P5. The loader moved in session 21 and the frame clock in 22; the ring
producer was the last policy living outside the machine. src/player/ring.i does
`aligned` placement, the descriptor ring, a prefill, 51.2's slack rule and a
seek, and the host keeps only the transport.

It needed a container change. `aligned` asks whether the next record fits
before the end of the ring -- a length asked BEFORE the record is fetched -- and
every reader in this tree answered that by walking the frame stream, which is
exactly what a player streaming off a disc cannot do. DLX4 carries nframes u16
record lengths in the scene header. Frame payloads are byte-identical to the
DLX3 encode, so no fitted constant moves; the scene header goes 5,920 to 6,164 B.

The producer reproduces the host's tiling exactly: 18 wraps, 14.7 KB mean hole,
pixel-exact, a third independent implementation of the same policy.

What it exposed is bigger than the item. A channel only moves bytes while it has
a request and only the CPU can issue one, so the disc stands still between
records by an amount the PLAYER sets, not the medium -- and no host-filled run
could see it. At 488 KB/s in a 256 KB ring a one-deep request queue gives away
6.8% of the pipe and underruns 59 of 120 frames; two-deep gives away 3.4% and
underruns none. The container's whole surplus over the wire is 8.7%, so the
player's own loop was spending most of the slack a branch point saves up.
Prefill is the weaker lever: six records of it still leaves 24 underruns.

Three silent bugs are recorded in FINDINGS 55.7 -- all produced wrong pixels or
a desync rather than a fault -- plus a rig one: MAME renders a screen line by
line, so snapshotting the frame the decoder finished in captures a tear that
reads exactly like a decoder bug.

check.sh gains the machine-owned ring and a seek with the decode after it.
decode.bin is unchanged at 1,296 B and a host-filled run executes none of the
new code, so every FINDINGS 49/51 figure stands. ALL GREEN before and after.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 21:45:05 -07:00
prosolis c419251266 Put the frame clock on the 68000, and find that the 12 fps frame does not exist 
ROADMAP P3 said "needs MFP timer or VBL" and neither can do it.  The MFP's
timer clock is 16 MHz/4, its prescalers stop at 200 and its data register is 8
bits, so the slowest tick any single timer can make is 78.125 Hz -- 6.5x faster
than a frame -- and 4e6/12 is not an integer, so no setting reaches 12 Hz at
all.  The raster has no whole divide near 12 either: 4 refreshes is 13.86 fps
and 5 is 11.09.  tools/analysis/23_frame_clock.py walks all 7x256 timer settings
rather than asserting it.

src/player/clock.i takes the V-DISP falling edge on MFP GPIP4 -- the start of
vertical blanking, which is when a player would present -- and adds fps*VTOTAL
per edge to a 16-bit accumulator, emitting a tick at 31,500 and keeping the
remainder.  The long-run rate is fps*VTOTAL/VTOTAL = 12.000000 fps exactly, and
both constants are read out of the CRTC at init, so the clock is derived from
the registers that generate the raster it counts.  Measured over 3,000
refreshes: 3,000 interrupts, 649 ticks where 649.1429 were due.

It costs 181.35 clocks per V-DISP, 838 per frame, 0.1006% of the budget -- timed
by the 68000 itself, because the host's granularity is 17.64 ms and the
interrupt is microseconds.  The loop's own cost was calibrated rather than
looked up and landed on 38.000002 clocks, which both licenses the subtraction
and confirms buscost.py's model; the 181.35 then decomposes exactly, leaving
43.99 clocks for the interrupt exception -- the textbook 44, measured.

THE ONE THAT MOVES SOMETHING: 12 fps on a 55.4577 Hz raster is 4.6215 refreshes,
so a frame is shown for 4 refreshes (72.13 ms) or 5 (90.16 ms), 37.9% of them
short.  The 833,333-clock budget every figure in this project is priced against
is the MEAN slot, and the short one is 13.4% under it.  The cadence was already
in the tree unnamed: stream.lua's tick is sampled at frame boundaries, so its
gaps were always 4 or 5, and every host-paced result in FINDINGS 49/51 carried
it.  P3 moved who produces it onto the machine and made it visible.  It is not a
dropped frame -- the pace gate lets an overrun eat the next frame's idle -- and
on the gate container it costs 4 frames of 120 their idle against 1 for the
nominal model, most of that the frame-0 transient at 111% of budget.  stream.s
counts it now, and the rig matches an offline model of the divider exactly.

Also struck: MAME's raster runs 2.22% fast.  refresh_mode() builds the frame
period from scr.max_x*scr.max_y with scr.max_x = m_htotal - 8, one character
cell short and an inclusive bound used as a count, so it runs at 56.6901 Hz
where the registers say 55.4577 -- agreeing to six digits with the arithmetic.
Every "1/55.46 s granularity" note in this tree was wrong and is 1/56.69 s,
corrected in six files with the derivation put once in crtc_mode.lua.  No
conclusion changes and no 68000 cycle figure moves; the CPU clock is unrelated
to the screen.  But anything paced by the raster runs fast under MAME, so the
rig reports both rates and prices the interrupt against the hardware's.

decode.s and frame.i are unchanged; decode.bin is still 1,296 B at the same MD5.
The pace gate's wait loop is byte-for-byte the one FINDINGS 51 measured and the
free-running path executes none of the new code.  check.sh gains two stages: the
clock's own measurement, and 120 frames decoded pixel-exact with nothing outside
the machine deciding when a frame may start.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 20:55:34 -07:00
prosolis 7179339bd2 Move the loader onto the 68000, and find 5,920 bytes nobody counted 
src/player/load.i expands both codebooks to word-per-pixel form and packs the
palette to GGGGGRRRRRBBBBBI out of the RAW container header, byte-exact against
tools/bench/dlxload.py on both CPU cores.  The palette half is gated on words
read back out of the palette registers at $E82000, so "the words reached the
hardware" is part of what passes.  ROADMAP P1 is done; P2's encoder half (a
reserved black entry, 23.4) is not, and is a re-encode rather than an edit.

A scene change costs 18.96 ms of 68000 time, 22.8% of one 12 fps frame; boot
costs 24.70 ms.  The scratch tables describe the CRTC, not the scene, so
pal_tables is a separate entry point built once at boot -- 5.29 ms off every
scene change.

The one that moves something: the scene header is 5,920 B that no rate table in
this tree included, because it belongs to no frame record.  In FINDINGS 51.3's
currency it is divided by the surplus pipe - wire, so it is hypersensitive:
138 ms of extra refill climb at 488 KB/s and 1.099 s at 451.4 KB/s, for the
same bytes.  tools/analysis/22_scene_load.py prices it across explicit rates.

Recorded as open: the two CPU cores agree to <3% on every stage but the table
build, where they differ by 16.4%.  px68k's C68K charges a flat 50 clocks for
MULU/MULS (c68kmacro.h:1869) where the 68000 charges 38+2n, which explains
4,608 of the 8,703 clock gap.  4,095 clocks are unexplained.  Nothing else in
src/player/ multiplies, so no figure in FINDINGS 24-52 is affected.

decode.s and stream.s are untouched; decode.bin is still 1,296 B at the same
MD5.  check.sh gains a stage that gates byte-exactness on both cores and
deliberately does not gate the cycle counts -- MAME's clock is 1/55.46 s and a
wall timing would make the green light host-sensitive.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
 2026-08-24 20:20:40 -07:00
98 changed files with 23435 additions and 437 deletions
Expand all files Collapse all files
.gitignore
+1
View File
@@ -11,6 +11,7 @@ roms/
__pycache__/
*.pyc
*.dlx
*.dlxp
a.out
tmp/
tools/bench/c68k/c68k_bench
README.md
+406 -239
View File
@@ -2,229 +2,398 @@
Porting Dragon's Lair to a stock X68000 (68000 @ 10MHz, 2MB, SCSI).
This is fundamentally a **video codec problem**, not a game-logic problem. The
game logic is a scene table with branching input windows; the difficulty is
pushing ~22 minutes of Don Bluth animation through a 10MHz 68000.
The game logic is a scene table with branching input windows. The difficulty is
pushing ~22 minutes of Don Bluth animation through a 10MHz 68000, and it is a
**delivery** problem rather than a compression one — bus cycles and bytes off a
disc.
**The live design has no decoder in it.** The disc holds packed 8-bit frames and
a DMA channel walks them into GVRAM with the CPU halted. Measured on the machine,
decoding 37,585 bytes costs more than not decoding 49,152: at the 9 clk/B
dual-address floor the codec is 110.4% of a 12 fps frame and the decoder-free
packed player is 54.9%.
A working codec is in the tree and is **parked** (USER DECISION). Both branches
are described below; the packed one is where the work goes.
## What it looks like
![Blu-ray source next to the 68000's output](docs/img/source-vs-decoded.png)
**The packed player, running off a real volume, with sound.** `src/player/packed.s`
on an emulated stock X68000: the 68000 brings up its own display, builds its own
193-entry DMA chain, keeps its own frame clock off the CRTC's V-DISP, and fetches
every record itself with `READ(10)` from a real MB89352. **There is no decoder in
the machine.** Source on the left; on the right, MAME's own snapshots of what the
68000 put on screen. **120 of 120 frames are pixel-exact against the container**,
every one compared, and the clip refuses to build otherwise.
Left, the Blu-ray frame cropped to 256x192. Right, the same frame **as the
emulated 68000 actually drew it**: 256 colours out of the X68000's 65536, one
16-colour-per-4x4-block codebook, decoded by `src/player/decode.s` from the
container. Not a re-render. These are the pixels MAME had on screen, pulled out
of its own snapshot, 2x nearest-neighbour, no filtering.
<video src="docs/img/packed-player.webm" controls muted loop width="100%"></video>
**The player, running.** 119 frames out of a **256 KB ring buffer on an emulated
stock 2 MB X68000**, paced to a 12 fps frame clock, streamed from a host file at
488 KB/s by `src/player/stream.s` with no Lua in the decode path. Source on the
left, the machine's screen on the right.
[`docs/img/packed-player.webm`](docs/img/packed-player.webm) (120 frames, 12 fps,
VP9, with the chip's own audio)
<video src="docs/img/player.webm" controls muted loop width="100%"></video>
**The clip is a composite of two runs and that is the point.** The picture is the
gate run — paced at half rate so each snapshot lands inside the write window,
cycle stealing, no sound. The sound is the audio run — the same container at
12 fps with the MSM6258 on channel 3, captured by MAME off the speaker, cut at
the first sample the chip produced and gated sample-exact against lump 0 before
anything is written. **What it is not is a real-time capture of the shipping
configuration**, which at this container's own burst rate would show a blank
layer for 99.5% of every slot (FINDINGS 64.2, below).
[`docs/img/player.webm`](docs/img/player.webm) (119 frames, 12 fps, VP9)
![Blu-ray source next to the packed player's own screen](docs/img/packed-player.png)
116 of those 119 frames are **pixel-exact** against `tools/encoder/dlx.py`'s
reference reconstruction. The other three are **torn**: the top of the picture
is frame *n* and the bottom still holds frame *n-1*, because MAME captured the
screen while the block loop was partway down it. That is not a rig artefact.
`decode.s` writes straight to the displayed page, so a real player tears the
same way. `tools/media/make_readme_media.py` asserts the tear rather than
trimming it: every differing pixel has to come from the previous frame, or it
refuses to build.
The still is the frame whose PSNR is closest to the mean over the gated window,
so it is not the flattering one: **33.13 dB against the window's mean of 33.10**,
which is the packed container's predicted GRB555 figure to the digit.
**What the decoder is doing.** The same window with the block-mode map beside
it. **Black is SKIP** (costs nothing, draws nothing, the previous frame stands),
**blue is V1** (one codebook index for a whole 4x4 block), **amber is V4** (four
indices), **red is RAW** (sixteen bytes verbatim). The mode mix is what every
cost table in `docs/FINDINGS.md` is really about: V4 costs 1.5x V1, and the mode
decision is charged both bytes *and* cycles, which is why a byte-rich profile
buys its way out to RAW rather than V4.
**Name the layer.** Everything here is **emulated**: MAME 0.277 `x68000`,
`-bios ipl10`, stock 10 MHz / 2 MB, with every 68000 cycle figure cross-checked
on a second CPU core (px68k's C68K) and the display path cross-checked against
px68k's own `gvram.c`. **Nothing in this project has run on real hardware yet.**
<video src="docs/img/modes.webm" controls muted loop width="100%"></video>
## How the packed player works
[`docs/img/modes.webm`](docs/img/modes.webm) (the same 119 frames, with the mode map)
**256-colour GVRAM throws away the high byte of every word a CPU writes**, so a
picture byte normally costs two disc bytes. **CRTC R20 bit 11 turns the masking
off**, and with the two 256-colour pages scrolled apart one word carries two
pixels (FINDINGS 46/47). That makes a frame 1.0 B/pixel — 49,152 bytes — and it
makes the frame a *literal*: no codebook, no recursion, no decode.
**Name the layer.** Everything above is **emulated**: MAME 0.277 `x68000`,
`-bios ipl10`, stock 10 MHz / 2 MB, cross-checked frame for frame on a second
CPU core (px68k's C68K). Nothing in this project has run on real hardware yet.
A record is that picture plus its own 256-entry palette: **49,664 bytes, which is
97 sectors exactly**. A packed record's length is geometry, so **the container
carries no index and no length word** — record *i* is at `off + i*rec` and a seek
is arithmetic. A DMA channel fills GVRAM in buffer mode straight off the disc and
**walks the 1,024-byte line stride itself** through array chaining, so a frame is
**one channel start and not 192** — and the chain crosses from the palette
registers at `$E82000` into GVRAM in the same start (FINDINGS 62). The array is
scene-constant, because the packed layout spends both 256-colour pages and there
is no page to flip.
**Audio rides a fixed cadence, not the record.** The MSM6258 wants 7,812.5 B/s
and 12 fps wants 651.0417 B a slot, a ratio with no arithmetic relationship, so
audio in the record would make records variable-length and bring the index back.
DLXP2 groups instead: **one 14-sector audio lump, then 11 records**, so
`record i = off_frm + i*rec + (i//11)*7168` — six instructions once a frame. The
lump is 7,168 bytes of space and eleven frames of audio is 7,161.4583…, so **the
payload alternates 7,161 and 7,162** and a player that fed the chip whole lumps
runs 0.09% fast: **1.25 s of lip-sync drift over the game's 22.8 minutes**,
predicted, and played at 1.26 (FINDINGS 67.2/68.3). The player carries one
accumulator instead, which is the frame clock's shape for the frame clock's
reason.
**The wire is fixed and there is no lever on it: 582.0 KB/s of picture +
7.64 KB/s of audio cadence = 589.6 KB/s.** A codec's bitrate is adjustable; a
literal frame's is geometry, and no scene costs less than another.
| | |
|---|---:|
| record | 49,664 B = 97 sectors |
| wire | **589.6 KB/s** |
| picture, as the display renders it | **33.10 dB** mean over the gate window |
| CPU per frame, video path | one channel start and one `READ(10)` — no paint |
| cost at the 9 clk/B dual-address floor | **54.9% of a 12 fps frame** (FINDINGS 64.2) |
| the codec, at the same floor | 110.4% (FINDINGS 61.4) |
## Where it stands
**The binding resource is the 68000's local BUS, not its clock.** The decoder
occupies 86.7% of it once instruction prefetch is counted, and 52 of the 53
frames that miss the 12fps budget miss on the bus (FINDINGS 38). Read that
before optimising anything for cycles.
**What runs, end to end, on the emulated machine off a real volume:**
**The decoder works and is measured.** `decode.s` draws blocks and v7 literal
spans pixel-exact under both CPU cores, and costs inside the player what the
standalone blit benchmark said it would, to 0.2% (FINDINGS 41).
- **120 of 120 frames pixel-exact**, in both palette orders, every frame compared
rather than the last — a packed frame is a literal, so frame 119 says nothing
about frame 60 (FINDINGS 64.1).
- **All 78,125 B of a container's audio, read back off the speaker**, sample-exact
against the four ADPCM axes carried in the container's own header, one
delivered byte at a time — because a player's counters all stay right through
a bug that overwrites the buffer the channel is reading (FINDINGS 68.1).
- **A mid-stream seek with sound on it**: `pg_aseek` rebuilds the lump index, the
stream position, the remainder accumulator and the byte offset into the group
and issues a second read — **132,162 B of spliced stream accounted for byte by
byte** across a branch deliberately not on a group boundary (FINDINGS 71.1).
- **The scene graph**: 40 scenes, 516 sequences, 906 input windows, imported from
the arcade's own graph into this project's schema (FINDINGS 56).
**The delivery path works too.** `stream.s` decodes the whole 120-frame window
out of a 256 KB ring on a stock 2 MB machine, final frame pixel-exact, with the
container in a host file rather than preloaded into RAM. The constraint is
**contiguity, not byte count**: the block loop reads with a monotonically
increasing `a0` and no bounds check, so the ring needs the whole next record
resident *and contiguous*, a condition no byte-counting buffer simulation can
see (FINDINGS 49).
**The binding resource is the 68000's local BUS, not its clock**, and every item
above is priced in one of four units:
**Seek slack is accumulated, not owned.** A ring's lookahead is built out of
`pipe - wire` and a seek spends all of it. At 488 KB/s a 256 KB ring needs 4.83
seconds of play to reach its 7-frame ceiling from empty, and 512 KB needs 8.42
seconds to reach 14, so a bigger ring raises the ceiling *and* lengthens the
climb. A branch point therefore asks "has there been enough play since the last
one", not "is the buffer big enough" (FINDINGS 51).
| resource | state |
|---|---|
| **68000 local bus** | the binding one. The codec's decoder occupies 86.7% of it, and 52 of the 53 frames that miss the 12 fps budget miss on the bus (FINDINGS 38). |
| **68000 clocks** | measured, on two independent cores. |
| **delivery rate** | **no working figure, deliberately** (FINDINGS 50, USER DECISION). `--bus`, `--kbps` and `DLX_STREAM_KBPS` are required arguments with no defaults, so no table can be scored against a rate its own output does not state. |
| **seek time** | **no figure at all, and never had one.** |
| **W, clocks stolen per delivered byte** | 5 single-address held, 9 dual held, 12 single arbitrated, 16..19 for the IPL ROM's own disk channel. **Still the largest open number.** The CPU doing the transfer itself is **87.28 clocks a byte, measured** — 4.6x the worst DMA configuration found here and 17.5x the best (FINDINGS 58). |
**There is no working delivery rate figure, deliberately.** `--bus`, `--kbps`
and `DLX_STREAM_KBPS` are required arguments with no defaults, so no table can
be scored against a rate its own output does not state. What replaces a constant
is a requirement: `tools/analysis/19_ring_stream.py` reports the **zero-prefill
pipe**, the rate a medium must clear for a container to need no prefill, which
is **513.2 KB/s** for the current candidate. That is a hardware acceptance test
to measure a BlueSCSI against (FINDINGS 50).
### The open question that decides which player gets built
**The largest open number is W, the clocks stolen per delivered byte.** The
MB89352 is an 8-bit SPC, so the DMAC pays per byte rather than per word, which
is a 2x correction the project has already paid for once (FINDINGS 43). What W
costs is set by how the player programs the DMAC: 5 clocks a byte single
address with the bus held, 9 dual address held, 12 single address arbitrating
per byte, 16..19 dual address arbitrating per byte. The design's fate changes
completely across that ladder, and it is ours to choose.
**The write window is the frame.** A packed write *requires* R20 bit 11; buffer
mode blanks the layer while it is set; and a DMAC-direct player holds the window
open for the whole data phase, because there is no second page to hide behind.
**The one worked example on the machine is expensive.** The X68000 IPL ROM
programs all four HD63450 channels itself, and
`tools/analysis/21_iplrom_dmac.py` decodes that configuration out of the ROM
image and gates on the bytes still being there. Both the audio channel and the
on-board disk channel are dual address, 8-bit port, cycle steal *without* hold,
one external request per byte: **16..19 clocks a byte**, the top of the ladder.
For audio that is a settled figure and a small one, 1.25%..1.48% of a frame. For
the disk it is where nothing fits at any container size. The ROM drives SASI
rather than the MB89352, so it does not settle W, but a cheap configuration is
now the thing that has to be shown rather than assumed (FINDINGS 52).
dark fraction of a slot = record bytes / (DATA-PHASE rate x slot)
**Current encode:** 496.7 KB/s at 29.19 dB, 1 frame of 120 over the 12fps
budget, and that one is frame 0, the intra frame, late on purpose.
**The rate in that expression is the BURST rate, not the sustained one**, and at
the container's own 582.0 KB/s the dark fraction is **1.0**: every frame
delivered, on time, pixel-exact, and none of them displayed. Free-running, the
run reported exactly that — **the window was open on 99.5% of host frames**.
**Green-light check:** `./tools/bench/check.sh` (~3 min, needs the Blu-ray
mounted) re-runs both display regression tests, the rate-control drift gate, the
display-path coherency counterexample, a 120-frame 68000 decode on two CPU
cores, the ring and paced-ring passes and the DMAC configuration gate, then
prints `ALL GREEN`.
It **reverses the ranking** in clocks, so there are two packed players and the
difference between them is *when* the window is open:
| | clocks (W=9) | on screen | RAM |
|---|---:|---:|---:|
| **A — DMAC-direct** (built, FINDINGS 64) | **54.9%** of a slot | 0% at the container's wire, 72.7% only at 2,131 KB/s | none |
| **B — DMA to RAM + CPU paint** (K4, not built) | 82.2% | **72.7% at any rate** | 99,328 B |
B's paint is **measured**: the packed `movem` blit is 227,553 clocks, 27.3% of a
slot, independent of the medium. They are equally visible only at **3.7x the
wire**. So B2 — *does buffer mode blank the display?* — decides which player
exists rather than how much headroom one has, and **K4 is not built until it is
answered** (FINDINGS 64.2, ROADMAP K4).
### What else a player has to carry
**A held channel costs the frame clock half its ticks, without the clock being
able to tell.** `clock.i` counts V-DISP interrupts; a held channel halts the
68000; the MFP's pending bit is one bit. Held at 12 fps, **487 of 1,038 edges are
lost** and the player reports **zero late frames**, because the tick it grades
itself against is advanced by the interrupt the channel stopped it from taking.
It believed it was at 12 fps; the screen was at **6.37** (FINDINGS 64.3).
**Held, the bus costs the audio 463 times the seam.** The MSM6258 has no
starvation state — it goes on decoding nibbles out of whatever byte its data
register still holds — so the interval between a channel counting out and the CPU
arming the next lump is a held nibble pair driving the predictor. Stealing, that
seam is **0.51 ms over ten seconds**, because the audio service runs from inside
`dma.i`'s transfer wait. Held, it is **236 ms, 2.31% of the audio**, with every
byte still correct (FINDINGS 68.2).
**The chip is not the datasheet and not ffmpeg, and it was asked.** Sixteen
candidate decoder models were fitted to what the emulated MSM6258 produced from a
designed nibble stream; **exactly one reproduces it sample-exact**, with a
negative control on every axis. The chip runs the datasheet's truncation, takes
the **low** nibble first, clamps its accumulator at **10 bits** and starts it at
**−2**. Getting the delta formula wrong costs −2.88 dB; **getting the nibble order
wrong costs −25.74 dB** (FINDINGS 66). The four axes ride in the container header
as fields rather than a version number.
**The audio level is measured off the whole disc and does not change.** All 201
streams that have audio, 21.5 minutes: the **disc peaks at 946 of 2048, 5.35 dB
over the chip's clamp**, in 402 events totalling 44.0 ms. Forty windows encoded at
six gains price the choice, and the disc's own level has the **best mean SNR
(22.03 dB)** — the gain that guarantees zero clamping costs 0.85 dB across the
game to buy back 1.90 dB on the 2.11 seconds that clamp, because the OKI step
table's floor is a constant and does not scale (FINDINGS 69).
**The predictor does not seek.** The MSM6258's accumulator is a pure integrator
with no leakage term, so a branch that hands the chip bytes chosen for a state it
is not in produces **a DC offset that does not decay**: playing through, −355 of
511 with AC 0.00, still −108 four seconds later; stopping and re-PLAYing, a single
permanent constant of −65. **A re-PLAY is 5.5x better and neither is zero.** The
only fix that reaches zero is the encoder's — reset the predictor where a branch
can land — and it costs **0.33 dB** (21.99 → 21.66) because the step table
re-converges in a few samples (FINDINGS 71).
**Branch points do not wait for the buffer.** 5.4% of the game's 612 branch
transitions open an input window on the first frame of a clip the disc *seeked
to*, so two seeks can fall back to back with no play between them. On the packed
branch there is **no refill climb at all** — a record goes straight into GVRAM and
the video lookahead is zero records — so the only consumer with any lookahead is
the audio one: **1.833 s of sound held against 0.000 s of picture** (FINDINGS
56/70.2). And a DLXP2 group puts its lump *in front* of its records, so a seek
lands with its audio behind it: **mean 416.5 ms of silence entering a branch**
over the arcade's 409 within-container targets, which is what `pg_aseek`'s second
read removes for 11.7 ms (FINDINGS 70.3).
### The parked branch: the codec
The codec is a Cinepak-style hybrid — each 4x4 block coded as SKIP, one 4x4
codeword, four 2x2 codewords, or RAW literal pixels, chosen per block by
rate-distortion, with a v7 literal-span mode over the top. It works, it is
measured, and it is **pixel-exact on the 68000 under two independent CPU cores**;
`stream.s` decodes a 120-frame window out of a 256 KB ring on a stock 2 MB
machine with the container in a file rather than in RAM.
<video src="docs/img/player.webm" controls muted loop width="100%"></video>
[`docs/img/player.webm`](docs/img/player.webm) — the codec player, 119 frames,
12 fps. 116 are pixel-exact against the reference decoder; **three are torn**,
frame *n* on top of frame *n-1*, because MAME captured the screen while the block
loop was partway down it. `decode.s` writes straight to the displayed page, so a
real player tears the same way, and the media builder **asserts the tear** rather
than trimming it. [`docs/img/modes.webm`](docs/img/modes.webm) is the same window
with the block-mode map beside it — black SKIP, blue V1, amber V4, red RAW —
which is what every cost table in FINDINGS is really about.
**Why it is parked.** Its remaining path is a conjunction and the packed one is
not: reaching a fitting rate needs a 35% byte reduction, two of its three levers
measure inert, and the reward on success is a design at ~100% of the frame
against the packed player's 54.9%. **It is kept on disk and not built on** — if
B2 comes back "buffer mode blanks", it is the only thing left (FINDINGS 61.8,
48.3).
## What is open
**Hardware — this list is the user's, and nothing here can be settled by an
emulator.**
- **B1. Measure the medium.** Three thresholds, not one: **sustained ≥ 589.6 KB/s**
or frames arrive late; **the data-phase BURST rate**, which decides how much of
the slot the picture is on screen; and **seek time**, which has no figure at
all. Plus what one extra SCSI command costs, which decides the audio cadence.
The 0.7–1.7 MB/s usually quoted for BlueSCSI on an X68000 is **folklore with no
published benchmark behind it**.
- **B2. Does buffer mode blank the display?** `probe_bit11_blank.lua` is written
and settles it in minutes on a real board. It decides A vs B above.
- **B3. Does a real CZ-6BS1 drive `#EXREQ`?** MAME's card has no request line to
the DMAC at all, so every configuration that can be run against it is
auto-requested and **charged by time rather than by byte**. A real request line
is what single-address 5 clk/B needs.
- **B4. Does a real palette register take a byte write?** A negative answer costs
0.28% of a frame and nothing else.
- **The MSM6258V, on silicon**: the four axes; whether it resets accumulator, step
index and nibble select on PLAY only when it was not already playing; and
whether it goes on asserting `#DRQ` while STOPped.
**Software, in order.**
1. **The predictor-reset container (DLXP3).** The cheapest thing that takes a
measured cost to zero rather than down. The player half already exists
(`PG_ARST`).
2. **The audio buffering depth.** `PG_ANBUF` is 3; two slots is one constant and
one run. The audio buffer is the packed branch's *only* buffer.
3. **The silent-clip and short-audio cases**: a scene whose audio is shorter than
its frames, and a scene with no audio track at all.
4. **The cadence pick.** With a working seek path the silence F=11 costs is zero,
so the trade is padding against RAM — plus one SCSI command per branch, which
is B1's.
**Parked, so it is not re-proposed:** the codec's remaining encoder work (E7, E4,
C1); `ring.i`, `xfer.i` and most of `stream.s`, which a DMAC-direct player has no
use for because it has no ring, and P4a's wiring with them. **K4 is blocked
on B2**, not parked.
## Reproducing this
**No media ships in this repo and none of it is redistributable.** Bring your
own Dragon's Lair Blu-ray. Everything else needed to rebuild every number and
every picture above is either here or is packaged.
You need:
**No media ships in this repo and none of it is redistributable.** Bring your own
Dragon's Lair Blu-ray. Everything else needed to rebuild every number and every
picture above is either here or is packaged.
| | |
|---|---|
| the disc | loop-mounted read-only: `udisksctl loop-setup -r -f DRAGONS_LAIR.iso`. The tree was built against a decrypted UDF 2.x image. 7-Zip cannot read UDF 2.x, so use the loop mount |
| the disc | loop-mounted read-only: `udisksctl loop-setup -r -f DRAGONS_LAIR.iso`. Built against a decrypted UDF 2.x image; 7-Zip cannot read UDF 2.x, so use the loop mount |
| `python3` | plus **numpy** and **Pillow**, and nothing else. The k-means is hand-rolled rather than pulling in sklearn |
| `ffmpeg` / `ffprobe` | frame extraction, and the clips above |
| **MAME** | tested on 0.277, with the `x68000` ROM set. The rigs drive it headless via `-autoboot_script` |
| vasm (m68k, Motorola syntax) | **vendored**: `tools/vasm/vasmm68k_mot` is a Linux x86-64 binary, with the source tarball beside it to rebuild elsewhere |
Then:
```sh
export DLX_BDROM=/path/to/your/mounted/bluray # if not /media/$USER/BDROM
./tools/bench/check.sh # ~3 min, prints ALL GREEN
./tools/bench/check.sh # the green light, prints ALL GREEN
```
`DLX_BDROM` is honoured by every tool that reads the disc. Two stages are
optional and **skip rather than fail** when their input is absent, because both
live outside this repo:
`check.sh` re-runs everything above that a host can re-run: both display
regression tests, the rate-control drift gate, the display-path coherency
counterexample, a 120-frame 68000 decode on two CPU cores, the ring and paced-ring
passes, the DMAC configuration gate, the load-time transforms on both cores, the
scene-graph import when a DirkSimple checkout is present, the packed container
(with audio, and a silent control beside it) rendered through px68k's own GVRAM
model, **the packed player for 120 frames off a real volume with every frame
compared**, the audio encoder against ffmpeg's decoder, the sixteen-way decoder
identification against the emulated chip, the DLXP2 container and each of its four
axes with a negative control on each, the audio level off every stream of the
game's own footage, and the refill climb against the arcade's own 612 branch
points. `DLX_BDROM` is honoured by every tool that reads the disc.
- `PX68K=/path/to/px68k` for the second-CPU-core gate. This is the cheapest
strong test in the tree (seconds, no MAME, no ROMs) and it is what licenses
the bus and cycle figures.
- `IPLROM=/path/to/iplrom.dat` for the DMAC configuration gate. Defaults to
`~/mame/roms/iplrom.dat`.
Two stages **skip rather than fail** when their input is absent, because both live
outside this repo: `PX68K=/path/to/px68k` for the second-CPU-core gate — the
cheapest strong test in the tree, and what licenses the bus and cycle figures —
and `IPLROM=/path/to/iplrom.dat` for the DMAC configuration gate.
To rebuild the stills and clips in `docs/img/` you also need a paced recording
run; see the header of `tools/media/make_readme_media.py`.
The player runs and the media are rebuilt from them:
```sh
bash tools/bench/packed_run.sh # the packed player, 7 runs
python3 tools/media/make_packed_media.py # docs/img/packed-player.{png,webm}
python3 tools/media/make_readme_media.py <c.dlx> # the parked codec's stills and clips
```
Both media builders **gate before they write**: a still or a clip of the player is
a claim that the player drew it, so every frame is checked pixel-exact against the
container first and the audio cut is checked sample-exact against lump 0. A README
that illustrated a pixel-exact player with an approximate picture would be a small
lie about the one property this project keeps testing.
**Scene selection is a hard-coded stream number, not a search.** The gates use
streams `00020` and `00223` of the disc's 224 `.m2ts` files. A different
pressing may number them differently, and if so the green light will extract the
wrong footage rather than fail, so check that `tmp/fr_singe/` looks like the
Singe encounter before trusting any figure.
streams `00020` and `00223` of the disc's 224 `.m2ts` files. A different pressing
may number them differently, and if so the green light will extract the wrong
footage rather than fail, so check that `tmp/fr_singe/` looks like the Singe
encounter before trusting any figure. **Not every large stream is game footage**:
`00216` is the feature with a burned-in commentary picture-in-picture and `00215`
is the commentary itself, the two largest files on the disc. The clean
9.4-minute animation is **`00223`** (FINDINGS 25.1).
**Not every large stream is game footage.** `00216` is the feature with a
burned-in commentary picture-in-picture and `00215` is the commentary itself,
the two largest files on the disc. The clean 9.4-minute animation is **`00223`**
(FINDINGS 25.1).
**Nothing outside-derived is committed here.** The scene graph is regenerated from
a reader's own clones into gitignored `tmp/`, and `tools/import/scenegraph.py` is
the single file in the repo coupled to those projects — everything downstream reads
`DLXSCENE1`, this project's own schema, with the sources' attribution carried in
it. DirkSimple is zlib (Ryan C. Gordon); the SNES chapter set is MIT (Chad
Doebelin) and, by its own README, *derived* from DirkSimple rather than an
independent transcription.
## Encoder
## Encoders
```
python3 tools/encoder/extract.py 00020 /tmp/fr 12 crop
python3 tools/encoder/encode.py /tmp/fr out.dlx --profile scsi --preview p.png
**The live one is the packed container**, and it has no rate control in it because
there is no rate lever:
```sh
# the gate window: stream 00223, 10.0 s from 539.4 s, the same seconds both times
python3 tools/encoder/extract.py 00223 tmp/fr_singe 12 crop 539.4 10.0
python3 tools/encoder/extract_audio.py 00223 tmp/au_singe.raw 15625 539.4 10.0
python3 tools/encoder/pack.py tmp/fr_singe out.dlxp --audio tmp/au_singe.raw
```
The codec is a Cinepak-style hybrid: each 4x4 block is coded as SKIP, one 4x4
codeword, four 2x2 codewords, or RAW literal pixels, chosen per block by
rate-distortion. The RAW escape means `lam=0` is pixel-exact against the
palettised frame, so the quality knob spans lossless to heavily compressed
without changing the bitstream.
`dlxp.py` is the format and the one place the interleave, the 97-sector record,
the audio cadence and the lump payload are stated. It encodes in ~3 seconds
because there is no k-means in it. `--scene-palette --no-palette` is the fallback
container: **30.79 dB, zero palette churn, 576.0 KB/s**, still ahead of the codec
as the display renders both, and it is what ships if B2 says the layer blanks —
because a per-frame palette changes **231 of 256 entries every frame**, and a
picture under the neighbouring frame's palette is **12.8 dB worse**
([`docs/img/palette-mismatch.png`](docs/img/palette-mismatch.png)).
**Two byte budgets, not one.** `--kbps` is the quality rate point and
`--span-kbps` is the ceiling the span pass may draw on. They are different
things: the profile is chosen, the pipe is hardware, and bytes between them buy
a better picture if spent on `lam`, the 68000's deadline if spent on spans, and
nothing if left unspent. Spans run before `mu` because a span pays in bytes and
`mu` pays in picture (FINDINGS 41.2).
`adpcm.py` is the MSM6258 codec and it carries **two decoders on purpose**: the
module defaults are ffmpeg's, so `tools/bench/verify_adpcm.py` stays a check
against an independent implementation, and `adpcm.CHIP` is the set measured out of
the machine's own chip. Anything that encodes *for* the machine passes `CHIP`
explicitly.
**Two ceilings, on two different axes.** The second is the 68000's decode
budget: `mu` is bisected per frame against 833,333 cycles so the frame also
*decodes* in time, which takes the worst sustained window from 37 frames over
budget to 1, for 0.62 dB at `scsi` (FINDINGS 31). It is on by default and
`--no-cpu-fit` turns it off. Unlike bytes, cycles have no bucket: there is no
double buffer to decode ahead into, so it is a hard per-frame ceiling.
**One profile, `scsi`, at 280 KB/s.** The 110 KB/s `sasi` profile was dropped on
capacity rather than bandwidth, since a SASI volume is limited to 40 MB and the
game's 22.8 minutes is 146 MiB even at that rate (FINDINGS 32). The rate point
may return under another name once the delivery medium is settled, because a 1x
CD-ROM sustains ~150 KB/s and CD-ROM is the only period medium with the
capacity.
The profile bitrate is a **ceiling**: `lam` is bisected per frame under a leaky
bucket, so the profile's `lam` is a quality floor rather than a setting
(`--fixed-lam` opts out). At `--spans all` none of that binds, though. A
32-frame bucket emits the same container byte for byte as an 8-frame one and
`lam` never leaves its floor on any frame of the reference window, because the
rate is set by the span pass and by `mu` (FINDINGS 44.3). Two known unit
inconsistencies on that side are implemented and default off because they
measure as a wash: `--joint-decide` prices a byte at `lam + mu*c` rather than
`lam`, and `--joint-bucket` stops the bucket lending clocks it cannot repay.
An encode is ~95% k-means. A 120-frame window is ~29 s, of which ~22 s is
training the two codebooks.
Profiles are derived from a bandwidth figure rather than chosen by eye:
**The parked codec's encoder is kept and still runs:**
```sh
python3 tools/encoder/encode.py tmp/fr_singe out.dlx --profile scsi --preview p.png
```
python3 tools/encoder/profile_gen.py --bw-mbps 4 --name scsi
```
`--kbps` is the quality rate point and `--span-kbps` the ceiling the span pass may
draw on; `mu` is bisected per frame against the 68000's own decode budget so a
frame also *decodes* in time. One profile, `scsi`; the 110 KB/s `sasi` profile was
dropped on capacity rather than bandwidth — a SASI volume is 40 MB and the
game is 146 MiB even at that rate. The RAW escape means `lam=0` is
pixel-exact against the palettised frame. See FINDINGS 31/41/44 before changing
any of it.
## Documentation
- **`docs/STATUS.md`** is the current state, working setup, blockers and next
steps. **Start here.** It also lists what has been explicitly abandoned, so
old ideas do not get re-proposed.
- **`docs/ROADMAP.md`** is the remaining work to a completion target, and which
milestone that target is. Read it with STATUS rather than instead of it:
STATUS holds the measurements, ROADMAP holds the shape and goes stale first.
- **`docs/FINDINGS.md`** is measured hardware facts, content statistics, the
codec decision, and a section on measurement traps that produced three
separate false results. Read §4 before trusting any pipeline number. It is
append-only and later sections overturn earlier ones; superseded sections
carry a blockquote pointing at the correction.
steps, newest session first. **Start here.** It also lists what has been
explicitly abandoned, so old ideas do not get re-proposed.
- **`docs/ROADMAP.md`** is the remaining work to a completion target. Read it with
STATUS rather than instead of it: STATUS holds the measurements, ROADMAP holds
the shape and goes stale first.
- **`docs/FINDINGS.md`** is measured hardware facts, content statistics, the codec
decision, and a section on measurement traps that produced three separate false
results. Read §4 before trusting any pipeline number. It is append-only and
later sections overturn earlier ones; superseded sections carry a blockquote
pointing at the correction.
- **`docs/BENCHMARK.md`** is how to measure the storage subsystem, and why a
bandwidth figure out of MAME would be meaningless.
- **`docs/HARDWARE.md`** is the X68000 GVRAM/CRTC reference.
@@ -232,78 +401,76 @@ python3 tools/encoder/profile_gen.py --bw-mbps 4 --name scsi
## Layout
```
docs/ findings, status, roadmap, hardware reference
docs/img/ the stills and clips above, built from a real emulated run
src/player/ packed.s THE LIVE PLAYER: display bring-up, the 193-entry DMA
chain, its own V-DISP clock, READ(10) off a real
volume, and the MSM6258 on channel 3 -- a lump ring,
the remainder accumulator that stops the padding
becoming drift, a service routine that runs from
INSIDE dma.i's transfer wait (0.51 ms of seam
against 236), and pg_aseek.
dma.i the HD63450 channel: array chaining, held and
stealing, and DM_HOOK.
scsi.i the MB89352: selection, READ(10), the data phase.
adpcm.i the audio transport, every register byte of it
decoded out of the IPL ROM rather than recalled.
clock.i the frame clock, off the CRTC's V-DISP.
geom.i the constants, in one place.
decode.s the parked codec's 68000 decoder (1,296 bytes,
stream.s asserted), the same decoder behind a bounded ring,
ring.i the ring producer, the transport under it, and the
xfer.i load-time codebook/palette transforms. OUT OF THE
load.i VIDEO PATH: a DMAC-direct player has no ring.
frame.i the block loop and span chain, included by both, so
there is exactly ONE copy of the bytes every cycle
constant is fitted to. The span pass is blit.s v7
verbatim -- do not tidy it.
tools/encoder/ dlxp.py THE LIVE CONTAINER (DLXP2) and pack.py writes it.
pack.py Nothing is shared with the codec's writer on purpose.
adpcm.py the MSM6258 codec, carrying ffmpeg's decoder and the
chip's measured one side by side.
dlx.py the codec's container and its REFERENCE DECODER,
encode.py ground truth for the 68000; the encoder, rate
spans.py control, and the v7 span geometry. Parked, kept.
extract.py frames and audio off the disc.
tools/bench/ check.sh the green light.
packed_run.sh the packed player: gate, rate, audio, held, seek.
verify_packed.py / verify_packed_audio.py
the two gates that matter. The audio one reads the
SPEAKER, one delivered byte at a time, because every
counter in the player stayed right through a bug
that overwrote the buffer the channel was reading.
crtc_mode.lua the SINGLE SOURCE OF TRUTH for CRTC R00-R08 and
R20. Do not write CRTC values anywhere else.
probe_bit11_blank.lua B2, ready to run on a real board.
c68k/ a SECOND emulator for every 68000 cycle figure:
px68k's CPU core, no SDL, no ROMs. It also counts
BUS cycles, which MAME cannot report.
gvpack/ the same argument for the DISPLAY: px68k's own
x68k/gvram.c, so the address decode, the R20 bit-11
write path and the scroll wrap are its code and not
a model of it.
blit.s the full-frame GVRAM blit, timed on the 68000
itself. V8 is the packed one: 227,553 clocks.
tools/analysis/ measurement scripts, numbered in the order they were written.
Run from the repo root; they import from tools/encoder/.
01 and 02 are marked BROKEN deliberately and kept as
regression references.
10 is a COUNTEREXAMPLE and exits non-zero by design: it
demonstrates that the two-display-path plan corrupts 70 of 120
frames, which is why decode.s has one display path.
15 measures how much of the 68000's local bus the decoder
occupies and exits non-zero if its derived model stops
matching the harness's measurement.
16 is the DLX3 span container round-trip gate: it encodes,
writes the container, reads it back with the reference decoder
and fails if a pixel differs, or if it emitted too few spans to
have tested anything.
19 models the ring's ADDRESSES rather than its occupancy,
because each record must be contiguous and not merely resident,
and reports the zero-prefill pipe.
20 is an independent Python re-derivation of the seek-slack
model, sharing no code with the Lua producer it checks.
21 decodes the IPL ROM's HD63450 configuration and gates on the
bytes being where it says they are.
buscost.py is the shared bus-cycle table. The per-block
constants live in tools/encoder/vq_hybrid.py and are imported,
never copied.
tools/bench/ MAME Lua injection harness and 68000 benchmark sources.
check.sh is the green light.
blit.s/blit.lua time the full-frame GVRAM blit on the 68000
itself. Not part of check.sh, because wall timings would make
the green light host-sensitive.
span.sh measures the literal-span mode the same way and
asserts that every one of its 36 timing configs drew a
pixel-exact frame, the count taken from generated metadata so
a new config cannot weaken the gate.
crtc_mode.lua is the single source of truth for CRTC R00-R08
and R20. Do not write CRTC values anywhere else.
prep_dlx.py/decode.lua/verify_decode.py load, time and verify
decode.s. prep_stream.py/stream.lua do the same for stream.s,
but lay the container out as a DISK in a host file and feed it
through a bounded ring at a modelled pipe rate, so the rig is
not bounded by the emulated machine's RAM and a stock 2 MB
machine runs the whole window. dlxload.py holds the
codebook/palette load-time maths both preps share.
tools/bench/c68k/ headless px68k C68K harness, a SECOND emulator for every
68000 cycle figure. Links only px68k's CPU core: no SDL, no
ROMs, no emulated machine. `make PX68K=~/src/px68k` then
run.sh; verify_c68k.py checks the decode is pixel-exact, which
is what licenses the cycle numbers. It also counts BUS cycles,
which MAME cannot report. The Makefile's -no-pie and the
harness's MAP_32BIT arena are load-bearing: C68K truncates
host pointers to 32 bits.
tools/media/ builds docs/img/ from a paced recording run
tools/vasm/ vasm m68k assembler, binary plus source tarball
tools/encoder/ hybrid VQ encoder and DLX3 container writer.
spans.py is the v7 span geometry, selection and serialiser,
and the single place the chain layout is stated on the encoder
side. It must match blit.s and decode.s: 11 coarse units of
24 px, 11 fine of 2.
DLX2 4-byte-aligns every frame record, because an odd move.l
is an ADDRESS ERROR on a 68000, not a slow read.
dlx.py is the reference DECODER, ground truth for the 68000.
src/player/ decode.s is the 68000 DLX3 decoder with a preloaded-stream
front-end. stream.s is the same decoder behind a bounded ring.
Both include frame.i (the block loop and span chain) and
geom.i (the constants), so there is exactly ONE copy of the
bytes every cycle constant is fitted to. The span pass is
blit.s v7 verbatim, the same instruction sequence the
66.0/9.143/9.978 clock fit was measured on, so do not tidy it.
check.sh asserts decode.s still assembles to the same 1,296
bytes.
assets/ extracted frames and audio (gitignored)
Run from the repo root. 01 and 02 are marked BROKEN
deliberately and kept as regression references; 10 is a
COUNTEREXAMPLE and exits non-zero by design, demonstrating that
the two-display-path plan corrupts 70 of 120 frames. 15 is the
bus occupancy model, 19 the ring's ADDRESSES (not its
occupancy), 21 the IPL ROM's DMAC configuration, 25 the scene
graph, 30/34 the packed containers, 31 the display duty, 35 the
audio level off the whole disc, 36 the climb against real
branch points, 37 what a branch costs the chip. buscost.py is
the shared bus-cycle table; the per-block constants live in
vq_hybrid.py and are imported, never copied.
tools/import/ scenegraph.py -- the ONLY code in this tree coupled to somebody
else's source. Output is gitignored derived data.
tools/media/ make_packed_media.py builds the live player's still and clip;
make_readme_media.py builds the parked codec's. Both gate
before they write.
tools/vasm/ vasm m68k assembler, binary plus source tarball.
docs/img/ the stills and clips above, built from real emulated runs.
assets/, tmp/ extracted frames, audio and run output (gitignored).
```
Source media (`DRAGONS_LAIR.iso`) and ROMs are gitignored. Supply your own.
docs/FINDINGS.md
+3435
View File
File diff suppressed because it is too large Load Diff
docs/ROADMAP.md
+845 -60
View File
@@ -1,6 +1,217 @@
# Roadmap — remaining work to a completion target
Written end of session 19 (2026-08-24), against a tree that is ALL GREEN.
Amended end of session 21: P1 done, P2 half done (FINDINGS 53).
Amended end of session 22: P3 done (FINDINGS 54).
Amended end of session 23: P5 done (FINDINGS 55).
Amended end of session 24: G1 done (FINDINGS 56).
Amended end of session 25: P4 HALF done (FINDINGS 57).
Amended end of session 26: P4b done, P4a is the last open item before M2
(FINDINGS 58).
Amended end of session 27: P4a done at the transport level; THE RE-ENCODE
BUNDLE under P2 is now the only thing between this tree and M2, because 59.4
made sector-aligned records a precondition the transport enforces rather than a
preference (FINDINGS 59). **And 59.7 re-ranks what is left: the frame affords
6.74 clocks a byte, a dual-address byte costs 9, so B3 stopped being a constant
to look up and became the question of whether the design fits at all.**
Amended end of session 28: **THE RE-ENCODE BUNDLE IS DONE — all four items, one
re-measurement (FINDINGS 60).** The container is DLX5, sector-aligned, and the
disc now moves exactly the records. Two of the four closed as NEGATIVES: E2's
`--spans all` default is refused on measurement, and E3's joint span/lam
selection emits byte-identical containers because `lam` never leaves its floor.
**What is left of M2 is P4a's wiring** — the DMA channel behind `ring.i`'s
mailbox — and the budget did not move: headroom 6.74 -> 6.69 clk/B, so every
conclusion in 59.7 stands.
Amended end of session 29: **THE DECODER-FREE PACKED PLAYER IS BACK, MEASURED,
AND IT FITS THE CLOCK BUDGET THE CODEC MISSES (FINDINGS 61).** The packed
full-frame blit is **27.3%** of a 12 fps frame — measured, not assumed — a
channel fills GVRAM in buffer mode off the disc with the CPU halted, and it
walks the 1,024 B line stride itself through array chaining. At the 9 clk/B
floor the codec is 110.4% and a decoder-free packed player is **55.2%**. It asks
**576 KB/s, fixed**, against E7's 327 KB/s target. **So B1 stopped setting how
much headroom the player has and started deciding WHICH PLAYER EXISTS**, and B2
stopped being a nice-to-have. The codec cannot be packed: 47.6.4 is closed and
the answer is no, both ways (61.3).
Amended end of session 30: **K1 IS DONE AND THE ANSWER IS THE GOOD ONE
(FINDINGS 62).** A channel writes the palette registers at `$E82000` byte-exact,
and ONE array-chained start crosses from device registers into GVRAM — so a
frame is a palette entry and 192 row entries, started once, with the CPU halted
throughout, and the array is scene-constant. **It opened B4**: MAME models the
palette as plain `COMBINE_DATA` storage with no handler to be wrong about, so
the run bounds the model and not the board, and what a real palette register
does with a byte write is UNMEASURED. B4 is the cheapest hardware item in the
project and a negative costs 0.28% of a frame. **K2, the packed container, is
next.**
Amended end of session 31: **K2 IS DONE, AND IT COST THE BRANCH SOMETHING
(FINDINGS 63).** DLXP1 is a 49,664 B record = 97 sectors exactly, no index, no
decoder, 582.0 KB/s — exactly 61.9's prediction — and px68k's own `gvram.c`
renders the container's bytes index-exact with the harness computing no
interleave. 61.9's picture claim survives the real builder: **34.05 dB against
its 34.08**, and the hardware GRB555 word is charged on top for the first time
in this project (0.53 dB, on every row, so it moves nothing). **But the control
row landed exactly on the codec's ceiling**, so the whole +2.31 dB is the
PER-FRAME PALETTE and nothing else — and 90% of that palette changes every
frame, which makes a mismatched paint **12.8 dB worse** than the correct
pairing. **B2 now decides which packed CONTAINER exists, not only which player**
(63.4). The fallback is a flag: `--scene-palette --no-palette` is 30.79 dB, zero
churn, 576.0 KB/s and still +2.07 dB on the shipping codec. **K3 is next.**
Amended end of session 32: **K3 IS DONE, AND IT FOUND THE THING THAT DECIDES
THE BRANCH (FINDINGS 64).** `src/player/packed.s` brings up its own display,
builds its own 193-entry chain, keeps its own clock off V-DISP and fetches every
record itself off a CZ-6BS1: **120 of 120 frames pixel-exact, every one
compared, in both palette orders.** Two things came with it. **(1) The write
window is the frame.** A packed write needs R20 bit 11, buffer mode blanks the
layer, and a DMAC-direct player holds the window open for the whole data phase
— so the dark fraction of a slot is `record / (DATA-PHASE rate x slot)` and a
medium that exactly meets the 582.0 KB/s sustained requirement **displays none
of the frames it delivers on time**. The rate that matters here is the BURST
rate, which is a third hardware number **B1 has no test for**. It also
**reverses 61.5's ranking**: a packed player that DMAs to RAM and paints with
the CPU opens the window only for the measured 27.3% blit, so it is on screen
72.7% of every slot at any rate, and the two are equal only at **2,131 KB/s =
3.7x the wire**. **(2) A held channel costs the frame clock half its ticks and
the clock cannot tell** — 46.9% of V-DISP edges lost, zero late frames reported,
the player believing 12 fps while the screen ran at 6.37. **The open item is now
K4.**
Amended end of session 33: **P6 HAS AN ENCODER AND A PRICE (FINDINGS 65).**
`tools/encoder/adpcm.py` is an MSM6258 codec whose in-loop decoder is gated
SAMPLE-EXACT against ffmpeg's `adpcm_ima_oki` — there is no reference encoder
for this format, so that is the only check available and it is the one that
catches an encoder agreeing with its own wrong decoder. The Singe window encodes
to **78,125 B at 21.97 dB**, which is 7,812.5 B/s to the byte and 52's figure
arriving from the other direction, and **normalising the disc's −13.4 dBFS level
buys 0.00 dB**, so the level is not a lever. Two things came with it. **(1) The
two published delta formulas disagree by at most 3 in 12-bit units and that is
worth 25 dB** — encode with one and decode with the other and the SNR goes from
21.97 dB to **−2.88 dB**, because ADPCM is recursive the way the codec is
temporally recursive (64.1). Which one the chip runs is now **P6a** and it is a
precondition on shipping any audio. **(2) The packed container's own best
property is what makes audio cost.** A record has no index BY DESIGN, so audio
cannot be per-record without making records variable; it rides a fixed cadence
`(F, A)`, the obvious cadence F=1 wastes **57.3%** of every audio sector, and the
pick is **F=11, A=14 — 0.09% padding, 14,336 B held, wire 582.0 → 589.6 KB/s**.
The codec container, which already has an index and variable records, pays
**zero** padding. That is the first cost anyone has found for the packed
branch's simplification, and 64's risk list predicted there would be one without
knowing what.
Amended end of session 34: **P6a IS DONE, AND THE ENCODER WAS WRONG ON FOUR AXES
RATHER THAN ONE (FINDINGS 66).** 68000 code programs HD63450 channel 3 with the
IPL ROM's own ADPCM bytes — dual address, 8-bit port, cycle steal, EXTERNAL
request — feeds the MSM6258 at the chip's own pace (**7,811.4 B/s against
7,812.5**), and one of **sixteen** candidate decoder models reproduces MAME's
capture **sample-exact over 1,678 consecutive samples**, with a negative control
on every axis. The chip runs **`terms`, LOW nibble first, a 10-bit accumulator,
starting at −2**, and `tools/encoder/adpcm.py` defaulted to the opposite of all
four. **65.2 named the wrong axis as the risk**: the delta formula is worth
−2.88 dB and the NIBBLE ORDER is worth **−25.74 dB**, and 65.1's "high first,
measured" was a measurement of ffmpeg's VOX file convention rather than of a
chip's data register. Two things came with it. **(1) The 10-bit clamp is free on
the Singe window and only because that window peaks at 435 of 511** — 1.4 dB of
headroom on a −13.4 dBFS passage, where the encoder had been clamping 12.1 dB
higher — so **the audio LEVEL is an open choice again**, downward, and the
loudest passage on the disc is unmeasured. **(2) The transport is P6b's, not
scaffolding**, and it worked first time. **P6b is next and its bytes are
decided: DLXP2 encodes with `adpcm.CHIP`.**
Amended end of session 35: **P6b IS DONE AND ITS PADDING WAS DRIFT (FINDINGS
67).** DLXP2 exists: a 64-byte header, groups of one `A`-sector audio lump then
`F` records, `record i = off_frm + i*rec + (i//F)*A*512` — **still no index and
still none needed**, which is the packed branch's whole claim surviving the one
change that could have ended it. The player carries it in **six instructions
once a frame**, and 120 of 120 records are still pixel-exact off a real volume
with the interleave in, against a **silent control** that says no picture byte
moved. **65.3's 0.09% was not waste, it was a rate error**: a lump is 7,168 B of
space and eleven frames of audio is 7,161.4583… B, so a player that fed the chip
the whole lump would run 0.09% fast — **1.25 s of lip-sync over the game's 22.8
minutes**. The payload is a remainder, `acc += 11*15625; n = acc//24; acc %= 24`,
which is `clock.i`'s shape for `clock.i`'s reason and the **third** time this
tree has met the pattern. The four ADPCM axes ride in the header and the gate
flips each one to prove they earn it (**order −31.99 dB, formula −24.86**). The
wire is **589.6 KB/s**, 65.3's figure to the tenth. ~~**What is left of P6 is the
last quarter: no audio has been played out of this container on any layer, and
two DMA channels have never run at once.**~~
Amended end of session 36: **P6c IS DONE, AND HOLDING THE BUS COSTS THE AUDIO
463x THE SEAM (FINDINGS 68).** All 78,125 B of the container's audio reached the
chip, in order, sample-exact against the four axes in its own header, out of a
player that was fetching records off the same disc at the same time. **Two DMA
channels have now run at once and the interaction is not in the bytes, it is in
the timing**: the MSM6258 has no starvation state, so the interval between a
channel counting out and the CPU arming the next lump is a held nibble pair
driving the predictor. Stealing, the seam is **0.51 ms over ten seconds**;
**held, it is 236 ms — 2.31% of the audio, every one of the ten lump boundaries,
worst 72.8 ms**, because a burst channel halts the 68000 and the only place a
68000 driving this video path has time is inside the transfer wait (250,000 of
250,240 service calls). That is FINDINGS 64.3 reaching the audio: **the held
configuration cannot feed a second consumer**, and B1's answer now decides a
sound as well as a picture. 67.2's drift is measured rather than derived —
predicted 1.25 s, played **1.26 s**. **What is left of P6 is the level (66.3)
and the refill climb with a second consumer through a real branch point.**
Amended end of session 37: **THE AUDIO LEVEL IS MEASURED, AND THE ANSWER IS
THAT IT DOES NOT CHANGE (FINDINGS 69).** Every stream of the game's own footage
(`00000`-`00201`) through `extract_audio.py`'s chain says the disc peaks at
**946 of 2048 = −6.71 dBFS**, which is **5.35 dB over** the chip's 10-bit clamp
— and the census behind that peak is **687 samples of 20.2 million, 402 events,
44.0 ms in 21.5 minutes**. Forty windows drawn over the game and encoded at six
gains then price the choice: the disc's own level has the best mean SNR
(**22.03 dB**) and the gain that guarantees zero clamping costs **0.85 dB across
the whole game** to buy back **1.90 dB on the 2.11 s that clamp**, because the
OKI step floor is a constant 16 and does not scale with the signal. **And
66.3's mechanism does not survive a control**: error after a clamp run is
elevated ~5x, and so is the same window at a gain that never clamps, read at the
same indices — worst ratio **1.28** — because `adpcm.encode` runs the chip's
clamp inside its own search and therefore never loses the chip's state. The
worry was right about the mechanism and aimed one layer too late; session 34
had already closed it. `--audio-gain` exists so the level is a parameter with a
measurement behind it, its default is 1.0, and the gate container is
byte-identical. **What is left of P6 is the refill climb with a second consumer
through a real branch point** — and 69.4 adds two small unbuilt cases, a scene
with no audio track (`00176`) and a scene whose audio is shorter than its video
(`00199`).
Amended end of session 38: **P6 IS CLOSED, AND THE LAST ITEM IN IT REOPENED THE
CADENCE (FINDINGS 70).** `tools/analysis/36_branch_audio.py` runs 51.3's climb
against 56.3's 612 real branch points with the second consumer on the wire. On
the codec branch **audio is 1.7% of the wire and up to 3.30x of the climb** —
6.70 s → 22.11 s at 451.4 KB/s, and 79% → 99% of the game's branch points
arriving under it — because slack accrues out of `pipe − wire` and that is a
small difference of two large numbers. On the packed branch **there is no climb
at all**: a record goes straight into GVRAM, video lookahead is zero records,
and the only consumer with any lookahead is the audio one (1.833 s of sound
against 0.000 s of picture). **At 589.6 KB/s — the acceptance figure — the
sounded packed container's surplus is exactly zero**, which is the sharpest
statement yet of why B1 must name a burst rate and not only a sustained one.
And a cost nobody had counted: a DLXP2 group is `lump k, then F records`, so a
seek into a group finds its audio BEHIND it — **mean 416.5 ms of silence at the
409 within-container branch points, worst 833.3 ms, 36 of 409 free**, while all
203 scene changes are free by construction. **The fix is one extra read of
7,168 B (11.7 ms against 416) and no player has an audio seek path**; the
alternative is F=1, which gives back 12,288 B of RAM for +4.36 KB/s of wire.
**The cadence pick is therefore reopened and the deciding number is B1's**: what
one extra SCSI command costs. **What is left of P6 is nothing** — 69.4's two
unbuilt cases and 70.3's seek path are player work, not open questions.
Amended end of session 39: **P6d IS DONE — THE PLAYER SEEKS WITH SOUND ON IT —
AND WHAT IT UNCOVERED IS AN ENCODER ITEM (FINDINGS 71).** `pg_aseek` rebuilds
the lump index, the stream position, the remainder accumulator and the byte
offset into the group, issues the second READ(10) 70.3 asked for, and re-arms
channel 3 part way into the buffer: **132,162 B of spliced stream accounted for
byte by byte in MAME's own capture**, across a branch at frame 37 that is
deliberately *not* on a group boundary, in both chip configurations. 70.3's
416.5 ms of silence is gone. **What replaces it is smaller and is not a player
problem: the MSM6258's accumulator is an integrator with no leak, so a branch is
a DC offset that does not decay** — −355 of 511 playing through (still −108 four
seconds later, AC 0.00), an exact permanent −65 if the chip is STOPped and
re-PLAYed. **A re-PLAY is 5.5x better and neither is zero.** The only fix that
reaches zero is the ENCODER's: reset the predictor where a branch can land, and
**resetting every frame makes all 119 of the container's branch points exact for
0.33 dB**. That is a DLXP3 and it is now the cheapest open item in the tree. The
seek also found a race that had passed this gate three times — `pg_ainit` waited
on a *read-back* MTC and could start the scene one byte in, silently — which is
71.2 and is fixed.
**THE COMPLETION TARGET IS M3, THE VERTICAL SLICE** (USER DECISION): one scene
tree — a decision point, two outcomes, a death clip — with audio, streaming from
@@ -27,7 +238,8 @@ these units:
| **68000 clocks** | measured, and the rate controller binds on them. |
| **Delivery rate** | **no working figure, deliberately** (FINDINGS 50, USER DECISION). Every tool REQUIRES an explicit rate. |
| **Seek time** | **no figure at all, and never had one.** 51.3/51.4 made it matter. |
| **W, clocks stolen per delivered byte** | 5 single-address held, 9 dual held, 12 single arbitrated; the IPL ROM's own disk channel is **16..19** (52.5). **The largest open number in the project.** |
| **W, clocks stolen per delivered byte** | 5 single-address held, 9 dual held, 12 single arbitrated; the IPL ROM's own disk channel is **16..19** (52.5). **The largest open number in the project.** Session 27 added the row underneath it: with **no external request line** on the card (59.2) the channel is auto-requested and is charged **by time rather than by byte**, so at 460 KB/s a 50% bus share costs **10.61 clk/B** and a smaller share cannot carry the rate at all (59.3). |
| **The frame's headroom for a transport** | **6.69 clk/B** — 30.2% of a 12 fps frame, after the MEASURED decode (68.6%) and best-case audio (1.25%), at the DLX5 gate container's **37,585 B delivered record** (session 28: the sector pad is delivered, so `15_bus_occupancy.py` charges it). **It is the number every row above is read against**, and a dual-address byte's floor is 9. It was 6.74 against the DLX4 container; the bundle moved it by 0.05 and moved no conclusion. |
---
@@ -53,10 +265,27 @@ None of these block M2 or M3 software work, because session 18 forced every rate
to be an explicit argument. They set constants, and two of them decide how much
headroom the finished player has.
**B1. Measure the BlueSCSI — throughput AND seek time.**
**B1. Measure the BlueSCSI — throughput, seek time AND the DATA-PHASE BURST
RATE.** The third one is session 32's (FINDINGS 64.2) and it is not a refinement
of the first: sustained throughput decides whether record *i* arrives before
slot *i*, and the **burst rate during the data phase** decides how much of the
slot the picture is on screen, because a DMAC-direct packed player holds the
GVRAM write window open for exactly as long as the transfer takes and buffer
mode blanks the layer. A drive with a read-ahead cache can pass the first and
fail the second. The acceptance test is **`record / (burst x slot)` = the dark
fraction**; at the container's own 582.0 KB/s it is 1.0, and the picture is never
displayed. **Session 29 gave this
a second acceptance test that is not a codec figure at all: 576 KB/s SUSTAINED,
which is what a decoder-free packed literal frame costs and cannot be talked down
from (FINDINGS 61.5).** A codec's bitrate is a lever; a literal frame's is
geometry. So the measurement now has three thresholds to be read against —
453.6 KB/s (the gate container needs no prefill), 327 KB/s (E7's target at the
dual-address floor) and 576 KB/s (no decoder at all) — and which of them the
medium clears decides which player gets built.
Throughput has an acceptance test already derived from real record sizes:
**513.2 KB/s** for the session-14 candidate, **451.4 KB/s** for the gate
container (`19_ring_stream.py`, FINDINGS 49.5). Seek time has nothing.
**513.2 KB/s** for the session-14 candidate, **453.6 KB/s** for the DLX5 gate
container (`19_ring_stream.py`, FINDINGS 49.5; the figure was 451.4 before
session 28's re-encode and the sector pad raised it). Seek time has nothing.
51.3/51.4 is why the second half matters: slack is *accumulated* out of
`pipe - wire`, so what a branch point costs is set by the rate and the time since
the last branch, not by the ring size. At 460 KB/s every ring from 192 KB to
@@ -69,12 +298,101 @@ and settles it in minutes on a real board. FINDINGS 48 shifted the prior toward
MAME and toward "unusable" — **do not pre-build on 1.0 B/pixel**. Same sitting:
the priority register `0xE82500` at `0x0000` (47.3).
**B3. Single-address vs dual-address DMA.** 242 KB/s and 0.69 dB. Needs
`scsiexrom.bin` (8 KB, CRC `7be488de`) sourced, then its DMAC init disassembled
for DCR's DTYP: `10`/`11` = single (5.0 clk/B), `00`/`01` = dual (9.0).
> **Session 29 raised what this is worth, and gave it a number to be worth
> (FINDINGS 61).** It used to gate a derived halving. It now gates a player that
> has been measured to fit a budget the shipping design misses — 55.2% of a frame
> against 110.4% at the 9 clk/B floor. And 61.6 found an asymmetry worth carrying
> to the board: the black interval is the PAINT, not the frame, so a CPU-painted
> packed player is dark for **27.3%** of a frame while the cheaper DMAC-direct one
> is dark for **30..113%**. Under MAME's reading the cheap architecture is the
> dark one. **B2 and B1 are now the same decision from two sides, and B2 is the
> five-minute half.**
>
> **Session 31 raised it again, from the other direction (FINDINGS 63.4).** If
> buffer mode does NOT blank, the packed player's per-frame palette is not
> merely visible during the paint — **231 of 256 entries change every frame**,
> and a picture under the neighbouring frame's palette is **12.8 dB worse** than
> the correct pairing, on screen for roughly half of every frame slot at 12 fps.
> So a non-blanking board does not just cost the packed player a blank interval,
> it may cost it the per-frame palette — which is **the whole +2.31 dB** the
> branch has over the codec's ceiling. The fallback is already an encoder flag
> (`pack.py --scene-palette --no-palette`: 30.79 dB, no churn, 576.0 KB/s), so
> what B2 decides is now **which packed container ships**, not whether one can.
**B4. Does a real palette register take a BYTE write?** (62.4, new session 30.)
`$E82000` is 256 16-bit registers. The decoder-free packed player's per-frame
palette rides the frame's array chain as a 193rd entry, which means a
dual-address channel with an 8-bit device port writes it **one byte at a time**,
even bytes into the high half of a register and odd into the low. **MAME cannot
be asked**: it maps the palette to `palette_device::read16/write16`, which is
`memory_array`'s `COMBINE_DATA` over plain storage — RAM that honours
`mem_mask`, with no handler that could refuse. So the run in 62 is a statement
about the model and not about the board.
**It is the cheapest item on this list, cheaper than B2**: write `$A5` to
`$E82000` and `$5A` to `$E82001` from the CPU and read the word back. If it
comes back `$A55A` the palette rides the chain. **The blast radius of a negative
is small and known** — the palette leaves the chain and the CPU writes 256 words
a frame at 61.9's derived 0.28% of a frame — which is why this is B4 and not
above B2.
**B3. Single-address vs dual-address DMA — and now, first, DOES THE CARD DRIVE
`#EXREQ`?** 242 KB/s and 0.69 dB was the old framing. **Session 27 promoted this
to the item that decides whether the design fits at all (59.7).** The frame
affords **6.74 clocks a byte**; a dual-address byte is **9** — a 4-clock read of
the device plus a 5-clock write to memory — so **no dual-address configuration
fits this container at 12 fps, at any delivery rate and any GCR share.** Single
address is 5 clk/B and fits at 92.2% of the frame with 7.8% to spare, and it
needs the device to ACK the DMAC directly, which needs the request line.
**MAME cannot answer it**: `x68k_scsiext.cpp`'s `drq_w` only stores a flag and
the expansion slot has no request path to the HD63450 at all (59.2). The slot
PINOUT has `#EXREQ` at B36 and `#EXACK` at B37, so the provision exists on the
real board. **What is wanted from hardware is therefore narrower and sharper
than "disassemble the ROM": does a CZ-6BS1 assert `#EXREQ` during a data phase,
and will the HD63450 run a single-address channel against it?** A scope or a
logic analyser answers the first; the second is a program the player can run.
Sourcing `scsiexrom.bin` (8 KB, CRC `7be488de`) and disassembling its DMAC init
is still the cheapest paper route to the same answer, because whatever Sharp's
own driver programs into DCR's DTYP is a statement about what the card supports.
FINDINGS 48.4. Not on this machine (checked, session 18).
**This is also P4's input** — the handshake the player drives is the same
question from the software side.
**Ranking, amended session 29.** It was: B1 sets how much headroom the player
has, B3 decides whether there is any. FINDINGS 61 adds a third reading — **B1
and B2 together decide which player exists.** If the medium clears 576 KB/s
sustained and buffer mode does not blank, the decoder-free packed literal fits at
the dual-address floor B3 cannot get under, and B3 stops mattering for video at
all. If it does not, B3 is still the question. The three hardware facts are no longer
independent, and B2 is by far the cheapest of them.
> **ENCODER WORK IS PARKED — USER DECISION, session 29.** The first draft of this
> amendment said "nothing here is a reason to stop work on the codec". That does
> not survive its own arithmetic. It rested on comparing the packed player's
> 576 KB/s against **E7's 327 KB/s target, which does not exist**: the codec that
> exists is 440 KB/s and 110.4% of a frame, so the real gap is 1.31x, not 1.76x.
> And the branches are not symmetric. **Packed needs two facts** — buffer mode
> does not blank, medium clears 576 KB/s. **The codec needs E7 to succeed** —
> unproven, and 60.4/60.5 measured two of its three levers inert — **AND** the
> medium to clear 327, **AND** it ships at ~100% of the frame with no margin,
> which is where 55.2% is now.
>
> **E7 and E4 are both parked**, E4 included: `H.build`'s k-means builds VQ
> codebooks and a literal player has no VQ. C1 is gated by E4 and follows.
>
> **The codec is KEPT AND NOT BUILT ON.** That is inventory, not work. B2 is
> unanswered and 48.1's prior leans against packing — an assertion against a
> silence — and if buffer mode blanks there is no version of the packed player
> that is merely expensive (48.3), at which point the codec is the only path
> left. Keeping a working decoder on disk costs nothing; building on it costs
> sessions.
**The older ranking, which still holds inside the codec branch:** B1 (throughput
and seek) sets how much headroom the finished player has. **B3 decides whether
there is any.** If the card drives `#EXREQ`, the ladder applies and the design fits with
room. If it does not, the fallback is limited-rate auto-request at a share the
player picks (P4c), and the container has to come down to **328 KB/s of payload**
to fit at the 9 clk/B floor — 34% below where the gate container sits (59.7).
> **Session 20 moved the prior hard, and it moved the wrong way (FINDINGS 52.5).**
> The IPL ROM *is* on this machine, and `tools/analysis/21_iplrom_dmac.py` reads
@@ -88,6 +406,97 @@ question from the software side.
---
## The packed branch — what building it means (session 29, USER DECISION)
**This is where the work goes now.** FINDINGS 61: a decoder-free packed literal
player is **55.2% of a frame at the 9 clk/B dual-address floor** against the
codec's 110.4%, and **+4.89 dB** on the shipping container because a literal
frame is not tied to a scene palette the codec's codewords index into. It costs
**582 KB/s, fixed, with no lever.** Three items, in order.
~~**K1. Can a DMA channel write the palette registers at `$E82000`?**~~
**DONE, session 30 — FINDINGS 62. YES, in this model.** `dmagate.s` runs 7–9:
512 B off the disc into the whole graphic palette, **byte-exact in 256
register words** read back out of `$E82000` by the 68000; the same transfer
aimed at RAM leaving the palette as the CPU poisoned it, which is what
attributes the first run to the channel's `MAR`; and **ONE array-chained start
crossing from device registers into GVRAM**, which is the shape of a whole
frame. The destination is POISONED first (62.1) because "it matches" was a
weak claim against a record that is mostly pad, and the host counts the
poison's discriminating power rather than assuming it: 511 of 512.
**And the array is SCENE-constant** (62.3) — the packed layout spends both
256-colour pages, so there is no page to flip and the 193 destinations never
change; the 1,158 B array is built once at scene setup. What is left on the
CPU per frame in the video path is the channel start and the READ(10), and
neither is priced — say "no per-frame PAINT work", not "no per-frame CPU work".
**It opened B4** (62.4): MAME models the palette as a generic `palette_device`
over `memory_array`, whose `write16` is a plain `COMBINE_DATA`, so it has no
handler to be wrong about and cannot discriminate. And it filed one open
design choice, 62.5: palette FIRST or 193rd is visible on screen and is not
decided. **Session 31 PRICED it and it is a wash — 20.32 dB against 20.33
(63.4) — so it is a container flag (`--palette-last`) and K3 runs both.**
~~**K2. A packed container.**~~
**DONE, session 31 — FINDINGS 63.** `tools/encoder/dlxp.py` is DLXP1 and
`pack.py` writes it: 254 colours with index 0 held free and black at 255,
column *i* interleaved with *i+128*, a per-frame palette, and a **49,664 B
record that is 97 sectors EXACTLY** — the alignment that cost session 28 a
re-encode is free here because a packed record's length is geometry. **No
index and no length word**, for the same reason: nothing has to be walked.
**582.0 KB/s, which is 61.9's prediction to the tenth.** Encodes in 3.3 s
because there is no k-means in it.
**The re-derivation is done and 61.9 survives it: 34.05 dB against 34.08**,
and the GRB555 word is charged for the first time (63.3) — 0.53 dB, on every
row of the table, so it moves no comparison. **The two reserved entries cost
0.0003 dB.**
**What it also found** (63.4): the SCENE-palette control lands exactly on the
codec's ceiling, so the whole +2.31 dB is the per-frame palette; 90% of that
palette changes every frame; and a mismatched paint is 12.8 dB worse, for
roughly half of every frame slot, if buffer mode does not blank.
- ~~**K3. End to end, off the disc.**~~ **DONE, session 32 — FINDINGS 64.**
`src/player/packed.s`, `tools/bench/packed.lua`, `tools/bench/packed_run.sh`,
`tools/bench/verify_packed.py`, `tools/analysis/31_display_duty.py`. Palette,
page-1 X-scroll 384, priority `vc1 = 0x0002`, R20 bit 11, one chained DMA a
frame, **120 of 120 pixel-exact in both palette orders** — and the gate checks
every frame rather than the last, because a packed frame is a literal and the
codec's recursion was what made one comparison audit 120.
**What it found is K4's whole content**, and it is in the amendment above: the
window is the frame, the burst rate decides visibility, and a held channel
eats the clock.
- **K4. THE PACKED PLAYER THAT IS ON SCREEN.** 64.2's option B: DMA the record
into one of two RAM buffers with the window SHUT, then paint it with the
packed `movem` blit (`blit.s` V8, **measured** at 227,553 clocks = 27.3% of a
slot). **82.2% of a frame at the 9 clk/B dual-address floor** against A's
54.9%, **99,328 B of RAM**, and a picture on screen **72.7% of every slot at
any delivery rate** instead of 0% at the container's own wire.
It is not a rewrite of K3: `packed.s` keeps its display bring-up, its clock,
its transport and its record arithmetic, and what changes is the chain's
destination and the addition of a paint. **The one thing in it that has never
been run is the overlap** — a channel filling buffer *i+1* while the CPU
paints buffer *i*, which is the first time in this project that the DMAC and
the 68000 have had to want the bus at the same time for a whole scene.
**K4 is conditional on B2 the same way K3's ranking was**, and the condition
now cuts the other way: if buffer mode does NOT blank, A is on screen the whole
slot and K4 is 27.3% of a frame spent on nothing. **Do not build K4 before B2
is answered** — that is the same rule 61.7 wrote for the codec, applied to the
branch that replaced it.
**What K3 deletes, and why that is a risk and not a win to be banked:** a
DMAC-direct packed player has **no ring** — `ring.i`, `xfer.i` and most of
`stream.s` leave the video path, and **P4a's wiring is parked with them.** A
simplification that large usually hides something, and 61.7.2 names the specific
untested thing: a chained transfer has never run back to back at 12 fps.
**K1 and K2 survive a bad answer to B2. K3 does not.** ~~Do K1 first.~~ ~~K1 is
done (session 30, FINDINGS 62); K2 is next.~~ **Both are done. K3 is next — and
63.4 added a second thing for it to run: BOTH palette orders, which is a flag
(`--palette-last`) and not a re-encode.**
---
## M2 — a player, as opposed to a decoder
`decode.s` draws pixel-exact frames from RAM Lua pre-loaded; `stream.s` decodes
@@ -96,24 +505,217 @@ out of a bounded ring fed by a host file on a paced clock. Neither is a player.
**Exit criterion: boots from a real SCSI volume on a stock 2 MB X68000, plays
one scene at 12 fps from disc, no host-file pipe, no Lua in the loop. Silent.**
**P1. Codebook expansion on the 68000.** `dlxload.py:19` expands CB1 to 32 B per
entry and CB4 to 8 B, host-side, because at the time it was a load-time cost that
would have flattered or damned the inner loop. The player must do it: **8 KB +
2 KB per scene**. Note where that lands — *at a scene change, when the ring is
empty because of the seek*. It compounds with 51.3 and should be priced against
the refill climb, not treated as free setup.
~~**P1. Codebook expansion on the 68000.**~~ **DONE, session 21 — FINDINGS 53.**
`src/player/load.i` expands both codebooks out of the raw container header,
byte-exact against `dlxload.py` on both CPU cores. **9.26 ms**, and it was
priced where it lands rather than treated as free setup: the scene header is
**5,920 B that no rate table in this tree counted**, and in the currency of
51.3 — accumulated slack — those bytes lengthen the refill climb by 138 ms at
488 KB/s and by **1.099 s at 451.4 KB/s**, because the surplus they are divided
by goes to zero. The whole fixed cost of a scene change is about a third of one
frame slot; what makes a branch point expensive is still the seek and the climb.
Shipping the codebooks pre-expanded was considered and refused: it trades
9.26 ms of CPU for 5,120 more header bytes, which is a wash in milliseconds and
not a wash in kind (53.6).
**P2. Palette packing on the 68000.** The encoder still emits RGB888; the X68000
word packing is Lua-side. Whatever writes real palette words must pick `I` per
entry by minimum squared error (**1.96 dB**, FINDINGS 23.3) and reserve index 0
as black with `I = 0` (23.4).
~~**P2. Palette packing on the 68000.**~~ **DONE, session 28 — FINDINGS 60.**
The half that was open was the encoder's, and it closed with the whole bundle
below. Session 21 — FINDINGS 53 — did the packing itself:
~~The encoder still emits RGB888; the X68000 word packing is Lua-side.~~ The
packing is on the 68000: `pal_pack` writes 256 words straight into `$E82000`
with `I` chosen per entry by minimum squared error (**1.96 dB**, 23.3), gated on
the words read back **out of the palette registers**. 9.70 ms per scene, plus
5.29 ms of scene-independent table build hoisted to boot (53.3).
**P3. A real frame clock.** `stream.s` has `PACE`/`PACEON` (`$18034`/`$18038`)
but the 12 fps tick comes from the Lua producer. Needs MFP timer or VBL. Keep
`PACEON=0` free-run working — the wrap gate uses it and every FINDINGS 49 figure
depends on it.
~~**What is left is the other half of the sentence: reserve index 0 as black with
`I = 0` (23.4).**~~ **DONE, session 28 — 60.3.** `VQ.scene_palette` quantises
the picture into 255 entries and holds index 0 at (0,0,0); `pack_palette` gives
it `I = 0` by its own minimum-squared-error rule, so 23.3's "the bars sit at
RGB (4,4,4)" needed no special case. **0.04 dB** on the Singe window, palette
ceiling unmoved. Black is reserved, not withheld — the mapper may still spend
index 0 on genuinely black pixels; what it buys is that index 0 is black
whatever the scene contains.
**P4. Real transport.** Drive the MB89352 instead of a host file. The `W`
~~**THE RE-ENCODE BUNDLE, collected here because it is now four items and they
share one re-measurement.**~~ **DONE, session 28 — FINDINGS 60. Two of the four
closed as NEGATIVES, which is the more useful half.** The four were:
1. ~~reserve palette index 0 as black, `I = 0`~~ **DONE** (23.4, 60.3);
2. ~~`--spans all` as the default~~ **MEASURED AND REFUSED** (E2, 60.4).
267.9 KB/s / 28.88 dB / 2 frames late at `need`, against 448.2 KB/s /
29.07 dB / 1 late at `all`: **+67% of the wire for +0.19 dB and one frame
of 120**, against a container the budget already says is 35% too big.
`need` stays the default. **It was called "the loaded lever on the byte
side" and it is — it is just loaded the wrong way**, and E7 is why. The
GATE container keeps `all`: it is a fixture, not a recipe;
3. ~~re-derive span selection jointly with `lam`~~ **IMPLEMENTED, MEASURED,
NO-OP** (E3, 60.5). `--joint-spans` hands the span pass's freed bytes back
to the lam search and re-spans; all four cells of `{need,all} x
{greedy,joint}` are BYTE-IDENTICAL, and so is `--rc-floor open`. **`lam`
never leaves its floor on any of 120 frames at either floor the encoder
offers** (44.3), so there is nothing to spend the freed bytes on. The code
stays, defaulted off, because a container that moved `lam` off its floor
would make the question live again;
4. ~~**sector-align every record**~~ **DONE — the container is DLX5** (58.3
option C, promoted to a precondition by 59.4; 60.1). Realised cost
**+0.48%** on the wire against the +0.43% predicted, zero clocks, and
**120/120 records start on a sector boundary** where 3/120 did. The disc
and the ring now move the SAME 4,510,208 B and check.sh gates on that
identity, both figures read out of the container instead of written into
the script — the old literals went red on the re-encode, correctly.
**The consumer had to be told too** (60.2): `stream.s` released the ring
to the last byte it READ, which strands up to 511 B of pad a record, and
the ring's own audit caught it on frame 0 while every frame still decoded
pixel-exact. The release rounds to `RECALN` now.
The letterbox no longer gets the palette's closest thing to black — item 1 put
true black at index 0, and `load.i` needed no change, as it said it would not.
~~**P3. A real frame clock.**~~ **DONE, session 22 — FINDINGS 54.**
`src/player/clock.i` derives the tick from the CRTC's own V-DISP through the
MFP, with a remainder-keeping divider whose two constants are read out of the
CRTC at init. **Exactly 12.000000 fps, by construction** — measured at 649 ticks
over 3,000 refreshes where 649.1429 were due, so the remainder still held and
nothing accumulated. **181.35 clocks per V-DISP, 838 per frame, 0.1006% of the
budget**, timed by the 68000 itself because the host's 17.64 ms granularity
cannot see it. `PACEON=0` free-run is untouched and so is the wait loop; the
free-running path executes none of the new code.
The item said "MFP timer or VBL" and **neither can do it alone**: 4e6/12 is not
an integer and no prescale/data pair reaches 12 Hz, while the slowest MFP tick
of any kind is 78.125 Hz; and the raster's 55.4577 Hz has no whole divide near
12 either (4 gives 13.86, 5 gives 11.09). `tools/analysis/23_frame_clock.py`
walks the whole space rather than asserting it.
**What it exposed is bigger than the item.** 12 fps on a 55.4577 Hz raster is
4.6215 refreshes, so a frame gets **4 refreshes (72.13 ms) or 5 (90.16 ms)** and
**there is no 83.33 ms frame** — that figure is the mean slot, and 37.9% of slots
are 13.4% under it. The cadence was ALREADY in every host-paced result in
FINDINGS 49/51, because `stream.lua`'s tick is sampled at frame boundaries and
its gaps were always 4 or 5; nothing had named it. On the gate container it
costs 4 frames of 120 their idle against 1 for the nominal model. **It is not a
dropped frame** — the pace gate lets an overrun eat the next frame's idle and
the clock recovers — but it means every budget in this project is priced against
a slot 37.9% of frames do not get. 54.4.
**Also struck: MAME's raster runs 2.22% fast** (`refresh_mode()` builds the frame
period from `htotal - 8`), so the tree's "1/55.46 s granularity" was 1/56.69 s
throughout. No 68000 cycle figure moves — the CPU clock is unrelated to the
screen — but anything paced by the raster does. 54.5.
**P4. Real transport. P4b DONE, session 26 — FINDINGS 58. P4a DONE at the
transport level, session 27 — FINDINGS 59. What is now between this tree and M2
is THE RE-ENCODE BUNDLE under P2, because the channel refuses a windowed read
(59.4) and 117 of 120 records need one.**
~~Drive the MB89352 instead of a host file.~~ `src/player/scsi.i` selects a SCSI
target and issues READ(10) on the 68000, with no IOCS and no host in the
transfer path: **4,096 B from LBA 0 and 2,048 B from LBA 1000, both byte-exact**
against the host's copy of the same volume.
**This item was listed as blocked and was not.** Session 21 recorded "MAME's
`x68000` has no MB89352 path"; `-exp1 cz6bs1` instantiates one, and FINDINGS
32.4 had read that card's DMA glue back in session 9. The real gap was the 8 KB
`scsiexrom.bin` MAME needs to instantiate the card and **the player never
executes**; a blank placeholder on a separate rompath settles it. **B3 still
wants the real ROM's bytes** and is untouched by this.
**What is left is the half that decides the project**, and it is now two pieces:
~~**P4a. A DMAC configuration that HOLDS THE BUS.**~~ **DONE at the transport
level, session 27 — FINDINGS 59.** `src/player/dma.i` programs HD63450 channel 1
and takes the DATA IN phase: **the same 2,048 B off the disc three ways — PIO,
held, stealing — all three byte-exact.** 57.3's warning was met rather than
worked around: the evidence never reads `$EA0015`. **MTC is sampled by the
instruction after the one that starts the channel, and held it reads zero of
2,048** — the whole transfer happened between two instructions, because the
68000 did not execute in between — against the full count and 426 CPU loop trips
for the stealing configuration. Put the stealing registers in the held slot and
every byte still arrives and the gate goes **red**, which is what says the
counter can come out different (58.3's vacuous-counter trap, avoided
deliberately).
**Three bounds on the apparatus, read out of MAME's source and not inferred**
(59.2): the card has **no request line to the DMAC** (its flow control is
DTACK), so external request — the mode the `W`=5 and `W`=12 rows assume —
cannot be run; **single address** cannot be run either (only channel 0 has
device callbacks); and **only burst is modelled as held**. Of the four rows of
the ladder exactly one, dual address held, has a code path here, and it is the
one demonstrated. The slot pinout has `#EXREQ` at B36, so a real card plausibly
drives it — **that is now B3's sharpest form**.
~~**What is left of P4a is downstream of the container, not of the DMAC**
(59.4)~~ — **and that block is GONE as of session 28.** `sc_in_data` refused a
windowed read because a channel cannot drop the 300 B in front of a record; the
container is DLX5 now and no record asks for a window. `xfer.i`'s sector
arithmetic already degenerates correctly — `SC_WSKIP` is 0 and `SC_WKEEP` is the
whole record on every one of the 120 — so **what is left of P4a is the wiring:
which loop moves the bytes.** `dma_run.sh`'s windowed-read refusal stays as a
negative control rather than as a description of the container. **This is the
next item, and it is the last one before M2** (60.9).
**P4c (new, and it is a DESIGN CHOICE the tree had not named).** Auto-request is
charged **by time, not by byte** — the channel spends its share of the bus
whether or not a byte is there, so halving the delivery rate DOUBLES the CPU
cost of the same record. The MC68450's GCR sets that share: `BT`/`BR`, four
values, 50/25/12.5/6.25%. `tools/analysis/28_autorequest_cost.py` prices it
against an explicit rate; at 460 KB/s **only the 50% share carries this
container**, at 10.61 clk/B and 47.6% of a frame per record, against 40.4% for
the `W`=9 row and 391.8% measured for PIO. **If B3 comes back saying the real
card drives `#EXREQ`, the ladder applies and this is the fallback; if it does
not, this IS the cost model** and the GCR pair is a number the player has to
choose.
~~**P4b. `scsi.i` behind `ring.i`'s `XF_*` mailbox.**~~ **DONE, session 26 —
FINDINGS 58.** `src/player/xfer.i` answers the mailbox with a real READ(10) per
record: **120 records, 4,488,588 B, pixel-exact, out of the same 256 KB ring,
with a real mid-stream seek in a second pass**. The tiling is the SAME 18 wraps
and 14.7 KB mean hole that 49.4's host producer and 55.4's modelled transport
produced — a third transport, same placement, which is the assertion that
`ring.i` could not tell which side of the seam answered it. The change above the
seam is two `bsr`s, and the one in `ring_seek`'s quiet-wait is not optional:
with the transport inside the machine, that loop is the only thing that can
retire an outstanding request.
**What it cost is the finding, and it re-prices P4a.** `tools/bench/
xfer_cost.sh` subtracts the same 120 frames run twice and gets **87.28 clocks
per delivered byte** — against the 68000's own cycle table for the loop, which
says **87.15**. **0.2% apart**, so it is the instruction stream and not MAME's
device model, and it is therefore the first number this rig has produced that
survives leaving the emulator. At this container's 37,405 B mean record that is
**391.8% of a 12 fps frame**, and the machine's own V-DISP clock agrees from the
other end: **2.57 fps**.
W = 5 single address, bus HELD ............................ 22.4%
W = 9 dual address, held .................................. 40.4%
W = 12 single address, arbitrated .......................... 53.9%
W = 19 dual address, arbitrated -- the IPL ROM's own (52.5) . 85.3%
PIO 87 MEASURED, session 26 ................................ 391.8%
**So P4a is worth 4.6x the worst DMA configuration in this tree and 17.5x the
best**, where before this session it was worth 9 against 19. `W` itself **did
not move by one clock** and is still the largest open number — but what depends
on it just got much larger.
**One more thing P4a inherits (58.3).** A record is not a sector: 117 of 120
start part way into one. PIO absorbs that for free because the CPU is already
touching every byte and simply does not store the ones outside the window — a
property that **disappears the moment the DMAC takes over**, because a channel
writes a contiguous run and cannot drop bytes. The three ways out price as
+1.34% wire and no DMA (windowed PIO), +1.34% wire and **+5 clk/B of copy**
(bounce buffer, which is exactly the cost `aligned` was chosen over `split` to
avoid), or **+0.43% wire and zero clocks** (sector-aligned records in the
container). The last one wins on both axes and is a **re-encode**; see the
bundle under P2. **P4a should be attempted against a sector-aligned container,
not against this one.** *(Session 27: it was, in the only sense that mattered —
the transport now REFUSES the windowed case rather than being trusted not to
reach it, so the bundle is a precondition rather than a plan. 59.4. Session 28:
the container IS one — the realised wire cost is +0.48% against the +0.43%
predicted here, and 120/120 records start on a sector boundary. 60.1.)*
*(original item, still the standing description of the `W` question:)*
Drive the MB89352 instead of a host file. **Session 23
added a second axis to it:** `W` is the clocks stolen per delivered byte, and
55.3 measured that the player's own request loop gives away 3-7% of the pipe
before `W` is even asked about. A transport design has to answer both. The `W`
handshake — clocks stolen per delivered byte, bracketed 5..12 by MC68450 Fig
4-25 — is listed in "Decisions locked" as UNDECIDED and as the thing that
decides the project: `W<=6` fits 0/120 frames, `W=8` misses 47/120. It is a
@@ -134,11 +736,36 @@ first job rather than its last.
clocks per WORD and FINDINGS 43 voided it; 52.5 cited it in byte units when
first written and strikes it.
**P5. Seek and branch.** Per-record index (the `aligned` producer needs one
anyway, 49.3), prefill policy, and the accumulated-slack rule from 51.3 made
explicit in the player rather than implied by the rig.
~~**P5. Seek and branch.**~~ **DONE, session 23 — FINDINGS 55.**
`src/player/ring.i` fills the ring on the 68000: `aligned` placement, the
descriptor ring, a prefill policy, 51.2's slack rule as arithmetic the player
can run (`ring_may_seek`), and a seek that quiets the channel and re-addresses
the stream out of the index. It reproduces the host producer's tiling exactly —
18 wraps, 14.7 KB mean hole, pixel-exact — and the host now AUDITS every
placement instead of making it.
The index is a **container change**: DLX4 carries `nframes` u16 record lengths
in the scene header, because `aligned` needs a record's length before it fetches
it and walking the stream is precisely what a player cannot do. Frame payloads
are byte-identical to the DLX3 encode; the scene header goes 5,920 to 6,164 B.
**What it exposed is bigger than the item.** A channel only moves bytes while it
has a request and only the CPU can issue one, so the disc **stands still between
records** by an amount set by the player's loop rather than by the medium — and
no host-filled run could see it. At 488 KB/s in a 256 KB ring, a one-deep
request queue gives away **6.8% of the pipe and underruns 59 of 120 frames**; a
two-deep one gives away 3.4% and underruns none. The container's whole surplus
over the wire at that rate is 8.7%, so the player's own loop was spending most
of the slack 51.3 accumulates. **Prefill is the weaker lever** — six records of
it still leaves 24 underruns at depth 1 — and the fix costs no clocks and no
bytes. 55.3, 55.4.
**P5a (open, and it belongs with P4).** The two-deep queue is modelled as two
mailbox slots. On the machine it is two DMAC channels or one channel with a
chained descriptor array, and which of those is affordable is a `W` question.
**P7. Boot.** The player as an executable loading from the SCSI volume.
Buildable, and empty until P4: there is nothing to boot from yet.
---
@@ -147,7 +774,15 @@ explicit in the player rather than implied by the rig.
**Exit criterion: one decision point, two outcomes, a death clip, with audio,
playing from disc on stock hardware.**
**P6. Audio — and it is the largest unpriced risk left in the project.**
**P6. Audio. P6a DONE, session 34 — FINDINGS 66.** ~~and it is the largest
unpriced risk left in the project.~~ Three quarters of P6 closed in session 33
and the fourth precondition — which decoder the chip runs — closed in 34, on the
machine, through the real DMA channel. ~~**What is left of P6 is P6b (DLXP2, a
container with sound in it) and the refill climb with a second consumer.**~~
**P6 IS CLOSED end of session 38** — P6b in 35, P6c in 36, the level in 37 and
the refill climb with a second consumer in 38 (FINDINGS 70).
*(original framing, kept because every figure below is still the live one:)*
MSM6258 ADPCM, 15.6 kHz mono, **7.8 KB/s**. That figure is in `ratectl.py`'s
budget and nowhere else: not extracted, not encoded, not interleaved into the
container, and **never priced on the bus**. Two reasons to treat it as a risk
@@ -176,26 +811,129 @@ charging audio the disk's 5. Both worries above resolve:
7.8 was decimal kB being multiplied by 1024; 2.4% high, now derived from the
sample rate in `buscost.ADPCM_BYTES_PER_S`.
**What is still open in P6 is everything except the bus:** extraction, encode,
container interleave, and what a second stream does to `wire` — and therefore to
`pipe - wire`, and therefore to 51.3's refill climb. That last one is the
interaction to price next, and it is E2's question with a second consumer in it.
~~**What is still open in P6 is everything except the bus:** extraction, encode,
container interleave, and what a second stream does to `wire`.~~ **THREE OF THE
FOUR ARE DONE, session 33 — FINDINGS 65.** `tools/encoder/extract_audio.py`
takes the same seconds of the same stream the frames come from;
`tools/encoder/adpcm.py` encodes them (**78,125 B, 21.97 dB**, gated against
ffmpeg's decoder sample-exact by `tools/bench/verify_adpcm.py`);
`tools/analysis/32_audio_wire.py` is the interleave and the wire. **The packed
container's cadence is `F=11, A=14`** — 0.09% padding, 14,336 B held, 582.0 →
**589.6 KB/s** — and the naive one-lump-per-record cadence would have wasted
57.3% of every audio sector. **The codec container pays zero padding**, because
it already has the index the packed one deleted (65.4).
~~**What is left in P6 is the fourth: the refill climb with a second consumer
through a real branch point** (51.3, 55.4). The slack table is in
`32_audio_wire.py` — at 582.0 KB/s exactly, silent breaks even and sounded
starves — but a branch point has not been run with audio on the wire.~~
**DONE, session 38 — FINDINGS 70**, and it produced three things the slack table
could not: the climb multiplier (up to **3.30x** for **1.7%** of the wire), the
fact that the packed branch has **no accumulator to climb**, and the group-entry
silence (**mean 416.5 ms** at 409 of the game's 612 branch points) that reopens
the cadence pick on a third column. `tools/analysis/36_branch_audio.py`.
**P6a. WHICH DELTA FORMULA DOES THE MSM6258 RUN? (new, session 33, FINDINGS
65.2 — and it is a precondition, not a refinement.)** ffmpeg computes
`((2*(n&7)+1) * step) >> 3`; the OKI datasheet's form truncates per term. They
differ by **at most 3 in 12-bit units per sample** and, because ADPCM is
recursive, **encoding for one and decoding on the other costs 25 dB — the noise
comes out louder than the signal.** No encoder output can be committed to a
container before this is answered.
**It does not need a board.** MAME's x68000 HAS the chip: `:okim6258`, with the
68000 reaching it at **`$E92001` and `$E92003`** — both read out of the machine's
own program map by `tools/bench/probe_adpcm.lua`, not from folklore. What did
NOT work is feeding it from Lua: `probe_adpcm2.lua` / `probe_adpcm3.lua` swept
control 0..3 against PPI port C 0..15 and `-wavwrite` recorded silence
throughout (65.5). The gap is the register semantics, and the way to close it is
**from 68000 code with the IPL ROM's own channel-3 DMAC configuration**, which
`21_iplrom_dmac.py` already reads out of the ROM — the one ADPCM path in this
machine that is known-correct because Sharp wrote it. That is also the real
design, so it is not scaffolding.
**P6b. A CONTAINER WITH SOUND IN IT. DONE, session 35 — FINDINGS 67.**
~~DLXP1 has no audio section. 65.3 is the arithmetic a DLXP2 is built from and no
byte of one is written.~~ DLXP2 is written, gated and carried by the player:
`tools/encoder/dlxp.py`, `pack.py --audio`, `tools/analysis/34_packed_audio.py`,
the third LBA term in `src/player/packed.s`. The cadence arithmetic came through
unchanged (F=11, A=14, 589.6 KB/s) and the **payload** did not: a lump's audio is
7,161 or 7,162 B of a 7,168 B sector run, and feeding the chip the whole lump
drifts 1.25 s over the game.
**P6c. AUDIO OUT OF THE CONTAINER, ON THE MACHINE. DONE, session 36 — FINDINGS
68.** The lump buffer is `PG_ABUF` (three slots, prefill two, and the minimum
depth is still unmeasured), the remainder accumulator is `pg_apay`, and the
service runs from inside `dma.i`'s transfer wait through `DM_HOOK` because once
a frame is a 90 ms seam by construction. **Two channels have run at once**: the
bytes are identical in both DMAC configurations and the sound is not — held
costs 236 ms of replayed byte against stealing's 0.51 ms. ~~Every
piece exists and none is joined up. The container carries the bytes (67); the
transport is `src/player/adpcm.i`'s channel-3 configuration, which is the IPL
ROM's own and worked first time (66.1). What is missing is **the lump buffer**
— 14,336 B, double-buffered, allocated by nobody — and **the remainder
accumulator**, which is three instructions and is not optional.~~
**And the interaction neither half has met: TWO CHANNELS AT ONCE.** The video
channel holds the bus and halts the 68000, which already costs the frame clock
half its ticks without the clock being able to tell (64.3). An audio channel
that must be serviced *during* that hold has never been run. 52's 1.25%..1.48%
of a frame is a figure measured in isolation, and `32_audio_wire.py` names what
it turns into here: audio does not merely cost clocks, it costs **darkness**.
**E6. Container v2** — audio interleave, per-record index, scene table. Depends
on P6's answer and on P5's index.
**G1. Import the scene graph — early, because it is a measurement input.**
SNES project `data/events/` (MIT, cleared) diffed against DirkSimple (zlib),
which transcribed the same data independently, to catch transcription errors
before anything reaches 68000 tables. **Neither is on this box** — both need
fetching.
**E7. A BYTE TARGET, AND IT COMES OUT OF THE BUS BUDGET RATHER THAN OUT OF
TASTE (new, session 27, FINDINGS 59.7; re-measured session 28, 60.7).** The
frame affords **6.69 clocks a byte** after the measured decode and the audio,
and a dual-address byte costs **9**. So *if* B3 comes back saying the card
cannot drive `#EXREQ`, the container has to reach **27,924 B a frame — 327 KB/s
of payload** to fit at 12 fps, where the DLX5 gate container delivers 37,585 B
and 440 KB/s: **35% too big.** (It was 6.74 / 328 / 34% against the DLX4
container. **The bundle moved the target by one KB/s and moved no conclusion**,
which is what a precondition is supposed to do.)
The reason to pull this ahead of the game logic that consumes it: 51.3 says
4.83 s of play to refill a 256 KB ring at 488 KB/s, and Dragon's Lair's decision
points are seconds apart. **Nothing in this tree can currently say what the worst
gap between consecutive decision points is** — only the scene table knows, and
until it is imported, whether this design survives a back-to-back branch is an
open question nobody is able to ask.
Three things make this less alarming than the number looks, and one makes it
worse:
- The gate container is **deliberately the heaviest thing the encoder emits**
(span-heavy, the 488 recipe, every block mode exercised). It is a test
fixture, not a shipping target.
- A lighter container **also decodes cheaper**, so the 68.5% decode term falls
with the byte term. 328 KB/s is the pessimistic reading of the lever.
- `rc_fr_singe_scsi_cpufit.dlx` already exists — the encoder has had a
CPU-fitting mode since session 11.
- **Worse:** `15_bus_occupancy.py` REFUSES to price the cpufit container,
correctly, because the C68K measurement it cross-checks against belongs to
the gate container. **So E7 starts with a harness re-run**
(`tools/bench/c68k/run.sh`) against whichever container is to be the target,
and until that is done "34% too big" is a statement about the fixture rather
than about the project.
~~**G1. Import the scene graph — early, because it is a measurement input.**~~
**DONE, session 24 — FINDINGS 56.** It was pulled ahead for exactly the reason
given, and it paid: **the worst gap between two consecutive decision points is
zero**, and 5.4% of the game's 612 branch transitions are. Two seeks can fall
back to back with no play between them, so 51.2's slack rule can be answered NO
by the content rather than by the buffer.
It does not break the design — a branch on an empty ring costs the prefill
(149.7 ms, 1.80 frame slots at 488 KB/s), not the climb — but it removes the
margin: at 488 KB/s in a 256 KB ring, **76% of this game's branch points arrive
before the ring has refilled**, and a 512 KB ring makes that 90%. **The ring is
not the lever; the surplus is.**
Two constraints on the input layer came with it: the arcade needs **eight
directions**, and the shortest input window is **98 ms** against a 72.13/90.16 ms
frame slot, so input cannot be polled on the frame tick (56.7).
**The cross-check plan was wrong and is struck.** The SNES chapters are
*derived* from DirkSimple, by their own README, so there is one transcription and
not two; the diff catches conversion errors only (56.2). **Nothing is vendored:**
`tools/import/scenegraph.py` is the one file coupled to those projects and it
writes this project's own `DLXSCENE1` schema into gitignored `tmp/`
(USER DECISION, session 24).
---
@@ -207,15 +945,24 @@ Listed for completeness; past M3 these are scope, not risk.
not menu vs content: the two largest streams are bonus material and look like
content by size, duration and bitrate alike (25.1). Run
`07_motion_survey.py` per stream first for a hot-window shortlist.
**Gated by E4.**
- **E4. `H.build` k-means**, 51 s of a 55 s run, once per scene. The thing to
attack before C1, and not anything in the per-frame path (27.6).
- **E2. `--spans all` as default.** Still a recommendation, not a measurement
(43.6.1), and the only loaded lever on the encoder's byte side (44.3). **It
spends every profitable byte, which raises `wire`, which shrinks `pipe - wire`,
which lengthens the refill climb after every branch.** That interaction is not
priced, and M3 is where it becomes measurable.
- **E3. Re-derive span selection jointly with `lam`** (39.3).
**Gated by E4, and parked with it (session 29).**
- ~~**E4. `H.build` k-means**, 51 s of a 55 s run, once per scene.~~ **PARKED,
session 29 (USER DECISION).** It was the thing to attack before C1. It builds
**VQ codebooks**, and a decoder-free packed player has no VQ — so this is
encoder work on the branch that is no longer being built on. It comes back if
and only if B2 goes MAME's way. C1 is gated by it and is parked with it.
- ~~**E2. `--spans all` as default.**~~ **MEASURED AND REFUSED, session 28 —
60.4.** It was "a recommendation, not a measurement" since 43.6.1 and it is a
measurement now: **+67% of the wire for +0.19 dB and one frame of 120.** It
IS the loaded lever on the byte side (44.3) — it is loaded the wrong way, and
E7 is why. What the entry predicted is exactly what it does: it raises `wire`,
which shrinks `pipe - wire`, which lengthens the refill climb. `need` stays
the default; the GATE container keeps `all` because it is a fixture.
- ~~**E3. Re-derive span selection jointly with `lam`** (39.3).~~
**IMPLEMENTED, MEASURED, NO-OP, session 28 — 60.5.** `--joint-spans` emits
byte-identical containers in all four `{need,all} x {greedy,joint}` cells and
at both lam floors, because **`lam` never leaves its floor on any of 120
frames**. Kept and defaulted off.
- **C2. Framing** — crop vs squash vs wide (FINDINGS 12). Needs an eyeball
against arcade reference, not a measurement. Cheap; blocks only final encodes.
- **C3. Disk image packaging**, ~1.09 GiB at the candidate rate.
@@ -226,16 +973,54 @@ Listed for completeness; past M3 these are scope, not risk.
## Dependency summary
```
B1 seek+rate ─┐
B3 DTYP ──────┴─> P4 transport ─┐
├─> M2 ─> M3 (COMPLETION TARGET) ─> M4
P1 P2 P3 P5 P7 ─────────────────┘ ^
│
P6 (bus cost DONE, 52) ──────────────────┤
G1 scene graph (fetch, do early) ─────────┘
B2 blanking ─> (page 1; do not pre-build on it)
P4a DONE (59): the channel drives the data phase and
holds the bus -- 391.7% of a frame becomes 40..95%
│
B3 #EXREQ? ──┬─ YES ─> single address, 5 clk/B, 92.4% ── FITS ──┐
│ │
└─ NO ──> auto-request, 9 clk/B FLOOR, 110.4% ──> E7 byte target
(the frame affords 6.69; 59.7, 60.7) 327 KB/s
│
P2 re-encode bundle DONE (60): DLX5, records ARE sectors ─────────┤
E2 refused on measurement, E3 a no-op -- 60.4, 60.5 │
E7/E4/C1 PARKED session 29: encoder work waits on B2 ───────────┤
│
K1 palette-register DMA? ─> K2 packed container ─> K3 end to end ──┤
K1 DONE s30 (62), K2 DONE s31 (63), K3 DONE s32 (64): │
120/120 pixel-exact, both palette orders, off a real volume │
└─> and K3 found that the WRITE WINDOW IS THE FRAME, so: │
K4 (DMA to RAM + the measured 27.3% paint) is the player │
that is ON SCREEN below a 2,131 KB/s BURST rate -- which │
is 3.7x the wire, so below every rate anyone has proposed │
P4a WIRING (the channel behind ring.i's mailbox) <- THE LAST ITEM ─┤
P1 P2 P3 P4b P5 P7, P6 bus cost (52), G1 scene graph (56) ────────┼─> M2 ─>
B1 seek+rate (sets HEADROOM, not fit) ─────────────────────────────┘ M3 ─> M4
B2 blanking ─┬─ NOT blanked ─> K3's DMAC-DIRECT player is the one: 54.9% of a
│ frame at the 9 clk/B floor against the codec's
│ 110.4%, on screen the whole slot, and K4's paint
│ would be 27.3% spent on nothing (61.4, 61.5)
└─ blanked ──────> K3's player is on screen for
1 - record/(BURST x slot) of every slot, which is
ZERO at the container's own wire -- so K4 is the
player, at 82.2% of a frame and 99,328 B of RAM
(64.2). Neither answer kills the branch and each
picks a different player.
B1 BURST rate (NEW, 64.2) ──> which of the two K3/K4 wins, if B2 blanks
P6 audio: encoder gated (65), P6a the chip's own decoder measured (66),
P6b DLXP2 written and gated, 589.6 KB/s on the wire (67),
P6c PLAYED -- 78,125 B to the chip beside the video channel (68)
└─> and the second consumer is now ANOTHER INPUT TO B1/B2:
stealing -> 0.51 ms of seam over 10 s \ same bytes,
held -> 236 ms, every lump boundary / different sound
└─> what is LEFT of P6: the LEVEL (66.3, oldest open item) and the
refill climb with the second consumer through a branch point
```
**Read that top-left branch as the project's live question.** Everything else
on the diagram is work; `#EXREQ` is a fact about a board nobody here has, and it
decides which of the two lower paths the player is on.
## Standing rules that apply to all of it
- **Green light first and last.** `./tools/bench/check.sh`, ALL GREEN, before and
docs/STATUS.md
+2292
View File
File diff suppressed because it is too large Load Diff
docs/img/packed-player.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 200 KiB

docs/img/packed-player.webm
BIN
View File
Binary file not shown.
docs/img/palette-mismatch.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 224 KiB

src/player/adpcm.i
+131
View File
@@ -0,0 +1,131 @@
; ---------------------------------------------------------------------------
; adpcm.i -- the MSM6258V, driven the way the machine's own ROM drives it.
; ROADMAP P6a, and it is P6b's transport arriving early rather than scaffolding.
;
; NOTHING HERE IS INVENTED. Every register value below is one that
; tools/analysis/21_iplrom_dmac.py decodes OUT OF THE IPL ROM's own bytes, at
; the addresses it prints: channel 3's DCR/SCR/MFC/CPR/DFC/DAR at $FF0C2E and
; the per-transfer OCR = $32 plus command $02 at $FF9A82. That is the one
; ADPCM path on this board that is known-correct because Sharp wrote it.
;
; WHY THIS FILE EXISTS AT ALL. Session 33 fed the chip from Lua and got
; silence, swept control 0..3 against port C 0..15, and stopped rather than
; guess (65.5). Two of the reasons are visible from here and neither is a
; register semantic anybody had to guess:
;
; * THE PPI'S PORT C IS NOT AN OUTPUT UNTIL IT IS TOLD TO BE. The ADPCM pan
; and the sample-rate divider are port C bits, and an i8255 in its reset
; state has every port an INPUT -- so a write to $E9A005 changes a latch
; nobody is reading and the pan never leaves wherever it was. Control word
; $92 (mode 0, A and B input, both halves of C OUTPUT) is what makes the
; other write mean anything.
; * AND FEEDING IT SLOWLY IS NOT FEEDING IT. The chip has no FIFO and no
; starvation state: it consumes a nibble every sample period out of whatever
; its data register last held, forever. A byte per host frame is not a
; quiet chip, it is the same two nibbles 130 times, which saturates in six
; samples. The feed has to be paced by the chip, which is what channel 3
; and its request line are FOR.
;
; THE CLOCK IS TWO WRITES AND THEY ARE IN DIFFERENT DEVICES. 15,625 Hz is
; 8 MHz / 512: the 8 MHz comes from CT1 in the YM2151's port register $1B, and
; the /512 from port C bits 3,2 = 10. Neither is readable, so the rate is
; verified from the OTHER end -- the capture's own sample count.
AD_CTRLR = $E92001 ; W: command R: status (bit7 = NOT playing)
AD_DATAR = $E92003 ; W: the byte the chip takes two nibbles from
AD_PLAY = $02 ; COMMAND_PLAY -- session 33's probes wrote $01,
AD_STOP = $01 ; which is COMMAND_STOP
PPI_PC = $E9A005
PPI_CTL = $E9A007
PPI_COUT = $92 ; mode 0, A/B input, BOTH halves of C output
PPI_RATE = $08 ; pan 00 = both, rate 10 = /512 = 15,625 Hz
YM_ADDR = $E90001
YM_DATA = $E90003
YM_CT = $1B ; CT1 in bit 1: 0 = ADPCM master clock 8 MHz
AD_DMAC = $E840C0 ; HD63450 channel 3 -- the ADPCM channel, and
A3_CSR = AD_DMAC+$00 ; the one the ROM points at $E92003
A3_CER = AD_DMAC+$01
A3_DCR = AD_DMAC+$04
A3_OCR = AD_DMAC+$05
A3_SCR = AD_DMAC+$06
A3_CCR = AD_DMAC+$07
A3_MTC = AD_DMAC+$0A
A3_MAR = AD_DMAC+$0C
A3_DAR = AD_DMAC+$14
A3_MFC = AD_DMAC+$29
A3_CPR = AD_DMAC+$2D
A3_DFC = AD_DMAC+$31
A3_DCRV = $80 ; XRM 10 cycle steal w/o hold, DTYP 00 dual
; address, DPS 0 8-bit port (IPL $FF0C2E)
A3_OCRV = $32 ; DIR memory->device, SIZE 11 byte unpacked,
; CHAIN 00, REQG 10 EXTERNAL REQUEST (IPL
; $FF9A82). External request is what makes the
; chip the pacemaker: one byte per #DRQ3, and
; #DRQ3 ticks at half the sample rate.
A3_SCRV = $04 ; MAC 01 memory increment, DAC 00 -- the device
; address is a REGISTER and must not walk
A3_CCRST = $80
; --------------------------------------------------------------- ad_setup
; The clock and the pan. No arguments, no result; trashes d0.
ad_setup:
move.b #YM_CT,YM_ADDR
moveq #60,d0 ; the YM2151 wants settling between the
.ymw: subq.l #1,d0 ; address write and the data write
bne.s .ymw
move.b #$00,YM_DATA ; CT1 = 0 -> ADPCM master clock 8 MHz
move.b #PPI_COUT,PPI_CTL ; ...and NOW port C drives something
move.b #PPI_RATE,PPI_PC ; pan both, /512
rts
; ----------------------------------------------------------------- ad_arm
; Arm channel 3 to feed (a1) for d1 bytes and start it. Trashes d0.
; The channel is started BEFORE the chip is told to play (see ad_play), so that
; byte 0 is already in the data register when the accumulator is reset.
ad_arm:
move.b #$FF,A3_CSR ; CSR is write-one-to-clear: a stale COC
; would pass the wait loop instantly
move.b #A3_DCRV,A3_DCR
move.b #A3_SCRV,A3_SCR
move.b #$05,A3_MFC
move.b #$05,A3_DFC
move.b #$01,A3_CPR ; the ROM's own priority: ADPCM outranks
; the disk at the arbiter (52.5 item 5)
move.l #AD_DATAR,A3_DAR
move.b #A3_OCRV,A3_OCR
move.l a1,A3_MAR
move.w d1,A3_MTC
move.b #A3_CCRST,A3_CCR
rts
ad_play:
move.b #AD_PLAY,AD_CTRLR
rts
ad_halt:
move.b #AD_STOP,AD_CTRLR
rts
; --------------------------------------------------------------- ad_abort
; STOP channel 3 where it stands, and clear what stopping it posts.
;
; A SEEK IS THE ONLY THING IN THIS PLAYER THAT NEEDS THIS. Every other re-arm
; happens at COC, where the channel has already counted itself out and there is
; nothing to stop; a seek arrives MID-LUMP, because a branch is a frame index
; and a frame does not know about the cadence (FINDINGS 70.3).
;
; SAB is CCR bit 4. CSR is then written $FF -- write-one-to-clear -- to take
; down COC and ERR together, because the abort posts a channel error (CER $11)
; and pg_aserv's whole test is "did the channel count out": a stale COC would
; make the very next service call arm a lump that is already playing.
;
; NAME THE LAYER. MAME's hd63450 does not NEED the abort -- its
; dma_transfer_start reloads MAR/MTC and restarts the timer whatever the channel
; was doing. The MC68450 does: STR written to an active channel is an operation
; timing error, and the transfer that is running is not the one that was asked
; for. The abort is here for the silicon, and the run below cannot tell.
ad_abort:
move.b #$10,A3_CCR ; SAB -- software abort
move.b #$FF,A3_CSR ; ...and the COC/ERR it posts
rts
src/player/adpcmgate.s
+112
View File
@@ -0,0 +1,112 @@
; Front-end for the ADPCM transport (ROADMAP P6a), for the rig.
;
; It plays ONE buffer of nibbles the host pushed into RAM and reports what the
; channel did. What is being measured is not this code -- it is the CHIP: which
; delta formula, which nibble of a byte first, where the accumulator clamps, and
; what it starts at. tools/bench/verify_adpcm_chip.py reads all four out of
; MAME's own -wavwrite capture.
;
; THE ONE THING THIS FILE HAS TO GET RIGHT is the order of the two starts. The
; chip resets its accumulator, its step index AND its nibble select when it is
; told to PLAY, and it begins consuming immediately out of whatever its data
; register holds. So the channel goes first and the CPU waits for MTC to move
; -- proof that a byte has actually been taken -- before the PLAY. The stream
; still has a prologue of unknown length, because the gap between PLAY and the
; NEXT #DRQ3 is not ours to set; the prologue is 16 zero nibbles for exactly
; that reason and the verifier reads its length off the capture.
include "src/player/geom.i"
AD_FLAG = $18600 ; u32 0 idle, 1 armed, 2 playing, $FF done, $EE error
AD_BUF = $18604 ; u32 where the nibble bytes are
AD_LEN = $18608 ; u32 how many bytes
AD_MTC0 = $1860C ; u32 MTC at the instant PLAY was written
AD_CSRF = $18610 ; u32 CSR at completion
AD_CERF = $18614 ; u32 CER with it
AD_MTCF = $18618 ; u32 MTC with it
AD_MARF = $1861C ; u32 MAR with it -- where the channel stopped
AD_SPIN = $18620 ; u32 trips round the wait loop
AD_STAT = $18624 ; u32 the chip's own status byte while playing
AD_PATIENCE = 60000000
AD_SETTLE = 60000 ; ~100 ms at 10 MHz, 18 clocks a trip ; the wait is bounded like every other
org $10000
start:
bsr ad_setup
; ---- SETTLE, and it is not superstition. The 8 MHz ADPCM clock is
; CT1 in the YM2151's port register, and this machine delivers that
; write to the ADPCM chip on the SOUND system's own schedule rather
; than at the instant of the store -- so a transfer started in the same
; breath as ad_setup plays its first ~17 ms at the PREVIOUS clock. The
; symptom is exact: the capture's first 130-odd samples come out in
; identical PAIRS, because the chip is clocking half as fast as the
; capture, and every model then fails to fit a stream that changed rate
; part way through. ~100 ms of nothing costs the gate nothing and a
; player sets its clock once at boot.
move.l #AD_SETTLE,d0
.settle:subq.l #1,d0
bne.s .settle
movea.l AD_BUF.l,a1
move.l AD_LEN.l,d1
bsr ad_arm
move.l #1,AD_FLAG.l
; ---- wait for the channel to actually take byte 0. Not a delay loop:
; the condition is MTC having moved, which is the channel's own account.
move.l #AD_PATIENCE,d3
.first: move.w A3_MTC,d0
andi.l #$FFFF,d0
cmp.l AD_LEN.l,d0
bne.s .go
subq.l #1,d3
bne.s .first
bra bad
.go: move.l d0,AD_MTC0.l
bsr ad_play
move.l #2,AD_FLAG.l
moveq #0,d0
move.b AD_CTRLR,d0 ; bit 7 clear = the chip says it is playing
move.l d0,AD_STAT.l
clr.l AD_SPIN.l
move.l #AD_PATIENCE,d3
.wait: addq.l #1,AD_SPIN.l
move.b A3_CSR,d4
btst #4,d4 ; ERR -- CER says which
bne.s bad
btst #7,d4 ; COC
bne.s .fin
subq.l #1,d3
bne.s .wait
bra.s bad
.fin: bsr report
; The chip is left PLAYING deliberately: it goes on replaying the last
; byte it was given, which the verifier ignores. Stopping here would
; put a silence in the capture at a point the host would then have to
; find, and the capture already has a length it knows.
move.l #$FF,AD_FLAG.l
hold: bra.s hold
bad: bsr report
move.l #$EE,AD_FLAG.l
bra.s hold
report:
moveq #0,d0
move.b A3_CSR,d0
move.l d0,AD_CSRF.l
moveq #0,d0
move.b A3_CER,d0
move.l d0,AD_CERF.l
move.w A3_MTC,d0
andi.l #$FFFF,d0
move.l d0,AD_MTCF.l
move.l A3_MAR,d0
move.l d0,AD_MARF.l
rts
include "src/player/adpcm.i"
src/player/clock.i
+202
View File
@@ -0,0 +1,202 @@
; ---------------------------------------------------------------------------
; clock.i -- the FRAME CLOCK, on the 68000 itself. ROADMAP P3.
;
; src/player/stream.s has a pace gate: frame i may not START before tick i, and
; PACE is the tick counter. Until now PACE was written by tools/bench/
; stream.lua, i.e. by the host, off the host's idea of what 12 fps means. That
; was honest for what FINDINGS 49/51 were measuring -- arrival times against a
; deadline -- and it is not a player. A player has no host. These are the
; bytes that replace it.
;
; WHAT THE MACHINE ACTUALLY OFFERS, because "use the MFP timer or vblank" hides
; a real constraint. The MC68901's timer clock on this board is 16 MHz / 4 =
; 4 MHz (MAME 0.277 src/mame/sharp/x68k.cpp:1027-1028), its prescaler ladder is
; {4, 10, 16, 50, 64, 100, 200} (src/devices/machine/mc68901.cpp:173) and its
; data register is 8 bits. The SLOWEST tick a single MFP timer can produce is
; therefore 4e6 / (200*256) = 78.125 Hz, and 4e6/12 = 333,333.33 is not even an
; integer -- so no prescaler/data pair ticks at 12 Hz, and no timer at any
; setting ticks as slowly as a 12 fps frame. A frame clock needs a divider in
; software whichever source it is built on. tools/analysis/23_frame_clock.py
; enumerates the whole space rather than asserting this.
;
; So the source is the RASTER, and that is a better answer than a timer anyway.
; GPIP4 on the MFP is V-DISP (x68k.cpp:1139, `m_crtc->vdisp_cb().set(i4_w)`),
; high while the display is active; the same signal is the MFP's Timer A event
; input (mc68901.cpp:167, GPIO_TIMER = {GPIP_4, GPIP_3}). Its interrupt is
; channel 6, IR_GPIP_4 = $40 in IERB/IPRB/IMRB (mc68901.cpp:76). We take the
; FALLING edge (AER bit 4 = 0), which is the start of vertical blanking -- the
; instant a player would present a finished frame, so the clock and the flip
; are the same event rather than two events with a phase between them.
;
; THE DIVIDER IS EXACT, AND IT IS EXACT BY CONSTRUCTION. The raster is
;
; 31,500 lines/s / (R04 + 1) lines/frame
;
; and 31,500 / 568 = 55.4577 Hz is not a multiple of 12, so a whole-number
; divide cannot do it: 4 refreshes is 13.87 fps and 5 is 11.09 fps. Instead
; each V-DISP adds `fps * (R04+1)` to an accumulator and a frame tick is emitted
; whenever it reaches 31,500, keeping the remainder:
;
; acc += fps*VTOTAL ; if acc >= HFREQ: acc -= HFREQ ; PACE += 1
;
; Over VTOTAL/gcd raster frames that emits exactly fps*VTOTAL/gcd ticks, so the
; long-run rate is fps*VTOTAL/VTOTAL = fps EXACTLY, with a bounded remainder and
; ZERO accumulated drift -- not 12.0001, not 11.9998. It holds for any fps and
; any vertical geometry, which is why the two constants are READ OUT OF THE
; CRTC at init rather than assembled in: the clock is derived from the same
; registers that generate the raster it is counting, so the two cannot disagree.
;
; WHAT IT COSTS IN CADENCE, WHICH IS THE PART THAT IS NOT FREE. 12 fps on a
; 55.4577 Hz raster is 4.6215 refreshes per frame, so a frame is shown for
; either 4 or 5 refreshes -- 72.13 ms or 90.16 ms. Nothing can change that;
; it is the display's quantisation, not the clock's error, and a timer-derived
; clock would have exactly the same cadence with an arbitrary phase against the
; raster on top. It does mean the slot a frame gets is NOT always the 83.33 ms
; every budget in this project is priced against, and the short slot is 13.4%
; under it. tools/analysis/23_frame_clock.py prices that; do not read this file
; as a claim that the clock made the budget bigger.
;
; INTERRUPTS, AND WHAT HAD TO BE TURNED OFF. The rigs launch the 68000 at
; SR=$2700 with everything masked, into a machine the IPL ROM has already booted
; -- so the MFP arrives with whatever IOCS enabled on it (keyboard receive,
; Timer C, its own V-DISP handler) and vectors pointing into IOCS. Lowering the
; mask without disarming the MFP would vector into code we did not put there.
; clk_init therefore writes IERA = IERB = 0 first, which on the MC68901 also
; clears the matching pending bits (mc68901.cpp REGISTER_IERA/B: `m_ipr &=
; m_ier`), and only then arms GPIP4 alone. Levels 1-5 stay masked at SR=$2500,
; so the DMAC (IRQ3) and the SCC (IRQ5) cannot get in either; level 7 is the
; front-panel NMI and is not ours to mask.
;
; The vector is the MFP's own: VR is written with the S bit CLEAR, so the
; in-service register is not used and an acknowledge clears the pending bit by
; itself (mc68901.cpp get_vector). No end-of-interrupt write in the handler.
; ---------------------------------------------------------------------------
; --- MFP registers. The device sits on D0-D7 of a 16-bit bus (x68k.cpp:793,
; `.umask16(0x00ff)`), so register n is one BYTE at $E88001 + 2n.
MFP = $E88001
MFP_GPIP = MFP+0*2
MFP_AER = MFP+1*2
MFP_DDR = MFP+2*2
MFP_IERA = MFP+3*2
MFP_IERB = MFP+4*2
MFP_IPRA = MFP+5*2
MFP_IPRB = MFP+6*2
MFP_ISRA = MFP+7*2
MFP_ISRB = MFP+8*2
MFP_IMRA = MFP+9*2
MFP_IMRB = MFP+10*2
MFP_VR = MFP+11*2
MFP_GPIP4 = 4 ; bit number of V-DISP in GPIP/AER/DDR
MFP_IVDISP = $40 ; IR_GPIP_4, channel 6, in IERB/IPRB/IMRB
MFP_VBASE = $40 ; vector base; S clear -> no in-service register
CLK_VEC = (MFP_VBASE+6)*4 ; $118: MFP channel 6 vector, as an ADDRESS
CRTC = $E80000
CRTC_R04 = CRTC+4*2 ; V total, in scanlines, minus one
CRTC_R20 = CRTC+20*2 ; mode; bit 4 = 31.5 kHz
HFREQ = 31500 ; lines/s in the 31.5 kHz modes. Exact: the
; 768-wide IPL mode is 34.776 MHz / 1104 dots
; and the 256-wide mode 11.592 MHz / 368, both
; 31500.0 (tools/bench/crtc_mode.lua).
; --- state. PACE is stream.s's, deliberately: the whole point is that the
; 68000 now writes the word the host used to write, and the pace gate that
; reads it does not change by a single byte.
CLK_PACE = $18034 ; == stream.s PACE
CLK_ACC = $18060 ; word: Bresenham remainder, < HFREQ
CLK_INCR = $18062 ; word: fps * (R04+1), computed by clk_init
CLK_VDISP = $18064 ; long: V-DISP edges taken. An INSTRUMENT --
; it is what lets a rig check that the tick
; count and the raster count are the same clock.
CLK_FPS = $18068 ; long: requested fps, an argument to clk_init
CLK_ERR = $1806C ; long: 0 ok / 1 not a 31.5 kHz mode
; / 2 fps*VTOTAL would overflow 16 bits
; ---------------------------------------------------------------------------
; clk_init -- arm the frame clock. Reads CLK_FPS, leaves CLK_ERR.
; Clobbers d0-d2. Leaves the CPU at SR=$2500 on success.
; ---------------------------------------------------------------------------
clk_init:
clr.l CLK_ERR.l
clr.l CLK_VDISP.l
clr.w CLK_ACC.l
clr.l CLK_PACE.l
; The mode has to be the one HFREQ describes. A 15 kHz mode would halve the
; line rate and the divider would run at double speed while looking correct,
; which is the failure this test exists to prevent.
move.w CRTC_R20.l,d0
btst #4,d0
bne.s .modeok
move.l #1,CLK_ERR.l
rts
.modeok:
; VTOTAL and the increment. Both out of the CRTC, so a change of mode changes
; the clock with it. acc is 16-bit and reaches at most HFREQ-1+incr, so incr
; must leave room: 65536 - 31500 = 34036. At VTOTAL=568 that is fps < 59.9,
; which is every rate this machine can display anyway -- but it is checked
; rather than argued.
move.w CRTC_R04.l,d0
addq.w #1,d0 ; VTOTAL scanlines
move.w d0,d1
move.w CLK_FPS+2.l,d2 ; low word of the long
mulu d2,d1 ; fps * VTOTAL (see FINDINGS 53.4 on
; C68K's flat MULU charge; this is boot
; code and is not cost-measured there)
cmp.l #65536-HFREQ,d1
bcs.s .fitok
move.l #2,CLK_ERR.l
rts
.fitok:
move.w d1,CLK_INCR.l
; The vector, before the source is armed.
move.l #clk_isr,CLK_VEC.w
; Disarm everything the IPL left running, then arm GPIP4 alone. Order matters:
; IER first (which clears IPR with it), then the edge, then the mask.
move.b #0,MFP_IERA.l
move.b #0,MFP_IERB.l
move.b #0,MFP_IMRA.l
move.b #MFP_VBASE,MFP_VR.l ; S clear: acknowledge clears pending
bclr #MFP_GPIP4,MFP_DDR.l ; V-DISP is an input
bclr #MFP_GPIP4,MFP_AER.l ; interrupt on the FALLING edge, i.e.
; at the start of vertical blanking
move.b #MFP_IVDISP,MFP_IERB.l
move.b #MFP_IVDISP,MFP_IMRB.l
move.w #$2500,sr ; let level 6 in; 1-5 stay masked
rts
; ---------------------------------------------------------------------------
; clk_stop -- disarm, and put the mask back where the rigs expect it.
; ---------------------------------------------------------------------------
clk_stop:
move.w #$2700,sr
move.b #0,MFP_IERB.l
move.b #0,MFP_IMRB.l
rts
; ---------------------------------------------------------------------------
; clk_isr -- one V-DISP. Every instruction here is charged to every frame the
; decoder draws, so it is deliberately the shortest thing that is still exact:
; four word operations and one long increment.
;
; Only the LOW WORD of d0 is touched, so only the low word is saved. The
; accumulator, the increment and the threshold all fit in 16 bits by the check
; in clk_init, which is what makes that legal.
; ---------------------------------------------------------------------------
clk_isr:
move.w d0,-(sp)
addq.l #1,CLK_VDISP.l
move.w CLK_ACC.l,d0
add.w CLK_INCR.l,d0
cmp.w #HFREQ,d0
bcs.s .nf
sub.w #HFREQ,d0
addq.l #1,CLK_PACE.l
.nf:
move.w d0,CLK_ACC.l
move.w (sp)+,d0
rte
src/player/clockgate.s
+59
View File
@@ -0,0 +1,59 @@
; Front-end for the frame clock (ROADMAP P3), for the rig.
;
; It exists to answer two questions that the streaming rig cannot answer on its
; own, because there the clock is buried under a decoder:
;
; 1. does the tick actually come from the raster, and at exactly the rate
; asked for -- measured over thousands of refreshes, not four;
; 2. WHAT IT COSTS, in clocks, per interrupt. This project's currency is
; 68000 clocks and the decoder already occupies 86.7% of the bus, so a
; frame clock is not free until someone has priced it.
;
; THE INSTRUMENT, and why it is a busy loop. MAME's Lua only sees the machine
; at frame boundaries, so it can time to 1/55.46 s and no finer -- 18 ms, where
; the whole per-frame cost of this clock is microseconds. Differencing two
; wall timings would measure nothing. So the 68000 counts instead: a loop with
; ONE instruction in its body runs for a fixed number of refreshes, and the
; iteration count is read out at both ends.
;
; clock off: iters0 * L = clocks in the window -> L
; clock on: iters1 * L + ints * H = clocks in the window -> H
;
; The window is an exact number of raster frames, so its length in clocks is
; exact and does not depend on the host at all. L is calibrated out by the
; first run rather than assumed from a cycle table, which matters: the point of
; the exercise is to price this code on the machine that will run it, and a
; table is the thing being checked. With ~2.7e7 iterations behind it, L carries
; enough digits that the interrupt cost -- 0.08% of the window -- survives the
; subtraction.
;
; The body is `addq.l #1,CGCNT.l` and nothing else: no compare, no counter in a
; register that an interrupt could be accused of disturbing, and a value that
; the host can read at any moment without stopping the CPU.
;
; The gate does NOT decode anything. What the clock does to a real frame is
; tools/bench/stream.lua's question, with DLX_PACE=2.
CGFLAG = $18070 ; 0 idle / 1 running / $EE clk_init refused
CGON = $18074 ; 1 = arm the frame clock, 0 = leave it off
CGCNT = $18078 ; <- loop iterations, read by the host at both
; ends of the window
org $10000
start:
clr.l CGCNT.l
move.l CGON.l,d0
beq.s noclk
bsr clk_init
tst.l CLK_ERR.l
bne.s bad
noclk:
move.l #1,CGFLAG.l ; the host starts its window here
loop:
addq.l #1,CGCNT.l
bra.s loop
bad:
move.l #$EE,CGFLAG.l
hold: bra.s hold
include "src/player/clock.i"
src/player/dma.i
+273
View File
@@ -0,0 +1,273 @@
; The HD63450 driving the SCSI data phase. ROADMAP P4a.
;
; WHAT P4a HAS TO SHOW, and why it needed a new kind of evidence. FINDINGS 58
; measured the CPU moving every byte itself at 87.28 clocks per delivered byte
; -- 391.8% of a 12 fps frame, against 22.4% for the cheapest DMA row of the
; ladder and 85.3% for the dearest. So the whole of what is left before M2 is a
; DMAC configuration that HOLDS THE BUS. 57.3 is why it could not simply be
; watched into existence: x68k_scsiext.cpp glues $EA0015 so that with the DMAC's
; OWN asserted -- which it is at idle on this machine -- MAME CANNOT DISTINGUISH
; a CPU-driven byte at that address from a DMAC-driven one. Watching the data
; register cannot answer the question it looks like it answers.
;
; THE DISCRIMINATOR USED HERE IS THE CPU'S OWN PROGRESS, and it never reads
; $EA0015 at all. A DMAC that holds the bus is one the CPU is not running
; against; so the witness is a single instruction:
;
; move.b #CCR_START,DM_CCR ; the channel is told to go
; move.w DM_MTC,d0 ; <- sampled by the VERY NEXT instruction
;
; If the bus was held for the transfer, the whole transfer happened between
; those two instructions and d0 reads ZERO. If it was not, d0 reads very nearly
; the full count and the CPU goes on to spin thousands of times while the
; channel trickles. Both configurations deliver the same bytes; what separates
; them is whether the 68000 got to execute anything meanwhile, which is exactly
; what "holds the bus" means and is not a fact about $EA0015.
;
; WHAT MAME CAN AND CANNOT BE ASKED, stated here because it bounds the claim and
; it is not obvious from the outside:
;
; * THE CARD HAS NO EXREQ PATH. x68k_scsiext.cpp's drq_w only stores a flag;
; the expansion slot has no request line to the DMAC at all (x68k.cpp wires
; drq0 from the FDC and drq3 from ADPCM, and nothing else). The card's flow
; control is DTACK: on a DMAC cycle with DRQ low the card NEGATES DTACK and
; the HD63450 discards that operand and retries. So every configuration
; below is AUTO-REQUEST; REQG=10, external request -- the mode the ladder's
; W=5 and W=12 rows assume -- has no wiring in this model and cannot be run.
; * SINGLE ADDRESS CANNOT BE RUN EITHER. hd63450.cpp only takes the implicit
; path when a channel has a dma_read/dma_write callback, and on this machine
; only channel 0 (the FDC) has one. DTYP=10/11 on channels 1..3 falls
; through to the dual-address code.
; * ONLY BURST IS MODELLED AS HELD. The device tests `(dcr & 0xc0) == 0`, so
; XRM=10 (cycle steal without hold) and XRM=11 (cycle steal WITH hold) are
; one code path. The bus is held, and the CPU halted, only for XRM=00 burst
; with REQG=01 max rate.
;
; So of the four rows of the per-byte ladder, exactly ONE -- dual address, bus
; held, 9 clk/B -- has a code path in this model, and it is the one demonstrated
; below. That is a bound on the apparatus and not a result about the board.
;
; AND IT IS STILL NOT A RATE. MAME's DMAC is configured in wall-clock attotimes
; (42.5), not per-operand cycles: set_burst_clocks gives channel 1 450 ns an
; operand no matter what the 68000 is doing. `W` is untouched by every line in
; this file and still wants a board (ROADMAP B1/B3).
; ---- the channel. 1, not 0: channel 0 is the FDC's and is the one channel
; with device callbacks, which would silently take the implicit-address path.
; Channel 1 is also the channel the IPL ROM points at the SASI data register
; (52.5), so this is the machine's own disk channel programmed differently.
DMA_CH = 1
DMACB = DMAC+DMA_CH*DMAC_CH ; $E84040
DM_CSR = DMACB+$00 ; channel status (write 1s to clear)
DM_CER = DMACB+$01 ; channel error (read only)
DM_DCR = DMACB+$04 ; device control
DM_OCR = DMACB+$05 ; operation control
DM_SCR = DMACB+$06 ; sequence control
DM_CCR = DMACB+$07 ; channel control
DM_MTC = DMACB+$0A ; memory transfer count, WORD
DM_MAR = DMACB+$0C ; memory address, LONG
DM_DAR = DMACB+$14 ; device address, LONG
DM_BTC = DMACB+$1A ; base transfer count, WORD (array chain)
DM_BAR = DMACB+$1C ; base address, LONG (array chain)
DM_MFC = DMACB+$29
DM_CPR = DMACB+$2D
DM_DFC = DMACB+$31
; CSR bits
CSR_COC = $80 ; channel operation complete
CSR_BTC = $40
CSR_NDT = $20 ; normal device termination
CSR_ERR = $10 ; channel error -- CER says which
CSR_ACT = $08 ; channel active
CCR_START = $80
; ---- the two configurations, as (DCR, OCR) pairs. Both are decoded by
; tools/analysis/27_dmac_config.py out of THESE bytes, using the same MC68450
; field tables 21_iplrom_dmac.py reads the IPL ROM's channels with -- so what
; the run claims it programmed and what it programmed cannot drift apart.
;
; HELD : DCR $00 = XRM 00 burst, DTYP 00 dual address, DPS 0 8-bit port
; OCR $81 = DIR device->memory, SIZE byte, no chain, REQG 01 max rate
; STEAL: DCR $80 = XRM 10 cycle steal WITHOUT hold, otherwise identical
; OCR $80 = REQG 00 auto-request at limited rate
DM_HELD_DCR = $00
DM_HELD_OCR = $81
DM_STEAL_DCR = $80
DM_STEAL_OCR = $80
; ---- what the run reports. Every one of these is a DMAC register or a count
; of the CPU's own instructions; none of them is a read of $EA0015.
; $18500 AND NOT $18300, WHICH IS WHERE THIS FIRST WENT. scsi.i's trace ends at
; $182FF and the next 160 bytes are the RING's: $18300 is ring.i's XF_SLOT
; mailbox, and tools/bench/stream.lua reads the same addresses from outside.
; dma.i is included by stream.s as well as by the gate, so DM_USE landed on the
; transfer request slot and the ring rig's first record request read as "use the
; DMAC" -- P4b's stage went red on a run that never reached its snapshot. The
; symptom was in a stage this session did not touch, which is the whole argument
; for check.sh being run before and after rather than only after.
DM_USE = $18500 ; u32 0 = PIO data phase, 1 = this file
DM_DCRV = $18504 ; u32 the DCR byte to program
DM_OCRV = $18508 ; u32 the OCR byte to program
DM_MTC0 = $1850C ; u32 MTC one instruction after START
DM_SPIN = $18510 ; u32 times the CPU went round the wait
DM_CSRF = $18514 ; u32 CSR when the channel finished
DM_CERF = $18518 ; u32 CER with it
DM_MTCF = $1851C ; u32 MTC with it
DM_MARF = $18520 ; u32 MAR with it -- where it stopped
DM_LEN = $18524 ; u32 bytes the channel was asked for
; ---- SEQUENTIAL ARRAY CHAINING, and it is OFF unless a caller asks for it.
; A device->GVRAM transfer cannot be one contiguous run: a picture row is 256 B
; of a 1024 B line stride, so 192 rows want 192 destinations. The MC68450 walks
; an array of 6-byte {u32 MAR, u16 MTC} entries for exactly this, and MAME's
; hd63450 implements it (`(ocr & 0x0c) == 0x08`, dma_transfer_start and
; dma_transfer_continue). DM_BARV = 0 means no chaining and NOTHING below
; changes, which is what stream.s gets: this file is included by the player as
; well as by the gate, and a mailbox that defaults to a new behaviour is how
; DM_USE landed on ring.i's slot and turned a stage red (above).
DM_BARV = $18528 ; u32 array base address, 0 = no chain
DM_BTCV = $1852C ; u32 array entry count (BTC)
; ---- THE SERVICE HOOK, and it is the whole of what a SECOND consumer needs
; from this file. ROADMAP P6c. A player that feeds ADPCM has to look at the
; audio channel more often than once a frame: the MSM6258 has no FIFO, so the
; instant its channel counts out the chip goes on replaying whatever byte its
; data register still holds -- and at 12 fps a once-a-frame re-arm makes that
; replay 90 ms long, which is not a gap, it is a buzz.
;
; THE ONLY PLACE A 68000 HAS TO SPARE IS INSIDE THIS WAIT. In the STEALING
; configuration the CPU goes round the loop below thousands of times per record
; (measured: 1,100,520 trips over 120 frames) and every one of them is time the
; disc is delivering and the CPU is not. So the hook is called from there, and
; the second consumer costs the video path nothing it was using.
;
; AND IN THE HELD CONFIGURATION THE HOOK CANNOT RUN AT ALL, which is not a bug
; in it: a burst channel HALTS the 68000 (dma_transfer_start asserts
; INPUT_LINE_HALT) and the CPU does not execute the loop, or anything else,
; until the record has landed. That asymmetry is the measurement -- FINDINGS
; 64.3 showed a held channel costs the frame CLOCK half its ticks, and this is
; the same fact reaching the audio.
;
; ZERO BY DEFAULT and every other front-end in this tree leaves it zero, so the
; cost to them is a `move.l` and a `beq` per trip. A mailbox that defaulted to
; a new behaviour is how DM_USE landed on ring.i's slot (above).
DM_HOOK = $18530 ; u32 0 = none, else a routine to call
; on every trip round the transfer wait.
; d0 is dead here and a0 is saved round
; the call, so the hook may trash both;
; it must preserve EVERYTHING else,
; because sc_in_dma's own d3/d4/d5 and
; scsi_read's a1 are live across it.
DM_PATIENCE = 4000000 ; the wait is bounded like every other
; ---------------------------------------------------------------- sc_in_dma
; Receive d1 bytes into (a1) in phase d2, WITHOUT the CPU touching one of them.
; Entered from sc_in_data when DM_USE is set; same registers, same contract.
;
; ORDER MATTERS AND IT IS NOT THE OBVIOUS ONE. The SPC is put into DMA transfer
; BEFORE the channel is started, because in the held configuration the 68000
; stops executing at the CCR write and does not run again until the transfer is
; over -- so anything the SPC needs to be told has to have been told already.
sc_in_dma:
movem.l d3-d5,-(sp)
move.l d1,d5 ; keep the length for the report
move.l d5,DM_LEN.l
move.b d2,SC_PCTL
move.l d1,d0
bsr sc_settc ; the SPC counts the same bytes down
; ---- the channel, quiet first: CSR is write-one-to-clear and a stale
; COC from a previous record would pass the wait loop instantly.
move.b #$FF,DM_CSR
move.l DM_DCRV.l,d0
move.b d0,DM_DCR
move.l DM_OCRV.l,d0
move.b d0,DM_OCR
move.b #$04,DM_SCR ; MAC 01 memory increment, DAC 00 none:
; the device address is a REGISTER and
; must not walk off it.
move.b #$05,DM_MFC ; the function codes the IPL ROM uses
move.b #$05,DM_DFC
move.b #$01,DM_CPR
move.l DM_BARV.l,d0
bne.s .chain
move.w d5,DM_MTC
move.l a1,DM_MAR
bra.s .darset
.chain:
; MAR and MTC are NOT written: the channel loads both from the array's
; first entry when it starts, and reloads them from the next entry at
; every count-out. Writing them here would be writing registers the
; hardware is about to overwrite, which reads like a contract and is not.
move.l d0,DM_BAR
move.l DM_BTCV.l,d0
move.w d0,DM_BTC
.darset:
move.l #SC_DREG,DM_DAR ; $EA0015 -- the DMAC's door, and now
; the DMAC is the one going through it
move.b #SCMD_XFER,SC_SCMD ; no PROGRAM bit: the SPC raises DRQ
move.l #11,SC_TAG.l ; 11 = channel armed, SPC in DMA mode
bsr sc_snap
; ---- START, and the witness immediately after it
move.b #CCR_START,DM_CCR
move.w DM_MTC,d0 ; THE DISCRIMINATOR. Held: zero.
andi.l #$FFFF,d0
move.l d0,DM_MTC0.l
; ---- wait for the channel, counting the CPU's own trips round the loop.
; In the held configuration this is one trip, because the CPU did not
; get to run until the transfer was over. In the stealing one it is
; thousands, and every one of them is a 68000 instruction that executed
; while the disc was delivering -- which is the whole point of P4a.
clr.l DM_SPIN.l
move.l #DM_PATIENCE,d3
.wait: addq.l #1,DM_SPIN.l
move.l DM_HOOK.l,d0 ; the second consumer's slot -- see above
beq.s .nohook
move.l a0,-(sp)
movea.l d0,a0
jsr (a0)
movea.l (sp)+,a0
.nohook:
move.b DM_CSR,d4
btst #4,d4 ; ERR
bne.s .err
btst #7,d4 ; COC
bne.s .fin
subq.l #1,d3
bne.s .wait
bsr .report
movem.l (sp)+,d3-d5
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
.err: bsr .report
movem.l (sp)+,d3-d5
move.l #SCE_TIMEOUT,SC_ERR.l ; a channel error is a dead transport
moveq #-1,d0
rts
.fin: bsr .report
move.b #$FF,DM_CSR ; leave the channel as we found it
move.l #12,SC_TAG.l ; 12 = channel reported COC
bsr sc_snap
movem.l (sp)+,d3-d5
bsr sc_xferend ; the SPC's own transfer, not the DMAC's
move.l d0,-(sp)
move.l #9,SC_TAG.l
bsr sc_snap
move.l (sp)+,d0
rts
; ---- the channel's own account of what it did, read out of its registers
.report:
moveq #0,d0
move.b DM_CSR,d0
move.l d0,DM_CSRF.l
moveq #0,d0
move.b DM_CER,d0
move.l d0,DM_CERF.l
move.w DM_MTC,d0
andi.l #$FFFF,d0
move.l d0,DM_MTCF.l
move.l DM_MAR,d0
move.l d0,DM_MARF.l
rts
src/player/dmagate.s
+403
View File
@@ -0,0 +1,403 @@
; Front-end for the HD63450 DATA PHASE (ROADMAP P4a), for the rig.
;
; THE QUESTION. FINDINGS 58 put the transport on the 68000 and priced it: the
; CPU moving every byte itself costs 87.28 clocks per delivered byte, 391.8% of
; a 12 fps frame. Against that, the cheapest DMA row of the ladder is 22.4% and
; the dearest is 85.3%, so everything left before M2 turns on getting the DMAC
; to drive the data phase with the bus HELD. 57.3 is why it cannot be shown by
; watching the data register: with the DMAC's OWN asserted, which it is at idle
; here, MAME cannot tell a CPU-driven byte at $EA0015 from a DMAC-driven one.
;
; THE EVIDENCE THIS GATE PRODUCES, and none of it is a read of $EA0015:
;
; 1. THE SAME BYTES. The same sectors are read three times -- once by the PIO
; path FINDINGS 58 measured, once by the channel with the bus held, once by
; the channel stealing cycles -- and the HOST compares all three against its
; own copy of the image. A transport that returns the wrong bytes without
; saying so is the failure a checksum-free ring cannot survive (49.2).
; 2. THE CPU'S OWN PROGRESS. MTC is sampled by the INSTRUCTION AFTER the one
; that starts the channel. Held, it reads zero: the entire transfer
; happened between two instructions, because the 68000 did not execute in
; between. Stealing, it reads nearly the full count and the CPU then goes
; round its wait loop thousands of times while the bytes arrive. That
; difference IS "the DMAC held the bus", and it is a fact about the CPU.
; 3. THE CHANNEL'S OWN ACCOUNT. CSR, CER, the final MTC and the final MAR:
; the channel says it completed without error, moved every byte, and left
; its memory pointer exactly one transfer-length past where it started.
; 4. THE WINDOW IS REFUSED. A windowed read (58.3: 117 of 120 records start
; part way into a sector) is rejected by the transport rather than silently
; delivering the neighbouring records' bytes into the ring. P4a's
; precondition is stated by the code that has it, not by a comment.
;
; WHAT IT DOES NOT SHOW. Not `W`. Not one clock of it. MAME's DMAC runs on
; wall-clock attotimes (42.5) and its burst mode halts the CPU outright rather
; than costing it cycles per operand, so this gate settles WHICH CONFIGURATION
; WORKS and not what one costs. See src/player/dma.i for the three ways this
; model bounds the question -- no EXREQ wiring, no single-address path, and only
; burst modelled as held.
DGFLAG = $18600 ; 0 idle / 1 done
DGREC = $18800 ; 9 x 32 B: rc, err, mtc0, spin, csr, cer, mtc, mar
; $18800 AND NOT $18610, WHERE THIS LIVED: nine
; records of 32 B run to $188FF, and from $18610
; they would have run over DGWIN at $18700 --
; the window run's own result, which run 10 then
; writes back. A silent overlap between two
; runs' evidence is the kind of thing that makes
; a gate report the wrong run's numbers.
DGREC_SZ = 32
DGWIN = $18700 ; u32 return of the WINDOWED dma read (want -1)
DGWERR = $18704 ; u32 SC_ERR after it (want SCE_WINDOW)
DGR20 = $18708 ; u32 R20 as it stood during run 4
DGR20N = $1870C ; u32 R20 as it stood during run 5 (the control)
DGR20C = $18710 ; u32 R20 as it stood during run 6 (chained)
DGR20P = $18714 ; u32 R20 as it stood during run 9 (palette+rows)
R20_BUF = $0916 ; 256-colour, 31.5 kHz, G-VRAM SET TO BUFFER
R20_MSK = $0116 ; the SAME, bit 11 CLEAR -- the negative control
DGDST4 = $C0C000 ; where the control writes
DGCHA = $19000 ; the array-chain table: 8 x {u32 MAR, u16 MTC}
DGCHN = 8 ; entries
DGCROW = 256 ; bytes an entry carries -- one packed picture row
DGDST5 = $C10000 ; and the 8 row bases, at the 1024 B line stride
CRTC20 = $E80028 ; CRTC R20
DGDST3 = $C08000 ; DMA straight into GVRAM, BUFFER MODE (47.6.2)
DGLBA = 1000 ; a NON-ZERO LBA throughout: a driver that emits
; a malformed LBA field still passes LBA 0
DGBLK = 4 ; 4 x 512 = 2,048 B
DGDST0 = $20000 ; PIO
DGDST1 = $24000 ; DMA, bus held
DGDST2 = $28000 ; DMA, cycle stealing
; ---- runs 7-9: THE PALETTE (ROADMAP K1, FINDINGS 61.9's first open item).
DGPAL = $E82000 ; the GRAPHIC palette: 256 words, GGGGGRRRRRBBBBBI
DGPALN = 256 ; words in it -- and 512 B is exactly ONE sector
DGPBLK = 1 ; so the whole palette is one block off the disc
DGPOIS = $A500 ; the poison: word i = DGPOIS|i. A palette that
; still reads this was not written by anything.
DGDST6 = $2C000 ; run 8's destination: RAM, so the palette is
; left alone and must still read poison
DGDST7 = $C14000 ; run 9's six GVRAM rows, at the 1024 B stride
DGCHA2 = $19100 ; run 9's array: 7 x {u32 MAR, u16 MTC}
DGCHN2 = 7 ; palette 512 B + 6 rows x 256 B = 2,048 B
DGCROW2 = 256
DGPS7 = $1A000 ; the palette as it stood after run 7 ...
DGPS8 = $1A200 ; ... after run 8 (the control: poison) ...
DGPS9 = $1A400 ; ... and after run 9 (chained). SNAPSHOTS, not
; a late read: each run overwrites the previous
; run's palette, so a host that looks once at
; the end sees only the last of the three.
org $10000
start:
clr.l DGFLAG.l
; ---- 1. the PIO path, unchanged, as the reference the other two are measured
; against. It is here so that a DMA failure cannot be confused with a SCSI
; protocol failure: if this one is wrong, nothing below is about the DMAC.
bsr scsi_init
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST0,a1
bsr scsi_read
lea DGREC,a0
bsr dg_save
; ---- 2. the channel, WITH THE BUS HELD
bsr scsi_init
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l ; after scsi_init, which clears it
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST1,a1
bsr scsi_read
lea DGREC+DGREC_SZ,a0
bsr dg_save
; ---- 3. the channel, STEALING CYCLES. Same bytes, same code, two register
; values different -- which is what makes the comparison a comparison.
bsr scsi_init
move.l #DM_STEAL_DCR,DM_DCRV.l
move.l #DM_STEAL_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST2,a1
bsr scsi_read
lea DGREC+2*DGREC_SZ,a0
bsr dg_save
; ---- 4. THE CHANNEL WRITING GVRAM, IN BUFFER MODE. 47.6.2: "the DMAC has not
; been near this" -- 44.7 costed a device->GVRAM transfer and 47 built the packed
; layout, and no run in this tree has ever pointed a channel at $C00000. Two
; separate things are being asked at once and both are write-path questions:
;
; a. can a channel write GVRAM AT ALL? Nothing says it cannot -- the DMAC
; writes through the same program address space the CPU does -- but a
; transport that silently drops its writes into a device handler is exactly
; the failure this gate exists to catch, and the SPC's own bytes make a
; better witness than a fill pattern.
; b. does a BYTE-wide channel fill the PACKED layout? 47.1 measured the write
; path with word writes from the CPU. A dual-address channel with an 8-bit
; device port writes BYTES, and MAME's gvram_w in buffer mode passes
; mem_mask straight through -- so an even byte should land in the HIGH half
; of its word (page 1) and an odd byte in the LOW half (page 0). If it
; does, a linear DMA of a stream interleaved (right<<8)|left -- which is
; exactly show_frame256_packed.lua's layout -- fills the screen with no CPU
; in the loop at all.
;
; R20 bit 11 is left SET across the readback on purpose: gvram_r returns the raw
; word in buffer mode and the masked byte outside it, so clearing it first would
; hide half of what is being measured.
;
; R20 IS WRITTEN OUTRIGHT AND NOT OR-ED INTO. The first cut of this run set bit
; 11 on top of whatever the IPL left, and the IPL leaves $0B16 (22.1) -- bit 11
; ALREADY SET, and COL = %11, the 65,536-colour setup, which writes whole words
; with or without the bit. The run passed and proved nothing: it was a test that
; could not fail. Run 5 below is the control that makes this one mean something,
; and the two differ in EXACTLY BIT 11.
bsr scsi_init
move.w #R20_BUF,CRTC20.l
move.l #R20_BUF,DGR20.l
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST3,a1
bsr scsi_read
lea DGREC+3*DGREC_SZ,a0
bsr dg_save
; ---- 5. THE NEGATIVE CONTROL: the same transfer with bit 11 CLEAR. In masked
; 256-colour mode gvram_w takes `data & 0x00ff` and IGNORES mem_mask, so a byte
; written to an EVEN address -- where the 68000 puts the MS byte, and where every
; other disc byte lands -- contributes nothing and cannot be read back. Half the
; transfer must be lost, and if it is not, run 4 was not measuring the bit.
bsr scsi_init
move.w #R20_MSK,CRTC20.l
move.l #R20_MSK,DGR20N.l
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST4,a1
bsr scsi_read
lea DGREC+4*DGREC_SZ,a0
bsr dg_save
move.w #R20_BUF,CRTC20.l ; back to buffer mode, so the host reads
; RAW WORDS out of both destinations
; ---- 6. THE ROW STRIDE, WHICH IS THE REAL SHAPE OF THE TRANSFER. Runs 4 and 5
; wrote 2,048 contiguous bytes, and no picture is contiguous: a packed row is
; 256 B of a 1024 B line stride, so a frame is 192 destinations and not one.
; 46.6 said "no stride for a DMAC to skip" about the bytes WITHIN a row and left
; the rows themselves unexamined; a channel cannot skip 768 B any more than it
; could skip the 300 B in front of a record (run 10 below).
;
; The MC68450 answers this with SEQUENTIAL ARRAY CHAINING -- an array of 6-byte
; {u32 MAR, u16 MTC} entries it walks by itself -- and MAME implements it. So
; the question is not whether the CPU can restart the channel 192 times a frame;
; it is whether it has to at all. Eight rows here, out of the same LBA and the
; same 2,048 B, so the ONLY difference from run 4 is where the bytes land.
bsr scsi_init
move.w #R20_BUF,CRTC20.l
move.l #R20_BUF,DGR20C.l
lea DGCHA,a0
lea DGDST5,a1
moveq #DGCHN-1,d5
dg_mkch:
move.l a1,(a0)+ ; MAR: this row's base
move.w #DGCROW,(a0)+ ; MTC: 256 bytes of it
lea 1024(a1),a1 ; the next row is a line stride away
dbra d5,dg_mkch
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR|$08,DM_OCRV.l ; OCR CHAIN = %10, array chain
move.l #DGCHA,DM_BARV.l
move.l #DGCHN,DM_BTCV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST5,a1 ; ignored under chaining; passed so the
; call site reads the same as the others
bsr scsi_read
lea DGREC+5*DGREC_SZ,a0
bsr dg_save
clr.l DM_BARV.l ; and OFF again, so run 7 is unchained
; ---- 7. THE PALETTE. Can a channel write $E82000? (FINDINGS 61.9, ROADMAP
; K1.) Runs 4-6 put the PICTURE on the channel; a packed frame is a picture AND
; a palette, and if the palette registers take a byte-wide DMA the way GVRAM
; does in buffer mode then the palette is a 193rd array-chain entry and ONE
; channel start paints a whole frame -- no per-frame CPU work in the video path
; at all. If they do not, the CPU writes 256 words a frame (61.9 derives that
; at ~2,370 clocks, 0.28% of a frame) and the architecture still stands; this is
; the difference between cheap and free, and it is worth one run to know which.
;
; 512 B IS THE WHOLE PALETTE AND EXACTLY ONE SECTOR, which is why this run reads
; one block where the others read four: a transfer that ran long would write
; $E82200 (the TEXT palette) and then $E82400 (the video controller's own
; registers, priority included), and a probe that reconfigures the video
; controller as a side effect is not a probe.
;
; THE PALETTE IS POISONED FIRST, and that is what stops this being run 4's trap
; a second time. A destination that already holds the right bytes cannot tell a
; channel that wrote them from a channel that did nothing; RAM at $20000 was
; zero and the record is mostly pad, so "it matches" has been a weak claim all
; session. Word i is set to $A500|i, which no 512 B of container matches by
; accident, and the host reports how many of the 512 positions the poison and
; the disc actually differ in rather than assuming all of them.
bsr scsi_init
bsr dg_poison
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGPBLK,d4
lea DGPAL,a1
bsr scsi_read
lea DGPS7,a1
bsr dg_palsnap ; before run 8 overwrites it
lea DGREC+6*DGREC_SZ,a0
bsr dg_save
; ---- 8. THE NEGATIVE CONTROL, and it is an ATTRIBUTION control rather than a
; mechanism one. Run 5's control could point at a mode bit; there is no mode
; bit here, so what has to be excluded is that run 7's palette held the disc's
; bytes for some reason OTHER than the channel having written them there --
; a readback that aliases somewhere else, the SPC's own path touching the
; registers, the poison never having landed.
;
; Same transfer, same channel, same bytes; ONE thing different, the destination
; address. The disc's bytes must appear at $2C000, and the palette must still
; read poison in all 256 words. Two claims from one run, and the second is the
; one that makes run 7 mean anything: it also proves the CPU's poison writes
; reach the registers the host reads back, which is the positive half.
bsr scsi_init
bsr dg_poison
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGPBLK,d4
lea DGDST6,a1
bsr scsi_read
lea DGPS8,a1
bsr dg_palsnap ; must be poison, word for word
lea DGREC+7*DGREC_SZ,a0
bsr dg_save
; ---- 9. THE 193rd ENTRY: one start, the palette AND the picture rows. This is
; the run K1 exists for. Runs 7 and 8 only show that a channel can write the
; palette registers; what the architecture needs is that ONE array chain can
; cross from a hardware register area into GVRAM without the CPU between them,
; because a frame is a palette entry followed by 192 row entries and the whole
; claim is that the CPU starts the channel once.
;
; Seven entries, 2,048 B, out of the same LBA as everything else: 512 B into the
; palette and then six rows of 256 B at the 1024 B line stride. The destination
; regions are of two different KINDS -- device registers and video RAM in buffer
; mode -- which is exactly the crossing that has never been run.
bsr scsi_init
bsr dg_poison
move.w #R20_BUF,CRTC20.l
move.l #R20_BUF,DGR20P.l
lea DGCHA2,a0
move.l #DGPAL,(a0)+ ; entry 0: the palette, a whole sector
move.w #DGPALN*2,(a0)+
lea DGDST7,a1
moveq #DGCHN2-2,d5 ; the remaining six are picture rows
dg_mkch2:
move.l a1,(a0)+
move.w #DGCROW2,(a0)+
lea 1024(a1),a1
dbra d5,dg_mkch2
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR|$08,DM_OCRV.l ; OCR CHAIN = %10, array chain
move.l #DGCHA2,DM_BARV.l
move.l #DGCHN2,DM_BTCV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST7,a1 ; ignored under chaining, as in run 6
bsr scsi_read
lea DGPS9,a1
bsr dg_palsnap
lea DGREC+8*DGREC_SZ,a0
bsr dg_save
clr.l DM_BARV.l ; and OFF again, so run 10 is unchained
; ---- 10. and a WINDOWED read through the channel, which must be REFUSED. This
; is the one test here that is expected to fail, and it has to fail LOUDLY: the
; alternative is a channel writing a whole sector into a ring that has room for
; a record, over the top of records the decoder has not finished with.
bsr scsi_init
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #300,SC_WSKIP.l ; a record that starts 300 B into a sector
move.l #1024,SC_WKEEP.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST2,a1
bsr scsi_read_win
move.l d0,DGWIN.l
move.l SC_ERR.l,DGWERR.l
move.l #1,DGFLAG.l
hold: bra.s hold
; ---- one config's result, copied out of the shared reporting words before the
; next run overwrites them. d0 = scsi_read's return; a0 = where it goes.
dg_save:
move.l d0,(a0)+
move.l SC_ERR.l,(a0)+
move.l DM_MTC0.l,(a0)+
move.l DM_SPIN.l,(a0)+
move.l DM_CSRF.l,(a0)+
move.l DM_CERF.l,(a0)+
move.l DM_MTCF.l,(a0)+
move.l DM_MARF.l,(a0)+
; and clear them, so a config that never reached the channel reports
; zeros of its own rather than the previous config's numbers.
clr.l DM_MTC0.l
clr.l DM_SPIN.l
clr.l DM_CSRF.l
clr.l DM_CERF.l
clr.l DM_MTCF.l
clr.l DM_MARF.l
rts
; ---- the poison. Word i of the graphic palette <- $A500|i, written by the
; 68000 itself. Two jobs: a destination that cannot hold the right answer by
; accident, and a known pattern the control run reads back out of the registers
; to show the readback path shows what is actually in them.
dg_poison:
movem.l d0-d2/a0,-(sp)
lea DGPAL,a0
moveq #0,d1 ; i
move.w #DGPALN-1,d0
.p: move.w d1,d2
ori.w #DGPOIS,d2 ; $A500|i, and i never exceeds 255
move.w d2,(a0)+
addq.w #1,d1
dbra d0,.p
movem.l (sp)+,d0-d2/a0
rts
; ---- the palette as it stands, copied to (a1) by the 68000 READING THE
; REGISTERS. A snapshot and not a late host read: each palette run overwrites
; the last one's result, so all three have to be kept while they are true. It
; is also the same shape of evidence 53.3 used for pal_pack -- the words come
; back out of $E82000 rather than out of the buffer they were built in.
dg_palsnap:
movem.l d0/a0-a1,-(sp)
lea DGPAL,a0
move.w #DGPALN-1,d0
.s: move.w (a0)+,(a1)+
dbra d0,.s
movem.l (sp)+,d0/a0-a1
rts
include "src/player/scsi.i"
src/player/geom.i
+14
View File
@@ -25,3 +25,17 @@ SPCU = 12 ; bytes of code per COARSE span unit (24 px)
SPCN = 11 ; coarse units: 11*24 = 264 px >= one row
SPFU = 2 ; bytes of code per FINE span unit (2 px)
SPFN = 11 ; fine units: 11*2 = 22 px > one coarse unit
; RECORD ALIGNMENT, and it is a property of the CONTAINER rather than of the
; 68000. DLX2 padded each record up to 4, which is all `move.l (a0)+` needs
; (FINDINGS 28.3). DLX5 pads up to a 512 B SECTOR, so that a DMA channel can
; read a record as whole sectors straight into the ring with no window and no
; bounce copy -- `sc_in_data` REFUSES a windowed read when the data phase is
; the channel's (59.4), and 117 of 120 records needed one under DLX4.
;
; The consumer has to know it too: the decoder releases the ring up to the end
; of the record it was handed, and a decoder that released only the bytes it
; READ would leave the pad unreclaimed and drift the producer's free-space
; arithmetic by up to RECALN-1 per record. The ring base must therefore be
; RECALN-aligned, which is asserted where the ring is placed.
RECALN = 512
src/player/load.i
+256
View File
@@ -0,0 +1,256 @@
; ---------------------------------------------------------------------------
; load.i -- the two LOAD-TIME transforms, on the 68000 itself. ROADMAP P1+P2.
;
; Until now both of these were done host-side, in tools/bench/dlxload.py, and
; the rigs pushed the RESULT into emulated RAM. That was the right call while
; the inner loop was the thing being measured -- charging a once-per-scene cost
; to the per-frame path would have flattered or damned it for no reason -- but
; a player has no host. These are the bytes that replace it.
;
; The reference is tools/bench/dlxload.py and it stays the reference: this code
; is gated BYTE-FOR-BYTE against it (tools/bench/verify_load.py), palette words
; and darkest-entry index included. If the two ever disagree, the symptom in a
; rig would be wrong colours rather than a crash, which is exactly the class of
; bug the split was made to prevent.
;
; WHAT IT READS. The RAW container as it comes off the disc. The DLX header is
; fixed-layout and big-endian (tools/encoder/dlx.py):
; +0 magic 'DLX3' +12 k1 u16 +16 off_pal u32
; +4 W u16 +14 k4 u16 +20 off_cb1 u32
; +6 H u16 +24 off_cb4 u32
; +8 fps u16 +28 off_frm u32
; +10 nframes u16
; The three offsets are container-relative, so every one of them is an add of
; the base the loader was handed. Nothing here parses a frame record.
;
; WHAT IT WRITES. CB1 (8 KB) and CB4 (2 KB) expanded to one WORD per pixel at
; the addresses geom.i names, and 256 packed palette words straight into the
; graphics palette at $E82000. It also reports the darkest entry, which is what
; the letterbox is filled with until the encoder reserves a black one (23.4,
; still open).
;
; WHY WORD-PER-PIXEL. The block loop movems codebook entries straight into
; GVRAM with no unpacking, and the high byte of a GVRAM word write is discarded
; by the hardware, so the high byte is left zero and never has to be cleared.
; It also makes index scaling a shift rather than a multiply (lsl.w #5 / #3).
;
; SCRATCH. Three tables, built here and dead the moment the palette is packed:
; P6TAB 64 B 6-bit level -> the 8-bit value the hardware renders it as
; SQTAB 256 B the square of that, so the darkest-entry search has no muls
; DTAB 512 B err(v, I=0) - err(v, I=1) per 8-bit channel value, signed
; DTAB is what turns P2's per-entry minimum-squared-error choice of the shared
; LSB into three table reads and a sign test. Choosing I per entry rather than
; fixing it is worth 1.96 dB (FINDINGS 23.3), and it is a per-ENTRY decision
; across three channels, so it cannot be folded into a per-channel table alone.
; ---------------------------------------------------------------------------
LFLAG = $18040 ; 0 idle / 1 running / $FF done / $EE bad header
LHDR = $18044 ; -> raw container base
LDARK = $18048 ; <- index of the darkest palette entry
LK1 = $1804C ; <- k1, as the 68000 read it out of the header
LK4 = $18050 ; <- k4
LMODE = $18054 ; bit0 codebooks, bit1 palette entries,
; bit2 the three scratch tables
LITER = $18058 ; repeat count, so a 55 Hz host clock can time it
P6TAB = $19000 ; 64 bytes
SQTAB = $19040 ; 64 longs
DTAB = $19140 ; 256 words
GPAL = $E82000 ; graphics palette, 256 words
; ---------------------------------------------------------------- do_load
; in: a0 = container base, d1 = mode bits: 1 codebooks, 2 palette entries,
; 4 the scratch tables. A player builds the tables ONCE at boot (they
; describe the hardware's colour rendering and nothing about the scene) and
; then loads each scene with 3.
; out: d0 = 0 ok, -1 not a DLX3/DLX4 container. a0-a4 clobbered, a5 = base.
;
; The magic is accepted as 'DLX' plus a version byte of '3' OR ABOVE rather than
; as one constant. DLX4 (ROADMAP P5) adds the per-record index and a fifth
; header offset at +32; every field this routine reads is at its DLX3 place, so
; the transforms are version-independent and the check should be too. A version
; this loader has never seen is still refused -- '3' or above, not "anything
; that begins DLX".
do_load:
movea.l a0,a5
move.l (a5),d0
andi.l #$FFFFFF00,d0
cmpi.l #$444C5800,d0 ; 'DLX'
bne .bad
cmpi.b #'3',3(a5) ; ... version 3 or above
bcs .bad
move.w 12(a5),d0
ext.l d0
move.l d0,LK1.l
move.w 14(a5),d0
ext.l d0
move.l d0,LK4.l
btst #2,d1
beq.s .notab
move.l d1,-(sp)
bsr pal_tables
move.l (sp)+,d1
.notab:
btst #0,d1
beq.s .nocb
moveq #0,d2 ; the count is built as a LONG and the
move.w 12(a5),d2 ; high word must not carry junk into it
lsl.l #4,d2 ; k1 entries x 16 source bytes
movea.l 20(a5),a0
adda.l a5,a0
lea CB1,a1
bsr expand
moveq #0,d2
move.w 14(a5),d2
lsl.l #2,d2 ; k4 entries x 4 source bytes
movea.l 24(a5),a0
adda.l a5,a0
lea CB4,a1
bsr expand
.nocb:
btst #1,d1
beq.s .nopal
bsr pal_pack
.nopal:
moveq #0,d0
rts
.bad: moveq #-1,d0
rts
; ---------------------------------------------------------------- expand
; One source byte -> one destination word, high byte zero.
; in: a0 src, a1 dst, d2 = source byte count. Always a multiple of 4: CB1 is
; k1*16 and CB4 is k4*4, so no remainder case can exist and none is written.
; A junk high word here is not a slow path, it is a WRONG one: `lsr.l #2` walks
; two of its bits down into the low word and the dbra count comes out long.
expand:
lsr.l #2,d2
subq.l #1,d2 ; k<=256, so the count fits a dbra
moveq #0,d0
.e1: move.b (a0)+,d0
move.w d0,(a1)+
move.b (a0)+,d0
move.w d0,(a1)+
move.b (a0)+,d0
move.w d0,(a1)+
move.b (a0)+,d0
move.w d0,(a1)+
dbra d2,.e1
rts
; ---------------------------------------------------------------- pal_tables
; The three scratch tables. SCENE-INDEPENDENT, every one of them: they describe
; how the CRTC renders a 5-bit channel plus the shared LSB, which is a property
; of the machine. A player builds them once at boot and never again, which is
; why they are a separate entry point rather than the head of pal_pack -- see
; FINDINGS 53.3 for what that is worth.
pal_tables:
; -- P6TAB[x] = ((x<<2)|(x>>4)) & $FF, and SQTAB[x] = P6TAB[x]^2
lea P6TAB,a0
lea SQTAB,a1
moveq #0,d1
.p1: move.w d1,d0
lsl.w #2,d0
move.w d1,d2
lsr.w #4,d2
or.w d2,d0
andi.w #$FF,d0
move.b d0,(a0)+
move.w d0,d2
mulu d2,d2
move.l d2,(a1)+
addq.w #1,d1
cmpi.w #64,d1
bne.s .p1
; -- DTAB[v] = (render(v,0)-v)^2 - (render(v,1)-v)^2, signed
lea P6TAB,a0
lea DTAB,a1
moveq #0,d1
.p2: move.w d1,d2
lsr.w #2,d2
andi.w #$3E,d2 ; x0 = (v>>3)<<1
moveq #0,d3
move.b 0(a0,d2.w),d3
sub.w d1,d3
muls d3,d3
moveq #0,d4
move.b 1(a0,d2.w),d4
sub.w d1,d4
muls d4,d4
sub.l d4,d3
move.w d3,(a1)+
addq.w #1,d1
cmpi.w #256,d1
bne.s .p2
rts
; ---------------------------------------------------------------- pal_pack
; 24-bit RGB -> GGGGGRRRRRBBBBBI, the shared LSB chosen per entry by minimum
; squared error, written to the palette registers. Identical arithmetic to
; dlxload.pack_palette, including its tie-breaks: I stays 0 when the two errors
; are equal, and the darkest entry is the FIRST index at the minimum.
; in: a5 = container base, and pal_tables already run.
pal_pack:
movea.l 16(a5),a0
adda.l a5,a0 ; -> 256 x RGB888
lea GPAL,a1
lea DTAB,a2
lea SQTAB,a4 ; P6TAB is not needed here: the rendered
; value is only ever wanted SQUARED
move.l #$7FFFFFFF,d6
clr.l LDARK.l
moveq #0,d7
.p3: moveq #0,d1
move.b (a0)+,d1 ; R
moveq #0,d2
move.b (a0)+,d2 ; G
moveq #0,d3
move.b (a0)+,d3 ; B
move.w d1,d0
add.w d0,d0
move.w 0(a2,d0.w),d4
move.w d2,d0
add.w d0,d0
add.w 0(a2,d0.w),d4
move.w d3,d0
add.w d0,d0
add.w 0(a2,d0.w),d4 ; sum of err0-err1 over the three
moveq #0,d5
tst.w d4
ble.s .p4
moveq #1,d5 ; I=1 only when it is STRICTLY better
.p4: lsr.w #3,d1 ; fR
lsr.w #3,d2 ; fG
lsr.w #3,d3 ; fB
move.w d2,d4
lsl.w #5,d4
or.w d1,d4
lsl.w #6,d4 ; (fG<<11)|(fR<<6)
move.w d3,d0
add.w d0,d0
or.w d0,d4
or.w d5,d4
move.w d4,(a1)+ ; -> the palette register
add.w d1,d1 ; x = (f<<1)|I, per channel
or.w d5,d1
add.w d2,d2
or.w d5,d2
add.w d3,d3
or.w d5,d3
lsl.w #2,d1 ; SQTAB holds longs
move.l 0(a4,d1.w),d0
lsl.w #2,d2
add.l 0(a4,d2.w),d0
lsl.w #2,d3
add.l 0(a4,d3.w),d0 ; squared distance from black
cmp.l d6,d0
bge.s .p5
move.l d0,d6
move.l d7,LDARK.l ; first index at the minimum wins
.p5: addq.w #1,d7
cmpi.w #256,d7
bne .p3
rts
src/player/loadgate.s
+39
View File
@@ -0,0 +1,39 @@
; Front-end for the load-time transforms (ROADMAP P1+P2), for the rig.
;
; It is to load.i what decode.s is to frame.i: a timing and control wrapper that
; does nothing the shipping player would not do, so that the bytes being
; measured are the bytes that will ship. The player's own boot path will call
; do_load once with the mode bits set to 3; this repeats it LITER times so a
; host clock with 1/56.69 s granularity (tools/bench/crtc_mode.lua) can time a
; job that takes milliseconds,
; and splits it by LMODE so the codebook expansion and the palette pack can be
; priced apart. A player calls do_load with mode 7 once at boot -- the three
; scratch tables describe the machine, not the scene -- and with mode 3 at every
; scene change after that.
;
; Repeating is honest here in a way it would not be for a frame: nothing in
; do_load is temporally recursive. Pass n writes exactly what pass n-1 wrote,
; over the top of it, out of the same source bytes.
include "src/player/geom.i"
org $10000
start:
move.l LMODE.l,d1
move.l LITER.l,d3
move.l #1,LFLAG.l ; timer starts here
loop:
movem.l d1/d3,-(sp)
movea.l LHDR.l,a0
bsr do_load
movem.l (sp)+,d1/d3
tst.l d0
bne.s bad
subq.l #1,d3
bne.s loop
move.l #$FF,LFLAG.l ; timer stops here
hold: bra.s hold
bad: move.l #$EE,LFLAG.l
bra.s hold
include "src/player/load.i"
src/player/packed.s
+1308
View File
File diff suppressed because it is too large Load Diff
src/player/ring.i
+506
View File
@@ -0,0 +1,506 @@
; ---------------------------------------------------------------- ring.i
; The RING PRODUCER, on the 68000. ROADMAP P5.
;
; WHAT MOVED. FINDINGS 49 and 51 measured a ring that a HOST filled:
; tools/bench/stream.lua held the record index, decided where each record went,
; wrote the descriptor and advertised it. The 68000 only ever consumed. That
; is the same shape session 21 found in the loader and session 22 in the frame
; clock -- a policy living outside the machine that has to run inside it -- and
; it is the last one in the delivery path. A player has no host to place its
; records.
;
; So the placement policy is here now, and the host keeps only the part that is
; genuinely not the CPU's: moving bytes off a disc at a rate. What the rig
; supplies is a TRANSPORT, one request at a time, which is what a single SPC and
; one DMAC channel are (FINDINGS 52.5); what this file supplies is every
; decision about WHICH record, WHERE in the ring, and WHEN it is safe to start.
;
; THE POLICY IS `aligned`, and it is the same one 19_ring_stream.py scored and
; 49.3 chose: never start a record that will not finish before the end of the
; ring; leave the hole, restart at the base. It costs a mean hole of 5.7% of
; the ring on the gate container and ZERO clocks in the block loop, against
; `split`'s 3.64% of every frame budget forever.
;
; WHY IT NEEDS AN INDEX, and why that is a container change (DLX4). `aligned`
; asks "does the NEXT record fit before the end of the ring", which is a
; question about a record's length asked BEFORE it is fetched. Every reader in
; this tree learned record lengths by walking the frame stream -- reading each
; record's length word to find the next -- and that is exactly what a player
; streaming off a disc cannot do: the length word of record i+1 is one of the
; bytes it has not fetched yet. DLX4 puts nframes u16 longword-counts in the
; scene header for this, and the same table gives a branch point the disc
; address of an arbitrary record without reading what lies between (`ring_seek`).
;
; THE HANDSHAKE WITH THE DECODER IS UNCHANGED, deliberately. FR_HEAD/FR_TAIL/
; DESC[] are the same words src/player/stream.s already reads, written in the
; same order, so the decoder cannot tell a host-filled ring from a self-filled
; one -- which is what makes the self-filled run a test of THIS file and not of
; a new rig. Two monotonic counters, single reader, single writer, no atomics.
;
; THE TRANSPORT MAILBOX, XF_QD outstanding requests:
; XF_SLOT[seq & 1] what to fetch, where to put it, how much, and which
; record it is
; XF_GO requests issued -- bumped LAST, after the slot
; XF_ACK requests completed, in order, by the transport
; In the player XF_* is an MB89352 command and a DMAC channel; here it is
; tools/bench/stream.lua delivering at a modelled rate. Either way the CPU
; issues and polls, and the bytes arrive on somebody else's time.
;
; AND THAT IS WHERE THE COST IS. The channel only moves bytes while a request
; is outstanding, and only the CPU can issue the next one. Between the
; completion of record i and the issue of record i+1 the disc is IDLE, and the
; length of that gap is a property of the PLAYER's loop, not of the medium.
; `ring_poll` is therefore called from the pace wait -- the idle the frame clock
; already creates -- rather than once a frame: once a frame would cap the fill
; at one record per slot, which is the wire rate exactly, and a ring that can
; only just keep up can never accumulate the slack a branch point spends
; (51.3). A frame that uses its whole slot does not merely present late
; (54.4); it stops the disc for a frame time. The rig counts that gap.
; ---- transport mailbox. TWO REQUEST SLOTS, and the depth is a knob.
; A channel only moves bytes while it has a request, and only the CPU can give
; it one. With ONE slot the disc stands still from the moment a transfer
; completes until the player next polls -- and a player polls in its idle, which
; is the end of a frame slot, so the gap is up to a whole frame's decode. With
; TWO the next request is already queued when the current one lands and the
; channel need never stop. XF_QD selects which, so the cost of the first is
; measurable against the second in one rig rather than argued about.
XF_SLOT = $18300 ; 2 x 16 B: u32 disc offset, u32 destination,
; u32 length, u32 record index
XF_SLSZ = 16
XF_SLM = 1 ; slot = sequence & XF_SLM
XF_GO = $18320 ; u32 requests ISSUED, written by the 68000
XF_ACK = $18324 ; u32 requests COMPLETED, written by transport
XF_QD = $18328 ; u32 queue depth, 1 or 2 (input)
; ---- producer state
RINGOWN = $1832C ; 1 = the 68000 owns placement (this file)
RNG_B = $18330 ; ring base address
RNG_SZ = $18334 ; ring size in bytes
IDX_B = $18338 ; base of the DLX4 record index, nframes u16
RQ_NEXT = $1833C ; next record to REQUEST
WCUR = $18340 ; write cursor, a ring OFFSET
RCUR = $18344 ; read cursor: ring offset of the oldest record
; the decoder has not finished with
RTAILN = $18348 ; records RCUR has stepped over; chases FR_TAIL
NRETIRE = $1834C ; requests this file has published; chases XF_ACK
DOFF = $18350 ; running disc offset of record RQ_NEXT
; ---- instruments. None of these is read by the policy.
N_HOLE = $18354 ; wraps that left a hole
N_HOLEB = $18358 ; total bytes in those holes
N_FULL = $1835C ; polls that refused for SPACE (ring-bound)
N_POLL = $18360 ; ring_poll calls
N_ISSUE = $18364 ; requests issued
SLK_MIN = $18368 ; least slack seen at a frame start, in records
SLK_AT = $1836C ; and the frame it was seen at
PF_REC = $18370 ; prefill target, in whole records (input)
PF_DONE = $18374 ; records resident when the prefill released
N_SEEK = $18378 ; ring_seek calls
SK_WAIT = $1837C ; polls spent waiting for the channel to go
; quiet before the last seek could start
ROFF = $19400 ; u32 per record: disc offset, built at load.
; The seek half of the index -- a running sum
; is enough to PLAY, but a branch point needs
; record j's address without summing to it.
ROFFMAX = 1024 ; entries; $19400..$1A400, below CB1 at $20000
RPOLLMAX = 4000000 ; ring_poll calls with no progress before the
; producer is declared wedged
; ---------------------------------------------------------------- ring_init
; in: IDX_B, RNG_B, RNG_SZ, NFR set by the caller.
; out: d0 = 0 ok, -1 the index is longer than ROFF can hold. Builds the disc
; offset table and leaves the ring empty at record 0.
; Clobbers d0-d2/a0-a1.
ring_init:
move.l NFR.l,d0
cmpi.l #ROFFMAX,d0
bhi .toobig
movea.l IDX_B.l,a0
lea ROFF.l,a1
moveq #0,d1 ; running disc offset
move.l d0,d2
beq.s .noidx
.sum: move.l d1,(a1)+
moveq #0,d0
move.w (a0)+,d0 ; longwords in this padded record
lsl.l #2,d0
add.l d0,d1
subq.l #1,d2
bne.s .sum
.noidx:
clr.l NRETIRE.l
clr.l N_HOLE.l
clr.l N_HOLEB.l
clr.l N_FULL.l
clr.l N_POLL.l
clr.l N_ISSUE.l
clr.l N_SEEK.l
clr.l SK_WAIT.l
clr.l XF_GO.l
clr.l XF_ACK.l
move.l #$7FFFFFFF,SLK_MIN.l
move.l #-1,SLK_AT.l
moveq #0,d0
bsr ring_seek ; a scene starts with a seek to record 0
moveq #0,d0
rts
.toobig:
moveq #-1,d0
rts
; ---------------------------------------------------------------- ring_seek
; in: d0 = record index to play from.
; out: the ring is empty, the cursors are at its base, and the next request
; will be for record d0. FR_HEAD is reset; the CALLER must reset FR_TAIL
; (it is the decoder's word, and this file never writes the decoder's).
;
; A SEEK CANNOT START WHILE THE CHANNEL IS BUSY. An outstanding request is
; bytes already on their way to an address this routine is about to declare
; free, so it is waited out and thrown away rather than cancelled -- a real
; SPC would need the transfer aborted and the bus handed back before a new
; command, and waiting is the version of that a rig can be honest about. What
; it costs is up to one record's delivery time, charged to the seek, and
; SK_WAIT counts the polls it took.
; Clobbers d0-d2/a0.
ring_seek:
movem.l d0-d2/a0,-(sp)
addq.l #1,N_SEEK.l
clr.l SK_WAIT.l
.wait: move.l XF_ACK.l,d1
cmp.l XF_GO.l,d1
beq.s .quiet
addq.l #1,SK_WAIT.l
bsr xf_service ; the transport hook, and here it is not
; optional: with the transport INSIDE
; the machine (src/player/xfer.i) the
; only thing that can retire the
; outstanding request is this loop, so
; without it a seek issued with a
; request in flight spins forever. A
; host transport retired it on its own
; time, which is exactly the kind of
; difference the seam exists to hide and
; this one it could not.
bra.s .wait
.quiet:
move.l XF_GO.l,NRETIRE.l ; whatever landed belongs to the scene
; we came FROM, and is discarded
move.l d0,RQ_NEXT.l
lsl.l #2,d0
lea ROFF.l,a0
move.l (a0,d0.l),DOFF.l ; the index's second job: record j's
; disc address without reading to it
clr.l WCUR.l
clr.l RCUR.l
clr.l RTAILN.l
clr.l FR_HEAD.l
movem.l (sp)+,d0-d2/a0
rts
; ---------------------------------------------------------------- ring_poll
; Advance the producer by at most one step: retire a completed request, catch
; the read cursor up with the decoder, and issue the next request if one fits.
; Preserves every register -- it is called from inside the decoder's wait loops
; and must be invisible to them.
ring_poll:
movem.l d0-d3/a0-a1,-(sp)
addq.l #1,N_POLL.l
; ---- 0. the TRANSPORT, if it lives in this machine. src/player/xfer.i answers
; at most one outstanding request per call and preserves every register;
; with XF_SCSI = 0 it is a tst and a branch, and the host is the transport
; exactly as it was in FINDINGS 55. It goes BEFORE the retire step so that
; a transfer completed here is published in the same poll.
bsr xf_service
; ---- 1. retire. The descriptor is written BEFORE the count that advertises
; it, which is the same order tools/bench/stream.lua used and the reason
; src/player/stream.s reads them the other way round.
move.l NRETIRE.l,d3
.retire:
cmp.l XF_ACK.l,d3
bcc.s .retired ; d3 >= XF_ACK: nothing new has landed
move.l d3,d0
and.l #XF_SLM,d0
lsl.l #4,d0 ; * XF_SLSZ
lea XF_SLOT.l,a1
adda.l d0,a1 ; a1 = the completed request's slot
move.l 12(a1),d0 ; its record index
lsl.l #2,d0
and.w #DESCM,d0
lea DESC.l,a0
move.l 4(a1),(a0,d0.w) ; its destination -> the descriptor
addq.l #1,FR_HEAD.l ; ...advertised only after the address
addq.l #1,d3
bra.s .retire
.retired:
move.l d3,NRETIRE.l
; ---- 2. catch the read cursor up. The decoder publishes FR_TAIL and nothing
; else the producer needs: RCUR walks the SAME placement rule the writer
; used, so it steps over the holes in exactly the places they were left.
; That is what makes the free space a single circular gap rather than a
; list of live records -- the host producer kept a list because it could
; afford to.
; THE RULE MUST BE THE WRITER'S, APPLIED TO THE SAME RECORD. Stepping
; the reader past record i lands on the END of record i, which is where
; record i+1 went only if i+1 FITTED there -- and if it did not, the writer
; put it at the ring base and left a hole. So the wrap is decided by the
; length of the record being stepped ONTO, exactly as the placement was.
;
; Deciding it with the wrong record's length was a real bug and not a
; conservative one: it left RCUR pointing into the hole, and one more
; retirement then pushed it past the end of the ring and wrapped it to a
; low address unrelated to any record. The live span computed from that is
; SHORTER than the truth, so the producer places on top of a record the
; decoder has not finished, and the symptom is a bitstream desync -- the
; decoder's a0 walking off the end of a record that changed underneath it.
move.l RTAILN.l,d1
.catch: cmp.l FR_TAIL.l,d1
bcc.s .caught
movea.l IDX_B.l,a0
move.l d1,d0
add.l d0,d0
moveq #0,d2
move.w (a0,d0.l),d2
lsl.l #2,d2 ; length of the record being retired
move.l RCUR.l,d0
add.l d2,d0 ; d0 = one past its end
addq.l #1,d1
cmp.l NFR.l,d1
bcc.s .last ; nothing follows it in this scene
move.l d1,d2
add.l d2,d2
moveq #0,d3
move.w (a0,d2.l),d3
lsl.l #2,d3 ; length of the record after it
add.l d0,d3
cmp.l RNG_SZ.l,d3
bls.s .last
moveq #0,d0 ; it did not fit: the writer restarted
; at the base, so the reader does too
.last:
move.l d0,RCUR.l
bra.s .catch
.caught:
move.l d1,RTAILN.l
; ---- 3. issue, if there is anything left and it fits.
move.l RQ_NEXT.l,d1
cmp.l NFR.l,d1
bcc .out ; whole scene requested
; ---- MEASURED AGAINST WHAT HAS BEEN RETIRED, NOT WHAT HAS BEEN ACKED, and
; the difference is a slot. A request's slot stays in use until this file
; has read the record index and destination out of it -- which happens in
; step 1 above, one poll later than the ack at the earliest. Gating on
; XF_ACK let the CPU write a slot whose descriptor had not been published
; yet: the transport had finished the transfer, the retire loop then read
; the OVERWRITTEN slot, and DESC for that frame stayed zero. The decoder
; duly decoded address zero and reported a bitstream desync.
move.l XF_GO.l,d0
sub.l NRETIRE.l,d0 ; slots still spoken for
cmp.l XF_QD.l,d0
bcc .out ; the queue is as deep as it may go
move.l d1,d0
add.l d0,d0
movea.l IDX_B.l,a0
moveq #0,d2
move.w (a0,d0.l),d2
lsl.l #2,d2 ; d2 = length to place
; ---- WHERE IT GOES, AND WHETHER IT MAY. The live bytes are the circular
; interval [RCUR, WCUR) -- oldest record the decoder has not finished with,
; up to the write cursor -- so the FREE bytes are its complement, and a
; record has to fit in ONE piece of it because the block loop reads with a
; monotonically increasing a0 (49.2).
;
; There are three shapes and they are not symmetric, which is the trap:
; empty the whole ring is free
; RCUR <= WCUR live is one run; free is [WCUR, SZ) THEN [0, RCUR),
; so a record that will not fit before the end may
; restart at the base -- this is `aligned`, and the
; skipped bytes are the hole
; RCUR > WCUR LIVE is the one that wraps; free is only [WCUR, RCUR)
; and the ring base is NOT ours -- a record that will
; not fit must simply wait
; Deciding the wrap from `WCUR + len > SZ` alone, before knowing which
; shape it is, was the second bug in this file: in the third shape it
; restarted at a base that was live and overwrote records the decoder had
; not read, and the symptom was a bitstream desync rather than a fault.
move.l WCUR.l,d3 ; d3 = candidate offset
moveq #0,d0 ; d0 = hole bytes, if any
move.l RQ_NEXT.l,d1
cmp.l RTAILN.l,d1
beq.s .isempty
move.l RCUR.l,d1
cmp.l d3,d1
beq .full ; RCUR == WCUR and not empty: the ring
; is completely full
bhi.s .freehi
; RCUR < WCUR: free is [WCUR, SZ) then [0, RCUR).
move.l d3,d0
add.l d2,d0
cmp.l RNG_SZ.l,d0
bls.s .nohole2 ; fits before the end of the ring
move.l RCUR.l,d0
cmp.l d2,d0
bcs .full ; it will not fit at the base either
move.l RNG_SZ.l,d0
sub.l d3,d0 ; the hole `aligned` is about to leave.
; Charged here, where the record is
; actually PLACED, and not where the
; wrap is decided: charging it at the
; decision counts one hole per retry
; while the decoder still owns the base,
; which is every poll of a fast pipe,
; and reported 105 wraps where there
; are 18.
moveq #0,d3
bra.s .place
.nohole2:
moveq #0,d0
bra.s .place
.freehi:
; RCUR > WCUR: the LIVE span wraps, so the only free run is [WCUR, RCUR).
move.l d3,d0
add.l d2,d0
cmp.l RCUR.l,d0
bhi .full
moveq #0,d0
bra.s .place
.isempty:
; Nothing live, so the whole ring is free and the live span is about to start
; here. Moving RCUR is what keeps the invariant true across a drained ring;
; without it the reader's cursor would still point at the last consumed record.
move.l d3,d0
add.l d2,d0
cmp.l RNG_SZ.l,d0
bls.s .enohole
move.l RNG_SZ.l,d0
sub.l d3,d0
moveq #0,d3
bra.s .esetr
.enohole:
moveq #0,d0
.esetr:
move.l d3,RCUR.l
.place:
tst.l d0
beq.s .nohole
addq.l #1,N_HOLE.l
add.l d0,N_HOLEB.l
.nohole:
move.l XF_GO.l,d1
and.l #XF_SLM,d1
lsl.l #4,d1 ; * XF_SLSZ
lea XF_SLOT.l,a1
adda.l d1,a1
move.l DOFF.l,(a1)
move.l RNG_B.l,d0
add.l d3,d0
move.l d0,4(a1)
move.l d2,8(a1)
move.l RQ_NEXT.l,12(a1)
add.l d2,d3
move.l d3,WCUR.l
add.l d2,DOFF.l
addq.l #1,RQ_NEXT.l
addq.l #1,N_ISSUE.l
addq.l #1,XF_GO.l ; LAST: the three words above must be
; visible before the request is
bra.s .out
.full:
addq.l #1,N_FULL.l
.out:
movem.l (sp)+,d0-d3/a0-a1
rts
; ---------------------------------------------------------------- ring_prefill
; Fill until PF_REC whole records are resident, then return. THE POLICY, not a
; convenience: the decoder must not be released at slack 1, because 51.2
; measured that n resident records buy n-1 frame times of stall -- the last one
; is spent covering the pipe's restart. Releasing at 1 therefore starts a scene
; with a stall budget of zero, and the first hiccup is an underrun.
;
; It is also the ONLY place a player can buy lookahead cheaply. 51.3: slack is
; accumulated out of `pipe - wire` over seconds of play, so a scene that starts
; empty climbs for 4.83 s at 488 KB/s before it can afford a branch. Bytes
; bought here are bought before the frame clock starts and cost nothing but the
; wait -- which is the one moment in a scene when the decoder has nothing else
; to do anyway.
;
; out: d0 = 0 ok, -1 the transport never delivered. PF_DONE = records resident.
ring_prefill:
movem.l d1-d2,-(sp)
moveq #0,d1
.loop: bsr ring_poll
move.l FR_HEAD.l,d0
sub.l FR_TAIL.l,d0
cmp.l PF_REC.l,d0
bcc.s .done
addq.l #1,d1
cmp.l #RPOLLMAX,d1
bcs.s .loop
movem.l (sp)+,d1-d2
moveq #-1,d0
rts
.done: move.l d0,PF_DONE.l
movem.l (sp)+,d1-d2
moveq #0,d0
rts
; ---------------------------------------------------------------- ring_slack
; out: d0 = whole records resident and unconsumed.
;
; This is the number FINDINGS 51 spent a session establishing the meaning of,
; and it is only worth what it is worth when the decoder is PACED: free-running,
; the decoder outruns any pipe and the ring never backs up, so the difference is
; a statement about earliness (49.7.2).
ring_slack:
move.l FR_HEAD.l,d0
sub.l FR_TAIL.l,d0
rts
; ---------------------------------------------------------------- ring_may_seek
; in: d0 = the stall a branch would cost, in whole frame times.
; out: d0 = 0 the ring can cover it, -1 it cannot. Z set on ok.
;
; THE RULE, from 51.2, measured and not assumed: n resident records buy n-1
; frame times, because the record due immediately after the pipe restarts is
; still arriving when its slot opens. A design that reads the resident count
; as its stall budget is over by one record every time.
;
; What a player does with a `no` is not this file's business -- delay the
; branch, take the outcome that needs no seek, or accept a late present -- but
; it must be able to ASK, and until now the answer only existed in the rig's
; log.
ring_may_seek:
move.l d1,-(sp)
move.l FR_HEAD.l,d1
sub.l FR_TAIL.l,d1
beq.s .no ; nothing resident: the subtraction
; below would wrap to $FFFFFFFF and an
; unsigned compare would then answer YES
; to any request, from an empty ring
subq.l #1,d1 ; the restart record is not spendable
cmp.l d0,d1
bcs.s .no
move.l (sp)+,d1
moveq #0,d0
rts
.no: move.l (sp)+,d1
moveq #-1,d0
rts
; ---------------------------------------------------------------- ring_mark
; Sample the slack at a frame boundary, for the instruments only.
; in: d0 = frame number. Clobbers nothing.
ring_mark:
movem.l d0-d1,-(sp)
move.l FR_HEAD.l,d1
sub.l FR_TAIL.l,d1
cmp.l SLK_MIN.l,d1
bcc.s .out
move.l d1,SLK_MIN.l
move.l d0,SLK_AT.l
.out: movem.l (sp)+,d0-d1
rts
src/player/scsi.i
+695
View File
@@ -0,0 +1,695 @@
; ---------------------------------------------------------------- scsi.i
; The MB89352 TRANSPORT, on the 68000. ROADMAP P4.
;
; WHAT THIS REPLACES. src/player/ring.i decides which record to fetch, where in
; the ring to put it and when it is safe; it hands that to a TRANSPORT through
; the XF_* mailbox and polls for completion. Until now the transport was
; tools/bench/stream.lua -- a host moving bytes at a modelled rate. A player has
; no host. This file is the transport: a Fujitsu MB89352 SPC and, later, one
; HD63450 channel.
;
; THE REGISTER MAP IS MEASURED, NOT ASSUMED. src/player/scsigate.s probes
; $EA0000..$EA003F one address at a time and survives the bus errors, and the
; map below is what answered:
;
; registers sit on the ODD bytes, $EA0001 + 2n, n = 0..14
; n=3 (TMOD) and n=15 (EXBF) BUS ERROR -- the MB89352 omits both, where the
; MB87030 has them, and MAME leaves HOLES rather than shifting the
; later indices down. DREG is index 10 at $EA0015 either way, which is the
; one address FINDINGS 32.4 had quoted.
; TEMP ($EA0017) took $A5 and gave it back, so these are registers and not a
; mirror of something.
;
; That last point is why the probe was worth a run: MAME's own device summary
; says the MB89352 "shifts subsequent indices", and the machine says it does
; not. The bytes win.
;
; THE DATA REGISTER IS DMA-ONLY, AND THAT IS NOT A CHOICE THIS CODE MADE.
; x68k_scsiext.cpp puts its own glue on $EA0015 and on no other address:
;
; write: if (exown()) { if (!drq) dtack_w(1); else dma_w(data); }
; else dreg_w(data);
;
; On this machine `exown()` -- the HD63450's OWN, fed back to the slot by
; x68k.cpp -- is asserted where a PIO write needs it not to be, so the `else`
; arm is unreachable and a byte written to $EA0015 with the SPC in PROGRAM
; transfer mode is DISCARDED. Silently: no error bit, no status change, no
; interrupt. It was measured rather than reasoned about -- scsigate.s writes
; $5A to $EA0015 and reads it straight back, and gets $00 with the FIFO still
; empty -- because ten command bytes vanishing without trace looks exactly like
; a target refusing a command.
;
; So every transfer here issues SCMD WITHOUT the PROGRAM bit, which puts the SPC
; in DMA mode and makes it raise DRQ; the CPU then moves the bytes through
; $EA0015 itself and they go in via `dma_w`/`dma_r`. The CPU is standing in for
; the DMAC, through the DMAC's own door.
;
; WHAT THAT COSTS THE ARGUMENT, stated because it is easy to overclaim here:
; with `exown` asserted at idle, MAME cannot distinguish a CPU-driven byte at
; $EA0015 from a DMAC-driven one. So this rig demonstrates THE DATA PATH and
; cannot, on its own, demonstrate that the HD63450 is the thing driving it.
; Whether a real CZ-6BS1 also refuses PIO here is NOT settled by this -- it is a
; property of MAME's model, and it wants a board (ROADMAP B1/B3).
;
; PIO FIRST, DMA SECOND, DELIBERATELY. The thing P4 has to demonstrate is a
; DMAC configuration that HOLDS THE BUS (ROADMAP: "getting the DMAC to hold the
; bus is the difference between 9 and 19 clocks per byte, and demonstrating a
; configuration that does it is P4's first job"). But a DMA bring-up that fails
; cannot tell "the SCSI protocol is wrong" from "the DMAC is misprogrammed". So
; the protocol is settled in PIO, where every byte is the CPU's and nothing else
; can be blamed, and only then does the data phase move to the channel.
;
; NOTHING HERE IS A RATE MEASUREMENT, and it cannot become one. MAME's device
; models are functional, not transfer-timing accurate (docs/BENCHMARK.md), and
; 42.5 reads its DMAC configured in wall-clock attotimes rather than per-operand
; cycles. `W` -- clocks stolen per delivered byte -- is untouched by every line
; below. What this settles is which handshake the player's own code provokes.
; ---- the SPC, at the CZ-6BS1's decode
SPCB = $EA0001 ; register 0; stride 2, odd lane
SC_BDID = SPCB+0 ; own ID (write the NUMBER; reads a MASK)
SC_SCTL = SPCB+2
SC_SCMD = SPCB+4
; SPCB+6 = TMOD, ABSENT on the MB89352 -- reading it BUS ERRORS
SC_INTS = SPCB+8
SC_PSNS = SPCB+10
SC_SSTS = SPCB+12
SC_SERR = SPCB+14
SC_PCTL = SPCB+16
SC_MBC = SPCB+18
SC_DREG = SPCB+20 ; $EA0015, and the DMAC's single address
SC_TEMP = SPCB+22
SC_TCH = SPCB+24
SC_TCM = SPCB+26
SC_TCL = SPCB+28
; SPCB+30 = EXBF, ABSENT -- reading it BUS ERRORS
; SCTL
SCTL_RESET = $80 ; reset & disable
; SCMD, command in bits 7-5
SCMD_RELEASE = $00 ; command 000, let go of the bus
SCMD_RSTACK = $C0 ; command 110, drop ACK/REQ
SCMD_SELECT = $20
SCMD_RSTATN = $40 ; command 010, drop ATN
SCMD_XFER = $80
SCMD_PROGRAM = $04 ; set = PIO, clear = DMA. NOT USED, and
; the reason is the whole of 57.x -- see
; "THE DATA REGISTER IS DMA-ONLY" above.
; INTS
INTS_RESET = $01
INTS_HARDERR = $02
INTS_TIMEOUT = $04
INTS_SERVICE = $08
INTS_CMDCOMP = $10
INTS_DISCON = $20
; SSTS
SSTS_DREG_E = $01 ; DREG empty
SSTS_DREG_F = $02 ; DREG full
SSTS_TC0 = $04
SSTS_BUSY = $20
SSTS_INITCON = $80
; SCSI bus phases, as PSNS bits 2..0 and as PCTL's low three
PH_DATAOUT = 0
PH_DATAIN = 1
PH_CMD = 2
PH_STATUS = 3
PH_MSGOUT = 6
PH_MSGIN = 7
SCSI_ID = 7 ; the player is the initiator
SCSI_TGT = 0 ; the disc
; ---- error codes, reported through SC_ERR
SCE_OK = 0
SCE_SELTMO = 1 ; the target never answered selection
SCE_PHASE = 2 ; the bus went somewhere unexpected
SCE_TIMEOUT = 3 ; a poll loop ran out of patience
SCE_STATUS = 4 ; the target returned non-zero status
SCE_WINDOW = 5 ; a WINDOWED read was asked of the DMAC,
; which cannot drop bytes (58.3/P4a)
SC_ERR = $18200 ; u32 last error
SC_STAT = $18204 ; u32 SCSI status byte from the last cmd
SC_PH = $18208 ; u32 phase we were in when it went wrong
SC_CDB = $18210 ; 12 B command block, built here
SC_MSG = $1821C ; 4 B message byte, either direction
; ---- THE RECORD WINDOW, and why a transport needs one. ROADMAP P4b.
; src/player/ring.i asks for a RECORD: a byte offset into the scene's frame
; stream and a length, both of them 4-byte aligned and neither of them a
; multiple of 512. A SCSI target deals in BLOCKS. On the gate container 117
; of 120 records start part way into a sector, so a transport that reads only
; whole sectors delivers the record plus up to 511 bytes in front of it and up
; to 511 behind, and those neighbouring bytes belong to records the decoder may
; still be reading -- the block loop walks a0 with no bounds check (49.2), so
; landing them in the ring is a corruption, not a waste.
;
; IN PIO THE FIX IS FREE, and that is the only reason this is affordable here:
; the CPU is already touching every byte, so it simply does not STORE the ones
; outside the window. SC_WSKIP bytes are pulled from the FIFO and dropped, the
; next SC_WKEEP are stored, the rest are pulled and dropped. Three loops rather
; than one steered loop, deliberately: the middle one is then byte-for-byte as
; tight as the un-windowed sc_in_pio, so the per-byte cost this rig reports is
; the transport's and not the window's.
;
; UNDER A DMAC IT IS NOT FREE, and that is P4a's problem arriving early. A
; channel writes a contiguous run to a contiguous address; it cannot be told to
; drop the first 300 bytes. So when the data phase moves to the HD63450 the
; choice is a bounce buffer plus a copy of every byte (the cost `aligned` was
; chosen to avoid, 49.3) or sector-aligned records in the container -- which is
; a re-encode. 57 measured which is cheaper; see FINDINGS 58.3.
SC_WSKIP = $18220 ; u32 bytes to drop before the window
SC_WKEEP = $18224 ; u32 bytes of window to store
; ---- a TRACE, because a SCSI bring-up cannot be debugged from one error code.
; Four registers at each interesting instant: SSTS, PSNS, INTS, SERR. MAME's
; SCMD_CMD_TRANSFER is a NO-OP unless SSTS_INIT_CONNECTED is set -- it `break`s
; out of the switch without complaint -- so "the transfer did nothing" and "the
; transfer went wrong" look identical from the outside. The trace separates
; them.
SC_TAG = $1822C ; u32 where the next snapshot came from
SC_TRN = $18230 ; u32 trace entries used
SC_TR = $18240 ; 24 x 8 B: SSTS PSNS INTS SERR TCH TCM TCL TAG
; A poll bound. Every wait in this file is bounded, because a SCSI phase that
; never arrives must be REPORTED -- an unbounded spin is indistinguishable from
; a wedged emulator, and 34.1 already cost this project fifteen minutes to that
; exact confusion.
SC_PATIENCE = 200000
; ---------------------------------------------------------------- sc_snap
; Append SSTS/PSNS/INTS/SERR to the trace. Clobbers nothing the callers use.
sc_snap:
movem.l d0/a0,-(sp)
move.l SC_TRN.l,d0
cmp.l #24,d0
bge.s sn_out
lea SC_TR.l,a0
lsl.l #3,d0
adda.l d0,a0
move.b SC_SSTS,(a0)+
move.b SC_PSNS,(a0)+
move.b SC_INTS,(a0)+
move.b SC_SERR,(a0)+
move.b SC_TCH,(a0)+
move.b SC_TCM,(a0)+
move.b SC_TCL,(a0)+
move.b SC_TAG+3,(a0)+ ; WHERE this snapshot was taken
addq.l #1,SC_TRN.l
sn_out: movem.l (sp)+,d0/a0
rts
; ---- the HD63450, so that PIO through the card's data register works at all.
; x68k_scsiext.cpp puts DMA-AWARE GLUE on $EA0015 and nowhere else:
;
; write: if (exown()) { if (!drq) dtack_w(1); else dma_w(data); }
; else dreg_w(data);
;
; With OWN asserted and DRQ low the byte is DROPPED, silently. That is the
; bring-up's fourth bug and the least guessable: ten command bytes went into
; $EA0015, the FIFO stayed empty, the transfer counter stayed at 10, and every
; register the SPC has said "waiting". Nothing reports a discarded write.
; OWN is the DMAC's, and the IPL ROM has been running for three seconds before
; the player's first instruction (52.5 reads its channel setup out of the ROM),
; so the player does not inherit a quiet DMAC -- it has to make one.
DMAC = $E84000
DMAC_CH = $40 ; channels are 64 B apart
dmac_quiet:
lea DMAC,a0
moveq #3,d1
dq1: move.b #0,7(a0) ; CCR: no operation
move.b #$FF,0(a0) ; CSR: write-one-to-clear
adda.w #DMAC_CH,a0
dbra d1,dq1
rts
; ---------------------------------------------------------------- scsi_init
; Reset the SPC and claim an initiator ID. Leaves interrupts DISABLED: the
; player polls, because the ring producer is already a polling loop living in
; the pace wait (ring.i) and an interrupt would buy it nothing it does not
; already have.
scsi_init:
bsr dmac_quiet
move.b #SCTL_RESET,SC_SCTL ; reset & disable
moveq #40,d0
sci1: nop
dbra d0,sci1
move.b #SCSI_ID,SC_BDID
move.b #0,SC_SCTL ; out of reset; no arbitration, no ints
move.b #$FF,SC_INTS ; INTS is cleared by writing its bits
move.b #0,SC_PCTL
clr.l SC_ERR.l
clr.l DM_USE.l ; PIO unless a caller asks otherwise,
; AFTER this call (src/player/dma.i)
clr.l DM_HOOK.l ; ...and no second consumer is being
; serviced until one installs itself,
; for the same reason and in the same
; direction: a mailbox that defaults to
; a behaviour is how DM_USE landed on
; ring.i's slot.
clr.l SC_TRN.l
move.l #0,SC_TAG.l
bsr sc_snap
rts
; ---------------------------------------------------------------- sc_settc
; d0 = 24-bit transfer count -> TCH/TCM/TCL
; Written LOW BYTE FIRST with lsr, not as a chain of rol.l #8. The rol version
; was the bring-up's second bug: three rotations put the ORIGINAL bits 31..24,
; 23..16 and 15..8 into TCH/TCM/TCL, so a count of 10 loaded a transfer counter
; of ZERO. MAME then completed the TRANSFER instantly and silently -- SSTS came
; back $85, TC0 set and XFER_IN_PROGRESS clear -- and the bus sat in command
; phase, which surfaced as the same `UNEXPECTED PHASE` as a protocol error.
sc_settc:
move.l d0,-(sp)
move.b d0,SC_TCL ; bits 7..0
lsr.l #8,d0
move.b d0,SC_TCM ; bits 15..8
lsr.l #8,d0
move.b d0,SC_TCH ; bits 23..16
move.l (sp)+,d0
rts
; ---------------------------------------------------------------- sc_waitreq
; Wait until the SPC reports a REQ with a phase, or patience runs out.
; Returns the phase in d0; sets SC_ERR and returns -1 on timeout.
sc_waitreq:
move.l #SC_PATIENCE,d1
swr1: move.b SC_PSNS,d0
btst #7,d0 ; REQ
bne.s swr2
subq.l #1,d1
bne.s swr1
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
swr2: and.l #7,d0
rts
; ---------------------------------------------------------------- sc_waitfree
; Wait for BUS FREE. A command is not over when its last message byte has been
; read: the target still has BSY asserted, and an initiator that starts
; arbitrating into that gets a selection timeout.
;
; This is the bring-up's fifth bug, and it only appeared once there were TWO
; reads. One read passed, byte-exact, and every conclusion drawn from it was
; sound; the SECOND could not select, because nothing had waited for the first
; to let go of the bus. A player issues one of these per record, so the failure
; would have been universal in the ring and invisible in the demonstration.
sc_waitfree:
move.l #SC_PATIENCE,d1
swf1: move.b SC_PSNS,d0
btst #3,d0 ; BSY
beq.s swf2
subq.l #1,d1
bne.s swf1
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
swf2: moveq #0,d0
rts
; ---------------------------------------------------------------- sc_select
; Select SCSI_TGT. The selection bitmask goes in TEMP -- both IDs, ours and
; theirs -- and the transfer counter doubles as the selection timeout (MAME:
; SelectionWaitBSY is derived from TC's upper bits, which is the datasheet's
; behaviour too).
sc_select:
move.b #$FF,SC_INTS
move.b #(1<<SCSI_ID)|(1<<SCSI_TGT),SC_TEMP
move.l #$002000,d0
bsr sc_settc
move.b #0,SC_PCTL
move.b #SCMD_SELECT,SC_SCMD
move.l #SC_PATIENCE,d1
ssel1: move.b SC_INTS,d0
btst #4,d0 ; COMMAND COMPLETE = selection won
bne.s ssel_ok
btst #2,d0 ; TIMEOUT = nobody there
bne.s ssel_tmo
subq.l #1,d1
bne.s ssel1
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
ssel_tmo:
move.b #$FF,SC_INTS
move.l #SCE_SELTMO,SC_ERR.l
moveq #-1,d0
rts
ssel_ok:
move.l #1,SC_TAG.l
bsr sc_snap
move.b #$FF,SC_INTS
move.l #2,SC_TAG.l
bsr sc_snap
moveq #0,d0
rts
; ---------------------------------------------------------------- sc_xferend
; Wait for the SPC to finish the TRANSFER it was given, rather than for the last
; byte to have been HANDED to it.
;
; This is the bring-up's one real bug and it is worth recording. Without it,
; sc_out_pio wrote all ten command bytes and returned, the caller immediately
; asked what phase the bus was in, and the answer was STILL COMMAND -- because
; the SPC had the last byte in its FIFO and had not yet run the REQ/ACK for it.
; The symptom was `UNEXPECTED PHASE, phase=2` at the DATA-IN check, which reads
; like a target refusing the command and is nothing of the kind. A byte handed
; to a FIFO is not a byte on the bus.
sc_xferend:
move.l #SC_PATIENCE,d3
sxe1: move.b SC_SSTS,d0
btst #4,d0 ; XFER IN PROGRESS
beq.s sxe2
subq.l #1,d3
bne.s sxe1
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
sxe2: moveq #0,d0
rts
; ---------------------------------------------------------------- sc_out_pio
; Send d1 bytes from (a0) in phase d2. Command blocks and nothing else, so it
; is the small, simple one.
sc_out_pio:
move.b d2,SC_PCTL
move.l d1,d0
bsr sc_settc
move.b #SCMD_XFER,SC_SCMD
move.l #4,SC_TAG.l ; 4 = TRANSFER issued for an OUT phase
bsr sc_snap
sop1: move.l #SC_PATIENCE,d3
sop2: move.b SC_SSTS,d0
btst #1,d0 ; DREG FULL -- wait for room
beq.s sop3
subq.l #1,d3
bne.s sop2
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
sop3: move.b (a0)+,SC_DREG
subq.l #1,d1
bne.s sop1
move.l #5,SC_TAG.l ; 5 = every byte handed to the FIFO
bsr sc_snap
bsr sc_xferend
move.l d0,-(sp)
move.l #6,SC_TAG.l ; 6 = after waiting for the transfer
bsr sc_snap
move.l (sp)+,d0
rts
; ---------------------------------------------------------------- sc_in_pio
; Receive d1 bytes into (a1) in phase d2. This is the path the DMA version
; replaces; it stays because it is what makes a DMA failure diagnosable.
sc_in_pio:
move.b d2,SC_PCTL
move.l d1,d0
bsr sc_settc
move.b #SCMD_XFER,SC_SCMD
move.l #7,SC_TAG.l ; 7 = TRANSFER issued for an IN phase
bsr sc_snap
sip1: move.l #SC_PATIENCE,d3
sip2: move.b SC_SSTS,d0
btst #0,d0 ; DREG EMPTY -- wait for a byte
beq.s sip3
subq.l #1,d3
bne.s sip2
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
sip3: move.b SC_DREG,(a1)+
subq.l #1,d1
bne.s sip1
move.l #8,SC_TAG.l ; 8 = every byte taken from the FIFO
bsr sc_snap
bsr sc_xferend
move.l d0,-(sp)
move.l #9,SC_TAG.l ; 9 = after waiting for the IN transfer
bsr sc_snap
move.l (sp)+,d0
rts
; ---------------------------------------------------------------- sc_in_data
; Receive d1 bytes in phase d2, storing only the WINDOW: drop SC_WSKIP, store
; SC_WKEEP at (a1), drop whatever is left. This is the DATA IN path; STATUS and
; MESSAGE IN keep sc_in_pio, which is one byte and has no window.
;
; THREE LOOPS, NOT ONE STEERED LOOP. A single loop with a `which third am I in`
; test per byte would cost ~20 clocks on every byte of every record, and the
; number this rig exists to produce is the transport's per-byte cost -- so the
; middle loop is byte-for-byte sc_in_pio's and the window is paid for once at
; each boundary instead of once per byte.
;
; A SPLIT DATA PHASE WOULD RE-SKIP. The counters are re-read from memory on
; every entry, so a target that broke one READ(10) across two DATA IN phases
; would drop the head of the second phase as well. This one does not split --
; the same limitation sc_in_pio's caller already carries -- and the fix is the
; same one: d5 has to become what each phase actually delivered.
sc_in_data:
; ---- P4a: the DATA IN phase can be handed to the HD63450 instead, and
; when it is, the CPU touches none of these bytes. src/player/dma.i.
; The window is REFUSED rather than ignored: a channel writes a
; contiguous run and cannot be told to drop the first 300 bytes, so a
; windowed DMA read would deliver the neighbours' bytes into the ring
; and the block loop has no bounds check to catch it (49.2, 58.3).
; Refusing it here is what makes "sector-aligned records" a PRECONDITION
; the transport states rather than an assumption it carries.
tst.l DM_USE.l
beq.s .pio
tst.l SC_WSKIP.l
bne.s .nowin
move.l SC_WKEEP.l,d0
cmp.l d1,d0
bne.s .nowin
bra sc_in_dma
.nowin: move.l #SCE_WINDOW,SC_ERR.l
moveq #-1,d0
rts
.pio:
movem.l d6-d7,-(sp)
move.b d2,SC_PCTL
move.l d1,d0
bsr sc_settc
move.b #SCMD_XFER,SC_SCMD
move.l #7,SC_TAG.l ; 7 = TRANSFER issued for an IN phase
bsr sc_snap
move.l SC_WSKIP.l,d6
move.l SC_WKEEP.l,d7
sub.l d6,d1
sub.l d7,d1 ; d1 = trailing bytes to drop
tst.l d6
beq.s .keep
.drop1: move.l #SC_PATIENCE,d3
.dw1: move.b SC_SSTS,d0
btst #0,d0 ; DREG EMPTY -- wait for a byte
beq.s .dg1
subq.l #1,d3
bne.s .dw1
bra .tmo
.dg1: tst.b SC_DREG ; popped and thrown away
subq.l #1,d6
bne.s .drop1
.keep: tst.l d7
beq.s .tail
.keep1: move.l #SC_PATIENCE,d3
.kw1: move.b SC_SSTS,d0
btst #0,d0
beq.s .kg1
subq.l #1,d3
bne.s .kw1
bra .tmo
.kg1: move.b SC_DREG,(a1)+
subq.l #1,d7
bne.s .keep1
.tail: tst.l d1
beq.s .fin
.tail1: move.l #SC_PATIENCE,d3
.tw1: move.b SC_SSTS,d0
btst #0,d0
beq.s .tg1
subq.l #1,d3
bne.s .tw1
bra.s .tmo
.tg1: tst.b SC_DREG
subq.l #1,d1
bne.s .tail1
.fin: movem.l (sp)+,d6-d7
move.l #8,SC_TAG.l ; 8 = every byte taken from the FIFO
bsr sc_snap
bsr sc_xferend
move.l d0,-(sp)
move.l #9,SC_TAG.l
bsr sc_snap
move.l (sp)+,d0
rts
.tmo: movem.l (sp)+,d6-d7
move.l #SCE_TIMEOUT,SC_ERR.l
moveq #-1,d0
rts
; ---------------------------------------------------------------- scsi_read
; READ(10) of d4 blocks from LBA d3 into (a1). READ(10) rather than READ(6)
; because a 21-bit LBA and a 256-block ceiling are limits this container will
; reach -- 4,488,588 B of frame records is already 8,767 sectors, and a full
; disc is 1.09 GiB (ROADMAP C3).
;
; DRIVEN BY THE PHASE, NOT BY A SCRIPT, and that is the third thing the bring-up
; taught. The first version ran a fixed sequence -- select, command, data,
; status, message -- and broke the moment the target asked for something else:
; it came up in MESSAGE OUT with ATN asserted and the driver, which "knew" the
; next phase was COMMAND, called it an unexpected phase and gave up. The bus
; decides the order. A driver that reads the phase and services whatever it
; finds is both shorter and correct, and it is what the target is entitled to.
scsi_read:
movem.l d3-d5/a1,-(sp)
move.l d4,d5
lsl.l #8,d5
lsl.l #1,d5 ; blocks * 512
clr.l SC_WSKIP.l ; no window: keep the whole transfer
move.l d5,SC_WKEEP.l
bra.s scr_body
; ---- the same read, delivering only SC_WSKIP..SC_WSKIP+SC_WKEEP of it. The
; caller sets the two words; everything else is identical, which is the point --
; a windowed read and a whole one must not be able to differ in the protocol.
scsi_read_win:
movem.l d3-d5/a1,-(sp)
move.l d4,d5
lsl.l #8,d5
lsl.l #1,d5
scr_body:
; ---- the command block, built before anything is on the bus
lea SC_CDB.l,a0
move.b #$28,(a0)+ ; READ(10)
clr.b (a0)+
move.l d3,d0 ; LBA, big-endian u32
rol.l #8,d0
move.b d0,(a0)+ ; 31..24
rol.l #8,d0
move.b d0,(a0)+ ; 23..16
rol.l #8,d0
move.b d0,(a0)+ ; 15..8
rol.l #8,d0
move.b d0,(a0)+ ; 7..0
clr.b (a0)+
move.l d4,d0 ; block count, big-endian u16. Same
lsr.l #8,d0 ; trap as sc_settc had: a rol chain here
move.b d0,(a0)+ ; would have emitted bits 31..24/23..16
move.b d4,(a0)+ ; of a count that lives in 15..0.
clr.b (a0)+
clr.l SC_STAT.l
bsr sc_select
tst.l d0
bmi scr_out
; Drop ATN. We have no message to send, so asking the target not to ask
; for one is cheaper than answering. The MSGOUT arm below still exists,
; because "cheaper" is not "guaranteed".
move.b #SCMD_RSTATN,SC_SCMD
; ---- service whatever the bus asks for, until the target ends the command
scr_ph:
bsr sc_waitreq
tst.l d0
bmi scr_out
move.l d0,-(sp)
move.l #3,SC_TAG.l ; 3 = the phase loop saw a REQ
bsr sc_snap
move.l (sp)+,d0
cmp.l #PH_CMD,d0
beq.s scr_cmd
cmp.l #PH_DATAIN,d0
beq.s scr_din
cmp.l #PH_STATUS,d0
beq.s scr_st
cmp.l #PH_MSGIN,d0
beq.s scr_min
cmp.l #PH_MSGOUT,d0
beq.s scr_mout
bra scr_phase
scr_cmd:
lea SC_CDB.l,a0
moveq #10,d1
moveq #PH_CMD,d2
bsr sc_out_pio
tst.l d0
bmi scr_out
bra scr_ph
scr_din:
; KNOWN LIMITATION, harmless here and not harmless forever: this asks
; for the WHOLE remaining count every time the bus enters DATA IN. A
; target that split one READ(10) across two data phases would be served
; the full length twice and overrun the caller's buffer. This one does
; not split -- 4,096 B and 2,048 B both arrive in a single phase -- but
; a real drive may, and P4b's mailbox integration is where d5 has to
; start being decremented by what each phase actually delivered.
move.l 12(sp),a1 ; the caller's destination. movem.l
; d3-d5/a1,-(sp) lays them out ASCENDING
; from sp as d3,d4,d5,a1 -- a1 is at 12.
move.l d5,d1
moveq #PH_DATAIN,d2
bsr sc_in_data
tst.l d0
bmi scr_out
bra scr_ph
scr_st:
lea SC_STAT.l,a1
addq.l #3,a1 ; the byte lands in the u32's low end
moveq #1,d1
moveq #PH_STATUS,d2
bsr sc_in_pio
tst.l d0
bmi scr_out
bra scr_ph
scr_min:
lea SC_MSG.l,a1
moveq #1,d1
moveq #PH_MSGIN,d2
bsr sc_in_pio
tst.l d0
bmi scr_out
; A message in ends the command. Anything non-zero in the status byte
; is the target refusing, and a transport that ignored it would hand the
; ring a buffer of stale bytes and call it a record.
move.l SC_STAT.l,d0
beq.s scr_ok
move.l #SCE_STATUS,SC_ERR.l
moveq #-1,d0
bra.s scr_out
scr_mout:
; Nothing to say: IDENTIFY, no disconnect, LUN 0.
lea SC_MSG.l,a1
move.b #$80,(a1)
move.l a1,a0
moveq #1,d1
moveq #PH_MSGOUT,d2
bsr sc_out_pio
tst.l d0
bmi scr_out
bra scr_ph
scr_ok: ; END OF COMMAND, and it takes two steps rather than one. After the
; final message byte the SPC is still holding ACK -- PSNS reads $4F,
; REQ low and ACK high -- and a target cannot drop BSY into that. So
; ACK is dropped explicitly, and only then is the bus released.
move.b #SCMD_RSTACK,SC_SCMD
move.b #SCMD_RELEASE,SC_SCMD
move.l #10,SC_TAG.l ; 10 = after the bus release command
bsr sc_snap
bsr sc_waitfree ; leave the bus as we found it
tst.l d0
bmi scr_out
moveq #0,d0
scr_out:
movem.l (sp)+,d3-d5/a1
rts
scr_phase:
move.l d0,SC_PH.l
move.l #SCE_PHASE,SC_ERR.l
moveq #-1,d0
bra.s scr_out
; ---- the DMAC side of the data phase, ROADMAP P4a. Included unconditionally
; so that there is ONE transport file: sc_in_data dispatches on DM_USE, which
; scsi_init clears, so a front-end that never sets it assembles the same PIO
; path FINDINGS 58 measured and executes not one instruction of the following.
include "src/player/dma.i"
src/player/scsigate.s
+150
View File
@@ -0,0 +1,150 @@
; Front-end for the MB89352 PROBE (ROADMAP P4, first step), for the rig.
;
; WHY A PROBE AND NOT A DRIVER. P4 replaces tools/bench/stream.lua's modelled
; transport with a real SPC behind the XF_* mailbox src/player/ring.i already
; talks to. Before any of that can be written, the register map has to be a
; FACT on the emulated machine rather than a reading of somebody's datasheet.
; FINDINGS 32.4 quotes MAME mapping the data register at $EA0015 -- register
; index 10 at a stride of 2 from $EA0001, i.e. registers on the ODD bytes. That
; is an inference from ONE address, and every access the driver makes rests on
; it.
;
; WHY IT ENUMERATES INSTEAD OF DUMPING. The first version of this walked
; $EA0000 upwards with a plain `move.b (a0)+`, and took a bus error at $EA0006 --
; at which point it knew one address was dead and nothing about the other 57.
; A sequential dump stops at the first hole and reports the hole as the answer.
; So this probes ONE ADDRESS AT A TIME with the index in MEMORY, and a bus error
; handler that records the fault, steps the index and re-enters the loop. A
; dead address costs an entry in the map rather than the rest of the run.
;
; The 68000 cannot resume a faulted instruction -- RTE re-runs it and faults
; again -- so the handler does not try. It restores a stack pointer saved
; before the loop and jumps back to the loop head, which reloads everything it
; needs from memory. Nothing lives in a register across a fault.
;
; This gate drives no SCSI bus and moves no data. It is the smallest thing that
; can turn "MAME instantiates an MB89352" into "the 68000 can reach it, HERE".
SCFLAG = $18080 ; 0 idle / 1 done
SCN = 64 ; addresses probed, from SPCBASE up
SCIDX = $18084 ; u32 probe index, lives in memory across faults
SCSAVSP = $18088 ; u32 stack pointer saved before the loop
SCVAL = $18100 ; SCN bytes: what each address read
SCOK = $18140 ; SCN bytes: 1 = answered, 0 = bus error
SCTMP = $180D0 ; u32: TEMP writeback -- $A5 written, read back
SCTMPOK = $180D4 ; u32: 1 = the writeback completed without fault
SCRD = $180D8 ; u32: scsi_read's return, 0 = the read worked
SCDRG = $180DC ; u32: $5A written to DREG then read straight back
SCDRS = $180E0 ; u32: SSTS between that write and that read
SCDST = $20000 ; where the read lands
SCBLKS = 8 ; 8 x 512 B, enough to cross a sector boundary
SCRD2 = $180E4 ; u32: the second read's return
SCER1 = $180E8 ; u32: SC_ERR as it stood after the first read
SCER2 = $180EC ; u32: ...and after the second
SCDST2 = $28000 ; where the second read lands
SCLBA2 = 1000 ; a NON-ZERO LBA: block 0 would pass even if the
; LBA bytes of the command block were ignored
SCBLK2 = 4
SPCBASE = $EA0000
org $10000
start:
move.l #buserr,$8.w ; vector 2
clr.l SCFLAG.l
clr.l SCTMP.l
clr.l SCTMPOK.l
clr.l SCIDX.l
move.l sp,SCSAVSP.l
; ---- probe SCN addresses, one at a time, surviving each fault
ploop:
move.l SCIDX.l,d0
cmp.l #SCN,d0
bge.s pdone
lea SPCBASE,a0
adda.l d0,a0
lea SCVAL.l,a1
lea SCOK.l,a2
move.b #1,0(a2,d0.l) ; assume it answers; the handler undoes
move.b (a0),d1 ; <- the access under test
move.b d1,0(a1,d0.l)
addq.l #1,SCIDX.l
bra.s ploop
pdone:
; ---- TEMP (register 11 on the believed map, $EA0017) is a scratch latch on a
; real MB89352. Writing a pattern and reading it back separates "these odd
; bytes are registers" from "these odd bytes are a mirror of something".
; Guarded the same way: if it faults, the handler lands in ploop with SCIDX
; already past the end, falls through here again, and SCTMPOK stays 0.
move.b #$A5,SPCBASE+23
moveq #0,d0
move.b SPCBASE+23,d0
move.l d0,SCTMP.l
move.l #1,SCTMPOK.l
; ---- DOES A WRITE TO THE DATA REGISTER REACH THE CHIP AT ALL?
; $EA0015 is the one address x68k_scsiext.cpp puts its own glue on, and that
; glue DROPS a write when the DMAC's OWN is asserted and DRQ is low. A dropped
; write is invisible: no error, no status bit, nothing. So it is tested
; directly, before any SCSI protocol can be blamed for it. dreg_w enqueues into
; the FIFO, so DREG_EMPTY must fall between the write and the read, and the read
; must give the byte back.
bsr scsi_init
move.b #$5A,SPCBASE+21
moveq #0,d0
move.b SPCBASE+13,d0 ; SSTS: is the FIFO still empty?
move.l d0,SCDRS.l
moveq #0,d0
move.b SPCBASE+21,d0
move.l d0,SCDRG.l
; ---- the SPC is reachable; now make it fetch something. A read of the first
; SCBLKS sectors, in PIO, verified BY THE HOST against the same bytes in
; tmp/dlxdisk.img. That is the whole of P4's correctness half in one line: the
; player's own code selected a target, issued a READ(10) and got the disc's
; bytes back, with no IOCS and no host in the path.
bsr scsi_init
moveq #0,d3 ; LBA 0
moveq #SCBLKS,d4
lea SCDST,a1
bsr scsi_read
move.l d0,SCRD.l
move.l SC_ERR.l,SCER1.l ; SC_ERR is the LAST error, so it is
; captured per read: reading it once at
; the end reported the second read's
; failure against the first read's name.
; ---- and again, somewhere else on the disc. A read of LBA 0 is passed by a
; driver that emits a malformed LBA field, because zero is what a malformed
; field usually is. This one is not.
move.l #SCLBA2,d3
moveq #SCBLK2,d4
lea SCDST2,a1
bsr scsi_read
move.l d0,SCRD2.l
move.l SC_ERR.l,SCER2.l
move.l #1,SCFLAG.l
hold: bra.s hold
; ---- bus error. Mark the address dead, step past it, re-enter the loop with a
; stack pointer that is known good. The stacked frame is abandoned deliberately:
; there is nothing in it worth more than the next 57 addresses.
buserr:
move.l SCSAVSP.l,sp
move.l SCIDX.l,d0
cmp.l #SCN,d0
bge.s btmp
lea SCOK.l,a2
clr.b 0(a2,d0.l)
lea SCVAL.l,a1
move.b #$FF,0(a1,d0.l)
addq.l #1,SCIDX.l
jmp ploop
btmp: ; the fault was the TEMP writeback
move.l #1,SCFLAG.l
be: bra.s be
include "src/player/scsi.i"
src/player/stream.s
+169 -5
View File
@@ -81,7 +81,22 @@ PACE = $18034 ; producer -> decoder: frame ticks elapsed since
; release. Frame i may not START before tick i.
PACEON = $18038 ; 1 = obey PACE. 0 leaves the loop free-running,
; byte for byte the loop FINDINGS 49 measured.
LATEFR = $18080 ; frames that reached the pace gate with their
; tick ALREADY ARRIVED, i.e. did not idle for a
; single poll -- the previous frame used its
; whole slot. See the gate below.
LATEMAX = $18084 ; the worst of those, in WHOLE ticks overrun
LATE1ST = $18088 ; index of the FIRST such frame, so that a
; count can be told apart from a start-up
; transient without re-running anything
CLKON = $1803C ; 1 = the 68000 paces ITSELF: src/player/clock.i
; drives PACE off the CRTC's V-DISP instead of
; the host writing it. Needs PACEON=1; the gate
; below cannot tell the two apart and must not.
DESC = $18100 ; DESCN x u32, record base addresses
; The producer's own words -- RINGOWN, the transport mailbox and its
; instruments -- are in src/player/ring.i, at $18300 and up, clear of DESC's
; 256 bytes.
DESCN = 64 ; power of two; the index is masked, not compared
DESCM = (DESCN-1)*4 ; mask for a BYTE offset into DESC
@@ -92,13 +107,76 @@ SPINMAX = 2000000 ; polls with no progress before giving up
org $10000
start:
; ---- the ring producer, if this run is asking the 68000 to fill its own ring
; (ROADMAP P5, src/player/ring.i). It goes FIRST because it only builds tables
; and touches no hardware: a run that cannot build them should not have armed an
; interrupt source first.
tst.l RINGOWN.l
beq.s noring
; ---- the transport, before the producer that will ask it for something.
; ring_init ends in a ring_seek and a seek WAITS for the channel to go quiet, so
; the thing that makes the channel quiet has to exist first. With XF_SCSI = 0
; this brings up nothing and the host is the transport (FINDINGS 55).
bsr xf_init
bsr ring_init
tst.l d0
bpl.s noring
move.l #$E3,FLAG.l ; the record index is longer than ROFF
bra hold
noring:
; ---- the frame clock, if this run is asking the 68000 to keep its own time.
; It goes here rather than inside the frame loop because clk_init CLEARS PACE:
; tick 0 has to be the instant the decoder was released, exactly as it is when
; the host writes PACE, or the first frame's deadline moves.
tst.l CLKON.l
beq.s noclk
bsr clk_init
tst.l CLK_ERR.l
beq.s noclk
move.l #$E2,FLAG.l ; the clock refused; CLK_ERR says why
bra hold
noclk:
move.l #1,FLAG.l ; timer starts here
outer:
move.l NFR.l,SCR_N.l
clr.l FR_TAIL.l
clr.l STALLS.l
clr.l SPINS.l
clr.l LATEFR.l
clr.l LATEMAX.l
move.l #-1,LATE1ST.l
; ---- SEEK AND PREFILL. Every pass starts with a seek to record 0 -- which on
; the first pass is just "start of scene" and on any later one is a REAL seek:
; the channel has to go quiet, the ring is declared empty, and the whole
; lookahead 51.3 says takes seconds of play to accumulate is thrown away and
; rebuilt from the prefill up. That is the branch point rehearsed with the one
; thing a rig can check afterwards -- the decode has to still be pixel-exact.
;
; The clock is REBASED here rather than at clk_init, because tick 0 must be the
; instant the decoder is released and the prefill happens before that. Under a
; host-written PACE this word belongs to the host, so it is only touched when
; the 68000 is keeping its own time; ITER>1 therefore needs CLKON.
tst.l RINGOWN.l
beq.s noseek
moveq #0,d0
bsr ring_seek
bsr ring_prefill
tst.l d0
bpl.s .pfok
move.l #$E1,FLAG.l ; the transport never delivered
bra hold
.pfok:
tst.l CLKON.l
beq.s noseek
clr.l PACE.l ; at most one tick is lost to a V-DISP
; landing between the ISR and here
noseek:
frameloop:
tst.l RINGOWN.l
beq.s nomark
move.l FR_TAIL.l,d0
bsr ring_mark
nomark:
; ---- PACE GATE (FINDINGS 49.7.2, and it is the whole point of this session).
; Free-running, this loop asks for record i the instant it finishes record i-1,
; so it outruns any finite pipe, the ring NEVER backs up, and the producer's
@@ -117,8 +195,63 @@ frameloop:
;
; It also makes STALLS mean something. Free-running, a stall is EARLINESS
; (49.6); paced, a frame that has to wait for its record is a real underrun.
;
; AND IT COUNTS THE FRAMES THAT WERE ALREADY LATE, which is a question only a
; REAL frame clock raises. 12 fps on a 55.4577 Hz raster is 4.6215 refreshes
; per frame, so the divider hands out slots of 4 refreshes (72.13 ms) and 5
; (90.16 ms), 37.9% of them short -- and the SHORT one is 13.4% under the
; 83.33 ms every budget in this project is priced against (FINDINGS 54). A
; frame that does not fit its slot does not fail here: PACE has already moved
; on, so the next frame starts the instant this one finishes and the clock
; catches up by itself. What it costs is one late PRESENT, and nothing in this
; tree counted those because until now the tick was a host model with no
; cadence in it at all.
;
; The test is "did this frame have to WAIT", not "is it a whole tick behind".
; Frame i waits while PACE < i; so PACE >= i on arrival means the decoder came
; to the gate with slot i already open and idled for zero polls, which is the
; same statement as "frame i-1 ran to the end of its slot". A whole tick of
; overrun -- PACE - FR_TAIL >= 1 -- is the much rarer case where it ran past
; the end of the NEXT one, and is reported separately as the worst seen.
;
; Frame 0 is excluded: it starts at tick 0 by definition and has no predecessor
; to have overrun. The wait loop below is untouched -- all of this is ahead of
; it, and the free-running path executes none of it.
tst.l PACEON.l
beq.s nopace
move.l PACE.l,d0
cmp.l FR_TAIL.l,d0 ; PACE < FR_TAIL: the slot has not come
bcs.s pacesel ; round yet, so this frame is EARLY
tst.l FR_TAIL.l
beq.s pacesel ; frame 0 starts AT tick 0 by definition
tst.l LATEFR.l
bne.s .nf1
move.l FR_TAIL.l,LATE1ST.l
.nf1:
addq.l #1,LATEFR.l
sub.l FR_TAIL.l,d0 ; whole ticks overrun; 0 = inside the
cmp.l LATEMAX.l,d0 ; slot but with nothing left of it
bls.s pacewait
move.l d0,LATEMAX.l
pacesel:
; Both branches above -- the early frame and frame 0 -- come here rather than
; jumping straight into the legacy wait, because the early frame is the COMMON
; case and it is the one with idle in it. Routing it past this test was a real
; bug and not a tidy-up: the producer then only ever ran from the record wait,
; about once a frame, and the disc spent most of the scene stopped.
tst.l RINGOWN.l
beq.s pacewait
; ---- THE IDLE IS WHERE THE DISC RUNS. ring_poll retires the completed
; request and issues the next one, and this loop is the only place in a paced
; player with time to spare. Polling once a FRAME instead would cap the fill at
; one record per slot -- the wire rate exactly -- and a ring that can only keep
; up never accumulates the slack a branch point spends (51.3).
pacewaitR:
bsr ring_poll
move.l PACE.l,d0
cmp.l FR_TAIL.l,d0
bcs.s pacewaitR
bra.s nopace
pacewait:
move.l PACE.l,d0
cmp.l FR_TAIL.l,d0 ; d0 - FR_TAIL; carry = tick not reached
@@ -128,6 +261,22 @@ nopace:
; d1 counts polls for this frame; a nonzero d1 on exit means the frame stalled.
moveq #0,d1
move.l FR_TAIL.l,d2
tst.l RINGOWN.l
beq.s waitrec
; ---- the same wait, with the producer inside it. Here d1 counts POLLS rather
; than spins, so STALLS still means "frames that had to wait" and SPINS is not
; comparable with a host-filled run's. A frame that waits here is a real
; underrun either way: paced, its slot has already opened.
waitrecR:
bsr ring_poll
move.l FR_HEAD.l,d0
cmp.l d2,d0
bhi gotrec
addq.l #1,d1
cmp.l #SPINMAX,d1
bcs.s waitrecR
move.l #$E1,FLAG.l
bra hold
waitrec:
move.l FR_HEAD.l,d0
cmp.l d2,d0
@@ -163,12 +312,16 @@ nostall:
cmpa.l SCR_END.l,a0 ; bitstream desync is silent otherwise
bne desync
; ---- release. Round up to 4 the same way decode.s does: the producer lays
; records on 4-byte boundaries, so the byte one past this record's padded
; end is the first byte the producer may reuse.
; ---- release. Round up to RECALN: the producer lays records on the
; container's own record boundaries (geom.i), so the byte one past this
; record's PADDED end is the first byte it may reuse. Releasing only the
; bytes actually read would strand up to RECALN-1 of pad per record and
; the producer's free-space arithmetic would drift by that much a frame.
; This is an absolute address, so it is only the record's padded end
; because the ring base is RECALN-aligned as well.
move.l a0,d0
addq.l #3,d0
and.b #$FC,d0
addi.l #RECALN-1,d0
andi.l #~(RECALN-1),d0
move.l d0,RD_PTR.l
addq.l #1,FR_TAIL.l
@@ -176,9 +329,20 @@ nostall:
bne frameloop
subq.l #1,ITER.l
bne outer
; ---- leave the MFP as it was found. A rig that exits with a live interrupt
; source and a lowered mask hands the next thing that runs an interrupt it
; has no vector for, and the failure would land somewhere else entirely.
tst.l CLKON.l
beq.s noclk2
bsr clk_stop
noclk2:
move.l #$FF,FLAG.l ; timer stops here
hold: bra.s hold
desync: move.l #$EE,FLAG.l
bra.s hold
include "src/player/frame.i"
include "src/player/clock.i"
include "src/player/ring.i"
include "src/player/scsi.i"
include "src/player/xfer.i"
src/player/xfer.i
+148
View File
@@ -0,0 +1,148 @@
; ---------------------------------------------------------------- xfer.i
; THE TRANSPORT BEHIND THE MAILBOX. ROADMAP P4b.
;
; src/player/ring.i has always ended at a seam: it decides which record to
; fetch, where in the ring to put it and when that is safe, writes those four
; words into XF_SLOT and bumps XF_GO, and then polls XF_ACK. On the other side
; of that seam, until now, was tools/bench/stream.lua -- a host moving bytes at
; a MODELLED rate, with XF_ACK synthesised out of emulated time. A player has
; no host. This file is the other side: XF_GO is answered by src/player/scsi.i
; issuing a real READ(10) to a real MB89352, and XF_ACK is a word the 68000
; bumps when the bytes have landed.
;
; NOTHING ABOVE THE SEAM CHANGED, and that is deliberate for the same reason it
; was in sessions 22 and 23: ring.i cannot tell which transport answered it, so
; a green run here is a test of THIS file rather than of a new producer. The
; two hooks in ring.i (one in ring_poll, one in ring_seek's quiet-wait) are the
; whole of the change on that side, and with XF_SCSI = 0 they are a tst and a
; branch.
;
; IT IS SYNCHRONOUS, AND THAT IS NOT A SHORTCUT -- IT IS THE FINDING. The
; modelled transport overlapped: a request issued at time t landed at t + len/
; rate while the 68000 got on with decoding, which is what a DMAC channel does.
; Here the CPU moves every byte itself through $EA0015 (57.3: a PIO write to
; that address is discarded, so even "PIO" runs the SPC in DMA mode with the CPU
; standing in for the channel), so `bsr xf_service` does not start a transfer,
; it PERFORMS one. A two-deep request queue therefore buys nothing at all: the
; second slot is filled and drained by the same instruction stream that would
; have been decoding. FINDINGS 55.3's whole result -- that a one-deep queue
; gives away 6.8% of the pipe -- is about a transport that runs in parallel with
; the CPU, and this one does not.
;
; So what this file is FOR is not to be the shipping transport. It is to make
; the shipping transport's cost measurable: the same 120 pixel-exact frames,
; delivered by the machine off a real volume, with the CPU cost of doing it
; charged where a rate model cannot hide it. P4a -- the HD63450 holding the bus
; -- is what makes the transfer overlap again, and until it exists this is the
; honest floor.
; ---- state. Above src/player/ring.i's instruments (last: SK_WAIT at $1837C)
; and below the disc-offset table at $19400.
XF_SCSI = $18380 ; 1 = the 68000 is the transport (input)
XS_LBA0 = $18384 ; LBA of byte 0 of the scene's frame stream
XS_NXFER = $18388 ; transfers completed
XS_NBYTE = $1838C ; record bytes delivered into the ring
XS_NWIRE = $18390 ; bytes actually read off the disc, sectors and
; all -- the two differ and 58.3 is why
XS_ERR = $18394 ; SC_ERR of the FIRST failure, 0 = none
XS_ERRAT = $18398 ; ...and the request index it failed on
; ---------------------------------------------------------------- xf_init
; Clears the instruments and brings the SPC up, if this run has one. Called
; before ring_init, because ring_init ends in a ring_seek and a seek waits on
; the transport.
xf_init:
clr.l XS_NXFER.l
clr.l XS_NBYTE.l
clr.l XS_NWIRE.l
clr.l XS_ERR.l
move.l #-1,XS_ERRAT.l
tst.l XF_SCSI.l
beq.s .out
bsr scsi_init
.out: rts
; ---------------------------------------------------------------- xf_service
; Answer at most ONE outstanding request, then return. Preserves every
; register: it is called from inside ring_poll, which is itself called from
; inside the decoder's wait loops and must be invisible to them.
;
; ONE PER CALL, not "drain the queue". ring_poll retires exactly one completed
; request per call as well, and a transport that answered both queued requests
; in one visit would hand the retire loop two acks it can only take one poll at
; a time -- which is legal, but it also means the decoder's wait loop would
; disappear for two record times instead of one. One per call keeps the two
; sides stepping at the same rate.
;
; A RECORD IS NOT A SECTOR, and this is where that is dealt with. ring.i asks
; for a byte offset and a length; the target answers in 512 B blocks. So the
; command covers the sectors the record lies in, and SC_WSKIP/SC_WKEEP tell
; src/player/scsi.i's DATA IN loop which of those bytes to store. The ones
; outside the window are pulled from the FIFO and dropped -- they are NOT
; written past the ends of the destination, because the bytes on either side of
; a record in the stream belong to records the decoder may still be reading and
; the block loop has no bounds check (49.2).
;
; XS_NWIRE counts what the disc actually moved and XS_NBYTE what the ring got.
; They are not the same number and the gap is a delivery cost, not an accounting
; detail: it is bytes on the wire that no frame contains.
xf_service:
tst.l XF_SCSI.l
beq.s .idle
move.l XF_ACK.l,d0
cmp.l XF_GO.l,d0
bcs.s .work ; XF_ACK < XF_GO: something outstanding
.idle: rts
.work:
movem.l d0-d7/a0-a2,-(sp)
move.l XF_ACK.l,d0
move.l d0,d1
and.l #XF_SLM,d1
lsl.l #4,d1 ; * XF_SLSZ
lea XF_SLOT.l,a0
adda.l d1,a0
move.l (a0),d1 ; disc byte offset within the stream
movea.l 4(a0),a1 ; destination in the ring
move.l 8(a0),d2 ; length
; ---- sector arithmetic
move.l d1,d3
and.l #511,d3 ; bytes of the first sector to drop
move.l d1,d4
lsr.l #8,d4
lsr.l #1,d4 ; sector index within the stream
add.l XS_LBA0.l,d4 ; ...and where the stream begins
move.l d3,d5
add.l d2,d5
addi.l #511,d5
lsr.l #8,d5
lsr.l #1,d5 ; sectors the record lies in
move.l d3,SC_WSKIP.l
move.l d2,SC_WKEEP.l
add.l d2,XS_NBYTE.l
move.l d5,d0
lsl.l #8,d0
lsl.l #1,d0
add.l d0,XS_NWIRE.l
move.l d4,d3 ; d3 = LBA
move.l d5,d4 ; d4 = blocks
bsr scsi_read_win
tst.l d0
bmi.s .err
addq.l #1,XS_NXFER.l
addq.l #1,XF_ACK.l ; LAST: the bytes are all in the ring
; before the request is called done
movem.l (sp)+,d0-d7/a0-a2
rts
; ---- a failed read is NOT acked. The record never becomes resident, the
; decoder spins out in waitrec and reports a stalled producer, and XS_ERR says
; which request failed and why. Acking a failed transfer would publish a
; descriptor for a buffer full of whatever was there before -- and the decoder
; would find a plausible-looking length word in it and desync somewhere else
; entirely.
.err:
tst.l XS_ERR.l
bne.s .err2
move.l SC_ERR.l,XS_ERR.l
move.l XF_ACK.l,XS_ERRAT.l
.err2: movem.l (sp)+,d0-d7/a0-a2
rts
tools/analysis/13_cpu_ratectl.py
+15 -3
View File
@@ -40,12 +40,24 @@ BUDGET = RC.FRAME_CYCLES
# H.build is ~55 s, nearly all k-means, and it does not depend on the profile:
# both ship k1=k4=256. One build, cached, serves every row of the table.
cache = a.cache or f"tmp/model_{os.path.basename(a.frames_dir.rstrip('/'))}.pkl"
# The build parameters are stored with the model and a mismatch rebuilds: the
# cache is keyed on the frames directory alone, and once H.build acquired an
# option (session 28's reserved black entry, 23.4) a stale pickle would quietly
# serve a model the shipping encoder no longer builds. Same guard as
# tools/analysis/16_span_roundtrip.py.
SIG = dict(k1=256, k4=256, iters=16, reserve_black=True)
m = None
if os.path.exists(cache):
m = pickle.load(open(cache, "rb"))
print(f"model from {cache}")
else:
if m.get("sig") != SIG:
print(f"{cache}: built with {m.get('sig')}, wanted {SIG} -- rebuilding")
m = None
else:
print(f"model from {cache}")
if m is None:
t = time.time()
m = H.build(a.frames_dir, k1=256, k4=256, iters=16)
m = H.build(a.frames_dir, **SIG)
m["sig"] = SIG
pickle.dump(m, open(cache, "wb"))
print(f"built model in {time.time()-t:.0f} s -> {cache}")
print(f"{a.frames_dir}: {len(m['idx'])} frames, {m['nb']} blocks, "
tools/analysis/15_bus_occupancy.py
+102 -8
View File
@@ -62,6 +62,11 @@ ap.add_argument("container", nargs="?", default="tmp/rc_fr_singe_scsi_cpufit.dlx
ap.add_argument("--csv", default="tmp/c68k_frames.csv",
help="per-frame output of tools/bench/c68k/run.sh")
ap.add_argument("--nframes", type=int, default=None)
ap.add_argument("--kbps", type=float, default=None,
help="delivery rate in KB/s. OPTIONAL and there is no default "
"(FINDINGS 50): supply it and the AUTO-REQUEST rows are "
"added, which are the only rows whose cost depends on how "
"long the record takes to arrive (59.3).")
a = ap.parse_args()
if not os.path.exists(a.container):
sys.exit(f"missing {a.container}")
@@ -162,7 +167,13 @@ print("THE OTHER TWO MASTERS -- what the DMAC takes out of the same frame\n")
FPS = d.fps
CPUHZ = 10e6 # stock X68000, MAME 0.277 x68k.cpp:1133
FRAME_CLK = CPUHZ / FPS
vid_bpf = sum(n + 4 for (_, n) in d.frames[:NF]) / NF # DLX2 record padding
# WHAT THE TRANSPORT MOVES, which is the PADDED record and not the payload.
# Under DLX2/3/4 the pad was 0..3 B and the distinction was noise; under DLX5 it
# is 0..511 B, and charging the payload would price the channel for bytes it
# does not carry while the disc carries them anyway. A budget that debits only
# the bytes a frame CONTAINS is the same incomplete accounting this project has
# been caught by before -- the pad is delivered, so the pad is charged.
vid_bpf = sum(d.record_lengths()[:NF]) / NF
aud_bpf = B.ADPCM_BYTES_PER_S / FPS
a_lo = aud_bpf * B.ADPCM_CLK_BYTE_BEST
a_hi = aud_bpf * B.ADPCM_CLK_BYTE_WORST
@@ -186,19 +197,102 @@ print(f" {'W (clk/byte)':<16}{'clk/frame':>12}{'% of frame':>12} "
f"{'CPU+audio+video':>18}")
for W, note in ((5.0, "single address, bus held (11_cpu_budget.py default)"),
(8.0, "FINDINGS 5's long-standing per-word ESTIMATE"),
(9.0, "DUAL address, bus held -- and the FLOOR of every "
"dual-address\n "
" configuration, auto-request included (59.3)"),
(12.0, "single address, arbitrated per byte"),
(16.0, "what the ROM programs for SASI (best case)"),
(19.0, "what the ROM programs for SASI (worst case)")):
(19.0, "what the ROM programs for SASI (worst case)"),
(87.28, "PIO -- MEASURED, FINDINGS 58.2, the CPU doing it itself")):
v = vid_bpf * W
tot_clk = (cpu_clk if cyc_t.any() else 0) + a_lo + v
print(f" {W:<16.0f}{v:>12,.0f}{100*v/FRAME_CLK:>11.1f}% "
print(f" {W:<16.6g}{v:>12,.0f}{100*v/FRAME_CLK:>11.1f}% "
f"{100*tot_clk/FRAME_CLK:>17.1f}% {note}")
print(f"\n (the last column adds the MEASURED mean decode and the BEST-CASE "
f"audio, so it is\n the optimistic end of every row. 100% is the frame "
f"deadline at {FPS:g} fps.)")
print(f"""
Audio is {100*a_lo/FRAME_CLK:.2f}%..{100*a_hi/FRAME_CLK:.2f}% of the frame and video is {vid_bpf*5/FRAME_CLK*100:.0f}%..{vid_bpf*19/FRAME_CLK*100:.0f}%. The unpriced audio
stream was never the risk P6 called it -- ON THE BUS. What the same reading of
the ROM found is that the DISK's per-byte cost has a worked example on this
machine, it is 16..19 clocks, and at that price this design does not fit at any
container size. W is the number to attack, and it is a PLAYER decision.""")
Audio is {100*a_lo/FRAME_CLK:.2f}%..{100*a_hi/FRAME_CLK:.2f}% of the frame and video is {vid_bpf*5/FRAME_CLK*100:.0f}%..{vid_bpf*19/FRAME_CLK*100:.0f}% over the ladder, against
{vid_bpf*87.28/FRAME_CLK*100:.0f}% for the PIO transport FINDINGS 58.2 measured. The unpriced audio stream
was never the risk P6 called it -- ON THE BUS.""")
# --- HEADROOM, AND THE FLOOR UNDER THE LADDER -----------------------------
# Added session 27. The sweep above answers "what does each W cost"; it never
# answered "what can this frame afford", and the two are not the same question.
# FINDINGS 59.2 is why it matters now: with no external request line the only
# configurations that can be run are dual-address, and a dual-address byte has
# a FLOOR -- one 4-clock read of the device plus one 5-clock write to memory,
# buscost.DMA_DUAL_BYTE_CLK. No GCR share and no delivery rate goes under it.
print("\n" + "=" * 72)
print("WHAT THE FRAME CAN AFFORD, AND THE FLOOR UNDER THE LADDER\n")
head_clk = FRAME_CLK - (cpu_clk if cyc_t.any() else 0) - a_lo
head_wb = head_clk / vid_bpf
print(f" headroom after the MEASURED decode and best-case audio: "
f"{head_clk:,.0f} clk = {100*head_clk/FRAME_CLK:.1f}%")
print(f" at {vid_bpf:,.0f} B a frame that is {head_wb:.2f} CLOCKS PER BYTE, and "
f"that is the number\n a transport has to come in under.\n")
floor = B.DMA_DUAL_BYTE_CLK
print(f" dual-address floor {floor} clk/B ({B.DMA_READ_CLK} read of the "
f"device + {B.DMA_WRITE_CLK} write to memory, Fig 4-25)")
print(f" single-address held {B.DMA_DISK_CLK_WORD_HELD} clk/B (one memory "
f"write; needs the device to ACK, i.e. a REQUEST LINE)")
if head_wb < floor:
print(f"""
SO DUAL ADDRESS DOES NOT FIT THIS CONTAINER AT {FPS:g} fps -- not at any
delivery rate and not at any GCR share, because {head_wb:.2f} < {floor}. A share
decides whether the channel sits AT the floor or above it; it cannot
go under it. That is FINDINGS 59.2's three bounds arriving in the
budget: the configurations this machine can run are exactly the ones
the frame cannot afford, and the one it can afford -- single address,
{B.DMA_DISK_CLK_WORD_HELD} clk/B, {100*vid_bpf*B.DMA_DISK_CLK_WORD_HELD/FRAME_CLK:.1f}% -- needs the request line ROADMAP B3 asks about.""")
for w, what in ((floor, "dual address"), (B.DMA_DISK_CLK_WORD_HELD, "single address")):
tgt = head_clk / w
print(f"\n TO FIT AT {w} clk/B ({what}) THIS CONTAINER MUST COME DOWN TO")
print(f" {tgt:,.0f} B a frame = {tgt*FPS/1024:,.0f} KB/s of payload "
f"(it is {vid_bpf:,.0f} B, {vid_bpf*FPS/1024:,.0f} KB/s)"
+ (" -- already met" if vid_bpf <= tgt else
f" -- {100*(vid_bpf/tgt-1):.0f}% too big"))
print(f"""
AND THAT IS THE PESSIMISTIC READING OF THE ENCODER LEVER: a lighter
container also DECODES cheaper, so the decode term above falls with
the byte term. The figure to re-derive it against is this tool run on
the lighter container -- with its OWN C68K measurement, because the
cross-check at the top is what licenses every number below it.""")
else:
print(f"\n The frame affords {head_wb:.2f} clk/B, which is at or above the "
f"{floor} clk/B dual-address floor.")
# --- AUTO-REQUEST, and only when a rate is supplied ------------------------
# These are the rows 59.3 added and they are the only ones here whose cost is
# not a property of the transfer: an auto-requested channel spends its share of
# the bus whether or not a byte is there, so what a record costs depends on how
# long it takes to ARRIVE. No default rate, deliberately (FINDINGS 50).
if a.kbps:
RATE = a.kbps * 1024.0
wire_clk = vid_bpf / RATE * CPUHZ
cap = 0.5 * CPUHZ / B.DMA_DUAL_BYTE_CLK # the 50% share's ceiling
print("\n" + "=" * 72)
print(f"AUTO-REQUEST AT {a.kbps:g} KB/s -- charged by TIME, not by byte "
f"(59.3)\n")
print(f" the record takes {wire_clk:,.0f} clk to arrive = "
f"{100*wire_clk/FRAME_CLK:.1f}% of a frame\n")
print(f" {'configuration':<34}{'clk/B':>8}{'% of frame':>12}"
f"{'CPU+audio+video':>18}")
rows = [("REQG 01, max rate (100% of the bus)", 1.0, None)]
for br, share in ((0, .5), (1, .25), (2, .125), (3, .0625)):
rows.append((f"REQG 00, LRAR BR={br:02b}, {share*100:g}% share", share,
share * CPUHZ / B.DMA_DUAL_BYTE_CLK))
for name, share, sustains in rows:
v = share * wire_clk
tot = (cpu_clk if cyc_t.any() else 0) + a_lo + v
flag = ""
if sustains is not None and sustains < RATE:
flag = f" cannot carry the rate ({sustains/1024:.0f} KB/s max)"
print(f" {name:<34}{v/vid_bpf:>8.2f}{100*v/FRAME_CLK:>11.1f}%"
f"{100*tot/FRAME_CLK:>17.1f}%{flag}")
print(f"""
A FASTER DISC MAKES AUTO-REQUEST CHEAPER, which no W does -- the share is
spent over a shorter wire time. But it cannot reach the floor: a 50% share
tops out at {cap/1024:,.0f} KB/s, above which the CHANNEL is the bottleneck and the
delivered rate falls back to it. At that ceiling the cost is exactly the
{B.DMA_DUAL_BYTE_CLK} clk/B floor, which is where the section above already put it.""")
tools/analysis/16_span_roundtrip.py
+27 -6
View File
@@ -43,12 +43,26 @@ ap.add_argument("--cache", default=None)
a = ap.parse_args()
cache = a.cache or f"tmp/model_{os.path.basename(a.frames_dir.rstrip('/'))}.pkl"
# The cache is keyed on the frames directory ALONE, which was fine while
# H.build had no options and became a trap the moment it did: session 28's
# reserved black entry (23.4) changes the palette, the codebooks and every
# index in the model, and a pickle from before it would have let this gate
# round-trip a container the shipping encoder no longer emits -- green, and
# testing the wrong artefact. So the build parameters are stored WITH the
# model and a mismatch rebuilds.
SIG = dict(k1=256, k4=256, iters=16, reserve_black=True)
m = None
if os.path.exists(cache):
m = pickle.load(open(cache, "rb"))
print(f"model from {cache}")
else:
if m.get("sig") != SIG:
print(f"{cache}: built with {m.get('sig')}, wanted {SIG} -- rebuilding")
m = None
else:
print(f"model from {cache}")
if m is None:
t = time.time()
m = H.build(a.frames_dir, k1=256, k4=256, iters=16)
m = H.build(a.frames_dir, **SIG)
m["sig"] = SIG
pickle.dump(m, open(cache, "wb"))
print(f"built model in {time.time()-t:.0f} s -> {cache}")
@@ -72,8 +86,15 @@ for span_mode in ("need", "all"):
f"({100*px/(len(recs)*m['H']*m['W']):.1f}% of all pixels)")
d = DLX(path)
if d.version != 3:
print(f"FAIL: container is DLX{d.version}, not DLX3"); bad += 1; continue
if not d.has_spans:
print(f"FAIL: container is DLX{d.version}, which has no span section")
bad += 1; continue
# DLX4 adds the record index and DLX() cross-checks it against its own walk
# of the frame stream, so simply constructing it above has already gated
# that. Said out loud here because it is easy to read this as version drift.
if d.has_index:
print(f" DLX{d.version}: record index agrees with the frame stream on all "
f"{d.nframes} records ({2*d.nframes:,} B of scene header)")
# The decoder's own walk of the span section must land exactly where the
# block payload starts, and blocks() already raises if the payload does not
@@ -104,5 +125,5 @@ print()
if bad:
print(f"FAILED: {bad} check(s)")
sys.exit(1)
print("OK the DLX3 span container round-trips: the reference decoder rebuilds "
print("OK the span container round-trips: the reference decoder rebuilds "
"the\n encoder's reconstruction exactly, from the emitted bytes.")
tools/analysis/19_ring_stream.py
+1 -1
View File
@@ -71,7 +71,7 @@ def records(path):
the ring must hold per frame are the padded ones, not the payload.
"""
d = DLX(path)
rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
rec = np.array(d.record_lengths(), np.int64)
return d, rec
tools/analysis/20_seek_slack.py
+1 -1
View File
@@ -32,7 +32,7 @@ SECTOR = 512
def records(path):
d = DLX(path)
rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
rec = np.array(d.record_lengths(), np.int64)
return d, rec
tools/analysis/21_iplrom_dmac.py
+2 -44
View File
@@ -31,6 +31,7 @@ registers itself. It is evidence about what Sharp's engineers could get the
board to do, from the vendor, for these exact devices.
"""
import sys, os, argparse, hashlib
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
BASE = 0xFE0000 # where the IPL ROM is mapped (and its 0xFF0000 alias)
@@ -41,50 +42,7 @@ KNOWN = {
"IPL 1.0 (MAME x68000 -bios ipl10), 131,072 B",
}
# --- MC68450 register map, by offset inside a channel's 0x40 block ----------
REG = {0x00: "CSR", 0x01: "CER", 0x04: "DCR", 0x05: "OCR", 0x06: "SCR",
0x07: "CCR", 0x0A: "MTC", 0x0C: "MAR", 0x14: "DAR", 0x1A: "BTC",
0x1C: "BAR", 0x25: "NIV", 0x27: "EIV", 0x29: "MFC", 0x2D: "CPR",
0x31: "DFC", 0x39: "BFC"}
XRM = {0: "burst",
1: "UNDEFINED",
2: "cycle steal WITHOUT hold (bus released between operands)",
3: "cycle steal with hold"}
DTYP = {0: "68000-compatible, EXPLICITLY addressed -> DUAL ADDRESS",
1: "6800-compatible, EXPLICITLY addressed -> DUAL ADDRESS",
2: "device with ACK, implicitly addressed -> SINGLE ADDRESS",
3: "device with ACK and RDY, implicit -> SINGLE ADDRESS"}
DPS = {0: "8-bit port", 1: "16-bit port"}
PCL = {0: "status input", 1: "status input with interrupt",
2: "start pulse", 3: "abort input"}
SIZE = {0: "byte", 1: "word", 2: "long word", 3: "byte, unpacked"}
CHAIN= {0: "none", 1: "UNDEFINED", 2: "array", 3: "linked array"}
REQG = {0: "auto-request at limited rate", 1: "auto-request at max rate",
2: "EXTERNAL request (one operand per device request)",
3: "auto-request first operand, external thereafter"}
def dcr(v):
return [f"XRM = {v>>6&3:02b} {XRM[v>>6&3]}",
f"DTYP = {v>>4&3:02b} {DTYP[v>>4&3]}",
f"DPS = {v>>3&1:b} {DPS[v>>3&1]}",
f"PCL = {v&3:02b} {PCL[v&3]}"]
def ocr(v):
return [f"DIR = {v>>7&1:b} " +
("device -> memory (read)" if v & 0x80 else "memory -> device (write)"),
f"SIZE = {v>>4&3:02b} {SIZE[v>>4&3]}",
f"CHAIN= {v>>2&3:02b} {CHAIN[v>>2&3]}",
f"REQG = {v&3:02b} {REQG[v&3]}"]
def scr(v):
m = {0: "no count", 1: "increment", 2: "decrement", 3: "UNDEFINED"}
return [f"MAC = {v>>2&3:02b} memory address {m[v>>2&3]}",
f"DAC = {v&3:02b} device address {m[v&3]}"]
from mc68450 import REG, XRM, DTYP, DPS, PCL, SIZE, CHAIN, REQG, dcr, ocr, scr
# --- the evidence ----------------------------------------------------------
# (address, expected bytes, one-line description). Every register value quoted
tools/analysis/22_scene_load.py
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""What a scene change costs, now that the loader runs on the 68000 (FINDINGS 53).
python3 tools/analysis/22_scene_load.py [container ...] --kbps R [R ...]
ROADMAP P1 asked for the codebook expansion to be priced "against the refill
climb, not treated as free setup", and that is the whole job of this file. A
scene change is the one moment where every cost in this project lands at once:
the ring is empty because of the seek, the header has to arrive before a single
frame can be drawn, and the 68000 cannot decode anything until it has expanded
the codebooks out of that header.
THREE COSTS, IN THREE DIFFERENT UNITS, and they are not interchangeable:
* BYTES. The container's header region -- palette, CB1, CB4 -- must be
delivered before frame 0 can be decoded. It is not part of any frame
record, so no rate table in this tree has ever counted it.
* CLOCKS. What src/player/load.i costs to turn that header into what the
block loop reads, MEASURED on the emulated 68000 by tools/bench/load.lua
and parsed out of its log rather than copied in here as a constant.
* ACCUMULATED SLACK. The bytes above are bytes the pipe did not spend
filling the ring, so they cost play-time at the surplus rate (pipe - wire),
which is the currency FINDINGS 51.3 established a branch point spends.
This is the one that compounds: it is charged on top of the seek itself.
`--kbps` is REQUIRED and takes no default, for the reason FINDINGS 50 gives.
Rates are decimal-KB per the rest of the tree's tooling; sizes are KiB.
"""
import sys, os, re, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
import ratectl as RC
FPS = 12
CPUHZ = 10_000_000
def rig_cycles(path):
"""The measured per-stage cost, out of tools/bench/load.lua's own log.
Parsed rather than pasted: a constant copied in here would go stale the
first time load.i changed, and it would go stale SILENTLY -- the arithmetic
below would keep working and keep being wrong.
"""
if not os.path.exists(path):
sys.exit(f"no rig log at {path} -- run tools/bench/load_run.sh first")
out = {}
for line in open(path, "rb").read().decode("utf-8", "replace").splitlines():
m = re.search(r"^\[LOD\]\s+(\S.*?)\s{2,}(\d+) cyc", line)
if m:
out[m.group(1).strip()] = int(m.group(2))
need = ("SCENE CHANGE: codebooks + palette", "scratch tables only (boot, once)")
for k in need:
if k not in out:
sys.exit(f"{path} has no '{k}' line -- is it a load.lua summary?")
return out
ap = argparse.ArgumentParser()
ap.add_argument("containers", nargs="*", default=["tmp/rc_fr_singe_scsi_span.dlx"])
ap.add_argument("--kbps", type=float, nargs="+", required=True,
help="delivered pipe rates, KB/s. REQUIRED, no default (FINDINGS 50)")
ap.add_argument("--log", default="tmp/load_check.log",
help="tools/bench/load.lua's log, for the measured cycle counts")
a = ap.parse_args()
cyc = rig_cycles(a.log)
scene_cyc = cyc["SCENE CHANGE: codebooks + palette"]
boot_cyc = cyc["scratch tables only (boot, once)"]
frame_cyc = CPUHZ / FPS
print(f"measured on the emulated 68000 ({a.log}):")
print(f" per scene change {scene_cyc:>8,} clocks = {1000*scene_cyc/CPUHZ:6.2f} ms "
f"= {100*scene_cyc/frame_cyc:.1f}% of one {FPS}fps frame")
print(f" once at boot {boot_cyc:>8,} clocks = {1000*boot_cyc/CPUHZ:6.2f} ms "
f" (the three scratch tables: scene-independent)")
for path in a.containers:
d = DLX(path)
hdr = int.from_bytes(d.raw[28:32], "big")
rec = np.array(d.record_lengths(), np.int64)
wire = rec.mean() * FPS / 1024 + RC.AUDIO_KBPS
print(f"\n=== {path}: header region {hdr:,} B "
f"(pal 768 + cb1 {d.k1*16:,} + cb4 {d.k4*4:,} + 32), wire {wire:.1f} KB/s")
print(f"{'pipe':>6} {'header ms':>10} {'+load ms':>9} {'total':>7} "
f"{'frames':>7} {'surplus':>9} {'slack s':>9}")
for kbps in a.kbps:
hdr_ms = 1000 * hdr / (kbps * 1024)
load_ms = 1000 * scene_cyc / CPUHZ
total = hdr_ms + load_ms
surplus = kbps - wire
# What the header costs in the currency of 51.3: play-time at the
# surplus rate. A negative surplus means the container does not fit the
# pipe at all and no amount of play buys the bytes back.
slack = f"{hdr/(surplus*1024):8.3f}" if surplus > 0 else " NEVER"
print(f"{kbps:>6.0f} {hdr_ms:>10.2f} {load_ms:>9.2f} {total:>7.2f} "
f"{total/(1000/FPS):>7.2f} {surplus:>9.1f} {slack:>9}")
print("""
Reading it. The 'frames' column is the scene change's FIXED cost in 12fps
frame slots, before the ring has been given a single frame of lookahead -- so it
is a floor under the black gap at a branch point, not the gap itself. The
'slack s' column is the one that compounds with FINDINGS 51.3: the header's
bytes are bytes that did not go into the ring, so they lengthen the climb back
to the seek-slack ceiling by that much play-time, every time.""")
tools/analysis/23_frame_clock.py
+254
View File
@@ -0,0 +1,254 @@
#!/usr/bin/env python3
"""What a real frame clock can be built from, and what its cadence costs.
python3 tools/analysis/23_frame_clock.py [--fps 12] [--vtotal 568]
[--csv tmp/c68k_frames.csv]
ROADMAP P3 says "needs MFP timer or VBL", which hides the fact that ONE OF
THOSE CANNOT DO IT and the other cannot do it either without a divider. This
file enumerates the space rather than asserting a conclusion, the way FINDINGS
47 had to be re-done once a hardware "no" turned out to be a claim about a whole
configuration space nobody had walked.
EVERY CONSTANT HERE IS SOURCED, and from a file on this machine:
MFP timer clock 16 MHz / 4 MAME 0.277 sharp/x68k.cpp:1027-1028
prescaler ladder 4,10,16,50,64,100,200 machine/mc68901.cpp:173
timer data reg 8 bits, 0 means 256 machine/mc68901.cpp (TCDR/TADR)
V-DISP -> GPIP4 x68k.cpp:1139, and it is also Timer A's event input,
mc68901.cpp:167 GPIO_TIMER = {GPIP_4, GPIP_3}
line rate 31,500 Hz exactly in both 31.5 kHz modes, derived in
tools/bench/crtc_mode.lua from the dot clocks
interrupt cost MEASURED, not tabled: tools/bench/clock_run.sh
THE PART THAT IS NOT A CLOCK PROBLEM AT ALL. 12 fps on a 55.4577 Hz raster is
4.6215 refreshes per frame, so every frame is shown for 4 refreshes or 5 --
72.13 ms or 90.16 ms -- and 37.9% of them get the short one. That is the
display's quantisation and no choice of clock changes it. What it changes is
the BUDGET: 833,333 clocks is the mean slot, not the slot, and the short slot is
721,270. With the per-frame decode costs in hand this file says exactly how
many frames do not fit theirs, which is a thing this project has never had to
ask because until now the tick came from a host that could not miss.
"""
import argparse
import csv
import os
import sys
MFP_HZ = 16_000_000 // 4 # x68k.cpp:1027-1028
PRESCALER = [4, 10, 16, 50, 64, 100, 200] # mc68901.cpp:173
HFREQ = 31500 # lines/s, crtc_mode.lua
CPUHZ = 10_000_000 # 40 MHz / 4, x68k.cpp:1133
def timer_space(fps):
"""Every (prescale, data) the MFP can be set to, against a target fps."""
slowest = MFP_HZ / (PRESCALER[-1] * 256)
print(f"\n=== 1. THE MFP TIMER, WHICH CANNOT DO IT ALONE")
print(f" timer clock {MFP_HZ:,} Hz, prescalers {PRESCALER}, data 1..256")
print(f" slowest tick any single timer can produce: "
f"{MFP_HZ}/({PRESCALER[-1]}*256) = {slowest:.3f} Hz")
print(f" a {fps} fps frame needs {fps} Hz, which is {slowest/fps:.1f}x "
f"slower than that -- so a software divider is REQUIRED whatever the "
f"source, and 'use an MFP timer' is not by itself an answer.")
exact = [(p, d) for p in PRESCALER for d in range(1, 257)
if (MFP_HZ * d * p) and (MFP_HZ % (p * d) == 0)
and (MFP_HZ // (p * d)) % fps == 0]
print(f" settings whose tick rate is a whole multiple of {fps} Hz, so that "
f"a plain counter would be exact: {len(exact)}")
if exact:
best = min(exact, key=lambda pd: MFP_HZ / (pd[0] * pd[1]))
p, d = best
tick = MFP_HZ / (p * d)
print(f" slowest of them: prescale /{p} data {d} = {tick:.4f} Hz, "
f"{tick/fps:.0f} ticks per frame")
print(f" -> {tick/fps:.0f} interrupts per frame, against 4.6215 for "
f"the raster: {tick/fps/(HFREQ/(fps*568)):.1f}x the cost, and its "
f"phase against the raster is arbitrary, so a frame would be "
f"presented mid-scan.")
else:
print(f" NONE. {MFP_HZ}/{fps} = {MFP_HZ/fps:,.2f} is not an "
f"integer, so no prescale/data pair divides to {fps} Hz at all.")
def raster_space(fps, vtotal):
hz = HFREQ / vtotal
print(f"\n=== 2. THE RASTER, WHICH IS THE RIGHT SOURCE AND IS ALSO NOT "
f"A WHOLE DIVIDE")
print(f" V-DISP is {HFREQ}/{vtotal} = {hz:.4f} Hz, and it is BOTH the "
f"GPIP4 interrupt and Timer A's event-count input")
print(f" whole divides -- all the MFP can do in hardware, no software:")
for n in (3, 4, 5, 6):
f = hz / n
print(f" Timer A event count = {n}: {f:7.4f} fps "
f"({100*(f/fps-1):+6.2f}% from {fps})")
print(f" {fps} fps needs {hz/fps:.4f} refreshes per frame, which is not a "
f"whole number, so no event-count setting is exact either.")
print(f"\n THE DIVIDER THAT IS EXACT: add fps*VTOTAL = {fps*vtotal} per "
f"V-DISP, emit a tick at {HFREQ}, keep the remainder.")
print(f" long-run rate = {fps}*{vtotal}/{vtotal} = {fps} fps EXACTLY, "
f"with a remainder that never accumulates")
print(f" the accumulator stays under {HFREQ + fps*vtotal:,}, so it is "
f"16-bit arithmetic on a 68000 (the ceiling is fps < "
f"{(65536-HFREQ)/vtotal:.1f} at this VTOTAL, and clk_init checks it)")
def divider_gaps(fps, vtotal, n):
"""The tick sequence src/player/clock.i emits, in refreshes per tick."""
acc, gaps, since = 0, [], 0
while len(gaps) < n:
acc += fps * vtotal
since += 1
if acc >= HFREQ:
acc -= HFREQ
gaps.append(since)
since = 0
return gaps
def host_gaps(fps, refresh_hz, n):
"""The tick sequence tools/bench/stream.lua's HOST clock emits.
It looks uniform in the source -- `floor((t - t_rel) * fps)` -- and is not.
Lua only sees the machine at frame boundaries, so tick k lands on the first
refresh at or after k/fps, and the gaps between ticks come out as the same
two whole numbers of refreshes the divider produces. The host-paced runs of
FINDINGS 49 and 51 therefore already had this cadence in them; what ROADMAP
P3 changes is who produces it, not whether it exists.
"""
at = [-(-int(k * refresh_hz * 1000000 // fps) // 1000000) for k in range(n + 1)]
return [at[k + 1] - at[k] for k in range(n)]
def cadence(fps, vtotal, costs, label, refresh_hz, gaps, uniform=False):
"""Charge each frame the slot it really gets, and run the pace gate."""
rpf = 1.0 if uniform else HFREQ / (fps * vtotal)
lo, hi = (1, 1) if uniform else (int(rpf), int(rpf) + 1)
slot_lo = lo / refresh_hz * CPUHZ
slot_hi = hi / refresh_hz * CPUHZ
nominal = CPUHZ / fps
n_lo = sum(1 for g in gaps[:len(costs)] if g == lo)
print(f"\n --- {label}: refresh {refresh_hz:.4f} Hz")
print(f" slots are {lo} refreshes = {slot_lo:,.0f} clk "
f"({1000*lo/refresh_hz:.2f} ms) or {hi} = {slot_hi:,.0f} clk "
f"({1000*hi/refresh_hz:.2f} ms)")
print(f" the nominal {fps} fps budget every figure in this project is "
f"priced against is {nominal:,.0f} clk; the SHORT slot is "
f"{100*(slot_lo/nominal-1):+.1f}% of it")
over_lo = sum(1 for c in costs if c > slot_lo)
over_hi = sum(1 for c in costs if c > slot_hi)
over_nom = sum(1 for c in costs if c > nominal)
print(f" frames that do not fit: {over_lo}/{len(costs)} the short "
f"slot, {over_hi}/{len(costs)} the long one, {over_nom}/{len(costs)} "
f"the nominal budget")
# The schedule, with the catch-up the pace gate actually performs: frame i
# starts at max(finish of i-1, tick i). A frame that overruns does not fail
# -- it eats the next frame's idle, and the clock catches up by itself.
t, tick, late, worst, first = 0.0, 0.0, 0, 0.0, None
for i, c in enumerate(costs):
if t <= tick:
t = tick # idled: on time
else:
if i: # frame 0 has no predecessor
late += 1
if first is None:
first = i
worst = max(worst, t - tick)
t += c
tick += gaps[i] / refresh_hz * CPUHZ
print(f" through the pace gate: {late}/{len(costs)} frames found "
f"their slot already open (first at frame {first}), worst start "
f"{worst:,.0f} clk = {1000*worst/CPUHZ:.1f} ms behind its tick")
if not uniform:
print(f" {n_lo}/{len(costs)} slots were the short one "
f"({100*n_lo/len(costs):.1f}%; the exact share is "
f"{100*(hi-rpf):.1f}%)")
return late
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--fps", type=int, default=12)
ap.add_argument("--vtotal", type=int, default=568,
help="CRTC R04+1; 568 is the 31.5 kHz 256-line mode")
ap.add_argument("--csv", default="tmp/c68k_frames.csv",
help="per-frame decode cost, from tools/bench/c68k/run.sh")
a = ap.parse_args()
timer_space(a.fps)
raster_space(a.fps, a.vtotal)
print(f"\n=== 3. WHAT THE CADENCE COSTS")
if not os.path.exists(a.csv):
print(f" {a.csv} not found -- run tools/bench/c68k/run.sh first. The "
f"cadence question CANNOT be answered from percentiles: it needs "
f"the per-frame series, because what matters is whether an "
f"expensive frame lands in a short slot and how long the catch-up "
f"takes afterwards.")
return 1
costs = [float(r["cycles"]) for r in csv.DictReader(open(a.csv))]
print(f" {len(costs)} frames from {a.csv}: mean {sum(costs)/len(costs):,.0f} "
f"clk, max {max(costs):,.0f} (frame {costs.index(max(costs))})")
hw = HFREQ / a.vtotal
n = len(costs)
# MAME's screen is fast by htotal/(htotal-8): refresh_mode() builds the
# frame period from scr.max_x*scr.max_y with scr.max_x = m_htotal - 8. Both
# rasters are run, because the rig measures against the fast one and the
# player will run on the other -- reporting only one would leave the rig's
# count and this file's differing with nobody able to say which was wrong.
htotal = 368
mame = hw * htotal / (htotal - 8)
# The budget model every figure in FINDINGS assumes: a slot of exactly
# 1/fps. No machine has this; it is the yardstick, run through the same
# schedule so that what the cadence ADDS can be read off.
cadence(a.fps, a.vtotal, costs, "the NOMINAL model (a slot of exactly "
"1/fps, which no raster produces)", float(a.fps),
[1] * (n + 1), uniform=True)
cadence(a.fps, a.vtotal, costs, "the HOST tick, as tools/bench/stream.lua "
"actually emits it", mame, host_gaps(a.fps, mame, n + 1))
cadence(a.fps, a.vtotal, costs, "the 68000's own clock on the HARDWARE "
"raster", hw, divider_gaps(a.fps, a.vtotal, n + 1))
cadence(a.fps, a.vtotal, costs, f"the 68000's own clock on MAME's raster "
f"(fast by {htotal}/{htotal-8})", mame,
divider_gaps(a.fps, a.vtotal, n + 1))
print(f"""
Reading it.
THE CADENCE WAS ALREADY THERE. The nominal row is the model every budget in
this project is priced against -- a slot of exactly 1/fps -- and no raster
produces it. The host row is what tools/bench/stream.lua has been emitting all
along: `floor((t - t_rel) * fps)` looks uniform, but Lua only sees the machine at
frame boundaries, so its ticks land on refreshes and its gaps are the same two
whole numbers. The host-paced results of FINDINGS 49 and 51 therefore already
carried a 4/5 cadence that nothing named. ROADMAP P3 did not introduce it; it
moved who produces it onto the machine, where it belongs, and made it visible.
THE SHORT SLOT IS REAL AND IT IS NOT A FAILURE. {sum(1 for c in costs if c > 721270)}/{len(costs)} frames do not fit
721,270 clocks. The pace gate only says "not before tick i", so a frame that
overruns spends the next frame's idle and the clock recovers by itself; the cost
is one frame presented a refresh late, not a dropped frame. What the counts
above measure is frames with no idle left, and the difference between the
nominal row and the raster rows -- 1 against 4 -- is the whole price of the
cadence on this container.
THE EXPENSIVE FRAME IS FRAME 0, at {max(costs)/(CPUHZ/a.fps)*100:.0f}% of the nominal budget: the first
frame of a scene has nothing to SKIP against, so it is the whole picture in one
slot. Most of what follows it in these counts is that transient draining, which
is why the first index is printed next to the total. It also means the cost is
paid AT A SCENE CHANGE, alongside the 18.96 ms of loader (FINDINGS 53.2) and the
seek -- not spread over the window.
SCOPE. Decode costs are C68K's, on zero-wait-state memory, so they are a lower
bound; real DRAM moves every row here in the same direction. The MAME rows are
the emulator's fast raster and exist to be compared with tools/bench/pace_run.sh,
not to describe hardware.""")
return 0
if __name__ == "__main__":
sys.exit(main())
tools/analysis/24_ring_owner.py
+305
View File
@@ -0,0 +1,305 @@
"""What the player's own fill loop costs the pipe (ROADMAP P5, FINDINGS 55).
19_ring_stream.py and 20_seek_slack.py model a ring whose producer is free to
act whenever it likes: bytes arrive at a rate and the only questions are where
they go and whether the ring can hold them. That is what a HOST-filled ring is,
and it is what every delivery figure in FINDINGS 49 and 51 was measured on.
A player has no host. The 68000 owns the ring (src/player/ring.i), and it can
only act when it is not decoding -- which turns the producer into a consumer of
the same resource the decoder is short of, and puts an idle CHANNEL between
every pair of records:
a transfer ends -> the disc has nothing to do -> the CPU next polls
-> it issues -> the disc starts again
The gap in the middle is bytes the medium could have delivered and did not, and
no rate table in this tree contains it. Its size is set by the PLAYER: how many
requests it may have outstanding (a DMAC channel takes one at a time; two slots
mean the next is already queued when the current lands), and when it polls.
This is that model, written from record sizes and per-frame decode costs, and
sharing no code with the Lua rig it is compared against -- the same arrangement
as 49.4 and 51.5. The rig drives a real 68000 through a real ring and is the
measurement; this says where to point it and what to expect.
python3 tools/analysis/24_ring_owner.py <container.dlx> --kbps R [R ...]
[--ring KB] [--qdepth N [N ...]] [--prefill RECORDS]
[--cadence raster|nominal]
`--kbps` is REQUIRED and has no default, for the reason FINDINGS 50 gives.
THE INDEX IS READ FROM THE CONTAINER, not derived by walking it. A DLX4
container carries nframes u16 record lengths in its scene header precisely
because the producer needs the length of a record it has not fetched; this model
reads the same table the 68000 does, so a container whose index disagreed with
its stream would be caught here as well as in dlx.py's constructor.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
import vq_hybrid as H
import spans as SP
CPUHZ = 10_000_000
HFREQ = 31500 # lines/s in the 31.5 kHz modes (src/player/clock.i)
VTOTAL = 568 # CRTC R04+1 in the 256x256 mode (tools/bench/crtc_mode.lua)
CLK_ISR = 181.35 # clocks per V-DISP, MEASURED (FINDINGS 54.3)
AUDIO_KBPS = 7.8 # ratectl.AUDIO_KBPS; the pipe carries it too
def frame_costs(d):
"""Per-frame decode cost in 68000 clocks: blocks plus v7 spans.
Both halves come from the encoder's own measured constants -- H.cycles is
the single source 11_cpu_budget.py uses, and SP.clocks is the v7 fit of
FINDINGS 40 -- so this is the same cost model the rate controller fits `mu`
against, applied to the emitted container rather than to a candidate.
"""
out = []
for f in range(d.nframes):
c = H.cycles(d.modes(f))
for _, _, px in d.spans(f)[0]:
c += SP.clocks(len(px))
out.append(c)
return np.array(out)
def ticks(n, fps):
"""Frame tick times from src/player/clock.i's divider, or a nominal clock.
The player's clock is the raster with a remainder: a frame gets 4 refreshes
(72.13 ms) or 5 (90.16 ms) and there is no 83.33 ms frame (FINDINGS 54.4).
A model that hands out uniform slots gives every frame 13.4% more time than
the short one really has, so the cadence is reproduced here rather than
averaged away.
"""
R = VTOTAL / HFREQ # one refresh, seconds
acc, out, t = 0, [0.0], 0.0
while len(out) < n:
t += R
acc += fps * VTOTAL
if acc >= HFREQ:
acc -= HFREQ
out.append(t)
return np.array(out)
def simulate(rec, dec, tick, bps, ringsz, qdepth, prefill):
"""One pass of the machine-owned ring. Returns a dict of instruments.
The rules are src/player/ring.i's, stated as events:
* the CPU polls whenever it is NOT decoding -- the pace wait and the
record wait both call ring_poll, and nothing else in the frame does;
* a request occupies a slot until it is RETIRED, which happens at a poll,
so the queue is measured against retirement and not against completion;
* placement is `aligned`: a record that will not fit before the end of the
ring restarts at the base, and only if the base is free;
* the channel serves one transfer at a time, in order.
"""
n = len(rec)
# ring state, in ring offsets
wcur = rcur = 0
rq = 0 # next record to request
retired = 0 # requests retired (== FR_HEAD)
consumed = 0 # records the decoder has finished (== FR_TAIL)
inflight = [] # [(record, done_time)] in issue order
chan_free = 0.0 # when the channel finishes what it has
gaps, gap_tot, gap_max = 0, 0.0, 0.0
busy = 0.0
full_refusals = 0
started = False # the first transfer has no gap before it
def live_empty():
return rq == consumed
def place(length):
"""Where the next record goes: (offset, hole) or None if it cannot."""
nonlocal full_refusals
if live_empty():
if wcur + length <= ringsz:
return wcur, 0
return 0, ringsz - wcur
if rcur == wcur:
return None # completely full
if rcur < wcur: # free is [wcur, SZ) then [0, rcur)
if wcur + length <= ringsz:
return wcur, 0
if length <= rcur:
return 0, ringsz - wcur
return None
if wcur + length <= rcur: # live wraps; free is [wcur, rcur)
return wcur, 0
return None
def issue(now):
"""Issue as many requests as the queue and the ring allow, at `now`."""
nonlocal wcur, rcur, rq, chan_free, gaps, gap_tot, gap_max, busy
nonlocal full_refusals, started
while rq < n and (rq - retired) < qdepth:
p = place(rec[rq])
if p is None:
full_refusals += 1
return
off, _hole = p
if live_empty():
rcur = off
start = max(now, chan_free)
if started:
g = start - chan_free
if g > 1e-12:
gaps += 1
gap_tot += g
gap_max = max(gap_max, g)
started = True
dur = rec[rq] / bps
busy += dur
chan_free = start + dur
inflight.append((rq, chan_free))
wcur = off + rec[rq]
rq += 1
def retire(now):
"""Publish every transfer that has landed by `now`. In order."""
nonlocal retired
while inflight and inflight[0][1] <= now:
inflight.pop(0)
retired += 1
def advance_reader():
"""Step rcur over the records the decoder has finished with."""
nonlocal rcur
i = consumed_seen[0]
while i < consumed:
end = rcur + rec[i]
if i + 1 < n and end + rec[i + 1] > ringsz:
end = 0
rcur = end
i += 1
consumed_seen[0] = i
consumed_seen = [0]
# ---- prefill. The decoder is not running, so the CPU polls continuously
# and the channel never waits for it: this is the one part of a scene where
# the request loop costs nothing.
now = 0.0
while retired < prefill and rq < n:
issue(now)
if not inflight:
break
now = inflight[0][1]
retire(now)
prefill_done = now
t0 = now
underruns, worst_late, noidle = 0, 0.0, 0
slack_series = []
for i in range(n):
deadline = t0 + tick[i]
# TWO WAYS A FRAME CAN START LATE, AND THEY ARE NOT THE SAME FAILURE.
# The decoder reaches the record wait at max(its own finish, the tick):
# if it got there after the tick, the PREVIOUS frame used its whole slot
# and this is the CPU (54.4's cadence). If it got there on time and the
# record was not resident, that is the PIPE. The rig counts them
# separately -- NO IDLE and UNDERRUNS -- so conflating them here would
# have made the model disagree with it for a reason that is not about
# delivery at all.
if now > deadline + 1e-9:
noidle += 1
arrive = max(now, deadline)
now = arrive
# the record wait: the CPU polls, so it retires and issues while it waits
starved = retired <= i
while retired <= i:
issue(now)
if not inflight:
break
now = max(now, inflight[0][1])
retire(now)
if starved:
underruns += 1
worst_late = max(worst_late, now - arrive)
slack_series.append(retired - consumed)
issue(now)
# ---- decode. No polls: whatever the channel finishes now waits.
now += dec[i] / CPUHZ
consumed += 1
advance_reader()
retire(now)
issue(now)
# ---- idle until the next tick. The CPU polls throughout, so every
# completion is retired and every free slot is refilled at once.
nxt = t0 + tick[i + 1] if i + 1 < n else now
while inflight and inflight[0][1] < nxt:
now = max(now, inflight[0][1])
retire(now)
issue(now)
now = max(now, min(nxt, now))
span = max(now - t0, 1e-9)
return dict(underruns=underruns, worst_late=worst_late, gaps=gaps,
noidle=noidle,
gap_tot=gap_tot, gap_max=gap_max, busy=busy, span=span,
ceiling=max(slack_series), mean_slack=float(np.mean(slack_series)),
full=full_refusals, prefill_s=prefill_done)
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--kbps", type=float, nargs="+", required=True,
help="delivery rates to model. REQUIRED: this tree has no "
"default rate (FINDINGS 50)")
ap.add_argument("--ring", type=int, default=256, help="ring size, KB")
ap.add_argument("--qdepth", type=int, nargs="+", default=[1, 2],
help="requests the player may have outstanding")
ap.add_argument("--prefill", type=int, default=2, help="records before release")
ap.add_argument("--cadence", choices=["raster", "nominal"], default="raster")
a = ap.parse_args()
d = DLX(a.container)
if not d.has_index:
sys.exit(f"{a.container} is DLX{d.version}: this model reads the record "
f"index the player reads, and only DLX4 carries one.")
rec = np.array([q * 4 for q in d.index], np.int64)
dec = frame_costs(d) + CLK_ISR * (HFREQ / VTOTAL) / d.fps # + the clock's own
if a.cadence == "raster":
tick = ticks(d.nframes + 1, d.fps)
else:
tick = np.arange(d.nframes + 1) / d.fps
wire = rec.mean() * d.fps / 1024
print(f"{a.container}: {d.nframes} records, {rec.mean()/1024:.1f} KB mean, "
f"{rec.max()/1024:.1f} KB max, wire {wire:.1f} KB/s")
print(f" index: {2*d.nframes:,} B of scene header -- read, not walked")
print(f" decode: mean {dec.mean():,.0f} clk/frame ({100*dec.mean()/(CPUHZ/d.fps):.1f}% "
f"of a mean slot), p90 {np.percentile(dec,90):,.0f}")
print(f" cadence: {a.cadence}"
+ (" (4 or 5 refreshes a frame, 72.13/90.16 ms -- FINDINGS 54.4)"
if a.cadence == "raster" else " (uniform 1/fps slots)"))
print(f" ring {a.ring} KB, prefill {a.prefill} records\n")
hdr = (f"{'pipe':>8} {'Q':>2} {'idle':>9} {'gaps':>5} {'worst':>8} "
f"{'under':>7} {'late by':>8} {'noidl':>5} {'ceil':>5} {'mean':>5} "
f"{'refus':>6}")
print(hdr)
print("-" * len(hdr))
for kb in a.kbps:
bps = (kb - AUDIO_KBPS) * 1024
for q in a.qdepth:
r = simulate(rec, dec, tick, bps, a.ring * 1024, q, a.prefill)
print(f"{kb:8.0f} {q:2d} {100*r['gap_tot']/r['span']:8.1f}% "
f"{r['gaps']:5d} {r['gap_max']*1000:7.1f}ms "
f"{r['underruns']:3d}/{d.nframes:<3d} {r['worst_late']*1000:7.1f}ms "
f"{r['noidle']:5d} {r['ceiling']:5d} {r['mean_slack']:5.1f} "
f"{r['full']:6d}")
print()
print("idle = the channel with no request to work on, as a fraction of the")
print(" window. Bytes the medium could have delivered and did not.")
print("under = frames whose record was not resident when the decoder asked")
print(" for it. The PIPE.")
print("noidl = frames that reached the gate after their tick, because the one")
print(" before used its whole slot. The CPU, and 54.4's cadence.")
print("ceil = most records resident and unconsumed at a frame start: what a")
print(" branch point could spend, minus one for the restart (51.2).")
print("refus = placements refused for SPACE. Nonzero means the ring filled.")
tools/analysis/25_scene_graph.py
+363
View File
@@ -0,0 +1,363 @@
"""The worst gap between two decision points, out of the scene graph (G1).
FINDINGS 51.3 measured that a ring's lookahead is ACCUMULATED out of
`pipe - wire` and that a seek spends all of it, so what a branch point costs is
set by the rate and by the time since the last branch. 55.5 rehearsed a seek on
the machine and could not ask the question that matters, because nothing in this
tree knew where the branch points ARE:
what is the WORST gap, in seconds of play, between two consecutive
decision points, and does the refill climb survive it?
Only the arcade scene graph knows. This answers it in the currency 51.3
established.
python3 tools/import/scenegraph.py # writes tmp/scenegraph.json
python3 tools/analysis/25_scene_graph.py --kbps R [R ...] [--ring KB [KB ...]]
`--kbps` is REQUIRED and takes no default, for the reason FINDINGS 50 gives.
THIS FILE KNOWS NOTHING ABOUT WHERE THE TABLE CAME FROM, deliberately. It reads
`DLXSCENE1`, which is this project's own schema; `tools/import/scenegraph.py` is
the single file in the tree that knows anything about the outside projects the
table is built from, and it carries their attribution. Nothing is vendored.
"""
import sys, os, json, argparse, importlib.util
sys.path.insert(0, "tools/encoder")
TABLE = os.environ.get("DLX_SCENEGRAPH", "tmp/scenegraph.json")
LD_FPS = 23.976 # the medium's frame rate; one frame is the comparison floor
# ------------------------------------------------------------------- graph
class Node:
"""One (scene, sequence): a clip, its exits, and whether entering it seeks."""
def __init__(self, scene, name, seq):
self.scene, self.name, self.seq = scene, name, seq
self.start = seq["start_ms"] # ms, or -1 for no seek
self.seeks = self.start >= 0
self.timeout_ms = seq["timeout_ms"]
self.exits = [(seq["timeout_ms"], "timeout", seq["timeout_next"])]
for a in seq["actions"]:
# The player may press as early as `from_ms`, so that is the least
# play this clip can deliver before the branch it leads to.
self.exits.append((a["from_ms"], "action", a["next"]))
@property
def key(self):
return f"{self.scene}.{self.name}"
def build_graph(scenes):
nodes = {}
for scene, seqs in scenes.items():
for name, seq in seqs.items():
n = Node(scene, name, seq)
nodes[n.key] = n
return nodes
def worst_gap(nodes):
"""Least play time, in ms, between one seek and the next.
A seek is entering a sequence whose start is >= 0; a negative start means
the disc keeps playing, so play ACCUMULATES across such sequences and the
gap is a shortest path over them. Bellman-Ford rather than Dijkstra
because a zero-length timeout is common (`start_alive` chains) and the
graph has cycles; all weights are non-negative so it terminates.
A null exit ends the scene: the player moves to another scene entirely,
which is a seek AND a container change (FINDINGS 53), so it counts as a
seek and is flagged.
"""
INF = float("inf")
dist = {k: (0.0 if n.seeks else INF) for k, n in nodes.items()}
for _ in range(len(nodes) + 1):
changed = False
for n in nodes.values():
if dist[n.key] == INF:
continue
for elapsed, kind, tgt in n.exits:
if tgt is None:
continue
tk = f"{n.scene}.{tgt}"
if tk not in nodes:
continue
d = dist[n.key] + max(0.0, float(elapsed))
if not nodes[tk].seeks and d < dist[tk] - 1e-9:
dist[tk], changed = d, True
if not changed:
break
best = {}
for n in nodes.values():
if dist[n.key] == INF:
continue
for elapsed, kind, tgt in n.exits:
tk = f"{n.scene}.{tgt}" if tgt is not None else None
ends_scene = tgt is None
if ends_scene or (tk in nodes and nodes[tk].seeks):
g = dist[n.key] + max(0.0, float(elapsed))
# One entry per (source, destination): several input windows can
# lead to the same clip and only the earliest of them binds.
k = (n.key, tgt)
if k not in best or g < best[k][0]:
best[k] = (g, n.key, tgt if tgt else "<end of scene>",
kind, ends_scene, n.scene)
return sorted(best.values()), dist
# ------------------------------------------------------ 51.3's currency
def slack_model(gap_s, kbps, wire_kbps, mean_rec_b):
"""Records of lookahead accrued in `gap_s` seconds of play at `kbps`, and
the seconds one record of lookahead costs.
This is 51.3's surplus model and nothing more: slack accrues at
(pipe - wire) bytes per second. The paced rig is the measurement; this
says whether the gap is even in the right order of magnitude.
"""
surplus = (kbps - wire_kbps) * 1024.0
if surplus <= 0:
return 0.0, float("inf")
return surplus * gap_s / mean_rec_b, mean_rec_b / surplus
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--kbps", type=float, nargs="+", required=True,
help="delivered pipe rates, KB/s. REQUIRED, no default "
"(FINDINGS 50).")
ap.add_argument("--table", default=TABLE, help="DLXSCENE1 scene table")
ap.add_argument("--container", default="tmp/rc_fr_singe_scsi_span.dlx",
help="container the wire demand and mean record come from")
ap.add_argument("--ring", type=float, nargs="+", default=[256, 512])
ap.add_argument("--top", type=int, default=12)
a = ap.parse_args()
if not os.path.exists(a.table):
print(f"no scene table at {a.table} -- run:\n"
f" python3 tools/import/scenegraph.py")
return 2
doc = json.load(open(a.table))
if doc.get("format") != "DLXSCENE1":
print(f"{a.table}: not a DLXSCENE1 table")
return 2
nodes = build_graph(doc["scenes"])
c = doc["counts"]
nseek = sum(1 for n in nodes.values() if n.seeks)
print(f"=== the scene graph ({a.table})")
for s in doc["sources"]:
print(f" from {s['name']} ({s['licence']}, {s['holder']}): {s['role']}")
print(f" scenes {c['scenes']}, sequences {c['sequences']}, "
f"input windows {c['windows']}")
print(f" sequences entered by a SEEK: {nseek}/{c['sequences']} "
f"({100*nseek/c['sequences']:.1f}%); the rest play on from where the "
f"disc is")
print(f" scene order: {len(doc['rows'])} rows x {len(doc['rows'][0])}")
# Gates. A parser that quietly dropped a branch would produce a SMALLER
# graph and a LONGER worst gap -- it would fail in the flattering direction.
assert c["sequences"] == 516, f"expected 516 sequences, got {c['sequences']}"
assert c["windows"] == 906, f"expected 906 input windows, got {c['windows']}"
# ---- the measurement
gaps, dist = worst_gap(nodes)
play = [g for g in gaps if g[5] != "attract_mode"]
zero = [g for g in play if g[0] <= 1e-9]
print()
print("=== the worst gap between two consecutive decision points")
print(" A 'gap' is the LEAST play time the disc delivers between one seek")
print(" and the next: the earliest an input window opens, chained across")
print(" sequences the disc plays through without seeking. One entry per")
print(" (source, destination); attract mode is excluded and reported")
print(" separately, because nothing branches there under a 12 fps budget.")
print(f" {'gap s':>7} from -> to")
for g, src, tgt, kind, ends, scene in play[:a.top]:
print(f" {g/1000:>7.3f} {src} -{kind}-> {tgt}"
+ (" [SCENE CHANGE]" if ends else ""))
sc_gaps = [g for g in play if g[4]]
print(f" ... {len(play)} distinct transitions into a seek "
f"({len(gaps)-len(play)} more in attract mode)")
worst = play[0][0] / 1000.0
med = play[len(play) // 2][0] / 1000.0
print(f" WORST {worst:.3f} s, median {med:.3f} s, "
f"best {play[-1][0]/1000:.3f} s")
print(f" SCENE CHANGES specifically ({len(sc_gaps)} of them, and each also")
print(f" needs a header before its frame 0, FINDINGS 53/55.1): worst "
f"{sc_gaps[0][0]/1000:.3f} s, median "
f"{sc_gaps[len(sc_gaps)//2][0]/1000:.3f} s")
print(f" ZERO-PLAY BRANCHES: {len(zero)} of {len(play)} "
f"({100*len(zero)/len(play):.1f}%) open an input window at t=0 of a")
print(" clip the disc SEEKED to, so two seeks can fall back to back with no")
print(" play between them at all. A rule of the form 'has there been")
print(" enough play since the last branch' (51.2's ring_may_seek) can be")
print(" answered NO by the content, not by the buffer.")
# ---- what the input layer has to survive, from the same table
inputs, windows = {}, []
for n in nodes.values():
for x in n.seq["actions"]:
inputs[x["input"]] = inputs.get(x["input"], 0) + 1
windows.append(x["to_ms"] - x["from_ms"])
windows.sort()
print()
print("=== what the input layer has to survive")
print(" " + ", ".join(f"{k} {v}" for k, v in
sorted(inputs.items(), key=lambda x: -x[1])))
diag = sum(v for k, v in inputs.items()
if k in ("upleft", "upright", "downleft", "downright"))
print(f" diagonals are {diag} windows of {len(windows)}: rare enough for a "
f"port to drop\n and not droppable by one aiming at the arcade")
print(f" window length: shortest {windows[0]:.0f} ms "
f"({windows[0]/(1000/12):.2f} frame slots at 12 fps), p10 "
f"{windows[len(windows)//10]:.0f} ms, median "
f"{windows[len(windows)//2]:.0f} ms")
print(" the floor is one to two frames wide (54.4: a slot is 72.13 or")
print(" 90.16 ms, never 83.33), so input cannot be polled on the frame tick")
# ---- 51.3's currency
if os.path.exists(a.container):
spec = importlib.util.spec_from_file_location(
"seek_slack", "tools/analysis/20_seek_slack.py")
m = importlib.util.module_from_spec(spec)
spec.loader.exec_module(m)
import ratectl as RC
d, rec = m.records(a.container)
mean_rec = float(rec.mean())
wire = mean_rec * 12 / 1024 + RC.AUDIO_KBPS
gs = [g[0] / 1000.0 for g in play]
print()
print(f"=== what that gap buys, at explicit rates "
f"({os.path.basename(a.container)}: mean record "
f"{mean_rec/1024:.1f} KB, wire {wire:.1f} KB/s)")
print(f" {'ring KB':>8} {'pipe':>7} {'ceiling':>8} {'climb s':>8} "
f"{'median gap':>11} {'accrued':>8} {'under climb':>12}")
for ring_kb in a.ring:
ring = int(ring_kb * 1024)
for kbps in a.kbps:
fill = (kbps - RC.AUDIO_KBPS) * 1024 / 12
lo, hi, ring_ref, rate_ref = m.paced_sim(rec, ring, fill)
ceiling = int(hi.max())
accrued, per_rec = slack_model(med, kbps, wire, mean_rec)
climb = ceiling * per_rec
under = sum(1 for g in gs if g < climb)
print(f" {ring_kb:>8.0f} {kbps:>7.1f} {ceiling:>8} "
f"{climb:>8.2f} {med:>11.3f} {accrued:>8.2f} "
f"{f'{under}/{len(gs)}':>12} {100*under/len(gs):.0f}%")
# What a branch costs when the gap before it bought nothing. The ring
# is empty after a seek and the decoder is released at the prefill depth
# (55.4's shipped policy is 2 records), so this is the stall the player
# eats every time -- not the climb to the ceiling, which is what it
# needs in order to TOLERATE the next one. A scene change additionally
# needs its header before frame 0; 22_scene_load.py prices that case
# properly, clocks included.
PREFILL_REC, HDR_B = 2, 6164
print()
print(f" A branch taken on an empty ring, at the shipped prefill of "
f"{PREFILL_REC} records:")
for kbps in a.kbps:
b = PREFILL_REC * mean_rec
ms = b / (kbps * 1024) * 1000
hms = (b + HDR_B) / (kbps * 1024) * 1000
print(f" {kbps:>7.1f} KB/s: {ms:>7.1f} ms "
f"({ms/(1000/12):.2f} frame slots), and {hms:>7.1f} ms "
f"({hms/(1000/12):.2f}) if it is a scene change carrying "
f"{HDR_B:,} header bytes")
print()
print(" 'climb s' is 51.3's: seconds of play to refill from empty to")
print(" the ceiling. 'under climb' is how many of this game's own")
print(" branch points arrive sooner than that, i.e. are reached with")
print(" LESS lookahead than the one before them. The worst gap is")
print(f" {worst:.3f} s and buys nothing at any rate in this table.")
else:
print(f"\n{a.container}: MISSING -- rate half skipped")
# ---- the cross-check, and what it is worth
print()
print("=== the second table, and why it is not a second transcription")
cross, meta = doc.get("crosscheck"), doc.get("crosscheck_meta")
if not cross:
print(" none in this table -- the import ran without it")
return 0
print(f" {meta['chapters']} chapters, {meta['scenes_mapped']} scenes")
print(f" PROVENANCE: {meta['provenance']}")
print(" FINDINGS 16 planned to diff two INDEPENDENT transcriptions to")
print(" catch transcription errors. There is only one transcription.")
print(" This diff catches CONVERSION errors and nothing more.")
FRAME_MS = 1000.0 / LD_FPS
offs, durs, same, diff, missing = [], [], 0, 0, 0
renames, inputs_differ, examples = 0, [], []
for scene, seqs in sorted(cross.items()):
for seq, ch in sorted(seqs.items()):
n = nodes.get(f"{scene}.{seq}")
if n is None:
missing += 1
continue
if n.seeks:
offs.append(ch["start_ms"] - n.start)
durs.append(((ch["end_ms"] - ch["start_ms"]) - n.timeout_ms,
ch["chapter"]))
ce = sorted((x["input"], x["next"]) for x in ch["actions"])
de = sorted((x["input"], x["next"]) for x in n.seq["actions"])
if ce == de:
same += 1
continue
diff += 1
ci = sorted(i for i, _ in ce)
di = sorted(i for i, _ in de)
if ci != di:
inputs_differ.append((ch["chapter"], ci, di))
else:
renames += 1
if len(examples) < 3:
examples.append((ch["chapter"], ce, de))
if offs:
offs.sort()
print(f" START TIMES are on different timelines and do not compare: "
f"{len(offs)} seeking chapters,")
print(f" offset spread {min(offs)/1000:,.1f} s .. "
f"{max(offs)/1000:,.1f} s, median {offs[len(offs)//2]/1000:,.1f} s"
f" -- not a constant, and not even one sign.")
if durs:
dd = sorted(x for x, _ in durs)
agree = sum(1 for x in dd if abs(x) <= FRAME_MS)
print(f" DURATIONS compare (offset-invariant): {agree}/{len(dd)} "
f"within one frame of the medium ({100*agree/len(dd):.1f}%), "
f"median {dd[len(dd)//2]:,.0f} ms")
for x, ch in sorted(durs, key=lambda x: -abs(x[0]))[:3]:
print(f" widest {ch}: {x/1000:+.3f} s")
if same + diff:
print(f" BRANCH STRUCTURE compares: {same}/{same+diff} chapters have "
f"the identical set of (input -> target) edges "
f"({100*same/(same+diff):.1f}%)")
if diff:
print(f" {renames} of the {diff} differ only in what a target "
f"sequence is NAMED")
DIAG = ("upleft", "upright", "downleft", "downright")
lost = [ch for ch, ci, di in inputs_differ
if set(di) - set(ci) and all(x in DIAG for x in set(di) - set(ci))]
print(f" {len(inputs_differ)} differ in the INPUT SET, and "
f"{len(lost)} of those are the other table dropping the arcade's")
print(" DIAGONALS: a controller decision, not a transcription "
"difference.")
for ch, ci, di in inputs_differ:
if ch not in lost:
print(f" the remaining one: {ch}, {ci} vs {di}")
for ch, ce, de in examples:
print(f" e.g. {ch}\n cross {ce}\n graph {de}")
if missing:
print(f" {missing} chapters have no sequence in the graph "
f"(the other project added them)")
return 0
if __name__ == "__main__":
sys.exit(main())
tools/analysis/26_sector_align.py
+150
View File
@@ -0,0 +1,150 @@
"""A record is not a sector: what the mismatch costs, three ways (P4b, 58.3).
src/player/ring.i asks the transport for a RECORD -- a byte offset into the
scene's frame stream and a length, both 4-byte aligned because that is what
`move.l (a0)+` needs (28.3) and neither of them a multiple of 512. A SCSI
target answers in 512 B BLOCKS. On the gate container 117 of 120 records start
part way into a sector, so something has to reconcile the two, and the three
ways of doing it are not close.
WHY IT IS NOT AN IMPLEMENTATION DETAIL. The bytes on either side of a record in
the stream belong to OTHER records -- ones the decoder may still be reading --
and the block loop walks a0 with no bounds check at all (49.2). So a transport
that reads whole sectors straight into the ring does not waste 500 bytes, it
CORRUPTS the neighbours, and the symptom is wrong pixels rather than a fault.
A. WINDOWED PIO. Read the sectors the record lies in, store only the record.
src/player/scsi.i does this and it is what FINDINGS 58 measured. It costs
nothing in clocks -- the CPU is touching every byte anyway -- and it costs
the extra sectors on the wire. It CANNOT be done by a DMAC: a channel
writes a contiguous run to a contiguous address and cannot be told to drop
the first 300 bytes.
B. BOUNCE BUFFER. Let the DMAC write whole sectors somewhere else, then copy
the record into the ring. Works under DMA, and costs a copy of every
delivered byte -- which is precisely the cost `aligned` was chosen over
`split` to avoid (49.3, 19_ring_stream.py).
C. SECTOR-ALIGNED RECORDS. Pad each record up to 512 in the container
instead of up to 4. Costs bytes on the disc and in every delivery, and
nothing else at all; the transport becomes a whole-sector read into the
ring with no window and no copy. It is a CONTAINER change -- a re-encode
and a re-measurement of every constant fitted to the gate container, which
is the class of change ROADMAP already has bundled with P2's other half.
python3 tools/analysis/26_sector_align.py <in.dlx> [--ring KB]
No rate is taken and none is needed: every figure here is a fraction of the
delivered bytes or a count of clocks, and both are rate-free. What a given
delivery rate does with them is 15_bus_occupancy.py's question.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
from dlx import DLX
SECTOR = 512
CPUHZ = 10_000_000
# 5.0 clocks/byte, and it is 19_ring_stream.py's constant rather than a new one:
# a 68000 `move.l (a0)+,(a1)+` moves 4 bytes in 20 clocks on a 16-bit bus. It
# is the OPTIMISTIC figure there and it is the optimistic figure here.
COPY_CLK_PER_BYTE = 5.0
# The windowed PIO loop in src/player/scsi.i, from the 68000's cycle table:
# 12 move.l #SC_PATIENCE,d3 patience reload
# 16 move.b SC_SSTS,d0 (xxx).L -> Dn
# 10 btst #0,d0
# 10 beq.s taken
# 20 move.b SC_DREG,(a1)+ (xxx).L -> (An)+
# 8 subq.l #1,d7
# 10 bne.s taken
# FINDINGS 58.2 measured 87.28 clocks per delivered byte against this loop's 86
# plus 1.15 for the dropped window bytes -- 0.2% apart, which is what says the
# cost is the instruction stream and not MAME's device model.
PIO_CLK_PER_BYTE = 86.0
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--ring", type=int, default=256, help="ring size in KB")
a = ap.parse_args()
d = DLX(a.container)
# The disc layout the 68000 walks: [u32 len][body], each record padded up to the
# container's own alignment -- 4 on DLX2/3/4, 512 on DLX5. Exactly
# tools/bench/prep_stream.py's, and it comes from the reader rather than from a
# second copy of the rule here, so pointing this tool at a DLX5 container asks
# it the RIGHT question: what does the mismatch still cost once the container
# has been changed to remove it? (The answer had better be nothing.)
off, recs = 0, []
for ln in d.record_lengths():
recs.append((off, ln))
off += ln
# The DENOMINATOR is the record bytes the decoder actually reads -- [u32 len]
# plus payload -- and NOT the padded length, because on a DLX5 container the
# padding IS the cost being measured. Scoring against the padded length would
# make an already-aligned container report +0.00% and look free.
payload = sum(4 + n for _, n in d.frames)
nfr = len(recs)
budget = CPUHZ / d.fps
print(f"{a.container}: {nfr} records, {payload:,} B, {d.fps} fps")
print(f" mean record {payload/nfr:,.0f} B; a {d.fps} fps frame is "
f"{budget:,.0f} clocks")
aligned0 = sum(1 for o, _ in recs if o % SECTOR == 0)
print(f" records that already start on a sector boundary: {aligned0}/{nfr}")
print()
# ---- A. windowed PIO: the sectors the record lies in, and only the record kept
wire_a = sum(((o % SECTOR) + ln + SECTOR - 1) // SECTOR for o, ln in recs) * SECTOR
drop_a = wire_a - payload
print("A. WINDOWED PIO (src/player/scsi.i, what FINDINGS 58 ran)")
print(f" wire {wire_a:,} B for {payload:,} B of record "
f"= +{100*drop_a/payload:.2f}%")
print(f" clocks {PIO_CLK_PER_BYTE:.0f}/B on EVERY byte off the FIFO, "
f"dropped ones included:")
print(f" {PIO_CLK_PER_BYTE*wire_a/nfr:,.0f} clk/frame "
f"= {100*PIO_CLK_PER_BYTE*wire_a/nfr/budget:.0f}% of the frame")
print( " and it does not survive the move to the DMAC at all: a channel "
"cannot drop bytes.")
print()
# ---- B. bounce buffer: DMA whole sectors elsewhere, copy the record in
print("B. BOUNCE BUFFER (whole sectors by DMA, then a copy)")
print(f" wire {wire_a:,} B, the same +{100*drop_a/payload:.2f}% -- the "
f"command is identical")
print(f" clocks {COPY_CLK_PER_BYTE:g}/B of copy on every DELIVERED byte, "
f"on top of whatever W the")
print(f" channel steals: {COPY_CLK_PER_BYTE*payload/nfr:,.0f} clk/frame "
f"= {100*COPY_CLK_PER_BYTE*payload/nfr/budget:.1f}% of the frame")
print( " which is the cost `aligned` was chosen over `split` to avoid "
"(49.3), arriving")
print( " by a different door and on every byte instead of on a wrap.")
print()
# ---- C. sector-aligned records in the container
cur, pad = 0, 0
for _, ln in recs:
if cur % SECTOR:
pad += SECTOR - (cur % SECTOR)
cur += SECTOR - (cur % SECTOR)
cur += ln
print("C. SECTOR-ALIGNED RECORDS (a container change; a re-encode)"
+ (" -- THIS CONTAINER ALREADY IS ONE" if d.sector_aligned else ""))
print(f" wire {cur:,} B for {payload:,} B of record = "
f"+{100*(cur-payload)/payload:.2f}%")
print( " clocks ZERO: the read is a whole-sector read straight into the "
"ring, no window,")
print( " no copy, and the DMAC can do it.")
print()
ringsz = a.ring * 1024
print(f" VERDICT, in the currency this project prices delivery in. C is "
f"cheaper on the wire")
print(f" than A and B by {100*(wire_a-cur)/payload:.2f} points of the payload "
f"({wire_a-cur:,} B on this scene),")
print(f" and it is the only one of the three a DMA channel can run without a "
f"copy. What it")
print(f" costs is a container revision and the re-measurement that comes with "
f"one.")
maxrec = max(ln for _, ln in recs)
maxpad = maxrec + (-maxrec) % SECTOR
print(f" It also grows the largest record from {maxrec:,} to {maxpad:,} B, "
f"which a {a.ring} KB")
print(f" ring still holds {ringsz//maxpad} times over.")
tools/analysis/27_dmac_config.py
+111
View File
@@ -0,0 +1,111 @@
#!/usr/bin/env python3
"""What the PLAYER programs into the DMAC -- read out of the assembler source.
python3 tools/analysis/27_dmac_config.py [src/player/dma.i]
ROADMAP P4a asks for a DMAC configuration that HOLDS THE BUS, and the whole
weight of the claim is in four register bytes. tools/analysis/21_iplrom_dmac.py
already reads the IPL ROM's four channels the same way, out of the shipping
image, and found Sharp's own disk channel at 16..19 clocks a byte (FINDINGS
52.5) -- above the entire bracket this project costs P4 in. This is the other
half of that comparison: the same MC68450 field tables (tools/analysis/
mc68450.py, one copy) applied to the bytes src/player/dma.i actually programs.
IT PARSES THE SOURCE RATHER THAN RESTATING IT. A constant typed into this file
would be a claim about the player that the player could quietly stop honouring;
the equates are read out of src/player/dma.i, so a change there changes what is
printed here and a mismatch between the two is not expressible.
IT IS A GATE. Each configuration is checked against what it is FOR -- held
must decode as a mode that keeps the bus, stealing must decode as one that does
not -- and a disagreement exits non-zero rather than printing a paragraph.
WHAT IT IS NOT: a rate. Nothing here is a measurement of anything. It says
which mode the player asks the chip for; tools/bench/dma_run.sh shows the
machine doing it, and `W` -- the clocks it costs on real silicon -- remains the
project's largest open number (ROADMAP B1/B3).
"""
import sys, os, re
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from mc68450 import dcr, ocr, scr, XRM, DTYP, REQG
src = sys.argv[1] if len(sys.argv) > 1 else "src/player/dma.i"
if not os.path.exists(src):
sys.exit(f"missing {src} -- run from the repo root.")
text = open(src).read()
def equ(name):
m = re.search(rf"^{name}\s*=\s*\$([0-9A-Fa-f]+)", text, re.M)
if not m:
sys.exit(f"{src} no longer defines {name}. This script reads the "
f"player's own equates; it does not keep a copy of them.")
return int(m.group(1), 16)
# The SCR the channel is given is written inline rather than equated, because it
# is the same for both configurations and there is nothing to choose about it.
m = re.search(r"move\.b\s+#\$([0-9A-Fa-f]+),DM_SCR", text)
if not m:
sys.exit(f"{src} no longer writes DM_SCR with a literal.")
SCR = int(m.group(1), 16)
CFG = [("BUS HELD", "DM_HELD_DCR", "DM_HELD_OCR"),
("CYCLE STEALING", "DM_STEAL_DCR", "DM_STEAL_OCR")]
print(f"WHAT src/player/dma.i PROGRAMS -- decoded from {src}\n")
bad = 0
for label, dn, on in CFG:
D, O = equ(dn), equ(on)
print(f" {label} DCR = ${D:02X} OCR = ${O:02X} SCR = ${SCR:02X}")
for line in dcr(D):
print(f" {line}")
for line in ocr(O):
print(f" {line}")
for line in scr(SCR):
print(f" {line}")
holds = (D >> 6 & 3) in (0, 3) # burst, or cycle steal WITH hold
dual = (D >> 4 & 3) in (0, 1)
tomem = bool(O & 0x80)
checks = [
(dual, "DTYP must be explicitly addressed: only channel 0 has device "
"callbacks in this machine, so an implicit-address DTYP on "
"channel 1 falls through to the dual-address path anyway"),
(tomem, "OCR DIR must be device -> memory; this is a READ"),
((O >> 4 & 3) == 0, "OCR SIZE must be byte: the SPC's port is 8 bits"),
((O >> 2 & 3) == 0, "OCR CHAIN must be none until P5a picks a chaining "
"scheme for the two-deep request queue (FINDINGS 55.3)"),
((SCR & 3) == 0, "SCR DAC must not count: the device address is a "
"REGISTER at $EA0015 and must not walk off it"),
((SCR >> 2 & 3) == 1, "SCR MAC must increment: the record is contiguous"),
((O & 3) in (0, 1), "OCR REQG must be an AUTO-request mode: the "
"expansion slot has no request line to the DMAC in "
"this machine, so external request cannot be run"),
]
if label == "BUS HELD":
checks.append((holds, "the held configuration must decode as a mode "
"that KEEPS the bus between operands"))
checks.append(((O & 3) == 1, "and as max-rate auto-request: MAME models "
"a held bus only for burst + REQG 01"))
else:
checks.append((not holds, "the stealing configuration must decode as a "
"mode that RELEASES the bus between operands "
"-- otherwise the two have no contrast"))
for ok, why in checks:
if not ok:
print(f" FAIL: {why}")
bad += 1
print()
print("""AGAINST THE MACHINE'S OWN DISK CHANNEL (21_iplrom_dmac.py, FINDINGS 52.5)
IPL ROM ch1, SASI DCR $80 OCR $B2 dual address, 8-bit port, cycle steal
WITHOUT hold, EXTERNAL request
-> a full arbitration per byte, 16..19
player, held DCR $00 OCR $81 dual address, 8-bit port, BURST,
auto-request at max rate
-> the ladder's dual-address held row, 9
Sharp's own configuration and the player's differ in exactly the field that
decides the project. That is 52.5's finding read the other way round: a cheaper
configuration IS reachable for an explicitly-addressed 8-bit port, and what it
costs on real silicon is still ROADMAP B3's question and not this file's.""")
sys.exit(1 if bad else 0)
tools/analysis/28_autorequest_cost.py
+172
View File
@@ -0,0 +1,172 @@
#!/usr/bin/env python3
"""What AUTO-REQUEST DMA costs the 68000, when there is no request line.
python3 tools/analysis/28_autorequest_cost.py --kbps 460 [--record 37405]
WHY THIS EXISTS. The project's per-byte ladder -- W = 5 single-address held, 9
dual held, 12 single arbitrated, 16..19 dual arbitrated (FINDINGS 42.4, 52.5) --
prices a transfer that the DEVICE asks for: one external request, one operand,
a known number of stolen clocks per delivered byte. Session 27 found that the
CZ-6BS1 as MAME models it has NO REQUEST LINE to the DMAC at all (FINDINGS
59.2): the card's flow control is DTACK, and every configuration that can be run
against it is AUTO-REQUEST, where the channel transfers because its own counter
says so and not because a byte has arrived.
THAT CHANGES THE CURRENCY, and it is the reason this file is not a line in
another one. An externally requested transfer is charged PER DELIVERED BYTE.
An auto-requested one is charged PER UNIT OF TIME THE CHANNEL IS ACTIVE, because
the channel has no way to know the device is not ready: it takes its allotted
share of the bus and spends it whether or not a byte comes back. So the cost of
delivering a record depends on HOW LONG THE RECORD TAKES TO ARRIVE -- i.e. on
the delivery rate, the figure this tree deliberately has no default for (FINDINGS
50) -- and the tool REQUIRES one rather than assuming it.
SOURCED: MC68450 Direct Memory Access Controller, Motorola, Jul 1989
(bitsavers), sections 3.8 and 5.2.3.3, the same document buscost.py's transfer
timings come from. Section 5.2.3.3.1: under maximum-rate auto-request "all
operands in the data block will be transferred in one burst, so that the DMAC
will use 100% of the available bus bandwidth" -- which is the datasheet saying,
in its own words, what session 27 measured MAME's model doing when it HALTED the
68000 for the whole data phase (FINDINGS 59.1).
THE ONE LOAD-BEARING ASSUMPTION, stated because the whole table rests on it:
that the channel SPENDS its allotted share whether or not the device has a byte.
Under auto-request a request is pending until MTC is exhausted, so the DMAC
takes the bus during every burst window it is entitled to; when the device is
not ready the cycle is stretched by wait states (a real CZ-6BS1 negating DTACK)
or retried later (MAME's model discards the operand), and either way the window
is gone from the CPU's point of view. If a real card instead lets the DMAC off
the bus early when no byte is there, these figures are UPPER BOUNDS. That is a
board question and it is ROADMAP B3's.
NOT A MEASUREMENT. Every figure below is arithmetic over datasheet constants
and an explicit rate. `W` is still unmeasured and still wants a board.
"""
import sys, os, argparse
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from buscost import DMA_DUAL_BYTE_CLK, DMA_READ_CLK, DMA_WRITE_CLK
CPU_HZ = 10_000_000.0 # the X68000 the whole tree is costed against
FPS = 12.0
ap = argparse.ArgumentParser()
ap.add_argument("--kbps", type=float, required=True,
help="delivery rate in KB/s. REQUIRED: this tree has no default "
"rate and the whole answer scales with it (FINDINGS 50).")
ap.add_argument("--record", type=int, default=37405,
help="mean record size in bytes (default: the gate container's)")
a = ap.parse_args()
RATE = a.kbps * 1024.0
# --- 3.8 GENERAL CONTROL REGISTER, decoded from the formulas in 5.2.3.3.2 ---
# burst time = 2^(BT + 4) clocks
# sample period = 2^(BT + BR + 5) clocks
# DMAC's share = 2^-(BR + 1)
# and Table 5-3 prints all sixteen combinations, so the formulas are GATED
# against the table rather than trusted.
TABLE = { # (BR, BT): (burst, MPU period, share, sample period)
(0,0):(16,16,.5,32), (0,1):(32,32,.5,64), (0,2):(64,64,.5,128), (0,3):(128,128,.5,256),
(1,0):(16,48,.25,64), (1,1):(32,96,.25,128), (1,2):(64,192,.25,256),(1,3):(128,384,.25,512),
(2,0):(16,112,.125,128),(2,1):(32,224,.125,256),(2,2):(64,448,.125,512),(2,3):(128,896,.125,1024),
(3,0):(16,240,.0625,256),(3,1):(32,480,.0625,512),(3,2):(64,960,.0625,1024),
(3,3):(128,1920,.0625,2048),
}
bad = 0
for (br, bt), (burst, mpu, share, sample) in sorted(TABLE.items()):
f_burst, f_sample, f_share = 2**(bt+4), 2**(bt+br+5), 2.0**-(br+1)
for got, want, what in ((f_burst, burst, "burst time"),
(f_sample, sample, "sample period"),
(f_share, share, "bandwidth share"),
(f_sample - f_burst, mpu, "MPU period")):
if got != want:
print(f" FAIL BR={br:02b} BT={bt:02b} {what}: formula {got}, "
f"Table 5-3 {want}")
bad += 1
if bad:
sys.exit(f"\n{bad} disagreements between 5.2.3.3.2's formulas and Table 5-3. "
"Everything below\nis those formulas, so it is not printed.")
print(f"MC68450 5.2.3.3.2's formulas reproduce all 16 rows of Table 5-3.\n")
BYTE_CLK = DMA_DUAL_BYTE_CLK # dual address, 8-bit port: a 4-clock read of
# $EA0015 and a 5-clock write to the ring
frame_clk = CPU_HZ / FPS
wire_s = a.record / RATE # how long the record takes to land
wire_clk = wire_s * CPU_HZ # ...in 68000 clocks
per_byte_wire = wire_clk / a.record # clocks of wall time per byte
print(f"THE RECORD: {a.record:,} B at {a.kbps:g} KB/s = {wire_s*1000:.2f} ms "
f"= {wire_clk:,.0f} clocks = {100*wire_clk/frame_clk:.1f}% of a "
f"{FPS:g} fps frame")
print(f" one byte of WIRE TIME is {per_byte_wire:.2f} clocks; one byte of DMAC "
f"WORK is {BYTE_CLK} ({DMA_READ_CLK} read + {DMA_WRITE_CLK} write, "
f"buscost.py)\n")
print("REQG 01, AUTO-REQUEST AT MAXIMUM RATE -- what session 27 demonstrated")
print(f" The channel holds the bus until MTC is exhausted (5.2.3.3.1: 100% of "
f"the\n bandwidth), so the CPU gets NOTHING for the whole delivery:")
print(f" cost to the 68000 = the whole {100*wire_clk/frame_clk:.1f}% of a "
f"frame, or {per_byte_wire:.2f} clk/B")
print(f" It is the cheapest configuration per BYTE MOVED and the dearest per "
f"byte\n DELIVERED, and the gap between those is the device's own "
f"slowness:\n {BYTE_CLK} clocks of work in {per_byte_wire:.1f} clocks "
f"of waiting = {100*BYTE_CLK/per_byte_wire:.1f}% of the held bus does "
f"anything.\n")
print("REQG 00, LIMITED-RATE AUTO-REQUEST -- the lever the GCR actually gives")
print(" The DMAC takes its programmed share of the bus and spends it whether "
"or not\n a byte is there, so the CPU pays the SHARE for the WHOLE "
"delivery -- and the\n share must also be big enough to carry the rate. "
"Both, or it does not fit.\n")
print(" BR share sustains clk/B charged % of a frame fits "
f"{a.kbps:g} KB/s?")
fits_any = []
for br in range(4):
burst, mpu, share, sample = TABLE[(br, 3)] # BT=11, the longest burst
# bytes the channel can move inside one burst window, and how often that
# window comes round
bytes_per_burst = burst // BYTE_CLK
sustains = bytes_per_burst * CPU_HZ / sample
charged = share * per_byte_wire # clocks the CPU loses per
# DELIVERED byte
pct = 100 * share * wire_clk / frame_clk
ok = sustains >= RATE
if ok:
fits_any.append((br, share, charged, pct))
print(f" {br:02b} {share*100:5.2f}% {sustains/1024:7.1f} KB/s "
f"{charged:9.2f} {pct:8.1f}% {'yes' if ok else 'NO'}")
print(f"\n (BT = 11 throughout: the longest burst, 128 clocks, which is the "
f"most\n favourable row -- a shorter burst moves fewer bytes per window "
f"at the same\n share and sustains proportionally less.)")
if not fits_any:
print(f"\n NOTHING FITS. At {a.kbps:g} KB/s no limited-rate share can "
f"carry the record,\n so the only auto-request configuration that "
f"delivers is maximum rate --\n and that one stops the CPU for the "
f"whole {100*wire_clk/frame_clk:.1f}% of a frame the record takes.")
else:
br, share, charged, pct = fits_any[0]
print(f"\n CHEAPEST THAT FITS: BR = {br:02b}, {share*100:g}% of the bus, "
f"{charged:.2f} clk/B charged to the\n 68000 -- {pct:.1f}% of a frame "
f"per record.")
print(f" Against the ladder: W=5 held costs {5*a.record/frame_clk*100:.1f}%, "
f"W=9 dual held {9*a.record/frame_clk*100:.1f}%,\n W=19 the IPL ROM's "
f"own {19*a.record/frame_clk*100:.1f}%, and PIO measured "
f"{87.28*a.record/frame_clk*100:.1f}% (FINDINGS 58.2).")
print(f"""
WHAT THIS SETTLES, AND WHAT IT DOES NOT
1. AUTO-REQUEST IS CHARGED BY TIME, NOT BY BYTE. Every W in this project is
clocks per DELIVERED byte, which presumes the device asks. With no request
line the channel spends its share of the bus at a rate it was told, so the
record's cost scales with how long the disc takes -- halve the delivery rate
and the CPU cost of the same record DOUBLES. No W does that.
2. THE GCR IS A DESIGN LEVER NOBODY HAD NAMED. BT and BR are two bits each and
they set what fraction of the bus the player gives away. That is the same
kind of choice as `aligned` vs `split` and it belongs in the same list.
3. IT IS STILL NOT A MEASUREMENT. These are datasheet constants and an explicit
rate. Whether the real CZ-6BS1 drives #EXREQ (pin B36 exists on the slot, and
MAME's model simply does not connect it) is ROADMAP B3's question, and if it
does, the ladder applies and this file is the fallback rather than the plan.""")
tools/analysis/29_packed_player.py
+405
View File
@@ -0,0 +1,405 @@
#!/usr/bin/env python3
"""The DECODER-FREE PACKED player, priced against the measured cost model.
python3 tools/analysis/29_packed_player.py [container.dlx] [--kbps R]
THE QUESTION, and why it is being asked again. FINDINGS 44.7 removed the codec
and asked what a player that just puts literal frames on screen would cost. It
answered "it fits the clocks and dies on the medium": 1,152 KB/s and 1.61 GB,
because 256-colour GVRAM's default write path throws away the high byte of every
word and a picture byte therefore costs two disc bytes. 46.5/47.1 then found
the off switch -- CRTC R20 bit 11 -- and 47.2 built the layout and rendered it
pixel-exactly on both emulators at 1.0 B/pixel. 47.5 re-derived the budget on
that and withdrew 44.7's conclusion CONDITIONALLY.
Everything in 47.5 is arithmetic over a cost model that has since been REPLACED.
When it was written the transport was an unmeasured `c`; sessions 25b-28 put the
transport on the 68000 and measured it (58.2: PIO is 87.28 clk/B), put it on the
DMAC and bounded it (59.2: this machine can run dual-address only, and a
dual-address byte has a 9 clk/B FLOOR), and re-derived what a frame can afford
(59.7/60.7: 6.69 clk/B on the gate container). 47.6.1 also filed the CPU paint
cost as an ASSUMPTION -- "the `movem` shape of the packed writes is an
assumption", no clock in 47.5 measured.
So this tool re-asks 44.7's question with:
* the paint MEASURED, not assumed -- tools/bench/blit.s V8 is V1 with 128
words a row instead of 256, and tools/bench/blit.lua times it next to V1,
V2 and V3 in the same run, so the packed number is quoted against a
variant whose value (53.6%) is a session-9 result that has not moved;
* the transport swept over the SAME `W` ladder 15_bus_occupancy.py uses,
every rung of it sourced or measured (buscost.py);
* the audio DMA charged, at the rate the IPL ROM's own channel-3 setup
implies (21_iplrom_dmac.py) -- 60.x's rule that a budget debits I/O;
* and the wire and the volume stated for each, because 44.7's answer was
never about clocks.
WHAT IT DOES NOT DO. It does not settle 47.4 -- whether buffer mode BLANKS the
graphics layer, which MAME asserts and px68k is silent about (48.1), and which
needs a real board. It PRICES both branches instead, and the blanking section
is where the measured paint earns its keep: the black interval is the paint, and
until now the paint was a range read off an unpacked measurement ("~27% to ~54%",
48.3) rather than a number.
"""
import sys, os, re, argparse, csv
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
from dlx import DLX
import buscost as B
CPUHZ = 10e6 # stock X68000, MAME 0.277 x68k.cpp:1133
GAME_S = 22.8 * 60 # the full-disc survey's runtime (ROADMAP C1)
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/rc_fr_singe_scsi_span.dlx",
help="the CODEC baseline this is compared against")
ap.add_argument("--csv", default="tmp/c68k_frames.csv",
help="per-frame C68K measurement of that container")
ap.add_argument("--blit-log", default="tmp/blit_v8.log",
help="tools/bench/blit.lua's log -- where the MEASURED paint "
"costs are read from. No defaults are compiled in.")
ap.add_argument("--kbps", type=float, default=None,
help="a delivery rate to score the wire against. OPTIONAL and "
"there is no default (FINDINGS 50).")
a = ap.parse_args()
# --- the measured paint, read out of the run's own log ---------------------
# NOT transcribed into this file. A constant copied out of a log is a constant
# that stops tracking the log, and this project has been caught by a stale
# number twice (60.8). If the log is not there the tool refuses rather than
# substituting a plausible one.
if not os.path.exists(a.blit_log):
sys.exit(f"missing {a.blit_log} -- run tools/bench/blit.lua first:\n"
f" cd tmp && mame x68000 -bios ipl10 -ramsize 2M -video soft "
f"-window -sound none -nothrottle -plugins \\\n"
f" -autoboot_script ../tools/bench/blit.lua -seconds_to_run 60")
blit = {}
for line in open(a.blit_log, errors="replace"):
m = re.search(r"V(\d+)\s+(\d+) cyc", line)
if m:
blit[int(m.group(1))] = int(m.group(2))
for v in (1, 2, 3, 4, 8, 9, 10):
if v not in blit:
sys.exit(f"{a.blit_log} has no V{v} result -- the summary is incomplete, "
f"so the run did not finish and nothing here can be quoted.")
# --- the codec baseline: the container, and its MEASURED decode ------------
d = DLX(a.container)
FPS = d.fps
FRAME_CLK = CPUHZ / FPS
meas = {}
if os.path.exists(a.csv):
for r in csv.DictReader(open(a.csv)):
meas[int(r["frame"])] = int(r["cycles"])
if not meas:
sys.exit(f"missing {a.csv} -- the codec row's decode term is MEASURED and "
f"there is no derived stand-in for it.")
NF = max(meas) + 1
codec_decode = np.mean([meas[f] for f in range(NF)])
codec_bpf = sum(d.record_lengths()[:NF]) / NF # the PADDED record (60.7)
# --- geometry, which is where the decoder-free rows come from -------------
W_PX, H_PX = d.W, d.H
NPX = W_PX * H_PX
UNPACKED_BPF = NPX * 2 # one pixel per word, high byte discarded
PACKED_BPF = NPX * 1 # R20 bit 11 + page scroll (47.2, measured)
aud_bpf = B.ADPCM_BYTES_PER_S / FPS
AUD_CLK = aud_bpf * B.ADPCM_CLK_BYTE_BEST # best case, so every row is
# the optimistic end
# A device->GVRAM channel cannot walk a 1024-byte line stride inside one
# transfer: it writes a contiguous run. 192 rows therefore need 192 array-chain
# entries -- and SESSION 29 RAN THAT, off the disc, through src/player/dma.i's
# DM_BARV/DM_BTCV: eight rows at the 1024 B stride landed from ONE channel start
# with the CPU halted throughout (tools/bench/dma_run.sh, `[chain]`). So the
# MECHANISM is demonstrated and the CPU does not restart the channel per row.
# The COST is still datasheet arithmetic -- 36 clocks an entry, Fig 4-25 sheet 1,
# buscost.DMA_CHAIN_CLK -- because MAME's DMAC runs on wall-clock attotimes and
# cannot be asked what anything costs (42.5).
CHAIN_CLK = H_PX * B.DMA_CHAIN_CLK
print(f"""{a.container}: {NF} frames of {W_PX}x{H_PX} at {FPS:g} fps
frame slot on a 10 MHz 68000: {FRAME_CLK:,.0f} clocks
paint costs MEASURED by tools/bench/blit.lua, read from {a.blit_log}:
V1 unpacked movem blit {blit[1]:>9,} clk {100*blit[1]/FRAME_CLK:5.1f}% (96 KB read + 96 KB write)
V2 byte-source expansion {blit[2]:>9,} clk {100*blit[2]/FRAME_CLK:5.1f}% (48 KB read + 96 KB write)
V3 write-only floor {blit[3]:>9,} clk {100*blit[3]/FRAME_CLK:5.1f}% (no source read at all)
V8 PACKED movem blit {blit[8]:>9,} clk {100*blit[8]/FRAME_CLK:5.1f}% (48 KB read + 48 KB write)
V8 is {100*blit[8]/blit[1]:.1f}% of V1 and {100*blit[8]/blit[3]:.1f}% of V3 -- so PACKED PAINT COSTS WHAT THE
UNPACKED PATH PAYS TO WRITE ALONE, with its source read thrown in free.
It is not exactly half of V1 because the 192-row loop does not halve with
the words: per word V1 is {blit[1]/(NPX):.3f} clk and V8 is {blit[8]/(NPX//2):.3f}.""")
# --- the architectures ----------------------------------------------------
# Each is (label, bytes on the wire per frame, CPU clocks per frame that are
# NOT the transport, and whether the transport lands in GVRAM or in RAM).
ARCH = [
("CODEC, CPU-decoded (the shipping design)", codec_bpf, codec_decode, "ring"),
("free / DMAC device->GVRAM / unpacked", UNPACKED_BPF, CHAIN_CLK, "gvram"),
("free / DMAC device->GVRAM / PACKED", PACKED_BPF, CHAIN_CLK, "gvram"),
("free / CPU-painted / unpacked, 2 B/px wire", UNPACKED_BPF, blit[1], "ring"),
("free / CPU-painted / unpacked, 1 B/px wire", PACKED_BPF, blit[2], "ring"),
("free / CPU-painted / PACKED", PACKED_BPF, blit[8], "ring"),
]
LADDER = [
(5.0, "single address, held -- needs a request line (B3)"),
(9.0, "dual address, held -- the FLOOR (59.2/59.7)"),
(12.0, "single address, arbitrated"),
(16.0, "what the ROM programs for SASI, best"),
(19.0, "what the ROM programs for SASI, worst"),
(87.28, "PIO -- MEASURED, 58.2"),
]
print("\n" + "=" * 78)
print("WHAT EACH ARCHITECTURE COSTS A FRAME, over the transport ladder\n")
print(f" audio DMA is charged in every row at {AUD_CLK:,.0f} clk "
f"({100*AUD_CLK/FRAME_CLK:.2f}%), best case.\n")
hdr = f" {'architecture':<44}{'B/frame':>9}" + "".join(f"{f'W={w:g}':>9}" for w, _ in LADDER)
print(hdr)
print(" " + "-" * (len(hdr) - 2))
for label, bpf, cpu, dest in ARCH:
cells = []
for w, _ in LADDER:
tot = bpf * w + cpu + AUD_CLK
pct = 100 * tot / FRAME_CLK
cells.append(f"{pct:>8.1f}%" if pct < 1000 else f"{pct:>8.0f}%")
print(f" {label:<44}{bpf:>9,.0f}" + "".join(cells))
print(f"""
100% is the frame deadline. Every cell is CPU work plus transport plus
best-case audio; none of them overlap, because the 68000 has no cache and a
two-word prefetch queue that empties at once (buscost.DMA_OVERLAPS = False).
THE TWO ROWS THAT MATTER ARE THE FLOOR COLUMN, W=9, because 59.2 found that
the only configurations this machine can be shown to run are dual-address,
and a dual-address byte is a 4-clock read of the device plus a 5-clock write
to memory. Every column left of it is a hardware fact nobody here has.""")
# --- the wire, which is what 44.7 actually died on ------------------------
print("\n" + "=" * 78)
print("THE WIRE AND THE MEDIUM -- 44.7's real objection\n")
print(f" {'architecture':<44}{'B/frame':>9}{'KB/s':>9}{'GB for 22.8 min':>18}")
print(" " + "-" * 78)
seen = set()
for label, bpf, cpu, dest in ARCH:
kbs = bpf * FPS / 1024
gb = bpf * FPS * GAME_S / 1e9
print(f" {label:<44}{bpf:>9,.0f}{kbs:>9.1f}{gb:>18.2f}")
print(f"""
The codec row is the gate container, which is deliberately the heaviest thing
the encoder emits (59.7). The default `need` recipe is 267.9 KB/s and E7's
byte target at the 9 clk/B floor is 327 KB/s (60.7).
SO THE PACKED DECODER-FREE PLAYER ASKS FOR {PACKED_BPF*FPS/1024:.0f} KB/s -- {PACKED_BPF*FPS/1024/327:.2f}x E7's target and
{PACKED_BPF*FPS/1024/(codec_bpf*FPS/1024):.2f}x the gate container -- and it asks for it AT A FIXED RATE. A codec's
bitrate is a lever; a literal frame's is geometry, and there is no scene in
the picture that costs less than another.""")
if a.kbps:
R = a.kbps * 1024
print(f"\n against a supplied {a.kbps:g} KB/s:")
for label, bpf, cpu, dest in ARCH:
need = bpf * FPS
print(f" {label:<44}{'FITS' if need <= R else 'SHORT BY '}"
f"{'' if need <= R else f'{(need-R)/1024:.0f} KB/s'}"
f" ({need/1024:.0f} KB/s wanted)")
# --- 47.4's two branches, priced -----------------------------------------
print("\n" + "=" * 78)
print("IF BUFFER MODE BLANKS THE LAYER (47.4 / 48, MAME's reading)\n")
print(""" R20 bit 11 only has to be SET across the GVRAM writes, so the black
interval is the paint and not the frame -- and which paint depends on where
the transport lands. That asymmetry has not been stated before:\n""")
print(f" {'architecture':<44}{'black interval':>16} {'set for':<14}")
print(" " + "-" * 78)
for label, bpf, cpu, dest in ARCH[1:]:
if dest == "gvram":
# the channel writes GVRAM, so the bit is set for the whole transfer
# DMA rungs only: a PIO transport is not a channel writing GVRAM, so
# 87.28 has no meaning in a device->GVRAM row.
rows = [bpf * w + CHAIN_CLK for w, _ in LADDER if w < 20]
span = f"{100*min(rows)/FRAME_CLK:.0f}%..{100*max(rows)/FRAME_CLK:.0f}%"
note = "the whole DMA"
else:
span = f"{100*cpu/FRAME_CLK:.1f}%"
note = "the blit only"
print(f" {label:<44}{span:>16} {note:<14}")
print(f"""
THE CPU-PAINTED PACKED PATH HAS THE SMALLEST BLACK WINDOW OF ANY OF THEM --
{100*blit[8]/FRAME_CLK:.1f}% -- because its transport lands in RAM, where bit 11 is irrelevant,
and only the {blit[8]:,}-clock blit needs the bit set. The DMAC-direct path,
which is cheaper in clocks at every rung of the ladder, is the one that must
hold the bit across its whole transfer. Under MAME's reading the cheap
architecture is the dark one.
Both are a strobe at the frame rate over the whole picture, and the packed
layout has no page to flip to: both 256-colour pages carry picture, which is
the entire point of it (48.3). {100*blit[8]/FRAME_CLK:.1f}% black at 12 Hz is not a tear.
IF PX68K IS RIGHT AND IT DOES NOT BLANK, every number above stands as
written. Neither emulator is authority and 48.1 is why the prior leans
MAME's way: MAME asserts the semantic twice and deliberately, px68k's
display path never reads the bit at all. That is an assertion against a
silence, not a tie, and it is settled by a board and the two-line probe in
tools/bench/probe_bit11_blank.lua.""")
# --- 47.6.4: does the CODEC survive the packed layout? --------------------
# Open since session 16 and never touched: "under the packed layout a word spans
# two columns 128 apart. Whether the existing codec survives that is untouched."
# There are exactly two ways it could, and blit.s V9 and V10 are them.
sk_blocks = sk_tot = pair_sk = pair_tot = 0
for f in range(NF):
m = d.modes(f).reshape(d.nby, d.nbx)
L, R = m[:, :d.nbx // 2], m[:, d.nbx // 2:]
sk_blocks += int((m == 0).sum()); sk_tot += m.size
pair_sk += int(((L == 0) & (R == 0)).sum()); pair_tot += L.size
paint_now = 1 - sk_blocks / sk_tot
paint_pair = 1 - pair_sk / pair_tot
print("\n" + "=" * 78)
print("CAN THE CODEC BE PACKED TOO? -- 47.6.4, open since session 16\n")
print(f""" A 4x4 block owns four bytes at STRIDE 2 under the packed layout, because
the high bytes of its four words belong to the block 128 columns away. So a
block decoder has two options and neither is free:
{'V4 block order, UNPACKED (the shipping shape)':<52}{blit[4]:>9,} clk {100*blit[4]/FRAME_CLK:5.1f}%
{'V9 block order, PACKED, 16 move.b at stride 2':<52}{blit[9]:>9,} clk {100*blit[9]/FRAME_CLK:5.1f}%
{'V10 block order, PACKED, blocks PAIRED (movem back)':<52}{blit[10]:>9,} clk {100*blit[10]/FRAME_CLK:5.1f}%
V9 IS {100*blit[9]/blit[4]-100:.0f}% DEARER THAN V4, not cheaper. Packing buys a block decoder
nothing on the wire either -- a codeword is already one byte a pixel -- so
that route buys NOTHING and costs {blit[9]-blit[4]:,} clocks a frame to buy it.
V10 halves the paint, and pays for it in the mode map. A pair skips only if
BOTH its blocks skip, and on this container:
SKIP blocks now {100*sk_blocks/sk_tot:5.1f}% painted now {100*paint_now:5.1f}%
SKIP block PAIRS {100*pair_sk/pair_tot:5.1f}% painted paired {100*paint_pair:5.1f}%
So pairing paints {paint_pair/paint_now:.2f}x as many blocks for {blit[10]/blit[4]:.2f}x the paint per block --
{100*(paint_pair/paint_now)*(blit[10]/blit[4])-100:+.0f}% on the clock, and about {100*(paint_pair/paint_now-1):+.0f}% on the BYTES, because a coded
block is bytes in the container whether its half of the pair changed or not.
E7 needs the bytes DOWN {100*(codec_bpf*FPS/1024)/327-100:.0f}%.
SO PACKING BELONGS TO THE LITERAL PLAYER AND ONLY TO IT. 47.6.4 is closed:
the packed layout is not an upgrade the existing codec can take, it is the
thing you get INSTEAD of the codec.""")
# --- the palette, which is where the literal player stops being a compromise --
# 46.3 measured these while pricing the TEXT PLANE and the 256-colour rows were
# only there for scale. They answer a question nobody put to them: a literal
# player has no codebooks, so it is not tied to a scene-wide palette the way the
# codec is (vq.scene_palette exists BECAUSE codewords are indices into it), and
# per-frame palettes become legal. Re-run 18_text_plane_16col.py to reproduce.
PSNR_SHIPPED = 29.19 # docs/STATUS.md, --spans all, c=5, 496.7 KB/s
PSNR_SCENE_256 = 31.33 # 18_text_plane_16col.py, tmp/fr_singe, 120 frames
PSNR_FRAME_256 = 34.08 # the same window, per-frame palettes
PAL_BYTES = 512 # 256 entries x 1 word
pal_bpf = PACKED_BPF + PAL_BYTES
# The palette write, DERIVED from a MEASURED per-word constant: V8 moves a word
# into GVRAM for blit[8]/(NPX//2) clocks and the palette is 256 consecutive
# words at $E82000 in the same movem shape.
pal_clk = 256 * blit[8] / (NPX // 2)
print("\n" + "=" * 78)
print("AND THE PICTURE IS BETTER, WHICH NOBODY HAD ASKED\n")
print(f""" PSNR against the 24-bit source, 18_text_plane_16col.py over the same
120-frame window the whole tree is measured on:
{'shipping container (the codec, as it ships)':<48}{PSNR_SHIPPED:6.2f} dB
{'256 colours, SCENE palette -- the codec CEILING':<48}{PSNR_SCENE_256:6.2f} dB
{'256 colours, PER-FRAME palette':<48}{PSNR_FRAME_256:6.2f} dB
THE MIDDLE ROW IS A CEILING AND NOT A RIVAL. Every codeword the codec emits
is an index INTO the scene palette, so no amount of bitrate takes it past
{PSNR_SCENE_256:.2f} dB; it spends {codec_bpf*FPS/1024:.0f} KB/s to get within {PSNR_SCENE_256-PSNR_SHIPPED:.2f} dB of it.
A LITERAL FRAME HAS NO CODEBOOKS, so the scene palette is not forced on it,
and the bottom row is what it simply IS -- {PSNR_FRAME_256-PSNR_SHIPPED:+.2f} dB on the shipping
container and {PSNR_FRAME_256-PSNR_SCENE_256:+.2f} dB past the ceiling the codec cannot cross.
WHAT THE PER-FRAME PALETTE COSTS:
on the wire {PAL_BYTES} B a frame -> {pal_bpf:,} B, {pal_bpf*FPS/1024:.1f} KB/s (+{100*PAL_BYTES/PACKED_BPF:.1f}%)
in clocks ~{pal_clk:,.0f} ({100*pal_clk/FRAME_CLK:.2f}% of a frame) if the CPU writes it, DERIVED
from V8's measured {blit[8]/(NPX//2):.3f} clk/word in the same movem shape
in colours 254, not 256: the packed layout spends index 0 on the
transparency key and puts black at 255 (47.2,
prep_frame.py --pack-transparent), against --reserve-black's
one entry. The tree has already measured a reserved entry at
0.04 dB (60.3), so this is noise against {PSNR_FRAME_256-PSNR_SHIPPED:+.2f}.
SETTLED IN SESSION 30, AND THE ANSWER IS YES (FINDINGS 62): a channel writes
the palette registers at $E82000 byte-exact, and ONE array-chained start
crosses from those registers into GVRAM -- so the palette IS a 193rd chain
entry and the clocks row above is what the CPU pays only if it does the write
itself. dmagate.s runs 7-9. What that does NOT settle is the board: MAME maps
the palette to palette_device over memory_array, whose write16 is a plain
COMBINE_DATA, so there is no handler that could refuse a byte write and the
model cannot discriminate. ROADMAP B4.
AND THE PSNR FIGURES ARE PIL's MEDIANCUT, not this project's own palette
builder (vq.scene_palette / H.build). The DIRECTION is measured and the
magnitude is about right; if the packed player gets built, re-derive the
per-frame number against the builder that will actually ship it.""")
# --- the answer ----------------------------------------------------------
w9 = 9.0
free_packed_dma = PACKED_BPF * w9 + CHAIN_CLK + AUD_CLK
free_packed_cpu = PACKED_BPF * w9 + blit[8] + AUD_CLK
# ... and the same two rows with the PER-FRAME PALETTE actually charged, which
# is what a player ships. The picture rows above are the comparison against the
# codec and are left alone so the published 55.2% / 81.6% do not drift; these
# are the shipping figures. Session 30 (FINDINGS 62) made the DMAC row's
# version legal: the palette is a 193rd chain ENTRY, so it costs 512 more
# delivered bytes and one more entry rather than 256 CPU word writes.
pal_dma = (PACKED_BPF + PAL_BYTES) * w9 + CHAIN_CLK + B.DMA_CHAIN_CLK + AUD_CLK
pal_cpu = (PACKED_BPF + PAL_BYTES) * w9 + blit[8] + pal_clk + AUD_CLK
codec_9 = codec_bpf * w9 + codec_decode + AUD_CLK
print("\n" + "=" * 78)
print(f"""THE ANSWER, AT THE ONE RUNG THIS MACHINE CAN BE SHOWN TO RUN (W=9)
CODEC, gate container {100*codec_9/FRAME_CLK:6.1f}% of the frame -- DOES NOT FIT
free / DMAC->GVRAM / PACKED {100*free_packed_dma/FRAME_CLK:6.1f}% -- FITS, with {100-100*free_packed_dma/FRAME_CLK:.0f}% to spare
free / CPU-painted / PACKED {100*free_packed_cpu/FRAME_CLK:6.1f}% -- FITS, with {100-100*free_packed_cpu/FRAME_CLK:.0f}% to spare
WITH THE PER-FRAME PALETTE CHARGED, which is what would ship:
DMAC-direct, palette on the CHAIN (62) {100*pal_dma/FRAME_CLK:6.1f}% of the frame, {(PACKED_BPF+PAL_BYTES)*FPS/1024:.0f} KB/s
CPU-painted, palette written by the CPU {100*pal_cpu/FRAME_CLK:6.1f}% of the frame, {(PACKED_BPF+PAL_BYTES)*FPS/1024:.0f} KB/s
The palette costs the same on the WIRE either way -- {PAL_BYTES} B a frame,
+{100*PAL_BYTES/PACKED_BPF:.1f}% -- and the wire is where this design is expensive. The gap
between the two rows is the PAINT, not the palette.
What session 30 bought is smaller than either and is worth stating exactly:
{pal_clk:,.0f} CPU clocks of palette writing replaced by one more chain entry at
{B.DMA_CHAIN_CLK} clocks, a net {100*(pal_clk-B.DMA_CHAIN_CLK)/FRAME_CLK:.2f}% of a frame -- plus the structural half,
which is that the video path then contains no per-frame PAINT at all. The
CPU still issues the READ(10) and starts the channel, and neither of those
is priced anywhere in this tree.
THE DECODER-FREE PACKED PLAYER FITS THE CLOCK BUDGET THAT THE CODEC MISSES.
That is not a small correction to 47.5, it is the reverse of the reason the
codec exists. 44.7 said it in advance and on a different cost model: "the
codec is not there to save CPU -- it is there to save the wire." The
measured model agrees, and now says the CPU side is not merely affordable
but strictly cheaper WITHOUT the codec: at the floor, decoding {codec_bpf:,.0f} bytes
costs {100*(codec_bpf*w9+codec_decode)/FRAME_CLK:.0f}% of a frame and NOT decoding {PACKED_BPF:,} costs {100*(PACKED_BPF*w9+blit[8])/FRAME_CLK:.0f}%.
SO THE QUESTION IS ENTIRELY A MEDIUM QUESTION, and it has two halves:
1. {PACKED_BPF*FPS/1024:.0f} KB/s SUSTAINED, with no lever to pull. ROADMAP B1 is
unmeasured; the 0.7-1.7 MB/s usually quoted for BlueSCSI on an X68000
is folklore with no published benchmark behind it. {PACKED_BPF*FPS/1024:.0f} KB/s sits
inside that range, which is exactly why the range has to be measured
rather than cited. A codec at 327 KB/s survives a slower answer; a
literal frame does not degrade, it drops.
2. {PACKED_BPF*FPS*GAME_S/1e9:.2f} GB for the whole game, against the codec's {codec_bpf*FPS*GAME_S/1e9:.2f} GB at the gate
recipe and ~{327*1024*GAME_S/1e9:.2f} GB at E7's target. That is a packaging fact (C3),
not a performance one.
AND 47.4 STILL SITS OVER ALL OF IT. Everything above assumes the layer is
visible while it is written. If it is not, the packed player is a {100*blit[8]/FRAME_CLK:.0f}% duty
strobe at best and there is no version of it that is merely expensive.""")
tools/analysis/30_packed_container.py
+290
View File
@@ -0,0 +1,290 @@
#!/usr/bin/env python3
"""The PACKED container: does it hold, and what is the picture actually worth?
python3 tools/analysis/30_packed_container.py [packed.dlxp]
[--frames tmp/fr_singe] [--codec tmp/rc_fr_singe_scsi_span.dlx]
ROADMAP K2. Two jobs, and they are different kinds of claim.
1. THE FORMAT HOLDS. A packed record is written into the palette registers and
GVRAM by a DMA channel with no bounds test anywhere -- the channel has no
opinion about what it is copying (FINDINGS 62) -- so "the geometry is right"
is not a tidiness check, it is the whole of the container's correctness.
Round-trip, sector geometry, and the two reserved indices are gated here.
2. THE PICTURE IS RE-DERIVED, and this is the number session 30 asked for.
FINDINGS 61.9 measured the packed player at 34.08 dB against the codec's
29.19 and filed TWO caveats: the quantiser was PIL's free 256-colour
MEDIANCUT rather than this project's builder, and the figure was quoted in
the RGB888 palette domain. Both are paid here:
* `vq.frame_palette` is what ships it -- 254 colours, because the packed
layout spends index 0 on the transparency key and 255 on black (47.2).
* the GRB555+I WORD is charged. A palette entry in a packed record is
already a hardware word; the display renders 5 bits a channel with one
shared LSB (23.3). Every PSNR in this project's encoder is measured
upstream of that, so the codec is charged it here too and the comparison
stays like for like.
And the scene-palette CONTROL is built and scored, because "per-frame
palettes became legal" is the mechanism 61.9 credits and an unrun control is
an assumption. The codec cannot take this row: every codeword it emits is an
index INTO `vq.scene_palette`, so 31.33 dB is its ceiling at any bitrate.
WHAT THIS DOES NOT DO. It does not put a packed frame on a machine -- that is
K3, and the layout itself was already rendered pixel-exactly on both emulators
in 47.2. It does not price clocks: 29_packed_player.py owns that, off the
MEASURED blit, and nothing here moves it. And it settles nothing about the
medium: 582.0 KB/s is geometry, and whether anything sustains it is B1.
"""
import argparse, glob, os, sys
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/bench")
import numpy as np
from PIL import Image
import vq as VQ
import dlxp as P
from dlx import DLX
from dlxload import pack_palette
ap = argparse.ArgumentParser()
ap.add_argument("packed", nargs="?", default="tmp/packed_singe.dlxp")
ap.add_argument("--frames", default="tmp/fr_singe")
ap.add_argument("--mismatch-png", default=None,
help="write the 62.5 mismatch as a picture: correct render | the "
"same frame under the NEXT frame's palette | the 24-bit "
"source. A dB is not a look, and this claim is about a look.")
ap.add_argument("--codec", default="tmp/rc_fr_singe_scsi_span.dlx",
help="the shipping container this replaces. Its PSNR is COMPUTED "
"from its own bytes, not transcribed from docs (60.8).")
a = ap.parse_args()
fail = []
d = P.DLXP(a.packed) # every format invariant is checked in here
print(f"{a.packed}: DLXP{d.version} {d.W}x{d.H} {d.fps}fps {d.nframes} frames")
print()
# --- 1. the format -----------------------------------------------------------
print("THE FORMAT, and why each line is a gate and not a courtesy check:")
print(f" record {d.rec_bytes:,} B = {d.rec_bytes // P.SECTOR} sectors exactly, "
f"palette {d.pal_bytes} B "
f"{'LAST' if d.palette_last else 'first'}, picture {d.pic_bytes:,} B")
print(f" 1.0 B/pixel: {d.pic_bytes} bytes carry {d.W * d.H} pixels "
f"(the unpacked path needs {2 * d.W * d.H:,})")
zero = black = 0
for f in range(d.nframes):
idx = d.indices(f)
# The channel copies bytes; a container whose interleave is a byte out does
# not fail, it paints. So the round trip is the assertion that the bytes in
# the record ARE the picture, in the order GVRAM wants them.
if P.pack_picture(idx).tobytes() != d._split(f)[1]:
fail.append(f"frame {f}: the record does not round-trip through the "
f"interleave -- the container is not what it says it is")
break
zero += int((idx == 0).sum())
black += int((idx == 255).sum())
if zero:
fail.append(f"index 0 appears in the picture {zero:,} times -- it is the "
f"TRANSPARENCY KEY of the top page and must stay unused (47.2)")
print(f" round-trip: {d.nframes} records unpack and re-pack byte-identical")
print(f" index 0 (transparency key) used {zero} times; "
f"index 255 (black) {black:,} times in the picture")
# THE PICTURE'S wire, and it is the one this file is about. DLXP2 puts audio on
# the same wire at a cadence (65.3, 67) and `d.kbps()` is both; what is asserted
# here is that the PICTURE's share is still exactly geometry, because that is
# 61.6's claim and a second stream is exactly the thing that could quietly
# dilute it.
kbps = d.video_kbps()
geom = d.rec_bytes * d.fps / 1024
if abs(kbps - geom) > 1e-6:
fail.append(f"wire {kbps} != geometry {geom}")
print(f" wire {kbps:.1f} KB/s = {d.rec_bytes:,} B x {d.fps} fps. FIXED. A codec's "
f"bitrate is a lever and a literal frame's is geometry (61.6)"
+ (f"\n ...and {d.audio_kbps():.2f} KB/s of audio rides beside it on the "
f"F={d.cad_f}/A={d.cad_a} cadence, for {d.kbps():.1f} KB/s total "
f"(tools/analysis/34_packed_audio.py)" if d.has_audio else ""))
print()
# --- 2. the picture ----------------------------------------------------------
files = sorted(glob.glob(f"{a.frames}/f*.png"))[:d.nframes]
if len(files) < d.nframes:
sys.exit(f"{a.frames}: {len(files)} frames, container has {d.nframes}")
src = [np.asarray(Image.open(f).convert("RGB")) for f in files]
def rendered(pal):
"""RGB888 as the DISPLAY produces it, from the same maths the loader uses."""
return pack_palette(np.asarray(pal, np.uint8))[2]
def score(name, pal_rgb, idx_frames, note=""):
"""Two columns: the palette domain every encoder PSNR in this tree is
quoted in, and the hardware word the display actually renders."""
ren = rendered(pal_rgb)
p_pal = np.mean([VQ.psnr(s, np.asarray(pal_rgb)[i])
for s, i in zip(src, idx_frames)])
p_hw = np.mean([VQ.psnr(s, ren[i]) for s, i in zip(src, idx_frames)])
print(f" {name:<44s} {p_pal:6.2f} {p_hw:6.2f} {note}")
return p_pal, p_hw
print("PSNR vs the 24-bit source, mean over frames:")
print(f" {'':<44s} {'RGB888':>6} {'GRB555':>6}")
codec_pal = codec_hw = None
if os.path.exists(a.codec):
c = DLX(a.codec)
if c.nframes < d.nframes:
print(f" (the codec container has {c.nframes} frames and this has "
f"{d.nframes} -- its row is skipped rather than compared over a "
f"different window)")
else:
# Its rate is printed with it because this is the GATE container -- the
# heaviest stream the encoder emits, `--kbps 280 --span-kbps 488
# --spans all` (check.sh) -- and NOT the 496.7 KB/s / 29.19 dB "current
# encode" of the README. Two containers, two numbers; a row that named
# neither would invite the difference to be read as a drift.
ckbps = sum(c.record_lengths()) * c.fps / c.nframes / 1024
codec_pal, codec_hw = score("CODEC, the GATE container", c.pal,
c.decode_all()[:d.nframes],
f"{ckbps:.1f} KB/s, "
f"{os.path.basename(a.codec)}")
else:
print(f" (no codec container at {a.codec} -- its row is skipped)")
# The codec's CEILING: 256 colours, one palette for the scene, no VQ loss at
# all. Not a rival, a bound -- no bitrate takes the codec past this row.
ref, spal = VQ.scene_palette(src, reserve_black=True)
sidx = VQ.palettise(src, ref)
ceil_pal, ceil_hw = score("256c SCENE palette -- the CODEC'S CEILING",
spal, sidx, "no bitrate crosses this")
# The control for the mechanism 61.9 credits: same LAYOUT and the same 254
# picture colours, one palette for the scene instead of one per frame. It is
# built to 255 with black reserved and then black is MOVED from 0 to 255, which
# is the packed layout's convention (47.2) rather than the codec's -- so the
# only variable between this row and the container's is per-frame vs scene-wide.
cref, c255 = VQ.scene_palette(src, colors=255, reserve_black=True)
cpal = np.vstack([np.zeros((1, 3), np.uint8), c255[1:],
np.zeros((1, 3), np.uint8)])
cidx = [np.where(i == 0, np.uint8(255), i)
for i in VQ.palettise(src, cref)]
ctl_pal, ctl_hw = score("PACKED, 254c SCENE palette [the CONTROL]", cpal, cidx)
# And the container itself. The right-hand column is read out of the CONTAINER'S
# OWN BYTES -- `DLXP.render` unpacks the GRB555 words the record carries -- and
# the left-hand one is recomputed from the encoder, because a packed record has
# no RGB888 palette in it to score. The two are tied together by a gate rather
# than by trust: the palettes the encoder builds here must reproduce the
# container's indices exactly, or the left column is describing a different file.
pk_idx, pk_pal_rgb, mismatch, palbad = [], [], 0, 0
for n, s in enumerate(src):
pal, idx = VQ.frame_palette(s)
if not np.array_equal(idx, d.indices(n)):
mismatch += 1
# And the WORD. The encoder packed GRB555+I with `dlxload.pack_palette` and
# `DLXP.palette_rgb` unpacks it: two separate pieces of maths over the same
# 23.3 rule, and a container is the only place they meet. Required to agree,
# not assumed to -- a wrong shared LSB is a 1.96 dB bug that still renders.
if not np.array_equal(rendered(pal), d.palette_rgb(n)):
palbad += 1
pk_idx.append(idx)
pk_pal_rgb.append(pal)
if palbad:
fail.append(f"{palbad} of {d.nframes} records carry palette words that do "
f"not unpack to the RGB the encoder packed -- pack_palette and "
f"DLXP.palette_rgb disagree about GRB555+I")
if mismatch:
fail.append(f"{mismatch} of {d.nframes} frames re-quantise to different "
f"indices than the container holds -- the RGB888 column would "
f"be scoring a file that is not this one")
pk_pal = np.mean([VQ.psnr(s, p[i]) for s, p, i in zip(src, pk_pal_rgb, pk_idx)])
pk_hw = np.mean([VQ.psnr(s, d.render(f)) for f, s in enumerate(src)])
print(f" {'PACKED CONTAINER, 254c PER-FRAME':<44s} {pk_pal:6.2f} {pk_hw:6.2f} "
f"GRB555 read out of {os.path.basename(a.packed)}")
print()
print(" The right-hand column is the PLAYER'S number. Every PSNR this project")
print(" has quoted -- 29.19, 31.33, 34.08 -- lives in the left one, upstream of")
print(" the 5-bit hardware word (23.3), and 61.9's 34.08 is directly comparable")
print(" to the packed row's left-hand entry and to nothing else.")
print()
# --- 3. what it means --------------------------------------------------------
if codec_hw is not None:
print(f" packed vs the codec gate container, as the DISPLAY renders both: "
f"{pk_hw - codec_hw:+.2f} dB")
print(f" packed vs the codec's CEILING: "
f"{pk_hw - ceil_hw:+.2f} dB")
print(f" what the PER-FRAME palette is worth (vs the control): "
f"{pk_hw - ctl_hw:+.2f} dB")
print(f" what the GRB555 word costs the ceiling row: "
f"{ceil_hw - ceil_pal:+.2f} dB")
# 60.3 measured ONE reserved entry at 0.04 dB; the packed layout spends two.
# Scored here at scene scale, where the control makes it a clean subtraction.
print(f" what the packed layout's TWO reserved entries cost: "
f"{ctl_pal - ceil_pal:+.4f} dB (256c -> 254c, scene palette, RGB888)")
print()
# The three claims 61.9 makes, restated as gates. A tree where any of these
# flipped has a different answer to ROADMAP K and should say so out loud.
if codec_hw is not None and pk_hw <= codec_hw:
fail.append(f"the packed container is {pk_hw:.2f} dB and the codec it "
f"replaces is {codec_hw:.2f} -- 61.9's headline is inverted")
if pk_hw <= ceil_hw:
fail.append(f"the packed container is {pk_hw:.2f} dB and the codec's own "
f"CEILING is {ceil_hw:.2f} -- the per-frame palette bought "
f"nothing, and 61.9's reason for building this branch is gone")
if pk_hw <= ctl_hw:
fail.append(f"per-frame {pk_hw:.2f} dB is not better than the SCENE-palette "
f"control {ctl_hw:.2f} -- the mechanism 61.9 credits is absent")
# --- 4. FINDINGS 62.5, which needed this encoder to exist ---------------------
# 62.5 filed the chain's order -- palette first or 193rd -- as a free choice with
# a visible consequence, and said the severity "depends on how much the palette
# moves between consecutive frames, which is a property of the encoder K2 has
# not been written yet". It is written now, so the number exists.
#
# The mismatch is a WIPE, not a flash: rows arrive top to bottom, so at any
# instant part of the screen is right. What is bounded here is the WORST
# instant of each order -- the whole screen wrong -- which is the start of the
# transfer for palette-first and the end of it for palette-last. The mean over
# the transfer is about half of each, because the wipe is linear in rows.
print("FINDINGS 62.5 PRICED -- palette FIRST vs LAST, at the worst instant of each:")
churn = np.mean([int((d.palette_words(n) != d.palette_words(n - 1)).sum())
for n in range(1, d.nframes)])
first = np.mean([VQ.psnr(src[n - 1], d.palette_rgb(n)[d.indices(n - 1)])
for n in range(1, d.nframes)])
last = np.mean([VQ.psnr(src[n], d.palette_rgb(n - 1)[d.indices(n)])
for n in range(1, d.nframes)])
correct = np.mean([VQ.psnr(src[n], d.render(n)) for n in range(1, d.nframes)])
print(f" palette entries that CHANGE frame to frame: {churn:.1f} of 256 "
f"({100 * churn / 256:.0f}%) -- a per-frame palette is not a small delta")
print(f" palette FIRST, old rows under the new palette: {first:6.2f} dB "
f"({first - correct:+.2f} against the correct pairing)")
print(f" palette LAST, new rows under the old palette: {last:6.2f} dB "
f"({last - correct:+.2f})")
print(f" the container is currently palette "
f"{'LAST' if d.palette_last else 'FIRST'} (dlxp.py, --palette-last)")
if a.mismatch_png:
# The frame whose mismatch is CLOSEST TO THE MEAN, so the picture is not an
# outlier picked to make the point look worse than the number.
mis = np.array([VQ.psnr(src[n - 1], d.palette_rgb(n)[d.indices(n - 1)])
for n in range(1, d.nframes)])
n = int(np.argmin(np.abs(mis - mis.mean()))) + 1
z = lambda x: np.repeat(np.repeat(x, 2, 0), 2, 1)
gap = np.full((d.H * 2, 6, 3), 30, np.uint8)
Image.fromarray(np.concatenate(
[z(d.render(n - 1)), gap, z(d.palette_rgb(n)[d.indices(n - 1)]), gap,
z(src[n - 1])], axis=1)).save(a.mismatch_png)
print(f" wrote {a.mismatch_png}: frame {n-1} correct | frame {n-1} under "
f"frame {n}'s palette ({mis[n-1]:.2f} dB) | the 24-bit source")
print(" Both are one paint, and both are MOOT if buffer mode blanks the layer")
print(" (47.4/B2). This bounds the cost of being wrong; it does not decide it,")
print(" because dB over a whole frame is not what an eye sees in a wipe.")
print()
for x in fail:
print("FAIL " + x)
sys.exit(1 if fail else 0)
tools/analysis/31_display_duty.py
+215
View File
@@ -0,0 +1,215 @@
#!/usr/bin/env python3
"""HOW LONG IS THE PICTURE ACTUALLY ON SCREEN? ROADMAP K3, FINDINGS 64.
python3 tools/analysis/31_display_duty.py [container.dlxp] [--rate KB/s ...]
THE QUESTION NOTHING IN THIS TREE HAD ASKED. Every budget in docs/FINDINGS.md
asks what a frame COSTS -- clocks, bus cycles, bytes on the wire. Session 32
built the packed player and ran it (src/player/packed.s), and the run reported a
number no budget has a column for: the write window was open on 99.5% of the
host frames, so the graphics layer was DARK for 99.5% of the scene. Every frame
was pixel-exact and almost none of them was visible.
WHY THAT IS ARITHMETIC AND NOT AN EMULATOR ARTEFACT. 256-colour GVRAM masks the
high byte of every write unless CRTC R20 bit 11 is set (46.5/47.1), and the
packed layout's whole 1.0 B/pixel claim is that one word carries two pixels --
so a packed write REQUIRES the bit. If buffer mode blanks the layer while the
bit is set (47.4/ROADMAP B2 -- MAME says it does, and 48.1's prior leans that
way), then the layer is dark for exactly as long as the window is open, and for
a DMAC-direct player the window is open for the whole transfer. There is no
second page to hide behind: the packed layout SPENDS both 256-colour pages,
which is the same fact that made a frame one channel start (FINDINGS 62).
dark fraction of a slot = record bytes / (data-phase rate x slot)
AND THE RATE IN THAT EXPRESSION IS THE BURST RATE, NOT THE SUSTAINED ONE. This
is the correction the session had to make to itself. The container's 582.0 KB/s
is a SUSTAINED requirement -- it decides whether record i arrives before slot i.
The dark fraction is set by how fast bytes move DURING THE DATA PHASE, which for
a drive with a read-ahead cache can be several times the sustained figure. The
two are independent, and a medium can pass one and fail the other:
sustained >= 582.0 KB/s or frames arrive late (B1, known)
data phase >> 582.0 KB/s or the frame is never displayed (NEW, and B1 has
no test for it)
THE OTHER PLAYER IN THE FAMILY DOES NOT HAVE THIS PROPERTY. A packed player
that DMAs the record into RAM and paints it with the CPU opens the window only
for the paint -- tools/bench/blit.s V8, MEASURED, not assumed -- which is a
fixed share of the slot no matter what the medium does. It costs more clocks
and 49 KB of RAM and it buys a picture that is on screen. FINDINGS 61.5 already
priced both in CLOCKS and ranked DMAC-direct first; this file is the column that
was missing from that table, and it reverses the ranking under B2-blanks.
"""
import argparse, os, re, sys
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from dlxp import DLXP
import buscost as B
CPUHZ = 10e6 # stock X68000, MAME 0.277 x68k.cpp:1133
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/packed_singe.dlxp")
ap.add_argument("--blit-log", default="tmp/blit_v8.log",
help="tools/bench/blit.lua's log -- the MEASURED packed paint. "
"Not a constant in this file: 47.6.1 filed the movem shape "
"as an assumption and session 29 measured it, and a second "
"copy of a measured number is how one of them goes stale.")
ap.add_argument("--rate", type=float, nargs="*", default=None,
help="data-phase rates to price, KB/s. REQUIRED to mean "
"anything: this project has no delivery figure and will "
"not default to one (FINDINGS 50).")
ap.add_argument("--run-log", default="tmp/packed_free_steal.log",
help="a free-running tools/bench/packed.lua log, for the "
"measured corroboration section")
a = ap.parse_args()
d = DLXP(a.container)
SLOT_S = 1.0 / d.fps
FRAME_CLK = CPUHZ * SLOT_S
wire = d.kbps() # KB/s, and fixed by geometry
blit = {}
if os.path.exists(a.blit_log):
for line in open(a.blit_log, errors="replace"):
m = re.search(r"V(\d+)\s+(\d+) cyc", line)
if m:
blit[int(m.group(1))] = int(m.group(2))
if 8 not in blit:
sys.exit(f"{a.blit_log} has no V8 result. The packed paint is a MEASUREMENT "
f"(tools/bench/blit.lua) and this tool will not substitute a "
f"constant for it -- run the blit bench, or point --blit-log at "
f"its log.")
PAINT_CLK = blit[8]
PAINT_FRAC = PAINT_CLK / FRAME_CLK
print(f"""THE CONTAINER (tools/encoder/dlxp.py)
{a.container}: {d.W}x{d.H} {d.fps} fps, {d.nframes} frames
record {d.rec_bytes:,} B = {d.rec_bytes // 512} sectors, palette \
{'LAST' if d.palette_last else 'FIRST'}
slot {SLOT_S*1000:.2f} ms = {FRAME_CLK:,.0f} clocks
wire {wire:.1f} KB/s -- FIXED by geometry. A codec's bitrate is a lever and a
literal frame's is not (61.6), so nothing an encoder does moves this.
THE TWO PACKED PLAYERS, and the difference is WHEN the write window is open
A. DMAC-DIRECT (src/player/packed.s, ROADMAP K3, and the one that is built).
One channel start, 193 destinations, the CPU halted or nearly. The window
must be open for the WHOLE data phase, because the channel writes when the
bytes arrive and the CPU cannot know when that is -- and a packed write
that lands with the bit clear is masked to its low byte and silently wrong.
B. DMA-TO-RAM + CPU PAINT. The record lands in RAM with the window shut; the
68000 then paints it with the packed movem blit. The window is open for the
PAINT and nothing else: {PAINT_CLK:,} clocks, {100*PAINT_FRAC:.1f}% of a slot,
MEASURED by tools/bench/blit.lua (V8), and INDEPENDENT of the medium.
Under 47.4-blanks the dark interval IS the open window, so B is on screen for
{100*(1-PAINT_FRAC):.1f}% of every slot at ANY rate that delivers the record at all,
and A's visibility is a function of the rate.
""")
rates = a.rate
if not rates:
print("""NO RATES GIVEN, so no table. This project retired its delivery
constant outright (FINDINGS 50, USER DECISION) and every tool requires an
explicit rate; a default here would be the same mistake in a new place. Pass
--rate with the figures you want priced. The three thresholds already derived
elsewhere, for reference and NOT as defaults:
453.6 KB/s the DLX5 codec gate container needs no prefill (49.5/60)
576.0 KB/s a packed container with no per-frame palette (61.5)
582.0 KB/s THIS container, palette included (63)
and note that all three are SUSTAINED figures. The dark fraction below is set by
the DATA-PHASE rate, which is a different measurement nothing has taken.""")
sys.exit(0)
print("A's VISIBILITY, against the DATA-PHASE rate\n")
print(f" {'data phase':>12} | {'transfer':>9} | {'window open':>11} | "
f"{'PICTURE ON SCREEN':>17} | vs B")
print(f" {'KB/s':>12} | {'ms':>9} | {'% of slot':>11} | "
f"{'% of slot':>17} |")
print(" " + "-"*12 + "-+-" + "-"*9 + "-+-" + "-"*11 + "-+-" + "-"*17 + "-+-----")
for R in sorted(rates):
t_ms = d.rec_bytes / (R * 1024) * 1000
openf = min(1.0, t_ms / (SLOT_S * 1000))
vis = 1.0 - openf
verdict = ("A wins" if vis > 1 - PAINT_FRAC else
"B wins" if vis < 1 - PAINT_FRAC else "equal")
late = " LATE" if R < wire else ""
print(f" {R:>12.1f} | {t_ms:>9.2f} | {100*openf:>11.1f} | "
f"{100*vis:>17.1f} | {verdict}{late}")
# The crossover, stated as a rate rather than left to be read off the table: it
# is the one number in here a hardware acceptance test can be written against.
cross = d.rec_bytes / (PAINT_FRAC * SLOT_S) / 1024
print(f"""
A and B show the picture for the same share of the slot at a data-phase rate
of {cross:,.0f} KB/s. Below that, THE PLAYER WITH THE CPU IN THE LOOP IS ON
SCREEN LONGER than the one without it -- which is the reverse of FINDINGS
61.5's ranking, and 61.5 is not wrong: it ranked them in CLOCKS, and this is
the column that table does not have.
{cross:,.0f} KB/s is {cross/wire:.1f}x the container's own wire. So a medium that exactly
meets the sustained requirement puts the DMAC-direct player's picture on
screen for {100*max(0.0, 1-wire/wire):.0f}% of every slot: it delivers every frame, on time,
pixel-exact, and displays none of them.""")
print(f"""
THE CPU SIDE, so the trade is priced on both axes (FINDINGS 61.5's ladder)
W is clocks stolen per delivered byte. Only the dual-address rungs have a code
path on this machine (59.2), and 9 is the floor: a 4-clock read of the device
plus a 5-clock write to memory.
""")
print(f" {'W':>3} | {'A: DMAC-direct':>15} | {'B: DMA + CPU paint':>19}")
print(" " + "-"*3 + "-+-" + "-"*15 + "-+-" + "-"*19)
AUDIO = B.ADPCM_BYTES_PER_S / d.fps * B.ADPCM_CLK_BYTE_BEST
for W in (5, 9, 12, 16, 19):
xfer = d.rec_bytes * W
ca = (xfer + AUDIO) / FRAME_CLK
cb = (xfer + AUDIO + PAINT_CLK) / FRAME_CLK
print(f" {W:>3} | {100*ca:>14.1f}% | {100*cb:>18.1f}%")
print(f"""
Both include the audio DMA at {AUDIO:,.0f} clocks a frame ({100*AUDIO/FRAME_CLK:.2f}%), charged from
the IPL ROM's own channel-3 setup (21_iplrom_dmac.py, 52.5). Neither includes
a decoder, because neither has one.
So B costs the paint -- {100*PAINT_FRAC:.1f}% of a frame -- and TWO record buffers,
{2*d.rec_bytes:,} B of RAM. Two and not one: at any rate near the wire the delivery
of record i+1 occupies most of the slot the paint of record i happens in, so
they overlap by construction. On a 2 MB machine that is {200*d.rec_bytes/(2*1024*1024):.1f}% of memory and
it is the resource this design has spare -- the ring the packed branch deleted
was 256 KB (FINDINGS 49). That is what a visible picture costs if 47.4 blanks.""")
# ---- the measured corroboration. It is a SEPARATE section and it is bounded
# on purpose: MAME's device models carry no transfer timing (docs/BENCHMARK.md,
# 42.5), so the run cannot supply a rate for the table above -- what it can do
# is show that the mechanism is real and that the arithmetic predicts it.
if os.path.exists(a.run_log):
txt = open(a.run_log, errors="replace").read()
m_rate = re.search(r"record lands in ([\d.]+) ms, i\.e\. ([\d.]+) KB/s", txt)
m_open = re.search(r"WRITE WINDOW OPEN on (\d+) of (\d+) host frames", txt)
if m_rate and m_open:
ms, kbps = float(m_rate.group(1)), float(m_rate.group(2))
op, tot = int(m_open.group(1)), int(m_open.group(2))
pred = min(1.0, d.rec_bytes / (kbps * 1024) / SLOT_S)
print(f"""
MEASURED, on the emulated machine (tools/bench/packed_run.sh, free-running)
{a.run_log}: a {d.rec_bytes:,} B record landed in {ms:.2f} ms = {kbps:.1f} KB/s, and the
write window was open on {op} of {tot} host frames = {100*op/tot:.1f}%.
The expression above predicts {100*pred:.1f}% at that rate.
THIS IS NOT A RATE MEASUREMENT AND {kbps:.0f} KB/s IS NOT A MEDIUM. MAME's
device models carry no transfer timing (42.5); the figure is a property of the
apparatus. What the run DOES establish is that the mechanism is the one the
arithmetic describes -- and one thing more that no arithmetic could have
given: the DMAC CONFIGURATION DOES NOT MOVE IT. Held and stealing delivered
the same record within 0.5% of each other, so what a channel configuration
buys is who owns the CPU, not when the picture appears.""")
else:
print(f"""
NO MEASURED SECTION: {a.run_log} is absent. Run
tools/bench/packed_run.sh to produce it. The arithmetic above stands without
it -- it is geometry -- but the run is what showed the effect was there to be
derived at all.""")
tools/analysis/32_audio_wire.py
+227
View File
@@ -0,0 +1,227 @@
#!/usr/bin/env python3
"""WHAT DOES AUDIO DO TO THE CONTAINER? ROADMAP P6, the half that is not the bus.
python3 tools/analysis/32_audio_wire.py [packed.dlxp] [--audio tmp/au_singe.raw]
[--rate KB/s ...]
Session 20 (FINDINGS 52) closed the bus half of P6: a second DMA consumer at
7,812.5 B/s is 1.25%..1.48% of a frame, about 4% of what the decoder leaves, and
the 7.8 kB/s figure survived with a unit correction. ROADMAP P6 then says, in
as many words, that EVERYTHING ELSE in the item is open: extraction, an encoder,
the container interleave, and what a second stream does to `wire` and therefore
to `pipe - wire` and therefore to 51.3's refill climb.
This file is the container interleave and the wire. It is arithmetic over the
real container's real geometry -- no MAME run, no board.
THE THING THAT MAKES IT INTERESTING, and it is a property of DLXP1 rather than
of audio: **a packed container has no index and cannot have one.** A record's
address is `LBA0 + i*97` because a literal frame's length is geometry (FINDINGS
63, 64.1). Audio is a stream at a rate that has nothing to do with the frame
rate, so the naive interleave -- give record i the audio bytes belonging to slot
i -- makes records VARIABLE LENGTH, and the moment records are variable length
the format needs an index and stops being the format.
So the interleave has to be a FIXED CADENCE: every F frames, A whole sectors of
audio, placed between records. Then
LBA(i) = LBA0 + i*RECSEC + floor(i/F)*A
which is still two multiplies and a divide -- arithmetic, no index, nothing
walked -- and the only cost is that A*512 must be at least F frames' worth of
audio, so the padding is whatever A*512 exceeds it by. Choosing (F, A) is a
rational-approximation problem and the answer is NOT the obvious cadence.
"""
import argparse, os, sys
from fractions import Fraction
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/analysis")
import buscost as B
from dlxp import DLXP, SECTOR
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/packed_singe.dlxp")
ap.add_argument("--audio", default="tmp/au_singe.raw",
help="raw s16le mono at the chip rate, from extract_audio.py")
ap.add_argument("--codec", default="tmp/rc_fr_singe_scsi_span.dlx",
help="the codec container, for the same arithmetic on the other branch")
ap.add_argument("--rate", type=float, nargs="*",
default=[453.6, 500.0, 582.0, 600.0, 650.0, 700.0],
help="explicit sustained delivery rates, KB/s")
a = ap.parse_args()
d = DLXP(a.container)
SLOT_S = 1.0 / d.fps
FRAME_CLK = B.CPU_HZ * SLOT_S if hasattr(B, "CPU_HZ") else 10_000_000 * SLOT_S
RECSEC = d.rec_bytes // SECTOR
print(f"""
=== THE STREAM =========================================================
The chip is an MSM6258V on an 8 MHz clock and it has three rates and no
others. Every budget in this tree is written against the first one.""")
RATES = {512: 15625.0, 768: 8_000_000/768, 1024: 7812.5}
print(f"\n {'divisor':>8} {'samples/s':>11} {'bytes/s':>10} {'B per 1/%d s slot' % d.fps:>19} exact?")
for div, hz in RATES.items():
bps = hz / 2
per = bps / d.fps
print(f" 8MHz/{div:<4} {hz:11,.1f} {bps:10,.1f} {per:19,.4f} "
f"{'yes' if per == int(per) else 'NO -- a remainder, like the frame clock (54)'}")
HZ = 15625.0
AU_BPS = HZ / 2 # 4 bits a sample, two samples to a byte
AU_FRAME = AU_BPS / d.fps # 651.0416... B, and the point is the dots
print(f"""
The shipping rate's per-slot figure is {AU_FRAME:,.4f} B and it is NOT an
integer -- 8 MHz / 512 / 2 / {d.fps} has a 12 in the denominator that 2**k
cannot clear. That is the same shape as FINDINGS 54's frame clock: what a
player carries is a remainder, not a count, and a container that rounds it
either drifts or underruns.""")
if os.path.exists(a.audio):
n16 = os.path.getsize(a.audio) // 2
secs = n16 / HZ
print(f"""
MEASURED, on the window this project gates everything on (00223 @539.4s,
{secs:.3f} s, tools/encoder/extract_audio.py):
{n16:,} samples -> {n16//2:,} B of ADPCM = {n16/2/secs:,.1f} B/s
which is {AU_BPS:,.1f} to the byte, so the rate is the rate.""")
print(f"""
=== THE INTERLEAVE, AND WHY THE OBVIOUS CADENCE IS THE WRONG ONE =======
A packed record is {d.rec_bytes:,} B = {RECSEC} sectors EXACTLY and its address is
arithmetic. Audio rides between records at a fixed cadence -- every F frames,
A whole sectors -- so that LBA(i) stays arithmetic. A must satisfy
A * {SECTOR} >= F * {AU_FRAME:,.4f} i.e. A/F >= {Fraction(int(AU_BPS*2), int(2*SECTOR*d.fps))} = {AU_FRAME/SECTOR:.9f}
and everything above that ratio is PADDING that the wire pays for and nothing
plays. Here is the whole small-F space, best A for each F:""")
target = Fraction(int(round(AU_BPS * 2)), 2 * SECTOR * d.fps) # sectors per frame, exact
rows, floor = [], None
for F in range(1, 241):
A = -(-(target.numerator * F) // target.denominator) # ceil(F * target)
have, need = A * SECTOR, F * AU_FRAME
waste = (have - need) / need
add = have / F * d.fps / 1024 # what the cadence puts on the wire, KB/s
rows.append((waste, F, A, have, need, add))
print(f"\n {'F':>4} {'A':>4} {'A*512 B':>10} {'needs':>12} {'padding':>9} {'waste':>7}"
f" {'wire adds':>10} {'player RAM':>11}")
seen = None
for waste, F, A, have, need, add in rows:
show = F <= 4 or seen is None or waste < seen - 1e-12
if seen is None or waste < seen: seen = waste
if show:
print(f" {F:4d} {A:4d} {have:10,} {need:12,.1f} {have-need:9,.1f} "
f"{100*waste:6.2f}% {add:9.2f} KB/s {have:9,} B")
best = sorted(rows)
w, F, A, have, need, add = best[0]
f1 = next(x for x in rows if x[1] == 1)
print(f""" THE FLOOR OF THAT SWEEP is F={F}, A={A}: {100*w:.3f}% padding, {add:.2f} KB/s of
wire for {AU_BPS/1024:.2f} KB/s of audio.
THE OBVIOUS CADENCE IS THE WORST ONE. F=1 -- one audio lump per record, which
is what "interleave the audio into the frame" means if nobody does the
arithmetic -- needs A={f1[2]} and costs {100*f1[0]:.1f}% padding: {AU_FRAME:,.1f} B rounded up to
{f1[3]:,}, so {f1[3]-AU_FRAME:,.1f} B of every record is nothing at all, and the wire pays
{f1[5]:.2f} KB/s for {AU_BPS/1024:.2f} KB/s of audio. That is {f1[5]-add:.2f} KB/s thrown away for
no reason but the cadence.
=== WHAT IT DOES TO THE WIRE ===========================================""")
vid_kbs = d.video_kbps()
for label, cad in (("F=1 (one lump a record)", f1), (f"F={F} (the floor)", best[0])):
tot = vid_kbs + cad[5]
print(f" {label:26s} video {vid_kbs:7.1f} + audio {cad[5]:5.2f} = {tot:7.1f} KB/s "
f"({100*(tot/vid_kbs-1):+.2f}%)")
print(f"""
And this is what B1's acceptance test becomes. The packed container's
sustained requirement was {vid_kbs:.1f} KB/s SILENT (FINDINGS 61.5, 63) and it is
{vid_kbs + add:.1f} KB/s with sound. A literal frame's bitrate is geometry and cannot
be talked down; the audio on top of it is {add:.2f} KB/s and can only be talked down
by choosing a worse chip rate.""")
f11 = next(x for x in rows if x[1] == 11)
print(f"""
AND THE CADENCE HAS A SECOND PRICE, WHICH IS RAM. A cadence of F frames means
the player is holding F frames of audio, and holding it TWICE -- the channel
fills lump n+1 while the chip drains lump n, the same reason K4 needs two
record buffers (64.2). So the floor of the sweep is not the answer:
F={f1[1]:<3} {f1[3]:>7,} B a lump, {2*f1[3]:>7,} B held {100*f1[0]:6.2f}% padding {f1[5]:5.2f} KB/s
F={f11[1]:<3} {f11[3]:>7,} B a lump, {2*f11[3]:>7,} B held {100*f11[0]:6.2f}% padding {f11[5]:5.2f} KB/s <- the pick
F={F:<3} {have:>7,} B a lump, {2*have:>7,} B held {100*w:6.2f}% padding {add:5.2f} KB/s
F={f11[1]} buys {100*(f1[0]-f11[0]):.1f} points of padding for {2*f11[3]-2*f1[3]:,} B of RAM, and F={F} buys the
last {100*(f11[0]-w):.2f} of a point for {2*have-2*f11[3]:,} B more. On a machine where K4 already
wants 99,328 B for two record buffers, the second trade is not one.
=== THE ASYMMETRY: THE CODEC CONTAINER PAYS NONE OF THIS ===============""")
if os.path.exists(a.codec):
sys.path.insert(0, "tools/encoder")
from dlx import DLX
c = DLX(a.codec)
lens = c.record_lengths() if callable(getattr(c, "record_lengths", None)) else c.record_lengths
cwire = sum(lens) / len(lens) * c.fps / 1024
print(f""" {os.path.basename(a.codec)}: {c.nframes} records, index {'PRESENT' if c.has_index else 'absent'},
records already VARIABLE ({min(lens):,}..{max(lens):,} B, mean {sum(lens)/len(lens):,.0f}) and
sector-aligned since DLX5 (60.1). A container that already carries an index
and already has variable records can put EXACTLY {AU_FRAME:,.1f} B of audio in record i
and pad only to the sector it was going to pad to anyway -- so its audio
padding is not 57.3% and not 1.11%, it is ZERO, and its wire goes
{cwire:.1f} -> {cwire + AU_BPS/1024:.1f} KB/s ({100*(AU_BPS/1024)/cwire:+.2f}%).
THAT IS THE FIRST COST THIS PROJECT HAS FOUND FOR THE PACKED BRANCH'S OWN
SIMPLIFICATION. "A record's length is geometry, so there is no index and none
can be needed" (63, 64.1) is what makes the packed player a page of arithmetic
instead of a parser -- and it is exactly the property that makes a second
stream at an unrelated rate cost padding, a cadence, and a buffer. It is a
small cost ({f11[5]-AU_BPS/1024:.2f} KB/s at the pick, {2*f11[3]:,} B of RAM) and it is not zero, and
nothing in FINDINGS 61-64 predicted it.""")
else:
print(f" SKIPPED: no codec container at {a.codec}")
print(f"""
=== WHAT IT DOES TO SLACK (51.3) =======================================
Slack is ACCUMULATED out of pipe - wire, so a second consumer does not cost a
fixed amount -- it costs the accumulation rate, and what a branch point costs is
set by that (51.3, 55.4). Silent vs sounded, at explicit rates:
{'pipe':>8} {'silent':>14} {'sounded':>14} what a second of play banks""")
for kbps in a.rate:
s_sl, a_sl = kbps - vid_kbs, kbps - (vid_kbs + add)
def fmt(x): return f"{x:+8.1f} KB/s" if x >= 0 else f"{x:+8.1f} KB/s"
print(f" {kbps:8.1f} {fmt(s_sl):>14} {fmt(a_sl):>14} "
+ ("both starve" if a_sl < 0 and s_sl < 0
else "SOUND IS WHAT BREAKS IT" if s_sl >= 0 > a_sl
else f"{a_sl/s_sl*100:.0f}% of the silent rate" if s_sl > 0 else ""))
AUCLK_LO = AU_FRAME * B.ADPCM_CLK_BYTE_BEST
AUCLK_HI = AU_FRAME * B.ADPCM_CLK_BYTE_WORST
print(f"""
=== AND WHAT IT DOES TO THE FRAME (the half session 20 already closed) ==
{AU_FRAME:,.1f} B a slot at {B.ADPCM_CLK_BYTE_BEST}..{B.ADPCM_CLK_BYTE_WORST} clocks a byte (the IPL ROM's OWN channel-3
configuration, read out of the ROM by 21_iplrom_dmac.py, not chosen here) is
{AUCLK_LO:,.0f}..{AUCLK_HI:,.0f} clocks = {100*AUCLK_LO/FRAME_CLK:.2f}%..{100*AUCLK_HI/FRAME_CLK:.2f}% of a {SLOT_S*1000:.2f} ms slot.
That reproduces FINDINGS 52 exactly, which is the point of printing it.
THE INTERACTION 52 COULD NOT HAVE HAD is with 64.2's write window. A
DMAC-direct packed player holds the GVRAM window open for the whole data
phase, and an audio channel stealing the bus during that phase makes the phase
LONGER -- so audio does not merely cost clocks, it costs DARKNESS:
extra dark per slot = {100*AUCLK_LO/FRAME_CLK:.2f}%..{100*AUCLK_HI/FRAME_CLK:.2f}% of the slot, on top of
record/(burst x slot), which is already 1.0 at the wire
It is small against a dark fraction that is already 1.0, and it is not small
against K4's {100*227553/FRAME_CLK:.1f}% paint. For the CPU-painted player the audio steals
from the paint and not from the picture, which is the third time this session
the two players have ranked differently on a column that is not clocks.
""")
tools/analysis/33_adpcm_model.py
+123
View File
@@ -0,0 +1,123 @@
#!/usr/bin/env python3
"""What the chip's own decoder model costs the encoder. ROADMAP P6a, after it.
tools/bench/adpcm_run.sh MEASURED four things about the MSM6258 as this machine
models it, and tools/encoder/adpcm.py had a different value for every one:
axis encoder default the chip how it was measured
feed both-high-first both-LOW-first
formula shift terms 1,678 samples, sample-exact
clamp 12-bit 10-bit one model of sixteen matched
init 0 -2
This file prices them, on the same ten seconds of the same stream every audio
figure in this project is quoted against (tmp/au_singe.raw, FINDINGS 65). It
takes an explicit source file rather than defaulting to one, for the same reason
every rate in this tree is an explicit argument (FINDINGS 50).
THE ONE THAT IS NOT A UNIT SLIP is the CLAMP. The other three are conventions:
get one wrong and the decode is wrong, get it right and nothing is lost. A
10-bit accumulator is a smaller container, and it is INSIDE the recursion -- the
predictor cannot represent what will not fit -- so it costs SNR even when the
encoder knows about it and encodes for it. That is a ceiling on this format on
this machine and it is not recoverable by encoding harder.
"""
import math, os, sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "encoder"))
import adpcm
RAW = sys.argv[1] if len(sys.argv) > 1 else "tmp/au_singe.raw"
CHIP = dict(variant="terms", order="low", bits=10, init=-2)
ENC = dict(variant="shift", order="high", bits=12, init=0)
def snr_db(ref, got):
num = sum(float(s) * s for s in ref)
den = sum((float(a) - b) ** 2 for a, b in zip(ref, got))
if den == 0: return float("inf")
return 10.0 * math.log10(num / den) if num else float("-inf")
def main():
import struct
if not os.path.exists(RAW):
print(f"no {RAW} -- run tools/encoder/extract_audio.py first")
return 2
pcm = struct.unpack("<%dh" % (os.path.getsize(RAW) // 2), open(RAW, "rb").read())
src12 = [max(-2048, min(2047, x >> 4)) for x in pcm]
print(f"{RAW}: {len(src12):,} samples, peak {max(abs(v) for v in src12)} "
f"in 12-bit units")
print()
print("1. THE COST OF ENCODING FOR THE WRONG CHIP, all four axes at once")
print(" Encode under the encoder's defaults; play it on the chip. The")
print(" nibble ORDER is not a decode parameter -- it decides which nibble")
print(" of each byte the chip takes -- so it is applied by re-reading the")
print(" encoder's own packed bytes the way the chip reads them.")
nib = adpcm.encode(src12, ENC["variant"], init=ENC["init"], bits=ENC["bits"])
same = adpcm.decode(nib, ENC["variant"], init=ENC["init"], bits=ENC["bits"])
data = adpcm.pack(nib, ENC["order"])
asread = list(adpcm.unpack(data, len(nib), CHIP["order"]))
cross = adpcm.decode(asread, CHIP["variant"], init=CHIP["init"], bits=CHIP["bits"])
print(f" encoded and decoded on the encoder's model : {snr_db(src12, same):7.2f} dB")
print(f" encoded on the encoder's, played on the chip: {snr_db(src12, cross):7.2f} dB")
print()
print("2. ONE AXIS AT A TIME, so the bill is itemised rather than lumped")
for name, key, val in (("nibble order", "order", CHIP["order"]),
("delta formula", "variant", CHIP["variant"]),
("clamp", "bits", CHIP["bits"]),
("initial accumulator", "init", CHIP["init"])):
m = dict(ENC); m[key] = val
d = adpcm.pack(nib, ENC["order"])
rd = list(adpcm.unpack(d, len(nib), m["order"]))
got = adpcm.decode(rd, m["variant"], init=m["init"], bits=m["bits"])
print(f" {name:22s} wrong only here: {snr_db(src12, got):7.2f} dB")
print()
print("3. AND THE ONE THAT IS NOT A CONVENTION. Encode FOR the chip -- the")
print(" encoder knows the model and searches against it -- and compare a")
print(" 10-bit accumulator with a 12-bit one on the same seconds.")
for bits in (12, 10):
n = adpcm.encode(src12, CHIP["variant"], init=CHIP["init"], bits=bits)
r = adpcm.decode(n, CHIP["variant"], init=CHIP["init"], bits=bits)
clip = sum(1 for v in r if v in adpcm.clamp_bounds(bits))
print(f" encoded and decoded at {bits}-bit: {snr_db(src12, r):7.2f} dB"
f" ({clip:,} of {len(r):,} samples sit ON the clamp)")
print()
print("4. WHAT THE LEVEL DOES NOW, and it did nothing before (65.1).")
print(" At 12 bits the disc's -13.4 dBFS peak had headroom to spare and")
print(" normalising bought 0.00 dB. A 10-bit accumulator is 4x smaller,")
print(" so the same signal is no longer comfortably inside it.")
peak = max(abs(v) for v in src12)
for name, g in (("as recorded", 1.0),
("scaled to fit 10 bits", 500.0 / peak),
("half of that", 250.0 / peak)):
sc = [max(-512, min(511, int(round(v * g)))) for v in src12]
n = adpcm.encode(sc, CHIP["variant"], init=CHIP["init"], bits=CHIP["bits"])
r = adpcm.decode(n, CHIP["variant"], init=CHIP["init"], bits=CHIP["bits"])
print(f" {name:24s} x{g:5.2f} peak {max(abs(v) for v in sc):4d} "
f"{snr_db(sc, r):7.2f} dB")
print()
print("5. THE HEADROOM, which is the part of this that will bite later.")
hd = 20 * math.log10(511.0 / peak)
print(f" This window peaks at {peak} of the 10-bit accumulator's 511, so it")
print(f" has {hd:.1f} dB of headroom left -- and it is a QUIET passage: the")
print(" disc peaks at -13.4 dBFS here (65.1). A 10-bit accumulator is")
print(f" {20*math.log10(2047.0/511.0):.1f} dB smaller than the 12-bit word the encoder was")
print(" clamping to, so a passage only a few dB louder than this one does")
print(" not fit and the predictor CLIPS inside the recursion. Nothing in")
print(" this project has measured the loudest passage on the disc; until")
print(" something does, the audio level is an OPEN choice and not a")
print(" settled one, and 65.1's `the level is not a lever` is now wrong")
print(" in one direction: it is not a lever UPWARD.")
print()
print(" The rows in 4 are NOT comparable as absolute quality")
print(" -- each is scored against its OWN scaled reference, so what they")
print(" compare is how well the format tracks a signal of that size.")
return 0
if __name__ == "__main__":
sys.exit(main())
tools/analysis/34_packed_audio.py
+217
View File
@@ -0,0 +1,217 @@
#!/usr/bin/env python3
"""DLXP2's GATE: a packed container with sound in it. ROADMAP P6b.
python3 tools/analysis/34_packed_audio.py [packed.dlxp] [--audio tmp/au_singe.raw]
WHAT THIS IS FOR. FINDINGS 65.3 did the arithmetic of putting audio in the
packed container and wrote no byte of one; 66 measured which of sixteen decoder
models the machine's chip runs and priced the axes at up to 25.7 dB. This is
the container those two produce, and the reason it needs a gate of its own is
that NOTHING PARSES A PACKED CONTAINER. A DMA channel copies bytes and has no
opinion about them (62), so a container whose lump is one sector out does not
fail -- it plays 512 B of picture as audio and 512 B of audio as picture, both
of which are things, and a gate that only checked for errors would pass it.
THE FOUR CLAIMS, and each is checked against something that is not the writer:
1. THE FILE IS ITS OWN ARITHMETIC. Every byte of the container is accounted
for by `off_frm + i*rec + (i//F)*A*512` and `off_aud + k*(F*rec + A*512)`
with no byte left over and no byte claimed twice. A per-record read cannot
catch an off-by-one that shifts everything after it; a partition can.
2. THE PICTURE DID NOT MOVE. Interleaving a second stream into a format whose
whole claim is "record i is at LBA0 + i*97" is exactly the change that can
break that claim, so every record is compared against a re-encode of the
same frames with `--audio` off. The silent container is the control.
3. THE BYTES ARE THE ENCODER'S. The lumps, concatenated, are byte-exact
against `adpcm.encode` run again on the same PCM with the same four axes.
4. THE HEADER'S AXES ARE LOAD-BEARING. The stream decodes to the source at
the SNR the encoder reported, and flipping any ONE of the four axes the
header carries collapses it. A header field nothing would notice being
wrong is a comment.
AND THE FINDING IT REPORTS (FINDINGS 67). The padding is not where a reader of
65.3 would put it. A lump is A*512 B of SPACE; F frames of audio is
F*hz/(2*fps) B, which at F=11 is 7,161.4583..., so the PAYLOAD alternates 7,161
and 7,162 and the sector run is 7,168 either way. A player that handed the chip
the whole lump -- the obvious implementation, and the one the phrase "14 sectors
of audio every 11 frames" invites -- would be feeding it 6.54 B a group too
much. That is not waste, which is what padding usually is. It is DRIFT.
"""
import argparse, math, os, struct, sys
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/analysis")
import adpcm
import dlxp as P
from dlxp import DLXP, SECTOR
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/packed_singe.dlxp")
ap.add_argument("--audio", default="tmp/au_singe.raw")
ap.add_argument("--silent", default="tmp/packed_singe_silent.dlxp",
help="the control: the same frames with --audio off. Built by "
"check.sh; skipped rather than faked when absent")
ap.add_argument("--game-min", type=float, default=22.8,
help="the game's running length, for what the drift comes to")
a = ap.parse_args()
fails = []
def ck(ok, msg):
print((" OK " if ok else " FAIL ") + msg)
if not ok:
fails.append(msg)
d = DLXP(a.container) # every format invariant is checked here
print(f"{a.container}: DLXP{d.version} {d.W}x{d.H} {d.fps}fps {d.nframes} frames, "
f"{'AUDIO' if d.has_audio else 'SILENT'}")
if not d.has_audio:
sys.exit(f"{a.container} carries no audio -- this gate has nothing to check. "
f"Build it with tools/encoder/pack.py --audio")
grp = d.cad_f * d.aud_hz / (2 * d.fps)
print(f"""
=== THE LAYOUT =========================================================
record {d.rec_bytes:,} B = {d.rec_bytes//SECTOR} sectors, lump {d.cad_a*SECTOR:,} B = {d.cad_a} sectors,
cadence F={d.cad_f} A={d.cad_a}, {d.n_lumps} lumps, {d.aud_bytes:,} B of ADPCM at {d.aud_hz:,} Hz
record i = {d.off_frm:,} + i*{d.rec_bytes:,} + (i//{d.cad_f})*{d.cad_a*SECTOR:,}
lump k = {d.off_aud:,} + k*{d.cad_f*d.rec_bytes + d.cad_a*SECTOR:,}
and NEITHER of those is a lookup. A packed record's length is geometry and a
lump's is a cadence, so DLXP2 still has no index and still needs none.""")
# --- 1. the file is its own arithmetic ------------------------------------
# Every byte, partitioned. Not "does record 7 read back" -- an off-by-one that
# shifts the whole stream reads back fine one record at a time.
spans = [(d.frame_off(i), d.rec_bytes, f"record {i}") for i in range(d.nframes)]
spans += [(d.lump_off(k), d.cad_a * SECTOR, f"lump {k}") for k in range(d.n_lumps)]
spans.sort()
pos, overlap, gap = SECTOR, [], []
for off, n, what in spans:
if off < pos: overlap.append(what)
elif off > pos: gap.append((pos, off, what))
pos = max(pos, off + n)
ck(not overlap, f"nothing overlaps ({len(spans)} spans: {d.nframes} records "
f"+ {d.n_lumps} lumps)" + (f" -- {overlap[:3]}" if overlap else ""))
ck(not gap, "no byte between the header and the end belongs to nothing"
+ (f" -- {gap[:3]}" if gap else ""))
ck(pos == len(d.raw), f"the arithmetic ends at {pos:,} and the file is "
f"{len(d.raw):,} B")
ck(all(off % SECTOR == 0 for off, _, _ in spans),
"every record and every lump starts on a 512 B sector -- 58.3/60.1's "
"precondition survives the interleave")
# --- 1b. and the cadence term is load-bearing -----------------------------
# THE FAILURE MODE THIS FORMAT HAS AND THE CODEC'S DOES NOT. A DLX record is
# found through an index and a player that read the wrong entry gets a length
# word that does not parse. A packed record is found by ARITHMETIC and nothing
# parses it, so a player that drops the `(i//F)*A` term reads 97 sectors
# starting 14 sectors early and paints them: the last 14 sectors of the previous
# record, then 83 of this one, shifted down the screen. It is a picture. Here
# is what the gate would be comparing if the term were missing, and it is only
# WRONG from frame F on -- the first group is exempt, which is how an off-by-one
# like this survives a rig that checks frame 0.
blind = [i for i in range(d.nframes)
if d.raw[d.off_frm + i*d.rec_bytes:d.off_frm + (i+1)*d.rec_bytes]
!= d.record(i)]
ck(blind == list(range(d.cad_f, d.nframes)),
f"a cadence-blind player reads the wrong bytes for {len(blind)} of "
f"{d.nframes} records, first at frame {blind[0] if blind else '-'} -- and "
f"frames 0..{d.cad_f-1} are IDENTICAL either way, so frame 0 proves nothing")
# --- 2. the picture did not move ------------------------------------------
if os.path.exists(a.silent):
q = DLXP(a.silent)
same = (q.nframes == d.nframes
and all(q.record(i) == d.record(i) for i in range(d.nframes)))
ck(same, f"all {d.nframes} records byte-exact against the SILENT control "
f"({os.path.basename(a.silent)}) -- interleaving audio moved no "
f"picture byte")
ck(q.has_audio is False and q.off_frm == SECTOR,
"and the control really is silent: no audio flag, record 0 at sector 1")
else:
print(f" SKIPPED: no silent control at {a.silent}")
# --- 3. the bytes are the encoder's ---------------------------------------
if os.path.exists(a.audio):
raw = open(a.audio, "rb").read()
pcm = struct.unpack("<%dh" % (len(raw) // 2), raw)
src = [max(-2048, min(2047, x >> 4)) for x in pcm]
axes = d.decoder()
ck(axes == adpcm.CHIP, f"the header's four axes ARE adpcm.CHIP: {axes}")
nib = adpcm.encode(src, variant=axes["variant"], init=axes["init"],
bits=axes["bits"])
want = adpcm.pack(nib, order=axes["order"])[:d.aud_bytes]
got = d.audio()
ck(got == want, f"the {len(got):,} B the lumps carry are byte-exact against "
f"adpcm.encode on the same PCM")
ck(all(d.lump(k, padding=True)[len(d.lump(k)):] == b"\0" * (
d.cad_a * SECTOR - len(d.lump(k))) for k in range(d.n_lumps)),
"and every lump's padding is zero, so a player that overruns the payload "
"feeds the chip silence rather than the next lump's first sample")
# --- 4. the header's axes are load-bearing ----------------------------
def snr(axes_):
rec = adpcm.decode(adpcm.unpack(got, len(src), order=axes_["order"]),
variant=axes_["variant"], init=axes_["init"],
bits=axes_["bits"])
n = min(len(rec), len(src))
e = sum((x - y) ** 2 for x, y in zip(src[:n], rec[:n]))
s = sum(x * x for x in src[:n])
return 10 * math.log10(s / e) if e else float("inf")
right = snr(axes)
ck(right > 20.0, f"decoded on the axes the header names: {right:.2f} dB")
print(f"\n AND EVERY AXIS IS A NEGATIVE CONTROL -- flip ONE and this is what\n"
f" a player that ignored the header would hear:\n")
print(f" {'axis':<12} {'header':>8} {'flipped to':>11} {'SNR':>9} cost")
flips = [("order", "high" if axes["order"] == "low" else "low"),
("variant", "terms" if axes["variant"] == "shift" else "shift"),
("bits", 12 if axes["bits"] == 10 else 10),
("init", 0 if axes["init"] else -2)]
for k, v in flips:
w = dict(axes); w[k] = v
s2 = snr(w)
print(f" {k:<12} {str(axes[k]):>8} {str(v):>11} {s2:9.2f} dB "
f"{s2-right:+.2f} dB")
if k in ("order", "variant"):
ck(s2 < right - 2.0, f"axis '{k}' is load-bearing: {s2-right:+.2f} dB")
else:
print(f" SKIPPED: no PCM at {a.audio} -- the bytes were not re-derived")
# --- the finding ----------------------------------------------------------
per = d.cad_a * SECTOR - grp
print(f"""
=== THE PAYLOAD IS NOT THE LUMP (FINDINGS 67) ==========================
A lump is {d.cad_a*SECTOR:,} B of SPACE. {d.cad_f} frames of audio is {grp:,.4f} B, so the
PAYLOAD is {P.lump_bytes(0, d.cad_f, d.fps, d.aud_hz):,} or {P.lump_bytes(2, d.cad_f, d.fps, d.aud_hz):,} -- the same remainder FINDINGS 54's frame
clock carries, one dimension over -- and the last {per:.4f} B are zero.
A PLAYER THAT FED THE CHIP THE WHOLE LUMP would hand it {per:.2f} B a group it
should not have. At {d.aud_hz:,} Hz that is {2*per/d.aud_hz*1000:.2f} ms of audio every
{d.cad_f/d.fps:.4f} s, which is {100*per/grp:.3f}% -- and it does not average out, it ACCUMULATES:""")
for mins in (1.0, a.game_min):
print(f" {mins:5.1f} min of play -> {mins*60*(per/grp):.2f} s of lip-sync error")
print(f""" so the cadence's {100*per/grp:.3f}% is not the waste figure 65.3 called it and left
at that. It is waste ON THE WIRE and DRIFT IN THE PLAYER, and the second is
the expensive one: {a.game_min:.1f} minutes is {a.game_min*60*(per/grp):.2f} s, which is a scene of dialogue
arriving after the mouth that spoke it.
WHAT A PLAYER CARRIES INSTEAD IS ONE ACCUMULATOR, and it is three
instructions rather than a table:
acc += {d.cad_f}*{d.aud_hz:,} ; = {d.cad_f*d.aud_hz:,}
n = acc // {2*d.fps} ; the MTC for this lump's channel
acc %= {2*d.fps}
which is exactly clock.i's shape (54) and for exactly the same reason: a rate
with a denominator of {2*d.fps} cannot be a count, so it is a remainder.
=== THE WIRE ===========================================================
video {d.video_kbps():7.1f} KB/s FIXED by geometry
audio {d.audio_kbps():7.2f} KB/s the CADENCE's, padding included -- the disc moves
whole sectors and the wire pays for the zero ones
total {d.kbps():7.1f} KB/s ({100*(d.kbps()/d.video_kbps()-1):+.2f}%), and 65.3 predicted {589.6:.1f}
""")
print(f"{'FAIL' if fails else 'OK'} 34_packed_audio: {len(fails)} failure(s)")
sys.exit(1 if fails else 0)
tools/analysis/35_audio_level.py
+496
View File
@@ -0,0 +1,496 @@
#!/usr/bin/env python3
"""HOW LOUD IS THE DISC? ROADMAP P6, the item FINDINGS 66.3 reopened.
python3 tools/analysis/35_audio_level.py [--streams 00000-00201] [--json out]
FINDINGS 66 asked MAME's MSM6258 which decoder it is and got four axes back.
The one with a bill attached is the CLAMP: the chip's accumulator is **10 bits**
and it clamps INSIDE the recursion, so the reachable set of reconstructed
samples is [-512, 511] in the 12-bit units everything in this project counts in
-- a quarter of the 12-bit word `adpcm.py` used to clamp at.
`pack.py` hands the encoder `s16 >> 4`, i.e. it maps the disc's full scale onto
the 12-bit word, and 66.3 measured the Singe window peaking at **435 of 511**.
That fit with 1.4 dB to spare, and it fit BY ACCIDENT: the window is a -13.4
dBFS passage. Any passage more than 1.4 dB louder does not merely distort at the
top, it drives the predictor -- a clamped accumulator is a WRONG STATE that the
next nibble is applied to, so the error outlives the loud sample.
So the level cannot be chosen from the ten seconds this project gates on. It has
to be chosen from the loudest thing the game will ever play, and this file
measures that: every stream of the unique scene footage -- `00000`-`00201`,
1366.6 s, FINDINGS 32.1 -- through the SAME chain `extract_audio.py` uses (AC-3
5.1, ffmpeg's default downmix matrix, mono, 15,625 Hz), because a level measured
through a different resampler is a level for a different encoder.
Two statistics, and the difference between them is the whole argument:
PEAK max |x| over the disc. What must fit under 511 for NOTHING to clamp.
PASSAGE the loudest ~1 s window's peak and RMS. What the ear gets. A single
sample 6 dB above everything else is a click and costs one clamp; a
passage 6 dB above the gate window is where the recursion lives for
fifteen thousand samples.
It prints the attenuation each choice implies, in dB and as the shift `pack.py`
would have to make, and it does NOT choose. Choosing needs the other half --
what attenuation costs at the quiet end, where the OKI step table's floor of 16
(12-bit units) does not scale with the signal -- and that is `--ladder`, which
encodes real passages at real gains with `adpcm.CHIP` and reports the SNR.
"""
import argparse, getpass, json, os, subprocess, sys
import numpy as np
sys.path.insert(0, "tools/encoder")
import adpcm
BDROM = os.environ.get("DLX_BDROM") or f"/media/{getpass.getuser()}/BDROM"
STREAM_DIR = f"{BDROM}/BDMV/STREAM"
HZ = 15625 # the chip's rate, and the only one budgeted for
FPS = 12
LUMP_FRAMES = 11 # FINDINGS 65.3's cadence: 11 frames of audio
WIN = LUMP_FRAMES * HZ // FPS # 14,322 samples ~ 0.917 s -- one audio lump
HOP = HZ // 4 # 0.25 s blocks; the window is 4 of them (rounded)
CLAMP_LO, CLAMP_HI = adpcm.clamp_bounds(adpcm.CHIP["bits"]) # -512, 511
FULL12 = 2048 # what `s16 >> 4` maps full scale to
def db(x, ref=FULL12):
return -np.inf if x <= 0 else 20 * np.log10(x / ref)
_PCM_CACHE = {}
def pcm12(stream, start=None, dur=None):
"""One stream as 12-bit signed samples, through extract_audio.py's chain.
Cached, because the scan, the census and the event walk are three passes
over the same 20 million samples and the whole game is 40 MB of int16.
"""
ck = (stream, start, dur)
if ck in _PCM_CACHE:
return _PCM_CACHE[ck]
cmd = ["ffmpeg", "-v", "error"]
if start is not None: cmd += ["-ss", str(start)]
if dur is not None: cmd += ["-t", str(dur)]
cmd += ["-i", f"{STREAM_DIR}/{stream}.m2ts", "-vn", "-ac", "1",
"-ar", str(HZ), "-f", "s16le", "-acodec", "pcm_s16le", "-"]
p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if p.returncode:
# 00176 is 3.0 s of mpeg2video with NO AUDIO TRACK AT ALL. That is a
# fact about the disc, not a failure here, so it is reported rather
# than swallowed -- but anything else is a real error.
if b"does not contain any stream" not in p.stderr:
raise SystemExit(f"ffmpeg failed on {stream}: "
f"{p.stderr.decode(errors='replace')[:400]}")
_PCM_CACHE[ck] = None
return None
x = np.frombuffer(p.stdout, "<i2").astype(np.int32)
# The SAME requantisation pack.py makes. It is a shift and not a divide, so
# it floors toward -inf, and that asymmetry is real: -1 >> 4 is -1.
out = np.clip(x >> 4, -FULL12, FULL12 - 1).astype(np.int16)
_PCM_CACHE[ck] = out
return out
def scan(streams):
"""Per-stream peak and loudest-passage statistics, in 12-bit units."""
rows, mute = [], []
for s in streams:
x = pcm12(s)
if x is None:
mute.append(s)
continue
if x.size == 0:
continue
a = np.abs(x).astype(np.float64)
nb = a.size // HOP
if nb:
bmax = a[:nb * HOP].reshape(nb, HOP).max(1)
bsq = (a[:nb * HOP].reshape(nb, HOP) ** 2).sum(1)
k = max(1, round(WIN / HOP))
if nb >= k:
# sliding sum over k blocks == the ~1 s lump window
cs = np.concatenate(([0.0], np.cumsum(bsq)))
wrms = np.sqrt((cs[k:] - cs[:-k]) / (k * HOP))
wpk = np.array([bmax[i:i + k].max() for i in range(nb - k + 1)])
else:
wrms = np.array([np.sqrt((a ** 2).mean())])
wpk = np.array([a.max()])
else:
wrms = np.array([np.sqrt((a ** 2).mean())])
wpk = np.array([a.max()])
ipk = int(np.argmax(a))
irms = int(np.argmax(wrms))
rows.append(dict(stream=s, n=int(x.size), secs=x.size / HZ,
peak=float(a.max()), peak_t=ipk / HZ,
rms=float(np.sqrt((a ** 2).mean())),
wpeak=float(wpk.max()),
wrms=float(wrms.max()), wrms_t=irms * HOP / HZ))
return rows, mute
def report(rows, mute):
tot = sum(r["secs"] for r in rows)
peak = max(rows, key=lambda r: r["peak"])
loud = max(rows, key=lambda r: r["wrms"])
disc_peak = peak["peak"]
print(f"=== THE DISC, {len(rows)} streams, {tot:,.1f} s = {tot/60:.1f} min "
f"(FINDINGS 32.1 says 1,366.6) ===\n")
if mute:
print(f" {len(mute)} stream(s) carry NO AUDIO TRACK: {', '.join(mute)}"
f" -- a fact about the disc, and a case a shipping encoder has\n"
f" to have an answer for (silence of the right length).\n")
print(f'{"stream":>8}{"secs":>8}{"peak":>7}{"dBFS":>8}{"passage pk":>12}'
f'{"passage rms":>13}{"dBFS":>8} at')
for r in sorted(rows, key=lambda r: -r["wrms"])[:12]:
print(f'{r["stream"]:>8}{r["secs"]:8.1f}{r["peak"]:7.0f}{db(r["peak"]):8.2f}'
f'{r["wpeak"]:12.0f}{r["wrms"]:13.1f}{db(r["wrms"]):8.2f}'
f' {r["wrms_t"]:6.2f} s')
print(" (the twelve loudest PASSAGES; the table is sorted by the window "
"RMS, not the peak)\n")
print(f" DISC PEAK {disc_peak:.0f} of {FULL12} = {db(disc_peak):.2f} dBFS"
f" ({peak['stream']} @ {peak['peak_t']:.2f} s)")
print(f" LOUDEST PASSAGE peak {loud['wpeak']:.0f}, rms {loud['wrms']:.1f}"
f" = {db(loud['wrms']):.2f} dBFS ({loud['stream']} @ {loud['wrms_t']:.2f} s)")
print(f" THE CLAMP +{CLAMP_HI} / {CLAMP_LO} (it is not symmetric), "
f"{db(CLAMP_HI):.2f} dBFS in the same units\n")
need = disc_peak / CLAMP_HI
print("=== WHAT THAT COSTS, AS A LEVEL ===\n")
print(f" `s16 >> 4` is what pack.py does today and it puts the disc's own")
print(f" peak at {disc_peak:.0f} against a clamp of {CLAMP_HI}: "
f"{'OVER by' if need > 1 else 'under by'} {abs(20*np.log10(need)):.2f} dB.")
print(f" Fitting the whole disc under the clamp with no sample clamped at")
print(f" all needs a gain of {1/need:.4f} = {-20*np.log10(need):.2f} dB, i.e.")
for sh in (4, 5, 6, 7):
pk = disc_peak / (1 << (sh - 4))
mark = " <- fits" if pk <= CLAMP_HI else ""
# ~ because a further right shift floors again and this halves; the
# difference is one count and the column is a signpost, not a spec.
print(f" s16 >> {sh} disc peak ~{pk:7.1f} "
f"{'clamps' if pk > CLAMP_HI else 'clear':>6} by "
f"{abs(20*np.log10(pk/CLAMP_HI)):5.2f} dB{mark}")
print()
# How much of the disc is actually above the clamp at today's level: the
# number that decides whether this is a level question or a limiter question.
return dict(rows=rows, disc_peak=disc_peak, peak_stream=peak["stream"],
peak_t=peak["peak_t"], loud_stream=loud["stream"],
loud_t=loud["wrms_t"], loud_rms=loud["wrms"],
loud_wpeak=loud["wpeak"], clamp=CLAMP_HI, mute=mute)
def clip_census(rows, streams, gains):
"""At each candidate gain, how many samples of the WHOLE DISC clamp?
A peak is one number and this is the distribution behind it. A gain that
clamps 12 samples in 22 minutes is a different object from one that clamps
thousands, and the peak alone cannot tell them apart.
"""
print("=== THE CENSUS: how much of the disc is ABOVE the clamp, by gain ===\n")
print(f'{"gain":>8}{"dB":>8}{"samples over":>14}{"of":>12}{"share":>10}'
f'{"worst over":>12}')
tot = 0
over = {g: 0 for g in gains}
worst = {g: 0.0 for g in gains}
for s in streams:
x = pcm12(s)
if x is None:
continue
a = np.abs(x).astype(np.float64)
tot += a.size
for g in gains:
# ROUNDED, exactly as the ladder and pack.py requantise. Comparing
# the float product instead makes 511/946 report one sample over
# its own clamp, which is arithmetic about floats and not about
# the disc.
v = np.round(a * g)
m = v > CLAMP_HI
over[g] += int(m.sum())
if m.any():
worst[g] = max(worst[g], float(v.max() / CLAMP_HI))
for g in gains:
w = f"{20*np.log10(worst[g]):.2f} dB" if worst[g] else "-"
print(f'{g:8.4f}{20*np.log10(g):8.2f}{over[g]:14,}{tot:12,}'
f'{100*over[g]/tot:9.4f}%{w:>12}')
print()
return dict(total=tot, over={f"{g:.4f}": over[g] for g in gains})
def clamp_events(streams, gain=1.0):
"""WHERE the over-clamp samples are, not just how many.
687 isolated samples in 22 minutes and one sustained 44 ms burst are the
same census row and completely different sounds, and a clamp inside a
recursion is not a clipped sample -- it is a wrong predictor state that the
next nibble is applied to. So the run lengths are the statistic.
"""
runs = []
for st in streams:
x = pcm12(st)
if x is None:
continue
m = np.round(np.abs(x).astype(np.float64) * gain) > CLAMP_HI
if not m.any():
continue
d = np.diff(np.concatenate(([0], m.view(np.int8), [0])))
beg = np.where(d == 1)[0]
end = np.where(d == -1)[0]
for b, e in zip(beg, end):
runs.append((int(e - b), st, b / HZ))
runs.sort(reverse=True)
n = sum(r[0] for r in runs)
print(f"=== WHERE THE CLAMPS ARE at gain {gain:.4f} "
f"({len(runs)} events, {n:,} samples = {1000*n/HZ:.1f} ms) ===\n")
print(f'{"run":>6}{"ms":>8} stream at')
for r, st, t in runs[:10]:
print(f'{r:6}{1000*r/HZ:8.2f} {st} {t:7.2f} s')
if runs:
print(f" longest run {runs[0][0]} samples = {1000*runs[0][0]/HZ:.2f} ms; "
f"median run {sorted(r[0] for r in runs)[len(runs)//2]}")
print()
return dict(events=len(runs), samples=n,
longest=runs[0][0] if runs else 0)
def ladder(where, gains, dur, label):
"""Encode a real passage at each gain with adpcm.CHIP and report the SNR.
This is the half a peak measurement cannot do. Attenuation buys headroom at
the top and spends resolution at the bottom, because the OKI step table's
floor is a constant 16 in 12-bit units and does not scale with the signal.
The SNR is reported against the SCALED source, which is the honest
comparison: the encoder's job is to reproduce what it was handed, and the
listener's volume knob is not this project's problem.
"""
stream, start = where
x = pcm12(stream, start, dur)
print(f"=== THE LADDER: {label} -- {stream} @ {start:.2f} s, {dur:.2f} s, "
f"{x.size:,} samples ===\n")
print(f'{"gain":>8}{"dB":>8}{"src peak":>10}{"clamped":>9}{"SNR dB":>9}'
f'{"vs 1.0":>8}')
base = None
out = []
for g in gains:
src = np.clip(np.round(x * g), -FULL12, FULL12 - 1).astype(int).tolist()
nib = adpcm.encode(src, variant=adpcm.CHIP["variant"],
init=adpcm.CHIP["init"], bits=adpcm.CHIP["bits"])
rec = adpcm.decode(nib, variant=adpcm.CHIP["variant"],
init=adpcm.CHIP["init"], bits=adpcm.CHIP["bits"])
s = np.array(src, dtype=np.float64)
r = np.array(rec, dtype=np.float64)
e = ((s - r) ** 2).sum()
snr = 10 * np.log10((s ** 2).sum() / e) if e else np.inf
nclamp = int((np.abs(s) > CLAMP_HI).sum())
if base is None:
base = snr
print(f'{g:8.4f}{20*np.log10(g):8.2f}{np.abs(s).max():10.0f}{nclamp:9,}'
f'{snr:9.2f}{snr-base:+8.2f}')
out.append(dict(gain=g, snr=snr, clamped=nclamp,
peak=float(np.abs(s).max())))
print()
return out
def survey(rows, gains, n, dur, seed=20260825):
"""THE DISC, not three passages of it.
Three hand-picked passages can be argued with; a sample cannot. `n` windows
are drawn uniformly over the game's own timeline -- weighted by stream
length, so a 24 s stream gets twenty times the draws of a 1.2 s one -- and
every one is encoded at every gain with `adpcm.CHIP`. What is reported is
the distribution: the mean SNR is what the level costs on average, and the
WORST window is what it costs where it matters, because a level is chosen
for the passage it fails on.
"""
rng = np.random.default_rng(seed)
pool = [r for r in rows if r["secs"] >= dur]
w = np.array([r["secs"] for r in pool], dtype=np.float64)
w /= w.sum()
picks = []
for _ in range(n):
r = pool[int(rng.choice(len(pool), p=w))]
t = float(rng.uniform(0, r["secs"] - dur))
picks.append((r["stream"], t))
print(f"=== THE SURVEY: {n} windows of {dur:.1f} s drawn over the whole "
f"{sum(r['secs'] for r in rows)/60:.1f} min, encoded at every gain ===\n")
src = [pcm12(st, t, dur) for st, t in picks]
print(f'{"gain":>8}{"dB":>8}{"mean SNR":>10}{"median":>9}{"WORST":>8}'
f'{"windows w/ clamp":>18}{"samples":>9}')
out = []
for g in gains:
snrs, nclamp, ncw = [], 0, 0
for x in src:
v = np.clip(np.round(x * g), -FULL12, FULL12 - 1).astype(int)
k = int((np.abs(v) > CLAMP_HI).sum())
nclamp += k
ncw += 1 if k else 0
nib = adpcm.encode(v.tolist(), variant=adpcm.CHIP["variant"],
init=adpcm.CHIP["init"], bits=adpcm.CHIP["bits"])
rec = np.array(adpcm.decode(nib, variant=adpcm.CHIP["variant"],
init=adpcm.CHIP["init"],
bits=adpcm.CHIP["bits"]), dtype=np.float64)
f = v.astype(np.float64)
e = ((f - rec) ** 2).sum()
snrs.append(10 * np.log10((f ** 2).sum() / e) if e else np.inf)
a = np.array(snrs)
print(f'{g:8.4f}{20*np.log10(g):8.2f}{a.mean():10.2f}'
f'{np.median(a):9.2f}{a.min():8.2f}{ncw:14} of {len(src)}{nclamp:9,}',
flush=True)
out.append(dict(gain=g, mean=float(a.mean()), median=float(np.median(a)),
worst=float(a.min()), clamped=nclamp, windows=ncw))
print()
return out
def recover(stream, gain, control, K=64):
"""DOES A CLAMP OUTLIVE THE SAMPLE IT HAPPENS ON? 66.3 said it would.
The worry was exact and it is the right worry for a recursive codec: a
clamped accumulator is a WRONG STATE and the next nibble is applied to it,
so the error should persist after the loud sample has gone. Measuring the
error after a clamp run does show it elevated -- and that is not evidence,
because the samples after a clamp run are LOUD samples, where the step is
large and the error is large anyway.
So the control is the same window at the gain that never clamps, rescaled
to the same units and read at the SAME sample indices. What the ratio
isolates is the clamp and nothing else.
"""
x = pcm12(stream).astype(np.float64)
def enc(g):
src = np.clip(np.round(x * g), -FULL12, FULL12 - 1).astype(int)
nib = adpcm.encode(src.tolist(), variant=adpcm.CHIP["variant"],
init=adpcm.CHIP["init"], bits=adpcm.CHIP["bits"])
rec = adpcm.decode(nib, variant=adpcm.CHIP["variant"],
init=adpcm.CHIP["init"], bits=adpcm.CHIP["bits"])
return src.astype(np.float64), np.array(rec, dtype=np.float64)
s1, r1 = enc(gain)
s2, r2 = enc(control)
e1 = np.abs(s1 - r1)
e2 = np.abs(s2 - r2) / control * gain # the control, in gain's units
m = np.abs(s1) > CLAMP_HI
d = np.diff(np.concatenate(([0], m.view(np.int8), [0])))
ends = [e for e in np.where(d == -1)[0] if e + K <= e1.size]
p1 = np.array([e1[e:e + K] for e in ends], dtype=np.float64).mean(0)
p2 = np.array([e2[e:e + K] for e in ends], dtype=np.float64).mean(0)
print(f"=== DOES THE CLAMP OUTLIVE THE SAMPLE? {stream}, gain {gain:g} "
f"against a control at {control:g} ===\n")
print(f" {int(m.sum())} samples clamp in {len(ends)} runs; the profile is "
f"the mean |error| at each\n offset after a run ENDS, in 12-bit units, "
f"against the same offsets of a\n window that never clamps at all.\n")
print(f'{"after":>7}{"clamped":>10}{"control":>10}{"ratio":>8}')
for i in (0, 1, 2, 4, 8, 16, 32, K - 1):
print(f'{"+" + str(i):>7}{p1[i]:10.2f}{p2[i]:10.2f}{p1[i]/p2[i]:8.2f}')
off = ~m
print(f'\n off-clamp mean |err| {e1[off].mean():.2f} vs {e2[off].mean():.2f}')
print(f' whole-window mean |err| {e1.mean():.2f} vs {e2.mean():.2f}')
print(f' worst ratio over the {K} offsets: {(p1/p2).max():.2f}\n')
return dict(stream=stream, gain=gain, control=control,
runs=len(ends), clamped=int(m.sum()),
worst_ratio=float((p1 / p2).max()),
mean_err=float(e1.mean()), mean_err_control=float(e2.mean()))
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--first", type=int, default=0)
ap.add_argument("--last", type=int, default=201,
help="the unique scene footage is 00000-00201 (FINDINGS 32.1); "
"00215/00216/00223 are compilations of the same material")
ap.add_argument("--ladder", action="store_true",
help="also encode the loudest and a quiet passage at each gain")
ap.add_argument("--dur", type=float, default=4.0, help="ladder passage seconds")
ap.add_argument("--survey", type=int, default=0,
help="encode N windows drawn over the whole game at each gain")
ap.add_argument("--survey-dur", type=float, default=2.0)
ap.add_argument("--recover", action="store_true",
help="does a clamp outlive its sample? 66.3 said it would")
ap.add_argument("--gate", action="store_true",
help="assert FINDINGS 69's headline numbers, exit 1 if not")
ap.add_argument("--json")
a = ap.parse_args()
streams = [f"{i:05d}" for i in range(a.first, a.last + 1)]
streams = [s for s in streams if os.path.exists(f"{STREAM_DIR}/{s}.m2ts")]
if not streams:
sys.exit(f"no streams under {STREAM_DIR} -- is the Blu-ray mounted? "
f"(DLX_BDROM)")
rows, mute = scan(streams)
summary = report(rows, mute)
# The odd one is not a round number and is not meant to be: it is
# CLAMP/disc peak, the gain at which the disc's own loudest sample lands
# EXACTLY on the clamp, computed from the scan rather than typed in.
exact = round(CLAMP_HI / summary["disc_peak"], 4)
gains = sorted({1.0, 0.7071, exact, 0.5, 0.3536, 0.25}, reverse=True)
summary["exact_gain"] = exact
summary["census"] = clip_census(rows, streams, gains)
summary["events"] = clamp_events(streams, 1.0)
if a.ladder:
# Three passages, because they answer three different questions.
# PEAK what CLAMPING costs, since this is the only place on the disc
# that clamps at today's level.
# LOUD the loudest sustained window that is long enough to encode.
# QUIET what ATTENUATION costs, which is the other end of the same
# decision and the reason -15 dB is not free.
long = [r for r in rows if r["secs"] >= a.dur]
pk = max(rows, key=lambda r: r["peak"]) # the DISC peak, however short
loud = max(long, key=lambda r: r["wrms"])
quiet = min(long, key=lambda r: r["wrms"])
at = lambda r, t: (r["stream"], min(max(0.0, t - a.dur / 2),
max(0.0, r["secs"] - a.dur)))
pkdur = min(a.dur, pk["secs"])
summary["ladder_peak"] = ladder(
(pk["stream"], min(max(0.0, pk["peak_t"] - pkdur / 2),
max(0.0, pk["secs"] - pkdur))),
gains, pkdur, "THE DISC PEAK ITSELF")
summary["ladder_loud"] = ladder(at(loud, loud["wrms_t"]), gains, a.dur,
"THE LOUDEST SUSTAINED PASSAGE")
summary["ladder_quiet"] = ladder(at(quiet, quiet["wrms_t"]), gains, a.dur,
"A QUIET PASSAGE, for the other end")
if a.survey:
summary["survey"] = survey(rows, gains, a.survey, a.survey_dur)
if a.recover:
summary["recover"] = recover(summary["peak_stream"], 1.0, exact)
if a.gate:
expect = dict(disc_peak=946.0, peak_stream="00200", clamp=511,
events=402, over=687)
bad = []
for k, v in expect.items():
got = (summary["events"]["events"] if k == "events" else
summary["events"]["samples"] if k == "over" else summary[k])
if got != v:
bad.append(f"{k}: expected {v}, measured {got}")
if bad:
print("LEVEL GATE RED -- the disc does not measure as FINDINGS 69 "
"recorded it:")
for b in bad:
print(" " + b)
print(" (a different pressing is a legitimate cause; a different "
"ffmpeg downmix is not)")
sys.exit(1)
print("LEVEL GATE GREEN: disc peak 946 of 2048 at 00200, 5.35 dB over "
"the chip's 511,\n 687 samples in 402 events = 44.0 ms of the "
"game's 21.5 min of audio.")
if a.json:
json.dump(summary, open(a.json, "w"), indent=1)
print(f"-> {a.json}")
if __name__ == "__main__":
main()
tools/analysis/36_branch_audio.py
+440
View File
@@ -0,0 +1,440 @@
#!/usr/bin/env python3
"""51.3's REFILL CLIMB WITH A SECOND CONSUMER, THROUGH A REAL BRANCH POINT.
python3 tools/analysis/36_branch_audio.py --kbps R [R ...]
[--ring KB ...] [--gate]
The oldest open item in ROADMAP P6, named by FINDINGS 65.6 and again by 67.6:
"the slack table is here, but 51.3's refill climb with a second consumer
through a real branch point is not."
Everything it needs already exists and none of it has ever been put in the same
room:
* 51.3 -- slack is ACCUMULATED out of `pipe - wire`, at `pipe - wire` bytes a
second, and a seek spends all of it. `tools/analysis/20_seek_slack.py`.
* 56.3 -- where the branch points ARE: 612 distinct transitions into a seek
over the arcade's own graph, worst gap 0.000 s, median 3.473 s.
`tools/analysis/25_scene_graph.py`, reading only DLXSCENE1.
* 56.4 -- the climb against that distribution, for the codec container. It
charged audio as `ratectl.AUDIO_KBPS`, a flat 7.8 KB/s placeholder that
predates any of the audio work.
* 65.3/67.1 -- what a second consumer ACTUALLY costs a container: a fixed
cadence of F frames per A sectors, because a packed record's address is
arithmetic and cannot be an index. `tools/analysis/32_audio_wire.py`.
* 68 -- the player that holds both streams at once, and its buffers.
Three questions, and the tree has never asked any of them:
1. What does the SECOND CONSUMER do to the climb? Not to the wire -- 32
answered that and it is 1.3% -- but to `pipe - wire`, which is a small
difference of two large numbers and is the thing the climb is made of.
2. What is the climb on the PACKED branch? This is the branch the player
runs (68) and the one B1's acceptance is written against.
3. What does the CADENCE do at a branch point? A group is `lump k, then F
records`, so lump k sits at a LOWER address than every record in its group
but the first. A seek to record i lands inside a group whose audio is
BEHIND it. Nobody has ever priced entering a group off-boundary, and the
game's own seek targets say how often it happens.
THE FRAME INDEX OF A SEEK TARGET IS A DESIGN ASSUMPTION AND IS LABELLED ONE.
DLXSCENE1 carries positions on the laserdisc timeline in ms. This tree's design
puts ONE CONTAINER PER SCENE -- 53 and 55.1 charge a scene change 6,164 header
bytes, and 56.3 counts 203 of the 612 transitions as container changes for
exactly that reason -- so a seek target's frame index inside its container is
`(target start - the scene's own earliest start) * fps / 1000`. If the design
ever puts one container per SEQUENCE instead, every seek lands on frame 0, the
group offset is always zero and section 3 collapses to nothing. That is the
assumption, said out loud, in the one place the answer depends on it.
"""
import sys, os, json, argparse, importlib.util
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/analysis")
TABLE = os.environ.get("DLX_SCENEGRAPH", "tmp/scenegraph.json")
FPS = 12
CHIP_HZ = 15625.0 # MSM6258V, 8 MHz / 512 (FINDINGS 32, 65)
AU_BPS = CHIP_HZ / 2 # 4 bits a sample, two samples to a byte
AU_FRAME = AU_BPS / FPS # 651.0416... B a slot, and the dots are 65.3
SECTOR = 512
def load(path, name):
spec = importlib.util.spec_from_file_location(name, path)
m = importlib.util.module_from_spec(spec)
spec.loader.exec_module(m)
return m
def cadence(F):
"""Best A for this F: the fewest whole sectors that hold F frames of chip."""
num, den = int(round(AU_BPS * 2)), 2 * SECTOR * FPS
A = -(-(num * F) // den) # ceil(F * bytes/sector)
return A, A * SECTOR
def climb_s(records, rec_b, pipe_kbps, wire_kbps):
"""Seconds of play to accumulate `records` records of lookahead (51.3)."""
surplus = (pipe_kbps - wire_kbps) * 1024.0
return float("inf") if surplus <= 0 else records * rec_b / surplus
# --------------------------------------------------------------- the branches
def branch_points(doc, nodes, sg):
"""The 612 transitions into a seek, plus each one's target frame index.
Returns (gaps_s, within, changes). `within` entries carry the frame index
the seek lands on inside its container; `changes` are scene changes, which
land on frame 0 of a new one.
"""
gaps, _ = sg.worst_gap(nodes)
play = [g for g in gaps if g[5] != "attract_mode"]
scene_start = {}
for scene, seqs in doc["scenes"].items():
st = [s["start_ms"] for s in seqs.values() if s["start_ms"] >= 0]
scene_start[scene] = min(st) if st else None
within, changes = [], []
for g, src, tgt, kind, ends, scene in play:
if ends:
changes.append((g / 1000.0, src))
continue
n = nodes.get(f"{scene}.{tgt}")
if n is None or not n.seeks or scene_start[scene] is None:
continue
i = int(round((n.start - scene_start[scene]) / 1000.0 * FPS))
within.append((g / 1000.0, max(0, i), f"{scene}.{tgt}"))
return sorted(g[0] / 1000.0 for g in play), within, changes
def silences(within, F):
"""Ms of silence entering each branch's group off-boundary, at cadence F.
A group is `lump k, then F records`. Seek to record i, take the next lump
that lies AHEAD of the read point -- lump k+1, which arrives at frame
(k+1)*F -- and the frames from i to (k+1)*F-1 have no audio. The chip's
second is a real second (15,625 samples, 2 to a byte), so the missing
time is exactly `(F - i mod F) mod F` frames of 1/12 s and none of
FINDINGS 54's frame-clock remainder gets into it.
"""
return sorted(((F - (i % F)) % F) / FPS * 1000.0 for _, i, _ in within)
def pct(xs, p):
return xs[min(len(xs) - 1, int(p * len(xs)))] if xs else float("nan")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--kbps", type=float, nargs="+",
help="delivered pipe rates, KB/s. REQUIRED (FINDINGS 50) "
"unless --gate, which carries its own.")
ap.add_argument("--ring", type=float, nargs="+", default=[256, 512])
ap.add_argument("--pkbps", type=float, nargs="+",
default=[589.6, 600.0, 650.0, 700.0, 900.0],
help="pipe rates for the PACKED table. Its own list, "
"because a packed record is 1.3x the codec's and "
"every rate that serves one starves the other.")
ap.add_argument("--table", default=TABLE)
ap.add_argument("--packed", default="tmp/packed_singe.dlxp")
ap.add_argument("--codec", default="tmp/rc_fr_singe_scsi_span.dlx")
ap.add_argument("--fsweep", type=int, default=12,
help="highest cadence F in the pick table")
ap.add_argument("--gate", action="store_true",
help="check.sh mode: fixed rates, and assert the structural "
"results rather than print the essay")
a = ap.parse_args()
if a.gate and not a.kbps:
a.kbps = [451.4, 488.0, 600.0]
if not a.kbps:
ap.error("--kbps is required and has no default (FINDINGS 50)")
if not os.path.exists(a.table):
print(f"no scene table at {a.table} -- run:\n"
f" python3 tools/import/scenegraph.py")
return 2
doc = json.load(open(a.table))
if doc.get("format") != "DLXSCENE1":
print(f"{a.table}: not a DLXSCENE1 table")
return 2
sg = load("tools/analysis/25_scene_graph.py", "scene_graph")
ss = load("tools/analysis/20_seek_slack.py", "seek_slack")
from dlxp import DLXP
import ratectl as RC
nodes = sg.build_graph(doc["scenes"])
gaps, within, changes = branch_points(doc, nodes, sg)
med = gaps[len(gaps) // 2]
d = DLXP(a.packed)
P_REC = d.rec_bytes
P_VID = P_REC * FPS / 1024
F0, A0 = d.cad_f, d.cad_a
P_AUD = A0 * SECTOR / F0 * FPS / 1024
print(f"""
=== WHAT EACH BRANCH ACTUALLY HOLDS ====================================
51.3's climb is a statement about an ACCUMULATOR. The two branches of this
project do not have the same one, and one of them does not have one at all.
codec ({os.path.basename(a.codec)})
a {a.ring[0]:.0f} KB ring of variable records with an index in front of it;
lookahead is whole records and the ceiling is what 20_seek_slack.py
simulates.
packed ({os.path.basename(a.packed)})
a record is {P_REC:,} B of literal picture and the channel puts it
STRAIGHT INTO GVRAM (FINDINGS 61, 62, 64). There is no record
buffer, so the VIDEO lookahead is ZERO records and there is nothing
to climb. The only consumer on that branch with any lookahead at
all is the AUDIO one: {d.cad_a * SECTOR:,} B a lump, PG_ANBUF={3} slots and
PG_APRE prefilled at {2} for the run 68 measured (it is a mailbox,
not a constant: src/player/packed.s, 68.6).
That is {2 * F0 / FPS:.3f} s of sound held against {0.0:.3f} s of picture.
The CPU-painted packed variant (64.2's column B) is the one that holds two
record buffers and 99,328 B, and it is the only packed configuration the
word "climb" applies to. Both are priced below.
=== THE BRANCH POINTS, OUT OF THE ARCADE'S OWN GRAPH ====================
{len(gaps)} transitions into a seek (attract mode excluded, 56.3)
worst {gaps[0]:.3f} s p10 {pct(gaps,.10):.3f} median {med:.3f} p90 {pct(gaps,.90):.3f}
{len(changes)} of them END THE SCENE and are container changes;
{len(within)} land INSIDE a container, at frame indices 0..{max(i for _, i, _ in within)}""")
# ---------------------------------------------------------------- 1
dc, rec = ss.records(a.codec)
C_REC = float(rec.mean())
C_VID = C_REC * FPS / 1024
print(f"""
=== 1. THE CLIMB WITH THE SECOND CONSUMER (the codec branch) ============
Audio is {AU_BPS/1024:.3f} KB/s and this container pays no padding for it (65.4: it
already has an index and already has variable records). Against a video wire
of {C_VID:.1f} KB/s that is {100*(AU_BPS/1024)/C_VID:+.2f}% -- and the climb is not built out of the wire,
it is built out of `pipe - wire`, so that is not the number that matters.
{'ring':>5} {'pipe':>7} {'ceil':>5} {'climb SILENT':>13} {'SOUNDED':>9} {'x':>6}"""
f" {'under, silent':>13} {'under, sounded':>14}")
tab1 = []
for ring_kb in a.ring:
ring = int(ring_kb * 1024)
for kbps in a.kbps:
fill = (kbps - AU_BPS / 1024) * 1024 / FPS
lo, hi, ring_ref, rate_ref = ss.paced_sim(rec, ring, fill)
ceil = int(hi.max())
cs = climb_s(ceil, C_REC, kbps, C_VID)
ca = climb_s(ceil, C_REC, kbps, C_VID + AU_BPS / 1024)
us = sum(1 for g in gaps if g < cs)
ua = sum(1 for g in gaps if g < ca)
ratio = ca / cs if cs not in (0.0, float("inf")) else float("inf")
tab1.append((ring_kb, kbps, ceil, cs, ca, ratio, us, ua))
print(f" {ring_kb:5.0f} {kbps:7.1f} {ceil:5d} {cs:12.2f}s "
f"{ca:8.2f}s {ratio:5.2f}x {f'{us}/{len(gaps)}':>13} "
f"{f'{ua}/{len(gaps)}':>14}")
worst = max(tab1, key=lambda r: r[5])
print(f"""
THE SECOND CONSUMER IS {100*(AU_BPS/1024)/C_VID:.1f}% OF THE WIRE AND UP TO {worst[5]:.2f}x OF THE CLIMB.
At {worst[1]:.1f} KB/s in a {worst[0]:.0f} KB ring the climb goes {worst[3]:.2f} s -> {worst[4]:.2f} s and the
branch points that arrive under it go {worst[6]}/{len(gaps)} -> {worst[7]}/{len(gaps)}
({100*worst[6]/len(gaps):.0f}% -> {100*worst[7]/len(gaps):.0f}%). Nothing about audio got bigger; the
DIFFERENCE it is subtracted from got smaller, and the climb is made of the
difference. This is why 51.4's rate/ring distinction matters more with a
second consumer than without one, and why quoting audio as a share of the
wire (32, and every budget before it) understates it at every rate close to
the wire.
A CORRECTION TO 56.4, and it is small: that table charged audio at
ratectl.AUDIO_KBPS = {RC.AUDIO_KBPS} KB/s, which is {AU_BPS:.1f} B/s expressed in decimal
kB (21_iplrom_dmac.py says so). In binary KB the figure is {AU_BPS/1024:.4f}, i.e.
{100*(RC.AUDIO_KBPS-AU_BPS/1024)/(AU_BPS/1024):+.2f}%. Every column of 56.4 moves in the flattering direction by
less than one part in six hundred of the wire. It is recorded because a
placeholder that turns out to be right is still a placeholder.""")
# ---------------------------------------------------------------- 2
print(f"""
=== 2. THE PACKED BRANCH: THERE IS NO CLIMB, AND THAT IS THE FINDING ====
video {P_VID:.1f} + audio {P_AUD:.4f} (F={F0}, A={A0}, {A0*SECTOR:,} B a lump) = {P_VID+P_AUD:.1f} KB/s,
which is B1's acceptance figure and is where it comes from.
DMAC-direct (what src/player/packed.s runs, FINDINGS 64/68):
video lookahead 0 records. The climb does not exist, the ceiling does not
exist, and the {len(gaps)} gaps buy it NOTHING -- there is no accumulator for
play to fill. Its acceptance is a PER-FRAME deadline: {P_REC:,} B must land
inside every slot, and a rate that averages {P_VID+P_AUD:.1f} KB/s over a second is
not the same claim. 56.4's alarming column -- most branch points arrive
with less lookahead than the one before them -- does not apply to it,
because every frame arrives with less lookahead than the one before it.
CPU-painted (64.2 column B, 99,328 B, two record buffers -> 1 record of
lookahead):
{'pipe':>7} {'climb SILENT':>13} {'SOUNDED':>9} {'x':>6} {'under, silent':>13} {'under, sounded':>14}""")
tab2 = []
def secs(x, w):
return f"{x:{w}.2f}s" if x != float("inf") else f"{'never':>{w+1}}"
for kbps in a.pkbps:
cs = climb_s(1, P_REC, kbps, P_VID)
ca = climb_s(1, P_REC, kbps, P_VID + P_AUD)
us = sum(1 for g in gaps if g < cs)
ua = sum(1 for g in gaps if g < ca)
r = ca / cs if cs not in (0.0, float("inf")) else float("inf")
tab2.append((kbps, cs, ca, r, us, ua))
print(f" {kbps:7.1f} {secs(cs,12)} {secs(ca,8)} "
f"{(f'{r:5.2f}x' if r != float('inf') else ' inf ')} "
f"{f'{us}/{len(gaps)}':>13} {f'{ua}/{len(gaps)}':>14}")
print(f" -- and EVERY rate in section 1's table is below {P_VID:.1f} KB/s, so "
f"none of\n them serves this container at all.")
print(f"""
READ THE FIRST ROW. At {P_VID+P_AUD:.1f} KB/s -- the acceptance figure this project
quotes -- the SILENT container still climbs its one record in {tab2[0][1]:.2f} s and
the SOUNDED one NEVER DOES, because {P_VID+P_AUD:.1f} is where its surplus is exactly
zero. The acceptance figure is the rate at which the sounded container has
no lookahead at any amount of play, which is a different thing from the rate
at which it plays.
A packed record is 1.3x the codec's mean record and the packed wire is 1.3x
the codec's, so a rate that is generous to one is tight for the other and the
same audio debit costs the packed climb more. ONE RECORD of lookahead is
1/12 s of tolerance and it takes seconds of play to earn.
THE TWO BRANCHES DIFFER HERE ON A COLUMN THAT IS NOT CLOCKS, which is the
third time (61.9, 64.2, and this). The packed branch spent its ring to
delete a decoder; what it bought with the RAM is a player with no tolerance
for a slow record at ANY time, not merely after a branch.""")
# ---------------------------------------------------------------- 3
sil = silences(within, F0)
free = sum(1 for x in sil if x == 0.0)
P_PIPE = a.pkbps[1] if len(a.pkbps) > 1 else a.pkbps[0]
lump_ms = A0 * SECTOR / (P_PIPE * 1024) * 1000
behind = (F0 - 1) * P_REC + A0 * SECTOR
print(f"""
=== 3. THE COST NOBODY HAD COUNTED: entering a group off-boundary =======
A DLXP2 group is `lump k, then F records` (dlxp.py), so lump k is at a LOWER
address than every record of its group except the first. Reading forward from
record i, the next lump to arrive is k+1, and it carries frame (k+1)*F. The
frames from i to (k+1)*F-1 therefore have picture and no sound.
Measured on the {len(within)} within-container seek targets of the arcade's own
graph, at the shipped cadence F={F0}:
mean {sum(sil)/len(sil):7.1f} ms of silence entering the branch
median {pct(sil,.50):7.1f} p90 {pct(sil,.90):7.1f} worst {sil[-1]:7.1f}
free {free}/{len(within)} land on a group boundary and cost nothing
The other {len(changes)} branch points -- the scene changes -- are FREE, and by
construction: lump 0 sits at sector 1 and record 0 at {d.off_frm:,}, so a
container's own first bytes are header, lump, record and a scene change reads
them in one forward pass. **The container's start is the one branch point
the cadence costs nothing at, and it is the only one anybody had looked at.**
THE FIX IS A SECOND READ AND NOBODY HAS ONE. Lump k is {behind:,} B behind
record i at worst, so it cannot be picked up by reading early -- it is a
separate command at a separate LBA, of {A0*SECTOR:,} B, which at {P_PIPE:.1f} KB/s is
{lump_ms:.1f} ms against a mean {sum(sil)/len(sil):.0f} ms of silence -- {sum(sil)/len(sil)/lump_ms:.0f}x cheaper in TIME,
one more command per branch, and the command overhead is B1's and unmeasured.
src/player/packed.s starts PG_AK and PG_AKF at lump 0 and has no audio seek
path at all; the player that branches needs one.
=== 4. THE CADENCE PICK, WITH THE THIRD COLUMN IT DID NOT HAVE ==========
32_audio_wire.py chose F={F0} on two columns, padding and RAM. Here is the same
sweep with the branch column, measured on the game's own seek targets rather
than assumed uniform:
{'F':>3} {'A':>3} {'lump B':>8} {'pad%':>7} {'aud KB/s':>9} {'RAM x2':>8} {'mean sil':>9} {'p90':>8} {'worst':>8} {'free':>10} {'vs uniform':>11}""")
for F in range(1, a.fsweep + 1):
A, lump = cadence(F)
need = F * AU_FRAME
s = silences(within, F)
uni = (F - 1) / 2 / FPS * 1000.0
mean = sum(s) / len(s)
mark = " <- shipped" if F == F0 else ""
print(f" {F:3d} {A:3d} {lump:8,} {100*(lump-need)/need:6.2f}% "
f"{lump/F*FPS/1024:8.3f} {2*lump:8,} {mean:8.1f} "
f"{pct(s,.90):8.1f} {s[-1]:8.1f} "
f"{f'{sum(1 for x in s if x == 0)}/{len(s)}':>10} "
f"{(mean/uni if uni else 1.0):10.2f}x{mark}")
ratios = []
for F in range(2, a.fsweep + 1):
sF_ = silences(within, F)
ratios.append((sum(sF_) / len(sF_)) / ((F - 1) / 2 / FPS * 1000.0))
min_r, max_r = min(ratios), max(ratios)
A1, l1 = cadence(1)
AF, lF = cadence(F0)
s1, sF = silences(within, 1), silences(within, F0)
print(f"""
F=1 -- "one lump a record", the cadence 32 called THE WORST ONE -- has no
group to enter off-boundary, no second read, no audio seek path and 2,048 B
of held lump instead of {2*lF:,}. It costs {l1/1*FPS/1024 - lF/F0*FPS/1024:+.3f} KB/s of wire, which is
{100*(l1/1*FPS/1024 - lF/F0*FPS/1024)/(P_VID+P_AUD):+.2f}% of the packed acceptance figure, and it BUYS BACK {2*lF-2*l1:,} B
of RAM on the branch whose whole argument is that RAM is what it has spare.
THE PICK IS THEREFORE REOPENED, and it is a real trade rather than an error:
padding is what F={F0} minimises and padding is not the only thing F sets.
A player that gets its audio seek right is indifferent; a player that does
not pays a mean {sum(sF)/len(sF):.0f} ms of silence at {len(within)} of the game's {len(gaps)} branch
points. Nothing here decides it -- the deciding number is the SCSI command
overhead of the extra read, and that is B1's.
AND THE CONTENT IS NOT UNIFORM MOD F. A uniform assumption would put the
mean at (F-1)/2 frames; the arcade's seek targets land where they land, and
the ratio column above runs {min_r:.2f}x..{max_r:.2f}x over the sweep, so a design
that assumed uniform would be out by a quarter at F=3. At the shipped F={F0} it is
{(sum(sF)/len(sF))/((F0-1)/2/FPS*1000):.2f}x, which is a coincidence and is reported as one.
=== 5. WHAT THIS DOES NOT ESTABLISH ====================================
1. NO RATE HERE IS MEASURED. Every pipe column is a sensitivity (FINDINGS
50), and B1 -- sustained AND data-phase burst -- is still the user's.
2. THE FRAME INDEX OF A SEEK TARGET IS A DESIGN ASSUMPTION. One container
per SCENE (53, 55.1, 56.3). One container per SEQUENCE makes section 3
zero and section 4 moot; nothing else in the file changes.
3. NOTHING RAN ON THE MACHINE. This is arithmetic over a scene table, two
containers and a player's own constants. 68's player has never seeked.
4. THE MECHANICAL SEEK IS STILL UNMODELLED (51.7.5) and is charged on top of
every millisecond here.
5. THE SILENCE IS A CONTAINER PROPERTY, NOT A CHIP ONE. What the MSM6258
does when it is not fed -- hold the last sample, or click -- is a board
question and belongs with session 34's fifth hardware item.""")
if a.gate:
# Structural assertions. Not the milliseconds -- those move with the
# scene table -- but the ORDER and the SIGNS, which are the finding.
ok = True
def check(cond, msg):
nonlocal ok
print(f" {'OK ' if cond else 'FAIL'} {msg}")
ok = ok and bool(cond)
print("\n=== GATE ===============================================")
check(len(gaps) == 612, f"612 transitions into a seek, got {len(gaps)}")
check(len(within) + len(changes) == len(gaps),
f"{len(within)} within + {len(changes)} scene changes = {len(gaps)}")
check(all(r[5] >= 1.0 for r in tab1),
"audio never SHORTENS the codec climb")
check(max(r[5] for r in tab1) > 1.5,
f"and at some rate it more than 1.5x's it "
f"({max(r[5] for r in tab1):.2f}x)")
check(all(r[7] >= r[6] for r in tab1),
"and never lowers the count of branch points under the climb")
check(free < len(within) // 2,
f"most within-container branches enter a group off-boundary "
f"({len(within)-free}/{len(within)})")
check(silences(within, 1) == [0.0] * len(within),
"F=1 has no off-boundary case at all")
check(sum(sil) / len(sil) > 10 * lump_ms,
f"the silence F={F0} costs is >10x the lump read that removes it "
f"({sum(sil)/len(sil):.0f} ms vs {lump_ms:.1f} ms)")
print(" " + ("BRANCH-AUDIO GATE GREEN" if ok else "BRANCH-AUDIO GATE RED"))
return 0 if ok else 1
return 0
if __name__ == "__main__":
sys.exit(main())
tools/analysis/37_audio_seek.py
+238
View File
@@ -0,0 +1,238 @@
#!/usr/bin/env python3
"""What a BRANCH costs the chip, and what an encoder could do about it.
python3 tools/analysis/37_audio_seek.py [container.dlxp] [--raw au.raw]
python3 tools/analysis/37_audio_seek.py --gate # the check.sh stage
FINDINGS 71. Session 39 put an audio seek path in src/player/packed.s and ran
it: 132,162 B across a real branch, every byte accounted for in MAME's own
capture. That settled the BYTES. This settles what is left, which is a
property of the codec rather than of the player and which no counter in the
player can reach.
THE MSM6258'S ACCUMULATOR HAS NO LEAKAGE TERM. It is a pure integrator of
deltas, clamped, and nothing pulls it back toward zero. So a branch that hands
the chip bytes chosen for a state it is not in does not produce a transient with
a time constant -- it produces a DC OFFSET THAT NEVER DECAYS. The machine run
measures both designs at one branch point; this measures the CENSUS, over every
frame boundary of the container, and prices the only fix that is worth anything,
which is in the encoder and not in the player.
* PLAY THROUGH the branch: the chip keeps whatever accumulator and step index
the previous scene's audio left it in. Unbounded, and its decay is the
signal's own clamping rather than the recursion forgetting.
* STOP and re-PLAY: the accumulator goes to the container's `init` and the
step index to 0 -- a state this script knows exactly, so the error is
EXACTLY `init - acc(target)`, constant, forever.
* ...and the third option is the ENCODER'S: encode the stream with the
predictor RESET at every point a branch can land on. Then a re-PLAYing
player is not close, it is exact. What that costs is a codec question and
is measured below.
NAME THE LAYER. Everything here is host arithmetic over one container and its
source PCM. The chip's four axes are the ones FINDINGS 66 measured on the
machine and 67.3 put in the header; the branch behaviour is the one session 39
ran. No emulator is involved and no rate is claimed.
"""
import argparse, math, os, statistics, sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "encoder"))
import adpcm
from dlxp import DLXP
def acc_trajectory(nibbles, dec):
"""The decoder's accumulator after every sample. This IS the encoder's
assumed state, because adpcm.encode runs its decoder inside its own search
loop -- the encoder cannot hold a state the decoder will not reach."""
lo, hi = adpcm.clamp_bounds(dec["bits"])
sig, idx = dec["init"], 0
out = []
for n in nibbles:
sig += adpcm.delta(n, adpcm.STEP[idx], dec["variant"])
sig = lo if sig < lo else (hi if sig > hi else sig)
idx += adpcm.INDEX_ADJUST[n & 7]
idx = 0 if idx < 0 else (48 if idx > 48 else idx)
out.append((sig, idx))
return out
def encode_reset(src12, dec, period):
"""adpcm.encode with the predictor forced back to (init, 0) every `period`
samples. period=None is the ordinary encode.
THIS CHANGES THE BYTES, so it is a container property and not a flag a
player can set. It is written here rather than in tools/encoder/adpcm.py
because nothing has decided to ship it: 71.5 is the trade and the deciding
number is a hardware one."""
if period is None:
return adpcm.encode(src12, variant=dec["variant"], init=dec["init"],
bits=dec["bits"])
out = bytearray()
for i in range(0, len(src12), period):
out += adpcm.encode(src12[i:i + period], variant=dec["variant"],
init=dec["init"], bits=dec["bits"])
return bytes(out)
def snr(ref, got):
n = min(len(ref), len(got))
sig = sum(x * x for x in ref[:n])
err = sum((ref[i] - got[i]) ** 2 for i in range(n))
if err == 0:
return float("inf")
return 10 * math.log10(sig / err) if sig else float("-inf")
def decode_reset(nib, dec, period):
if period is None:
return list(adpcm.decode_state(nib, variant=dec["variant"],
init=dec["init"], bits=dec["bits"])[0])
out = []
for i in range(0, len(nib), period):
out += list(adpcm.decode_state(nib[i:i + period], variant=dec["variant"],
init=dec["init"], bits=dec["bits"])[0])
return out
def main():
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/packed_singe.dlxp")
ap.add_argument("--raw", default="tmp/au_singe.raw")
ap.add_argument("--gate", action="store_true")
a = ap.parse_args()
d = DLXP(a.container)
if not d.has_audio:
sys.exit("this container is silent -- there is no branch to price")
dec = d.decoder()
lo, hi = adpcm.clamp_bounds(dec["bits"])
data = d.audio()
nib = adpcm.unpack(data, len(data) * 2, order=dec["order"])
traj = acc_trajectory(nib, dec)
fails = []
def ck(ok, msg):
print(("OK " if ok else "FAIL ") + msg)
if not ok:
fails.append(msg)
# ---- 1. THE CENSUS. What a re-PLAY costs at every frame boundary the
# container has, which is every point a branch in this design can land on:
# 56.3's targets are frame indices and this player seeks to a frame.
den = 2 * d.fps
# THE INDEX IS A NIBBLE INDEX AND THE POSITION IS A BYTE ONE, which is the
# one conversion in this file and it is worth the line: getting it wrong
# reads the trajectory at HALF the target and produces a census that is
# entirely plausible -- a distribution of the right shape over the wrong
# instants. The cross-check against the machine's own branch point below is
# what caught it.
pos = lambda f: 2 * (f * d.aud_hz // den) - 1
frames = [f for f in range(1, d.nframes) if pos(f) < len(traj)]
dcs = [abs(dec["init"] - traj[pos(f)][0]) for f in frames]
idxs = [traj[pos(f)][1] for f in frames]
dcs_s = sorted(dcs)
p = lambda q: dcs_s[min(len(dcs_s) - 1, int(q * len(dcs_s)))]
print(f"--- 1. A RE-PLAYED BRANCH COSTS `init - acc(target)`, EXACTLY AND "
f"FOREVER. {len(frames)} frame boundaries of {a.container}:")
print(f" |DC| against the {hi}-unit clamp: mean {statistics.mean(dcs):.1f} "
f"({statistics.mean(dcs)*100/hi:.1f}%), median {statistics.median(dcs):.0f}, "
f"p90 {p(0.90)}, worst {max(dcs)} ({max(dcs)*100/hi:.1f}%)")
print(f" ...and the step index the encoder assumed at those points runs "
f"{min(idxs)}..{max(idxs)} of 48, median {statistics.median(idxs):.0f} "
f"-- a re-PLAY sets it to 0, so a branch into a LOUD passage gets the "
f"offset AND a step index that has to climb back")
# The machine run's own branch, so the two layers are checked against each
# other rather than merely agreeing in prose.
F37 = 37
dc37 = dec["init"] - traj[pos(F37)][0]
print(f" frame {F37}, the branch tools/bench/packed_run.sh runs on the "
f"machine: DC {dc37} -- and MAME's capture measured the chip at "
f"exactly that, constant over 62,500 samples (FINDINGS 71.3)")
ck(abs(dc37) == 65,
f"the host's arithmetic for the machine's own branch point is {abs(dc37)} "
f"and the capture said 65 -- one number, two layers")
# ---- 2. THE DECAY THAT ISN'T. A re-PLAY's error is constant BY
# CONSTRUCTION -- same step index, same nibbles, one offset -- and playing
# through is not, because the step indices differ too. The point of
# measuring it here is that the constancy is a PROPERTY OF THE PREDICTOR
# and not of the ten seconds this container happens to hold.
print(f"--- 2. AND IT DOES NOT DECAY. The accumulator is an integrator with "
f"no leak: a re-PLAY changes the STARTING value and nothing else, so "
f"the same nibbles produce the same deltas and the offset is carried "
f"to the end of the stream. The machine agrees -- AC 0.00 over four "
f"seconds (FINDINGS 71.3). Playing THROUGH the branch is worse and is "
f"not constant, because the step index differs as well: -355 falling "
f"to -108 over four seconds, which is clamping and not forgetting.")
# ---- 3. THE ENCODER'S FIX, PRICED. Reset the predictor where a branch can
# land and a re-PLAYing player is EXACT rather than close.
if not os.path.exists(a.raw):
print(f" (no {a.raw}: the encoder trade below needs the source PCM)")
return 1 if fails else 0
import struct
pcm = struct.unpack("<%dh" % (os.path.getsize(a.raw) // 2),
open(a.raw, "rb").read())
src12 = [max(-2048, min(2047, x >> 4)) for x in pcm][:len(nib)]
per_frame = d.aud_hz // den * 2 # samples in one frame slot
print(f"--- 3. THE ONLY FIX THAT MAKES A BRANCH FREE IS THE ENCODER'S, and "
f"here is its bill. Reset the predictor every N frames when encoding; "
f"a player that re-PLAYs at a branch landing on one of those points is "
f"then EXACT, not close:")
print(f" {"reset every":>24} {'SNR dB':>8} {'vs shipped':>10} "
f"{'branch points made free':>24}")
base = None
rows = []
for label, period in [("never (shipped)", None),
(f"{d.cad_f} frames (the cadence)", d.cad_f * per_frame),
("1 frame", per_frame)]:
nb = encode_reset(src12, dec, period)
got = decode_reset(nb, dec, period)
v = snr(src12, got)
if base is None:
base = v
free = (0 if period is None
else (len(frames) // d.cad_f if period != per_frame
else len(frames)))
rows.append((label, v, v - base, free))
print(f" {label:>24} {v:8.2f} {v-base:+10.2f} "
f"{free:>15} of {len(frames)}")
# THE ASSERTION IS THE ORDER AND THE SIGN, not the decibel: the source PCM
# is a property of the disc and the encoder is greedy, so the exact figures
# move with the window. What must not move is that resetting COSTS SNR and
# that resetting more often costs more -- if it ever came out free, the
# predictor would not be doing anything and the codec would be pointless.
ck(rows[1][1] <= rows[0][1] + 1e-9 and rows[2][1] <= rows[1][1] + 1e-9,
f"resetting the predictor costs SNR, and resetting it more often costs "
f"more: {rows[0][1]:.2f} -> {rows[1][1]:.2f} -> {rows[2][1]:.2f} dB")
ck(rows[2][1] > rows[0][1] - 3.0,
f"...and a reset EVERY FRAME is {rows[0][1]-rows[2][1]:.2f} dB, which is "
f"the price of making all {len(frames)} of this container's branch points "
f"exact. The step table's floor is a constant 16 and the recursion "
f"re-converges in a few samples, which is why twelve resets a second is "
f"not twelve times anything")
print(f"--- 4. WHAT THIS DOES NOT SETTLE.")
print(f" * Nothing here is a rate and nothing here ran on silicon. The "
f"branch behaviour is MAME's okim6258 -- PLAY sets the accumulator to "
f"-2, the step index to 0 and the nibble select to 0 -- which is the "
f"model FINDINGS 66 fitted to the machine and NOT a measurement of an "
f"MSM6258V. It joins session 34's fifth hardware item.")
print(f" * The census is ONE container, ten seconds, one passage at "
f"-13.4 dBFS (FINDINGS 69). The offset a re-PLAY costs is the signal's "
f"own value at the cut, so a louder passage costs more, up to the "
f"clamp -- and the disc peaks at 946 of 2048 (69.2).")
print(f" * The reset-every-frame encode is NOT in tools/encoder. It is "
f"a container change (a DLXP3), it costs bytes nothing and SNR "
f"something, and what decides it is whether a branch is allowed to "
f"land anywhere or only on frames the encoder was told about.")
print("AUDIO SEEK GATE " + ("GREEN" if not fails else f"RED: {len(fails)}"))
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main())
tools/analysis/mc68450.py
+59
View File
@@ -0,0 +1,59 @@
"""MC68450 / HD63450 register field layouts, in ONE copy.
Read by tools/analysis/21_iplrom_dmac.py, which decodes what the X68000's IPL
ROM programs into the DMAC, and by tools/analysis/27_dmac_config.py, which
decodes what src/player/dma.i programs into it. The two exist to be COMPARED
-- the ROM's own disk channel costs 16..19 clocks a byte (FINDINGS 52.5) and
the player's job is to be cheaper -- and a comparison between two decodings
that used two copies of these tables would not be one. This tree has already
paid twice for a transform with two copies of itself (FINDINGS 49.7.5).
SOURCED: MC68450 Direct Memory Access Controller, Motorola, Jul 1989
(bitsavers) -- the same document FINDINGS 39 cites for the transfer timings in
tools/analysis/buscost.py.
"""
# --- MC68450 register map, by offset inside a channel's 0x40 block ----------
REG = {0x00: "CSR", 0x01: "CER", 0x04: "DCR", 0x05: "OCR", 0x06: "SCR",
0x07: "CCR", 0x0A: "MTC", 0x0C: "MAR", 0x14: "DAR", 0x1A: "BTC",
0x1C: "BAR", 0x25: "NIV", 0x27: "EIV", 0x29: "MFC", 0x2D: "CPR",
0x31: "DFC", 0x39: "BFC"}
XRM = {0: "burst",
1: "UNDEFINED",
2: "cycle steal WITHOUT hold (bus released between operands)",
3: "cycle steal with hold"}
DTYP = {0: "68000-compatible, EXPLICITLY addressed -> DUAL ADDRESS",
1: "6800-compatible, EXPLICITLY addressed -> DUAL ADDRESS",
2: "device with ACK, implicitly addressed -> SINGLE ADDRESS",
3: "device with ACK and RDY, implicit -> SINGLE ADDRESS"}
DPS = {0: "8-bit port", 1: "16-bit port"}
PCL = {0: "status input", 1: "status input with interrupt",
2: "start pulse", 3: "abort input"}
SIZE = {0: "byte", 1: "word", 2: "long word", 3: "byte, unpacked"}
CHAIN= {0: "none", 1: "UNDEFINED", 2: "array", 3: "linked array"}
REQG = {0: "auto-request at limited rate", 1: "auto-request at max rate",
2: "EXTERNAL request (one operand per device request)",
3: "auto-request first operand, external thereafter"}
def dcr(v):
return [f"XRM = {v>>6&3:02b} {XRM[v>>6&3]}",
f"DTYP = {v>>4&3:02b} {DTYP[v>>4&3]}",
f"DPS = {v>>3&1:b} {DPS[v>>3&1]}",
f"PCL = {v&3:02b} {PCL[v&3]}"]
def ocr(v):
return [f"DIR = {v>>7&1:b} " +
("device -> memory (read)" if v & 0x80 else "memory -> device (write)"),
f"SIZE = {v>>4&3:02b} {SIZE[v>>4&3]}",
f"CHAIN= {v>>2&3:02b} {CHAIN[v>>2&3]}",
f"REQG = {v&3:02b} {REQG[v&3]}"]
def scr(v):
m = {0: "no count", 1: "increment", 2: "decrement", 3: "UNDEFINED"}
return [f"MAC = {v>>2&3:02b} memory address {m[v>>2&3]}",
f"DAC = {v&3:02b} device address {m[v&3]}"]
tools/bench/adpcm.lua
+88
View File
@@ -0,0 +1,88 @@
-- Play one buffer of ADPCM nibbles on the emulated MSM6258V, from 68000 code
-- (ROADMAP P6a). The Lua here does what Lua is allowed to do in this tree:
-- push bytes in, start the CPU, read the mailbox out. It is NOT in the feed
-- path -- session 33's probe was, and a host that writes the data register at
-- host-frame rate is not feeding a chip that consumes at 15,625 Hz (65.5).
--
-- WHAT THE MEASUREMENT IS. MAME's -wavwrite capture, at a sample rate chosen
-- to EQUAL the chip's stream rate so nothing resamples it, is the chip's own
-- output. tools/bench/verify_adpcm_chip.py reads the four model axes out of it.
M = manager.machine
SP = M.devices[":maincpu"].spaces["program"]
local META = loadfile("adpcm_meta.lua")()
local AD_FLAG, AD_BUF, AD_LEN = 0x18600, 0x18604, 0x18608
local AD_MTC0, AD_CSRF, AD_CERF = 0x1860C, 0x18610, 0x18614
local AD_MTCF, AD_MARF, AD_SPIN, AD_STAT = 0x18618, 0x1861C, 0x18620, 0x18624
local code do local f=io.open("adpcmgate.bin","rb"); code=f:read("a"); f:close() end
local data do local f=io.open("adpcm_data.bin","rb"); data=f:read("a"); f:close() end
local function P(s) print("[ADP] "..s) end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local st, t0, tplay = "boot", nil, nil
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
local t = T()
if st == "boot" then
if t < 3.0 then return end
for i = 1, #data do SP:write_u8(META.buf + i - 1, string.byte(data, i)) end
for i = 1, #code do SP:write_u8(0x10000 + i - 1, string.byte(code, i)) end
SP:write_u32(AD_FLAG, 0)
SP:write_u32(AD_BUF, META.buf)
SP:write_u32(AD_LEN, META.nbytes)
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700 -- supervisor, ALL interrupts masked
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
P(string.format("pushed %d B of code and %d B of nibbles at 0x%X",
#code, META.nbytes, META.buf))
-- THE CAPTURE'S OWN CLOCK. The wav starts at t=0 of the run, so the host
-- has to know when PLAY happened to find the stream in it -- but it is
-- NOT used as the alignment: the verifier searches a small window around
-- it, because a host frame is 17.6 ms and a sample is 64 us.
st, t0 = "running", t
return
end
if st == "running" then
local fl = SP:read_u32(AD_FLAG)
if fl == 2 and not tplay then
tplay = t
P(string.format("PLAY at t=%.4f s, chip status $%02X (bit7 clear = playing), "
.."MTC then = %d of %d", t, SP:read_u32(AD_STAT),
SP:read_u32(AD_MTC0), META.nbytes))
end
if fl == 0xFF or fl == 0xEE then
P(string.format("channel finished: CSR=$%02X CER=$%02X MTC=%d MAR=$%06X "
.."spin=%d", SP:read_u32(AD_CSRF), SP:read_u32(AD_CERF),
SP:read_u32(AD_MTCF), SP:read_u32(AD_MARF),
SP:read_u32(AD_SPIN)))
local dt = t - (tplay or t)
P(string.format("%d bytes took %.4f s = %.1f B/s "
.."(15,625 nibbles/s wants 7,812.5)",
META.nbytes, dt, META.nbytes/dt))
if fl == 0xEE then P("ERROR: the gate flagged a channel error or a timeout") end
local f = io.open("adpcm_run.lua", "w")
f:write(string.format("return { tplay = %.9f, ok = %s, nbytes = %d,\n"
.." csr = %d, cer = %d, mtc = %d, spin = %d }\n",
tplay or -1, tostring(fl == 0xFF), META.nbytes,
SP:read_u32(AD_CSRF), SP:read_u32(AD_CERF),
SP:read_u32(AD_MTCF), SP:read_u32(AD_SPIN)))
f:close()
st = "drain"; t0 = t
return
end
if t - t0 > 60 then P("TIMEOUT flag="..string.format("%08X", fl)); M:exit() end
return
end
if st == "drain" then
-- let the capture run past the end of the stream, so a truncated wav is
-- never mistaken for a short stream
if t - t0 < 0.3 then return end
P("done")
M:exit()
end
end)
if not ok then print("[ADP] LUA ERROR: "..tostring(err)); M:exit() end
end)
tools/bench/adpcm_run.sh
Executable
+56
View File
@@ -0,0 +1,56 @@
#!/bin/bash
# ROADMAP P6a: ask the machine's own MSM6258V which decoder it is.
#
# tools/bench/adpcm_run.sh
#
# WHAT A GREEN RUN MEANS: 68000 code programmed HD63450 channel 3 exactly as the
# IPL ROM programs it -- dual address, 8-bit port, cycle steal, EXTERNAL request
# -- fed the chip a designed nibble stream at the chip's own pace, and exactly
# ONE of sixteen candidate decoder models reproduces MAME's capture of the
# result SAMPLE-EXACT, with every one of the four axes shown to matter.
#
# WHAT IT DOES NOT MEAN: anything about an MSM6258. This is MAME's device model
# measured end to end through the machine's real transport. It settles the RIG.
# The silicon stays on the hardware list.
set -e
cd "$(dirname "$0")/../.."
tools/vasm/vasmm68k_mot -Fbin -o tmp/adpcmgate.bin src/player/adpcmgate.s > /dev/null
python3 tools/bench/prep_adpcm.py
# -samplerate 15625 is not a preference: it is the chip's own stream rate
# (8 MHz / 512), and equal rates are what keep MAME's resampler from filtering
# the thing being measured. The first cut of this ran at the default 48000 and
# every reconstructed sample arrived as an interpolated pair.
( cd tmp && SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 300 \
mame x68000 -bios ipl10 -ramsize 2M -video soft -window \
-samplerate 15625 -wavwrite adpcm.wav -nothrottle -plugins \
-autoboot_script ../tools/bench/adpcm.lua \
-seconds_to_run 12 > adpcm_run.log 2>&1 )
grep -aq "^\[ADP\] done" tmp/adpcm_run.log || {
echo "FAIL: the ADPCM gate did not finish -- no completion marker."
tail -8 tmp/adpcm_run.log; exit 1; }
grep -a "^\[ADP\]" tmp/adpcm_run.log | sed 's/^\[ADP\] / /'
fail() { echo "FAIL: $1"; exit 1; }
if grep -aq "^\[ADP\] ERROR" tmp/adpcm_run.log; then
fail "the channel reported an error or the gate timed out -- see CSR/CER above."
fi
grep -aq "bit7 clear = playing" tmp/adpcm_run.log || \
fail "the chip never reported itself playing."
# THE FEED RATE IS A GATE, not a note. The chip is the pacemaker: one byte per
# #DRQ3 and #DRQ3 at half the sample rate. If the bytes went out at some other
# rate then the channel was NOT being paced by the device, and every sample
# below is of a stream that arrived faster or slower than it was consumed --
# which is precisely the failure session 33 hit from Lua.
RATE=$(sed -n 's/.*= \([0-9.]*\) B\/s .*/\1/p' tmp/adpcm_run.log | head -1)
python3 - "$RATE" <<'PY' || fail "the feed was not paced by the chip (see above)."
import sys
r = float(sys.argv[1])
want = 7812.5
print(f" feed rate {r:,.1f} B/s against the chip's own {want:,.1f} B/s "
f"({100*(r-want)/want:+.2f}%)")
sys.exit(0 if abs(r - want) / want < 0.02 else 1)
PY
python3 tools/bench/verify_adpcm_chip.py tmp/adpcm.wav tmp/adpcm_seq.json
tools/bench/blit.lua
+25 -2
View File
@@ -11,7 +11,8 @@
-- a LOWER BOUND, not a prediction. Interrupts are masked (SR=$2700) so the
-- IPL's timer and VBL handlers cannot steal cycles into the measurement.
--
-- Timing resolution is one video frame (1/55.46 s = 18.03 ms), because Lua
-- Timing resolution is one video frame (1/56.69 s = 17.64 ms -- MAME's, not
-- the hardware's 55.46; see crtc_mode.lua), because Lua
-- gets no cycle counter -- luaengine.cpp exposes machine.time and nothing
-- from device_execute_interface. Each variant therefore loops enough times
-- to run ~4 emulated seconds, putting the granularity error near 0.4%.
@@ -28,7 +29,7 @@ end
local MODE = load_mode()
local FLAG, VAR, ITER = 0x18000, 0x18004, 0x18008
local SRCW, SRCB = 0x60000, 0x80000
local SRCW, SRCB, SRCP = 0x60000, 0x80000, 0x90000
local GVRAM, GPAL = 0xC00000, 0xE82000
local CPUHZ = 10000000 -- x68k.cpp:1133, 40_MHz_XTAL/4
local FRAME12 = CPUHZ / 12 -- 833333 cycles at 12 fps
@@ -39,6 +40,16 @@ local PLAN = {
{var=2, iter= 50, name="V2 naive byte-source expansion (move.b/move.w per pixel)"},
{var=3, iter=200, name="V3 write-only floor (no source read at all)"},
{var=4, iter= 60, name="V4 same 96KB of writes, issued in 4x4 BLOCK order (decoder access pattern)"},
-- V8 is V1 with R20 bit 11's packing: 48KB read + 48KB write for the SAME
-- 49,152 pixels. It is the per-frame work of a decoder-free packed player
-- (FINDINGS 44.7 / 46.6 / 47.5), and 47.6.1 filed its `movem` shape as an
-- ASSUMPTION -- this is the measurement that assumption was standing in for.
{var=8, iter=200, name="V8 PACKED movem.l blit (48KB read + 48KB write, same 49,152 pixels as V1)"},
-- V9/V10 are the two ways a BLOCK decoder could survive the packed layout
-- (47.6.4, open since session 16): sixteen move.b at stride 2 per block, or
-- pair the blocks 128 columns apart in the encoder and get V4's movem back.
{var= 9, iter= 40, name="V9 PACKED block order, 16 move.b at stride 2 per 4x4 block"},
{var=10, iter=120, name="V10 PACKED block order, blocks PAIRED so a movem writes whole words"},
}
local code do
@@ -96,6 +107,18 @@ local function setup()
SP:write_u8 (SRCB + y*256 + x, px)
end
end
-- SRCP: the PACKED frame, interleaved the way tools/bench/show_frame256_packed.lua
-- lays it out -- word i of a row is (column i+128) << 8 | (column i), because
-- page 0 is the low byte at screen column i and page 1 the high byte at i+128.
-- Only V8 reads it, and only its SIZE (128 words a row) affects the timing;
-- the interleave is written correctly so the buffer is the real artefact and
-- not a same-sized stand-in.
for y = 0, H-1 do
local row = PIX0 + y*W
for i = 0, (W//2)-1 do
SP:write_u16(SRCP + y*(W//2)*2 + i*2, (B(row+i+W//2) << 8) | B(row+i))
end
end
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
P(string.format("loaded blit.bin=%d bytes, source frame %dx%d at yoff=%d", #code, W, H, YOFF))
end
tools/bench/blit.s
+146
View File
@@ -104,6 +104,9 @@ ITER = $18008 ; iteration count, written by Lua
SPTR = $1800C ; V5 span stream pointer, written by Lua
SRCW = $60000 ; word-expanded frame 192*512 = 96KB
SRCB = $80000 ; byte-per-pixel frame 192*256 = 48KB
SRCP = $90000 ; PACKED frame 192*256 = 48KB (V8): two picture
; bytes per word, already interleaved by the
; encoder, so the blit is a straight copy
DST0 = $C08000 ; GVRAM + 32*1024 (first picture row)
DSTE = $C38000 ; GVRAM + 224*1024 (one past last)
ROWS = 192 ; picture rows a V5 stream describes
@@ -130,6 +133,12 @@ start:
beq v6
cmp.l #7,d0
beq v7
cmp.l #8,d0
beq v8
cmp.l #9,d0
beq v9
cmp.l #10,d0
beq v10
bra v3
; ---------------------------------------------------------------- V1
@@ -372,5 +381,142 @@ v7fh:
bne v7
bra done
; ---------------------------------------------------------------- V8
; THE PACKED FULL-FRAME BLIT (FINDINGS 46.6/47.2). Identical in shape to V1 --
; a row-linear movem.l chain out of a RAM frame into GVRAM -- and different in
; exactly one thing: a row is 128 WORDS, not 256, because R20 bit 11 lets one
; word carry two picture bytes. 256 = 5*48 + 16, so five 12-register bursts
; and a 4-register tail, against V1's ten and one.
;
; TIMING ONLY, and it does not set bit 11. MAME's gvram_w carries no timing in
; either arm (blit.lua's header), so the bit cannot move a cycle here; what it
; moves is the PICTURE, and the picture is what tools/bench/show_frame256_packed.lua
; and tools/bench/gvpack already verify pixel-exactly. Setting it here would
; make this variant's snapshot right and its measurement no different, and
; would put a display-mode change inside a timing loop for no gain.
;
; The source is PRE-INTERLEAVED by the host, which is the honest half of the
; claim: the packing is an encoder-side transform (46.3's argument for the text
; plane, and the same one here), so the decoder-free player's per-frame work is
; this copy and nothing else. If the interleave had to happen at run time this
; variant would be V2, not V1.
v8: lea SRCP,a0
lea DST0,a1
lea DSTE,a6
v8row: movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,48(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,96(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,144(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,192(a1)
movem.l (a0)+,d0-d3
movem.l d0-d3,240(a1)
lea 1024(a1),a1
cmpa.l a6,a1
bne v8row
subq.l #1,ITER.l
bne v8
bra done
; ---------------------------------------------------------------- V9
; WHAT THE PACKED LAYOUT COSTS A BLOCK DECODER (FINDINGS 47.6.4, open).
;
; V4 is the access pattern of a decoder that writes 4x4 codewords straight into
; GVRAM: 4 rows of 8 contiguous bytes at a 1024-byte stride, so each row is one
; `movem.l` of two registers. Under the packed layout that pattern is GONE.
; A block at columns x..x+3 owns the LOW bytes of four consecutive words -- four
; bytes at STRIDE 2 -- and the high bytes of those same words belong to the
; block 128 columns away. There is no burst that writes every other byte, so
; the block is sixteen `move.b`s.
;
; V9 does the pair together, low block then high block off one base, so it
; writes every byte it touches and covers the same 49,152 pixels V1/V4/V8 do.
; It is the HONEST version of "keep the codec and pack the screen": the mode
; map is unchanged, SKIP still works per block, and the writes go byte at a
; time. V10 below is the other option, and the comparison is the point.
v9: lea SRCB,a0
lea DST0,a3
lea DSTE,a4
v9brow: move.l a3,a1
lea 256(a3),a5 ; 32 block PAIRS * 8 bytes
v9blk:
move.b (a0)+,(a1)
move.b (a0)+,2(a1)
move.b (a0)+,4(a1)
move.b (a0)+,6(a1)
move.b (a0)+,1024(a1)
move.b (a0)+,1026(a1)
move.b (a0)+,1028(a1)
move.b (a0)+,1030(a1)
move.b (a0)+,2048(a1)
move.b (a0)+,2050(a1)
move.b (a0)+,2052(a1)
move.b (a0)+,2054(a1)
move.b (a0)+,3072(a1)
move.b (a0)+,3074(a1)
move.b (a0)+,3076(a1)
move.b (a0)+,3078(a1)
move.b (a0)+,1(a1)
move.b (a0)+,3(a1)
move.b (a0)+,5(a1)
move.b (a0)+,7(a1)
move.b (a0)+,1025(a1)
move.b (a0)+,1027(a1)
move.b (a0)+,1029(a1)
move.b (a0)+,1031(a1)
move.b (a0)+,2049(a1)
move.b (a0)+,2051(a1)
move.b (a0)+,2053(a1)
move.b (a0)+,2055(a1)
move.b (a0)+,3073(a1)
move.b (a0)+,3075(a1)
move.b (a0)+,3077(a1)
move.b (a0)+,3079(a1)
addq.l #8,a1
cmpa.l a5,a1
bne v9blk
lea 4096(a3),a3
cmpa.l a4,a3
bne v9brow
subq.l #1,ITER.l
bne v9
bra done
; ---------------------------------------------------------------- V10
; THE OTHER OPTION: PAIR THE BLOCKS IN THE ENCODER. If the codec codes the
; block at x and the block at x+128 as ONE unit, the destination is whole words
; again and V4's `movem.l` shape comes straight back -- the same instructions,
; the same 32 bytes of source per unit, and TWICE the pixels, because a word now
; carries two of them. So V10 is V4's inner loop run half as many times.
;
; WHAT IT COSTS IS NOT IN THIS MEASUREMENT. A pair skips only if BOTH of its
; blocks skip, and the two are 128 columns apart with nothing in the picture
; relating them. That is a CONTAINER question -- what fraction of the mode map
; survives pairing -- and 08_mode_map.py has the data to answer it. V10 prices
; the paint; it does not price the SKIPs the pairing loses.
v10: lea SRCP,a0
lea DST0,a3
lea DSTE,a4
v10brow: move.l a3,a1
lea 256(a3),a5 ; 32 block PAIRS * 8 bytes
v10blk: movem.l (a0)+,d0-d7 ; 32 bytes = one PAIR of 4x4 blocks
movem.l d0-d1,(a1)
movem.l d2-d3,1024(a1)
movem.l d4-d5,2048(a1)
movem.l d6-d7,3072(a1)
addq.l #8,a1
cmpa.l a5,a1
bne.s v10blk
lea 4096(a3),a3
cmpa.l a4,a3
bne v10brow
subq.l #1,ITER.l
bne v10
bra done
done: move.l #$FF,FLAG.l ; timer stops here
halt: bra.s halt
tools/bench/c68k/harness.c
+124 -3
View File
@@ -55,6 +55,12 @@ void p6logd(const char *fmt, ...) { (void)fmt; }
#define GV_HI 0xC80000u
#define FLAG 0x18000u
#define LFLAG 0x18040u /* src/player/load.i's control block */
#define LHDR 0x18044u
#define LDARK 0x18048u
#define LMODE 0x18054u
#define LITER 0x18058u
#define GPAL 0xE82000u
#define ITER 0x18008u
#define NFR 0x1800Cu
#define FPTR 0x18010u
@@ -106,9 +112,16 @@ static void wr8(unsigned int a, unsigned char d)
* momentarily reads back as $FF again. Without in_exec that transient
* recorded a run's stop cycle before the run had started, and every frame
* after the first came out as the whole slice. */
/* Which flag word the run watches. decode.s and stream.s use FLAG; the
* load-time transforms of src/player/load.i use their own, so that a player
* could eventually contain both without one clearing the other's state. The
* VALUES mean the same thing in both (1 running, $FF done, $EE failed), which
* is why one hook serves both. */
static unsigned int flag_adr = FLAG;
static void note_flag(void)
{
unsigned int v = rd32(FLAG);
unsigned int v = rd32(flag_adr);
long long now = slice - C68K.ICount;
if (!in_exec) return;
if (v == 1 && cyc_start < 0) cyc_start = now;
@@ -124,7 +137,7 @@ static void wr16(unsigned int a, unsigned short d)
a &= ADRMASK;
if (a >= GV_LO && a < GV_HI) { buf[a] = (unsigned char)d; buf[a+1] = 0; return; }
buf[a] = (unsigned char)d; buf[a+1] = (unsigned char)(d >> 8);
if (a >= FLAG && a < FLAG + 4) note_flag();
if (a >= flag_adr && a < flag_adr + 4) note_flag();
}
static void wr32(unsigned int a, unsigned int d){ wr16(a, (unsigned short)(d >> 16)); wr16(a+2, (unsigned short)d); }
@@ -185,12 +198,104 @@ static long long run(unsigned int off, unsigned int nfr, unsigned int iter)
return cyc_stop - cyc_start;
}
/* ---- the load-time transforms (ROADMAP P1+P2, FINDINGS 53) --------------
* The same question this harness asks of the decoder, asked of the loader: does
* a SECOND 68000 core, with its own cycle table and its own memory model,
* produce the same bytes and agree about what they cost? It also counts BUS
* cycles, which MAME cannot report -- and the bus is the resource this project
* established is the binding one (FINDINGS 38).
*/
static int run_load(const char *fcode, const char *fraw, const char *dump,
unsigned int mode, unsigned int iter,
unsigned int cb1_len, unsigned int cb4_len)
{
size_t nc, nr;
unsigned char *code = slurp(fcode, &nc), *raw = slurp(fraw, &nr);
push(STREAM, raw, nr); /* the RAW container header */
push(CODE, code, nc);
/* Poison every destination, so that a transform which writes NOTHING
* cannot pass by leaving the harness's own zeros in place. */
for (unsigned int a = CB1; a < CB1 + cb1_len; a += 2) wr16(a, 0xDEAD);
for (unsigned int a = CB4; a < CB4 + cb4_len; a += 2) wr16(a, 0xDEAD);
for (unsigned int c = 0; c < 256; c++) wr16(GPAL + c*2, 0xDEAD);
wr32(LDARK, 0xFFFFFFFFu);
/* The three scratch tables are poisoned only before a run that claims to
* build them. A run that only PACKS the palette is entitled to find them
* already built -- that is the point of pricing it separately -- so when
* this process is asked for one, it does the boot pass first, untimed,
* exactly as a player would have done at boot. Without that the pack runs
* on zeros: every entry then takes the same branch and the darkest entry
* comes out 0, which is a measurement of nothing. */
if (mode & 4)
for (unsigned int a = 0x19000; a < 0x19340; a += 2) wr16(a, 0xDEAD);
flag_adr = LFLAG;
if ((mode & 2) && !(mode & 4)) {
cyc_start = cyc_stop = -1; desync = 0;
wr32(LFLAG, 0); wr32(LHDR, STREAM); wr32(LMODE, 4); wr32(LITER, 1);
C68k_Reset(&C68K);
C68k_Set_Reg(&C68K, C68K_SR, 0x2700);
C68k_Set_Reg(&C68K, C68K_A7, STACK);
C68k_Set_Reg(&C68K, C68K_PC, CODE);
slice = 2000000000LL; in_exec = 1;
C68k_Exec(&C68K, (INT32)slice);
in_exec = 0;
if (cyc_stop < 0) { fprintf(stderr, "TIMEOUT in the table pre-pass\n"); return 4; }
}
cyc_start = cyc_stop = -1; desync = 0; bus_r = bus_w = 0;
wr32(LFLAG, 0); wr32(LHDR, STREAM); wr32(LMODE, mode); wr32(LITER, iter);
C68k_Reset(&C68K);
C68k_Set_Reg(&C68K, C68K_SR, 0x2700);
C68k_Set_Reg(&C68K, C68K_A7, STACK);
C68k_Set_Reg(&C68K, C68K_PC, CODE);
slice = 2000000000LL;
in_exec = 1;
C68k_Exec(&C68K, (INT32)slice);
in_exec = 0;
if (cyc_stop < 0) { fprintf(stderr, "TIMEOUT -- loader never set LFLAG\n"); return 4; }
if (desync) { fprintf(stderr, "BAD HEADER -- load.i found no 'DLX3' magic\n"); return 5; }
long long cyc = (cyc_stop - cyc_start) / (iter ? iter : 1);
fprintf(stderr, "[C68K] load mode %u: %lld cyc/pass (%.2f ms at 10MHz, "
"%.1f%% of a 12fps frame), dark=%u\n", mode, cyc, cyc / 10000.0,
100.0 * cyc / (10000000.0 / 12), rd32(LDARK));
/* A 68000 bus cycle is 4 clocks. Prefetch is not counted (C68K reads
* opcodes straight through the fetch pointer), so this is a LOWER bound on
* occupancy and the headroom it implies is an UPPER bound -- same caveat as
* the decoder's figure above. */
{
double slots = (double)cyc / 4.0;
double used = (double)(bus_r + bus_w) / (iter ? iter : 1);
fprintf(stderr, "[C68K] data bus: %.0f reads + %.0f writes = %.0f of "
"%.0f cycles = %.1f%% occupied (prefetch NOT counted)\n",
(double)bus_r / iter, (double)bus_w / iter, used, slots,
100.0 * used / slots);
}
if (dump) {
FILE *g = fopen(dump, "wb");
if (!g) { perror(dump); return 2; }
for (unsigned int a = CB1; a < CB1 + cb1_len; a++) { unsigned char b = rd8(a); fwrite(&b,1,1,g); }
for (unsigned int a = CB4; a < CB4 + cb4_len; a++) { unsigned char b = rd8(a); fwrite(&b,1,1,g); }
for (unsigned int c = 0; c < 256; c++) {
unsigned short w = rd16(GPAL + c*2);
unsigned char b[2] = { (unsigned char)(w >> 8), (unsigned char)w };
fwrite(b, 1, 2, g);
}
fclose(g);
fprintf(stderr, "[C68K] load output dumped to %s (%u B)\n",
dump, cb1_len + cb4_len + 512);
}
return 0;
}
int main(int argc, char **argv)
{
const char *fcode = "tmp/decode.bin", *fdata = "tmp/decode_data.bin", *dump = NULL;
unsigned int cb1_len=0, cb4_len=0, pal_len=0, stream_len=0, nframes=0, H=192, W=256, fps=12;
unsigned int dark = 255;
unsigned int anch[32]; int nanch = 0;
const char *fraw = NULL, *loaddump = NULL;
unsigned int loadmode = 7, loaditer = 1;
for (int i = 1; i < argc; i++) {
if (!strcmp(argv[i], "--code")) fcode = argv[++i];
else if (!strcmp(argv[i], "--data")) fdata = argv[++i];
@@ -204,10 +309,15 @@ int main(int argc, char **argv)
else if (!strcmp(argv[i], "--H")) H = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--fps")) fps = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--dark")) dark = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--loadraw")) fraw = argv[++i];
else if (!strcmp(argv[i], "--loaddump")) loaddump = argv[++i];
else if (!strcmp(argv[i], "--loadmode")) loadmode = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--loaditer")) loaditer = (unsigned)atoi(argv[++i]);
else if (!strcmp(argv[i], "--anchor")) { if (nanch < 32) anch[nanch++] = (unsigned)strtoul(argv[++i], NULL, 10); }
else { fprintf(stderr, "unknown arg %s\n", argv[i]); return 2; }
}
if (!nframes || !stream_len) { fprintf(stderr, "need --nframes and --stream (from decode_meta.lua)\n"); return 2; }
if (!fraw && (!nframes || !stream_len)) {
fprintf(stderr, "need --nframes and --stream (from decode_meta.lua)\n"); return 2; }
/* MAP_32BIT: C68K keeps its fetch base in a UINT32, so the arena must live
* below 4 GB or every opcode fetch reads a truncated pointer. */
@@ -216,6 +326,17 @@ int main(int argc, char **argv)
if (buf == MAP_FAILED) { perror("mmap MAP_32BIT"); return 2; }
fprintf(stderr, "[C68K] arena at %p\n", (void *)buf);
if (fraw) {
C68k_Init(&C68K);
C68k_Set_ReadB (&C68K, rd8);
C68k_Set_ReadW (&C68K, rd16);
C68k_Set_WriteB(&C68K, wr8);
C68k_Set_WriteW(&C68K, wr16);
C68k_Set_Fetch (&C68K, 0x000000, 0xFFFFFF, (UINT32)(unsigned long)buf);
return run_load(fcode, fraw, loaddump, loadmode, loaditer,
cb1_len ? cb1_len : 8192, cb4_len ? cb4_len : 2048);
}
size_t nc, nd;
unsigned char *code = slurp(fcode, &nc), *data = slurp(fdata, &nd);
size_t need = (size_t)cb1_len + cb4_len + pal_len + stream_len;
tools/bench/check.sh
+634
View File
@@ -270,4 +270,638 @@ grep -q "ceiling 8 frames" tmp/pace_check.log || {
grep -q "^OK" tmp/pace_check.log || { echo "FAIL: paced pass not pixel-exact";
tail -4 tmp/pace_check.log; exit 1; }
echo "--- session 21: the 68000 builds its own codebooks and palette (FINDINGS 53) ---"
# ROADMAP P1+P2. Until now tools/bench/dlxload.py expanded the codebooks and
# packed the palette HOST-SIDE and the rigs pushed the result into emulated RAM.
# A player has no host. src/player/load.i does both on the 68000, out of the RAW
# container header, and this gates it byte-for-byte against dlxload.py -- which
# stays the reference, because what changed is where the transforms RUN, not
# what they produce.
#
# Byte-for-byte and not "close enough": a wrong codebook byte is a wrong colour
# in every block that uses that codeword, and a wrong shared LSB is a slightly
# wrong colour that looks like a codec artefact rather than a loader bug.
# The palette half is read back out of the PALETTE REGISTERS at $E82000, so
# "the words reached the hardware" is part of what passes.
#
# NOT gated on the cycle counts, and the reason is NOT the one blit.s has. These
# are emulated time and reproduce exactly run to run; what they are not is
# sharp, because MAME samples them on a 1/56.69 s clock and the job takes
# milliseconds. Nothing in the tree's cost models depends on them either. A
# change in them is a re-derivation in FINDINGS 53, not a red light here.
bash tools/bench/load_run.sh "$DLX" > tmp/load_gate.log 2>&1 || {
echo "FAIL: the load-time transforms did not pass."; tail -12 tmp/load_gate.log
exit 1; }
grep -aE "^ *OK|both CPU cores|SCENE CHANGE" tmp/load_gate.log | sed 's/^ *//;s/^/ /'
echo "--- session 22: the 68000 keeps its own frame clock (FINDINGS 54) ---"
# ROADMAP P3. Until now the 12 fps tick came from tools/bench/stream.lua -- a
# host writing a word into emulated RAM. A player has no host. src/player/
# clock.i derives the tick from the CRTC's own V-DISP output through the MFP,
# with a remainder-keeping divider whose two constants are READ OUT OF THE CRTC
# at init, so the clock and the raster it counts cannot disagree.
#
# WHAT IS GATED, and it is deliberately structural rather than numeric:
# * the interrupt count equals the raster frame count -- the tick IS the
# raster, not something that merely resembles it;
# * the divider does not accumulate drift, stated in TICKS (a remainder can
# hold back at most one) rather than in ppm, which would let a longer
# window advertise a tighter clock for free;
# * every frame tick waits 4 or 5 refreshes and nothing else, which is what a
# remainder-keeping divider can produce and a broken one cannot.
# The interrupt COST is printed and not gated, for the same reason FINDINGS 53's
# cycle counts are not: it is a measurement, and a change in it is a
# re-derivation in FINDINGS 54 rather than a red light here.
bash tools/bench/clock_run.sh 3000 12 > tmp/clock_gate.log 2>&1 || {
echo "FAIL: the frame clock did not pass."; tail -12 tmp/clock_gate.log
exit 1; }
grep -aE "INTERRUPT:|PER FRAME:|DRIFT:|CADENCE:" tmp/clock_gate.log
grep -q "V-DISP interrupts 3000" tmp/clock_gate.log || {
echo "FAIL: the tick is not the raster -- the interrupt count and the frame"
echo " count disagree. Everything else in FINDINGS 54 rests on that."
exit 1; }
echo "--- session 22: 120 frames decoded on the machine's own clock (FINDINGS 54) ---"
# The strongest form of the claim: the same pixel-exact 120-frame decode out of
# the same 256 KB ring, with NOTHING outside the machine deciding when a frame
# may start. The pace gate in src/player/stream.s is byte-for-byte the one
# FINDINGS 51 measured -- it cannot tell a host-written tick from a machine-
# written one, which is why this is a test of the clock and not of a new rig.
DLX_PACE=2 bash tools/bench/pace_run.sh 256 0 > tmp/selfpace_check.log 2>&1 || {
echo "FAIL: the self-paced pass did not complete."; tail -8 tmp/selfpace_check.log
exit 1; }
grep -aE "decoder SELF-PACED|FRAME CLOCK|UNDERRUNS|NO IDLE" tmp/selfpace_check.log
grep -q "UNDERRUNS: 0/120" tmp/selfpace_check.log || {
echo "FAIL: the self-paced decoder underran."; exit 1; }
grep -q "^OK" tmp/selfpace_check.log || {
echo "FAIL: the self-paced pass was not pixel-exact. The clock changed WHEN"
echo " frames start; if it changed WHAT they draw, the interrupt is"
echo " corrupting decoder state."; tail -4 tmp/selfpace_check.log; exit 1; }
echo "--- session 23: the 68000 fills its own ring (FINDINGS 55) ---"
# ROADMAP P5. Until now the RING was filled by tools/bench/stream.lua: the host
# held the record index, chose where each record went, wrote the descriptor and
# advertised it. A player has no host. src/player/ring.i does all of that on the
# 68000, out of the DLX4 record index in the scene header, and this script keeps
# only the part that is not the CPU's -- a transport that answers one request at
# a time at a modelled rate.
#
# WHAT IS GATED:
# * pixel-exact, which is the only test that can see a wrong placement: the
# block loop reads with a monotonically increasing a0 and no bounds check,
# so a record placed over one the decoder has not finished corrupts pixels
# rather than faulting (49.2);
# * the host AUDITS every placement against its own index and its own list of
# live records, and refuses the run on the first disagreement;
# * the wrap policy still produces the SAME 18 wraps the
# host producer produced in FINDINGS 49.4 -- a third independent
# implementation of `aligned` landing on the same tiling;
# * zero underruns at a two-deep request queue, which is the finding: a
# one-deep queue leaves the channel idle between records and underran 59 of
# 120 frames on this same container and rate.
DLX_PACE=2 DLX_RINGOWN=1 DLX_QDEPTH=2 bash tools/bench/pace_run.sh 256 488 \
> tmp/ringown_check.log 2>&1 || {
echo "FAIL: the machine-owned ring pass did not complete."
tail -10 tmp/ringown_check.log; exit 1; }
grep -aE "MACHINE-OWNED|PREFILL:|CHANNEL IDLE|UNDERRUNS|SEEK SLACK" tmp/ringown_check.log \
| sed "s/\[STR\] / /"
grep -q "MISPLACED" tmp/ringown_check.log && {
echo "FAIL: the 68000 placed a record over one the decoder still owned."
exit 1; }
grep -q "UNDERRUNS: 0/120" tmp/ringown_check.log || {
echo "FAIL: the machine-owned ring underran at a two-deep queue. That is the"
echo " configuration FINDINGS 55 says keeps the channel busy; if it no"
echo " longer does, the poll site in src/player/stream.s moved."
exit 1; }
grep -q "ring: 18 wraps" tmp/ringown_check.log || {
echo "FAIL: the machine's own \`aligned\` no longer tiles this container the"
echo " way FINDINGS 49.4's host producer did (18 wraps). The policy is"
echo " meant to be the SAME policy in a different place."
exit 1; }
grep -q "^OK" tmp/ringown_check.log || {
echo "FAIL: the machine-owned ring pass was not pixel-exact -- a record was"
echo " placed or described wrongly."; tail -4 tmp/ringown_check.log
exit 1; }
echo "--- session 23: a seek, and the decode after it (FINDINGS 55) ---"
# The branch point rehearsed. A second pass over the scene begins with a real
# seek in src/player/ring.i: the channel is waited quiet, the ring is declared
# empty, the disc address of record 0 comes out of the index rather than from a
# walk, and the whole lookahead 51.3 says takes seconds of play to accumulate is
# thrown away and rebuilt from the prefill. What is gated afterwards is the one
# thing that can see a wrong seek: the last frame of the SECOND pass has to be
# pixel-exact, and a SKIP block is a claim about the previous frame, so it is
# only right if every frame after the seek was.
DLX_PACE=2 DLX_RINGOWN=1 DLX_QDEPTH=2 DLX_ITER=2 \
bash tools/bench/pace_run.sh 256 488 > tmp/ringseek_check.log 2>&1 || {
echo "FAIL: the seek pass did not complete."; tail -10 tmp/ringseek_check.log
exit 1; }
grep -aE "SEEK PASS|CHANNEL IDLE|UNDERRUNS" tmp/ringseek_check.log | sed "s/\[STR\] / /"
grep -q "SEEK PASS 2" tmp/ringseek_check.log || {
echo "FAIL: no second pass -- the seek never happened, so this gated nothing."
exit 1; }
grep -q "UNDERRUNS: 0/120" tmp/ringseek_check.log || {
echo "FAIL: the pass after the seek underran."; exit 1; }
grep -q "^OK" tmp/ringseek_check.log || {
echo "FAIL: the decode after the seek was not pixel-exact."
tail -4 tmp/ringseek_check.log; exit 1; }
echo "--- session 25: the 68000 reads the disc itself (FINDINGS 57) ---"
# ROADMAP P4, first half. Until now every byte the player consumed was placed in
# emulated RAM by a host: decode.lua preloaded a container, stream.lua answered a
# mailbox at a modelled rate. A player has no host. src/player/scsi.i selects a
# SCSI target on a real MB89352 and issues READ(10) itself.
#
# Session 21 recorded this as blocked -- "MAME's x68000 has no MB89352 path" --
# and that was wrong: `-exp1 cz6bs1` instantiates one, and FINDINGS 32.4 had
# already read its DMA glue in session 9. What was actually missing was the
# card's 8 KB boot ROM, which MAME requires to instantiate the device and which
# the player never executes. scsi_run.sh supplies a blank one on its own rompath.
#
# WHAT IS GATED: the register window (60 of 64 addresses -- the two holes ARE the
# MB89352's missing TMOD and EXBF, and they are what put DREG at $EA0015), and
# two READ(10)s verified byte-for-byte against the host's copy of the same image,
# one of them at a NON-ZERO LBA. Nothing here is gated on rate, and nothing here
# can be: MAME's device models are functional, not transfer-timing accurate.
# Skipped rather than failed when chdman is absent -- it ships with mame-tools.
if command -v chdman > /dev/null; then
bash tools/bench/scsi_run.sh "$DLX" > tmp/scsi_gate.log 2>&1 || {
echo "FAIL: the 68000 could not read the disc."; tail -14 tmp/scsi_gate.log
exit 1; }
grep -aE "ANSWERED|READ\(10\) OK" tmp/scsi_gate.log
else
echo " SKIPPED: no chdman (ships with mame-tools) -- cannot build the volume"
fi
echo "--- session 26: the ring is filled off a real SCSI volume (FINDINGS 58) ---"
# ROADMAP P4b. The stage above shows the 68000 can READ the disc. This shows it
# can RUN off it: src/player/xfer.i sits behind src/player/ring.i's XF_* mailbox
# in place of tools/bench/stream.lua's modelled transport, and the same 120
# frames are decoded out of the same 256 KB ring with NOTHING outside the
# machine in the transfer path -- no host file, no modelled rate, no synthesised
# ack.
#
# WHAT IS GATED, and it is correctness rather than rate on purpose:
# * pixel-exact, which is the only test that can see a wrong record: the
# window in scsi.i decides which of a sector's bytes reach the ring, and a
# window off by one byte desyncs the bitstream rather than faulting (49.2);
# * the SAME 18 wraps -- ring.i's placement policy must not be able to tell
# which transport answered it, and this is the assertion that says it could
# not. The WRAP COUNT is gated and the mean hole is only reported: DLX5's
# records are up to 511 B longer than DLX4's, so the hole moved (14.7 KB ->
# 13.5 KB) while the tiling did not. Gating a number that the container's
# record lengths move would gate the container, not the policy;
# * every record accounted for: 120 READ(10)s, and the bytes into the ring
# EQUAL to the bytes off the disc -- both read out of the container rather
# than written here. Under DLX4 they differed by 1.34% because a record was
# not a sector (58.3); DLX5 aligns records to sectors and the covering-sector
# read disappears, so the gate is now their IDENTITY;
# * a real mid-stream SEEK with the real transport, in the second pass. This
# is the one path that could not exist before: ring_seek waits for the
# channel to go quiet, and with the transport INSIDE the machine the only
# thing that can retire an outstanding request is that wait loop itself.
#
# NOTHING HERE IS GATED ON RATE and nothing here can be. What the run DOES cost
# is printed by tools/bench/xfer_cost.sh and recorded in FINDINGS 58.2; it is a
# measurement, and a change in it is a re-derivation there rather than a red
# light here. Skipped rather than failed when chdman is absent.
if command -v chdman > /dev/null; then
DLX_PACE=0 DLX_RINGOWN=1 DLX_QDEPTH=2 DLX_XFER=scsi \
bash tools/bench/pace_run.sh 256 0 > tmp/p4b_check.log 2>&1 || {
echo "FAIL: the 68000 could not run the ring off a real SCSI volume."
tail -12 tmp/p4b_check.log; exit 1; }
grep -aE "REAL TRANSPORT:|SECTOR OVERHEAD|ring: " tmp/p4b_check.log \
| sed "s/^ *//;s/^/ /"
grep -aq "TRANSPORT FAILED" tmp/p4b_check.log && {
echo "FAIL: a record's READ(10) reported an error."; exit 1; }
# THE BYTE COUNTS COME OUT OF THE CONTAINER, not out of this file. They were
# two hardcoded constants fitted to the DLX4 gate container, and session 28's
# re-encode went red on both of them for the right reason -- the container had
# changed and the expectation had not. A gate whose expected value is a
# literal tests the literal.
EXPECT_B=$(python3 -c "
import sys; sys.path.insert(0, 'tools/encoder')
from dlx import DLX
print(sum(DLX('$DLX').record_lengths()))")
grep -aq "REAL TRANSPORT: 120 READ(10)s by the 68000, $EXPECT_B B into the ring" \
tmp/p4b_check.log || {
echo "FAIL: the 68000 did not fetch all 120 records, or did not fetch"
echo " $EXPECT_B B of them. A short record is a desync, not a shortfall."
exit 1; }
# DLX5 MAKES THESE THE SAME NUMBER, and that identity IS the finding (59.4,
# and 58.3 option C): a sector-aligned container has no covering-sector read,
# so the disc moves exactly the records and nothing else. Under DLX4 they
# differed by 1.34% and both were gated so neither could drift into the other;
# under DLX5 the gate is that they are EQUAL. If a windowed read ever came
# back -- a container that was not aligned, or a layout that lost the
# alignment -- the disc figure would exceed the ring figure and this goes red.
grep -aq "SECTOR OVERHEAD: $EXPECT_B B off the disc for $EXPECT_B B of record = 0.00%" \
tmp/p4b_check.log || {
echo "FAIL: the disc no longer moves EXACTLY the records. On a sector-"
echo " aligned container (DLX5) there is no covering-sector read at"
echo " all, so these two counts must be the same $EXPECT_B B. If they"
echo " differ, either the container lost its alignment or scsi.i is"
echo " windowing again -- and a DMA channel cannot window (59.4)."
exit 1; }
grep -aq "ring: 18 wraps" tmp/p4b_check.log || {
echo "FAIL: the placement policy tiled this container differently with a"
echo " real transport behind it than with a modelled one. ring.i is"
echo " not supposed to be able to tell them apart."; exit 1; }
grep -aq "^OK" tmp/p4b_check.log || {
echo "FAIL: the pass off the SCSI volume was not pixel-exact."
tail -4 tmp/p4b_check.log; exit 1; }
DLX_PACE=2 DLX_RINGOWN=1 DLX_QDEPTH=2 DLX_ITER=2 DLX_XFER=scsi \
DLX_SECONDS=240 bash tools/bench/pace_run.sh 256 0 \
> tmp/p4b_seek_check.log 2>&1 || {
echo "FAIL: the seek pass off the SCSI volume did not complete."
tail -12 tmp/p4b_seek_check.log; exit 1; }
grep -aE "SEEK PASS|IS VACUOUS" tmp/p4b_seek_check.log | sed "s/^ *//;s/^/ /"
grep -aq "SEEK PASS 2" tmp/p4b_seek_check.log || {
echo "FAIL: no real seek -- the second pass never threw its ring away, so"
echo " ring_seek's quiet-wait was never asked to retire an outstanding"
echo " transfer and this gated nothing."; exit 1; }
grep -aq "^OK" tmp/p4b_seek_check.log || {
echo "FAIL: the decode after a seek off the SCSI volume was not pixel-exact."
tail -4 tmp/p4b_seek_check.log; exit 1; }
else
echo " SKIPPED: no chdman (ships with mame-tools) -- cannot build the volume"
fi
echo "--- session 27: the DMAC drives the data phase, and holds the bus (FINDINGS 59) ---"
# ROADMAP P4a, the last item before M2. The two stages above have the CPU moving
# every byte itself, at the 87.28 clocks per delivered byte FINDINGS 58.2
# measured -- 391.8% of a 12 fps frame. This one hands the DATA IN phase to the
# HD63450 and gates on the thing 57.3 said would be hard to show: that the DMAC,
# and not the CPU, is driving it.
#
# IT IS GATED WITHOUT LOOKING AT $EA0015, and that is the design. With the
# DMAC's OWN asserted -- which it is at idle here -- MAME cannot distinguish a
# CPU-driven byte at that address from a DMAC-driven one, so watching it proves
# nothing. What is gated instead is THE CPU'S OWN PROGRESS:
# * the same 2,048 B off the disc three ways -- PIO, held, stealing -- all
# three byte-exact against the host's copy, so the configuration is being
# compared against a delivery that works and not against nothing;
# * MTC sampled by the INSTRUCTION AFTER the one that starts the channel: zero
# in the held configuration (the whole transfer happened between two
# instructions, because the 68000 did not execute in between) and the full
# count in the stealing one;
# * the CPU's own trip count round its wait loop: 1 against hundreds. A
# counter that CANNOT come out different is 58.3's vacuous "UNDERRUNS: 0/120"
# again, so the run asserts the contrast and not just the held value;
# * the channel's own CSR/CER/MTC/MAR, which must say it moved every byte
# without error;
# * THE PALETTE REGISTERS AT $E82000 (session 30, ROADMAP K1): the same
# transfer aimed at the palette, byte-exact into 256 register words read
# back by the 68000; the SAME transfer aimed 20 KB away leaving the palette
# as the CPU poisoned it, which is what attributes the first run to the
# channel's MAR; and ONE array-chained start crossing from the registers
# into GVRAM, which is the shape of a whole frame -- a palette entry and
# 192 row entries, started once. What this does NOT settle is the board:
# MAME models the palette as a generic palette_device over memory_array,
# whose write16 is a plain COMBINE_DATA, so it cannot tell a register file
# that takes byte writes from one that does not (FINDINGS 62.4);
# * and a WINDOWED read through the channel REFUSED. 117 of 120 records start
# part way into a sector (58.3); a channel writes a contiguous run and cannot
# drop the bytes in front of one, so it would write the neighbouring records
# into the ring with no bounds check to catch it (49.2). The refusal is what
# makes "sector-aligned container" a precondition the transport states.
#
# NOT GATED ON RATE, and it cannot be: MAME's DMAC runs on wall-clock attotimes
# (42.5) and models a held bus by HALTING the CPU rather than by charging it
# cycles per operand. `W` is untouched. tools/analysis/28_autorequest_cost.py
# prices what this configuration costs, from the datasheet and an explicit rate.
# Skipped rather than failed when chdman is absent.
if command -v chdman > /dev/null; then
bash tools/bench/dma_run.sh "$DLX" > tmp/dma_gate.log 2>&1 || {
echo "FAIL: the DMAC did not drive the SCSI data phase."
tail -16 tmp/dma_gate.log; exit 1; }
grep -aE "BYTES OK|MTC one instruction|trips round|REFUSED|PALETTE|ONE START" \
tmp/dma_gate.log \
| sed 's/^ *//;s/^/ /'
else
echo " SKIPPED: no chdman (ships with mame-tools) -- cannot build the volume"
fi
echo "--- session 24: the scene graph, and the gap between branch points (FINDINGS 56) ---"
# The arcade scene graph is not in this repo and is not redistributable from
# here. tools/import/scenegraph.py is the ONE file in the tree that knows the
# outside projects exist; it writes tmp/scenegraph.json in this project's own
# DLXSCENE1 schema and everything downstream reads only that.
# What is gated is the IMPORT, not the numbers: 516 sequences and 906 input
# windows, and the four timing helpers still being the formulas the importer
# evaluates. Skipped when the checkout is absent.
DIRKSIMPLE=${DLX_DIRKSIMPLE:-tmp/scenegraph/DirkSimple}
if [ -f "$DIRKSIMPLE/data/games/lair/game.lua" ]; then
DLX_DIRKSIMPLE="$DIRKSIMPLE" python3 tools/import/scenegraph.py \
-o tmp/scenegraph.json > tmp/scenegraph_import.log 2>&1 \
|| { cat tmp/scenegraph_import.log; exit 1; }
sed "s/^/ /" tmp/scenegraph_import.log
grep -q "516 sequences, 906 input windows" tmp/scenegraph_import.log || {
echo "FAIL: the scene graph did not import to 516/906 -- upstream changed,"
echo " or the parser silently dropped branches."; exit 1; }
python3 tools/analysis/25_scene_graph.py --kbps 488 --ring 256 \
> tmp/scenegraph_check.log 2>&1 \
|| { tail -20 tmp/scenegraph_check.log; exit 1; }
grep -aE "^ WORST |^ ZERO-PLAY|^ BRANCH STRUCTURE" tmp/scenegraph_check.log
else
echo " SKIPPED: no DirkSimple checkout at $DIRKSIMPLE"
echo " (git clone --depth 1 https://github.com/icculus/DirkSimple)"
fi
echo "--- session 29: the packed paint, and what it does to the codec (FINDINGS 61) ---"
# tools/bench/blit.s gained V8/V9/V10 -- the packed full-frame blit, and the two
# ways a 4x4 BLOCK decoder could survive the packed layout. 47.6.1 had filed the
# packed paint's `movem` shape as an ASSUMPTION since session 16; this measures
# it, in the same run as V1/V3/V4 so it is quoted against numbers that have not
# moved since session 9.
#
# WHAT IS GATED IS STRUCTURAL, not numeric, for the reason the load stage gives:
# MAME samples these on a 1/56.69 s clock and no cost model in the tree depends
# on their exact value. What DOES depend on them is the ORDER, and the order is
# the whole of FINDINGS 61:
# V8 < V1 packing halves the full-frame literal paint
# V9 > V4 packing makes a BLOCK decoder DEARER, not cheaper
# V10 < V4 unless the blocks are paired, which costs SKIPs instead
# A tree where any of those flipped has a different answer to 44.7 and should
# say so out loud rather than let 29_packed_player.py narrate the old one.
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
rm -f tmp/blit_v8.log
( cd tmp && SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 300 mame x68000 \
-bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
-autoboot_script ../tools/bench/blit.lua -seconds_to_run 120 \
> blit_v8.log 2>&1 )
grep -aq "summary (instruction cycles only" tmp/blit_v8.log || {
echo "FAIL: the blit timing run produced no summary -- it did not finish."
tail -8 tmp/blit_v8.log; exit 1; }
python3 - <<'EOF' || exit 1
import re, sys
v = {}
for line in open("tmp/blit_v8.log", errors="replace"):
m = re.search(r"V(\d+)\s+(\d+) cyc", line)
if m: v[int(m.group(1))] = int(m.group(2))
need = (1, 2, 3, 4, 8, 9, 10)
missing = [n for n in need if n not in v]
if missing: sys.exit(f"FAIL: blit.lua reported no V{missing} -- run incomplete.")
for a, op, b, why in ((8, "<", 1, "packing did not halve the literal paint"),
(9, ">", 4, "packed BLOCK order came out CHEAPER than "
"unpacked -- 61.3's conclusion is inverted"),
(10, "<", 4, "pairing the blocks did not buy back the "
"movem shape")):
ok = v[a] < v[b] if op == "<" else v[a] > v[b]
if not ok:
sys.exit(f"FAIL: V{a} {v[a]:,} is not {op} V{b} {v[b]:,} -- {why}.")
print(f" V1 {v[1]:,} / V8 PACKED {v[8]:,} = {100*v[8]/v[1]:.0f}% -- "
f"and V3, the unpacked WRITE-ONLY floor, is {v[3]:,}")
print(f" V4 {v[4]:,} / V9 packed-block {v[9]:,} = {100*v[9]/v[4]:.0f}% -- "
f"packing costs a BLOCK decoder {100*v[9]/v[4]-100:.0f}%")
print(f" V10 paired blocks {v[10]:,} = {100*v[10]/v[4]:.0f}% of V4, and pairing "
f"is paid for in SKIPs")
EOF
python3 tools/analysis/29_packed_player.py "$DLX" > tmp/packed_player.log 2>&1 \
|| { tail -20 tmp/packed_player.log; exit 1; }
grep -aE "SKIP block PAIRS|free / DMAC->GVRAM / PACKED|^ CODEC, gate" \
tmp/packed_player.log
echo "--- session 31: the PACKED container, and the picture re-derived (FINDINGS 63) ---"
# ROADMAP K2. The container is REBUILT every run rather than reused when
# present, the way the codec's gate container is: a packed encode is 3 seconds
# because there is no k-means in it, so there is no reason to let a stale file
# stand between the encoder and the gate.
# The window's audio has to exist before the container can carry it, so the
# extraction that used to live in session 33's stage moves up here. Same seconds
# as the frames, and that is not a convenience: an audio stream that is not the
# same seconds as the picture is not this project's audio.
[ -f tmp/au_singe.raw ] || python3 tools/encoder/extract_audio.py 00223 tmp/au_singe.raw 15625 539.4 10.0
python3 tools/encoder/pack.py tmp/fr_singe tmp/packed_singe.dlxp \
--nframes "$NF" --audio tmp/au_singe.raw > tmp/pack_encode.log 2>&1 \
|| { cat tmp/pack_encode.log; exit 1; }
grep -aE "^ (record|wire|DLXP2|the four axes|lump payload)" tmp/pack_encode.log
# ...and the SILENT control beside it, which is what says the interleave moved no
# picture byte. It is the same encode with one flag off, and a packed encode is
# four seconds because there is no k-means in it, so the control is cheap enough
# to build every run rather than reason about.
python3 tools/encoder/pack.py tmp/fr_singe tmp/packed_singe_silent.dlxp \
--nframes "$NF" > tmp/pack_encode_silent.log 2>&1 \
|| { cat tmp/pack_encode_silent.log; exit 1; }
# WHAT IS GATED. Four format invariants that a DMA channel cannot check for
# itself -- it copies bytes and has no opinion about them (FINDINGS 62) -- and
# the three quality claims FINDINGS 61.9 rests the whole packed branch on:
# round-trip, sector geometry, index 0 unused, palette words agree
# packed > the shipping codec / > the codec's CEILING / > a SCENE-palette control
# A tree where any of the last three flipped has a different answer to ROADMAP K
# and should say so out loud rather than let the branch keep building.
python3 tools/analysis/30_packed_container.py tmp/packed_singe.dlxp \
--codec "$DLX" > tmp/packed_container.log 2>&1 \
|| { cat tmp/packed_container.log; exit 1; }
sed -n '/^PSNR/,$p' tmp/packed_container.log | grep -aE "RGB888|CODEC|CEILING|CONTROL|PER-FRAME|packed vs|worth|costs"
# And the container's OWN BYTES through px68k's real gvram.c, with the harness
# computing no interleave -- the only test that can catch an encoder whose byte
# order is wrong, because the container round-trips against its own inverse
# either way. Same skip-not-fail rule as the C68K stage: px68k is not in here.
if [ -f "$PX68K/x68k/gvram.c" ]; then
make -s -C tools/bench/gvpack PX68K="$PX68K"
# Frame 0 carries the two negative controls; the other two are there because
# one frame rendering does not say the container's 120th record is placed
# right, and record placement is arithmetic this format has no index to check.
python3 tools/bench/gvpack/verify_dlxp.py tmp/packed_singe.dlxp 0 --controls || exit 1
for f in $((NF / 2)) $((NF - 1)); do
# NOT piped into head: `set -e` reads a pipeline's status from its LAST
# command, so a piped verifier that failed would be reported by head's zero.
python3 tools/bench/gvpack/verify_dlxp.py tmp/packed_singe.dlxp "$f" \
> "tmp/dlxp_f$f.log" 2>&1 || { cat "tmp/dlxp_f$f.log"; exit 1; }
head -1 "tmp/dlxp_f$f.log"
done
else
echo " SKIPPED: no px68k at $PX68K -- the container's bytes were not rendered"
fi
echo "--- session 32: the PACKED PLAYER, end to end off the disc (FINDINGS 64) ---"
# ROADMAP K3. src/player/packed.s brings up its own display, builds its own
# 193-entry DMA chain, keeps its own frame clock off V-DISP and fetches every
# record itself with READ(10) off a CZ-6BS1. This script writes no picture byte
# and no palette entry.
#
# WHY EVERY FRAME IS COMPARED AND THE CODEC'S GATE COMPARES ONE. The codec is
# temporally recursive -- a SKIP block is a claim about the previous frame -- so
# its last frame audits all 120. A packed frame is a LITERAL and frame 119 says
# nothing about frame 60. The simplification that deleted the ring also deleted
# the gate's free lunch.
#
# AND WHY IT IS PACED AT HALF RATE. The write window has to be OPEN for the
# whole transfer and buffer mode blanks the graphics layer, so at the
# container's own 12 fps there is no instant at which a complete frame is
# displayable and there is nothing to snapshot (FINDINGS 64.2 -- that is the
# session's finding, not a rig limitation being worked around). Half rate opens
# a display interval without changing one byte of the transfer.
#
# GATE ONLY: runs 2-4 of packed_run.sh measure the apparatus rather than gate
# it, and they are three more MAME jobs for numbers that cannot change unless
# MAME does. tools/bench/packed_run.sh with no DLX_PK_GATE_ONLY runs all four.
if [ -f "$HOME/mame/roms/x68000.zip" ] || [ -d "$HOME/mame/roms/x68000" ]; then
DLX_PK_GATE_ONLY=1 DLX_PK_NFR="$NF" bash tools/bench/packed_run.sh \
tmp/packed_singe.dlxp > tmp/packed_gate_stage.log 2>&1 \
|| { cat tmp/packed_gate_stage.log; exit 1; }
# SPLIT AT RUN 5, because the audio run below is in the same log and its
# verifier emits OK lines too -- one grep over the whole file would print
# session 36's result under session 31's heading.
sed -n '1,/--- 5. THE AUDIO/p' tmp/packed_gate_stage.log | \
grep -aE "^ (FLAG|array|chain|frame clock|late frames|WRITE WINDOW)|^OK "
else
echo " SKIPPED: no x68000 romset -- the player was not run"
fi
echo "--- session 36: THE CONTAINER'S OWN AUDIO, OUT OF THE CHIP (FINDINGS 68) ---"
# ROADMAP P6c. Every piece of this existed before this stage did and none of it
# was joined up: the container has carried the lumps since 67, the transport is
# the IPL ROM's channel-3 configuration from 66, and what was missing was the
# lump buffer and the remainder accumulator in a player. This is the run where
# a byte of the container reaches the speaker.
#
# WHY THE GATE IS A WAV AND NOT A COUNTER. src/player/packed.s reports lumps
# armed, lumps fetched, payload bytes and starves, and EVERY ONE OF THOSE CAN BE
# RIGHT WHILE THE SOUND IS WRONG -- which is not hypothetical, it is the bug
# this session shipped: the refill ran one lump too far ahead and overwrote the
# buffer channel 3 was reading out of, and the player's account of it was
# 11 of 11 armed, 11 fetched, 78,125 B, no starve. Nothing parses a packed
# container (FINDINGS 67.4), so a wrong byte is not an error, it is a sound.
# tools/bench/verify_packed_audio.py accounts for all 78,125 bytes against
# MAME's own capture, one delivered byte at a time.
#
# The run is part of packed_run.sh's gate half, so DLX_PK_GATE_ONLY takes it.
if [ -f tmp/packed_gate_stage.log ] && \
grep -aq -- "--- 5. THE AUDIO" tmp/packed_gate_stage.log; then
sed -n '/--- 5. THE AUDIO/,/--- 7. THE SEEK/p' tmp/packed_gate_stage.log | \
grep -aE "^(OK|FAIL) |^ AUDIO:|^ {5}(nibbles per|worst|payload|-> |the player)"
else
echo " SKIPPED: no x68000 romset, or the container is silent"
fi
echo "--- session 33: AUDIO -- the encoder, and what it does to the wire (FINDINGS 65) ---"
# ROADMAP P6, everything in it except the bus half session 20 closed. The audio
# is the SAME WINDOW as the frames -- 00223 from 539.4 s for 10 s -- because an
# audio stream that is not the same seconds as the picture is not this project's
# audio, and a gate that lets the two drift apart would never say so.
# tmp/au_singe.raw was extracted by session 31's stage, which needs it to build
# the container.
# There is NO ffmpeg encoder for this format -- adpcm_ima_oki is decode-only --
# so the encoder cannot be checked against a reference. What is checked is that
# the decoder our encoder runs in its own loop IS ffmpeg's, sample for sample.
# An encoder that agrees with its own wrong decoder is the failure this catches.
python3 tools/bench/verify_adpcm.py tmp/au_singe.raw || exit 1
# And the container arithmetic. The interesting line is the padding: a packed
# record has no index BY DESIGN, so audio has to ride a fixed cadence, and the
# obvious cadence throws away a third of every audio sector.
python3 tools/analysis/32_audio_wire.py tmp/packed_singe.dlxp \
> tmp/audio_wire.log 2>&1 || { cat tmp/audio_wire.log; exit 1; }
grep -aE "^ ( 1| 11| 81) |THE FLOOR|F=1 |F=11|is ZERO|SOUND IS WHAT" tmp/audio_wire.log
echo "--- session 34: THE CHIP'S OWN DECODER, off the machine (FINDINGS 66) ---"
# ROADMAP P6a. 68000 code programs HD63450 channel 3 exactly as the IPL ROM
# programs it and feeds the MSM6258 a designed nibble stream at the chip's own
# pace; ONE of sixteen candidate decoder models reproduces MAME's capture
# sample-exact, and every axis has a negative control. The encoder disagreed
# with the chip on ALL FOUR axes, and the largest of them is not the delta
# formula 65 named -- it is the NIBBLE ORDER, at -25.7 dB.
bash tools/bench/adpcm_run.sh > tmp/adpcm_gate.log 2>&1 || {
cat tmp/adpcm_gate.log; exit 1; }
grep -aE "^(OK|FAIL) |^ (feed|nibbles|delta|clamp|accumulator)" tmp/adpcm_gate.log \
| sed 's/^/ /'
# And the bill, on the same ten seconds every other audio figure is quoted on.
python3 tools/analysis/33_adpcm_model.py tmp/au_singe.raw > tmp/adpcm_model.log 2>&1 \
|| { cat tmp/adpcm_model.log; exit 1; }
grep -aE "wrong only here|played on the chip|decoded on the encoder|headroom left" \
tmp/adpcm_model.log
echo "--- session 35: DLXP2 -- a packed container with sound in it (FINDINGS 67) ---"
# ROADMAP P6b. 65.3 did the arithmetic and wrote no byte; 66 measured which of
# sixteen decoder models the chip runs. This is the container both produce, and
# it needs a gate of its own because NOTHING PARSES A PACKED CONTAINER: a lump
# one sector out does not fail, it paints 512 B of audio and plays 512 B of
# picture, and a gate that only looked for errors would pass it.
#
# The picture side of it is gated twice over: here against a SILENT control
# built from the same frames, and above by the packed player itself, which now
# carries the `(i//F)*A` term and still gets all 120 frames pixel-exact off a
# real volume.
python3 tools/analysis/34_packed_audio.py tmp/packed_singe.dlxp \
> tmp/packed_audio.log 2>&1 || { cat tmp/packed_audio.log; exit 1; }
grep -aE "^ (OK|FAIL) |^ (order|variant|bits|init) |min of play ->" \
tmp/packed_audio.log
echo "--- session 37: THE AUDIO LEVEL, measured off the whole disc (FINDINGS 69) ---"
# ROADMAP P6, the item 66.3 reopened and two sessions deferred. The chip clamps
# its accumulator at 10 bits INSIDE the recursion, and the ten seconds every
# audio figure in this tree is quoted on peak at 435 of 511 -- which fits, and
# fits BY ACCIDENT, because that window is a -13.4 dBFS passage.
#
# So the level is measured against the loudest thing the game can play, which
# means every stream of the unique scene footage (00000-00201, FINDINGS 32.1)
# through extract_audio.py's own chain. The gate asserts the disc's peak and
# the census behind it; a different pressing is a legitimate reason for it to
# go red, a different ffmpeg downmix is not.
#
# ~18 s, and it needs the Blu-ray mounted like every other stage here.
python3 tools/analysis/35_audio_level.py --gate > tmp/audio_level.log 2>&1 \
|| { cat tmp/audio_level.log; exit 1; }
grep -aE "DISC PEAK|LOUDEST PASSAGE|THE CLAMP|LEVEL GATE|NO AUDIO TRACK|^ 1\.0000|^ 0\.5" \
tmp/audio_level.log
echo "--- session 39: THE SEEK, on the machine, with sound across it (FINDINGS 71) ---"
# Run 7 of packed_run.sh, split out of the same log for the same reason session
# 36's stage is split out of it: three verifiers in one file all emit OK lines,
# and one grep over the whole thing prints this session's result under session
# 31's heading.
if [ -f tmp/packed_gate_stage.log ] && \
grep -aq -- "--- 7. THE SEEK" tmp/packed_gate_stage.log; then
sed -n '/--- 7. THE SEEK/,$p' tmp/packed_gate_stage.log | \
grep -aE "^(OK|FAIL) |^ {4}-- the chip|^ {5}(the chip was|error against|-> )|DC +-?[0-9.]+ +AC"
else
echo " SKIPPED: no x68000 romset, or the container is silent"
fi
echo "--- session 39: the audio seek path, run across a real branch (FINDINGS 71) ---"
# ROADMAP P6d, and it is the item session 38's handoff put first: 70.3 named
# what was missing -- "src/player/packed.s starts PG_AK/PG_AKF at lump 0 and has
# no audio seek path at all" -- and priced its absence at a mean 416.5 ms of
# silence over the arcade's 409 within-container seek targets.
#
# The MACHINE half of this is run 7 of tools/bench/packed_run.sh, above: two
# passes over the container with the second starting at frame 37, which is four
# frames into lump 3 and therefore NOT on a group boundary, so the byte offset
# inside the lump is load-bearing. It runs in both chip configurations.
#
# THIS STAGE IS THE HOST HALF, and what it adds is the census the one branch
# point on the machine cannot give. The chip's accumulator is an integrator with
# NO LEAKAGE TERM, so the state error a branch creates is a DC offset that never
# decays -- and the only fix that makes a branch free is in the ENCODER, not the
# player. Both are priced here, and section 1 checks the host's arithmetic for
# the machine's own branch point against what MAME's capture measured.
python3 tools/analysis/37_audio_seek.py > tmp/audio_seek.log 2>&1 \
|| { cat tmp/audio_seek.log; exit 1; }
grep -aE "^(OK|FAIL) |RE-PLAYED BRANCH|\|DC\| against|step index the encoder|frame 37, the branch|reset every|never \(shipped\)|frames \(the cadence\)|1 frame |ONLY FIX" \
tmp/audio_seek.log
echo "--- session 38: the refill climb with a second consumer, through a real branch point (FINDINGS 70) ---"
# ROADMAP P6, the oldest item in it: 65.6 and 67.6 both recorded that the slack
# table existed and that 51.3's climb had never met a branch point with audio on
# the wire. This is that run. It needs the scene graph, so it skips with the
# session-24 stage when there is no checkout.
#
# WHAT IS GATED IS STRUCTURAL, and deliberately not the milliseconds: the
# silences move with the scene table and with the one-container-per-scene
# assumption the tool prints in its own section 5. What must not move is the
# ORDER and the SIGNS -- audio never shortens a climb, the shipped cadence
# strands most within-container branch points off a group boundary, F=1 strands
# none, and the lump read that removes the silence is an order of magnitude
# cheaper than the silence. A tree where any of those flipped has a different
# answer to the cadence pick.
if [ -f "$DIRKSIMPLE/data/games/lair/game.lua" ]; then
python3 tools/analysis/36_branch_audio.py --gate \
> tmp/branch_audio.log 2>&1 || { cat tmp/branch_audio.log; exit 1; }
grep -aE "^ (OK|FAIL) |SECOND CONSUMER IS|BRANCH-AUDIO GATE|^ mean |^ free " \
tmp/branch_audio.log
else
echo " SKIPPED: no DirkSimple checkout at $DIRKSIMPLE"
fi
echo "ALL GREEN"
tools/bench/clock.lua
+240
View File
@@ -0,0 +1,240 @@
-- Drive src/player/clockgate.s: measure the 68000's own FRAME CLOCK.
-- ROADMAP P3.
--
-- Two things are being measured and they need different instruments.
--
-- THE RATE AND THE CADENCE are counted, not timed. The clock's tick is a
-- V-DISP interrupt, and MAME's Lua sees the machine once per screen frame --
-- which is once per V-DISP. So the host's sampling granularity is exactly the
-- clock's own granularity, and the cadence comes out as integers: how many
-- refreshes each frame tick waited. There is no timing error to argue about
-- in a count of 4s and 5s.
--
-- THE COST IS TIMED BY THE 68000, because the host cannot. 1/55.46 s of host
-- granularity is 18 ms and the interrupt costs microseconds. So the 68000 runs
-- a one-instruction loop for a window of thousands of refreshes and the host
-- reads the iteration count at both ends; the interrupt cost falls out of the
-- difference between a run with the clock armed and one without. See the head
-- of src/player/clockgate.s for the arithmetic. This script emits the raw
-- counts; tools/bench/clock_cost.py does the subtraction, so that the two runs
-- it needs can be separate MAME invocations.
--
-- MEASUREMENT SCOPE. This is MAME 0.277's emulated X68000, not real hardware.
-- What is being priced is the interrupt sequence of MAME's cycle-accurate
-- M68000 core (src/devices/cpu/m68000, the `M68000` device x68k.cpp:1133 asks
-- for) against zero-wait-state RAM. Real DRAM adds wait states to the six bus
-- cycles of the exception and the four of the handler alike, so this is a LOWER
-- BOUND in the same way every other 68000 figure in this project is.
--
-- Env:
-- DLX_CLK_ON 1 = arm the frame clock, 0 = leave it off (the calibration
-- run). REQUIRED -- the two runs are not interchangeable and a
-- default would let one be reported as the other.
-- DLX_CLK_FPS frame rate to ask clk_init for (default 12)
-- DLX_CLK_WIN measurement window, in raster frames (default 3000 = 54.1 s)
-- DLX_CLK_OUT where to write the raw counts (default tmp/clock_run.txt)
M = manager.machine
SP = M.devices[":maincpu"].spaces["program"]
local function findfile(n)
for _,p in ipairs{"../tools/bench/"..n, "tools/bench/"..n, n} do
local f = io.open(p,"rb"); if f then f:close(); return p end
end
error(n.." not found")
end
local MODE = loadfile(findfile("crtc_mode.lua"))()
local CGFLAG, CGON, CGCNT = 0x18070, 0x18074, 0x18078
local CLK_PACE = 0x18034
local CLK_ACC, CLK_INCR = 0x18060, 0x18062
local CLK_VDISP, CLK_FPS = 0x18064, 0x18068
local CLK_ERR = 0x1806C
local CPUHZ = 10000000
local ONS = os.getenv("DLX_CLK_ON")
local FPS = tonumber(os.getenv("DLX_CLK_FPS") or "") or 12
local WIN = tonumber(os.getenv("DLX_CLK_WIN") or "") or 3000
local OUT = os.getenv("DLX_CLK_OUT") or "clock_run.txt"
local function P(s) print("[CLK] "..s) end
if ONS ~= "0" and ONS ~= "1" then
P("DLX_CLK_ON must be 0 (calibration, clock off) or 1 (clock armed). The "
.."cost figure is the DIFFERENCE between the two runs, so neither is "
.."meaningful alone and neither gets to be the default.")
M:exit()
return
end
local ON = (ONS == "1")
local code do local f=io.open("clockgate.bin","rb"); code=f:read("a"); f:close() end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
-- Settling frames between the gate reporting `running` and the window opening.
-- The CPU may still be inside clk_init when the host first sees CGFLAG=1, and
-- the first V-DISP edge after arming lands wherever the raster happens to be.
-- Two frames puts the window entirely inside the steady state.
local SETTLE = 2
local st, n = "boot", 0
local f_ready, f0, f1 = nil, nil, nil
local c0, c1, v0, v1, p0, p1, t0, t1
-- Cadence: refreshes between consecutive frame ticks. Recorded as a histogram
-- and as the raw first few, because the interesting claim is not the mean (the
-- divider makes that exact by construction) but that the SPREAD is only ever
-- the two values either side of fps*VTOTAL/HFREQ.
local last_pace, last_pace_f, cad, seen_tick = nil, nil, {}, false
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
n = n + 1
if st == "boot" then
if T() < 3.0 then return end
MODE.apply(SP)
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
SP:write_u32(CGFLAG, 0)
SP:write_u32(CGON, ON and 1 or 0)
SP:write_u32(CLK_FPS, FPS)
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700 -- clk_init lowers it to $2500 itself
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
P(string.format("clockgate.bin=%d B, clock %s, asking for %d fps, "
.."window %d raster frames", #code,
ON and "ARMED" or "OFF (calibration run)", FPS, WIN))
st = "wait"; return
end
if st == "wait" then
local fl = SP:read_u32(CGFLAG)
if fl == 0xEE then
local e = SP:read_u32(CLK_ERR)
P("clk_init REFUSED: CLK_ERR="..e..(e == 1 and
" (CRTC is not in a 31.5 kHz mode, so HFREQ=31500 would be wrong)" or
e == 2 and " (fps*VTOTAL does not fit the 16-bit accumulator)" or ""))
M:exit(); return
end
if fl ~= 1 then
if T() > 60 then P("TIMEOUT: the gate never started"); M:exit() end
return
end
f_ready = n; st = "settle"; return
end
if st == "settle" then
if n < f_ready + SETTLE then return end
f0, t0 = n, T()
c0 = SP:read_u32(CGCNT)
v0 = SP:read_u32(CLK_VDISP)
p0 = SP:read_u32(CLK_PACE)
last_pace, last_pace_f = p0, n
if ON then
P(string.format("armed: incr=%d (fps*VTOTAL), acc=%d, first tick "
.."pending", SP:read_u16(CLK_INCR),
SP:read_u16(CLK_ACC)))
end
st = "run"; return
end
if st == "run" then
if ON then
local pc = SP:read_u32(CLK_PACE)
if pc ~= last_pace then
-- The FIRST change is dropped. Its interval runs from the window
-- opening rather than from a tick, so it measures where the window
-- happened to start and would show up as a spurious short bucket.
if seen_tick then
-- More than one tick in a single refresh would mean fps above the
-- raster rate; give it its own bucket rather than averaging it in.
local gap = n - last_pace_f
if pc - last_pace > 1 then gap = 0 end
cad[gap] = (cad[gap] or 0) + 1
end
seen_tick = true
last_pace, last_pace_f = pc, n
end
end
if n < f0 + WIN then return end
f1, t1 = n, T()
c1 = SP:read_u32(CGCNT)
v1 = SP:read_u32(CLK_VDISP)
p1 = SP:read_u32(CLK_PACE)
st = "done"
local frames = f1 - f0
local secs = t1 - t0
local clocks = secs * CPUHZ
local iters = c1 - c0
local ints = v1 - v0
local ticks = p1 - p0
P(string.format("window: %d raster frames, %.6f s emulated -> %.0f "
.."68000 clocks", frames, secs, clocks))
-- THE INSTRUMENT IS 2.22% FAST AND IT IS WORTH SAYING SO EVERY RUN.
-- The CRTC registers describe a 31,500 lines/s raster of VTOTAL lines.
-- MAME does not run it at that rate: x68k_crtc.cpp refresh_mode()
-- computes the frame period as (scr.max_x * scr.max_y) dots with
-- scr.max_x = m_htotal - 8, one character cell short and an INCLUSIVE
-- rectangle bound used as a count. So the emulated raster is fast by
-- htotal/(htotal-8) -- 368/360 in this mode -- and every rate derived
-- from it here is fast by the same factor. The divider under test is
-- built on the registers, so its HARDWARE rate is the asked-for one and
-- what this rig can check is that it tracks whatever raster it is given.
local vtotal = SP:read_u16(0xE80008) + 1
local htotal = (SP:read_u16(0xE80000) + 1) * 8
local hw_hz = 31500 / vtotal
local skew = htotal / (htotal - 8)
P(string.format(" raster period %.4f ms = %.4f Hz", 1000*secs/frames,
frames/secs))
P(string.format(" the CRTC registers describe 31500/%d = %.4f Hz; "
.."MAME is fast by htotal/(htotal-8) = %d/%d = %.4f",
vtotal, hw_hz, htotal, htotal-8, skew))
P(string.format(" loop iterations %d", iters))
if ON then
P(string.format(" V-DISP interrupts %d, frame ticks %d", ints,
ticks))
-- The self-check that makes the rest of it worth reading: the interrupt
-- count and the host's screen-frame count are supposed to be the SAME
-- clock seen from two sides. If they disagree by more than the one
-- edge the window boundaries can straddle, the tick is not the raster.
if math.abs(ints - frames) > 1 then
P(string.format("FAIL: %d V-DISP interrupts over %d raster frames. "
.."The tick is not coming from the raster.", ints,
frames))
M:exit(); return
end
-- Two numbers, and confusing them is the whole trap. The measured rate
-- is against MAME's fast raster; dividing the skew out gives the rate
-- the same code produces on a machine whose raster matches its own
-- registers, which is the number the player is judged on.
local meas = ticks/secs
P(string.format(" measured rate %.6f fps against MAME's raster "
.."(%+.0f ppm vs the asked %d)", meas,
1e6*(meas/FPS - 1), FPS))
P(string.format(" de-skewed %.6f fps -> %+.1f ppm from %d, "
.."which is the tick quantisation of %d ticks and not "
.."drift", meas/skew, 1e6*(meas/skew/FPS - 1), FPS,
ticks))
local ks = {}
for k in pairs(cad) do ks[#ks+1] = k end
table.sort(ks)
local s = ""
for _,k in ipairs(ks) do
s = s .. string.format("%d:%d ", k, cad[k])
end
P(" cadence, refreshes per frame tick: "..s)
end
local fh = io.open(OUT, "w")
fh:write(string.format("on %d\nfps %d\nframes %d\nsecs %.15g\n"
.."clocks %.15g\niters %d\nints %d\nticks %d\n"
.."vtotal %d\nhtotal %d\nhw_hz %.15g\nskew %.15g\n",
ON and 1 or 0, FPS, frames, secs, clocks, iters,
ints, ticks, vtotal, htotal, hw_hz, skew))
for k, v in pairs(cad) do fh:write(string.format("cad %d %d\n", k, v)) end
fh:close()
P("counts -> "..OUT)
P("done")
M:exit(); return
end
end)
if not ok then print("[CLK] LUA ERROR: "..tostring(err)); M:exit() end
end)
tools/bench/clock_cost.py
+132
View File
@@ -0,0 +1,132 @@
"""What the 68000's own frame clock costs, out of the two clock.lua runs.
ROADMAP P3. Usage: clock_cost.py <off-run.txt> <on-run.txt>
THE SUBTRACTION. Both runs execute the same one-instruction loop over a window
of the same number of raster frames, so the window is the same number of 68000
clocks in both. With the clock off, every clock in the window went into loop
iterations:
L = clocks / iters_off clocks per iteration
With it armed, the interrupts took some of them:
H = (clocks - iters_on * L) / ints clocks per V-DISP interrupt
L is CALIBRATED rather than looked up. That is the point: this project's cost
model (tools/analysis/buscost.py) says a 68000 bus cycle is 4 clocks and an
instruction costs 4 * (instruction words + data accesses), and the whole reason
to measure is to avoid scoring the clock against the table the table is meant to
be checked by. L falling on a whole number of clocks is therefore a RESULT, not
an assumption, and it is reported as one.
WHAT THE FIGURE IS PER FRAME. Not H -- the interrupt fires once per refresh and
a frame is several refreshes. On the hardware raster that is 31500/VTOTAL over
fps interrupts per frame, and the de-skewed rate is the one to use: MAME's
raster is fast by htotal/(htotal-8) (see tools/bench/clock.lua), and charging
the player the emulator's extra interrupts would overstate the cost by that
same 2.2%.
"""
import sys
def read(path):
d, cad = {}, {}
for line in open(path):
f = line.split()
if f[0] == "cad":
cad[int(f[1])] = int(f[2])
else:
d[f[0]] = float(f[1])
d["cad"] = cad
return d
def main(off_path, on_path):
off, on = read(off_path), read(on_path)
if off["on"] != 0 or on["on"] != 1:
sys.exit("FAIL: expected the calibration run first and the armed run "
"second; got on=%d then on=%d" % (off["on"], on["on"]))
for k in ("frames", "clocks", "fps", "vtotal"):
if off[k] != on[k]:
sys.exit("FAIL: the two runs do not share a window: %s is %g in "
"the calibration run and %g in the armed one"
% (k, off[k], on[k]))
clocks = off["clocks"]
L = clocks / off["iters"]
ints = on["ints"]
H = (clocks - on["iters"] * L) / ints
# The self-check that licenses the subtraction: the interrupt count must be
# the raster frame count. clock.lua already fails on this, restated here
# because this file is also read on its own.
if abs(ints - on["frames"]) > 1:
sys.exit("FAIL: %d interrupts over %g raster frames -- not the raster"
% (ints, on["frames"]))
fps, skew = on["fps"], on["skew"]
hw_hz = on["hw_hz"]
per_frame_ints = hw_hz / fps
per_frame = H * per_frame_ints
FRAME_CLK = 10e6 / fps
print(" calibration: %.6f clocks per loop iteration over %d iterations"
% (L, off["iters"]))
print(" (%s a whole number of clocks -- the loop is one "
"`addq.l #1,abs.l` at 7 bus cycles plus a `bra.s`)"
% ("lands on" if abs(L - round(L)) < 1e-3 else "does NOT land on"))
print(" INTERRUPT: %.2f clocks per V-DISP, measured over %d of them"
% (H, ints))
print(" PER FRAME: %.2f interrupts x %.2f = %.0f clocks = %.4f%% of a "
"%g fps frame" % (per_frame_ints, H, per_frame,
100 * per_frame / FRAME_CLK, fps))
print(" (%.4f refreshes per frame on the HARDWARE raster of "
"31500/%d = %.4f Hz, not on MAME's, which is %.4fx fast)"
% (per_frame_ints, on["vtotal"], hw_hz, skew))
# THE DRIFT GATE, and it is stated in TICKS rather than in ppm on purpose.
# A remainder-keeping divider emits floor() or ceil() of the exact tick
# count over any window and never accumulates -- so the only honest
# tolerance is one tick, and any ppm figure is that one tick divided by
# however long the window happened to be. Quoting ppm would let a longer
# window advertise a tighter clock for no reason.
want = on["frames"] * fps * on["vtotal"] / 31500.0
ticks = on["ticks"]
print(" DRIFT: %d ticks over %d refreshes; exact is %.4f, so the "
"error is %+.4f ticks" % (ticks, on["frames"], want, ticks - want))
if abs(ticks - want) > 1.0:
sys.exit("FAIL: %d ticks where %.4f were due -- off by %.2f, which is "
"more than the one tick a remainder can hold back. The "
"divider is accumulating drift." % (ticks, want, ticks - want))
cad = on["cad"]
tot = sum(cad.values())
if tot:
# Refreshes per frame is 31500 / (fps * VTOTAL) exactly -- the divider's
# own ratio, upside down. A remainder-keeping divider can only ever
# emit the two whole numbers either side of it, so anything else in the
# histogram is a bug in the divider and not a rounding taste.
rpf = 31500.0 / (fps * on["vtotal"])
lo, hi = int(rpf), int(rpf) + 1
print(" CADENCE: %s (%d intervals; %.4f refreshes per frame, so "
"only %d and %d are possible)"
% (", ".join("%dx%d (%.1f%%)" % (k, v, 100.0 * v / tot)
for k, v in sorted(cad.items())), tot, rpf, lo, hi))
for k in cad:
if k not in (lo, hi):
sys.exit("FAIL: a frame tick waited %d refreshes, which a "
"remainder-keeping divider cannot produce" % k)
# The mix is forced too: lo*a + hi*b = refreshes, a + b = ticks.
b = tot * rpf - lo * tot
print(" expected %d:%d split %.1f%% / %.1f%%, got "
"%.1f%% / %.1f%%"
% (lo, hi, 100 * (tot - b) / tot, 100 * b / tot,
100.0 * cad.get(lo, 0) / tot, 100.0 * cad.get(hi, 0) / tot))
return 0
if __name__ == "__main__":
if len(sys.argv) != 3:
sys.exit(__doc__)
sys.exit(main(sys.argv[1], sys.argv[2]))
tools/bench/clock_run.sh
Executable
+36
View File
@@ -0,0 +1,36 @@
#!/bin/bash
# One frame-clock run: the 68000 derives its own 12 fps tick from the raster
# (ROADMAP P3, FINDINGS 54).
#
# tools/bench/clock_run.sh [window-in-raster-frames] [fps]
#
# TWO MAME INVOCATIONS, and they are not interchangeable. The first leaves the
# clock off and calibrates the cost of the gate's own loop; the second arms it.
# The interrupt cost is the difference, so a run that reported only the second
# would be reporting a number it cannot compute. See src/player/clockgate.s.
#
# Only MAME can run this: the frame clock is an MFP interrupt driven by the
# CRTC's V-DISP output, and tools/bench/c68k has neither device. That is why
# this stage has no second-core half, unlike load_run.sh.
set -e
cd "$(dirname "$0")/../.."
WIN=${1:-3000}
FPS=${2:-12}
tools/vasm/vasmm68k_mot -Fbin -o tmp/clockgate.bin src/player/clockgate.s > /dev/null
for ON in 0 1; do
# stdbuf -oL: without it a long MAME run is unobservable until it exits, and
# a run that is merely finishing looks exactly like one that is wedged (34.1).
( cd tmp && DLX_CLK_ON=$ON DLX_CLK_FPS=$FPS DLX_CLK_WIN=$WIN \
DLX_CLK_OUT=clock_$ON.txt SDL_VIDEODRIVER=dummy stdbuf -oL \
timeout -k 5 600 mame x68000 -bios ipl10 -ramsize 2M -video soft -window \
-sound none -nothrottle -plugins -autoboot_script ../tools/bench/clock.lua \
-seconds_to_run 240 > clock_$ON.log 2>&1 )
# A run that never reached the counts must fail as that, not as bad arithmetic.
grep -q "^\[CLK\] done" tmp/clock_$ON.log || {
echo "FAIL: the clock rig did not finish run ON=$ON -- no completion marker."
tail -8 tmp/clock_$ON.log; exit 1; }
done
grep -a "^\[CLK\]" tmp/clock_1.log | sed -n '/window:/,/cadence/p' | sed 's/\[CLK\] / /'
python3 tools/bench/clock_cost.py tmp/clock_0.txt tmp/clock_1.txt
tools/bench/crtc_mode.lua
+20
View File
@@ -23,6 +23,26 @@
-- Total blanking time is identical to the 768 mode (112 dots @ 11.592MHz =
-- 336 dots @ 34.776MHz = 9.66us), which is what a real monitor needs.
--
-- THE EMULATOR DOES NOT RUN THE RASTER THESE REGISTERS DESCRIBE, and every
-- rig in this tree samples the machine at ITS rate, not at the hardware's.
-- x68k_crtc.cpp refresh_mode() builds the frame period as
--
-- (scr.max_x * scr.max_y) dots / dotclock, scr.max_x = m_htotal - 8
--
-- which is one character cell short AND uses an inclusive rectangle bound as a
-- count. So MAME's refresh is fast by htotal/(htotal-8) = 368/360 = 1.02222:
-- 56.6901 Hz where the registers say 55.4577. MEASURED, not read off the
-- source alone -- tools/bench/clock.lua reports both every run, and they agree
-- to six digits (FINDINGS 54.5).
--
-- It matters in exactly two places and is harmless in the rest. Anything timed
-- by counting host frames has 1/56.69 s of granularity, not 1/55.46; and
-- anything PACED by the raster runs 2.22% fast under MAME. It does NOT touch
-- 68000 cycle figures: the CPU clock is 40 MHz/4 and has nothing to do with the
-- screen. Do not "correct" the 55.4577 below to match a measurement -- it is
-- the hardware's, derived from the dot clocks above, and it is what
-- src/player/clock.i builds its divider on.
--
-- VERTICAL registers are NOT halved. The CRTC still generates a 568-line
-- 31.5kHz raster (31500/568 = 55.46 Hz); "256 lines" is a graphics-layer
-- double-scan (draw_gfx() halves gfxrect, x68k_v.cpp:401). Halving them would
tools/bench/decode.lua
+1 -1
View File
@@ -93,7 +93,7 @@ end
-- The plan: one sequential correctness pass, then the cost anchors, then a
-- full pass timed. Iteration counts target ~4 emulated seconds each so the
-- 1/55.46 s timing granularity costs under 0.5%.
-- 1/56.69 s timing granularity (crtc_mode.lua) costs under 0.5%.
-- DLX_VERIFY_ONLY=1 drops the cost anchors and runs only the correctness pass,
-- so tools/bench/check.sh can gate the decoder without paying for ~2 minutes of
-- timing runs that would make the green light sensitive to host load anyway.
tools/bench/dlxload.py
+11 -4
View File
@@ -36,11 +36,18 @@ def pack_palette(d):
which is the point: the verifier and the loader must agree or a colour bug
reads as a decoder bug.
Returns (palette bytes 256x2 big-endian, index of the darkest entry). The
encoder does not yet reserve a black entry (docs/STATUS.md, encoder gaps),
so the letterbox gets the closest thing to black the palette has.
Returns (palette bytes 256x2 big-endian, index of the darkest entry, and the
RGB888 the hardware actually RENDERS from those words). The encoder does not
reserve a black entry in the CODEC container (docs/STATUS.md, encoder gaps),
so the letterbox gets the closest thing to black the palette has; the PACKED
container does reserve one (tools/encoder/dlxp.py, index 255).
`d` is a DLX container OR a bare (256,3) uint8 palette. The packed path has
no codebooks and so no DLX object to carry a palette on, and this had to stay
the ONE copy of the GRB555+I maths -- the verifier, the loader and now the
packed encoder all have to agree or a colour bug reads as a decoder bug.
"""
pal = d.pal.astype(int)
pal = (d if isinstance(d, np.ndarray) else d.pal).astype(int)
p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
f = pal >> 3
render = lambda I: p6((f << 1) | I[:, None])
tools/bench/dma.lua
+312
View File
@@ -0,0 +1,312 @@
-- Drive src/player/dmagate.s: does the HD63450 drive the SCSI data phase, and
-- does it HOLD THE BUS? (ROADMAP P4a)
--
-- THE APPARATUS is tools/bench/scsi_run.sh's, unchanged and stated again
-- because it is two substitutions deep: `x68000 -exp1 cz6bs1` (the board 42.5
-- says to benchmark, never x68ksupr, whose internal SCSI is PIO-only in MAME),
-- and a ZERO-FILLED scsiexrom.bin on a private rompath, which is honest only
-- because the player drives the SPC registers directly and never executes a
-- byte of that ROM.
--
-- WHAT THIS RIG DOES NOT DO, and it is the point of the whole design: it never
-- looks at $EA0015. 57.3 showed that address cannot answer the question --
-- with the DMAC's OWN asserted MAME cannot tell a CPU-driven byte there from a
-- DMAC-driven one. What separates the two configurations below is whether the
-- 68000 EXECUTED ANYTHING while the bytes were arriving, which is a fact about
-- the CPU and is read out of the DMAC's own registers plus a counter the
-- machine incremented itself.
--
-- AND IT IS NOT A RATE. MAME's DMAC is configured in wall-clock attotimes
-- (42.5); its burst mode halts the CPU outright rather than charging it cycles
-- per operand. `W` is untouched here and still wants a board.
local M = manager.machine
local SP = M.devices[":maincpu"].spaces["program"]
local function P(s) print("[DMA] "..s) end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local DGFLAG, DGREC, DGREC_SZ = 0x18600, 0x18800, 32
local DGWIN, DGWERR, DGR20, DGR20N, DGR20C = 0x18700, 0x18704, 0x18708, 0x1870C, 0x18710
local DGR20P = 0x18714
local CHROW, CHN, CHBASE = 256, 8, 0xC10000
local R20OF -- filled in after the mailbox addresses are known
local GV = 0xC00000
local DGLBA, DGBLK = 1000, 4
-- the palette runs (ROADMAP K1). PS7/PS8/PS9 are the SNAPSHOTS dmagate.s takes
-- by reading $E82000 back with the 68000 after each run; the registers
-- themselves hold only the last of the three by the time the host looks.
local PAL, PALN, PALB, POIS = 0xE82000, 256, 512, 0xA500
local PS7, PS8, PS9 = 0x1A000, 0x1A200, 0x1A400
local CHROW2, CH2BASE, CHN2ROWS = 256, 0xC14000, 6
local DST = {0x20000, 0x24000, 0x28000, 0xC08000, 0xC0C000, 0xC10000,
PAL, 0x2C000, CH2BASE}
local NAME = {"PIO (the path FINDINGS 58 measured)",
"DMA, BUS HELD (DCR $00 burst, OCR $81 max rate)",
"DMA, STEALING (DCR $80 cycle steal, OCR $80 limited)",
"DMA -> GVRAM (bus held, R20 bit 11 = BUFFER MODE) [47.6.2]",
"DMA -> GVRAM (the SAME, bit 11 CLEAR -- NEGATIVE CONTROL)",
"DMA -> GVRAM (ARRAY CHAINED, 8 rows at the 1024 B line stride)",
"DMA -> PALETTE (bus held, 512 B into $E82000) [K1, 61.9]",
"DMA -> RAM (the SAME read aimed elsewhere -- NEGATIVE CONTROL:"
.." the palette must still read poison)",
"DMA -> PALETTE + SIX ROWS (ONE array-chained start across two"
.." kinds of destination)"}
local SHORT = {"pio", "held", "steal", "gvram", "masked", "chain",
"pal", "palctl", "palchain"}
local LENOF = {[6]=PALB, [7]=PALB} -- everything else is DGBLK*512
local ERRNAME = {[0]="OK", "SELECTION TIMEOUT -- no target answered",
"UNEXPECTED PHASE", "POLL TIMEOUT -- a phase never arrived",
"NON-ZERO SCSI STATUS",
"WINDOWED READ REFUSED -- a channel cannot drop bytes"}
R20OF = {[3]=DGR20, [4]=DGR20N, [5]=DGR20C, [8]=DGR20P}
local DISK = os.getenv("DLX_SCSI_IMG") or "dlxdisk.img"
local code do local f=io.open("dmagate.bin","rb"); code=f:read("a"); f:close() end
-- the disc's own bytes, once, for all three comparisons
local want do
local f = io.open(DISK, "rb")
if f then f:seek("set", DGLBA*512); want = f:read(DGBLK*512); f:close() end
end
local st = "boot"
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
if st == "boot" then
if T() < 3.0 then return end
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
SP:write_u32(DGFLAG, 0)
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
P(string.format("dmagate.bin=%d B loaded at $10000; reading LBA %d, %d B, "
.."three ways, then once more into GVRAM",
#code, DGLBA, DGBLK*512))
st = "wait"; return
end
if st == "wait" then
if SP:read_u32(DGFLAG) ~= 1 then
if T() > 60 then P("TIMEOUT: the gate never finished"); P("done"); M:exit() end
return
end
if not want then P("no "..DISK.." to check against"); P("done"); M:exit(); return end
for i = 0, 8 do
local LEN = LENOF[i] or DGBLK*512
local b = DGREC + i*DGREC_SZ
local rc = SP:read_u32(b)
local e = SP:read_u32(b+4)
local mtc0 = SP:read_u32(b+8)
local spin = SP:read_u32(b+12)
local csr = SP:read_u32(b+16)
local cer = SP:read_u32(b+20)
local mtcf = SP:read_u32(b+24)
local marf = SP:read_u32(b+28)
P(NAME[i+1])
if rc ~= 0 then
P(string.format(" FAILED: err=%d (%s)", e, ERRNAME[e] or "?"))
else
local bad, first = 0, nil
-- The GVRAM run is read back a WORD at a time and split by hand.
-- SP:read_u8 on $C00000 goes through gvram_r, which in buffer mode
-- returns the whole word; asking for one byte of it would hand back
-- whichever half MAME's address space happens to hand over, and the
-- question here is precisely WHICH HALF each disc byte landed in.
-- Even disc byte -> high half (page 1), odd -> low half (page 0),
-- because the 68000 is big-endian and an even address is the MS byte.
local pg1, pg0, bad_hi, bad_lo = 0, 0, 0, 0
for k = 1, LEN do
local got
-- `a` is set for every destination that has to be read a WORD at a
-- time and split by hand -- GVRAM in buffer mode, and the palette
-- snapshots, whose words are what the 68000 read back out of
-- $E82000. Where it stays nil the destination is plain RAM.
local off, a
if i == 5 then
-- The chained run's destination is not linear: byte k of the
-- transfer is byte k%256 of row k//256, and the rows are a full
-- 1024 B line stride apart. If the channel had ignored the array
-- and run contiguously, every byte past the first row would be
-- in the wrong place and this comparison would say so.
off = (k-1) % CHROW
a = CHBASE + ((k-1) // CHROW) * 1024 + (off & ~1)
elseif i == 6 then
off = (k-1) % 2
a = PS7 + ((k-1) & ~1)
elseif i == 8 then
-- ONE transfer across two kinds of destination: the first sector
-- is the palette, the rest is six picture rows at the line
-- stride. The split is the array's, and this walks it the same
-- way the channel was told to.
if k <= PALB then
off = (k-1) % 2
a = PS9 + ((k-1) & ~1)
else
local idx = k - PALB - 1
off = idx % 2
a = CH2BASE + (idx // CHROW2) * 1024 + ((idx % CHROW2) & ~1)
end
elseif i >= 3 and i ~= 7 then
off = (k-1) % 2
a = DST[i+1] + ((k-1) & ~1)
end
if a then
local w = SP:read_u16(a)
if (off % 2) == 0 then got = (w >> 8) & 0xff; pg1 = pg1 + 1
else got = w & 0xff; pg0 = pg0 + 1 end
else
got = SP:read_u8(DST[i+1]+k-1)
end
if got ~= string.byte(want, k) then
bad = bad + 1; first = first or (k-1)
if ((k-1) % 2) == 0 then bad_hi = bad_hi + 1
else bad_lo = bad_lo + 1 end
end
end
if R20OF[i] then
P(string.format(" R20 during the run = $%04X (bit 11 %s); %d bytes "
.."read back out of the HIGH half of a destination "
.."word and %d out of the LOW half",
SP:read_u32(R20OF[i]),
((SP:read_u32(R20OF[i]) & 0x0800) ~= 0)
and "SET" or "CLEAR",
pg1, pg0))
end
-- THE PALETTE RUNS' OWN VACUITY CHECK. Run 7's destination was
-- poisoned by the 68000 first, so "it matches the disc" cannot be
-- satisfied by a channel that did nothing -- but only if the poison
-- and the disc actually differ everywhere they are compared. That is
-- a property of THIS record and is counted rather than assumed.
if i == 6 then
local diff = 0
for j = 0, PALN-1 do
local w = (POIS | j) & 0xffff
if ((w >> 8) & 0xff) ~= string.byte(want, 2*j+1) then diff = diff + 1 end
if (w & 0xff) ~= string.byte(want, 2*j+2) then diff = diff + 1 end
end
P(string.format(" PALETTE POISON IS A DISCRIMINATOR: %d of %d "
.."positions differ from the disc's bytes -- a "
.."channel that wrote nothing could not have passed "
.."in those.", diff, PALB))
P(string.format(" %d bytes read back out of the HIGH half of a "
.."palette word (G and the top of R) and %d out of "
.."the LOW half", pg1, pg0))
end
if bad == 0 then
P(string.format(" BYTES OK: %d B from LBA %d match %s byte for byte "
.."[%s]", LEN, DGLBA, DISK, SHORT[i+1]))
if i == 3 then
P(" A CHANNEL FILLS THE PACKED LAYOUT: every disc byte landed in "
.."its own half of a GVRAM word, with the CPU halted -- so a "
.."stream interleaved (right<<8)|left goes from disc to screen "
.."with no CPU in the loop (47.6.2, first half).")
end
if i == 5 then
P(string.format(" THE CHANNEL WALKED THE ARRAY ITSELF: %d rows of "
.."%d B landed at a %d B line stride from ONE start, CPU halted "
.."throughout. A frame is %d such entries; the CPU does not "
.."restart the channel per row.", CHN, CHROW, 1024, 192))
end
if i == 4 then
P(" CONTROL DID NOT FAIL: the masked write path delivered every "
.."byte too, so the run above is not evidence about R20 bit 11.")
end
if i == 6 then
P(" A CHANNEL WRITES THE PALETTE REGISTERS: 512 B off the disc "
.."became 256 palette words, read back OUT OF $E82000 by the "
.."68000 itself, with the CPU halted for the transfer. Each "
.."disc byte landed in its own half of a register word, so a "
.."per-frame palette needs no CPU (61.9, ROADMAP K1).")
end
if i == 8 then
P(string.format(" ONE START PAINTED THE PALETTE AND %d ROWS: a "
.."single array-chained transfer crossed from device registers "
.."at $%06X into GVRAM at $%06X, %d B in %d entries, CPU halted "
.."throughout. A frame is that shape with %d row entries "
.."instead of %d.", CHN2ROWS, PAL, CH2BASE, LEN, CHN2ROWS+1,
192, CHN2ROWS))
end
elseif i == 6 or i == 8 then
P(string.format(" PALETTE WRONG [%s]: %d of %d differ, first at "
.."+%d -- %d at EVEN offsets (the HIGH half of a "
.."word), %d at ODD.",
SHORT[i+1], bad, LEN, first, bad_hi, bad_lo))
elseif i == 4 then
-- THE CLAIM IS NOT "half the bytes differ". In masked 256-colour
-- mode gvram_w takes `data & 0x00ff` and ignores mem_mask, so a byte
-- written to an EVEN address is never stored and the high half keeps
-- whatever it held; some of those stale halves match the disc by
-- coincidence, and this record is full of pad, so a lot of them do.
-- The mechanism's signature is WHERE the damage is, not how much:
-- every ODD byte must survive and only EVEN ones may be lost.
P(string.format(" BYTES LOST [masked]: %d of %d differ (first at "
.."+%d) -- %d at EVEN offsets, %d at ODD.",
bad, LEN, first, bad_hi, bad_lo))
if bad_lo == 0 and bad_hi > 0 then
P(string.format(" EXACTLY THE MECHANISM: all %d survivors of the "
.."high half are stale GVRAM that happens to match "
.."(this record is mostly pad); not one of the %d "
.."ODD bytes was harmed. Bit 11 is what carried the "
.."even ones in the run above.", LEN//2 - bad_hi, LEN//2))
end
else
P(string.format(" BYTES WRONG [%s]: %d of %d differ, first at +%d",
SHORT[i+1], bad, LEN, first))
end
end
if i == 7 then
-- THE ATTRIBUTION CONTROL'S SECOND CLAIM, and the one that makes run
-- 7 mean something: the same transfer aimed 20 KB away leaves the
-- palette exactly as the 68000 poisoned it. If this comes back with
-- the disc's bytes in it, something other than the channel's MAR
-- decides what reaches $E82000 and run 7 measured that instead.
local stale, first_s = 0, nil
for j = 0, PALN-1 do
if SP:read_u16(PS8 + 2*j) ~= ((POIS | j) & 0xffff) then
stale = stale + 1; first_s = first_s or j
end
end
if stale == 0 then
P(string.format(" PALETTE UNTOUCHED BY THE CONTROL: %d of %d words "
.."still read the poison the 68000 wrote, so the "
.."bytes in run 7 got there because the channel's "
.."MAR pointed at $%06X.", PALN, PALN, PAL))
else
P(string.format(" CONTROL DID NOT FAIL [palctl]: %d of %d palette "
.."words are no longer poison (first at entry %d) "
.."-- the palette changed without a channel aimed "
.."at it.", stale, PALN, first_s))
end
end
if i > 0 then
-- THE DISCRIMINATOR. MTC as the instruction after START saw it, and
-- the number of times the CPU went round its own wait loop.
P(string.format(" MTC one instruction after START: %d of %d -> the "
.."CPU %s while the transfer ran [%s]",
mtc0, LEN,
(mtc0 == 0) and "NEVER EXECUTED" or "kept executing",
SHORT[i+1]))
P(string.format(" CPU trips round the wait loop: %d [%s]", spin, SHORT[i+1]))
P(string.format(" channel: CSR=$%02X (%s%s%s) CER=$%02X MTC=%d "
.."MAR=$%06X (+%d) [%s]",
csr,
((csr & 0x80) ~= 0) and "COC " or "",
((csr & 0x10) ~= 0) and "ERR " or "",
((csr & 0x08) ~= 0) and "ACT" or "idle",
cer, mtcf, marf, marf - DST[i+1], SHORT[i+1]))
end
end
-- The refusal. Expected to fail, and the run is only green if it did.
local w, we = SP:read_u32(DGWIN), SP:read_u32(DGWERR)
if w == 0xFFFFFFFF and we == 5 then
P("WINDOWED DMA READ REFUSED, as it must be: a channel writes a "
.."contiguous run and cannot drop the 300 B in front of the record "
.."(58.3). P4a's precondition is a SECTOR-ALIGNED container.")
else
P(string.format("WINDOW NOT REFUSED: rc=%d err=%d -- the transport would "
.."have written the neighbours' bytes into the ring.", w, we))
end
P("done"); M:exit(); return
end
end)
if not ok then P("LUA ERROR: "..tostring(err)); P("done"); M:exit() end
end)
tools/bench/dma_run.sh
Executable
+154
View File
@@ -0,0 +1,154 @@
#!/bin/bash
# One HD63450 data-phase run: does the DMAC drive the SCSI data phase, and does
# it HOLD THE BUS? (ROADMAP P4a, the last item before M2.)
#
# tools/bench/dma_run.sh [container.dlx]
#
# The apparatus is tools/bench/scsi_run.sh's -- `x68000 -exp1 cz6bs1` and a
# zero-filled scsiexrom.bin on a private rompath -- and the volume is
# tools/bench/mkvol.sh's, the same bytes the host-file ring rig reads.
#
# WHAT A GREEN RUN MEANS: the same 2,048 B came off the disc three ways -- PIO,
# the channel with the bus held, the channel stealing cycles -- all three
# byte-exact against the host's copy; and in the held configuration THE WHOLE
# TRANSFER HAPPENED BETWEEN TWO INSTRUCTIONS, which is what holding the bus
# means and is not a claim about $EA0015 (57.3).
#
# WHAT IT DOES NOT MEAN: anything about `W`. MAME's DMAC runs on wall-clock
# attotimes (42.5) and models a held bus by HALTING the CPU rather than by
# charging it cycles per operand. This settles which configuration works.
set -e
cd "$(dirname "$0")/../.."
DLX=${1:-tmp/rc_fr_singe_scsi_span.dlx}
bash tools/bench/mkvol.sh "$DLX"
tools/vasm/vasmm68k_mot -Fbin -o tmp/dmagate.bin src/player/dmagate.s > /dev/null
# What the player will program, decoded out of the same constants it programs.
python3 tools/analysis/27_dmac_config.py
# stdbuf -oL: without it a long MAME run is unobservable until it exits, and a
# run that is merely finishing looks exactly like one that is wedged (34.1).
( cd tmp && SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 300 \
mame x68000 -bios ipl10 -exp1 cz6bs1 \
-rompath "$HOME/mame/roms;./p4roms" -hard dlxdisk.chd \
-ramsize 2M -video soft -window -sound none -nothrottle -plugins \
-autoboot_script ../tools/bench/dma.lua \
-seconds_to_run 90 > dma_run.log 2>&1 )
grep -aq "^\[DMA\] done" tmp/dma_run.log || {
echo "FAIL: the DMA gate did not finish -- no completion marker."
tail -8 tmp/dma_run.log; exit 1; }
grep -a "^\[DMA\]" tmp/dma_run.log | sed 's/^\[DMA\] / /'
# THE ASSERTIONS. Printing a result and gating on it are different things.
fail() { echo "FAIL: $1"; exit 1; }
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[pio\]" tmp/dma_run.log || \
fail "the PIO reference read did not match -- nothing below is about the DMAC."
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[held\]" tmp/dma_run.log || \
fail "the bus-held DMA read did not deliver the disc's bytes."
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[steal\]" tmp/dma_run.log || \
fail "the cycle-stealing DMA read did not deliver the disc's bytes."
grep -aq "MTC one instruction after START: 0 of 2048 .*NEVER EXECUTED .*\[held\]" \
tmp/dma_run.log || \
fail "the bus was NOT held: the CPU executed while the channel ran, so this is
not the configuration ROADMAP P4a asks for. That MTC is the whole of the
evidence that does not come from watching \$EA0015 (57.3)."
grep -aq "CPU trips round the wait loop: 1 \[held\]" tmp/dma_run.log || \
fail "the held configuration's CPU went round its wait loop more than once --
it was running, so the bus was not held for the whole transfer."
# A NEGATIVE ASSERTION IS WRITTEN AS AN `if`, not as `grep ... && fail`: under
# `set -e` a failing grep in an AND-list takes the whole script's exit status
# with it, so the run would report the failure it was looking for as a pass.
SPIN=$(sed -n 's/.*CPU trips round the wait loop: \([0-9]*\) \[steal\].*/\1/p' \
tmp/dma_run.log)
[ -n "$SPIN" ] && [ "$SPIN" -ge 100 ] || \
fail "the cycle-stealing configuration did not leave the CPU running (spin
= ${SPIN:-none}) -- the two configurations are meant to DIFFER in exactly
that, and a contrast of one against one is not a contrast."
if grep -aq "MTC one instruction after START: 0 of 2048 .*\[steal\]" tmp/dma_run.log
then
fail "the cycle-stealing configuration also finished between two instructions,
so the comparison has no contrast in it and the discriminator is measuring
something other than bus ownership."
fi
grep -aq "COC .*CER=\$00 MTC=0 .*(+2048) \[held\]" tmp/dma_run.log || \
fail "the held channel did not report a clean completion of every byte."
grep -aq "COC .*CER=\$00 MTC=0 .*(+2048) \[steal\]" tmp/dma_run.log || \
fail "the stealing channel did not report a clean completion of every byte."
# ---- the GVRAM run and its control (47.6.2). A channel that writes GVRAM in
# buffer mode is the decoder-free packed player's entire per-frame path, and a
# run with no control is 58.3's vacuous "UNDERRUNS: 0/120" again -- the IPL
# leaves R20 = $0B16, bit 11 ALREADY SET, so the first cut of this test could
# not have failed.
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[gvram\]" tmp/dma_run.log || \
fail "the channel did not fill GVRAM in buffer mode -- a device->GVRAM
transfer is the whole of the decoder-free packed player's frame."
grep -aq "R20 during the run = \$0916 (bit 11 SET)" tmp/dma_run.log || \
fail "the GVRAM run did not run in buffer mode with a KNOWN R20."
grep -aq "R20 during the run = \$0116 (bit 11 CLEAR)" tmp/dma_run.log || \
fail "the negative control did not run with bit 11 clear."
if grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[masked\]" tmp/dma_run.log
then
fail "the MASKED control delivered every byte, so the run above is not a
measurement of R20 bit 11 -- it is a measurement of nothing."
fi
grep -aq "EXACTLY THE MECHANISM" tmp/dma_run.log || \
fail "the masked control lost bytes at ODD offsets too, or lost none at all.
The claim is not a COUNT -- stale GVRAM matches the disc by coincidence
wherever the record is pad -- it is a PLACE: gvram_w's 256-colour arm
drops what the channel wrote to EVEN addresses and stores what it wrote
to odd ones. Damage anywhere else is a different mechanism."
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[chain\]" tmp/dma_run.log || \
fail "the array-chained run did not put the bytes at the row bases its array
named. A picture row is 256 B of a 1024 B line stride, so a frame is 192
destinations; if the channel cannot walk them the CPU has to restart it
per row and the decoder-free path costs a per-row front end."
grep -aq "THE CHANNEL WALKED THE ARRAY ITSELF" tmp/dma_run.log || \
fail "the chained run did not report walking its own array."
# ---- THE PALETTE (ROADMAP K1, FINDINGS 61.9). If the registers at $E82000 take
# a byte-wide DMA the way GVRAM does in buffer mode, a per-frame palette is a
# 193rd array-chain entry and ONE channel start paints a whole frame; if they do
# not, the CPU writes 256 words a frame and the architecture still stands. The
# run is poisoned first and controlled twice -- once by aiming the same transfer
# elsewhere, once by counting how many of the 512 positions the poison and the
# disc actually differ in.
grep -aq "BYTES OK: 512 B from LBA 1000 .*\[pal\]" tmp/dma_run.log || \
fail "the channel did not write the palette registers at \$E82000 -- so a
per-frame palette costs the CPU 256 word writes and cannot ride the
frame's array chain (61.9). That is a RESULT, not a broken run: check the
PALETTE WRONG line above for whether the bytes were dropped or misplaced."
DIFF=$(sed -n 's/.*PALETTE POISON IS A DISCRIMINATOR: \([0-9]*\) of 512.*/\1/p' \
tmp/dma_run.log)
[ -n "$DIFF" ] && [ "$DIFF" -ge 500 ] || \
fail "the poison and the disc's bytes agree in ${DIFF:-?} of 512 positions, so
the palette run could have passed without a channel writing anything --
this is run 4's could-not-fail trap in a new place. Change DGPOIS."
grep -aq "BYTES OK: 512 B from LBA 1000 .*\[palctl\]" tmp/dma_run.log || \
fail "the ATTRIBUTION control's read did not land in RAM, so its palette claim
is about a transfer that did not happen."
grep -aq "PALETTE UNTOUCHED BY THE CONTROL: 256 of 256 words" tmp/dma_run.log || \
fail "the palette changed during a transfer aimed 20 KB away from it. Then
what reached \$E82000 in the run above was not decided by the channel's
MAR, and that run measured something else."
if grep -aq "CONTROL DID NOT FAIL \[palctl\]" tmp/dma_run.log
then
fail "the control reported its own failure -- see the line above it."
fi
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[palchain\]" tmp/dma_run.log || \
fail "ONE array-chained start could not cross from the palette registers into
GVRAM. A frame is one palette entry and 192 row entries; if the two kinds
of destination cannot share a chain, the CPU is back in the video path
once a frame to start the second half of it."
grep -aq "ONE START PAINTED THE PALETTE AND 6 ROWS" tmp/dma_run.log || \
fail "the palette+rows run did not report the crossing it exists to show."
grep -aq "WINDOWED DMA READ REFUSED" tmp/dma_run.log || \
fail "a WINDOWED read through the channel was not refused. 117 of 120 records
start part way into a sector (58.3), and a channel cannot drop the bytes
in front of one -- so it would write the neighbouring records into the
ring, over data the decoder has not finished with, with no bounds check
to catch it (49.2)."
exit 0
tools/bench/gvpack/gvpack
BIN
View File
Binary file not shown.
tools/bench/gvpack/harness.c
+28 -2
View File
@@ -97,6 +97,17 @@ int main(int argc, char **argv)
int iw = (d[4] << 8) | d[5], ih = (d[6] << 8) | d[7];
if (n < (size_t)(8 + 768 + iw * ih)) { fprintf(stderr, "short blob\n"); return 2; }
const BYTE *pix = d + 8 + 768;
/* 'DLXQ' -- the blob is PRE-INTERLEAVED: `pix` is already the bytes a DLXP1
* record carries, in GVRAM order. The ordinary 'DLXR' path computes the
* interleave here, which tests the LAYOUT; this path tests the CONTAINER,
* by writing its bytes verbatim and asking px68k's own gvram.c what they
* display as. The two agreeing is the claim ROADMAP K2 has to make: the
* encoder's byte order is the one 47.2 verified as a picture. */
int prepacked = (d[3] == 'Q');
if (prepacked && !packed) {
fprintf(stderr, "a pre-interleaved blob has no unpacked form\n");
return 2;
}
int yoff = (H - ih) / 2;
const BYTE BLACK = 255;
@@ -121,8 +132,23 @@ int main(int argc, char **argv)
for (int y = 0; y < H; y++) {
DWORD base = 0xC00000 + y * 1024;
for (int i = 128; i < 512; i++) wr16(base + i * 2, 0);
for (int i = 0; i < 128; i++)
wr16(base + i * 2, (WORD)((PIX(y, i + 128) << 8) | PIX(y, i)));
for (int i = 0; i < 128; i++) {
WORD w;
if (prepacked) {
/* The letterbox rows are STATIC SETUP and are not in a
* record (dlxp.py), so they are supplied here, the way a
* player's scene setup supplies them: both halves BLACK. */
if (y < yoff || y >= yoff + ih)
w = (WORD)((BLACK << 8) | BLACK);
else {
const BYTE *row = pix + (y - yoff) * iw;
w = (WORD)((row[i * 2] << 8) | row[i * 2 + 1]);
}
} else {
w = (WORD)((PIX(y, i + 128) << 8) | PIX(y, i));
}
wr16(base + i * 2, w);
}
}
if (!keepbuf) set_r20(R20_DISPLAY); /* back to display */
} else {
tools/bench/gvpack/verify_dlxp.py
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/env python3
"""The PACKED CONTAINER's own bytes, through px68k's real GVRAM code.
python3 tools/bench/gvpack/verify_dlxp.py [packed.dlxp] [frame] [--controls]
`verify_gvpack.py` checks the LAYOUT: it hands the harness a picture and lets
the harness compute the interleave, so what it proves is that FINDINGS 47.2's
scheme renders. This checks the CONTAINER: it writes a DLXP1 record's bytes
into GVRAM VERBATIM -- no interleave computed anywhere in the harness -- and
asks px68k what they display as. That is the only way to test a format whose
whole design is that nothing parses it (dlxp.py): if the encoder's byte order
were wrong, every check upstream of the display would still pass, because the
container round-trips against its own inverse.
It is the same second-emulator argument tools/bench/c68k makes for cycles: the
address decode, the R20 bit-11 write path, the page-byte selection, the scroll
wrap and the index-0 transparency test are px68k's own `x68k/gvram.c`.
Two negative controls, because a test that cannot fail proves nothing, and both
are mechanisms this container depends on rather than decoration:
--nobuffer R20 bit 11 CLEAR -- the high byte of every word is masked away,
so page 1 (columns 128..255) never gets written
--noscroll page 1 unscrolled -- its storage sits under the wrong columns
"""
import os, struct, subprocess, sys
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlxp import DLXP
args = [x for x in sys.argv[1:] if not x.startswith("--")]
path = args[0] if args else "tmp/packed_singe.dlxp"
frame = int(args[1]) if len(args) > 1 else 0
controls = "--controls" in sys.argv
d = DLXP(path)
_, pic = d._split(frame)
blob = b"DLXQ" + struct.pack(">HH", d.W, d.H) + b"\0" * 768 + pic
open("tmp/dlxp_gvpack.bin", "wb").write(blob)
BIN = "tools/bench/gvpack/gvpack"
if not os.path.exists(BIN):
sys.exit(f"{BIN} not built -- make -C tools/bench/gvpack PX68K=...")
def run(extra=None):
cmd = [BIN, "tmp/dlxp_gvpack.bin", "tmp/dlxp_gvpack.raw", "--packed", "0x02"]
if extra:
cmd.append(extra)
subprocess.run(cmd, check=True, stderr=subprocess.DEVNULL)
g = np.frombuffer(open("tmp/dlxp_gvpack.raw", "rb").read(), np.uint8)
return g.reshape(256, 256)
want = d.indices(frame)
yoff = (256 - d.H) // 2
g = run()
act = g[yoff:yoff + d.H]
fail = []
if not np.array_equal(act, want):
bad = act != want
fail.append(f"{bad.sum()} px differ (left half {bad[:, :128].sum()}, "
f"right half {bad[:, 128:].sum()})")
bars = np.concatenate([g[:yoff], g[yoff + d.H:]])
if bars.size and (bars != 255).any():
fail.append(f"letterbox is not index 255: {(bars != 255).sum()} px")
if (act == 0).any():
fail.append(f"index 0 reached the screen: {(act == 0).sum()} px")
if controls and not fail:
for flag, why in (("--nobuffer", "R20 bit 11 clear"),
("--noscroll", "page 1 unscrolled")):
c = run(flag)[yoff:yoff + d.H]
n = int((c != want).sum())
print(f" control {flag:<11s} ({why}): {n:,} px differ"
+ ("" if n else " <-- IT DID NOT FAIL"))
if not n:
fail.append(f"control {flag} passed -- the test cannot fail on it")
for x in fail:
print("FAIL " + x)
if fail:
sys.exit(1)
print(f"OK {os.path.basename(path)} frame {frame}: px68k's own gvram.c renders "
f"the container's {d.pic_bytes:,} bytes index-exact over {d.W}x{d.H},")
print(f" letterbox on the reserved black, and the transparency key never "
f"reaches the screen. The harness computed no interleave.")
tools/bench/load.lua
+178
View File
@@ -0,0 +1,178 @@
-- Time and verify src/player/load.i on the emulated 68000 (ROADMAP P1+P2).
--
-- Two questions, one run, exactly as decode.lua asks them of the decoder:
-- 1. CORRECTNESS. Does the 68000 produce, out of the RAW container header,
-- byte for byte what tools/bench/dlxload.py produces host-side? The
-- expanded codebooks are read back out of RAM and the palette out of the
-- PALETTE REGISTERS -- not out of a RAM shadow, because "the words reached
-- $E82000" is the claim being tested. tools/bench/verify_load.py does the
-- comparison against dlxload.py, so the ground truth stays in one place.
-- 2. COST. How long does it take, split into the codebook expansion and the
-- palette pack, and what is that as a fraction of a 12 fps frame -- the
-- only unit this project prices anything in.
--
-- Nothing here is pre-chewed: the blob pushed into RAM is the first 5,920 bytes
-- of the container as they come off the disc. That is the whole point of the
-- exercise, and it is also, not incidentally, exactly the read a player has to
-- complete at a scene change before it can draw a single frame.
--
-- MEASUREMENT SCOPE, unchanged from decode.lua: MAME's memory carries no wait
-- states, so these are pure 68000 instruction cycles -- a LOWER BOUND on real
-- hardware. Interrupts are masked (SR=$2700). The host clock has 1/56.69 s
-- granularity and the job takes milliseconds, so each configuration is repeated
-- LITER times and divided; repeating is honest because do_load is not
-- temporally recursive -- every pass rewrites what the last one wrote, from the
-- same source bytes.
M = manager.machine
SP = M.devices[":maincpu"].spaces["program"]
local function findfile(n)
for _,p in ipairs{"../tools/bench/"..n, "tools/bench/"..n, n} do
local f = io.open(p,"rb"); if f then f:close(); return p end
end
error(n.." not found")
end
local MODE = loadfile(findfile("crtc_mode.lua"))()
local META = loadfile("load_meta.lua")()
local LFLAG, LHDR, LDARK = 0x18040, 0x18044, 0x18048
local LK1, LK4, LMODE, LITER = 0x1804C, 0x18050, 0x18054, 0x18058
local CB1, CB4, RAW = 0x20000, 0x22000, 0x30000
local GPAL = 0xE82000
local CPUHZ = 10000000 -- x68k.cpp:1133, 40_MHz_XTAL/4
local FPS = 12
local FRAME12 = CPUHZ / FPS
local ITER = tonumber(os.getenv("DLX_LOAD_ITER") or "40")
local code do local f=io.open("loadgate.bin","rb"); code=f:read("a"); f:close() end
local data do local f=io.open("load_data.bin","rb"); data=f:read("a"); f:close() end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local function P(s) print("[LOD] "..s) end
local function push(addr, s, from, len)
local i, n = from, len
while n >= 4 do
SP:write_u32(addr, (string.unpack(">I4", s, i)))
addr, i, n = addr+4, i+4, n-4
end
while n > 0 do
SP:write_u8(addr, string.byte(s,i)); addr, i, n = addr+1, i+1, n-1
end
end
-- Poison every destination before each run. Without this a stage that wrote
-- NOTHING would still compare equal to the previous stage's output, and the
-- palette-only run would "pass" the codebook check for free.
--
-- The three scratch tables are poisoned only before a run that CLAIMS to build
-- them (mode bit 2). They are scene-independent, so the palette-entry stage is
-- entitled to find them already there -- that is the whole point of measuring
-- it separately -- but a stage that says it builds them must be shown to.
local P6TAB, TABEND = 0x19000, 0x19340
local function poison(mode)
for a = CB1, CB1 + META.cb1_len - 2, 2 do SP:write_u16(a, 0xDEAD) end
for a = CB4, CB4 + META.cb4_len - 2, 2 do SP:write_u16(a, 0xDEAD) end
for c = 0, 255 do SP:write_u16(GPAL + c*2, 0xDEAD) end
SP:write_u32(LDARK, 0xFFFFFFFF)
if mode & 4 ~= 0 then
for a = P6TAB, TABEND - 2, 2 do SP:write_u16(a, 0xDEAD) end
end
end
local function setup()
MODE.apply(SP)
push(RAW, data, 1, META.raw_len)
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
P(string.format("loaded loadgate.bin=%d B, raw container header %d B at 0x%X",
#code, META.raw_len, RAW))
end
local function launch(mode, iter)
poison(mode)
SP:write_u32(LFLAG, 0)
SP:write_u32(LHDR, RAW)
SP:write_u32(LMODE, mode)
SP:write_u32(LITER, iter)
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700 -- supervisor, ALL interrupts masked
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
end
-- Written after the mode-3 run, and only after it: it is the output of ONE
-- do_load call over the whole header, which is what the player does.
local function dump()
local out = io.open("load_out.bin", "wb")
for a = CB1, CB1 + META.cb1_len - 1 do out:write(string.char(SP:read_u8(a))) end
for a = CB4, CB4 + META.cb4_len - 1 do out:write(string.char(SP:read_u8(a))) end
for c = 0, 255 do out:write(string.pack(">I2", SP:read_u16(GPAL + c*2) & 0xFFFF)) end
out:close()
P(string.format("dumped %d B of 68000 output to tmp/load_out.bin",
META.cb1_len + META.cb4_len + 512))
P(string.format("DARK=%d (host-side dlxload.py says %d), K1=%d K4=%d",
SP:read_u32(LDARK), META.dark, SP:read_u32(LK1), SP:read_u32(LK4)))
end
-- Order matters: the scratch tables are built by the first stage and the
-- palette-entry stage runs on them, which is exactly how a player would be
-- arranged. The two stages that stand for real player events -- boot, and a
-- scene change -- come last, and the dump the verifier checks is taken from the
-- BOOT one, so the path that is proved correct is the one that builds
-- everything from nothing.
local PLAN = {
{name="scratch tables only (boot, once)", mode=4, iter=ITER},
{name="codebook expansion only (P1)", mode=1, iter=ITER},
{name="palette entries only (P2)", mode=2, iter=ITER},
{name="BOOT: tables + codebooks + palette", mode=7, iter=ITER, dump=true},
{name="SCENE CHANGE: codebooks + palette", mode=3, iter=ITER},
}
local step, st, t0 = 0, "boot", nil
local results = {}
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
local t = T()
if st == "boot" then
if t < 3.0 then return end
setup(); step = 1; launch(PLAN[1].mode, PLAN[1].iter)
st, t0 = "running", nil; return
end
if st == "running" then
local fl = SP:read_u32(LFLAG)
if fl == 1 and not t0 then t0 = t; return end
if fl == 0xEE then
P("BAD HEADER -- load.i did not find the 'DLX3' magic at LHDR")
M:exit(); return
end
if fl == 0xFF then
local p = PLAN[step]
local dt = t - (t0 or t)
local cyc = dt * CPUHZ / p.iter
results[#results+1] = {name=p.name, cyc=cyc}
P(string.format("%s: %d passes in %.4f s -> %.0f cycles = %.1f%% of a "
.."%dfps frame (%.2f ms)", p.name, p.iter, dt, cyc,
100*cyc/FRAME12, FPS, 1000*cyc/CPUHZ))
if p.dump then dump() end
step = step + 1
if PLAN[step] then launch(PLAN[step].mode, PLAN[step].iter); st, t0 = "running", nil
else st = "finish" end
return
end
if t > 400 then P("TIMEOUT flag="..string.format("%08X",fl)); M:exit() end
return
end
if st == "finish" then
P("---- summary (instruction cycles only; real RAM adds wait states) ----")
for _,r in ipairs(results) do
P(string.format(" %-44s %8.0f cyc %5.1f%% of a frame %6.2f ms",
r.name, r.cyc, 100*r.cyc/FRAME12, 1000*r.cyc/CPUHZ))
end
P("done")
M:exit()
end
end)
if not ok then print("[LOD] LUA ERROR: "..tostring(err)); M:exit() end
end)
tools/bench/load_run.sh
Executable
+54
View File
@@ -0,0 +1,54 @@
#!/bin/bash
# One load-time transform run: the 68000 builds its own codebooks and palette
# out of the RAW container header, on both CPU cores (ROADMAP P1+P2, FINDINGS
# 53).
#
# tools/bench/load_run.sh [container]
#
# Both instruments run the same loadgate.bin over the same header bytes:
# * MAME, which is the only one of the two with real PALETTE REGISTERS -- the
# packed words are read back out of $E82000, not out of a RAM shadow, so
# "the words reached the hardware" is part of what passes.
# * px68k's C68K, which is exact to the cycle and counts BUS cycles, and is a
# second opinion on the cost from a separately written cycle table.
# Both outputs are compared byte-for-byte against tools/bench/dlxload.py, which
# stays the reference: this code replaces where those transforms RUN, not what
# they produce.
set -e
cd "$(dirname "$0")/../.."
DLX=${1:-tmp/rc_fr_singe_scsi_span.dlx}
PX68K=${PX68K:-$HOME/src/px68k}
ITER=${DLX_LOAD_ITER:-40}
tools/vasm/vasmm68k_mot -Fbin -o tmp/loadgate.bin src/player/loadgate.s > /dev/null
python3 tools/bench/prep_load.py "$DLX" > tmp/prep_load.log
cat tmp/prep_load.log
# stdbuf -oL: without it a long MAME run is unobservable until it exits, and a
# run that is merely finishing looks exactly like one that is wedged (34.1).
( cd tmp && DLX_LOAD_ITER=$ITER SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 180 \
mame x68000 -bios ipl10 -ramsize 2M -video soft -window -sound none \
-nothrottle -plugins -autoboot_script ../tools/bench/load.lua \
-seconds_to_run 30 > load_check.log 2>&1 )
# A run that never reached the dump must fail as that, not as a byte mismatch.
grep -q "^\[LOD\] done" tmp/load_check.log || {
echo "FAIL: the load rig did not finish -- no completion marker."
tail -6 tmp/load_check.log; exit 1; }
grep -a "^\[LOD\]" tmp/load_check.log | sed -n '/summary/,$p' | sed 's/\[LOD\] / /'
python3 tools/bench/verify_load.py "$DLX"
if [ -f "$PX68K/m68000/c68k.c" ]; then
make -s -C tools/bench/c68k PX68K="$PX68K" 2>/dev/null
for M in 4 1 2 7 3; do
tools/bench/c68k/c68k_bench --code tmp/loadgate.bin --loadraw tmp/load_data.bin \
--loadmode $M --loaditer 1 --cb1 8192 --cb4 2048 \
$([ $M = 3 ] && echo "--loaddump tmp/load_c68k.bin") 2>&1 >/dev/null \
| grep -av arena | sed 's/\[C68K\] / /'
done
# The second core's bytes are held to the same standard as the first's.
cmp -s tmp/load_c68k.bin tmp/load_out.bin || {
echo "FAIL: the two CPU cores produced DIFFERENT load-time output."; exit 1; }
echo " OK both CPU cores produced the same $(stat -c%s tmp/load_out.bin) B"
else
echo " SKIPPED: no px68k at $PX68K (set PX68K= to point at a checkout)"
fi
tools/bench/mkvol.sh
Executable
+76
View File
@@ -0,0 +1,76 @@
#!/bin/bash
# Build the SCSI VOLUME the P4 rigs read, and the blank card ROM MAME needs to
# instantiate the card. Sourced-by-calling from tools/bench/scsi_run.sh and
# tools/bench/pace_run.sh so there is ONE copy of the layout.
#
# tools/bench/mkvol.sh [container.dlx | container.dlxp]
#
# TWO CONTAINERS, ONE VOLUME BUILDER. A DLX volume is tools/bench/prep_stream.py's
# disk image -- the codec's records, laid down from sector 0. A DLXP volume
# (ROADMAP K2/K3) needs no preparation at all: the container is ALREADY a
# sector-aligned image of itself -- a 512 B header, then fixed 97-sector records
# -- so the file IS the volume and copying it is the whole build. That is not a
# convenience, it is the format's central claim (tools/encoder/dlxp.py) arriving
# at the disc, and a builder that transformed it on the way would be hiding the
# claim rather than testing it.
#
# The two get DIFFERENT CHD NAMES. Alternating between the packed rig and the
# codec rig would otherwise rebuild the volume on every run, and -- much worse --
# a stale CHD under the name the other rig expected would serve one container's
# bytes to the other's gate, which reads as a decode failure and is not one.
#
# ONE COPY, ON PURPOSE. The volume is tmp/stream_disk.bin -- byte for byte the
# file the host-file ring rig reads -- laid out as 512 B sectors. If two scripts
# each built it, a difference between the SCSI rig and the modelled-transport rig
# could be a difference in what they were reading, and the whole value of running
# both is that it cannot be. This tree has already paid twice for a transform
# with two copies of itself (FINDINGS 49.7.5, and check.sh's dlxload note).
#
# THE BLANK BOOT ROM is the substitution session 25 argued for and it is
# unchanged: MAME refuses to instantiate the CZ-6BS1 without an 8 KB
# `scsiexrom.bin` (CRC 7be488de) that the player never executes, so a zero-filled
# placeholder goes on a SEPARATE rompath and the user's romset is untouched.
# MAME prints WRONG CHECKSUMS, as it should. DO NOT reuse this rompath for
# anything that boots from the card or calls SCSI IOCS -- those DO execute it.
set -e
cd "$(dirname "$0")/../.."
DLX=${1:-tmp/rc_fr_singe_scsi_span.dlx}
case "$DLX" in
*.dlxp) SRC="$DLX"
IMG=tmp/dlxpdisk.img; CHD=tmp/dlxpdisk.chd ;;
*) SRC=tmp/stream_disk.bin
IMG=tmp/dlxdisk.img; CHD=tmp/dlxdisk.chd
[ -f "$SRC" ] || python3 tools/bench/prep_stream.py "$DLX" > /dev/null ;;
esac
if [ ! -f "$CHD" ] || [ "$SRC" -nt "$CHD" ]; then
SRC="$SRC" IMG="$IMG" python3 - <<'PY'
import os
src, img = os.environ["SRC"], os.environ["IMG"]
d = open(src, "rb").read()
n = (len(d) + 511) // 512
open(img, "wb").write(d + b"\0" * (n * 512 - len(d)))
print(f" disc image: {len(d)} B of records -> {n} sectors")
PY
rm -f "$CHD"
# -c none IS LOAD-BEARING, and it was found by a gate rather than by taste.
# Session 28, on the DLX5 volume: with the default (lzma/zlib/huff/flac) MAME
# 0.277 served the CHD FILE'S OWN BYTES as sector data -- the destination
# buffer after READ(10) at LBA 0 was byte-for-byte the first 4,096 bytes of
# dlxdisk.chd, starting "MComprHD" -- while `chdman verify` reported both SHA1s
# correct. Uncompressed, the identical image reads byte-exact. The trigger is
# the image's CONTENT: the same 8,768-sector length that works for the DLX4
# volume fails for the DLX5 one, a conventional 16x63 geometry fails too, and
# `-c zlib` alone fails as well. The MAME-side cause is NOT diagnosed; what is
# measured is that compression decides it and uncompressed is sound.
# Costs 4.5 MB in tmp/ against 1.6 MB. DO NOT restore compression to save the
# disc space: the failure is SILENT at the transport layer -- every READ(10)
# reports success and returns the wrong bytes -- and only the byte comparison
# in tools/bench/scsi.lua catches it.
chdman createhd -i "$IMG" -o "$CHD" -ss 512 -c none > /dev/null 2>&1
fi
mkdir -p tmp/p4roms/x68k_cz6bs1
[ -f tmp/p4roms/x68k_cz6bs1/scsiexrom.bin ] || \
head -c 8192 /dev/zero > tmp/p4roms/x68k_cz6bs1/scsiexrom.bin
tools/bench/pace_run.sh
+77 -4
View File
@@ -15,12 +15,81 @@
#
# kbps 0 = unlimited pipe. There is no default rate anywhere in this tree
# (FINDINGS 50) and there is none here either.
#
# DLX_RINGOWN=1 hands the RING to the 68000 as well (ROADMAP P5,
# src/player/ring.i): this script's Lua stops placing records and becomes a
# transport that answers one request at a time. DLX_PREFILL_FR is then the
# prefill policy, in whole records. It needs a DLX4 container, because the
# machine cannot learn a record's length by walking a stream it has not fetched.
#
# DLX_ITER=2 runs the scene TWICE, which under DLX_RINGOWN means a real seek
# between the passes: the channel goes quiet, the ring is declared empty and the
# whole accumulated lookahead is thrown away and rebuilt from the prefill. It
# needs DLX_PACE=2, because rebasing the frame clock across a pass is the
# machine's to do and a host-written tick would carry on counting.
#
# DLX_XFER=scsi replaces the MODELLED transport with a real one (ROADMAP P4b,
# src/player/xfer.i): the machine gets a CZ-6BS1 and the same volume the SCSI
# gate reads, this script stops moving bytes altogether, and every record is
# fetched by the 68000 with READ(10). It needs DLX_RINGOWN=1 -- the mailbox it
# answers is ring.i's -- and it FORBIDS a modelled rate, because there is no
# longer anything for one to model.
#
# DLX_PACE selects WHO KEEPS THE TIME: 1 (default) is the host writing the tick,
# 2 is the 68000 writing it off the CRTC's V-DISP (ROADMAP P3, FINDINGS 54).
# Everything else about the run is identical, which is the whole point -- the
# pace gate in src/player/stream.s cannot tell them apart, so a difference in
# the result is a difference in the CLOCK and not in the rig.
set -e
cd "$(dirname "$0")/../.."
RING=${1:?ring KB}; KBPS=${2:?pipe KB/s, or 0 for unlimited}
CUT_AT=$3; CUT_FR=${4:-1}
DLX=${DLX:-tmp/rc_fr_singe_scsi_span.dlx}
PACE=${DLX_PACE:-1}
OWN=${DLX_RINGOWN:-0}
ITERS=${DLX_ITER:-1}
XFER=${DLX_XFER:-model}
# EMULATED seconds the run is allowed. A pass that is cut short compares a
# half-drawn screen and reads as a wrap bug, so this is raised deliberately
# rather than left to a timeout: a DLX_XFER=scsi pass costs ~46 s of emulated
# time against the modelled transport's ~7, because the CPU moves every byte
# itself (FINDINGS 58.2), and two of them do not fit in 90.
SECS=${DLX_SECONDS:-90}
if [ "$XFER" = scsi ]; then
[ "$OWN" = 1 ] || { echo "DLX_XFER=scsi needs DLX_RINGOWN=1: the transport in"
echo "src/player/xfer.i answers src/player/ring.i's mailbox, and with the"
echo "host owning the ring there is no mailbox to answer."; exit 2; }
# A rate is not merely ignored here, it is REFUSED. The bytes now arrive on
# the emulated machine's own time, and a run labelled "488 KB/s" that did not
# deliver at 488 KB/s is exactly the kind of number this project has twice
# paid for. There is no rate in a DLX_XFER=scsi run, and the log says so.
[ "$KBPS" = 0 ] || { echo "DLX_XFER=scsi takes kbps 0. The transport is real,"
echo "so nothing here delivers at a modelled rate -- and MAME's device"
echo "models are functional, not transfer-timing accurate, so the rate it"
echo "DOES deliver at is not a measurement either (docs/BENCHMARK.md)."
exit 2; }
command -v chdman > /dev/null || { echo "DLX_XFER=scsi needs chdman (ships"
echo "with mame-tools) to build the volume."; exit 2; }
bash tools/bench/mkvol.sh "$DLX"
fi
if [ "$OWN" = 1 ] && [ "$ITERS" != 1 ] && [ "$PACE" != 2 ]; then
echo "DLX_ITER>1 needs DLX_PACE=2: the frame clock is rebased per pass by"
echo "src/player/stream.s, and a host-written tick would go on counting"
echo "through the seek and open every slot of the second pass at once."
exit 2
fi
TAG="r${RING}_k${KBPS}${CUT_AT:+_cut${CUT_AT}x${CUT_FR}}"
# The default tag is left ALONE when the host keeps the time: tools/bench/
# pace_sweep.sh reads tmp/pace_r<ring>_k<kbps>.log by name, and renaming the
# host-paced logs would break a sweep that has nothing to do with this option.
if [ "$PACE" != 1 ]; then TAG="${TAG}_p$PACE"; fi
if [ "$OWN" = 1 ]; then TAG="${TAG}_own"; fi
if [ "$ITERS" != 1 ]; then TAG="${TAG}_x$ITERS"; fi
if [ "$XFER" != model ]; then TAG="${TAG}_$XFER"; fi
MAMEX=()
if [ "$XFER" = scsi ]; then
MAMEX=(-exp1 cz6bs1 -rompath "$HOME/mame/roms;./p4roms" -hard dlxdisk.chd)
fi
tools/vasm/vasmm68k_mot -Fbin -o tmp/stream.bin src/player/stream.s > /dev/null
[ -f tmp/stream_disk.bin ] || python3 tools/bench/prep_stream.py "$DLX" > tmp/prep_stream.log
@@ -29,12 +98,16 @@ mkdir -p "tmp/snap_pace_$TAG"; rm -f "tmp/snap_pace_$TAG/x68000"/*.png
# of a prefix is not an assignment token, so bash takes the next word as the
# command and the run dies with "SDL_VIDEODRIVER=dummy: command not found".
CUTENV=(); [ -n "$CUT_AT" ] && CUTENV=(DLX_CUT_AT="$CUT_AT" DLX_CUT_FR="$CUT_FR")
( cd tmp && env DLX_PACE=1 DLX_RING_KB=$RING DLX_STREAM_KBPS=$KBPS \
( cd tmp && env DLX_PACE=$PACE DLX_RING_KB=$RING DLX_STREAM_KBPS=$KBPS \
DLX_RINGOWN=$OWN DLX_ITER=$ITERS DLX_XFER=$XFER \
${DLX_PREFILL_FR:+DLX_PREFILL_FR=$DLX_PREFILL_FR} \
"${CUTENV[@]}" DLX_SLACK_CSV="slack_$TAG.csv" \
SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 900 \
mame x68000 -bios ipl10 -ramsize 2M -video soft -window -sound none \
mame x68000 -bios ipl10 "${MAMEX[@]}" -ramsize 2M -video soft -window \
-sound none \
-nothrottle -plugins -autoboot_script ../tools/bench/stream.lua \
-snapshot_directory "./snap_pace_$TAG" -snapview native -seconds_to_run 90 \
-snapshot_directory "./snap_pace_$TAG" -snapview native \
-seconds_to_run $SECS \
> "pace_$TAG.log" 2>&1 )
# The completion marker is not optional: a run killed mid-decode compares a
# half-drawn screen and reads as a wrap bug rather than as a truncated run.
@@ -42,6 +115,6 @@ grep -q "snapshot taken" "tmp/pace_$TAG.log" || {
echo "FAIL($TAG): no snapshot marker -- the pass did not complete."
tail -6 "tmp/pace_$TAG.log"; exit 1; }
echo "=== $TAG"
grep -aE "decoder (PACED|FREE)|ring: |UNDERRUNS|SEEK SLACK|RING-BOUND|RATE-BOUND|BUILD TIME|PIPE CUT|DEADLINE|REQUIRED" \
grep -aE "decoder (SELF-PACED|PACED|FREE)|FRAME CLOCK|ring: |UNDERRUNS|NO IDLE|SEEK SLACK|RING-BOUND|RATE-BOUND|BUILD TIME|PIPE CUT|DEADLINE|REQUIRED|MACHINE-OWNED|PREFILL:|CHANNEL IDLE|MISPLACED|SEEK PASS|REAL TRANSPORT|SECTOR OVERHEAD|TRANSPORT FAILED|IS VACUOUS" \
"tmp/pace_$TAG.log" | sed "s/\[STR\] / /"
python3 tools/bench/verify_decode.py "$DLX" --snap "tmp/snap_pace_$TAG" | tail -2
tools/bench/packed.lua
+509
View File
@@ -0,0 +1,509 @@
-- Drive src/player/packed.s: THE DECODER-FREE PACKED PLAYER, END TO END,
-- OFF A REAL VOLUME. ROADMAP K3.
--
-- WHAT THIS SCRIPT DOES NOT DO IS THE POINT OF IT. tools/bench/stream.lua
-- pushes expanded codebooks and a packed palette into RAM, plays a transport at
-- a modelled byte rate, and writes the frame tick. This one pushes 2,898 bytes
-- of 68000 code and eleven mailbox words, and then READS. It moves no picture
-- byte, models no rate, sets no CRTC register and writes no palette entry: the
-- machine brings up its own display, builds its own chain, keeps its own clock
-- off V-DISP and fetches every record itself with READ(10) off a CZ-6BS1.
--
-- SO THE GATE IS NOT THE ONE THE CODEC USES, AND IT HAD TO CHANGE.
-- tools/bench/verify_decode.py checks ONE frame -- the last -- and that audits
-- the whole run because the codec is temporally recursive: a SKIP block is a
-- claim about the previous frame still being on screen, so the final frame is
-- only right if all 120 were. A packed frame is a LITERAL. Frame 119 being
-- pixel-exact says nothing whatever about frame 60. This script therefore
-- snapshots EVERY frame and tools/bench/verify_packed.py compares all of them;
-- the simplification that deleted the ring also deleted the gate's free lunch.
--
-- WHEN A SNAPSHOT IS TAKEN, and why not on the frame it changed. PG_SHOWN is
-- bumped by the 68000 after it clears R20 bit 11, so a change means "a complete
-- frame is now displayable". But MAME's screen bitmap for the host frame in
-- progress was drawn partly before that instant, so snapshotting immediately
-- would sample the write window -- which BLANKS the graphics layer -- for part
-- of the picture. A 12 fps frame lasts 4 or 5 host refreshes at 56.69 Hz, so
-- waiting SNAP_DELAY whole host frames is safely inside the slot and safely
-- after the window closed.
--
-- Env:
-- DLX_PK_HELD 1 = the channel HOLDS THE BUS (burst, max rate), 0 = it
-- steals cycles. Not two speeds of one thing: 59.3 showed an
-- auto-requested channel is charged by TIME, so held is the
-- 68000 stopped for as long as the record takes to arrive.
-- Default 1.
-- DLX_PK_PACE 1 = the machine holds itself to the container's fps off
-- V-DISP (default). 0 free-runs, which tests the CHAIN with
-- the clock out of the way.
-- DLX_PK_ITER passes over the scene (default 1). >1 exercises the SEEK.
-- DLX_PK_SEEK the frame passes after the first start at (default 0, i.e.
-- a replay). On the video path the seek IS arithmetic and
-- nothing else; on the audio path it is a second read at a
-- separate LBA, because a DLXP2 group puts lump k in FRONT of
-- its records and a branch lands `f mod F` frames into it
-- (FINDINGS 70.3). Setting this to a frame that is NOT a
-- multiple of the cadence is the point: 36 of the arcade's
-- 409 within-container seek targets land on a boundary and
-- 373 do not.
-- DLX_PK_NFR play only the first N frames (default: all of them)
-- DLX_PK_FPS pace at this rate instead of the container's. NOT a
-- cosmetic knob and not a way to make a number look better:
-- under MAME the emulated transport takes about a whole 12 fps
-- slot to deliver a 49,664 B record, and the write window has
-- to be OPEN for all of it -- so at 12 fps there is no instant
-- at which a complete frame is displayable and nothing can be
-- snapshotted. Pacing slower opens a display interval without
-- changing one byte of the transfer, which is what lets the
-- PIXEL-EXACTNESS of all 120 frames be gated separately from
-- the RATE the emulated transport happens to run at. The two
-- are different questions and this is the knob that separates
-- them.
-- DLX_PK_CSV write the per-frame arrival series here
M = manager.machine
SP = M.devices[":maincpu"].spaces["program"]
local function findfile(n)
for _,p in ipairs{"../tools/bench/"..n, "tools/bench/"..n, n} do
local f = io.open(p,"rb"); if f then f:close(); return p end
end
error(n.." not found")
end
local META = loadfile("packed_meta.lua")()
-- src/player/packed.s. Inputs first, then outputs; the split is the file's.
local PG_FLAG, PG_NFR, PG_FPS, PG_LBA0 = 0x18900, 0x18904, 0x18908, 0x1890C
local PG_RECS, PG_PALL, PG_HELD = 0x18910, 0x18914, 0x18918
local PG_PACEON, PG_ITER = 0x1891C, 0x18920
local PG_CADF, PG_CADA, PG_AUDON = 0x18924, 0x18928, 0x1892C
local PG_SHOWN, PG_ERR, PG_ERRAT = 0x18930, 0x18934, 0x18938
local PG_LATE, PG_LATE1, PG_LATEM = 0x1893C, 0x18940, 0x18944
local PG_VDISP, PG_VD0, PG_TSPIN = 0x18948, 0x1894C, 0x18950
local PG_GSPIN, PG_LOSTV, PG_ARRN = 0x18954, 0x18958, 0x1895C
-- ROADMAP P6c: the audio path's inputs and its own account of what it did.
local PG_AFPS, PG_AHZ, PG_ALBA0 = 0x18960, 0x18964, 0x18968
local PG_NLUMP, PG_ABYTES, PG_APRE = 0x1896C, 0x18970, 0x18974
local PG_SEEKF, PG_ARST = 0x18978, 0x1897C
local PG_APOS, PG_ASKIP = 0x189C8, 0x189CC
local PG_ASKN, PG_ASKB, PG_TSEQ = 0x189D0, 0x189D4, 0x189DC
local PG_AARM, PG_AFET, PG_ABYT = 0x18980, 0x18984, 0x18988
local PG_ADRY, PG_ASEAM, PG_ASRV = 0x1898C, 0x18990, 0x18994
local PG_ACSR, PG_ACER, PG_ALATE = 0x18998, 0x1899C, 0x189A0
local PG_AMTC0, PG_AK, PG_AKF, PG_AACC = 0x189A4, 0x189B0, 0x189B4, 0x189B8
local PG_AFERR, PG_AFERA = 0x189A8, 0x189AC
local PG_ARR = 0x1B000
-- src/player/clock.i and src/player/scsi.i, read for diagnosis only.
local CLK_PACE, CLK_VDISP, CLK_ERR = 0x18034, 0x18064, 0x1806C
local SC_ERR = 0x18200
local CRTC_R20 = 0xE80028
local HELD = (os.getenv("DLX_PK_HELD") or "1") == "1"
local PACED = (os.getenv("DLX_PK_PACE") or "1") == "1"
local ITERS = tonumber(os.getenv("DLX_PK_ITER") or "") or 1
local SEEKF = tonumber(os.getenv("DLX_PK_SEEK") or "") or 0
-- ROADMAP P6d. 1 = STOP and re-PLAY the chip at a branch. Not a tidiness
-- knob: the chip's accumulator has no leak, so the two settings are a large
-- decaying error against a small permanent one (FINDINGS 71.3).
local ARST = (os.getenv("DLX_PK_ARST") or "0") == "1"
local NFR = tonumber(os.getenv("DLX_PK_NFR") or "") or META.nframes
local FPS = tonumber(os.getenv("DLX_PK_FPS") or "") or META.fps
local CSV = os.getenv("DLX_PK_CSV")
-- ROADMAP P6c. OFF by default and it is not a convenience: a run with the chip
-- silent is the CONTROL this one is read against, and every gate that existed
-- before session 36 is that control. DLX_PK_APRE is the lumps fetched before
-- frame 0 -- see src/player/packed.s on why the answer is not 1.
local AUDIO = (os.getenv("DLX_PK_AUD") or "0") == "1"
local APRE = tonumber(os.getenv("DLX_PK_APRE") or "") or 2
local AJSON = os.getenv("DLX_PK_AJSON")
local SNAP_DELAY = 2
local SCERRNAME = {[0]="OK", "SELECTION TIMEOUT -- no target answered",
"UNEXPECTED PHASE", "POLL TIMEOUT -- a phase never arrived",
"NON-ZERO SCSI STATUS",
"WINDOWED READ REFUSED -- a channel cannot drop bytes"}
local code do local f=assert(io.open("packed.bin","rb")); code=f:read("a"); f:close() end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local function P(s) print("[PK] "..s) end
local function setup()
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
SP:write_u32(PG_FLAG, 0)
SP:write_u32(PG_NFR, NFR)
SP:write_u32(PG_FPS, FPS)
SP:write_u32(PG_LBA0, META.lba0)
SP:write_u32(PG_RECS, META.rec_sectors)
SP:write_u32(PG_PALL, META.palette_last)
SP:write_u32(PG_HELD, HELD and 1 or 0)
SP:write_u32(PG_PACEON, PACED and 1 or 0)
SP:write_u32(PG_ITER, ITERS)
SP:write_u32(PG_SEEKF, SEEKF)
SP:write_u32(PG_ARST, ARST and 1 or 0)
-- DLXP2's cadence. Zero for a silent container, and the 68000 branches on the
-- zero: a player told the wrong cadence does not fail, it reads an audio lump
-- as a record and paints it.
SP:write_u32(PG_CADF, META.cad_f or 0)
SP:write_u32(PG_CADA, META.cad_a or 0)
SP:write_u32(PG_SHOWN, 0)
-- P6c. AUDON is separate from the cadence on purpose: the LBA arithmetic has
-- to skip the lumps whether or not a chip is being fed, and a run that skips
-- them without playing them is the control this one is measured against.
SP:write_u32(PG_AUDON, (AUDIO and (META.has_audio or 0) == 1) and 1 or 0)
SP:write_u32(PG_AFPS, META.fps) -- the CONTAINER's, NOT the pace
SP:write_u32(PG_AHZ, META.aud_hz or 0)
SP:write_u32(PG_ALBA0, META.lba_aud or 0)
SP:write_u32(PG_NLUMP, META.n_lumps or 0)
SP:write_u32(PG_ABYTES, META.aud_bytes or 0)
SP:write_u32(PG_APRE, APRE)
if ITERS > 1 then
local grp = (META.cad_f or 0) > 0 and (SEEKF % META.cad_f) or 0
P(string.format("SEEK: %d passes, and passes 2..%d start at FRAME %d%s",
ITERS, ITERS, SEEKF,
(META.cad_f or 0) == 0 and " (silent container)"
or string.format(" -- lump %d, %d frame(s) into its group "
.."of %d, so the audio needs a second read and a byte "
.."offset (FINDINGS 70.3)%s",
SEEKF // META.cad_f, grp, META.cad_f,
grp == 0 and " -- ON a group boundary, the free case"
or "")))
P(ARST and " the chip is STOPPED and re-PLAYED at the branch: its "
.."accumulator goes to the container's own init and its step "
.."index to 0"
or " the chip PLAYS THROUGH the branch: it keeps the predictor "
.."state the previous scene's audio left it in")
end
P(string.format("packed.bin=%d B, %dx%d %d fps, %d of %d frames, %d passes",
#code, META.W, META.H, META.fps, NFR, META.nframes, ITERS))
local cad = ""
if (META.cad_f or 0) > 0 then
cad = string.format(" + (i//%d)*%d", META.cad_f, META.cad_a)
end
P(string.format("record %d B = %d sectors at LBA %d + i*%d%s, palette %s",
META.rec_bytes, META.rec_sectors, META.lba0,
META.rec_sectors, cad,
META.palette_last == 1 and "LAST" or "FIRST"))
if (META.cad_f or 0) > 0 then
P(string.format("DLXP2: %d B of ADPCM at %d Hz rides in %d lumps of %d sectors, "
.."one in front of every %d records -- the third term above is the "
.."whole cost of it on the video path",
META.aud_bytes, META.aud_hz, META.n_lumps, META.cad_a, META.cad_f))
end
if AUDIO and (META.has_audio or 0) == 1 then
P(string.format("AUDIO ON: lump k at LBA %d + k*%d, payload 11*%d/24 B a "
.."group -- the PAYLOAD and not the %d B lump (FINDINGS "
.."67.2). Decoder from the header: %s/%s, %d-bit clamp, "
.."accumulator %d at PLAY. Prefill %d lumps of %d.",
META.lba_aud, META.cad_f*META.rec_sectors + META.cad_a,
META.aud_hz, META.cad_a*512, META.aud_variant,
META.aud_order, META.aud_bits, META.aud_init,
APRE, META.n_lumps))
elseif (META.has_audio or 0) == 1 then
P("audio present in the container and NOT played -- this is the silent "
.."control (DLX_PK_AUD=1 plays it)")
end
P(string.format("channel: %s, %s",
HELD and "BUS HELD (burst, max rate)" or "CYCLE STEALING",
PACED and ("SELF-PACED at "..FPS.." fps off V-DISP"
..(FPS ~= META.fps and (" -- NOT the container's "
..META.fps..", see DLX_PK_FPS") or ""))
or "FREE-RUNNING (tests the chain, not the clock)"))
P("this script writes NO picture byte, NO palette entry and NO CRTC register: "
.."the machine brings up its own display and fetches its own records.")
end
local function launch()
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700 -- supervisor, all interrupts masked;
cpu.state["SP"].value = 0x8000 -- clk_init lowers it to $2500 itself
cpu.state["PC"].value = 0x10000
end
local st, t0 = "boot", nil
local shown, pending, snaps = 0, nil, 0
local arrive, hostfr, missed = {}, 0, 0
-- WHICH FRAME EACH SNAPSHOT IS. MAME numbers snapshots 0000, 0001, ... in the
-- order they were taken, and a frame that could not be sampled leaves no gap in
-- that sequence -- so the file name is NOT the frame index and a verifier that
-- assumed it was would compare frame 61 against record 60 and report a codec
-- bug that is really a bookkeeping one. This is the map, written out for it.
local snapfr = {}
local r20seen = {}
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
local t = T()
if st == "boot" then
if t < 3.0 then return end
setup(); launch(); t0 = t; st = "running"; return
end
if st ~= "running" then return end
hostfr = hostfr + 1
-- WHAT THE SCREEN MODE WAS, sampled every host frame. R20 bit 11 blanks
-- the graphics layer, so this is the only way to see the shutter the player
-- is running: the fraction of host frames that found the window OPEN is the
-- fraction of the scene the display spent dark, and it is a MEASUREMENT of
-- 47.4's cost under MAME rather than a restatement of the prior.
local r20 = SP:read_u16(CRTC_R20)
r20seen[#r20seen+1] = ((r20 >> 11) & 1)
local s = SP:read_u32(PG_SHOWN)
if s > shown then
-- Only the LAST change matters if several landed in one host frame; that
-- cannot happen at 12 fps on a 56.69 Hz raster, and if it ever does the
-- gate below catches it as a missing snapshot rather than a wrong one.
arrive[#arrive+1] = {n = s, t = t - t0}
shown = s
pending = SNAP_DELAY
end
if pending then
pending = pending - 1
if pending <= 0 then
pending = nil
if (SP:read_u16(CRTC_R20) >> 11) & 1 == 0 then
M.video:snapshot(); snaps = snaps + 1
snapfr[#snapfr+1] = shown - 1
else
-- The window was open again when the delay expired: the frame we
-- meant to sample is being overwritten. COUNTED, NOT PRINTED -- when
-- the transfer is longer than the slot EVERY frame misses, and 119
-- identical lines bury the four numbers the run exists to report.
-- The count is reported once at the end and the gate reads it there.
missed = missed + 1
end
end
end
local flag = SP:read_u32(PG_FLAG)
if flag ~= 1 and pending then
-- THE LAST FRAME IS STILL PENDING. packed.s spins in pg_hold with the
-- window CLOSED once the scene is over, so the delay can simply run out;
-- returning here rather than reporting is what stops the final frame
-- being the one frame the gate never sees.
return
end
if flag ~= 1 then
st = "done"
local wall = t - t0
P(string.format("FLAG=$%02X after %.3f s, %d frames shown, %d snapshots, "
.."%d frames NOT SAMPLED (the write window had reopened "
.."-- the transfer is longer than the display interval)",
flag, wall, shown, snaps, missed))
local err = SP:read_u32(PG_ERR)
if err ~= 0 then
P(string.format("TRANSPORT FAILED on frame %d: %s",
SP:read_u32(PG_ERRAT), SCERRNAME[err] or ("code "..err)))
end
P(string.format("array: the 68000 built %d entries (the container wants "
.."%d)", SP:read_u32(PG_ARRN), META.entries))
-- The first entry, read back out of the machine's own RAM. It is the one
-- place palette-first and palette-last are visible as a FACT rather than
-- as a flag the rig passed in and the rig read back.
P(string.format("chain[0] = MAR $%06X MTC %d ; chain[1] = MAR $%06X MTC %d",
SP:read_u32(PG_ARR), SP:read_u16(PG_ARR+4),
SP:read_u32(PG_ARR+6), SP:read_u16(PG_ARR+10)))
if PACED then
local vd = SP:read_u32(PG_VDISP) - SP:read_u32(PG_VD0)
-- THE CLOCK, AGAINST THE RASTER THAT DROVE IT. CLK_VDISP counts the
-- edges the 68000's ISR SAW. hostfr counts the frames MAME actually
-- drew. A held channel halts the CPU, and the MFP's pending bit is one
-- bit, so an edge that falls inside a transfer long enough to span two
-- of them is an edge the machine can never count. Nothing in this
-- project has ever run a transfer and a clock at once, so nothing could
-- have seen this before.
P(string.format("frame clock: PACE=%d ticks, V-DISP edges SEEN=%d, "
.."host frames drawn=%d -> %d edges LOST (%.1f%%)",
SP:read_u32(CLK_PACE), vd, hostfr, hostfr - vd,
hostfr > 0 and (hostfr-vd)*100/hostfr or 0))
local nlate = SP:read_u32(PG_LATE)
P(string.format("late frames (tick already past at the gate): %d%s",
nlate, nlate > 0 and string.format(", first %d, worst "
.."%d ticks", SP:read_u32(PG_LATE1),
SP:read_u32(PG_LATEM)) or ""))
-- AND WHY `late = 0` IS NOT `on time`. The gate compares the frame
-- index against PACE, and PACE is advanced by the ISR that the held
-- channel stops the CPU from running. A clock that loses edges loses
-- them from BOTH sides of the comparison, so a player whose own clock
-- has halved still reports every frame early. The LOST figure above is
-- the only thing in this run that can contradict it, and it comes from
-- the host's raster count rather than from the machine.
if hostfr - vd > 0 then
P(string.format(" ...and %d of those ticks were never "
.."issued, so `late=%d` is measured against a clock "
.."running at %.1f%% of the raster. The player "
.."believes it is at %d fps and the screen is at "
.."%.2f.", hostfr - vd, nlate, vd*100/hostfr, FPS,
FPS * vd / hostfr))
end
end
P(string.format("CPU: %d trips round the TRANSFER wait in total, %d round "
.."the PACE gate", SP:read_u32(PG_TSPIN),
SP:read_u32(PG_GSPIN)))
if AUDIO and (META.has_audio or 0) == 1 then
local armed, fet = SP:read_u32(PG_AARM), SP:read_u32(PG_AFET)
local byt, dry = SP:read_u32(PG_ABYT), SP:read_u32(PG_ADRY)
local seam, srv = SP:read_u32(PG_ASEAM), SP:read_u32(PG_ASRV)
local late, acc = SP:read_u32(PG_ALATE), SP:read_u32(PG_AACC)
P(string.format("AUDIO: %d of %d lumps armed, %d fetched, %d B of "
.."payload handed to the chip (the stream is %d B)",
armed, META.n_lumps, fet, byt, META.aud_bytes))
-- THE PADDING, CHARGED. A player that fed the chip the whole lump
-- would have handed it n_lumps*A*512 B; the difference is the drift
-- FINDINGS 67.2 priced at 1.25 s over the game, and printing both
-- numbers is the only way the accumulator is visible from outside.
-- FULL groups only. The last lump is short when the scene's frame
-- count is not a multiple of F, and averaging that in reports the
-- scene's TAIL as though it were the cadence -- which is a different
-- number from the drift and looks like a worse one. The percentage
-- lives in verify_packed_audio.py, which knows each lump's payload.
local nfull = META.n_lumps
if NFR % META.cad_f ~= 0 then nfull = nfull - 1 end
local lumpb = nfull * META.cad_a * 512
P(string.format(" %d whole groups: %d B of lump space for the "
.."payload the accumulator asked for. The whole-lump "
.."player feeds the chip that space, and the excess is "
.."DRIFT and not waste (67.2). Accumulator left at "
.."%d/%d.", nfull, lumpb, acc, 2*META.fps))
P(string.format(" service: %d calls, %d found the channel "
.."counted out, %d of those had NO lump ready (a "
.."STARVE -- the chip replays its last byte)",
srv, seam, dry))
P(string.format(" re-arms with MTC still non-zero: %d (bytes "
.."fetched and never played; 0 is the correct value)",
late))
-- THE SEEK'S OWN ACCOUNT. PG_ASKN is the second reads and PG_ASKB the
-- bytes they skipped at the head of a lump: a seek path with no offset
-- term would report the first and zero for the second, and would be
-- indistinguishable from a correct one on any target that happened to
-- land on a group boundary.
local skn, skb = SP:read_u32(PG_ASKN), SP:read_u32(PG_ASKB)
if skn > 0 then
P(string.format(" SEEK: %d audio seek(s), %d B skipped into "
.."the head of a lump. Stream position ended at %d B "
.."and the chip was handed %d -- they differ BY the "
.."skip, which is the whole reason they are two cells "
.."(FINDINGS 71).", skn, skb, SP:read_u32(PG_APOS),
byt))
end
local ferr = SP:read_u32(PG_AFERR)
if ferr ~= 0 then
P(string.format(" A LUMP FETCH FAILED on lump %d: %s -- the "
.."picture is unaffected and the sound is gone, which "
.."is why this has its own error word",
SP:read_u32(PG_AFERA), SCERRNAME[ferr] or ("code "..ferr)))
end
P(string.format(" channel 3 at the end: CSR=$%02X CER=$%02X, "
.."MTC one instruction after the first START = %d",
SP:read_u32(PG_ACSR), SP:read_u32(PG_ACER),
SP:read_u32(PG_AMTC0)))
-- WHAT HOLDING THE BUS COSTS A SECOND CONSUMER, and it is this line.
-- Stealing, pg_aserv runs from inside dma.i's transfer wait as well as
-- twice a frame; held, the 68000 is HALTED for the whole transfer and
-- the two frame-loop calls are all it gets. The ratio is the audio's
-- half of FINDINGS 64.3.
P(string.format(" -> %.1f service calls per frame shown. %s",
shown > 0 and srv/shown or 0,
HELD and ("BUS HELD: the 68000 is halted for the whole "
.."transfer, so DM_HOOK never runs and this is the "
.."two frame-loop calls and nothing else.")
or ("CYCLE STEALING: DM_HOOK ran from inside the "
.."transfer wait, which is where a 68000 driving this "
.."video path has any time at all.")))
if AJSON then
local f = io.open(AJSON, "w")
f:write(string.format('{"armed":%d,"fetched":%d,"bytes":%d,'
..'"starve":%d,"seam":%d,"serv":%d,"late":%d,"acc":%d,'
..'"csr":%d,"cer":%d,"held":%s,"fps":%d,"shown":%d,'
..'"seekn":%d,"seekb":%d,"pos":%d,"seekf":%d,"iters":%d,'
..'"arst":%s}\n',
armed, fet, byt, dry, seam, srv, late, acc,
SP:read_u32(PG_ACSR), SP:read_u32(PG_ACER),
HELD and "true" or "false", FPS, shown,
SP:read_u32(PG_ASKN), SP:read_u32(PG_ASKB),
SP:read_u32(PG_APOS), SEEKF, ITERS,
ARST and "true" or "false"))
f:close()
P("audio counters -> "..AJSON)
end
end
local open = 0
for _,v in ipairs(r20seen) do open = open + v end
P(string.format("WRITE WINDOW OPEN on %d of %d host frames (%.1f%%) -- "
.."buffer mode blanks the graphics layer, so that is the "
.."share of the scene the display spent DARK under MAME",
open, #r20seen, #r20seen > 0 and open*100/#r20seen or 0))
if #arrive >= 2 then
local dts, first, last = {}, arrive[1].t, arrive[#arrive].t
for i = 2, #arrive do dts[#dts+1] = arrive[i].t - arrive[i-1].t end
table.sort(dts)
-- CADENCE, AND THE GRANULARITY IT IS MEASURED AT. PG_SHOWN is sampled
-- once per host frame, so a single inter-frame figure is quantised to
-- 1/56.69 s = 17.6 ms and the min/median/max below are multiples of it.
-- The MEAN over the whole run is not: the quantisation error is bounded
-- by one host frame at each END, so over n-1 intervals it is 35 ms/(n-1)
-- -- 0.30 ms a frame over 120. Read the mean; the spread is the
-- sampler's, not the player's.
local mean = (last-first)/(#arrive-1)
P(string.format("cadence: %d frames in %.3f s = %.3f fps, mean "
.."%.2f ms/frame (+/- %.2f ms, the sampler's); "
.."inter-frame min %.1f median %.1f max %.1f ms",
#arrive, last-first, (#arrive-1)/(last-first),
mean*1000, 35.3/(#arrive-1),
dts[1]*1000, dts[math.ceil(#dts/2)]*1000,
dts[#dts]*1000))
-- THE TRANSPORT'S TIME IS ONLY READABLE OFF A FREE-RUNNING RUN.
-- Paced, the mean inter-frame IS THE PACE PERIOD: the player waits for
-- its tick and the record's delivery hides inside the slot, so dividing
-- the record by that mean reports the clock rather than the transport
-- and reports it as a rate. The first cut of this script printed
-- "297.4 KB/s" off a 6 fps gate run, which is the pace and not the
-- disc. Free-running there is no gate and the loop is transfer-bound,
-- so the mean is the transfer.
if not PACED then
P(string.format(" -> FREE-RUNNING, so the mean IS the "
.."transport: a %d B record lands in %.2f ms, i.e. "
.."%.1f KB/s and %.1f%% of a %d fps slot. MAME's "
.."device models carry no transfer timing "
.."(docs/BENCHMARK.md, 42.5), so this is a property "
.."of the APPARATUS -- it is not W and it is not a "
.."measurement of any medium.",
META.rec_bytes, mean*1000,
META.rec_bytes/mean/1024,
mean*META.fps*100, META.fps))
else
-- What a paced run CAN say about the transfer, and it says it from
-- the display rather than from the clock: the window is open for
-- exactly as long as the record takes, so the open fraction times the
-- slot is the transfer time, sampled at the host's frame rate.
P(string.format(" -> PACED, so this mean is the PACE and "
.."NOT the transport. What the run does bound is the "
.."transfer: the window was open %.1f%% of a "
.."%.2f ms slot = %.1f ms a record.",
open*100/#r20seen, mean*1000,
open/#r20seen*mean*1000))
end
end
if CSV then
local f = io.open(CSV, "w")
f:write("frame,t_s\n")
for _,a in ipairs(arrive) do f:write(string.format("%d,%.6f\n", a.n-1, a.t)) end
f:close()
P("arrivals -> "..CSV)
end
do
local f = assert(io.open("packed_snaps.csv", "w"))
f:write("snapshot,frame\n")
for i, fr in ipairs(snapfr) do
f:write(string.format("%04d,%d\n", i-1, fr))
end
f:close()
P(string.format("%d snapshots -> tmp/packed_snaps.csv", #snapfr))
end
P("done")
M:exit()
end
end)
if not ok then print("[PK] LUA ERROR: "..tostring(err)); M:exit() end
end)
tools/bench/packed_run.sh
Executable
+290
View File
@@ -0,0 +1,290 @@
#!/bin/bash
# THE PACKED PLAYER, END TO END, OFF A REAL VOLUME. ROADMAP K3.
#
# tools/bench/packed_run.sh [container.dlxp]
#
# Four runs of src/player/packed.s, and each answers a different question. They
# are separate runs because the questions interfere: the write window has to be
# OPEN for the whole transfer and buffer mode blanks the graphics layer, so at
# the container's own 12 fps there is no instant at which a complete frame is
# displayable and the pixel gate has nothing to sample. Pacing slower opens a
# display interval without changing one byte of the transfer. Reporting the
# rate off the gate run instead would have been the flattering shortcut, and it
# reports the PACE rather than the disc -- the first cut of packed.lua did
# exactly that and printed 297 KB/s off a 6 fps run.
#
# 1. GATE, stealing, paced at half rate: 120 records, 120 snapshots, every one
# compared. A packed frame is a LITERAL, so unlike the codec's gate the last
# frame audits nothing and all 120 have to be checked (verify_packed.py).
# 2. RATE, stealing, FREE-RUNNING: the loop is transfer-bound, so the mean
# inter-frame IS the emulated transport's time for a record.
# 3. RATE, held, FREE-RUNNING: the same, with the bus held.
# 4. CLOCK, held, paced at the container's fps: what holding the bus does to a
# frame clock built on counting V-DISP interrupts.
#
# DLX_PK_GATE_ONLY=1 runs 1 alone. That is what tools/bench/check.sh takes: the
# green light's job is to catch a regression in the PLAYER, and runs 2-4 measure
# the apparatus rather than gate it -- three more MAME jobs for numbers that
# cannot change unless MAME does.
#
# THE APPARATUS is tools/bench/dma_run.sh's -- `x68000 -exp1 cz6bs1` and a
# zero-filled scsiexrom.bin on a private rompath -- and the volume is
# tools/bench/mkvol.sh's, which for a DLXP container is the container itself.
#
# WHAT NO RUN HERE MEASURES: `W`, and any rate a real medium would deliver.
# MAME's device models carry no transfer timing (docs/BENCHMARK.md, 42.5). What
# is measured is the SHAPE -- one channel start, 193 destinations, 120 times,
# on a clock the machine keeps itself, with every frame pixel-exact.
set -e
cd "$(dirname "$0")/../.."
DLXP=${1:-tmp/packed_singe.dlxp}
NFR=${DLX_PK_NFR:-120}
GATE_FPS=${DLX_PK_GATE_FPS:-6}
bash tools/bench/mkvol.sh "$DLXP"
python3 tools/bench/prep_packed.py "$DLXP"
# WHICH ORDER THIS CONTAINER USES, read out of the container rather than
# assumed. FINDINGS 62.5/63.4 priced palette-first and palette-last at -12.8 dB
# for one paint apiece and could not choose between them, so the format records
# it (dlxp.py flags bit 1) and BOTH have to pass this gate. The chain assertion
# below is the only place the difference is visible from outside the machine,
# and hard-coding either order there would turn "K3 ran both" into "K3 ran one
# and the other could not have failed".
PALLAST=$(sed -n 's/.*palette_last = \([01]\),.*/\1/p' tmp/packed_meta.lua)
if [ "$PALLAST" = "1" ]; then
CHAIN0='chain\[0\] = MAR \$C08000 MTC 256'
ORDER="palette LAST -- the 193rd entry"
else
CHAIN0='chain\[0\] = MAR \$E82000 MTC 512 ; chain\[1\] = MAR \$C08000 MTC 256'
ORDER="palette FIRST -- entry 0, then 192 rows"
fi
echo " container order: $ORDER"
tools/vasm/vasmm68k_mot -Fbin -o tmp/packed.bin src/player/packed.s > /dev/null
# One run. $1 names the log, the rest are environment.
run() {
local tag=$1; shift
rm -rf "tmp/snap_packed_$tag"; mkdir -p "tmp/snap_packed_$tag"
# stdbuf -oL: without it a long MAME run is unobservable until it exits, and a
# run that is merely finishing looks exactly like one that is wedged (34.1).
( cd tmp && env SDL_VIDEODRIVER=dummy "$@" stdbuf -oL timeout -k 5 900 \
mame x68000 -bios ipl10 -exp1 cz6bs1 \
-rompath "$HOME/mame/roms;./p4roms" -hard dlxpdisk.chd \
-ramsize 2M -video soft -window $SOUNDARGS -nothrottle -plugins \
-autoboot_script ../tools/bench/packed.lua \
-snapshot_directory "./snap_packed_$tag" -snapview native \
-seconds_to_run "$SECS" > "packed_$tag.log" 2>&1 )
grep -aq "^\[PK\] done" "tmp/packed_$tag.log" || {
echo "FAIL: the $tag run did not finish -- no completion marker."
tail -12 "tmp/packed_$tag.log"; exit 1; }
grep -a "^\[PK\]" "tmp/packed_$tag.log" | sed 's/^\[PK\] / /'
}
fail() { echo "FAIL: $1"; exit 1; }
# -sound none for every run that is not about sound, which is all of them until
# run 5. 15,625 is not a preference there: it is the chip's own stream rate
# (8 MHz / 512), and equal rates are what keep MAME's resampler from filtering
# the thing being measured (FINDINGS 66, adpcm_run.sh).
SOUNDARGS="-sound none"
echo "--- 1. THE GATE: $NFR records, paced at $GATE_FPS fps, channel stealing ---"
SECS=$(( NFR / GATE_FPS + 25 ))
run gate DLX_PK_HELD=0 DLX_PK_PACE=1 DLX_PK_FPS=$GATE_FPS DLX_PK_NFR=$NFR
cp tmp/packed_snaps.csv tmp/packed_snaps_gate.csv
# THE ASSERTIONS. Printing a result and gating on it are different things.
grep -aq "^\[PK\] FLAG=\$FF" tmp/packed_gate.log || \
fail "the player did not reach the end of the scene. FLAG=\$E1 is a CRTC mode
the frame clock cannot divide, \$E2 is a transport failure -- and the
TRANSPORT FAILED line above names which."
grep -aq "array: the 68000 built 193 entries (the container wants 193)" \
tmp/packed_gate.log || \
fail "the 68000 built a chain of the wrong length. One entry short delivers a
picture with its last row missing, which looks like a decode bug and is a
layout bug; the container's geometry and the player's arithmetic are two
independent statements of one number and they have to agree."
grep -aq "$CHAIN0" tmp/packed_gate.log || \
fail "the chain does not have the shape this container asks for ($ORDER).
The crossing from the palette registers into GVRAM IS the packed frame
(FINDINGS 62) -- a palette entry and 192 row entries, one start, the CPU
halted throughout -- and an array built the other way round from the
record feeding it does not fail: it paints 192 rows of picture into the
palette registers and 512 B of palette across the top of the screen."
grep -aq "late frames (tick already past at the gate): 0$" tmp/packed_gate.log || \
fail "a frame missed its slot in the GATE run, which is paced at half rate on
purpose. That is not a rate result -- it means the transfer did not fit in
a slot twice as long as the container's, and the pixel comparison below is
then sampling frames the player was still overwriting."
grep -aq "0 frames NOT SAMPLED" tmp/packed_gate.log || \
fail "the gate run could not sample every frame: the write window reopened
before the snapshot on at least one. At half the container's rate the
transfer must fit inside the display interval with room to spare, and if
it does not the comparison below is checking frames the player was still
overwriting."
python3 tools/bench/verify_packed.py "$DLXP" --snap tmp/snap_packed_gate \
--map tmp/packed_snaps_gate.csv --min-frames "$NFR"
# ---------------------------------------------------------------------------
# 5. THE AUDIO. ROADMAP P6c: the container's own bytes, out of channel 3,
# beside the video channel. Paced at the CONTAINER's rate rather than the
# gate's half rate, because the audio was cut at 12 fps and a 6 fps run would
# starve the chip for half of every group -- the picture can be slowed down and
# a crystal cannot. Nothing is snapshotted; the instrument is the WAV, and
# tools/bench/verify_packed_audio.py accounts for every byte of the stream in
# it. The tag is `aud` and not `audio` because tmp/packed_audio.log is
# tools/analysis/34_packed_audio.py's, in check.sh.
#
# CYCLE STEALING, and that is a result rather than a setting -- run 6 below is
# the same run with the bus held and it is a CONTRAST, not a gate.
AUDIO_ON=$(sed -n 's/.*has_audio = \([01]\),.*/\1/p' tmp/packed_meta.lua)
CFPS=$(sed -n 's/^ fps = \([0-9]*\),.*/\1/p' tmp/packed_meta.lua)
if [ "$AUDIO_ON" = "1" ]; then
echo
echo "--- 5. THE AUDIO: the container's own lumps, out of channel 3, while"
echo " the video channel is on the same bus (ROADMAP P6c) ---"
SECS=$(( NFR / 8 + 30 ))
SOUNDARGS="-samplerate 15625 -wavwrite packed_aud.wav"
run aud DLX_PK_HELD=0 DLX_PK_PACE=1 DLX_PK_FPS="$CFPS" DLX_PK_NFR=$NFR \
DLX_PK_AUD=1 DLX_PK_AJSON=packed_aud.json
SOUNDARGS="-sound none"
grep -aq "^\[PK\] FLAG=\$FF" tmp/packed_aud.log || \
fail "the audio run did not reach the end of the scene."
python3 tools/bench/verify_packed_audio.py "$DLXP" tmp/packed_aud.wav \
tmp/packed_aud.json || \
fail "the chip did not play the container. The counters above can all be
right while this fails -- nothing parses a packed container, so a lump
fetched into a buffer that is still being read is not an error, it is a
sound (FINDINGS 67.4, and it is the bug session 36 shipped and caught)."
fi
# ---------------------------------------------------------------------------
# 7. THE SEEK, WITH SOUND ON IT. FINDINGS 70.3 asked for this and named what
# was missing: "src/player/packed.s starts PG_AK/PG_AKF at lump 0 and has no
# audio seek path at all". This is that path, run.
#
# THE FRAME IS CHOSEN NOT TO BE A MULTIPLE OF THE CADENCE, and that is the whole
# design of the run. A DLXP2 group is `lump k, then F records`, so a branch
# that lands on a group boundary needs no offset and no second read -- and 36 of
# the arcade's 409 within-container seek targets do land on one. A run that
# picked one of those would exercise the arithmetic that was already there and
# report success. DLX_PK_SEEK defaults below to a frame `f mod F != 0`, so the
# byte offset inside the lump is load-bearing: get it wrong and the chip is fed
# a stream that starts up to F frames early, which is not an error, it is a
# rate, and only verify_packed_audio.py's spliced walk can see it.
#
# AND THE SECOND RESULT IS ONE NO COUNTER CAN REACH: the chip's predictor does
# not seek. Every byte can arrive, in order, exactly -- and the samples still
# be wrong, because the encoder chose them for a state a continuous play would
# have been in. The verifier measures that against its own control.
if [ "$AUDIO_ON" = "1" ] && [ "${DLX_PK_NOSEEK:-0}" != "1" ]; then
SEEKF=${DLX_PK_SEEK:-37}
CADF=$(sed -n 's/^ cad_f = \([0-9]*\),.*/\1/p' tmp/packed_meta.lua)
[ $((SEEKF % CADF)) -ne 0 ] || \
fail "the seek frame $SEEKF is a multiple of the cadence $CADF, so it lands
ON a group boundary -- the one case that needs no byte offset and would
pass with the offset arithmetic deleted (FINDINGS 70.3)."
echo
echo "--- 7. THE SEEK: two passes, the second starting at frame $SEEKF --"
echo " lump $((SEEKF / CADF)), $((SEEKF % CADF)) frame(s) into its group"
echo " of $CADF (FINDINGS 70.3/71) ---"
TOTFR=$(( NFR + NFR - SEEKF ))
SECS=$(( TOTFR / 8 + 35 ))
# BOTH CONFIGURATIONS, and they are not two speeds of one thing. The chip's
# accumulator is an integrator with no leak, so what a branch costs is set by
# the state it lands in: play THROUGH and the chip keeps whatever the previous
# scene left it in; STOP and re-PLAY and it goes to the container's own `init`
# with the step index at 0. Neither is zero and they are 5.5x apart, so the
# run measures both and FINDINGS 71.3 chooses.
for M in 0 1; do
TAG=seek; [ "$M" = 1 ] && TAG=seek_rst
echo " -- the chip $([ "$M" = 1 ] && echo 'STOPPED and re-PLAYED' \
|| echo 'PLAYING THROUGH') the branch"
SOUNDARGS="-samplerate 15625 -wavwrite packed_$TAG.wav"
run $TAG DLX_PK_HELD=0 DLX_PK_PACE=1 DLX_PK_FPS="$CFPS" DLX_PK_NFR=$NFR \
DLX_PK_AUD=1 DLX_PK_AJSON=packed_$TAG.json \
DLX_PK_ITER=2 DLX_PK_SEEK=$SEEKF DLX_PK_ARST=$M
SOUNDARGS="-sound none"
grep -aq "^\[PK\] FLAG=\$FF" "tmp/packed_$TAG.log" || \
fail "the $TAG run did not reach the end of the second pass."
python3 tools/bench/verify_packed_audio.py "$DLXP" "tmp/packed_$TAG.wav" \
"tmp/packed_$TAG.json" --seek "$SEEKF" --iters 2 || \
fail "the player did not play the container ACROSS A BRANCH ($TAG).
Nothing here parses anything (FINDINGS 67.4): a seek that fetched the
wrong lump plays 7,168 B of the wrong part of the scene, and one that
dropped the byte offset plays the right lump from up to $((CADF-1))
frames too early -- neither is an error and both are a sound."
done
fi
if [ "${DLX_PK_GATE_ONLY:-0}" = "1" ]; then exit 0; fi
echo
echo "--- 2/3. THE RATE: free-running, both channel configurations ---"
SECS=$(( NFR / 8 + 25 ))
run free_steal DLX_PK_HELD=0 DLX_PK_PACE=0 DLX_PK_NFR=$NFR
run free_held DLX_PK_HELD=1 DLX_PK_PACE=0 DLX_PK_NFR=$NFR
for t in free_steal free_held; do
grep -aq "FREE-RUNNING, so the mean IS the transport" "tmp/packed_$t.log" || \
fail "the $t run did not report a transport time, so there is no rate here
to read -- and a rate read off a PACED run is the pace."
done
echo
echo "--- 4. THE CLOCK: held, paced at the container's own rate ---"
SECS=$(( NFR / 12 + 25 ))
run held_paced DLX_PK_HELD=1 DLX_PK_PACE=1 DLX_PK_NFR=$NFR
# THE FINDING THIS RUN EXISTS FOR, asserted rather than admired. A held channel
# halts the 68000, and the frame clock is an INTERRUPT off V-DISP whose pending
# bit is ONE BIT -- so every edge that falls inside a transfer spanning two of
# them is an edge the machine can never count. If this ever comes back at zero,
# either the transfer got short enough to fit between two rasters or the held
# configuration stopped halting the CPU, and both change what the run means.
LOST=$(sed -n 's/.*-> \([0-9]*\) edges LOST.*/\1/p' tmp/packed_held_paced.log | head -1)
[ -n "$LOST" ] && [ "$LOST" -gt 0 ] || \
fail "the held run lost no V-DISP edges (${LOST:-none}). Either the bus is no
longer being held for the transfer, or the transfer now fits between two
rasters -- and the comparison with the stealing run below is then a
comparison of two configurations that do the same thing."
LOSTS=$(sed -n 's/.*-> \([0-9]*\) edges LOST.*/\1/p' tmp/packed_gate.log | head -1)
# ---------------------------------------------------------------------------
# 6. THE SAME AUDIO WITH THE BUS HELD, and this is the interaction ROADMAP P6c
# said neither half had met. A burst channel HALTS the 68000 for the whole
# 88 ms record, so the audio service cannot run during it -- src/player/dma.i's
# DM_HOOK is never reached and the chip is looked at twice a frame instead of
# two thousand times. The bytes are the same bytes either way; what changes is
# WHEN the next lump is armed, and the chip has no starvation state: what it
# does in between is replay the byte pair it is holding.
#
# NOT A GATE. Both configurations play the container byte for byte and the
# verifier passes on both; the difference is entirely in the seams, and a seam
# is a design cost rather than a correctness one.
if [ "$AUDIO_ON" = "1" ]; then
echo
echo "--- 6. THE AUDIO AGAIN, WITH THE BUS HELD (the contrast, not a gate) ---"
SECS=$(( NFR / 6 + 30 ))
SOUNDARGS="-samplerate 15625 -wavwrite packed_aud_held.wav"
run aud_held DLX_PK_HELD=1 DLX_PK_PACE=1 DLX_PK_FPS="$CFPS" DLX_PK_NFR=$NFR \
DLX_PK_AUD=1 DLX_PK_AJSON=packed_aud_held.json
SOUNDARGS="-sound none"
python3 tools/bench/verify_packed_audio.py "$DLXP" tmp/packed_aud_held.wav \
tmp/packed_aud_held.json || fail "the held run did not play the
container. The bytes are not what holding the bus was expected to cost."
echo
echo " THE SEAM, STEALING AGAINST HELD -- audio does not merely cost clocks:"
for t in aud aud_held; do
printf ' %-9s ' "$t"
python3 tools/bench/verify_packed_audio.py "$DLXP" "tmp/packed_$t.wav" \
| sed -n 's/^ worst \(.*\)$/\1/p' | head -1
done
echo " Stealing, the 68000 sees the channel from inside dma.i's transfer"
echo " wait. Held, it is halted for the whole record and cannot look at all."
fi
echo
echo " V-DISP edges lost: $LOST held at 12 fps, $LOSTS stealing at $GATE_FPS fps."
echo " A player keeps a clock, reads a stick and feeds ADPCM. Which of the two"
echo " configurations can do any of that is a DESIGN question, and it is the"
echo " one this run answers; neither figure is W."
exit 0
tools/bench/prep_adpcm.py
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
"""Build the nibble stream that asks the MSM6258 which decoder it is.
ROADMAP P6a.
WHAT HAS TO BE DISCRIMINATED, and it is four things rather than the one
FINDINGS 65 named:
1. DELTA FORMULA -- 'shift' (ffmpeg's adpcm_ima_oki) against 'terms' (the
datasheet's per-term truncation). Worth 25 dB (65.2).
2. NIBBLE ORDER -- which half of a byte handed to the data register is played
FIRST. 65.1 measured 'high' AGAINST FFMPEG, which is a fact about the VOX
file convention and not about a chip's data register.
3. THE CLAMP -- the accumulator saturates somewhere, and where is inside
the recursion, so it is not an output scaling that can be undone.
4. THE INITIAL ACCUMULATOR at the instant of PLAY.
The stream is in three parts and each part exists for a reason:
PROLOGUE, 16 zero nibbles. Nibble 0 moves the step index DOWN, so it stays
pinned at 0 and the delta is a constant +2 under every candidate. That makes
the prologue a RAMP that both formulas agree on, which is what absorbs the one
thing this rig cannot control: how many times the chip consumes byte 0 before
the channel delivers byte 1. The verifier reads that count off the capture
instead of assuming it.
SEGMENT A, a quiet sine, encoded by tools/encoder/adpcm.py itself. Amplitude
300 keeps it clear of even the 10-bit clamp, so A discriminates the FORMULA
and the ORDER without the clamp confounding either. Using the shipping
encoder rather than a hand-written pattern is deliberate: the nibbles the chip
is asked about are the kind of nibbles it will be sent.
SEGMENT B, loud bursts. It exists ONLY to cross the 10-bit clamp, which
segment A is built never to reach, and it is last because a clamp is
irreversible state and everything after it would be measuring segment B.
"""
import json, math, os, sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "encoder"))
import adpcm
PRO_NIB = 16 # prologue nibbles (byte 0 = $00, so a repeat costs nothing)
A_SAMPLES = 1500 # segment A, one nibble each
A_AMP = 300 # clear of the 10-bit clamp at 511 with room for the ramp
A_HZ = 61.0 # ~256 samples a cycle at 15,625 Hz: many step indices
RATE = 15625.0
BUF = 0x30000 # where the harness pushes the bytes
OUT_BIN = "tmp/adpcm_data.bin"
OUT_META = "tmp/adpcm_meta.lua"
OUT_SEQ = "tmp/adpcm_seq.json"
# THE TRIGGER, and it is here because the first cut of this file did not have
# one and measured ONE differing sample in 1,676. The two formulas are
# IDENTICAL whenever the step value is a multiple of 8:
#
# terms - shift = b2*floor(r/2) + b3*floor(r/4) - floor((4*b2+2*b3+1)*r/8)
#
# with r = step mod 8 and (b1,b2,b3) the nibble's low three bits. It is zero
# for r = 0, and the step table STARTS at 16. A quiet signal never moves the
# step index off its floor, so a probe made of quiet nibbles asks the chip a
# question that has the same answer either way.
#
# nibble 4 at step 16: delta 18 under both, and it moves the index to 2
# nibble 3 at step 19: shift 16, terms 15 <- the two states part company
#
# After that they never rejoin, because the delta is added to a running
# predictor -- so ONE two-nibble trigger converts the rest of the stream into
# discriminating evidence. That is the same recursion 65.2 priced at 25 dB,
# used deliberately instead of suffered.
TRIGGER = [4, 3]
def segment_a():
"""The trigger, then a sine encoded by the shipping encoder. The model the
sine is encoded under does not matter for discrimination -- once the trigger
has parted the two states, any nibble stream keeps them apart -- so the
defaults are used and the choice is recorded rather than tuned. Using the
shipping encoder rather than a hand-written pattern is the point: the
nibbles the chip is asked about are the kind of nibbles it will be sent."""
sig = [int(round(A_AMP * math.sin(2 * math.pi * A_HZ * i / RATE)))
for i in range(A_SAMPLES)]
return TRIGGER + list(adpcm.encode(sig, "shift"))
def segment_b():
"""Loud, and alternating in sign so the step index does not simply pin: 40
up, 40 down, twice. Under a 10-bit accumulator this saturates; under a
12-bit one it does not, and that difference is the whole point of it."""
return ([7] * 40 + [15] * 40) * 2
def main():
core = segment_a() + segment_b()
nibs = [0] * PRO_NIB + core
data = adpcm.pack(nibs, "high") # HIGH first: the encoder's convention,
# which is one of the things on trial
os.makedirs("tmp", exist_ok=True)
open(OUT_BIN, "wb").write(data)
# HOW MUCH DISCRIMINATING POWER IS IN IT, counted rather than asserted. A
# probe that cannot separate two candidates reports a match against both and
# a gate that did not count this would call that a result.
ref = adpcm.decode(nibs, "shift", init=-2, bits=10)
axes = {}
for name, kw in (("formula", dict(variant="terms")),
("order", dict(order="low")),
("clamp", dict(bits=12)),
("init", dict(init=0))):
order = kw.pop("order", "high")
n2 = ([0] * PRO_NIB
+ list(adpcm.unpack(data, len(nibs), order))[PRO_NIB:]) \
if order != "high" else nibs
n2 = list(adpcm.unpack(data, len(nibs), order))
alt = adpcm.decode(n2, kw.get("variant", "shift"),
init=kw.get("init", -2), bits=kw.get("bits", 10))
d = sum(1 for a, b in zip(ref, alt) if a != b)
axes[name] = d
seq = {"nibbles": nibs, "core": core, "pro": PRO_NIB,
"bytes": len(data), "buf": BUF, "axes": axes,
"a_samples": A_SAMPLES, "a_amp": A_AMP, "a_hz": A_HZ}
json.dump(seq, open(OUT_SEQ, "w"))
with open(OUT_META, "w") as f:
f.write("return {\n")
f.write(f" buf = 0x{BUF:X},\n")
f.write(f" nbytes = {len(data)},\n")
f.write(f" nnibs = {len(nibs)},\n")
f.write("}\n")
print(f" probe stream: {len(nibs)} nibbles = {len(data)} B "
f"= {len(nibs)/RATE*1000:.1f} ms at 15,625 Hz")
print(f" prologue {PRO_NIB} zero nibbles, segment A {len(segment_a())} "
f"(sine {A_AMP} @ {A_HZ} Hz), segment B {len(segment_b())} (loud)")
print(" DISCRIMINATING POWER -- samples that change when ONE axis is "
"flipped away from MAME's own model:")
for k, v in axes.items():
print(f" {k:8s} {v:5d} of {len(ref)}")
if __name__ == "__main__":
main()
tools/bench/prep_load.py
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Lay out the LOAD-TIME test: raw container header in, expected results out.
python3 tools/bench/prep_load.py <in.dlx> [--out tmp/load]
src/player/load.i does on the 68000 what tools/bench/dlxload.py has been doing
host-side since session 1: expand the two codebooks to word-per-pixel form and
pack the 24-bit palette into GGGGGRRRRRBBBBBI with the shared LSB chosen per
entry (ROADMAP P1 and P2). This writes both halves of that comparison.
<out>_data.bin the container's HEADER REGION, byte for byte as it comes
off the disc: magic, geometry, the three section offsets,
the 768-byte palette, CB1 and CB4. Nothing is pre-chewed --
that is the entire point. It ends where the frame stream
begins, so it is also exactly what a player would have to
read before it could draw anything.
<out>_expect.bin what dlxload.py says the 68000 must produce: expanded CB1,
expanded CB4, then 256 big-endian palette words.
<out>_meta.lua sizes, k1/k4, and the expected darkest-entry index.
The expectation is generated by the SAME module the two decode rigs load
through, so this cannot pass by agreeing with a second copy of the maths.
"""
import sys, argparse
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/bench")
from dlx import DLX
import dlxload as DL
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--out", default="tmp/load")
a = ap.parse_args()
d = DLX(a.container)
if d.version < 3:
sys.exit(f"{a.container} is DLX{d.version}: load.i wants DLX3 or DLX4")
if d.idx_bytes != 1:
sys.exit("2-byte codebook indices: load.i expands one source byte per pixel")
off_frm = int.from_bytes(d.raw[28:32], "big")
raw = d.raw[:off_frm]
cb1, cb4 = DL.expand_codebooks(d)
palb, dark, _ = DL.pack_palette(d)
open(a.out + "_data.bin", "wb").write(raw)
open(a.out + "_expect.bin", "wb").write(cb1.tobytes() + cb4.tobytes() + palb.tobytes())
with open(a.out + "_meta.lua", "w") as fh:
fh.write("-- generated by tools/bench/prep_load.py -- do not edit\nreturn {\n")
fh.write(f" k1={d.k1}, k4={d.k4}, dark={dark},\n")
fh.write(f" raw_len={len(raw)}, cb1_len={cb1.nbytes}, cb4_len={cb4.nbytes},\n")
fh.write(f" pal_len={palb.nbytes},\n}}\n")
print(f"{a.container}: k1={d.k1} k4={d.k4}, header region {len(raw)} B "
f"(pal 768 + cb1 {d.k1*16} + cb4 {d.k4*4} + 32)")
print(f" the 68000 must produce {cb1.nbytes} + {cb4.nbytes} B of expanded "
f"codebook and {palb.nbytes} B of palette, darkest entry {dark}")
tools/bench/prep_packed.py
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""Everything tools/bench/packed.lua needs to know about a DLXP container.
python3 tools/bench/prep_packed.py <in.dlxp> -> tmp/packed_meta.lua
THERE IS NO BLOB TO PREPARE, and that is the whole difference from
`prep_stream.py`. The codec's rig has to hand the machine expanded codebooks, a
packed palette and a record index, because a DLX record cannot be found or drawn
without them; `prep_dlx.py` and `prep_stream.py` exist for that and FINDINGS
49.7.5 records what it cost to have two copies of one of those transforms. A
packed container carries no such thing: record `i` is at sector 1 + i*97 by
geometry and its bytes are already in the order GVRAM wants them (dlxp.py). So
this file emits METADATA ONLY -- six numbers the rig would otherwise have to
hard-code, every one of them read out of the container's own header.
The volume is the container itself; tools/bench/mkvol.sh copies it.
"""
import os, sys
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)),
"..", "encoder"))
from dlxp import DLXP, SECTOR
if len(sys.argv) != 2:
sys.exit(__doc__)
d = DLXP(sys.argv[1])
# The record's sector count, and the array chain's entry count, DERIVED here and
# asserted by the 68000 (PG_ARRN). Two independent statements of one geometry
# is the only way a container and a player can be caught disagreeing about it --
# a chain one entry short delivers a picture with its last row missing, which
# looks like a decode bug and is a layout bug.
recs = d.rec_bytes // SECTOR
rows = d.H
entries = rows + (1 if d.has_palette else 0)
out = "tmp/packed_meta.lua"
with open(out, "w") as fh:
fh.write("-- generated by tools/bench/prep_packed.py; do not edit\n")
fh.write("return {\n")
for k, v in [("W", d.W), ("H", d.H), ("fps", d.fps), ("nframes", d.nframes),
("rec_bytes", d.rec_bytes), ("rec_sectors", recs),
("pal_bytes", d.pal_bytes), ("pic_bytes", d.pic_bytes),
("lba0", d.off_frm // SECTOR),
("palette_last", int(d.palette_last)),
("has_palette", int(d.has_palette)),
# DLXP2. Zero in a silent container, and the player branches on
# the zero rather than being built two ways.
("cad_f", d.cad_f), ("cad_a", d.cad_a),
("has_audio", int(d.has_audio)),
("aud_bytes", d.aud_bytes), ("aud_hz", d.aud_hz),
("n_lumps", d.n_lumps),
# ROADMAP P6c. lba_aud is off_aud/512 and is a HEADER field
# rather than the constant 1 it happens to equal: a shipping
# volume has a filesystem in front of the stream and the
# player adds PG_LBA0's base to neither of them by accident.
("lba_aud", d.off_aud // SECTOR if d.has_audio else 0),
# the four axes, so the rig can print what the container says
# it was encoded for and the verifier can decode with it
("aud_variant", '"%s"' % d.decoder()["variant"]
if d.has_audio else '""'),
("aud_order", '"%s"' % d.decoder()["order"]
if d.has_audio else '""'),
("aud_bits", d.aud_bits), ("aud_init", d.aud_init),
("entries", entries)]:
fh.write(f" {k} = {v},\n")
fh.write("}\n")
print(f"{sys.argv[1]}: DLXP{d.version} {d.W}x{d.H} {d.fps}fps {d.nframes} frames"
+ (f", AUDIO F={d.cad_f} A={d.cad_a}" if d.has_audio else ", silent"))
print(f" record {d.rec_bytes:,} B = {recs} sectors, palette "
f"{'LAST' if d.palette_last else 'FIRST'}, {d.pal_bytes} B")
cad = (f" + (i//{d.cad_f})*{d.cad_a}" if d.has_audio else "")
print(f" record i is at LBA {d.off_frm // SECTOR} + i*{recs}{cad} -- ARITHMETIC. "
f"There is no index in this container and none can be needed.")
print(f" the chain the 68000 must build: {entries} entries "
f"({rows} rows{' + 1 palette' if d.has_palette else ''})")
print(f" wire {d.video_kbps():.1f}"
+ (f" + {d.audio_kbps():.2f} = {d.kbps():.1f}" if d.has_audio else "")
+ f" KB/s, FIXED by geometry -> {out}")
tools/bench/prep_stream.py
+33 -1
View File
@@ -18,6 +18,10 @@ This writes three files instead:
filesystem and feeds it into a bounded ring, so the emulated
machine's RAM stops bounding how much of a window can be
tested. A stock 2 MB machine can now run all 120 frames.
<out>_idx.bin the DLX4 RECORD INDEX, nframes u16 big-endian, straight out
of the container's scene header. This is what the 68000
producer reads (src/player/ring.i); it is not derived here,
because deriving it is precisely what a player cannot do.
<out>_meta.lua geometry, and the record index.
THE RECORD INDEX IS NOT A CONVENIENCE. src/player/stream.s takes each frame's
@@ -66,17 +70,42 @@ disk, index = bytearray(), []
for (o, n) in d.frames:
start = len(disk)
disk += n.to_bytes(4, "big") + d.raw[o:o + n]
while len(disk) % 4:
# The container's own alignment rule, not this script's copy of it: DLX5
# pads to 512 so a DMA channel can read whole sectors into the ring, DLX4
# to 4 so `move.l (a0)+` does not take an address error (28.3).
while len(disk) % d.rec_align:
disk += b"\0"
index.append((start, len(disk) - start))
open(a.out + "_disk.bin", "wb").write(bytes(disk))
# THE CONTAINER'S OWN INDEX, and it is checked against this layout rather than
# regenerated from it. src/player/ring.i walks the disk with a running sum of
# these lengths and never reads a record's length word before fetching it, so a
# container index that disagreed with the disk image by one byte would place
# every later record at the wrong address -- and the block loop reads without a
# bounds check (49.2), so the symptom would be wrong pixels, not a fault.
if d.has_index:
want = [ln // 4 for _, ln in index]
if d.index != want:
bad = next(i for i in range(len(want)) if d.index[i] != want[i])
sys.exit(f"{a.container}: the DLX4 index disagrees with this disk "
f"layout at record {bad}: {d.index[bad]} vs {want[bad]} "
f"longwords")
open(a.out + "_idx.bin", "wb").write(
b"".join(q.to_bytes(2, "big") for q in d.index))
else:
if os.path.exists(a.out + "_idx.bin"):
os.remove(a.out + "_idx.bin") # a stale index is worse than none
rec = np.array([n for _, n in index])
with open(a.out + "_meta.lua", "w") as fh:
fh.write("-- generated by tools/bench/prep_stream.py -- do not edit\nreturn {\n")
fh.write(f" W={d.W}, H={d.H}, fps={d.fps}, nframes={d.nframes}, dark={dark},\n")
fh.write(f" cb1_len={cb1.nbytes}, cb4_len={cb4.nbytes}, pal_len={palb.nbytes},\n")
fh.write(f" disk_len={len(disk)}, maxrec={int(rec.max())},\n")
fh.write(f" dlx_version={d.version}, "
f"has_index={'true' if d.has_index else 'false'},\n")
fh.write(f" padrec={{{','.join(str(ln) for _, ln in index)}}},\n")
fh.write(" index={\n")
for off, ln in index:
fh.write(f" {{off={off}, len={ln}}},\n")
@@ -89,5 +118,8 @@ print(f" disk image {len(disk):,} B -> {a.out}_disk.bin "
f"(records: min {rec.min():,} median {int(np.median(rec)):,} "
f"max {rec.max():,})")
print(f" wire rate {rec.mean()*d.fps/1024:.1f} KB/s video at {d.fps} fps")
if d.has_index:
print(f" DLX4 record index: {2*d.nframes:,} B of scene header, and it "
f"agrees with the disk image on all {d.nframes} records")
print(f" A ring must hold one whole record contiguously: >= {rec.max():,} B "
f"({rec.max()/1024:.1f} KB) before any policy or prefill.")
tools/bench/probe_adpcm.lua
+28
View File
@@ -0,0 +1,28 @@
-- Ask MAME what the x68000's ADPCM device is, and where the 68000 reaches it.
-- FINDINGS 64.4 recorded that no MAME source tree is on this machine; this is
-- the way to ask the same question without one.
M = manager.machine
local done = false
SUB = emu.add_machine_frame_notifier(function()
if done then return end
done = true
print("[AD] === devices whose tag looks like an ADPCM chip ===")
for tag, dev in pairs(M.devices) do
local t = tag:lower()
if t:find("adpcm") or t:find("oki") or t:find("msm") or t:find("6258") then
print(string.format("[AD] DEV %-24s shortname=%s", tag, tostring(dev.shortname)))
end
end
local sp = M.devices[":maincpu"].spaces["program"]
print("[AD] === program map, $E90000..$EA0000 ===")
local ok, err = pcall(function()
for _, e in ipairs(sp.map.entries) do
if e.address_start >= 0xE90000 and e.address_start < 0xEA0000 then
print(string.format("[AD] MAP %08X-%08X", e.address_start, e.address_end))
end
end
end)
if not ok then print("[AD] MAP unavailable: " .. tostring(err)) end
print("[AD] done")
M:exit()
end)
tools/bench/probe_adpcm2.lua
+36
View File
@@ -0,0 +1,36 @@
-- Feed the x68000's OWN okim6258 a known nibble stream and let MAME record what
-- comes out, so that "which delta formula does the chip use" is a MEASUREMENT
-- and not a reading of source code that is not on this machine (64.4).
--
-- The feed is deliberately SLOW -- a byte every host frame, where real time
-- wants ~138 -- because the question is not the rate. A starved chip holds its
-- last sample, so the wave is a STAIRCASE of the reconstructed values, which is
-- exactly the sequence the two candidate formulas disagree about.
M = manager.machine
local sp = M.devices[":maincpu"].spaces["program"]
local CTRL, DATA = 0xE92001, 0xE92003
local CMD = tonumber(os.getenv("AD_CMD") or "1")
-- 12 loud nibbles to climb the step index, then every nibble in turn: the pairs
-- where the two formulas differ are all at step indices above the floor.
local nibs = {}
for i = 1, 12 do nibs[#nibs+1] = 7 end
for i = 0, 15 do nibs[#nibs+1] = i end
for i = 0, 15 do nibs[#nibs+1] = i end
local bytes = {}
for i = 1, #nibs, 2 do bytes[#bytes+1] = nibs[i] * 16 + nibs[i+1] end
local n, started = 0, false
SUB = emu.add_machine_frame_notifier(function()
n = n + 1
if n == 30 then
sp:write_u8(CTRL, CMD)
started = true
print(string.format("[AD] ctrl $%02X written to $%06X", CMD, CTRL))
elseif started and n > 30 and (n - 30) <= #bytes then
sp:write_u8(DATA, bytes[n - 30])
elseif started and (n - 30) == #bytes + 20 then
print("[AD] fed " .. #bytes .. " bytes = " .. #nibs .. " nibbles")
print("[AD] done")
M:exit()
end
end)
tools/bench/probe_adpcm3.lua
+24
View File
@@ -0,0 +1,24 @@
-- Sweep the PPI's port C -- which is where the X68000 puts ADPCM pan and the
-- chip's clock divider -- and feed a loud burst under each value, so that the
-- WAV says which value un-mutes the chip. Nothing here is assumed: the segment
-- boundaries are printed and the analysis reads the wave against them.
M = manager.machine
local sp = M.devices[":maincpu"].spaces["program"]
local CTRL, DATA, PPIC = 0xE92001, 0xE92003, 0xE9A005
local SEG = 40 -- host frames per segment
local n = 0
SUB = emu.add_machine_frame_notifier(function()
n = n + 1
if n <= 20 then return end
local k = n - 20
local seg = math.floor((k - 1) / SEG)
local off = (k - 1) % SEG
if seg > 15 then print("[AD] done"); M:exit(); return end
if off == 0 then
sp:write_u8(PPIC, seg)
sp:write_u8(CTRL, 1)
print(string.format("[AD] seg %2d portC=$%02X starts at host frame %d", seg, seg, n))
elseif off <= 30 then
sp:write_u8(DATA, 0x77) -- two loud positive nibbles
end
end)
tools/bench/probe_adpcm4.lua
+33
View File
@@ -0,0 +1,33 @@
-- SPIKE (session 34). Not a gate: it exists to find out whether the chip can be
-- driven at all, what scale its samples arrive at in a -wavwrite capture, and
-- where it clamps. It sets EVERY thing session 33's probes left to the IPL:
-- * YM2151 reg $1B bit1 = 0 -> CT1 = 0 -> ADPCM master clock 8 MHz
-- * PPI control $92 -> port C is an OUTPUT (without this the i8255's
-- out_pc_callback never fires and the pan and
-- divider writes go nowhere)
-- * PPI port C $08 -> pan 00 = BOTH, rate 10 = /512 -> 15,625 Hz
-- * ctrl $02 = COMMAND_PLAY -- session 33's probes wrote $01, COMMAND_STOP.
M = manager.machine
local sp = M.devices[":maincpu"].spaces["program"]
local YMA, YMD = 0xE90001, 0xE90003
local PPIC, PPICTL = 0xE9A005, 0xE9A007
local CTRL, DATA = 0xE92001, 0xE92003
local BYTE = tonumber(os.getenv("AD_BYTE") or "0x77")
local n = 0
SUB = emu.add_machine_frame_notifier(function()
n = n + 1
if n == 40 then
sp:write_u8(YMA, 0x1B); sp:write_u8(YMD, 0x00)
elseif n == 60 then
sp:write_u8(YMA, 0x1B); sp:write_u8(YMD, 0x00)
sp:write_u8(PPICTL, 0x92)
sp:write_u8(PPIC, 0x08)
print(string.format("[AD4] portC readback = $%02X", sp:read_u8(PPIC)))
sp:write_u8(CTRL, 0x02)
sp:write_u8(DATA, BYTE)
print(string.format("[AD4] PLAY, data $%02X, status = $%02X",
BYTE, sp:read_u8(CTRL)))
elseif n == 90 then
print("[AD4] done"); M:exit()
end
end)
tools/bench/scsi.lua
+170
View File
@@ -0,0 +1,170 @@
-- Drive src/player/scsigate.s: can the 68000 reach the MB89352? (ROADMAP P4)
--
-- Bus errors are EXPECTED and are data: the gate probes one address at a time
-- and survives each fault, so a hole in the decode costs an entry in the map
-- rather than the rest of the run. See src/player/scsigate.s.
--
-- THE APPARATUS, STATED UP FRONT. This runs `x68000 -exp1 cz6bs1`, which is
-- the board FINDINGS 42.5 says to benchmark and never `x68ksupr` (whose
-- internal SCSI is PIO-only in MAME -- `// TODO: duplicate DMA glue from
-- CZ-6BS1`). MAME refuses to instantiate the card without an 8 KB
-- `scsiexrom.bin`, which is not in this tree; the rig supplies a ZERO-FILLED
-- placeholder on a SEPARATE rompath so the user's own romset is untouched.
--
-- THAT PLACEHOLDER IS HONEST HERE AND WOULD NOT BE EVERYWHERE. The CZ-6BS1's
-- boot ROM exists to make the card bootable through IOCS. This player drives
-- the SPC registers directly and never executes a byte of it -- that was
-- already the plan in docs/BENCHMARK.md item 4, long before the ROM was missing
-- -- so a blank one changes nothing this rig measures. What it WOULD change is
-- anything that booted from the card or called SCSI IOCS; do not reuse it for
-- that. The run prints the substitution rather than burying it.
--
-- WHAT A GREEN RUN MEANS, and what it does not. It means the 68000 reaches the
-- SPC and the register map is the one the driver will be written against. It
-- says NOTHING about rate: MAME's device models are functional, not
-- transfer-timing accurate (docs/BENCHMARK.md), and 42.5 shows its DMAC is
-- configured in wall-clock attotimes rather than per-operand cycles. `W` is
-- untouched by anything here.
local M = manager.machine
local SP = M.devices[":maincpu"].spaces["program"]
local function P(s) print("[SCSI] "..s) end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local SCFLAG, SCN, SCVAL, SCOK, SCTMP, SCTMPOK =
0x18080, 64, 0x18100, 0x18140, 0x180D0, 0x180D4
local SCRD, SCDST, SCBLKS = 0x180D8, 0x20000, 8
local SC_ERR, SC_STAT, SC_PH = 0x18200, 0x18204, 0x18208
local ERRNAME = {[0]="OK", "SELECTION TIMEOUT -- no target answered",
"UNEXPECTED PHASE", "POLL TIMEOUT -- a phase never arrived",
"NON-ZERO SCSI STATUS"}
local DISK = os.getenv("DLX_SCSI_IMG") or "dlxdisk.img"
local code do local f=io.open("scsigate.bin","rb"); code=f:read("a"); f:close() end
local st = "boot"
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
if st == "boot" then
if T() < 3.0 then return end
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
SP:write_u32(SCFLAG, 0)
local cpu = M.devices[":maincpu"]
cpu.state["SR"].value = 0x2700
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
P(string.format("scsigate.bin=%d B loaded at $10000", #code))
st = "wait"; return
end
if st == "wait" then
local fl = SP:read_u32(SCFLAG)
if fl ~= 1 then
if T() > 30 then P("TIMEOUT: the gate never finished"); P("done"); M:exit() end
return
end
-- THE MAP, address by address. A dead address is a bus error the
-- gate survived, not a run that stopped.
local live, dead = {}, {}
for i = 0, SCN-1 do
if SP:read_u8(SCOK+i) == 1 then live[#live+1] = i else dead[#dead+1] = i end
end
for row = 0, (SCN/16)-1 do
local t = {}
for i = 0, 15 do
local a = row*16 + i
t[#t+1] = (SP:read_u8(SCOK+a) == 1)
and string.format("%02X", SP:read_u8(SCVAL+a)) or "--"
end
P(string.format(" $EA%04X: %s", row*16, table.concat(t, " ")))
end
P(string.format("ANSWERED %d of %d addresses; \"--\" is a bus error.",
#live, SCN))
-- The lane question, decided by which addresses answer.
local odd_live, even_live = 0, 0
for _, a in ipairs(live) do
if a % 2 == 1 then odd_live = odd_live + 1 else even_live = even_live + 1 end
end
P(string.format("LANES: %d odd addresses answer, %d even.", odd_live, even_live))
if SP:read_u32(SCTMPOK) == 1 then
P(string.format("TEMP writeback ($EA0017): wrote $A5, read back $%02X",
SP:read_u32(SCTMP) & 0xFF))
else
P("TEMP writeback ($EA0017): BUS ERROR -- not a writable register here.")
end
P(string.format("DREG writeback ($EA0015): wrote $5A, SSTS then $%02X "
.."(FIFO %s), read back $%02X",
SP:read_u32(0x180E0) & 0xFF,
((SP:read_u32(0x180E0) & 1) == 1) and "EMPTY -- the write "
.."never reached dreg_w" or "has a byte",
SP:read_u32(0x180DC) & 0xFF))
-- The trace, whatever happened.
local ntr = SP:read_u32(0x18230)
if ntr > 0 then
P("trace (SSTS PSNS INTS SERR | TCH TCM TCL | where):")
for i = 0, ntr-1 do
local b = {}
for k = 0, 7 do b[k+1] = string.format("%02X", SP:read_u8(0x18240+i*8+k)) end
local WH = {[0]="init", "selected", "INTS cleared", "phase loop saw REQ",
"TRANSFER issued (out)", "bytes handed over",
"after xfer wait", "TRANSFER issued (in)",
"bytes taken from FIFO", "after IN xfer wait",
"after bus release"}
P(string.format(" %2d: %s %s %s %s | %s %s %s | %s", i,
b[1], b[2], b[3], b[4], b[5], b[6], b[7],
WH[tonumber(b[8], 16)] or b[8]))
end
end
-- THE READ. Verified against the host's copy of the same image: a
-- transport that returns the wrong bytes without saying so is exactly the
-- failure a checksum-free ring cannot survive (49.2).
local rd = SP:read_u32(SCRD)
local e = SP:read_u32(0x180E8)
if rd ~= 0 or e ~= 0 then
P(string.format("scsi_read FAILED: err=%d (%s), status=$%02X, phase=%d",
e, ERRNAME[e] or "?", SP:read_u32(SC_STAT) & 0xFF,
SP:read_u32(SC_PH)))
else
local f = io.open(DISK, "rb")
if not f then
P("scsi_read returned OK but "..DISK.." is not here to check it against.")
else
local want = f:read(SCBLKS * 512); f:close()
local bad, first = 0, nil
for i = 1, #want do
if SP:read_u8(SCDST + i - 1) ~= string.byte(want, i) then
bad = bad + 1; first = first or (i-1)
end
end
if bad == 0 then
P(string.format("READ(10) OK: %d B from LBA 0 match %s byte for byte.",
#want, DISK))
else
P(string.format("READ(10) WRONG: %d of %d bytes differ, first at +%d.",
bad, #want, first))
end
end
end
-- The second read, at a non-zero LBA.
local rd2 = SP:read_u32(0x180E4)
if rd2 ~= 0 then
P(string.format("second scsi_read FAILED: err=%d (%s)",
SP:read_u32(0x180EC),
ERRNAME[SP:read_u32(0x180EC)] or "?"))
else
local f = io.open(DISK, "rb")
if f then
f:seek("set", 1000 * 512)
local want = f:read(4 * 512); f:close()
local bad = 0
for i = 1, #want do
if SP:read_u8(0x28000 + i - 1) ~= string.byte(want, i) then bad = bad + 1 end
end
P(bad == 0
and string.format("READ(10) OK: %d B from LBA 1000 match too.", #want)
or string.format("READ(10) WRONG at LBA 1000: %d of %d differ.", bad, #want))
end
end
P("done"); M:exit(); return
end
end)
if not ok then P("LUA ERROR: "..tostring(err)); P("done"); M:exit() end
end)
tools/bench/scsi_run.sh
Executable
+79
View File
@@ -0,0 +1,79 @@
#!/bin/bash
# One MB89352 transport run: the 68000 selects a SCSI target and reads the disc
# itself (ROADMAP P4, first half).
#
# tools/bench/scsi_run.sh [container.dlx]
#
# THE APPARATUS, AND ITS TWO SUBSTITUTIONS, BOTH DELIBERATE AND BOTH LABELLED.
#
# 1. THE BOARD. `x68000 -exp1 cz6bs1`, which is what FINDINGS 42.5 says to
# benchmark and never `x68ksupr` -- the Super/XVI internal SCSI is PIO-only in
# MAME (`// TODO: duplicate DMA glue from CZ-6BS1`), so it would measure a
# fallback the real machine does not have.
#
# 2. THE BOOT ROM. MAME refuses to instantiate the card without an 8 KB
# `scsiexrom.bin` (CRC 7be488de), which is not in this tree and is not on this
# machine. This script writes a ZERO-FILLED placeholder into its own rompath,
# so the user's romset is untouched, and MAME prints WRONG CHECKSUMS as it
# should. That is honest HERE and would not be everywhere: the player drives
# the SPC registers directly and never executes a byte of that ROM -- which
# was already the plan in docs/BENCHMARK.md item 4, long before the ROM turned
# out to be missing. DO NOT reuse this rompath for anything that boots from
# the card or calls SCSI IOCS; those DO execute it.
#
# WHAT A GREEN RUN MEANS: the 68000 reached the SPC, selected a target, issued
# READ(10) twice -- at LBA 0 and at a non-zero LBA -- and both came back byte for
# byte identical to the host's copy of the same image. No IOCS, no host in the
# transfer path.
#
# WHAT IT DOES NOT MEAN: anything at all about RATE. MAME's device models are
# functional, not transfer-timing accurate (docs/BENCHMARK.md), and 42.5 reads
# its DMAC configured in wall-clock attotimes rather than per-operand cycles.
# `W` -- clocks stolen per delivered byte, the project's largest open number --
# is untouched by this script.
set -e
cd "$(dirname "$0")/../.."
DLX=${1:-tmp/rc_fr_singe_scsi_span.dlx}
# The container's frame records, laid out as a disc, and the blank card ROM.
# Both are tools/bench/mkvol.sh's, shared with tools/bench/pace_run.sh so the
# SCSI rig and the host-file ring rig cannot be reading different bytes.
bash tools/bench/mkvol.sh "$DLX"
tools/vasm/vasmm68k_mot -Fbin -o tmp/scsigate.bin src/player/scsigate.s > /dev/null
# stdbuf -oL: without it a long MAME run is unobservable until it exits, and a
# run that is merely finishing looks exactly like one that is wedged (34.1).
( cd tmp && SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 300 \
mame x68000 -bios ipl10 -exp1 cz6bs1 \
-rompath "$HOME/mame/roms;./p4roms" -hard dlxdisk.chd \
-ramsize 2M -video soft -window -sound none -nothrottle -plugins \
-autoboot_script ../tools/bench/scsi.lua \
-seconds_to_run 60 > scsi_run.log 2>&1 )
# A run that never reached the report must fail as that, not as a missing line.
grep -aq "^\[SCSI\] done" tmp/scsi_run.log || {
echo "FAIL: the SCSI gate did not finish -- no completion marker."
tail -8 tmp/scsi_run.log; exit 1; }
grep -a "^\[SCSI\]" tmp/scsi_run.log | grep -avE "^\[SCSI\] +[0-9]+:" \
| sed 's/^\[SCSI\] / /'
# THE ASSERTIONS. Printing a result and gating on it are different things, and
# this project has already paid once for a stage that printed.
grep -aq "ANSWERED 60 of 64" tmp/scsi_run.log || {
echo "FAIL: the card's register window is no longer 60 of 64 addresses."
echo " The MB89352 omits TMOD (index 3) and EXBF (index 15) where the"
echo " MB87030 has them, and MAME leaves HOLES rather than shifting the"
echo " later indices down -- which is what puts DREG at \$EA0015. If this"
echo " moved, src/player/scsi.i's whole register map moved with it."
exit 1; }
grep -aq "READ(10) OK: 4096 B from LBA 0" tmp/scsi_run.log || {
echo "FAIL: the 68000 did not read LBA 0 off the SCSI volume byte-exact."
exit 1; }
grep -aq "READ(10) OK: 2048 B from LBA 1000" tmp/scsi_run.log || {
echo "FAIL: the read at a NON-ZERO LBA did not match. A driver that emits a"
echo " malformed LBA field still passes LBA 0, because zero is what a"
echo " malformed field usually is -- so this is the half that matters."
exit 1; }
grep -aq "scsi_read FAILED" tmp/scsi_run.log && {
echo "FAIL: a read reported an error."; exit 1; }
exit 0
tools/bench/span.lua
+2 -1
View File
@@ -107,7 +107,8 @@ end
local function launch(cfg)
push(STREAM, blob, cfg.off+1, cfg.len)
clear_picture()
-- ~4 emulated seconds per config: 1/55.46 s granularity costs under 0.5%.
-- ~4 emulated seconds per config: 1/56.69 s granularity (crtc_mode.lua)
-- costs under 0.5%.
local est = cfg.nspans*(cfg.var == 5 and 60 or (cfg.var == 6 and 50 or 70))
+ cfg.npix*10
cfg.iter = math.max(4, math.floor(4*CPUHZ/est))
tools/bench/stream.lua
+650 -24
View File
@@ -52,8 +52,29 @@
-- no provenance that was never a bus measurement at all
-- (FINDINGS 42.1). A default is how a folklore number ends up
-- silently underneath a table nobody restates it in.
-- DLX_PREFILL_KB bytes to deliver before releasing the CPU (default 0)
-- DLX_PACE 1 = hold the decoder to META.fps (default 0 = free-run)
-- DLX_PREFILL_KB bytes to deliver before releasing the CPU (default 0).
-- Host-owned ring only: with DLX_RINGOWN the machine does
-- its own prefill and DLX_PREFILL_FR is the knob.
-- DLX_RINGOWN 1 = THE 68000 OWNS THE RING (ROADMAP P5,
-- src/player/ring.i). This script stops placing records and
-- becomes a TRANSPORT: it answers one request at a time,
-- delivering XF_LEN bytes from XF_OFF to XF_DST in
-- len/rate seconds, and the machine decides which record,
-- where in the ring, and when. It also AUDITS every
-- placement the machine makes against its own index.
-- DLX_PREFILL_FR whole records the machine's prefill waits for before it
-- releases the decoder (default 2; see ring_prefill)
-- DLX_PACE 1 = hold the decoder to META.fps, with the HOST writing
-- the tick (default 0 = free-run)
-- 2 = hold it to META.fps with the 68000 writing its own
-- tick, off the CRTC's V-DISP (src/player/clock.i, ROADMAP
-- P3). Same gate, same ring, same deadlines; what changes
-- is that nothing outside the machine decides when a frame
-- may start. Deadlines are then taken from the ticks the
-- machine actually emitted rather than from a host model of
-- 12 fps -- which matters, because MAME's raster runs 2.22%
-- fast (see tools/bench/clock.lua) and a host model would
-- quietly grade every arrival against the wrong clock.
-- DLX_CUT_AT frame tick at which the pipe stops dead (a seek). Needs
-- DLX_PACE; unset = no cut.
-- DLX_CUT_FR how many frame times the cut lasts (default 1)
@@ -88,7 +109,33 @@ local META = loadfile("stream_meta.lua")()
local FLAG, ITER, NFR = 0x18000, 0x18008, 0x1800C
local RD_PTR, FR_HEAD, FR_TAIL = 0x18020, 0x18024, 0x18028
local STALLS, SPINS, DESC = 0x1802C, 0x18030, 0x18100
local PACE, PACEON = 0x18034, 0x18038
local PACE, PACEON, CLKON = 0x18034, 0x18038, 0x1803C
local CLK_VDISP, CLK_FPS, CLK_ERR = 0x18064, 0x18068, 0x1806C
local LATEFR, LATEMAX, LATE1ST = 0x18080, 0x18084, 0x18088
-- src/player/ring.i. The transport mailbox, then the producer's state and its
-- instruments. Nothing below 0x18300 changed: the decoder's handshake with the
-- ring (FR_HEAD/FR_TAIL/DESC) is the same one FINDINGS 49 and 51 measured, which
-- is what makes a self-filled run a test of ring.i and not of a new rig.
local XF_SLOT, XF_SLSZ = 0x18300, 16
local XF_GO, XF_ACK, XF_QD = 0x18320, 0x18324, 0x18328
local A_RINGOWN, A_RNG_B, A_RNG_SZ, A_IDX_B = 0x1832C, 0x18330, 0x18334, 0x18338
local A_RQ_NEXT, A_WCUR, A_RCUR = 0x1833C, 0x18340, 0x18344
local A_NHOLE, A_NHOLEB, A_NFULL = 0x18354, 0x18358, 0x1835C
local A_NPOLL, A_NISSUE = 0x18360, 0x18364
local A_SLKMIN, A_SLKAT = 0x18368, 0x1836C
local A_PFREC, A_PFDONE = 0x18370, 0x18374
local A_NSEEK, A_SKWAIT = 0x18378, 0x1837C
-- src/player/xfer.i, the transport that lives in the machine.
local A_XFSCSI, A_XSLBA0 = 0x18380, 0x18384
local A_XSNXFER, A_XSNBYTE = 0x18388, 0x1838C
local A_XSNWIRE, A_XSERR, A_XSERRAT = 0x18390, 0x18394, 0x18398
-- The scene header's record index, pushed into RAM before the CPU is launched.
-- It sits ABOVE the codebooks rather than in the low RAM around ring.i's own
-- tables: setup() runs one raster frame before launch() and the IPL is still
-- executing in that gap, so anything written below ~$20000 can be overwritten
-- before the 68000 ever sees it. That cost an hour: the machine read a table of
-- zeroes and asked the transport for a 0-byte record.
local IDXRAM = 0x30000 -- the DLX4 index, in RAM
local DESCN = 64
local CB1, CB4 = 0x20000, 0x22000
local RING = 0x40000
@@ -108,10 +155,45 @@ if KBPS == nil then
return
end
local PREFILL = (tonumber(os.getenv("DLX_PREFILL_KB") or "") or 0) * 1024
local PACED = (os.getenv("DLX_PACE") == "1")
local SELFCLK = (os.getenv("DLX_PACE") == "2")
local PACED = (os.getenv("DLX_PACE") == "1") or SELFCLK
local CUT_AT = tonumber(os.getenv("DLX_CUT_AT") or "")
local CUT_FR = tonumber(os.getenv("DLX_CUT_FR") or "") or 1
local SLACK_CSV = os.getenv("DLX_SLACK_CSV")
local RINGOWN = (os.getenv("DLX_RINGOWN") == "1")
local PREFILL_FR = tonumber(os.getenv("DLX_PREFILL_FR") or "") or 2
-- Passes over the scene. With DLX_RINGOWN each pass after the first begins
-- with a REAL seek in src/player/stream.s: the channel is waited quiet, the
-- ring is declared empty and the lookahead 51.3 says takes seconds of play to
-- accumulate is thrown away and rebuilt from the prefill. The check that
-- matters afterwards is the one this rig always makes -- the last frame of the
-- last pass has to be pixel-exact, and a SKIP block is a claim about the
-- previous frame, so it is only right if every frame after the seek was.
local ITERS = tonumber(os.getenv("DLX_ITER") or "") or 1
-- How many requests the player may have outstanding at once. 1 is the obvious
-- loop and leaves the channel idle from every completion until the next poll;
-- 2 keeps the next request queued so it never stops. Both are real designs and
-- the difference between them is what this rig measures.
local QDEPTH = tonumber(os.getenv("DLX_QDEPTH") or "") or 2
-- WHO MOVES THE BYTES. "model" is this script: a transport that delivers
-- XF_LEN bytes in XF_LEN/rate seconds and acks from emulated time, which is
-- what FINDINGS 49/51/55 were all measured through. "scsi" is ROADMAP P4b --
-- src/player/xfer.i and src/player/scsi.i on the 68000, a real MB89352, a real
-- volume, and NOTHING of this script in the transfer path.
--
-- The two are not interchangeable and the log must never let them look it. The
-- modelled transport OVERLAPS with the CPU, which is what a DMAC channel does;
-- the real one here does not, because 57.3 leaves the CPU moving every byte
-- itself. So a "scsi" run measures CORRECTNESS off a real volume and the CPU
-- cost of a PIO transport, and it measures NO rate: every rate-shaped number in
-- this script's report is suppressed rather than printed against a transport
-- that has no model behind it.
local XFER = os.getenv("DLX_XFER") or "model"
local SCSIX = (XFER == "scsi")
-- src/player/scsi.i's SCE_* codes, so a failure names itself.
local SCERRNAME = {[0]="OK", "SELECTION TIMEOUT -- no target answered",
"UNEXPECTED PHASE", "POLL TIMEOUT -- a phase never arrived",
"NON-ZERO SCSI STATUS"}
-- One snapshot per frame tick instead of one at the end of the run. This is a
-- DOCUMENTATION artefact -- 120 PNGs of a paced player, for a recording -- and
-- it is deliberately not on any path tools/bench/check.sh takes. Needs
@@ -127,6 +209,16 @@ local BPS = (KBPS > 0) and (KBPS - AUDIO_KBPS) * 1024 or math.huge
local code do local f=io.open("stream.bin","rb"); code=f:read("a"); f:close() end
local cb do local f=io.open("stream_cb.bin","rb"); cb=f:read("a"); f:close() end
local DISK = assert(io.open("stream_disk.bin","rb"))
local idxblob do
local f = io.open("stream_idx.bin","rb")
if f then idxblob = f:read("a"); f:close() end
end
if RINGOWN and not idxblob then
print("[STR] DLX_RINGOWN needs the DLX4 record index (stream_idx.bin). "
.."Re-run tools/bench/prep_stream.py on a DLX4 container: the machine "
.."cannot derive record lengths by walking a stream it has not fetched.")
manager.machine:exit(); return
end
local YOFF = (MODE.height - META.H) // 2
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
@@ -251,6 +343,202 @@ local function produce(now)
end
end
-- -------------------------------------------------------------- transport
-- THE OTHER HALF OF ROADMAP P5. With DLX_RINGOWN the producer above is not
-- used at all: src/player/ring.i decides which record to fetch, where to put it
-- and when it is safe, and this becomes the part that is genuinely not the
-- CPU's -- an SPC and one DMAC channel moving bytes at a rate.
--
-- IT IS BUILT OUT OF MEMORY TAPS, not out of the machine-frame notifier, and
-- that is a measurement decision rather than a stylistic one. A notifier sees
-- the machine once per raster frame, 17.64 ms (54.5), and the thing being
-- measured here is the gap between a transfer COMPLETING and the CPU issuing
-- the next one -- which is a fraction of a frame slot. Sampling it at 17.64 ms
-- would have quantised the very quantity in question, and in the flattering
-- direction if the ack were early or the pessimistic one if late.
--
-- write tap on XF_GO fires inside the 68000's write, so the issue time is
-- exact. The bytes are copied in there and then: the
-- CPU cannot observe them before its ack anyway, because
-- it does not advertise the record until it retires.
-- read tap on XF_ACK synthesises the completion word from emulated time --
-- `done` iff now >= t_done -- so the CPU learns of the
-- completion on the exact cycle it happens, the way it
-- would from a status register.
-- A 68000 bus access is 16 bits, so a longword is two tap calls: the write is
-- acted on at the LOW half (written second, so the whole value is there) and
-- the read latches its answer at the HIGH half so the two halves cannot
-- straddle a completion.
local xf = {go = 0, ack = 0, busy = 0.0, gap = 0.0, gapmax = 0.0, ngap = 0,
latch = 0, tfree = 0, copylate = 0.0, ncopylate = 0}
-- q[n] = the n-th request (n counts from 1): when the CPU issued it, when the
-- transport will have finished it, and whether its bytes have been moved yet.
local q = {}
local npass = 1
local function xf_deliver(off, dst, len, now)
DISK:seek("set", off)
push(dst, DISK:read(len), 1, len)
end
-- The host as AUDITOR rather than as producer. Every placement the machine
-- makes is checked against this script's own record index and its own list of
-- records the decoder has not finished with -- the same overlap test the host
-- producer used to make its decisions with, now used only to grade them. A
-- wrong placement corrupts pixels rather than faulting (49.2), so it has to be
-- caught where it is made.
local function audit(idx, off, dst, len)
local rec = META.index[idx + 1]
if not rec then return string.format("record %d does not exist", idx) end
if off ~= rec.off then
return string.format("record %d: machine asked for disc offset %d, the "
.."index says %d", idx, off, rec.off) end
if len ~= rec.len then
return string.format("record %d: machine asked for %d B, the index says %d",
idx, len, rec.len) end
local w = dst - RING
if w < 0 or w + len > RINGSZ then
return string.format("record %d: placed at ring offset %d..%d, outside a "
.."%d B ring", idx, w, w + len, RINGSZ) end
local tail = SP:read_u32(FR_TAIL)
local i = 1
while i <= #live do
if live[i].idx < tail then table.remove(live, i) else i = i + 1 end
end
for _, r in ipairs(live) do
if w < r.off + r.len and r.off < w + len then
return string.format("record %d at %d..%d overlaps record %d at %d..%d, "
.."which the decoder has not consumed",
idx, w, w + len, r.idx, r.off, r.off + r.len) end
end
live[#live+1] = {idx = idx, off = w, len = len}
return nil
end
-- THE TAP OBJECTS ARE KEPT ALIVE HERE ON PURPOSE. install_*_tap returns a
-- handle and the tap dies with it: dropping it on the floor leaves the taps
-- working until Lua's collector next runs, and then the mailbox silently stops
-- answering. That looks exactly like a wedged producer -- the machine polled
-- 2.4 million times for an ack that had already been computed.
-- THE TAP OBJECTS ARE KEPT ALIVE HERE ON PURPOSE. install_*_tap returns a
-- handle and the tap dies with it: dropping it on the floor leaves the taps
-- working until Lua's collector next runs, and then the mailbox silently stops
-- answering. That looks exactly like a wedged producer -- the machine polled
-- 2.4 million times for an ack that had already been computed.
local TAPS = {}
local function install_transport()
-- NOTHING IN A TAP CALLBACK TOUCHES THE MEMORY SPACE. MAME 0.277 segfaults
-- if a tap reads or writes the space it was triggered from -- a nested access
-- part way through the CPU's own -- and it does it intermittently, which is
-- the worst way to find out. So the taps do pure Lua: the write tap stamps
-- the EXACT emulated time the 68000 issued a request, the read tap answers
-- the completion count from those stamps, and every actual memory access is
-- done from the machine-frame notifier.
--
-- What that buys is the thing worth having: issue and completion times are
-- both exact, so the gap between a transfer finishing and the next one
-- starting -- the disc standing still because nobody has asked it for
-- anything -- is measured at cycle resolution rather than at 17.64 ms.
TAPS[#TAPS+1] = SP:install_write_tap(XF_GO, XF_GO + 3, "dlx_xf_go",
function(offset, data, mask)
if offset ~= XF_GO + 2 then return data end -- low half, written second
-- ...and it must be the INCREMENT, not any other write to the word.
-- ring_init clears the mailbox at boot, which lands on this address with
-- the request fields still empty; without this test the transport answered
-- that clear as if it were a request and audited a 0-byte record 0.
if data ~= ((xf.go + 1) & 0xFFFF) then return data end
xf.go = xf.go + 1
q[xf.go] = {tissue = T(), served = false}
return data
end)
TAPS[#TAPS+1] = SP:install_read_tap(XF_ACK, XF_ACK + 3, "dlx_xf_ack",
function(offset, data, mask)
if offset == XF_ACK then
-- Latched on the high half so the two halves of one longword read cannot
-- straddle a completion. Requests complete IN ORDER: one channel, one
-- transfer at a time, however many are queued.
local now = T()
while true do
local r = q[xf.ack + 1]
if not (r and r.served and now >= r.tdone) then break end
xf.ack = xf.ack + 1
end
xf.latch = xf.ack
return (xf.latch >> 16) & 0xFFFF
end
return xf.latch & 0xFFFF
end)
end
-- Service every request the machine has issued but this script has not yet
-- looked at: read what was asked for, grade the placement, move the bytes, and
-- work out when the channel will have finished it. Called from the
-- machine-frame notifier, which is where this script may touch memory.
local function transport_service(t)
local n = xf.ack + 1
while q[n] do
local r = q[n]
if r.served then n = n + 1; goto continue end
local slot = XF_SLOT + ((n - 1) % 2) * XF_SLSZ
local off, dst = SP:read_u32(slot), SP:read_u32(slot + 4)
local len, idx = SP:read_u32(slot + 8), SP:read_u32(slot + 12)
if idx == 0 and n > 1 then
-- A new pass: everything the auditor knows about the ring is about the
-- scene we just left. The decoder has not consumed the new records and
-- the old ones are no longer anybody's.
live, nsent, arrival = {}, 0, {}
npass = npass + 1
P(string.format("SEEK PASS %d: the machine seeked back to record 0 at "
.."%.3f s and is refilling from empty", npass, r.tissue))
end
local err = audit(idx, off, dst, len)
if err then
P("MISPLACED: "..err)
P(" The machine owns the ring in this run, so this is a policy bug in "
.."src/player/ring.i, not a rig one -- and it would have shown up as "
.."wrong pixels, because the block loop reads without a bounds check.")
M:exit(); return
end
-- ONE CHANNEL: a transfer starts when the channel is free AND the request
-- exists, so a queued request starts the instant its predecessor lands and
-- a late one starts when it is issued. The difference is the gap, and the
-- gap is the player's, not the medium's -- no host-filled run could see it,
-- because the host producer placed records whenever it liked.
local startt = math.max(r.tissue, xf.tfree)
if xf.tfree > 0 then
local g = startt - xf.tfree
if g > 0 and nsent < META.nframes then
xf.gap, xf.ngap = xf.gap + g, xf.ngap + 1
if g > xf.gapmax then xf.gapmax = g end
end
end
local dur = (BPS == math.huge) and 0 or (len / BPS)
local done = startt + dur
-- A cut is a seek: the transfer freezes for its duration rather than
-- carrying on invisibly. Freezing is the conservative reading and the one
-- 51.6 settled on for the host producer -- a drive that is repositioning is
-- not banking bytes it will burst on arrival.
if cut_t0 and done > cut_t0 and startt < cut_t1 then
done = done + (cut_t1 - math.max(startt, cut_t0))
end
xf_deliver(off, dst, len, t)
r.tdone, r.served = done, true
xf.tfree = done
xf.busy = xf.busy + dur
nsent = math.max(nsent, idx + 1)
arrival[idx + 1] = done -- resident when the last byte lands
delivered = delivered + len
-- The one place this rig is coarser than the machine: a record whose
-- modelled transfer was shorter than the wait for this notifier was already
-- "done" by the time the bytes could be moved. Counted, not absorbed.
if t > done then
xf.copylate, xf.ncopylate = xf.copylate + (t - done), xf.ncopylate + 1
end
n = n + 1
::continue::
end
end
-- ------------------------------------------------------------------ setup
local function setup()
MODE.apply(SP)
@@ -267,17 +555,62 @@ local function setup()
end
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
SP:write_u32(FLAG, 0); SP:write_u32(FR_HEAD, 0); SP:write_u32(FR_TAIL, 0)
SP:write_u32(ITER, 1); SP:write_u32(NFR, META.nframes)
SP:write_u32(ITER, ITERS); SP:write_u32(NFR, META.nframes)
SP:write_u32(PACE, 0); SP:write_u32(PACEON, PACED and 1 or 0)
SP:write_u32(CLKON, SELFCLK and 1 or 0)
SP:write_u32(CLK_FPS, META.fps)
SP:write_u32(A_RINGOWN, RINGOWN and 1 or 0)
if RINGOWN then
-- The scene header the machine reads: the DLX4 record index, exactly the
-- bytes the container carries, pushed into RAM the way a loaded scene
-- header would be. ring_init turns it into the disc-offset table a seek
-- needs; nothing here derives a record boundary for the machine.
push(IDXRAM, idxblob, 1, #idxblob)
SP:write_u32(A_IDX_B, IDXRAM)
SP:write_u32(A_RNG_B, RING)
SP:write_u32(A_RNG_SZ, RINGSZ)
SP:write_u32(A_PFREC, PREFILL_FR)
SP:write_u32(XF_QD, QDEPTH)
if SCSIX then
-- The stream begins at LBA 0 of the volume, because tools/bench/mkvol.sh
-- lays tmp/stream_disk.bin down from sector 0 and nothing else is on it.
-- It is a WORD the machine reads rather than a constant in xfer.i: a
-- shipping volume has a filesystem in front of the stream, and the base
-- LBA is the one number that changes when it does.
SP:write_u32(A_XFSCSI, 1)
SP:write_u32(A_XSLBA0, 0)
else
install_transport()
end
end
P(string.format("stream.bin=%d B, codebooks %d+%d B, disk %d B, %d frames",
#code, META.cb1_len, META.cb4_len, META.disk_len, META.nframes))
P(string.format("decoder %s%s", PACED and ("PACED at "..META.fps.." fps")
P(string.format("decoder %s%s", SELFCLK
and ("SELF-PACED at "..META.fps.." fps off V-DISP")
or PACED and ("PACED at "..META.fps.." fps by the host")
or "FREE-RUNNING (tests wrap, not buffering -- 49.7.2)",
CUT_AT and string.format(", pipe cut at tick %d for %.2f fr",
CUT_AT, CUT_FR) or ""))
P(string.format("ring %d KB at %06X, pipe %s, prefill %d KB, maxrec %d B",
P(string.format("ring %d KB at %06X, pipe %s, prefill %s, maxrec %d B",
RING_KB, RING, (KBPS > 0) and (KBPS.." KB/s") or "unlimited",
PREFILL // 1024, META.maxrec))
RINGOWN and (PREFILL_FR.." records (the machine's own)")
or ((PREFILL // 1024).." KB"), META.maxrec))
if RINGOWN then
if SCSIX then
P(string.format("REAL TRANSPORT (ROADMAP P4b, src/player/xfer.i): the "
.."68000 fetches every record itself with READ(10) off a "
.."CZ-6BS1. This script moves NO bytes and models NO "
.."rate. %d B of DLX4 index at %06X.", #idxblob, IDXRAM))
P(" the queue depth cannot deepen anything here: the transport is "
.."SYNCHRONOUS (xfer.i), so a queued request is drained by the same "
.."instruction stream that would be decoding. FINDINGS 55.3's 1-vs-2 "
.."result is about a transport that OVERLAPS; this one does not.")
else
P(string.format("ring OWNED BY THE 68000 (src/player/ring.i): this script "
.."is a transport with a %d-deep request queue, %d B of "
.."DLX4 index at %06X", QDEPTH, #idxblob, IDXRAM))
end
end
if META.maxrec > RINGSZ then
P("RING TOO SMALL: one record does not fit. stream.s needs a whole record "
.."contiguous."); M:exit()
@@ -292,20 +625,66 @@ local function launch()
end
local st, t0, t_rel = "boot", nil, 0
-- tick_t[i+1] = emulated time of frame tick i, filled in as they are observed.
-- Under a self-clock this replaces the host's `t_rel + i/fps` as the deadline:
-- the machine's clock is the one the player is actually held to, and MAME's
-- raster is 2.22% fast, so grading arrivals against a host model of 12 fps
-- would flatter every record by that much.
local tick_t = {}
local pace, min_ahead, min_at = -1, math.huge, -1
-- DLX_XFER=scsi: the machine's seek counter, and the transfer count it stood at
-- when that seek happened. -1 means "not sampled yet", so the first sample
-- rebases without announcing a pass that did not happen.
local xs_nseek, xs_base = -1, 0
local sum_ahead, n_ahead = 0, 0
local slack_series = {}
SUB = emu.add_machine_frame_notifier(function()
local ok, err = pcall(function()
local t = T()
if RINGOWN and not SCSIX then transport_service(t) end
if SCSIX and st ~= "boot" then
-- The machine's own count of completed transfers, which is what `nsent`
-- means in a run where this script sends nothing. Sampled rather than
-- derived, so the slack window still closes when the producer runs out of
-- records to place instead of running on into the drain tail.
--
-- REBASED AT EVERY SEEK, and it has to be: XS_NXFER is cumulative over
-- the whole run, so on a second pass it is already >= nframes and the
-- slack sampling below -- gated on `nsent < META.nframes` -- would never
-- fire again. It did not, and the second pass reported a ceiling of 0
-- frames and a build time of -1 ticks, which is an empty series printing
-- as a result. ring.i's own seek counter is the exact place to rebase:
-- a seek discards every outstanding request by definition.
local ns = SP:read_u32(A_NSEEK)
if ns ~= xs_nseek then
-- ANNOUNCE ONLY A SEEK WITH PLAY BEHIND IT. A pass opens with two
-- seeks that are not branch points -- ring_init's, and stream.s's at
-- the top of `outer` -- and both happen before a single record has been
-- fetched. Announcing those numbered the first pass "SEEK PASS 2" and
-- the real one "3". XS_NXFER separates them exactly: a seek taken with
-- transfers already behind it is the one that threw a ring away.
if xs_nseek >= 0 and SP:read_u32(A_XSNXFER) > 0 then
npass = npass + 1
P(string.format("SEEK PASS %d: the machine seeked at %.3f s and is "
.."refilling from empty", npass, t))
end
xs_nseek, xs_base = ns, SP:read_u32(A_XSNXFER)
end
nsent = SP:read_u32(A_XSNXFER) - xs_base
if nsent < 0 then nsent = 0 end
end
if st == "boot" then
if t < 3.0 then return end
setup(); last_t, pf_t0 = t, t; st = "prefill"; return
end
if st == "prefill" then
produce(t)
if delivered >= PREFILL then
if not RINGOWN then produce(t) end
-- RINGOWN: there is nothing for the host to prefill. The machine issues
-- its own requests, so it has to be running first; ring_prefill then
-- holds the decoder until PF_REC records are resident, which is the same
-- policy in the place a player would keep it.
if RINGOWN or delivered >= PREFILL then
P(string.format("prefill done: %.1f KB in %.3f s, releasing the CPU",
delivered/1024, t - pf_t0))
launch(); t_rel = t; st = "running"
@@ -314,10 +693,29 @@ SUB = emu.add_machine_frame_notifier(function()
end
if st == "running" then
if PACED then
local tick = math.floor((t - t_rel) * META.fps)
-- WHO WRITES THE TICK. Host-paced, the tick is a host model of
-- META.fps and this script advances it. Self-paced, the 68000 has
-- already advanced it off the raster and this script only READS it --
-- the frame gate in src/player/stream.s cannot tell the two apart,
-- which is the point: the same bytes are gated either way.
local tick = SELFCLK and SP:read_u32(PACE)
or math.floor((t - t_rel) * META.fps)
if tick < pace then
-- src/player/stream.s rebased the clock for a new pass: tick 0 is the
-- instant the decoder was released after the seek's prefill, not the
-- start of the run. Everything sampled per tick starts again with it.
pace, tick_t = -1, {}
min_ahead, min_at = math.huge, -1
sum_ahead, n_ahead, slack_series = 0, 0, {}
end
if tick > pace then
pace = tick
SP:write_u32(PACE, pace)
if not SELFCLK then SP:write_u32(PACE, pace) end
-- The emulated time this tick actually happened, which is what a
-- record's deadline is measured against under a self-clock. Ticks
-- can arrive more than one apart if the host misses a frame, so the
-- whole run is filled rather than just the newest.
for k = #tick_t, tick do tick_t[k+1] = t end
-- Taken BEFORE this tick's frame is decoded, so snapshot n is the
-- finished picture of frame n-1. tick 0 is skipped: nothing has been
-- drawn yet and it would record a black screen as a decoded frame.
@@ -338,7 +736,12 @@ SUB = emu.add_machine_frame_notifier(function()
if nsent < META.nframes then
if ahead < min_ahead then min_ahead, min_at = ahead, tick end
sum_ahead, n_ahead = sum_ahead + ahead, n_ahead + 1
slack_series[#slack_series+1] = {tick, ahead, n_ring, n_rate}
-- Column 3 is "has the producer been refused for space yet",
-- which is what makes a later minimum meaningful: before the first
-- refusal the ring is still filling. With the ring owned by the
-- machine that counter is ring.i's, not this script's.
slack_series[#slack_series+1] =
{tick, ahead, RINGOWN and SP:read_u32(A_NFULL) or n_ring, n_rate}
end
if CUT_AT and tick >= CUT_AT and not cut_t0 then
cut_t0, cut_t1 = t, t + CUT_FR / META.fps
@@ -348,16 +751,55 @@ SUB = emu.add_machine_frame_notifier(function()
end
end
end
produce(t)
if not RINGOWN then produce(t) end
if RINGOWN and os.getenv("DLX_RINGDBG") == "1" then
dbg_n = (dbg_n or 0) + 1
if dbg_n % 30 == 0 then
P(string.format("DBG t=%.2f flag=%X go=%d ack=%d tfree=%.3f queued=%s "
.."head=%d tail=%d rq=%d poll=%d full=%d wcur=%d rcur=%d",
t, SP:read_u32(FLAG), xf.go, xf.ack, xf.tfree,
tostring(q[xf.ack+1] ~= nil), SP:read_u32(FR_HEAD),
SP:read_u32(FR_TAIL), SP:read_u32(A_RQ_NEXT),
SP:read_u32(A_NPOLL), SP:read_u32(A_NFULL),
SP:read_u32(A_WCUR), SP:read_u32(A_RCUR)))
end
end
if cut_t0 and not cut_done and t >= cut_t1 then cut_done = true end
local fl = SP:read_u32(FLAG)
if fl == 1 and not t0 then t0 = t; return end
if fl == 0xEE then
P("BITSTREAM DESYNC -- the decoder consumed the wrong number of bytes.")
if RINGOWN then
local tail, head = SP:read_u32(FR_TAIL), SP:read_u32(FR_HEAD)
P(string.format(" tail=%d head=%d rq=%d wcur=%d rcur=%d go=%d ack=%d",
tail, head, SP:read_u32(A_RQ_NEXT), SP:read_u32(A_WCUR),
SP:read_u32(A_RCUR), xf.go, xf.ack))
P(string.format(" DESC[%d]=%08X, the index says record %d is at ring "
.."offset ? len %d", tail,
SP:read_u32(DESC + (tail % 64)*4), tail,
META.index[tail+1] and META.index[tail+1].len or -1))
local ls = {}
for _, r in ipairs(live) do
ls[#ls+1] = string.format("%d@%d+%d", r.idx, r.off, r.len) end
P(" host thinks live: "..table.concat(ls, " "))
end
P(" Under a ring that is the whole point: it means a record was placed "
.."or described wrongly, not that the codec changed.")
M:exit(); return
end
if fl == 0xE2 then
local e = SP:read_u32(CLK_ERR)
P("FRAME CLOCK REFUSED: CLK_ERR="..e..(e == 1 and
" (the CRTC is not in a 31.5 kHz mode, so the divider's 31500 "
.."lines/s would be wrong)" or e == 2 and
" (fps*VTOTAL does not fit the 16-bit accumulator)" or ""))
M:exit(); return
end
if fl == 0xE3 then
P("RING INIT REFUSED: the record index has more entries than "
.."src/player/ring.i's disc-offset table (ROFFMAX) can hold.")
M:exit(); return
end
if fl == 0xE1 then
P("PRODUCER STALLED OUT -- stream.s spun SPINMAX times with no new "
.."record. Delivered "..nsent.."/"..META.nframes..".")
@@ -382,10 +824,117 @@ SUB = emu.add_machine_frame_notifier(function()
META.nframes, dt, dt*CPUHZ/META.nframes,
100*(dt*CPUHZ/META.nframes)/FRAME12, META.fps))
end
if SELFCLK then
-- The clock's own report, in the run where it actually paced a
-- decoder rather than a busy loop. The rate is against MAME's fast
-- raster; tools/bench/clock_run.sh is where it gets de-skewed.
local vd = SP:read_u32(CLK_VDISP)
local vt = SP:read_u16(0xE80008) + 1 -- VTOTAL, as clk_init read it
-- DELIBERATELY NOT A RATE. 120 ticks is far too short a window to
-- quote fps from: the run's start and end each straddle a tick, so
-- +/-1 on 119 intervals is +/-8000 ppm and would read as drift the
-- clock does not have. What IS exact here is a count of refreshes
-- against a count of ticks. The rate figure comes from
-- tools/bench/clock_run.sh, over thousands of them.
P(string.format("FRAME CLOCK: %d V-DISP interrupts drove %d ticks "
.."(%.4f refreshes/frame; the divider's own ratio is "
.."31500/(%d*%d) = %.4f), %.0f clocks/frame of "
.."interrupt at FINDINGS 54's 181.35 each",
vd, pace + 1, vd/(pace + 1), META.fps, vt,
31500/(META.fps*vt), 181.35*vd/(pace+1)))
end
local rholes, rhole_b = holes, hole_bytes
if RINGOWN then
rholes, rhole_b = SP:read_u32(A_NHOLE), SP:read_u32(A_NHOLEB)
end
P(string.format("ring: %d wraps, %d B of hole (mean %.1f KB, %.1f%% of "
.."the ring)", holes, hole_bytes,
holes > 0 and hole_bytes/holes/1024 or 0,
100*(holes > 0 and hole_bytes/holes or 0)/RINGSZ))
.."the ring)", rholes, rhole_b,
rholes > 0 and rhole_b/rholes/1024 or 0,
100*(rholes > 0 and rhole_b/rholes or 0)/RINGSZ))
if RINGOWN then
-- The producer's own account of itself. These are read out of the
-- machine's RAM, not kept here: the point of the run is that this
-- script did not make any of these decisions.
local pf = SP:read_u32(A_PFDONE)
local npoll, nissue = SP:read_u32(A_NPOLL), SP:read_u32(A_NISSUE)
local nfull = SP:read_u32(A_NFULL)
local slk, slkat = SP:read_u32(A_SLKMIN), SP:read_u32(A_SLKAT)
local nseek, skw = SP:read_u32(A_NSEEK), SP:read_u32(A_SKWAIT)
P(string.format("MACHINE-OWNED RING: %d records placed by the 68000, "
.."%d polls, %d refused for space (ring-bound), "
.."%d seeks", nissue, npoll, nfull, nseek))
P(string.format("PREFILL: released the decoder at %d records "
.."(policy: %d); LEAST SLACK %d records at frame %d",
pf, PREFILL_FR, slk, slkat))
-- THE NUMBER THIS RIG EXISTS TO PRODUCE. The channel only moves
-- bytes while a request is outstanding and only the CPU can issue the
-- next one, so the disc stands still between the completion of one
-- record and the issue of the next. That gap is a property of the
-- PLAYER's loop, not of the medium, and no host-filled run could see
-- it -- the host producer placed records whenever it liked.
local span = (arrival[nissue] or t) - (t0 or t)
if SCSIX then
-- WHAT A REAL TRANSPORT CAN AND CANNOT BE ASKED HERE. The modelled
-- transport knew when every byte landed because it decided; this one
-- does not report, and MAME's device models are functional rather
-- than transfer-timing accurate, so even if it did the number would
-- not be a rate (docs/BENCHMARK.md, 42.5). So CHANNEL IDLE,
-- DEADLINE and REQUIRED PREFILL are not printed at all rather than
-- printed as zeros -- a zero here reads as "the channel never
-- stopped", which would be a claim about a medium this rig has
-- never timed.
local nb, nw = SP:read_u32(A_XSNBYTE), SP:read_u32(A_XSNWIRE)
local nx = SP:read_u32(A_XSNXFER)
P(string.format("REAL TRANSPORT: %d READ(10)s by the 68000, %d B "
.."into the ring", nx, nb))
-- THE COST THE CONTAINER PAYS FOR NOT BEING SECTOR-ALIGNED. A
-- record starts wherever the previous one ended, rounded up to 4;
-- a target answers in 512 B blocks. So the command covers the
-- sectors the record lies in and src/player/scsi.i's window drops
-- the rest. In PIO those bytes cost nothing but wire; they are
-- still bytes the disc moved that no frame contains.
P(string.format("SECTOR OVERHEAD: %d B off the disc for %d B of "
.."record = %.2f%% the medium moved and no frame "
.."contains", nw, nb,
nb > 0 and 100*(nw - nb)/nb or 0))
local e, eat = SP:read_u32(A_XSERR), SP:read_u32(A_XSERRAT)
if e ~= 0 then
P(string.format("TRANSPORT FAILED: SC_ERR=%d on request %d (%s)",
e, eat, SCERRNAME[e] or "see src/player/scsi.i"))
end
if skw > 0 then
P(string.format(" the last seek waited %d polls for the channel "
.."to go quiet before it could start", skw))
end
goto ringdone
end
P(string.format("CHANNEL IDLE: %.1f ms over %d gaps (worst %.1f ms) = "
.."%.1f%% of the %.2f s the transport was needed for; "
.."busy %.1f ms",
xf.gap*1000, xf.ngap, xf.gapmax*1000,
span > 0 and 100*xf.gap/span or 0, span,
xf.busy*1000))
-- THE RIG'S OWN RESOLUTION, PRINTED RATHER THAN LEFT IMPLICIT.
-- The transport's timing is exact (memory taps), but the BYTES are
-- moved from the machine-frame notifier, so a record whose modelled
-- transfer is shorter than the wait for that notifier is acked later
-- than the model says it landed. Every millisecond here is a
-- millisecond the decoder may have waited that the medium would not
-- have made it wait.
if xf.ncopylate > 0 then
P(string.format("RIG RESOLUTION: %d of %d transfers were acked late "
.."because the bytes are moved on the notifier, by "
.."%.1f ms in total (mean %.1f ms) -- an artefact of "
.."this rig, not of the player",
xf.ncopylate, nissue, xf.copylate*1000,
xf.copylate*1000/xf.ncopylate))
end
if skw > 0 then
P(string.format(" the last seek waited %d polls for the channel "
.."to go quiet before it could start", skw))
end
::ringdone::
end
-- The decoder's own spin counter, kept for what it is: evidence that
-- stream.s outran the pipe, NOT evidence of an underrun. See the note
-- on `arrival` above.
@@ -395,6 +944,38 @@ SUB = emu.add_machine_frame_notifier(function()
-- opposite thing, so the two are never printed in the same words.
P(string.format("UNDERRUNS: %d/%d frames waited past their %d fps slot "
.."(%d polls)", stalls, META.nframes, META.fps, spins))
if SCSIX then
-- AND IT IS VACUOUS HERE, WHICH IS WORTH MORE THAN THE ZERO IS.
-- A synchronous transport cannot underrun by construction: the
-- decoder does not go on until the record is in its hand, because
-- it fetched the record itself. What it does instead is MISS THE
-- SLOT, and that is the NO IDLE line above -- 442 whole ticks of
-- overrun is not a late present, it is a player running at a
-- fraction of its frame rate. Quoting "0 underruns" from a
-- DLX_XFER=scsi run without this line would be the most flattering
-- possible reading of the least flattering result in the tree.
local ticks = pace + 1
P(string.format(" 0 IS VACUOUS: the transport is synchronous, so a "
.."frame cannot start before its record has landed "
.."-- the decoder IS the transport. The honest rate "
.."is %d frames in %d slots of %d fps = %.2f fps.",
META.nframes, ticks, META.fps,
ticks > 0 and META.fps*META.nframes/ticks or 0))
end
-- The other way a paced frame can go wrong, and it is not the same
-- failure. An UNDERRUN is the pipe: the record was not there. This
-- is the CPU: the record was there, the slot was already open, and
-- the decoder had no idle left in the frame before. Under a
-- host-written tick every slot is 83.33 ms; under the machine's own
-- clock they alternate 72.13 and 90.16 ms, 37.9% of them short, and
-- this is what the short ones cost (FINDINGS 54).
local late, latemax = SP:read_u32(LATEFR), SP:read_u32(LATEMAX)
local late1 = SP:read_u32(LATE1ST)
P(string.format("NO IDLE: %d/%d frames found their slot already open "
.."-- the frame before used all of it; worst overrun "
.."%d whole tick%s, first at frame %d", late,
META.nframes, latemax, latemax == 1 and "" or "s",
late1 == 0xFFFFFFFF and -1 or late1))
-- WHAT THE MINIMUM OVER A RUN IS, AND IS NOT. At release the ring
-- holds only what the prefill put there, so the early ticks report a
-- buffer that has not been built yet, not a ring or a pipe that
@@ -415,15 +996,22 @@ SUB = emu.add_machine_frame_notifier(function()
for _,e in ipairs(slack_series) do
if e[3] > 0 and e[2] < ss_min then ss_min, ss_at = e[2], e[1] end
end
if ss_min == math.huge then ss_min, ss_at = -1, -1 end
P(string.format("SEEK SLACK: ceiling %d frames (%.0f ms), first "
.."reached at tick %d; mean %.1f over the window",
ceiling, 1000*ceiling/META.fps, ceil_at,
sum_ahead/math.max(1,n_ahead)))
if n_ring > 0 then
-- WHOSE REFUSAL COUNT. With the ring owned by the machine the host
-- producer's counters are not merely stale, they are zero -- it never
-- placed anything -- so the classification has to come from
-- src/player/ring.i's own N_FULL. Reading n_ring here in a RINGOWN
-- run would have reported RATE-BOUND on every run by construction.
local ring_ref = RINGOWN and SP:read_u32(A_NFULL) or n_ring
if ring_ref > 0 then
P(string.format(" RING-BOUND: the ring filled (%d refusals). Once "
.."full it survives delivery stopped dead for %d "
.."frames = %.0f ms; min after first fill %d "
.."(tick %d)", n_ring, ceiling,
.."(tick %d)", ring_ref, ceiling,
1000*ceiling/META.fps, ss_min, ss_at))
else
P(string.format(" RATE-BOUND: the ring NEVER filled in %d frames. "
@@ -451,11 +1039,35 @@ SUB = emu.add_machine_frame_notifier(function()
end
-- The delivery result. Deadline for record i is release + i/fps.
-- Only this script can compute it, because only this script knows when
-- a byte landed -- and in a DLX_XFER=scsi run it does not. Printing it
-- from an empty arrival table would report "0/120 records late", which
-- is the most flattering possible reading of no data at all.
if SCSIX then
P("DEADLINE: not measured. The transport is in the machine and does "
.."not report arrival times, and MAME's SCSI is not transfer-timing "
.."accurate, so there is no honest deadline to grade against here. "
.."The sharp instrument that DOES survive is the decoder's own "
.."UNDERRUNS count above -- the machine takes it, and it is exact.")
st = "settle"; return
end
local misses, worst, prefill_s = 0, 0.0, 0.0
for i = 0, META.nframes-1 do
local a = arrival[i+1]
if a then
local late = a - (t_rel + i / META.fps)
-- INSTRUMENT NOTE, and it only bites under the self-clock.
-- tick_t[] is when this script OBSERVED the machine's tick, which
-- is the next machine-frame boundary after it -- up to 17.64 ms
-- late (54.5). So these deadlines are generous by that much and
-- this count UNDERSTATES lateness. The sharp instrument for the
-- same question is the decoder's own stall counter, which is exact
-- because the machine takes it: at 488 KB/s with a one-deep request
-- queue this reported 6 records late where the 68000 counted 59
-- frames that had to wait. Host-paced (DLX_PACE=1) the deadline is
-- a host model and is exact, so FINDINGS 49.6's table is unaffected.
local due = SELFCLK and tick_t[i+1] or (t_rel + i / META.fps)
if not due then due = t_rel + i / META.fps end
local late = a - due
if late > 0 then
misses = misses + 1
if late > worst then worst = late end
@@ -473,14 +1085,28 @@ SUB = emu.add_machine_frame_notifier(function()
.."at %s", prefill_s*1000, prefill_s*META.fps,
(KBPS > 0) and (prefill_s * BPS / 1024) or 0.0,
(KBPS > 0) and (KBPS.." KB/s") or "an unlimited pipe"))
M.video:snapshot()
P("snapshot taken after the sequential pass -- last frame, decoded "
.."entirely out of a "..RING_KB.." KB ring")
st = "snapped"; return
-- ONE FRAME OF SETTLE BEFORE THE CAPTURE, and it is not cosmetic.
-- MAME renders a screen scanline by scanline and the machine-frame
-- notifier fires at the END of that frame, so a bitmap for a frame in
-- which GVRAM changed holds some lines drawn before the change and some
-- after. Snapshotting it captures a TEAR -- and the symptom is a
-- pixel-exactness failure in the bottom blocks plus a broken double-scan
-- pairing, which reads exactly like a decoder bug and is not one. It
-- only bites when the decoder finishes its last frame LATE in a screen
-- frame, which is why it never showed up until a run with underruns in
-- it. Waiting one whole frame renders the finished picture with nothing
-- writing to GVRAM.
st = "settle"; return
end
if t > 900 then P("TIMEOUT flag="..string.format("%08X",fl)); M:exit() end
return
end
if st == "settle" then
M.video:snapshot()
P("snapshot taken after the sequential pass -- last frame, decoded "
.."entirely out of a "..RING_KB.." KB ring")
st = "snapped"; return
end
if st == "snapped" then M:exit(); return end
end)
if not ok then print("[STR] LUA ERROR: "..tostring(err)); M:exit() end
tools/bench/verify_adpcm.py
+140
View File
@@ -0,0 +1,140 @@
#!/usr/bin/env python3
"""Gate tools/encoder/adpcm.py against the only independent decoder on this
machine: ffmpeg's `adpcm_ima_oki`.
THIS FILE IS NOT ABOUT THE X68000's CHIP and after session 34 that distinction
is load-bearing. It checks this implementation against an independent one, so
its parameters stay ffmpeg's -- variant 'shift', high nibble first, a 12-bit
clamp, accumulator from 0. What the MACHINE's MSM6258 does is measured by
tools/bench/adpcm_run.sh and it is a different set of four values on all four
axes (adpcm.CHIP, FINDINGS 66). Do not "fix" the defaults here to match it: a
reference check whose reference has been adjusted to agree is not a check.
There is no ffmpeg ENCODER for this format -- `adpcm_ima_oki` is decode-only --
so the encoder here cannot be checked against a reference implementation. What
CAN be checked, and is, is that the decoder our encoder runs in its own loop is
byte-for-byte the decoder that ships in ffmpeg. An encoder that agrees with its
own wrong decoder is exactly the failure this catches.
Usage: verify_adpcm.py [wav_or_raw12 ...]
"""
import os, struct, subprocess, sys, random, math
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "encoder"))
import adpcm
TMP = "tmp/adpcm_gate"
# The step table as it is printed in the OKI datasheet and in every
# implementation of this format. adpcm.py BUILDS its table from 16*1.1**k; if
# the two ever disagree, one of them is a typo and this says which.
CANON = [16,17,19,21,23,25,28,31,34,37,41,45,50,55,60,66,73,80,88,97,107,118,
130,143,157,173,190,209,230,253,279,307,337,371,408,449,494,544,598,
658,724,796,876,963,1060,1166,1282,1411,1552]
fails = []
def ck(ok, msg):
print(("OK " if ok else "FAIL ") + msg)
if not ok: fails.append(msg)
def ffmpeg_decode(data, rate=15625):
"""Decode packed OKI ADPCM through ffmpeg, by wrapping it in a WAV whose
format tag is 0x0010 (WAVE_FORMAT_OKI_ADPCM). Returns 16-bit samples."""
os.makedirs(TMP, exist_ok=True)
fmt = struct.pack("<HHIIHHH", 0x0010, 1, rate, rate, 1, 4, 0)
body = (b"WAVE" + b"fmt " + struct.pack("<I", len(fmt)) + fmt
+ b"data" + struct.pack("<I", len(data)) + data)
w = f"{TMP}/probe.wav"
open(w, "wb").write(b"RIFF" + struct.pack("<I", len(body)) + body)
raw = subprocess.check_output(
["ffmpeg", "-v", "error", "-i", w, "-f", "s16le", "-acodec", "pcm_s16le", "-"])
return list(struct.unpack("<%dh" % (len(raw) // 2), raw))
def snr_db(ref, got):
"""Signal-to-noise over the 12-bit sample word."""
num = sum(float(s) * s for s in ref)
den = sum((float(a) - b) ** 2 for a, b in zip(ref, got))
if den == 0: return float("inf")
return 10.0 * math.log10(num / den) if num else float("-inf")
print("--- the step table ---")
ck(adpcm.STEP == CANON, f"49 entries, built = published (16*1.1**k), {adpcm.STEP[0]}..{adpcm.STEP[-1]}")
print("--- our decoder vs ffmpeg's adpcm_ima_oki ---")
random.seed(1234)
nibs = ([7]*12 + [i % 16 for i in range(256)]
+ [random.randrange(16) for _ in range(4000)])
data = adpcm.pack(nibs)
ff = ffmpeg_decode(data)
ours = [v * 16 for v in adpcm.decode(adpcm.unpack(data, len(nibs)), "shift")]
ck(len(ff) == len(ours), f"sample count {len(ff)} = {len(ours)}")
ck(ff == ours, f"variant 'shift' is SAMPLE-EXACT vs ffmpeg over {len(nibs)} nibbles")
# NEGATIVE CONTROL. A gate that passes whatever it is handed proves nothing;
# reading the nibbles the other way round has to go red, or "high nibble first"
# is an assertion rather than a measurement.
lowfirst = [adpcm.unpack(data, len(nibs))[i ^ 1] for i in range(len(nibs))]
bad = [v * 16 for v in adpcm.decode(lowfirst, "shift")]
ndiff = sum(1 for a, b in zip(ff, bad) if a != b)
ck(ndiff > 0, f"low-nibble-first DISAGREES on {ndiff}/{len(ff)} -- so what ffmpeg reads is measured, not assumed")
# AND IT IS A FACT ABOUT A FILE FORMAT, NOT ABOUT A CHIP. Session 33 recorded
# this line as "nibble order: HIGH FIRST, measured", which it is -- of the VOX
# convention ffmpeg implements. Session 34 asked the machine's own MSM6258 the
# same question through HD63450 channel 3 and got the OTHER answer: the chip
# takes the LOW nibble of a delivered byte first (FINDINGS 66), and encoding
# for the wrong one of the two costs -25.7 dB on the Singe window. The two
# claims do not conflict; they are about different things, and only one of them
# is about the machine this is being ported to.
print("--- and the second variant is not the same decoder ---")
terms = [v * 16 for v in adpcm.decode(adpcm.unpack(data, len(nibs)), "terms")]
d = [abs(a - b) // 16 for a, b in zip(ff, terms)]
nd = sum(1 for x in d if x)
ck(nd > 0, f"variant 'terms' differs on {nd}/{len(d)} samples, max {max(d)} in 12-bit units"
" -- and 'terms' is the one the machine runs (adpcm_run.sh, FINDINGS 66)")
print("--- and getting the variant wrong is NOT a rounding error ---")
# THE MEASUREMENT THAT CHANGED THIS FROM A FOOTNOTE INTO AN OPEN ITEM. The two
# variants differ by at most 3 in 12-bit units PER SAMPLE, which reads like
# something nobody could hear. ADPCM is RECURSIVE -- the delta is added to a
# running predictor and the nibble also moves the step index -- so the
# disagreement does not stay where it happens. It is the same shape as the
# codec's temporal recursion (64.1), one dimension down.
if os.path.exists("tmp/au_singe.raw"):
raw = open("tmp/au_singe.raw", "rb").read()
pcm = struct.unpack("<%dh" % (len(raw) // 2), raw)
ref = [max(-2048, min(2047, x >> 4)) for x in pcm]
nib = adpcm.encode(ref, "shift")
same, cross = adpcm.decode(nib, "shift"), adpcm.decode(nib, "terms")
err = [abs(x - y) for x, y in zip(same, cross)]
print(f" encoded 'shift', decoded 'shift': SNR {snr_db(ref, same):6.2f} dB")
print(f" encoded 'shift', decoded 'terms': SNR {snr_db(ref, cross):6.2f} dB"
f" <- the noise is LOUDER THAN THE SIGNAL")
print(f" per-sample disagreement over {len(err):,} samples: max {max(err)}, "
f"mean {sum(err)/len(err):.1f} in 12-bit units")
ck(snr_db(ref, cross) < 0,
"a 3-LSB formula disagreement costs ~25 dB, because ADPCM is RECURSIVE")
else:
print(" SKIPPED: no tmp/au_singe.raw (tools/encoder/extract_audio.py)")
print("--- the encoder, through the gated decoder ---")
for path in (sys.argv[1:] or []):
raw = open(path, "rb").read()
if raw[:4] == b"RIFF":
raw = subprocess.check_output(["ffmpeg", "-v", "error", "-i", path,
"-f", "s16le", "-ac", "1", "-ar", "15625", "-"])
pcm16 = struct.unpack("<%dh" % (len(raw) // 2), raw)
src = [max(-2048, min(2047, s >> 4)) for s in pcm16]
nib = adpcm.encode(src, "shift")
packed = adpcm.pack(nib)
rec_ff = [v // 16 for v in ffmpeg_decode(packed)][:len(src)]
rec_us = adpcm.decode(nib, "shift")
ck(rec_ff == rec_us,
f"{os.path.basename(path)}: encoder's own reconstruction = ffmpeg's, {len(src)} samples")
print(f" {len(src)} samples, {len(packed)} B, SNR {snr_db(src, rec_us):.2f} dB "
f"(12-bit word; the source is already quantised to it)")
print("ADPCM GATE " + ("GREEN" if not fails else f"RED: {len(fails)} failed"))
sys.exit(1 if fails else 0)
tools/bench/verify_adpcm_chip.py
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env python3
"""Read the MSM6258's own decoder out of a MAME capture. ROADMAP P6a.
FIVE THINGS ARE UNKNOWN, not the one FINDINGS 65 named:
feed which nibble(s) of a delivered BYTE the chip actually plays
variant 'shift' | 'terms' the delta formula (65.2, worth 25 dB)
bits 12 | 10 where the accumulator clamps
init 0 | -2 the accumulator at PLAY
(and the capture's own decimation, below)
`feed` is on the list because the run MEASURED it and it is not what anybody
assumed. Fed by HD63450 channel 3 in the IPL ROM's own configuration, this
machine plays ONE nibble per delivered byte -- the prologue is 16 zero nibbles
in 8 bytes and the accumulator climbs by 8 steps, not 16. A probe that had
assumed two would have found no model that fit and reported a broken rig. So
the hypothesis is enumerated with the others and the capture picks.
THE CAPTURE'S DECIMATION is enumerated for the same reason. MAME resamples the
chip's stream to the wav's rate, and a filtered 2x upsample is not recognisable
from a single sample -- on a slow ramp it looks like an exact repeat and on a
step it does not. So (factor, phase) is searched over {1x, 2x phase 0, 2x phase
1}, and the SCALE RESIDUAL is then checked on the decimated stream: MAME's
okim6258 puts `signal << 4` into a stream scaled to 32768 and the machine routes
it to the speaker at gain 0.50, so a chip sample is `signal * 8`. If the
winning decimation does not land within a couple of counts of a multiple of 8 on
every sample, it is not the chip's own stream and the run says so instead of
rounding to the nearest story.
WHAT THIS DOES NOT SETTLE. It measures MAME's device model driven through the
machine's real transport. It settles the RIG -- an emulated audio test encoded
against the wrong model is 25 dB of nothing -- and it leaves the silicon where
it was: needing a board or a datasheet.
"""
import json, os, struct, sys, wave
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "encoder"))
import adpcm
WAV = sys.argv[1] if len(sys.argv) > 1 else "tmp/adpcm.wav"
SEQ = sys.argv[2] if len(sys.argv) > 2 else "tmp/adpcm_seq.json"
SCALE = 8 # okim6258's <<4, times the machine's 0.50 speaker route
PRO_MAX = 40 # zero nibbles the prologue is allowed to have grown by
SKEW = 8 # samples of slack on where PLAY lands in the capture
DECIM = ((1, 0), (2, 0), (2, 1))
FEEDS = ("both-high-first", "both-low-first", "low-only", "high-only")
fails = []
def ck(ok, msg):
print(("OK " if ok else "FAIL ") + msg)
if not ok: fails.append(msg)
def nibbles_for(data, feed):
"""The nibble sequence the chip is hypothesised to have PLAYED, given the
bytes the channel delivered."""
if feed == "both-high-first":
return list(adpcm.unpack(data, None, "high"))
if feed == "both-low-first":
return list(adpcm.unpack(data, None, "low"))
if feed == "low-only":
return [b & 15 for b in data]
return [b >> 4 for b in data]
def main():
seq = json.load(open(SEQ))
pro, core = seq["pro"], seq["core"]
data = adpcm.pack([0] * pro + core, "high")
pro_bytes = pro // 2 # bytes of prologue, all $00
w = wave.open(WAV)
rate = w.getframerate()
n, ch = w.getnframes(), w.getnchannels()
s = struct.unpack("<%dh" % (n * ch), w.readframes(n))
left, right = list(s[0::ch]), list(s[1::ch])
ck(left == right, "both speakers carry the same samples (pan 00 = BOTH)")
ck(any(left), "the capture contains a signal at all")
if not any(left):
return 1
# ---- search: (decimation) x (feed) x (variant, bits, init) x (prologue)
results = {}
for fac, ph in DECIM:
rec = [round(v / SCALE) for v in left[ph::fac]]
nz = next((i for i, v in enumerate(rec) if v), None)
if nz is None:
continue
lo, hi = max(0, nz - SKEW), nz + 1
for feed in FEEDS:
base = nibbles_for(data, feed)
# a repeat of byte 0 costs whole nibbles under 'both' and one nibble
# under 'low-only'/'high-only'; either way it is zeros
for extra in range(PRO_MAX):
for variant in ("shift", "terms"):
for bits in (12, 10):
for init in (0, -2):
want = adpcm.decode([0] * extra + base, variant,
init=init, bits=bits)
for off in range(lo, hi):
if rec[off:off + len(want)] == want:
results.setdefault(
(feed, variant, bits, init),
(fac, ph, extra, off, len(want)))
print("--- candidates that reproduce the capture SAMPLE-EXACT ---")
for k, v in results.items():
print(f" feed={k[0]:<15s} variant={k[1]:<5s} bits={k[2]} init={k[3]:<2d}"
f" decimation {v[0]}x phase {v[1]}, prologue +{v[2]}, "
f"{v[4]:,} samples")
ck(len(results) == 1,
f"exactly one model reproduces the capture ({len(results)} did)")
if len(results) != 1:
return 1
model, (fac, ph, extra, off, ln) = next(iter(results.items()))
feed, variant, bits, init = model
# ---- the scale residual, on the stream the winner actually matched
seg = left[ph::fac][off:off + ln]
worst = max(abs(v - SCALE * round(v / SCALE)) for v in seg)
ck(worst <= 2,
f"every matched sample is within {worst} of a multiple of {SCALE} -- so "
f"`signal = round(sample/{SCALE})` is a recovery and not a rounding")
print("--- THE CHIP, AS THIS MACHINE MODELS IT ---")
print(f" nibbles played {feed}")
print(f" delta formula {variant}")
print(f" clamp {bits}-bit accumulator "
f"{adpcm.clamp_bounds(bits)}")
print(f" accumulator at PLAY {init}")
print(f" matched {ln:,} consecutive samples, "
f"capture decimated {fac}x at phase {ph}")
print(f" chip stream rate = {rate}/{fac} = {rate/fac:,.1f} Hz, and the "
f"channel delivered {len(data):,} B")
# ---- THE NEGATIVE CONTROLS. Flip one axis alone; the match must die.
# Without these an axis the probe is BLIND to reads exactly like an axis it
# has settled, which is 58.3's vacuous-counter trap in a new place.
print("--- and every axis was actually asked (flip one, the match dies) ---")
rec = [round(v / SCALE) for v in left[ph::fac]]
flips = {"feed": [f for f in FEEDS if f != feed],
"formula": ["terms" if variant == "shift" else "shift"],
"clamp": [12 if bits == 10 else 10],
"init": [0 if init == -2 else -2]}
for name, alts in flips.items():
worst_axis = None
for a in alts:
m = dict(feed=feed, variant=variant, bits=bits, init=init)
m[{"feed": "feed", "formula": "variant",
"clamp": "bits", "init": "init"}[name]] = a
want = adpcm.decode([0] * extra + nibbles_for(data, m["feed"]),
m["variant"], init=m["init"], bits=m["bits"])
got = rec[off:off + len(want)]
d = sum(1 for x, y in zip(got, want) if x != y)
worst_axis = d if worst_axis is None else min(worst_axis, d)
ck(worst_axis > 0,
f"{name:8s} flipped: the closest alternative still disagrees on "
f"{worst_axis:,} of {ln:,} samples")
print("--- against what tools/encoder/adpcm.py DEFAULTS to ---")
cur = {"feed": "both-high-first", "formula": "shift", "clamp": 12, "init": 0}
got = {"feed": feed, "formula": variant, "clamp": bits, "init": init}
for k in cur:
print(f" {k:8s} encoder {str(cur[k]):<15s} chip {str(got[k]):<15s}"
f" {'agree' if cur[k] == got[k] else 'DISAGREE'}")
json.dump({"feed": feed, "variant": variant, "bits": bits, "init": init,
"decimation": fac, "phase": ph, "extra": extra,
"matched": ln, "chip_rate": rate / fac},
open("tmp/adpcm_model.json", "w"))
print("ADPCM CHIP GATE " + ("GREEN" if not fails else f"RED: {len(fails)} failed"))
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main())
tools/bench/verify_load.py
+73
View File
@@ -0,0 +1,73 @@
#!/usr/bin/env python3
"""Check the 68000's load-time output against tools/bench/dlxload.py, byte for byte.
python3 tools/bench/verify_load.py <in.dlx> [--out tmp/load]
The 68000 ran src/player/load.i over the RAW container header; tools/bench/
load.lua read the results back out of emulated RAM and out of the PALETTE
REGISTERS. This compares them with what the host-side transforms produce.
Byte-for-byte and not "close enough", for both halves:
* the codebooks are indices, so a single wrong byte is a wrong COLOUR in
every block that uses that codeword, in every frame of the scene.
* the palette words carry the shared LSB the encoder's 1.96 dB (FINDINGS
23.3) depends on, and a wrong choice of it is invisible in a diff of the
picture's SHAPE -- it is a slightly wrong colour, which is exactly the sort
of thing that gets attributed to the codec.
The darkest-entry index is checked too: it is what the letterbox is filled
with until the encoder reserves a black entry (23.4, still open), and it comes
out of an argmin whose tie-break has to match numpy's -- first index wins.
"""
import sys, argparse
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, "tools/bench")
from dlx import DLX
import dlxload as DL
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--out", default="tmp/load")
ap.add_argument("--log", default="tmp/load_check.log",
help="the rig's log, for the DARK= line it printed")
a = ap.parse_args()
d = DLX(a.container)
cb1, cb4 = DL.expand_codebooks(d)
palb, dark, _ = DL.pack_palette(d)
want = cb1.tobytes() + cb4.tobytes() + palb.tobytes()
got = open(a.out + "_out.bin", "rb").read()
if len(got) != len(want):
sys.exit(f"FAIL: the 68000 produced {len(got)} B, expected {len(want)}")
n1, n4 = cb1.nbytes, cb4.nbytes
sections = (("CB1", 0, n1), ("CB4", n1, n1 + n4), ("palette", n1 + n4, len(want)))
bad = 0
for name, lo, hi in sections:
diff = [i for i in range(lo, hi) if got[i] != want[i]]
if diff:
bad += len(diff)
i = diff[0]
print(f"FAIL: {name}: {len(diff)}/{hi-lo} bytes differ; first at "
f"+{i-lo} (68000 {got[i]:#04x}, dlxload {want[i]:#04x})")
else:
print(f" OK {name}: {hi-lo} B identical to dlxload.py")
# The rig prints the index the 68000 chose; parse it rather than re-deriving,
# so a rig that failed to read LDARK cannot pass by silence.
got_dark = None
for line in open(a.log, "rb").read().decode("utf-8", "replace").splitlines():
if "DARK=" in line:
got_dark = int(line.split("DARK=")[1].split()[0].rstrip(","))
if got_dark is None:
sys.exit("FAIL: the rig printed no DARK= line -- it did not reach the dump")
if got_dark != dark:
sys.exit(f"FAIL: darkest palette entry: 68000 says {got_dark}, dlxload says {dark}")
print(f" OK darkest entry {dark}, chosen by the same argmin tie-break")
if bad:
sys.exit(f"FAIL: {bad} bytes differ in total")
print(f"OK the 68000 reproduced all {len(want)} B of load-time output exactly "
f"(P1 codebooks, P2 palette)")
tools/bench/verify_packed.py
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Is EVERY frame the packed player put on screen pixel-exact? ROADMAP K3.
python3 tools/bench/verify_packed.py <in.dlxp> [--snap tmp/snap_packed]
[--map tmp/packed_snaps.csv]
[--min-frames N]
WHY THIS CHECKS ALL OF THEM AND tools/bench/verify_decode.py CHECKS ONE. The
codec is temporally recursive: a SKIP block is a claim that the previous frame is
still in GVRAM, so the last frame of a sequential run is only correct if every
frame before it was, and one comparison audits 120. A packed frame is a
LITERAL -- 192 rows of picture and a whole new palette, written over whatever
was there. Frame 119 being right says nothing at all about frame 60. The
simplification that deleted the ring, the codebooks and the decoder also deleted
the gate's free lunch, and this is the bill.
WHAT IS COMPARED. MAME's own screen, through MAME's own video code: the
snapshot is what the display produced out of GVRAM and the palette REGISTERS.
Nothing here re-implements the packed interleave -- that is deliberate and it is
the same rule tools/bench/gvpack/verify_dlxp.py was built on, because a
container round-trips against its own inverse whether or not its byte order is
the one the hardware wants. The reference is dlxp.render(i), which is the
palette in the record applied to the indices in the record.
THE LETTERBOX IS CHECKED TOO, and it is not padding. The picture is 192 rows of
a 256-row screen; the other 64 rows are STATIC SETUP the 68000 wrote once at
scene start (packed.s pg_static) and the channel never touches again. If they
were wrong -- or if they decayed as the per-frame palette moved under them --
the picture would still be pixel-exact and the screen would not be. Index 255
is black in every frame's palette by construction (vq.frame_palette), so this
also gates that reservation across all 120 records.
"""
import argparse, csv, os, sys
sys.path.insert(0, "tools/encoder")
import numpy as np
from PIL import Image
from dlxp import DLXP
ap = argparse.ArgumentParser()
ap.add_argument("container")
ap.add_argument("--snap", default="tmp/snap_packed")
ap.add_argument("--map", default="tmp/packed_snaps.csv")
ap.add_argument("--min-frames", type=int, default=1,
help="fail if fewer than this many frames were sampled -- a "
"run that displayed nothing must not pass as a run with "
"no mismatches in it")
a = ap.parse_args()
d = DLXP(a.container)
if not d.has_palette:
# A --no-palette container leaves the palette registers holding whatever the
# scene setup put there, and this rig's player writes none -- so there is no
# reference for what the screen should show. Say so rather than compare
# against an assumption.
sys.exit(f"{a.container} carries no palette; this gate has no reference "
f"for what the display should have produced.")
with open(a.map) as fh:
pairs = [(r["snapshot"], int(r["frame"])) for r in csv.DictReader(fh)]
if len(pairs) < a.min_frames:
print(f"FAIL 0. only {len(pairs)} frames were sampled, --min-frames is "
f"{a.min_frames}. A player whose write window never closed displays "
f"nothing, and an empty comparison is not a pass.")
sys.exit(1)
SCRH, SCRW = 256, 256
YOFF = (SCRH - d.H) // 2
fails, checked = [], 0
for name, fr in pairs:
path = f"{a.snap}/x68000/{name}.png"
if not os.path.exists(path):
fails.append(f"snapshot {name} (frame {fr}) is missing from {a.snap}")
continue
s = np.asarray(Image.open(path).convert("RGB")).astype(int)
if s.shape[:2] != (2 * SCRH, SCRW):
fails.append(f"frame {fr}: geometry {s.shape[1]}x{s.shape[0]}, "
f"expected {SCRW}x{2*SCRH}")
continue
if not all(np.array_equal(s[i], s[i + 1]) for i in range(1, s.shape[0] - 1, 2)):
fails.append(f"frame {fr}: double-scan pairing (1,2),(3,4),... broken")
continue
g = s[0::2]
pal = d.palette_rgb(fr)
exp = np.empty((SCRH, SCRW, 3), int)
exp[:] = pal[255] # the letterbox, and the reservation
exp[YOFF:YOFF + d.H] = d.render(fr)
checked += 1
if np.array_equal(g, exp):
continue
bad = (g != exp).any(2)
by, bx = np.where(bad)
inpic = ((by >= YOFF) & (by < YOFF + d.H)).sum()
fails.append(f"frame {fr} (snapshot {name}): {bad.sum()} px differ "
f"({inpic} in the picture, {bad.sum()-inpic} in the "
f"letterbox), first at y={by[0]} x={bx[0]}, maxdiff "
f"{abs(g-exp).max()}")
for f in fails[:12]:
print("FAIL " + f)
if len(fails) > 12:
print(f"FAIL ... and {len(fails)-12} more")
if fails:
print(f" {checked-len([f for f in fails])} of {len(pairs)} sampled "
f"frames compared clean")
sys.exit(1)
lo, hi = min(f for _, f in pairs), max(f for _, f in pairs)
print(f"OK {checked} frames of {a.container} pixel-exact on the emulated "
f"68000, frames {lo}..{hi} of {d.nframes}")
print(f" every one of them a LITERAL: no decoder, no codebook, no ring. "
f"Screen {SCRW}x{SCRH}, picture {d.W}x{d.H} at y={YOFF}, letterbox on "
f"the reserved index 255.")
print(f" palette {'LAST' if d.palette_last else 'FIRST'} in the record, "
f"{d.pal_bytes} B, compared as the DISPLAY renders it (GRB555+I out of "
f"the palette registers)")
tools/bench/verify_packed_audio.py
+598
View File
@@ -0,0 +1,598 @@
#!/usr/bin/env python3
"""The container's own audio, READ BACK OFF THE SPEAKER. ROADMAP P6c.
python3 tools/bench/verify_packed_audio.py <in.dlxp> <capture.wav> \
[counters.json] [--seek FRAME --iters N]
WHY THIS READS THE CAPTURE AND NOT THE PLAYER'S COUNTERS. src/player/packed.s
reports how many lumps it armed and how many payload bytes it handed the chip,
and every one of those numbers can be right while the sound is wrong. Nothing
in this format parses anything (FINDINGS 67.4): a lump fetched one sector out is
not an error, it is 7,168 B of noise played at the right length; a payload one
byte long is not an error either, it is a rate. The only instrument that can
tell those apart from a correct run is the stream the chip actually produced.
WHAT IS CHECKED, and it is the whole scene rather than a sample of it:
1. every lump's payload, decoded with the FOUR AXES OUT OF THE CONTAINER'S OWN
HEADER (FINDINGS 66/67.3), appears in the capture SAMPLE-EXACT and in
order. Not "close": the recursion is exact arithmetic and MAME's okim6258
puts `signal << 4` into a stream the machine routes to the speaker at gain
0.50, so a chip sample is `signal * 8` and recovering it is a division and
not a rounding. The residual is asserted.
2. the PAYLOAD lengths are the accumulator's and not the lump's. A player
that fed the chip the whole A*512 B lump runs 0.09% fast -- 1.25 s of
lip-sync over the game (67.2) -- and the difference between the two is 6.54
B a group, which is 13 samples. So this is checked by LENGTH: lump k's run
of matched samples must be exactly 2*lump_bytes(k), and that alternates
14,322 / 14,324 rather than being 14,336 every time.
3. THE SEAMS, measured rather than assumed. Channel 3 counts out at the end
of a lump and the chip has no FIFO and no starvation state -- it goes on
decoding whatever byte its data register still holds, alternating that
byte's low and high nibbles, until the CPU arms the next lump. Those
samples are NOT silence, they are the recursion running on a repeated byte,
and the state they leave behind is what lump k+1 decodes from. So the
search below carries the state across the seam and reports its LENGTH,
which is the audible cost of every design decision on the video path.
4. THE SEEK, when the run made one (--seek). A branch is a frame index and
a DLXP2 group puts lump k in FRONT of its records, so the player has to
issue a second read and enter the lump `f mod F` frames in (FINDINGS 70.3).
The stream the chip should then have been fed is the container's own bytes
SPLICED -- everything, then everything from byte floor(f*hz/(2*fps)) on --
and the walk above accounts for it as one continuous run, which is the
check: a player that dropped the byte offset feeds a stream that starts up
to F frames early and NOTHING ABOUT IT IS AN ERROR. It is a rate.
AND THE THING THE PLAYER CANNOT SEE. An MSM6258 has no seek: its
accumulator and step index are the product of every nibble since PLAY, so a
seek hands it bytes the encoder chose for a state it is not in. The
samples are then wrong while the recursion re-converges, and they are wrong
WITHOUT ANY BYTE BEING WRONG -- every counter in the player stays right and
the walk above still passes, because the walk tracks the chip rather than
the intent. This measures it: the chip's own samples across the splice
against the samples the ENCODER meant, which are the same bytes decoded
from the state a continuous play would have been in.
THE NEGATIVE CONTROL IS BUILT IN. A seam is found by searching for the repeat
count that makes the next lump match; if the player had fed the wrong bytes, no
repeat count would make it match and the run fails rather than sliding.
"""
import json, os, struct, sys, wave
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)),
"..", "encoder"))
import adpcm
from dlxp import DLXP, lump_bytes as _lump_bytes
def dlxp_lump_bytes(d, k):
"""What the CADENCE gives lump k, before the stream's end
truncates it -- so a lump that is short because the scene ran
out can be told from one that is short because the remainder
arithmetic said so."""
return _lump_bytes(k, d.cad_f, d.fps, d.aud_hz)
SCALE = 8 # okim6258's `signal << 4` into a 32768 stream, times the
# machine's 0.50 speaker route. verify_adpcm_chip.py's.
RESID = 2 # counts of slack on that recovery, as 66 measured it
LOOK = 32 # samples of continuation a candidate run has to survive
MAXRUN = 60000 # nibbles one delivered byte may be stretched over: 3.8 s
# at 15,625 Hz, far past any seam a working player makes.
# A bound is what makes a failure say "not found" rather
# than run until the host is bored.
LEAD = 400000 # samples of silence before PLAY
def stepper(dec):
"""One sample of the recursion, as a closure over the container's own four
axes. adpcm.decode_state is the same arithmetic and is what the whole-lump
paths use; this exists because the walk below needs it ONE NIBBLE AT A TIME
and a function call per sample over 78,125 bytes is the difference between
a gate that runs in seconds and one that does not."""
lo, hi = adpcm.clamp_bounds(dec["bits"])
variant, step, adj = dec["variant"], adpcm.STEP, adpcm.INDEX_ADJUST
def one(st, n):
sig, idx = st
sig += adpcm.delta(n, step[idx], variant)
sig = lo if sig < lo else (hi if sig > hi else sig)
idx += adj[n & 7]
idx = 0 if idx < 0 else (48 if idx > 48 else idx)
return sig, (sig, idx)
return one
def walk(rec, pos, st, data, dec):
"""Account for EVERY byte the player handed the chip, in order.
WHY A WALK AND NOT A COMPARISON. A whole-stream `decode(data) == capture`
is the check this wanted to be and it does not survive contact with the
machine. The MSM6258 has no FIFO and no handshake at all: the DMA channel
writes a byte into the data register whenever #DRQ3 asks, and the chip
decodes a nibble out of whatever is in there on every sample tick. Those
are two clocks -- 7,812.5 B/s and 15,625 Hz -- and MAME's okim6258 data_w
RESETS the nibble select on every write. So a byte is normally played as
two nibbles, and near a boundary it can be played as one (the high nibble
dropped) or as three or more (the low/high pair repeated) purely from where
the write lands inside a sound-stream slice.
THE MODEL IS THEREFORE ONE LINE: byte b was played as `c` nibbles taken from
the cycle (b&15, b>>4), c >= 1. This walk finds `c` for every byte, and the
HISTOGRAM of c is the result -- c=2 everywhere is a chip being fed exactly
at its own rate, and every c>2 is the chip replaying a byte while the 68000
was somewhere else, which is what a SEAM is.
It is not a loose check. Every one of the `c` samples has to be exactly
right, the run has to be followed by LOOK samples that are exactly right,
and a byte the player never sent leaves no c at all. Returns
(fail_index, pos, runs); fail_index is None on success.
"""
one = stepper(dec)
runs = []
n = len(data)
pos_ = pos
for i in range(n):
b = data[i]
pair = (b & 15, b >> 4)
nxt = data[i + 1] & 15 if i + 1 < n else None
# DEEP FIRST, THEN SHALLOW. LOOK samples of continuation is what tells
# a real seam from a coincidence -- a repeated pair can agree with the
# next lump's first nibble for one sample and does not for thirty-two.
# But the window is 16 bytes wide and a SECOND stretched byte inside it
# rejects the true answer as well as the false ones, so a byte that no
# candidate survives is retried with a shallow window rather than
# failing -- and the retry window is TWO samples, the next byte's own
# pair, because a four-sample one reaches into the stretched byte
# itself. The cost of resolving it wrong is a failed walk, not a pass.
got = _pick(rec, pos_, st, pair, nxt, data, i, one, LOOK)
if got is None:
got = _pick(rec, pos_, st, pair, nxt, data, i, one, 2)
if got is None:
return i, pos_, runs
c, st = got
pos_ += c
runs.append(c)
return None, pos_, runs
def _pick(rec, pos, st, pair, nxt, data, i, one, look):
"""The run length for one byte: every candidate `c` whose samples are exact
and whose continuation survives `look`, resolved to c=2 where c=2 is one of
them. Two nibbles a byte is what the two clocks agree on; anything else is
an event and an event needs the evidence, which is what `look` is."""
cands = []
s2, c = st, 0
while c < MAXRUN and pos + c < len(rec):
smp, s3 = one(s2, pair[c & 1])
if rec[pos + c] != smp:
break
c += 1
s2 = s3
if nxt is None:
cands.append((c, s2))
break
smp2, _ = one(s2, nxt)
if pos + c < len(rec) and rec[pos + c] == smp2 \
and _look(rec, pos + c, s2, data, i + 1, one, look):
cands.append((c, s2))
if c > 2 and len(cands) >= 2:
break
if not cands:
return None
for cc in cands:
if cc[0] == 2:
return cc
return cands[0]
def _look(rec, pos, st, data, i, one, look=LOOK):
"""`look` samples of continuation, assuming two nibbles a byte from here.
This is what tells a real seam from a coincidence. At a lump boundary the
repeated pair can happen to agree with the next lump's first nibble for one
sample; it does not go on agreeing for thirty-two.
"""
n, k = len(data), 0
while k < look and i < n:
b = data[i]
for nib in (b & 15, b >> 4):
if pos >= len(rec):
return True
smp, st = one(st, nib)
if rec[pos] != smp:
return False
pos += 1
k += 1
i += 1
return True
def stream_pos(d, frame):
"""The byte of the ADPCM stream frame `frame` starts at. Exact, and the
exactness is the same one FINDINGS 67.2 is about: 15,625 samples a second
over 24 half-frames does not divide, and a seek that rounded it would be a
rate error rather than a byte error."""
return frame * d.aud_hz // (2 * d.fps)
def spliced(d, seek, iters):
"""The stream the player should have fed, as (label, bytes) chunks.
One pass is the container's lumps in order. A pass after a SEEK starts at
stream byte stream_pos(seek): the tail of lump k = seek//F, then every lump
after it. The chunks are kept separate because their BOUNDARIES are what
the stretched-byte assertion is measured against -- the chip replays its
last byte at a lump end and at a seek, and nowhere else.
"""
out = [(f"lump {k}", d.lump(k)) for k in range(d.n_lumps)]
if iters <= 1 or seek is None:
return out
b0 = stream_pos(d, seek)
for it in range(1, iters):
off = 0
for k in range(d.n_lumps):
lb = d.lump(k)
if off + len(lb) <= b0:
off += len(lb)
continue
cut = max(0, b0 - off)
out.append((f"pass {it+1} lump {k}" + (f" +{cut} B" if cut else ""),
lb[cut:]))
off += len(lb)
return out
def walk_reset(rec, start, data, dec, sp):
"""The walk, SPLIT AT A CHIP RESET.
A STOP/PLAY at the branch is a discontinuity the continuous model cannot
cross: the accumulator, the step index and the nibble select all go back to
the container's own start state, so no repeat count of the last byte of
pass 1 can be followed by pass 2's first sample. The walk would report
"byte 78,124 does not account for capture sample N", which is true and is
the wrong complaint.
So pass 1 is walked without its final byte, the RESTART is searched for --
the first capture position at which pass 2 walks cleanly from `init` -- and
pass 2 is walked from there. The final byte's run length is then the gap,
which is exactly what it is: the chip replaying it while the 68000 fetched
a lump and stopped the chip.
The search is cheap because the first post-reset sample is DETERMINED: one
nibble of a known byte from a known state. Only positions carrying that
value are tried at all.
"""
one = stepper(dec)
bad, pos, runs = walk(rec, start, (dec["init"], 0), data[:sp - 1], dec)
if bad is not None:
return bad, pos, runs, None
first, _ = one((dec["init"], 0), data[sp] & 15)
K = 64 # bytes of pass 2 a candidate must
for p in range(pos, min(pos + MAXRUN, len(rec))): # survive before the
if rec[p] != first: # full walk is run
continue
b2, _, _ = walk(rec, p, (dec["init"], 0), data[sp:sp + K], dec)
if b2 is None:
b3, pos3, runs3 = walk(rec, p, (dec["init"], 0), data[sp:], dec)
if b3 is None:
return None, pos3, runs + [p - pos] + runs3, p
return sp - 1, pos, runs, None
def main():
argv = [a for a in sys.argv[1:]]
seek, iters = None, 1
while "--seek" in argv:
i = argv.index("--seek"); seek = int(argv[i+1]); del argv[i:i+2]
while "--iters" in argv:
i = argv.index("--iters"); iters = int(argv[i+1]); del argv[i:i+2]
if len(argv) < 2:
sys.exit(__doc__)
sys.argv = [sys.argv[0]] + argv
d = DLXP(sys.argv[1])
counters = json.load(open(sys.argv[3])) if len(sys.argv) > 3 else None
if not d.has_audio:
sys.exit("this container is silent -- there is nothing to have heard")
dec = d.decoder()
fails = []
def ck(ok, msg):
print(("OK " if ok else "FAIL ") + msg)
if not ok:
fails.append(msg)
w = wave.open(sys.argv[2])
n, ch, rate = w.getnframes(), w.getnchannels(), w.getframerate()
s = struct.unpack("<%dh" % (n * ch), w.readframes(n))
left, right = s[0::ch], s[1::ch]
ck(rate == d.aud_hz,
f"the capture is at {rate:,} Hz and the chip's stream is {d.aud_hz:,} -- "
f"equal rates are what keep MAME's resampler out of the measurement")
ck(list(left) == list(right), "both speakers carry the same samples (pan 00)")
ck(any(left), "the capture contains a signal at all")
if not any(left) or rate != d.aud_hz:
return 1
rec = [round(v / SCALE) for v in left]
start = next(i for i, v in enumerate(rec) if v)
ck(start < LEAD, f"the chip starts playing {start/rate:.2f} s in")
print(f"--- {sys.argv[1]}: {d.n_lumps} lumps, decoder {dec['variant']}/"
f"{dec['order']}, {dec['bits']}-bit clamp, accumulator {dec['init']} "
f"at PLAY -- ALL FOUR out of the header (67.3)")
# ---- THE STREAM, ACCOUNTED FOR BYTE BY BYTE. d.audio() is the container's
# own lumps reassembled BY PAYLOAD -- 67.2's accumulator, not the padded
# sector runs -- so a player that fed the chip whole lumps does not merely
# score worse here, it fails to walk: the 6.54 B of zero at the end of a
# lump are nibbles that are not in this stream.
chunks = spliced(d, seek, iters)
data = b"".join(c for _, c in chunks)
if seek is not None and iters > 1:
print(f"--- THE SEEK: {iters} passes, passes 2..{iters} start at frame "
f"{seek} = stream byte {stream_pos(d, seek):,}, which is "
f"{seek % d.cad_f} frame(s) into lump {seek // d.cad_f} "
f"(FINDINGS 70.3). The stream below is SPLICED and is "
f"{len(data):,} B against the container's {d.aud_bytes:,}.")
replay = bool(counters and counters.get("arst"))
if replay and seek is not None and iters > 1:
sp0 = sum(len(c) for _, c in chunks[:d.n_lumps])
bad, pos, runs, rp = walk_reset(rec, start, data, dec, sp0)
if bad is None:
print(f" the chip was STOPPED and re-PLAYED at the branch: the "
f"walk is SPLIT there, and pass 2 restarts at capture sample "
f"{rp:,}, {runs[sp0-1]:,} samples after the last byte of pass "
f"1 was handed over ({runs[sp0-1]/rate*1000:.0f} ms of "
f"replayed byte and stopped chip)")
else:
bad, pos, runs = walk(rec, start, (dec["init"], 0), data, dec)
ck(bad is None,
f"all {len(data):,} bytes of the container's audio reached the chip, in "
f"order, and every sample the chip produced from them is exact"
+ ("" if bad is None else f" -- byte {bad:,} of {len(data):,} does not "
f"account for capture sample {pos:,}"))
if bad is not None:
k = bad * 2 * d.fps // (d.cad_f * d.aud_hz)
print(f" that is inside lump {k}, {bad - sum(len(d.lump(j)) for j in range(k)):,} B in")
return 1
matched = pos - start
worst = max(abs(v - SCALE * round(v / SCALE))
for v in left[start:start + matched])
ck(worst <= RESID,
f"every one of {matched:,} matched samples is within {worst} of a "
f"multiple of {SCALE} -- so `signal = sample/{SCALE}` recovers the chip's "
f"own stream rather than rounding to the nearest story")
# ---- 1. THE FEED. c=2 is a chip being fed at exactly its own rate.
hist = {}
for c in runs:
hist[c] = hist.get(c, 0) + 1
two = hist.get(2, 0)
ck(two * 1000 >= len(runs) * 999,
f"{two:,} of {len(runs):,} bytes ({two*100/len(runs):.3f}%) were played "
f"as exactly two nibbles -- the chip was paced by its own #DRQ3 and not "
f"by the CPU")
print(f" nibbles per delivered byte: "
+ ", ".join(f"{c}x{v:,}" for c, v in sorted(hist.items())))
# ---- 2. THE SEAMS, which is what every c > 2 is. A lump's channel counts
# out and the chip goes on replaying the last byte until the CPU arms the
# next one; the excess nibbles ARE that interval, measured in the only place
# it exists, which is the sound.
seams = [(i, c - 2) for i, c in enumerate(runs) if c > 2]
print(f"--- THE SEAMS: {len(seams)} byte(s) were stretched, out of "
f"{d.n_lumps - 1} lump boundaries")
if seams:
ex = sum(c for _, c in seams)
print(f" worst {max(c for _, c in seams)} samples = "
f"{max(c for _, c in seams)/rate*1000:.2f} ms; total {ex} samples "
f"= {ex/rate*1000:.2f} ms of replayed byte over "
f"{matched/rate:.2f} s of audio ({ex*100/matched:.4f}%)")
# and every stretched byte must BE a lump boundary -- a stretch anywhere
# else is the CPU losing the chip in the middle of a buffer.
ends = set()
off = 0
for _, c in chunks:
off += len(c)
ends.add(off - 1)
stray = [i for i, _ in seams if i not in ends]
ck(not stray,
f"every stretched byte is the LAST byte of a lump ({len(stray)} were not)"
+ ("" if not stray else f" -- first at byte {stray[0]:,}, which is the "
f"chip running dry in the middle of a buffer"))
# ---- 3. THE PAYLOAD IS THE ACCUMULATOR'S (FINDINGS 67.2). The walk
# already proves it -- a whole-lump player's stream contains the padding and
# would not walk -- so what is left is to price what was avoided. The LAST
# lump is left out: 120 frames is not a multiple of F=11, so it carries ten
# frames of audio and is short for an arithmetic reason and not a rate one.
full = [k for k in range(d.n_lumps)
if len(d.lump(k)) == dlxp_lump_bytes(d, k)]
ck(len(full) >= d.n_lumps - 1,
f"{len(full)} of {d.n_lumps} lumps carry a whole group of audio")
if len(full) > 1:
pad = d.cad_a * 512
got = sum(len(d.lump(k)) for k in full)
over = pad * len(full) - got
secs = got * 2 / rate
print(f"--- THE PADDING IS DRIFT (FINDINGS 67.2), over the {len(full)} "
f"lumps that carry a whole group")
print(f" payload {got:,} B against {pad*len(full):,} B of lump "
f"space: {over:,} B more, {over*100/got:.3f}%, "
f"{over*2/rate*1000:.2f} ms over {secs:.2f} s of audio")
print(f" -> {over*2/rate/secs*22.8*60:.2f} s of lip-sync over the "
f"game's 22.8 min, and the accumulator in pg_apay is the three "
f"lines that do not spend it")
# ---- 4. WHAT A SEEK COSTS THE CHIP, and it is the thing no counter in the
# player can reach. The walk above proves every BYTE arrived; this asks
# whether the SAMPLES the chip made out of them are the ones the encoder
# meant. They are not, and they cannot be: the MSM6258's accumulator is a
# pure integrator with NO LEAKAGE TERM, so a state mismatch at a branch is a
# DC offset that does not decay -- it is not a transient with a time
# constant, and calling it one would be the flattering reading.
#
# THE REFERENCE IS THE ENCODER'S OWN STATE, not a fresh one. adpcm.py
# encoded the stream in one pass, so the state it chose byte B(f)'s nibbles
# for is the state a CONTINUOUS play reaches at B(f). The FRESH control is
# the other design -- STOP the chip and PLAY it again at the branch, which
# is what DLX_PK_ARST does -- and the two numbers are what choose between
# them. Under --replay the fresh series is not a control at all: it is the
# prediction, and it has to be sample-exact.
if seek is not None and iters > 1 and bad is None:
one = stepper(dec)
sp = sum(len(c) for _, c in chunks[:d.n_lumps])
b0 = stream_pos(d, seek)
whole = d.audio()
def run_from(st, i0, nmax):
"""The samples the chip WOULD have made from byte i0 on, had it been
in state `st` -- with the run lengths it actually used, so the two
series are sample-aligned across a seam as well as a byte."""
out, i = [], i0
while i < len(data) and len(out) < nmax:
b, c = data[i], runs[i]
for j in range(c):
smp, st = one(st, (b & 15) if j % 2 == 0 else (b >> 4))
out.append(smp)
i += 1
return out
if replay:
st = (dec["init"], 0) # by construction: PLAY sets
else: # both, and the run above
st = (dec["init"], 0) # proved it sample-exact
for i in range(sp): # ...otherwise the chip is
b, c = data[i], runs[i] # wherever the previous
for j in range(c): # scene's audio left it,
_, st = one(st, (b & 15) if j % 2 == 0 else (b >> 4))
pos0 = start + sum(runs[:sp])
N = min(4 * rate, len(rec) - pos0)
ref = (dec["init"], 0) # ...and the ENCODER's state
for by in whole[:b0]: # at the SAME stream byte,
for nib in (by & 15, by >> 4): # reached continuously
_, ref = one(ref, nib)
got = rec[pos0:pos0 + N]
want = run_from(ref, sp, N)
fresh = run_from((dec["init"], 0), sp, N)
n = min(len(got), len(want), len(fresh))
e = [got[i] - want[i] for i in range(n)]
def band(lo, hi):
seg = e[lo:min(hi, n)]
if not seg:
return None
m = sum(seg) / len(seg)
ac = (sum((x - m) ** 2 for x in seg) / len(seg)) ** 0.5
return m, ac, max(abs(x) for x in seg)
lo_c, hi_c = adpcm.clamp_bounds(dec["bits"])
print(f"--- THE PREDICTOR DOES NOT SEEK (FINDINGS 71). The chip's state "
f"at the branch is accumulator {st[0]}, step index {st[1]}; the "
f"encoder chose byte {b0:,}'s nibbles for accumulator {ref[0]}, "
f"step index {ref[1]}.")
print(f" error against the encoder's intent, DC and AC separately "
f"(full scale is {hi_c}):")
for lo, hi, lab in [(0, 100, "0-6 ms"), (100, 1000, "6-64 ms"),
(1000, 5000, "64-320 ms"), (5000, rate, "0.3-1.0 s"),
(rate, 2*rate, "1-2 s"), (2*rate, 4*rate, "2-4 s")]:
r = band(lo, hi)
if r:
print(f" {lab:>10} DC {r[0]:8.1f} AC {r[1]:7.2f} "
f"|max| {r[2]}")
if replay:
# THE STRONGEST FORM THIS CAN TAKE. A STOP/PLAY puts the chip in a
# state this script knows exactly, so the prediction is not "close",
# it is every sample. A run that claimed to reset and did not fails
# here and passes everything else on the page.
diff = [i for i in range(n) if got[i] != fresh[i]]
ck(not diff,
f"the chip was STOPPED and re-PLAYED at the branch, so all "
f"{n:,} post-seek samples are EXACTLY a decode from the "
f"container's own accumulator ({dec['init']}) and step index 0"
+ ("" if not diff else f" -- {len(diff):,} differ, first at "
f"sample {diff[0]}"))
dc = [x for x in e]
const = len(set(dc)) == 1
ck(const,
f"...and the whole error against the encoder's intent is the "
f"SINGLE CONSTANT {dc[0]}"
+ ("" if const else f" -- it takes {len(set(dc))} values, so the "
f"step indices differ too and this is distortion, not offset")
)
if const:
print(f" -> a re-PLAYED branch costs a PERMANENT DC offset "
f"of {dc[0]} = {abs(dc[0])*100/hi_c:.1f}% of full scale "
f"and {abs(dc[0])*100/511:.1f}% of the 10-bit clamp's "
f"headroom. It is inaudible as a tone and it is not free: "
f"it is headroom, and it clicks once at the branch.")
else:
r0, r4 = band(0, 100), band(2*rate, 4*rate)
ck(band(1000, 5000)[1] < abs(band(1000, 5000)[0]),
f"the error at the branch is an OFFSET and not distortion: over "
f"64-320 ms its DC is {band(1000,5000)[0]:.1f} and its AC is "
f"{band(1000,5000)[1]:.2f}, so the chip is decoding the right "
f"shape from the wrong ground")
if r4:
print(f" -> playing THROUGH the branch, the offset is still "
f"{r4[0]:.0f} four seconds later ({abs(r4[0])*100/hi_c:.0f}%"
f" of full scale). There is no leakage term in this "
f"predictor; what decay there is comes from the signal's "
f"own clamping, not from the recursion forgetting.")
# AND THE CONTROL THAT MAKES EITHER READING MEAN ANYTHING.
ck(any(want[i] != fresh[i] for i in range(n)),
f"the two references are distinguishable over these {n:,} samples, "
f"so 'carry the predictor' and 'reset it' are different runs and this "
f"measurement has a subject")
if counters:
ck(counters["late"] == 0,
f"the player never re-armed a channel that still had bytes to send "
f"({counters['late']} did)")
ck(counters["starve"] == 0,
f"the player never found the channel counted out with no lump ready "
f"({counters['starve']} times it did)")
ck(counters["bytes"] == len(data),
f"the player's own byte count ({counters['bytes']:,}) is the whole "
f"stream it should have fed ({len(data):,})")
if seek is not None and iters > 1:
# THE TWO CELLS, ASSERTED APART. A player that kept one counter for
# "where the stream is" and "what the chip got" reports a number
# that is right for neither, and the symptom is a LAST LUMP that is
# long by the skip -- which is not an error, it is a rate.
want_skip = (iters - 1) * (stream_pos(d, seek)
- stream_pos(d, seek - seek % d.cad_f))
ck(counters.get("seekn") == iters - 1,
f"the player made {counters.get('seekn')} audio seek(s) for "
f"{iters-1} branch point(s) -- the second read FINDINGS 70.3 "
f"asks for, at a separate LBA")
ck(counters.get("seekb") == want_skip,
f"it skipped {counters.get('seekb')} B into the head of a lump "
f"and the cadence says {want_skip} -- the byte offset is what "
f"makes a branch land on its own frame instead of up to "
f"{d.cad_f-1} frames early")
ck(counters["pos"] == d.aud_bytes,
f"the stream POSITION ended at {counters['pos']:,} B, the "
f"container's whole stream ({d.aud_bytes:,}) -- while the chip "
f"was handed {counters['bytes']:,}. TWO CELLS FOR TWO FACTS: the "
f"position is where the container has got to and is what the "
f"last lump's length is measured against; the byte count is what "
f"the capture has to account for, and a seek moves one and not "
f"the other")
print(f" the player: {counters['armed']} lumps armed, "
f"{counters['fetched']} fetched, {counters['serv']:,} service "
f"calls over {counters['shown']} frames "
f"({'BUS HELD' if counters['held'] else 'CYCLE STEALING'})")
print("PACKED AUDIO GATE " + ("GREEN" if not fails
else f"RED: {len(fails)} failed"))
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main())
tools/bench/xfer_cost.sh
Executable
+112
View File
@@ -0,0 +1,112 @@
#!/bin/bash
# What the PIO transport costs the 68000, by subtraction (ROADMAP P4b,
# FINDINGS 58.2).
#
# tools/bench/xfer_cost.sh [ring_kb]
#
# THE MEASUREMENT IS AN A/B AND HAS TO BE. Nothing in the machine can time
# itself finely enough: src/player/clock.i counts V-DISP at 56.69 Hz and the
# thing being priced is a per-BYTE cost. So the same 120 frames are decoded
# twice, from the same ring, by the same src/player/stream.s, with the same
# src/player/ring.i placing every record -- and the ONLY difference is which
# side of the XF_* mailbox answers:
#
# model tools/bench/stream.lua, an unlimited modelled pipe. Bytes appear
# in the ring for free; the emulated time is decode plus ring_poll.
# scsi src/player/xfer.i and src/player/scsi.i, a real MB89352 and a real
# volume. The CPU moves every byte itself.
#
# Both runs are FREE-RUNNING (DLX_PACE=0). A paced run would measure the pace,
# not the work, and the difference this script exists for would vanish into the
# wait loop.
#
# WHAT THE SUBTRACTION IS HONEST ABOUT, and it is worth stating twice: this is a
# count of 68000 CLOCKS, not a delivery rate. It says what the player pays to
# move a byte with its own instruction stream. It says nothing about how fast
# the medium can supply one -- MAME's device models are functional, not
# transfer-timing accurate (docs/BENCHMARK.md, 42.5), and `W`, the clocks a DMAC
# steals per delivered byte, is untouched by every line here.
#
# The two run boundaries are sampled on the machine-frame notifier, so each end
# is up to 17.64 ms coarse (54.5). Against a difference of tens of seconds that
# is under a tenth of a percent, and the analytic cross-check below is what
# actually establishes the figure -- the subtraction only has to agree with it.
set -e
cd "$(dirname "$0")/../.."
RING=${1:-256}
command -v chdman > /dev/null || { echo "needs chdman (mame-tools)"; exit 2; }
echo "=== modelled transport, unlimited pipe"
DLX_PACE=0 DLX_RINGOWN=1 DLX_QDEPTH=2 bash tools/bench/pace_run.sh "$RING" 0 \
> tmp/xfer_cost_model.log 2>&1
echo "=== real transport, a CZ-6BS1 and a real volume"
DLX_PACE=0 DLX_RINGOWN=1 DLX_QDEPTH=2 DLX_XFER=scsi \
bash tools/bench/pace_run.sh "$RING" 0 > tmp/xfer_cost_scsi.log 2>&1
python3 - <<'PY'
import re, sys
def one(tag, log):
t = open(log, "rb").read().decode("latin1")
m = re.search(r"decoded (\d+) frames in ([0-9.]+) s emulated", t)
if not m: sys.exit(f"{log}: no completion line -- the pass did not finish")
return int(m.group(1)), float(m.group(2)), t
nfr, t_model, _ = one("model", "tmp/pace_r256_k0_p0_own.log")
nfr2, t_scsi, ts = one("scsi", "tmp/pace_r256_k0_p0_own_scsi.log")
assert nfr == nfr2, "the two runs decoded different frame counts"
nb = int(re.search(r"REAL TRANSPORT: \d+ READ\(10\)s by the 68000, (\d+) B", ts).group(1))
nw = int(re.search(r"SECTOR OVERHEAD: (\d+) B off the disc", ts).group(1))
CPUHZ = 10_000_000
FPS = 12
budget = CPUHZ / FPS
d = (t_scsi - t_model) * CPUHZ
print(f" {nfr} frames, {nb:,} B of record, {nw:,} B off the disc")
print(f" decode + ring_poll alone {t_model:8.4f} s emulated "
f"= {100*t_model*CPUHZ/nfr/budget:5.1f}% of a {FPS} fps frame")
print(f" ...with the real transport {t_scsi:8.4f} s emulated "
f"= {100*t_scsi*CPUHZ/nfr/budget:5.1f}%")
print(f" TRANSPORT COST: {d:,.0f} clocks = {d/nb:.2f} per delivered byte, "
f"{d/nw:.2f} per byte off the FIFO")
print(f" {d/nfr:,.0f} clk/frame = "
f"{100*d/nfr/budget:.1f}% of a {FPS} fps frame")
# ---- THE CROSS-CHECK, and it is the half that makes the number portable.
# If the measured cost is the 68000's own instruction stream, it must equal the
# 68000's cycle table for the loop in src/player/scsi.i. If it does NOT, the
# difference is time spent waiting on MAME's SPC model -- which is emulator
# behaviour and would not survive contact with a board.
LOOP = [("move.l #SC_PATIENCE,d3", 12), ("move.b SC_SSTS,d0 (xxx).L->Dn", 16),
("btst #0,d0", 10), ("beq.s taken", 10),
("move.b SC_DREG,(a1)+ (xxx).L->(An)+", 20),
("subq.l #1,d7", 8), ("bne.s taken", 10)]
loop = sum(c for _, c in LOOP)
print(f"\n the keep loop in src/player/scsi.i, from the 68000's cycle table:")
for n, c in LOOP: print(f" {c:3d} {n}")
pred = loop * nw / nb
print(f" --- {loop} clk per byte off the FIFO, and the FIFO carries the "
f"dropped\n window bytes too: {pred:.2f} per DELIVERED byte")
err = 100 * abs(pred - d/nb) / (d/nb)
print(f" MEASURED {d/nb:.2f} vs PREDICTED {pred:.2f} -- {err:.1f}% apart, so "
f"the cost is the\n instruction stream and not MAME's SPC: it is a "
f"figure a real board would also pay.")
resid = d/nb - pred
print(f" residual {resid:+.2f} clk/B = {resid*nb/nfr:,.0f} clk/record of "
f"select, CDB, status,\n message and xf_service -- the per-COMMAND cost, "
f"which is the part that does not\n scale with the record.")
print(f"\n AGAINST THE LADDER, in the same units (clocks charged to the CPU "
f"per delivered\n byte). W is the DMAC's steal, 52.5's bracket:")
for W, name in [(5, "single address, bus HELD"), (9, "dual address, held"),
(12, "single address, arbitrated"),
(19, "dual address, arbitrated -- the IPL ROM's own disk ch")]:
print(f" W = {W:2d} {name:52s} {100*W*nb/nfr/budget:5.1f}% of the frame")
print(f" PIO {d/nb:.0f} this rig, measured "
f" {100*d/nfr/budget:15.1f}%")
print(f"\n So the PIO transport is {d/nb/19:.1f}x the WORST DMA configuration "
f"this project has\n found and {d/nb/5:.1f}x the best. P4a is not an "
f"optimisation of this; it is the\n difference between a player and a "
f"slideshow.")
PY
tools/encoder/adpcm.py
+186
View File
@@ -0,0 +1,186 @@
#!/usr/bin/env python3
"""MSM6258 (OKI/Dialogic) 4-bit ADPCM -- encoder, decoder, and the fact that
there are TWO decoders and they are not the same one.
The X68000's ADPCM is an OKI MSM6258V clocked at 8 MHz, dividing to 15,625 /
10,417 / 7,812.5 samples a second, 4 bits each, two samples to a byte
(FINDINGS 52, buscost.ADPCM_SAMPLE_HZ). The sample word is 12 bits signed.
WHY THIS FILE HAS TWO DECODERS. Nothing in this repo can be trusted to say what
the chip does, and the two references available on this machine DISAGREE:
VARIANT 'shift' delta = ((2*(n&7) + 1) * step) >> 3
This is ffmpeg's `adpcm_ima_oki`, and `gate_vs_ffmpeg()`
reproduces it SAMPLE-EXACT, so it is not a reading of source
code -- it is a measurement of the decoder that ships.
VARIANT 'terms' delta = step/8 + (n&4 ? step : 0) + (n&2 ? step/2 : 0)
+ (n&1 ? step/4 : 0), each term truncated
This is the OKI datasheet's own form, the one an ADPCM chip
can actually build out of shifts and adds, and it is what
MAME's okim6258 is understood to compute. NOT VERIFIED HERE:
no MAME source tree is on this machine (FINDINGS 64.4).
They differ on 445 of 2,268 sampled nibbles, by up to 4 in 12-bit units --
small, and small is not zero. Which one the machine runs is an open question
with an experiment attached: MAME's x68000 HAS an okim6258, so it can be asked
rather than argued about.
Nibble order is HIGH NIBBLE FIRST within a byte -- measured, not assumed, by the
same gate: reading low-first mismatches ffmpeg on 1,728 of 2,268 samples.
"""
# The 49-entry OKI step table. floor(16 * 1.1**k) for k in 0..48 -- built rather
# than pasted, so a transcription slip is not one of the things that can be
# wrong here.
STEP = [int(16 * 1.1**k) for k in range(49)]
# The nibble magnitude's effect on the step index. Four quiet nibbles walk it
# down one, four loud ones walk it up by more.
INDEX_ADJUST = (-1, -1, -1, -1, 2, 4, 6, 8)
SAMPLE_MIN, SAMPLE_MAX = -2048, 2047 # the 12-bit DAC word
VARIANTS = ("shift", "terms")
# ---------------------------------------------------------------------------
# THE THREE AXES THAT WERE FIXED CONSTANTS UNTIL SESSION 34, and every one of
# them turned out to be a real choice that a decoder can get wrong. FINDINGS 65
# priced the `variant` axis at 25 dB and left the other three unnamed; MAME's
# okim6258 disagrees with this file on ALL THREE, so they are parameters now and
# tools/bench/adpcm_run.sh measures which values the emulated chip runs.
#
# ORDER which nibble of a byte is played FIRST. 'high' is the Dialogic VOX
# file convention and is what ffmpeg's adpcm_ima_oki reads, which is
# what 65.1 measured. That is a fact about a FILE FORMAT. What the
# chip does with a byte handed to its data register is a different
# question and MAME answers it 'low'.
# INIT the accumulator at the instant the chip is told to PLAY. This file
# started it at 0; MAME's okim6258 resets it to -2.
# BITS where the accumulator CLAMPS. This file clamped at the 12-bit ADPCM
# word; the MSM6258's own D/A is 10-bit and MAME clamps there, INSIDE
# the recursion, so it is not a post-hoc output scaling.
#
# Defaults are unchanged, so tools/bench/verify_adpcm.py still measures exactly
# what it measured in session 33: ffmpeg's decoder, high nibble first.
ORDERS = ("high", "low")
# WHAT THE MACHINE'S OWN CHIP DOES, MEASURED -- tools/bench/adpcm_run.sh, one
# model of sixteen reproducing 1,678 consecutive samples of a MAME capture
# sample-exact, with a negative control on every axis (FINDINGS 66). It is a
# measurement of MAME's device model driven through the real transport, not of
# an MSM6258; the silicon is still a hardware item.
#
# THE DEFAULTS ABOVE ARE DELIBERATELY *NOT* THESE. The defaults are ffmpeg's
# adpcm_ima_oki, because tools/bench/verify_adpcm.py's whole value is that it
# checks this file against an independent implementation, and a default that
# had drifted to match the thing under test would end that. Anything that
# ENCODES FOR THE MACHINE passes CHIP explicitly.
CHIP = dict(variant="terms", order="low", bits=10, init=-2)
def clamp_bounds(bits):
"""The accumulator's clamp, as MAME's okim6258 computes it: max = 2^(b-1)-1,
min = -2^(b-1). Note it is NOT symmetric, and the asymmetry is load-bearing
on a signal that saturates."""
return -(1 << (bits - 1)), (1 << (bits - 1)) - 1
def delta(nibble, step, variant):
"""The reconstruction step for one nibble, in 12-bit units."""
if variant == "shift":
d = ((2 * (nibble & 7) + 1) * step) >> 3
elif variant == "terms":
d = step // 8
if nibble & 4: d += step
if nibble & 2: d += step // 2
if nibble & 1: d += step // 4
else:
raise ValueError(f"unknown variant {variant!r}")
return -d if nibble & 8 else d
def decode_state(nibbles, variant="shift", state=None, init=0, bits=12):
"""The recursion, with its STATE in and out. (samples, (signal, idx)).
`decode` is this with the state thrown away, and it is written this way
round rather than duplicated because a stream that STOPS and RESUMES is not
a hypothetical here: an MSM6258 fed by a DMA channel goes on decoding
whatever byte its data register still holds when the channel counts out
(MAME okim6258 sound_stream_update reads m_data_in unconditionally while
PLAYING), so the samples between one lump and the next are the recursion
continuing over a repeated byte. tools/bench/verify_packed_audio.py has to
carry that state across the seam to check the lump on the far side of it,
and a second copy of the loop is a second place the clamp can drift.
"""
lo, hi = clamp_bounds(bits)
signal, idx = state if state is not None else (init, 0)
out = []
for n in nibbles:
signal += delta(n, STEP[idx], variant)
signal = lo if signal < lo else (hi if signal > hi else signal)
idx += INDEX_ADJUST[n & 7]
idx = 0 if idx < 0 else (48 if idx > 48 else idx)
out.append(signal)
return out, (signal, idx)
def decode(nibbles, variant="shift", init=0, bits=12):
"""Nibbles -> signed samples. State is (signal, step index); the step index
is 0 at the start of a stream and `init` is where the accumulator starts."""
return decode_state(nibbles, variant, None, init, bits)[0]
def encode(samples, variant="shift", init=0, bits=12):
"""12-bit signed samples -> nibbles.
The nibble is chosen by EXHAUSTIVE SEARCH over all sixteen, minimising the
reconstruction error of this sample. That is greedy rather than optimal --
a nibble also moves the step index, so a locally worse choice can pay later
-- but it is what a chip-matched encoder is expected to do and it costs
nothing offline. The decoder is run INSIDE the loop, so the encoder can
never drift away from what the decoder will reconstruct.
"""
lo, hi = clamp_bounds(bits)
signal, idx, out = init, 0, bytearray()
for s in samples:
step = STEP[idx]
best, best_err = 0, None
for n in range(16):
v = signal + delta(n, step, variant)
v = lo if v < lo else (hi if v > hi else v)
err = (v - s) ** 2
if best_err is None or err < best_err:
best, best_err = n, err
signal += delta(best, step, variant)
signal = lo if signal < lo else (hi if signal > hi else signal)
idx += INDEX_ADJUST[best & 7]
idx = 0 if idx < 0 else (48 if idx > 48 else idx)
out.append(best)
return bytes(out)
def pack(nibbles, order="high"):
"""Nibbles -> bytes. `order` names which nibble of a byte is played FIRST;
'high' is the VOX file convention. An odd count pads with a 0 nibble, which
is the quietest one the format has (delta = step/8)."""
if order not in ORDERS:
raise ValueError(f"unknown nibble order {order!r}")
n = bytes(nibbles)
if len(n) & 1:
n += b"\0"
if order == "high":
return bytes((n[i] << 4) | n[i + 1] for i in range(0, len(n), 2))
return bytes((n[i + 1] << 4) | n[i] for i in range(0, len(n), 2))
def unpack(data, count=None, order="high"):
if order not in ORDERS:
raise ValueError(f"unknown nibble order {order!r}")
out = bytearray()
for b in data:
if order == "high":
out.append(b >> 4); out.append(b & 15)
else:
out.append(b & 15); out.append(b >> 4)
return bytes(out[:count] if count is not None else out)
tools/encoder/dlx.py
+72 -9
View File
@@ -29,6 +29,13 @@ import spans as SP
MODE_SKIP, MODE_V1, MODE_V4, MODE_RAW = 0, 1, 2, 3
# A SCSI target answers in 512-byte blocks and a record is not a sector: on the
# DLX4 gate container 117 of 120 records start part way into one. DLX5 makes
# the container agree with the medium instead of making the transport reconcile
# them (tools/analysis/26_sector_align.py prices all three ways).
SECTOR = 512
class DLX:
def __init__(self, path):
self.raw = open(path, "rb").read()
@@ -38,14 +45,25 @@ class DLX:
# (FINDINGS 28.3), so the padding is part of the format, not a loader
# convenience -- but DLX1 containers stay readable, because every
# measurement in FINDINGS 28-31 was taken on one.
if b[:4] not in (b"DLX1", b"DLX2", b"DLX3"):
if b[:4] not in (b"DLX1", b"DLX2", b"DLX3", b"DLX4", b"DLX5"):
raise ValueError(f"{path}: not a DLX container")
self.version = int(b[3:4])
self.aligned = self.version >= 2
self.has_spans = self.version >= 3
self.has_index = self.version >= 4
# DLX5: every record starts on a 512-BYTE SECTOR boundary, and so does
# the frame stream itself. That is not a tidier version of DLX4's
# 4-byte rule -- it is what lets a DMA channel read a record as whole
# sectors straight into the ring, with no window and no bounce copy
# (FINDINGS 58.3 option C, and 59.4 made it a precondition: sc_in_data
# REFUSES a windowed read when the data phase is the channel's).
self.sector_aligned = self.version >= 5
self.rec_align = SECTOR if self.sector_aligned else (4 if self.aligned
else 1)
(self.W, self.H, self.fps, self.nframes,
self.k1, self.k4) = struct.unpack(">HHHHHH", b[4:16])
off_pal, off_cb1, off_cb4, off_frm = struct.unpack(">IIII", b[16:32])
off_idx = struct.unpack(">I", b[32:36])[0] if self.has_index else None
self.pal = np.frombuffer(b, np.uint8, 256 * 3, off_pal).reshape(256, 3)
self.cb1 = np.frombuffer(b, np.uint8, self.k1 * 16,
@@ -59,24 +77,69 @@ class DLX:
self.mode_bytes = (self.nb * 2 + 7) // 8
# frame directory: (offset of the mode header, payload length)
if self.aligned and off_frm % 4:
raise ValueError(f"{path}: DLX2 frame stream starts at {off_frm}, "
f"which is not 4-byte aligned")
if off_frm % self.rec_align:
raise ValueError(f"{path}: DLX{self.version} frame stream starts at "
f"{off_frm}, which is not {self.rec_align}-byte "
f"aligned")
self.frames = []
p = off_frm
for _ in range(self.nframes):
(n,) = struct.unpack(">I", b[p:p + 4])
self.frames.append((p + 4, n))
p += 4 + n
if self.aligned:
p += -p % 4 # skip the pad to the next record
# The writer does not pad after the LAST record -- nothing follows it --
# so `p` may have advanced past the end by up to 3 bytes there.
p += -p % self.rec_align # skip the pad to the next record
# DLX2/DLX3 do not pad after the LAST record -- nothing follows it --
# so `p` may have advanced past the end by up to 3 bytes there. DLX4
# and DLX5 DO pad it, because a producer that trusts the index fetches
# a whole padded record for the last frame like any other.
slack = len(b) - p
if not (slack == 0 or (self.aligned and -3 <= slack < 0)):
if not (slack == 0 or (self.aligned and not self.has_index
and -(self.rec_align - 1) <= slack < 0)):
raise ValueError(f"{path}: {slack} trailing bytes after "
f"{self.nframes} frames")
# DLX4's record index, and it is CHECKED rather than trusted. The
# walk above is what every reader in this tree did before there was an
# index -- read a record's length word to find the next one -- and it is
# exactly what a player streaming off a disc cannot do, because the
# length word of record i+1 is one of the bytes it has not fetched. So
# the two are computed independently here and required to agree: the
# index is the producer's only source of record geometry, and an index
# that disagrees with the stream places records at wrong addresses,
# which the block loop reads without a bounds check (49.2).
self.index = None
if self.has_index:
self.index = list(struct.unpack(
f">{self.nframes}H", b[off_idx:off_idx + 2 * self.nframes]))
walked = [self._padded(n) // 4 for _, n in self.frames]
if self.index != walked:
bad = next(i for i in range(self.nframes)
if self.index[i] != walked[i])
raise ValueError(
f"{path}: record index disagrees with the frame stream at "
f"frame {bad}: index says {self.index[bad]} longwords, the "
f"stream is {walked[bad]}")
if off_frm + 4 * sum(self.index) != len(b):
raise ValueError(
f"{path}: the index accounts for "
f"{off_frm + 4 * sum(self.index)} bytes and the file is "
f"{len(b)} -- a producer trusting it would run off the end")
def _padded(self, n):
"""Bytes one record of `n` payload bytes occupies, pad included."""
ln = 4 + n
return ln + (-ln % self.rec_align)
def record_lengths(self):
"""Padded record lengths in BYTES, in stream order.
The one place the container's alignment rule is applied. Every caller
that used to write `4 + n + (-(4+n) % 4)` was carrying its own copy of
that rule, which is exactly the kind of duplication that made DLX5 a
multi-file change instead of a one-line one.
"""
return [self._padded(n) for _, n in self.frames]
def modes(self, f):
o, _ = self.frames[f]
h = np.frombuffer(self.raw, np.uint8, self.mode_bytes, o)
tools/encoder/dlxp.py
+463
View File
@@ -0,0 +1,463 @@
#!/usr/bin/env python3
"""DLXP1 -- the PACKED container, and the one place its layout rule is written.
from dlxp import DLXP, pack_picture, unpack_picture, write
THIS CONTAINER HAS NO DECODER. That is the point of it (FINDINGS 61): a record
is the bytes a DMA channel puts straight into the palette registers and GVRAM,
so the "reference decoder" here is not a decoder at all -- it is an assertion
about where each byte lands. `dlx.py` exists because a 68000 has to PARSE the
codec's container and can get it wrong; this file exists because a DMA channel
must NOT have to parse anything, and the format is what makes that true.
The layout, all of it fixed, all of it verified as a picture in FINDINGS 47.2:
* 256-colour GVRAM normally throws away the high byte of every word a CPU
writes, so a picture byte costs two disc bytes. CRTC R20 bit 11 turns the
masking off (46.5/47.1), and with the two 256-colour pages scrolled apart by
384 one word carries TWO pixels: word `i` of a row is
(pix[y][i+128] << 8) | pix[y][i]
-- page 1 (X-scrolled 384, transparent top) shows columns 128..255, page 0
(unscrolled, opaque bottom) shows columns 0..127.
* so a row is 128 words = 256 BYTES for 256 pixels, and big-endian storage
makes the byte order `pix[128], pix[0], pix[129], pix[1], ...`. That byte
order is not a serialisation choice: it is what the 68000's bus puts on the
high half of the word, and the channel copies bytes.
* 192 rows -> 49,152 B of picture, 1.0 B/pixel against the unpacked 2.0.
* index 0 is the TRANSPARENCY KEY and never appears; black is 255, which is
what the letterbox rows display (`vq.frame_palette`).
* words 128..511 of each row, and the letterbox rows themselves, are STATIC
SETUP -- written once at scene setup, never per frame -- so they are not in
the container. FINDINGS 47.2 lists them; keeping them out is what makes the
per-frame payload exactly the picture.
* the GVRAM line stride is 1,024 B and a row is 256 B, so the container's rows
are CONTIGUOUS and the 1,024 B step is the channel's, walked by array
chaining from one start (FINDINGS 62). A container that carried the stride
would be 4x the size and would say nothing extra.
header, 64 bytes, big-endian, then zero pad to the first sector:
0 'DLXP'
4 u16 version (2)
6 u16 flags bit 0: a per-frame palette is present
bit 1: the palette is at the END of the record
bit 2: an audio stream is interleaved (DLXP2)
8 u16 width, u16 height
12 u16 fps, u16 nframes
16 u32 record bytes fixed, and a whole number of 512 B sectors
20 u32 palette bytes 512 (256 GRB555+I words), or 0
24 u32 picture bytes 49,152
28 u32 frames offset offset of RECORD 0, lump 0 already skipped
-- DLXP2 adds, and every field is zero in a silent container:
32 u32 audio offset offset of LUMP 0, 512 B, i.e. sector 1
36 u32 audio sample Hz 15,625 -- the chip's, not the disc's
40 u32 audio bytes the ADPCM payload, padding NOT counted
44 u16 cadence F frames between one lump and the next
46 u16 cadence A whole sectors in a lump
48 u16 audio format bit 0: LOW nibble first. bit 1: the OKI
datasheet's per-term delta ('terms')
50 u16 audio clamp bits where the accumulator saturates, 10 or 12
52 i16 audio init the accumulator at PLAY, -2 on this chip
54 u16 reserved (0)
56 u32 reserved (0)
60 u32 reserved (0)
then, from sector 1, GROUPS: one audio lump of A sectors, then F records.
A record is 49,664 B = 97 sectors EXACTLY and a lump is A*512 B EXACTLY, so
record i is at off_frm + i*rec_bytes + (i//F)*A*512
lump k is at off_aud + k*(F*rec_bytes + A*512)
-- still arithmetic, still no index, still nothing walked.
WHY AUDIO IS A CADENCE AND NOT A FIELD IN THE RECORD (FINDINGS 65.3). 15,625
samples a second, two to a byte, is 651.0416... B per 12 fps slot, and the dots
are the whole problem: put slot i's audio in record i and records become
VARIABLE LENGTH, which needs an index, which ends the format. A fixed cadence
of F frames per A sectors keeps `LBA(i)` arithmetic and pays instead in padding,
and the padding is a rational-approximation problem whose answer is not the
obvious cadence: F=1 wastes 57.3% of every audio sector and F=11, A=14 wastes
0.09%.
AND THE PADDING IS NOT WHERE THE BYTES ARE (FINDINGS 67). A lump is A*512 B of
SPACE and it does NOT carry A*512 B of audio: F frames need F*15625/24 B, which
is 7,161.4583... at F=11, so a lump's PAYLOAD alternates 7,161 and 7,162 by the
same remainder arithmetic FINDINGS 54's frame clock carries, and the rest of the
sector is zero. A player that fed the chip the whole lump would be handing it
6.54 B a group it should not have -- 0.09% too much audio, which is not waste,
it is DRIFT: 0.83 ms a group, 1.2 s of lip-sync over the game's 22 minutes. So
`lump_bytes(k)` below is the format, not a convenience, and a player computes it
with one accumulator: `acc += F*aud_hz; n = acc // (2*fps); acc %= 2*fps`.
THE FOUR ADPCM AXES ARE IN THE HEADER BECAUSE GETTING ONE WRONG COSTS 25 dB
(FINDINGS 66). Nibble order, delta formula, clamp width and the accumulator's
value at PLAY are properties of the DECODER, and a container encoded for one
decoder and played on another comes out with the noise louder than the signal.
They are four fields rather than a version number so that a mismatch is legible
in a hexdump rather than inferred from a container's age.
THERE IS NO RECORD INDEX AND NO LENGTH WORD, and that is the difference DLX4's
index was invented for (49.3): a codec record's length is content-dependent, so
a producer cannot know where record i+1 starts without being told. A packed
record's length is GEOMETRY -- 192 rows of 256 B plus a palette -- so record `i`
is at `off_frm + i * rec_bytes` and a seek is arithmetic. Nothing in this
format has to be walked, which is also why the packed player has no ring
(ROADMAP K3): there is no variable-length thing to keep contiguous.
THE PALETTE ORDER IS A DECISION, NOT AN ACCIDENT (FINDINGS 62.5). Palette first
or 193rd is visible on screen for one paint -- old rows under the new palette, or
new rows under the old one -- and it is moot if buffer mode blanks the layer
(47.4/B2). It is a CONTAINER property here, chosen at encode time by
`--palette-last` and recorded in flags bit 1, so K3 can measure both without a
re-encode being an argument about which one the format assumed.
"""
import struct
import numpy as np
MAGIC = b"DLXP"
VERSION = 2
SECTOR = 512 # same rule and the same reason as dlx.SECTOR
PAL_BYTES = 512 # 256 entries, one GRB555+I word each
HDR_BYTES = 64
FLAG_PALETTE = 1 << 0
FLAG_PALETTE_LAST = 1 << 1
FLAG_AUDIO = 1 << 2
# The two ADPCM axes that are booleans. The other two -- the clamp width and
# the accumulator at PLAY -- are numbers and get their own fields, because
# encoding them as flags would mean this file deciding which values are legal.
AFMT_ORDER_LOW = 1 << 0
AFMT_VARIANT_TERMS = 1 << 1
# The default cadence, and it is a MEASUREMENT rather than a taste (FINDINGS
# 65.3): the sweep's floor is F=81 and costs 91,136 B more of player RAM for the
# last 0.09 of a point of padding, on a machine where two record buffers already
# want 99,328 B.
CADENCE_F = 11
def audio_format(variant, order, bits, init):
"""adpcm.py's four axes -> the three header fields that carry them.
This file does not import adpcm.py and must not: a container format that
depended on an encoder would be a format that could not be read without one.
What it carries is the DESCRIPTION, and `tools/analysis/34_packed_audio.py`
is what checks the description against the encoder that wrote the bytes.
"""
if variant not in ("shift", "terms") or order not in ("high", "low"):
raise ValueError(f"unknown ADPCM decoder ({variant!r}, {order!r})")
fmt = ((AFMT_ORDER_LOW if order == "low" else 0)
| (AFMT_VARIANT_TERMS if variant == "terms" else 0))
return fmt, int(bits), int(init)
def audio_decoder(fmt, bits, init):
"""The inverse: what a player, or a gate, has to run to hear the bytes."""
return dict(variant="terms" if fmt & AFMT_VARIANT_TERMS else "shift",
order="low" if fmt & AFMT_ORDER_LOW else "high",
bits=int(bits), init=int(init))
def cadence(fps, hz, F=CADENCE_F):
"""(F, A): F frames of audio rounded UP to whole sectors.
A*512 must cover F frames or the chip runs dry, so A is a ceiling and the
excess is padding the wire pays for and nothing plays. Integer arithmetic
throughout: the whole point of 65.3 is that this ratio has a remainder, and
a float here would hide the case where it does not.
"""
num, den = F * hz, 2 * fps # bytes per group = num/den
return F, -(-num // (den * SECTOR))
def lump_bytes(k, F, fps, hz, total=None):
"""The PAYLOAD of lump k -- what is handed to the chip, padding excluded.
Exact, and exactness is the finding (FINDINGS 67): floor((k+1)*F*hz/(2*fps))
- floor(k*F*hz/(2*fps)) alternates 7,161 and 7,162 at F=11, and a player
that fed the chip the whole A*512 B lump instead would run 0.09% fast --
1.2 s of lip-sync over 22 minutes.
"""
den = 2 * fps
n = ((k + 1) * F * hz) // den - (k * F * hz) // den
if total is not None: # the last lump is short, not padded
n = max(0, min(n, total - (k * F * hz) // den))
return n
def n_lumps(nframes, F):
return -(-nframes // F)
def pack_picture(idx):
"""(H,W) palette indices -> the bytes GVRAM wants, in GVRAM order.
The ONE place the interleave rule is applied, for the same reason
`dlx.record_lengths` is the one place the alignment rule is: every caller
that carries its own copy of a layout rule is a place the layout can drift.
"""
H, W = idx.shape
if W % 2:
raise ValueError(f"packed layout needs an even width, got {W}")
half = W // 2
left, right = idx[:, :half], idx[:, half:]
out = np.empty((H, half, 2), np.uint8)
out[:, :, 0] = right # high byte of the word -> page 1 -> col i+128
out[:, :, 1] = left # low byte -> page 0 -> col i
return out.reshape(H, half * 2)
def unpack_picture(buf, W, H):
"""The inverse, and the assertion that `pack_picture` is reversible."""
b = np.frombuffer(buf, np.uint8, W * H).reshape(H, W // 2, 2)
idx = np.empty((H, W), np.uint8)
idx[:, W // 2:] = b[:, :, 0]
idx[:, :W // 2] = b[:, :, 1]
return idx
def record_bytes(W, H, palette=True):
n = W * H + (PAL_BYTES if palette else 0)
if n % SECTOR:
raise ValueError(f"a {W}x{H} packed record is {n} B, which is not a "
f"whole number of {SECTOR} B sectors")
return n
def write(path, W, H, fps, frames, palette_last=False, audio=None):
"""`frames` is a sequence of (palette_words_bytes | None, picture_bytes).
`audio`, when given, is a dict: `data` the packed ADPCM bytes, `hz` the
chip's sample rate, `variant`/`order`/`bits`/`init` the four axes FINDINGS
66 measured, and optionally `F`. The stream is CUT UP HERE and nowhere
else, by `lump_bytes`, for the same reason `pack_picture` is the one place
the interleave lives: a second copy of a layout rule is a place the layout
can drift.
"""
frames = list(frames)
pal_b = PAL_BYTES if frames and frames[0][0] is not None else 0
pic_b = W * H
rec_b = record_bytes(W, H, palette=bool(pal_b))
flags = ((FLAG_PALETTE if pal_b else 0)
| (FLAG_PALETTE_LAST if palette_last and pal_b else 0)
| (FLAG_AUDIO if audio else 0))
if audio:
hz = int(audio["hz"])
F, A = cadence(fps, hz, audio.get("F", CADENCE_F))
au = audio["data"]
fmt, bits, init = audio_format(audio["variant"], audio["order"],
audio["bits"], audio["init"])
lumps = []
got = 0
for k in range(n_lumps(len(frames), F)):
n = lump_bytes(k, F, fps, hz, total=len(au))
if n > A * SECTOR:
raise ValueError(f"lump {k} needs {n} B and the cadence gives "
f"{A*SECTOR}")
lumps.append(au[got:got + n].ljust(A * SECTOR, b"\0"))
got += n
# A container whose audio runs out before its pictures do is a container
# that goes silent part way through, which is exactly the failure a
# writer should refuse rather than a player discover.
if got < len(au):
raise ValueError(f"{len(au)-got} B of audio have no lump to ride in "
f"-- {len(frames)} frames hold {got} B")
if got < min(len(au), (len(frames) * hz) // (2 * fps)):
raise ValueError(f"the audio stream is short: {len(au)} B for "
f"{len(frames)} frames at {fps} fps")
off_aud, off_frm = SECTOR, SECTOR + A * SECTOR
atail = struct.pack(">IIIHHHHhHII", off_aud, hz, len(au), F, A,
fmt, bits, init, 0, 0, 0)
else:
F = A = 0
lumps = []
off_aud, off_frm = 0, SECTOR
atail = struct.pack(">IIIHHHHhHII", 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
hdr = (MAGIC + struct.pack(">HHHHHH", VERSION, flags, W, H, fps, len(frames))
+ struct.pack(">IIII", rec_b, pal_b, pic_b, off_frm) + atail)
assert len(hdr) == HDR_BYTES, len(hdr)
with open(path, "wb") as fh:
fh.write(hdr + b"\0" * (SECTOR - HDR_BYTES))
for i, (pw, pic) in enumerate(frames):
if len(pic) != pic_b or (pal_b and len(pw) != pal_b):
raise ValueError(f"frame {i}: record parts are the wrong size")
# The lump goes BEFORE the group it feeds, which is the one place
# this format makes a choice audio forced on it. 65.3's formula put
# it after; a stream is read forwards, so bytes that arrive after
# the slot they belong to are bytes a player has to have fetched
# early anyway. Placing it first makes the fetch order the play
# order and costs one lump of offset in the arithmetic.
if audio and i % F == 0:
fh.write(lumps[i // F])
rec = pic if not pal_b else (pic + pw if palette_last else pw + pic)
fh.write(rec)
return rec_b
class DLXP:
"""Reader, and every invariant the format claims, CHECKED rather than read.
The checks are not defensive coding. A packed record is written into GVRAM
and the palette registers with no bounds test anywhere -- the channel has no
opinion about what it is copying -- so a container whose geometry is a byte
wrong does not fail, it paints.
"""
def __init__(self, path):
b = self.raw = open(path, "rb").read()
if b[:4] != MAGIC:
raise ValueError(f"{path}: not a DLXP container")
(self.version, self.flags, self.W, self.H, self.fps,
self.nframes) = struct.unpack(">HHHHHH", b[4:16])
(self.rec_bytes, self.pal_bytes,
self.pic_bytes, self.off_frm) = struct.unpack(">IIII", b[16:32])
if self.version != VERSION:
raise ValueError(f"{path}: DLXP version {self.version}")
self.has_palette = bool(self.flags & FLAG_PALETTE)
self.palette_last = bool(self.flags & FLAG_PALETTE_LAST)
self.has_audio = bool(self.flags & FLAG_AUDIO)
(self.off_aud, self.aud_hz, self.aud_bytes, self.cad_f, self.cad_a,
self.aud_fmt, self.aud_bits, self.aud_init,
_r0, _r1, _r2) = struct.unpack(">IIIHHHHhHII", b[32:64])
if not self.has_audio:
if any((self.off_aud, self.aud_hz, self.aud_bytes, self.cad_f,
self.cad_a, self.aud_fmt, self.aud_bits, self.aud_init)):
raise ValueError(f"{path}: silent container with audio fields set")
else:
if (self.cad_f, self.cad_a) != cadence(self.fps, self.aud_hz,
self.cad_f):
raise ValueError(f"{path}: cadence F={self.cad_f} A={self.cad_a} "
f"does not cover {self.cad_f} frames of "
f"{self.aud_hz} Hz audio at {self.fps} fps")
if self.off_aud != SECTOR or self.off_aud % SECTOR:
raise ValueError(f"{path}: audio starts at {self.off_aud}")
if self.aud_bits not in (10, 12):
raise ValueError(f"{path}: ADPCM clamp is {self.aud_bits} bits")
# The stream has to fill the groups it is cut into, or the last
# frames of the scene play in silence and nothing says so.
want = sum(lump_bytes(k, self.cad_f, self.fps, self.aud_hz)
for k in range(n_lumps(self.nframes, self.cad_f)))
if not (want - self.cad_f * self.aud_hz // (2 * self.fps)
<= self.aud_bytes <= want):
raise ValueError(f"{path}: {self.aud_bytes} B of audio for "
f"{self.nframes} frames, geometry wants {want}")
if self.pic_bytes != self.W * self.H:
raise ValueError(f"{path}: picture is {self.pic_bytes} B for "
f"{self.W}x{self.H} -- the packed layout is 1.0 B/px")
if self.pal_bytes != (PAL_BYTES if self.has_palette else 0):
raise ValueError(f"{path}: palette section is {self.pal_bytes} B")
if self.rec_bytes != self.pal_bytes + self.pic_bytes:
raise ValueError(f"{path}: record is {self.rec_bytes} B, parts are "
f"{self.pal_bytes} + {self.pic_bytes}")
# The whole reason for DLX5 (58.3/59.4), and here it is free rather than
# a re-encode: the record is a fixed multiple of a sector by geometry.
if self.off_frm % SECTOR or self.rec_bytes % SECTOR:
raise ValueError(f"{path}: not sector-aligned -- stream at "
f"{self.off_frm}, record {self.rec_bytes}")
if self.off_frm != (SECTOR + (self.cad_a * SECTOR if self.has_audio
else 0)):
raise ValueError(f"{path}: record 0 is at {self.off_frm}, and a "
f"{self.cad_a}-sector lump comes before it")
want = (self.off_frm + self.nframes * self.rec_bytes
+ (n_lumps(self.nframes, self.cad_f) - 1) * self.cad_a * SECTOR
if self.has_audio else
self.off_frm + self.nframes * self.rec_bytes)
if len(b) != want:
raise ValueError(f"{path}: {len(b)} bytes, geometry says {want}")
def frame_off(self, i):
"""The arithmetic, and the ONE place it is written on the host side --
src/player/packed.s is the other, in six instructions, and
tools/analysis/34_packed_audio.py is what makes the two agree."""
if not (0 <= i < self.nframes):
raise IndexError(i)
o = self.off_frm + i * self.rec_bytes
return o + (i // self.cad_f) * self.cad_a * SECTOR if self.has_audio else o
def lump_off(self, k):
if not self.has_audio or not (0 <= k < self.n_lumps):
raise IndexError(k)
return self.off_aud + k * (self.cad_f * self.rec_bytes
+ self.cad_a * SECTOR)
@property
def n_lumps(self):
return n_lumps(self.nframes, self.cad_f) if self.has_audio else 0
def lump(self, k, padding=False):
"""Lump k's PAYLOAD -- what the chip is fed. `padding=True` returns the
whole A*512 B sector run instead, which is what the disc moves and what
a player must NOT hand to the chip (FINDINGS 67)."""
o = self.lump_off(k)
if padding:
return self.raw[o:o + self.cad_a * SECTOR]
n = lump_bytes(k, self.cad_f, self.fps, self.aud_hz, total=self.aud_bytes)
return self.raw[o:o + n]
def audio(self):
"""The whole ADPCM stream, reassembled from its lumps."""
return b"".join(self.lump(k) for k in range(self.n_lumps))
def decoder(self):
"""The four axes the bytes were encoded for, as adpcm.decode's kwargs."""
return audio_decoder(self.aud_fmt, self.aud_bits, self.aud_init)
def record(self, i):
o = self.frame_off(i)
return self.raw[o:o + self.rec_bytes]
def _split(self, i):
r = self.record(i)
if not self.has_palette:
return None, r
if self.palette_last:
return r[self.pic_bytes:], r[:self.pic_bytes]
return r[:self.pal_bytes], r[self.pal_bytes:]
def palette_words(self, i):
pw, _ = self._split(i)
if pw is None:
raise ValueError("this container carries no palette -- it was made "
"with --no-palette, and the palette a player would "
"display is not in the file to be read back")
return np.frombuffer(pw, ">u2").astype(np.uint16)
def indices(self, i):
_, pic = self._split(i)
return unpack_picture(pic, self.W, self.H)
def palette_rgb(self, i):
"""(256,3) uint8 -- what the DISPLAY produces, not what the encoder meant.
The palette in a record is already a GRB555+I word, so this is where the
5-bit hardware quantisation gets charged. Everything upstream of the
container is in RGB888 and 61.9's +4.89 dB was quoted there; a player's
number has to come from here. Same maths as `dlxload.pack_palette` and
`tools/bench/verify_frame256.py` -- the shared LSB `I` is a bit in the
word, so unpacking it needs no choice made.
"""
w = self.palette_words(i).astype(int)
f = np.stack([(w >> 6) & 31, (w >> 11) & 31, (w >> 1) & 31], 1) # R,G,B
p6 = lambda v: ((v << 2) | (v >> 4)) & 0xFF
return p6((f << 1) | (w & 1)[:, None]).astype(np.uint8)
def render(self, i):
"""(H,W,3) uint8 -- the frame as the display produces it."""
return self.palette_rgb(i)[self.indices(i)]
def video_kbps(self):
"""The picture on the wire. FIXED by geometry -- there is no lever."""
return self.rec_bytes * self.fps / 1024
def audio_kbps(self):
"""What the CADENCE costs, padding included, which is the honest figure:
the disc moves whole sectors and the wire pays for the ones that are
zero as well as the ones that are audio (FINDINGS 65.3)."""
if not self.has_audio:
return 0.0
return self.cad_a * SECTOR / self.cad_f * self.fps / 1024
def kbps(self):
"""Both, over the scene's own duration."""
return self.video_kbps() + self.audio_kbps()
tools/encoder/encode.py
+86 -12
View File
@@ -15,7 +15,7 @@ lets quiet frames spend the whole allowance and lands the mean ON target.
Container (little-endian is WRONG here -- the 68000 is big-endian, so every
multi-byte field is big-endian and the decoder can read it with a plain move.w):
header, 32 bytes
header, 32 bytes ('DLX4': 36 -- one more offset, see below)
0 'DLX2' magic ('DLX1' = the same, unaligned; still read)
4 u16 width, u16 height
8 u16 fps, u16 nframes
@@ -25,6 +25,23 @@ multi-byte field is big-endian and the decoder can read it with a plain move.w):
20 u32 cb1 offset (k1 * 16 bytes of palette indices)
24 u32 cb4 offset (k4 * 4 bytes)
28 u32 frames offset
32 u32 record index offset DLX4 ONLY. nframes * u16, each the length of
that frame's PADDED record in LONGWORDS --
i.e. (4 + payload + pad) / 4, the whole thing
the ring producer must place contiguously.
THE INDEX IS NOT A CONVENIENCE, AND IT IS NOT DERIVABLE ON THE MACHINE. The
`aligned` wrap policy (FINDINGS 49.3) requires the producer to know how long
the next record is BEFORE it fetches it, because that is what decides whether
it fits before the end of the ring or leaves a hole and restarts at the base.
Every reader in this tree up to DLX3 learned record boundaries by WALKING the
frame stream -- reading each record's length word to find the next -- which a
host with the whole file mapped can do and a player streaming off a disc
cannot: the length word of record i+1 is exactly one of the bytes it has not
fetched yet. A branching game needs the same structure a second time, to seek
to a branch point without reading what lies between.
Lengths are stored rather than offsets: 2 bytes a frame instead of 4, and the
offsets are a running sum the player builds once at scene load (u16 caps a
record at 262,140 B, asserted at write time).
then, per frame, each record starting on a 4-BYTE BOUNDARY (0-3 zero pad
bytes before it; a 68000 takes an address error, not a slow read, on an odd
`move.l` -- FINDINGS 28.3):
@@ -49,6 +66,7 @@ import argparse, struct, sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import vq as VQ, vq_hybrid as H, ratectl as RC, spans as SP
from dlx import SECTOR
# Measured on the emulated 68000, FINDINGS 24. Instruction cycles against
# zero-wait-state memory, so these are floors, not hardware predictions.
@@ -117,8 +135,12 @@ def build_records(m, enc, span_mode):
return out
def write_container(path, m, frames, fps, k1, k4, span_mode):
"""Write the whole container. Returns (total bytes, video bytes, pad)."""
def write_container(path, m, frames, fps, k1, k4, span_mode, sector=True):
"""Write the whole container. Returns (total bytes, video bytes, pad).
`sector` selects DLX5's 512-byte record alignment over DLX4's 4-byte one.
"""
align = SECTOR if sector else 4
palette = m["pal"][:256]
if len(palette) < 256:
palette = np.vstack([palette, np.zeros((256 - len(palette), 3), np.uint8)])
@@ -126,31 +148,63 @@ def write_container(path, m, frames, fps, k1, k4, span_mode):
cb1_b = m["cb1"].astype(np.uint8).tobytes()
cb4_b = m["cb4"].astype(np.uint8).tobytes()
off_pal = 32
off_pal = 36 if span_mode else 32
if not span_mode:
sector = False # DLX2 has no index and no sector rule
align = 4
off_cb1 = off_pal + len(pal_b)
off_cb4 = off_cb1 + len(cb1_b)
off_frm = off_cb4 + len(cb4_b)
# DLX4: the record index sits with the palette and the codebooks, ahead of
# the frame stream, because it is part of what has to ARRIVE before frame 0
# can be decoded -- FINDINGS 53.5's scene header, and this adds to it.
idx_b = b""
if span_mode:
qlens = []
for i, rec in enumerate(frames):
n = 4 + len(rec)
n += -n % align
q = n // 4
assert q <= 0xFFFF, (f"frame {i} is {n} B: a u16 longword count "
f"caps a record at 262,140 B")
qlens.append(q)
idx_b = struct.pack(f">{len(qlens)}H", *qlens)
off_idx = off_cb4 + len(cb4_b)
off_frm = off_idx + len(idx_b)
# DLX2: every frame record starts on a 4-byte boundary, including the
# first. Payload lengths are arbitrary, so end-to-end records land on odd
# addresses -- and `move.l (a0)+` at an odd address is an ADDRESS ERROR on
# a 68000, not a slow read. It vectors into the IPL and looks exactly like
# an infinite loop (FINDINGS 28.3). tools/bench/prep_dlx.py has been
# realigning at load time; the container now carries it.
tbl_pad = -off_frm % 4
# DLX5 aligns the frame stream itself as well as the records inside it, so
# the whole container can be laid on a volume at a sector boundary and every
# record lands on one. Aligning the records to each other and not the run
# they sit in would leave 117 of 120 of them off-sector again the moment the
# scene header changed length by a byte.
tbl_pad = -off_frm % align
off_frm += tbl_pad
hdr = ((b"DLX3" if span_mode else b"DLX2")
magic = (b"DLX5" if sector else b"DLX4") if span_mode else b"DLX2"
hdr = (magic
+ struct.pack(">HHHHHH", m["W"], m["H"], fps, len(frames), k1, k4)
+ struct.pack(">IIII", off_pal, off_cb1, off_cb4, off_frm))
assert len(hdr) == 32, len(hdr)
if span_mode:
hdr += struct.pack(">I", off_idx)
assert len(hdr) == (36 if span_mode else 32), len(hdr)
frm_pad = 0
with open(path, "wb") as fh:
fh.write(hdr); fh.write(pal_b); fh.write(cb1_b); fh.write(cb4_b)
fh.write(b"\0" * tbl_pad)
fh.write(idx_b); fh.write(b"\0" * tbl_pad)
for i, rec in enumerate(frames):
fh.write(struct.pack(">I", len(rec))); fh.write(rec)
if i + 1 < len(frames): # nothing follows the last record
n = -(4 + len(rec)) % 4
# DLX2/3 skipped the pad after the LAST record because nothing
# followed it. DLX4's index describes PADDED records, and a producer
# that trusts the index fetches that many bytes -- so the last
# record is padded too, and the file ends where the index says it
# does rather than up to 3 bytes short of it.
if span_mode or i + 1 < len(frames):
n = -(4 + len(rec)) % align
fh.write(b"\0" * n); frm_pad += n
total = os.path.getsize(path)
return total, sum(len(r) + 4 for r in frames) + frm_pad, frm_pad
@@ -212,6 +266,13 @@ def main():
"Default OFF: measured, it is worth one frame of 120 "
"at --spans need and a 2%% regression at --spans all "
"(FINDINGS 44)")
ap.add_argument("--joint-spans", action="store_true",
help="re-run the lam search with the bytes the span pass "
"freed, then re-span (E3, FINDINGS 39.3 item 5). The "
"span pass removes the block payload of every block "
"it covers, so without this the frame lands under its "
"allowance and the blocks that were NOT spanned were "
"priced as if those bytes were still needed.")
ap.add_argument("--no-cpu-fit", action="store_true",
help="drop the per-frame 68000 decode ceiling (session 7 "
"behaviour: 31%% of frames on hard content do not fit)")
@@ -219,6 +280,13 @@ def main():
help="how full the player's buffer is assumed to be at "
"scene start, as a fraction of the bucket (0 = cold "
"buffer after a seek, the conservative assumption)")
ap.add_argument("--no-reserve-black", action="store_true",
help="let the scene palette spend all 256 entries on the "
"picture. The default RESERVES index 0 as true black "
"(FINDINGS 23.4), because GVRAM cleared to zero shows "
"entry 0 and the 256x192 picture sits in a 256x256 "
"mode -- so a free palette letterboxes the frame in "
"whatever colour mediancut happened to put first.")
ap.add_argument("--preview")
a = ap.parse_args()
@@ -239,6 +307,7 @@ def main():
if a.disk_clk_byte is not None:
RC.DISK_CLK_BYTE = a.disk_clk_byte
RC.JOINT_DECIDE = a.joint_decide
RC.JOINT_SPANS = a.joint_spans
RC.JOINT_BUCKET = a.joint_bucket
cyc_budget = None if a.no_cpu_fit else RC.FRAME_CYCLES
span_mode = None if (a.spans == "off" or not rc) else a.spans
@@ -261,8 +330,13 @@ def main():
else:
print(f" target {prof['kbps']} KB/s, FIXED lam={lam} (no rate control)")
print(f" k1={k1} k4={k4}, {_IDX_BYTES}-byte indices")
print(f" palette: " + ("255 picture colours, index 0 RESERVED as true "
"black for the letterbox (23.4)" if not a.no_reserve_black
else "all 256 entries to the picture (--no-reserve-black); index 0 "
"is whatever mediancut put there, and the letterbox with it"))
m = H.build(a.frames_dir, k1=k1, k4=k4, iters=a.iters)
m = H.build(a.frames_dir, k1=k1, k4=k4, iters=a.iters,
reserve_black=not a.no_reserve_black)
if rc:
enc = RC.encode_rate_controlled(m, prof["kbps"], fps=a.fps,
bucket_frames=a.bucket_frames,
tools/encoder/extract_audio.py
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env python3
"""Extract the audio of a Blu-ray window as mono PCM at an MSM6258 sample rate.
The video side of this window is tools/encoder/extract.py; the arguments mean
the same things and are meant to be given the same values, because an audio
stream that is not the same seconds as the frames is not this project's audio.
The disc is AC-3 5.1 at 48 kHz. The arcade original is MONO, so this downmixes
-- ffmpeg's default matrix, dialogue from the centre channel included -- and
resamples to the chip's rate. Nothing here shapes, gates or normalises the
level: what the ADPCM encoder is handed is what the disc has, so that the SNR
it reports is the codec's and not a gain stage's.
"""
import getpass, os, subprocess, sys
BDROM = os.environ.get("DLX_BDROM") or f"/media/{getpass.getuser()}/BDROM"
STREAM_DIR = f"{BDROM}/BDMV/STREAM"
# 8 MHz / {512, 768, 1024}. The chip has no other rates and 15,625 is the one
# every budget in this project is written against (FINDINGS 52).
RATES = {15625: 512, 10417: 768, 7813: 1024}
def extract(stream, out, rate=15625, start=None, dur=None):
if rate not in RATES:
raise SystemExit(f"{rate} is not an MSM6258 rate: {sorted(RATES)}")
src = f"{STREAM_DIR}/{stream}.m2ts"
cmd = ["ffmpeg", "-v", "error"]
if start is not None: cmd += ["-ss", str(start)]
if dur is not None: cmd += ["-t", str(dur)]
cmd += ["-i", src, "-vn", "-ac", "1", "-ar", str(rate),
"-f", "s16le", "-acodec", "pcm_s16le", out, "-y"]
subprocess.check_call(cmd)
n = os.path.getsize(out) // 2
print(f"{stream}: {n} samples @ {rate} Hz mono = {n/rate:.3f} s -> {out}")
return n
if __name__ == "__main__":
# extract_audio.py <stream> <out.raw> [rate] [start_s] [dur_s]
stream, out = sys.argv[1], sys.argv[2]
rate = int(sys.argv[3]) if len(sys.argv) > 3 else 15625
start = float(sys.argv[4]) if len(sys.argv) > 4 else None
dur = float(sys.argv[5]) if len(sys.argv) > 5 else None
extract(stream, out, rate, start, dur)
tools/encoder/pack.py
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env python3
"""Encode one scene to the PACKED container -- ROADMAP K2.
python3 tools/encoder/pack.py <frames_dir> <out.dlxp> [--fps 12]
[--nframes N] [--palette-last] [--no-palette]
[--scene-palette] [--audio tmp/au_singe.raw]
WHAT IS NOT HERE IS THE POINT. No VQ, no codebooks, no mode map, no rate
control, no `lam`, no leaky bucket, no span geometry -- `encode.py` is 452 lines
and 95% of its wall clock is k-means. A packed frame is a palette and a
picture, and both are geometry. FINDINGS 61: at the 9 clk/B dual-address floor
the codec is 110.4% of a 12 fps frame and this is 55.2%, so the thing that
replaces the codec is also the thing that is simpler than it.
THERE IS NO RATE CONTROL BECAUSE THERE IS NO RATE LEVER. A codec's bitrate is
adjustable; a literal frame's is geometry. The wire cost of this container is
fixed by W, H and fps and nothing an encoder does can move it, which is exactly
why FINDINGS 61.6 says the medium question decides which player exists. A
`--kbps` argument here would be a lie of the shape FINDINGS 50 removed from the
rest of the tree.
THE QUANTISER IS THIS PROJECT'S, NOT PIL'S DEFAULT PATH. 61.9's +4.89 dB was
measured with `18_text_plane_16col.py`'s free 256-colour MEDIANCUT and was filed
as "a direction, not the player's number" (risk 2 in the session 30 handoff).
`vq.frame_palette` is what actually ships it: 254 colours, index 0 held free for
the transparency key, black at 255. `tools/analysis/30_packed_container.py`
re-derives the figure against this and charges the GRB555 word on top.
AND `--audio` MAKES IT A DLXP2, WHICH IS THE ONE PLACE THE FOUR ADPCM AXES ARE
CHOSEN. It encodes for `adpcm.CHIP` -- the datasheet's per-term delta, the LOW
nibble of a byte first, a 10-bit clamp, the accumulator at -2 -- because that is
the set FINDINGS 66 measured out of the machine's own chip through the machine's
own DMA channel, and encoding for any other set costs up to 25.7 dB. It does
NOT use adpcm.py's module defaults, which are ffmpeg's on purpose so that
tools/bench/verify_adpcm.py stays a check against an independent implementation.
The axes go in the header, so a player never has to be told.
"""
import argparse, glob, math, os, sys
import numpy as np
from PIL import Image
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)),
"..", "bench"))
import vq as VQ
import dlxp as P
import adpcm
from dlxload import pack_palette
ap = argparse.ArgumentParser()
ap.add_argument("frames_dir")
ap.add_argument("out")
ap.add_argument("--fps", type=int, default=12)
ap.add_argument("--nframes", type=int, default=None,
help="encode only the first N frames (the gate window is 120)")
ap.add_argument("--palette-last", action="store_true",
help="put the palette after the picture in every record. "
"FINDINGS 62.5 -- a design choice, not a default to inherit")
ap.add_argument("--no-palette", action="store_true",
help="picture only, 49,152 B a record. NOT a shipping option: "
"it re-imposes the scene palette the codec is capped by")
ap.add_argument("--scene-palette", action="store_true",
help="one palette for the whole scene, repeated in every record "
"-- the CONTROL for 61.9's per-frame claim")
ap.add_argument("--audio", default=None,
help="raw s16le mono at --audio-hz, from extract_audio.py. "
"Makes the output a DLXP2: the stream is encoded for "
"adpcm.CHIP and interleaved on the 65.3 cadence")
ap.add_argument("--audio-hz", type=int, default=15625,
help="the CHIP's rate, and the rate the .raw was resampled to")
ap.add_argument("--cadence", type=int, default=P.CADENCE_F,
help="frames between audio lumps (65.3's sweep picks 11)")
ap.add_argument("--audio-gain", type=float, default=1.0,
help="LEVEL, applied before the 12-bit requantisation. 1.0 is "
"s16>>4, the disc's own level, and it is the MEASURED "
"choice: FINDINGS 69 encoded windows drawn over the whole "
"game at six gains and every attenuation that buys "
"headroom under the chip's 10-bit clamp costs more SNR "
"than the clamping it avoids. Below 1.0 is a fallback")
a = ap.parse_args()
files = sorted(glob.glob(f"{a.frames_dir}/f*.png"))
if a.nframes:
files = files[:a.nframes]
if not files:
sys.exit(f"no f*.png in {a.frames_dir}")
rgb = [np.asarray(Image.open(f).convert("RGB")) for f in files]
H, W = rgb[0].shape[:2]
if any(r.shape[:2] != (H, W) for r in rgb):
sys.exit("frames are not all the same size")
# The scene-palette control shares ONE palette across every record, which is the
# constraint the codec cannot escape (61.9) and this format merely chooses not
# to inherit. It is built by the same routine the codec uses, with black at 0
# moved to 255 and index 0 vacated, so the only variable between the two runs is
# per-frame versus scene-wide.
scene = None
if a.scene_palette:
# 255, not 256: reserve_black spends one entry on black and the packed
# layout needs the OTHER end free too, so the picture gets 254 either way.
ref, spal = VQ.scene_palette(rgb, colors=255, reserve_black=True)
sidx = VQ.palettise(rgb, ref)
# 0 -> 255: the packed layout needs index 0 free and black displayed at 255.
spal = np.vstack([np.zeros((1, 3), np.uint8), spal[1:],
np.zeros((1, 3), np.uint8)])
scene = (spal, [np.where(i == 0, np.uint8(255), i) for i in sidx])
recs, psnr_pal = [], []
for n, src in enumerate(rgb):
if scene is not None:
pal, idx = scene[0], scene[1][n]
else:
pal, idx = VQ.frame_palette(src)
if (idx == 0).any():
sys.exit(f"frame {n}: index 0 is the transparency key and got used")
palw = None
if not a.no_palette:
palb, _dark, rendered = pack_palette(pal)
palw = palb.tobytes()
psnr_pal.append(VQ.psnr(src, pal[idx]))
recs.append((palw, P.pack_picture(idx).tobytes()))
# THE AUDIO, AND IT IS ENCODED FOR THE CHIP AND NOT FOR ffmpeg. The source is
# s16le because that is what ffmpeg resamples to; the chip's word is 12 bits
# signed, so the shift is a requantisation and not a format conversion, and it
# is the same one tools/bench/verify_adpcm.py makes.
audio = None
if a.audio:
import struct as _struct
raw = open(a.audio, "rb").read()
pcm = _struct.unpack("<%dh" % (len(raw) // 2), raw)
# THE LEVEL. `>> 4` maps the disc's full scale onto the 12-bit word and is
# what every container in this tree has been encoded at; the gain is a
# multiply BEFORE it, so gain 1.0 is byte-identical to what shipped. The
# chip clamps at 10 bits INSIDE the recursion (adpcm.CHIP), so anything the
# gain puts above 511 is unreachable -- and FINDINGS 69 measured that the
# attenuation which avoids that costs more than the clamping does.
g = a.audio_gain
src12 = [max(-2048, min(2047, int(math.floor(x * g)) >> 4)) for x in pcm]
lo12, hi12 = adpcm.clamp_bounds(adpcm.CHIP["bits"])
nclamp = sum(1 for v in src12 if v > hi12 or v < lo12)
need = len(files) * a.audio_hz // (2 * a.fps)
nib = adpcm.encode(src12, variant=adpcm.CHIP["variant"],
init=adpcm.CHIP["init"], bits=adpcm.CHIP["bits"])
data = adpcm.pack(nib, order=adpcm.CHIP["order"])[:need]
# What the encoder thinks it wrote, checked by DECODING it with the same
# four axes -- the encoder runs its decoder inside its own loop, so this is
# not circular in the way it looks: it is the PACKED bytes going back
# through unpack(), which is where a nibble-order slip would land.
back = adpcm.decode(adpcm.unpack(data, len(src12), order=adpcm.CHIP["order"]),
variant=adpcm.CHIP["variant"], init=adpcm.CHIP["init"],
bits=adpcm.CHIP["bits"])
ref = src12[:len(back)]
e = [(x - y) ** 2 for x, y in zip(ref, back)]
sig = sum(x * x for x in ref)
snr = 10 * np.log10(sig / sum(e)) if sum(e) else float("inf")
audio = dict(data=data, hz=a.audio_hz, F=a.cadence, **adpcm.CHIP)
rec_b = P.write(a.out, W, H, a.fps, recs, palette_last=a.palette_last,
audio=audio)
d = P.DLXP(a.out) # re-read: every invariant is checked
if d.nframes != len(recs):
sys.exit("writer and reader disagree about the frame count")
kind = ("scene palette" if a.scene_palette else "per-frame palette")
if a.no_palette:
kind += ", NONE in the record"
print(f"{a.out}: DLXP{P.VERSION} {W}x{H} {a.fps}fps {d.nframes} frames, {kind}"
f"{', palette LAST' if a.palette_last else ''}")
print(f" record {rec_b:,} B = {rec_b // P.SECTOR} sectors exactly, "
f"file {os.path.getsize(a.out):,} B")
if d.has_audio:
print(f" DLXP2: audio {d.aud_bytes:,} B at {d.aud_hz:,} Hz, SNR {snr:.2f} dB, "
f"cadence F={d.cad_f} A={d.cad_a} ({d.n_lumps} lumps)")
print(f" level gain {a.audio_gain:g}, source peak {max(abs(v) for v in src12)}"
f" of the chip's {hi12}: {nclamp:,} of {len(src12):,} samples "
f"({100*nclamp/len(src12):.4f}%) are above the clamp and cannot be "
f"reached (FINDINGS 69)")
print(f" the four axes, in the header: "
+ ", ".join(f"{k}={v}" for k, v in d.decoder().items()))
# STEADY STATE, not the file: the last lump of a 120-frame window feeds 4
# frames out of 11 and occupies 14 sectors either way, so a whole-file
# padding figure is a boundary effect of the WINDOW and would change with
# its length. 65.3's 0.09% is the cadence's, and the cadence is the thing.
grp = d.cad_f * d.aud_hz / (2 * d.fps)
print(f" lump payload {P.lump_bytes(0, d.cad_f, d.fps, d.aud_hz):,}.."
f"{P.lump_bytes(2, d.cad_f, d.fps, d.aud_hz):,} B of {d.cad_a*P.SECTOR:,} "
f"-- {100*(d.cad_a*P.SECTOR-grp)/grp:.3f}% padding steady state, and the "
f"payload is NOT the lump (FINDINGS 67)")
print(f" wire {d.video_kbps():.1f} + {d.audio_kbps():.2f} = {d.kbps():.1f} KB/s "
f"-- FIXED by geometry, there is no lever")
else:
print(f" wire {d.kbps():.1f} KB/s -- FIXED by geometry, there is no lever")
print(f" palette-domain PSNR vs the 24-bit source: "
f"{np.mean(psnr_pal):.2f} dB (min {np.min(psnr_pal):.2f})")
tools/encoder/ratectl.py
+59
View File
@@ -157,6 +157,23 @@ JOINT_DECIDE = False
# `--joint-bucket` turns it on.
JOINT_BUCKET = False
# E3 / FINDINGS 39.3 item 5: SPAN SELECTION IS GREEDY AFTER `lam`, and this is
# the switch that makes the two joint. The lam bisection picks a mode map
# against a byte allowance, and the span pass then REMOVES the block payload of
# every block it covers -- so the frame lands under the allowance by exactly
# the bytes the spans freed, and the blocks that were NOT spanned were priced
# at a lam chosen as if those bytes were still needed. Joint mode hands the
# freed bytes back to the lam search and re-spans the result, to a fixed point
# or two rounds, whichever comes first.
#
# It is a REFINEMENT, not a different objective: lam can only fall (the
# allowance only grows), so the un-spanned blocks can only improve, and a round
# is kept only if the frame still fits both ceilings it was already fitting.
# Default OFF until measured, which is 44.3's lesson -- ask whether the lever
# is loaded before pulling it.
JOINT_SPANS = False
JOINT_SPAN_ROUNDS = 2
def _byte_clk():
"""The debit the mode decision is allowed to see (0 = the old decision)."""
@@ -287,6 +304,39 @@ def _fit_spans(m, ctx, mode, sz, room, cyc_budget, span_mode, ib):
return nmode, nsz, H.cycles(nmode) + sel["clocks"], sel
def _refit_joint(m, ctx, allow, span_allow, lam_lo, lam_hi, cyc_budget,
span_mode, ib, mode_pre, mode, sz, cyc, sel, mu=0.0):
"""Give the lam search back the bytes the span pass freed, then re-span.
`mode_pre` is the mode map BEFORE spanning and `mode` the one after, so the
difference in frame_bytes is exactly what the spans made unnecessary. The
ceiling the result is judged against is the one _fit_spans was already
working to, so a kept round is never a frame that grew past a budget it was
inside.
"""
ceiling = span_allow if span_allow is not None else allow
for _ in range(JOINT_SPAN_ROUNDS):
if sel is None:
break
freed = (H.frame_bytes(mode_pre, ctx["nb"], ib)
- H.frame_bytes(mode, ctx["nb"], ib))
if freed <= 0:
break
lam2, mode2, sz2, _ = _search_lam(ctx, allow + freed, lam_lo, lam_hi, mu=mu)
if sz2 <= H.frame_bytes(mode_pre, ctx["nb"], ib):
break # lam did not move: already at the floor
n_pre, n_mode, n_sz, n_cyc, n_sel = (
mode2, *_fit_spans(m, ctx, mode2, sz2, span_allow if span_allow
is not None else allow, cyc_budget, span_mode, ib))
if n_sel is None or n_sz > ceiling:
break
if cyc_budget is not None and n_cyc + DISK_CLK_BYTE * n_sz > cyc_budget \
and cyc + DISK_CLK_BYTE * sz <= cyc_budget:
break # round 1 made the deadline and this does not
mode_pre, mode, sz, cyc, sel = n_pre, n_mode, n_sz, n_cyc, n_sel
return mode_pre, mode, sz, cyc, sel
def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
lam_lo=1.0, lam_hi=LAM_CLIFF, prefill=0.0,
steps=None, verbose=False, cycle_budget=None,
@@ -364,6 +414,10 @@ def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
mode_pre = mode
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
cycle_budget, span_mode, ib)
if JOINT_SPANS:
mode_pre, mode, sz, cyc, sel = _refit_joint(
m, ctx, allow, span_budget and span_allow, lam_lo, lam_hi,
cycle_budget, span_mode, ib, mode_pre, mode, sz, cyc, sel)
if cycle_budget is not None and cyc + DISK_CLK_BYTE * sz > cycle_budget:
# The byte allowance could not buy the frame's deadline, so fall
# back to the controller that pays in picture -- and then offer
@@ -374,6 +428,11 @@ def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
mode_pre = mode
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
cycle_budget, span_mode, ib)
if JOINT_SPANS:
mode_pre, mode, sz, cyc, sel = _refit_joint(
m, ctx, allow, span_budget and span_allow, lam_lo,
lam_hi, cycle_budget, span_mode, ib, mode_pre, mode,
sz, cyc, sel, mu=mu)
late = cyc + DISK_CLK_BYTE * sz > cycle_budget
# Paint from the mode map as it was BEFORE spanning. A spanned run's
# blocks read SKIP in the emitted header, but SKIP means "hold the
tools/encoder/vq.py
+68 -5
View File
@@ -28,15 +28,78 @@ def load_frames(d):
return [np.asarray(Image.open(f).convert("RGB")) for f in fs]
def scene_palette(rgb, colors=256, stride=3):
"""One shared palette for the whole scene, no dithering (cel art is flat)."""
def scene_palette(rgb, colors=256, stride=3, reserve_black=True):
"""One shared palette for the whole scene, no dithering (cel art is flat).
`reserve_black` puts TRUE BLACK at index 0 and quantises the picture into
the other 255 entries. It is not a cosmetic default (FINDINGS 23.4): the
picture is 256x192 inside a 256x256 mode, GVRAM cleared to zero displays
palette entry 0, and a free mediancut palette puts a real image colour
there -- on 00020 f0001 it was (206,192,176), used by 210 image pixels, so
the 64 blank rows of letterbox came out beige. Entry 0 also needs `I = 0`
in the X68000's GRB555 word or the bars sit at RGB (4,4,4) (23.3); that
half is `dlxload.pack_palette`'s and it needs no special case, because a
(0,0,0) entry picks I=0 by its own minimum-squared-error rule.
Black is RESERVED, not withheld: the mapper may still spend index 0 on
genuinely black pixels, which is the entry it would have wanted anyway.
What the reservation buys is that index 0 is black REGARDLESS of what the
scene contains, which is what the letterbox needs and what a free palette
cannot promise.
"""
samp = np.concatenate([r.reshape(-1, 3) for r in rgb[::stride]])
ref = Image.fromarray(samp.reshape(-1, 1, 3)).quantize(
colors=colors, method=Image.MEDIANCUT, dither=Image.NONE)
pal = np.array(ref.getpalette()[:colors * 3], dtype=np.uint8).reshape(-1, 3)
n = colors - 1 if reserve_black else colors
q = Image.fromarray(samp.reshape(-1, 1, 3)).quantize(
colors=n, method=Image.MEDIANCUT, dither=Image.NONE)
pal = np.array(q.getpalette()[:n * 3], dtype=np.uint8).reshape(-1, 3)
if not reserve_black:
return q, pal
pal = np.vstack([np.zeros((1, 3), np.uint8), pal])
# The quantiser above cannot be reused as the mapping reference: its
# palette is the 255 it chose, at the wrong indices. A P-mode image
# carrying the FINAL palette is what every frame is then mapped against,
# so the indices in the container and the entries in the container's
# palette section are the same table by construction.
ref = Image.new("P", (1, 1))
ref.putpalette(pal.tobytes().ljust(768, b"\0"))
return ref, pal
def frame_palette(rgb1, colors=254):
"""`scene_palette`'s sibling, for the PACKED layout: ONE FRAME, 254 colours.
The codec cannot have this. Every codeword it emits is an index INTO
`scene_palette`, so its palette is shared scene-wide and 31.33 dB is a
ceiling no bitrate crosses (FINDINGS 61.9). A literal frame has no
codebooks, so nothing forces a shared palette on it.
The layout spends TWO entries where `--reserve-black` spends one (47.2):
index 0 is the TRANSPARENCY KEY of the top graphics page and must never
appear in the picture, and black therefore lives at 255 for the letterbox.
So the picture gets 254.
The +1 shift is the whole mechanism, and it is why this does NOT go through
a P-mode reference image the way `scene_palette` does. `palettise` maps
against the FINAL 256-entry table, and that table has (0,0,0) at both 0 and
255 -- a nearest-colour mapper is free to pick either, and there is no way to
forbid the one that must stay unused. Quantising to 254 and shifting keeps
index 0 free BY CONSTRUCTION rather than by hoping the mapper agrees, and it
is still exact: `pal[idx]` reproduces the quantiser's own rendering.
Returns (pal (256,3) uint8, idx (H,W) uint8 in 1..254).
"""
q = Image.fromarray(rgb1).quantize(colors=colors, method=Image.MEDIANCUT,
dither=Image.NONE)
raw = q.getpalette()
if len(raw) < colors * 3:
raise ValueError(f"quantiser returned {len(raw)//3} entries, wanted {colors}")
pal = np.array(raw[:colors * 3], dtype=np.uint8).reshape(-1, 3)
pal = np.vstack([np.zeros((1, 3), np.uint8), pal, np.zeros((1, 3), np.uint8)])
idx = np.asarray(q, dtype=np.uint8) + np.uint8(1)
if idx.min() < 1 or idx.max() > colors:
raise ValueError("index 0 (transparency key) or 255 (black) got used")
return pal, idx
def palettise(rgb, ref):
return [np.asarray(Image.fromarray(r).quantize(palette=ref, dither=Image.NONE),
dtype=np.uint8) for r in rgb]
tools/encoder/vq_hybrid.py
+2 -2
View File
@@ -113,10 +113,10 @@ def blocks_of(idx, pal, bw, bh):
return VQ.blockify(idx, pal, bw, bh)
def build(frames_dir, k1=256, k4=256, iters=16, lam=0.0):
def build(frames_dir, k1=256, k4=256, iters=16, lam=0.0, reserve_black=True):
rgb = VQ.load_frames(frames_dir)
H, W = rgb[0].shape[:2]
ref, pal = VQ.scene_palette(rgb)
ref, pal = VQ.scene_palette(rgb, reserve_black=reserve_black)
idx = VQ.palettise(rgb, ref)
# --- two codebooks, trained on the whole scene ---
tools/import/scenegraph.py
+436
View File
@@ -0,0 +1,436 @@
"""The ONE file in this tree that knows anything about somebody else's source.
Everything downstream -- `tools/analysis/25_scene_graph.py` and whatever the
game-logic layer eventually becomes -- reads the neutral table this writes and
knows nothing about where it came from. That is deliberate. Two outside
projects are read here, neither is vendored, neither ships in this repo, and
their file layouts, table names, timing formulas and magic constants are wired
into THIS file and nowhere else. When one of them moves, one file breaks.
python3 tools/import/scenegraph.py [-o tmp/scenegraph.json]
Sources, both permissively licensed and both cloned by the reader:
icculus/DirkSimple -- zlib, Ryan C. Gordon. `data/games/lair/game.lua` holds
the arcade scene graph, transcribed from the arcade ROM's own data table.
THE source; the measurement is made on this.
DLX_DIRKSIMPLE=/path (default tmp/scenegraph/DirkSimple)
astrobleem/SNES-SuperDragonsLairArcade -- MIT, Chad Doebelin.
`data/events/*.xml` holds 516 chapter definitions. Read as a CROSS-CHECK
only, and a weak one: their own README states the chapters are "derived
from DirkSimple game data", so this is a second COPY, not a second
transcription (FINDINGS 56.2). Optional; skipped when absent.
DLX_SNESLAIR=/path (default tmp/scenegraph/SNES-SuperDragonsLairArcade)
WHAT COMES OUT is `DLXSCENE1`, our own schema, in one JSON file:
{"format": "DLXSCENE1",
"sources": [{"name", "url", "licence", "holder", "role"} ...],
"scenes": {scene: {sequence: {
"start_ms": float, absolute position on the medium, or -1 for
"keep playing from where the disc already is",
"timeout_ms": float, how long the clip runs unbranched,
"kills", "single_frame": bool,
"timeout_next": sequence name or null,
"actions": [{"input", "from_ms", "to_ms", "next"}]}}},
"rows": [[scene, scene, scene] ...], the arcade's 13x3 scene order
"counts": {"scenes", "sequences", "windows"},
"crosscheck": {scene: {sequence: {"start_ms", "end_ms", "actions": [...],
"chapter"}}} or null}
Nothing in the schema is a copy of either project's structure: it is what this
project's own analysis needs, which is a clip's length, its exits and the
earliest instant each of them can fire.
If a scene table is ever COMMITTED to this repo rather than regenerated into
gitignored tmp/, it becomes redistribution of derived data and the attribution
in "sources" has to travel with it. FINDINGS 16, 56.5.
"""
import sys, os, re, json, argparse
import xml.etree.ElementTree as ET
DIRK = os.environ.get("DLX_DIRKSIMPLE", "tmp/scenegraph/DirkSimple")
SNES = os.environ.get("DLX_SNESLAIR", "tmp/scenegraph/SNES-SuperDragonsLairArcade")
# DirkSimple's own constants, from data/games/lair/game.lua. Restated here
# because this file evaluates its table; both are gated below against the text
# of the file they came from, so they cannot go stale silently.
LD_FPS = 23.976 # laserdisc_frame_to_ms
ROM_OFFSET_MS = 6297.0 # time_laserdisc_frame's "magic millisecond offset"
SOURCES = [
{"name": "icculus/DirkSimple",
"url": "https://github.com/icculus/DirkSimple",
"licence": "zlib", "holder": "Ryan C. Gordon",
"role": "the scene graph itself, transcribed from the arcade ROM"},
{"name": "astrobleem/SNES-SuperDragonsLairArcade",
"url": "https://github.com/astrobleem/SNES-SuperDragonsLairArcade",
"licence": "MIT", "holder": "Chad Doebelin",
"role": "cross-check only; itself derived from DirkSimple (FINDINGS 56.2)"},
]
# ---------------------------------------------------------------- Lua subset
class LuaParser:
"""Just enough Lua to read game.lua's `scenes` table constructor.
Not a Lua interpreter and not trying to be: the grammar is table
constructors, string/number/boolean/nil literals, bare identifiers (used
only for function references like `interrupt=game_over_complete`), calls to
four known helpers, and + / - between them. Anything else is a parse error
rather than a silent skip, and the caller asserts the whole table was
consumed, so a change upstream shows up as a failure and not as a smaller
scene graph.
"""
def __init__(self, text, helpers):
self.s, self.i, self.helpers = text, 0, helpers
def error(self, msg):
line = self.s.count("\n", 0, self.i) + 1
raise SyntaxError(f"{msg} at line {line}: {self.s[self.i:self.i+60]!r}")
def ws(self):
while self.i < len(self.s):
c = self.s[self.i]
if c in " \t\r\n":
self.i += 1
elif self.s.startswith("--", self.i):
self.i = self.s.find("\n", self.i)
if self.i < 0:
self.i = len(self.s)
else:
return
def take(self, lit):
self.ws()
if self.s.startswith(lit, self.i):
self.i += len(lit)
return True
return False
def expect(self, lit):
if not self.take(lit):
self.error(f"expected {lit!r}")
def name(self):
self.ws()
m = re.compile(r"[A-Za-z_][A-Za-z0-9_]*").match(self.s, self.i)
if not m:
return None
self.i = m.end()
return m.group(0)
def value(self):
self.ws()
if self.take("{"):
return self.table()
m = re.compile(r"-?\d+(\.\d+)?").match(self.s, self.i)
if m:
self.i = m.end()
v = float(m.group(0))
return self.arith(int(v) if v.is_integer() else v)
if self.s[self.i] in "\"'":
q = self.s[self.i]
j = self.s.index(q, self.i + 1)
v = self.s[self.i + 1:j]
self.i = j + 1
return v
n = self.name()
if n is None:
self.error("expected a value")
if n == "nil":
return None
if n in ("true", "false"):
return n == "true"
if self.take("("): # a call
args = []
if not self.take(")"):
while True:
args.append(self.value())
if self.take(")"):
break
self.expect(",")
if n not in self.helpers:
self.error(f"unknown helper {n!r}")
return self.arith(self.helpers[n](*args))
return Symbol(n) # a function reference
def arith(self, left):
"""+ and - between numbers. Present in the source and load-bearing:
e.g. `time_laserdisc_frame(1823) - laserdisc_frame_to_ms(2)`."""
while True:
self.ws()
if self.i < len(self.s) and self.s[self.i] in "+-":
op = self.s[self.i]
self.i += 1
right = self.value()
left = left + right if op == "+" else left - right
else:
return left
def table(self):
d, arr = {}, []
while True:
self.ws()
if self.take("}"):
break
save = self.i
k = self.name()
if k is not None and self.take("="):
d[k] = self.value()
else:
self.i = save
arr.append(self.value())
if not (self.take(",") or self.take(";")):
self.expect("}")
break
if d and arr:
self.error("mixed array/hash table")
return arr if arr or not d else d
class Symbol(str):
"""A bare Lua identifier used as a value (a function reference)."""
def load_dirksimple(path):
"""Parse lair/game.lua's `scenes` and `scene_manager` tables.
Gates the four timing helpers against the text that defines them, so this
tool cannot keep evaluating a formula upstream has changed.
"""
src = open(os.path.join(path, "data/games/lair/game.lua"), encoding="utf-8").read()
def gate(pat, what):
if not re.search(pat, src):
raise AssertionError(f"DirkSimple's {what} is not what this tool "
f"evaluates any more (looked for {pat!r})")
gate(r"frame\s*/\s*23\.976", "laserdisc_frame_to_ms")
gate(r"-\s*6297\.0", "time_laserdisc_frame's ROM offset")
gate(r"time_laserdisc_noseek.*?\n\s*return -1", "time_laserdisc_noseek")
gate(r"\(seconds \* 1000\) \+ ms", "time_to_ms")
helpers = {
"laserdisc_frame_to_ms": lambda f: (f / LD_FPS) * 1000.0,
"time_laserdisc_frame": lambda f: (f / LD_FPS) * 1000.0 - ROM_OFFSET_MS,
"time_laserdisc_noseek": lambda: -1,
# time_to_ms takes (seconds, ms); the source calls it with a third
# argument exactly once, which Lua discards.
"time_to_ms": lambda s, ms=0, *_: s * 1000 + ms,
}
out = {}
for var in ("scenes", "scene_manager"):
m = re.search(rf"^{var} = \{{", src, re.M)
if not m:
raise AssertionError(f"{var} table not found in game.lua")
p = LuaParser(src, helpers)
p.i = m.end()
out[var] = p.table()
# The parser must have consumed the WHOLE constructor: table() returns
# with the cursor just past the matching close brace, and in this file
# every top-level table ends on a `}` in column 0. A parser that
# stopped early would return a smaller graph, which is the direction
# that flatters the measurement, so this is checked rather than assumed.
if src[p.i - 1] != "}":
raise AssertionError(f"{var}: parser did not end on a close brace")
if src.rfind("\n", 0, p.i) != p.i - 2:
raise AssertionError(f"{var}: the constructor ended mid-line at "
f"{p.i}, so the parse stopped early")
return out["scenes"], out["scene_manager"]
# ------------------------------------------------- the SNES cross-check
def load_snes(path):
"""Parse the SNES project's chapter XMLs into {chapter: (start_ms, end_ms,
[(input, from_ms, to_ms, target)])}, plus the provenance line."""
ev = os.path.join(path, "data/events")
readme = os.path.join(ev, "README.md")
prov = None
if os.path.exists(readme):
txt = open(readme, encoding="utf-8").read()
if "derived from DirkSimple" in txt:
prov = "derived from DirkSimple game data (data/events/README.md)"
def ms(e):
return (int(e.get("min", 0)) * 60000 + int(e.get("second", 0)) * 1000
+ int(e.get("ms", 0)))
chapters = {}
for f in sorted(os.listdir(ev)):
if not f.endswith(".xml"):
continue
root = ET.parse(os.path.join(ev, f)).getroot()
tl = root.find("timeline")
s = tl.find("timestart")
e = tl.find("timeend")
acts = []
for evt in root.findall("./events/event"):
if evt.get("type") != "direction":
continue
t = evt.find("timeline")
p = evt.find("./params/str[@key='type']")
r = evt.find("./result/playchapter")
acts.append((p.get("value") if p is not None else "?",
ms(t.find("timestart")),
ms(t.find("timeend")) if t.find("timeend") is not None else -1,
r.get("name") if r is not None else None))
chapters[f[:-4]] = (ms(s) if s is not None else -1,
ms(e) if e is not None else -1, acts)
return chapters, prov
def map_abbrevs(chapters, scenes):
"""Match each SNES scene abbreviation to a DirkSimple scene from the DATA,
not from a hand-written table, so a wrong match is visible rather than
assumed. Three signals, in order:
1. the abbreviation's letters must be a SUBSEQUENCE of the scene name
(`snkr` is snake_room, and cannot be black_knight however similar
their `seqN` names are -- overlap alone got that one wrong);
2. the Jaccard overlap of the two sequence-name sets;
3. the reversed-scene rule: Dragon's Lair mirrors thirteen scenes, and
where BOTH `x` and `xr` exist as abbreviations, `xr` is the
`_reversed` scene and `x` is not. Their sequence names are identical,
so nothing else separates them.
Assignment is 1:1 and greedy on that ranking.
"""
def subseq(a, b):
it = iter(b)
return all(c in it for c in a)
by_abbr = {}
for ch in chapters:
abbr, _, seq = ch.partition("_")
by_abbr.setdefault(abbr, set()).add(seq)
pairs = []
for abbr, seqs in by_abbr.items():
mirror = abbr.endswith("r") and abbr[:-1] in by_abbr
for scene, s in scenes.items():
if not subseq(abbr, scene.replace("_", "")):
continue
union = len(seqs | set(s))
j = len(seqs & set(s)) / union if union else 0.0
rev = scene.endswith("_reversed")
pairs.append((j, 1 if rev == mirror else 0, -len(scene),
abbr, scene, len(seqs)))
pairs.sort(reverse=True)
mapping, used = {}, set()
for j, _, _, abbr, scene, n in pairs:
if abbr in mapping or scene in used:
continue
mapping[abbr] = (scene, j, n)
used.add(scene)
unmatched = [(a, None, 0.0, len(s)) for a, s in by_abbr.items()
if a not in mapping]
return mapping, unmatched
# ------------------------------------------------------------------- emit
def to_neutral(scenes, mgr):
"""DirkSimple's tables -> DLXSCENE1. The one place their shape is read."""
out = {}
for scene, seqs in scenes.items():
s = out.setdefault(scene, {})
for name, seq in seqs.items():
t = seq["timeout"]
s[name] = {
"start_ms": float(seq["start_time"]),
"timeout_ms": float(t["when"]),
"timeout_next": t.get("nextsequence"),
"kills": bool(seq.get("kills_player")),
"single_frame": bool(seq.get("is_single_frame")),
"actions": [{"input": a["input"],
"from_ms": float(a["from"]),
"to_ms": float(a["to"]),
"next": a.get("nextsequence")}
for a in seq.get("actions", [])],
}
rows = [list(r) for r in mgr["rows"]]
return out, rows
def crosscheck_neutral(chapters, scenes):
"""The SNES chapters, keyed the way OUR table is keyed.
Their `abbr_sequence` naming is resolved to a scene here, from the data
(see map_abbrevs), so nothing downstream has to know an abbreviation
exists. Chapters the mapping cannot place are dropped and counted.
"""
mapping, unmatched = map_abbrevs(chapters, scenes)
out, dropped = {}, 0
for ch, (s_ms, e_ms, acts) in chapters.items():
abbr, _, seq = ch.partition("_")
if abbr not in mapping:
dropped += 1
continue
scene = mapping[abbr][0]
pre = abbr + "_"
out.setdefault(scene, {})[seq] = {
"chapter": ch,
"start_ms": float(s_ms),
"end_ms": float(e_ms),
"actions": [{"input": i, "from_ms": float(f), "to_ms": float(t2),
"next": (n[len(pre):] if n and n.startswith(pre) else n)}
for i, f, t2, n in acts],
}
weak = sorted((round(j, 3), ab, sc) for ab, (sc, j, _n) in mapping.items()
if j < 1.0)
return out, {"chapters": len(chapters), "dropped": dropped,
"scenes_mapped": len(mapping),
"weakest_mappings": weak[:6]}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("-o", "--out", default="tmp/scenegraph.json")
a = ap.parse_args()
if not os.path.isdir(DIRK):
sys.exit(f"no DirkSimple checkout at {DIRK} -- set DLX_DIRKSIMPLE, or\n"
f" git clone --depth 1 https://github.com/icculus/DirkSimple")
scenes, mgr = load_dirksimple(DIRK)
neutral, rows = to_neutral(scenes, mgr)
nseq = sum(len(s) for s in neutral.values())
nwin = sum(len(q["actions"]) for s in neutral.values() for q in s.values())
cross, cross_meta = None, None
if os.path.isdir(SNES):
chapters, prov = load_snes(SNES)
cross, cross_meta = crosscheck_neutral(chapters, scenes)
cross_meta["provenance"] = prov
# 56.2: this is the sentence the whole "second source" claim turned on.
# If upstream ever removes it, say so rather than silently promoting a
# copy back to an independent transcription.
if not prov:
cross_meta["provenance"] = ("UNSTATED -- data/events/README.md no "
"longer says where the chapters came "
"from; re-check before trusting this "
"as anything.")
doc = {"format": "DLXSCENE1", "sources": SOURCES,
"scenes": neutral, "rows": rows,
"counts": {"scenes": len(neutral), "sequences": nseq,
"windows": nwin},
"crosscheck": cross, "crosscheck_meta": cross_meta}
os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True)
with open(a.out, "w") as f:
json.dump(doc, f, indent=1, sort_keys=True)
print(f"{a.out}: {len(neutral)} scenes, {nseq} sequences, {nwin} input "
f"windows, {len(rows)} rows"
+ (f", cross-check {cross_meta['chapters']} chapters"
if cross_meta else ", no cross-check (SNES tree absent)"))
return 0
if __name__ == "__main__":
sys.exit(main())
tools/media/make_packed_media.py
+235
View File
@@ -0,0 +1,235 @@
#!/usr/bin/env python3
"""The README still for the PACKED player -- ROADMAP K3, FINDINGS 64.
python3 tools/media/make_packed_media.py [container.dlxp]
[--snap tmp/snap_packed_gate] [--map tmp/packed_snaps_gate.csv]
[--src tmp/fr_singe] [--frame N] [--out docs/img/packed-player.png]
Blu-ray source | what the emulated 68000 actually put on screen. The right-hand
panel is MAME's own snapshot, de-double-scanned and cropped to the picture -- not
a re-render, not `dlxp.render`. It is the same rule the codec's still is built on
(tools/media/make_readme_media.py) and it is the only reason the picture is worth
printing: an encoder can be checked against its own inverse, and a screen cannot.
THE FRAME IS CHOSEN, NOT PICKED. --frame defaults to the one whose PSNR against
the source is CLOSEST TO THE MEAN over the whole gated window, so the still is
representative rather than flattering. The chosen frame and its distance from the
mean are printed, so a reader can see it was not the best one.
"""
import argparse, csv, os, shutil, struct, subprocess, sys, wave
sys.path.insert(0, "tools/encoder")
import numpy as np
from PIL import Image, ImageDraw
from dlxp import DLXP
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/packed_singe.dlxp")
ap.add_argument("--snap", default="tmp/snap_packed_gate")
ap.add_argument("--map", default="tmp/packed_snaps_gate.csv")
ap.add_argument("--src", default="tmp/fr_singe")
ap.add_argument("--frame", type=int, default=None)
ap.add_argument("--out", default="docs/img/packed-player.png")
ap.add_argument("--webm", default="docs/img/packed-player.webm")
ap.add_argument("--wav", default="tmp/packed_aud.wav",
help="MAME's own -wavwrite capture from run 5 of "
"packed_run.sh; --no-audio drops it")
ap.add_argument("--no-audio", action="store_true")
ap.add_argument("--no-webm", action="store_true")
ap.add_argument("--fps", type=float, default=None,
help="clip rate; defaults to the container's own")
a = ap.parse_args()
d = DLXP(a.container)
SNAP_W, SNAP_H = 256, 512
with open(a.map) as fh:
shot = {int(r["frame"]): r["snapshot"] for r in csv.DictReader(fh)}
if not shot:
sys.exit(f"{a.map} is empty -- run tools/bench/packed_run.sh first")
def screen(fr):
"""The 256x192 picture out of one MAME native snapshot."""
p = f"{a.snap}/x68000/{shot[fr]}.png"
s = np.asarray(Image.open(p).convert("RGB"))
if s.shape[:2] != (SNAP_H, SNAP_W):
sys.exit(f"{p}: expected {SNAP_W}x{SNAP_H}, got {s.shape[1]}x{s.shape[0]}")
g = s[0::2] # undo the double scan
y = (g.shape[0] - d.H) // 2 # the picture is centred
return g[y:y + d.H]
def source(fr):
p = f"{a.src}/f{fr+1:04d}.png"
if not os.path.exists(p):
sys.exit(f"missing {p} -- re-extract the frames the container was built "
f"from, or point --src at them")
return np.asarray(Image.open(p).convert("RGB"))
def psnr(x, y):
e = ((x.astype(float) - y.astype(float)) ** 2).mean()
return float("inf") if e == 0 else 10 * np.log10(255.0 ** 2 / e)
frames = sorted(shot)
scores = {f: psnr(source(f), screen(f)) for f in frames}
mean = float(np.mean(list(scores.values())))
if a.frame is None:
pick = min(scores, key=lambda f: abs(scores[f] - mean))
else:
pick = a.frame
if pick not in scores:
sys.exit(f"frame {pick} was not sampled by that run")
# THE PANEL IS GATED, not just drawn. A still of the player is a claim that the
# player drew it, and the snapshot has to still be pixel-exact against the
# container for that claim to hold -- verify_packed.py checks all of them and
# this checks the one being printed, so the picture cannot outlive the result.
ref = d.render(pick)
if not np.array_equal(screen(pick), ref):
sys.exit(f"frame {pick} is NOT pixel-exact against {a.container}. The still "
f"is not being written: it would be a picture of a failure with a "
f"caption saying otherwise.")
Z, BAR = 2, 22
def captioned(img, text):
up = np.repeat(np.repeat(img, Z, 0), Z, 1)
out = Image.new("RGB", (up.shape[1], up.shape[0] + BAR), (16, 16, 18))
out.paste(Image.fromarray(up), (0, BAR))
ImageDraw.Draw(out).text((6, 6), text, fill=(190, 190, 196))
return out
left = captioned(source(pick), "Blu-ray source, cropped 256x192")
right = captioned(screen(pick),
"emulated 68000, MAME's own snapshot, no decoder")
out = Image.new("RGB", (left.width + right.width + 8, left.height), (16, 16, 18))
out.paste(left, (0, 0))
out.paste(right, (left.width + 8, 0))
os.makedirs(os.path.dirname(a.out), exist_ok=True)
out.save(a.out)
print(f"{a.out}: frame {pick} of {d.nframes}, {scores[pick]:.2f} dB against the "
f"24-bit source")
print(f" chosen as the frame CLOSEST TO THE MEAN ({mean:.2f} dB over "
f"{len(frames)} gated frames), {abs(scores[pick]-mean):.3f} dB from it -- "
f"best in the window is {max(scores.values()):.2f}, worst "
f"{min(scores.values()):.2f}")
print(f" and it is pixel-exact against {a.container}, checked before writing")
# ---------------------------------------------------------------------------
# THE CLIP. ROADMAP K3's result is a SEQUENCE -- 120 records off a real volume,
# every one of them a literal -- and a still cannot show the one property that
# distinguishes this branch from the codec's: there is no recursion here, so
# frame 119 says nothing about frame 60 and every frame has to be its own claim.
# This writes all of them, and gates all of them before writing any (64.1).
#
# WHAT IT IS, EXACTLY, because two runs of packed_run.sh are in it:
# picture run 1, the GATE run -- paced at half the container's rate so the
# snapshot lands inside the write window, cycle stealing, no sound.
# Every panel is MAME's own snapshot, 2x nearest, no filtering.
# sound run 5, the AUDIO run -- the same container at 12 fps with the
# MSM6258 on channel 3, captured by MAME's -wavwrite off the
# speaker. It is the chip's stream, not the encoder's.
# They are two runs because they have to be: the gate run is at 6 fps and audio
# cut at 12 fps played at 6 is not this scene. The clip is therefore a
# COMPOSITE, and saying so is the point -- what it is NOT is a real-time capture
# of the shipping configuration, which at the container's own burst rate would
# show a blank layer for 99.5% of every slot (FINDINGS 64.2).
if not a.no_webm:
if not shutil.which("ffmpeg"):
sys.exit("ffmpeg not found -- needed for the webm (--no-webm skips it)")
fps = a.fps or d.fps
# EVERY frame gated, not the printed one. A packed frame is a literal: the
# codec's last-frame test audits 120 through its own recursion and nothing
# here does, so a clip of 120 frames is 120 separate claims.
bad = [f for f in frames if not np.array_equal(screen(f), d.render(f))]
if bad:
sys.exit(f"{len(bad)} of {len(frames)} frames are NOT pixel-exact "
f"against {a.container} (first {bad[0]}). The clip is not "
f"being written: it would be a recording of a failure.")
aud = None
if not a.no_audio and d.has_audio and os.path.exists(a.wav):
# THE ALIGNMENT IS FOUND, NOT ASSUMED, and then CHECKED. The capture
# opens with the machine booting, so the stream starts at the first
# non-zero sample -- and "first non-zero" is exactly the kind of thing
# that is off by one byte forever with every counter in the player
# agreeing (FINDINGS 71.5). So lump 0 is decoded with the FOUR AXES OUT
# OF THE CONTAINER'S OWN HEADER and required to be sample-exact from
# there. Only lump 0: past it the seams need the walk in
# tools/bench/verify_packed_audio.py, which is what gates the whole
# stream in check.sh. This gates the cut.
sys.path.insert(0, "tools/encoder")
import adpcm
SCALE = 8 # okim6258's signal<<4 at gain 0.50
w = wave.open(a.wav)
nfr, ch, rate, sw = (w.getnframes(), w.getnchannels(),
w.getframerate(), w.getsampwidth())
if rate != d.aud_hz or sw != 2:
sys.exit(f"{a.wav}: {rate} Hz / {sw*8}-bit -- the capture has to be "
f"the chip's own {d.aud_hz} Hz or a resampler is in the "
f"measurement")
raw = w.readframes(nfr)
left = struct.unpack("<%dh" % (nfr * ch), raw)[0::ch]
rec = [round(v / SCALE) for v in left]
start = next((i for i, v in enumerate(rec) if v), None)
if start is None:
sys.exit(f"{a.wav} is silent -- run 5 of packed_run.sh writes it")
dec = d.decoder()
l0 = d.lump(0)
want = adpcm.decode(adpcm.unpack(l0, order=dec["order"]),
variant=dec["variant"], init=dec["init"],
bits=dec["bits"])
got = rec[start:start + len(want)]
if got != list(want):
n = sum(x != y for x, y in zip(got, want))
sys.exit(f"the cut at capture sample {start:,} does not decode as "
f"lump 0: {n:,} of {len(want):,} samples differ. The clip "
f"is not being written -- the sound would be the right "
f"scene from the wrong byte.")
n_out = int(round(len(frames) / fps * rate))
aud = "tmp/_packed_media_audio.wav"
ow = wave.open(aud, "wb")
ow.setnchannels(ch); ow.setsampwidth(sw); ow.setframerate(rate)
ow.writeframes(raw[start * ch * sw:(start + n_out) * ch * sw])
ow.close()
print(f" audio: {a.wav}, cut at sample {start:,} "
f"({start/rate:.2f} s of boot dropped), {n_out/rate:.2f} s -- "
f"lump 0 sample-exact against the container's own axes "
f"({dec['variant']}/{dec['order']}, {dec['bits']}-bit, "
f"init {dec['init']})")
elif not a.no_audio:
print(f" no audio: {a.wav} is missing or the container is silent")
tmpd = "tmp/_packed_media_frames"
shutil.rmtree(tmpd, ignore_errors=True)
os.makedirs(tmpd)
for n, f in enumerate(frames):
l = captioned(source(f), "Blu-ray source, cropped 256x192")
r = captioned(screen(f), "emulated 68000, MAME's own snapshot, "
"no decoder in the machine")
im = Image.new("RGB", (l.width + r.width + 8, l.height), (16, 16, 18))
im.paste(l, (0, 0)); im.paste(r, (l.width + 8, 0))
im.save(f"{tmpd}/{n:04d}.png")
# VP9 near-lossless: this is 256x192 palettised pixel art scaled by an
# integer, and a codec that smooths a colour boundary would be editorialising
# about the one thing the picture is evidence of.
cmd = ["ffmpeg", "-y", "-loglevel", "error", "-framerate", f"{fps:g}",
"-i", f"{tmpd}/%04d.png"]
if aud:
cmd += ["-i", aud, "-c:a", "libopus", "-b:a", "96k", "-shortest"]
cmd += ["-c:v", "libvpx-vp9", "-crf", "12", "-b:v", "0",
"-pix_fmt", "yuv444p", "-row-mt", "1", a.webm]
subprocess.run(cmd, check=True)
shutil.rmtree(tmpd)
if aud:
os.remove(aud)
print(f"{a.webm}: {len(frames)} frames @ {fps:g} fps, "
f"{os.path.getsize(a.webm)/1024:.0f} KB, every frame pixel-exact "
f"against {a.container}"
+ (" -- with the chip's own audio" if aud else " -- silent"))