drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Align the container to the disc, and find the decoder-free packed player fits

Browse Source 
Two sessions, unrecorded until now, committed together because their edits
share files and cannot be split cleanly after the fact.

Session 28 (FINDINGS 60): the container is DLX5 -- every record sector-aligned,
120/120 starting on a boundary where 3/120 did, +0.48% on the wire and zero
clocks -- and the ring's release rounds to RECALN so no pad is stranded.  Two
encoder levers measured and refused: `--spans all` buys +0.19 dB for +67% of
the wire, and joint span/lam selection emits byte-identical containers because
`lam` never leaves its floor on any of 120 frames.

Session 29 (FINDINGS 61): the packed full-frame blit is 27.3% of a 12 fps
frame, a channel fills GVRAM in buffer mode off the disc with the CPU halted,
and it walks the 1,024 B line stride itself through array chaining.  At the
9 clk/B dual-address floor the codec is 110.4% of a frame and a decoder-free
packed literal player is 55.2%, at +4.89 dB -- 2.75 dB past a ceiling the
codec's scene-wide palette cannot cross.  Encoder work is parked; the codec is
kept and not built on.

check.sh is ALL GREEN before and after, plus one new stage that gates the ORDER
of the measured paint costs rather than their values.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-25 06:54:27 -07:00
parent 8800d8f8c0
commit 1be428c270
28 changed files with 2203 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+52 -11
View File
@@ -189,19 +189,56 @@ payload to **328** — which is an encoder target, entirely inside this project,
and measured against the heaviest container the encoder emits rather than
against a shipping one (FINDINGS 59.7).
**A record is not a sector, and the cheapest fix is a re-encode.** 117 of 120
records start part way into a 512 B block, and reading whole blocks into the
ring corrupts the neighbouring records rather than merely wasting bytes — the
block loop reads with no bounds check. PIO absorbs this for free by simply not
storing the bytes outside the window, which is a property that disappears the
**A record was not a sector, and the fix was a re-encode — it is done.** 117 of
120 records used to start part way into a 512 B block, and reading whole blocks
into the ring corrupts the neighbouring records rather than merely wasting bytes
— the block loop reads with no bounds check. PIO absorbed this for free by
simply not storing the bytes outside the window, a property that disappears the
moment a DMA channel takes over. Priced three ways: windowed PIO is +1.34% on
the wire and cannot be done by a channel at all; a bounce buffer is +1.34% and
**+5 clocks on every delivered byte**, 22.4% of a frame; sector-aligning records
in the container is **+0.43% and zero clocks**. The last wins on both axes and
joins the re-encode bundle (FINDINGS 58.3). **Session 27 made it a
precondition rather than a preference**: the transport now *refuses* a windowed
read when the data phase is the channel's, so the container has to meet it
before the DMAC can sit behind the ring (FINDINGS 59.4).
in the container is **+0.43% and zero clocks** (FINDINGS 58.3). Session 27 made
it a precondition rather than a preference — the transport *refuses* a windowed
read when the data phase is the channel's (59.4) — and **session 28 met it: the
container is DLX5, every record is padded to 512 B and the frame stream starts
on a sector boundary. 120 of 120 records are aligned, the realised wire cost is
+0.48%, and the disc now moves exactly the records** — the bytes off the disc
and the bytes into the ring are the same number, which is what check.sh gates on
(FINDINGS 60.1).
**And the player that has no decoder at all fits the budget the codec misses.**
256-colour GVRAM throws away the high byte of every word a CPU writes, so a
picture byte normally costs two disc bytes — but CRTC R20 bit 11 turns the
masking off, and with the two 256-colour pages scrolled apart one word carries
two pixels (FINDINGS 46/47). Session 29 measured what that is worth. The packed
full-frame blit is **227,553 clocks, 27.3% of a 12 fps frame** — 51% of the
unpacked one, and the *same* as the unpacked path's write-only floor, so packing
buys back the whole of the source read. A DMA channel fills GVRAM in buffer mode
straight off the disc with the CPU halted, and **walks the 1,024-byte line stride
itself** through array chaining, so a frame is one channel start and not 192. At
the 9 clk/B dual-address floor — the only configuration this machine can be shown
to run — **the shipping codec is 110.4% of a frame and a decoder-free packed
player is 55.2%.** Decoding 37,585 bytes costs more than not decoding 49,152.
What it costs is the wire: **576 KB/s, fixed, with no lever** — a codec's bitrate
is adjustable and a literal frame's is geometry — against 327 KB/s for the codec
at the same floor. So the two open hardware facts changed character: **whether
the medium sustains 576 KB/s, and whether buffer mode blanks the layer while it
is being written, now decide which player exists** rather than how much headroom
one has. The codec cannot take the packing either way: writing 4×4 blocks a byte
at a time is **28% dearer** than the shipping shape, and pairing the blocks 128
columns apart to get the burst back drops SKIP from 66.3% of blocks to 46.1% of
pairs — about **+60% on the bytes**, against a target that needs them 35% lower
(FINDINGS 61).
**So encoder work is PARKED (USER DECISION, session 29).** Not because the codec
is wrong, but because its remaining path is a conjunction and the packed one is
not. The codec that exists is 440 KB/s and 110.4% of a frame; reaching E7's
327 KB/s needs a 35% byte reduction after two of its three levers were measured
and found inert (60.4, 60.5), and the reward on success is a design at ~100% of
the frame. The packed player is at 55.2% today. **The codec is kept on disk and
not built on**, because B2 is unanswered and 48.1's prior leans against packing —
if buffer mode blanks, it is the only thing left (48.3).
**The scene graph is in, and the worst gap between two decision points is
zero.** `tools/import/scenegraph.py` imports the arcade scene graph — 40 scenes,
@@ -434,7 +471,11 @@ tools/encoder/ hybrid VQ encoder and DLX3 container writer.
side. It must match blit.s and decode.s: 11 coarse units of
24 px, 11 fine of 2.
DLX2 4-byte-aligns every frame record, because an odd move.l
is an ADDRESS ERROR on a 68000, not a slow read.
is an ADDRESS ERROR on a 68000, not a slow read. DLX5 aligns
them to 512 B sectors instead, so a DMA channel can read a
record as whole sectors straight into the ring with no window
and no bounce copy; dlx.record_lengths() is the one place that
rule is applied.
dlx.py is the reference DECODER, ground truth for the 68000.
24 models the ring with the 68000 owning it: the request
queue, the poll-only-when-not-decoding rule and 54.4's frame
docs/FINDINGS.md
+507
View File
@@ -5803,3 +5803,510 @@ cross-check at the top of the tool is what licenses every number under it.
**Re-deriving 59.7 against a lighter container needs that harness re-run first**,
and until it is, "34% too big" is a statement about the heaviest container and
not about the project.
---
## 60. The re-encode bundle: the container agrees with the medium, and two encoder levers turn out not to be levers (session 28)
**Emulated** — MAME 0.277, `x68000 -exp1 cz6bs1 -ramsize 2M/6M`, plus px68k's
C68K core in `tools/bench/c68k`. **No real hardware ran.** `./tools/bench/
check.sh` was ALL GREEN before this session and is ALL GREEN after it, including
a full re-encode of the gate container.
ROADMAP's re-encode bundle was four items collected under P2 because they shared
one re-measurement. All four are now closed, and **two of them close as
negatives** — which is the more useful half of the session.
### 60.1 DLX5: the container is laid out in sectors, and the disc now moves exactly the records
`tools/encoder/encode.py` pads every record up to **512 bytes** instead of 4,
and pads the scene header so the frame stream itself starts on a sector
boundary. `tools/encoder/dlx.py` reads it as `DLX5`; `rec_align` is a property
of the container version and `record_lengths()` is the one place the rule is
applied, replacing four hand-copied `4 + n + (-(4+n) % 4)` expressions in the
analysis tools.
**Why 4 was not enough** is 58.3 option C and 59.4: a SCSI target answers in
512 B blocks, 117 of 120 DLX4 records started part way into one, and
`sc_in_data` **refuses** a windowed read when the data phase belongs to the
DMAC, because a channel writes a contiguous run and cannot drop the 300 bytes in
front of a record. Windowed PIO absorbed that for free and does not survive the
move to the channel.
| on the gate container | DLX4 | **DLX5** |
|---|---:|---:|
| records starting on a sector boundary | 3/120 | **120/120** |
| bytes off the disc, for 4,488,577 B of record | 4,548,608 (+1.34%) | **4,510,208 (+0.48%)** |
| bytes into the ring | 4,488,588 | **4,510,208** |
| clocks of window or bounce copy | 0 (PIO only) / +5 clk/B (DMA) | **0, and a channel can run it** |
| largest record | 40,984 B | 41,472 B — a 256 KB ring still holds it 6 times |
**The disc figure and the ring figure are now the same number, and check.sh
gates on that identity** rather than on two constants. Both are read out of the
container at check time: the old gate had 4,488,588 and 4,548,608 written into
it as literals, and the re-encode went red on both — correctly, because the
container had changed and the expectation had not. **A gate whose expected value
is a literal tests the literal.**
### 60.2 The consumer had to be told, and the failure was a byte-exact wrong answer
`stream.s` released the ring up to the last byte it *read*, rounded to 4. Under
DLX5 that strands up to 511 bytes of pad per record, and the ring's own audit
caught it on frame 0: `RD_PTR MISMATCH: decoder released 00040F08, record ends
00041000`. The release now rounds to `RECALN` (geom.i), which is the record
alignment the container guarantees, and is correct only because the ring base is
`RECALN`-aligned too — stated where the constant is defined rather than assumed.
**This is the shape of the whole item.** Sector alignment is not a tidier
version of 4-byte alignment; it is a contract with *two* sides, and the producer
side alone would have drifted the free-space arithmetic by up to 511 B a frame
with every frame still decoding pixel-exact.
### 60.3 Reserved black at index 0 — the letterbox is true black, and it costs 0.04 dB
23.4, open since session 5, was the other half of P2. `VQ.scene_palette` now
quantises the picture into **255 entries and reserves index 0 as (0,0,0)**;
`dlxload.pack_palette` gives it `I = 0` by its own minimum-squared-error rule
with no special case, so 23.3's "the bars sit at RGB (4,4,4)" goes away for
free. GVRAM cleared to zero displays entry 0, and a free mediancut palette put a
real image colour there — on 00020 f0001, `(206,192,176)`, used by 210 image
pixels.
Black is **reserved, not withheld**: the mapper may still spend index 0 on
genuinely black pixels. What the reservation buys is that index 0 is black
whatever the scene contains, which is what the letterbox needs and what a free
palette cannot promise. Measured cost on the Singe window: **28.92 dB → 28.88 dB
at `--spans need`**, and the palette ceiling is unmoved at 31.32 dB.
### 60.4 `--spans all` as the default: MEASURED, and the recommendation is REFUSED
E2 has been "a recommendation, not a measurement" since 43.6.1. It is now a
measurement, on the DLX5 container, 120 frames, the gate recipe
(`--kbps 280 --span-kbps 488`):
| | KB/s incl. audio | PSNR | frames that miss the 12 fps deadline |
|---|---:|---:|---:|
| `--spans need` (shipped default) | **267.9** | 28.88 | 2/120 |
| `--spans all` | 448.2 | 29.07 | 1/120 |
**+67% of the wire, for +0.19 dB and one frame of 120.** Session 27 established
that the frame affords 6.69 clocks a byte and that a dual-address byte costs 9,
so the container's problem is that it is too big; spending 180 KB/s to buy a
fifth of a decibel is the wrong direction, and `need` stays the default.
**The item is closed, and the answer is no.** The gate container keeps
`--spans all` — it is a fixture chosen to exercise the newest path in the
decoder, not a shipping recipe, and check.sh says so.
### 60.5 Joint span/lam selection: implemented, measured, and it is a NO-OP for a reason
E3 (39.3 item 5) asked for span selection to be re-derived jointly with `lam`
instead of greedily after it. The argument is sound on paper: the span pass
*removes* the block payload of every block it covers, so the frame lands under
its byte allowance and the blocks that were **not** spanned were priced at a
`lam` chosen as if those bytes were still needed.
`ratectl._refit_joint` (`--joint-spans`) hands the freed bytes back to the lam
search and re-spans, to a fixed point or two rounds. A round is kept only if the
frame still fits both ceilings it was already fitting, so `lam` can only fall
and the un-spanned blocks can only improve.
**It emits byte-identical containers.** All four cells of `{need, all} x {greedy,
joint}` produce two md5s, not four — and so does `--rc-floor open`:
| | lam, 120 frames | container |
|---|---|---|
| `need` greedy / joint | 10.0 min, median, p90, max | **identical** |
| `all` greedy / joint | 10.0 min, median, p90, max | **identical** |
| `all --rc-floor open` greedy / joint | 1.0 min, median, p90, max | **identical** |
**The reason is 44.3, and it is structural rather than incidental: `lam` never
leaves its floor on any frame, at either floor the encoder offers.** The block
coder at the profile floor already lands under the per-frame byte budget, so
there is nothing for a joint re-search to spend the freed bytes on. E3 is not a
lever, and it is not a lever for the same reason `--kbps` and the bucket are
not: **the byte side of this encoder is governed by the span pass and by `mu`.**
The code stays, defaulted off, because the measurement is the finding and a
future container that moved `lam` off its floor would make the question live
again. That is 44's pattern kept deliberately: ask whether the lever is loaded
before pulling it.
### 60.6 The apparatus lied, silently, and only a byte comparison caught it
**MAME 0.277 served a compressed CHD's own file bytes as sector data.** On the
DLX5 volume, every `READ(10)` reported success and returned the wrong bytes: the
destination buffer after a 4,096 B read at LBA 0 was **byte-for-byte the first
4,096 bytes of `dlxdisk.chd`, starting `MComprHD`** — the CHD file header — while
`chdman verify` reported both SHA1s correct.
Isolated by experiment, and the trigger is the image's **content**:
| volume | compression | result |
|---|---|---|
| DLX4 gate, 8,768 sectors | default (lzma/zlib/huff/flac) | **byte-exact** |
| DLX5 gate, 8,809 sectors | default | CHD header served as LBA 0 |
| DLX5 gate, padded to 9,072 sectors, 16x63 geometry | default | same failure |
| DLX5 gate, **truncated to 8,768 sectors** — the working length | default | same failure |
| DLX5 gate | `-c zlib` alone | same failure |
| DLX5 gate | **`-c none`** | **byte-exact** |
So it is not size, not geometry and not the codec; compression decides it and
uncompressed is sound. **The MAME-side cause is not diagnosed** and is not
claimed here. `tools/bench/mkvol.sh` now builds the volume `-c none`, at 4.5 MB
in `tmp/` against 1.6 MB, with the measurement written next to the flag.
**What is worth keeping is not the workaround.** The failure was *silent at the
transport layer* — the SPC handshake completed, the phase sequence was correct,
the byte count was right, and `scsi_read` returned 0. The only thing in the tree
that could see it was `tools/bench/scsi.lua` comparing every delivered byte
against the host's copy of the same image. A transport gate that checked status
and length would have been green all session, and FINDINGS 58's "byte-exact
against the host's copy" would have been the load-bearing phrase nobody noticed.
### 60.7 What the bundle did to the budget: almost nothing, which is the result
The re-measurement the four items shared, on the rebuilt gate container, with
`tmp/c68k_frames.csv` regenerated by the same `check.sh` run:
| | before (DLX4, session 27) | **after (DLX5, session 28)** |
|---|---:|---:|
| decoder, MEASURED on C68K | 68.5% of the frame | **68.6%** |
| delivered bytes a frame | 37,403 | **37,585** (the pad is delivered, so it is charged) |
| headroom after decode + best-case audio | 6.74 clk/B | **6.69 clk/B** |
| single address held, `W`=5, total | 92.2% | **92.4%** |
| dual address held, `W`=9, total | 110.2% | **110.4%** |
| E7's byte target at the 9 clk/B floor | 328 KB/s, 34% too big | **327 KB/s, 35% too big** |
| B1's zero-prefill delivery requirement | 451.4 KB/s | **453.6 KB/s** |
| scene header (P1/53.5, 22_scene_load) | 6,164 B | **6,656 B** |
**Every conclusion in 59.7 survives unchanged**, which is what a precondition is
supposed to do: it removes an obstacle without moving the arithmetic. The two
numbers that did move are both the alignment pad being paid honestly —
`15_bus_occupancy.py` now charges the **padded** record rather than the payload,
because the disc carries the pad whether or not a frame contains it.
### 60.8 Two caches that would have tested the wrong artefact
`H.build` acquired an option this session (`reserve_black`), and two tools cache
its output in a pickle **keyed on the frames directory alone** —
`16_span_roundtrip.py`, which is the span container's round-trip gate, and
`13_cpu_ratectl.py`. A stale pickle would have let the gate round-trip a
container built from the old palette while the shipping encoder emitted the new
one: green, and testing an artefact that no longer exists. Both now store the
build parameters with the model and rebuild on a mismatch.
### 60.9 What is now true, and what P4a still owes
**M2's re-encode bundle is done and the container is what the DMA path needs.**
`dma_run.sh` still gates on the *refusal* — `WINDOWED DMA READ REFUSED, as it
must be` — which is now a negative control rather than a description of the
container, because the container no longer asks for a windowed read.
What is left of P4a is the wiring: putting the channel behind `ring.i`'s `XF_*`
mailbox in place of the PIO loop in `xfer.i`, which 59.4 blocked on exactly this
precondition. `xfer.i`'s sector arithmetic already degenerates correctly —
`SC_WSKIP` is 0 and `SC_WKEEP` is the whole record on every record — so what
changes is which loop moves the bytes, not what is asked for.
## 61. The decoder-free packed player fits the clocks the codec misses, and the codec cannot have the packing (session 29)
**Emulated.** MAME 0.277, `x68000 -bios ipl10`, and for the transport runs
`-exp1 cz6bs1` with a zero-filled `scsiexrom.bin`. No real hardware ran, and
47.4 — does buffer mode blank the layer — still wants a board.
FINDINGS 44.7 asked what a player with no codec at all would cost and answered
"it fits the clocks and dies on the medium". 46.5/47.1 found the off switch for
the 2:1 GVRAM tax, 47.2 built the packed layout and rendered it pixel-exactly on
both emulators, and **47.5 re-derived the budget on a cost model that has since
been replaced in full** — the transport was an unmeasured `c` when it was
written, and sessions 25b–28 measured it (58.2), bounded it (59.2) and priced
what a frame can afford against it (59.7/60.7). 47.6.1 filed the packed paint's
`movem` shape as an *assumption*; 47.6.2 said the DMAC "has not been near this";
47.6.4 left the codec's own survival under the layout untouched.
All four are answered here. `tools/analysis/29_packed_player.py` is the
arithmetic, `tools/bench/blit.s` V8/V9/V10 are the paint measurements, and
`src/player/dmagate.s` runs 4–6 are the transport ones.
### 61.1 The packed paint, MEASURED: it costs what the unpacked path pays to WRITE
`blit.s` V8 is V1 with one thing different — a row is 128 words instead of 256,
because R20 bit 11 lets a word carry two picture bytes — and `blit.lua` times it
in the same run as V1, V2 and V3, so it is quoted against numbers that have not
moved since session 9.
| variant | clk/frame | % of a 12 fps frame | what it moves |
|---|---:|---:|---|
| V1 unpacked `movem` blit | 446,286 | 53.6% | 96 KB read + 96 KB write |
| V2 byte-source expansion | 1,284,174 | 154.1% | 48 KB read + 96 KB write |
| V3 unpacked WRITE-ONLY floor | 225,789 | 27.1% | no source read at all |
| **V8 PACKED `movem` blit** | **227,553** | **27.3%** | **48 KB read + 48 KB write** |
All four put the same 49,152 pixels on screen. **V8 is 51.0% of V1 and 100.8% of
V3: the packed blit costs what the unpacked one pays to write alone, with its
source read thrown in free.** It is not exactly half of V1 because the 192-row
loop does not halve with the words — per word V1 is 9.080 clocks and V8 is
9.259, and the difference is the row loop amortised over half as many bursts.
V2 is worth keeping in view: it is the "send 1 byte a pixel and let the CPU
expand it" trade, and at 154.1% it is not a trade, which is why the unpacked
path has no cheap wire.
V8 is a **timing** variant and does not set bit 11. MAME's `gvram_w` carries no
timing in either arm, so the bit cannot move a cycle; what it moves is the
picture, and the picture is what `show_frame256_packed.lua` and `gvpack` already
verify pixel-exactly (47.2). The source is pre-interleaved host-side, which is
the honest half of the claim — the packing is an encoder-side transform, the
same argument 46.3 made for the text plane.
### 61.2 The channel writes GVRAM in buffer mode, and it walks the line stride itself
47.6.2 was right that nothing in this tree had ever pointed a channel at
`$C00000`. `dmagate.s` now does, three times, and `dma_run.sh` gates all three:
| run | R20 | result |
|---|---|---|
| **4. DMA → GVRAM, bus held, bit 11 SET** | `$0916` | **2,048 B byte-exact against the disc**; 1,024 landed in page 1 (high half), 1,024 in page 0 |
| **5. the SAME, bit 11 CLEAR** | `$0116` | **457 bytes lost, every one of them at an EVEN offset; not one ODD byte harmed** |
| **6. ARRAY CHAINED, 8 rows at the 1024 B stride** | `$0916` | **2,048 B byte-exact at eight separate row bases from ONE start**; `MAR` ended at `+7,424` |
In all three the discriminator reads the same as 59.1's: **`MTC` sampled by the
instruction after `START` is 0 of 2,048 and the CPU went round its wait loop
once** — the whole transfer happened between two instructions.
**Run 5 exists because the first cut of run 4 was a test that could not fail.**
It OR-ed bit 11 on top of whatever the IPL left, and the IPL leaves `$0B16`
(22.1) — bit 11 **already set**, and `COL = %11`, the 65,536-colour setup, which
writes whole words with or without it. It passed and proved nothing. Both runs
now write R20 outright and differ in exactly that bit.
**And the control's signature is a PLACE, not a count.** The obvious assertion —
"half the bytes must differ" — is wrong, and asserting it turned the gate red on
a correct run. In masked 256-colour mode `gvram_w` takes `data & 0x00ff` and
ignores `mem_mask`, so a byte written to an even address is never stored and the
high half keeps what it held; where the record is pad, the stale half matches the
disc by coincidence, and 567 of 1,024 did. What the mechanism says is that
**every odd byte survives and only even ones can be lost**, and that is what the
gate asserts.
Run 6 is the one that changes an architecture. A picture row is 256 B of a
1,024 B line stride, so a frame is **192 destinations, not one**, and a channel
writes a contiguous run — the same limitation that makes it refuse a windowed
read (58.3). 46.6 said "no stride for a DMAC to skip" about the bytes *within* a
row and never examined the rows. The MC68450 answers with sequential array
chaining, `src/player/dma.i` now programs it behind a `DM_BARV` mailbox that is
**zero unless a caller asks**, and the channel walked an 8-entry array on its
own. **The CPU does not restart the channel per row.**
### 61.3 The codec cannot have the packing — 47.6.4, closed
A 4×4 block owns four bytes at **stride 2** under the packed layout, because the
high bytes of its four words belong to the block 128 columns away. There are
exactly two ways a block decoder could live with that, and `blit.s` V9 and V10
are them:
| variant | clk/frame | % of frame | vs V4 |
|---|---:|---:|---:|
| V4 block order, UNPACKED (the shipping shape) | 637,971 | 76.6% | — |
| **V9 block order, PACKED, 16 `move.b` at stride 2** | **815,839** | **97.9%** | **+28%** |
| **V10 block order, PACKED, blocks PAIRED** | **320,456** | **38.5%** | **−50%** |
**V9 is 28% DEARER than V4, and buys nothing on the wire** — a codeword is
already one byte a pixel, so the packed layout has no bytes left to save a block
decoder. That route costs 177,868 clocks a frame for nothing.
**V10 halves the paint and pays for it in the mode map.** Pairing the block at
x with the block at x+128 gives V4's `movem` shape back, and a pair skips only
if *both* its blocks skip. On the gate container:
| | now | paired |
|---|---:|---:|
| SKIP | **66.3%** of blocks | **46.1%** of pairs |
| painted | **33.7%** | **53.9%** |
So pairing paints **1.60×** as many blocks for **0.50×** the paint per block:
**−20% on the clock, and about +60% on the BYTES**, because a coded block is
bytes in the container whether its half of the pair changed or not. E7 needs the
bytes **down 35%**.
**The packed layout is not an upgrade the codec can take. It is the thing you
get instead of the codec.**
### 61.4 The budget, re-derived on the measured model
Every cell below is CPU work plus transport plus best-case audio (10,417 clk,
1.25%, from the IPL ROM's own channel-3 setup — 21_iplrom_dmac.py). None of them
overlap: `buscost.DMA_OVERLAPS = False`.
| architecture | B/frame | W=5 | **W=9** | W=12 | W=16 | W=19 |
|---|---:|---:|---:|---:|---:|---:|
| CODEC, CPU-decoded (the shipping design) | 37,585 | 92.4% | **110.4%** | 124.0% | 142.0% | 155.5% |
| free / DMAC device→GVRAM / unpacked | 98,304 | 61.1% | 108.2% | 143.6% | 190.8% | 226.2% |
| **free / DMAC device→GVRAM / PACKED** | **49,152** | 31.6% | **55.2%** | 72.9% | 96.5% | 114.1% |
| free / CPU-painted / unpacked, 2 B/px wire | 98,304 | 113.8% | 161.0% | 196.4% | 243.5% | 278.9% |
| free / CPU-painted / unpacked, 1 B/px wire | 49,152 | 184.8% | 208.4% | 226.1% | 249.7% | 267.4% |
| **free / CPU-painted / PACKED** | **49,152** | 58.0% | **81.6%** | 99.3% | 122.9% | 140.6% |
**W=9 is the column that matters**, because 59.2 found the only configurations
this machine can be shown to run are dual-address, and a dual-address byte is a
4-clock read of the device plus a 5-clock write to memory. Everything left of it
is a hardware fact nobody here has (ROADMAP B3).
**At the floor the CODEC misses by 10.4% and the DECODER-FREE PACKED PLAYER FITS
WITH 45% TO SPARE.** Decoding 37,585 bytes costs 109% of a frame; not decoding
49,152 costs 54%.
That is not a small correction to 47.5. **44.7 said it in advance, on a cost
model that has since been thrown away: "the codec is not there to save CPU — it
is there to save the wire."** The measured model agrees and goes further: the
CPU side is not merely affordable without the codec, it is *strictly cheaper*.
### 61.5 So it is entirely a medium question, and the medium is the unmeasured one
| architecture | B/frame | KB/s | GB for 22.8 min |
|---|---:|---:|---:|
| CODEC, gate container | 37,585 | 440.4 | 0.62 |
| CODEC, `need` default | — | 267.9 | ~0.38 |
| CODEC, E7's target at the 9 clk/B floor | 27,924 | 327 | 0.46 |
| **decoder-free PACKED, either architecture** | **49,152** | **576.0** | **0.81** |
| decoder-free unpacked | 98,304 | 1,152.0 | 1.61 |
**576 KB/s, sustained, with no lever to pull.** A codec's bitrate is a lever; a
literal frame's is geometry, and no scene in the picture costs less than another.
ROADMAP B1 is unmeasured, and the 0.7–1.7 MB/s usually quoted for BlueSCSI on an
X68000 is **folklore with no published benchmark behind it**. 576 KB/s sits
inside that range, which is exactly the reason the range has to be measured
rather than cited: **a codec at 327 KB/s survives a slower answer and a literal
frame does not degrade, it drops.**
**B1 has therefore changed character.** It used to set how much headroom the
player has. It now decides which player exists.
### 61.6 And under MAME's blanking reading, the cheap architecture is the dark one
47.4's question is untouched by any of this and it still needs a board. What the
measured paint does is turn 48.3's range into numbers, and it exposes an
asymmetry nobody had stated: **R20 bit 11 only has to be set across the GVRAM
writes, and where those writes come from decides how long that is.**
| architecture | black interval | bit 11 set for |
|---|---:|---|
| free / DMAC device→GVRAM / unpacked | 60%..225% of a frame | the whole DMA |
| free / DMAC device→GVRAM / PACKED | 30%..113% of a frame | the whole DMA |
| free / CPU-painted / unpacked | 53.6% | the blit only |
| **free / CPU-painted / PACKED** | **27.3%** | **the blit only** |
The CPU-painted packed path has the smallest black window of any of them,
because its transport lands in RAM where bit 11 is irrelevant and only the
227,553-clock blit needs the bit. **The DMAC-direct path, which is cheaper in
clocks at every rung of the ladder, is the one that must hold the bit across its
whole transfer.** If MAME is right, the cheap architecture is the dark one and
the expensive one is merely dim.
Both are a strobe at the frame rate over the whole picture, and the packed layout
has no page to flip to: both 256-colour pages carry picture, which is the entire
point of it (48.3). If px68k is right, every number in 61.4 and 61.5 stands as
written. 48.1's prior — an assertion against a silence — has not moved.
### 61.7 What this does NOT establish
1. **No clock here is a transport clock.** MAME's DMAC runs on wall-clock
attotimes and models a held bus by halting the CPU (42.5), so runs 4–6 settle
which configurations *work* and not what one costs. Every `W` in 61.4 is
datasheet arithmetic except PIO, and the array-chain entry at 36 clocks is
Fig 4-25 sheet 1, not a measurement.
2. **One frame's worth of transport, not a stream.** Run 6 chained eight rows,
not 192, and nothing here ran a chained transfer back to back at 12 fps or
through `ring.i`. A decoder-free player has no ring at all in the DMAC-direct
form, which is a simplification this tree has not tested.
3. **The paint is a lower bound.** Every `blit.s` number is instruction cycles
against zero-wait-state memory; real GVRAM stalls the CPU and neither
emulator models it (`blit.lua`'s header, and 45's note on C68K).
4. **61.3's pairing cost is one container's.** 66.3% / 46.1% is the gate
container, deliberately the heaviest thing the encoder emits. A lighter
container has a different SKIP map and would pair differently — though not,
at that margin, differently enough to change the sign.
5. **No audio, no branching, no seek.** 61.4 charges the audio DMA and nothing
else. A decoder-free player still has to make 56's branch decisions and 51's
seeks, and at 576 KB/s it has less slack to make them in, not more.
### 61.8 Encoder work is parked (session 29, USER DECISION)
61.4 first closed with "nothing here is a reason to stop work on the codec — it
is the only branch that survives a slow answer to B1." **That does not survive
its own arithmetic and is withdrawn.**
It rested on 576 KB/s against **E7's 327 KB/s target, which does not exist.** The
codec that exists is **440 KB/s and 110.4% of a frame**, so the gap is **1.31x,
not 1.76x** — and reaching 327 needs a 35% byte reduction after 60.4 and 60.5
measured two of the encoder's three byte levers and found neither is a lever. The
reward on success is a design at ~100% of the frame with no margin, which is where
55.2% already is.
The two branches are not symmetric, and counting what each NEEDS is the argument:
| | facts it needs |
|---|---|
| **decoder-free packed** | buffer mode does not blank (B2); medium clears 576 KB/s (B1) |
| **codec** | E7 succeeds (unproven, two levers dead); **and** medium clears 327; **and** it ships at ~100% of a frame |
**So E7 and E4 are parked, and C1 with them.** E4 is included deliberately: it is
`H.build`'s k-means, it builds **VQ codebooks**, and a literal player has no VQ.
**What is NOT parked is the codec itself.** It stays on disk, gated by check.sh,
and nothing is built on it. 48.1's prior leans against packing — MAME asserts the
blanking semantic twice and deliberately, px68k's display path never reads the
bit at all, and an assertion against a silence is not a tie — and 48.3 stands:
if buffer mode blanks there is no version of the packed player that is merely
expensive. In that branch the codec is the only path left. **Keeping a working
decoder is inventory; building on it is work, and the work waits on B2.**
### 61.9 And the picture is BETTER — the codec is capped below the thing replacing it
Asked while scoping what a packed player would take, and it is the finding that
makes the branch worth building rather than merely affordable. The numbers are
`18_text_plane_16col.py`'s, over the same 120-frame window everything else is
measured on; 46.3 computed them to give the *text plane* something to be scored
against and never turned them on the 256-colour path itself.
| | PSNR vs the 24-bit source |
|---|---:|
| shipping container (the codec, as it ships) | **29.19 dB** |
| 256 colours, SCENE palette — **the codec's CEILING** | **31.33 dB** |
| 256 colours, **PER-FRAME** palette | **34.08 dB** |
**The middle row is a ceiling, not a rival.** Every codeword the codec emits is
an index *into* `vq.scene_palette`, so no bitrate takes it past 31.33 dB; it
spends 440 KB/s getting within 2.14 dB of it.
**A literal frame has no codebooks, so nothing forces the scene palette on it,
and per-frame palettes become legal.** 46.3 stated the constraint in as many
words while arguing the *other* side — "the 256-colour path cannot do this: its
palette is shared scene-wide because the codec's codebooks are indices INTO it."
Remove the codec and the constraint goes with it.
**So the decoder-free packed player is +4.89 dB on the shipping container and
+2.75 dB past a ceiling the codec cannot cross** — while costing 55.2% of a frame
against 110.4%. It is not a quality compromise bought with clocks. It is better
on both, and the whole of its cost is on the wire.
What the per-frame palette costs:
| | |
|---|---|
| wire | 512 B/frame → 49,664 B, **582.0 KB/s (+1.0%)** |
| clocks | ~2,370, **0.28% of a frame** if the CPU writes it — DERIVED from V8's measured 9.259 clk/word in the same `movem` shape |
| colours | **254, not 256** — the packed layout spends index 0 on the transparency key and puts black at 255 (47.2, `prep_frame.py --pack-transparent`), where `--reserve-black` spends one. A reserved entry measured 0.04 dB in 60.3, so this is noise against +4.89 |
**Two things this does NOT settle.**
1. **Whether a channel can write `$E82000`.** If the palette registers take a
byte-wide DMA the way GVRAM does in buffer mode, the palette is a 193rd array
entry and costs the CPU nothing at all — one channel start still paints a
whole frame. 61.2 only ever pointed a channel at GVRAM. **This is the next
probe, and it is the same shape as the ones that worked.**
2. **The quantiser is PIL's MEDIANCUT, not this project's.** `vq.scene_palette`
and `H.build` are what would actually ship the palette. The direction is
measured and the magnitude is about right; re-derive the per-frame figure
against the real builder before quoting it as the player's number.
docs/ROADMAP.md
+208 -52
View File
@@ -14,6 +14,24 @@ made sector-aligned records a precondition the transport enforces rather than a
preference (FINDINGS 59). **And 59.7 re-ranks what is left: the frame affords
6.74 clocks a byte, a dual-address byte costs 9, so B3 stopped being a constant
to look up and became the question of whether the design fits at all.**
Amended end of session 28: **THE RE-ENCODE BUNDLE IS DONE — all four items, one
re-measurement (FINDINGS 60).** The container is DLX5, sector-aligned, and the
disc now moves exactly the records. Two of the four closed as NEGATIVES: E2's
`--spans all` default is refused on measurement, and E3's joint span/lam
selection emits byte-identical containers because `lam` never leaves its floor.
**What is left of M2 is P4a's wiring** — the DMA channel behind `ring.i`'s
mailbox — and the budget did not move: headroom 6.74 -> 6.69 clk/B, so every
conclusion in 59.7 stands.
Amended end of session 29: **THE DECODER-FREE PACKED PLAYER IS BACK, MEASURED,
AND IT FITS THE CLOCK BUDGET THE CODEC MISSES (FINDINGS 61).** The packed
full-frame blit is **27.3%** of a 12 fps frame — measured, not assumed — a
channel fills GVRAM in buffer mode off the disc with the CPU halted, and it
walks the 1,024 B line stride itself through array chaining. At the 9 clk/B
floor the codec is 110.4% and a decoder-free packed player is **55.2%**. It asks
**576 KB/s, fixed**, against E7's 327 KB/s target. **So B1 stopped setting how
much headroom the player has and started deciding WHICH PLAYER EXISTS**, and B2
stopped being a nice-to-have. The codec cannot be packed: 47.6.4 is closed and
the answer is no, both ways (61.3).
**THE COMPLETION TARGET IS M3, THE VERTICAL SLICE** (USER DECISION): one scene
tree — a decision point, two outcomes, a death clip — with audio, streaming from
@@ -41,7 +59,7 @@ these units:
| **Delivery rate** | **no working figure, deliberately** (FINDINGS 50, USER DECISION). Every tool REQUIRES an explicit rate. |
| **Seek time** | **no figure at all, and never had one.** 51.3/51.4 made it matter. |
| **W, clocks stolen per delivered byte** | 5 single-address held, 9 dual held, 12 single arbitrated; the IPL ROM's own disk channel is **16..19** (52.5). **The largest open number in the project.** Session 27 added the row underneath it: with **no external request line** on the card (59.2) the channel is auto-requested and is charged **by time rather than by byte**, so at 460 KB/s a 50% bus share costs **10.61 clk/B** and a smaller share cannot carry the rate at all (59.3). |
| **The frame's headroom for a transport** | **6.74 clk/B** — 30.2% of a 12 fps frame, after the MEASURED decode (68.5%) and best-case audio (1.25%), at the gate container's 37,403 B record. **New in session 27, 59.7, and it is the number every row above is now read against.** A dual-address byte's floor is 9. |
| **The frame's headroom for a transport** | **6.69 clk/B** — 30.2% of a 12 fps frame, after the MEASURED decode (68.6%) and best-case audio (1.25%), at the DLX5 gate container's **37,585 B delivered record** (session 28: the sector pad is delivered, so `15_bus_occupancy.py` charges it). **It is the number every row above is read against**, and a dual-address byte's floor is 9. It was 6.74 against the DLX4 container; the bundle moved it by 0.05 and moved no conclusion. |
---
@@ -67,10 +85,18 @@ None of these block M2 or M3 software work, because session 18 forced every rate
to be an explicit argument. They set constants, and two of them decide how much
headroom the finished player has.
**B1. Measure the BlueSCSI — throughput AND seek time.**
**B1. Measure the BlueSCSI — throughput AND seek time.** **Session 29 gave this
a second acceptance test that is not a codec figure at all: 576 KB/s SUSTAINED,
which is what a decoder-free packed literal frame costs and cannot be talked down
from (FINDINGS 61.5).** A codec's bitrate is a lever; a literal frame's is
geometry. So the measurement now has three thresholds to be read against —
453.6 KB/s (the gate container needs no prefill), 327 KB/s (E7's target at the
dual-address floor) and 576 KB/s (no decoder at all) — and which of them the
medium clears decides which player gets built.
Throughput has an acceptance test already derived from real record sizes:
**513.2 KB/s** for the session-14 candidate, **451.4 KB/s** for the gate
container (`19_ring_stream.py`, FINDINGS 49.5). Seek time has nothing.
**513.2 KB/s** for the session-14 candidate, **453.6 KB/s** for the DLX5 gate
container (`19_ring_stream.py`, FINDINGS 49.5; the figure was 451.4 before
session 28's re-encode and the sector pad raised it). Seek time has nothing.
51.3/51.4 is why the second half matters: slack is *accumulated* out of
`pipe - wire`, so what a branch point costs is set by the rate and the time since
the last branch, not by the ring size. At 460 KB/s every ring from 192 KB to
@@ -83,6 +109,16 @@ and settles it in minutes on a real board. FINDINGS 48 shifted the prior toward
MAME and toward "unusable" — **do not pre-build on 1.0 B/pixel**. Same sitting:
the priority register `0xE82500` at `0x0000` (47.3).
> **Session 29 raised what this is worth, and gave it a number to be worth
> (FINDINGS 61).** It used to gate a derived halving. It now gates a player that
> has been measured to fit a budget the shipping design misses — 55.2% of a frame
> against 110.4% at the 9 clk/B floor. And 61.6 found an asymmetry worth carrying
> to the board: the black interval is the PAINT, not the frame, so a CPU-painted
> packed player is dark for **27.3%** of a frame while the cheaper DMAC-direct one
> is dark for **30..113%**. Under MAME's reading the cheap architecture is the
> dark one. **B2 and B1 are now the same decision from two sides, and B2 is the
> five-minute half.**
**B3. Single-address vs dual-address DMA — and now, first, DOES THE CARD DRIVE
`#EXREQ`?** 242 KB/s and 0.69 dB was the old framing. **Session 27 promoted this
to the item that decides whether the design fits at all (59.7).** The frame
@@ -105,9 +141,38 @@ is still the cheapest paper route to the same answer, because whatever Sharp's
own driver programs into DCR's DTYP is a statement about what the card supports.
FINDINGS 48.4. Not on this machine (checked, session 18).
**Ranking, stated because B1 used to outrank this:** B1 (throughput and seek)
sets how much headroom the finished player has. **B3 decides whether there is
any.** If the card drives `#EXREQ`, the ladder applies and the design fits with
**Ranking, amended session 29.** It was: B1 sets how much headroom the player
has, B3 decides whether there is any. FINDINGS 61 adds a third reading — **B1
and B2 together decide which player exists.** If the medium clears 576 KB/s
sustained and buffer mode does not blank, the decoder-free packed literal fits at
the dual-address floor B3 cannot get under, and B3 stops mattering for video at
all. If it does not, B3 is still the question. The three hardware facts are no longer
independent, and B2 is by far the cheapest of them.
> **ENCODER WORK IS PARKED — USER DECISION, session 29.** The first draft of this
> amendment said "nothing here is a reason to stop work on the codec". That does
> not survive its own arithmetic. It rested on comparing the packed player's
> 576 KB/s against **E7's 327 KB/s target, which does not exist**: the codec that
> exists is 440 KB/s and 110.4% of a frame, so the real gap is 1.31x, not 1.76x.
> And the branches are not symmetric. **Packed needs two facts** — buffer mode
> does not blank, medium clears 576 KB/s. **The codec needs E7 to succeed** —
> unproven, and 60.4/60.5 measured two of its three levers inert — **AND** the
> medium to clear 327, **AND** it ships at ~100% of the frame with no margin,
> which is where 55.2% is now.
>
> **E7 and E4 are both parked**, E4 included: `H.build`'s k-means builds VQ
> codebooks and a literal player has no VQ. C1 is gated by E4 and follows.
>
> **The codec is KEPT AND NOT BUILT ON.** That is inventory, not work. B2 is
> unanswered and 48.1's prior leans against packing — an assertion against a
> silence — and if buffer mode blanks there is no version of the packed player
> that is merely expensive (48.3), at which point the codec is the only path
> left. Keeping a working decoder on disk costs nothing; building on it costs
> sessions.
**The older ranking, which still holds inside the codec branch:** B1 (throughput
and seek) sets how much headroom the finished player has. **B3 decides whether
there is any.** If the card drives `#EXREQ`, the ladder applies and the design fits with
room. If it does not, the fallback is limited-rate auto-request at a share the
player picks (P4c), and the container has to come down to **328 KB/s of payload**
to fit at the 9 clk/B floor — 34% below where the gate container sits (59.7).
@@ -124,6 +189,43 @@ to fit at the 9 clk/B floor — 34% below where the gate container sits (59.7).
---
## The packed branch — what building it means (session 29, USER DECISION)
**This is where the work goes now.** FINDINGS 61: a decoder-free packed literal
player is **55.2% of a frame at the 9 clk/B dual-address floor** against the
codec's 110.4%, and **+4.89 dB** on the shipping container because a literal
frame is not tied to a scene palette the codec's codewords index into. It costs
**582 KB/s, fixed, with no lever.** Three items, in order.
- **K1. Can a DMA channel write the palette registers at `$E82000`?** (61.9.)
The cheapest item in the project and the one with the largest structural
payoff: if the palette takes a byte-wide DMA the way GVRAM does in buffer mode,
it becomes a **193rd array-chain entry** and one channel start paints a whole
frame — palette included, CPU halted throughout, **no per-frame CPU work in the
video path at all**. Same shape as 61.2's three runs: a run in `dmagate.s`, a
host comparison in `dma.lua`, an assertion in `dma_run.sh`, **and a negative
control.**
- **K2. A packed container.** Quantise to **254** colours (index 0 is the
transparency key, black at 255 — 47.2), interleave column *i* with *i+128*,
emit a per-frame palette, sector-align the record. Simpler than a coded one:
no VQ, no rate control, no mode map, no `lam`. `encode.py` already aligns
(DLX5) and `prep_frame.py --pack-transparent` already emits the layout.
**Re-derive 61.9's 34.08 dB against `vq.scene_palette` here** — the measured
figure is PIL's MEDIANCUT and is a direction, not the player's number.
- **K3. End to end, off the disc.** Palette, page-1 X-scroll 384, priority
`vc1 = 0x0002`, R20 bit 11; one chained DMA a frame. Gated pixel-exact over the
same 120-frame window the decoder is gated on.
**What K3 deletes, and why that is a risk and not a win to be banked:** a
DMAC-direct packed player has **no ring** — `ring.i`, `xfer.i` and most of
`stream.s` leave the video path, and **P4a's wiring is parked with them.** A
simplification that large usually hides something, and 61.7.2 names the specific
untested thing: a chained transfer has never run back to back at 12 fps.
**K1 and K2 survive a bad answer to B2. K3 does not.** Do K1 first.
---
## M2 — a player, as opposed to a decoder
`decode.s` draws pixel-exact frames from RAM Lua pre-loaded; `stream.s` decodes
@@ -145,32 +247,57 @@ Shipping the codebooks pre-expanded was considered and refused: it trades
9.26 ms of CPU for 5,120 more header bytes, which is a wash in milliseconds and
not a wash in kind (53.6).
**P2. Palette packing on the 68000. HALF DONE, session 21 — FINDINGS 53.**
~~**P2. Palette packing on the 68000.**~~ **DONE, session 28 — FINDINGS 60.**
The half that was open was the encoder's, and it closed with the whole bundle
below. Session 21 — FINDINGS 53 — did the packing itself:
~~The encoder still emits RGB888; the X68000 word packing is Lua-side.~~ The
packing is on the 68000: `pal_pack` writes 256 words straight into `$E82000`
with `I` chosen per entry by minimum squared error (**1.96 dB**, 23.3), gated on
the words read back **out of the palette registers**. 9.70 ms per scene, plus
5.29 ms of scene-independent table build hoisted to boot (53.3).
**What is left is the other half of the sentence: reserve index 0 as black with
`I = 0` (23.4).** That is ENCODER-side, it changes the container, and it moves
every constant fitted to the gate container, so it is a re-encode plus a
re-measurement rather than an edit.
~~**What is left is the other half of the sentence: reserve index 0 as black with
`I = 0` (23.4).**~~ **DONE, session 28 — 60.3.** `VQ.scene_palette` quantises
the picture into 255 entries and holds index 0 at (0,0,0); `pack_palette` gives
it `I = 0` by its own minimum-squared-error rule, so 23.3's "the bars sit at
RGB (4,4,4)" needed no special case. **0.04 dB** on the Singe window, palette
ceiling unmoved. Black is reserved, not withheld — the mapper may still spend
index 0 on genuinely black pixels; what it buys is that index 0 is black
whatever the scene contains.
**THE RE-ENCODE BUNDLE, collected here because it is now four items and they
share one re-measurement.** Nothing below is worth a container revision on its
own; together they are one:
1. reserve palette index 0 as black, `I = 0` (23.4, this item);
2. `--spans all` as the default (E2, and it is the loaded lever on the byte
side);
3. re-derive span selection jointly with `lam` (E3);
4. **sector-align every record (58.3; PROMOTED TO A PRECONDITION in session
27, 59.4 — `sc_in_data` refuses a windowed read under the DMAC, so this is
what the channel is waiting for)** — +0.43% on the
wire, zero clocks, and it is what lets P4a's DMA channel write straight
into the ring with no window and no bounce copy. Until then the letterbox gets the palette's
closest thing to black (index 255 on the gate container); `load.i` reports
whichever index that is and needs no change when it becomes 0.
~~**THE RE-ENCODE BUNDLE, collected here because it is now four items and they
share one re-measurement.**~~ **DONE, session 28 — FINDINGS 60. Two of the four
closed as NEGATIVES, which is the more useful half.** The four were:
1. ~~reserve palette index 0 as black, `I = 0`~~ **DONE** (23.4, 60.3);
2. ~~`--spans all` as the default~~ **MEASURED AND REFUSED** (E2, 60.4).
267.9 KB/s / 28.88 dB / 2 frames late at `need`, against 448.2 KB/s /
29.07 dB / 1 late at `all`: **+67% of the wire for +0.19 dB and one frame
of 120**, against a container the budget already says is 35% too big.
`need` stays the default. **It was called "the loaded lever on the byte
side" and it is — it is just loaded the wrong way**, and E7 is why. The
GATE container keeps `all`: it is a fixture, not a recipe;
3. ~~re-derive span selection jointly with `lam`~~ **IMPLEMENTED, MEASURED,
NO-OP** (E3, 60.5). `--joint-spans` hands the span pass's freed bytes back
to the lam search and re-spans; all four cells of `{need,all} x
{greedy,joint}` are BYTE-IDENTICAL, and so is `--rc-floor open`. **`lam`
never leaves its floor on any of 120 frames at either floor the encoder
offers** (44.3), so there is nothing to spend the freed bytes on. The code
stays, defaulted off, because a container that moved `lam` off its floor
would make the question live again;
4. ~~**sector-align every record**~~ **DONE — the container is DLX5** (58.3
option C, promoted to a precondition by 59.4; 60.1). Realised cost
**+0.48%** on the wire against the +0.43% predicted, zero clocks, and
**120/120 records start on a sector boundary** where 3/120 did. The disc
and the ring now move the SAME 4,510,208 B and check.sh gates on that
identity, both figures read out of the container instead of written into
the script — the old literals went red on the re-encode, correctly.
**The consumer had to be told too** (60.2): `stream.s` released the ring
to the last byte it READ, which strands up to 511 B of pad a record, and
the ring's own audit caught it on frame 0 while every frame still decoded
pixel-exact. The release rounds to `RECALN` now.
The letterbox no longer gets the palette's closest thing to black — item 1 put
true black at index 0, and `load.i` needed no change, as it said it would not.
~~**P3. A real frame clock.**~~ **DONE, session 22 — FINDINGS 54.**
`src/player/clock.i` derives the tick from the CRTC's own V-DISP through the
@@ -244,11 +371,15 @@ the ladder exactly one, dual address held, has a code path here, and it is the
one demonstrated. The slot pinout has `#EXREQ` at B36, so a real card plausibly
drives it — **that is now B3's sharpest form**.
**What is left of P4a is downstream of the container, not of the DMAC** (59.4):
`sc_in_data` **refuses** a windowed read when the data phase is the channel's,
because a channel writes a contiguous run and cannot drop the 300 B in front of
a record. So putting the channel behind `ring.i`'s mailbox waits on the
re-encode bundle.
~~**What is left of P4a is downstream of the container, not of the DMAC**
(59.4)~~ — **and that block is GONE as of session 28.** `sc_in_data` refused a
windowed read because a channel cannot drop the 300 B in front of a record; the
container is DLX5 now and no record asks for a window. `xfer.i`'s sector
arithmetic already degenerates correctly — `SC_WSKIP` is 0 and `SC_WKEEP` is the
whole record on every one of the 120 — so **what is left of P4a is the wiring:
which loop moves the bytes.** `dma_run.sh`'s windowed-read refusal stays as a
negative control rather than as a description of the container. **This is the
next item, and it is the last one before M2** (60.9).
**P4c (new, and it is a DESIGN CHOICE the tree had not named).** Auto-request is
charged **by time, not by byte** — the channel spends its share of the bus
@@ -305,7 +436,9 @@ container). The last one wins on both axes and is a **re-encode**; see the
bundle under P2. **P4a should be attempted against a sector-aligned container,
not against this one.** *(Session 27: it was, in the only sense that mattered —
the transport now REFUSES the windowed case rather than being trusted not to
reach it, so the bundle is a precondition rather than a plan. 59.4.)*
reach it, so the bundle is a precondition rather than a plan. 59.4. Session 28:
the container IS one — the realised wire cost is +0.48% against the +0.43%
predicted here, and 120/120 records start on a sector boundary. 60.1.)*
*(original item, still the standing description of the `W` question:)*
Drive the MB89352 instead of a host file. **Session 23
@@ -408,11 +541,14 @@ interaction to price next, and it is E2's question with a second consumer in it.
on P6's answer and on P5's index.
**E7. A BYTE TARGET, AND IT COMES OUT OF THE BUS BUDGET RATHER THAN OUT OF
TASTE (new, session 27, FINDINGS 59.7).** The frame affords **6.74 clocks a
byte** after the measured decode and the audio, and a dual-address byte costs
**9**. So *if* B3 comes back saying the card cannot drive `#EXREQ`, the container
has to reach **27,995 B a frame — 328 KB/s of payload** to fit at 12 fps, where
the gate container sits at 37,403 B and 438 KB/s: **34% too big.**
TASTE (new, session 27, FINDINGS 59.7; re-measured session 28, 60.7).** The
frame affords **6.69 clocks a byte** after the measured decode and the audio,
and a dual-address byte costs **9**. So *if* B3 comes back saying the card
cannot drive `#EXREQ`, the container has to reach **27,924 B a frame — 327 KB/s
of payload** to fit at 12 fps, where the DLX5 gate container delivers 37,585 B
and 440 KB/s: **35% too big.** (It was 6.74 / 328 / 34% against the DLX4
container. **The bundle moved the target by one KB/s and moved no conclusion**,
which is what a precondition is supposed to do.)
Three things make this less alarming than the number looks, and one makes it
worse:
@@ -465,15 +601,24 @@ Listed for completeness; past M3 these are scope, not risk.
not menu vs content: the two largest streams are bonus material and look like
content by size, duration and bitrate alike (25.1). Run
`07_motion_survey.py` per stream first for a hot-window shortlist.
**Gated by E4.**
- **E4. `H.build` k-means**, 51 s of a 55 s run, once per scene. The thing to
attack before C1, and not anything in the per-frame path (27.6).
- **E2. `--spans all` as default.** *(re-encode bundle item 2; see P2.)* Still a recommendation, not a measurement
(43.6.1), and the only loaded lever on the encoder's byte side (44.3). **It
spends every profitable byte, which raises `wire`, which shrinks `pipe - wire`,
which lengthens the refill climb after every branch.** That interaction is not
priced, and M3 is where it becomes measurable.
- **E3. Re-derive span selection jointly with `lam`** (39.3). *(bundle item 3.)*
**Gated by E4, and parked with it (session 29).**
- ~~**E4. `H.build` k-means**, 51 s of a 55 s run, once per scene.~~ **PARKED,
session 29 (USER DECISION).** It was the thing to attack before C1. It builds
**VQ codebooks**, and a decoder-free packed player has no VQ — so this is
encoder work on the branch that is no longer being built on. It comes back if
and only if B2 goes MAME's way. C1 is gated by it and is parked with it.
- ~~**E2. `--spans all` as default.**~~ **MEASURED AND REFUSED, session 28 —
60.4.** It was "a recommendation, not a measurement" since 43.6.1 and it is a
measurement now: **+67% of the wire for +0.19 dB and one frame of 120.** It
IS the loaded lever on the byte side (44.3) — it is loaded the wrong way, and
E7 is why. What the entry predicted is exactly what it does: it raises `wire`,
which shrinks `pipe - wire`, which lengthens the refill climb. `need` stays
the default; the GATE container keeps `all` because it is a fixture.
- ~~**E3. Re-derive span selection jointly with `lam`** (39.3).~~
**IMPLEMENTED, MEASURED, NO-OP, session 28 — 60.5.** `--joint-spans` emits
byte-identical containers in all four `{need,all} x {greedy,joint}` cells and
at both lam floors, because **`lam` never leaves its floor on any of 120
frames**. Kept and defaulted off.
- **C2. Framing** — crop vs squash vs wide (FINDINGS 12). Needs an eyeball
against arcade reference, not a measurement. Cheap; blocks only final encodes.
- **C3. Disk image packaging**, ~1.09 GiB at the candidate rate.
@@ -487,15 +632,26 @@ Listed for completeness; past M3 these are scope, not risk.
P4a DONE (59): the channel drives the data phase and
holds the bus -- 391.7% of a frame becomes 40..95%
│
B3 #EXREQ? ──┬─ YES ─> single address, 5 clk/B, 92.2% ── FITS ──┐
B3 #EXREQ? ──┬─ YES ─> single address, 5 clk/B, 92.4% ── FITS ──┐
│ │
└─ NO ──> auto-request, 9 clk/B FLOOR, 110.2% ──> E7 byte target
(the frame affords 6.74; 59.7) 328 KB/s
└─ NO ──> auto-request, 9 clk/B FLOOR, 110.4% ──> E7 byte target
(the frame affords 6.69; 59.7, 60.7) 327 KB/s
│
P2 re-encode bundle (sector alignment is now a PRECONDITION, 59.4) ┤
P1 P3 P4b P5 P7, P6 bus cost (52), G1 scene graph (56) ────────────┼─> M2 ─>
P2 re-encode bundle DONE (60): DLX5, records ARE sectors ─────────┤
E2 refused on measurement, E3 a no-op -- 60.4, 60.5 │
E7/E4/C1 PARKED session 29: encoder work waits on B2 ───────────┤
│
K1 palette-register DMA? ─> K2 packed container ─> K3 end to end ──┤
(61.9; K1/K2 survive a bad B2, K3 does not) │
P4a WIRING (the channel behind ring.i's mailbox) <- THE LAST ITEM ─┤
P1 P2 P3 P4b P5 P7, P6 bus cost (52), G1 scene graph (56) ────────┼─> M2 ─>
B1 seek+rate (sets HEADROOM, not fit) ─────────────────────────────┘ M3 ─> M4
B2 blanking ─> (page 1; do not pre-build on it)
B2 blanking ─┬─ NOT blanked ─> the DECODER-FREE PACKED player is live (61):
│ 55.2% of a frame at the 9 clk/B FLOOR, against
│ the codec's 110.4% -- and then B1 decides it,
│ because it wants 576 KB/s FIXED (61.4, 61.5)
└─ blanked ──────> a 27..113% duty strobe; there is no version of
it that is merely expensive (48.3, 61.6)
```
**Read that top-left branch as the project's live question.** Everything else
docs/STATUS.md
+258 -1
View File
@@ -1,4 +1,261 @@
# Status & next-session handoff — end of session 27 (2026-08-24)
# Status & next-session handoff — end of session 29 (2026-08-25)
## Session 29: the decoder-free packed player fits the budget the codec misses
**Green light first and last: `./tools/bench/check.sh` was ALL GREEN before any
of this and ALL GREEN after**, plus one new stage.
**FINDINGS 61.** **Emulated** — MAME 0.277, `x68000 -bios ipl10`, and
`-exp1 cz6bs1` for the transport runs. No real hardware ran. 47.4 — does buffer
mode blank the layer — is still the board question and is still open.
**THE HEADLINE.** At the 9 clk/B dual-address floor, which 59.2 established is
the only configuration this machine can be shown to run:
| architecture | B/frame | % of a 12 fps frame at W=9 | KB/s |
|---|---:|---:|---:|
| **CODEC, the shipping design** | 37,585 | **110.4% — MISSES** | 440.4 |
| **decoder-free PACKED, DMAC device→GVRAM** | 49,152 | **55.2% — FITS** | 576.0 |
| **decoder-free PACKED, CPU-painted** | 49,152 | **81.6% — FITS** | 576.0 |
**Decoding 37,585 bytes costs 109% of a frame; NOT decoding 49,152 costs 54%.**
44.7 said this in advance on a cost model that has since been thrown away — "the
codec is not there to save CPU, it is there to save the wire" — and the measured
model agrees and goes further. **The question is now entirely a medium question.**
**1. THE PACKED PAINT IS MEASURED, and 47.6.1's assumption is retired.**
`blit.s` V8 is V1 with 128 words a row instead of 256. **227,553 clocks, 27.3%
of a frame** — 51.0% of V1, and **100.8% of V3, the unpacked WRITE-ONLY floor**.
The packed blit costs what the unpacked path pays to write alone, source read
thrown in free. V1 came out at 53.6% again, unmoved since session 9, which is
what licenses quoting V8 next to it.
**2. A CHANNEL FILLS GVRAM OFF THE DISC, AND WALKS THE LINE STRIDE ITSELF.**
Three new runs in `dmagate.s`, all gated: bit 11 SET → 2,048 B byte-exact, half
into page 1 and half into page 0, CPU never executed; bit 11 CLEAR → **457 bytes
lost, every one at an EVEN offset and not one ODD byte harmed**; and **array
chained, 8 rows at the 1,024 B line stride from ONE start.** 47.6.2 is answered.
**3. THE FIRST CUT OF THAT TEST COULD NOT HAVE FAILED.** It OR-ed bit 11 onto
whatever the IPL left, and the IPL leaves `$0B16` — bit 11 already set, COL
`%11`, which writes whole words either way. It passed and proved nothing. Both
runs now write R20 outright and differ in exactly one bit.
**4. AND THE CONTROL'S SIGNATURE IS A PLACE, NOT A COUNT.** "Half the bytes must
differ" turned the gate RED on a correct run: `gvram_w` ignores `mem_mask` in the
masked arm, so the high half keeps stale GVRAM, and where the record is pad the
stale half matches by coincidence — 567 of 1,024 did. The claim the mechanism
actually makes is that odd bytes survive and only even ones are lost.
**5. THE CODEC CANNOT HAVE THE PACKING — 47.6.4 CLOSED, both ways.** A 4×4 block
owns four bytes at stride 2. Writing them one at a time (V9) is **97.9% of a
frame against V4's 76.6% — 28% DEARER, for no saving on the wire at all.**
Pairing the blocks 128 columns apart (V10) gets the `movem` back at 38.5%, and
pays in the mode map: **SKIP falls from 66.3% of blocks to 46.1% of pairs**, so
1.60× as many blocks are painted — about **+60% on the bytes**, against an E7
target that needs them **down 35%**.
**6. WHAT B1 AND B2 NOW DECIDE.** 576 KB/s sustained, fixed, with no lever —
against 453.6 KB/s (gate container, zero prefill), 327 KB/s (E7's target) and
folklore that says 0.7–1.7 MB/s with no benchmark behind it. And under MAME's
blanking reading the **cheap architecture is the dark one**: a CPU-painted packed
player is black for 27.3% of a frame, the cheaper DMAC-direct one for 30–113%,
because the black interval is the paint and the CPU's transport lands in RAM
where bit 11 is irrelevant.
**7. AND THE PICTURE IS BETTER, WHICH NOBODY HAD ASKED (61.9).** The codec is
**capped at 31.33 dB** — every codeword is an index into the scene palette — and
delivers 29.19. A literal frame has no codebooks, so per-frame palettes are legal
and it simply **IS 34.08 dB: +4.89 dB on the shipping container, +2.75 dB past a
ceiling the codec cannot cross**, for +1.0% on the wire and 0.28% of a frame.
**The packed player is not a quality compromise bought with clocks. It is better
on both, and its whole cost is on the wire.**
**ENCODER WORK IS PARKED (USER DECISION).** The handoff first said this was no
reason to stop codec work. It does not survive its own arithmetic: that rested on
576 KB/s against **E7's 327, which does not exist**, when the codec that exists is
440 KB/s and 110.4% of a frame — a 1.31x gap, not 1.76x. Packed needs two facts;
the codec needs E7 to succeed (two of three levers already measured inert, 60.4
and 60.5) AND the medium AND a ship at ~100% of the frame with no margin. **E7 and
E4 are both parked** — E4 too, because `H.build`'s k-means builds VQ codebooks and
a literal player has no VQ — and C1 follows E4. **The codec stays on disk and
nothing is built on it**, because 48.1's prior leans against packing and if
buffer mode blanks it is the only path left (48.3). That is inventory, not work.
---
## HANDOFF — start here
**THE TREE IS ALL GREEN** (`tmp/check_s29_handoff.log`, and four times before it this session), and **DIRTY WITH TWO
SESSIONS IN IT.** Session 28's work was never committed either, so `git status`
shows 26 modified files spanning both sessions plus one new file. The last
commit is session 27's (`8800d8f`, "Price the transport against the frame").
**Nothing here is half-finished — it is unrecorded, which is a different
problem.** Commit before starting anything, or the next green light cannot be
attributed to a session.
**THE DIRECTION CHANGED (USER DECISION, session 29): build the DECODER-FREE
PACKED player, keep the codec on disk and build nothing on it.** The reasoning is
61.8; the numbers that forced it are 61.4 (55.2% of a frame against the codec's
110.4%) and 61.9 (+4.89 dB, and the codec is *capped* 2.75 dB below it).
### The work, in the order it should be done
**1. CAN A DMA CHANNEL WRITE THE PALETTE REGISTERS AT `$E82000`? (61.9, open)**
The cheapest item and the one with the largest structural payoff. If the palette
takes a byte-wide DMA the way GVRAM does in buffer mode, it becomes a **193rd
array-chain entry** and ONE channel start paints a whole frame — palette
included, CPU halted throughout, no per-frame CPU work in the video path at all.
It is the same shape as the three probes that worked this session: another run in
`src/player/dmagate.s`, host-side byte comparison in `tools/bench/dma.lua`, an
assertion in `tools/bench/dma_run.sh`, **and a negative control** — see the traps
below, this is exactly where session 29 nearly shipped a vacuous pass.
**2. A PACKED CONTAINER.** Simpler than a coded one, not harder: quantise to 254
colours, interleave column *i* with column *i+128*, emit a per-frame palette,
sector-align the record. No VQ, no rate control, no mode map, no `lam`.
`encode.py` already sector-aligns (DLX5) and
`tools/bench/prep_frame.py --pack-transparent` already emits the exact pixel
layout 47.2 verified pixel-exactly on both emulators.
**3. END TO END, OFF THE DISC.** Scene setup — palette, page-1 X-scroll **384**,
priority `vc1 = 0x0002`, R20 bit 11 — then one chained DMA a frame. Gate it the
way the existing decoder is gated: pixel-exact over the same 120-frame window
against `dlx.py`'s reconstruction.
### What is PARKED, so it is not re-derived
**E7, E4 and C1** (61.8). E4 is included on purpose: `H.build`'s k-means builds
**VQ codebooks**, and a literal player has no VQ. **P4a's wiring** — the channel
behind `ring.i`'s `XF_*` mailbox — is parked *for the video path* too: a
DMAC-direct packed player has **no ring at all**. Do not delete any of it; B2 is
unanswered and 48.1's prior leans against packing.
### Risks that are OURS, not hardware
1. **A chained transfer has never run back to back at 12 fps.** 61.2's run 6 was
eight rows, once. A frame is 192 entries and a second is twelve frames.
2. **34.08 dB is PIL's MEDIANCUT**, not `vq.scene_palette` / `H.build`. The
direction is measured; re-derive the number against the builder that ships it
before quoting it as the player's.
3. **The packed player deletes `ring.i`, `xfer.i` and most of `stream.s` from the
video path.** A simplification that large usually hides something.
4. **B2 sits over all of it.** Items 1 and 2 survive a bad answer; item 3 does
not.
### Traps this session stepped in, both caught by the tree's own rules
**A test that could not fail.** The first GVRAM run OR-ed R20 bit 11 onto
whatever the IPL left — and the IPL leaves `$0B16`, bit 11 **already set**, COL
`%11`, which writes whole words either way. It passed and proved nothing. Both
runs now write R20 outright and differ in exactly one bit.
**An assertion that was right about the mechanism and wrong about the number.**
"The masked control must lose half the bytes" turned the gate RED on a correct
run: `gvram_w` ignores `mem_mask` in the masked arm, so the high half keeps stale
GVRAM, and where the record is pad it matches the disc by coincidence — 567 of
1,024 did. **The mechanism's signature was a PLACE, not a count**: odd bytes
survive, only even ones can be lost. That is what the gate asserts now.
### Reproducing this session
./tools/bench/check.sh # ALL GREEN, ~5 min
python3 tools/analysis/29_packed_player.py tmp/rc_fr_singe_scsi_span.dlx
bash tools/bench/dma_run.sh # runs 4-6, with controls
python3 tools/analysis/18_text_plane_16col.py # the PSNR rows of 61.9
`check.sh` gained one stage that runs `blit.lua` and gates the ORDER of the paint
costs, not their values — V8 < V1, V9 > V4, V10 < V4 — because a tree where any
of those flipped has a different answer to 44.7 and should say so out loud.
**WHAT IS NEXT.** Item 1 above: the palette-register DMA probe.
---
# Status & next-session handoff — end of session 28 (2026-08-25)
## Session 28: the container agrees with the medium, and two encoder levers are not levers
**Green light first and last: `./tools/bench/check.sh` was ALL GREEN before any
of this and ALL GREEN after** — and after a full re-encode of the gate
container, which is the first time this tree has replaced it.
**ROADMAP's RE-ENCODE BUNDLE (all four items under P2) is DONE. FINDINGS 60.**
**Emulated** — MAME 0.277 and px68k's C68K core. No real hardware ran.
**1. DLX5 — RECORDS ARE SECTORS.** `encode.py` pads every record to 512 B and
the scene header with it, so the frame stream starts on a sector boundary and
every record inside it does. `dlx.py` reads `DLX5`, and `record_lengths()` is
now the ONE place the alignment rule lives — it replaced four hand-copied
`4 + n + (-(4+n) % 4)` expressions in the analysis tools.
| gate container | DLX4 | **DLX5** |
|---|---:|---:|
| records starting on a sector boundary | 3/120 | **120/120** |
| off the disc, for 4,488,577 B of record | 4,548,608 (+1.34%) | **4,510,208 (+0.48%)** |
| into the ring | 4,488,588 | **4,510,208** |
| window or bounce copy | 0 PIO / +5 clk/B DMA | **0, and a channel can run it** |
The two byte counts are now the SAME NUMBER and check.sh gates on that
identity, both read out of the container rather than written into the script.
**2. THE CONSUMER HAD TO BE TOLD, and the ring's own audit caught it.**
`stream.s` released up to the last byte it READ, rounded to 4; under DLX5 that
strands up to 511 B of pad per record. `RD_PTR MISMATCH after frame 0` on the
first run. The release rounds to `RECALN` (geom.i) now. Every frame still
decoded pixel-exact while the free-space arithmetic was wrong, which is the
point: alignment is a contract with two sides.
**3. RESERVED BLACK (23.4, open since session 5).** The scene palette is 255
picture colours with index 0 held at (0,0,0), so the letterbox is true black and
`I = 0` falls out of `pack_palette`'s existing rule with no special case.
**0.04 dB** on the Singe window; the palette ceiling does not move.
**4. `--spans all` AS THE DEFAULT: MEASURED, AND REFUSED.** 267.9 KB/s / 28.88
dB / 2 frames late at `need`, against 448.2 KB/s / 29.07 dB / 1 late at `all`.
**+67% of the wire for +0.19 dB and one frame of 120**, against a container the
frame budget already says is 35% too big. `need` stays the default; the GATE
container keeps `all`, because it is a fixture and not a recipe.
**5. JOINT SPAN/LAM (E3): implemented, and it is a NO-OP.** `--joint-spans`
hands the span pass's freed bytes back to the lam search and re-spans. All four
cells of `{need,all} x {greedy,joint}` are BYTE-IDENTICAL, and so is
`--rc-floor open`. The reason is structural: **`lam` never leaves its floor on
any of 120 frames at either floor the encoder offers** (44.3), so there is
nothing to spend the freed bytes on. Kept, defaulted off.
**6. THE APPARATUS LIED SILENTLY.** MAME 0.277 served the compressed CHD's OWN
FILE BYTES as sector data — the buffer after READ(10) at LBA 0 was byte-for-byte
`dlxdisk.chd`'s first 4,096 bytes, starting `MComprHD` — while every READ
reported success and `chdman verify` passed. Not size, not geometry, not the
codec: the DLX5 image fails compressed at the working volume's exact length and
with a conventional geometry, and passes `-c none`. `mkvol.sh` builds
uncompressed now. **Only the per-byte comparison in `scsi.lua` could see it**; a
gate checking status and length would have been green all session.
**7. THE BUDGET BARELY MOVED, which is what a precondition should do.**
Decode 68.5% → **68.6%**, headroom 6.74 → **6.69 clk/B**, `W`=5 total 92.2% →
**92.4%**, E7's target 328 → **327 KB/s**, B1's zero-prefill requirement 451.4 →
**453.6 KB/s**. Every conclusion in 59.7 survives. `15_bus_occupancy.py` now
charges the **padded** record, because the disc carries the pad whether or not a
frame contains it.
**8. TWO CACHES WOULD HAVE TESTED THE WRONG ARTEFACT.** `16_span_roundtrip.py`
and `13_cpu_ratectl.py` cache `H.build` in a pickle keyed on the frames
directory alone; `H.build` gained an option this session. Both now store the
build parameters with the model and rebuild on a mismatch.
**WHAT IS NEXT.** P4a's wiring: the DMA channel behind `ring.i`'s `XF_*` mailbox
in place of `xfer.i`'s PIO loop, which 59.4 blocked on exactly this
precondition. `xfer.i`'s sector arithmetic already degenerates correctly —
`SC_WSKIP` is 0 on every record now — so what changes is which loop moves the
bytes. `dma_run.sh`'s windowed-read refusal becomes a negative control.
---
# Status at end of session 27 (2026-08-24)
## Session 27: the DMAC drives the data phase, and auto-request is charged by time
src/player/dma.i
+25
View File
@@ -67,6 +67,8 @@ DM_CCR = DMACB+$07 ; channel control
DM_MTC = DMACB+$0A ; memory transfer count, WORD
DM_MAR = DMACB+$0C ; memory address, LONG
DM_DAR = DMACB+$14 ; device address, LONG
DM_BTC = DMACB+$1A ; base transfer count, WORD (array chain)
DM_BAR = DMACB+$1C ; base address, LONG (array chain)
DM_MFC = DMACB+$29
DM_CPR = DMACB+$2D
DM_DFC = DMACB+$31
@@ -113,6 +115,17 @@ DM_CERF = $18518 ; u32 CER with it
DM_MTCF = $1851C ; u32 MTC with it
DM_MARF = $18520 ; u32 MAR with it -- where it stopped
DM_LEN = $18524 ; u32 bytes the channel was asked for
; ---- SEQUENTIAL ARRAY CHAINING, and it is OFF unless a caller asks for it.
; A device->GVRAM transfer cannot be one contiguous run: a picture row is 256 B
; of a 1024 B line stride, so 192 rows want 192 destinations. The MC68450 walks
; an array of 6-byte {u32 MAR, u16 MTC} entries for exactly this, and MAME's
; hd63450 implements it (`(ocr & 0x0c) == 0x08`, dma_transfer_start and
; dma_transfer_continue). DM_BARV = 0 means no chaining and NOTHING below
; changes, which is what stream.s gets: this file is included by the player as
; well as by the gate, and a mailbox that defaults to a new behaviour is how
; DM_USE landed on ring.i's slot and turned a stage red (above).
DM_BARV = $18528 ; u32 array base address, 0 = no chain
DM_BTCV = $1852C ; u32 array entry count (BTC)
DM_PATIENCE = 4000000 ; the wait is bounded like every other
; ---------------------------------------------------------------- sc_in_dma
@@ -144,8 +157,20 @@ sc_in_dma:
move.b #$05,DM_MFC ; the function codes the IPL ROM uses
move.b #$05,DM_DFC
move.b #$01,DM_CPR
move.l DM_BARV.l,d0
bne.s .chain
move.w d5,DM_MTC
move.l a1,DM_MAR
bra.s .darset
.chain:
; MAR and MTC are NOT written: the channel loads both from the array's
; first entry when it starts, and reloads them from the next entry at
; every count-out. Writing them here would be writing registers the
; hardware is about to overwrite, which reads like a contract and is not.
move.l d0,DM_BAR
move.l DM_BTCV.l,d0
move.w d0,DM_BTC
.darset:
move.l #SC_DREG,DM_DAR ; $EA0015 -- the DMAC's door, and now
; the DMAC is the one going through it
move.b #SCMD_XFER,SC_SCMD ; no PROGRAM bit: the SPC raises DRQ
src/player/dmagate.s
+115 -4
View File
@@ -37,10 +37,22 @@
; burst modelled as held.
DGFLAG = $18600 ; 0 idle / 1 done
DGREC = $18610 ; 3 x 32 B: rc, err, mtc0, spin, csr, cer, mtc, mar
DGREC = $18610 ; 6 x 32 B: rc, err, mtc0, spin, csr, cer, mtc, mar
DGREC_SZ = 32
DGWIN = $18680 ; u32 return of the WINDOWED dma read (want -1)
DGWERR = $18684 ; u32 SC_ERR after it (want SCE_WINDOW)
DGWIN = $18700 ; u32 return of the WINDOWED dma read (want -1)
DGWERR = $18704 ; u32 SC_ERR after it (want SCE_WINDOW)
DGR20 = $18708 ; u32 R20 as it stood during run 4
DGR20N = $1870C ; u32 R20 as it stood during run 5 (the control)
DGR20C = $18710 ; u32 R20 as it stood during run 6 (chained)
R20_BUF = $0916 ; 256-colour, 31.5 kHz, G-VRAM SET TO BUFFER
R20_MSK = $0116 ; the SAME, bit 11 CLEAR -- the negative control
DGDST4 = $C0C000 ; where the control writes
DGCHA = $19000 ; the array-chain table: 8 x {u32 MAR, u16 MTC}
DGCHN = 8 ; entries
DGCROW = 256 ; bytes an entry carries -- one packed picture row
DGDST5 = $C10000 ; and the 8 row bases, at the 1024 B line stride
CRTC20 = $E80028 ; CRTC R20
DGDST3 = $C08000 ; DMA straight into GVRAM, BUFFER MODE (47.6.2)
DGLBA = 1000 ; a NON-ZERO LBA throughout: a driver that emits
; a malformed LBA field still passes LBA 0
DGBLK = 4 ; 4 x 512 = 2,048 B
@@ -88,7 +100,106 @@ start:
lea DGREC+2*DGREC_SZ,a0
bsr dg_save
; ---- 4. and a WINDOWED read through the channel, which must be REFUSED. This
; ---- 4. THE CHANNEL WRITING GVRAM, IN BUFFER MODE. 47.6.2: "the DMAC has not
; been near this" -- 44.7 costed a device->GVRAM transfer and 47 built the packed
; layout, and no run in this tree has ever pointed a channel at $C00000. Two
; separate things are being asked at once and both are write-path questions:
;
; a. can a channel write GVRAM AT ALL? Nothing says it cannot -- the DMAC
; writes through the same program address space the CPU does -- but a
; transport that silently drops its writes into a device handler is exactly
; the failure this gate exists to catch, and the SPC's own bytes make a
; better witness than a fill pattern.
; b. does a BYTE-wide channel fill the PACKED layout? 47.1 measured the write
; path with word writes from the CPU. A dual-address channel with an 8-bit
; device port writes BYTES, and MAME's gvram_w in buffer mode passes
; mem_mask straight through -- so an even byte should land in the HIGH half
; of its word (page 1) and an odd byte in the LOW half (page 0). If it
; does, a linear DMA of a stream interleaved (right<<8)|left -- which is
; exactly show_frame256_packed.lua's layout -- fills the screen with no CPU
; in the loop at all.
;
; R20 bit 11 is left SET across the readback on purpose: gvram_r returns the raw
; word in buffer mode and the masked byte outside it, so clearing it first would
; hide half of what is being measured.
;
; R20 IS WRITTEN OUTRIGHT AND NOT OR-ED INTO. The first cut of this run set bit
; 11 on top of whatever the IPL left, and the IPL leaves $0B16 (22.1) -- bit 11
; ALREADY SET, and COL = %11, the 65,536-colour setup, which writes whole words
; with or without the bit. The run passed and proved nothing: it was a test that
; could not fail. Run 5 below is the control that makes this one mean something,
; and the two differ in EXACTLY BIT 11.
bsr scsi_init
move.w #R20_BUF,CRTC20.l
move.l #R20_BUF,DGR20.l
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST3,a1
bsr scsi_read
lea DGREC+3*DGREC_SZ,a0
bsr dg_save
; ---- 5. THE NEGATIVE CONTROL: the same transfer with bit 11 CLEAR. In masked
; 256-colour mode gvram_w takes `data & 0x00ff` and IGNORES mem_mask, so a byte
; written to an EVEN address -- where the 68000 puts the MS byte, and where every
; other disc byte lands -- contributes nothing and cannot be read back. Half the
; transfer must be lost, and if it is not, run 4 was not measuring the bit.
bsr scsi_init
move.w #R20_MSK,CRTC20.l
move.l #R20_MSK,DGR20N.l
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR,DM_OCRV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST4,a1
bsr scsi_read
lea DGREC+4*DGREC_SZ,a0
bsr dg_save
move.w #R20_BUF,CRTC20.l ; back to buffer mode, so the host reads
; RAW WORDS out of both destinations
; ---- 6. THE ROW STRIDE, WHICH IS THE REAL SHAPE OF THE TRANSFER. Runs 4 and 5
; wrote 2,048 contiguous bytes, and no picture is contiguous: a packed row is
; 256 B of a 1024 B line stride, so a frame is 192 destinations and not one.
; 46.6 said "no stride for a DMAC to skip" about the bytes WITHIN a row and left
; the rows themselves unexamined; a channel cannot skip 768 B any more than it
; could skip the 300 B in front of a record (run 7 below).
;
; The MC68450 answers this with SEQUENTIAL ARRAY CHAINING -- an array of 6-byte
; {u32 MAR, u16 MTC} entries it walks by itself -- and MAME implements it. So
; the question is not whether the CPU can restart the channel 192 times a frame;
; it is whether it has to at all. Eight rows here, out of the same LBA and the
; same 2,048 B, so the ONLY difference from run 4 is where the bytes land.
bsr scsi_init
move.w #R20_BUF,CRTC20.l
move.l #R20_BUF,DGR20C.l
lea DGCHA,a0
lea DGDST5,a1
moveq #DGCHN-1,d5
dg_mkch:
move.l a1,(a0)+ ; MAR: this row's base
move.w #DGCROW,(a0)+ ; MTC: 256 bytes of it
lea 1024(a1),a1 ; the next row is a line stride away
dbra d5,dg_mkch
move.l #DM_HELD_DCR,DM_DCRV.l
move.l #DM_HELD_OCR|$08,DM_OCRV.l ; OCR CHAIN = %10, array chain
move.l #DGCHA,DM_BARV.l
move.l #DGCHN,DM_BTCV.l
move.l #1,DM_USE.l
move.l #DGLBA,d3
moveq #DGBLK,d4
lea DGDST5,a1 ; ignored under chaining; passed so the
; call site reads the same as the others
bsr scsi_read
lea DGREC+5*DGREC_SZ,a0
bsr dg_save
clr.l DM_BARV.l ; and OFF again, so run 7 is unchained
; ---- 7. and a WINDOWED read through the channel, which must be REFUSED. This
; is the one test here that is expected to fail, and it has to fail LOUDLY: the
; alternative is a channel writing a whole sector into a ring that has room for
; a record, over the top of records the decoder has not finished with.
src/player/geom.i
+14
View File
@@ -25,3 +25,17 @@ SPCU = 12 ; bytes of code per COARSE span unit (24 px)
SPCN = 11 ; coarse units: 11*24 = 264 px >= one row
SPFU = 2 ; bytes of code per FINE span unit (2 px)
SPFN = 11 ; fine units: 11*2 = 22 px > one coarse unit
; RECORD ALIGNMENT, and it is a property of the CONTAINER rather than of the
; 68000. DLX2 padded each record up to 4, which is all `move.l (a0)+` needs
; (FINDINGS 28.3). DLX5 pads up to a 512 B SECTOR, so that a DMA channel can
; read a record as whole sectors straight into the ring with no window and no
; bounce copy -- `sc_in_data` REFUSES a windowed read when the data phase is
; the channel's (59.4), and 117 of 120 records needed one under DLX4.
;
; The consumer has to know it too: the decoder releases the ring up to the end
; of the record it was handed, and a decoder that released only the bytes it
; READ would leave the pad unreclaimed and drift the producer's free-space
; arithmetic by up to RECALN-1 per record. The ring base must therefore be
; RECALN-aligned, which is asserted where the ring is placed.
RECALN = 512
src/player/stream.s
+9 -5
View File
@@ -312,12 +312,16 @@ nostall:
cmpa.l SCR_END.l,a0 ; bitstream desync is silent otherwise
bne desync
; ---- release. Round up to 4 the same way decode.s does: the producer lays
; records on 4-byte boundaries, so the byte one past this record's padded
; end is the first byte the producer may reuse.
; ---- release. Round up to RECALN: the producer lays records on the
; container's own record boundaries (geom.i), so the byte one past this
; record's PADDED end is the first byte it may reuse. Releasing only the
; bytes actually read would strand up to RECALN-1 of pad per record and
; the producer's free-space arithmetic would drift by that much a frame.
; This is an absolute address, so it is only the record's padded end
; because the ring base is RECALN-aligned as well.
move.l a0,d0
addq.l #3,d0
and.b #$FC,d0
addi.l #RECALN-1,d0
andi.l #~(RECALN-1),d0
move.l d0,RD_PTR.l
addq.l #1,FR_TAIL.l
tools/analysis/13_cpu_ratectl.py
+14 -2
View File
@@ -40,12 +40,24 @@ BUDGET = RC.FRAME_CYCLES
# H.build is ~55 s, nearly all k-means, and it does not depend on the profile:
# both ship k1=k4=256. One build, cached, serves every row of the table.
cache = a.cache or f"tmp/model_{os.path.basename(a.frames_dir.rstrip('/'))}.pkl"
# The build parameters are stored with the model and a mismatch rebuilds: the
# cache is keyed on the frames directory alone, and once H.build acquired an
# option (session 28's reserved black entry, 23.4) a stale pickle would quietly
# serve a model the shipping encoder no longer builds. Same guard as
# tools/analysis/16_span_roundtrip.py.
SIG = dict(k1=256, k4=256, iters=16, reserve_black=True)
m = None
if os.path.exists(cache):
m = pickle.load(open(cache, "rb"))
print(f"model from {cache}")
if m.get("sig") != SIG:
print(f"{cache}: built with {m.get('sig')}, wanted {SIG} -- rebuilding")
m = None
else:
print(f"model from {cache}")
if m is None:
t = time.time()
m = H.build(a.frames_dir, k1=256, k4=256, iters=16)
m = H.build(a.frames_dir, **SIG)
m["sig"] = SIG
pickle.dump(m, open(cache, "wb"))
print(f"built model in {time.time()-t:.0f} s -> {cache}")
print(f"{a.frames_dir}: {len(m['idx'])} frames, {m['nb']} blocks, "
tools/analysis/15_bus_occupancy.py
+7 -1
View File
@@ -167,7 +167,13 @@ print("THE OTHER TWO MASTERS -- what the DMAC takes out of the same frame\n")
FPS = d.fps
CPUHZ = 10e6 # stock X68000, MAME 0.277 x68k.cpp:1133
FRAME_CLK = CPUHZ / FPS
vid_bpf = sum(n + 4 for (_, n) in d.frames[:NF]) / NF # DLX2 record padding
# WHAT THE TRANSPORT MOVES, which is the PADDED record and not the payload.
# Under DLX2/3/4 the pad was 0..3 B and the distinction was noise; under DLX5 it
# is 0..511 B, and charging the payload would price the channel for bytes it
# does not carry while the disc carries them anyway. A budget that debits only
# the bytes a frame CONTAINS is the same incomplete accounting this project has
# been caught by before -- the pad is delivered, so the pad is charged.
vid_bpf = sum(d.record_lengths()[:NF]) / NF
aud_bpf = B.ADPCM_BYTES_PER_S / FPS
a_lo = aud_bpf * B.ADPCM_CLK_BYTE_BEST
a_hi = aud_bpf * B.ADPCM_CLK_BYTE_WORST
tools/analysis/16_span_roundtrip.py
+17 -3
View File
@@ -43,12 +43,26 @@ ap.add_argument("--cache", default=None)
a = ap.parse_args()
cache = a.cache or f"tmp/model_{os.path.basename(a.frames_dir.rstrip('/'))}.pkl"
# The cache is keyed on the frames directory ALONE, which was fine while
# H.build had no options and became a trap the moment it did: session 28's
# reserved black entry (23.4) changes the palette, the codebooks and every
# index in the model, and a pickle from before it would have let this gate
# round-trip a container the shipping encoder no longer emits -- green, and
# testing the wrong artefact. So the build parameters are stored WITH the
# model and a mismatch rebuilds.
SIG = dict(k1=256, k4=256, iters=16, reserve_black=True)
m = None
if os.path.exists(cache):
m = pickle.load(open(cache, "rb"))
print(f"model from {cache}")
if m.get("sig") != SIG:
print(f"{cache}: built with {m.get('sig')}, wanted {SIG} -- rebuilding")
m = None
else:
print(f"model from {cache}")
if m is None:
t = time.time()
m = H.build(a.frames_dir, k1=256, k4=256, iters=16)
m = H.build(a.frames_dir, **SIG)
m["sig"] = SIG
pickle.dump(m, open(cache, "wb"))
print(f"built model in {time.time()-t:.0f} s -> {cache}")
@@ -79,7 +93,7 @@ for span_mode in ("need", "all"):
# of the frame stream, so simply constructing it above has already gated
# that. Said out loud here because it is easy to read this as version drift.
if d.has_index:
print(f" DLX4: record index agrees with the frame stream on all "
print(f" DLX{d.version}: record index agrees with the frame stream on all "
f"{d.nframes} records ({2*d.nframes:,} B of scene header)")
# The decoder's own walk of the span section must land exactly where the
tools/analysis/19_ring_stream.py
+1 -1
View File
@@ -71,7 +71,7 @@ def records(path):
the ring must hold per frame are the padded ones, not the payload.
"""
d = DLX(path)
rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
rec = np.array(d.record_lengths(), np.int64)
return d, rec
tools/analysis/20_seek_slack.py
+1 -1
View File
@@ -32,7 +32,7 @@ SECTOR = 512
def records(path):
d = DLX(path)
rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
rec = np.array(d.record_lengths(), np.int64)
return d, rec
tools/analysis/22_scene_load.py
+1 -1
View File
@@ -79,7 +79,7 @@ print(f" once at boot {boot_cyc:>8,} clocks = {1000*boot_cyc/CPUHZ:6.2f}
for path in a.containers:
d = DLX(path)
hdr = int.from_bytes(d.raw[28:32], "big")
rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
rec = np.array(d.record_lengths(), np.int64)
wire = rec.mean() * FPS / 1024 + RC.AUDIO_KBPS
print(f"\n=== {path}: header region {hdr:,} B "
f"(pal 768 + cb1 {d.k1*16:,} + cb4 {d.k4*4:,} + 32), wire {wire:.1f} KB/s")
tools/analysis/26_sector_align.py
+18 -11
View File
@@ -66,16 +66,21 @@ a = ap.parse_args()
d = DLX(a.container)
# The disc layout the 68000 walks: [u32 len][body], each record padded up to 4.
# Exactly tools/bench/prep_stream.py's, and it is rebuilt here rather than read
# from tmp/ so this tool works on any container.
# The disc layout the 68000 walks: [u32 len][body], each record padded up to the
# container's own alignment -- 4 on DLX2/3/4, 512 on DLX5. Exactly
# tools/bench/prep_stream.py's, and it comes from the reader rather than from a
# second copy of the rule here, so pointing this tool at a DLX5 container asks
# it the RIGHT question: what does the mismatch still cost once the container
# has been changed to remove it? (The answer had better be nothing.)
off, recs = 0, []
for (o, n) in d.frames:
ln = 4 + n
ln += (-ln) % 4
for ln in d.record_lengths():
recs.append((off, ln))
off += ln
payload = sum(ln for _, ln in recs)
# The DENOMINATOR is the record bytes the decoder actually reads -- [u32 len]
# plus payload -- and NOT the padded length, because on a DLX5 container the
# padding IS the cost being measured. Scoring against the padded length would
# make an already-aligned container report +0.00% and look free.
payload = sum(4 + n for _, n in d.frames)
nfr = len(recs)
budget = CPUHZ / d.fps
@@ -120,8 +125,10 @@ for _, ln in recs:
pad += SECTOR - (cur % SECTOR)
cur += SECTOR - (cur % SECTOR)
cur += ln
print("C. SECTOR-ALIGNED RECORDS (a container change; a re-encode)")
print(f" wire {cur:,} B for {payload:,} B of record = +{100*pad/payload:.2f}%")
print("C. SECTOR-ALIGNED RECORDS (a container change; a re-encode)"
+ (" -- THIS CONTAINER ALREADY IS ONE" if d.sector_aligned else ""))
print(f" wire {cur:,} B for {payload:,} B of record = "
f"+{100*(cur-payload)/payload:.2f}%")
print( " clocks ZERO: the read is a whole-sector read straight into the "
"ring, no window,")
print( " no copy, and the DMAC can do it.")
@@ -130,8 +137,8 @@ print()
ringsz = a.ring * 1024
print(f" VERDICT, in the currency this project prices delivery in. C is "
f"cheaper on the wire")
print(f" than A and B by {100*(drop_a-pad)/payload:.2f} points of the payload "
f"({drop_a-pad:,} B on this scene),")
print(f" than A and B by {100*(wire_a-cur)/payload:.2f} points of the payload "
f"({wire_a-cur:,} B on this scene),")
print(f" and it is the only one of the three a DMA channel can run without a "
f"copy. What it")
print(f" costs is a container revision and the re-measurement that comes with "
tools/analysis/29_packed_player.py
+377
View File
@@ -0,0 +1,377 @@
#!/usr/bin/env python3
"""The DECODER-FREE PACKED player, priced against the measured cost model.
python3 tools/analysis/29_packed_player.py [container.dlx] [--kbps R]
THE QUESTION, and why it is being asked again. FINDINGS 44.7 removed the codec
and asked what a player that just puts literal frames on screen would cost. It
answered "it fits the clocks and dies on the medium": 1,152 KB/s and 1.61 GB,
because 256-colour GVRAM's default write path throws away the high byte of every
word and a picture byte therefore costs two disc bytes. 46.5/47.1 then found
the off switch -- CRTC R20 bit 11 -- and 47.2 built the layout and rendered it
pixel-exactly on both emulators at 1.0 B/pixel. 47.5 re-derived the budget on
that and withdrew 44.7's conclusion CONDITIONALLY.
Everything in 47.5 is arithmetic over a cost model that has since been REPLACED.
When it was written the transport was an unmeasured `c`; sessions 25b-28 put the
transport on the 68000 and measured it (58.2: PIO is 87.28 clk/B), put it on the
DMAC and bounded it (59.2: this machine can run dual-address only, and a
dual-address byte has a 9 clk/B FLOOR), and re-derived what a frame can afford
(59.7/60.7: 6.69 clk/B on the gate container). 47.6.1 also filed the CPU paint
cost as an ASSUMPTION -- "the `movem` shape of the packed writes is an
assumption", no clock in 47.5 measured.
So this tool re-asks 44.7's question with:
* the paint MEASURED, not assumed -- tools/bench/blit.s V8 is V1 with 128
words a row instead of 256, and tools/bench/blit.lua times it next to V1,
V2 and V3 in the same run, so the packed number is quoted against a
variant whose value (53.6%) is a session-9 result that has not moved;
* the transport swept over the SAME `W` ladder 15_bus_occupancy.py uses,
every rung of it sourced or measured (buscost.py);
* the audio DMA charged, at the rate the IPL ROM's own channel-3 setup
implies (21_iplrom_dmac.py) -- 60.x's rule that a budget debits I/O;
* and the wire and the volume stated for each, because 44.7's answer was
never about clocks.
WHAT IT DOES NOT DO. It does not settle 47.4 -- whether buffer mode BLANKS the
graphics layer, which MAME asserts and px68k is silent about (48.1), and which
needs a real board. It PRICES both branches instead, and the blanking section
is where the measured paint earns its keep: the black interval is the paint, and
until now the paint was a range read off an unpacked measurement ("~27% to ~54%",
48.3) rather than a number.
"""
import sys, os, re, argparse, csv
sys.path.insert(0, "tools/encoder")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
from dlx import DLX
import buscost as B
CPUHZ = 10e6 # stock X68000, MAME 0.277 x68k.cpp:1133
GAME_S = 22.8 * 60 # the full-disc survey's runtime (ROADMAP C1)
ap = argparse.ArgumentParser()
ap.add_argument("container", nargs="?", default="tmp/rc_fr_singe_scsi_span.dlx",
help="the CODEC baseline this is compared against")
ap.add_argument("--csv", default="tmp/c68k_frames.csv",
help="per-frame C68K measurement of that container")
ap.add_argument("--blit-log", default="tmp/blit_v8.log",
help="tools/bench/blit.lua's log -- where the MEASURED paint "
"costs are read from. No defaults are compiled in.")
ap.add_argument("--kbps", type=float, default=None,
help="a delivery rate to score the wire against. OPTIONAL and "
"there is no default (FINDINGS 50).")
a = ap.parse_args()
# --- the measured paint, read out of the run's own log ---------------------
# NOT transcribed into this file. A constant copied out of a log is a constant
# that stops tracking the log, and this project has been caught by a stale
# number twice (60.8). If the log is not there the tool refuses rather than
# substituting a plausible one.
if not os.path.exists(a.blit_log):
sys.exit(f"missing {a.blit_log} -- run tools/bench/blit.lua first:\n"
f" cd tmp && mame x68000 -bios ipl10 -ramsize 2M -video soft "
f"-window -sound none -nothrottle -plugins \\\n"
f" -autoboot_script ../tools/bench/blit.lua -seconds_to_run 60")
blit = {}
for line in open(a.blit_log, errors="replace"):
m = re.search(r"V(\d+)\s+(\d+) cyc", line)
if m:
blit[int(m.group(1))] = int(m.group(2))
for v in (1, 2, 3, 4, 8, 9, 10):
if v not in blit:
sys.exit(f"{a.blit_log} has no V{v} result -- the summary is incomplete, "
f"so the run did not finish and nothing here can be quoted.")
# --- the codec baseline: the container, and its MEASURED decode ------------
d = DLX(a.container)
FPS = d.fps
FRAME_CLK = CPUHZ / FPS
meas = {}
if os.path.exists(a.csv):
for r in csv.DictReader(open(a.csv)):
meas[int(r["frame"])] = int(r["cycles"])
if not meas:
sys.exit(f"missing {a.csv} -- the codec row's decode term is MEASURED and "
f"there is no derived stand-in for it.")
NF = max(meas) + 1
codec_decode = np.mean([meas[f] for f in range(NF)])
codec_bpf = sum(d.record_lengths()[:NF]) / NF # the PADDED record (60.7)
# --- geometry, which is where the decoder-free rows come from -------------
W_PX, H_PX = d.W, d.H
NPX = W_PX * H_PX
UNPACKED_BPF = NPX * 2 # one pixel per word, high byte discarded
PACKED_BPF = NPX * 1 # R20 bit 11 + page scroll (47.2, measured)
aud_bpf = B.ADPCM_BYTES_PER_S / FPS
AUD_CLK = aud_bpf * B.ADPCM_CLK_BYTE_BEST # best case, so every row is
# the optimistic end
# A device->GVRAM channel cannot walk a 1024-byte line stride inside one
# transfer: it writes a contiguous run. 192 rows therefore need 192 array-chain
# entries -- and SESSION 29 RAN THAT, off the disc, through src/player/dma.i's
# DM_BARV/DM_BTCV: eight rows at the 1024 B stride landed from ONE channel start
# with the CPU halted throughout (tools/bench/dma_run.sh, `[chain]`). So the
# MECHANISM is demonstrated and the CPU does not restart the channel per row.
# The COST is still datasheet arithmetic -- 36 clocks an entry, Fig 4-25 sheet 1,
# buscost.DMA_CHAIN_CLK -- because MAME's DMAC runs on wall-clock attotimes and
# cannot be asked what anything costs (42.5).
CHAIN_CLK = H_PX * B.DMA_CHAIN_CLK
print(f"""{a.container}: {NF} frames of {W_PX}x{H_PX} at {FPS:g} fps
frame slot on a 10 MHz 68000: {FRAME_CLK:,.0f} clocks
paint costs MEASURED by tools/bench/blit.lua, read from {a.blit_log}:
V1 unpacked movem blit {blit[1]:>9,} clk {100*blit[1]/FRAME_CLK:5.1f}% (96 KB read + 96 KB write)
V2 byte-source expansion {blit[2]:>9,} clk {100*blit[2]/FRAME_CLK:5.1f}% (48 KB read + 96 KB write)
V3 write-only floor {blit[3]:>9,} clk {100*blit[3]/FRAME_CLK:5.1f}% (no source read at all)
V8 PACKED movem blit {blit[8]:>9,} clk {100*blit[8]/FRAME_CLK:5.1f}% (48 KB read + 48 KB write)
V8 is {100*blit[8]/blit[1]:.1f}% of V1 and {100*blit[8]/blit[3]:.1f}% of V3 -- so PACKED PAINT COSTS WHAT THE
UNPACKED PATH PAYS TO WRITE ALONE, with its source read thrown in free.
It is not exactly half of V1 because the 192-row loop does not halve with
the words: per word V1 is {blit[1]/(NPX):.3f} clk and V8 is {blit[8]/(NPX//2):.3f}.""")
# --- the architectures ----------------------------------------------------
# Each is (label, bytes on the wire per frame, CPU clocks per frame that are
# NOT the transport, and whether the transport lands in GVRAM or in RAM).
ARCH = [
("CODEC, CPU-decoded (the shipping design)", codec_bpf, codec_decode, "ring"),
("free / DMAC device->GVRAM / unpacked", UNPACKED_BPF, CHAIN_CLK, "gvram"),
("free / DMAC device->GVRAM / PACKED", PACKED_BPF, CHAIN_CLK, "gvram"),
("free / CPU-painted / unpacked, 2 B/px wire", UNPACKED_BPF, blit[1], "ring"),
("free / CPU-painted / unpacked, 1 B/px wire", PACKED_BPF, blit[2], "ring"),
("free / CPU-painted / PACKED", PACKED_BPF, blit[8], "ring"),
]
LADDER = [
(5.0, "single address, held -- needs a request line (B3)"),
(9.0, "dual address, held -- the FLOOR (59.2/59.7)"),
(12.0, "single address, arbitrated"),
(16.0, "what the ROM programs for SASI, best"),
(19.0, "what the ROM programs for SASI, worst"),
(87.28, "PIO -- MEASURED, 58.2"),
]
print("\n" + "=" * 78)
print("WHAT EACH ARCHITECTURE COSTS A FRAME, over the transport ladder\n")
print(f" audio DMA is charged in every row at {AUD_CLK:,.0f} clk "
f"({100*AUD_CLK/FRAME_CLK:.2f}%), best case.\n")
hdr = f" {'architecture':<44}{'B/frame':>9}" + "".join(f"{f'W={w:g}':>9}" for w, _ in LADDER)
print(hdr)
print(" " + "-" * (len(hdr) - 2))
for label, bpf, cpu, dest in ARCH:
cells = []
for w, _ in LADDER:
tot = bpf * w + cpu + AUD_CLK
pct = 100 * tot / FRAME_CLK
cells.append(f"{pct:>8.1f}%" if pct < 1000 else f"{pct:>8.0f}%")
print(f" {label:<44}{bpf:>9,.0f}" + "".join(cells))
print(f"""
100% is the frame deadline. Every cell is CPU work plus transport plus
best-case audio; none of them overlap, because the 68000 has no cache and a
two-word prefetch queue that empties at once (buscost.DMA_OVERLAPS = False).
THE TWO ROWS THAT MATTER ARE THE FLOOR COLUMN, W=9, because 59.2 found that
the only configurations this machine can be shown to run are dual-address,
and a dual-address byte is a 4-clock read of the device plus a 5-clock write
to memory. Every column left of it is a hardware fact nobody here has.""")
# --- the wire, which is what 44.7 actually died on ------------------------
print("\n" + "=" * 78)
print("THE WIRE AND THE MEDIUM -- 44.7's real objection\n")
print(f" {'architecture':<44}{'B/frame':>9}{'KB/s':>9}{'GB for 22.8 min':>18}")
print(" " + "-" * 78)
seen = set()
for label, bpf, cpu, dest in ARCH:
kbs = bpf * FPS / 1024
gb = bpf * FPS * GAME_S / 1e9
print(f" {label:<44}{bpf:>9,.0f}{kbs:>9.1f}{gb:>18.2f}")
print(f"""
The codec row is the gate container, which is deliberately the heaviest thing
the encoder emits (59.7). The default `need` recipe is 267.9 KB/s and E7's
byte target at the 9 clk/B floor is 327 KB/s (60.7).
SO THE PACKED DECODER-FREE PLAYER ASKS FOR {PACKED_BPF*FPS/1024:.0f} KB/s -- {PACKED_BPF*FPS/1024/327:.2f}x E7's target and
{PACKED_BPF*FPS/1024/(codec_bpf*FPS/1024):.2f}x the gate container -- and it asks for it AT A FIXED RATE. A codec's
bitrate is a lever; a literal frame's is geometry, and there is no scene in
the picture that costs less than another.""")
if a.kbps:
R = a.kbps * 1024
print(f"\n against a supplied {a.kbps:g} KB/s:")
for label, bpf, cpu, dest in ARCH:
need = bpf * FPS
print(f" {label:<44}{'FITS' if need <= R else 'SHORT BY '}"
f"{'' if need <= R else f'{(need-R)/1024:.0f} KB/s'}"
f" ({need/1024:.0f} KB/s wanted)")
# --- 47.4's two branches, priced -----------------------------------------
print("\n" + "=" * 78)
print("IF BUFFER MODE BLANKS THE LAYER (47.4 / 48, MAME's reading)\n")
print(""" R20 bit 11 only has to be SET across the GVRAM writes, so the black
interval is the paint and not the frame -- and which paint depends on where
the transport lands. That asymmetry has not been stated before:\n""")
print(f" {'architecture':<44}{'black interval':>16} {'set for':<14}")
print(" " + "-" * 78)
for label, bpf, cpu, dest in ARCH[1:]:
if dest == "gvram":
# the channel writes GVRAM, so the bit is set for the whole transfer
# DMA rungs only: a PIO transport is not a channel writing GVRAM, so
# 87.28 has no meaning in a device->GVRAM row.
rows = [bpf * w + CHAIN_CLK for w, _ in LADDER if w < 20]
span = f"{100*min(rows)/FRAME_CLK:.0f}%..{100*max(rows)/FRAME_CLK:.0f}%"
note = "the whole DMA"
else:
span = f"{100*cpu/FRAME_CLK:.1f}%"
note = "the blit only"
print(f" {label:<44}{span:>16} {note:<14}")
print(f"""
THE CPU-PAINTED PACKED PATH HAS THE SMALLEST BLACK WINDOW OF ANY OF THEM --
{100*blit[8]/FRAME_CLK:.1f}% -- because its transport lands in RAM, where bit 11 is irrelevant,
and only the {blit[8]:,}-clock blit needs the bit set. The DMAC-direct path,
which is cheaper in clocks at every rung of the ladder, is the one that must
hold the bit across its whole transfer. Under MAME's reading the cheap
architecture is the dark one.
Both are a strobe at the frame rate over the whole picture, and the packed
layout has no page to flip to: both 256-colour pages carry picture, which is
the entire point of it (48.3). {100*blit[8]/FRAME_CLK:.1f}% black at 12 Hz is not a tear.
IF PX68K IS RIGHT AND IT DOES NOT BLANK, every number above stands as
written. Neither emulator is authority and 48.1 is why the prior leans
MAME's way: MAME asserts the semantic twice and deliberately, px68k's
display path never reads the bit at all. That is an assertion against a
silence, not a tie, and it is settled by a board and the two-line probe in
tools/bench/probe_bit11_blank.lua.""")
# --- 47.6.4: does the CODEC survive the packed layout? --------------------
# Open since session 16 and never touched: "under the packed layout a word spans
# two columns 128 apart. Whether the existing codec survives that is untouched."
# There are exactly two ways it could, and blit.s V9 and V10 are them.
sk_blocks = sk_tot = pair_sk = pair_tot = 0
for f in range(NF):
m = d.modes(f).reshape(d.nby, d.nbx)
L, R = m[:, :d.nbx // 2], m[:, d.nbx // 2:]
sk_blocks += int((m == 0).sum()); sk_tot += m.size
pair_sk += int(((L == 0) & (R == 0)).sum()); pair_tot += L.size
paint_now = 1 - sk_blocks / sk_tot
paint_pair = 1 - pair_sk / pair_tot
print("\n" + "=" * 78)
print("CAN THE CODEC BE PACKED TOO? -- 47.6.4, open since session 16\n")
print(f""" A 4x4 block owns four bytes at STRIDE 2 under the packed layout, because
the high bytes of its four words belong to the block 128 columns away. So a
block decoder has two options and neither is free:
{'V4 block order, UNPACKED (the shipping shape)':<52}{blit[4]:>9,} clk {100*blit[4]/FRAME_CLK:5.1f}%
{'V9 block order, PACKED, 16 move.b at stride 2':<52}{blit[9]:>9,} clk {100*blit[9]/FRAME_CLK:5.1f}%
{'V10 block order, PACKED, blocks PAIRED (movem back)':<52}{blit[10]:>9,} clk {100*blit[10]/FRAME_CLK:5.1f}%
V9 IS {100*blit[9]/blit[4]-100:.0f}% DEARER THAN V4, not cheaper. Packing buys a block decoder
nothing on the wire either -- a codeword is already one byte a pixel -- so
that route buys NOTHING and costs {blit[9]-blit[4]:,} clocks a frame to buy it.
V10 halves the paint, and pays for it in the mode map. A pair skips only if
BOTH its blocks skip, and on this container:
SKIP blocks now {100*sk_blocks/sk_tot:5.1f}% painted now {100*paint_now:5.1f}%
SKIP block PAIRS {100*pair_sk/pair_tot:5.1f}% painted paired {100*paint_pair:5.1f}%
So pairing paints {paint_pair/paint_now:.2f}x as many blocks for {blit[10]/blit[4]:.2f}x the paint per block --
{100*(paint_pair/paint_now)*(blit[10]/blit[4])-100:+.0f}% on the clock, and about {100*(paint_pair/paint_now-1):+.0f}% on the BYTES, because a coded
block is bytes in the container whether its half of the pair changed or not.
E7 needs the bytes DOWN {100*(codec_bpf*FPS/1024)/327-100:.0f}%.
SO PACKING BELONGS TO THE LITERAL PLAYER AND ONLY TO IT. 47.6.4 is closed:
the packed layout is not an upgrade the existing codec can take, it is the
thing you get INSTEAD of the codec.""")
# --- the palette, which is where the literal player stops being a compromise --
# 46.3 measured these while pricing the TEXT PLANE and the 256-colour rows were
# only there for scale. They answer a question nobody put to them: a literal
# player has no codebooks, so it is not tied to a scene-wide palette the way the
# codec is (vq.scene_palette exists BECAUSE codewords are indices into it), and
# per-frame palettes become legal. Re-run 18_text_plane_16col.py to reproduce.
PSNR_SHIPPED = 29.19 # docs/STATUS.md, --spans all, c=5, 496.7 KB/s
PSNR_SCENE_256 = 31.33 # 18_text_plane_16col.py, tmp/fr_singe, 120 frames
PSNR_FRAME_256 = 34.08 # the same window, per-frame palettes
PAL_BYTES = 512 # 256 entries x 1 word
pal_bpf = PACKED_BPF + PAL_BYTES
# The palette write, DERIVED from a MEASURED per-word constant: V8 moves a word
# into GVRAM for blit[8]/(NPX//2) clocks and the palette is 256 consecutive
# words at $E82000 in the same movem shape.
pal_clk = 256 * blit[8] / (NPX // 2)
print("\n" + "=" * 78)
print("AND THE PICTURE IS BETTER, WHICH NOBODY HAD ASKED\n")
print(f""" PSNR against the 24-bit source, 18_text_plane_16col.py over the same
120-frame window the whole tree is measured on:
{'shipping container (the codec, as it ships)':<48}{PSNR_SHIPPED:6.2f} dB
{'256 colours, SCENE palette -- the codec CEILING':<48}{PSNR_SCENE_256:6.2f} dB
{'256 colours, PER-FRAME palette':<48}{PSNR_FRAME_256:6.2f} dB
THE MIDDLE ROW IS A CEILING AND NOT A RIVAL. Every codeword the codec emits
is an index INTO the scene palette, so no amount of bitrate takes it past
{PSNR_SCENE_256:.2f} dB; it spends {codec_bpf*FPS/1024:.0f} KB/s to get within {PSNR_SCENE_256-PSNR_SHIPPED:.2f} dB of it.
A LITERAL FRAME HAS NO CODEBOOKS, so the scene palette is not forced on it,
and the bottom row is what it simply IS -- {PSNR_FRAME_256-PSNR_SHIPPED:+.2f} dB on the shipping
container and {PSNR_FRAME_256-PSNR_SCENE_256:+.2f} dB past the ceiling the codec cannot cross.
WHAT THE PER-FRAME PALETTE COSTS:
on the wire {PAL_BYTES} B a frame -> {pal_bpf:,} B, {pal_bpf*FPS/1024:.1f} KB/s (+{100*PAL_BYTES/PACKED_BPF:.1f}%)
in clocks ~{pal_clk:,.0f} ({100*pal_clk/FRAME_CLK:.2f}% of a frame) if the CPU writes it, DERIVED
from V8's measured {blit[8]/(NPX//2):.3f} clk/word in the same movem shape
in colours 254, not 256: the packed layout spends index 0 on the
transparency key and puts black at 255 (47.2,
prep_frame.py --pack-transparent), against --reserve-black's
one entry. The tree has already measured a reserved entry at
0.04 dB (60.3), so this is noise against {PSNR_FRAME_256-PSNR_SHIPPED:+.2f}.
NOT SETTLED, and it is the obvious next probe: whether a DMA CHANNEL can write
the palette registers at $E82000, which would make the palette a 193rd chain
entry and cost the CPU nothing at all. Untested; 61.2 only ever pointed a
channel at GVRAM.
AND THE PSNR FIGURES ARE PIL's MEDIANCUT, not this project's own palette
builder (vq.scene_palette / H.build). The DIRECTION is measured and the
magnitude is about right; if the packed player gets built, re-derive the
per-frame number against the builder that will actually ship it.""")
# --- the answer ----------------------------------------------------------
w9 = 9.0
free_packed_dma = PACKED_BPF * w9 + CHAIN_CLK + AUD_CLK
free_packed_cpu = PACKED_BPF * w9 + blit[8] + AUD_CLK
codec_9 = codec_bpf * w9 + codec_decode + AUD_CLK
print("\n" + "=" * 78)
print(f"""THE ANSWER, AT THE ONE RUNG THIS MACHINE CAN BE SHOWN TO RUN (W=9)
CODEC, gate container {100*codec_9/FRAME_CLK:6.1f}% of the frame -- DOES NOT FIT
free / DMAC->GVRAM / PACKED {100*free_packed_dma/FRAME_CLK:6.1f}% -- FITS, with {100-100*free_packed_dma/FRAME_CLK:.0f}% to spare
free / CPU-painted / PACKED {100*free_packed_cpu/FRAME_CLK:6.1f}% -- FITS, with {100-100*free_packed_cpu/FRAME_CLK:.0f}% to spare
THE DECODER-FREE PACKED PLAYER FITS THE CLOCK BUDGET THAT THE CODEC MISSES.
That is not a small correction to 47.5, it is the reverse of the reason the
codec exists. 44.7 said it in advance and on a different cost model: "the
codec is not there to save CPU -- it is there to save the wire." The
measured model agrees, and now says the CPU side is not merely affordable
but strictly cheaper WITHOUT the codec: at the floor, decoding {codec_bpf:,.0f} bytes
costs {100*(codec_bpf*w9+codec_decode)/FRAME_CLK:.0f}% of a frame and NOT decoding {PACKED_BPF:,} costs {100*(PACKED_BPF*w9+blit[8])/FRAME_CLK:.0f}%.
SO THE QUESTION IS ENTIRELY A MEDIUM QUESTION, and it has two halves:
1. {PACKED_BPF*FPS/1024:.0f} KB/s SUSTAINED, with no lever to pull. ROADMAP B1 is
unmeasured; the 0.7-1.7 MB/s usually quoted for BlueSCSI on an X68000
is folklore with no published benchmark behind it. {PACKED_BPF*FPS/1024:.0f} KB/s sits
inside that range, which is exactly why the range has to be measured
rather than cited. A codec at 327 KB/s survives a slower answer; a
literal frame does not degrade, it drops.
2. {PACKED_BPF*FPS*GAME_S/1e9:.2f} GB for the whole game, against the codec's {codec_bpf*FPS*GAME_S/1e9:.2f} GB at the gate
recipe and ~{327*1024*GAME_S/1e9:.2f} GB at E7's target. That is a packaging fact (C3),
not a performance one.
AND 47.4 STILL SITS OVER ALL OF IT. Everything above assumes the layer is
visible while it is written. If it is not, the packed player is a {100*blit[8]/FRAME_CLK:.0f}% duty
strobe at best and there is no version of it that is merely expensive.""")
tools/bench/blit.lua
+23 -1
View File
@@ -29,7 +29,7 @@ end
local MODE = load_mode()
local FLAG, VAR, ITER = 0x18000, 0x18004, 0x18008
local SRCW, SRCB = 0x60000, 0x80000
local SRCW, SRCB, SRCP = 0x60000, 0x80000, 0x90000
local GVRAM, GPAL = 0xC00000, 0xE82000
local CPUHZ = 10000000 -- x68k.cpp:1133, 40_MHz_XTAL/4
local FRAME12 = CPUHZ / 12 -- 833333 cycles at 12 fps
@@ -40,6 +40,16 @@ local PLAN = {
{var=2, iter= 50, name="V2 naive byte-source expansion (move.b/move.w per pixel)"},
{var=3, iter=200, name="V3 write-only floor (no source read at all)"},
{var=4, iter= 60, name="V4 same 96KB of writes, issued in 4x4 BLOCK order (decoder access pattern)"},
-- V8 is V1 with R20 bit 11's packing: 48KB read + 48KB write for the SAME
-- 49,152 pixels. It is the per-frame work of a decoder-free packed player
-- (FINDINGS 44.7 / 46.6 / 47.5), and 47.6.1 filed its `movem` shape as an
-- ASSUMPTION -- this is the measurement that assumption was standing in for.
{var=8, iter=200, name="V8 PACKED movem.l blit (48KB read + 48KB write, same 49,152 pixels as V1)"},
-- V9/V10 are the two ways a BLOCK decoder could survive the packed layout
-- (47.6.4, open since session 16): sixteen move.b at stride 2 per block, or
-- pair the blocks 128 columns apart in the encoder and get V4's movem back.
{var= 9, iter= 40, name="V9 PACKED block order, 16 move.b at stride 2 per 4x4 block"},
{var=10, iter=120, name="V10 PACKED block order, blocks PAIRED so a movem writes whole words"},
}
local code do
@@ -97,6 +107,18 @@ local function setup()
SP:write_u8 (SRCB + y*256 + x, px)
end
end
-- SRCP: the PACKED frame, interleaved the way tools/bench/show_frame256_packed.lua
-- lays it out -- word i of a row is (column i+128) << 8 | (column i), because
-- page 0 is the low byte at screen column i and page 1 the high byte at i+128.
-- Only V8 reads it, and only its SIZE (128 words a row) affects the timing;
-- the interleave is written correctly so the buffer is the real artefact and
-- not a same-sized stand-in.
for y = 0, H-1 do
local row = PIX0 + y*W
for i = 0, (W//2)-1 do
SP:write_u16(SRCP + y*(W//2)*2 + i*2, (B(row+i+W//2) << 8) | B(row+i))
end
end
for i = 1, #code do SP:write_u8(0x10000+i-1, string.byte(code,i)) end
P(string.format("loaded blit.bin=%d bytes, source frame %dx%d at yoff=%d", #code, W, H, YOFF))
end
tools/bench/blit.s
+146
View File
@@ -104,6 +104,9 @@ ITER = $18008 ; iteration count, written by Lua
SPTR = $1800C ; V5 span stream pointer, written by Lua
SRCW = $60000 ; word-expanded frame 192*512 = 96KB
SRCB = $80000 ; byte-per-pixel frame 192*256 = 48KB
SRCP = $90000 ; PACKED frame 192*256 = 48KB (V8): two picture
; bytes per word, already interleaved by the
; encoder, so the blit is a straight copy
DST0 = $C08000 ; GVRAM + 32*1024 (first picture row)
DSTE = $C38000 ; GVRAM + 224*1024 (one past last)
ROWS = 192 ; picture rows a V5 stream describes
@@ -130,6 +133,12 @@ start:
beq v6
cmp.l #7,d0
beq v7
cmp.l #8,d0
beq v8
cmp.l #9,d0
beq v9
cmp.l #10,d0
beq v10
bra v3
; ---------------------------------------------------------------- V1
@@ -372,5 +381,142 @@ v7fh:
bne v7
bra done
; ---------------------------------------------------------------- V8
; THE PACKED FULL-FRAME BLIT (FINDINGS 46.6/47.2). Identical in shape to V1 --
; a row-linear movem.l chain out of a RAM frame into GVRAM -- and different in
; exactly one thing: a row is 128 WORDS, not 256, because R20 bit 11 lets one
; word carry two picture bytes. 256 = 5*48 + 16, so five 12-register bursts
; and a 4-register tail, against V1's ten and one.
;
; TIMING ONLY, and it does not set bit 11. MAME's gvram_w carries no timing in
; either arm (blit.lua's header), so the bit cannot move a cycle here; what it
; moves is the PICTURE, and the picture is what tools/bench/show_frame256_packed.lua
; and tools/bench/gvpack already verify pixel-exactly. Setting it here would
; make this variant's snapshot right and its measurement no different, and
; would put a display-mode change inside a timing loop for no gain.
;
; The source is PRE-INTERLEAVED by the host, which is the honest half of the
; claim: the packing is an encoder-side transform (46.3's argument for the text
; plane, and the same one here), so the decoder-free player's per-frame work is
; this copy and nothing else. If the interleave had to happen at run time this
; variant would be V2, not V1.
v8: lea SRCP,a0
lea DST0,a1
lea DSTE,a6
v8row: movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,48(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,96(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,144(a1)
movem.l (a0)+,d0-d7/a2-a5
movem.l d0-d7/a2-a5,192(a1)
movem.l (a0)+,d0-d3
movem.l d0-d3,240(a1)
lea 1024(a1),a1
cmpa.l a6,a1
bne v8row
subq.l #1,ITER.l
bne v8
bra done
; ---------------------------------------------------------------- V9
; WHAT THE PACKED LAYOUT COSTS A BLOCK DECODER (FINDINGS 47.6.4, open).
;
; V4 is the access pattern of a decoder that writes 4x4 codewords straight into
; GVRAM: 4 rows of 8 contiguous bytes at a 1024-byte stride, so each row is one
; `movem.l` of two registers. Under the packed layout that pattern is GONE.
; A block at columns x..x+3 owns the LOW bytes of four consecutive words -- four
; bytes at STRIDE 2 -- and the high bytes of those same words belong to the
; block 128 columns away. There is no burst that writes every other byte, so
; the block is sixteen `move.b`s.
;
; V9 does the pair together, low block then high block off one base, so it
; writes every byte it touches and covers the same 49,152 pixels V1/V4/V8 do.
; It is the HONEST version of "keep the codec and pack the screen": the mode
; map is unchanged, SKIP still works per block, and the writes go byte at a
; time. V10 below is the other option, and the comparison is the point.
v9: lea SRCB,a0
lea DST0,a3
lea DSTE,a4
v9brow: move.l a3,a1
lea 256(a3),a5 ; 32 block PAIRS * 8 bytes
v9blk:
move.b (a0)+,(a1)
move.b (a0)+,2(a1)
move.b (a0)+,4(a1)
move.b (a0)+,6(a1)
move.b (a0)+,1024(a1)
move.b (a0)+,1026(a1)
move.b (a0)+,1028(a1)
move.b (a0)+,1030(a1)
move.b (a0)+,2048(a1)
move.b (a0)+,2050(a1)
move.b (a0)+,2052(a1)
move.b (a0)+,2054(a1)
move.b (a0)+,3072(a1)
move.b (a0)+,3074(a1)
move.b (a0)+,3076(a1)
move.b (a0)+,3078(a1)
move.b (a0)+,1(a1)
move.b (a0)+,3(a1)
move.b (a0)+,5(a1)
move.b (a0)+,7(a1)
move.b (a0)+,1025(a1)
move.b (a0)+,1027(a1)
move.b (a0)+,1029(a1)
move.b (a0)+,1031(a1)
move.b (a0)+,2049(a1)
move.b (a0)+,2051(a1)
move.b (a0)+,2053(a1)
move.b (a0)+,2055(a1)
move.b (a0)+,3073(a1)
move.b (a0)+,3075(a1)
move.b (a0)+,3077(a1)
move.b (a0)+,3079(a1)
addq.l #8,a1
cmpa.l a5,a1
bne v9blk
lea 4096(a3),a3
cmpa.l a4,a3
bne v9brow
subq.l #1,ITER.l
bne v9
bra done
; ---------------------------------------------------------------- V10
; THE OTHER OPTION: PAIR THE BLOCKS IN THE ENCODER. If the codec codes the
; block at x and the block at x+128 as ONE unit, the destination is whole words
; again and V4's `movem.l` shape comes straight back -- the same instructions,
; the same 32 bytes of source per unit, and TWICE the pixels, because a word now
; carries two of them. So V10 is V4's inner loop run half as many times.
;
; WHAT IT COSTS IS NOT IN THIS MEASUREMENT. A pair skips only if BOTH of its
; blocks skip, and the two are 128 columns apart with nothing in the picture
; relating them. That is a CONTAINER question -- what fraction of the mode map
; survives pairing -- and 08_mode_map.py has the data to answer it. V10 prices
; the paint; it does not price the SKIPs the pairing loses.
v10: lea SRCP,a0
lea DST0,a3
lea DSTE,a4
v10brow: move.l a3,a1
lea 256(a3),a5 ; 32 block PAIRS * 8 bytes
v10blk: movem.l (a0)+,d0-d7 ; 32 bytes = one PAIR of 4x4 blocks
movem.l d0-d1,(a1)
movem.l d2-d3,1024(a1)
movem.l d4-d5,2048(a1)
movem.l d6-d7,3072(a1)
addq.l #8,a1
cmpa.l a5,a1
bne.s v10blk
lea 4096(a3),a3
cmpa.l a4,a3
bne v10brow
subq.l #1,ITER.l
bne v10
bra done
done: move.l #$FF,FLAG.l ; timer stops here
halt: bra.s halt
tools/bench/check.sh
+91 -14
View File
@@ -353,7 +353,7 @@ echo "--- session 23: the 68000 fills its own ring (FINDINGS 55) ---"
# rather than faulting (49.2);
# * the host AUDITS every placement against its own index and its own list of
# live records, and refuses the run on the first disagreement;
# * the wrap policy still produces the SAME 18 wraps and 14.7 KB mean hole the
# * the wrap policy still produces the SAME 18 wraps the
# host producer produced in FINDINGS 49.4 -- a third independent
# implementation of `aligned` landing on the same tiling;
# * zero underruns at a two-deep request queue, which is the finding: a
@@ -445,13 +445,17 @@ echo "--- session 26: the ring is filled off a real SCSI volume (FINDINGS 58) --
# * pixel-exact, which is the only test that can see a wrong record: the
# window in scsi.i decides which of a sector's bytes reach the ring, and a
# window off by one byte desyncs the bitstream rather than faulting (49.2);
# * the SAME 18 wraps and 14.7 KB mean hole -- ring.i's placement policy must
# not be able to tell which transport answered it, and this is the assertion
# that says it could not;
# * every record accounted for: 120 READ(10)s, 4,488,588 B into the ring, and
# 4,548,608 B off the disc. The two byte counts differ by 1.34% because a
# record is not a sector, and that gap is a delivery cost (58.3) -- gating
# both numbers means neither can drift silently into the other;
# * the SAME 18 wraps -- ring.i's placement policy must not be able to tell
# which transport answered it, and this is the assertion that says it could
# not. The WRAP COUNT is gated and the mean hole is only reported: DLX5's
# records are up to 511 B longer than DLX4's, so the hole moved (14.7 KB ->
# 13.5 KB) while the tiling did not. Gating a number that the container's
# record lengths move would gate the container, not the policy;
# * every record accounted for: 120 READ(10)s, and the bytes into the ring
# EQUAL to the bytes off the disc -- both read out of the container rather
# than written here. Under DLX4 they differed by 1.34% because a record was
# not a sector (58.3); DLX5 aligns records to sectors and the covering-sector
# read disappears, so the gate is now their IDENTITY;
# * a real mid-stream SEEK with the real transport, in the second pass. This
# is the one path that could not exist before: ring_seek waits for the
# channel to go quiet, and with the transport INSIDE the machine the only
@@ -470,15 +474,34 @@ if command -v chdman > /dev/null; then
| sed "s/^ *//;s/^/ /"
grep -aq "TRANSPORT FAILED" tmp/p4b_check.log && {
echo "FAIL: a record's READ(10) reported an error."; exit 1; }
grep -aq "REAL TRANSPORT: 120 READ(10)s by the 68000, 4488588 B into the ring" \
# THE BYTE COUNTS COME OUT OF THE CONTAINER, not out of this file. They were
# two hardcoded constants fitted to the DLX4 gate container, and session 28's
# re-encode went red on both of them for the right reason -- the container had
# changed and the expectation had not. A gate whose expected value is a
# literal tests the literal.
EXPECT_B=$(python3 -c "
import sys; sys.path.insert(0, 'tools/encoder')
from dlx import DLX
print(sum(DLX('$DLX').record_lengths()))")
grep -aq "REAL TRANSPORT: 120 READ(10)s by the 68000, $EXPECT_B B into the ring" \
tmp/p4b_check.log || {
echo "FAIL: the 68000 did not fetch all 120 records, or did not fetch"
echo " 4,488,588 B of them. A short record is a desync, not a shortfall."
echo " $EXPECT_B B of them. A short record is a desync, not a shortfall."
exit 1; }
grep -aq "SECTOR OVERHEAD: 4548608 B off the disc" tmp/p4b_check.log || {
echo "FAIL: the bytes the DISC moved are no longer 4,548,608. A record is"
echo " not a sector; this is the covering-sector read, and if it moved"
echo " then either the layout or scsi.i's window did. See FINDINGS 58.3."
# DLX5 MAKES THESE THE SAME NUMBER, and that identity IS the finding (59.4,
# and 58.3 option C): a sector-aligned container has no covering-sector read,
# so the disc moves exactly the records and nothing else. Under DLX4 they
# differed by 1.34% and both were gated so neither could drift into the other;
# under DLX5 the gate is that they are EQUAL. If a windowed read ever came
# back -- a container that was not aligned, or a layout that lost the
# alignment -- the disc figure would exceed the ring figure and this goes red.
grep -aq "SECTOR OVERHEAD: $EXPECT_B B off the disc for $EXPECT_B B of record = 0.00%" \
tmp/p4b_check.log || {
echo "FAIL: the disc no longer moves EXACTLY the records. On a sector-"
echo " aligned container (DLX5) there is no covering-sector read at"
echo " all, so these two counts must be the same $EXPECT_B B. If they"
echo " differ, either the container lost its alignment or scsi.i is"
echo " windowing again -- and a DMA channel cannot window (59.4)."
exit 1; }
grep -aq "ring: 18 wraps" tmp/p4b_check.log || {
echo "FAIL: the placement policy tiled this container differently with a"
@@ -574,4 +597,58 @@ else
echo " (git clone --depth 1 https://github.com/icculus/DirkSimple)"
fi
echo "--- session 29: the packed paint, and what it does to the codec (FINDINGS 61) ---"
# tools/bench/blit.s gained V8/V9/V10 -- the packed full-frame blit, and the two
# ways a 4x4 BLOCK decoder could survive the packed layout. 47.6.1 had filed the
# packed paint's `movem` shape as an ASSUMPTION since session 16; this measures
# it, in the same run as V1/V3/V4 so it is quoted against numbers that have not
# moved since session 9.
#
# WHAT IS GATED IS STRUCTURAL, not numeric, for the reason the load stage gives:
# MAME samples these on a 1/56.69 s clock and no cost model in the tree depends
# on their exact value. What DOES depend on them is the ORDER, and the order is
# the whole of FINDINGS 61:
# V8 < V1 packing halves the full-frame literal paint
# V9 > V4 packing makes a BLOCK decoder DEARER, not cheaper
# V10 < V4 unless the blocks are paired, which costs SKIPs instead
# A tree where any of those flipped has a different answer to 44.7 and should
# say so out loud rather than let 29_packed_player.py narrate the old one.
python3 tools/bench/prep_frame.py tmp/fr_00020 tmp/frame256.bin 0 --reserve-black
rm -f tmp/blit_v8.log
( cd tmp && SDL_VIDEODRIVER=dummy stdbuf -oL timeout -k 5 300 mame x68000 \
-bios ipl10 -ramsize 2M -video soft -window -sound none -nothrottle -plugins \
-autoboot_script ../tools/bench/blit.lua -seconds_to_run 120 \
> blit_v8.log 2>&1 )
grep -aq "summary (instruction cycles only" tmp/blit_v8.log || {
echo "FAIL: the blit timing run produced no summary -- it did not finish."
tail -8 tmp/blit_v8.log; exit 1; }
python3 - <<'EOF' || exit 1
import re, sys
v = {}
for line in open("tmp/blit_v8.log", errors="replace"):
m = re.search(r"V(\d+)\s+(\d+) cyc", line)
if m: v[int(m.group(1))] = int(m.group(2))
need = (1, 2, 3, 4, 8, 9, 10)
missing = [n for n in need if n not in v]
if missing: sys.exit(f"FAIL: blit.lua reported no V{missing} -- run incomplete.")
for a, op, b, why in ((8, "<", 1, "packing did not halve the literal paint"),
(9, ">", 4, "packed BLOCK order came out CHEAPER than "
"unpacked -- 61.3's conclusion is inverted"),
(10, "<", 4, "pairing the blocks did not buy back the "
"movem shape")):
ok = v[a] < v[b] if op == "<" else v[a] > v[b]
if not ok:
sys.exit(f"FAIL: V{a} {v[a]:,} is not {op} V{b} {v[b]:,} -- {why}.")
print(f" V1 {v[1]:,} / V8 PACKED {v[8]:,} = {100*v[8]/v[1]:.0f}% -- "
f"and V3, the unpacked WRITE-ONLY floor, is {v[3]:,}")
print(f" V4 {v[4]:,} / V9 packed-block {v[9]:,} = {100*v[9]/v[4]:.0f}% -- "
f"packing costs a BLOCK decoder {100*v[9]/v[4]-100:.0f}%")
print(f" V10 paired blocks {v[10]:,} = {100*v[10]/v[4]:.0f}% of V4, and pairing "
f"is paid for in SKIPs")
EOF
python3 tools/analysis/29_packed_player.py "$DLX" > tmp/packed_player.log 2>&1 \
|| { tail -20 tmp/packed_player.log; exit 1; }
grep -aE "SKIP block PAIRS|free / DMAC->GVRAM / PACKED|^ CODEC, gate" \
tmp/packed_player.log
echo "ALL GREEN"
tools/bench/dma.lua
+87 -8
View File
@@ -25,17 +25,24 @@ local function P(s) print("[DMA] "..s) end
local function T() local t=M.time; return t.seconds + t.attoseconds/1e18 end
local DGFLAG, DGREC, DGREC_SZ = 0x18600, 0x18610, 32
local DGWIN, DGWERR = 0x18680, 0x18684
local DGWIN, DGWERR, DGR20, DGR20N, DGR20C = 0x18700, 0x18704, 0x18708, 0x1870C, 0x18710
local CHROW, CHN, CHBASE = 256, 8, 0xC10000
local R20OF -- filled in after the mailbox addresses are known
local GV = 0xC00000
local DGLBA, DGBLK = 1000, 4
local DST = {0x20000, 0x24000, 0x28000}
local DST = {0x20000, 0x24000, 0x28000, 0xC08000, 0xC0C000, 0xC10000}
local NAME = {"PIO (the path FINDINGS 58 measured)",
"DMA, BUS HELD (DCR $00 burst, OCR $81 max rate)",
"DMA, STEALING (DCR $80 cycle steal, OCR $80 limited)"}
local SHORT = {"pio", "held", "steal"}
"DMA, STEALING (DCR $80 cycle steal, OCR $80 limited)",
"DMA -> GVRAM (bus held, R20 bit 11 = BUFFER MODE) [47.6.2]",
"DMA -> GVRAM (the SAME, bit 11 CLEAR -- NEGATIVE CONTROL)",
"DMA -> GVRAM (ARRAY CHAINED, 8 rows at the 1024 B line stride)"}
local SHORT = {"pio", "held", "steal", "gvram", "masked", "chain"}
local ERRNAME = {[0]="OK", "SELECTION TIMEOUT -- no target answered",
"UNEXPECTED PHASE", "POLL TIMEOUT -- a phase never arrived",
"NON-ZERO SCSI STATUS",
"WINDOWED READ REFUSED -- a channel cannot drop bytes"}
R20OF = {[3]=DGR20, [4]=DGR20N, [5]=DGR20C}
local DISK = os.getenv("DLX_SCSI_IMG") or "dlxdisk.img"
local code do local f=io.open("dmagate.bin","rb"); code=f:read("a"); f:close() end
@@ -58,7 +65,8 @@ SUB = emu.add_machine_frame_notifier(function()
cpu.state["SP"].value = 0x8000
cpu.state["PC"].value = 0x10000
P(string.format("dmagate.bin=%d B loaded at $10000; reading LBA %d, %d B, "
.."three ways", #code, DGLBA, DGBLK*512))
.."three ways, then once more into GVRAM",
#code, DGLBA, DGBLK*512))
st = "wait"; return
end
if st == "wait" then
@@ -68,7 +76,7 @@ SUB = emu.add_machine_frame_notifier(function()
end
if not want then P("no "..DISK.." to check against"); P("done"); M:exit(); return end
local LEN = DGBLK*512
for i = 0, 2 do
for i = 0, 5 do
local b = DGREC + i*DGREC_SZ
local rc = SP:read_u32(b)
local e = SP:read_u32(b+4)
@@ -83,14 +91,85 @@ SUB = emu.add_machine_frame_notifier(function()
P(string.format(" FAILED: err=%d (%s)", e, ERRNAME[e] or "?"))
else
local bad, first = 0, nil
-- The GVRAM run is read back a WORD at a time and split by hand.
-- SP:read_u8 on $C00000 goes through gvram_r, which in buffer mode
-- returns the whole word; asking for one byte of it would hand back
-- whichever half MAME's address space happens to hand over, and the
-- question here is precisely WHICH HALF each disc byte landed in.
-- Even disc byte -> high half (page 1), odd -> low half (page 0),
-- because the 68000 is big-endian and an even address is the MS byte.
local pg1, pg0, bad_hi, bad_lo = 0, 0, 0, 0
for k = 1, LEN do
if SP:read_u8(DST[i+1]+k-1) ~= string.byte(want, k) then
bad = bad + 1; first = first or (k-1)
local got
if i == 5 then
-- The chained run's destination is not linear: byte k of the
-- transfer is byte k%256 of row k//256, and the rows are a full
-- 1024 B line stride apart. If the channel had ignored the array
-- and run contiguously, every byte past the first row would be
-- in the wrong place and this comparison would say so.
local off = (k-1) % CHROW
local a = CHBASE + ((k-1) // CHROW) * 1024 + (off & ~1)
local w = SP:read_u16(a)
if (off % 2) == 0 then got = (w >> 8) & 0xff; pg1 = pg1 + 1
else got = w & 0xff; pg0 = pg0 + 1 end
elseif i >= 3 then
local w = SP:read_u16(DST[i+1] + ((k-1) & ~1))
if ((k-1) % 2) == 0 then got = (w >> 8) & 0xff; pg1 = pg1 + 1
else got = w & 0xff; pg0 = pg0 + 1 end
else
got = SP:read_u8(DST[i+1]+k-1)
end
if got ~= string.byte(want, k) then
bad = bad + 1; first = first or (k-1)
if ((k-1) % 2) == 0 then bad_hi = bad_hi + 1
else bad_lo = bad_lo + 1 end
end
end
if i >= 3 then
P(string.format(" R20 during the run = $%04X (bit 11 %s); %d bytes "
.."read back as page 1 (high half) and %d as page 0",
SP:read_u32(R20OF[i]),
((SP:read_u32(R20OF[i]) & 0x0800) ~= 0)
and "SET" or "CLEAR",
pg1, pg0))
end
if bad == 0 then
P(string.format(" BYTES OK: %d B from LBA %d match %s byte for byte "
.."[%s]", LEN, DGLBA, DISK, SHORT[i+1]))
if i == 3 then
P(" A CHANNEL FILLS THE PACKED LAYOUT: every disc byte landed in "
.."its own half of a GVRAM word, with the CPU halted -- so a "
.."stream interleaved (right<<8)|left goes from disc to screen "
.."with no CPU in the loop (47.6.2, first half).")
end
if i == 5 then
P(string.format(" THE CHANNEL WALKED THE ARRAY ITSELF: %d rows of "
.."%d B landed at a %d B line stride from ONE start, CPU halted "
.."throughout. A frame is %d such entries; the CPU does not "
.."restart the channel per row.", CHN, CHROW, 1024, 192))
end
if i == 4 then
P(" CONTROL DID NOT FAIL: the masked write path delivered every "
.."byte too, so the run above is not evidence about R20 bit 11.")
end
elseif i == 4 then
-- THE CLAIM IS NOT "half the bytes differ". In masked 256-colour
-- mode gvram_w takes `data & 0x00ff` and ignores mem_mask, so a byte
-- written to an EVEN address is never stored and the high half keeps
-- whatever it held; some of those stale halves match the disc by
-- coincidence, and this record is full of pad, so a lot of them do.
-- The mechanism's signature is WHERE the damage is, not how much:
-- every ODD byte must survive and only EVEN ones may be lost.
P(string.format(" BYTES LOST [masked]: %d of %d differ (first at "
.."+%d) -- %d at EVEN offsets, %d at ODD.",
bad, LEN, first, bad_hi, bad_lo))
if bad_lo == 0 and bad_hi > 0 then
P(string.format(" EXACTLY THE MECHANISM: all %d survivors of the "
.."high half are stale GVRAM that happens to match "
.."(this record is mostly pad); not one of the %d "
.."ODD bytes was harmed. Bit 11 is what carried the "
.."even ones in the run above.", LEN//2 - bad_hi, LEN//2))
end
else
P(string.format(" BYTES WRONG [%s]: %d of %d differ, first at +%d",
SHORT[i+1], bad, LEN, first))
tools/bench/dma_run.sh
+32
View File
@@ -76,6 +76,38 @@ grep -aq "COC .*CER=\$00 MTC=0 .*(+2048) \[held\]" tmp/dma_run.log || \
fail "the held channel did not report a clean completion of every byte."
grep -aq "COC .*CER=\$00 MTC=0 .*(+2048) \[steal\]" tmp/dma_run.log || \
fail "the stealing channel did not report a clean completion of every byte."
# ---- the GVRAM run and its control (47.6.2). A channel that writes GVRAM in
# buffer mode is the decoder-free packed player's entire per-frame path, and a
# run with no control is 58.3's vacuous "UNDERRUNS: 0/120" again -- the IPL
# leaves R20 = $0B16, bit 11 ALREADY SET, so the first cut of this test could
# not have failed.
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[gvram\]" tmp/dma_run.log || \
fail "the channel did not fill GVRAM in buffer mode -- a device->GVRAM
transfer is the whole of the decoder-free packed player's frame."
grep -aq "R20 during the run = \$0916 (bit 11 SET)" tmp/dma_run.log || \
fail "the GVRAM run did not run in buffer mode with a KNOWN R20."
grep -aq "R20 during the run = \$0116 (bit 11 CLEAR)" tmp/dma_run.log || \
fail "the negative control did not run with bit 11 clear."
if grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[masked\]" tmp/dma_run.log
then
fail "the MASKED control delivered every byte, so the run above is not a
measurement of R20 bit 11 -- it is a measurement of nothing."
fi
grep -aq "EXACTLY THE MECHANISM" tmp/dma_run.log || \
fail "the masked control lost bytes at ODD offsets too, or lost none at all.
The claim is not a COUNT -- stale GVRAM matches the disc by coincidence
wherever the record is pad -- it is a PLACE: gvram_w's 256-colour arm
drops what the channel wrote to EVEN addresses and stores what it wrote
to odd ones. Damage anywhere else is a different mechanism."
grep -aq "BYTES OK: 2048 B from LBA 1000 .*\[chain\]" tmp/dma_run.log || \
fail "the array-chained run did not put the bytes at the row bases its array
named. A picture row is 256 B of a 1024 B line stride, so a frame is 192
destinations; if the channel cannot walk them the CPU has to restart it
per row and the decoder-free path costs a per-row front end."
grep -aq "THE CHANNEL WALKED THE ARRAY ITSELF" tmp/dma_run.log || \
fail "the chained run did not report walking its own array."
grep -aq "WINDOWED DMA READ REFUSED" tmp/dma_run.log || \
fail "a WINDOWED read through the channel was not refused. 117 of 120 records
start part way into a sector (58.3), and a channel cannot drop the bytes
tools/bench/mkvol.sh
+15 -1
View File
@@ -31,7 +31,21 @@ open("tmp/dlxdisk.img", "wb").write(d + b"\0" * (n * 512 - len(d)))
print(f" disc image: {len(d)} B of records -> {n} sectors")
PY
rm -f tmp/dlxdisk.chd
chdman createhd -i tmp/dlxdisk.img -o tmp/dlxdisk.chd -ss 512 > /dev/null 2>&1
# -c none IS LOAD-BEARING, and it was found by a gate rather than by taste.
# Session 28, on the DLX5 volume: with the default (lzma/zlib/huff/flac) MAME
# 0.277 served the CHD FILE'S OWN BYTES as sector data -- the destination
# buffer after READ(10) at LBA 0 was byte-for-byte the first 4,096 bytes of
# dlxdisk.chd, starting "MComprHD" -- while `chdman verify` reported both SHA1s
# correct. Uncompressed, the identical image reads byte-exact. The trigger is
# the image's CONTENT: the same 8,768-sector length that works for the DLX4
# volume fails for the DLX5 one, a conventional 16x63 geometry fails too, and
# `-c zlib` alone fails as well. The MAME-side cause is NOT diagnosed; what is
# measured is that compression decides it and uncompressed is sound.
# Costs 4.5 MB in tmp/ against 1.6 MB. DO NOT restore compression to save the
# disc space: the failure is SILENT at the transport layer -- every READ(10)
# reports success and returns the wrong bytes -- and only the byte comparison
# in tools/bench/scsi.lua catches it.
chdman createhd -i tmp/dlxdisk.img -o tmp/dlxdisk.chd -ss 512 -c none > /dev/null 2>&1
fi
mkdir -p tmp/p4roms/x68k_cz6bs1
tools/bench/prep_stream.py
+4 -1
View File
@@ -70,7 +70,10 @@ disk, index = bytearray(), []
for (o, n) in d.frames:
start = len(disk)
disk += n.to_bytes(4, "big") + d.raw[o:o + n]
while len(disk) % 4:
# The container's own alignment rule, not this script's copy of it: DLX5
# pads to 512 so a DMA channel can read whole sectors into the ring, DLX4
# to 4 so `move.l (a0)+` does not take an address error (28.3).
while len(disk) % d.rec_align:
disk += b"\0"
index.append((start, len(disk) - start))
open(a.out + "_disk.bin", "wb").write(bytes(disk))
tools/encoder/dlx.py
+44 -10
View File
@@ -29,6 +29,13 @@ import spans as SP
MODE_SKIP, MODE_V1, MODE_V4, MODE_RAW = 0, 1, 2, 3
# A SCSI target answers in 512-byte blocks and a record is not a sector: on the
# DLX4 gate container 117 of 120 records start part way into one. DLX5 makes
# the container agree with the medium instead of making the transport reconcile
# them (tools/analysis/26_sector_align.py prices all three ways).
SECTOR = 512
class DLX:
def __init__(self, path):
self.raw = open(path, "rb").read()
@@ -38,12 +45,21 @@ class DLX:
# (FINDINGS 28.3), so the padding is part of the format, not a loader
# convenience -- but DLX1 containers stay readable, because every
# measurement in FINDINGS 28-31 was taken on one.
if b[:4] not in (b"DLX1", b"DLX2", b"DLX3", b"DLX4"):
if b[:4] not in (b"DLX1", b"DLX2", b"DLX3", b"DLX4", b"DLX5"):
raise ValueError(f"{path}: not a DLX container")
self.version = int(b[3:4])
self.aligned = self.version >= 2
self.has_spans = self.version >= 3
self.has_index = self.version >= 4
# DLX5: every record starts on a 512-BYTE SECTOR boundary, and so does
# the frame stream itself. That is not a tidier version of DLX4's
# 4-byte rule -- it is what lets a DMA channel read a record as whole
# sectors straight into the ring, with no window and no bounce copy
# (FINDINGS 58.3 option C, and 59.4 made it a precondition: sc_in_data
# REFUSES a windowed read when the data phase is the channel's).
self.sector_aligned = self.version >= 5
self.rec_align = SECTOR if self.sector_aligned else (4 if self.aligned
else 1)
(self.W, self.H, self.fps, self.nframes,
self.k1, self.k4) = struct.unpack(">HHHHHH", b[4:16])
off_pal, off_cb1, off_cb4, off_frm = struct.unpack(">IIII", b[16:32])
@@ -61,21 +77,24 @@ class DLX:
self.mode_bytes = (self.nb * 2 + 7) // 8
# frame directory: (offset of the mode header, payload length)
if self.aligned and off_frm % 4:
raise ValueError(f"{path}: DLX2 frame stream starts at {off_frm}, "
f"which is not 4-byte aligned")
if off_frm % self.rec_align:
raise ValueError(f"{path}: DLX{self.version} frame stream starts at "
f"{off_frm}, which is not {self.rec_align}-byte "
f"aligned")
self.frames = []
p = off_frm
for _ in range(self.nframes):
(n,) = struct.unpack(">I", b[p:p + 4])
self.frames.append((p + 4, n))
p += 4 + n
if self.aligned:
p += -p % 4 # skip the pad to the next record
# The writer does not pad after the LAST record -- nothing follows it --
# so `p` may have advanced past the end by up to 3 bytes there.
p += -p % self.rec_align # skip the pad to the next record
# DLX2/DLX3 do not pad after the LAST record -- nothing follows it --
# so `p` may have advanced past the end by up to 3 bytes there. DLX4
# and DLX5 DO pad it, because a producer that trusts the index fetches
# a whole padded record for the last frame like any other.
slack = len(b) - p
if not (slack == 0 or (self.aligned and -3 <= slack < 0)):
if not (slack == 0 or (self.aligned and not self.has_index
and -(self.rec_align - 1) <= slack < 0)):
raise ValueError(f"{path}: {slack} trailing bytes after "
f"{self.nframes} frames")
@@ -92,7 +111,7 @@ class DLX:
if self.has_index:
self.index = list(struct.unpack(
f">{self.nframes}H", b[off_idx:off_idx + 2 * self.nframes]))
walked = [(-(4 + n) % 4 + 4 + n) // 4 for _, n in self.frames]
walked = [self._padded(n) // 4 for _, n in self.frames]
if self.index != walked:
bad = next(i for i in range(self.nframes)
if self.index[i] != walked[i])
@@ -106,6 +125,21 @@ class DLX:
f"{off_frm + 4 * sum(self.index)} bytes and the file is "
f"{len(b)} -- a producer trusting it would run off the end")
def _padded(self, n):
"""Bytes one record of `n` payload bytes occupies, pad included."""
ln = 4 + n
return ln + (-ln % self.rec_align)
def record_lengths(self):
"""Padded record lengths in BYTES, in stream order.
The one place the container's alignment rule is applied. Every caller
that used to write `4 + n + (-(4+n) % 4)` was carrying its own copy of
that rule, which is exactly the kind of duplication that made DLX5 a
multi-file change instead of a one-line one.
"""
return [self._padded(n) for _, n in self.frames]
def modes(self, f):
o, _ = self.frames[f]
h = np.frombuffer(self.raw, np.uint8, self.mode_bytes, o)
tools/encoder/encode.py
+41 -7
View File
@@ -66,6 +66,7 @@ import argparse, struct, sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import vq as VQ, vq_hybrid as H, ratectl as RC, spans as SP
from dlx import SECTOR
# Measured on the emulated 68000, FINDINGS 24. Instruction cycles against
# zero-wait-state memory, so these are floors, not hardware predictions.
@@ -134,8 +135,12 @@ def build_records(m, enc, span_mode):
return out
def write_container(path, m, frames, fps, k1, k4, span_mode):
"""Write the whole container. Returns (total bytes, video bytes, pad)."""
def write_container(path, m, frames, fps, k1, k4, span_mode, sector=True):
"""Write the whole container. Returns (total bytes, video bytes, pad).
`sector` selects DLX5's 512-byte record alignment over DLX4's 4-byte one.
"""
align = SECTOR if sector else 4
palette = m["pal"][:256]
if len(palette) < 256:
palette = np.vstack([palette, np.zeros((256 - len(palette), 3), np.uint8)])
@@ -144,6 +149,9 @@ def write_container(path, m, frames, fps, k1, k4, span_mode):
cb4_b = m["cb4"].astype(np.uint8).tobytes()
off_pal = 36 if span_mode else 32
if not span_mode:
sector = False # DLX2 has no index and no sector rule
align = 4
off_cb1 = off_pal + len(pal_b)
off_cb4 = off_cb1 + len(cb1_b)
# DLX4: the record index sits with the palette and the codebooks, ahead of
@@ -154,7 +162,7 @@ def write_container(path, m, frames, fps, k1, k4, span_mode):
qlens = []
for i, rec in enumerate(frames):
n = 4 + len(rec)
n += -n % 4
n += -n % align
q = n // 4
assert q <= 0xFFFF, (f"frame {i} is {n} B: a u16 longword count "
f"caps a record at 262,140 B")
@@ -169,9 +177,15 @@ def write_container(path, m, frames, fps, k1, k4, span_mode):
# a 68000, not a slow read. It vectors into the IPL and looks exactly like
# an infinite loop (FINDINGS 28.3). tools/bench/prep_dlx.py has been
# realigning at load time; the container now carries it.
tbl_pad = -off_frm % 4
# DLX5 aligns the frame stream itself as well as the records inside it, so
# the whole container can be laid on a volume at a sector boundary and every
# record lands on one. Aligning the records to each other and not the run
# they sit in would leave 117 of 120 of them off-sector again the moment the
# scene header changed length by a byte.
tbl_pad = -off_frm % align
off_frm += tbl_pad
hdr = ((b"DLX4" if span_mode else b"DLX2")
magic = (b"DLX5" if sector else b"DLX4") if span_mode else b"DLX2"
hdr = (magic
+ struct.pack(">HHHHHH", m["W"], m["H"], fps, len(frames), k1, k4)
+ struct.pack(">IIII", off_pal, off_cb1, off_cb4, off_frm))
if span_mode:
@@ -190,7 +204,7 @@ def write_container(path, m, frames, fps, k1, k4, span_mode):
# record is padded too, and the file ends where the index says it
# does rather than up to 3 bytes short of it.
if span_mode or i + 1 < len(frames):
n = -(4 + len(rec)) % 4
n = -(4 + len(rec)) % align
fh.write(b"\0" * n); frm_pad += n
total = os.path.getsize(path)
return total, sum(len(r) + 4 for r in frames) + frm_pad, frm_pad
@@ -252,6 +266,13 @@ def main():
"Default OFF: measured, it is worth one frame of 120 "
"at --spans need and a 2%% regression at --spans all "
"(FINDINGS 44)")
ap.add_argument("--joint-spans", action="store_true",
help="re-run the lam search with the bytes the span pass "
"freed, then re-span (E3, FINDINGS 39.3 item 5). The "
"span pass removes the block payload of every block "
"it covers, so without this the frame lands under its "
"allowance and the blocks that were NOT spanned were "
"priced as if those bytes were still needed.")
ap.add_argument("--no-cpu-fit", action="store_true",
help="drop the per-frame 68000 decode ceiling (session 7 "
"behaviour: 31%% of frames on hard content do not fit)")
@@ -259,6 +280,13 @@ def main():
help="how full the player's buffer is assumed to be at "
"scene start, as a fraction of the bucket (0 = cold "
"buffer after a seek, the conservative assumption)")
ap.add_argument("--no-reserve-black", action="store_true",
help="let the scene palette spend all 256 entries on the "
"picture. The default RESERVES index 0 as true black "
"(FINDINGS 23.4), because GVRAM cleared to zero shows "
"entry 0 and the 256x192 picture sits in a 256x256 "
"mode -- so a free palette letterboxes the frame in "
"whatever colour mediancut happened to put first.")
ap.add_argument("--preview")
a = ap.parse_args()
@@ -279,6 +307,7 @@ def main():
if a.disk_clk_byte is not None:
RC.DISK_CLK_BYTE = a.disk_clk_byte
RC.JOINT_DECIDE = a.joint_decide
RC.JOINT_SPANS = a.joint_spans
RC.JOINT_BUCKET = a.joint_bucket
cyc_budget = None if a.no_cpu_fit else RC.FRAME_CYCLES
span_mode = None if (a.spans == "off" or not rc) else a.spans
@@ -301,8 +330,13 @@ def main():
else:
print(f" target {prof['kbps']} KB/s, FIXED lam={lam} (no rate control)")
print(f" k1={k1} k4={k4}, {_IDX_BYTES}-byte indices")
print(f" palette: " + ("255 picture colours, index 0 RESERVED as true "
"black for the letterbox (23.4)" if not a.no_reserve_black
else "all 256 entries to the picture (--no-reserve-black); index 0 "
"is whatever mediancut put there, and the letterbox with it"))
m = H.build(a.frames_dir, k1=k1, k4=k4, iters=a.iters)
m = H.build(a.frames_dir, k1=k1, k4=k4, iters=a.iters,
reserve_black=not a.no_reserve_black)
if rc:
enc = RC.encode_rate_controlled(m, prof["kbps"], fps=a.fps,
bucket_frames=a.bucket_frames,
tools/encoder/ratectl.py
+59
View File
@@ -157,6 +157,23 @@ JOINT_DECIDE = False
# `--joint-bucket` turns it on.
JOINT_BUCKET = False
# E3 / FINDINGS 39.3 item 5: SPAN SELECTION IS GREEDY AFTER `lam`, and this is
# the switch that makes the two joint. The lam bisection picks a mode map
# against a byte allowance, and the span pass then REMOVES the block payload of
# every block it covers -- so the frame lands under the allowance by exactly
# the bytes the spans freed, and the blocks that were NOT spanned were priced
# at a lam chosen as if those bytes were still needed. Joint mode hands the
# freed bytes back to the lam search and re-spans the result, to a fixed point
# or two rounds, whichever comes first.
#
# It is a REFINEMENT, not a different objective: lam can only fall (the
# allowance only grows), so the un-spanned blocks can only improve, and a round
# is kept only if the frame still fits both ceilings it was already fitting.
# Default OFF until measured, which is 44.3's lesson -- ask whether the lever
# is loaded before pulling it.
JOINT_SPANS = False
JOINT_SPAN_ROUNDS = 2
def _byte_clk():
"""The debit the mode decision is allowed to see (0 = the old decision)."""
@@ -287,6 +304,39 @@ def _fit_spans(m, ctx, mode, sz, room, cyc_budget, span_mode, ib):
return nmode, nsz, H.cycles(nmode) + sel["clocks"], sel
def _refit_joint(m, ctx, allow, span_allow, lam_lo, lam_hi, cyc_budget,
span_mode, ib, mode_pre, mode, sz, cyc, sel, mu=0.0):
"""Give the lam search back the bytes the span pass freed, then re-span.
`mode_pre` is the mode map BEFORE spanning and `mode` the one after, so the
difference in frame_bytes is exactly what the spans made unnecessary. The
ceiling the result is judged against is the one _fit_spans was already
working to, so a kept round is never a frame that grew past a budget it was
inside.
"""
ceiling = span_allow if span_allow is not None else allow
for _ in range(JOINT_SPAN_ROUNDS):
if sel is None:
break
freed = (H.frame_bytes(mode_pre, ctx["nb"], ib)
- H.frame_bytes(mode, ctx["nb"], ib))
if freed <= 0:
break
lam2, mode2, sz2, _ = _search_lam(ctx, allow + freed, lam_lo, lam_hi, mu=mu)
if sz2 <= H.frame_bytes(mode_pre, ctx["nb"], ib):
break # lam did not move: already at the floor
n_pre, n_mode, n_sz, n_cyc, n_sel = (
mode2, *_fit_spans(m, ctx, mode2, sz2, span_allow if span_allow
is not None else allow, cyc_budget, span_mode, ib))
if n_sel is None or n_sz > ceiling:
break
if cyc_budget is not None and n_cyc + DISK_CLK_BYTE * n_sz > cyc_budget \
and cyc + DISK_CLK_BYTE * sz <= cyc_budget:
break # round 1 made the deadline and this does not
mode_pre, mode, sz, cyc, sel = n_pre, n_mode, n_sz, n_cyc, n_sel
return mode_pre, mode, sz, cyc, sel
def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
lam_lo=1.0, lam_hi=LAM_CLIFF, prefill=0.0,
steps=None, verbose=False, cycle_budget=None,
@@ -364,6 +414,10 @@ def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
mode_pre = mode
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
cycle_budget, span_mode, ib)
if JOINT_SPANS:
mode_pre, mode, sz, cyc, sel = _refit_joint(
m, ctx, allow, span_budget and span_allow, lam_lo, lam_hi,
cycle_budget, span_mode, ib, mode_pre, mode, sz, cyc, sel)
if cycle_budget is not None and cyc + DISK_CLK_BYTE * sz > cycle_budget:
# The byte allowance could not buy the frame's deadline, so fall
# back to the controller that pays in picture -- and then offer
@@ -374,6 +428,11 @@ def encode_rate_controlled(m, target_kbps, fps=12, bucket_frames=8,
mode_pre = mode
mode, sz, cyc, sel = _fit_spans(m, ctx, mode, sz, span_allow,
cycle_budget, span_mode, ib)
if JOINT_SPANS:
mode_pre, mode, sz, cyc, sel = _refit_joint(
m, ctx, allow, span_budget and span_allow, lam_lo,
lam_hi, cycle_budget, span_mode, ib, mode_pre, mode,
sz, cyc, sel, mu=mu)
late = cyc + DISK_CLK_BYTE * sz > cycle_budget
# Paint from the mode map as it was BEFORE spanning. A spanned run's
# blocks read SKIP in the emitted header, but SKIP means "hold the
tools/encoder/vq.py
+33 -5
View File
@@ -28,12 +28,40 @@ def load_frames(d):
return [np.asarray(Image.open(f).convert("RGB")) for f in fs]
def scene_palette(rgb, colors=256, stride=3):
"""One shared palette for the whole scene, no dithering (cel art is flat)."""
def scene_palette(rgb, colors=256, stride=3, reserve_black=True):
"""One shared palette for the whole scene, no dithering (cel art is flat).
`reserve_black` puts TRUE BLACK at index 0 and quantises the picture into
the other 255 entries. It is not a cosmetic default (FINDINGS 23.4): the
picture is 256x192 inside a 256x256 mode, GVRAM cleared to zero displays
palette entry 0, and a free mediancut palette puts a real image colour
there -- on 00020 f0001 it was (206,192,176), used by 210 image pixels, so
the 64 blank rows of letterbox came out beige. Entry 0 also needs `I = 0`
in the X68000's GRB555 word or the bars sit at RGB (4,4,4) (23.3); that
half is `dlxload.pack_palette`'s and it needs no special case, because a
(0,0,0) entry picks I=0 by its own minimum-squared-error rule.
Black is RESERVED, not withheld: the mapper may still spend index 0 on
genuinely black pixels, which is the entry it would have wanted anyway.
What the reservation buys is that index 0 is black REGARDLESS of what the
scene contains, which is what the letterbox needs and what a free palette
cannot promise.
"""
samp = np.concatenate([r.reshape(-1, 3) for r in rgb[::stride]])
ref = Image.fromarray(samp.reshape(-1, 1, 3)).quantize(
colors=colors, method=Image.MEDIANCUT, dither=Image.NONE)
pal = np.array(ref.getpalette()[:colors * 3], dtype=np.uint8).reshape(-1, 3)
n = colors - 1 if reserve_black else colors
q = Image.fromarray(samp.reshape(-1, 1, 3)).quantize(
colors=n, method=Image.MEDIANCUT, dither=Image.NONE)
pal = np.array(q.getpalette()[:n * 3], dtype=np.uint8).reshape(-1, 3)
if not reserve_black:
return q, pal
pal = np.vstack([np.zeros((1, 3), np.uint8), pal])
# The quantiser above cannot be reused as the mapping reference: its
# palette is the 255 it chose, at the wrong indices. A P-mode image
# carrying the FINAL palette is what every frame is then mapped against,
# so the indices in the container and the entries in the container's
# palette section are the same table by construction.
ref = Image.new("P", (1, 1))
ref.putpalette(pal.tobytes().ljust(768, b"\0"))
return ref, pal
tools/encoder/vq_hybrid.py
+2 -2
View File
@@ -113,10 +113,10 @@ def blocks_of(idx, pal, bw, bh):
return VQ.blockify(idx, pal, bw, bh)
def build(frames_dir, k1=256, k4=256, iters=16, lam=0.0):
def build(frames_dir, k1=256, k4=256, iters=16, lam=0.0, reserve_black=True):
rgb = VQ.load_frames(frames_dir)
H, W = rgb[0].shape[:2]
ref, pal = VQ.scene_palette(rgb)
ref, pal = VQ.scene_palette(rgb, reserve_black=reserve_black)
idx = VQ.palettise(rgb, ref)
# --- two codebooks, trained on the whole scene ---