Session 1: hardware research, content measurement, codec decision, MAME harness

Verified GVRAM is one word-access per pixel in ALL color modes; chose 256-color
256x192 with movem.l bursts (page 1 sacrificed as double-buffer).

Measured 8 scenes from the Blu-ray source: blit costs under 8% of the 12fps
cycle budget, so I/O is the bottleneck, not CPU. Naive delta+RLE reaches only
3.2:1 (365 KB/s, 470MB) -> decision to use 4x4 vector quantization (~30 KB/s).

"Shot on twos" assumption failed: the transfer has zero duplicate frames, so
12fps requires explicit decimation.

Documents three false measurement results and their root causes (per-frame
Floyd-Steinberg dithering, temporal denoise, exact-match dedupe on noisy source).

MAME Lua injection harness works and is reusable for cycle-cost measurement;
the IOCS _B_READ disk benchmark is blocked returning -1.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
reala-misaki
2026-08-23 11:23:49 -07:00
commit 65112b9305
19 changed files with 842 additions and 0 deletions
+190
View File
@@ -0,0 +1,190 @@
# Findings — session 1 (2026-08-23)
All numbers here are MEASURED unless marked ESTIMATE or FOLKLORE.
---
## 1. Source material
`DRAGONS_LAIR.iso` — 16 GB, UDF 2.x, **decrypted** (no AACS dir).
Loop-mounted read-only at `/media/reala-misaki/BDROM` via `udisksctl loop-setup -r -f`.
(7-Zip cannot read UDF 2.x; use the loop mount.)
- **224 `.m2ts` streams**, 1920x1080, **MPEG-2, progressive, 23.976 fps**
- Size histogram: 47 <5MB, 138 5-50MB, 22 50-150MB, 14 150-400MB, 3 >400MB
- The 185 sub-50MB streams are the **arcade branching scenes already split into
individual clips** — we get scene boundaries for free.
- Big streams are full-feature playthroughs: 00215 (1376s), 00216 (1151s), 00223 (566s)
- Typical scene clip ~60s (00203/00205/00199), some ~100s (00164/00212)
**Gotcha:** clip durations vary wildly. Always read `format=duration` and seek
relative to it. Seeking to a fixed offset silently yields 0 frames on short clips.
---
## 2. GVRAM layout [verified — see HARDWARE.md for source]
**One 16-bit word per pixel position in EVERY color mode.** Bit depth does not
change VRAM bandwidth; it only subdivides the word.
`addr = page_base + y*1024 + x*2` — adjacent pixels are 2 bytes apart in all modes.
Consequence: low bit depth buys **no speed**. 16-color mode is strictly worse than
256-color (same bus traffic, 1/16 the palette). Page-alias writes are hardware
auto-masked, so 16-color needs no software read-modify-write — but it's still
one word-access per pixel.
**Chosen: 256 colors, 256x192 active area.**
In 256-color mode P0=low byte, P1=high byte of each word. Sacrificing page 1 as a
double-buffer lets a `move.l` cover two pixel positions, enabling `movem.l` bursts
(12 regs = 48 bytes = 24 pixels). Identical blit cost to 65536-color mode but
**half the on-disk data**.
---
## 3. Content measurements (8 scenes sampled, 5s each at 40% into each clip)
| metric | mean | p90 |
|---|---|---|
| pixels changed / frame | 20.1% | 30.2% |
| **blit cost** | **~64k cycles** | **~97k cycles** |
| naive delta+RLE frame size | 15.5 KB | 19.6 KB |
Budget is **833,333 cycles/frame** @ 12fps on a 10MHz 68000.
### => THE CPU IS NOT THE BOTTLENECK. I/O IS.
Blit uses **under 8%** of budget. The naive row-span+RLE codec achieves only
**3.2:1**, giving **365 KB/s / 470 MB** at 24fps (~183 KB/s / 235 MB at 12fps).
Per-scene variance is extreme: static dialogue ~30 KB/s, action ~700 KB/s.
Any codec needs a hard bitrate ceiling, not just a good average.
### "Shot on twos" — ASSUMPTION FAILED
Dedupe found **zero** duplicate frames across all 8 scenes (`uniq=120/120`,
24.0 fps effective). This Blu-ray is a restoration where every frame is unique.
We do NOT get halved data for free. **Decimation to 12fps must be explicit.**
A weak alternation signature does exist (even-index pairs 40.7% vs odd 27.5%,
ratio 1.5x, with occasional true-duplicate pairs at 0.03-0.19%), but it is
irregular — Bluth mixed ones and twos; action is animated on ones.
---
## 4. MEASUREMENT TRAPS — read before trusting any pipeline number
Three separate false results were produced and caught this session. All three
looked plausible. Guard against them:
1. **Per-frame Floyd-Steinberg dithering destroys temporal coherence.**
Error diffusion is chaotic: a +/-1 input change cascades across the row and
produces a completely different index pattern. First run reported 31.5% pixels
changed with near-zero variance (median 31.6, p90 32.3, max 32.7) while source
mean-abs-diff was 0.09 — i.e. visually identical frames. That flat variance is
the tell: **real animation has scene-dependent variance; noise does not.**
Use no dithering (cel art is flat) or ordered/Bayer (spatially fixed, temporally stable).
2. **Temporal denoise smears motion.** `hqdn3d=4:3:6:4` — the `6:4` are temporal
params. It flattened real motion, which then measured as "no motion" and
produced an absurd 0.8 fps / 4 MB result. **Use spatial-only: `hqdn3d=4:3:0:0`.**
3. **Exact-match dedupe fails on a noisy source.** MPEG-2 grain means near-duplicate
frames differ by +/-1 and are never bit-exact. Use a threshold on
"% pixels differing by more than N levels", and pick the threshold from the
observed distribution, not a guess. A 2% threshold ate genuine animation when
mean consecutive change was only 0.9%.
**Sanity rule: if a result has suspiciously low variance, or is suspiciously
good, it is probably an artifact of the measurement, not a property of the content.**
Scripts kept in `tools/analysis/` — 01 and 02 are marked BROKEN deliberately as
regression references; 03 and 04 are the correct ones.
---
## 5. Storage interface — the SASI/SCSI split
[Yasuma, X68030 internal SCSI controller]
- Interface: **SCSI-1**, 50-pin, 5 MB/s bus spec
- Controller: **Fujitsu MB89352** SPC
- Transfer mode: **DMA** (via **HD63450** DMAC)
- Bus: X68000 original bus, **16-bit @ 10MHz**
**Even on the X68030, SCSI runs at 10MHz 16-bit DMA.** Storage bandwidth does
NOT scale with CPU — the controller sits on the original bus. HD63450's 12.5MHz
official ceiling is why the X68030 runs at 25MHz. An "HSCSI" TSR forces PIO/FIFO
transfer instead of DMA but was marginal even at 25MHz.
Because it's DMA, **streaming costs essentially no CPU** — this stacks with the
8% blit utilisation. The 68000 really is nearly idle.
### Model split — IMPORTANT
**The 10MHz models (original X68000, ACE, PRO, EXPERT) use SASI, not SCSI.**
Built-in SCSI starts at the X68000 **Super** (1990) and continues through XVI,
Compact, X68030. SCSI on earlier machines needs the **Sharp CZ-6BS1** board
in an I/O slot (MAME models this: `-exp1 cz6bs1`).
| target | bandwidth | naive codec (365 KB/s) | VQ codec (~30 KB/s) |
|---|---|---|---|
| SASI (stock ACE/EXPERT) | ~300-500 KB/s FOLKLORE | infeasible | comfortable |
| SCSI (Super+, or CZ-6BS1) | ~1 MB/s FOLKLORE | tight but viable | trivial |
Derived bounds (ESTIMATE): 16-bit @10MHz with 4-clock bus cycle = 5 MB/s absolute
ceiling; HD63450 single-address DMA ~8 clocks/word => ~2.5 MB/s practical ceiling,
before SCSI-1 async handshake and drive latency.
**No measured benchmark was obtained — see STATUS.md.** The ~300-500 KB/s and
~1 MB/s figures are folklore-grade; I could not find a primary measurement.
---
## 6. Codec decision: vector quantization (Cinepak-style)
Given ~8x CPU headroom and an I/O ceiling, spend CPU to buy bandwidth.
- Split frame into 4x4 blocks, encode each as a 1-byte index into a per-scene codebook
- Decode = 16-byte copy from a lookup table: nearly free
- A **full** frame = 256*192/16 = **3,072 bytes** — a hard 16:1 floor before delta
- Add block-level delta on top; action scenes ~2-3 KB/frame
- => roughly **30 KB/s, ~40 MB total**, with a *deterministic* bitrate ceiling
Divergence from the SNES project (below): use a **per-scene codebook with delta
updates**, not a per-frame rebuild. We trade adaptivity for bandwidth because we
have 2MB RAM to keep a codebook resident and CPU to spare.
**Risk not yet evaluated:** 4x4 VQ with a 256-entry codebook will visibly soften
detail. Bluth's fine ink linework is what suffers. Prototype and eyeball before committing.
---
## 7. Comparison: astrobleem/SNES-SuperDragonsLairArcade
Reached the **same core architecture independently** — "512 tiles per frame" is
vector quantization (8x8 codebook + tilemap). Good validation.
But: the SNES PPU has **no bitmap mode**, so tiles are forced on them by display
hardware. The X68000 has a real linear framebuffer, so VQ is a *compression
choice* we can tune or drop per-scene.
**MSU-1 is a bandwidth cheat we don't have.** It's a modern flash-cart coprocessor
giving memory-mapped streaming the real SNES never had. Their budget: 512 tiles x
32 bytes (4bpp 8x8) + tilemap ~= 18 KB/frame => **~430 KB/s** at 23.976fps.
That's *higher* than the 365 KB/s we'd reject on SASI. (ESTIMATE: my arithmetic on
their stated tile budget, not a measured figure.)
Where we're ahead: 256 simultaneous colors from a 65536 palette vs their 4bpp
sub-palettes needing a tile-aware palette optimizer plus a spatial smoothing pass
to hide 8x8 palette seams. That problem doesn't exist for us. Plus 68000@10MHz
vs 65816@3.58MHz, and 2MB vs 128KB.
**Most valuable thing in that repo is NOT the codec — it's `data/events/`:**
516 chapter definitions across 29 scenes as XML, plus
`data/chapter_event_inventory.md`. That's the arcade scene graph and input-timing
structure, entirely hardware-independent — the whole game-logic layer we'd
otherwise reverse-engineer from the arcade ROM.
**TODO: check their license before planning to reuse it.**
Their 516 chapters are finer-grained than our 224 Blu-ray streams, so mapping
their event table onto our footage means subdividing streams by timecode.
Caveat: all of the above is from README/repo-tree summaries, not their source.
+52
View File
@@ -0,0 +1,52 @@
# X68000 hardware facts (verified, not assumed)
Target baseline: X68000 ACE/Expert class. 68000 @ 10MHz, 2MB RAM, SCSI HDD.
## Cycle budget
Content is cel animation shot on twos -> 12fps effective.
10,000,000 / 12 = **833,333 cycles per frame**. This is the wall.
## GVRAM layout [verified: JC-000/x68000-dev-guide docs/graphics.md]
- 512KB physical, mapped at $C00000-$DFFFFF as aliased page windows.
- **One 16-bit word per pixel position, in EVERY color mode.**
Bit depth does NOT change VRAM bandwidth. It only subdivides the word.
- Address formula (512-wide page): `addr = page_base + y*1024 + x*2`
- Horizontally adjacent pixels are 2 bytes apart in all modes.
| Mode | Pages | Word subdivision | Page aliases |
|-------|-------|-----------------------------|-----------------------------------|
| 16 | 4 | nibble per page (P3..P0) | $C00000/$C80000/$D00000/$D80000 |
| 256 | 2 | byte per page (P0=lo,P1=hi) | $C00000 (P0) / $C80000 (P1) |
| 65536 | 1 | whole word | $C00000 |
Writes via a page alias are **auto-masked and shifted by hardware** into that
page's field. No software read-modify-write is needed for 16-color mode.
## Chosen mode: 256 colors, 256x192 active area
Rationale: since every mode is one word-access per pixel, low bit depth buys
no speed. 256-color halves on-disk data vs 65536-color for identical blit cost.
Coalescing trick: in 256-color mode a `move.l` spans two pixel positions across
BOTH pages. We sacrifice page 1 as a double-buffer and let it take duplicate
data, which lets us use `movem.l` bursts:
movem.l d0-d7/a0-a3,(a6) ; 12 regs = 48 bytes = 24 pixels
Full-frame refresh cost: 49,152 px / 24 = 2,048 bursts
~104 cyc/burst + VRAM waits (~1.5x) = ~320k cycles << 833k budget
A hard scene cut fits. Delta frames cost far less.
## Audio
MSM6258 ADPCM via HD63450 DMAC. ~7.8KB/s at 15.6kHz mono, near-zero CPU.
22 min ~= 10MB.
## Storage estimate
~15,800 frames. Target ~3-5KB/frame compressed -> 50-80MB video + 10MB audio.
Stream rate ~40-60KB/s. Well within SCSI sustained throughput.
---
## Corrections applied after session-1 measurement
- Storage estimate below was optimistic. Measured naive codec gives ~470MB @24fps.
See FINDINGS.md §3. VQ codec targets ~40MB.
- "Content is cel animation shot on twos -> 12fps effective" — **the transfer has
zero duplicate frames**; 12fps requires explicit decimation. See FINDINGS.md §3.
- 10MHz models are **SASI**, not SCSI. See FINDINGS.md §5.
+139
View File
@@ -0,0 +1,139 @@
# Status & next-session handoff — end of session 1 (2026-08-23)
## Decisions locked
| decision | value | why |
|---|---|---|
| Target CPU | 68000 @ 10MHz (stock) | hardest honest constraint |
| Display mode | 256 colors, 256x192 in 256x256 CRTC mode | every mode is 1 word-access/pixel, so 256c is free vs 16c |
| Double buffer | **none** — page 1 sacrificed | enables `movem.l` 24px bursts; delta coding needs a RAM reference frame anyway |
| Codec | 4x4 vector quantization, per-scene codebook + block delta | CPU is idle, I/O is the ceiling — spend cycles to buy bandwidth |
| Framerate | 12 fps, **explicit decimation** | source has zero duplicate frames; no free "twos" win |
| Medium | SCSI HDD image (.hds) | but see SASI/SCSI split below |
| Emulator | MAME 0.277 x68000 | accurate enough that measured cycles mean something |
**OPEN QUESTION for the user:** stock 10MHz machines are **SASI**, not SCSI.
Three options, not yet chosen:
1. Stock 10MHz + SASI (purist) — VQ becomes mandatory
2. Stock 10MHz + CZ-6BS1 SCSI board — relieves I/O, keeps CPU honest
3. Super/XVI baseline — built-in SCSI, still a 10MHz 68000
Recommendation: make the codec's bitrate ceiling a **build parameter**, so one
encoder serves all three and the target is chosen at package time.
---
## Working setup
**MAME ROMs**`~/mame/roms/x68000.zip` (present, working).
Must pass **`-bios ipl10`**; the default BIOS is `cz600ce`, whose split
even/odd IPL halves (`rh-ix0897cezz.ic12` / `rh-ix0898cezz.ic11`) are absent.
`-verifyroms` will still report those two as missing — this is expected and harmless.
Boots headless at ~430-480% speed:
```
mame x68000 -bios ipl10 -video none -sound none -nothrottle -seconds_to_run 3
```
**Assembler** — vasm built from source, binary at `tools/vasm/vasmm68k_mot`
(source tarball alongside it). Verified correct 68000 output.
```
tools/vasm/vasmm68k_mot -Fbin -o out.bin in.s
```
**Blu-ray** — mount with:
```
udisksctl loop-setup -r -f DRAGONS_LAIR.iso # -> /media/reala-misaki/BDROM
```
NOTE: this loop mount is still active from session 1. Re-mount if the machine rebooted.
---
## MAME Lua harness — WORKING, reusable
`tools/bench/*.lua` inject 68000 machine code straight into emulated RAM and time
it against the emulated clock. No bootable disk or OS required. This is the
measurement rig for all future cycle-cost work (blit timing, decoder benchmarks).
Pattern:
```
mame x68000 -bios ipl10 -video none -sound none -nothrottle \
-seconds_to_run 30 -plugins -autoboot_script yourscript.lua
```
### Three MAME Lua gotchas — all cost real time, all now solved
1. **Retain the notifier subscription.** `emu.add_machine_frame_notifier()` returns
a token; if you drop it into a chunk-local it is garbage-collected and the
callback **silently stops firing**. Assign it to a **global** (`SUB = ...`).
2. **The stack pointer is `SP`, not `A7`** in `cpu.state[...]`.
Full list: A0-A6, D0-D7, PC, SP, SR, USP, CURPC, CURFLAGS, IR.
3. **`autoboot_script` fires at time=0, before boot** (PC=0). Wait until
`machine.time` >= ~5s before injecting, or IOCS is not yet initialised.
Also: piping MAME through `grep` block-buffers output — write raw to a file when
backgrounding, or you will see an empty log and assume a hang.
And never `pkill -f 'mame x68000'` — the pattern matches your own shell and kills it
(exit 144). Use `pkill -x mame`.
---
## BLOCKED: disk throughput benchmark
**Goal:** measure real SASI/SCSI KB/s to replace the folklore figures in FINDINGS.md §5.
**Status:** harness fully working; the IOCS call itself fails.
`IOCS _B_READ` ($46 via `TRAP #15`; d1.hb=PDA, d2.l=position, d3.l=bytes, a1=buffer)
returns **`FFFFFFFF` (-1), zero reads**, uniformly across:
- all 16 PDA values $80-$8F
- both d1 encodings (PDA in bits 31-24 and bits 15-8)
- image sizes 10MB / 20MB / 40MB
The uniformity is the diagnostic: calls are **dispatched and cleanly rejected**,
so `TRAP #15` and IOCS are reachable. MAME does mount the image
(`:x68k_hdc: opened image file bench.hdf`).
**Untested hypotheses, in rough order of likelihood:**
1. The raw image has no X68000 SASI format, so the IPL's boot scan never registered
a usable drive and IOCS refuses. Would need Human68k to format one — **we have
no Human68k image on this system.**
2. MAME's `x68k_hdc` SASI implementation may be too partial for IOCS-level reads.
3. `SP=$8000` may put the injected stack on top of the IOCS work area in low RAM.
Try a much higher stack.
4. The **SCSI path was never tried** — this is the obvious next move and is more
relevant to the target anyway:
`-exp1 cz6bs1 -hard disk.chd` with `exp1:cz6bs1:scsi:0 harddisk`
(`-listmedia` gains a `harddisk` slot accepting .chd/.hd/.hdv/.2mg/.hdi).
**Honest assessment: this benchmark is NOT on the critical path.** The VQ codec
(~30 KB/s) is correct whether SASI does 300 or 600 KB/s. Do not let it block the
encoder. Its real value is deciding whether the *simpler* row-span codec could
have sufficed.
Caveat if resumed: MAME idealizes drive seek latency. That's acceptable because the
realistic deployment is BlueSCSI/SCSI2SD (SD-backed, no mechanical seek), so what
gets measured is the bus/DMAC/controller path — the genuine ceiling. The caveat
only bites for a real period spinning drive.
---
## Next steps, in priority order
1. **Build the VQ encoder** (`tools/encoder/`) — 4x4 blocks, per-scene codebook,
block delta. Emit sample PNGs for visual evaluation. **The open question is
whether VQ softens Bluth's ink linework unacceptably — decide by eye before
committing to the architecture.**
2. **Full-disc survey** — all 224 streams, not 5s samples, to firm up bitrate
(current numbers are +/-30%) and map streams onto the arcade scene graph.
3. **Check the SNES project's license**, then evaluate reusing `data/events/`
(516 chapters / 29 scenes) as the scene-graph and input-timing layer.
4. Resolve the SASI/SCSI target question with the user.
5. Optionally unblock the disk benchmark via the SCSI path.
6. 68000 player skeleton: CRTC init for 256x192x256c, `movem.l` blitter,
ADPCM via HD63450 DMA.
## Not yet started
- Any 68000 player code
- ADPCM audio extraction/encoding (MSM6258, 15.6kHz mono, ~7.8 KB/s, ~10MB for 22min)
- Disk image packaging / container format
- Game logic (scene branching, input windows, death clips)