drwily/Dragon-s-Lair-X68k
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Size against 4 Mbps: peaks break the scsi profile; DMA steal is not free

Browse Source 
User clarified the bandwidth figure is 4 Mbps (488 KB/s), not 4 MB/s -- ~8x
tighter than the previous commit reasoned against. Two consequences, plus a
correction to session 1.

1. The scsi profile committed in f0f2f80 DOES NOT FIT. Its mean is a
   comfortable 52% of the pipe but it PEAKS at 96.4% (470.8 KB/s on scene
   00020), and a frame that arrives late is a dropped frame, not a slow one.
   Peak/mean is 1.4-1.9x even on 1.2-1.7s clips. Sizing a real-time stream on
   the mean was the error. Flagged in STATUS rather than silently retuned,
   because the fix is rate control, not a lower lam.

   This promotes ratectl.py -- written in session 2, never wired into
   encode.py -- from a loose end to the highest-value work in the repo. It is
   worth a full step on the quality ladder (lam=25 -> lam=10, +0.7/+1.2 dB)
   because it allows sizing for the mean instead of the peak.

2. Pixel-exact is off the table at this bandwidth: lam=0 needs 92-97% of the
   pipe. The previous commit's "if SCSI sustains >=800 KB/s, ship transparent"
   conclusion only applies at roughly double the user's figure.

3. FINDINGS 5 said that because transfers are DMA, streaming "costs essentially
   no CPU" and the 68000 is "nearly idle". That is wrong. The HD63450 steals
   ~8 clocks per 16-bit word: 10-20% of the machine at the rates the profiles
   now use, on top of a 38% full-frame blit. Bandwidth and CPU are one budget.

Adds tools/encoder/profile_gen.py, which derives lam FROM a bandwidth figure
(accounting for audio, peak/mean and DMA steal) instead of reading it off the
knee of the RD curve, and docs/BENCHMARK.md covering how to actually measure
the subsystem -- including why MAME cannot answer the bandwidth question and
would be the same class of error as the FINDINGS 4 traps.

The 4 Mbps figure is user-supplied and its provenance is not recorded; every
profile now hangs off it.

Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
2026-08-23 12:12:34 -07:00
parent f0f2f807a4
commit 48e912de8b
4 changed files with 324 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/BENCHMARK.md
+133
View File
@@ -0,0 +1,133 @@
# Benchmarking the storage subsystem, and deriving profiles from it
Written session 2, in answer to "how do we benchmark the SCSI subsystem itself
and base our performance profiles around that?"
## The short answer
**You cannot set a bitrate profile from MAME.** MAME's `x68k_hdc` (SASI) and
`mb89352`/`cz6bs1` (SCSI) are *functional* models — they move the right bytes
and raise the right interrupts, but they are not transfer-timing accurate. A
throughput number out of MAME measures how fast the emulator's device model
hands over a buffer, which is an artefact of MAME's scheduling, not of a
Fujitsu MB89352 on a 10MHz bus.
So split the question in two, because they need different instruments:
| question | instrument | what it settles |
|---|---|---|
| does our read path work at all? | MAME | correctness, IOCS vs direct SPC, DMA setup |
| what rate does the hardware sustain? | derivation + real hardware | the profile bitrates |
Using MAME for the second is the same class of error as FINDINGS 4: a number
that looks like a measurement but is an artefact of the apparatus.
## Tier 1 — MAME: validate the path, not the speed
This is what `tools/bench/` already does, and what is currently blocked
(`IOCS _B_READ` returns -1 uniformly). Its value is that it proves the
request/DMA/completion loop is correct before any of it is burned into 68000
player code.
Next moves, in order — the SCSI path was never tried and is more relevant to
the target anyway:
1. **SCSI instead of SASI.** `-exp1 cz6bs1 -hard disk.chd`, with
`exp1:cz6bs1:scsi:0 harddisk`. Use IOCS `_S_READ` ($F5) rather than
`_B_READ` ($46).
2. **Move the stack.** `SP=$8000` may sit on top of the IOCS work area in low
RAM; put it at $200000+ (hypothesis 3 from session 1).
3. **Format the image.** Hypothesis 1 — a raw image has no X68000 partition
structure, so the IPL's boot scan never registers a drive and IOCS refuses.
Needs a Human68k image, which this machine does not have.
4. **Bypass IOCS entirely** and drive the MB89352 SPC registers directly. This
is what the shipping player will do anyway, since we want DMA straight into
a ring buffer with no OS in the path. If direct SPC works while IOCS does
not, that is a complete answer to the blocker and we simply skip IOCS.
Record from MAME: bytes transferred, completion status, and whether DMA or PIO
was used. **Do not record KB/s and treat it as a hardware figure.**
## Tier 2 — derivation: the defensible ceiling
Already partly in FINDINGS 5. Bounds worth tightening from datasheets:
- 68000 bus cycle: 4 clocks @ 10MHz, 16-bit => **5 MB/s** absolute ceiling
- HD63450 single-address DMA, ~8 clocks/word => **~2.5 MB/s** practical ceiling
- SCSI-1 asynchronous REQ/ACK handshake per byte, plus MB89352 FIFO depth
=> the real limiter, and the number we do not have from a primary source
The user's working figure is **4 Mbps = 488 KB/s**, which sits sensibly between
the derived DMA ceiling and observed period-drive rates. **Provenance not yet
recorded — worth pinning down, because every profile now hangs off it.**
### The coupling nobody had counted
Cycle-stealing DMA is not free DMA. At ~8 clocks per 16-bit word:
| stream | CPU stolen | + full-frame blit (38.3%) |
|---|---|---|
| 110 KB/s | 4.5% | 42.8% |
| 250 KB/s | 10.2% | 48.5% |
| 450 KB/s | 18.4% | 56.7% |
| 488 KB/s | 20.0% | 58.3% |
FINDINGS 5 concluded that because transfers are DMA, "streaming costs
essentially no CPU". **That is wrong.** It costs up to a fifth of the machine at
the rates we now care about. Bandwidth and CPU are one budget, not two.
## Tier 3 — real hardware: the only thing that settles it
An X68000 (ACE/EXPERT for SASI, Super/XVI or a CZ-6BS1-equipped 10MHz machine
for SCSI) with a **BlueSCSI or SCSI2SD**, which is the realistic deployment
anyway and removes mechanical seek from the measurement.
The benchmark must measure **what the player actually does**, not a synthetic
bulk read:
1. Sequential read into a ring buffer, in the chunk size the player will use.
2. **With the decoder running** — so DMA/CPU contention is included. An idle-CPU
bulk read will overstate the sustained rate by roughly the blit percentage.
3. Timed with the machine's own timer (MFP timer-C or the 1/100s system clock),
not a stopwatch.
4. Reported as sustained KB/s over >=30s, plus the worst 1-second window. The
worst window is what the profile must survive, since a frame that arrives
late is a dropped frame.
Deliverable: a `.x` executable and its source in `tools/bench/`, runnable on
real hardware and reporting a single number.
## Feeding the result back into the profiles
`tools/encoder/profile_gen.py` inverts the dependency — give it a bandwidth and
it returns the lam that fits, from the MEASURED rate-distortion points in
FINDINGS 17.4:
```
python3 tools/encoder/profile_gen.py --bw-mbps 4 --name scsi
```
It accounts for the three things that eat the pipe before video sees any of it:
audio (7.8 KB/s), peak-to-mean burstiness (measured 1.4-1.9x), and it reports
the DMA cycle-steal so the CPU coupling stays visible.
At 4 Mbps it currently returns:
| | lam | mean | peak | 00020 | 00146 | CPU |
|---|---|---|---|---|---|---|
| today (no rate control) | 25 | 194 KB/s | 368 KB/s | -1.22 dB | -4.21 dB | 53% |
| with rate control wired | 10 | 305 KB/s | 305 KB/s | -0.52 dB | -2.98 dB | 51% |
**Rate control is worth a full step on the quality ladder** — it is not a
tidiness feature, it is the difference between sizing for the peak and sizing
for the mean. That is the strongest argument yet for wiring up `ratectl.py`.
## What would change the design
- **If sustained is much below 4 Mbps** (say 2 Mbps / 244 KB/s): `scsi`
collapses toward today's `sasi`, and the two profiles stop being meaningfully
different. At that point reconsider 10 fps, or a narrower active area.
- **If sustained is much above** (>=8 Mbps / 976 KB/s): `lam=0` fits with
margin and the port ships **pixel-exact** video on SCSI.
- **If DMA cannot be used** and transfers fall back to PIO, the CPU cost rises
from ~15% to something far larger and CPU becomes the binding constraint.
This is the single worst outcome and is worth checking early in Tier 1.
docs/FINDINGS.md
+61
View File
@@ -450,3 +450,64 @@ now sit at 110 and 280 KB/s, close enough to the folklore ceilings that the
error bars matter, and **if SCSI sustains >=800 KB/s the correct `scsi` profile
is lam=0 — pixel-exact video.** Whether this port ships transparent or lossy on
SCSI is now waiting on one measurement.
## 18. Peak-to-mean burstiness — the mean was hiding the problem
Prompted by the user clarifying that the bandwidth figure is **4 Mbps = 488 KB/s**,
not 4 MB/s. That is ~8x tighter than what 17 was reasoning against, and it
changes the answer.
Per-frame instantaneous rate (video + 7.8 KB/s audio), 12 fps:
| scene | lam | mean | p90 | **max** | peak/mean | max as % of 488 KB/s |
|---|---|---|---|---|---|---|
| 00010 | 60 | 95.0 | 127.3 | 138.8 | 1.46 | 28.4% |
| 00010 | 10 | 198.9 | 266.1 | 284.0 | 1.43 | 58.2% |
| 00020 | 60 | 115.8 | 155.4 | 222.3 | 1.92 | 45.5% |
| 00020 | 10 | 255.9 | 391.2 | **470.8** | 1.84 | **96.4%** |
**The `scsi` profile as committed in f0f2f80 does not fit 4 Mbps.** Its mean is a
comfortable 52% of the pipe, but it peaks at 96.4% — and a frame that arrives
late is a *dropped frame*, not a slow one. Sizing a real-time stream on the mean
is the mistake; peak/mean is 1.4-1.9x on 1.2-1.7s clips and will be worse across
a full scene.
Two ways out, and only one is good:
- Size for the peak: `lam=25`, mean 194 KB/s. Costs a full step of quality.
- **Rate-control to the mean and carry a leaky bucket:** `lam=10` fits, and buys
back +0.7 dB (00020) / +1.2 dB (00146).
`ratectl.py` was written in session 2 but **never wired into `encode.py`**. This
demotes that from a loose end to the highest-value unfinished work in the repo.
## 19. Cycle-stealing DMA is not free DMA — 5 was wrong
FINDINGS 5 concluded "because it's DMA, streaming costs essentially no CPU —
this stacks with the 8% blit utilisation. The 68000 really is nearly idle."
The HD63450 steals bus cycles from the 68000 at roughly 8 clocks per 16-bit word:
| stream | words/s | clocks/s | CPU stolen | + full-frame blit |
|---|---|---|---|---|
| 110 KB/s | 56,320 | 450,560 | 4.5% | 42.8% |
| 250 KB/s | 128,000 | 1,024,000 | 10.2% | 48.5% |
| 450 KB/s | 230,400 | 1,843,200 | 18.4% | 56.7% |
| 488 KB/s | 249,856 | 1,998,848 | 20.0% | 58.3% |
At the rates the profiles now use, streaming costs **10-20% of the machine**.
Still affordable — nothing here breaks — but **bandwidth and CPU are one budget,
not two**, and any future headroom argument has to spend from both. The
"nearly idle" framing should not be reused.
(The 8 clocks/word figure is session 1's ESTIMATE from HD63450 timing, not a
measurement. It is the weakest link in this table.)
## 20. Where the profiles should come from
`tools/encoder/profile_gen.py` now derives lam from a bandwidth figure rather
than from the shape of the RD curve, accounting for audio, peak/mean, and
reporting DMA steal. Full benchmarking methodology — and why MAME cannot answer
the bandwidth question — is in `docs/BENCHMARK.md`.
The 4 Mbps figure itself is **user-supplied and its provenance is not recorded**.
Every profile now hangs off it, so it is worth pinning down.
docs/STATUS.md
+13 -5
View File
@@ -18,10 +18,16 @@ Session 1 left "which machine do we target" open. The user's answer: **ship both
as two quality profiles. This is now implemented rather than hypothetical — the
bitrate ceiling is a build parameter in `tools/encoder/ratectl.py`:
| profile | target | lam | quality (00020 / 00146) | bus utilisation | machine |
| profile | target | lam | quality (00020 / 00146) | machine |
|---|---|---|---|---|---|
| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | 35% of 300 KB/s | stock 10MHz ACE/EXPERT |
| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | 28% of 1 MB/s | Super/XVI, or CZ-6BS1 board |
| `sasi` | 110 KB/s | 60 | 36.9 / 29.6 dB | stock 10MHz ACE/EXPERT |
| `scsi` | 280 KB/s | 10 | 39.4 / 32.3 dB | Super/XVI, or CZ-6BS1 board |
**WARNING — `scsi` does not currently fit 4 Mbps.** The user's working bandwidth
figure is **4 Mbps = 488 KB/s**. `scsi` means 52% of that but **peaks at 96.4%**
(FINDINGS 18), and a late frame is a dropped frame. Until `ratectl.py` is wired
into `encode.py`, `scsi` must either drop to `lam=25` (194 KB/s mean) or not
ship. Derive profiles with `tools/encoder/profile_gen.py --bw-mbps 4`, not by eye.
`scsi` is now within **0.5 dB of the palette ceiling** on 00020. These were
initially set at 45 / 75 KB/s, which was 12% / 7% bus utilisation — read off the
@@ -126,8 +132,10 @@ Next move is the untried SCSI path: `-exp1 cz6bs1 -hard disk.chd`.
## Next steps, in priority order
1. **Wire rate control into `encode.py`** and validate that the hard ceiling
actually holds on an action scene (the whole point of choosing VQ).
1. **Wire rate control into `encode.py`** — now the highest-value work in the
repo, not a loose end. It is worth a full step on the quality ladder
(lam=25 -> lam=10, +0.7/+1.2 dB) because it lets us size for the mean
instead of the peak. Validate the bucket holds on an action scene.
2. ~~Entropy-code the payload~~ — **ABANDONED, see FINDINGS 17.2.** Deflate
decode is ~216% of the frame budget on a 68000 and LZ4 is ~54%; there is no
room beside a 38% blit. All bitrates are raw payload. This also demotes the