diff --git a/README.md b/README.md index 912b649..a1e7473 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,28 @@ at 488 KB/s a one-deep request queue gives away **6.8% of the pipe and underruns 59 of 120 frames**, a two-deep one gives away 3.4% and underruns none — on a container whose whole surplus over the wire is 8.7% (FINDINGS 55). +**The scene graph is in, and the worst gap between two decision points is +zero.** `tools/import/scenegraph.py` imports the arcade scene graph — 40 scenes, +516 sequences, 906 input windows — and 5.4% of the game's 612 branch transitions +open an input window on the first frame of a clip the disc *seeked to*, so two +seeks can fall back to back with no play between them. A rule of the form "has +there been enough play since the last branch" can therefore be answered no by +the **content**, not by the buffer. It does not break the design: a branch on an +empty ring costs the 2-record prefill, **149.7 ms at 488 KB/s**, not the climb. +What it removes is margin — at that rate in a 256 KB ring, **76% of this game's +branch points arrive before the ring has refilled**, and a 512 KB ring makes it +90%, because doubling the ceiling does not touch `pipe - wire` (FINDINGS 56). + +**Nothing outside-derived is committed here.** The scene graph is not +redistributable from this tree; it is regenerated from a reader's own clones +into gitignored `tmp/`, and `tools/import/scenegraph.py` is the single file in +the repo coupled to those projects — everything downstream reads `DLXSCENE1`, +this project's own schema, with the sources' attribution carried in it. +DirkSimple is zlib (Ryan C. Gordon); the SNES chapter set is MIT (Chad +Doebelin) and, by its own README, *derived* from DirkSimple rather than an +independent transcription, which struck a cross-check this project had planned +on for eight sessions. + **Current encode:** 496.7 KB/s at 29.19 dB, 1 frame of 120 over the 12fps budget, and that one is frame 0, the intra frame, late on purpose. @@ -133,7 +155,8 @@ budget, and that one is frame 0, the intra frame, late on purpose. mounted) re-runs both display regression tests, the rate-control drift gate, the display-path coherency counterexample, a 120-frame 68000 decode on two CPU cores, the ring and paced-ring passes, the DMAC configuration gate and the -load-time transforms on both cores, then prints `ALL GREEN`. +load-time transforms on both cores, then imports and gates the scene graph +when a DirkSimple checkout is present, then prints `ALL GREEN`. ## Reproducing this @@ -316,6 +339,16 @@ tools/bench/c68k/ headless px68k C68K harness, a SECOND emulator for every which MAME cannot report. The Makefile's -no-pie and the harness's MAP_32BIT arena are load-bearing: C68K truncates host pointers to 32 bits. + 25 imports nothing itself: it reads the DLXSCENE1 scene + table and reports the worst gap between two decision points, + what the input layer has to survive, and what both cost in + 51.3's accumulated slack across explicit rates. +tools/import/ the ONLY code in this tree coupled to somebody else's source. + scenegraph.py reads a DirkSimple checkout (and optionally the + SNES chapter XMLs) and writes tmp/scenegraph.json in this + project's own DLXSCENE1 schema, with the sources' licences and + attribution inside it. Nothing is vendored and the output is + gitignored derived data. tools/media/ builds docs/img/ from a paced recording run tools/vasm/ vasm m68k assembler, binary plus source tarball tools/encoder/ hybrid VQ encoder and DLX3 container writer. diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index d44010a..63ce8ff 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -5017,3 +5017,215 @@ and are exact, so 49.6's table is unaffected. two-instruction test at the top of the pace wait that routes a self-filled run into a polling wait loop. The legacy wait loops are byte for byte the ones FINDINGS 51 measured and a host-filled run executes none of the new code. + +--- + +# Findings — session 24 (2026-08-24) + +## 56. The scene graph is in, and the worst gap between two decision points is zero (session 24) + +ROADMAP G1, and it was scheduled early *because it is a measurement input*. +FINDINGS 51.3 established that a ring's lookahead is accumulated out of +`pipe - wire` and that a seek spends all of it, so what a branch point costs is +set by the rate and by **the time since the last branch**. 55.5 rehearsed a seek +on the machine and then said out loud that it could not ask the question that +matters, because nothing in this tree knew where the branch points *are*. + +Now it does, in two files with a hard line between them (USER DECISION, +session 24): + +- **`tools/import/scenegraph.py` is the only file in this tree that knows + anything about somebody else's source.** Their file layout, table names, + timing formulas and magic constants are wired into it and nowhere else. It + writes **`DLXSCENE1`**, this project's own schema — a clip's length, its exits + and the earliest instant each of them can fire — into gitignored `tmp/`. +- **`tools/analysis/25_scene_graph.py` reads only DLXSCENE1** and could not name + an outside project if it wanted to. + +Nothing is vendored and nothing outside-derived is committed. That line is worth +drawing before the game-logic layer exists rather than after: when one of those +projects moves, exactly one file in this tree breaks, and when a scene table is +eventually committed rather than regenerated, there is one place the attribution +already lives. Restructuring changed no number in this section — the split was +made after the measurement and the whole output was re-run byte for byte. + +### 56.1 What was imported, and what gates it + +**40 scenes, 516 sequences, 906 input windows**, plus `scene_manager.rows` — the +13x3 scene order the arcade walks. **282 of the 516 sequences (54.7%) are +entered by a seek**; the rest play on from wherever the disc already is, which +is the distinction the whole finding rests on. + +The source table is Lua, and there is no Lua interpreter on this box, so the +importer contains a **deliberately small Lua-subset parser**: table constructors, +literals, bare identifiers, the file's four timing helpers and `+`/`-` between +them. Anything else is a parse error rather than a silent skip. Three gates +stand behind that choice: + +1. the four timing helpers are **matched against the text that defines them** — + `frame / 23.976`, the `- 6297.0` ROM offset, `noseek` returning `-1`, + `(seconds * 1000) + ms`. If upstream changes a formula, the tool stops + rather than keeps evaluating the old one. +2. the parse must reach **516 sequences and 906 input windows** exactly. A + parser that quietly dropped a branch would produce a *smaller* graph and a + *longer* worst gap, i.e. it would fail in the flattering direction. +3. `check.sh` runs the import and then the analysis, and **skips rather than + fails** when there is no checkout, like every other outside-this-repo input + in the tree. + +### 56.2 CORRECTION to FINDINGS 16: there is only one transcription + +> 16 cleared two permissively licensed transcriptions and planned to **diff them +> against each other to catch transcription errors**. That plan does not work. + +The SNES project's own `data/events/README.md` states its 516 chapter XMLs are +"**derived from DirkSimple game data**". They are a *conversion* of the same +transcription, not a second one. The diff below is still worth running — it +catches conversion errors — but it **cannot** catch a transcription error, +because there is nothing independent to compare against. DirkSimple is the +single source, and its own provenance is the arcade ROM's data table. + +That correction was cheap to make and it was one sentence of a README away from +never being made at all. It belongs with FINDINGS 42.1: a plausible source with +no provenance is folklore, and so is a plausible cross-check. + +### 56.3 THE MEASUREMENT: the worst gap is zero, and 5.4% of branches are + +Chaining play across non-seeking sequences and taking the **earliest** moment an +input window opens (the least play a clip can deliver before the branch it leads +to), over 612 distinct transitions into a seek, excluding attract mode: + +| | seconds of play | +|---|---:| +| worst | **0.000** | +| p10 | 0.950 | +| p25 | 1.966 | +| median | 3.473 | +| p75 | 5.800 | +| p90 | 9.548 | +| best | 82.497 | + +**33 of the 612 (5.4%) are zero**: an input window that opens at t=0 of a clip +the disc *seeked to*, so two seeks can fall back to back with no play between +them at all. `flaming_ropes.enter_room -> fall_to_death` is one — press right on +the frame the clip starts and the player dies immediately. + +This is not an edge case to be designed around; it is the game. **A rule of the +form "has there been enough play since the last branch" — 51.2's slack rule, +`ring_may_seek` in `src/player/ring.i` — can be answered NO by the content, +not by the buffer**, and no amount of ring is a defence. + +**203 of the 612 end the scene**, which in this design is also a container +change and needs 6,164 header bytes before frame 0 (53, 55.1). The worst +scene-change gap is **0.541 s** and the median is 2.501 s. + +### 56.4 The median branch point arrives before the ring has refilled + +51.3's climb, against the game's own gap distribution. Gate container +(`rc_fr_singe_scsi_span.dlx`, mean record 36.5 KB, wire 446.1 KB/s); rates are +explicit arguments and every one of them is a sensitivity, not a claim: + +| ring KB | pipe KB/s | ceiling | climb s | branch points under the climb | +|---:|---:|---:|---:|---:| +| 256 | 451.4 | 3 | 20.83 | 601/612 (98%) | +| 256 | 488.0 | 7 | 6.11 | 468/612 (76%) | +| 256 | 513.2 | 7 | 3.81 | 370/612 (60%) | +| 256 | 600.0 | 7 | 1.66 | 129/612 (21%) | +| 512 | 488.0 | 11 | 9.60 | 551/612 (90%) | +| 512 | 513.2 | 14 | 7.63 | 512/612 (84%) | +| 512 | 600.0 | 14 | 3.32 | 301/612 (49%) | + +Two things fall out, and the second is the one that costs something. + +1. **At every rate this tree has considered, most branch points arrive with less + lookahead than the one before them.** At 488 KB/s in a 256 KB ring — the + configuration `check.sh` gates — that is 76%. +2. **A bigger ring makes this metric worse, and now content says so too.** 51.3 + derived it from the surplus alone; here the same rate goes from 76% under the + climb at 256 KB to 90% at 512 KB, because doubling the ring doubles the + ceiling without touching `pipe - wire`. **The ring is not the lever. The + surplus is.** + +### 56.5 What a branch actually costs when the gap bought nothing + +The climb is what a player needs to *tolerate* the next branch. What it *pays* +at one is the prefill, because the ring is empty after a seek and 55.4's shipped +policy releases the decoder at 2 records: + +| pipe KB/s | 2-record prefill | as a scene change (+6,164 B) | +|---:|---:|---:| +| 451.4 | 161.8 ms (1.94 frame slots) | 175.2 ms (2.10) | +| 488.0 | 149.7 ms (1.80) | 162.0 ms (1.94) | + +So a zero-play branch is **not** a failure: it costs about two frame slots of +black, every time. What it removes is margin. A player that branches at p10 +(0.950 s) has spent its whole lookahead and rebuilt almost none of it, and the +next slow record has nothing behind it. **The finding is not "this breaks", it +is "this design runs permanently at minimum lookahead, and the arcade content is +what puts it there."** 22_scene_load.py prices the scene-change case properly, +clocks included; the mechanical seek is still unmodelled (B1) and is charged on +top of all of it. + +### 56.6 What the cross-check IS worth, now that it is not a cross-check + +Run anyway, against 518 SNES chapter XMLs, with the 40 scene abbreviations +matched to DirkSimple scenes **from the data** — the abbreviation's letters must +be a subsequence of the scene name, ranked by sequence-name overlap, with a +reversed-scene rule for the thirteen mirrored scenes. (Overlap alone mapped +`snkr` to `black_knight`, because two scenes full of `seqN` names look alike.) + +- **Start times do not compare at all.** The SNES XMLs are on their own + extraction's timeline; DirkSimple is on arcade ROM frames minus 6,297 ms. The + difference is not a constant and **does not even keep its sign** (spread + -8.0 s .. +6.4 s). Reported as a spread rather than as a check, because a + check it is not. +- **Durations do compare** — they are offset-invariant. 388/505 agree within one + laserdisc frame (76.8%), median difference 0 ms. +- **Branch structure compares**: 470/505 chapters (93.1%) carry the identical + set of (input -> target) edges. Of the 35 that differ, **16 are renames** + (`captured_by_ghouls` -> `ghoul_capture`) and **18 of the remaining 19 are the + SNES conversion dropping the arcade's diagonals**. The 19th is + `intr_castle_exterior`, where the SNES *added* a skip. + +**Zero transcription discrepancies were found, and none could have been.** What +the diff produced is one useful fact about our own input layer, below. + +### 56.7 Two constraints on the input layer, from the same import + +1. **The arcade uses eight directions plus action and start.** Counted over the + 906 windows: left 233, right 217, up 209, down 152, action 72, **upleft 13, + upright 4, downleft 3, downright 1**, start 2. The diagonals are 21 windows + out of 906 — rare enough to be dropped by a port that had to (the SNES one + did) and **not** droppable by one aiming at the arcade. +2. **The shortest input window is 98 ms.** Median 950 ms (11.4 frame slots), p10 + 393 ms, but the floor is 98 ms — and by 54.4 a 12 fps frame slot is 72.13 ms + or 90.16 ms, never 83.33. **A 98 ms window is one or two frames wide.** + Polling input on the frame tick is therefore marginal by construction: the + input layer has to run off something faster than the frame clock, and + `src/player/clock.i` already owns an MFP interrupt at 181.35 clocks a + V-DISP (54.2) that is 8.6x faster and costs 0.1% of the budget. + +### 56.8 What this does NOT establish + +1. **No rate here is measured.** Every column is a sensitivity across explicit + rates (FINDINGS 50). B1 is still open and the mechanical seek still has no + figure at all. +2. **The gap model is a lower bound by construction.** It takes the earliest + instant an input window opens, so it is what an *expert* player can force, + not what a typical one produces. That is the right bound for a buffer + design and the wrong one for describing play. +3. **It is the arcade's graph, not this port's.** The 612 transitions assume + the port reproduces every branch. Nothing has been mapped onto our 224 + Blu-ray streams yet — the SNES project's 516 chapters are finer-grained than + our streams, and that mapping is still C1's problem. +4. **Nothing ran on the 68000 this session.** This is host-side analysis of an + imported table. `decode.s`, `stream.s`, `ring.i`, `clock.i` and `load.i` are + untouched, `decode.bin` is still 1,296 B at the same MD5, and the green light + was ALL GREEN before and after. +5. **The scene graph is not vendored, and the coupling is contained.** Neither + repo ships here and neither is redistributable from this tree; both are + permissive (DirkSimple zlib, Ryan C. Gordon; SNES project MIT, Chad + Doebelin) and both are cloned by the reader. The `sources` block of every + generated table carries the attribution. **`tmp/scenegraph.json` is + generated, gitignored and derived data**: committing it, or any table built + from it, is redistribution and the attribution has to travel with it. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index db4c78e..6e3f2f3 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -3,6 +3,8 @@ Written end of session 19 (2026-08-24), against a tree that is ALL GREEN. Amended end of session 21: P1 done, P2 half done (FINDINGS 53). Amended end of session 22: P3 done (FINDINGS 54). +Amended end of session 23: P5 done (FINDINGS 55). +Amended end of session 24: G1 done (FINDINGS 56). **THE COMPLETION TARGET IS M3, THE VERTICAL SLICE** (USER DECISION): one scene tree — a decision point, two outcomes, a death clip — with audio, streaming from @@ -256,18 +258,29 @@ interaction to price next, and it is E2's question with a second consumer in it. **E6. Container v2** — audio interleave, per-record index, scene table. Depends on P6's answer and on P5's index. -**G1. Import the scene graph — early, because it is a measurement input.** -SNES project `data/events/` (MIT, cleared) diffed against DirkSimple (zlib), -which transcribed the same data independently, to catch transcription errors -before anything reaches 68000 tables. **Neither is on this box** — both need -fetching. +~~**G1. Import the scene graph — early, because it is a measurement input.**~~ +**DONE, session 24 — FINDINGS 56.** It was pulled ahead for exactly the reason +given, and it paid: **the worst gap between two consecutive decision points is +zero**, and 5.4% of the game's 612 branch transitions are. Two seeks can fall +back to back with no play between them, so 51.2's slack rule can be answered NO +by the content rather than by the buffer. -The reason to pull this ahead of the game logic that consumes it: 51.3 says -4.83 s of play to refill a 256 KB ring at 488 KB/s, and Dragon's Lair's decision -points are seconds apart. **Nothing in this tree can currently say what the worst -gap between consecutive decision points is** — only the scene table knows, and -until it is imported, whether this design survives a back-to-back branch is an -open question nobody is able to ask. +It does not break the design — a branch on an empty ring costs the prefill +(149.7 ms, 1.80 frame slots at 488 KB/s), not the climb — but it removes the +margin: at 488 KB/s in a 256 KB ring, **76% of this game's branch points arrive +before the ring has refilled**, and a 512 KB ring makes that 90%. **The ring is +not the lever; the surplus is.** + +Two constraints on the input layer came with it: the arcade needs **eight +directions**, and the shortest input window is **98 ms** against a 72.13/90.16 ms +frame slot, so input cannot be polled on the frame tick (56.7). + +**The cross-check plan was wrong and is struck.** The SNES chapters are +*derived* from DirkSimple, by their own README, so there is one transcription and +not two; the diff catches conversion errors only (56.2). **Nothing is vendored:** +`tools/import/scenegraph.py` is the one file coupled to those projects and it +writes this project's own `DLXSCENE1` schema into gitignored `tmp/` +(USER DECISION, session 24). --- @@ -304,7 +317,7 @@ B3 DTYP ──────┴─> P4 transport ─┐ P1 P2(half) P3 P5 P7 ───────────┘ ^ │ P6 (bus cost DONE, 52) ──────────────────┤ -G1 scene graph (fetch, do early) ─────────┘ +G1 scene graph (DONE, 56) ───────────────┘ B2 blanking ─> (page 1; do not pre-build on it) ``` diff --git a/docs/STATUS.md b/docs/STATUS.md index b2759df..fdc4f03 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1,3 +1,94 @@ +# Status & next-session handoff — end of session 24 (2026-08-24) + +## Session 24: the scene graph is in, and the worst gap between two decision points is zero + +**Green light first and last: `./tools/bench/check.sh` was ALL GREEN before any +of this and ALL GREEN after**, plus a new import stage. + +**ROADMAP G1 is DONE. FINDINGS 56.** G1 was scheduled early because it is a +*measurement input*, and it paid for that immediately. + +**1. THE ANSWER: the worst gap is zero.** Over 612 distinct transitions into a +seek, taking the earliest instant each input window opens and chaining play +across sequences the disc plays through without seeking: + +| | worst | p10 | median | p75 | p90 | best | +|---|---:|---:|---:|---:|---:|---:| +| seconds of play between two seeks | **0.000** | 0.950 | 3.473 | 5.800 | 9.548 | 82.497 | + +**33 of the 612 (5.4%) are zero** — an input window that opens at t=0 of a clip +the disc seeked to, so two seeks can fall back to back with no play at all +(`flaming_ropes.enter_room -> fall_to_death`: press right on the first frame and +you die). **51.2's slack rule can be answered NO by the content, not by the +buffer**, and no amount of ring is a defence. 56.3. + +**2. Most branch points arrive before the ring has refilled, at every rate this +tree has considered.** 51.3's climb against the game's own gaps, gate container: + +| ring KB | pipe KB/s | ceiling | climb s | branch points under the climb | +|---:|---:|---:|---:|---:| +| 256 | 451.4 | 3 | 20.83 | 601/612 (98%) | +| 256 | 488.0 | 7 | 6.11 | **468/612 (76%)** | +| 256 | 513.2 | 7 | 3.81 | 370/612 (60%) | +| 256 | 600.0 | 7 | 1.66 | 129/612 (21%) | +| 512 | 488.0 | 11 | 9.60 | **551/612 (90%)** | + +**A bigger ring makes this worse and now content says so too**: same rate, 76% +at 256 KB and 90% at 512 KB, because doubling the ceiling does not touch +`pipe - wire`. **The ring is not the lever; the surplus is.** 56.4. + +**3. It does not break — it removes margin.** A branch on an empty ring costs +the prefill, not the climb: **149.7 ms (1.80 frame slots) at 488 KB/s**, 162.0 ms +if it is a scene change carrying the 6,164-byte header. So the finding is not +"this fails", it is **"this design runs permanently at minimum lookahead, and +the arcade content is what puts it there"**. The mechanical seek is still +unmodelled (B1) and is charged on top. 56.5. + +**4. CORRECTION to FINDINGS 16: there is only one transcription.** 16 cleared +two permissively licensed sources and planned to diff them "to catch +transcription errors". The SNES project's own `data/events/README.md` says its +chapters are "derived from DirkSimple game data" — a second *copy*, not a second +transcription. The diff runs anyway and catches conversion errors: durations +agree 388/505 within one frame, branch structure 470/505, and of the 35 +differences **16 are renames and 18 of the other 19 are the SNES conversion +dropping the arcade's diagonals**. Zero transcription discrepancies were found +and none could have been. 56.2, 56.6. + +**5. Two constraints on the input layer, free with the import.** The arcade uses +**eight directions plus action and start** (diagonals are 21 of 906 windows — +droppable by a port that must, not by one aiming at the arcade), and the +**shortest input window is 98 ms** against a frame slot of 72.13 or 90.16 ms +(54.4). **Input cannot be polled on the frame tick**; `clock.i`'s V-DISP +interrupt already runs 8.6x faster at 0.1% of the budget. 56.7. + +**6. The coupling to outside source is contained to one file (USER DECISION).** +`tools/import/scenegraph.py` is the only file in this tree that knows those +projects exist — their paths, table names, timing formulas, constants — and it +writes **`DLXSCENE1`**, our own schema, into gitignored `tmp/`. +`tools/analysis/25_scene_graph.py` reads only that. Nothing is vendored, nothing +outside-derived is committed, and the generated table carries its own `sources` +attribution block. The split was made after the measurement and changed no +number in it. + +**New in the tree:** `tools/import/` (new directory, one file), +`tools/analysis/25_scene_graph.py`, and a `check.sh` stage that imports, gates +on 516 sequences / 906 input windows, and runs the analysis — skipped when there +is no checkout, like the px68k and IPL ROM stages. + +**No 68000 code ran and none changed.** `decode.bin` is still 1,296 B at the +same MD5. + +**Next:** **P4** (drive the MB89352, settle `W`) still decides the project and +still needs hardware or a MAME that models the SPC. What 56 changes about it: +the transport now has a *content* requirement as well as a rate one — it has to +survive a branch with an empty ring at zero notice, 5.4% of the time. **P2's +remaining half** (reserve index 0 as black) is unchanged and still bundled with +the two other re-encode-class questions from 55: the delivered-rate rate point +and 54.4's short slot. All three are still one re-encode plus one +re-measurement, and still want deciding together. + +--- + # Status & next-session handoff — end of session 23 (2026-08-24) ## Session 23: the 68000 fills its own ring, and the player's request loop turns out to cost more than the medium does diff --git a/tools/analysis/25_scene_graph.py b/tools/analysis/25_scene_graph.py new file mode 100644 index 0000000..ef1fc8a --- /dev/null +++ b/tools/analysis/25_scene_graph.py @@ -0,0 +1,363 @@ +"""The worst gap between two decision points, out of the scene graph (G1). + +FINDINGS 51.3 measured that a ring's lookahead is ACCUMULATED out of +`pipe - wire` and that a seek spends all of it, so what a branch point costs is +set by the rate and by the time since the last branch. 55.5 rehearsed a seek on +the machine and could not ask the question that matters, because nothing in this +tree knew where the branch points ARE: + + what is the WORST gap, in seconds of play, between two consecutive + decision points, and does the refill climb survive it? + +Only the arcade scene graph knows. This answers it in the currency 51.3 +established. + + python3 tools/import/scenegraph.py # writes tmp/scenegraph.json + python3 tools/analysis/25_scene_graph.py --kbps R [R ...] [--ring KB [KB ...]] + +`--kbps` is REQUIRED and takes no default, for the reason FINDINGS 50 gives. + +THIS FILE KNOWS NOTHING ABOUT WHERE THE TABLE CAME FROM, deliberately. It reads +`DLXSCENE1`, which is this project's own schema; `tools/import/scenegraph.py` is +the single file in the tree that knows anything about the outside projects the +table is built from, and it carries their attribution. Nothing is vendored. +""" +import sys, os, json, argparse, importlib.util + +sys.path.insert(0, "tools/encoder") + +TABLE = os.environ.get("DLX_SCENEGRAPH", "tmp/scenegraph.json") +LD_FPS = 23.976 # the medium's frame rate; one frame is the comparison floor + + +# ------------------------------------------------------------------- graph + +class Node: + """One (scene, sequence): a clip, its exits, and whether entering it seeks.""" + + def __init__(self, scene, name, seq): + self.scene, self.name, self.seq = scene, name, seq + self.start = seq["start_ms"] # ms, or -1 for no seek + self.seeks = self.start >= 0 + self.timeout_ms = seq["timeout_ms"] + self.exits = [(seq["timeout_ms"], "timeout", seq["timeout_next"])] + for a in seq["actions"]: + # The player may press as early as `from_ms`, so that is the least + # play this clip can deliver before the branch it leads to. + self.exits.append((a["from_ms"], "action", a["next"])) + + @property + def key(self): + return f"{self.scene}.{self.name}" + + +def build_graph(scenes): + nodes = {} + for scene, seqs in scenes.items(): + for name, seq in seqs.items(): + n = Node(scene, name, seq) + nodes[n.key] = n + return nodes + + +def worst_gap(nodes): + """Least play time, in ms, between one seek and the next. + + A seek is entering a sequence whose start is >= 0; a negative start means + the disc keeps playing, so play ACCUMULATES across such sequences and the + gap is a shortest path over them. Bellman-Ford rather than Dijkstra + because a zero-length timeout is common (`start_alive` chains) and the + graph has cycles; all weights are non-negative so it terminates. + + A null exit ends the scene: the player moves to another scene entirely, + which is a seek AND a container change (FINDINGS 53), so it counts as a + seek and is flagged. + """ + INF = float("inf") + dist = {k: (0.0 if n.seeks else INF) for k, n in nodes.items()} + for _ in range(len(nodes) + 1): + changed = False + for n in nodes.values(): + if dist[n.key] == INF: + continue + for elapsed, kind, tgt in n.exits: + if tgt is None: + continue + tk = f"{n.scene}.{tgt}" + if tk not in nodes: + continue + d = dist[n.key] + max(0.0, float(elapsed)) + if not nodes[tk].seeks and d < dist[tk] - 1e-9: + dist[tk], changed = d, True + if not changed: + break + + best = {} + for n in nodes.values(): + if dist[n.key] == INF: + continue + for elapsed, kind, tgt in n.exits: + tk = f"{n.scene}.{tgt}" if tgt is not None else None + ends_scene = tgt is None + if ends_scene or (tk in nodes and nodes[tk].seeks): + g = dist[n.key] + max(0.0, float(elapsed)) + # One entry per (source, destination): several input windows can + # lead to the same clip and only the earliest of them binds. + k = (n.key, tgt) + if k not in best or g < best[k][0]: + best[k] = (g, n.key, tgt if tgt else "", + kind, ends_scene, n.scene) + return sorted(best.values()), dist + + +# ------------------------------------------------------ 51.3's currency + +def slack_model(gap_s, kbps, wire_kbps, mean_rec_b): + """Records of lookahead accrued in `gap_s` seconds of play at `kbps`, and + the seconds one record of lookahead costs. + + This is 51.3's surplus model and nothing more: slack accrues at + (pipe - wire) bytes per second. The paced rig is the measurement; this + says whether the gap is even in the right order of magnitude. + """ + surplus = (kbps - wire_kbps) * 1024.0 + if surplus <= 0: + return 0.0, float("inf") + return surplus * gap_s / mean_rec_b, mean_rec_b / surplus + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--kbps", type=float, nargs="+", required=True, + help="delivered pipe rates, KB/s. REQUIRED, no default " + "(FINDINGS 50).") + ap.add_argument("--table", default=TABLE, help="DLXSCENE1 scene table") + ap.add_argument("--container", default="tmp/rc_fr_singe_scsi_span.dlx", + help="container the wire demand and mean record come from") + ap.add_argument("--ring", type=float, nargs="+", default=[256, 512]) + ap.add_argument("--top", type=int, default=12) + a = ap.parse_args() + + if not os.path.exists(a.table): + print(f"no scene table at {a.table} -- run:\n" + f" python3 tools/import/scenegraph.py") + return 2 + doc = json.load(open(a.table)) + if doc.get("format") != "DLXSCENE1": + print(f"{a.table}: not a DLXSCENE1 table") + return 2 + + nodes = build_graph(doc["scenes"]) + c = doc["counts"] + nseek = sum(1 for n in nodes.values() if n.seeks) + + print(f"=== the scene graph ({a.table})") + for s in doc["sources"]: + print(f" from {s['name']} ({s['licence']}, {s['holder']}): {s['role']}") + print(f" scenes {c['scenes']}, sequences {c['sequences']}, " + f"input windows {c['windows']}") + print(f" sequences entered by a SEEK: {nseek}/{c['sequences']} " + f"({100*nseek/c['sequences']:.1f}%); the rest play on from where the " + f"disc is") + print(f" scene order: {len(doc['rows'])} rows x {len(doc['rows'][0])}") + # Gates. A parser that quietly dropped a branch would produce a SMALLER + # graph and a LONGER worst gap -- it would fail in the flattering direction. + assert c["sequences"] == 516, f"expected 516 sequences, got {c['sequences']}" + assert c["windows"] == 906, f"expected 906 input windows, got {c['windows']}" + + # ---- the measurement + gaps, dist = worst_gap(nodes) + play = [g for g in gaps if g[5] != "attract_mode"] + zero = [g for g in play if g[0] <= 1e-9] + print() + print("=== the worst gap between two consecutive decision points") + print(" A 'gap' is the LEAST play time the disc delivers between one seek") + print(" and the next: the earliest an input window opens, chained across") + print(" sequences the disc plays through without seeking. One entry per") + print(" (source, destination); attract mode is excluded and reported") + print(" separately, because nothing branches there under a 12 fps budget.") + print(f" {'gap s':>7} from -> to") + for g, src, tgt, kind, ends, scene in play[:a.top]: + print(f" {g/1000:>7.3f} {src} -{kind}-> {tgt}" + + (" [SCENE CHANGE]" if ends else "")) + sc_gaps = [g for g in play if g[4]] + print(f" ... {len(play)} distinct transitions into a seek " + f"({len(gaps)-len(play)} more in attract mode)") + worst = play[0][0] / 1000.0 + med = play[len(play) // 2][0] / 1000.0 + print(f" WORST {worst:.3f} s, median {med:.3f} s, " + f"best {play[-1][0]/1000:.3f} s") + print(f" SCENE CHANGES specifically ({len(sc_gaps)} of them, and each also") + print(f" needs a header before its frame 0, FINDINGS 53/55.1): worst " + f"{sc_gaps[0][0]/1000:.3f} s, median " + f"{sc_gaps[len(sc_gaps)//2][0]/1000:.3f} s") + print(f" ZERO-PLAY BRANCHES: {len(zero)} of {len(play)} " + f"({100*len(zero)/len(play):.1f}%) open an input window at t=0 of a") + print(" clip the disc SEEKED to, so two seeks can fall back to back with no") + print(" play between them at all. A rule of the form 'has there been") + print(" enough play since the last branch' (51.2's ring_may_seek) can be") + print(" answered NO by the content, not by the buffer.") + + # ---- what the input layer has to survive, from the same table + inputs, windows = {}, [] + for n in nodes.values(): + for x in n.seq["actions"]: + inputs[x["input"]] = inputs.get(x["input"], 0) + 1 + windows.append(x["to_ms"] - x["from_ms"]) + windows.sort() + print() + print("=== what the input layer has to survive") + print(" " + ", ".join(f"{k} {v}" for k, v in + sorted(inputs.items(), key=lambda x: -x[1]))) + diag = sum(v for k, v in inputs.items() + if k in ("upleft", "upright", "downleft", "downright")) + print(f" diagonals are {diag} windows of {len(windows)}: rare enough for a " + f"port to drop\n and not droppable by one aiming at the arcade") + print(f" window length: shortest {windows[0]:.0f} ms " + f"({windows[0]/(1000/12):.2f} frame slots at 12 fps), p10 " + f"{windows[len(windows)//10]:.0f} ms, median " + f"{windows[len(windows)//2]:.0f} ms") + print(" the floor is one to two frames wide (54.4: a slot is 72.13 or") + print(" 90.16 ms, never 83.33), so input cannot be polled on the frame tick") + + # ---- 51.3's currency + if os.path.exists(a.container): + spec = importlib.util.spec_from_file_location( + "seek_slack", "tools/analysis/20_seek_slack.py") + m = importlib.util.module_from_spec(spec) + spec.loader.exec_module(m) + import ratectl as RC + d, rec = m.records(a.container) + mean_rec = float(rec.mean()) + wire = mean_rec * 12 / 1024 + RC.AUDIO_KBPS + gs = [g[0] / 1000.0 for g in play] + print() + print(f"=== what that gap buys, at explicit rates " + f"({os.path.basename(a.container)}: mean record " + f"{mean_rec/1024:.1f} KB, wire {wire:.1f} KB/s)") + print(f" {'ring KB':>8} {'pipe':>7} {'ceiling':>8} {'climb s':>8} " + f"{'median gap':>11} {'accrued':>8} {'under climb':>12}") + for ring_kb in a.ring: + ring = int(ring_kb * 1024) + for kbps in a.kbps: + fill = (kbps - RC.AUDIO_KBPS) * 1024 / 12 + lo, hi, ring_ref, rate_ref = m.paced_sim(rec, ring, fill) + ceiling = int(hi.max()) + accrued, per_rec = slack_model(med, kbps, wire, mean_rec) + climb = ceiling * per_rec + under = sum(1 for g in gs if g < climb) + print(f" {ring_kb:>8.0f} {kbps:>7.1f} {ceiling:>8} " + f"{climb:>8.2f} {med:>11.3f} {accrued:>8.2f} " + f"{f'{under}/{len(gs)}':>12} {100*under/len(gs):.0f}%") + + # What a branch costs when the gap before it bought nothing. The ring + # is empty after a seek and the decoder is released at the prefill depth + # (55.4's shipped policy is 2 records), so this is the stall the player + # eats every time -- not the climb to the ceiling, which is what it + # needs in order to TOLERATE the next one. A scene change additionally + # needs its header before frame 0; 22_scene_load.py prices that case + # properly, clocks included. + PREFILL_REC, HDR_B = 2, 6164 + print() + print(f" A branch taken on an empty ring, at the shipped prefill of " + f"{PREFILL_REC} records:") + for kbps in a.kbps: + b = PREFILL_REC * mean_rec + ms = b / (kbps * 1024) * 1000 + hms = (b + HDR_B) / (kbps * 1024) * 1000 + print(f" {kbps:>7.1f} KB/s: {ms:>7.1f} ms " + f"({ms/(1000/12):.2f} frame slots), and {hms:>7.1f} ms " + f"({hms/(1000/12):.2f}) if it is a scene change carrying " + f"{HDR_B:,} header bytes") + print() + print(" 'climb s' is 51.3's: seconds of play to refill from empty to") + print(" the ceiling. 'under climb' is how many of this game's own") + print(" branch points arrive sooner than that, i.e. are reached with") + print(" LESS lookahead than the one before them. The worst gap is") + print(f" {worst:.3f} s and buys nothing at any rate in this table.") + else: + print(f"\n{a.container}: MISSING -- rate half skipped") + + # ---- the cross-check, and what it is worth + print() + print("=== the second table, and why it is not a second transcription") + cross, meta = doc.get("crosscheck"), doc.get("crosscheck_meta") + if not cross: + print(" none in this table -- the import ran without it") + return 0 + print(f" {meta['chapters']} chapters, {meta['scenes_mapped']} scenes") + print(f" PROVENANCE: {meta['provenance']}") + print(" FINDINGS 16 planned to diff two INDEPENDENT transcriptions to") + print(" catch transcription errors. There is only one transcription.") + print(" This diff catches CONVERSION errors and nothing more.") + + FRAME_MS = 1000.0 / LD_FPS + offs, durs, same, diff, missing = [], [], 0, 0, 0 + renames, inputs_differ, examples = 0, [], [] + for scene, seqs in sorted(cross.items()): + for seq, ch in sorted(seqs.items()): + n = nodes.get(f"{scene}.{seq}") + if n is None: + missing += 1 + continue + if n.seeks: + offs.append(ch["start_ms"] - n.start) + durs.append(((ch["end_ms"] - ch["start_ms"]) - n.timeout_ms, + ch["chapter"])) + ce = sorted((x["input"], x["next"]) for x in ch["actions"]) + de = sorted((x["input"], x["next"]) for x in n.seq["actions"]) + if ce == de: + same += 1 + continue + diff += 1 + ci = sorted(i for i, _ in ce) + di = sorted(i for i, _ in de) + if ci != di: + inputs_differ.append((ch["chapter"], ci, di)) + else: + renames += 1 + if len(examples) < 3: + examples.append((ch["chapter"], ce, de)) + + if offs: + offs.sort() + print(f" START TIMES are on different timelines and do not compare: " + f"{len(offs)} seeking chapters,") + print(f" offset spread {min(offs)/1000:,.1f} s .. " + f"{max(offs)/1000:,.1f} s, median {offs[len(offs)//2]/1000:,.1f} s" + f" -- not a constant, and not even one sign.") + if durs: + dd = sorted(x for x, _ in durs) + agree = sum(1 for x in dd if abs(x) <= FRAME_MS) + print(f" DURATIONS compare (offset-invariant): {agree}/{len(dd)} " + f"within one frame of the medium ({100*agree/len(dd):.1f}%), " + f"median {dd[len(dd)//2]:,.0f} ms") + for x, ch in sorted(durs, key=lambda x: -abs(x[0]))[:3]: + print(f" widest {ch}: {x/1000:+.3f} s") + if same + diff: + print(f" BRANCH STRUCTURE compares: {same}/{same+diff} chapters have " + f"the identical set of (input -> target) edges " + f"({100*same/(same+diff):.1f}%)") + if diff: + print(f" {renames} of the {diff} differ only in what a target " + f"sequence is NAMED") + DIAG = ("upleft", "upright", "downleft", "downright") + lost = [ch for ch, ci, di in inputs_differ + if set(di) - set(ci) and all(x in DIAG for x in set(di) - set(ci))] + print(f" {len(inputs_differ)} differ in the INPUT SET, and " + f"{len(lost)} of those are the other table dropping the arcade's") + print(" DIAGONALS: a controller decision, not a transcription " + "difference.") + for ch, ci, di in inputs_differ: + if ch not in lost: + print(f" the remaining one: {ch}, {ci} vs {di}") + for ch, ce, de in examples: + print(f" e.g. {ch}\n cross {ce}\n graph {de}") + if missing: + print(f" {missing} chapters have no sequence in the graph " + f"(the other project added them)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/bench/check.sh b/tools/bench/check.sh index ecf498a..d265172 100755 --- a/tools/bench/check.sh +++ b/tools/bench/check.sh @@ -406,4 +406,30 @@ grep -q "^OK" tmp/ringseek_check.log || { echo "FAIL: the decode after the seek was not pixel-exact." tail -4 tmp/ringseek_check.log; exit 1; } +echo "--- session 24: the scene graph, and the gap between branch points (FINDINGS 56) ---" +# The arcade scene graph is not in this repo and is not redistributable from +# here. tools/import/scenegraph.py is the ONE file in the tree that knows the +# outside projects exist; it writes tmp/scenegraph.json in this project's own +# DLXSCENE1 schema and everything downstream reads only that. +# What is gated is the IMPORT, not the numbers: 516 sequences and 906 input +# windows, and the four timing helpers still being the formulas the importer +# evaluates. Skipped when the checkout is absent. +DIRKSIMPLE=${DLX_DIRKSIMPLE:-tmp/scenegraph/DirkSimple} +if [ -f "$DIRKSIMPLE/data/games/lair/game.lua" ]; then + DLX_DIRKSIMPLE="$DIRKSIMPLE" python3 tools/import/scenegraph.py \ + -o tmp/scenegraph.json > tmp/scenegraph_import.log 2>&1 \ + || { cat tmp/scenegraph_import.log; exit 1; } + sed "s/^/ /" tmp/scenegraph_import.log + grep -q "516 sequences, 906 input windows" tmp/scenegraph_import.log || { + echo "FAIL: the scene graph did not import to 516/906 -- upstream changed," + echo " or the parser silently dropped branches."; exit 1; } + python3 tools/analysis/25_scene_graph.py --kbps 488 --ring 256 \ + > tmp/scenegraph_check.log 2>&1 \ + || { tail -20 tmp/scenegraph_check.log; exit 1; } + grep -aE "^ WORST |^ ZERO-PLAY|^ BRANCH STRUCTURE" tmp/scenegraph_check.log +else + echo " SKIPPED: no DirkSimple checkout at $DIRKSIMPLE" + echo " (git clone --depth 1 https://github.com/icculus/DirkSimple)" +fi + echo "ALL GREEN" diff --git a/tools/import/scenegraph.py b/tools/import/scenegraph.py new file mode 100644 index 0000000..e508dc5 --- /dev/null +++ b/tools/import/scenegraph.py @@ -0,0 +1,436 @@ +"""The ONE file in this tree that knows anything about somebody else's source. + +Everything downstream -- `tools/analysis/25_scene_graph.py` and whatever the +game-logic layer eventually becomes -- reads the neutral table this writes and +knows nothing about where it came from. That is deliberate. Two outside +projects are read here, neither is vendored, neither ships in this repo, and +their file layouts, table names, timing formulas and magic constants are wired +into THIS file and nowhere else. When one of them moves, one file breaks. + + python3 tools/import/scenegraph.py [-o tmp/scenegraph.json] + +Sources, both permissively licensed and both cloned by the reader: + + icculus/DirkSimple -- zlib, Ryan C. Gordon. `data/games/lair/game.lua` holds + the arcade scene graph, transcribed from the arcade ROM's own data table. + THE source; the measurement is made on this. + DLX_DIRKSIMPLE=/path (default tmp/scenegraph/DirkSimple) + + astrobleem/SNES-SuperDragonsLairArcade -- MIT, Chad Doebelin. + `data/events/*.xml` holds 516 chapter definitions. Read as a CROSS-CHECK + only, and a weak one: their own README states the chapters are "derived + from DirkSimple game data", so this is a second COPY, not a second + transcription (FINDINGS 56.2). Optional; skipped when absent. + DLX_SNESLAIR=/path (default tmp/scenegraph/SNES-SuperDragonsLairArcade) + +WHAT COMES OUT is `DLXSCENE1`, our own schema, in one JSON file: + + {"format": "DLXSCENE1", + "sources": [{"name", "url", "licence", "holder", "role"} ...], + "scenes": {scene: {sequence: { + "start_ms": float, absolute position on the medium, or -1 for + "keep playing from where the disc already is", + "timeout_ms": float, how long the clip runs unbranched, + "kills", "single_frame": bool, + "timeout_next": sequence name or null, + "actions": [{"input", "from_ms", "to_ms", "next"}]}}}, + "rows": [[scene, scene, scene] ...], the arcade's 13x3 scene order + "counts": {"scenes", "sequences", "windows"}, + "crosscheck": {scene: {sequence: {"start_ms", "end_ms", "actions": [...], + "chapter"}}} or null} + +Nothing in the schema is a copy of either project's structure: it is what this +project's own analysis needs, which is a clip's length, its exits and the +earliest instant each of them can fire. + +If a scene table is ever COMMITTED to this repo rather than regenerated into +gitignored tmp/, it becomes redistribution of derived data and the attribution +in "sources" has to travel with it. FINDINGS 16, 56.5. +""" +import sys, os, re, json, argparse +import xml.etree.ElementTree as ET + +DIRK = os.environ.get("DLX_DIRKSIMPLE", "tmp/scenegraph/DirkSimple") +SNES = os.environ.get("DLX_SNESLAIR", "tmp/scenegraph/SNES-SuperDragonsLairArcade") + +# DirkSimple's own constants, from data/games/lair/game.lua. Restated here +# because this file evaluates its table; both are gated below against the text +# of the file they came from, so they cannot go stale silently. +LD_FPS = 23.976 # laserdisc_frame_to_ms +ROM_OFFSET_MS = 6297.0 # time_laserdisc_frame's "magic millisecond offset" + +SOURCES = [ + {"name": "icculus/DirkSimple", + "url": "https://github.com/icculus/DirkSimple", + "licence": "zlib", "holder": "Ryan C. Gordon", + "role": "the scene graph itself, transcribed from the arcade ROM"}, + {"name": "astrobleem/SNES-SuperDragonsLairArcade", + "url": "https://github.com/astrobleem/SNES-SuperDragonsLairArcade", + "licence": "MIT", "holder": "Chad Doebelin", + "role": "cross-check only; itself derived from DirkSimple (FINDINGS 56.2)"}, +] + +# ---------------------------------------------------------------- Lua subset + +class LuaParser: + """Just enough Lua to read game.lua's `scenes` table constructor. + + Not a Lua interpreter and not trying to be: the grammar is table + constructors, string/number/boolean/nil literals, bare identifiers (used + only for function references like `interrupt=game_over_complete`), calls to + four known helpers, and + / - between them. Anything else is a parse error + rather than a silent skip, and the caller asserts the whole table was + consumed, so a change upstream shows up as a failure and not as a smaller + scene graph. + """ + + def __init__(self, text, helpers): + self.s, self.i, self.helpers = text, 0, helpers + + def error(self, msg): + line = self.s.count("\n", 0, self.i) + 1 + raise SyntaxError(f"{msg} at line {line}: {self.s[self.i:self.i+60]!r}") + + def ws(self): + while self.i < len(self.s): + c = self.s[self.i] + if c in " \t\r\n": + self.i += 1 + elif self.s.startswith("--", self.i): + self.i = self.s.find("\n", self.i) + if self.i < 0: + self.i = len(self.s) + else: + return + + def take(self, lit): + self.ws() + if self.s.startswith(lit, self.i): + self.i += len(lit) + return True + return False + + def expect(self, lit): + if not self.take(lit): + self.error(f"expected {lit!r}") + + def name(self): + self.ws() + m = re.compile(r"[A-Za-z_][A-Za-z0-9_]*").match(self.s, self.i) + if not m: + return None + self.i = m.end() + return m.group(0) + + def value(self): + self.ws() + if self.take("{"): + return self.table() + m = re.compile(r"-?\d+(\.\d+)?").match(self.s, self.i) + if m: + self.i = m.end() + v = float(m.group(0)) + return self.arith(int(v) if v.is_integer() else v) + if self.s[self.i] in "\"'": + q = self.s[self.i] + j = self.s.index(q, self.i + 1) + v = self.s[self.i + 1:j] + self.i = j + 1 + return v + n = self.name() + if n is None: + self.error("expected a value") + if n == "nil": + return None + if n in ("true", "false"): + return n == "true" + if self.take("("): # a call + args = [] + if not self.take(")"): + while True: + args.append(self.value()) + if self.take(")"): + break + self.expect(",") + if n not in self.helpers: + self.error(f"unknown helper {n!r}") + return self.arith(self.helpers[n](*args)) + return Symbol(n) # a function reference + + def arith(self, left): + """+ and - between numbers. Present in the source and load-bearing: + e.g. `time_laserdisc_frame(1823) - laserdisc_frame_to_ms(2)`.""" + while True: + self.ws() + if self.i < len(self.s) and self.s[self.i] in "+-": + op = self.s[self.i] + self.i += 1 + right = self.value() + left = left + right if op == "+" else left - right + else: + return left + + def table(self): + d, arr = {}, [] + while True: + self.ws() + if self.take("}"): + break + save = self.i + k = self.name() + if k is not None and self.take("="): + d[k] = self.value() + else: + self.i = save + arr.append(self.value()) + if not (self.take(",") or self.take(";")): + self.expect("}") + break + if d and arr: + self.error("mixed array/hash table") + return arr if arr or not d else d + + +class Symbol(str): + """A bare Lua identifier used as a value (a function reference).""" + + +def load_dirksimple(path): + """Parse lair/game.lua's `scenes` and `scene_manager` tables. + + Gates the four timing helpers against the text that defines them, so this + tool cannot keep evaluating a formula upstream has changed. + """ + src = open(os.path.join(path, "data/games/lair/game.lua"), encoding="utf-8").read() + + def gate(pat, what): + if not re.search(pat, src): + raise AssertionError(f"DirkSimple's {what} is not what this tool " + f"evaluates any more (looked for {pat!r})") + gate(r"frame\s*/\s*23\.976", "laserdisc_frame_to_ms") + gate(r"-\s*6297\.0", "time_laserdisc_frame's ROM offset") + gate(r"time_laserdisc_noseek.*?\n\s*return -1", "time_laserdisc_noseek") + gate(r"\(seconds \* 1000\) \+ ms", "time_to_ms") + + helpers = { + "laserdisc_frame_to_ms": lambda f: (f / LD_FPS) * 1000.0, + "time_laserdisc_frame": lambda f: (f / LD_FPS) * 1000.0 - ROM_OFFSET_MS, + "time_laserdisc_noseek": lambda: -1, + # time_to_ms takes (seconds, ms); the source calls it with a third + # argument exactly once, which Lua discards. + "time_to_ms": lambda s, ms=0, *_: s * 1000 + ms, + } + + out = {} + for var in ("scenes", "scene_manager"): + m = re.search(rf"^{var} = \{{", src, re.M) + if not m: + raise AssertionError(f"{var} table not found in game.lua") + p = LuaParser(src, helpers) + p.i = m.end() + out[var] = p.table() + # The parser must have consumed the WHOLE constructor: table() returns + # with the cursor just past the matching close brace, and in this file + # every top-level table ends on a `}` in column 0. A parser that + # stopped early would return a smaller graph, which is the direction + # that flatters the measurement, so this is checked rather than assumed. + if src[p.i - 1] != "}": + raise AssertionError(f"{var}: parser did not end on a close brace") + if src.rfind("\n", 0, p.i) != p.i - 2: + raise AssertionError(f"{var}: the constructor ended mid-line at " + f"{p.i}, so the parse stopped early") + return out["scenes"], out["scene_manager"] + + +# ------------------------------------------------- the SNES cross-check + +def load_snes(path): + """Parse the SNES project's chapter XMLs into {chapter: (start_ms, end_ms, + [(input, from_ms, to_ms, target)])}, plus the provenance line.""" + ev = os.path.join(path, "data/events") + readme = os.path.join(ev, "README.md") + prov = None + if os.path.exists(readme): + txt = open(readme, encoding="utf-8").read() + if "derived from DirkSimple" in txt: + prov = "derived from DirkSimple game data (data/events/README.md)" + + def ms(e): + return (int(e.get("min", 0)) * 60000 + int(e.get("second", 0)) * 1000 + + int(e.get("ms", 0))) + + chapters = {} + for f in sorted(os.listdir(ev)): + if not f.endswith(".xml"): + continue + root = ET.parse(os.path.join(ev, f)).getroot() + tl = root.find("timeline") + s = tl.find("timestart") + e = tl.find("timeend") + acts = [] + for evt in root.findall("./events/event"): + if evt.get("type") != "direction": + continue + t = evt.find("timeline") + p = evt.find("./params/str[@key='type']") + r = evt.find("./result/playchapter") + acts.append((p.get("value") if p is not None else "?", + ms(t.find("timestart")), + ms(t.find("timeend")) if t.find("timeend") is not None else -1, + r.get("name") if r is not None else None)) + chapters[f[:-4]] = (ms(s) if s is not None else -1, + ms(e) if e is not None else -1, acts) + return chapters, prov + + +def map_abbrevs(chapters, scenes): + """Match each SNES scene abbreviation to a DirkSimple scene from the DATA, + not from a hand-written table, so a wrong match is visible rather than + assumed. Three signals, in order: + + 1. the abbreviation's letters must be a SUBSEQUENCE of the scene name + (`snkr` is snake_room, and cannot be black_knight however similar + their `seqN` names are -- overlap alone got that one wrong); + 2. the Jaccard overlap of the two sequence-name sets; + 3. the reversed-scene rule: Dragon's Lair mirrors thirteen scenes, and + where BOTH `x` and `xr` exist as abbreviations, `xr` is the + `_reversed` scene and `x` is not. Their sequence names are identical, + so nothing else separates them. + + Assignment is 1:1 and greedy on that ranking. + """ + def subseq(a, b): + it = iter(b) + return all(c in it for c in a) + + by_abbr = {} + for ch in chapters: + abbr, _, seq = ch.partition("_") + by_abbr.setdefault(abbr, set()).add(seq) + + pairs = [] + for abbr, seqs in by_abbr.items(): + mirror = abbr.endswith("r") and abbr[:-1] in by_abbr + for scene, s in scenes.items(): + if not subseq(abbr, scene.replace("_", "")): + continue + union = len(seqs | set(s)) + j = len(seqs & set(s)) / union if union else 0.0 + rev = scene.endswith("_reversed") + pairs.append((j, 1 if rev == mirror else 0, -len(scene), + abbr, scene, len(seqs))) + pairs.sort(reverse=True) + + mapping, used = {}, set() + for j, _, _, abbr, scene, n in pairs: + if abbr in mapping or scene in used: + continue + mapping[abbr] = (scene, j, n) + used.add(scene) + unmatched = [(a, None, 0.0, len(s)) for a, s in by_abbr.items() + if a not in mapping] + return mapping, unmatched + + + + +# ------------------------------------------------------------------- emit + +def to_neutral(scenes, mgr): + """DirkSimple's tables -> DLXSCENE1. The one place their shape is read.""" + out = {} + for scene, seqs in scenes.items(): + s = out.setdefault(scene, {}) + for name, seq in seqs.items(): + t = seq["timeout"] + s[name] = { + "start_ms": float(seq["start_time"]), + "timeout_ms": float(t["when"]), + "timeout_next": t.get("nextsequence"), + "kills": bool(seq.get("kills_player")), + "single_frame": bool(seq.get("is_single_frame")), + "actions": [{"input": a["input"], + "from_ms": float(a["from"]), + "to_ms": float(a["to"]), + "next": a.get("nextsequence")} + for a in seq.get("actions", [])], + } + rows = [list(r) for r in mgr["rows"]] + return out, rows + + +def crosscheck_neutral(chapters, scenes): + """The SNES chapters, keyed the way OUR table is keyed. + + Their `abbr_sequence` naming is resolved to a scene here, from the data + (see map_abbrevs), so nothing downstream has to know an abbreviation + exists. Chapters the mapping cannot place are dropped and counted. + """ + mapping, unmatched = map_abbrevs(chapters, scenes) + out, dropped = {}, 0 + for ch, (s_ms, e_ms, acts) in chapters.items(): + abbr, _, seq = ch.partition("_") + if abbr not in mapping: + dropped += 1 + continue + scene = mapping[abbr][0] + pre = abbr + "_" + out.setdefault(scene, {})[seq] = { + "chapter": ch, + "start_ms": float(s_ms), + "end_ms": float(e_ms), + "actions": [{"input": i, "from_ms": float(f), "to_ms": float(t2), + "next": (n[len(pre):] if n and n.startswith(pre) else n)} + for i, f, t2, n in acts], + } + weak = sorted((round(j, 3), ab, sc) for ab, (sc, j, _n) in mapping.items() + if j < 1.0) + return out, {"chapters": len(chapters), "dropped": dropped, + "scenes_mapped": len(mapping), + "weakest_mappings": weak[:6]} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("-o", "--out", default="tmp/scenegraph.json") + a = ap.parse_args() + + if not os.path.isdir(DIRK): + sys.exit(f"no DirkSimple checkout at {DIRK} -- set DLX_DIRKSIMPLE, or\n" + f" git clone --depth 1 https://github.com/icculus/DirkSimple") + scenes, mgr = load_dirksimple(DIRK) + neutral, rows = to_neutral(scenes, mgr) + nseq = sum(len(s) for s in neutral.values()) + nwin = sum(len(q["actions"]) for s in neutral.values() for q in s.values()) + + cross, cross_meta = None, None + if os.path.isdir(SNES): + chapters, prov = load_snes(SNES) + cross, cross_meta = crosscheck_neutral(chapters, scenes) + cross_meta["provenance"] = prov + # 56.2: this is the sentence the whole "second source" claim turned on. + # If upstream ever removes it, say so rather than silently promoting a + # copy back to an independent transcription. + if not prov: + cross_meta["provenance"] = ("UNSTATED -- data/events/README.md no " + "longer says where the chapters came " + "from; re-check before trusting this " + "as anything.") + + doc = {"format": "DLXSCENE1", "sources": SOURCES, + "scenes": neutral, "rows": rows, + "counts": {"scenes": len(neutral), "sequences": nseq, + "windows": nwin}, + "crosscheck": cross, "crosscheck_meta": cross_meta} + os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True) + with open(a.out, "w") as f: + json.dump(doc, f, indent=1, sort_keys=True) + print(f"{a.out}: {len(neutral)} scenes, {nseq} sequences, {nwin} input " + f"windows, {len(rows)} rows" + + (f", cross-check {cross_meta['chapters']} chapters" + if cross_meta else ", no cross-check (SNES tree absent)")) + return 0 + + +if __name__ == "__main__": + sys.exit(main())