Import the scene graph, and find the worst gap between two branch points is zero
ROADMAP G1, scheduled early because it is a measurement input, and it paid for that immediately. 51.3 established that a ring's lookahead is accumulated out of pipe - wire and that a seek spends all of it, so what a branch point costs is set by the time since the last one. 55.5 rehearsed a seek on the machine and said out loud that it could not ask the question, because nothing here knew where the branch points are. They are everywhere, and 5.4% of them are free of charge to the player and not to us. Over 612 distinct transitions into a seek, taking the earliest instant each input window opens: worst 0.000 s, p10 0.950, median 3.473, best 82.497. 33 open on the first frame of a clip the disc SEEKED to -- press right as flaming_ropes.enter_room appears and you are in fall_to_death, two seeks with no play between them. 51.2's slack rule can therefore be answered NO by the content rather than by the buffer, and no amount of ring is a defence. It does not break the design. A branch on an empty ring costs the 2-record prefill, 149.7 ms at 488 KB/s, not the climb. What it removes is margin: at that rate in a 256 KB ring, 76% of this game's branch points arrive before the ring has refilled, and a 512 KB ring makes that 90%, because doubling the ceiling does not touch the surplus. The ring is not the lever; the surplus is. CORRECTION to FINDINGS 16: there is only one transcription. The SNES chapter set says in its own README that it is derived from DirkSimple, so the planned diff of two independent sources catches conversion errors only. Run anyway: durations agree 388/505 within a frame, branch structure 470/505, and of the 35 differences 16 are renames and 18 of the other 19 are that port dropping the arcade's diagonals. Zero transcription discrepancies, and none were findable. Two constraints on the input layer come free: the arcade needs eight directions, and the shortest input window is 98 ms against 54.4's 72.13/90.16 ms frame slot, so input cannot be polled on the frame tick. The coupling to outside source is contained to one file (USER DECISION). tools/import/scenegraph.py is the only code here that knows those projects exist -- their paths, table names, timing formulas, constants -- and it writes DLXSCENE1, this project's own schema, into gitignored tmp/ with the sources' licences inside it. tools/analysis/25_scene_graph.py reads only that. Nothing is vendored and nothing outside-derived is committed. The split was made after the measurement and the whole output was re-run byte for byte to show it moved no number. Both import gates are negative-tested: deleting one sequence upstream fails the 516/906 count, and closing the table early fails the constructor-end check, which replaced one that was vacuous. No 68000 code ran or changed; decode.bin is still 1,296 B at the same MD5. check.sh gains an import stage that skips when there is no checkout. ALL GREEN before and after. Claude-Session: https://claude.ai/code/session_01194oWYW8DQXK1SZ2DnChW6
This commit is contained in:
prosolis
@@ -0,0 +1,363 @@
|
||||
"""The worst gap between two decision points, out of the scene graph (G1).
|
||||
|
||||
FINDINGS 51.3 measured that a ring's lookahead is ACCUMULATED out of
|
||||
`pipe - wire` and that a seek spends all of it, so what a branch point costs is
|
||||
set by the rate and by the time since the last branch. 55.5 rehearsed a seek on
|
||||
the machine and could not ask the question that matters, because nothing in this
|
||||
tree knew where the branch points ARE:
|
||||
|
||||
what is the WORST gap, in seconds of play, between two consecutive
|
||||
decision points, and does the refill climb survive it?
|
||||
|
||||
Only the arcade scene graph knows. This answers it in the currency 51.3
|
||||
established.
|
||||
|
||||
python3 tools/import/scenegraph.py # writes tmp/scenegraph.json
|
||||
python3 tools/analysis/25_scene_graph.py --kbps R [R ...] [--ring KB [KB ...]]
|
||||
|
||||
`--kbps` is REQUIRED and takes no default, for the reason FINDINGS 50 gives.
|
||||
|
||||
THIS FILE KNOWS NOTHING ABOUT WHERE THE TABLE CAME FROM, deliberately. It reads
|
||||
`DLXSCENE1`, which is this project's own schema; `tools/import/scenegraph.py` is
|
||||
the single file in the tree that knows anything about the outside projects the
|
||||
table is built from, and it carries their attribution. Nothing is vendored.
|
||||
"""
|
||||
import sys, os, json, argparse, importlib.util
|
||||
|
||||
sys.path.insert(0, "tools/encoder")
|
||||
|
||||
TABLE = os.environ.get("DLX_SCENEGRAPH", "tmp/scenegraph.json")
|
||||
LD_FPS = 23.976 # the medium's frame rate; one frame is the comparison floor
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- graph
|
||||
|
||||
class Node:
|
||||
"""One (scene, sequence): a clip, its exits, and whether entering it seeks."""
|
||||
|
||||
def __init__(self, scene, name, seq):
|
||||
self.scene, self.name, self.seq = scene, name, seq
|
||||
self.start = seq["start_ms"] # ms, or -1 for no seek
|
||||
self.seeks = self.start >= 0
|
||||
self.timeout_ms = seq["timeout_ms"]
|
||||
self.exits = [(seq["timeout_ms"], "timeout", seq["timeout_next"])]
|
||||
for a in seq["actions"]:
|
||||
# The player may press as early as `from_ms`, so that is the least
|
||||
# play this clip can deliver before the branch it leads to.
|
||||
self.exits.append((a["from_ms"], "action", a["next"]))
|
||||
|
||||
@property
|
||||
def key(self):
|
||||
return f"{self.scene}.{self.name}"
|
||||
|
||||
|
||||
def build_graph(scenes):
|
||||
nodes = {}
|
||||
for scene, seqs in scenes.items():
|
||||
for name, seq in seqs.items():
|
||||
n = Node(scene, name, seq)
|
||||
nodes[n.key] = n
|
||||
return nodes
|
||||
|
||||
|
||||
def worst_gap(nodes):
|
||||
"""Least play time, in ms, between one seek and the next.
|
||||
|
||||
A seek is entering a sequence whose start is >= 0; a negative start means
|
||||
the disc keeps playing, so play ACCUMULATES across such sequences and the
|
||||
gap is a shortest path over them. Bellman-Ford rather than Dijkstra
|
||||
because a zero-length timeout is common (`start_alive` chains) and the
|
||||
graph has cycles; all weights are non-negative so it terminates.
|
||||
|
||||
A null exit ends the scene: the player moves to another scene entirely,
|
||||
which is a seek AND a container change (FINDINGS 53), so it counts as a
|
||||
seek and is flagged.
|
||||
"""
|
||||
INF = float("inf")
|
||||
dist = {k: (0.0 if n.seeks else INF) for k, n in nodes.items()}
|
||||
for _ in range(len(nodes) + 1):
|
||||
changed = False
|
||||
for n in nodes.values():
|
||||
if dist[n.key] == INF:
|
||||
continue
|
||||
for elapsed, kind, tgt in n.exits:
|
||||
if tgt is None:
|
||||
continue
|
||||
tk = f"{n.scene}.{tgt}"
|
||||
if tk not in nodes:
|
||||
continue
|
||||
d = dist[n.key] + max(0.0, float(elapsed))
|
||||
if not nodes[tk].seeks and d < dist[tk] - 1e-9:
|
||||
dist[tk], changed = d, True
|
||||
if not changed:
|
||||
break
|
||||
|
||||
best = {}
|
||||
for n in nodes.values():
|
||||
if dist[n.key] == INF:
|
||||
continue
|
||||
for elapsed, kind, tgt in n.exits:
|
||||
tk = f"{n.scene}.{tgt}" if tgt is not None else None
|
||||
ends_scene = tgt is None
|
||||
if ends_scene or (tk in nodes and nodes[tk].seeks):
|
||||
g = dist[n.key] + max(0.0, float(elapsed))
|
||||
# One entry per (source, destination): several input windows can
|
||||
# lead to the same clip and only the earliest of them binds.
|
||||
k = (n.key, tgt)
|
||||
if k not in best or g < best[k][0]:
|
||||
best[k] = (g, n.key, tgt if tgt else "<end of scene>",
|
||||
kind, ends_scene, n.scene)
|
||||
return sorted(best.values()), dist
|
||||
|
||||
|
||||
# ------------------------------------------------------ 51.3's currency
|
||||
|
||||
def slack_model(gap_s, kbps, wire_kbps, mean_rec_b):
|
||||
"""Records of lookahead accrued in `gap_s` seconds of play at `kbps`, and
|
||||
the seconds one record of lookahead costs.
|
||||
|
||||
This is 51.3's surplus model and nothing more: slack accrues at
|
||||
(pipe - wire) bytes per second. The paced rig is the measurement; this
|
||||
says whether the gap is even in the right order of magnitude.
|
||||
"""
|
||||
surplus = (kbps - wire_kbps) * 1024.0
|
||||
if surplus <= 0:
|
||||
return 0.0, float("inf")
|
||||
return surplus * gap_s / mean_rec_b, mean_rec_b / surplus
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--kbps", type=float, nargs="+", required=True,
|
||||
help="delivered pipe rates, KB/s. REQUIRED, no default "
|
||||
"(FINDINGS 50).")
|
||||
ap.add_argument("--table", default=TABLE, help="DLXSCENE1 scene table")
|
||||
ap.add_argument("--container", default="tmp/rc_fr_singe_scsi_span.dlx",
|
||||
help="container the wire demand and mean record come from")
|
||||
ap.add_argument("--ring", type=float, nargs="+", default=[256, 512])
|
||||
ap.add_argument("--top", type=int, default=12)
|
||||
a = ap.parse_args()
|
||||
|
||||
if not os.path.exists(a.table):
|
||||
print(f"no scene table at {a.table} -- run:\n"
|
||||
f" python3 tools/import/scenegraph.py")
|
||||
return 2
|
||||
doc = json.load(open(a.table))
|
||||
if doc.get("format") != "DLXSCENE1":
|
||||
print(f"{a.table}: not a DLXSCENE1 table")
|
||||
return 2
|
||||
|
||||
nodes = build_graph(doc["scenes"])
|
||||
c = doc["counts"]
|
||||
nseek = sum(1 for n in nodes.values() if n.seeks)
|
||||
|
||||
print(f"=== the scene graph ({a.table})")
|
||||
for s in doc["sources"]:
|
||||
print(f" from {s['name']} ({s['licence']}, {s['holder']}): {s['role']}")
|
||||
print(f" scenes {c['scenes']}, sequences {c['sequences']}, "
|
||||
f"input windows {c['windows']}")
|
||||
print(f" sequences entered by a SEEK: {nseek}/{c['sequences']} "
|
||||
f"({100*nseek/c['sequences']:.1f}%); the rest play on from where the "
|
||||
f"disc is")
|
||||
print(f" scene order: {len(doc['rows'])} rows x {len(doc['rows'][0])}")
|
||||
# Gates. A parser that quietly dropped a branch would produce a SMALLER
|
||||
# graph and a LONGER worst gap -- it would fail in the flattering direction.
|
||||
assert c["sequences"] == 516, f"expected 516 sequences, got {c['sequences']}"
|
||||
assert c["windows"] == 906, f"expected 906 input windows, got {c['windows']}"
|
||||
|
||||
# ---- the measurement
|
||||
gaps, dist = worst_gap(nodes)
|
||||
play = [g for g in gaps if g[5] != "attract_mode"]
|
||||
zero = [g for g in play if g[0] <= 1e-9]
|
||||
print()
|
||||
print("=== the worst gap between two consecutive decision points")
|
||||
print(" A 'gap' is the LEAST play time the disc delivers between one seek")
|
||||
print(" and the next: the earliest an input window opens, chained across")
|
||||
print(" sequences the disc plays through without seeking. One entry per")
|
||||
print(" (source, destination); attract mode is excluded and reported")
|
||||
print(" separately, because nothing branches there under a 12 fps budget.")
|
||||
print(f" {'gap s':>7} from -> to")
|
||||
for g, src, tgt, kind, ends, scene in play[:a.top]:
|
||||
print(f" {g/1000:>7.3f} {src} -{kind}-> {tgt}"
|
||||
+ (" [SCENE CHANGE]" if ends else ""))
|
||||
sc_gaps = [g for g in play if g[4]]
|
||||
print(f" ... {len(play)} distinct transitions into a seek "
|
||||
f"({len(gaps)-len(play)} more in attract mode)")
|
||||
worst = play[0][0] / 1000.0
|
||||
med = play[len(play) // 2][0] / 1000.0
|
||||
print(f" WORST {worst:.3f} s, median {med:.3f} s, "
|
||||
f"best {play[-1][0]/1000:.3f} s")
|
||||
print(f" SCENE CHANGES specifically ({len(sc_gaps)} of them, and each also")
|
||||
print(f" needs a header before its frame 0, FINDINGS 53/55.1): worst "
|
||||
f"{sc_gaps[0][0]/1000:.3f} s, median "
|
||||
f"{sc_gaps[len(sc_gaps)//2][0]/1000:.3f} s")
|
||||
print(f" ZERO-PLAY BRANCHES: {len(zero)} of {len(play)} "
|
||||
f"({100*len(zero)/len(play):.1f}%) open an input window at t=0 of a")
|
||||
print(" clip the disc SEEKED to, so two seeks can fall back to back with no")
|
||||
print(" play between them at all. A rule of the form 'has there been")
|
||||
print(" enough play since the last branch' (51.2's ring_may_seek) can be")
|
||||
print(" answered NO by the content, not by the buffer.")
|
||||
|
||||
# ---- what the input layer has to survive, from the same table
|
||||
inputs, windows = {}, []
|
||||
for n in nodes.values():
|
||||
for x in n.seq["actions"]:
|
||||
inputs[x["input"]] = inputs.get(x["input"], 0) + 1
|
||||
windows.append(x["to_ms"] - x["from_ms"])
|
||||
windows.sort()
|
||||
print()
|
||||
print("=== what the input layer has to survive")
|
||||
print(" " + ", ".join(f"{k} {v}" for k, v in
|
||||
sorted(inputs.items(), key=lambda x: -x[1])))
|
||||
diag = sum(v for k, v in inputs.items()
|
||||
if k in ("upleft", "upright", "downleft", "downright"))
|
||||
print(f" diagonals are {diag} windows of {len(windows)}: rare enough for a "
|
||||
f"port to drop\n and not droppable by one aiming at the arcade")
|
||||
print(f" window length: shortest {windows[0]:.0f} ms "
|
||||
f"({windows[0]/(1000/12):.2f} frame slots at 12 fps), p10 "
|
||||
f"{windows[len(windows)//10]:.0f} ms, median "
|
||||
f"{windows[len(windows)//2]:.0f} ms")
|
||||
print(" the floor is one to two frames wide (54.4: a slot is 72.13 or")
|
||||
print(" 90.16 ms, never 83.33), so input cannot be polled on the frame tick")
|
||||
|
||||
# ---- 51.3's currency
|
||||
if os.path.exists(a.container):
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"seek_slack", "tools/analysis/20_seek_slack.py")
|
||||
m = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(m)
|
||||
import ratectl as RC
|
||||
d, rec = m.records(a.container)
|
||||
mean_rec = float(rec.mean())
|
||||
wire = mean_rec * 12 / 1024 + RC.AUDIO_KBPS
|
||||
gs = [g[0] / 1000.0 for g in play]
|
||||
print()
|
||||
print(f"=== what that gap buys, at explicit rates "
|
||||
f"({os.path.basename(a.container)}: mean record "
|
||||
f"{mean_rec/1024:.1f} KB, wire {wire:.1f} KB/s)")
|
||||
print(f" {'ring KB':>8} {'pipe':>7} {'ceiling':>8} {'climb s':>8} "
|
||||
f"{'median gap':>11} {'accrued':>8} {'under climb':>12}")
|
||||
for ring_kb in a.ring:
|
||||
ring = int(ring_kb * 1024)
|
||||
for kbps in a.kbps:
|
||||
fill = (kbps - RC.AUDIO_KBPS) * 1024 / 12
|
||||
lo, hi, ring_ref, rate_ref = m.paced_sim(rec, ring, fill)
|
||||
ceiling = int(hi.max())
|
||||
accrued, per_rec = slack_model(med, kbps, wire, mean_rec)
|
||||
climb = ceiling * per_rec
|
||||
under = sum(1 for g in gs if g < climb)
|
||||
print(f" {ring_kb:>8.0f} {kbps:>7.1f} {ceiling:>8} "
|
||||
f"{climb:>8.2f} {med:>11.3f} {accrued:>8.2f} "
|
||||
f"{f'{under}/{len(gs)}':>12} {100*under/len(gs):.0f}%")
|
||||
|
||||
# What a branch costs when the gap before it bought nothing. The ring
|
||||
# is empty after a seek and the decoder is released at the prefill depth
|
||||
# (55.4's shipped policy is 2 records), so this is the stall the player
|
||||
# eats every time -- not the climb to the ceiling, which is what it
|
||||
# needs in order to TOLERATE the next one. A scene change additionally
|
||||
# needs its header before frame 0; 22_scene_load.py prices that case
|
||||
# properly, clocks included.
|
||||
PREFILL_REC, HDR_B = 2, 6164
|
||||
print()
|
||||
print(f" A branch taken on an empty ring, at the shipped prefill of "
|
||||
f"{PREFILL_REC} records:")
|
||||
for kbps in a.kbps:
|
||||
b = PREFILL_REC * mean_rec
|
||||
ms = b / (kbps * 1024) * 1000
|
||||
hms = (b + HDR_B) / (kbps * 1024) * 1000
|
||||
print(f" {kbps:>7.1f} KB/s: {ms:>7.1f} ms "
|
||||
f"({ms/(1000/12):.2f} frame slots), and {hms:>7.1f} ms "
|
||||
f"({hms/(1000/12):.2f}) if it is a scene change carrying "
|
||||
f"{HDR_B:,} header bytes")
|
||||
print()
|
||||
print(" 'climb s' is 51.3's: seconds of play to refill from empty to")
|
||||
print(" the ceiling. 'under climb' is how many of this game's own")
|
||||
print(" branch points arrive sooner than that, i.e. are reached with")
|
||||
print(" LESS lookahead than the one before them. The worst gap is")
|
||||
print(f" {worst:.3f} s and buys nothing at any rate in this table.")
|
||||
else:
|
||||
print(f"\n{a.container}: MISSING -- rate half skipped")
|
||||
|
||||
# ---- the cross-check, and what it is worth
|
||||
print()
|
||||
print("=== the second table, and why it is not a second transcription")
|
||||
cross, meta = doc.get("crosscheck"), doc.get("crosscheck_meta")
|
||||
if not cross:
|
||||
print(" none in this table -- the import ran without it")
|
||||
return 0
|
||||
print(f" {meta['chapters']} chapters, {meta['scenes_mapped']} scenes")
|
||||
print(f" PROVENANCE: {meta['provenance']}")
|
||||
print(" FINDINGS 16 planned to diff two INDEPENDENT transcriptions to")
|
||||
print(" catch transcription errors. There is only one transcription.")
|
||||
print(" This diff catches CONVERSION errors and nothing more.")
|
||||
|
||||
FRAME_MS = 1000.0 / LD_FPS
|
||||
offs, durs, same, diff, missing = [], [], 0, 0, 0
|
||||
renames, inputs_differ, examples = 0, [], []
|
||||
for scene, seqs in sorted(cross.items()):
|
||||
for seq, ch in sorted(seqs.items()):
|
||||
n = nodes.get(f"{scene}.{seq}")
|
||||
if n is None:
|
||||
missing += 1
|
||||
continue
|
||||
if n.seeks:
|
||||
offs.append(ch["start_ms"] - n.start)
|
||||
durs.append(((ch["end_ms"] - ch["start_ms"]) - n.timeout_ms,
|
||||
ch["chapter"]))
|
||||
ce = sorted((x["input"], x["next"]) for x in ch["actions"])
|
||||
de = sorted((x["input"], x["next"]) for x in n.seq["actions"])
|
||||
if ce == de:
|
||||
same += 1
|
||||
continue
|
||||
diff += 1
|
||||
ci = sorted(i for i, _ in ce)
|
||||
di = sorted(i for i, _ in de)
|
||||
if ci != di:
|
||||
inputs_differ.append((ch["chapter"], ci, di))
|
||||
else:
|
||||
renames += 1
|
||||
if len(examples) < 3:
|
||||
examples.append((ch["chapter"], ce, de))
|
||||
|
||||
if offs:
|
||||
offs.sort()
|
||||
print(f" START TIMES are on different timelines and do not compare: "
|
||||
f"{len(offs)} seeking chapters,")
|
||||
print(f" offset spread {min(offs)/1000:,.1f} s .. "
|
||||
f"{max(offs)/1000:,.1f} s, median {offs[len(offs)//2]/1000:,.1f} s"
|
||||
f" -- not a constant, and not even one sign.")
|
||||
if durs:
|
||||
dd = sorted(x for x, _ in durs)
|
||||
agree = sum(1 for x in dd if abs(x) <= FRAME_MS)
|
||||
print(f" DURATIONS compare (offset-invariant): {agree}/{len(dd)} "
|
||||
f"within one frame of the medium ({100*agree/len(dd):.1f}%), "
|
||||
f"median {dd[len(dd)//2]:,.0f} ms")
|
||||
for x, ch in sorted(durs, key=lambda x: -abs(x[0]))[:3]:
|
||||
print(f" widest {ch}: {x/1000:+.3f} s")
|
||||
if same + diff:
|
||||
print(f" BRANCH STRUCTURE compares: {same}/{same+diff} chapters have "
|
||||
f"the identical set of (input -> target) edges "
|
||||
f"({100*same/(same+diff):.1f}%)")
|
||||
if diff:
|
||||
print(f" {renames} of the {diff} differ only in what a target "
|
||||
f"sequence is NAMED")
|
||||
DIAG = ("upleft", "upright", "downleft", "downright")
|
||||
lost = [ch for ch, ci, di in inputs_differ
|
||||
if set(di) - set(ci) and all(x in DIAG for x in set(di) - set(ci))]
|
||||
print(f" {len(inputs_differ)} differ in the INPUT SET, and "
|
||||
f"{len(lost)} of those are the other table dropping the arcade's")
|
||||
print(" DIAGONALS: a controller decision, not a transcription "
|
||||
"difference.")
|
||||
for ch, ci, di in inputs_differ:
|
||||
if ch not in lost:
|
||||
print(f" the remaining one: {ch}, {ci} vs {di}")
|
||||
for ch, ce, de in examples:
|
||||
print(f" e.g. {ch}\n cross {ce}\n graph {de}")
|
||||
if missing:
|
||||
print(f" {missing} chapters have no sequence in the graph "
|
||||
f"(the other project added them)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -406,4 +406,30 @@ grep -q "^OK" tmp/ringseek_check.log || {
|
||||
echo "FAIL: the decode after the seek was not pixel-exact."
|
||||
tail -4 tmp/ringseek_check.log; exit 1; }
|
||||
|
||||
echo "--- session 24: the scene graph, and the gap between branch points (FINDINGS 56) ---"
|
||||
# The arcade scene graph is not in this repo and is not redistributable from
|
||||
# here. tools/import/scenegraph.py is the ONE file in the tree that knows the
|
||||
# outside projects exist; it writes tmp/scenegraph.json in this project's own
|
||||
# DLXSCENE1 schema and everything downstream reads only that.
|
||||
# What is gated is the IMPORT, not the numbers: 516 sequences and 906 input
|
||||
# windows, and the four timing helpers still being the formulas the importer
|
||||
# evaluates. Skipped when the checkout is absent.
|
||||
DIRKSIMPLE=${DLX_DIRKSIMPLE:-tmp/scenegraph/DirkSimple}
|
||||
if [ -f "$DIRKSIMPLE/data/games/lair/game.lua" ]; then
|
||||
DLX_DIRKSIMPLE="$DIRKSIMPLE" python3 tools/import/scenegraph.py \
|
||||
-o tmp/scenegraph.json > tmp/scenegraph_import.log 2>&1 \
|
||||
|| { cat tmp/scenegraph_import.log; exit 1; }
|
||||
sed "s/^/ /" tmp/scenegraph_import.log
|
||||
grep -q "516 sequences, 906 input windows" tmp/scenegraph_import.log || {
|
||||
echo "FAIL: the scene graph did not import to 516/906 -- upstream changed,"
|
||||
echo " or the parser silently dropped branches."; exit 1; }
|
||||
python3 tools/analysis/25_scene_graph.py --kbps 488 --ring 256 \
|
||||
> tmp/scenegraph_check.log 2>&1 \
|
||||
|| { tail -20 tmp/scenegraph_check.log; exit 1; }
|
||||
grep -aE "^ WORST |^ ZERO-PLAY|^ BRANCH STRUCTURE" tmp/scenegraph_check.log
|
||||
else
|
||||
echo " SKIPPED: no DirkSimple checkout at $DIRKSIMPLE"
|
||||
echo " (git clone --depth 1 https://github.com/icculus/DirkSimple)"
|
||||
fi
|
||||
|
||||
echo "ALL GREEN"
|
||||
|
||||
@@ -0,0 +1,436 @@
|
||||
"""The ONE file in this tree that knows anything about somebody else's source.
|
||||
|
||||
Everything downstream -- `tools/analysis/25_scene_graph.py` and whatever the
|
||||
game-logic layer eventually becomes -- reads the neutral table this writes and
|
||||
knows nothing about where it came from. That is deliberate. Two outside
|
||||
projects are read here, neither is vendored, neither ships in this repo, and
|
||||
their file layouts, table names, timing formulas and magic constants are wired
|
||||
into THIS file and nowhere else. When one of them moves, one file breaks.
|
||||
|
||||
python3 tools/import/scenegraph.py [-o tmp/scenegraph.json]
|
||||
|
||||
Sources, both permissively licensed and both cloned by the reader:
|
||||
|
||||
icculus/DirkSimple -- zlib, Ryan C. Gordon. `data/games/lair/game.lua` holds
|
||||
the arcade scene graph, transcribed from the arcade ROM's own data table.
|
||||
THE source; the measurement is made on this.
|
||||
DLX_DIRKSIMPLE=/path (default tmp/scenegraph/DirkSimple)
|
||||
|
||||
astrobleem/SNES-SuperDragonsLairArcade -- MIT, Chad Doebelin.
|
||||
`data/events/*.xml` holds 516 chapter definitions. Read as a CROSS-CHECK
|
||||
only, and a weak one: their own README states the chapters are "derived
|
||||
from DirkSimple game data", so this is a second COPY, not a second
|
||||
transcription (FINDINGS 56.2). Optional; skipped when absent.
|
||||
DLX_SNESLAIR=/path (default tmp/scenegraph/SNES-SuperDragonsLairArcade)
|
||||
|
||||
WHAT COMES OUT is `DLXSCENE1`, our own schema, in one JSON file:
|
||||
|
||||
{"format": "DLXSCENE1",
|
||||
"sources": [{"name", "url", "licence", "holder", "role"} ...],
|
||||
"scenes": {scene: {sequence: {
|
||||
"start_ms": float, absolute position on the medium, or -1 for
|
||||
"keep playing from where the disc already is",
|
||||
"timeout_ms": float, how long the clip runs unbranched,
|
||||
"kills", "single_frame": bool,
|
||||
"timeout_next": sequence name or null,
|
||||
"actions": [{"input", "from_ms", "to_ms", "next"}]}}},
|
||||
"rows": [[scene, scene, scene] ...], the arcade's 13x3 scene order
|
||||
"counts": {"scenes", "sequences", "windows"},
|
||||
"crosscheck": {scene: {sequence: {"start_ms", "end_ms", "actions": [...],
|
||||
"chapter"}}} or null}
|
||||
|
||||
Nothing in the schema is a copy of either project's structure: it is what this
|
||||
project's own analysis needs, which is a clip's length, its exits and the
|
||||
earliest instant each of them can fire.
|
||||
|
||||
If a scene table is ever COMMITTED to this repo rather than regenerated into
|
||||
gitignored tmp/, it becomes redistribution of derived data and the attribution
|
||||
in "sources" has to travel with it. FINDINGS 16, 56.5.
|
||||
"""
|
||||
import sys, os, re, json, argparse
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
DIRK = os.environ.get("DLX_DIRKSIMPLE", "tmp/scenegraph/DirkSimple")
|
||||
SNES = os.environ.get("DLX_SNESLAIR", "tmp/scenegraph/SNES-SuperDragonsLairArcade")
|
||||
|
||||
# DirkSimple's own constants, from data/games/lair/game.lua. Restated here
|
||||
# because this file evaluates its table; both are gated below against the text
|
||||
# of the file they came from, so they cannot go stale silently.
|
||||
LD_FPS = 23.976 # laserdisc_frame_to_ms
|
||||
ROM_OFFSET_MS = 6297.0 # time_laserdisc_frame's "magic millisecond offset"
|
||||
|
||||
SOURCES = [
|
||||
{"name": "icculus/DirkSimple",
|
||||
"url": "https://github.com/icculus/DirkSimple",
|
||||
"licence": "zlib", "holder": "Ryan C. Gordon",
|
||||
"role": "the scene graph itself, transcribed from the arcade ROM"},
|
||||
{"name": "astrobleem/SNES-SuperDragonsLairArcade",
|
||||
"url": "https://github.com/astrobleem/SNES-SuperDragonsLairArcade",
|
||||
"licence": "MIT", "holder": "Chad Doebelin",
|
||||
"role": "cross-check only; itself derived from DirkSimple (FINDINGS 56.2)"},
|
||||
]
|
||||
|
||||
# ---------------------------------------------------------------- Lua subset
|
||||
|
||||
class LuaParser:
|
||||
"""Just enough Lua to read game.lua's `scenes` table constructor.
|
||||
|
||||
Not a Lua interpreter and not trying to be: the grammar is table
|
||||
constructors, string/number/boolean/nil literals, bare identifiers (used
|
||||
only for function references like `interrupt=game_over_complete`), calls to
|
||||
four known helpers, and + / - between them. Anything else is a parse error
|
||||
rather than a silent skip, and the caller asserts the whole table was
|
||||
consumed, so a change upstream shows up as a failure and not as a smaller
|
||||
scene graph.
|
||||
"""
|
||||
|
||||
def __init__(self, text, helpers):
|
||||
self.s, self.i, self.helpers = text, 0, helpers
|
||||
|
||||
def error(self, msg):
|
||||
line = self.s.count("\n", 0, self.i) + 1
|
||||
raise SyntaxError(f"{msg} at line {line}: {self.s[self.i:self.i+60]!r}")
|
||||
|
||||
def ws(self):
|
||||
while self.i < len(self.s):
|
||||
c = self.s[self.i]
|
||||
if c in " \t\r\n":
|
||||
self.i += 1
|
||||
elif self.s.startswith("--", self.i):
|
||||
self.i = self.s.find("\n", self.i)
|
||||
if self.i < 0:
|
||||
self.i = len(self.s)
|
||||
else:
|
||||
return
|
||||
|
||||
def take(self, lit):
|
||||
self.ws()
|
||||
if self.s.startswith(lit, self.i):
|
||||
self.i += len(lit)
|
||||
return True
|
||||
return False
|
||||
|
||||
def expect(self, lit):
|
||||
if not self.take(lit):
|
||||
self.error(f"expected {lit!r}")
|
||||
|
||||
def name(self):
|
||||
self.ws()
|
||||
m = re.compile(r"[A-Za-z_][A-Za-z0-9_]*").match(self.s, self.i)
|
||||
if not m:
|
||||
return None
|
||||
self.i = m.end()
|
||||
return m.group(0)
|
||||
|
||||
def value(self):
|
||||
self.ws()
|
||||
if self.take("{"):
|
||||
return self.table()
|
||||
m = re.compile(r"-?\d+(\.\d+)?").match(self.s, self.i)
|
||||
if m:
|
||||
self.i = m.end()
|
||||
v = float(m.group(0))
|
||||
return self.arith(int(v) if v.is_integer() else v)
|
||||
if self.s[self.i] in "\"'":
|
||||
q = self.s[self.i]
|
||||
j = self.s.index(q, self.i + 1)
|
||||
v = self.s[self.i + 1:j]
|
||||
self.i = j + 1
|
||||
return v
|
||||
n = self.name()
|
||||
if n is None:
|
||||
self.error("expected a value")
|
||||
if n == "nil":
|
||||
return None
|
||||
if n in ("true", "false"):
|
||||
return n == "true"
|
||||
if self.take("("): # a call
|
||||
args = []
|
||||
if not self.take(")"):
|
||||
while True:
|
||||
args.append(self.value())
|
||||
if self.take(")"):
|
||||
break
|
||||
self.expect(",")
|
||||
if n not in self.helpers:
|
||||
self.error(f"unknown helper {n!r}")
|
||||
return self.arith(self.helpers[n](*args))
|
||||
return Symbol(n) # a function reference
|
||||
|
||||
def arith(self, left):
|
||||
"""+ and - between numbers. Present in the source and load-bearing:
|
||||
e.g. `time_laserdisc_frame(1823) - laserdisc_frame_to_ms(2)`."""
|
||||
while True:
|
||||
self.ws()
|
||||
if self.i < len(self.s) and self.s[self.i] in "+-":
|
||||
op = self.s[self.i]
|
||||
self.i += 1
|
||||
right = self.value()
|
||||
left = left + right if op == "+" else left - right
|
||||
else:
|
||||
return left
|
||||
|
||||
def table(self):
|
||||
d, arr = {}, []
|
||||
while True:
|
||||
self.ws()
|
||||
if self.take("}"):
|
||||
break
|
||||
save = self.i
|
||||
k = self.name()
|
||||
if k is not None and self.take("="):
|
||||
d[k] = self.value()
|
||||
else:
|
||||
self.i = save
|
||||
arr.append(self.value())
|
||||
if not (self.take(",") or self.take(";")):
|
||||
self.expect("}")
|
||||
break
|
||||
if d and arr:
|
||||
self.error("mixed array/hash table")
|
||||
return arr if arr or not d else d
|
||||
|
||||
|
||||
class Symbol(str):
|
||||
"""A bare Lua identifier used as a value (a function reference)."""
|
||||
|
||||
|
||||
def load_dirksimple(path):
|
||||
"""Parse lair/game.lua's `scenes` and `scene_manager` tables.
|
||||
|
||||
Gates the four timing helpers against the text that defines them, so this
|
||||
tool cannot keep evaluating a formula upstream has changed.
|
||||
"""
|
||||
src = open(os.path.join(path, "data/games/lair/game.lua"), encoding="utf-8").read()
|
||||
|
||||
def gate(pat, what):
|
||||
if not re.search(pat, src):
|
||||
raise AssertionError(f"DirkSimple's {what} is not what this tool "
|
||||
f"evaluates any more (looked for {pat!r})")
|
||||
gate(r"frame\s*/\s*23\.976", "laserdisc_frame_to_ms")
|
||||
gate(r"-\s*6297\.0", "time_laserdisc_frame's ROM offset")
|
||||
gate(r"time_laserdisc_noseek.*?\n\s*return -1", "time_laserdisc_noseek")
|
||||
gate(r"\(seconds \* 1000\) \+ ms", "time_to_ms")
|
||||
|
||||
helpers = {
|
||||
"laserdisc_frame_to_ms": lambda f: (f / LD_FPS) * 1000.0,
|
||||
"time_laserdisc_frame": lambda f: (f / LD_FPS) * 1000.0 - ROM_OFFSET_MS,
|
||||
"time_laserdisc_noseek": lambda: -1,
|
||||
# time_to_ms takes (seconds, ms); the source calls it with a third
|
||||
# argument exactly once, which Lua discards.
|
||||
"time_to_ms": lambda s, ms=0, *_: s * 1000 + ms,
|
||||
}
|
||||
|
||||
out = {}
|
||||
for var in ("scenes", "scene_manager"):
|
||||
m = re.search(rf"^{var} = \{{", src, re.M)
|
||||
if not m:
|
||||
raise AssertionError(f"{var} table not found in game.lua")
|
||||
p = LuaParser(src, helpers)
|
||||
p.i = m.end()
|
||||
out[var] = p.table()
|
||||
# The parser must have consumed the WHOLE constructor: table() returns
|
||||
# with the cursor just past the matching close brace, and in this file
|
||||
# every top-level table ends on a `}` in column 0. A parser that
|
||||
# stopped early would return a smaller graph, which is the direction
|
||||
# that flatters the measurement, so this is checked rather than assumed.
|
||||
if src[p.i - 1] != "}":
|
||||
raise AssertionError(f"{var}: parser did not end on a close brace")
|
||||
if src.rfind("\n", 0, p.i) != p.i - 2:
|
||||
raise AssertionError(f"{var}: the constructor ended mid-line at "
|
||||
f"{p.i}, so the parse stopped early")
|
||||
return out["scenes"], out["scene_manager"]
|
||||
|
||||
|
||||
# ------------------------------------------------- the SNES cross-check
|
||||
|
||||
def load_snes(path):
|
||||
"""Parse the SNES project's chapter XMLs into {chapter: (start_ms, end_ms,
|
||||
[(input, from_ms, to_ms, target)])}, plus the provenance line."""
|
||||
ev = os.path.join(path, "data/events")
|
||||
readme = os.path.join(ev, "README.md")
|
||||
prov = None
|
||||
if os.path.exists(readme):
|
||||
txt = open(readme, encoding="utf-8").read()
|
||||
if "derived from DirkSimple" in txt:
|
||||
prov = "derived from DirkSimple game data (data/events/README.md)"
|
||||
|
||||
def ms(e):
|
||||
return (int(e.get("min", 0)) * 60000 + int(e.get("second", 0)) * 1000
|
||||
+ int(e.get("ms", 0)))
|
||||
|
||||
chapters = {}
|
||||
for f in sorted(os.listdir(ev)):
|
||||
if not f.endswith(".xml"):
|
||||
continue
|
||||
root = ET.parse(os.path.join(ev, f)).getroot()
|
||||
tl = root.find("timeline")
|
||||
s = tl.find("timestart")
|
||||
e = tl.find("timeend")
|
||||
acts = []
|
||||
for evt in root.findall("./events/event"):
|
||||
if evt.get("type") != "direction":
|
||||
continue
|
||||
t = evt.find("timeline")
|
||||
p = evt.find("./params/str[@key='type']")
|
||||
r = evt.find("./result/playchapter")
|
||||
acts.append((p.get("value") if p is not None else "?",
|
||||
ms(t.find("timestart")),
|
||||
ms(t.find("timeend")) if t.find("timeend") is not None else -1,
|
||||
r.get("name") if r is not None else None))
|
||||
chapters[f[:-4]] = (ms(s) if s is not None else -1,
|
||||
ms(e) if e is not None else -1, acts)
|
||||
return chapters, prov
|
||||
|
||||
|
||||
def map_abbrevs(chapters, scenes):
|
||||
"""Match each SNES scene abbreviation to a DirkSimple scene from the DATA,
|
||||
not from a hand-written table, so a wrong match is visible rather than
|
||||
assumed. Three signals, in order:
|
||||
|
||||
1. the abbreviation's letters must be a SUBSEQUENCE of the scene name
|
||||
(`snkr` is snake_room, and cannot be black_knight however similar
|
||||
their `seqN` names are -- overlap alone got that one wrong);
|
||||
2. the Jaccard overlap of the two sequence-name sets;
|
||||
3. the reversed-scene rule: Dragon's Lair mirrors thirteen scenes, and
|
||||
where BOTH `x` and `xr` exist as abbreviations, `xr` is the
|
||||
`_reversed` scene and `x` is not. Their sequence names are identical,
|
||||
so nothing else separates them.
|
||||
|
||||
Assignment is 1:1 and greedy on that ranking.
|
||||
"""
|
||||
def subseq(a, b):
|
||||
it = iter(b)
|
||||
return all(c in it for c in a)
|
||||
|
||||
by_abbr = {}
|
||||
for ch in chapters:
|
||||
abbr, _, seq = ch.partition("_")
|
||||
by_abbr.setdefault(abbr, set()).add(seq)
|
||||
|
||||
pairs = []
|
||||
for abbr, seqs in by_abbr.items():
|
||||
mirror = abbr.endswith("r") and abbr[:-1] in by_abbr
|
||||
for scene, s in scenes.items():
|
||||
if not subseq(abbr, scene.replace("_", "")):
|
||||
continue
|
||||
union = len(seqs | set(s))
|
||||
j = len(seqs & set(s)) / union if union else 0.0
|
||||
rev = scene.endswith("_reversed")
|
||||
pairs.append((j, 1 if rev == mirror else 0, -len(scene),
|
||||
abbr, scene, len(seqs)))
|
||||
pairs.sort(reverse=True)
|
||||
|
||||
mapping, used = {}, set()
|
||||
for j, _, _, abbr, scene, n in pairs:
|
||||
if abbr in mapping or scene in used:
|
||||
continue
|
||||
mapping[abbr] = (scene, j, n)
|
||||
used.add(scene)
|
||||
unmatched = [(a, None, 0.0, len(s)) for a, s in by_abbr.items()
|
||||
if a not in mapping]
|
||||
return mapping, unmatched
|
||||
|
||||
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- emit
|
||||
|
||||
def to_neutral(scenes, mgr):
|
||||
"""DirkSimple's tables -> DLXSCENE1. The one place their shape is read."""
|
||||
out = {}
|
||||
for scene, seqs in scenes.items():
|
||||
s = out.setdefault(scene, {})
|
||||
for name, seq in seqs.items():
|
||||
t = seq["timeout"]
|
||||
s[name] = {
|
||||
"start_ms": float(seq["start_time"]),
|
||||
"timeout_ms": float(t["when"]),
|
||||
"timeout_next": t.get("nextsequence"),
|
||||
"kills": bool(seq.get("kills_player")),
|
||||
"single_frame": bool(seq.get("is_single_frame")),
|
||||
"actions": [{"input": a["input"],
|
||||
"from_ms": float(a["from"]),
|
||||
"to_ms": float(a["to"]),
|
||||
"next": a.get("nextsequence")}
|
||||
for a in seq.get("actions", [])],
|
||||
}
|
||||
rows = [list(r) for r in mgr["rows"]]
|
||||
return out, rows
|
||||
|
||||
|
||||
def crosscheck_neutral(chapters, scenes):
|
||||
"""The SNES chapters, keyed the way OUR table is keyed.
|
||||
|
||||
Their `abbr_sequence` naming is resolved to a scene here, from the data
|
||||
(see map_abbrevs), so nothing downstream has to know an abbreviation
|
||||
exists. Chapters the mapping cannot place are dropped and counted.
|
||||
"""
|
||||
mapping, unmatched = map_abbrevs(chapters, scenes)
|
||||
out, dropped = {}, 0
|
||||
for ch, (s_ms, e_ms, acts) in chapters.items():
|
||||
abbr, _, seq = ch.partition("_")
|
||||
if abbr not in mapping:
|
||||
dropped += 1
|
||||
continue
|
||||
scene = mapping[abbr][0]
|
||||
pre = abbr + "_"
|
||||
out.setdefault(scene, {})[seq] = {
|
||||
"chapter": ch,
|
||||
"start_ms": float(s_ms),
|
||||
"end_ms": float(e_ms),
|
||||
"actions": [{"input": i, "from_ms": float(f), "to_ms": float(t2),
|
||||
"next": (n[len(pre):] if n and n.startswith(pre) else n)}
|
||||
for i, f, t2, n in acts],
|
||||
}
|
||||
weak = sorted((round(j, 3), ab, sc) for ab, (sc, j, _n) in mapping.items()
|
||||
if j < 1.0)
|
||||
return out, {"chapters": len(chapters), "dropped": dropped,
|
||||
"scenes_mapped": len(mapping),
|
||||
"weakest_mappings": weak[:6]}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("-o", "--out", default="tmp/scenegraph.json")
|
||||
a = ap.parse_args()
|
||||
|
||||
if not os.path.isdir(DIRK):
|
||||
sys.exit(f"no DirkSimple checkout at {DIRK} -- set DLX_DIRKSIMPLE, or\n"
|
||||
f" git clone --depth 1 https://github.com/icculus/DirkSimple")
|
||||
scenes, mgr = load_dirksimple(DIRK)
|
||||
neutral, rows = to_neutral(scenes, mgr)
|
||||
nseq = sum(len(s) for s in neutral.values())
|
||||
nwin = sum(len(q["actions"]) for s in neutral.values() for q in s.values())
|
||||
|
||||
cross, cross_meta = None, None
|
||||
if os.path.isdir(SNES):
|
||||
chapters, prov = load_snes(SNES)
|
||||
cross, cross_meta = crosscheck_neutral(chapters, scenes)
|
||||
cross_meta["provenance"] = prov
|
||||
# 56.2: this is the sentence the whole "second source" claim turned on.
|
||||
# If upstream ever removes it, say so rather than silently promoting a
|
||||
# copy back to an independent transcription.
|
||||
if not prov:
|
||||
cross_meta["provenance"] = ("UNSTATED -- data/events/README.md no "
|
||||
"longer says where the chapters came "
|
||||
"from; re-check before trusting this "
|
||||
"as anything.")
|
||||
|
||||
doc = {"format": "DLXSCENE1", "sources": SOURCES,
|
||||
"scenes": neutral, "rows": rows,
|
||||
"counts": {"scenes": len(neutral), "sequences": nseq,
|
||||
"windows": nwin},
|
||||
"crosscheck": cross, "crosscheck_meta": cross_meta}
|
||||
os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True)
|
||||
with open(a.out, "w") as f:
|
||||
json.dump(doc, f, indent=1, sort_keys=True)
|
||||
print(f"{a.out}: {len(neutral)} scenes, {nseq} sequences, {nwin} input "
|
||||
f"windows, {len(rows)} rows"
|
||||
+ (f", cross-check {cross_meta['chapters']} chapters"
|
||||
if cross_meta else ", no cross-check (SNES tree absent)"))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user