Dragon-s-Lair-X68k/tools/analysis/19_ring_stream.py

"""Ring-buffer streaming simulation, against the CONTIGUITY constraint (STATUS 3/4).

09_buffer_sim.py asked one question -- does cumulative supply ever fall behind
cumulative demand -- and answered it in BYTES.  FINDINGS 21 got "zero required
prefill" out of it at 110 and 280 KB/s.  That test is necessary and not
sufficient, and the missing half is the whole of STATUS item 3:

  src/player/decode.s reads a frame record with a MONOTONICALLY INCREASING a0
  and no bounds check anywhere.  `move.l (a0)+,d0` for the length, `lea
  MODEB(a0),a0` for the span section, eleven unrolled `movem.l (a0)+` chains,
  `move.b (a0)+` per block index.  Nothing in it can survive an address that
  wraps mid-record.  So the buffer does not merely need ENOUGH BYTES resident
  by the deadline -- it needs the WHOLE NEXT RECORD resident and CONTIGUOUS.

Having enough bytes and having them contiguous are different conditions, and a
byte-counting simulation cannot tell them apart.  This one models the ring's
addresses, not just its occupancy.

THREE WRAP POLICIES, and the point of the tool is that they are not equivalent:

  split   the writer wraps mid-record; the reader cannot.  Requires a SHADOW of
          the ring's first MAXREC bytes mirrored past its end, so any record
          start can be read linearly for MAXREC bytes.  Every byte landing in
          that first MAXREC is written twice.  Costs 68000 CLOCKS, forever, at a
          rate set by MAXREC/ring -- and those clocks come out of the same
          budget the decoder is already spending 77.0% of (FINDINGS 45).

  aligned the writer refuses to start a record it cannot finish before the end
          of the ring; it leaves a hole and restarts at 0.  Costs RAM (the mean
          hole) and nothing else -- no copy, no per-byte work.  Needs a frame
          INDEX so the fill side knows record boundaries, which a branching
          laserdisc game needs anyway to seek to a branch point.

  none    the decoder handles the wrap itself.  Priced here only to show what it
          would cost: a bounds test in the block loop is inside the sequence
          FINDINGS 30.4/40 fitted, so it does not cost a branch -- it costs
          every span and per-block constant in the tree being re-measured.
          Not simulated; see the note printed at the end.

DEADLINE MODEL, and it is the conservative one: record i must be wholly
resident when frame i's decode BEGINS.  The decoder in fact reads a record
progressively over ~77% of a frame time, so a byte arriving mid-frame would in
practice be in time -- but that is a race between the DMAC's fill address and
a0, and this tool refuses to certify a design on a race it cannot see.

Fill is quantised to 512-byte SCSI blocks: a partial sector is not resident.

    python3 tools/analysis/19_ring_stream.py [container ...] --kbps R [--ring KB]

`--kbps` is REQUIRED and has no default -- see the argument's help text.
"""
import sys, os, argparse
sys.path.insert(0, "tools/encoder")
import numpy as np
from dlx import DLX
import ratectl as RC

SECTOR = 512
# 5 clocks/byte for a 68000 `move.l (a0)+,(a1)+` copy: 20 clocks moves 4 bytes
# on a 16-bit bus (2 read + 2 write bus cycles at 4 clocks, plus the fetch it
# shares with the loop).  Deliberately the OPTIMISTIC figure -- a movem-shaped
# copy is what the shadow would really use, and it is the same 5.0.
COPY_CLK_PER_BYTE = 5.0
CPUHZ = 10_000_000


def records(path):
    """Padded record sizes, exactly as the 68000 walks them.

    prep_dlx.py rounds each record START up to 4 (FINDINGS 28.3), so the bytes
    the ring must hold per frame are the padded ones, not the payload.
    """
    d = DLX(path)
    rec = np.array([4 + n + (-(4 + n) % 4) for _, n in d.frames], np.int64)
    return d, rec


def simulate(rec, fill_per_frame, ring, policy, maxrec):
    """Address-level ring simulation.  Returns a dict of results.

    The ring is modelled as a write cursor and a read cursor over `ring` bytes.
    Supply arrives at `fill_per_frame` bytes per frame time, sector-quantised.
    Record i is due at the start of frame i.
    """
    n = len(rec)
    resident = 0.0          # bytes fully arrived and not yet consumed
    carry = 0.0             # sub-sector remainder of the fill
    wcur = 0                # write cursor within the ring
    holes = []              # bytes wasted per wrap, `aligned` policy
    shadow_bytes = 0        # bytes double-written, `split` policy
    occ = []
    prefill = 0.0
    late = []
    free = ring

    # Required prefill is solved rather than searched: run once with an infinite
    # head start to find the worst deficit, exactly as 09_buffer_sim does, then
    # assert the ring can hold it.
    deficit = np.maximum.accumulate(np.cumsum(rec - fill_per_frame))
    prefill = float(max(0.0, deficit.max()))

    for i, r in enumerate(rec):
        # --- supply for this frame time, sector-quantised
        avail = carry + fill_per_frame
        sectors = int(avail // SECTOR)
        got = sectors * SECTOR
        carry = avail - got

        # --- placement: does this frame's arriving data cross the ring end?
        if policy == "aligned":
            # The writer will not start a record it cannot finish.  Charge the
            # hole when the NEXT record would not fit in the tail.
            if wcur + r > ring:
                holes.append(ring - wcur)
                wcur = 0
            wcur += r
        else:  # split
            end = wcur + r
            if end > ring:
                wcur = end - ring
                # every byte that landed in the first MAXREC of the ring is
                # mirrored into the shadow
                shadow_bytes += min(wcur, maxrec)
            else:
                wcur = end
                if wcur <= maxrec:
                    shadow_bytes += r
                elif wcur - r < maxrec:
                    shadow_bytes += maxrec - (wcur - r)

        resident += got
        if resident + 1e-9 < r:
            late.append((i, float(r - resident)))
        resident -= r
        occ.append(resident)

    hole_mean = float(np.mean(holes)) if holes else 0.0
    usable = ring - hole_mean if policy == "aligned" else ring
    copy_clk = shadow_bytes * COPY_CLK_PER_BYTE / max(1, n)
    return dict(prefill=prefill, late=late, occ=np.array(occ),
                holes=holes, hole_mean=hole_mean, usable=usable,
                shadow_bytes=shadow_bytes, copy_clk_per_frame=copy_clk,
                wraps=len(holes) if policy == "aligned" else None)


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("containers", nargs="*",
                    default=["tmp/s14_d5_all1500.dlx",
                             "tmp/rc_fr_singe_scsi_span.dlx"])
    ap.add_argument("--kbps", type=float, required=True,
                    help="delivered pipe, KB/s. REQUIRED, and deliberately has "
                         "no default: the delivery rate is a property of the "
                         "medium and this project has never measured it. The "
                         "figure that used to sit here was a user-supplied "
                         "'4 Mbps' with no provenance and was never a bus "
                         "measurement (FINDINGS 42.1); leaving it as a default "
                         "let table after table be scored against it without "
                         "anyone restating what it was.")
    ap.add_argument("--ring", type=float, default=256.0,
                    help="ring size in KB (default 256, FINDINGS 21's sizing)")
    a = ap.parse_args()

    FPS = 12
    print(f"ring {a.ring:.0f} KB   sector {SECTOR} B   "
          f"audio {RC.AUDIO_KBPS} KB/s debited from the pipe\n")

    for path in a.containers:
        if not os.path.exists(path):
            print(f"{path}: MISSING -- skipped\n"); continue
        d, rec = records(path)
        maxrec = int(rec.max())
        ring = int(a.ring * 1024)
        wire = rec.mean() * FPS / 1024 + RC.AUDIO_KBPS

        print(f"=== {path}")
        print(f"    {d.nframes} frames @ {d.fps}fps, record bytes "
              f"min {rec.min():,} median {int(np.median(rec)):,} max {maxrec:,}")
        print(f"    wire demand {wire:.1f} KB/s "
              f"(video {rec.mean()*FPS/1024:.1f} + audio {RC.AUDIO_KBPS}), "
              f"including the u32 length and the 4-byte record pad")

        # A required prefill is only a startup cost if the window's MEAN demand
        # is under the pipe.  If the mean is over, the deficit grows for as long
        # as the scene runs and the prefill this window reports is just how far
        # it got in 120 frames -- no ring size fixes that, and quoting a KB
        # figure for it would be the most flattering possible way to state a
        # sustained overrun.  FINDINGS 21's "zero prefill" never had to make
        # this distinction because it ran far under the pipe it assumed.
        if wire > a.kbps:
            over = wire - a.kbps
            print(f"    !! SUSTAINED OVERRUN at the {a.kbps:.0f} KB/s pipe: "
                  f"demand exceeds supply by {over:.1f} KB/s on the MEAN, not "
                  f"on a burst.")
            print(f"       The deficit grows {over*1024/FPS:,.0f} B per frame "
                  f"for as long as the scene runs -- {over*1024*120/FPS/1024:.0f} "
                  f"KB over this 120-frame window, {over*60:.0f} KB per minute "
                  f"of play. Prefill below is where it got in 120 frames, NOT a "
                  f"startup cost that fixes it.")

        if maxrec > ring:
            print(f"    !! MAXREC {maxrec:,} > ring {ring:,}: no policy works. "
                  f"decode.s needs one whole record contiguous.\n")
            continue

        # --- the requirement on the medium, which is the useful output, and
        #     the reason this tool takes no default rate.  There is no measured
        #     pipe figure to score against (42.1), and the intent is to measure
        #     a BlueSCSI directly -- so the tool reports the THRESHOLD to
        #     measure against.  The sweep is anchored to the container's own
        #     wire demand rather than to a list of fixed rates, so it stays
        #     meaningful for any container and privileges no constant.
        print(f"    {'pipe KB/s':>10} {'vs wire':>8} {'prefill KB':>11} "
              f"{'records':>8} {'seek slack':>11}")
        for mult in (0.90, 0.95, 1.00, 1.02, 1.05, 1.10, 1.25, 1.50, 2.00):
            kbps = wire * mult
            fill = (kbps - RC.AUDIO_KBPS) * 1024 / FPS
            r = simulate(rec, fill, ring, "aligned", maxrec)
            pf = r["prefill"]
            # Branch-point seek slack, STATICALLY: with the ring FULL, how many
            # frame times can the fill be zero before the next record is not
            # resident?  It is an upper bound and it assumes the premise that
            # FINDINGS 51.3 took apart -- the ring is NOT full at a branch
            # point, it is empty, and refilling it takes seconds of play.  For
            # the measured figure use tools/analysis/20_seek_slack.py, or the
            # rig itself (tools/bench/pace_run.sh).  Kept here as the ceiling
            # this container's record sizes allow, which is what the rest of
            # this row is about.
            slack = (r["usable"] - maxrec) / rec.mean()
            flag = ""
            if pf + maxrec > r["usable"]:
                flag = "  <- does not fit the ring"
            print(f"    {kbps:>10.1f} {mult:>7.2f}x {pf/1024:>11.1f} "
                  f"{pf/rec.mean():>8.2f} {slack:>8.1f} fr{flag}")
        # smallest pipe needing zero prefill, to 0.1 KB/s
        lo, hi = wire, wire + 400
        for _ in range(40):
            mid = (lo + hi) / 2
            f = (mid - RC.AUDIO_KBPS) * 1024 / FPS
            if simulate(rec, f, ring, "aligned", maxrec)["prefill"] > 0:
                lo = mid
            else:
                hi = mid
        print(f"    ZERO-PREFILL PIPE: {hi:.1f} KB/s   "
              f"({hi - wire:+.1f} KB/s over the wire demand, "
              f"{100*hi/wire - 100:+.1f}%)")
        print(f"    ^ this is the number to measure a medium against. It is a "
              f"REQUIREMENT, not a verdict.")

        # --- the policy trade, at the default pipe
        fill = (a.kbps - RC.AUDIO_KBPS) * 1024 / FPS
        print(f"    wrap policy, at pipe {a.kbps:.0f} KB/s:")
        for policy in ("aligned", "split"):
            r = simulate(rec, fill, ring, policy, maxrec)
            if policy == "aligned":
                print(f"      aligned  wraps {r['wraps']:3}  mean hole "
                      f"{r['hole_mean']/1024:6.1f} KB  usable ring "
                      f"{r['usable']/1024:6.1f} KB "
                      f"({100*r['usable']/ring:.1f}%)  CPU cost 0")
            else:
                pct = 100 * r["copy_clk_per_frame"] / (CPUHZ / FPS)
                print(f"      split    shadow {r['shadow_bytes']/1024:8.1f} KB "
                      f"= {r['copy_clk_per_frame']:8.0f} clk/frame = "
                      f"{pct:.2f}% of the frame budget, forever  RAM cost 0")
        print()

    print("The `none` policy -- decoder wraps its own reads -- is not simulated.")
    print("It has no RAM or copy cost and it is still the expensive one: the")
    print("bounds test lands inside the exact instruction sequences FINDINGS")
    print("30.4 and 40 fitted, so it does not cost a branch, it costs every span")
    print("and per-block constant in the tree being re-measured. FINDINGS 28.3.")


if __name__ == "__main__":
    main()