feedwatch: tell the owner when a feed breaks, comes back, or goes quiet

Every poll's outcome has always been written to source_health and shown on
/status, but only to someone who went and looked. Nobody did: Siliconera has
been refused on 3,769 polls in a row and Digital Foundry on 1,900, and
neither has ever fetched successfully. SendAdminWarning existed and nothing
called it.

A watcher now reads source_health every 15 minutes and sends one notice to
matrix.admin_room covering everything that changed since it last spoke:
feeds that have failed 5 polls in a row, feeds that have fetched since they
were reported down, and feeds that fetch fine but have kept nothing new for 7
days (every healthy source produces something inside about 4). What has been
said is stored on the row (down_alerted_at, quiet_alerted_at), so each
episode is reported once and a restart doesn't repeat it. A notice that
fails to send is not marked, and goes out on the next pass.

last_new_at is new, moved only when a poll keeps a story. Existing rows start
from their newest retained story, or from now if they have none, and a new
source starts at its first poll: nothing on disk says whether a source with
no stories is long dead or was added yesterday, so it gets a full window
before it can be called quiet.

With no admin room configured the watcher doesn't run at all, rather than
marking alerts as delivered to nowhere.
This commit is contained in:
prosolis
2026-09-22 23:14:12 -07:00
parent 517019d4a4
commit c6c107b936
7 changed files with 533 additions and 16 deletions
+8
View File
@@ -96,6 +96,14 @@ func runMigrations(d *sql.DB) error {
addColumnIfMissing(d, "post_log", "url_canonical", "TEXT")
addColumnIfMissing(d, "post_log", "forced", "INTEGER NOT NULL DEFAULT 0")
addColumnIfMissing(d, "round_robin_state", "last_channel", "TEXT")
// Feed alerts. last_new_at is when a source last produced a story Pete kept;
// the two *_alerted_at stamps remember that the owner has already been told a
// feed is down or has gone quiet, so an alert fires once per episode and
// survives a restart. See backfillLastNewAt for what existing rows start with.
addColumnIfMissing(d, "source_health", "last_new_at", "INTEGER")
addColumnIfMissing(d, "source_health", "down_alerted_at", "INTEGER")
addColumnIfMissing(d, "source_health", "quiet_alerted_at", "INTEGER")
backfillLastNewAt(d)
// Occupancy of a shared table. Rows written before the casino went multiplayer
// are solo games and read as NULL, which is exactly what they are.
addColumnIfMissing(d, "game_live_hands", "table_id", "TEXT")