A multi-session build turning Pete's read-only web UI into something people
return to. Five phases, signed-in features keyed off the OIDC subject; anonymous
visitors keep the reverse-chron feed and localStorage-only state.
Phase 1 — per-user read + bookmark state: user_story_state table +
storage/userstate.go; auth-gated /api/read, /api/bookmark, /api/state and a
/bookmarks page; reader.js syncs state server-side for signed-in users. Also
hides the Matrix-posting UI when posting.enabled=false (web-only mode).
Phase 2 — outbound feeds: storage.ListForFeed + web/feed.go hand-build RSS 2.0
(content:encoded) and JSON Feed 1.1 (no new dep); /feed.xml, /feed.json and
per-channel variants; <link rel=alternate> discovery tags.
Phase 3 — "For you" + related: storage/rank.go scores recent unread candidates
by channel/source affinity + recency decay; RelatedStories via FTS5. ForYou rail
+ /for-you page; public /api/related feeds the reader's "You might also like".
Phase 4 — source-health dashboard: source_health table + storage/sourcehealth.go
(RecordPollResult, ListSourceHealth, SourceContentStats), written by the poller;
admin-gated /status page behind web.admin_subs.
Phase 5 — PWA + offline reader + Web Push: root-scoped manifest.webmanifest and
sw.js (app-shell precache, /api/article runtime cache for offline reading,
offline fallback, push/notificationclick handlers); PNG icons from pete.avif;
pwa.js registers the SW and drives a notifications toggle. Web Push adds
webpush-go, a [web.push] config block (pete -genvapid mints VAPID keys), a
push_subscriptions table, auth-gated subscribe/unsubscribe endpoints, and a
digest sender that pings each subscriber "N new stories" past their watermark,
honoring disabled-sources and pruning gone endpoints.
Tests added beside each new storage/web file; go test ./... and go vet clean.
Reader mode presents the stories on a page one at a time in a focused
overlay, marking each read as it's shown. Left/right arrows (or the header
book button / `f`) page through them; read stories dim on the grid. Read
state is device-local in localStorage.
Backing this required actually capturing article bodies, which Pete wasn't
doing — it kept only the RSS <description> lede and discarded content:encoded:
- stories.content column (idempotent migration; old rows fall back to lede)
- parser keeps content:encoded as paragraph-preserving text
- article fetch already done for paywall detection now also returns its body,
so ingest stores the richer of feed-content vs scraped body with no extra
request (prefers the archive snapshot body for paywalled stories)
- GET /api/article?id= serves the stored text; card queries now select id and
expose it as data-id for the reader
Tests cover content extraction, the storage round-trip, and the article
endpoint + card rendering end to end.
The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest
bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an
optional per-source user_agent that falls back to the default bot UA, and
set The Portugal News to a browser UA. Not load-related — 30-min polls and
the IP isn't banned.
When a source sets language = "en", drop items whose per-item
language tag is present and doesn't prefix-match. Items without
a language tag pass through unchanged. Politico Europe is the
motivating case — same headlines appear in en, fr, and de.
LWN's subscriber-only articles have no public version reachable by
archive snapshots or bypass UAs, so stamping them as paywalled produces
clicks that always dead-end. Detect the marker text and skip ingestion
entirely.
Bypass-UA retry (Googlebot + Google referer) for soft paywalls, JSON-LD
gating scoped to Article-typed nodes, HTTP 402 treated as explicit
paywall, Wayback freshness filter (30d cap), archive.today as secondary
archive fallback, and transport failures no longer trigger snapshot
swaps. When gating is detected and no archive workaround succeeds, the
story is stored with paywalled=1 and the web card renders a diagonal
red rubber-stamp overlay so readers know the link is gated.
- Detect explicit paywall markers (article:content_tier meta, JSON-LD
isAccessibleForFree) so metered articles fall back to Wayback even
when body length is above threshold
- Blank-import go.mau.fi/util/dbutil/litestream so the sqlite3-fk-wal
driver mautrix's cryptohelper depends on is registered
- Fix /img route: Go ServeMux requires {wildcard} to be a whole segment,
so capture {name} and strip the .avif suffix in the handler
Pete moves to a remote host without Ollama access. Every source must
declare a direct_route channel; the classifier, explainer, semantic
dedup, !explain summaries, feed_hint, and the recent_headlines /
classification_log tables are gone. Deterministic dedup (canonical URL,
headline_norm, per-channel cooldown) remains.
OllamaClient.Generate/GenerateText/call now take ctx and build the HTTP
request via NewRequestWithContext, so an in-flight LLM call is aborted
when the parent context is cancelled (Ctrl-C). Classifier.Classify and
its tier helpers take ctx too. ProcessFunc gets a ctx parameter so the
poller can forward its cancellable context down to classification.
Explainer.summarize manages its own 60s context since reaction-driven
flow has no parent ctx to inherit.
Four related fixes after Pete flooded a channel and ignored Ctrl-C:
1. Global daily cap (posting.daily_cap_total, default 5): hard ceiling on
posts across ALL channels in a rolling 24h window. Checked before the
per-channel min-interval and burst-cap.
2. Shutdown no longer flushes the queue. Previous drainAll posted every
remaining item with rate limits disabled — which was literally the
flood. Replaced with dropOnShutdown that clears queues and logs
the count.
3. Poller respects ctx mid-loop. pollOnceWithErr now takes ctx and bails
between items, so Ctrl-C doesn't have to wait for ~30s of network
per pending story before shutdown can complete.
4. Double-image fix. PostStory now reports imageSent; the queue clears
ImageURL before retry so a text-send failure after a successful
image upload doesn't re-post the image.
Fetch each new article once and measure visible <p> text. If body is below
500 chars (or fetch fails), resolve a Wayback snapshot via the
archive.org/wayback/available API and use that URL for both the og:image
fallback and the posted link. Dedup keys stay derived from the original
URL so paywalled/non-paywalled hits collide as before.
- New: internal/ingestion/article.go (FetchArticleMeta via goquery)
- New: internal/ingestion/wayback.go (ResolveWayback)
- Removed: internal/ingestion/og.go (folded into article.go)
- poller.go: dedup first, then one article fetch, then snapshot fallback
