Datawrapper embed resizers (and other inline scripts) were leaking into
reader mode as literal text. Two extraction paths were affected:
- parser.go extractContentText/extractLede stripped only script tags via
htmlTagRe, leaving the JS body behind. This is the path that usually
wins for feeds shipping full content:encoded (e.g. Politico).
- article.go goquery paths call .Text(), which concatenates script source.
Both now drop whole <script>/<style>/<noscript> elements before pulling
text. Paywall detection (JSON-LD) still runs before the goquery strip.
Code review of the personalization/feeds/PWA/push work surfaced ten
confirmed issues, now fixed:
- Web Push delivery bypassed the SSRF guard (unguarded default client);
now routes through safehttp.NewClient with a hard timeout, and the
subscribe handler validates the endpoint URL.
- Push unsubscribe deleted by endpoint with no owner check; added
RemovePushSubscriptionForUser scoped to the signed-in user.
- Byte-slice body/content truncation could split a UTF-8 rune and break
the RSS content:encoded XML; added a rune-safe truncateUTF8 helper.
- Digest sender could permanently starve a user who hid a high-volume
source; step the watermark past a full hidden-source scan window.
- Service worker cached personalized HTML navigations into a shared
cache (identity leak across PWA users); navigations are now
network-only, CACHE_VERSION bumped to v2 to purge stale pages.
- Public /api/article leaked discarded/unclassified bodies; filter to
classified, non-sentinel stories.
- runLocal never started the push sender; digests now fire in -local.
- Push client had no timeout, so one hung endpoint stalled all sends.
- Reader migration resurrected cross-device-cleared reads; gate it
behind a one-time flag so the server stays authoritative.
- Bookmarks count didn't match the classified list filter.
A multi-session build turning Pete's read-only web UI into something people
return to. Five phases, signed-in features keyed off the OIDC subject; anonymous
visitors keep the reverse-chron feed and localStorage-only state.
Phase 1 — per-user read + bookmark state: user_story_state table +
storage/userstate.go; auth-gated /api/read, /api/bookmark, /api/state and a
/bookmarks page; reader.js syncs state server-side for signed-in users. Also
hides the Matrix-posting UI when posting.enabled=false (web-only mode).
Phase 2 — outbound feeds: storage.ListForFeed + web/feed.go hand-build RSS 2.0
(content:encoded) and JSON Feed 1.1 (no new dep); /feed.xml, /feed.json and
per-channel variants; <link rel=alternate> discovery tags.
Phase 3 — "For you" + related: storage/rank.go scores recent unread candidates
by channel/source affinity + recency decay; RelatedStories via FTS5. ForYou rail
+ /for-you page; public /api/related feeds the reader's "You might also like".
Phase 4 — source-health dashboard: source_health table + storage/sourcehealth.go
(RecordPollResult, ListSourceHealth, SourceContentStats), written by the poller;
admin-gated /status page behind web.admin_subs.
Phase 5 — PWA + offline reader + Web Push: root-scoped manifest.webmanifest and
sw.js (app-shell precache, /api/article runtime cache for offline reading,
offline fallback, push/notificationclick handlers); PNG icons from pete.avif;
pwa.js registers the SW and drives a notifications toggle. Web Push adds
webpush-go, a [web.push] config block (pete -genvapid mints VAPID keys), a
push_subscriptions table, auth-gated subscribe/unsubscribe endpoints, and a
digest sender that pings each subscriber "N new stories" past their watermark,
honoring disabled-sources and pruning gone endpoints.
Tests added beside each new storage/web file; go test ./... and go vet clean.
Reader mode presents the stories on a page one at a time in a focused
overlay, marking each read as it's shown. Left/right arrows (or the header
book button / `f`) page through them; read stories dim on the grid. Read
state is device-local in localStorage.
Backing this required actually capturing article bodies, which Pete wasn't
doing — it kept only the RSS <description> lede and discarded content:encoded:
- stories.content column (idempotent migration; old rows fall back to lede)
- parser keeps content:encoded as paragraph-preserving text
- article fetch already done for paywall detection now also returns its body,
so ingest stores the richer of feed-content vs scraped body with no extra
request (prefers the archive snapshot body for paywalled stories)
- GET /api/article?id= serves the stored text; card queries now select id and
expose it as data-id for the reader
Tests cover content extraction, the storage round-trip, and the article
endpoint + card rendering end to end.
The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest
bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an
optional per-source user_agent that falls back to the default bot UA, and
set The Portugal News to a browser UA. Not load-related — 30-min polls and
the IP isn't banned.
- Route image-validation, Matrix image-download, and wayback/archive.today
fetches through safehttp so feed-controlled URLs can't reach loopback/
RFC1918/cloud-metadata IPs (incl. via redirects).
- Cap feed body size with safehttp.LimitedBody (16 MiB) to prevent OOM.
- Fail closed if matrix.pickle_key is unset/<16 chars; drop the hardcoded
"pete_pickle_key" default that silently weakened E2EE-at-rest.
- Gate !post behind a matrix.admins allowlist; empty = disabled (channel
rooms are public, so empty must not mean anyone).
- Reject /\host open-redirect bypass in post-login safeNext.
- Allowlist http(s) schemes for the Matrix link href; escape JSON embedded
in inline <script> (prefs/sources blobs).
When a source sets language = "en", drop items whose per-item
language tag is present and doesn't prefix-match. Items without
a language tag pass through unchanged. Politico Europe is the
motivating case — same headlines appear in en, fr, and de.
Two ingestion changes:
- extractLede replaced HTML tags with empty string, so adjacent
block tags like </p><p> fused words across paragraphs. Replace
tags with a space and collapse whitespace.
- Pull each item's <language> tag (or dc:language) into FeedItem
so the poller can filter on it. Politico Europe publishes the
same story in en / fr / de side-by-side and we want to keep
only one language per source.
LWN's subscriber-only articles have no public version reachable by
archive snapshots or bypass UAs, so stamping them as paywalled produces
clicks that always dead-end. Detect the marker text and skip ingestion
entirely.
ANN wraps article bodies in <div class="KonaBody"> with no semantic
<article>/<main> tags, so body-length extraction fell below the 500-char
threshold and the poller flagged stories as gated. Broaden the container
fallback to also try itemprop="articleBody", common content-container
classes, and ANN's KonaBody.
- Add internal/safehttp: hardened HTTP client (DNS-resolved dial guard
blocking loopback/RFC1918/CGNAT/link-local, redirect re-validation,
body-size cap) and rewire article/feed/thumb clients through it
- Cap goquery body at 5 MiB so a hostile origin can't OOM the process
- search.js: reject non-http(s) hrefs to block stored XSS via javascript:
- dedup: tracking-param key "CMP" was unreachable (lookup lowercases);
fixed to "cmp" so CMP= is actually stripped from canonical URLs
- ForcePost: postItem now returns bool; on dedup-skip ForcePost returns
false so !post falls back to DB lookup instead of silently consuming
- Bound reaction callbacks behind an 8-slot semaphore; drop overflow
- Add stories indexes on (channel, classified, seen_at DESC),
(classified, seen_at DESC), and partial image_url to kill full scans
in IsKnownImageURL and ORDER BY seen_at hot paths
- Surface FTS5 probe Scan error instead of swallowing it
- Body extractor falls back to <article>/<main> container text when
<p> extraction is sparse, catching <br>-separated bodies (Phoronix).
- Detect Cloudflare bot-block / JS-challenge pages on 403/503 and
treat them as transport failures rather than paywalls (Brooklyn Vegan).
- og:image extractor falls back to img.wp-post-image and the first
content <img> in <article>/<main>, with lazy-load placeholder
handling via data-src / data-lazy-src (Hardcore Gaming 101).
- New -backfill-paywall flag re-checks paywalled=1 rows with the
current logic, clearing false positives and filling missing thumbs.
Bypass-UA retry (Googlebot + Google referer) for soft paywalls, JSON-LD
gating scoped to Article-typed nodes, HTTP 402 treated as explicit
paywall, Wayback freshness filter (30d cap), archive.today as secondary
archive fallback, and transport failures no longer trigger snapshot
swaps. When gating is detected and no archive workaround succeeds, the
story is stored with paywalled=1 and the web card renders a diagonal
red rubber-stamp overlay so readers know the link is gated.
- Detect explicit paywall markers (article:content_tier meta, JSON-LD
isAccessibleForFree) so metered articles fall back to Wayback even
when body length is above threshold
- Blank-import go.mau.fi/util/dbutil/litestream so the sqlite3-fk-wal
driver mautrix's cryptohelper depends on is registered
- Fix /img route: Go ServeMux requires {wildcard} to be a whole segment,
so capture {name} and strip the .avif suffix in the handler
- Add -local flag: web/RSS-only mode that skips Matrix login and posting,
so the web UI can be exercised against live feeds without credentials
- Add Makefile (build/local/seed/test/clean) that handles Tailwind +
go build in one shot
- Fix Guardian thumbnails: NormalizeImageURL was rewriting width=1200
onto signed i.guim.co.uk URLs, invalidating the s= signature and
returning 401. Leave signed URLs alone and pick the widest
media:content variant up front instead
- Use pete.avif as the header logo, favicon, and footer mark; drop the
unused leaf.svg
Pete moves to a remote host without Ollama access. Every source must
declare a direct_route channel; the classifier, explainer, semantic
dedup, !explain summaries, feed_hint, and the recent_headlines /
classification_log tables are gone. Deterministic dedup (canonical URL,
headline_norm, per-channel cooldown) remains.
- !post falls back to newest unposted story for the channel when the
in-memory queue is empty (the steady state under round-robin).
- Accept ❓️/❔️ (U+FE0F variation selector) as question reactions —
the bare codepoints alone missed clients that render the colored emoji.
- Rewrite Guardian i.guim.co.uk thumbnails to width=1200 so we stop
rejecting real images as "tracking pixels"; relabel the size warning.
- Log decrypt failures and reactions on events not in post_log so future
silent drops surface instead of vanishing.
OllamaClient.Generate/GenerateText/call now take ctx and build the HTTP
request via NewRequestWithContext, so an in-flight LLM call is aborted
when the parent context is cancelled (Ctrl-C). Classifier.Classify and
its tier helpers take ctx too. ProcessFunc gets a ctx parameter so the
poller can forward its cancellable context down to classification.
Explainer.summarize manages its own 60s context since reaction-driven
flow has no parent ctx to inherit.
Four related fixes after Pete flooded a channel and ignored Ctrl-C:
1. Global daily cap (posting.daily_cap_total, default 5): hard ceiling on
posts across ALL channels in a rolling 24h window. Checked before the
per-channel min-interval and burst-cap.
2. Shutdown no longer flushes the queue. Previous drainAll posted every
remaining item with rate limits disabled — which was literally the
flood. Replaced with dropOnShutdown that clears queues and logs
the count.
3. Poller respects ctx mid-loop. pollOnceWithErr now takes ctx and bails
between items, so Ctrl-C doesn't have to wait for ~30s of network
per pending story before shutdown can complete.
4. Double-image fix. PostStory now reports imageSent; the queue clears
ImageURL before retry so a text-send failure after a successful
image upload doesn't re-post the image.
When a user reacts ❓ on one of Pete's posts, fetch the article body,
ask Ollama for a 3-bullet summary, and post it as a threaded reply
rooted at the original story event. Per-process cooldown of 5min per
story keeps repeated reactions from re-summarizing.
- ingestion.FetchArticleBody: visible <p> text capped at 8000 chars
- classifier.OllamaClient.GenerateText: non-JSON variant
- storage.GetStoryByGUID: full row lookup
- matrix.PostThreadedReply: m.thread + m.in_reply_to fallback
- poster.SetReactionCallback: optional hook fired after recording
- New package: internal/explainer
Fetch each new article once and measure visible <p> text. If body is below
500 chars (or fetch fails), resolve a Wayback snapshot via the
archive.org/wayback/available API and use that URL for both the og:image
fallback and the posted link. Dedup keys stay derived from the original
URL so paywalled/non-paywalled hits collide as before.
- New: internal/ingestion/article.go (FetchArticleMeta via goquery)
- New: internal/ingestion/wayback.go (ResolveWayback)
- Removed: internal/ingestion/og.go (folded into article.go)
- poller.go: dedup first, then one article fetch, then snapshot fallback
