Datawrapper embed resizers (and other inline scripts) were leaking into
reader mode as literal text. Two extraction paths were affected:
- parser.go extractContentText/extractLede stripped only script tags via
htmlTagRe, leaving the JS body behind. This is the path that usually
wins for feeds shipping full content:encoded (e.g. Politico).
- article.go goquery paths call .Text(), which concatenates script source.
Both now drop whole <script>/<style>/<noscript> elements before pulling
text. Paywall detection (JSON-LD) still runs before the goquery strip.
Code review of the personalization/feeds/PWA/push work surfaced ten
confirmed issues, now fixed:
- Web Push delivery bypassed the SSRF guard (unguarded default client);
now routes through safehttp.NewClient with a hard timeout, and the
subscribe handler validates the endpoint URL.
- Push unsubscribe deleted by endpoint with no owner check; added
RemovePushSubscriptionForUser scoped to the signed-in user.
- Byte-slice body/content truncation could split a UTF-8 rune and break
the RSS content:encoded XML; added a rune-safe truncateUTF8 helper.
- Digest sender could permanently starve a user who hid a high-volume
source; step the watermark past a full hidden-source scan window.
- Service worker cached personalized HTML navigations into a shared
cache (identity leak across PWA users); navigations are now
network-only, CACHE_VERSION bumped to v2 to purge stale pages.
- Public /api/article leaked discarded/unclassified bodies; filter to
classified, non-sentinel stories.
- runLocal never started the push sender; digests now fire in -local.
- Push client had no timeout, so one hung endpoint stalled all sends.
- Reader migration resurrected cross-device-cleared reads; gate it
behind a one-time flag so the server stays authoritative.
- Bookmarks count didn't match the classified list filter.
Reader mode presents the stories on a page one at a time in a focused
overlay, marking each read as it's shown. Left/right arrows (or the header
book button / `f`) page through them; read stories dim on the grid. Read
state is device-local in localStorage.
Backing this required actually capturing article bodies, which Pete wasn't
doing — it kept only the RSS <description> lede and discarded content:encoded:
- stories.content column (idempotent migration; old rows fall back to lede)
- parser keeps content:encoded as paragraph-preserving text
- article fetch already done for paywall detection now also returns its body,
so ingest stores the richer of feed-content vs scraped body with no extra
request (prefers the archive snapshot body for paywalled stories)
- GET /api/article?id= serves the stored text; card queries now select id and
expose it as data-id for the reader
Tests cover content extraction, the storage round-trip, and the article
endpoint + card rendering end to end.
The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest
bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an
optional per-source user_agent that falls back to the default bot UA, and
set The Portugal News to a browser UA. Not load-related — 30-min polls and
the IP isn't banned.
- Route image-validation, Matrix image-download, and wayback/archive.today
fetches through safehttp so feed-controlled URLs can't reach loopback/
RFC1918/cloud-metadata IPs (incl. via redirects).
- Cap feed body size with safehttp.LimitedBody (16 MiB) to prevent OOM.
- Fail closed if matrix.pickle_key is unset/<16 chars; drop the hardcoded
"pete_pickle_key" default that silently weakened E2EE-at-rest.
- Gate !post behind a matrix.admins allowlist; empty = disabled (channel
rooms are public, so empty must not mean anyone).
- Reject /\host open-redirect bypass in post-login safeNext.
- Allowlist http(s) schemes for the Matrix link href; escape JSON embedded
in inline <script> (prefs/sources blobs).
Two ingestion changes:
- extractLede replaced HTML tags with empty string, so adjacent
block tags like </p><p> fused words across paragraphs. Replace
tags with a space and collapse whitespace.
- Pull each item's <language> tag (or dc:language) into FeedItem
so the poller can filter on it. Politico Europe publishes the
same story in en / fr / de side-by-side and we want to keep
only one language per source.
- Add internal/safehttp: hardened HTTP client (DNS-resolved dial guard
blocking loopback/RFC1918/CGNAT/link-local, redirect re-validation,
body-size cap) and rewire article/feed/thumb clients through it
- Cap goquery body at 5 MiB so a hostile origin can't OOM the process
- search.js: reject non-http(s) hrefs to block stored XSS via javascript:
- dedup: tracking-param key "CMP" was unreachable (lookup lowercases);
fixed to "cmp" so CMP= is actually stripped from canonical URLs
- ForcePost: postItem now returns bool; on dedup-skip ForcePost returns
false so !post falls back to DB lookup instead of silently consuming
- Bound reaction callbacks behind an 8-slot semaphore; drop overflow
- Add stories indexes on (channel, classified, seen_at DESC),
(classified, seen_at DESC), and partial image_url to kill full scans
in IsKnownImageURL and ORDER BY seen_at hot paths
- Surface FTS5 probe Scan error instead of swallowing it
- Add -local flag: web/RSS-only mode that skips Matrix login and posting,
so the web UI can be exercised against live feeds without credentials
- Add Makefile (build/local/seed/test/clean) that handles Tailwind +
go build in one shot
- Fix Guardian thumbnails: NormalizeImageURL was rewriting width=1200
onto signed i.guim.co.uk URLs, invalidating the s= signature and
returning 401. Leave signed URLs alone and pick the widest
media:content variant up front instead
- Use pete.avif as the header logo, favicon, and footer mark; drop the
unused leaf.svg
Pete moves to a remote host without Ollama access. Every source must
declare a direct_route channel; the classifier, explainer, semantic
dedup, !explain summaries, feed_hint, and the recent_headlines /
classification_log tables are gone. Deterministic dedup (canonical URL,
headline_norm, per-channel cooldown) remains.
- !post falls back to newest unposted story for the channel when the
in-memory queue is empty (the steady state under round-robin).
- Accept ❓️/❔️ (U+FE0F variation selector) as question reactions —
the bare codepoints alone missed clients that render the colored emoji.
- Rewrite Guardian i.guim.co.uk thumbnails to width=1200 so we stop
rejecting real images as "tracking pixels"; relabel the size warning.
- Log decrypt failures and reactions on events not in post_log so future
silent drops surface instead of vanishing.
