Two ingestion changes:
- extractLede replaced HTML tags with empty string, so adjacent
block tags like </p><p> fused words across paragraphs. Replace
tags with a space and collapse whitespace.
- Pull each item's <language> tag (or dc:language) into FeedItem
so the poller can filter on it. Politico Europe publishes the
same story in en / fr / de side-by-side and we want to keep
only one language per source.
- Add internal/safehttp: hardened HTTP client (DNS-resolved dial guard
blocking loopback/RFC1918/CGNAT/link-local, redirect re-validation,
body-size cap) and rewire article/feed/thumb clients through it
- Cap goquery body at 5 MiB so a hostile origin can't OOM the process
- search.js: reject non-http(s) hrefs to block stored XSS via javascript:
- dedup: tracking-param key "CMP" was unreachable (lookup lowercases);
fixed to "cmp" so CMP= is actually stripped from canonical URLs
- ForcePost: postItem now returns bool; on dedup-skip ForcePost returns
false so !post falls back to DB lookup instead of silently consuming
- Bound reaction callbacks behind an 8-slot semaphore; drop overflow
- Add stories indexes on (channel, classified, seen_at DESC),
(classified, seen_at DESC), and partial image_url to kill full scans
in IsKnownImageURL and ORDER BY seen_at hot paths
- Surface FTS5 probe Scan error instead of swallowing it
- Add -local flag: web/RSS-only mode that skips Matrix login and posting,
so the web UI can be exercised against live feeds without credentials
- Add Makefile (build/local/seed/test/clean) that handles Tailwind +
go build in one shot
- Fix Guardian thumbnails: NormalizeImageURL was rewriting width=1200
onto signed i.guim.co.uk URLs, invalidating the s= signature and
returning 401. Leave signed URLs alone and pick the widest
media:content variant up front instead
- Use pete.avif as the header logo, favicon, and footer mark; drop the
unused leaf.svg
Pete moves to a remote host without Ollama access. Every source must
declare a direct_route channel; the classifier, explainer, semantic
dedup, !explain summaries, feed_hint, and the recent_headlines /
classification_log tables are gone. Deterministic dedup (canonical URL,
headline_norm, per-channel cooldown) remains.
- !post falls back to newest unposted story for the channel when the
in-memory queue is empty (the steady state under round-robin).
- Accept ❓️/❔️ (U+FE0F variation selector) as question reactions —
the bare codepoints alone missed clients that render the colored emoji.
- Rewrite Guardian i.guim.co.uk thumbnails to width=1200 so we stop
rejecting real images as "tracking pixels"; relabel the size warning.
- Log decrypt failures and reactions on events not in post_log so future
silent drops surface instead of vanishing.
