Pete's side of the Adventure news feed. Receives structured game-event
facts from gogobee, templates them in Pete's warm-reporter voice, and
publishes to a new /adventure section + live Matrix posts.
- adventure.go: bearer ingest + fact-guard + 13 event templates;
/adventure/{guid} permalink (story.html); per-event SVG emblems at
/adventure/art/{type}.svg (card image + og:image); NoPush suppresses
the live Matrix post (cold-start backfill).
- adventure_digest.go: daily BULLETIN roundup at DigestHour (UTC);
unposted-in-48h = bulletins; marks them digested; per-day ?digest= URL
avoids canonical dedup.
- config AdventureConfig (enabled/ingest_token/channel/digest_hour);
web.New takes the seam + a priority poster; started in main.
- adventure theme colors; thumbURL passes through local emblem paths;
adventure pages are noindex (player-named; gap #5).
Claude-Session: https://claude.ai/code/session_017mEwUmmS7aQTP2NQXj6rUa
The baked atlas sprites (blurred-circle clouds, gradient moon with flat
maria blobs) read as cheap. Clouds are now built per-instance in the
particle fragment shader from fbm density with a flattened base and lit
tops; the moon renders in the sky shader with sphere shading, emboss
relief, maria and a halo. Stars skip the moon disc since the sky pass
draws first.
The background weather is now GPU-rendered: one instanced-quad draw call
over a baked sprite atlas plus a fullscreen sky shader (fog, Saharan
haze, aurora, sun rays, storm gloom and lightning flash). The old
Canvas2D renderer stays as weather-2d.js and kicks in automatically
when WebGL2 is missing; weather.js is now a thin controller that owns
the toggle, prefs and the PeteWeather API.
Effect upgrades: shared wind with gust pulses leans the whole scene
together, rain gets depth, splash pops and velocity-aligned streaks,
storms grow procedural branched lightning bolts, snow mixes soft motes
with spinning six-arm crystals, clouds drift in two parallax layers,
clear nights get a moon with maria, twinkling stars and the occasional
shooting star, blossoms and leaves tumble with a faked 3D flip.
New variants: haze (Saharan calima), wind (autumn gusts with streak
lines), hail (bouncing stones with drizzle) and aurora. The /weather
demo page switches variant, intensity and phase in place without a
reload and shows the active renderer plus an FPS meter.
The reader overlay opened without moving focus, so Arrow-Up/Down and
PageUp/Down scrolled the page behind it until the user clicked into the
text. Make the scroll container focusable (tabindex=-1) and focus it on
open, and drop its focus outline.
Reader read-aloud now streams neural WAV audio from a new POST /api/tts
endpoint that shells out to Piper, instead of the browser's Web Speech
voice. Each paragraph is synthesized on demand with the next one
prefetched during playback, keeping the existing highlight/scroll sync.
Voices are configured under [web.tts] (piper binary + voices_dir + a
labelled voice list) and exposed to the client as window.PETE_TTS; the
reader gets a Voice selector in the Aa menu, persisted per-device. Still
a signed-in-only perk and gated on auth.
The .pete-reader-typemenu class sets display:flex, which outranks the
UA [hidden]{display:none} rule, so toggling typeMenu.hidden never hid
the popover. Add an explicit [hidden] guard.
The N-min-read chip derived reading time via LENGTH(content) over the
full article-body TEXT column on every listing render. LENGTH can't use
an index, so SQLite read each row's whole body per request on the hottest
path. Cache the character count in a content_chars column filled at insert
time (backfilled for existing rows), and point StoryContentLengths at it.
- reader.js: guard read-aloud against the loading placeholder (bodyReady)
and invalidate stale speechSynthesis callbacks with a generation token
- storage: check rows.Err() after iterating story-view/content-length reads
- metrics: reuse placeholders() instead of duplicating the IN-clause builder
Surface read counts and sharpen the reader:
- story_views table + RecordStoryView on /api/article (background,
filter-guarded); "Popular this week" home rail via TrendingStories;
read-count badge and reading-time chip decorated onto every listing
- reader: signed-in-only read-aloud (TTS), native share/copy, and an
Aa typography popover (size/serif/sepia) persisted per device
- real alt text on card/reader/related/search images; time-of-day Pete
greeting on the home hero
- harden exec() to skip (not panic) on a nil DB so background writes
can't crash on a closed handle
Tests: story_views_test.go, trending_test.go. Suite green, CSS rebuilt.
Datawrapper embed resizers (and other inline scripts) were leaking into
reader mode as literal text. Two extraction paths were affected:
- parser.go extractContentText/extractLede stripped only script tags via
htmlTagRe, leaving the JS body behind. This is the path that usually
wins for feeds shipping full content:encoded (e.g. Politico).
- article.go goquery paths call .Text(), which concatenates script source.
Both now drop whole <script>/<style>/<noscript> elements before pulling
text. Paywall detection (JSON-LD) still runs before the goquery strip.
The account/logout button was a loose justify-between flex child alongside
all the utility icons, so on mobile it wrapped onto a row by itself. Split
the header into a brand bar (logo + account) and a controls bar (utilities +
search + nav) so the account is pinned top-right and never orphaned.
/status was admin-only (404 for everyone else). Serve it to all: a
reader view with per-feed live/idle/delayed status and last-update time,
while admins additionally get poll cadence, item/story counts, paywall
rates, posting times, and raw fetch errors. Error strings are stripped
server-side for non-admins so feed-specific workarounds and upstream URLs
never reach the public payload. Nav status link now shows for everyone.
- push digest queries now exclude _duplicate channel like every other
visibility query (bookmarks list/count and NewClassifiedSince)
- advance push watermark to newest scanned story, not pass-start now, so
stories arriving during a long send pass aren't re-counted next pass
- replace hand-rolled escapeHTMLText with stdlib html.EscapeString
- drop em-dashes from user-facing copy; bump PWA CACHE_VERSION so clients
pick up the changed shell assets
Code review of the personalization/feeds/PWA/push work surfaced ten
confirmed issues, now fixed:
- Web Push delivery bypassed the SSRF guard (unguarded default client);
now routes through safehttp.NewClient with a hard timeout, and the
subscribe handler validates the endpoint URL.
- Push unsubscribe deleted by endpoint with no owner check; added
RemovePushSubscriptionForUser scoped to the signed-in user.
- Byte-slice body/content truncation could split a UTF-8 rune and break
the RSS content:encoded XML; added a rune-safe truncateUTF8 helper.
- Digest sender could permanently starve a user who hid a high-volume
source; step the watermark past a full hidden-source scan window.
- Service worker cached personalized HTML navigations into a shared
cache (identity leak across PWA users); navigations are now
network-only, CACHE_VERSION bumped to v2 to purge stale pages.
- Public /api/article leaked discarded/unclassified bodies; filter to
classified, non-sentinel stories.
- runLocal never started the push sender; digests now fire in -local.
- Push client had no timeout, so one hung endpoint stalled all sends.
- Reader migration resurrected cross-device-cleared reads; gate it
behind a one-time flag so the server stays authoritative.
- Bookmarks count didn't match the classified list filter.
A multi-session build turning Pete's read-only web UI into something people
return to. Five phases, signed-in features keyed off the OIDC subject; anonymous
visitors keep the reverse-chron feed and localStorage-only state.
Phase 1 — per-user read + bookmark state: user_story_state table +
storage/userstate.go; auth-gated /api/read, /api/bookmark, /api/state and a
/bookmarks page; reader.js syncs state server-side for signed-in users. Also
hides the Matrix-posting UI when posting.enabled=false (web-only mode).
Phase 2 — outbound feeds: storage.ListForFeed + web/feed.go hand-build RSS 2.0
(content:encoded) and JSON Feed 1.1 (no new dep); /feed.xml, /feed.json and
per-channel variants; <link rel=alternate> discovery tags.
Phase 3 — "For you" + related: storage/rank.go scores recent unread candidates
by channel/source affinity + recency decay; RelatedStories via FTS5. ForYou rail
+ /for-you page; public /api/related feeds the reader's "You might also like".
Phase 4 — source-health dashboard: source_health table + storage/sourcehealth.go
(RecordPollResult, ListSourceHealth, SourceContentStats), written by the poller;
admin-gated /status page behind web.admin_subs.
Phase 5 — PWA + offline reader + Web Push: root-scoped manifest.webmanifest and
sw.js (app-shell precache, /api/article runtime cache for offline reading,
offline fallback, push/notificationclick handlers); PNG icons from pete.avif;
pwa.js registers the SW and drives a notifications toggle. Web Push adds
webpush-go, a [web.push] config block (pete -genvapid mints VAPID keys), a
push_subscriptions table, auth-gated subscribe/unsubscribe endpoints, and a
digest sender that pings each subscriber "N new stories" past their watermark,
honoring disabled-sources and pruning gone endpoints.
Tests added beside each new storage/web file; go test ./... and go vet clean.
Reader mode presents the stories on a page one at a time in a focused
overlay, marking each read as it's shown. Left/right arrows (or the header
book button / `f`) page through them; read stories dim on the grid. Read
state is device-local in localStorage.
Backing this required actually capturing article bodies, which Pete wasn't
doing — it kept only the RSS <description> lede and discarded content:encoded:
- stories.content column (idempotent migration; old rows fall back to lede)
- parser keeps content:encoded as paragraph-preserving text
- article fetch already done for paywall detection now also returns its body,
so ingest stores the richer of feed-content vs scraped body with no extra
request (prefers the archive snapshot body for paywalled stories)
- GET /api/article?id= serves the stored text; card queries now select id and
expose it as data-id for the reader
Tests cover content extraction, the storage round-trip, and the article
endpoint + card rendering end to end.
Stories are still ingested, classified, and served to the web UI; only
automatic Matrix posting is gated. Command replies (!post, !petestats)
still work. Pointer-bool so an absent key defaults to posting-on.
Track per-page/per-channel view counts and a privacy-preserving daily
unique-visitor estimate (salted IP+UA hash, salt rotated daily and never
persisted). No third-party analytics, no JS beacon. Surfaced via the
admin-gated !petestats Matrix command (named to avoid an existing !stats
bot in the rooms).
Reuses the saved weather location's lat/lon to fetch us_aqi from
Open-Meteo's air-quality API (no key), folded into the existing 2h
forecast cache (bumped v1->v2). AQI is best-effort: a failed fetch
never sinks the forecast. Shows as a header chip and a colored row
on the forecast card; both self-hide when there is no reading.
The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest
bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an
optional per-source user_agent that falls back to the default bot UA, and
set The Portugal News to a browser UA. Not load-related — 30-min polls and
the IP isn't banned.
- Route image-validation, Matrix image-download, and wayback/archive.today
fetches through safehttp so feed-controlled URLs can't reach loopback/
RFC1918/cloud-metadata IPs (incl. via redirects).
- Cap feed body size with safehttp.LimitedBody (16 MiB) to prevent OOM.
- Fail closed if matrix.pickle_key is unset/<16 chars; drop the hardcoded
"pete_pickle_key" default that silently weakened E2EE-at-rest.
- Gate !post behind a matrix.admins allowlist; empty = disabled (channel
rooms are public, so empty must not mean anyone).
- Reject /\host open-redirect bypass in post-login safeNext.
- Allowlist http(s) schemes for the Matrix link href; escape JSON embedded
in inline <script> (prefs/sources blobs).
Signed-in users get their preferences (hidden feeds, weather location,
weather toggle) stored server-side keyed by their OIDC subject and synced
across devices. Anonymous visitors keep using browser localStorage, so the
site stays public. First sign-in migrates existing localStorage prefs up.
- config: [web.auth] section (issuer, client_id/secret, redirect, session_secret)
- storage: user_preferences table + Get/PutUserPrefs
- web/auth: OIDC code flow, HMAC-signed session cookie, CSRF state + nonce
- web/prefs_api: GET/PUT /api/preferences (auth-gated, 64KB cap)
- frontend: prefs.js sync layer seeds localStorage from server, pushes on write
- header: sign-in / account control
OIDC discovery is non-fatal at boot: if Authentik is down, Pete serves
anonymously rather than refusing to start.
Visitors can save a postal code (international, via Zippopotam) to get the
local forecast from Open-Meteo — a header chip + a 5-day home-page card —
and the canvas background switches from the seasonal effect to live
conditions. Entirely client-side: no keys, no server logic. Geocode cached
permanently, forecast cached 2h. Celsius by default, Fahrenheit opt-in.
New canvas effects: clear (sun by day, shaded moon + stars at night),
clouds (blurred drifting sprites), snow, fog, and storm (rain + lightning).
Seasonal effects remain the no-location fallback.
LEGO channel (🧱, red theme) posts to Matrix with Brickset + The Brick Fan
feeds. Finance gains MarketBeat (analyst ratings/upgrades) and Seeking Alpha
(tier 2; expect paywall stamps).
Two bugs prevented Pete's device from ever cross-signing itself:
1. Double login / split-brain: New() did a manual mx.Login() AND set
ch.LoginAs, so cryptohelper.Init() logged in a second time on a fresh
crypto store, minting a separate device. device.json recorded one device
while the olm account belonged to another, and every cold start leaked an
orphan device. Pete's outgoing events were attributed to a device whose
keys no client could verify. Removed ch.LoginAs (auth is already handled by
the manual login + isTokenValid re-login path).
2. Unreachable bootstrap: the reset gate used
IsDeviceTrusted(mach.OwnIdentity()), but OwnIdentity() hard-codes
Trust=TrustStateVerified, so it always returns true and the bootstrap/reset
branch was never entered. Replaced with GetOwnVerificationStatus(), which
actually checks whether the current device key is signed by our
self-signing key.
New web-only category mirroring the EU channel: shows in the web UI but
does not post to Matrix. Adds the channel entry, emerald (money green)
theme utilities, and two macro/finance feeds (Naked Capitalism, Wolf
Street) to the example config.
- New kids channel (/kids) with theme color, surfaced in the web UI
and routed to by BBC Newsround, DOGO News, Science News Explores,
and NASA for Students in the example config.
- README and config.example.toml now mention the optional
per-source `language` filter added in the previous commit.
The 0.5s seek often landed on the initial keyframe or a fade-in,
producing blocky thumbnails. Aim ~5s in instead, with an ffprobe
midpoint fallback for clips shorter than that and a 2s → 0
retry chain if the seek still overshoots.
btoa() throws InvalidCharacterError on non-Latin1 input, so feed
names with em-dashes ("The Guardian — World") killed render()
before the dialog's hidden class was removed and nothing visibly
happened on click. Use the loop index for the row id instead.
When a source sets language = "en", drop items whose per-item
language tag is present and doesn't prefix-match. Items without
a language tag pass through unchanged. Politico Europe is the
motivating case — same headlines appear in en, fr, and de.
Two ingestion changes:
- extractLede replaced HTML tags with empty string, so adjacent
block tags like </p><p> fused words across paragraphs. Replace
tags with a space and collapse whitespace.
- Pull each item's <language> tag (or dc:language) into FeedItem
so the poller can filter on it. Politico Europe publishes the
same story in en / fr / de side-by-side and we want to keep
only one language per source.
Manual !post overrides were counted toward daily_cap_total, so a few
forced posts could starve the round-robin rotation for the rest of the
day. Tag forced rows in post_log and skip them in CountAllPostsInWindow
so the cap only meters the auto-rotation.
Visitors can hide individual feeds via a gear-icon panel in the header.
Preferences live in localStorage; the server ships the full source list
(name + channel) as window.PETE_SOURCES so the panel lists every feed,
grouped by channel, regardless of what's on the current page.
Go's stdlib image/jpeg refuses some valid-but-rare features such as
4:1:1 luma/chroma subsampling (ANN's CDN serves these). When the
in-process decode fails, route the bytes through ffmpeg the same way
we already do for video sources.
Channel pill row outgrew the viewport once more channels were added,
pushing the whole page sideways. Let the header wrap, scroll the nav
inside its own pill, and clip body overflow as a safety net.
mp4/webm/mov/m4v/mkv URLs now route to a frame-extraction path with a
larger 64 MiB download cap, and ffmpeg pulls a single resized frame
that the existing avifenc step turns into the cached thumb. ffmpeg is
optional: if missing, we fall through like any other build failure
and the handler redirects to the source URL.
LWN's subscriber-only articles have no public version reachable by
archive snapshots or bypass UAs, so stamping them as paywalled produces
clicks that always dead-end. Detect the marker text and skip ingestion
entirely.
ANN wraps article bodies in <div class="KonaBody"> with no semantic
<article>/<main> tags, so body-length extraction fell below the 500-char
threshold and the poller flagged stories as gated. Broaden the container
fallback to also try itemprop="articleBody", common content-container
classes, and ANN's KonaBody.
- Add internal/safehttp: hardened HTTP client (DNS-resolved dial guard
blocking loopback/RFC1918/CGNAT/link-local, redirect re-validation,
body-size cap) and rewire article/feed/thumb clients through it
- Cap goquery body at 5 MiB so a hostile origin can't OOM the process
- search.js: reject non-http(s) hrefs to block stored XSS via javascript:
- dedup: tracking-param key "CMP" was unreachable (lookup lowercases);
fixed to "cmp" so CMP= is actually stripped from canonical URLs
- ForcePost: postItem now returns bool; on dedup-skip ForcePost returns
false so !post falls back to DB lookup instead of silently consuming
- Bound reaction callbacks behind an 8-slot semaphore; drop overflow
- Add stories indexes on (channel, classified, seen_at DESC),
(classified, seen_at DESC), and partial image_url to kill full scans
in IsKnownImageURL and ORDER BY seen_at hot paths
- Surface FTS5 probe Scan error instead of swallowing it
- Body extractor falls back to <article>/<main> container text when
<p> extraction is sparse, catching <br>-separated bodies (Phoronix).
- Detect Cloudflare bot-block / JS-challenge pages on 403/503 and
treat them as transport failures rather than paywalls (Brooklyn Vegan).
- og:image extractor falls back to img.wp-post-image and the first
content <img> in <article>/<main>, with lazy-load placeholder
handling via data-src / data-lazy-src (Hardcore Gaming 101).
- New -backfill-paywall flag re-checks paywalled=1 rows with the
current logic, clearing false positives and filling missing thumbs.
Two new web-only channels alongside EU: anime (🌸 sakura pink #ec5e8a)
and foss (🐧 amber #d97706). Both get the full bg/text/decoration/
border/glow theme classes.
Search: new FTS5-backed SearchStories query, /search JSON endpoint,
client-side overlay (search.js) wired into the layout header. EU
channel gets its own theme color (#003399) across bg/text/border/glow
classes. Sources can now route to non-Matrix channels without
validation error (web-only mode); a warning still flags typos.
Bypass-UA retry (Googlebot + Google referer) for soft paywalls, JSON-LD
gating scoped to Article-typed nodes, HTTP 402 treated as explicit
paywall, Wayback freshness filter (30d cap), archive.today as secondary
archive fallback, and transport failures no longer trigger snapshot
swaps. When gating is detected and no archive workaround succeeds, the
story is stored with paywalled=1 and the web card renders a diagonal
red rubber-stamp overlay so readers know the link is gated.
Decode AVIF via avifdec when Go's image.Decode can't, and pass-through
small AVIFs (<=800px wide) by caching the original bytes instead of
re-encoding.
Server picks variant from Lisbon-local date (rain/petals/jacaranda/motes/leaves)
and renders behind content via a canvas particle layer. Each variant has a
hand-drawn silhouette so shapes are recognizable. /weather demo route exposes
variant + intensity + phase pickers, locking the time-of-day phase override.
