Compare commits

...

4 Commits

Author SHA1 Message Date
prosolis
35850eaf73 Fix push-digest sentinel filter, watermark, and cleanups from review
- push digest queries now exclude _duplicate channel like every other
  visibility query (bookmarks list/count and NewClassifiedSince)
- advance push watermark to newest scanned story, not pass-start now, so
  stories arriving during a long send pass aren't re-counted next pass
- replace hand-rolled escapeHTMLText with stdlib html.EscapeString
- drop em-dashes from user-facing copy; bump PWA CACHE_VERSION so clients
  pick up the changed shell assets
2026-07-07 01:25:23 -07:00
prosolis
8863b75916 Fix push SSRF, cross-user unsub, and personalization edge cases
Code review of the personalization/feeds/PWA/push work surfaced ten
confirmed issues, now fixed:

- Web Push delivery bypassed the SSRF guard (unguarded default client);
  now routes through safehttp.NewClient with a hard timeout, and the
  subscribe handler validates the endpoint URL.
- Push unsubscribe deleted by endpoint with no owner check; added
  RemovePushSubscriptionForUser scoped to the signed-in user.
- Byte-slice body/content truncation could split a UTF-8 rune and break
  the RSS content:encoded XML; added a rune-safe truncateUTF8 helper.
- Digest sender could permanently starve a user who hid a high-volume
  source; step the watermark past a full hidden-source scan window.
- Service worker cached personalized HTML navigations into a shared
  cache (identity leak across PWA users); navigations are now
  network-only, CACHE_VERSION bumped to v2 to purge stale pages.
- Public /api/article leaked discarded/unclassified bodies; filter to
  classified, non-sentinel stories.
- runLocal never started the push sender; digests now fire in -local.
- Push client had no timeout, so one hung endpoint stalled all sends.
- Reader migration resurrected cross-device-cleared reads; gate it
  behind a one-time flag so the server stays authoritative.
- Bookmarks count didn't match the classified list filter.
2026-07-07 01:08:42 -07:00
prosolis
71f7050f41 Add personalization, outbound feeds, and PWA/push to the web UI
A multi-session build turning Pete's read-only web UI into something people
return to. Five phases, signed-in features keyed off the OIDC subject; anonymous
visitors keep the reverse-chron feed and localStorage-only state.

Phase 1 — per-user read + bookmark state: user_story_state table +
storage/userstate.go; auth-gated /api/read, /api/bookmark, /api/state and a
/bookmarks page; reader.js syncs state server-side for signed-in users. Also
hides the Matrix-posting UI when posting.enabled=false (web-only mode).

Phase 2 — outbound feeds: storage.ListForFeed + web/feed.go hand-build RSS 2.0
(content:encoded) and JSON Feed 1.1 (no new dep); /feed.xml, /feed.json and
per-channel variants; <link rel=alternate> discovery tags.

Phase 3 — "For you" + related: storage/rank.go scores recent unread candidates
by channel/source affinity + recency decay; RelatedStories via FTS5. ForYou rail
+ /for-you page; public /api/related feeds the reader's "You might also like".

Phase 4 — source-health dashboard: source_health table + storage/sourcehealth.go
(RecordPollResult, ListSourceHealth, SourceContentStats), written by the poller;
admin-gated /status page behind web.admin_subs.

Phase 5 — PWA + offline reader + Web Push: root-scoped manifest.webmanifest and
sw.js (app-shell precache, /api/article runtime cache for offline reading,
offline fallback, push/notificationclick handlers); PNG icons from pete.avif;
pwa.js registers the SW and drives a notifications toggle. Web Push adds
webpush-go, a [web.push] config block (pete -genvapid mints VAPID keys), a
push_subscriptions table, auth-gated subscribe/unsubscribe endpoints, and a
digest sender that pings each subscriber "N new stories" past their watermark,
honoring disabled-sources and pruning gone endpoints.

Tests added beside each new storage/web file; go test ./... and go vet clean.
2026-07-07 00:07:19 -07:00
prosolis
55aa167151 Add feed/reader mode with full-article capture at ingest
Reader mode presents the stories on a page one at a time in a focused
overlay, marking each read as it's shown. Left/right arrows (or the header
book button / `f`) page through them; read stories dim on the grid. Read
state is device-local in localStorage.

Backing this required actually capturing article bodies, which Pete wasn't
doing — it kept only the RSS <description> lede and discarded content:encoded:

- stories.content column (idempotent migration; old rows fall back to lede)
- parser keeps content:encoded as paragraph-preserving text
- article fetch already done for paywall detection now also returns its body,
  so ingest stores the richer of feed-content vs scraped body with no extra
  request (prefers the archive snapshot body for paywalled stories)
- GET /api/article?id= serves the stored text; card queries now select id and
  expose it as data-id for the reader

Tests cover content extraction, the storage round-trip, and the article
endpoint + card rendering end to end.
2026-07-06 22:46:14 -07:00
50 changed files with 4529 additions and 51 deletions

View File

@@ -37,6 +37,12 @@ listen_addr = ":8080"
site_title = "Pete" site_title = "Pete"
base_url = "https://news.parodia.dev" base_url = "https://news.parodia.dev"
# OIDC subjects allowed to view the owner-facing source-health dashboard at
# /status (per-feed poll status, failures, content stats). Requires web.auth
# below. Empty = /status returns 404 for everyone. Find a user's subject in the
# server logs ("auth: user signed in" sub=...) after they sign in once.
admin_subs = []
# Optional OIDC sign-in (Authentik). When enabled, signed-in users get their # Optional OIDC sign-in (Authentik). When enabled, signed-in users get their
# preferences (hidden feeds, weather location, toggles) stored server-side keyed # preferences (hidden feeds, weather location, toggles) stored server-side keyed
# by their OIDC subject and synced across devices. Anonymous visitors keep using # by their OIDC subject and synced across devices. Anonymous visitors keep using
@@ -53,6 +59,24 @@ redirect_url = "https://news.parodia.dev/auth/callback"
# HMAC key that signs the session cookie. Generate with: openssl rand -hex 32 # HMAC key that signs the session cookie. Generate with: openssl rand -hex 32
session_secret = "${PETE_SESSION_SECRET}" session_secret = "${PETE_SESSION_SECRET}"
# Optional Web Push digests. When enabled, signed-in users can opt in (from the
# feed-settings panel) to a periodic "N new stories" notification, delivered via
# a service worker so it also works when the site is installed as a PWA. Push is
# signed-in only, so it needs web.auth above; it does nothing otherwise.
# Generate the VAPID keypair once with: pete -genvapid
# then paste the two keys here (the private key is a secret — treat it like a
# password). subject identifies you to the push service (a mailto: or https: URL).
[web.push]
enabled = false
vapid_public_key = "${PETE_VAPID_PUBLIC_KEY}"
vapid_private_key = "${PETE_VAPID_PRIVATE_KEY}"
subject = "mailto:admin@parodia.dev"
# How often the sender wakes to look for new stories per subscriber (minutes).
interval_minutes = 360
# Smallest number of new (non-hidden) stories that triggers a digest, so a lone
# item doesn't ping everyone.
min_stories = 3
# Every enabled source MUST set direct_route to a key from [matrix.channels] above. # Every enabled source MUST set direct_route to a key from [matrix.channels] above.
# There is no automatic classification — Pete posts each story to its configured channel. # There is no automatic classification — Pete posts each story to its configured channel.
# Optional: language = "en" drops feed items whose per-item <language> tag is # Optional: language = "en" drops feed items whose per-item <language> tag is

10
go.mod
View File

@@ -5,7 +5,12 @@ go 1.25.0
require ( require (
github.com/BurntSushi/toml v1.6.0 github.com/BurntSushi/toml v1.6.0
github.com/PuerkitoBio/goquery v1.12.0 github.com/PuerkitoBio/goquery v1.12.0
github.com/SherClockHolmes/webpush-go v1.4.0
github.com/coreos/go-oidc/v3 v3.19.0
github.com/mmcdole/gofeed v1.3.0 github.com/mmcdole/gofeed v1.3.0
go.mau.fi/util v0.9.9
golang.org/x/image v0.41.0
golang.org/x/oauth2 v0.36.0
maunium.net/go/mautrix v0.28.0 maunium.net/go/mautrix v0.28.0
modernc.org/sqlite v1.50.1 modernc.org/sqlite v1.50.1
) )
@@ -13,9 +18,9 @@ require (
require ( require (
filippo.io/edwards25519 v1.2.0 // indirect filippo.io/edwards25519 v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/coreos/go-oidc/v3 v3.19.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/golang-jwt/jwt/v5 v5.2.1 // indirect
github.com/google/uuid v1.6.0 // indirect github.com/google/uuid v1.6.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect github.com/json-iterator/go v1.1.12 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect github.com/mattn/go-colorable v0.1.14 // indirect
@@ -32,12 +37,9 @@ require (
github.com/tidwall/match v1.2.0 // indirect github.com/tidwall/match v1.2.0 // indirect
github.com/tidwall/pretty v1.2.1 // indirect github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/sjson v1.2.5 // indirect github.com/tidwall/sjson v1.2.5 // indirect
go.mau.fi/util v0.9.9 // indirect
golang.org/x/crypto v0.52.0 // indirect golang.org/x/crypto v0.52.0 // indirect
golang.org/x/exp v0.0.0-20260508232706-74f9aab9d74a // indirect golang.org/x/exp v0.0.0-20260508232706-74f9aab9d74a // indirect
golang.org/x/image v0.41.0 // indirect
golang.org/x/net v0.55.0 // indirect golang.org/x/net v0.55.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sys v0.45.0 // indirect golang.org/x/sys v0.45.0 // indirect
golang.org/x/text v0.37.0 // indirect golang.org/x/text v0.37.0 // indirect
modernc.org/libc v1.72.3 // indirect modernc.org/libc v1.72.3 // indirect

4
go.sum
View File

@@ -6,6 +6,8 @@ github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7Oputl
github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU= github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU=
github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo= github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo=
github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ= github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ=
github.com/SherClockHolmes/webpush-go v1.4.0 h1:ocnzNKWN23T9nvHi6IfyrQjkIc0oJWv1B1pULsf9i3s=
github.com/SherClockHolmes/webpush-go v1.4.0/go.mod h1:XSq8pKX11vNV8MJEMwjrlTkxhAj1zKfxmyhdV7Pd6UA=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM= github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA= github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/coreos/go-oidc/v3 v3.19.0 h1:F/xyOi3x1UnG1U27YVnM1N6bHiL1K2upi6U/0qr8r+I= github.com/coreos/go-oidc/v3 v3.19.0 h1:F/xyOi3x1UnG1U27YVnM1N6bHiL1K2upi6U/0qr8r+I=
@@ -17,6 +19,8 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA= github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08= github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs= github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=

View File

@@ -27,6 +27,30 @@ type WebConfig struct {
SiteTitle string `toml:"site_title"` // display name in the header SiteTitle string `toml:"site_title"` // display name in the header
BaseURL string `toml:"base_url"` // public URL (used in metadata only) BaseURL string `toml:"base_url"` // public URL (used in metadata only)
Auth AuthConfig `toml:"auth"` // optional OIDC sign-in (Authentik) Auth AuthConfig `toml:"auth"` // optional OIDC sign-in (Authentik)
Push PushConfig `toml:"push"` // optional Web Push digests (VAPID)
// AdminSubs is the allowlist of OIDC subjects allowed to view the
// owner-facing source-health dashboard at /status. Empty means the page is
// inaccessible to everyone (returns 404). Requires auth to be enabled.
AdminSubs []string `toml:"admin_subs"`
}
// PushConfig wires the Web Push digest sender. Push is signed-in only (it keys
// off the OIDC subject) so it does nothing unless auth is also enabled. Generate
// a VAPID keypair once with `pete -genvapid` and paste the two keys here.
type PushConfig struct {
Enabled bool `toml:"enabled"`
VAPIDPublicKey string `toml:"vapid_public_key"`
VAPIDPrivateKey string `toml:"vapid_private_key"`
// Subject identifies the sender to the push service; a mailto: or https: URL
// per RFC 8292. Defaults to mailto:admin@<base_url host> is not attempted —
// set it explicitly.
Subject string `toml:"subject"`
// IntervalMinutes is how often the digest sender wakes to look for new
// stories per subscriber. Defaults to 360 (6h).
IntervalMinutes int `toml:"interval_minutes"`
// MinStories is the smallest number of new stories that triggers a digest
// for a subscriber, so they aren't pinged for a single item. Defaults to 3.
MinStories int `toml:"min_stories"`
} }
// AuthConfig wires Pete's web UI to an OIDC provider (our Authentik instance). // AuthConfig wires Pete's web UI to an OIDC provider (our Authentik instance).
@@ -167,6 +191,19 @@ func (c *Config) validate() error {
} }
} }
if c.Web.Push.Enabled {
if !c.Web.Auth.Enabled {
return fmt.Errorf("web.push requires web.auth to be enabled (push is signed-in only)")
}
p := c.Web.Push
if p.VAPIDPublicKey == "" || p.VAPIDPrivateKey == "" {
return fmt.Errorf("web.push requires vapid_public_key and vapid_private_key (generate with: pete -genvapid)")
}
if p.Subject == "" {
return fmt.Errorf("web.push.subject is required (a mailto: or https: URL identifying the sender)")
}
}
for i, s := range c.Sources { for i, s := range c.Sources {
if s.Name == "" { if s.Name == "" {
return fmt.Errorf("sources[%d].name is required", i) return fmt.Errorf("sources[%d].name is required", i)
@@ -232,6 +269,12 @@ func (c *Config) applyDefaults() {
if c.Web.SiteTitle == "" { if c.Web.SiteTitle == "" {
c.Web.SiteTitle = "Pete" c.Web.SiteTitle = "Pete"
} }
if c.Web.Push.IntervalMinutes == 0 {
c.Web.Push.IntervalMinutes = 360
}
if c.Web.Push.MinStories == 0 {
c.Web.Push.MinStories = 3
}
for i := range c.Sources { for i := range c.Sources {
if c.Sources[i].PollIntervalMinutes == 0 { if c.Sources[i].PollIntervalMinutes == 0 {
c.Sources[i].PollIntervalMinutes = 20 c.Sources[i].PollIntervalMinutes = 20

View File

@@ -211,6 +211,43 @@ feed_url = "https://e.com/rss"
tier = 1 tier = 1
poll_interval_minutes = 20 poll_interval_minutes = 20
enabled = true enabled = true
`},
{"push enabled without auth", `
[matrix]
homeserver = "https://h"
user_id = "@p:h"
password = "pw"
pickle_key = "test_pickle_key_1234"
[matrix.channels]
tech = "!t:e"
[storage]
db_path = "/tmp/t.db"
[web.push]
enabled = true
vapid_public_key = "pub"
vapid_private_key = "priv"
subject = "mailto:a@b.c"
`},
{"push enabled missing keys", `
[matrix]
homeserver = "https://h"
user_id = "@p:h"
password = "pw"
pickle_key = "test_pickle_key_1234"
[matrix.channels]
tech = "!t:e"
[storage]
db_path = "/tmp/t.db"
[web.auth]
enabled = true
issuer = "https://i"
client_id = "id"
client_secret = "secret"
redirect_url = "https://r/cb"
session_secret = "session_secret_16chars_long"
[web.push]
enabled = true
subject = "mailto:a@b.c"
`}, `},
} }

View File

@@ -65,6 +65,7 @@ const googlebotUA = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.
// ArticleMeta is what we can learn from fetching an article page directly. // ArticleMeta is what we can learn from fetching an article page directly.
type ArticleMeta struct { type ArticleMeta struct {
ImageURL string // og:image or twitter:image, absolute URL ImageURL string // og:image or twitter:image, absolute URL
BodyText string // extracted visible article body text (capped at MaxBodyChars)
BodyChars int // length of extracted visible body text BodyChars int // length of extracted visible body text
Fetched bool // true if we got an HTTP 200 with HTML Fetched bool // true if we got an HTTP 200 with HTML
FetchError bool // true if the fetch failed at the network/HTTP layer FetchError bool // true if the fetch failed at the network/HTTP layer
@@ -192,7 +193,10 @@ func fetchArticleMetaOnce(articleURL, ua, referer string) ArticleMeta {
meta.Fetched = true meta.Fetched = true
meta.ImageURL = extractOGImage(doc, articleURL) meta.ImageURL = extractOGImage(doc, articleURL)
meta.BodyChars = extractBodyChars(doc) // One body extraction serves both purposes: the text feeds reader mode and
// its length is the paywall/thin-body signal.
meta.BodyText = extractBodyText(doc)
meta.BodyChars = len(meta.BodyText)
meta.Paywalled = detectPaywall(doc) meta.Paywalled = detectPaywall(doc)
meta.SubscriberOnly = detectSubscriberOnly(articleURL, doc) meta.SubscriberOnly = detectSubscriberOnly(articleURL, doc)
return meta return meta
@@ -410,10 +414,7 @@ func extractBodyText(doc *goquery.Document) string {
out = alt out = alt
} }
} }
if len(out) > MaxBodyChars { return truncateUTF8(out, MaxBodyChars)
out = out[:MaxBodyChars]
}
return out
} }
// bodyContainerSelectors is the list of selectors we try when <article>/<main> // bodyContainerSelectors is the list of selectors we try when <article>/<main>

View File

@@ -10,6 +10,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"time" "time"
"unicode/utf8"
"github.com/mmcdole/gofeed" "github.com/mmcdole/gofeed"
ext "github.com/mmcdole/gofeed/extensions" ext "github.com/mmcdole/gofeed/extensions"
@@ -31,6 +32,7 @@ type FeedItem struct {
GUID string GUID string
Headline string Headline string
Lede string Lede string
Content string // full article text from content:encoded, block structure preserved; "" when the feed ships only a snippet
ImageURL string ImageURL string
ArticleURL string ArticleURL string
Source string Source string
@@ -43,8 +45,19 @@ var (
htmlTagRe = regexp.MustCompile(`<[^>]*>`) htmlTagRe = regexp.MustCompile(`<[^>]*>`)
wsRe = regexp.MustCompile(`\s+`) wsRe = regexp.MustCompile(`\s+`)
imgSrcRe = regexp.MustCompile(`(?i)<img[^>]+src=["']([^"']+)["']`) imgSrcRe = regexp.MustCompile(`(?i)<img[^>]+src=["']([^"']+)["']`)
// Used by extractContentText to preserve paragraph structure when turning
// content:encoded HTML into plain text.
blockCloseRe = regexp.MustCompile(`(?i)</(p|div|li|h[1-6]|blockquote|article|section|figure|figcaption|ul|ol|table|tr|pre)>`)
brRe = regexp.MustCompile(`(?i)<br\s*/?>`)
hspaceRe = regexp.MustCompile(`[ \t\f\r]+`)
manyNewlineRe = regexp.MustCompile(`\n{3,}`)
) )
// maxContentChars bounds the stored article text. Generous enough for any real
// article while keeping row size and the reader payload sane.
const maxContentChars = 20000
// FetchFeed fetches and parses an RSS/Atom feed, returning items. ua overrides // FetchFeed fetches and parses an RSS/Atom feed, returning items. ua overrides
// the User-Agent header; pass "" to use Pete's honest default bot UA. // the User-Agent header; pass "" to use Pete's honest default bot UA.
func FetchFeed(feedURL, ua string) ([]FeedItem, error) { func FetchFeed(feedURL, ua string) ([]FeedItem, error) {
@@ -79,6 +92,7 @@ func FetchFeed(feedURL, ua string) ([]FeedItem, error) {
GUID: itemGUID(item), GUID: itemGUID(item),
Headline: strings.TrimSpace(item.Title), Headline: strings.TrimSpace(item.Title),
Lede: extractLede(item.Description), Lede: extractLede(item.Description),
Content: extractContentText(item.Content),
ImageURL: NormalizeImageURL(extractImageURL(item)), ImageURL: NormalizeImageURL(extractImageURL(item)),
ArticleURL: strings.TrimSpace(item.Link), ArticleURL: strings.TrimSpace(item.Link),
Language: itemLanguage(item), Language: itemLanguage(item),
@@ -142,6 +156,47 @@ func extractLede(desc string) string {
return strings.TrimSpace(text) return strings.TrimSpace(text)
} }
// extractContentText converts a feed's content:encoded HTML into plain text
// with paragraph breaks preserved, for reader mode. Block-level boundaries
// (</p>, </div>, <br>, list items, headings…) become newlines before the
// remaining tags are stripped, so the reader can re-wrap the text into
// paragraphs. Returns "" when the feed carried no content:encoded.
func extractContentText(raw string) string {
if strings.TrimSpace(raw) == "" {
return ""
}
s := brRe.ReplaceAllString(raw, "\n")
s = blockCloseRe.ReplaceAllString(s, "\n\n")
s = htmlTagRe.ReplaceAllString(s, "")
s = html.UnescapeString(s)
lines := strings.Split(s, "\n")
for i, ln := range lines {
lines[i] = strings.TrimSpace(hspaceRe.ReplaceAllString(ln, " "))
}
s = manyNewlineRe.ReplaceAllString(strings.Join(lines, "\n"), "\n\n")
s = strings.TrimSpace(s)
return truncateUTF8(s, maxContentChars)
}
// truncateUTF8 caps s at max bytes without splitting a multi-byte rune. A plain
// byte slice can leave a partial trailing rune, which is invalid UTF-8; that
// corrupt byte then breaks the RSS content:encoded XML (Go's xml encoder writes
// it raw) and shows as a replacement char in JSON. Trim back to a rune boundary.
func truncateUTF8(s string, max int) string {
if len(s) <= max {
return s
}
s = s[:max]
for len(s) > 0 {
if r, size := utf8.DecodeLastRuneInString(s); r == utf8.RuneError && size <= 1 {
s = s[:len(s)-1] // strip one byte of an incomplete trailing rune
continue
}
break
}
return s
}
// extractImageURL tries to find an image URL from feed item metadata. // extractImageURL tries to find an image URL from feed item metadata.
// Order: media:content/thumbnail, enclosures, item.Image, then <img> tags // Order: media:content/thumbnail, enclosures, item.Image, then <img> tags
// scraped from content:encoded / description (catches feeds like Ars that // scraped from content:encoded / description (catches feeds like Ars that

View File

@@ -1,6 +1,9 @@
package ingestion package ingestion
import "testing" import (
"strings"
"testing"
)
func TestExtractLede(t *testing.T) { func TestExtractLede(t *testing.T) {
tests := []struct { tests := []struct {
@@ -69,3 +72,26 @@ func TestExtractLede(t *testing.T) {
}) })
} }
} }
func TestExtractContentText(t *testing.T) {
if got := extractContentText(""); got != "" {
t.Errorf("empty input = %q, want empty", got)
}
if got := extractContentText(" \n "); got != "" {
t.Errorf("whitespace-only input = %q, want empty", got)
}
in := `<p>First paragraph with <a href="/x">a link</a> &amp; entity.</p>` +
`<p>Second line one.<br>line two.</p><ul><li>item</li></ul>`
got := extractContentText(in)
want := "First paragraph with a link & entity.\n\nSecond line one.\nline two.\n\nitem"
if got != want {
t.Errorf("extractContentText:\n got %q\n want %q", got, want)
}
// Output is capped so a runaway body can't bloat the row.
long := "<p>" + strings.Repeat("x", maxContentChars+500) + "</p>"
if n := len(extractContentText(long)); n > maxContentChars {
t.Errorf("capped length = %d, want <= %d", n, maxContentChars)
}
}

View File

@@ -97,8 +97,16 @@ func (p *Poller) pollOnce(ctx context.Context, src config.SourceConfig) {
func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) error { func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) error {
items, err := FetchFeed(src.FeedURL, src.UserAgent) items, err := FetchFeed(src.FeedURL, src.UserAgent)
if err != nil { if err != nil {
// Persist the failure for the owner-facing source-health dashboard. The
// in-memory consecutiveFailures in pollSource drives log warnings; this
// row is the authoritative record the dashboard reads.
storage.RecordPollResult(src.Name, false, 0, err)
return err return err
} }
// A successful fetch: record health now (before per-item processing) with
// the number of items the feed returned, so the dashboard reflects feed
// reachability even if individual items are later dropped as duplicates.
storage.RecordPollResult(src.Name, true, len(items), nil)
newCount := 0 newCount := 0
for i := range items { for i := range items {
@@ -160,6 +168,10 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
"guid", items[i].GUID, "url", originalURL, "source", src.Name) "guid", items[i].GUID, "url", originalURL, "source", src.Name)
continue continue
} }
// bodyText is the article text we may store for reader mode. It starts
// as the body scraped during the fetch above and is upgraded to the
// snapshot's body if we end up swapping in an archive snapshot.
bodyText := meta.BodyText
gated := meta.Gated() || (meta.Fetched && meta.BodyChars < PaywallBodyThreshold) gated := meta.Gated() || (meta.Fetched && meta.BodyChars < PaywallBodyThreshold)
// paywalled tracks whether the link the user will click is still // paywalled tracks whether the link the user will click is still
// gated — i.e. gating was detected AND no archive workaround worked. // gated — i.e. gating was detected AND no archive workaround worked.
@@ -175,6 +187,9 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
if items[i].ImageURL == "" && snapMeta.ImageURL != "" { if items[i].ImageURL == "" && snapMeta.ImageURL != "" {
items[i].ImageURL = snapMeta.ImageURL items[i].ImageURL = snapMeta.ImageURL
} }
if snapMeta.BodyText != "" {
bodyText = snapMeta.BodyText
}
workedAround = true workedAround = true
slog.Info("paywall detected, using archive snapshot", slog.Info("paywall detected, using archive snapshot",
"guid", items[i].GUID, "original", originalURL, "guid", items[i].GUID, "original", originalURL,
@@ -209,10 +224,18 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
if publishedAt > seenAt { if publishedAt > seenAt {
publishedAt = seenAt publishedAt = seenAt
} }
// Reader-mode body: prefer whichever source gave us more complete text.
// Feeds that ship full content:encoded usually win; feeds that only
// syndicate a snippet fall back to the scraped article body.
content := items[i].Content
if len(bodyText) > len(content) {
content = bodyText
}
if err := storage.InsertStory(&storage.Story{ if err := storage.InsertStory(&storage.Story{
GUID: items[i].GUID, GUID: items[i].GUID,
Headline: items[i].Headline, Headline: items[i].Headline,
Lede: items[i].Lede, Lede: items[i].Lede,
Content: content,
ImageURL: items[i].ImageURL, ImageURL: items[i].ImageURL,
ArticleURL: items[i].ArticleURL, ArticleURL: items[i].ArticleURL,
URLCanonical: canonical, URLCanonical: canonical,

View File

@@ -81,6 +81,10 @@ func runMigrations(d *sql.DB) error {
addColumnIfMissing(d, "stories", "url_canonical", "TEXT") addColumnIfMissing(d, "stories", "url_canonical", "TEXT")
addColumnIfMissing(d, "stories", "headline_norm", "TEXT") addColumnIfMissing(d, "stories", "headline_norm", "TEXT")
addColumnIfMissing(d, "stories", "paywalled", "INTEGER NOT NULL DEFAULT 0") addColumnIfMissing(d, "stories", "paywalled", "INTEGER NOT NULL DEFAULT 0")
// content holds the full article text (feed content:encoded when present,
// else the body scraped during paywall detection) for reader mode. Stories
// ingested before this column existed simply have NULL and fall back to lede.
addColumnIfMissing(d, "stories", "content", "TEXT")
addColumnIfMissing(d, "stories", "published_at", "INTEGER") addColumnIfMissing(d, "stories", "published_at", "INTEGER")
addColumnIfMissing(d, "post_log", "url_canonical", "TEXT") addColumnIfMissing(d, "post_log", "url_canonical", "TEXT")
addColumnIfMissing(d, "post_log", "forced", "INTEGER NOT NULL DEFAULT 0") addColumnIfMissing(d, "post_log", "forced", "INTEGER NOT NULL DEFAULT 0")
@@ -116,6 +120,11 @@ func RunMaintenance() {
exec("prune old reactions", exec("prune old reactions",
`DELETE FROM reactions WHERE reacted_at < ?`, storyCutoff) `DELETE FROM reactions WHERE reacted_at < ?`, storyCutoff)
// Drop per-user read/bookmark rows whose story has been pruned above, so the
// table can't accumulate dangling references as stories age out.
exec("prune orphan user_story_state",
`DELETE FROM user_story_state WHERE story_id NOT IN (SELECT id FROM stories)`)
// Daily unique tokens are only useful for the recent window; their salts are // Daily unique tokens are only useful for the recent window; their salts are
// long gone. page_views is kept forever (tiny aggregate, all-time totals). // long gone. page_views is kept forever (tiny aggregate, all-time totals).
exec("prune old daily_visitors", exec("prune old daily_visitors",

115
internal/storage/push.go Normal file
View File

@@ -0,0 +1,115 @@
package storage
import "fmt"
// PushSubscription is one browser/device endpoint a signed-in user has opted in
// for Web Push digests. See the push_subscriptions schema for the field roles.
type PushSubscription struct {
Endpoint string
UserSub string
P256dh string
Auth string
CreatedAt int64
LastNotifiedAt int64
}
// AddPushSubscription records (or refreshes) a push endpoint for a user. The
// endpoint is the primary key, so a re-subscribe from the same browser updates
// the keys and resets the digest watermark to now — the user shouldn't be
// paged for everything published before they opted in.
func AddPushSubscription(sub, endpoint, p256dh, auth string) error {
now := nowUnix()
_, err := Get().Exec(`
INSERT INTO push_subscriptions (endpoint, user_sub, p256dh, auth, created_at, last_notified_at)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(endpoint) DO UPDATE SET
user_sub = excluded.user_sub,
p256dh = excluded.p256dh,
auth = excluded.auth,
last_notified_at = excluded.last_notified_at`,
endpoint, sub, p256dh, auth, now, now)
if err != nil {
return fmt.Errorf("add push subscription: %w", err)
}
return nil
}
// RemovePushSubscription drops one endpoint regardless of owner. Reserved for
// the digest sender's prune path, where a push service has reported the endpoint
// gone (404/410) and there's no caller identity to scope by. User-initiated
// opt-outs must use RemovePushSubscriptionForUser.
func RemovePushSubscription(endpoint string) error {
_, err := Get().Exec(`DELETE FROM push_subscriptions WHERE endpoint = ?`, endpoint)
if err != nil {
return fmt.Errorf("remove push subscription: %w", err)
}
return nil
}
// RemovePushSubscriptionForUser drops an endpoint only if it belongs to sub, so
// a signed-in user can't unsubscribe another account's device by presenting its
// endpoint string. A no-op (no matching row) is not an error.
func RemovePushSubscriptionForUser(sub, endpoint string) error {
_, err := Get().Exec(
`DELETE FROM push_subscriptions WHERE endpoint = ? AND user_sub = ?`, endpoint, sub)
if err != nil {
return fmt.Errorf("remove push subscription: %w", err)
}
return nil
}
// ListPushSubscriptions returns every stored subscription, for the digest sender.
func ListPushSubscriptions() ([]PushSubscription, error) {
rows, err := Get().Query(
`SELECT endpoint, user_sub, p256dh, auth, created_at, last_notified_at
FROM push_subscriptions`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []PushSubscription
for rows.Next() {
var p PushSubscription
if err := rows.Scan(&p.Endpoint, &p.UserSub, &p.P256dh, &p.Auth, &p.CreatedAt, &p.LastNotifiedAt); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// TouchPushSubscription advances an endpoint's digest watermark so its next
// digest only considers stories seen after ts.
func TouchPushSubscription(endpoint string, ts int64) error {
_, err := Get().Exec(
`UPDATE push_subscriptions SET last_notified_at = ? WHERE endpoint = ?`, ts, endpoint)
if err != nil {
return fmt.Errorf("touch push subscription: %w", err)
}
return nil
}
// NewClassifiedSince returns classified stories first seen after sinceUnix,
// newest first, capped at limit. The digest sender uses it to count and preview
// what's new for a subscriber; it carries just the fields a digest needs.
func NewClassifiedSince(sinceUnix int64, limit int) ([]Story, error) {
rows, err := Get().Query(
`SELECT id, headline, source, channel, seen_at
FROM stories
WHERE classified = 1 AND channel NOT IN ('_discarded', '_duplicate') AND seen_at > ?
ORDER BY seen_at DESC
LIMIT ?`, sinceUnix, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.ID, &s.Headline, &s.Source, &s.Channel, &s.SeenAt); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}

View File

@@ -0,0 +1,105 @@
package storage
import "testing"
func TestPushSubscriptionLifecycle(t *testing.T) {
setupTestDB(t)
if err := AddPushSubscription("sub-1", "https://push.example/ep-a", "p256-a", "auth-a"); err != nil {
t.Fatal(err)
}
// A second endpoint for the same user (e.g. a second device).
if err := AddPushSubscription("sub-1", "https://push.example/ep-b", "p256-b", "auth-b"); err != nil {
t.Fatal(err)
}
// A different user.
if err := AddPushSubscription("sub-2", "https://push.example/ep-c", "p256-c", "auth-c"); err != nil {
t.Fatal(err)
}
subs, err := ListPushSubscriptions()
if err != nil {
t.Fatal(err)
}
if len(subs) != 3 {
t.Fatalf("got %d subscriptions, want 3", len(subs))
}
// Re-subscribing the same endpoint updates keys in place, not a new row.
if err := AddPushSubscription("sub-1", "https://push.example/ep-a", "p256-a2", "auth-a2"); err != nil {
t.Fatal(err)
}
subs, _ = ListPushSubscriptions()
if len(subs) != 3 {
t.Fatalf("after re-subscribe got %d rows, want 3 (upsert, not insert)", len(subs))
}
var epA PushSubscription
for _, s := range subs {
if s.Endpoint == "https://push.example/ep-a" {
epA = s
}
}
if epA.P256dh != "p256-a2" || epA.Auth != "auth-a2" {
t.Errorf("re-subscribe did not refresh keys: %+v", epA)
}
if err := RemovePushSubscription("https://push.example/ep-a"); err != nil {
t.Fatal(err)
}
subs, _ = ListPushSubscriptions()
if len(subs) != 2 {
t.Fatalf("after remove got %d rows, want 2", len(subs))
}
}
func TestTouchPushSubscriptionAdvancesWatermark(t *testing.T) {
setupTestDB(t)
if err := AddPushSubscription("sub-1", "https://push.example/ep", "p", "a"); err != nil {
t.Fatal(err)
}
subs, _ := ListPushSubscriptions()
orig := subs[0].LastNotifiedAt
want := orig + 5000
if err := TouchPushSubscription("https://push.example/ep", want); err != nil {
t.Fatal(err)
}
subs, _ = ListPushSubscriptions()
if subs[0].LastNotifiedAt != want {
t.Errorf("watermark = %d, want %d", subs[0].LastNotifiedAt, want)
}
}
func TestNewClassifiedSince(t *testing.T) {
setupTestDB(t)
now := nowUnix()
insert := func(guid, headline, source, channel string, classified bool, seenAt int64) {
if err := InsertStory(&Story{
GUID: guid, Headline: headline, ArticleURL: "https://x/" + guid,
Source: source, Channel: channel, Classified: classified, SeenAt: seenAt,
}); err != nil {
t.Fatal(err)
}
}
insert("old", "Old news", "Feed A", "tech", true, now-1000)
insert("fresh1", "Fresh one", "Feed A", "tech", true, now-100)
insert("fresh2", "Fresh two", "Feed B", "gaming", true, now-50)
insert("unclassified", "Pending", "Feed A", "", false, now-10)
insert("discarded", "Junk", "Feed A", "_discarded", true, now-10)
got, err := NewClassifiedSince(now-500, 10)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 {
t.Fatalf("got %d stories, want 2 (fresh classified, non-discarded, after watermark)", len(got))
}
// Newest first.
if got[0].GUID != "" && got[0].Headline != "Fresh two" {
t.Errorf("first result = %q, want newest 'Fresh two'", got[0].Headline)
}
if got[0].Source != "Feed B" || got[1].Source != "Feed A" {
t.Errorf("unexpected order/sources: %q then %q", got[0].Source, got[1].Source)
}
}

View File

@@ -38,13 +38,33 @@ func InsertStory(s *Story) error {
publishedAt = s.PublishedAt publishedAt = s.PublishedAt
} }
_, err := Get().Exec( _, err := Get().Exec(
`INSERT INTO stories (guid, headline, lede, image_url, article_url, url_canonical, headline_norm, source, platforms, channel, classified, paywalled, seen_at, published_at) `INSERT INTO stories (guid, headline, lede, content, image_url, article_url, url_canonical, headline_norm, source, platforms, channel, classified, paywalled, seen_at, published_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
s.GUID, s.Headline, s.Lede, s.ImageURL, s.ArticleURL, nullIfEmpty(s.URLCanonical), nullIfEmpty(s.HeadlineNorm), s.Source, s.Platforms, s.Channel, classified, paywalled, s.SeenAt, publishedAt, s.GUID, s.Headline, s.Lede, nullIfEmpty(s.Content), s.ImageURL, s.ArticleURL, nullIfEmpty(s.URLCanonical), nullIfEmpty(s.HeadlineNorm), s.Source, s.Platforms, s.Channel, classified, paywalled, s.SeenAt, publishedAt,
) )
return err return err
} }
// GetStoryReaderText returns the stored full article text and lede for a single
// story, for reader mode. found is false when no story has that id. The query
// mirrors the visible-story filter (classified, non-sentinel channel) so the
// public /api/article endpoint can't be enumerated to pull the captured bodies
// of discarded or not-yet-classified stories that never surface in the UI.
func GetStoryReaderText(id int64) (content, lede string, found bool, err error) {
var c sql.NullString
var l sql.NullString
row := Get().QueryRow(
`SELECT content, lede FROM stories
WHERE id = ? AND classified = 1 AND channel NOT IN ('_discarded', '_duplicate')`, id)
switch err = row.Scan(&c, &l); {
case errors.Is(err, sql.ErrNoRows):
return "", "", false, nil
case err != nil:
return "", "", false, err
}
return c.String, l.String, true, nil
}
// nullIfEmpty returns a SQL NULL for empty strings so partial unique indexes // nullIfEmpty returns a SQL NULL for empty strings so partial unique indexes
// (WHERE col IS NOT NULL) can hold multiple "unknown" rows without conflict. // (WHERE col IS NOT NULL) can hold multiple "unknown" rows without conflict.
func nullIfEmpty(s string) any { func nullIfEmpty(s string) any {
@@ -236,7 +256,7 @@ func GetNewestPostableStoryByChannel(channel string) (*Story, error) {
// channels (_discarded, _duplicate) are excluded. // channels (_discarded, _duplicate) are excluded.
func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error) { func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error) {
rows, err := Get().Query( rows, err := Get().Query(
`SELECT s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at, `SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories s FROM stories s
WHERE s.classified = 1 WHERE s.classified = 1
@@ -250,7 +270,7 @@ func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error)
var out []Story var out []Story
for rows.Next() { for rows.Next() {
var s Story var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil { if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err return nil, err
} }
out = append(out, s) out = append(out, s)
@@ -262,7 +282,7 @@ func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error)
// channels, newest first. Sentinel channels are excluded. // channels, newest first. Sentinel channels are excluded.
func ListAllClassified(limit, offset int) ([]Story, error) { func ListAllClassified(limit, offset int) ([]Story, error) {
rows, err := Get().Query( rows, err := Get().Query(
`SELECT s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at, `SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories s FROM stories s
WHERE s.classified = 1 WHERE s.classified = 1
@@ -277,7 +297,7 @@ func ListAllClassified(limit, offset int) ([]Story, error) {
var out []Story var out []Story
for rows.Next() { for rows.Next() {
var s Story var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil { if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err return nil, err
} }
out = append(out, s) out = append(out, s)
@@ -285,11 +305,60 @@ func ListAllClassified(limit, offset int) ([]Story, error) {
return out, rows.Err() return out, rows.Err()
} }
// ListForFeed returns up to `limit` classified stories for the outbound RSS and
// JSON feeds, newest first. Unlike the web list queries it also selects the full
// `content` and `published_at`, which the feeds need for content:encoded bodies
// and real pubDates. channel == "" spans all real channels; otherwise it scopes
// to that one channel. Sentinel channels are always excluded.
func ListForFeed(channel string, limit int) ([]Story, error) {
const cols = `s.id, s.guid, s.headline, s.lede, s.content, s.image_url, s.article_url, s.url_canonical, s.source, s.channel, s.seen_at, s.published_at`
var (
rows *sql.Rows
err error
)
if channel == "" {
rows, err = Get().Query(
`SELECT `+cols+`
FROM stories s
WHERE s.classified = 1
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')
ORDER BY COALESCE(s.published_at, s.seen_at) DESC
LIMIT ?`, limit)
} else {
rows, err = Get().Query(
`SELECT `+cols+`
FROM stories s
WHERE s.classified = 1
AND s.channel = ?
ORDER BY COALESCE(s.published_at, s.seen_at) DESC
LIMIT ?`, channel, limit)
}
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
var content, canonical sql.NullString
var published sql.NullInt64
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &content, &s.ImageURL, &s.ArticleURL, &canonical, &s.Source, &s.Channel, &s.SeenAt, &published); err != nil {
return nil, err
}
s.Content = content.String
s.URLCanonical = canonical.String
s.PublishedAt = published.Int64
out = append(out, s)
}
return out, rows.Err()
}
// ListRecentlyPosted returns up to `limit` stories that have been posted to // ListRecentlyPosted returns up to `limit` stories that have been posted to
// Matrix, ordered by post time (newest first). Posted is always true. // Matrix, ordered by post time (newest first). Posted is always true.
func ListRecentlyPosted(limit int) ([]Story, error) { func ListRecentlyPosted(limit int) ([]Story, error) {
rows, err := Get().Query( rows, err := Get().Query(
`SELECT s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at `SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at
FROM stories s FROM stories s
JOIN post_log p ON p.guid = s.guid JOIN post_log p ON p.guid = s.guid
GROUP BY s.guid GROUP BY s.guid
@@ -302,7 +371,7 @@ func ListRecentlyPosted(limit int) ([]Story, error) {
var out []Story var out []Story
for rows.Next() { for rows.Next() {
var s Story var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt); err != nil { if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt); err != nil {
return nil, err return nil, err
} }
s.Posted = true s.Posted = true

275
internal/storage/rank.go Normal file
View File

@@ -0,0 +1,275 @@
package storage
import (
"database/sql"
"errors"
"math"
"sort"
"strings"
)
// Ranking weights for the "For you" feed. Bookmarks are a stronger signal of
// interest than a plain read, and channel affinity outweighs source affinity
// (a reader follows topics more than outlets). Recency keeps the feed fresh so
// a heavily-read channel doesn't surface week-old stories over today's news.
const (
affinityReadWeight = 1.0
affinityBookmarkWeight = 3.0
forYouChannelWeight = 1.0
forYouSourceWeight = 0.7
forYouRecencyWeight = 0.9
// forYouCandidatePool bounds how many recent unread stories we score in Go.
forYouCandidatePool = 400
// forYouRecencyHalfLifeHours sets how fast the recency term decays.
forYouRecencyHalfLifeHours = 48.0
)
// userAffinity builds normalized channel and source affinity scores (each 0..1)
// for a signed-in user from their read + bookmark history. Bookmarks count for
// more than plain reads. total is the number of signal-bearing rows; when it is
// zero the user has no history yet and both maps are empty.
func userAffinity(sub string) (channel, source map[string]float64, total int, err error) {
channel = make(map[string]float64)
source = make(map[string]float64)
if sub == "" {
return channel, source, 0, nil
}
rows, err := Get().Query(
`SELECT s.channel, s.source, u.read_at, u.bookmarked_at
FROM user_story_state u
JOIN stories s ON s.id = u.story_id
WHERE u.user_sub = ?
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')`, sub)
if err != nil {
return nil, nil, 0, err
}
defer rows.Close()
for rows.Next() {
var ch, src string
var readAt, markAt sql.NullInt64
if err := rows.Scan(&ch, &src, &readAt, &markAt); err != nil {
return nil, nil, 0, err
}
w := 0.0
if readAt.Valid {
w += affinityReadWeight
}
if markAt.Valid {
w += affinityBookmarkWeight
}
if w == 0 {
continue
}
channel[ch] += w
source[src] += w
total++
}
if err := rows.Err(); err != nil {
return nil, nil, 0, err
}
normalizeMax(channel)
normalizeMax(source)
return channel, source, total, nil
}
// normalizeMax scales a map's values so the largest becomes 1.0, leaving an
// all-zero (or empty) map untouched.
func normalizeMax(m map[string]float64) {
var max float64
for _, v := range m {
if v > max {
max = v
}
}
if max == 0 {
return
}
for k := range m {
m[k] /= max
}
}
// ForYou returns a personalized ranking of recent, unread stories for a
// signed-in user, blending channel affinity, source affinity, and recency. It
// returns (nil, nil) when the user has no read/bookmark history yet, so callers
// can fall back to the plain latest feed.
func ForYou(sub string, limit int) ([]Story, error) {
if limit <= 0 {
return nil, nil
}
chAff, srcAff, total, err := userAffinity(sub)
if err != nil {
return nil, err
}
if total == 0 {
return nil, nil
}
rows, err := Get().Query(
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at, COALESCE(s.published_at, s.seen_at),
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories s
WHERE s.classified = 1
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')
AND s.id NOT IN (
SELECT story_id FROM user_story_state
WHERE user_sub = ? AND read_at IS NOT NULL)
ORDER BY COALESCE(s.published_at, s.seen_at) DESC
LIMIT ?`, sub, forYouCandidatePool)
if err != nil {
return nil, err
}
defer rows.Close()
now := float64(nowUnix())
type scored struct {
s Story
score float64
}
var cands []scored
for rows.Next() {
var s Story
var effectiveAt int64
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &effectiveAt, &s.Posted); err != nil {
return nil, err
}
ageHours := (now - float64(effectiveAt)) / 3600.0
if ageHours < 0 {
ageHours = 0
}
recency := math.Exp(-ageHours / forYouRecencyHalfLifeHours)
score := forYouChannelWeight*chAff[s.Channel] +
forYouSourceWeight*srcAff[s.Source] +
forYouRecencyWeight*recency
cands = append(cands, scored{s, score})
}
if err := rows.Err(); err != nil {
return nil, err
}
// Candidates arrive newest-first, and a stable sort keeps that order for
// equal scores, so ties break toward the more recent story.
sort.SliceStable(cands, func(i, j int) bool { return cands[i].score > cands[j].score })
if len(cands) > limit {
cands = cands[:limit]
}
out := make([]Story, 0, len(cands))
for _, c := range cands {
out = append(out, c.s)
}
return out, nil
}
// relatedRecencyWindowSeconds bounds "related" results to roughly the same
// window the story feed lives in (stories are pruned at 30 days anyway).
const relatedRecencyWindowSeconds = 30 * 86400
// relatedStopwords are high-frequency, low-signal tokens dropped when building
// the "related" FTS query so matches key off the story's actual subject matter.
var relatedStopwords = map[string]bool{
"the": true, "and": true, "for": true, "with": true, "that": true,
"this": true, "from": true, "has": true, "have": true, "are": true,
"was": true, "were": true, "will": true, "its": true, "into": true,
"out": true, "how": true, "why": true, "what": true, "who": true,
"you": true, "your": true, "not": true, "but": true, "all": true,
"new": true, "now": true, "more": true, "after": true, "over": true,
"about": true, "says": true, "said": true, "of": true, "to": true,
"in": true, "is": true, "on": true, "at": true, "by": true, "as": true,
"an": true, "be": true, "it": true, "or": true, "if": true, "we": true,
"he": true, "do": true, "up": true, "so": true, "no": true, "my": true,
}
// RelatedStories returns classified stories textually similar to the given
// story, ranked by FTS5 relevance and excluding the story itself. The match
// query is an OR of significant tokens from the seed story's headline and lede,
// so it surfaces stories that share subject matter rather than requiring every
// term (which is what SearchStories' implicit-AND query would demand).
func RelatedStories(storyID int64, limit int) ([]Story, error) {
if storyID <= 0 || limit <= 0 {
return nil, nil
}
var headline, lede sql.NullString
err := Get().QueryRow(`SELECT headline, lede FROM stories WHERE id = ?`, storyID).Scan(&headline, &lede)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return nil, nil
}
return nil, err
}
fts := buildRelatedFTSQuery(headline.String + " " + lede.String)
if fts == "" {
return nil, nil
}
cutoff := nowUnix() - relatedRecencyWindowSeconds
rows, err := Get().Query(
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories_fts f
JOIN stories s ON s.id = f.rowid
WHERE f.stories_fts MATCH ?
AND s.id != ?
AND s.classified = 1
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')
AND s.seen_at >= ?
ORDER BY bm25(stories_fts) ASC, s.seen_at DESC
LIMIT ?`, fts, storyID, cutoff, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// buildRelatedFTSQuery tokenizes a story's headline+lede into a de-duplicated,
// stopword-filtered OR query of quoted terms, capped so a long lede can't build
// a pathological query. Returns "" when nothing usable remains.
func buildRelatedFTSQuery(raw string) string {
var tokens []string
var cur strings.Builder
seen := make(map[string]bool)
flush := func() {
if cur.Len() == 0 {
return
}
t := strings.ToLower(cur.String())
cur.Reset()
if len([]rune(t)) < 2 || relatedStopwords[t] || seen[t] {
return
}
seen[t] = true
tokens = append(tokens, t)
}
for _, r := range raw {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
cur.WriteRune(r)
case r > 127:
cur.WriteRune(r)
default:
flush()
}
}
flush()
if len(tokens) == 0 {
return ""
}
const maxTokens = 16
if len(tokens) > maxTokens {
tokens = tokens[:maxTokens]
}
for i, t := range tokens {
tokens[i] = `"` + t + `"`
}
return strings.Join(tokens, " OR ")
}

View File

@@ -0,0 +1,141 @@
package storage
import (
"strings"
"testing"
)
// seedStoryFull inserts a story with an explicit channel + source so ranking
// tests can build a skewed affinity profile.
func seedStoryFull(t *testing.T, guid, channel, source, headline, lede string) int64 {
t.Helper()
s := &Story{
GUID: guid,
Headline: headline,
Lede: lede,
ArticleURL: "https://example.com/" + guid,
Source: source,
Channel: channel,
Classified: true,
SeenAt: nowUnix(),
}
if err := InsertStory(s); err != nil {
t.Fatalf("insert story %s: %v", guid, err)
}
var id int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, guid).Scan(&id); err != nil {
t.Fatalf("lookup id %s: %v", guid, err)
}
return id
}
func TestForYou_LeansToAffinityAndExcludesRead(t *testing.T) {
setupTestDB(t)
const sub = "sub-1"
// A skewed history: the user reads/bookmarks gaming stories.
g1 := seedStoryFull(t, "g1", "gaming", "GameSite", "Game one", "")
g2 := seedStoryFull(t, "g2", "gaming", "GameSite", "Game two", "")
// Candidates to rank (all unread until we mark some).
gc := seedStoryFull(t, "gc", "gaming", "GameSite", "Fresh gaming story", "")
tc := seedStoryFull(t, "tc", "tech", "TechSite", "Fresh tech story", "")
if err := SetRead(sub, g1, true); err != nil {
t.Fatalf("read g1: %v", err)
}
if err := SetBookmark(sub, g2, true); err != nil {
t.Fatalf("bookmark g2: %v", err)
}
got, err := ForYou(sub, 10)
if err != nil {
t.Fatalf("ForYou: %v", err)
}
if len(got) == 0 {
t.Fatal("expected results")
}
// Affinity leans gaming, so a gaming story tops the list and the tech story
// sinks to the bottom. (gc and the unread-but-bookmarked g2 both qualify and
// tie on score, so we assert on channel rather than a specific id.)
if got[0].Channel != "gaming" {
t.Fatalf("expected a gaming story first, got channel %q (id %d)", got[0].Channel, got[0].ID)
}
if got[len(got)-1].ID != tc {
t.Fatalf("expected tech candidate ranked last, got id %d", got[len(got)-1].ID)
}
// Read stories must never appear.
for _, s := range got {
if s.ID == g1 {
t.Fatalf("read story g1 leaked into ForYou")
}
}
// Sanity: the gaming candidate that was never touched should be present.
var sawGC bool
for _, s := range got {
if s.ID == gc {
sawGC = true
}
}
if !sawGC {
t.Fatalf("expected unread gaming candidate gc in results")
}
}
func TestForYou_NoHistoryReturnsNil(t *testing.T) {
setupTestDB(t)
seedStoryFull(t, "a", "tech", "TechSite", "Something", "")
got, err := ForYou("nobody", 10)
if err != nil {
t.Fatalf("ForYou: %v", err)
}
if got != nil {
t.Fatalf("expected nil for a user with no history, got %d rows", len(got))
}
}
func TestRelatedStories_OnTopicExcludesSelf(t *testing.T) {
setupTestDB(t)
seed := seedStoryFull(t, "seed", "tech", "TechSite",
"Apple unveils new iPhone camera", "The new camera sensor is larger.")
rel := seedStoryFull(t, "rel", "tech", "OtherSite",
"iPhone camera teardown reveals sensor", "A look at the new iPhone camera.")
seedStoryFull(t, "off", "gaming", "GameSite",
"Nintendo announces handheld", "A brand new portable console.")
got, err := RelatedStories(seed, 5)
if err != nil {
t.Fatalf("RelatedStories: %v", err)
}
if len(got) == 0 {
t.Fatal("expected at least one related story")
}
for _, s := range got {
if s.ID == seed {
t.Fatal("seed story returned as its own related")
}
}
if got[0].ID != rel {
t.Fatalf("expected the on-topic iPhone story first, got id %d", got[0].ID)
}
}
func TestBuildRelatedFTSQuery(t *testing.T) {
// Stopwords and 1-char tokens drop out; the rest become an OR of quoted terms.
q := buildRelatedFTSQuery("The new iPhone camera is a big deal")
if q == "" {
t.Fatal("expected a non-empty query")
}
for _, bad := range []string{`"the"`, `"is"`, `"new"`} {
if strings.Contains(q, bad) {
t.Fatalf("stopword leaked into query: %s in %q", bad, q)
}
}
for _, want := range []string{`"iphone"`, `"camera"`} {
if !strings.Contains(q, want) {
t.Fatalf("expected %s in query %q", want, q)
}
}
if buildRelatedFTSQuery("the a of to") != "" {
t.Fatal("expected empty query when only stopwords remain")
}
}

View File

@@ -6,6 +6,7 @@ CREATE TABLE IF NOT EXISTS stories (
guid TEXT UNIQUE NOT NULL, guid TEXT UNIQUE NOT NULL,
headline TEXT NOT NULL, headline TEXT NOT NULL,
lede TEXT, lede TEXT,
content TEXT,
image_url TEXT, image_url TEXT,
article_url TEXT NOT NULL, article_url TEXT NOT NULL,
url_canonical TEXT, url_canonical TEXT,
@@ -53,6 +54,18 @@ CREATE TABLE IF NOT EXISTS user_preferences (
updated_at INTEGER NOT NULL updated_at INTEGER NOT NULL
); );
-- Per-user read + bookmark state for signed-in visitors, keyed by OIDC subject.
-- One row carries both signals; a NULL timestamp means "not set". A row with
-- both timestamps NULL is meaningless and is pruned, so presence of a row means
-- the story is read, bookmarked, or both.
CREATE TABLE IF NOT EXISTS user_story_state (
user_sub TEXT NOT NULL,
story_id INTEGER NOT NULL,
read_at INTEGER,
bookmarked_at INTEGER,
PRIMARY KEY (user_sub, story_id)
);
-- Aggregate web usage. page_views holds running view counts keyed by a coarse -- Aggregate web usage. page_views holds running view counts keyed by a coarse
-- path label ("home", channel slug, …) and the UTC day, so we can report both -- path label ("home", channel slug, …) and the UTC day, so we can report both
-- all-time totals and per-day breakdowns without storing any per-request rows. -- all-time totals and per-day breakdowns without storing any per-request rows.
@@ -63,6 +76,35 @@ CREATE TABLE IF NOT EXISTS page_views (
PRIMARY KEY (path, day) PRIMARY KEY (path, day)
); );
-- Per-source poll health, one row per configured feed (keyed by source name).
-- Written on every poll (success and failure) so the owner-facing dashboard can
-- show which feeds are healthy without keeping the poller's in-memory state.
-- last_success_at / last_item_count survive failures so a broken feed still
-- shows when it last worked and how much it last returned.
CREATE TABLE IF NOT EXISTS source_health (
source TEXT PRIMARY KEY,
last_poll_at INTEGER, -- unix, most recent poll attempt
last_success_at INTEGER, -- unix, most recent successful fetch
last_error TEXT, -- last failure message ('' when healthy)
consecutive_failures INTEGER NOT NULL DEFAULT 0,
last_item_count INTEGER NOT NULL DEFAULT 0, -- items in the last successful fetch
updated_at INTEGER NOT NULL
);
-- Web Push subscriptions for signed-in users, one row per browser/device
-- endpoint. p256dh + auth are the client's encryption keys (RFC 8291); the
-- server needs them to encrypt each push. last_notified_at is the per-endpoint
-- digest watermark: the sender only counts stories seen after it. A user can
-- have several endpoints (phone, desktop) — each is notified independently.
CREATE TABLE IF NOT EXISTS push_subscriptions (
endpoint TEXT PRIMARY KEY,
user_sub TEXT NOT NULL,
p256dh TEXT NOT NULL,
auth TEXT NOT NULL,
created_at INTEGER NOT NULL,
last_notified_at INTEGER NOT NULL
);
-- Privacy-preserving daily unique estimate. visitor is a salted hash of -- Privacy-preserving daily unique estimate. visitor is a salted hash of
-- IP+User-Agent; the salt rotates every UTC day and is never persisted, so the -- IP+User-Agent; the salt rotates every UTC day and is never persisted, so the
-- hashes are irreversible and cannot be linked across days. We keep only enough -- hashes are irreversible and cannot be linked across days. We keep only enough
@@ -87,6 +129,9 @@ CREATE INDEX IF NOT EXISTS idx_reactions_post_guid ON reactions(post_guid);
CREATE UNIQUE INDEX IF NOT EXISTS idx_reactions_event_id ON reactions(event_id); CREATE UNIQUE INDEX IF NOT EXISTS idx_reactions_event_id ON reactions(event_id);
CREATE INDEX IF NOT EXISTS idx_page_views_day ON page_views(day); CREATE INDEX IF NOT EXISTS idx_page_views_day ON page_views(day);
CREATE INDEX IF NOT EXISTS idx_daily_visitors_day ON daily_visitors(day); CREATE INDEX IF NOT EXISTS idx_daily_visitors_day ON daily_visitors(day);
CREATE INDEX IF NOT EXISTS idx_user_state_bookmarks ON user_story_state(user_sub, bookmarked_at) WHERE bookmarked_at IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_user_state_reads ON user_story_state(user_sub, read_at) WHERE read_at IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_push_sub_user ON push_subscriptions(user_sub);
` `
const ftsSchema = ` const ftsSchema = `

View File

@@ -0,0 +1,149 @@
package storage
import (
"database/sql"
"fmt"
)
// SourceHealth is one source's persisted poll health, as written by the poller.
type SourceHealth struct {
Source string
LastPollAt int64 // 0 = never polled
LastSuccessAt int64 // 0 = never succeeded
LastError string
ConsecutiveFailures int
LastItemCount int
UpdatedAt int64
}
// SourceContentStat is per-source content derived from the stories table (plus
// post_log for last-posted), independent of poll health.
type SourceContentStat struct {
Source string
Total int // stories currently retained for this source
Classified int // of those, how many are classified
Paywalled int // of those, how many are gated
LastSeenAt int64 // MAX(seen_at); 0 = none
LastPostedAt int64 // MAX(post_log.posted_at) joined by guid; 0 = never posted
}
// RecordPollResult upserts a source's health row after a poll attempt. On
// success it clears the error and failure counter and records the item count;
// on failure it bumps consecutive_failures and stores the message while
// preserving the last successful timestamp and item count. Fire-and-forget:
// a failure here must never disrupt polling, so errors are logged and swallowed.
func RecordPollResult(source string, ok bool, itemCount int, pollErr error) {
now := nowUnix()
if ok {
exec("record poll success",
`INSERT INTO source_health
(source, last_poll_at, last_success_at, last_error, consecutive_failures, last_item_count, updated_at)
VALUES (?, ?, ?, '', 0, ?, ?)
ON CONFLICT(source) DO UPDATE SET
last_poll_at = excluded.last_poll_at,
last_success_at = excluded.last_success_at,
last_error = '',
consecutive_failures = 0,
last_item_count = excluded.last_item_count,
updated_at = excluded.updated_at`,
source, now, now, itemCount, now)
return
}
msg := ""
if pollErr != nil {
msg = pollErr.Error()
}
exec("record poll failure",
`INSERT INTO source_health
(source, last_poll_at, last_success_at, last_error, consecutive_failures, last_item_count, updated_at)
VALUES (?, ?, NULL, ?, 1, 0, ?)
ON CONFLICT(source) DO UPDATE SET
last_poll_at = excluded.last_poll_at,
last_error = excluded.last_error,
consecutive_failures = source_health.consecutive_failures + 1,
updated_at = excluded.updated_at`,
source, now, msg, now)
}
// ListSourceHealth returns the poll-health row for every source that has been
// polled at least once, keyed by source name.
func ListSourceHealth() (map[string]SourceHealth, error) {
rows, err := Get().Query(
`SELECT source, last_poll_at, last_success_at, last_error,
consecutive_failures, last_item_count, updated_at
FROM source_health`)
if err != nil {
return nil, fmt.Errorf("list source health: %w", err)
}
defer rows.Close()
out := make(map[string]SourceHealth)
for rows.Next() {
var h SourceHealth
var lastPoll, lastSuccess sql.NullInt64
var lastErr sql.NullString
if err := rows.Scan(&h.Source, &lastPoll, &lastSuccess, &lastErr,
&h.ConsecutiveFailures, &h.LastItemCount, &h.UpdatedAt); err != nil {
return nil, err
}
h.LastPollAt = lastPoll.Int64
h.LastSuccessAt = lastSuccess.Int64
h.LastError = lastErr.String
out[h.Source] = h
}
return out, rows.Err()
}
// SourceContentStats derives per-source content counts from the stories table,
// with last-posted joined from post_log by guid. Keyed by source name. Only
// sources with at least one retained story appear; the caller pads out the rest
// from its configured source list.
func SourceContentStats() (map[string]SourceContentStat, error) {
out := make(map[string]SourceContentStat)
rows, err := Get().Query(
`SELECT source,
COUNT(*),
COALESCE(SUM(classified), 0),
COALESCE(SUM(paywalled), 0),
COALESCE(MAX(seen_at), 0)
FROM stories
GROUP BY source`)
if err != nil {
return nil, fmt.Errorf("source content stats: %w", err)
}
defer rows.Close()
for rows.Next() {
var st SourceContentStat
if err := rows.Scan(&st.Source, &st.Total, &st.Classified, &st.Paywalled, &st.LastSeenAt); err != nil {
return nil, err
}
out[st.Source] = st
}
if err := rows.Err(); err != nil {
return nil, err
}
// Last-posted per source, joined by guid. Kept separate so sources with
// stories but no posts still appear above with a zero last-posted.
prows, err := Get().Query(
`SELECT s.source, MAX(p.posted_at)
FROM post_log p
JOIN stories s ON s.guid = p.guid
GROUP BY s.source`)
if err != nil {
return nil, fmt.Errorf("source last-posted: %w", err)
}
defer prows.Close()
for prows.Next() {
var source string
var lastPosted int64
if err := prows.Scan(&source, &lastPosted); err != nil {
return nil, err
}
st := out[source]
st.Source = source
st.LastPostedAt = lastPosted
out[source] = st
}
return out, prows.Err()
}

View File

@@ -0,0 +1,104 @@
package storage
import (
"errors"
"testing"
)
func TestRecordPollResult_SuccessThenFailure(t *testing.T) {
setupTestDB(t)
// First a successful poll returning 7 items.
RecordPollResult("Feed A", true, 7, nil)
h, err := ListSourceHealth()
if err != nil {
t.Fatal(err)
}
a, ok := h["Feed A"]
if !ok {
t.Fatal("expected a health row for Feed A")
}
if a.LastItemCount != 7 || a.ConsecutiveFailures != 0 || a.LastError != "" {
t.Errorf("after success: items=%d fails=%d err=%q", a.LastItemCount, a.ConsecutiveFailures, a.LastError)
}
if a.LastPollAt == 0 || a.LastSuccessAt == 0 {
t.Errorf("after success: last_poll=%d last_success=%d, want both set", a.LastPollAt, a.LastSuccessAt)
}
successTS := a.LastSuccessAt
// Two failures in a row: failures accumulate, but the last-success timestamp
// and item count are preserved so the dashboard still shows when it last worked.
RecordPollResult("Feed A", false, 0, errors.New("dial tcp: timeout"))
RecordPollResult("Feed A", false, 0, errors.New("502 bad gateway"))
h, _ = ListSourceHealth()
a = h["Feed A"]
if a.ConsecutiveFailures != 2 {
t.Errorf("consecutive failures = %d, want 2", a.ConsecutiveFailures)
}
if a.LastError != "502 bad gateway" {
t.Errorf("last error = %q, want %q", a.LastError, "502 bad gateway")
}
if a.LastSuccessAt != successTS {
t.Errorf("last success = %d, want preserved %d", a.LastSuccessAt, successTS)
}
if a.LastItemCount != 7 {
t.Errorf("last item count = %d, want preserved 7", a.LastItemCount)
}
// Recovery clears the error and resets the counter.
RecordPollResult("Feed A", true, 3, nil)
h, _ = ListSourceHealth()
a = h["Feed A"]
if a.ConsecutiveFailures != 0 || a.LastError != "" || a.LastItemCount != 3 {
t.Errorf("after recovery: fails=%d err=%q items=%d", a.ConsecutiveFailures, a.LastError, a.LastItemCount)
}
}
func TestSourceContentStats(t *testing.T) {
setupTestDB(t)
// Source X: two classified stories, one paywalled; one posted.
InsertStory(&Story{GUID: "x1", Headline: "X one", ArticleURL: "https://x.com/1", Source: "X", Channel: "tech", Classified: true, SeenAt: 100})
InsertStory(&Story{GUID: "x2", Headline: "X two", ArticleURL: "https://x.com/2", Source: "X", Channel: "tech", Classified: true, Paywalled: true, SeenAt: 200})
// Source Y: one unclassified story, never posted.
InsertStory(&Story{GUID: "y1", Headline: "Y one", ArticleURL: "https://y.com/1", Source: "Y", SeenAt: 150})
InsertPostLog("x1", "tech", "$e1", "", false)
// Backdate/forward the post so we can assert MAX(posted_at).
Get().Exec(`UPDATE post_log SET posted_at = ? WHERE guid = ?`, int64(500), "x1")
stats, err := SourceContentStats()
if err != nil {
t.Fatal(err)
}
x := stats["X"]
if x.Total != 2 || x.Classified != 2 || x.Paywalled != 1 {
t.Errorf("X: total=%d classified=%d paywalled=%d, want 2/2/1", x.Total, x.Classified, x.Paywalled)
}
if x.LastSeenAt != 200 {
t.Errorf("X last seen = %d, want 200", x.LastSeenAt)
}
if x.LastPostedAt != 500 {
t.Errorf("X last posted = %d, want 500", x.LastPostedAt)
}
y := stats["Y"]
if y.Total != 1 || y.Classified != 0 || y.Paywalled != 0 {
t.Errorf("Y: total=%d classified=%d paywalled=%d, want 1/0/0", y.Total, y.Classified, y.Paywalled)
}
if y.LastPostedAt != 0 {
t.Errorf("Y last posted = %d, want 0 (never posted)", y.LastPostedAt)
}
}
func TestListSourceHealth_Empty(t *testing.T) {
setupTestDB(t)
h, err := ListSourceHealth()
if err != nil {
t.Fatal(err)
}
if len(h) != 0 {
t.Errorf("expected no health rows, got %d", len(h))
}
}

View File

@@ -61,6 +61,85 @@ func TestInsertStoryAndGUIDSeen(t *testing.T) {
} }
} }
func TestGetStoryReaderText(t *testing.T) {
setupTestDB(t)
// Reader text is only served for classified, non-sentinel stories — the same
// filter the public /api/article endpoint applies so discarded/unclassified
// bodies can't be pulled by id enumeration.
s := &Story{
GUID: "reader-guid",
Headline: "Reader Headline",
Lede: "Short lede.",
Content: "First paragraph.\n\nSecond paragraph.",
ArticleURL: "https://example.com/reader",
Source: "Src",
Channel: "tech",
Classified: true,
SeenAt: 1700000000,
}
if err := InsertStory(s); err != nil {
t.Fatal(err)
}
var id int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, s.GUID).Scan(&id); err != nil {
t.Fatal(err)
}
content, lede, found, err := GetStoryReaderText(id)
if err != nil {
t.Fatal(err)
}
if !found {
t.Fatal("expected story to be found")
}
if content != s.Content {
t.Errorf("content = %q, want %q", content, s.Content)
}
if lede != s.Lede {
t.Errorf("lede = %q, want %q", lede, s.Lede)
}
// A story ingested before content capture (no content) is still found, with
// an empty content string so the reader falls back to the lede.
bare := &Story{GUID: "bare-guid", Headline: "H", Lede: "Only a lede.", ArticleURL: "https://a.com", Source: "S", Channel: "tech", Classified: true, SeenAt: 1}
if err := InsertStory(bare); err != nil {
t.Fatal(err)
}
var bareID int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, bare.GUID).Scan(&bareID); err != nil {
t.Fatal(err)
}
content, lede, found, err = GetStoryReaderText(bareID)
if err != nil || !found {
t.Fatalf("bare story: found=%v err=%v", found, err)
}
if content != "" {
t.Errorf("bare content = %q, want empty", content)
}
if lede != "Only a lede." {
t.Errorf("bare lede = %q", lede)
}
// Unknown id reports not-found rather than erroring.
if _, _, found, err = GetStoryReaderText(999999); err != nil || found {
t.Errorf("unknown id: found=%v err=%v, want found=false err=nil", found, err)
}
// A discarded story must not be readable via /api/article, even by direct id.
discarded := &Story{GUID: "disc-guid", Headline: "H", Lede: "hidden", Content: "secret body", ArticleURL: "https://d.com", Source: "S", Channel: "_discarded", Classified: true, SeenAt: 1}
if err := InsertStory(discarded); err != nil {
t.Fatal(err)
}
var discID int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, discarded.GUID).Scan(&discID); err != nil {
t.Fatal(err)
}
if _, _, found, err = GetStoryReaderText(discID); err != nil || found {
t.Errorf("discarded story: found=%v err=%v, want found=false", found, err)
}
}
func TestInsertStoryDuplicateGUID(t *testing.T) { func TestInsertStoryDuplicateGUID(t *testing.T) {
setupTestDB(t) setupTestDB(t)

View File

@@ -6,6 +6,7 @@ type Story struct {
GUID string GUID string
Headline string Headline string
Lede string Lede string
Content string // full article text for reader mode (feed content:encoded or scraped body); "" when unavailable
ImageURL string ImageURL string
ArticleURL string ArticleURL string
URLCanonical string URLCanonical string

View File

@@ -0,0 +1,145 @@
package storage
import (
"database/sql"
"fmt"
"strings"
)
// SetRead marks (read=true) or clears (read=false) the read flag for one story
// and one signed-in user (keyed by OIDC subject). Read and bookmark state share
// a row; clearing the last remaining flag removes the row.
func SetRead(sub string, storyID int64, read bool) error {
var ts any
if read {
ts = nowUnix()
}
_, err := Get().Exec(`
INSERT INTO user_story_state (user_sub, story_id, read_at, bookmarked_at)
VALUES (?, ?, ?, NULL)
ON CONFLICT(user_sub, story_id) DO UPDATE SET read_at = excluded.read_at`,
sub, storyID, ts)
if err != nil {
return fmt.Errorf("set read: %w", err)
}
return pruneEmptyState(sub, storyID)
}
// SetBookmark adds (on=true) or removes (on=false) a bookmark for one story and
// one signed-in user. See SetRead for the shared-row semantics.
func SetBookmark(sub string, storyID int64, on bool) error {
var ts any
if on {
ts = nowUnix()
}
_, err := Get().Exec(`
INSERT INTO user_story_state (user_sub, story_id, read_at, bookmarked_at)
VALUES (?, ?, NULL, ?)
ON CONFLICT(user_sub, story_id) DO UPDATE SET bookmarked_at = excluded.bookmarked_at`,
sub, storyID, ts)
if err != nil {
return fmt.Errorf("set bookmark: %w", err)
}
return pruneEmptyState(sub, storyID)
}
// pruneEmptyState drops a row once neither flag is set, keeping the table to
// only meaningful state.
func pruneEmptyState(sub string, storyID int64) error {
_, err := Get().Exec(
`DELETE FROM user_story_state
WHERE user_sub = ? AND story_id = ? AND read_at IS NULL AND bookmarked_at IS NULL`,
sub, storyID)
if err != nil {
return fmt.Errorf("prune user state: %w", err)
}
return nil
}
// UserStoryState reports, for the given story ids, which are read and which are
// bookmarked by this user. Both maps contain only ids whose flag is set, so a
// missing key means false. An empty sub or id list returns empty maps.
func UserStoryState(sub string, ids []int64) (read, bookmarked map[int64]bool, err error) {
read = make(map[int64]bool)
bookmarked = make(map[int64]bool)
if sub == "" || len(ids) == 0 {
return read, bookmarked, nil
}
q := `SELECT story_id, read_at, bookmarked_at FROM user_story_state
WHERE user_sub = ? AND story_id IN (` + placeholders(len(ids)) + `)`
args := make([]any, 0, len(ids)+1)
args = append(args, sub)
for _, id := range ids {
args = append(args, id)
}
rows, err := Get().Query(q, args...)
if err != nil {
return nil, nil, fmt.Errorf("user story state: %w", err)
}
defer rows.Close()
for rows.Next() {
var id int64
var r, b sql.NullInt64
if err := rows.Scan(&id, &r, &b); err != nil {
return nil, nil, err
}
if r.Valid {
read[id] = true
}
if b.Valid {
bookmarked[id] = true
}
}
return read, bookmarked, rows.Err()
}
// ListBookmarks returns the user's bookmarked stories, most recently bookmarked
// first, restricted to still-classified stories.
func ListBookmarks(sub string, limit, offset int) ([]Story, error) {
rows, err := Get().Query(
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM user_story_state u
JOIN stories s ON s.id = u.story_id
WHERE u.user_sub = ?
AND u.bookmarked_at IS NOT NULL
AND s.classified = 1
AND s.channel NOT IN ('_discarded', '_duplicate')
ORDER BY u.bookmarked_at DESC, u.story_id DESC
LIMIT ? OFFSET ?`, sub, limit, offset)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// CountBookmarks returns how many bookmarked stories the user would see on the
// bookmarks page. It applies the same classified filter as ListBookmarks so the
// "N saved" header can't exceed the number of rendered cards.
func CountBookmarks(sub string) (int, error) {
var n int
err := Get().QueryRow(
`SELECT COUNT(*) FROM user_story_state u
JOIN stories s ON s.id = u.story_id
WHERE u.user_sub = ? AND u.bookmarked_at IS NOT NULL AND s.classified = 1
AND s.channel NOT IN ('_discarded', '_duplicate')`,
sub).Scan(&n)
return n, err
}
// placeholders returns "?, ?, …" with n slots for an IN clause.
func placeholders(n int) string {
if n <= 0 {
return ""
}
return strings.Repeat("?, ", n-1) + "?"
}

View File

@@ -0,0 +1,126 @@
package storage
import "testing"
func seedStory(t *testing.T, guid string) int64 {
t.Helper()
s := &Story{
GUID: guid,
Headline: "Headline " + guid,
Lede: "Lede.",
ArticleURL: "https://example.com/" + guid,
Source: "Test Source",
Channel: "tech",
Classified: true,
SeenAt: 1700000000,
}
if err := InsertStory(s); err != nil {
t.Fatalf("insert story %s: %v", guid, err)
}
var id int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, guid).Scan(&id); err != nil {
t.Fatalf("lookup id %s: %v", guid, err)
}
return id
}
func TestUserStoryState_ReadAndBookmark(t *testing.T) {
setupTestDB(t)
id1 := seedStory(t, "s1")
id2 := seedStory(t, "s2")
const sub = "ak-sub-1"
// Nothing set yet.
read, marks, err := UserStoryState(sub, []int64{id1, id2})
if err != nil {
t.Fatalf("initial state: %v", err)
}
if len(read) != 0 || len(marks) != 0 {
t.Fatalf("expected empty state, got read=%v bookmarked=%v", read, marks)
}
// Mark id1 read, bookmark id1 and id2.
if err := SetRead(sub, id1, true); err != nil {
t.Fatalf("set read: %v", err)
}
if err := SetBookmark(sub, id1, true); err != nil {
t.Fatalf("set bookmark id1: %v", err)
}
if err := SetBookmark(sub, id2, true); err != nil {
t.Fatalf("set bookmark id2: %v", err)
}
read, marks, _ = UserStoryState(sub, []int64{id1, id2})
if !read[id1] || read[id2] {
t.Fatalf("read map wrong: %v", read)
}
if !marks[id1] || !marks[id2] {
t.Fatalf("bookmark map wrong: %v", marks)
}
// id1 carries both flags in a single row.
if !read[id1] || !marks[id1] {
t.Fatalf("expected id1 read+bookmarked, read=%v marks=%v", read, marks)
}
// ListBookmarks orders newest bookmark first, tiebreaking on story id so the
// result is deterministic within a single second.
bm, err := ListBookmarks(sub, 10, 0)
if err != nil {
t.Fatalf("list bookmarks: %v", err)
}
if len(bm) != 2 {
t.Fatalf("expected 2 bookmarks, got %d", len(bm))
}
if bm[0].ID != id2 || bm[1].ID != id1 {
t.Fatalf("bookmark order wrong: %d then %d", bm[0].ID, bm[1].ID)
}
if n, _ := CountBookmarks(sub); n != 2 {
t.Fatalf("count bookmarks = %d, want 2", n)
}
}
func TestUserStoryState_ClearRemovesRow(t *testing.T) {
setupTestDB(t)
id := seedStory(t, "s1")
const sub = "ak-sub-1"
if err := SetRead(sub, id, true); err != nil {
t.Fatalf("set read: %v", err)
}
if err := SetRead(sub, id, false); err != nil {
t.Fatalf("unset read: %v", err)
}
// Row should be gone since neither flag is set.
var n int
if err := Get().QueryRow(`SELECT COUNT(*) FROM user_story_state WHERE user_sub = ?`, sub).Scan(&n); err != nil {
t.Fatalf("count rows: %v", err)
}
if n != 0 {
t.Fatalf("expected row pruned, found %d", n)
}
// Setting read then bookmark then clearing only read must keep the row.
_ = SetRead(sub, id, true)
_ = SetBookmark(sub, id, true)
_ = SetRead(sub, id, false)
read, marks, _ := UserStoryState(sub, []int64{id})
if read[id] {
t.Fatalf("expected not read")
}
if !marks[id] {
t.Fatalf("expected still bookmarked after clearing read")
}
}
func TestUserStoryState_ScopedBySub(t *testing.T) {
setupTestDB(t)
id := seedStory(t, "s1")
_ = SetBookmark("user-a", id, true)
read, marks, _ := UserStoryState("user-b", []int64{id})
if len(read) != 0 || len(marks) != 0 {
t.Fatalf("user-b should see no state, got read=%v marks=%v", read, marks)
}
if n, _ := CountBookmarks("user-b"); n != 0 {
t.Fatalf("user-b count = %d, want 0", n)
}
}

292
internal/web/feed.go Normal file
View File

@@ -0,0 +1,292 @@
package web
import (
"encoding/json"
"encoding/xml"
"html"
"log/slog"
"net/http"
"strings"
"time"
"pete/internal/storage"
)
// feedItemLimit caps how many stories each outbound feed carries.
const feedItemLimit = 50
// feedCacheControl is a short cache window; feeds refresh a few times an hour.
const feedCacheControl = "public, max-age=300"
const (
rssContentNS = "http://purl.org/rss/1.0/modules/content/"
rssAtomNS = "http://www.w3.org/2005/Atom"
jsonFeedVer = "https://jsonfeed.org/version/1.1"
)
// --- RSS 2.0 ---
type rssFeedXML struct {
XMLName xml.Name `xml:"rss"`
Version string `xml:"version,attr"`
ContentNS string `xml:"xmlns:content,attr"`
AtomNS string `xml:"xmlns:atom,attr"`
Channel rssChannel `xml:"channel"`
}
type rssChannel struct {
Title string `xml:"title"`
Link string `xml:"link"`
Description string `xml:"description"`
Language string `xml:"language,omitempty"`
LastBuildDate string `xml:"lastBuildDate,omitempty"`
Generator string `xml:"generator,omitempty"`
AtomLink rssAtomLink `xml:"atom:link"`
Items []rssItem `xml:"item"`
}
type rssAtomLink struct {
Href string `xml:"href,attr"`
Rel string `xml:"rel,attr"`
Type string `xml:"type,attr"`
}
type rssItem struct {
Title string `xml:"title"`
Link string `xml:"link"`
GUID rssGUID `xml:"guid"`
PubDate string `xml:"pubDate,omitempty"`
Category string `xml:"category,omitempty"`
Description string `xml:"description"`
Content *rssContent `xml:"content:encoded,omitempty"`
}
type rssGUID struct {
IsPermaLink string `xml:"isPermaLink,attr"`
Value string `xml:",chardata"`
}
type rssContent struct {
Value string `xml:",cdata"`
}
// --- JSON Feed 1.1 ---
type jsonFeed struct {
Version string `json:"version"`
Title string `json:"title"`
HomePageURL string `json:"home_page_url"`
FeedURL string `json:"feed_url"`
Description string `json:"description,omitempty"`
Items []jsonFeedItem `json:"items"`
}
type jsonFeedItem struct {
ID string `json:"id"`
URL string `json:"url"`
Title string `json:"title"`
ContentText string `json:"content_text,omitempty"`
Summary string `json:"summary,omitempty"`
Image string `json:"image,omitempty"`
DatePublished string `json:"date_published,omitempty"`
Authors []jsonFeedAuthor `json:"authors,omitempty"`
Tags []string `json:"tags,omitempty"`
}
type jsonFeedAuthor struct {
Name string `json:"name"`
}
// handleFeedXML serves an RSS 2.0 feed for the given channel ("" = all channels).
func (s *Server) handleFeedXML(w http.ResponseWriter, r *http.Request, channel string) {
stories, err := storage.ListForFeed(channel, feedItemLimit)
if err != nil {
slog.Error("web: feed query failed", "channel", channel, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
base := s.publicBase(r)
title, homeURL, desc := s.feedMeta(channel, base)
selfURL := base + feedPath(channel, "xml")
feed := rssFeedXML{
Version: "2.0",
ContentNS: rssContentNS,
AtomNS: rssAtomNS,
Channel: rssChannel{
Title: title,
Link: homeURL,
Description: desc,
Language: "en",
Generator: "Pete",
AtomLink: rssAtomLink{Href: selfURL, Rel: "self", Type: "application/rss+xml"},
Items: make([]rssItem, 0, len(stories)),
},
}
if len(stories) > 0 {
feed.Channel.LastBuildDate = feedTime(stories[0]).UTC().Format(time.RFC1123Z)
}
for _, st := range stories {
link := storyLink(st)
item := rssItem{
Title: st.Headline,
Link: link,
GUID: rssGUID{IsPermaLink: "false", Value: st.GUID},
PubDate: feedTime(st).UTC().Format(time.RFC1123Z),
Category: st.Channel,
Description: st.Lede,
}
if html := contentToHTML(st.Content); html != "" {
item.Content = &rssContent{Value: html}
}
feed.Channel.Items = append(feed.Channel.Items, item)
}
w.Header().Set("Content-Type", "application/rss+xml; charset=utf-8")
w.Header().Set("Cache-Control", feedCacheControl)
if _, err := w.Write([]byte(xml.Header)); err != nil {
return
}
enc := xml.NewEncoder(w)
enc.Indent("", " ")
if err := enc.Encode(feed); err != nil {
slog.Error("web: rss encode failed", "err", err)
}
}
// handleFeedJSON serves a JSON Feed 1.1 for the given channel ("" = all).
func (s *Server) handleFeedJSON(w http.ResponseWriter, r *http.Request, channel string) {
stories, err := storage.ListForFeed(channel, feedItemLimit)
if err != nil {
slog.Error("web: feed query failed", "channel", channel, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
base := s.publicBase(r)
title, homeURL, desc := s.feedMeta(channel, base)
feed := jsonFeed{
Version: jsonFeedVer,
Title: title,
HomePageURL: homeURL,
FeedURL: base + feedPath(channel, "json"),
Description: desc,
Items: make([]jsonFeedItem, 0, len(stories)),
}
for _, st := range stories {
item := jsonFeedItem{
ID: st.GUID,
URL: storyLink(st),
Title: st.Headline,
ContentText: strings.TrimSpace(st.Content),
Summary: st.Lede,
Image: st.ImageURL,
DatePublished: feedTime(st).UTC().Format(time.RFC3339),
Tags: []string{st.Channel},
}
if item.ContentText == "" {
item.ContentText = st.Lede
}
if st.Source != "" {
item.Authors = []jsonFeedAuthor{{Name: st.Source}}
}
feed.Items = append(feed.Items, item)
}
w.Header().Set("Content-Type", "application/feed+json; charset=utf-8")
w.Header().Set("Cache-Control", feedCacheControl)
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
if err := enc.Encode(feed); err != nil {
slog.Error("web: json feed encode failed", "err", err)
}
}
// feedMeta returns the title, home-page URL, and description for a feed. An
// empty channel is the site-wide feed; a slug scopes to that channel.
func (s *Server) feedMeta(channel, base string) (title, homeURL, desc string) {
site := s.cfg.SiteTitle
if site == "" {
site = "Pete"
}
if channel == "" {
return site, base + "/", "The latest across every channel."
}
ch := channelBySlug(channel)
return site + " — " + ch.Title, base + "/" + channel, ch.Blurb
}
// feedPath is the site-relative path of a feed, e.g. "/feed.xml" or
// "/gaming/feed.json".
func feedPath(channel, ext string) string {
if channel == "" {
return "/feed." + ext
}
return "/" + channel + "/feed." + ext
}
// storyLink is the canonical outbound link for a story: its canonical URL when
// known, otherwise the raw article URL.
func storyLink(s storage.Story) string {
if s.URLCanonical != "" {
return s.URLCanonical
}
return s.ArticleURL
}
// feedTime picks the best timestamp for a story: its published date when
// present, otherwise when Pete first saw it.
func feedTime(s storage.Story) time.Time {
if s.PublishedAt > 0 {
return time.Unix(s.PublishedAt, 0)
}
return time.Unix(s.SeenAt, 0)
}
// channelBySlug looks up a channel by slug, falling back to a bare title-cased
// entry so an unknown slug never panics.
func channelBySlug(slug string) Channel {
for _, ch := range channels {
if ch.Slug == slug {
return ch
}
}
return Channel{Slug: slug, Title: slug, Theme: slug}
}
// publicBase returns the site's public origin (scheme://host, no trailing
// slash). It prefers the configured base_url and falls back to the request's
// host so feeds still carry absolute URLs when base_url is unset.
func (s *Server) publicBase(r *http.Request) string {
if b := strings.TrimRight(s.cfg.BaseURL, "/"); b != "" {
return b
}
scheme := "http"
if r.TLS != nil || strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https") {
scheme = "https"
}
return scheme + "://" + r.Host
}
// contentToHTML turns Pete's stored plain-text article body (paragraphs split by
// blank lines, soft line breaks inside) into simple, escaped HTML for RSS
// content:encoded. Returns "" when there is no body.
func contentToHTML(text string) string {
text = strings.TrimSpace(text)
if text == "" {
return ""
}
var b strings.Builder
for _, para := range strings.Split(text, "\n\n") {
para = strings.TrimSpace(para)
if para == "" {
continue
}
b.WriteString("<p>")
b.WriteString(strings.ReplaceAll(html.EscapeString(para), "\n", "<br>"))
b.WriteString("</p>")
}
return b.String()
}

210
internal/web/feed_test.go Normal file
View File

@@ -0,0 +1,210 @@
package web
import (
"encoding/json"
"encoding/xml"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"pete/internal/config"
"pete/internal/storage"
)
// TestFeeds exercises the outbound RSS and JSON feeds end to end: a classified
// story appears in both formats, carrying its canonical link, full content, and
// a real pubDate; a channel-scoped feed excludes stories from other channels.
func TestFeeds(t *testing.T) {
storage.Close()
if err := storage.Init(filepath.Join(t.TempDir(), "feed.db")); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { storage.Close() })
pub := time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC).Unix()
tech := &storage.Story{
GUID: "feed-tech-1",
Headline: "Chips & Dips: A Tech Tale",
Lede: "The lede for the tech story.",
Content: "First paragraph.\n\nSecond paragraph with <html> & symbols.",
ArticleURL: "https://example.com/tech/story?utm=x",
URLCanonical: "https://example.com/tech/story",
Source: "Example Wire",
Channel: "tech",
Classified: true,
SeenAt: time.Now().Unix(),
PublishedAt: pub,
}
gaming := &storage.Story{
GUID: "feed-gaming-1",
Headline: "A Gaming Headline",
Lede: "Gaming lede.",
ArticleURL: "https://example.com/gaming/story",
Source: "Play Wire",
Channel: "gaming",
Classified: true,
SeenAt: time.Now().Unix(),
}
for _, st := range []*storage.Story{tech, gaming} {
if err := storage.InsertStory(st); err != nil {
t.Fatal(err)
}
}
s, err := New(config.WebConfig{SiteTitle: "Pete", ListenAddr: ":0", BaseURL: "https://news.example/"}, nil, true)
if err != nil {
t.Fatal(err)
}
// --- Site-wide RSS ---
rw := httptest.NewRecorder()
s.handleFeedXML(rw, httptest.NewRequest("GET", "/feed.xml", nil), "")
if rw.Code != 200 {
t.Fatalf("rss status = %d", rw.Code)
}
if ct := rw.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/rss+xml") {
t.Errorf("rss content-type = %q", ct)
}
var rss struct {
Channel struct {
Title string `xml:"title"`
AtomLink struct {
Href string `xml:"href,attr"`
} `xml:"link"`
Items []struct {
Title string `xml:"title"`
Link string `xml:"link"`
GUID string `xml:"guid"`
PubDate string `xml:"pubDate"`
Content string `xml:"encoded"` // content:encoded
} `xml:"item"`
} `xml:"channel"`
}
if err := xml.Unmarshal(rw.Body.Bytes(), &rss); err != nil {
t.Fatalf("rss did not parse as XML: %v\n%s", err, rw.Body.String())
}
if len(rss.Channel.Items) != 2 {
t.Fatalf("rss items = %d, want 2", len(rss.Channel.Items))
}
// Newest-first: PublishedAt March 2026 vs gaming's seen_at (now) — gaming is
// newer, so it sorts first. Find the tech item to assert on.
var techItem *struct {
Title string `xml:"title"`
Link string `xml:"link"`
GUID string `xml:"guid"`
PubDate string `xml:"pubDate"`
Content string `xml:"encoded"`
}
for i := range rss.Channel.Items {
if rss.Channel.Items[i].GUID == "feed-tech-1" {
techItem = &rss.Channel.Items[i]
}
}
if techItem == nil {
t.Fatal("tech item missing from rss")
}
if techItem.Link != "https://example.com/tech/story" {
t.Errorf("rss link = %q, want canonical", techItem.Link)
}
if techItem.Title != "Chips & Dips: A Tech Tale" {
t.Errorf("rss title = %q", techItem.Title)
}
if !strings.Contains(techItem.Content, "<p>First paragraph.</p>") {
t.Errorf("rss content missing paragraph markup: %q", techItem.Content)
}
if !strings.Contains(techItem.Content, "&lt;html&gt;") {
t.Errorf("rss content did not escape embedded markup: %q", techItem.Content)
}
if !strings.Contains(techItem.PubDate, "2026") {
t.Errorf("rss pubDate = %q, want the published date", techItem.PubDate)
}
if !strings.Contains(rw.Body.String(), `href="https://news.example/feed.xml"`) {
t.Errorf("rss missing atom self link with base_url")
}
// --- Site-wide JSON Feed ---
rw = httptest.NewRecorder()
s.handleFeedJSON(rw, httptest.NewRequest("GET", "/feed.json", nil), "")
if rw.Code != 200 {
t.Fatalf("json status = %d", rw.Code)
}
if ct := rw.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/feed+json") {
t.Errorf("json content-type = %q", ct)
}
var jf struct {
Version string `json:"version"`
FeedURL string `json:"feed_url"`
HomePageURL string `json:"home_page_url"`
Items []struct {
ID string `json:"id"`
URL string `json:"url"`
Title string `json:"title"`
ContentText string `json:"content_text"`
Tags []string `json:"tags"`
Authors []struct {
Name string `json:"name"`
} `json:"authors"`
} `json:"items"`
}
if err := json.Unmarshal(rw.Body.Bytes(), &jf); err != nil {
t.Fatalf("json feed did not parse: %v", err)
}
if !strings.Contains(jf.Version, "jsonfeed.org") {
t.Errorf("json version = %q", jf.Version)
}
if jf.FeedURL != "https://news.example/feed.json" {
t.Errorf("json feed_url = %q", jf.FeedURL)
}
if len(jf.Items) != 2 {
t.Fatalf("json items = %d, want 2", len(jf.Items))
}
var jTech *struct {
ID string `json:"id"`
URL string `json:"url"`
Title string `json:"title"`
ContentText string `json:"content_text"`
Tags []string `json:"tags"`
Authors []struct {
Name string `json:"name"`
} `json:"authors"`
}
for i := range jf.Items {
if jf.Items[i].ID == "feed-tech-1" {
jTech = &jf.Items[i]
}
}
if jTech == nil {
t.Fatal("tech item missing from json feed")
}
if jTech.URL != "https://example.com/tech/story" {
t.Errorf("json url = %q, want canonical", jTech.URL)
}
if !strings.Contains(jTech.ContentText, "Second paragraph") {
t.Errorf("json content_text = %q", jTech.ContentText)
}
if len(jTech.Tags) != 1 || jTech.Tags[0] != "tech" {
t.Errorf("json tags = %v", jTech.Tags)
}
if len(jTech.Authors) != 1 || jTech.Authors[0].Name != "Example Wire" {
t.Errorf("json authors = %v", jTech.Authors)
}
// --- Channel-scoped RSS excludes other channels ---
rw = httptest.NewRecorder()
s.handleFeedXML(rw, httptest.NewRequest("GET", "/tech/feed.xml", nil), "tech")
rss.Channel.Items = nil // xml.Unmarshal appends; clear the site-wide items
if err := xml.Unmarshal(rw.Body.Bytes(), &rss); err != nil {
t.Fatalf("channel rss parse: %v", err)
}
if len(rss.Channel.Items) != 1 {
t.Fatalf("channel rss items = %d, want 1", len(rss.Channel.Items))
}
if rss.Channel.Items[0].GUID != "feed-tech-1" {
t.Errorf("channel rss wrong item: %q", rss.Channel.Items[0].GUID)
}
if !strings.Contains(rss.Channel.Title, "Tech") {
t.Errorf("channel rss title = %q, want channel name", rss.Channel.Title)
}
}

View File

@@ -17,6 +17,7 @@ const pageSize = 24
// StoryView is the trimmed-down record used in templates. // StoryView is the trimmed-down record used in templates.
type StoryView struct { type StoryView struct {
ID int64
Headline string Headline string
Lede string Lede string
ImageURL string ImageURL string
@@ -30,6 +31,7 @@ type StoryView struct {
func toView(s storage.Story) StoryView { func toView(s storage.Story) StoryView {
return StoryView{ return StoryView{
ID: s.ID,
Headline: s.Headline, Headline: s.Headline,
Lede: s.Lede, Lede: s.Lede,
ImageURL: s.ImageURL, ImageURL: s.ImageURL,
@@ -43,16 +45,20 @@ func toView(s storage.Story) StoryView {
} }
type pageData struct { type pageData struct {
SiteTitle string SiteTitle string
Channels []Channel Channels []Channel
Active string // slug of active channel, "" for landing Active string // slug of active channel, "" for landing
Weather Weather Weather Weather
Phase string // when set, forces day/dawn/dusk/night and disables the clock-driven phase JS Phase string // when set, forces day/dawn/dusk/night and disables the clock-driven phase JS
AllSources template.JS // JSON array of {name, channel} for the settings panel AllSources template.JS // JSON array of {name, channel} for the settings panel
AuthEnabled bool // sign-in is available AuthEnabled bool // sign-in is available
User *SessionUser // nil for anonymous visitors User *SessionUser // nil for anonymous visitors
UserPrefs template.JS // signed-in user's stored prefs blob (JSON), or "null" UserPrefs template.JS // signed-in user's stored prefs blob (JSON), or "null"
Path string // current request path, for post-login return Path string // current request path, for post-login return
PostingEnabled bool // false = web-only mode; hide Matrix-posting UI
IsAdmin bool // signed-in user is on the admin allowlist (shows /status link)
PushEnabled bool // Web Push is configured (shows the notifications toggle to signed-in users)
PushPublicKey string // VAPID public key handed to the client to subscribe
} }
type channelPage struct { type channelPage struct {
@@ -72,6 +78,7 @@ type indexPage struct {
JustPosted []StoryView JustPosted []StoryView
Stats []channelStat Stats []channelStat
Latest []StoryView Latest []StoryView
ForYou []StoryView // personalized rail; empty for anon / no-history users
} }
type channelStat struct { type channelStat struct {
@@ -103,17 +110,21 @@ func (s *Server) base(r *http.Request) pageData {
srcJSON = []byte("[]") srcJSON = []byte("[]")
} }
d := pageData{ d := pageData{
SiteTitle: s.cfg.SiteTitle, SiteTitle: s.cfg.SiteTitle,
Channels: channels, Channels: channels,
Weather: currentWeather(time.Now()), Weather: currentWeather(time.Now()),
AllSources: jsForScript(srcJSON), AllSources: jsForScript(srcJSON),
AuthEnabled: s.auth != nil, AuthEnabled: s.auth != nil,
UserPrefs: template.JS("null"), UserPrefs: template.JS("null"),
Path: r.URL.Path, Path: r.URL.Path,
PostingEnabled: s.postingEnabled,
PushEnabled: s.auth != nil && s.cfg.Push.Enabled,
PushPublicKey: s.cfg.Push.VAPIDPublicKey,
} }
if s.auth != nil { if s.auth != nil {
if u := s.auth.userFromRequest(r); u != nil { if u := s.auth.userFromRequest(r); u != nil {
d.User = u d.User = u
d.IsAdmin = s.adminSubs[u.Sub]
if blob, err := storage.GetUserPrefs(u.Sub); err == nil && blob != "" { if blob, err := storage.GetUserPrefs(u.Sub); err == nil && blob != "" {
d.UserPrefs = jsForScript([]byte(blob)) d.UserPrefs = jsForScript([]byte(blob))
} }
@@ -174,9 +185,54 @@ func (s *Server) handleIndex(w http.ResponseWriter, r *http.Request) {
Stats: stats, Stats: stats,
Latest: latest, Latest: latest,
} }
// For signed-in users with some read/bookmark history, lead with a small
// personalized rail. ForYou returns nothing for anon / no-history users, so
// the section simply doesn't render for them.
if s.auth != nil {
if u := s.auth.userFromRequest(r); u != nil {
const forYouLimit = 8
if fy, err := storage.ForYou(u.Sub, forYouLimit); err != nil {
slog.Error("web: for-you rail failed", "sub", u.Sub, "err", err)
} else {
for _, row := range fy {
data.ForYou = append(data.ForYou, toView(row))
}
}
}
}
s.render(w, "index", data) s.render(w, "index", data)
} }
type forYouPage struct {
pageData
Stories []StoryView
}
// handleForYou renders the dedicated personalized feed. Anonymous visitors are
// sent to sign-in (the route is only registered when auth is on).
func (s *Server) handleForYou(w http.ResponseWriter, r *http.Request) {
u := s.auth.userFromRequest(r)
if u == nil {
http.Redirect(w, r, "/auth/login?next=/for-you", http.StatusSeeOther)
return
}
s.track(r, "for-you")
const limit = 30
rows, err := storage.ForYou(u.Sub, limit)
if err != nil {
slog.Error("web: for-you page failed", "sub", u.Sub, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
views := make([]StoryView, 0, len(rows))
for _, row := range rows {
views = append(views, toView(row))
}
base := s.base(r)
base.Active = "for-you"
s.render(w, "for-you", forYouPage{pageData: base, Stories: views})
}
func (s *Server) handleChannel(w http.ResponseWriter, r *http.Request, ch Channel) { func (s *Server) handleChannel(w http.ResponseWriter, r *http.Request, ch Channel) {
s.track(r, ch.Slug) s.track(r, ch.Slug)
page := 1 page := 1
@@ -218,6 +274,64 @@ func (s *Server) handleChannel(w http.ResponseWriter, r *http.Request, ch Channe
s.render(w, "channel", data) s.render(w, "channel", data)
} }
type bookmarksPage struct {
pageData
Stories []StoryView
Page int
HasPrev bool
HasNext bool
PrevURL string
NextURL string
Total int
}
// handleBookmarks lists the signed-in user's bookmarked stories. Anonymous
// visitors are sent to sign-in (the route is only registered when auth is on).
func (s *Server) handleBookmarks(w http.ResponseWriter, r *http.Request) {
u := s.auth.userFromRequest(r)
if u == nil {
http.Redirect(w, r, "/auth/login?next=/bookmarks", http.StatusSeeOther)
return
}
s.track(r, "bookmarks")
page := 1
if p := r.URL.Query().Get("page"); p != "" {
if n, err := strconv.Atoi(p); err == nil && n > 0 {
page = n
}
}
offset := (page - 1) * pageSize
rows, err := storage.ListBookmarks(u.Sub, pageSize+1, offset)
if err != nil {
slog.Error("web: list bookmarks failed", "sub", u.Sub, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
hasNext := len(rows) > pageSize
if hasNext {
rows = rows[:pageSize]
}
views := make([]StoryView, 0, len(rows))
for _, row := range rows {
views = append(views, toView(row))
}
total, _ := storage.CountBookmarks(u.Sub)
base := s.base(r)
base.Active = "bookmarks"
data := bookmarksPage{
pageData: base,
Stories: views,
Page: page,
HasPrev: page > 1,
HasNext: hasNext,
PrevURL: fmt.Sprintf("/bookmarks?page=%d", page-1),
NextURL: fmt.Sprintf("/bookmarks?page=%d", page+1),
Total: total,
}
s.render(w, "bookmarks", data)
}
var ( var (
demoVariants = []string{"rain", "petals", "jacaranda", "motes", "leaves", "clear", "clouds", "snow", "fog", "storm"} demoVariants = []string{"rain", "petals", "jacaranda", "motes", "leaves", "clear", "clouds", "snow", "fog", "storm"}
demoIntensities = []string{"light", "medium", "heavy"} demoIntensities = []string{"light", "medium", "heavy"}
@@ -307,6 +421,12 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
return return
} }
_ = json.NewEncoder(w).Encode(map[string]any{"results": toSearchResults(rows)})
}
// toSearchResults maps stories to the JSON shape shared by /api/search and
// /api/related, resolving each story's channel to its display metadata.
func toSearchResults(rows []storage.Story) []searchResult {
channelByName := make(map[string]Channel, len(channels)) channelByName := make(map[string]Channel, len(channels))
for _, ch := range channels { for _, ch := range channels {
channelByName[ch.Slug] = ch channelByName[ch.Slug] = ch
@@ -331,7 +451,75 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
Posted: row.Posted, Posted: row.Posted,
}) })
} }
_ = json.NewEncoder(w).Encode(map[string]any{"results": results}) return results
}
// handleRelated returns stories textually similar to a given story, for the
// "You might also like" rail in reader mode. It is public (reader mode works
// for anonymous visitors too); for signed-in users it drops already-read
// stories so recommendations stay fresh.
func (s *Server) handleRelated(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Cache-Control", "no-store")
id, err := strconv.ParseInt(strings.TrimSpace(r.URL.Query().Get("id")), 10, 64)
if err != nil || id <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
const relatedLimit = 6
// Over-fetch so dropping already-read stories doesn't starve the rail.
rows, err := storage.RelatedStories(id, relatedLimit*2)
if err != nil {
slog.Error("web: related failed", "id", id, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
if s.auth != nil && len(rows) > 0 {
if u := s.auth.userFromRequest(r); u != nil {
ids := make([]int64, 0, len(rows))
for _, row := range rows {
ids = append(ids, row.ID)
}
if read, _, err := storage.UserStoryState(u.Sub, ids); err == nil {
kept := rows[:0]
for _, row := range rows {
if !read[row.ID] {
kept = append(kept, row)
}
}
rows = kept
}
}
}
if len(rows) > relatedLimit {
rows = rows[:relatedLimit]
}
_ = json.NewEncoder(w).Encode(map[string]any{"results": toSearchResults(rows)})
}
// handleArticle serves the stored full text of a single story for reader mode.
// The client already has headline/image/source/time from the card's data
// attributes, so this returns just the body text (and the lede as a fallback
// for stories ingested before content was captured).
func (s *Server) handleArticle(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Cache-Control", "no-store")
id, err := strconv.ParseInt(strings.TrimSpace(r.URL.Query().Get("id")), 10, 64)
if err != nil || id <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
content, lede, found, err := storage.GetStoryReaderText(id)
if err != nil {
slog.Error("web: article read failed", "id", id, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
if !found {
http.Error(w, `{"error":"not found"}`, http.StatusNotFound)
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"content": content, "lede": lede})
} }
func shortTimeAgo(t time.Time) string { func shortTimeAgo(t time.Time) string {

236
internal/web/push_sender.go Normal file
View File

@@ -0,0 +1,236 @@
package web
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"net/http"
"time"
"pete/internal/safehttp"
"pete/internal/storage"
webpush "github.com/SherClockHolmes/webpush-go"
)
// digestScan caps how many new stories the sender inspects per subscriber in one
// pass. Well past MinStories; the digest only needs a count and one headline.
const digestScan = 60
// pushSendTimeout bounds one push delivery. The endpoint is user-supplied, so a
// hostile or dead push service must not be able to wedge the (serial) digest
// loop and starve every other subscriber.
const pushSendTimeout = 15 * time.Second
// runPushSender periodically builds and delivers a "N new stories" digest to
// each subscriber, respecting their disabled-sources preference. It's started
// only when push is configured. Best-effort throughout: a failed send never
// stops the loop, and a permanently-gone endpoint is pruned.
func (s *Server) runPushSender(ctx context.Context) {
interval := time.Duration(s.cfg.Push.IntervalMinutes) * time.Minute
if interval <= 0 {
interval = 6 * time.Hour
}
slog.Info("web: push digest sender started", "interval", interval, "min_stories", s.cfg.Push.MinStories)
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
s.sendDigests()
}
}
}
// sendDigests walks every subscription once, notifying those with enough new
// stories since their last digest.
func (s *Server) sendDigests() {
subs, err := storage.ListPushSubscriptions()
if err != nil {
slog.Error("push: list subscriptions failed", "err", err)
return
}
if len(subs) == 0 {
return
}
// Disabled-source sets are per user; cache within a pass so a user with
// several devices only parses prefs once.
disabledByUser := make(map[string]map[string]bool)
sent, pruned := 0, 0
for _, sub := range subs {
stories, err := storage.NewClassifiedSince(sub.LastNotifiedAt, digestScan)
if err != nil {
slog.Error("push: scan new stories failed", "sub", sub.UserSub, "err", err)
continue
}
if len(stories) == 0 {
continue
}
disabled, ok := disabledByUser[sub.UserSub]
if !ok {
disabled = disabledSourcesFor(sub.UserSub)
disabledByUser[sub.UserSub] = disabled
}
count, top := 0, ""
for _, st := range stories {
if disabled[st.Source] {
continue
}
if count == 0 {
top = st.Headline
}
count++
}
if count < s.cfg.Push.MinStories {
// If a full scan window filled entirely with stories the user has
// hidden, the non-hidden count can stay below the threshold forever
// while the watermark never advances — the same hidden window is
// re-scanned every pass and the subscriber is permanently starved of
// digests. When the window was capped (a genuine glut, not a quiet
// spell), step the watermark past it so the next pass sees fresh
// stories. A non-full window is a real lull; leave it to accumulate.
if len(stories) == digestScan {
if derr := storage.TouchPushSubscription(sub.Endpoint, stories[0].SeenAt); derr != nil {
slog.Error("push: advance watermark past hidden glut failed", "err", derr)
}
}
continue
}
payload := buildDigestPayload(count, top)
gone, err := s.sendPush(sub, payload)
if gone {
if derr := storage.RemovePushSubscription(sub.Endpoint); derr != nil {
slog.Error("push: prune gone subscription failed", "err", derr)
} else {
pruned++
}
continue
}
if err != nil {
slog.Warn("push: send failed", "sub", sub.UserSub, "err", err)
continue
}
// Advance to the newest story this digest actually accounted for, not a
// pass-start "now": the loop can run for minutes (one slow endpoint per
// send), so stories arriving mid-pass would otherwise be re-counted next
// pass. stories[0] is the newest in the scanned window (seen_at DESC).
if derr := storage.TouchPushSubscription(sub.Endpoint, stories[0].SeenAt); derr != nil {
slog.Error("push: advance watermark failed", "err", derr)
}
sent++
}
if sent > 0 || pruned > 0 {
slog.Info("push: digest pass complete", "sent", sent, "pruned", pruned, "subscriptions", len(subs))
}
}
// buildDigestPayload renders the notification JSON the service worker expects.
func buildDigestPayload(count int, top string) []byte {
body := fmt.Sprintf("%d new stories", count)
if count == 1 {
body = "1 new story"
}
if top != "" {
body += ": " + top
}
b, _ := json.Marshal(map[string]string{
"title": "Pete has fresh news",
"body": body,
"url": "/",
"tag": "pete-digest",
})
return b
}
// sendPush encrypts and delivers one notification. It reports gone=true when the
// push service says the endpoint no longer exists (404/410) so the caller can
// prune it; err is set for other, likely-transient failures.
func (s *Server) sendPush(sub storage.PushSubscription, payload []byte) (gone bool, err error) {
resp, err := webpush.SendNotification(payload, &webpush.Subscription{
Endpoint: sub.Endpoint,
Keys: webpush.Keys{P256dh: sub.P256dh, Auth: sub.Auth},
}, &webpush.Options{
// The endpoint URL comes from the browser and is attacker-influenceable,
// so deliver through the SSRF-guarded client (blocks loopback/RFC1918/
// link-local/metadata targets) with a hard timeout — never the library's
// unguarded default http.Client.
HTTPClient: s.pushClient(),
Subscriber: s.cfg.Push.Subject,
VAPIDPublicKey: s.cfg.Push.VAPIDPublicKey,
VAPIDPrivateKey: s.cfg.Push.VAPIDPrivateKey,
TTL: 24 * 60 * 60,
Urgency: webpush.UrgencyNormal,
})
if err != nil {
return false, err
}
defer resp.Body.Close()
if resp.StatusCode == 404 || resp.StatusCode == 410 {
return true, nil
}
if resp.StatusCode >= 400 {
return false, fmt.Errorf("push service returned %d", resp.StatusCode)
}
return false, nil
}
// disabledSourcesFor returns the set of source names a user has hidden, read
// from their stored prefs blob. The blob mirrors localStorage: a JSON object
// whose "pete.disabledSources.v1" value is itself a JSON string encoding a
// {sourceName: true} map. Any parse failure yields an empty (deny-nothing) set.
func disabledSourcesFor(sub string) map[string]bool {
out := map[string]bool{}
blob, err := storage.GetUserPrefs(sub)
if err != nil || blob == "" {
return out
}
var prefs map[string]json.RawMessage
if err := json.Unmarshal([]byte(blob), &prefs); err != nil {
return out
}
raw, ok := prefs["pete.disabledSources.v1"]
if !ok {
return out
}
// The value is normally a JSON *string* containing JSON; unwrap that first,
// but tolerate a bare object too.
inner := []byte(raw)
var asStr string
if err := json.Unmarshal(raw, &asStr); err == nil {
inner = []byte(asStr)
}
var set map[string]bool
if err := json.Unmarshal(inner, &set); err != nil {
return out
}
for name, on := range set {
if on {
out[name] = true
}
}
return out
}
// pushClient returns the shared SSRF-guarded, timeout-bounded HTTP client used
// for Web Push delivery, building it once on first use. The digest loop is a
// single goroutine, so the lazy init needs no lock.
func (s *Server) pushClient() *http.Client {
if s.pushHTTP == nil {
s.pushHTTP = safehttp.NewClient(pushSendTimeout)
}
return s.pushHTTP
}
// StartPushSender launches the digest loop if push is enabled. Safe to call
// unconditionally; it's a no-op when push is off.
func (s *Server) StartPushSender(ctx context.Context) {
if !s.cfg.Push.Enabled || s.auth == nil {
return
}
go s.runPushSender(ctx)
}

View File

@@ -0,0 +1,58 @@
package web
import (
"path/filepath"
"strings"
"testing"
"pete/internal/storage"
)
func TestDisabledSourcesFor(t *testing.T) {
storage.Close()
if err := storage.Init(filepath.Join(t.TempDir(), "push.db")); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { storage.Close() })
// The stored blob mirrors localStorage: the disabledSources value is itself a
// JSON *string* encoding a {source: true} map (see prefs.js snapshot()).
blob := `{"pete.disabledSources.v1":"{\"Feed A\":true,\"Feed B\":false}","pete.weather.loc.v1":"1000-100"}`
if err := storage.PutUserPrefs("sub-1", blob, "u", "u@x"); err != nil {
t.Fatal(err)
}
got := disabledSourcesFor("sub-1")
if !got["Feed A"] {
t.Error("Feed A should be disabled")
}
if got["Feed B"] {
t.Error("Feed B is false in prefs and must not be treated as disabled")
}
if len(got) != 1 {
t.Errorf("disabled set = %v, want just {Feed A}", got)
}
// A user with no prefs disables nothing.
if len(disabledSourcesFor("nobody")) != 0 {
t.Error("unknown user should have an empty disabled set")
}
}
func TestBuildDigestPayload(t *testing.T) {
single := string(buildDigestPayload(1, "Only Story"))
if !strings.Contains(single, "1 new story") || strings.Contains(single, "1 new stories") {
t.Errorf("singular wording wrong: %s", single)
}
if !strings.Contains(single, "Only Story") {
t.Errorf("top headline missing: %s", single)
}
many := string(buildDigestPayload(7, "Big One"))
if !strings.Contains(many, "7 new stories") || !strings.Contains(many, "Big One") {
t.Errorf("plural payload wrong: %s", many)
}
if !strings.Contains(many, `"tag":"pete-digest"`) {
t.Errorf("expected digest tag in payload: %s", many)
}
}

115
internal/web/pwa.go Normal file
View File

@@ -0,0 +1,115 @@
package web
import (
"io/fs"
"log/slog"
"net/http"
"pete/internal/safehttp"
"pete/internal/storage"
)
// maxPushBodyBytes caps a subscription payload. A PushSubscription JSON is an
// endpoint URL plus two short base64 keys — a few hundred bytes — so 4 KiB is
// generous headroom for long endpoint URLs.
const maxPushBodyBytes = 4096
// handlePushSubscribe stores the caller's Web Push subscription. The body is the
// browser's PushSubscription.toJSON() shape: {endpoint, keys:{p256dh, auth}}.
func (s *Server) handlePushSubscribe(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
if !s.cfg.Push.Enabled {
http.Error(w, `{"error":"push disabled"}`, http.StatusNotFound)
return
}
var req struct {
Endpoint string `json:"endpoint"`
Keys struct {
P256dh string `json:"p256dh"`
Auth string `json:"auth"`
} `json:"keys"`
}
if !decodeStateBodyN(w, r, &req, maxPushBodyBytes) {
return
}
if req.Endpoint == "" || req.Keys.P256dh == "" || req.Keys.Auth == "" {
http.Error(w, `{"error":"incomplete subscription"}`, http.StatusBadRequest)
return
}
// The endpoint is delivered to server-side; reject non-http(s) schemes here so
// a client can't stash a file:// or gopher:// target. The digest sender's
// SSRF-guarded client blocks non-public hosts at dial time, but keeping bad
// endpoints out of the table avoids storing garbage in the first place.
if err := safehttp.ValidateURL(req.Endpoint); err != nil {
http.Error(w, `{"error":"invalid endpoint"}`, http.StatusBadRequest)
return
}
if err := storage.AddPushSubscription(u.Sub, req.Endpoint, req.Keys.P256dh, req.Keys.Auth); err != nil {
slog.Error("push: subscribe failed", "sub", u.Sub, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handlePushUnsubscribe drops the caller's own stored subscription by endpoint.
// The delete is scoped to the signed-in user so one account can't remove
// another's subscription by presenting its endpoint string.
func (s *Server) handlePushUnsubscribe(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
var req struct {
Endpoint string `json:"endpoint"`
}
if !decodeStateBodyN(w, r, &req, maxPushBodyBytes) {
return
}
if req.Endpoint == "" {
http.Error(w, `{"error":"missing endpoint"}`, http.StatusBadRequest)
return
}
if err := storage.RemovePushSubscriptionForUser(u.Sub, req.Endpoint); err != nil {
slog.Error("push: unsubscribe failed", "sub", u.Sub, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleManifest serves the web app manifest from the embedded static tree. It
// lives at the root so the installable scope covers the whole origin.
func (s *Server) handleManifest(w http.ResponseWriter, r *http.Request) {
s.serveEmbedded(w, r, "manifest.webmanifest", "application/manifest+json; charset=utf-8", "public, max-age=3600")
}
// handleServiceWorker serves /sw.js from the root. Serving it here rather than
// under /static/ lets its scope be the whole origin (a worker's default scope
// is its own path), and we set Service-Worker-Allowed as a belt-and-braces in
// case it's ever moved. no-cache keeps updated workers from being pinned by the
// HTTP cache — the browser still byte-compares to decide whether to install.
func (s *Server) handleServiceWorker(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Service-Worker-Allowed", "/")
s.serveEmbedded(w, r, "sw.js", "text/javascript; charset=utf-8", "no-cache")
}
// serveEmbedded writes a file from the embedded static FS with explicit headers.
func (s *Server) serveEmbedded(w http.ResponseWriter, _ *http.Request, name, contentType, cacheControl string) {
sub, err := fs.Sub(staticFS, "static")
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
b, err := fs.ReadFile(sub, name)
if err != nil {
http.NotFound(w, nil)
return
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Cache-Control", cacheControl)
_, _ = w.Write(b)
}

View File

@@ -0,0 +1,95 @@
package web
import (
"encoding/json"
"net/http/httptest"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"pete/internal/config"
"pete/internal/storage"
)
// TestReaderCardDataAndArticleAPI exercises the reader-mode path end to end: a
// classified story renders a card carrying its id + headline data attributes,
// and /api/article returns the stored full text for that id.
func TestReaderCardDataAndArticleAPI(t *testing.T) {
storage.Close()
if err := storage.Init(filepath.Join(t.TempDir(), "reader.db")); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { storage.Close() })
story := &storage.Story{
GUID: "reader-e2e",
Headline: "A Distinctive Reader Headline",
Lede: "The lede.",
Content: "Opening paragraph of the piece.\n\nA second paragraph with more detail.",
ArticleURL: "https://example.com/story",
Source: "Example Wire",
Channel: "tech",
Classified: true,
SeenAt: time.Now().Unix(),
}
if err := storage.InsertStory(story); err != nil {
t.Fatal(err)
}
var id int64
if err := storage.Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, story.GUID).Scan(&id); err != nil {
t.Fatal(err)
}
s, err := New(config.WebConfig{SiteTitle: "Pete", ListenAddr: ":0"}, nil, true)
if err != nil {
t.Fatal(err)
}
// Index page renders the card with the reader data attributes.
rw := httptest.NewRecorder()
s.handleIndex(rw, httptest.NewRequest("GET", "/", nil))
body := rw.Body.String()
if rw.Code != 200 {
t.Fatalf("index status = %d", rw.Code)
}
for _, want := range []string{
`data-id="` + strconv.FormatInt(id, 10) + `"`,
`data-headline="A Distinctive Reader Headline"`,
`data-story-card`,
} {
if !strings.Contains(body, want) {
t.Errorf("index HTML missing %q", want)
}
}
// The article endpoint returns the stored content for that id.
rw2 := httptest.NewRecorder()
s.handleArticle(rw2, httptest.NewRequest("GET", "/api/article?id="+strconv.FormatInt(id, 10), nil))
if rw2.Code != 200 {
t.Fatalf("article status = %d body=%s", rw2.Code, rw2.Body.String())
}
var got struct {
Content string `json:"content"`
Lede string `json:"lede"`
}
if err := json.Unmarshal(rw2.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.Content != story.Content {
t.Errorf("content = %q, want %q", got.Content, story.Content)
}
// A bad id is a 400, an unknown id is a 404.
for _, tc := range []struct {
q string
code int
}{{"id=0", 400}, {"id=abc", 400}, {"id=999999", 404}} {
w := httptest.NewRecorder()
s.handleArticle(w, httptest.NewRequest("GET", "/api/article?"+tc.q, nil))
if w.Code != tc.code {
t.Errorf("article?%s status = %d, want %d", tc.q, w.Code, tc.code)
}
}
}

View File

@@ -52,11 +52,14 @@ type SourceInfo struct {
// Server is the HTTP server for Pete's web UI. // Server is the HTTP server for Pete's web UI.
type Server struct { type Server struct {
cfg config.WebConfig cfg config.WebConfig
sources []SourceInfo sources []SourceInfo
srv *http.Server postingEnabled bool // false = web-only mode; hides Matrix-posting UI
tpls map[string]*template.Template // keyed by page name (e.g. "index", "channel") srv *http.Server
auth *Authenticator // nil when sign-in is disabled or unavailable tpls map[string]*template.Template // keyed by page name (e.g. "index", "channel")
auth *Authenticator // nil when sign-in is disabled or unavailable
adminSubs map[string]bool // OIDC subjects allowed to view /status
pushHTTP *http.Client // SSRF-guarded client for Web Push delivery; built lazily by pushClient()
// Daily-rotated salt for the privacy-preserving unique-visitor estimate. // Daily-rotated salt for the privacy-preserving unique-visitor estimate.
// Guarded by metricsMu; never persisted (see metrics.go). // Guarded by metricsMu; never persisted (see metrics.go).
@@ -68,8 +71,8 @@ type Server struct {
// New builds the server. Templates are parsed once at startup. Each page // New builds the server. Templates are parsed once at startup. Each page
// gets its own template set sharing layout.html + _card.html, which avoids // gets its own template set sharing layout.html + _card.html, which avoids
// `main` define collisions between pages. // `main` define collisions between pages.
func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) { func New(cfg config.WebConfig, sources []config.SourceConfig, postingEnabled bool) (*Server, error) {
pages := []string{"index", "channel", "weather"} pages := []string{"index", "channel", "weather", "bookmarks", "for-you", "status"}
tpls := make(map[string]*template.Template, len(pages)) tpls := make(map[string]*template.Template, len(pages))
for _, p := range pages { for _, p := range pages {
t, err := template.New("").Funcs(funcs).ParseFS(templateFS, t, err := template.New("").Funcs(funcs).ParseFS(templateFS,
@@ -89,7 +92,13 @@ func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) {
} }
infos = append(infos, SourceInfo{Name: sc.Name, Channel: sc.DirectRoute}) infos = append(infos, SourceInfo{Name: sc.Name, Channel: sc.DirectRoute})
} }
s := &Server{cfg: cfg, sources: infos, tpls: tpls} adminSubs := make(map[string]bool, len(cfg.AdminSubs))
for _, sub := range cfg.AdminSubs {
if sub != "" {
adminSubs[sub] = true
}
}
s := &Server{cfg: cfg, sources: infos, postingEnabled: postingEnabled, tpls: tpls, adminSubs: adminSubs}
// Optional OIDC sign-in (Authentik). Discovery is a network call; if the // Optional OIDC sign-in (Authentik). Discovery is a network call; if the
// provider is unreachable at boot we log and serve anonymously rather than // provider is unreachable at boot we log and serve anonymously rather than
@@ -115,14 +124,23 @@ func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) {
mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.FS(staticSub)))) mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.FS(staticSub))))
mux.HandleFunc("GET /img/{name}", s.handleImg) mux.HandleFunc("GET /img/{name}", s.handleImg)
mux.HandleFunc("GET /manifest.webmanifest", s.handleManifest)
mux.HandleFunc("GET /sw.js", s.handleServiceWorker)
mux.HandleFunc("GET /{$}", s.handleIndex) mux.HandleFunc("GET /{$}", s.handleIndex)
mux.HandleFunc("GET /weather", s.handleWeatherDemo) mux.HandleFunc("GET /weather", s.handleWeatherDemo)
mux.HandleFunc("GET /api/search", s.handleSearch) mux.HandleFunc("GET /api/search", s.handleSearch)
mux.HandleFunc("GET /api/article", s.handleArticle)
mux.HandleFunc("GET /api/related", s.handleRelated)
mux.HandleFunc("GET /feed.xml", func(w http.ResponseWriter, r *http.Request) { s.handleFeedXML(w, r, "") })
mux.HandleFunc("GET /feed.json", func(w http.ResponseWriter, r *http.Request) { s.handleFeedJSON(w, r, "") })
for _, ch := range channels { for _, ch := range channels {
ch := ch ch := ch
mux.HandleFunc("GET /"+ch.Slug, func(w http.ResponseWriter, r *http.Request) { mux.HandleFunc("GET /"+ch.Slug, func(w http.ResponseWriter, r *http.Request) {
s.handleChannel(w, r, ch) s.handleChannel(w, r, ch)
}) })
mux.HandleFunc("GET /"+ch.Slug+"/feed.xml", func(w http.ResponseWriter, r *http.Request) { s.handleFeedXML(w, r, ch.Slug) })
mux.HandleFunc("GET /"+ch.Slug+"/feed.json", func(w http.ResponseWriter, r *http.Request) { s.handleFeedJSON(w, r, ch.Slug) })
} }
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
@@ -135,6 +153,16 @@ func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) {
mux.HandleFunc("GET /auth/logout", s.auth.handleLogout) mux.HandleFunc("GET /auth/logout", s.auth.handleLogout)
mux.HandleFunc("GET /api/preferences", s.handlePrefs) mux.HandleFunc("GET /api/preferences", s.handlePrefs)
mux.HandleFunc("PUT /api/preferences", s.handlePrefs) mux.HandleFunc("PUT /api/preferences", s.handlePrefs)
mux.HandleFunc("POST /api/read", s.handleRead)
mux.HandleFunc("POST /api/bookmark", s.handleBookmark)
mux.HandleFunc("GET /api/state", s.handleState)
mux.HandleFunc("GET /bookmarks", s.handleBookmarks)
mux.HandleFunc("GET /for-you", s.handleForYou)
mux.HandleFunc("GET /status", s.handleStatus)
if s.cfg.Push.Enabled {
mux.HandleFunc("POST /api/push/subscribe", s.handlePushSubscribe)
mux.HandleFunc("POST /api/push/unsubscribe", s.handlePushUnsubscribe)
}
} }
s.srv = &http.Server{ s.srv = &http.Server{

View File

@@ -169,3 +169,241 @@ html[data-phase="night"] {
filter: contrast(1.05); filter: contrast(1.05);
} }
} }
/* ----------------------------------------------------------------------------
Feed / reader mode. A focused, one-story-at-a-time overlay driven by
reader.js. Left/right arrows page through the stories currently on screen,
marking each read as it's shown. Styling leans on the phase palette vars so
it recolours with day/night like everything else.
---------------------------------------------------------------------------- */
@layer components {
.pete-reader-backdrop {
position: absolute;
inset: 0;
background: rgba(20, 14, 6, 0.55);
backdrop-filter: blur(6px);
}
html[data-phase="night"] .pete-reader-backdrop { background: rgba(6, 8, 20, 0.65); }
.pete-reader-shell {
position: relative;
height: 100%;
width: 100%;
max-width: 44rem;
margin: 0 auto;
display: flex;
flex-direction: column;
padding: max(env(safe-area-inset-top), 0.75rem) 1rem 0.75rem;
gap: 0.75rem;
}
.pete-reader-bar {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.75rem;
color: #fff;
}
.pete-reader-progress {
font-family: "Fredoka", "Nunito", system-ui, sans-serif;
font-weight: 600;
font-size: 0.85rem;
letter-spacing: 0.02em;
text-shadow: 0 1px 2px rgba(0, 0, 0, 0.4);
}
.pete-reader-nav { display: flex; align-items: center; gap: 0.4rem; }
.pete-reader-btn {
display: inline-flex;
align-items: center;
justify-content: center;
min-width: 2.25rem;
height: 2.25rem;
padding: 0 0.7rem;
border-radius: 9999px;
background: rgba(255, 255, 255, 0.14);
color: #fff;
font-weight: 700;
font-size: 0.95rem;
border: 2px solid rgba(255, 255, 255, 0.18);
cursor: pointer;
transition: background 0.15s ease, transform 0.15s ease, opacity 0.15s ease;
}
.pete-reader-btn:hover { background: rgba(255, 255, 255, 0.28); transform: translateY(-1px); }
.pete-reader-btn:disabled { opacity: 0.35; cursor: default; transform: none; }
.pete-reader-btn-open { background: var(--accent); border-color: transparent; color: #1c1305; text-decoration: none; }
.pete-reader-btn-open:hover { background: var(--accent); filter: brightness(1.08); }
.pete-reader-scroll {
flex: 1 1 auto;
overflow-y: auto;
-webkit-overflow-scrolling: touch;
border-radius: 1.75rem;
}
.pete-reader-article {
background: var(--card);
color: var(--ink);
border-radius: 1.75rem;
border: 2px solid rgba(0, 0, 0, 0.06);
box-shadow: 0 6px 0 rgba(60, 40, 20, 0.12), 0 16px 32px rgba(60, 40, 20, 0.18);
padding: clamp(1.25rem, 4vw, 2.5rem);
}
html[data-phase="night"] .pete-reader-article { border-color: rgba(255, 255, 255, 0.08); }
.pete-reader-eyebrow {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 0.5rem;
font-size: 0.75rem;
margin-bottom: 0.85rem;
}
.pete-reader-chip {
display: inline-flex;
align-items: center;
gap: 0.35rem;
border-radius: 9999px;
padding: 0.15rem 0.65rem;
color: #fff;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.06em;
}
.pete-reader-source { font-weight: 700; opacity: 0.75; }
.pete-reader-time { opacity: 0.55; }
.pete-reader-title {
font-family: "Fredoka", "Nunito", system-ui, sans-serif;
font-weight: 700;
line-height: 1.15;
letter-spacing: -0.01em;
font-size: clamp(1.6rem, 4.5vw, 2.4rem);
margin-bottom: 1rem;
}
.pete-reader-hero {
width: 100%;
border-radius: 1.1rem;
margin-bottom: 1.25rem;
background: rgba(0, 0, 0, 0.05);
}
.pete-reader-body { font-size: 1.08rem; line-height: 1.75; }
.pete-reader-body p { margin-bottom: 1.05em; }
.pete-reader-body p:last-child { margin-bottom: 0; }
.pete-reader-note {
margin-top: 1.25rem;
padding-top: 1rem;
border-top: 1px solid rgba(0, 0, 0, 0.1);
font-size: 0.85rem;
opacity: 0.7;
}
html[data-phase="night"] .pete-reader-note { border-color: rgba(255, 255, 255, 0.12); }
.pete-reader-note a { color: var(--accent); font-weight: 700; }
.pete-reader-empty {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
text-align: center;
gap: 0.75rem;
padding: clamp(2rem, 8vw, 4rem) 1rem;
color: var(--ink);
}
.pete-reader-empty-emoji { font-size: 2.5rem; }
.pete-reader-hint {
text-align: center;
color: rgba(255, 255, 255, 0.85);
font-size: 0.75rem;
text-shadow: 0 1px 2px rgba(0, 0, 0, 0.4);
}
.pete-reader-hint kbd {
font-family: ui-monospace, monospace;
background: rgba(255, 255, 255, 0.16);
border-radius: 0.3rem;
padding: 0.05rem 0.3rem;
}
@media (max-width: 640px) {
.pete-reader-hint { display: none; }
}
/* "You might also like" rail, shown under the article in feed mode. Lives
inside the reader's scroll area, below the article card. */
.pete-reader-related { width: 100%; margin: 0.85rem auto 0; }
.pete-reader-related-title {
font-family: "Fredoka", "Nunito", system-ui, sans-serif;
font-weight: 700;
font-size: 0.95rem;
color: #fff;
margin: 0 0 0.6rem;
text-shadow: 0 1px 2px rgba(0, 0, 0, 0.4);
}
.pete-reader-related-grid { display: grid; gap: 0.6rem; }
.pete-reader-related-card {
display: flex;
align-items: center;
gap: 0.75rem;
background: var(--card);
color: var(--ink);
border-radius: 1rem;
border: 2px solid rgba(0, 0, 0, 0.06);
padding: 0.6rem;
text-decoration: none;
transition: transform 0.15s ease, box-shadow 0.15s ease;
}
.pete-reader-related-card:hover {
transform: translateY(-1px);
box-shadow: 0 6px 16px rgba(60, 40, 20, 0.16);
}
html[data-phase="night"] .pete-reader-related-card { border-color: rgba(255, 255, 255, 0.08); }
.pete-reader-related-thumb {
width: 4.5rem;
height: 3.25rem;
flex-shrink: 0;
object-fit: cover;
border-radius: 0.6rem;
background: rgba(0, 0, 0, 0.06);
}
.pete-reader-related-meta { min-width: 0; display: flex; flex-direction: column; gap: 0.25rem; }
.pete-reader-related-eyebrow {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 0.4rem;
font-size: 0.7rem;
}
.pete-reader-related-source { font-weight: 700; opacity: 0.7; }
.pete-reader-related-headline {
font-weight: 700;
line-height: 1.25;
font-size: 0.92rem;
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
/* Grid treatment for stories already read in feed mode: dimmed, with a small
corner check. Hovering restores full opacity so nothing feels lost. */
[data-story-card][data-read="1"] { opacity: 0.5; transition: opacity 0.2s ease; }
[data-story-card][data-read="1"]:hover { opacity: 1; }
[data-story-card][data-read="1"]::after {
content: "✓";
position: absolute;
top: 0.6rem;
right: 0.6rem;
z-index: 6;
display: grid;
place-items: center;
height: 1.5rem;
width: 1.5rem;
border-radius: 9999px;
background: rgba(20, 14, 6, 0.72);
color: #fff;
font-size: 0.8rem;
font-weight: 700;
}
}

File diff suppressed because one or more lines are too long

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 154 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

View File

@@ -0,0 +1,116 @@
// PWA glue: register the service worker, and (for signed-in users on a
// push-enabled server) drive the notification opt-in toggle in the settings
// panel. Anonymous visitors still get the offline reader — only the push
// controls are gated behind sign-in + a configured VAPID key.
(function () {
if (!("serviceWorker" in navigator)) return;
var CFG = window.PETE_PUSH || null; // { enabled, publicKey } or null
var reg = null;
navigator.serviceWorker.register("/sw.js").then(function (r) {
reg = r;
if (canPush()) initPushUI();
}).catch(function () {});
function canPush() {
return !!(CFG && CFG.enabled && CFG.publicKey && window.PETE_USER &&
"PushManager" in window && "Notification" in window);
}
// ---- push subscription ----------------------------------------------------
function urlBase64ToUint8Array(base64String) {
var padding = "=".repeat((4 - (base64String.length % 4)) % 4);
var base64 = (base64String + padding).replace(/-/g, "+").replace(/_/g, "/");
var raw = atob(base64);
var out = new Uint8Array(raw.length);
for (var i = 0; i < raw.length; i++) out[i] = raw.charCodeAt(i);
return out;
}
function currentSub() {
if (!reg) return Promise.resolve(null);
return reg.pushManager.getSubscription();
}
function subscribe() {
return reg.pushManager.subscribe({
userVisibleOnly: true,
applicationServerKey: urlBase64ToUint8Array(CFG.publicKey),
}).then(function (sub) {
return fetch("/api/push/subscribe", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(sub.toJSON()),
credentials: "same-origin",
}).then(function (res) {
if (!res.ok) throw new Error("subscribe rejected");
return sub;
});
});
}
function unsubscribe() {
return currentSub().then(function (sub) {
if (!sub) return;
var endpoint = sub.endpoint;
return sub.unsubscribe().then(function () {
return fetch("/api/push/unsubscribe", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ endpoint: endpoint }),
credentials: "same-origin",
}).catch(function () {});
});
});
}
// ---- settings-panel toggle ------------------------------------------------
function initPushUI() {
var slot = document.querySelector("[data-push-section]");
if (!slot) return;
slot.hidden = false;
var btn = slot.querySelector("[data-push-toggle]");
var note = slot.querySelector("[data-push-note]");
if (!btn) return;
var busy = false;
function paint(on, text) {
btn.setAttribute("aria-pressed", on ? "true" : "false");
btn.textContent = on ? "Notifications on" : "Turn on notifications";
if (note && text != null) note.textContent = text;
}
function refresh() {
if (Notification.permission === "denied") {
btn.disabled = true;
paint(false, "Notifications are blocked in your browser settings.");
return;
}
currentSub().then(function (sub) {
paint(!!sub, sub ? "You'll get a nudge when new stories land." : "Get a nudge when new stories land.");
});
}
btn.addEventListener("click", function () {
if (busy) return;
busy = true;
btn.disabled = true;
currentSub().then(function (sub) {
if (sub) return unsubscribe().then(function () { paint(false, "Notifications off."); });
return Notification.requestPermission().then(function (perm) {
if (perm !== "granted") { paint(false, "Permission denied."); return; }
return subscribe().then(function () { paint(true, "You're all set. New stories will nudge you."); });
});
}).catch(function () {
paint(false, "Something went wrong. Try again.");
}).finally(function () {
busy = false;
btn.disabled = Notification.permission === "denied";
});
});
refresh();
}
})();

View File

@@ -0,0 +1,488 @@
// Feed / reader mode. Presents the stories currently on the page one at a
// time in a focused overlay, marking each read as it's shown. Left/right arrow
// keys (or the on-screen buttons) page through them; the full article text is
// whatever Pete captured at ingest (content:encoded or the scraped body),
// fetched on demand from /api/article and cached for the session.
//
// Read state lives in localStorage under pete.read.v1. It's deliberately
// device-local — unlike the small prefs blob, the read set grows unbounded, so
// it isn't mirrored through PetePrefs to the account.
(function () {
var overlay = document.getElementById("pete-reader");
if (!overlay) return;
var READ_KEY = "pete.read.v1";
var articleEl = overlay.querySelector("[data-reader-article]");
var scrollEl = overlay.querySelector("[data-reader-scroll]");
var progressEl = overlay.querySelector("[data-reader-progress]");
var linkEl = overlay.querySelector("[data-reader-link]");
var prevBtn = overlay.querySelector("[data-reader-prev]");
var nextBtn = overlay.querySelector("[data-reader-next]");
var closeBtn = overlay.querySelector("[data-reader-close]");
var backdrop = overlay.querySelector("[data-reader-backdrop]");
var readerBookmarkBtn = overlay.querySelector("[data-reader-bookmark]");
var relatedEl = overlay.querySelector("[data-reader-related]");
var relatedCache = {}; // id -> results array
// Signed-in users (Authentik) get read + bookmark state synced server-side;
// window.PETE_USER is non-null for them. Anonymous visitors keep the
// localStorage-only behaviour and never see the bookmark controls.
var SIGNED_IN = !!(window.PETE_USER);
var bookmarkSet = Object.create(null); // id -> 1 for bookmarked stories
var items = []; // {id, url, headline, lede, image, time, source, chTitle, chEmoji, chTheme, posted, paywalled}
var index = 0;
var open = false;
var contentCache = {}; // id -> {content, lede}
var inflight = null;
// ---- read-state store -----------------------------------------------------
function loadRead() {
try {
var raw = localStorage.getItem(READ_KEY);
var obj = raw ? JSON.parse(raw) : {};
return obj && typeof obj === "object" ? obj : {};
} catch (e) { return {}; }
}
function saveRead(set) {
try { localStorage.setItem(READ_KEY, JSON.stringify(set)); } catch (e) {}
}
var readSet = loadRead();
function isRead(id) { return !!readSet[id]; }
function setRead(id, on) {
if (on) readSet[id] = 1; else delete readSet[id];
saveRead(readSet);
paintCard(id, on);
if (SIGNED_IN) postState("/api/read", { id: Number(id), read: !!on });
}
// ---- server sync (signed-in only) -----------------------------------------
function postState(url, body) {
try {
fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
credentials: "same-origin",
keepalive: true
}).catch(function () {});
} catch (e) {}
}
function isBookmarked(id) { return !!bookmarkSet[id]; }
// setBookmark updates memory, paints every matching control, and persists.
function setBookmark(id, on) {
if (on) bookmarkSet[id] = 1; else delete bookmarkSet[id];
paintBookmark(id, on);
postState("/api/bookmark", { id: Number(id), on: !!on });
// On the bookmarks page, an un-bookmark should drop the card immediately.
if (!on && location.pathname === "/bookmarks") {
document.querySelectorAll('[data-story-card][data-id="' + cssEsc(id) + '"]').forEach(function (c) {
c.parentNode && c.parentNode.removeChild(c);
});
}
}
// setBookmarkQuiet applies server-provided state without echoing it back.
function setBookmarkQuiet(id, on) {
if (on) bookmarkSet[id] = 1; else delete bookmarkSet[id];
paintBookmark(id, on);
}
function paintBookmark(id, on) {
document.querySelectorAll('[data-bookmark-btn][data-story-id="' + cssEsc(id) + '"]').forEach(function (b) {
applyCardBookmark(b, on);
});
if (readerBookmarkBtn && items[index] && String(items[index].id) === String(id)) {
applyReaderBookmark(on);
}
}
function applyCardBookmark(btn, on) {
btn.setAttribute("aria-pressed", on ? "true" : "false");
var svg = btn.querySelector("svg");
if (on) {
btn.style.background = "var(--accent)";
btn.style.color = "#1c1305";
if (svg) svg.setAttribute("fill", "currentColor");
} else {
btn.style.background = "rgba(20,14,6,.62)";
btn.style.color = "#fff";
if (svg) svg.setAttribute("fill", "none");
}
}
function applyReaderBookmark(on) {
if (!readerBookmarkBtn) return;
readerBookmarkBtn.setAttribute("aria-pressed", on ? "true" : "false");
readerBookmarkBtn.textContent = on ? "🔖 Saved" : "🔖 Save";
}
// initUserState reveals the bookmark controls and pulls the signed-in user's
// read + bookmark state for the stories on this page, painting them and
// migrating any device-local reads the account doesn't have yet.
function initUserState() {
if (!SIGNED_IN) return;
document.querySelectorAll("[data-bookmark-btn]").forEach(function (b) {
b.style.display = "inline-flex";
});
var ids = [];
document.querySelectorAll("[data-story-card]").forEach(function (c) {
var id = c.getAttribute("data-id");
if (id) ids.push(id);
});
if (!ids.length) return;
fetch("/api/state?ids=" + encodeURIComponent(ids.join(",")), { credentials: "same-origin" })
.then(function (r) { return r.ok ? r.json() : null; })
.then(function (data) {
if (!data) return;
var serverRead = Object.create(null);
(data.read || []).forEach(function (id) {
serverRead[id] = 1; readSet[id] = 1; paintCard(id, true);
});
(data.bookmarked || []).forEach(function (id) { setBookmarkQuiet(id, true); });
// Migrate device-local reads the account doesn't have yet — but only
// once per device. After the first sync the server is authoritative, so
// a story the user later marks unread on another device stays unread
// instead of being perpetually resurrected from this device's stale
// local set on every page load.
var MIGRATED_KEY = "pete.readMigrated.v1";
var migrated = false;
try { migrated = localStorage.getItem(MIGRATED_KEY) === "1"; } catch (e) {}
if (!migrated) {
ids.forEach(function (id) {
if (readSet[id] && !serverRead[id]) postState("/api/read", { id: Number(id), read: true });
});
try { localStorage.setItem(MIGRATED_KEY, "1"); } catch (e) {}
}
saveRead(readSet);
})
.catch(function () {});
}
// Reflect read state onto every matching card on the page (a story can appear
// in more than one section on the home page).
function paintCard(id, on) {
document.querySelectorAll('[data-story-card][data-id="' + cssEsc(id) + '"]').forEach(function (c) {
if (on) c.setAttribute("data-read", "1");
else c.removeAttribute("data-read");
});
}
function cssEsc(s) {
return String(s).replace(/["\\]/g, "\\$&");
}
// ---- collecting the page's stories ----------------------------------------
function isVisible(el) {
return window.getComputedStyle(el).display !== "none";
}
function collect() {
var seen = Object.create(null);
var out = [];
document.querySelectorAll("[data-story-card]").forEach(function (c) {
var id = c.getAttribute("data-id");
if (!id || seen[id] || !isVisible(c)) return;
var url = c.getAttribute("data-url") || c.getAttribute("href") || "";
if (!/^https?:\/\//i.test(url)) url = "";
seen[id] = true;
out.push({
id: id,
url: url,
headline: c.getAttribute("data-headline") || "",
lede: c.getAttribute("data-lede") || "",
image: c.getAttribute("data-image") || "",
time: c.getAttribute("data-time") || "",
source: c.getAttribute("data-source") || "",
chTitle: c.getAttribute("data-ch-title") || "",
chEmoji: c.getAttribute("data-ch-emoji") || "",
chTheme: c.getAttribute("data-ch-theme") || "",
paywalled: c.getAttribute("data-paywalled") === "1"
});
});
return out;
}
// ---- rendering ------------------------------------------------------------
function escapeHTML(s) {
return String(s == null ? "" : s)
.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;")
.replace(/"/g, "&quot;").replace(/'/g, "&#39;");
}
function safeURL(s) {
var raw = String(s == null ? "" : s).trim();
return /^https?:\/\//i.test(raw) ? raw : "";
}
// Turn stored plain text (paragraphs separated by blank lines) into escaped
// <p> blocks. Single newlines inside a paragraph become spaces.
function paragraphs(text) {
var blocks = String(text || "").split(/\n{2,}/);
var html = "";
for (var i = 0; i < blocks.length; i++) {
var t = blocks[i].replace(/\s*\n\s*/g, " ").trim();
if (t) html += "<p>" + escapeHTML(t) + "</p>";
}
return html;
}
function eyebrow(it) {
var chip = it.chTheme
? '<span class="pete-reader-chip bg-theme-' + escapeHTML(it.chTheme) + '">' +
(it.chEmoji ? '<span aria-hidden="true">' + escapeHTML(it.chEmoji) + "</span>" : "") +
escapeHTML(it.chTitle) + "</span>"
: "";
var src = it.source ? '<span class="pete-reader-source">' + escapeHTML(it.source) + "</span>" : "";
var time = it.time ? '<span class="pete-reader-time">' + escapeHTML(it.time) + "</span>" : "";
return '<div class="pete-reader-eyebrow">' + chip + src + time + "</div>";
}
function renderBody(it, bodyHTML) {
var hero = it.image
? '<img class="pete-reader-hero" src="' + escapeHTML(it.image) + '" alt="" loading="lazy" decoding="async">'
: "";
var href = safeURL(it.url);
var note = '<p class="pete-reader-note">' +
(it.paywalled ? "This source is paywalled, so the text above may be partial. " : "") +
(href ? 'Read it at the source: <a href="' + escapeHTML(href) + '" target="_blank" rel="noopener noreferrer">' +
escapeHTML(it.source || "original article") + " ↗</a>." : "") +
"</p>";
articleEl.innerHTML =
eyebrow(it) +
'<h1 class="pete-reader-title">' + escapeHTML(it.headline) + "</h1>" +
hero +
'<div class="pete-reader-body">' + bodyHTML + "</div>" +
note;
}
function loadingBody(it) {
renderBody(it, '<p style="opacity:.55">Loading the full story…</p>');
}
function fillContent(it, data) {
var text = (data && data.content) ? data.content : ((data && data.lede) || it.lede);
var html = paragraphs(text);
if (!html) {
html = '<p style="opacity:.7">No article text was captured for this story. Open the original to read it.</p>';
}
renderBody(it, html);
}
function fetchContent(it) {
if (contentCache[it.id]) { fillContent(it, contentCache[it.id]); return; }
loadingBody(it);
if (inflight) inflight.abort();
var ctrl = new AbortController();
inflight = ctrl;
var reqId = it.id;
fetch("/api/article?id=" + encodeURIComponent(it.id), { signal: ctrl.signal, credentials: "same-origin" })
.then(function (r) { if (!r.ok) throw new Error("status " + r.status); return r.json(); })
.then(function (data) {
contentCache[reqId] = data;
if (ctrl !== inflight) return; // superseded by a newer navigation
if (items[index] && items[index].id === reqId) fillContent(it, data);
})
.catch(function (err) {
if (err.name === "AbortError") return;
if (items[index] && items[index].id === reqId) fillContent(it, null); // fall back to lede
})
.finally(function () { if (ctrl === inflight) inflight = null; });
}
// ---- related ("you might also like") --------------------------------------
function clearRelated() {
if (!relatedEl) return;
relatedEl.innerHTML = "";
relatedEl.hidden = true;
}
function renderRelated(reqId, results) {
if (!relatedEl) return;
// Ignore a response that arrived after the user moved on.
if (!items[index] || String(items[index].id) !== String(reqId)) return;
if (!results || !results.length) { clearRelated(); return; }
var html = '<h2 class="pete-reader-related-title">You might also like</h2>' +
'<div class="pete-reader-related-grid">';
for (var i = 0; i < results.length; i++) {
var it = results[i];
var href = safeURL(it.article_url);
if (!href) continue;
var chip = it.channel_theme
? '<span class="pete-reader-chip bg-theme-' + escapeHTML(it.channel_theme) + '">' +
(it.channel_emoji ? '<span aria-hidden="true">' + escapeHTML(it.channel_emoji) + "</span>" : "") +
escapeHTML(it.channel_title) + "</span>"
: "";
var thumb = it.thumb_url
? '<img class="pete-reader-related-thumb" src="' + escapeHTML(it.thumb_url) + '" alt="" loading="lazy" decoding="async">'
: '<div class="pete-reader-related-thumb pete-reader-related-thumb-empty"></div>';
html += '<a class="pete-reader-related-card" href="' + escapeHTML(href) + '" target="_blank" rel="noopener noreferrer">' +
thumb +
'<div class="pete-reader-related-meta">' +
'<div class="pete-reader-related-eyebrow">' + chip +
(it.source ? '<span class="pete-reader-related-source">' + escapeHTML(it.source) + "</span>" : "") +
"</div>" +
'<div class="pete-reader-related-headline">' + escapeHTML(it.headline) + "</div>" +
"</div></a>";
}
html += "</div>";
relatedEl.innerHTML = html;
relatedEl.hidden = false;
}
function fetchRelated(it) {
if (!relatedEl) return;
clearRelated();
var reqId = it.id;
if (relatedCache[reqId]) { renderRelated(reqId, relatedCache[reqId]); return; }
fetch("/api/related?id=" + encodeURIComponent(it.id), { credentials: "same-origin" })
.then(function (r) { return r.ok ? r.json() : null; })
.then(function (data) {
var results = (data && data.results) || [];
relatedCache[reqId] = results;
renderRelated(reqId, results);
})
.catch(function () {});
}
// ---- navigation -----------------------------------------------------------
function show(i) {
index = i;
if (index >= items.length) { renderDone(); return; }
var it = items[index];
progressEl.textContent = (index + 1) + " / " + items.length;
var href = safeURL(it.url);
if (href) { linkEl.href = href; linkEl.style.display = ""; }
else linkEl.style.display = "none";
prevBtn.disabled = index === 0;
nextBtn.disabled = false;
nextBtn.textContent = index === items.length - 1 ? "done ✓" : "→";
if (scrollEl) scrollEl.scrollTop = 0;
if (readerBookmarkBtn) {
readerBookmarkBtn.style.display = SIGNED_IN ? "" : "none";
applyReaderBookmark(isBookmarked(it.id));
}
fetchContent(it);
fetchRelated(it);
setRead(it.id, true); // presenting a story marks it read
}
function renderDone() {
clearRelated();
progressEl.textContent = items.length + " / " + items.length;
linkEl.style.display = "none";
prevBtn.disabled = items.length === 0;
nextBtn.disabled = true;
nextBtn.textContent = "→";
if (scrollEl) scrollEl.scrollTop = 0;
articleEl.innerHTML =
'<div class="pete-reader-empty">' +
'<span class="pete-reader-empty-emoji" aria-hidden="true">🎉</span>' +
'<h1 class="pete-reader-title" style="margin:0">You\'re all caught up</h1>' +
'<p style="opacity:.7;max-width:28rem">That\'s every story on this page. ' +
"Head back to browse more, or press ← to revisit the last one.</p>" +
"</div>";
}
function next() { if (index < items.length) show(index + 1); }
function prev() { if (index > 0) show(index - 1); }
function firstUnread() {
for (var i = 0; i < items.length; i++) if (!isRead(items[i].id)) return i;
return 0;
}
// ---- open / close ---------------------------------------------------------
function openReader() {
items = collect();
if (items.length === 0) return;
open = true;
overlay.classList.remove("hidden");
document.body.classList.add("overflow-hidden");
show(firstUnread());
}
function closeReader() {
open = false;
if (inflight) { inflight.abort(); inflight = null; }
clearRelated();
overlay.classList.add("hidden");
document.body.classList.remove("overflow-hidden");
}
// ---- wiring ---------------------------------------------------------------
document.addEventListener("DOMContentLoaded", function () {
// Dim stories already read on this device.
document.querySelectorAll("[data-story-card]").forEach(function (c) {
var id = c.getAttribute("data-id");
if (id && isRead(id)) c.setAttribute("data-read", "1");
});
document.querySelectorAll("[data-reader-open]").forEach(function (b) {
b.addEventListener("click", function (e) { e.preventDefault(); openReader(); });
});
if (prevBtn) prevBtn.addEventListener("click", prev);
if (nextBtn) nextBtn.addEventListener("click", next);
if (closeBtn) closeBtn.addEventListener("click", closeReader);
if (backdrop) backdrop.addEventListener("click", closeReader);
if (readerBookmarkBtn) readerBookmarkBtn.addEventListener("click", function () {
var it = items[index];
if (it) setBookmark(it.id, !isBookmarked(it.id));
});
initUserState();
});
// Bookmark buttons live inside the card's <a>; intercept so a tap toggles the
// bookmark instead of following the link. Delegated so it also covers cards
// that are added or removed after load.
document.addEventListener("click", function (e) {
var btn = e.target.closest && e.target.closest("[data-bookmark-btn]");
if (!btn) return;
e.preventDefault();
e.stopPropagation();
var id = btn.getAttribute("data-story-id");
if (id) setBookmark(id, !isBookmarked(id));
});
document.addEventListener("keydown", function (e) {
if (e.key !== "Enter" && e.key !== " ") return;
var btn = e.target.closest && e.target.closest("[data-bookmark-btn]");
if (!btn) return;
e.preventDefault();
var id = btn.getAttribute("data-story-id");
if (id) setBookmark(id, !isBookmarked(id));
});
document.addEventListener("keydown", function (e) {
// Enter feed mode with `f` when nothing is focused and no other overlay is up.
if (!open && (e.key === "f" || e.key === "F") && !e.metaKey && !e.ctrlKey && !e.altKey) {
var tag = (document.activeElement && document.activeElement.tagName) || "";
if (tag === "INPUT" || tag === "TEXTAREA") return;
var searchOpen = document.getElementById("pete-search");
if (searchOpen && !searchOpen.classList.contains("hidden")) return;
e.preventDefault();
openReader();
return;
}
if (!open) return;
switch (e.key) {
case "ArrowRight": case "l": case " ": e.preventDefault(); next(); break;
case "ArrowLeft": case "h": e.preventDefault(); prev(); break;
case "Escape": e.preventDefault(); closeReader(); break;
case "o": case "Enter": {
var it = items[index];
var href = it && safeURL(it.url);
if (href) { e.preventDefault(); window.open(href, "_blank", "noopener"); }
break;
}
case "u": {
var cur = items[index];
if (cur) setRead(cur.id, false); // let the user undo an accidental read
break;
}
case "b": case "B": {
if (!SIGNED_IN) break;
var it = items[index];
if (it) { e.preventDefault(); setBookmark(it.id, !isBookmarked(it.id)); }
break;
}
}
});
})();

View File

@@ -0,0 +1,22 @@
{
"name": "Pete — friendly news",
"short_name": "Pete",
"description": "A calm, read-one-at-a-time news reader. Bookmarks, a personalized feed, and offline reading.",
"id": "/",
"start_url": "/?source=pwa",
"scope": "/",
"display": "standalone",
"orientation": "portrait-primary",
"background_color": "#fbf3e3",
"theme_color": "#fbf3e3",
"categories": ["news", "productivity"],
"icons": [
{ "src": "/static/img/icon-192.png", "sizes": "192x192", "type": "image/png", "purpose": "any" },
{ "src": "/static/img/icon-512.png", "sizes": "512x512", "type": "image/png", "purpose": "any" },
{ "src": "/static/img/maskable-512.png", "sizes": "512x512", "type": "image/png", "purpose": "maskable" }
],
"shortcuts": [
{ "name": "For you", "short_name": "For you", "url": "/for-you", "description": "Your personalized feed" },
{ "name": "Bookmarks", "short_name": "Saved", "url": "/bookmarks", "description": "Stories you saved" }
]
}

193
internal/web/static/sw.js Normal file
View File

@@ -0,0 +1,193 @@
// Pete's service worker: an installable-PWA shell, an offline reader, and the
// Web Push receiver. Served from the root (/sw.js) so its scope is the whole
// origin — it can intercept navigations and /api/article the same as any page.
//
// Bump CACHE_VERSION whenever the precached shell assets change; activate()
// drops every cache that doesn't match the current version.
var CACHE_VERSION = "v3";
var SHELL_CACHE = "pete-shell-" + CACHE_VERSION;
var RUNTIME_CACHE = "pete-runtime-" + CACHE_VERSION;
// App shell: the static assets every page needs. Versioned by URL content only
// loosely, so we lean on network-first for HTML and cache-first for these.
var SHELL_ASSETS = [
"/static/css/output.css",
"/static/js/prefs.js",
"/static/js/weather.js",
"/static/js/weather-forecast.js",
"/static/js/search.js",
"/static/js/settings.js",
"/static/js/reader.js",
"/static/js/pwa.js",
"/static/img/pete.avif",
"/static/img/icon-192.png",
"/static/img/icon-512.png",
];
// How many visited-article responses to keep for offline reading before the
// oldest are evicted. Reader articles are small JSON blobs.
var RUNTIME_MAX = 60;
self.addEventListener("install", function (event) {
event.waitUntil(
caches.open(SHELL_CACHE).then(function (cache) {
// addAll is atomic-ish: if one asset 404s the whole install fails, so keep
// this list to assets we know are served. Individual failures are tolerated
// by falling back to per-asset puts.
return Promise.all(
SHELL_ASSETS.map(function (url) {
return cache.add(url).catch(function () {});
})
);
}).then(function () {
return self.skipWaiting();
})
);
});
self.addEventListener("activate", function (event) {
event.waitUntil(
caches.keys().then(function (keys) {
return Promise.all(
keys.map(function (k) {
if (k !== SHELL_CACHE && k !== RUNTIME_CACHE) return caches.delete(k);
})
);
}).then(function () {
return self.clients.claim();
})
);
});
// trimCache evicts the oldest entries once a runtime cache passes its cap.
function trimCache(cacheName, max) {
caches.open(cacheName).then(function (cache) {
cache.keys().then(function (keys) {
if (keys.length <= max) return;
for (var i = 0; i < keys.length - max; i++) cache.delete(keys[i]);
});
});
}
// A minimal offline page for navigations we have nothing cached for.
function offlineFallback() {
return new Response(
"<!doctype html><meta charset=utf-8><meta name=viewport content='width=device-width,initial-scale=1'>" +
"<title>Offline · Pete</title>" +
"<div style=\"font-family:system-ui,sans-serif;max-width:32rem;margin:20vh auto;padding:0 1.5rem;text-align:center;color:#3a2f1a\">" +
"<div style=font-size:3rem>🦆</div>" +
"<h1 style=font-size:1.4rem>You're offline</h1>" +
"<p style=opacity:.7>Pete can't reach the news right now. Articles you've already opened are still readable, so head back and try one of those.</p>" +
"</div>",
{ headers: { "Content-Type": "text/html; charset=utf-8" }, status: 503 }
);
}
self.addEventListener("fetch", function (event) {
var req = event.request;
if (req.method !== "GET") return;
var url = new URL(req.url);
if (url.origin !== self.location.origin) return; // never touch cross-origin
// Visited articles: network-first so text stays fresh, but cache every success
// so the reader still works offline for stories the user has already opened.
if (url.pathname === "/api/article") {
event.respondWith(
fetch(req).then(function (res) {
if (res && res.ok) {
var copy = res.clone();
caches.open(RUNTIME_CACHE).then(function (cache) {
cache.put(req, copy);
trimCache(RUNTIME_CACHE, RUNTIME_MAX);
});
}
return res;
}).catch(function () {
return caches.match(req).then(function (hit) {
return hit || new Response(JSON.stringify({ error: "offline" }), {
status: 503,
headers: { "Content-Type": "application/json" },
});
});
})
);
return;
}
// Static assets: cache-first (they're versioned by deploy), fill the cache on
// first miss so a later offline visit has them.
if (url.pathname.indexOf("/static/") === 0) {
event.respondWith(
caches.match(req).then(function (hit) {
return hit || fetch(req).then(function (res) {
if (res && res.ok) {
var copy = res.clone();
caches.open(SHELL_CACHE).then(function (cache) { cache.put(req, copy); });
}
return res;
});
})
);
return;
}
// Page navigations: network-only, falling back to the offline card when the
// network is unreachable. We deliberately do NOT cache HTML responses: pages
// are personalized (they embed the signed-in user's name/email and a "For you"
// rail), and the runtime cache is shared across everyone who uses this
// installed PWA. Caching a navigation would let a signed-out visitor — or a
// second person on the same device — be served the previous user's identity
// and personalized stories offline. Offline reading still works: the reader
// fetches cached /api/article JSON on top of the cached static shell.
if (req.mode === "navigate") {
event.respondWith(
fetch(req).catch(function () {
return offlineFallback();
})
);
return;
}
// Everything else (other /api/* calls): straight to the network. These are
// personalized/stateful and must not be served stale.
});
// ---- Web Push -------------------------------------------------------------
// The server sends a JSON payload {title, body, url, tag}. Missing fields fall
// back to sensible defaults so a malformed push still shows something useful.
self.addEventListener("push", function (event) {
var data = {};
if (event.data) {
try { data = event.data.json(); } catch (e) { data = { body: event.data.text() }; }
}
var title = data.title || "Pete";
var options = {
body: data.body || "New stories are waiting.",
icon: "/static/img/icon-192.png",
badge: "/static/img/icon-192.png",
tag: data.tag || "pete-digest",
renotify: true,
data: { url: data.url || "/" },
};
event.waitUntil(self.registration.showNotification(title, options));
});
self.addEventListener("notificationclick", function (event) {
event.notification.close();
var target = (event.notification.data && event.notification.data.url) || "/";
event.waitUntil(
self.clients.matchAll({ type: "window", includeUncontrolled: true }).then(function (clients) {
for (var i = 0; i < clients.length; i++) {
var c = clients[i];
// Focus an existing Pete tab and route it to the target if we can.
if ("focus" in c) {
c.focus();
if ("navigate" in c && target !== "/") { try { c.navigate(target); } catch (e) {} }
return;
}
}
if (self.clients.openWindow) return self.clients.openWindow(target);
})
);
});

128
internal/web/status.go Normal file
View File

@@ -0,0 +1,128 @@
package web
import (
"log/slog"
"net/http"
"sort"
"time"
"pete/internal/storage"
)
// isAdmin reports whether the request carries a signed-in session whose OIDC
// subject is on the admin allowlist. False when auth is off, the allowlist is
// empty, or the visitor is anonymous.
func (s *Server) isAdmin(r *http.Request) bool {
if s.auth == nil || len(s.adminSubs) == 0 {
return false
}
u := s.auth.userFromRequest(r)
if u == nil {
return false
}
return s.adminSubs[u.Sub]
}
// sourceStatus is one row of the source-health dashboard: the configured feed
// plus its persisted poll health and derived content stats.
type sourceStatus struct {
Name string
Channel string
Healthy bool // last poll succeeded (no consecutive failures)
NeverRun bool // no poll recorded yet
LastPollAt time.Time
LastSuccessAt time.Time
LastError string
Failures int
LastItemCount int
Total int
Classified int
Paywalled int
PaywallRate int // percent of retained stories that are gated
LastSeenAt time.Time
LastPostedAt time.Time
}
type statusPage struct {
pageData
Sources []sourceStatus
DegradedCnt int // sources currently failing
}
// handleStatus renders the owner-facing source-health dashboard. Access is
// restricted to admin subjects; everyone else gets a 404 so the page's
// existence isn't advertised.
func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) {
if !s.isAdmin(r) {
http.NotFound(w, r)
return
}
s.track(r, "status")
health, err := storage.ListSourceHealth()
if err != nil {
slog.Error("web: source health query failed", "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
stats, err := storage.SourceContentStats()
if err != nil {
slog.Error("web: source content stats failed", "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
rows := make([]sourceStatus, 0, len(s.sources))
degraded := 0
for _, src := range s.sources {
h, hasHealth := health[src.Name]
st := stats[src.Name]
row := sourceStatus{
Name: src.Name,
Channel: src.Channel,
NeverRun: !hasHealth || h.LastPollAt == 0,
LastError: h.LastError,
Failures: h.ConsecutiveFailures,
LastItemCount: h.LastItemCount,
Total: st.Total,
Classified: st.Classified,
Paywalled: st.Paywalled,
}
row.Healthy = hasHealth && h.ConsecutiveFailures == 0
if h.LastPollAt > 0 {
row.LastPollAt = time.Unix(h.LastPollAt, 0)
}
if h.LastSuccessAt > 0 {
row.LastSuccessAt = time.Unix(h.LastSuccessAt, 0)
}
if st.LastSeenAt > 0 {
row.LastSeenAt = time.Unix(st.LastSeenAt, 0)
}
if st.LastPostedAt > 0 {
row.LastPostedAt = time.Unix(st.LastPostedAt, 0)
}
if st.Total > 0 {
row.PaywallRate = st.Paywalled * 100 / st.Total
}
if !row.NeverRun && !row.Healthy {
degraded++
}
rows = append(rows, row)
}
// Failing sources first (most consecutive failures), then healthy ones by
// name, so the owner's eye lands on what needs attention.
sort.SliceStable(rows, func(i, j int) bool {
if rows[i].Failures != rows[j].Failures {
return rows[i].Failures > rows[j].Failures
}
return rows[i].Name < rows[j].Name
})
base := s.base(r)
base.Active = "status"
s.render(w, "status", statusPage{pageData: base, Sources: rows, DegradedCnt: degraded})
}

View File

@@ -0,0 +1,38 @@
package web
import (
"strings"
"testing"
"time"
"pete/internal/config"
)
func TestStatusTemplateExecutes(t *testing.T) {
s, err := New(config.WebConfig{SiteTitle: "Pete", ListenAddr: ":0"}, nil, true)
if err != nil {
t.Fatal(err)
}
data := statusPage{
pageData: pageData{SiteTitle: "Pete", Channels: channels},
DegradedCnt: 1,
Sources: []sourceStatus{
{Name: "Broken Feed", Channel: "tech", Failures: 3, LastError: "dial tcp: timeout",
LastPollAt: time.Now(), NeverRun: false, Healthy: false, Total: 4, Paywalled: 2, PaywallRate: 50},
{Name: "Good Feed", Channel: "eu", Healthy: true, LastPollAt: time.Now(),
LastSuccessAt: time.Now(), LastItemCount: 12, Total: 30, Classified: 28,
LastSeenAt: time.Now(), LastPostedAt: time.Now()},
{Name: "Idle Feed", Channel: "music", NeverRun: true},
},
}
var b strings.Builder
if err := s.tpls["status"].ExecuteTemplate(&b, "layout", data); err != nil {
t.Fatal(err)
}
out := b.String()
for _, want := range []string{"Broken Feed", "Good Feed", "dial tcp: timeout", "1 degraded", "Source health"} {
if !strings.Contains(out, want) {
t.Errorf("rendered status page missing %q", want)
}
}
}

View File

@@ -1,7 +1,23 @@
{{define "card"}} {{define "card"}}{{$ch := channel .Story.Channel}}
<a href="{{.Story.ArticleURL}}" target="_blank" rel="noopener noreferrer" <a href="{{.Story.ArticleURL}}" target="_blank" rel="noopener noreferrer"
data-story-card data-source="{{.Story.Source}}" data-channel="{{.Story.Channel}}" data-story-card data-source="{{.Story.Source}}" data-channel="{{.Story.Channel}}"
data-id="{{.Story.ID}}"
data-headline="{{.Story.Headline}}"
data-lede="{{.Story.Lede}}"
data-url="{{.Story.ArticleURL}}"
data-image="{{if .Story.ImageURL}}{{thumb .Story.ImageURL}}{{end}}"
data-time="{{timeAgo .Story.SeenAt}}"
data-ch-title="{{$ch.Title}}" data-ch-emoji="{{$ch.Emoji}}" data-ch-theme="{{$ch.Theme}}"
data-posted="{{if .Story.Posted}}1{{end}}"
data-paywalled="{{if .Story.Paywalled}}1{{end}}"
class="group relative block rounded-3xl bg-[color:var(--card)] border-2 {{if .Story.Posted}}border-theme-{{.Theme}} glow-theme-{{.Theme}}{{else}}border-[color:var(--ink)]/10{{end}} shadow-pete overflow-hidden hover:-translate-y-1 hover:shadow-pete-lg transition"> class="group relative block rounded-3xl bg-[color:var(--card)] border-2 {{if .Story.Posted}}border-theme-{{.Theme}} glow-theme-{{.Theme}}{{else}}border-[color:var(--ink)]/10{{end}} shadow-pete overflow-hidden hover:-translate-y-1 hover:shadow-pete-lg transition">
<span role="button" tabindex="0" data-bookmark-btn data-story-id="{{.Story.ID}}"
aria-label="Bookmark this story" aria-pressed="false"
style="display:none;position:absolute;top:.6rem;left:.6rem;z-index:6;height:1.9rem;width:1.9rem;align-items:center;justify-content:center;border-radius:9999px;background:rgba(20,14,6,.62);color:#fff;-webkit-backdrop-filter:blur(2px);backdrop-filter:blur(2px)">
<svg viewBox="0 0 24 24" width="15" height="15" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<path d="M6 2h12a1 1 0 0 1 1 1v18l-7-4-7 4V3a1 1 0 0 1 1-1z"></path>
</svg>
</span>
{{if .Story.ImageURL}} {{if .Story.ImageURL}}
<div class="aspect-[16/10] w-full overflow-hidden bg-[color:var(--ink)]/5"> <div class="aspect-[16/10] w-full overflow-hidden bg-[color:var(--ink)]/5">
<img src="{{thumb .Story.ImageURL}}" alt="" loading="lazy" decoding="async" <img src="{{thumb .Story.ImageURL}}" alt="" loading="lazy" decoding="async"

View File

@@ -0,0 +1,37 @@
{{define "title"}}Bookmarks — {{.SiteTitle}}{{end}}
{{define "main"}}
<section class="mt-2 mb-8">
<div class="rounded-3xl bg-[color:var(--accent)] text-[#1c1305] p-6 sm:p-10 shadow-pete relative overflow-hidden">
<div class="absolute -top-6 -right-6 text-[12rem] opacity-20 select-none" aria-hidden="true">🔖</div>
<div class="relative">
<p class="text-sm uppercase tracking-[0.2em] opacity-70">saved for later</p>
<h1 class="font-display text-4xl sm:text-5xl font-bold mt-1">Bookmarks</h1>
<p class="mt-2 max-w-xl opacity-80">Stories you've tucked away. Tap the bookmark on any card to add or remove one.</p>
<p class="mt-4 text-xs uppercase tracking-wider opacity-75">{{.Total}} saved</p>
</div>
</div>
</section>
{{if .Stories}}
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 gap-5">
{{range .Stories}}
{{template "card" dict "Story" . "Theme" .Channel "ShowChannel" true}}
{{end}}
</div>
<nav class="mt-10 flex items-center justify-between">
{{if .HasPrev}}
<a href="{{.PrevURL}}" class="rounded-full bg-[color:var(--card)] border-2 border-[color:var(--ink)]/10 px-5 py-2 font-semibold shadow-pete hover:-translate-y-0.5 transition">← newer</a>
{{else}}<span></span>{{end}}
<span class="text-sm text-[color:var(--ink)]/60">page {{.Page}}</span>
{{if .HasNext}}
<a href="{{.NextURL}}" class="rounded-full bg-[color:var(--accent)] text-[#1c1305] px-5 py-2 font-semibold shadow-pete hover:-translate-y-0.5 transition">older →</a>
{{else}}<span></span>{{end}}
</nav>
{{else}}
<div class="rounded-3xl bg-[color:var(--card)] border-2 border-dashed border-[color:var(--ink)]/15 p-10 text-center text-[color:var(--ink)]/60">
Nothing saved yet. Browse the feed and tap the 🔖 on a story to keep it here.
</div>
{{end}}
{{end}}

View File

@@ -0,0 +1,26 @@
{{define "title"}}For you — {{.SiteTitle}}{{end}}
{{define "main"}}
<section class="mt-2 mb-8">
<div class="rounded-3xl bg-[color:var(--accent)] text-[#1c1305] p-6 sm:p-10 shadow-pete relative overflow-hidden">
<div class="absolute -top-6 -right-6 text-[12rem] opacity-20 select-none" aria-hidden="true"></div>
<div class="relative">
<p class="text-sm uppercase tracking-[0.2em] opacity-70">picked for you</p>
<h1 class="font-display text-4xl sm:text-5xl font-bold mt-1">For you</h1>
<p class="mt-2 max-w-xl opacity-80">Fresh stories weighted toward the channels and sources you read and bookmark most. The more you read, the sharper this gets.</p>
</div>
</div>
</section>
{{if .Stories}}
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 gap-5">
{{range .Stories}}
{{template "card" dict "Story" . "Theme" .Channel "ShowChannel" true}}
{{end}}
</div>
{{else}}
<div class="rounded-3xl bg-[color:var(--card)] border-2 border-dashed border-[color:var(--ink)]/15 p-10 text-center text-[color:var(--ink)]/60">
Nothing to tune yet. Read a few stories or bookmark the ones you like, and Pete will start building a feed around them.
</div>
{{end}}
{{end}}

View File

@@ -6,14 +6,34 @@
a friendlier news feed. a friendlier news feed.
</h1> </h1>
<p class="mx-auto mt-3 max-w-xl text-base sm:text-lg text-[color:var(--ink)]/70"> <p class="mx-auto mt-3 max-w-xl text-base sm:text-lg text-[color:var(--ink)]/70">
{{if .PostingEnabled}}
Pete reads RSS, sorts stories, and posts them to Matrix. Pete reads RSS, sorts stories, and posts them to Matrix.
Glowing cards are the ones he's posted. Glowing cards are the ones he's posted.
{{else}}
Pete reads RSS and sorts stories into channels, fresh from his feeds.
{{end}}
</p> </p>
</section> </section>
<section data-weather-card class="mb-10 empty:hidden"></section> <section data-weather-card class="mb-10 empty:hidden"></section>
{{if .JustPosted}} {{if .ForYou}}
<section class="mb-10">
<div class="flex items-end justify-between gap-3 mb-4">
<h2 class="font-display text-xl sm:text-2xl font-bold">
<span aria-hidden="true"></span> For you
</h2>
<a href="/for-you" class="text-xs font-semibold text-[color:var(--ink)]/60 hover:text-[color:var(--ink)]">more →</a>
</div>
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-4">
{{range .ForYou}}
{{template "card" dict "Story" . "Theme" .Channel "ShowChannel" true}}
{{end}}
</div>
</section>
{{end}}
{{if and .PostingEnabled .JustPosted}}
<section class="mb-10"> <section class="mb-10">
<div class="flex items-end justify-between gap-3 mb-4"> <div class="flex items-end justify-between gap-3 mb-4">
<h2 class="font-display text-xl sm:text-2xl font-bold"> <h2 class="font-display text-xl sm:text-2xl font-bold">
@@ -34,7 +54,7 @@
<h2 class="font-display text-xl sm:text-2xl font-bold"> <h2 class="font-display text-xl sm:text-2xl font-bold">
<span aria-hidden="true">📊</span> Channels <span aria-hidden="true">📊</span> Channels
</h2> </h2>
<span class="text-xs text-[color:var(--ink)]/50">last 24 hours</span> <span class="text-xs text-[color:var(--ink)]/50">{{if .PostingEnabled}}last 24 hours{{else}}what Pete's tracking{{end}}</span>
</div> </div>
<div class="grid grid-cols-2 lg:grid-cols-4 gap-3 sm:gap-4"> <div class="grid grid-cols-2 lg:grid-cols-4 gap-3 sm:gap-4">
{{range .Stats}} {{range .Stats}}
@@ -47,6 +67,7 @@
<span class="font-display text-lg font-semibold">{{.Channel.Title}}</span> <span class="font-display text-lg font-semibold">{{.Channel.Title}}</span>
</div> </div>
<dl class="mt-3 space-y-1 text-xs text-[color:var(--ink)]/70"> <dl class="mt-3 space-y-1 text-xs text-[color:var(--ink)]/70">
{{if $.PostingEnabled}}
<div class="flex justify-between gap-2"> <div class="flex justify-between gap-2">
<dt>last post</dt> <dt>last post</dt>
<dd class="font-semibold text-[color:var(--ink)]"> <dd class="font-semibold text-[color:var(--ink)]">
@@ -57,6 +78,7 @@
<dt>posted 24h</dt> <dt>posted 24h</dt>
<dd class="font-semibold text-[color:var(--ink)]">{{.PostsToday}}</dd> <dd class="font-semibold text-[color:var(--ink)]">{{.PostsToday}}</dd>
</div> </div>
{{end}}
<div class="flex justify-between gap-2"> <div class="flex justify-between gap-2">
<dt>stories</dt> <dt>stories</dt>
<dd class="font-semibold text-[color:var(--ink)]">{{.Total}}</dd> <dd class="font-semibold text-[color:var(--ink)]">{{.Total}}</dd>

View File

@@ -9,6 +9,17 @@
<link href="https://fonts.googleapis.com/css2?family=Fredoka:wght@400;500;600;700&family=Nunito:wght@400;600;700&display=swap" rel="stylesheet"> <link href="https://fonts.googleapis.com/css2?family=Fredoka:wght@400;500;600;700&family=Nunito:wght@400;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="/static/css/output.css"> <link rel="stylesheet" href="/static/css/output.css">
<link rel="icon" href="/static/img/pete.avif" type="image/avif"> <link rel="icon" href="/static/img/pete.avif" type="image/avif">
<link rel="manifest" href="/manifest.webmanifest">
<meta name="theme-color" content="#fbf3e3">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="default">
<meta name="apple-mobile-web-app-title" content="Pete">
<link rel="apple-touch-icon" href="/static/img/apple-touch-icon.png">
<link rel="alternate" type="application/rss+xml" title="{{.SiteTitle}} — all stories" href="/feed.xml">
<link rel="alternate" type="application/feed+json" title="{{.SiteTitle}} — all stories" href="/feed.json">
{{if .Active}}{{if ne .Active "bookmarks"}}
<link rel="alternate" type="application/rss+xml" title="{{.SiteTitle}} — {{.Active}}" href="/{{.Active}}/feed.xml">
{{end}}{{end}}
<script> <script>
// Pick a palette phase from the browser clock and update it each minute. // Pick a palette phase from the browser clock and update it each minute.
(function () { (function () {
@@ -49,6 +60,16 @@
</svg> </svg>
<span class="sr-only">Feed settings</span> <span class="sr-only">Feed settings</span>
</button> </button>
<button type="button" data-reader-open
title="Feed mode — read one story at a time (press f)"
class="inline-flex shrink-0 items-center justify-center rounded-full bg-[color:var(--card)] p-2 shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round" class="h-5 w-5">
<path d="M2 4h6a3 3 0 0 1 3 3v13a2.5 2.5 0 0 0-2.5-2.5H2z"></path>
<path d="M22 4h-6a3 3 0 0 0-3 3v13a2.5 2.5 0 0 1 2.5-2.5H22z"></path>
</svg>
<span class="sr-only">Feed mode</span>
</button>
{{if .Weather.Variant}} {{if .Weather.Variant}}
<button type="button" data-weather-toggle aria-pressed="true" <button type="button" data-weather-toggle aria-pressed="true"
title="Toggle weather animation" title="Toggle weather animation"
@@ -69,6 +90,26 @@
class="hidden shrink-0 items-center gap-1.5 rounded-full bg-[color:var(--card)] px-3 py-2 text-sm font-semibold shadow-pete border-2 border-[color:var(--ink)]/10 tabular-nums"></span> class="hidden shrink-0 items-center gap-1.5 rounded-full bg-[color:var(--card)] px-3 py-2 text-sm font-semibold shadow-pete border-2 border-[color:var(--ink)]/10 tabular-nums"></span>
{{if .AuthEnabled}} {{if .AuthEnabled}}
{{if .User}} {{if .User}}
<a href="/bookmarks" data-bookmarks-link
title="Your bookmarks"
class="inline-flex shrink-0 items-center justify-center rounded-full bg-[color:var(--card)] p-2 shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition{{if eq .Active "bookmarks"}} bg-[color:var(--accent)]/20{{end}}">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round" class="h-5 w-5">
<path d="M6 2h12a1 1 0 0 1 1 1v18l-7-4-7 4V3a1 1 0 0 1 1-1z"></path>
</svg>
<span class="sr-only">Bookmarks</span>
</a>
{{if .IsAdmin}}
<a href="/status" data-status-link
title="Source health"
class="inline-flex shrink-0 items-center justify-center rounded-full bg-[color:var(--card)] p-2 shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition{{if eq .Active "status"}} bg-[color:var(--accent)]/20{{end}}">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round" class="h-5 w-5">
<path d="M3 12h4l2 6 4-14 2 8h6"></path>
</svg>
<span class="sr-only">Source health</span>
</a>
{{end}}
<a href="/auth/logout" data-account <a href="/auth/logout" data-account
title="Signed in as {{.User.Display}}{{if .User.Email}} · {{.User.Email}}{{end}} — sign out" title="Signed in as {{.User.Display}}{{if .User.Email}} · {{.User.Email}}{{end}} — sign out"
class="inline-flex shrink-0 items-center gap-2 rounded-full bg-[color:var(--card)] px-2.5 py-1.5 text-sm font-semibold shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition"> class="inline-flex shrink-0 items-center gap-2 rounded-full bg-[color:var(--card)] px-2.5 py-1.5 text-sm font-semibold shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition">
@@ -102,6 +143,14 @@
</button> </button>
<nav class="pete-channel-nav flex min-w-0 items-center gap-1 sm:gap-2 overflow-x-auto rounded-full bg-[color:var(--card)] p-1 shadow-pete border-2 border-[color:var(--ink)]/10"> <nav class="pete-channel-nav flex min-w-0 items-center gap-1 sm:gap-2 overflow-x-auto rounded-full bg-[color:var(--card)] p-1 shadow-pete border-2 border-[color:var(--ink)]/10">
{{$active := .Active}} {{$active := .Active}}
{{if .User}}
<a href="/for-you"
title="Your personalized feed"
class="shrink-0 rounded-full px-3 py-2 text-sm font-semibold transition
{{if eq $active "for-you"}}bg-[color:var(--accent)] text-[#1c1305] shadow-sm{{else}}hover:bg-[color:var(--ink)]/5{{end}}">
<span aria-hidden="true" class="mr-1"></span><span class="hidden sm:inline">For you</span>
</a>
{{end}}
{{range .Channels}} {{range .Channels}}
<a href="/{{.Slug}}" <a href="/{{.Slug}}"
class="shrink-0 rounded-full px-3 py-2 text-sm font-semibold transition class="shrink-0 rounded-full px-3 py-2 text-sm font-semibold transition
@@ -136,6 +185,20 @@
<h2 class="font-display text-lg font-bold">Feed settings</h2> <h2 class="font-display text-lg font-bold">Feed settings</h2>
<button type="button" data-settings-close class="rounded-full px-2 py-1 text-sm text-[color:var(--ink)]/60 hover:bg-[color:var(--ink)]/5">esc</button> <button type="button" data-settings-close class="rounded-full px-2 py-1 text-sm text-[color:var(--ink)]/60 hover:bg-[color:var(--ink)]/5">esc</button>
</div> </div>
{{if .PushEnabled}}{{if .User}}
<div data-push-section hidden class="px-5 pt-4 pb-1">
<div class="flex items-center justify-between gap-3 rounded-2xl bg-[color:var(--ink)]/5 px-4 py-3">
<div class="min-w-0">
<div class="text-sm font-bold">🔔 Notifications</div>
<div data-push-note class="text-xs text-[color:var(--ink)]/60">Get a nudge when new stories land.</div>
</div>
<button type="button" data-push-toggle aria-pressed="false"
class="shrink-0 rounded-full bg-[color:var(--accent)] px-3 py-2 text-sm font-semibold text-[#1c1305] shadow-sm hover:brightness-105 transition disabled:opacity-50 disabled:cursor-not-allowed">
Turn on notifications
</button>
</div>
</div>
{{end}}{{end}}
<p class="px-5 pt-3 text-xs text-[color:var(--ink)]/60">Uncheck a feed to hide its stories. <span data-storage-note>Saved in this browser.</span></p> <p class="px-5 pt-3 text-xs text-[color:var(--ink)]/60">Uncheck a feed to hide its stories. <span data-storage-note>Saved in this browser.</span></p>
<div data-settings-list class="max-h-[55vh] overflow-y-auto p-3 space-y-1"></div> <div data-settings-list class="max-h-[55vh] overflow-y-auto p-3 space-y-1"></div>
<div class="px-5 py-3 border-t border-[color:var(--ink)]/10 flex items-center justify-between text-xs"> <div class="px-5 py-3 border-t border-[color:var(--ink)]/10 flex items-center justify-between text-xs">
@@ -145,6 +208,29 @@
</div> </div>
</div> </div>
<div id="pete-reader" class="hidden fixed inset-0 z-50" aria-modal="true" role="dialog" aria-label="Feed mode reader">
<div class="pete-reader-backdrop" data-reader-backdrop></div>
<div class="pete-reader-shell">
<div class="pete-reader-bar">
<span class="pete-reader-progress tabular-nums" data-reader-progress></span>
<div class="pete-reader-nav">
<button type="button" data-reader-prev class="pete-reader-btn" title="Previous (←)" aria-label="Previous article"></button>
<button type="button" data-reader-next class="pete-reader-btn" title="Next (→)" aria-label="Next article"></button>
<button type="button" data-reader-bookmark class="pete-reader-btn" style="display:none" title="Bookmark (b)" aria-label="Bookmark this story" aria-pressed="false">🔖 Save</button>
<a data-reader-link target="_blank" rel="noopener noreferrer" class="pete-reader-btn pete-reader-btn-open" title="Open original (o)">Open&nbsp;</a>
<button type="button" data-reader-close class="pete-reader-btn" title="Close (esc)" aria-label="Close reader">esc</button>
</div>
</div>
<div class="pete-reader-scroll" data-reader-scroll>
<article class="pete-reader-article" data-reader-article></article>
<div class="pete-reader-related" data-reader-related hidden></div>
</div>
<div class="pete-reader-hint">
<span><kbd></kbd> <kbd></kbd> navigate · <kbd>o</kbd> open · <kbd>u</kbd> mark unread · <kbd>esc</kbd> close</span>
</div>
</div>
</div>
<main class="mx-auto max-w-6xl px-4 pb-24"> <main class="mx-auto max-w-6xl px-4 pb-24">
{{block "main" .}}{{end}} {{block "main" .}}{{end}}
</main> </main>
@@ -176,11 +262,14 @@
window.PETE_SOURCES = {{.AllSources}}; window.PETE_SOURCES = {{.AllSources}};
window.PETE_USER = {{if .User}}{ name: {{.User.Display}}, email: {{.User.Email}} }{{else}}null{{end}}; window.PETE_USER = {{if .User}}{ name: {{.User.Display}}, email: {{.User.Email}} }{{else}}null{{end}};
window.PETE_PREFS = {{.UserPrefs}}; window.PETE_PREFS = {{.UserPrefs}};
window.PETE_PUSH = {{if .PushEnabled}}{ enabled: true, publicKey: {{.PushPublicKey}} }{{else}}null{{end}};
</script> </script>
<script src="/static/js/prefs.js" defer></script> <script src="/static/js/prefs.js" defer></script>
<script src="/static/js/weather.js" defer></script> <script src="/static/js/weather.js" defer></script>
<script src="/static/js/weather-forecast.js" defer></script> <script src="/static/js/weather-forecast.js" defer></script>
<script src="/static/js/search.js" defer></script> <script src="/static/js/search.js" defer></script>
<script src="/static/js/settings.js" defer></script> <script src="/static/js/settings.js" defer></script>
<script src="/static/js/reader.js" defer></script>
<script src="/static/js/pwa.js" defer></script>
</body> </body>
</html>{{end}} </html>{{end}}

View File

@@ -0,0 +1,80 @@
{{define "title"}}Source health — {{.SiteTitle}}{{end}}
{{define "main"}}
<section class="mt-2 mb-8">
<div class="rounded-3xl bg-[color:var(--card)] border-2 border-[color:var(--ink)]/10 p-6 sm:p-8 shadow-pete">
<p class="text-sm uppercase tracking-[0.2em] text-[color:var(--ink)]/50">owner view</p>
<h1 class="font-display text-3xl sm:text-4xl font-bold mt-1">Source health</h1>
<p class="mt-2 max-w-2xl text-[color:var(--ink)]/70">
Per-feed poll status and content stats. Rows that are currently failing float to the top.
</p>
<div class="mt-4 flex flex-wrap gap-2">
<span class="inline-flex items-center gap-1.5 rounded-full bg-[color:var(--ink)]/5 px-3 py-1 text-sm font-semibold tabular-nums">
{{len .Sources}} sources
</span>
{{if .DegradedCnt}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-red-500/15 text-red-700 dark:text-red-300 px-3 py-1 text-sm font-semibold tabular-nums">
⚠ {{.DegradedCnt}} degraded
</span>
{{else}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-emerald-500/15 text-emerald-700 dark:text-emerald-300 px-3 py-1 text-sm font-semibold">
✓ all healthy
</span>
{{end}}
</div>
</div>
</section>
<div class="overflow-x-auto rounded-3xl bg-[color:var(--card)] border-2 border-[color:var(--ink)]/10 shadow-pete">
<table class="w-full min-w-[52rem] text-sm">
<thead>
<tr class="text-left text-[color:var(--ink)]/50 uppercase text-xs tracking-wider border-b-2 border-[color:var(--ink)]/10">
<th class="px-4 py-3 font-semibold">Source</th>
<th class="px-4 py-3 font-semibold">Channel</th>
<th class="px-4 py-3 font-semibold">Status</th>
<th class="px-4 py-3 font-semibold">Last poll</th>
<th class="px-4 py-3 font-semibold">Last success</th>
<th class="px-4 py-3 font-semibold text-right">Items</th>
<th class="px-4 py-3 font-semibold text-right">Stories</th>
<th class="px-4 py-3 font-semibold text-right">Paywall</th>
<th class="px-4 py-3 font-semibold">Last story</th>
<th class="px-4 py-3 font-semibold">Last posted</th>
</tr>
</thead>
<tbody>
{{range .Sources}}
<tr class="border-b border-[color:var(--ink)]/5 last:border-0 align-top">
<td class="px-4 py-3 font-semibold whitespace-nowrap">{{.Name}}</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{.Channel}}</td>
<td class="px-4 py-3 whitespace-nowrap">
{{if .NeverRun}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-[color:var(--ink)]/10 px-2.5 py-1 text-xs font-semibold">◌ idle</span>
{{else if .Healthy}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-emerald-500/15 text-emerald-700 dark:text-emerald-300 px-2.5 py-1 text-xs font-semibold">● ok</span>
{{else}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-red-500/15 text-red-700 dark:text-red-300 px-2.5 py-1 text-xs font-semibold"
title="{{.LastError}}">✕ {{.Failures}} fail{{if ne .Failures 1}}s{{end}}</span>
{{end}}
</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastPollAt.IsZero}}never{{else}}{{timeAgo .LastPollAt}}{{end}}</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastSuccessAt.IsZero}}never{{else}}{{timeAgo .LastSuccessAt}}{{end}}</td>
<td class="px-4 py-3 text-right tabular-nums">{{.LastItemCount}}</td>
<td class="px-4 py-3 text-right tabular-nums" title="{{.Classified}} classified of {{.Total}}">{{.Total}}</td>
<td class="px-4 py-3 text-right tabular-nums {{if gt .PaywallRate 50}}text-amber-600 dark:text-amber-400 font-semibold{{end}}">
{{if .Total}}{{.PaywallRate}}%{{else}}—{{end}}
</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastSeenAt.IsZero}}—{{else}}{{timeAgo .LastSeenAt}}{{end}}</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastPostedAt.IsZero}}—{{else}}{{timeAgo .LastPostedAt}}{{end}}</td>
</tr>
{{if .LastError}}{{if not .Healthy}}
<tr class="border-b border-[color:var(--ink)]/5">
<td colspan="10" class="px-4 pb-3 -mt-1 text-xs text-red-600 dark:text-red-400 font-mono break-all">{{.LastError}}</td>
</tr>
{{end}}{{end}}
{{else}}
<tr><td colspan="10" class="px-4 py-8 text-center text-[color:var(--ink)]/50">No sources configured.</td></tr>
{{end}}
</tbody>
</table>
</div>
{{end}}

View File

@@ -0,0 +1,159 @@
package web
import (
"encoding/json"
"io"
"log/slog"
"net/http"
"strconv"
"strings"
"pete/internal/storage"
)
// maxStateBodyBytes caps read/bookmark request bodies. These carry a single id
// and a boolean, so this is generous headroom.
const maxStateBodyBytes = 1024
// maxStateIDs bounds how many ids /api/state will look up in one call. A page
// renders a few dozen cards; this is well clear of that.
const maxStateIDs = 500
// requireUser resolves the signed-in user or writes the appropriate error and
// returns nil. It mirrors handlePrefs: 404 when auth is disabled entirely, 401
// for anonymous callers (the client's cue to stay on localStorage).
func (s *Server) requireUser(w http.ResponseWriter, r *http.Request) *SessionUser {
if s.auth == nil {
http.Error(w, "auth disabled", http.StatusNotFound)
return nil
}
u := s.auth.userFromRequest(r)
if u == nil {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
http.Error(w, `{"error":"not signed in"}`, http.StatusUnauthorized)
return nil
}
return u
}
// decodeStateBody reads a small JSON body into dst, writing a 400 on failure.
func decodeStateBody(w http.ResponseWriter, r *http.Request, dst any) bool {
return decodeStateBodyN(w, r, dst, maxStateBodyBytes)
}
// decodeStateBodyN is decodeStateBody with an explicit byte cap, for endpoints
// (like push subscribe) whose payloads run larger than a single id + flag.
func decodeStateBodyN(w http.ResponseWriter, r *http.Request, dst any, limit int64) bool {
body, err := io.ReadAll(io.LimitReader(r.Body, limit+1))
if err != nil || int64(len(body)) > limit {
http.Error(w, `{"error":"bad request"}`, http.StatusBadRequest)
return false
}
if err := json.Unmarshal(body, dst); err != nil {
http.Error(w, `{"error":"invalid JSON"}`, http.StatusBadRequest)
return false
}
return true
}
// handleRead records or clears a story's read flag for the signed-in user.
func (s *Server) handleRead(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
var req struct {
ID int64 `json:"id"`
Read bool `json:"read"`
}
if !decodeStateBody(w, r, &req) {
return
}
if req.ID <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
if err := storage.SetRead(u.Sub, req.ID, req.Read); err != nil {
slog.Error("state: set read failed", "sub", u.Sub, "id", req.ID, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleBookmark adds or removes a bookmark for the signed-in user.
func (s *Server) handleBookmark(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
var req struct {
ID int64 `json:"id"`
On bool `json:"on"`
}
if !decodeStateBody(w, r, &req) {
return
}
if req.ID <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
if err := storage.SetBookmark(u.Sub, req.ID, req.On); err != nil {
slog.Error("state: set bookmark failed", "sub", u.Sub, "id", req.ID, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleState returns which of the given story ids are read and bookmarked, so
// the client can paint a freshly rendered page in one round-trip.
func (s *Server) handleState(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
ids := parseIDList(r.URL.Query().Get("ids"))
read, bookmarked, err := storage.UserStoryState(u.Sub, ids)
if err != nil {
slog.Error("state: lookup failed", "sub", u.Sub, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Cache-Control", "no-store")
_ = json.NewEncoder(w).Encode(map[string]any{
"read": keysOf(read),
"bookmarked": keysOf(bookmarked),
})
}
// parseIDList turns "1,2,3" into a bounded, deduplicated slice of positive ids.
func parseIDList(raw string) []int64 {
if raw == "" {
return nil
}
seen := make(map[int64]bool)
out := make([]int64, 0)
for _, part := range strings.Split(raw, ",") {
id, err := strconv.ParseInt(strings.TrimSpace(part), 10, 64)
if err != nil || id <= 0 || seen[id] {
continue
}
seen[id] = true
out = append(out, id)
if len(out) >= maxStateIDs {
break
}
}
return out
}
// keysOf returns the keys of a set as a slice (order unspecified).
func keysOf(m map[int64]bool) []int64 {
out := make([]int64, 0, len(m))
for k := range m {
out = append(out, k)
}
return out
}

30
main.go
View File

@@ -21,9 +21,22 @@ import (
"pete/internal/storage" "pete/internal/storage"
"pete/internal/web" "pete/internal/web"
webpush "github.com/SherClockHolmes/webpush-go"
"maunium.net/go/mautrix/id" "maunium.net/go/mautrix/id"
) )
// runGenVAPID prints a fresh VAPID keypair for the [web.push] config block.
func runGenVAPID() {
priv, pub, err := webpush.GenerateVAPIDKeys()
if err != nil {
fmt.Fprintln(os.Stderr, "genvapid: failed:", err)
os.Exit(1)
}
fmt.Println("# Add these under [web.push] in your config:")
fmt.Printf("vapid_public_key = %q\n", pub)
fmt.Printf("vapid_private_key = %q\n", priv)
}
func main() { func main() {
configPath := flag.String("config", "config.toml", "path to config file") configPath := flag.String("config", "config.toml", "path to config file")
seed := flag.Bool("seed", false, "ingest current feed items as seen without posting, then exit") seed := flag.Bool("seed", false, "ingest current feed items as seen without posting, then exit")
@@ -31,8 +44,15 @@ func main() {
testSource := flag.String("test-source", "", "source name to use for -test (default: first enabled)") testSource := flag.String("test-source", "", "source name to use for -test (default: first enabled)")
local := flag.Bool("local", false, "web/RSS-only mode: poll feeds and serve the web UI, no Matrix login or posting") local := flag.Bool("local", false, "web/RSS-only mode: poll feeds and serve the web UI, no Matrix login or posting")
backfillPaywall := flag.Bool("backfill-paywall", false, "re-check paywalled rows with current detection logic; clear false positives and fill missing thumbnails, then exit") backfillPaywall := flag.Bool("backfill-paywall", false, "re-check paywalled rows with current detection logic; clear false positives and fill missing thumbnails, then exit")
genVAPID := flag.Bool("genvapid", false, "generate a VAPID keypair for web.push and print it, then exit")
flag.Parse() flag.Parse()
// -genvapid needs neither config nor database; handle it before anything else.
if *genVAPID {
runGenVAPID()
return
}
slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{ slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{
Level: slog.LevelInfo, Level: slog.LevelInfo,
}))) })))
@@ -240,11 +260,12 @@ func main() {
// Start the read-only web UI alongside the Matrix bot. // Start the read-only web UI alongside the Matrix bot.
if cfg.Web.Enabled { if cfg.Web.Enabled {
ws, err := web.New(cfg.Web, cfg.Sources) ws, err := web.New(cfg.Web, cfg.Sources, postingEnabled)
if err != nil { if err != nil {
slog.Error("web server init failed", "err", err) slog.Error("web server init failed", "err", err)
} else { } else {
go ws.Start(ctx) go ws.Start(ctx)
ws.StartPushSender(ctx)
} }
} }
@@ -349,12 +370,17 @@ func runLocal(cfg *config.Config) {
poller.Start(ctx) poller.Start(ctx)
slog.Info("local: pollers started") slog.Info("local: pollers started")
ws, err := web.New(cfg.Web, cfg.Sources) // Local mode never connects to Matrix, so it's always web-only.
ws, err := web.New(cfg.Web, cfg.Sources, false)
if err != nil { if err != nil {
slog.Error("local: web server init failed", "err", err) slog.Error("local: web server init failed", "err", err)
os.Exit(1) os.Exit(1)
} }
go ws.Start(ctx) go ws.Start(ctx)
// Push only needs the web auth layer, not Matrix, so a web-only deployment
// still runs the digest sender — otherwise subscriptions accumulate but no
// digest ever fires.
ws.StartPushSender(ctx)
slog.Info("local: web UI listening", "addr", cfg.Web.ListenAddr) slog.Info("local: web UI listening", "addr", cfg.Web.ListenAddr)
storage.RunMaintenance() storage.RunMaintenance()