Compare commits

..

4 Commits

Author SHA1 Message Date
prosolis
35850eaf73 Fix push-digest sentinel filter, watermark, and cleanups from review
- push digest queries now exclude _duplicate channel like every other
  visibility query (bookmarks list/count and NewClassifiedSince)
- advance push watermark to newest scanned story, not pass-start now, so
  stories arriving during a long send pass aren't re-counted next pass
- replace hand-rolled escapeHTMLText with stdlib html.EscapeString
- drop em-dashes from user-facing copy; bump PWA CACHE_VERSION so clients
  pick up the changed shell assets
2026-07-07 01:25:23 -07:00
prosolis
8863b75916 Fix push SSRF, cross-user unsub, and personalization edge cases
Code review of the personalization/feeds/PWA/push work surfaced ten
confirmed issues, now fixed:

- Web Push delivery bypassed the SSRF guard (unguarded default client);
  now routes through safehttp.NewClient with a hard timeout, and the
  subscribe handler validates the endpoint URL.
- Push unsubscribe deleted by endpoint with no owner check; added
  RemovePushSubscriptionForUser scoped to the signed-in user.
- Byte-slice body/content truncation could split a UTF-8 rune and break
  the RSS content:encoded XML; added a rune-safe truncateUTF8 helper.
- Digest sender could permanently starve a user who hid a high-volume
  source; step the watermark past a full hidden-source scan window.
- Service worker cached personalized HTML navigations into a shared
  cache (identity leak across PWA users); navigations are now
  network-only, CACHE_VERSION bumped to v2 to purge stale pages.
- Public /api/article leaked discarded/unclassified bodies; filter to
  classified, non-sentinel stories.
- runLocal never started the push sender; digests now fire in -local.
- Push client had no timeout, so one hung endpoint stalled all sends.
- Reader migration resurrected cross-device-cleared reads; gate it
  behind a one-time flag so the server stays authoritative.
- Bookmarks count didn't match the classified list filter.
2026-07-07 01:08:42 -07:00
prosolis
71f7050f41 Add personalization, outbound feeds, and PWA/push to the web UI
A multi-session build turning Pete's read-only web UI into something people
return to. Five phases, signed-in features keyed off the OIDC subject; anonymous
visitors keep the reverse-chron feed and localStorage-only state.

Phase 1 — per-user read + bookmark state: user_story_state table +
storage/userstate.go; auth-gated /api/read, /api/bookmark, /api/state and a
/bookmarks page; reader.js syncs state server-side for signed-in users. Also
hides the Matrix-posting UI when posting.enabled=false (web-only mode).

Phase 2 — outbound feeds: storage.ListForFeed + web/feed.go hand-build RSS 2.0
(content:encoded) and JSON Feed 1.1 (no new dep); /feed.xml, /feed.json and
per-channel variants; <link rel=alternate> discovery tags.

Phase 3 — "For you" + related: storage/rank.go scores recent unread candidates
by channel/source affinity + recency decay; RelatedStories via FTS5. ForYou rail
+ /for-you page; public /api/related feeds the reader's "You might also like".

Phase 4 — source-health dashboard: source_health table + storage/sourcehealth.go
(RecordPollResult, ListSourceHealth, SourceContentStats), written by the poller;
admin-gated /status page behind web.admin_subs.

Phase 5 — PWA + offline reader + Web Push: root-scoped manifest.webmanifest and
sw.js (app-shell precache, /api/article runtime cache for offline reading,
offline fallback, push/notificationclick handlers); PNG icons from pete.avif;
pwa.js registers the SW and drives a notifications toggle. Web Push adds
webpush-go, a [web.push] config block (pete -genvapid mints VAPID keys), a
push_subscriptions table, auth-gated subscribe/unsubscribe endpoints, and a
digest sender that pings each subscriber "N new stories" past their watermark,
honoring disabled-sources and pruning gone endpoints.

Tests added beside each new storage/web file; go test ./... and go vet clean.
2026-07-07 00:07:19 -07:00
prosolis
55aa167151 Add feed/reader mode with full-article capture at ingest
Reader mode presents the stories on a page one at a time in a focused
overlay, marking each read as it's shown. Left/right arrows (or the header
book button / `f`) page through them; read stories dim on the grid. Read
state is device-local in localStorage.

Backing this required actually capturing article bodies, which Pete wasn't
doing — it kept only the RSS <description> lede and discarded content:encoded:

- stories.content column (idempotent migration; old rows fall back to lede)
- parser keeps content:encoded as paragraph-preserving text
- article fetch already done for paywall detection now also returns its body,
  so ingest stores the richer of feed-content vs scraped body with no extra
  request (prefers the archive snapshot body for paywalled stories)
- GET /api/article?id= serves the stored text; card queries now select id and
  expose it as data-id for the reader

Tests cover content extraction, the storage round-trip, and the article
endpoint + card rendering end to end.
2026-07-06 22:46:14 -07:00
50 changed files with 4529 additions and 51 deletions

View File

@@ -37,6 +37,12 @@ listen_addr = ":8080"
site_title = "Pete"
base_url = "https://news.parodia.dev"
# OIDC subjects allowed to view the owner-facing source-health dashboard at
# /status (per-feed poll status, failures, content stats). Requires web.auth
# below. Empty = /status returns 404 for everyone. Find a user's subject in the
# server logs ("auth: user signed in" sub=...) after they sign in once.
admin_subs = []
# Optional OIDC sign-in (Authentik). When enabled, signed-in users get their
# preferences (hidden feeds, weather location, toggles) stored server-side keyed
# by their OIDC subject and synced across devices. Anonymous visitors keep using
@@ -53,6 +59,24 @@ redirect_url = "https://news.parodia.dev/auth/callback"
# HMAC key that signs the session cookie. Generate with: openssl rand -hex 32
session_secret = "${PETE_SESSION_SECRET}"
# Optional Web Push digests. When enabled, signed-in users can opt in (from the
# feed-settings panel) to a periodic "N new stories" notification, delivered via
# a service worker so it also works when the site is installed as a PWA. Push is
# signed-in only, so it needs web.auth above; it does nothing otherwise.
# Generate the VAPID keypair once with: pete -genvapid
# then paste the two keys here (the private key is a secret — treat it like a
# password). subject identifies you to the push service (a mailto: or https: URL).
[web.push]
enabled = false
vapid_public_key = "${PETE_VAPID_PUBLIC_KEY}"
vapid_private_key = "${PETE_VAPID_PRIVATE_KEY}"
subject = "mailto:admin@parodia.dev"
# How often the sender wakes to look for new stories per subscriber (minutes).
interval_minutes = 360
# Smallest number of new (non-hidden) stories that triggers a digest, so a lone
# item doesn't ping everyone.
min_stories = 3
# Every enabled source MUST set direct_route to a key from [matrix.channels] above.
# There is no automatic classification — Pete posts each story to its configured channel.
# Optional: language = "en" drops feed items whose per-item <language> tag is

10
go.mod
View File

@@ -5,7 +5,12 @@ go 1.25.0
require (
github.com/BurntSushi/toml v1.6.0
github.com/PuerkitoBio/goquery v1.12.0
github.com/SherClockHolmes/webpush-go v1.4.0
github.com/coreos/go-oidc/v3 v3.19.0
github.com/mmcdole/gofeed v1.3.0
go.mau.fi/util v0.9.9
golang.org/x/image v0.41.0
golang.org/x/oauth2 v0.36.0
maunium.net/go/mautrix v0.28.0
modernc.org/sqlite v1.50.1
)
@@ -13,9 +18,9 @@ require (
require (
filippo.io/edwards25519 v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/coreos/go-oidc/v3 v3.19.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/golang-jwt/jwt/v5 v5.2.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
@@ -32,12 +37,9 @@ require (
github.com/tidwall/match v1.2.0 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/sjson v1.2.5 // indirect
go.mau.fi/util v0.9.9 // indirect
golang.org/x/crypto v0.52.0 // indirect
golang.org/x/exp v0.0.0-20260508232706-74f9aab9d74a // indirect
golang.org/x/image v0.41.0 // indirect
golang.org/x/net v0.55.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sys v0.45.0 // indirect
golang.org/x/text v0.37.0 // indirect
modernc.org/libc v1.72.3 // indirect

4
go.sum
View File

@@ -6,6 +6,8 @@ github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7Oputl
github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU=
github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo=
github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ=
github.com/SherClockHolmes/webpush-go v1.4.0 h1:ocnzNKWN23T9nvHi6IfyrQjkIc0oJWv1B1pULsf9i3s=
github.com/SherClockHolmes/webpush-go v1.4.0/go.mod h1:XSq8pKX11vNV8MJEMwjrlTkxhAj1zKfxmyhdV7Pd6UA=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/coreos/go-oidc/v3 v3.19.0 h1:F/xyOi3x1UnG1U27YVnM1N6bHiL1K2upi6U/0qr8r+I=
@@ -17,6 +19,8 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=

View File

@@ -27,6 +27,30 @@ type WebConfig struct {
SiteTitle string `toml:"site_title"` // display name in the header
BaseURL string `toml:"base_url"` // public URL (used in metadata only)
Auth AuthConfig `toml:"auth"` // optional OIDC sign-in (Authentik)
Push PushConfig `toml:"push"` // optional Web Push digests (VAPID)
// AdminSubs is the allowlist of OIDC subjects allowed to view the
// owner-facing source-health dashboard at /status. Empty means the page is
// inaccessible to everyone (returns 404). Requires auth to be enabled.
AdminSubs []string `toml:"admin_subs"`
}
// PushConfig wires the Web Push digest sender. Push is signed-in only (it keys
// off the OIDC subject) so it does nothing unless auth is also enabled. Generate
// a VAPID keypair once with `pete -genvapid` and paste the two keys here.
type PushConfig struct {
Enabled bool `toml:"enabled"`
VAPIDPublicKey string `toml:"vapid_public_key"`
VAPIDPrivateKey string `toml:"vapid_private_key"`
// Subject identifies the sender to the push service; a mailto: or https: URL
// per RFC 8292. Defaults to mailto:admin@<base_url host> is not attempted —
// set it explicitly.
Subject string `toml:"subject"`
// IntervalMinutes is how often the digest sender wakes to look for new
// stories per subscriber. Defaults to 360 (6h).
IntervalMinutes int `toml:"interval_minutes"`
// MinStories is the smallest number of new stories that triggers a digest
// for a subscriber, so they aren't pinged for a single item. Defaults to 3.
MinStories int `toml:"min_stories"`
}
// AuthConfig wires Pete's web UI to an OIDC provider (our Authentik instance).
@@ -167,6 +191,19 @@ func (c *Config) validate() error {
}
}
if c.Web.Push.Enabled {
if !c.Web.Auth.Enabled {
return fmt.Errorf("web.push requires web.auth to be enabled (push is signed-in only)")
}
p := c.Web.Push
if p.VAPIDPublicKey == "" || p.VAPIDPrivateKey == "" {
return fmt.Errorf("web.push requires vapid_public_key and vapid_private_key (generate with: pete -genvapid)")
}
if p.Subject == "" {
return fmt.Errorf("web.push.subject is required (a mailto: or https: URL identifying the sender)")
}
}
for i, s := range c.Sources {
if s.Name == "" {
return fmt.Errorf("sources[%d].name is required", i)
@@ -232,6 +269,12 @@ func (c *Config) applyDefaults() {
if c.Web.SiteTitle == "" {
c.Web.SiteTitle = "Pete"
}
if c.Web.Push.IntervalMinutes == 0 {
c.Web.Push.IntervalMinutes = 360
}
if c.Web.Push.MinStories == 0 {
c.Web.Push.MinStories = 3
}
for i := range c.Sources {
if c.Sources[i].PollIntervalMinutes == 0 {
c.Sources[i].PollIntervalMinutes = 20

View File

@@ -211,6 +211,43 @@ feed_url = "https://e.com/rss"
tier = 1
poll_interval_minutes = 20
enabled = true
`},
{"push enabled without auth", `
[matrix]
homeserver = "https://h"
user_id = "@p:h"
password = "pw"
pickle_key = "test_pickle_key_1234"
[matrix.channels]
tech = "!t:e"
[storage]
db_path = "/tmp/t.db"
[web.push]
enabled = true
vapid_public_key = "pub"
vapid_private_key = "priv"
subject = "mailto:a@b.c"
`},
{"push enabled missing keys", `
[matrix]
homeserver = "https://h"
user_id = "@p:h"
password = "pw"
pickle_key = "test_pickle_key_1234"
[matrix.channels]
tech = "!t:e"
[storage]
db_path = "/tmp/t.db"
[web.auth]
enabled = true
issuer = "https://i"
client_id = "id"
client_secret = "secret"
redirect_url = "https://r/cb"
session_secret = "session_secret_16chars_long"
[web.push]
enabled = true
subject = "mailto:a@b.c"
`},
}

View File

@@ -65,6 +65,7 @@ const googlebotUA = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.
// ArticleMeta is what we can learn from fetching an article page directly.
type ArticleMeta struct {
ImageURL string // og:image or twitter:image, absolute URL
BodyText string // extracted visible article body text (capped at MaxBodyChars)
BodyChars int // length of extracted visible body text
Fetched bool // true if we got an HTTP 200 with HTML
FetchError bool // true if the fetch failed at the network/HTTP layer
@@ -192,7 +193,10 @@ func fetchArticleMetaOnce(articleURL, ua, referer string) ArticleMeta {
meta.Fetched = true
meta.ImageURL = extractOGImage(doc, articleURL)
meta.BodyChars = extractBodyChars(doc)
// One body extraction serves both purposes: the text feeds reader mode and
// its length is the paywall/thin-body signal.
meta.BodyText = extractBodyText(doc)
meta.BodyChars = len(meta.BodyText)
meta.Paywalled = detectPaywall(doc)
meta.SubscriberOnly = detectSubscriberOnly(articleURL, doc)
return meta
@@ -410,10 +414,7 @@ func extractBodyText(doc *goquery.Document) string {
out = alt
}
}
if len(out) > MaxBodyChars {
out = out[:MaxBodyChars]
}
return out
return truncateUTF8(out, MaxBodyChars)
}
// bodyContainerSelectors is the list of selectors we try when <article>/<main>

View File

@@ -10,6 +10,7 @@ import (
"strconv"
"strings"
"time"
"unicode/utf8"
"github.com/mmcdole/gofeed"
ext "github.com/mmcdole/gofeed/extensions"
@@ -31,6 +32,7 @@ type FeedItem struct {
GUID string
Headline string
Lede string
Content string // full article text from content:encoded, block structure preserved; "" when the feed ships only a snippet
ImageURL string
ArticleURL string
Source string
@@ -43,8 +45,19 @@ var (
htmlTagRe = regexp.MustCompile(`<[^>]*>`)
wsRe = regexp.MustCompile(`\s+`)
imgSrcRe = regexp.MustCompile(`(?i)<img[^>]+src=["']([^"']+)["']`)
// Used by extractContentText to preserve paragraph structure when turning
// content:encoded HTML into plain text.
blockCloseRe = regexp.MustCompile(`(?i)</(p|div|li|h[1-6]|blockquote|article|section|figure|figcaption|ul|ol|table|tr|pre)>`)
brRe = regexp.MustCompile(`(?i)<br\s*/?>`)
hspaceRe = regexp.MustCompile(`[ \t\f\r]+`)
manyNewlineRe = regexp.MustCompile(`\n{3,}`)
)
// maxContentChars bounds the stored article text. Generous enough for any real
// article while keeping row size and the reader payload sane.
const maxContentChars = 20000
// FetchFeed fetches and parses an RSS/Atom feed, returning items. ua overrides
// the User-Agent header; pass "" to use Pete's honest default bot UA.
func FetchFeed(feedURL, ua string) ([]FeedItem, error) {
@@ -79,6 +92,7 @@ func FetchFeed(feedURL, ua string) ([]FeedItem, error) {
GUID: itemGUID(item),
Headline: strings.TrimSpace(item.Title),
Lede: extractLede(item.Description),
Content: extractContentText(item.Content),
ImageURL: NormalizeImageURL(extractImageURL(item)),
ArticleURL: strings.TrimSpace(item.Link),
Language: itemLanguage(item),
@@ -142,6 +156,47 @@ func extractLede(desc string) string {
return strings.TrimSpace(text)
}
// extractContentText converts a feed's content:encoded HTML into plain text
// with paragraph breaks preserved, for reader mode. Block-level boundaries
// (</p>, </div>, <br>, list items, headings…) become newlines before the
// remaining tags are stripped, so the reader can re-wrap the text into
// paragraphs. Returns "" when the feed carried no content:encoded.
func extractContentText(raw string) string {
if strings.TrimSpace(raw) == "" {
return ""
}
s := brRe.ReplaceAllString(raw, "\n")
s = blockCloseRe.ReplaceAllString(s, "\n\n")
s = htmlTagRe.ReplaceAllString(s, "")
s = html.UnescapeString(s)
lines := strings.Split(s, "\n")
for i, ln := range lines {
lines[i] = strings.TrimSpace(hspaceRe.ReplaceAllString(ln, " "))
}
s = manyNewlineRe.ReplaceAllString(strings.Join(lines, "\n"), "\n\n")
s = strings.TrimSpace(s)
return truncateUTF8(s, maxContentChars)
}
// truncateUTF8 caps s at max bytes without splitting a multi-byte rune. A plain
// byte slice can leave a partial trailing rune, which is invalid UTF-8; that
// corrupt byte then breaks the RSS content:encoded XML (Go's xml encoder writes
// it raw) and shows as a replacement char in JSON. Trim back to a rune boundary.
func truncateUTF8(s string, max int) string {
if len(s) <= max {
return s
}
s = s[:max]
for len(s) > 0 {
if r, size := utf8.DecodeLastRuneInString(s); r == utf8.RuneError && size <= 1 {
s = s[:len(s)-1] // strip one byte of an incomplete trailing rune
continue
}
break
}
return s
}
// extractImageURL tries to find an image URL from feed item metadata.
// Order: media:content/thumbnail, enclosures, item.Image, then <img> tags
// scraped from content:encoded / description (catches feeds like Ars that

View File

@@ -1,6 +1,9 @@
package ingestion
import "testing"
import (
"strings"
"testing"
)
func TestExtractLede(t *testing.T) {
tests := []struct {
@@ -69,3 +72,26 @@ func TestExtractLede(t *testing.T) {
})
}
}
func TestExtractContentText(t *testing.T) {
if got := extractContentText(""); got != "" {
t.Errorf("empty input = %q, want empty", got)
}
if got := extractContentText(" \n "); got != "" {
t.Errorf("whitespace-only input = %q, want empty", got)
}
in := `<p>First paragraph with <a href="/x">a link</a> &amp; entity.</p>` +
`<p>Second line one.<br>line two.</p><ul><li>item</li></ul>`
got := extractContentText(in)
want := "First paragraph with a link & entity.\n\nSecond line one.\nline two.\n\nitem"
if got != want {
t.Errorf("extractContentText:\n got %q\n want %q", got, want)
}
// Output is capped so a runaway body can't bloat the row.
long := "<p>" + strings.Repeat("x", maxContentChars+500) + "</p>"
if n := len(extractContentText(long)); n > maxContentChars {
t.Errorf("capped length = %d, want <= %d", n, maxContentChars)
}
}

View File

@@ -97,8 +97,16 @@ func (p *Poller) pollOnce(ctx context.Context, src config.SourceConfig) {
func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) error {
items, err := FetchFeed(src.FeedURL, src.UserAgent)
if err != nil {
// Persist the failure for the owner-facing source-health dashboard. The
// in-memory consecutiveFailures in pollSource drives log warnings; this
// row is the authoritative record the dashboard reads.
storage.RecordPollResult(src.Name, false, 0, err)
return err
}
// A successful fetch: record health now (before per-item processing) with
// the number of items the feed returned, so the dashboard reflects feed
// reachability even if individual items are later dropped as duplicates.
storage.RecordPollResult(src.Name, true, len(items), nil)
newCount := 0
for i := range items {
@@ -160,6 +168,10 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
"guid", items[i].GUID, "url", originalURL, "source", src.Name)
continue
}
// bodyText is the article text we may store for reader mode. It starts
// as the body scraped during the fetch above and is upgraded to the
// snapshot's body if we end up swapping in an archive snapshot.
bodyText := meta.BodyText
gated := meta.Gated() || (meta.Fetched && meta.BodyChars < PaywallBodyThreshold)
// paywalled tracks whether the link the user will click is still
// gated — i.e. gating was detected AND no archive workaround worked.
@@ -175,6 +187,9 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
if items[i].ImageURL == "" && snapMeta.ImageURL != "" {
items[i].ImageURL = snapMeta.ImageURL
}
if snapMeta.BodyText != "" {
bodyText = snapMeta.BodyText
}
workedAround = true
slog.Info("paywall detected, using archive snapshot",
"guid", items[i].GUID, "original", originalURL,
@@ -209,10 +224,18 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
if publishedAt > seenAt {
publishedAt = seenAt
}
// Reader-mode body: prefer whichever source gave us more complete text.
// Feeds that ship full content:encoded usually win; feeds that only
// syndicate a snippet fall back to the scraped article body.
content := items[i].Content
if len(bodyText) > len(content) {
content = bodyText
}
if err := storage.InsertStory(&storage.Story{
GUID: items[i].GUID,
Headline: items[i].Headline,
Lede: items[i].Lede,
Content: content,
ImageURL: items[i].ImageURL,
ArticleURL: items[i].ArticleURL,
URLCanonical: canonical,

View File

@@ -81,6 +81,10 @@ func runMigrations(d *sql.DB) error {
addColumnIfMissing(d, "stories", "url_canonical", "TEXT")
addColumnIfMissing(d, "stories", "headline_norm", "TEXT")
addColumnIfMissing(d, "stories", "paywalled", "INTEGER NOT NULL DEFAULT 0")
// content holds the full article text (feed content:encoded when present,
// else the body scraped during paywall detection) for reader mode. Stories
// ingested before this column existed simply have NULL and fall back to lede.
addColumnIfMissing(d, "stories", "content", "TEXT")
addColumnIfMissing(d, "stories", "published_at", "INTEGER")
addColumnIfMissing(d, "post_log", "url_canonical", "TEXT")
addColumnIfMissing(d, "post_log", "forced", "INTEGER NOT NULL DEFAULT 0")
@@ -116,6 +120,11 @@ func RunMaintenance() {
exec("prune old reactions",
`DELETE FROM reactions WHERE reacted_at < ?`, storyCutoff)
// Drop per-user read/bookmark rows whose story has been pruned above, so the
// table can't accumulate dangling references as stories age out.
exec("prune orphan user_story_state",
`DELETE FROM user_story_state WHERE story_id NOT IN (SELECT id FROM stories)`)
// Daily unique tokens are only useful for the recent window; their salts are
// long gone. page_views is kept forever (tiny aggregate, all-time totals).
exec("prune old daily_visitors",

115
internal/storage/push.go Normal file
View File

@@ -0,0 +1,115 @@
package storage
import "fmt"
// PushSubscription is one browser/device endpoint a signed-in user has opted in
// for Web Push digests. See the push_subscriptions schema for the field roles.
type PushSubscription struct {
Endpoint string
UserSub string
P256dh string
Auth string
CreatedAt int64
LastNotifiedAt int64
}
// AddPushSubscription records (or refreshes) a push endpoint for a user. The
// endpoint is the primary key, so a re-subscribe from the same browser updates
// the keys and resets the digest watermark to now — the user shouldn't be
// paged for everything published before they opted in.
func AddPushSubscription(sub, endpoint, p256dh, auth string) error {
now := nowUnix()
_, err := Get().Exec(`
INSERT INTO push_subscriptions (endpoint, user_sub, p256dh, auth, created_at, last_notified_at)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(endpoint) DO UPDATE SET
user_sub = excluded.user_sub,
p256dh = excluded.p256dh,
auth = excluded.auth,
last_notified_at = excluded.last_notified_at`,
endpoint, sub, p256dh, auth, now, now)
if err != nil {
return fmt.Errorf("add push subscription: %w", err)
}
return nil
}
// RemovePushSubscription drops one endpoint regardless of owner. Reserved for
// the digest sender's prune path, where a push service has reported the endpoint
// gone (404/410) and there's no caller identity to scope by. User-initiated
// opt-outs must use RemovePushSubscriptionForUser.
func RemovePushSubscription(endpoint string) error {
_, err := Get().Exec(`DELETE FROM push_subscriptions WHERE endpoint = ?`, endpoint)
if err != nil {
return fmt.Errorf("remove push subscription: %w", err)
}
return nil
}
// RemovePushSubscriptionForUser drops an endpoint only if it belongs to sub, so
// a signed-in user can't unsubscribe another account's device by presenting its
// endpoint string. A no-op (no matching row) is not an error.
func RemovePushSubscriptionForUser(sub, endpoint string) error {
_, err := Get().Exec(
`DELETE FROM push_subscriptions WHERE endpoint = ? AND user_sub = ?`, endpoint, sub)
if err != nil {
return fmt.Errorf("remove push subscription: %w", err)
}
return nil
}
// ListPushSubscriptions returns every stored subscription, for the digest sender.
func ListPushSubscriptions() ([]PushSubscription, error) {
rows, err := Get().Query(
`SELECT endpoint, user_sub, p256dh, auth, created_at, last_notified_at
FROM push_subscriptions`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []PushSubscription
for rows.Next() {
var p PushSubscription
if err := rows.Scan(&p.Endpoint, &p.UserSub, &p.P256dh, &p.Auth, &p.CreatedAt, &p.LastNotifiedAt); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// TouchPushSubscription advances an endpoint's digest watermark so its next
// digest only considers stories seen after ts.
func TouchPushSubscription(endpoint string, ts int64) error {
_, err := Get().Exec(
`UPDATE push_subscriptions SET last_notified_at = ? WHERE endpoint = ?`, ts, endpoint)
if err != nil {
return fmt.Errorf("touch push subscription: %w", err)
}
return nil
}
// NewClassifiedSince returns classified stories first seen after sinceUnix,
// newest first, capped at limit. The digest sender uses it to count and preview
// what's new for a subscriber; it carries just the fields a digest needs.
func NewClassifiedSince(sinceUnix int64, limit int) ([]Story, error) {
rows, err := Get().Query(
`SELECT id, headline, source, channel, seen_at
FROM stories
WHERE classified = 1 AND channel NOT IN ('_discarded', '_duplicate') AND seen_at > ?
ORDER BY seen_at DESC
LIMIT ?`, sinceUnix, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.ID, &s.Headline, &s.Source, &s.Channel, &s.SeenAt); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}

View File

@@ -0,0 +1,105 @@
package storage
import "testing"
func TestPushSubscriptionLifecycle(t *testing.T) {
setupTestDB(t)
if err := AddPushSubscription("sub-1", "https://push.example/ep-a", "p256-a", "auth-a"); err != nil {
t.Fatal(err)
}
// A second endpoint for the same user (e.g. a second device).
if err := AddPushSubscription("sub-1", "https://push.example/ep-b", "p256-b", "auth-b"); err != nil {
t.Fatal(err)
}
// A different user.
if err := AddPushSubscription("sub-2", "https://push.example/ep-c", "p256-c", "auth-c"); err != nil {
t.Fatal(err)
}
subs, err := ListPushSubscriptions()
if err != nil {
t.Fatal(err)
}
if len(subs) != 3 {
t.Fatalf("got %d subscriptions, want 3", len(subs))
}
// Re-subscribing the same endpoint updates keys in place, not a new row.
if err := AddPushSubscription("sub-1", "https://push.example/ep-a", "p256-a2", "auth-a2"); err != nil {
t.Fatal(err)
}
subs, _ = ListPushSubscriptions()
if len(subs) != 3 {
t.Fatalf("after re-subscribe got %d rows, want 3 (upsert, not insert)", len(subs))
}
var epA PushSubscription
for _, s := range subs {
if s.Endpoint == "https://push.example/ep-a" {
epA = s
}
}
if epA.P256dh != "p256-a2" || epA.Auth != "auth-a2" {
t.Errorf("re-subscribe did not refresh keys: %+v", epA)
}
if err := RemovePushSubscription("https://push.example/ep-a"); err != nil {
t.Fatal(err)
}
subs, _ = ListPushSubscriptions()
if len(subs) != 2 {
t.Fatalf("after remove got %d rows, want 2", len(subs))
}
}
func TestTouchPushSubscriptionAdvancesWatermark(t *testing.T) {
setupTestDB(t)
if err := AddPushSubscription("sub-1", "https://push.example/ep", "p", "a"); err != nil {
t.Fatal(err)
}
subs, _ := ListPushSubscriptions()
orig := subs[0].LastNotifiedAt
want := orig + 5000
if err := TouchPushSubscription("https://push.example/ep", want); err != nil {
t.Fatal(err)
}
subs, _ = ListPushSubscriptions()
if subs[0].LastNotifiedAt != want {
t.Errorf("watermark = %d, want %d", subs[0].LastNotifiedAt, want)
}
}
func TestNewClassifiedSince(t *testing.T) {
setupTestDB(t)
now := nowUnix()
insert := func(guid, headline, source, channel string, classified bool, seenAt int64) {
if err := InsertStory(&Story{
GUID: guid, Headline: headline, ArticleURL: "https://x/" + guid,
Source: source, Channel: channel, Classified: classified, SeenAt: seenAt,
}); err != nil {
t.Fatal(err)
}
}
insert("old", "Old news", "Feed A", "tech", true, now-1000)
insert("fresh1", "Fresh one", "Feed A", "tech", true, now-100)
insert("fresh2", "Fresh two", "Feed B", "gaming", true, now-50)
insert("unclassified", "Pending", "Feed A", "", false, now-10)
insert("discarded", "Junk", "Feed A", "_discarded", true, now-10)
got, err := NewClassifiedSince(now-500, 10)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 {
t.Fatalf("got %d stories, want 2 (fresh classified, non-discarded, after watermark)", len(got))
}
// Newest first.
if got[0].GUID != "" && got[0].Headline != "Fresh two" {
t.Errorf("first result = %q, want newest 'Fresh two'", got[0].Headline)
}
if got[0].Source != "Feed B" || got[1].Source != "Feed A" {
t.Errorf("unexpected order/sources: %q then %q", got[0].Source, got[1].Source)
}
}

View File

@@ -38,13 +38,33 @@ func InsertStory(s *Story) error {
publishedAt = s.PublishedAt
}
_, err := Get().Exec(
`INSERT INTO stories (guid, headline, lede, image_url, article_url, url_canonical, headline_norm, source, platforms, channel, classified, paywalled, seen_at, published_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
s.GUID, s.Headline, s.Lede, s.ImageURL, s.ArticleURL, nullIfEmpty(s.URLCanonical), nullIfEmpty(s.HeadlineNorm), s.Source, s.Platforms, s.Channel, classified, paywalled, s.SeenAt, publishedAt,
`INSERT INTO stories (guid, headline, lede, content, image_url, article_url, url_canonical, headline_norm, source, platforms, channel, classified, paywalled, seen_at, published_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
s.GUID, s.Headline, s.Lede, nullIfEmpty(s.Content), s.ImageURL, s.ArticleURL, nullIfEmpty(s.URLCanonical), nullIfEmpty(s.HeadlineNorm), s.Source, s.Platforms, s.Channel, classified, paywalled, s.SeenAt, publishedAt,
)
return err
}
// GetStoryReaderText returns the stored full article text and lede for a single
// story, for reader mode. found is false when no story has that id. The query
// mirrors the visible-story filter (classified, non-sentinel channel) so the
// public /api/article endpoint can't be enumerated to pull the captured bodies
// of discarded or not-yet-classified stories that never surface in the UI.
func GetStoryReaderText(id int64) (content, lede string, found bool, err error) {
var c sql.NullString
var l sql.NullString
row := Get().QueryRow(
`SELECT content, lede FROM stories
WHERE id = ? AND classified = 1 AND channel NOT IN ('_discarded', '_duplicate')`, id)
switch err = row.Scan(&c, &l); {
case errors.Is(err, sql.ErrNoRows):
return "", "", false, nil
case err != nil:
return "", "", false, err
}
return c.String, l.String, true, nil
}
// nullIfEmpty returns a SQL NULL for empty strings so partial unique indexes
// (WHERE col IS NOT NULL) can hold multiple "unknown" rows without conflict.
func nullIfEmpty(s string) any {
@@ -236,7 +256,7 @@ func GetNewestPostableStoryByChannel(channel string) (*Story, error) {
// channels (_discarded, _duplicate) are excluded.
func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error) {
rows, err := Get().Query(
`SELECT s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories s
WHERE s.classified = 1
@@ -250,7 +270,7 @@ func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error)
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err
}
out = append(out, s)
@@ -262,7 +282,7 @@ func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error)
// channels, newest first. Sentinel channels are excluded.
func ListAllClassified(limit, offset int) ([]Story, error) {
rows, err := Get().Query(
`SELECT s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories s
WHERE s.classified = 1
@@ -277,7 +297,7 @@ func ListAllClassified(limit, offset int) ([]Story, error) {
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err
}
out = append(out, s)
@@ -285,11 +305,60 @@ func ListAllClassified(limit, offset int) ([]Story, error) {
return out, rows.Err()
}
// ListForFeed returns up to `limit` classified stories for the outbound RSS and
// JSON feeds, newest first. Unlike the web list queries it also selects the full
// `content` and `published_at`, which the feeds need for content:encoded bodies
// and real pubDates. channel == "" spans all real channels; otherwise it scopes
// to that one channel. Sentinel channels are always excluded.
func ListForFeed(channel string, limit int) ([]Story, error) {
const cols = `s.id, s.guid, s.headline, s.lede, s.content, s.image_url, s.article_url, s.url_canonical, s.source, s.channel, s.seen_at, s.published_at`
var (
rows *sql.Rows
err error
)
if channel == "" {
rows, err = Get().Query(
`SELECT `+cols+`
FROM stories s
WHERE s.classified = 1
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')
ORDER BY COALESCE(s.published_at, s.seen_at) DESC
LIMIT ?`, limit)
} else {
rows, err = Get().Query(
`SELECT `+cols+`
FROM stories s
WHERE s.classified = 1
AND s.channel = ?
ORDER BY COALESCE(s.published_at, s.seen_at) DESC
LIMIT ?`, channel, limit)
}
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
var content, canonical sql.NullString
var published sql.NullInt64
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &content, &s.ImageURL, &s.ArticleURL, &canonical, &s.Source, &s.Channel, &s.SeenAt, &published); err != nil {
return nil, err
}
s.Content = content.String
s.URLCanonical = canonical.String
s.PublishedAt = published.Int64
out = append(out, s)
}
return out, rows.Err()
}
// ListRecentlyPosted returns up to `limit` stories that have been posted to
// Matrix, ordered by post time (newest first). Posted is always true.
func ListRecentlyPosted(limit int) ([]Story, error) {
rows, err := Get().Query(
`SELECT s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at
FROM stories s
JOIN post_log p ON p.guid = s.guid
GROUP BY s.guid
@@ -302,7 +371,7 @@ func ListRecentlyPosted(limit int) ([]Story, error) {
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt); err != nil {
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt); err != nil {
return nil, err
}
s.Posted = true

275
internal/storage/rank.go Normal file
View File

@@ -0,0 +1,275 @@
package storage
import (
"database/sql"
"errors"
"math"
"sort"
"strings"
)
// Ranking weights for the "For you" feed. Bookmarks are a stronger signal of
// interest than a plain read, and channel affinity outweighs source affinity
// (a reader follows topics more than outlets). Recency keeps the feed fresh so
// a heavily-read channel doesn't surface week-old stories over today's news.
const (
affinityReadWeight = 1.0
affinityBookmarkWeight = 3.0
forYouChannelWeight = 1.0
forYouSourceWeight = 0.7
forYouRecencyWeight = 0.9
// forYouCandidatePool bounds how many recent unread stories we score in Go.
forYouCandidatePool = 400
// forYouRecencyHalfLifeHours sets how fast the recency term decays.
forYouRecencyHalfLifeHours = 48.0
)
// userAffinity builds normalized channel and source affinity scores (each 0..1)
// for a signed-in user from their read + bookmark history. Bookmarks count for
// more than plain reads. total is the number of signal-bearing rows; when it is
// zero the user has no history yet and both maps are empty.
func userAffinity(sub string) (channel, source map[string]float64, total int, err error) {
channel = make(map[string]float64)
source = make(map[string]float64)
if sub == "" {
return channel, source, 0, nil
}
rows, err := Get().Query(
`SELECT s.channel, s.source, u.read_at, u.bookmarked_at
FROM user_story_state u
JOIN stories s ON s.id = u.story_id
WHERE u.user_sub = ?
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')`, sub)
if err != nil {
return nil, nil, 0, err
}
defer rows.Close()
for rows.Next() {
var ch, src string
var readAt, markAt sql.NullInt64
if err := rows.Scan(&ch, &src, &readAt, &markAt); err != nil {
return nil, nil, 0, err
}
w := 0.0
if readAt.Valid {
w += affinityReadWeight
}
if markAt.Valid {
w += affinityBookmarkWeight
}
if w == 0 {
continue
}
channel[ch] += w
source[src] += w
total++
}
if err := rows.Err(); err != nil {
return nil, nil, 0, err
}
normalizeMax(channel)
normalizeMax(source)
return channel, source, total, nil
}
// normalizeMax scales a map's values so the largest becomes 1.0, leaving an
// all-zero (or empty) map untouched.
func normalizeMax(m map[string]float64) {
var max float64
for _, v := range m {
if v > max {
max = v
}
}
if max == 0 {
return
}
for k := range m {
m[k] /= max
}
}
// ForYou returns a personalized ranking of recent, unread stories for a
// signed-in user, blending channel affinity, source affinity, and recency. It
// returns (nil, nil) when the user has no read/bookmark history yet, so callers
// can fall back to the plain latest feed.
func ForYou(sub string, limit int) ([]Story, error) {
if limit <= 0 {
return nil, nil
}
chAff, srcAff, total, err := userAffinity(sub)
if err != nil {
return nil, err
}
if total == 0 {
return nil, nil
}
rows, err := Get().Query(
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at, COALESCE(s.published_at, s.seen_at),
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories s
WHERE s.classified = 1
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')
AND s.id NOT IN (
SELECT story_id FROM user_story_state
WHERE user_sub = ? AND read_at IS NOT NULL)
ORDER BY COALESCE(s.published_at, s.seen_at) DESC
LIMIT ?`, sub, forYouCandidatePool)
if err != nil {
return nil, err
}
defer rows.Close()
now := float64(nowUnix())
type scored struct {
s Story
score float64
}
var cands []scored
for rows.Next() {
var s Story
var effectiveAt int64
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &effectiveAt, &s.Posted); err != nil {
return nil, err
}
ageHours := (now - float64(effectiveAt)) / 3600.0
if ageHours < 0 {
ageHours = 0
}
recency := math.Exp(-ageHours / forYouRecencyHalfLifeHours)
score := forYouChannelWeight*chAff[s.Channel] +
forYouSourceWeight*srcAff[s.Source] +
forYouRecencyWeight*recency
cands = append(cands, scored{s, score})
}
if err := rows.Err(); err != nil {
return nil, err
}
// Candidates arrive newest-first, and a stable sort keeps that order for
// equal scores, so ties break toward the more recent story.
sort.SliceStable(cands, func(i, j int) bool { return cands[i].score > cands[j].score })
if len(cands) > limit {
cands = cands[:limit]
}
out := make([]Story, 0, len(cands))
for _, c := range cands {
out = append(out, c.s)
}
return out, nil
}
// relatedRecencyWindowSeconds bounds "related" results to roughly the same
// window the story feed lives in (stories are pruned at 30 days anyway).
const relatedRecencyWindowSeconds = 30 * 86400
// relatedStopwords are high-frequency, low-signal tokens dropped when building
// the "related" FTS query so matches key off the story's actual subject matter.
var relatedStopwords = map[string]bool{
"the": true, "and": true, "for": true, "with": true, "that": true,
"this": true, "from": true, "has": true, "have": true, "are": true,
"was": true, "were": true, "will": true, "its": true, "into": true,
"out": true, "how": true, "why": true, "what": true, "who": true,
"you": true, "your": true, "not": true, "but": true, "all": true,
"new": true, "now": true, "more": true, "after": true, "over": true,
"about": true, "says": true, "said": true, "of": true, "to": true,
"in": true, "is": true, "on": true, "at": true, "by": true, "as": true,
"an": true, "be": true, "it": true, "or": true, "if": true, "we": true,
"he": true, "do": true, "up": true, "so": true, "no": true, "my": true,
}
// RelatedStories returns classified stories textually similar to the given
// story, ranked by FTS5 relevance and excluding the story itself. The match
// query is an OR of significant tokens from the seed story's headline and lede,
// so it surfaces stories that share subject matter rather than requiring every
// term (which is what SearchStories' implicit-AND query would demand).
func RelatedStories(storyID int64, limit int) ([]Story, error) {
if storyID <= 0 || limit <= 0 {
return nil, nil
}
var headline, lede sql.NullString
err := Get().QueryRow(`SELECT headline, lede FROM stories WHERE id = ?`, storyID).Scan(&headline, &lede)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return nil, nil
}
return nil, err
}
fts := buildRelatedFTSQuery(headline.String + " " + lede.String)
if fts == "" {
return nil, nil
}
cutoff := nowUnix() - relatedRecencyWindowSeconds
rows, err := Get().Query(
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM stories_fts f
JOIN stories s ON s.id = f.rowid
WHERE f.stories_fts MATCH ?
AND s.id != ?
AND s.classified = 1
AND s.channel IS NOT NULL
AND s.channel NOT IN ('_discarded', '_duplicate')
AND s.seen_at >= ?
ORDER BY bm25(stories_fts) ASC, s.seen_at DESC
LIMIT ?`, fts, storyID, cutoff, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// buildRelatedFTSQuery tokenizes a story's headline+lede into a de-duplicated,
// stopword-filtered OR query of quoted terms, capped so a long lede can't build
// a pathological query. Returns "" when nothing usable remains.
func buildRelatedFTSQuery(raw string) string {
var tokens []string
var cur strings.Builder
seen := make(map[string]bool)
flush := func() {
if cur.Len() == 0 {
return
}
t := strings.ToLower(cur.String())
cur.Reset()
if len([]rune(t)) < 2 || relatedStopwords[t] || seen[t] {
return
}
seen[t] = true
tokens = append(tokens, t)
}
for _, r := range raw {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
cur.WriteRune(r)
case r > 127:
cur.WriteRune(r)
default:
flush()
}
}
flush()
if len(tokens) == 0 {
return ""
}
const maxTokens = 16
if len(tokens) > maxTokens {
tokens = tokens[:maxTokens]
}
for i, t := range tokens {
tokens[i] = `"` + t + `"`
}
return strings.Join(tokens, " OR ")
}

View File

@@ -0,0 +1,141 @@
package storage
import (
"strings"
"testing"
)
// seedStoryFull inserts a story with an explicit channel + source so ranking
// tests can build a skewed affinity profile.
func seedStoryFull(t *testing.T, guid, channel, source, headline, lede string) int64 {
t.Helper()
s := &Story{
GUID: guid,
Headline: headline,
Lede: lede,
ArticleURL: "https://example.com/" + guid,
Source: source,
Channel: channel,
Classified: true,
SeenAt: nowUnix(),
}
if err := InsertStory(s); err != nil {
t.Fatalf("insert story %s: %v", guid, err)
}
var id int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, guid).Scan(&id); err != nil {
t.Fatalf("lookup id %s: %v", guid, err)
}
return id
}
func TestForYou_LeansToAffinityAndExcludesRead(t *testing.T) {
setupTestDB(t)
const sub = "sub-1"
// A skewed history: the user reads/bookmarks gaming stories.
g1 := seedStoryFull(t, "g1", "gaming", "GameSite", "Game one", "")
g2 := seedStoryFull(t, "g2", "gaming", "GameSite", "Game two", "")
// Candidates to rank (all unread until we mark some).
gc := seedStoryFull(t, "gc", "gaming", "GameSite", "Fresh gaming story", "")
tc := seedStoryFull(t, "tc", "tech", "TechSite", "Fresh tech story", "")
if err := SetRead(sub, g1, true); err != nil {
t.Fatalf("read g1: %v", err)
}
if err := SetBookmark(sub, g2, true); err != nil {
t.Fatalf("bookmark g2: %v", err)
}
got, err := ForYou(sub, 10)
if err != nil {
t.Fatalf("ForYou: %v", err)
}
if len(got) == 0 {
t.Fatal("expected results")
}
// Affinity leans gaming, so a gaming story tops the list and the tech story
// sinks to the bottom. (gc and the unread-but-bookmarked g2 both qualify and
// tie on score, so we assert on channel rather than a specific id.)
if got[0].Channel != "gaming" {
t.Fatalf("expected a gaming story first, got channel %q (id %d)", got[0].Channel, got[0].ID)
}
if got[len(got)-1].ID != tc {
t.Fatalf("expected tech candidate ranked last, got id %d", got[len(got)-1].ID)
}
// Read stories must never appear.
for _, s := range got {
if s.ID == g1 {
t.Fatalf("read story g1 leaked into ForYou")
}
}
// Sanity: the gaming candidate that was never touched should be present.
var sawGC bool
for _, s := range got {
if s.ID == gc {
sawGC = true
}
}
if !sawGC {
t.Fatalf("expected unread gaming candidate gc in results")
}
}
func TestForYou_NoHistoryReturnsNil(t *testing.T) {
setupTestDB(t)
seedStoryFull(t, "a", "tech", "TechSite", "Something", "")
got, err := ForYou("nobody", 10)
if err != nil {
t.Fatalf("ForYou: %v", err)
}
if got != nil {
t.Fatalf("expected nil for a user with no history, got %d rows", len(got))
}
}
func TestRelatedStories_OnTopicExcludesSelf(t *testing.T) {
setupTestDB(t)
seed := seedStoryFull(t, "seed", "tech", "TechSite",
"Apple unveils new iPhone camera", "The new camera sensor is larger.")
rel := seedStoryFull(t, "rel", "tech", "OtherSite",
"iPhone camera teardown reveals sensor", "A look at the new iPhone camera.")
seedStoryFull(t, "off", "gaming", "GameSite",
"Nintendo announces handheld", "A brand new portable console.")
got, err := RelatedStories(seed, 5)
if err != nil {
t.Fatalf("RelatedStories: %v", err)
}
if len(got) == 0 {
t.Fatal("expected at least one related story")
}
for _, s := range got {
if s.ID == seed {
t.Fatal("seed story returned as its own related")
}
}
if got[0].ID != rel {
t.Fatalf("expected the on-topic iPhone story first, got id %d", got[0].ID)
}
}
func TestBuildRelatedFTSQuery(t *testing.T) {
// Stopwords and 1-char tokens drop out; the rest become an OR of quoted terms.
q := buildRelatedFTSQuery("The new iPhone camera is a big deal")
if q == "" {
t.Fatal("expected a non-empty query")
}
for _, bad := range []string{`"the"`, `"is"`, `"new"`} {
if strings.Contains(q, bad) {
t.Fatalf("stopword leaked into query: %s in %q", bad, q)
}
}
for _, want := range []string{`"iphone"`, `"camera"`} {
if !strings.Contains(q, want) {
t.Fatalf("expected %s in query %q", want, q)
}
}
if buildRelatedFTSQuery("the a of to") != "" {
t.Fatal("expected empty query when only stopwords remain")
}
}

View File

@@ -6,6 +6,7 @@ CREATE TABLE IF NOT EXISTS stories (
guid TEXT UNIQUE NOT NULL,
headline TEXT NOT NULL,
lede TEXT,
content TEXT,
image_url TEXT,
article_url TEXT NOT NULL,
url_canonical TEXT,
@@ -53,6 +54,18 @@ CREATE TABLE IF NOT EXISTS user_preferences (
updated_at INTEGER NOT NULL
);
-- Per-user read + bookmark state for signed-in visitors, keyed by OIDC subject.
-- One row carries both signals; a NULL timestamp means "not set". A row with
-- both timestamps NULL is meaningless and is pruned, so presence of a row means
-- the story is read, bookmarked, or both.
CREATE TABLE IF NOT EXISTS user_story_state (
user_sub TEXT NOT NULL,
story_id INTEGER NOT NULL,
read_at INTEGER,
bookmarked_at INTEGER,
PRIMARY KEY (user_sub, story_id)
);
-- Aggregate web usage. page_views holds running view counts keyed by a coarse
-- path label ("home", channel slug, …) and the UTC day, so we can report both
-- all-time totals and per-day breakdowns without storing any per-request rows.
@@ -63,6 +76,35 @@ CREATE TABLE IF NOT EXISTS page_views (
PRIMARY KEY (path, day)
);
-- Per-source poll health, one row per configured feed (keyed by source name).
-- Written on every poll (success and failure) so the owner-facing dashboard can
-- show which feeds are healthy without keeping the poller's in-memory state.
-- last_success_at / last_item_count survive failures so a broken feed still
-- shows when it last worked and how much it last returned.
CREATE TABLE IF NOT EXISTS source_health (
source TEXT PRIMARY KEY,
last_poll_at INTEGER, -- unix, most recent poll attempt
last_success_at INTEGER, -- unix, most recent successful fetch
last_error TEXT, -- last failure message ('' when healthy)
consecutive_failures INTEGER NOT NULL DEFAULT 0,
last_item_count INTEGER NOT NULL DEFAULT 0, -- items in the last successful fetch
updated_at INTEGER NOT NULL
);
-- Web Push subscriptions for signed-in users, one row per browser/device
-- endpoint. p256dh + auth are the client's encryption keys (RFC 8291); the
-- server needs them to encrypt each push. last_notified_at is the per-endpoint
-- digest watermark: the sender only counts stories seen after it. A user can
-- have several endpoints (phone, desktop) — each is notified independently.
CREATE TABLE IF NOT EXISTS push_subscriptions (
endpoint TEXT PRIMARY KEY,
user_sub TEXT NOT NULL,
p256dh TEXT NOT NULL,
auth TEXT NOT NULL,
created_at INTEGER NOT NULL,
last_notified_at INTEGER NOT NULL
);
-- Privacy-preserving daily unique estimate. visitor is a salted hash of
-- IP+User-Agent; the salt rotates every UTC day and is never persisted, so the
-- hashes are irreversible and cannot be linked across days. We keep only enough
@@ -87,6 +129,9 @@ CREATE INDEX IF NOT EXISTS idx_reactions_post_guid ON reactions(post_guid);
CREATE UNIQUE INDEX IF NOT EXISTS idx_reactions_event_id ON reactions(event_id);
CREATE INDEX IF NOT EXISTS idx_page_views_day ON page_views(day);
CREATE INDEX IF NOT EXISTS idx_daily_visitors_day ON daily_visitors(day);
CREATE INDEX IF NOT EXISTS idx_user_state_bookmarks ON user_story_state(user_sub, bookmarked_at) WHERE bookmarked_at IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_user_state_reads ON user_story_state(user_sub, read_at) WHERE read_at IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_push_sub_user ON push_subscriptions(user_sub);
`
const ftsSchema = `

View File

@@ -0,0 +1,149 @@
package storage
import (
"database/sql"
"fmt"
)
// SourceHealth is one source's persisted poll health, as written by the poller.
type SourceHealth struct {
Source string
LastPollAt int64 // 0 = never polled
LastSuccessAt int64 // 0 = never succeeded
LastError string
ConsecutiveFailures int
LastItemCount int
UpdatedAt int64
}
// SourceContentStat is per-source content derived from the stories table (plus
// post_log for last-posted), independent of poll health.
type SourceContentStat struct {
Source string
Total int // stories currently retained for this source
Classified int // of those, how many are classified
Paywalled int // of those, how many are gated
LastSeenAt int64 // MAX(seen_at); 0 = none
LastPostedAt int64 // MAX(post_log.posted_at) joined by guid; 0 = never posted
}
// RecordPollResult upserts a source's health row after a poll attempt. On
// success it clears the error and failure counter and records the item count;
// on failure it bumps consecutive_failures and stores the message while
// preserving the last successful timestamp and item count. Fire-and-forget:
// a failure here must never disrupt polling, so errors are logged and swallowed.
func RecordPollResult(source string, ok bool, itemCount int, pollErr error) {
now := nowUnix()
if ok {
exec("record poll success",
`INSERT INTO source_health
(source, last_poll_at, last_success_at, last_error, consecutive_failures, last_item_count, updated_at)
VALUES (?, ?, ?, '', 0, ?, ?)
ON CONFLICT(source) DO UPDATE SET
last_poll_at = excluded.last_poll_at,
last_success_at = excluded.last_success_at,
last_error = '',
consecutive_failures = 0,
last_item_count = excluded.last_item_count,
updated_at = excluded.updated_at`,
source, now, now, itemCount, now)
return
}
msg := ""
if pollErr != nil {
msg = pollErr.Error()
}
exec("record poll failure",
`INSERT INTO source_health
(source, last_poll_at, last_success_at, last_error, consecutive_failures, last_item_count, updated_at)
VALUES (?, ?, NULL, ?, 1, 0, ?)
ON CONFLICT(source) DO UPDATE SET
last_poll_at = excluded.last_poll_at,
last_error = excluded.last_error,
consecutive_failures = source_health.consecutive_failures + 1,
updated_at = excluded.updated_at`,
source, now, msg, now)
}
// ListSourceHealth returns the poll-health row for every source that has been
// polled at least once, keyed by source name.
func ListSourceHealth() (map[string]SourceHealth, error) {
rows, err := Get().Query(
`SELECT source, last_poll_at, last_success_at, last_error,
consecutive_failures, last_item_count, updated_at
FROM source_health`)
if err != nil {
return nil, fmt.Errorf("list source health: %w", err)
}
defer rows.Close()
out := make(map[string]SourceHealth)
for rows.Next() {
var h SourceHealth
var lastPoll, lastSuccess sql.NullInt64
var lastErr sql.NullString
if err := rows.Scan(&h.Source, &lastPoll, &lastSuccess, &lastErr,
&h.ConsecutiveFailures, &h.LastItemCount, &h.UpdatedAt); err != nil {
return nil, err
}
h.LastPollAt = lastPoll.Int64
h.LastSuccessAt = lastSuccess.Int64
h.LastError = lastErr.String
out[h.Source] = h
}
return out, rows.Err()
}
// SourceContentStats derives per-source content counts from the stories table,
// with last-posted joined from post_log by guid. Keyed by source name. Only
// sources with at least one retained story appear; the caller pads out the rest
// from its configured source list.
func SourceContentStats() (map[string]SourceContentStat, error) {
out := make(map[string]SourceContentStat)
rows, err := Get().Query(
`SELECT source,
COUNT(*),
COALESCE(SUM(classified), 0),
COALESCE(SUM(paywalled), 0),
COALESCE(MAX(seen_at), 0)
FROM stories
GROUP BY source`)
if err != nil {
return nil, fmt.Errorf("source content stats: %w", err)
}
defer rows.Close()
for rows.Next() {
var st SourceContentStat
if err := rows.Scan(&st.Source, &st.Total, &st.Classified, &st.Paywalled, &st.LastSeenAt); err != nil {
return nil, err
}
out[st.Source] = st
}
if err := rows.Err(); err != nil {
return nil, err
}
// Last-posted per source, joined by guid. Kept separate so sources with
// stories but no posts still appear above with a zero last-posted.
prows, err := Get().Query(
`SELECT s.source, MAX(p.posted_at)
FROM post_log p
JOIN stories s ON s.guid = p.guid
GROUP BY s.source`)
if err != nil {
return nil, fmt.Errorf("source last-posted: %w", err)
}
defer prows.Close()
for prows.Next() {
var source string
var lastPosted int64
if err := prows.Scan(&source, &lastPosted); err != nil {
return nil, err
}
st := out[source]
st.Source = source
st.LastPostedAt = lastPosted
out[source] = st
}
return out, prows.Err()
}

View File

@@ -0,0 +1,104 @@
package storage
import (
"errors"
"testing"
)
func TestRecordPollResult_SuccessThenFailure(t *testing.T) {
setupTestDB(t)
// First a successful poll returning 7 items.
RecordPollResult("Feed A", true, 7, nil)
h, err := ListSourceHealth()
if err != nil {
t.Fatal(err)
}
a, ok := h["Feed A"]
if !ok {
t.Fatal("expected a health row for Feed A")
}
if a.LastItemCount != 7 || a.ConsecutiveFailures != 0 || a.LastError != "" {
t.Errorf("after success: items=%d fails=%d err=%q", a.LastItemCount, a.ConsecutiveFailures, a.LastError)
}
if a.LastPollAt == 0 || a.LastSuccessAt == 0 {
t.Errorf("after success: last_poll=%d last_success=%d, want both set", a.LastPollAt, a.LastSuccessAt)
}
successTS := a.LastSuccessAt
// Two failures in a row: failures accumulate, but the last-success timestamp
// and item count are preserved so the dashboard still shows when it last worked.
RecordPollResult("Feed A", false, 0, errors.New("dial tcp: timeout"))
RecordPollResult("Feed A", false, 0, errors.New("502 bad gateway"))
h, _ = ListSourceHealth()
a = h["Feed A"]
if a.ConsecutiveFailures != 2 {
t.Errorf("consecutive failures = %d, want 2", a.ConsecutiveFailures)
}
if a.LastError != "502 bad gateway" {
t.Errorf("last error = %q, want %q", a.LastError, "502 bad gateway")
}
if a.LastSuccessAt != successTS {
t.Errorf("last success = %d, want preserved %d", a.LastSuccessAt, successTS)
}
if a.LastItemCount != 7 {
t.Errorf("last item count = %d, want preserved 7", a.LastItemCount)
}
// Recovery clears the error and resets the counter.
RecordPollResult("Feed A", true, 3, nil)
h, _ = ListSourceHealth()
a = h["Feed A"]
if a.ConsecutiveFailures != 0 || a.LastError != "" || a.LastItemCount != 3 {
t.Errorf("after recovery: fails=%d err=%q items=%d", a.ConsecutiveFailures, a.LastError, a.LastItemCount)
}
}
func TestSourceContentStats(t *testing.T) {
setupTestDB(t)
// Source X: two classified stories, one paywalled; one posted.
InsertStory(&Story{GUID: "x1", Headline: "X one", ArticleURL: "https://x.com/1", Source: "X", Channel: "tech", Classified: true, SeenAt: 100})
InsertStory(&Story{GUID: "x2", Headline: "X two", ArticleURL: "https://x.com/2", Source: "X", Channel: "tech", Classified: true, Paywalled: true, SeenAt: 200})
// Source Y: one unclassified story, never posted.
InsertStory(&Story{GUID: "y1", Headline: "Y one", ArticleURL: "https://y.com/1", Source: "Y", SeenAt: 150})
InsertPostLog("x1", "tech", "$e1", "", false)
// Backdate/forward the post so we can assert MAX(posted_at).
Get().Exec(`UPDATE post_log SET posted_at = ? WHERE guid = ?`, int64(500), "x1")
stats, err := SourceContentStats()
if err != nil {
t.Fatal(err)
}
x := stats["X"]
if x.Total != 2 || x.Classified != 2 || x.Paywalled != 1 {
t.Errorf("X: total=%d classified=%d paywalled=%d, want 2/2/1", x.Total, x.Classified, x.Paywalled)
}
if x.LastSeenAt != 200 {
t.Errorf("X last seen = %d, want 200", x.LastSeenAt)
}
if x.LastPostedAt != 500 {
t.Errorf("X last posted = %d, want 500", x.LastPostedAt)
}
y := stats["Y"]
if y.Total != 1 || y.Classified != 0 || y.Paywalled != 0 {
t.Errorf("Y: total=%d classified=%d paywalled=%d, want 1/0/0", y.Total, y.Classified, y.Paywalled)
}
if y.LastPostedAt != 0 {
t.Errorf("Y last posted = %d, want 0 (never posted)", y.LastPostedAt)
}
}
func TestListSourceHealth_Empty(t *testing.T) {
setupTestDB(t)
h, err := ListSourceHealth()
if err != nil {
t.Fatal(err)
}
if len(h) != 0 {
t.Errorf("expected no health rows, got %d", len(h))
}
}

View File

@@ -61,6 +61,85 @@ func TestInsertStoryAndGUIDSeen(t *testing.T) {
}
}
func TestGetStoryReaderText(t *testing.T) {
setupTestDB(t)
// Reader text is only served for classified, non-sentinel stories — the same
// filter the public /api/article endpoint applies so discarded/unclassified
// bodies can't be pulled by id enumeration.
s := &Story{
GUID: "reader-guid",
Headline: "Reader Headline",
Lede: "Short lede.",
Content: "First paragraph.\n\nSecond paragraph.",
ArticleURL: "https://example.com/reader",
Source: "Src",
Channel: "tech",
Classified: true,
SeenAt: 1700000000,
}
if err := InsertStory(s); err != nil {
t.Fatal(err)
}
var id int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, s.GUID).Scan(&id); err != nil {
t.Fatal(err)
}
content, lede, found, err := GetStoryReaderText(id)
if err != nil {
t.Fatal(err)
}
if !found {
t.Fatal("expected story to be found")
}
if content != s.Content {
t.Errorf("content = %q, want %q", content, s.Content)
}
if lede != s.Lede {
t.Errorf("lede = %q, want %q", lede, s.Lede)
}
// A story ingested before content capture (no content) is still found, with
// an empty content string so the reader falls back to the lede.
bare := &Story{GUID: "bare-guid", Headline: "H", Lede: "Only a lede.", ArticleURL: "https://a.com", Source: "S", Channel: "tech", Classified: true, SeenAt: 1}
if err := InsertStory(bare); err != nil {
t.Fatal(err)
}
var bareID int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, bare.GUID).Scan(&bareID); err != nil {
t.Fatal(err)
}
content, lede, found, err = GetStoryReaderText(bareID)
if err != nil || !found {
t.Fatalf("bare story: found=%v err=%v", found, err)
}
if content != "" {
t.Errorf("bare content = %q, want empty", content)
}
if lede != "Only a lede." {
t.Errorf("bare lede = %q", lede)
}
// Unknown id reports not-found rather than erroring.
if _, _, found, err = GetStoryReaderText(999999); err != nil || found {
t.Errorf("unknown id: found=%v err=%v, want found=false err=nil", found, err)
}
// A discarded story must not be readable via /api/article, even by direct id.
discarded := &Story{GUID: "disc-guid", Headline: "H", Lede: "hidden", Content: "secret body", ArticleURL: "https://d.com", Source: "S", Channel: "_discarded", Classified: true, SeenAt: 1}
if err := InsertStory(discarded); err != nil {
t.Fatal(err)
}
var discID int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, discarded.GUID).Scan(&discID); err != nil {
t.Fatal(err)
}
if _, _, found, err = GetStoryReaderText(discID); err != nil || found {
t.Errorf("discarded story: found=%v err=%v, want found=false", found, err)
}
}
func TestInsertStoryDuplicateGUID(t *testing.T) {
setupTestDB(t)

View File

@@ -6,6 +6,7 @@ type Story struct {
GUID string
Headline string
Lede string
Content string // full article text for reader mode (feed content:encoded or scraped body); "" when unavailable
ImageURL string
ArticleURL string
URLCanonical string

View File

@@ -0,0 +1,145 @@
package storage
import (
"database/sql"
"fmt"
"strings"
)
// SetRead marks (read=true) or clears (read=false) the read flag for one story
// and one signed-in user (keyed by OIDC subject). Read and bookmark state share
// a row; clearing the last remaining flag removes the row.
func SetRead(sub string, storyID int64, read bool) error {
var ts any
if read {
ts = nowUnix()
}
_, err := Get().Exec(`
INSERT INTO user_story_state (user_sub, story_id, read_at, bookmarked_at)
VALUES (?, ?, ?, NULL)
ON CONFLICT(user_sub, story_id) DO UPDATE SET read_at = excluded.read_at`,
sub, storyID, ts)
if err != nil {
return fmt.Errorf("set read: %w", err)
}
return pruneEmptyState(sub, storyID)
}
// SetBookmark adds (on=true) or removes (on=false) a bookmark for one story and
// one signed-in user. See SetRead for the shared-row semantics.
func SetBookmark(sub string, storyID int64, on bool) error {
var ts any
if on {
ts = nowUnix()
}
_, err := Get().Exec(`
INSERT INTO user_story_state (user_sub, story_id, read_at, bookmarked_at)
VALUES (?, ?, NULL, ?)
ON CONFLICT(user_sub, story_id) DO UPDATE SET bookmarked_at = excluded.bookmarked_at`,
sub, storyID, ts)
if err != nil {
return fmt.Errorf("set bookmark: %w", err)
}
return pruneEmptyState(sub, storyID)
}
// pruneEmptyState drops a row once neither flag is set, keeping the table to
// only meaningful state.
func pruneEmptyState(sub string, storyID int64) error {
_, err := Get().Exec(
`DELETE FROM user_story_state
WHERE user_sub = ? AND story_id = ? AND read_at IS NULL AND bookmarked_at IS NULL`,
sub, storyID)
if err != nil {
return fmt.Errorf("prune user state: %w", err)
}
return nil
}
// UserStoryState reports, for the given story ids, which are read and which are
// bookmarked by this user. Both maps contain only ids whose flag is set, so a
// missing key means false. An empty sub or id list returns empty maps.
func UserStoryState(sub string, ids []int64) (read, bookmarked map[int64]bool, err error) {
read = make(map[int64]bool)
bookmarked = make(map[int64]bool)
if sub == "" || len(ids) == 0 {
return read, bookmarked, nil
}
q := `SELECT story_id, read_at, bookmarked_at FROM user_story_state
WHERE user_sub = ? AND story_id IN (` + placeholders(len(ids)) + `)`
args := make([]any, 0, len(ids)+1)
args = append(args, sub)
for _, id := range ids {
args = append(args, id)
}
rows, err := Get().Query(q, args...)
if err != nil {
return nil, nil, fmt.Errorf("user story state: %w", err)
}
defer rows.Close()
for rows.Next() {
var id int64
var r, b sql.NullInt64
if err := rows.Scan(&id, &r, &b); err != nil {
return nil, nil, err
}
if r.Valid {
read[id] = true
}
if b.Valid {
bookmarked[id] = true
}
}
return read, bookmarked, rows.Err()
}
// ListBookmarks returns the user's bookmarked stories, most recently bookmarked
// first, restricted to still-classified stories.
func ListBookmarks(sub string, limit, offset int) ([]Story, error) {
rows, err := Get().Query(
`SELECT s.id, s.guid, s.headline, s.lede, s.image_url, s.article_url, s.source, s.platforms, s.channel, s.paywalled, s.seen_at,
EXISTS(SELECT 1 FROM post_log p WHERE p.guid = s.guid) AS posted
FROM user_story_state u
JOIN stories s ON s.id = u.story_id
WHERE u.user_sub = ?
AND u.bookmarked_at IS NOT NULL
AND s.classified = 1
AND s.channel NOT IN ('_discarded', '_duplicate')
ORDER BY u.bookmarked_at DESC, u.story_id DESC
LIMIT ? OFFSET ?`, sub, limit, offset)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.ID, &s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.Paywalled, &s.SeenAt, &s.Posted); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// CountBookmarks returns how many bookmarked stories the user would see on the
// bookmarks page. It applies the same classified filter as ListBookmarks so the
// "N saved" header can't exceed the number of rendered cards.
func CountBookmarks(sub string) (int, error) {
var n int
err := Get().QueryRow(
`SELECT COUNT(*) FROM user_story_state u
JOIN stories s ON s.id = u.story_id
WHERE u.user_sub = ? AND u.bookmarked_at IS NOT NULL AND s.classified = 1
AND s.channel NOT IN ('_discarded', '_duplicate')`,
sub).Scan(&n)
return n, err
}
// placeholders returns "?, ?, …" with n slots for an IN clause.
func placeholders(n int) string {
if n <= 0 {
return ""
}
return strings.Repeat("?, ", n-1) + "?"
}

View File

@@ -0,0 +1,126 @@
package storage
import "testing"
func seedStory(t *testing.T, guid string) int64 {
t.Helper()
s := &Story{
GUID: guid,
Headline: "Headline " + guid,
Lede: "Lede.",
ArticleURL: "https://example.com/" + guid,
Source: "Test Source",
Channel: "tech",
Classified: true,
SeenAt: 1700000000,
}
if err := InsertStory(s); err != nil {
t.Fatalf("insert story %s: %v", guid, err)
}
var id int64
if err := Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, guid).Scan(&id); err != nil {
t.Fatalf("lookup id %s: %v", guid, err)
}
return id
}
func TestUserStoryState_ReadAndBookmark(t *testing.T) {
setupTestDB(t)
id1 := seedStory(t, "s1")
id2 := seedStory(t, "s2")
const sub = "ak-sub-1"
// Nothing set yet.
read, marks, err := UserStoryState(sub, []int64{id1, id2})
if err != nil {
t.Fatalf("initial state: %v", err)
}
if len(read) != 0 || len(marks) != 0 {
t.Fatalf("expected empty state, got read=%v bookmarked=%v", read, marks)
}
// Mark id1 read, bookmark id1 and id2.
if err := SetRead(sub, id1, true); err != nil {
t.Fatalf("set read: %v", err)
}
if err := SetBookmark(sub, id1, true); err != nil {
t.Fatalf("set bookmark id1: %v", err)
}
if err := SetBookmark(sub, id2, true); err != nil {
t.Fatalf("set bookmark id2: %v", err)
}
read, marks, _ = UserStoryState(sub, []int64{id1, id2})
if !read[id1] || read[id2] {
t.Fatalf("read map wrong: %v", read)
}
if !marks[id1] || !marks[id2] {
t.Fatalf("bookmark map wrong: %v", marks)
}
// id1 carries both flags in a single row.
if !read[id1] || !marks[id1] {
t.Fatalf("expected id1 read+bookmarked, read=%v marks=%v", read, marks)
}
// ListBookmarks orders newest bookmark first, tiebreaking on story id so the
// result is deterministic within a single second.
bm, err := ListBookmarks(sub, 10, 0)
if err != nil {
t.Fatalf("list bookmarks: %v", err)
}
if len(bm) != 2 {
t.Fatalf("expected 2 bookmarks, got %d", len(bm))
}
if bm[0].ID != id2 || bm[1].ID != id1 {
t.Fatalf("bookmark order wrong: %d then %d", bm[0].ID, bm[1].ID)
}
if n, _ := CountBookmarks(sub); n != 2 {
t.Fatalf("count bookmarks = %d, want 2", n)
}
}
func TestUserStoryState_ClearRemovesRow(t *testing.T) {
setupTestDB(t)
id := seedStory(t, "s1")
const sub = "ak-sub-1"
if err := SetRead(sub, id, true); err != nil {
t.Fatalf("set read: %v", err)
}
if err := SetRead(sub, id, false); err != nil {
t.Fatalf("unset read: %v", err)
}
// Row should be gone since neither flag is set.
var n int
if err := Get().QueryRow(`SELECT COUNT(*) FROM user_story_state WHERE user_sub = ?`, sub).Scan(&n); err != nil {
t.Fatalf("count rows: %v", err)
}
if n != 0 {
t.Fatalf("expected row pruned, found %d", n)
}
// Setting read then bookmark then clearing only read must keep the row.
_ = SetRead(sub, id, true)
_ = SetBookmark(sub, id, true)
_ = SetRead(sub, id, false)
read, marks, _ := UserStoryState(sub, []int64{id})
if read[id] {
t.Fatalf("expected not read")
}
if !marks[id] {
t.Fatalf("expected still bookmarked after clearing read")
}
}
func TestUserStoryState_ScopedBySub(t *testing.T) {
setupTestDB(t)
id := seedStory(t, "s1")
_ = SetBookmark("user-a", id, true)
read, marks, _ := UserStoryState("user-b", []int64{id})
if len(read) != 0 || len(marks) != 0 {
t.Fatalf("user-b should see no state, got read=%v marks=%v", read, marks)
}
if n, _ := CountBookmarks("user-b"); n != 0 {
t.Fatalf("user-b count = %d, want 0", n)
}
}

292
internal/web/feed.go Normal file
View File

@@ -0,0 +1,292 @@
package web
import (
"encoding/json"
"encoding/xml"
"html"
"log/slog"
"net/http"
"strings"
"time"
"pete/internal/storage"
)
// feedItemLimit caps how many stories each outbound feed carries.
const feedItemLimit = 50
// feedCacheControl is a short cache window; feeds refresh a few times an hour.
const feedCacheControl = "public, max-age=300"
const (
rssContentNS = "http://purl.org/rss/1.0/modules/content/"
rssAtomNS = "http://www.w3.org/2005/Atom"
jsonFeedVer = "https://jsonfeed.org/version/1.1"
)
// --- RSS 2.0 ---
type rssFeedXML struct {
XMLName xml.Name `xml:"rss"`
Version string `xml:"version,attr"`
ContentNS string `xml:"xmlns:content,attr"`
AtomNS string `xml:"xmlns:atom,attr"`
Channel rssChannel `xml:"channel"`
}
type rssChannel struct {
Title string `xml:"title"`
Link string `xml:"link"`
Description string `xml:"description"`
Language string `xml:"language,omitempty"`
LastBuildDate string `xml:"lastBuildDate,omitempty"`
Generator string `xml:"generator,omitempty"`
AtomLink rssAtomLink `xml:"atom:link"`
Items []rssItem `xml:"item"`
}
type rssAtomLink struct {
Href string `xml:"href,attr"`
Rel string `xml:"rel,attr"`
Type string `xml:"type,attr"`
}
type rssItem struct {
Title string `xml:"title"`
Link string `xml:"link"`
GUID rssGUID `xml:"guid"`
PubDate string `xml:"pubDate,omitempty"`
Category string `xml:"category,omitempty"`
Description string `xml:"description"`
Content *rssContent `xml:"content:encoded,omitempty"`
}
type rssGUID struct {
IsPermaLink string `xml:"isPermaLink,attr"`
Value string `xml:",chardata"`
}
type rssContent struct {
Value string `xml:",cdata"`
}
// --- JSON Feed 1.1 ---
type jsonFeed struct {
Version string `json:"version"`
Title string `json:"title"`
HomePageURL string `json:"home_page_url"`
FeedURL string `json:"feed_url"`
Description string `json:"description,omitempty"`
Items []jsonFeedItem `json:"items"`
}
type jsonFeedItem struct {
ID string `json:"id"`
URL string `json:"url"`
Title string `json:"title"`
ContentText string `json:"content_text,omitempty"`
Summary string `json:"summary,omitempty"`
Image string `json:"image,omitempty"`
DatePublished string `json:"date_published,omitempty"`
Authors []jsonFeedAuthor `json:"authors,omitempty"`
Tags []string `json:"tags,omitempty"`
}
type jsonFeedAuthor struct {
Name string `json:"name"`
}
// handleFeedXML serves an RSS 2.0 feed for the given channel ("" = all channels).
func (s *Server) handleFeedXML(w http.ResponseWriter, r *http.Request, channel string) {
stories, err := storage.ListForFeed(channel, feedItemLimit)
if err != nil {
slog.Error("web: feed query failed", "channel", channel, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
base := s.publicBase(r)
title, homeURL, desc := s.feedMeta(channel, base)
selfURL := base + feedPath(channel, "xml")
feed := rssFeedXML{
Version: "2.0",
ContentNS: rssContentNS,
AtomNS: rssAtomNS,
Channel: rssChannel{
Title: title,
Link: homeURL,
Description: desc,
Language: "en",
Generator: "Pete",
AtomLink: rssAtomLink{Href: selfURL, Rel: "self", Type: "application/rss+xml"},
Items: make([]rssItem, 0, len(stories)),
},
}
if len(stories) > 0 {
feed.Channel.LastBuildDate = feedTime(stories[0]).UTC().Format(time.RFC1123Z)
}
for _, st := range stories {
link := storyLink(st)
item := rssItem{
Title: st.Headline,
Link: link,
GUID: rssGUID{IsPermaLink: "false", Value: st.GUID},
PubDate: feedTime(st).UTC().Format(time.RFC1123Z),
Category: st.Channel,
Description: st.Lede,
}
if html := contentToHTML(st.Content); html != "" {
item.Content = &rssContent{Value: html}
}
feed.Channel.Items = append(feed.Channel.Items, item)
}
w.Header().Set("Content-Type", "application/rss+xml; charset=utf-8")
w.Header().Set("Cache-Control", feedCacheControl)
if _, err := w.Write([]byte(xml.Header)); err != nil {
return
}
enc := xml.NewEncoder(w)
enc.Indent("", " ")
if err := enc.Encode(feed); err != nil {
slog.Error("web: rss encode failed", "err", err)
}
}
// handleFeedJSON serves a JSON Feed 1.1 for the given channel ("" = all).
func (s *Server) handleFeedJSON(w http.ResponseWriter, r *http.Request, channel string) {
stories, err := storage.ListForFeed(channel, feedItemLimit)
if err != nil {
slog.Error("web: feed query failed", "channel", channel, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
base := s.publicBase(r)
title, homeURL, desc := s.feedMeta(channel, base)
feed := jsonFeed{
Version: jsonFeedVer,
Title: title,
HomePageURL: homeURL,
FeedURL: base + feedPath(channel, "json"),
Description: desc,
Items: make([]jsonFeedItem, 0, len(stories)),
}
for _, st := range stories {
item := jsonFeedItem{
ID: st.GUID,
URL: storyLink(st),
Title: st.Headline,
ContentText: strings.TrimSpace(st.Content),
Summary: st.Lede,
Image: st.ImageURL,
DatePublished: feedTime(st).UTC().Format(time.RFC3339),
Tags: []string{st.Channel},
}
if item.ContentText == "" {
item.ContentText = st.Lede
}
if st.Source != "" {
item.Authors = []jsonFeedAuthor{{Name: st.Source}}
}
feed.Items = append(feed.Items, item)
}
w.Header().Set("Content-Type", "application/feed+json; charset=utf-8")
w.Header().Set("Cache-Control", feedCacheControl)
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
if err := enc.Encode(feed); err != nil {
slog.Error("web: json feed encode failed", "err", err)
}
}
// feedMeta returns the title, home-page URL, and description for a feed. An
// empty channel is the site-wide feed; a slug scopes to that channel.
func (s *Server) feedMeta(channel, base string) (title, homeURL, desc string) {
site := s.cfg.SiteTitle
if site == "" {
site = "Pete"
}
if channel == "" {
return site, base + "/", "The latest across every channel."
}
ch := channelBySlug(channel)
return site + " — " + ch.Title, base + "/" + channel, ch.Blurb
}
// feedPath is the site-relative path of a feed, e.g. "/feed.xml" or
// "/gaming/feed.json".
func feedPath(channel, ext string) string {
if channel == "" {
return "/feed." + ext
}
return "/" + channel + "/feed." + ext
}
// storyLink is the canonical outbound link for a story: its canonical URL when
// known, otherwise the raw article URL.
func storyLink(s storage.Story) string {
if s.URLCanonical != "" {
return s.URLCanonical
}
return s.ArticleURL
}
// feedTime picks the best timestamp for a story: its published date when
// present, otherwise when Pete first saw it.
func feedTime(s storage.Story) time.Time {
if s.PublishedAt > 0 {
return time.Unix(s.PublishedAt, 0)
}
return time.Unix(s.SeenAt, 0)
}
// channelBySlug looks up a channel by slug, falling back to a bare title-cased
// entry so an unknown slug never panics.
func channelBySlug(slug string) Channel {
for _, ch := range channels {
if ch.Slug == slug {
return ch
}
}
return Channel{Slug: slug, Title: slug, Theme: slug}
}
// publicBase returns the site's public origin (scheme://host, no trailing
// slash). It prefers the configured base_url and falls back to the request's
// host so feeds still carry absolute URLs when base_url is unset.
func (s *Server) publicBase(r *http.Request) string {
if b := strings.TrimRight(s.cfg.BaseURL, "/"); b != "" {
return b
}
scheme := "http"
if r.TLS != nil || strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https") {
scheme = "https"
}
return scheme + "://" + r.Host
}
// contentToHTML turns Pete's stored plain-text article body (paragraphs split by
// blank lines, soft line breaks inside) into simple, escaped HTML for RSS
// content:encoded. Returns "" when there is no body.
func contentToHTML(text string) string {
text = strings.TrimSpace(text)
if text == "" {
return ""
}
var b strings.Builder
for _, para := range strings.Split(text, "\n\n") {
para = strings.TrimSpace(para)
if para == "" {
continue
}
b.WriteString("<p>")
b.WriteString(strings.ReplaceAll(html.EscapeString(para), "\n", "<br>"))
b.WriteString("</p>")
}
return b.String()
}

210
internal/web/feed_test.go Normal file
View File

@@ -0,0 +1,210 @@
package web
import (
"encoding/json"
"encoding/xml"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"pete/internal/config"
"pete/internal/storage"
)
// TestFeeds exercises the outbound RSS and JSON feeds end to end: a classified
// story appears in both formats, carrying its canonical link, full content, and
// a real pubDate; a channel-scoped feed excludes stories from other channels.
func TestFeeds(t *testing.T) {
storage.Close()
if err := storage.Init(filepath.Join(t.TempDir(), "feed.db")); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { storage.Close() })
pub := time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC).Unix()
tech := &storage.Story{
GUID: "feed-tech-1",
Headline: "Chips & Dips: A Tech Tale",
Lede: "The lede for the tech story.",
Content: "First paragraph.\n\nSecond paragraph with <html> & symbols.",
ArticleURL: "https://example.com/tech/story?utm=x",
URLCanonical: "https://example.com/tech/story",
Source: "Example Wire",
Channel: "tech",
Classified: true,
SeenAt: time.Now().Unix(),
PublishedAt: pub,
}
gaming := &storage.Story{
GUID: "feed-gaming-1",
Headline: "A Gaming Headline",
Lede: "Gaming lede.",
ArticleURL: "https://example.com/gaming/story",
Source: "Play Wire",
Channel: "gaming",
Classified: true,
SeenAt: time.Now().Unix(),
}
for _, st := range []*storage.Story{tech, gaming} {
if err := storage.InsertStory(st); err != nil {
t.Fatal(err)
}
}
s, err := New(config.WebConfig{SiteTitle: "Pete", ListenAddr: ":0", BaseURL: "https://news.example/"}, nil, true)
if err != nil {
t.Fatal(err)
}
// --- Site-wide RSS ---
rw := httptest.NewRecorder()
s.handleFeedXML(rw, httptest.NewRequest("GET", "/feed.xml", nil), "")
if rw.Code != 200 {
t.Fatalf("rss status = %d", rw.Code)
}
if ct := rw.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/rss+xml") {
t.Errorf("rss content-type = %q", ct)
}
var rss struct {
Channel struct {
Title string `xml:"title"`
AtomLink struct {
Href string `xml:"href,attr"`
} `xml:"link"`
Items []struct {
Title string `xml:"title"`
Link string `xml:"link"`
GUID string `xml:"guid"`
PubDate string `xml:"pubDate"`
Content string `xml:"encoded"` // content:encoded
} `xml:"item"`
} `xml:"channel"`
}
if err := xml.Unmarshal(rw.Body.Bytes(), &rss); err != nil {
t.Fatalf("rss did not parse as XML: %v\n%s", err, rw.Body.String())
}
if len(rss.Channel.Items) != 2 {
t.Fatalf("rss items = %d, want 2", len(rss.Channel.Items))
}
// Newest-first: PublishedAt March 2026 vs gaming's seen_at (now) — gaming is
// newer, so it sorts first. Find the tech item to assert on.
var techItem *struct {
Title string `xml:"title"`
Link string `xml:"link"`
GUID string `xml:"guid"`
PubDate string `xml:"pubDate"`
Content string `xml:"encoded"`
}
for i := range rss.Channel.Items {
if rss.Channel.Items[i].GUID == "feed-tech-1" {
techItem = &rss.Channel.Items[i]
}
}
if techItem == nil {
t.Fatal("tech item missing from rss")
}
if techItem.Link != "https://example.com/tech/story" {
t.Errorf("rss link = %q, want canonical", techItem.Link)
}
if techItem.Title != "Chips & Dips: A Tech Tale" {
t.Errorf("rss title = %q", techItem.Title)
}
if !strings.Contains(techItem.Content, "<p>First paragraph.</p>") {
t.Errorf("rss content missing paragraph markup: %q", techItem.Content)
}
if !strings.Contains(techItem.Content, "&lt;html&gt;") {
t.Errorf("rss content did not escape embedded markup: %q", techItem.Content)
}
if !strings.Contains(techItem.PubDate, "2026") {
t.Errorf("rss pubDate = %q, want the published date", techItem.PubDate)
}
if !strings.Contains(rw.Body.String(), `href="https://news.example/feed.xml"`) {
t.Errorf("rss missing atom self link with base_url")
}
// --- Site-wide JSON Feed ---
rw = httptest.NewRecorder()
s.handleFeedJSON(rw, httptest.NewRequest("GET", "/feed.json", nil), "")
if rw.Code != 200 {
t.Fatalf("json status = %d", rw.Code)
}
if ct := rw.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/feed+json") {
t.Errorf("json content-type = %q", ct)
}
var jf struct {
Version string `json:"version"`
FeedURL string `json:"feed_url"`
HomePageURL string `json:"home_page_url"`
Items []struct {
ID string `json:"id"`
URL string `json:"url"`
Title string `json:"title"`
ContentText string `json:"content_text"`
Tags []string `json:"tags"`
Authors []struct {
Name string `json:"name"`
} `json:"authors"`
} `json:"items"`
}
if err := json.Unmarshal(rw.Body.Bytes(), &jf); err != nil {
t.Fatalf("json feed did not parse: %v", err)
}
if !strings.Contains(jf.Version, "jsonfeed.org") {
t.Errorf("json version = %q", jf.Version)
}
if jf.FeedURL != "https://news.example/feed.json" {
t.Errorf("json feed_url = %q", jf.FeedURL)
}
if len(jf.Items) != 2 {
t.Fatalf("json items = %d, want 2", len(jf.Items))
}
var jTech *struct {
ID string `json:"id"`
URL string `json:"url"`
Title string `json:"title"`
ContentText string `json:"content_text"`
Tags []string `json:"tags"`
Authors []struct {
Name string `json:"name"`
} `json:"authors"`
}
for i := range jf.Items {
if jf.Items[i].ID == "feed-tech-1" {
jTech = &jf.Items[i]
}
}
if jTech == nil {
t.Fatal("tech item missing from json feed")
}
if jTech.URL != "https://example.com/tech/story" {
t.Errorf("json url = %q, want canonical", jTech.URL)
}
if !strings.Contains(jTech.ContentText, "Second paragraph") {
t.Errorf("json content_text = %q", jTech.ContentText)
}
if len(jTech.Tags) != 1 || jTech.Tags[0] != "tech" {
t.Errorf("json tags = %v", jTech.Tags)
}
if len(jTech.Authors) != 1 || jTech.Authors[0].Name != "Example Wire" {
t.Errorf("json authors = %v", jTech.Authors)
}
// --- Channel-scoped RSS excludes other channels ---
rw = httptest.NewRecorder()
s.handleFeedXML(rw, httptest.NewRequest("GET", "/tech/feed.xml", nil), "tech")
rss.Channel.Items = nil // xml.Unmarshal appends; clear the site-wide items
if err := xml.Unmarshal(rw.Body.Bytes(), &rss); err != nil {
t.Fatalf("channel rss parse: %v", err)
}
if len(rss.Channel.Items) != 1 {
t.Fatalf("channel rss items = %d, want 1", len(rss.Channel.Items))
}
if rss.Channel.Items[0].GUID != "feed-tech-1" {
t.Errorf("channel rss wrong item: %q", rss.Channel.Items[0].GUID)
}
if !strings.Contains(rss.Channel.Title, "Tech") {
t.Errorf("channel rss title = %q, want channel name", rss.Channel.Title)
}
}

View File

@@ -17,6 +17,7 @@ const pageSize = 24
// StoryView is the trimmed-down record used in templates.
type StoryView struct {
ID int64
Headline string
Lede string
ImageURL string
@@ -30,6 +31,7 @@ type StoryView struct {
func toView(s storage.Story) StoryView {
return StoryView{
ID: s.ID,
Headline: s.Headline,
Lede: s.Lede,
ImageURL: s.ImageURL,
@@ -43,16 +45,20 @@ func toView(s storage.Story) StoryView {
}
type pageData struct {
SiteTitle string
Channels []Channel
Active string // slug of active channel, "" for landing
Weather Weather
Phase string // when set, forces day/dawn/dusk/night and disables the clock-driven phase JS
AllSources template.JS // JSON array of {name, channel} for the settings panel
AuthEnabled bool // sign-in is available
User *SessionUser // nil for anonymous visitors
UserPrefs template.JS // signed-in user's stored prefs blob (JSON), or "null"
Path string // current request path, for post-login return
SiteTitle string
Channels []Channel
Active string // slug of active channel, "" for landing
Weather Weather
Phase string // when set, forces day/dawn/dusk/night and disables the clock-driven phase JS
AllSources template.JS // JSON array of {name, channel} for the settings panel
AuthEnabled bool // sign-in is available
User *SessionUser // nil for anonymous visitors
UserPrefs template.JS // signed-in user's stored prefs blob (JSON), or "null"
Path string // current request path, for post-login return
PostingEnabled bool // false = web-only mode; hide Matrix-posting UI
IsAdmin bool // signed-in user is on the admin allowlist (shows /status link)
PushEnabled bool // Web Push is configured (shows the notifications toggle to signed-in users)
PushPublicKey string // VAPID public key handed to the client to subscribe
}
type channelPage struct {
@@ -72,6 +78,7 @@ type indexPage struct {
JustPosted []StoryView
Stats []channelStat
Latest []StoryView
ForYou []StoryView // personalized rail; empty for anon / no-history users
}
type channelStat struct {
@@ -103,17 +110,21 @@ func (s *Server) base(r *http.Request) pageData {
srcJSON = []byte("[]")
}
d := pageData{
SiteTitle: s.cfg.SiteTitle,
Channels: channels,
Weather: currentWeather(time.Now()),
AllSources: jsForScript(srcJSON),
AuthEnabled: s.auth != nil,
UserPrefs: template.JS("null"),
Path: r.URL.Path,
SiteTitle: s.cfg.SiteTitle,
Channels: channels,
Weather: currentWeather(time.Now()),
AllSources: jsForScript(srcJSON),
AuthEnabled: s.auth != nil,
UserPrefs: template.JS("null"),
Path: r.URL.Path,
PostingEnabled: s.postingEnabled,
PushEnabled: s.auth != nil && s.cfg.Push.Enabled,
PushPublicKey: s.cfg.Push.VAPIDPublicKey,
}
if s.auth != nil {
if u := s.auth.userFromRequest(r); u != nil {
d.User = u
d.IsAdmin = s.adminSubs[u.Sub]
if blob, err := storage.GetUserPrefs(u.Sub); err == nil && blob != "" {
d.UserPrefs = jsForScript([]byte(blob))
}
@@ -174,9 +185,54 @@ func (s *Server) handleIndex(w http.ResponseWriter, r *http.Request) {
Stats: stats,
Latest: latest,
}
// For signed-in users with some read/bookmark history, lead with a small
// personalized rail. ForYou returns nothing for anon / no-history users, so
// the section simply doesn't render for them.
if s.auth != nil {
if u := s.auth.userFromRequest(r); u != nil {
const forYouLimit = 8
if fy, err := storage.ForYou(u.Sub, forYouLimit); err != nil {
slog.Error("web: for-you rail failed", "sub", u.Sub, "err", err)
} else {
for _, row := range fy {
data.ForYou = append(data.ForYou, toView(row))
}
}
}
}
s.render(w, "index", data)
}
type forYouPage struct {
pageData
Stories []StoryView
}
// handleForYou renders the dedicated personalized feed. Anonymous visitors are
// sent to sign-in (the route is only registered when auth is on).
func (s *Server) handleForYou(w http.ResponseWriter, r *http.Request) {
u := s.auth.userFromRequest(r)
if u == nil {
http.Redirect(w, r, "/auth/login?next=/for-you", http.StatusSeeOther)
return
}
s.track(r, "for-you")
const limit = 30
rows, err := storage.ForYou(u.Sub, limit)
if err != nil {
slog.Error("web: for-you page failed", "sub", u.Sub, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
views := make([]StoryView, 0, len(rows))
for _, row := range rows {
views = append(views, toView(row))
}
base := s.base(r)
base.Active = "for-you"
s.render(w, "for-you", forYouPage{pageData: base, Stories: views})
}
func (s *Server) handleChannel(w http.ResponseWriter, r *http.Request, ch Channel) {
s.track(r, ch.Slug)
page := 1
@@ -218,6 +274,64 @@ func (s *Server) handleChannel(w http.ResponseWriter, r *http.Request, ch Channe
s.render(w, "channel", data)
}
type bookmarksPage struct {
pageData
Stories []StoryView
Page int
HasPrev bool
HasNext bool
PrevURL string
NextURL string
Total int
}
// handleBookmarks lists the signed-in user's bookmarked stories. Anonymous
// visitors are sent to sign-in (the route is only registered when auth is on).
func (s *Server) handleBookmarks(w http.ResponseWriter, r *http.Request) {
u := s.auth.userFromRequest(r)
if u == nil {
http.Redirect(w, r, "/auth/login?next=/bookmarks", http.StatusSeeOther)
return
}
s.track(r, "bookmarks")
page := 1
if p := r.URL.Query().Get("page"); p != "" {
if n, err := strconv.Atoi(p); err == nil && n > 0 {
page = n
}
}
offset := (page - 1) * pageSize
rows, err := storage.ListBookmarks(u.Sub, pageSize+1, offset)
if err != nil {
slog.Error("web: list bookmarks failed", "sub", u.Sub, "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
hasNext := len(rows) > pageSize
if hasNext {
rows = rows[:pageSize]
}
views := make([]StoryView, 0, len(rows))
for _, row := range rows {
views = append(views, toView(row))
}
total, _ := storage.CountBookmarks(u.Sub)
base := s.base(r)
base.Active = "bookmarks"
data := bookmarksPage{
pageData: base,
Stories: views,
Page: page,
HasPrev: page > 1,
HasNext: hasNext,
PrevURL: fmt.Sprintf("/bookmarks?page=%d", page-1),
NextURL: fmt.Sprintf("/bookmarks?page=%d", page+1),
Total: total,
}
s.render(w, "bookmarks", data)
}
var (
demoVariants = []string{"rain", "petals", "jacaranda", "motes", "leaves", "clear", "clouds", "snow", "fog", "storm"}
demoIntensities = []string{"light", "medium", "heavy"}
@@ -307,6 +421,12 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"results": toSearchResults(rows)})
}
// toSearchResults maps stories to the JSON shape shared by /api/search and
// /api/related, resolving each story's channel to its display metadata.
func toSearchResults(rows []storage.Story) []searchResult {
channelByName := make(map[string]Channel, len(channels))
for _, ch := range channels {
channelByName[ch.Slug] = ch
@@ -331,7 +451,75 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
Posted: row.Posted,
})
}
_ = json.NewEncoder(w).Encode(map[string]any{"results": results})
return results
}
// handleRelated returns stories textually similar to a given story, for the
// "You might also like" rail in reader mode. It is public (reader mode works
// for anonymous visitors too); for signed-in users it drops already-read
// stories so recommendations stay fresh.
func (s *Server) handleRelated(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Cache-Control", "no-store")
id, err := strconv.ParseInt(strings.TrimSpace(r.URL.Query().Get("id")), 10, 64)
if err != nil || id <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
const relatedLimit = 6
// Over-fetch so dropping already-read stories doesn't starve the rail.
rows, err := storage.RelatedStories(id, relatedLimit*2)
if err != nil {
slog.Error("web: related failed", "id", id, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
if s.auth != nil && len(rows) > 0 {
if u := s.auth.userFromRequest(r); u != nil {
ids := make([]int64, 0, len(rows))
for _, row := range rows {
ids = append(ids, row.ID)
}
if read, _, err := storage.UserStoryState(u.Sub, ids); err == nil {
kept := rows[:0]
for _, row := range rows {
if !read[row.ID] {
kept = append(kept, row)
}
}
rows = kept
}
}
}
if len(rows) > relatedLimit {
rows = rows[:relatedLimit]
}
_ = json.NewEncoder(w).Encode(map[string]any{"results": toSearchResults(rows)})
}
// handleArticle serves the stored full text of a single story for reader mode.
// The client already has headline/image/source/time from the card's data
// attributes, so this returns just the body text (and the lede as a fallback
// for stories ingested before content was captured).
func (s *Server) handleArticle(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Cache-Control", "no-store")
id, err := strconv.ParseInt(strings.TrimSpace(r.URL.Query().Get("id")), 10, 64)
if err != nil || id <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
content, lede, found, err := storage.GetStoryReaderText(id)
if err != nil {
slog.Error("web: article read failed", "id", id, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
if !found {
http.Error(w, `{"error":"not found"}`, http.StatusNotFound)
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"content": content, "lede": lede})
}
func shortTimeAgo(t time.Time) string {

236
internal/web/push_sender.go Normal file
View File

@@ -0,0 +1,236 @@
package web
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"net/http"
"time"
"pete/internal/safehttp"
"pete/internal/storage"
webpush "github.com/SherClockHolmes/webpush-go"
)
// digestScan caps how many new stories the sender inspects per subscriber in one
// pass. Well past MinStories; the digest only needs a count and one headline.
const digestScan = 60
// pushSendTimeout bounds one push delivery. The endpoint is user-supplied, so a
// hostile or dead push service must not be able to wedge the (serial) digest
// loop and starve every other subscriber.
const pushSendTimeout = 15 * time.Second
// runPushSender periodically builds and delivers a "N new stories" digest to
// each subscriber, respecting their disabled-sources preference. It's started
// only when push is configured. Best-effort throughout: a failed send never
// stops the loop, and a permanently-gone endpoint is pruned.
func (s *Server) runPushSender(ctx context.Context) {
interval := time.Duration(s.cfg.Push.IntervalMinutes) * time.Minute
if interval <= 0 {
interval = 6 * time.Hour
}
slog.Info("web: push digest sender started", "interval", interval, "min_stories", s.cfg.Push.MinStories)
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
s.sendDigests()
}
}
}
// sendDigests walks every subscription once, notifying those with enough new
// stories since their last digest.
func (s *Server) sendDigests() {
subs, err := storage.ListPushSubscriptions()
if err != nil {
slog.Error("push: list subscriptions failed", "err", err)
return
}
if len(subs) == 0 {
return
}
// Disabled-source sets are per user; cache within a pass so a user with
// several devices only parses prefs once.
disabledByUser := make(map[string]map[string]bool)
sent, pruned := 0, 0
for _, sub := range subs {
stories, err := storage.NewClassifiedSince(sub.LastNotifiedAt, digestScan)
if err != nil {
slog.Error("push: scan new stories failed", "sub", sub.UserSub, "err", err)
continue
}
if len(stories) == 0 {
continue
}
disabled, ok := disabledByUser[sub.UserSub]
if !ok {
disabled = disabledSourcesFor(sub.UserSub)
disabledByUser[sub.UserSub] = disabled
}
count, top := 0, ""
for _, st := range stories {
if disabled[st.Source] {
continue
}
if count == 0 {
top = st.Headline
}
count++
}
if count < s.cfg.Push.MinStories {
// If a full scan window filled entirely with stories the user has
// hidden, the non-hidden count can stay below the threshold forever
// while the watermark never advances — the same hidden window is
// re-scanned every pass and the subscriber is permanently starved of
// digests. When the window was capped (a genuine glut, not a quiet
// spell), step the watermark past it so the next pass sees fresh
// stories. A non-full window is a real lull; leave it to accumulate.
if len(stories) == digestScan {
if derr := storage.TouchPushSubscription(sub.Endpoint, stories[0].SeenAt); derr != nil {
slog.Error("push: advance watermark past hidden glut failed", "err", derr)
}
}
continue
}
payload := buildDigestPayload(count, top)
gone, err := s.sendPush(sub, payload)
if gone {
if derr := storage.RemovePushSubscription(sub.Endpoint); derr != nil {
slog.Error("push: prune gone subscription failed", "err", derr)
} else {
pruned++
}
continue
}
if err != nil {
slog.Warn("push: send failed", "sub", sub.UserSub, "err", err)
continue
}
// Advance to the newest story this digest actually accounted for, not a
// pass-start "now": the loop can run for minutes (one slow endpoint per
// send), so stories arriving mid-pass would otherwise be re-counted next
// pass. stories[0] is the newest in the scanned window (seen_at DESC).
if derr := storage.TouchPushSubscription(sub.Endpoint, stories[0].SeenAt); derr != nil {
slog.Error("push: advance watermark failed", "err", derr)
}
sent++
}
if sent > 0 || pruned > 0 {
slog.Info("push: digest pass complete", "sent", sent, "pruned", pruned, "subscriptions", len(subs))
}
}
// buildDigestPayload renders the notification JSON the service worker expects.
func buildDigestPayload(count int, top string) []byte {
body := fmt.Sprintf("%d new stories", count)
if count == 1 {
body = "1 new story"
}
if top != "" {
body += ": " + top
}
b, _ := json.Marshal(map[string]string{
"title": "Pete has fresh news",
"body": body,
"url": "/",
"tag": "pete-digest",
})
return b
}
// sendPush encrypts and delivers one notification. It reports gone=true when the
// push service says the endpoint no longer exists (404/410) so the caller can
// prune it; err is set for other, likely-transient failures.
func (s *Server) sendPush(sub storage.PushSubscription, payload []byte) (gone bool, err error) {
resp, err := webpush.SendNotification(payload, &webpush.Subscription{
Endpoint: sub.Endpoint,
Keys: webpush.Keys{P256dh: sub.P256dh, Auth: sub.Auth},
}, &webpush.Options{
// The endpoint URL comes from the browser and is attacker-influenceable,
// so deliver through the SSRF-guarded client (blocks loopback/RFC1918/
// link-local/metadata targets) with a hard timeout — never the library's
// unguarded default http.Client.
HTTPClient: s.pushClient(),
Subscriber: s.cfg.Push.Subject,
VAPIDPublicKey: s.cfg.Push.VAPIDPublicKey,
VAPIDPrivateKey: s.cfg.Push.VAPIDPrivateKey,
TTL: 24 * 60 * 60,
Urgency: webpush.UrgencyNormal,
})
if err != nil {
return false, err
}
defer resp.Body.Close()
if resp.StatusCode == 404 || resp.StatusCode == 410 {
return true, nil
}
if resp.StatusCode >= 400 {
return false, fmt.Errorf("push service returned %d", resp.StatusCode)
}
return false, nil
}
// disabledSourcesFor returns the set of source names a user has hidden, read
// from their stored prefs blob. The blob mirrors localStorage: a JSON object
// whose "pete.disabledSources.v1" value is itself a JSON string encoding a
// {sourceName: true} map. Any parse failure yields an empty (deny-nothing) set.
func disabledSourcesFor(sub string) map[string]bool {
out := map[string]bool{}
blob, err := storage.GetUserPrefs(sub)
if err != nil || blob == "" {
return out
}
var prefs map[string]json.RawMessage
if err := json.Unmarshal([]byte(blob), &prefs); err != nil {
return out
}
raw, ok := prefs["pete.disabledSources.v1"]
if !ok {
return out
}
// The value is normally a JSON *string* containing JSON; unwrap that first,
// but tolerate a bare object too.
inner := []byte(raw)
var asStr string
if err := json.Unmarshal(raw, &asStr); err == nil {
inner = []byte(asStr)
}
var set map[string]bool
if err := json.Unmarshal(inner, &set); err != nil {
return out
}
for name, on := range set {
if on {
out[name] = true
}
}
return out
}
// pushClient returns the shared SSRF-guarded, timeout-bounded HTTP client used
// for Web Push delivery, building it once on first use. The digest loop is a
// single goroutine, so the lazy init needs no lock.
func (s *Server) pushClient() *http.Client {
if s.pushHTTP == nil {
s.pushHTTP = safehttp.NewClient(pushSendTimeout)
}
return s.pushHTTP
}
// StartPushSender launches the digest loop if push is enabled. Safe to call
// unconditionally; it's a no-op when push is off.
func (s *Server) StartPushSender(ctx context.Context) {
if !s.cfg.Push.Enabled || s.auth == nil {
return
}
go s.runPushSender(ctx)
}

View File

@@ -0,0 +1,58 @@
package web
import (
"path/filepath"
"strings"
"testing"
"pete/internal/storage"
)
func TestDisabledSourcesFor(t *testing.T) {
storage.Close()
if err := storage.Init(filepath.Join(t.TempDir(), "push.db")); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { storage.Close() })
// The stored blob mirrors localStorage: the disabledSources value is itself a
// JSON *string* encoding a {source: true} map (see prefs.js snapshot()).
blob := `{"pete.disabledSources.v1":"{\"Feed A\":true,\"Feed B\":false}","pete.weather.loc.v1":"1000-100"}`
if err := storage.PutUserPrefs("sub-1", blob, "u", "u@x"); err != nil {
t.Fatal(err)
}
got := disabledSourcesFor("sub-1")
if !got["Feed A"] {
t.Error("Feed A should be disabled")
}
if got["Feed B"] {
t.Error("Feed B is false in prefs and must not be treated as disabled")
}
if len(got) != 1 {
t.Errorf("disabled set = %v, want just {Feed A}", got)
}
// A user with no prefs disables nothing.
if len(disabledSourcesFor("nobody")) != 0 {
t.Error("unknown user should have an empty disabled set")
}
}
func TestBuildDigestPayload(t *testing.T) {
single := string(buildDigestPayload(1, "Only Story"))
if !strings.Contains(single, "1 new story") || strings.Contains(single, "1 new stories") {
t.Errorf("singular wording wrong: %s", single)
}
if !strings.Contains(single, "Only Story") {
t.Errorf("top headline missing: %s", single)
}
many := string(buildDigestPayload(7, "Big One"))
if !strings.Contains(many, "7 new stories") || !strings.Contains(many, "Big One") {
t.Errorf("plural payload wrong: %s", many)
}
if !strings.Contains(many, `"tag":"pete-digest"`) {
t.Errorf("expected digest tag in payload: %s", many)
}
}

115
internal/web/pwa.go Normal file
View File

@@ -0,0 +1,115 @@
package web
import (
"io/fs"
"log/slog"
"net/http"
"pete/internal/safehttp"
"pete/internal/storage"
)
// maxPushBodyBytes caps a subscription payload. A PushSubscription JSON is an
// endpoint URL plus two short base64 keys — a few hundred bytes — so 4 KiB is
// generous headroom for long endpoint URLs.
const maxPushBodyBytes = 4096
// handlePushSubscribe stores the caller's Web Push subscription. The body is the
// browser's PushSubscription.toJSON() shape: {endpoint, keys:{p256dh, auth}}.
func (s *Server) handlePushSubscribe(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
if !s.cfg.Push.Enabled {
http.Error(w, `{"error":"push disabled"}`, http.StatusNotFound)
return
}
var req struct {
Endpoint string `json:"endpoint"`
Keys struct {
P256dh string `json:"p256dh"`
Auth string `json:"auth"`
} `json:"keys"`
}
if !decodeStateBodyN(w, r, &req, maxPushBodyBytes) {
return
}
if req.Endpoint == "" || req.Keys.P256dh == "" || req.Keys.Auth == "" {
http.Error(w, `{"error":"incomplete subscription"}`, http.StatusBadRequest)
return
}
// The endpoint is delivered to server-side; reject non-http(s) schemes here so
// a client can't stash a file:// or gopher:// target. The digest sender's
// SSRF-guarded client blocks non-public hosts at dial time, but keeping bad
// endpoints out of the table avoids storing garbage in the first place.
if err := safehttp.ValidateURL(req.Endpoint); err != nil {
http.Error(w, `{"error":"invalid endpoint"}`, http.StatusBadRequest)
return
}
if err := storage.AddPushSubscription(u.Sub, req.Endpoint, req.Keys.P256dh, req.Keys.Auth); err != nil {
slog.Error("push: subscribe failed", "sub", u.Sub, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handlePushUnsubscribe drops the caller's own stored subscription by endpoint.
// The delete is scoped to the signed-in user so one account can't remove
// another's subscription by presenting its endpoint string.
func (s *Server) handlePushUnsubscribe(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
var req struct {
Endpoint string `json:"endpoint"`
}
if !decodeStateBodyN(w, r, &req, maxPushBodyBytes) {
return
}
if req.Endpoint == "" {
http.Error(w, `{"error":"missing endpoint"}`, http.StatusBadRequest)
return
}
if err := storage.RemovePushSubscriptionForUser(u.Sub, req.Endpoint); err != nil {
slog.Error("push: unsubscribe failed", "sub", u.Sub, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleManifest serves the web app manifest from the embedded static tree. It
// lives at the root so the installable scope covers the whole origin.
func (s *Server) handleManifest(w http.ResponseWriter, r *http.Request) {
s.serveEmbedded(w, r, "manifest.webmanifest", "application/manifest+json; charset=utf-8", "public, max-age=3600")
}
// handleServiceWorker serves /sw.js from the root. Serving it here rather than
// under /static/ lets its scope be the whole origin (a worker's default scope
// is its own path), and we set Service-Worker-Allowed as a belt-and-braces in
// case it's ever moved. no-cache keeps updated workers from being pinned by the
// HTTP cache — the browser still byte-compares to decide whether to install.
func (s *Server) handleServiceWorker(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Service-Worker-Allowed", "/")
s.serveEmbedded(w, r, "sw.js", "text/javascript; charset=utf-8", "no-cache")
}
// serveEmbedded writes a file from the embedded static FS with explicit headers.
func (s *Server) serveEmbedded(w http.ResponseWriter, _ *http.Request, name, contentType, cacheControl string) {
sub, err := fs.Sub(staticFS, "static")
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
b, err := fs.ReadFile(sub, name)
if err != nil {
http.NotFound(w, nil)
return
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Cache-Control", cacheControl)
_, _ = w.Write(b)
}

View File

@@ -0,0 +1,95 @@
package web
import (
"encoding/json"
"net/http/httptest"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"pete/internal/config"
"pete/internal/storage"
)
// TestReaderCardDataAndArticleAPI exercises the reader-mode path end to end: a
// classified story renders a card carrying its id + headline data attributes,
// and /api/article returns the stored full text for that id.
func TestReaderCardDataAndArticleAPI(t *testing.T) {
storage.Close()
if err := storage.Init(filepath.Join(t.TempDir(), "reader.db")); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { storage.Close() })
story := &storage.Story{
GUID: "reader-e2e",
Headline: "A Distinctive Reader Headline",
Lede: "The lede.",
Content: "Opening paragraph of the piece.\n\nA second paragraph with more detail.",
ArticleURL: "https://example.com/story",
Source: "Example Wire",
Channel: "tech",
Classified: true,
SeenAt: time.Now().Unix(),
}
if err := storage.InsertStory(story); err != nil {
t.Fatal(err)
}
var id int64
if err := storage.Get().QueryRow(`SELECT id FROM stories WHERE guid = ?`, story.GUID).Scan(&id); err != nil {
t.Fatal(err)
}
s, err := New(config.WebConfig{SiteTitle: "Pete", ListenAddr: ":0"}, nil, true)
if err != nil {
t.Fatal(err)
}
// Index page renders the card with the reader data attributes.
rw := httptest.NewRecorder()
s.handleIndex(rw, httptest.NewRequest("GET", "/", nil))
body := rw.Body.String()
if rw.Code != 200 {
t.Fatalf("index status = %d", rw.Code)
}
for _, want := range []string{
`data-id="` + strconv.FormatInt(id, 10) + `"`,
`data-headline="A Distinctive Reader Headline"`,
`data-story-card`,
} {
if !strings.Contains(body, want) {
t.Errorf("index HTML missing %q", want)
}
}
// The article endpoint returns the stored content for that id.
rw2 := httptest.NewRecorder()
s.handleArticle(rw2, httptest.NewRequest("GET", "/api/article?id="+strconv.FormatInt(id, 10), nil))
if rw2.Code != 200 {
t.Fatalf("article status = %d body=%s", rw2.Code, rw2.Body.String())
}
var got struct {
Content string `json:"content"`
Lede string `json:"lede"`
}
if err := json.Unmarshal(rw2.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.Content != story.Content {
t.Errorf("content = %q, want %q", got.Content, story.Content)
}
// A bad id is a 400, an unknown id is a 404.
for _, tc := range []struct {
q string
code int
}{{"id=0", 400}, {"id=abc", 400}, {"id=999999", 404}} {
w := httptest.NewRecorder()
s.handleArticle(w, httptest.NewRequest("GET", "/api/article?"+tc.q, nil))
if w.Code != tc.code {
t.Errorf("article?%s status = %d, want %d", tc.q, w.Code, tc.code)
}
}
}

View File

@@ -52,11 +52,14 @@ type SourceInfo struct {
// Server is the HTTP server for Pete's web UI.
type Server struct {
cfg config.WebConfig
sources []SourceInfo
srv *http.Server
tpls map[string]*template.Template // keyed by page name (e.g. "index", "channel")
auth *Authenticator // nil when sign-in is disabled or unavailable
cfg config.WebConfig
sources []SourceInfo
postingEnabled bool // false = web-only mode; hides Matrix-posting UI
srv *http.Server
tpls map[string]*template.Template // keyed by page name (e.g. "index", "channel")
auth *Authenticator // nil when sign-in is disabled or unavailable
adminSubs map[string]bool // OIDC subjects allowed to view /status
pushHTTP *http.Client // SSRF-guarded client for Web Push delivery; built lazily by pushClient()
// Daily-rotated salt for the privacy-preserving unique-visitor estimate.
// Guarded by metricsMu; never persisted (see metrics.go).
@@ -68,8 +71,8 @@ type Server struct {
// New builds the server. Templates are parsed once at startup. Each page
// gets its own template set sharing layout.html + _card.html, which avoids
// `main` define collisions between pages.
func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) {
pages := []string{"index", "channel", "weather"}
func New(cfg config.WebConfig, sources []config.SourceConfig, postingEnabled bool) (*Server, error) {
pages := []string{"index", "channel", "weather", "bookmarks", "for-you", "status"}
tpls := make(map[string]*template.Template, len(pages))
for _, p := range pages {
t, err := template.New("").Funcs(funcs).ParseFS(templateFS,
@@ -89,7 +92,13 @@ func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) {
}
infos = append(infos, SourceInfo{Name: sc.Name, Channel: sc.DirectRoute})
}
s := &Server{cfg: cfg, sources: infos, tpls: tpls}
adminSubs := make(map[string]bool, len(cfg.AdminSubs))
for _, sub := range cfg.AdminSubs {
if sub != "" {
adminSubs[sub] = true
}
}
s := &Server{cfg: cfg, sources: infos, postingEnabled: postingEnabled, tpls: tpls, adminSubs: adminSubs}
// Optional OIDC sign-in (Authentik). Discovery is a network call; if the
// provider is unreachable at boot we log and serve anonymously rather than
@@ -115,14 +124,23 @@ func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) {
mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.FS(staticSub))))
mux.HandleFunc("GET /img/{name}", s.handleImg)
mux.HandleFunc("GET /manifest.webmanifest", s.handleManifest)
mux.HandleFunc("GET /sw.js", s.handleServiceWorker)
mux.HandleFunc("GET /{$}", s.handleIndex)
mux.HandleFunc("GET /weather", s.handleWeatherDemo)
mux.HandleFunc("GET /api/search", s.handleSearch)
mux.HandleFunc("GET /api/article", s.handleArticle)
mux.HandleFunc("GET /api/related", s.handleRelated)
mux.HandleFunc("GET /feed.xml", func(w http.ResponseWriter, r *http.Request) { s.handleFeedXML(w, r, "") })
mux.HandleFunc("GET /feed.json", func(w http.ResponseWriter, r *http.Request) { s.handleFeedJSON(w, r, "") })
for _, ch := range channels {
ch := ch
mux.HandleFunc("GET /"+ch.Slug, func(w http.ResponseWriter, r *http.Request) {
s.handleChannel(w, r, ch)
})
mux.HandleFunc("GET /"+ch.Slug+"/feed.xml", func(w http.ResponseWriter, r *http.Request) { s.handleFeedXML(w, r, ch.Slug) })
mux.HandleFunc("GET /"+ch.Slug+"/feed.json", func(w http.ResponseWriter, r *http.Request) { s.handleFeedJSON(w, r, ch.Slug) })
}
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
@@ -135,6 +153,16 @@ func New(cfg config.WebConfig, sources []config.SourceConfig) (*Server, error) {
mux.HandleFunc("GET /auth/logout", s.auth.handleLogout)
mux.HandleFunc("GET /api/preferences", s.handlePrefs)
mux.HandleFunc("PUT /api/preferences", s.handlePrefs)
mux.HandleFunc("POST /api/read", s.handleRead)
mux.HandleFunc("POST /api/bookmark", s.handleBookmark)
mux.HandleFunc("GET /api/state", s.handleState)
mux.HandleFunc("GET /bookmarks", s.handleBookmarks)
mux.HandleFunc("GET /for-you", s.handleForYou)
mux.HandleFunc("GET /status", s.handleStatus)
if s.cfg.Push.Enabled {
mux.HandleFunc("POST /api/push/subscribe", s.handlePushSubscribe)
mux.HandleFunc("POST /api/push/unsubscribe", s.handlePushUnsubscribe)
}
}
s.srv = &http.Server{

View File

@@ -169,3 +169,241 @@ html[data-phase="night"] {
filter: contrast(1.05);
}
}
/* ----------------------------------------------------------------------------
Feed / reader mode. A focused, one-story-at-a-time overlay driven by
reader.js. Left/right arrows page through the stories currently on screen,
marking each read as it's shown. Styling leans on the phase palette vars so
it recolours with day/night like everything else.
---------------------------------------------------------------------------- */
@layer components {
.pete-reader-backdrop {
position: absolute;
inset: 0;
background: rgba(20, 14, 6, 0.55);
backdrop-filter: blur(6px);
}
html[data-phase="night"] .pete-reader-backdrop { background: rgba(6, 8, 20, 0.65); }
.pete-reader-shell {
position: relative;
height: 100%;
width: 100%;
max-width: 44rem;
margin: 0 auto;
display: flex;
flex-direction: column;
padding: max(env(safe-area-inset-top), 0.75rem) 1rem 0.75rem;
gap: 0.75rem;
}
.pete-reader-bar {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.75rem;
color: #fff;
}
.pete-reader-progress {
font-family: "Fredoka", "Nunito", system-ui, sans-serif;
font-weight: 600;
font-size: 0.85rem;
letter-spacing: 0.02em;
text-shadow: 0 1px 2px rgba(0, 0, 0, 0.4);
}
.pete-reader-nav { display: flex; align-items: center; gap: 0.4rem; }
.pete-reader-btn {
display: inline-flex;
align-items: center;
justify-content: center;
min-width: 2.25rem;
height: 2.25rem;
padding: 0 0.7rem;
border-radius: 9999px;
background: rgba(255, 255, 255, 0.14);
color: #fff;
font-weight: 700;
font-size: 0.95rem;
border: 2px solid rgba(255, 255, 255, 0.18);
cursor: pointer;
transition: background 0.15s ease, transform 0.15s ease, opacity 0.15s ease;
}
.pete-reader-btn:hover { background: rgba(255, 255, 255, 0.28); transform: translateY(-1px); }
.pete-reader-btn:disabled { opacity: 0.35; cursor: default; transform: none; }
.pete-reader-btn-open { background: var(--accent); border-color: transparent; color: #1c1305; text-decoration: none; }
.pete-reader-btn-open:hover { background: var(--accent); filter: brightness(1.08); }
.pete-reader-scroll {
flex: 1 1 auto;
overflow-y: auto;
-webkit-overflow-scrolling: touch;
border-radius: 1.75rem;
}
.pete-reader-article {
background: var(--card);
color: var(--ink);
border-radius: 1.75rem;
border: 2px solid rgba(0, 0, 0, 0.06);
box-shadow: 0 6px 0 rgba(60, 40, 20, 0.12), 0 16px 32px rgba(60, 40, 20, 0.18);
padding: clamp(1.25rem, 4vw, 2.5rem);
}
html[data-phase="night"] .pete-reader-article { border-color: rgba(255, 255, 255, 0.08); }
.pete-reader-eyebrow {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 0.5rem;
font-size: 0.75rem;
margin-bottom: 0.85rem;
}
.pete-reader-chip {
display: inline-flex;
align-items: center;
gap: 0.35rem;
border-radius: 9999px;
padding: 0.15rem 0.65rem;
color: #fff;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.06em;
}
.pete-reader-source { font-weight: 700; opacity: 0.75; }
.pete-reader-time { opacity: 0.55; }
.pete-reader-title {
font-family: "Fredoka", "Nunito", system-ui, sans-serif;
font-weight: 700;
line-height: 1.15;
letter-spacing: -0.01em;
font-size: clamp(1.6rem, 4.5vw, 2.4rem);
margin-bottom: 1rem;
}
.pete-reader-hero {
width: 100%;
border-radius: 1.1rem;
margin-bottom: 1.25rem;
background: rgba(0, 0, 0, 0.05);
}
.pete-reader-body { font-size: 1.08rem; line-height: 1.75; }
.pete-reader-body p { margin-bottom: 1.05em; }
.pete-reader-body p:last-child { margin-bottom: 0; }
.pete-reader-note {
margin-top: 1.25rem;
padding-top: 1rem;
border-top: 1px solid rgba(0, 0, 0, 0.1);
font-size: 0.85rem;
opacity: 0.7;
}
html[data-phase="night"] .pete-reader-note { border-color: rgba(255, 255, 255, 0.12); }
.pete-reader-note a { color: var(--accent); font-weight: 700; }
.pete-reader-empty {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
text-align: center;
gap: 0.75rem;
padding: clamp(2rem, 8vw, 4rem) 1rem;
color: var(--ink);
}
.pete-reader-empty-emoji { font-size: 2.5rem; }
.pete-reader-hint {
text-align: center;
color: rgba(255, 255, 255, 0.85);
font-size: 0.75rem;
text-shadow: 0 1px 2px rgba(0, 0, 0, 0.4);
}
.pete-reader-hint kbd {
font-family: ui-monospace, monospace;
background: rgba(255, 255, 255, 0.16);
border-radius: 0.3rem;
padding: 0.05rem 0.3rem;
}
@media (max-width: 640px) {
.pete-reader-hint { display: none; }
}
/* "You might also like" rail, shown under the article in feed mode. Lives
inside the reader's scroll area, below the article card. */
.pete-reader-related { width: 100%; margin: 0.85rem auto 0; }
.pete-reader-related-title {
font-family: "Fredoka", "Nunito", system-ui, sans-serif;
font-weight: 700;
font-size: 0.95rem;
color: #fff;
margin: 0 0 0.6rem;
text-shadow: 0 1px 2px rgba(0, 0, 0, 0.4);
}
.pete-reader-related-grid { display: grid; gap: 0.6rem; }
.pete-reader-related-card {
display: flex;
align-items: center;
gap: 0.75rem;
background: var(--card);
color: var(--ink);
border-radius: 1rem;
border: 2px solid rgba(0, 0, 0, 0.06);
padding: 0.6rem;
text-decoration: none;
transition: transform 0.15s ease, box-shadow 0.15s ease;
}
.pete-reader-related-card:hover {
transform: translateY(-1px);
box-shadow: 0 6px 16px rgba(60, 40, 20, 0.16);
}
html[data-phase="night"] .pete-reader-related-card { border-color: rgba(255, 255, 255, 0.08); }
.pete-reader-related-thumb {
width: 4.5rem;
height: 3.25rem;
flex-shrink: 0;
object-fit: cover;
border-radius: 0.6rem;
background: rgba(0, 0, 0, 0.06);
}
.pete-reader-related-meta { min-width: 0; display: flex; flex-direction: column; gap: 0.25rem; }
.pete-reader-related-eyebrow {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 0.4rem;
font-size: 0.7rem;
}
.pete-reader-related-source { font-weight: 700; opacity: 0.7; }
.pete-reader-related-headline {
font-weight: 700;
line-height: 1.25;
font-size: 0.92rem;
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
/* Grid treatment for stories already read in feed mode: dimmed, with a small
corner check. Hovering restores full opacity so nothing feels lost. */
[data-story-card][data-read="1"] { opacity: 0.5; transition: opacity 0.2s ease; }
[data-story-card][data-read="1"]:hover { opacity: 1; }
[data-story-card][data-read="1"]::after {
content: "✓";
position: absolute;
top: 0.6rem;
right: 0.6rem;
z-index: 6;
display: grid;
place-items: center;
height: 1.5rem;
width: 1.5rem;
border-radius: 9999px;
background: rgba(20, 14, 6, 0.72);
color: #fff;
font-size: 0.8rem;
font-weight: 700;
}
}

File diff suppressed because one or more lines are too long

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 154 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

View File

@@ -0,0 +1,116 @@
// PWA glue: register the service worker, and (for signed-in users on a
// push-enabled server) drive the notification opt-in toggle in the settings
// panel. Anonymous visitors still get the offline reader — only the push
// controls are gated behind sign-in + a configured VAPID key.
(function () {
if (!("serviceWorker" in navigator)) return;
var CFG = window.PETE_PUSH || null; // { enabled, publicKey } or null
var reg = null;
navigator.serviceWorker.register("/sw.js").then(function (r) {
reg = r;
if (canPush()) initPushUI();
}).catch(function () {});
function canPush() {
return !!(CFG && CFG.enabled && CFG.publicKey && window.PETE_USER &&
"PushManager" in window && "Notification" in window);
}
// ---- push subscription ----------------------------------------------------
function urlBase64ToUint8Array(base64String) {
var padding = "=".repeat((4 - (base64String.length % 4)) % 4);
var base64 = (base64String + padding).replace(/-/g, "+").replace(/_/g, "/");
var raw = atob(base64);
var out = new Uint8Array(raw.length);
for (var i = 0; i < raw.length; i++) out[i] = raw.charCodeAt(i);
return out;
}
function currentSub() {
if (!reg) return Promise.resolve(null);
return reg.pushManager.getSubscription();
}
function subscribe() {
return reg.pushManager.subscribe({
userVisibleOnly: true,
applicationServerKey: urlBase64ToUint8Array(CFG.publicKey),
}).then(function (sub) {
return fetch("/api/push/subscribe", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(sub.toJSON()),
credentials: "same-origin",
}).then(function (res) {
if (!res.ok) throw new Error("subscribe rejected");
return sub;
});
});
}
function unsubscribe() {
return currentSub().then(function (sub) {
if (!sub) return;
var endpoint = sub.endpoint;
return sub.unsubscribe().then(function () {
return fetch("/api/push/unsubscribe", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ endpoint: endpoint }),
credentials: "same-origin",
}).catch(function () {});
});
});
}
// ---- settings-panel toggle ------------------------------------------------
function initPushUI() {
var slot = document.querySelector("[data-push-section]");
if (!slot) return;
slot.hidden = false;
var btn = slot.querySelector("[data-push-toggle]");
var note = slot.querySelector("[data-push-note]");
if (!btn) return;
var busy = false;
function paint(on, text) {
btn.setAttribute("aria-pressed", on ? "true" : "false");
btn.textContent = on ? "Notifications on" : "Turn on notifications";
if (note && text != null) note.textContent = text;
}
function refresh() {
if (Notification.permission === "denied") {
btn.disabled = true;
paint(false, "Notifications are blocked in your browser settings.");
return;
}
currentSub().then(function (sub) {
paint(!!sub, sub ? "You'll get a nudge when new stories land." : "Get a nudge when new stories land.");
});
}
btn.addEventListener("click", function () {
if (busy) return;
busy = true;
btn.disabled = true;
currentSub().then(function (sub) {
if (sub) return unsubscribe().then(function () { paint(false, "Notifications off."); });
return Notification.requestPermission().then(function (perm) {
if (perm !== "granted") { paint(false, "Permission denied."); return; }
return subscribe().then(function () { paint(true, "You're all set. New stories will nudge you."); });
});
}).catch(function () {
paint(false, "Something went wrong. Try again.");
}).finally(function () {
busy = false;
btn.disabled = Notification.permission === "denied";
});
});
refresh();
}
})();

View File

@@ -0,0 +1,488 @@
// Feed / reader mode. Presents the stories currently on the page one at a
// time in a focused overlay, marking each read as it's shown. Left/right arrow
// keys (or the on-screen buttons) page through them; the full article text is
// whatever Pete captured at ingest (content:encoded or the scraped body),
// fetched on demand from /api/article and cached for the session.
//
// Read state lives in localStorage under pete.read.v1. It's deliberately
// device-local — unlike the small prefs blob, the read set grows unbounded, so
// it isn't mirrored through PetePrefs to the account.
(function () {
var overlay = document.getElementById("pete-reader");
if (!overlay) return;
var READ_KEY = "pete.read.v1";
var articleEl = overlay.querySelector("[data-reader-article]");
var scrollEl = overlay.querySelector("[data-reader-scroll]");
var progressEl = overlay.querySelector("[data-reader-progress]");
var linkEl = overlay.querySelector("[data-reader-link]");
var prevBtn = overlay.querySelector("[data-reader-prev]");
var nextBtn = overlay.querySelector("[data-reader-next]");
var closeBtn = overlay.querySelector("[data-reader-close]");
var backdrop = overlay.querySelector("[data-reader-backdrop]");
var readerBookmarkBtn = overlay.querySelector("[data-reader-bookmark]");
var relatedEl = overlay.querySelector("[data-reader-related]");
var relatedCache = {}; // id -> results array
// Signed-in users (Authentik) get read + bookmark state synced server-side;
// window.PETE_USER is non-null for them. Anonymous visitors keep the
// localStorage-only behaviour and never see the bookmark controls.
var SIGNED_IN = !!(window.PETE_USER);
var bookmarkSet = Object.create(null); // id -> 1 for bookmarked stories
var items = []; // {id, url, headline, lede, image, time, source, chTitle, chEmoji, chTheme, posted, paywalled}
var index = 0;
var open = false;
var contentCache = {}; // id -> {content, lede}
var inflight = null;
// ---- read-state store -----------------------------------------------------
function loadRead() {
try {
var raw = localStorage.getItem(READ_KEY);
var obj = raw ? JSON.parse(raw) : {};
return obj && typeof obj === "object" ? obj : {};
} catch (e) { return {}; }
}
function saveRead(set) {
try { localStorage.setItem(READ_KEY, JSON.stringify(set)); } catch (e) {}
}
var readSet = loadRead();
function isRead(id) { return !!readSet[id]; }
function setRead(id, on) {
if (on) readSet[id] = 1; else delete readSet[id];
saveRead(readSet);
paintCard(id, on);
if (SIGNED_IN) postState("/api/read", { id: Number(id), read: !!on });
}
// ---- server sync (signed-in only) -----------------------------------------
function postState(url, body) {
try {
fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
credentials: "same-origin",
keepalive: true
}).catch(function () {});
} catch (e) {}
}
function isBookmarked(id) { return !!bookmarkSet[id]; }
// setBookmark updates memory, paints every matching control, and persists.
function setBookmark(id, on) {
if (on) bookmarkSet[id] = 1; else delete bookmarkSet[id];
paintBookmark(id, on);
postState("/api/bookmark", { id: Number(id), on: !!on });
// On the bookmarks page, an un-bookmark should drop the card immediately.
if (!on && location.pathname === "/bookmarks") {
document.querySelectorAll('[data-story-card][data-id="' + cssEsc(id) + '"]').forEach(function (c) {
c.parentNode && c.parentNode.removeChild(c);
});
}
}
// setBookmarkQuiet applies server-provided state without echoing it back.
function setBookmarkQuiet(id, on) {
if (on) bookmarkSet[id] = 1; else delete bookmarkSet[id];
paintBookmark(id, on);
}
function paintBookmark(id, on) {
document.querySelectorAll('[data-bookmark-btn][data-story-id="' + cssEsc(id) + '"]').forEach(function (b) {
applyCardBookmark(b, on);
});
if (readerBookmarkBtn && items[index] && String(items[index].id) === String(id)) {
applyReaderBookmark(on);
}
}
function applyCardBookmark(btn, on) {
btn.setAttribute("aria-pressed", on ? "true" : "false");
var svg = btn.querySelector("svg");
if (on) {
btn.style.background = "var(--accent)";
btn.style.color = "#1c1305";
if (svg) svg.setAttribute("fill", "currentColor");
} else {
btn.style.background = "rgba(20,14,6,.62)";
btn.style.color = "#fff";
if (svg) svg.setAttribute("fill", "none");
}
}
function applyReaderBookmark(on) {
if (!readerBookmarkBtn) return;
readerBookmarkBtn.setAttribute("aria-pressed", on ? "true" : "false");
readerBookmarkBtn.textContent = on ? "🔖 Saved" : "🔖 Save";
}
// initUserState reveals the bookmark controls and pulls the signed-in user's
// read + bookmark state for the stories on this page, painting them and
// migrating any device-local reads the account doesn't have yet.
function initUserState() {
if (!SIGNED_IN) return;
document.querySelectorAll("[data-bookmark-btn]").forEach(function (b) {
b.style.display = "inline-flex";
});
var ids = [];
document.querySelectorAll("[data-story-card]").forEach(function (c) {
var id = c.getAttribute("data-id");
if (id) ids.push(id);
});
if (!ids.length) return;
fetch("/api/state?ids=" + encodeURIComponent(ids.join(",")), { credentials: "same-origin" })
.then(function (r) { return r.ok ? r.json() : null; })
.then(function (data) {
if (!data) return;
var serverRead = Object.create(null);
(data.read || []).forEach(function (id) {
serverRead[id] = 1; readSet[id] = 1; paintCard(id, true);
});
(data.bookmarked || []).forEach(function (id) { setBookmarkQuiet(id, true); });
// Migrate device-local reads the account doesn't have yet — but only
// once per device. After the first sync the server is authoritative, so
// a story the user later marks unread on another device stays unread
// instead of being perpetually resurrected from this device's stale
// local set on every page load.
var MIGRATED_KEY = "pete.readMigrated.v1";
var migrated = false;
try { migrated = localStorage.getItem(MIGRATED_KEY) === "1"; } catch (e) {}
if (!migrated) {
ids.forEach(function (id) {
if (readSet[id] && !serverRead[id]) postState("/api/read", { id: Number(id), read: true });
});
try { localStorage.setItem(MIGRATED_KEY, "1"); } catch (e) {}
}
saveRead(readSet);
})
.catch(function () {});
}
// Reflect read state onto every matching card on the page (a story can appear
// in more than one section on the home page).
function paintCard(id, on) {
document.querySelectorAll('[data-story-card][data-id="' + cssEsc(id) + '"]').forEach(function (c) {
if (on) c.setAttribute("data-read", "1");
else c.removeAttribute("data-read");
});
}
function cssEsc(s) {
return String(s).replace(/["\\]/g, "\\$&");
}
// ---- collecting the page's stories ----------------------------------------
function isVisible(el) {
return window.getComputedStyle(el).display !== "none";
}
function collect() {
var seen = Object.create(null);
var out = [];
document.querySelectorAll("[data-story-card]").forEach(function (c) {
var id = c.getAttribute("data-id");
if (!id || seen[id] || !isVisible(c)) return;
var url = c.getAttribute("data-url") || c.getAttribute("href") || "";
if (!/^https?:\/\//i.test(url)) url = "";
seen[id] = true;
out.push({
id: id,
url: url,
headline: c.getAttribute("data-headline") || "",
lede: c.getAttribute("data-lede") || "",
image: c.getAttribute("data-image") || "",
time: c.getAttribute("data-time") || "",
source: c.getAttribute("data-source") || "",
chTitle: c.getAttribute("data-ch-title") || "",
chEmoji: c.getAttribute("data-ch-emoji") || "",
chTheme: c.getAttribute("data-ch-theme") || "",
paywalled: c.getAttribute("data-paywalled") === "1"
});
});
return out;
}
// ---- rendering ------------------------------------------------------------
function escapeHTML(s) {
return String(s == null ? "" : s)
.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;")
.replace(/"/g, "&quot;").replace(/'/g, "&#39;");
}
function safeURL(s) {
var raw = String(s == null ? "" : s).trim();
return /^https?:\/\//i.test(raw) ? raw : "";
}
// Turn stored plain text (paragraphs separated by blank lines) into escaped
// <p> blocks. Single newlines inside a paragraph become spaces.
function paragraphs(text) {
var blocks = String(text || "").split(/\n{2,}/);
var html = "";
for (var i = 0; i < blocks.length; i++) {
var t = blocks[i].replace(/\s*\n\s*/g, " ").trim();
if (t) html += "<p>" + escapeHTML(t) + "</p>";
}
return html;
}
function eyebrow(it) {
var chip = it.chTheme
? '<span class="pete-reader-chip bg-theme-' + escapeHTML(it.chTheme) + '">' +
(it.chEmoji ? '<span aria-hidden="true">' + escapeHTML(it.chEmoji) + "</span>" : "") +
escapeHTML(it.chTitle) + "</span>"
: "";
var src = it.source ? '<span class="pete-reader-source">' + escapeHTML(it.source) + "</span>" : "";
var time = it.time ? '<span class="pete-reader-time">' + escapeHTML(it.time) + "</span>" : "";
return '<div class="pete-reader-eyebrow">' + chip + src + time + "</div>";
}
function renderBody(it, bodyHTML) {
var hero = it.image
? '<img class="pete-reader-hero" src="' + escapeHTML(it.image) + '" alt="" loading="lazy" decoding="async">'
: "";
var href = safeURL(it.url);
var note = '<p class="pete-reader-note">' +
(it.paywalled ? "This source is paywalled, so the text above may be partial. " : "") +
(href ? 'Read it at the source: <a href="' + escapeHTML(href) + '" target="_blank" rel="noopener noreferrer">' +
escapeHTML(it.source || "original article") + " ↗</a>." : "") +
"</p>";
articleEl.innerHTML =
eyebrow(it) +
'<h1 class="pete-reader-title">' + escapeHTML(it.headline) + "</h1>" +
hero +
'<div class="pete-reader-body">' + bodyHTML + "</div>" +
note;
}
function loadingBody(it) {
renderBody(it, '<p style="opacity:.55">Loading the full story…</p>');
}
function fillContent(it, data) {
var text = (data && data.content) ? data.content : ((data && data.lede) || it.lede);
var html = paragraphs(text);
if (!html) {
html = '<p style="opacity:.7">No article text was captured for this story. Open the original to read it.</p>';
}
renderBody(it, html);
}
function fetchContent(it) {
if (contentCache[it.id]) { fillContent(it, contentCache[it.id]); return; }
loadingBody(it);
if (inflight) inflight.abort();
var ctrl = new AbortController();
inflight = ctrl;
var reqId = it.id;
fetch("/api/article?id=" + encodeURIComponent(it.id), { signal: ctrl.signal, credentials: "same-origin" })
.then(function (r) { if (!r.ok) throw new Error("status " + r.status); return r.json(); })
.then(function (data) {
contentCache[reqId] = data;
if (ctrl !== inflight) return; // superseded by a newer navigation
if (items[index] && items[index].id === reqId) fillContent(it, data);
})
.catch(function (err) {
if (err.name === "AbortError") return;
if (items[index] && items[index].id === reqId) fillContent(it, null); // fall back to lede
})
.finally(function () { if (ctrl === inflight) inflight = null; });
}
// ---- related ("you might also like") --------------------------------------
function clearRelated() {
if (!relatedEl) return;
relatedEl.innerHTML = "";
relatedEl.hidden = true;
}
function renderRelated(reqId, results) {
if (!relatedEl) return;
// Ignore a response that arrived after the user moved on.
if (!items[index] || String(items[index].id) !== String(reqId)) return;
if (!results || !results.length) { clearRelated(); return; }
var html = '<h2 class="pete-reader-related-title">You might also like</h2>' +
'<div class="pete-reader-related-grid">';
for (var i = 0; i < results.length; i++) {
var it = results[i];
var href = safeURL(it.article_url);
if (!href) continue;
var chip = it.channel_theme
? '<span class="pete-reader-chip bg-theme-' + escapeHTML(it.channel_theme) + '">' +
(it.channel_emoji ? '<span aria-hidden="true">' + escapeHTML(it.channel_emoji) + "</span>" : "") +
escapeHTML(it.channel_title) + "</span>"
: "";
var thumb = it.thumb_url
? '<img class="pete-reader-related-thumb" src="' + escapeHTML(it.thumb_url) + '" alt="" loading="lazy" decoding="async">'
: '<div class="pete-reader-related-thumb pete-reader-related-thumb-empty"></div>';
html += '<a class="pete-reader-related-card" href="' + escapeHTML(href) + '" target="_blank" rel="noopener noreferrer">' +
thumb +
'<div class="pete-reader-related-meta">' +
'<div class="pete-reader-related-eyebrow">' + chip +
(it.source ? '<span class="pete-reader-related-source">' + escapeHTML(it.source) + "</span>" : "") +
"</div>" +
'<div class="pete-reader-related-headline">' + escapeHTML(it.headline) + "</div>" +
"</div></a>";
}
html += "</div>";
relatedEl.innerHTML = html;
relatedEl.hidden = false;
}
function fetchRelated(it) {
if (!relatedEl) return;
clearRelated();
var reqId = it.id;
if (relatedCache[reqId]) { renderRelated(reqId, relatedCache[reqId]); return; }
fetch("/api/related?id=" + encodeURIComponent(it.id), { credentials: "same-origin" })
.then(function (r) { return r.ok ? r.json() : null; })
.then(function (data) {
var results = (data && data.results) || [];
relatedCache[reqId] = results;
renderRelated(reqId, results);
})
.catch(function () {});
}
// ---- navigation -----------------------------------------------------------
function show(i) {
index = i;
if (index >= items.length) { renderDone(); return; }
var it = items[index];
progressEl.textContent = (index + 1) + " / " + items.length;
var href = safeURL(it.url);
if (href) { linkEl.href = href; linkEl.style.display = ""; }
else linkEl.style.display = "none";
prevBtn.disabled = index === 0;
nextBtn.disabled = false;
nextBtn.textContent = index === items.length - 1 ? "done ✓" : "→";
if (scrollEl) scrollEl.scrollTop = 0;
if (readerBookmarkBtn) {
readerBookmarkBtn.style.display = SIGNED_IN ? "" : "none";
applyReaderBookmark(isBookmarked(it.id));
}
fetchContent(it);
fetchRelated(it);
setRead(it.id, true); // presenting a story marks it read
}
function renderDone() {
clearRelated();
progressEl.textContent = items.length + " / " + items.length;
linkEl.style.display = "none";
prevBtn.disabled = items.length === 0;
nextBtn.disabled = true;
nextBtn.textContent = "→";
if (scrollEl) scrollEl.scrollTop = 0;
articleEl.innerHTML =
'<div class="pete-reader-empty">' +
'<span class="pete-reader-empty-emoji" aria-hidden="true">🎉</span>' +
'<h1 class="pete-reader-title" style="margin:0">You\'re all caught up</h1>' +
'<p style="opacity:.7;max-width:28rem">That\'s every story on this page. ' +
"Head back to browse more, or press ← to revisit the last one.</p>" +
"</div>";
}
function next() { if (index < items.length) show(index + 1); }
function prev() { if (index > 0) show(index - 1); }
function firstUnread() {
for (var i = 0; i < items.length; i++) if (!isRead(items[i].id)) return i;
return 0;
}
// ---- open / close ---------------------------------------------------------
function openReader() {
items = collect();
if (items.length === 0) return;
open = true;
overlay.classList.remove("hidden");
document.body.classList.add("overflow-hidden");
show(firstUnread());
}
function closeReader() {
open = false;
if (inflight) { inflight.abort(); inflight = null; }
clearRelated();
overlay.classList.add("hidden");
document.body.classList.remove("overflow-hidden");
}
// ---- wiring ---------------------------------------------------------------
document.addEventListener("DOMContentLoaded", function () {
// Dim stories already read on this device.
document.querySelectorAll("[data-story-card]").forEach(function (c) {
var id = c.getAttribute("data-id");
if (id && isRead(id)) c.setAttribute("data-read", "1");
});
document.querySelectorAll("[data-reader-open]").forEach(function (b) {
b.addEventListener("click", function (e) { e.preventDefault(); openReader(); });
});
if (prevBtn) prevBtn.addEventListener("click", prev);
if (nextBtn) nextBtn.addEventListener("click", next);
if (closeBtn) closeBtn.addEventListener("click", closeReader);
if (backdrop) backdrop.addEventListener("click", closeReader);
if (readerBookmarkBtn) readerBookmarkBtn.addEventListener("click", function () {
var it = items[index];
if (it) setBookmark(it.id, !isBookmarked(it.id));
});
initUserState();
});
// Bookmark buttons live inside the card's <a>; intercept so a tap toggles the
// bookmark instead of following the link. Delegated so it also covers cards
// that are added or removed after load.
document.addEventListener("click", function (e) {
var btn = e.target.closest && e.target.closest("[data-bookmark-btn]");
if (!btn) return;
e.preventDefault();
e.stopPropagation();
var id = btn.getAttribute("data-story-id");
if (id) setBookmark(id, !isBookmarked(id));
});
document.addEventListener("keydown", function (e) {
if (e.key !== "Enter" && e.key !== " ") return;
var btn = e.target.closest && e.target.closest("[data-bookmark-btn]");
if (!btn) return;
e.preventDefault();
var id = btn.getAttribute("data-story-id");
if (id) setBookmark(id, !isBookmarked(id));
});
document.addEventListener("keydown", function (e) {
// Enter feed mode with `f` when nothing is focused and no other overlay is up.
if (!open && (e.key === "f" || e.key === "F") && !e.metaKey && !e.ctrlKey && !e.altKey) {
var tag = (document.activeElement && document.activeElement.tagName) || "";
if (tag === "INPUT" || tag === "TEXTAREA") return;
var searchOpen = document.getElementById("pete-search");
if (searchOpen && !searchOpen.classList.contains("hidden")) return;
e.preventDefault();
openReader();
return;
}
if (!open) return;
switch (e.key) {
case "ArrowRight": case "l": case " ": e.preventDefault(); next(); break;
case "ArrowLeft": case "h": e.preventDefault(); prev(); break;
case "Escape": e.preventDefault(); closeReader(); break;
case "o": case "Enter": {
var it = items[index];
var href = it && safeURL(it.url);
if (href) { e.preventDefault(); window.open(href, "_blank", "noopener"); }
break;
}
case "u": {
var cur = items[index];
if (cur) setRead(cur.id, false); // let the user undo an accidental read
break;
}
case "b": case "B": {
if (!SIGNED_IN) break;
var it = items[index];
if (it) { e.preventDefault(); setBookmark(it.id, !isBookmarked(it.id)); }
break;
}
}
});
})();

View File

@@ -0,0 +1,22 @@
{
"name": "Pete — friendly news",
"short_name": "Pete",
"description": "A calm, read-one-at-a-time news reader. Bookmarks, a personalized feed, and offline reading.",
"id": "/",
"start_url": "/?source=pwa",
"scope": "/",
"display": "standalone",
"orientation": "portrait-primary",
"background_color": "#fbf3e3",
"theme_color": "#fbf3e3",
"categories": ["news", "productivity"],
"icons": [
{ "src": "/static/img/icon-192.png", "sizes": "192x192", "type": "image/png", "purpose": "any" },
{ "src": "/static/img/icon-512.png", "sizes": "512x512", "type": "image/png", "purpose": "any" },
{ "src": "/static/img/maskable-512.png", "sizes": "512x512", "type": "image/png", "purpose": "maskable" }
],
"shortcuts": [
{ "name": "For you", "short_name": "For you", "url": "/for-you", "description": "Your personalized feed" },
{ "name": "Bookmarks", "short_name": "Saved", "url": "/bookmarks", "description": "Stories you saved" }
]
}

193
internal/web/static/sw.js Normal file
View File

@@ -0,0 +1,193 @@
// Pete's service worker: an installable-PWA shell, an offline reader, and the
// Web Push receiver. Served from the root (/sw.js) so its scope is the whole
// origin — it can intercept navigations and /api/article the same as any page.
//
// Bump CACHE_VERSION whenever the precached shell assets change; activate()
// drops every cache that doesn't match the current version.
var CACHE_VERSION = "v3";
var SHELL_CACHE = "pete-shell-" + CACHE_VERSION;
var RUNTIME_CACHE = "pete-runtime-" + CACHE_VERSION;
// App shell: the static assets every page needs. Versioned by URL content only
// loosely, so we lean on network-first for HTML and cache-first for these.
var SHELL_ASSETS = [
"/static/css/output.css",
"/static/js/prefs.js",
"/static/js/weather.js",
"/static/js/weather-forecast.js",
"/static/js/search.js",
"/static/js/settings.js",
"/static/js/reader.js",
"/static/js/pwa.js",
"/static/img/pete.avif",
"/static/img/icon-192.png",
"/static/img/icon-512.png",
];
// How many visited-article responses to keep for offline reading before the
// oldest are evicted. Reader articles are small JSON blobs.
var RUNTIME_MAX = 60;
self.addEventListener("install", function (event) {
event.waitUntil(
caches.open(SHELL_CACHE).then(function (cache) {
// addAll is atomic-ish: if one asset 404s the whole install fails, so keep
// this list to assets we know are served. Individual failures are tolerated
// by falling back to per-asset puts.
return Promise.all(
SHELL_ASSETS.map(function (url) {
return cache.add(url).catch(function () {});
})
);
}).then(function () {
return self.skipWaiting();
})
);
});
self.addEventListener("activate", function (event) {
event.waitUntil(
caches.keys().then(function (keys) {
return Promise.all(
keys.map(function (k) {
if (k !== SHELL_CACHE && k !== RUNTIME_CACHE) return caches.delete(k);
})
);
}).then(function () {
return self.clients.claim();
})
);
});
// trimCache evicts the oldest entries once a runtime cache passes its cap.
function trimCache(cacheName, max) {
caches.open(cacheName).then(function (cache) {
cache.keys().then(function (keys) {
if (keys.length <= max) return;
for (var i = 0; i < keys.length - max; i++) cache.delete(keys[i]);
});
});
}
// A minimal offline page for navigations we have nothing cached for.
function offlineFallback() {
return new Response(
"<!doctype html><meta charset=utf-8><meta name=viewport content='width=device-width,initial-scale=1'>" +
"<title>Offline · Pete</title>" +
"<div style=\"font-family:system-ui,sans-serif;max-width:32rem;margin:20vh auto;padding:0 1.5rem;text-align:center;color:#3a2f1a\">" +
"<div style=font-size:3rem>🦆</div>" +
"<h1 style=font-size:1.4rem>You're offline</h1>" +
"<p style=opacity:.7>Pete can't reach the news right now. Articles you've already opened are still readable, so head back and try one of those.</p>" +
"</div>",
{ headers: { "Content-Type": "text/html; charset=utf-8" }, status: 503 }
);
}
self.addEventListener("fetch", function (event) {
var req = event.request;
if (req.method !== "GET") return;
var url = new URL(req.url);
if (url.origin !== self.location.origin) return; // never touch cross-origin
// Visited articles: network-first so text stays fresh, but cache every success
// so the reader still works offline for stories the user has already opened.
if (url.pathname === "/api/article") {
event.respondWith(
fetch(req).then(function (res) {
if (res && res.ok) {
var copy = res.clone();
caches.open(RUNTIME_CACHE).then(function (cache) {
cache.put(req, copy);
trimCache(RUNTIME_CACHE, RUNTIME_MAX);
});
}
return res;
}).catch(function () {
return caches.match(req).then(function (hit) {
return hit || new Response(JSON.stringify({ error: "offline" }), {
status: 503,
headers: { "Content-Type": "application/json" },
});
});
})
);
return;
}
// Static assets: cache-first (they're versioned by deploy), fill the cache on
// first miss so a later offline visit has them.
if (url.pathname.indexOf("/static/") === 0) {
event.respondWith(
caches.match(req).then(function (hit) {
return hit || fetch(req).then(function (res) {
if (res && res.ok) {
var copy = res.clone();
caches.open(SHELL_CACHE).then(function (cache) { cache.put(req, copy); });
}
return res;
});
})
);
return;
}
// Page navigations: network-only, falling back to the offline card when the
// network is unreachable. We deliberately do NOT cache HTML responses: pages
// are personalized (they embed the signed-in user's name/email and a "For you"
// rail), and the runtime cache is shared across everyone who uses this
// installed PWA. Caching a navigation would let a signed-out visitor — or a
// second person on the same device — be served the previous user's identity
// and personalized stories offline. Offline reading still works: the reader
// fetches cached /api/article JSON on top of the cached static shell.
if (req.mode === "navigate") {
event.respondWith(
fetch(req).catch(function () {
return offlineFallback();
})
);
return;
}
// Everything else (other /api/* calls): straight to the network. These are
// personalized/stateful and must not be served stale.
});
// ---- Web Push -------------------------------------------------------------
// The server sends a JSON payload {title, body, url, tag}. Missing fields fall
// back to sensible defaults so a malformed push still shows something useful.
self.addEventListener("push", function (event) {
var data = {};
if (event.data) {
try { data = event.data.json(); } catch (e) { data = { body: event.data.text() }; }
}
var title = data.title || "Pete";
var options = {
body: data.body || "New stories are waiting.",
icon: "/static/img/icon-192.png",
badge: "/static/img/icon-192.png",
tag: data.tag || "pete-digest",
renotify: true,
data: { url: data.url || "/" },
};
event.waitUntil(self.registration.showNotification(title, options));
});
self.addEventListener("notificationclick", function (event) {
event.notification.close();
var target = (event.notification.data && event.notification.data.url) || "/";
event.waitUntil(
self.clients.matchAll({ type: "window", includeUncontrolled: true }).then(function (clients) {
for (var i = 0; i < clients.length; i++) {
var c = clients[i];
// Focus an existing Pete tab and route it to the target if we can.
if ("focus" in c) {
c.focus();
if ("navigate" in c && target !== "/") { try { c.navigate(target); } catch (e) {} }
return;
}
}
if (self.clients.openWindow) return self.clients.openWindow(target);
})
);
});

128
internal/web/status.go Normal file
View File

@@ -0,0 +1,128 @@
package web
import (
"log/slog"
"net/http"
"sort"
"time"
"pete/internal/storage"
)
// isAdmin reports whether the request carries a signed-in session whose OIDC
// subject is on the admin allowlist. False when auth is off, the allowlist is
// empty, or the visitor is anonymous.
func (s *Server) isAdmin(r *http.Request) bool {
if s.auth == nil || len(s.adminSubs) == 0 {
return false
}
u := s.auth.userFromRequest(r)
if u == nil {
return false
}
return s.adminSubs[u.Sub]
}
// sourceStatus is one row of the source-health dashboard: the configured feed
// plus its persisted poll health and derived content stats.
type sourceStatus struct {
Name string
Channel string
Healthy bool // last poll succeeded (no consecutive failures)
NeverRun bool // no poll recorded yet
LastPollAt time.Time
LastSuccessAt time.Time
LastError string
Failures int
LastItemCount int
Total int
Classified int
Paywalled int
PaywallRate int // percent of retained stories that are gated
LastSeenAt time.Time
LastPostedAt time.Time
}
type statusPage struct {
pageData
Sources []sourceStatus
DegradedCnt int // sources currently failing
}
// handleStatus renders the owner-facing source-health dashboard. Access is
// restricted to admin subjects; everyone else gets a 404 so the page's
// existence isn't advertised.
func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) {
if !s.isAdmin(r) {
http.NotFound(w, r)
return
}
s.track(r, "status")
health, err := storage.ListSourceHealth()
if err != nil {
slog.Error("web: source health query failed", "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
stats, err := storage.SourceContentStats()
if err != nil {
slog.Error("web: source content stats failed", "err", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
rows := make([]sourceStatus, 0, len(s.sources))
degraded := 0
for _, src := range s.sources {
h, hasHealth := health[src.Name]
st := stats[src.Name]
row := sourceStatus{
Name: src.Name,
Channel: src.Channel,
NeverRun: !hasHealth || h.LastPollAt == 0,
LastError: h.LastError,
Failures: h.ConsecutiveFailures,
LastItemCount: h.LastItemCount,
Total: st.Total,
Classified: st.Classified,
Paywalled: st.Paywalled,
}
row.Healthy = hasHealth && h.ConsecutiveFailures == 0
if h.LastPollAt > 0 {
row.LastPollAt = time.Unix(h.LastPollAt, 0)
}
if h.LastSuccessAt > 0 {
row.LastSuccessAt = time.Unix(h.LastSuccessAt, 0)
}
if st.LastSeenAt > 0 {
row.LastSeenAt = time.Unix(st.LastSeenAt, 0)
}
if st.LastPostedAt > 0 {
row.LastPostedAt = time.Unix(st.LastPostedAt, 0)
}
if st.Total > 0 {
row.PaywallRate = st.Paywalled * 100 / st.Total
}
if !row.NeverRun && !row.Healthy {
degraded++
}
rows = append(rows, row)
}
// Failing sources first (most consecutive failures), then healthy ones by
// name, so the owner's eye lands on what needs attention.
sort.SliceStable(rows, func(i, j int) bool {
if rows[i].Failures != rows[j].Failures {
return rows[i].Failures > rows[j].Failures
}
return rows[i].Name < rows[j].Name
})
base := s.base(r)
base.Active = "status"
s.render(w, "status", statusPage{pageData: base, Sources: rows, DegradedCnt: degraded})
}

View File

@@ -0,0 +1,38 @@
package web
import (
"strings"
"testing"
"time"
"pete/internal/config"
)
func TestStatusTemplateExecutes(t *testing.T) {
s, err := New(config.WebConfig{SiteTitle: "Pete", ListenAddr: ":0"}, nil, true)
if err != nil {
t.Fatal(err)
}
data := statusPage{
pageData: pageData{SiteTitle: "Pete", Channels: channels},
DegradedCnt: 1,
Sources: []sourceStatus{
{Name: "Broken Feed", Channel: "tech", Failures: 3, LastError: "dial tcp: timeout",
LastPollAt: time.Now(), NeverRun: false, Healthy: false, Total: 4, Paywalled: 2, PaywallRate: 50},
{Name: "Good Feed", Channel: "eu", Healthy: true, LastPollAt: time.Now(),
LastSuccessAt: time.Now(), LastItemCount: 12, Total: 30, Classified: 28,
LastSeenAt: time.Now(), LastPostedAt: time.Now()},
{Name: "Idle Feed", Channel: "music", NeverRun: true},
},
}
var b strings.Builder
if err := s.tpls["status"].ExecuteTemplate(&b, "layout", data); err != nil {
t.Fatal(err)
}
out := b.String()
for _, want := range []string{"Broken Feed", "Good Feed", "dial tcp: timeout", "1 degraded", "Source health"} {
if !strings.Contains(out, want) {
t.Errorf("rendered status page missing %q", want)
}
}
}

View File

@@ -1,7 +1,23 @@
{{define "card"}}
{{define "card"}}{{$ch := channel .Story.Channel}}
<a href="{{.Story.ArticleURL}}" target="_blank" rel="noopener noreferrer"
data-story-card data-source="{{.Story.Source}}" data-channel="{{.Story.Channel}}"
data-id="{{.Story.ID}}"
data-headline="{{.Story.Headline}}"
data-lede="{{.Story.Lede}}"
data-url="{{.Story.ArticleURL}}"
data-image="{{if .Story.ImageURL}}{{thumb .Story.ImageURL}}{{end}}"
data-time="{{timeAgo .Story.SeenAt}}"
data-ch-title="{{$ch.Title}}" data-ch-emoji="{{$ch.Emoji}}" data-ch-theme="{{$ch.Theme}}"
data-posted="{{if .Story.Posted}}1{{end}}"
data-paywalled="{{if .Story.Paywalled}}1{{end}}"
class="group relative block rounded-3xl bg-[color:var(--card)] border-2 {{if .Story.Posted}}border-theme-{{.Theme}} glow-theme-{{.Theme}}{{else}}border-[color:var(--ink)]/10{{end}} shadow-pete overflow-hidden hover:-translate-y-1 hover:shadow-pete-lg transition">
<span role="button" tabindex="0" data-bookmark-btn data-story-id="{{.Story.ID}}"
aria-label="Bookmark this story" aria-pressed="false"
style="display:none;position:absolute;top:.6rem;left:.6rem;z-index:6;height:1.9rem;width:1.9rem;align-items:center;justify-content:center;border-radius:9999px;background:rgba(20,14,6,.62);color:#fff;-webkit-backdrop-filter:blur(2px);backdrop-filter:blur(2px)">
<svg viewBox="0 0 24 24" width="15" height="15" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<path d="M6 2h12a1 1 0 0 1 1 1v18l-7-4-7 4V3a1 1 0 0 1 1-1z"></path>
</svg>
</span>
{{if .Story.ImageURL}}
<div class="aspect-[16/10] w-full overflow-hidden bg-[color:var(--ink)]/5">
<img src="{{thumb .Story.ImageURL}}" alt="" loading="lazy" decoding="async"

View File

@@ -0,0 +1,37 @@
{{define "title"}}Bookmarks — {{.SiteTitle}}{{end}}
{{define "main"}}
<section class="mt-2 mb-8">
<div class="rounded-3xl bg-[color:var(--accent)] text-[#1c1305] p-6 sm:p-10 shadow-pete relative overflow-hidden">
<div class="absolute -top-6 -right-6 text-[12rem] opacity-20 select-none" aria-hidden="true">🔖</div>
<div class="relative">
<p class="text-sm uppercase tracking-[0.2em] opacity-70">saved for later</p>
<h1 class="font-display text-4xl sm:text-5xl font-bold mt-1">Bookmarks</h1>
<p class="mt-2 max-w-xl opacity-80">Stories you've tucked away. Tap the bookmark on any card to add or remove one.</p>
<p class="mt-4 text-xs uppercase tracking-wider opacity-75">{{.Total}} saved</p>
</div>
</div>
</section>
{{if .Stories}}
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 gap-5">
{{range .Stories}}
{{template "card" dict "Story" . "Theme" .Channel "ShowChannel" true}}
{{end}}
</div>
<nav class="mt-10 flex items-center justify-between">
{{if .HasPrev}}
<a href="{{.PrevURL}}" class="rounded-full bg-[color:var(--card)] border-2 border-[color:var(--ink)]/10 px-5 py-2 font-semibold shadow-pete hover:-translate-y-0.5 transition">← newer</a>
{{else}}<span></span>{{end}}
<span class="text-sm text-[color:var(--ink)]/60">page {{.Page}}</span>
{{if .HasNext}}
<a href="{{.NextURL}}" class="rounded-full bg-[color:var(--accent)] text-[#1c1305] px-5 py-2 font-semibold shadow-pete hover:-translate-y-0.5 transition">older →</a>
{{else}}<span></span>{{end}}
</nav>
{{else}}
<div class="rounded-3xl bg-[color:var(--card)] border-2 border-dashed border-[color:var(--ink)]/15 p-10 text-center text-[color:var(--ink)]/60">
Nothing saved yet. Browse the feed and tap the 🔖 on a story to keep it here.
</div>
{{end}}
{{end}}

View File

@@ -0,0 +1,26 @@
{{define "title"}}For you — {{.SiteTitle}}{{end}}
{{define "main"}}
<section class="mt-2 mb-8">
<div class="rounded-3xl bg-[color:var(--accent)] text-[#1c1305] p-6 sm:p-10 shadow-pete relative overflow-hidden">
<div class="absolute -top-6 -right-6 text-[12rem] opacity-20 select-none" aria-hidden="true"></div>
<div class="relative">
<p class="text-sm uppercase tracking-[0.2em] opacity-70">picked for you</p>
<h1 class="font-display text-4xl sm:text-5xl font-bold mt-1">For you</h1>
<p class="mt-2 max-w-xl opacity-80">Fresh stories weighted toward the channels and sources you read and bookmark most. The more you read, the sharper this gets.</p>
</div>
</div>
</section>
{{if .Stories}}
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 gap-5">
{{range .Stories}}
{{template "card" dict "Story" . "Theme" .Channel "ShowChannel" true}}
{{end}}
</div>
{{else}}
<div class="rounded-3xl bg-[color:var(--card)] border-2 border-dashed border-[color:var(--ink)]/15 p-10 text-center text-[color:var(--ink)]/60">
Nothing to tune yet. Read a few stories or bookmark the ones you like, and Pete will start building a feed around them.
</div>
{{end}}
{{end}}

View File

@@ -6,14 +6,34 @@
a friendlier news feed.
</h1>
<p class="mx-auto mt-3 max-w-xl text-base sm:text-lg text-[color:var(--ink)]/70">
{{if .PostingEnabled}}
Pete reads RSS, sorts stories, and posts them to Matrix.
Glowing cards are the ones he's posted.
{{else}}
Pete reads RSS and sorts stories into channels, fresh from his feeds.
{{end}}
</p>
</section>
<section data-weather-card class="mb-10 empty:hidden"></section>
{{if .JustPosted}}
{{if .ForYou}}
<section class="mb-10">
<div class="flex items-end justify-between gap-3 mb-4">
<h2 class="font-display text-xl sm:text-2xl font-bold">
<span aria-hidden="true"></span> For you
</h2>
<a href="/for-you" class="text-xs font-semibold text-[color:var(--ink)]/60 hover:text-[color:var(--ink)]">more →</a>
</div>
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-4">
{{range .ForYou}}
{{template "card" dict "Story" . "Theme" .Channel "ShowChannel" true}}
{{end}}
</div>
</section>
{{end}}
{{if and .PostingEnabled .JustPosted}}
<section class="mb-10">
<div class="flex items-end justify-between gap-3 mb-4">
<h2 class="font-display text-xl sm:text-2xl font-bold">
@@ -34,7 +54,7 @@
<h2 class="font-display text-xl sm:text-2xl font-bold">
<span aria-hidden="true">📊</span> Channels
</h2>
<span class="text-xs text-[color:var(--ink)]/50">last 24 hours</span>
<span class="text-xs text-[color:var(--ink)]/50">{{if .PostingEnabled}}last 24 hours{{else}}what Pete's tracking{{end}}</span>
</div>
<div class="grid grid-cols-2 lg:grid-cols-4 gap-3 sm:gap-4">
{{range .Stats}}
@@ -47,6 +67,7 @@
<span class="font-display text-lg font-semibold">{{.Channel.Title}}</span>
</div>
<dl class="mt-3 space-y-1 text-xs text-[color:var(--ink)]/70">
{{if $.PostingEnabled}}
<div class="flex justify-between gap-2">
<dt>last post</dt>
<dd class="font-semibold text-[color:var(--ink)]">
@@ -57,6 +78,7 @@
<dt>posted 24h</dt>
<dd class="font-semibold text-[color:var(--ink)]">{{.PostsToday}}</dd>
</div>
{{end}}
<div class="flex justify-between gap-2">
<dt>stories</dt>
<dd class="font-semibold text-[color:var(--ink)]">{{.Total}}</dd>

View File

@@ -9,6 +9,17 @@
<link href="https://fonts.googleapis.com/css2?family=Fredoka:wght@400;500;600;700&family=Nunito:wght@400;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="/static/css/output.css">
<link rel="icon" href="/static/img/pete.avif" type="image/avif">
<link rel="manifest" href="/manifest.webmanifest">
<meta name="theme-color" content="#fbf3e3">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="default">
<meta name="apple-mobile-web-app-title" content="Pete">
<link rel="apple-touch-icon" href="/static/img/apple-touch-icon.png">
<link rel="alternate" type="application/rss+xml" title="{{.SiteTitle}} — all stories" href="/feed.xml">
<link rel="alternate" type="application/feed+json" title="{{.SiteTitle}} — all stories" href="/feed.json">
{{if .Active}}{{if ne .Active "bookmarks"}}
<link rel="alternate" type="application/rss+xml" title="{{.SiteTitle}} — {{.Active}}" href="/{{.Active}}/feed.xml">
{{end}}{{end}}
<script>
// Pick a palette phase from the browser clock and update it each minute.
(function () {
@@ -49,6 +60,16 @@
</svg>
<span class="sr-only">Feed settings</span>
</button>
<button type="button" data-reader-open
title="Feed mode — read one story at a time (press f)"
class="inline-flex shrink-0 items-center justify-center rounded-full bg-[color:var(--card)] p-2 shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round" class="h-5 w-5">
<path d="M2 4h6a3 3 0 0 1 3 3v13a2.5 2.5 0 0 0-2.5-2.5H2z"></path>
<path d="M22 4h-6a3 3 0 0 0-3 3v13a2.5 2.5 0 0 1 2.5-2.5H22z"></path>
</svg>
<span class="sr-only">Feed mode</span>
</button>
{{if .Weather.Variant}}
<button type="button" data-weather-toggle aria-pressed="true"
title="Toggle weather animation"
@@ -69,6 +90,26 @@
class="hidden shrink-0 items-center gap-1.5 rounded-full bg-[color:var(--card)] px-3 py-2 text-sm font-semibold shadow-pete border-2 border-[color:var(--ink)]/10 tabular-nums"></span>
{{if .AuthEnabled}}
{{if .User}}
<a href="/bookmarks" data-bookmarks-link
title="Your bookmarks"
class="inline-flex shrink-0 items-center justify-center rounded-full bg-[color:var(--card)] p-2 shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition{{if eq .Active "bookmarks"}} bg-[color:var(--accent)]/20{{end}}">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round" class="h-5 w-5">
<path d="M6 2h12a1 1 0 0 1 1 1v18l-7-4-7 4V3a1 1 0 0 1 1-1z"></path>
</svg>
<span class="sr-only">Bookmarks</span>
</a>
{{if .IsAdmin}}
<a href="/status" data-status-link
title="Source health"
class="inline-flex shrink-0 items-center justify-center rounded-full bg-[color:var(--card)] p-2 shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition{{if eq .Active "status"}} bg-[color:var(--accent)]/20{{end}}">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round" class="h-5 w-5">
<path d="M3 12h4l2 6 4-14 2 8h6"></path>
</svg>
<span class="sr-only">Source health</span>
</a>
{{end}}
<a href="/auth/logout" data-account
title="Signed in as {{.User.Display}}{{if .User.Email}} · {{.User.Email}}{{end}} — sign out"
class="inline-flex shrink-0 items-center gap-2 rounded-full bg-[color:var(--card)] px-2.5 py-1.5 text-sm font-semibold shadow-pete border-2 border-[color:var(--ink)]/10 hover:bg-[color:var(--ink)]/5 transition">
@@ -102,6 +143,14 @@
</button>
<nav class="pete-channel-nav flex min-w-0 items-center gap-1 sm:gap-2 overflow-x-auto rounded-full bg-[color:var(--card)] p-1 shadow-pete border-2 border-[color:var(--ink)]/10">
{{$active := .Active}}
{{if .User}}
<a href="/for-you"
title="Your personalized feed"
class="shrink-0 rounded-full px-3 py-2 text-sm font-semibold transition
{{if eq $active "for-you"}}bg-[color:var(--accent)] text-[#1c1305] shadow-sm{{else}}hover:bg-[color:var(--ink)]/5{{end}}">
<span aria-hidden="true" class="mr-1"></span><span class="hidden sm:inline">For you</span>
</a>
{{end}}
{{range .Channels}}
<a href="/{{.Slug}}"
class="shrink-0 rounded-full px-3 py-2 text-sm font-semibold transition
@@ -136,6 +185,20 @@
<h2 class="font-display text-lg font-bold">Feed settings</h2>
<button type="button" data-settings-close class="rounded-full px-2 py-1 text-sm text-[color:var(--ink)]/60 hover:bg-[color:var(--ink)]/5">esc</button>
</div>
{{if .PushEnabled}}{{if .User}}
<div data-push-section hidden class="px-5 pt-4 pb-1">
<div class="flex items-center justify-between gap-3 rounded-2xl bg-[color:var(--ink)]/5 px-4 py-3">
<div class="min-w-0">
<div class="text-sm font-bold">🔔 Notifications</div>
<div data-push-note class="text-xs text-[color:var(--ink)]/60">Get a nudge when new stories land.</div>
</div>
<button type="button" data-push-toggle aria-pressed="false"
class="shrink-0 rounded-full bg-[color:var(--accent)] px-3 py-2 text-sm font-semibold text-[#1c1305] shadow-sm hover:brightness-105 transition disabled:opacity-50 disabled:cursor-not-allowed">
Turn on notifications
</button>
</div>
</div>
{{end}}{{end}}
<p class="px-5 pt-3 text-xs text-[color:var(--ink)]/60">Uncheck a feed to hide its stories. <span data-storage-note>Saved in this browser.</span></p>
<div data-settings-list class="max-h-[55vh] overflow-y-auto p-3 space-y-1"></div>
<div class="px-5 py-3 border-t border-[color:var(--ink)]/10 flex items-center justify-between text-xs">
@@ -145,6 +208,29 @@
</div>
</div>
<div id="pete-reader" class="hidden fixed inset-0 z-50" aria-modal="true" role="dialog" aria-label="Feed mode reader">
<div class="pete-reader-backdrop" data-reader-backdrop></div>
<div class="pete-reader-shell">
<div class="pete-reader-bar">
<span class="pete-reader-progress tabular-nums" data-reader-progress></span>
<div class="pete-reader-nav">
<button type="button" data-reader-prev class="pete-reader-btn" title="Previous (←)" aria-label="Previous article"></button>
<button type="button" data-reader-next class="pete-reader-btn" title="Next (→)" aria-label="Next article"></button>
<button type="button" data-reader-bookmark class="pete-reader-btn" style="display:none" title="Bookmark (b)" aria-label="Bookmark this story" aria-pressed="false">🔖 Save</button>
<a data-reader-link target="_blank" rel="noopener noreferrer" class="pete-reader-btn pete-reader-btn-open" title="Open original (o)">Open&nbsp;</a>
<button type="button" data-reader-close class="pete-reader-btn" title="Close (esc)" aria-label="Close reader">esc</button>
</div>
</div>
<div class="pete-reader-scroll" data-reader-scroll>
<article class="pete-reader-article" data-reader-article></article>
<div class="pete-reader-related" data-reader-related hidden></div>
</div>
<div class="pete-reader-hint">
<span><kbd></kbd> <kbd></kbd> navigate · <kbd>o</kbd> open · <kbd>u</kbd> mark unread · <kbd>esc</kbd> close</span>
</div>
</div>
</div>
<main class="mx-auto max-w-6xl px-4 pb-24">
{{block "main" .}}{{end}}
</main>
@@ -176,11 +262,14 @@
window.PETE_SOURCES = {{.AllSources}};
window.PETE_USER = {{if .User}}{ name: {{.User.Display}}, email: {{.User.Email}} }{{else}}null{{end}};
window.PETE_PREFS = {{.UserPrefs}};
window.PETE_PUSH = {{if .PushEnabled}}{ enabled: true, publicKey: {{.PushPublicKey}} }{{else}}null{{end}};
</script>
<script src="/static/js/prefs.js" defer></script>
<script src="/static/js/weather.js" defer></script>
<script src="/static/js/weather-forecast.js" defer></script>
<script src="/static/js/search.js" defer></script>
<script src="/static/js/settings.js" defer></script>
<script src="/static/js/reader.js" defer></script>
<script src="/static/js/pwa.js" defer></script>
</body>
</html>{{end}}

View File

@@ -0,0 +1,80 @@
{{define "title"}}Source health — {{.SiteTitle}}{{end}}
{{define "main"}}
<section class="mt-2 mb-8">
<div class="rounded-3xl bg-[color:var(--card)] border-2 border-[color:var(--ink)]/10 p-6 sm:p-8 shadow-pete">
<p class="text-sm uppercase tracking-[0.2em] text-[color:var(--ink)]/50">owner view</p>
<h1 class="font-display text-3xl sm:text-4xl font-bold mt-1">Source health</h1>
<p class="mt-2 max-w-2xl text-[color:var(--ink)]/70">
Per-feed poll status and content stats. Rows that are currently failing float to the top.
</p>
<div class="mt-4 flex flex-wrap gap-2">
<span class="inline-flex items-center gap-1.5 rounded-full bg-[color:var(--ink)]/5 px-3 py-1 text-sm font-semibold tabular-nums">
{{len .Sources}} sources
</span>
{{if .DegradedCnt}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-red-500/15 text-red-700 dark:text-red-300 px-3 py-1 text-sm font-semibold tabular-nums">
⚠ {{.DegradedCnt}} degraded
</span>
{{else}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-emerald-500/15 text-emerald-700 dark:text-emerald-300 px-3 py-1 text-sm font-semibold">
✓ all healthy
</span>
{{end}}
</div>
</div>
</section>
<div class="overflow-x-auto rounded-3xl bg-[color:var(--card)] border-2 border-[color:var(--ink)]/10 shadow-pete">
<table class="w-full min-w-[52rem] text-sm">
<thead>
<tr class="text-left text-[color:var(--ink)]/50 uppercase text-xs tracking-wider border-b-2 border-[color:var(--ink)]/10">
<th class="px-4 py-3 font-semibold">Source</th>
<th class="px-4 py-3 font-semibold">Channel</th>
<th class="px-4 py-3 font-semibold">Status</th>
<th class="px-4 py-3 font-semibold">Last poll</th>
<th class="px-4 py-3 font-semibold">Last success</th>
<th class="px-4 py-3 font-semibold text-right">Items</th>
<th class="px-4 py-3 font-semibold text-right">Stories</th>
<th class="px-4 py-3 font-semibold text-right">Paywall</th>
<th class="px-4 py-3 font-semibold">Last story</th>
<th class="px-4 py-3 font-semibold">Last posted</th>
</tr>
</thead>
<tbody>
{{range .Sources}}
<tr class="border-b border-[color:var(--ink)]/5 last:border-0 align-top">
<td class="px-4 py-3 font-semibold whitespace-nowrap">{{.Name}}</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{.Channel}}</td>
<td class="px-4 py-3 whitespace-nowrap">
{{if .NeverRun}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-[color:var(--ink)]/10 px-2.5 py-1 text-xs font-semibold">◌ idle</span>
{{else if .Healthy}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-emerald-500/15 text-emerald-700 dark:text-emerald-300 px-2.5 py-1 text-xs font-semibold">● ok</span>
{{else}}
<span class="inline-flex items-center gap-1.5 rounded-full bg-red-500/15 text-red-700 dark:text-red-300 px-2.5 py-1 text-xs font-semibold"
title="{{.LastError}}">✕ {{.Failures}} fail{{if ne .Failures 1}}s{{end}}</span>
{{end}}
</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastPollAt.IsZero}}never{{else}}{{timeAgo .LastPollAt}}{{end}}</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastSuccessAt.IsZero}}never{{else}}{{timeAgo .LastSuccessAt}}{{end}}</td>
<td class="px-4 py-3 text-right tabular-nums">{{.LastItemCount}}</td>
<td class="px-4 py-3 text-right tabular-nums" title="{{.Classified}} classified of {{.Total}}">{{.Total}}</td>
<td class="px-4 py-3 text-right tabular-nums {{if gt .PaywallRate 50}}text-amber-600 dark:text-amber-400 font-semibold{{end}}">
{{if .Total}}{{.PaywallRate}}%{{else}}—{{end}}
</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastSeenAt.IsZero}}—{{else}}{{timeAgo .LastSeenAt}}{{end}}</td>
<td class="px-4 py-3 text-[color:var(--ink)]/70 whitespace-nowrap">{{if .LastPostedAt.IsZero}}—{{else}}{{timeAgo .LastPostedAt}}{{end}}</td>
</tr>
{{if .LastError}}{{if not .Healthy}}
<tr class="border-b border-[color:var(--ink)]/5">
<td colspan="10" class="px-4 pb-3 -mt-1 text-xs text-red-600 dark:text-red-400 font-mono break-all">{{.LastError}}</td>
</tr>
{{end}}{{end}}
{{else}}
<tr><td colspan="10" class="px-4 py-8 text-center text-[color:var(--ink)]/50">No sources configured.</td></tr>
{{end}}
</tbody>
</table>
</div>
{{end}}

View File

@@ -0,0 +1,159 @@
package web
import (
"encoding/json"
"io"
"log/slog"
"net/http"
"strconv"
"strings"
"pete/internal/storage"
)
// maxStateBodyBytes caps read/bookmark request bodies. These carry a single id
// and a boolean, so this is generous headroom.
const maxStateBodyBytes = 1024
// maxStateIDs bounds how many ids /api/state will look up in one call. A page
// renders a few dozen cards; this is well clear of that.
const maxStateIDs = 500
// requireUser resolves the signed-in user or writes the appropriate error and
// returns nil. It mirrors handlePrefs: 404 when auth is disabled entirely, 401
// for anonymous callers (the client's cue to stay on localStorage).
func (s *Server) requireUser(w http.ResponseWriter, r *http.Request) *SessionUser {
if s.auth == nil {
http.Error(w, "auth disabled", http.StatusNotFound)
return nil
}
u := s.auth.userFromRequest(r)
if u == nil {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
http.Error(w, `{"error":"not signed in"}`, http.StatusUnauthorized)
return nil
}
return u
}
// decodeStateBody reads a small JSON body into dst, writing a 400 on failure.
func decodeStateBody(w http.ResponseWriter, r *http.Request, dst any) bool {
return decodeStateBodyN(w, r, dst, maxStateBodyBytes)
}
// decodeStateBodyN is decodeStateBody with an explicit byte cap, for endpoints
// (like push subscribe) whose payloads run larger than a single id + flag.
func decodeStateBodyN(w http.ResponseWriter, r *http.Request, dst any, limit int64) bool {
body, err := io.ReadAll(io.LimitReader(r.Body, limit+1))
if err != nil || int64(len(body)) > limit {
http.Error(w, `{"error":"bad request"}`, http.StatusBadRequest)
return false
}
if err := json.Unmarshal(body, dst); err != nil {
http.Error(w, `{"error":"invalid JSON"}`, http.StatusBadRequest)
return false
}
return true
}
// handleRead records or clears a story's read flag for the signed-in user.
func (s *Server) handleRead(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
var req struct {
ID int64 `json:"id"`
Read bool `json:"read"`
}
if !decodeStateBody(w, r, &req) {
return
}
if req.ID <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
if err := storage.SetRead(u.Sub, req.ID, req.Read); err != nil {
slog.Error("state: set read failed", "sub", u.Sub, "id", req.ID, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleBookmark adds or removes a bookmark for the signed-in user.
func (s *Server) handleBookmark(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
var req struct {
ID int64 `json:"id"`
On bool `json:"on"`
}
if !decodeStateBody(w, r, &req) {
return
}
if req.ID <= 0 {
http.Error(w, `{"error":"bad id"}`, http.StatusBadRequest)
return
}
if err := storage.SetBookmark(u.Sub, req.ID, req.On); err != nil {
slog.Error("state: set bookmark failed", "sub", u.Sub, "id", req.ID, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleState returns which of the given story ids are read and bookmarked, so
// the client can paint a freshly rendered page in one round-trip.
func (s *Server) handleState(w http.ResponseWriter, r *http.Request) {
u := s.requireUser(w, r)
if u == nil {
return
}
ids := parseIDList(r.URL.Query().Get("ids"))
read, bookmarked, err := storage.UserStoryState(u.Sub, ids)
if err != nil {
slog.Error("state: lookup failed", "sub", u.Sub, "err", err)
http.Error(w, `{"error":"internal error"}`, http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Cache-Control", "no-store")
_ = json.NewEncoder(w).Encode(map[string]any{
"read": keysOf(read),
"bookmarked": keysOf(bookmarked),
})
}
// parseIDList turns "1,2,3" into a bounded, deduplicated slice of positive ids.
func parseIDList(raw string) []int64 {
if raw == "" {
return nil
}
seen := make(map[int64]bool)
out := make([]int64, 0)
for _, part := range strings.Split(raw, ",") {
id, err := strconv.ParseInt(strings.TrimSpace(part), 10, 64)
if err != nil || id <= 0 || seen[id] {
continue
}
seen[id] = true
out = append(out, id)
if len(out) >= maxStateIDs {
break
}
}
return out
}
// keysOf returns the keys of a set as a slice (order unspecified).
func keysOf(m map[int64]bool) []int64 {
out := make([]int64, 0, len(m))
for k := range m {
out = append(out, k)
}
return out
}

30
main.go
View File

@@ -21,9 +21,22 @@ import (
"pete/internal/storage"
"pete/internal/web"
webpush "github.com/SherClockHolmes/webpush-go"
"maunium.net/go/mautrix/id"
)
// runGenVAPID prints a fresh VAPID keypair for the [web.push] config block.
func runGenVAPID() {
priv, pub, err := webpush.GenerateVAPIDKeys()
if err != nil {
fmt.Fprintln(os.Stderr, "genvapid: failed:", err)
os.Exit(1)
}
fmt.Println("# Add these under [web.push] in your config:")
fmt.Printf("vapid_public_key = %q\n", pub)
fmt.Printf("vapid_private_key = %q\n", priv)
}
func main() {
configPath := flag.String("config", "config.toml", "path to config file")
seed := flag.Bool("seed", false, "ingest current feed items as seen without posting, then exit")
@@ -31,8 +44,15 @@ func main() {
testSource := flag.String("test-source", "", "source name to use for -test (default: first enabled)")
local := flag.Bool("local", false, "web/RSS-only mode: poll feeds and serve the web UI, no Matrix login or posting")
backfillPaywall := flag.Bool("backfill-paywall", false, "re-check paywalled rows with current detection logic; clear false positives and fill missing thumbnails, then exit")
genVAPID := flag.Bool("genvapid", false, "generate a VAPID keypair for web.push and print it, then exit")
flag.Parse()
// -genvapid needs neither config nor database; handle it before anything else.
if *genVAPID {
runGenVAPID()
return
}
slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{
Level: slog.LevelInfo,
})))
@@ -240,11 +260,12 @@ func main() {
// Start the read-only web UI alongside the Matrix bot.
if cfg.Web.Enabled {
ws, err := web.New(cfg.Web, cfg.Sources)
ws, err := web.New(cfg.Web, cfg.Sources, postingEnabled)
if err != nil {
slog.Error("web server init failed", "err", err)
} else {
go ws.Start(ctx)
ws.StartPushSender(ctx)
}
}
@@ -349,12 +370,17 @@ func runLocal(cfg *config.Config) {
poller.Start(ctx)
slog.Info("local: pollers started")
ws, err := web.New(cfg.Web, cfg.Sources)
// Local mode never connects to Matrix, so it's always web-only.
ws, err := web.New(cfg.Web, cfg.Sources, false)
if err != nil {
slog.Error("local: web server init failed", "err", err)
os.Exit(1)
}
go ws.Start(ctx)
// Push only needs the web auth layer, not Matrix, so a web-only deployment
// still runs the digest sender — otherwise subscriptions accumulate but no
// digest ever fires.
ws.StartPushSender(ctx)
slog.Info("local: web UI listening", "addr", cfg.Web.ListenAddr)
storage.RunMaintenance()