Add feed/reader mode with full-article capture at ingest

Reader mode presents the stories on a page one at a time in a focused
overlay, marking each read as it's shown. Left/right arrows (or the header
book button / `f`) page through them; read stories dim on the grid. Read
state is device-local in localStorage.

Backing this required actually capturing article bodies, which Pete wasn't
doing — it kept only the RSS <description> lede and discarded content:encoded:

- stories.content column (idempotent migration; old rows fall back to lede)
- parser keeps content:encoded as paragraph-preserving text
- article fetch already done for paywall detection now also returns its body,
  so ingest stores the richer of feed-content vs scraped body with no extra
  request (prefers the archive snapshot body for paywalled stories)
- GET /api/article?id= serves the stored text; card queries now select id and
  expose it as data-id for the reader

Tests cover content extraction, the storage round-trip, and the article
endpoint + card rendering end to end.
This commit is contained in:
prosolis
2026-07-06 22:46:14 -07:00
parent 410f8dda0a
commit 55aa167151
17 changed files with 803 additions and 13 deletions

View File

@@ -160,6 +160,10 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
"guid", items[i].GUID, "url", originalURL, "source", src.Name)
continue
}
// bodyText is the article text we may store for reader mode. It starts
// as the body scraped during the fetch above and is upgraded to the
// snapshot's body if we end up swapping in an archive snapshot.
bodyText := meta.BodyText
gated := meta.Gated() || (meta.Fetched && meta.BodyChars < PaywallBodyThreshold)
// paywalled tracks whether the link the user will click is still
// gated — i.e. gating was detected AND no archive workaround worked.
@@ -175,6 +179,9 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
if items[i].ImageURL == "" && snapMeta.ImageURL != "" {
items[i].ImageURL = snapMeta.ImageURL
}
if snapMeta.BodyText != "" {
bodyText = snapMeta.BodyText
}
workedAround = true
slog.Info("paywall detected, using archive snapshot",
"guid", items[i].GUID, "original", originalURL,
@@ -209,10 +216,18 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
if publishedAt > seenAt {
publishedAt = seenAt
}
// Reader-mode body: prefer whichever source gave us more complete text.
// Feeds that ship full content:encoded usually win; feeds that only
// syndicate a snippet fall back to the scraped article body.
content := items[i].Content
if len(bodyText) > len(content) {
content = bodyText
}
if err := storage.InsertStory(&storage.Story{
GUID: items[i].GUID,
Headline: items[i].Headline,
Lede: items[i].Lede,
Content: content,
ImageURL: items[i].ImageURL,
ArticleURL: items[i].ArticleURL,
URLCanonical: canonical,