Drop LWN subscriber-only articles instead of stamping paywalled

LWN's subscriber-only articles have no public version reachable by
archive snapshots or bypass UAs, so stamping them as paywalled produces
clicks that always dead-end. Detect the marker text and skip ingestion
entirely.
This commit is contained in:
prosolis
2026-05-25 14:31:08 -07:00
parent dd99f2c7ab
commit 83b1216541
3 changed files with 78 additions and 1 deletions

View File

@@ -137,6 +137,17 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
// swapping in a stale snapshot for a transient blip is worse than
// posting the live link.
meta := FetchArticleMeta(originalURL)
// Some sources (e.g. LWN's subscriber-only articles) have no public
// version reachable by any workaround. Drop these from view entirely
// rather than stamping them as paywalled.
if meta.SubscriberOnly {
// No DB row written, so the next poll will re-fetch and re-detect
// this URL — fine in practice since LWN's feed is tiny and the
// alternative (a tombstone row) leaks into every story query.
slog.Info("dropping subscriber-only story",
"guid", items[i].GUID, "url", originalURL, "source", src.Name)
continue
}
gated := meta.Gated() || (meta.Fetched && meta.BodyChars < PaywallBodyThreshold)
// paywalled tracks whether the link the user will click is still
// gated — i.e. gating was detected AND no archive workaround worked.