Drop LWN subscriber-only articles instead of stamping paywalled
LWN's subscriber-only articles have no public version reachable by archive snapshots or bypass UAs, so stamping them as paywalled produces clicks that always dead-end. Detect the marker text and skip ingestion entirely.
This commit is contained in:
@@ -137,6 +137,17 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
|
||||
// swapping in a stale snapshot for a transient blip is worse than
|
||||
// posting the live link.
|
||||
meta := FetchArticleMeta(originalURL)
|
||||
// Some sources (e.g. LWN's subscriber-only articles) have no public
|
||||
// version reachable by any workaround. Drop these from view entirely
|
||||
// rather than stamping them as paywalled.
|
||||
if meta.SubscriberOnly {
|
||||
// No DB row written, so the next poll will re-fetch and re-detect
|
||||
// this URL — fine in practice since LWN's feed is tiny and the
|
||||
// alternative (a tombstone row) leaks into every story query.
|
||||
slog.Info("dropping subscriber-only story",
|
||||
"guid", items[i].GUID, "url", originalURL, "source", src.Name)
|
||||
continue
|
||||
}
|
||||
gated := meta.Gated() || (meta.Fetched && meta.BodyChars < PaywallBodyThreshold)
|
||||
// paywalled tracks whether the link the user will click is still
|
||||
// gated — i.e. gating was detected AND no archive workaround worked.
|
||||
|
||||
Reference in New Issue
Block a user