Paywall detection: swap to Wayback snapshot when article body is gated
Fetch each new article once and measure visible <p> text. If body is below 500 chars (or fetch fails), resolve a Wayback snapshot via the archive.org/wayback/available API and use that URL for both the og:image fallback and the posted link. Dedup keys stay derived from the original URL so paywalled/non-paywalled hits collide as before. - New: internal/ingestion/article.go (FetchArticleMeta via goquery) - New: internal/ingestion/wayback.go (ResolveWayback) - Removed: internal/ingestion/og.go (folded into article.go) - poller.go: dedup first, then one article fetch, then snapshot fallback
This commit is contained in:
2
go.mod
2
go.mod
@@ -3,6 +3,7 @@ module pete
|
||||
go 1.25.0
|
||||
|
||||
require (
|
||||
github.com/PuerkitoBio/goquery v1.12.0
|
||||
github.com/mmcdole/gofeed v1.3.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
maunium.net/go/mautrix v0.28.0
|
||||
@@ -11,7 +12,6 @@ require (
|
||||
|
||||
require (
|
||||
filippo.io/edwards25519 v1.2.0 // indirect
|
||||
github.com/PuerkitoBio/goquery v1.12.0 // indirect
|
||||
github.com/andybalholm/cascadia v1.3.3 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
|
||||
Reference in New Issue
Block a user