drwily/Pete
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add per-source user_agent override for WAF-gated feeds

Browse Source 
The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest
bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an
optional per-source user_agent that falls back to the default bot UA, and
set The Portugal News to a browser UA. Not load-related — 30-min polls and
the IP isn't banned.
This commit is contained in:
prosolis
2026-06-21 16:40:05 -07:00
parent d5d0656dba
commit 4bdf9a7615
5 changed files with 18 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
internal/ingestion/poller.go
View File

@@ -95,7 +95,7 @@ func (p *Poller) pollOnce(ctx context.Context, src config.SourceConfig) {
}
func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) error {
items, err := FetchFeed(src.FeedURL)
items, err := FetchFeed(src.FeedURL, src.UserAgent)
if err != nil {
return err
}