drwily/Pete
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add per-source user_agent override for WAF-gated feeds

Browse Source 
The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest
bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an
optional per-source user_agent that falls back to the default bot UA, and
set The Portugal News to a browser UA. Not load-related — 30-min polls and
the IP isn't banned.
This commit is contained in:
prosolis
2026-06-21 16:40:05 -07:00
parent d5d0656dba
commit 4bdf9a7615
5 changed files with 18 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
config.example.toml
View File

@@ -65,6 +65,9 @@ tier = 1
poll_interval_minutes = 20
direct_route = "politics"
enabled = true
# Optional: override the User-Agent for this feed only. Leave unset to use
# Pete's honest bot UA; set a browser string for feeds whose WAF blocks bots.
# user_agent = "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
[[sources]]
name = "The Guardian — Politics"