Add per-source user_agent override for WAF-gated feeds

The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest
bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an
optional per-source user_agent that falls back to the default bot UA, and
set The Portugal News to a browser UA. Not load-related — 30-min polls and
the IP isn't banned.
This commit is contained in:
prosolis
2026-06-21 16:40:05 -07:00
parent d5d0656dba
commit 4bdf9a7615
5 changed files with 18 additions and 8 deletions

View File

@@ -91,6 +91,10 @@ type SourceConfig struct {
// present and does not match (prefix). Useful for multilingual feeds
// like Politico Europe that publish English + French side-by-side.
Language string `toml:"language"`
// UserAgent overrides the User-Agent sent when fetching this feed. Empty
// uses Pete's honest default bot UA. Set a browser-like string only for
// sources whose WAF (e.g. AWS WAF on The Portugal News) blocks bot UAs.
UserAgent string `toml:"user_agent"`
}
func Load(path string) (*Config, error) {