Add per-source user_agent override for WAF-gated feeds
The Portugal News moved its RSS behind AWS WAF, which 405s Pete's honest bot UA (confirmed: bot UA 0/6, browser UA 6/6 from the server IP). Add an optional per-source user_agent that falls back to the default bot UA, and set The Portugal News to a browser UA. Not load-related — 30-min polls and the IP isn't banned.
This commit is contained in:
@@ -91,6 +91,10 @@ type SourceConfig struct {
|
||||
// present and does not match (prefix). Useful for multilingual feeds
|
||||
// like Politico Europe that publish English + French side-by-side.
|
||||
Language string `toml:"language"`
|
||||
// UserAgent overrides the User-Agent sent when fetching this feed. Empty
|
||||
// uses Pete's honest default bot UA. Set a browser-like string only for
|
||||
// sources whose WAF (e.g. AWS WAF on The Portugal News) blocks bot UAs.
|
||||
UserAgent string `toml:"user_agent"`
|
||||
}
|
||||
|
||||
func Load(path string) (*Config, error) {
|
||||
|
||||
Reference in New Issue
Block a user