mirror of
https://github.com/prosolis/gogobee.git
synced 2026-07-15 08:32:41 +00:00
ba7b20dfe5
Two independent causes, both silent: LimitedBody returned an error once the cap was hit, so scrapeOG failed the whole goquery parse on any page over 2 MiB — even though og: tags live in <head>, near the top. Hitting the cap is a truncation, not a failure: return io.EOF and let the parser decide whether it found what it needed. Sized against the pages actually posted here (n=14): og:title landed within the first 7 KiB on twelve, worst case ~600 KiB. validateImageURL HEAD-probed the image and bailed on non-200. Some publisher CDNs (dims.apnews.com among them) answer HEAD with 403 while serving the same URL over GET, so their thumbnails were always dropped. Probe with HEAD first, fall back to a ranged GET asking for the first KiB. A 206 declares the full size in Content-Range's "/119070" suffix rather than Content-Length, so the tracking-pixel filter reads size from there.
1.4 KiB
1.4 KiB