drwily/gogobee
1
0
Fork 0
mirror of https://github.com/prosolis/gogobee.git synced 2026-07-16 08:52:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

url previews: stop dropping thumbnails on body cap and HEAD-403 CDNs

Browse Source 
Two independent causes, both silent:

LimitedBody returned an error once the cap was hit, so scrapeOG failed the
whole goquery parse on any page over 2 MiB — even though og: tags live in
<head>, near the top. Hitting the cap is a truncation, not a failure: return
io.EOF and let the parser decide whether it found what it needed. Sized
against the pages actually posted here (n=14): og:title landed within the
first 7 KiB on twelve, worst case ~600 KiB.

validateImageURL HEAD-probed the image and bailed on non-200. Some publisher
CDNs (dims.apnews.com among them) answer HEAD with 403 while serving the same
URL over GET, so their thumbnails were always dropped. Probe with HEAD first,
fall back to a ranged GET asking for the first KiB. A 206 declares the full
size in Content-Range's "/119070" suffix rather than Content-Length, so the
tracking-pixel filter reads size from there.
This commit is contained in:
prosolis
2026-07-09 18:52:32 -07:00
parent b5493a0e79
commit ba7b20dfe5
5 changed files with 201 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
internal/plugin/urls.go
View File

@@ -187,7 +187,14 @@ func (p *URLsPlugin) scrapeOG(rawURL string) (string, string, string, error) {
return "", "", "", fmt.Errorf("status %d", resp.StatusCode)
}
// Cap the parsed body at 2 MiB — og: tags live in <head>, near the top.
// Cap the parsed body at 2 MiB: big enough that <head> always fits, small
// enough to bound memory against an origin streaming an endless body. Reaching
// the cap truncates rather than failing, so an oversized tail costs nothing.
//
// Sized against the pages actually posted here (2026-07-09, n=14): og:title
// landed within the first 7 KiB on twelve of them, worst case ~600 KiB, on
// pages up to 1.44 MiB. Note failed scrapes never reach url_cache, so that
// sample can't tell you how big the pages that *blew* the old cap were.
doc, err := goquery.NewDocumentFromReader(safehttp.LimitedBody(resp.Body, 2*1024*1024))
if err != nil {
return "", "", "", fmt.Errorf("parse HTML: %w", err)