drwily/Pete
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Strip inline scripts from reader body extraction

Browse Source 
Datawrapper embed resizers (and other inline scripts) were leaking into
reader mode as literal text. Two extraction paths were affected:

- parser.go extractContentText/extractLede stripped only script tags via
  htmlTagRe, leaving the JS body behind. This is the path that usually
  wins for feeds shipping full content:encoded (e.g. Politico).
- article.go goquery paths call .Text(), which concatenates script source.

Both now drop whole <script>/<style>/<noscript> elements before pulling
text. Paywall detection (JSON-LD) still runs before the goquery strip.
This commit is contained in:
prosolis
2026-07-07 21:33:35 -07:00
parent ff3a0e87be
commit 2ea5f7a6f7
4 changed files with 79 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
internal/ingestion/og_test.go
View File

@@ -105,6 +105,31 @@ func TestExtractBodyCharsANNLayout(t *testing.T) {
}
}
// TestExtractBodyTextStripsInlineScript guards against inline embed scripts
// (Datawrapper iframe resizers, analytics) leaking into reader mode. goquery's
// .Text() concatenates the source of <script> nodes, so extraction must strip
// them first. The script here sits between two prose paragraphs, mirroring the
// Politico/Datawrapper layout that surfaced the bug.
func TestExtractBodyTextStripsInlineScript(t *testing.T) {
body := `<html><body><article>
<p>While he has an overall net trust rating of +11 percent nationally, this masks a large regional gap between the north and south of the country.</p>
<figure><script type="text/javascript">!function(){"use strict";window.addEventListener("message",(function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r=0;r<e.length;r++);}}))}();</script></figure>
<p>Burnham is on the cusp of becoming the U.K.'s seventh prime minister in 10 years and will enter Number 10 later this month after being elected unopposed.</p>
</article></body></html>`
doc, err := goquery.NewDocumentFromReader(strings.NewReader(body))
if err != nil {
t.Fatalf("parse: %v", err)
}
stripNonContent(doc)
out := extractBodyText(doc)
if strings.Contains(out, "datawrapper-height") || strings.Contains(out, "addEventListener") {
t.Fatalf("inline script leaked into body text:\n%s", out)
}
if !strings.Contains(out, "net trust rating") || !strings.Contains(out, "seventh prime minister") {
t.Fatalf("prose paragraphs missing from body text:\n%s", out)
}
}
func TestDetectSubscriberOnly(t *testing.T) {
cases := []struct {
name string