Fix glued-together RSS ledes; parse per-item language

Two ingestion changes:

- extractLede replaced HTML tags with empty string, so adjacent
  block tags like </p><p> fused words across paragraphs. Replace
  tags with a space and collapse whitespace.
- Pull each item's <language> tag (or dc:language) into FeedItem
  so the poller can filter on it. Politico Europe publishes the
  same story in en / fr / de side-by-side and we want to keep
  only one language per source.
This commit is contained in:
prosolis
2026-05-26 23:02:46 -07:00
parent 59658e7ebb
commit 7d469cf8c5
2 changed files with 27 additions and 1 deletions

View File

@@ -48,6 +48,11 @@ func TestExtractLede(t *testing.T) {
"It&#39;s a new era&#8212;one of change.",
"It's a new era\u2014one of change.",
},
{
"adjacent block tags inject a space",
"<p>...end of moments</p><p>Clarence B Jones, a former...</p>",
"...end of moments Clarence B Jones, a former...",
},
{
"tags and entities combined",
"<p>Oil prices rose &amp; gas fell by &gt;5%.</p>",