Fix glued-together RSS ledes; parse per-item language
Two ingestion changes: - extractLede replaced HTML tags with empty string, so adjacent block tags like </p><p> fused words across paragraphs. Replace tags with a space and collapse whitespace. - Pull each item's <language> tag (or dc:language) into FeedItem so the poller can filter on it. Politico Europe publishes the same story in en / fr / de side-by-side and we want to keep only one language per source.
This commit is contained in:
@@ -48,6 +48,11 @@ func TestExtractLede(t *testing.T) {
|
||||
"It's a new era—one of change.",
|
||||
"It's a new era\u2014one of change.",
|
||||
},
|
||||
{
|
||||
"adjacent block tags inject a space",
|
||||
"<p>...end of moments</p><p>Clarence B Jones, a former...</p>",
|
||||
"...end of moments Clarence B Jones, a former...",
|
||||
},
|
||||
{
|
||||
"tags and entities combined",
|
||||
"<p>Oil prices rose & gas fell by >5%.</p>",
|
||||
|
||||
Reference in New Issue
Block a user