storage: keep the stories people saved, and prune more than once a deploy
Maintenance deleted every story older than 30 days and then swept up any user_story_state row pointing at a story that no longer existed. A bookmark is one of those rows, so every saved story quietly vanished a month after Pete first saw it, which is exactly what saving it was meant to prevent. A bookmarked story is now exempt from the prune; read markers on pruned stories still go. Maintenance also only ran at startup, on a service that can go weeks between deploys, so nothing was pruned for weeks and then everything was at once. It runs at startup and then daily. Pruning daily makes one old hole steady-state: the stories row is Pete's only memory of a GUID, so a slow feed still listing month-old items would meet them again as new the day after their rows went. The poller now skips items published before the retention window. Undated items can't be judged and still pass.
This commit is contained in:
@@ -118,6 +118,16 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
|
||||
continue
|
||||
}
|
||||
|
||||
// An item older than the retention window is one Pete has either
|
||||
// already shown and since pruned, or never needs to: maintenance would
|
||||
// delete it on the next pass. Without this, a slow feed that still
|
||||
// lists month-old items re-ingests them as new the day after their row
|
||||
// is pruned, since that row was the only memory of them. Items with no
|
||||
// date can't be judged and pass through.
|
||||
if items[i].PublishedAt > 0 && time.Since(time.Unix(items[i].PublishedAt, 0)) > storage.StoryRetention {
|
||||
continue
|
||||
}
|
||||
|
||||
// Drop items in languages this source isn't configured to surface.
|
||||
// Items without a language tag pass through — only filter when the
|
||||
// feed explicitly marks an item with a non-matching language.
|
||||
|
||||
Reference in New Issue
Block a user