Update README: !explain, paywall/Wayback fallback, explainer package

This commit is contained in:
prosolis
2026-05-22 18:38:37 -07:00
parent d41eaa6504
commit 8d1e6ed568

View File

@@ -9,6 +9,8 @@ A Matrix news bot that ingests RSS feeds from curated sources, classifies storie
- **Semantic deduplication** — GUID exact match + LLM-based cross-source duplicate detection
- **Metered release** — per-channel post queues with minimum interval (5min) and burst cap (3/30min)
- **Reaction tracking** — records emoji reactions on posts for classifier tuning
- **`!explain` via ❓ reaction** — react with `❓` (or `❔ ⁉ 🤔 ?`) on any post; Pete fetches the article body and replies in-thread with a 3-bullet Ollama-generated TL;DR
- **Paywall detection** — if an article's visible body text is below threshold, Pete swaps in a Wayback Machine snapshot URL for both the lead image and the posted link
- **FTS5 search** — full-text search across headlines and ledes
- **Image validation** — HEAD-based checks filter tracking pixels, uploads valid images via MXC
@@ -80,13 +82,15 @@ Set `PETE_PASSWORD` in your environment or a `.env` file for the Matrix password
## Architecture
```
RSS Feed → Poller → GUID Dedup → Store → Classifier → Semantic Dedup → Image Validate → Queue → Matrix
Direct Route (no LLM)
or
Ollama /api/chat (Tier 1 routing)
or
Keyword Gating → Ollama (Tier 2)
RSS Feed → Poller → GUID Dedup → Canonical/Headline Dedup → Article Fetch → Store → Classifier → Semantic Dedup → Image Validate → Queue → Matrix
Paywall? → Wayback Direct Route (no LLM)
or
Ollama /api/chat (Tier 1 routing)
or
Keyword Gating → Ollama (Tier 2)
Posted message ← ❓ reaction → Explainer → Article Fetch → Ollama summary → Threaded reply
```
### Packages
@@ -97,8 +101,9 @@ RSS Feed → Poller → GUID Dedup → Store → Classifier → Semantic Dedup
| `internal/storage` | SQLite with WAL, FTS5, all queries |
| `internal/ingestion` | Per-source RSS polling, feed parsing, image validation |
| `internal/classifier` | Ollama client, Tier 1/2 prompts, JSON repair, keyword gating |
| `internal/matrix` | Password auth with device persistence, posting, reaction listener |
| `internal/poster` | Per-channel metered release queue, reaction tracking |
| `internal/matrix` | Password auth with device persistence, posting, threaded replies, reaction listener |
| `internal/poster` | Per-channel metered release queue, reaction tracking, callback hook |
| `internal/explainer` | ❓-reaction → article fetch → Ollama summary → threaded reply |
## Post Format