Update README: !explain, paywall/Wayback fallback, explainer package
This commit is contained in:
23
README.md
23
README.md
@@ -9,6 +9,8 @@ A Matrix news bot that ingests RSS feeds from curated sources, classifies storie
|
||||
- **Semantic deduplication** — GUID exact match + LLM-based cross-source duplicate detection
|
||||
- **Metered release** — per-channel post queues with minimum interval (5min) and burst cap (3/30min)
|
||||
- **Reaction tracking** — records emoji reactions on posts for classifier tuning
|
||||
- **`!explain` via ❓ reaction** — react with `❓` (or `❔ ⁉ 🤔 ?`) on any post; Pete fetches the article body and replies in-thread with a 3-bullet Ollama-generated TL;DR
|
||||
- **Paywall detection** — if an article's visible body text is below threshold, Pete swaps in a Wayback Machine snapshot URL for both the lead image and the posted link
|
||||
- **FTS5 search** — full-text search across headlines and ledes
|
||||
- **Image validation** — HEAD-based checks filter tracking pixels, uploads valid images via MXC
|
||||
|
||||
@@ -80,13 +82,15 @@ Set `PETE_PASSWORD` in your environment or a `.env` file for the Matrix password
|
||||
## Architecture
|
||||
|
||||
```
|
||||
RSS Feed → Poller → GUID Dedup → Store → Classifier → Semantic Dedup → Image Validate → Queue → Matrix
|
||||
↓
|
||||
Direct Route (no LLM)
|
||||
or
|
||||
Ollama /api/chat (Tier 1 routing)
|
||||
or
|
||||
Keyword Gating → Ollama (Tier 2)
|
||||
RSS Feed → Poller → GUID Dedup → Canonical/Headline Dedup → Article Fetch → Store → Classifier → Semantic Dedup → Image Validate → Queue → Matrix
|
||||
↓ ↓
|
||||
Paywall? → Wayback Direct Route (no LLM)
|
||||
or
|
||||
Ollama /api/chat (Tier 1 routing)
|
||||
or
|
||||
Keyword Gating → Ollama (Tier 2)
|
||||
|
||||
Posted message ← ❓ reaction → Explainer → Article Fetch → Ollama summary → Threaded reply
|
||||
```
|
||||
|
||||
### Packages
|
||||
@@ -97,8 +101,9 @@ RSS Feed → Poller → GUID Dedup → Store → Classifier → Semantic Dedup
|
||||
| `internal/storage` | SQLite with WAL, FTS5, all queries |
|
||||
| `internal/ingestion` | Per-source RSS polling, feed parsing, image validation |
|
||||
| `internal/classifier` | Ollama client, Tier 1/2 prompts, JSON repair, keyword gating |
|
||||
| `internal/matrix` | Password auth with device persistence, posting, reaction listener |
|
||||
| `internal/poster` | Per-channel metered release queue, reaction tracking |
|
||||
| `internal/matrix` | Password auth with device persistence, posting, threaded replies, reaction listener |
|
||||
| `internal/poster` | Per-channel metered release queue, reaction tracking, callback hook |
|
||||
| `internal/explainer` | ❓-reaction → article fetch → Ollama summary → threaded reply |
|
||||
|
||||
## Post Format
|
||||
|
||||
|
||||
Reference in New Issue
Block a user