drwily/Pete
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rip out Ollama: direct_route only, no LLM layer

Browse Source 
Pete moves to a remote host without Ollama access. Every source must
declare a direct_route channel; the classifier, explainer, semantic
dedup, !explain summaries, feed_hint, and the recent_headlines /
classification_log tables are gone. Deterministic dedup (canonical URL,
headline_norm, per-channel cooldown) remains.
This commit is contained in:
prosolis
2026-05-24 22:07:13 -07:00
parent fbd4b84eaf
commit e88483526d
23 changed files with 146 additions and 1675 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
README.md
View File

@@ -1,16 +1,15 @@
# Pete
A Matrix news bot that ingests RSS feeds from curated sources, classifies stories using a local LLM, deduplicates semantically, and routes posts to the appropriate channel.
A Matrix news bot that ingests RSS feeds from curated sources and routes each story to a configured channel.
## Features
- **RSS ingestion** from configurable sources (Guardian, Ars Technica) with per-source polling intervals
- **Two-tier classification** — Tier 1 sources use direct routing or lightweight LLM routing; Tier 2 (wire services) use a keyword gating pipeline before LLM
- **Semantic deduplication** — GUID exact match + LLM-based cross-source duplicate detection
- **RSS ingestion** from configurable sources (Guardian, Ars Technica, Time Extension, …) with per-source polling intervals
- **Direct routing** — every source declares a `direct_route` channel; Pete posts there. No LLM, no classification step
- **Deduplication** — GUID exact match, canonical-URL match, normalized-headline match, and per-channel canonical-URL cooldown
- **Metered release** — per-channel post queues with minimum interval (5min) and burst cap (3/30min)
- **Round-robin mode** — opt-in pacing: one story per N hours (default 4), cycling through channels in sorted order, skip-and-advance over empty channels, state persists across restarts
- **Reaction tracking** — records emoji reactions on posts for classifier tuning
- **`!explain` via ❓ reaction** — react with `❓` (or `❔ ⁉ 🤔 ?`) on any post; Pete fetches the article body and replies in-thread with a 3-bullet Ollama-generated TL;DR
- **Reaction tracking** — records emoji reactions on posts
- **`!post` on demand** — type `!post` in any configured channel room and Pete force-publishes the next queued story for that channel, bypassing min-interval, burst cap, and daily cap (canonical-URL dedup still applies); replies in-thread if the queue is empty
- **Paywall detection** — if an article's visible body text is below threshold, Pete swaps in a Wayback Machine snapshot URL for both the lead image and the posted link
- **FTS5 search** — full-text search across headlines and ledes
@@ -22,13 +21,12 @@ A Matrix news bot that ingests RSS feeds from curated sources, classifies storie
| Channel | Purpose |
|---|---|
| `tech` | Technology news, product/industry stories |
| `politics` | Political, policy, current events (default/catch-all) |
| `gaming` | Gaming news, releases, platform announcements (no active sources yet) |
| `politics` | Political, policy, current events |
| `gaming` | Gaming news, releases, platform announcements |
## Requirements
- Go 1.25+
- [Ollama](https://ollama.ai) running locally or on a reachable host
- A Matrix account for Pete with access to target rooms
- SQLite (bundled via pure Go driver, no CGo)
- No CGo dependencies — E2EE uses pure Go crypto (goolm) via mautrix v0.28
@@ -42,7 +40,7 @@ go build -tags goolm .
# Create config from example
cp config.example.yaml config.yaml
# Edit config.yaml — fill in Matrix credentials, room IDs, Ollama endpoint
# Edit config.yaml — fill in Matrix credentials, room IDs, and a direct_route for every source
# First run: seed current feed items as seen (prevents flood)
./pete -config config.yaml -seed
@@ -59,10 +57,9 @@ cp config.example.yaml config.yaml
See `config.example.yaml` for the full structure. Key sections:
- `matrix` — homeserver, credentials, channel room IDs, optional admin room
- `ollama` — base URL, model name, timeout
- `posting` — rate limiting (min interval, burst cap, daily cap), optional `round_robin` block
- `storage` — database path, retention windows
- `sources` — RSS feeds with tier, polling interval, feed hint, optional direct routing
- `sources` — RSS feeds with tier, polling interval, and **required** `direct_route` (must match a key in `matrix.channels`)
- `web` — read-only HTTP UI (enabled toggle, listen address, site title, public base URL)
Environment variables can be referenced with `${VAR}` syntax in the YAML.
@@ -86,7 +83,7 @@ Front the server with a reverse proxy (Caddy, nginx, Traefik) terminating TLS fo
### Round-robin mode
Set `posting.round_robin.enabled: true` to switch Pete from "post on classify" to a paced rotation. On each tick (`interval_hours`, default 4) Pete picks the newest unposted classified story routed to the next channel in sorted order, posting through the existing queue. Empty channels are skipped; the rotation pointer advances to whichever channel actually posted, and `last_channel` / `last_tick_at` are persisted so restarts don't reset the cycle. Rotating by channel (not by source) guarantees variety even when one channel has many more feeds than the others.
Set `posting.round_robin.enabled: true` to switch Pete from "post immediately" to a paced rotation. On each tick (`interval_hours`, default 4) Pete picks the newest unposted story routed to the next channel in sorted order, posting through the existing queue. Empty channels are skipped; the rotation pointer advances to whichever channel actually posted, and `last_channel` / `last_tick_at` are persisted so restarts don't reset the cycle. Rotating by channel (not by source) guarantees variety even when one channel has many more feeds than the others.
With a 4-hour cadence Pete will post at most 6 stories/day, so set `posting.daily_cap_total` to 0 (or ≥6) when enabling this — otherwise the daily cap can silently swallow a tick.
@@ -109,15 +106,9 @@ Set `PETE_PASSWORD` in your environment or a `.env` file for the Matrix password
## Architecture
```
RSS Feed → Poller → GUID Dedup → Canonical/Headline Dedup → Article Fetch → Store → Classifier → Semantic Dedup → Image Validate → Queue → Matrix
↓ ↓
Paywall? → Wayback Direct Route (no LLM)
or
Ollama /api/chat (Tier 1 routing)
or
Keyword Gating → Ollama (Tier 2)
Posted message ← ❓ reaction → Explainer → Article Fetch → Ollama summary → Threaded reply
RSS Feed → Poller → GUID/Canonical/Headline Dedup → Article Fetch → Store → Route by direct_route → Image Validate → Queue → Matrix
↓
Paywall? → Wayback snapshot
```
### Packages
@@ -126,11 +117,10 @@ Posted message ← ❓ reaction → Explainer → Article Fetch → Ollama summa
|---|---|
| `internal/config` | YAML config loading with `${ENV}` expansion |
| `internal/storage` | SQLite with WAL, FTS5, all queries |
| `internal/ingestion` | Per-source RSS polling, feed parsing, image validation |
| `internal/classifier` | Ollama client, Tier 1/2 prompts, JSON repair, keyword gating |
| `internal/ingestion` | Per-source RSS polling, feed parsing, image validation, paywall detection |
| `internal/dedup` | Canonical URL + headline normalization helpers |
| `internal/matrix` | Password auth with device persistence, posting, threaded replies, reaction + message listener |
| `internal/poster` | Per-channel metered release queue, reaction tracking, callback hook |
| `internal/explainer` | ❓-reaction → article fetch → Ollama summary → threaded reply |
| `internal/poster` | Per-channel metered release queue, reaction tracking |
| `internal/scheduler` | Round-robin posting scheduler: paced rotation across channels when enabled |
| `internal/web` | Read-only HTTP UI: Tailwind templates, day/night palette, per-channel feed pages |
@@ -143,8 +133,6 @@ Lede text from feed
`source name`
```
Gaming stories append platform tags: `` `source` · `nintendo-switch` · `pc` ``
## Testing
```bash