119 lines
3.9 KiB
Markdown
119 lines
3.9 KiB
Markdown
# Pete
|
|
|
|
A Matrix news bot that ingests RSS feeds from curated sources, classifies stories using a local LLM, deduplicates semantically, and routes posts to the appropriate channel.
|
|
|
|
## Features
|
|
|
|
- **RSS ingestion** from configurable sources (Guardian, Ars Technica) with per-source polling intervals
|
|
- **Two-tier classification** — Tier 1 sources use direct routing or lightweight LLM routing; Tier 2 (wire services) use a keyword gating pipeline before LLM
|
|
- **Semantic deduplication** — GUID exact match + LLM-based cross-source duplicate detection
|
|
- **Metered release** — per-channel post queues with minimum interval (5min) and burst cap (3/30min)
|
|
- **Reaction tracking** — records emoji reactions on posts for classifier tuning
|
|
- **FTS5 search** — full-text search across headlines and ledes
|
|
- **Image validation** — HEAD-based checks filter tracking pixels, uploads valid images via MXC
|
|
|
|
## Channels
|
|
|
|
| Channel | Purpose |
|
|
|---|---|
|
|
| `tech` | Technology news, product/industry stories |
|
|
| `politics` | Political, policy, current events (default/catch-all) |
|
|
| `gaming` | Gaming news, releases, platform announcements (no active sources yet) |
|
|
|
|
## Requirements
|
|
|
|
- Go 1.25+
|
|
- [Ollama](https://ollama.ai) running locally or on a reachable host
|
|
- A Matrix account for Pete with access to target rooms
|
|
- SQLite (bundled via pure Go driver, no CGo)
|
|
- No CGo dependencies — E2EE uses pure Go crypto (goolm) via mautrix v0.28
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
# Clone and build
|
|
git clone <repo-url> && cd pete
|
|
go build -tags goolm .
|
|
|
|
# Create config from example
|
|
cp config.example.yaml config.yaml
|
|
# Edit config.yaml — fill in Matrix credentials, room IDs, Ollama endpoint
|
|
|
|
# First run: seed current feed items as seen (prevents flood)
|
|
./pete -config config.yaml -seed
|
|
|
|
# Post one story to verify the pipeline
|
|
./pete -config config.yaml -test
|
|
|
|
# Run
|
|
./pete -config config.yaml
|
|
```
|
|
|
|
## Configuration
|
|
|
|
See `config.example.yaml` for the full structure. Key sections:
|
|
|
|
- `matrix` — homeserver, credentials, channel room IDs, optional admin room
|
|
- `ollama` — base URL, model name, timeout
|
|
- `posting` — rate limiting (min interval, burst cap)
|
|
- `storage` — database path, retention windows
|
|
- `sources` — RSS feeds with tier, polling interval, feed hint, optional direct routing
|
|
|
|
Environment variables can be referenced with `${VAR}` syntax in the YAML.
|
|
|
|
## Flags
|
|
|
|
| Flag | Description |
|
|
|---|---|
|
|
| `-config <path>` | Path to config file (default: `config.yaml`) |
|
|
| `-seed` | Ingest all current feed items as seen without posting, then exit |
|
|
| `-test` | Post one story to verify the full pipeline, then exit |
|
|
|
|
## Docker
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
Set `PETE_PASSWORD` in your environment or a `.env` file for the Matrix password.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
RSS Feed → Poller → GUID Dedup → Store → Classifier → Semantic Dedup → Image Validate → Queue → Matrix
|
|
↓
|
|
Direct Route (no LLM)
|
|
or
|
|
Ollama /api/chat (Tier 1 routing)
|
|
or
|
|
Keyword Gating → Ollama (Tier 2)
|
|
```
|
|
|
|
### Packages
|
|
|
|
| Package | Purpose |
|
|
|---|---|
|
|
| `internal/config` | YAML config loading with `${ENV}` expansion |
|
|
| `internal/storage` | SQLite with WAL, FTS5, all queries |
|
|
| `internal/ingestion` | Per-source RSS polling, feed parsing, image validation |
|
|
| `internal/classifier` | Ollama client, Tier 1/2 prompts, JSON repair, keyword gating |
|
|
| `internal/matrix` | Password auth with device persistence, posting, reaction listener |
|
|
| `internal/poster` | Per-channel metered release queue, reaction tracking |
|
|
|
|
## Post Format
|
|
|
|
```
|
|
[lead image if valid]
|
|
**Headline** → article_url
|
|
Lede text from feed
|
|
`source name`
|
|
```
|
|
|
|
Gaming stories append platform tags: `` `source` · `nintendo-switch` · `pc` ``
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
go test ./...
|
|
```
|