Rip out Ollama: direct_route only, no LLM layer

Pete moves to a remote host without Ollama access. Every source must
declare a direct_route channel; the classifier, explainer, semantic
dedup, !explain summaries, feed_hint, and the recent_headlines /
classification_log tables are gone. Deterministic dedup (canonical URL,
headline_norm, per-channel cooldown) remains.
This commit is contained in:
prosolis
2026-05-24 22:07:13 -07:00
parent fbd4b84eaf
commit e88483526d
23 changed files with 146 additions and 1675 deletions

View File

@@ -1,16 +1,15 @@
# Pete # Pete
A Matrix news bot that ingests RSS feeds from curated sources, classifies stories using a local LLM, deduplicates semantically, and routes posts to the appropriate channel. A Matrix news bot that ingests RSS feeds from curated sources and routes each story to a configured channel.
## Features ## Features
- **RSS ingestion** from configurable sources (Guardian, Ars Technica) with per-source polling intervals - **RSS ingestion** from configurable sources (Guardian, Ars Technica, Time Extension, …) with per-source polling intervals
- **Two-tier classification** — Tier 1 sources use direct routing or lightweight LLM routing; Tier 2 (wire services) use a keyword gating pipeline before LLM - **Direct routing** — every source declares a `direct_route` channel; Pete posts there. No LLM, no classification step
- **Semantic deduplication** — GUID exact match + LLM-based cross-source duplicate detection - **Deduplication** — GUID exact match, canonical-URL match, normalized-headline match, and per-channel canonical-URL cooldown
- **Metered release** — per-channel post queues with minimum interval (5min) and burst cap (3/30min) - **Metered release** — per-channel post queues with minimum interval (5min) and burst cap (3/30min)
- **Round-robin mode** — opt-in pacing: one story per N hours (default 4), cycling through channels in sorted order, skip-and-advance over empty channels, state persists across restarts - **Round-robin mode** — opt-in pacing: one story per N hours (default 4), cycling through channels in sorted order, skip-and-advance over empty channels, state persists across restarts
- **Reaction tracking** — records emoji reactions on posts for classifier tuning - **Reaction tracking** — records emoji reactions on posts
- **`!explain` via ❓ reaction** — react with `❓` (or `❔ ⁉ 🤔 ?`) on any post; Pete fetches the article body and replies in-thread with a 3-bullet Ollama-generated TL;DR
- **`!post` on demand** — type `!post` in any configured channel room and Pete force-publishes the next queued story for that channel, bypassing min-interval, burst cap, and daily cap (canonical-URL dedup still applies); replies in-thread if the queue is empty - **`!post` on demand** — type `!post` in any configured channel room and Pete force-publishes the next queued story for that channel, bypassing min-interval, burst cap, and daily cap (canonical-URL dedup still applies); replies in-thread if the queue is empty
- **Paywall detection** — if an article's visible body text is below threshold, Pete swaps in a Wayback Machine snapshot URL for both the lead image and the posted link - **Paywall detection** — if an article's visible body text is below threshold, Pete swaps in a Wayback Machine snapshot URL for both the lead image and the posted link
- **FTS5 search** — full-text search across headlines and ledes - **FTS5 search** — full-text search across headlines and ledes
@@ -22,13 +21,12 @@ A Matrix news bot that ingests RSS feeds from curated sources, classifies storie
| Channel | Purpose | | Channel | Purpose |
|---|---| |---|---|
| `tech` | Technology news, product/industry stories | | `tech` | Technology news, product/industry stories |
| `politics` | Political, policy, current events (default/catch-all) | | `politics` | Political, policy, current events |
| `gaming` | Gaming news, releases, platform announcements (no active sources yet) | | `gaming` | Gaming news, releases, platform announcements |
## Requirements ## Requirements
- Go 1.25+ - Go 1.25+
- [Ollama](https://ollama.ai) running locally or on a reachable host
- A Matrix account for Pete with access to target rooms - A Matrix account for Pete with access to target rooms
- SQLite (bundled via pure Go driver, no CGo) - SQLite (bundled via pure Go driver, no CGo)
- No CGo dependencies — E2EE uses pure Go crypto (goolm) via mautrix v0.28 - No CGo dependencies — E2EE uses pure Go crypto (goolm) via mautrix v0.28
@@ -42,7 +40,7 @@ go build -tags goolm .
# Create config from example # Create config from example
cp config.example.yaml config.yaml cp config.example.yaml config.yaml
# Edit config.yaml — fill in Matrix credentials, room IDs, Ollama endpoint # Edit config.yaml — fill in Matrix credentials, room IDs, and a direct_route for every source
# First run: seed current feed items as seen (prevents flood) # First run: seed current feed items as seen (prevents flood)
./pete -config config.yaml -seed ./pete -config config.yaml -seed
@@ -59,10 +57,9 @@ cp config.example.yaml config.yaml
See `config.example.yaml` for the full structure. Key sections: See `config.example.yaml` for the full structure. Key sections:
- `matrix` — homeserver, credentials, channel room IDs, optional admin room - `matrix` — homeserver, credentials, channel room IDs, optional admin room
- `ollama` — base URL, model name, timeout
- `posting` — rate limiting (min interval, burst cap, daily cap), optional `round_robin` block - `posting` — rate limiting (min interval, burst cap, daily cap), optional `round_robin` block
- `storage` — database path, retention windows - `storage` — database path, retention windows
- `sources` — RSS feeds with tier, polling interval, feed hint, optional direct routing - `sources` — RSS feeds with tier, polling interval, and **required** `direct_route` (must match a key in `matrix.channels`)
- `web` — read-only HTTP UI (enabled toggle, listen address, site title, public base URL) - `web` — read-only HTTP UI (enabled toggle, listen address, site title, public base URL)
Environment variables can be referenced with `${VAR}` syntax in the YAML. Environment variables can be referenced with `${VAR}` syntax in the YAML.
@@ -86,7 +83,7 @@ Front the server with a reverse proxy (Caddy, nginx, Traefik) terminating TLS fo
### Round-robin mode ### Round-robin mode
Set `posting.round_robin.enabled: true` to switch Pete from "post on classify" to a paced rotation. On each tick (`interval_hours`, default 4) Pete picks the newest unposted classified story routed to the next channel in sorted order, posting through the existing queue. Empty channels are skipped; the rotation pointer advances to whichever channel actually posted, and `last_channel` / `last_tick_at` are persisted so restarts don't reset the cycle. Rotating by channel (not by source) guarantees variety even when one channel has many more feeds than the others. Set `posting.round_robin.enabled: true` to switch Pete from "post immediately" to a paced rotation. On each tick (`interval_hours`, default 4) Pete picks the newest unposted story routed to the next channel in sorted order, posting through the existing queue. Empty channels are skipped; the rotation pointer advances to whichever channel actually posted, and `last_channel` / `last_tick_at` are persisted so restarts don't reset the cycle. Rotating by channel (not by source) guarantees variety even when one channel has many more feeds than the others.
With a 4-hour cadence Pete will post at most 6 stories/day, so set `posting.daily_cap_total` to 0 (or ≥6) when enabling this — otherwise the daily cap can silently swallow a tick. With a 4-hour cadence Pete will post at most 6 stories/day, so set `posting.daily_cap_total` to 0 (or ≥6) when enabling this — otherwise the daily cap can silently swallow a tick.
@@ -109,15 +106,9 @@ Set `PETE_PASSWORD` in your environment or a `.env` file for the Matrix password
## Architecture ## Architecture
``` ```
RSS Feed → Poller → GUID Dedup → Canonical/Headline Dedup → Article Fetch → Store → Classifier → Semantic Dedup → Image Validate → Queue → Matrix RSS Feed → Poller → GUID/Canonical/Headline Dedup → Article Fetch → Store → Route by direct_route → Image Validate → Queue → Matrix
Paywall? → Wayback Direct Route (no LLM) Paywall? → Wayback snapshot
or
Ollama /api/chat (Tier 1 routing)
or
Keyword Gating → Ollama (Tier 2)
Posted message ← ❓ reaction → Explainer → Article Fetch → Ollama summary → Threaded reply
``` ```
### Packages ### Packages
@@ -126,11 +117,10 @@ Posted message ← ❓ reaction → Explainer → Article Fetch → Ollama summa
|---|---| |---|---|
| `internal/config` | YAML config loading with `${ENV}` expansion | | `internal/config` | YAML config loading with `${ENV}` expansion |
| `internal/storage` | SQLite with WAL, FTS5, all queries | | `internal/storage` | SQLite with WAL, FTS5, all queries |
| `internal/ingestion` | Per-source RSS polling, feed parsing, image validation | | `internal/ingestion` | Per-source RSS polling, feed parsing, image validation, paywall detection |
| `internal/classifier` | Ollama client, Tier 1/2 prompts, JSON repair, keyword gating | | `internal/dedup` | Canonical URL + headline normalization helpers |
| `internal/matrix` | Password auth with device persistence, posting, threaded replies, reaction + message listener | | `internal/matrix` | Password auth with device persistence, posting, threaded replies, reaction + message listener |
| `internal/poster` | Per-channel metered release queue, reaction tracking, callback hook | | `internal/poster` | Per-channel metered release queue, reaction tracking |
| `internal/explainer` | ❓-reaction → article fetch → Ollama summary → threaded reply |
| `internal/scheduler` | Round-robin posting scheduler: paced rotation across channels when enabled | | `internal/scheduler` | Round-robin posting scheduler: paced rotation across channels when enabled |
| `internal/web` | Read-only HTTP UI: Tailwind templates, day/night palette, per-channel feed pages | | `internal/web` | Read-only HTTP UI: Tailwind templates, day/night palette, per-channel feed pages |
@@ -143,8 +133,6 @@ Lede text from feed
`source name` `source name`
``` ```
Gaming stories append platform tags: `` `source` · `nintendo-switch` · `pc` ``
## Testing ## Testing
```bash ```bash

View File

@@ -11,11 +11,6 @@ matrix:
politics: "!politicsroomid:matrix.example.org" politics: "!politicsroomid:matrix.example.org"
gaming: "!gamingroomid:matrix.example.org" gaming: "!gamingroomid:matrix.example.org"
ollama:
base_url: "http://localhost:11434"
model: "gemma4:27b"
timeout_seconds: 30
posting: posting:
min_interval_seconds: 300 min_interval_seconds: 300
burst_cap_count: 3 burst_cap_count: 3
@@ -23,12 +18,11 @@ posting:
daily_cap_total: 5 # hard global cap across ALL channels (rolling 24h); 0 disables daily_cap_total: 5 # hard global cap across ALL channels (rolling 24h); 0 disables
round_robin: round_robin:
enabled: false # when true, replaces immediate posting with paced rotation enabled: false # when true, replaces immediate posting with paced rotation
interval_hours: 4 # one story per N hours, cycling through sources in config order interval_hours: 4 # one story per N hours, cycling through channels in sorted order
storage: storage:
db_path: "./data/pete.db" db_path: "./data/pete.db"
recent_window_hours: 24 recent_window_hours: 24
classification_log_days: 7
web: web:
enabled: true enabled: true
@@ -36,28 +30,20 @@ web:
site_title: "Pete" site_title: "Pete"
base_url: "https://news.parodia.dev" base_url: "https://news.parodia.dev"
# Every enabled source MUST set direct_route to a key from matrix.channels above.
# There is no automatic classification — Pete posts each story to its configured channel.
sources: sources:
- name: "The Guardian — World" - name: "The Guardian — World"
feed_url: "https://www.theguardian.com/world/rss" feed_url: "https://www.theguardian.com/world/rss"
tier: 1 tier: 1
poll_interval_minutes: 20 poll_interval_minutes: 20
feed_hint: "politics"
direct_route: "politics" direct_route: "politics"
enabled: true enabled: true
- name: "The Guardian — Technology"
feed_url: "https://www.theguardian.com/technology/rss"
tier: 1
poll_interval_minutes: 20
feed_hint: "tech"
direct_route: null
enabled: true
- name: "The Guardian — Politics" - name: "The Guardian — Politics"
feed_url: "https://www.theguardian.com/politics/rss" feed_url: "https://www.theguardian.com/politics/rss"
tier: 1 tier: 1
poll_interval_minutes: 20 poll_interval_minutes: 20
feed_hint: "politics"
direct_route: "politics" direct_route: "politics"
enabled: true enabled: true
@@ -65,7 +51,6 @@ sources:
feed_url: "https://www.theguardian.com/us-news/rss" feed_url: "https://www.theguardian.com/us-news/rss"
tier: 1 tier: 1
poll_interval_minutes: 20 poll_interval_minutes: 20
feed_hint: "politics"
direct_route: "politics" direct_route: "politics"
enabled: true enabled: true
@@ -73,22 +58,12 @@ sources:
feed_url: "https://feeds.arstechnica.com/arstechnica/index" feed_url: "https://feeds.arstechnica.com/arstechnica/index"
tier: 1 tier: 1
poll_interval_minutes: 20 poll_interval_minutes: 20
feed_hint: "tech" direct_route: "tech"
direct_route: null
enabled: true
- name: "Ars Technica — Policy"
feed_url: "https://feeds.arstechnica.com/arstechnica/tech-policy"
tier: 1
poll_interval_minutes: 20
feed_hint: "tech+politics"
direct_route: null
enabled: true enabled: true
- name: "Time Extension" - name: "Time Extension"
feed_url: "https://www.timeextension.com/feeds/news" feed_url: "https://www.timeextension.com/feeds/news"
tier: 1 tier: 1
poll_interval_minutes: 20 poll_interval_minutes: 20
feed_hint: "gaming"
direct_route: "gaming" direct_route: "gaming"
enabled: true enabled: true

View File

@@ -1,171 +0,0 @@
package classifier
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"sync/atomic"
"pete/internal/ingestion"
"pete/internal/matrix"
"pete/internal/storage"
)
const recentHeadlineLimit = 50
// ClassifyResult is the unified output of the classification pipeline.
type ClassifyResult struct {
Channel string
Platforms []string
DuplicateOf *string
ShouldPost bool
Rationale string
}
// Classifier runs the classification pipeline for incoming stories.
type Classifier struct {
ollama *OllamaClient
matrixClient *matrix.Client
consecutiveFailures atomic.Int32
}
// NewClassifier creates a new classifier.
func NewClassifier(ollama *OllamaClient, matrixClient *matrix.Client) *Classifier {
return &Classifier{
ollama: ollama,
matrixClient: matrixClient,
}
}
// Classify runs the appropriate classification pipeline for a feed item.
// ctx is checked before any LLM call so shutdown aborts in-flight work.
func (c *Classifier) Classify(ctx context.Context, item *ingestion.FeedItem) (*ClassifyResult, error) {
// Direct routing — no LLM call needed
if item.DirectRoute != nil {
slog.Debug("direct route", "guid", item.GUID, "channel", *item.DirectRoute)
// Still check GUID dedup via recent headlines for semantic dedup
// but direct-routed stories always post
return &ClassifyResult{
Channel: *item.DirectRoute,
ShouldPost: true,
Rationale: "direct route from source config",
}, nil
}
// Get recent headlines for dedup context
recent, err := storage.GetRecentHeadlines(recentHeadlineLimit)
if err != nil {
slog.Error("failed to get recent headlines", "err", err)
recent = nil // continue without dedup context
}
if item.Tier == 2 {
return c.classifyTier2(ctx, item, recent)
}
return c.classifyTier1(ctx, item, recent)
}
func (c *Classifier) classifyTier1(ctx context.Context, item *ingestion.FeedItem, recent []storage.RecentHeadline) (*ClassifyResult, error) {
prompt := BuildTier1Prompt(item, recent)
raw, err := c.ollama.Generate(ctx, prompt)
if err != nil {
c.handleOllamaFailure(err)
return nil, fmt.Errorf("ollama generate: %w", err)
}
c.handleOllamaSuccess()
result, err := ParseTier1Response(raw)
if err != nil {
slog.Error("malformed LLM response", "guid", item.GUID, "raw", raw, "err", err)
return nil, fmt.Errorf("parse tier1: %w", err)
}
// Log classification
logResult(item.GUID, raw)
return &ClassifyResult{
Channel: result.Channel,
Platforms: result.Platforms,
DuplicateOf: result.DuplicateOf,
ShouldPost: true, // Tier 1 always posts
Rationale: result.Rationale,
}, nil
}
func (c *Classifier) classifyTier2(ctx context.Context, item *ingestion.FeedItem, recent []storage.RecentHeadline) (*ClassifyResult, error) {
// Run keyword gating first
gatingResult, reason := RunGating(item.Headline, item.Lede)
switch gatingResult {
case GatingDiscard:
slog.Info("tier2 gating: discard", "guid", item.GUID, "reason", reason)
logResult(item.GUID, fmt.Sprintf(`{"gating":"discard","reason":%q}`, reason))
return &ClassifyResult{
ShouldPost: false,
Rationale: reason,
}, nil
case GatingPost:
slog.Info("tier2 gating: definite post, routing via LLM", "guid", item.GUID, "reason", reason)
// Fall through to LLM for routing (but post=true is guaranteed)
}
// Stage 3 (ambiguous) or Stage 1 (definite post needing routing) — send to LLM
prompt := BuildTier2Prompt(item, recent)
raw, err := c.ollama.Generate(ctx, prompt)
if err != nil {
c.handleOllamaFailure(err)
return nil, fmt.Errorf("ollama generate: %w", err)
}
c.handleOllamaSuccess()
result, err := ParseTier2Response(raw)
if err != nil {
slog.Error("malformed LLM response", "guid", item.GUID, "raw", raw, "err", err)
return nil, fmt.Errorf("parse tier2: %w", err)
}
logResult(item.GUID, raw)
// If gating said definite post, override LLM's post decision
shouldPost := result.Post
if gatingResult == GatingPost {
shouldPost = true
}
return &ClassifyResult{
Channel: result.Channel,
Platforms: result.Platforms,
DuplicateOf: result.DuplicateOf,
ShouldPost: shouldPost,
Rationale: result.Rationale,
}, nil
}
func (c *Classifier) handleOllamaFailure(err error) {
n := c.consecutiveFailures.Add(1)
slog.Error("ollama failure", "err", err, "consecutive", n)
if n == 3 && c.matrixClient != nil {
c.matrixClient.SendAdminWarning(
fmt.Sprintf("Ollama unavailable for 3 consecutive attempts: %v", err))
}
}
func (c *Classifier) handleOllamaSuccess() {
c.consecutiveFailures.Store(0)
}
func logResult(guid, raw string) {
// Try to compact the JSON for cleaner logging
var compact json.RawMessage
if err := json.Unmarshal([]byte(raw), &compact); err == nil {
storage.InsertClassificationLog(guid, string(compact))
} else {
storage.InsertClassificationLog(guid, raw)
}
}

View File

@@ -1,77 +0,0 @@
package classifier
import "regexp"
// GatingResult indicates the outcome of keyword-based gating for Tier 2 stories.
type GatingResult int
const (
GatingPost GatingResult = iota // Stage 1: definite post
GatingDiscard // Stage 2: definite discard
GatingAmbiguous // Stage 3: escalate to LLM
)
// Stage 1: Definite post patterns — stories with clear public significance.
// Patterns use multi-word context to reduce false positives (e.g. "Star Wars" won't match).
var postPatterns = []*regexp.Regexp{
// Named heads of state / major international bodies
regexp.MustCompile(`(?i)\b(president|prime minister|chancellor|pope)\s+\w+`),
regexp.MustCompile(`(?i)\b(UN|United Nations|WHO|NATO|EU|European Union|IMF|World Bank)\b`),
// Casualty language with numbers
regexp.MustCompile(`(?i)\b\d+\s*(dead|killed|injured|casualties|wounded)\b`),
regexp.MustCompile(`(?i)\b(dead|killed|injured|casualties|wounded)\s*\d+\b`),
// Disaster language — require a location-like context word nearby
regexp.MustCompile(`(?i)\b(earthquake|hurricane|typhoon|tsunami|wildfire|tornado)\b.*\b(hit|struck|destroyed|devastated|kills|deaths)\b`),
// Conflict language — require multi-word phrases to avoid "Star Wars", "culture war"
regexp.MustCompile(`(?i)\b(military invasion|armed coup|airstrike\s+\w+|assassination\s+\w+|car bombing|suicide bombing)\b`),
regexp.MustCompile(`(?i)\bwar\s+(in|on|with|against|between)\b`),
// Crisis language — require context word nearby
regexp.MustCompile(`(?i)\b(state of emergency|economic crisis|economic collapse|disease outbreak|pandemic declared|pandemic phase|famine)\b`),
// Major financial scale
regexp.MustCompile(`(?i)\$\d+\s*(bn|tn|billion|trillion)\b`),
regexp.MustCompile(`(?i)\b(trillion|billion)\s*(dollar|euro|pound)\b`),
}
// Stage 2: Definite discard patterns — stories unlikely to warrant posting.
var discardPatterns = []*regexp.Regexp{
// Local/municipal scope
regexp.MustCompile(`(?i)\b(city council|municipal|town hall|zoning|local police)\b`),
// Sports scores and routine results
regexp.MustCompile(`(?i)\b(final score|match result|league standings|defeated|innings)\b`),
regexp.MustCompile(`(?i)\b\d+-\d+\s*(win|loss|defeat|victory)\b`),
// Entertainment/celebrity lifestyle
regexp.MustCompile(`(?i)\b(red carpet|award show|celebrity gossip|dating rumor|relationship drama|album release)\b`),
// Routine corporate earnings
regexp.MustCompile(`(?i)\b(quarterly earnings|Q[1-4] results|earnings call|fiscal quarter)\b`),
}
// RunGating evaluates Tier 2 gating stages on headline + lede.
// Returns the gating result and a reason string for logging.
func RunGating(headline, lede string) (GatingResult, string) {
combined := headline + " " + lede
// Stage 1: Definite post
for _, re := range postPatterns {
if re.MatchString(combined) {
return GatingPost, "stage1: " + re.String()
}
}
// Stage 2: Definite discard
for _, re := range discardPatterns {
if re.MatchString(combined) {
return GatingDiscard, "stage2: " + re.String()
}
}
// Stage 3: Ambiguous — escalate to LLM
return GatingAmbiguous, "stage3: no keyword match"
}

View File

@@ -1,74 +0,0 @@
package classifier
import "testing"
func TestGating_Stage1_DefinitePost(t *testing.T) {
tests := []struct {
name string
headline string
lede string
}{
{"head of state", "President signs new climate bill", "The president signed legislation today."},
{"international body", "WHO declares new pandemic phase", "The World Health Organization announced a new phase."},
{"casualties", "12 killed in factory explosion", "Authorities report 12 dead after an explosion."},
{"disaster", "Earthquake strikes northern Japan", "A major earthquake hit the region early Tuesday."},
{"conflict", "Airstrike hits hospital in conflict zone", "An airstrike damaged a hospital."},
{"crisis", "Economic crisis deepens in Argentina", "The ongoing crisis has intensified."},
{"financial scale", "EU approves $50bn climate fund", "The European Union approved funding."},
{"assassination", "Assassination attempt on opposition leader", "An armed assailant attacked the leader."},
{"NATO", "NATO expands eastern flank deployment", "Alliance members agreed to new deployments."},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result, reason := RunGating(tt.headline, tt.lede)
if result != GatingPost {
t.Errorf("expected GatingPost, got %d (reason: %s)", result, reason)
}
})
}
}
func TestGating_Stage2_DefiniteDiscard(t *testing.T) {
tests := []struct {
name string
headline string
lede string
}{
{"local government", "City council approves new parking meters", "The council voted 5-3 in favor."},
{"sports scores", "Lakers defeat Celtics 112-98 in regular season", "Final score 112-98."},
{"celebrity gossip", "Celebrity couple spotted on red carpet at Cannes", "The pair arrived together."},
{"routine earnings", "Regional bank reports Q2 results slightly above estimates", "Quarterly earnings were released."},
{"entertainment", "Pop star announces new album release date", "Fans have been waiting since 2023."},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result, reason := RunGating(tt.headline, tt.lede)
if result != GatingDiscard {
t.Errorf("expected GatingDiscard, got %d (reason: %s)", result, reason)
}
})
}
}
func TestGating_Stage3_Ambiguous(t *testing.T) {
tests := []struct {
name string
headline string
lede string
}{
{"tech product", "Apple unveils new MacBook Pro with M5 chip", "The new laptops feature significant performance gains."},
{"policy nuanced", "Government considers new data privacy framework", "Ministers met to discuss potential changes."},
{"science", "Researchers discover high-temperature superconductor", "A team at MIT published findings in Nature."},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result, reason := RunGating(tt.headline, tt.lede)
if result != GatingAmbiguous {
t.Errorf("expected GatingAmbiguous, got %d (reason: %s)", result, reason)
}
})
}
}

View File

@@ -1,120 +0,0 @@
package classifier
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"strings"
"time"
"pete/internal/config"
)
// OllamaClient handles communication with the Ollama API.
type OllamaClient struct {
baseURL string
model string
httpClient *http.Client
}
type chatRequest struct {
Model string `json:"model"`
Messages []chatMessage `json:"messages"`
Stream bool `json:"stream"`
Format string `json:"format"`
}
type chatMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatResponse struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
}
// NewOllamaClient creates an Ollama API client from config.
func NewOllamaClient(cfg config.OllamaConfig) *OllamaClient {
return &OllamaClient{
baseURL: strings.TrimRight(cfg.BaseURL, "/"),
model: cfg.Model,
httpClient: &http.Client{
Timeout: time.Duration(cfg.TimeoutSeconds) * time.Second,
},
}
}
// Generate sends a prompt to Ollama via the chat API and returns the raw
// response text. Honors ctx cancellation — callers can abort an in-flight
// call instead of waiting for the configured timeout.
func (o *OllamaClient) Generate(ctx context.Context, prompt string) (string, error) {
return o.call(ctx,
"You are a JSON-only news classification API. Respond with valid JSON only. No preamble, no explanation, no markdown.",
prompt,
"json",
)
}
// GenerateText sends a system + user prompt to Ollama and returns the raw
// text response. Unlike Generate, no JSON mode is forced.
func (o *OllamaClient) GenerateText(ctx context.Context, systemPrompt, userPrompt string) (string, error) {
return o.call(ctx, systemPrompt, userPrompt, "")
}
func (o *OllamaClient) call(ctx context.Context, systemPrompt, userPrompt, format string) (string, error) {
reqBody := chatRequest{
Model: o.model,
Messages: []chatMessage{
{Role: "system", Content: systemPrompt},
{Role: "user", Content: userPrompt},
},
Stream: false,
Format: format,
}
body, err := json.Marshal(reqBody)
if err != nil {
return "", fmt.Errorf("marshal request: %w", err)
}
url := o.baseURL + "/api/chat"
slog.Debug("ollama: calling", "url", url, "model", o.model)
req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
if err != nil {
return "", fmt.Errorf("ollama request build: %w", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := o.httpClient.Do(req)
if err != nil {
return "", fmt.Errorf("ollama request: %w", err)
}
defer resp.Body.Close()
respBody, err := io.ReadAll(io.LimitReader(resp.Body, 1*1024*1024)) // 1MB cap
if err != nil {
return "", fmt.Errorf("read ollama response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("ollama status %d: %s", resp.StatusCode, string(respBody))
}
var chatResp chatResponse
if err := json.Unmarshal(respBody, &chatResp); err != nil {
return "", fmt.Errorf("decode ollama response: %w (body: %s)", err, truncate(string(respBody), 200))
}
result := strings.TrimSpace(chatResp.Message.Content)
if result == "" {
return "", fmt.Errorf("ollama returned empty response (body: %s)", truncate(string(respBody), 200))
}
return result, nil
}

View File

@@ -1,123 +0,0 @@
package classifier
import (
"encoding/json"
"fmt"
"regexp"
"strings"
)
// Tier1Result is the LLM response for Tier 1 (routing only) stories.
type Tier1Result struct {
Channel string `json:"channel"`
Platforms []string `json:"platforms"`
DuplicateOf *string `json:"duplicate_of"`
Rationale string `json:"rationale"`
}
// Tier2Result is the LLM response for Tier 2 (gating + routing) stories.
type Tier2Result struct {
Post bool `json:"post"`
Channel string `json:"channel"`
Platforms []string `json:"platforms"`
DuplicateOf *string `json:"duplicate_of"`
Rationale string `json:"rationale"`
}
var validChannels = map[string]bool{
"tech": true,
"politics": true,
"gaming": true,
}
var validPlatforms = map[string]bool{
"nintendo-switch": true,
"playstation": true,
"xbox": true,
"pc": true,
"retro": true,
"mobile": true,
"multi": true,
}
var thinkRe = regexp.MustCompile(`(?s)<think>.*?</think>`)
var fenceRe = regexp.MustCompile("(?s)```(?:json)?\\s*(.+?)\\s*```")
var trailingCommaRe = regexp.MustCompile(`,\s*([}\]])`)
// ParseTier1Response parses an LLM response into a Tier1Result.
func ParseTier1Response(raw string) (*Tier1Result, error) {
cleaned := repairJSON(raw)
var result Tier1Result
if err := json.Unmarshal([]byte(cleaned), &result); err != nil {
return nil, fmt.Errorf("parse tier1 response: %w (raw: %s)", err, truncate(raw, 200))
}
if !validChannels[result.Channel] {
return nil, fmt.Errorf("invalid channel: %q", result.Channel)
}
result.Platforms = filterPlatforms(result.Platforms)
return &result, nil
}
// ParseTier2Response parses an LLM response into a Tier2Result.
func ParseTier2Response(raw string) (*Tier2Result, error) {
cleaned := repairJSON(raw)
var result Tier2Result
if err := json.Unmarshal([]byte(cleaned), &result); err != nil {
return nil, fmt.Errorf("parse tier2 response: %w (raw: %s)", err, truncate(raw, 200))
}
// Always validate channel — gating may override post=false to true later
if !validChannels[result.Channel] {
if result.Post {
return nil, fmt.Errorf("invalid channel: %q", result.Channel)
}
// Default to "politics" for discarded stories with garbage channels
result.Channel = "politics"
}
result.Platforms = filterPlatforms(result.Platforms)
return &result, nil
}
// repairJSON attempts to fix common LLM output issues.
func repairJSON(raw string) string {
s := strings.TrimSpace(raw)
// Strip <think>...</think> blocks (some models include reasoning)
s = thinkRe.ReplaceAllString(s, "")
// Strip markdown code fences
if matches := fenceRe.FindStringSubmatch(s); len(matches) > 1 {
s = matches[1]
}
s = strings.TrimSpace(s)
// Remove trailing commas before closing braces/brackets (outside quoted strings)
s = trailingCommaRe.ReplaceAllString(s, "$1")
return s
}
func filterPlatforms(platforms []string) []string {
var valid []string
for _, p := range platforms {
p = strings.TrimSpace(strings.ToLower(p))
if validPlatforms[p] {
valid = append(valid, p)
}
}
return valid
}
func truncate(s string, n int) string {
runes := []rune(s)
if len(runes) <= n {
return s
}
return string(runes[:n]) + "..."
}

View File

@@ -1,156 +0,0 @@
package classifier
import (
"testing"
)
func TestParseTier1Response_Clean(t *testing.T) {
raw := `{"channel": "tech", "platforms": [], "duplicate_of": null, "rationale": "GPU launch story"}`
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if result.Channel != "tech" {
t.Errorf("channel = %q, want tech", result.Channel)
}
if len(result.Platforms) != 0 {
t.Errorf("platforms = %v, want empty", result.Platforms)
}
if result.DuplicateOf != nil {
t.Errorf("duplicate_of = %v, want nil", result.DuplicateOf)
}
}
func TestParseTier1Response_WithPlatforms(t *testing.T) {
raw := `{"channel":"gaming","platforms":["nintendo-switch","pc"],"duplicate_of":null,"rationale":"Switch 2 launch"}`
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if result.Channel != "gaming" {
t.Errorf("channel = %q", result.Channel)
}
if len(result.Platforms) != 2 {
t.Fatalf("platforms = %v, want 2", result.Platforms)
}
if result.Platforms[0] != "nintendo-switch" || result.Platforms[1] != "pc" {
t.Errorf("platforms = %v", result.Platforms)
}
}
func TestParseTier1Response_WithDuplicate(t *testing.T) {
raw := `{"channel":"politics","platforms":[],"duplicate_of":"guid-abc-123","rationale":"same event"}`
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if result.DuplicateOf == nil || *result.DuplicateOf != "guid-abc-123" {
t.Errorf("duplicate_of = %v", result.DuplicateOf)
}
}
func TestParseTier1Response_InvalidChannel(t *testing.T) {
raw := `{"channel":"sports","platforms":[],"duplicate_of":null,"rationale":"test"}`
_, err := ParseTier1Response(raw)
if err == nil {
t.Error("expected error for invalid channel")
}
}
func TestParseTier1Response_FiltersInvalidPlatforms(t *testing.T) {
raw := `{"channel":"gaming","platforms":["nintendo-switch","fake-platform","pc","also-fake"],"duplicate_of":null,"rationale":"test"}`
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if len(result.Platforms) != 2 {
t.Fatalf("platforms = %v, want [nintendo-switch pc]", result.Platforms)
}
}
func TestParseTier2Response_PostTrue(t *testing.T) {
raw := `{"post":true,"channel":"politics","platforms":[],"duplicate_of":null,"rationale":"major world event"}`
result, err := ParseTier2Response(raw)
if err != nil {
t.Fatal(err)
}
if !result.Post {
t.Error("post = false, want true")
}
if result.Channel != "politics" {
t.Errorf("channel = %q", result.Channel)
}
}
func TestParseTier2Response_PostFalse(t *testing.T) {
raw := `{"post":false,"platforms":[],"duplicate_of":null,"rationale":"local news"}`
result, err := ParseTier2Response(raw)
if err != nil {
t.Fatal(err)
}
if result.Post {
t.Error("post = true, want false")
}
}
func TestRepairJSON_MarkdownFences(t *testing.T) {
raw := "```json\n{\"channel\": \"tech\", \"platforms\": [], \"duplicate_of\": null, \"rationale\": \"test\"}\n```"
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if result.Channel != "tech" {
t.Errorf("channel = %q", result.Channel)
}
}
func TestRepairJSON_ThinkBlocks(t *testing.T) {
raw := "<think>I need to classify this as tech because it's about GPUs</think>\n{\"channel\": \"tech\", \"platforms\": [], \"duplicate_of\": null, \"rationale\": \"GPU story\"}"
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if result.Channel != "tech" {
t.Errorf("channel = %q", result.Channel)
}
}
func TestRepairJSON_TrailingCommas(t *testing.T) {
raw := `{"channel": "politics", "platforms": [], "duplicate_of": null, "rationale": "test",}`
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if result.Channel != "politics" {
t.Errorf("channel = %q", result.Channel)
}
}
func TestRepairJSON_TrailingCommaInArray(t *testing.T) {
raw := `{"channel": "gaming", "platforms": ["pc", "xbox",], "duplicate_of": null, "rationale": "test"}`
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if len(result.Platforms) != 2 {
t.Errorf("platforms = %v", result.Platforms)
}
}
func TestRepairJSON_CombinedIssues(t *testing.T) {
raw := "<think>reasoning here</think>\n```json\n{\"channel\": \"tech\", \"platforms\": [],\"duplicate_of\": null, \"rationale\": \"test\",}\n```"
result, err := ParseTier1Response(raw)
if err != nil {
t.Fatal(err)
}
if result.Channel != "tech" {
t.Errorf("channel = %q", result.Channel)
}
}
func TestRepairJSON_GarbageInput(t *testing.T) {
_, err := ParseTier1Response("I think this should go to tech because reasons")
if err == nil {
t.Error("expected error for garbage input")
}
}

View File

@@ -1,110 +0,0 @@
package classifier
import (
"fmt"
"strings"
"pete/internal/ingestion"
"pete/internal/storage"
)
// BuildTier1Prompt builds the routing-only prompt for Tier 1 stories.
func BuildTier1Prompt(item *ingestion.FeedItem, recent []storage.RecentHeadline) string {
return fmt.Sprintf(`You are a news router for a community discussion platform with three channels: tech, politics, and gaming.
This story comes from a curated Tier 1 source. It will be posted. Your only job is to decide which channel it belongs in.
Choose exactly one channel:
- tech: technology products, software, hardware, industry — where technology itself is the primary subject
- gaming: games, gaming platforms, gaming industry, esports
- politics: everything else — political, policy, current events, public interest, significant world events
Politics is the default. When in doubt, choose politics.
Routing guidance:
- GPU launch, software release, hardware review → tech
- Game release, platform announcement, studio news → gaming
- Election, legislation, court ruling → politics
- AI regulation, antitrust action, surveillance policy → politics (policy is the story, not the technology)
- Death of a public figure → politics
- Violent crime, assassination attempt, arrest of public figure → politics
- Natural disaster, humanitarian crisis → politics
- Celebrity gossip, lifestyle with no broader significance → politics (still gets posted, still needs a home)
For gaming stories only: identify which platforms are relevant from this list:
nintendo-switch, playstation, xbox, pc, retro, mobile, multi
Return an empty array if no specific platform applies.
Also check: does this story cover the same underlying event as any story in the recent window? Different outlets covering the same event counts as a duplicate. If yes, set duplicate_of to that story's GUID. If no match, set duplicate_of to null.
Respond ONLY with valid JSON. No preamble, no explanation outside the JSON:
{
"channel": "tech|politics|gaming",
"platforms": [string],
"duplicate_of": string | null,
"rationale": "one sentence max, for logging only"
}
INCOMING STORY:
Source: %s
Tier: %d
Feed hint: %s
Headline: %s
Lede: %s
RECENT STORIES (last 24hr):
%s`, item.Source, item.Tier, item.FeedHint, item.Headline, item.Lede, formatRecentStories(recent))
}
// BuildTier2Prompt builds the gating + routing prompt for Tier 2 stories.
func BuildTier2Prompt(item *ingestion.FeedItem, recent []storage.RecentHeadline) string {
return fmt.Sprintf(`You are a news classifier for a community discussion platform with three channels: tech, politics, and gaming.
This story comes from a wire service. Unlike curated sources, wire feeds publish everything — not all stories warrant posting. First decide if this story is worth posting, then decide where.
Step 1 — Should this be posted?
Post if the story has clear public significance: major world events, policy decisions, significant deaths, natural disasters, major technology developments. Do not post: local news with no broader relevance, routine financial filings, minor sports results, entertainment fluff.
Step 2 — If posting, choose exactly one channel:
- tech: technology products, software, hardware, industry
- gaming: games, gaming platforms, gaming industry
- politics: everything else — the default
Politics is the default when in doubt.
For gaming stories only: identify relevant platforms from: nintendo-switch, playstation, xbox, pc, retro, mobile, multi
Also check for duplicates against the recent window. If this story covers the same underlying event as a recent story, set duplicate_of to that story's GUID.
Respond ONLY with valid JSON:
{
"post": true|false,
"channel": "tech|politics|gaming",
"platforms": [string],
"duplicate_of": string | null,
"rationale": "one sentence max, for logging only"
}
If post is false, channel may be omitted.
INCOMING STORY:
Source: %s
Tier: %d
Feed hint: %s
Headline: %s
Lede: %s
RECENT STORIES (last 24hr):
%s`, item.Source, item.Tier, item.FeedHint, item.Headline, item.Lede, formatRecentStories(recent))
}
func formatRecentStories(recent []storage.RecentHeadline) string {
if len(recent) == 0 {
return "(none)"
}
var sb strings.Builder
for _, h := range recent {
fmt.Fprintf(&sb, "[%s] %s — %s\n", h.GUID, h.Headline, h.Source)
}
return sb.String()
}

View File

@@ -14,7 +14,6 @@ var envBracketRe = regexp.MustCompile(`\$\{([^}]+)\}`)
type Config struct { type Config struct {
Matrix MatrixConfig `yaml:"matrix"` Matrix MatrixConfig `yaml:"matrix"`
Ollama OllamaConfig `yaml:"ollama"`
Posting PostingConfig `yaml:"posting"` Posting PostingConfig `yaml:"posting"`
Storage StorageConfig `yaml:"storage"` Storage StorageConfig `yaml:"storage"`
Web WebConfig `yaml:"web"` Web WebConfig `yaml:"web"`
@@ -40,12 +39,6 @@ type MatrixConfig struct {
Channels map[string]string `yaml:"channels"` Channels map[string]string `yaml:"channels"`
} }
type OllamaConfig struct {
BaseURL string `yaml:"base_url"`
Model string `yaml:"model"`
TimeoutSeconds int `yaml:"timeout_seconds"`
}
type PostingConfig struct { type PostingConfig struct {
MinIntervalSeconds int `yaml:"min_interval_seconds"` MinIntervalSeconds int `yaml:"min_interval_seconds"`
BurstCapCount int `yaml:"burst_cap_count"` BurstCapCount int `yaml:"burst_cap_count"`
@@ -58,27 +51,25 @@ type PostingConfig struct {
} }
// RoundRobinConfig switches Pete from immediate-on-classify posting to a // RoundRobinConfig switches Pete from immediate-on-classify posting to a
// paced rotation: one story per IntervalHours, picking the next source in // paced rotation: one story per IntervalHours, picking the next channel in
// config order that has a postable story (skip-and-advance, newest-first). // rotation order that has a postable story (skip-and-advance, newest-first).
type RoundRobinConfig struct { type RoundRobinConfig struct {
Enabled bool `yaml:"enabled"` Enabled bool `yaml:"enabled"`
IntervalHours int `yaml:"interval_hours"` IntervalHours int `yaml:"interval_hours"`
} }
type StorageConfig struct { type StorageConfig struct {
DBPath string `yaml:"db_path"` DBPath string `yaml:"db_path"`
RecentWindowHours int `yaml:"recent_window_hours"` RecentWindowHours int `yaml:"recent_window_hours"`
ClassificationLogDays int `yaml:"classification_log_days"`
} }
type SourceConfig struct { type SourceConfig struct {
Name string `yaml:"name"` Name string `yaml:"name"`
FeedURL string `yaml:"feed_url"` FeedURL string `yaml:"feed_url"`
Tier int `yaml:"tier"` Tier int `yaml:"tier"`
PollIntervalMinutes int `yaml:"poll_interval_minutes"` PollIntervalMinutes int `yaml:"poll_interval_minutes"`
FeedHint string `yaml:"feed_hint"` DirectRoute string `yaml:"direct_route"`
DirectRoute *string `yaml:"direct_route"` Enabled bool `yaml:"enabled"`
Enabled bool `yaml:"enabled"`
} }
func Load(path string) (*Config, error) { func Load(path string) (*Config, error) {
@@ -125,12 +116,6 @@ func (c *Config) validate() error {
if len(c.Matrix.Channels) == 0 { if len(c.Matrix.Channels) == 0 {
return fmt.Errorf("matrix.channels must have at least one entry") return fmt.Errorf("matrix.channels must have at least one entry")
} }
if c.Ollama.BaseURL == "" {
return fmt.Errorf("ollama.base_url is required")
}
if c.Ollama.Model == "" {
return fmt.Errorf("ollama.model is required")
}
if c.Storage.DBPath == "" { if c.Storage.DBPath == "" {
return fmt.Errorf("storage.db_path is required") return fmt.Errorf("storage.db_path is required")
} }
@@ -148,6 +133,15 @@ func (c *Config) validate() error {
if s.PollIntervalMinutes <= 0 { if s.PollIntervalMinutes <= 0 {
return fmt.Errorf("sources[%d].poll_interval_minutes must be > 0", i) return fmt.Errorf("sources[%d].poll_interval_minutes must be > 0", i)
} }
if !s.Enabled {
continue
}
if s.DirectRoute == "" {
return fmt.Errorf("sources[%d] (%s): direct_route is required for enabled sources", i, s.Name)
}
if _, ok := c.Matrix.Channels[s.DirectRoute]; !ok {
return fmt.Errorf("sources[%d] (%s): direct_route %q is not a configured matrix.channels key", i, s.Name, s.DirectRoute)
}
} }
return nil return nil
@@ -163,9 +157,6 @@ func (c *Config) applyDefaults() {
if c.Matrix.PickleKey == "" { if c.Matrix.PickleKey == "" {
c.Matrix.PickleKey = "pete_pickle_key" c.Matrix.PickleKey = "pete_pickle_key"
} }
if c.Ollama.TimeoutSeconds == 0 {
c.Ollama.TimeoutSeconds = 30
}
if c.Posting.MinIntervalSeconds == 0 { if c.Posting.MinIntervalSeconds == 0 {
c.Posting.MinIntervalSeconds = 300 c.Posting.MinIntervalSeconds = 300
} }
@@ -184,9 +175,6 @@ func (c *Config) applyDefaults() {
if c.Storage.RecentWindowHours == 0 { if c.Storage.RecentWindowHours == 0 {
c.Storage.RecentWindowHours = 24 c.Storage.RecentWindowHours = 24
} }
if c.Storage.ClassificationLogDays == 0 {
c.Storage.ClassificationLogDays = 7
}
if c.Web.ListenAddr == "" { if c.Web.ListenAddr == "" {
c.Web.ListenAddr = ":8080" c.Web.ListenAddr = ":8080"
} }

View File

@@ -19,16 +19,14 @@ matrix:
func TestLoadValid(t *testing.T) { func TestLoadValid(t *testing.T) {
yaml := validMatrix + ` yaml := validMatrix + `
ollama:
base_url: "http://localhost:11434"
model: "gemma4:27b"
timeout_seconds: 60
storage: storage:
db_path: "/tmp/test.db" db_path: "/tmp/test.db"
sources: sources:
- name: "Test Source" - name: "Test Source"
feed_url: "https://example.com/rss" feed_url: "https://example.com/rss"
tier: 1 tier: 1
poll_interval_minutes: 20
direct_route: "tech"
enabled: true enabled: true
` `
cfg := loadFromString(t, yaml) cfg := loadFromString(t, yaml)
@@ -39,15 +37,15 @@ sources:
if cfg.Matrix.UserID != "@pete:example.org" { if cfg.Matrix.UserID != "@pete:example.org" {
t.Errorf("user_id = %q", cfg.Matrix.UserID) t.Errorf("user_id = %q", cfg.Matrix.UserID)
} }
if cfg.Ollama.TimeoutSeconds != 60 {
t.Errorf("timeout = %d, want 60", cfg.Ollama.TimeoutSeconds)
}
if len(cfg.Sources) != 1 { if len(cfg.Sources) != 1 {
t.Fatalf("sources = %d, want 1", len(cfg.Sources)) t.Fatalf("sources = %d, want 1", len(cfg.Sources))
} }
if cfg.Sources[0].Name != "Test Source" { if cfg.Sources[0].Name != "Test Source" {
t.Errorf("source name = %q", cfg.Sources[0].Name) t.Errorf("source name = %q", cfg.Sources[0].Name)
} }
if cfg.Sources[0].DirectRoute != "tech" {
t.Errorf("direct_route = %q, want tech", cfg.Sources[0].DirectRoute)
}
} }
func TestEnvVarExpansion(t *testing.T) { func TestEnvVarExpansion(t *testing.T) {
@@ -60,9 +58,6 @@ matrix:
password: "${TEST_PETE_PASS}" password: "${TEST_PETE_PASS}"
channels: channels:
tech: "!tech:example.org" tech: "!tech:example.org"
ollama:
base_url: "http://localhost:11434"
model: "gemma4:27b"
storage: storage:
db_path: "/tmp/test.db" db_path: "/tmp/test.db"
sources: [] sources: []
@@ -76,15 +71,14 @@ sources: []
func TestDefaults(t *testing.T) { func TestDefaults(t *testing.T) {
yaml := validMatrix + ` yaml := validMatrix + `
ollama:
base_url: "http://localhost:11434"
model: "gemma4:27b"
storage: storage:
db_path: "/tmp/test.db" db_path: "/tmp/test.db"
sources: sources:
- name: "S" - name: "S"
feed_url: "https://example.com/rss" feed_url: "https://example.com/rss"
tier: 1 tier: 1
poll_interval_minutes: 20
direct_route: "tech"
enabled: true enabled: true
` `
cfg := loadFromString(t, yaml) cfg := loadFromString(t, yaml)
@@ -95,9 +89,6 @@ sources:
if cfg.Matrix.DisplayName != "Pete" { if cfg.Matrix.DisplayName != "Pete" {
t.Errorf("display_name default = %q, want Pete", cfg.Matrix.DisplayName) t.Errorf("display_name default = %q, want Pete", cfg.Matrix.DisplayName)
} }
if cfg.Ollama.TimeoutSeconds != 30 {
t.Errorf("timeout default = %d, want 30", cfg.Ollama.TimeoutSeconds)
}
if cfg.Posting.MinIntervalSeconds != 300 { if cfg.Posting.MinIntervalSeconds != 300 {
t.Errorf("min_interval default = %d, want 300", cfg.Posting.MinIntervalSeconds) t.Errorf("min_interval default = %d, want 300", cfg.Posting.MinIntervalSeconds)
} }
@@ -110,27 +101,6 @@ sources:
if cfg.Storage.RecentWindowHours != 24 { if cfg.Storage.RecentWindowHours != 24 {
t.Errorf("recent_window default = %d, want 24", cfg.Storage.RecentWindowHours) t.Errorf("recent_window default = %d, want 24", cfg.Storage.RecentWindowHours)
} }
if cfg.Storage.ClassificationLogDays != 7 {
t.Errorf("classification_log default = %d, want 7", cfg.Storage.ClassificationLogDays)
}
if cfg.Sources[0].PollIntervalMinutes != 20 {
t.Errorf("poll_interval default = %d, want 20", cfg.Sources[0].PollIntervalMinutes)
}
}
// shortMatrix returns a minimal valid matrix block for validation error tests.
func shortMatrix(overrides string) string {
base := `
matrix:
homeserver: "https://h"
user_id: "@p:h"
password: "pw"
channels: {tech: "!t:e"}
`
if overrides != "" {
return overrides
}
return base
} }
func TestValidationErrors(t *testing.T) { func TestValidationErrors(t *testing.T) {
@@ -143,7 +113,6 @@ matrix:
user_id: "@p:h" user_id: "@p:h"
password: "pw" password: "pw"
channels: {tech: "!t:e"} channels: {tech: "!t:e"}
ollama: {base_url: "http://l", model: "m"}
storage: {db_path: "/tmp/t.db"} storage: {db_path: "/tmp/t.db"}
`}, `},
{"missing user_id", ` {"missing user_id", `
@@ -151,7 +120,6 @@ matrix:
homeserver: "https://h" homeserver: "https://h"
password: "pw" password: "pw"
channels: {tech: "!t:e"} channels: {tech: "!t:e"}
ollama: {base_url: "http://l", model: "m"}
storage: {db_path: "/tmp/t.db"} storage: {db_path: "/tmp/t.db"}
`}, `},
{"missing password", ` {"missing password", `
@@ -159,7 +127,6 @@ matrix:
homeserver: "https://h" homeserver: "https://h"
user_id: "@p:h" user_id: "@p:h"
channels: {tech: "!t:e"} channels: {tech: "!t:e"}
ollama: {base_url: "http://l", model: "m"}
storage: {db_path: "/tmp/t.db"} storage: {db_path: "/tmp/t.db"}
`}, `},
{"no channels", ` {"no channels", `
@@ -167,25 +134,6 @@ matrix:
homeserver: "https://h" homeserver: "https://h"
user_id: "@p:h" user_id: "@p:h"
password: "pw" password: "pw"
ollama: {base_url: "http://l", model: "m"}
storage: {db_path: "/tmp/t.db"}
`},
{"missing ollama base_url", `
matrix:
homeserver: "https://h"
user_id: "@p:h"
password: "pw"
channels: {tech: "!t:e"}
ollama: {model: "m"}
storage: {db_path: "/tmp/t.db"}
`},
{"missing ollama model", `
matrix:
homeserver: "https://h"
user_id: "@p:h"
password: "pw"
channels: {tech: "!t:e"}
ollama: {base_url: "http://l"}
storage: {db_path: "/tmp/t.db"} storage: {db_path: "/tmp/t.db"}
`}, `},
{"missing db_path", ` {"missing db_path", `
@@ -194,7 +142,6 @@ matrix:
user_id: "@p:h" user_id: "@p:h"
password: "pw" password: "pw"
channels: {tech: "!t:e"} channels: {tech: "!t:e"}
ollama: {base_url: "http://l", model: "m"}
storage: {} storage: {}
`}, `},
{"invalid tier", ` {"invalid tier", `
@@ -203,12 +150,14 @@ matrix:
user_id: "@p:h" user_id: "@p:h"
password: "pw" password: "pw"
channels: {tech: "!t:e"} channels: {tech: "!t:e"}
ollama: {base_url: "http://l", model: "m"}
storage: {db_path: "/tmp/t.db"} storage: {db_path: "/tmp/t.db"}
sources: sources:
- name: "S" - name: "S"
feed_url: "https://e.com/rss" feed_url: "https://e.com/rss"
tier: 5 tier: 5
poll_interval_minutes: 20
direct_route: "tech"
enabled: true
`}, `},
{"missing source name", ` {"missing source name", `
matrix: matrix:
@@ -216,11 +165,42 @@ matrix:
user_id: "@p:h" user_id: "@p:h"
password: "pw" password: "pw"
channels: {tech: "!t:e"} channels: {tech: "!t:e"}
ollama: {base_url: "http://l", model: "m"}
storage: {db_path: "/tmp/t.db"} storage: {db_path: "/tmp/t.db"}
sources: sources:
- feed_url: "https://e.com/rss" - feed_url: "https://e.com/rss"
tier: 1 tier: 1
poll_interval_minutes: 20
direct_route: "tech"
enabled: true
`},
{"enabled source missing direct_route", `
matrix:
homeserver: "https://h"
user_id: "@p:h"
password: "pw"
channels: {tech: "!t:e"}
storage: {db_path: "/tmp/t.db"}
sources:
- name: "S"
feed_url: "https://e.com/rss"
tier: 1
poll_interval_minutes: 20
enabled: true
`},
{"direct_route not in channels", `
matrix:
homeserver: "https://h"
user_id: "@p:h"
password: "pw"
channels: {tech: "!t:e"}
storage: {db_path: "/tmp/t.db"}
sources:
- name: "S"
feed_url: "https://e.com/rss"
tier: 1
poll_interval_minutes: 20
direct_route: "gaming"
enabled: true
`}, `},
} }
@@ -237,37 +217,25 @@ sources:
} }
} }
func TestDirectRouteNullable(t *testing.T) { func TestDisabledSourceSkipsDirectRouteCheck(t *testing.T) {
// A disabled source without direct_route must not fail validation.
yaml := ` yaml := `
matrix: matrix:
homeserver: "https://h" homeserver: "https://h"
user_id: "@p:h" user_id: "@p:h"
password: "pw" password: "pw"
channels: {tech: "!t:e"} channels: {tech: "!t:e"}
ollama: {base_url: "http://l", model: "m"}
storage: {db_path: "/tmp/t.db"} storage: {db_path: "/tmp/t.db"}
sources: sources:
- name: "Direct" - name: "Off"
feed_url: "https://e.com/rss" feed_url: "https://e.com/rss"
tier: 1 tier: 1
direct_route: "politics" poll_interval_minutes: 20
enabled: true enabled: false
- name: "LLM"
feed_url: "https://e.com/rss2"
tier: 1
direct_route: null
enabled: true
` `
cfg := loadFromString(t, yaml) cfg := loadFromString(t, yaml)
if cfg.Sources[0].DirectRoute != "" {
if cfg.Sources[0].DirectRoute == nil { t.Errorf("expected empty direct_route, got %q", cfg.Sources[0].DirectRoute)
t.Fatal("direct source should have non-nil DirectRoute")
}
if *cfg.Sources[0].DirectRoute != "politics" {
t.Errorf("direct_route = %q, want politics", *cfg.Sources[0].DirectRoute)
}
if cfg.Sources[1].DirectRoute != nil {
t.Errorf("null source should have nil DirectRoute, got %v", cfg.Sources[1].DirectRoute)
} }
} }

View File

@@ -1,196 +0,0 @@
// Package explainer summarizes a posted story on demand: when a user reacts ❓
// on one of Pete's posts, Pete fetches the article body, asks Ollama for a
// 3-bullet TL;DR, and replies in a thread rooted at the original post.
package explainer
import (
"context"
"fmt"
"html"
"log/slog"
"strings"
"sync"
"time"
"pete/internal/classifier"
"pete/internal/ingestion"
"pete/internal/matrix"
"pete/internal/storage"
"maunium.net/go/mautrix/id"
)
// questionReactions is the set of reaction keys that trigger a summary.
// Matrix delivers the key verbatim; we accept several question-y glyphs plus
// plain "?" so users don't have to hunt for the exact red question mark.
var questionReactions = map[string]bool{
"❓": true, // U+2753 red question mark
"❓️": true, // U+2753 + VS16 (emoji presentation)
"❔": true, // U+2754 white question mark
"❔️": true, // U+2754 + VS16
"⁉": true, // U+2049 exclamation question mark
"⁉️": true, // U+2049 + VS16 (emoji presentation)
"🤔": true, // U+1F914 thinking face
"?": true, // plain ascii
"": true, // U+FF1F fullwidth question mark
}
// IsQuestionReaction reports whether a reaction key should trigger a summary.
func IsQuestionReaction(key string) bool {
return questionReactions[key]
}
// cooldown for repeat-explain on the same story (per process).
const explainCooldown = 5 * time.Minute
// Explainer holds the dependencies needed to produce a story summary.
type Explainer struct {
mx *matrix.Client
ollama *classifier.OllamaClient
mu sync.Mutex
lastSeen map[string]time.Time // guid -> last explanation time
}
// New builds an Explainer wired to the Matrix client and Ollama backend.
func New(mx *matrix.Client, ollama *classifier.OllamaClient) *Explainer {
return &Explainer{
mx: mx,
ollama: ollama,
lastSeen: make(map[string]time.Time),
}
}
// Handle is the entry point used as a callback from poster.HandleReaction.
// It runs synchronously — callers should invoke it in a goroutine.
func (e *Explainer) Handle(roomID id.RoomID, rootEventID id.EventID, guid, channel, emoji string) {
if !IsQuestionReaction(emoji) {
return
}
if !e.acquire(guid) {
slog.Debug("explain: skipping recent duplicate", "guid", guid)
return
}
story, err := storage.GetStoryByGUID(guid)
if err != nil || story == nil {
slog.Warn("explain: story lookup failed", "guid", guid, "err", err)
return
}
body := ingestion.FetchArticleBody(story.ArticleURL)
if body == "" {
// Couldn't get the body — fall back to lede so the user at least gets something.
body = story.Lede
}
if body == "" {
slog.Warn("explain: no body or lede available", "guid", guid, "url", story.ArticleURL)
_ = e.mx.PostThreadedReply(channel, rootEventID,
"Couldn't fetch the article body to summarize.",
"<em>Couldn't fetch the article body to summarize.</em>")
return
}
summary, err := e.summarize(story.Headline, body)
if err != nil {
slog.Error("explain: ollama failed", "guid", guid, "err", err)
_ = e.mx.PostThreadedReply(channel, rootEventID,
"Sorry, I couldn't generate a summary right now.",
"<em>Sorry, I couldn't generate a summary right now.</em>")
return
}
plain, htmlBody := formatSummary(summary)
if err := e.mx.PostThreadedReply(channel, rootEventID, plain, htmlBody); err != nil {
slog.Error("explain: send reply failed", "guid", guid, "err", err)
return
}
slog.Info("explain: summary posted", "guid", guid, "channel", channel)
}
// acquire returns true if this guid hasn't been explained within the cooldown.
// Updates the timestamp on success.
func (e *Explainer) acquire(guid string) bool {
e.mu.Lock()
defer e.mu.Unlock()
if t, ok := e.lastSeen[guid]; ok && time.Since(t) < explainCooldown {
return false
}
e.lastSeen[guid] = time.Now()
return true
}
const summarySystem = `You summarize news articles in exactly 3 short bullet points for a chat reader.
Rules:
- Output ONLY the 3 bullets, each on its own line, prefixed with "- ".
- No preamble, no headings, no closing remarks.
- Each bullet is one concise sentence (under 25 words).
- Stick strictly to facts in the article. Do not speculate.`
func (e *Explainer) summarize(headline, body string) (string, error) {
// Reaction-driven flow has no parent ctx; bound the LLM call ourselves.
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
prompt := fmt.Sprintf("Headline: %s\n\nArticle body:\n%s", headline, body)
out, err := e.ollama.GenerateText(ctx, summarySystem, prompt)
if err != nil {
return "", err
}
return strings.TrimSpace(out), nil
}
// formatSummary converts the LLM's "- bullet\n- bullet" output into a Matrix
// plain + HTML pair. Any stray prefixes ("Summary:", "•", "*") are normalized
// to "-" bullets.
func formatSummary(raw string) (plain, htmlBody string) {
lines := strings.Split(raw, "\n")
var bullets []string
sawMarker := false
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" {
continue
}
// Skip header-ish lines the model sometimes adds
low := strings.ToLower(line)
if strings.HasPrefix(low, "summary:") || strings.HasPrefix(low, "tl;dr") {
continue
}
// Normalize common bullet markers; only count this line if it had one.
hadMarker := false
for _, prefix := range []string{"- ", "* ", "• ", "·"} {
if strings.HasPrefix(line, prefix) {
line = strings.TrimSpace(line[len(prefix):])
hadMarker = true
break
}
}
if !hadMarker {
continue
}
sawMarker = true
if line == "" {
continue
}
bullets = append(bullets, line)
}
if !sawMarker || len(bullets) == 0 {
// Model produced something but we couldn't parse bullets — pass through.
return raw, "<pre>" + html.EscapeString(raw) + "</pre>"
}
var pb, hb strings.Builder
hb.WriteString("<ul>")
for i, b := range bullets {
if i > 0 {
pb.WriteByte('\n')
}
pb.WriteString("• ")
pb.WriteString(b)
hb.WriteString("<li>")
hb.WriteString(html.EscapeString(b))
hb.WriteString("</li>")
}
hb.WriteString("</ul>")
return pb.String(), hb.String()
}

View File

@@ -1,110 +0,0 @@
package explainer
import (
"strings"
"testing"
)
func TestFormatSummary(t *testing.T) {
cases := []struct {
name string
in string
wantPlain string
wantHTML string
}{
{
name: "dash bullets",
in: "- First point.\n- Second point.\n- Third point.",
wantPlain: "• First point.\n• Second point.\n• Third point.",
wantHTML: "<ul><li>First point.</li><li>Second point.</li><li>Third point.</li></ul>",
},
{
name: "asterisk bullets",
in: "* alpha\n* beta",
wantPlain: "• alpha\n• beta",
wantHTML: "<ul><li>alpha</li><li>beta</li></ul>",
},
{
name: "unicode bullets",
in: "• one\n• two",
wantPlain: "• one\n• two",
wantHTML: "<ul><li>one</li><li>two</li></ul>",
},
{
name: "blank lines ignored",
in: "- a\n\n- b\n\n\n- c",
wantPlain: "• a\n• b\n• c",
wantHTML: "<ul><li>a</li><li>b</li><li>c</li></ul>",
},
{
name: "summary header stripped",
in: "Summary:\n- first\n- second",
wantPlain: "• first\n• second",
wantHTML: "<ul><li>first</li><li>second</li></ul>",
},
{
name: "tldr header stripped",
in: "TL;DR of the article\n- key point\n- other point",
wantPlain: "• key point\n• other point",
wantHTML: "<ul><li>key point</li><li>other point</li></ul>",
},
{
name: "single bullet",
in: "- just one",
wantPlain: "• just one",
wantHTML: "<ul><li>just one</li></ul>",
},
{
name: "html escaped in bullets",
in: `- A "quote" & <tag>`,
wantPlain: `• A "quote" & <tag>`,
wantHTML: "<ul><li>A &#34;quote&#34; &amp; &lt;tag&gt;</li></ul>",
},
{
name: "leading whitespace trimmed",
in: " - first\n - second ",
wantPlain: "• first\n• second",
wantHTML: "<ul><li>first</li><li>second</li></ul>",
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
gotPlain, gotHTML := formatSummary(c.in)
if gotPlain != c.wantPlain {
t.Errorf("plain:\n got %q\n want %q", gotPlain, c.wantPlain)
}
if gotHTML != c.wantHTML {
t.Errorf("html:\n got %q\n want %q", gotHTML, c.wantHTML)
}
})
}
}
func TestFormatSummary_NoBulletsPassthrough(t *testing.T) {
in := "Just a single paragraph with no bullets at all."
plain, htmlBody := formatSummary(in)
if plain != in {
t.Errorf("plain should pass through raw, got %q", plain)
}
if !strings.HasPrefix(htmlBody, "<pre>") || !strings.HasSuffix(htmlBody, "</pre>") {
t.Errorf("html should wrap raw in <pre>, got %q", htmlBody)
}
if !strings.Contains(htmlBody, "single paragraph") {
t.Errorf("html should contain the body text, got %q", htmlBody)
}
}
func TestIsQuestionReaction(t *testing.T) {
want := []string{"❓", "❓️", "❔", "❔️", "⁉", "⁉️", "🤔", "?", ""}
for _, k := range want {
if !IsQuestionReaction(k) {
t.Errorf("expected %q to trigger explain", k)
}
}
notQuestion := []string{"👍", "👎", "🔥", "", "??", "❤️"}
for _, k := range notQuestion {
if IsQuestionReaction(k) {
t.Errorf("expected %q to NOT trigger explain", k)
}
}
}

View File

@@ -16,7 +16,7 @@ const userAgent = "Pete/1.0 (newsbot; +https://github.com/reala-misaki/pete)"
var feedClient = &http.Client{Timeout: 30 * time.Second} var feedClient = &http.Client{Timeout: 30 * time.Second}
// FeedItem represents a parsed RSS item ready for classification. // FeedItem represents a parsed RSS item ready for routing.
type FeedItem struct { type FeedItem struct {
GUID string GUID string
Headline string Headline string
@@ -24,9 +24,7 @@ type FeedItem struct {
ImageURL string ImageURL string
ArticleURL string ArticleURL string
Source string Source string
FeedHint string DirectRoute string
Tier int
DirectRoute *string
} }
var ( var (

View File

@@ -11,9 +11,7 @@ import (
"pete/internal/storage" "pete/internal/storage"
) )
// ProcessFunc is called for each new feed item that needs classification. // ProcessFunc is called for each new feed item that needs routing.
// ctx is the poller's parent context; callbacks should pass it into any
// downstream LLM/HTTP work so shutdown aborts in-flight calls.
type ProcessFunc func(ctx context.Context, item *FeedItem) type ProcessFunc func(ctx context.Context, item *FeedItem)
// Poller manages per-source RSS polling goroutines. // Poller manages per-source RSS polling goroutines.
@@ -81,7 +79,6 @@ func (p *Poller) pollSource(ctx context.Context, src config.SourceConfig) {
"source", src.Name, "source", src.Name,
"failures", consecutiveFailures, "failures", consecutiveFailures,
) )
// TODO: send admin room warning via matrix client
} }
} else { } else {
consecutiveFailures = 0 consecutiveFailures = 0
@@ -163,11 +160,9 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
// Stamp source metadata onto the item // Stamp source metadata onto the item
items[i].Source = src.Name items[i].Source = src.Name
items[i].FeedHint = src.FeedHint
items[i].Tier = src.Tier
items[i].DirectRoute = src.DirectRoute items[i].DirectRoute = src.DirectRoute
// Store immediately (before classification) to prevent re-ingestion // Store immediately (before routing) to prevent re-ingestion
if err := storage.InsertStory(&storage.Story{ if err := storage.InsertStory(&storage.Story{
GUID: items[i].GUID, GUID: items[i].GUID,
Headline: items[i].Headline, Headline: items[i].Headline,
@@ -177,7 +172,6 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
URLCanonical: canonical, URLCanonical: canonical,
HeadlineNorm: headlineNorm, HeadlineNorm: headlineNorm,
Source: items[i].Source, Source: items[i].Source,
FeedHint: items[i].FeedHint,
SeenAt: time.Now().Unix(), SeenAt: time.Now().Unix(),
}); err != nil { }); err != nil {
slog.Error("failed to insert story", "guid", items[i].GUID, "err", err) slog.Error("failed to insert story", "guid", items[i].GUID, "err", err)
@@ -192,40 +186,5 @@ func (p *Poller) pollOnceWithErr(ctx context.Context, src config.SourceConfig) e
slog.Info("ingested new stories", "source", src.Name, "count", newCount) slog.Info("ingested new stories", "source", src.Name, "count", newCount)
} }
// Retry unclassified stories from this source only (cap at 20 to avoid runaway retries)
unclassified, err := storage.GetUnclassifiedStories(src.Name)
if err != nil {
slog.Error("failed to get unclassified stories", "err", err)
return nil
}
for _, s := range unclassified {
if ctx.Err() != nil {
return nil
}
// Skip stories we just ingested (they're already being processed above)
alreadyProcessed := false
for _, item := range items {
if item.GUID == s.GUID {
alreadyProcessed = true
break
}
}
if alreadyProcessed {
continue
}
p.process(ctx, &FeedItem{
GUID: s.GUID,
Headline: s.Headline,
Lede: s.Lede,
ImageURL: s.ImageURL,
ArticleURL: s.ArticleURL,
Source: s.Source,
FeedHint: s.FeedHint,
Tier: src.Tier,
DirectRoute: src.DirectRoute,
})
}
return nil return nil
} }

View File

@@ -33,7 +33,6 @@ func insertPostable(t *testing.T, guid, source, channel string, seenAt int64) {
Lede: "Lede for " + guid, Lede: "Lede for " + guid,
ArticleURL: "https://example.com/" + guid, ArticleURL: "https://example.com/" + guid,
Source: source, Source: source,
FeedHint: "tech",
SeenAt: seenAt, SeenAt: seenAt,
}); err != nil { }); err != nil {
t.Fatalf("InsertStory(%s): %v", guid, err) t.Fatalf("InsertStory(%s): %v", guid, err)

View File

@@ -101,15 +101,7 @@ func runMigrations(d *sql.DB) error {
} }
// RunMaintenance prunes stale data. Called periodically. // RunMaintenance prunes stale data. Called periodically.
func RunMaintenance(recentWindowHours, classificationLogDays int) { func RunMaintenance() {
recentCutoff := nowUnix() - int64(recentWindowHours*3600)
exec("prune recent_headlines",
`DELETE FROM recent_headlines WHERE seen_at < ?`, recentCutoff)
logCutoff := nowUnix() - int64(classificationLogDays*86400)
exec("prune classification_log",
`DELETE FROM classification_log WHERE logged_at < ?`, logCutoff)
// Prune old stories (30 days) and their post logs / reactions // Prune old stories (30 days) and their post logs / reactions
storyCutoff := nowUnix() - int64(30*86400) storyCutoff := nowUnix() - int64(30*86400)
exec("prune old stories", exec("prune old stories",

View File

@@ -29,9 +29,9 @@ func InsertStory(s *Story) error {
classified = 1 classified = 1
} }
_, err := Get().Exec( _, err := Get().Exec(
`INSERT INTO stories (guid, headline, lede, image_url, article_url, url_canonical, headline_norm, source, feed_hint, platforms, channel, classified, seen_at) `INSERT INTO stories (guid, headline, lede, image_url, article_url, url_canonical, headline_norm, source, platforms, channel, classified, seen_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
s.GUID, s.Headline, s.Lede, s.ImageURL, s.ArticleURL, nullIfEmpty(s.URLCanonical), nullIfEmpty(s.HeadlineNorm), s.Source, s.FeedHint, s.Platforms, s.Channel, classified, s.SeenAt, s.GUID, s.Headline, s.Lede, s.ImageURL, s.ArticleURL, nullIfEmpty(s.URLCanonical), nullIfEmpty(s.HeadlineNorm), s.Source, s.Platforms, s.Channel, classified, s.SeenAt,
) )
return err return err
} }
@@ -99,73 +99,18 @@ func MarkClassified(guid, channel, platforms string) {
channel, platforms, guid) channel, platforms, guid)
} }
// GetUnclassifiedStories returns stories from a specific source that were ingested but not yet classified.
func GetUnclassifiedStories(source string) ([]Story, error) {
rows, err := Get().Query(
`SELECT guid, headline, lede, image_url, article_url, source, feed_hint, platforms, seen_at
FROM stories WHERE classified = 0 AND source = ? ORDER BY seen_at ASC LIMIT 20`, source)
if err != nil {
return nil, err
}
defer rows.Close()
var stories []Story
for rows.Next() {
var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.FeedHint, &s.Platforms, &s.SeenAt); err != nil {
return nil, err
}
stories = append(stories, s)
}
return stories, rows.Err()
}
// GetStoryByGUID returns the full story record for a GUID, or nil if not found. // GetStoryByGUID returns the full story record for a GUID, or nil if not found.
func GetStoryByGUID(guid string) (*Story, error) { func GetStoryByGUID(guid string) (*Story, error) {
row := Get().QueryRow( row := Get().QueryRow(
`SELECT guid, headline, lede, image_url, article_url, source, feed_hint, platforms, channel, seen_at `SELECT guid, headline, lede, image_url, article_url, source, platforms, channel, seen_at
FROM stories WHERE guid = ?`, guid) FROM stories WHERE guid = ?`, guid)
var s Story var s Story
if err := row.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.FeedHint, &s.Platforms, &s.Channel, &s.SeenAt); err != nil { if err := row.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.SeenAt); err != nil {
return nil, err return nil, err
} }
return &s, nil return &s, nil
} }
// InsertRecentHeadline adds a headline to the dedup context window.
func InsertRecentHeadline(guid, headline, source string) {
exec("insert recent_headline",
`INSERT OR REPLACE INTO recent_headlines (guid, headline, source, seen_at) VALUES (?, ?, ?, ?)`,
guid, headline, source, nowUnix())
}
// GetRecentHeadlines returns the most recent headlines for LLM dedup context, capped at limit.
func GetRecentHeadlines(limit int) ([]RecentHeadline, error) {
rows, err := Get().Query(
`SELECT guid, headline, source, seen_at FROM recent_headlines ORDER BY seen_at DESC LIMIT ?`, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var headlines []RecentHeadline
for rows.Next() {
var h RecentHeadline
if err := rows.Scan(&h.GUID, &h.Headline, &h.Source, &h.SeenAt); err != nil {
return nil, err
}
headlines = append(headlines, h)
}
return headlines, rows.Err()
}
// InsertClassificationLog stores a classification result for tuning review.
func InsertClassificationLog(guid, resultJSON string) {
exec("insert classification_log",
`INSERT OR REPLACE INTO classification_log (guid, result, logged_at) VALUES (?, ?, ?)`,
guid, resultJSON, nowUnix())
}
// InsertPostLog records that a story was posted to a channel. // InsertPostLog records that a story was posted to a channel.
// Uses OR IGNORE to prevent duplicate posts for the same story+channel. // Uses OR IGNORE to prevent duplicate posts for the same story+channel.
func InsertPostLog(guid, channel, eventID, urlCanonical string) { func InsertPostLog(guid, channel, eventID, urlCanonical string) {
@@ -228,7 +173,7 @@ func InsertReaction(postGUID, channel, eventID, emoji, userID string, reactedAt
// row in post_log). Returns (nil, nil) when nothing qualifies. // row in post_log). Returns (nil, nil) when nothing qualifies.
func GetNewestPostableStory(source string) (*Story, error) { func GetNewestPostableStory(source string) (*Story, error) {
row := Get().QueryRow( row := Get().QueryRow(
`SELECT guid, headline, lede, image_url, article_url, source, feed_hint, platforms, channel, seen_at `SELECT guid, headline, lede, image_url, article_url, source, platforms, channel, seen_at
FROM stories FROM stories
WHERE classified = 1 WHERE classified = 1
AND source = ? AND source = ?
@@ -238,7 +183,7 @@ func GetNewestPostableStory(source string) (*Story, error) {
ORDER BY seen_at DESC ORDER BY seen_at DESC
LIMIT 1`, source) LIMIT 1`, source)
var s Story var s Story
if err := row.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.FeedHint, &s.Platforms, &s.Channel, &s.SeenAt); err != nil { if err := row.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.SeenAt); err != nil {
if errors.Is(err, sql.ErrNoRows) { if errors.Is(err, sql.ErrNoRows) {
return nil, nil return nil, nil
} }
@@ -253,7 +198,7 @@ func GetNewestPostableStory(source string) (*Story, error) {
// between ticks). Returns (nil, nil) when nothing qualifies. // between ticks). Returns (nil, nil) when nothing qualifies.
func GetNewestPostableStoryByChannel(channel string) (*Story, error) { func GetNewestPostableStoryByChannel(channel string) (*Story, error) {
row := Get().QueryRow( row := Get().QueryRow(
`SELECT guid, headline, lede, image_url, article_url, source, feed_hint, platforms, channel, seen_at `SELECT guid, headline, lede, image_url, article_url, source, platforms, channel, seen_at
FROM stories FROM stories
WHERE classified = 1 WHERE classified = 1
AND channel = ? AND channel = ?
@@ -261,7 +206,7 @@ func GetNewestPostableStoryByChannel(channel string) (*Story, error) {
ORDER BY seen_at DESC ORDER BY seen_at DESC
LIMIT 1`, channel) LIMIT 1`, channel)
var s Story var s Story
if err := row.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.FeedHint, &s.Platforms, &s.Channel, &s.SeenAt); err != nil { if err := row.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.SeenAt); err != nil {
if errors.Is(err, sql.ErrNoRows) { if errors.Is(err, sql.ErrNoRows) {
return nil, nil return nil, nil
} }
@@ -275,7 +220,7 @@ func GetNewestPostableStoryByChannel(channel string) (*Story, error) {
// channels (_discarded, _duplicate) are excluded. // channels (_discarded, _duplicate) are excluded.
func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error) { func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error) {
rows, err := Get().Query( rows, err := Get().Query(
`SELECT guid, headline, lede, image_url, article_url, source, feed_hint, platforms, channel, seen_at `SELECT guid, headline, lede, image_url, article_url, source, platforms, channel, seen_at
FROM stories FROM stories
WHERE classified = 1 WHERE classified = 1
AND channel = ? AND channel = ?
@@ -288,7 +233,7 @@ func ListClassifiedByChannel(channel string, limit, offset int) ([]Story, error)
var out []Story var out []Story
for rows.Next() { for rows.Next() {
var s Story var s Story
if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.FeedHint, &s.Platforms, &s.Channel, &s.SeenAt); err != nil { if err := rows.Scan(&s.GUID, &s.Headline, &s.Lede, &s.ImageURL, &s.ArticleURL, &s.Source, &s.Platforms, &s.Channel, &s.SeenAt); err != nil {
return nil, err return nil, err
} }
out = append(out, s) out = append(out, s)

View File

@@ -11,26 +11,12 @@ CREATE TABLE IF NOT EXISTS stories (
url_canonical TEXT, url_canonical TEXT,
headline_norm TEXT, headline_norm TEXT,
source TEXT NOT NULL, source TEXT NOT NULL,
feed_hint TEXT,
platforms TEXT, platforms TEXT,
channel TEXT, channel TEXT,
classified INTEGER NOT NULL DEFAULT 0, classified INTEGER NOT NULL DEFAULT 0,
seen_at INTEGER NOT NULL seen_at INTEGER NOT NULL
); );
CREATE TABLE IF NOT EXISTS recent_headlines (
guid TEXT PRIMARY KEY,
headline TEXT NOT NULL,
source TEXT NOT NULL,
seen_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS classification_log (
guid TEXT PRIMARY KEY,
result TEXT NOT NULL,
logged_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS post_log ( CREATE TABLE IF NOT EXISTS post_log (
id INTEGER PRIMARY KEY AUTOINCREMENT, id INTEGER PRIMARY KEY AUTOINCREMENT,
guid TEXT NOT NULL, guid TEXT NOT NULL,

View File

@@ -1,7 +1,6 @@
package storage package storage
import ( import (
"fmt"
"path/filepath" "path/filepath"
"testing" "testing"
) )
@@ -48,7 +47,6 @@ func TestInsertStoryAndGUIDSeen(t *testing.T) {
Lede: "Test lede sentence.", Lede: "Test lede sentence.",
ArticleURL: "https://example.com/article", ArticleURL: "https://example.com/article",
Source: "Test Source", Source: "Test Source",
FeedHint: "tech",
SeenAt: 1700000000, SeenAt: 1700000000,
} }
if err := InsertStory(s); err != nil { if err := InsertStory(s); err != nil {
@@ -79,63 +77,21 @@ func TestInsertStoryDuplicateGUID(t *testing.T) {
} }
} }
func TestMarkClassifiedAndGetUnclassified(t *testing.T) { func TestMarkClassified(t *testing.T) {
setupTestDB(t) setupTestDB(t)
InsertStory(&Story{ InsertStory(&Story{
GUID: "g1", Headline: "H1", ArticleURL: "https://a.com", Source: "S", SeenAt: 1, GUID: "g1", Headline: "H1", ArticleURL: "https://a.com", Source: "S", SeenAt: 1,
}) })
InsertStory(&Story{
GUID: "g2", Headline: "H2", ArticleURL: "https://b.com", Source: "S", SeenAt: 2,
})
unclassified, err := GetUnclassifiedStories("S")
if err != nil {
t.Fatal(err)
}
if len(unclassified) != 2 {
t.Fatalf("unclassified = %d, want 2", len(unclassified))
}
MarkClassified("g1", "tech", "[]") MarkClassified("g1", "tech", "[]")
unclassified, err = GetUnclassifiedStories("S") var channel string
if err != nil { var classified int
t.Fatal(err) Get().QueryRow(`SELECT channel, classified FROM stories WHERE guid = ?`, "g1").
} Scan(&channel, &classified)
if len(unclassified) != 1 { if channel != "tech" || classified != 1 {
t.Fatalf("unclassified = %d, want 1", len(unclassified)) t.Errorf("after MarkClassified: channel=%q classified=%d, want tech/1", channel, classified)
}
if unclassified[0].GUID != "g2" {
t.Errorf("remaining = %q, want g2", unclassified[0].GUID)
}
}
func TestRecentHeadlines(t *testing.T) {
setupTestDB(t)
// Insert with explicit timestamps to ensure ordering
db := Get()
db.Exec(`INSERT OR REPLACE INTO recent_headlines (guid, headline, source, seen_at) VALUES (?, ?, ?, ?)`,
"g1", "Headline One", "Source A", 1000)
db.Exec(`INSERT OR REPLACE INTO recent_headlines (guid, headline, source, seen_at) VALUES (?, ?, ?, ?)`,
"g2", "Headline Two", "Source B", 2000)
db.Exec(`INSERT OR REPLACE INTO recent_headlines (guid, headline, source, seen_at) VALUES (?, ?, ?, ?)`,
"g3", "Headline Three", "Source A", 3000)
headlines, err := GetRecentHeadlines(2)
if err != nil {
t.Fatal(err)
}
if len(headlines) != 2 {
t.Fatalf("headlines = %d, want 2", len(headlines))
}
// Should be most recent first
if headlines[0].GUID != "g3" {
t.Errorf("first = %q, want g3", headlines[0].GUID)
}
if headlines[1].GUID != "g2" {
t.Errorf("second = %q, want g2", headlines[1].GUID)
} }
} }
@@ -269,57 +225,6 @@ func TestInsertReaction_DuplicateIgnored(t *testing.T) {
} }
} }
func TestGetUnclassifiedStories_ScopedBySource(t *testing.T) {
setupTestDB(t)
InsertStory(&Story{GUID: "g1", Headline: "H1", ArticleURL: "https://a.com", Source: "SourceA", SeenAt: 1})
InsertStory(&Story{GUID: "g2", Headline: "H2", ArticleURL: "https://b.com", Source: "SourceB", SeenAt: 2})
InsertStory(&Story{GUID: "g3", Headline: "H3", ArticleURL: "https://c.com", Source: "SourceA", SeenAt: 3})
got, err := GetUnclassifiedStories("SourceA")
if err != nil {
t.Fatal(err)
}
if len(got) != 2 {
t.Fatalf("expected 2 stories for SourceA, got %d", len(got))
}
got, err = GetUnclassifiedStories("SourceB")
if err != nil {
t.Fatal(err)
}
if len(got) != 1 {
t.Fatalf("expected 1 story for SourceB, got %d", len(got))
}
got, err = GetUnclassifiedStories("SourceC")
if err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("expected 0 stories for SourceC, got %d", len(got))
}
}
func TestGetUnclassifiedStories_Limit20(t *testing.T) {
setupTestDB(t)
for i := 0; i < 30; i++ {
InsertStory(&Story{
GUID: fmt.Sprintf("g%d", i), Headline: fmt.Sprintf("H%d", i),
ArticleURL: "https://a.com", Source: "S", SeenAt: int64(i),
})
}
got, err := GetUnclassifiedStories("S")
if err != nil {
t.Fatal(err)
}
if len(got) != 20 {
t.Fatalf("expected 20 (capped), got %d", len(got))
}
}
func TestRunMaintenance(t *testing.T) { func TestRunMaintenance(t *testing.T) {
setupTestDB(t) setupTestDB(t)
db := Get() db := Get()
@@ -341,11 +246,7 @@ func TestRunMaintenance(t *testing.T) {
InsertReaction("old", "tech", "$r1", "👍", "@u:x", old) InsertReaction("old", "tech", "$r1", "👍", "@u:x", old)
InsertReaction("new", "tech", "$r2", "👍", "@u:x", now) InsertReaction("new", "tech", "$r2", "👍", "@u:x", now)
// Insert old + recent headlines RunMaintenance()
db.Exec(`INSERT OR REPLACE INTO recent_headlines (guid, headline, source, seen_at) VALUES (?, ?, ?, ?)`, "old", "Old", "S", old)
db.Exec(`INSERT OR REPLACE INTO recent_headlines (guid, headline, source, seen_at) VALUES (?, ?, ?, ?)`, "new", "New", "S", now)
RunMaintenance(24, 7)
// Old stories should be pruned // Old stories should be pruned
var count int var count int
@@ -363,11 +264,6 @@ func TestRunMaintenance(t *testing.T) {
if count != 1 { if count != 1 {
t.Errorf("reactions: expected 1, got %d", count) t.Errorf("reactions: expected 1, got %d", count)
} }
db.QueryRow(`SELECT COUNT(*) FROM recent_headlines`).Scan(&count)
if count != 1 {
t.Errorf("recent_headlines: expected 1, got %d", count)
}
} }
func TestFTS5Search(t *testing.T) { func TestFTS5Search(t *testing.T) {

View File

@@ -2,26 +2,17 @@ package storage
// Story is the core record for an ingested news item. // Story is the core record for an ingested news item.
type Story struct { type Story struct {
ID int64 ID int64
GUID string GUID string
Headline string Headline string
Lede string Lede string
ImageURL string ImageURL string
ArticleURL string ArticleURL string
URLCanonical string URLCanonical string
HeadlineNorm string HeadlineNorm string
Source string Source string
FeedHint string Platforms string // JSON array e.g. '["nintendo-switch","multi"]'
Platforms string // JSON array e.g. '["nintendo-switch","multi"]' Channel string
Channel string Classified bool
Classified bool SeenAt int64
SeenAt int64
}
// RecentHeadline is a lightweight record for the LLM dedup context window.
type RecentHeadline struct {
GUID string
Headline string
Source string
SeenAt int64
} }

84
main.go
View File

@@ -10,9 +10,7 @@ import (
"syscall" "syscall"
"time" "time"
"pete/internal/classifier"
"pete/internal/config" "pete/internal/config"
"pete/internal/explainer"
"pete/internal/ingestion" "pete/internal/ingestion"
"pete/internal/matrix" "pete/internal/matrix"
"pete/internal/poster" "pete/internal/poster"
@@ -43,7 +41,6 @@ func main() {
slog.Info("config loaded", slog.Info("config loaded",
"sources", len(cfg.Sources), "sources", len(cfg.Sources),
"channels", len(cfg.Matrix.Channels), "channels", len(cfg.Matrix.Channels),
"ollama_model", cfg.Ollama.Model,
) )
// Initialize database // Initialize database
@@ -72,17 +69,11 @@ func main() {
os.Exit(1) os.Exit(1)
} }
// Create Ollama client and classifier
ollamaClient := classifier.NewOllamaClient(cfg.Ollama)
cls := classifier.NewClassifier(ollamaClient, mx)
// Create post queue // Create post queue
queue := poster.NewQueue(cfg.Posting, mx) queue := poster.NewQueue(cfg.Posting, mx)
// Wire reaction handler + ❓ → summary hook // Wire reaction handler
mx.SetReactionHandler(poster.HandleReaction) mx.SetReactionHandler(poster.HandleReaction)
exp := explainer.New(mx, ollamaClient)
poster.SetReactionCallback(exp.Handle)
// Wire !post command: force-publish next queued story for the room's channel. // Wire !post command: force-publish next queued story for the room's channel.
mx.SetMessageHandler(func(channel string, roomID id.RoomID, eventID id.EventID, sender id.UserID, body string) { mx.SetMessageHandler(func(channel string, roomID id.RoomID, eventID id.EventID, sender id.UserID, body string) {
@@ -142,70 +133,37 @@ func main() {
roundRobinMode := cfg.Posting.RoundRobin.Enabled roundRobinMode := cfg.Posting.RoundRobin.Enabled
// Build the pipeline callback: classify → enqueue (or just classify+store // Pipeline callback: route the story to its direct_route channel, then
// when round-robin mode is enabled; the scheduler does the enqueueing). // either enqueue immediately or leave it for the round-robin scheduler.
processItem := func(ctx context.Context, item *ingestion.FeedItem) { processItem := func(_ context.Context, item *ingestion.FeedItem) {
result, err := cls.Classify(ctx, item) if item.DirectRoute == "" {
if err != nil { slog.Error("item has no direct_route, skipping", "guid", item.GUID, "source", item.Source)
slog.Error("classification failed, will retry next cycle",
"guid", item.GUID,
"err", err,
)
return // story stays unclassified for retry
}
// Add to recent headlines for future dedup
storage.InsertRecentHeadline(item.GUID, item.Headline, item.Source)
if !result.ShouldPost {
// Mark classified with sentinel channel so it's not retried
storage.MarkClassified(item.GUID, "_discarded", "[]") storage.MarkClassified(item.GUID, "_discarded", "[]")
slog.Info("story gated out",
"guid", item.GUID,
"rationale", result.Rationale,
)
return return
} }
// Mark classified with actual channel channel := item.DirectRoute
platforms := storage.MarshalPlatforms(result.Platforms) storage.MarkClassified(item.GUID, channel, "[]")
storage.MarkClassified(item.GUID, result.Channel, platforms)
if result.DuplicateOf != nil {
// Overwrite with sentinel so the round-robin scheduler doesn't pick
// this up later. Immediate-mode posting already returned above.
storage.MarkClassified(item.GUID, "_duplicate", "[]")
slog.Info("story deduplicated",
"guid", item.GUID,
"duplicate_of", *result.DuplicateOf,
)
return
}
if roundRobinMode { if roundRobinMode {
// Story stays in DB classified+postable; the scheduler will pick slog.Info("story routed, awaiting round-robin tick",
// it up on the next rotation tick. "guid", item.GUID, "source", item.Source, "channel", channel)
slog.Info("story classified, awaiting round-robin tick",
"guid", item.GUID, "source", item.Source, "channel", result.Channel)
return return
} }
// Validate image
imageURL := "" imageURL := ""
if item.ImageURL != "" && ingestion.ValidateImageURL(item.ImageURL) { if item.ImageURL != "" && ingestion.ValidateImageURL(item.ImageURL) {
imageURL = item.ImageURL imageURL = item.ImageURL
} }
// Enqueue for posting
queue.Enqueue(poster.QueueItem{ queue.Enqueue(poster.QueueItem{
GUID: item.GUID, GUID: item.GUID,
Headline: item.Headline, Headline: item.Headline,
Lede: item.Lede, Lede: item.Lede,
ImageURL: imageURL, ImageURL: imageURL,
ArticleURL: item.ArticleURL, ArticleURL: item.ArticleURL,
Source: item.Source, Source: item.Source,
Channel: result.Channel, Channel: channel,
Platforms: result.Platforms,
}) })
} }
@@ -238,7 +196,7 @@ func main() {
} }
// Run maintenance on startup // Run maintenance on startup
storage.RunMaintenance(cfg.Storage.RecentWindowHours, cfg.Storage.ClassificationLogDays) storage.RunMaintenance()
// Wait for shutdown signal // Wait for shutdown signal
sig := <-sigCh sig := <-sigCh
@@ -272,15 +230,13 @@ func runSeed(cfg *config.Config) {
Lede: item.Lede, Lede: item.Lede,
ImageURL: item.ImageURL, ImageURL: item.ImageURL,
ArticleURL: item.ArticleURL, ArticleURL: item.ArticleURL,
Source: src.Name, Source: src.Name,
FeedHint: src.FeedHint,
Classified: true, Classified: true,
SeenAt: timeNow(), SeenAt: timeNow(),
}); err != nil { }); err != nil {
slog.Error("seed: insert failed", "guid", item.GUID, "err", err) slog.Error("seed: insert failed", "guid", item.GUID, "err", err)
continue continue
} }
storage.InsertRecentHeadline(item.GUID, item.Headline, src.Name)
total++ total++
} }
slog.Info("seed: source done", "source", src.Name, "items", len(items)) slog.Info("seed: source done", "source", src.Name, "items", len(items))
@@ -307,8 +263,6 @@ func runTest(cfg *config.Config, sourceName string) {
if len(items) > 0 { if len(items) > 0 {
item := items[0] item := items[0]
item.Source = s.Name item.Source = s.Name
item.FeedHint = s.FeedHint
item.Tier = s.Tier
item.DirectRoute = s.DirectRoute item.DirectRoute = s.DirectRoute
testItem = &item testItem = &item
src = s src = s
@@ -326,10 +280,10 @@ func runTest(cfg *config.Config, sourceName string) {
"guid", testItem.GUID, "guid", testItem.GUID,
) )
// Determine channel channel := testItem.DirectRoute
channel := "politics" // default if channel == "" {
if testItem.DirectRoute != nil { slog.Error("test: source has no direct_route", "source", src.Name)
channel = *testItem.DirectRoute os.Exit(1)
} }
// Validate image // Validate image
@@ -351,7 +305,7 @@ func runTest(cfg *config.Config, sourceName string) {
Headline: testItem.Headline, Headline: testItem.Headline,
ArticleURL: testItem.ArticleURL, ArticleURL: testItem.ArticleURL,
Lede: testItem.Lede, Lede: testItem.Lede,
Source: testItem.Source, Source: testItem.Source,
Channel: channel, Channel: channel,
} }

View File

@@ -58,7 +58,7 @@ func TestRunSeed_IngestsStories(t *testing.T) {
FeedURL: ts.URL, FeedURL: ts.URL,
Tier: 1, Tier: 1,
PollIntervalMinutes: 20, PollIntervalMinutes: 20,
FeedHint: "tech", DirectRoute: "tech",
Enabled: true, Enabled: true,
}, },
}, },
@@ -74,13 +74,11 @@ func TestRunSeed_IngestsStories(t *testing.T) {
} }
} }
// Should have no unclassified stories (seed marks classified=true) // Seed marks rows classified=1; confirm none remain unclassified.
unclassified, err := storage.GetUnclassifiedStories("Test Feed") var unclassified int
if err != nil { storage.Get().QueryRow(`SELECT COUNT(*) FROM stories WHERE classified = 0`).Scan(&unclassified)
t.Fatal(err) if unclassified != 0 {
} t.Errorf("expected 0 unclassified after seed, got %d", unclassified)
if len(unclassified) != 0 {
t.Errorf("expected 0 unclassified after seed, got %d", len(unclassified))
} }
} }
@@ -96,7 +94,7 @@ func TestRunSeed_Idempotent(t *testing.T) {
FeedURL: ts.URL, FeedURL: ts.URL,
Tier: 1, Tier: 1,
PollIntervalMinutes: 20, PollIntervalMinutes: 20,
FeedHint: "tech", DirectRoute: "tech",
Enabled: true, Enabled: true,
}, },
}, },
@@ -139,35 +137,6 @@ func TestRunSeed_SkipsDisabledSources(t *testing.T) {
} }
} }
func TestRunSeed_RecentHeadlinesPopulated(t *testing.T) {
setupTestDB(t)
ts := serveFeed(t, 2)
defer ts.Close()
cfg := &config.Config{
Sources: []config.SourceConfig{
{
Name: "Test Feed",
FeedURL: ts.URL,
Tier: 1,
PollIntervalMinutes: 20,
FeedHint: "tech",
Enabled: true,
},
},
}
runSeed(cfg)
headlines, err := storage.GetRecentHeadlines(10)
if err != nil {
t.Fatal(err)
}
if len(headlines) != 2 {
t.Errorf("expected 2 recent headlines after seed, got %d", len(headlines))
}
}
func TestRunSeed_BadFeedURL(t *testing.T) { func TestRunSeed_BadFeedURL(t *testing.T) {
setupTestDB(t) setupTestDB(t)