Pete/newsbot-spec.md

NewsBot Spec — Claude Code Handoff

Sibling bot to GogoBee, built on a shared Matrix/infrastructure library

---

Overview

A Matrix news bot that ingests RSS feeds from curated Tier 1 sources, classifies stories using a local LLM, deduplicates semantically, and routes posts to the appropriate channel with a consistent post format. Designed as a sibling to GogoBee, sharing core plumbing via an extracted shared library.

---

Architecture

Option Selected: Shared Library (Option B)

Extract reusable GogoBee infrastructure into a shared Go library. The news bot is a separate binary that consumes the shared library. GogoBee remains unchanged initially but can migrate to use the shared library over time.

Rationale: Two consumers validate the library's abstraction quality. Clean separation means the news bot can be deployed independently without the full GogoBee stack. Forces good abstractions at the seams.

Repository Structure

/
├── lib/                        # Shared library (new)
│   ├── matrix/                 # Matrix room/event handling primitives
│   ├── storage/                # SQLite wrapper patterns
│   ├── config/                 # Config loading / env scaffolding
│   ├── http/                   # HTTP client + retry/backoff utilities
│   └── ollama/                 # Ollama API client + polling loop pattern
├── gogobee/                    # Existing bot (unchanged initially)
└── newsbot/                    # New bot (this spec)
    ├── main.go
    ├── config/
    ├── ingestion/              # RSS feed polling + parsing
    ├── classifier/             # LLM classification + dedup
    ├── storage/                # SQLite schemas for this bot
    └── poster/                 # Matrix post formatting + delivery

---

Shared Library Surface Area

Extract from GogoBee (do not duplicate):

Package	Contents
lib/matrix	Room/event handling primitives, connection lifecycle
lib/storage	SQLite open/close, migration support, typed query helpers
lib/config	Env var loading, config struct patterns, validation
lib/http	HTTP client with retry, backoff, timeout defaults
lib/ollama	Ollama API client, polling loop pattern, JSON response handling

Rule: Anything that goes into the shared library must work for two consumers without modification. If it needs a GogoBee-specific assumption, it stays in GogoBee.

---

Sources

Initial Source List

Source	Feed URL	Pre-classification hint
The Guardian — World	https://www.theguardian.com/world/rss	politics
The Guardian — Technology	https://www.theguardian.com/technology/rss	tech
The Guardian — Politics	https://www.theguardian.com/politics/rss	politics
The Guardian — US News	https://www.theguardian.com/us-news/rss	politics
Ars Technica — Main	https://feeds.arstechnica.com/arstechnica/index	tech
Ars Technica — Policy	https://feeds.arstechnica.com/arstechnica/policy	tech+politics

Platform Taxonomy

Used for gaming stories. Stored as a JSON array in SQLite, filtered via SQL alongside FTS5 search results.

Tag	Covers
nintendo-switch	Switch, Switch 2
playstation	PS5, PS4, PSN
xbox	Xbox Series X/S, Game Pass
pc	Steam, Epic, PC gaming broadly
retro	Pre-7th gen, emulation, preservation
mobile	iOS/Android gaming
multi	Cross-platform releases

Platform tags are classifier output — derived from headline and lede, not hardcoded per source. A story about a multiplatform release gets ["multi"]. A Switch exclusive gets ["nintendo-switch"]. Retro content gets ["retro"] regardless of original platform.

Source Metadata (per source in config)

sources:
  - name: "The Guardian — Technology"
    feed_url: "https://www.theguardian.com/technology/rss"
    tier: 1
    poll_interval_minutes: 20
    feed_hint: "tech"
    direct_route: null          # null = LLM routing
    enabled: true

The direct_route field bypasses LLM classification entirely and routes straight to the named channel. Set only when feed sampling confirms the source is clean enough to trust. This is a rare configuration change — sources don't get added often and each new source should be sampled before a routing decision is made.

Source Tier Taxonomy

Sources are classified on two axes: funding model and political lean. Both matter independently.

• Funding model determines editorial incentive alignment. Subscriber/trust-funded outlets optimize for reader retention and trust, not click-through rate. Ad-funded outlets face structural pressure toward volume, outrage, and shareability regardless of journalist quality.
• Political lean filters for intent and framing. Far-right lean correlates strongly with content produced to cultivate grievance and dehumanize — high production values and deliberate disinformation are not mutually exclusive.

Tier	Funding	Lean	Criteria	Examples
1	Subscriber/trust funded	Center-left	Strong editorial standards, high signal-to-noise	Guardian, Ars Technica, ProPublica, The Markup, Mother Jones
2	Institutional / non-commercial	Lean-agnostic by design	Wire services optimizing for factual accuracy over narrative	AP, Reuters, AFP
3	Domain trade press	Lean largely irrelevant	Specialist audience expertise enforces accuracy	Eurogamer, Time Extension, Rock Paper Shotgun (future gaming channel)

The tier field drives classification pipeline behaviour. Tier 1 sources use direct routing or lightweight LLM routing only. Tier 2 sources go through the full gating pipeline before routing.

---

Channels

Three Matrix rooms:

Channel	Purpose
tech	Technology news, product/industry stories
politics	Political, policy, and significant current events
gaming	Gaming news, releases, platform announcements, industry

Each story is routed to exactly one channel. Politics is the catch-all — when a story doesn't clearly belong in tech or gaming, it goes to politics. This applies to tech×policy overlap stories too: an AI regulation story goes to politics, not tech.

The gaming channel is defined in config from day one but will not have active sources until gaming feeds are added. The channel infrastructure, platform tagging, and FTS5 search are implemented now.

---

Ingestion Pipeline

Poll Loop

• Configurable interval per source (default 20 minutes)
• Uses lib/ollama polling loop pattern from shared library
• On each tick: fetch feed → parse items → filter seen GUIDs → pass new items to classifier

Feed Parsing

• Standard RSS/Atom via github.com/mmcdole/gofeed or equivalent
• Extract per item: GUID, headline, lede (first sentence or description field), image URL, article URL, source name, feed hint
• Store seen GUIDs in SQLite immediately on receipt (before classification) to prevent reprocessing on next poll

Image Validation

For each story with an image URL:

1. HEAD request to image URL — must return 200
2. Content-Type must be image/*
3. Content-Length (if present) must be > 5KB (filters 1x1 tracking pixels)
4. On validation failure: post story without image, log warning

Tier 1 sources are trusted editorially — no LLM image relevance check needed.

---

Post Format

Every story Pete posts follows a consistent structure:

[lead image — rendered inline by Matrix]
**{headline}** → {article_url}
{lede}
`{source_name}`

Rules

• Lead image is included if validation passes (see Image Validation). Omitted silently on failure — no placeholder, no error message to the room.
• Headline is bold and hyperlinked to the article URL.
• Lede is the first sentence of the story description from the feed. Not truncated, not summarized — verbatim from the feed.
• Source tag is monospaced, lowercase, matches the name field from source config.
• No additional commentary, no bot chatter. Pete posts and stays quiet.

Gaming Stories

Gaming stories append platform tags after the source tag if platforms are non-empty:

[lead image]
**{headline}** → {article_url}
{lede}
`{source_name}` · `nintendo-switch` · `retro`

---

Classification

Model

• Model: Gemma4 27B (config value — never hardcode model name)
• Runtime: Ollama, local hardware
• Config key: ollama.model

Single Inference Pass

Tier 1 — direct routing (no LLM): Guardian Politics, Guardian World, and Guardian US News route directly to politics. Feed sampling confirmed these are clean enough that LLM adds no value. GUID dedup check only.

Tier 1 — LLM routing: Guardian Technology and Ars Technica feeds go through LLM routing. Feed sampling showed meaningful politics bleed in both — Ars Technica is cleaner but not clean enough to skip classification. One LLM call handles routing (exactly one channel) + platform tagging (gaming only) + semantic dedup.

Tier 2 — all stories: Three-stage gating pipeline (see Tier 2 Gating Logic), then LLM routing for stories that pass.

Input Context

Incoming story:
  Source: {source_name}
  Feed hint: {feed_hint}         ← pre-classification signal from source config
  Headline: {headline}
  Lede: {lede}

Recent stories (last 24hr, headline + GUID only):
  [{guid}] {headline} — {source_name}
  [{guid}] {headline} — {source_name}
  ...

Cap recent stories at 50 items to manage token budget. Oldest items dropped first.

Prompt

Tier 1 Prompt (routing only)

You are a news router for a community discussion platform with three channels: tech, politics, and gaming.

This story comes from a curated Tier 1 source. It will be posted. Your only job is to decide which channel it belongs in.

Choose exactly one channel:
- tech: technology products, software, hardware, industry — where technology itself is the primary subject
- gaming: games, gaming platforms, gaming industry, esports
- politics: everything else — political, policy, current events, public interest, significant world events

Politics is the default. When in doubt, choose politics.

Routing guidance:
- GPU launch, software release, hardware review → tech
- Game release, platform announcement, studio news → gaming
- Election, legislation, court ruling → politics
- AI regulation, antitrust action, surveillance policy → politics (policy is the story, not the technology)
- Death of a public figure → politics
- Violent crime, assassination attempt, arrest of public figure → politics
- Natural disaster, humanitarian crisis → politics
- Celebrity gossip, lifestyle with no broader significance → politics (still gets posted, still needs a home)

For gaming stories only: identify which platforms are relevant from this list:
nintendo-switch, playstation, xbox, pc, retro, mobile, multi
Return an empty array if no specific platform applies.

Also check: does this story cover the same underlying event as any story in the recent window? Different outlets covering the same event counts as a duplicate. If yes, set duplicate_of to that story's GUID. If no match, set duplicate_of to null.

Respond ONLY with valid JSON. No preamble, no explanation outside the JSON:
{
  "channel": "tech|politics|gaming",
  "platforms": [string],
  "duplicate_of": string | null,
  "rationale": "one sentence max, for logging only"
}

INCOMING STORY:
Source: {source}
Tier: {tier}
Feed hint: {feed_hint}
Headline: {headline}
Lede: {lede}

RECENT STORIES (last 24hr):
{recent_stories}

Tier 2 Prompt (gating + routing)

You are a news classifier for a community discussion platform with three channels: tech, politics, and gaming.

This story comes from a wire service. Unlike curated sources, wire feeds publish everything — not all stories warrant posting. First decide if this story is worth posting, then decide where.

Step 1 — Should this be posted?
Post if the story has clear public significance: major world events, policy decisions, significant deaths, natural disasters, major technology developments. Do not post: local news with no broader relevance, routine financial filings, minor sports results, entertainment fluff.

Step 2 — If posting, choose exactly one channel:
- tech: technology products, software, hardware, industry
- gaming: games, gaming platforms, gaming industry
- politics: everything else — the default

Politics is the default when in doubt.

For gaming stories only: identify relevant platforms from: nintendo-switch, playstation, xbox, pc, retro, mobile, multi

Also check for duplicates against the recent window. If this story covers the same underlying event as a recent story, set duplicate_of to that story's GUID.

Respond ONLY with valid JSON:
{
  "post": true|false,
  "channel": "tech|politics|gaming",
  "platforms": [string],
  "duplicate_of": string | null,
  "rationale": "one sentence max, for logging only"
}

If post is false, channel may be omitted.

INCOMING STORY:
Source: {source}
Tier: {tier}
Feed hint: {feed_hint}
Headline: {headline}
Lede: {lede}

RECENT STORIES (last 24hr):
{recent_stories}

Output Handling

// Tier 1 — routing only
type Tier1Result struct {
    Channel     string   `json:"channel"`      // "tech" | "politics" | "gaming"
    Platforms   []string `json:"platforms"`
    DuplicateOf *string  `json:"duplicate_of"`
    Rationale   string   `json:"rationale"`
}

// Tier 2 — gating + routing
type Tier2Result struct {
    Post        bool     `json:"post"`
    Channel     string   `json:"channel"`      // "tech" | "politics" | "gaming"
    Platforms   []string `json:"platforms"`
    DuplicateOf *string  `json:"duplicate_of"`
    Rationale   string   `json:"rationale"`
}

Parse-with-fallback pattern (mandatory):

• Wrap all JSON parsing in try/catch
• On parse failure: log raw LLM output + story GUID → discard story → continue polling loop
• On scores out of range [0.0, 1.0]: clamp, log warning
• On missing fields: treat as discard
• Never let a malformed LLM response block the polling loop

Enable Ollama format: json parameter to constrain output to valid JSON.

Routing Logic

Tier 1 stories are always posted — the classifier decides channel only. No threshold, no gating.

Tier 2 stories require the classifier to explicitly set post: true before routing proceeds. The rationale field in the classification log is the primary tool for evaluating whether Tier 2 gating is calibrated correctly during the initial period.

---

Deduplication

Two-Stage Approach

Stage 1 — GUID dedup (exact):

• Store all seen GUIDs in SQLite on ingestion
• Fast O(1) lookup, catches same story from same source

Stage 2 — LLM semantic dedup (approximate):

• Handled inside the classification call (see above)
• duplicate_of field — if non-null, discard story regardless of scores
• Rolling 24-hour window of recent headlines passed as context
• LLM identifies same underlying event across different outlets/URLs

No embedding similarity layer in v1 — the two-stage approach (GUID + LLM) is sufficient for two sources with low story velocity. Embeddings can be added later if source count grows significantly.

SQLite Dedup Schema

table: seen_guids
  guid TEXT PRIMARY KEY
  seen_at INTEGER

table: recent_headlines
  guid TEXT PRIMARY KEY
  headline TEXT
  source TEXT
  seen_at INTEGER

Prune recent_headlines entries older than 24 hours on each poll cycle.

---

Post Format

[image — rendered inline by Matrix]
**{headline}** → {article_url}
{lede sentence}
`{source_name}`

Cross-channel Posts

When a story routes to both channels, post to each independently with a tag:

[image]
**{headline}** → {article_url}
{lede}
`{source_name}` · `tech × policy`

Rate Limiting

• Minimum 5-minute gap between posts per channel
• If multiple stories clear threshold in the same poll cycle, queue and release at interval
• Prevents burst spam during breaking news events that generate many near-duplicate stories

---

Storage Schema (SQLite)

-- Core story record
CREATE TABLE stories (
    id           INTEGER PRIMARY KEY AUTOINCREMENT,
    guid         TEXT UNIQUE NOT NULL,
    headline     TEXT NOT NULL,
    lede         TEXT,
    image_url    TEXT,
    article_url  TEXT NOT NULL,
    source       TEXT NOT NULL,
    feed_hint    TEXT,
    platforms    TEXT,              -- JSON array e.g. '["nintendo-switch","multi"]'
    seen_at      INTEGER NOT NULL   -- unix timestamp
);

-- Dedup window (last 24hr headlines for LLM context)
CREATE TABLE recent_headlines (
    guid       TEXT PRIMARY KEY,
    headline   TEXT NOT NULL,
    source     TEXT NOT NULL,
    seen_at    INTEGER NOT NULL
);

-- Classification log (7 day retention, for threshold tuning)
CREATE TABLE classification_log (
    guid       TEXT PRIMARY KEY,
    result     TEXT NOT NULL,       -- JSON ClassificationResult
    logged_at  INTEGER NOT NULL
);

-- Post log (rate limiting + audit)
CREATE TABLE post_log (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    guid       TEXT NOT NULL,
    channel    TEXT NOT NULL,
    posted_at  INTEGER NOT NULL
);

-- Reactions (see Reaction Tracking section)
CREATE TABLE reactions (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    post_guid  TEXT NOT NULL,
    channel    TEXT NOT NULL,
    event_id   TEXT NOT NULL,
    emoji      TEXT NOT NULL,
    user_id    TEXT NOT NULL,
    reacted_at INTEGER NOT NULL
);
CREATE INDEX idx_reactions_post_guid ON reactions(post_guid);

-- FTS5 full-text search index
CREATE VIRTUAL TABLE stories_fts USING fts5(
    guid UNINDEXED,
    headline,
    lede,
    source UNINDEXED,
    platforms UNINDEXED,
    content='stories',
    content_rowid='id'
);

FTS5 Search

FTS5 is the right tool for freetext search against headlines and ledes — specifically for cases like "show me everything about Elden Ring" where the game title is embedded in headline text. Pure SQL LIKE queries would be slow and miss ranking entirely.

Example queries:

-- Search by game title
SELECT s.* FROM stories s
JOIN stories_fts f ON s.id = f.rowid
WHERE stories_fts MATCH 'Elden Ring'
ORDER BY rank;

-- Search by game title filtered to gaming channel posts
SELECT s.* FROM stories s
JOIN stories_fts f ON s.id = f.rowid
JOIN post_log p ON p.guid = s.guid
WHERE stories_fts MATCH 'Metroid Prime'
AND p.channel = 'gaming'
ORDER BY rank;

-- Filter by platform tag (structured SQL, no FTS5 needed)
SELECT s.* FROM stories s
JOIN post_log p ON p.guid = s.guid
WHERE p.channel = 'gaming'
AND json_each(s.platforms) LIKE '%nintendo-switch%'
ORDER BY s.seen_at DESC;

Note on tokenization: FTS5 default tokenizer handles game titles (proper nouns, consistent spelling) well. Platform abbreviations like "PS5" and "Xbox" may need a synonym map in v2 once real search patterns are known. Don't over-engineer tokenization before you have real queries to learn from.

---

Metered Release

When multiple stories clear threshold in the same poll cycle for a single channel, Pete queues and releases them at controlled intervals rather than burst posting.

Rules

• Minimum interval — 5 minutes between posts per channel (configured via posting.min_interval_seconds)
• Burst cap — maximum 3 stories per channel per 30 minutes regardless of queue depth
• Queue strategy — FIFO within each channel queue; arrival order is respected
• Per-channel isolation — tech and politics queues are independent; a burst in one channel does not affect the other

Queue Behaviour

Stories that clear the threshold are added to an in-memory per-channel queue immediately. A ticker drains each queue independently at the minimum interval, subject to the burst cap. Stories that arrive while the burst cap is active wait in queue — they are not discarded.

Configuration

posting:
  min_interval_seconds: 300       # 5 minutes between posts per channel
  burst_cap_count: 3              # max posts per channel per burst window
  burst_cap_window_seconds: 1800  # 30 minute burst window

Future Consideration

RSS feed metadata (pubDate recency, outlet-supplied category tags, GUID update frequency) may provide a legitimate urgency signal worth incorporating into queue priority in a future version. Not implemented in v1 — evaluate once real burst patterns are observed.

---

Configuration

matrix:
  homeserver: "https://matrix.example.org"
  access_token: "${MATRIX_ACCESS_TOKEN}"
  channels:
    tech: "!roomid:matrix.example.org"
    politics: "!roomid:matrix.example.org"
    gaming: "!roomid:matrix.example.org"

ollama:
  base_url: "http://localhost:11434"
  model: "gemma4:27b"           # always a config value, never hardcoded
  timeout_seconds: 30

posting:
  min_interval_seconds: 300       # 5 minutes between posts per channel
  burst_cap_count: 3              # max posts per channel per burst window
  burst_cap_window_seconds: 1800  # 30 minute burst window

storage:
  db_path: "/var/lib/newsbot/newsbot.db"
  recent_window_hours: 24
  classification_log_days: 7

sources:
  - name: "The Guardian — World"
    feed_url: "https://www.theguardian.com/world/rss"
    tier: 1
    poll_interval_minutes: 20
    feed_hint: "politics"
    direct_route: "politics"    # confirmed clean via feed sampling
    enabled: true
  - name: "The Guardian — Technology"
    feed_url: "https://www.theguardian.com/technology/rss"
    tier: 1
    poll_interval_minutes: 20
    feed_hint: "tech"
    direct_route: null          # LLM routing — significant politics bleed confirmed
    enabled: true
  - name: "The Guardian — Politics"
    feed_url: "https://www.theguardian.com/politics/rss"
    tier: 1
    poll_interval_minutes: 20
    feed_hint: "politics"
    direct_route: "politics"    # confirmed clean via feed sampling
    enabled: true
  - name: "The Guardian — US News"
    feed_url: "https://www.theguardian.com/us-news/rss"
    tier: 1
    poll_interval_minutes: 20
    feed_hint: "politics"
    direct_route: "politics"    # confirmed clean via feed sampling
    enabled: true
  - name: "Ars Technica"
    feed_url: "https://feeds.arstechnica.com/arstechnica/index"
    tier: 1
    poll_interval_minutes: 20
    feed_hint: "tech"
    direct_route: null          # LLM routing — politics bleed confirmed (~20%)
    enabled: true
  - name: "Ars Technica — Policy"
    feed_url: "https://feeds.arstechnica.com/arstechnica/policy"
    tier: 1
    poll_interval_minutes: 20
    feed_hint: "tech+politics"
    direct_route: null          # LLM routing — feed empty at sampling time
    enabled: true

---

Deployment

Pete is deployment-agnostic. Recommended approach:

• Docker Compose service alongside your existing stack
• No HTTP ingress required (outbound only — no reverse proxy needed)
• Config via environment variables + mounted config file
• Ollama API on localhost or reachable host, shared with other consumers if applicable

Matrix Identity

Pete requires a dedicated Matrix account:

• Display name: Pete
• Avatar: configured via Matrix client before deployment — not managed by the bot
• Bot account should be invited to each channel room and granted appropriate permissions (post messages, read reactions)
• Optionally configure an admin room in config for operational warnings (feed failures, Ollama unavailability)

matrix:
  homeserver: "https://matrix.example.org"
  access_token: "${MATRIX_ACCESS_TOKEN}"
  admin_room: "!roomid:matrix.example.org"   # optional — operational warnings only
  channels:
    tech: "!roomid:matrix.example.org"
    politics: "!roomid:matrix.example.org"
    gaming: "!roomid:matrix.example.org"

---

Reaction Tracking

Pete listens for m.reaction events referencing his post event IDs and aggregates them per story in SQLite.

Reaction Categories

Emojis are mapped to categories at query time, not write time. Raw emoji is always stored so categories can be remapped without data loss.

Category	Emojis	Signal
Relevant	📰	Classifier validation — story belonged in this channel
Noise	🗑️	Classifier validation — story should not have been posted
Positive	👍 🔥 ❤️ 😂 🎉	Story generated a positive feeling
Negative	👎 😢 😡 😤 😱	Story generated a negative feeling
Sad	😢 💔 🕯️	Grief, loss, mourning
Angry	😡 😤 🤬	Outrage, injustice, frustration
Shock	😱 😨 🤯	Disbelief, surprise, breaking news

Categories intentionally overlap — 😢 contributes to both Negative and Sad, 😡 to both Negative and Angry. They measure different dimensions simultaneously.

Engagement Signals

• Relevance — 📰 vs 🗑️ ratio: is the classifier making good calls?
• Sentiment — Positive vs Negative ratio: how did the story land emotionally?
• Tone — Sad / Angry / Shock breakdown: what kind of negative response?
• Engagement volume — total reaction count regardless of type: did the story resonate at all?
• Silence — no reactions: possible threshold miscalibration or source fatigue

A story about a beloved figure dying may generate zero 📰 or 🗑️ but a flood of 😢 💔 — high engagement, clearly worth posting, classifier did its job.

SQLite Schema

CREATE TABLE reactions (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    post_guid   TEXT NOT NULL,
    channel     TEXT NOT NULL,
    event_id    TEXT NOT NULL,       -- Matrix event ID of the reaction
    emoji       TEXT NOT NULL,
    user_id     TEXT NOT NULL,       -- Matrix user ID
    reacted_at  INTEGER NOT NULL     -- unix timestamp
);

CREATE INDEX idx_reactions_post_guid ON reactions(post_guid);

Categories are derived at query time by joining against an in-memory emoji→category mapping. No category column stored — remapping is free.

Classifier Tuning Use

• Consistently high 🗑️ from a specific source → flag source for review
• Consistently high 🗑️ on a specific feed_hint → routing prompt or direct routing logic may need calibration
• High engagement (any emoji) with low 🗑️ → source and classifier performing well
• High Angry reactions on stories that cleared politics threshold → expected, not a misfire

---

Error Handling

Ollama Unavailable

• On connection failure: log error, skip classification for current poll cycle
• Do not discard stories — they will be reprocessed on next poll cycle if not yet seen
• If Ollama is unavailable for 3 consecutive poll cycles: log a warning to Matrix bot admin room (if configured)
• Never crash the poll loop on Ollama failure

Matrix Connection Drop

• Use mautrix-go reconnection handling from shared library
• On reconnect: resume normal operation — no replay of missed stories
• Stories that arrived during disconnect are processed normally on next poll cycle

Feed Fetch Failure

• On HTTP error or timeout: log warning, skip source for this cycle
• If a source fails 5 consecutive polls: log persistent failure warning to admin room
• Other sources continue unaffected — per-source failure isolation

Image Validation Failure

• Post story without image, log warning
• Never block story posting on image failure

Malformed LLM Response

• Log raw response + story GUID
• Discard story for this cycle — do not post unclassified content
• Story is not marked as seen — will be reprocessed on next poll cycle
• Never crash the poll loop on malformed LLM output

General Principle

The poll loop must never crash. All errors are logged, isolated, and recovered from. A degraded Pete that posts fewer stories is always preferable to a Pete that stops entirely.

---

Tier 2 Gating Logic

Wire services publish hundreds of stories daily — not all warrant posting. Gating uses a three-stage pipeline to minimize unnecessary LLM calls and keep decisions deterministic where possible.

Stage 1 — Definite Post (no LLM)

If any of the following match in the headline or lede, post without LLM gating. Route via Tier 2 prompt as normal.

• Named head of state, head of government, or major international body (UN, WHO, NATO, EU, IMF, World Bank) as primary subject
• Casualty language: "dead", "killed", "injured", "casualties" with a number
• Disaster language: "earthquake", "hurricane", "tsunami", "flood", "wildfire" with named location
• Conflict language: "war", "invasion", "coup", "airstrike", "assassination", "siege"
• Crisis language: "emergency", "crisis", "collapse", "outbreak", "pandemic"
• Major financial scale: "$Xbn", "$Xtn", "trillion" in context of policy or market event

Stage 2 — Definite Discard (no LLM)

If any of the following match, discard without LLM call. Log reason.

• Purely local scope: city council, municipal government, local crime, regional weather with no national angle
• Sports scores, match results, league standings with no broader significance
• Entertainment and celebrity lifestyle: awards, relationships, appearances, album releases
• Routine corporate earnings with no systemic significance (not major banks, not Big Tech)
• Obituaries of non-public figures

Stage 3 — Ambiguous (escalate to LLM)

Everything that clears neither Stage 1 nor Stage 2 goes to Gemma4 via the Tier 2 prompt. The post: true|false decision in the LLM response applies only to this tier — Stages 1 and 2 never reach the model for gating.

Pipeline Flow

Wire story arrives
  ↓
Stage 1: matches definite post criteria?
  → Yes: skip gating, send to Tier 2 routing prompt
  ↓
Stage 2: matches definite discard criteria?
  → Yes: discard, log reason
  ↓
Stage 3: ambiguous — send to Tier 2 prompt (gating + routing)
  → post: false: discard, log rationale
  → post: true: route to channel

Implementation Notes

• Stage 1 and Stage 2 matching is keyword/pattern based — fast, no external calls
• Keyword lists are config-driven, not hardcoded — adjustable without redeployment
• All discards are logged with stage and reason for calibration review
• Stage 3 LLM calls log the full rationale field — primary tool for tuning keyword lists over time

Future Consideration

As Tier 2 source volume and patterns become known, Stage 1 and Stage 2 keyword lists should be reviewed against the classification log. Patterns that consistently produce correct LLM decisions can be promoted to deterministic rules, reducing LLM calls further.

---

Out of Scope (v1)

• Reaction data aggregation dashboards and automated threshold tuning based on reaction signals (future)
• Embedding similarity dedup layer (future, if source count grows)
• Gaming sources (channel infrastructure is implemented in v1, sources populated later)
• LLM image relevance scoring (unnecessary for Tier 1 sources)
• Web scraping (RSS-only in v1)
• Admin Matrix commands (future — enable/disable sources, adjust thresholds)

---

Open Questions for Engineering Review

1. Shared library extraction scope — which GogoBee packages are cleanest to extract first?
2. gofeed vs. alternative RSS library — any existing preference in the codebase?
3. SQLite table naming conventions — follow GogoBee patterns?
4. Ollama format: json parameter — confirm supported by current Ollama version on host
5. Matrix image attachment method — m.image event or inline URL in m.text?