Initial commit
This commit is contained in:
786
veola-spec.md
Normal file
786
veola-spec.md
Normal file
@@ -0,0 +1,786 @@
|
||||
# Veola — Claude Code Reference Specification
|
||||
|
||||
> DO NOT REWRITE, SUMMARIZE, OR SHORTEN ANY ENTRIES IN THIS FILE
|
||||
|
||||
This document is the authoritative specification for a self-hosted Go web application that tracks items across e-commerce platforms and delivers deal alerts via push notification. It is written for Claude Code and should be followed precisely.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A Go web application named Veola for tracking specific items across multiple e-commerce platforms (eBay, Amazon, etc.) via the Apify API. The operator adds items to watch via a web interface, configures per-item alert thresholds, and receives push notifications via a self-hosted Ntfy instance when deals are found. All configuration, results, and history are managed through the web UI.
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Layer | Choice | Notes |
|
||||
|---|---|---|
|
||||
| Language | Go 1.22+ | Standard library preferred where possible |
|
||||
| Templates | [a-h/templ](https://github.com/a-h/templ) | Type-safe HTML components |
|
||||
| Reactivity | [HTMX](https://htmx.org) | Via CDN, no build step |
|
||||
| CSS | [Tailwind CSS](https://tailwindcss.com) | Via CDN Play CDN for development; document where to swap for production build |
|
||||
| Database | SQLite | Use `modernc.org/sqlite` (pure Go, no CGO required) |
|
||||
| HTTP router | `net/http` + `chi` | `github.com/go-chi/chi/v5` |
|
||||
| Config | TOML config file | Use `github.com/BurntSushi/toml` |
|
||||
| Scheduler | `robfig/cron` | `github.com/robfig/cron/v3` |
|
||||
|
||||
---
|
||||
|
||||
## Visual Design Direction
|
||||
|
||||
The app is named Veola, after the magic shop owner from Atelier Iris: Eternal Mana. Taciturn, precise, and better at the job than anyone else in the room. The aesthetic should reflect that: no-nonsense, visually distinctive, purpose-built.
|
||||
|
||||
**Direction: Sega blue**
|
||||
|
||||
The reference point is the Sega Genesis/Mega Drive hardware era — the distinctive saturated blue of the console body, the bright Sega logo blue, and the high-contrast white text of that period's UI. Not dark mode, not light mode — blue mode.
|
||||
|
||||
Color palette:
|
||||
- Background: `#1a2b6d` (Genesis console blue — deep, saturated, not navy)
|
||||
- Surface/cards: `#1f3380` (slightly lighter, enough contrast to lift cards off the background)
|
||||
- Primary accent: `#00a4e4` (Sega logo blue — bright, electric)
|
||||
- Secondary accent: `#f5c400` (Sega arcade button yellow — used very sparingly for price highlights and active states)
|
||||
- Text primary: `#ffffff`
|
||||
- Text secondary: `#a8c0f0` (desaturated blue-white for secondary labels and timestamps)
|
||||
- Danger/error: `#e84040` (bright red — high contrast against blue)
|
||||
- Success/below-target price: `#00e4a4` (mint, complementary to the Sega blue)
|
||||
- Border: `rgba(255, 255, 255, 0.12)` (subtle white rule)
|
||||
|
||||
Typography:
|
||||
- Monospaced font for prices and data values: `JetBrains Mono` or `Fira Code` via Google Fonts
|
||||
- UI chrome: `Outfit` or `DM Sans` — clean, slightly geometric, fits the era
|
||||
- Do not use Inter, Roboto, or system fonts
|
||||
|
||||
Surface treatment:
|
||||
- Cards: solid `#1f3380` background, `1px` border at `rgba(255,255,255,0.12)`, `border-radius: 8px`
|
||||
- No glassmorphism, no blur effects, no texture — solid surfaces only
|
||||
- No texture effects — clean flat surfaces throughout
|
||||
- Box shadows use blue tones, not black: `0 4px 16px rgba(0, 0, 80, 0.4)`
|
||||
|
||||
Component specifics:
|
||||
- Status pills: Active (`#00a4e4` background, white text), Paused (white/10 background, `#a8c0f0` text), Error (`#e84040` background, white text)
|
||||
- Prices: `JetBrains Mono`, large, `#f5c400` when at or below target, `#ffffff` otherwise
|
||||
- Primary buttons: `#00a4e4` background, white text, no rounded pill — `border-radius: 6px`
|
||||
- Navigation active state: `#00a4e4` left border accent, slightly lighter surface
|
||||
|
||||
Do not introduce purple. Do not use gradients on backgrounds. Do not produce a generic dark SaaS dashboard with blue accents — the background itself is blue, which is the whole point.
|
||||
|
||||
---
|
||||
|
||||
## Application Structure
|
||||
|
||||
```
|
||||
veola/
|
||||
├── main.go
|
||||
├── config.toml.example
|
||||
├── internal/
|
||||
│ ├── config/
|
||||
│ │ └── config.go # TOML config loading and validation
|
||||
│ ├── db/
|
||||
│ │ ├── db.go # SQLite init, migrations
|
||||
│ │ └── queries.go # All DB operations
|
||||
│ ├── models/
|
||||
│ │ └── models.go # Item, Result, PricePoint, Setting structs
|
||||
│ ├── apify/
|
||||
│ │ └── client.go # Apify API client
|
||||
│ ├── ntfy/
|
||||
│ │ └── client.go # Ntfy push client
|
||||
│ ├── auth/
|
||||
│ │ └── auth.go # Session management, bcrypt, middleware
|
||||
│ ├── crypto/
|
||||
│ │ └── crypto.go # AES-256-GCM encrypt/decrypt, key derivation
|
||||
│ ├── scheduler/
|
||||
│ │ └── scheduler.go # Cron-based poll scheduler
|
||||
│ └── handlers/
|
||||
│ ├── items.go # Item CRUD + preview handlers
|
||||
│ ├── results.go # Results view handlers
|
||||
│ ├── settings.go # Settings handlers
|
||||
│ ├── auth.go # Login/logout handlers
|
||||
│ └── dashboard.go # Dashboard handler
|
||||
├── templates/
|
||||
│ ├── layout.templ # Base layout, nav
|
||||
│ ├── login.templ # Login page
|
||||
│ ├── dashboard.templ
|
||||
│ ├── items.templ
|
||||
│ ├── item_form.templ
|
||||
│ ├── item_preview.templ # Preview results before confirming add
|
||||
│ ├── results.templ
|
||||
│ └── settings.templ
|
||||
└── static/
|
||||
└── (any local static assets if needed)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
All migrations run at startup via embedded SQL. Use `modernc.org/sqlite` with WAL mode enabled.
|
||||
|
||||
```sql
|
||||
PRAGMA journal_mode=WAL;
|
||||
PRAGMA foreign_keys=ON;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
username TEXT NOT NULL UNIQUE,
|
||||
password_hash TEXT NOT NULL, -- bcrypt hash, cost factor 12
|
||||
role TEXT NOT NULL DEFAULT 'user', -- 'admin' or 'user'
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS items (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL,
|
||||
search_query TEXT, -- free text search (optional)
|
||||
url TEXT, -- specific product URL (optional)
|
||||
category TEXT, -- user-defined label
|
||||
target_price REAL, -- alert threshold (NULL = alert on any result)
|
||||
ntfy_topic TEXT NOT NULL, -- per-item ntfy topic slug
|
||||
ntfy_priority TEXT DEFAULT 'default', -- min, low, default, high, urgent
|
||||
poll_interval_minutes INTEGER DEFAULT 60,
|
||||
include_out_of_stock INTEGER DEFAULT 0,
|
||||
active INTEGER DEFAULT 1, -- 0 = paused
|
||||
last_polled_at DATETIME, -- timestamp of last completed poll
|
||||
last_poll_error TEXT, -- most recent error message, NULL if last poll succeeded
|
||||
best_price REAL, -- lowest price seen in most recent poll
|
||||
best_price_store TEXT, -- store name for best_price
|
||||
best_price_url TEXT, -- direct listing URL for best_price
|
||||
best_price_image_url TEXT, -- product image URL from best result
|
||||
best_price_title TEXT, -- product title from best result
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS results (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
item_id INTEGER NOT NULL REFERENCES items(id) ON DELETE CASCADE,
|
||||
title TEXT,
|
||||
price REAL,
|
||||
currency TEXT NOT NULL, -- ISO 4217 code: USD, JPY, GBP, EUR, etc. Never assume USD.
|
||||
url TEXT,
|
||||
source TEXT, -- e.g. "ebay", "amazon"
|
||||
image_url TEXT,
|
||||
alerted INTEGER DEFAULT 0, -- 1 = ntfy notification sent
|
||||
found_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- price_history records the best price observed per item per poll cycle.
|
||||
-- Used to render the price history chart. One row per poll run.
|
||||
CREATE TABLE IF NOT EXISTS price_history (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
item_id INTEGER NOT NULL REFERENCES items(id) ON DELETE CASCADE,
|
||||
price REAL NOT NULL, -- best (lowest) price seen in this poll
|
||||
store TEXT,
|
||||
polled_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS settings (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Seed default settings keys (values populated via UI or env)
|
||||
INSERT OR IGNORE INTO settings (key, value) VALUES
|
||||
('apify_api_key', ''),
|
||||
('ntfy_base_url', ''),
|
||||
('ntfy_default_topic', 'veola'),
|
||||
('global_poll_interval_minutes', '60'),
|
||||
('match_confidence_threshold', '0.6');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Config is loaded from a TOML file. The path defaults to `./config.toml` and can be overridden with the `-config` flag at startup:
|
||||
|
||||
```
|
||||
./veola -config /etc/veola/config.toml
|
||||
```
|
||||
|
||||
The config file is the single source of truth. There are no environment variable overrides and no DB-stored settings that duplicate what is in the file — the Settings UI stores only Apify and Ntfy operational settings in the DB (API keys, topics, thresholds). Server and security settings live in the file only.
|
||||
|
||||
`config.toml.example`:
|
||||
|
||||
```toml
|
||||
[server]
|
||||
port = 8080
|
||||
db_path = "./veola.db"
|
||||
|
||||
[security]
|
||||
session_secret = "change-this-to-a-random-32-byte-string"
|
||||
encryption_key = "change-this-to-a-different-random-32-byte-string"
|
||||
|
||||
[apify]
|
||||
api_key = ""
|
||||
|
||||
[apify.actors]
|
||||
active_listings = "harvestlab/ebay-scraper"
|
||||
sold_listings = "automation-lab/ebay-sold-scraper"
|
||||
price_comparison = "junipr/price-comparison"
|
||||
|
||||
[ntfy]
|
||||
base_url = "https://ntfy.yourdomain.com"
|
||||
default_topic = "veola"
|
||||
|
||||
[scheduler]
|
||||
global_poll_interval_minutes = 60
|
||||
match_confidence_threshold = 0.6
|
||||
```
|
||||
|
||||
The Go config struct mirrors this structure exactly:
|
||||
|
||||
```go
|
||||
type Config struct {
|
||||
Server ServerConfig `toml:"server"`
|
||||
Security SecurityConfig `toml:"security"`
|
||||
Apify ApifyConfig `toml:"apify"`
|
||||
Ntfy NtfyConfig `toml:"ntfy"`
|
||||
Scheduler SchedulerConfig `toml:"scheduler"`
|
||||
}
|
||||
```
|
||||
|
||||
Startup validation (exit on failure):
|
||||
- `security.session_secret` must be at least 32 bytes
|
||||
- `security.encryption_key` must be at least 32 bytes
|
||||
- `security.session_secret` and `security.encryption_key` must not be equal
|
||||
- `server.db_path` must be a writable path
|
||||
- Config file must exist and parse without error; missing file is a fatal error with a clear message pointing to `config.toml.example`
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
Single-user setup for v1 but the schema and auth layer support multiple users from the start. All users share the same item list — items are not scoped per user. Each user manages their own ntfy topic subscription independently.
|
||||
|
||||
**User roles:**
|
||||
- `admin` — can manage users (add, remove, reset passwords), access all settings
|
||||
- `user` — can manage items, view results, view settings (read-only)
|
||||
|
||||
The first account created via `/setup` is automatically assigned the `admin` role. Subsequent accounts are created by an admin from the Settings page — no self-registration.
|
||||
|
||||
**Package: `internal/auth/auth.go`**
|
||||
|
||||
- Password hashing: `bcrypt` at cost factor 12 (`golang.org/x/crypto/bcrypt`)
|
||||
- Sessions: signed cookies using `gorilla/sessions` with `SESSION_SECRET` from env
|
||||
- Session lifetime: 7 days, renewed on each request
|
||||
- All routes except `/login` and `/setup` are protected by auth middleware
|
||||
- Middleware checks for a valid session cookie; redirects to `/login` on failure
|
||||
|
||||
**First-run setup:**
|
||||
- On startup, if the `users` table is empty, the app redirects all requests to `/setup`
|
||||
- `/setup` renders a form to create the initial username and password
|
||||
- Password must be at least 12 characters — validate server-side
|
||||
- After setup completes, redirect to `/login`
|
||||
- `/setup` is inaccessible once a user exists (return 404)
|
||||
|
||||
**Login (`/login`):**
|
||||
- Simple form: username + password
|
||||
- On success: create session, redirect to `/`
|
||||
- On failure: show "Invalid username or password" (do not indicate which field was wrong)
|
||||
- No rate limiting required for v1 (single-user, self-hosted)
|
||||
|
||||
**Logout (`/logout`):**
|
||||
- POST only (CSRF-safe)
|
||||
- Clears session, redirects to `/login`
|
||||
|
||||
**Password change:**
|
||||
- Available in Settings page
|
||||
- Requires current password confirmation before accepting new password
|
||||
|
||||
---
|
||||
|
||||
## Data Encryption
|
||||
|
||||
Application-level column encryption using AES-256-GCM. The SQLite file remains unencrypted at the filesystem level; sensitive values are encrypted before write and decrypted after read in the query layer. No CGO required.
|
||||
|
||||
**Package: `internal/crypto/crypto.go`**
|
||||
|
||||
Key derivation:
|
||||
- Raw key material comes from `security.encryption_key` in `config.toml` (must be at least 32 bytes; app exits on startup if absent or too short)
|
||||
- Derive a 32-byte AES key using HKDF-SHA256 (`golang.org/x/crypto/hkdf`) with a fixed info string of `"veola-v1"`
|
||||
- The derived key is held in memory for the lifetime of the process; never written to disk
|
||||
|
||||
Encryption scheme:
|
||||
- Algorithm: AES-256-GCM (`crypto/aes`, `crypto/cipher`)
|
||||
- Nonce: 12 bytes, randomly generated per encryption call (`crypto/rand`)
|
||||
- Storage format: base64(nonce + ciphertext + tag), stored as TEXT in SQLite
|
||||
- Prefix encrypted values with the string `enc:` so the query layer can distinguish encrypted from plaintext values (relevant during any future migration)
|
||||
|
||||
```go
|
||||
// Public interface
|
||||
func Encrypt(key []byte, plaintext string) (string, error)
|
||||
func Decrypt(key []byte, ciphertext string) (string, error)
|
||||
func IsEncrypted(value string) bool // checks for "enc:" prefix
|
||||
```
|
||||
|
||||
**Encrypted fields:**
|
||||
|
||||
| Table | Column |
|
||||
|---|---|
|
||||
| `settings` | `value` (all rows) |
|
||||
| `items` | `search_query`, `url`, `ntfy_topic`, `best_price_url`, `best_price_image_url`, `best_price_title`, `last_poll_error` |
|
||||
| `results` | `title`, `url`, `image_url` |
|
||||
| `price_history` | `store` |
|
||||
| `users` | `username` |
|
||||
|
||||
Fields not encrypted: all numeric values (`price`, `id`, `poll_interval_minutes`, etc.), boolean flags, timestamps, and `currency`. These carry no sensitive information and encrypting them would break sorting and range queries.
|
||||
|
||||
**Query layer integration:**
|
||||
- All encrypt/decrypt calls happen in `internal/db/queries.go` — never in handlers or templates
|
||||
- Structs in `internal/models/models.go` always hold decrypted plaintext values
|
||||
- If decryption fails for a field, log the error and substitute an empty string — do not crash or return an error to the UI for individual field failures
|
||||
- On write: encrypt then store. On read: read then decrypt.
|
||||
|
||||
**Key rotation (document only, do not implement in v1):**
|
||||
- Future: add a `/admin/reencrypt` endpoint that reads all encrypted rows with the old key and rewrites them with a new key. Not in scope for v1.
|
||||
|
||||
**Startup validation** is handled by the config loader (see Configuration section) — the crypto package receives an already-validated key and may assume it is correct length. `SESSION_SECRET` and `ENCRYPTION_KEY` must be different values — validated at config load time.
|
||||
|
||||
---
|
||||
|
||||
## Apify Integration
|
||||
|
||||
**Client: `internal/apify/client.go`**
|
||||
|
||||
Veola uses two distinct actor types per item, serving different purposes. Both share the same underlying API client but with different input schemas and result parsers.
|
||||
|
||||
### API Flow (both actor types)
|
||||
|
||||
1. Start a run via `POST https://api.apify.com/v2/acts/{actorId}/runs`
|
||||
2. Poll run status via `GET https://api.apify.com/v2/actor-runs/{runId}` until `SUCCEEDED` or `FAILED`
|
||||
3. Fetch results via `GET https://api.apify.com/v2/actor-runs/{runId}/dataset/items`
|
||||
4. Return structured results
|
||||
|
||||
All requests authenticated via `?token={apiKey}` query parameter. Run timeout: 5 minutes. API key and all actor IDs loaded from config — never hardcoded.
|
||||
|
||||
### Actor 1 — Active Listings (primary, polling)
|
||||
|
||||
Default actor: `harvestlab/ebay-scraper`
|
||||
|
||||
Used on every scheduled poll cycle. Finds currently available listings — what you can actually buy right now.
|
||||
|
||||
```go
|
||||
type ActiveListingInput struct {
|
||||
Query string `json:"query,omitempty"`
|
||||
URL string `json:"url,omitempty"`
|
||||
Marketplace string `json:"marketplace,omitempty"` // e.g. "ebay.com", "ebay.co.uk"
|
||||
ListingType string `json:"listingType,omitempty"` // "BIN", "auction", "all"
|
||||
MaxResults int `json:"maxResults,omitempty"`
|
||||
}
|
||||
|
||||
type ActiveListingResult struct {
|
||||
Title string `json:"title"`
|
||||
Price float64 `json:"price"`
|
||||
Currency string `json:"currency"`
|
||||
URL string `json:"url"`
|
||||
Store string `json:"store"`
|
||||
ImageURL string `json:"imageUrl"`
|
||||
Condition string `json:"condition"`
|
||||
ListingType string `json:"listingType"`
|
||||
ShippingPrice float64 `json:"shippingPrice"`
|
||||
FreeShipping bool `json:"freeShipping"`
|
||||
Marketplace string `json:"marketplace"`
|
||||
MatchConfidence float64 `json:"matchConfidence"`
|
||||
}
|
||||
```
|
||||
|
||||
- Default marketplace: `ebay.com` (configurable per item)
|
||||
- Default listing type: `all` (configurable per item)
|
||||
- Minimum `matchConfidence`: 0.6 (configurable in settings)
|
||||
- Filter out out-of-stock results by default (configurable per item)
|
||||
|
||||
### Actor 2 — Sold Listings (historical baseline)
|
||||
|
||||
Default actor: `automation-lab/ebay-sold-scraper`
|
||||
|
||||
Used in two scenarios only:
|
||||
1. On item creation — seeds `price_history` with real completed sale prices before the first active listing poll runs
|
||||
2. Weekly refresh — keeps the historical baseline current with recent sold data
|
||||
|
||||
This ensures deal quality badges ("X% below 30-day average") are based on what things actually sell for, not asking price snapshots.
|
||||
|
||||
```go
|
||||
type SoldListingInput struct {
|
||||
Query string `json:"query"`
|
||||
Marketplace string `json:"marketplace,omitempty"`
|
||||
MaxResults int `json:"maxResults,omitempty"`
|
||||
DaysBack int `json:"daysBack,omitempty"` // default 30
|
||||
}
|
||||
|
||||
type SoldListingResult struct {
|
||||
Title string `json:"title"`
|
||||
SoldPrice float64 `json:"soldPrice"`
|
||||
Currency string `json:"soldCurrency"`
|
||||
SoldAt string `json:"endedAt"`
|
||||
Condition string `json:"condition"`
|
||||
ListingType string `json:"listingType"`
|
||||
ShippingPrice float64 `json:"shippingPrice"`
|
||||
URL string `json:"url"`
|
||||
}
|
||||
```
|
||||
|
||||
On item creation, insert one `price_history` row per sold result using `soldPrice` and `endedAt` as the timestamp. This gives deal quality badges real data immediately rather than waiting weeks for polling history to accumulate.
|
||||
|
||||
### Actor 3 — General Price Comparison (secondary, optional)
|
||||
|
||||
Default actor: `junipr/price-comparison`
|
||||
|
||||
Covers Amazon, Walmart, Target, Best Buy, Home Depot. Not eBay. Used as a supplementary source for items where broader retail coverage is wanted alongside eBay. Not enabled by default — configurable per item in the advanced section of the item form.
|
||||
|
||||
```go
|
||||
type PriceComparisonInput struct {
|
||||
Query string `json:"query,omitempty"`
|
||||
URL string `json:"url,omitempty"`
|
||||
MatchStrictness string `json:"matchStrictness,omitempty"` // "strict", "normal", "loose"
|
||||
}
|
||||
|
||||
type PriceComparisonResult struct {
|
||||
Title string `json:"title"`
|
||||
Price float64 `json:"price"`
|
||||
Currency string `json:"currency"`
|
||||
URL string `json:"url"`
|
||||
Store string `json:"store"`
|
||||
ImageURL string `json:"imageUrl"`
|
||||
Availability string `json:"availability"`
|
||||
MatchConfidence float64 `json:"matchConfidence"`
|
||||
}
|
||||
```
|
||||
|
||||
### Actor 4 — Yahoo Auctions Japan / JDirectItems (Japanese marketplace)
|
||||
|
||||
Default actor: `styleindexamerica/jp-yahooauctions-scraper`
|
||||
|
||||
Scrapes `auctions.yahoo.co.jp` directly. Known internationally as JDirectItems Auction since January 2025 — the underlying site and URL are unchanged.
|
||||
|
||||
**Important:** Yahoo Auctions Japan is Japanese-language only. Search queries for this marketplace must be in Japanese (kanji/kana) or romaji to return meaningful results. The item form must display a visible warning when this actor is selected: "Yahoo Auctions Japan searches in Japanese. English queries will return few or no results."
|
||||
|
||||
**CJK input requirements:** All text input fields in the item form must handle Japanese IME input correctly. Specifically:
|
||||
- Do not trigger search, validation, or HTMX requests on `keydown` or `keyup` events — use `change` or explicit button actions only. IME composition fires intermediate key events that will break mid-input if the app reacts to them.
|
||||
- Set `lang` attribute appropriately on inputs where Japanese is expected.
|
||||
- Do not truncate, strip, or re-encode non-ASCII characters at any layer — Go, SQLite, and the template layer must all pass CJK strings through unchanged.
|
||||
- Test strings: `ツインビー`, `グラディウス`, `パロディウス` — if these round-trip through the form, DB, and display correctly the implementation is sound.
|
||||
|
||||
A companion sold-price actor is also available for historical baseline seeding: `contented_eyebrow/yafuoku-closedsearch-crawler`
|
||||
|
||||
```go
|
||||
type YahooAuctionsJPInput struct {
|
||||
Query string `json:"query"`
|
||||
MaxItems int `json:"maxItems,omitempty"`
|
||||
}
|
||||
|
||||
type YahooAuctionsJPResult struct {
|
||||
Title string `json:"name"`
|
||||
Price float64 `json:"currentPrice"`
|
||||
Currency string `json:"currency"` // JPY
|
||||
URL string `json:"url"`
|
||||
ImageURL string `json:"imageUrl"`
|
||||
BidsCount int `json:"bidsCount"`
|
||||
TimeLeft string `json:"timeLeft"`
|
||||
EndTime string `json:"endTime"`
|
||||
}
|
||||
```
|
||||
|
||||
Currency will be JPY — the results layer must store and display the original currency rather than assuming USD.
|
||||
|
||||
### Actor 5 — Mercari Japan
|
||||
|
||||
Default actor: community Mercari Japan scraper (verify current best actor on Apify Store before implementation — search "mercari japan" sorted by run count, pick highest with 90%+ success rate and update within 90 days)
|
||||
|
||||
Mercari Japan is a large secondhand marketplace popular for figures, games, and collectibles. Same Japanese-language caveat applies as Yahoo Auctions Japan.
|
||||
|
||||
### Config
|
||||
|
||||
```toml
|
||||
[apify]
|
||||
api_key = ""
|
||||
|
||||
[apify.actors]
|
||||
active_listings = "harvestlab/ebay-scraper"
|
||||
sold_listings = "automation-lab/ebay-sold-scraper"
|
||||
price_comparison = "junipr/price-comparison"
|
||||
yahoo_auctions_jp = "styleindexamerica/jp-yahooauctions-scraper"
|
||||
yahoo_auctions_jp_sold = "contented_eyebrow/yafuoku-closedsearch-crawler"
|
||||
mercari_jp = "" # populate before use — see actor selection note above
|
||||
```
|
||||
|
||||
All three actor IDs are overridable per item in the item form (advanced section, collapsed by default).
|
||||
|
||||
---
|
||||
|
||||
## Ntfy Integration
|
||||
|
||||
**Client: `internal/ntfy/client.go`**
|
||||
|
||||
Post a JSON notification to the self-hosted Ntfy instance.
|
||||
|
||||
```go
|
||||
type Notification struct {
|
||||
Topic string `json:"topic"`
|
||||
Title string `json:"title"`
|
||||
Message string `json:"message"`
|
||||
Priority string `json:"priority"` // min, low, default, high, urgent
|
||||
Tags []string `json:"tags"`
|
||||
Click string `json:"click,omitempty"` // URL to open on tap
|
||||
}
|
||||
```
|
||||
|
||||
- Base URL is configurable (self-hosted)
|
||||
- Topic is per-item
|
||||
- `Click` should be the product listing URL so tapping the notification opens the deal
|
||||
- Tags: use `["shopping_cart", "tada"]` on a price-threshold hit, `["mag"]` on a general result
|
||||
- Include price and source in the message body
|
||||
- Mark `result.alerted = 1` in DB after successful send
|
||||
|
||||
Example notification for a hit:
|
||||
```
|
||||
Title: Veola Alert: TwinBee Famicom CIB
|
||||
Message: eBay — $42.00 (target: $60.00)
|
||||
Buy It Now · Free Shipping
|
||||
Click: https://ebay.com/itm/...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scheduler
|
||||
|
||||
**`internal/scheduler/scheduler.go`**
|
||||
|
||||
- Use `robfig/cron/v3` with second-level precision disabled (minute granularity is fine)
|
||||
- On startup, load all active items from DB and register a cron job per item based on `poll_interval_minutes`
|
||||
- When an item is created, updated, or toggled active/paused, re-register or remove its cron job without restarting the app
|
||||
- Each job: invoke Apify search, store results in DB, evaluate alert conditions, send Ntfy if triggered
|
||||
- Log each poll cycle with item name, result count, and whether an alert was sent
|
||||
- Jobs run with a context that is cancelled on app shutdown (graceful shutdown required)
|
||||
|
||||
Alert condition logic:
|
||||
```
|
||||
if item.TargetPrice == nil:
|
||||
alert on every new result not previously seen
|
||||
else:
|
||||
alert only when result.Price <= item.TargetPrice
|
||||
```
|
||||
|
||||
Deduplication: a result is "previously seen" if a row exists in `results` with the same `item_id` and `url`. Do not re-alert the same listing.
|
||||
|
||||
---
|
||||
|
||||
## Web UI — Pages and Handlers
|
||||
|
||||
### Layout (`layout.templ`)
|
||||
- Fixed left sidebar navigation on desktop, collapsible on mobile
|
||||
- Nav items: Dashboard, Items, Results, Settings
|
||||
- Header shows app name "Veola" with a small bee emoji or custom SVG icon
|
||||
- Active nav item highlighted with the primary accent color
|
||||
- No top navbar — sidebar only
|
||||
|
||||
### Dashboard (`/`)
|
||||
- Summary cards: Total Items Tracked, Active Items, Results Today, Alerts Sent Today
|
||||
- **Potential Spend card**: sum of `best_price` across all active items with a known best price. Label: "Potential Spend" with a subline showing the item count contributing to the total (e.g. "across 12 items"). Displayed in large monospaced font. Items with no best price yet (never polled or no results) are excluded from the total and noted: "3 items not yet priced."
|
||||
- **Money Saved card**: sum of `(historical_average_price - best_price)` across all active items where both values are known. Historical average is the mean of all `price_history` rows for that item. Only include items where `best_price` is below their historical average — negative savings are not counted. Subline shows contributing item count. Displayed in large monospaced font in the success color (`#00e4a4`).
|
||||
- Recent Results table: last 20 results across all items, showing name, price, source, found time, alert status
|
||||
- Recent Alerts section: last 5 ntfy alerts sent, with item name and price
|
||||
- All data loaded server-side; HTMX auto-refresh every 60 seconds (`hx-trigger="every 60s"`)
|
||||
|
||||
### Items List (`/items`)
|
||||
- Table/card list of all tracked items
|
||||
- Columns: Name, Category, Target Price, Best Price (with store name as a link to the listing), Last Polled, Status (Active/Paused/Error), Actions
|
||||
- Best Price cell: show price in large monospaced font, store name beneath it as a link, highlighted mint green if at or below target price
|
||||
- Last Polled cell: human-readable relative time (e.g. "12 minutes ago"); tooltip shows exact timestamp
|
||||
- Error badge: if `last_poll_error` is set, show a small coral warning badge on the row; clicking it shows the error message inline via HTMX
|
||||
- Actions: Edit, Pause/Resume toggle (HTMX inline toggle, no page reload), Delete (with confirm), Run Now
|
||||
- "Add Item" button opens Step 1 of the item form (HTMX swap into a modal or slide panel)
|
||||
- Filter by category (simple dropdown, no JS required — HTMX get)
|
||||
|
||||
### Add Item Flow (Two-Step)
|
||||
|
||||
The add item flow is a two-step process. The item is not saved to the database until the user confirms after seeing the preview.
|
||||
|
||||
**Step 1 — Entry form (`/items/new`)**
|
||||
|
||||
Fields:
|
||||
- Name (required)
|
||||
- Category (text, optional — HTMX autocomplete from existing categories)
|
||||
- Search Query (textarea, optional)
|
||||
- Product URL (text input, optional)
|
||||
- At least one of Search Query or URL is required — validate server-side before running preview
|
||||
- Target Price (number, optional — blank means alert on any result)
|
||||
- Ntfy Topic (pre-populated as a slug from Name, editable)
|
||||
- Ntfy Priority (select: min / low / default / high / urgent)
|
||||
- Poll Interval (select: 15min / 30min / 1hr / 2hr / 6hr / 12hr / 24hr)
|
||||
- Include Out of Stock (checkbox, default unchecked)
|
||||
|
||||
The primary action button is "Preview" — not "Save". Clicking it submits the form via HTMX POST to `/items/preview`.
|
||||
|
||||
**Step 2 — Preview (`POST /items/preview`)**
|
||||
|
||||
Server validates the form, runs an Apify search immediately, and returns a preview partial (`item_preview.templ`) swapped into the modal/panel. The original form values are preserved as hidden fields in the confirmation form.
|
||||
|
||||
The preview partial displays:
|
||||
- A header: "Found {N} results for '{query}'"
|
||||
- Best result card (largest, top of preview):
|
||||
- Product image (if available) on the left
|
||||
- Title, store name (as a link to the listing), and price on the right
|
||||
- Price in large monospaced font with the currency symbol
|
||||
- "Best Price" label above the price in mint green
|
||||
- Remaining results as a compact list below the best result card:
|
||||
- Each row: image thumbnail, title (truncated), store (linked), price
|
||||
- Show up to 5 additional results
|
||||
- If more than 6 results exist, show "and {N} more" below the list
|
||||
- Price range summary line: "Prices range from $X.XX to $Y.YY across {N} stores"
|
||||
- If no results found: show a warning card "No results found. Try a broader search query." with a Back button only
|
||||
- Two action buttons at the bottom:
|
||||
- "Back" — returns to the entry form with fields repopulated (HTMX swap)
|
||||
- "Confirm and Track" — POST to `/items` to save the item and start tracking
|
||||
|
||||
**Step 2 error states:**
|
||||
- Apify timeout or API error: show error message with a Back button
|
||||
- Zero results after filtering: show warning with Back button
|
||||
|
||||
**`POST /items` (confirm save)**
|
||||
- Validates all fields again server-side
|
||||
- Saves item to DB
|
||||
- Registers cron job in scheduler
|
||||
- Records initial `best_price`, `best_price_store`, `best_price_url`, `best_price_image_url`, `best_price_title` from the preview results (passed as hidden fields or re-fetched — re-fetching is acceptable)
|
||||
- Writes one row to `price_history` for the initial poll
|
||||
- Sets `last_polled_at` to now
|
||||
- Redirects to the new item's results page
|
||||
|
||||
### Edit Item (`/items/{id}/edit`)
|
||||
- Same fields as Step 1 of the add flow
|
||||
- No preview step on edit — changes save directly
|
||||
- Validation errors shown inline via HTMX
|
||||
|
||||
### Results and Price History (`/items/{id}/results`)
|
||||
- Item summary header: name, category, target price, best current price (store + link), last polled timestamp
|
||||
- **Deal Quality badge**: displayed prominently in the header next to the best price. Show the single highest-priority badge that applies, evaluated in this order:
|
||||
1. "All-time low" (mint green, `#00e4a4`) — current `best_price` is the lowest value ever recorded in `price_history` for this item
|
||||
2. "X% below 30-day avg" (Sega blue accent, `#00a4e4`) — current `best_price` is at least 10% below the mean of `price_history` rows from the last 30 days. X is the calculated percentage, rounded to the nearest whole number.
|
||||
3. "X% below target" (yellow, `#f5c400`) — a `target_price` is set and `best_price` is below it. X is the percentage difference.
|
||||
- Show no badge if none of the conditions are met. Do not show multiple badges simultaneously.
|
||||
- Price History chart: line chart rendered via Chart.js (CDN). X axis is poll date/time, Y axis is price. One data point per `price_history` row. Chart is dark-themed to match the app aesthetic (dark background, mint green line, coral point markers). If fewer than 2 data points exist, show "Not enough history yet" instead of the chart.
|
||||
- Results table below the chart:
|
||||
- Columns: Image (thumbnail), Title, Price, Store (linked), Found, Alert Sent
|
||||
- Price highlighted in mint green if at or below target
|
||||
- Sortable by price and found date (server-side, via HTMX get with sort params)
|
||||
- Pagination (20 per page)
|
||||
- "Run Now" button in the header — triggers an immediate Apify poll, updates best price fields, appends to price_history, refreshes the chart and table via HTMX
|
||||
|
||||
### Global Results (`/results`)
|
||||
- All results across all items, most recent first
|
||||
- Columns: Item Name, Title, Price, Store, Found, Alert Sent
|
||||
- Filterable by item and date range
|
||||
|
||||
### Settings (`/settings`)
|
||||
- Form fields for all configurable settings:
|
||||
- Apify API Key (password input, masked)
|
||||
- Ntfy Base URL
|
||||
- Ntfy Default Topic
|
||||
- Global Poll Interval
|
||||
- Match Confidence Threshold (slider or number input, 0.0–1.0)
|
||||
- Note: actor IDs are configured in `config.toml` and in per-item advanced settings, not here
|
||||
- "Test Ntfy" button — sends a test notification to the default topic, result shown inline
|
||||
- "Test Apify" button — runs a sample search ("test query"), shows raw result count inline
|
||||
- Save persists to DB settings table
|
||||
- Password Change section (separate from the above):
|
||||
- Current Password
|
||||
- New Password (min 12 characters)
|
||||
- Confirm New Password
|
||||
- Saved independently from other settings
|
||||
- User Management section (admin only):
|
||||
- List of all users with username, role, and created date
|
||||
- "Add User" form: username, role (admin/user), initial password
|
||||
- "Remove User" button per user (cannot remove yourself)
|
||||
- "Reset Password" button per user — admin sets a new password directly, no current password required
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
- All Apify failures: log error, mark item `last_poll_error` (add TEXT column to items), surface in UI as a warning badge
|
||||
- All Ntfy failures: log error, do not mark result as alerted, retry on next poll cycle
|
||||
- DB errors: log and return 500 with a friendly error page (no stack traces in UI)
|
||||
- Context cancellation on shutdown: in-progress Apify polls should be abandoned cleanly
|
||||
|
||||
---
|
||||
|
||||
## Graceful Shutdown
|
||||
|
||||
- Listen for SIGINT/SIGTERM
|
||||
- Stop cron scheduler (waits for running jobs to complete)
|
||||
- Close DB connection
|
||||
- Log "Veola shutting down"
|
||||
|
||||
---
|
||||
|
||||
## Editorial and Code Style Rules
|
||||
|
||||
- No em dashes anywhere in UI copy or log messages. Use a plain hyphen or restructure the sentence.
|
||||
- No explaining jokes or results to the user. Show the data, let them decide.
|
||||
- Log lines should be structured (key=value pairs), not prose sentences.
|
||||
- Do not add unrequested features. Build exactly what is specified.
|
||||
- Do not rewrite or restructure this document.
|
||||
- Comments in Go code should be terse and factual.
|
||||
- All configurable values (API keys, URLs, actor IDs, model names) must flow through the config struct — never hardcoded.
|
||||
- Templ components should be small and composable. Avoid monolithic template files.
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope (Do Not Implement)
|
||||
|
||||
- Matrix posting (future phase)
|
||||
- Discord integration
|
||||
- Email alerting
|
||||
- Mobile app
|
||||
|
||||
---
|
||||
|
||||
## Suggested Implementation Order for Claude Code
|
||||
|
||||
1. DB schema and migrations
|
||||
2. Config loading
|
||||
3. Crypto package (AES-256-GCM, key derivation, startup validation)
|
||||
4. Auth package (bcrypt, sessions, middleware)
|
||||
5. Login/setup/logout handlers and templates
|
||||
6. Models and DB query layer (encrypt/decrypt wired in)
|
||||
7. Apify client (with a test harness)
|
||||
8. Ntfy client
|
||||
9. Templ layout + base styles (auth-aware nav)
|
||||
10. Items list handler and template
|
||||
11. Add item Step 1 form
|
||||
12. Add item Step 2 preview handler and template
|
||||
13. Confirm save handler (wires to scheduler)
|
||||
14. Edit item handler
|
||||
15. Settings page (including password change)
|
||||
16. Scheduler wired to Apify + Ntfy + price_history writes
|
||||
17. Dashboard
|
||||
18. Per-item results page with Chart.js price history
|
||||
19. Global results view
|
||||
20. "Run Now" HTMX endpoint
|
||||
21. Graceful shutdown
|
||||
|
||||
---
|
||||
|
||||
## Agreed Deviations from this Spec
|
||||
|
||||
These items were discussed with the operator on 2026-05-10 before implementation and override the corresponding sections above. They are appended rather than edited in place to preserve the original spec text.
|
||||
|
||||
1. **Encrypted-columns list pruned.** Random-nonce AES-256-GCM produces non-deterministic ciphertext, which breaks the two equality lookups the spec itself requires: login (`SELECT ... WHERE username = ?`) and result deduplication (`SELECT ... WHERE item_id = ? AND url = ?`, spec line 565). Therefore drop encryption from these columns:
|
||||
- `users.username`
|
||||
- `items.url`
|
||||
- `results.url`
|
||||
|
||||
Keep encryption on all other columns listed in the Data Encryption table (lines 320-326): `settings.value`, `items.search_query`, `items.ntfy_topic`, `items.best_price_url`, `items.best_price_image_url`, `items.best_price_title`, `items.last_poll_error`, `results.title`, `results.image_url`, `price_history.store`.
|
||||
|
||||
2. **Sessions use `github.com/alexedwards/scs/v2`** with `github.com/alexedwards/scs/sqlite3store`, not `gorilla/sessions`. `gorilla/sessions` is in maintenance mode; `scs` is actively maintained and its sqlite store fits the existing DB. The `security.session_secret` config value is still the source of the signing key.
|
||||
|
||||
3. **CSRF protection is implemented as per-session synchronizer tokens plus `SameSite=Lax` session cookies.** The spec's note that POST-only logout is "CSRF-safe" (line 286) is incorrect on its own; a token system is required for all state-changing forms (item create/edit/delete, settings save, password change, user management, logout, pause/resume, run-now). Middleware validates the token on every non-idempotent method.
|
||||
|
||||
4. **Vendor HTMX and Chart.js into `static/`** rather than loading from a CDN. The app is self-hosted and should not break when the host has no internet. Tailwind remains on the Play CDN for v1 (per spec line 22), with a documented path to swap for a built CSS file. Google Fonts (JetBrains Mono, Outfit) remain on CDN.
|
||||
|
||||
5. **Preview results are cached in-memory for 10 minutes,** keyed on the tuple `(query, url, marketplace, listing_type, max_results)`. A re-preview within the window reuses the cached results and does not call Apify. This avoids burning Apify credits when the operator iterates on the add-item form. Cache is process-local, lost on restart.
|
||||
|
||||
6. **Tests are written for these areas only:** crypto round-trip and tamper-detection; scheduler alert-condition logic; result deduplication; deal-quality badge selection. Other code is untested for v1.
|
||||
|
||||
7. **Apify actor IDs are verified at build time before populating `config.toml.example`.** Each ID listed in the spec (lines 360, 402, 428, 453, 466, 502-505) is checked against `apify.com/store`. If an actor cannot be confirmed to exist with reasonable activity and success rate, its config key is left blank and the app fails fast at startup with a message naming the unset key. The Mercari JP actor was already flagged for verification by the spec itself (lines 488-489).
|
||||
Reference in New Issue
Block a user