diff --git a/README.md b/README.md index ec1f8a1..d1ae0cd 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,14 @@ # Veola -Self-hosted Go web app that tracks items across e-commerce platforms (eBay, Amazon family, Yahoo Auctions JP, Mercari JP) via the [Apify](https://apify.com) scraping API and pushes deal alerts to a self-hosted [ntfy](https://ntfy.sh) instance. +Self-hosted Go web app that tracks items across e-commerce platforms and pushes deal alerts to a self-hosted [ntfy](https://ntfy.sh) instance. eBay marketplaces are polled through eBay's official [Browse API](https://developer.ebay.com/api-docs/buy/browse/overview.html); Amazon family, Yahoo Auctions JP, and Mercari JP go through the [Apify](https://apify.com) scraping API. Track. Watch. Notice. ## Features - Watch arbitrary items across multiple marketplaces with per-item search queries, target prices, and poll intervals -- Active-listing, sold-listing, and price-comparison actors per item +- eBay via the official eBay Browse API, with a per-day call quota tracked and enforced — polling halts before the limit and resets on eBay's Pacific-time clock +- Apify active-listing, sold-listing, and price-comparison actors for the non-eBay marketplaces - Price-history chart and best-price badge once enough history accumulates - Deal alerts pushed to ntfy when current price falls at or below target - Single-binary deploy, SQLite storage, no CGO @@ -17,22 +18,30 @@ See [`veola-spec.md`](veola-spec.md) for the full specification. ## Requirements - Go 1.22+ (developed against 1.25) -- An [Apify](https://apify.com) account + API key - A reachable [ntfy](https://ntfy.sh) instance (self-hosted or ntfy.sh) +- An [eBay developer](https://developer.ebay.com) keyset (App ID + Cert ID) — for eBay marketplaces +- An [Apify](https://apify.com) account + API key — for the non-eBay marketplaces +- To build from source: the [`templ`](https://templ.guide) CLI. The Tailwind standalone CLI is fetched automatically by the Makefile — no Node toolchain required. ## Build ```sh -go build -o veola-bin . +make build ``` +This runs `templ generate`, compiles Tailwind, and produces `veola-bin`. Makefile targets: + +| Target | What it does | +| --- | --- | +| `make generate` | Regenerate templ Go from `.templ` sources | +| `make css` | Compile `static/css/tailwind.css` from `static/css/input.css` (fetches the Tailwind standalone CLI into `bin/` on first run) | +| `make build` | `generate` + `css` + `go build -o veola-bin .` | +| `make run` | `build`, then run against `config.toml` | +| `make test` | `go test ./...` | + The binary is named `veola-bin` rather than `veola` because the module is also `veola` — `go build` cannot write a binary with the same name as the module dir. -If you change any `.templ` files, regenerate first: - -```sh -~/go/bin/templ generate -``` +`static/css/tailwind.css` is committed, so a deploy box can `go build -o veola-bin .` without the Tailwind CLI. Re-run `make css` whenever you change templates or `static/css/input.css`. The hand-written component layer in `static/css/app.css` is loaded separately and needs no rebuild. ## Configure @@ -48,7 +57,12 @@ Both `session_secret` and `encryption_key` must be at least 32 bytes and differe openssl rand -hex 32 ``` -`encryption_key` encrypts secrets at rest in SQLite (Apify keys, ntfy settings). Rotating it invalidates stored secrets — re-enter them through `/settings` after rotation. +`encryption_key` encrypts secrets at rest in SQLite (Apify key, eBay credentials, ntfy settings). Rotating it invalidates stored secrets — re-enter them through `/settings` after rotation. + +Other notable config: + +- `[ebay]` — `client_id` (App ID), `client_secret` (Cert ID), `environment` (`production` or `sandbox`), and `daily_call_limit` (Browse API calls per day; default 5000). All of `client_id` / `client_secret` / the limit can also be set or overridden at runtime via `/settings`. +- `server.secure_cookies` — sets the `Secure` attribute on the session cookie. Defaults to `true`; keep it on for any HTTPS-reachable deployment, **including behind a TLS-terminating proxy**. Set `false` only for local plain-HTTP testing on a non-localhost address. ## Run @@ -61,7 +75,7 @@ First-run flow: 1. Visit `http://localhost:8080/`. With no users, you are redirected to `/setup`. 2. Create the admin account. 3. Log in at `/login`. -4. Add items at `/items/new`. Optionally fill in your Apify key and ntfy URL via `/settings` if you didn't put them in `config.toml`. +4. Add items at `/items/new`. Optionally fill in your eBay/Apify credentials and ntfy URL via `/settings` if you didn't put them in `config.toml`. The Settings page also shows the running eBay API call count for the day. The scheduler starts with the server and polls each active item on its configured interval. The bottom-of-hour global poll runs every `scheduler.global_poll_interval_minutes`. @@ -75,12 +89,15 @@ internal/ db/ SQLite schema, migrations, store models/ domain types apify/ Apify API client + ebay/ eBay Browse API client (OAuth2 + item search) ntfy/ ntfy push client auth/ session + CSRF scheduler/ poll loop, alert/dedup/badge logic handlers/ HTTP handlers templates/ templ components -static/ CSS, vendored htmx +static/ compiled Tailwind + app.css, vendored htmx/Chart.js, JS +tailwind.config.js Tailwind content globs +Makefile build targets ``` ## Test @@ -89,14 +106,23 @@ static/ CSS, vendored htmx go test ./... ``` -Unit tests cover crypto round-trip, db round-trip and dedup, and scheduler alert/badge logic. No handler-level tests yet. +Unit tests cover crypto round-trip, db round-trip and dedup, scheduler alert/badge logic, and eBay marketplace/filter mapping. No handler-level tests yet. ## Operate - The SQLite file lives at `server.db_path` (default `./veola.db`). Back this up — it holds your watched items, history, encrypted secrets, and user accounts. +- `config.toml` and `veola.db` (plus its `-wal`/`-shm`) hold secrets and live session tokens — keep them `chmod 600` and owned by the service user. - The process responds to `SIGINT` / `SIGTERM` with a graceful HTTP shutdown (30s timeout) followed by scheduler stop. - Logs go to stdout as structured `log/slog` records. +### Deploying publicly + +Veola speaks plain HTTP and is meant to sit behind a TLS-terminating reverse proxy (e.g. Traefik, Caddy, nginx). + +- Keep `server.secure_cookies = true` (the default). +- Terminate TLS at the proxy and set HSTS there — Veola does not emit HSTS itself. +- Veola sets `Content-Security-Policy`, `X-Frame-Options`, `X-Content-Type-Options`, and `Referrer-Policy` on every response, and trusts `X-Forwarded-For` for client IPs — configure the proxy to strip client-supplied `X-Forwarded-*` headers so they cannot be spoofed. + ## Aesthetic Sega Genesis blue. Not dark mode, not light mode — blue mode. See the visual design section of `veola-spec.md` for the palette. diff --git a/internal/handlers/dashboard.go b/internal/handlers/dashboard.go index 825df01..fcdd532 100644 --- a/internal/handlers/dashboard.go +++ b/internal/handlers/dashboard.go @@ -24,10 +24,11 @@ func (a *App) GetDashboardRefresh(w http.ResponseWriter, r *http.Request) { http.Error(w, err.Error(), http.StatusInternalServerError) return } - // Render only the inner block by reusing the full body component; the - // outer hx-swap="outerHTML" replaces the same wrapper. The full Dashboard - // template is overkill but keeps a single source of truth. - render(w, r, templates.Dashboard(d)) + // Render ONLY the inner body. The hx-swap="outerHTML" on DashboardBody's + // root div replaces it with this fresh copy. Rendering templates.Dashboard + // here would return the whole Layout — sidebar included — nested inside + // the div, producing a duplicate nav bar on every refresh. + render(w, r, templates.DashboardBody(d)) } func (a *App) dashboardData(r *http.Request) (templates.DashboardData, error) { diff --git a/internal/handlers/items.go b/internal/handlers/items.go index 5d38d66..412b19a 100644 --- a/internal/handlers/items.go +++ b/internal/handlers/items.go @@ -2,13 +2,13 @@ package handlers import ( "context" - "encoding/json" "fmt" "log/slog" "net/http" "sort" "strconv" "strings" + "time" "veola/internal/apify" "veola/internal/models" @@ -261,46 +261,16 @@ func (a *App) runPreview(ctx context.Context, it models.Item) ([]apify.UnifiedRe var merged []apify.UnifiedResult primarySource := "" for _, p := range plans { - actorID := p.ActorID() - if actorID == "" { - continue - } - raw, err := a.Apify.Run(ctx, actorID, p.Input()) + decoded, err := a.Scheduler.ExecutePlan(ctx, p) if err != nil { - slog.Warn("preview run failed", "actor", actorID, "query", p.Query(), "err", err) + slog.Warn("preview plan failed", + "provider", p.Provider(), + "marketplace", p.Marketplace(), + "query", p.Query(), + "err", err, + ) continue } - decoded, _ := apify.Decode(raw, p.Source()) - for i := range decoded { - decoded[i].MatchedQuery = p.Query() - } - usable := 0 - for _, r := range decoded { - if r.URL != "" && r.Price > 0 { - usable++ - } - } - slog.Info("preview decoded", - "marketplace", previewMarket, - "actor", actorID, - "query", p.Query(), - "raw", len(raw), - "decoded", len(decoded), - "usable", usable, - ) - if usable == 0 && len(raw) > 0 { - var sample map[string]any - if err := json.Unmarshal(raw[0], &sample); err == nil { - ks := make([]string, 0, len(sample)) - for k := range sample { - ks = append(ks, k) - } - slog.Warn("preview decoded zero usable rows; raw item keys", - "actor", actorID, - "keys", ks, - ) - } - } merged = append(merged, decoded...) if primarySource == "" { primarySource = p.Source() @@ -431,9 +401,40 @@ func (a *App) PostRunItem(w http.ResponseWriter, r *http.Request) { http.NotFound(w, r) return } - go a.Scheduler.RunPoll(context.Background(), *it) - // Re-render the row immediately so HTMX has something to swap in. - render(w, r, templates.ItemRow(*it, a.Auth.CSRFToken(r.Context()))) + + // Run synchronously so the response reflects the finished poll. Bounded so + // a slow Apify actor run can't tie the request up indefinitely (eBay + // Browse API polls finish in seconds, well within this). Detached from the + // request context so a client disconnect mid-run doesn't abort DB writes. + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + a.Scheduler.RunPoll(ctx, *it) + + // RunPoll writes best price, last_polled_at, and last_poll_error; re-fetch + // so the rendered partial shows the post-poll state. + fresh, err := a.Store.GetItem(r.Context(), id) + if err != nil || fresh == nil { + http.Error(w, "could not reload item after run", http.StatusInternalServerError) + return + } + + // The results page asks for a refreshed listing table; the items list + // asks for a refreshed row. Both POST to this same endpoint. + if r.PostFormValue("from") == "results" { + d, err := a.buildItemResultsData(r, fresh, 1, "found_desc") + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + if fresh.LastPollError != "" { + d.RunError = "Run finished with errors: " + fresh.LastPollError + } else { + d.RunMsg = fmt.Sprintf("Run complete. Showing %d listing(s).", len(d.Results)) + } + render(w, r, templates.ItemResultsTable(d)) + return + } + render(w, r, templates.ItemRow(*fresh, a.Auth.CSRFToken(r.Context()))) } func (a *App) GetItemError(w http.ResponseWriter, r *http.Request) { diff --git a/internal/handlers/results.go b/internal/handlers/results.go index 6620999..d6dbc92 100644 --- a/internal/handlers/results.go +++ b/internal/handlers/results.go @@ -21,20 +21,29 @@ func (a *App) GetItemResults(w http.ResponseWriter, r *http.Request) { http.NotFound(w, r) return } + page, _ := strconv.Atoi(r.URL.Query().Get("page")) + d, err := a.buildItemResultsData(r, it, page, r.URL.Query().Get("order")) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + render(w, r, templates.ItemResults(d)) +} - order := r.URL.Query().Get("order") +// buildItemResultsData assembles the per-item results view: paginated results, +// price history, badge, and chart JSON. Shared by GetItemResults and the +// "Run Now" handler so both render identical data. +func (a *App) buildItemResultsData(r *http.Request, it *models.Item, page int, order string) (templates.ItemResultsData, error) { if order == "" { order = "found_desc" } - page, _ := strconv.Atoi(r.URL.Query().Get("page")) if page < 1 { page = 1 } - total, err := a.Store.CountResults(r.Context(), id) + total, err := a.Store.CountResults(r.Context(), it.ID) if err != nil { - http.Error(w, err.Error(), http.StatusInternalServerError) - return + return templates.ItemResultsData{}, err } totalPages := (total + resultsPerPage - 1) / resultsPerPage if totalPages < 1 { @@ -45,36 +54,31 @@ func (a *App) GetItemResults(w http.ResponseWriter, r *http.Request) { } results, err := a.Store.ListResults(r.Context(), db.ResultsQuery{ - ItemID: id, + ItemID: it.ID, Limit: resultsPerPage, Offset: (page - 1) * resultsPerPage, Order: order, }) if err != nil { - http.Error(w, err.Error(), http.StatusInternalServerError) - return + return templates.ItemResultsData{}, err } - history, err := a.Store.ListPriceHistory(r.Context(), id) + history, err := a.Store.ListPriceHistory(r.Context(), it.ID) if err != nil { - http.Error(w, err.Error(), http.StatusInternalServerError) - return + return templates.ItemResultsData{}, err } - badge := scheduler.PickBadge(*it, history, time.Now()) - chart := buildChartJSON(history) - - render(w, r, templates.ItemResults(templates.ItemResultsData{ + return templates.ItemResultsData{ Page: a.page(r, it.Name, "items"), Item: *it, - Badge: badge, + Badge: scheduler.PickBadge(*it, history, time.Now()), History: history, Results: results, Page_: page, TotalPages: totalPages, Order: order, - HistoryChartJSON: chart, - })) + HistoryChartJSON: buildChartJSON(history), + }, nil } func buildChartJSON(history []models.PricePoint) string { diff --git a/static/css/app.css b/static/css/app.css index 4f9b992..c3a2f21 100644 --- a/static/css/app.css +++ b/static/css/app.css @@ -193,11 +193,6 @@ table.v-table tr:hover td { background: rgba(255,255,255,0.03); } margin-bottom: 1rem; } -/* htmx indicator: hidden by default, visible during in-flight requests. */ -.htmx-indicator { display: none; } -.htmx-request .htmx-indicator, -.htmx-request.htmx-indicator { display: inline-flex; } - .v-spinner { display: inline-block; width: 14px; height: 14px; @@ -207,3 +202,11 @@ table.v-table tr:hover td { background: rgba(255,255,255,0.03); } animation: v-spin 0.8s linear infinite; } @keyframes v-spin { to { transform: rotate(360deg); } } + +/* htmx indicator: hidden by default, visible during in-flight requests. + Declared AFTER .v-spinner on purpose: an element carrying both classes + (e.g.