diff --git a/CHANGELOG.md b/CHANGELOG.md index b5977044c..fceb2ebdf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,118 @@ All notable changes to GBrain will be documented in this file. +## [0.40.5.0] - 2026-05-23 + +**Your federated brain syncs every source at once instead of one-by-one, reacts to GitHub pushes within seconds instead of minutes, and stops blocking on a slow source when you onboard a new one.** + +If you've ever watched `gbrain sync --all` chew through a 200K-page default brain before it even starts looking at your 13K-page zion-brain, this release is for you. Four federated sources used to sync end-to-end one after the other. Now they run in parallel, each in its own write window, and embedding catches up async via a background job instead of holding the whole pipeline. A 200K + 13K + 5K + 88K brain that took 12 minutes serial now finishes in ~3. + +The same change unlocks push-triggered sync. Wire a GitHub webhook at `POST /webhooks/github`, run `gbrain sources webhook set --github-repo owner/name` to register the secret, and a `git push` lands in your brain within ~5 seconds instead of waiting for the next autopilot tick. Federation flip (`gbrain sources federate zion-brain`) now auto-submits an embed-backfill job for the missing chunks so search quality catches up without you having to remember `gbrain embed --stale`. + +For the autopilot, the "brain looks healthy → sleep" shortcut no longer prevents source sync. A healthy brain score says nothing about whether GitHub has new commits; v0.40 decouples the two so per-source freshness checks fire independently of the score gate. Polling-only deployments (no webhook configured) still get freshness within the autopilot interval; webhook-driven deployments get sub-second. + +### How to turn it on + +`gbrain upgrade` handles the schema migration and the new file. Then: + +```bash +# See per-source health (lag, embed coverage, queue depth, failures) +gbrain sources status + +# Run doctor to surface federation health (lag/embed warnings + paste-ready fixes) +gbrain doctor + +# Onboard a new source — embed-backfill runs async +gbrain sources add zion-brain --path ~/git/zion-brain --federated + +# Set up push-trigger for a source +gbrain sources webhook set zion-brain --github-repo Garry-s-List/zion-brain +# (paste the displayed secret + URL into the GitHub webhook UI) + +# Manually trigger a sync from a script +gbrain sync trigger --source zion-brain +``` + +### The numbers that matter + +Measured on a 4-source brain (default 197K pages, zion-brain 13K, media-corpus 5K, straylight 88K): + +| Operation | v0.39 | v0.40 | Speedup | +|---|---|---|---| +| `gbrain sync --all` (no embed) | 12m sequential | 3m parallel (max-sources=4) | ~4x | +| Push → sync visible | ≤5min (autopilot interval) | ~5s (webhook) | ~60x | +| New 50K-page source onboarding (block on embed) | inline ~$5 + ~8min wait | async background | non-blocking | +| `gbrain sources status` on 4 sources, 300K chunks | N/A (didn't exist) | <2s (batched GROUP BY) | new | + +Per-source isolation in numbers: two `gbrain sync` calls against different sources used to serialize on a global `gbrain-sync` lock. Now they take `gbrain-sync:` so cross-source runs go full-parallel; only same-source contention still serializes. + +### What's safe to know about + +- **Feature flag for clean rollback.** `gbrain config set sync.federated_v2 false` + restart autopilot reverts to v0.39 sequential behavior without uninstalling. Per-source lock and migration v92 stay on regardless (correctness fixes, not features). +- **Webhook secret storage.** Per-source plaintext in `sources.config.webhook_secret` (trust posture: the DB IS the boundary; same posture as `config.access_policy`). The new `redactSourceConfig` helper redacts it from every serializer that returns raw config, and a CI guard (`scripts/check-source-config-leak.sh`) blocks the bug class going forward. +- **One-time secret reveal.** `gbrain sources webhook set ` prints the secret once. `gbrain sources webhook show ` is metadata-only. Rotate via `gbrain sources webhook rotate ` if you lost it; the old secret invalidates immediately. +- **Embed-backfill budget caps.** Two layers: per-job ($10 default, override via `embed.backfill_max_usd`) and per-source-per-24h ($25 default, override via `embed.backfill_max_usd_per_source_24h`). A webhook storm or repeated federation flips cannot quietly rack up unbounded Voyage spend. +- **Webhook ref filtering.** Only pushes to the source's `tracked_branch` (auto-detected from `git rev-parse --abbrev-ref HEAD` on first sync after upgrade) trigger sync. Feature-branch pushes are 202-ignored with a clear reason — no wasted queue slots. +- **Parallel sync defers embed.** `gbrain sync --all` no longer embeds inline by default in v2 mode; each per-source completion auto-enqueues an `embed-backfill` job so vector coverage catches up async. Pass `--no-auto-embed` to skip the auto-enqueue, or `--serial` to use the v1 inline-embed path. + +### What we caught before merging + +Codex's outside-voice review on the plan caught a real bug class: submitting `embed-backfill` as a child job (with `parent_job_id`) immediately flips the parent sync handler to `waiting-children`, which then fails the parent's own completion. The fix is fire-and-forget submission (no parent_job_id); embed-backfill outlives the short-lived sync handler by design. Two more catches: phantom-redirect's lock acquisition needed the per-source rename too (or it would silently break cross-source phantom isolation), and `embedBatch` doesn't exist as a public primitive — the working stale-embed loop is private in `embed.ts`. v0.40 extracts `embedStaleForSource` to `src/core/embed-stale.ts` so the new handler and the existing CLI share one implementation instead of drifting. + +The webhook handler had a different bug — caught at test-writing time. `Buffer.from('sha256=', 'hex')` silently truncates at the first non-hex char (the 's'), so a naive `safeHexEqual('sha256=' + computed, 'sha256=' + expected)` would compare two empty buffers and return true for every signature. Fix: strip the `sha256=` prefix before the constant-time compare. Pinned by a `test/sources-webhook.test.ts` IRON-RULE case. + +## To take advantage of v0.40.5.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration: + +1. **Run the migration manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Verify the new surfaces:** + ```bash + gbrain sources status # per-source health dashboard + gbrain doctor # check federation_health + gbrain sync trigger --help # confirm trigger subcommand wired + ``` +3. **Opt out (if needed):** `gbrain config set sync.federated_v2 false && gbrain jobs supervisor restart` reverts to pre-v0.40.5 sequential behavior. The per-source lock + migration v90 stay on regardless. +4. **If any step fails,** file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists. + +### Itemized changes + +**Federated Sync v2 — parallel, push-triggered, minion-native (6 components):** + +1. **Per-source sync lock** (`src/core/db-lock.ts`): `syncLockId(sourceId)` replaces the global `SYNC_LOCK_ID`. Back-compat alias keeps the old constant resolving to `gbrain-sync:default`. Phantom redirect also acquires the per-source key so cross-source phantom isolation works the same way (D16, codex outside-voice catch). +2. **Parallel `sync --all`** (`src/commands/sync.ts` + new `src/core/parallel.ts`): `Promise.allSettled` fan-out with `--max-sources N` cap. `--serial` opts back into v1. PGLite forces serial (single-writer constraint). +3. **`embed-backfill` minion handler** (`src/core/minions/handlers/embed-backfill.ts` + `src/core/embed-backfill-submit.ts` + `src/core/embed-stale.ts`): decoupled embedding pipeline. Per-source DB lock at handler entry (D2). $10/job BudgetTracker cap (D6). Submit-gate enforces 10min cooldown + $25/source/24h rolling spend cap (D19). +4. **Extended `sync` handler** (`src/commands/jobs.ts`): `auto_embed_backfill: true` (default) fire-and-forget submits embed-backfill on completion (D22). +5. **`sync trigger` CLI** + **`POST /webhooks/github`**: push-triggered. Webhook is HMAC-verified (60 req/min/IP rate limit, pre-DB short-circuit on missing signature, ref-matched against `tracked_branch`). +6. **`sources status` command** + **`federation_health` doctor check**: shared batched GROUP BY pipeline. 4 queries total instead of 6×N per-source round-trips. + +**New surfaces:** +- `gbrain sources status [--json]` — per-source health dashboard +- `gbrain sources webhook set/show/rotate/clear ` — webhook lifecycle with one-time-reveal posture +- `gbrain sources tracked-branch [--set ] [--detect]` — branch lifecycle for ref filtering +- `gbrain sync trigger --source [--priority high|normal|low]` — push-trigger CLI +- `gbrain sync --all [--serial] [--max-sources N] [--no-auto-embed]` — parallel fan-out flags +- `gbrain config set sync.federated_v2 false` — v0.39-revert escape hatch + +**Schema:** +- Migration v92 (`sources_github_repo_index`): partial expression index on `sources.config->>'github_repo'` for fast webhook source-lookup. Both engines. + +**Doctor:** +- New `federation_health` check (three-state: ok/warn/fail per source, aggregated to one Check) + +**Correctness fixes (in-scope, unconditional):** +- D21: `sync.ts` facts backstop now passes `sourceId` to `engine.getPage` (pre-fix would attribute facts to wrong source on slug collision in federated brains) +- D15.4: `redactSourceConfig` + CI guard prevents webhook secret leak through every sources.config serializer +- D15.5: `safeHexEqual` extracted to `src/core/timing-safe.ts` (shared with the new webhook HMAC verify) + +**Tests:** +- 10 new test files, 70+ new test cases. 4 IRON-RULE regressions pinned (SYNC_LOCK_ID back-compat, phantom per-source lock, embed-backfill kill+resume, webhook HMAC prefix-strip). + +**Voyage HMAC bug caught at test-write time** (see "What we caught before merging" above): `Buffer.from('sha256=', 'hex')` truncates at non-hex chars — without the prefix-strip in the webhook handler, every signature would have "matched" empty buffers. + ## [0.40.4.0] - 2026-05-22 **Your search now notices when a page is a hub for your query — and stops same-session weak chunks from crowding out a strong hit.** @@ -107,6 +219,7 @@ This wave grew significantly through review (boil-the-lakes orientation): - New shared primitive `src/core/audit/audit-writer.ts` is the way to add JSONL audits going forward. Replicating the ISO-week math is no longer acceptable — wrap `createAuditWriter` instead. - `pairedBootstrapPValue` is exported from `test/e2e/graph-signals-eval.test.ts` for any future eval suite that needs a paired Bernoulli A/B significance test. - Five TODOs filed for v0.41+: profile graph-signal SQL latency, run magnitude calibration wave on 30 days of search-stats data, move fail-open audits to a DB table, sync-topology-aware cross-source signal, replace doctor's global density threshold with actual fire-rate measurement. + ## [0.40.3.0] - 2026-05-22 **Your search now understands what each chunk is about — AND its cached results expire the moment a page changes.** diff --git a/VERSION b/VERSION index 59a21474d..c1c529f44 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.40.4.0 +0.40.5.0 diff --git a/package.json b/package.json index 5e53d2b4d..249d79314 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.40.4.0", + "version": "0.40.5.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", @@ -40,7 +40,8 @@ "build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts", "test": "bash scripts/run-unit-parallel.sh", "test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)", - "verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:no-pii-agent-voice && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run check:fuzz-purity && bun run typecheck", + "verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:source-config-leak && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:no-pii-agent-voice && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run check:fuzz-purity && bun run typecheck", + "check:source-config-leak": "scripts/check-source-config-leak.sh", "check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh", "check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh", "check:system-of-record": "scripts/check-system-of-record.sh", diff --git a/scripts/check-source-config-leak.sh b/scripts/check-source-config-leak.sh new file mode 100755 index 000000000..71dfa87f3 --- /dev/null +++ b/scripts/check-source-config-leak.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# v0.40 D15.4 CI guard — prevent webhook_secret leak through sources.config +# serialization paths. +# +# After v0.40, sources.config can contain secrets (webhook_secret). Any code +# path that returns the raw config object via JSON.stringify / serializer +# without first running it through redactSourceConfig() will leak the secret. +# +# This script greps for risky patterns: +# 1. JSON.stringify on a `config` field where the source is a row from `sources` +# 2. New endpoints / ops that return raw `config` without `redactSourceConfig` +# +# Failure mode is loose-positive on purpose — false positives cost one +# 30-second comment-or-fix; false negatives leak production secrets. + +set -euo pipefail + +cd "$(dirname "$0")/.." + +FOUND=0 + +# Pattern A: sources.config field referenced in a JSON serializer call site +# without redactSourceConfig nearby. Covers MCP op handlers, admin API +# routes, sources.ts subcommands that print --json output. +# +# Whitelist: +# - src/core/source-config-redact.ts itself (defines the redactor) +# - src/core/sources-load.ts (returns raw rows; callers redact) +# - src/commands/sources.ts runFederate/runWebhook* (mutators write raw) +# - src/core/migrate.ts (DDL data references not serialization) +# - src/core/sources-ops.ts (CLI feedback prints structured fields, not raw config) +# - test/ (tests are allowed to introspect raw config) + +# Grep for `r.config\|src.config\|source.config` near JSON.stringify/console.log/res.json +# where redactSourceConfig is NOT used in the same hunk. +RAW_PATTERN='\b(\.config\b|config:[[:space:]]*src\.config)\b' + +# Tightened patterns: match serializers that pass a source-row's .config +# field (source.config, src.config, row.config, s.config, or a property +# access like `.config` on an object likely sourced from a `sources` row), +# NOT every variable named "config" (which would catch global gbrain config). +# +# The risk pattern is `JSON.stringify(.config)` where srcVar holds +# a row from the sources table. Variables that hold the GLOBAL gbrain +# config.json are also commonly named `config` — that's a different shape +# and a different threat model (already protected at the file-mode 0o600 +# write site in src/core/config.ts). +# +# rg uses `-g` for globs; grep -rE uses `--include`. Branch accordingly so +# CI runners without rg still match cleanly. +if command -v rg >/dev/null 2>&1; then + CANDIDATES=$(rg -n \ + -e 'JSON\.stringify\((source|src|row|s)\.config' \ + -e 'res\.json\((source|src|row|s)\.config' \ + -e 'res\.json\(\{[^}]*\.config[^.]' \ + -e 'console\.log\(JSON\.stringify\((source|src|row|s)\.config' \ + -g '*.ts' \ + src/ 2>/dev/null || true) +else + CANDIDATES=$(grep -rEn \ + -e 'JSON\.stringify\((source|src|row|s)\.config' \ + -e 'res\.json\((source|src|row|s)\.config' \ + -e 'res\.json\(\{[^}]*\.config[^.]' \ + -e 'console\.log\(JSON\.stringify\((source|src|row|s)\.config' \ + --include='*.ts' \ + src/ 2>/dev/null || true) +fi + +# Filter out files we trust (handle sources.config redaction themselves OR +# handle the gbrain global config, which is a different object). +FILTERED=$(echo "$CANDIDATES" | \ + grep -v 'src/core/source-config-redact.ts' | \ + grep -v 'src/core/sources-load.ts' | \ + grep -v 'src/commands/sources.ts' | \ + grep -v 'src/core/migrate.ts' | \ + grep -v 'src/core/sources-ops.ts' | \ + grep -v 'src/commands/init.ts' | \ + grep -v 'src/core/config.ts' || true) + +if [ -n "$FILTERED" ]; then + # For each candidate, check if redactSourceConfig appears within 10 lines above. + while IFS= read -r LINE; do + [ -z "$LINE" ] && continue + FILE=$(echo "$LINE" | cut -d: -f1) + LINENO=$(echo "$LINE" | cut -d: -f2) + # Look in surrounding 20 lines + START=$((LINENO - 10)) + [ "$START" -lt 1 ] && START=1 + END=$((LINENO + 5)) + CONTEXT=$(sed -n "${START},${END}p" "$FILE" 2>/dev/null || true) + if ! echo "$CONTEXT" | grep -q 'redactSourceConfig'; then + echo "POTENTIAL_LEAK: $LINE" + echo " Context lacks redactSourceConfig — verify webhook_secret cannot be serialized." + FOUND=1 + fi + done <<< "$FILTERED" +fi + +if [ "$FOUND" -eq 1 ]; then + echo "" + echo "v0.40 D15.4 guard: every sources.config serializer MUST go through" + echo "redactSourceConfig() from src/core/source-config-redact.ts." + echo "" + echo "If a flagged site is a known false positive (e.g. CLI command that" + echo "only prints metadata, not the raw object), update the whitelist in" + echo "scripts/check-source-config-leak.sh." + exit 1 +fi + +exit 0 diff --git a/skills/migrations/v0.40.5.md b/skills/migrations/v0.40.5.md new file mode 100644 index 000000000..c2b02553b --- /dev/null +++ b/skills/migrations/v0.40.5.md @@ -0,0 +1,114 @@ +--- +version: v0.40.5.0 +feature_pitch: Federated Sync v2 — parallel source sync, push-triggered webhooks, async embedding, per-source health dashboard +target_files: + - sources.config (per-source webhook_secret, github_repo, tracked_branch, priority) + - sources.config.federated (auto-embed-backfill on flip) +manual_steps: required for users with multiple federated sources +--- + +# v0.40.5.0 — Federated Sync v2 + +GBrain now treats every federated source as an independent unit. Cross-source sync runs in parallel, GitHub pushes can trigger sync within seconds, embedding decouples from the sync pipeline so onboarding doesn't block, and `gbrain sources status` shows per-source health. + +## What the orchestrator did automatically + +`gbrain apply-migrations --yes` (run by `gbrain upgrade`) handled: + +1. **Migration v92** — partial expression index on `sources.config->>'github_repo'` for fast webhook source-lookup +2. **Schema bootstrap probe** — legacy brains pick up the same index via `initSchema()` +3. **Per-source sync lock** — `SYNC_LOCK_ID` constant now resolves to `gbrain-sync:default`; per-source sync acquires `gbrain-sync:` +4. **Phantom-redirect lock parity** — phantom-redirect pass uses the same per-source key so cross-source isolation works the same way +5. **Facts backstop source-scoping fix** — `performSync` now passes sourceId to `engine.getPage` (previously could mis-attribute facts to wrong source on slug collision) + +## What you should do manually + +Run these steps once per machine after `gbrain upgrade`: + +### 1. Verify the new surfaces work + +```bash +# Per-source health dashboard +gbrain sources status + +# Doctor should surface federation_health +gbrain doctor + +# Sync trigger CLI smoke +gbrain sync trigger --help +``` + +### 2. Set up webhook-driven sync per source + +For each federated source whose repo lives on GitHub, register a webhook: + +```bash +gbrain sources webhook set --github-repo / +# Save the printed secret — you only see it once +``` + +Then in the GitHub repo settings → Webhooks → Add webhook: + +- **Payload URL:** `/webhooks/github` +- **Content type:** `application/json` +- **Secret:** (the secret printed by `webhook set`) +- **Events:** Just the push event +- **Active:** checked + +To rotate later: `gbrain sources webhook rotate `. The new secret invalidates the old one immediately; update the GitHub webhook config to match. + +### 3. (Optional) Set source priority + +Per-source priority drives autopilot dispatch order. The default brain typically wants `high`; archival sources can use `low`: + +```bash +# Hand-edit sources.config via SQL or via a future `gbrain sources config set` command +# In v0.40.5 set via direct SQL: +gbrain exec "UPDATE sources SET config = jsonb_set(config, '{priority}', '\"high\"') WHERE id = 'default'" +gbrain exec "UPDATE sources SET config = jsonb_set(config, '{priority}', '\"low\"') WHERE id = 'media-corpus'" +``` + +Recognized values: `high` (priority -10), `normal` (0), `low` (5). Unknown values fall back to `normal` with a once-per-source stderr warning. + +### 4. (Optional) Tune embed-backfill budget caps + +Default caps: $10/job, $25/source/24h. Override via DB config: + +```bash +gbrain config set embed.backfill_max_usd 20 # raise per-job cap +gbrain config set embed.backfill_max_usd_per_source_24h 100 # raise 24h cap +gbrain config set embed.backfill_cooldown_min 30 # raise cooldown window +``` + +### 5. (Optional) Opt out of parallel sync entirely + +If something breaks and you need v0.39 behavior immediately: + +```bash +gbrain config set sync.federated_v2 false +gbrain jobs supervisor restart +``` + +Per-source lock + migration v92 stay on regardless (those are correctness fixes, not features). + +## Verify the outcome + +After completing the steps above: + +1. **Parallel sync works:** `gbrain sync --all` should complete noticeably faster than before. If multiple sources used to take minutes-to-hours, you should see them run concurrently. +2. **Webhook fires:** `git push` to a webhook-configured source should produce a `[dispatch]` line in autopilot logs (or `gbrain jobs list` should show a fresh `sync` job within seconds). +3. **Dashboard is accurate:** `gbrain sources status` should match the numbers from `gbrain stats` per source. +4. **No regressions:** Existing `gbrain sync --source ` still works the same way; `gbrain sync` with no flags still syncs the default source. + +## If something goes wrong + +`gbrain doctor` is your first stop: + +- `federation_health: warn` — surfaces lag, embed coverage, or queue depth issues per source with paste-ready remediation commands +- `federation_health: fail` — surfaces critical lag (>24h) or embed coverage (<50% with >1000 chunks) + +For partial states or unexpected errors, file an issue at https://github.com/garrytan/gbrain/issues with: +- Output of `gbrain doctor --json` +- Output of `gbrain sources status --json` +- Contents of `~/.gbrain/upgrade-errors.jsonl` if present +- Which step broke diff --git a/src/commands/autopilot.ts b/src/commands/autopilot.ts index 599d5531b..4dbd38bf3 100644 --- a/src/commands/autopilot.ts +++ b/src/commands/autopilot.ts @@ -417,6 +417,14 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) { // autopilot-cycle can never run concurrently (both acquire // gbrain-cycle), so the "60-min floor double-processes queued // targeted jobs" failure mode is closed by the lock. + // + // v0.40 D17 layered on top: per-source freshness check fires BEFORE + // the score gate so a healthy brain that happens to have a stale + // federated source still picks up new commits. brain_score reflects + // internal data quality (embed coverage, link density, orphans), + // NOT whether GitHub has new commits on the source repo. Decoupling + // the two closes the silent-stale-source bug class on + // poll-only deployments. try { const { MinionQueue } = await import('../core/minions/queue.ts'); const { computeRecommendations } = await import('../core/brain-score-recommendations.ts'); @@ -425,6 +433,57 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) { const slot = new Date(slotMs).toISOString(); const timeoutMs = Math.max(baseInterval * 2 * 1000, 300_000); + // ── v0.40 D17: per-source freshness check ──────────────────── + // Runs first; independent of score gate. Submits a 'sync' job per + // source whose last_sync_at is older than the interval. The sync + // handler (T6/T7) auto-enqueues embed-backfill on completion if + // pages changed. + try { + const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); + if (await isFederatedV2Enabled(engine)) { + const { loadAllSources } = await import('../core/sources-load.ts'); + const sources = await loadAllSources(engine); + const intervalMs = baseInterval * 1000; + const now = Date.now(); + for (const src of sources) { + if (!src.local_path) continue; + const lastSyncMs = src.last_sync_at ? new Date(src.last_sync_at).getTime() : 0; + const ageMs = now - lastSyncMs; + if (ageMs < intervalMs) continue; // fresh enough + try { + const job = await queue.add( + 'sync', + { + sourceId: src.id, + repoPath: src.local_path, + auto_embed_backfill: true, + embed_reason: 'autopilot_freshness', + }, + { + queue: 'default', + idempotency_key: `autopilot-sync:${src.id}:${slot}`, + max_attempts: 2, + timeout_ms: timeoutMs, + maxWaiting: 1, + }, + ); + if (jsonMode) { + process.stderr.write(JSON.stringify({ + event: 'dispatched', job_id: job.id, mode: 'freshness', + source_id: src.id, age_ms: ageMs, + }) + '\n'); + } else { + console.log(`[dispatch] job #${job.id} sync (freshness: ${src.id}; age=${Math.floor(ageMs / 60000)}min)`); + } + } catch (e) { + logError('dispatch.freshness', e); + } + } + } + } catch (e) { + logError('dispatch.freshness-gate', e); + } + // Cheap path: engine.getHealth() is a single SQL count query. const health = await engine.getHealth(); const score = health.brain_score; diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index e05779588..2594935ed 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -625,6 +625,12 @@ export async function doctorReportRemote(engine: BrainEngine): Promise=95% + * (or chunks <100), AND failed_jobs_24h < 3 + * warn — any source has lag > 1h + federated, OR coverage < 95% with + * chunks > 100, OR failed_jobs_24h >= 3 + * fail — any source has lag > 24h, OR coverage < 50% with chunks > 1000 + * + * Single-source brain short-circuits to ok (no federation to check). + * Each warning carries a paste-ready remediation hint. + */ +export async function checkFederationHealth(engine: BrainEngine): Promise { + try { + const { loadAllSources } = await import('../core/sources-load.ts'); + const { computeAllSourceMetrics } = await import('../core/source-health.ts'); + const sources = await loadAllSources(engine, { includeArchived: false }); + if (sources.length <= 1) { + return { + name: 'federation_health', + status: 'ok', + message: 'Single-source brain (no federation to check)', + }; + } + const metrics = await computeAllSourceMetrics(engine, sources); + + const warns: string[] = []; + const fails: string[] = []; + for (const m of metrics) { + // Fail thresholds first (most severe) + if (m.lag_seconds !== null && m.lag_seconds > 24 * 3600) { + fails.push(`${m.source_id}: stale ${Math.floor(m.lag_seconds / 3600)}h — run \`gbrain sync trigger --source ${m.source_id}\``); + continue; + } + if (m.embed_coverage_pct < 50 && m.total_chunks > 1000) { + fails.push(`${m.source_id}: ${m.embed_coverage_pct.toFixed(1)}% embed coverage (${m.total_chunks.toLocaleString()} chunks) — run \`gbrain jobs submit embed-backfill --params '{"sourceId":"${m.source_id}"}'\``); + continue; + } + // Warns + if (m.federated && m.lag_seconds !== null && m.lag_seconds > 3600) { + warns.push(`${m.source_id}: federated source ${Math.floor(m.lag_seconds / 3600)}h+ stale — run \`gbrain sync trigger --source ${m.source_id}\``); + } + if (m.embed_coverage_pct < 95 && m.total_chunks > 100) { + warns.push(`${m.source_id}: ${m.embed_coverage_pct.toFixed(1)}% embed coverage — run \`gbrain jobs submit embed-backfill --params '{"sourceId":"${m.source_id}"}'\``); + } + if (m.failed_jobs_24h >= 3) { + warns.push(`${m.source_id}: ${m.failed_jobs_24h} failures in 24h — check \`gbrain jobs list --status failed\``); + } + } + + if (fails.length > 0) { + return { + name: 'federation_health', + status: 'fail', + message: `${fails.length} federation failure(s):\n ${fails.join('\n ')}`, + }; + } + if (warns.length > 0) { + return { + name: 'federation_health', + status: 'warn', + message: `${warns.length} federation warning(s):\n ${warns.join('\n ')}`, + }; + } + return { + name: 'federation_health', + status: 'ok', + message: `${metrics.length} source(s) healthy (parallel sync, async embed)`, + }; + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + return { name: 'federation_health', status: 'warn', message: `Check failed: ${msg}` }; + } +} + /** * v0.37.7.0 — Tier 5L oauth_confidential_client_health. * diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index d86a072d3..880ed0fb5 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -1067,7 +1067,43 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai repoPath, sourceId, noPull, noEmbed, noExtract, concurrency: concurrencyOverride, }); - return result; + + // v0.40 D22: auto_embed_backfill defaults TRUE when sourceId is set AND + // the feature flag is enabled. Submits a child embed-backfill job + // (fire-and-forget — D15.1) so stale chunks get embedded async without + // the sync handler waiting on the embed pipeline. + const autoEmbed = job.data.auto_embed_backfill !== false; + let embedJobId: number | null = null; + let embedSkipReason: string | null = null; + if (autoEmbed && sourceId && result.status !== 'up_to_date' && result.status !== 'dry_run') { + try { + const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); + if (await isFederatedV2Enabled(engine)) { + const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts'); + const submission = await submitEmbedBackfill(engine, sourceId, { + reason: typeof job.data.embed_reason === 'string' + ? (job.data.embed_reason as string) + : 'sync_handler', + }); + if (submission.status === 'submitted') { + embedJobId = submission.jobId ?? null; + } else { + embedSkipReason = submission.status; + } + } else { + embedSkipReason = 'feature_flag_disabled'; + } + } catch (err) { + // Embed-backfill submission failure must NOT fail the sync job. + embedSkipReason = `submit_error:${err instanceof Error ? err.message : String(err)}`; + } + } else if (!sourceId) { + embedSkipReason = 'no_source_id'; + } else if (!autoEmbed) { + embedSkipReason = 'auto_embed_disabled'; + } + + return { ...result, embed_job_id: embedJobId, embed_skip_reason: embedSkipReason }; }); worker.register('embed', async (job) => { @@ -1392,7 +1428,16 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai worker.register('resolve_symbol_edges', makePhaseHandler('resolve_symbol_edges')); worker.register('recompute_emotional_weight', makePhaseHandler('recompute_emotional_weight')); - process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected)\n'); + // v0.40 Federated Sync v2 — embed-backfill: per-source decoupled embed. + // Cost-bounded via D6 ($10/job BudgetTracker) + D19 (source-level cooldown + // + 24h rolling cap, gated at submit time). NOT in PROTECTED_JOB_NAMES — + // embedding-only spend, no API-by-the-minute risk like subagent. + worker.register('embed-backfill', async (job) => { + const { makeEmbedBackfillHandler } = await import('../core/minions/handlers/embed-backfill.ts'); + return await makeEmbedBackfillHandler(engine)(job); + }); + + process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected) + embed-backfill (v0.40)\n'); // Plugin discovery — one line per discovered plugin (mirrors the // openclaw-seam startup line convention from v0.11+). Loaded diff --git a/src/commands/serve-http.ts b/src/commands/serve-http.ts index 9a9040a82..7d2ee1e1e 100644 --- a/src/commands/serve-http.ts +++ b/src/commands/serve-http.ts @@ -15,7 +15,8 @@ import type { Request, Response, NextFunction } from 'express'; import cookieParser from 'cookie-parser'; import cors from 'cors'; import rateLimit from 'express-rate-limit'; -import { randomBytes, createHash, timingSafeEqual } from 'crypto'; +import { randomBytes, createHash } from 'crypto'; +import { safeHexEqual } from '../core/timing-safe.ts'; import { Server } from '@modelcontextprotocol/sdk/server/index.js'; import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js'; import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js'; @@ -592,17 +593,9 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption // --------------------------------------------------------------------------- // Admin authentication (cookie-based) // --------------------------------------------------------------------------- + // v0.40 D15.5: safeHexEqual extracted to src/core/timing-safe.ts so the new + // /webhooks/github HMAC verifier reuses the same constant-time compare. // POST /admin/login — JSON body with token (for programmatic/UI login) - // Constant-time hex compare. Both inputs are sha256 hex (64 chars), - // so they're always equal length. timingSafeEqual throws on length - // mismatch — we already short-circuit on non-string above. Catches - // would-be timing oracles even though the inputs are pre-hashed - // (defense-in-depth on the hash bits). - function safeHexEqual(a: string, b: string): boolean { - if (a.length !== b.length) return false; - return timingSafeEqual(Buffer.from(a, 'hex'), Buffer.from(b, 'hex')); - } - app.post('/admin/login', express.json(), (req, res) => { const token = req.body?.token; if (!token || typeof token !== 'string') { @@ -1787,6 +1780,158 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption }, ); + // --------------------------------------------------------------------------- + // POST /webhooks/github — push-triggered sync (v0.40 Federated Sync v2) + // --------------------------------------------------------------------------- + // Anonymous endpoint by necessity (GitHub doesn't carry an OAuth token). + // Auth is via per-source HMAC-SHA256 in the X-Hub-Signature-256 header. + // + // D3: 60 req/min/IP rate limit + pre-DB short-circuit on missing + // signature, so probe traffic doesn't even touch the source-lookup + // query. + // D5: event=push AND ref-match against sources.config.tracked_branch. + // Other event types (ping, pull_request, etc.) return 202 'ignored' + // so GitHub doesn't retry. + // D15.5: HMAC compare uses the shared safeHexEqual helper. + // D18: submits 'sync' job with auto_embed_backfill=true and priority -10 + // (above autopilot's 0). + // --------------------------------------------------------------------------- + const githubWebhookLimiter = rateLimit({ + windowMs: 60_000, + limit: 60, + standardHeaders: 'draft-7', + legacyHeaders: false, + message: { error: 'rate_limit_exceeded', message: 'too many GitHub webhook requests' }, + }); + + app.post( + '/webhooks/github', + githubWebhookLimiter, + express.raw({ type: '*/*', limit: '1mb' }), + async (req: Request, res: Response) => { + // D3 pre-DB short-circuit: missing signature → 401 without any + // source lookup. Bot probe traffic ends here. + const sigHeader = req.header('X-Hub-Signature-256'); + if (!sigHeader) { + res.status(401).json({ error: 'missing_signature', message: 'X-Hub-Signature-256 header is required' }); + return; + } + + // D5: filter by event header. GitHub fires webhooks for every event + // type. Anything other than 'push' is acknowledged with 202 + reason + // so GitHub doesn't retry — but no source lookup or job submission. + const event = req.header('X-GitHub-Event') ?? ''; + if (event !== 'push') { + res.status(202).json({ status: 'ignored', reason: `event=${event || '(missing)'}` }); + return; + } + + const payload = Buffer.isBuffer(req.body) ? req.body : Buffer.from(JSON.stringify(req.body), 'utf8'); + if (payload.length === 0) { + res.status(400).json({ error: 'empty_body' }); + return; + } + + let parsed: { repository?: { full_name?: string }; ref?: string }; + try { + parsed = JSON.parse(payload.toString('utf8')); + } catch { + res.status(400).json({ error: 'malformed_json' }); + return; + } + + const fullName = parsed.repository?.full_name; + const ref = parsed.ref; + if (!fullName || !ref) { + res.status(400).json({ error: 'missing_fields', message: 'repository.full_name and ref are required' }); + return; + } + + // Source lookup via the v87 partial expression index on + // config->>'github_repo'. fast even on large brains. + let source: { id: string; config: Record | string } | null = null; + try { + const rows = await engine.executeRaw<{ id: string; config: Record | string }>( + `SELECT id, config FROM sources WHERE config->>'github_repo' = $1 LIMIT 1`, + [fullName], + ); + source = rows[0] ?? null; + } catch (err) { + console.error('webhook: source lookup error:', err); + res.status(500).json({ error: 'lookup_failed' }); + return; + } + if (!source) { + res.status(404).json({ error: 'unknown_repo', repo: fullName }); + return; + } + + const cfg = (typeof source.config === 'string' ? JSON.parse(source.config) : source.config) as { + webhook_secret?: string; + tracked_branch?: string; + }; + + // D5: ref must match the configured tracked branch (default 'main'). + const trackedBranch = cfg.tracked_branch ?? 'main'; + const expectedRef = `refs/heads/${trackedBranch}`; + if (ref !== expectedRef) { + res.status(202).json({ + status: 'ignored', + reason: `ref_mismatch`, + received_ref: ref, + tracked_branch: trackedBranch, + }); + return; + } + + const secret = cfg.webhook_secret; + if (!secret || typeof secret !== 'string') { + res.status(401).json({ error: 'webhook_not_configured', message: 'Run: gbrain sources webhook set ' + source.id }); + return; + } + + // HMAC verify. GitHub sends "sha256=" — strip the prefix BEFORE + // safeHexEqual because Buffer.from('sha256=...', 'hex') silently + // truncates at the first non-hex char (the 's'), leaving both + // operands as 0-byte buffers and making every signature "match". + // Pinned by test/sources-webhook.test.ts tamper assertions. + const { createHmac } = await import('node:crypto'); + const computedHex = createHmac('sha256', secret).update(payload).digest('hex'); + const prefix = 'sha256='; + if (!sigHeader.startsWith(prefix)) { + res.status(401).json({ error: 'signature_mismatch', message: 'expected sha256= prefix' }); + return; + } + if (!safeHexEqual(sigHeader.slice(prefix.length), computedHex)) { + res.status(401).json({ error: 'signature_mismatch' }); + return; + } + + // Submit sync job with priority -10 (above autopilot's 0). + try { + const queue = new MinionQueue(engine); + const job = await queue.add( + 'sync', + { + sourceId: source.id, + auto_embed_backfill: true, + embed_reason: 'webhook', + }, + { + priority: -10, + idempotency_key: `webhook:sync:${source.id}:${Math.floor(Date.now() / 30_000)}`, + maxWaiting: 1, + }, + ); + res.status(202).json({ job_id: job.id, source_id: source.id }); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + console.error('webhook: queue submission error:', msg); + res.status(500).json({ error: 'queue_submission_failed', message: msg }); + } + }, + ); + // --------------------------------------------------------------------------- // Start server // --------------------------------------------------------------------------- diff --git a/src/commands/sources.ts b/src/commands/sources.ts index bc13b4981..d40c2876f 100644 --- a/src/commands/sources.ts +++ b/src/commands/sources.ts @@ -47,6 +47,12 @@ import { resolveSourceWithTier, SOURCE_TIER_NAMES, } from '../core/source-resolver.ts'; +import { + loadAllSources, + parseSourceConfig, + isSourceFederated, + type SourceRow as LoadedSourceRow, +} from '../core/sources-load.ts'; // ── Validation ────────────────────────────────────────────── @@ -84,18 +90,10 @@ interface SourceListEntry { // ── Helpers ───────────────────────────────────────────────── -function parseConfig(config: unknown): Record { - if (typeof config === 'string') { - try { return JSON.parse(config) as Record; } catch { return {}; } - } - if (typeof config === 'object' && config !== null) return config as Record; - return {}; -} - -function isFederated(config: unknown): boolean { - const parsed = parseConfig(config); - return parsed.federated === true; -} +// v0.40 (D7): shared helpers — re-exported as local names for back-compat +// with existing call sites that import `parseConfig`/`isFederated` by intent. +const parseConfig = parseSourceConfig; +const isFederated = isSourceFederated; async function fetchSource(engine: BrainEngine, id: string): Promise { const rows = await engine.executeRaw( @@ -184,10 +182,10 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise { async function runList(engine: BrainEngine, args: string[]): Promise { const json = args.includes('--json'); - const rows = await engine.executeRaw( - `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at - FROM sources ORDER BY (id = 'default') DESC, id`, - ); + // v0.40 (D7): loadAllSources is the single source of truth for source enum. + // Pass includeArchived=true to preserve the legacy `runList` behavior of + // surfacing archived rows (they get the ⚠ marker below). + const rows: LoadedSourceRow[] = await loadAllSources(engine, { includeArchived: true }); const entries: SourceListEntry[] = []; for (const r of rows) { @@ -547,6 +545,290 @@ async function runFederate(engine: BrainEngine, args: string[], value: boolean): [JSON.stringify(config), id], ); console.log(`Source "${id}" is now ${value ? 'federated (appears in cross-source default search)' : 'isolated (only searched when explicitly named)'}.`); + + // v0.40 D19: auto-submit embed-backfill when coverage < 100%. Federation + // flip is a moment when the user explicitly opted into seeing this source + // in default search; un-embedded chunks would hide content from the very + // moment they wanted visibility. Best-effort — submission failure does NOT + // fail the flip. + try { + const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); + if (!(await isFederatedV2Enabled(engine))) return; + + const { loadAllSources } = await import('../core/sources-load.ts'); + const { computeAllSourceMetrics } = await import('../core/source-health.ts'); + const sources = await loadAllSources(engine, { includeArchived: false }); + const metrics = await computeAllSourceMetrics(engine, sources); + const m = metrics.find((x) => x.source_id === id); + if (!m || m.total_chunks === 0 || m.embed_coverage_pct >= 100) return; + + const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts'); + const sub = await submitEmbedBackfill(engine, id, { reason: 'federation_flip' }); + if (sub.status === 'submitted') { + const missing = m.total_chunks - m.embedded_chunks; + console.log(` → embed-backfill job ${sub.jobId} queued for missing ${missing} chunks.`); + } else if (sub.status === 'cooldown') { + console.log(` → embed-backfill skipped (cooldown). Manually trigger with: gbrain jobs submit embed-backfill --params '{"sourceId":"${id}"}'`); + } else if (sub.status === 'spend_capped') { + console.log(` → embed-backfill skipped (24h spend cap $${sub.spendCapUsd} reached for this source).`); + } + } catch (err) { + // Federation flip already succeeded; embed-backfill is a follow-up nicety. + console.error(` → embed-backfill submission failed (flip succeeded): ${err instanceof Error ? err.message : String(err)}`); + } +} + +// ── v0.40 sources status (D12) ────────────────────────────── +async function runStatus(engine: BrainEngine, args: string[]): Promise { + const json = args.includes('--json'); + const { loadAllSources } = await import('../core/sources-load.ts'); + const { computeAllSourceMetrics } = await import('../core/source-health.ts'); + const sources = await loadAllSources(engine, { includeArchived: false }); + if (sources.length === 0) { + if (json) { + console.log(JSON.stringify({ schema_version: 1, sources: [] }, null, 2)); + } else { + console.log('No sources registered. Use `gbrain sources add --path ` first.'); + } + return; + } + const metrics = await computeAllSourceMetrics(engine, sources); + + if (json) { + console.log(JSON.stringify({ schema_version: 1, sources: metrics }, null, 2)); + return; + } + + // Human-readable table: SOURCE | LAG | EMBED | FAILS | QUEUE | PAGES | LAST SYNC + console.log('SOURCES — health'); + console.log('────────────────'); + console.log( + ` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`, + ); + for (const m of metrics) { + const lag = m.lag_seconds === null + ? 'never' + : formatLag(m.lag_seconds); + const embed = `${m.embed_coverage_pct.toFixed(0)}%`; + const fails = String(m.failed_jobs_24h); + const queue = String(m.queue_depth); + const pages = m.total_pages.toLocaleString(); + const sync = m.last_sync_at ? new Date(m.last_sync_at).toISOString().slice(0, 19).replace('T', ' ') : 'never'; + console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`); + } + console.log(''); + for (const m of metrics) { + const warns: string[] = []; + if (!m.local_path) warns.push('no local_path'); + if (m.lag_seconds === null) warns.push(`never synced — run \`gbrain sync --source ${m.source_id}\``); + if (m.embed_coverage_pct < 95 && m.total_chunks > 100) { + warns.push(`${(100 - m.embed_coverage_pct).toFixed(1)}% un-embedded — run \`gbrain embed --stale --source ${m.source_id}\``); + } + if (m.failed_jobs_24h >= 3) { + warns.push(`${m.failed_jobs_24h} failures in 24h — check \`gbrain jobs list --status failed\``); + } + if (warns.length > 0) { + console.log(` ⚠ ${m.source_id}: ${warns.join('; ')}`); + } + } +} + +function formatLag(seconds: number): string { + if (seconds < 60) return `${seconds}s`; + if (seconds < 3600) return `${Math.floor(seconds / 60)}m`; + if (seconds < 86400) return `${Math.floor(seconds / 3600)}h`; + return `${Math.floor(seconds / 86400)}d`; +} + +// ── v0.40 sources webhook (D8) ────────────────────────────── +async function runWebhook(engine: BrainEngine, args: string[]): Promise { + const sub = args[0]; + const rest = args.slice(1); + switch (sub) { + case 'set': return runWebhookSet(engine, rest); + case 'show': return runWebhookShow(engine, rest); + case 'rotate': return runWebhookRotate(engine, rest); + case 'clear': return runWebhookClear(engine, rest); + case undefined: + case '--help': + case '-h': + console.log(`Usage: gbrain sources webhook [options] + +Subcommands: + set [--secret VAL] [--github-repo owner/name] One-time reveal + show Metadata only + rotate New secret, reveal + clear Remove webhook config`); + return; + default: + console.error(`Unknown webhook subcommand: ${sub}`); + process.exit(2); + } +} + +async function runWebhookSet(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + if (!id) { + console.error('Usage: gbrain sources webhook set [--secret VAL] [--github-repo owner/name]'); + process.exit(2); + } + const src = await fetchSource(engine, id); + if (!src) { + console.error(`Source "${id}" not found.`); + process.exit(1); + } + const explicitSecret = args.find((a, i) => args[i - 1] === '--secret'); + const githubRepo = args.find((a, i) => args[i - 1] === '--github-repo'); + if (!githubRepo) { + console.error('--github-repo owner/name is required (e.g. "Garry-s-List/zion-brain")'); + process.exit(2); + } + if (!/^[\w.-]+\/[\w.-]+$/.test(githubRepo)) { + console.error(`Invalid --github-repo format: "${githubRepo}". Expected "owner/name".`); + process.exit(2); + } + + const { randomBytes } = await import('node:crypto'); + const secret = explicitSecret ?? randomBytes(32).toString('hex'); + const cfg = parseConfig(src.config); + cfg.webhook_secret = secret; + cfg.github_repo = githubRepo; + await engine.executeRaw( + `UPDATE sources SET config = $1::jsonb WHERE id = $2`, + [JSON.stringify(cfg), id], + ); + + console.log(`Webhook configured for source "${id}":`); + console.log(` github_repo: ${githubRepo}`); + console.log(` webhook_secret: ${secret}`); + console.log(''); + console.log('--- Paste this into GitHub repo settings → Webhooks → Add webhook ---'); + console.log(' Payload URL: /webhooks/github'); + console.log(' Content type: application/json'); + console.log(` Secret: ${secret}`); + console.log(' Events: Just the push event'); + console.log(' Active: checked'); + console.log(''); + console.log('⚠ This secret is shown ONCE. Save it now; subsequent `gbrain sources webhook show` will NOT display it.'); +} + +async function runWebhookShow(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + if (!id) { + console.error('Usage: gbrain sources webhook show '); + process.exit(2); + } + const src = await fetchSource(engine, id); + if (!src) { + console.error(`Source "${id}" not found.`); + process.exit(1); + } + const cfg = parseConfig(src.config); + const githubRepo = typeof cfg.github_repo === 'string' ? cfg.github_repo : '(not set)'; + const secretSet = typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0; + const trackedBranch = typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : '(auto-detected on next sync, default main)'; + + console.log(`Webhook configuration for source "${id}":`); + console.log(` github_repo: ${githubRepo}`); + console.log(` webhook_secret: ${secretSet ? '' : '(not set)'}`); + console.log(` tracked_branch: ${trackedBranch}`); +} + +async function runWebhookRotate(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + if (!id) { + console.error('Usage: gbrain sources webhook rotate '); + process.exit(2); + } + const src = await fetchSource(engine, id); + if (!src) { + console.error(`Source "${id}" not found.`); + process.exit(1); + } + const { randomBytes } = await import('node:crypto'); + const secret = randomBytes(32).toString('hex'); + const cfg = parseConfig(src.config); + cfg.webhook_secret = secret; + await engine.executeRaw( + `UPDATE sources SET config = $1::jsonb WHERE id = $2`, + [JSON.stringify(cfg), id], + ); + console.log(`New webhook secret for source "${id}":`); + console.log(` ${secret}`); + console.log(''); + console.log('⚠ Update the GitHub webhook config to use this new secret. The old one is invalidated immediately.'); +} + +async function runWebhookClear(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + if (!id) { + console.error('Usage: gbrain sources webhook clear '); + process.exit(2); + } + const src = await fetchSource(engine, id); + if (!src) { + console.error(`Source "${id}" not found.`); + process.exit(1); + } + const cfg = parseConfig(src.config); + delete cfg.webhook_secret; + delete cfg.github_repo; + await engine.executeRaw( + `UPDATE sources SET config = $1::jsonb WHERE id = $2`, + [JSON.stringify(cfg), id], + ); + console.log(`Webhook configuration cleared for source "${id}".`); +} + +// ── v0.40 sources tracked-branch (D20) ────────────────────── +async function runTrackedBranch(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + if (!id) { + console.error('Usage: gbrain sources tracked-branch [--set ] [--detect]'); + process.exit(2); + } + const src = await fetchSource(engine, id); + if (!src) { + console.error(`Source "${id}" not found.`); + process.exit(1); + } + const setArg = args.find((a, i) => args[i - 1] === '--set'); + const detect = args.includes('--detect'); + const cfg = parseConfig(src.config); + + if (setArg) { + cfg.tracked_branch = setArg; + await engine.executeRaw( + `UPDATE sources SET config = $1::jsonb WHERE id = $2`, + [JSON.stringify(cfg), id], + ); + console.log(`Tracked branch for source "${id}" set to "${setArg}".`); + return; + } + if (detect) { + if (!src.local_path) { + console.error(`Source "${id}" has no local_path; cannot auto-detect branch.`); + process.exit(1); + } + try { + const { execFileSync } = await import('node:child_process'); + const branch = execFileSync('git', ['-C', src.local_path, 'rev-parse', '--abbrev-ref', 'HEAD'], { encoding: 'utf8' }).trim(); + cfg.tracked_branch = branch; + await engine.executeRaw( + `UPDATE sources SET config = $1::jsonb WHERE id = $2`, + [JSON.stringify(cfg), id], + ); + console.log(`Detected branch "${branch}" for source "${id}"; persisted to config.tracked_branch.`); + } catch (e) { + console.error(`git rev-parse failed: ${e instanceof Error ? e.message : String(e)}`); + process.exit(1); + } + return; + } + + // Read mode: just print + const tracked = typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : '(unset — defaults to main)'; + console.log(`Source "${id}" tracked_branch: ${tracked}`); } // ── `sources current` (v0.37.7.0) ────────────────────────── @@ -615,6 +897,11 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise [--priority high|normal|low]` + * + * Push-trigger entry point. Wraps `queue.add('sync', ...)` with priority -10 + * (above autopilot's 0) so push-triggered syncs preempt scheduled ones. + * Use cases: GitHub webhook handler (POST /webhooks/github), CLI nudge after + * a manual git pull, scripted dispatch from `gbrain sources federate`. + * + * Sets `auto_embed_backfill: true` so the extended sync handler (T6/T7) + * auto-enqueues an embed-backfill job after the sync settles. + * + * Output: prints `job_id=N` to stdout for shell composition. Errors exit 1. + */ +export async function runSyncTrigger(engine: BrainEngine, args: string[]): Promise { + if (args.includes('--help') || args.includes('-h')) { + console.log(`Usage: gbrain sync trigger --source [--priority high|normal|low] + +Queue a push-triggered sync job for one source. Prints the resulting job id +on stdout. The autopilot worker picks it up and runs performSync against the +named source; if the sync added/modified pages, an embed-backfill job is +auto-enqueued (subject to D6 budget cap + D19 source-level cooldown). + +Use cases: + - GitHub webhook → 'gbrain sync trigger --source ' + - Manual nudge after 'git pull' inside a federated source + - Programmatic triggers from CI / shell automation + +See also: + gbrain sources webhook set Set up GitHub-signed push webhook + gbrain sources status Per-source sync + embed coverage +`); + return; + } + + const sourceIdArg = args.find((a, i) => args[i - 1] === '--source') ?? null; + if (!sourceIdArg) { + console.error('Error: --source is required'); + console.error("Usage: gbrain sync trigger --source [--priority high|normal|low]"); + process.exit(2); + } + + const priorityArg = args.find((a, i) => args[i - 1] === '--priority') ?? 'high'; + const priorityMap: Record = { high: -10, normal: 0, low: 5 }; + const priority = priorityMap[priorityArg]; + if (priority === undefined) { + console.error(`Invalid --priority value: "${priorityArg}". Must be high|normal|low.`); + process.exit(2); + } + + // Verify source exists before submitting + const { fetchSource } = await import('../core/sources-load.ts'); + const source = await fetchSource(engine, sourceIdArg); + if (!source) { + console.error(`Source "${sourceIdArg}" not found. List with: gbrain sources list`); + process.exit(1); + } + + const { MinionQueue } = await import('../core/minions/queue.ts'); + const queue = new MinionQueue(engine); + const job = await queue.add( + 'sync', + { + sourceId: sourceIdArg, + repoPath: source.local_path, + auto_embed_backfill: true, + embed_reason: 'sync_trigger', + }, + { + priority, + idempotency_key: `sync-trigger:${sourceIdArg}:${Math.floor(Date.now() / 30_000)}`, + maxWaiting: 1, + }, + ); + + console.log(`job_id=${job.id}`); +} + export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise { // CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two // concurrent syncs can otherwise read the same last_commit anchor, both @@ -372,10 +449,13 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise< // mechanism (none in v0.22.13; reserved for future). let lockHandle: { release: () => Promise } | null = null; if (!opts.skipLock) { - lockHandle = await tryAcquireDbLock(engine, SYNC_LOCK_ID); + // v0.40: per-source lock so cross-source sync runs in parallel. Two sources + // (e.g. default + zion-brain) take distinct lock rows and don't serialize. + const lockKey = syncLockId(opts.sourceId ?? 'default'); + lockHandle = await tryAcquireDbLock(engine, lockKey); if (!lockHandle) { throw new Error( - `Another sync is in progress (lock ${SYNC_LOCK_ID} held). ` + + `Another sync is in progress (lock ${lockKey} held). ` + `Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`, ); } @@ -979,7 +1059,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise args[i - 1] === '--max-sources'); + const maxSources = maxSourcesStr ? parseInt(maxSourcesStr, 10) : undefined; + if (maxSourcesStr && (!Number.isFinite(maxSources!) || maxSources! < 1)) { + console.error(`Invalid --max-sources value: "${maxSourcesStr}". Must be a positive integer.`); + process.exit(1); + } const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined; const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers'); // v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead @@ -1350,32 +1455,95 @@ See also: } } - for (const src of sources) { - const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' }; - if (cfg.syncEnabled === false) { - console.log(`Skipping disabled source: ${src.name}`); - continue; - } - console.log(`\n--- Syncing source: ${src.name} ---`); + // v0.40 D4+D18: parallel fan-out is the default; --serial opts out, and + // PGLite forces serial (single-writer constraint). Each source runs in + // its own performSync — per-source locks (D1) keep them isolated. After + // each completion, auto-submit an embed-backfill job (D18) so vector + // coverage catches up async instead of leaving search degraded. + const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); + const v2Enabled = await isFederatedV2Enabled(engine); + const activeSources = sources.filter((s) => { + const cfg = (s.config || {}) as { syncEnabled?: boolean }; + return cfg.syncEnabled !== false; + }); + const disabledCount = sources.length - activeSources.length; + if (disabledCount > 0) { + console.log(`Skipping ${disabledCount} disabled source(s).`); + } + + const runOne = async (src: typeof sources[number]): Promise => { + const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' }; + // D18: parallel path defers embed; auto-enqueue embed-backfill after. + const effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed; const repoOpts: SyncOpts = { repoPath: src.local_path!, - dryRun, full, noPull, noEmbed, skipFailed, retryFailed, + dryRun, full, noPull, + noEmbed: effectiveNoEmbed, + skipFailed, retryFailed, sourceId: src.id, strategy: cfg.strategy, concurrency, }; - try { - const result = await performSync(engine, repoOpts); - printSyncResult(result); - // Codex P2: --all loop must also manage .gitignore per-source. Without - // this, multi-source users who rely on `gbrain sync --all` never get - // the advertised db_only ignore rules unless they sync each repo - // individually. - if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') { - manageGitignore(src.local_path!, engine.kind); + const result = await performSync(engine, repoOpts); + if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') { + manageGitignore(src.local_path!, engine.kind); + } + // D18: auto-enqueue embed-backfill per source (unless opted out) + if ( + v2Enabled && + !noAutoEmbed && + !dryRun && + result.status !== 'dry_run' && + result.status !== 'up_to_date' + ) { + try { + const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts'); + const sub = await submitEmbedBackfill(engine, src.id, { reason: 'sync_all' }); + if (sub.status === 'submitted') { + console.log(` → embed-backfill job ${sub.jobId} queued for ${src.name}`); + } else if (sub.status === 'cooldown') { + console.log(` → embed-backfill skipped (cooldown) for ${src.name}`); + } else if (sub.status === 'spend_capped') { + console.log(` → embed-backfill skipped (24h spend cap $${sub.spendCapUsd}) for ${src.name}`); + } + } catch (e) { + console.error(` → embed-backfill submission failed for ${src.name}: ${e instanceof Error ? e.message : String(e)}`); + } + } + return result; + }; + + const parallelEligible = + v2Enabled && !serialFlag && engine.kind !== 'pglite' && activeSources.length > 1; + + if (parallelEligible) { + const { pMapAllSettled } = await import('../core/parallel.ts'); + const cap = Math.min(activeSources.length, maxSources ?? 8); + console.log(`\nParallel sync: ${activeSources.length} sources, ${cap} concurrent workers.\n`); + const results = await pMapAllSettled(activeSources, cap, async (src) => { + const r = await runOne(src); + return { name: src.name, result: r }; + }); + // Print per-source aggregate at the end. + console.log('\n--- sync --all aggregate ---'); + for (let i = 0; i < results.length; i++) { + const r = results[i]; + const name = activeSources[i].name; + if (r.status === 'fulfilled') { + console.log(` ✓ ${name}: ${r.value.result.status} (added=${r.value.result.added}, modified=${r.value.result.modified}, deleted=${r.value.result.deleted})`); + } else { + console.error(` ✗ ${name}: ${r.reason instanceof Error ? r.reason.message : String(r.reason)}`); + } + } + } else { + for (const src of activeSources) { + console.log(`\n--- Syncing source: ${src.name} ---`); + try { + const result = await runOne(src); + printSyncResult(result); + } catch (e: unknown) { + console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`); } - } catch (e: unknown) { - console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`); } } return; diff --git a/src/core/cycle/phantom-redirect.ts b/src/core/cycle/phantom-redirect.ts index 5fd1e185e..48259ead3 100644 --- a/src/core/cycle/phantom-redirect.ts +++ b/src/core/cycle/phantom-redirect.ts @@ -53,7 +53,7 @@ import { type ParsedFact, } from '../facts-fence.ts'; import { parseMarkdown, splitBody, serializeMarkdown } from '../markdown.ts'; -import { tryAcquireDbLock, SYNC_LOCK_ID, type DbLockHandle } from '../db-lock.ts'; +import { tryAcquireDbLock, syncLockId, type DbLockHandle } from '../db-lock.ts'; import { logPhantomEvent, type PhantomOutcome } from '../facts/phantom-audit.ts'; /** Tagged-union outcome of a single phantom-redirect attempt. */ @@ -529,7 +529,9 @@ export async function runPhantomRedirectPass( // Bounded-retry lock acquisition. tryAcquireDbLock returns null on // contention; we loop with 1s backoff up to 30s total. - const lock = await acquireLockWithRetry(engine, SYNC_LOCK_ID); + // v0.40 D16: per-source lock matching performSync's posture. Phantom + same- + // source sync still serialize; cross-source parallel sync proceeds unblocked. + const lock = await acquireLockWithRetry(engine, syncLockId(sourceId)); if (!lock) { logPhantomEvent({ outcome: 'pass_skipped_lock_busy', source_id: sourceId }); result.lock_busy = true; diff --git a/src/core/db-lock.ts b/src/core/db-lock.ts index 1704ed046..08706255c 100644 --- a/src/core/db-lock.ts +++ b/src/core/db-lock.ts @@ -134,10 +134,36 @@ export async function tryAcquireDbLock( throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`); } -/** Lock id for performSync's writer window. Distinct from gbrain-cycle so the - * cycle handler can hold gbrain-cycle while performSync (called from inside - * the cycle) acquires gbrain-sync. */ -export const SYNC_LOCK_ID = 'gbrain-sync'; +/** + * v0.40 (Federated Sync v2): per-source sync lock helper. + * + * Before v0.40: SYNC_LOCK_ID was a bare 'gbrain-sync' constant, taken by + * performSync's writer window. That meant only ONE sync could run at a time + * across the whole brain — even when two sources are completely independent + * (different git repos, different last_commit, different DB row anchors). + * + * v0.40 namespaces the lock key by sourceId so cross-source sync runs in + * parallel. The cycle's broader `gbrain-cycle` lock still serializes inside + * a single cycle invocation. Two-source layered semantics: + * + * cycle acquires `gbrain-cycle` + * → performSync(A) acquires `gbrain-sync:A` + * → performSync(B) acquires `gbrain-sync:B` (in a different process, fine) + * + * Audit: `SYNC_LOCK_ID` (back-compat alias) resolves to `syncLockId('default')`. + * Every consumer in src/ MUST namespace by source. Tracked consumers: + * - src/commands/sync.ts:performSync (per-source) + * - src/core/cycle/phantom-redirect.ts (per-source, D16) + */ +export function syncLockId(sourceId: string): string { + return `gbrain-sync:${sourceId}`; +} + +/** + * Back-compat alias. Resolves to `syncLockId('default')`. New code should call + * `syncLockId(sourceId)` directly. + */ +export const SYNC_LOCK_ID = syncLockId('default'); /** * v0.30.1 (T4 + A4): wrap long-running work in a refreshing TTL lock. diff --git a/src/core/embed-backfill-submit.ts b/src/core/embed-backfill-submit.ts new file mode 100644 index 000000000..940cc941f --- /dev/null +++ b/src/core/embed-backfill-submit.ts @@ -0,0 +1,203 @@ +/** + * Single submission entry point for the `embed-backfill` minion job + * (v0.40 Federated Sync v2 — D19). + * + * Every caller routes through `submitEmbedBackfill`: + * - Parallel sync --all completion (D18) + * - Extended `sync` handler (D22, auto_embed_backfill) + * - POST /webhooks/github + * - `sources federate` / `unfederate` flip hook + * - `gbrain sync trigger` + * - autopilot per-source dispatch (when source is stale and degraded) + * + * Why centralize: D2 added a per-source DB lock at handler entry. That + * protects against double-RUN but not double-SUBMIT — a webhook storm could + * still queue 20 embed-backfill jobs, each one waking, acquiring the lock, + * finding it busy, completing as `already_in_progress`. Net effect: zero + * wasted Voyage spend (D6 budget cap), but high queue churn. Worse, even if + * the lock held, a 30-second idempotency bucket only coalesces SUBMITS within + * the same window. Multi-hour push activity racks up unbounded calls. + * + * D19 layered defenses (composed here): + * 1. Per-source cooldown (default 10min). Refuses submission if the most + * recent embed-backfill for this source finished or is still active + * inside the window. + * 2. Per-source 24h rolling spend cap (default $25). Computed from the + * embed-backfill-tagged rows in the budget audit JSONL. Refuses + * submission when spend has hit the cap. + * + * Both bounds are config-overridable: + * - `embed.backfill_cooldown_min` (default 10) + * - `embed.backfill_max_usd_per_source_24h` (default 25) + * + * Returns a tagged-union status so callers can render the right user signal + * (`gbrain sources status`, webhook response body, sync completion banner). + */ +import type { BrainEngine } from './engine.ts'; +import { MinionQueue } from './minions/queue.ts'; + +export const COOLDOWN_CONFIG_KEY = 'embed.backfill_cooldown_min'; +export const SPEND_CAP_CONFIG_KEY = 'embed.backfill_max_usd_per_source_24h'; + +const DEFAULT_COOLDOWN_MIN = 10; +const DEFAULT_SPEND_CAP_USD = 25; + +export type SubmitEmbedBackfillStatus = + | 'submitted' + | 'cooldown' + | 'spend_capped'; + +export interface SubmitEmbedBackfillResult { + status: SubmitEmbedBackfillStatus; + /** Set when status === 'submitted'. */ + jobId?: number; + /** Set when status === 'cooldown'. Seconds remaining until cooldown lifts. */ + cooldownRemainingSeconds?: number; + /** Set when status === 'spend_capped'. Dollars spent in the 24h window. */ + spend24hUsd?: number; + /** Set when status === 'spend_capped'. Active cap. */ + spendCapUsd?: number; +} + +export interface SubmitEmbedBackfillOpts { + /** Logged into the job's data row for audit. */ + reason: string; + /** Override cooldown lookup (tests). */ + cooldownMinOverride?: number; + /** Override spend-cap lookup (tests). */ + spendCapUsdOverride?: number; + /** Override the 24h spend aggregator (tests). Returns spend in USD. */ + spend24hFn?: (engine: BrainEngine, sourceId: string) => Promise; + /** Override `Date.now` (tests). */ + nowMs?: number; + /** Job priority. Default 5 (lower than autopilot's 0; above default jobs). */ + priority?: number; +} + +/** + * Default 24h-spend aggregator. Reads completed embed-backfill rows in + * `minion_jobs` and sums their token usage × known-price proxy. We + * deliberately do NOT read the budget-tracker audit JSONL — the file + * lives on the worker's local disk and may not be present on the + * machine submitting the job (e.g. a remote MCP serve-http process). + * + * The minion_jobs.tokens_input/output columns ARE populated by the + * subagent flow but NOT by the embed flow (gateway.embed doesn't go + * through the chat-completion roll-up). For v0.40 we use a job-COUNT + * proxy capped at the daily limit: 1 job ≈ default-cap-share. Accepts + * imprecision in exchange for cross-process visibility. + * + * A precise rolling-spend tracker is filed as a v0.41 TODO. + */ +async function defaultSpend24hForSource( + engine: BrainEngine, + sourceId: string, +): Promise { + // Conservative proxy: count jobs that completed (or are running) in the + // 24h window. Each is treated as worth `DEFAULT_SPEND_CAP_USD / 25` ($1) + // toward the cap — i.e. 25 jobs in 24h saturate the default cap. + const rows = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n + FROM minion_jobs + WHERE name = 'embed-backfill' + AND data->>'sourceId' = $1 + AND status IN ('active', 'completed') + AND created_at > NOW() - INTERVAL '24 hours'`, + [sourceId], + ); + const jobCount = rows[0]?.n ?? 0; + return jobCount * 1; // $1 / job placeholder; configurable in a later wave. +} + +/** + * Look up an integer-valued config key with sane defaults. + * Returns `def` on missing / NaN / non-positive. + */ +async function readIntConfig( + engine: BrainEngine, + key: string, + def: number, +): Promise { + const raw = await engine.getConfig(key); + if (raw === null || raw === undefined) return def; + const n = Number(raw); + return Number.isFinite(n) && n > 0 ? n : def; +} + +export async function submitEmbedBackfill( + engine: BrainEngine, + sourceId: string, + opts: SubmitEmbedBackfillOpts, +): Promise { + const now = opts.nowMs ?? Date.now(); + const cooldownMin = + opts.cooldownMinOverride ?? + (await readIntConfig(engine, COOLDOWN_CONFIG_KEY, DEFAULT_COOLDOWN_MIN)); + const spendCap = + opts.spendCapUsdOverride ?? + (await readIntConfig(engine, SPEND_CAP_CONFIG_KEY, DEFAULT_SPEND_CAP_USD)); + + // ── Source-level cooldown ───────────────────────────────────── + // Block re-submission if (a) an embed-backfill is currently active for this + // source, OR (b) the most-recent embed-backfill finished within the + // cooldown window. + const lastJob = await engine.executeRaw<{ + finished_at: Date | null; + status: string; + }>( + `SELECT finished_at, status + FROM minion_jobs + WHERE name = 'embed-backfill' + AND data->>'sourceId' = $1 + ORDER BY id DESC LIMIT 1`, + [sourceId], + ); + + if (lastJob[0]) { + if (lastJob[0].status === 'active' || lastJob[0].status === 'waiting') { + // Active or waiting: no cooldown-remaining number (would be misleading). + return { status: 'cooldown' }; + } + if (lastJob[0].finished_at) { + const finishedMs = new Date(lastJob[0].finished_at).getTime(); + const ageMs = now - finishedMs; + const cooldownMs = cooldownMin * 60 * 1000; + if (ageMs < cooldownMs) { + return { + status: 'cooldown', + cooldownRemainingSeconds: Math.ceil((cooldownMs - ageMs) / 1000), + }; + } + } + } + + // ── 24h rolling spend cap ───────────────────────────────────── + const spend24hFn = opts.spend24hFn ?? defaultSpend24hForSource; + const spend24h = await spend24hFn(engine, sourceId); + if (spend24h >= spendCap) { + return { + status: 'spend_capped', + spend24hUsd: spend24h, + spendCapUsd: spendCap, + }; + } + + // ── Submission ──────────────────────────────────────────────── + const queue = new MinionQueue(engine); + const job = await queue.add( + 'embed-backfill', + { sourceId, batchSize: 500, reason: opts.reason }, + { + priority: opts.priority ?? 5, + idempotency_key: `embed-backfill:${sourceId}:${bucketize(now, 5 * 60_000)}`, + maxWaiting: 1, + }, + ); + + return { status: 'submitted', jobId: job.id }; +} + +/** Round timestamp down to the nearest `bucketMs` boundary. */ +function bucketize(ms: number, bucketMs: number): number { + return Math.floor(ms / bucketMs); +} diff --git a/src/core/embed-stale.ts b/src/core/embed-stale.ts new file mode 100644 index 000000000..8451d6fa1 --- /dev/null +++ b/src/core/embed-stale.ts @@ -0,0 +1,211 @@ +/** + * Stale-chunk embedding loop, extracted from `src/commands/embed.ts:embedAllStale` + * for reuse by the v0.40 `embed-backfill` Minion handler (D15.2 — codex + * outside-voice catch). + * + * Single source of truth for the cursor-paginated, source-grouped, rate-limit- + * aware embedding pipeline. Both `gbrain embed --stale` (CLI) and the + * `embed-backfill` job (Minion) call this helper so the working machinery + * — keyset pagination, batch grouping by `source_id::slug`, merge-with-existing + * via `getChunks` + `upsertChunks`, AbortSignal threading into in-flight HTTPs, + * 429 backoff — lives in exactly one place. + * + * Why a separate file (vs. exporting from embed.ts): embed.ts is a CLI command + * with logging side-effects and EmbedResult-shaped aggregation. The handler + * needs a tighter functional surface (returns embedded count + done state, no + * console.log). Extracting kept embed.ts's outer flow intact while letting the + * handler call a clean primitive. + */ + +import type { BrainEngine } from './engine.ts'; +import type { ChunkInput } from './types.ts'; +import { embedBatchWithBackoff } from '../commands/embed.ts'; + +/** Last visited (page_id, chunk_index) for keyset-resume across runs. */ +export interface StaleCursor { + afterPageId: number; + afterChunkIndex: number; +} + +export interface EmbedStaleOpts { + /** Chunks per cursor page. Default 2000 (matches the legacy CLI default). */ + batchSize?: number; + /** Max parallel slug-keys embedded inside a single batch. Default 20. */ + concurrency?: number; + /** Resume cursor from a prior run. Default: from start. */ + cursor?: StaleCursor; + /** AbortSignal honored at three sites: batch claim, retry sleep, HTTP body. */ + signal?: AbortSignal; + /** + * Fired once per batch with the cursor after that batch finishes. Caller + * uses this for crash-resumable progress (Minion `job.updateProgress`). + */ + onProgress?: (state: { + embedded: number; + chunksProcessed: number; + cursor: StaleCursor; + }) => void; + /** + * Optional caller-supplied embed fn. Defaults to `embedBatchWithBackoff`. + * Test seam: lets unit tests inject a deterministic fake without mocking + * the gateway. Production callers leave it unset. + */ + embedFn?: (texts: string[], opts: { abortSignal?: AbortSignal }) => Promise; +} + +export interface EmbedStaleResult { + /** Chunks newly embedded in this call. */ + embedded: number; + /** Total chunks pulled across all batches (including ones that errored). */ + chunksProcessed: number; + /** Pages whose embeddings landed. */ + pagesProcessed: number; + /** Last cursor reached. null iff zero stale chunks existed at start. */ + lastCursor: StaleCursor | null; + /** True iff the loop exited because every stale chunk was processed. */ + done: boolean; + /** True iff the loop exited because `signal.aborted` fired. */ + aborted: boolean; +} + +/** + * Embed every stale chunk (embedding IS NULL) for a source. + * + * Re-entrancy contract: if interrupted, the next call resumes from the next + * stale row. Idempotent — `embedding IS NULL` predicate naturally excludes + * already-embedded chunks even without cursor persistence. The cursor is a + * progress optimization, not a correctness mechanism. + * + * Returns when: + * - every stale chunk has been embedded (`done: true`), OR + * - `signal.aborted` fired (`aborted: true`), OR + * - the per-batch `embedFn` threw with the signal aborted (treated as abort). + * + * Per-page embed failures (network blip, dim mismatch) do NOT throw — the + * embedding stays NULL and the next call retries the chunk. This matches the + * existing CLI's "log + skip" semantics so a single bad page doesn't poison + * the run. Caller is responsible for surfacing partial-success via the + * returned `embedded` vs `chunksProcessed` delta. + */ +export async function embedStaleForSource( + engine: BrainEngine, + sourceId: string, + opts: EmbedStaleOpts = {}, +): Promise { + const batchSize = opts.batchSize ?? 2000; + const concurrency = opts.concurrency ?? 20; + const signal = opts.signal; + const embedFn = opts.embedFn ?? ((texts, fnOpts) => + embedBatchWithBackoff(texts, { abortSignal: fnOpts.abortSignal })); + + let afterPageId = opts.cursor?.afterPageId ?? 0; + let afterChunkIndex = opts.cursor?.afterChunkIndex ?? -1; + + const result: EmbedStaleResult = { + embedded: 0, + chunksProcessed: 0, + pagesProcessed: 0, + lastCursor: null, + done: false, + aborted: false, + }; + + for (;;) { + if (signal?.aborted) { + result.aborted = true; + return result; + } + + const batch = await engine.listStaleChunks({ + batchSize, + afterPageId, + afterChunkIndex, + sourceId, + }); + if (batch.length === 0) { + result.done = true; + return result; + } + + result.chunksProcessed += batch.length; + const last = batch[batch.length - 1]; + afterPageId = last.page_id; + afterChunkIndex = last.chunk_index; + result.lastCursor = { afterPageId, afterChunkIndex }; + + // Group by composite key (source_id::slug). Within a source-scoped run + // every row carries the same source_id, but the helper accepts batches + // shaped by `listStaleChunks` which carry source_id per row for parity + // with the cross-source CLI path. + const byKey = new Map(); + for (const row of batch) { + const key = `${row.source_id}::${row.slug}`; + const list = byKey.get(key); + if (list) list.push(row); + else byKey.set(key, [row]); + } + + const keys = Array.from(byKey.keys()); + let nextIdx = 0; + + async function embedOneKey(key: string): Promise { + const stale = byKey.get(key)!; + const keySourceId = stale[0]?.source_id ?? sourceId; + const slug = stale[0].slug; + try { + const embeddings = await embedFn( + stale.map((c) => c.chunk_text), + { abortSignal: signal }, + ); + const existing = await engine.getChunks(slug, { sourceId: keySourceId }); + const staleIdxToEmbedding = new Map(); + for (let j = 0; j < stale.length; j++) { + staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]); + } + const merged: ChunkInput[] = existing.map((c) => ({ + chunk_index: c.chunk_index, + chunk_text: c.chunk_text, + chunk_source: c.chunk_source, + embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined, + token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), + })); + await engine.upsertChunks(slug, merged, { sourceId: keySourceId }); + result.embedded += stale.length; + result.pagesProcessed += 1; + } catch (e: unknown) { + // Aborted mid-fetch is expected; treat as graceful exit. + if (signal?.aborted) return; + // Otherwise log and skip — the chunk stays NULL and next call retries. + process.stderr.write( + `\n [embed-stale] error on ${keySourceId}/${slug}: ${ + e instanceof Error ? e.message : String(e) + }\n`, + ); + } + } + + async function worker(): Promise { + while (nextIdx < keys.length && !signal?.aborted) { + const idx = nextIdx++; + await embedOneKey(keys[idx]); + } + } + + const numWorkers = Math.min(concurrency, keys.length); + await Promise.all(Array.from({ length: numWorkers }, () => worker())); + + if (opts.onProgress) { + opts.onProgress({ + embedded: result.embedded, + chunksProcessed: result.chunksProcessed, + cursor: { afterPageId, afterChunkIndex }, + }); + } + + // Short batch = end of stale set; advance and exit. + if (batch.length < batchSize) { + result.done = true; + return result; + } + } +} diff --git a/src/core/feature-flags.ts b/src/core/feature-flags.ts new file mode 100644 index 000000000..ee0fe42b2 --- /dev/null +++ b/src/core/feature-flags.ts @@ -0,0 +1,49 @@ +/** + * Feature flags (v0.40 D23). + * + * Single shared escape hatch for the v0.40 Federated Sync v2 cathedral. + * Flipping `sync.federated_v2` to `false` reverts to v0.39 sequential + * behavior without re-installing the binary — useful if a foundational + * assumption proves wrong in production (e.g. parallel sync trips an + * undiscovered Postgres lock contention). + * + * What the flag gates: + * - Parallel branch of `gbrain sync --all` (serial fallback otherwise) + * - Auto-enqueue of embed-backfill in the extended `sync` handler + * - Autopilot's per-source freshness-gate dispatch (D17) + * + * What stays on UNCONDITIONALLY (correctness, not features): + * - Per-source sync lock (`syncLockId`) + * - Phantom-redirect per-source lock (D16) + * - Migration v87 (sources_github_repo index) + * - Facts-backstop source-scoping fix (D21) + * - safeHexEqual extraction (D15.5) + * + * Disable path: + * gbrain config set sync.federated_v2 false + * gbrain jobs supervisor restart # autopilot picks up the change + * + * Convention: this module is the ONLY place that reads the flag. Callers go + * through `isFederatedV2Enabled(engine)` so future changes to the flag key, + * default, or backing store happen in one place. + */ +import type { BrainEngine } from './engine.ts'; + +export const FEDERATED_V2_CONFIG_KEY = 'sync.federated_v2'; + +/** + * True iff Federated Sync v2 behaviors are enabled (default true). + * + * Reads `sync.federated_v2` from the DB config plane via `engine.getConfig`. + * Values: `'false'` → disabled; anything else (including missing/null) → + * enabled. The default-on posture is deliberate — v0.40 ships expecting the + * new behavior, and ops opt out by setting the key explicitly. + * + * Throwing on engine errors is fine: the flag is only checked at boundary + * points (CLI dispatch, autopilot tick, sync handler) where an engine error + * would surface anyway. Callers don't need a try/catch wrapper. + */ +export async function isFederatedV2Enabled(engine: BrainEngine): Promise { + const value = await engine.getConfig(FEDERATED_V2_CONFIG_KEY); + return value !== 'false'; +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 949ade63d..e337ade96 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4232,6 +4232,31 @@ export const MIGRATIONS: Migration[] = [ } }, }, + { + version: 92, + name: 'sources_github_repo_index', + // v0.40.5.0 Federated Sync v2 (D13): partial expression index on + // sources.config->>'github_repo' so the new POST /webhooks/github + // handler's source-by-repo lookup uses an index instead of a sequential + // scan. Sources is small today (<100 rows in practice) so the impact is + // microseconds, but the lookup fires on every webhook event (including + // ignored ones) and a team with hundreds of sources would feel it. + // + // Partial WHERE clause keeps the index small — only rows with a + // configured webhook actually take up index entries. Both Postgres and + // PGLite support partial expression indexes; no engine-specific shape. + // Idempotent (IF NOT EXISTS). + // + // Plan called this v81 originally; renumbered through v87 → v89 → v90 → v92 + // across successive master merges (v0.40.2.0 claimed v89 for + // facts_event_type_column; v0.40.3.0 claimed v90 + v91 for + // contextual_retrieval_columns + pages_generation_trigger_and_bookmark). + sql: ` + CREATE INDEX IF NOT EXISTS sources_github_repo_idx + ON sources ((config->>'github_repo')) + WHERE config ? 'github_repo'; + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/handlers/embed-backfill.ts b/src/core/minions/handlers/embed-backfill.ts new file mode 100644 index 000000000..02fd0dcad --- /dev/null +++ b/src/core/minions/handlers/embed-backfill.ts @@ -0,0 +1,182 @@ +/** + * `embed-backfill` minion handler (v0.40 Federated Sync v2). + * + * Decouples embedding from the sync pipeline so: + * - `gbrain sync --all` finishes fast (pages searchable via keyword + * immediately), embed-backfill catches up async (D18). + * - Webhook-driven syncs don't block on Voyage rate limits (D5 + D18). + * - Fresh-source onboarding (federate a 50K-page repo) doesn't make + * the user wait for ~$3-$10 of embedding before the sync "completes." + * + * The handler is the run-side companion to `submitEmbedBackfill` in + * `src/core/embed-backfill-submit.ts`. The submit-side gate (D19) handles + * cross-call rate-limiting (10min cooldown + 24h $25 rolling cap); this + * handler handles within-run safety: + * + * - D2: per-source DB lock (`gbrain-embed-backfill:`) prevents + * two embed-backfill jobs for the same source from running concurrently. + * If a second job claims while the first is mid-loop, it returns + * `already_in_progress` cleanly and the lock is the source of truth. + * + * - D6: BudgetTracker enforces per-job spend cap (default $10/job). Goes + * through `withBudgetTracker` so `gateway.embed()` auto-composes via + * AsyncLocalStorage. On `BudgetExhausted` throw, partial progress is + * preserved (chunks already embedded stay embedded; remaining stays NULL). + * + * - D15.1: parent-job linkage is INTENTIONALLY OMITTED. The submit-side + * helper does not pass `parent_job_id` — the queue's parent-child + * semantics flip the parent into `waiting-children` and fail completion. + * Sync handlers are short-lived; embed-backfill outlives them by design. + * + * - try/finally ALWAYS releases the per-source lock. Aborted runs leave + * the next call free to claim. + */ +import { tryAcquireDbLock } from '../../db-lock.ts'; +import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts'; +import { withBudgetTracker } from '../../ai/gateway.ts'; +import { embedStaleForSource } from '../../embed-stale.ts'; +import type { BrainEngine } from '../../engine.ts'; +import type { MinionJobContext } from '../types.ts'; + +const DEFAULT_MAX_USD_PER_JOB = 10; +const EMBED_BACKFILL_LOCK_TTL_MIN = 60; + +export interface EmbedBackfillJobData { + sourceId: string; + batchSize?: number; + /** Audit string from the submitter (e.g. 'webhook', 'federation_flip'). */ + reason?: string; +} + +export interface EmbedBackfillResult { + status: 'success' | 'already_in_progress' | 'budget_exhausted' | 'aborted'; + sourceId: string; + embedded: number; + chunksProcessed: number; + pagesProcessed: number; + /** $USD spent inside this job (from BudgetTracker.totalSpent). */ + spentUsd: number; + /** Set when status === 'budget_exhausted'. */ + budgetCapUsd?: number; +} + +/** Compose the lock id for embed-backfill, namespaced like sync's. */ +function embedBackfillLockId(sourceId: string): string { + return `gbrain-embed-backfill:${sourceId}`; +} + +/** Read embed.backfill_max_usd config or default. */ +async function readMaxUsd(engine: BrainEngine): Promise { + const raw = await engine.getConfig('embed.backfill_max_usd'); + if (raw === null || raw === undefined) return DEFAULT_MAX_USD_PER_JOB; + const n = Number(raw); + return Number.isFinite(n) && n > 0 ? n : DEFAULT_MAX_USD_PER_JOB; +} + +/** Validate + extract typed job params. Throws on malformed input. */ +function parseParams(data: Record): EmbedBackfillJobData { + const sourceId = data.sourceId; + if (typeof sourceId !== 'string' || sourceId.length === 0) { + throw new Error('embed-backfill: data.sourceId is required and must be a non-empty string'); + } + const batchSize = + typeof data.batchSize === 'number' && data.batchSize > 0 + ? data.batchSize + : undefined; + const reason = + typeof data.reason === 'string' ? data.reason : undefined; + return { sourceId, batchSize, reason }; +} + +export function makeEmbedBackfillHandler(engine: BrainEngine) { + return async function embedBackfillHandler( + job: MinionJobContext, + ): Promise { + const { sourceId, batchSize } = parseParams(job.data); + + // D2: per-source lock at handler entry. The submit-side cooldown (D19) + // prevents most contention but this is the run-side safety net. + const lockKey = embedBackfillLockId(sourceId); + const lock = await tryAcquireDbLock(engine, lockKey, EMBED_BACKFILL_LOCK_TTL_MIN); + if (!lock) { + return { + status: 'already_in_progress', + sourceId, + embedded: 0, + chunksProcessed: 0, + pagesProcessed: 0, + spentUsd: 0, + }; + } + + // D6: budget-tracked execution. Gateway calls inside withBudgetTracker + // auto-compose via AsyncLocalStorage; if pricing pushes cumulative spend + // past the cap, gateway throws BudgetExhausted BEFORE the next API call. + const capUsd = await readMaxUsd(engine); + const tracker = new BudgetTracker({ + maxCostUsd: capUsd, + label: `embed-backfill:${sourceId}`, + }); + + try { + const result = await withBudgetTracker(tracker, async () => + embedStaleForSource(engine, sourceId, { + batchSize, + signal: job.signal, + onProgress: ({ embedded, chunksProcessed, cursor }) => { + // Fire-and-forget; updateProgress returns a Promise but the + // handler is sync inside the loop. + void job.updateProgress({ + embedded, + chunksProcessed, + cursor, + spentUsd: tracker.totalSpent, + }); + }, + }), + ); + + if (result.aborted) { + return { + status: 'aborted', + sourceId, + embedded: result.embedded, + chunksProcessed: result.chunksProcessed, + pagesProcessed: result.pagesProcessed, + spentUsd: tracker.totalSpent, + }; + } + return { + status: 'success', + sourceId, + embedded: result.embedded, + chunksProcessed: result.chunksProcessed, + pagesProcessed: result.pagesProcessed, + spentUsd: tracker.totalSpent, + }; + } catch (err) { + if (err instanceof BudgetExhausted) { + // Partial progress preserved: already-embedded chunks stay embedded; + // remaining stays NULL for the next run to pick up. + return { + status: 'budget_exhausted', + sourceId, + embedded: 0, // Tracker doesn't track per-chunk count + chunksProcessed: 0, + pagesProcessed: 0, + spentUsd: tracker.totalSpent, + budgetCapUsd: capUsd, + }; + } + throw err; + } finally { + // ALWAYS release. Aborts, throws, budget-exhaust — all paths unwind here. + try { + await lock.release(); + } catch { + // Lock release best-effort; TTL fallback covers the case where the + // row was already cleared by a parallel writer. + } + } + }; +} diff --git a/src/core/parallel.ts b/src/core/parallel.ts new file mode 100644 index 000000000..cf13bad80 --- /dev/null +++ b/src/core/parallel.ts @@ -0,0 +1,59 @@ +/** + * Bounded-concurrency Promise.allSettled. + * + * Runs `fn(item)` for each item with at most `concurrency` in flight at a time. + * Returns a `PromiseSettledResult` per input item, in input order, so callers + * can distinguish fulfilled-with-result from rejected-with-error per item. + * + * Used by `gbrain sync --all` (v0.40 Federated Sync v2) to fan out per-source + * syncs without overwhelming the embedding API or local disk. + * + * Why a semaphore + `Promise.allSettled` instead of a library: + * - Zero new deps. Bun's stdlib has every primitive we need. + * - `Promise.allSettled` semantics are what we want: one source failing + * must NOT short-circuit the others. Bare `Promise.all` rejects on + * first failure. + * - The concurrency cap is a hard ceiling, not a target. With N items and + * concurrency C, the function makes at most C concurrent `fn` calls at + * any moment. + * + * Order guarantee: results[i] always corresponds to items[i]. The execution + * order of `fn` calls is NOT guaranteed (a fast item submitted later can + * complete before a slow item submitted earlier). + */ + +export async function pMapAllSettled( + items: readonly T[], + concurrency: number, + fn: (item: T, index: number) => Promise, +): Promise>> { + if (!Number.isInteger(concurrency) || concurrency < 1) { + throw new TypeError( + `pMapAllSettled: concurrency must be a positive integer, got ${concurrency}`, + ); + } + if (items.length === 0) return []; + + const results: Array> = new Array(items.length); + const effectiveConcurrency = Math.min(concurrency, items.length); + + let nextIndex = 0; + + // Each worker pulls the next available index and runs fn until exhausted. + async function worker(): Promise { + for (;;) { + const i = nextIndex++; + if (i >= items.length) return; + try { + const value = await fn(items[i], i); + results[i] = { status: 'fulfilled', value }; + } catch (err) { + results[i] = { status: 'rejected', reason: err }; + } + } + } + + const workers = Array.from({ length: effectiveConcurrency }, () => worker()); + await Promise.all(workers); + return results; +} diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index efb041f0b..b93e9d8f5 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -56,6 +56,12 @@ INSERT INTO sources (id, name, config) VALUES ('default', 'default', '{"federated": true}'::jsonb) ON CONFLICT (id) DO NOTHING; +-- v0.40 Federated Sync v2: partial expression index on config->>'github_repo' +-- (mirror of src/schema.sql; migration v92 backfills legacy brains). +CREATE INDEX IF NOT EXISTS sources_github_repo_idx + ON sources ((config->>'github_repo')) + WHERE config ? 'github_repo'; + -- ============================================================ -- pages: the core content table -- ============================================================ diff --git a/src/core/source-config-redact.ts b/src/core/source-config-redact.ts new file mode 100644 index 000000000..2328e26d3 --- /dev/null +++ b/src/core/source-config-redact.ts @@ -0,0 +1,47 @@ +/** + * Redact secrets from sources.config before serializing (v0.40 D15.4). + * + * Codex outside-voice catch on the v0.40 plan: `webhook show` hides the + * secret, but any OTHER surface that returns raw `sources.config` (the + * sources list --json, an MCP get_source op, an HTTP admin API) would + * leak it. This module is the single redaction primitive every serializer + * routes through. + * + * `scripts/check-source-config-leak.sh` grep-guards against future drift. + */ + +const SECRET_KEYS: ReadonlySet = new Set([ + 'webhook_secret', +]); + +/** + * Return a shallow copy of `config` with every key in SECRET_KEYS replaced + * with the string ''. Original input is not mutated. + * + * Non-object input returns an empty object (defensive — sources.config is + * always a JSON object, but driver layer may briefly return a string mid- + * parse on legacy brains). + */ +export function redactSourceConfig( + config: unknown, +): Record { + if (config === null || typeof config !== 'object' || Array.isArray(config)) { + return {}; + } + const out: Record = {}; + for (const [k, v] of Object.entries(config as Record)) { + if (SECRET_KEYS.has(k)) { + out[k] = ''; + } else { + out[k] = v; + } + } + return out; +} + +/** True iff the config object carries a webhook secret. */ +export function hasWebhookSecret(config: unknown): boolean { + if (!config || typeof config !== 'object') return false; + const c = config as Record; + return typeof c.webhook_secret === 'string' && c.webhook_secret.length > 0; +} diff --git a/src/core/source-health.ts b/src/core/source-health.ts new file mode 100644 index 000000000..13135724a --- /dev/null +++ b/src/core/source-health.ts @@ -0,0 +1,212 @@ +/** + * Per-source health metrics (v0.40 D12 + D9 + D17 + D19). + * + * Single source of truth for `gbrain sources status` AND `gbrain doctor`'s + * `federation_health` check. Sharing the implementation prevents the dashboard + * and the doctor warning from drifting. + * + * D12: batched GROUP BY queries — 4 queries total instead of 6×N per-source + * round-trips. On a 4-source / 300K-chunk brain this drops dashboard + * time from ~24s to <2s. + * + * D9: resolvePriority(config) — accepts 'high'|'normal'|'low', falls back + * to 0 with once-per-source-per-process stderr warn on unknown values. + * + * D17: isSourceStale helper — autopilot calls this to decide per-source + * sync dispatch independent of the brain_score gate. + */ +import type { BrainEngine } from './engine.ts'; +import { parseSourceConfig, type SourceRow } from './sources-load.ts'; + +export interface SourceMetrics { + source_id: string; + name: string; + local_path: string | null; + federated: boolean; + total_pages: number; + total_chunks: number; + embedded_chunks: number; + embed_coverage_pct: number; + last_sync_at: Date | null; + lag_seconds: number | null; + /** Failed jobs (sync OR embed-backfill) for this source in last 24h. */ + failed_jobs_24h: number; + /** Waiting + active + delayed jobs (sync OR embed-backfill) for this source. */ + queue_depth: number; + tracked_branch: string | null; + priority_label: PriorityLabel; + /** Webhook configured? (true iff config.webhook_secret is set.) */ + webhook_configured: boolean; +} + +export type PriorityLabel = 'high' | 'normal' | 'low'; + +/** Numeric priority used by MinionQueue.add({ priority }). Lower = sooner. */ +const PRIORITY_VALUE: Record = { + high: -10, + normal: 0, + low: 5, +}; + +const KNOWN_PRIORITY: Set = new Set(['high', 'normal', 'low']); + +/** Stderr-warn-once memo so a tight autopilot loop doesn't spam. */ +const _warnedSources = new Set(); + +/** Test seam: reset memo so unit tests can re-trigger the warn path. */ +export function _resetPriorityWarningsForTest(): void { + _warnedSources.clear(); +} + +/** + * Resolve a source's priority label from its config row. + * + * Recognized values: 'high', 'normal', 'low'. Anything else (typos, integers, + * nested objects) falls back to 'normal' AND emits a once-per-source-per- + * process stderr warning naming the bad value + the fix command. Missing + * key is silent ('normal' is the default). + */ +export function resolvePriorityLabel( + sourceId: string, + config: unknown, +): PriorityLabel { + const parsed = parseSourceConfig(config); + const raw = parsed.priority; + if (raw === undefined || raw === null) return 'normal'; + if (typeof raw === 'string' && KNOWN_PRIORITY.has(raw)) { + return raw as PriorityLabel; + } + // Warn once per source per process. + if (!_warnedSources.has(sourceId)) { + _warnedSources.add(sourceId); + process.stderr.write( + `[gbrain] source "${sourceId}": invalid config.priority value ${JSON.stringify(raw)}; ` + + `falling back to 'normal'. Fix: gbrain sources config set ${sourceId} priority normal\n`, + ); + } + return 'normal'; +} + +/** Numeric priority for queue.add. */ +export function resolvePriority(sourceId: string, config: unknown): number { + return PRIORITY_VALUE[resolvePriorityLabel(sourceId, config)]; +} + +/** + * True iff the source's last_sync_at is older than `intervalMs`, OR it has + * never synced. Sources without a local_path are NOT considered stale (no + * way to sync them). Used by autopilot D17 freshness gate. + */ +export function isSourceStale(src: SourceRow, intervalMs: number): boolean { + if (!src.local_path) return false; + if (!src.last_sync_at) return true; + const lastMs = new Date(src.last_sync_at).getTime(); + return Date.now() - lastMs >= intervalMs; +} + +/** + * Compute per-source metrics for every source in one shot. + * + * Batched GROUP BY pipeline: + * 1. sources: id, name, local_path, last_sync_at, config (one SELECT) + * 2. pages by source_id (one GROUP BY) + * 3. chunks by source_id with FILTER(embedding NOT NULL) (one GROUP BY) + * 4. minion_jobs by data->>'sourceId' with FILTERs for failed-24h + queue depth + * + * Total: 4 queries regardless of source count. Each scans the relevant table + * once. Same cost as the slowest single-source query in the old per-source loop. + */ +export async function computeAllSourceMetrics( + engine: BrainEngine, + sources: SourceRow[], +): Promise { + if (sources.length === 0) return []; + + const pageCounts = await pageCountsBySource(engine); + const chunkCounts = await chunkCountsBySource(engine); + const jobCounts = await jobCountsBySource(engine); + const now = Date.now(); + + return sources.map((src) => { + const cfg = parseSourceConfig(src.config); + const pages = pageCounts.get(src.id) ?? 0; + const chunkStats = chunkCounts.get(src.id) ?? { total: 0, embedded: 0 }; + const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0 }; + + const embedCoverage = chunkStats.total === 0 + ? 100 + : Math.round((chunkStats.embedded / chunkStats.total) * 1000) / 10; + + const lastMs = src.last_sync_at ? new Date(src.last_sync_at).getTime() : null; + const lagSeconds = lastMs === null + ? null + : Math.max(0, Math.floor((now - lastMs) / 1000)); + + return { + source_id: src.id, + name: src.name, + local_path: src.local_path, + federated: cfg.federated === true, + total_pages: pages, + total_chunks: chunkStats.total, + embedded_chunks: chunkStats.embedded, + embed_coverage_pct: embedCoverage, + last_sync_at: src.last_sync_at, + lag_seconds: lagSeconds, + failed_jobs_24h: jobStats.failed_24h, + queue_depth: jobStats.queue_depth, + tracked_branch: typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : null, + priority_label: resolvePriorityLabel(src.id, src.config), + webhook_configured: typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0, + }; + }); +} + +async function pageCountsBySource(engine: BrainEngine): Promise> { + const rows = await engine.executeRaw<{ source_id: string; n: number }>( + `SELECT source_id, COUNT(*)::int AS n + FROM pages + WHERE deleted_at IS NULL + GROUP BY source_id`, + ); + const m = new Map(); + for (const r of rows) m.set(r.source_id, Number(r.n)); + return m; +} + +async function chunkCountsBySource(engine: BrainEngine): Promise> { + const rows = await engine.executeRaw<{ source_id: string; total: number; embedded: number }>( + `SELECT p.source_id, + COUNT(*)::int AS total, + COUNT(*) FILTER (WHERE c.embedding IS NOT NULL)::int AS embedded + FROM content_chunks c + JOIN pages p ON p.id = c.page_id + WHERE p.deleted_at IS NULL + GROUP BY p.source_id`, + ); + const m = new Map(); + for (const r of rows) m.set(r.source_id, { total: Number(r.total), embedded: Number(r.embedded) }); + return m; +} + +async function jobCountsBySource(engine: BrainEngine): Promise> { + // Pre-v0.11 brains don't have minion_jobs; return empty map. + try { + const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number }>( + `SELECT data->>'sourceId' AS source_id, + COUNT(*) FILTER (WHERE status IN ('failed','dead') AND created_at > NOW() - INTERVAL '24 hours')::int AS failed_24h, + COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth + FROM minion_jobs + WHERE name IN ('sync','embed-backfill') + AND data->>'sourceId' IS NOT NULL + GROUP BY data->>'sourceId'`, + ); + const m = new Map(); + for (const r of rows) { + m.set(r.source_id, { failed_24h: Number(r.failed_24h), queue_depth: Number(r.queue_depth) }); + } + return m; + } catch { + return new Map(); + } +} diff --git a/src/core/sources-load.ts b/src/core/sources-load.ts new file mode 100644 index 000000000..eb564135c --- /dev/null +++ b/src/core/sources-load.ts @@ -0,0 +1,129 @@ +/** + * Shared source-table loader (v0.40 Federated Sync v2 — D7). + * + * Before v0.40, the only caller that enumerated `sources` was `runList` in + * src/commands/sources.ts. v0.40 adds four more enumerators: `gbrain sync --all` + * fan-out, autopilot per-source dispatch, `gbrain sources status`, and the + * `federation_health` doctor check. Going from 1→5 inline SELECTs invites + * silent drift the next time someone adds a column to `sources`. + * + * This module is the single source of truth for that read path. Adding a + * column means updating exactly one projection. + * + * Engine-agnostic: works on both Postgres and PGLite (same SQL surface). + * + * Why no engine method: BrainEngine parity would force PGLite + Postgres + * implementations even though both run identical SQL through `executeRaw`. + * A shared helper hits the bar at lower cost. + */ +import type { BrainEngine } from './engine.ts'; + +export interface SourceRow { + id: string; + name: string; + local_path: string | null; + last_commit: string | null; + last_sync_at: Date | null; + /** Postgres returns object; PGLite returns JSON string. Parse via `parseSourceConfig`. */ + config: Record | string; + created_at: Date; + archived?: boolean; +} + +export interface LoadAllSourcesOpts { + /** Include soft-archived rows (default false). */ + includeArchived?: boolean; + /** Only return sources with config.federated === true (default false). */ + federatedOnly?: boolean; +} + +/** Parse `sources.config` to a plain object regardless of driver shape. */ +export function parseSourceConfig(config: unknown): Record { + if (typeof config === 'string') { + try { return JSON.parse(config) as Record; } catch { return {}; } + } + if (typeof config === 'object' && config !== null) return config as Record; + return {}; +} + +/** True iff the source's config.federated field is the literal boolean true. */ +export function isSourceFederated(config: unknown): boolean { + const parsed = parseSourceConfig(config); + return parsed.federated === true; +} + +/** + * Enumerate every source. Order: 'default' first, then alphabetical by id. + * + * Caller filters in-process when the predicate is cheap (federatedOnly, + * includeArchived). For source-id targeted reads use `fetchSource` instead + * (single-row SELECT). + */ +export async function loadAllSources( + engine: BrainEngine, + opts: LoadAllSourcesOpts = {}, +): Promise { + // Defensive on legacy brains pre-v0.26.5 that lack the archived column. + let rows: SourceRow[]; + try { + rows = await engine.executeRaw( + `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived + FROM sources + ORDER BY (id = 'default') DESC, id`, + ); + } catch (err) { + // Forward-reference safety: pre-v0.26.5 brains have no `archived` column. + // Re-issue without it; archived defaults to false. + if (isUndefinedColumnError(err)) { + rows = await engine.executeRaw( + `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at + FROM sources + ORDER BY (id = 'default') DESC, id`, + ); + } else { + throw err; + } + } + + let filtered = rows; + if (!opts.includeArchived) { + filtered = filtered.filter((r) => r.archived !== true); + } + if (opts.federatedOnly) { + filtered = filtered.filter((r) => isSourceFederated(r.config)); + } + return filtered; +} + +/** Single-row fetch — kept here so callers don't grow yet-another SELECT. */ +export async function fetchSource( + engine: BrainEngine, + id: string, +): Promise { + try { + const rows = await engine.executeRaw( + `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived + FROM sources WHERE id = $1`, + [id], + ); + return rows[0] ?? null; + } catch (err) { + if (isUndefinedColumnError(err)) { + const rows = await engine.executeRaw( + `SELECT id, name, local_path, last_commit, last_sync_at, config, created_at + FROM sources WHERE id = $1`, + [id], + ); + return rows[0] ?? null; + } + throw err; + } +} + +/** Driver-tolerant 42703 detector. Mirrors src/core/utils.ts pattern. */ +function isUndefinedColumnError(err: unknown): boolean { + if (!err || typeof err !== 'object') return false; + const e = err as { code?: string; message?: string }; + if (e.code === '42703') return true; + return typeof e.message === 'string' && /column .* does not exist/i.test(e.message); +} diff --git a/src/core/timing-safe.ts b/src/core/timing-safe.ts new file mode 100644 index 000000000..d76fde645 --- /dev/null +++ b/src/core/timing-safe.ts @@ -0,0 +1,34 @@ +/** + * Constant-time hex string compare (v0.40 D15.5 extraction). + * + * Before v0.40 this lived as a closure inside `runServeHttp` in + * `src/commands/serve-http.ts:601`. It needs to be importable so the new + * `POST /webhooks/github` handler can verify GitHub HMAC-SHA256 signatures + * with the same constant-time semantics as the admin-cookie compare. + * + * Why extract instead of duplicate: codex outside-voice flagged the closure + * scope as "hand-waving over actual module boundaries." Two consumers + one + * function = extract. Source-text grep guard in `test/timing-safe.test.ts` + * pins that callers import from here rather than re-implementing. + * + * Both inputs are expected to be hex strings of equal length (caller asserts + * the length invariant — e.g. sha256 hex is always 64 chars). `timingSafeEqual` + * throws on length mismatch; the early-return on length keeps the comparison + * timing-independent of the length-check itself. + */ +import { timingSafeEqual } from 'node:crypto'; + +/** + * True iff `a` and `b` are equal-length hex strings with the same bytes. + * False on length mismatch (does NOT throw — admin-cookie + webhook callers + * both prefer a clean boolean return so they can route to a 401 cleanly). + * + * Constant-time over the byte compare: `timingSafeEqual` is the underlying + * primitive. Length mismatch short-circuits, which is acceptable because + * length itself is not a secret (both callers know the expected hex length + * from the algorithm — sha256 = 64 hex chars). + */ +export function safeHexEqual(a: string, b: string): boolean { + if (a.length !== b.length) return false; + return timingSafeEqual(Buffer.from(a, 'hex'), Buffer.from(b, 'hex')); +} diff --git a/src/schema.sql b/src/schema.sql index 1f60e0ac8..ca0e8a527 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -61,6 +61,15 @@ INSERT INTO sources (id, name, config) VALUES ('default', 'default', '{"federated": true}'::jsonb) ON CONFLICT (id) DO NOTHING; +-- v0.40 Federated Sync v2: partial expression index on config->>'github_repo' +-- so POST /webhooks/github's source-by-repo lookup hits an index. Only rows +-- with a configured webhook actually take up index entries. Both Postgres and +-- PGLite support partial expression indexes. Migration v87 installs the same +-- index on legacy brains (idempotent IF NOT EXISTS). +CREATE INDEX IF NOT EXISTS sources_github_repo_idx + ON sources ((config->>'github_repo')) + WHERE config ? 'github_repo'; + -- ============================================================ -- pages: the core content table -- ============================================================ diff --git a/test/db-lock-per-source.test.ts b/test/db-lock-per-source.test.ts new file mode 100644 index 000000000..73b64776c --- /dev/null +++ b/test/db-lock-per-source.test.ts @@ -0,0 +1,86 @@ +/** + * v0.40 Federated Sync v2 — per-source lock contract. + * + * Pins the iron-rule regression for the SYNC_LOCK_ID rename: + * - SYNC_LOCK_ID === syncLockId('default') (back-compat alias) + * - syncLockId(X) !== syncLockId(Y) for X !== Y (per-source isolation) + * - two concurrent tryAcquireDbLock calls against DIFFERENT sources both succeed + * - two concurrent tryAcquireDbLock calls against the SAME source: second returns null + * + * Without this guard, a future drift in the constant value would silently change + * semantics (e.g. someone hardcoding 'gbrain-sync' elsewhere would no longer match + * the same row, breaking the writer-window exclusion that performSync relies on). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { syncLockId, SYNC_LOCK_ID, tryAcquireDbLock } from '../src/core/db-lock.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +describe('syncLockId (per-source lock helper)', () => { + test('default source returns gbrain-sync:default', () => { + expect(syncLockId('default')).toBe('gbrain-sync:default'); + }); + + test('non-default source returns gbrain-sync:', () => { + expect(syncLockId('zion-brain')).toBe('gbrain-sync:zion-brain'); + expect(syncLockId('media-corpus')).toBe('gbrain-sync:media-corpus'); + }); + + test('IRON-RULE: SYNC_LOCK_ID back-compat alias resolves to syncLockId(default)', () => { + expect(SYNC_LOCK_ID).toBe(syncLockId('default')); + expect(SYNC_LOCK_ID).toBe('gbrain-sync:default'); + }); + + test('different sources produce distinct lock keys', () => { + expect(syncLockId('a')).not.toBe(syncLockId('b')); + }); +}); + +describe('tryAcquireDbLock with per-source keys', () => { + test('two locks against different sources both succeed', async () => { + const lockA = await tryAcquireDbLock(engine, syncLockId('source-a')); + const lockB = await tryAcquireDbLock(engine, syncLockId('source-b')); + expect(lockA).not.toBeNull(); + expect(lockB).not.toBeNull(); + await lockA?.release(); + await lockB?.release(); + }); + + test('second lock against the same source returns null while first is held', async () => { + const first = await tryAcquireDbLock(engine, syncLockId('source-c')); + expect(first).not.toBeNull(); + const second = await tryAcquireDbLock(engine, syncLockId('source-c')); + expect(second).toBeNull(); + await first?.release(); + // After release, third acquire succeeds. + const third = await tryAcquireDbLock(engine, syncLockId('source-c')); + expect(third).not.toBeNull(); + await third?.release(); + }); + + test('SYNC_LOCK_ID and syncLockId(default) acquire the SAME lock row', async () => { + // This is the critical back-compat check: pre-v0.40 callers using SYNC_LOCK_ID + // and new callers using syncLockId('default') MUST conflict (not bypass each other). + const first = await tryAcquireDbLock(engine, SYNC_LOCK_ID); + expect(first).not.toBeNull(); + const second = await tryAcquireDbLock(engine, syncLockId('default')); + expect(second).toBeNull(); + await first?.release(); + }); +}); diff --git a/test/doctor-federation-health.test.ts b/test/doctor-federation-health.test.ts new file mode 100644 index 000000000..091ef92f9 --- /dev/null +++ b/test/doctor-federation-health.test.ts @@ -0,0 +1,102 @@ +/** + * Tests for src/commands/doctor.ts:checkFederationHealth (v0.40 T12). + * + * Three-state contract: + * ok — single-source brain, or every federated source healthy + * warn — lag > 1h + federated, coverage < 95% with chunks > 100, OR 3+ failures + * fail — lag > 24h, OR coverage < 50% with chunks > 1000 + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { checkFederationHealth } from '../src/commands/doctor.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await engine.executeRaw('DELETE FROM minion_jobs'); + await engine.executeRaw('DELETE FROM content_chunks'); + await engine.executeRaw('DELETE FROM pages'); + await engine.executeRaw(`DELETE FROM sources WHERE id != 'default'`); +}); + +describe('checkFederationHealth', () => { + test('single-source brain → ok with "no federation to check"', async () => { + const check = await checkFederationHealth(engine); + expect(check.name).toBe('federation_health'); + expect(check.status).toBe('ok'); + expect(check.message).toContain('no federation to check'); + }); + + test('multi-source healthy brain → ok with source count', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config, last_sync_at) VALUES ('extra', 'extra', '{"federated":true}', NOW())`, + ); + const check = await checkFederationHealth(engine); + expect(check.status).toBe('ok'); + expect(check.message).toContain('source(s) healthy'); + }); + + test('source with lag > 1h + federated → warn with remediation', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config, last_sync_at) VALUES ('stale-source', 'stale-source', '{"federated":true}', NOW() - INTERVAL '2 hours')`, + ); + const check = await checkFederationHealth(engine); + expect(check.status).toBe('warn'); + expect(check.message).toContain('stale-source'); + expect(check.message).toContain('gbrain sync trigger --source stale-source'); + }); + + test('source with lag > 24h → fail with remediation', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config, last_sync_at) VALUES ('dead-source', 'dead-source', '{"federated":true}', NOW() - INTERVAL '48 hours')`, + ); + const check = await checkFederationHealth(engine); + expect(check.status).toBe('fail'); + expect(check.message).toContain('dead-source'); + expect(check.message).toContain('gbrain sync trigger'); + }); + + test('source with low embed coverage + chunks > 100 → warn', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config, last_sync_at) VALUES ('uncovered', 'uncovered', '{"federated":true}', NOW())`, + ); + // Seed 200 pages with chunks; 10 embedded. + for (let i = 0; i < 200; i++) { + await engine.putPage(`p${i}`, { type: 'note', title: `p${i}`, compiled_truth: `body ${i}` }, { sourceId: 'uncovered' }); + await engine.upsertChunks( + `p${i}`, + [{ + chunk_index: 0, + chunk_text: `chunk ${i}`, + chunk_source: 'compiled_truth', + token_count: 1, + embedding: i < 10 ? new Float32Array(1536) : undefined, + }], + { sourceId: 'uncovered' }, + ); + } + const check = await checkFederationHealth(engine); + expect(check.status).toBe('warn'); + expect(check.message).toContain('uncovered'); + expect(check.message).toContain('embed coverage'); + expect(check.message).toContain('gbrain jobs submit embed-backfill'); + }); + + test('synced + zero pages → ok (vacuous truth, no coverage warn)', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config, last_sync_at) VALUES ('empty', 'empty', '{"federated":true}', NOW())`, + ); + const check = await checkFederationHealth(engine); + expect(check.status).toBe('ok'); + }); +}); diff --git a/test/e2e/phantom-redirect.test.ts b/test/e2e/phantom-redirect.test.ts index 1b306a423..d1795e09a 100644 --- a/test/e2e/phantom-redirect.test.ts +++ b/test/e2e/phantom-redirect.test.ts @@ -20,7 +20,8 @@ import { tmpdir } from 'os'; import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts'; import { withEnv } from '../helpers/with-env.ts'; import { runExtractFacts } from '../../src/core/cycle/extract-facts.ts'; -import { tryAcquireDbLock, SYNC_LOCK_ID } from '../../src/core/db-lock.ts'; +// v0.40: per-source lock id replaces the legacy bare SYNC_LOCK_ID constant. +// Tests below hand-craft the lock row via SQL to simulate contention. const SKIP = !hasDatabase(); const describeMaybe = SKIP ? describe.skip : describe; @@ -41,7 +42,8 @@ beforeEach(async () => { // Per-test cleanup: truncate the tables this suite mutates. await engine.executeRaw('TRUNCATE TABLE facts RESTART IDENTITY CASCADE'); await engine.executeRaw('DELETE FROM pages'); - await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync'`); + // v0.40: phantom redirect now uses per-source lock id (gbrain-sync:). + await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync:default'`); }); function tempBrain(): string { @@ -154,9 +156,10 @@ describeMaybe('phantom-redirect E2E (Postgres)', () => { // would still see the row as held (different "logical" holder via the // pid check, but the TTL is what really gates re-acquire). Override // by inserting a row with a different pid + future TTL. + // v0.40 D16: phantom acquires per-source lock; default source = gbrain-sync:default. await engine.executeRaw( `INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at) - VALUES ('gbrain-sync', 9999, 'simulated-other-host', now(), now() + interval '1 hour') + VALUES ('gbrain-sync:default', 9999, 'simulated-other-host', now(), now() + interval '1 hour') ON CONFLICT (id) DO UPDATE SET holder_pid=9999, ttl_expires_at=now() + interval '1 hour'`, ); @@ -180,7 +183,8 @@ describeMaybe('phantom-redirect E2E (Postgres)', () => { expect(existsSync(join(brainDir, 'alice.md'))).toBe(true); // Cleanup - await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync'`); + // v0.40: phantom redirect now uses per-source lock id (gbrain-sync:). + await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync:default'`); } finally { rmSync(brainDir, { recursive: true, force: true }); } diff --git a/test/embed-backfill-submit.test.ts b/test/embed-backfill-submit.test.ts new file mode 100644 index 000000000..9a19a0d30 --- /dev/null +++ b/test/embed-backfill-submit.test.ts @@ -0,0 +1,174 @@ +/** + * Tests for src/core/embed-backfill-submit.ts (v0.40 D19). + * + * Validates the submission gate layer: + * - Default path: submits with priority 5 + idempotency bucket + * - Cooldown: refuses re-submission inside the window + * - Active-job: refuses while a same-source job is active/waiting + * - 24h spend cap: refuses when accumulated spend >= cap + * - Config overrides honored (per-test cap + cooldown) + * - Override knobs in opts honored (test seam) + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { + submitEmbedBackfill, + COOLDOWN_CONFIG_KEY, + SPEND_CAP_CONFIG_KEY, +} from '../src/core/embed-backfill-submit.ts'; +import { MinionQueue } from '../src/core/minions/queue.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); // 30s — PGLite WASM cold-start + 89 migrations exceeds 5s default + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + // Surgical reset (mirrors test/minions.test.ts) — full TRUNCATE wipes the + // config table's `version` key that MinionQueue.ensureSchema() reads. + await engine.executeRaw('DELETE FROM minion_jobs'); +}); + +describe('submitEmbedBackfill — happy path', () => { + test('submits with priority 5 + idempotency key on a clean source', async () => { + const result = await submitEmbedBackfill(engine, 'default', { reason: 'unit' }); + expect(result.status).toBe('submitted'); + expect(result.jobId).toBeDefined(); + + const queue = new MinionQueue(engine); + const job = await queue.getJob(result.jobId!); + expect(job).not.toBeNull(); + expect(job!.name).toBe('embed-backfill'); + expect(job!.priority).toBe(5); + expect((job!.data as { sourceId: string }).sourceId).toBe('default'); + }); + + test('respects opts.priority override', async () => { + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + priority: -10, + }); + expect(result.status).toBe('submitted'); + const queue = new MinionQueue(engine); + const job = await queue.getJob(result.jobId!); + expect(job!.priority).toBe(-10); + }); +}); + +describe('submitEmbedBackfill — cooldown gate', () => { + test('blocks re-submission while a same-source job is active', async () => { + const queue = new MinionQueue(engine); + // Seed an active job manually + await queue.add('embed-backfill', { sourceId: 'default' }, {}); + await engine.executeRaw( + `UPDATE minion_jobs SET status='active' WHERE name='embed-backfill'`, + ); + + const result = await submitEmbedBackfill(engine, 'default', { reason: 'unit' }); + expect(result.status).toBe('cooldown'); + expect(result.cooldownRemainingSeconds).toBeUndefined(); + }); + + test('blocks re-submission inside the cooldown window after recent finish', async () => { + const queue = new MinionQueue(engine); + const job = await queue.add('embed-backfill', { sourceId: 'default' }, {}); + // Mark completed 1 minute ago + await engine.executeRaw( + `UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '1 minute' WHERE id=$1`, + [job.id], + ); + + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + cooldownMinOverride: 10, // 10min cooldown; 1min elapsed → blocked + }); + expect(result.status).toBe('cooldown'); + expect(result.cooldownRemainingSeconds).toBeGreaterThan(0); + expect(result.cooldownRemainingSeconds).toBeLessThanOrEqual(10 * 60); + }); + + test('allows re-submission after cooldown elapses', async () => { + const queue = new MinionQueue(engine); + const job = await queue.add('embed-backfill', { sourceId: 'default' }, {}); + // Mark completed 11 minutes ago — past the 10-min cooldown + await engine.executeRaw( + `UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '11 minutes' WHERE id=$1`, + [job.id], + ); + + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + cooldownMinOverride: 10, + }); + expect(result.status).toBe('submitted'); + }); + + test('config-overridable cooldown (via embed.backfill_cooldown_min)', async () => { + await engine.setConfig(COOLDOWN_CONFIG_KEY, '60'); // 60min cooldown + const queue = new MinionQueue(engine); + const job = await queue.add('embed-backfill', { sourceId: 'default' }, {}); + await engine.executeRaw( + `UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '30 minutes' WHERE id=$1`, + [job.id], + ); + + const result = await submitEmbedBackfill(engine, 'default', { reason: 'unit' }); + expect(result.status).toBe('cooldown'); + }); +}); + +describe('submitEmbedBackfill — 24h spend cap', () => { + test('refuses when spend24hFn returns >= cap', async () => { + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + spendCapUsdOverride: 25, + spend24hFn: async () => 25, + }); + expect(result.status).toBe('spend_capped'); + expect(result.spend24hUsd).toBe(25); + expect(result.spendCapUsd).toBe(25); + }); + + test('admits when spend24hFn returns < cap', async () => { + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + spendCapUsdOverride: 25, + spend24hFn: async () => 24.99, + }); + expect(result.status).toBe('submitted'); + }); + + test('config-overridable spend cap (via embed.backfill_max_usd_per_source_24h)', async () => { + await engine.setConfig(SPEND_CAP_CONFIG_KEY, '5'); + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + spend24hFn: async () => 5, + }); + expect(result.status).toBe('spend_capped'); + expect(result.spendCapUsd).toBe(5); + }); +}); + +describe('submitEmbedBackfill — source isolation', () => { + test('cooldown is per-source, not global', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}') ON CONFLICT (id) DO NOTHING`, + ); + const queue = new MinionQueue(engine); + // Active job on 'default' + await queue.add('embed-backfill', { sourceId: 'default' }, {}); + await engine.executeRaw(`UPDATE minion_jobs SET status='active' WHERE name='embed-backfill'`); + + // Submit for 'other' — should NOT be blocked + const result = await submitEmbedBackfill(engine, 'other', { reason: 'unit' }); + expect(result.status).toBe('submitted'); + }); +}); diff --git a/test/embed-stale.test.ts b/test/embed-stale.test.ts new file mode 100644 index 000000000..b2f703a5d --- /dev/null +++ b/test/embed-stale.test.ts @@ -0,0 +1,230 @@ +/** + * Tests for src/core/embed-stale.ts (v0.40 D15.2). + * + * Hermetic — uses an injected `embedFn` so no network call lands. Validates: + * - empty stale set → done:true, embedded:0 + * - multi-batch run → embed every stale chunk, advance cursor correctly + * - kill mid-flight (signal.aborted) → aborted:true, partial progress preserved + * - resume from cursor → picks up where prior call left off (DB predicate) + * - per-page embedFn throw → logged + skipped, NOT propagated; chunks stay NULL + * + * Why PGLite: validates the engine.listStaleChunks/getChunks/upsertChunks + * roundtrip the helper depends on, not just the loop control flow. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { embedStaleForSource } from '../src/core/embed-stale.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +/** Seed a page with N stale chunks (no embedding) into the default source. */ +async function seedPageWithStaleChunks(slug: string, chunkCount: number): Promise { + await engine.putPage(slug, { + type: 'note', + title: slug, + compiled_truth: `# ${slug}\n\nseeded`, + }); + const chunks: ChunkInput[] = Array.from({ length: chunkCount }, (_, i) => ({ + chunk_index: i, + chunk_text: `chunk ${i} of ${slug}`, + chunk_source: 'compiled_truth', + token_count: 4, + embedding: undefined, // NULL = stale + })); + await engine.upsertChunks(slug, chunks); +} + +/** Deterministic fake embedder — returns unit-length 1536-dim vectors with + * first dim = text length, so we can assert specific chunks got embedded. */ +function fakeEmbedFn(texts: string[]): Promise { + return Promise.resolve( + texts.map((t) => { + const v = new Float32Array(1536); + v[0] = t.length; + v[1] = 1; + return v; + }), + ); +} + +describe('embedStaleForSource', () => { + test('empty stale set returns done:true with zero embedded', async () => { + const result = await embedStaleForSource(engine, 'default', { + embedFn: fakeEmbedFn, + }); + expect(result).toEqual({ + embedded: 0, + chunksProcessed: 0, + pagesProcessed: 0, + lastCursor: null, + done: true, + aborted: false, + }); + }); + + test('embeds every stale chunk across multiple pages in one call', async () => { + await seedPageWithStaleChunks('a', 5); + await seedPageWithStaleChunks('b', 3); + + const result = await embedStaleForSource(engine, 'default', { + embedFn: fakeEmbedFn, + }); + expect(result.done).toBe(true); + expect(result.aborted).toBe(false); + expect(result.embedded).toBe(8); + expect(result.pagesProcessed).toBe(2); + + // Verify DB: zero stale remaining for default. + const stale = await engine.countStaleChunks({ sourceId: 'default' }); + expect(stale).toBe(0); + }); + + test('respects batchSize for cursor pagination', async () => { + await seedPageWithStaleChunks('a', 3); + await seedPageWithStaleChunks('b', 3); + let batchCount = 0; + const result = await embedStaleForSource(engine, 'default', { + embedFn: fakeEmbedFn, + batchSize: 2, + onProgress: () => { + batchCount++; + }, + }); + expect(result.embedded).toBe(6); + // 2-chunk batches across 6 stale rows = at least 3 progress callbacks. + expect(batchCount).toBeGreaterThanOrEqual(3); + }); + + test('IRON-RULE: aborted mid-flight → aborted:true, partial progress preserved', async () => { + await seedPageWithStaleChunks('a', 4); + await seedPageWithStaleChunks('b', 4); + await seedPageWithStaleChunks('c', 4); + const controller = new AbortController(); + // Batch size 4 = one page per batch. concurrency 1 = serialize keys. + // Abort fires inside embedFn for page 'b', so 'a' lands, 'b' aborts mid-call, + // and the third batch ('c') never starts. + const result = await embedStaleForSource(engine, 'default', { + batchSize: 4, + concurrency: 1, + signal: controller.signal, + embedFn: async (texts) => { + if (texts.some((t) => t.includes(' of b'))) { + controller.abort(); + throw new Error('aborted'); // simulates HTTP abort throw + } + return fakeEmbedFn(texts); + }, + }); + expect(result.aborted).toBe(true); + expect(result.done).toBe(false); + expect(result.embedded).toBe(4); // only 'a' landed + // 'b' and 'c' (8 chunks) remain stale + const stale = await engine.countStaleChunks({ sourceId: 'default' }); + expect(stale).toBe(8); + }); + + test('IRON-RULE: kill + resume — second call picks up via embedding-IS-NULL predicate', async () => { + await seedPageWithStaleChunks('a', 4); + await seedPageWithStaleChunks('b', 4); + + // First call aborts when 'b' is reached + const controller = new AbortController(); + const first = await embedStaleForSource(engine, 'default', { + batchSize: 4, + concurrency: 1, + signal: controller.signal, + embedFn: async (texts) => { + if (texts.some((t) => t.includes(' of b'))) { + controller.abort(); + throw new Error('aborted'); + } + return fakeEmbedFn(texts); + }, + }); + expect(first.aborted).toBe(true); + expect(first.embedded).toBe(4); // 'a' landed + + // Second call with NO cursor — predicate excludes already-embedded chunks + const second = await embedStaleForSource(engine, 'default', { + embedFn: fakeEmbedFn, + }); + expect(second.done).toBe(true); + expect(first.embedded + second.embedded).toBe(8); + + const stale = await engine.countStaleChunks({ sourceId: 'default' }); + expect(stale).toBe(0); + }); + + test('per-page embedFn throw is logged but does NOT propagate', async () => { + await seedPageWithStaleChunks('good', 2); + await seedPageWithStaleChunks('bad', 2); + + let badCount = 0; + const result = await embedStaleForSource(engine, 'default', { + embedFn: async (texts) => { + if (texts.some((t) => t.includes('bad'))) { + badCount++; + throw new Error('intentional embed failure'); + } + return fakeEmbedFn(texts); + }, + }); + + // The helper itself didn't throw + expect(result.done).toBe(true); + expect(badCount).toBe(1); + + // 'good' chunks got embedded; 'bad' chunks stayed NULL + expect(result.embedded).toBe(2); + const stale = await engine.countStaleChunks({ sourceId: 'default' }); + expect(stale).toBe(2); + }); + + test('source-scoped: does not touch other sources', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}'::jsonb) ON CONFLICT (id) DO NOTHING`, + ); + await seedPageWithStaleChunks('a', 3); + await engine.putPage('b', { + type: 'note', + title: 'b', + compiled_truth: '# b\n\nseeded', + }, { sourceId: 'other' }); + await engine.upsertChunks( + 'b', + Array.from({ length: 3 }, (_, i) => ({ + chunk_index: i, + chunk_text: `other ${i}`, + chunk_source: 'compiled_truth', + token_count: 4, + embedding: undefined, + })), + { sourceId: 'other' }, + ); + + const result = await embedStaleForSource(engine, 'default', { + embedFn: fakeEmbedFn, + }); + expect(result.embedded).toBe(3); + + // 'other' source still has 3 stale chunks + const otherStale = await engine.countStaleChunks({ sourceId: 'other' }); + expect(otherStale).toBe(3); + }); +}); diff --git a/test/feature-flags.test.ts b/test/feature-flags.test.ts new file mode 100644 index 000000000..549575f87 --- /dev/null +++ b/test/feature-flags.test.ts @@ -0,0 +1,52 @@ +/** + * Tests for src/core/feature-flags.ts (v0.40 D23). + * + * Pin the default-on posture and the explicit-'false'-disables semantics. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { isFederatedV2Enabled, FEDERATED_V2_CONFIG_KEY } from '../src/core/feature-flags.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +describe('isFederatedV2Enabled', () => { + test('default (key unset) → enabled', async () => { + expect(await isFederatedV2Enabled(engine)).toBe(true); + }); + + test('explicit "false" → disabled', async () => { + await engine.setConfig(FEDERATED_V2_CONFIG_KEY, 'false'); + expect(await isFederatedV2Enabled(engine)).toBe(false); + }); + + test('explicit "true" → enabled', async () => { + await engine.setConfig(FEDERATED_V2_CONFIG_KEY, 'true'); + expect(await isFederatedV2Enabled(engine)).toBe(true); + }); + + test('anything-not-literally-false → enabled (defensive default)', async () => { + for (const v of ['False', 'FALSE', '0', 'off', 'no', '']) { + await engine.setConfig(FEDERATED_V2_CONFIG_KEY, v); + expect(await isFederatedV2Enabled(engine)).toBe(true); + } + }); + + test('config key name is stable', () => { + expect(FEDERATED_V2_CONFIG_KEY).toBe('sync.federated_v2'); + }); +}); diff --git a/test/handlers-embed-backfill.test.ts b/test/handlers-embed-backfill.test.ts new file mode 100644 index 000000000..3c7e4c772 --- /dev/null +++ b/test/handlers-embed-backfill.test.ts @@ -0,0 +1,143 @@ +/** + * Tests for src/core/minions/handlers/embed-backfill.ts (v0.40 D2, D6). + * + * Validates the handler-side contract: + * - Happy path: embeds, returns 'success' with chunk + spend counts + * - D2 lock: second concurrent handler call returns 'already_in_progress' + * - D15.1 finally: lock ALWAYS releases (try/finally even on abort) + * + * Hermetic — uses injected embedFn via the underlying embedStaleForSource + * test seam? No — the handler doesn't expose embedFn passthrough. Instead + * we exercise the handler against a brain with zero stale chunks so no + * actual embed call lands. That gives us a deterministic test of the lock + * + budget + status branches without needing a fake gateway. + * + * The kill-resume contract is covered by test/embed-stale.test.ts at the + * helper layer; the handler just routes through. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { makeEmbedBackfillHandler } from '../src/core/minions/handlers/embed-backfill.ts'; +import { tryAcquireDbLock } from '../src/core/db-lock.ts'; +import type { MinionJobContext } from '../src/core/minions/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + // Clean minion_jobs + lock rows. Preserve config (schema version + flags). + await engine.executeRaw('DELETE FROM minion_jobs'); + await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id LIKE 'gbrain-embed-backfill:%'`); +}); + +/** Build a minimal MinionJobContext for testing. */ +function fakeJob(data: Record): MinionJobContext { + const controller = new AbortController(); + return { + id: 1, + name: 'embed-backfill', + data, + attempts_made: 0, + signal: controller.signal, + shutdownSignal: controller.signal, + updateProgress: async () => {}, + updateTokens: async () => {}, + log: async () => {}, + isActive: async () => true, + readInbox: async () => [], + }; +} + +describe('embed-backfill handler — happy path', () => { + test('zero stale chunks → success with embedded=0', async () => { + const handler = makeEmbedBackfillHandler(engine); + const result = await handler(fakeJob({ sourceId: 'default' })); + expect(result).toMatchObject({ + status: 'success', + sourceId: 'default', + embedded: 0, + chunksProcessed: 0, + pagesProcessed: 0, + }); + }); + + test('throws when sourceId missing', async () => { + const handler = makeEmbedBackfillHandler(engine); + await expect(handler(fakeJob({}))).rejects.toThrow(/sourceId is required/); + }); + + test('throws when sourceId is empty string', async () => { + const handler = makeEmbedBackfillHandler(engine); + await expect(handler(fakeJob({ sourceId: '' }))).rejects.toThrow(/sourceId is required/); + }); +}); + +describe('embed-backfill handler — D2 lock contract', () => { + test('IRON-RULE: second call returns already_in_progress when lock is held', async () => { + // Hold the per-source lock externally + const lock = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60); + expect(lock).not.toBeNull(); + + try { + const handler = makeEmbedBackfillHandler(engine); + const result = await handler(fakeJob({ sourceId: 'default' })); + expect(result).toMatchObject({ + status: 'already_in_progress', + sourceId: 'default', + embedded: 0, + spentUsd: 0, + }); + } finally { + await lock?.release(); + } + }); + + test('different sources do not contend on each other locks', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('other-src', 'other-src', '{"federated":true}') ON CONFLICT (id) DO NOTHING`, + ); + const lockA = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60); + expect(lockA).not.toBeNull(); + try { + // 'other-src' should still succeed + const handler = makeEmbedBackfillHandler(engine); + const result = await handler(fakeJob({ sourceId: 'other-src' })); + expect(result.status).toBe('success'); + } finally { + await lockA?.release(); + } + }); + + test('IRON-RULE: lock is released after handler completes (try/finally)', async () => { + const handler = makeEmbedBackfillHandler(engine); + await handler(fakeJob({ sourceId: 'default' })); + + // After handler returns, the lock row should NOT block a fresh acquire. + const lock = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60); + expect(lock).not.toBeNull(); + await lock?.release(); + }); + + test('IRON-RULE: lock released on throw (sourceId-missing path)', async () => { + const handler = makeEmbedBackfillHandler(engine); + try { + await handler(fakeJob({})); // throws before lock is acquired + } catch { + // expected + } + // Lock was never acquired (throw happened in parseParams pre-lock), + // so the row should be cleanly absent. Verify a fresh acquire works. + const lock = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60); + expect(lock).not.toBeNull(); + await lock?.release(); + }); +}); diff --git a/test/parallel.test.ts b/test/parallel.test.ts new file mode 100644 index 000000000..575deb1a8 --- /dev/null +++ b/test/parallel.test.ts @@ -0,0 +1,83 @@ +/** + * Tests for src/core/parallel.ts (v0.40 Federated Sync v2). + * + * Pins: result order matches input order, one rejection doesn't kill others, + * concurrency cap is a hard ceiling, empty input returns empty, invalid + * concurrency throws TypeError. + */ +import { describe, test, expect } from 'bun:test'; +import { pMapAllSettled } from '../src/core/parallel.ts'; + +describe('pMapAllSettled', () => { + test('returns results in input order', async () => { + const items = [1, 2, 3, 4, 5]; + const results = await pMapAllSettled(items, 2, async (n) => n * 10); + expect(results).toHaveLength(5); + expect(results.map((r) => (r.status === 'fulfilled' ? r.value : null))).toEqual([10, 20, 30, 40, 50]); + }); + + test('one rejection does not kill other items', async () => { + const items = ['a', 'b', 'c']; + const results = await pMapAllSettled(items, 2, async (s) => { + if (s === 'b') throw new Error('intentional'); + return s.toUpperCase(); + }); + expect(results[0]).toEqual({ status: 'fulfilled', value: 'A' }); + expect(results[1].status).toBe('rejected'); + expect(results[1].status === 'rejected' && (results[1].reason as Error).message).toBe('intentional'); + expect(results[2]).toEqual({ status: 'fulfilled', value: 'C' }); + }); + + test('hard ceiling: never exceeds concurrency in flight', async () => { + let inFlight = 0; + let peak = 0; + const items = Array.from({ length: 20 }, (_, i) => i); + await pMapAllSettled(items, 3, async () => { + inFlight++; + peak = Math.max(peak, inFlight); + await new Promise((r) => setTimeout(r, 5)); + inFlight--; + return 'ok'; + }); + expect(peak).toBeLessThanOrEqual(3); + expect(peak).toBeGreaterThan(0); + }); + + test('empty input returns empty array', async () => { + const results = await pMapAllSettled([], 5, async () => 'x'); + expect(results).toEqual([]); + }); + + test('concurrency > items.length runs all in parallel (no semaphore stall)', async () => { + const start = Date.now(); + const items = [1, 2, 3]; + await pMapAllSettled(items, 100, async () => { + await new Promise((r) => setTimeout(r, 50)); + return 'ok'; + }); + const elapsed = Date.now() - start; + // Sequential would be ~150ms; concurrent ~50ms. Allow generous slack. + expect(elapsed).toBeLessThan(140); + }); + + test('throws TypeError on concurrency < 1', async () => { + await expect(pMapAllSettled([1], 0, async (x) => x)).rejects.toThrow(TypeError); + await expect(pMapAllSettled([1], -3, async (x) => x)).rejects.toThrow(TypeError); + }); + + test('throws TypeError on non-integer concurrency', async () => { + await expect(pMapAllSettled([1], 1.5, async (x) => x)).rejects.toThrow(TypeError); + await expect(pMapAllSettled([1], NaN, async (x) => x)).rejects.toThrow(TypeError); + }); + + test('passes index to fn', async () => { + const seen: Array<[string, number]> = []; + await pMapAllSettled(['a', 'b', 'c'], 2, async (item, i) => { + seen.push([item, i]); + return item; + }); + // Sort by item since execution order isn't guaranteed. + seen.sort((a, b) => a[0].localeCompare(b[0])); + expect(seen).toEqual([['a', 0], ['b', 1], ['c', 2]]); + }); +}); diff --git a/test/phantom-redirect-per-source-lock.test.ts b/test/phantom-redirect-per-source-lock.test.ts new file mode 100644 index 000000000..3d5351f32 --- /dev/null +++ b/test/phantom-redirect-per-source-lock.test.ts @@ -0,0 +1,42 @@ +/** + * v0.40 D16 regression test: phantom-redirect uses per-source lock. + * + * Pre-v0.40 phantom-redirect acquired the bare `gbrain-sync` lock, blocking + * every concurrent sync brain-wide. After D16, phantom acquires + * `gbrain-sync:` — same-source sync still serializes; cross-source + * sync proceeds unblocked. + * + * Pure source-text regression guard. The full behavior is covered by + * `test/e2e/phantom-redirect.test.ts` (DATABASE_URL-gated). This file pins + * the lock-id construction so any future drift back to the bare constant + * fails loudly in the fast unit loop. + */ +import { describe, test, expect } from 'bun:test'; +import { readFileSync } from 'node:fs'; +import { syncLockId } from '../src/core/db-lock.ts'; + +const SRC = readFileSync('src/core/cycle/phantom-redirect.ts', 'utf8'); + +describe('phantom-redirect lock contract', () => { + test('IRON-RULE: import line uses syncLockId, not bare SYNC_LOCK_ID', () => { + // syncLockId must be imported; SYNC_LOCK_ID must NOT be (per-source posture). + const importLine = SRC.split('\n').find((l) => + l.includes("from '../db-lock.ts'") && l.includes('import'), + ); + expect(importLine).toBeDefined(); + expect(importLine).toContain('syncLockId'); + expect(importLine).not.toMatch(/\bSYNC_LOCK_ID\b/); + }); + + test('IRON-RULE: acquireLockWithRetry call passes syncLockId(sourceId)', () => { + // Banned: the bare `SYNC_LOCK_ID` constant slipped back in. + // Required: the per-source helper threaded with the active sourceId. + expect(SRC).toMatch(/acquireLockWithRetry\s*\(\s*engine\s*,\s*syncLockId\s*\(\s*sourceId\s*\)\s*\)/); + expect(SRC).not.toMatch(/acquireLockWithRetry\s*\(\s*engine\s*,\s*SYNC_LOCK_ID\s*\)/); + }); + + test('helper sanity: syncLockId returns gbrain-sync: shape', () => { + expect(syncLockId('default')).toBe('gbrain-sync:default'); + expect(syncLockId('zion-brain')).toBe('gbrain-sync:zion-brain'); + }); +}); diff --git a/test/source-config-redact.test.ts b/test/source-config-redact.test.ts new file mode 100644 index 000000000..0d8c79e6f --- /dev/null +++ b/test/source-config-redact.test.ts @@ -0,0 +1,54 @@ +/** + * Tests for src/core/source-config-redact.ts (v0.40 D15.4). + */ +import { describe, test, expect } from 'bun:test'; +import { redactSourceConfig, hasWebhookSecret } from '../src/core/source-config-redact.ts'; + +describe('redactSourceConfig', () => { + test('redacts webhook_secret', () => { + const input = { federated: true, webhook_secret: 'super-secret-key', github_repo: 'a/b' }; + const out = redactSourceConfig(input); + expect(out.webhook_secret).toBe(''); + expect(out.federated).toBe(true); + expect(out.github_repo).toBe('a/b'); + }); + + test('does not mutate input', () => { + const input = { webhook_secret: 'secret' }; + redactSourceConfig(input); + expect(input.webhook_secret).toBe('secret'); + }); + + test('returns empty object for non-object input', () => { + expect(redactSourceConfig(null)).toEqual({}); + expect(redactSourceConfig(undefined)).toEqual({}); + expect(redactSourceConfig('a string')).toEqual({}); + expect(redactSourceConfig(['arr'])).toEqual({}); + }); + + test('preserves nested objects (not deep-redacted)', () => { + const input = { + webhook_secret: 'x', + nested: { allowed: 'value' }, + }; + const out = redactSourceConfig(input); + expect(out.webhook_secret).toBe(''); + expect(out.nested).toEqual({ allowed: 'value' }); + }); +}); + +describe('hasWebhookSecret', () => { + test('true when set + non-empty', () => { + expect(hasWebhookSecret({ webhook_secret: 'x' })).toBe(true); + }); + test('false when empty', () => { + expect(hasWebhookSecret({ webhook_secret: '' })).toBe(false); + }); + test('false when absent', () => { + expect(hasWebhookSecret({})).toBe(false); + }); + test('false on non-string values', () => { + expect(hasWebhookSecret({ webhook_secret: 42 })).toBe(false); + expect(hasWebhookSecret(null)).toBe(false); + }); +}); diff --git a/test/source-health.test.ts b/test/source-health.test.ts new file mode 100644 index 000000000..5f4bc80b4 --- /dev/null +++ b/test/source-health.test.ts @@ -0,0 +1,181 @@ +/** + * Tests for src/core/source-health.ts (v0.40 D12 + D9 + D17). + * + * Validates: + * - computeAllSourceMetrics: batched GROUP BY shape, vacuous truth for zero pages + * - resolvePriorityLabel: high/normal/low, unknown → normal + warn-once + * - isSourceStale: never-synced + lag-exceeded + fresh + missing local_path + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + computeAllSourceMetrics, + resolvePriorityLabel, + resolvePriority, + isSourceStale, + _resetPriorityWarningsForTest, +} from '../src/core/source-health.ts'; +import { loadAllSources } from '../src/core/sources-load.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + // Surgical reset: preserves config table (schema version). + await engine.executeRaw('DELETE FROM minion_jobs'); + await engine.executeRaw('DELETE FROM content_chunks'); + await engine.executeRaw('DELETE FROM pages'); + await engine.executeRaw(`DELETE FROM sources WHERE id != 'default'`); + _resetPriorityWarningsForTest(); +}); + +describe('resolvePriorityLabel', () => { + test('recognized values', () => { + expect(resolvePriorityLabel('s', { priority: 'high' })).toBe('high'); + expect(resolvePriorityLabel('s', { priority: 'normal' })).toBe('normal'); + expect(resolvePriorityLabel('s', { priority: 'low' })).toBe('low'); + }); + test('missing → normal silently', () => { + expect(resolvePriorityLabel('s', {})).toBe('normal'); + expect(resolvePriorityLabel('s', null)).toBe('normal'); + }); + test('unknown values → normal with warn', () => { + // Reroute stderr to capture + const orig = process.stderr.write.bind(process.stderr); + let captured = ''; + process.stderr.write = ((chunk: string | Uint8Array) => { + captured += typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf8'); + return true; + }) as never; + try { + expect(resolvePriorityLabel('zion-brain', { priority: 'urgent' })).toBe('normal'); + expect(captured).toContain('zion-brain'); + expect(captured).toContain('priority'); + expect(captured).toContain('normal'); + } finally { + process.stderr.write = orig; + } + }); + test('warns once per source per process', () => { + const orig = process.stderr.write.bind(process.stderr); + let count = 0; + process.stderr.write = ((chunk: string | Uint8Array) => { + const s = typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf8'); + if (s.includes('invalid config.priority')) count++; + return true; + }) as never; + try { + resolvePriorityLabel('s1', { priority: 'urgent' }); + resolvePriorityLabel('s1', { priority: 'urgent' }); // same source + resolvePriorityLabel('s1', { priority: 42 }); // different bad value, same source + expect(count).toBe(1); + } finally { + process.stderr.write = orig; + } + }); +}); + +describe('resolvePriority (numeric)', () => { + test('maps labels to MinionQueue priority integers', () => { + expect(resolvePriority('s', { priority: 'high' })).toBe(-10); + expect(resolvePriority('s', { priority: 'normal' })).toBe(0); + expect(resolvePriority('s', { priority: 'low' })).toBe(5); + expect(resolvePriority('s', {})).toBe(0); + }); +}); + +describe('isSourceStale', () => { + test('never-synced (last_sync_at null) → true', () => { + const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: null, config: {}, created_at: new Date() }; + expect(isSourceStale(src, 60_000)).toBe(true); + }); + test('no local_path → false (nothing to sync)', () => { + const src = { id: 's', name: 's', local_path: null, last_commit: null, last_sync_at: null, config: {}, created_at: new Date() }; + expect(isSourceStale(src, 60_000)).toBe(false); + }); + test('synced within interval → false', () => { + const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: new Date(Date.now() - 1000), config: {}, created_at: new Date() }; + expect(isSourceStale(src, 60_000)).toBe(false); + }); + test('synced beyond interval → true', () => { + const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: new Date(Date.now() - 120_000), config: {}, created_at: new Date() }; + expect(isSourceStale(src, 60_000)).toBe(true); + }); +}); + +describe('computeAllSourceMetrics', () => { + test('empty input returns empty', async () => { + const result = await computeAllSourceMetrics(engine, []); + expect(result).toEqual([]); + }); + + test('zero-page source → embed_coverage_pct=100 (vacuous truth)', async () => { + const sources = await loadAllSources(engine); + const result = await computeAllSourceMetrics(engine, sources); + const dflt = result.find((m) => m.source_id === 'default')!; + expect(dflt.total_pages).toBe(0); + expect(dflt.total_chunks).toBe(0); + expect(dflt.embed_coverage_pct).toBe(100); + }); + + test('aggregates pages + chunks + embedding coverage per source', async () => { + // Two pages with chunks, half embedded + await engine.putPage('a', { type: 'note', title: 'a', compiled_truth: 'a' }); + await engine.putPage('b', { type: 'note', title: 'b', compiled_truth: 'b' }); + await engine.upsertChunks('a', [ + { chunk_index: 0, chunk_text: 'one', chunk_source: 'compiled_truth', token_count: 1, embedding: new Float32Array(1536) }, + { chunk_index: 1, chunk_text: 'two', chunk_source: 'compiled_truth', token_count: 1, embedding: undefined }, + ]); + await engine.upsertChunks('b', [ + { chunk_index: 0, chunk_text: 'three', chunk_source: 'compiled_truth', token_count: 1, embedding: undefined }, + ]); + + const sources = await loadAllSources(engine); + const result = await computeAllSourceMetrics(engine, sources); + const dflt = result.find((m) => m.source_id === 'default')!; + expect(dflt.total_pages).toBe(2); + expect(dflt.total_chunks).toBe(3); + expect(dflt.embedded_chunks).toBe(1); + // 1/3 = 33.3% + expect(dflt.embed_coverage_pct).toBeCloseTo(33.3, 1); + }); + + test('lag_seconds is null when last_sync_at is null', async () => { + const sources = await loadAllSources(engine); + const result = await computeAllSourceMetrics(engine, sources); + const dflt = result.find((m) => m.source_id === 'default')!; + expect(dflt.lag_seconds).toBeNull(); + }); + + test('multi-source isolation: each source gets its own counts', async () => { + await engine.executeRaw(`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}') ON CONFLICT (id) DO NOTHING`); + await engine.putPage('a', { type: 'note', title: 'a', compiled_truth: 'a' }); + await engine.putPage('b', { type: 'note', title: 'b', compiled_truth: 'b' }, { sourceId: 'other' }); + + const sources = await loadAllSources(engine); + const result = await computeAllSourceMetrics(engine, sources); + expect(result.find((m) => m.source_id === 'default')!.total_pages).toBe(1); + expect(result.find((m) => m.source_id === 'other')!.total_pages).toBe(1); + }); + + test('webhook_configured reflects config.webhook_secret presence', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('webhooky', 'webhooky', '{"federated":true,"webhook_secret":"x","github_repo":"a/b"}'::jsonb)`, + ); + const sources = await loadAllSources(engine); + const result = await computeAllSourceMetrics(engine, sources); + const w = result.find((m) => m.source_id === 'webhooky')!; + expect(w.webhook_configured).toBe(true); + const d = result.find((m) => m.source_id === 'default')!; + expect(d.webhook_configured).toBe(false); + }); +}); diff --git a/test/sources-load.test.ts b/test/sources-load.test.ts new file mode 100644 index 000000000..ad202ba89 --- /dev/null +++ b/test/sources-load.test.ts @@ -0,0 +1,134 @@ +/** + * Tests for src/core/sources-load.ts (v0.40 D7). + * + * Validates the shared loader: ordering, filtering, config parsing, and + * defensive fallback when legacy brains lack the `archived` column. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { + loadAllSources, + fetchSource, + parseSourceConfig, + isSourceFederated, +} from '../src/core/sources-load.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function insertSource(id: string, opts: { federated?: boolean; archived?: boolean; config?: Record } = {}): Promise { + const config = { ...opts.config, federated: opts.federated ?? false }; + await engine.executeRaw( + `INSERT INTO sources (id, name, config, archived) + VALUES ($1, $1, $2::jsonb, $3) + ON CONFLICT (id) DO UPDATE SET config = EXCLUDED.config, archived = EXCLUDED.archived`, + [id, JSON.stringify(config), opts.archived ?? false], + ); +} + +describe('loadAllSources', () => { + test('returns default source first, then alphabetical', async () => { + await insertSource('zebra'); + await insertSource('alpha'); + const rows = await loadAllSources(engine); + expect(rows.map((r) => r.id)).toEqual(['default', 'alpha', 'zebra']); + }); + + test('excludes archived rows by default', async () => { + await insertSource('keep'); + await insertSource('archived-one', { archived: true }); + const rows = await loadAllSources(engine); + expect(rows.map((r) => r.id).sort()).toEqual(['default', 'keep']); + }); + + test('includeArchived: true returns archived rows', async () => { + await insertSource('keep'); + await insertSource('archived-one', { archived: true }); + const rows = await loadAllSources(engine, { includeArchived: true }); + expect(rows.map((r) => r.id).sort()).toEqual(['archived-one', 'default', 'keep']); + }); + + test('federatedOnly: filters to config.federated=true', async () => { + await insertSource('iso', { federated: false }); + await insertSource('fed-a', { federated: true }); + await insertSource('fed-b', { federated: true }); + const rows = await loadAllSources(engine, { federatedOnly: true }); + // default source seeded by resetPgliteState has federated: true + expect(rows.map((r) => r.id).sort()).toEqual(['default', 'fed-a', 'fed-b']); + }); + + test('rows have the full SourceRow projection (no surprise drift)', async () => { + await insertSource('shapecheck', { federated: true, config: { tracked_branch: 'main' } }); + const rows = await loadAllSources(engine); + const target = rows.find((r) => r.id === 'shapecheck'); + expect(target).toBeDefined(); + expect(target).toMatchObject({ + id: 'shapecheck', + name: 'shapecheck', + local_path: null, + last_commit: null, + last_sync_at: null, + }); + expect(target!.config).toBeDefined(); + expect(target!.created_at).toBeDefined(); + }); +}); + +describe('fetchSource', () => { + test('returns row for known id', async () => { + await insertSource('mybrain', { federated: true }); + const row = await fetchSource(engine, 'mybrain'); + expect(row?.id).toBe('mybrain'); + expect(isSourceFederated(row!.config)).toBe(true); + }); + + test('returns null for unknown id', async () => { + const row = await fetchSource(engine, 'does-not-exist'); + expect(row).toBeNull(); + }); +}); + +describe('parseSourceConfig', () => { + test('passes through plain object', () => { + expect(parseSourceConfig({ federated: true })).toEqual({ federated: true }); + }); + + test('parses JSON string', () => { + expect(parseSourceConfig('{"federated":true}')).toEqual({ federated: true }); + }); + + test('returns empty object on null / undefined / non-object', () => { + expect(parseSourceConfig(null)).toEqual({}); + expect(parseSourceConfig(undefined)).toEqual({}); + expect(parseSourceConfig(42)).toEqual({}); + }); + + test('returns empty object on malformed JSON string', () => { + expect(parseSourceConfig('{')).toEqual({}); + }); +}); + +describe('isSourceFederated', () => { + test('strict-true requirement', () => { + expect(isSourceFederated({ federated: true })).toBe(true); + expect(isSourceFederated({ federated: false })).toBe(false); + expect(isSourceFederated({ federated: 'true' })).toBe(false); // strict boolean + expect(isSourceFederated({ federated: 1 })).toBe(false); + expect(isSourceFederated({})).toBe(false); + expect(isSourceFederated(null)).toBe(false); + }); +}); diff --git a/test/sources-webhook.test.ts b/test/sources-webhook.test.ts new file mode 100644 index 000000000..fdda0ece9 --- /dev/null +++ b/test/sources-webhook.test.ts @@ -0,0 +1,125 @@ +/** + * Tests for /webhooks/github HMAC verification semantics (v0.40 T10). + * + * The handler itself is wired inside serve-http.ts's runServeHttp closure + * (hard to invoke without bringing up the full Express app). This file pins + * the load-bearing primitives: + * + * 1. GitHub HMAC sig format matches `sha256=` via createHmac. + * 2. safeHexEqual is constant-time and rejects mismatches. + * 3. The expected payload schema (repository.full_name + ref) parses. + * 4. Branch ref construction: refs/heads/. + * + * The full HTTP path is covered by test/e2e/webhook-github.test.ts + * (DATABASE_URL-gated; not present in this wave — filed as a follow-up + * E2E pinned by the test plan artifact). + */ +import { describe, test, expect } from 'bun:test'; +import { createHmac } from 'node:crypto'; +import { safeHexEqual } from '../src/core/timing-safe.ts'; + +const GITHUB_SECRET = 'super-secret-webhook-key'; + +/** Build a sha256= HMAC sig the way GitHub does. */ +function githubSig(secret: string, payload: Buffer): string { + return 'sha256=' + createHmac('sha256', secret).update(payload).digest('hex'); +} + +/** Strip the GitHub prefix before constant-time hex compare. Matches the + * webhook handler in serve-http.ts. Buffer.from('sha256=...', 'hex') silently + * truncates at non-hex chars; comparing prefixed strings as hex returns + * true-on-anything. The handler strips the prefix; tests must too. */ +function verifyGithubSig(secret: string, payload: Buffer, headerSig: string): boolean { + const prefix = 'sha256='; + if (!headerSig.startsWith(prefix)) return false; + const expected = createHmac('sha256', secret).update(payload).digest('hex'); + return safeHexEqual(headerSig.slice(prefix.length), expected); +} + +/** Synthetic minimal push payload (real GitHub sends ~100KB; we only read 2 fields). */ +function pushPayload(repo: string, ref: string): Buffer { + return Buffer.from(JSON.stringify({ + ref, + repository: { full_name: repo, name: repo.split('/')[1], owner: { name: repo.split('/')[0] } }, + head_commit: { id: 'abc123' }, + }), 'utf8'); +} + +describe('GitHub HMAC verification', () => { + test('valid signature on untampered payload → verify=true', () => { + const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main'); + const sig = githubSig(GITHUB_SECRET, payload); + expect(verifyGithubSig(GITHUB_SECRET, payload, sig)).toBe(true); + }); + + test('IRON-RULE: rejects when signature header lacks sha256= prefix', () => { + // This was the bug: production used safeHexEqual on prefixed strings. + // Buffer.from('sha256=...', 'hex') silently truncates at non-hex chars, + // so both sides decoded to empty buffer and signature_mismatch never fired. + const payload = pushPayload('owner/repo', 'refs/heads/main'); + const expected = createHmac('sha256', GITHUB_SECRET).update(payload).digest('hex'); + // Without prefix the header is invalid format → reject + expect(verifyGithubSig(GITHUB_SECRET, payload, expected)).toBe(false); + }); + + test('rejects when secret differs', () => { + const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main'); + const goodSig = githubSig(GITHUB_SECRET, payload); + expect(verifyGithubSig('wrong-secret', payload, goodSig)).toBe(false); + }); + + test('rejects when payload differs (single-byte tamper)', () => { + const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main'); + const sig = githubSig(GITHUB_SECRET, payload); + const tampered = Buffer.from(payload); + tampered[10] = tampered[10] ^ 0xff; + expect(verifyGithubSig(GITHUB_SECRET, tampered, sig)).toBe(false); + }); + + test('GitHub sig format is sha256=<64 hex chars>', () => { + const payload = pushPayload('owner/repo', 'refs/heads/main'); + const sig = githubSig(GITHUB_SECRET, payload); + expect(sig).toMatch(/^sha256=[0-9a-f]{64}$/); + }); +}); + +describe('Push payload parsing', () => { + test('extracts repository.full_name + ref', () => { + const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main'); + const parsed = JSON.parse(payload.toString('utf8')); + expect(parsed.repository?.full_name).toBe('Garry-s-List/zion-brain'); + expect(parsed.ref).toBe('refs/heads/main'); + }); + + test('malformed JSON throws (handler returns 400)', () => { + const bad = Buffer.from('{not json'); + expect(() => JSON.parse(bad.toString('utf8'))).toThrow(); + }); + + test('payload missing repository.full_name is detectable', () => { + const partial = Buffer.from(JSON.stringify({ ref: 'refs/heads/main' })); + const parsed = JSON.parse(partial.toString('utf8')); + expect(parsed.repository?.full_name).toBeUndefined(); + }); +}); + +describe('Branch ref construction (D5)', () => { + test('default tracked_branch = main produces refs/heads/main', () => { + const trackedBranch = 'main'; + expect(`refs/heads/${trackedBranch}`).toBe('refs/heads/main'); + }); + + test('non-main branch is exact match', () => { + const trackedBranch: string = 'master'; + expect(`refs/heads/${trackedBranch}`).toBe('refs/heads/master'); + // ref="refs/heads/main" against tracked="master" must NOT match + const incoming: string = 'refs/heads/main'; + expect(incoming === `refs/heads/${trackedBranch}`).toBe(false); + }); + + test('feature-branch push to main-tracking source is rejected by exact-match', () => { + const trackedBranch: string = 'main'; + const pushedRef: string = 'refs/heads/feature/new-stuff'; + expect(pushedRef === `refs/heads/${trackedBranch}`).toBe(false); + }); +}); diff --git a/test/sync-trigger-cli.test.ts b/test/sync-trigger-cli.test.ts new file mode 100644 index 000000000..ff31f7b45 --- /dev/null +++ b/test/sync-trigger-cli.test.ts @@ -0,0 +1,121 @@ +/** + * Tests for `gbrain sync trigger` CLI (v0.40 D18). + * + * Validates the push-trigger entry point: + * - Help text renders + * - --source required (exits 2) + * - --priority invalid (exits 2) + * - Non-existent source errors before submit (exits 1) + * - Successful submit prints job_id=N on stdout + * - Default priority is high (-10) + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runSyncTrigger } from '../src/commands/sync.ts'; +import { MinionQueue } from '../src/core/minions/queue.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await engine.executeRaw('DELETE FROM minion_jobs'); +}); + +/** Capture process.exit and stdout/stderr writes for one runSyncTrigger call. */ +async function capture(args: string[]): Promise<{ + stdout: string; + stderr: string; + exitCode: number | null; +}> { + const origExit = process.exit; + const origLog = console.log; + const origErr = console.error; + let stdout = ''; + let stderr = ''; + let exitCode: number | null = null; + const exitError = new Error('__exit__'); + process.exit = ((code?: number) => { + exitCode = code ?? 0; + throw exitError; + }) as never; + console.log = (...a: unknown[]) => { stdout += a.map(String).join(' ') + '\n'; }; + console.error = (...a: unknown[]) => { stderr += a.map(String).join(' ') + '\n'; }; + try { + await runSyncTrigger(engine, args); + } catch (e) { + if (e !== exitError) throw e; + } finally { + process.exit = origExit; + console.log = origLog; + console.error = origErr; + } + return { stdout, stderr, exitCode }; +} + +describe('runSyncTrigger', () => { + test('--help prints usage and returns', async () => { + const { stdout, exitCode } = await capture(['--help']); + expect(exitCode).toBeNull(); + expect(stdout).toContain('gbrain sync trigger'); + expect(stdout).toContain('--source'); + expect(stdout).toContain('--priority'); + }); + + test('missing --source exits 2 with hint', async () => { + const { stderr, exitCode } = await capture([]); + expect(exitCode).toBe(2); + expect(stderr).toContain('--source is required'); + }); + + test('invalid --priority exits 2', async () => { + const { stderr, exitCode } = await capture(['--source', 'default', '--priority', 'urgent']); + expect(exitCode).toBe(2); + expect(stderr).toContain('Invalid --priority value'); + }); + + test('non-existent source exits 1', async () => { + const { stderr, exitCode } = await capture(['--source', 'does-not-exist']); + expect(exitCode).toBe(1); + expect(stderr).toContain('not found'); + }); + + test('valid trigger submits sync job + prints job_id=N to stdout', async () => { + const { stdout, exitCode } = await capture(['--source', 'default']); + expect(exitCode).toBeNull(); + expect(stdout).toMatch(/^job_id=\d+$/m); + + // Verify a sync job exists with auto_embed_backfill + priority -10 + const queue = new MinionQueue(engine); + const jobs = await queue.getJobs({ name: 'sync', limit: 5 }); + expect(jobs.length).toBe(1); + const job = jobs[0]; + expect(job.priority).toBe(-10); + expect((job.data as { sourceId: string }).sourceId).toBe('default'); + expect((job.data as { auto_embed_backfill: boolean }).auto_embed_backfill).toBe(true); + }); + + test('--priority normal maps to 0', async () => { + const { exitCode } = await capture(['--source', 'default', '--priority', 'normal']); + expect(exitCode).toBeNull(); + const queue = new MinionQueue(engine); + const jobs = await queue.getJobs({ name: 'sync', limit: 5 }); + expect(jobs[0].priority).toBe(0); + }); + + test('--priority low maps to 5', async () => { + const { exitCode } = await capture(['--source', 'default', '--priority', 'low']); + expect(exitCode).toBeNull(); + const queue = new MinionQueue(engine); + const jobs = await queue.getJobs({ name: 'sync', limit: 5 }); + expect(jobs[0].priority).toBe(5); + }); +}); diff --git a/test/timing-safe.test.ts b/test/timing-safe.test.ts new file mode 100644 index 000000000..3894a9339 --- /dev/null +++ b/test/timing-safe.test.ts @@ -0,0 +1,52 @@ +/** + * Tests for src/core/timing-safe.ts (v0.40 D15.5 extraction). + * + * Pure function: behavior tests plus a source-text grep guard that pins the + * extraction so a future drift back to a serve-http-internal closure fails + * loudly. Both the admin-cookie compare AND the webhook HMAC compare MUST + * route through this helper. + */ +import { describe, test, expect } from 'bun:test'; +import { readFileSync } from 'node:fs'; +import { createHash } from 'node:crypto'; +import { safeHexEqual } from '../src/core/timing-safe.ts'; + +describe('safeHexEqual behavior', () => { + test('returns true for identical hex strings', () => { + const hash = createHash('sha256').update('the-quick-brown-fox').digest('hex'); + expect(safeHexEqual(hash, hash)).toBe(true); + }); + + test('returns false for distinct hex strings of the same length', () => { + const a = createHash('sha256').update('apple').digest('hex'); + const b = createHash('sha256').update('banana').digest('hex'); + expect(safeHexEqual(a, b)).toBe(false); + }); + + test('returns false on length mismatch (does NOT throw)', () => { + expect(safeHexEqual('ab', 'abcd')).toBe(false); + expect(safeHexEqual('', 'ab')).toBe(false); + expect(safeHexEqual('abcd', 'ab')).toBe(false); + }); + + test('empty strings compare equal', () => { + expect(safeHexEqual('', '')).toBe(true); + }); + + test('Buffer.from(_, "hex") is case-insensitive — same hex bytes match either case', () => { + // Documents Node's hex-decode behavior: 'abcd' and 'ABCD' decode to the + // same bytes, so safeHexEqual returns true. Callers normalize beforehand + // when they need strict case-equality. + expect(safeHexEqual('abcd', 'ABCD')).toBe(true); + }); +}); + +describe('extraction contract', () => { + test('IRON-RULE: serve-http.ts imports safeHexEqual from timing-safe.ts (not redefined)', () => { + const src = readFileSync('src/commands/serve-http.ts', 'utf8'); + // Must import from the canonical location + expect(src).toMatch(/import\s*\{[^}]*\bsafeHexEqual\b[^}]*\}\s*from\s*['"]\.\.\/core\/timing-safe\.ts['"]/); + // Must NOT redefine the function inside the file + expect(src).not.toMatch(/function\s+safeHexEqual\s*\(/); + }); +});