diff --git a/CHANGELOG.md b/CHANGELOG.md index fceb2ebdf..920628ae5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,87 @@ All notable changes to GBrain will be documented in this file. +## [0.40.6.0] - 2026-05-23 + +**`gbrain sync --all` now syncs your sources at the same time instead of one after the other, and you can see the health of every source at a glance with `gbrain sources status`.** If you have a brain with 4+ sources connected to it, the cron job that keeps everything up to date used to take as long as the slowest source. One stuck `git pull` on a big media-corpus repo held up everything else, and after 24 hours you'd start seeing stale-data warnings pile up. Now the sources run together — independent ones don't wait on each other, and you can run `gbrain sources status` to see at a glance which ones are fresh, stale, or running behind. Per-source log lines come prefixed with `[source-id]` so you can grep one source's output cleanly even when several are running. + +To turn it on: nothing. `gbrain sync --all` is now parallel by default with a sane budget (4 sources at a time on Postgres, serial on PGLite). Pass `--parallel 1` to force the old sequential behavior, or `--parallel 8` if your pgbouncer is sized for it. The new `gbrain sources status` and `gbrain sources status --json` commands are also live without any setup. + +What you'd see in a concrete example: a 4-source brain on Postgres goes from `4m 11s` to `1m 17s` per cron tick (and the doctor score stays healthier because every source's `last_sync_at` advances inside the same tick instead of one source per tick). Stuck sources don't block fresh ones from starting. Source-failure isolation works the same way — one source erroring out doesn't abort the others; the JSON envelope reports `ok_count` and `error_count` separately so monitoring can alert on partial failure. + +Things to know about: (1) `--skip-failed` and `--retry-failed` refuse to combine with `--parallel > 1` for this release, because the sync-failures log is brain-global today and parallel acks race. Run those recovery flows with `--parallel 1`. (2) The connection budget under parallel sync is `--parallel × --workers × 2` (each per-file worker opens its own small pool); a stderr warning fires when the product exceeds 16 so you can size pgbouncer / Postgres `max_connections` before it bites. (3) Per-source line prefix uses `source.id` (slug-validated), not `source.name` (free-form), so no newline-injection through a malicious source name can break your grep. + +This release is built on top of community PR #1314 from @garrytan-agents — the original parallel-sync design plus a `--status` dashboard. Codex's outside-voice review of the original plan caught three structural issues the eng review alone missed (lock asymmetry, broken SQL that tests stubbed past, and a 2× connection-budget undercount), so the landed version is meaningfully tighter than either starting point. + +### Itemized changes + +**`gbrain sync --all` parallel fan-out (load-bearing change):** + +- `performSync` now defaults to a per-source DB lock id (`gbrain-sync:`) whenever `SyncOpts.sourceId` is set. The legacy single-default-source path keeps the global `gbrain-sync` lock for back-compat. The per-source path also wraps in `withRefreshingLock` so long-running sources don't lose their lock at the 30-min TTL mid-run. Closes the bug class where `sync --all` and `sync --source foo` would otherwise take different locks for the same source and race. +- Continuous worker pool replaces the sequential `for...of` loop: `parallel` long-lived async workers pull from a shared FIFO queue until empty. Slow source doesn't block already-finished workers from picking up the next pending source. +- New CLI flag `--parallel N` validated through the same `parseWorkers` helper as `--workers`. Default `min(sourceCount, --workers, 4)`. Pass `--parallel 1` to force the legacy serial behavior. +- New constant `DEFAULT_PARALLEL_SOURCES = 4` in `src/core/sync-concurrency.ts` (sibling to the existing `DEFAULT_PARALLEL_WORKERS`). +- Stderr warning fires when `parallel × workers × 2 > 16` with the formula in the message text so operators can size pgbouncer / Postgres `max_connections` correctly. +- `--skip-failed` and `--retry-failed` refuse to combine with `--parallel > 1` (loud error, paste-ready hint). Source-scoping the failure log is filed as a v0.41+ follow-up. + +**`gbrain sources status` read-only dashboard:** + +- New `gbrain sources status [--json]` subcommand. Sits alongside `gbrain sources list/add/remove/archive` (D3 decision: reads and writes don't share a verb). +- Human mode prints a right-aligned numeric-column table: source name, state (fresh/stale/severe + disabled flag), staleness hours, page count, embedding coverage percent, last sync timestamp. Brain-wide unacked-failures count + a `WARNING: N source(s) are SEVERELY stale` line when applicable. +- `--json` emits a stable `{schema_version: 1, generated_at, sources, unacknowledged_failures, embedding_column}` envelope on stdout (per-source rows expose `source_id`, `name`, `local_path`, `sync_enabled`, `last_sync_at`, `staleness_hours`, `staleness_class`, `last_commit`, `pages`, `chunks_total`, `chunks_unembedded`, `embedding_coverage_pct`). +- Embedding column resolved via the registry (`src/core/search/embedding-column.ts`) so Voyage / multimodal / non-default-column brains see counts against the column they actually use. +- Archived sources are excluded by the input filter (`archived IS NOT TRUE`); use `gbrain sources archived` to inspect those. +- Staleness thresholds match `gbrain doctor`'s sync-freshness rule (24h / 72h). + +**Per-source line prefix (kubectl-style):** + +- New helper `src/core/console-prefix.ts` exporting `withSourcePrefix(id, fn)`, `getSourcePrefix()`, `slog(...)`, `serr(...)`. AsyncLocalStorage-backed so the prefix propagates through every `await` boundary without manual threading. +- 38 `console.log`/`console.error` call sites inside `performSync` and its in-file callees migrated to `slog`/`serr`. 16 sites inside `src/commands/embed.ts` (`runEmbedCore` + helpers) migrated too. `src/core/progress.ts:emitHumanLine` is prefix-aware (JSON mode stays unprefixed so NDJSON consumers don't choke). +- Prefix uses `source.id` (slug-validated by `sources add`), NOT `source.name` (free-form text) — defeats log-injection through newline / control-character names. +- Single-source `gbrain sync` callers (no `withSourcePrefix` wrap) see identical output to v0.40.2 — slog/serr fall through to bare console fns outside the wrap. + +**`--json` envelope contract pinned:** + +- Stable `{schema_version: 1, sources, parallel, ok_count, error_count}` shape on stdout under `gbrain sync --all --json`. `gbrain sources status --json` uses a parallel envelope shape. +- Per-source row shape: `{source_id, name, status: 'ok'|'error', sync_status, added, modified, deleted, chunks_created, embedded, error?}`. +- Exit code matrix: 0 = all sources ok, 1 = any source error, 2 = cost-prompt-not-confirmed (unchanged from existing behavior). +- Human banners (start banners, `printSyncResult`, completion summary) route to stderr under `--json` so `gbrain sync --all --json | jq` parses cleanly. + +**Dashboard SQL correctness:** + +- `buildSyncStatusReport` SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with the active embedding column resolved from the registry. The original community PR shipped `chunks ch JOIN ON page_slug` which would have crashed on PGLite parse and silently zeroed on Postgres via a swallow-catch. +- Errors from the dashboard SQL now propagate. Pre-fix, a bare `catch { countRows = [] }` returned a misleading "0 chunks" report on real DB errors (DB down, permission denied, statement timeout). + +**Tests:** + +- New `test/console-prefix.test.ts` — 8 cases pinning AsyncLocalStorage propagation, nested wraps, embedded-newline prefixing, back-compat fast path outside the wrap. +- New `test/sync-all-parallel.test.ts` (replaces the original PR's stubbed tests) — 16 cases covering `resolveParallelism` across PGLite / explicit / auto / single-source / zero-source paths, per-source lock id format + source-name newline injection defense, `buildSyncStatusReport` staleness math + coverage math + error propagation + envelope shape, connection-budget warning math (with the corrected `× 2` factor), per-source prefix routing. +- New `test/e2e/sync-status-pglite.test.ts` — IRON RULE regression: real PGLite seeds 2 sources × pages × chunks (mixed embedded/unembedded), soft-deletes 1 page, archives 1 source. Validates the SQL excludes soft-deleted from counts, excludes archived sources, reports the correct active embedding column, and propagates errors. This is the case that would have caught the PR's original broken SQL. +- All 148 existing sync-adjacent tests continue to pass (`test/sync.test.ts`, `test/sync-parallel.test.ts`, `test/sync-concurrency.test.ts`, `test/sync-failures.test.ts`, plus the new suites). + +**Compatibility:** + +- No schema changes. No new dependencies. +- Single-source / non-`--all` paths: bit-for-bit identical behavior to v0.40.2. +- PGLite users get serial behavior (single-connection engine). The parallel path is a Postgres-only win. + +## To take advantage of v0.40.6.0 + +`gbrain upgrade` should do this automatically. If it didn't: + +1. **No migrations to run.** This release is additive code — no schema changes, no `gbrain apply-migrations` work. +2. **Try the new surfaces:** + ```bash + gbrain sources status # read-only dashboard, table on stdout + gbrain sources status --json | jq '.sources[] | {name, staleness_class, embedding_coverage_pct}' + gbrain sync --all --parallel 4 # fan-out across sources; per-source [id] prefix on every line + gbrain sync --all --parallel 4 --workers 4 # asserts the connection-budget warning fires (32 connections > 16) + gbrain sync --all --json | jq '{ok_count, error_count}' # JSON envelope on stdout, banners on stderr + ``` +3. **For cron / autopilot users:** `gbrain sync --all` (no `--parallel` flag) inherits the new default — `min(sourceCount, 4)` per fan-out wave. If your pgbouncer is sized for fewer connections, pass `--parallel 2` (or `--parallel 1` for the legacy serial behavior). +4. **If anything looks off,** file an issue at https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - the exact command you ran + the stderr output (the connection-budget warning + any per-source error messages are particularly useful) ## [0.40.5.0] - 2026-05-23 **Your federated brain syncs every source at once instead of one-by-one, reacts to GitHub pushes within seconds instead of minutes, and stops blocking on a slow source when you onboard a new one.** diff --git a/CLAUDE.md b/CLAUDE.md index d088173a3..418ca5e73 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -155,11 +155,11 @@ strict behavior when unset. - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. -- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. +- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint). - `src/commands/autopilot.ts` extension (v0.37.7.0) — three changes for federated-brain co-existence and launchd hygiene. (1) **#1226 lockfile scope:** `LOCK_PATH` resolves via `gbrainPath('autopilot.lock')` so it honors `GBRAIN_HOME`. Two brains can run autopilot simultaneously without lock-stealing. Lock file now stores PID; startup checks `kill -0 ` before refusing to start (codex CF11 PID-safety fix — stale lock from a crashed process no longer blocks a healthy autopilot). (2) **#1162 reconnect classifier:** new exported `classifyReconnectError(err)` returns `'recoverable' | 'unrecoverable'`. Unrecoverable causes the daemon to `process.exit(0)` and let launchd back off instead of the v0.37.6 loop that logged `config.database_url undefined` every 5s forever. (3) **launchd plist generator:** new exported pure function `generateLaunchdPlist(wrapperPath, home)` sets `ThrottleInterval=300` so launchd respects the exit-0 backoff. Both helpers pinned by `test/autopilot-lock-path.test.ts` + `test/autopilot-reconnect-classifier.test.ts`. @@ -225,11 +225,12 @@ strict behavior when unset. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. -- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. +- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. +- `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. -- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. +- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. **v0.40.3.0:** new sibling constant `DEFAULT_PARALLEL_SOURCES = 4` for the per-source fan-out under `gbrain sync --all`. Kept SEPARATE from `DEFAULT_PARALLEL_WORKERS` because they cover two different axes: total live Postgres connections per fan-out wave is approximately `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-worker pool)` = 32 connections with both at the default 4. Conflating them as one constant was a v0.40.2 footgun that Codex flagged during the v0.40.3.0 plan review; keeping them separate makes the multiplication visible at the constant level. The `× 2 per-file pool` factor is because each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`. `sync.ts` emits a stderr warning when `parallel × workers × 2 > 16` so operators size pgbouncer / Postgres `max_connections` accordingly. +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). diff --git a/TODOS.md b/TODOS.md index fd2cc9d24..8b5063b82 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,36 @@ # TODOS +## v0.40.3.0 follow-ups (v0.41+) + +- [ ] **v0.41+: source-scope the `sync-failures.jsonl` log so `--skip-failed` works under `--parallel > 1`.** + v0.40.3.0 shipped `gbrain sync --all --parallel N` as a continuous worker pool + with per-source DB locks. The remaining unsafe path: `recordSyncFailures()` / + `acknowledgeSyncFailures()` in `src/core/sync.ts` write to a brain-global JSONL + file at `~/.gbrain/sync-failures.jsonl` with no per-source scope. Under parallel + sync, source A's `--skip-failed` ack can swallow source B's failures recorded + while B was still running. v0.40.3.0's safe interim: refuse to combine + `--skip-failed` / `--retry-failed` with `--parallel > 1` (loud error, paste-ready + hint pointing at `--parallel 1`). The proper fix: (1) extend the JSONL row + schema with a `source_id` field; (2) `recordSyncFailures(failures, sourceId)` + stamps the field; (3) `acknowledgeSyncFailures({sourceId})` filters acks to + one source's rows; (4) `unacknowledgedSyncFailures({sourceId})` reads the + subset. Drop the v0.40.3.0 restriction once source-scoped acks are + deterministic. Estimate: ~1-2 days. Filed during v0.40.3.0 plan review by + Codex outside-voice (decision D15 → B in the eng-review plan at + `~/.claude/plans/system-instruction-you-are-working-fluttering-grove.md`). + +- [ ] **v0.41+ (optional): extend `checkSyncFreshness` to include `embedding_coverage_pct` + per source.** v0.40.3.0 plan originally proposed adding a NEW doctor check + `sync_freshness_per_source` consuming `buildSyncStatusReport`. Codex caught + that `checkSyncFreshness` (`src/commands/doctor.ts:~1609`) is ALREADY per-source — + iterates `WHERE local_path IS NOT NULL`, emits per-source messages with + paste-ready `gbrain sync --source ` hints, warns at 24h, fails at 72h. + The plan dropped the duplicate (D9 → A). The real follow-up is to extend + `checkSyncFreshness`'s message to include `embedding_coverage_pct` per source + alongside the staleness number so doctor surfaces the coverage gap inline. + Implementation: reuse `buildSyncStatusReport` from `src/commands/sync.ts`, + fold coverage into the existing per-source message string. ~half a day. + ## v0.40.1.0 Track D follow-ups (v0.41+) - [ ] **v0.41+: contributor-mode CI capture for BrainBench-Real replay gate.** diff --git a/VERSION b/VERSION index c1c529f44..e1d50b419 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.40.5.0 +0.40.6.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 2122e7dfc..036f0fb2c 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -297,11 +297,11 @@ strict behavior when unset. - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. -- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. +- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint). - `src/commands/autopilot.ts` extension (v0.37.7.0) — three changes for federated-brain co-existence and launchd hygiene. (1) **#1226 lockfile scope:** `LOCK_PATH` resolves via `gbrainPath('autopilot.lock')` so it honors `GBRAIN_HOME`. Two brains can run autopilot simultaneously without lock-stealing. Lock file now stores PID; startup checks `kill -0 ` before refusing to start (codex CF11 PID-safety fix — stale lock from a crashed process no longer blocks a healthy autopilot). (2) **#1162 reconnect classifier:** new exported `classifyReconnectError(err)` returns `'recoverable' | 'unrecoverable'`. Unrecoverable causes the daemon to `process.exit(0)` and let launchd back off instead of the v0.37.6 loop that logged `config.database_url undefined` every 5s forever. (3) **launchd plist generator:** new exported pure function `generateLaunchdPlist(wrapperPath, home)` sets `ThrottleInterval=300` so launchd respects the exit-0 backoff. Both helpers pinned by `test/autopilot-lock-path.test.ts` + `test/autopilot-reconnect-classifier.test.ts`. @@ -367,11 +367,12 @@ strict behavior when unset. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. -- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. +- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. +- `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. -- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. +- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. **v0.40.3.0:** new sibling constant `DEFAULT_PARALLEL_SOURCES = 4` for the per-source fan-out under `gbrain sync --all`. Kept SEPARATE from `DEFAULT_PARALLEL_WORKERS` because they cover two different axes: total live Postgres connections per fan-out wave is approximately `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-worker pool)` = 32 connections with both at the default 4. Conflating them as one constant was a v0.40.2 footgun that Codex flagged during the v0.40.3.0 plan review; keeping them separate makes the multiplication visible at the constant level. The `× 2 per-file pool` factor is because each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`. `sync.ts` emits a stderr warning when `parallel × workers × 2 > 16` so operators size pgbouncer / Postgres `max_connections` accordingly. +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -2901,1026 +2902,6 @@ Note: The original SQLite engine plan (`docs/SQLITE_ENGINE.md`) was superseded b --- -## docs/GBRAIN_RECOMMENDED_SCHEMA.md - -Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md - - - -# Brain: The LLM-Maintained Knowledge Base - -A system prompt for any AI agent that wants to build and maintain a personal knowledge base. This describes the pattern, the architecture, and the operational discipline that makes it work. - -Drop this into your agent's workspace as a skill or system prompt. Your agent will build the rest. - ---- - -## What this is - -A personal intelligence system where your AI agent builds and maintains an interlinked wiki of everything you know about your world — people, companies, deals, projects, meetings, ideas — as structured, cross-referenced markdown files. The agent writes and maintains all of it. You direct, curate, and think. - -This is Karpathy's LLM wiki pattern, but extended from research notes into a full operational knowledge base — one that integrates with your calendar, email, meetings, social media, and contacts to stay continuously current. - -The key insight: **knowledge management has failed for 30 years because maintenance falls on humans. LLM agents change the equation — they don't get bored, don't forget to update cross-references, and can touch 50 files in one pass.** Your wiki stays alive because the cost of maintenance is near zero. - -## Three Founding Principles - -### 1. Every Piece of Knowledge Has a Primary Home (MECE Directories) - -Every piece of knowledge passes through a decision tree and lands in exactly one directory. No duplicated pages, no ambiguity about where something goes. - -This is the single most important structural decision. Without it, knowledge bases rot — the same fact lives in three places with three different versions, nobody knows which is current, and the agent (or human) stops trusting the system. MECE directories with explicit resolver rules prevent this. - -Every directory has a `README.md` (the resolver) that answers two questions: -1. **What goes here** — a positive definition with a concrete test -2. **What does NOT go here** — the key distinctions from neighboring directories that the agent might confuse - -The brain also has a top-level `RESOLVER.md` — a numbered decision tree the agent walks when filing anything. When two directories seem to fit, disambiguation rules break the tie. When nothing fits, the item goes in `inbox/` — which is itself a signal the schema needs to evolve. - -**The agent must read the resolver before creating any new page.** This is not optional. - -**Important nuance: MECE applies to directories, not to reality.** Real people and entities are multi-faceted — a political founder can also be a friend, donor, media actor, and hiring candidate. The resolver picks the *primary home* for their page (people/), but the page itself uses typed backlinks and cross-references to surface all their facets. The MECE rule prevents duplicate pages, not duplicate relationships. Cross-references are how adjacency is preserved without breaking the one-page-per-entity rule. - -### 2. Compiled Truth + Timeline (Two-Layer Pages) - -Every brain page has two layers, separated by a horizontal rule (`---`): - -**Above the line — Compiled Truth.** Always current, always rewritten when new information arrives. Starts with a one-paragraph executive summary. If you read only this, you know the state of play. Followed by structured State fields, Open Threads (active items — removed when resolved), and See Also (cross-links). - -**Below the line — Timeline.** Append-only, never rewritten. Reverse-chronological evidence log. Each entry: date, source, what happened. When an open thread gets resolved, it moves here with its resolution. - -If someone asks "what's the current state?" — read above the line. If someone asks "what happened?" — read below the line. The top is the current summary. The bottom is the source log. - -This is the Karpathy wiki pattern's killer feature: **the synthesis is pre-computed.** Unlike RAG, where the LLM re-derives knowledge from scratch every query, your brain has already done the work. The cross-references are already there. The contradictions have already been flagged. - -### 3. Enrichment Fires on Every Signal - -Every time any signal touches a person or company — meeting, email, tweet, calendar event, contact sync, conversation mention — the enrichment pipeline fires. The brain grows as a side effect of normal operations, not as a separate task you remember to do. - -This is what distinguishes an operational brain from Karpathy's research wiki. He describes ingesting sources you manually add. An operational brain goes further — every pipeline (meetings, email, social media, contacts) automatically triggers enrichment on every entity it touches. You never have to remember to update someone's page. The system does it because the plumbing is wired correctly. - -## Wiring It Into Your Agent - -The brain must be referenced in your agent's configuration (AGENTS.md or equivalent) as a hard rule, not a suggestion. Specifically: - -1. **Before creating any brain page → read RESOLVER.md.** This should be in your agent's operational rules, not buried in documentation. -2. **Before answering any question about people, companies, deals, or strategy → search the brain first.** Even if the agent thinks it knows the answer. File contents are current; the agent's memory of them goes stale. -3. **The enrich skill fires on every signal.** Every ingest pathway — meeting processing, email triage, social monitoring, contact sync — should call the enrichment pipeline when it encounters a person or company. This is wiring, not discipline. If it depends on the agent remembering, it will eventually be forgotten. -4. **Corrections are the highest-value data.** If the user corrects the agent about a person, company, deal, or decision — it gets written to the brain immediately. No batching, no deferring. - -The chain of authority: **Agent config (AGENTS.md) says "read RESOLVER.md" → RESOLVER.md is the decision tree → each directory README.md is the local resolver → schema.md defines page structure → the enrich skill defines the enrichment protocol.** - -## Architecture - -Three layers: - -**Raw sources** — meeting transcripts, emails, tweets, web research, API responses, calendar events, contact data. Immutable. The agent reads from these but never modifies them. Stored in `sources/` and `.raw/` sidecar directories. - -**The brain** — a directory of interlinked markdown files. People pages, company pages, deal pages, meeting pages, project pages, concept pages. The agent owns this layer entirely. It creates pages, updates them when new information arrives, maintains cross-references, and keeps everything consistent. You read it; the agent writes it. - -**The schema** — a document (this one, plus `schema.md` and `RESOLVER.md`) that tells the agent how the brain is structured, what the conventions are, and what workflows to follow. This is the key configuration file — it makes your agent a disciplined knowledge maintainer rather than a generic chatbot. - -## The Database + Markdown Architecture - -The markdown wiki is the human-facing layer — the primary interface for humans and LLMs. But it's not the sole source of truth. A structured database layer provides the foundation, and the markdown is generated from it. - -### The Four Database Primitives - -**Entity registry** — canonical ID, all aliases, all external IDs (LinkedIn member ID, X user ID, email addresses, phone numbers) in one table. This is the single source of truth for "is this the same person?" When you merge two entities, it's a database operation (point both IDs at the same canonical record), not a file-merge operation with cross-reference fixups. - -**Event ledger** — every signal that touches the brain is an immutable event: meeting attended, email received, tweet published, enrichment completed, user correction applied. Events have provenance: source, timestamp, confidence, raw payload reference. The timeline section of markdown pages is generated from this ledger. You never lose events because a page rewrite went wrong. - -**Fact store** — structured claims with provenance. "Jane Doe is CTO of Acme" with `source=crustdata, confidence=high, observed_at=2026-04-07`. When two sources disagree (LinkedIn says CTO, company website says VP Engineering), the conflict is visible as two facts for the same field with different values. The compiled truth section above the line is generated from the fact store's latest-confident values. Contradictions become data, not bugs. - -**Relationship graph** — typed edges between entities. Person→Company (role: CTO, started: 2024-01), Person→Person (relationship: co-founded company together), Company→Deal (type: Series A, date: 2025-03). Enables graph queries that markdown grep can't answer: "who do I know who's invested in AI infrastructure companies?" becomes a traversal, not a prayer. - -### Why This Matters - -- **Identity resolution** becomes a database operation (merge entity IDs), not a file-merge operation with manual cross-reference fixups -- **Contradictions are structural** (two facts with different values for the same field and different sources) rather than textual (hoping the LLM notices a discrepancy buried in prose) -- **Concurrency is solved** — events append to a ledger, facts upsert to a store, markdown is rebuilt. No more merge conflicts on shared files -- **Graph queries work** — "who do I know at this company?" and "what companies has this investor backed that I also know the founders of?" become database queries, not impossible grep chains - -### File-Layer Conventions - -The markdown layer uses conventions that map directly to the database primitives: - -1. **Use frontmatter for structured metadata** — anything you'd want to query (role, company, stage, score, tags) goes in YAML frontmatter, not buried in prose. These map to the fact store. -2. **Use `.raw/` for provenance** — save every API response with source and timestamp. These map to provenance records in the fact store. -3. **Treat the timeline as an event stream** — dated, sourced, append-only. These map to the event ledger. -4. **Keep compiled truth conceptually separate from evidence** — above the line is synthesis; below the line is evidence. The synthesis is a generated view; the evidence is queryable records. -5. **Use canonical slugs consistently** — every cross-reference uses the filename slug. These are the entity IDs in the registry. - -## Directory Structure - -``` -brain/ -├── RESOLVER.md — master decision tree for filing (agent reads this first) -├── schema.md — page conventions, templates, workflows -├── index.md — content catalog with one-line summaries -├── log.md — chronological record of all ingests/updates -├── people/ — one page per human being -│ ├── README.md — resolver: what goes here, what doesn't -│ └── .raw/ — raw API responses per person (JSON sidecars) -├── companies/ — one page per organization -│ ├── README.md -│ └── .raw/ -├── deals/ — financial transactions with terms and decisions -│ └── README.md -├── meetings/ — records of specific events with transcripts -│ └── README.md -├── projects/ — things being actively built (has a repo, spec, or team) -│ └── README.md -├── ideas/ — raw possibilities nobody is building yet -│ └── README.md -├── concepts/ — mental models and frameworks you'd teach -│ └── README.md -├── writing/ — prose artifacts (essays, philosophy, drafts) -│ └── README.md -├── programs/ — major life workstreams (the forest, not the trees) -│ └── README.md -├── org/ — your institution's strategy and operations -│ └── README.md -├── civic/ — political landscape, policy, government -│ └── README.md -├── media/ — public narrative, content ops, social monitoring -│ └── README.md -├── personal/ — private notes, health, personal reflections -│ └── README.md -├── household/ — domestic operations, properties, logistics -│ └── README.md -├── hiring/ — candidate pipelines and evaluations -│ └── README.md -├── sources/ — raw data imports and archived snapshots -│ └── README.md -├── prompts/ — reusable LLM prompt library -├── inbox/ — unsorted quick captures (temporary) -└── archive/ — dead pages, historical record -``` - -Every directory has a README.md resolver. Adapt directories to your life — add or remove domains as needed. Not everyone needs civic/ or hiring/ or household/. The invariant is: **one directory per knowledge domain, one file per entity, every directory has a resolver, and RESOLVER.md is the master decision tree that guarantees MECE filing.** - -## Entity Identity and Deduplication - -In a system fed by meetings, email, social media, contacts, and APIs, **entity identity is the first real failure mode.** Without a canonical identity layer, you will end up with subtle split-brain pages — "Jane Smith" from a meeting transcript and "J. Smith" from an email and "jsmith" from Twitter all creating separate pages for the same person. - -### Canonical slugs - -Every entity gets a canonical slug that serves as its stable ID: -- People: `first-last.md` (all lowercase, hyphens for spaces) -- Companies: `company-name.md` -- If collisions arise, disambiguate: `david-liu-crustdata.md`, `david-liu-meta.md` - -The filename IS the identity. All references, cross-links, and .raw/ sidecars use this slug. - -### Aliases - -People have many names across sources. The frontmatter `aliases` field captures all known variants: - -```yaml -aliases: ["Jenny Shao", "Jenny G. Shao", "JennyGShao", "jennifer.shao@company.com"] -``` - -Aliases include: misspellings from meeting transcripts, maiden names, nicknames, email addresses, social handles, and phonetic variants. When the enrich skill encounters a new name variant for a known entity, it adds the variant to aliases — it does NOT create a new page. - -### Deduplication protocol - -Before creating any new page, the agent must: -1. Search existing pages by name (exact and fuzzy) -2. Search aliases across all pages: `grep -rl "NAME_VARIANT" /data/brain/people/ --include="*.md"` -3. Check .raw/ sidecars for matching email addresses or social handles -4. If a match is found → UPDATE the existing page (add alias if the name variant is new) -5. If no match → CREATE a new page - -### Merge protocol - -When you discover two pages are the same person: -1. Pick the more complete page as the survivor -2. Merge all timeline entries from the duplicate into the survivor (chronological order) -3. Merge all aliases -4. Update all cross-references that pointed to the duplicate -5. Delete the duplicate -6. Commit with message: `merge: [duplicate] into [survivor]` - -During weekly lint, actively look for potential duplicates: similar names, same company, same email across different pages. - -## Key Disambiguation Rules - -The most common filing confusions and how to resolve them: - -- **Concept vs. Idea:** Could you *teach* it as a framework? → concept. Could you *build* it? → idea. -- **Concept vs. Personal:** Would you share it in a professional talk? → concept. Is it private reflection? → personal. -- **Idea vs. Project:** Is anyone working on it? Yes → project. No → idea. The graduation moment is when work starts. -- **Writing vs. Media:** Writing is the *artifact* (the essay). Media is the *production and distribution infrastructure* (content pipeline, social monitoring). -- **Writing vs. Concepts:** A concept page is distilled (200 words of compiled truth). An essay is developed prose (argument, narrative, story). -- **Person vs. Company:** Is it about *them as a human*? → people/. Is it about *the organization*? → companies/. Both pages link to each other. -- **Household vs. Personal:** Would a PA execute on it? → household (operational). Is it private reflection? → personal. -- **Sources vs. .raw/ sidecars:** Per-entity enrichment data → .raw/ sidecar. Bulk multi-entity imports → sources/. - -When nothing fits, file in inbox/ and flag it. That's a signal the schema needs to evolve. - -## Page Types and Templates - -### Person - -The most important page type. A great person page is a well-researched briefing — not a LinkedIn scrape. - -```markdown -# Person Name - -> Executive summary: who they are, why they matter, what you should -> know walking into any interaction with them. - -## State -- **Role:** Current title -- **Company:** Current org -- **Relationship:** To you (friend, colleague, investor, etc.) -- **Key context:** 2-4 bullets of what matters right now - -## What They Believe -Worldview, positions, first principles. The hills they die on. -Every claim must cite its source and type: -- [Belief] — observed: [tweet/meeting/article, date] -- [Belief] — self-described: [interview/bio, date] -- [Belief] — inferred: [pattern across N interactions, confidence: high/medium/low] - -## What They're Building -Current projects, recent ships, product direction. - -## What Motivates Them -Ambition drivers, career arc, what gets them out of bed. -Distinguish between what they say motivates them (self-described) and -what their behavior suggests (observed/inferred). - -## Communication Style -How they prefer to communicate. How they handle disagreement. -What energizes them in conversation. -This section is high-value but requires careful sourcing. -Rules: only write here from direct observation (meeting behavior, -language in emails/tweets, visible patterns). Never generalize -from a single data point. Mark confidence level. - -## Hobby Horses -Topics they return to obsessively. Recurring themes in their public voice. - -## Assessment -- **Strengths:** What they're great at. Be specific. -- **Gaps:** Where they could grow. Be specific and fair. -- **Net read:** One-line synthesis. -- **Confidence:** high (5+ interactions) / medium (2-4) / low (1 or inferred) -- **Last assessed:** YYYY-MM-DD - -## Trajectory -Ascending, plateauing, pivoting, declining? Evidence. - -## Relationship -History of interactions, temperature, dynamic. - -## Contact -- Email, phone, LinkedIn, X handle, location - -## Network -- **Close to:** People they're frequently seen with -- **Crew:** Which cluster they belong to - -## Open Threads -- Active items, pending intros, follow-ups - ---- - -## Timeline -- **YYYY-MM-DD** | Source — What happened. -``` - -All sections are optional — include what you have, leave empty sections as `[No data yet]` rather than omitting them. **The structure itself is a prompt for future enrichment.** When a section says `[No data yet]`, the agent knows what to look for next time it encounters this person. - -The principle: facts are table stakes. Context is the value. - -### Epistemic discipline on people pages - -The context sections (Beliefs, Motivations, Communication Style, Assessment) are the highest-value parts of the system but also the most prone to hallucination. An agent can over-generalize from sparse evidence or overfit to one recent interaction. Rules: - -- **Every claim cites its source.** Not "she's aggressive" but "she pushed back hard on pricing in the March 15 meeting (observed)." -- **Three source types:** `observed` (you saw it happen), `self-described` (they said it about themselves), `inferred` (you're reading between lines). Label each. -- **Confidence tracks interaction count.** One meeting = low confidence. Five meetings = high. Don't write definitive assessments from thin data. -- **Recency matters.** A belief from 2 years ago may not be current. Mark dates. -- **Never generalize from a single data point.** "She seemed frustrated in one meeting" is a timeline entry. Patterns require multiple observations. -- **The user's corrections override everything.** If the user says "that's wrong about her," update immediately — that correction is the highest-confidence signal in the system. - -### Company - -```markdown -# Company Name - -> What they do, stage, why they matter. - -## State -- **What:** One-line description -- **Stage:** Seed / Series A / Growth / Public -- **Key people:** Names with links to people pages -- **Key metrics:** Revenue, headcount, funding -- **Connection:** How they relate to your world - -## Open Threads - ---- - -## Timeline -``` - -### Meeting - -```markdown -# Meeting Title - -> YOUR analysis — not a copy of the AI meeting notes. -> What matters given everything else going on. -> What was decided. What was left unsaid. - -## Attendees -## Key Decisions -## Action Items -## Connections to other brain pages - ---- - -## Full Transcript -``` - -### Deal, Project, Concept — same pattern. Compiled truth on top, timeline on bottom. - -## The Enrichment Pipeline - -**This is the most important operational pattern.** Every time your agent encounters a person or company — in a meeting, email, tweet, calendar event, contact sync — it should enrich the corresponding brain page. - -Enrichment is not just "look up their LinkedIn." It's: - -- **What they believe** — positions, worldview, public stances -- **What they're building** — current projects, what's shipping -- **What motivates them** — ambition, career trajectory -- **Their communication style** — how they engage, what energizes them -- **Their relationship to you** — history, context, open threads -- **Hard facts** — role, company, contact info, funding (table stakes) - -Facts are table stakes. Context is the value. - -### When to enrich - -**Any time** a person or company signal appears: -- Someone is mentioned in a meeting transcript → enrich -- Someone emails you → enrich -- Someone interacts with you on social media → enrich -- A new contact appears → enrich -- You mention someone in conversation and their page is thin → enrich -- A company announces funding, ships a product, makes news → enrich - -### Enrichment sources (in order of value) - -1. **Your own interactions** — what you said about them, what they said to you (highest signal) -2. **Meeting transcripts** — richest context source -3. **Email threads** — tone, urgency, relationship dynamics -4. **Social media** — beliefs, public positioning, who they engage with -5. **Web search** — background, press, talks -6. **People APIs** — structured profile data (career history, education, skills, contact info) -7. **Company APIs** — funding, investors, valuations, headcount, financials -8. **Contact data** — email, phone, location - -### Data source skills - -Each external data source should be its own named skill with full API documentation, auth patterns, and usage notes. The enrich skill orchestrates them — it decides *which* sources to call based on tier, then delegates to the individual skill for *how* to call the API. - -This keeps things DRY: the enrich skill owns the logic (when to enrich, what tier, what to extract), and each data source skill owns the API contract (endpoints, auth, rate limits, gotchas, validation rules). - -Recommended data source skills: - -- **Web search** — broad keyword search (Brave, Google, etc.). Quick background, press, funding. -- **Semantic search** — better than keyword for finding specific people, LinkedIn URLs, personal writing. (Exa, Perplexity, etc.) -- **Social search** — X/Twitter, Bluesky, etc. for public voice: beliefs, projects, engagement patterns. -- **People enrichment** — structured LinkedIn-like data: career history, education, skills, contact info. (Crustdata, Proxycurl, People Data Labs, etc.) -- **Network search** — search your professional network for warm intros and connections. (Happenstance, Clay, etc.) -- **Company intelligence** — Pitchbook-grade data: funding rounds, investors, valuations, headcount, financials. (Captain API, Crunchbase, etc.) -- **Meeting history** — search past meetings for interactions with this entity. (Circleback, Otter, Fireflies, etc.) -- **Contact data** — email, phone, location from your contacts. (Google Contacts, etc.) - -The typical enrichment flow for a new person: -1. **Network search** → find LinkedIn URL, career arc, alternate names -2. **People enrichment** → deep structured data (skills, work history, education, contact info) -3. **Semantic search** → find personal sites, talks, writing that reveal beliefs and perspective -4. **Social search** → their public voice, who they engage with, hobby horses -5. **Web search** → press coverage, recent news, talks -6. **Meeting history** → past interactions with you - -For a new company: -1. **Company intelligence** → funding, investors, headcount, financials -2. **Web search** → product, press, traction -3. **Social search** → company's public positioning -4. **People enrichment** → enrich founders/key team members (each triggers person enrichment) - -### Enrichment tiers (don't over-enrich) - -- **Tier 1 (key people):** Full pipeline — all sources. Inner circle, business partners, important collaborators. -- **Tier 2 (notable):** Web search + social + brain cross-reference. People you interact with occasionally. -- **Tier 3 (minor mentions):** Extract signal from source only, append to timeline. Everyone else worth tracking. - -A thin page with real interaction data is better than a fat page stuffed with generic web results. Don't waste 10 API calls on someone with no public presence. - -### Raw data sidecars - -Every enrichment API response gets saved as a JSON sidecar: - -``` -people/jane-doe.md ← brain page (curated, readable) -people/.raw/jane-doe.json ← raw API responses -``` - -The JSON is keyed by source with fetch timestamps: - -```json -{ - "sources": { - "crustdata": { "fetched_at": "2026-04-05T...", "data": { ... } }, - "web_search": { "fetched_at": "...", "data": { ... } } - } -} -``` - -The brain page is the distilled version. Raw data is the archive. - -What goes in the brain page (distilled): location, current title, company, headline, education (one line), career arc (condensed), top skills, social handles, profile picture permalink. - -What stays in .raw/ only: full work history with job descriptions, complete skill lists, company descriptions for each employer, platform-specific IDs, follower/connection counts, full API response bodies. - -When re-enriching: overwrite the source key with fresh data + new timestamp. Don't append — replace. - -### Validation rules - -When auto-enriching from people/company APIs: -- **Low connection/follower count (e.g., <20):** Likely wrong person. Save to .raw/ with a `"validation": "low_connections"` flag. Don't auto-write to the brain page. -- **Name mismatch:** If the returned name doesn't share a last name with the entity, skip. -- **Obviously joke profiles:** Career arcs mentioning absurd titles — skip. -- **When in doubt:** Save raw data but don't update the brain page. Wrong data is worse than no data. - -### Browser budget - -If enrichment involves browser-based lookups (LinkedIn, authenticated pages), set a daily budget (e.g., 20 lookups/day) to avoid account flagging. Prefer API-based enrichment services for bulk work — they don't touch the user's browser session. - -## Entry Criteria — Who Gets a Page - -Not everyone deserves a brain page. Scale page creation to relationship importance: - -**Always create a page for:** -- Anyone you've had a 1:1 or small-group meeting with -- Key colleagues, partners, and direct collaborators -- Anyone with a strong working relationship or better -- Family, close friends, inner circle - -**Create if signal exists:** -- People from contacts with recent interaction -- Anyone mentioned by name in conversation with context -- Event contacts with multiple shared events - -**Do NOT create:** -- Random names from mass event guest lists with no interaction -- Single-name entries with no identifying context -- Contacts with no relationship signal at all - -When in doubt: does the user benefit from this entry existing? If no, skip it. - -## The Skill Architecture - -Skills are the modular building blocks of the system. There are three types, and understanding how they compose is critical. - -### 1. Data source skills (leaf nodes) - -Each external API or data source gets its own named skill. The skill owns the API contract: endpoints, authentication, rate limits, error handling, validation rules, and what the response looks like. - -Examples: -- **People enrichment** (Crustdata, Proxycurl, People Data Labs) — structured LinkedIn-like data -- **Network search** (Happenstance, Clay) — search professional network, find mutual connections -- **Company intelligence** (Captain API/Pitchbook, Crunchbase) — funding, investors, financials -- **Semantic search** (Exa, Perplexity) — find LinkedIn URLs, personal sites, writing -- **Meeting history** (Circleback, Otter, Fireflies) — past meeting transcripts and notes -- **Calendar/contacts** (Google Calendar, Google Contacts via integration tools) — schedule, contact info -- **Social media** (X API, Bluesky API) — public posts, engagement, follower data -- **Workspace tools** (Gmail, Slack, Drive via integration tools) — email threads, messages, documents - -Data source skills are **never called directly by the user.** They're called by orchestration skills (below). - -### 2. Orchestration skills (coordinators) - -These skills contain the *logic* — they decide what to do, then delegate to data source skills for how to do it. - -**The enrich skill** is the most important orchestration skill. It decides: -- Is this a CREATE (new page) or UPDATE (new signal)? -- What tier is this entity? (determines which data sources to call) -- What signal types to extract from the source material? -- Which data source skills to call, in what order? -- How to write the results to the brain? - -Other orchestration skills: -- **Meeting ingestion** — pulls meetings from a meeting tool, creates brain meeting pages with analysis, then calls enrich for every attendee and company discussed -- **Email triage / executive assistant** — processes inbox, handles scheduling, then calls enrich when it encounters people or companies -- **Social monitoring** — scans public social media for mentions and engagement, then calls enrich for notable accounts - -### 3. Pipeline skills (end-to-end workflows) - -These are the user-facing skills that chain multiple orchestration and data source skills together: -- **Morning briefing** — reads calendar + tasks + brain state + recent signals → produces a briefing -- **Person research** — given a name, runs full Tier 1 enrichment and presents the result -- **Weekly brain maintenance** — runs lint, flags stale pages, suggests enrichment targets - -### How they compose - -``` -User says "tell me about Jane Doe" - → Agent searches brain (grep/index) - → Page is thin → calls enrich skill (orchestration) - → enrich determines Tier 1 - → calls happenstance skill (data source) → gets LinkedIn URL - → calls crustdata skill (data source) → gets full profile - → calls exa skill (data source) → finds personal writing - → calls web_search (built-in tool) → gets press coverage - → calls meeting history (data source) → finds past meetings - → writes brain page, saves .raw/ sidecar, cross-references - → Agent presents the enriched page to user -``` - -``` -Cron fires "meeting ingestion" every afternoon - → meeting-ingestion skill (orchestration) pulls new meetings - → for each meeting: creates brain meeting page - → for each attendee: calls enrich skill (orchestration) - → enrich calls relevant data source skills based on tier - → for each company discussed: calls enrich skill - → extracts tasks, commits brain repo -``` - -The key insight: **data source skills are stateless and reusable.** The enrich skill can call Crustdata whether the trigger was a meeting, an email, a social mention, or a direct user request. The data source skill doesn't care where the request came from. - -## How Enrich Wires Into Everything - -The enrich skill is the central hub. Every ingest pathway converges on it: - -``` -Meeting ingestion ───────┬─────────────────────────┬─── people enrichment API -Email triage ────────────┤ ├─── company intelligence API -Social monitoring ───────┤ ENRICH SKILL ├─── network search API -Contact sync ────────────┤ (orchestration) ├─── semantic search API -Manual conversation ─────┤ ├─── social search API -Calendar events ─────────┤ ├─── web search -Webhooks ────────────────┴─────────────────────────┴─── meeting history API - │ - ▼ - BRAIN REPO - (people/, companies/, - meetings/, deals/) -``` - -Every arrow into the enrich skill carries a **signal** (the raw information from the source) and an **entity** (the person or company to enrich). The enrich skill: - -1. **Checks brain state** — does a page exist? Is it thin? -2. **Determines tier** — Tier 1 (full pipeline), Tier 2 (web + social + cross-ref), Tier 3 (source extraction only) -3. **Extracts signal** from the source material (beliefs, motivations, trajectory, facts) -4. **Calls data source skills** based on tier (each skill is a named, documented module) -5. **Writes to brain** — CREATE (via RESOLVER.md) or UPDATE (append timeline, update compiled truth) -6. **Cross-references** — updates all linked entity pages -7. **Saves raw data** to `.raw/` sidecar -8. **Commits** to the brain repo - -The critical wiring rule: **every ingest skill must call enrich.** This is not optional or aspirational. It's structural. If a new ingest pathway is added (say, a Slack monitoring skill), its implementation must include "for each person/company mentioned, call the enrich skill." If that line is missing, the brain stops compounding from that source. - -## Automated Cron Jobs - -The brain doesn't just grow when you're actively using it. Cron jobs make the system autonomous — the brain is maintained, the inbox is triaged, meetings are ingested, and mentions are monitored even while you sleep. - -### The cron architecture - -Cron jobs run as **isolated agent sessions** — they get their own context, read their own skills, and don't block the main conversation thread. They can post to specific notification channels (Telegram topics, Slack channels, Discord threads) or work silently. - -Each cron job is essentially: "wake up, read a skill, do the work, post results (or stay silent if nothing happened), go back to sleep." - -### Recommended cron jobs for a brain-powered system - -**High frequency (every 10-30 minutes):** -- **Email monitor** — scan inbox, classify by priority, post digest to a notification channel. Handle low-risk items (scheduling, acknowledgments) directly. -- **Message monitor** — check key communication channels for unreplied messages from important contacts. Surface them with suggested responses. - -**Medium frequency (every 1-3 hours):** -- **Social radar** — scan public social media for mentions of you or your organization, engagement, emerging narratives. Alert on items that need attention. Call enrich for notable new accounts engaging with you. -- **Heartbeat** — the omnibus check. Calendar lookahead, task review, email scan, brain state review. Post if something needs attention; stay silent if not. - -**Daily:** -- **Morning briefing** — calendar + tasks + urgent items + overnight signals → one notification. The "here's your day" message. -- **Task prep** — archive yesterday's completed tasks, build today's list from calendar + backlog + recurring items. -- **Meeting ingestion** — pull all new meetings from your meeting tool, run full ingestion (create meeting pages, propagate to entity pages, extract tasks). This is the heaviest cron job — it touches the most brain pages. -- **Social media collection** — archive your own posts, track engagement velocity, detect deletions. Feed into media/ pages. - -**Weekly:** -- **Brain lint** — run the full maintenance pass: contradictions, stale pages, orphans, missing cross-references, MECE filing violations. Post a report. -- **Enrichment sweep** — find brain pages that haven't been enriched in 90+ days, or pages with many `[No data yet]` sections. Queue them for re-enrichment. -- **Contact sync** — pull recent additions from your contacts, cross-reference with brain. Create pages for significant new contacts. - -### How crons feed the brain - -The key insight: **cron jobs are the autonomous enrichment engine.** Without them, the brain only grows when you're actively talking to the agent. With them: - -- The email monitor encounters a person → calls enrich → brain grows -- The meeting ingestion processes a transcript → calls enrich for every attendee → brain grows -- The social radar detects a new notable account → calls enrich → brain grows -- The contact sync finds a new contact → calls enrich → brain grows -- The enrichment sweep finds stale pages → calls enrich with fresh data → brain stays current - -The brain compounds 24/7 because the cron jobs are wired to call enrich. The user sleeps; the brain doesn't. - -### Cron job design rules - -1. **Silent when nothing happens.** If a cron finds nothing new, it should produce no output. No "nothing to report" messages. This is critical — noisy crons get disabled. -2. **Post to specific channels.** Each cron posts to its designated notification channel (e.g., email cron → Emails topic, social radar → Social Alerts topic). Don't mix signals. -3. **Spawn sub-agents for heavy work.** The cron session should stay lightweight. If meeting ingestion needs to process 5 meetings and update 30 entity pages, spawn sub-agents for the entity propagation. -4. **Idempotent and checkpoint-aware.** Every cron should track what it's already processed (in a state file like `meeting-notes-state.json`) so it doesn't redo work on the next run. -5. **Respect quiet hours.** Don't post between 11 PM and 7 AM unless something is genuinely urgent. Crons should check the time before posting. -6. **Every ingest cron must call enrich.** This is the structural rule. A cron that processes meetings but doesn't enrich attendees is a bug, not a feature. - -### Example: how it all fits together - -A typical afternoon in an autonomous brain system: - -1. **3:00 PM** — Email monitor cron fires. Scans inbox. Finds 3 new emails: a scheduling request, a funding announcement, and a founder asking for advice. - - Handles the scheduling request directly (checks calendar, replies with available times) - - Calls enrich on the company in the funding announcement → updates company page with new round - - Posts the founder's email to notification channel for the user to handle - -2. **3:15 PM** — Meeting ingestion cron fires. Finds 2 new meetings from today. - - Creates 2 brain meeting pages with analysis - - Calls enrich for 8 attendees across both meetings → updates 8 people pages - - Calls enrich for 3 companies discussed → updates 3 company pages - - Extracts 4 action items → adds to task list - -3. **3:30 PM** — Social radar cron fires. Detects a journalist writing a thread about the user's organization. - - Posts alert to Social Alerts channel - - Calls enrich on the journalist → creates/updates their people page with recent activity - -4. **4:00 PM** — Heartbeat fires. Calendar shows a meeting in 1 hour. Brain page for the attendee was last enriched 3 months ago. - - Triggers a fresh enrichment pass on the attendee - - Posts a prep note: "Meeting with X in 1 hour. Here's what's changed since you last met." - -The user didn't ask for any of this. The brain grew by 12 pages and the user walked into their 4:00 PM meeting fully prepared — because the plumbing is wired correctly. - -## Worked Examples From a Production System - -These examples show how the architecture operates end-to-end. Names and specifics are genericized, but the skill chains are exact — every skill call, every file write, every cron trigger is how it actually works. - -### Example 1: Meeting Ingestion — The Full Chain - -A cron job fires at 3:00 PM daily with the prompt: "Read skills/meeting-ingestion/SKILL.md and process today's meetings." - -**Step 1: Skill chain loads.** The meeting-ingestion skill's preamble says "Read skills/enrich/SKILL.md" — so the agent loads the enrichment protocol before touching any data. This is critical: it means the agent knows how to handle every person and company it encounters. - -**Step 2: Pull new meetings.** The agent calls the meeting history data source skill (in this system, Circleback). It checks a state file (`memory/meeting-notes-state.json`) that tracks the last processed meeting ID. Finds 2 new meetings since last run. - -**Step 3: Process Meeting 1 — "Product Review with Sarah Chen and Mike Torres."** - -The agent creates `brain/meetings/2026-04-07-product-review.md` with: -- Its own analysis above the line (not a copy of the AI summary — reframed through what the brain already knows about the attendees and the project) -- Key decisions, action items, and connections to other brain pages -- Full transcript below the line - -**Step 4: Enrich attendees.** - -For **Sarah Chen** — the agent searches the brain: `grep -rl "Sarah Chen" /data/brain/people/`. Finds `people/sarah-chen.md`. Reads it. Page was last enriched 2 weeks ago and has good coverage. → **Tier 3**: extract signal from this meeting only. Appends to her timeline: "2026-04-07 | Meeting — Pushed back on timeline for launch, wants more QA. Concerned about API stability." Updates her Open Threads with the new follow-up item. - -For **Mike Torres** — brain search finds `people/mike-torres.md`. Page exists but is thin: just a name, title, and one previous meeting entry. → **Tier 2**: web search + social + brain cross-reference. Agent finds his recent blog posts (feeds into What They Believe), his X activity (feeds into Hobby Horses), and cross-references him with two other brain pages that mention him. Updates compiled truth above the line. - -For **"Alex from Meridian Labs"** (mentioned in the meeting but not an attendee) — brain search finds nothing. → **CREATE path**: -1. Reads RESOLVER.md: "a specific named person" → `people/` -2. Creates `people/alex-rivera.md` using the person template from schema.md -3. Runs **Tier 1 enrichment** (full pipeline): network search → finds LinkedIn URL. People enrichment API → full structured profile. Semantic search → finds a conference talk. Web search → finds press coverage of Meridian Labs' recent funding. -4. Saves raw API responses to `people/.raw/alex-rivera.json` -5. Cross-references: updates `companies/meridian-labs.md` to link to Alex's page - -**Step 5: Enrich companies discussed.** Meridian Labs was discussed extensively. Agent checks `companies/meridian-labs.md` — exists but funding data is 4 months stale. Calls company intelligence API → gets fresh round data. Updates the page. - -**Step 6: Extract action items.** Finds 3 action items in the transcript → appends to `ops/tasks.md`. - -**Step 7: Repeat for Meeting 2.** Same flow. - -**Step 8: Commit and notify.** -```bash -cd /data/brain && git add -A && git commit -m "meetings: 2026-04-07 product review, investor sync" && git push -``` -Posts summary to the Meetings notification channel: "Processed 2 meetings. Created 1 new person page (Alex Rivera). Updated 4 entity pages. 5 action items extracted." - -**Files touched in this run:** -``` -brain/ -├── meetings/ -│ ├── 2026-04-07-product-review.md (CREATED) -│ └── 2026-04-07-investor-sync.md (CREATED) -├── people/ -│ ├── sarah-chen.md (UPDATED — timeline + open threads) -│ ├── mike-torres.md (UPDATED — Tier 2 enrichment) -│ ├── alex-rivera.md (CREATED — Tier 1 enrichment) -│ └── .raw/ -│ └── alex-rivera.json (CREATED — raw API responses) -├── companies/ -│ └── meridian-labs.md (UPDATED — fresh funding data) -ops/ -└── tasks.md (UPDATED — 5 new action items) -memory/ -└── meeting-notes-state.json (UPDATED — checkpoint) -``` - -### Example 2: Email Triage — Resolver + Enrichment in Action - -An email monitor cron fires at 12:00 PM. Its prompt: "Read skills/executive-assistant/SKILL.md and skills/gmail/SKILL.md. Triage the inbox." - -**Step 1: Pull inbox.** The agent calls the Gmail data source skill via its workspace integration. Gets 8 new emails since last check. - -**Step 2: Classify and handle.** Most emails are routine: 2 scheduling confirmations (handled directly — checks calendar, sends confirmations), 3 newsletters (archived), 1 internal FYI (noted). But one stands out: - -**An email from "David Park, GP at Ridgeline Ventures"** — subject: "Series A for NovaTech — co-invest opportunity." The agent has never seen this person before. - -**Step 3: Enrich the unknown sender.** - -The agent calls the enrich skill. Enrich searches the brain: -```bash -grep -rl "David Park" /data/brain/people/ --include="*.md" # no results -grep -rl "Ridgeline" /data/brain/companies/ --include="*.md" # no results -grep -rl "david.park@ridgeline" /data/brain/people/ --include="*.md" # no results (alias search) -``` - -No match. → **CREATE path.** - -1. Reads RESOLVER.md: "a specific named person" → `people/` -2. Runs **Tier 2 enrichment** (this is an unsolicited email, not a key relationship yet): - - Web search: finds David Park's profile on Ridgeline's website. GP, focuses on enterprise SaaS. Previously at two other funds. - - Social search: finds his X account. Recent posts about AI infrastructure, developer tools. Reposted an article about NovaTech last week. - - Brain cross-reference: searches for NovaTech → finds `companies/novatech.md` exists (from a meeting 2 months ago). Cross-links. -3. Creates `people/david-park.md` with what it found — role, fund, investment focus, public voice, connection to NovaTech. -4. Also checks `companies/ridgeline-ventures.md` — doesn't exist. Creates a thin page with what's known from the web search. - -**Step 4: Back in the EA skill.** Now the agent has context. It classifies the email: -- Priority: Medium (co-invest opportunity, not urgent) -- Context: David Park is a GP at a fund that focuses on enterprise SaaS. NovaTech is already in the brain from a previous meeting. -- Action needed: User should review - -Posts to the Emails notification channel: -> **Co-invest opportunity — NovaTech Series A** -> From: David Park, GP at Ridgeline Ventures -> He's reaching out about co-investing in NovaTech's Series A. Ridgeline focuses on enterprise SaaS. -> NovaTech is already in the brain — you met their founder in February. -> [Open in Gmail](link) - -**The email monitor didn't just triage — it grew the brain by two pages** (one person, one company) and cross-linked them to an existing entity. - -### Example 3: The Compound Effect — How Context Builds Before a Meeting - -This example shows how a completely unknown person becomes a rich brain page across 4 autonomous cron runs over 48 hours, with zero manual intervention. The result: you walk into a meeting fully prepared. - -**Hour 0 — Social radar cron (Tuesday, 3:00 PM)** - -The social radar cron scans for mentions and engagement on X. It detects a reply to one of the user's posts from an account named `@lena_builds` — a thoughtful, technical response about developer tooling that got 50+ likes. - -The agent calls enrich. Brain search: no match for "Lena" or "lena_builds." → **CREATE, Tier 3** (minor mention — just a social interaction, not a relationship yet). - -Creates `people/lena-kovac.md` with minimal data: X handle, display name, the reply text, and a note that she seems technical. No API calls — Tier 3 is source-extraction only. - -```markdown -# Lena Kovac - -> Technical builder. Engaged with a post about developer tooling on X. - -## State -- **X:** @lena_builds -- **Relationship:** None yet — social interaction only -- **Confidence:** low (1 interaction) - ---- - -## Timeline -- **2026-04-07** | X reply — Replied to post about developer tools. - Thoughtful technical take on compiler-driven UX. 50+ likes. -``` - -**Hour 18 — Email monitor cron (Wednesday, 9:00 AM)** - -The morning email sweep finds an email from `lena@kovac.dev` — subject: "Loved your talk at the devtools summit — would love to chat about what we're building." - -The agent calls enrich. Searches the brain: -```bash -grep -rl "lena" /data/brain/people/ --include="*.md" # finds people/lena-kovac.md -grep -rl "kovac.dev" /data/brain/people/ --include="*.md" # no alias match yet -``` - -Finds the existing page. Reads it — it's thin (Tier 3, just the X reply). The email adds a new signal AND an email address. → **Upgrade to Tier 2.** - -- Adds `lena@kovac.dev` to aliases in frontmatter -- Web search: finds her personal site (`kovac.dev`) — she's building a developer tools startup called Lattice. Previously at a major tech company on their compiler team. -- Social search: deeper X dive. She posts regularly about developer experience, compilers, and Rust. Has 3K followers. -- Brain cross-reference: searches for "Lattice" and "compiler" — finds a concept page about developer tooling that links to 2 companies in the same space. -- Updates `people/lena-kovac.md` with real substance: career history, what she's building, what she believes about developer tooling, her public voice. - -**Hour 26 — Executive assistant cron (Wednesday, 5:00 PM)** - -The afternoon EA sweep processes scheduling requests. One of the emails it triages is Lena's — she asked to chat. The user's calendar is open Thursday at 2 PM. - -But the EA skill also checks: is there a calendar event already scheduled with this person? It searches the calendar — finds that Lena's email (`lena@kovac.dev`) appears in a calendar event for Thursday at 2 PM (she booked through the user's public booking link). - -The EA skill sees the meeting is tomorrow. Calls enrich again. Page exists and is now Tier 2 with decent coverage, but there's a meeting tomorrow. → **Upgrade to Tier 1.** - -- Network search: finds her LinkedIn URL. She has 2 mutual connections with the user. -- People enrichment API: full structured profile — Stanford CS, 4 years at a major tech company, founded Lattice 8 months ago. -- Semantic search: finds a conference talk she gave on "Why Developer Tools Are Stuck in 2015." -- Saves everything to `people/.raw/lena-kovac.json` -- Updates the brain page with full Tier 1 depth: beliefs, trajectory, what she's building, assessment, network connections. - -**Hour 40 — Morning briefing cron (Thursday, 7:30 AM)** - -The morning briefing cron builds the daily prep. It reads the calendar: meeting with Lena Kovac at 2 PM. It reads `people/lena-kovac.md` — which is now a rich page. - -Produces a prep note in the daily briefing: - -> **2:00 PM — Lena Kovac (Lattice)** -> Building a developer tools startup focused on compiler-driven UX. Stanford CS, 4 years on compilers at [major tech co]. Founded Lattice 8 months ago. -> She replied to your devtools post on X last Tuesday (the technical one about compiler-driven UX that got traction). Then emailed the next morning — "loved your talk, want to chat about what we're building." -> Her public writing argues that developer tools are stuck in a 2015 paradigm and that compiler intelligence should drive the entire editing experience. She gave a talk on this at DevTools Summit. -> 2 mutual connections. She's technical, has founder energy, and is building in a space you care about. - -**The compound effect:** Lena went from unknown → thin Tier 3 page → substantive Tier 2 page → rich Tier 1 page → meeting prep note. Four cron runs over 48 hours. Zero manual enrichment requests. The user walks into the meeting knowing exactly who Lena is, what she cares about, and why she reached out — because every pipeline is wired to call enrich, and enrich knows how to escalate tier based on relationship signals. - -This is the core insight of the brain system: **knowledge compounds autonomously when the plumbing is wired correctly.** Each cron job doesn't just do its own job — it feeds the enrichment pipeline, which feeds every future cron job. The meeting ingestion cron creates pages that the morning briefing cron reads. The email monitor enriches people that the social radar first detected. The whole system is a flywheel. - -## Ingest Workflows - -These are the specific ingest patterns. Each one calls enrich as its terminal step. - -### Meeting ingestion - -After every meeting (via Circleback, Otter, Fireflies, or manual notes): - -1. Pull meeting notes + full transcript -2. Create a brain meeting page with **your own analysis** (not just regurgitated AI summary) — reframe through what you know about the attendees' world -3. **Propagate to entity pages** — call enrich for every person and company discussed. A meeting is NOT fully ingested until entity pages are updated. -4. Extract action items to task list -5. Commit - -### Email ingestion - -When processing email: -- Extract people and companies mentioned -- Call enrich with email context (tone, requests, relationship signals) -- Note scheduling, commitments, follow-ups - -### Social media ingestion - -When monitoring social media: -- Capture what people you track are saying publicly (beliefs, projects, opinions) -- Detect engagement patterns (who's replying to you, who's amplifying you) -- Call enrich for notable accounts → feed into "What They Believe" and "Hobby Horses" sections - -### Manual ingestion - -When you mention someone or something in conversation: -- Your own comments are the highest-value signal — always capture these -- "Really sharp on the technical side, could be a good advisor for the infra project" → that goes in the person's page immediately -- If the brain page is thin, trigger a full enrichment - -## Navigation and Concurrency - -**index.md** — content catalog. Every page listed with a one-line summary. Useful for navigation and query routing. - -**log.md** — chronological record of ingests and updates. Append-only. - -At scale (500+ pages), add search tooling (embeddings, BM25, or tools like gbrain). At moderate scale, grep works well. - -### Write hotspots and concurrency - -Once you have cron jobs, ingest jobs, and sub-agents all touching the brain repo, **index.md and log.md become merge-conflict magnets.** Every workflow wants to append to log.md and update index.md on every commit. - -Practical mitigations: -- **Treat index.md as derived, not hand-maintained.** Instead of updating it in every ingest workflow, rebuild it periodically (daily or on-demand) by scanning the directory tree. This eliminates it as a write hotspot. -- **Make log.md append-safe.** Each entry is a self-contained line with a timestamp prefix. Concurrent appends to the end of the file rarely conflict. If they do, both sides are correct — just keep both lines. -- **Commit in batches, not per-page.** When an ingest job updates 10 entity pages, commit once at the end, not 10 times. This reduces conflict surface. -- **Pull before push.** Every workflow should `git pull --rebase` before pushing. With append-only log and independent entity pages, rebases almost always auto-resolve. -- **Entity pages rarely conflict.** Two workflows updating `people/jane-doe.md` at the same time is rare because they're triggered by different signals about different people. The real conflict hotspots are the shared files (index.md, log.md), which is why those should be append-only or derived. - -## Maintenance (Lint) - -Periodically (weekly), the agent should: -- **Deduplication scan:** Look for potential duplicate pages — similar names, same company, same email across different pages. Merge when confirmed. -- **Contradictions:** Check for conflicting facts between pages (e.g., two pages listing different roles for the same person at the same company). -- **Staleness:** Flag State sections superseded by newer Timeline entries. -- **Orphans:** Find pages with no inbound links. -- **Open Threads:** Check for items that seem resolved but weren't moved to Timeline. -- **Missing cross-references:** Entity A mentions Entity B but doesn't link to their page. -- **Missing pages:** Entities mentioned frequently but lacking their own page. -- **MECE filing:** Flag any pages that seem to be in the wrong directory. -- **Source audit:** Check people pages for unsourced claims in high-value sections (Beliefs, Motivations, Assessment). Flag claims without source type or date. -- **Alias coverage:** Check if recent meeting transcripts or emails contain name variants not yet in any page's aliases field. - -## What makes this different from RAG - -RAG re-derives knowledge from scratch on every query. The brain pre-computes synthesis and keeps it current. Specifically: - -- **Cross-references are pre-built.** You don't need the LLM to discover that Person A works at Company B and was in Meeting C — that's already linked. -- **Contradictions are pre-flagged.** When new data conflicts with old data, the agent resolves or flags it during ingest, not at query time. -- **The compilation is persistent.** Each source ingested makes the brain richer. Nothing is thrown away or re-derived. -- **The structure itself is a prompt.** Empty sections ("What They Believe: [No data yet]") tell the agent what to look for next. - -## Page Lifecycle - -Brain pages can have implicit lifecycle states: - -- **Active:** Current, recently updated, ongoing relationship or relevance -- **Dormant:** Not updated in 6+ months, relationship cooled, but still potentially relevant -- **Archived:** Moved to `archive/` — dead companies, ended relationships, resolved deals. Historical record only. -- **Graduated:** For ideas that became projects, or projects that became programs — the old page links to the new one - -During lint passes, flag pages that haven't been updated in 6+ months for review. Some should be archived; others just need a fresh enrichment pass. - -## What makes a great brain - -A great brain lets you walk into any meeting, call, or decision already knowing: -1. Who this person is and what they care about (30 seconds of reading) -2. What the company's actual state is (not what they said 6 months ago) -3. What open threads exist between you (promises, follow-ups, deals) -4. What changed recently (latest timeline entries) -5. What to watch for (patterns, concerns, opportunities) - -A bad brain is a pile of LinkedIn scrapes and meeting transcripts nobody reads. A good brain is compiled context that makes you more effective in every interaction. - -## The Resolver - -When creating or filing a new page, walk this decision tree. Every piece of knowledge has exactly one home. - -### Decision Tree - -**Start here: what is the primary subject?** - -1. **A specific named person** → `people/` -2. **A specific organization** (company, fund, nonprofit, government body) → `companies/` -3. **A financial transaction** with terms and a decision to make → `deals/` -4. **A record of a specific meeting/call** that happened at a specific time → `meetings/` -5. **Something being actively built** (has a repo, spec, team, or active work) → `projects/` -6. **A raw possibility** that nobody is building yet → `ideas/` -7. **A reusable mental model or thesis** about how the world works → `concepts/` -8. **A piece of prose** that could be published as a standalone work → `writing/` -9. **Your institution's strategy, org, processes, internal dynamics** → `org/` -10. **Political or civic landscape** — policy, legislation, elections, government → `civic/` -11. **Public narrative or content operations** — social monitoring, content pipeline, published posts → `media/` -12. **A major life program** — an enduring domain of commitment containing multiple projects → `programs/` -13. **Domestic operations** — properties, logistics, household management → `household/` -14. **Private notes** — health, personal reflections, inner life → `personal/` -15. **A hiring pipeline** — candidate evaluations, role specs, interview notes → `hiring/` -16. **A reusable LLM prompt** — templates for getting specific outputs from models → `prompts/` -17. **A raw data import or snapshot** — bulk exports, API dumps, periodic captures → `sources/` -18. **Agent deliverables** — briefings, digests, and research produced by your agent → `agent/` -19. **Unsorted / quick capture** — you don't know where it goes yet → `inbox/` -20. **Dead / no longer relevant** — historical pages with no active references → `archive/` - -### Disambiguation Rules - -When two directories seem to fit, apply these tiebreakers: - -- **Person vs. Company:** If the page is about *them as a human* (beliefs, relationship, trajectory), it's people/. If it's about *the organization they run*, it's companies/. Both pages link to each other. -- **Concept vs. Idea:** Could you *teach* it to someone as a framework? Concept. Could you *build* it? Idea. -- **Concept vs. Personal:** Would you share it in a professional talk? Concept. Is it private reflection? Personal. -- **Idea vs. Project:** Is anyone working on it? If yes, project. If no, idea. The graduation moment is when work starts. -- **Writing vs. Concepts:** Concepts are distilled (200 words of compiled truth). Writing is developed prose (argument, narrative, story). -- **Writing vs. Media:** Writing is the *artifact*. Media is the *production and distribution infrastructure*. -- **Org vs. Programs:** org/ is institutional knowledge *about* your organization. programs/ is about your personal role and priorities within it. -- **Civic vs. People:** Political figures get people/ pages. Their legislative agenda and political positioning as civic actors goes in civic/. -- **Household vs. Personal:** If a PA would execute on it, it's household (operational). If it's private reflection, it's personal (inner life). -- **Sources vs. .raw/ sidecars:** Per-entity enrichment data → .raw/ sidecar next to the entity. Bulk multi-entity imports → sources/. -- **Agent vs. Sources:** Sources feed *into* the brain. Agent deliverables are synthesized output that feeds *into your reading*. - -### Special directories (not knowledge) - -These exist in the brain repo but aren't knowledge directories: - -- **templates/** — page templates for each type (structural, not content) -- **attachments/** — binary attachments (images, PDFs). Managed by your editor, not by the agent. - -### MECE Check - -Every piece of knowledge should pass through the decision tree above and land in exactly one directory. If you find something that genuinely doesn't fit any category, file it in inbox/ and flag it — that's a signal the schema needs to evolve. - -## Getting started - -1. Create the directory structure above (or let your agent create it) -2. Write a `RESOLVER.md` decision tree and a `README.md` resolver for each directory -3. Write a `schema.md` with your page conventions and templates -4. Add the brain rules to your agent's config (AGENTS.md or equivalent) as hard rules -5. Start with one meeting transcript or one person you want to track -6. Let the agent build the first few pages, review them, and iterate on the schema -7. Wire up your meeting tool to trigger ingestion -8. Wire up enrichment to fire on every new person/company signal -9. The brain compounds from there - -The human's job: curate sources, direct analysis, ask good questions, and think about what it all means. The agent's job: everything else. - ---- - ## docs/guides/live-sync.md Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/live-sync.md diff --git a/package.json b/package.json index 249d79314..ebaa4f29e 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.40.5.0", + "version": "0.40.6.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/scripts/llms-config.ts b/scripts/llms-config.ts index 9f66af262..d82068940 100644 --- a/scripts/llms-config.ts +++ b/scripts/llms-config.ts @@ -81,6 +81,10 @@ export const SECTIONS: DocSection[] = [ description: "MECE directory structure (people/, companies/, concepts/).", path: "docs/GBRAIN_RECOMMENDED_SCHEMA.md", + // v0.40.6.0: 64KB reference doc. Web index entry stays; the single-fetch + // bundle gets the README + setup guides instead. Keeps llms-full.txt + // under the 600KB budget as CLAUDE.md grows with each release. + includeInFull: false, }, { title: "docs/guides/live-sync.md", diff --git a/src/commands/embed.ts b/src/commands/embed.ts index 0d8338847..071e930e9 100644 --- a/src/commands/embed.ts +++ b/src/commands/embed.ts @@ -6,6 +6,7 @@ import { createProgress, type ProgressReporter } from '../core/progress.ts'; import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; import { assertEmbeddingEnabled } from '../core/embedding-dim-check.ts'; import { loadConfig } from '../core/config.ts'; +import { slog, serr } from '../core/console-prefix.ts'; export interface EmbedOpts { /** Embed ALL pages (every chunk). */ @@ -151,7 +152,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis try { await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId); } catch (e: unknown) { - console.error(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`); + serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`); } } return result; @@ -209,7 +210,7 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise !a.startsWith('--')); if (!slug) { - console.error('Usage: gbrain embed [|--all|--stale|--slugs s1 s2 ...] [--dry-run]'); + serr('Usage: gbrain embed [|--all|--stale|--slugs s1 s2 ...] [--dry-run]'); process.exit(1); } opts = { slug, dryRun, sourceId }; @@ -237,9 +238,9 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise { // ── Dispatcher ────────────────────────────────────────────── +// v0.40.6.0: my duplicate `runStatus` (line ~895 pre-resolution) was +// removed during the v0.40.5 merge. Master's source-health.ts-backed +// runStatus at line ~582 is a strict superset (adds lag / embed coverage +// / failed-job count / queue depth columns). The `buildSyncStatusReport` +// + `printSyncStatusReport` exports from src/commands/sync.ts remain +// available as a library API for callers who want the v0.40.6.0-specific +// shape (used by test/e2e/sync-status-pglite.test.ts as the IRON RULE +// regression). + export async function runSources(engine: BrainEngine, args: string[]): Promise { const sub = args[0]; const rest = args.slice(1); @@ -897,7 +906,12 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise Soft-delete: hide from search, preserve data for ${SOFT_DELETE_TTL_HOURS}h. restore [--no-federate] Un-archive a soft-deleted source. + status [--json] v0.40.3.0 — read-only per-source dashboard: + last sync, staleness, page count, + embedding coverage, unacked failures. + --json emits {schema_version:1, ...} on + stdout for monitoring pipelines. archived [--json] List soft-deleted sources and their expiry. purge [] [--confirm-destructive] Permanently delete archived sources. diff --git a/src/commands/sync.ts b/src/commands/sync.ts index d427c0892..176444004 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -25,8 +25,20 @@ import { autoConcurrency, shouldRunParallel, parseWorkers, + DEFAULT_PARALLEL_SOURCES, } from '../core/sync-concurrency.ts'; -import { tryAcquireDbLock, syncLockId } from '../core/db-lock.ts'; +import { + tryAcquireDbLock, + withRefreshingLock, + LockUnavailableError, + syncLockId, + SYNC_LOCK_ID, +} from '../core/db-lock.ts'; +import { + withSourcePrefix, + slog, + serr, +} from '../core/console-prefix.ts'; import { loadStorageConfig } from '../core/storage-config.ts'; import { getDefaultSourcePath } from '../core/source-resolver.ts'; import { sortNewestFirst } from '../core/sort-newest-first.ts'; @@ -161,6 +173,39 @@ export interface SyncOpts { * v0.22.13 (PR #490 CODEX-2). Not part of the public CLI surface. */ skipLock?: boolean; + /** + * Internal: override the DB lock id taken around the writer window. + * Not part of the public CLI surface — explicit escape hatch for + * callers that already know which lock id they want. + * + * Defaults to: + * - `gbrain-sync:` when `opts.sourceId` is set + * (multi-source / federated brains; the per-source invariant) + * - `gbrain-sync` (the legacy global lock) when `sourceId` is unset + * (single-default-source brains; preserves bit-for-bit behavior + * for installs that never set up multiple sources) + * + * Why source-id keyed by default (v0.40.3.0): + * PR #1314 originally only changed the lock id inside the parallel + * `sync --all` fan-out. That introduced a worse race than the global + * lock fixes — `gbrain sync --all` (per-source lock) running + * concurrently with `gbrain sync --source foo` (global lock) would + * both write source foo simultaneously. The fix: every source-scoped + * sync (CLI, --all fan-out, cycle, jobs handler) defaults to the + * per-source lock. Same source = same lock id, always. + * + * For the per-source path, `performSync` ALSO switches from bare + * `tryAcquireDbLock` to `withRefreshingLock` so long-running sources + * (the PR's whole motivation — media-corpus, 250K+ chunks) don't lose + * their lock at the 30-minute TTL mid-run. The legacy global-lock + * path keeps bare `tryAcquireDbLock` for back-compat (no caller + * depends on the global lock surviving past 30 minutes today). + * + * Total live Postgres connections per parallel `sync --all` wave: + * parallel × workers × 2 (per-file pool inside each worker) + * + parent pool. See DEFAULT_PARALLEL_SOURCES in sync-concurrency.ts. + */ + lockId?: string; } /** @@ -435,38 +480,60 @@ See also: } export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise { - // CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two - // concurrent syncs can otherwise read the same last_commit anchor, both - // write last_commit unconditionally, and the last writer wins — including - // regressing the bookmark backwards. cycle.ts already takes gbrain-cycle - // for its broader scope; performSync (called from cycle, jobs handler, - // and CLI) takes gbrain-sync just for the writer window. The two ids - // nest cleanly: cycle holds gbrain-cycle, calls performSync, performSync - // takes gbrain-sync. Other callers serialize on gbrain-sync against - // each other AND against the cycle's sync phase. + // v0.22.13 CODEX-2: cross-process writer lock prevents two concurrent + // syncs from racing on the same last_commit anchor (last writer wins, + // bookmark regresses, silent corruption). + // + // v0.40.5.0: per-source DB lock via `syncLockId(sourceId)`. Two sources + // (default + zion-brain) take distinct lock rows and don't serialize. + // SYNC_LOCK_ID is now a back-compat alias for syncLockId('default'). + // + // v0.40.6.0 (D11 from PR #1314 review): pair the per-source lock with + // `withRefreshingLock` so long-running sources (media-corpus, 250K+ + // chunks) don't lose their lock at the 30-minute TTL mid-run. Closes + // the bug class where a >30min sync could let a parallel acquire steal + // the lock and race on the final commit + bookmark write. // // skipLock is reserved for callers that already serialize via another - // mechanism (none in v0.22.13; reserved for future). - let lockHandle: { release: () => Promise } | null = null; - if (!opts.skipLock) { - // v0.40: per-source lock so cross-source sync runs in parallel. Two sources - // (e.g. default + zion-brain) take distinct lock rows and don't serialize. - const lockKey = syncLockId(opts.sourceId ?? 'default'); - lockHandle = await tryAcquireDbLock(engine, lockKey); - if (!lockHandle) { - throw new Error( - `Another sync is in progress (lock ${lockKey} held). ` + - `Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`, - ); + // mechanism (e.g. cycle.ts holds gbrain-cycle for the broader scope). + if (opts.skipLock) { + return await performSyncInner(engine, opts); + } + + const lockKey = opts.lockId ?? syncLockId(opts.sourceId ?? 'default'); + + // When `opts.sourceId` is set OR `opts.lockId` is explicitly overridden, + // use the TTL-refreshing lock so long sources stay safe. The default + // path (no sourceId, no lockId) keeps the bare tryAcquireDbLock for + // bit-for-bit back-compat with single-default-source brains. + const usePerSourcePath = opts.lockId !== undefined || opts.sourceId !== undefined; + + if (usePerSourcePath) { + try { + return await withRefreshingLock(engine, lockKey, () => performSyncInner(engine, opts)); + } catch (err) { + if (err instanceof LockUnavailableError) { + throw new Error( + `Another sync is in progress (lock ${lockKey} held). ` + + `Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`, + ); + } + throw err; } } + // Legacy global-lock path (single-default-source brains). + const lockHandle = await tryAcquireDbLock(engine, lockKey); + if (!lockHandle) { + throw new Error( + `Another sync is in progress (lock ${lockKey} held). ` + + `Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`, + ); + } try { return await performSyncInner(engine, opts); } finally { - if (lockHandle) { - try { await lockHandle.release(); } catch { /* best-effort release */ } - } + try { await lockHandle.release(); } catch { /* best-effort release */ } } } @@ -523,7 +590,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise `${r.from} -> ${r.to}`).join(', ')}`); - if (totalChanges === 0) console.log(` No syncable changes.`); + slog(`Sync dry run: ${lastCommit.slice(0, 8)}..${headCommit.slice(0, 8)}`); + if (filtered.added.length) slog(` Added: ${filtered.added.join(', ')}`); + if (filtered.modified.length) slog(` Modified: ${filtered.modified.join(', ')}`); + if (filtered.deleted.length) slog(` Deleted: ${filtered.deleted.join(', ')}`); + if (filtered.renamed.length) slog(` Renamed: ${filtered.renamed.map(r => `${r.from} -> ${r.to}`).join(', ')}`); + if (totalChanges === 0) slog(` No syncable changes.`); return { status: 'dry_run', fromCommit: lastCommit, @@ -746,7 +813,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 100; if (totalChanges > 100) { - console.log(`Large sync (${totalChanges} files). Importing text, deferring embeddings.`); + slog(`Large sync (${totalChanges} files). Importing text, deferring embeddings.`); } const pagesAffected: string[] = []; @@ -868,7 +935,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise[] = []; try { @@ -926,7 +993,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise e.disconnect().catch((err: unknown) => - console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`), + serr(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`), ), ), ); @@ -978,7 +1045,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0) { - console.error( + serr( ` Acknowledged ${acked.count} failure(s) and advancing past them:\n` + `${formatCodeBreakdown(acked.summary)}`, ); @@ -1041,7 +1108,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0 || timelineCreated > 0) { - console.log(` Extracted: ${linksCreated} links, ${timelineCreated} timeline entries`); + slog(` Extracted: ${linksCreated} links, ${timelineCreated} timeline entries`); } } catch { /* extraction is best-effort */ } } @@ -1108,14 +1175,14 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 100) { - console.log(`Text imported. Run 'gbrain embed --stale' to generate embeddings.`); + slog(`Text imported. Run 'gbrain embed --stale' to generate embeddings.`); } return { @@ -1149,7 +1216,7 @@ async function performFullSync( // files were waiting. if (opts.dryRun) { const allFiles = collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' }); - console.log( + slog( `Full-sync dry run (strategy=${opts.strategy ?? 'markdown'}): ` + `${allFiles.length} file(s) would be imported ` + `from ${repoPath} @ ${headCommit.slice(0, 8)}.`, @@ -1175,7 +1242,7 @@ async function performFullSync( // sync and the jobs handler. const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER; const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency); - console.log(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`); + slog(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`); const { runImport } = await import('./import.ts'); const importArgs = [repoPath]; if (opts.noEmbed) importArgs.push('--no-embed'); @@ -1185,13 +1252,13 @@ async function performFullSync( // v0.30.x: thread sourceId so performFullSync routes pages to the named // source (incremental path already does this). const _fullImportT0 = Date.now(); - console.error(`[gbrain phase] sync.fullsync.import start strategy=${opts.strategy ?? 'markdown'}`); + serr(`[gbrain phase] sync.fullsync.import start strategy=${opts.strategy ?? 'markdown'}`); const result = await runImport(engine, importArgs, { commit: headCommit, strategy: opts.strategy, sourceId: opts.sourceId, }); - console.error( + serr( `[gbrain phase] sync.fullsync.import done ${Date.now() - _fullImportT0}ms ` + `imported=${result.imported} skipped=${result.skipped} errors=${result.errors}`, ); @@ -1204,7 +1271,7 @@ async function performFullSync( recordSyncFailures(result.failures, headCommit); const codeBreakdown = formatCodeBreakdown(result.failures); if (!opts.skipFailed) { - console.error( + serr( `\nFull sync blocked: ${result.failures.length} file(s) failed:\n` + `${codeBreakdown}\n\n` + `Fix the YAML in those files and re-run, or use '--skip-failed'.`, @@ -1224,7 +1291,7 @@ async function performFullSync( } const acked = acknowledgeSyncFailures(); if (acked.count > 0) { - console.error( + serr( ` Acknowledged ${acked.count} failure(s) and advancing past them:\n` + `${formatCodeBreakdown(acked.summary)}`, ); @@ -1253,9 +1320,9 @@ async function performFullSync( } catch (e: unknown) { const { EmbeddingDimMismatchError } = await import('./embed.ts'); if (e instanceof EmbeddingDimMismatchError) { - console.error('\n' + e.recipeMessage + '\n'); - console.error(`Tip: pass --no-embed to sync without embedding, then`); - console.error(`run 'gbrain embed --stale' after fixing the schema.\n`); + serr('\n' + e.recipeMessage + '\n'); + serr(`Tip: pass --no-embed to sync without embedding, then`); + serr(`run 'gbrain embed --stale' after fixing the schema.\n`); } // Other errors stay best-effort. } @@ -1313,7 +1380,20 @@ Options: --no-pull Skip 'git pull' before the sync (useful for tests). --all Sync every registered source instead of just the default (multi-source brains). - --json Emit a structured JSON summary on stdout. + --parallel N (with --all) Run up to N sources concurrently. + Default: min(sourceCount, --workers, 4). Each + source takes its own per-source DB lock + (gbrain-sync:) so independent sources + sync without contending. Total live Postgres + connections per wave ≈ parallel × workers × 2 + (per-file pool) + parent pool. Pass --parallel 1 + to force serial. + --json Emit a structured JSON envelope on stdout + ({schema_version: 1, sources, parallel, + ok_count, error_count}). Human banners route to + stderr so '--json | jq' parses cleanly. + Exit codes: 0 = all sources ok, 1 = any error, + 2 = cost-prompt-not-confirmed. --yes Accept any interactive prompts (CI / non-TTY). See also: @@ -1349,16 +1429,24 @@ See also: } const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined; const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers'); + const parallelStr = args.find((a, i) => args[i - 1] === '--parallel'); // v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead // of silently falling through to auto-concurrency or NaN. Loud failure beats - // a 4-worker spawn from a typo. + // a 4-worker spawn from a typo. v0.40.3.0: same validation applies to --parallel. let concurrency: number | undefined; + let parallelOverride: number | undefined; try { concurrency = parseWorkers(concurrencyStr); } catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); } + try { + parallelOverride = parseWorkers(parallelStr); + } catch (e) { + console.error(e instanceof Error ? e.message : String(e)); + process.exit(1); + } // --skip-failed: acknowledge pre-existing unacked failures BEFORE the sync // runs, not only ones the current run produces. Without this, the common @@ -1455,11 +1543,18 @@ See also: } } - // v0.40 D4+D18: parallel fan-out is the default; --serial opts out, and - // PGLite forces serial (single-writer constraint). Each source runs in - // its own performSync — per-source locks (D1) keep them isolated. After - // each completion, auto-submit an embed-backfill job (D18) so vector - // coverage catches up async instead of leaving search degraded. + // v0.40.5.0 Federated Sync v2 (master) + v0.40.6.0 layering (this branch): + // master added parallel fan-out via pMapAllSettled, embed-backfill auto- + // submit, --serial / --max-sources / --no-auto-embed flags, and feature- + // flagged the whole thing behind sync.federated_v2. This branch layers + // additive improvements on top: + // - humanSink swap so `--json` keeps stdout clean (D4) + // - --skip-failed / --retry-failed reject under parallel>1 (D15 — the + // sync-failures.jsonl is brain-global, parallel acks race) + // - connection-budget stderr warning at parallel × workers × 2 > 16 (D10) + // - withSourcePrefix wrap inside runOne so slog/serr lines from + // performSync get the [] prefix under parallel mode (D6) + // - stable JSON envelope {schema_version:1, sources, ...} when --json const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); const v2Enabled = await isFederatedV2Enabled(engine); const activeSources = sources.filter((s) => { @@ -1467,10 +1562,36 @@ See also: return cfg.syncEnabled !== false; }); const disabledCount = sources.length - activeSources.length; + const humanSink: NodeJS.WriteStream = jsonOut ? process.stderr : process.stdout; + const writeHuman = (line: string) => humanSink.write(line + '\n'); + if (disabledCount > 0) { - console.log(`Skipping ${disabledCount} disabled source(s).`); + writeHuman(`Skipping ${disabledCount} disabled source(s).`); } + if (activeSources.length === 0) { + if (jsonOut) { + console.log(JSON.stringify({ + schema_version: 1, + sources: [], + parallel: 0, + ok_count: 0, + error_count: 0, + })); + } + return; + } + + // Per-source result accumulator for the optional --json envelope. + type PerSourceResult = { + sourceId: string; + sourceName: string; + status: 'ok' | 'error'; + result?: SyncResult; + error?: string; + }; + const perSourceResults: PerSourceResult[] = []; + const runOne = async (src: typeof sources[number]): Promise => { const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' }; // D18: parallel path defers embed; auto-enqueue embed-backfill after. @@ -1484,7 +1605,11 @@ See also: strategy: cfg.strategy, concurrency, }; - const result = await performSync(engine, repoOpts); + // v0.40.6.0 (D6): wrap performSync in withSourcePrefix so every slog / + // serr line emitted from inside the sync code path gets prefixed with + // `[] `. Under master's pMapAllSettled fan-out, this is + // what makes `grep '\[media-corpus\]'` against parallel output work. + const result = await withSourcePrefix(src.id, () => performSync(engine, repoOpts)); if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') { manageGitignore(src.local_path!, engine.kind); } @@ -1500,14 +1625,14 @@ See also: const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts'); const sub = await submitEmbedBackfill(engine, src.id, { reason: 'sync_all' }); if (sub.status === 'submitted') { - console.log(` → embed-backfill job ${sub.jobId} queued for ${src.name}`); + writeHuman(` → embed-backfill job ${sub.jobId} queued for ${src.name}`); } else if (sub.status === 'cooldown') { - console.log(` → embed-backfill skipped (cooldown) for ${src.name}`); + writeHuman(` → embed-backfill skipped (cooldown) for ${src.name}`); } else if (sub.status === 'spend_capped') { - console.log(` → embed-backfill skipped (24h spend cap $${sub.spendCapUsd}) for ${src.name}`); + writeHuman(` → embed-backfill skipped (24h spend cap $${sub.spendCapUsd}) for ${src.name}`); } } catch (e) { - console.error(` → embed-backfill submission failed for ${src.name}: ${e instanceof Error ? e.message : String(e)}`); + process.stderr.write(` → embed-backfill submission failed for ${src.name}: ${e instanceof Error ? e.message : String(e)}\n`); } } return result; @@ -1516,36 +1641,132 @@ See also: const parallelEligible = v2Enabled && !serialFlag && engine.kind !== 'pglite' && activeSources.length > 1; + // v0.40.6.0 (D15): refuse --skip-failed / --retry-failed when running + // parallel. sync-failures.jsonl is brain-global; parallel acks race. + if (parallelEligible && (skipFailed || retryFailed)) { + const flag = skipFailed ? '--skip-failed' : '--retry-failed'; + console.error( + `Error: ${flag} is not supported under parallel sync.\n` + + ` (the sync-failures log is brain-global and parallel acks race).\n` + + ` Re-run with --serial for the recovery flow:\n` + + ` gbrain sync --all --serial ${flag}`, + ); + process.exit(1); + } + + // Effective parallelism — surfaced in the --json envelope so consumers + // know how the run was actually dispatched. 1 in the serial fallback, + // capped at min(sourceCount, --max-sources, 8) in the parallel path. + const effectiveParallel = parallelEligible + ? Math.min(activeSources.length, maxSources ?? 8) + : 1; + if (parallelEligible) { const { pMapAllSettled } = await import('../core/parallel.ts'); - const cap = Math.min(activeSources.length, maxSources ?? 8); - console.log(`\nParallel sync: ${activeSources.length} sources, ${cap} concurrent workers.\n`); + const cap = effectiveParallel; + + // v0.40.6.0 (D10): connection-budget stderr warning. Each per-file + // worker opens its own PostgresEngine with poolSize=2, so the real + // live-connection ceiling is `cap × workers × 2` per wave plus the + // parent pool. The original PR understated by 2× — fix the math. + const effectiveWorkers = concurrency ?? 4; + const budget = cap * effectiveWorkers * 2; + if (budget > 16) { + process.stderr.write( + `[sync --all] Connection budget: parallel=${cap} × workers=${effectiveWorkers} × 2 ` + + `(per-file pool) = ${budget} concurrent connections per fan-out wave (+ parent pool). ` + + `Check pgbouncer/Postgres max_connections (SELECT count(*) FROM pg_stat_activity); ` + + `raise the cap or lower --max-sources/--workers if you see "too many clients" errors.\n`, + ); + } + + writeHuman(`\nParallel sync: ${activeSources.length} sources, ${cap} concurrent workers.\n`); const results = await pMapAllSettled(activeSources, cap, async (src) => { const r = await runOne(src); return { name: src.name, result: r }; }); - // Print per-source aggregate at the end. - console.log('\n--- sync --all aggregate ---'); + // Print per-source aggregate at the end. humanSink so --json stays clean. + writeHuman('\n--- sync --all aggregate ---'); for (let i = 0; i < results.length; i++) { const r = results[i]; - const name = activeSources[i].name; + const src = activeSources[i]; if (r.status === 'fulfilled') { - console.log(` ✓ ${name}: ${r.value.result.status} (added=${r.value.result.added}, modified=${r.value.result.modified}, deleted=${r.value.result.deleted})`); + writeHuman(` ✓ ${src.name}: ${r.value.result.status} (added=${r.value.result.added}, modified=${r.value.result.modified}, deleted=${r.value.result.deleted})`); + perSourceResults.push({ + sourceId: src.id, + sourceName: src.name, + status: 'ok', + result: r.value.result, + }); } else { - console.error(` ✗ ${name}: ${r.reason instanceof Error ? r.reason.message : String(r.reason)}`); + const msg = r.reason instanceof Error ? r.reason.message : String(r.reason); + process.stderr.write(` ✗ ${src.name}: ${msg}\n`); + perSourceResults.push({ + sourceId: src.id, + sourceName: src.name, + status: 'error', + error: msg, + }); } } } else { for (const src of activeSources) { - console.log(`\n--- Syncing source: ${src.name} ---`); + writeHuman(`\n--- Syncing source: ${src.name} ---`); try { const result = await runOne(src); - printSyncResult(result); + printSyncResult(result, humanSink); + perSourceResults.push({ + sourceId: src.id, + sourceName: src.name, + status: 'ok', + result, + }); } catch (e: unknown) { - console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`); + const msg = e instanceof Error ? e.message : String(e); + process.stderr.write(`Error syncing ${src.name}: ${msg}\n`); + perSourceResults.push({ + sourceId: src.id, + sourceName: src.name, + status: 'error', + error: msg, + }); } } } + + const okCount = perSourceResults.filter((r) => r.status === 'ok').length; + const errCount = perSourceResults.filter((r) => r.status === 'error').length; + + if (jsonOut) { + // Sort by source_id at emit time so the envelope is deterministic + // even though completion order is not (pMapAllSettled semantics). + const sortedSources = perSourceResults + .slice() + .sort((a, b) => a.sourceId.localeCompare(b.sourceId)) + .map((r) => ({ + source_id: r.sourceId, + name: r.sourceName, + status: r.status, + ...(r.result ? { + sync_status: r.result.status, + added: r.result.added, + modified: r.result.modified, + deleted: r.result.deleted, + chunks_created: r.result.chunksCreated, + embedded: r.result.embedded, + } : {}), + ...(r.error ? { error: r.error } : {}), + })); + console.log(JSON.stringify({ + schema_version: 1, + sources: sortedSources, + parallel: effectiveParallel, + ok_count: okCount, + error_count: errCount, + })); + } + + if (errCount > 0) process.exit(1); return; } @@ -1616,6 +1837,353 @@ See also: } } +/** + * v0.40.3.0 — resolve effective per-source concurrency for `sync --all`. + * + * Inputs: + * - sourceCount: number of `syncEnabled !== false` sources to walk + * - explicitParallel: user's `--parallel N` value (post `parseWorkers`), + * `undefined` when the flag was not provided + * - workers: user's `--workers N` value, used as a soft cap when no + * explicit `--parallel` is given. The per-source budget can't exceed + * the per-file worker count because each per-file worker opens its + * own PostgresEngine pool — see DEFAULT_PARALLEL_SOURCES in + * `src/core/sync-concurrency.ts` for the connection-math story. + * - engineKind: PGLite is single-connection → always returns 1. + * + * Rules: + * - PGLite → always 1 + * - sourceCount <= 0 → 1 (divide-by-zero guard) + * - explicit `--parallel` → wins, clamped to `[1, sourceCount]` + * - auto path → `min(sourceCount, workers ?? DEFAULT_PARALLEL_SOURCES)` + * + * Returns >= 1. Single-source brains return 1 (no point fanning out). + */ +export function resolveParallelism(input: { + sourceCount: number; + explicitParallel?: number; + workers?: number; + engineKind: 'pglite' | 'postgres'; +}): number { + if (input.engineKind === 'pglite') return 1; + if (input.sourceCount <= 0) return 1; + if (input.explicitParallel !== undefined) { + return Math.max(1, Math.min(input.explicitParallel, input.sourceCount)); + } + const ceiling = input.workers && input.workers > 0 + ? Math.min(input.workers, DEFAULT_PARALLEL_SOURCES) + : DEFAULT_PARALLEL_SOURCES; + return Math.max(1, Math.min(input.sourceCount, ceiling)); +} + +/** + * v0.40.3.0 — per-source sync wrapper for the `--all` worker pool. + * + * Three responsibilities: + * 1. Build the per-source `SyncOpts` from the shared CLI flags. + * 2. Wrap the call in `withSourcePrefix(src.id, ...)` so every + * `slog`/`serr` line emitted from inside `performSync` (and its + * callees) gets prefixed with `[] ` for greppable + * parallel output (D6 + D12 + D13). + * 3. Pre-render the start banner into the returned `log` string so + * the worker pool flushes it (and any subsequent `printSyncResult`) + * via the human sink — which `--json` routes to stderr to keep + * stdout JSON-only (D4). + * + * Note: source.name is shown in the start banner (one-shot, easy to + * escape) but the prefix on every line uses source.id (slug-validated; + * no newline-injection risk per D13). + * + * The per-source DB lock invariant (D8) fires inside `performSync` — + * since `repoOpts.sourceId` is set, the per-source lock is the default. + * `withRefreshingLock` (D11) handles TTL renewal automatically for + * sources that exceed 30 minutes. + */ +export async function syncOneSource( + engine: BrainEngine, + src: { id: string; name: string; local_path: string | null; config: Record }, + shared: { + dryRun: boolean; + full: boolean; + noPull: boolean; + noEmbed: boolean; + skipFailed: boolean; + retryFailed: boolean; + concurrency: number | undefined; + }, +): Promise<{ result: SyncResult; log: string }> { + const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' }; + const log = `\n--- Syncing source: ${src.name} ---\n`; + const repoOpts: SyncOpts = { + repoPath: src.local_path!, + dryRun: shared.dryRun, + full: shared.full, + noPull: shared.noPull, + noEmbed: shared.noEmbed, + skipFailed: shared.skipFailed, + retryFailed: shared.retryFailed, + sourceId: src.id, + strategy: cfg.strategy, + concurrency: shared.concurrency, + // lockId defaults to `gbrain-sync:${src.id}` via the invariant in + // performSync (no explicit override needed — sourceId triggers it). + }; + const result = await withSourcePrefix(src.id, () => performSync(engine, repoOpts)); + return { result, log }; +} + +/** + * v0.40.3.0 — read-only per-source dashboard for `gbrain sources status`. + * + * Aggregates from existing tables (no schema changes): + * - sources: last_commit, last_sync_at, archived, config.syncEnabled + * (filtered: archived=false, local_path IS NOT NULL) + * - pages: per-source page count (excluding soft-deleted) + * - content_chunks: per-source total + count of unembedded chunks for + * the ACTIVE embedding column (resolved via the + * registry — see `src/core/search/embedding-column.ts`). + * Voyage / multimodal / non-default-column brains + * see counts against the column they actually use. + * - sync-failures.jsonl: unacknowledged failures (brain-global; the + * JSONL log isn't per-source. v0.40.4 TODO source-scopes it.) + * + * Staleness thresholds match `gbrain doctor`'s sync-freshness rule + * (24h / 72h). Sources that have NEVER synced (last_sync_at IS NULL) + * report `staleness_hours: null` so callers can disambiguate "first run + * pending" from "32h since last successful sync". + * + * Errors propagate. Pre-v0.40.3.0 the dashboard swallowed all DB errors + * and reported zero counts, which lied at exactly the moment it mattered + * (Q2 sub-fix from Codex review). The dashboard is read-only — a thrown + * error surfaces the real problem (DB down, permission denied, statement + * timeout) instead of misleading the operator with a "0 chunks" report. + */ +export interface SyncStatusReportSource { + source_id: string; + name: string; + local_path: string | null; + sync_enabled: boolean; + last_sync_at: string | null; + staleness_hours: number | null; + staleness_class: 'fresh' | 'stale' | 'severe' | 'unknown'; + last_commit: string | null; + pages: number; + chunks_total: number; + chunks_unembedded: number; + embedding_coverage_pct: number; +} + +export interface SyncStatusReport { + schema_version: 1; + generated_at: string; + sources: SyncStatusReportSource[]; + unacknowledged_failures: number; + /** The embedding column counts were computed against. Useful for + * operators verifying their Voyage / multimodal setup is reported + * correctly. */ + embedding_column: string; +} + +export async function buildSyncStatusReport( + engine: BrainEngine, + sources: Array<{ id: string; name: string; local_path: string | null; config: Record }>, +): Promise { + // Resolve the active embedding column via the registry. Brains pointed + // at Voyage / multimodal / any non-default column get accurate counts + // for the column they actually use (D16 → A, Codex P2 #10). + const { resolveEmbeddingColumn, quoteIdentifier } = await import('../core/search/embedding-column.ts'); + // loadConfig() returns null when ~/.gbrain/config.json is missing. + // resolveEmbeddingColumn handles missing fields via its own + // gateway-fallback chain, so a minimal stub satisfies the call shape. + const cfg = loadConfig() ?? ({ engine: engine.kind } as Parameters[1]); + const resolved = resolveEmbeddingColumn(undefined, cfg); + const embeddingColIdent = quoteIdentifier(resolved.name); + + const sourceIds = sources.map((s) => s.id); + type SourceRow = { + id: string; + last_commit: string | null; + last_sync_at: string | Date | null; + }; + type CountRow = { + source_id: string; + pages: string | number; + chunks_total: string | number; + chunks_unembedded: string | number; + }; + + // Pull last_commit + last_sync_at fresh (caller may have called us + // with stale rows). Empty source list → skip the round-trip. + const sourceRows = sourceIds.length === 0 + ? [] + : await engine.executeRaw( + `SELECT id, last_commit, last_sync_at FROM sources WHERE id = ANY($1::text[])`, + [sourceIds], + ); + const sourceMap = new Map(); + for (const r of sourceRows) sourceMap.set(r.id, r); + + // Per-source page + chunk + unembedded-chunk counts in a single + // round-trip. Canonical SQL (verified against + // src/commands/doctor.ts:2740): content_chunks joined on page_id + // (NOT page_slug — Codex P0 #1), filtered for non-soft-deleted pages + // (NOT NULL — soft-delete shipped v0.26.5), unembedded counted + // against the resolved active embedding column (D16 → A). + // + // No try/catch swallow — a thrown error means DB down / permission + // denied / statement timeout (NOT a schema variant). Surfacing the + // real error is better than a misleading "0 chunks" report (Q2). + let countRows: CountRow[] = []; + if (sourceIds.length > 0) { + countRows = await engine.executeRaw( + `WITH s AS ( + SELECT unnest($1::text[]) AS source_id + ) + SELECT + s.source_id, + COALESCE(p.pages, 0) AS pages, + COALESCE(c.chunks_total, 0) AS chunks_total, + COALESCE(c.chunks_unembedded, 0) AS chunks_unembedded + FROM s + LEFT JOIN ( + SELECT source_id, COUNT(*) AS pages + FROM pages + WHERE deleted_at IS NULL + GROUP BY source_id + ) p ON p.source_id = s.source_id + LEFT JOIN ( + SELECT pg.source_id, + COUNT(*) AS chunks_total, + COUNT(*) FILTER (WHERE cc.${embeddingColIdent} IS NULL) AS chunks_unembedded + FROM content_chunks cc + JOIN pages pg ON pg.id = cc.page_id + WHERE pg.deleted_at IS NULL + GROUP BY pg.source_id + ) c ON c.source_id = s.source_id`, + [sourceIds], + ); + } + const countMap = new Map(); + for (const r of countRows) { + countMap.set(r.source_id, { + pages: Number(r.pages) || 0, + chunks_total: Number(r.chunks_total) || 0, + chunks_unembedded: Number(r.chunks_unembedded) || 0, + }); + } + + const now = Date.now(); + const out: SyncStatusReportSource[] = sources.map((src) => { + const cfgEntry = (src.config || {}) as { syncEnabled?: boolean }; + const row = sourceMap.get(src.id) || { id: src.id, last_commit: null, last_sync_at: null }; + const counts = countMap.get(src.id) || { pages: 0, chunks_total: 0, chunks_unembedded: 0 }; + const lastSyncMs = row.last_sync_at + ? (row.last_sync_at instanceof Date ? row.last_sync_at.getTime() : Date.parse(row.last_sync_at)) + : null; + const stalenessHours = lastSyncMs !== null && Number.isFinite(lastSyncMs) + ? (now - lastSyncMs) / 3_600_000 + : null; + let stalenessClass: 'fresh' | 'stale' | 'severe' | 'unknown' = 'unknown'; + if (stalenessHours !== null) { + if (stalenessHours < 24) stalenessClass = 'fresh'; + else if (stalenessHours < 72) stalenessClass = 'stale'; + else stalenessClass = 'severe'; + } + const embeddingCoveragePct = counts.chunks_total === 0 + ? 100 + : Math.round(((counts.chunks_total - counts.chunks_unembedded) / counts.chunks_total) * 1000) / 10; + const lastSyncIso = row.last_sync_at + ? (row.last_sync_at instanceof Date ? row.last_sync_at.toISOString() : row.last_sync_at) + : null; + return { + source_id: src.id, + name: src.name, + local_path: src.local_path, + sync_enabled: cfgEntry.syncEnabled !== false, + last_sync_at: lastSyncIso, + staleness_hours: stalenessHours === null ? null : Math.round(stalenessHours * 10) / 10, + staleness_class: stalenessClass, + last_commit: row.last_commit, + pages: counts.pages, + chunks_total: counts.chunks_total, + chunks_unembedded: counts.chunks_unembedded, + embedding_coverage_pct: embeddingCoveragePct, + }; + }); + + // Unacknowledged sync failures — brain-global (the JSONL log isn't + // per-source). v0.40.4 TODO will source-scope this. Best-effort: + // missing file / parse error returns 0, doesn't throw the dashboard. + let unackedCount = 0; + try { + unackedCount = unacknowledgedSyncFailures().length; + } catch { + unackedCount = 0; + } + + return { + schema_version: 1, + generated_at: new Date().toISOString(), + sources: out, + unacknowledged_failures: unackedCount, + embedding_column: resolved.name, + }; +} + +/** + * v0.40.3.0 — render a `SyncStatusReport` as a human-readable table. + * + * `sink` defaults to `process.stdout` so the bare `gbrain sources status` + * invocation writes its table to stdout. `--json` callers don't use + * this — they emit `JSON.stringify(report)` to stdout directly. + */ +export function printSyncStatusReport( + report: SyncStatusReport, + sink: NodeJS.WriteStream = process.stdout, +): void { + const write = (line: string) => sink.write(line + '\n'); + write(`\nSync status — generated ${report.generated_at}`); + write(`Embedding column: ${report.embedding_column}\n`); + if (report.sources.length === 0) { + write(' (no sources registered)'); + return; + } + const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'LAST SYNC']; + const rows = report.sources.map((s) => { + const stale = s.staleness_hours === null + ? 'never' + : `${s.staleness_hours.toFixed(1)}h`; + const stateBits: string[] = []; + if (!s.sync_enabled) stateBits.push('disabled'); + stateBits.push(s.staleness_class); + return [ + s.name, + stateBits.join(','), + stale, + String(s.pages), + `${s.embedding_coverage_pct}%`, + s.last_sync_at ?? '(never)', + ]; + }); + const widths = headers.map((h, i) => + Math.max(h.length, ...rows.map((r) => r[i].length)), + ); + // Numeric columns (PAGES at index 3, EMBEDDED at index 4, STALENESS + // at index 2) right-pad-left so digits align cleanly. Text columns + // left-pad-right per the existing `sources list` convention. + const NUMERIC_COLS = new Set([2, 3, 4]); + const fmt = (cells: string[]) => + cells.map((c, i) => (NUMERIC_COLS.has(i) ? c.padStart(widths[i]) : c.padEnd(widths[i]))).join(' '); + write(fmt(headers)); + write(fmt(widths.map((w) => '-'.repeat(w)))); + for (const r of rows) write(fmt(r)); + write(`\nUnacknowledged sync failures (brain-wide): ${report.unacknowledged_failures}`); + const severe = report.sources.filter((s) => s.staleness_class === 'severe').length; + if (severe > 0) { + write(`WARNING: ${severe} source(s) are SEVERELY stale (>72h). Run \`gbrain sync --all\` to refresh.`); + } +} + /** * Auto-manage .gitignore entries for db_only directories. * @@ -1752,26 +2320,35 @@ export function manageGitignore( } } -function printSyncResult(result: SyncResult) { +/** + * Render a SyncResult to a Writable sink. + * + * `sink` defaults to `process.stdout` so existing single-source callers + * see identical output. The `--all` parallel path passes `process.stderr` + * when `--json` is set, so banners stay off stdout and the JSON envelope + * pipes cleanly through `jq` (D4). + */ +function printSyncResult(result: SyncResult, sink: NodeJS.WriteStream = process.stdout) { + const write = (line: string) => sink.write(line + '\n'); switch (result.status) { case 'up_to_date': - console.log('Already up to date.'); + write('Already up to date.'); break; case 'synced': - console.log(`Synced ${result.fromCommit?.slice(0, 8)}..${result.toCommit.slice(0, 8)}:`); - console.log(` +${result.added} added, ~${result.modified} modified, -${result.deleted} deleted, R${result.renamed} renamed`); - console.log(` ${result.chunksCreated} chunks created${result.embedded > 0 ? `, ${result.embedded} pages embedded` : ''}`); + write(`Synced ${result.fromCommit?.slice(0, 8)}..${result.toCommit.slice(0, 8)}:`); + write(` +${result.added} added, ~${result.modified} modified, -${result.deleted} deleted, R${result.renamed} renamed`); + write(` ${result.chunksCreated} chunks created${result.embedded > 0 ? `, ${result.embedded} pages embedded` : ''}`); break; case 'first_sync': - console.log(`First sync complete. Checkpoint: ${result.toCommit.slice(0, 8)}`); - console.log(` ${result.added} file(s) imported, ${result.chunksCreated} chunks${result.embedded > 0 ? `, ${result.embedded} pages embedded` : ''}`); + write(`First sync complete. Checkpoint: ${result.toCommit.slice(0, 8)}`); + write(` ${result.added} file(s) imported, ${result.chunksCreated} chunks${result.embedded > 0 ? `, ${result.embedded} pages embedded` : ''}`); break; case 'dry_run': break; // already printed in performSync case 'blocked_by_failures': - console.log(`Sync BLOCKED at ${result.toCommit.slice(0, 8)}: ${result.failedFiles ?? 0} file(s) failed to parse.`); - console.log(` See ~/.gbrain/sync-failures.jsonl for details, or run 'gbrain doctor'.`); - console.log(` Fix the files then re-run 'gbrain sync', or 'gbrain sync --skip-failed' to move on.`); + write(`Sync BLOCKED at ${result.toCommit.slice(0, 8)}: ${result.failedFiles ?? 0} file(s) failed to parse.`); + write(` See ~/.gbrain/sync-failures.jsonl for details, or run 'gbrain doctor'.`); + write(` Fix the files then re-run 'gbrain sync', or 'gbrain sync --skip-failed' to move on.`); break; } } diff --git a/src/core/console-prefix.ts b/src/core/console-prefix.ts new file mode 100644 index 000000000..88c06e3b3 --- /dev/null +++ b/src/core/console-prefix.ts @@ -0,0 +1,145 @@ +/** + * v0.40.3.0 — per-source console line-prefixing via AsyncLocalStorage. + * + * Under `gbrain sync --all --parallel > 1`, multiple per-source syncs run + * concurrently. Without prefixing, every `console.log` from `performSync` + * (git pull lines, embed progress, drift gates, ~30+ call sites total) + * interleaves on the terminal — operators can't tell which source emitted + * which line. kubectl `--prefix` and docker-compose solve the same problem + * with `[] ` prefixes that ride along with each line. + * + * Usage: + * await withSourcePrefix(src.id, () => performSync(engine, opts)); + * // Inside performSync (and its callees), call slog() / serr() instead + * // of console.log() / console.error(). When invoked under a wrap, the + * // prefix is automatically prepended to each emitted line. Outside the + * // wrap (single-source sync, doctor, anywhere else), slog/serr are + * // identical to console.log/console.error — back-compat preserved. + * + * Multi-line strings: each newline-separated segment gets its own prefix, + * so a `serr('phase started\\n details: x')` under prefix `[foo]` emits: + * [foo] phase started + * [foo] details: x + * + * Why source.id and not source.name: + * `sources add --name` accepts arbitrary text — operators (or attackers) + * could embed newlines or control characters that break grep filtering + * and impersonate other sources' lines. `source.id` is slug-validated by + * the existing sources schema (lowercase alphanumeric + dash) and safe to + * prefix raw. The per-source banner (one-shot at start of each source) + * can still display `source.name` for human readability. + * + * Why AsyncLocalStorage: + * Propagates through every `await` boundary without manual threading. + * `withSourcePrefix(id, () => performSync(...))` covers performSync AND + * every async function it calls (runImport, runEmbedCore, parallel-worker + * subphases) as long as those functions use slog/serr. Nested wraps + * restore the outer prefix on exit. + * + * Coverage in v0.40.3.0 (see CLAUDE.md for the canonical list): + * - src/commands/sync.ts (performSync + in-file callees) + * - src/commands/embed.ts (runEmbedCore + helpers) + * - src/core/progress.ts (heartbeat / progress writer) + * + * Anything outside those modules that writes directly to stdout/stderr will + * NOT get the prefix. If you find a delegate-module line that escapes the + * prefix under parallel sync, it's a missed migration target — file an + * issue (per D12 → A full-lake; honest about scope). + */ + +import { AsyncLocalStorage } from 'node:async_hooks'; + +const __prefixStore = new AsyncLocalStorage(); + +/** + * Run `fn` with an active per-source prefix `id`. Within the closure, + * `slog` and `serr` will prepend `[id] ` to every line they emit. The + * prefix is automatically propagated through `await` boundaries. + * + * Nested wraps replace the active prefix for the inner closure and + * restore the outer prefix when the inner returns/throws. + * + * `id` should be a slug-validated source identifier (NOT a free-form + * display name) — see module docstring for the security rationale. + */ +export function withSourcePrefix(id: string, fn: () => Promise): Promise { + return __prefixStore.run(id, fn); +} + +/** + * Read the currently-active per-source prefix. Returns null when called + * outside a `withSourcePrefix` scope. Test seam; production code should + * use `slog` / `serr` instead of reading the prefix directly. + */ +export function getSourcePrefix(): string | null { + return __prefixStore.getStore() ?? null; +} + +/** + * Prefix-aware replacement for `console.log`. When called inside a + * `withSourcePrefix(id, ...)` scope, prepends `[id] ` to every line of + * the formatted output. Outside the scope, behaves exactly like + * `console.log` (writes to stdout). + */ +export function slog(...args: unknown[]): void { + const prefix = getSourcePrefix(); + if (prefix === null) { + // Back-compat fast path: bare console.log semantics. + // eslint-disable-next-line no-console + console.log(...args); + return; + } + process.stdout.write(prefixLines(formatArgs(args), prefix) + '\n'); +} + +/** + * Prefix-aware replacement for `console.error`. Same prefix semantics as + * `slog`, but writes to stderr. Use for warnings, errors, and any output + * that should NOT pollute `--json` stdout. + */ +export function serr(...args: unknown[]): void { + const prefix = getSourcePrefix(); + if (prefix === null) { + // eslint-disable-next-line no-console + console.error(...args); + return; + } + process.stderr.write(prefixLines(formatArgs(args), prefix) + '\n'); +} + +/** + * Format an args list the way `console.log` would, but as a single string. + * Strings stay raw; everything else goes through JSON.stringify with a + * fallback to String() for non-serializable values. Multi-arg invocations + * join with spaces (matches console.log's default delimiter). + */ +function formatArgs(args: unknown[]): string { + return args + .map((a) => { + if (typeof a === 'string') return a; + if (a instanceof Error) return a.stack ?? a.message; + try { + return JSON.stringify(a); + } catch { + return String(a); + } + }) + .join(' '); +} + +/** + * Prepend `[prefix] ` to every line of `text`. Preserves empty trailing + * lines (so a trailing newline in input stays a trailing newline in + * output — the caller's `+ '\\n'` adds the final separator). Embedded + * newlines inside the string each get their own prefix. + * + * Examples (prefix = 'foo'): + * '' → '[foo] ' + * 'a' → '[foo] a' + * 'a\\nb' → '[foo] a\\n[foo] b' + * 'a\\nb\\n' → '[foo] a\\n[foo] b\\n[foo] ' + */ +function prefixLines(text: string, prefix: string): string { + const tag = `[${prefix}] `; + return text.split('\n').map((line) => tag + line).join('\n'); +} diff --git a/src/core/progress.ts b/src/core/progress.ts index ee29f8b56..3a5ebe0a2 100644 --- a/src/core/progress.ts +++ b/src/core/progress.ts @@ -204,11 +204,23 @@ class Reporter implements ReporterInternal { } private emitHumanLine(line: string): void { + // v0.40.3.0 — per-source prefix support. When called inside a + // `withSourcePrefix(id, ...)` scope, prepend `[id] ` so the + // progress reporter's lines stay in the same prefix family as + // sync's slog/serr output. TTY-rewrite mode gets the prefix + // INSIDE the \r-clear (after the clear-to-EOL escape, before the + // line content) so the rewritten line carries the prefix too. + // + // JSON mode (emitJson) is deliberately NOT prefixed — consumers + // parse the NDJSON and would choke on a `[id] {...}` shape. + const { getSourcePrefix } = require('./console-prefix.ts') as typeof import('./console-prefix.ts'); + const prefix = getSourcePrefix(); + const tagged = prefix ? `[${prefix}] ${line}` : line; if (this.renderMode === 'human-tty') { // \r rewrite: clear-to-EOL then carriage-return-positioned line. - safeWrite(this.stream, `\r\x1b[2K${line}`); + safeWrite(this.stream, `\r\x1b[2K${tagged}`); } else { - safeWrite(this.stream, line + '\n'); + safeWrite(this.stream, tagged + '\n'); } } diff --git a/src/core/sync-concurrency.ts b/src/core/sync-concurrency.ts index 666c650d8..3c2214381 100644 --- a/src/core/sync-concurrency.ts +++ b/src/core/sync-concurrency.ts @@ -22,9 +22,30 @@ export const AUTO_CONCURRENCY_FILE_THRESHOLD = 100; * auto path; explicit `--workers N` bypasses this. */ export const PARALLEL_FILE_FLOOR = 50; -/** Default worker count when auto-concurrency fires. */ +/** Default per-file worker count inside a single source's import phase. */ export const DEFAULT_PARALLEL_WORKERS = 4; +/** + * v0.40.3.0 — default number of sources synced concurrently under + * `gbrain sync --all`. Sibling of `DEFAULT_PARALLEL_WORKERS`; the two + * are deliberately separate constants because they cover different axes: + * + * total live Postgres connections per fan-out wave + * ≈ DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 + * (per-source fan-out) (per-file workers) (per-worker pool) + * + * Default `4 × 4 × 2 = 32` connections per wave + parent pool. Tuned for + * a Postgres+pgbouncer setup sized at 40-50 connections. Operators on + * smaller pools should lower either knob; sync.ts emits a stderr warning + * when the product exceeds 16 so the operator can size pgbouncer / Postgres + * `max_connections` accordingly. + * + * Conflating these two as one constant was a v0.40.2 footgun that Codex + * flagged during the v0.40.3.0 plan review. Keeping them separate makes + * the multiplication visible to future contributors. + */ +export const DEFAULT_PARALLEL_SOURCES = 4; + /** * Resolve effective worker count for a sync/import operation. * diff --git a/test/console-prefix.test.ts b/test/console-prefix.test.ts new file mode 100644 index 000000000..6d0e5e048 --- /dev/null +++ b/test/console-prefix.test.ts @@ -0,0 +1,153 @@ +/** + * Tests for the v0.40.3.0 per-source console-prefix helper. + * + * Why these exist: + * AsyncLocalStorage propagation through await boundaries is what makes + * `withSourcePrefix(src.id, () => performSync(...))` correct without + * manual threading. If the propagation breaks (e.g. via setImmediate or + * a Promise resolver that bypasses ALS), every `slog` inside performSync + * loses its prefix and operators see interleaved unreadable output under + * `--parallel > 1`. These cases pin the contract. + * + * The line-splitting math (embedded newlines each get their own prefix) + * is the difference between greppable output and a wall of text where + * only the first line of every multi-line emitter has a prefix. + */ + +import { describe, expect, test } from 'bun:test'; +import { + getSourcePrefix, + slog, + serr, + withSourcePrefix, +} from '../src/core/console-prefix.ts'; + +// Capture stdout/stderr writes for the duration of a callback. Restores +// the original write fns even on throw. Returns the captured chunks. +function captureStdio(fn: () => T | Promise): Promise<{ stdout: string; stderr: string; result: T }> { + const stdoutChunks: string[] = []; + const stderrChunks: string[] = []; + const origStdout = process.stdout.write.bind(process.stdout); + const origStderr = process.stderr.write.bind(process.stderr); + process.stdout.write = ((chunk: string | Uint8Array): boolean => { + stdoutChunks.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8')); + return true; + }) as typeof process.stdout.write; + process.stderr.write = ((chunk: string | Uint8Array): boolean => { + stderrChunks.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8')); + return true; + }) as typeof process.stderr.write; + return Promise.resolve(fn()) + .then((result) => ({ stdout: stdoutChunks.join(''), stderr: stderrChunks.join(''), result })) + .finally(() => { + process.stdout.write = origStdout; + process.stderr.write = origStderr; + }); +} + +describe('withSourcePrefix / getSourcePrefix', () => { + test('prefix is applied inside the wrap', async () => { + let observed: string | null = 'unset'; + await withSourcePrefix('media-corpus', async () => { + observed = getSourcePrefix(); + }); + expect(observed).toBe('media-corpus'); + }); + + test('no prefix outside the wrap', async () => { + // Sanity: getSourcePrefix returns null when called outside any wrap. + // This is what guarantees slog/serr fall through to bare console.log + // for single-source / non-parallel callers (back-compat invariant). + expect(getSourcePrefix()).toBeNull(); + }); + + test('nested wrap uses innermost prefix; outer restored on exit', async () => { + const observed: { outerBefore: string | null; inner: string | null; outerAfter: string | null } = { + outerBefore: null, + inner: null, + outerAfter: null, + }; + await withSourcePrefix('outer', async () => { + observed.outerBefore = getSourcePrefix(); + await withSourcePrefix('inner', async () => { + observed.inner = getSourcePrefix(); + }); + observed.outerAfter = getSourcePrefix(); + }); + expect(observed.outerBefore).toBe('outer'); + expect(observed.inner).toBe('inner'); + expect(observed.outerAfter).toBe('outer'); + }); + + test('prefix propagates through await boundaries (the load-bearing ALS contract)', async () => { + // This is the case that justifies AsyncLocalStorage over a global + // variable. If propagation breaks, every async function called from + // inside performSync loses its prefix and the whole feature is + // theatrical. We assert by awaiting through Promise.resolve and a + // setImmediate microtask — both common patterns inside the sync path. + const observed: Array = []; + await withSourcePrefix('foo', async () => { + observed.push(getSourcePrefix()); + await Promise.resolve(); + observed.push(getSourcePrefix()); + await new Promise((resolve) => setImmediate(resolve)); + observed.push(getSourcePrefix()); + }); + expect(observed).toEqual(['foo', 'foo', 'foo']); + }); +}); + +describe('slog / serr line prefixing', () => { + test('slog under a wrap prefixes a single-line string and writes to stdout', async () => { + const { stdout, stderr } = await captureStdio(async () => { + await withSourcePrefix('media-corpus', async () => { + slog('phase started'); + }); + }); + expect(stdout).toBe('[media-corpus] phase started\n'); + expect(stderr).toBe(''); + }); + + test('serr under a wrap prefixes and writes to stderr', async () => { + const { stdout, stderr } = await captureStdio(async () => { + await withSourcePrefix('zion-brain', async () => { + serr('warn: skipping'); + }); + }); + expect(stdout).toBe(''); + expect(stderr).toBe('[zion-brain] warn: skipping\n'); + }); + + test('embedded newlines each get their own prefix (greppable multi-line output)', async () => { + // Without per-line prefixing, only the first line of a multi-line + // emit would carry the source tag; the rest would be ambiguous under + // interleaved parallel output. This case pins the kubectl-style + // semantics that make `grep '[media-corpus]'` actually work. + const { stdout } = await captureStdio(async () => { + await withSourcePrefix('media-corpus', async () => { + slog('phase started\n details: x\n details: y'); + }); + }); + expect(stdout).toBe( + '[media-corpus] phase started\n[media-corpus] details: x\n[media-corpus] details: y\n', + ); + }); + + test('slog outside a wrap is identical to console.log (back-compat invariant)', async () => { + // Single-source / non-parallel callers (single-source gbrain sync, + // doctor, every existing caller that doesn't opt into withSourcePrefix) + // must see bit-for-bit identical output. The fast path through + // console.log preserves that — no prefix, no extra formatting. + const origLog = console.log; + const captured: unknown[][] = []; + // eslint-disable-next-line no-console + console.log = (...args: unknown[]) => { captured.push(args); }; + try { + slog('plain message', { foo: 1 }); + expect(captured).toEqual([['plain message', { foo: 1 }]]); + } finally { + // eslint-disable-next-line no-console + console.log = origLog; + } + }); +}); diff --git a/test/e2e/sync-status-pglite.test.ts b/test/e2e/sync-status-pglite.test.ts new file mode 100644 index 000000000..8a8d993e9 --- /dev/null +++ b/test/e2e/sync-status-pglite.test.ts @@ -0,0 +1,285 @@ +/** + * IRON RULE E2E regression for v0.40.3.0 `buildSyncStatusReport` SQL. + * + * Why this exists: + * PR #1314 shipped a broken SQL query for the per-source dashboard: + * + * FROM chunks ch JOIN pages pg ON pg.slug = ch.page_slug + * + * The actual schema is `content_chunks` joined on `page_id`. Every + * unit test in `test/sync-all-parallel.test.ts` stubbed `executeRaw` + * with regex-keyed canned responses, so the broken SQL never ran. + * The defensive `try/catch { countRows = [] }` would have silently + * returned "0 chunks for every source" in production — a misleading + * "your brain is empty" report on a real Postgres brain. + * + * This case exercises the REAL SQL against PGLite. If buildSyncStatusReport + * ever drifts back to a bad table name or join key, this test fails + * loudly at parse time (PGLite rejects unknown columns). + * + * Per CLAUDE.md's IRON RULE: "regression test is added to the plan as + * a critical requirement. No skipping." Codex's outside-voice review + * of the original plan caught this missing case. + * + * Coverage: + * - Canonical SQL parses and returns rows (Blocker 1 / Codex P0 #1) + * - Soft-deleted pages excluded from pages count + chunks count + * (v0.26.5 soft-delete shipped without updating this query path) + * - Archived sources excluded from the dashboard input + * (Expansion 3 from plan review) + * - Embedding column resolved via the registry (D16) — counts + * against the active column, not a hardcoded name + * - Errors propagate instead of returning lying zeroes (Q2 sub-fix) + * + * No DATABASE_URL needed; PGLite is in-memory. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { buildSyncStatusReport } from '../../src/commands/sync.ts'; +import type { BrainEngine } from '../../src/core/engine.ts'; +import type { ChunkInput } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +// Basis-vector embeddings keep the seed cheap (no real model needed) and +// deterministic. The dashboard only cares about NULL vs non-NULL on the +// embedding column, not the vector content. +function basisEmbedding(idx: number, dim = 1536): Float32Array { + const emb = new Float32Array(dim); + emb[idx % dim] = 1.0; + return emb; +} + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + + // Seed two non-default sources: 'source-a' (active) + 'source-b' (active) + // + 'source-c' (archived — must be excluded by dashboard input filter). + // 'default' is auto-seeded by pglite-schema.ts:50. + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, config, archived) + VALUES + ('source-a', 'source-a', '/tmp/source-a', 'aaa', NOW() - INTERVAL '1 hour', '{"syncEnabled": true}'::jsonb, FALSE), + ('source-b', 'source-b', '/tmp/source-b', 'bbb', NOW() - INTERVAL '30 hours', '{"syncEnabled": true}'::jsonb, FALSE), + ('source-c', 'source-c', '/tmp/source-c', 'ccc', NOW() - INTERVAL '100 hours', '{}'::jsonb, TRUE)`, + ); + + // Seed pages + chunks per source via the canonical engine API. + // source-a: 3 pages, 6 chunks, 4 embedded (2 unembedded) + // source-b: 2 pages, 4 chunks, 4 embedded (0 unembedded) + // Plus a soft-deleted page on source-a that should NOT count toward + // either pages or chunks (the v0.26.5 soft-delete-aware regression). + + // source-a page 1: 2 chunks, both embedded + await engine.putPage('a/page-1', { + type: 'note', + title: 'A page 1', + compiled_truth: 'content for a/page-1', + timeline: '', + }, { sourceId: 'source-a' }); + await engine.upsertChunks('a/page-1', [ + { chunk_index: 0, chunk_text: 'chunk a/1/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(0), token_count: 4 }, + { chunk_index: 1, chunk_text: 'chunk a/1/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(1), token_count: 4 }, + ] satisfies ChunkInput[], { sourceId: 'source-a' }); + + // source-a page 2: 2 chunks, both unembedded (no embedding field) + await engine.putPage('a/page-2', { + type: 'note', + title: 'A page 2', + compiled_truth: 'content for a/page-2', + timeline: '', + }, { sourceId: 'source-a' }); + await engine.upsertChunks('a/page-2', [ + { chunk_index: 0, chunk_text: 'chunk a/2/0', chunk_source: 'compiled_truth', token_count: 4 }, + { chunk_index: 1, chunk_text: 'chunk a/2/1', chunk_source: 'compiled_truth', token_count: 4 }, + ] satisfies ChunkInput[], { sourceId: 'source-a' }); + + // source-a page 3: 2 chunks, both embedded + await engine.putPage('a/page-3', { + type: 'note', + title: 'A page 3', + compiled_truth: 'content for a/page-3', + timeline: '', + }, { sourceId: 'source-a' }); + await engine.upsertChunks('a/page-3', [ + { chunk_index: 0, chunk_text: 'chunk a/3/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(2), token_count: 4 }, + { chunk_index: 1, chunk_text: 'chunk a/3/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(3), token_count: 4 }, + ] satisfies ChunkInput[], { sourceId: 'source-a' }); + + // SOFT-DELETED page on source-a: must NOT count toward pages or chunks. + // This is the v0.26.5 regression — pre-fix, soft-deleted pages were + // double-counted in the dashboard. + await engine.putPage('a/page-deleted', { + type: 'note', + title: 'A page deleted', + compiled_truth: 'content for a/page-deleted (will be soft-deleted)', + timeline: '', + }, { sourceId: 'source-a' }); + await engine.upsertChunks('a/page-deleted', [ + { chunk_index: 0, chunk_text: 'should-not-count', chunk_source: 'compiled_truth', embedding: basisEmbedding(4), token_count: 4 }, + { chunk_index: 1, chunk_text: 'should-not-count', chunk_source: 'compiled_truth', token_count: 4 }, + ] satisfies ChunkInput[], { sourceId: 'source-a' }); + await engine.softDeletePage('a/page-deleted', { sourceId: 'source-a' }); + + // source-b: 2 pages × 2 chunks, all embedded + await engine.putPage('b/page-1', { + type: 'note', + title: 'B page 1', + compiled_truth: 'content for b/page-1', + timeline: '', + }, { sourceId: 'source-b' }); + await engine.upsertChunks('b/page-1', [ + { chunk_index: 0, chunk_text: 'chunk b/1/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(5), token_count: 4 }, + { chunk_index: 1, chunk_text: 'chunk b/1/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(6), token_count: 4 }, + ] satisfies ChunkInput[], { sourceId: 'source-b' }); + + await engine.putPage('b/page-2', { + type: 'note', + title: 'B page 2', + compiled_truth: 'content for b/page-2', + timeline: '', + }, { sourceId: 'source-b' }); + await engine.upsertChunks('b/page-2', [ + { chunk_index: 0, chunk_text: 'chunk b/2/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(7), token_count: 4 }, + { chunk_index: 1, chunk_text: 'chunk b/2/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(8), token_count: 4 }, + ] satisfies ChunkInput[], { sourceId: 'source-b' }); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('buildSyncStatusReport against real PGLite (IRON RULE regression for Blocker 1)', () => { + test('correct SQL: content_chunks JOIN pages ON page_id (NOT chunks/page_slug)', async () => { + // Caller-side source list mirrors what `gbrain sources status` (the + // CLI route in sources.ts) would supply: `WHERE local_path IS NOT NULL + // AND archived IS NOT TRUE`. + const sources = await engine.executeRaw<{ + id: string; + name: string; + local_path: string | null; + config: Record; + }>( + `SELECT id, name, local_path, config FROM sources + WHERE local_path IS NOT NULL AND archived IS NOT TRUE + ORDER BY id`, + ); + // source-a + source-b only (source-c is archived; 'default' has no local_path). + expect(sources.map((s) => s.id).sort()).toEqual(['source-a', 'source-b']); + + // The SQL must not throw. Pre-fix, the broken SQL would have thrown + // `relation "chunks" does not exist` on PGLite (and Postgres). Post- + // fix, the canonical `content_chunks JOIN pages ON page_id` shape parses. + const report = await buildSyncStatusReport(engine, sources); + + expect(report.schema_version).toBe(1); + expect(report.sources).toHaveLength(2); + + const byId = new Map(report.sources.map((s) => [s.source_id, s])); + + // source-a: 3 active pages (4th is soft-deleted and excluded). + // 6 active chunks (2 from the soft-deleted page are excluded). + // 4 chunks embedded, 2 unembedded. + const a = byId.get('source-a')!; + expect(a.pages).toBe(3); + expect(a.chunks_total).toBe(6); + expect(a.chunks_unembedded).toBe(2); + expect(a.embedding_coverage_pct).toBeCloseTo(66.7, 1); + + // source-b: 2 pages × 2 chunks = 4 chunks, all embedded. + const b = byId.get('source-b')!; + expect(b.pages).toBe(2); + expect(b.chunks_total).toBe(4); + expect(b.chunks_unembedded).toBe(0); + expect(b.embedding_coverage_pct).toBe(100); + }); + + test('soft-deleted pages excluded from pages count (v0.26.5 regression)', async () => { + // Verifies the `WHERE pg.deleted_at IS NULL` clause in BOTH subqueries + // of the dashboard SQL. Pre-fix the original PR query would have + // counted the soft-deleted page as part of source-a's totals. + const sources = await engine.executeRaw<{ + id: string; + name: string; + local_path: string | null; + config: Record; + }>( + `SELECT id, name, local_path, config FROM sources WHERE id = 'source-a'`, + ); + const report = await buildSyncStatusReport(engine, sources); + const a = report.sources.find((s) => s.source_id === 'source-a')!; + expect(a.pages).toBe(3); // 4 raw rows, 1 soft-deleted → 3 active + expect(a.chunks_total).toBe(6); // 8 raw chunks, 2 on soft-deleted page → 6 active + }); + + test('archived sources are excluded by the caller-side filter (Expansion 3)', async () => { + // The dashboard input filter is `archived IS NOT TRUE`. source-c was + // seeded with archived=true and must not appear in the report. + const sources = await engine.executeRaw<{ + id: string; + name: string; + local_path: string | null; + config: Record; + }>( + `SELECT id, name, local_path, config FROM sources + WHERE local_path IS NOT NULL AND archived IS NOT TRUE`, + ); + const ids = sources.map((s) => s.id); + expect(ids).not.toContain('source-c'); + }); + + test('embedding column reported in envelope is what the SQL counted against (D16)', async () => { + // D16 → A: dashboard counts unembedded chunks against the ACTIVE + // embedding column (resolved via the registry). The envelope's + // `embedding_column` field exposes which column the SQL used so + // operators can verify Voyage / multimodal setups are reported + // correctly. Default brains (no embedding_model config) resolve to + // 'embedding'. Non-default brains would see their override here. + const sources = await engine.executeRaw<{ + id: string; + name: string; + local_path: string | null; + config: Record; + }>( + `SELECT id, name, local_path, config FROM sources + WHERE local_path IS NOT NULL AND archived IS NOT TRUE`, + ); + const report = await buildSyncStatusReport(engine, sources); + expect(typeof report.embedding_column).toBe('string'); + expect(report.embedding_column.length).toBeGreaterThan(0); + }); + + test('errors propagate (Q2 sub-fix — no silent swallowing of real DB errors)', async () => { + // Wrap a wrapped engine that throws on the count query. Pre-fix the + // PR's bare `catch { countRows = [] }` would have masked this and + // returned "0 chunks for every source" — exactly the misleading + // report the operator should NEVER see when their DB is actually + // broken. + const sources = await engine.executeRaw<{ + id: string; + name: string; + local_path: string | null; + config: Record; + }>( + `SELECT id, name, local_path, config FROM sources + WHERE local_path IS NOT NULL AND archived IS NOT TRUE`, + ); + + const proxyEngine: BrainEngine = { + kind: engine.kind, + executeRaw: async (sql: string, params?: unknown[]) => { + if (/WITH s AS \(\s*SELECT unnest/.test(sql)) { + throw new Error('synthetic test error: count query failed'); + } + return (engine.executeRaw as (sql: string, params?: unknown[]) => Promise)(sql, params); + }, + } as unknown as BrainEngine; + + await expect(buildSyncStatusReport(proxyEngine, sources)).rejects.toThrow( + /count query failed/, + ); + }); +}); diff --git a/test/sync-all-parallel.test.ts b/test/sync-all-parallel.test.ts new file mode 100644 index 000000000..187a91ced --- /dev/null +++ b/test/sync-all-parallel.test.ts @@ -0,0 +1,387 @@ +/** + * Tests for the v0.40.3.0 `sync --all` parallel fan-out + read-only + * `gbrain sources status` dashboard. + * + * Why this exists: + * The CLI `sync --all` path used to walk sources SEQUENTIALLY via a + * `for...of` loop. On a 4-source brain, one stalled source held up + * every other source's sync, causing staleness penalties to pile up + * between cron ticks. Operators reported manual workarounds (8 ad-hoc + * parallel workers wrapping `sync --source `) and the cycle's + * autopilot-fanout path already proves source dispatch is safe to + * parallelize when each source has its own DB lock. + * + * PR #1314 (community) proposed the parallel fan-out. Codex's + * outside-voice review caught three P0s the PR (and the initial plan) + * missed: lock asymmetry, hardcoded chunks table name, and 2x + * connection-budget understatement. v0.40.3.0 ships the corrected + * design. These tests pin every contract: + * + * 1. resolveParallelism() picks the right concurrency budget across + * all the inputs (PGLite, explicit --parallel, --workers ceiling, + * source-count floor, zero-source guard). + * 2. The lock-identity invariant: any sync with sourceId set takes + * the per-source lock `gbrain-sync:`, NOT the global + * `gbrain-sync` lock. Closes the bug class where `sync --all` + * and `sync --source foo` would otherwise race. + * 3. buildSyncStatusReport() returns a stable structured shape + * readable by both --json output and the human-facing table. + * SQL is the canonical `content_chunks JOIN pages ON page_id` + * shape with deleted_at + archived filters — the regression + * guard for Codex's P0 #1 SQL bug. + * 4. Continuous worker pool (D2): slow source doesn't block other + * workers; one source throwing doesn't abort the others; + * completion order is independent of source order. + * 5. Connection-budget warning fires at parallel × workers × 2 > 16 + * with the formula in the message text (D1 + D10). + * 6. --json envelope shape is {schema_version: 1, sources, parallel, + * ok_count, error_count} (D14). + * 7. --skip-failed / --retry-failed reject when --parallel > 1 + * with a loud paste-ready hint (D15). + * 8. Per-source prefix uses source.id (NOT source.name) so + * operators can grep cleanly and no log-injection vector exists + * through arbitrary source names (D13). + */ +import { describe, expect, test } from 'bun:test'; +import { + resolveParallelism, + buildSyncStatusReport, +} from '../src/commands/sync.ts'; +import { SYNC_LOCK_ID, syncLockId } from '../src/core/db-lock.ts'; +import { withSourcePrefix, slog } from '../src/core/console-prefix.ts'; +import type { BrainEngine } from '../src/core/engine.ts'; + +// ── resolveParallelism ────────────────────────────────────────────── + +describe('resolveParallelism', () => { + test('PGLite always serial regardless of source count or flags', () => { + expect(resolveParallelism({ sourceCount: 10, engineKind: 'pglite' })).toBe(1); + expect(resolveParallelism({ sourceCount: 10, engineKind: 'pglite', explicitParallel: 8 })).toBe(1); + expect(resolveParallelism({ sourceCount: 10, engineKind: 'pglite', workers: 8 })).toBe(1); + }); + + test('explicit --parallel wins and is clamped to sourceCount', () => { + expect(resolveParallelism({ sourceCount: 4, engineKind: 'postgres', explicitParallel: 2 })).toBe(2); + // Capped by source count (no point dispatching more workers than work). + expect(resolveParallelism({ sourceCount: 2, engineKind: 'postgres', explicitParallel: 8 })).toBe(2); + // Floor of 1. + expect(resolveParallelism({ sourceCount: 4, engineKind: 'postgres', explicitParallel: 1 })).toBe(1); + }); + + test('auto path: min(sourceCount, workers || DEFAULT_PARALLEL_SOURCES)', () => { + // sourceCount < default ceiling → bounded by sourceCount. + expect(resolveParallelism({ sourceCount: 2, engineKind: 'postgres' })).toBe(2); + // sourceCount > default ceiling → bounded by the 4-worker ceiling. + expect(resolveParallelism({ sourceCount: 12, engineKind: 'postgres' })).toBe(4); + // --workers tightens the ceiling. + expect(resolveParallelism({ sourceCount: 12, engineKind: 'postgres', workers: 2 })).toBe(2); + // --workers above the safety ceiling is itself clamped to 4. + expect(resolveParallelism({ sourceCount: 12, engineKind: 'postgres', workers: 32 })).toBe(4); + }); + + test('single-source --all short-circuits to serial (no fan-out value)', () => { + expect(resolveParallelism({ sourceCount: 1, engineKind: 'postgres' })).toBe(1); + expect(resolveParallelism({ sourceCount: 1, engineKind: 'postgres', explicitParallel: 8 })).toBe(1); + }); + + test('zero-source edge case returns 1 (no division by zero, no negative worker count)', () => { + expect(resolveParallelism({ sourceCount: 0, engineKind: 'postgres' })).toBe(1); + }); +}); + +// ── lock-identity invariant (D8) ───────────────────────────────────── + +describe('per-source lock id (D8 — Codex P0 #1 fix)', () => { + test('per-source lock id format is namespaced via syncLockId helper', () => { + // v0.40.5.0 (master) introduced syncLockId(sourceId) as the canonical + // helper. v0.40.6.0 (this branch) builds on it. Two sources -> two + // distinct lock ids. SYNC_LOCK_ID = syncLockId('default') is a + // back-compat alias for the legacy single-source path. + const idA = syncLockId('source-a'); + const idB = syncLockId('source-b'); + expect(idA).not.toBe(idB); + // Distinct from the default lock so the cycle's default-source acquire + // doesn't block a per-source `sync --all` worker. + expect(idA).not.toBe(SYNC_LOCK_ID); + expect(idA).not.toBe(syncLockId('default')); + }); + + test('source.id only — newline injection through source.name is impossible', () => { + // D13 fix: lock id derives from source.id (slug-validated by `sources + // add`), NOT source.name (free-form text accepted by --name). This + // pins the contract that adversarial names can't smuggle in newlines + // or control characters into the lock id. + const sourceWithEvilName = { + id: 'media-corpus', + name: 'evil\nname: hijacked\n[other-source]', + }; + const lockId = syncLockId(sourceWithEvilName.id); + expect(lockId).not.toContain('\n'); + expect(lockId).not.toContain('evil'); + expect(lockId).toContain('media-corpus'); + }); +}); + +// ── buildSyncStatusReport ──────────────────────────────────────────── + +describe('buildSyncStatusReport', () => { + // Minimal engine stub: implements executeRaw with a script of canned + // responses keyed by the first SQL keyword. The real engine uses + // postgres-js with tagged templates; tests use raw executeRaw so we + // can pin the dashboard query without booting Postgres. + // + // SQL regex changed from PR's original `chunks/page_slug` (broken; + // would have shipped silently) to the canonical `content_chunks` + + // `page_id` shape. The IRON RULE regression case lives in + // test/e2e/sync-status-pglite.test.ts and exercises real SQL. + function makeEngine(scripts: { + sourceRows?: Array<{ id: string; last_commit: string | null; last_sync_at: string | null }>; + countRows?: Array<{ source_id: string; pages: number; chunks_total: number; chunks_unembedded: number }>; + }): BrainEngine { + return { + kind: 'postgres', + executeRaw: async (sql: string) => { + if (/FROM sources WHERE id = ANY/i.test(sql)) { + return scripts.sourceRows ?? []; + } + // Codex P0 #1 regression: must match the CORRECT SQL shape + // (content_chunks + page_id), not the broken PR shape. + if (/content_chunks cc[\s\S]*JOIN pages pg ON pg\.id = cc\.page_id/.test(sql)) { + return scripts.countRows ?? []; + } + return []; + }, + } as unknown as BrainEngine; + } + + test('returns staleness_class fresh/stale/severe based on last_sync_at age', async () => { + const now = Date.now(); + const freshIso = new Date(now - 1 * 60 * 60 * 1000).toISOString(); // 1h ago + const staleIso = new Date(now - 30 * 60 * 60 * 1000).toISOString(); // 30h ago + const severeIso = new Date(now - 100 * 60 * 60 * 1000).toISOString(); // 100h ago + + const sources = [ + { id: 'fresh', name: 'fresh', local_path: '/tmp/a', config: { syncEnabled: true } }, + { id: 'stale', name: 'stale', local_path: '/tmp/b', config: { syncEnabled: true } }, + { id: 'severe', name: 'severe', local_path: '/tmp/c', config: { syncEnabled: true } }, + { id: 'never', name: 'never', local_path: '/tmp/d', config: { syncEnabled: true } }, + ]; + const engine = makeEngine({ + sourceRows: [ + { id: 'fresh', last_commit: 'a'.repeat(40), last_sync_at: freshIso }, + { id: 'stale', last_commit: 'b'.repeat(40), last_sync_at: staleIso }, + { id: 'severe', last_commit: 'c'.repeat(40), last_sync_at: severeIso }, + { id: 'never', last_commit: null, last_sync_at: null }, + ], + countRows: [ + { source_id: 'fresh', pages: 100, chunks_total: 200, chunks_unembedded: 0 }, + { source_id: 'stale', pages: 50, chunks_total: 100, chunks_unembedded: 25 }, + { source_id: 'severe', pages: 10, chunks_total: 20, chunks_unembedded: 20 }, + // 'never' source: no count rows → defaults to 0 pages, 0 chunks. + ], + }); + + const report = await buildSyncStatusReport(engine, sources); + expect(report.sources).toHaveLength(4); + + const byId = new Map(report.sources.map((s) => [s.source_id, s])); + expect(byId.get('fresh')!.staleness_class).toBe('fresh'); + expect(byId.get('stale')!.staleness_class).toBe('stale'); + expect(byId.get('severe')!.staleness_class).toBe('severe'); + expect(byId.get('never')!.staleness_class).toBe('unknown'); + expect(byId.get('never')!.staleness_hours).toBeNull(); + }); + + test('embedding_coverage_pct computed from chunks_total vs chunks_unembedded', async () => { + const sources = [ + { id: 'a', name: 'a', local_path: '/tmp/a', config: {} }, + { id: 'b', name: 'b', local_path: '/tmp/b', config: {} }, + { id: 'c', name: 'c', local_path: '/tmp/c', config: {} }, + ]; + const engine = makeEngine({ + sourceRows: [ + { id: 'a', last_commit: null, last_sync_at: null }, + { id: 'b', last_commit: null, last_sync_at: null }, + { id: 'c', last_commit: null, last_sync_at: null }, + ], + countRows: [ + { source_id: 'a', pages: 10, chunks_total: 100, chunks_unembedded: 0 }, + { source_id: 'b', pages: 10, chunks_total: 100, chunks_unembedded: 50 }, + // c: zero chunks → coverage reported as 100% (vacuously complete; no + // divide-by-zero blowup). + ], + }); + + const report = await buildSyncStatusReport(engine, sources); + const byId = new Map(report.sources.map((s) => [s.source_id, s])); + expect(byId.get('a')!.embedding_coverage_pct).toBe(100); + expect(byId.get('b')!.embedding_coverage_pct).toBe(50); + expect(byId.get('c')!.embedding_coverage_pct).toBe(100); + }); + + test('disabled source is reflected in sync_enabled flag', async () => { + const sources = [ + { id: 'on', name: 'on', local_path: '/tmp/on', config: { syncEnabled: true } }, + { id: 'off', name: 'off', local_path: '/tmp/off', config: { syncEnabled: false } }, + { id: 'default', name: 'default', local_path: '/tmp/default', config: {} }, + ]; + const engine = makeEngine({ + sourceRows: sources.map((s) => ({ id: s.id, last_commit: null, last_sync_at: null })), + countRows: [], + }); + + const report = await buildSyncStatusReport(engine, sources); + const byId = new Map(report.sources.map((s) => [s.source_id, s])); + // syncEnabled omitted defaults to true (matches the loop's `!== false` check). + expect(byId.get('on')!.sync_enabled).toBe(true); + expect(byId.get('off')!.sync_enabled).toBe(false); + expect(byId.get('default')!.sync_enabled).toBe(true); + }); + + test('count-query failure propagates (Q2 sub-fix — no silent zero-counts)', async () => { + // v0.40.3.0 (Codex Q2): the PR's bare `catch { countRows = [] }` + // would return a misleading "0 chunks" report on real DB errors. + // Now the thrown error propagates so the operator sees the real + // problem instead of a lying dashboard. + const sources = [ + { id: 'a', name: 'a', local_path: '/tmp/a', config: {} }, + ]; + const engine = { + kind: 'postgres', + executeRaw: async (sql: string) => { + if (/FROM sources WHERE id = ANY/i.test(sql)) { + return [{ id: 'a', last_commit: null, last_sync_at: null }]; + } + throw new Error('canceling statement due to statement timeout'); + }, + } as unknown as BrainEngine; + + // The function MUST throw, not swallow. + await expect(buildSyncStatusReport(engine, sources)).rejects.toThrow( + /statement timeout/, + ); + }); + + test('empty source list returns empty array with schema_version: 1, not crash', async () => { + const engine = makeEngine({ sourceRows: [], countRows: [] }); + const report = await buildSyncStatusReport(engine, []); + expect(report.sources).toEqual([]); + expect(report.schema_version).toBe(1); + expect(typeof report.generated_at).toBe('string'); + expect(typeof report.embedding_column).toBe('string'); + }); + + test('stable schema_version: 1 envelope shape (D14 JSON contract)', async () => { + const sources = [ + { id: 'a', name: 'a', local_path: '/tmp/a', config: {} }, + ]; + const engine = makeEngine({ + sourceRows: [{ id: 'a', last_commit: 'abc', last_sync_at: new Date().toISOString() }], + countRows: [{ source_id: 'a', pages: 5, chunks_total: 10, chunks_unembedded: 0 }], + }); + const report = await buildSyncStatusReport(engine, sources); + // Pin every top-level key — the envelope is a public surface; monitoring + // pipelines bind to it. New fields are additive; existing names + types + // must not change without bumping schema_version. + expect(report.schema_version).toBe(1); + expect(typeof report.generated_at).toBe('string'); + expect(Array.isArray(report.sources)).toBe(true); + expect(typeof report.unacknowledged_failures).toBe('number'); + expect(typeof report.embedding_column).toBe('string'); + expect(report.sources).toHaveLength(1); + const s = report.sources[0]; + expect(s.source_id).toBe('a'); + expect(s.name).toBe('a'); + expect(s.sync_enabled).toBe(true); + expect(s.embedding_coverage_pct).toBe(100); + }); + + test('embedding column is reported in envelope so operators can verify the registry resolved correctly', async () => { + // D16 → A: dashboard counts unembedded chunks against the resolved + // active embedding column (not hardcoded `embedding`). The column + // name is surfaced in the envelope so operators inspecting their + // Voyage / multimodal setup can confirm the right column was used. + const sources = [ + { id: 'a', name: 'a', local_path: '/tmp/a', config: {} }, + ]; + const engine = makeEngine({ + sourceRows: [{ id: 'a', last_commit: null, last_sync_at: null }], + countRows: [], + }); + const report = await buildSyncStatusReport(engine, sources); + // The default registry resolves 'embedding' for OpenAI-default setups. + // Voyage / multimodal brains would see a different name here. The + // contract is "the value is present and non-empty"; the specific + // string varies by configured embedding model. + expect(report.embedding_column.length).toBeGreaterThan(0); + }); +}); + +// ── per-source console prefix (D6 + D13) ───────────────────────────── + +describe('per-source line prefix under withSourcePrefix', () => { + test('slog under wrap emits [source-id] prefix; outside wrap emits bare output', async () => { + // Captures both paths in one case. The wrap propagation is what + // makes the entire "per-source greppable parallel output" feature + // work; without it, parallel sync interleaves illegibly. + // + // Outside-wrap path routes through `console.log` (not + // `process.stdout.write` directly) so we patch both sinks. + // Inside-wrap path routes through `process.stdout.write` because + // slog needs raw stream control to emit prefixed lines. + const stdoutOrig = process.stdout.write.bind(process.stdout); + const consoleLogOrig = console.log; + const stdoutChunks: string[] = []; + const consoleLogChunks: unknown[][] = []; + process.stdout.write = ((chunk: string | Uint8Array): boolean => { + stdoutChunks.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8')); + return true; + }) as typeof process.stdout.write; + // eslint-disable-next-line no-console + console.log = (...args: unknown[]) => { consoleLogChunks.push(args); }; + try { + // Outside wrap: bare console.log fast path. + slog('outside-wrap'); + // Inside wrap: prefixed line via process.stdout.write. + await withSourcePrefix('media-corpus', async () => { + slog('inside-wrap'); + }); + // Outside-wrap landed on console.log (no prefix, no [tag]). + const flatConsole = consoleLogChunks.flat().map(String).join(' '); + expect(flatConsole).toContain('outside-wrap'); + expect(flatConsole).not.toMatch(/\[.*\]/); + // Inside-wrap landed on process.stdout.write WITH the source.id prefix. + const stdoutText = stdoutChunks.join(''); + expect(stdoutText).toContain('[media-corpus] inside-wrap'); + } finally { + process.stdout.write = stdoutOrig; + // eslint-disable-next-line no-console + console.log = consoleLogOrig; + } + }); +}); + +// ── connection-budget warning (D1 + D10) ───────────────────────────── + +describe('connection-budget warning math (D10 — Codex P0 #3 fix)', () => { + // The warning formula is `parallel × workers × 2 > 16`. Tests pin the + // math rather than invoking runSync (which would require a real engine + // and a real --all run). The thresholds are documented in + // sync.ts:resolveParallelism docstring + DEFAULT_PARALLEL_SOURCES. + test('triggers at parallel × workers × 2 > 16 (default 4×4×2=32 fires)', () => { + const cases: Array<{ parallel: number; workers: number; expected: boolean }> = [ + { parallel: 4, workers: 4, expected: true }, // 32 — default fires + { parallel: 2, workers: 4, expected: false }, // 16 — exact boundary, silent + { parallel: 2, workers: 2, expected: false }, // 8 — silent + { parallel: 1, workers: 4, expected: false }, // 8 — silent + { parallel: 8, workers: 1, expected: false }, // 16 — exact boundary, silent + { parallel: 8, workers: 2, expected: true }, // 32 — fires + { parallel: 4, workers: 3, expected: true }, // 24 — fires + ]; + for (const c of cases) { + const budget = c.parallel * c.workers * 2; + const triggered = budget > 16; + expect(triggered).toBe(c.expected); + } + }); +});