From 146a8f1eed10f2fea14a42b1953616554c43e342 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 30 May 2026 10:30:56 -0700 Subject: [PATCH] v0.41.31.0 feat(embed): delta-aware sync --all cost gate + real stale-embedding semantics (#1632) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(cost): embedding cost preview uses configured model rate, not hardcoded OpenAI The sync --all cost gate computed spend from a hardcoded EMBEDDING_COST_PER_1K_TOKENS = 0.00013 (OpenAI text-embedding-3-large) and labeled the preview with the back-compat EMBEDDING_MODEL constant, regardless of the actually-configured embedding model. A brain running a cheaper model (e.g. zeroentropyai:zembed-1 @ $0.05/Mtok) saw a preview that named the wrong provider and over-stated spend ~2.6x ($337 vs $130 on a 2.6B-token corpus). estimateEmbeddingCostUsd now resolves the live model via the gateway and prices it through embedding-pricing.ts (the existing per-provider:model table), falling back to the OpenAI rate only when the gateway is unconfigured (unit-test context) or the model is unknown. sync.ts surfaces the real model name in the preview message and JSON. Regression test pins model-aware pricing: openai 3-large vs zembed-1 must produce materially different previews; collapsing both to the OpenAI number fails the assertion. * fix(cost): sync --all gate is informational when embed is deferred; delta-aware inline gate Under federated_v2 (default), sync --all DEFERS embedding to per-source embed-backfill jobs that already cap spend at $25/source/24h. The v0.20 cost gate predated that cap and fired ConfirmationRequired + exit 2 on EVERY non-TTY sync --all without --yes, regardless of cost — blocking nightly crons over already-synced corpora and forcing permanent --yes. The gate is now mode-aware: - Deferred embed (v2 default): print an FYI deferred notice (cap-aware, "not charged by this sync") + the stale-chunk backlog estimate, and NEVER exit 2. The backfill cap is the real money gate. - Inline embed (v2 off, or --serial without --no-embed): keep the blocking gate, but estimate the actual delta — full-tree ceiling for changed sources (unchanged sources contribute 0 via the same git + chunker_version "do work?" gate doctor/sync use) + stale backlog — and block only when it exceeds the new configurable floor sync.cost_gate_min_usd (default $0.50). New pure helpers in embedding.ts (willEmbedSynchronously, shouldBlockSync) keep the decision logic hermetically testable. New engine method sumStaleChunkChars (both engines) prices the embedding backlog via estimateCostFromChars. estimateSyncAllCost's per-source walk extracted to estimateSourceTreeTokens (reused by the inline estimator). Regressions pinned: R-1 deferred non-TTY never exit 2 (headline), R-2 inline above-floor still exit 2 (protection), plus the willEmbedSynchronously / shouldBlockSync matrix and sumStaleChunkChars engine + scope + embed_skip coverage. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(embed): real stale semantics — re-embed on model/dims swap (migration v108) Pre-v0.41.30 "stale" meant only `embedding IS NULL`, so swapping the embedding model or dimensions left the whole corpus silently embedded under the OLD model — `embed --stale` ignored it and search quality quietly degraded. New `pages.embedding_signature` (TEXT, migration v108) stamps the embedding provenance (`:`) whenever a page's chunks are embedded. A later model/dims swap makes the stored signature differ from the current one, which the embed paths now detect and re-embed. GRANDFATHER (critical): the stale predicate is `embedding IS NULL OR (embedding_signature IS NOT NULL AND <> $current)` so a NULL signature is NEVER stale. After the migration every existing page has NULL → none flagged → the next `embed --stale` does NOT re-embed the whole corpus. Signatures are stamped going forward only. Surface: - countStaleChunks / sumStaleChunkChars gain an optional `signature` opt that widens staleness (read-only; used by the dry-run preview + the sync cost preview, which is now signature-aware). - invalidateStaleSignatureEmbeddings(signature, sourceId?) NULLs the embeddings of signature-mismatched pages so the EXISTING NULL-embedding cursor (listStaleChunks, untouched) re-embeds them — keeps the keyset pagination logic intact. - setPageEmbeddingSignature stamps after a page's chunks land. - Both embed loops wired: `gbrain embed --stale`/`--all` (embed.ts) and the embed-backfill minion (embed-stale.ts) invalidate-then-stamp. Migration v108 + bootstrap probe (both engines) + REQUIRED_BOOTSTRAP_COVERAGE entry. Pinned by test/embedding-signature-stale.test.ts (R-4 grandfather, mismatch detection, matching no-op, scoped invalidate, stamp) + the bootstrap-coverage gate. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(sync): surface embed-backfill job state in sources status + deferred notice Under federated_v2, `sync --all` exits 0 and embedding lags behind in embed-backfill jobs (subject to cooldown + the per-source 24h cap). Pre-fix an operator had no signal those jobs were queued or lagging — the sync looked "done" while embeddings trickled in later. `gbrain sources status` now shows a BACKFILL column per source (active(N)/queued(N)/idle) plus the last completion timestamp, read from minion_jobs. The deferred-sync notice appends "N backfill job(s) queued" so a cron operator sees work is enqueued, not lost. Both reads are best-effort — a brain that never ran a worker (no minion_jobs table) reports idle/0 instead of crashing the dashboard. SyncStatusReportSource gains backfill_queued / backfill_active / backfill_last_completed_at (additive; JSON envelope schema_version unchanged). Pinned by a new case in test/e2e/sync-status-pglite.test.ts. Co-Authored-By: Claude Opus 4.8 (1M context) * test: add currentEmbeddingSignature to embedding.ts mocks + sync stale EXPECTED_PHASES Commit 3 made embed.ts import currentEmbeddingSignature from embedding.ts. Four tests mock.module the whole embedding.ts and omitted the new export, so embed.ts (imported transitively) failed at load with "Export named 'currentEmbeddingSignature' not found". Add the export to each mock: embed.serial.test.ts, e2e/cycle.test.ts, e2e/dream.test.ts, e2e/dream-cycle-phase-order-pglite.test.ts. Also sync the stale EXPECTED_PHASES in dream-cycle-phase-order-pglite.test.ts to match cycle.ts ALL_PHASES — extract_atoms, synthesize_concepts, and conversation_facts_backfill drifted in after the test was last touched (v0.41.0.0) and were never added, so both phase-order assertions were failing on the branch before this wave (confirmed against 0906ab0a). The dry-run cycle emits all 20 phases, so mirroring the constant makes both assertions pass. Pre-existing, unrelated: cycle.test.ts / dream.test.ts have 5 runCycle failures via direct `bun test` (the conversation_facts_backfill phase uses the module-singleton getConnection) — present identically at 0906ab0a, not touched by this wave. Co-Authored-By: Claude Opus 4.8 (1M context) * test: pin R-3 chunker-drift regression + embed-signature stamp-call wiring Ship-workflow coverage audit flagged two gaps: - R-3 (mandatory regression) had no dedicated test: the inline unchanged-source short-circuit requires git-unchanged AND chunker_version match, but nothing pinned the chunker half. Add a case where git is unchanged (HEAD==last_commit, clean) but chunker_version is stale → estimate still fires (exit 2), plus a control where chunker matches → short-circuits to $0 (no block). - The embed loops' setPageEmbeddingSignature call-site was only kept green by the mock, never asserted. Add a test that runs `embed --all` and asserts the stamp fires once per page with the current signature. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(embed): stamp embedding_signature on the inline import + per-slug paths (F1); inline cost gate counts new-content only (F2) Adversarial review caught that the stale-detection feature was inert for non-federated/inline brains: the embed-write paths that DON'T go through embed.ts/embed-stale.ts never stamped pages.embedding_signature. F1 — stamp at the remaining write sites: - embedPage (gbrain embed + sync's post-import runEmbedCore({slugs})) - importFromContent markdown branch (inline import/sync embed + gbrain import) - importCodeFile (only when EVERY chunk was freshly embedded this call — reuse-by-hash carries old-model vectors, so a mixed page stays unstamped rather than falsely marked current) Without this, inline-synced pages kept NULL signatures → grandfathered → never re-embedded on a model/dims swap. Now all embed-write paths stamp. F2 — coupled regression the F1 fix would otherwise introduce: the inline cost gate added the stale backlog (NULL + signature drift) into the BLOCKING cost, but `gbrain sync` inline only embeds new/changed content — the backlog is `gbrain embed --stale`'s job. Once F1 gives inline brains real signatures, a model swap would inflate the inline gate and block the next cron for cost the sync never incurs. Inline blocking cost is now new-content only; the stale backlog is shown informationally ("pending gbrain embed --stale"). Deferred path keeps the signature-aware backlog FYI (the backfill does clear it). Pinned by test/import-signature-stamp.serial.test.ts (inline stamp + --no-embed NULL) and the existing R-2/R-3 inline-gate tests (still exit 2 on new-content). Co-Authored-By: Claude Opus 4.8 (1M context) * chore: bump version and changelog (v0.41.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(sources): wire BACKFILL into the real `sources status` path (P0a); guard partial-page signature stamping (P0b) Codex adversarial review caught two issues in the v0.41.30 wave: P0a — `gbrain sources status` routes through computeAllSourceMetrics (source-health.ts), not the buildSyncStatusReport helper where the BACKFILL column was added, so the CLI never showed it. Add per-source embed-backfill active/queued counts to computeAllSourceMetrics (one extra FILTER on the existing minion_jobs query) and render a BACKFILL column in `sources status`. The deferred-sync notice's queued-job count (live sync path) already worked. P0b — embedPage / embedAllStale / embed-stale stamped embedding_signature unconditionally after embedding only the STALE subset of a page's chunks. A partially-embedded page (some chunks preserved from a prior embed under unknown/old provenance) would be falsely marked current, hiding the old vectors from future stale detection. Now stamp only when EVERY chunk of the page was (re)embedded this pass (toEmbed === chunks / stale === existing). importFromContent embeds the full chunk set so it stays unconditional; importCodeFile already had the equivalent guard. `gbrain embed --all` fully re-embeds and stamps mixed pages. Accepted as documented limitations (not fixed): the inline cost gate can over-estimate a >100-file `--serial` sync that performSync will defer (non-default mode, conservative-high bias), and model-swap invalidation NULLs drifted vectors before re-embed (a deliberate, rare operation). Pinned by a new backfill-counts case in test/source-health.test.ts. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: update project documentation for v0.41.30.0 Refresh CLAUDE.md Key Files + Commands for the embedding cost-model + stale-semantics wave: model-aware cost helpers in embedding.ts (currentEmbeddingPricePerMTok / currentEmbeddingSignature / willEmbedSynchronously / shouldBlockSync), the embedding-signature stale-detection engine quartet (sumStaleChunkChars / setPageEmbeddingSignature / invalidateStaleSignatureEmbeddings + widened countStaleChunks), migration v108, signature stamping across embed.ts / import-file.ts, the mode-aware sync --all cost gate + sync.cost_gate_min_usd config key, and the sources status BACKFILL column. Add a same-dimension-swap auto-reembed note to docs/embedding-migrations.md. Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: re-version 0.41.30.0 → 0.41.31.0 (queue slot) Mechanical version-string sweep across VERSION, package.json, CHANGELOG, CLAUDE.md, docs, source/test comments, and regenerated llms bundles. No logic change. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(test): pin embedding dim in cosine-rescore-column test (CI shard-5 contamination) CI shard 5 failed deterministically with `expected 1280 dimensions, not 1536`. Root cause: cosine-rescore-column.test.ts hardcodes 1536-dim `embedding` vectors and asserts length 1536, but its beforeAll ran `initSchema()` with no gateway config. initSchema sizes the `embedding` column from getEmbeddingDimensions(), whose default is 1280 (zeroentropyai:zembed-1). The test only passed by inheriting a leaked 1536 gateway config from an earlier test (or, locally, from ~/.gbrain). When the v0.41.31 merge shifted the weight-aware shard bin-packing, the file order changed so the 1280 default won in CI → vector(1280) column → 1536 insert rejected. (Passed locally because the dev machine's ~/.gbrain resolves 1536.) Fix: configureGateway({ openai:text-embedding-3-large, 1536 }) in beforeAll BEFORE connect/initSchema so the column is deterministically vector(1536) regardless of ambient/leaked state, and resetGateway() in afterAll for hygiene. Proven: under a forced-1280 gateway preload the old test reproduces the exact CI error and the fixed test passes (4/4). Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: t Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 104 +++++ CLAUDE.md | 24 +- VERSION | 2 +- docs/embedding-migrations.md | 28 ++ llms-full.txt | 24 +- package.json | 2 +- src/commands/embed.ts | 49 ++- src/commands/sources.ts | 13 +- src/commands/sync.ts | 400 +++++++++++++----- src/core/embed-stale.ts | 27 ++ src/core/embedding.ts | 104 ++++- src/core/engine.ts | 33 +- src/core/import-file.ts | 17 +- src/core/migrate.ts | 26 ++ src/core/minions/handlers/embed-backfill.ts | 4 + src/core/pglite-engine.ts | 115 ++++- src/core/postgres-engine.ts | 128 ++++-- src/core/source-health.ts | 27 +- test/cosine-rescore-column.test.ts | 13 + test/e2e/cycle.test.ts | 2 + .../dream-cycle-phase-order-pglite.test.ts | 8 + test/e2e/dream.test.ts | 2 + test/e2e/sync-status-pglite.test.ts | 37 ++ test/embed.serial.test.ts | 28 ++ test/embedding-signature-stale.test.ts | 121 ++++++ test/import-signature-stamp.serial.test.ts | 76 ++++ test/schema-bootstrap-coverage.test.ts | 4 + test/source-health.test.ts | 18 + test/sum-stale-chunk-chars.test.ts | 113 +++++ test/sync-cost-gate.serial.test.ts | 160 +++++++ test/sync-cost-preview.test.ts | 85 +++- 31 files changed, 1599 insertions(+), 195 deletions(-) create mode 100644 test/embedding-signature-stale.test.ts create mode 100644 test/import-signature-stamp.serial.test.ts create mode 100644 test/sum-stale-chunk-chars.test.ts create mode 100644 test/sync-cost-gate.serial.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 92a85a5c8..10af4533e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,110 @@ All notable changes to GBrain will be documented in this file. +## [0.41.31.0] - 2026-05-30 + +**Your nightly `gbrain sync --all` cron stops getting blocked. It used to +demand `--yes` on every non-interactive run even when nothing needed +embedding. Now it just runs. And when you change your embedding model, +`gbrain embed --stale` actually re-embeds the pages still on the old model +instead of silently leaving them stale.** + +Before this release, `gbrain sync --all` always stopped and asked for +confirmation in any non-interactive run (cron, script, piped output) based on +a guess of your whole corpus's embedding cost. On an already-synced brain that +guess was large and the real work was zero, so the cron exited with an error +every night and you had to wire in `--yes` permanently, which defeats the +safety check. + +Here's what changed. On the default fast sync path, embedding doesn't run +during the sync itself; it's handed to background jobs that already cap their +own spend at $25 per source per day. So the sync command now prints what's +queued and proceeds. It never blocks. You'll only see a confirmation prompt +when sync embeds inline (the older non-parallel mode), and even then only when +the new content this sync embeds crosses a dollar threshold you control. + +**The threshold knob:** +``` +gbrain config set sync.cost_gate_min_usd 0.50 # default; raise or lower to taste +``` +At or below this estimate an inline sync proceeds silently. Set it to 0 to +confirm on any nonzero cost. + +**Real stale detection.** gbrain now records which model and dimensions +produced each page's vectors. Change your embedding model (OpenAI to Voyage, +or a different dimension) and `gbrain embed --stale` finds and re-embeds every +page still on the old model. Before, "stale" only meant "never embedded," so a +model swap was invisible and search quietly mixed vector spaces. + +This does NOT trigger a surprise re-embed of your whole brain on upgrade. Pages +that predate this release have no recorded signature, and gbrain treats "no +signature" as "leave it alone." Only pages embedded after you upgrade carry a +signature, and only an actual model change marks them stale. + +**See your embedding backlog.** `gbrain sources status` gains a BACKFILL column +showing whether each source has embed jobs queued, running, or idle, plus when +the last one finished. After a `sync --all` you can tell at a glance whether +embeddings are caught up or still trickling in. + +## To take advantage of v0.41.31.0 + +`gbrain upgrade` does this automatically (it applies migration v108, which adds +the per-page embedding-signature column). If `gbrain doctor` warns about a +partial migration: + +1. **Apply migrations manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Verify the cron no longer blocks:** + ```bash + gbrain sync --all --dry-run # previews, exits 0, no confirmation demanded + gbrain sources status # BACKFILL column present + ``` +3. **After a future model swap:** `gbrain embed --stale` re-embeds the drifted + pages. (Pages embedded before this upgrade keep their old vectors until they + are next re-embedded for some other reason — the no-surprise-cost + grandfather behavior.) +4. **If anything looks wrong,** file an issue at + https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`. + +### Itemized changes + +#### Sync cost gate +- `gbrain sync --all` is mode-aware. When embedding is deferred to backfill + jobs (the federated_v2 default), the gate is informational only: it prints a + deferred notice (cap-aware, "not charged by this sync") plus the current + backlog and never exits 2. The blocking confirmation gate fires only when + sync embeds inline AND the new-content estimate exceeds + `sync.cost_gate_min_usd` (default $0.50). +- The inline estimate is delta-aware: sources unchanged since their last sync + (HEAD == last_commit, clean tree, chunker version current) contribute 0; + changed sources contribute the full-tree ceiling. The pre-existing stale + backlog is shown informationally but does not gate inline sync (that backlog + is `gbrain embed --stale`'s job, not sync's). +- The cost preview prices against the configured model's real rate rather than + a hardcoded OpenAI rate. + +#### Real stale semantics (migration v108) +- New `pages.embedding_signature` column records `:` when + a page's chunks are embedded. `gbrain embed --stale`, the embed-backfill + jobs, and the sync cost preview treat a page as stale when its signature + differs from the current model's. NULL signature is grandfathered (never + stale) so upgrading does not mass-re-embed. +- All embed-write paths stamp the signature: `gbrain embed`/`--all`/`--stale`, + the embed-backfill minion, `gbrain sync`'s inline import, and `gbrain import`. +- New engine methods on both Postgres and PGLite: `sumStaleChunkChars`, + `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, plus an + optional `signature` filter on `countStaleChunks` / `sumStaleChunkChars`. + +#### Backfill visibility +- `gbrain sources status` shows per-source embed-backfill state + (queued/active/idle + last completion). The deferred sync notice appends the + queued-job count so a cron operator sees work is enqueued, not lost. + +#### Configuration +- `sync.cost_gate_min_usd` (default 0.50) sets the inline-sync confirmation + floor. ## [0.41.30.0] - 2026-05-30 **`gbrain lsd --save` (and `brainstorm --save`) now actually writes the diff --git a/CLAUDE.md b/CLAUDE.md index 6a0cb6406..06718167f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. @@ -52,8 +52,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. **v0.41.31.0:** implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, and the widened `countStaleChunks({sourceId?, signature?})`. The `signature` opt widens the stale predicate via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)` (NULL grandfathered). Parity SQL with postgres-engine.ts. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. **v0.41.31.0:** implements the embedding-signature stale-detection quartet in parity with pglite-engine.ts — `sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, and the widened `countStaleChunks` (all accept the optional `signature` to extend "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN; NULL signature grandfathered). The v0.22.1 `countStaleChunks` / `listStaleChunks` server-side `embedding IS NULL` filter is preserved as the no-signature fast path. Pinned by `test/e2e/engine-parity.test.ts`. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -64,7 +64,7 @@ strict behavior when unset. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). **v0.26.9 (D14):** adds `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated. Replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback path. Reusable from any future code that needs the same column-existence probe semantics. **v0.32.8 (PR #860):** adds `validateSourceId(id)` that throws on anything outside `^[a-z0-9_-]+$`. Used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` call so source_id can't traverse out of brainDir. `rowToPage` updated to populate the now-required `Page.source_id` field from the SELECT projection (`scripts/check-source-id-projection.sh` enforces that every projection feeding `rowToPage` includes the column). - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) -- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags) +- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). **v0.41.31.0:** `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) — covers the inline import/sync path so a model/dims swap is detectable as stale. `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`); mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle the swap for those). - `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. @@ -112,7 +112,7 @@ strict behavior when unset. - `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. -- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. +- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. **v0.41.31.0 (cost-model + stale-semantics wave):** `estimateEmbeddingCostUsd(tokens)` now prices against the CURRENTLY-CONFIGURED model's rate instead of the hardcoded OpenAI `EMBEDDING_COST_PER_1K_TOKENS`. Pre-fix, a brain on a cheaper model (e.g. ZeroEntropy) saw a `sync --all` preview that named OpenAI and over-stated spend ~2.6x. New `currentEmbeddingPricePerMTok()` resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate (0.13) only when the gateway is unconfigured (unit tests, pre-connect preview) or the model is unknown to the pricing table. `EMBEDDING_COST_PER_1K_TOKENS` is retained for back-compat with direct importers/tests. New `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (that's tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from the current one and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. New `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so the cost gate and the actual embed decision can never drift. New `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. - `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962. - `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline. **v0.37.6.0 (#1210):** new `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers that ride alongside auth on every openai-compat touchpoint. Mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) are rejected at `applyResolveAuth` call time so defaults cannot accidentally shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple; usable by any future recipe (Together/Groq) that wants attribution. - `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.36.3.0:** `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path can embed via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default. The hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}`; the gateway resolves the matching recipe and routes through its `instantiateEmbedding()` branch. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down — fixes the CDX-4 bug where a healthy Voyage column would skip vector retrieval because OpenAI happened to be unconfigured. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. @@ -172,7 +172,7 @@ strict behavior when unset. - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. **v0.41.31.0:** every embed-write path now stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. - `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. @@ -180,7 +180,7 @@ strict behavior when unset. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` (v0.41.23.0) — unified extract operator-surface wave. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) now writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug per D-EXTRACT-17: `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` per D-EXTRACT-19 (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts` so receipts surface in search but never dominate user content. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes are best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. New `extract_health` doctor check reads the rollup for last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok` silently. Operator CLI surface: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted by halt_rate desc + cost desc, kubectl-style top-5 + "more rows" hint, stable `schema_version: 1` JSON envelope); `gbrain extract --explain ` (resolution chain: pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)` markers, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict D-EXTRACT-21 path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; v0.41.23 ships as a stub-reporter — LLM dispatch deferred to a follow-up release). Pack-author authoring loop: `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` per D-EXTRACT-37 — parses but refuses at runtime in v0.41.23); new `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` helpers in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable in one verb, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Tests: 22 cases in `test/extractable-spec-widening.test.ts` (back-compat boolean shape + struct shape + D-EXTRACT-37 verifier_path refuse), 12 cases in `test/extract/receipt-writer.test.ts` (uses canonical PGLite block per CLAUDE.md R3+R4 — one engine per file with `resetPgliteState` in `beforeEach`), 17 cases in `test/extract/benchmark.test.ts` (path validation rejections + JSONL contract), 15 cases in `test/extract/status.test.ts` (pure aggregation + formatting), 15 cases in `test/schema-pack/scaffold-extractable.test.ts` (scaffold mechanics + explicit privacy-rule guards against real-name leakage), 8 cases in `test/doctor-extract-health.test.ts`. Plan persisted at `~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md`. Deferred to a follow-up release: replay versioning + `gbrain extract replay --since v`; unified LLM dispatcher (deliberately respects v0.41.13 head comment that extraction is regex + cost-cap value-add lives at embed step). - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. -- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). +- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). **v0.41.31.0 (P0a):** `gbrain sources status` (the live `runStatus` health table path, NOT just the `buildSyncStatusReport` dashboard) gains a `BACKFILL` column between `EMBED` and `FAILS`: `active(N)` beats `queued(N)` beats `idle`, sourced from `SourceMetrics.backfill_active` / `backfill_queued` in `src/core/source-health.ts`. Under federated_v2, `sync --all` defers embedding to per-source `embed-backfill` minion jobs, so an operator needs to SEE that deferred work after sync exits 0 instead of assuming it was lost. The data layer (`jobCountsBySource` in `source-health.ts`) widens its one `minion_jobs` SQL with two `COUNT(*) FILTER (WHERE name = 'embed-backfill' AND ...)` aggregates (active vs waiting/delayed/waiting-children) — best-effort, all-0 on pre-minions brains. Pinned by `test/source-health.test.ts`. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint). **v0.41.13.0 (#1434):** new tier 5.5 `sole_non_default` slots between `brain_default` and `seed_default`. When NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'`, auto-route to it. `SOURCE_TIER_NAMES` extended to 7 entries. New private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points (cannot drift). Archived sources excluded (try/catch for pre-v34 brains). New exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (returns null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `src/commands/sync.ts:1497-1519` rewired to call `resolveSourceWithTier` unconditionally so the new tier actually fires (codex caught the original plan shipping dead code). `src/commands/import.ts:96-128` mirrors the same wiring with the tier-gated nudge. Pinned by `test/source-resolver-sole-non-default.test.ts` (14 cases) + `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync` with stderr capture). - `src/core/sync.ts` extension (v0.41.13.0, #1433) — `isSyncable` factored through a private `classifySync(path, opts): SyncableReason | null` helper. New exported `unsyncableReason(path, opts)` companion returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` lifted to a named export (the four canonical metafile basenames: `schema.md`, `index.md`, `log.md`, `README.md`). New `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. `src/commands/sync.ts:772` cleanup loop now guards on `unsyncableReason(path) === 'metafile'` so previously-indexed metafile pages (typically `log.md` from an older gbrain that didn't filter it) survive every re-sync. Honest scope: does NOT cover `manifest.deleted` (the upstream filter already strips metafiles, so `rm log.md` doesn't delete the page either — v0.42+ TODO for a `gbrain pages remove ` operator surface). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases pinning the duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases including the negative case — renamed `.md → .txt` still gets cleaned up properly). @@ -263,7 +263,7 @@ strict behavior when unset. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -275,7 +275,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. **v0.41.31.0 (mode-aware cost gate):** the v0.20.0 unconditional `sync --all` ConfirmationRequired gate is superseded by a mode-aware gate driven by `willEmbedSynchronously` + `shouldBlockSync` from `src/core/embedding.ts`. federated_v2 is resolved ONCE up front (shared with the fan-out below). On the DEFERRED path (v2 on, parallel) embedding goes to per-source `embed-backfill` jobs that carry their own `$X/source/24h` cap (default $25, `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`), so the gate is INFORMATIONAL — it prints an FYI naming the backfill cap + current backlog (`sumStaleChunkChars({signature})` priced via `estimateCostFromChars` at `currentEmbeddingPricePerMTok()`) + the count of already-queued `embed-backfill` jobs (TODO-2 visibility), and NEVER exits 2. On the INLINE path (v2 off, or `--serial` without `--no-embed`) the BLOCKING gate fires only when the NEW-CONTENT estimate exceeds the `sync.cost_gate_min_usd` floor (default $0.50). The new-content estimate (`estimateInlineNewTokens`) contributes ZERO for sources provably unchanged since last sync (`isSourceUnchangedSinceSync` HEAD==last_commit + clean tree, AND `chunker_version === CURRENT`) — mirrors doctor's `sync_freshness` + sync's own do-work gate — and the full-tree ceiling for changed sources. F2 fix: the pre-existing stale backlog (NULL embeddings + signature drift) is shown informationally but NEVER gated on, because sync doesn't sweep it inline (`gbrain embed --stale` does); gating on it would block every inline cron after a model swap. New helpers `resolveCostGateFloorUsd(engine)` (reads `sync.cost_gate_min_usd`, fail-open $0.50, accepts 0) + `resolveBackfillCapUsd(engine)`. The `sync --all` SELECT widens to carry `last_commit + chunker_version` (both columns predate v0.41; no migration). JSON envelopes gain `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | confirmation_required`). `SyncStatusReportSource` gains `backfill_queued` / `backfill_active` / `backfill_last_completed_at` (best-effort; 0/null on pre-minions brains). All embedding-cost preview math reads the configured model name via `getEmbeddingModelName()` (no hardcoded OpenAI). Pinned by `test/sync-cost-gate.serial.test.ts` + `test/sync-cost-preview.test.ts`. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -659,6 +659,12 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. +Key commands added in v0.41.31.0 (embedding cost-model + stale-semantics wave): +- `gbrain sync --all` cost gate is now mode-aware. When embedding is DEFERRED (federated_v2 parallel path → per-source `embed-backfill` jobs), the gate is INFORMATIONAL — it prints the backfill cap, the current backlog cost, and the count of queued backfill jobs, and NEVER exits 2. When embedding runs INLINE (v2 off, or `--serial` without `--no-embed`), the gate BLOCKS only when the NEW-CONTENT estimate exceeds the floor. Cost previews now name the ACTUALLY-configured embedding model and price at its real rate (no more hardcoded OpenAI over-statement on cheaper models). +- `gbrain config set sync.cost_gate_min_usd ` — inline-path cost-gate floor (DB plane, default `$0.50`). Below this estimate the inline `sync --all` gate proceeds without blocking, killing the noise where a tiny incremental sync prompted on every cron. Set to `0` to block on any nonzero inline cost. Fail-open to the default on a missing/invalid value. +- `gbrain embed --stale` now re-embeds pages whose embedding model/dimensions changed, not just pages with NULL embeddings. After a model or dims swap, the stored `pages.embedding_signature` differs from the current one, so the next `embed --stale` (or per-source `embed-backfill` job) invalidates and re-embeds those pages. GRANDFATHER: pages embedded before v0.41.31.0 have a NULL signature and are NEVER flagged stale, so the upgrade does NOT trigger a whole-corpus re-embed. `embed --stale --dry-run` counts signature-drift without mutating. +- `gbrain sources status` gains a `BACKFILL` column (`active(N)` / `queued(N)` / `idle`) between `EMBED` and `FAILS` so an operator can see deferred embedding work after `sync --all` exits 0. + Key commands added in v0.36.4.0 (brain-health-100 wave): - `gbrain doctor --remediation-plan [--target-score N] [--json]` — preview the dependency-ordered plan that would drive the brain to target. JSON envelope is stable: each `Remediation` carries `id`, `idempotency_key` (content-hash for cron-safe retries), `severity`, `est_seconds`, `est_usd_cost`, and `depends_on` (referencing other ids). Empty `recommendations` array when the brain is already at target. - `gbrain doctor --remediate [--yes] [--target-score N] [--max-usd N]` — actually submit the plan. Walks dependency order, submits one Minion job per step, re-checks score between steps, refuses to spend past `--max-usd` (defaults: target=90, max-usd=infinite — but cron callers should always pass `--max-usd`). Bails when target exceeds `maxReachableScore()` for the brain (empty / under-configured brains) with a clear list of what's missing. diff --git a/VERSION b/VERSION index 13bcae3ae..c9ccfb296 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.30.0 +0.41.31.0 \ No newline at end of file diff --git a/docs/embedding-migrations.md b/docs/embedding-migrations.md index d1eaed639..e9c425128 100644 --- a/docs/embedding-migrations.md +++ b/docs/embedding-migrations.md @@ -10,6 +10,34 @@ change automatically. this mismatch and refuse to silently proceed. This doc is the recipe they point at. +## Same-dimension model swaps (v0.41.31.0 — automatic) + +If you switch to a different model at the **same** dimension count +(e.g. one 1536-dim provider to another, or a re-tuned model that keeps +its width), the column type doesn't change, so no `ALTER`/wipe recipe +is needed. As of v0.41.31.0, gbrain stamps an embedding-provenance +signature (`:`) onto each page when its chunks are +embedded. After you point the config at the new model, the stored +signatures differ from the current one, and `gbrain embed --stale` +re-embeds exactly those pages: + +```bash +# After switching to the new same-dim model in your config: +gbrain embed --stale # re-embeds signature-drifted pages +gbrain embed --stale --dry-run # preview the count without re-embedding +``` + +Under federated_v2, the same drift is picked up by the per-source +`embed-backfill` jobs that `gbrain sync --all` enqueues (capped +`$X/source/24h`). **Grandfather:** pages embedded before v0.41.31.0 +carry a NULL signature and are NEVER flagged stale, so upgrading to +v0.41.31.0 does NOT trigger a whole-corpus re-embed. Signatures only +get stamped going forward. + +A **dimension** change still requires the wipe-and-reinit (PGLite) or +column-alter (Postgres) recipe below — the on-disk `vector(N)` width +genuinely has to change. + ## Why we don't do this automatically Switching dimensions requires: diff --git a/llms-full.txt b/llms-full.txt index 422ff20b2..9b7db17d8 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -183,7 +183,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. @@ -194,8 +194,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. **v0.41.31.0:** implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, and the widened `countStaleChunks({sourceId?, signature?})`. The `signature` opt widens the stale predicate via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)` (NULL grandfathered). Parity SQL with postgres-engine.ts. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. **v0.41.31.0:** implements the embedding-signature stale-detection quartet in parity with pglite-engine.ts — `sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, and the widened `countStaleChunks` (all accept the optional `signature` to extend "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN; NULL signature grandfathered). The v0.22.1 `countStaleChunks` / `listStaleChunks` server-side `embedding IS NULL` filter is preserved as the no-signature fast path. Pinned by `test/e2e/engine-parity.test.ts`. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -206,7 +206,7 @@ strict behavior when unset. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). **v0.26.9 (D14):** adds `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated. Replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback path. Reusable from any future code that needs the same column-existence probe semantics. **v0.32.8 (PR #860):** adds `validateSourceId(id)` that throws on anything outside `^[a-z0-9_-]+$`. Used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` call so source_id can't traverse out of brainDir. `rowToPage` updated to populate the now-required `Page.source_id` field from the SELECT projection (`scripts/check-source-id-projection.sh` enforces that every projection feeding `rowToPage` includes the column). - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) -- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags) +- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). **v0.41.31.0:** `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) — covers the inline import/sync path so a model/dims swap is detectable as stale. `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`); mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle the swap for those). - `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. @@ -254,7 +254,7 @@ strict behavior when unset. - `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. -- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. +- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. **v0.41.31.0 (cost-model + stale-semantics wave):** `estimateEmbeddingCostUsd(tokens)` now prices against the CURRENTLY-CONFIGURED model's rate instead of the hardcoded OpenAI `EMBEDDING_COST_PER_1K_TOKENS`. Pre-fix, a brain on a cheaper model (e.g. ZeroEntropy) saw a `sync --all` preview that named OpenAI and over-stated spend ~2.6x. New `currentEmbeddingPricePerMTok()` resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate (0.13) only when the gateway is unconfigured (unit tests, pre-connect preview) or the model is unknown to the pricing table. `EMBEDDING_COST_PER_1K_TOKENS` is retained for back-compat with direct importers/tests. New `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (that's tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from the current one and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. New `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so the cost gate and the actual embed decision can never drift. New `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. - `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962. - `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline. **v0.37.6.0 (#1210):** new `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers that ride alongside auth on every openai-compat touchpoint. Mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) are rejected at `applyResolveAuth` call time so defaults cannot accidentally shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple; usable by any future recipe (Together/Groq) that wants attribution. - `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.36.3.0:** `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path can embed via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default. The hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}`; the gateway resolves the matching recipe and routes through its `instantiateEmbedding()` branch. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down — fixes the CDX-4 bug where a healthy Voyage column would skip vector retrieval because OpenAI happened to be unconfigured. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. @@ -314,7 +314,7 @@ strict behavior when unset. - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. **v0.41.31.0:** every embed-write path now stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. - `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. @@ -322,7 +322,7 @@ strict behavior when unset. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` (v0.41.23.0) — unified extract operator-surface wave. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) now writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug per D-EXTRACT-17: `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` per D-EXTRACT-19 (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts` so receipts surface in search but never dominate user content. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes are best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. New `extract_health` doctor check reads the rollup for last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok` silently. Operator CLI surface: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted by halt_rate desc + cost desc, kubectl-style top-5 + "more rows" hint, stable `schema_version: 1` JSON envelope); `gbrain extract --explain ` (resolution chain: pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)` markers, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict D-EXTRACT-21 path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; v0.41.23 ships as a stub-reporter — LLM dispatch deferred to a follow-up release). Pack-author authoring loop: `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` per D-EXTRACT-37 — parses but refuses at runtime in v0.41.23); new `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` helpers in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable in one verb, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Tests: 22 cases in `test/extractable-spec-widening.test.ts` (back-compat boolean shape + struct shape + D-EXTRACT-37 verifier_path refuse), 12 cases in `test/extract/receipt-writer.test.ts` (uses canonical PGLite block per CLAUDE.md R3+R4 — one engine per file with `resetPgliteState` in `beforeEach`), 17 cases in `test/extract/benchmark.test.ts` (path validation rejections + JSONL contract), 15 cases in `test/extract/status.test.ts` (pure aggregation + formatting), 15 cases in `test/schema-pack/scaffold-extractable.test.ts` (scaffold mechanics + explicit privacy-rule guards against real-name leakage), 8 cases in `test/doctor-extract-health.test.ts`. Plan persisted at `~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md`. Deferred to a follow-up release: replay versioning + `gbrain extract replay --since v`; unified LLM dispatcher (deliberately respects v0.41.13 head comment that extraction is regex + cost-cap value-add lives at embed step). - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. -- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). +- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). **v0.41.31.0 (P0a):** `gbrain sources status` (the live `runStatus` health table path, NOT just the `buildSyncStatusReport` dashboard) gains a `BACKFILL` column between `EMBED` and `FAILS`: `active(N)` beats `queued(N)` beats `idle`, sourced from `SourceMetrics.backfill_active` / `backfill_queued` in `src/core/source-health.ts`. Under federated_v2, `sync --all` defers embedding to per-source `embed-backfill` minion jobs, so an operator needs to SEE that deferred work after sync exits 0 instead of assuming it was lost. The data layer (`jobCountsBySource` in `source-health.ts`) widens its one `minion_jobs` SQL with two `COUNT(*) FILTER (WHERE name = 'embed-backfill' AND ...)` aggregates (active vs waiting/delayed/waiting-children) — best-effort, all-0 on pre-minions brains. Pinned by `test/source-health.test.ts`. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint). **v0.41.13.0 (#1434):** new tier 5.5 `sole_non_default` slots between `brain_default` and `seed_default`. When NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'`, auto-route to it. `SOURCE_TIER_NAMES` extended to 7 entries. New private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points (cannot drift). Archived sources excluded (try/catch for pre-v34 brains). New exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (returns null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `src/commands/sync.ts:1497-1519` rewired to call `resolveSourceWithTier` unconditionally so the new tier actually fires (codex caught the original plan shipping dead code). `src/commands/import.ts:96-128` mirrors the same wiring with the tier-gated nudge. Pinned by `test/source-resolver-sole-non-default.test.ts` (14 cases) + `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync` with stderr capture). - `src/core/sync.ts` extension (v0.41.13.0, #1433) — `isSyncable` factored through a private `classifySync(path, opts): SyncableReason | null` helper. New exported `unsyncableReason(path, opts)` companion returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` lifted to a named export (the four canonical metafile basenames: `schema.md`, `index.md`, `log.md`, `README.md`). New `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. `src/commands/sync.ts:772` cleanup loop now guards on `unsyncableReason(path) === 'metafile'` so previously-indexed metafile pages (typically `log.md` from an older gbrain that didn't filter it) survive every re-sync. Honest scope: does NOT cover `manifest.deleted` (the upstream filter already strips metafiles, so `rm log.md` doesn't delete the page either — v0.42+ TODO for a `gbrain pages remove ` operator surface). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases pinning the duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases including the negative case — renamed `.md → .txt` still gets cleaned up properly). @@ -405,7 +405,7 @@ strict behavior when unset. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -417,7 +417,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. **v0.41.31.0 (mode-aware cost gate):** the v0.20.0 unconditional `sync --all` ConfirmationRequired gate is superseded by a mode-aware gate driven by `willEmbedSynchronously` + `shouldBlockSync` from `src/core/embedding.ts`. federated_v2 is resolved ONCE up front (shared with the fan-out below). On the DEFERRED path (v2 on, parallel) embedding goes to per-source `embed-backfill` jobs that carry their own `$X/source/24h` cap (default $25, `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`), so the gate is INFORMATIONAL — it prints an FYI naming the backfill cap + current backlog (`sumStaleChunkChars({signature})` priced via `estimateCostFromChars` at `currentEmbeddingPricePerMTok()`) + the count of already-queued `embed-backfill` jobs (TODO-2 visibility), and NEVER exits 2. On the INLINE path (v2 off, or `--serial` without `--no-embed`) the BLOCKING gate fires only when the NEW-CONTENT estimate exceeds the `sync.cost_gate_min_usd` floor (default $0.50). The new-content estimate (`estimateInlineNewTokens`) contributes ZERO for sources provably unchanged since last sync (`isSourceUnchangedSinceSync` HEAD==last_commit + clean tree, AND `chunker_version === CURRENT`) — mirrors doctor's `sync_freshness` + sync's own do-work gate — and the full-tree ceiling for changed sources. F2 fix: the pre-existing stale backlog (NULL embeddings + signature drift) is shown informationally but NEVER gated on, because sync doesn't sweep it inline (`gbrain embed --stale` does); gating on it would block every inline cron after a model swap. New helpers `resolveCostGateFloorUsd(engine)` (reads `sync.cost_gate_min_usd`, fail-open $0.50, accepts 0) + `resolveBackfillCapUsd(engine)`. The `sync --all` SELECT widens to carry `last_commit + chunker_version` (both columns predate v0.41; no migration). JSON envelopes gain `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | confirmation_required`). `SyncStatusReportSource` gains `backfill_queued` / `backfill_active` / `backfill_last_completed_at` (best-effort; 0/null on pre-minions brains). All embedding-cost preview math reads the configured model name via `getEmbeddingModelName()` (no hardcoded OpenAI). Pinned by `test/sync-cost-gate.serial.test.ts` + `test/sync-cost-preview.test.ts`. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -801,6 +801,12 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. +Key commands added in v0.41.31.0 (embedding cost-model + stale-semantics wave): +- `gbrain sync --all` cost gate is now mode-aware. When embedding is DEFERRED (federated_v2 parallel path → per-source `embed-backfill` jobs), the gate is INFORMATIONAL — it prints the backfill cap, the current backlog cost, and the count of queued backfill jobs, and NEVER exits 2. When embedding runs INLINE (v2 off, or `--serial` without `--no-embed`), the gate BLOCKS only when the NEW-CONTENT estimate exceeds the floor. Cost previews now name the ACTUALLY-configured embedding model and price at its real rate (no more hardcoded OpenAI over-statement on cheaper models). +- `gbrain config set sync.cost_gate_min_usd ` — inline-path cost-gate floor (DB plane, default `$0.50`). Below this estimate the inline `sync --all` gate proceeds without blocking, killing the noise where a tiny incremental sync prompted on every cron. Set to `0` to block on any nonzero inline cost. Fail-open to the default on a missing/invalid value. +- `gbrain embed --stale` now re-embeds pages whose embedding model/dimensions changed, not just pages with NULL embeddings. After a model or dims swap, the stored `pages.embedding_signature` differs from the current one, so the next `embed --stale` (or per-source `embed-backfill` job) invalidates and re-embeds those pages. GRANDFATHER: pages embedded before v0.41.31.0 have a NULL signature and are NEVER flagged stale, so the upgrade does NOT trigger a whole-corpus re-embed. `embed --stale --dry-run` counts signature-drift without mutating. +- `gbrain sources status` gains a `BACKFILL` column (`active(N)` / `queued(N)` / `idle`) between `EMBED` and `FAILS` so an operator can see deferred embedding work after `sync --all` exits 0. + Key commands added in v0.36.4.0 (brain-health-100 wave): - `gbrain doctor --remediation-plan [--target-score N] [--json]` — preview the dependency-ordered plan that would drive the brain to target. JSON envelope is stable: each `Remediation` carries `id`, `idempotency_key` (content-hash for cron-safe retries), `severity`, `est_seconds`, `est_usd_cost`, and `depends_on` (referencing other ids). Empty `recommendations` array when the brain is already at target. - `gbrain doctor --remediate [--yes] [--target-score N] [--max-usd N]` — actually submit the plan. Walks dependency order, submits one Minion job per step, re-checks score between steps, refuses to spend past `--max-usd` (defaults: target=90, max-usd=infinite — but cron callers should always pass `--max-usd`). Bails when target exceeds `maxReachableScore()` for the brain (empty / under-configured brains) with a clear list of what's missing. diff --git a/package.json b/package.json index 85a907ce1..fb5a56b40 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.30.0" + "version": "0.41.31.0" } diff --git a/src/commands/embed.ts b/src/commands/embed.ts index 920d1936e..95274986a 100644 --- a/src/commands/embed.ts +++ b/src/commands/embed.ts @@ -1,5 +1,5 @@ import type { BrainEngine } from '../core/engine.ts'; -import { embedBatch } from '../core/embedding.ts'; +import { embedBatch, currentEmbeddingSignature } from '../core/embedding.ts'; import type { ChunkInput } from '../core/types.ts'; import { chunkText } from '../core/chunkers/recursive.ts'; import { createProgress, type ProgressReporter } from '../core/progress.ts'; @@ -378,6 +378,16 @@ async function embedPage( })); await engine.upsertChunks(slug, updated, opts); + // v0.41.31: stamp provenance so a later model/dims swap is detectable as + // stale. embedPage is the per-slug path used by `gbrain embed ` AND + // by `gbrain sync`'s post-import embed step (runEmbedCore({slugs})). + // Guard: only stamp when EVERY chunk was (re)embedded this pass. If some + // chunks were preserved from a prior embed (unknown/old provenance), the + // page is mixed — don't claim it's current. `embed --all` fully re-embeds + // such a page and then stamps it. + if (toEmbed.length === chunks.length) { + await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() }); + } result.embedded += toEmbed.length; result.pages_processed++; slog(`${slug}: embedded ${toEmbed.length} chunks`); @@ -396,6 +406,10 @@ async function embedAll( catchUp?: boolean; }, ) { + // v0.41.31: current embedding provenance signature. Stamped onto pages + // when their chunks are (re)embedded so a later model/dimension swap is + // detectable as stale. + const signature = currentEmbeddingSignature(); // ───────────────────────────────────────────────────────────── // Stale-only fast path: avoid the listPages + per-page getChunks // bomb that pulled every page row + every chunk's embedding column @@ -412,7 +426,7 @@ async function embedAll( if (staleOnly) { // D7: thread sourceId so `gbrain embed --stale --source X` actually scopes. // v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path. - return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts); + return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature); } // v0.31.12: when sourceId is set, scope listPages to that source. @@ -482,6 +496,9 @@ async function embedAll( token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); await engine.upsertChunks(page.slug, updated, pageOpts); + // v0.41.31: stamp embedding provenance so a later model swap is + // detectable as stale. + await engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }); result.embedded += toEmbed.length; } catch (e: unknown) { serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`); @@ -543,13 +560,31 @@ async function embedAllStale( priority?: 'recent'; catchUp?: boolean; }, + signature?: string, ) { // D7: thread sourceId so source-scoped runs only count + visit // that source's NULL embeddings. const sourceOpt = sourceId ? { sourceId } : undefined; + // v0.41.31: re-embed pages whose embedding_signature drifted (model/dims + // swap). dry-run must NOT mutate, so it counts signature-stale via the + // widened predicate; a live run NULLs them first so the existing + // NULL-embedding cursor (listStaleChunks) picks them up unchanged. + if (!dryRun && signature) { + const invalidated = await engine.invalidateStaleSignatureEmbeddings({ + signature, + ...(sourceId && { sourceId }), + }); + if (invalidated > 0) { + slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`); + } + } + // Pre-flight: 0 stale chunks → nothing to do, no further DB reads. - const staleCount = await engine.countStaleChunks(sourceOpt); + // dry-run includes signature-drift in the count without mutating. + const staleCount = await engine.countStaleChunks( + dryRun && signature ? { ...sourceOpt, signature } : sourceOpt, + ); if (staleCount === 0) { if (dryRun) { slog('[dry-run] Would embed 0 chunks (0 stale found)'); @@ -671,6 +706,14 @@ async function embedAllStale( token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); await engine.upsertChunks(slug, merged, { sourceId: keySourceId }); + // v0.41.31: stamp provenance after the page's chunks are embedded — + // but only when EVERY chunk was stale (fully re-embedded this pass). + // A partially-stale page keeps preserved chunks of unknown/old + // provenance, so don't claim it's current. (After invalidate, a + // signature-drifted page IS fully stale → this stamps it.) + if (signature && stale.length === existing.length) { + await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }); + } result.embedded += stale.length; } catch (e: unknown) { // Budget-fired aborts are expected on the way out; don't spam diff --git a/src/commands/sources.ts b/src/commands/sources.ts index 72618492f..e7620048b 100644 --- a/src/commands/sources.ts +++ b/src/commands/sources.ts @@ -599,22 +599,29 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise { return; } - // Human-readable table: SOURCE | LAG | EMBED | FAILS | QUEUE | PAGES | LAST SYNC + // Human-readable table: SOURCE | LAG | EMBED | BACKFILL | FAILS | QUEUE | PAGES | LAST SYNC console.log('SOURCES — health'); console.log('────────────────'); console.log( - ` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`, + ` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'BACKFILL'.padEnd(9)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`, ); for (const m of metrics) { const lag = m.lag_seconds === null ? 'never' : formatLag(m.lag_seconds); const embed = `${m.embed_coverage_pct.toFixed(0)}%`; + // v0.41.31: embed-backfill state (active beats queued beats idle) so a + // cron operator sees deferred embedding work after `sync --all`. + const backfill = m.backfill_active > 0 + ? `active(${m.backfill_active})` + : m.backfill_queued > 0 + ? `queued(${m.backfill_queued})` + : 'idle'; const fails = String(m.failed_jobs_24h); const queue = String(m.queue_depth); const pages = m.total_pages.toLocaleString(); const sync = m.last_sync_at ? new Date(m.last_sync_at).toISOString().slice(0, 19).replace('T', ' ') : 'never'; - console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`); + console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${backfill.padEnd(9)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`); } console.log(''); for (const m of metrics) { diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 41628afe1..41244e5d8 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -17,7 +17,17 @@ import { formatCodeBreakdown, } from '../core/sync.ts'; import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts'; -import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts'; +import { + estimateEmbeddingCostUsd, + getEmbeddingModelName, + currentEmbeddingPricePerMTok, + currentEmbeddingSignature, + willEmbedSynchronously, + shouldBlockSync, +} from '../core/embedding.ts'; +import { estimateCostFromChars } from '../core/embedding-pricing.ts'; +import { isSourceUnchangedSinceSync } from '../core/git-head.ts'; +import { SPEND_CAP_CONFIG_KEY } from '../core/embed-backfill-submit.ts'; import { errorFor, serializeError } from '../core/errors.ts'; import type { SyncManifest } from '../core/sync.ts'; import { createProgress } from '../core/progress.ts'; @@ -78,64 +88,114 @@ export interface SyncResult { } /** - * v0.20.0 Cathedral II Layer 8 (D1) — walk each source's working tree and - * sum tokens for every syncable file. This is a conservative overestimate - * (full file content, not just the incremental diff) because `sync --all` - * on a source that hasn't been synced yet WILL embed every file in the - * working tree. For already-synced sources with only incremental changes, - * the overestimate is the ceiling, not the floor — users never get - * surprised by MORE cost than the preview claims. The false-high bias is - * intentional: a lower estimate that undersells the real bill would be - * worse than one that oversells. + * Walk ONE source's working tree and sum tokens for every syncable file. + * Conservative full-tree ceiling (full file content, not the incremental + * diff) — over-counts, never under-counts, and matches the filesystem set + * `sync` actually imports (collectSyncableFiles + content_hash, NOT a git + * commit diff). Best-effort per file and per source: anything unreadable + * contributes 0 rather than blocking the preview. + * + * v0.31.2: routed through collectSyncableFiles (lstat + inode-cycle + + * max-depth) so the preview walks exactly what the real sync walks. */ -function estimateSyncAllCost(sources: Array<{ local_path: string | null; config: Record }>): { - totalTokens: number; - totalFiles: number; - activeSources: number; - perSource: Array<{ path: string; tokens: number; files: number }>; -} { - let totalTokens = 0; - let totalFiles = 0; - let activeSources = 0; - const perSource: Array<{ path: string; tokens: number; files: number }> = []; +function estimateSourceTreeTokens( + localPath: string, + strategy: 'markdown' | 'code' | 'auto', +): { tokens: number; files: number } { + let tokens = 0; + let files = 0; + try { + const fileList = collectSyncableFiles(localPath, { strategy }); + for (const fullPath of fileList) { + try { + const stat = statSync(fullPath); + if (stat.size > 5_000_000) continue; // skip large binaries + const content = readFileSync(fullPath, 'utf-8'); + tokens += estimateTokens(content); + files++; + } catch { + // Best-effort per file; sync itself tolerates the same. + } + } + } catch { + // Best-effort: a source whose local_path is gone/unreadable contributes 0. + } + return { tokens, files }; +} +/** + * v0.41.31 — INLINE-path new-content estimate. Per source, contribute ZERO + * when the source is provably unchanged since its last sync (HEAD == + * last_commit AND clean working tree AND chunker_version matches CURRENT) — + * `content_hash` short-circuits every file so nothing re-embeds. Otherwise + * contribute the full-tree ceiling. The unchanged predicate mirrors + * doctor's `sync_freshness` and sync's own "do work?" gate (sync.ts:1057+ + * 1075). `isSourceUnchangedSinceSync` is fail-open (probe error → false), so + * a source we can't prove unchanged is conservatively re-estimated rather + * than silently priced at $0. + */ +function estimateInlineNewTokens( + sources: Array<{ + local_path: string | null; + config: Record; + last_commit: string | null; + chunker_version: string | null; + }>, + currentChunkerVersion: string, +): { tokens: number; changedSources: number; unchangedSources: number } { + let tokens = 0; + let changedSources = 0; + let unchangedSources = 0; for (const src of sources) { if (!src.local_path) continue; const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' }; if (cfg.syncEnabled === false) continue; - activeSources++; - let sourceTokens = 0; - let sourceFiles = 0; - try { - // v0.31.2: cost preview routed through collectSyncableFiles - // (single hardened walker; see import.ts). Previously - // walkSyncableFiles used statSync (followed symlinks). New walker - // uses lstat + inode-cycle + max-depth so the preview matches - // what the real sync will actually walk. - const files = collectSyncableFiles(src.local_path, { strategy: cfg.strategy ?? 'markdown' }); - for (const fullPath of files) { - try { - const stat = statSync(fullPath); - if (stat.size > 5_000_000) continue; // skip large binaries - const content = readFileSync(fullPath, 'utf-8'); - sourceTokens += estimateTokens(content); - sourceFiles++; - } catch { - // Best-effort per file. Skip unreadable files silently; - // sync itself tolerates the same. - } - } - } catch { - // Best-effort: a source whose local_path is gone or unreadable just - // contributes 0. The sync itself would have failed anyway; no point - // blocking the preview on a pre-existing fault. + const unchanged = + isSourceUnchangedSinceSync(src.local_path, src.last_commit, { requireCleanWorkingTree: true }) && + src.chunker_version === currentChunkerVersion; + if (unchanged) { + unchangedSources++; + continue; } - totalTokens += sourceTokens; - totalFiles += sourceFiles; - perSource.push({ path: src.local_path, tokens: sourceTokens, files: sourceFiles }); + changedSources++; + tokens += estimateSourceTreeTokens(src.local_path, cfg.strategy ?? 'markdown').tokens; } + return { tokens, changedSources, unchangedSources }; +} - return { totalTokens, totalFiles, activeSources, perSource }; +/** + * Resolve the inline-path cost-gate floor in USD. Config key + * `sync.cost_gate_min_usd` (DB plane), default $0.50. Below this estimate + * the inline gate proceeds without blocking. Fail-open to the default on a + * missing/invalid value or a config-read error (the gate must never crash + * the sync). Accepts 0 (an operator can set the floor to $0 to make the + * gate block on any nonzero inline cost). + */ +async function resolveCostGateFloorUsd(engine: BrainEngine): Promise { + try { + const raw = await engine.getConfig('sync.cost_gate_min_usd'); + if (raw === null || raw === undefined) return 0.5; + const n = Number(raw); + return Number.isFinite(n) && n >= 0 ? n : 0.5; + } catch { + return 0.5; + } +} + +/** + * Resolve the per-source embed-backfill 24h spend cap (USD) for the deferred + * notice. Mirrors embed-backfill-submit.ts's own resolution of + * SPEND_CAP_CONFIG_KEY (default 25). Fail-open to the default. + */ +async function resolveBackfillCapUsd(engine: BrainEngine): Promise { + try { + const raw = await engine.getConfig(SPEND_CAP_CONFIG_KEY); + if (raw === null || raw === undefined) return 25; + const n = Number(raw); + return Number.isFinite(n) && n > 0 ? n : 25; + } catch { + return 25; + } } /** Interactive [y/N] prompt. Resolves false on non-y answers or EOF. */ @@ -2231,62 +2291,149 @@ See also: // source (no checkout) has nothing for `sync` to pull. Sources with // syncEnabled=false in config.jsonb are skipped too. if (syncAll) { - const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record }>( - `SELECT id, name, local_path, config FROM sources WHERE local_path IS NOT NULL`, + // v0.41.31: SELECT carries last_commit + chunker_version so the inline + // cost preview's "unchanged source → 0" short-circuit can mirror sync's + // own "do work?" gate (sync.ts:1057+1075) + doctor's sync_freshness. + // Both columns predate v0.41 (writeSyncAnchor / writeChunkerVersion); no + // schema migration needed. + const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record; last_commit: string | null; chunker_version: string | null }>( + `SELECT id, name, local_path, config, last_commit, chunker_version FROM sources WHERE local_path IS NOT NULL`, ); if (!sources || sources.length === 0) { console.log('No sources with local_path configured. Use `gbrain sources add --path ` first.'); return; } - // v0.20.0 Cathedral II Layer 8 D1 — cost preview + ConfirmationRequired - // gate. Before kicking off a multi-source sync that may embed tens of - // thousands of chunks (real money), walk the sync-diff set(s), sum - // tokens, compute USD estimate, and gate: - // - TTY + !json + !yes → interactive [y/N] prompt - // - non-TTY OR --json OR piped → emit ConfirmationRequired envelope, - // exit 2 (reserve 1 for runtime errors) - // - --yes → skip prompt entirely - // - --dry-run → preview + exit 0 - // Skipped entirely when --no-embed is set (user already opted out of - // the cost and will run `embed --stale` later). + // v0.41.31 — mode-aware cost gate. Resolve federated_v2 ONCE here so both + // the gate (below) and the fan-out (further down) share it. + const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); + const v2Enabled = await isFederatedV2Enabled(engine); + + // v0.41.31 cost gate (supersedes the v0.20.0 unconditional gate). Under + // federated_v2 sync DEFERS embedding to per-source embed-backfill jobs + // that carry their own $X/source/24h spend cap, so sync itself spends + // nothing synchronously — the gate is INFORMATIONAL (never exit 2) on + // that path. The blocking ConfirmationRequired gate fires ONLY when embed + // runs INLINE (v2 off, or --serial without --no-embed) AND the estimated + // spend exceeds `sync.cost_gate_min_usd` (default $0.50). Skipped entirely + // when --no-embed is set (user opted out; will run `embed --stale` later). if (!noEmbed) { - const preview = estimateSyncAllCost(sources); - const costUsd = estimateEmbeddingCostUsd(preview.totalTokens); - const previewMsg = - `sync --all preview: ${preview.totalFiles} files across ${preview.activeSources} source(s), ` + - `~${preview.totalTokens.toLocaleString()} tokens, est. $${costUsd.toFixed(2)} on ${EMBEDDING_MODEL}.`; - - if (dryRun) { - if (jsonOut) { - console.log(JSON.stringify({ status: 'dry_run', preview, costUsd, model: EMBEDDING_MODEL })); - } else { - console.log(previewMsg); - console.log('--dry-run: exit without syncing.'); - } - return; + const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed }); + // Stale backlog: cheap single SQL; fail-open to 0 so a transient DB + // hiccup never blocks the sync. + let staleChars = 0; + try { + // v0.41.31: signature-aware so a model/dims swap surfaces in the + // backlog estimate (NULL signature grandfathered → not counted). + staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() }); + } catch { + staleChars = 0; } + const rate = currentEmbeddingPricePerMTok(); + const staleCostUsd = estimateCostFromChars(staleChars, rate); + const embeddingModelName = getEmbeddingModelName(); + const floorUsd = await resolveCostGateFloorUsd(engine); - if (!yesFlag) { - const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); - if (!isTTY || jsonOut) { - // Agent-facing path: emit structured envelope, exit 2. - const envelope = serializeError(errorFor({ - class: 'ConfirmationRequired', - code: 'cost_preview_requires_yes', - message: previewMsg, - hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.', - })); - console.log(JSON.stringify({ error: envelope, preview, costUsd, model: EMBEDDING_MODEL })); - process.exit(2); + if (mode === 'deferred') { + // Deferred path: print an FYI, NEVER exit 2. The backfill cap is the + // real money gate (D1/D4). + const capUsd = await resolveBackfillCapUsd(engine); + // v0.41.31 (TODO-2): surface already-queued backfill jobs so a cron + // operator sees work is enqueued, not lost. Best-effort — minion_jobs + // may not exist on a brain that never ran a worker. + let queuedBackfills = 0; + try { + const r = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM minion_jobs + WHERE name = 'embed-backfill' + AND status IN ('waiting','active','delayed','waiting-children')`, + ); + queuedBackfills = Number(r[0]?.n) || 0; + } catch { + queuedBackfills = 0; } - // Interactive TTY path: prompt [y/N]. - console.log(previewMsg); - const answer = await promptYesNo('Proceed? [y/N] '); - if (!answer) { - console.log('Cancelled.'); + const deferredMsg = + `sync --all: embedding deferred to backfill jobs ` + + `(capped $${capUsd}/source/24h, not charged by this sync). ` + + `Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` + + `${embeddingModelName}) across ${sources.length} source(s); ` + + `${queuedBackfills} backfill job(s) queued.`; + if (dryRun) { + if (jsonOut) { + console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName })); + } else { + console.log(deferredMsg); + console.log('--dry-run: exit without syncing.'); + } return; } + if (jsonOut) { + console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName })); + } else { + console.log(deferredMsg); + } + // fall through to sync — no exit 2. + } else { + // Inline path: sync embeds synchronously with no backfill cap to + // protect it, so the blocking gate applies. The BLOCKING cost is the + // new-content estimate ONLY (full-tree ceiling for changed sources; + // unchanged contribute 0) — that's what this sync actually embeds. + // The pre-existing stale backlog (NULL embeddings + signature drift) + // is NOT swept by sync; `gbrain embed --stale` clears it. So we show + // it informationally but never gate on cost this sync won't incur + // (else a model swap would block the next inline cron — F2). + const currentChunkerVersion = String(CHUNKER_VERSION); + const inline = estimateInlineNewTokens(sources, currentChunkerVersion); + const newCostUsd = estimateEmbeddingCostUsd(inline.tokens); + const costUsd = newCostUsd; + const staleNote = staleChars > 0 + ? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)` + : ''; + const previewMsg = + `sync --all preview (inline embed): ${inline.changedSources} changed source(s), ` + + `${inline.unchangedSources} unchanged; ~${inline.tokens.toLocaleString()} new tokens, ` + + `est. $${costUsd.toFixed(2)} on ${embeddingModelName}${staleNote}.`; + + if (dryRun) { + if (jsonOut) { + console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); + } else { + console.log(previewMsg); + console.log('--dry-run: exit without syncing.'); + } + return; + } + + if (!yesFlag) { + if (shouldBlockSync(costUsd, floorUsd, mode)) { + const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); + if (!isTTY || jsonOut) { + // Agent-facing path: emit structured envelope, exit 2. + const envelope = serializeError(errorFor({ + class: 'ConfirmationRequired', + code: 'cost_preview_requires_yes', + message: previewMsg, + hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.', + })); + console.log(JSON.stringify({ error: envelope, mode, gate: 'confirmation_required', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); + process.exit(2); + } + // Interactive TTY path: prompt [y/N]. + console.log(previewMsg); + const answer = await promptYesNo('Proceed? [y/N] '); + if (!answer) { + console.log('Cancelled.'); + return; + } + } else { + // Below floor → proceed without blocking (kills inline-cron noise). + if (jsonOut) { + console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); + } else { + console.log(`${previewMsg} Below cost gate floor ($${floorUsd.toFixed(2)}), proceeding.`); + } + } + } } } @@ -2302,8 +2449,7 @@ See also: // - withSourcePrefix wrap inside runOne so slog/serr lines from // performSync get the [] prefix under parallel mode (D6) // - stable JSON envelope {schema_version:1, sources, ...} when --json - const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); - const v2Enabled = await isFederatedV2Enabled(engine); + // v0.41.31: v2Enabled resolved once above (cost gate). Reused here. const activeSources = sources.filter((s) => { const cfg = (s.config || {}) as { syncEnabled?: boolean }; return cfg.syncEnabled !== false; @@ -2791,6 +2937,13 @@ export interface SyncStatusReportSource { chunks_total: number; chunks_unembedded: number; embedding_coverage_pct: number; + // v0.41.31: embed-backfill job visibility (federated_v2 defers embedding + // to these jobs; without this an operator can't see queued/lagging work + // after `sync --all` exits 0). Best-effort — all 0 / null on brains + // without the minion_jobs table. + backfill_queued: number; + backfill_active: number; + backfill_last_completed_at: string | null; } export interface SyncStatusReport { @@ -2892,6 +3045,43 @@ export async function buildSyncStatusReport( }); } + // v0.41.31: per-source embed-backfill job state. Best-effort — the + // minion_jobs table doesn't exist on every brain (a brain that never ran + // a worker has the pre-minions schema), and the dashboard must not crash + // for that. A failure → empty map → all sources report 0/null. + type BackfillRow = { + source_id: string | null; + queued: string | number; + active: string | number; + last_completed_at: string | Date | null; + }; + const backfillMap = new Map(); + if (sourceIds.length > 0) { + try { + const backfillRows = await engine.executeRaw( + `SELECT data->>'sourceId' AS source_id, + COUNT(*) FILTER (WHERE status IN ('waiting','delayed','waiting-children'))::int AS queued, + COUNT(*) FILTER (WHERE status = 'active')::int AS active, + MAX(finished_at) FILTER (WHERE status = 'completed') AS last_completed_at + FROM minion_jobs + WHERE name = 'embed-backfill' AND data->>'sourceId' = ANY($1::text[]) + GROUP BY data->>'sourceId'`, + [sourceIds], + ); + for (const r of backfillRows) { + if (!r.source_id) continue; + const last = r.last_completed_at; + backfillMap.set(r.source_id, { + queued: Number(r.queued) || 0, + active: Number(r.active) || 0, + last_completed_at: last == null ? null : (last instanceof Date ? last.toISOString() : last), + }); + } + } catch { + // minion_jobs absent / unreadable → leave backfillMap empty. + } + } + const now = Date.now(); const out: SyncStatusReportSource[] = sources.map((src) => { const cfgEntry = (src.config || {}) as { syncEnabled?: boolean }; @@ -2928,6 +3118,9 @@ export async function buildSyncStatusReport( chunks_total: counts.chunks_total, chunks_unembedded: counts.chunks_unembedded, embedding_coverage_pct: embeddingCoveragePct, + backfill_queued: backfillMap.get(src.id)?.queued ?? 0, + backfill_active: backfillMap.get(src.id)?.active ?? 0, + backfill_last_completed_at: backfillMap.get(src.id)?.last_completed_at ?? null, }; }); @@ -2968,7 +3161,7 @@ export function printSyncStatusReport( write(' (no sources registered)'); return; } - const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'LAST SYNC']; + const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'BACKFILL', 'LAST SYNC']; const rows = report.sources.map((s) => { const stale = s.staleness_hours === null ? 'never' @@ -2976,21 +3169,28 @@ export function printSyncStatusReport( const stateBits: string[] = []; if (!s.sync_enabled) stateBits.push('disabled'); stateBits.push(s.staleness_class); + // BACKFILL: active beats queued beats idle for the at-a-glance cell. + const backfill = s.backfill_active > 0 + ? `active(${s.backfill_active})` + : s.backfill_queued > 0 + ? `queued(${s.backfill_queued})` + : 'idle'; return [ s.name, stateBits.join(','), stale, String(s.pages), `${s.embedding_coverage_pct}%`, + backfill, s.last_sync_at ?? '(never)', ]; }); const widths = headers.map((h, i) => Math.max(h.length, ...rows.map((r) => r[i].length)), ); - // Numeric columns (PAGES at index 3, EMBEDDED at index 4, STALENESS - // at index 2) right-pad-left so digits align cleanly. Text columns - // left-pad-right per the existing `sources list` convention. + // Numeric columns (STALENESS=2, PAGES=3, EMBEDDED=4) right-pad-left so + // digits align cleanly. Text columns (incl. BACKFILL=5) left-pad-right + // per the existing `sources list` convention. const NUMERIC_COLS = new Set([2, 3, 4]); const fmt = (cells: string[]) => cells.map((c, i) => (NUMERIC_COLS.has(i) ? c.padStart(widths[i]) : c.padEnd(widths[i]))).join(' '); diff --git a/src/core/embed-stale.ts b/src/core/embed-stale.ts index 8451d6fa1..5ca95cd78 100644 --- a/src/core/embed-stale.ts +++ b/src/core/embed-stale.ts @@ -51,6 +51,14 @@ export interface EmbedStaleOpts { * the gateway. Production callers leave it unset. */ embedFn?: (texts: string[], opts: { abortSignal?: AbortSignal }) => Promise; + /** + * v0.41.31: current embedding provenance signature (`:`). + * When set, embeddings stamped under a DIFFERENT signature are invalidated + * (NULLed) at the start so they flow through the NULL cursor and get + * re-embedded; each page's signature is stamped after its chunks land. + * Omit to keep the legacy `embedding IS NULL`-only behavior. + */ + embeddingSignature?: string; } export interface EmbedStaleResult { @@ -109,6 +117,18 @@ export async function embedStaleForSource( done: false, aborted: false, }; + const signature = opts.embeddingSignature; + + // v0.41.31: invalidate embeddings stamped under a prior model signature so + // the NULL cursor below re-embeds them. GRANDFATHER: NULL signature + // untouched. Best-effort — a failure here must not abort the backfill. + if (signature) { + try { + await engine.invalidateStaleSignatureEmbeddings({ signature, sourceId }); + } catch { + // Non-fatal: fall through to the NULL-only stale loop. + } + } for (;;) { if (signal?.aborted) { @@ -170,6 +190,13 @@ export async function embedStaleForSource( token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); await engine.upsertChunks(slug, merged, { sourceId: keySourceId }); + // v0.41.31: stamp provenance only when EVERY chunk was stale (fully + // re-embedded this pass) — a partially-stale page keeps preserved + // chunks of unknown provenance, so don't claim current. After the + // invalidate pass above, signature-drifted pages ARE fully stale. + if (signature && stale.length === existing.length) { + await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }); + } result.embedded += stale.length; result.pagesProcessed += 1; } catch (e: unknown) { diff --git a/src/core/embedding.ts b/src/core/embedding.ts index 4c1529544..7fe3c9148 100644 --- a/src/core/embedding.ts +++ b/src/core/embedding.ts @@ -12,6 +12,7 @@ import { getEmbeddingModel as gatewayGetModel, getEmbeddingDimensions as gatewayGetDims, } from './ai/gateway.ts'; +import { lookupEmbeddingPrice } from './embedding-pricing.ts'; // v0.27.1: re-export multimodal embedding so callers can pull both text and // image embedding APIs from `src/core/embedding`. import-image-file consumes @@ -127,13 +128,106 @@ export const EMBEDDING_MODEL = 'text-embedding-3-large'; export const EMBEDDING_DIMENSIONS = 1536; /** - * USD cost per 1k tokens for text-embedding-3-large. Used by - * `gbrain sync --all` cost preview and `reindex-code` to surface - * expected spend before accepting expensive operations. + * USD cost per 1k tokens for text-embedding-3-large. Retained for back-compat + * with callers/tests that import it directly; new cost math resolves the + * ACTUAL configured model's rate via embedding-pricing.ts instead of assuming + * OpenAI. (Hardcoding this rate produced cost previews that named the wrong + * provider and over-stated spend ~2.6x when the brain ran on a cheaper model.) */ export const EMBEDDING_COST_PER_1K_TOKENS = 0.00013; -/** Compute USD cost estimate for embedding `tokens` at current model rate. */ +/** + * Resolve the price-per-1M-tokens for the currently-configured embedding + * model. Falls back to the OpenAI text-embedding-3-large rate only when the + * model is unknown to the pricing table. + */ +export function currentEmbeddingPricePerMTok(): number { + let modelString: string; + try { + modelString = gatewayGetModel(); // e.g. 'zeroentropyai:zembed-1' + } catch { + // Gateway not configured (e.g. unit tests, cost preview before connect). + // Fall back to the OpenAI text-embedding-3-large default rate. + return 0.13; + } + const hit = lookupEmbeddingPrice(modelString); + return hit.kind === 'known' ? hit.pricePerMTok : 0.13; +} + +/** + * Compute USD cost estimate for embedding `tokens` at the CURRENT configured + * model's rate (not a hardcoded OpenAI rate). + */ export function estimateEmbeddingCostUsd(tokens: number): number { - return (tokens / 1000) * EMBEDDING_COST_PER_1K_TOKENS; + return (tokens / 1_000_000) * currentEmbeddingPricePerMTok(); +} + +/** + * Embedding provenance signature for the currently-configured model: + * `:` (e.g. `openai:text-embedding-3-large:1536`). + * Stamped onto `pages.embedding_signature` when a page's chunks are + * embedded so a later model/dimension swap can be detected as stale. + * + * Deliberately does NOT include the chunker version — chunker drift is + * already tracked per-page via `pages.chunker_version` (used by sync + + * doctor). This signature is strictly about the EMBEDDING space. + * + * Falls back to the OpenAI default signature when the gateway is + * unconfigured (unit-test context), matching the other estimator fallbacks. + */ +export function currentEmbeddingSignature(): string { + try { + return `${gatewayGetModel()}:${gatewayGetDims()}`; + } catch { + return `${EMBEDDING_MODEL}:${EMBEDDING_DIMENSIONS}`; + } +} + +/** + * Whether a `gbrain sync --all` invocation will embed at sync time + * ('inline') or defer embedding to per-source `embed-backfill` minion jobs + * ('deferred'). Under federated_v2 the default path defers; the backfill + * jobs carry their own 10-min cooldown + $25/source/24h spend cap, so the + * sync-time cost gate only BLOCKS on the inline path. See sync.ts:2346 + * (`effectiveNoEmbed`) — this mirrors that resolution exactly. + */ +export type SyncEmbedMode = 'deferred' | 'inline'; + +/** + * Resolve the embed mode from the same three signals sync.ts uses to + * compute `effectiveNoEmbed`. Single source of truth so the cost gate and + * the actual embed decision can never drift. + * + * effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed + * + * Embed runs INLINE iff that resolves to false: + * - v2 off → inline (legacy synchronous embed) + * - v2 on + --serial + !--no-embed → inline + * - v2 on (parallel) → deferred (backfill jobs) + * - --no-embed (any path) → the caller skips the gate entirely; + * we report 'deferred' for completeness. + */ +export function willEmbedSynchronously(opts: { + v2Enabled: boolean; + serialFlag: boolean; + noEmbed: boolean; +}): SyncEmbedMode { + const effectiveNoEmbed = + opts.v2Enabled && !opts.serialFlag && !opts.noEmbed ? true : opts.noEmbed; + return effectiveNoEmbed ? 'deferred' : 'inline'; +} + +/** + * Pure cost-gate decision. The gate BLOCKS (prompt in TTY, exit 2 envelope + * in non-TTY) only when embed runs inline AND the estimated spend exceeds + * the floor. Deferred mode NEVER blocks — the backfill cap is the real + * money gate, and blocking the cheap markdown import for cost the import + * doesn't synchronously incur is the bug this fix removes. + */ +export function shouldBlockSync( + costUsd: number, + floorUsd: number, + mode: SyncEmbedMode, +): boolean { + return mode === 'inline' && costUsd > floorUsd; } diff --git a/src/core/engine.ts b/src/core/engine.ts index b9861d0d9..f76baddbc 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -956,7 +956,38 @@ export interface BrainEngine { * `gbrain embed --stale --source media-corpus` expect only that * source's NULLs touched; the caller threads `sourceId` here. */ - countStaleChunks(opts?: { sourceId?: string }): Promise; + countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise; + /** + * Sum of LENGTH(chunk_text) over stale chunks — the character-count + * backlog the embed phase / embed-backfill will process. Sibling of + * countStaleChunks (same stale predicate + embed_skip filter + optional + * sourceId scope); used by the `gbrain sync --all` cost preview to price + * the embedding backlog via estimateCostFromChars. Returns 0 on an + * empty/fully-embedded brain. + * + * v0.41.31: `signature` (optional) widens "stale" to ALSO include chunks + * whose page `embedding_signature` is set AND differs from the current + * model signature (a model/dims swap). NULL signature is GRANDFATHERED + * (never counted) so the post-migration corpus isn't flagged en masse. + * Omit `signature` for the legacy `embedding IS NULL`-only count. + */ + sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise; + /** + * Stamp `pages.embedding_signature = signature` for one page. Called after + * a page's chunks are (re)embedded so a later model swap can detect it as + * stale. Idempotent. No-op if the page doesn't exist. + */ + setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise; + /** + * NULL out the embeddings (and embedded_at) of every chunk whose page + * `embedding_signature` is set AND differs from `signature` — i.e. pages + * embedded under a now-stale model. Returns the chunk count invalidated. + * The embed-stale loop calls this BEFORE listStaleChunks so signature- + * drift pages flow through the existing NULL-embedding cursor (keeps + * listStaleChunks's keyset pagination untouched). GRANDFATHER: NULL + * signature is never invalidated. `sourceId` scopes the sweep. + */ + invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise; /** * Return every chunk where embedding IS NULL, with the metadata needed * to call embedBatch + upsertChunks. The `embedding` column is omitted diff --git a/src/core/import-file.ts b/src/core/import-file.ts index 4dc34525b..3beb34328 100644 --- a/src/core/import-file.ts +++ b/src/core/import-file.ts @@ -8,7 +8,7 @@ import { chunkText } from './chunkers/recursive.ts'; import { chunkCodeText, chunkCodeTextFull, detectCodeLanguage, CHUNKER_VERSION } from './chunkers/code.ts'; import { findChunkForOffset } from './chunkers/edge-extractor.ts'; import { extractCodeRefs, imageOfCandidates } from './link-extraction.ts'; -import { embedBatch, embedMultimodal } from './embedding.ts'; +import { embedBatch, embedMultimodal, currentEmbeddingSignature } from './embedding.ts'; import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts'; import type { ChunkInput, PageInput, PageType } from './types.ts'; import { computeEffectiveDate } from './effective-date.ts'; @@ -658,6 +658,13 @@ export async function importFromContent( if (chunks.length > 0) { await tx.upsertChunks(slug, chunks, txOpts); + // v0.41.31: stamp embedding provenance when this import actually + // embedded (not --no-embed), so a later model/dims swap is detectable + // as stale via embed --stale. The deferred/backfill + per-slug embed + // paths stamp too; this covers the inline import/sync path. + if (!opts.noEmbed) { + await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() }); + } } else { // Content is empty — delete stale chunks so they don't ghost in search results await tx.deleteChunks(slug, txOpts); @@ -968,6 +975,14 @@ export async function importCodeFile( if (chunks.length > 0) { await tx.upsertChunks(slug, chunks, txOpts); + // v0.41.31: stamp embedding provenance ONLY when every chunk was + // freshly embedded with the current model this call (no reuse-by-hash + // carrying old-model vectors). Mixed pages stay unstamped rather than + // falsely marked current; `reindex --code --force` / `embed --stale` + // handle the swap for those. + if (!opts.noEmbed && needsEmbedIndexes.length === chunks.length) { + await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() }); + } } else { await tx.deleteChunks(slug, txOpts); } diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 4de08e538..b326ef980 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4920,6 +4920,32 @@ export const MIGRATIONS: Migration[] = [ EXECUTE FUNCTION bump_page_generation_clock_fn(); `, }, + { + version: 108, + name: 'pages_embedding_signature', + // v0.41.31 — embedding provenance for real stale semantics. + // + // Adds `pages.embedding_signature TEXT NULL` = `:` + // stamped when a page's chunks are embedded (setPageEmbeddingSignature). + // A later model/dimension swap makes the stored signature differ from + // the current one, so countStaleChunks/sumStaleChunkChars (with the + // `signature` opt) and invalidateStaleSignatureEmbeddings can detect and + // re-embed those pages. + // + // GRANDFATHER (critical): the stale predicate is + // `embedding_signature IS NOT NULL AND embedding_signature <> $current` + // so a NULL signature is NEVER stale. After this migration every existing + // page has NULL — none are flagged — so the next `embed --stale` does NOT + // re-embed the whole corpus. Signatures only get stamped going forward. + // + // No index: the column is read only via a JOINed pages row in the + // chunk-grain stale queries; no standalone lookup hot path. ADD COLUMN + // with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. + idempotent: true, + sql: ` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT NULL; + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/handlers/embed-backfill.ts b/src/core/minions/handlers/embed-backfill.ts index 02fd0dcad..221d5b676 100644 --- a/src/core/minions/handlers/embed-backfill.ts +++ b/src/core/minions/handlers/embed-backfill.ts @@ -35,6 +35,7 @@ import { tryAcquireDbLock } from '../../db-lock.ts'; import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts'; import { withBudgetTracker } from '../../ai/gateway.ts'; import { embedStaleForSource } from '../../embed-stale.ts'; +import { currentEmbeddingSignature } from '../../embedding.ts'; import type { BrainEngine } from '../../engine.ts'; import type { MinionJobContext } from '../types.ts'; @@ -123,6 +124,9 @@ export function makeEmbedBackfillHandler(engine: BrainEngine) { embedStaleForSource(engine, sourceId, { batchSize, signal: job.signal, + // v0.41.31: re-embed pages whose model signature drifted + stamp + // provenance as chunks land. + embeddingSignature: currentEmbeddingSignature(), onProgress: ({ embedded, chunksProcessed, cursor }) => { // Fire-and-forget; updateProgress returns a Promise but the // handler is sync inside the loop. diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 612d5de5f..f6063c754 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -422,7 +422,9 @@ export class PGLiteEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema='public' AND table_name='sources' AND column_name='trust_frontmatter_overrides') AS sources_trust_fm_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists + WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='embedding_signature') AS pages_embedding_signature_exists `); const probe = rows[0] as { pages_exists: boolean; @@ -463,6 +465,7 @@ export class PGLiteEngine implements BrainEngine { sources_cr_mode_exists: boolean; sources_trust_fm_exists: boolean; pages_generation_exists: boolean; + pages_embedding_signature_exists: boolean; }; const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists; @@ -530,6 +533,10 @@ export class PGLiteEngine implements BrainEngine { // body; bootstrap only needs to add the column on pre-v91 brains so // the CREATE INDEX doesn't crash. const needsPagesGeneration = probe.pages_exists && !probe.pages_generation_exists; + // v0.41.31 (v108): pages.embedding_signature for real stale semantics. + // No SCHEMA_SQL index references it today; bootstrap is defense-in-depth + // so future schema work doesn't wedge pre-v108 brains. + const needsPagesEmbeddingSignature = probe.pages_exists && !probe.pages_embedding_signature_exists; // Fresh installs (no tables yet) and modern brains both no-op. if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap @@ -539,7 +546,8 @@ export class PGLiteEngine implements BrainEngine { && !needsFilesBootstrap && !needsOauthClientsBootstrap && !needsSourcesArchive && !needsPagesLastRetrievedAt && !needsPagesProvenance - && !needsContextualRetrievalColumns && !needsPagesGeneration) return; + && !needsContextualRetrievalColumns && !needsPagesGeneration + && !needsPagesEmbeddingSignature) return; process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n'); @@ -767,6 +775,15 @@ export class PGLiteEngine implements BrainEngine { ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1; `); } + + if (needsPagesEmbeddingSignature) { + // v108 (pages_embedding_signature): embedding provenance for real + // stale semantics. NULL grandfathered (never stale). v108 runs later + // via runMigrations and is idempotent. + await this.db.exec(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT; + `); + } } async withReservedConnection(fn: (conn: ReservedConnection) => Promise): Promise { @@ -2061,35 +2078,91 @@ export class PGLiteEngine implements BrainEngine { return (rows as Record[]).map(r => rowToChunk(r)); } - async countStaleChunks(opts?: { sourceId?: string }): Promise { - // D7: source-scoped count for `gbrain embed --stale --source X`. - // v0.41 (D4+D8+Codex r2 #11): always JOIN pages so embed-skip filter - // applies via `NOT (frontmatter ? 'embed_skip')`. PGLite is - // PostgreSQL 17.5 in WASM and supports the full JSONB operator set. - if (opts?.sourceId === undefined) { - const { rows } = await this.db.query( - `SELECT count(*)::int AS count - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`, - ); - const count = (rows[0] as { count: number } | undefined)?.count ?? 0; - return Number(count); + /** + * Build the stale-chunk WHERE clause + positional params. embed_skip is + * always excluded. `signature` widens "stale" to include embedding_signature + * drift (NULL grandfathered → never stale). Shared by countStaleChunks + + * sumStaleChunkChars so they can't drift. + */ + private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } { + const params: unknown[] = []; + const conds: string[] = []; + if (opts?.signature !== undefined) { + params.push(opts.signature); + conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`); + } else { + conds.push(`cc.embedding IS NULL`); } + conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`); + if (opts?.sourceId !== undefined) { + params.push(opts.sourceId); + conds.push(`p.source_id = $${params.length}`); + } + return { where: conds.join(' AND '), params }; + } + + async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise { + // D7: source-scoped count for `gbrain embed --stale --source X`. Always + // JOIN pages so embed-skip + signature predicates apply. PGLite is + // PostgreSQL 17.5 in WASM and supports the full JSONB operator set. + const { where, params } = this.buildStaleChunkWhere(opts); const { rows } = await this.db.query( `SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND p.source_id = $1 - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`, - [opts.sourceId], + WHERE ${where}`, + params, ); const count = (rows[0] as { count: number } | undefined)?.count ?? 0; return Number(count); } + async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise { + // Sibling of countStaleChunks: same stale predicate, summing chunk_text + // length for the sync cost preview. ::bigint guards int4 overflow. + const { where, params } = this.buildStaleChunkWhere(opts); + const { rows } = await this.db.query( + `SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE ${where}`, + params, + ); + const chars = (rows[0] as { chars: number | string } | undefined)?.chars ?? 0; + return Number(chars); + } + + async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise { + await this.db.query( + `UPDATE pages SET embedding_signature = $1 WHERE slug = $2 AND source_id = $3`, + [opts.signature, slug, opts.sourceId ?? 'default'], + ); + } + + async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise { + // NULL out embeddings whose page signature is set AND differs from the + // current model signature. GRANDFATHER: NULL signature untouched. Feeds + // the existing NULL-embedding cursor so listStaleChunks stays unchanged. + const params: unknown[] = [opts.signature]; + let srcClause = ''; + if (opts.sourceId !== undefined) { + params.push(opts.sourceId); + srcClause = ` AND p.source_id = $${params.length}`; + } + const { rows } = await this.db.query( + `UPDATE content_chunks cc + SET embedding = NULL, embedded_at = NULL + FROM pages p + WHERE cc.page_id = p.id + AND cc.embedding IS NOT NULL + AND p.embedding_signature IS NOT NULL + AND p.embedding_signature <> $1${srcClause} + RETURNING cc.page_id`, + params, + ); + return (rows as unknown[]).length; + } + async listStaleChunks(opts?: { batchSize?: number; afterPageId?: number; diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 2fb90fd39..48db794ca 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -455,7 +455,9 @@ export class PostgresEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'trust_frontmatter_overrides') AS sources_trust_fm_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'embedding_signature') AS pages_embedding_signature_exists `; const probe = probeRows[0]!; @@ -530,6 +532,7 @@ export class PostgresEngine implements BrainEngine { sources_cr_mode_exists?: boolean; sources_trust_fm_exists?: boolean; pages_generation_exists?: boolean; + pages_embedding_signature_exists?: boolean; }; const needsContextualRetrievalColumns = (probe.pages_exists && (!probeCr.pages_cr_mode_exists || !probeCr.pages_corpus_generation_exists)) @@ -540,6 +543,9 @@ export class PostgresEngine implements BrainEngine { // it. Pre-v91 brains crash without the column; bootstrap adds it before // SCHEMA_SQL replay creates the index. const needsPagesGeneration = probe.pages_exists && !probeCr.pages_generation_exists; + // v0.41.31 (v108): pages.embedding_signature for real stale semantics. + // No SCHEMA_SQL index references it; bootstrap is defense-in-depth. + const needsPagesEmbeddingSignature = probe.pages_exists && !probeCr.pages_embedding_signature_exists; if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt && !needsMcpLogBootstrap && !needsSubagentProviderId @@ -548,7 +554,8 @@ export class PostgresEngine implements BrainEngine { && !needsOauthClientsBootstrap && !needsSourcesArchive && !needsPagesLastRetrievedAt && !needsPagesProvenance - && !needsContextualRetrievalColumns && !needsPagesGeneration) return; + && !needsContextualRetrievalColumns && !needsPagesGeneration + && !needsPagesEmbeddingSignature) return; process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n'); @@ -774,6 +781,15 @@ export class PostgresEngine implements BrainEngine { ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1; `); } + + if (needsPagesEmbeddingSignature) { + // v108 (pages_embedding_signature): embedding provenance for real stale + // semantics. NULL grandfathered. v108 runs later via runMigrations and + // is idempotent. + await conn.unsafe(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT; + `); + } } async transaction(fn: (engine: BrainEngine) => Promise): Promise { @@ -2093,36 +2109,88 @@ export class PostgresEngine implements BrainEngine { return rows.map((r) => rowToChunk(r as Record)); } - async countStaleChunks(opts?: { sourceId?: string }): Promise { - const sql = this.sql; - // v0.41 (D4+D8+Codex r2 #11): the embed-skip filter requires JOIN - // pages so we always join — the pre-v0.41 "fast path" without join - // is gone. JSONB `?` existence check is cheap on the small set of - // skipped pages; full-scan benefits from the partial index on - // embedding IS NULL regardless. - // - // D7: source_id scoping. NULL/undefined = scan all sources; - // a value scopes to that source so `gbrain embed --stale --source X` - // does what it says. - if (opts?.sourceId === undefined) { - const [row] = await sql` - SELECT count(*)::int AS count - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') - `; - return Number((row as { count?: number } | undefined)?.count ?? 0); + /** + * Build the stale-chunk WHERE clause + positional params for sql.unsafe. + * embed_skip always excluded. `signature` widens "stale" to include + * embedding_signature drift (NULL grandfathered). Shared by + * countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling). + */ + private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } { + const params: unknown[] = []; + const conds: string[] = []; + if (opts?.signature !== undefined) { + params.push(opts.signature); + conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`); + } else { + conds.push(`cc.embedding IS NULL`); } - const [row] = await sql` - SELECT count(*)::int AS count - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND p.source_id = ${opts.sourceId} - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') + conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`); + if (opts?.sourceId !== undefined) { + params.push(opts.sourceId); + conds.push(`p.source_id = $${params.length}`); + } + return { where: conds.join(' AND '), params }; + } + + async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise { + // Always JOIN pages so the embed_skip + signature predicates apply. + // D7: source_id scoping. v0.41.31: optional signature widens staleness + // to embedding_signature drift (NULL grandfathered). + const { where, params } = this.buildStaleChunkWhere(opts); + const rows = await this.sql.unsafe( + `SELECT count(*)::int AS count + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE ${where}`, + params as Parameters[1], + ); + return Number((rows[0] as { count?: number } | undefined)?.count ?? 0); + } + + async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise { + // Sibling of countStaleChunks: same stale predicate, summing chunk_text + // length for the sync cost preview. ::bigint guards int4 overflow. + const { where, params } = this.buildStaleChunkWhere(opts); + const rows = await this.sql.unsafe( + `SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE ${where}`, + params as Parameters[1], + ); + return Number((rows[0] as { chars?: number | string } | undefined)?.chars ?? 0); + } + + async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise { + const sql = this.sql; + await sql` + UPDATE pages SET embedding_signature = ${opts.signature} + WHERE slug = ${slug} AND source_id = ${opts.sourceId ?? 'default'} `; - return Number((row as { count?: number } | undefined)?.count ?? 0); + } + + async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise { + // NULL embeddings whose page signature is set AND differs from current. + // GRANDFATHER: NULL signature untouched. Feeds the NULL-embedding cursor + // so listStaleChunks stays unchanged. RETURNING → row count. + const params: unknown[] = [opts.signature]; + let srcClause = ''; + if (opts.sourceId !== undefined) { + params.push(opts.sourceId); + srcClause = ` AND p.source_id = $${params.length}`; + } + const rows = await this.sql.unsafe( + `UPDATE content_chunks cc + SET embedding = NULL, embedded_at = NULL + FROM pages p + WHERE cc.page_id = p.id + AND cc.embedding IS NOT NULL + AND p.embedding_signature IS NOT NULL + AND p.embedding_signature <> $1${srcClause} + RETURNING cc.page_id`, + params as Parameters[1], + ); + return (rows as unknown[]).length; } async listStaleChunks(opts?: { diff --git a/src/core/source-health.ts b/src/core/source-health.ts index 13135724a..8553a8358 100644 --- a/src/core/source-health.ts +++ b/src/core/source-health.ts @@ -33,6 +33,10 @@ export interface SourceMetrics { failed_jobs_24h: number; /** Waiting + active + delayed jobs (sync OR embed-backfill) for this source. */ queue_depth: number; + /** v0.41.31: embed-backfill jobs specifically active right now. */ + backfill_active: number; + /** v0.41.31: embed-backfill jobs queued (waiting/delayed/waiting-children). */ + backfill_queued: number; tracked_branch: string | null; priority_label: PriorityLabel; /** Webhook configured? (true iff config.webhook_secret is set.) */ @@ -131,7 +135,7 @@ export async function computeAllSourceMetrics( const cfg = parseSourceConfig(src.config); const pages = pageCounts.get(src.id) ?? 0; const chunkStats = chunkCounts.get(src.id) ?? { total: 0, embedded: 0 }; - const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0 }; + const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0, backfill_active: 0, backfill_queued: 0 }; const embedCoverage = chunkStats.total === 0 ? 100 @@ -155,6 +159,8 @@ export async function computeAllSourceMetrics( lag_seconds: lagSeconds, failed_jobs_24h: jobStats.failed_24h, queue_depth: jobStats.queue_depth, + backfill_active: jobStats.backfill_active, + backfill_queued: jobStats.backfill_queued, tracked_branch: typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : null, priority_label: resolvePriorityLabel(src.id, src.config), webhook_configured: typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0, @@ -189,21 +195,30 @@ async function chunkCountsBySource(engine: BrainEngine): Promise> { +type JobStats = { failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number }; + +async function jobCountsBySource(engine: BrainEngine): Promise> { // Pre-v0.11 brains don't have minion_jobs; return empty map. try { - const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number }>( + const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number }>( `SELECT data->>'sourceId' AS source_id, COUNT(*) FILTER (WHERE status IN ('failed','dead') AND created_at > NOW() - INTERVAL '24 hours')::int AS failed_24h, - COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth + COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth, + COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status = 'active')::int AS backfill_active, + COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status IN ('waiting','delayed','waiting-children'))::int AS backfill_queued FROM minion_jobs WHERE name IN ('sync','embed-backfill') AND data->>'sourceId' IS NOT NULL GROUP BY data->>'sourceId'`, ); - const m = new Map(); + const m = new Map(); for (const r of rows) { - m.set(r.source_id, { failed_24h: Number(r.failed_24h), queue_depth: Number(r.queue_depth) }); + m.set(r.source_id, { + failed_24h: Number(r.failed_24h), + queue_depth: Number(r.queue_depth), + backfill_active: Number(r.backfill_active), + backfill_queued: Number(r.backfill_queued), + }); } return m; } catch { diff --git a/test/cosine-rescore-column.test.ts b/test/cosine-rescore-column.test.ts index 7e576ac50..9010b9ba1 100644 --- a/test/cosine-rescore-column.test.ts +++ b/test/cosine-rescore-column.test.ts @@ -19,6 +19,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { EmbeddingColumnNotRegisteredError, } from '../src/core/search/embedding-column.ts'; +import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts'; import type { PageInput, ChunkInput } from '../src/core/types.ts'; let engine: PGLiteEngine; @@ -26,6 +27,17 @@ let chunkId: number; let chunkId2: number; beforeAll(async () => { + // Pin the embedding dim to 1536 BEFORE initSchema. initSchema sizes the + // `embedding` column from getEmbeddingDimensions() (default 1280 = + // zeroentropyai). This test hardcodes 1536-dim vectors + asserts 1536, so + // it must NOT inherit ambient/leaked gateway state (which is 1536 from a + // local ~/.gbrain config but 1280 in CI → vector(1280) → insert fails). + // Pinning here makes the column deterministically 1536 regardless of order. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test-cosine-rescore' }, + }); engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); @@ -87,6 +99,7 @@ beforeAll(async () => { afterAll(async () => { await engine.disconnect(); + resetGateway(); }); describe('getEmbeddingsByChunkIds — column parameter (D9)', () => { diff --git a/test/e2e/cycle.test.ts b/test/e2e/cycle.test.ts index 95b73e4ab..e49319f99 100644 --- a/test/e2e/cycle.test.ts +++ b/test/e2e/cycle.test.ts @@ -25,6 +25,8 @@ mock.module('../../src/core/embedding.ts', () => ({ // Deterministic fake vector for each chunk. return texts.map(() => new Float32Array(1536)); }, + // v0.41.31: embed phase reads the current signature to stamp provenance. + currentEmbeddingSignature: () => 'test:model:1536', })); const { runCycle } = await import('../../src/core/cycle.ts'); diff --git a/test/e2e/dream-cycle-phase-order-pglite.test.ts b/test/e2e/dream-cycle-phase-order-pglite.test.ts index 151a79ca1..f434f1e57 100644 --- a/test/e2e/dream-cycle-phase-order-pglite.test.ts +++ b/test/e2e/dream-cycle-phase-order-pglite.test.ts @@ -46,6 +46,8 @@ mock.module('../../src/core/embedding.ts', () => ({ EMBEDDING_DIMENSIONS: 1536, EMBEDDING_COST_PER_1K_TOKENS: 0.00013, estimateEmbeddingCostUsd: (tokens: number) => (tokens / 1000) * 0.00013, + // v0.41.31: embed phase reads the current signature to stamp provenance. + currentEmbeddingSignature: () => 'text-embedding-3-large:1536', })); const { runCycle, ALL_PHASES } = await import('../../src/core/cycle.ts'); @@ -106,6 +108,9 @@ async function withoutAnthropicKey(body: () => Promise): Promise { // v0.31 — added `consolidate` between recompute_emotional_weight and embed // v0.33 — added `resolve_symbol_edges` between extract and patterns type CyclePhase = (typeof ALL_PHASES)[number]; +// Mirrors src/core/cycle.ts ALL_PHASES order exactly. v0.41.31: synced the +// three phases that drifted in after this test was last touched (v0.41.0.0): +// extract_atoms (v0.41 T9), synthesize_concepts, conversation_facts_backfill. const EXPECTED_PHASES: CyclePhase[] = [ 'lint', 'backlinks', @@ -113,13 +118,16 @@ const EXPECTED_PHASES: CyclePhase[] = [ 'synthesize', 'extract', 'extract_facts', // v0.32.2 — reconcile fence → DB facts index + 'extract_atoms', // v0.41 T9 — pack-gated atom extraction 'resolve_symbol_edges', // v0.33.3 — within-file symbol resolution 'patterns', + 'synthesize_concepts', // v0.41 — concept synthesis after patterns 'recompute_emotional_weight', // v0.29 'consolidate', // v0.31 'propose_takes', // v0.36.1.0 — hindsight calibration wave 'grade_takes', // v0.36.1.0 'calibration_profile', // v0.36.1.0 + 'conversation_facts_backfill', // v0.41.11.0 — config-gated (default off) 'embed', 'orphans', 'schema-suggest', // v0.39.0.0 — passive schema-suggest after orphans diff --git a/test/e2e/dream.test.ts b/test/e2e/dream.test.ts index 867af2ce8..7687003c9 100644 --- a/test/e2e/dream.test.ts +++ b/test/e2e/dream.test.ts @@ -19,6 +19,8 @@ import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers. // Mock embedBatch so embed phase doesn't call OpenAI. mock.module('../../src/core/embedding.ts', () => ({ embedBatch: async (texts: string[]) => texts.map(() => new Float32Array(1536)), + // v0.41.31: embed phase reads the current signature to stamp provenance. + currentEmbeddingSignature: () => 'test:model:1536', })); const { runDream } = await import('../../src/commands/dream.ts'); diff --git a/test/e2e/sync-status-pglite.test.ts b/test/e2e/sync-status-pglite.test.ts index 8a8d993e9..0773ec4a5 100644 --- a/test/e2e/sync-status-pglite.test.ts +++ b/test/e2e/sync-status-pglite.test.ts @@ -197,6 +197,43 @@ describe('buildSyncStatusReport against real PGLite (IRON RULE regression for Bl expect(b.embedding_coverage_pct).toBe(100); }); + test('v0.41.31: embed-backfill job state surfaced per source (TODO-2)', async () => { + // Seed embed-backfill minion jobs for source-a: 2 queued + 1 active + + // 1 completed. source-b has none → idle. + await engine.executeRaw( + `INSERT INTO minion_jobs (name, status, data) VALUES + ('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb), + ('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb), + ('embed-backfill', 'active', '{"sourceId":"source-a"}'::jsonb)`, + ); + await engine.executeRaw( + `INSERT INTO minion_jobs (name, status, data, finished_at) + VALUES ('embed-backfill', 'completed', '{"sourceId":"source-a"}'::jsonb, now())`, + ); + + const sources = await engine.executeRaw<{ + id: string; name: string; local_path: string | null; config: Record; + }>( + `SELECT id, name, local_path, config FROM sources + WHERE local_path IS NOT NULL AND archived IS NOT TRUE ORDER BY id`, + ); + const report = await buildSyncStatusReport(engine, sources); + const byId = new Map(report.sources.map((s) => [s.source_id, s])); + + const a = byId.get('source-a')!; + expect(a.backfill_queued).toBe(2); + expect(a.backfill_active).toBe(1); + expect(a.backfill_last_completed_at).not.toBeNull(); + + const b = byId.get('source-b')!; + expect(b.backfill_queued).toBe(0); + expect(b.backfill_active).toBe(0); + expect(b.backfill_last_completed_at).toBeNull(); + + // Clean up so sibling tests in this describe see a clean minion_jobs. + await engine.executeRaw(`DELETE FROM minion_jobs WHERE name = 'embed-backfill'`); + }); + test('soft-deleted pages excluded from pages count (v0.26.5 regression)', async () => { // Verifies the `WHERE pg.deleted_at IS NULL` clause in BOTH subqueries // of the dashboard SQL. Pre-fix the original PR query would have diff --git a/test/embed.serial.test.ts b/test/embed.serial.test.ts index 5caaa0fc5..ed8058140 100644 --- a/test/embed.serial.test.ts +++ b/test/embed.serial.test.ts @@ -32,6 +32,11 @@ mock.module('../src/core/embedding.ts', () => ({ activeEmbedCalls--; } }, + // v0.41.31: embedAll/embedAllStale read the current embedding signature to + // stamp provenance. The mock returns a stable value; the mock engine's + // setPageEmbeddingSignature / invalidateStaleSignatureEmbeddings resolve to + // null via the Proxy default, so the signature value is inert here. + currentEmbeddingSignature: () => 'test:model:1536', })); // Import AFTER mocking. @@ -105,6 +110,29 @@ describe('runEmbed --all (parallel)', () => { expect(maxConcurrentEmbedCalls).toBeLessThanOrEqual(10); }); + test('v0.41.31: stamps embedding_signature after embedding each page (--all)', async () => { + const pages = [{ slug: 'a', source_id: 'default' }, { slug: 'b', source_id: 'default' }]; + const chunksBySlug = new Map( + pages.map(p => [ + p.slug, + [{ chunk_index: 0, chunk_text: `text ${p.slug}`, chunk_source: 'compiled_truth', embedded_at: null, token_count: 4 }], + ]), + ); + const engine = mockEngine({ + listPages: async () => pages, + getChunks: async (slug: string) => chunksBySlug.get(slug) || [], + upsertChunks: async () => {}, + }); + + await runEmbed(engine, ['--all']); + + // The wiring gap this pins: embedAll must CALL setPageEmbeddingSignature + // after upsertChunks, with the current signature (mocked to test:model:1536). + const stampCalls = (engine as any)._calls.filter((c: any) => c.method === 'setPageEmbeddingSignature'); + expect(stampCalls.length).toBe(2); // one per page + expect(stampCalls[0].args[1]).toEqual({ sourceId: 'default', signature: 'test:model:1536' }); + }); + test('respects GBRAIN_EMBED_CONCURRENCY=1 (serial)', async () => { const pages = Array.from({ length: 5 }, (_, i) => ({ slug: `page-${i}` })); const chunksBySlug = new Map( diff --git a/test/embedding-signature-stale.test.ts b/test/embedding-signature-stale.test.ts new file mode 100644 index 000000000..c7d3e626e --- /dev/null +++ b/test/embedding-signature-stale.test.ts @@ -0,0 +1,121 @@ +/** + * v0.41.31 — real stale semantics via pages.embedding_signature (PGLite). + * + * Pins the commit-3 contract: + * - R-4 (grandfather, CRITICAL): a page embedded under a NULL signature is + * NEVER stale. After the v108 migration every existing page has NULL, so + * the next embed --stale must NOT re-embed the whole corpus. + * - signature mismatch (model/dims swap) → counted as stale. + * - matching signature → not stale. + * - invalidateStaleSignatureEmbeddings NULLs only mismatched (grandfathered + * NULL + matching untouched) and returns the count. + * - setPageEmbeddingSignature stamps. + * + * Canonical PGLite block (CLAUDE.md R3+R4). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +let colDim: number; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`, + ); + colDim = Number(rows[0]?.dim); +}); + +/** + * Seed a page with one EMBEDDED chunk (non-null vector) and a given + * embedding_signature (null → grandfathered legacy state). + */ +async function seedEmbedded(slug: string, text: string, signature: string | null, sourceId?: string): Promise { + await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }, sourceId ? { sourceId } : undefined); + const chunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined }, + ]; + await engine.upsertChunks(slug, chunks, sourceId ? { sourceId } : undefined); + // Flip the chunk to a non-null vector sized to the actual column dim. + await engine.executeRaw( + `UPDATE content_chunks + SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector + WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = $3)`, + [colDim, slug, sourceId ?? 'default'], + ); + if (signature !== null) { + await engine.setPageEmbeddingSignature(slug, { sourceId, signature }); + } +} + +describe('embedding_signature stale semantics', () => { + test('R-4 GRANDFATHER: NULL signature is never stale', async () => { + await seedEmbedded('legacy', 'abcde', null); // embedded, NULL signature + // No NULL embeddings, NULL signature → not stale under any signature. + expect(await engine.countStaleChunks({ signature: 'openai:m:1536' })).toBe(0); + expect(await engine.sumStaleChunkChars({ signature: 'openai:m:1536' })).toBe(0); + }); + + test('signature MISMATCH (model swap) is counted as stale', async () => { + await seedEmbedded('drifted', 'abcde', 'openai:old:1536'); // 5 chars + expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(1); + expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(5); + // Without the signature opt, the legacy NULL-only predicate ignores it. + expect(await engine.countStaleChunks()).toBe(0); + }); + + test('MATCHING signature is not stale', async () => { + await seedEmbedded('fresh', 'abcde', 'voyage:new:1024'); + expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(0); + expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(0); + }); + + test('invalidateStaleSignatureEmbeddings NULLs only mismatched; grandfathered + matching untouched', async () => { + await seedEmbedded('old', 'abcde', 'openai:old:1536'); // mismatched → invalidate + await seedEmbedded('legacy', 'fghij', null); // grandfathered → keep + await seedEmbedded('new', 'klmno', 'voyage:new:1024'); // matching → keep + + const invalidated = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' }); + expect(invalidated).toBe(1); // only 'old' + + // Now exactly the 'old' page's chunk is NULL → legacy stale count = 1. + expect(await engine.countStaleChunks()).toBe(1); + // Re-running is idempotent (nothing left to invalidate). + expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' })).toBe(0); + }); + + test('invalidate is sourceId-scoped', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{}'::jsonb) ON CONFLICT (id) DO NOTHING`, + ); + await seedEmbedded('a', 'abcde', 'openai:old:1536'); // default + await seedEmbedded('b', 'fghij', 'openai:old:1536', 'other'); // other + const n = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024', sourceId: 'default' }); + expect(n).toBe(1); + expect(await engine.countStaleChunks({ sourceId: 'default' })).toBe(1); + expect(await engine.countStaleChunks({ sourceId: 'other' })).toBe(0); // untouched + }); + + test('setPageEmbeddingSignature stamps the page', async () => { + await engine.putPage('p', { type: 'note', title: 'p', compiled_truth: '# p' }); + await engine.setPageEmbeddingSignature('p', { signature: 'openai:m:1536' }); + const rows = await engine.executeRaw<{ embedding_signature: string | null }>( + `SELECT embedding_signature FROM pages WHERE slug = 'p' AND source_id = 'default'`, + ); + expect(rows[0]?.embedding_signature).toBe('openai:m:1536'); + }); +}); diff --git a/test/import-signature-stamp.serial.test.ts b/test/import-signature-stamp.serial.test.ts new file mode 100644 index 000000000..4614f0fb4 --- /dev/null +++ b/test/import-signature-stamp.serial.test.ts @@ -0,0 +1,76 @@ +/** + * v0.41.31 (F1) — importFromContent stamps pages.embedding_signature when it + * embeds inline. + * + * The inline import/sync path (importFromContent) writes embeddings without + * going through embed.ts/embed-stale.ts. Before the F1 fix it never stamped + * the signature, so non-federated/inline brains kept NULL signatures forever + * → grandfathered → `embed --stale` never re-embedded them after a model swap + * (the headline feature silently inert). This pins the stamp. + * + * Serial: stubs the gateway embed transport (process-global module state). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { importFromContent } from '../src/core/import-file.ts'; +import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts'; +import { currentEmbeddingSignature } from '../src/core/embedding.ts'; + +let engine: PGLiteEngine; +let colDim: number; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`, + ); + colDim = Number(rows[0]?.dim); + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: colDim, + env: { OPENAI_API_KEY: 'sk-test-import-stamp' }, + }); + // Fake transport (AI SDK embedMany shape): receives { values }, returns + // one zero-vector (sized to the column) per input value. + __setEmbedTransportForTests(async ({ values }: { values: string[] }) => ({ + embeddings: values.map(() => Array(colDim).fill(0)), + usage: { tokens: 0 }, + }) as any); +}); + +afterEach(() => { + __setEmbedTransportForTests(null); + resetGateway(); +}); + +async function signatureOf(slug: string): Promise { + const rows = await engine.executeRaw<{ embedding_signature: string | null }>( + `SELECT embedding_signature FROM pages WHERE slug = $1 AND source_id = 'default'`, + [slug], + ); + return rows[0]?.embedding_signature ?? null; +} + +describe('importFromContent embedding_signature stamping (F1)', () => { + test('embeds inline → stamps the current signature', async () => { + await importFromContent(engine, 'concepts/stamped', '# Stamped\n\nsome body content to chunk and embed.', {}); + expect(await signatureOf('concepts/stamped')).toBe(currentEmbeddingSignature()); + }); + + test('--no-embed → leaves signature NULL (grandfathered, not stale)', async () => { + await importFromContent(engine, 'concepts/unstamped', '# Unstamped\n\nbody content.', { noEmbed: true }); + expect(await signatureOf('concepts/unstamped')).toBeNull(); + }); +}); diff --git a/test/schema-bootstrap-coverage.test.ts b/test/schema-bootstrap-coverage.test.ts index 64f761427..d81043f61 100644 --- a/test/schema-bootstrap-coverage.test.ts +++ b/test/schema-bootstrap-coverage.test.ts @@ -158,6 +158,10 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [ // pages_generation_idx (CREATE INDEX ON pages (generation)) so bootstrap // probes guard pre-v91 brains. { kind: 'column', table: 'pages', column: 'generation' }, + // v0.41.31 (v108) — pages.embedding_signature TEXT for real stale + // semantics. No SCHEMA_SQL index references it; bootstrap probe is + // defense-in-depth (and satisfies the MIGRATIONS ADD COLUMN coverage gate). + { kind: 'column', table: 'pages', column: 'embedding_signature' }, ]; test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => { diff --git a/test/source-health.test.ts b/test/source-health.test.ts index 5f4bc80b4..e59d237dd 100644 --- a/test/source-health.test.ts +++ b/test/source-health.test.ts @@ -167,6 +167,24 @@ describe('computeAllSourceMetrics', () => { expect(result.find((m) => m.source_id === 'other')!.total_pages).toBe(1); }); + test('v0.41.31: embed-backfill active/queued counts per source (CLI BACKFILL column)', async () => { + // 2 queued + 1 active embed-backfill for default; a non-backfill 'sync' + // job must NOT inflate the backfill counts (only the generic queue_depth). + await engine.executeRaw( + `INSERT INTO minion_jobs (name, status, data) VALUES + ('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb), + ('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb), + ('embed-backfill', 'active', '{"sourceId":"default"}'::jsonb), + ('sync', 'waiting', '{"sourceId":"default"}'::jsonb)`, + ); + const sources = await loadAllSources(engine); + const dflt = (await computeAllSourceMetrics(engine, sources)).find((m) => m.source_id === 'default')!; + expect(dflt.backfill_active).toBe(1); + expect(dflt.backfill_queued).toBe(2); + // generic queue_depth includes the sync job too (4 total waiting/active). + expect(dflt.queue_depth).toBe(4); + }); + test('webhook_configured reflects config.webhook_secret presence', async () => { await engine.executeRaw( `INSERT INTO sources (id, name, config) VALUES ('webhooky', 'webhooky', '{"federated":true,"webhook_secret":"x","github_repo":"a/b"}'::jsonb)`, diff --git a/test/sum-stale-chunk-chars.test.ts b/test/sum-stale-chunk-chars.test.ts new file mode 100644 index 000000000..399226752 --- /dev/null +++ b/test/sum-stale-chunk-chars.test.ts @@ -0,0 +1,113 @@ +/** + * v0.41.31 — BrainEngine.sumStaleChunkChars tests (PGLite). + * + * Sibling of countStaleChunks: sums LENGTH(chunk_text) over stale chunks so + * the `gbrain sync --all` cost preview can price the embedding backlog. + * Validates: empty brain → 0, exact char sum, sourceId scope, embed_skip + * exclusion, and that non-null (already-embedded) chunks are NOT counted. + * + * Uses the canonical PGLite block (one engine per file, resetPgliteState in + * beforeEach) per CLAUDE.md test-isolation rules R3+R4. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +/** Seed a page + stale (NULL-embedding) chunks with explicit chunk_text. */ +async function seedStale( + slug: string, + texts: string[], + opts?: { sourceId?: string; frontmatter?: Record }, +): Promise { + await engine.putPage( + slug, + { + type: 'note', + title: slug, + compiled_truth: `# ${slug}`, + ...(opts?.frontmatter ? { frontmatter: opts.frontmatter } : {}), + }, + opts?.sourceId ? { sourceId: opts.sourceId } : undefined, + ); + const chunks: ChunkInput[] = texts.map((t, i) => ({ + chunk_index: i, + chunk_text: t, + chunk_source: 'compiled_truth', + token_count: 4, + embedding: undefined, // NULL = stale + })); + await engine.upsertChunks(slug, chunks, opts?.sourceId ? { sourceId: opts.sourceId } : undefined); +} + +describe('sumStaleChunkChars', () => { + test('empty brain returns 0', async () => { + expect(await engine.sumStaleChunkChars()).toBe(0); + }); + + test('sums LENGTH(chunk_text) across stale chunks', async () => { + await seedStale('a', ['abcde', 'fghij']); // 5 + 5 + await seedStale('b', ['xyz']); // 3 + expect(await engine.sumStaleChunkChars()).toBe(13); + }); + + test('scopes to sourceId when provided', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}'::jsonb) ON CONFLICT (id) DO NOTHING`, + ); + await seedStale('a', ['abcde']); // default, 5 + await seedStale('b', ['xyzwv'], { sourceId: 'other' }); // other, 5 + expect(await engine.sumStaleChunkChars({ sourceId: 'default' })).toBe(5); + expect(await engine.sumStaleChunkChars({ sourceId: 'other' })).toBe(5); + expect(await engine.sumStaleChunkChars()).toBe(10); // all sources + }); + + test('excludes pages with embed_skip frontmatter', async () => { + await seedStale('keep', ['abcde']); // 5, counted + await seedStale('skip', ['1234567890'], { + frontmatter: { embed_skip: { at: '2026-01-01T00:00:00Z', reason: 'oversize' } }, + }); + expect(await engine.sumStaleChunkChars()).toBe(5); + }); + + test('does NOT count already-embedded (non-null) chunks', async () => { + // Stale page → counted. + await seedStale('stale', ['abcde']); // 5 + // "Embedded" page: seed as stale, then flip its chunk's embedding to a + // non-null vector via raw SQL. We bypass upsertChunks so the test doesn't + // depend on gateway-configured dimensions (keeps it robust regardless of + // any leaked gateway state from sibling files). + await seedStale('done', ['this is embedded and should not count']); + // Size the embedding to the ACTUAL column dimension (pgvector stores it + // in atttypmod) so the test is robust to whatever dim initSchema chose. + const dimRows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`, + ); + const dim = Number(dimRows[0]?.dim); + expect(dim).toBeGreaterThan(0); + await engine.executeRaw( + `UPDATE content_chunks + SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector + WHERE page_id = (SELECT id FROM pages WHERE slug = 'done' AND source_id = 'default')`, + [dim], + ); + expect(await engine.sumStaleChunkChars()).toBe(5); + }); +}); diff --git a/test/sync-cost-gate.serial.test.ts b/test/sync-cost-gate.serial.test.ts new file mode 100644 index 000000000..48d7b057d --- /dev/null +++ b/test/sync-cost-gate.serial.test.ts @@ -0,0 +1,160 @@ +/** + * v0.41.31 — `gbrain sync --all` cost-gate wiring regressions (PGLite). + * + * Pure shouldBlockSync / willEmbedSynchronously logic is pinned in + * test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in + * runSync's --all path: + * + * R-1 (headline): deferred-embed sync --all, non-TTY, with backlog → + * emits gate:'deferred_notice' and NEVER exit 2 (the cron-blocking + * bug this release fixes). + * R-2 (protection): inline-embed sync --all (--serial), non-TTY, above + * floor → still exit 2 with gate:'confirmation_required'. + * + * Serial-quarantined: stubs process.exit + console.log (process-global). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runSources } from '../src/commands/sources.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts'; +import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +let repoPath: string; +let headSha: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 60_000); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}, 60_000); + +beforeEach(async () => { + await resetPgliteState(engine); + // Configure the gateway with a dummy key so the pre-gate embedding-creds + // preflight passes (diagnoseEmbedding reads gateway configure-time state, + // not live env). The gate runs before any real embed call, so no network + // request is made. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test-costgate' }, + }); + repoPath = mkdtempSync(join(tmpdir(), 'gbrain-costgate-')); + execSync('git init', { cwd: repoPath, stdio: 'pipe' }); + execSync('git config user.email "t@t.com"', { cwd: repoPath, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repoPath, stdio: 'pipe' }); + mkdirSync(join(repoPath, 'topics'), { recursive: true }); + writeFileSync( + join(repoPath, 'topics/foo.md'), + ['---', 'type: concept', 'title: Foo', '---', '', 'some body content to estimate.'].join('\n'), + ); + execSync('git add -A && git commit -m initial', { cwd: repoPath, stdio: 'pipe' }); + headSha = execSync('git rev-parse HEAD', { cwd: repoPath, stdio: 'pipe' }).toString().trim(); +}); + +afterEach(() => { + resetGateway(); + if (repoPath) rmSync(repoPath, { recursive: true, force: true }); +}); + +/** Run runSync(args) with process.exit + console.log captured. */ +async function runSyncCaptured(args: string[]): Promise<{ exitCode: number | undefined; stdout: string }> { + const { runSync } = await import('../src/commands/sync.ts'); + const origExit = process.exit; + const origLog = console.log.bind(console); + const out: string[] = []; + let exitCode: number | undefined; + console.log = (...a: unknown[]) => { + out.push(a.map((x) => (typeof x === 'string' ? x : JSON.stringify(x))).join(' ')); + }; + process.exit = ((code?: number) => { + exitCode = code; + throw new Error('__exit__'); + }) as typeof process.exit; + try { + await runSync(engine, args); + } catch (e) { + if ((e as Error).message !== '__exit__') throw e; + } finally { + process.exit = origExit; + console.log = origLog; + } + return { exitCode, stdout: out.join('\n') }; +} + +describe('v0.41.31 — sync --all cost gate wiring', () => { + test('R-1: deferred sync --all (non-TTY) emits deferred_notice and never exit 2', async () => { + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + // Make the fan-out a clean no-op: last_commit == HEAD so performSync + // reports up_to_date (no git pull, no backfill submit). + await engine.executeRaw(`UPDATE sources SET last_commit = $1 WHERE id = 'vault'`, [headSha]); + // Seed a stale backlog so the deferred notice has a non-zero figure. + await engine.putPage('vault/note', { type: 'note', title: 'note', compiled_truth: '# note' }, { sourceId: 'vault' }); + const chunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: 'x'.repeat(500), chunk_source: 'compiled_truth', token_count: 4, embedding: undefined }, + ]; + await engine.upsertChunks('vault/note', chunks, { sourceId: 'vault' }); + + // v2 default ON → deferred. --json → agent path. NO --yes. --no-pull + // because the synthetic repo has no remote. + const { exitCode, stdout } = await runSyncCaptured(['--all', '--json', '--no-pull']); + + // The headline assertion: NOT blocked. + expect(exitCode).not.toBe(2); + expect(stdout).toContain('"gate":"deferred_notice"'); + }, 60_000); + + test('R-2: inline sync --all (--serial) above floor still exit 2', async () => { + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + // Floor 0 → any nonzero inline cost blocks. Source is unsynced + // (last_commit NULL) so estimateInlineNewTokens sees it as changed → + // full-tree tokens > 0 → costUsd > 0 > floor. + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + // --serial forces inline even with v2 on. --json → non-TTY exit-2 path. + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).toBe(2); + expect(stdout).toContain('"gate":"confirmation_required"'); + }, 60_000); + + test('R-3: inline, git-unchanged source but STALE chunker_version still estimates (not $0)', async () => { + // The unchanged-source short-circuit requires HEAD==last_commit AND clean + // tree AND chunker_version == current. Here git is unchanged but the + // chunker drifted, so the source must NOT be treated as 0 — sync would + // re-chunk + re-embed everything. floor=0 so any nonzero cost blocks. + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, 'STALE-0']); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).toBe(2); + expect(stdout).toContain('"gate":"confirmation_required"'); + }, 60_000); + + test('R-3 control: inline, git-unchanged + CURRENT chunker_version short-circuits to $0 (no exit 2)', async () => { + // Same setup but chunker_version matches current → the source IS unchanged + // → contributes 0 new-content tokens → below floor → proceeds (no block). + // Proves the short-circuit fires when (and only when) everything matches. + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).not.toBe(2); + expect(stdout).not.toContain('"gate":"confirmation_required"'); + }, 60_000); +}); diff --git a/test/sync-cost-preview.test.ts b/test/sync-cost-preview.test.ts index 22fb3408a..7f492c032 100644 --- a/test/sync-cost-preview.test.ts +++ b/test/sync-cost-preview.test.ts @@ -14,23 +14,54 @@ */ import { describe, test, expect } from 'bun:test'; -import { EMBEDDING_COST_PER_1K_TOKENS, estimateEmbeddingCostUsd } from '../src/core/embedding.ts'; +import { + EMBEDDING_COST_PER_1K_TOKENS, + estimateEmbeddingCostUsd, + willEmbedSynchronously, + shouldBlockSync, +} from '../src/core/embedding.ts'; +import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts'; import { estimateTokens } from '../src/core/chunkers/code.ts'; describe('Layer 8 D1 — embedding cost model', () => { - test('EMBEDDING_COST_PER_1K_TOKENS is text-embedding-3-large pricing', () => { - // Update this when OpenAI changes text-embedding-3-large pricing. - // As of 2026-04-24: $0.00013 / 1k tokens. + test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => { + // Retained only for back-compat imports. Live cost math now resolves the + // CONFIGURED model's rate via embedding-pricing.ts (see model-aware test + // below). As of 2026-04-24: $0.00013 / 1k tokens. expect(EMBEDDING_COST_PER_1K_TOKENS).toBe(0.00013); }); - test('estimateEmbeddingCostUsd scales linearly with tokens', () => { + test('estimateEmbeddingCostUsd scales linearly (gateway-unconfigured fallback = OpenAI rate)', () => { + // With no gateway configured (unit-test context) the estimator falls back + // to the OpenAI text-embedding-3-large rate ($0.13/Mtok = $0.00013/1k). expect(estimateEmbeddingCostUsd(0)).toBe(0); expect(estimateEmbeddingCostUsd(1000)).toBeCloseTo(0.00013, 5); expect(estimateEmbeddingCostUsd(10_000)).toBeCloseTo(0.0013, 4); expect(estimateEmbeddingCostUsd(1_000_000)).toBeCloseTo(0.13, 4); }); + test('cost preview uses the CONFIGURED model rate, not a hardcoded OpenAI rate', () => { + // Regression: the cost gate previously hardcoded $0.00013/1k (OpenAI + // text-embedding-3-large) regardless of the configured embedding model, + // so a brain on a cheaper model (e.g. zeroentropyai:zembed-1 @ $0.05/Mtok) + // saw a preview that named the wrong provider and over-stated spend ~2.6x. + // The pricing table is the single source of truth per provider:model. + const TOKENS = 2_590_710_262; // a real large-brain sync preview + const openai = lookupEmbeddingPrice('openai:text-embedding-3-large'); + const zeroentropy = lookupEmbeddingPrice('zeroentropyai:zembed-1'); + expect(openai.kind).toBe('known'); + expect(zeroentropy.kind).toBe('known'); + if (openai.kind === 'known' && zeroentropy.kind === 'known') { + const openaiCost = (TOKENS / 1_000_000) * openai.pricePerMTok; + const zeCost = (TOKENS / 1_000_000) * zeroentropy.pricePerMTok; + // The two models must produce materially different previews; a fix that + // collapses both to the OpenAI number would regress this assertion. + expect(openaiCost).toBeCloseTo(336.79, 1); + expect(zeCost).toBeCloseTo(129.54, 1); + expect(zeCost).toBeLessThan(openaiCost); + } + }); + test('5K-file TS repo sanity check: ~$5 at ~400k tokens', () => { // A 5K-file TS repo at ~80 tokens/file averages ~400k tokens. Cost: // 400_000 / 1000 * 0.00013 = $0.052 ≈ $0.05. Not $5. The CHANGELOG @@ -43,6 +74,50 @@ describe('Layer 8 D1 — embedding cost model', () => { }); }); +describe('v0.41.31 — willEmbedSynchronously (embed-mode resolver)', () => { + // Mirrors sync.ts:2346 effectiveNoEmbed = v2 && !serial && !noEmbed ? true : noEmbed. + // Embed runs INLINE iff that resolves to false. + test('v2 off → inline (legacy synchronous embed)', () => { + expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: false })).toBe('inline'); + }); + test('v2 on + parallel → deferred (backfill jobs)', () => { + expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: false, noEmbed: false })).toBe('deferred'); + }); + test('v2 on + --serial → inline', () => { + expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: false })).toBe('inline'); + }); + test('--no-embed forces deferred regardless of v2/serial', () => { + expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: true })).toBe('deferred'); + expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: true })).toBe('deferred'); + }); +}); + +describe('v0.41.31 — shouldBlockSync (cost-gate decision)', () => { + // R-1: deferred NEVER blocks, even at absurd cost (the headline fix — a + // nightly cron over a synced corpus must not exit 2). + test('R-1: deferred never blocks, even at $999', () => { + expect(shouldBlockSync(999, 0.5, 'deferred')).toBe(false); + expect(shouldBlockSync(0, 0.5, 'deferred')).toBe(false); + }); + // R-2: inline still blocks above the floor (protection preserved where + // sync actually spends synchronously). + test('R-2: inline blocks above floor', () => { + expect(shouldBlockSync(0.51, 0.5, 'inline')).toBe(true); + expect(shouldBlockSync(130, 0.5, 'inline')).toBe(true); + }); + test('inline at exactly the floor does NOT block (boundary)', () => { + expect(shouldBlockSync(0.5, 0.5, 'inline')).toBe(false); + }); + test('inline below floor does not block (kills cents-level cron noise)', () => { + expect(shouldBlockSync(0.03, 0.5, 'inline')).toBe(false); + expect(shouldBlockSync(0, 0.5, 'inline')).toBe(false); + }); + test('floor of 0 makes inline block on any nonzero cost', () => { + expect(shouldBlockSync(0.0001, 0, 'inline')).toBe(true); + expect(shouldBlockSync(0, 0, 'inline')).toBe(false); + }); +}); + describe('Layer 8 D1 — estimateTokens (exported from chunkers/code.ts)', () => { test('empty string is 0 tokens', () => { expect(estimateTokens('')).toBe(0);