diff --git a/CHANGELOG.md b/CHANGELOG.md index 92a85a5c8..10af4533e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,110 @@ All notable changes to GBrain will be documented in this file. +## [0.41.31.0] - 2026-05-30 + +**Your nightly `gbrain sync --all` cron stops getting blocked. It used to +demand `--yes` on every non-interactive run even when nothing needed +embedding. Now it just runs. And when you change your embedding model, +`gbrain embed --stale` actually re-embeds the pages still on the old model +instead of silently leaving them stale.** + +Before this release, `gbrain sync --all` always stopped and asked for +confirmation in any non-interactive run (cron, script, piped output) based on +a guess of your whole corpus's embedding cost. On an already-synced brain that +guess was large and the real work was zero, so the cron exited with an error +every night and you had to wire in `--yes` permanently, which defeats the +safety check. + +Here's what changed. On the default fast sync path, embedding doesn't run +during the sync itself; it's handed to background jobs that already cap their +own spend at $25 per source per day. So the sync command now prints what's +queued and proceeds. It never blocks. You'll only see a confirmation prompt +when sync embeds inline (the older non-parallel mode), and even then only when +the new content this sync embeds crosses a dollar threshold you control. + +**The threshold knob:** +``` +gbrain config set sync.cost_gate_min_usd 0.50 # default; raise or lower to taste +``` +At or below this estimate an inline sync proceeds silently. Set it to 0 to +confirm on any nonzero cost. + +**Real stale detection.** gbrain now records which model and dimensions +produced each page's vectors. Change your embedding model (OpenAI to Voyage, +or a different dimension) and `gbrain embed --stale` finds and re-embeds every +page still on the old model. Before, "stale" only meant "never embedded," so a +model swap was invisible and search quietly mixed vector spaces. + +This does NOT trigger a surprise re-embed of your whole brain on upgrade. Pages +that predate this release have no recorded signature, and gbrain treats "no +signature" as "leave it alone." Only pages embedded after you upgrade carry a +signature, and only an actual model change marks them stale. + +**See your embedding backlog.** `gbrain sources status` gains a BACKFILL column +showing whether each source has embed jobs queued, running, or idle, plus when +the last one finished. After a `sync --all` you can tell at a glance whether +embeddings are caught up or still trickling in. + +## To take advantage of v0.41.31.0 + +`gbrain upgrade` does this automatically (it applies migration v108, which adds +the per-page embedding-signature column). If `gbrain doctor` warns about a +partial migration: + +1. **Apply migrations manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Verify the cron no longer blocks:** + ```bash + gbrain sync --all --dry-run # previews, exits 0, no confirmation demanded + gbrain sources status # BACKFILL column present + ``` +3. **After a future model swap:** `gbrain embed --stale` re-embeds the drifted + pages. (Pages embedded before this upgrade keep their old vectors until they + are next re-embedded for some other reason — the no-surprise-cost + grandfather behavior.) +4. **If anything looks wrong,** file an issue at + https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`. + +### Itemized changes + +#### Sync cost gate +- `gbrain sync --all` is mode-aware. When embedding is deferred to backfill + jobs (the federated_v2 default), the gate is informational only: it prints a + deferred notice (cap-aware, "not charged by this sync") plus the current + backlog and never exits 2. The blocking confirmation gate fires only when + sync embeds inline AND the new-content estimate exceeds + `sync.cost_gate_min_usd` (default $0.50). +- The inline estimate is delta-aware: sources unchanged since their last sync + (HEAD == last_commit, clean tree, chunker version current) contribute 0; + changed sources contribute the full-tree ceiling. The pre-existing stale + backlog is shown informationally but does not gate inline sync (that backlog + is `gbrain embed --stale`'s job, not sync's). +- The cost preview prices against the configured model's real rate rather than + a hardcoded OpenAI rate. + +#### Real stale semantics (migration v108) +- New `pages.embedding_signature` column records `:` when + a page's chunks are embedded. `gbrain embed --stale`, the embed-backfill + jobs, and the sync cost preview treat a page as stale when its signature + differs from the current model's. NULL signature is grandfathered (never + stale) so upgrading does not mass-re-embed. +- All embed-write paths stamp the signature: `gbrain embed`/`--all`/`--stale`, + the embed-backfill minion, `gbrain sync`'s inline import, and `gbrain import`. +- New engine methods on both Postgres and PGLite: `sumStaleChunkChars`, + `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, plus an + optional `signature` filter on `countStaleChunks` / `sumStaleChunkChars`. + +#### Backfill visibility +- `gbrain sources status` shows per-source embed-backfill state + (queued/active/idle + last completion). The deferred sync notice appends the + queued-job count so a cron operator sees work is enqueued, not lost. + +#### Configuration +- `sync.cost_gate_min_usd` (default 0.50) sets the inline-sync confirmation + floor. ## [0.41.30.0] - 2026-05-30 **`gbrain lsd --save` (and `brainstorm --save`) now actually writes the diff --git a/CLAUDE.md b/CLAUDE.md index 6a0cb6406..06718167f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. @@ -52,8 +52,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. **v0.41.31.0:** implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, and the widened `countStaleChunks({sourceId?, signature?})`. The `signature` opt widens the stale predicate via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)` (NULL grandfathered). Parity SQL with postgres-engine.ts. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. **v0.41.31.0:** implements the embedding-signature stale-detection quartet in parity with pglite-engine.ts — `sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, and the widened `countStaleChunks` (all accept the optional `signature` to extend "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN; NULL signature grandfathered). The v0.22.1 `countStaleChunks` / `listStaleChunks` server-side `embedding IS NULL` filter is preserved as the no-signature fast path. Pinned by `test/e2e/engine-parity.test.ts`. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -64,7 +64,7 @@ strict behavior when unset. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). **v0.26.9 (D14):** adds `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated. Replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback path. Reusable from any future code that needs the same column-existence probe semantics. **v0.32.8 (PR #860):** adds `validateSourceId(id)` that throws on anything outside `^[a-z0-9_-]+$`. Used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` call so source_id can't traverse out of brainDir. `rowToPage` updated to populate the now-required `Page.source_id` field from the SELECT projection (`scripts/check-source-id-projection.sh` enforces that every projection feeding `rowToPage` includes the column). - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) -- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags) +- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). **v0.41.31.0:** `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) — covers the inline import/sync path so a model/dims swap is detectable as stale. `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`); mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle the swap for those). - `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. @@ -112,7 +112,7 @@ strict behavior when unset. - `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. -- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. +- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. **v0.41.31.0 (cost-model + stale-semantics wave):** `estimateEmbeddingCostUsd(tokens)` now prices against the CURRENTLY-CONFIGURED model's rate instead of the hardcoded OpenAI `EMBEDDING_COST_PER_1K_TOKENS`. Pre-fix, a brain on a cheaper model (e.g. ZeroEntropy) saw a `sync --all` preview that named OpenAI and over-stated spend ~2.6x. New `currentEmbeddingPricePerMTok()` resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate (0.13) only when the gateway is unconfigured (unit tests, pre-connect preview) or the model is unknown to the pricing table. `EMBEDDING_COST_PER_1K_TOKENS` is retained for back-compat with direct importers/tests. New `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (that's tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from the current one and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. New `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so the cost gate and the actual embed decision can never drift. New `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. - `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962. - `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline. **v0.37.6.0 (#1210):** new `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers that ride alongside auth on every openai-compat touchpoint. Mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) are rejected at `applyResolveAuth` call time so defaults cannot accidentally shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple; usable by any future recipe (Together/Groq) that wants attribution. - `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.36.3.0:** `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path can embed via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default. The hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}`; the gateway resolves the matching recipe and routes through its `instantiateEmbedding()` branch. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down — fixes the CDX-4 bug where a healthy Voyage column would skip vector retrieval because OpenAI happened to be unconfigured. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. @@ -172,7 +172,7 @@ strict behavior when unset. - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. **v0.41.31.0:** every embed-write path now stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. - `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. @@ -180,7 +180,7 @@ strict behavior when unset. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` (v0.41.23.0) — unified extract operator-surface wave. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) now writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug per D-EXTRACT-17: `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` per D-EXTRACT-19 (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts` so receipts surface in search but never dominate user content. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes are best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. New `extract_health` doctor check reads the rollup for last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok` silently. Operator CLI surface: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted by halt_rate desc + cost desc, kubectl-style top-5 + "more rows" hint, stable `schema_version: 1` JSON envelope); `gbrain extract --explain ` (resolution chain: pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)` markers, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict D-EXTRACT-21 path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; v0.41.23 ships as a stub-reporter — LLM dispatch deferred to a follow-up release). Pack-author authoring loop: `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` per D-EXTRACT-37 — parses but refuses at runtime in v0.41.23); new `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` helpers in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable in one verb, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Tests: 22 cases in `test/extractable-spec-widening.test.ts` (back-compat boolean shape + struct shape + D-EXTRACT-37 verifier_path refuse), 12 cases in `test/extract/receipt-writer.test.ts` (uses canonical PGLite block per CLAUDE.md R3+R4 — one engine per file with `resetPgliteState` in `beforeEach`), 17 cases in `test/extract/benchmark.test.ts` (path validation rejections + JSONL contract), 15 cases in `test/extract/status.test.ts` (pure aggregation + formatting), 15 cases in `test/schema-pack/scaffold-extractable.test.ts` (scaffold mechanics + explicit privacy-rule guards against real-name leakage), 8 cases in `test/doctor-extract-health.test.ts`. Plan persisted at `~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md`. Deferred to a follow-up release: replay versioning + `gbrain extract replay --since v`; unified LLM dispatcher (deliberately respects v0.41.13 head comment that extraction is regex + cost-cap value-add lives at embed step). - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. -- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). +- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). **v0.41.31.0 (P0a):** `gbrain sources status` (the live `runStatus` health table path, NOT just the `buildSyncStatusReport` dashboard) gains a `BACKFILL` column between `EMBED` and `FAILS`: `active(N)` beats `queued(N)` beats `idle`, sourced from `SourceMetrics.backfill_active` / `backfill_queued` in `src/core/source-health.ts`. Under federated_v2, `sync --all` defers embedding to per-source `embed-backfill` minion jobs, so an operator needs to SEE that deferred work after sync exits 0 instead of assuming it was lost. The data layer (`jobCountsBySource` in `source-health.ts`) widens its one `minion_jobs` SQL with two `COUNT(*) FILTER (WHERE name = 'embed-backfill' AND ...)` aggregates (active vs waiting/delayed/waiting-children) — best-effort, all-0 on pre-minions brains. Pinned by `test/source-health.test.ts`. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint). **v0.41.13.0 (#1434):** new tier 5.5 `sole_non_default` slots between `brain_default` and `seed_default`. When NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'`, auto-route to it. `SOURCE_TIER_NAMES` extended to 7 entries. New private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points (cannot drift). Archived sources excluded (try/catch for pre-v34 brains). New exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (returns null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `src/commands/sync.ts:1497-1519` rewired to call `resolveSourceWithTier` unconditionally so the new tier actually fires (codex caught the original plan shipping dead code). `src/commands/import.ts:96-128` mirrors the same wiring with the tier-gated nudge. Pinned by `test/source-resolver-sole-non-default.test.ts` (14 cases) + `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync` with stderr capture). - `src/core/sync.ts` extension (v0.41.13.0, #1433) — `isSyncable` factored through a private `classifySync(path, opts): SyncableReason | null` helper. New exported `unsyncableReason(path, opts)` companion returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` lifted to a named export (the four canonical metafile basenames: `schema.md`, `index.md`, `log.md`, `README.md`). New `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. `src/commands/sync.ts:772` cleanup loop now guards on `unsyncableReason(path) === 'metafile'` so previously-indexed metafile pages (typically `log.md` from an older gbrain that didn't filter it) survive every re-sync. Honest scope: does NOT cover `manifest.deleted` (the upstream filter already strips metafiles, so `rm log.md` doesn't delete the page either — v0.42+ TODO for a `gbrain pages remove ` operator surface). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases pinning the duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases including the negative case — renamed `.md → .txt` still gets cleaned up properly). @@ -263,7 +263,7 @@ strict behavior when unset. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -275,7 +275,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. **v0.41.31.0 (mode-aware cost gate):** the v0.20.0 unconditional `sync --all` ConfirmationRequired gate is superseded by a mode-aware gate driven by `willEmbedSynchronously` + `shouldBlockSync` from `src/core/embedding.ts`. federated_v2 is resolved ONCE up front (shared with the fan-out below). On the DEFERRED path (v2 on, parallel) embedding goes to per-source `embed-backfill` jobs that carry their own `$X/source/24h` cap (default $25, `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`), so the gate is INFORMATIONAL — it prints an FYI naming the backfill cap + current backlog (`sumStaleChunkChars({signature})` priced via `estimateCostFromChars` at `currentEmbeddingPricePerMTok()`) + the count of already-queued `embed-backfill` jobs (TODO-2 visibility), and NEVER exits 2. On the INLINE path (v2 off, or `--serial` without `--no-embed`) the BLOCKING gate fires only when the NEW-CONTENT estimate exceeds the `sync.cost_gate_min_usd` floor (default $0.50). The new-content estimate (`estimateInlineNewTokens`) contributes ZERO for sources provably unchanged since last sync (`isSourceUnchangedSinceSync` HEAD==last_commit + clean tree, AND `chunker_version === CURRENT`) — mirrors doctor's `sync_freshness` + sync's own do-work gate — and the full-tree ceiling for changed sources. F2 fix: the pre-existing stale backlog (NULL embeddings + signature drift) is shown informationally but NEVER gated on, because sync doesn't sweep it inline (`gbrain embed --stale` does); gating on it would block every inline cron after a model swap. New helpers `resolveCostGateFloorUsd(engine)` (reads `sync.cost_gate_min_usd`, fail-open $0.50, accepts 0) + `resolveBackfillCapUsd(engine)`. The `sync --all` SELECT widens to carry `last_commit + chunker_version` (both columns predate v0.41; no migration). JSON envelopes gain `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | confirmation_required`). `SyncStatusReportSource` gains `backfill_queued` / `backfill_active` / `backfill_last_completed_at` (best-effort; 0/null on pre-minions brains). All embedding-cost preview math reads the configured model name via `getEmbeddingModelName()` (no hardcoded OpenAI). Pinned by `test/sync-cost-gate.serial.test.ts` + `test/sync-cost-preview.test.ts`. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -659,6 +659,12 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. +Key commands added in v0.41.31.0 (embedding cost-model + stale-semantics wave): +- `gbrain sync --all` cost gate is now mode-aware. When embedding is DEFERRED (federated_v2 parallel path → per-source `embed-backfill` jobs), the gate is INFORMATIONAL — it prints the backfill cap, the current backlog cost, and the count of queued backfill jobs, and NEVER exits 2. When embedding runs INLINE (v2 off, or `--serial` without `--no-embed`), the gate BLOCKS only when the NEW-CONTENT estimate exceeds the floor. Cost previews now name the ACTUALLY-configured embedding model and price at its real rate (no more hardcoded OpenAI over-statement on cheaper models). +- `gbrain config set sync.cost_gate_min_usd ` — inline-path cost-gate floor (DB plane, default `$0.50`). Below this estimate the inline `sync --all` gate proceeds without blocking, killing the noise where a tiny incremental sync prompted on every cron. Set to `0` to block on any nonzero inline cost. Fail-open to the default on a missing/invalid value. +- `gbrain embed --stale` now re-embeds pages whose embedding model/dimensions changed, not just pages with NULL embeddings. After a model or dims swap, the stored `pages.embedding_signature` differs from the current one, so the next `embed --stale` (or per-source `embed-backfill` job) invalidates and re-embeds those pages. GRANDFATHER: pages embedded before v0.41.31.0 have a NULL signature and are NEVER flagged stale, so the upgrade does NOT trigger a whole-corpus re-embed. `embed --stale --dry-run` counts signature-drift without mutating. +- `gbrain sources status` gains a `BACKFILL` column (`active(N)` / `queued(N)` / `idle`) between `EMBED` and `FAILS` so an operator can see deferred embedding work after `sync --all` exits 0. + Key commands added in v0.36.4.0 (brain-health-100 wave): - `gbrain doctor --remediation-plan [--target-score N] [--json]` — preview the dependency-ordered plan that would drive the brain to target. JSON envelope is stable: each `Remediation` carries `id`, `idempotency_key` (content-hash for cron-safe retries), `severity`, `est_seconds`, `est_usd_cost`, and `depends_on` (referencing other ids). Empty `recommendations` array when the brain is already at target. - `gbrain doctor --remediate [--yes] [--target-score N] [--max-usd N]` — actually submit the plan. Walks dependency order, submits one Minion job per step, re-checks score between steps, refuses to spend past `--max-usd` (defaults: target=90, max-usd=infinite — but cron callers should always pass `--max-usd`). Bails when target exceeds `maxReachableScore()` for the brain (empty / under-configured brains) with a clear list of what's missing. diff --git a/VERSION b/VERSION index 13bcae3ae..c9ccfb296 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.30.0 +0.41.31.0 \ No newline at end of file diff --git a/docs/embedding-migrations.md b/docs/embedding-migrations.md index d1eaed639..e9c425128 100644 --- a/docs/embedding-migrations.md +++ b/docs/embedding-migrations.md @@ -10,6 +10,34 @@ change automatically. this mismatch and refuse to silently proceed. This doc is the recipe they point at. +## Same-dimension model swaps (v0.41.31.0 — automatic) + +If you switch to a different model at the **same** dimension count +(e.g. one 1536-dim provider to another, or a re-tuned model that keeps +its width), the column type doesn't change, so no `ALTER`/wipe recipe +is needed. As of v0.41.31.0, gbrain stamps an embedding-provenance +signature (`:`) onto each page when its chunks are +embedded. After you point the config at the new model, the stored +signatures differ from the current one, and `gbrain embed --stale` +re-embeds exactly those pages: + +```bash +# After switching to the new same-dim model in your config: +gbrain embed --stale # re-embeds signature-drifted pages +gbrain embed --stale --dry-run # preview the count without re-embedding +``` + +Under federated_v2, the same drift is picked up by the per-source +`embed-backfill` jobs that `gbrain sync --all` enqueues (capped +`$X/source/24h`). **Grandfather:** pages embedded before v0.41.31.0 +carry a NULL signature and are NEVER flagged stale, so upgrading to +v0.41.31.0 does NOT trigger a whole-corpus re-embed. Signatures only +get stamped going forward. + +A **dimension** change still requires the wipe-and-reinit (PGLite) or +column-alter (Postgres) recipe below — the on-disk `vector(N)` width +genuinely has to change. + ## Why we don't do this automatically Switching dimensions requires: diff --git a/llms-full.txt b/llms-full.txt index 422ff20b2..9b7db17d8 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -183,7 +183,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. @@ -194,8 +194,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. **v0.41.31.0:** implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, and the widened `countStaleChunks({sourceId?, signature?})`. The `signature` opt widens the stale predicate via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)` (NULL grandfathered). Parity SQL with postgres-engine.ts. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. **v0.41.31.0:** implements the embedding-signature stale-detection quartet in parity with pglite-engine.ts — `sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, and the widened `countStaleChunks` (all accept the optional `signature` to extend "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN; NULL signature grandfathered). The v0.22.1 `countStaleChunks` / `listStaleChunks` server-side `embedding IS NULL` filter is preserved as the no-signature fast path. Pinned by `test/e2e/engine-parity.test.ts`. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -206,7 +206,7 @@ strict behavior when unset. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). **v0.26.9 (D14):** adds `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated. Replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback path. Reusable from any future code that needs the same column-existence probe semantics. **v0.32.8 (PR #860):** adds `validateSourceId(id)` that throws on anything outside `^[a-z0-9_-]+$`. Used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` call so source_id can't traverse out of brainDir. `rowToPage` updated to populate the now-required `Page.source_id` field from the SELECT projection (`scripts/check-source-id-projection.sh` enforces that every projection feeding `rowToPage` includes the column). - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) -- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags) +- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). **v0.41.31.0:** `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) — covers the inline import/sync path so a model/dims swap is detectable as stale. `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`); mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle the swap for those). - `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. @@ -254,7 +254,7 @@ strict behavior when unset. - `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. -- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. +- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. **v0.41.31.0 (cost-model + stale-semantics wave):** `estimateEmbeddingCostUsd(tokens)` now prices against the CURRENTLY-CONFIGURED model's rate instead of the hardcoded OpenAI `EMBEDDING_COST_PER_1K_TOKENS`. Pre-fix, a brain on a cheaper model (e.g. ZeroEntropy) saw a `sync --all` preview that named OpenAI and over-stated spend ~2.6x. New `currentEmbeddingPricePerMTok()` resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate (0.13) only when the gateway is unconfigured (unit tests, pre-connect preview) or the model is unknown to the pricing table. `EMBEDDING_COST_PER_1K_TOKENS` is retained for back-compat with direct importers/tests. New `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (that's tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from the current one and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. New `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so the cost gate and the actual embed decision can never drift. New `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. - `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962. - `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline. **v0.37.6.0 (#1210):** new `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers that ride alongside auth on every openai-compat touchpoint. Mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) are rejected at `applyResolveAuth` call time so defaults cannot accidentally shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple; usable by any future recipe (Together/Groq) that wants attribution. - `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.36.3.0:** `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path can embed via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default. The hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}`; the gateway resolves the matching recipe and routes through its `instantiateEmbedding()` branch. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down — fixes the CDX-4 bug where a healthy Voyage column would skip vector retrieval because OpenAI happened to be unconfigured. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. @@ -314,7 +314,7 @@ strict behavior when unset. - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. **v0.41.31.0:** every embed-write path now stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. - `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. @@ -322,7 +322,7 @@ strict behavior when unset. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` (v0.41.23.0) — unified extract operator-surface wave. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) now writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug per D-EXTRACT-17: `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` per D-EXTRACT-19 (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts` so receipts surface in search but never dominate user content. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes are best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. New `extract_health` doctor check reads the rollup for last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok` silently. Operator CLI surface: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted by halt_rate desc + cost desc, kubectl-style top-5 + "more rows" hint, stable `schema_version: 1` JSON envelope); `gbrain extract --explain ` (resolution chain: pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)` markers, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict D-EXTRACT-21 path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; v0.41.23 ships as a stub-reporter — LLM dispatch deferred to a follow-up release). Pack-author authoring loop: `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` per D-EXTRACT-37 — parses but refuses at runtime in v0.41.23); new `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` helpers in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable in one verb, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Tests: 22 cases in `test/extractable-spec-widening.test.ts` (back-compat boolean shape + struct shape + D-EXTRACT-37 verifier_path refuse), 12 cases in `test/extract/receipt-writer.test.ts` (uses canonical PGLite block per CLAUDE.md R3+R4 — one engine per file with `resetPgliteState` in `beforeEach`), 17 cases in `test/extract/benchmark.test.ts` (path validation rejections + JSONL contract), 15 cases in `test/extract/status.test.ts` (pure aggregation + formatting), 15 cases in `test/schema-pack/scaffold-extractable.test.ts` (scaffold mechanics + explicit privacy-rule guards against real-name leakage), 8 cases in `test/doctor-extract-health.test.ts`. Plan persisted at `~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md`. Deferred to a follow-up release: replay versioning + `gbrain extract replay --since v`; unified LLM dispatcher (deliberately respects v0.41.13 head comment that extraction is regex + cost-cap value-add lives at embed step). - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. -- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). +- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). **v0.41.31.0 (P0a):** `gbrain sources status` (the live `runStatus` health table path, NOT just the `buildSyncStatusReport` dashboard) gains a `BACKFILL` column between `EMBED` and `FAILS`: `active(N)` beats `queued(N)` beats `idle`, sourced from `SourceMetrics.backfill_active` / `backfill_queued` in `src/core/source-health.ts`. Under federated_v2, `sync --all` defers embedding to per-source `embed-backfill` minion jobs, so an operator needs to SEE that deferred work after sync exits 0 instead of assuming it was lost. The data layer (`jobCountsBySource` in `source-health.ts`) widens its one `minion_jobs` SQL with two `COUNT(*) FILTER (WHERE name = 'embed-backfill' AND ...)` aggregates (active vs waiting/delayed/waiting-children) — best-effort, all-0 on pre-minions brains. Pinned by `test/source-health.test.ts`. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint). **v0.41.13.0 (#1434):** new tier 5.5 `sole_non_default` slots between `brain_default` and `seed_default`. When NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'`, auto-route to it. `SOURCE_TIER_NAMES` extended to 7 entries. New private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points (cannot drift). Archived sources excluded (try/catch for pre-v34 brains). New exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (returns null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `src/commands/sync.ts:1497-1519` rewired to call `resolveSourceWithTier` unconditionally so the new tier actually fires (codex caught the original plan shipping dead code). `src/commands/import.ts:96-128` mirrors the same wiring with the tier-gated nudge. Pinned by `test/source-resolver-sole-non-default.test.ts` (14 cases) + `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync` with stderr capture). - `src/core/sync.ts` extension (v0.41.13.0, #1433) — `isSyncable` factored through a private `classifySync(path, opts): SyncableReason | null` helper. New exported `unsyncableReason(path, opts)` companion returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` lifted to a named export (the four canonical metafile basenames: `schema.md`, `index.md`, `log.md`, `README.md`). New `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. `src/commands/sync.ts:772` cleanup loop now guards on `unsyncableReason(path) === 'metafile'` so previously-indexed metafile pages (typically `log.md` from an older gbrain that didn't filter it) survive every re-sync. Honest scope: does NOT cover `manifest.deleted` (the upstream filter already strips metafiles, so `rm log.md` doesn't delete the page either — v0.42+ TODO for a `gbrain pages remove ` operator surface). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases pinning the duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases including the negative case — renamed `.md → .txt` still gets cleaned up properly). @@ -405,7 +405,7 @@ strict behavior when unset. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -417,7 +417,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. **v0.41.31.0 (mode-aware cost gate):** the v0.20.0 unconditional `sync --all` ConfirmationRequired gate is superseded by a mode-aware gate driven by `willEmbedSynchronously` + `shouldBlockSync` from `src/core/embedding.ts`. federated_v2 is resolved ONCE up front (shared with the fan-out below). On the DEFERRED path (v2 on, parallel) embedding goes to per-source `embed-backfill` jobs that carry their own `$X/source/24h` cap (default $25, `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`), so the gate is INFORMATIONAL — it prints an FYI naming the backfill cap + current backlog (`sumStaleChunkChars({signature})` priced via `estimateCostFromChars` at `currentEmbeddingPricePerMTok()`) + the count of already-queued `embed-backfill` jobs (TODO-2 visibility), and NEVER exits 2. On the INLINE path (v2 off, or `--serial` without `--no-embed`) the BLOCKING gate fires only when the NEW-CONTENT estimate exceeds the `sync.cost_gate_min_usd` floor (default $0.50). The new-content estimate (`estimateInlineNewTokens`) contributes ZERO for sources provably unchanged since last sync (`isSourceUnchangedSinceSync` HEAD==last_commit + clean tree, AND `chunker_version === CURRENT`) — mirrors doctor's `sync_freshness` + sync's own do-work gate — and the full-tree ceiling for changed sources. F2 fix: the pre-existing stale backlog (NULL embeddings + signature drift) is shown informationally but NEVER gated on, because sync doesn't sweep it inline (`gbrain embed --stale` does); gating on it would block every inline cron after a model swap. New helpers `resolveCostGateFloorUsd(engine)` (reads `sync.cost_gate_min_usd`, fail-open $0.50, accepts 0) + `resolveBackfillCapUsd(engine)`. The `sync --all` SELECT widens to carry `last_commit + chunker_version` (both columns predate v0.41; no migration). JSON envelopes gain `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | confirmation_required`). `SyncStatusReportSource` gains `backfill_queued` / `backfill_active` / `backfill_last_completed_at` (best-effort; 0/null on pre-minions brains). All embedding-cost preview math reads the configured model name via `getEmbeddingModelName()` (no hardcoded OpenAI). Pinned by `test/sync-cost-gate.serial.test.ts` + `test/sync-cost-preview.test.ts`. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -801,6 +801,12 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. +Key commands added in v0.41.31.0 (embedding cost-model + stale-semantics wave): +- `gbrain sync --all` cost gate is now mode-aware. When embedding is DEFERRED (federated_v2 parallel path → per-source `embed-backfill` jobs), the gate is INFORMATIONAL — it prints the backfill cap, the current backlog cost, and the count of queued backfill jobs, and NEVER exits 2. When embedding runs INLINE (v2 off, or `--serial` without `--no-embed`), the gate BLOCKS only when the NEW-CONTENT estimate exceeds the floor. Cost previews now name the ACTUALLY-configured embedding model and price at its real rate (no more hardcoded OpenAI over-statement on cheaper models). +- `gbrain config set sync.cost_gate_min_usd ` — inline-path cost-gate floor (DB plane, default `$0.50`). Below this estimate the inline `sync --all` gate proceeds without blocking, killing the noise where a tiny incremental sync prompted on every cron. Set to `0` to block on any nonzero inline cost. Fail-open to the default on a missing/invalid value. +- `gbrain embed --stale` now re-embeds pages whose embedding model/dimensions changed, not just pages with NULL embeddings. After a model or dims swap, the stored `pages.embedding_signature` differs from the current one, so the next `embed --stale` (or per-source `embed-backfill` job) invalidates and re-embeds those pages. GRANDFATHER: pages embedded before v0.41.31.0 have a NULL signature and are NEVER flagged stale, so the upgrade does NOT trigger a whole-corpus re-embed. `embed --stale --dry-run` counts signature-drift without mutating. +- `gbrain sources status` gains a `BACKFILL` column (`active(N)` / `queued(N)` / `idle`) between `EMBED` and `FAILS` so an operator can see deferred embedding work after `sync --all` exits 0. + Key commands added in v0.36.4.0 (brain-health-100 wave): - `gbrain doctor --remediation-plan [--target-score N] [--json]` — preview the dependency-ordered plan that would drive the brain to target. JSON envelope is stable: each `Remediation` carries `id`, `idempotency_key` (content-hash for cron-safe retries), `severity`, `est_seconds`, `est_usd_cost`, and `depends_on` (referencing other ids). Empty `recommendations` array when the brain is already at target. - `gbrain doctor --remediate [--yes] [--target-score N] [--max-usd N]` — actually submit the plan. Walks dependency order, submits one Minion job per step, re-checks score between steps, refuses to spend past `--max-usd` (defaults: target=90, max-usd=infinite — but cron callers should always pass `--max-usd`). Bails when target exceeds `maxReachableScore()` for the brain (empty / under-configured brains) with a clear list of what's missing. diff --git a/package.json b/package.json index 85a907ce1..fb5a56b40 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.30.0" + "version": "0.41.31.0" } diff --git a/src/commands/embed.ts b/src/commands/embed.ts index 920d1936e..95274986a 100644 --- a/src/commands/embed.ts +++ b/src/commands/embed.ts @@ -1,5 +1,5 @@ import type { BrainEngine } from '../core/engine.ts'; -import { embedBatch } from '../core/embedding.ts'; +import { embedBatch, currentEmbeddingSignature } from '../core/embedding.ts'; import type { ChunkInput } from '../core/types.ts'; import { chunkText } from '../core/chunkers/recursive.ts'; import { createProgress, type ProgressReporter } from '../core/progress.ts'; @@ -378,6 +378,16 @@ async function embedPage( })); await engine.upsertChunks(slug, updated, opts); + // v0.41.31: stamp provenance so a later model/dims swap is detectable as + // stale. embedPage is the per-slug path used by `gbrain embed ` AND + // by `gbrain sync`'s post-import embed step (runEmbedCore({slugs})). + // Guard: only stamp when EVERY chunk was (re)embedded this pass. If some + // chunks were preserved from a prior embed (unknown/old provenance), the + // page is mixed — don't claim it's current. `embed --all` fully re-embeds + // such a page and then stamps it. + if (toEmbed.length === chunks.length) { + await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() }); + } result.embedded += toEmbed.length; result.pages_processed++; slog(`${slug}: embedded ${toEmbed.length} chunks`); @@ -396,6 +406,10 @@ async function embedAll( catchUp?: boolean; }, ) { + // v0.41.31: current embedding provenance signature. Stamped onto pages + // when their chunks are (re)embedded so a later model/dimension swap is + // detectable as stale. + const signature = currentEmbeddingSignature(); // ───────────────────────────────────────────────────────────── // Stale-only fast path: avoid the listPages + per-page getChunks // bomb that pulled every page row + every chunk's embedding column @@ -412,7 +426,7 @@ async function embedAll( if (staleOnly) { // D7: thread sourceId so `gbrain embed --stale --source X` actually scopes. // v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path. - return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts); + return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature); } // v0.31.12: when sourceId is set, scope listPages to that source. @@ -482,6 +496,9 @@ async function embedAll( token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); await engine.upsertChunks(page.slug, updated, pageOpts); + // v0.41.31: stamp embedding provenance so a later model swap is + // detectable as stale. + await engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }); result.embedded += toEmbed.length; } catch (e: unknown) { serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`); @@ -543,13 +560,31 @@ async function embedAllStale( priority?: 'recent'; catchUp?: boolean; }, + signature?: string, ) { // D7: thread sourceId so source-scoped runs only count + visit // that source's NULL embeddings. const sourceOpt = sourceId ? { sourceId } : undefined; + // v0.41.31: re-embed pages whose embedding_signature drifted (model/dims + // swap). dry-run must NOT mutate, so it counts signature-stale via the + // widened predicate; a live run NULLs them first so the existing + // NULL-embedding cursor (listStaleChunks) picks them up unchanged. + if (!dryRun && signature) { + const invalidated = await engine.invalidateStaleSignatureEmbeddings({ + signature, + ...(sourceId && { sourceId }), + }); + if (invalidated > 0) { + slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`); + } + } + // Pre-flight: 0 stale chunks → nothing to do, no further DB reads. - const staleCount = await engine.countStaleChunks(sourceOpt); + // dry-run includes signature-drift in the count without mutating. + const staleCount = await engine.countStaleChunks( + dryRun && signature ? { ...sourceOpt, signature } : sourceOpt, + ); if (staleCount === 0) { if (dryRun) { slog('[dry-run] Would embed 0 chunks (0 stale found)'); @@ -671,6 +706,14 @@ async function embedAllStale( token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); await engine.upsertChunks(slug, merged, { sourceId: keySourceId }); + // v0.41.31: stamp provenance after the page's chunks are embedded — + // but only when EVERY chunk was stale (fully re-embedded this pass). + // A partially-stale page keeps preserved chunks of unknown/old + // provenance, so don't claim it's current. (After invalidate, a + // signature-drifted page IS fully stale → this stamps it.) + if (signature && stale.length === existing.length) { + await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }); + } result.embedded += stale.length; } catch (e: unknown) { // Budget-fired aborts are expected on the way out; don't spam diff --git a/src/commands/sources.ts b/src/commands/sources.ts index 72618492f..e7620048b 100644 --- a/src/commands/sources.ts +++ b/src/commands/sources.ts @@ -599,22 +599,29 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise { return; } - // Human-readable table: SOURCE | LAG | EMBED | FAILS | QUEUE | PAGES | LAST SYNC + // Human-readable table: SOURCE | LAG | EMBED | BACKFILL | FAILS | QUEUE | PAGES | LAST SYNC console.log('SOURCES — health'); console.log('────────────────'); console.log( - ` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`, + ` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'BACKFILL'.padEnd(9)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`, ); for (const m of metrics) { const lag = m.lag_seconds === null ? 'never' : formatLag(m.lag_seconds); const embed = `${m.embed_coverage_pct.toFixed(0)}%`; + // v0.41.31: embed-backfill state (active beats queued beats idle) so a + // cron operator sees deferred embedding work after `sync --all`. + const backfill = m.backfill_active > 0 + ? `active(${m.backfill_active})` + : m.backfill_queued > 0 + ? `queued(${m.backfill_queued})` + : 'idle'; const fails = String(m.failed_jobs_24h); const queue = String(m.queue_depth); const pages = m.total_pages.toLocaleString(); const sync = m.last_sync_at ? new Date(m.last_sync_at).toISOString().slice(0, 19).replace('T', ' ') : 'never'; - console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`); + console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${backfill.padEnd(9)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`); } console.log(''); for (const m of metrics) { diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 41628afe1..41244e5d8 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -17,7 +17,17 @@ import { formatCodeBreakdown, } from '../core/sync.ts'; import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts'; -import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts'; +import { + estimateEmbeddingCostUsd, + getEmbeddingModelName, + currentEmbeddingPricePerMTok, + currentEmbeddingSignature, + willEmbedSynchronously, + shouldBlockSync, +} from '../core/embedding.ts'; +import { estimateCostFromChars } from '../core/embedding-pricing.ts'; +import { isSourceUnchangedSinceSync } from '../core/git-head.ts'; +import { SPEND_CAP_CONFIG_KEY } from '../core/embed-backfill-submit.ts'; import { errorFor, serializeError } from '../core/errors.ts'; import type { SyncManifest } from '../core/sync.ts'; import { createProgress } from '../core/progress.ts'; @@ -78,64 +88,114 @@ export interface SyncResult { } /** - * v0.20.0 Cathedral II Layer 8 (D1) — walk each source's working tree and - * sum tokens for every syncable file. This is a conservative overestimate - * (full file content, not just the incremental diff) because `sync --all` - * on a source that hasn't been synced yet WILL embed every file in the - * working tree. For already-synced sources with only incremental changes, - * the overestimate is the ceiling, not the floor — users never get - * surprised by MORE cost than the preview claims. The false-high bias is - * intentional: a lower estimate that undersells the real bill would be - * worse than one that oversells. + * Walk ONE source's working tree and sum tokens for every syncable file. + * Conservative full-tree ceiling (full file content, not the incremental + * diff) — over-counts, never under-counts, and matches the filesystem set + * `sync` actually imports (collectSyncableFiles + content_hash, NOT a git + * commit diff). Best-effort per file and per source: anything unreadable + * contributes 0 rather than blocking the preview. + * + * v0.31.2: routed through collectSyncableFiles (lstat + inode-cycle + + * max-depth) so the preview walks exactly what the real sync walks. */ -function estimateSyncAllCost(sources: Array<{ local_path: string | null; config: Record }>): { - totalTokens: number; - totalFiles: number; - activeSources: number; - perSource: Array<{ path: string; tokens: number; files: number }>; -} { - let totalTokens = 0; - let totalFiles = 0; - let activeSources = 0; - const perSource: Array<{ path: string; tokens: number; files: number }> = []; +function estimateSourceTreeTokens( + localPath: string, + strategy: 'markdown' | 'code' | 'auto', +): { tokens: number; files: number } { + let tokens = 0; + let files = 0; + try { + const fileList = collectSyncableFiles(localPath, { strategy }); + for (const fullPath of fileList) { + try { + const stat = statSync(fullPath); + if (stat.size > 5_000_000) continue; // skip large binaries + const content = readFileSync(fullPath, 'utf-8'); + tokens += estimateTokens(content); + files++; + } catch { + // Best-effort per file; sync itself tolerates the same. + } + } + } catch { + // Best-effort: a source whose local_path is gone/unreadable contributes 0. + } + return { tokens, files }; +} +/** + * v0.41.31 — INLINE-path new-content estimate. Per source, contribute ZERO + * when the source is provably unchanged since its last sync (HEAD == + * last_commit AND clean working tree AND chunker_version matches CURRENT) — + * `content_hash` short-circuits every file so nothing re-embeds. Otherwise + * contribute the full-tree ceiling. The unchanged predicate mirrors + * doctor's `sync_freshness` and sync's own "do work?" gate (sync.ts:1057+ + * 1075). `isSourceUnchangedSinceSync` is fail-open (probe error → false), so + * a source we can't prove unchanged is conservatively re-estimated rather + * than silently priced at $0. + */ +function estimateInlineNewTokens( + sources: Array<{ + local_path: string | null; + config: Record; + last_commit: string | null; + chunker_version: string | null; + }>, + currentChunkerVersion: string, +): { tokens: number; changedSources: number; unchangedSources: number } { + let tokens = 0; + let changedSources = 0; + let unchangedSources = 0; for (const src of sources) { if (!src.local_path) continue; const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' }; if (cfg.syncEnabled === false) continue; - activeSources++; - let sourceTokens = 0; - let sourceFiles = 0; - try { - // v0.31.2: cost preview routed through collectSyncableFiles - // (single hardened walker; see import.ts). Previously - // walkSyncableFiles used statSync (followed symlinks). New walker - // uses lstat + inode-cycle + max-depth so the preview matches - // what the real sync will actually walk. - const files = collectSyncableFiles(src.local_path, { strategy: cfg.strategy ?? 'markdown' }); - for (const fullPath of files) { - try { - const stat = statSync(fullPath); - if (stat.size > 5_000_000) continue; // skip large binaries - const content = readFileSync(fullPath, 'utf-8'); - sourceTokens += estimateTokens(content); - sourceFiles++; - } catch { - // Best-effort per file. Skip unreadable files silently; - // sync itself tolerates the same. - } - } - } catch { - // Best-effort: a source whose local_path is gone or unreadable just - // contributes 0. The sync itself would have failed anyway; no point - // blocking the preview on a pre-existing fault. + const unchanged = + isSourceUnchangedSinceSync(src.local_path, src.last_commit, { requireCleanWorkingTree: true }) && + src.chunker_version === currentChunkerVersion; + if (unchanged) { + unchangedSources++; + continue; } - totalTokens += sourceTokens; - totalFiles += sourceFiles; - perSource.push({ path: src.local_path, tokens: sourceTokens, files: sourceFiles }); + changedSources++; + tokens += estimateSourceTreeTokens(src.local_path, cfg.strategy ?? 'markdown').tokens; } + return { tokens, changedSources, unchangedSources }; +} - return { totalTokens, totalFiles, activeSources, perSource }; +/** + * Resolve the inline-path cost-gate floor in USD. Config key + * `sync.cost_gate_min_usd` (DB plane), default $0.50. Below this estimate + * the inline gate proceeds without blocking. Fail-open to the default on a + * missing/invalid value or a config-read error (the gate must never crash + * the sync). Accepts 0 (an operator can set the floor to $0 to make the + * gate block on any nonzero inline cost). + */ +async function resolveCostGateFloorUsd(engine: BrainEngine): Promise { + try { + const raw = await engine.getConfig('sync.cost_gate_min_usd'); + if (raw === null || raw === undefined) return 0.5; + const n = Number(raw); + return Number.isFinite(n) && n >= 0 ? n : 0.5; + } catch { + return 0.5; + } +} + +/** + * Resolve the per-source embed-backfill 24h spend cap (USD) for the deferred + * notice. Mirrors embed-backfill-submit.ts's own resolution of + * SPEND_CAP_CONFIG_KEY (default 25). Fail-open to the default. + */ +async function resolveBackfillCapUsd(engine: BrainEngine): Promise { + try { + const raw = await engine.getConfig(SPEND_CAP_CONFIG_KEY); + if (raw === null || raw === undefined) return 25; + const n = Number(raw); + return Number.isFinite(n) && n > 0 ? n : 25; + } catch { + return 25; + } } /** Interactive [y/N] prompt. Resolves false on non-y answers or EOF. */ @@ -2231,62 +2291,149 @@ See also: // source (no checkout) has nothing for `sync` to pull. Sources with // syncEnabled=false in config.jsonb are skipped too. if (syncAll) { - const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record }>( - `SELECT id, name, local_path, config FROM sources WHERE local_path IS NOT NULL`, + // v0.41.31: SELECT carries last_commit + chunker_version so the inline + // cost preview's "unchanged source → 0" short-circuit can mirror sync's + // own "do work?" gate (sync.ts:1057+1075) + doctor's sync_freshness. + // Both columns predate v0.41 (writeSyncAnchor / writeChunkerVersion); no + // schema migration needed. + const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record; last_commit: string | null; chunker_version: string | null }>( + `SELECT id, name, local_path, config, last_commit, chunker_version FROM sources WHERE local_path IS NOT NULL`, ); if (!sources || sources.length === 0) { console.log('No sources with local_path configured. Use `gbrain sources add --path ` first.'); return; } - // v0.20.0 Cathedral II Layer 8 D1 — cost preview + ConfirmationRequired - // gate. Before kicking off a multi-source sync that may embed tens of - // thousands of chunks (real money), walk the sync-diff set(s), sum - // tokens, compute USD estimate, and gate: - // - TTY + !json + !yes → interactive [y/N] prompt - // - non-TTY OR --json OR piped → emit ConfirmationRequired envelope, - // exit 2 (reserve 1 for runtime errors) - // - --yes → skip prompt entirely - // - --dry-run → preview + exit 0 - // Skipped entirely when --no-embed is set (user already opted out of - // the cost and will run `embed --stale` later). + // v0.41.31 — mode-aware cost gate. Resolve federated_v2 ONCE here so both + // the gate (below) and the fan-out (further down) share it. + const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); + const v2Enabled = await isFederatedV2Enabled(engine); + + // v0.41.31 cost gate (supersedes the v0.20.0 unconditional gate). Under + // federated_v2 sync DEFERS embedding to per-source embed-backfill jobs + // that carry their own $X/source/24h spend cap, so sync itself spends + // nothing synchronously — the gate is INFORMATIONAL (never exit 2) on + // that path. The blocking ConfirmationRequired gate fires ONLY when embed + // runs INLINE (v2 off, or --serial without --no-embed) AND the estimated + // spend exceeds `sync.cost_gate_min_usd` (default $0.50). Skipped entirely + // when --no-embed is set (user opted out; will run `embed --stale` later). if (!noEmbed) { - const preview = estimateSyncAllCost(sources); - const costUsd = estimateEmbeddingCostUsd(preview.totalTokens); - const previewMsg = - `sync --all preview: ${preview.totalFiles} files across ${preview.activeSources} source(s), ` + - `~${preview.totalTokens.toLocaleString()} tokens, est. $${costUsd.toFixed(2)} on ${EMBEDDING_MODEL}.`; - - if (dryRun) { - if (jsonOut) { - console.log(JSON.stringify({ status: 'dry_run', preview, costUsd, model: EMBEDDING_MODEL })); - } else { - console.log(previewMsg); - console.log('--dry-run: exit without syncing.'); - } - return; + const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed }); + // Stale backlog: cheap single SQL; fail-open to 0 so a transient DB + // hiccup never blocks the sync. + let staleChars = 0; + try { + // v0.41.31: signature-aware so a model/dims swap surfaces in the + // backlog estimate (NULL signature grandfathered → not counted). + staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() }); + } catch { + staleChars = 0; } + const rate = currentEmbeddingPricePerMTok(); + const staleCostUsd = estimateCostFromChars(staleChars, rate); + const embeddingModelName = getEmbeddingModelName(); + const floorUsd = await resolveCostGateFloorUsd(engine); - if (!yesFlag) { - const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); - if (!isTTY || jsonOut) { - // Agent-facing path: emit structured envelope, exit 2. - const envelope = serializeError(errorFor({ - class: 'ConfirmationRequired', - code: 'cost_preview_requires_yes', - message: previewMsg, - hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.', - })); - console.log(JSON.stringify({ error: envelope, preview, costUsd, model: EMBEDDING_MODEL })); - process.exit(2); + if (mode === 'deferred') { + // Deferred path: print an FYI, NEVER exit 2. The backfill cap is the + // real money gate (D1/D4). + const capUsd = await resolveBackfillCapUsd(engine); + // v0.41.31 (TODO-2): surface already-queued backfill jobs so a cron + // operator sees work is enqueued, not lost. Best-effort — minion_jobs + // may not exist on a brain that never ran a worker. + let queuedBackfills = 0; + try { + const r = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM minion_jobs + WHERE name = 'embed-backfill' + AND status IN ('waiting','active','delayed','waiting-children')`, + ); + queuedBackfills = Number(r[0]?.n) || 0; + } catch { + queuedBackfills = 0; } - // Interactive TTY path: prompt [y/N]. - console.log(previewMsg); - const answer = await promptYesNo('Proceed? [y/N] '); - if (!answer) { - console.log('Cancelled.'); + const deferredMsg = + `sync --all: embedding deferred to backfill jobs ` + + `(capped $${capUsd}/source/24h, not charged by this sync). ` + + `Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` + + `${embeddingModelName}) across ${sources.length} source(s); ` + + `${queuedBackfills} backfill job(s) queued.`; + if (dryRun) { + if (jsonOut) { + console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName })); + } else { + console.log(deferredMsg); + console.log('--dry-run: exit without syncing.'); + } return; } + if (jsonOut) { + console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName })); + } else { + console.log(deferredMsg); + } + // fall through to sync — no exit 2. + } else { + // Inline path: sync embeds synchronously with no backfill cap to + // protect it, so the blocking gate applies. The BLOCKING cost is the + // new-content estimate ONLY (full-tree ceiling for changed sources; + // unchanged contribute 0) — that's what this sync actually embeds. + // The pre-existing stale backlog (NULL embeddings + signature drift) + // is NOT swept by sync; `gbrain embed --stale` clears it. So we show + // it informationally but never gate on cost this sync won't incur + // (else a model swap would block the next inline cron — F2). + const currentChunkerVersion = String(CHUNKER_VERSION); + const inline = estimateInlineNewTokens(sources, currentChunkerVersion); + const newCostUsd = estimateEmbeddingCostUsd(inline.tokens); + const costUsd = newCostUsd; + const staleNote = staleChars > 0 + ? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)` + : ''; + const previewMsg = + `sync --all preview (inline embed): ${inline.changedSources} changed source(s), ` + + `${inline.unchangedSources} unchanged; ~${inline.tokens.toLocaleString()} new tokens, ` + + `est. $${costUsd.toFixed(2)} on ${embeddingModelName}${staleNote}.`; + + if (dryRun) { + if (jsonOut) { + console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); + } else { + console.log(previewMsg); + console.log('--dry-run: exit without syncing.'); + } + return; + } + + if (!yesFlag) { + if (shouldBlockSync(costUsd, floorUsd, mode)) { + const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); + if (!isTTY || jsonOut) { + // Agent-facing path: emit structured envelope, exit 2. + const envelope = serializeError(errorFor({ + class: 'ConfirmationRequired', + code: 'cost_preview_requires_yes', + message: previewMsg, + hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.', + })); + console.log(JSON.stringify({ error: envelope, mode, gate: 'confirmation_required', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); + process.exit(2); + } + // Interactive TTY path: prompt [y/N]. + console.log(previewMsg); + const answer = await promptYesNo('Proceed? [y/N] '); + if (!answer) { + console.log('Cancelled.'); + return; + } + } else { + // Below floor → proceed without blocking (kills inline-cron noise). + if (jsonOut) { + console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); + } else { + console.log(`${previewMsg} Below cost gate floor ($${floorUsd.toFixed(2)}), proceeding.`); + } + } + } } } @@ -2302,8 +2449,7 @@ See also: // - withSourcePrefix wrap inside runOne so slog/serr lines from // performSync get the [] prefix under parallel mode (D6) // - stable JSON envelope {schema_version:1, sources, ...} when --json - const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); - const v2Enabled = await isFederatedV2Enabled(engine); + // v0.41.31: v2Enabled resolved once above (cost gate). Reused here. const activeSources = sources.filter((s) => { const cfg = (s.config || {}) as { syncEnabled?: boolean }; return cfg.syncEnabled !== false; @@ -2791,6 +2937,13 @@ export interface SyncStatusReportSource { chunks_total: number; chunks_unembedded: number; embedding_coverage_pct: number; + // v0.41.31: embed-backfill job visibility (federated_v2 defers embedding + // to these jobs; without this an operator can't see queued/lagging work + // after `sync --all` exits 0). Best-effort — all 0 / null on brains + // without the minion_jobs table. + backfill_queued: number; + backfill_active: number; + backfill_last_completed_at: string | null; } export interface SyncStatusReport { @@ -2892,6 +3045,43 @@ export async function buildSyncStatusReport( }); } + // v0.41.31: per-source embed-backfill job state. Best-effort — the + // minion_jobs table doesn't exist on every brain (a brain that never ran + // a worker has the pre-minions schema), and the dashboard must not crash + // for that. A failure → empty map → all sources report 0/null. + type BackfillRow = { + source_id: string | null; + queued: string | number; + active: string | number; + last_completed_at: string | Date | null; + }; + const backfillMap = new Map(); + if (sourceIds.length > 0) { + try { + const backfillRows = await engine.executeRaw( + `SELECT data->>'sourceId' AS source_id, + COUNT(*) FILTER (WHERE status IN ('waiting','delayed','waiting-children'))::int AS queued, + COUNT(*) FILTER (WHERE status = 'active')::int AS active, + MAX(finished_at) FILTER (WHERE status = 'completed') AS last_completed_at + FROM minion_jobs + WHERE name = 'embed-backfill' AND data->>'sourceId' = ANY($1::text[]) + GROUP BY data->>'sourceId'`, + [sourceIds], + ); + for (const r of backfillRows) { + if (!r.source_id) continue; + const last = r.last_completed_at; + backfillMap.set(r.source_id, { + queued: Number(r.queued) || 0, + active: Number(r.active) || 0, + last_completed_at: last == null ? null : (last instanceof Date ? last.toISOString() : last), + }); + } + } catch { + // minion_jobs absent / unreadable → leave backfillMap empty. + } + } + const now = Date.now(); const out: SyncStatusReportSource[] = sources.map((src) => { const cfgEntry = (src.config || {}) as { syncEnabled?: boolean }; @@ -2928,6 +3118,9 @@ export async function buildSyncStatusReport( chunks_total: counts.chunks_total, chunks_unembedded: counts.chunks_unembedded, embedding_coverage_pct: embeddingCoveragePct, + backfill_queued: backfillMap.get(src.id)?.queued ?? 0, + backfill_active: backfillMap.get(src.id)?.active ?? 0, + backfill_last_completed_at: backfillMap.get(src.id)?.last_completed_at ?? null, }; }); @@ -2968,7 +3161,7 @@ export function printSyncStatusReport( write(' (no sources registered)'); return; } - const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'LAST SYNC']; + const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'BACKFILL', 'LAST SYNC']; const rows = report.sources.map((s) => { const stale = s.staleness_hours === null ? 'never' @@ -2976,21 +3169,28 @@ export function printSyncStatusReport( const stateBits: string[] = []; if (!s.sync_enabled) stateBits.push('disabled'); stateBits.push(s.staleness_class); + // BACKFILL: active beats queued beats idle for the at-a-glance cell. + const backfill = s.backfill_active > 0 + ? `active(${s.backfill_active})` + : s.backfill_queued > 0 + ? `queued(${s.backfill_queued})` + : 'idle'; return [ s.name, stateBits.join(','), stale, String(s.pages), `${s.embedding_coverage_pct}%`, + backfill, s.last_sync_at ?? '(never)', ]; }); const widths = headers.map((h, i) => Math.max(h.length, ...rows.map((r) => r[i].length)), ); - // Numeric columns (PAGES at index 3, EMBEDDED at index 4, STALENESS - // at index 2) right-pad-left so digits align cleanly. Text columns - // left-pad-right per the existing `sources list` convention. + // Numeric columns (STALENESS=2, PAGES=3, EMBEDDED=4) right-pad-left so + // digits align cleanly. Text columns (incl. BACKFILL=5) left-pad-right + // per the existing `sources list` convention. const NUMERIC_COLS = new Set([2, 3, 4]); const fmt = (cells: string[]) => cells.map((c, i) => (NUMERIC_COLS.has(i) ? c.padStart(widths[i]) : c.padEnd(widths[i]))).join(' '); diff --git a/src/core/embed-stale.ts b/src/core/embed-stale.ts index 8451d6fa1..5ca95cd78 100644 --- a/src/core/embed-stale.ts +++ b/src/core/embed-stale.ts @@ -51,6 +51,14 @@ export interface EmbedStaleOpts { * the gateway. Production callers leave it unset. */ embedFn?: (texts: string[], opts: { abortSignal?: AbortSignal }) => Promise; + /** + * v0.41.31: current embedding provenance signature (`:`). + * When set, embeddings stamped under a DIFFERENT signature are invalidated + * (NULLed) at the start so they flow through the NULL cursor and get + * re-embedded; each page's signature is stamped after its chunks land. + * Omit to keep the legacy `embedding IS NULL`-only behavior. + */ + embeddingSignature?: string; } export interface EmbedStaleResult { @@ -109,6 +117,18 @@ export async function embedStaleForSource( done: false, aborted: false, }; + const signature = opts.embeddingSignature; + + // v0.41.31: invalidate embeddings stamped under a prior model signature so + // the NULL cursor below re-embeds them. GRANDFATHER: NULL signature + // untouched. Best-effort — a failure here must not abort the backfill. + if (signature) { + try { + await engine.invalidateStaleSignatureEmbeddings({ signature, sourceId }); + } catch { + // Non-fatal: fall through to the NULL-only stale loop. + } + } for (;;) { if (signal?.aborted) { @@ -170,6 +190,13 @@ export async function embedStaleForSource( token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); await engine.upsertChunks(slug, merged, { sourceId: keySourceId }); + // v0.41.31: stamp provenance only when EVERY chunk was stale (fully + // re-embedded this pass) — a partially-stale page keeps preserved + // chunks of unknown provenance, so don't claim current. After the + // invalidate pass above, signature-drifted pages ARE fully stale. + if (signature && stale.length === existing.length) { + await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }); + } result.embedded += stale.length; result.pagesProcessed += 1; } catch (e: unknown) { diff --git a/src/core/embedding.ts b/src/core/embedding.ts index 4c1529544..7fe3c9148 100644 --- a/src/core/embedding.ts +++ b/src/core/embedding.ts @@ -12,6 +12,7 @@ import { getEmbeddingModel as gatewayGetModel, getEmbeddingDimensions as gatewayGetDims, } from './ai/gateway.ts'; +import { lookupEmbeddingPrice } from './embedding-pricing.ts'; // v0.27.1: re-export multimodal embedding so callers can pull both text and // image embedding APIs from `src/core/embedding`. import-image-file consumes @@ -127,13 +128,106 @@ export const EMBEDDING_MODEL = 'text-embedding-3-large'; export const EMBEDDING_DIMENSIONS = 1536; /** - * USD cost per 1k tokens for text-embedding-3-large. Used by - * `gbrain sync --all` cost preview and `reindex-code` to surface - * expected spend before accepting expensive operations. + * USD cost per 1k tokens for text-embedding-3-large. Retained for back-compat + * with callers/tests that import it directly; new cost math resolves the + * ACTUAL configured model's rate via embedding-pricing.ts instead of assuming + * OpenAI. (Hardcoding this rate produced cost previews that named the wrong + * provider and over-stated spend ~2.6x when the brain ran on a cheaper model.) */ export const EMBEDDING_COST_PER_1K_TOKENS = 0.00013; -/** Compute USD cost estimate for embedding `tokens` at current model rate. */ +/** + * Resolve the price-per-1M-tokens for the currently-configured embedding + * model. Falls back to the OpenAI text-embedding-3-large rate only when the + * model is unknown to the pricing table. + */ +export function currentEmbeddingPricePerMTok(): number { + let modelString: string; + try { + modelString = gatewayGetModel(); // e.g. 'zeroentropyai:zembed-1' + } catch { + // Gateway not configured (e.g. unit tests, cost preview before connect). + // Fall back to the OpenAI text-embedding-3-large default rate. + return 0.13; + } + const hit = lookupEmbeddingPrice(modelString); + return hit.kind === 'known' ? hit.pricePerMTok : 0.13; +} + +/** + * Compute USD cost estimate for embedding `tokens` at the CURRENT configured + * model's rate (not a hardcoded OpenAI rate). + */ export function estimateEmbeddingCostUsd(tokens: number): number { - return (tokens / 1000) * EMBEDDING_COST_PER_1K_TOKENS; + return (tokens / 1_000_000) * currentEmbeddingPricePerMTok(); +} + +/** + * Embedding provenance signature for the currently-configured model: + * `:` (e.g. `openai:text-embedding-3-large:1536`). + * Stamped onto `pages.embedding_signature` when a page's chunks are + * embedded so a later model/dimension swap can be detected as stale. + * + * Deliberately does NOT include the chunker version — chunker drift is + * already tracked per-page via `pages.chunker_version` (used by sync + + * doctor). This signature is strictly about the EMBEDDING space. + * + * Falls back to the OpenAI default signature when the gateway is + * unconfigured (unit-test context), matching the other estimator fallbacks. + */ +export function currentEmbeddingSignature(): string { + try { + return `${gatewayGetModel()}:${gatewayGetDims()}`; + } catch { + return `${EMBEDDING_MODEL}:${EMBEDDING_DIMENSIONS}`; + } +} + +/** + * Whether a `gbrain sync --all` invocation will embed at sync time + * ('inline') or defer embedding to per-source `embed-backfill` minion jobs + * ('deferred'). Under federated_v2 the default path defers; the backfill + * jobs carry their own 10-min cooldown + $25/source/24h spend cap, so the + * sync-time cost gate only BLOCKS on the inline path. See sync.ts:2346 + * (`effectiveNoEmbed`) — this mirrors that resolution exactly. + */ +export type SyncEmbedMode = 'deferred' | 'inline'; + +/** + * Resolve the embed mode from the same three signals sync.ts uses to + * compute `effectiveNoEmbed`. Single source of truth so the cost gate and + * the actual embed decision can never drift. + * + * effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed + * + * Embed runs INLINE iff that resolves to false: + * - v2 off → inline (legacy synchronous embed) + * - v2 on + --serial + !--no-embed → inline + * - v2 on (parallel) → deferred (backfill jobs) + * - --no-embed (any path) → the caller skips the gate entirely; + * we report 'deferred' for completeness. + */ +export function willEmbedSynchronously(opts: { + v2Enabled: boolean; + serialFlag: boolean; + noEmbed: boolean; +}): SyncEmbedMode { + const effectiveNoEmbed = + opts.v2Enabled && !opts.serialFlag && !opts.noEmbed ? true : opts.noEmbed; + return effectiveNoEmbed ? 'deferred' : 'inline'; +} + +/** + * Pure cost-gate decision. The gate BLOCKS (prompt in TTY, exit 2 envelope + * in non-TTY) only when embed runs inline AND the estimated spend exceeds + * the floor. Deferred mode NEVER blocks — the backfill cap is the real + * money gate, and blocking the cheap markdown import for cost the import + * doesn't synchronously incur is the bug this fix removes. + */ +export function shouldBlockSync( + costUsd: number, + floorUsd: number, + mode: SyncEmbedMode, +): boolean { + return mode === 'inline' && costUsd > floorUsd; } diff --git a/src/core/engine.ts b/src/core/engine.ts index b9861d0d9..f76baddbc 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -956,7 +956,38 @@ export interface BrainEngine { * `gbrain embed --stale --source media-corpus` expect only that * source's NULLs touched; the caller threads `sourceId` here. */ - countStaleChunks(opts?: { sourceId?: string }): Promise; + countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise; + /** + * Sum of LENGTH(chunk_text) over stale chunks — the character-count + * backlog the embed phase / embed-backfill will process. Sibling of + * countStaleChunks (same stale predicate + embed_skip filter + optional + * sourceId scope); used by the `gbrain sync --all` cost preview to price + * the embedding backlog via estimateCostFromChars. Returns 0 on an + * empty/fully-embedded brain. + * + * v0.41.31: `signature` (optional) widens "stale" to ALSO include chunks + * whose page `embedding_signature` is set AND differs from the current + * model signature (a model/dims swap). NULL signature is GRANDFATHERED + * (never counted) so the post-migration corpus isn't flagged en masse. + * Omit `signature` for the legacy `embedding IS NULL`-only count. + */ + sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise; + /** + * Stamp `pages.embedding_signature = signature` for one page. Called after + * a page's chunks are (re)embedded so a later model swap can detect it as + * stale. Idempotent. No-op if the page doesn't exist. + */ + setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise; + /** + * NULL out the embeddings (and embedded_at) of every chunk whose page + * `embedding_signature` is set AND differs from `signature` — i.e. pages + * embedded under a now-stale model. Returns the chunk count invalidated. + * The embed-stale loop calls this BEFORE listStaleChunks so signature- + * drift pages flow through the existing NULL-embedding cursor (keeps + * listStaleChunks's keyset pagination untouched). GRANDFATHER: NULL + * signature is never invalidated. `sourceId` scopes the sweep. + */ + invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise; /** * Return every chunk where embedding IS NULL, with the metadata needed * to call embedBatch + upsertChunks. The `embedding` column is omitted diff --git a/src/core/import-file.ts b/src/core/import-file.ts index 4dc34525b..3beb34328 100644 --- a/src/core/import-file.ts +++ b/src/core/import-file.ts @@ -8,7 +8,7 @@ import { chunkText } from './chunkers/recursive.ts'; import { chunkCodeText, chunkCodeTextFull, detectCodeLanguage, CHUNKER_VERSION } from './chunkers/code.ts'; import { findChunkForOffset } from './chunkers/edge-extractor.ts'; import { extractCodeRefs, imageOfCandidates } from './link-extraction.ts'; -import { embedBatch, embedMultimodal } from './embedding.ts'; +import { embedBatch, embedMultimodal, currentEmbeddingSignature } from './embedding.ts'; import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts'; import type { ChunkInput, PageInput, PageType } from './types.ts'; import { computeEffectiveDate } from './effective-date.ts'; @@ -658,6 +658,13 @@ export async function importFromContent( if (chunks.length > 0) { await tx.upsertChunks(slug, chunks, txOpts); + // v0.41.31: stamp embedding provenance when this import actually + // embedded (not --no-embed), so a later model/dims swap is detectable + // as stale via embed --stale. The deferred/backfill + per-slug embed + // paths stamp too; this covers the inline import/sync path. + if (!opts.noEmbed) { + await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() }); + } } else { // Content is empty — delete stale chunks so they don't ghost in search results await tx.deleteChunks(slug, txOpts); @@ -968,6 +975,14 @@ export async function importCodeFile( if (chunks.length > 0) { await tx.upsertChunks(slug, chunks, txOpts); + // v0.41.31: stamp embedding provenance ONLY when every chunk was + // freshly embedded with the current model this call (no reuse-by-hash + // carrying old-model vectors). Mixed pages stay unstamped rather than + // falsely marked current; `reindex --code --force` / `embed --stale` + // handle the swap for those. + if (!opts.noEmbed && needsEmbedIndexes.length === chunks.length) { + await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() }); + } } else { await tx.deleteChunks(slug, txOpts); } diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 4de08e538..b326ef980 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4920,6 +4920,32 @@ export const MIGRATIONS: Migration[] = [ EXECUTE FUNCTION bump_page_generation_clock_fn(); `, }, + { + version: 108, + name: 'pages_embedding_signature', + // v0.41.31 — embedding provenance for real stale semantics. + // + // Adds `pages.embedding_signature TEXT NULL` = `:` + // stamped when a page's chunks are embedded (setPageEmbeddingSignature). + // A later model/dimension swap makes the stored signature differ from + // the current one, so countStaleChunks/sumStaleChunkChars (with the + // `signature` opt) and invalidateStaleSignatureEmbeddings can detect and + // re-embed those pages. + // + // GRANDFATHER (critical): the stale predicate is + // `embedding_signature IS NOT NULL AND embedding_signature <> $current` + // so a NULL signature is NEVER stale. After this migration every existing + // page has NULL — none are flagged — so the next `embed --stale` does NOT + // re-embed the whole corpus. Signatures only get stamped going forward. + // + // No index: the column is read only via a JOINed pages row in the + // chunk-grain stale queries; no standalone lookup hot path. ADD COLUMN + // with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. + idempotent: true, + sql: ` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT NULL; + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/handlers/embed-backfill.ts b/src/core/minions/handlers/embed-backfill.ts index 02fd0dcad..221d5b676 100644 --- a/src/core/minions/handlers/embed-backfill.ts +++ b/src/core/minions/handlers/embed-backfill.ts @@ -35,6 +35,7 @@ import { tryAcquireDbLock } from '../../db-lock.ts'; import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts'; import { withBudgetTracker } from '../../ai/gateway.ts'; import { embedStaleForSource } from '../../embed-stale.ts'; +import { currentEmbeddingSignature } from '../../embedding.ts'; import type { BrainEngine } from '../../engine.ts'; import type { MinionJobContext } from '../types.ts'; @@ -123,6 +124,9 @@ export function makeEmbedBackfillHandler(engine: BrainEngine) { embedStaleForSource(engine, sourceId, { batchSize, signal: job.signal, + // v0.41.31: re-embed pages whose model signature drifted + stamp + // provenance as chunks land. + embeddingSignature: currentEmbeddingSignature(), onProgress: ({ embedded, chunksProcessed, cursor }) => { // Fire-and-forget; updateProgress returns a Promise but the // handler is sync inside the loop. diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 612d5de5f..f6063c754 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -422,7 +422,9 @@ export class PGLiteEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema='public' AND table_name='sources' AND column_name='trust_frontmatter_overrides') AS sources_trust_fm_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists + WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='embedding_signature') AS pages_embedding_signature_exists `); const probe = rows[0] as { pages_exists: boolean; @@ -463,6 +465,7 @@ export class PGLiteEngine implements BrainEngine { sources_cr_mode_exists: boolean; sources_trust_fm_exists: boolean; pages_generation_exists: boolean; + pages_embedding_signature_exists: boolean; }; const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists; @@ -530,6 +533,10 @@ export class PGLiteEngine implements BrainEngine { // body; bootstrap only needs to add the column on pre-v91 brains so // the CREATE INDEX doesn't crash. const needsPagesGeneration = probe.pages_exists && !probe.pages_generation_exists; + // v0.41.31 (v108): pages.embedding_signature for real stale semantics. + // No SCHEMA_SQL index references it today; bootstrap is defense-in-depth + // so future schema work doesn't wedge pre-v108 brains. + const needsPagesEmbeddingSignature = probe.pages_exists && !probe.pages_embedding_signature_exists; // Fresh installs (no tables yet) and modern brains both no-op. if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap @@ -539,7 +546,8 @@ export class PGLiteEngine implements BrainEngine { && !needsFilesBootstrap && !needsOauthClientsBootstrap && !needsSourcesArchive && !needsPagesLastRetrievedAt && !needsPagesProvenance - && !needsContextualRetrievalColumns && !needsPagesGeneration) return; + && !needsContextualRetrievalColumns && !needsPagesGeneration + && !needsPagesEmbeddingSignature) return; process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n'); @@ -767,6 +775,15 @@ export class PGLiteEngine implements BrainEngine { ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1; `); } + + if (needsPagesEmbeddingSignature) { + // v108 (pages_embedding_signature): embedding provenance for real + // stale semantics. NULL grandfathered (never stale). v108 runs later + // via runMigrations and is idempotent. + await this.db.exec(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT; + `); + } } async withReservedConnection(fn: (conn: ReservedConnection) => Promise): Promise { @@ -2061,35 +2078,91 @@ export class PGLiteEngine implements BrainEngine { return (rows as Record[]).map(r => rowToChunk(r)); } - async countStaleChunks(opts?: { sourceId?: string }): Promise { - // D7: source-scoped count for `gbrain embed --stale --source X`. - // v0.41 (D4+D8+Codex r2 #11): always JOIN pages so embed-skip filter - // applies via `NOT (frontmatter ? 'embed_skip')`. PGLite is - // PostgreSQL 17.5 in WASM and supports the full JSONB operator set. - if (opts?.sourceId === undefined) { - const { rows } = await this.db.query( - `SELECT count(*)::int AS count - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`, - ); - const count = (rows[0] as { count: number } | undefined)?.count ?? 0; - return Number(count); + /** + * Build the stale-chunk WHERE clause + positional params. embed_skip is + * always excluded. `signature` widens "stale" to include embedding_signature + * drift (NULL grandfathered → never stale). Shared by countStaleChunks + + * sumStaleChunkChars so they can't drift. + */ + private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } { + const params: unknown[] = []; + const conds: string[] = []; + if (opts?.signature !== undefined) { + params.push(opts.signature); + conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`); + } else { + conds.push(`cc.embedding IS NULL`); } + conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`); + if (opts?.sourceId !== undefined) { + params.push(opts.sourceId); + conds.push(`p.source_id = $${params.length}`); + } + return { where: conds.join(' AND '), params }; + } + + async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise { + // D7: source-scoped count for `gbrain embed --stale --source X`. Always + // JOIN pages so embed-skip + signature predicates apply. PGLite is + // PostgreSQL 17.5 in WASM and supports the full JSONB operator set. + const { where, params } = this.buildStaleChunkWhere(opts); const { rows } = await this.db.query( `SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND p.source_id = $1 - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`, - [opts.sourceId], + WHERE ${where}`, + params, ); const count = (rows[0] as { count: number } | undefined)?.count ?? 0; return Number(count); } + async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise { + // Sibling of countStaleChunks: same stale predicate, summing chunk_text + // length for the sync cost preview. ::bigint guards int4 overflow. + const { where, params } = this.buildStaleChunkWhere(opts); + const { rows } = await this.db.query( + `SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE ${where}`, + params, + ); + const chars = (rows[0] as { chars: number | string } | undefined)?.chars ?? 0; + return Number(chars); + } + + async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise { + await this.db.query( + `UPDATE pages SET embedding_signature = $1 WHERE slug = $2 AND source_id = $3`, + [opts.signature, slug, opts.sourceId ?? 'default'], + ); + } + + async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise { + // NULL out embeddings whose page signature is set AND differs from the + // current model signature. GRANDFATHER: NULL signature untouched. Feeds + // the existing NULL-embedding cursor so listStaleChunks stays unchanged. + const params: unknown[] = [opts.signature]; + let srcClause = ''; + if (opts.sourceId !== undefined) { + params.push(opts.sourceId); + srcClause = ` AND p.source_id = $${params.length}`; + } + const { rows } = await this.db.query( + `UPDATE content_chunks cc + SET embedding = NULL, embedded_at = NULL + FROM pages p + WHERE cc.page_id = p.id + AND cc.embedding IS NOT NULL + AND p.embedding_signature IS NOT NULL + AND p.embedding_signature <> $1${srcClause} + RETURNING cc.page_id`, + params, + ); + return (rows as unknown[]).length; + } + async listStaleChunks(opts?: { batchSize?: number; afterPageId?: number; diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 2fb90fd39..48db794ca 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -455,7 +455,9 @@ export class PostgresEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'trust_frontmatter_overrides') AS sources_trust_fm_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'embedding_signature') AS pages_embedding_signature_exists `; const probe = probeRows[0]!; @@ -530,6 +532,7 @@ export class PostgresEngine implements BrainEngine { sources_cr_mode_exists?: boolean; sources_trust_fm_exists?: boolean; pages_generation_exists?: boolean; + pages_embedding_signature_exists?: boolean; }; const needsContextualRetrievalColumns = (probe.pages_exists && (!probeCr.pages_cr_mode_exists || !probeCr.pages_corpus_generation_exists)) @@ -540,6 +543,9 @@ export class PostgresEngine implements BrainEngine { // it. Pre-v91 brains crash without the column; bootstrap adds it before // SCHEMA_SQL replay creates the index. const needsPagesGeneration = probe.pages_exists && !probeCr.pages_generation_exists; + // v0.41.31 (v108): pages.embedding_signature for real stale semantics. + // No SCHEMA_SQL index references it; bootstrap is defense-in-depth. + const needsPagesEmbeddingSignature = probe.pages_exists && !probeCr.pages_embedding_signature_exists; if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt && !needsMcpLogBootstrap && !needsSubagentProviderId @@ -548,7 +554,8 @@ export class PostgresEngine implements BrainEngine { && !needsOauthClientsBootstrap && !needsSourcesArchive && !needsPagesLastRetrievedAt && !needsPagesProvenance - && !needsContextualRetrievalColumns && !needsPagesGeneration) return; + && !needsContextualRetrievalColumns && !needsPagesGeneration + && !needsPagesEmbeddingSignature) return; process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n'); @@ -774,6 +781,15 @@ export class PostgresEngine implements BrainEngine { ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1; `); } + + if (needsPagesEmbeddingSignature) { + // v108 (pages_embedding_signature): embedding provenance for real stale + // semantics. NULL grandfathered. v108 runs later via runMigrations and + // is idempotent. + await conn.unsafe(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT; + `); + } } async transaction(fn: (engine: BrainEngine) => Promise): Promise { @@ -2093,36 +2109,88 @@ export class PostgresEngine implements BrainEngine { return rows.map((r) => rowToChunk(r as Record)); } - async countStaleChunks(opts?: { sourceId?: string }): Promise { - const sql = this.sql; - // v0.41 (D4+D8+Codex r2 #11): the embed-skip filter requires JOIN - // pages so we always join — the pre-v0.41 "fast path" without join - // is gone. JSONB `?` existence check is cheap on the small set of - // skipped pages; full-scan benefits from the partial index on - // embedding IS NULL regardless. - // - // D7: source_id scoping. NULL/undefined = scan all sources; - // a value scopes to that source so `gbrain embed --stale --source X` - // does what it says. - if (opts?.sourceId === undefined) { - const [row] = await sql` - SELECT count(*)::int AS count - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') - `; - return Number((row as { count?: number } | undefined)?.count ?? 0); + /** + * Build the stale-chunk WHERE clause + positional params for sql.unsafe. + * embed_skip always excluded. `signature` widens "stale" to include + * embedding_signature drift (NULL grandfathered). Shared by + * countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling). + */ + private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } { + const params: unknown[] = []; + const conds: string[] = []; + if (opts?.signature !== undefined) { + params.push(opts.signature); + conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`); + } else { + conds.push(`cc.embedding IS NULL`); } - const [row] = await sql` - SELECT count(*)::int AS count - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND p.source_id = ${opts.sourceId} - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') + conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`); + if (opts?.sourceId !== undefined) { + params.push(opts.sourceId); + conds.push(`p.source_id = $${params.length}`); + } + return { where: conds.join(' AND '), params }; + } + + async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise { + // Always JOIN pages so the embed_skip + signature predicates apply. + // D7: source_id scoping. v0.41.31: optional signature widens staleness + // to embedding_signature drift (NULL grandfathered). + const { where, params } = this.buildStaleChunkWhere(opts); + const rows = await this.sql.unsafe( + `SELECT count(*)::int AS count + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE ${where}`, + params as Parameters[1], + ); + return Number((rows[0] as { count?: number } | undefined)?.count ?? 0); + } + + async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise { + // Sibling of countStaleChunks: same stale predicate, summing chunk_text + // length for the sync cost preview. ::bigint guards int4 overflow. + const { where, params } = this.buildStaleChunkWhere(opts); + const rows = await this.sql.unsafe( + `SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE ${where}`, + params as Parameters[1], + ); + return Number((rows[0] as { chars?: number | string } | undefined)?.chars ?? 0); + } + + async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise { + const sql = this.sql; + await sql` + UPDATE pages SET embedding_signature = ${opts.signature} + WHERE slug = ${slug} AND source_id = ${opts.sourceId ?? 'default'} `; - return Number((row as { count?: number } | undefined)?.count ?? 0); + } + + async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise { + // NULL embeddings whose page signature is set AND differs from current. + // GRANDFATHER: NULL signature untouched. Feeds the NULL-embedding cursor + // so listStaleChunks stays unchanged. RETURNING → row count. + const params: unknown[] = [opts.signature]; + let srcClause = ''; + if (opts.sourceId !== undefined) { + params.push(opts.sourceId); + srcClause = ` AND p.source_id = $${params.length}`; + } + const rows = await this.sql.unsafe( + `UPDATE content_chunks cc + SET embedding = NULL, embedded_at = NULL + FROM pages p + WHERE cc.page_id = p.id + AND cc.embedding IS NOT NULL + AND p.embedding_signature IS NOT NULL + AND p.embedding_signature <> $1${srcClause} + RETURNING cc.page_id`, + params as Parameters[1], + ); + return (rows as unknown[]).length; } async listStaleChunks(opts?: { diff --git a/src/core/source-health.ts b/src/core/source-health.ts index 13135724a..8553a8358 100644 --- a/src/core/source-health.ts +++ b/src/core/source-health.ts @@ -33,6 +33,10 @@ export interface SourceMetrics { failed_jobs_24h: number; /** Waiting + active + delayed jobs (sync OR embed-backfill) for this source. */ queue_depth: number; + /** v0.41.31: embed-backfill jobs specifically active right now. */ + backfill_active: number; + /** v0.41.31: embed-backfill jobs queued (waiting/delayed/waiting-children). */ + backfill_queued: number; tracked_branch: string | null; priority_label: PriorityLabel; /** Webhook configured? (true iff config.webhook_secret is set.) */ @@ -131,7 +135,7 @@ export async function computeAllSourceMetrics( const cfg = parseSourceConfig(src.config); const pages = pageCounts.get(src.id) ?? 0; const chunkStats = chunkCounts.get(src.id) ?? { total: 0, embedded: 0 }; - const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0 }; + const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0, backfill_active: 0, backfill_queued: 0 }; const embedCoverage = chunkStats.total === 0 ? 100 @@ -155,6 +159,8 @@ export async function computeAllSourceMetrics( lag_seconds: lagSeconds, failed_jobs_24h: jobStats.failed_24h, queue_depth: jobStats.queue_depth, + backfill_active: jobStats.backfill_active, + backfill_queued: jobStats.backfill_queued, tracked_branch: typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : null, priority_label: resolvePriorityLabel(src.id, src.config), webhook_configured: typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0, @@ -189,21 +195,30 @@ async function chunkCountsBySource(engine: BrainEngine): Promise> { +type JobStats = { failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number }; + +async function jobCountsBySource(engine: BrainEngine): Promise> { // Pre-v0.11 brains don't have minion_jobs; return empty map. try { - const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number }>( + const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number }>( `SELECT data->>'sourceId' AS source_id, COUNT(*) FILTER (WHERE status IN ('failed','dead') AND created_at > NOW() - INTERVAL '24 hours')::int AS failed_24h, - COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth + COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth, + COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status = 'active')::int AS backfill_active, + COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status IN ('waiting','delayed','waiting-children'))::int AS backfill_queued FROM minion_jobs WHERE name IN ('sync','embed-backfill') AND data->>'sourceId' IS NOT NULL GROUP BY data->>'sourceId'`, ); - const m = new Map(); + const m = new Map(); for (const r of rows) { - m.set(r.source_id, { failed_24h: Number(r.failed_24h), queue_depth: Number(r.queue_depth) }); + m.set(r.source_id, { + failed_24h: Number(r.failed_24h), + queue_depth: Number(r.queue_depth), + backfill_active: Number(r.backfill_active), + backfill_queued: Number(r.backfill_queued), + }); } return m; } catch { diff --git a/test/cosine-rescore-column.test.ts b/test/cosine-rescore-column.test.ts index 7e576ac50..9010b9ba1 100644 --- a/test/cosine-rescore-column.test.ts +++ b/test/cosine-rescore-column.test.ts @@ -19,6 +19,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { EmbeddingColumnNotRegisteredError, } from '../src/core/search/embedding-column.ts'; +import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts'; import type { PageInput, ChunkInput } from '../src/core/types.ts'; let engine: PGLiteEngine; @@ -26,6 +27,17 @@ let chunkId: number; let chunkId2: number; beforeAll(async () => { + // Pin the embedding dim to 1536 BEFORE initSchema. initSchema sizes the + // `embedding` column from getEmbeddingDimensions() (default 1280 = + // zeroentropyai). This test hardcodes 1536-dim vectors + asserts 1536, so + // it must NOT inherit ambient/leaked gateway state (which is 1536 from a + // local ~/.gbrain config but 1280 in CI → vector(1280) → insert fails). + // Pinning here makes the column deterministically 1536 regardless of order. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test-cosine-rescore' }, + }); engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); @@ -87,6 +99,7 @@ beforeAll(async () => { afterAll(async () => { await engine.disconnect(); + resetGateway(); }); describe('getEmbeddingsByChunkIds — column parameter (D9)', () => { diff --git a/test/e2e/cycle.test.ts b/test/e2e/cycle.test.ts index 95b73e4ab..e49319f99 100644 --- a/test/e2e/cycle.test.ts +++ b/test/e2e/cycle.test.ts @@ -25,6 +25,8 @@ mock.module('../../src/core/embedding.ts', () => ({ // Deterministic fake vector for each chunk. return texts.map(() => new Float32Array(1536)); }, + // v0.41.31: embed phase reads the current signature to stamp provenance. + currentEmbeddingSignature: () => 'test:model:1536', })); const { runCycle } = await import('../../src/core/cycle.ts'); diff --git a/test/e2e/dream-cycle-phase-order-pglite.test.ts b/test/e2e/dream-cycle-phase-order-pglite.test.ts index 151a79ca1..f434f1e57 100644 --- a/test/e2e/dream-cycle-phase-order-pglite.test.ts +++ b/test/e2e/dream-cycle-phase-order-pglite.test.ts @@ -46,6 +46,8 @@ mock.module('../../src/core/embedding.ts', () => ({ EMBEDDING_DIMENSIONS: 1536, EMBEDDING_COST_PER_1K_TOKENS: 0.00013, estimateEmbeddingCostUsd: (tokens: number) => (tokens / 1000) * 0.00013, + // v0.41.31: embed phase reads the current signature to stamp provenance. + currentEmbeddingSignature: () => 'text-embedding-3-large:1536', })); const { runCycle, ALL_PHASES } = await import('../../src/core/cycle.ts'); @@ -106,6 +108,9 @@ async function withoutAnthropicKey(body: () => Promise): Promise { // v0.31 — added `consolidate` between recompute_emotional_weight and embed // v0.33 — added `resolve_symbol_edges` between extract and patterns type CyclePhase = (typeof ALL_PHASES)[number]; +// Mirrors src/core/cycle.ts ALL_PHASES order exactly. v0.41.31: synced the +// three phases that drifted in after this test was last touched (v0.41.0.0): +// extract_atoms (v0.41 T9), synthesize_concepts, conversation_facts_backfill. const EXPECTED_PHASES: CyclePhase[] = [ 'lint', 'backlinks', @@ -113,13 +118,16 @@ const EXPECTED_PHASES: CyclePhase[] = [ 'synthesize', 'extract', 'extract_facts', // v0.32.2 — reconcile fence → DB facts index + 'extract_atoms', // v0.41 T9 — pack-gated atom extraction 'resolve_symbol_edges', // v0.33.3 — within-file symbol resolution 'patterns', + 'synthesize_concepts', // v0.41 — concept synthesis after patterns 'recompute_emotional_weight', // v0.29 'consolidate', // v0.31 'propose_takes', // v0.36.1.0 — hindsight calibration wave 'grade_takes', // v0.36.1.0 'calibration_profile', // v0.36.1.0 + 'conversation_facts_backfill', // v0.41.11.0 — config-gated (default off) 'embed', 'orphans', 'schema-suggest', // v0.39.0.0 — passive schema-suggest after orphans diff --git a/test/e2e/dream.test.ts b/test/e2e/dream.test.ts index 867af2ce8..7687003c9 100644 --- a/test/e2e/dream.test.ts +++ b/test/e2e/dream.test.ts @@ -19,6 +19,8 @@ import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers. // Mock embedBatch so embed phase doesn't call OpenAI. mock.module('../../src/core/embedding.ts', () => ({ embedBatch: async (texts: string[]) => texts.map(() => new Float32Array(1536)), + // v0.41.31: embed phase reads the current signature to stamp provenance. + currentEmbeddingSignature: () => 'test:model:1536', })); const { runDream } = await import('../../src/commands/dream.ts'); diff --git a/test/e2e/sync-status-pglite.test.ts b/test/e2e/sync-status-pglite.test.ts index 8a8d993e9..0773ec4a5 100644 --- a/test/e2e/sync-status-pglite.test.ts +++ b/test/e2e/sync-status-pglite.test.ts @@ -197,6 +197,43 @@ describe('buildSyncStatusReport against real PGLite (IRON RULE regression for Bl expect(b.embedding_coverage_pct).toBe(100); }); + test('v0.41.31: embed-backfill job state surfaced per source (TODO-2)', async () => { + // Seed embed-backfill minion jobs for source-a: 2 queued + 1 active + + // 1 completed. source-b has none → idle. + await engine.executeRaw( + `INSERT INTO minion_jobs (name, status, data) VALUES + ('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb), + ('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb), + ('embed-backfill', 'active', '{"sourceId":"source-a"}'::jsonb)`, + ); + await engine.executeRaw( + `INSERT INTO minion_jobs (name, status, data, finished_at) + VALUES ('embed-backfill', 'completed', '{"sourceId":"source-a"}'::jsonb, now())`, + ); + + const sources = await engine.executeRaw<{ + id: string; name: string; local_path: string | null; config: Record; + }>( + `SELECT id, name, local_path, config FROM sources + WHERE local_path IS NOT NULL AND archived IS NOT TRUE ORDER BY id`, + ); + const report = await buildSyncStatusReport(engine, sources); + const byId = new Map(report.sources.map((s) => [s.source_id, s])); + + const a = byId.get('source-a')!; + expect(a.backfill_queued).toBe(2); + expect(a.backfill_active).toBe(1); + expect(a.backfill_last_completed_at).not.toBeNull(); + + const b = byId.get('source-b')!; + expect(b.backfill_queued).toBe(0); + expect(b.backfill_active).toBe(0); + expect(b.backfill_last_completed_at).toBeNull(); + + // Clean up so sibling tests in this describe see a clean minion_jobs. + await engine.executeRaw(`DELETE FROM minion_jobs WHERE name = 'embed-backfill'`); + }); + test('soft-deleted pages excluded from pages count (v0.26.5 regression)', async () => { // Verifies the `WHERE pg.deleted_at IS NULL` clause in BOTH subqueries // of the dashboard SQL. Pre-fix the original PR query would have diff --git a/test/embed.serial.test.ts b/test/embed.serial.test.ts index 5caaa0fc5..ed8058140 100644 --- a/test/embed.serial.test.ts +++ b/test/embed.serial.test.ts @@ -32,6 +32,11 @@ mock.module('../src/core/embedding.ts', () => ({ activeEmbedCalls--; } }, + // v0.41.31: embedAll/embedAllStale read the current embedding signature to + // stamp provenance. The mock returns a stable value; the mock engine's + // setPageEmbeddingSignature / invalidateStaleSignatureEmbeddings resolve to + // null via the Proxy default, so the signature value is inert here. + currentEmbeddingSignature: () => 'test:model:1536', })); // Import AFTER mocking. @@ -105,6 +110,29 @@ describe('runEmbed --all (parallel)', () => { expect(maxConcurrentEmbedCalls).toBeLessThanOrEqual(10); }); + test('v0.41.31: stamps embedding_signature after embedding each page (--all)', async () => { + const pages = [{ slug: 'a', source_id: 'default' }, { slug: 'b', source_id: 'default' }]; + const chunksBySlug = new Map( + pages.map(p => [ + p.slug, + [{ chunk_index: 0, chunk_text: `text ${p.slug}`, chunk_source: 'compiled_truth', embedded_at: null, token_count: 4 }], + ]), + ); + const engine = mockEngine({ + listPages: async () => pages, + getChunks: async (slug: string) => chunksBySlug.get(slug) || [], + upsertChunks: async () => {}, + }); + + await runEmbed(engine, ['--all']); + + // The wiring gap this pins: embedAll must CALL setPageEmbeddingSignature + // after upsertChunks, with the current signature (mocked to test:model:1536). + const stampCalls = (engine as any)._calls.filter((c: any) => c.method === 'setPageEmbeddingSignature'); + expect(stampCalls.length).toBe(2); // one per page + expect(stampCalls[0].args[1]).toEqual({ sourceId: 'default', signature: 'test:model:1536' }); + }); + test('respects GBRAIN_EMBED_CONCURRENCY=1 (serial)', async () => { const pages = Array.from({ length: 5 }, (_, i) => ({ slug: `page-${i}` })); const chunksBySlug = new Map( diff --git a/test/embedding-signature-stale.test.ts b/test/embedding-signature-stale.test.ts new file mode 100644 index 000000000..c7d3e626e --- /dev/null +++ b/test/embedding-signature-stale.test.ts @@ -0,0 +1,121 @@ +/** + * v0.41.31 — real stale semantics via pages.embedding_signature (PGLite). + * + * Pins the commit-3 contract: + * - R-4 (grandfather, CRITICAL): a page embedded under a NULL signature is + * NEVER stale. After the v108 migration every existing page has NULL, so + * the next embed --stale must NOT re-embed the whole corpus. + * - signature mismatch (model/dims swap) → counted as stale. + * - matching signature → not stale. + * - invalidateStaleSignatureEmbeddings NULLs only mismatched (grandfathered + * NULL + matching untouched) and returns the count. + * - setPageEmbeddingSignature stamps. + * + * Canonical PGLite block (CLAUDE.md R3+R4). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +let colDim: number; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`, + ); + colDim = Number(rows[0]?.dim); +}); + +/** + * Seed a page with one EMBEDDED chunk (non-null vector) and a given + * embedding_signature (null → grandfathered legacy state). + */ +async function seedEmbedded(slug: string, text: string, signature: string | null, sourceId?: string): Promise { + await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }, sourceId ? { sourceId } : undefined); + const chunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined }, + ]; + await engine.upsertChunks(slug, chunks, sourceId ? { sourceId } : undefined); + // Flip the chunk to a non-null vector sized to the actual column dim. + await engine.executeRaw( + `UPDATE content_chunks + SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector + WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = $3)`, + [colDim, slug, sourceId ?? 'default'], + ); + if (signature !== null) { + await engine.setPageEmbeddingSignature(slug, { sourceId, signature }); + } +} + +describe('embedding_signature stale semantics', () => { + test('R-4 GRANDFATHER: NULL signature is never stale', async () => { + await seedEmbedded('legacy', 'abcde', null); // embedded, NULL signature + // No NULL embeddings, NULL signature → not stale under any signature. + expect(await engine.countStaleChunks({ signature: 'openai:m:1536' })).toBe(0); + expect(await engine.sumStaleChunkChars({ signature: 'openai:m:1536' })).toBe(0); + }); + + test('signature MISMATCH (model swap) is counted as stale', async () => { + await seedEmbedded('drifted', 'abcde', 'openai:old:1536'); // 5 chars + expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(1); + expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(5); + // Without the signature opt, the legacy NULL-only predicate ignores it. + expect(await engine.countStaleChunks()).toBe(0); + }); + + test('MATCHING signature is not stale', async () => { + await seedEmbedded('fresh', 'abcde', 'voyage:new:1024'); + expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(0); + expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(0); + }); + + test('invalidateStaleSignatureEmbeddings NULLs only mismatched; grandfathered + matching untouched', async () => { + await seedEmbedded('old', 'abcde', 'openai:old:1536'); // mismatched → invalidate + await seedEmbedded('legacy', 'fghij', null); // grandfathered → keep + await seedEmbedded('new', 'klmno', 'voyage:new:1024'); // matching → keep + + const invalidated = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' }); + expect(invalidated).toBe(1); // only 'old' + + // Now exactly the 'old' page's chunk is NULL → legacy stale count = 1. + expect(await engine.countStaleChunks()).toBe(1); + // Re-running is idempotent (nothing left to invalidate). + expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' })).toBe(0); + }); + + test('invalidate is sourceId-scoped', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{}'::jsonb) ON CONFLICT (id) DO NOTHING`, + ); + await seedEmbedded('a', 'abcde', 'openai:old:1536'); // default + await seedEmbedded('b', 'fghij', 'openai:old:1536', 'other'); // other + const n = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024', sourceId: 'default' }); + expect(n).toBe(1); + expect(await engine.countStaleChunks({ sourceId: 'default' })).toBe(1); + expect(await engine.countStaleChunks({ sourceId: 'other' })).toBe(0); // untouched + }); + + test('setPageEmbeddingSignature stamps the page', async () => { + await engine.putPage('p', { type: 'note', title: 'p', compiled_truth: '# p' }); + await engine.setPageEmbeddingSignature('p', { signature: 'openai:m:1536' }); + const rows = await engine.executeRaw<{ embedding_signature: string | null }>( + `SELECT embedding_signature FROM pages WHERE slug = 'p' AND source_id = 'default'`, + ); + expect(rows[0]?.embedding_signature).toBe('openai:m:1536'); + }); +}); diff --git a/test/import-signature-stamp.serial.test.ts b/test/import-signature-stamp.serial.test.ts new file mode 100644 index 000000000..4614f0fb4 --- /dev/null +++ b/test/import-signature-stamp.serial.test.ts @@ -0,0 +1,76 @@ +/** + * v0.41.31 (F1) — importFromContent stamps pages.embedding_signature when it + * embeds inline. + * + * The inline import/sync path (importFromContent) writes embeddings without + * going through embed.ts/embed-stale.ts. Before the F1 fix it never stamped + * the signature, so non-federated/inline brains kept NULL signatures forever + * → grandfathered → `embed --stale` never re-embedded them after a model swap + * (the headline feature silently inert). This pins the stamp. + * + * Serial: stubs the gateway embed transport (process-global module state). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { importFromContent } from '../src/core/import-file.ts'; +import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts'; +import { currentEmbeddingSignature } from '../src/core/embedding.ts'; + +let engine: PGLiteEngine; +let colDim: number; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`, + ); + colDim = Number(rows[0]?.dim); + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: colDim, + env: { OPENAI_API_KEY: 'sk-test-import-stamp' }, + }); + // Fake transport (AI SDK embedMany shape): receives { values }, returns + // one zero-vector (sized to the column) per input value. + __setEmbedTransportForTests(async ({ values }: { values: string[] }) => ({ + embeddings: values.map(() => Array(colDim).fill(0)), + usage: { tokens: 0 }, + }) as any); +}); + +afterEach(() => { + __setEmbedTransportForTests(null); + resetGateway(); +}); + +async function signatureOf(slug: string): Promise { + const rows = await engine.executeRaw<{ embedding_signature: string | null }>( + `SELECT embedding_signature FROM pages WHERE slug = $1 AND source_id = 'default'`, + [slug], + ); + return rows[0]?.embedding_signature ?? null; +} + +describe('importFromContent embedding_signature stamping (F1)', () => { + test('embeds inline → stamps the current signature', async () => { + await importFromContent(engine, 'concepts/stamped', '# Stamped\n\nsome body content to chunk and embed.', {}); + expect(await signatureOf('concepts/stamped')).toBe(currentEmbeddingSignature()); + }); + + test('--no-embed → leaves signature NULL (grandfathered, not stale)', async () => { + await importFromContent(engine, 'concepts/unstamped', '# Unstamped\n\nbody content.', { noEmbed: true }); + expect(await signatureOf('concepts/unstamped')).toBeNull(); + }); +}); diff --git a/test/schema-bootstrap-coverage.test.ts b/test/schema-bootstrap-coverage.test.ts index 64f761427..d81043f61 100644 --- a/test/schema-bootstrap-coverage.test.ts +++ b/test/schema-bootstrap-coverage.test.ts @@ -158,6 +158,10 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [ // pages_generation_idx (CREATE INDEX ON pages (generation)) so bootstrap // probes guard pre-v91 brains. { kind: 'column', table: 'pages', column: 'generation' }, + // v0.41.31 (v108) — pages.embedding_signature TEXT for real stale + // semantics. No SCHEMA_SQL index references it; bootstrap probe is + // defense-in-depth (and satisfies the MIGRATIONS ADD COLUMN coverage gate). + { kind: 'column', table: 'pages', column: 'embedding_signature' }, ]; test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => { diff --git a/test/source-health.test.ts b/test/source-health.test.ts index 5f4bc80b4..e59d237dd 100644 --- a/test/source-health.test.ts +++ b/test/source-health.test.ts @@ -167,6 +167,24 @@ describe('computeAllSourceMetrics', () => { expect(result.find((m) => m.source_id === 'other')!.total_pages).toBe(1); }); + test('v0.41.31: embed-backfill active/queued counts per source (CLI BACKFILL column)', async () => { + // 2 queued + 1 active embed-backfill for default; a non-backfill 'sync' + // job must NOT inflate the backfill counts (only the generic queue_depth). + await engine.executeRaw( + `INSERT INTO minion_jobs (name, status, data) VALUES + ('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb), + ('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb), + ('embed-backfill', 'active', '{"sourceId":"default"}'::jsonb), + ('sync', 'waiting', '{"sourceId":"default"}'::jsonb)`, + ); + const sources = await loadAllSources(engine); + const dflt = (await computeAllSourceMetrics(engine, sources)).find((m) => m.source_id === 'default')!; + expect(dflt.backfill_active).toBe(1); + expect(dflt.backfill_queued).toBe(2); + // generic queue_depth includes the sync job too (4 total waiting/active). + expect(dflt.queue_depth).toBe(4); + }); + test('webhook_configured reflects config.webhook_secret presence', async () => { await engine.executeRaw( `INSERT INTO sources (id, name, config) VALUES ('webhooky', 'webhooky', '{"federated":true,"webhook_secret":"x","github_repo":"a/b"}'::jsonb)`, diff --git a/test/sum-stale-chunk-chars.test.ts b/test/sum-stale-chunk-chars.test.ts new file mode 100644 index 000000000..399226752 --- /dev/null +++ b/test/sum-stale-chunk-chars.test.ts @@ -0,0 +1,113 @@ +/** + * v0.41.31 — BrainEngine.sumStaleChunkChars tests (PGLite). + * + * Sibling of countStaleChunks: sums LENGTH(chunk_text) over stale chunks so + * the `gbrain sync --all` cost preview can price the embedding backlog. + * Validates: empty brain → 0, exact char sum, sourceId scope, embed_skip + * exclusion, and that non-null (already-embedded) chunks are NOT counted. + * + * Uses the canonical PGLite block (one engine per file, resetPgliteState in + * beforeEach) per CLAUDE.md test-isolation rules R3+R4. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +/** Seed a page + stale (NULL-embedding) chunks with explicit chunk_text. */ +async function seedStale( + slug: string, + texts: string[], + opts?: { sourceId?: string; frontmatter?: Record }, +): Promise { + await engine.putPage( + slug, + { + type: 'note', + title: slug, + compiled_truth: `# ${slug}`, + ...(opts?.frontmatter ? { frontmatter: opts.frontmatter } : {}), + }, + opts?.sourceId ? { sourceId: opts.sourceId } : undefined, + ); + const chunks: ChunkInput[] = texts.map((t, i) => ({ + chunk_index: i, + chunk_text: t, + chunk_source: 'compiled_truth', + token_count: 4, + embedding: undefined, // NULL = stale + })); + await engine.upsertChunks(slug, chunks, opts?.sourceId ? { sourceId: opts.sourceId } : undefined); +} + +describe('sumStaleChunkChars', () => { + test('empty brain returns 0', async () => { + expect(await engine.sumStaleChunkChars()).toBe(0); + }); + + test('sums LENGTH(chunk_text) across stale chunks', async () => { + await seedStale('a', ['abcde', 'fghij']); // 5 + 5 + await seedStale('b', ['xyz']); // 3 + expect(await engine.sumStaleChunkChars()).toBe(13); + }); + + test('scopes to sourceId when provided', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}'::jsonb) ON CONFLICT (id) DO NOTHING`, + ); + await seedStale('a', ['abcde']); // default, 5 + await seedStale('b', ['xyzwv'], { sourceId: 'other' }); // other, 5 + expect(await engine.sumStaleChunkChars({ sourceId: 'default' })).toBe(5); + expect(await engine.sumStaleChunkChars({ sourceId: 'other' })).toBe(5); + expect(await engine.sumStaleChunkChars()).toBe(10); // all sources + }); + + test('excludes pages with embed_skip frontmatter', async () => { + await seedStale('keep', ['abcde']); // 5, counted + await seedStale('skip', ['1234567890'], { + frontmatter: { embed_skip: { at: '2026-01-01T00:00:00Z', reason: 'oversize' } }, + }); + expect(await engine.sumStaleChunkChars()).toBe(5); + }); + + test('does NOT count already-embedded (non-null) chunks', async () => { + // Stale page → counted. + await seedStale('stale', ['abcde']); // 5 + // "Embedded" page: seed as stale, then flip its chunk's embedding to a + // non-null vector via raw SQL. We bypass upsertChunks so the test doesn't + // depend on gateway-configured dimensions (keeps it robust regardless of + // any leaked gateway state from sibling files). + await seedStale('done', ['this is embedded and should not count']); + // Size the embedding to the ACTUAL column dimension (pgvector stores it + // in atttypmod) so the test is robust to whatever dim initSchema chose. + const dimRows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`, + ); + const dim = Number(dimRows[0]?.dim); + expect(dim).toBeGreaterThan(0); + await engine.executeRaw( + `UPDATE content_chunks + SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector + WHERE page_id = (SELECT id FROM pages WHERE slug = 'done' AND source_id = 'default')`, + [dim], + ); + expect(await engine.sumStaleChunkChars()).toBe(5); + }); +}); diff --git a/test/sync-cost-gate.serial.test.ts b/test/sync-cost-gate.serial.test.ts new file mode 100644 index 000000000..48d7b057d --- /dev/null +++ b/test/sync-cost-gate.serial.test.ts @@ -0,0 +1,160 @@ +/** + * v0.41.31 — `gbrain sync --all` cost-gate wiring regressions (PGLite). + * + * Pure shouldBlockSync / willEmbedSynchronously logic is pinned in + * test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in + * runSync's --all path: + * + * R-1 (headline): deferred-embed sync --all, non-TTY, with backlog → + * emits gate:'deferred_notice' and NEVER exit 2 (the cron-blocking + * bug this release fixes). + * R-2 (protection): inline-embed sync --all (--serial), non-TTY, above + * floor → still exit 2 with gate:'confirmation_required'. + * + * Serial-quarantined: stubs process.exit + console.log (process-global). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runSources } from '../src/commands/sources.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts'; +import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +let repoPath: string; +let headSha: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 60_000); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}, 60_000); + +beforeEach(async () => { + await resetPgliteState(engine); + // Configure the gateway with a dummy key so the pre-gate embedding-creds + // preflight passes (diagnoseEmbedding reads gateway configure-time state, + // not live env). The gate runs before any real embed call, so no network + // request is made. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test-costgate' }, + }); + repoPath = mkdtempSync(join(tmpdir(), 'gbrain-costgate-')); + execSync('git init', { cwd: repoPath, stdio: 'pipe' }); + execSync('git config user.email "t@t.com"', { cwd: repoPath, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repoPath, stdio: 'pipe' }); + mkdirSync(join(repoPath, 'topics'), { recursive: true }); + writeFileSync( + join(repoPath, 'topics/foo.md'), + ['---', 'type: concept', 'title: Foo', '---', '', 'some body content to estimate.'].join('\n'), + ); + execSync('git add -A && git commit -m initial', { cwd: repoPath, stdio: 'pipe' }); + headSha = execSync('git rev-parse HEAD', { cwd: repoPath, stdio: 'pipe' }).toString().trim(); +}); + +afterEach(() => { + resetGateway(); + if (repoPath) rmSync(repoPath, { recursive: true, force: true }); +}); + +/** Run runSync(args) with process.exit + console.log captured. */ +async function runSyncCaptured(args: string[]): Promise<{ exitCode: number | undefined; stdout: string }> { + const { runSync } = await import('../src/commands/sync.ts'); + const origExit = process.exit; + const origLog = console.log.bind(console); + const out: string[] = []; + let exitCode: number | undefined; + console.log = (...a: unknown[]) => { + out.push(a.map((x) => (typeof x === 'string' ? x : JSON.stringify(x))).join(' ')); + }; + process.exit = ((code?: number) => { + exitCode = code; + throw new Error('__exit__'); + }) as typeof process.exit; + try { + await runSync(engine, args); + } catch (e) { + if ((e as Error).message !== '__exit__') throw e; + } finally { + process.exit = origExit; + console.log = origLog; + } + return { exitCode, stdout: out.join('\n') }; +} + +describe('v0.41.31 — sync --all cost gate wiring', () => { + test('R-1: deferred sync --all (non-TTY) emits deferred_notice and never exit 2', async () => { + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + // Make the fan-out a clean no-op: last_commit == HEAD so performSync + // reports up_to_date (no git pull, no backfill submit). + await engine.executeRaw(`UPDATE sources SET last_commit = $1 WHERE id = 'vault'`, [headSha]); + // Seed a stale backlog so the deferred notice has a non-zero figure. + await engine.putPage('vault/note', { type: 'note', title: 'note', compiled_truth: '# note' }, { sourceId: 'vault' }); + const chunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: 'x'.repeat(500), chunk_source: 'compiled_truth', token_count: 4, embedding: undefined }, + ]; + await engine.upsertChunks('vault/note', chunks, { sourceId: 'vault' }); + + // v2 default ON → deferred. --json → agent path. NO --yes. --no-pull + // because the synthetic repo has no remote. + const { exitCode, stdout } = await runSyncCaptured(['--all', '--json', '--no-pull']); + + // The headline assertion: NOT blocked. + expect(exitCode).not.toBe(2); + expect(stdout).toContain('"gate":"deferred_notice"'); + }, 60_000); + + test('R-2: inline sync --all (--serial) above floor still exit 2', async () => { + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + // Floor 0 → any nonzero inline cost blocks. Source is unsynced + // (last_commit NULL) so estimateInlineNewTokens sees it as changed → + // full-tree tokens > 0 → costUsd > 0 > floor. + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + // --serial forces inline even with v2 on. --json → non-TTY exit-2 path. + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).toBe(2); + expect(stdout).toContain('"gate":"confirmation_required"'); + }, 60_000); + + test('R-3: inline, git-unchanged source but STALE chunker_version still estimates (not $0)', async () => { + // The unchanged-source short-circuit requires HEAD==last_commit AND clean + // tree AND chunker_version == current. Here git is unchanged but the + // chunker drifted, so the source must NOT be treated as 0 — sync would + // re-chunk + re-embed everything. floor=0 so any nonzero cost blocks. + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, 'STALE-0']); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).toBe(2); + expect(stdout).toContain('"gate":"confirmation_required"'); + }, 60_000); + + test('R-3 control: inline, git-unchanged + CURRENT chunker_version short-circuits to $0 (no exit 2)', async () => { + // Same setup but chunker_version matches current → the source IS unchanged + // → contributes 0 new-content tokens → below floor → proceeds (no block). + // Proves the short-circuit fires when (and only when) everything matches. + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).not.toBe(2); + expect(stdout).not.toContain('"gate":"confirmation_required"'); + }, 60_000); +}); diff --git a/test/sync-cost-preview.test.ts b/test/sync-cost-preview.test.ts index 22fb3408a..7f492c032 100644 --- a/test/sync-cost-preview.test.ts +++ b/test/sync-cost-preview.test.ts @@ -14,23 +14,54 @@ */ import { describe, test, expect } from 'bun:test'; -import { EMBEDDING_COST_PER_1K_TOKENS, estimateEmbeddingCostUsd } from '../src/core/embedding.ts'; +import { + EMBEDDING_COST_PER_1K_TOKENS, + estimateEmbeddingCostUsd, + willEmbedSynchronously, + shouldBlockSync, +} from '../src/core/embedding.ts'; +import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts'; import { estimateTokens } from '../src/core/chunkers/code.ts'; describe('Layer 8 D1 — embedding cost model', () => { - test('EMBEDDING_COST_PER_1K_TOKENS is text-embedding-3-large pricing', () => { - // Update this when OpenAI changes text-embedding-3-large pricing. - // As of 2026-04-24: $0.00013 / 1k tokens. + test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => { + // Retained only for back-compat imports. Live cost math now resolves the + // CONFIGURED model's rate via embedding-pricing.ts (see model-aware test + // below). As of 2026-04-24: $0.00013 / 1k tokens. expect(EMBEDDING_COST_PER_1K_TOKENS).toBe(0.00013); }); - test('estimateEmbeddingCostUsd scales linearly with tokens', () => { + test('estimateEmbeddingCostUsd scales linearly (gateway-unconfigured fallback = OpenAI rate)', () => { + // With no gateway configured (unit-test context) the estimator falls back + // to the OpenAI text-embedding-3-large rate ($0.13/Mtok = $0.00013/1k). expect(estimateEmbeddingCostUsd(0)).toBe(0); expect(estimateEmbeddingCostUsd(1000)).toBeCloseTo(0.00013, 5); expect(estimateEmbeddingCostUsd(10_000)).toBeCloseTo(0.0013, 4); expect(estimateEmbeddingCostUsd(1_000_000)).toBeCloseTo(0.13, 4); }); + test('cost preview uses the CONFIGURED model rate, not a hardcoded OpenAI rate', () => { + // Regression: the cost gate previously hardcoded $0.00013/1k (OpenAI + // text-embedding-3-large) regardless of the configured embedding model, + // so a brain on a cheaper model (e.g. zeroentropyai:zembed-1 @ $0.05/Mtok) + // saw a preview that named the wrong provider and over-stated spend ~2.6x. + // The pricing table is the single source of truth per provider:model. + const TOKENS = 2_590_710_262; // a real large-brain sync preview + const openai = lookupEmbeddingPrice('openai:text-embedding-3-large'); + const zeroentropy = lookupEmbeddingPrice('zeroentropyai:zembed-1'); + expect(openai.kind).toBe('known'); + expect(zeroentropy.kind).toBe('known'); + if (openai.kind === 'known' && zeroentropy.kind === 'known') { + const openaiCost = (TOKENS / 1_000_000) * openai.pricePerMTok; + const zeCost = (TOKENS / 1_000_000) * zeroentropy.pricePerMTok; + // The two models must produce materially different previews; a fix that + // collapses both to the OpenAI number would regress this assertion. + expect(openaiCost).toBeCloseTo(336.79, 1); + expect(zeCost).toBeCloseTo(129.54, 1); + expect(zeCost).toBeLessThan(openaiCost); + } + }); + test('5K-file TS repo sanity check: ~$5 at ~400k tokens', () => { // A 5K-file TS repo at ~80 tokens/file averages ~400k tokens. Cost: // 400_000 / 1000 * 0.00013 = $0.052 ≈ $0.05. Not $5. The CHANGELOG @@ -43,6 +74,50 @@ describe('Layer 8 D1 — embedding cost model', () => { }); }); +describe('v0.41.31 — willEmbedSynchronously (embed-mode resolver)', () => { + // Mirrors sync.ts:2346 effectiveNoEmbed = v2 && !serial && !noEmbed ? true : noEmbed. + // Embed runs INLINE iff that resolves to false. + test('v2 off → inline (legacy synchronous embed)', () => { + expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: false })).toBe('inline'); + }); + test('v2 on + parallel → deferred (backfill jobs)', () => { + expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: false, noEmbed: false })).toBe('deferred'); + }); + test('v2 on + --serial → inline', () => { + expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: false })).toBe('inline'); + }); + test('--no-embed forces deferred regardless of v2/serial', () => { + expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: true })).toBe('deferred'); + expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: true })).toBe('deferred'); + }); +}); + +describe('v0.41.31 — shouldBlockSync (cost-gate decision)', () => { + // R-1: deferred NEVER blocks, even at absurd cost (the headline fix — a + // nightly cron over a synced corpus must not exit 2). + test('R-1: deferred never blocks, even at $999', () => { + expect(shouldBlockSync(999, 0.5, 'deferred')).toBe(false); + expect(shouldBlockSync(0, 0.5, 'deferred')).toBe(false); + }); + // R-2: inline still blocks above the floor (protection preserved where + // sync actually spends synchronously). + test('R-2: inline blocks above floor', () => { + expect(shouldBlockSync(0.51, 0.5, 'inline')).toBe(true); + expect(shouldBlockSync(130, 0.5, 'inline')).toBe(true); + }); + test('inline at exactly the floor does NOT block (boundary)', () => { + expect(shouldBlockSync(0.5, 0.5, 'inline')).toBe(false); + }); + test('inline below floor does not block (kills cents-level cron noise)', () => { + expect(shouldBlockSync(0.03, 0.5, 'inline')).toBe(false); + expect(shouldBlockSync(0, 0.5, 'inline')).toBe(false); + }); + test('floor of 0 makes inline block on any nonzero cost', () => { + expect(shouldBlockSync(0.0001, 0, 'inline')).toBe(true); + expect(shouldBlockSync(0, 0, 'inline')).toBe(false); + }); +}); + describe('Layer 8 D1 — estimateTokens (exported from chunkers/code.ts)', () => { test('empty string is 0 tokens', () => { expect(estimateTokens('')).toBe(0);