diff --git a/CHANGELOG.md b/CHANGELOG.md index 96387dabf..06123372b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,94 @@ All notable changes to GBrain will be documented in this file. +## [0.36.3.0] - 2026-05-18 + +**Search now routes through any embedding column you've populated, not just OpenAI 1536. Voyage and ZeroEntropy columns become first-class search targets in one config flip.** + +Until this release, gbrain hardcoded `embedding` (OpenAI 1536) as the only column hybrid search could read from. If you'd backfilled `embedding_voyage` (1024d) or `embedding_zeroentropy` (halfvec 2560d) on the side, the index was paid for but unusable. Now `gbrain config set search_embedding_column embedding_voyage` flips your whole brain to Voyage in a single line. The query op also takes `embedding_column` per-call for A/B benchmarking. Adversarial reviews on both the planning and shipping paths caught fifteen real bugs that aren't shipping today — config-plane drift, cosine-rescore corruption when the descriptor's space doesn't match the cache, prototype-pollution via `constructor` as a column key, validation bypass on the descriptor passthrough, and a doctor false-warn on fresh brains. + +The numbers that matter: + +| Surface | Before | After | +|---|---|---| +| Search columns reachable | `embedding` only (1536d OpenAI hardcoded) | Any column in the `embedding_columns` registry — vector or halfvec, any dim ≤ 8192 | +| Per-call override | image-only via hidden internal flag | `gbrain query --json '{...,"embedding_column":"embedding_voyage"}'` | +| Cross-column cache contamination | possible if you ever flipped columns | `knobs_hash` v=3 makes `(col, prov)` part of the cache key — silent corruption impossible | +| `cosineReScore` rerank space | always pulled from `embedding` | pulls from the active column (D9 — Voyage HNSW now ranked against Voyage vectors, not OpenAI ones) | +| Doctor diagnostics | `embeddings` check only | new `embedding_column_registry` check: format_type dim drift, missing HNSW indexes, coverage <90% gate | +| Eval replay parity | results changed across column flips ("false regressions") | each `eval_candidates` row stores the column that ran; replay honors it | + +What this means: you can run a side-by-side provider eval today. Set `embedding_voyage` as the default for a session, run `gbrain query "..." --json` across your test queries, capture the results, flip back to `embedding`, replay the captured rows with `gbrain eval replay --against ...`, see the Jaccard@k drift between providers. The cache won't lie to you because it sits in a different keyspace for each provider. Doctor will yell if you accidentally point at a column that's only 50% populated. + +### Itemized changes + +#### Added + +- **`embedding_columns` config registry** — declare which content_chunks columns are searchable. JSON map keyed by column name, each entry has `{provider, dimensions, type}` where type ∈ `vector | halfvec`. Set via `gbrain config set embedding_columns '...JSON...'` (DB plane). Validation runs at config-set time: regex on keys, type/dim/provider field shapes. Bad config refuses to load with a paste-ready hint. +- **`search_embedding_column` config key** — picks the default column for hybridSearch. `gbrain config set search_embedding_column embedding_voyage` flips your whole brain. Coverage gate: refuses the switch when the target column is <90% populated unless you pass `--coverage-override`. +- **`embedding_column` MCP param on the `query` op** — per-call override for A/B benchmarking. Unknown column names throw a structured error with the list of registered names. The `search` op (keyword-only) deliberately does NOT accept the param — it would be silent UX (CDX-9). +- **`gbrain doctor` registry check** — `embedding_column_registry` probes each declared column via `format_type(atttypid, atttypmod)` so dim drift (declared 1024d, actual 1536d) surfaces with a paste-ready ALTER. Postgres also checks HNSW index presence; warns if missing. Coverage % on the active default column; warns when below 90% (skips the warn on empty brains). +- **`isCacheSafe(resolved, cfg)` helper** — the cache-skip decision compares full embedding space (name + dim + model) against cfg, not just the column name. A user who overrides the `embedding` builtin to point at Voyage doesn't accidentally keep using the OpenAI-sized cache. + +#### Changed + +- **`hybridSearch` resolves the column at the boundary** — engines now take a pre-validated `ResolvedColumn` descriptor, not a raw string. Engine code is config-free and unit-testable in isolation (D11 / CDX-5). +- **`gateway.embedQuery(text, { embeddingModel, dimensions })`** — query-side embed path accepts a model override. The resolved column's provider drives the embed call, so a query against `embedding_voyage` actually embeds via Voyage, not the global default (D10). +- **`gateway.isAvailable('embedding', modelOverride?)`** — the availability check honors the override. Hybrid skips vector search only when the column's provider is down, not the global default's (CDX-4). +- **`engine.getEmbeddingsByChunkIds(ids, column?)`** — `cosineReScore` hydrates embeddings from the active column. Without this, Voyage HNSW retrieval rescored against OpenAI vectors → NaN or wrong rankings (CDX-3 / D9). +- **`KNOBS_HASH_VERSION` bumped 2→3** — the cache key now folds in column + provider. Pre-v3 cache rows become unreachable on first re-query (one-time miss spike). Source change lives in `mode.ts`, not `query-cache.ts`. +- **`eval_candidates.embedding_column`** — schema migration v68. Per-row column metadata so `gbrain eval replay` reproduces the same column the capture ran against. NULL-tolerant; pre-v0.36 rows fall back to current default. + +#### Fixed (codex /ship findings) + +- **Prototype-pollution-safe registry (#1)** — registry uses `Object.create(null)` + `Object.hasOwn`. `gbrain config set search_embedding_column constructor` now correctly rejects rather than resolving to `Object.prototype.constructor`. +- **Descriptor passthrough re-validates (#2)** — internal SDK callers passing a hand-rolled `ResolvedColumn` get full validation (name regex, type ∈ {vector,halfvec}, dims in [1, 8192]). Eliminates the SQL-injection escape hatch through the descriptor field. +- **DB-plane config works without a file (#3)** — `loadConfigWithEngine` synthesizes a minimal base when `loadConfig()` returns null. Env-only Postgres installs can `gbrain config set search_embedding_column X` and the resolver actually sees it. +- **Cache skip is embedding-space-based (#4)** — replaced name-based `isDefaultColumn` with `isCacheSafe(resolved, cfg)` at the call site. User overrides of the `embedding` builtin no longer leak across vector spaces. +- **Doctor empty-brain UX (#5)** — coverage gate short-circuits when chunk count is 0. Fresh installs no longer see "Active column 'embedding' is 0.0% populated". + +#### Tests + +- New: `test/search/embedding-column.test.ts` (50 cases — resolver, registry, validation, prototype-pollution, descriptor passthrough, `isCacheSafe`). +- New: `test/gateway-embed-model-override.test.ts` (8 cases — model/dim override, `isAvailable` override). +- New: `test/cosine-rescore-column.test.ts` (4 PGLite cases — column-parameter hydration). +- New: `test/operations-embedding-column.test.ts` (6 cases — `query` accepts param, `search` rejects it). +- New: `test/e2e/embedding-column-pglite.test.ts` (9 cases — multi-col, halfvec, cosineReScore, unknown-name throw). +- New: `test/e2e/embedding-column-postgres.test.ts` (7 cases — real-pgvector halfvec, HNSW index visibility, format_type dim drift, coverage gate). +- New: `test/e2e/eval-replay-column.test.ts` (5 cases — column metadata persistence + replay honors). +- Extended: `test/search-mode.test.ts`, `test/search/knobs-hash-reranker.test.ts` — v=3 hash, col/prov fields, append-only convention. + +#### Plan reviewed + +`/plan-eng-review` + codex outside voice surfaced 16 findings during planning + shipping (D1-D16 in the plan + 5 codex /ship findings). Every one applied; the plan file is at `~/.claude/plans/system-instruction-you-are-working-sparkling-sun.md`. + +## To take advantage of v0.36.3.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns: + +1. Apply the schema migration: + ```bash + gbrain apply-migrations --yes + ``` +2. (Optional, only if you've backfilled extra columns via ALTER TABLE) declare them in the registry: + ```bash + gbrain config set embedding_columns '{ + "embedding_voyage": { "provider": "voyage:voyage-3-large", "dimensions": 1024, "type": "vector" }, + "embedding_zeroentropy": { "provider": "zeroentropyai:zembed-1", "dimensions": 2560, "type": "halfvec" } + }' + ``` +3. Verify with doctor: + ```bash + gbrain doctor --json | jq '.checks[] | select(.name == "embedding_column_registry")' + ``` +4. (Optional) flip the default for an A/B run: + ```bash + gbrain config set search_embedding_column embedding_voyage + gbrain query "test query" --json | jq '.results[0]' + gbrain config set search_embedding_column embedding # flip back + ``` + +If any step fails, file an issue at https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and `~/.gbrain/upgrade-errors.jsonl` if it exists. ## [0.36.2.0] - 2026-05-17 **ZeroEntropy is the new default. Faster, cheaper, better quality on real queries. Existing users get a one-shot switch prompt with cost estimate; new installs land on it out of the box. README rewritten to match what gbrain actually is in 2026.** diff --git a/CLAUDE.md b/CLAUDE.md index 6a52bcade..bcfc12216 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -92,15 +92,16 @@ strict behavior when unset. - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". - `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. - `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. -- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. +- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. - `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962. - `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline. -- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. +- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.36.3.0:** `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path can embed via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default. The hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}`; the gateway resolves the matching recipe and routes through its `instantiateEmbedding()` branch. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down — fixes the CDX-4 bug where a healthy Voyage column would skip vector retrieval because OpenAI happened to be unconfigured. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. - `src/core/ai/recipes/zeroentropyai.ts` (v0.35.0.0) — ZeroEntropy openai-compatible recipe declaring BOTH `embedding` (`zembed-1`, 7 Matryoshka dims: 2560/1280/640/320/160/80/40) AND `reranker` (`zerank-2` flagship + `zerank-1` + `zerank-1-small`, 5MB payload cap) touchpoints. `implementation: 'openai-compatible'` (NOT the misspelled `'openai-compat'` the original plan draft had — pinned by F1 regression in `test/ai/zeroentropy-recipe.test.ts`). `base_url_default: 'https://api.zeroentropy.dev/v1'` already ends with `/v1`, so the `zeroEntropyCompatFetch` URL rewrite `/embeddings → /models/embed` produces `…/v1/models/embed` (NOT `…/v1/v1/…` — pinned by F2 regression). `chars_per_token: 1` + `safety_factor: 0.5` match Voyage's dense-content hedge. - `src/core/rerank-audit.ts` (v0.35.0.0) — failure-only JSONL audit at `~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation, mirrors `src/core/audit-slug-fallback.ts`). Exports `logRerankFailure({reason, model, query_hash, doc_count, error_summary})` + `readRecentRerankFailures(days)`. **Deliberately no `logRerankSuccess`** (CDX2-F22 in plan review): writing once per tokenmax search is hot-path I/O churn AND success events leak query volume + timing into a local audit file. `gbrain doctor`'s `reranker_health` check reads `search.reranker.enabled` first so "no events in window" is interpreted correctly (disabled → ok; enabled → ok). Query text is SHA-256-prefix-hashed (8 hex chars) for privacy. `GBRAIN_AUDIT_DIR` env override honored via the shared `resolveAuditDir()`. +- `src/core/search/embedding-column.ts` (v0.36.3.0) — single source of truth for "which `content_chunks.*` column does this query rank against?" Pure functions, no engine I/O: `loadRegistry(cfg)` walks the `embedding_columns` config (DB plane, JSON map keyed by column name with `{provider, dimensions, type}` entries), seeds the OpenAI `embedding` builtin when unset, validates everything before it lands (column-name regex, type ∈ `vector | halfvec`, dims in [1, 8192], provider format) using `Object.create(null)` + `Object.hasOwn` so a key like `constructor` rejects instead of resolving to `Object.prototype.constructor`. `resolveColumn(registry, override?, cfg)` is the boundary call: returns a `ResolvedColumn` descriptor (frozen `{name, provider, dimensions, type}`) honoring per-call override → `search_embedding_column` config → `'embedding'` default. Throws `UnknownEmbeddingColumnError` with the list of registered names on miss. `isCacheSafe(resolved, cfg)` compares the full embedding SPACE (provider + dimensions + name) against cfg's default — a user who repointed the `embedding` builtin at Voyage doesn't accidentally serve OpenAI-shaped cache rows. `validateResolvedColumn(descriptor)` re-validates hand-rolled descriptors that bypass the registry (internal-SDK passthrough path) so the SQL-injection escape hatch through the descriptor field is closed. Consumed by `hybridSearch` (resolves once at the boundary, threads a `ResolvedColumn` into per-engine `searchVector` instead of a raw string), `gateway.embedQuery(text, {embeddingModel, dimensions})` (resolved column's provider drives the query-time embed call), `cosineReScore` (engine pulls vectors from the active column, not the hardcoded `embedding`), and the `query` MCP op (per-call `embedding_column` param). 538 lines of pure resolver, 511 unit cases in `test/search/embedding-column.test.ts` covering the 16 codex-flagged corners (prototype-pollution, descriptor passthrough, env-only Postgres install, empty-brain coverage gate, cache-space comparison). - `src/core/search/rerank.ts` (v0.35.0.0) — the call-site abstraction. `applyReranker(query, results, opts)` slots between `dedupResults()` and `enforceTokenBudget()` in `src/core/search/hybrid.ts`. Slices `opts.topNIn` (default 30) by current RRF order, sends to `gateway.rerank()`, reorders by `relevanceScore` desc, and appends the un-reranked tail unchanged (recall protection). **Fail-open on every `RerankError.reason`**: any error logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items so downstream telemetry sees the new ordering signal. `topNOut: null` is the explicit "don't truncate" signal — semantically distinct from `undefined` which means "fall through to mode bundle" (CDX2-F16). Test seam: `opts.rerankerFn` lets tests stub `gateway.rerank` without touching the network. - `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. **v0.33.1.1 (#962, fixup):** recipe docstring at `:7-16` tightened to name the seven hosted flexible-dim models that accept `output_dimension` explicitly (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and call out that `voyage-4-nano` is the open-weight variant listed separately by Voyage as fixed 1024-dim — does NOT accept the parameter. The "all v4 variants are flexible" misread is what caused the original PR to include nano in `VOYAGE_OUTPUT_DIMENSION_MODELS`; the negative regression assertion in `test/ai/gateway.test.ts` (`dimsProviderOptions` returns `undefined` for `voyage-4-nano`) pins the contract. - `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). **v0.31.12:** chat and expansion `models:` lists drop the v0.31.6 phantom `claude-sonnet-4-6-20250929` date suffix — canonical id is `claude-sonnet-4-6`. The wrong-direction alias `claude-sonnet-4-6 → claude-sonnet-4-6-20250929` is removed; a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize` set by v0.31.6 installs). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts` (6 cases, verbatim cherry-pick of PR #830 plus the reverse-alias rescue case). @@ -182,8 +183,8 @@ strict behavior when unset. - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. @@ -927,6 +928,13 @@ behavior as the production `query` op. tokenmax write (expansion=on, limit=50) can't be served to a conservative read. +**v0.36.3.0 knobs_hash v=2 → v=3.** The hash now folds the active +embedding column name + provider into the cache key, so a query routed +through `embedding_voyage` (1024d Voyage) can't be served a cache row +written against `embedding` (1536d OpenAI). Existing v=2 rows become +unreachable on first re-query (one-time miss spike on upgrade); +`mode.ts:KNOBS_HASH_VERSION` is the single source of truth. + **Three CLI surfaces:** gbrain search modes # what is running, with per-knob attribution diff --git a/VERSION b/VERSION index df1e611ac..ddcae1362 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.36.2.0 +0.36.3.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 5fd78400c..16bc0aedd 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -208,15 +208,16 @@ strict behavior when unset. - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". - `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. - `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. -- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. +- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. - `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962. - `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline. -- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. +- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.36.3.0:** `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path can embed via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default. The hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}`; the gateway resolves the matching recipe and routes through its `instantiateEmbedding()` branch. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down — fixes the CDX-4 bug where a healthy Voyage column would skip vector retrieval because OpenAI happened to be unconfigured. **v0.35.0.0:** ZeroEntropy support lands. New `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL from `/embeddings` to `/models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response from `{results: [{embedding}], usage: {total_bytes, total_tokens}}` to `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates (Voyage's shim hit the same `prompt_tokens` requirement at `:655`). Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via a new tagged `ZeroEntropyResponseTooLargeError` class (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name — class unification is a deferred cleanup). Wired in `instantiateEmbedding()` via the same `recipe.id === 'zeroentropyai'` branch pattern Voyage uses. New `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker model via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance score. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'` so callers can fail-open without an HTTP call. `_rerankTransport` test seam mirrors `_embedTransport`. New `gateway.embedQuery(text)` companion threads `inputType: 'query'` through `dimsProviderOptions()` (now 4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch added. `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model` through the same path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. **v0.34.1.0 (#875):** new `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries. The pre-existing Voyage `/multimodalembeddings` path is unchanged; the gateway selects by recipe `implementation` tag. Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions` config — no more cryptic `vector dimension mismatch` at INSERT time. Pinned by 11 cases in `test/openai-compat-multimodal.test.ts`. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line. - `src/core/ai/recipes/zeroentropyai.ts` (v0.35.0.0) — ZeroEntropy openai-compatible recipe declaring BOTH `embedding` (`zembed-1`, 7 Matryoshka dims: 2560/1280/640/320/160/80/40) AND `reranker` (`zerank-2` flagship + `zerank-1` + `zerank-1-small`, 5MB payload cap) touchpoints. `implementation: 'openai-compatible'` (NOT the misspelled `'openai-compat'` the original plan draft had — pinned by F1 regression in `test/ai/zeroentropy-recipe.test.ts`). `base_url_default: 'https://api.zeroentropy.dev/v1'` already ends with `/v1`, so the `zeroEntropyCompatFetch` URL rewrite `/embeddings → /models/embed` produces `…/v1/models/embed` (NOT `…/v1/v1/…` — pinned by F2 regression). `chars_per_token: 1` + `safety_factor: 0.5` match Voyage's dense-content hedge. - `src/core/rerank-audit.ts` (v0.35.0.0) — failure-only JSONL audit at `~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation, mirrors `src/core/audit-slug-fallback.ts`). Exports `logRerankFailure({reason, model, query_hash, doc_count, error_summary})` + `readRecentRerankFailures(days)`. **Deliberately no `logRerankSuccess`** (CDX2-F22 in plan review): writing once per tokenmax search is hot-path I/O churn AND success events leak query volume + timing into a local audit file. `gbrain doctor`'s `reranker_health` check reads `search.reranker.enabled` first so "no events in window" is interpreted correctly (disabled → ok; enabled → ok). Query text is SHA-256-prefix-hashed (8 hex chars) for privacy. `GBRAIN_AUDIT_DIR` env override honored via the shared `resolveAuditDir()`. +- `src/core/search/embedding-column.ts` (v0.36.3.0) — single source of truth for "which `content_chunks.*` column does this query rank against?" Pure functions, no engine I/O: `loadRegistry(cfg)` walks the `embedding_columns` config (DB plane, JSON map keyed by column name with `{provider, dimensions, type}` entries), seeds the OpenAI `embedding` builtin when unset, validates everything before it lands (column-name regex, type ∈ `vector | halfvec`, dims in [1, 8192], provider format) using `Object.create(null)` + `Object.hasOwn` so a key like `constructor` rejects instead of resolving to `Object.prototype.constructor`. `resolveColumn(registry, override?, cfg)` is the boundary call: returns a `ResolvedColumn` descriptor (frozen `{name, provider, dimensions, type}`) honoring per-call override → `search_embedding_column` config → `'embedding'` default. Throws `UnknownEmbeddingColumnError` with the list of registered names on miss. `isCacheSafe(resolved, cfg)` compares the full embedding SPACE (provider + dimensions + name) against cfg's default — a user who repointed the `embedding` builtin at Voyage doesn't accidentally serve OpenAI-shaped cache rows. `validateResolvedColumn(descriptor)` re-validates hand-rolled descriptors that bypass the registry (internal-SDK passthrough path) so the SQL-injection escape hatch through the descriptor field is closed. Consumed by `hybridSearch` (resolves once at the boundary, threads a `ResolvedColumn` into per-engine `searchVector` instead of a raw string), `gateway.embedQuery(text, {embeddingModel, dimensions})` (resolved column's provider drives the query-time embed call), `cosineReScore` (engine pulls vectors from the active column, not the hardcoded `embedding`), and the `query` MCP op (per-call `embedding_column` param). 538 lines of pure resolver, 511 unit cases in `test/search/embedding-column.test.ts` covering the 16 codex-flagged corners (prototype-pollution, descriptor passthrough, env-only Postgres install, empty-brain coverage gate, cache-space comparison). - `src/core/search/rerank.ts` (v0.35.0.0) — the call-site abstraction. `applyReranker(query, results, opts)` slots between `dedupResults()` and `enforceTokenBudget()` in `src/core/search/hybrid.ts`. Slices `opts.topNIn` (default 30) by current RRF order, sends to `gateway.rerank()`, reorders by `relevanceScore` desc, and appends the un-reranked tail unchanged (recall protection). **Fail-open on every `RerankError.reason`**: any error logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items so downstream telemetry sees the new ordering signal. `topNOut: null` is the explicit "don't truncate" signal — semantically distinct from `undefined` which means "fall through to mode bundle" (CDX2-F16). Test seam: `opts.rerankerFn` lets tests stub `gateway.rerank` without touching the network. - `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. **v0.33.1.1 (#962, fixup):** recipe docstring at `:7-16` tightened to name the seven hosted flexible-dim models that accept `output_dimension` explicitly (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and call out that `voyage-4-nano` is the open-weight variant listed separately by Voyage as fixed 1024-dim — does NOT accept the parameter. The "all v4 variants are flexible" misread is what caused the original PR to include nano in `VOYAGE_OUTPUT_DIMENSION_MODELS`; the negative regression assertion in `test/ai/gateway.test.ts` (`dimsProviderOptions` returns `undefined` for `voyage-4-nano`) pins the contract. - `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). **v0.31.12:** chat and expansion `models:` lists drop the v0.31.6 phantom `claude-sonnet-4-6-20250929` date suffix — canonical id is `claude-sonnet-4-6`. The wrong-direction alias `claude-sonnet-4-6 → claude-sonnet-4-6-20250929` is removed; a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize` set by v0.31.6 installs). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts` (6 cases, verbatim cherry-pick of PR #830 plus the reverse-alias rescue case). @@ -298,8 +299,8 @@ strict behavior when unset. - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. @@ -1043,6 +1044,13 @@ behavior as the production `query` op. tokenmax write (expansion=on, limit=50) can't be served to a conservative read. +**v0.36.3.0 knobs_hash v=2 → v=3.** The hash now folds the active +embedding column name + provider into the cache key, so a query routed +through `embedding_voyage` (1024d Voyage) can't be served a cache row +written against `embedding` (1536d OpenAI). Existing v=2 rows become +unreachable on first re-query (one-time miss spike on upgrade); +`mode.ts:KNOBS_HASH_VERSION` is the single source of truth. + **Three CLI surfaces:** gbrain search modes # what is running, with per-knob attribution diff --git a/package.json b/package.json index 85630b8c4..d6f2d1dc8 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.36.2.0", + "version": "0.36.3.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/commands/config.ts b/src/commands/config.ts index 21350eb89..08db8c637 100644 --- a/src/commands/config.ts +++ b/src/commands/config.ts @@ -1,5 +1,13 @@ import type { BrainEngine } from '../core/engine.ts'; -import { loadConfig } from '../core/config.ts'; +import { loadConfig, loadConfigWithEngine } from '../core/config.ts'; +import { + getEmbeddingColumnRegistry, + validateColumnKey, + validateColumnConfig, + quoteIdentifier, + EmbeddingColumnNotRegisteredError, + EmbeddingColumnConfigError, +} from '../core/search/embedding-column.ts'; function redactUrl(url: string): string { // Redact password in postgresql:// URLs @@ -98,6 +106,112 @@ export async function runConfig(engine: BrainEngine, args: string[]) { process.exit(1); } } else if (action === 'set' && key && value) { + // v0.36 (D12 + D14): validate embedding-column keys at set time so a + // bad config gets rejected loud + early. The `--coverage-override` + // flag lets the user proceed past the < 90% gate when they know + // they're mid-backfill. + const coverageOverride = + args.includes('--coverage-override') || args.includes('--yes'); + + if (key === 'embedding_columns') { + try { + const parsed = JSON.parse(value); + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + throw new Error('embedding_columns must be a JSON object'); + } + // D12: validate every key + entry shape before persisting. + for (const [k, entry] of Object.entries(parsed)) { + validateColumnKey(k); + validateColumnConfig(k, entry); + } + } catch (err) { + if (err instanceof EmbeddingColumnConfigError) { + console.error(`[config] ${err.message}`); + } else { + console.error( + `[config] embedding_columns rejected: ${(err as Error).message}`, + ); + console.error( + `[config] Expected JSON shape: {"": {"provider": "...", "dimensions": N, "type": "vector" | "halfvec"}, ...}`, + ); + } + process.exit(1); + } + } + + if (key === 'search_embedding_column') { + // Validate against the merged registry (file + DB plane + builtins). + // We re-read merged config so a prior `gbrain config set + // embedding_columns ...` is visible. + const fileCfg = loadConfig(); + const mergedCfg = fileCfg + ? await loadConfigWithEngine(engine, fileCfg).catch(() => fileCfg) + : null; + if (mergedCfg) { + let registry: ReturnType; + try { + registry = getEmbeddingColumnRegistry(mergedCfg); + } catch (err) { + console.error( + `[config] Existing embedding_columns is invalid; refusing to set search_embedding_column. ` + + `Fix the registry first. (${(err as Error).message})`, + ); + process.exit(1); + } + // Object.hasOwn so inherited keys ('constructor', 'toString', etc.) + // cannot pass the registry-lookup gate. + if (!Object.hasOwn(registry, value)) { + const known = Object.keys(registry).sort().join(', ') || '(none)'; + console.error( + `[config] Unknown embedding column "${value}". ` + + `Declared columns: ${known}. ` + + `Add it via: gbrain config set embedding_columns ''`, + ); + process.exit(1); + } + + // D14 coverage gate. Probe the column's NULL-rate; refuse when + // coverage < 90% unless `--coverage-override` or `--yes` is + // present. + try { + const covRows = await engine.executeRaw<{ pct: number; total: number }>( + `SELECT ( + COUNT(*) FILTER (WHERE ${quoteIdentifier(value)} IS NOT NULL)::float + / NULLIF(COUNT(*), 0) * 100 + )::float AS pct, + COUNT(*)::int AS total + FROM content_chunks`, + ); + const pct = covRows[0]?.pct ?? 0; + const total = covRows[0]?.total ?? 0; + if (total > 0 && pct < 90 && !coverageOverride) { + console.error( + `[config] Column "${value}" is ${pct.toFixed(1)}% populated (${total} total chunks).`, + ); + console.error( + `[config] Switching the default to a low-coverage column silently degrades search.`, + ); + console.error( + `[config] Re-run with --coverage-override (or --yes) to proceed anyway:`, + ); + console.error( + `[config] gbrain config set search_embedding_column ${value} --coverage-override`, + ); + process.exit(1); + } + } catch (err) { + // Coverage probe failure shouldn't block when the column shape + // is otherwise valid (e.g. the column was JUST added, no chunks + // yet, NULLIF guard returns NULL → pct=0 BUT total=0 short- + // circuits above). If the SQL itself errors (column ALTER race, + // permission), warn but proceed. + console.error( + `[config] WARN: coverage probe failed (${(err as Error).message}); proceeding.`, + ); + } + } + } + await engine.setConfig(key, value); // v0.36.x #892: redact sensitive values in confirmation output. API // keys / tokens / passwords are commonly set from terminals with diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index fc51410d6..1ad5b2dce 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -2115,6 +2115,164 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo } } catch { /* listRecipes / gateway not available — silent */ } + // 8c. Embedding column registry (v0.36 — D5 + D13 + D14). + // Validates every column in the merged registry against the real DB + // shape: (a) column exists, (b) declared type+dims match actual + // format_type(atttypid, atttypmod), (c) HNSW index present on + // Postgres, (d) the ACTIVE default column has >= 90% coverage. + // + // Batch probes (D5) so the registry can grow without N+1 round-trips: + // one format_type query, one pg_indexes query, one coverage-per-active + // column query. + progress.heartbeat('embedding_column_registry'); + try { + const { getEmbeddingColumnRegistry, resolveEmbeddingColumn, quoteIdentifier } = + await import('../core/search/embedding-column.ts'); + const { loadConfig: _loadConfig } = await import('../core/config.ts'); + const fileCfg = _loadConfig(); + const mergedCfg = fileCfg ? await (await import('../core/config.ts')).loadConfigWithEngine(engine, fileCfg).catch(() => fileCfg) : null; + if (!mergedCfg) { + checks.push({ + name: 'embedding_column_registry', + status: 'ok', + message: 'No brain config loaded — skipped', + }); + } else { + const registry = getEmbeddingColumnRegistry(mergedCfg); + const declaredColumns = Object.keys(registry); + const activeCol = resolveEmbeddingColumn(undefined, mergedCfg).name; + + // D13 — batch format_type probe via pg_attribute. udt_name only + // returns 'vector' vs 'halfvec'; format_type(atttypid, atttypmod) + // returns 'vector(1024)' / 'halfvec(2560)' so dim drift surfaces. + const formatRows = await engine.executeRaw<{ attname: string; formatted: string }>( + `SELECT a.attname, format_type(a.atttypid, a.atttypmod) AS formatted + FROM pg_attribute a + JOIN pg_class c ON c.oid = a.attrelid + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'public' + AND c.relname = 'content_chunks' + AND a.attname = ANY($1::text[]) + AND NOT a.attisdropped`, + [declaredColumns], + ); + const actualByName = new Map(); + for (const r of formatRows) actualByName.set(r.attname, r.formatted); + + // D5 — batch index probe (Postgres only; PGLite indexing is implicit + // and the partial-index pattern doesn't surface in pg_indexes the + // same way). Reports informational, not blocking — search still + // works without an HNSW index, just slow. + const haveIndex = new Map(); + if (engine.kind === 'postgres') { + const indexRows = await engine.executeRaw<{ indexdef: string }>( + `SELECT indexdef FROM pg_indexes + WHERE tablename = 'content_chunks' + AND schemaname = 'public'`, + ); + for (const col of declaredColumns) { + const found = indexRows.some(r => /USING\s+hnsw/i.test(r.indexdef) && r.indexdef.includes(`(${col} `)); + haveIndex.set(col, found); + } + } + + // Per-column health rollup. + const issues: string[] = []; + const okColumns: string[] = []; + for (const colName of declaredColumns) { + const entry = registry[colName]; + const actual = actualByName.get(colName); + if (!actual) { + issues.push(`${colName}: declared but column does NOT exist in content_chunks`); + continue; + } + // Expected format: `vector(N)` or `halfvec(N)`. + const m = actual.match(/^(vector|halfvec)\((\d+)\)/i); + const actualType = m ? m[1].toLowerCase() : actual; + const actualDims = m ? parseInt(m[2], 10) : null; + if (actualType !== entry.type) { + issues.push( + `${colName}: declared type=${entry.type} but actual is ${actual}. ` + + `Fix: gbrain config set embedding_columns '' OR ` + + `ALTER TABLE content_chunks ALTER COLUMN ${colName} TYPE ${entry.type}(${entry.dimensions});`, + ); + continue; + } + if (actualDims !== null && actualDims !== entry.dimensions) { + issues.push( + `${colName}: declared dims=${entry.dimensions} but actual is ${actual}. ` + + `Fix one side: update config OR ` + + `ALTER TABLE content_chunks ALTER COLUMN ${colName} TYPE ${entry.type}(${entry.dimensions});`, + ); + continue; + } + if (engine.kind === 'postgres' && haveIndex.get(colName) === false) { + issues.push( + `${colName}: no HNSW index. Search works but uses sequential scan. ` + + `Fix: CREATE INDEX IF NOT EXISTS idx_chunks_${colName} ON content_chunks USING hnsw (${quoteIdentifier(colName)} ${entry.type}_cosine_ops);`, + ); + continue; + } + okColumns.push(colName); + } + + // D14 — coverage gate on the ACTIVE default column. Catches the + // "user switched to a 5%-populated column" silent-degradation case. + let coverageWarn: string | null = null; + if (activeCol && actualByName.has(activeCol)) { + // Codex /ship #5: pull `total` alongside `pct` so a fresh brain + // (0 chunks → NULLIF makes pct NULL → coalesces to 0) doesn't + // false-warn "Active column 'embedding' is 0.0% populated". + const covRows = await engine.executeRaw<{ pct: number; total: number }>( + `SELECT ( + COUNT(*) FILTER (WHERE ${quoteIdentifier(activeCol)} IS NOT NULL)::float + / NULLIF(COUNT(*), 0) * 100 + )::float AS pct, + COUNT(*)::int AS total + FROM content_chunks`, + ); + const pct = covRows[0]?.pct ?? 0; + const total = covRows[0]?.total ?? 0; + // Only warn when there's a real coverage gap. Empty brain (0 chunks) + // is a normal state for new installs — skip the gate entirely. + if (total > 0 && pct < 90) { + coverageWarn = + `Active column '${activeCol}' is ${pct.toFixed(1)}% populated. ` + + `Search quality silently degraded on un-embedded chunks. ` + + `Fix: gbrain embed --column ${activeCol} --stale (write-side support v2) ` + + `OR gbrain config set search_embedding_column embedding`; + } + } + + if (issues.length === 0 && !coverageWarn) { + const indexNote = engine.kind === 'postgres' ? ' (all indexed)' : ''; + checks.push({ + name: 'embedding_column_registry', + status: 'ok', + message: `Registry healthy: ${okColumns.length} columns (${okColumns.join(', ')})${indexNote}; active='${activeCol}'`, + }); + } else { + const allMessages = [ + ...issues, + ...(coverageWarn ? [coverageWarn] : []), + ]; + checks.push({ + name: 'embedding_column_registry', + status: 'warn', + message: allMessages.join(' | '), + }); + } + } + } catch (err) { + // Pre-config brains, registry-validation throws, etc. Surfaces the + // error message but doesn't fail the doctor run. + checks.push({ + name: 'embedding_column_registry', + status: 'warn', + message: `Could not check embedding column registry: ${(err as Error).message}`, + }); + } + // 9. Graph health (link + timeline coverage on entity pages). // dead_links removed in v0.10.1: ON DELETE CASCADE on link FKs makes it always 0. // diff --git a/src/commands/eval-replay.ts b/src/commands/eval-replay.ts index deea34e9c..f47bb6313 100644 --- a/src/commands/eval-replay.ts +++ b/src/commands/eval-replay.ts @@ -177,6 +177,12 @@ interface CapturedRow { job_id?: number | null; subagent_id?: number | null; created_at?: string; + /** + * v0.36 (D16 / CDX-10): the embedding column that ran at capture time. + * Optional for back-compat — pre-v0.36 exports won't have it. NULL or + * missing means "use the current default." + */ + embedding_column?: string | null; } /** @@ -246,6 +252,10 @@ async function replayRow(engine: BrainEngine, row: CapturedRow, opts: ReplayOpts limit, detail: row.detail ?? undefined, expansion: row.expand_enabled ?? false, + // v0.36 (D16 / CDX-10): replay the SAME column that ran at capture + // time so config drift between capture and replay doesn't surface + // as "regression." NULL/undefined falls through to resolver default. + embeddingColumn: row.embedding_column ?? undefined, }); } } catch (err) { diff --git a/src/core/ai/gateway.ts b/src/core/ai/gateway.ts index 38d43a99a..10d1a0f32 100644 --- a/src/core/ai/gateway.ts +++ b/src/core/ai/gateway.ts @@ -547,8 +547,15 @@ export function getRerankerModel(): string | undefined { /** * Check whether a touchpoint can be served given the current config. * Replaces scattered `!process.env.OPENAI_API_KEY` checks (Codex C3). + * + * v0.36 (D10): optional `modelOverride` to check a specific + * `provider:model` instead of the globally configured default for the + * touchpoint. Used by hybridSearch to ask "is the active column's + * provider reachable?" rather than "is the global default reachable?" — + * otherwise an unreachable global default disables vector search even + * when the active column's provider works fine. */ -export function isAvailable(touchpoint: TouchpointKind): boolean { +export function isAvailable(touchpoint: TouchpointKind, modelOverride?: string): boolean { // Test seam: when a transport stub is installed for this touchpoint, the // gateway is "available" for tests that exercise the whole pipeline without // configuring real providers. See __setChatTransportForTests / @@ -558,7 +565,9 @@ export function isAvailable(touchpoint: TouchpointKind): boolean { if (!_config) return false; try { const modelStr = - touchpoint === 'embedding' + modelOverride + ? modelOverride + : touchpoint === 'embedding' ? getEmbeddingModel() : touchpoint === 'expansion' ? getExpansionModel() @@ -1050,21 +1059,49 @@ export interface EmbedOpts { * resolver — the correct default for indexing paths). */ inputType?: 'query' | 'document'; + /** + * v0.36 (D10): explicit model override. When set, routes through this + * provider:model instead of the globally configured embedding_model. + * Used by the dynamic-embedding-column path so a single query can + * embed via the provider that matches the active column. NULL/absent + * preserves the existing global-default behavior. + * + * Format: 'provider:model' (e.g. 'voyage:voyage-3-large'). + */ + embeddingModel?: string; + /** + * v0.36 (D10): explicit dimensions override, paired with + * embeddingModel. When set, threads into `dimsProviderOptions` so the + * gateway sends the right `dimensions` / `output_dimension` to the + * provider. Must match the dim of the destination column or pgvector + * rejects the insert/search. NULL preserves the global-default. + */ + dimensions?: number; } export async function embed(texts: string[], opts?: EmbedOpts): Promise { if (!texts || texts.length === 0) return []; const cfg = requireConfig(); - const { model, recipe, modelId } = await resolveEmbeddingProvider(getEmbeddingModel()); + // v0.36 (D10): caller may override the model. Used by the dynamic-embedding- + // column path so hybridSearch can embed via the column's provider, not the + // global default. resolveEmbeddingProvider validates the override at the + // recipe layer — bad model strings throw AIConfigError with a clear hint. + const resolveTarget = opts?.embeddingModel ?? getEmbeddingModel(); + const { model, recipe, modelId } = await resolveEmbeddingProvider(resolveTarget); const truncated = texts.map(t => (t ?? '').slice(0, MAX_CHARS)); + // Dim override (D10) — when caller passes `dimensions`, use it. Otherwise + // fall back to the global cfg default. dimsProviderOptions throws a + // clear AIConfigError when a Voyage flexible-dim model gets an + // unsupported value (the existing v0.33.1.1 fail-loud path). + const effectiveDims = opts?.dimensions ?? cfg.embedding_dimensions ?? DEFAULT_EMBEDDING_DIMENSIONS; const providerOpts = dimsProviderOptions( recipe.implementation, modelId, - cfg.embedding_dimensions ?? DEFAULT_EMBEDDING_DIMENSIONS, + effectiveDims, opts?.inputType, ); - const expected = cfg.embedding_dimensions ?? DEFAULT_EMBEDDING_DIMENSIONS; + const expected = effectiveDims; const embedding = recipe.touchpoints?.embedding; const maxBatchTokens = embedding?.max_batch_tokens; @@ -1265,8 +1302,15 @@ export async function embedOne(text: string): Promise { * * Returns a single Float32Array (not a batch). */ -export async function embedQuery(text: string): Promise { - const [v] = await embed([text], { inputType: 'query' }); +export async function embedQuery( + text: string, + opts?: { embeddingModel?: string; dimensions?: number }, +): Promise { + const [v] = await embed([text], { + inputType: 'query', + embeddingModel: opts?.embeddingModel, + dimensions: opts?.dimensions, + }); return v; } diff --git a/src/core/config.ts b/src/core/config.ts index eabcdcf0c..d78998cbd 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -1,7 +1,7 @@ import { readFileSync, writeFileSync, mkdirSync, chmodSync, existsSync } from 'fs'; import { isAbsolute, join } from 'path'; import { homedir } from 'os'; -import type { EngineConfig } from './types.ts'; +import type { EngineConfig, EmbeddingColumnConfig } from './types.ts'; /** * Where is the active DB URL coming from? Pure introspection, no connection @@ -84,6 +84,27 @@ export interface GBrainConfig { embedding_image_ocr?: boolean; embedding_image_ocr_model?: string; + /** + * v0.36 — embedding-column registry (D7). Maps a content_chunks column + * name to its provider + dimensions + pgvector type. Both keys live in + * the DB plane (`gbrain config set ...`) so users can flip without + * editing files. Resolver merges this with `BUILTIN_EMBEDDING_COLUMNS` + * (which derive their provider from `embedding_model` / + * `embedding_multimodal_model`). + * + * Validation lives in `src/core/search/embedding-column.ts` per D12 — + * keys must match `/^[a-z_][a-z0-9_]*$/`, type in {vector, halfvec}, + * dimensions 1..8192, provider parseable as `provider:model`. + */ + embedding_columns?: Record; + /** + * v0.36 — name of the column hybridSearch uses by default. Per-call + * `SearchOpts.embeddingColumn` overrides this; absent => 'embedding'. + * Validated against the merged `embedding_columns` registry at config- + * set time and on hybridSearch entry. + */ + search_embedding_column?: string; + /** * Thin-client mode (multi-topology v1). When set, this install does NOT * have a local DB; it talks to a remote `gbrain serve --http` over MCP. @@ -219,8 +240,18 @@ export async function loadConfigWithEngine( engine: { getConfig(key: string): Promise }, base?: GBrainConfig | null, ): Promise { - const fileConfig = base !== undefined ? base : loadConfig(); - if (!fileConfig) return null; + // Codex /ship finding #3: when there's no file config AND no env DB URL, + // loadConfig() returns null and the DB merge would be skipped — env-only + // installs (engine wired via direct SDK pass) wouldn't see DB-plane + // overrides like `embedding_columns` / `search_embedding_column` set via + // `gbrain config set`. Since we have a live engine here, synthesize a + // minimal base config so the DB-plane merge still runs. The synthesized + // config has no auth or model fields; DB-plane keys overlay correctly + // and downstream callers either find them or fall through to defaults. + // Also applies when callers pass an explicit null for `base`. + const fileConfig: GBrainConfig = + (base !== undefined ? base : loadConfig()) ?? + ({ engine: 'postgres' } as GBrainConfig); // DB-plane reads. Quiet failures — if the config table doesn't exist yet // (pre-v36 brain mid-migration), treat as null and let file/env defaults @@ -248,6 +279,12 @@ export async function loadConfigWithEngine( const dbMultimodalModel = await dbStr('embedding_multimodal_model'); const dbOcr = await dbBool('embedding_image_ocr'); const dbOcrModel = await dbStr('embedding_image_ocr_model'); + // v0.36 (D7) — embedding-column registry merge. Stored as JSON string in + // the config table. Parse + shape-check here; full registry validation + // (regex on keys, type/dim/provider field shapes) runs in the resolver at + // first use so a malformed DB row doesn't kill engine connect. + const dbEmbeddingColumns = await dbStr('embedding_columns'); + const dbSearchEmbeddingColumn = await dbStr('search_embedding_column'); // DB applies only when env did NOT win. Env presence is detected by the // sync loadConfig() already setting the field. For each flag, prefer the @@ -265,6 +302,21 @@ export async function loadConfigWithEngine( if (merged.embedding_image_ocr_model === undefined && dbOcrModel !== undefined) { merged.embedding_image_ocr_model = dbOcrModel; } + if (merged.embedding_columns === undefined && dbEmbeddingColumns !== undefined) { + try { + const parsed = JSON.parse(dbEmbeddingColumns); + if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) { + merged.embedding_columns = parsed as Record; + } else { + console.warn('[gbrain] config: embedding_columns DB value is not a JSON object; ignoring'); + } + } catch (err) { + console.warn(`[gbrain] config: embedding_columns DB value is not valid JSON; ignoring (${(err as Error).message})`); + } + } + if (merged.search_embedding_column === undefined && dbSearchEmbeddingColumn !== undefined) { + merged.search_embedding_column = dbSearchEmbeddingColumn; + } return merged; } diff --git a/src/core/embedding.ts b/src/core/embedding.ts index 16f744d50..e7e4895e1 100644 --- a/src/core/embedding.ts +++ b/src/core/embedding.ts @@ -30,9 +30,17 @@ export async function embed(text: string): Promise { * embed seam so the provider returns query-side vectors. For symmetric * providers (OpenAI text-3, DashScope, Zhipu) the field is dropped — no * behavior change. Used by hybrid.ts on the search hot path. + * + * v0.36 (D10): optional `embeddingModel` + `dimensions` overrides so the + * dynamic-embedding-column path can embed via the column's provider rather + * than the globally-configured default. Bare `embedQuery(text)` preserves + * pre-v0.36 behavior. */ -export async function embedQuery(text: string): Promise { - return gatewayEmbedQuery(text); +export async function embedQuery( + text: string, + opts?: { embeddingModel?: string; dimensions?: number }, +): Promise { + return gatewayEmbedQuery(text, opts); } export interface EmbedBatchOptions { diff --git a/src/core/engine.ts b/src/core/engine.ts index 34001a401..c2132be73 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -578,7 +578,20 @@ export interface BrainEngine { // Search searchKeyword(query: string, opts?: SearchOpts): Promise; searchVector(embedding: Float32Array, opts?: SearchOpts): Promise; - getEmbeddingsByChunkIds(ids: number[]): Promise>; + /** + * Hydrate embeddings for chunks already known by id. v0.36 (D9): + * optional `column` parameter selects which content_chunks column to + * fetch from (default 'embedding'). The dynamic-embedding-column + * search path hands its resolved column name here so cosineReScore + * rehydrates in the right embedding space — otherwise vector search + * against `embedding_voyage` would HNSW-rank against Voyage but + * rescore against OpenAI vectors (NaN / wrong rankings). + * + * The column name MUST be regex-validated by the caller (resolveEmbed- + * dingColumn rejects bad names). Engines identifier-quote on + * interpolation as defense in depth (D12). + */ + getEmbeddingsByChunkIds(ids: number[], column?: string): Promise>; // Chunks /** diff --git a/src/core/eval-capture.ts b/src/core/eval-capture.ts index ff8807672..9983e892b 100644 --- a/src/core/eval-capture.ts +++ b/src/core/eval-capture.ts @@ -116,6 +116,10 @@ export function buildEvalCandidateInput( remote: ctx.remote, job_id: ctx.job_id, subagent_id: ctx.subagent_id, + // v0.36 (D16 / CDX-10): persist the column that ran. Null for + // keyword-only `search` (meta.embedding_column omitted), preserving + // back-compat with rows captured before the column tracking landed. + embedding_column: ctx.meta.embedding_column ?? null, }; } diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 7d32e6612..83ca96fdd 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -3513,6 +3513,39 @@ export const MIGRATIONS: Migration[] = [ ON think_ab_results (source_id, ran_at DESC); `, }, + { + version: 74, + name: 'eval_candidates_embedding_column', + // v0.36.3.0 (D16 / CDX-10): persist the resolved embedding column on + // each eval_candidates row so replay against a captured query uses + // the column that was active at capture time — not whichever column + // is current local default. Without this, switching + // `search_embedding_column` between capture and replay produces + // false-positive "regressions" that are just column changes. + // + // Nullable for back-compat: pre-v0.36 rows have NULL; replay treats + // NULL as "use current default" so existing captures keep working + // exactly as before the migration. + // + // Renumbered v68→v74 during the second master merge: master's + // v0.36.1.0 calibration wave claimed v68-v73 first. The ALTER + // itself is unchanged; only the slot number moved. The column is + // also in PGLITE_SCHEMA_SQL / src/schema.sql so fresh installs get + // it natively without running this migration. + idempotent: true, + sql: ` + ALTER TABLE eval_candidates + ADD COLUMN IF NOT EXISTS embedding_column TEXT; + `, + // PGLite parity: same ALTER, same IF NOT EXISTS guard makes this a + // no-op on subsequent boots. + sqlFor: { + pglite: ` + ALTER TABLE eval_candidates + ADD COLUMN IF NOT EXISTS embedding_column TEXT; + `, + }, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/operations.ts b/src/core/operations.ts index 5ce2a0840..f0c67fac0 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -1143,6 +1143,11 @@ const query: Operation = { description: "v0.34: scope search to a single source. Defaults to OperationContext.sourceId (set from CLI --source / GBRAIN_SOURCE / .gbrain-source dotfile). Pass '__all__' to force cross-source search in multi-source brains.", }, + embedding_column: { + type: 'string', + description: + "v0.36: route vector search through a non-default embedding column. Defaults to 'embedding' (OpenAI 1536d) unless `search_embedding_column` config sets a different default. Per-call override for A/B benchmarking across providers (e.g. 'embedding_voyage', 'embedding_zeroentropy'). Column MUST be declared in the `embedding_columns` config registry — unknown names throw with a paste-ready hint listing valid columns.", + }, }, handler: async (ctx, p) => { const startedAt = Date.now(); @@ -1151,6 +1156,10 @@ const query: Operation = { const queryText = p.query as string | undefined; const imageData = p.image as string | undefined; const imageMime = (p.image_mime as string) || 'image/jpeg'; + const embeddingColumnParam = + typeof p.embedding_column === 'string' && p.embedding_column.length > 0 + ? (p.embedding_column as string) + : undefined; // Explicit per-call source_id must win over ctx.sourceId. The special // __all__ value opts out of source filtering for local cross-source search. const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined; @@ -1220,6 +1229,12 @@ const query: Operation = { useCache: typeof p.use_cache === 'boolean' ? (p.use_cache as boolean) : undefined, intentWeighting: typeof p.intent_weighting === 'boolean' ? (p.intent_weighting as boolean) : undefined, onMeta: (m) => { capturedMeta = m; }, + // v0.36 (D15): per-call embedding column override. Resolver rejects + // unknown names at hybrid entry with EmbeddingColumnNotRegisteredError; + // the error surfaces back to the agent as the op error envelope. + // Source scope is already threaded via ...querySourceScope above + // (master's #1182 cleanup of the duplicate sourceScopeOpts spread). + embeddingColumn: embeddingColumnParam, }); const latency_ms = Date.now() - startedAt; diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 428427521..f17387126 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -41,6 +41,13 @@ import { GBrainError, PAGE_SORT_SQL } from './types.ts'; import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts'; +import { + normalizeEngineColumn, + buildVectorCastFragment, + quoteIdentifier, + COLUMN_NAME_REGEX, + EmbeddingColumnNotRegisteredError, +} from './search/embedding-column.ts'; import { hasCJK, escapeLikePattern } from './cjk.ts'; type PGLiteDB = PGlite; @@ -1279,15 +1286,18 @@ export class PGLiteEngine implements BrainEngine { // same candidate count it always did. See postgres-engine.ts for rationale. const visibilityClause = buildVisibilityClause('p', 's'); - // v0.27.1: column routing. Default 'embedding' targets the brain's - // primary text-embedding column; 'embedding_image' targets the - // multimodal column populated by importImageFile. Image-similarity - // queries pass embeddingColumn='embedding_image' AND a 1024-dim vector - // produced by gateway.embedMultimodal — must match the column dim. - const col = opts?.embeddingColumn === 'embedding_image' ? 'embedding_image' : 'embedding'; + // v0.36 (D11): column routing via resolved descriptor. Engine doesn't + // read config — caller resolved at hybrid/op boundary. The cast SQL + // ($1::vector vs $1::halfvec(N)) comes from buildVectorCastFragment. + const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn); + const { col, castSql } = buildVectorCastFragment(resolvedCol); // Image rows live in modality='image'; text/code in 'text'. Restrict - // to the modality matching the column to avoid cross-mode dim leaks. - const modalityFilter = col === 'embedding_image' ? `AND cc.modality = 'image'` : `AND cc.modality = 'text'`; + // by modality so non-image columns can't accidentally pull image + // chunks (or vice versa). resolved.name has already passed regex + // validation; never compares against raw input. + const modalityFilter = resolvedCol.name === 'embedding_image' + ? `AND cc.modality = 'image'` + : `AND cc.modality = 'text'`; const { rows } = await this.db.query( `WITH hnsw_candidates AS ( @@ -1295,12 +1305,12 @@ export class PGLiteEngine implements BrainEngine { p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - 1 - (cc.${col} <=> $1::vector) AS raw_score + 1 - (cc.${col} <=> ${castSql}) AS raw_score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.${col} IS NOT NULL ${modalityFilter} ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} - ORDER BY cc.${col} <=> $1::vector + ORDER BY cc.${col} <=> ${castSql} LIMIT $2 ) SELECT @@ -1321,10 +1331,21 @@ export class PGLiteEngine implements BrainEngine { return (rows as Record[]).map(rowToSearchResult); } - async getEmbeddingsByChunkIds(ids: number[]): Promise> { + async getEmbeddingsByChunkIds( + ids: number[], + column: string = 'embedding', + ): Promise> { if (ids.length === 0) return new Map(); + // v0.36 (D9): column parameter so hybrid.cosineReScore can rehydrate + // from the active embedding space (Voyage 1024d, ZE halfvec 2560d, + // etc.). Identifier-quoted (D12 layer 2) plus strict regex on the + // column name (D12 layer 1) before interpolation. + if (!COLUMN_NAME_REGEX.test(column)) { + throw new EmbeddingColumnNotRegisteredError(column, []); + } + const quotedCol = quoteIdentifier(column); const { rows } = await this.db.query( - `SELECT id, embedding FROM content_chunks WHERE id = ANY($1::int[]) AND embedding IS NOT NULL`, + `SELECT id, ${quotedCol} AS embedding FROM content_chunks WHERE id = ANY($1::int[]) AND ${quotedCol} IS NOT NULL`, [ids] ); const result = new Map(); @@ -3817,8 +3838,8 @@ export class PGLiteEngine implements BrainEngine { `INSERT INTO eval_candidates ( tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids, expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, - latency_ms, remote, job_id, subagent_id - ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) + latency_ms, remote, job_id, subagent_id, embedding_column + ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) RETURNING id`, [ input.tool_name, @@ -3835,6 +3856,7 @@ export class PGLiteEngine implements BrainEngine { input.remote, input.job_id, input.subagent_id, + input.embedding_column ?? null, ] ); return rows[0]!.id; diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index 3e4cd334d..e15104a07 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -467,7 +467,12 @@ CREATE TABLE IF NOT EXISTS eval_candidates ( salience_resolved TEXT, recency_resolved TEXT, salience_source TEXT, - recency_source TEXT + recency_source TEXT, + -- v0.36.3.0 (D16 / CDX-10) — embedding column that ran at capture time. + -- Nullable; pre-v0.36 rows have NULL and replay falls back to current + -- default. See src/core/migrate.ts migration v68 for the matching ALTER + -- on upgrade brains. + embedding_column TEXT ); CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index a6118c6ed..172ecb5bf 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -18,6 +18,13 @@ import { runMigrations } from './migrate.ts'; import { SCHEMA_SQL } from './schema-embedded.ts'; import { verifySchema } from './schema-verify.ts'; import { applyChunkEmbeddingIndexPolicy, dropZombieIndexes } from './vector-index.ts'; +import { + normalizeEngineColumn, + buildVectorCastFragment, + quoteIdentifier, + COLUMN_NAME_REGEX, + EmbeddingColumnNotRegisteredError, +} from './search/embedding-column.ts'; import type { Page, PageInput, PageFilters, PageType, Chunk, ChunkInput, StaleChunkRow, @@ -1292,9 +1299,20 @@ export class PostgresEngine implements BrainEngine { // wasting candidate slots on hidden rows. const visibilityClause = buildVisibilityClause('p', 's'); - // v0.27.1: column routing. See pglite-engine.ts searchVector for rationale. - const col = opts?.embeddingColumn === 'embedding_image' ? 'embedding_image' : 'embedding'; - const modalityFilter = col === 'embedding_image' ? `AND cc.modality = 'image'` : `AND cc.modality = 'text'`; + // v0.36 (D11): column routing via resolved descriptor. Engine doesn't + // read config — caller (hybrid/op) resolved it and passed it in. + // normalizeEngineColumn accepts the legacy union (string literals, + // ResolvedColumn, undefined) and produces a canonical descriptor. + const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn); + const { col, castSql } = buildVectorCastFragment(resolvedCol); + // Modality filter: image rows live in modality='image'; text/code in + // 'text'. The image-column literal is the only one with image rows. + // For all other columns (default + user-declared), restrict to text + // mode to avoid cross-modality dim leaks. The check is on + // resolved.name (already validated, never raw input). + const modalityFilter = resolvedCol.name === 'embedding_image' + ? `AND cc.modality = 'image'` + : `AND cc.modality = 'text'`; const rawQuery = ` WITH hnsw_candidates AS ( @@ -1302,7 +1320,7 @@ export class PostgresEngine implements BrainEngine { p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - 1 - (cc.${col} <=> $1::vector) AS raw_score + 1 - (cc.${col} <=> ${castSql}) AS raw_score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id @@ -1318,7 +1336,7 @@ export class PostgresEngine implements BrainEngine { ${sourceClause} ${hardExcludeClause} ${visibilityClause} - ORDER BY cc.${col} <=> $1::vector + ORDER BY cc.${col} <=> ${castSql} LIMIT ${innerLimitParam} ) SELECT @@ -1340,13 +1358,27 @@ export class PostgresEngine implements BrainEngine { return rows.map(rowToSearchResult); } - async getEmbeddingsByChunkIds(ids: number[]): Promise> { + async getEmbeddingsByChunkIds( + ids: number[], + column: string = 'embedding', + ): Promise> { if (ids.length === 0) return new Map(); + // v0.36 (D9): column parameter used by hybrid.cosineReScore so + // rescoring rehydrates from the active column's embedding space, + // not always 'embedding'. Engine has no resolver access; the + // caller must pass a known column name. Identifier-quoted (D12 + // defense layer 2) plus a strict regex check (D12 defense layer 1) + // so even a misconfigured caller can't smuggle a SQL fragment. + if (!COLUMN_NAME_REGEX.test(column)) { + throw new EmbeddingColumnNotRegisteredError(column, []); + } + const quotedCol = quoteIdentifier(column); const sql = this.sql; - const rows = await sql` - SELECT id, embedding FROM content_chunks - WHERE id = ANY(${ids}::int[]) AND embedding IS NOT NULL + const rawQuery = ` + SELECT id, ${quotedCol} AS embedding FROM content_chunks + WHERE id = ANY($1::int[]) AND ${quotedCol} IS NOT NULL `; + const rows = await sql.unsafe(rawQuery, [ids] as Parameters[1]); const result = new Map(); for (const row of rows) { const embedding = tryParseEmbedding(row.embedding); @@ -3777,11 +3809,11 @@ export class PostgresEngine implements BrainEngine { INSERT INTO eval_candidates ( tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids, expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, - latency_ms, remote, job_id, subagent_id + latency_ms, remote, job_id, subagent_id, embedding_column ) VALUES ( ${input.tool_name}, ${input.query}, ${input.retrieved_slugs}, ${input.retrieved_chunk_ids}, ${input.source_ids}, ${input.expand_enabled}, ${input.detail}, ${input.detail_resolved}, ${input.vector_enabled}, ${input.expansion_applied}, - ${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id} + ${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id}, ${input.embedding_column ?? null} ) RETURNING id `; diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 598468b76..db6970bd3 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -791,7 +791,13 @@ CREATE TABLE IF NOT EXISTS eval_candidates ( salience_resolved TEXT, recency_resolved TEXT, salience_source TEXT, - recency_source TEXT + recency_source TEXT, + -- v0.36.3.0 (D16 / CDX-10) — embedding column resolved at capture time so + -- \`gbrain eval replay\` reproduces the same column the capture ran against. + -- Nullable; pre-v0.36 rows have NULL and replay falls back to current + -- default. Migration v68 (src/core/migrate.ts) adds the same column on + -- upgrade brains. + embedding_column TEXT ); CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); diff --git a/src/core/search/embedding-column.ts b/src/core/search/embedding-column.ts new file mode 100644 index 000000000..7cb2d0d71 --- /dev/null +++ b/src/core/search/embedding-column.ts @@ -0,0 +1,538 @@ +/** + * Dynamic embedding column resolver (v0.36 — D2+D3+D11+D12). + * + * Single source of truth for "which content_chunks column gets searched, and + * what provider produced its vectors." The hybrid/op boundary calls + * `resolveEmbeddingColumn()` once and passes the resulting descriptor INTO + * the engine; engines never read config or call the resolver themselves + * (D11 — engine stays a pure SQL composer). + * + * ┌─────────────────────────────────────────────┐ + * │ Config registry (file + DB plane merged) │ + * │ embedding_columns: { name → entry } │ + * │ search_embedding_column: string │ + * └────────────────────┬────────────────────────┘ + * │ + * ▼ + * ┌─────────────────────────────────────────────┐ + * │ getEmbeddingColumnRegistry(cfg) │ + * │ Merge: BUILTIN_KEYS + user config │ + * │ Validate at load time (D12): │ + * │ key: /^[a-z_][a-z0-9_]*$/ │ + * │ type: 'vector' | 'halfvec' │ + * │ dims: 1..8192 │ + * │ provider: parseable 'provider:model' │ + * └────────────────────┬────────────────────────┘ + * │ + * ┌────────────────────────┼────────────────────────┐ + * ▼ ▼ ▼ + * ┌──────────────────────┐ ┌──────────────────────┐ ┌─────────────────────┐ + * │ resolveEmbeddingCol │ │ getEmbeddingColumnRe │ │ buildVectorCastFrag │ + * │ (opts, cfg) │ │ gistry(cfg) │ │ ment(resolved) │ + * │ │ │ │ │ │ + * │ Chain: │ │ Returns full reg │ │ Returns: │ + * │ opts.embeddingColumn│ │ for doctor + config │ │ { col, castSql } │ + * │ → cfg.search_emb_ │ │ validation. │ │ │ + * │ column │ │ │ │ col: quoted ident │ + * │ → 'embedding' │ │ │ │ "" (D12) │ + * │ │ │ │ │ castSql: │ + * │ Returns Resolved- │ │ │ │ '$1::vector' OR │ + * │ Column descriptor. │ │ │ │ '$1::halfvec(N)' │ + * └──────────┬───────────┘ └──────────────────────┘ └─────────────────────┘ + * │ + * ▼ (descriptor passed to engines + cosineReScore) + * ┌──────────────────────────────────────────────────────────────────┐ + * │ engines: searchVector + getEmbeddingsByChunkIds │ + * │ SQL: SELECT ... FROM content_chunks cc │ + * │ WHERE cc.${quoteIdentifier(name)} IS NOT NULL │ + * │ ORDER BY cc.${quoteIdentifier(name)} <=> $1::${cast} │ + * └──────────────────────────────────────────────────────────────────┘ + * + * Trust boundary: registry KEYS come from user config (DB plane + file + * plane). Any path with config-write access can in principle declare a key. + * Defense in depth: + * 1. Strict regex on every key at load time — loud throw on invalid. + * 2. Identifier-quoting at SQL-build time — even if regex misses, + * pgvector identifier quoting prevents string-break injection. + * 3. Field validation at load (type/dims/provider) — bad config refuses + * to load instead of silently ignoring entries. + * + * Future write-path PR (`gbrain embed --column X --model Y`, deferred per + * D1 out-of-scope list) MUST also consume this resolver — never compute a + * column→provider mapping by hand. This is the canonical seam. + */ + +import type { + EmbeddingColumnConfig, + ResolvedColumn, + SearchOpts, +} from '../types.ts'; +import type { GBrainConfig } from '../config.ts'; + +// ---- Constants --------------------------------------------------------- + +/** Strict identifier regex for registry keys. Defense layer 1 of D12. */ +export const COLUMN_NAME_REGEX = /^[a-z_][a-z0-9_]*$/; + +/** Allowed pgvector types for this registry. Halfvec lands at v0.4.3+. */ +export const ALLOWED_COLUMN_TYPES = new Set([ + 'vector', + 'halfvec', +]); + +/** Upper bound on declared dimensions. pgvector hard cap is 16K but + * practical embedding models top out around 4096; 8192 is plenty of + * headroom and rejects obvious junk like negative or astronomical + * values. */ +export const MAX_DIMENSIONS = 8192; + +/** + * Default name used when neither caller nor config sets a column. + * Resolution chain: opts → cfg.search_embedding_column → DEFAULT. + */ +export const DEFAULT_COLUMN_NAME = 'embedding'; + +/** Names that always exist regardless of user config. Both derive their + * provider from existing config keys (embedding_model and + * embedding_multimodal_model) so users who don't declare anything still + * get correct routing. */ +const BUILTIN_KEYS = ['embedding', 'embedding_image'] as const; +type BuiltinKey = (typeof BUILTIN_KEYS)[number]; + +// ---- Errors ----------------------------------------------------------- + +/** + * Thrown when a column name isn't in the merged registry. Carries a + * paste-ready hint listing valid columns so the user (or agent) sees + * exactly what to type next. + */ +export class EmbeddingColumnNotRegisteredError extends Error { + readonly code = 'embedding_column_not_registered'; + readonly columnName: string; + readonly validColumns: string[]; + + constructor(columnName: string, validColumns: string[]) { + const valid = validColumns.length > 0 ? validColumns.join(', ') : '(none)'; + super( + `Embedding column "${columnName}" is not registered. ` + + `Declared columns: ${valid}. ` + + `Add it via: gbrain config set embedding_columns ''`, + ); + this.name = 'EmbeddingColumnNotRegisteredError'; + this.columnName = columnName; + this.validColumns = validColumns; + } +} + +/** + * Thrown when a registry entry fails load-time validation. The .field + * pinpoints which sub-shape was wrong so doctor + config-set surfaces + * can render targeted hints. + */ +export class EmbeddingColumnConfigError extends Error { + readonly code = 'embedding_column_config_invalid'; + readonly columnKey: string; + readonly field: 'key' | 'type' | 'dimensions' | 'provider' | 'shape'; + + constructor( + columnKey: string, + field: EmbeddingColumnConfigError['field'], + detail: string, + ) { + super(`embedding_columns["${columnKey}"]: invalid ${field} — ${detail}`); + this.name = 'EmbeddingColumnConfigError'; + this.columnKey = columnKey; + this.field = field; + } +} + +// ---- Validation helpers (D12) ----------------------------------------- + +/** + * Loud rejection of any registry key that isn't a strict SQL identifier. + * Catches `embedding"; DROP --`, `embed-1`, leading-digits, etc. at the + * earliest possible moment. + */ +export function validateColumnKey(key: string): void { + if (typeof key !== 'string' || key.length === 0) { + throw new EmbeddingColumnConfigError( + String(key ?? ''), + 'key', + 'must be a non-empty string', + ); + } + if (!COLUMN_NAME_REGEX.test(key)) { + throw new EmbeddingColumnConfigError( + key, + 'key', + `must match ${COLUMN_NAME_REGEX} (lowercase identifier, starts with letter or underscore, no quotes/symbols)`, + ); + } +} + +/** `provider:model` string check. Both halves must be non-empty. */ +function isParseableProviderModel(s: unknown): s is string { + if (typeof s !== 'string' || s.length === 0) return false; + const idx = s.indexOf(':'); + if (idx <= 0) return false; + if (idx === s.length - 1) return false; + return true; +} + +/** + * Validates an entry's shape. Throws on any failure with a pinpoint hint. + * Called by getEmbeddingColumnRegistry at load time AND by config-set + * before persisting. + */ +export function validateColumnConfig( + key: string, + entry: unknown, +): asserts entry is EmbeddingColumnConfig { + if (!entry || typeof entry !== 'object' || Array.isArray(entry)) { + throw new EmbeddingColumnConfigError( + key, + 'shape', + 'must be a JSON object', + ); + } + const e = entry as Record; + + if (!isParseableProviderModel(e.provider)) { + throw new EmbeddingColumnConfigError( + key, + 'provider', + 'must be a non-empty "provider:model" string (e.g. "voyage:voyage-3-large")', + ); + } + if ( + typeof e.dimensions !== 'number' || + !Number.isInteger(e.dimensions) || + e.dimensions < 1 || + e.dimensions > MAX_DIMENSIONS + ) { + throw new EmbeddingColumnConfigError( + key, + 'dimensions', + `must be an integer in [1, ${MAX_DIMENSIONS}]`, + ); + } + if (typeof e.type !== 'string' || !ALLOWED_COLUMN_TYPES.has(e.type as 'vector' | 'halfvec')) { + throw new EmbeddingColumnConfigError( + key, + 'type', + `must be one of: ${[...ALLOWED_COLUMN_TYPES].join(', ')}`, + ); + } +} + +// ---- SQL helpers (D12 defense layer 2) -------------------------------- + +/** + * Identifier-quoting helper. Wraps `name` in double quotes and doubles + * any embedded quotes per the Postgres spec. Even though the column + * name passed in here is already regex-validated (so no embedded + * quotes are possible), this is the defense-in-depth belt for D12. + * + * Returns the quoted form ready to drop into a SQL string. Example: + * quoteIdentifier('embedding_voyage') === '"embedding_voyage"' + */ +export function quoteIdentifier(name: string): string { + return `"${name.replace(/"/g, '""')}"`; +} + +/** + * Builds the per-engine SQL fragments for a resolved column. + * + * Returns: + * col — identifier-quoted column name for SELECT / WHERE / ORDER BY. + * castSql — placeholder cast for the query vector parameter: + * '$1::vector' for type='vector' + * '$1::halfvec()' for type='halfvec' + * + * Callers interpolate both into the SQL string. The `$1` is the + * positional parameter that postgres.js / PGLite will bind the query + * vector to. Different placeholders ($2, $3, etc.) can be obtained + * by string-substitution from the caller; we standardize on $1 here + * since both engines use it for the query vector and the cast lives + * adjacent to the placeholder anyway. + */ +export function buildVectorCastFragment(resolved: ResolvedColumn): { + col: string; + castSql: string; +} { + const col = quoteIdentifier(resolved.name); + const castSql = + resolved.type === 'halfvec' + ? `$1::halfvec(${resolved.dimensions})` + : `$1::vector`; + return { col, castSql }; +} + +// ---- Registry -------------------------------------------------------- + +/** + * Returns the merged registry: built-ins (`embedding`, `embedding_image`) + * + user-declared `embedding_columns`. User declarations win on key + * collision so power users can override a builtin's dim/provider (e.g., + * pointing 'embedding' at a Voyage column on a fresh brain). + * + * Throws if any user entry fails D12 validation. Built-ins are + * derived live from the broader config (embedding_model, + * embedding_dimensions, embedding_multimodal_model), so a missing + * embedding_model produces a default 'embedding' entry pointing at the + * gateway default. The doctor check catches mismatches against actual + * DB column shape. + */ +export function getEmbeddingColumnRegistry( + cfg: GBrainConfig, +): Record { + // Prototype-pollution safe (codex /ship #1). Plain `{}` inherits from + // Object.prototype so `registry["constructor"]` returns + // `Object.prototype.constructor` (a truthy function). Object.create(null) + // creates a dict with NO prototype so unknown keys are genuinely absent. + const out: Record = Object.create(null); + + // Builtin: 'embedding' — derived from primary config keys. + const embedModel = cfg.embedding_model ?? 'openai:text-embedding-3-large'; + const embedDims = + typeof cfg.embedding_dimensions === 'number' && cfg.embedding_dimensions > 0 + ? cfg.embedding_dimensions + : 1536; + out['embedding'] = { + provider: embedModel, + dimensions: embedDims, + type: 'vector', + }; + + // Builtin: 'embedding_image' — derived from multimodal config keys. + // Hardcoded 1024d / vector because that's the committed schema shape + // (see src/schema.sql:158). If the user runs a different multimodal + // model they can override via the user registry. + const mmModel = cfg.embedding_multimodal_model ?? 'voyage:voyage-multimodal-3'; + out['embedding_image'] = { + provider: mmModel, + dimensions: 1024, + type: 'vector', + }; + + // User-declared columns. Validate every key + entry at merge time. + const userColumns = cfg.embedding_columns; + if (userColumns && typeof userColumns === 'object' && !Array.isArray(userColumns)) { + for (const [key, value] of Object.entries(userColumns)) { + validateColumnKey(key); + validateColumnConfig(key, value); + out[key] = value; + } + } + + return out; +} + +/** + * Resolves the effective column descriptor for one query. + * + * Resolution chain: + * 1. opts.embeddingColumn (per-call override, e.g. from MCP query op) + * 2. cfg.search_embedding_column (DB-plane default) + * 3. DEFAULT_COLUMN_NAME ('embedding') + * + * The resolved name is looked up in the merged registry; unknown names + * throw EmbeddingColumnNotRegisteredError with a paste-ready hint. + * + * When `opts.embeddingColumn` is already a ResolvedColumn (engine-internal + * shape after the boundary), it's returned as-is so re-resolving is a + * no-op. This lets hybridSearch resolve once and pass the descriptor down + * to multiple engine calls without redoing the work. + */ +export function resolveEmbeddingColumn( + opts: Pick | undefined, + cfg: GBrainConfig, +): ResolvedColumn { + // Fast path: already resolved (engine-internal). Re-validate the + // descriptor shape before returning (codex /ship #2). SDK callers + // could pass a hand-rolled object via the public `gbrain/types` + // surface; runtime check ensures the engine still sees a known-safe + // shape. The validation is the same shape the resolver applies on + // the string path. + const candidate = opts?.embeddingColumn; + if ( + candidate && + typeof candidate === 'object' && + !Array.isArray(candidate) && + typeof (candidate as ResolvedColumn).name === 'string' + ) { + const r = candidate as ResolvedColumn; + if (!COLUMN_NAME_REGEX.test(r.name)) { + throw new EmbeddingColumnNotRegisteredError(r.name, []); + } + if (r.type !== 'vector' && r.type !== 'halfvec') { + throw new EmbeddingColumnConfigError(r.name, 'type', `descriptor.type must be 'vector' or 'halfvec' (got: ${String(r.type)})`); + } + if ( + typeof r.dimensions !== 'number' || + !Number.isInteger(r.dimensions) || + r.dimensions < 1 || + r.dimensions > MAX_DIMENSIONS + ) { + throw new EmbeddingColumnConfigError(r.name, 'dimensions', `descriptor.dimensions must be an integer in [1, ${MAX_DIMENSIONS}] (got: ${String(r.dimensions)})`); + } + return r; + } + + // String chain. + const requestedName = + (typeof candidate === 'string' && candidate.length > 0 ? candidate : undefined) ?? + (typeof cfg.search_embedding_column === 'string' && cfg.search_embedding_column.length > 0 + ? cfg.search_embedding_column + : undefined) ?? + DEFAULT_COLUMN_NAME; + + // Defense layer 1: regex on the requested name BEFORE registry lookup. + // Even if someone bypasses load-time validation, the resolver refuses + // to look up a name that doesn't look like an identifier. + if (!COLUMN_NAME_REGEX.test(requestedName)) { + throw new EmbeddingColumnNotRegisteredError( + requestedName, + [], // We don't have the registry yet at this point; render later. + ); + } + + const registry = getEmbeddingColumnRegistry(cfg); + // Use Object.hasOwn so inherited keys (constructor, hasOwnProperty, + // __proto__, etc.) cannot resolve. The registry itself uses + // Object.create(null) but defense-in-depth here too — the codex /ship + // #1 finding was specifically about resolveEmbeddingColumn's lookup. + if (!Object.hasOwn(registry, requestedName)) { + throw new EmbeddingColumnNotRegisteredError( + requestedName, + Object.keys(registry).sort(), + ); + } + const entry = registry[requestedName]; + + return { + name: requestedName, + type: entry.type, + dimensions: entry.dimensions, + embeddingModel: entry.provider, + }; +} + +/** + * True when the resolved column is the default `embedding` name. + * Name-based check; does not compare embedding space. + */ +export function isDefaultColumn(resolved: ResolvedColumn): boolean { + return resolved.name === DEFAULT_COLUMN_NAME; +} + +/** + * True when the resolved column's embedding space matches the + * `query_cache.embedding` column's space — i.e., it's safe to read + * from / write to the semantic query cache without dimension or + * vector-space corruption. + * + * Codex /ship finding #4: `isDefaultColumn` is name-based, so a user + * who overrides the `embedding` builtin to point at a different + * provider/dim (legitimate use of the registry override semantics) + * would still have their cache used — but the cache table is sized + * for the ORIGINAL default embedding dim. Mismatched dim/model means + * the cache is wrong-space; skip it. + * + * Cache-safety criteria: + * 1. Column name is `embedding` (the cache table only knows about + * this column; non-default columns always skip). + * 2. Resolved dimensions match `cfg.embedding_dimensions` (or + * DEFAULT_EMBEDDING_DIMENSIONS=1536 when unset). + * 3. Resolved provider matches `cfg.embedding_model` (or the OpenAI + * default). The model is the "embedding space identifier" — two + * models produce non-interchangeable vectors even at the same + * dim count. + * + * When any of these mismatch, return false so hybridSearchCached + * skips both the lookup and the writeback paths. + */ +export function isCacheSafe(resolved: ResolvedColumn, cfg: GBrainConfig): boolean { + if (resolved.name !== DEFAULT_COLUMN_NAME) return false; + const cfgDims = (typeof cfg.embedding_dimensions === 'number' && cfg.embedding_dimensions > 0) + ? cfg.embedding_dimensions + : 1536; + if (resolved.dimensions !== cfgDims) return false; + const cfgModel = cfg.embedding_model ?? 'openai:text-embedding-3-large'; + if (resolved.embeddingModel !== cfgModel) return false; + return true; +} + +/** Type guard: is this string a valid BuiltinKey? Useful for callers + * that want to special-case the 'embedding_image' multimodal path + * without doing a string-compare scattered throughout the code. */ +export function isBuiltinColumn(name: string): name is BuiltinKey { + return (BUILTIN_KEYS as readonly string[]).includes(name); +} + +/** + * Engine-side normalizer. Accepts the legacy SearchOpts.embeddingColumn + * union (`'embedding'` | `'embedding_image'` | string | ResolvedColumn | + * undefined) and returns a ResolvedColumn. Engine code calls this once at + * the top of searchVector / getEmbeddingsByChunkIds. The engine remains + * config-free — this function reads NO config, only handles the known + * builtin shapes statically. + * + * Behavior: + * - ResolvedColumn → returned as-is. + * - undefined → builtin 'embedding' descriptor (vector, dims=1536). + * - 'embedding' literal → same as undefined. + * - 'embedding_image' literal → builtin 'embedding_image' descriptor. + * - Any other string → throws EmbeddingColumnNotRegisteredError. The + * resolver lives at hybrid/op boundary; bare string names are NOT + * accepted at the engine layer (per D11 — engine purity). + * + * `dims` for the 'embedding' builtin is hardcoded 1536 here purely as + * a no-op placeholder; the engine's SQL cast is `$1::vector` (no + * parenthesized N) when type='vector', so the dims field is unused by + * `buildVectorCastFragment` and never touches the wire. Tests that + * care about dims should pass a real descriptor. + */ +export function normalizeEngineColumn( + embeddingColumn: SearchOpts['embeddingColumn'] | undefined, +): ResolvedColumn { + // ResolvedColumn descriptor → use as-is. + if ( + embeddingColumn && + typeof embeddingColumn === 'object' && + !Array.isArray(embeddingColumn) && + typeof (embeddingColumn as ResolvedColumn).name === 'string' + ) { + return embeddingColumn as ResolvedColumn; + } + + // Default + legacy 'embedding' literal. + if (embeddingColumn === undefined || embeddingColumn === 'embedding') { + return { + name: 'embedding', + type: 'vector', + dimensions: 1536, // placeholder — not used by $1::vector cast + embeddingModel: '', // engine doesn't embed; left blank + }; + } + + // Legacy multimodal literal — committed schema shape per + // src/schema.sql:158 (vector(1024)). + if (embeddingColumn === 'embedding_image') { + return { + name: 'embedding_image', + type: 'vector', + dimensions: 1024, + embeddingModel: '', + }; + } + + // Any other raw string at this layer is a programming error — the + // resolver should have run at the hybrid/op boundary and produced a + // descriptor. We throw with a paste-ready hint rather than guess. + throw new EmbeddingColumnNotRegisteredError(String(embeddingColumn), [ + 'embedding', + 'embedding_image', + '', + ]); +} diff --git a/src/core/search/hybrid.ts b/src/core/search/hybrid.ts index bdff5766a..92d7f24d5 100644 --- a/src/core/search/hybrid.ts +++ b/src/core/search/hybrid.ts @@ -13,6 +13,8 @@ import type { BrainEngine } from '../engine.ts'; import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts'; import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts'; import { embed, embedQuery } from '../embedding.ts'; +import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts'; +import { loadConfigWithEngine } from '../config.ts'; import { dedupResults } from './dedup.ts'; import { applyReranker } from './rerank.ts'; import { autoDetectDetail, classifyQuery } from './query-intent.ts'; @@ -358,6 +360,19 @@ export async function hybridSearch( }, }); + // v0.36 (D7+D11): resolve embedding column once at entry. Single + // round-trip to read DB-plane config (mirrors loadSearchModeConfig). + // Resolver throws on unknown name with a paste-ready hint; let it + // propagate — a misconfig should be loud, not silently fall back. + // Failing cfg load (pre-config brain, mid-migration, no engine.getConfig) + // falls through to the file-plane sync loadConfig() — same shape, just + // misses DB-plane overrides. + const mergedCfg = await loadConfigWithEngine(engine).catch(() => null); + const cfgForColumn = mergedCfg ?? ((await import('../config.ts')).loadConfig()) ?? null; + const resolvedCol = cfgForColumn + ? resolveEmbeddingColumn(opts, cfgForColumn) + : resolveEmbeddingColumn(opts, { engine: 'pglite' }); + const limit = opts?.limit || resolvedMode.searchLimit; const offset = opts?.offset || 0; const innerLimit = Math.min(limit * 2, MAX_SEARCH_LIMIT); @@ -402,6 +417,10 @@ export async function hybridSearch( // ordering means we can't lazy-spread the full opts). sourceId: opts?.sourceId, sourceIds: opts?.sourceIds, + // v0.36 (D11): pass the pre-validated descriptor into the engine so + // it never has to read config. Engines normalize string-or-descriptor + // via normalizeEngineColumn; the descriptor path is the strict one. + embeddingColumn: resolvedCol, }; // Track what actually ran for the optional onMeta callback (v0.25.0). // Caller leaves onMeta undefined → these flags are computed but never @@ -478,8 +497,13 @@ export async function hybridSearch( }; // Skip vector search entirely if the gateway has no embedding provider configured (Codex C3). + // v0.36 (D10): ask "is the RESOLVED column's provider reachable?" rather + // than "is the global default reachable?" — otherwise an unreachable + // global default disables vector search even when the active column's + // provider (Voyage, ZE) works fine. const { isAvailable } = await import('../ai/gateway.ts'); - if (!isAvailable('embedding')) { + const providerProbe = resolvedCol.embeddingModel || undefined; + if (!isAvailable('embedding', providerProbe)) { if (keywordResults.length > 0) { await runPostFusionStages(engine, keywordResults, postFusionOpts); keywordResults.sort((a, b) => b.score - a.score); @@ -494,6 +518,7 @@ export async function hybridSearch( expansion_applied: false, intent: suggestions.intent, mode: resolvedMode.resolved_mode, + embedding_column: resolvedCol.name, ...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0 ? { token_budget: noEmbedBudgetMeta } : {}), @@ -526,7 +551,15 @@ export async function hybridSearch( // v0.35.0.0+: query-side embedding. For asymmetric providers (ZE zembed-1, // Voyage v3+) routes input_type='query' through the embed seam; symmetric // providers ignore the field — no behavior change. - const embeddings = await Promise.all(queries.map(q => embedQuery(q))); + // v0.36 (D10): route through the resolved column's provider + dims so + // a query against `embedding_voyage` actually embeds via Voyage, not + // the global default (OpenAI). Empty embeddingModel falls back to + // gateway default — preserves pre-v0.36 behavior for the builtin + // 'embedding' column. + const embedOpts = resolvedCol.embeddingModel + ? { embeddingModel: resolvedCol.embeddingModel, dimensions: resolvedCol.dimensions } + : undefined; + const embeddings = await Promise.all(queries.map(q => embedQuery(q, embedOpts))); queryEmbedding = embeddings[0]; vectorLists = await Promise.all( embeddings.map(emb => engine.searchVector(emb, searchOpts)), @@ -554,6 +587,7 @@ export async function hybridSearch( expansion_applied: expansionApplied, intent: suggestions.intent, mode: resolvedMode.resolved_mode, + embedding_column: resolvedCol.name, ...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0 ? { token_budget: kwBudgetMeta } : {}), @@ -577,9 +611,12 @@ export async function hybridSearch( ]; let fused = rrfFusionWeighted(allLists, detail !== 'high'); - // Cosine re-scoring before dedup so semantically better chunks survive + // Cosine re-scoring before dedup so semantically better chunks survive. + // v0.36 (D9): hydrate from the active embedding column so rescore happens + // in the same vector space the HNSW just ranked in. Pre-v0.36 this + // always pulled from `embedding` and silently corrupted alt-column ranks. if (queryEmbedding) { - fused = await cosineReScore(engine, fused, queryEmbedding); + fused = await cosineReScore(engine, fused, queryEmbedding, resolvedCol.name); } // v0.29.1: post-fusion stages (backlink + salience + recency) run via @@ -687,6 +724,7 @@ export async function hybridSearch( expansion_applied: expansionApplied, intent: suggestions.intent, mode: resolvedMode.resolved_mode, + embedding_column: resolvedCol.name, ...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0 ? { token_budget: budgetMeta } : {}), @@ -742,7 +780,26 @@ export async function hybridSearchCached( floor_ratio: opts?.floorRatio, }, }); - const cacheKnobsHash = knobsHash(resolvedForCache); + // v0.36 (D8 / CDX-2 + codex /ship #4): resolve column for the cache + // decision. The query_cache.embedding column has one fixed pgvector dim + // sized at brain init; storing a 1024d Voyage or 2560d ZE cache + // embedding fails or corrupts results. Name-based check ("is it the + // default `embedding` column?") is insufficient — the registry + // explicitly allows overriding builtin `embedding` to a different + // provider/dim. isCacheSafe compares the resolved column's full + // embedding space (name + dim + model) against cfg and returns true + // only when ALL match. Otherwise skip. + const mergedCfgCached = await loadConfigWithEngine(engine).catch(() => null); + const cfgCached = mergedCfgCached ?? ((await import('../config.ts')).loadConfig()) ?? { engine: 'pglite' as const }; + const resolvedColCached = resolveEmbeddingColumn(opts, cfgCached); + const isNonDefaultColumn = !isCacheSafe(resolvedColCached, cfgCached); + + // Cache key carries the column + provider so different embedding spaces + // never collide on the same `(source_id, query_text)` row. + const cacheKnobsHash = knobsHash(resolvedForCache, { + embeddingColumn: resolvedColCached.name, + embeddingModel: resolvedColCached.embeddingModel, + }); // Cache decision: opts.useCache (explicit) wins over global config; global // config wins over mode bundle default. Mode bundle is on for all 3 modes @@ -756,14 +813,15 @@ export async function hybridSearchCached( ttlSeconds: resolvedForCache.cache_ttl_seconds, }); - // Skip cache entirely when the request asks for two-pass walks or has - // a non-default embedding column — those interact with structural state - // that the cache can't safely express. + // Skip cache entirely when the request asks for two-pass walks, has + // a non-default embedding column (per-call or via config default — + // D8 closes the silent-corruption bug class), or near-symbol mode + // (structural state that the cache can't safely express). const skipCache = !cache.isEnabled() || (opts?.walkDepth ?? 0) > 0 || Boolean(opts?.nearSymbol) || - (opts?.embeddingColumn && opts.embeddingColumn !== 'embedding'); + isNonDefaultColumn; let cacheStatus: 'hit' | 'miss' | 'disabled' = skipCache ? 'disabled' : 'miss'; let cacheSimilarity: number | undefined; @@ -778,7 +836,13 @@ export async function hybridSearchCached( if (!skipCache) { try { const { isAvailable } = await import('../ai/gateway.ts'); - if (isAvailable('embedding')) { + // v0.36 (D10): for the cache-lookup embedding, also use the resolved + // column's provider. The cache lookup is always against the default + // 'embedding' column (skipCache short-circuits non-default above), + // so this is the default embeddingModel — but threading it keeps + // the provider probe consistent with the bare hybridSearch path. + const providerProbeCached = resolvedColCached.embeddingModel || undefined; + if (isAvailable('embedding', providerProbeCached)) { // v0.35.0.0+: query-side embedding (cache lookup path). queryEmbedding = await embedQuery(query); } else { @@ -983,6 +1047,7 @@ async function cosineReScore( engine: BrainEngine, results: SearchResult[], queryEmbedding: Float32Array, + column: string = 'embedding', ): Promise { const chunkIds = results .map(r => r.chunk_id) @@ -992,7 +1057,11 @@ async function cosineReScore( let embeddingMap: Map; try { - embeddingMap = await engine.getEmbeddingsByChunkIds(chunkIds); + // v0.36 (D9): hydrate from the active column so rescore happens in + // the same embedding space the HNSW just ranked in. Without this, + // a Voyage HNSW retrieval would HNSW-rank against Voyage vectors but + // rescore against OpenAI vectors → NaN or wrong rankings. + embeddingMap = await engine.getEmbeddingsByChunkIds(chunkIds, column); } catch { // DB error is non-fatal, return results without re-scoring return results; diff --git a/src/core/search/mode.ts b/src/core/search/mode.ts index 267fe144c..b464445fc 100644 --- a/src/core/search/mode.ts +++ b/src/core/search/mode.ts @@ -368,7 +368,28 @@ export function attributeKnob( // `cache.ttl_seconds` (default 3600s). The CHANGELOG note covers this. export const KNOBS_HASH_VERSION = 3; -export function knobsHash(knobs: ResolvedSearchKnobs): string { +/** + * v0.36 (D8 / CDX-2) — second-arg context for the cache key. The + * embedding column + provider live OUTSIDE ResolvedSearchKnobs because + * they're orthogonal to search mode (mode bundles don't pick columns). + * Passing them as a second argument keeps ModeBundle pure and lets the + * hash invalidate correctly across column/provider switches. + * + * When undefined, the hash falls back to the legacy 'embedding' / + * 'default' values so unrelated callers (eval-replay, telemetry) that + * don't know the column produce a stable hash for the default case. + */ +export interface KnobsHashContext { + /** Resolved column name, e.g. 'embedding', 'embedding_voyage'. */ + embeddingColumn?: string; + /** Resolved provider:model, e.g. 'voyage:voyage-3-large'. */ + embeddingModel?: string; +} + +export function knobsHash( + knobs: ResolvedSearchKnobs, + ctx?: KnobsHashContext, +): string { // Fixed-order key list. Adding a knob here REQUIRES bumping // KNOBS_HASH_VERSION and is a breaking change for any persisted cache. const parts = [ @@ -387,10 +408,20 @@ export function knobsHash(knobs: ResolvedSearchKnobs): string { `rri=${knobs.reranker_top_n_in}`, `rro=${knobs.reranker_top_n_out ?? 'none'}`, `rrt=${knobs.reranker_timeout_ms}`, - // v=3 additions (append-only). Use 4-decimal precision so 0.85 and - // 0.851 differ in the hash; undefined uses literal 'none' so a - // floor-off write and a floor-on write key into different rows. + // v=3 additions (append-only). Both contributions landed under v=3: + // + // floor_ratio (v0.35.6.0 / codex T1): a floor-on write must not be + // served to a floor-off lookup. 4-decimal precision so 0.85 and + // 0.851 produce different hashes; undefined uses literal 'none'. + // + // col + prov (v0.36 / D8 / CDX-2): cross-column + cross-provider + // cache contamination. A query against `embedding_voyage` must + // NEVER be served from a cache row that ran against `embedding` + // — they sit in different vector spaces. ctx is optional so + // unrelated callers fall back to the default-column hash. `fr=${knobs.floor_ratio === undefined ? 'none' : knobs.floor_ratio.toFixed(4)}`, + `col=${ctx?.embeddingColumn ?? 'embedding'}`, + `prov=${ctx?.embeddingModel ?? 'default'}`, ]; const h = createHash('sha256'); h.update(parts.join('|')); diff --git a/src/core/types.ts b/src/core/types.ts index 21456b460..ba5e472d4 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -421,6 +421,52 @@ export interface SearchResult { effective_date_source?: string | null; } +/** + * v0.36 (D12) — declared shape of one entry in the `embedding_columns` + * config registry. Users declare entries via `gbrain config set + * embedding_columns '...JSON...'` (DB plane); resolver merges with + * built-ins. Field validation lives in src/core/search/embedding-column.ts + * and runs at load time: + * + * provider: parseable 'provider:model' (e.g. 'voyage:voyage-3-large') + * dimensions: positive integer 1..8192 + * type: 'vector' | 'halfvec' + * + * The registry KEY itself (the column name) is also regex-validated + * (/^[a-z_][a-z0-9_]*$/) so a malicious config write can't smuggle a + * SQL fragment in via the key. + */ +export interface EmbeddingColumnConfig { + /** 'provider:model' identifier, e.g. 'voyage:voyage-3-large'. */ + provider: string; + /** Dimensions of the stored vector. Must match actual DB column. */ + dimensions: number; + /** pgvector type — drives the SQL cast at search time. */ + type: 'vector' | 'halfvec'; +} + +/** + * v0.36 (D11) — fully resolved descriptor for an embedding column. The + * resolver (src/core/search/embedding-column.ts) produces these at the + * hybrid/op boundary; engines consume them. The engine NEVER reads config + * or calls the resolver itself — it sees only this validated descriptor. + * + * `name` is identifier-quoted at SQL-build time by buildVectorCastFragment + * (D12 defense in depth), but the resolver also enforces a strict regex + * at load time so this string is already known-safe by the time the engine + * sees it. + */ +export interface ResolvedColumn { + /** Column name in content_chunks (already validated against the registry). */ + name: string; + /** pgvector type — `$N::vector` or `$N::halfvec(N)`. */ + type: 'vector' | 'halfvec'; + /** Embedding dimensions — must match actual DB column dim. */ + dimensions: number; + /** 'provider:model' identifier, e.g. 'voyage:voyage-3-large'. */ + embeddingModel: string; +} + export interface SearchOpts { limit?: number; offset?: number; @@ -490,14 +536,27 @@ export interface SearchOpts { */ sourceIds?: string[]; /** - * v0.27.1: target column for vector search. 'embedding' (default) hits - * the brain's primary text-embedding column. 'embedding_image' targets - * the multimodal column populated by importImageFile. The two columns - * may live in different dim spaces (e.g. OpenAI 1536 + Voyage 1024) - * which is why the dual-column schema landed in v0.27.1. searchKeyword - * is unaffected — modality filtering on the keyword path is independent. + * v0.27.1 / v0.36 (D11): target column for vector search. Two shapes: + * + * 1. String name (legacy + user-facing). Engine and hybridSearch convert + * to ResolvedColumn at the boundary via `resolveEmbeddingColumn()`. + * Built-in names: 'embedding' (default, text), 'embedding_image' + * (multimodal). Custom user-declared names also accepted when + * registered in `embedding_columns` config. + * + * 2. ResolvedColumn descriptor (internal). The engine ONLY accepts + * this shape — hybridSearch resolves once at entry and passes the + * descriptor in so the engine stays config-independent (D11 — engine + * is a pure SQL composer). + * + * The two-shape union is the transition seam. External callers + * (operations.ts image branch, tests, gbrain-evals) pass strings; + * hybridSearch resolves; engine sees descriptor. + * + * searchKeyword is unaffected — modality filtering on the keyword path + * is independent. */ - embeddingColumn?: 'embedding' | 'embedding_image'; + embeddingColumn?: 'embedding' | 'embedding_image' | string | ResolvedColumn; /** * @deprecated v0.29.1: use `since` instead. Removed in v0.30. * v0.27.0: filter results to pages updated/created after this date. ISO-8601 string. @@ -872,6 +931,19 @@ export interface EvalCandidateInput { remote: boolean; job_id: number | null; subagent_id: number | null; + /** + * v0.36 (D16 / CDX-10): the embedding column resolved at capture time. + * Optional for back-compat with pre-v0.36 callers + test fixtures. + * Engines coalesce undefined to NULL at insert time so the column is + * always either a known column name or DB NULL — never JS undefined. + * + * Replay (`gbrain eval replay`) re-runs the captured query against the + * brain. Without this field, a replay after the user flipped + * `search_embedding_column` produces "regressions" that are just + * column changes. Replay reads this and uses it as a per-row override + * so capture and replay run in the same embedding space. + */ + embedding_column?: string | null; } export interface EvalCandidate extends EvalCandidateInput { @@ -945,6 +1017,14 @@ export interface HybridSearchMeta { * the operator's `config.search.mode` setting if per-call overrides win). */ mode?: 'conservative' | 'balanced' | 'tokenmax'; + /** + * v0.36 (D16 / CDX-10): the embedding column that actually ran this + * search. Threaded through to eval_candidates capture so replay can + * use the same column even after `search_embedding_column` is + * flipped. Always set on hybridSearch paths; omitted on + * non-hybridSearch capture paths (keyword-only `search` op). + */ + embedding_column?: string; } // Config diff --git a/src/schema.sql b/src/schema.sql index a8ab448f6..80f973e98 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -787,7 +787,13 @@ CREATE TABLE IF NOT EXISTS eval_candidates ( salience_resolved TEXT, recency_resolved TEXT, salience_source TEXT, - recency_source TEXT + recency_source TEXT, + -- v0.36.3.0 (D16 / CDX-10) — embedding column resolved at capture time so + -- `gbrain eval replay` reproduces the same column the capture ran against. + -- Nullable; pre-v0.36 rows have NULL and replay falls back to current + -- default. Migration v68 (src/core/migrate.ts) adds the same column on + -- upgrade brains. + embedding_column TEXT ); CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); diff --git a/test/cosine-rescore-column.test.ts b/test/cosine-rescore-column.test.ts new file mode 100644 index 000000000..7e576ac50 --- /dev/null +++ b/test/cosine-rescore-column.test.ts @@ -0,0 +1,126 @@ +/** + * v0.36 (D9 / CDX-3) — getEmbeddingsByChunkIds column-parameter tests. + * + * Pins: + * - Default param value 'embedding' preserves pre-v0.36 behavior. + * - Custom column parameter (e.g. 'embedding_voyage') hydrates from + * that column. + * - Invalid column name (regex-failing) throws + * EmbeddingColumnNotRegisteredError BEFORE any SQL runs. + * - Identifier-quoting safely interpolates the column name. + * + * Uses PGLite in-memory engine. ALTER TABLE adds an ad-hoc + * `embedding_voyage` column to mimic the user's production state + * where columns get added outside the committed schema. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + EmbeddingColumnNotRegisteredError, +} from '../src/core/search/embedding-column.ts'; +import type { PageInput, ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +let chunkId: number; +let chunkId2: number; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + + // Add an ad-hoc voyage column at the same shape Garry's brain has — + // outside the committed schema, declared per-instance. + await (engine as any).db.exec( + `ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_voyage vector(1024)`, + ); + + // Seed a page with two chunks. + const page: PageInput = { + type: 'concept', + title: 'Cosine Rescore Test', + compiled_truth: 'Two chunks for rescore.', + }; + await engine.putPage('test/cosine-rescore', page); + + const chunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: 'first chunk text', chunk_source: 'compiled_truth' }, + { chunk_index: 1, chunk_text: 'second chunk text', chunk_source: 'compiled_truth' }, + ]; + await engine.upsertChunks('test/cosine-rescore', chunks); + + // Read back chunk ids. + const rows = await engine.executeRaw<{ id: number; chunk_index: number }>( + `SELECT cc.id, cc.chunk_index FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE p.slug = 'test/cosine-rescore' + ORDER BY cc.chunk_index`, + ); + chunkId = rows[0].id; + chunkId2 = rows[1].id; + + // Plant distinct vectors in 'embedding' (1536d) and 'embedding_voyage' + // (1024d) so we can prove the column parameter actually selects. + const v1536a = new Array(1536).fill(0).map(() => 0.001).join(','); + const v1536b = new Array(1536).fill(0).map(() => 0.002).join(','); + const v1024a = new Array(1024).fill(0).map(() => 0.5).join(','); + const v1024b = new Array(1024).fill(0).map(() => 0.6).join(','); + + await (engine as any).db.query( + `UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`, + [`[${v1536a}]`, chunkId], + ); + await (engine as any).db.query( + `UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`, + [`[${v1536b}]`, chunkId2], + ); + await (engine as any).db.query( + `UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`, + [`[${v1024a}]`, chunkId], + ); + await (engine as any).db.query( + `UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`, + [`[${v1024b}]`, chunkId2], + ); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('getEmbeddingsByChunkIds — column parameter (D9)', () => { + test('default param fetches from "embedding" — preserves pre-v0.36 behavior', async () => { + const map = await engine.getEmbeddingsByChunkIds([chunkId, chunkId2]); + expect(map.size).toBe(2); + expect(map.get(chunkId)!.length).toBe(1536); + expect(map.get(chunkId2)!.length).toBe(1536); + // First-element matches what we seeded for the primary column. + expect(map.get(chunkId)![0]).toBeCloseTo(0.001, 5); + }); + + test('column="embedding_voyage" fetches from the alt column', async () => { + const map = await engine.getEmbeddingsByChunkIds([chunkId, chunkId2], 'embedding_voyage'); + expect(map.size).toBe(2); + expect(map.get(chunkId)!.length).toBe(1024); + expect(map.get(chunkId2)!.length).toBe(1024); + // First-element matches what we seeded for voyage. + expect(map.get(chunkId)![0]).toBeCloseTo(0.5, 5); + }); + + test('invalid column param throws BEFORE SQL runs (regex-rejected)', async () => { + let threw: Error | null = null; + try { + await engine.getEmbeddingsByChunkIds([chunkId], 'embedding"; DROP --'); + } catch (e) { + threw = e as Error; + } + expect(threw).toBeTruthy(); + expect(threw).toBeInstanceOf(EmbeddingColumnNotRegisteredError); + }); + + test('empty id list short-circuits (no SQL run)', async () => { + const map = await engine.getEmbeddingsByChunkIds([], 'embedding_voyage'); + expect(map.size).toBe(0); + }); +}); diff --git a/test/e2e/embedding-column-pglite.test.ts b/test/e2e/embedding-column-pglite.test.ts new file mode 100644 index 000000000..86ba41693 --- /dev/null +++ b/test/e2e/embedding-column-pglite.test.ts @@ -0,0 +1,243 @@ +/** + * v0.36 E2E — dynamic embedding column selection (PGLite). + * + * Covers (per D4 + D9 + D11 + D12 + CDX-2 + CDX-3 + CDX-7 + CDX-8 + CDX-10): + * - Multi-column search: same query against `embedding` and against an + * ad-hoc `embedding_voyage` column produces different orderings + * consistent with the seeded vectors. + * - Halfvec column: ALTER TABLE ADD `embedding_ze halfvec(2560)` and + * confirm the `$1::halfvec(2560)` cast works. + * - Image branch unaffected: `embedding_image` still works via the + * existing operations.ts path. + * - cosineReScore reads from the active column, not the default + * (D9 — pre-fix, rescore against Voyage HNSW used OpenAI vectors). + * - Unknown column at hybridSearch entry throws loud. + * - Mid-session column switch invalidates the cache (knobs_hash v=3). + * + * No DATABASE_URL needed — PGLite in-memory. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { hybridSearch } from '../../src/core/search/hybrid.ts'; +import { + buildVectorCastFragment, + EmbeddingColumnNotRegisteredError, +} from '../../src/core/search/embedding-column.ts'; +import { + configureGateway, + resetGateway, + __setEmbedTransportForTests, +} from '../../src/core/ai/gateway.ts'; +import type { ResolvedColumn } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; +let chunkIdA: number; +let chunkIdB: number; + +const VEC1536_A = new Array(1536).fill(0).map((_, i) => 0.001 * (i % 10)); +const VEC1536_B = new Array(1536).fill(0).map((_, i) => 0.002 * (i % 10)); +const VEC1024_A = new Array(1024).fill(0).map((_, i) => 0.5 - 0.001 * (i % 10)); +const VEC1024_B = new Array(1024).fill(0).map((_, i) => 0.4 + 0.001 * (i % 10)); + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + + // Add the ad-hoc Voyage + ZE columns the way a user with a multi-provider + // brain has done it (outside the committed schema, per-instance ALTER). + await (engine as any).db.exec( + `ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_voyage vector(1024)`, + ); + await (engine as any).db.exec( + `ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_ze halfvec(2560)`, + ); + + // Two pages with one chunk each. + await engine.putPage('docs/page-a', { + type: 'concept', + title: 'Page A — about cats', + compiled_truth: 'Page A discusses cats and their behavior.', + }); + await engine.putPage('docs/page-b', { + type: 'concept', + title: 'Page B — about dogs', + compiled_truth: 'Page B discusses dogs and their habits.', + }); + + await engine.upsertChunks('docs/page-a', [ + { chunk_index: 0, chunk_text: 'cats behavior chunk A', chunk_source: 'compiled_truth' }, + ]); + await engine.upsertChunks('docs/page-b', [ + { chunk_index: 0, chunk_text: 'dogs habits chunk B', chunk_source: 'compiled_truth' }, + ]); + + // Look up chunk ids. + const rows = await engine.executeRaw<{ id: number; slug: string }>( + `SELECT cc.id, p.slug FROM content_chunks cc JOIN pages p ON p.id = cc.page_id ORDER BY p.slug`, + ); + chunkIdA = rows.find(r => r.slug === 'docs/page-a')!.id; + chunkIdB = rows.find(r => r.slug === 'docs/page-b')!.id; + + // Seed vectors. Vectors are intentionally distinct between columns so + // search orderings depend on which column the engine actually reads. + const vecLit = (arr: number[]) => `[${arr.join(',')}]`; + await (engine as any).db.query( + `UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`, + [vecLit(VEC1536_A), chunkIdA], + ); + await (engine as any).db.query( + `UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`, + [vecLit(VEC1536_B), chunkIdB], + ); + await (engine as any).db.query( + `UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`, + [vecLit(VEC1024_A), chunkIdA], + ); + await (engine as any).db.query( + `UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`, + [vecLit(VEC1024_B), chunkIdB], + ); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); + __setEmbedTransportForTests(null); + resetGateway(); +}); + +describe('PGLite engine: searchVector accepts ResolvedColumn descriptor (D11)', () => { + test('vector cast routes to correct column when descriptor names embedding_voyage', async () => { + const queryVec = new Float32Array(VEC1024_A); + const descriptor: ResolvedColumn = { + name: 'embedding_voyage', + type: 'vector', + dimensions: 1024, + embeddingModel: 'voyage:voyage-3-large', + }; + const results = await engine.searchVector(queryVec, { + embeddingColumn: descriptor, + limit: 5, + }); + // Both pages have voyage embeddings; cosine to VEC1024_A is closer to + // page-a (identical) than page-b. Verify ordering. + expect(results.length).toBeGreaterThanOrEqual(1); + expect(results[0].slug).toBe('docs/page-a'); + }); + + test('halfvec cast accepted: ALTER TABLE column + $1::halfvec(N)', async () => { + // Seed halfvec values via direct cast. + const ze1 = `[${new Array(2560).fill(0.5).join(',')}]`; + const ze2 = `[${new Array(2560).fill(0.6).join(',')}]`; + await (engine as any).db.query( + `UPDATE content_chunks SET embedding_ze = $1::halfvec WHERE id = $2`, + [ze1, chunkIdA], + ); + await (engine as any).db.query( + `UPDATE content_chunks SET embedding_ze = $1::halfvec WHERE id = $2`, + [ze2, chunkIdB], + ); + + const queryVec = new Float32Array(2560).fill(0.5); + const descriptor: ResolvedColumn = { + name: 'embedding_ze', + type: 'halfvec', + dimensions: 2560, + embeddingModel: 'zeroentropyai:zembed-1', + }; + const results = await engine.searchVector(queryVec, { + embeddingColumn: descriptor, + limit: 5, + }); + expect(results.length).toBeGreaterThanOrEqual(1); + // Page A's halfvec is closer to the all-0.5 query. + expect(results[0].slug).toBe('docs/page-a'); + }); + + test('legacy embedding_image literal still routes correctly', async () => { + // We never seeded embedding_image so we expect zero results, but the + // query MUST NOT throw — the legacy-literal path must still work + // (no regression on the existing image branch). + const v = new Float32Array(1024).fill(0.1); + const results = await engine.searchVector(v, { + embeddingColumn: 'embedding_image', + limit: 5, + }); + expect(Array.isArray(results)).toBe(true); + }); +}); + +describe('PGLite engine: getEmbeddingsByChunkIds column param (D9)', () => { + test('default fetches from embedding (back-compat)', async () => { + const map = await engine.getEmbeddingsByChunkIds([chunkIdA, chunkIdB]); + expect(map.get(chunkIdA)!.length).toBe(1536); + }); + + test('column="embedding_voyage" fetches from voyage column', async () => { + const map = await engine.getEmbeddingsByChunkIds([chunkIdA, chunkIdB], 'embedding_voyage'); + expect(map.get(chunkIdA)!.length).toBe(1024); + }); + + test('invalid column rejected at engine layer (regex guard)', async () => { + let threw: Error | null = null; + try { + await engine.getEmbeddingsByChunkIds([chunkIdA], 'embed-bad-name'); + } catch (e) { + threw = e as Error; + } + expect(threw).toBeInstanceOf(EmbeddingColumnNotRegisteredError); + }); +}); + +describe('hybridSearch + resolver — unknown column at entry (D11)', () => { + test('unknown name in opts.embeddingColumn throws via resolver', async () => { + // configureGateway with a transport stub so we don't hit a real API. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test' }, + }); + __setEmbedTransportForTests(async () => ({ + embeddings: [new Array(1536).fill(0)], + usage: { tokens: 0 }, + } as any)); + + let threw: Error | null = null; + try { + await hybridSearch(engine, 'cats', { + embeddingColumn: 'nonexistent_column', + limit: 5, + }); + } catch (e) { + threw = e as Error; + } + expect(threw).toBeInstanceOf(EmbeddingColumnNotRegisteredError); + }); +}); + +describe('buildVectorCastFragment — engine SQL composer (D3)', () => { + test('vector descriptor emits $1::vector', () => { + const r: ResolvedColumn = { + name: 'embedding', + type: 'vector', + dimensions: 1536, + embeddingModel: '', + }; + const { col, castSql } = buildVectorCastFragment(r); + expect(col).toBe('"embedding"'); + expect(castSql).toBe('$1::vector'); + }); + + test('halfvec descriptor emits $1::halfvec(N) with parenthesized N', () => { + const r: ResolvedColumn = { + name: 'embedding_ze', + type: 'halfvec', + dimensions: 2560, + embeddingModel: 'zeroentropyai:zembed-1', + }; + const { col, castSql } = buildVectorCastFragment(r); + expect(col).toBe('"embedding_ze"'); + expect(castSql).toBe('$1::halfvec(2560)'); + }); +}); diff --git a/test/e2e/embedding-column-postgres.test.ts b/test/e2e/embedding-column-postgres.test.ts new file mode 100644 index 000000000..849a2df17 --- /dev/null +++ b/test/e2e/embedding-column-postgres.test.ts @@ -0,0 +1,227 @@ +/** + * v0.36 E2E (real Postgres) — halfvec + HNSW + doctor SQL probes. + * + * Skipped gracefully when DATABASE_URL is unset. When present, exercises + * the real pgvector + Postgres code paths that PGLite can't fully cover: + * + * - halfvec(2560) cast accepted by real pgvector via searchVector. + * - HNSW index on the alternative column is visible in EXPLAIN. + * - The format_type SQL the doctor check uses (D13) correctly distinguishes + * `vector(1024)` vs `vector(1536)` so dim drift can be detected. + * - The coverage SQL the doctor check uses (D14) computes accurate + * percentage and the < 90% gate fires when expected. + * + * Tests the SQL the doctor check issues, not `runDoctor` itself (which + * calls process.exit). This is the regression-catching layer that + * matters; the doctor wrapping just renders the result. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PostgresEngine } from '../../src/core/postgres-engine.ts'; +import { quoteIdentifier } from '../../src/core/search/embedding-column.ts'; +import type { ResolvedColumn } from '../../src/core/types.ts'; + +const dbUrl = process.env.DATABASE_URL; +if (!dbUrl) { + describe.skip('postgres E2E — embedding column (skipped: DATABASE_URL unset)', () => { + test('skipped', () => { expect(true).toBe(true); }); + }); +} else { + let engine: PostgresEngine; + let catId: number; + let dogId: number; + + beforeAll(async () => { + engine = new PostgresEngine(); + await engine.connect({ database_url: dbUrl } as never); + await engine.initSchema(); + + // Wipe state from prior runs. + await engine.executeRaw(`DELETE FROM content_chunks`); + await engine.executeRaw(`DELETE FROM pages WHERE slug LIKE 'docs/%'`); + + // Add the ad-hoc Voyage + ZE columns + HNSW indexes. + await engine.executeRaw( + `ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_voyage vector(1024)`, + ); + await engine.executeRaw( + `ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_ze halfvec(2560)`, + ); + await engine.executeRaw( + `CREATE INDEX IF NOT EXISTS idx_chunks_embedding_voyage + ON content_chunks USING hnsw (embedding_voyage vector_cosine_ops)`, + ); + + // Seed two pages and two chunks. + await engine.putPage('docs/cat', { + type: 'concept', + title: 'Cat doc', + compiled_truth: 'Cat doc compiled truth.', + }); + await engine.putPage('docs/dog', { + type: 'concept', + title: 'Dog doc', + compiled_truth: 'Dog doc compiled truth.', + }); + await engine.upsertChunks('docs/cat', [ + { chunk_index: 0, chunk_text: 'cat chunk', chunk_source: 'compiled_truth' }, + ]); + await engine.upsertChunks('docs/dog', [ + { chunk_index: 0, chunk_text: 'dog chunk', chunk_source: 'compiled_truth' }, + ]); + + const idRows = await engine.executeRaw<{ id: number; slug: string }>( + `SELECT cc.id, p.slug FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.slug LIKE 'docs/%' ORDER BY p.slug`, + ); + catId = idRows.find(r => r.slug === 'docs/cat')!.id; + dogId = idRows.find(r => r.slug === 'docs/dog')!.id; + + const vec1024 = (v: number) => `[${new Array(1024).fill(v).join(',')}]`; + const vec2560 = (v: number) => `[${new Array(2560).fill(v).join(',')}]`; + await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = '${vec1024(0.5)}'::vector WHERE id = ${catId}`); + await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = '${vec1024(0.7)}'::vector WHERE id = ${dogId}`); + await engine.executeRaw(`UPDATE content_chunks SET embedding_ze = '${vec2560(0.5)}'::halfvec WHERE id = ${catId}`); + await engine.executeRaw(`UPDATE content_chunks SET embedding_ze = '${vec2560(0.7)}'::halfvec WHERE id = ${dogId}`); + }); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }); + + describe('Postgres: searchVector with halfvec descriptor', () => { + test('halfvec(2560) cast accepted; results returned in expected cosine order', async () => { + const queryVec = new Float32Array(2560).fill(0.5); + const descriptor: ResolvedColumn = { + name: 'embedding_ze', + type: 'halfvec', + dimensions: 2560, + embeddingModel: 'zeroentropyai:zembed-1', + }; + const results = await engine.searchVector(queryVec, { + embeddingColumn: descriptor, + limit: 5, + }); + expect(results.length).toBeGreaterThanOrEqual(1); + // Cat's halfvec is identical to the query — closer than dog. + expect(results[0].slug).toBe('docs/cat'); + }); + + test('vector(1024) cast on embedding_voyage routes correctly', async () => { + const queryVec = new Float32Array(1024).fill(0.5); + const descriptor: ResolvedColumn = { + name: 'embedding_voyage', + type: 'vector', + dimensions: 1024, + embeddingModel: 'voyage:voyage-3-large', + }; + const results = await engine.searchVector(queryVec, { + embeddingColumn: descriptor, + limit: 5, + }); + expect(results.length).toBeGreaterThanOrEqual(1); + expect(results[0].slug).toBe('docs/cat'); + }); + }); + + describe('Postgres: HNSW index visible to planner', () => { + test('pg_indexes shows the hnsw index on embedding_voyage', async () => { + const rows = await engine.executeRaw<{ indexname: string; indexdef: string }>( + `SELECT indexname, indexdef FROM pg_indexes + WHERE tablename = 'content_chunks' + AND schemaname = 'public'`, + ); + const voyageHnsw = rows.find(r => + r.indexname === 'idx_chunks_embedding_voyage' && /USING\s+hnsw/i.test(r.indexdef), + ); + expect(voyageHnsw).toBeDefined(); + expect(voyageHnsw!.indexdef).toMatch(/embedding_voyage/); + }); + }); + + describe('Postgres: doctor SQL probes for dim drift (D13)', () => { + test('format_type returns parenthesized dim — distinguishes vector(1024) from vector(1536)', async () => { + // This is the exact SQL shape doctor's embedding_column_registry + // check uses. If pg_attribute or format_type semantics ever change, + // this test fails loud. + const rows = await engine.executeRaw<{ attname: string; formatted: string }>( + `SELECT a.attname, format_type(a.atttypid, a.atttypmod) AS formatted + FROM pg_attribute a + JOIN pg_class c ON c.oid = a.attrelid + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'public' + AND c.relname = 'content_chunks' + AND a.attname = ANY($1::text[]) + AND NOT a.attisdropped`, + [['embedding', 'embedding_voyage', 'embedding_ze']], + ); + const byName = new Map(); + for (const r of rows) byName.set(r.attname, r.formatted); + + // Default 'embedding' is vector(1536) by committed schema. + expect(byName.get('embedding')).toMatch(/^vector\(\d+\)/); + // Voyage is vector(1024) per the ALTER above. + expect(byName.get('embedding_voyage')).toBe('vector(1024)'); + // ZE is halfvec(2560) per the ALTER above. + expect(byName.get('embedding_ze')).toBe('halfvec(2560)'); + }); + + test('format_type catches dim drift: declared 1536 vs actual 1024', async () => { + // Simulate the user declaring 1536 in their registry but the column + // is actually 1024d. Doctor's regex-parse of the formatted string + // is the layer that catches this. + const rows = await engine.executeRaw<{ formatted: string }>( + `SELECT format_type(atttypid, atttypmod) AS formatted + FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass + AND attname = 'embedding_voyage' + AND NOT attisdropped`, + ); + const formatted = rows[0]?.formatted ?? ''; + const m = formatted.match(/^(vector|halfvec)\((\d+)\)$/); + expect(m).toBeTruthy(); + const declaredDims = 1536; + const actualDims = parseInt(m![2], 10); + expect(actualDims).not.toBe(declaredDims); // The drift is detectable. + expect(actualDims).toBe(1024); + }); + }); + + describe('Postgres: doctor SQL probe for coverage (D14)', () => { + test('coverage % computed correctly on fully-populated column', async () => { + const col = quoteIdentifier('embedding_voyage'); + const rows = await engine.executeRaw<{ pct: number; total: number }>( + `SELECT ( + COUNT(*) FILTER (WHERE ${col} IS NOT NULL)::float + / NULLIF(COUNT(*), 0) * 100 + )::float AS pct, + COUNT(*)::int AS total + FROM content_chunks`, + ); + expect(rows[0].total).toBe(2); + expect(rows[0].pct).toBe(100); + }); + + test('coverage % drops to 50 after a partial wipe; gate at < 90 fires', async () => { + // Clear voyage on one chunk to simulate partial backfill. + await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = NULL WHERE id = ${dogId}`); + + const col = quoteIdentifier('embedding_voyage'); + const rows = await engine.executeRaw<{ pct: number; total: number }>( + `SELECT ( + COUNT(*) FILTER (WHERE ${col} IS NOT NULL)::float + / NULLIF(COUNT(*), 0) * 100 + )::float AS pct, + COUNT(*)::int AS total + FROM content_chunks`, + ); + expect(rows[0].total).toBe(2); + expect(rows[0].pct).toBe(50); + // The < 90 gate that doctor + config-set use should fire. + expect(rows[0].pct < 90).toBe(true); + + // Restore so subsequent tests see the original fixture. + const v = `[${new Array(1024).fill(0.7).join(',')}]`; + await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = '${v}'::vector WHERE id = ${dogId}`); + }); + }); +} diff --git a/test/e2e/eval-replay-column.test.ts b/test/e2e/eval-replay-column.test.ts new file mode 100644 index 000000000..368e102a1 --- /dev/null +++ b/test/e2e/eval-replay-column.test.ts @@ -0,0 +1,115 @@ +/** + * v0.36 (D16 / CDX-10) — eval_candidates.embedding_column round-trip. + * + * Pins: + * - logEvalCandidate persists `embedding_column` when set. + * - listEvalCandidates reads it back (SELECT * carries the new column). + * - Migration v67 applied: ALTER TABLE eval_candidates ADD COLUMN + * IF NOT EXISTS embedding_column TEXT. + * - Back-compat: rows inserted without embedding_column store NULL, + * and replay treats NULL as "use current default." + * + * The integration of replay with hybridSearch's column-override path is + * tested via the embeddingColumn option being honored — we don't run + * the full eval-replay CLI subcommand here because that brings in CLI + * arg parsing + filesystem reading. Unit-level surface is sufficient + * for the persist-and-read contract. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import type { EvalCandidateInput } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +const baseRow: Omit = { + tool_name: 'query', + query: 'test query', + retrieved_slugs: ['docs/a', 'docs/b'], + retrieved_chunk_ids: [1, 2], + source_ids: ['default'], + expand_enabled: true, + detail: null, + detail_resolved: 'medium', + vector_enabled: true, + expansion_applied: false, + latency_ms: 42, + remote: false, + job_id: null, + subagent_id: null, +}; + +describe('eval_candidates.embedding_column persistence (D16)', () => { + test('migration v67 added embedding_column TEXT column', async () => { + const rows = await engine.executeRaw<{ column_name: string; data_type: string; is_nullable: string }>( + `SELECT column_name, data_type, is_nullable + FROM information_schema.columns + WHERE table_name = 'eval_candidates' AND column_name = 'embedding_column'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].column_name).toBe('embedding_column'); + expect(rows[0].data_type).toBe('text'); + expect(rows[0].is_nullable).toBe('YES'); + }); + + test('logEvalCandidate stores embedding_column value', async () => { + const id = await engine.logEvalCandidate({ + ...baseRow, + embedding_column: 'embedding_voyage', + }); + const rows = await engine.executeRaw<{ embedding_column: string | null }>( + `SELECT embedding_column FROM eval_candidates WHERE id = $1`, + [id], + ); + expect(rows[0].embedding_column).toBe('embedding_voyage'); + }); + + test('listEvalCandidates reads embedding_column back (round-trip via SELECT *)', async () => { + const id = await engine.logEvalCandidate({ + ...baseRow, + query: 'list-readback test', + embedding_column: 'embedding_ze', + }); + const list = await engine.listEvalCandidates({ limit: 100 }); + const found = list.find(r => r.id === id); + expect(found).toBeDefined(); + // Cast-through-unknown rowToEvalCandidate doesn't exist; SELECT * + // carries `embedding_column` as a column with the same key name. + expect((found as any).embedding_column).toBe('embedding_ze'); + }); + + test('back-compat: missing embedding_column coalesces to NULL at insert', async () => { + // Build a row WITHOUT embedding_column to mimic pre-v0.36 callers. + const input: EvalCandidateInput = { ...baseRow, query: 'pre-v036 fixture' }; + expect(input.embedding_column).toBeUndefined(); + const id = await engine.logEvalCandidate(input); + const rows = await engine.executeRaw<{ embedding_column: string | null }>( + `SELECT embedding_column FROM eval_candidates WHERE id = $1`, + [id], + ); + expect(rows[0].embedding_column).toBeNull(); + }); + + test('back-compat: explicit null persists as DB NULL (not the literal string "null")', async () => { + const id = await engine.logEvalCandidate({ + ...baseRow, + query: 'explicit-null fixture', + embedding_column: null, + }); + const rows = await engine.executeRaw<{ embedding_column: string | null }>( + `SELECT embedding_column FROM eval_candidates WHERE id = $1`, + [id], + ); + expect(rows[0].embedding_column).toBeNull(); + }); +}); diff --git a/test/gateway-embed-model-override.test.ts b/test/gateway-embed-model-override.test.ts new file mode 100644 index 000000000..9f008e6c3 --- /dev/null +++ b/test/gateway-embed-model-override.test.ts @@ -0,0 +1,176 @@ +/** + * v0.36 (D10) — gateway embed model override path. + * + * Pins: + * - embedQuery(text, { embeddingModel }) routes through THAT provider, + * not the global default. + * - embedQuery(text, { dimensions }) flows into dimsProviderOptions + * so providers that accept output_dimension see the override. + * - Bare embedQuery(text) continues to use the configured default. + * - Unknown override model throws (resolveEmbeddingProvider's + * AIConfigError shape with a hint). + * - isAvailable('embedding', modelOverride) probes the override's + * recipe, not the global default's. + */ + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { + configureGateway, + embedQuery, + isAvailable, + resetGateway, + __setEmbedTransportForTests, +} from '../src/core/ai/gateway.ts'; + +interface TransportCall { + modelString: string; + values: string[]; + providerOptions: Record; +} + +const calls: TransportCall[] = []; + +function installCaptureTransport(makeVector: (dims: number) => number[]) { + __setEmbedTransportForTests(async ({ model, values, providerOptions }: any) => { + // The AI SDK's `model` object exposes `.modelId` (string) on every + // openai-compatible model. We capture it so tests can assert the + // gateway routed to the correct provider:model. + const modelString = (model?.modelId ?? '') as string; + calls.push({ modelString, values: [...values], providerOptions: { ...(providerOptions ?? {}) } }); + // Pick the dim from providerOpts when present (Voyage flexible-dim + // path emits openaiCompatible.dimensions); otherwise default 1536. + const oc = (providerOptions?.openaiCompatible ?? {}) as Record; + const dims = typeof oc.dimensions === 'number' ? (oc.dimensions as number) : 1536; + return { + embeddings: values.map(() => makeVector(dims)), + usage: { tokens: 0 }, + } as any; + }); +} + +beforeEach(() => { + calls.length = 0; + resetGateway(); +}); + +afterEach(() => { + __setEmbedTransportForTests(null); + resetGateway(); +}); + +describe('embedQuery — bare (no opts)', () => { + test('bare call uses the globally configured embedding_model', async () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test' }, + }); + installCaptureTransport(d => new Array(d).fill(0).map((_, i) => i * 0.001)); + + const v = await embedQuery('hello'); + expect(v.length).toBe(1536); + expect(calls.length).toBe(1); + expect(calls[0].modelString).toBe('text-embedding-3-large'); + }); +}); + +describe('embedQuery — { embeddingModel } override', () => { + test('routes through the override provider, not the global default', async () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test', VOYAGE_API_KEY: 'voy-test' }, + }); + installCaptureTransport(d => new Array(d).fill(0).map((_, i) => i * 0.002)); + + const v = await embedQuery('hello', { + embeddingModel: 'voyage:voyage-3-large', + dimensions: 1024, + }); + expect(v.length).toBe(1024); + expect(calls.length).toBe(1); + expect(calls[0].modelString).toBe('voyage-3-large'); + }); + + test('dimensions override flows into providerOptions', async () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test', VOYAGE_API_KEY: 'voy-test' }, + }); + installCaptureTransport(d => new Array(d).fill(0).map(() => 0.1)); + + await embedQuery('hello', { embeddingModel: 'voyage:voyage-3-large', dimensions: 2048 }); + const opts = calls[0].providerOptions as { openaiCompatible?: Record }; + // Voyage flexible-dim models emit `dimensions` into the openaiCompatible + // providerOptions block; the shim translates to output_dimension on + // the wire. Either way, the gateway honored the caller's dim override. + expect(opts.openaiCompatible).toBeDefined(); + expect(opts.openaiCompatible!.dimensions).toBe(2048); + }); + + test('unknown override model throws AIConfigError with a useful hint', async () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test' }, + }); + installCaptureTransport(d => new Array(d).fill(0)); + + let threw: Error | null = null; + try { + await embedQuery('hello', { embeddingModel: 'nonexistent:bogus' }); + } catch (e) { + threw = e as Error; + } + expect(threw).toBeTruthy(); + // Error message names the provider or model so the user knows what failed. + expect(threw!.message.toLowerCase()).toMatch(/nonexistent|provider|recipe|model/); + }); +}); + +describe('isAvailable(touchpoint, modelOverride) — D10', () => { + test('global default available + Voyage override key present → both available', () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test', VOYAGE_API_KEY: 'voy-test' }, + }); + expect(isAvailable('embedding')).toBe(true); + expect(isAvailable('embedding', 'voyage:voyage-3-large')).toBe(true); + }); + + test('global default key missing but override key present → override is available', () => { + // The single-OPENAI scenario: user removed OPENAI_API_KEY but + // configured Voyage for the alt column. Pre-D10, isAvailable would + // have said embedding is unavailable globally and hybridSearch + // would skip vector search ENTIRELY. With the override, hybrid asks + // about the active column's provider and gets a green light. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { VOYAGE_API_KEY: 'voy-test' }, // no OPENAI_API_KEY + }); + expect(isAvailable('embedding')).toBe(false); + expect(isAvailable('embedding', 'voyage:voyage-3-large')).toBe(true); + }); + + test('global default available but override key missing → override is unavailable', () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test' }, // no VOYAGE_API_KEY + }); + expect(isAvailable('embedding')).toBe(true); + expect(isAvailable('embedding', 'voyage:voyage-3-large')).toBe(false); + }); + + test('override against an unknown model returns false', () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test' }, + }); + expect(isAvailable('embedding', 'totally:unknown')).toBe(false); + }); +}); diff --git a/test/loadConfig-merge.test.ts b/test/loadConfig-merge.test.ts index 6af6e649e..7056efd9e 100644 --- a/test/loadConfig-merge.test.ts +++ b/test/loadConfig-merge.test.ts @@ -20,9 +20,31 @@ function makeEngine(map: Record): FakeEngine } describe('loadConfigWithEngine (Phase 4 / F3)', () => { - test('returns null when base config is null', async () => { + test('synthesizes a minimal base when base config is null (v0.36 codex /ship #3)', async () => { + // Pre-v0.36 this returned null and skipped DB-plane merge entirely. + // That meant env-only Postgres installs (no file config) couldn't see + // DB-plane overrides set via `gbrain config set` — the documented + // smoke test for `search_embedding_column` would silently fail. + // The fix synthesizes a minimal `{ engine: 'postgres' }` base so DB + // merge still runs; downstream callers either find the DB key or + // fall through to defaults. const result = await loadConfigWithEngine(makeEngine({}), null); - expect(result).toBeNull(); + expect(result).not.toBeNull(); + expect(result?.engine).toBe('postgres'); + }); + + test('DB-plane embedding_columns merge works even with null base (codex /ship #3 round-trip)', async () => { + // The whole point of the synthesized fallback: env-only installs + // calling `gbrain config set embedding_columns '...'` get those keys + // back when the resolver re-reads config. Verifies the merge path + // actually runs (not just that the function returns truthy). + const engine = makeEngine({ + search_embedding_column: 'embedding_voyage', + embedding_columns: '{"embedding_voyage":{"provider":"voyage:voyage-3-large","dimensions":1024,"type":"vector"}}', + }); + const merged = await loadConfigWithEngine(engine, null); + expect(merged?.search_embedding_column).toBe('embedding_voyage'); + expect(merged?.embedding_columns?.embedding_voyage?.dimensions).toBe(1024); }); test('DB flag fills in when file/env did not set it', async () => { diff --git a/test/operations-embedding-column.test.ts b/test/operations-embedding-column.test.ts new file mode 100644 index 000000000..4b54358c7 --- /dev/null +++ b/test/operations-embedding-column.test.ts @@ -0,0 +1,54 @@ +/** + * v0.36 (D15 / CDX-9) — MCP op `embedding_column` param surface. + * + * Pins: + * - `query` op declares `embedding_column` in its params allowlist + * (caller can pass it). + * - `search` op does NOT declare it (CDX-9: search is keyword-only; + * adding the field would silently change semantics). + * - Param description names the registry + the override semantics so + * agents discover it via tool definitions. + */ + +import { describe, test, expect } from 'bun:test'; +import { operationsByName } from '../src/core/operations.ts'; + +describe('query op — embedding_column param (D15)', () => { + const queryOp = operationsByName.query; + + test('exists', () => { + expect(queryOp).toBeDefined(); + }); + + test('declares embedding_column in params allowlist', () => { + expect(queryOp.params.embedding_column).toBeDefined(); + expect(queryOp.params.embedding_column.type).toBe('string'); + }); + + test('description names registry + override semantics', () => { + const desc = queryOp.params.embedding_column.description ?? ''; + expect(desc.toLowerCase()).toMatch(/embedding/); + // Description should give the agent enough context to understand + // when to use the param and where the registry lives. + expect(desc.toLowerCase()).toMatch(/registry|embedding_columns|column/); + }); + + test('embedding_column is NOT required (per-call override is optional)', () => { + expect(queryOp.params.embedding_column.required).not.toBe(true); + }); +}); + +describe('search op — does NOT declare embedding_column (CDX-9)', () => { + const searchOp = operationsByName.search; + + test('exists', () => { + expect(searchOp).toBeDefined(); + }); + + test('does NOT include embedding_column in params (search is keyword-only)', () => { + // Adding embedding_column to the keyword-only `search` op would + // either be silently ignored (footgun for agents) or change the op's + // semantics from keyword to hybrid. Both bad. Keep it on `query` only. + expect(searchOp.params.embedding_column).toBeUndefined(); + }); +}); diff --git a/test/search-mode.test.ts b/test/search-mode.test.ts index eb2e887ce..5aa33aef5 100644 --- a/test/search-mode.test.ts +++ b/test/search-mode.test.ts @@ -272,8 +272,11 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => { test('KNOBS_HASH_VERSION constant exposed for migrations to bump on schema change', () => { // v0.35.0.0+ bumped 1→2 to fold reranker fields into the cache key. - // v0.35.6.0 bumped 2→3 to fold floor_ratio into the cache key - // (codex outside-voice T1 — preventing cross-floor cache contamination). + // v0.35.6.0 bumped 2→3 to fold floor_ratio into the cache key + // (codex T1 — preventing cross-floor cache contamination). + // v0.36 also extends v=3 with embedding column + provider (D8 / CDX-2) + // so a query against `embedding_voyage` never shares a cache row with + // `embedding`, even when all other knobs match. expect(KNOBS_HASH_VERSION).toBe(3); }); diff --git a/test/search/embedding-column.test.ts b/test/search/embedding-column.test.ts new file mode 100644 index 000000000..36ed2077c --- /dev/null +++ b/test/search/embedding-column.test.ts @@ -0,0 +1,511 @@ +/** + * v0.36 — embedding column resolver tests. + * + * Pins: + * - D2/D11: resolver returns descriptor (name, type, dimensions, + * embeddingModel). + * - D3: buildVectorCastFragment produces correct cast string per type. + * - D11: builtins (`embedding`, `embedding_image`) always present. + * - D12: registry-key regex + field validation reject malicious input. + * - D12: identifier-quoting handles embedded quotes safely. + * - Resolution chain: opts > cfg.search_embedding_column > 'embedding'. + * - normalizeEngineColumn: descriptor-passthrough + legacy literals + + * throw on unknown string. + */ + +import { describe, test, expect } from 'bun:test'; +import { + resolveEmbeddingColumn, + getEmbeddingColumnRegistry, + buildVectorCastFragment, + quoteIdentifier, + validateColumnKey, + validateColumnConfig, + normalizeEngineColumn, + EmbeddingColumnNotRegisteredError, + EmbeddingColumnConfigError, + COLUMN_NAME_REGEX, + ALLOWED_COLUMN_TYPES, + MAX_DIMENSIONS, + DEFAULT_COLUMN_NAME, + isDefaultColumn, + isCacheSafe, + isBuiltinColumn, +} from '../../src/core/search/embedding-column.ts'; +import type { GBrainConfig } from '../../src/core/config.ts'; +import type { ResolvedColumn } from '../../src/core/types.ts'; + +function cfg(overrides: Partial = {}): GBrainConfig { + return { engine: 'pglite', ...overrides }; +} + +describe('resolveEmbeddingColumn — resolution chain', () => { + test('default fallback returns "embedding"', () => { + const r = resolveEmbeddingColumn(undefined, cfg()); + expect(r.name).toBe('embedding'); + expect(r.type).toBe('vector'); + }); + + test('cfg.search_embedding_column wins over default', () => { + const r = resolveEmbeddingColumn(undefined, cfg({ + search_embedding_column: 'embedding_voyage', + embedding_columns: { + embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' }, + }, + })); + expect(r.name).toBe('embedding_voyage'); + expect(r.embeddingModel).toBe('voyage:voyage-3-large'); + expect(r.dimensions).toBe(1024); + }); + + test('opts.embeddingColumn wins over cfg.search_embedding_column', () => { + const r = resolveEmbeddingColumn( + { embeddingColumn: 'embedding_voyage' }, + cfg({ + search_embedding_column: 'embedding_zeroentropy', + embedding_columns: { + embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' }, + embedding_zeroentropy: { provider: 'zeroentropyai:zembed-1', dimensions: 2560, type: 'halfvec' }, + }, + }), + ); + expect(r.name).toBe('embedding_voyage'); + }); + + test('unknown name throws EmbeddingColumnNotRegisteredError with hint', () => { + let err: EmbeddingColumnNotRegisteredError | null = null; + try { + resolveEmbeddingColumn({ embeddingColumn: 'nonexistent' }, cfg()); + } catch (e) { + err = e as EmbeddingColumnNotRegisteredError; + } + expect(err).toBeTruthy(); + expect(err?.code).toBe('embedding_column_not_registered'); + expect(err?.columnName).toBe('nonexistent'); + expect(err?.validColumns).toEqual(['embedding', 'embedding_image']); + expect(err?.message).toContain('Declared columns:'); + expect(err?.message).toContain('gbrain config set'); + }); + + test('SQL-injection-shaped name rejected before registry lookup', () => { + expect(() => + resolveEmbeddingColumn( + { embeddingColumn: 'embedding"; DROP TABLE pages; --' }, + cfg(), + ), + ).toThrow(EmbeddingColumnNotRegisteredError); + }); + + test('descriptor passthrough: ResolvedColumn returned as-is', () => { + const descriptor: ResolvedColumn = { + name: 'embedding_custom', + type: 'halfvec', + dimensions: 2560, + embeddingModel: 'zeroentropyai:zembed-1', + }; + const r = resolveEmbeddingColumn({ embeddingColumn: descriptor }, cfg()); + expect(r).toEqual(descriptor); + }); +}); + +describe('getEmbeddingColumnRegistry — builtins + merge', () => { + test('builtin embedding always present even with empty user config', () => { + const reg = getEmbeddingColumnRegistry(cfg()); + expect(reg.embedding).toBeDefined(); + expect(reg.embedding!.type).toBe('vector'); + expect(reg.embedding!.dimensions).toBe(1536); + }); + + test('builtin embedding_image always present with 1024d vector', () => { + const reg = getEmbeddingColumnRegistry(cfg()); + expect(reg.embedding_image).toBeDefined(); + expect(reg.embedding_image!.type).toBe('vector'); + expect(reg.embedding_image!.dimensions).toBe(1024); + }); + + test('builtin embedding derives provider from cfg.embedding_model', () => { + const reg = getEmbeddingColumnRegistry( + cfg({ embedding_model: 'voyage:voyage-3-large', embedding_dimensions: 1024 }), + ); + expect(reg.embedding!.provider).toBe('voyage:voyage-3-large'); + expect(reg.embedding!.dimensions).toBe(1024); + }); + + test('builtin embedding_image derives provider from cfg.embedding_multimodal_model', () => { + const reg = getEmbeddingColumnRegistry( + cfg({ embedding_multimodal_model: 'voyage:voyage-multimodal-3' }), + ); + expect(reg.embedding_image!.provider).toBe('voyage:voyage-multimodal-3'); + }); + + test('user-declared columns merge with builtins', () => { + const reg = getEmbeddingColumnRegistry( + cfg({ + embedding_columns: { + embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' }, + }, + }), + ); + expect(Object.keys(reg).sort()).toEqual(['embedding', 'embedding_image', 'embedding_voyage']); + }); + + test('user override wins on conflict (override embedding builtin)', () => { + const reg = getEmbeddingColumnRegistry( + cfg({ + embedding_columns: { + embedding: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' }, + }, + }), + ); + expect(reg.embedding!.provider).toBe('voyage:voyage-3-large'); + expect(reg.embedding!.dimensions).toBe(1024); + }); + + test('halfvec column with high dim accepted', () => { + const reg = getEmbeddingColumnRegistry( + cfg({ + embedding_columns: { + embedding_ze: { provider: 'zeroentropyai:zembed-1', dimensions: 2560, type: 'halfvec' }, + }, + }), + ); + expect(reg.embedding_ze!.type).toBe('halfvec'); + expect(reg.embedding_ze!.dimensions).toBe(2560); + }); +}); + +describe('D12 — defense-in-depth validation', () => { + describe('validateColumnKey', () => { + test('accepts lowercase identifier', () => { + expect(() => validateColumnKey('embedding_voyage')).not.toThrow(); + expect(() => validateColumnKey('a')).not.toThrow(); + expect(() => validateColumnKey('_underscore_first')).not.toThrow(); + expect(() => validateColumnKey('mix_of_letters_and_123')).not.toThrow(); + }); + + test('rejects keys with quotes (SQL injection vector)', () => { + expect(() => validateColumnKey('embedding"; DROP --')).toThrow(EmbeddingColumnConfigError); + expect(() => validateColumnKey("embedding'")).toThrow(EmbeddingColumnConfigError); + }); + + test('rejects keys with uppercase', () => { + expect(() => validateColumnKey('Embedding')).toThrow(EmbeddingColumnConfigError); + expect(() => validateColumnKey('EMBEDDING_VOYAGE')).toThrow(EmbeddingColumnConfigError); + }); + + test('rejects keys starting with digits', () => { + expect(() => validateColumnKey('1embedding')).toThrow(EmbeddingColumnConfigError); + }); + + test('rejects keys with hyphens, spaces, special chars', () => { + expect(() => validateColumnKey('embed-voyage')).toThrow(EmbeddingColumnConfigError); + expect(() => validateColumnKey('embed voyage')).toThrow(EmbeddingColumnConfigError); + expect(() => validateColumnKey('embed.voyage')).toThrow(EmbeddingColumnConfigError); + }); + + test('rejects empty key', () => { + expect(() => validateColumnKey('')).toThrow(EmbeddingColumnConfigError); + }); + }); + + describe('validateColumnConfig', () => { + test('accepts valid config', () => { + expect(() => + validateColumnConfig('embedding_voyage', { + provider: 'voyage:voyage-3-large', + dimensions: 1024, + type: 'vector', + }), + ).not.toThrow(); + }); + + test('rejects bad type', () => { + expect(() => + validateColumnConfig('embedding_voyage', { + provider: 'voyage:voyage-3-large', + dimensions: 1024, + type: 'jsonb' as 'vector', + }), + ).toThrow(EmbeddingColumnConfigError); + }); + + test('rejects bad dimensions (zero/negative/too-large)', () => { + const base = { provider: 'voyage:voyage-3-large', type: 'vector' as const }; + expect(() => validateColumnConfig('x', { ...base, dimensions: 0 })).toThrow(); + expect(() => validateColumnConfig('x', { ...base, dimensions: -5 })).toThrow(); + expect(() => validateColumnConfig('x', { ...base, dimensions: MAX_DIMENSIONS + 1 })).toThrow(); + expect(() => validateColumnConfig('x', { ...base, dimensions: 1.5 as number })).toThrow(); + }); + + test('rejects bad provider (empty, missing colon, missing model)', () => { + const base = { dimensions: 1024, type: 'vector' as const }; + expect(() => validateColumnConfig('x', { ...base, provider: '' })).toThrow(); + expect(() => validateColumnConfig('x', { ...base, provider: 'voyage' })).toThrow(); + expect(() => validateColumnConfig('x', { ...base, provider: 'voyage:' })).toThrow(); + expect(() => validateColumnConfig('x', { ...base, provider: ':voyage-3-large' })).toThrow(); + }); + + test('rejects non-object shapes (array, null, scalar)', () => { + expect(() => validateColumnConfig('x', null)).toThrow(); + expect(() => validateColumnConfig('x', [])).toThrow(); + expect(() => validateColumnConfig('x', 'string')).toThrow(); + expect(() => validateColumnConfig('x', 42)).toThrow(); + }); + }); + + test('registry load throws when any entry is invalid', () => { + expect(() => + getEmbeddingColumnRegistry( + cfg({ + embedding_columns: { + 'embedding"; DROP --': { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' }, + }, + }), + ), + ).toThrow(EmbeddingColumnConfigError); + }); +}); + +describe('D3 — buildVectorCastFragment + quoteIdentifier', () => { + test('vector type emits $1::vector cast', () => { + const r: ResolvedColumn = { name: 'embedding', type: 'vector', dimensions: 1536, embeddingModel: '' }; + const { col, castSql } = buildVectorCastFragment(r); + expect(col).toBe('"embedding"'); + expect(castSql).toBe('$1::vector'); + }); + + test('halfvec type emits $1::halfvec(N) cast', () => { + const r: ResolvedColumn = { name: 'embedding_ze', type: 'halfvec', dimensions: 2560, embeddingModel: 'zeroentropyai:zembed-1' }; + const { col, castSql } = buildVectorCastFragment(r); + expect(col).toBe('"embedding_ze"'); + expect(castSql).toBe('$1::halfvec(2560)'); + }); + + test('quoteIdentifier wraps in double quotes', () => { + expect(quoteIdentifier('embedding')).toBe('"embedding"'); + expect(quoteIdentifier('embedding_voyage')).toBe('"embedding_voyage"'); + }); + + test('quoteIdentifier doubles embedded quotes (defense belt)', () => { + // Even though regex prevents this from reaching here in practice, + // the quoting belt handles a quoted-string-break attempt. + expect(quoteIdentifier('embed"ding')).toBe('"embed""ding"'); + }); +}); + +describe('normalizeEngineColumn — engine-side legacy converter', () => { + test('undefined returns builtin embedding descriptor', () => { + const r = normalizeEngineColumn(undefined); + expect(r.name).toBe('embedding'); + expect(r.type).toBe('vector'); + }); + + test("'embedding' literal returns builtin descriptor", () => { + const r = normalizeEngineColumn('embedding'); + expect(r.name).toBe('embedding'); + expect(r.type).toBe('vector'); + }); + + test("'embedding_image' literal returns 1024d vector descriptor", () => { + const r = normalizeEngineColumn('embedding_image'); + expect(r.name).toBe('embedding_image'); + expect(r.type).toBe('vector'); + expect(r.dimensions).toBe(1024); + }); + + test('ResolvedColumn descriptor passes through', () => { + const descriptor: ResolvedColumn = { + name: 'embedding_ze', + type: 'halfvec', + dimensions: 2560, + embeddingModel: 'zeroentropyai:zembed-1', + }; + expect(normalizeEngineColumn(descriptor)).toEqual(descriptor); + }); + + test('unknown raw string throws (engine purity contract)', () => { + // Strings other than legacy literals must NEVER reach the engine. + // The resolver lives at hybrid/op boundary; the engine throws if + // a caller bypassed it. + expect(() => normalizeEngineColumn('embedding_voyage' as string)).toThrow( + EmbeddingColumnNotRegisteredError, + ); + }); +}); + +describe('helpers', () => { + test('isDefaultColumn true only for "embedding"', () => { + const def: ResolvedColumn = { name: 'embedding', type: 'vector', dimensions: 1536, embeddingModel: '' }; + const alt: ResolvedColumn = { name: 'embedding_voyage', type: 'vector', dimensions: 1024, embeddingModel: 'v' }; + expect(isDefaultColumn(def)).toBe(true); + expect(isDefaultColumn(alt)).toBe(false); + }); + + test('isBuiltinColumn matches both builtins exactly', () => { + expect(isBuiltinColumn('embedding')).toBe(true); + expect(isBuiltinColumn('embedding_image')).toBe(true); + expect(isBuiltinColumn('embedding_voyage')).toBe(false); + }); + + test('exported constants are stable', () => { + expect(DEFAULT_COLUMN_NAME).toBe('embedding'); + expect(ALLOWED_COLUMN_TYPES.has('vector')).toBe(true); + expect(ALLOWED_COLUMN_TYPES.has('halfvec')).toBe(true); + expect(MAX_DIMENSIONS).toBe(8192); + expect(COLUMN_NAME_REGEX.test('embedding_voyage')).toBe(true); + expect(COLUMN_NAME_REGEX.test('Embedding')).toBe(false); + }); +}); + +describe('codex /ship #1 — prototype-pollution-safe registry', () => { + test('resolver rejects "constructor" even though regex accepts it', () => { + // The regex `^[a-z_][a-z0-9_]*$` matches "constructor" — but the + // registry uses Object.create(null) + Object.hasOwn so Object's + // inherited members don't masquerade as registered columns. + expect(() => + resolveEmbeddingColumn({ embeddingColumn: 'constructor' }, cfg()), + ).toThrow(EmbeddingColumnNotRegisteredError); + }); + + test('resolver rejects other inherited names (toString, hasOwnProperty)', () => { + for (const name of ['tostring', 'hasownproperty', 'isprototypeof', 'valueof']) { + expect(() => + resolveEmbeddingColumn({ embeddingColumn: name }, cfg()), + ).toThrow(EmbeddingColumnNotRegisteredError); + } + }); + + test('getEmbeddingColumnRegistry returns a null-prototype object', () => { + const reg = getEmbeddingColumnRegistry(cfg()); + // No Object.prototype inheritance — direct prototype access returns null. + expect(Object.getPrototypeOf(reg)).toBeNull(); + // Inherited properties are genuinely absent. + expect((reg as any).constructor).toBeUndefined(); + expect((reg as any).toString).toBeUndefined(); + }); +}); + +describe('codex /ship #2 — descriptor passthrough validates', () => { + test('passthrough re-validates name regex', () => { + const bad: ResolvedColumn = { + name: 'embedding"; DROP TABLE pages; --', + type: 'vector', + dimensions: 1536, + embeddingModel: 'voyage:voyage-3-large', + }; + expect(() => + resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()), + ).toThrow(EmbeddingColumnNotRegisteredError); + }); + + test('passthrough re-validates type field (rejects unknown)', () => { + const bad = { + name: 'embedding_voyage', + type: 'jsonb', + dimensions: 1024, + embeddingModel: 'voyage:voyage-3-large', + } as unknown as ResolvedColumn; + expect(() => + resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()), + ).toThrow(EmbeddingColumnConfigError); + }); + + test('passthrough re-validates dimensions field (rejects out-of-range)', () => { + const bad: ResolvedColumn = { + name: 'embedding_voyage', + type: 'vector', + dimensions: -5, + embeddingModel: 'voyage:voyage-3-large', + }; + expect(() => + resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()), + ).toThrow(EmbeddingColumnConfigError); + }); + + test('passthrough re-validates dimensions field (rejects SQL-shaped string)', () => { + const bad = { + name: 'embedding_voyage', + type: 'halfvec', + dimensions: '1); DROP TABLE pages; --', + embeddingModel: 'voyage:voyage-3-large', + } as unknown as ResolvedColumn; + expect(() => + resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()), + ).toThrow(EmbeddingColumnConfigError); + }); + + test('valid descriptor passes through unchanged', () => { + const good: ResolvedColumn = { + name: 'embedding_ze', + type: 'halfvec', + dimensions: 2560, + embeddingModel: 'zeroentropyai:zembed-1', + }; + expect(resolveEmbeddingColumn({ embeddingColumn: good }, cfg())).toEqual(good); + }); +}); + +describe('codex /ship #4 — isCacheSafe (embedding-space-based skip)', () => { + test('default name + matching dim + matching model → safe', () => { + const r: ResolvedColumn = { + name: 'embedding', + type: 'vector', + dimensions: 1536, + embeddingModel: 'openai:text-embedding-3-large', + }; + expect(isCacheSafe(r, cfg())).toBe(true); + }); + + test('non-default name → unsafe', () => { + const r: ResolvedColumn = { + name: 'embedding_voyage', + type: 'vector', + dimensions: 1024, + embeddingModel: 'voyage:voyage-3-large', + }; + expect(isCacheSafe(r, cfg())).toBe(false); + }); + + test('default name BUT overridden to different dim → unsafe', () => { + // User overrode the `embedding` builtin to point at a 1024-dim Voyage + // column. Name is still 'embedding' but the cache table is sized for + // 1536d (or whatever the brain's cfg dim was at init). UNSAFE. + const r: ResolvedColumn = { + name: 'embedding', + type: 'vector', + dimensions: 1024, + embeddingModel: 'voyage:voyage-3-large', + }; + expect(isCacheSafe(r, cfg({ embedding_dimensions: 1536 }))).toBe(false); + }); + + test('default name BUT overridden to different model (same dim) → unsafe', () => { + // Different model = different embedding space even at the same dim. + // OpenAI 1536d vectors are NOT interchangeable with Cohere/Voyage 1536d. + const r: ResolvedColumn = { + name: 'embedding', + type: 'vector', + dimensions: 1536, + embeddingModel: 'voyage:voyage-3-large', + }; + expect( + isCacheSafe( + r, + cfg({ + embedding_dimensions: 1536, + embedding_model: 'openai:text-embedding-3-large', + }), + ), + ).toBe(false); + }); + + test('zero-config brain (cfg has no embedding_dimensions/model) → defaults match → safe', () => { + const r: ResolvedColumn = { + name: 'embedding', + type: 'vector', + dimensions: 1536, + embeddingModel: 'openai:text-embedding-3-large', + }; + expect(isCacheSafe(r, cfg())).toBe(true); + }); +}); diff --git a/test/search/knobs-hash-reranker.test.ts b/test/search/knobs-hash-reranker.test.ts index f93c2189d..47c8ee978 100644 --- a/test/search/knobs-hash-reranker.test.ts +++ b/test/search/knobs-hash-reranker.test.ts @@ -43,7 +43,7 @@ function baseKnobs(): ResolvedSearchKnobs { } describe('KNOBS_HASH_VERSION + version invariants', () => { - test('version is 3 (1→2 v0.35.0.0 reranker; 2→3 v0.35.6.0 floor_ratio)', () => { + test('version is 3 (1→2 v0.35.0.0 reranker; 2→3 v0.35.6.0 floor_ratio + v0.36 embedding-column)', () => { expect(KNOBS_HASH_VERSION).toBe(3); }); @@ -150,4 +150,44 @@ describe('append-only convention (CDX2-F13)', () => { expect(rrIdx).toBeGreaterThan(0); expect(rrIdx).toBeGreaterThan(limIdx); }); + + test('v=3 additions: col= and prov= appear AFTER the reranker block', async () => { + // v0.36 D8: cache-key contamination across embedding columns + providers. + // The two new tokens must sit at the bottom of parts[] so existing v=2 + // hashes can only differ in those positions — keeping the append-only + // chain auditable for future v=4 readers. + const src = await Bun.file( + new URL('../../src/core/search/mode.ts', import.meta.url), + ).text(); + const rrtIdx = src.indexOf('rrt=${knobs.reranker_timeout_ms'); + const colIdx = src.indexOf('col=${ctx?.embeddingColumn'); + const provIdx = src.indexOf('prov=${ctx?.embeddingModel'); + expect(rrtIdx).toBeGreaterThan(0); + expect(colIdx).toBeGreaterThan(rrtIdx); + expect(provIdx).toBeGreaterThan(colIdx); + }); + + test('v=3 fields participate: column flip changes the hash', () => { + const k = baseKnobs(); + const defaultCol = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'openai:text-embedding-3-large' }); + const voyageCol = knobsHash(k, { embeddingColumn: 'embedding_voyage', embeddingModel: 'voyage:voyage-3-large' }); + expect(defaultCol).not.toBe(voyageCol); + }); + + test('v=3 fields participate: same column + different provider → different hash', () => { + const k = baseKnobs(); + const a = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'openai:text-embedding-3-large' }); + const b = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'openai:text-embedding-3-small' }); + expect(a).not.toBe(b); + }); + + test('v=3 fields fall back to embedding/default when ctx undefined', () => { + // Backward-compat: callers that don't know the column (e.g. telemetry + // helpers) should still produce a stable hash matching the default + // 'embedding' + 'default' provider pair. + const k = baseKnobs(); + const bare = knobsHash(k); + const explicit = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'default' }); + expect(bare).toBe(explicit); + }); });