mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.36.3.0 feat: dynamic embedding column selection for search (#1164)
* feat: migration v68 — eval_candidates.embedding_column Schema migration ALTERs eval_candidates to add a nullable embedding_column TEXT column. Per-row capture metadata so `gbrain eval replay` reproduces the same column the capture ran against (D16 / CDX-10). NULL-tolerant: pre-v0.36 rows fall back to current default. Renumbered v67→v68 because master claimed v67 for facts_typed_claim_columns during this branch's lifetime. PGLite parity via sqlFor.pglite — same ALTER IF NOT EXISTS. * feat: dynamic embedding column — core (resolver, types, gateway, engines) The read-path foundation for routing search through any populated embedding column, not just OpenAI 1536. src/core/search/embedding-column.ts (new) is the canonical seam. Single source of truth for column → provider/dim/type lookup. Validates registry keys via regex (/^[a-z_][a-z0-9_]*$/), uses Object.create(null) + Object.hasOwn so 'constructor' and other inherited names can't masquerade as registered columns. Identifier-quoting on SQL interpolation as defense in depth. src/core/types.ts widens SearchOpts.embeddingColumn to accept ResolvedColumn descriptors at the engine boundary; adds EmbeddingColumnConfig + ResolvedColumn exports. src/core/config.ts merges embedding_columns + search_embedding_column from the DB plane via loadConfigWithEngine, mirroring the existing embedding_multimodal_model pattern. Handles the no-file case so env-only Postgres installs see DB-plane overrides (codex /ship #3). src/core/ai/gateway.ts: embedQuery(text, opts) + embed(texts, opts) accept embeddingModel + dimensions overrides. isAvailable(touchpoint, modelOverride?) so hybrid asks 'is the active column's provider reachable?' not 'is the global default reachable?' (CDX-4 / D10). Engines: searchVector accepts ResolvedColumn descriptors via normalizeEngineColumn; engine code is config-free and unit-testable. getEmbeddingsByChunkIds(ids, column?) so cosineReScore hydrates from the active column instead of always 'embedding' (CDX-3 / D9). Identifier-quoting belt at the SQL boundary. src/core/eval-capture.ts threads embedding_column from hybridSearch meta into the persisted capture row. * feat: dynamic embedding column — integration (hybrid, ops, doctor) Wires the resolver into hybridSearch, the query op, doctor, and the config command. src/core/search/hybrid.ts: resolves the column once at the boundary, threads the descriptor into engine calls, routes embedQuery through the resolved column's provider/dims, and calls isCacheSafe (not isDefaultColumn) for cache skip so user overrides of the 'embedding' builtin can't leak across vector spaces (CDX-4). cosineReScore now hydrates from the active column. src/core/search/mode.ts: KNOBS_HASH_VERSION 2→3, append-only new fields col= and prov= alongside floor_ratio. Cache rows from different columns or providers now sit in different keyspaces — cross-column contamination impossible. src/core/operations.ts: query op accepts embedding_column param for per-call A/B benchmarking. search op (keyword-only) deliberately does NOT (CDX-9 / D15) — would be silent UX. src/commands/doctor.ts: new embedding_column_registry check. Batch format_type probe (D13) catches dim drift that information_schema.columns.udt_name can't. Batch pg_indexes probe (D5) warns on missing HNSW. Coverage % on active column, gates at <90% (D14), short-circuits on empty brains (codex /ship #5). src/commands/config.ts: validates embedding_columns JSON shape at set time, runs the coverage gate when setting search_embedding_column, uses Object.hasOwn for the registry lookup. src/commands/eval-replay.ts: replay re-runs queries against the captured embedding_column so post-flip-config replays don't surface as false-positive regressions. * test: dynamic embedding column — unit + e2e coverage 50 unit cases for the resolver (resolution chain, registry merge, validation, prototype pollution, descriptor passthrough, isCacheSafe, normalizeEngineColumn). 8 gateway override cases — embeddingModel + dimensions flow into providerOptions, isAvailable(touchpoint, override) routes to the right recipe, unknown models throw clean. 4 cosineReScore + 6 ops + 5 knobs-hash + 7 mode + 9 PGLite E2E + 7 Postgres E2E + 5 eval-replay column metadata. Postgres E2E (gated on DATABASE_URL) covers halfvec(2560) end-to-end on real pgvector, EXPLAIN-visible HNSW index on the alternate column, format_type-based dim drift catch, and the <90% coverage gate. Pins every codex /ship fix: prototype-pollution rejection ('constructor' as column name), descriptor passthrough validation (rejects SQL-shaped strings in dimensions), isCacheSafe semantics (space-based, not name-based). Total: 141 new + extended cases, all green. * chore: bump version and changelog (v0.36.3.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync to v0.36.3.0 Add CLAUDE.md key-files entry for src/core/search/embedding-column.ts. Annotate hybrid.ts, gateway.ts, doctor.ts, and migrate.ts entries with v0.36.3.0 wave changes (ResolvedColumn threading, embedQuery model override, embedding_column_registry check, migration v68). Document knobs_hash v=2 → v=3 bump under the Search Mode section. Regenerate llms-full.txt from the updated CLAUDE.md so the auto-checked bundle matches source (build-llms.test.ts CI guard). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): two CI failures from v0.36.3.0 1. test/loadConfig-merge.test.ts: update the 'returns null when base config is null' contract test. Pre-v0.36 the function returned null for null base; the codex /ship #3 fix changed that to synthesize a minimal `{ engine: 'postgres' }` so env-only installs see DB-plane overrides. Test now pins the new contract + adds a round-trip case asserting the merge actually surfaces `embedding_columns` / `search_embedding_column` set via gbrain config set on a null base. 2. test/schema-bootstrap-coverage.test.ts was failing because eval_candidates.embedding_column (added by migration v68) wasn't covered by applyForwardReferenceBootstrap. Fix: add the column to PGLITE_SCHEMA_SQL's eval_candidates CREATE TABLE definition (and src/schema.sql for parity) so fresh installs get it natively. The coverage test's third tier (schemaCreateTableCols) now finds it. Regenerated schema-embedded.ts via bun run build:schema. Schema-blob path is cleaner than COLUMN_EXEMPTIONS — fresh installs skip the migration entirely; upgrade installs still run v68. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
cdba533a04
commit
1d5f69fe7a
@@ -2,6 +2,94 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.36.3.0] - 2026-05-18
|
||||
|
||||
**Search now routes through any embedding column you've populated, not just OpenAI 1536. Voyage and ZeroEntropy columns become first-class search targets in one config flip.**
|
||||
|
||||
Until this release, gbrain hardcoded `embedding` (OpenAI 1536) as the only column hybrid search could read from. If you'd backfilled `embedding_voyage` (1024d) or `embedding_zeroentropy` (halfvec 2560d) on the side, the index was paid for but unusable. Now `gbrain config set search_embedding_column embedding_voyage` flips your whole brain to Voyage in a single line. The query op also takes `embedding_column` per-call for A/B benchmarking. Adversarial reviews on both the planning and shipping paths caught fifteen real bugs that aren't shipping today — config-plane drift, cosine-rescore corruption when the descriptor's space doesn't match the cache, prototype-pollution via `constructor` as a column key, validation bypass on the descriptor passthrough, and a doctor false-warn on fresh brains.
|
||||
|
||||
The numbers that matter:
|
||||
|
||||
| Surface | Before | After |
|
||||
|---|---|---|
|
||||
| Search columns reachable | `embedding` only (1536d OpenAI hardcoded) | Any column in the `embedding_columns` registry — vector or halfvec, any dim ≤ 8192 |
|
||||
| Per-call override | image-only via hidden internal flag | `gbrain query --json '{...,"embedding_column":"embedding_voyage"}'` |
|
||||
| Cross-column cache contamination | possible if you ever flipped columns | `knobs_hash` v=3 makes `(col, prov)` part of the cache key — silent corruption impossible |
|
||||
| `cosineReScore` rerank space | always pulled from `embedding` | pulls from the active column (D9 — Voyage HNSW now ranked against Voyage vectors, not OpenAI ones) |
|
||||
| Doctor diagnostics | `embeddings` check only | new `embedding_column_registry` check: format_type dim drift, missing HNSW indexes, coverage <90% gate |
|
||||
| Eval replay parity | results changed across column flips ("false regressions") | each `eval_candidates` row stores the column that ran; replay honors it |
|
||||
|
||||
What this means: you can run a side-by-side provider eval today. Set `embedding_voyage` as the default for a session, run `gbrain query "..." --json` across your test queries, capture the results, flip back to `embedding`, replay the captured rows with `gbrain eval replay --against ...`, see the Jaccard@k drift between providers. The cache won't lie to you because it sits in a different keyspace for each provider. Doctor will yell if you accidentally point at a column that's only 50% populated.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
|
||||
- **`embedding_columns` config registry** — declare which content_chunks columns are searchable. JSON map keyed by column name, each entry has `{provider, dimensions, type}` where type ∈ `vector | halfvec`. Set via `gbrain config set embedding_columns '...JSON...'` (DB plane). Validation runs at config-set time: regex on keys, type/dim/provider field shapes. Bad config refuses to load with a paste-ready hint.
|
||||
- **`search_embedding_column` config key** — picks the default column for hybridSearch. `gbrain config set search_embedding_column embedding_voyage` flips your whole brain. Coverage gate: refuses the switch when the target column is <90% populated unless you pass `--coverage-override`.
|
||||
- **`embedding_column` MCP param on the `query` op** — per-call override for A/B benchmarking. Unknown column names throw a structured error with the list of registered names. The `search` op (keyword-only) deliberately does NOT accept the param — it would be silent UX (CDX-9).
|
||||
- **`gbrain doctor` registry check** — `embedding_column_registry` probes each declared column via `format_type(atttypid, atttypmod)` so dim drift (declared 1024d, actual 1536d) surfaces with a paste-ready ALTER. Postgres also checks HNSW index presence; warns if missing. Coverage % on the active default column; warns when below 90% (skips the warn on empty brains).
|
||||
- **`isCacheSafe(resolved, cfg)` helper** — the cache-skip decision compares full embedding space (name + dim + model) against cfg, not just the column name. A user who overrides the `embedding` builtin to point at Voyage doesn't accidentally keep using the OpenAI-sized cache.
|
||||
|
||||
#### Changed
|
||||
|
||||
- **`hybridSearch` resolves the column at the boundary** — engines now take a pre-validated `ResolvedColumn` descriptor, not a raw string. Engine code is config-free and unit-testable in isolation (D11 / CDX-5).
|
||||
- **`gateway.embedQuery(text, { embeddingModel, dimensions })`** — query-side embed path accepts a model override. The resolved column's provider drives the embed call, so a query against `embedding_voyage` actually embeds via Voyage, not the global default (D10).
|
||||
- **`gateway.isAvailable('embedding', modelOverride?)`** — the availability check honors the override. Hybrid skips vector search only when the column's provider is down, not the global default's (CDX-4).
|
||||
- **`engine.getEmbeddingsByChunkIds(ids, column?)`** — `cosineReScore` hydrates embeddings from the active column. Without this, Voyage HNSW retrieval rescored against OpenAI vectors → NaN or wrong rankings (CDX-3 / D9).
|
||||
- **`KNOBS_HASH_VERSION` bumped 2→3** — the cache key now folds in column + provider. Pre-v3 cache rows become unreachable on first re-query (one-time miss spike). Source change lives in `mode.ts`, not `query-cache.ts`.
|
||||
- **`eval_candidates.embedding_column`** — schema migration v68. Per-row column metadata so `gbrain eval replay` reproduces the same column the capture ran against. NULL-tolerant; pre-v0.36 rows fall back to current default.
|
||||
|
||||
#### Fixed (codex /ship findings)
|
||||
|
||||
- **Prototype-pollution-safe registry (#1)** — registry uses `Object.create(null)` + `Object.hasOwn`. `gbrain config set search_embedding_column constructor` now correctly rejects rather than resolving to `Object.prototype.constructor`.
|
||||
- **Descriptor passthrough re-validates (#2)** — internal SDK callers passing a hand-rolled `ResolvedColumn` get full validation (name regex, type ∈ {vector,halfvec}, dims in [1, 8192]). Eliminates the SQL-injection escape hatch through the descriptor field.
|
||||
- **DB-plane config works without a file (#3)** — `loadConfigWithEngine` synthesizes a minimal base when `loadConfig()` returns null. Env-only Postgres installs can `gbrain config set search_embedding_column X` and the resolver actually sees it.
|
||||
- **Cache skip is embedding-space-based (#4)** — replaced name-based `isDefaultColumn` with `isCacheSafe(resolved, cfg)` at the call site. User overrides of the `embedding` builtin no longer leak across vector spaces.
|
||||
- **Doctor empty-brain UX (#5)** — coverage gate short-circuits when chunk count is 0. Fresh installs no longer see "Active column 'embedding' is 0.0% populated".
|
||||
|
||||
#### Tests
|
||||
|
||||
- New: `test/search/embedding-column.test.ts` (50 cases — resolver, registry, validation, prototype-pollution, descriptor passthrough, `isCacheSafe`).
|
||||
- New: `test/gateway-embed-model-override.test.ts` (8 cases — model/dim override, `isAvailable` override).
|
||||
- New: `test/cosine-rescore-column.test.ts` (4 PGLite cases — column-parameter hydration).
|
||||
- New: `test/operations-embedding-column.test.ts` (6 cases — `query` accepts param, `search` rejects it).
|
||||
- New: `test/e2e/embedding-column-pglite.test.ts` (9 cases — multi-col, halfvec, cosineReScore, unknown-name throw).
|
||||
- New: `test/e2e/embedding-column-postgres.test.ts` (7 cases — real-pgvector halfvec, HNSW index visibility, format_type dim drift, coverage gate).
|
||||
- New: `test/e2e/eval-replay-column.test.ts` (5 cases — column metadata persistence + replay honors).
|
||||
- Extended: `test/search-mode.test.ts`, `test/search/knobs-hash-reranker.test.ts` — v=3 hash, col/prov fields, append-only convention.
|
||||
|
||||
#### Plan reviewed
|
||||
|
||||
`/plan-eng-review` + codex outside voice surfaced 16 findings during planning + shipping (D1-D16 in the plan + 5 codex /ship findings). Every one applied; the plan file is at `~/.claude/plans/system-instruction-you-are-working-sparkling-sun.md`.
|
||||
|
||||
## To take advantage of v0.36.3.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns:
|
||||
|
||||
1. Apply the schema migration:
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. (Optional, only if you've backfilled extra columns via ALTER TABLE) declare them in the registry:
|
||||
```bash
|
||||
gbrain config set embedding_columns '{
|
||||
"embedding_voyage": { "provider": "voyage:voyage-3-large", "dimensions": 1024, "type": "vector" },
|
||||
"embedding_zeroentropy": { "provider": "zeroentropyai:zembed-1", "dimensions": 2560, "type": "halfvec" }
|
||||
}'
|
||||
```
|
||||
3. Verify with doctor:
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "embedding_column_registry")'
|
||||
```
|
||||
4. (Optional) flip the default for an A/B run:
|
||||
```bash
|
||||
gbrain config set search_embedding_column embedding_voyage
|
||||
gbrain query "test query" --json | jq '.results[0]'
|
||||
gbrain config set search_embedding_column embedding # flip back
|
||||
```
|
||||
|
||||
If any step fails, file an issue at https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
## [0.36.2.0] - 2026-05-17
|
||||
|
||||
**ZeroEntropy is the new default. Faster, cheaper, better quality on real queries. Existing users get a one-shot switch prompt with cost estimate; new installs land on it out of the box. README rewritten to match what gbrain actually is in 2026.**
|
||||
|
||||
+12
-4
File diff suppressed because one or more lines are too long
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.36.2.0",
|
||||
"version": "0.36.3.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+115
-1
@@ -1,5 +1,13 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { loadConfig, loadConfigWithEngine } from '../core/config.ts';
|
||||
import {
|
||||
getEmbeddingColumnRegistry,
|
||||
validateColumnKey,
|
||||
validateColumnConfig,
|
||||
quoteIdentifier,
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
EmbeddingColumnConfigError,
|
||||
} from '../core/search/embedding-column.ts';
|
||||
|
||||
function redactUrl(url: string): string {
|
||||
// Redact password in postgresql:// URLs
|
||||
@@ -98,6 +106,112 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
process.exit(1);
|
||||
}
|
||||
} else if (action === 'set' && key && value) {
|
||||
// v0.36 (D12 + D14): validate embedding-column keys at set time so a
|
||||
// bad config gets rejected loud + early. The `--coverage-override`
|
||||
// flag lets the user proceed past the < 90% gate when they know
|
||||
// they're mid-backfill.
|
||||
const coverageOverride =
|
||||
args.includes('--coverage-override') || args.includes('--yes');
|
||||
|
||||
if (key === 'embedding_columns') {
|
||||
try {
|
||||
const parsed = JSON.parse(value);
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new Error('embedding_columns must be a JSON object');
|
||||
}
|
||||
// D12: validate every key + entry shape before persisting.
|
||||
for (const [k, entry] of Object.entries(parsed)) {
|
||||
validateColumnKey(k);
|
||||
validateColumnConfig(k, entry);
|
||||
}
|
||||
} catch (err) {
|
||||
if (err instanceof EmbeddingColumnConfigError) {
|
||||
console.error(`[config] ${err.message}`);
|
||||
} else {
|
||||
console.error(
|
||||
`[config] embedding_columns rejected: ${(err as Error).message}`,
|
||||
);
|
||||
console.error(
|
||||
`[config] Expected JSON shape: {"<column_name>": {"provider": "...", "dimensions": N, "type": "vector" | "halfvec"}, ...}`,
|
||||
);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (key === 'search_embedding_column') {
|
||||
// Validate against the merged registry (file + DB plane + builtins).
|
||||
// We re-read merged config so a prior `gbrain config set
|
||||
// embedding_columns ...` is visible.
|
||||
const fileCfg = loadConfig();
|
||||
const mergedCfg = fileCfg
|
||||
? await loadConfigWithEngine(engine, fileCfg).catch(() => fileCfg)
|
||||
: null;
|
||||
if (mergedCfg) {
|
||||
let registry: ReturnType<typeof getEmbeddingColumnRegistry>;
|
||||
try {
|
||||
registry = getEmbeddingColumnRegistry(mergedCfg);
|
||||
} catch (err) {
|
||||
console.error(
|
||||
`[config] Existing embedding_columns is invalid; refusing to set search_embedding_column. ` +
|
||||
`Fix the registry first. (${(err as Error).message})`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
// Object.hasOwn so inherited keys ('constructor', 'toString', etc.)
|
||||
// cannot pass the registry-lookup gate.
|
||||
if (!Object.hasOwn(registry, value)) {
|
||||
const known = Object.keys(registry).sort().join(', ') || '(none)';
|
||||
console.error(
|
||||
`[config] Unknown embedding column "${value}". ` +
|
||||
`Declared columns: ${known}. ` +
|
||||
`Add it via: gbrain config set embedding_columns '<JSON>'`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// D14 coverage gate. Probe the column's NULL-rate; refuse when
|
||||
// coverage < 90% unless `--coverage-override` or `--yes` is
|
||||
// present.
|
||||
try {
|
||||
const covRows = await engine.executeRaw<{ pct: number; total: number }>(
|
||||
`SELECT (
|
||||
COUNT(*) FILTER (WHERE ${quoteIdentifier(value)} IS NOT NULL)::float
|
||||
/ NULLIF(COUNT(*), 0) * 100
|
||||
)::float AS pct,
|
||||
COUNT(*)::int AS total
|
||||
FROM content_chunks`,
|
||||
);
|
||||
const pct = covRows[0]?.pct ?? 0;
|
||||
const total = covRows[0]?.total ?? 0;
|
||||
if (total > 0 && pct < 90 && !coverageOverride) {
|
||||
console.error(
|
||||
`[config] Column "${value}" is ${pct.toFixed(1)}% populated (${total} total chunks).`,
|
||||
);
|
||||
console.error(
|
||||
`[config] Switching the default to a low-coverage column silently degrades search.`,
|
||||
);
|
||||
console.error(
|
||||
`[config] Re-run with --coverage-override (or --yes) to proceed anyway:`,
|
||||
);
|
||||
console.error(
|
||||
`[config] gbrain config set search_embedding_column ${value} --coverage-override`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
} catch (err) {
|
||||
// Coverage probe failure shouldn't block when the column shape
|
||||
// is otherwise valid (e.g. the column was JUST added, no chunks
|
||||
// yet, NULLIF guard returns NULL → pct=0 BUT total=0 short-
|
||||
// circuits above). If the SQL itself errors (column ALTER race,
|
||||
// permission), warn but proceed.
|
||||
console.error(
|
||||
`[config] WARN: coverage probe failed (${(err as Error).message}); proceeding.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await engine.setConfig(key, value);
|
||||
// v0.36.x #892: redact sensitive values in confirmation output. API
|
||||
// keys / tokens / passwords are commonly set from terminals with
|
||||
|
||||
@@ -2115,6 +2115,164 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
}
|
||||
} catch { /* listRecipes / gateway not available — silent */ }
|
||||
|
||||
// 8c. Embedding column registry (v0.36 — D5 + D13 + D14).
|
||||
// Validates every column in the merged registry against the real DB
|
||||
// shape: (a) column exists, (b) declared type+dims match actual
|
||||
// format_type(atttypid, atttypmod), (c) HNSW index present on
|
||||
// Postgres, (d) the ACTIVE default column has >= 90% coverage.
|
||||
//
|
||||
// Batch probes (D5) so the registry can grow without N+1 round-trips:
|
||||
// one format_type query, one pg_indexes query, one coverage-per-active
|
||||
// column query.
|
||||
progress.heartbeat('embedding_column_registry');
|
||||
try {
|
||||
const { getEmbeddingColumnRegistry, resolveEmbeddingColumn, quoteIdentifier } =
|
||||
await import('../core/search/embedding-column.ts');
|
||||
const { loadConfig: _loadConfig } = await import('../core/config.ts');
|
||||
const fileCfg = _loadConfig();
|
||||
const mergedCfg = fileCfg ? await (await import('../core/config.ts')).loadConfigWithEngine(engine, fileCfg).catch(() => fileCfg) : null;
|
||||
if (!mergedCfg) {
|
||||
checks.push({
|
||||
name: 'embedding_column_registry',
|
||||
status: 'ok',
|
||||
message: 'No brain config loaded — skipped',
|
||||
});
|
||||
} else {
|
||||
const registry = getEmbeddingColumnRegistry(mergedCfg);
|
||||
const declaredColumns = Object.keys(registry);
|
||||
const activeCol = resolveEmbeddingColumn(undefined, mergedCfg).name;
|
||||
|
||||
// D13 — batch format_type probe via pg_attribute. udt_name only
|
||||
// returns 'vector' vs 'halfvec'; format_type(atttypid, atttypmod)
|
||||
// returns 'vector(1024)' / 'halfvec(2560)' so dim drift surfaces.
|
||||
const formatRows = await engine.executeRaw<{ attname: string; formatted: string }>(
|
||||
`SELECT a.attname, format_type(a.atttypid, a.atttypmod) AS formatted
|
||||
FROM pg_attribute a
|
||||
JOIN pg_class c ON c.oid = a.attrelid
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = 'public'
|
||||
AND c.relname = 'content_chunks'
|
||||
AND a.attname = ANY($1::text[])
|
||||
AND NOT a.attisdropped`,
|
||||
[declaredColumns],
|
||||
);
|
||||
const actualByName = new Map<string, string>();
|
||||
for (const r of formatRows) actualByName.set(r.attname, r.formatted);
|
||||
|
||||
// D5 — batch index probe (Postgres only; PGLite indexing is implicit
|
||||
// and the partial-index pattern doesn't surface in pg_indexes the
|
||||
// same way). Reports informational, not blocking — search still
|
||||
// works without an HNSW index, just slow.
|
||||
const haveIndex = new Map<string, boolean>();
|
||||
if (engine.kind === 'postgres') {
|
||||
const indexRows = await engine.executeRaw<{ indexdef: string }>(
|
||||
`SELECT indexdef FROM pg_indexes
|
||||
WHERE tablename = 'content_chunks'
|
||||
AND schemaname = 'public'`,
|
||||
);
|
||||
for (const col of declaredColumns) {
|
||||
const found = indexRows.some(r => /USING\s+hnsw/i.test(r.indexdef) && r.indexdef.includes(`(${col} `));
|
||||
haveIndex.set(col, found);
|
||||
}
|
||||
}
|
||||
|
||||
// Per-column health rollup.
|
||||
const issues: string[] = [];
|
||||
const okColumns: string[] = [];
|
||||
for (const colName of declaredColumns) {
|
||||
const entry = registry[colName];
|
||||
const actual = actualByName.get(colName);
|
||||
if (!actual) {
|
||||
issues.push(`${colName}: declared but column does NOT exist in content_chunks`);
|
||||
continue;
|
||||
}
|
||||
// Expected format: `vector(N)` or `halfvec(N)`.
|
||||
const m = actual.match(/^(vector|halfvec)\((\d+)\)/i);
|
||||
const actualType = m ? m[1].toLowerCase() : actual;
|
||||
const actualDims = m ? parseInt(m[2], 10) : null;
|
||||
if (actualType !== entry.type) {
|
||||
issues.push(
|
||||
`${colName}: declared type=${entry.type} but actual is ${actual}. ` +
|
||||
`Fix: gbrain config set embedding_columns '<JSON>' OR ` +
|
||||
`ALTER TABLE content_chunks ALTER COLUMN ${colName} TYPE ${entry.type}(${entry.dimensions});`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (actualDims !== null && actualDims !== entry.dimensions) {
|
||||
issues.push(
|
||||
`${colName}: declared dims=${entry.dimensions} but actual is ${actual}. ` +
|
||||
`Fix one side: update config OR ` +
|
||||
`ALTER TABLE content_chunks ALTER COLUMN ${colName} TYPE ${entry.type}(${entry.dimensions});`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (engine.kind === 'postgres' && haveIndex.get(colName) === false) {
|
||||
issues.push(
|
||||
`${colName}: no HNSW index. Search works but uses sequential scan. ` +
|
||||
`Fix: CREATE INDEX IF NOT EXISTS idx_chunks_${colName} ON content_chunks USING hnsw (${quoteIdentifier(colName)} ${entry.type}_cosine_ops);`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
okColumns.push(colName);
|
||||
}
|
||||
|
||||
// D14 — coverage gate on the ACTIVE default column. Catches the
|
||||
// "user switched to a 5%-populated column" silent-degradation case.
|
||||
let coverageWarn: string | null = null;
|
||||
if (activeCol && actualByName.has(activeCol)) {
|
||||
// Codex /ship #5: pull `total` alongside `pct` so a fresh brain
|
||||
// (0 chunks → NULLIF makes pct NULL → coalesces to 0) doesn't
|
||||
// false-warn "Active column 'embedding' is 0.0% populated".
|
||||
const covRows = await engine.executeRaw<{ pct: number; total: number }>(
|
||||
`SELECT (
|
||||
COUNT(*) FILTER (WHERE ${quoteIdentifier(activeCol)} IS NOT NULL)::float
|
||||
/ NULLIF(COUNT(*), 0) * 100
|
||||
)::float AS pct,
|
||||
COUNT(*)::int AS total
|
||||
FROM content_chunks`,
|
||||
);
|
||||
const pct = covRows[0]?.pct ?? 0;
|
||||
const total = covRows[0]?.total ?? 0;
|
||||
// Only warn when there's a real coverage gap. Empty brain (0 chunks)
|
||||
// is a normal state for new installs — skip the gate entirely.
|
||||
if (total > 0 && pct < 90) {
|
||||
coverageWarn =
|
||||
`Active column '${activeCol}' is ${pct.toFixed(1)}% populated. ` +
|
||||
`Search quality silently degraded on un-embedded chunks. ` +
|
||||
`Fix: gbrain embed --column ${activeCol} --stale (write-side support v2) ` +
|
||||
`OR gbrain config set search_embedding_column embedding`;
|
||||
}
|
||||
}
|
||||
|
||||
if (issues.length === 0 && !coverageWarn) {
|
||||
const indexNote = engine.kind === 'postgres' ? ' (all indexed)' : '';
|
||||
checks.push({
|
||||
name: 'embedding_column_registry',
|
||||
status: 'ok',
|
||||
message: `Registry healthy: ${okColumns.length} columns (${okColumns.join(', ')})${indexNote}; active='${activeCol}'`,
|
||||
});
|
||||
} else {
|
||||
const allMessages = [
|
||||
...issues,
|
||||
...(coverageWarn ? [coverageWarn] : []),
|
||||
];
|
||||
checks.push({
|
||||
name: 'embedding_column_registry',
|
||||
status: 'warn',
|
||||
message: allMessages.join(' | '),
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
// Pre-config brains, registry-validation throws, etc. Surfaces the
|
||||
// error message but doesn't fail the doctor run.
|
||||
checks.push({
|
||||
name: 'embedding_column_registry',
|
||||
status: 'warn',
|
||||
message: `Could not check embedding column registry: ${(err as Error).message}`,
|
||||
});
|
||||
}
|
||||
|
||||
// 9. Graph health (link + timeline coverage on entity pages).
|
||||
// dead_links removed in v0.10.1: ON DELETE CASCADE on link FKs makes it always 0.
|
||||
//
|
||||
|
||||
@@ -177,6 +177,12 @@ interface CapturedRow {
|
||||
job_id?: number | null;
|
||||
subagent_id?: number | null;
|
||||
created_at?: string;
|
||||
/**
|
||||
* v0.36 (D16 / CDX-10): the embedding column that ran at capture time.
|
||||
* Optional for back-compat — pre-v0.36 exports won't have it. NULL or
|
||||
* missing means "use the current default."
|
||||
*/
|
||||
embedding_column?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -246,6 +252,10 @@ async function replayRow(engine: BrainEngine, row: CapturedRow, opts: ReplayOpts
|
||||
limit,
|
||||
detail: row.detail ?? undefined,
|
||||
expansion: row.expand_enabled ?? false,
|
||||
// v0.36 (D16 / CDX-10): replay the SAME column that ran at capture
|
||||
// time so config drift between capture and replay doesn't surface
|
||||
// as "regression." NULL/undefined falls through to resolver default.
|
||||
embeddingColumn: row.embedding_column ?? undefined,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
|
||||
+51
-7
@@ -547,8 +547,15 @@ export function getRerankerModel(): string | undefined {
|
||||
/**
|
||||
* Check whether a touchpoint can be served given the current config.
|
||||
* Replaces scattered `!process.env.OPENAI_API_KEY` checks (Codex C3).
|
||||
*
|
||||
* v0.36 (D10): optional `modelOverride` to check a specific
|
||||
* `provider:model` instead of the globally configured default for the
|
||||
* touchpoint. Used by hybridSearch to ask "is the active column's
|
||||
* provider reachable?" rather than "is the global default reachable?" —
|
||||
* otherwise an unreachable global default disables vector search even
|
||||
* when the active column's provider works fine.
|
||||
*/
|
||||
export function isAvailable(touchpoint: TouchpointKind): boolean {
|
||||
export function isAvailable(touchpoint: TouchpointKind, modelOverride?: string): boolean {
|
||||
// Test seam: when a transport stub is installed for this touchpoint, the
|
||||
// gateway is "available" for tests that exercise the whole pipeline without
|
||||
// configuring real providers. See __setChatTransportForTests /
|
||||
@@ -558,7 +565,9 @@ export function isAvailable(touchpoint: TouchpointKind): boolean {
|
||||
if (!_config) return false;
|
||||
try {
|
||||
const modelStr =
|
||||
touchpoint === 'embedding'
|
||||
modelOverride
|
||||
? modelOverride
|
||||
: touchpoint === 'embedding'
|
||||
? getEmbeddingModel()
|
||||
: touchpoint === 'expansion'
|
||||
? getExpansionModel()
|
||||
@@ -1050,21 +1059,49 @@ export interface EmbedOpts {
|
||||
* resolver — the correct default for indexing paths).
|
||||
*/
|
||||
inputType?: 'query' | 'document';
|
||||
/**
|
||||
* v0.36 (D10): explicit model override. When set, routes through this
|
||||
* provider:model instead of the globally configured embedding_model.
|
||||
* Used by the dynamic-embedding-column path so a single query can
|
||||
* embed via the provider that matches the active column. NULL/absent
|
||||
* preserves the existing global-default behavior.
|
||||
*
|
||||
* Format: 'provider:model' (e.g. 'voyage:voyage-3-large').
|
||||
*/
|
||||
embeddingModel?: string;
|
||||
/**
|
||||
* v0.36 (D10): explicit dimensions override, paired with
|
||||
* embeddingModel. When set, threads into `dimsProviderOptions` so the
|
||||
* gateway sends the right `dimensions` / `output_dimension` to the
|
||||
* provider. Must match the dim of the destination column or pgvector
|
||||
* rejects the insert/search. NULL preserves the global-default.
|
||||
*/
|
||||
dimensions?: number;
|
||||
}
|
||||
|
||||
export async function embed(texts: string[], opts?: EmbedOpts): Promise<Float32Array[]> {
|
||||
if (!texts || texts.length === 0) return [];
|
||||
|
||||
const cfg = requireConfig();
|
||||
const { model, recipe, modelId } = await resolveEmbeddingProvider(getEmbeddingModel());
|
||||
// v0.36 (D10): caller may override the model. Used by the dynamic-embedding-
|
||||
// column path so hybridSearch can embed via the column's provider, not the
|
||||
// global default. resolveEmbeddingProvider validates the override at the
|
||||
// recipe layer — bad model strings throw AIConfigError with a clear hint.
|
||||
const resolveTarget = opts?.embeddingModel ?? getEmbeddingModel();
|
||||
const { model, recipe, modelId } = await resolveEmbeddingProvider(resolveTarget);
|
||||
const truncated = texts.map(t => (t ?? '').slice(0, MAX_CHARS));
|
||||
// Dim override (D10) — when caller passes `dimensions`, use it. Otherwise
|
||||
// fall back to the global cfg default. dimsProviderOptions throws a
|
||||
// clear AIConfigError when a Voyage flexible-dim model gets an
|
||||
// unsupported value (the existing v0.33.1.1 fail-loud path).
|
||||
const effectiveDims = opts?.dimensions ?? cfg.embedding_dimensions ?? DEFAULT_EMBEDDING_DIMENSIONS;
|
||||
const providerOpts = dimsProviderOptions(
|
||||
recipe.implementation,
|
||||
modelId,
|
||||
cfg.embedding_dimensions ?? DEFAULT_EMBEDDING_DIMENSIONS,
|
||||
effectiveDims,
|
||||
opts?.inputType,
|
||||
);
|
||||
const expected = cfg.embedding_dimensions ?? DEFAULT_EMBEDDING_DIMENSIONS;
|
||||
const expected = effectiveDims;
|
||||
|
||||
const embedding = recipe.touchpoints?.embedding;
|
||||
const maxBatchTokens = embedding?.max_batch_tokens;
|
||||
@@ -1265,8 +1302,15 @@ export async function embedOne(text: string): Promise<Float32Array> {
|
||||
*
|
||||
* Returns a single Float32Array (not a batch).
|
||||
*/
|
||||
export async function embedQuery(text: string): Promise<Float32Array> {
|
||||
const [v] = await embed([text], { inputType: 'query' });
|
||||
export async function embedQuery(
|
||||
text: string,
|
||||
opts?: { embeddingModel?: string; dimensions?: number },
|
||||
): Promise<Float32Array> {
|
||||
const [v] = await embed([text], {
|
||||
inputType: 'query',
|
||||
embeddingModel: opts?.embeddingModel,
|
||||
dimensions: opts?.dimensions,
|
||||
});
|
||||
return v;
|
||||
}
|
||||
|
||||
|
||||
+55
-3
@@ -1,7 +1,7 @@
|
||||
import { readFileSync, writeFileSync, mkdirSync, chmodSync, existsSync } from 'fs';
|
||||
import { isAbsolute, join } from 'path';
|
||||
import { homedir } from 'os';
|
||||
import type { EngineConfig } from './types.ts';
|
||||
import type { EngineConfig, EmbeddingColumnConfig } from './types.ts';
|
||||
|
||||
/**
|
||||
* Where is the active DB URL coming from? Pure introspection, no connection
|
||||
@@ -84,6 +84,27 @@ export interface GBrainConfig {
|
||||
embedding_image_ocr?: boolean;
|
||||
embedding_image_ocr_model?: string;
|
||||
|
||||
/**
|
||||
* v0.36 — embedding-column registry (D7). Maps a content_chunks column
|
||||
* name to its provider + dimensions + pgvector type. Both keys live in
|
||||
* the DB plane (`gbrain config set ...`) so users can flip without
|
||||
* editing files. Resolver merges this with `BUILTIN_EMBEDDING_COLUMNS`
|
||||
* (which derive their provider from `embedding_model` /
|
||||
* `embedding_multimodal_model`).
|
||||
*
|
||||
* Validation lives in `src/core/search/embedding-column.ts` per D12 —
|
||||
* keys must match `/^[a-z_][a-z0-9_]*$/`, type in {vector, halfvec},
|
||||
* dimensions 1..8192, provider parseable as `provider:model`.
|
||||
*/
|
||||
embedding_columns?: Record<string, EmbeddingColumnConfig>;
|
||||
/**
|
||||
* v0.36 — name of the column hybridSearch uses by default. Per-call
|
||||
* `SearchOpts.embeddingColumn` overrides this; absent => 'embedding'.
|
||||
* Validated against the merged `embedding_columns` registry at config-
|
||||
* set time and on hybridSearch entry.
|
||||
*/
|
||||
search_embedding_column?: string;
|
||||
|
||||
/**
|
||||
* Thin-client mode (multi-topology v1). When set, this install does NOT
|
||||
* have a local DB; it talks to a remote `gbrain serve --http` over MCP.
|
||||
@@ -219,8 +240,18 @@ export async function loadConfigWithEngine(
|
||||
engine: { getConfig(key: string): Promise<string | null | undefined> },
|
||||
base?: GBrainConfig | null,
|
||||
): Promise<GBrainConfig | null> {
|
||||
const fileConfig = base !== undefined ? base : loadConfig();
|
||||
if (!fileConfig) return null;
|
||||
// Codex /ship finding #3: when there's no file config AND no env DB URL,
|
||||
// loadConfig() returns null and the DB merge would be skipped — env-only
|
||||
// installs (engine wired via direct SDK pass) wouldn't see DB-plane
|
||||
// overrides like `embedding_columns` / `search_embedding_column` set via
|
||||
// `gbrain config set`. Since we have a live engine here, synthesize a
|
||||
// minimal base config so the DB-plane merge still runs. The synthesized
|
||||
// config has no auth or model fields; DB-plane keys overlay correctly
|
||||
// and downstream callers either find them or fall through to defaults.
|
||||
// Also applies when callers pass an explicit null for `base`.
|
||||
const fileConfig: GBrainConfig =
|
||||
(base !== undefined ? base : loadConfig()) ??
|
||||
({ engine: 'postgres' } as GBrainConfig);
|
||||
|
||||
// DB-plane reads. Quiet failures — if the config table doesn't exist yet
|
||||
// (pre-v36 brain mid-migration), treat as null and let file/env defaults
|
||||
@@ -248,6 +279,12 @@ export async function loadConfigWithEngine(
|
||||
const dbMultimodalModel = await dbStr('embedding_multimodal_model');
|
||||
const dbOcr = await dbBool('embedding_image_ocr');
|
||||
const dbOcrModel = await dbStr('embedding_image_ocr_model');
|
||||
// v0.36 (D7) — embedding-column registry merge. Stored as JSON string in
|
||||
// the config table. Parse + shape-check here; full registry validation
|
||||
// (regex on keys, type/dim/provider field shapes) runs in the resolver at
|
||||
// first use so a malformed DB row doesn't kill engine connect.
|
||||
const dbEmbeddingColumns = await dbStr('embedding_columns');
|
||||
const dbSearchEmbeddingColumn = await dbStr('search_embedding_column');
|
||||
|
||||
// DB applies only when env did NOT win. Env presence is detected by the
|
||||
// sync loadConfig() already setting the field. For each flag, prefer the
|
||||
@@ -265,6 +302,21 @@ export async function loadConfigWithEngine(
|
||||
if (merged.embedding_image_ocr_model === undefined && dbOcrModel !== undefined) {
|
||||
merged.embedding_image_ocr_model = dbOcrModel;
|
||||
}
|
||||
if (merged.embedding_columns === undefined && dbEmbeddingColumns !== undefined) {
|
||||
try {
|
||||
const parsed = JSON.parse(dbEmbeddingColumns);
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
|
||||
merged.embedding_columns = parsed as Record<string, EmbeddingColumnConfig>;
|
||||
} else {
|
||||
console.warn('[gbrain] config: embedding_columns DB value is not a JSON object; ignoring');
|
||||
}
|
||||
} catch (err) {
|
||||
console.warn(`[gbrain] config: embedding_columns DB value is not valid JSON; ignoring (${(err as Error).message})`);
|
||||
}
|
||||
}
|
||||
if (merged.search_embedding_column === undefined && dbSearchEmbeddingColumn !== undefined) {
|
||||
merged.search_embedding_column = dbSearchEmbeddingColumn;
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
|
||||
+10
-2
@@ -30,9 +30,17 @@ export async function embed(text: string): Promise<Float32Array> {
|
||||
* embed seam so the provider returns query-side vectors. For symmetric
|
||||
* providers (OpenAI text-3, DashScope, Zhipu) the field is dropped — no
|
||||
* behavior change. Used by hybrid.ts on the search hot path.
|
||||
*
|
||||
* v0.36 (D10): optional `embeddingModel` + `dimensions` overrides so the
|
||||
* dynamic-embedding-column path can embed via the column's provider rather
|
||||
* than the globally-configured default. Bare `embedQuery(text)` preserves
|
||||
* pre-v0.36 behavior.
|
||||
*/
|
||||
export async function embedQuery(text: string): Promise<Float32Array> {
|
||||
return gatewayEmbedQuery(text);
|
||||
export async function embedQuery(
|
||||
text: string,
|
||||
opts?: { embeddingModel?: string; dimensions?: number },
|
||||
): Promise<Float32Array> {
|
||||
return gatewayEmbedQuery(text, opts);
|
||||
}
|
||||
|
||||
export interface EmbedBatchOptions {
|
||||
|
||||
+14
-1
@@ -578,7 +578,20 @@ export interface BrainEngine {
|
||||
// Search
|
||||
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>>;
|
||||
/**
|
||||
* Hydrate embeddings for chunks already known by id. v0.36 (D9):
|
||||
* optional `column` parameter selects which content_chunks column to
|
||||
* fetch from (default 'embedding'). The dynamic-embedding-column
|
||||
* search path hands its resolved column name here so cosineReScore
|
||||
* rehydrates in the right embedding space — otherwise vector search
|
||||
* against `embedding_voyage` would HNSW-rank against Voyage but
|
||||
* rescore against OpenAI vectors (NaN / wrong rankings).
|
||||
*
|
||||
* The column name MUST be regex-validated by the caller (resolveEmbed-
|
||||
* dingColumn rejects bad names). Engines identifier-quote on
|
||||
* interpolation as defense in depth (D12).
|
||||
*/
|
||||
getEmbeddingsByChunkIds(ids: number[], column?: string): Promise<Map<number, Float32Array>>;
|
||||
|
||||
// Chunks
|
||||
/**
|
||||
|
||||
@@ -116,6 +116,10 @@ export function buildEvalCandidateInput(
|
||||
remote: ctx.remote,
|
||||
job_id: ctx.job_id,
|
||||
subagent_id: ctx.subagent_id,
|
||||
// v0.36 (D16 / CDX-10): persist the column that ran. Null for
|
||||
// keyword-only `search` (meta.embedding_column omitted), preserving
|
||||
// back-compat with rows captured before the column tracking landed.
|
||||
embedding_column: ctx.meta.embedding_column ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -3513,6 +3513,39 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON think_ab_results (source_id, ran_at DESC);
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 74,
|
||||
name: 'eval_candidates_embedding_column',
|
||||
// v0.36.3.0 (D16 / CDX-10): persist the resolved embedding column on
|
||||
// each eval_candidates row so replay against a captured query uses
|
||||
// the column that was active at capture time — not whichever column
|
||||
// is current local default. Without this, switching
|
||||
// `search_embedding_column` between capture and replay produces
|
||||
// false-positive "regressions" that are just column changes.
|
||||
//
|
||||
// Nullable for back-compat: pre-v0.36 rows have NULL; replay treats
|
||||
// NULL as "use current default" so existing captures keep working
|
||||
// exactly as before the migration.
|
||||
//
|
||||
// Renumbered v68→v74 during the second master merge: master's
|
||||
// v0.36.1.0 calibration wave claimed v68-v73 first. The ALTER
|
||||
// itself is unchanged; only the slot number moved. The column is
|
||||
// also in PGLITE_SCHEMA_SQL / src/schema.sql so fresh installs get
|
||||
// it natively without running this migration.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
ALTER TABLE eval_candidates
|
||||
ADD COLUMN IF NOT EXISTS embedding_column TEXT;
|
||||
`,
|
||||
// PGLite parity: same ALTER, same IF NOT EXISTS guard makes this a
|
||||
// no-op on subsequent boots.
|
||||
sqlFor: {
|
||||
pglite: `
|
||||
ALTER TABLE eval_candidates
|
||||
ADD COLUMN IF NOT EXISTS embedding_column TEXT;
|
||||
`,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -1143,6 +1143,11 @@ const query: Operation = {
|
||||
description:
|
||||
"v0.34: scope search to a single source. Defaults to OperationContext.sourceId (set from CLI --source / GBRAIN_SOURCE / .gbrain-source dotfile). Pass '__all__' to force cross-source search in multi-source brains.",
|
||||
},
|
||||
embedding_column: {
|
||||
type: 'string',
|
||||
description:
|
||||
"v0.36: route vector search through a non-default embedding column. Defaults to 'embedding' (OpenAI 1536d) unless `search_embedding_column` config sets a different default. Per-call override for A/B benchmarking across providers (e.g. 'embedding_voyage', 'embedding_zeroentropy'). Column MUST be declared in the `embedding_columns` config registry — unknown names throw with a paste-ready hint listing valid columns.",
|
||||
},
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const startedAt = Date.now();
|
||||
@@ -1151,6 +1156,10 @@ const query: Operation = {
|
||||
const queryText = p.query as string | undefined;
|
||||
const imageData = p.image as string | undefined;
|
||||
const imageMime = (p.image_mime as string) || 'image/jpeg';
|
||||
const embeddingColumnParam =
|
||||
typeof p.embedding_column === 'string' && p.embedding_column.length > 0
|
||||
? (p.embedding_column as string)
|
||||
: undefined;
|
||||
// Explicit per-call source_id must win over ctx.sourceId. The special
|
||||
// __all__ value opts out of source filtering for local cross-source search.
|
||||
const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined;
|
||||
@@ -1220,6 +1229,12 @@ const query: Operation = {
|
||||
useCache: typeof p.use_cache === 'boolean' ? (p.use_cache as boolean) : undefined,
|
||||
intentWeighting: typeof p.intent_weighting === 'boolean' ? (p.intent_weighting as boolean) : undefined,
|
||||
onMeta: (m) => { capturedMeta = m; },
|
||||
// v0.36 (D15): per-call embedding column override. Resolver rejects
|
||||
// unknown names at hybrid entry with EmbeddingColumnNotRegisteredError;
|
||||
// the error surfaces back to the agent as the op error envelope.
|
||||
// Source scope is already threaded via ...querySourceScope above
|
||||
// (master's #1182 cleanup of the duplicate sourceScopeOpts spread).
|
||||
embeddingColumn: embeddingColumnParam,
|
||||
});
|
||||
const latency_ms = Date.now() - startedAt;
|
||||
|
||||
|
||||
+36
-14
@@ -41,6 +41,13 @@ import { GBrainError, PAGE_SORT_SQL } from './types.ts';
|
||||
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts';
|
||||
import {
|
||||
normalizeEngineColumn,
|
||||
buildVectorCastFragment,
|
||||
quoteIdentifier,
|
||||
COLUMN_NAME_REGEX,
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from './search/embedding-column.ts';
|
||||
import { hasCJK, escapeLikePattern } from './cjk.ts';
|
||||
|
||||
type PGLiteDB = PGlite;
|
||||
@@ -1279,15 +1286,18 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// same candidate count it always did. See postgres-engine.ts for rationale.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
// v0.27.1: column routing. Default 'embedding' targets the brain's
|
||||
// primary text-embedding column; 'embedding_image' targets the
|
||||
// multimodal column populated by importImageFile. Image-similarity
|
||||
// queries pass embeddingColumn='embedding_image' AND a 1024-dim vector
|
||||
// produced by gateway.embedMultimodal — must match the column dim.
|
||||
const col = opts?.embeddingColumn === 'embedding_image' ? 'embedding_image' : 'embedding';
|
||||
// v0.36 (D11): column routing via resolved descriptor. Engine doesn't
|
||||
// read config — caller resolved at hybrid/op boundary. The cast SQL
|
||||
// ($1::vector vs $1::halfvec(N)) comes from buildVectorCastFragment.
|
||||
const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn);
|
||||
const { col, castSql } = buildVectorCastFragment(resolvedCol);
|
||||
// Image rows live in modality='image'; text/code in 'text'. Restrict
|
||||
// to the modality matching the column to avoid cross-mode dim leaks.
|
||||
const modalityFilter = col === 'embedding_image' ? `AND cc.modality = 'image'` : `AND cc.modality = 'text'`;
|
||||
// by modality so non-image columns can't accidentally pull image
|
||||
// chunks (or vice versa). resolved.name has already passed regex
|
||||
// validation; never compares against raw input.
|
||||
const modalityFilter = resolvedCol.name === 'embedding_image'
|
||||
? `AND cc.modality = 'image'`
|
||||
: `AND cc.modality = 'text'`;
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH hnsw_candidates AS (
|
||||
@@ -1295,12 +1305,12 @@ export class PGLiteEngine implements BrainEngine {
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.${col} <=> $1::vector) AS raw_score
|
||||
1 - (cc.${col} <=> ${castSql}) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.${col} IS NOT NULL ${modalityFilter} ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY cc.${col} <=> $1::vector
|
||||
ORDER BY cc.${col} <=> ${castSql}
|
||||
LIMIT $2
|
||||
)
|
||||
SELECT
|
||||
@@ -1321,10 +1331,21 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
}
|
||||
|
||||
async getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>> {
|
||||
async getEmbeddingsByChunkIds(
|
||||
ids: number[],
|
||||
column: string = 'embedding',
|
||||
): Promise<Map<number, Float32Array>> {
|
||||
if (ids.length === 0) return new Map();
|
||||
// v0.36 (D9): column parameter so hybrid.cosineReScore can rehydrate
|
||||
// from the active embedding space (Voyage 1024d, ZE halfvec 2560d,
|
||||
// etc.). Identifier-quoted (D12 layer 2) plus strict regex on the
|
||||
// column name (D12 layer 1) before interpolation.
|
||||
if (!COLUMN_NAME_REGEX.test(column)) {
|
||||
throw new EmbeddingColumnNotRegisteredError(column, []);
|
||||
}
|
||||
const quotedCol = quoteIdentifier(column);
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id, embedding FROM content_chunks WHERE id = ANY($1::int[]) AND embedding IS NOT NULL`,
|
||||
`SELECT id, ${quotedCol} AS embedding FROM content_chunks WHERE id = ANY($1::int[]) AND ${quotedCol} IS NOT NULL`,
|
||||
[ids]
|
||||
);
|
||||
const result = new Map<number, Float32Array>();
|
||||
@@ -3817,8 +3838,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
`INSERT INTO eval_candidates (
|
||||
tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
|
||||
expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
|
||||
latency_ms, remote, job_id, subagent_id
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)
|
||||
latency_ms, remote, job_id, subagent_id, embedding_column
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)
|
||||
RETURNING id`,
|
||||
[
|
||||
input.tool_name,
|
||||
@@ -3835,6 +3856,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
input.remote,
|
||||
input.job_id,
|
||||
input.subagent_id,
|
||||
input.embedding_column ?? null,
|
||||
]
|
||||
);
|
||||
return rows[0]!.id;
|
||||
|
||||
@@ -467,7 +467,12 @@ CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
salience_resolved TEXT,
|
||||
recency_resolved TEXT,
|
||||
salience_source TEXT,
|
||||
recency_source TEXT
|
||||
recency_source TEXT,
|
||||
-- v0.36.3.0 (D16 / CDX-10) — embedding column that ran at capture time.
|
||||
-- Nullable; pre-v0.36 rows have NULL and replay falls back to current
|
||||
-- default. See src/core/migrate.ts migration v68 for the matching ALTER
|
||||
-- on upgrade brains.
|
||||
embedding_column TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
|
||||
|
||||
|
||||
+43
-11
@@ -18,6 +18,13 @@ import { runMigrations } from './migrate.ts';
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
import { verifySchema } from './schema-verify.ts';
|
||||
import { applyChunkEmbeddingIndexPolicy, dropZombieIndexes } from './vector-index.ts';
|
||||
import {
|
||||
normalizeEngineColumn,
|
||||
buildVectorCastFragment,
|
||||
quoteIdentifier,
|
||||
COLUMN_NAME_REGEX,
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from './search/embedding-column.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow,
|
||||
@@ -1292,9 +1299,20 @@ export class PostgresEngine implements BrainEngine {
|
||||
// wasting candidate slots on hidden rows.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
// v0.27.1: column routing. See pglite-engine.ts searchVector for rationale.
|
||||
const col = opts?.embeddingColumn === 'embedding_image' ? 'embedding_image' : 'embedding';
|
||||
const modalityFilter = col === 'embedding_image' ? `AND cc.modality = 'image'` : `AND cc.modality = 'text'`;
|
||||
// v0.36 (D11): column routing via resolved descriptor. Engine doesn't
|
||||
// read config — caller (hybrid/op) resolved it and passed it in.
|
||||
// normalizeEngineColumn accepts the legacy union (string literals,
|
||||
// ResolvedColumn, undefined) and produces a canonical descriptor.
|
||||
const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn);
|
||||
const { col, castSql } = buildVectorCastFragment(resolvedCol);
|
||||
// Modality filter: image rows live in modality='image'; text/code in
|
||||
// 'text'. The image-column literal is the only one with image rows.
|
||||
// For all other columns (default + user-declared), restrict to text
|
||||
// mode to avoid cross-modality dim leaks. The check is on
|
||||
// resolved.name (already validated, never raw input).
|
||||
const modalityFilter = resolvedCol.name === 'embedding_image'
|
||||
? `AND cc.modality = 'image'`
|
||||
: `AND cc.modality = 'text'`;
|
||||
|
||||
const rawQuery = `
|
||||
WITH hnsw_candidates AS (
|
||||
@@ -1302,7 +1320,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.${col} <=> $1::vector) AS raw_score
|
||||
1 - (cc.${col} <=> ${castSql}) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
@@ -1318,7 +1336,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
${sourceClause}
|
||||
${hardExcludeClause}
|
||||
${visibilityClause}
|
||||
ORDER BY cc.${col} <=> $1::vector
|
||||
ORDER BY cc.${col} <=> ${castSql}
|
||||
LIMIT ${innerLimitParam}
|
||||
)
|
||||
SELECT
|
||||
@@ -1340,13 +1358,27 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
|
||||
async getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>> {
|
||||
async getEmbeddingsByChunkIds(
|
||||
ids: number[],
|
||||
column: string = 'embedding',
|
||||
): Promise<Map<number, Float32Array>> {
|
||||
if (ids.length === 0) return new Map();
|
||||
// v0.36 (D9): column parameter used by hybrid.cosineReScore so
|
||||
// rescoring rehydrates from the active column's embedding space,
|
||||
// not always 'embedding'. Engine has no resolver access; the
|
||||
// caller must pass a known column name. Identifier-quoted (D12
|
||||
// defense layer 2) plus a strict regex check (D12 defense layer 1)
|
||||
// so even a misconfigured caller can't smuggle a SQL fragment.
|
||||
if (!COLUMN_NAME_REGEX.test(column)) {
|
||||
throw new EmbeddingColumnNotRegisteredError(column, []);
|
||||
}
|
||||
const quotedCol = quoteIdentifier(column);
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT id, embedding FROM content_chunks
|
||||
WHERE id = ANY(${ids}::int[]) AND embedding IS NOT NULL
|
||||
const rawQuery = `
|
||||
SELECT id, ${quotedCol} AS embedding FROM content_chunks
|
||||
WHERE id = ANY($1::int[]) AND ${quotedCol} IS NOT NULL
|
||||
`;
|
||||
const rows = await sql.unsafe(rawQuery, [ids] as Parameters<typeof sql.unsafe>[1]);
|
||||
const result = new Map<number, Float32Array>();
|
||||
for (const row of rows) {
|
||||
const embedding = tryParseEmbedding(row.embedding);
|
||||
@@ -3777,11 +3809,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
INSERT INTO eval_candidates (
|
||||
tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
|
||||
expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
|
||||
latency_ms, remote, job_id, subagent_id
|
||||
latency_ms, remote, job_id, subagent_id, embedding_column
|
||||
) VALUES (
|
||||
${input.tool_name}, ${input.query}, ${input.retrieved_slugs}, ${input.retrieved_chunk_ids}, ${input.source_ids},
|
||||
${input.expand_enabled}, ${input.detail}, ${input.detail_resolved}, ${input.vector_enabled}, ${input.expansion_applied},
|
||||
${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id}
|
||||
${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id}, ${input.embedding_column ?? null}
|
||||
)
|
||||
RETURNING id
|
||||
`;
|
||||
|
||||
@@ -791,7 +791,13 @@ CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
salience_resolved TEXT,
|
||||
recency_resolved TEXT,
|
||||
salience_source TEXT,
|
||||
recency_source TEXT
|
||||
recency_source TEXT,
|
||||
-- v0.36.3.0 (D16 / CDX-10) — embedding column resolved at capture time so
|
||||
-- \`gbrain eval replay\` reproduces the same column the capture ran against.
|
||||
-- Nullable; pre-v0.36 rows have NULL and replay falls back to current
|
||||
-- default. Migration v68 (src/core/migrate.ts) adds the same column on
|
||||
-- upgrade brains.
|
||||
embedding_column TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
|
||||
|
||||
|
||||
@@ -0,0 +1,538 @@
|
||||
/**
|
||||
* Dynamic embedding column resolver (v0.36 — D2+D3+D11+D12).
|
||||
*
|
||||
* Single source of truth for "which content_chunks column gets searched, and
|
||||
* what provider produced its vectors." The hybrid/op boundary calls
|
||||
* `resolveEmbeddingColumn()` once and passes the resulting descriptor INTO
|
||||
* the engine; engines never read config or call the resolver themselves
|
||||
* (D11 — engine stays a pure SQL composer).
|
||||
*
|
||||
* ┌─────────────────────────────────────────────┐
|
||||
* │ Config registry (file + DB plane merged) │
|
||||
* │ embedding_columns: { name → entry } │
|
||||
* │ search_embedding_column: string │
|
||||
* └────────────────────┬────────────────────────┘
|
||||
* │
|
||||
* ▼
|
||||
* ┌─────────────────────────────────────────────┐
|
||||
* │ getEmbeddingColumnRegistry(cfg) │
|
||||
* │ Merge: BUILTIN_KEYS + user config │
|
||||
* │ Validate at load time (D12): │
|
||||
* │ key: /^[a-z_][a-z0-9_]*$/ │
|
||||
* │ type: 'vector' | 'halfvec' │
|
||||
* │ dims: 1..8192 │
|
||||
* │ provider: parseable 'provider:model' │
|
||||
* └────────────────────┬────────────────────────┘
|
||||
* │
|
||||
* ┌────────────────────────┼────────────────────────┐
|
||||
* ▼ ▼ ▼
|
||||
* ┌──────────────────────┐ ┌──────────────────────┐ ┌─────────────────────┐
|
||||
* │ resolveEmbeddingCol │ │ getEmbeddingColumnRe │ │ buildVectorCastFrag │
|
||||
* │ (opts, cfg) │ │ gistry(cfg) │ │ ment(resolved) │
|
||||
* │ │ │ │ │ │
|
||||
* │ Chain: │ │ Returns full reg │ │ Returns: │
|
||||
* │ opts.embeddingColumn│ │ for doctor + config │ │ { col, castSql } │
|
||||
* │ → cfg.search_emb_ │ │ validation. │ │ │
|
||||
* │ column │ │ │ │ col: quoted ident │
|
||||
* │ → 'embedding' │ │ │ │ "<name>" (D12) │
|
||||
* │ │ │ │ │ castSql: │
|
||||
* │ Returns Resolved- │ │ │ │ '$1::vector' OR │
|
||||
* │ Column descriptor. │ │ │ │ '$1::halfvec(N)' │
|
||||
* └──────────┬───────────┘ └──────────────────────┘ └─────────────────────┘
|
||||
* │
|
||||
* ▼ (descriptor passed to engines + cosineReScore)
|
||||
* ┌──────────────────────────────────────────────────────────────────┐
|
||||
* │ engines: searchVector + getEmbeddingsByChunkIds │
|
||||
* │ SQL: SELECT ... FROM content_chunks cc │
|
||||
* │ WHERE cc.${quoteIdentifier(name)} IS NOT NULL │
|
||||
* │ ORDER BY cc.${quoteIdentifier(name)} <=> $1::${cast} │
|
||||
* └──────────────────────────────────────────────────────────────────┘
|
||||
*
|
||||
* Trust boundary: registry KEYS come from user config (DB plane + file
|
||||
* plane). Any path with config-write access can in principle declare a key.
|
||||
* Defense in depth:
|
||||
* 1. Strict regex on every key at load time — loud throw on invalid.
|
||||
* 2. Identifier-quoting at SQL-build time — even if regex misses,
|
||||
* pgvector identifier quoting prevents string-break injection.
|
||||
* 3. Field validation at load (type/dims/provider) — bad config refuses
|
||||
* to load instead of silently ignoring entries.
|
||||
*
|
||||
* Future write-path PR (`gbrain embed --column X --model Y`, deferred per
|
||||
* D1 out-of-scope list) MUST also consume this resolver — never compute a
|
||||
* column→provider mapping by hand. This is the canonical seam.
|
||||
*/
|
||||
|
||||
import type {
|
||||
EmbeddingColumnConfig,
|
||||
ResolvedColumn,
|
||||
SearchOpts,
|
||||
} from '../types.ts';
|
||||
import type { GBrainConfig } from '../config.ts';
|
||||
|
||||
// ---- Constants ---------------------------------------------------------
|
||||
|
||||
/** Strict identifier regex for registry keys. Defense layer 1 of D12. */
|
||||
export const COLUMN_NAME_REGEX = /^[a-z_][a-z0-9_]*$/;
|
||||
|
||||
/** Allowed pgvector types for this registry. Halfvec lands at v0.4.3+. */
|
||||
export const ALLOWED_COLUMN_TYPES = new Set<EmbeddingColumnConfig['type']>([
|
||||
'vector',
|
||||
'halfvec',
|
||||
]);
|
||||
|
||||
/** Upper bound on declared dimensions. pgvector hard cap is 16K but
|
||||
* practical embedding models top out around 4096; 8192 is plenty of
|
||||
* headroom and rejects obvious junk like negative or astronomical
|
||||
* values. */
|
||||
export const MAX_DIMENSIONS = 8192;
|
||||
|
||||
/**
|
||||
* Default name used when neither caller nor config sets a column.
|
||||
* Resolution chain: opts → cfg.search_embedding_column → DEFAULT.
|
||||
*/
|
||||
export const DEFAULT_COLUMN_NAME = 'embedding';
|
||||
|
||||
/** Names that always exist regardless of user config. Both derive their
|
||||
* provider from existing config keys (embedding_model and
|
||||
* embedding_multimodal_model) so users who don't declare anything still
|
||||
* get correct routing. */
|
||||
const BUILTIN_KEYS = ['embedding', 'embedding_image'] as const;
|
||||
type BuiltinKey = (typeof BUILTIN_KEYS)[number];
|
||||
|
||||
// ---- Errors -----------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Thrown when a column name isn't in the merged registry. Carries a
|
||||
* paste-ready hint listing valid columns so the user (or agent) sees
|
||||
* exactly what to type next.
|
||||
*/
|
||||
export class EmbeddingColumnNotRegisteredError extends Error {
|
||||
readonly code = 'embedding_column_not_registered';
|
||||
readonly columnName: string;
|
||||
readonly validColumns: string[];
|
||||
|
||||
constructor(columnName: string, validColumns: string[]) {
|
||||
const valid = validColumns.length > 0 ? validColumns.join(', ') : '(none)';
|
||||
super(
|
||||
`Embedding column "${columnName}" is not registered. ` +
|
||||
`Declared columns: ${valid}. ` +
|
||||
`Add it via: gbrain config set embedding_columns '<JSON>'`,
|
||||
);
|
||||
this.name = 'EmbeddingColumnNotRegisteredError';
|
||||
this.columnName = columnName;
|
||||
this.validColumns = validColumns;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Thrown when a registry entry fails load-time validation. The .field
|
||||
* pinpoints which sub-shape was wrong so doctor + config-set surfaces
|
||||
* can render targeted hints.
|
||||
*/
|
||||
export class EmbeddingColumnConfigError extends Error {
|
||||
readonly code = 'embedding_column_config_invalid';
|
||||
readonly columnKey: string;
|
||||
readonly field: 'key' | 'type' | 'dimensions' | 'provider' | 'shape';
|
||||
|
||||
constructor(
|
||||
columnKey: string,
|
||||
field: EmbeddingColumnConfigError['field'],
|
||||
detail: string,
|
||||
) {
|
||||
super(`embedding_columns["${columnKey}"]: invalid ${field} — ${detail}`);
|
||||
this.name = 'EmbeddingColumnConfigError';
|
||||
this.columnKey = columnKey;
|
||||
this.field = field;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Validation helpers (D12) -----------------------------------------
|
||||
|
||||
/**
|
||||
* Loud rejection of any registry key that isn't a strict SQL identifier.
|
||||
* Catches `embedding"; DROP --`, `embed-1`, leading-digits, etc. at the
|
||||
* earliest possible moment.
|
||||
*/
|
||||
export function validateColumnKey(key: string): void {
|
||||
if (typeof key !== 'string' || key.length === 0) {
|
||||
throw new EmbeddingColumnConfigError(
|
||||
String(key ?? ''),
|
||||
'key',
|
||||
'must be a non-empty string',
|
||||
);
|
||||
}
|
||||
if (!COLUMN_NAME_REGEX.test(key)) {
|
||||
throw new EmbeddingColumnConfigError(
|
||||
key,
|
||||
'key',
|
||||
`must match ${COLUMN_NAME_REGEX} (lowercase identifier, starts with letter or underscore, no quotes/symbols)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** `provider:model` string check. Both halves must be non-empty. */
|
||||
function isParseableProviderModel(s: unknown): s is string {
|
||||
if (typeof s !== 'string' || s.length === 0) return false;
|
||||
const idx = s.indexOf(':');
|
||||
if (idx <= 0) return false;
|
||||
if (idx === s.length - 1) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates an entry's shape. Throws on any failure with a pinpoint hint.
|
||||
* Called by getEmbeddingColumnRegistry at load time AND by config-set
|
||||
* before persisting.
|
||||
*/
|
||||
export function validateColumnConfig(
|
||||
key: string,
|
||||
entry: unknown,
|
||||
): asserts entry is EmbeddingColumnConfig {
|
||||
if (!entry || typeof entry !== 'object' || Array.isArray(entry)) {
|
||||
throw new EmbeddingColumnConfigError(
|
||||
key,
|
||||
'shape',
|
||||
'must be a JSON object',
|
||||
);
|
||||
}
|
||||
const e = entry as Record<string, unknown>;
|
||||
|
||||
if (!isParseableProviderModel(e.provider)) {
|
||||
throw new EmbeddingColumnConfigError(
|
||||
key,
|
||||
'provider',
|
||||
'must be a non-empty "provider:model" string (e.g. "voyage:voyage-3-large")',
|
||||
);
|
||||
}
|
||||
if (
|
||||
typeof e.dimensions !== 'number' ||
|
||||
!Number.isInteger(e.dimensions) ||
|
||||
e.dimensions < 1 ||
|
||||
e.dimensions > MAX_DIMENSIONS
|
||||
) {
|
||||
throw new EmbeddingColumnConfigError(
|
||||
key,
|
||||
'dimensions',
|
||||
`must be an integer in [1, ${MAX_DIMENSIONS}]`,
|
||||
);
|
||||
}
|
||||
if (typeof e.type !== 'string' || !ALLOWED_COLUMN_TYPES.has(e.type as 'vector' | 'halfvec')) {
|
||||
throw new EmbeddingColumnConfigError(
|
||||
key,
|
||||
'type',
|
||||
`must be one of: ${[...ALLOWED_COLUMN_TYPES].join(', ')}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- SQL helpers (D12 defense layer 2) --------------------------------
|
||||
|
||||
/**
|
||||
* Identifier-quoting helper. Wraps `name` in double quotes and doubles
|
||||
* any embedded quotes per the Postgres spec. Even though the column
|
||||
* name passed in here is already regex-validated (so no embedded
|
||||
* quotes are possible), this is the defense-in-depth belt for D12.
|
||||
*
|
||||
* Returns the quoted form ready to drop into a SQL string. Example:
|
||||
* quoteIdentifier('embedding_voyage') === '"embedding_voyage"'
|
||||
*/
|
||||
export function quoteIdentifier(name: string): string {
|
||||
return `"${name.replace(/"/g, '""')}"`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds the per-engine SQL fragments for a resolved column.
|
||||
*
|
||||
* Returns:
|
||||
* col — identifier-quoted column name for SELECT / WHERE / ORDER BY.
|
||||
* castSql — placeholder cast for the query vector parameter:
|
||||
* '$1::vector' for type='vector'
|
||||
* '$1::halfvec(<dims>)' for type='halfvec'
|
||||
*
|
||||
* Callers interpolate both into the SQL string. The `$1` is the
|
||||
* positional parameter that postgres.js / PGLite will bind the query
|
||||
* vector to. Different placeholders ($2, $3, etc.) can be obtained
|
||||
* by string-substitution from the caller; we standardize on $1 here
|
||||
* since both engines use it for the query vector and the cast lives
|
||||
* adjacent to the placeholder anyway.
|
||||
*/
|
||||
export function buildVectorCastFragment(resolved: ResolvedColumn): {
|
||||
col: string;
|
||||
castSql: string;
|
||||
} {
|
||||
const col = quoteIdentifier(resolved.name);
|
||||
const castSql =
|
||||
resolved.type === 'halfvec'
|
||||
? `$1::halfvec(${resolved.dimensions})`
|
||||
: `$1::vector`;
|
||||
return { col, castSql };
|
||||
}
|
||||
|
||||
// ---- Registry --------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Returns the merged registry: built-ins (`embedding`, `embedding_image`)
|
||||
* + user-declared `embedding_columns`. User declarations win on key
|
||||
* collision so power users can override a builtin's dim/provider (e.g.,
|
||||
* pointing 'embedding' at a Voyage column on a fresh brain).
|
||||
*
|
||||
* Throws if any user entry fails D12 validation. Built-ins are
|
||||
* derived live from the broader config (embedding_model,
|
||||
* embedding_dimensions, embedding_multimodal_model), so a missing
|
||||
* embedding_model produces a default 'embedding' entry pointing at the
|
||||
* gateway default. The doctor check catches mismatches against actual
|
||||
* DB column shape.
|
||||
*/
|
||||
export function getEmbeddingColumnRegistry(
|
||||
cfg: GBrainConfig,
|
||||
): Record<string, EmbeddingColumnConfig> {
|
||||
// Prototype-pollution safe (codex /ship #1). Plain `{}` inherits from
|
||||
// Object.prototype so `registry["constructor"]` returns
|
||||
// `Object.prototype.constructor` (a truthy function). Object.create(null)
|
||||
// creates a dict with NO prototype so unknown keys are genuinely absent.
|
||||
const out: Record<string, EmbeddingColumnConfig> = Object.create(null);
|
||||
|
||||
// Builtin: 'embedding' — derived from primary config keys.
|
||||
const embedModel = cfg.embedding_model ?? 'openai:text-embedding-3-large';
|
||||
const embedDims =
|
||||
typeof cfg.embedding_dimensions === 'number' && cfg.embedding_dimensions > 0
|
||||
? cfg.embedding_dimensions
|
||||
: 1536;
|
||||
out['embedding'] = {
|
||||
provider: embedModel,
|
||||
dimensions: embedDims,
|
||||
type: 'vector',
|
||||
};
|
||||
|
||||
// Builtin: 'embedding_image' — derived from multimodal config keys.
|
||||
// Hardcoded 1024d / vector because that's the committed schema shape
|
||||
// (see src/schema.sql:158). If the user runs a different multimodal
|
||||
// model they can override via the user registry.
|
||||
const mmModel = cfg.embedding_multimodal_model ?? 'voyage:voyage-multimodal-3';
|
||||
out['embedding_image'] = {
|
||||
provider: mmModel,
|
||||
dimensions: 1024,
|
||||
type: 'vector',
|
||||
};
|
||||
|
||||
// User-declared columns. Validate every key + entry at merge time.
|
||||
const userColumns = cfg.embedding_columns;
|
||||
if (userColumns && typeof userColumns === 'object' && !Array.isArray(userColumns)) {
|
||||
for (const [key, value] of Object.entries(userColumns)) {
|
||||
validateColumnKey(key);
|
||||
validateColumnConfig(key, value);
|
||||
out[key] = value;
|
||||
}
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the effective column descriptor for one query.
|
||||
*
|
||||
* Resolution chain:
|
||||
* 1. opts.embeddingColumn (per-call override, e.g. from MCP query op)
|
||||
* 2. cfg.search_embedding_column (DB-plane default)
|
||||
* 3. DEFAULT_COLUMN_NAME ('embedding')
|
||||
*
|
||||
* The resolved name is looked up in the merged registry; unknown names
|
||||
* throw EmbeddingColumnNotRegisteredError with a paste-ready hint.
|
||||
*
|
||||
* When `opts.embeddingColumn` is already a ResolvedColumn (engine-internal
|
||||
* shape after the boundary), it's returned as-is so re-resolving is a
|
||||
* no-op. This lets hybridSearch resolve once and pass the descriptor down
|
||||
* to multiple engine calls without redoing the work.
|
||||
*/
|
||||
export function resolveEmbeddingColumn(
|
||||
opts: Pick<SearchOpts, 'embeddingColumn'> | undefined,
|
||||
cfg: GBrainConfig,
|
||||
): ResolvedColumn {
|
||||
// Fast path: already resolved (engine-internal). Re-validate the
|
||||
// descriptor shape before returning (codex /ship #2). SDK callers
|
||||
// could pass a hand-rolled object via the public `gbrain/types`
|
||||
// surface; runtime check ensures the engine still sees a known-safe
|
||||
// shape. The validation is the same shape the resolver applies on
|
||||
// the string path.
|
||||
const candidate = opts?.embeddingColumn;
|
||||
if (
|
||||
candidate &&
|
||||
typeof candidate === 'object' &&
|
||||
!Array.isArray(candidate) &&
|
||||
typeof (candidate as ResolvedColumn).name === 'string'
|
||||
) {
|
||||
const r = candidate as ResolvedColumn;
|
||||
if (!COLUMN_NAME_REGEX.test(r.name)) {
|
||||
throw new EmbeddingColumnNotRegisteredError(r.name, []);
|
||||
}
|
||||
if (r.type !== 'vector' && r.type !== 'halfvec') {
|
||||
throw new EmbeddingColumnConfigError(r.name, 'type', `descriptor.type must be 'vector' or 'halfvec' (got: ${String(r.type)})`);
|
||||
}
|
||||
if (
|
||||
typeof r.dimensions !== 'number' ||
|
||||
!Number.isInteger(r.dimensions) ||
|
||||
r.dimensions < 1 ||
|
||||
r.dimensions > MAX_DIMENSIONS
|
||||
) {
|
||||
throw new EmbeddingColumnConfigError(r.name, 'dimensions', `descriptor.dimensions must be an integer in [1, ${MAX_DIMENSIONS}] (got: ${String(r.dimensions)})`);
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
// String chain.
|
||||
const requestedName =
|
||||
(typeof candidate === 'string' && candidate.length > 0 ? candidate : undefined) ??
|
||||
(typeof cfg.search_embedding_column === 'string' && cfg.search_embedding_column.length > 0
|
||||
? cfg.search_embedding_column
|
||||
: undefined) ??
|
||||
DEFAULT_COLUMN_NAME;
|
||||
|
||||
// Defense layer 1: regex on the requested name BEFORE registry lookup.
|
||||
// Even if someone bypasses load-time validation, the resolver refuses
|
||||
// to look up a name that doesn't look like an identifier.
|
||||
if (!COLUMN_NAME_REGEX.test(requestedName)) {
|
||||
throw new EmbeddingColumnNotRegisteredError(
|
||||
requestedName,
|
||||
[], // We don't have the registry yet at this point; render later.
|
||||
);
|
||||
}
|
||||
|
||||
const registry = getEmbeddingColumnRegistry(cfg);
|
||||
// Use Object.hasOwn so inherited keys (constructor, hasOwnProperty,
|
||||
// __proto__, etc.) cannot resolve. The registry itself uses
|
||||
// Object.create(null) but defense-in-depth here too — the codex /ship
|
||||
// #1 finding was specifically about resolveEmbeddingColumn's lookup.
|
||||
if (!Object.hasOwn(registry, requestedName)) {
|
||||
throw new EmbeddingColumnNotRegisteredError(
|
||||
requestedName,
|
||||
Object.keys(registry).sort(),
|
||||
);
|
||||
}
|
||||
const entry = registry[requestedName];
|
||||
|
||||
return {
|
||||
name: requestedName,
|
||||
type: entry.type,
|
||||
dimensions: entry.dimensions,
|
||||
embeddingModel: entry.provider,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the resolved column is the default `embedding` name.
|
||||
* Name-based check; does not compare embedding space.
|
||||
*/
|
||||
export function isDefaultColumn(resolved: ResolvedColumn): boolean {
|
||||
return resolved.name === DEFAULT_COLUMN_NAME;
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the resolved column's embedding space matches the
|
||||
* `query_cache.embedding` column's space — i.e., it's safe to read
|
||||
* from / write to the semantic query cache without dimension or
|
||||
* vector-space corruption.
|
||||
*
|
||||
* Codex /ship finding #4: `isDefaultColumn` is name-based, so a user
|
||||
* who overrides the `embedding` builtin to point at a different
|
||||
* provider/dim (legitimate use of the registry override semantics)
|
||||
* would still have their cache used — but the cache table is sized
|
||||
* for the ORIGINAL default embedding dim. Mismatched dim/model means
|
||||
* the cache is wrong-space; skip it.
|
||||
*
|
||||
* Cache-safety criteria:
|
||||
* 1. Column name is `embedding` (the cache table only knows about
|
||||
* this column; non-default columns always skip).
|
||||
* 2. Resolved dimensions match `cfg.embedding_dimensions` (or
|
||||
* DEFAULT_EMBEDDING_DIMENSIONS=1536 when unset).
|
||||
* 3. Resolved provider matches `cfg.embedding_model` (or the OpenAI
|
||||
* default). The model is the "embedding space identifier" — two
|
||||
* models produce non-interchangeable vectors even at the same
|
||||
* dim count.
|
||||
*
|
||||
* When any of these mismatch, return false so hybridSearchCached
|
||||
* skips both the lookup and the writeback paths.
|
||||
*/
|
||||
export function isCacheSafe(resolved: ResolvedColumn, cfg: GBrainConfig): boolean {
|
||||
if (resolved.name !== DEFAULT_COLUMN_NAME) return false;
|
||||
const cfgDims = (typeof cfg.embedding_dimensions === 'number' && cfg.embedding_dimensions > 0)
|
||||
? cfg.embedding_dimensions
|
||||
: 1536;
|
||||
if (resolved.dimensions !== cfgDims) return false;
|
||||
const cfgModel = cfg.embedding_model ?? 'openai:text-embedding-3-large';
|
||||
if (resolved.embeddingModel !== cfgModel) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Type guard: is this string a valid BuiltinKey? Useful for callers
|
||||
* that want to special-case the 'embedding_image' multimodal path
|
||||
* without doing a string-compare scattered throughout the code. */
|
||||
export function isBuiltinColumn(name: string): name is BuiltinKey {
|
||||
return (BUILTIN_KEYS as readonly string[]).includes(name);
|
||||
}
|
||||
|
||||
/**
|
||||
* Engine-side normalizer. Accepts the legacy SearchOpts.embeddingColumn
|
||||
* union (`'embedding'` | `'embedding_image'` | string | ResolvedColumn |
|
||||
* undefined) and returns a ResolvedColumn. Engine code calls this once at
|
||||
* the top of searchVector / getEmbeddingsByChunkIds. The engine remains
|
||||
* config-free — this function reads NO config, only handles the known
|
||||
* builtin shapes statically.
|
||||
*
|
||||
* Behavior:
|
||||
* - ResolvedColumn → returned as-is.
|
||||
* - undefined → builtin 'embedding' descriptor (vector, dims=1536).
|
||||
* - 'embedding' literal → same as undefined.
|
||||
* - 'embedding_image' literal → builtin 'embedding_image' descriptor.
|
||||
* - Any other string → throws EmbeddingColumnNotRegisteredError. The
|
||||
* resolver lives at hybrid/op boundary; bare string names are NOT
|
||||
* accepted at the engine layer (per D11 — engine purity).
|
||||
*
|
||||
* `dims` for the 'embedding' builtin is hardcoded 1536 here purely as
|
||||
* a no-op placeholder; the engine's SQL cast is `$1::vector` (no
|
||||
* parenthesized N) when type='vector', so the dims field is unused by
|
||||
* `buildVectorCastFragment` and never touches the wire. Tests that
|
||||
* care about dims should pass a real descriptor.
|
||||
*/
|
||||
export function normalizeEngineColumn(
|
||||
embeddingColumn: SearchOpts['embeddingColumn'] | undefined,
|
||||
): ResolvedColumn {
|
||||
// ResolvedColumn descriptor → use as-is.
|
||||
if (
|
||||
embeddingColumn &&
|
||||
typeof embeddingColumn === 'object' &&
|
||||
!Array.isArray(embeddingColumn) &&
|
||||
typeof (embeddingColumn as ResolvedColumn).name === 'string'
|
||||
) {
|
||||
return embeddingColumn as ResolvedColumn;
|
||||
}
|
||||
|
||||
// Default + legacy 'embedding' literal.
|
||||
if (embeddingColumn === undefined || embeddingColumn === 'embedding') {
|
||||
return {
|
||||
name: 'embedding',
|
||||
type: 'vector',
|
||||
dimensions: 1536, // placeholder — not used by $1::vector cast
|
||||
embeddingModel: '', // engine doesn't embed; left blank
|
||||
};
|
||||
}
|
||||
|
||||
// Legacy multimodal literal — committed schema shape per
|
||||
// src/schema.sql:158 (vector(1024)).
|
||||
if (embeddingColumn === 'embedding_image') {
|
||||
return {
|
||||
name: 'embedding_image',
|
||||
type: 'vector',
|
||||
dimensions: 1024,
|
||||
embeddingModel: '',
|
||||
};
|
||||
}
|
||||
|
||||
// Any other raw string at this layer is a programming error — the
|
||||
// resolver should have run at the hybrid/op boundary and produced a
|
||||
// descriptor. We throw with a paste-ready hint rather than guess.
|
||||
throw new EmbeddingColumnNotRegisteredError(String(embeddingColumn), [
|
||||
'embedding',
|
||||
'embedding_image',
|
||||
'<custom name via resolveEmbeddingColumn at hybrid/op boundary>',
|
||||
]);
|
||||
}
|
||||
+80
-11
@@ -13,6 +13,8 @@ import type { BrainEngine } from '../engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts';
|
||||
import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts';
|
||||
import { embed, embedQuery } from '../embedding.ts';
|
||||
import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts';
|
||||
import { loadConfigWithEngine } from '../config.ts';
|
||||
import { dedupResults } from './dedup.ts';
|
||||
import { applyReranker } from './rerank.ts';
|
||||
import { autoDetectDetail, classifyQuery } from './query-intent.ts';
|
||||
@@ -358,6 +360,19 @@ export async function hybridSearch(
|
||||
},
|
||||
});
|
||||
|
||||
// v0.36 (D7+D11): resolve embedding column once at entry. Single
|
||||
// round-trip to read DB-plane config (mirrors loadSearchModeConfig).
|
||||
// Resolver throws on unknown name with a paste-ready hint; let it
|
||||
// propagate — a misconfig should be loud, not silently fall back.
|
||||
// Failing cfg load (pre-config brain, mid-migration, no engine.getConfig)
|
||||
// falls through to the file-plane sync loadConfig() — same shape, just
|
||||
// misses DB-plane overrides.
|
||||
const mergedCfg = await loadConfigWithEngine(engine).catch(() => null);
|
||||
const cfgForColumn = mergedCfg ?? ((await import('../config.ts')).loadConfig()) ?? null;
|
||||
const resolvedCol = cfgForColumn
|
||||
? resolveEmbeddingColumn(opts, cfgForColumn)
|
||||
: resolveEmbeddingColumn(opts, { engine: 'pglite' });
|
||||
|
||||
const limit = opts?.limit || resolvedMode.searchLimit;
|
||||
const offset = opts?.offset || 0;
|
||||
const innerLimit = Math.min(limit * 2, MAX_SEARCH_LIMIT);
|
||||
@@ -402,6 +417,10 @@ export async function hybridSearch(
|
||||
// ordering means we can't lazy-spread the full opts).
|
||||
sourceId: opts?.sourceId,
|
||||
sourceIds: opts?.sourceIds,
|
||||
// v0.36 (D11): pass the pre-validated descriptor into the engine so
|
||||
// it never has to read config. Engines normalize string-or-descriptor
|
||||
// via normalizeEngineColumn; the descriptor path is the strict one.
|
||||
embeddingColumn: resolvedCol,
|
||||
};
|
||||
// Track what actually ran for the optional onMeta callback (v0.25.0).
|
||||
// Caller leaves onMeta undefined → these flags are computed but never
|
||||
@@ -478,8 +497,13 @@ export async function hybridSearch(
|
||||
};
|
||||
|
||||
// Skip vector search entirely if the gateway has no embedding provider configured (Codex C3).
|
||||
// v0.36 (D10): ask "is the RESOLVED column's provider reachable?" rather
|
||||
// than "is the global default reachable?" — otherwise an unreachable
|
||||
// global default disables vector search even when the active column's
|
||||
// provider (Voyage, ZE) works fine.
|
||||
const { isAvailable } = await import('../ai/gateway.ts');
|
||||
if (!isAvailable('embedding')) {
|
||||
const providerProbe = resolvedCol.embeddingModel || undefined;
|
||||
if (!isAvailable('embedding', providerProbe)) {
|
||||
if (keywordResults.length > 0) {
|
||||
await runPostFusionStages(engine, keywordResults, postFusionOpts);
|
||||
keywordResults.sort((a, b) => b.score - a.score);
|
||||
@@ -494,6 +518,7 @@ export async function hybridSearch(
|
||||
expansion_applied: false,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
embedding_column: resolvedCol.name,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: noEmbedBudgetMeta }
|
||||
: {}),
|
||||
@@ -526,7 +551,15 @@ export async function hybridSearch(
|
||||
// v0.35.0.0+: query-side embedding. For asymmetric providers (ZE zembed-1,
|
||||
// Voyage v3+) routes input_type='query' through the embed seam; symmetric
|
||||
// providers ignore the field — no behavior change.
|
||||
const embeddings = await Promise.all(queries.map(q => embedQuery(q)));
|
||||
// v0.36 (D10): route through the resolved column's provider + dims so
|
||||
// a query against `embedding_voyage` actually embeds via Voyage, not
|
||||
// the global default (OpenAI). Empty embeddingModel falls back to
|
||||
// gateway default — preserves pre-v0.36 behavior for the builtin
|
||||
// 'embedding' column.
|
||||
const embedOpts = resolvedCol.embeddingModel
|
||||
? { embeddingModel: resolvedCol.embeddingModel, dimensions: resolvedCol.dimensions }
|
||||
: undefined;
|
||||
const embeddings = await Promise.all(queries.map(q => embedQuery(q, embedOpts)));
|
||||
queryEmbedding = embeddings[0];
|
||||
vectorLists = await Promise.all(
|
||||
embeddings.map(emb => engine.searchVector(emb, searchOpts)),
|
||||
@@ -554,6 +587,7 @@ export async function hybridSearch(
|
||||
expansion_applied: expansionApplied,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
embedding_column: resolvedCol.name,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: kwBudgetMeta }
|
||||
: {}),
|
||||
@@ -577,9 +611,12 @@ export async function hybridSearch(
|
||||
];
|
||||
let fused = rrfFusionWeighted(allLists, detail !== 'high');
|
||||
|
||||
// Cosine re-scoring before dedup so semantically better chunks survive
|
||||
// Cosine re-scoring before dedup so semantically better chunks survive.
|
||||
// v0.36 (D9): hydrate from the active embedding column so rescore happens
|
||||
// in the same vector space the HNSW just ranked in. Pre-v0.36 this
|
||||
// always pulled from `embedding` and silently corrupted alt-column ranks.
|
||||
if (queryEmbedding) {
|
||||
fused = await cosineReScore(engine, fused, queryEmbedding);
|
||||
fused = await cosineReScore(engine, fused, queryEmbedding, resolvedCol.name);
|
||||
}
|
||||
|
||||
// v0.29.1: post-fusion stages (backlink + salience + recency) run via
|
||||
@@ -687,6 +724,7 @@ export async function hybridSearch(
|
||||
expansion_applied: expansionApplied,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
embedding_column: resolvedCol.name,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
: {}),
|
||||
@@ -742,7 +780,26 @@ export async function hybridSearchCached(
|
||||
floor_ratio: opts?.floorRatio,
|
||||
},
|
||||
});
|
||||
const cacheKnobsHash = knobsHash(resolvedForCache);
|
||||
// v0.36 (D8 / CDX-2 + codex /ship #4): resolve column for the cache
|
||||
// decision. The query_cache.embedding column has one fixed pgvector dim
|
||||
// sized at brain init; storing a 1024d Voyage or 2560d ZE cache
|
||||
// embedding fails or corrupts results. Name-based check ("is it the
|
||||
// default `embedding` column?") is insufficient — the registry
|
||||
// explicitly allows overriding builtin `embedding` to a different
|
||||
// provider/dim. isCacheSafe compares the resolved column's full
|
||||
// embedding space (name + dim + model) against cfg and returns true
|
||||
// only when ALL match. Otherwise skip.
|
||||
const mergedCfgCached = await loadConfigWithEngine(engine).catch(() => null);
|
||||
const cfgCached = mergedCfgCached ?? ((await import('../config.ts')).loadConfig()) ?? { engine: 'pglite' as const };
|
||||
const resolvedColCached = resolveEmbeddingColumn(opts, cfgCached);
|
||||
const isNonDefaultColumn = !isCacheSafe(resolvedColCached, cfgCached);
|
||||
|
||||
// Cache key carries the column + provider so different embedding spaces
|
||||
// never collide on the same `(source_id, query_text)` row.
|
||||
const cacheKnobsHash = knobsHash(resolvedForCache, {
|
||||
embeddingColumn: resolvedColCached.name,
|
||||
embeddingModel: resolvedColCached.embeddingModel,
|
||||
});
|
||||
|
||||
// Cache decision: opts.useCache (explicit) wins over global config; global
|
||||
// config wins over mode bundle default. Mode bundle is on for all 3 modes
|
||||
@@ -756,14 +813,15 @@ export async function hybridSearchCached(
|
||||
ttlSeconds: resolvedForCache.cache_ttl_seconds,
|
||||
});
|
||||
|
||||
// Skip cache entirely when the request asks for two-pass walks or has
|
||||
// a non-default embedding column — those interact with structural state
|
||||
// that the cache can't safely express.
|
||||
// Skip cache entirely when the request asks for two-pass walks, has
|
||||
// a non-default embedding column (per-call or via config default —
|
||||
// D8 closes the silent-corruption bug class), or near-symbol mode
|
||||
// (structural state that the cache can't safely express).
|
||||
const skipCache =
|
||||
!cache.isEnabled() ||
|
||||
(opts?.walkDepth ?? 0) > 0 ||
|
||||
Boolean(opts?.nearSymbol) ||
|
||||
(opts?.embeddingColumn && opts.embeddingColumn !== 'embedding');
|
||||
isNonDefaultColumn;
|
||||
|
||||
let cacheStatus: 'hit' | 'miss' | 'disabled' = skipCache ? 'disabled' : 'miss';
|
||||
let cacheSimilarity: number | undefined;
|
||||
@@ -778,7 +836,13 @@ export async function hybridSearchCached(
|
||||
if (!skipCache) {
|
||||
try {
|
||||
const { isAvailable } = await import('../ai/gateway.ts');
|
||||
if (isAvailable('embedding')) {
|
||||
// v0.36 (D10): for the cache-lookup embedding, also use the resolved
|
||||
// column's provider. The cache lookup is always against the default
|
||||
// 'embedding' column (skipCache short-circuits non-default above),
|
||||
// so this is the default embeddingModel — but threading it keeps
|
||||
// the provider probe consistent with the bare hybridSearch path.
|
||||
const providerProbeCached = resolvedColCached.embeddingModel || undefined;
|
||||
if (isAvailable('embedding', providerProbeCached)) {
|
||||
// v0.35.0.0+: query-side embedding (cache lookup path).
|
||||
queryEmbedding = await embedQuery(query);
|
||||
} else {
|
||||
@@ -983,6 +1047,7 @@ async function cosineReScore(
|
||||
engine: BrainEngine,
|
||||
results: SearchResult[],
|
||||
queryEmbedding: Float32Array,
|
||||
column: string = 'embedding',
|
||||
): Promise<SearchResult[]> {
|
||||
const chunkIds = results
|
||||
.map(r => r.chunk_id)
|
||||
@@ -992,7 +1057,11 @@ async function cosineReScore(
|
||||
|
||||
let embeddingMap: Map<number, Float32Array>;
|
||||
try {
|
||||
embeddingMap = await engine.getEmbeddingsByChunkIds(chunkIds);
|
||||
// v0.36 (D9): hydrate from the active column so rescore happens in
|
||||
// the same embedding space the HNSW just ranked in. Without this,
|
||||
// a Voyage HNSW retrieval would HNSW-rank against Voyage vectors but
|
||||
// rescore against OpenAI vectors → NaN or wrong rankings.
|
||||
embeddingMap = await engine.getEmbeddingsByChunkIds(chunkIds, column);
|
||||
} catch {
|
||||
// DB error is non-fatal, return results without re-scoring
|
||||
return results;
|
||||
|
||||
+35
-4
@@ -368,7 +368,28 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// `cache.ttl_seconds` (default 3600s). The CHANGELOG note covers this.
|
||||
export const KNOBS_HASH_VERSION = 3;
|
||||
|
||||
export function knobsHash(knobs: ResolvedSearchKnobs): string {
|
||||
/**
|
||||
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
|
||||
* embedding column + provider live OUTSIDE ResolvedSearchKnobs because
|
||||
* they're orthogonal to search mode (mode bundles don't pick columns).
|
||||
* Passing them as a second argument keeps ModeBundle pure and lets the
|
||||
* hash invalidate correctly across column/provider switches.
|
||||
*
|
||||
* When undefined, the hash falls back to the legacy 'embedding' /
|
||||
* 'default' values so unrelated callers (eval-replay, telemetry) that
|
||||
* don't know the column produce a stable hash for the default case.
|
||||
*/
|
||||
export interface KnobsHashContext {
|
||||
/** Resolved column name, e.g. 'embedding', 'embedding_voyage'. */
|
||||
embeddingColumn?: string;
|
||||
/** Resolved provider:model, e.g. 'voyage:voyage-3-large'. */
|
||||
embeddingModel?: string;
|
||||
}
|
||||
|
||||
export function knobsHash(
|
||||
knobs: ResolvedSearchKnobs,
|
||||
ctx?: KnobsHashContext,
|
||||
): string {
|
||||
// Fixed-order key list. Adding a knob here REQUIRES bumping
|
||||
// KNOBS_HASH_VERSION and is a breaking change for any persisted cache.
|
||||
const parts = [
|
||||
@@ -387,10 +408,20 @@ export function knobsHash(knobs: ResolvedSearchKnobs): string {
|
||||
`rri=${knobs.reranker_top_n_in}`,
|
||||
`rro=${knobs.reranker_top_n_out ?? 'none'}`,
|
||||
`rrt=${knobs.reranker_timeout_ms}`,
|
||||
// v=3 additions (append-only). Use 4-decimal precision so 0.85 and
|
||||
// 0.851 differ in the hash; undefined uses literal 'none' so a
|
||||
// floor-off write and a floor-on write key into different rows.
|
||||
// v=3 additions (append-only). Both contributions landed under v=3:
|
||||
//
|
||||
// floor_ratio (v0.35.6.0 / codex T1): a floor-on write must not be
|
||||
// served to a floor-off lookup. 4-decimal precision so 0.85 and
|
||||
// 0.851 produce different hashes; undefined uses literal 'none'.
|
||||
//
|
||||
// col + prov (v0.36 / D8 / CDX-2): cross-column + cross-provider
|
||||
// cache contamination. A query against `embedding_voyage` must
|
||||
// NEVER be served from a cache row that ran against `embedding`
|
||||
// — they sit in different vector spaces. ctx is optional so
|
||||
// unrelated callers fall back to the default-column hash.
|
||||
`fr=${knobs.floor_ratio === undefined ? 'none' : knobs.floor_ratio.toFixed(4)}`,
|
||||
`col=${ctx?.embeddingColumn ?? 'embedding'}`,
|
||||
`prov=${ctx?.embeddingModel ?? 'default'}`,
|
||||
];
|
||||
const h = createHash('sha256');
|
||||
h.update(parts.join('|'));
|
||||
|
||||
+87
-7
@@ -421,6 +421,52 @@ export interface SearchResult {
|
||||
effective_date_source?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36 (D12) — declared shape of one entry in the `embedding_columns`
|
||||
* config registry. Users declare entries via `gbrain config set
|
||||
* embedding_columns '...JSON...'` (DB plane); resolver merges with
|
||||
* built-ins. Field validation lives in src/core/search/embedding-column.ts
|
||||
* and runs at load time:
|
||||
*
|
||||
* provider: parseable 'provider:model' (e.g. 'voyage:voyage-3-large')
|
||||
* dimensions: positive integer 1..8192
|
||||
* type: 'vector' | 'halfvec'
|
||||
*
|
||||
* The registry KEY itself (the column name) is also regex-validated
|
||||
* (/^[a-z_][a-z0-9_]*$/) so a malicious config write can't smuggle a
|
||||
* SQL fragment in via the key.
|
||||
*/
|
||||
export interface EmbeddingColumnConfig {
|
||||
/** 'provider:model' identifier, e.g. 'voyage:voyage-3-large'. */
|
||||
provider: string;
|
||||
/** Dimensions of the stored vector. Must match actual DB column. */
|
||||
dimensions: number;
|
||||
/** pgvector type — drives the SQL cast at search time. */
|
||||
type: 'vector' | 'halfvec';
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36 (D11) — fully resolved descriptor for an embedding column. The
|
||||
* resolver (src/core/search/embedding-column.ts) produces these at the
|
||||
* hybrid/op boundary; engines consume them. The engine NEVER reads config
|
||||
* or calls the resolver itself — it sees only this validated descriptor.
|
||||
*
|
||||
* `name` is identifier-quoted at SQL-build time by buildVectorCastFragment
|
||||
* (D12 defense in depth), but the resolver also enforces a strict regex
|
||||
* at load time so this string is already known-safe by the time the engine
|
||||
* sees it.
|
||||
*/
|
||||
export interface ResolvedColumn {
|
||||
/** Column name in content_chunks (already validated against the registry). */
|
||||
name: string;
|
||||
/** pgvector type — `$N::vector` or `$N::halfvec(N)`. */
|
||||
type: 'vector' | 'halfvec';
|
||||
/** Embedding dimensions — must match actual DB column dim. */
|
||||
dimensions: number;
|
||||
/** 'provider:model' identifier, e.g. 'voyage:voyage-3-large'. */
|
||||
embeddingModel: string;
|
||||
}
|
||||
|
||||
export interface SearchOpts {
|
||||
limit?: number;
|
||||
offset?: number;
|
||||
@@ -490,14 +536,27 @@ export interface SearchOpts {
|
||||
*/
|
||||
sourceIds?: string[];
|
||||
/**
|
||||
* v0.27.1: target column for vector search. 'embedding' (default) hits
|
||||
* the brain's primary text-embedding column. 'embedding_image' targets
|
||||
* the multimodal column populated by importImageFile. The two columns
|
||||
* may live in different dim spaces (e.g. OpenAI 1536 + Voyage 1024)
|
||||
* which is why the dual-column schema landed in v0.27.1. searchKeyword
|
||||
* is unaffected — modality filtering on the keyword path is independent.
|
||||
* v0.27.1 / v0.36 (D11): target column for vector search. Two shapes:
|
||||
*
|
||||
* 1. String name (legacy + user-facing). Engine and hybridSearch convert
|
||||
* to ResolvedColumn at the boundary via `resolveEmbeddingColumn()`.
|
||||
* Built-in names: 'embedding' (default, text), 'embedding_image'
|
||||
* (multimodal). Custom user-declared names also accepted when
|
||||
* registered in `embedding_columns` config.
|
||||
*
|
||||
* 2. ResolvedColumn descriptor (internal). The engine ONLY accepts
|
||||
* this shape — hybridSearch resolves once at entry and passes the
|
||||
* descriptor in so the engine stays config-independent (D11 — engine
|
||||
* is a pure SQL composer).
|
||||
*
|
||||
* The two-shape union is the transition seam. External callers
|
||||
* (operations.ts image branch, tests, gbrain-evals) pass strings;
|
||||
* hybridSearch resolves; engine sees descriptor.
|
||||
*
|
||||
* searchKeyword is unaffected — modality filtering on the keyword path
|
||||
* is independent.
|
||||
*/
|
||||
embeddingColumn?: 'embedding' | 'embedding_image';
|
||||
embeddingColumn?: 'embedding' | 'embedding_image' | string | ResolvedColumn;
|
||||
/**
|
||||
* @deprecated v0.29.1: use `since` instead. Removed in v0.30.
|
||||
* v0.27.0: filter results to pages updated/created after this date. ISO-8601 string.
|
||||
@@ -872,6 +931,19 @@ export interface EvalCandidateInput {
|
||||
remote: boolean;
|
||||
job_id: number | null;
|
||||
subagent_id: number | null;
|
||||
/**
|
||||
* v0.36 (D16 / CDX-10): the embedding column resolved at capture time.
|
||||
* Optional for back-compat with pre-v0.36 callers + test fixtures.
|
||||
* Engines coalesce undefined to NULL at insert time so the column is
|
||||
* always either a known column name or DB NULL — never JS undefined.
|
||||
*
|
||||
* Replay (`gbrain eval replay`) re-runs the captured query against the
|
||||
* brain. Without this field, a replay after the user flipped
|
||||
* `search_embedding_column` produces "regressions" that are just
|
||||
* column changes. Replay reads this and uses it as a per-row override
|
||||
* so capture and replay run in the same embedding space.
|
||||
*/
|
||||
embedding_column?: string | null;
|
||||
}
|
||||
|
||||
export interface EvalCandidate extends EvalCandidateInput {
|
||||
@@ -945,6 +1017,14 @@ export interface HybridSearchMeta {
|
||||
* the operator's `config.search.mode` setting if per-call overrides win).
|
||||
*/
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
/**
|
||||
* v0.36 (D16 / CDX-10): the embedding column that actually ran this
|
||||
* search. Threaded through to eval_candidates capture so replay can
|
||||
* use the same column even after `search_embedding_column` is
|
||||
* flipped. Always set on hybridSearch paths; omitted on
|
||||
* non-hybridSearch capture paths (keyword-only `search` op).
|
||||
*/
|
||||
embedding_column?: string;
|
||||
}
|
||||
|
||||
// Config
|
||||
|
||||
+7
-1
@@ -787,7 +787,13 @@ CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
salience_resolved TEXT,
|
||||
recency_resolved TEXT,
|
||||
salience_source TEXT,
|
||||
recency_source TEXT
|
||||
recency_source TEXT,
|
||||
-- v0.36.3.0 (D16 / CDX-10) — embedding column resolved at capture time so
|
||||
-- `gbrain eval replay` reproduces the same column the capture ran against.
|
||||
-- Nullable; pre-v0.36 rows have NULL and replay falls back to current
|
||||
-- default. Migration v68 (src/core/migrate.ts) adds the same column on
|
||||
-- upgrade brains.
|
||||
embedding_column TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
|
||||
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
/**
|
||||
* v0.36 (D9 / CDX-3) — getEmbeddingsByChunkIds column-parameter tests.
|
||||
*
|
||||
* Pins:
|
||||
* - Default param value 'embedding' preserves pre-v0.36 behavior.
|
||||
* - Custom column parameter (e.g. 'embedding_voyage') hydrates from
|
||||
* that column.
|
||||
* - Invalid column name (regex-failing) throws
|
||||
* EmbeddingColumnNotRegisteredError BEFORE any SQL runs.
|
||||
* - Identifier-quoting safely interpolates the column name.
|
||||
*
|
||||
* Uses PGLite in-memory engine. ALTER TABLE adds an ad-hoc
|
||||
* `embedding_voyage` column to mimic the user's production state
|
||||
* where columns get added outside the committed schema.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from '../src/core/search/embedding-column.ts';
|
||||
import type { PageInput, ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let chunkId: number;
|
||||
let chunkId2: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Add an ad-hoc voyage column at the same shape Garry's brain has —
|
||||
// outside the committed schema, declared per-instance.
|
||||
await (engine as any).db.exec(
|
||||
`ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_voyage vector(1024)`,
|
||||
);
|
||||
|
||||
// Seed a page with two chunks.
|
||||
const page: PageInput = {
|
||||
type: 'concept',
|
||||
title: 'Cosine Rescore Test',
|
||||
compiled_truth: 'Two chunks for rescore.',
|
||||
};
|
||||
await engine.putPage('test/cosine-rescore', page);
|
||||
|
||||
const chunks: ChunkInput[] = [
|
||||
{ chunk_index: 0, chunk_text: 'first chunk text', chunk_source: 'compiled_truth' },
|
||||
{ chunk_index: 1, chunk_text: 'second chunk text', chunk_source: 'compiled_truth' },
|
||||
];
|
||||
await engine.upsertChunks('test/cosine-rescore', chunks);
|
||||
|
||||
// Read back chunk ids.
|
||||
const rows = await engine.executeRaw<{ id: number; chunk_index: number }>(
|
||||
`SELECT cc.id, cc.chunk_index FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.slug = 'test/cosine-rescore'
|
||||
ORDER BY cc.chunk_index`,
|
||||
);
|
||||
chunkId = rows[0].id;
|
||||
chunkId2 = rows[1].id;
|
||||
|
||||
// Plant distinct vectors in 'embedding' (1536d) and 'embedding_voyage'
|
||||
// (1024d) so we can prove the column parameter actually selects.
|
||||
const v1536a = new Array(1536).fill(0).map(() => 0.001).join(',');
|
||||
const v1536b = new Array(1536).fill(0).map(() => 0.002).join(',');
|
||||
const v1024a = new Array(1024).fill(0).map(() => 0.5).join(',');
|
||||
const v1024b = new Array(1024).fill(0).map(() => 0.6).join(',');
|
||||
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`,
|
||||
[`[${v1536a}]`, chunkId],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`,
|
||||
[`[${v1536b}]`, chunkId2],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`,
|
||||
[`[${v1024a}]`, chunkId],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`,
|
||||
[`[${v1024b}]`, chunkId2],
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('getEmbeddingsByChunkIds — column parameter (D9)', () => {
|
||||
test('default param fetches from "embedding" — preserves pre-v0.36 behavior', async () => {
|
||||
const map = await engine.getEmbeddingsByChunkIds([chunkId, chunkId2]);
|
||||
expect(map.size).toBe(2);
|
||||
expect(map.get(chunkId)!.length).toBe(1536);
|
||||
expect(map.get(chunkId2)!.length).toBe(1536);
|
||||
// First-element matches what we seeded for the primary column.
|
||||
expect(map.get(chunkId)![0]).toBeCloseTo(0.001, 5);
|
||||
});
|
||||
|
||||
test('column="embedding_voyage" fetches from the alt column', async () => {
|
||||
const map = await engine.getEmbeddingsByChunkIds([chunkId, chunkId2], 'embedding_voyage');
|
||||
expect(map.size).toBe(2);
|
||||
expect(map.get(chunkId)!.length).toBe(1024);
|
||||
expect(map.get(chunkId2)!.length).toBe(1024);
|
||||
// First-element matches what we seeded for voyage.
|
||||
expect(map.get(chunkId)![0]).toBeCloseTo(0.5, 5);
|
||||
});
|
||||
|
||||
test('invalid column param throws BEFORE SQL runs (regex-rejected)', async () => {
|
||||
let threw: Error | null = null;
|
||||
try {
|
||||
await engine.getEmbeddingsByChunkIds([chunkId], 'embedding"; DROP --');
|
||||
} catch (e) {
|
||||
threw = e as Error;
|
||||
}
|
||||
expect(threw).toBeTruthy();
|
||||
expect(threw).toBeInstanceOf(EmbeddingColumnNotRegisteredError);
|
||||
});
|
||||
|
||||
test('empty id list short-circuits (no SQL run)', async () => {
|
||||
const map = await engine.getEmbeddingsByChunkIds([], 'embedding_voyage');
|
||||
expect(map.size).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,243 @@
|
||||
/**
|
||||
* v0.36 E2E — dynamic embedding column selection (PGLite).
|
||||
*
|
||||
* Covers (per D4 + D9 + D11 + D12 + CDX-2 + CDX-3 + CDX-7 + CDX-8 + CDX-10):
|
||||
* - Multi-column search: same query against `embedding` and against an
|
||||
* ad-hoc `embedding_voyage` column produces different orderings
|
||||
* consistent with the seeded vectors.
|
||||
* - Halfvec column: ALTER TABLE ADD `embedding_ze halfvec(2560)` and
|
||||
* confirm the `$1::halfvec(2560)` cast works.
|
||||
* - Image branch unaffected: `embedding_image` still works via the
|
||||
* existing operations.ts path.
|
||||
* - cosineReScore reads from the active column, not the default
|
||||
* (D9 — pre-fix, rescore against Voyage HNSW used OpenAI vectors).
|
||||
* - Unknown column at hybridSearch entry throws loud.
|
||||
* - Mid-session column switch invalidates the cache (knobs_hash v=3).
|
||||
*
|
||||
* No DATABASE_URL needed — PGLite in-memory.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { hybridSearch } from '../../src/core/search/hybrid.ts';
|
||||
import {
|
||||
buildVectorCastFragment,
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from '../../src/core/search/embedding-column.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setEmbedTransportForTests,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
import type { ResolvedColumn } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let chunkIdA: number;
|
||||
let chunkIdB: number;
|
||||
|
||||
const VEC1536_A = new Array(1536).fill(0).map((_, i) => 0.001 * (i % 10));
|
||||
const VEC1536_B = new Array(1536).fill(0).map((_, i) => 0.002 * (i % 10));
|
||||
const VEC1024_A = new Array(1024).fill(0).map((_, i) => 0.5 - 0.001 * (i % 10));
|
||||
const VEC1024_B = new Array(1024).fill(0).map((_, i) => 0.4 + 0.001 * (i % 10));
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Add the ad-hoc Voyage + ZE columns the way a user with a multi-provider
|
||||
// brain has done it (outside the committed schema, per-instance ALTER).
|
||||
await (engine as any).db.exec(
|
||||
`ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_voyage vector(1024)`,
|
||||
);
|
||||
await (engine as any).db.exec(
|
||||
`ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_ze halfvec(2560)`,
|
||||
);
|
||||
|
||||
// Two pages with one chunk each.
|
||||
await engine.putPage('docs/page-a', {
|
||||
type: 'concept',
|
||||
title: 'Page A — about cats',
|
||||
compiled_truth: 'Page A discusses cats and their behavior.',
|
||||
});
|
||||
await engine.putPage('docs/page-b', {
|
||||
type: 'concept',
|
||||
title: 'Page B — about dogs',
|
||||
compiled_truth: 'Page B discusses dogs and their habits.',
|
||||
});
|
||||
|
||||
await engine.upsertChunks('docs/page-a', [
|
||||
{ chunk_index: 0, chunk_text: 'cats behavior chunk A', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.upsertChunks('docs/page-b', [
|
||||
{ chunk_index: 0, chunk_text: 'dogs habits chunk B', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
|
||||
// Look up chunk ids.
|
||||
const rows = await engine.executeRaw<{ id: number; slug: string }>(
|
||||
`SELECT cc.id, p.slug FROM content_chunks cc JOIN pages p ON p.id = cc.page_id ORDER BY p.slug`,
|
||||
);
|
||||
chunkIdA = rows.find(r => r.slug === 'docs/page-a')!.id;
|
||||
chunkIdB = rows.find(r => r.slug === 'docs/page-b')!.id;
|
||||
|
||||
// Seed vectors. Vectors are intentionally distinct between columns so
|
||||
// search orderings depend on which column the engine actually reads.
|
||||
const vecLit = (arr: number[]) => `[${arr.join(',')}]`;
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`,
|
||||
[vecLit(VEC1536_A), chunkIdA],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding = $1::vector WHERE id = $2`,
|
||||
[vecLit(VEC1536_B), chunkIdB],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`,
|
||||
[vecLit(VEC1024_A), chunkIdA],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding_voyage = $1::vector WHERE id = $2`,
|
||||
[vecLit(VEC1024_B), chunkIdB],
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('PGLite engine: searchVector accepts ResolvedColumn descriptor (D11)', () => {
|
||||
test('vector cast routes to correct column when descriptor names embedding_voyage', async () => {
|
||||
const queryVec = new Float32Array(VEC1024_A);
|
||||
const descriptor: ResolvedColumn = {
|
||||
name: 'embedding_voyage',
|
||||
type: 'vector',
|
||||
dimensions: 1024,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
};
|
||||
const results = await engine.searchVector(queryVec, {
|
||||
embeddingColumn: descriptor,
|
||||
limit: 5,
|
||||
});
|
||||
// Both pages have voyage embeddings; cosine to VEC1024_A is closer to
|
||||
// page-a (identical) than page-b. Verify ordering.
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
expect(results[0].slug).toBe('docs/page-a');
|
||||
});
|
||||
|
||||
test('halfvec cast accepted: ALTER TABLE column + $1::halfvec(N)', async () => {
|
||||
// Seed halfvec values via direct cast.
|
||||
const ze1 = `[${new Array(2560).fill(0.5).join(',')}]`;
|
||||
const ze2 = `[${new Array(2560).fill(0.6).join(',')}]`;
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding_ze = $1::halfvec WHERE id = $2`,
|
||||
[ze1, chunkIdA],
|
||||
);
|
||||
await (engine as any).db.query(
|
||||
`UPDATE content_chunks SET embedding_ze = $1::halfvec WHERE id = $2`,
|
||||
[ze2, chunkIdB],
|
||||
);
|
||||
|
||||
const queryVec = new Float32Array(2560).fill(0.5);
|
||||
const descriptor: ResolvedColumn = {
|
||||
name: 'embedding_ze',
|
||||
type: 'halfvec',
|
||||
dimensions: 2560,
|
||||
embeddingModel: 'zeroentropyai:zembed-1',
|
||||
};
|
||||
const results = await engine.searchVector(queryVec, {
|
||||
embeddingColumn: descriptor,
|
||||
limit: 5,
|
||||
});
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
// Page A's halfvec is closer to the all-0.5 query.
|
||||
expect(results[0].slug).toBe('docs/page-a');
|
||||
});
|
||||
|
||||
test('legacy embedding_image literal still routes correctly', async () => {
|
||||
// We never seeded embedding_image so we expect zero results, but the
|
||||
// query MUST NOT throw — the legacy-literal path must still work
|
||||
// (no regression on the existing image branch).
|
||||
const v = new Float32Array(1024).fill(0.1);
|
||||
const results = await engine.searchVector(v, {
|
||||
embeddingColumn: 'embedding_image',
|
||||
limit: 5,
|
||||
});
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('PGLite engine: getEmbeddingsByChunkIds column param (D9)', () => {
|
||||
test('default fetches from embedding (back-compat)', async () => {
|
||||
const map = await engine.getEmbeddingsByChunkIds([chunkIdA, chunkIdB]);
|
||||
expect(map.get(chunkIdA)!.length).toBe(1536);
|
||||
});
|
||||
|
||||
test('column="embedding_voyage" fetches from voyage column', async () => {
|
||||
const map = await engine.getEmbeddingsByChunkIds([chunkIdA, chunkIdB], 'embedding_voyage');
|
||||
expect(map.get(chunkIdA)!.length).toBe(1024);
|
||||
});
|
||||
|
||||
test('invalid column rejected at engine layer (regex guard)', async () => {
|
||||
let threw: Error | null = null;
|
||||
try {
|
||||
await engine.getEmbeddingsByChunkIds([chunkIdA], 'embed-bad-name');
|
||||
} catch (e) {
|
||||
threw = e as Error;
|
||||
}
|
||||
expect(threw).toBeInstanceOf(EmbeddingColumnNotRegisteredError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch + resolver — unknown column at entry (D11)', () => {
|
||||
test('unknown name in opts.embeddingColumn throws via resolver', async () => {
|
||||
// configureGateway with a transport stub so we don't hit a real API.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test' },
|
||||
});
|
||||
__setEmbedTransportForTests(async () => ({
|
||||
embeddings: [new Array(1536).fill(0)],
|
||||
usage: { tokens: 0 },
|
||||
} as any));
|
||||
|
||||
let threw: Error | null = null;
|
||||
try {
|
||||
await hybridSearch(engine, 'cats', {
|
||||
embeddingColumn: 'nonexistent_column',
|
||||
limit: 5,
|
||||
});
|
||||
} catch (e) {
|
||||
threw = e as Error;
|
||||
}
|
||||
expect(threw).toBeInstanceOf(EmbeddingColumnNotRegisteredError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildVectorCastFragment — engine SQL composer (D3)', () => {
|
||||
test('vector descriptor emits $1::vector', () => {
|
||||
const r: ResolvedColumn = {
|
||||
name: 'embedding',
|
||||
type: 'vector',
|
||||
dimensions: 1536,
|
||||
embeddingModel: '',
|
||||
};
|
||||
const { col, castSql } = buildVectorCastFragment(r);
|
||||
expect(col).toBe('"embedding"');
|
||||
expect(castSql).toBe('$1::vector');
|
||||
});
|
||||
|
||||
test('halfvec descriptor emits $1::halfvec(N) with parenthesized N', () => {
|
||||
const r: ResolvedColumn = {
|
||||
name: 'embedding_ze',
|
||||
type: 'halfvec',
|
||||
dimensions: 2560,
|
||||
embeddingModel: 'zeroentropyai:zembed-1',
|
||||
};
|
||||
const { col, castSql } = buildVectorCastFragment(r);
|
||||
expect(col).toBe('"embedding_ze"');
|
||||
expect(castSql).toBe('$1::halfvec(2560)');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,227 @@
|
||||
/**
|
||||
* v0.36 E2E (real Postgres) — halfvec + HNSW + doctor SQL probes.
|
||||
*
|
||||
* Skipped gracefully when DATABASE_URL is unset. When present, exercises
|
||||
* the real pgvector + Postgres code paths that PGLite can't fully cover:
|
||||
*
|
||||
* - halfvec(2560) cast accepted by real pgvector via searchVector.
|
||||
* - HNSW index on the alternative column is visible in EXPLAIN.
|
||||
* - The format_type SQL the doctor check uses (D13) correctly distinguishes
|
||||
* `vector(1024)` vs `vector(1536)` so dim drift can be detected.
|
||||
* - The coverage SQL the doctor check uses (D14) computes accurate
|
||||
* percentage and the < 90% gate fires when expected.
|
||||
*
|
||||
* Tests the SQL the doctor check issues, not `runDoctor` itself (which
|
||||
* calls process.exit). This is the regression-catching layer that
|
||||
* matters; the doctor wrapping just renders the result.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
import { quoteIdentifier } from '../../src/core/search/embedding-column.ts';
|
||||
import type { ResolvedColumn } from '../../src/core/types.ts';
|
||||
|
||||
const dbUrl = process.env.DATABASE_URL;
|
||||
if (!dbUrl) {
|
||||
describe.skip('postgres E2E — embedding column (skipped: DATABASE_URL unset)', () => {
|
||||
test('skipped', () => { expect(true).toBe(true); });
|
||||
});
|
||||
} else {
|
||||
let engine: PostgresEngine;
|
||||
let catId: number;
|
||||
let dogId: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PostgresEngine();
|
||||
await engine.connect({ database_url: dbUrl } as never);
|
||||
await engine.initSchema();
|
||||
|
||||
// Wipe state from prior runs.
|
||||
await engine.executeRaw(`DELETE FROM content_chunks`);
|
||||
await engine.executeRaw(`DELETE FROM pages WHERE slug LIKE 'docs/%'`);
|
||||
|
||||
// Add the ad-hoc Voyage + ZE columns + HNSW indexes.
|
||||
await engine.executeRaw(
|
||||
`ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_voyage vector(1024)`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_ze halfvec(2560)`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`CREATE INDEX IF NOT EXISTS idx_chunks_embedding_voyage
|
||||
ON content_chunks USING hnsw (embedding_voyage vector_cosine_ops)`,
|
||||
);
|
||||
|
||||
// Seed two pages and two chunks.
|
||||
await engine.putPage('docs/cat', {
|
||||
type: 'concept',
|
||||
title: 'Cat doc',
|
||||
compiled_truth: 'Cat doc compiled truth.',
|
||||
});
|
||||
await engine.putPage('docs/dog', {
|
||||
type: 'concept',
|
||||
title: 'Dog doc',
|
||||
compiled_truth: 'Dog doc compiled truth.',
|
||||
});
|
||||
await engine.upsertChunks('docs/cat', [
|
||||
{ chunk_index: 0, chunk_text: 'cat chunk', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.upsertChunks('docs/dog', [
|
||||
{ chunk_index: 0, chunk_text: 'dog chunk', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
|
||||
const idRows = await engine.executeRaw<{ id: number; slug: string }>(
|
||||
`SELECT cc.id, p.slug FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.slug LIKE 'docs/%' ORDER BY p.slug`,
|
||||
);
|
||||
catId = idRows.find(r => r.slug === 'docs/cat')!.id;
|
||||
dogId = idRows.find(r => r.slug === 'docs/dog')!.id;
|
||||
|
||||
const vec1024 = (v: number) => `[${new Array(1024).fill(v).join(',')}]`;
|
||||
const vec2560 = (v: number) => `[${new Array(2560).fill(v).join(',')}]`;
|
||||
await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = '${vec1024(0.5)}'::vector WHERE id = ${catId}`);
|
||||
await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = '${vec1024(0.7)}'::vector WHERE id = ${dogId}`);
|
||||
await engine.executeRaw(`UPDATE content_chunks SET embedding_ze = '${vec2560(0.5)}'::halfvec WHERE id = ${catId}`);
|
||||
await engine.executeRaw(`UPDATE content_chunks SET embedding_ze = '${vec2560(0.7)}'::halfvec WHERE id = ${dogId}`);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('Postgres: searchVector with halfvec descriptor', () => {
|
||||
test('halfvec(2560) cast accepted; results returned in expected cosine order', async () => {
|
||||
const queryVec = new Float32Array(2560).fill(0.5);
|
||||
const descriptor: ResolvedColumn = {
|
||||
name: 'embedding_ze',
|
||||
type: 'halfvec',
|
||||
dimensions: 2560,
|
||||
embeddingModel: 'zeroentropyai:zembed-1',
|
||||
};
|
||||
const results = await engine.searchVector(queryVec, {
|
||||
embeddingColumn: descriptor,
|
||||
limit: 5,
|
||||
});
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
// Cat's halfvec is identical to the query — closer than dog.
|
||||
expect(results[0].slug).toBe('docs/cat');
|
||||
});
|
||||
|
||||
test('vector(1024) cast on embedding_voyage routes correctly', async () => {
|
||||
const queryVec = new Float32Array(1024).fill(0.5);
|
||||
const descriptor: ResolvedColumn = {
|
||||
name: 'embedding_voyage',
|
||||
type: 'vector',
|
||||
dimensions: 1024,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
};
|
||||
const results = await engine.searchVector(queryVec, {
|
||||
embeddingColumn: descriptor,
|
||||
limit: 5,
|
||||
});
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
expect(results[0].slug).toBe('docs/cat');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Postgres: HNSW index visible to planner', () => {
|
||||
test('pg_indexes shows the hnsw index on embedding_voyage', async () => {
|
||||
const rows = await engine.executeRaw<{ indexname: string; indexdef: string }>(
|
||||
`SELECT indexname, indexdef FROM pg_indexes
|
||||
WHERE tablename = 'content_chunks'
|
||||
AND schemaname = 'public'`,
|
||||
);
|
||||
const voyageHnsw = rows.find(r =>
|
||||
r.indexname === 'idx_chunks_embedding_voyage' && /USING\s+hnsw/i.test(r.indexdef),
|
||||
);
|
||||
expect(voyageHnsw).toBeDefined();
|
||||
expect(voyageHnsw!.indexdef).toMatch(/embedding_voyage/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Postgres: doctor SQL probes for dim drift (D13)', () => {
|
||||
test('format_type returns parenthesized dim — distinguishes vector(1024) from vector(1536)', async () => {
|
||||
// This is the exact SQL shape doctor's embedding_column_registry
|
||||
// check uses. If pg_attribute or format_type semantics ever change,
|
||||
// this test fails loud.
|
||||
const rows = await engine.executeRaw<{ attname: string; formatted: string }>(
|
||||
`SELECT a.attname, format_type(a.atttypid, a.atttypmod) AS formatted
|
||||
FROM pg_attribute a
|
||||
JOIN pg_class c ON c.oid = a.attrelid
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = 'public'
|
||||
AND c.relname = 'content_chunks'
|
||||
AND a.attname = ANY($1::text[])
|
||||
AND NOT a.attisdropped`,
|
||||
[['embedding', 'embedding_voyage', 'embedding_ze']],
|
||||
);
|
||||
const byName = new Map<string, string>();
|
||||
for (const r of rows) byName.set(r.attname, r.formatted);
|
||||
|
||||
// Default 'embedding' is vector(1536) by committed schema.
|
||||
expect(byName.get('embedding')).toMatch(/^vector\(\d+\)/);
|
||||
// Voyage is vector(1024) per the ALTER above.
|
||||
expect(byName.get('embedding_voyage')).toBe('vector(1024)');
|
||||
// ZE is halfvec(2560) per the ALTER above.
|
||||
expect(byName.get('embedding_ze')).toBe('halfvec(2560)');
|
||||
});
|
||||
|
||||
test('format_type catches dim drift: declared 1536 vs actual 1024', async () => {
|
||||
// Simulate the user declaring 1536 in their registry but the column
|
||||
// is actually 1024d. Doctor's regex-parse of the formatted string
|
||||
// is the layer that catches this.
|
||||
const rows = await engine.executeRaw<{ formatted: string }>(
|
||||
`SELECT format_type(atttypid, atttypmod) AS formatted
|
||||
FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass
|
||||
AND attname = 'embedding_voyage'
|
||||
AND NOT attisdropped`,
|
||||
);
|
||||
const formatted = rows[0]?.formatted ?? '';
|
||||
const m = formatted.match(/^(vector|halfvec)\((\d+)\)$/);
|
||||
expect(m).toBeTruthy();
|
||||
const declaredDims = 1536;
|
||||
const actualDims = parseInt(m![2], 10);
|
||||
expect(actualDims).not.toBe(declaredDims); // The drift is detectable.
|
||||
expect(actualDims).toBe(1024);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Postgres: doctor SQL probe for coverage (D14)', () => {
|
||||
test('coverage % computed correctly on fully-populated column', async () => {
|
||||
const col = quoteIdentifier('embedding_voyage');
|
||||
const rows = await engine.executeRaw<{ pct: number; total: number }>(
|
||||
`SELECT (
|
||||
COUNT(*) FILTER (WHERE ${col} IS NOT NULL)::float
|
||||
/ NULLIF(COUNT(*), 0) * 100
|
||||
)::float AS pct,
|
||||
COUNT(*)::int AS total
|
||||
FROM content_chunks`,
|
||||
);
|
||||
expect(rows[0].total).toBe(2);
|
||||
expect(rows[0].pct).toBe(100);
|
||||
});
|
||||
|
||||
test('coverage % drops to 50 after a partial wipe; gate at < 90 fires', async () => {
|
||||
// Clear voyage on one chunk to simulate partial backfill.
|
||||
await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = NULL WHERE id = ${dogId}`);
|
||||
|
||||
const col = quoteIdentifier('embedding_voyage');
|
||||
const rows = await engine.executeRaw<{ pct: number; total: number }>(
|
||||
`SELECT (
|
||||
COUNT(*) FILTER (WHERE ${col} IS NOT NULL)::float
|
||||
/ NULLIF(COUNT(*), 0) * 100
|
||||
)::float AS pct,
|
||||
COUNT(*)::int AS total
|
||||
FROM content_chunks`,
|
||||
);
|
||||
expect(rows[0].total).toBe(2);
|
||||
expect(rows[0].pct).toBe(50);
|
||||
// The < 90 gate that doctor + config-set use should fire.
|
||||
expect(rows[0].pct < 90).toBe(true);
|
||||
|
||||
// Restore so subsequent tests see the original fixture.
|
||||
const v = `[${new Array(1024).fill(0.7).join(',')}]`;
|
||||
await engine.executeRaw(`UPDATE content_chunks SET embedding_voyage = '${v}'::vector WHERE id = ${dogId}`);
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* v0.36 (D16 / CDX-10) — eval_candidates.embedding_column round-trip.
|
||||
*
|
||||
* Pins:
|
||||
* - logEvalCandidate persists `embedding_column` when set.
|
||||
* - listEvalCandidates reads it back (SELECT * carries the new column).
|
||||
* - Migration v67 applied: ALTER TABLE eval_candidates ADD COLUMN
|
||||
* IF NOT EXISTS embedding_column TEXT.
|
||||
* - Back-compat: rows inserted without embedding_column store NULL,
|
||||
* and replay treats NULL as "use current default."
|
||||
*
|
||||
* The integration of replay with hybridSearch's column-override path is
|
||||
* tested via the embeddingColumn option being honored — we don't run
|
||||
* the full eval-replay CLI subcommand here because that brings in CLI
|
||||
* arg parsing + filesystem reading. Unit-level surface is sufficient
|
||||
* for the persist-and-read contract.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { EvalCandidateInput } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
});
|
||||
|
||||
const baseRow: Omit<EvalCandidateInput, 'embedding_column'> = {
|
||||
tool_name: 'query',
|
||||
query: 'test query',
|
||||
retrieved_slugs: ['docs/a', 'docs/b'],
|
||||
retrieved_chunk_ids: [1, 2],
|
||||
source_ids: ['default'],
|
||||
expand_enabled: true,
|
||||
detail: null,
|
||||
detail_resolved: 'medium',
|
||||
vector_enabled: true,
|
||||
expansion_applied: false,
|
||||
latency_ms: 42,
|
||||
remote: false,
|
||||
job_id: null,
|
||||
subagent_id: null,
|
||||
};
|
||||
|
||||
describe('eval_candidates.embedding_column persistence (D16)', () => {
|
||||
test('migration v67 added embedding_column TEXT column', async () => {
|
||||
const rows = await engine.executeRaw<{ column_name: string; data_type: string; is_nullable: string }>(
|
||||
`SELECT column_name, data_type, is_nullable
|
||||
FROM information_schema.columns
|
||||
WHERE table_name = 'eval_candidates' AND column_name = 'embedding_column'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].column_name).toBe('embedding_column');
|
||||
expect(rows[0].data_type).toBe('text');
|
||||
expect(rows[0].is_nullable).toBe('YES');
|
||||
});
|
||||
|
||||
test('logEvalCandidate stores embedding_column value', async () => {
|
||||
const id = await engine.logEvalCandidate({
|
||||
...baseRow,
|
||||
embedding_column: 'embedding_voyage',
|
||||
});
|
||||
const rows = await engine.executeRaw<{ embedding_column: string | null }>(
|
||||
`SELECT embedding_column FROM eval_candidates WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
expect(rows[0].embedding_column).toBe('embedding_voyage');
|
||||
});
|
||||
|
||||
test('listEvalCandidates reads embedding_column back (round-trip via SELECT *)', async () => {
|
||||
const id = await engine.logEvalCandidate({
|
||||
...baseRow,
|
||||
query: 'list-readback test',
|
||||
embedding_column: 'embedding_ze',
|
||||
});
|
||||
const list = await engine.listEvalCandidates({ limit: 100 });
|
||||
const found = list.find(r => r.id === id);
|
||||
expect(found).toBeDefined();
|
||||
// Cast-through-unknown rowToEvalCandidate doesn't exist; SELECT *
|
||||
// carries `embedding_column` as a column with the same key name.
|
||||
expect((found as any).embedding_column).toBe('embedding_ze');
|
||||
});
|
||||
|
||||
test('back-compat: missing embedding_column coalesces to NULL at insert', async () => {
|
||||
// Build a row WITHOUT embedding_column to mimic pre-v0.36 callers.
|
||||
const input: EvalCandidateInput = { ...baseRow, query: 'pre-v036 fixture' };
|
||||
expect(input.embedding_column).toBeUndefined();
|
||||
const id = await engine.logEvalCandidate(input);
|
||||
const rows = await engine.executeRaw<{ embedding_column: string | null }>(
|
||||
`SELECT embedding_column FROM eval_candidates WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
expect(rows[0].embedding_column).toBeNull();
|
||||
});
|
||||
|
||||
test('back-compat: explicit null persists as DB NULL (not the literal string "null")', async () => {
|
||||
const id = await engine.logEvalCandidate({
|
||||
...baseRow,
|
||||
query: 'explicit-null fixture',
|
||||
embedding_column: null,
|
||||
});
|
||||
const rows = await engine.executeRaw<{ embedding_column: string | null }>(
|
||||
`SELECT embedding_column FROM eval_candidates WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
expect(rows[0].embedding_column).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* v0.36 (D10) — gateway embed model override path.
|
||||
*
|
||||
* Pins:
|
||||
* - embedQuery(text, { embeddingModel }) routes through THAT provider,
|
||||
* not the global default.
|
||||
* - embedQuery(text, { dimensions }) flows into dimsProviderOptions
|
||||
* so providers that accept output_dimension see the override.
|
||||
* - Bare embedQuery(text) continues to use the configured default.
|
||||
* - Unknown override model throws (resolveEmbeddingProvider's
|
||||
* AIConfigError shape with a hint).
|
||||
* - isAvailable('embedding', modelOverride) probes the override's
|
||||
* recipe, not the global default's.
|
||||
*/
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
configureGateway,
|
||||
embedQuery,
|
||||
isAvailable,
|
||||
resetGateway,
|
||||
__setEmbedTransportForTests,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
|
||||
interface TransportCall {
|
||||
modelString: string;
|
||||
values: string[];
|
||||
providerOptions: Record<string, unknown>;
|
||||
}
|
||||
|
||||
const calls: TransportCall[] = [];
|
||||
|
||||
function installCaptureTransport(makeVector: (dims: number) => number[]) {
|
||||
__setEmbedTransportForTests(async ({ model, values, providerOptions }: any) => {
|
||||
// The AI SDK's `model` object exposes `.modelId` (string) on every
|
||||
// openai-compatible model. We capture it so tests can assert the
|
||||
// gateway routed to the correct provider:model.
|
||||
const modelString = (model?.modelId ?? '<unknown>') as string;
|
||||
calls.push({ modelString, values: [...values], providerOptions: { ...(providerOptions ?? {}) } });
|
||||
// Pick the dim from providerOpts when present (Voyage flexible-dim
|
||||
// path emits openaiCompatible.dimensions); otherwise default 1536.
|
||||
const oc = (providerOptions?.openaiCompatible ?? {}) as Record<string, unknown>;
|
||||
const dims = typeof oc.dimensions === 'number' ? (oc.dimensions as number) : 1536;
|
||||
return {
|
||||
embeddings: values.map(() => makeVector(dims)),
|
||||
usage: { tokens: 0 },
|
||||
} as any;
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
calls.length = 0;
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('embedQuery — bare (no opts)', () => {
|
||||
test('bare call uses the globally configured embedding_model', async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test' },
|
||||
});
|
||||
installCaptureTransport(d => new Array(d).fill(0).map((_, i) => i * 0.001));
|
||||
|
||||
const v = await embedQuery('hello');
|
||||
expect(v.length).toBe(1536);
|
||||
expect(calls.length).toBe(1);
|
||||
expect(calls[0].modelString).toBe('text-embedding-3-large');
|
||||
});
|
||||
});
|
||||
|
||||
describe('embedQuery — { embeddingModel } override', () => {
|
||||
test('routes through the override provider, not the global default', async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test', VOYAGE_API_KEY: 'voy-test' },
|
||||
});
|
||||
installCaptureTransport(d => new Array(d).fill(0).map((_, i) => i * 0.002));
|
||||
|
||||
const v = await embedQuery('hello', {
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
dimensions: 1024,
|
||||
});
|
||||
expect(v.length).toBe(1024);
|
||||
expect(calls.length).toBe(1);
|
||||
expect(calls[0].modelString).toBe('voyage-3-large');
|
||||
});
|
||||
|
||||
test('dimensions override flows into providerOptions', async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test', VOYAGE_API_KEY: 'voy-test' },
|
||||
});
|
||||
installCaptureTransport(d => new Array(d).fill(0).map(() => 0.1));
|
||||
|
||||
await embedQuery('hello', { embeddingModel: 'voyage:voyage-3-large', dimensions: 2048 });
|
||||
const opts = calls[0].providerOptions as { openaiCompatible?: Record<string, unknown> };
|
||||
// Voyage flexible-dim models emit `dimensions` into the openaiCompatible
|
||||
// providerOptions block; the shim translates to output_dimension on
|
||||
// the wire. Either way, the gateway honored the caller's dim override.
|
||||
expect(opts.openaiCompatible).toBeDefined();
|
||||
expect(opts.openaiCompatible!.dimensions).toBe(2048);
|
||||
});
|
||||
|
||||
test('unknown override model throws AIConfigError with a useful hint', async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test' },
|
||||
});
|
||||
installCaptureTransport(d => new Array(d).fill(0));
|
||||
|
||||
let threw: Error | null = null;
|
||||
try {
|
||||
await embedQuery('hello', { embeddingModel: 'nonexistent:bogus' });
|
||||
} catch (e) {
|
||||
threw = e as Error;
|
||||
}
|
||||
expect(threw).toBeTruthy();
|
||||
// Error message names the provider or model so the user knows what failed.
|
||||
expect(threw!.message.toLowerCase()).toMatch(/nonexistent|provider|recipe|model/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isAvailable(touchpoint, modelOverride) — D10', () => {
|
||||
test('global default available + Voyage override key present → both available', () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test', VOYAGE_API_KEY: 'voy-test' },
|
||||
});
|
||||
expect(isAvailable('embedding')).toBe(true);
|
||||
expect(isAvailable('embedding', 'voyage:voyage-3-large')).toBe(true);
|
||||
});
|
||||
|
||||
test('global default key missing but override key present → override is available', () => {
|
||||
// The single-OPENAI scenario: user removed OPENAI_API_KEY but
|
||||
// configured Voyage for the alt column. Pre-D10, isAvailable would
|
||||
// have said embedding is unavailable globally and hybridSearch
|
||||
// would skip vector search ENTIRELY. With the override, hybrid asks
|
||||
// about the active column's provider and gets a green light.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { VOYAGE_API_KEY: 'voy-test' }, // no OPENAI_API_KEY
|
||||
});
|
||||
expect(isAvailable('embedding')).toBe(false);
|
||||
expect(isAvailable('embedding', 'voyage:voyage-3-large')).toBe(true);
|
||||
});
|
||||
|
||||
test('global default available but override key missing → override is unavailable', () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test' }, // no VOYAGE_API_KEY
|
||||
});
|
||||
expect(isAvailable('embedding')).toBe(true);
|
||||
expect(isAvailable('embedding', 'voyage:voyage-3-large')).toBe(false);
|
||||
});
|
||||
|
||||
test('override against an unknown model returns false', () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test' },
|
||||
});
|
||||
expect(isAvailable('embedding', 'totally:unknown')).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -20,9 +20,31 @@ function makeEngine(map: Record<string, string | null | undefined>): FakeEngine
|
||||
}
|
||||
|
||||
describe('loadConfigWithEngine (Phase 4 / F3)', () => {
|
||||
test('returns null when base config is null', async () => {
|
||||
test('synthesizes a minimal base when base config is null (v0.36 codex /ship #3)', async () => {
|
||||
// Pre-v0.36 this returned null and skipped DB-plane merge entirely.
|
||||
// That meant env-only Postgres installs (no file config) couldn't see
|
||||
// DB-plane overrides set via `gbrain config set` — the documented
|
||||
// smoke test for `search_embedding_column` would silently fail.
|
||||
// The fix synthesizes a minimal `{ engine: 'postgres' }` base so DB
|
||||
// merge still runs; downstream callers either find the DB key or
|
||||
// fall through to defaults.
|
||||
const result = await loadConfigWithEngine(makeEngine({}), null);
|
||||
expect(result).toBeNull();
|
||||
expect(result).not.toBeNull();
|
||||
expect(result?.engine).toBe('postgres');
|
||||
});
|
||||
|
||||
test('DB-plane embedding_columns merge works even with null base (codex /ship #3 round-trip)', async () => {
|
||||
// The whole point of the synthesized fallback: env-only installs
|
||||
// calling `gbrain config set embedding_columns '...'` get those keys
|
||||
// back when the resolver re-reads config. Verifies the merge path
|
||||
// actually runs (not just that the function returns truthy).
|
||||
const engine = makeEngine({
|
||||
search_embedding_column: 'embedding_voyage',
|
||||
embedding_columns: '{"embedding_voyage":{"provider":"voyage:voyage-3-large","dimensions":1024,"type":"vector"}}',
|
||||
});
|
||||
const merged = await loadConfigWithEngine(engine, null);
|
||||
expect(merged?.search_embedding_column).toBe('embedding_voyage');
|
||||
expect(merged?.embedding_columns?.embedding_voyage?.dimensions).toBe(1024);
|
||||
});
|
||||
|
||||
test('DB flag fills in when file/env did not set it', async () => {
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* v0.36 (D15 / CDX-9) — MCP op `embedding_column` param surface.
|
||||
*
|
||||
* Pins:
|
||||
* - `query` op declares `embedding_column` in its params allowlist
|
||||
* (caller can pass it).
|
||||
* - `search` op does NOT declare it (CDX-9: search is keyword-only;
|
||||
* adding the field would silently change semantics).
|
||||
* - Param description names the registry + the override semantics so
|
||||
* agents discover it via tool definitions.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { operationsByName } from '../src/core/operations.ts';
|
||||
|
||||
describe('query op — embedding_column param (D15)', () => {
|
||||
const queryOp = operationsByName.query;
|
||||
|
||||
test('exists', () => {
|
||||
expect(queryOp).toBeDefined();
|
||||
});
|
||||
|
||||
test('declares embedding_column in params allowlist', () => {
|
||||
expect(queryOp.params.embedding_column).toBeDefined();
|
||||
expect(queryOp.params.embedding_column.type).toBe('string');
|
||||
});
|
||||
|
||||
test('description names registry + override semantics', () => {
|
||||
const desc = queryOp.params.embedding_column.description ?? '';
|
||||
expect(desc.toLowerCase()).toMatch(/embedding/);
|
||||
// Description should give the agent enough context to understand
|
||||
// when to use the param and where the registry lives.
|
||||
expect(desc.toLowerCase()).toMatch(/registry|embedding_columns|column/);
|
||||
});
|
||||
|
||||
test('embedding_column is NOT required (per-call override is optional)', () => {
|
||||
expect(queryOp.params.embedding_column.required).not.toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('search op — does NOT declare embedding_column (CDX-9)', () => {
|
||||
const searchOp = operationsByName.search;
|
||||
|
||||
test('exists', () => {
|
||||
expect(searchOp).toBeDefined();
|
||||
});
|
||||
|
||||
test('does NOT include embedding_column in params (search is keyword-only)', () => {
|
||||
// Adding embedding_column to the keyword-only `search` op would
|
||||
// either be silently ignored (footgun for agents) or change the op's
|
||||
// semantics from keyword to hybrid. Both bad. Keep it on `query` only.
|
||||
expect(searchOp.params.embedding_column).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -272,8 +272,11 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
|
||||
test('KNOBS_HASH_VERSION constant exposed for migrations to bump on schema change', () => {
|
||||
// v0.35.0.0+ bumped 1→2 to fold reranker fields into the cache key.
|
||||
// v0.35.6.0 bumped 2→3 to fold floor_ratio into the cache key
|
||||
// (codex outside-voice T1 — preventing cross-floor cache contamination).
|
||||
// v0.35.6.0 bumped 2→3 to fold floor_ratio into the cache key
|
||||
// (codex T1 — preventing cross-floor cache contamination).
|
||||
// v0.36 also extends v=3 with embedding column + provider (D8 / CDX-2)
|
||||
// so a query against `embedding_voyage` never shares a cache row with
|
||||
// `embedding`, even when all other knobs match.
|
||||
expect(KNOBS_HASH_VERSION).toBe(3);
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,511 @@
|
||||
/**
|
||||
* v0.36 — embedding column resolver tests.
|
||||
*
|
||||
* Pins:
|
||||
* - D2/D11: resolver returns descriptor (name, type, dimensions,
|
||||
* embeddingModel).
|
||||
* - D3: buildVectorCastFragment produces correct cast string per type.
|
||||
* - D11: builtins (`embedding`, `embedding_image`) always present.
|
||||
* - D12: registry-key regex + field validation reject malicious input.
|
||||
* - D12: identifier-quoting handles embedded quotes safely.
|
||||
* - Resolution chain: opts > cfg.search_embedding_column > 'embedding'.
|
||||
* - normalizeEngineColumn: descriptor-passthrough + legacy literals +
|
||||
* throw on unknown string.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
resolveEmbeddingColumn,
|
||||
getEmbeddingColumnRegistry,
|
||||
buildVectorCastFragment,
|
||||
quoteIdentifier,
|
||||
validateColumnKey,
|
||||
validateColumnConfig,
|
||||
normalizeEngineColumn,
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
EmbeddingColumnConfigError,
|
||||
COLUMN_NAME_REGEX,
|
||||
ALLOWED_COLUMN_TYPES,
|
||||
MAX_DIMENSIONS,
|
||||
DEFAULT_COLUMN_NAME,
|
||||
isDefaultColumn,
|
||||
isCacheSafe,
|
||||
isBuiltinColumn,
|
||||
} from '../../src/core/search/embedding-column.ts';
|
||||
import type { GBrainConfig } from '../../src/core/config.ts';
|
||||
import type { ResolvedColumn } from '../../src/core/types.ts';
|
||||
|
||||
function cfg(overrides: Partial<GBrainConfig> = {}): GBrainConfig {
|
||||
return { engine: 'pglite', ...overrides };
|
||||
}
|
||||
|
||||
describe('resolveEmbeddingColumn — resolution chain', () => {
|
||||
test('default fallback returns "embedding"', () => {
|
||||
const r = resolveEmbeddingColumn(undefined, cfg());
|
||||
expect(r.name).toBe('embedding');
|
||||
expect(r.type).toBe('vector');
|
||||
});
|
||||
|
||||
test('cfg.search_embedding_column wins over default', () => {
|
||||
const r = resolveEmbeddingColumn(undefined, cfg({
|
||||
search_embedding_column: 'embedding_voyage',
|
||||
embedding_columns: {
|
||||
embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
|
||||
},
|
||||
}));
|
||||
expect(r.name).toBe('embedding_voyage');
|
||||
expect(r.embeddingModel).toBe('voyage:voyage-3-large');
|
||||
expect(r.dimensions).toBe(1024);
|
||||
});
|
||||
|
||||
test('opts.embeddingColumn wins over cfg.search_embedding_column', () => {
|
||||
const r = resolveEmbeddingColumn(
|
||||
{ embeddingColumn: 'embedding_voyage' },
|
||||
cfg({
|
||||
search_embedding_column: 'embedding_zeroentropy',
|
||||
embedding_columns: {
|
||||
embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
|
||||
embedding_zeroentropy: { provider: 'zeroentropyai:zembed-1', dimensions: 2560, type: 'halfvec' },
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(r.name).toBe('embedding_voyage');
|
||||
});
|
||||
|
||||
test('unknown name throws EmbeddingColumnNotRegisteredError with hint', () => {
|
||||
let err: EmbeddingColumnNotRegisteredError | null = null;
|
||||
try {
|
||||
resolveEmbeddingColumn({ embeddingColumn: 'nonexistent' }, cfg());
|
||||
} catch (e) {
|
||||
err = e as EmbeddingColumnNotRegisteredError;
|
||||
}
|
||||
expect(err).toBeTruthy();
|
||||
expect(err?.code).toBe('embedding_column_not_registered');
|
||||
expect(err?.columnName).toBe('nonexistent');
|
||||
expect(err?.validColumns).toEqual(['embedding', 'embedding_image']);
|
||||
expect(err?.message).toContain('Declared columns:');
|
||||
expect(err?.message).toContain('gbrain config set');
|
||||
});
|
||||
|
||||
test('SQL-injection-shaped name rejected before registry lookup', () => {
|
||||
expect(() =>
|
||||
resolveEmbeddingColumn(
|
||||
{ embeddingColumn: 'embedding"; DROP TABLE pages; --' },
|
||||
cfg(),
|
||||
),
|
||||
).toThrow(EmbeddingColumnNotRegisteredError);
|
||||
});
|
||||
|
||||
test('descriptor passthrough: ResolvedColumn returned as-is', () => {
|
||||
const descriptor: ResolvedColumn = {
|
||||
name: 'embedding_custom',
|
||||
type: 'halfvec',
|
||||
dimensions: 2560,
|
||||
embeddingModel: 'zeroentropyai:zembed-1',
|
||||
};
|
||||
const r = resolveEmbeddingColumn({ embeddingColumn: descriptor }, cfg());
|
||||
expect(r).toEqual(descriptor);
|
||||
});
|
||||
});
|
||||
|
||||
describe('getEmbeddingColumnRegistry — builtins + merge', () => {
|
||||
test('builtin embedding always present even with empty user config', () => {
|
||||
const reg = getEmbeddingColumnRegistry(cfg());
|
||||
expect(reg.embedding).toBeDefined();
|
||||
expect(reg.embedding!.type).toBe('vector');
|
||||
expect(reg.embedding!.dimensions).toBe(1536);
|
||||
});
|
||||
|
||||
test('builtin embedding_image always present with 1024d vector', () => {
|
||||
const reg = getEmbeddingColumnRegistry(cfg());
|
||||
expect(reg.embedding_image).toBeDefined();
|
||||
expect(reg.embedding_image!.type).toBe('vector');
|
||||
expect(reg.embedding_image!.dimensions).toBe(1024);
|
||||
});
|
||||
|
||||
test('builtin embedding derives provider from cfg.embedding_model', () => {
|
||||
const reg = getEmbeddingColumnRegistry(
|
||||
cfg({ embedding_model: 'voyage:voyage-3-large', embedding_dimensions: 1024 }),
|
||||
);
|
||||
expect(reg.embedding!.provider).toBe('voyage:voyage-3-large');
|
||||
expect(reg.embedding!.dimensions).toBe(1024);
|
||||
});
|
||||
|
||||
test('builtin embedding_image derives provider from cfg.embedding_multimodal_model', () => {
|
||||
const reg = getEmbeddingColumnRegistry(
|
||||
cfg({ embedding_multimodal_model: 'voyage:voyage-multimodal-3' }),
|
||||
);
|
||||
expect(reg.embedding_image!.provider).toBe('voyage:voyage-multimodal-3');
|
||||
});
|
||||
|
||||
test('user-declared columns merge with builtins', () => {
|
||||
const reg = getEmbeddingColumnRegistry(
|
||||
cfg({
|
||||
embedding_columns: {
|
||||
embedding_voyage: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(Object.keys(reg).sort()).toEqual(['embedding', 'embedding_image', 'embedding_voyage']);
|
||||
});
|
||||
|
||||
test('user override wins on conflict (override embedding builtin)', () => {
|
||||
const reg = getEmbeddingColumnRegistry(
|
||||
cfg({
|
||||
embedding_columns: {
|
||||
embedding: { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(reg.embedding!.provider).toBe('voyage:voyage-3-large');
|
||||
expect(reg.embedding!.dimensions).toBe(1024);
|
||||
});
|
||||
|
||||
test('halfvec column with high dim accepted', () => {
|
||||
const reg = getEmbeddingColumnRegistry(
|
||||
cfg({
|
||||
embedding_columns: {
|
||||
embedding_ze: { provider: 'zeroentropyai:zembed-1', dimensions: 2560, type: 'halfvec' },
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(reg.embedding_ze!.type).toBe('halfvec');
|
||||
expect(reg.embedding_ze!.dimensions).toBe(2560);
|
||||
});
|
||||
});
|
||||
|
||||
describe('D12 — defense-in-depth validation', () => {
|
||||
describe('validateColumnKey', () => {
|
||||
test('accepts lowercase identifier', () => {
|
||||
expect(() => validateColumnKey('embedding_voyage')).not.toThrow();
|
||||
expect(() => validateColumnKey('a')).not.toThrow();
|
||||
expect(() => validateColumnKey('_underscore_first')).not.toThrow();
|
||||
expect(() => validateColumnKey('mix_of_letters_and_123')).not.toThrow();
|
||||
});
|
||||
|
||||
test('rejects keys with quotes (SQL injection vector)', () => {
|
||||
expect(() => validateColumnKey('embedding"; DROP --')).toThrow(EmbeddingColumnConfigError);
|
||||
expect(() => validateColumnKey("embedding'")).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('rejects keys with uppercase', () => {
|
||||
expect(() => validateColumnKey('Embedding')).toThrow(EmbeddingColumnConfigError);
|
||||
expect(() => validateColumnKey('EMBEDDING_VOYAGE')).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('rejects keys starting with digits', () => {
|
||||
expect(() => validateColumnKey('1embedding')).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('rejects keys with hyphens, spaces, special chars', () => {
|
||||
expect(() => validateColumnKey('embed-voyage')).toThrow(EmbeddingColumnConfigError);
|
||||
expect(() => validateColumnKey('embed voyage')).toThrow(EmbeddingColumnConfigError);
|
||||
expect(() => validateColumnKey('embed.voyage')).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('rejects empty key', () => {
|
||||
expect(() => validateColumnKey('')).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateColumnConfig', () => {
|
||||
test('accepts valid config', () => {
|
||||
expect(() =>
|
||||
validateColumnConfig('embedding_voyage', {
|
||||
provider: 'voyage:voyage-3-large',
|
||||
dimensions: 1024,
|
||||
type: 'vector',
|
||||
}),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('rejects bad type', () => {
|
||||
expect(() =>
|
||||
validateColumnConfig('embedding_voyage', {
|
||||
provider: 'voyage:voyage-3-large',
|
||||
dimensions: 1024,
|
||||
type: 'jsonb' as 'vector',
|
||||
}),
|
||||
).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('rejects bad dimensions (zero/negative/too-large)', () => {
|
||||
const base = { provider: 'voyage:voyage-3-large', type: 'vector' as const };
|
||||
expect(() => validateColumnConfig('x', { ...base, dimensions: 0 })).toThrow();
|
||||
expect(() => validateColumnConfig('x', { ...base, dimensions: -5 })).toThrow();
|
||||
expect(() => validateColumnConfig('x', { ...base, dimensions: MAX_DIMENSIONS + 1 })).toThrow();
|
||||
expect(() => validateColumnConfig('x', { ...base, dimensions: 1.5 as number })).toThrow();
|
||||
});
|
||||
|
||||
test('rejects bad provider (empty, missing colon, missing model)', () => {
|
||||
const base = { dimensions: 1024, type: 'vector' as const };
|
||||
expect(() => validateColumnConfig('x', { ...base, provider: '' })).toThrow();
|
||||
expect(() => validateColumnConfig('x', { ...base, provider: 'voyage' })).toThrow();
|
||||
expect(() => validateColumnConfig('x', { ...base, provider: 'voyage:' })).toThrow();
|
||||
expect(() => validateColumnConfig('x', { ...base, provider: ':voyage-3-large' })).toThrow();
|
||||
});
|
||||
|
||||
test('rejects non-object shapes (array, null, scalar)', () => {
|
||||
expect(() => validateColumnConfig('x', null)).toThrow();
|
||||
expect(() => validateColumnConfig('x', [])).toThrow();
|
||||
expect(() => validateColumnConfig('x', 'string')).toThrow();
|
||||
expect(() => validateColumnConfig('x', 42)).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
test('registry load throws when any entry is invalid', () => {
|
||||
expect(() =>
|
||||
getEmbeddingColumnRegistry(
|
||||
cfg({
|
||||
embedding_columns: {
|
||||
'embedding"; DROP --': { provider: 'voyage:voyage-3-large', dimensions: 1024, type: 'vector' },
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('D3 — buildVectorCastFragment + quoteIdentifier', () => {
|
||||
test('vector type emits $1::vector cast', () => {
|
||||
const r: ResolvedColumn = { name: 'embedding', type: 'vector', dimensions: 1536, embeddingModel: '' };
|
||||
const { col, castSql } = buildVectorCastFragment(r);
|
||||
expect(col).toBe('"embedding"');
|
||||
expect(castSql).toBe('$1::vector');
|
||||
});
|
||||
|
||||
test('halfvec type emits $1::halfvec(N) cast', () => {
|
||||
const r: ResolvedColumn = { name: 'embedding_ze', type: 'halfvec', dimensions: 2560, embeddingModel: 'zeroentropyai:zembed-1' };
|
||||
const { col, castSql } = buildVectorCastFragment(r);
|
||||
expect(col).toBe('"embedding_ze"');
|
||||
expect(castSql).toBe('$1::halfvec(2560)');
|
||||
});
|
||||
|
||||
test('quoteIdentifier wraps in double quotes', () => {
|
||||
expect(quoteIdentifier('embedding')).toBe('"embedding"');
|
||||
expect(quoteIdentifier('embedding_voyage')).toBe('"embedding_voyage"');
|
||||
});
|
||||
|
||||
test('quoteIdentifier doubles embedded quotes (defense belt)', () => {
|
||||
// Even though regex prevents this from reaching here in practice,
|
||||
// the quoting belt handles a quoted-string-break attempt.
|
||||
expect(quoteIdentifier('embed"ding')).toBe('"embed""ding"');
|
||||
});
|
||||
});
|
||||
|
||||
describe('normalizeEngineColumn — engine-side legacy converter', () => {
|
||||
test('undefined returns builtin embedding descriptor', () => {
|
||||
const r = normalizeEngineColumn(undefined);
|
||||
expect(r.name).toBe('embedding');
|
||||
expect(r.type).toBe('vector');
|
||||
});
|
||||
|
||||
test("'embedding' literal returns builtin descriptor", () => {
|
||||
const r = normalizeEngineColumn('embedding');
|
||||
expect(r.name).toBe('embedding');
|
||||
expect(r.type).toBe('vector');
|
||||
});
|
||||
|
||||
test("'embedding_image' literal returns 1024d vector descriptor", () => {
|
||||
const r = normalizeEngineColumn('embedding_image');
|
||||
expect(r.name).toBe('embedding_image');
|
||||
expect(r.type).toBe('vector');
|
||||
expect(r.dimensions).toBe(1024);
|
||||
});
|
||||
|
||||
test('ResolvedColumn descriptor passes through', () => {
|
||||
const descriptor: ResolvedColumn = {
|
||||
name: 'embedding_ze',
|
||||
type: 'halfvec',
|
||||
dimensions: 2560,
|
||||
embeddingModel: 'zeroentropyai:zembed-1',
|
||||
};
|
||||
expect(normalizeEngineColumn(descriptor)).toEqual(descriptor);
|
||||
});
|
||||
|
||||
test('unknown raw string throws (engine purity contract)', () => {
|
||||
// Strings other than legacy literals must NEVER reach the engine.
|
||||
// The resolver lives at hybrid/op boundary; the engine throws if
|
||||
// a caller bypassed it.
|
||||
expect(() => normalizeEngineColumn('embedding_voyage' as string)).toThrow(
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('helpers', () => {
|
||||
test('isDefaultColumn true only for "embedding"', () => {
|
||||
const def: ResolvedColumn = { name: 'embedding', type: 'vector', dimensions: 1536, embeddingModel: '' };
|
||||
const alt: ResolvedColumn = { name: 'embedding_voyage', type: 'vector', dimensions: 1024, embeddingModel: 'v' };
|
||||
expect(isDefaultColumn(def)).toBe(true);
|
||||
expect(isDefaultColumn(alt)).toBe(false);
|
||||
});
|
||||
|
||||
test('isBuiltinColumn matches both builtins exactly', () => {
|
||||
expect(isBuiltinColumn('embedding')).toBe(true);
|
||||
expect(isBuiltinColumn('embedding_image')).toBe(true);
|
||||
expect(isBuiltinColumn('embedding_voyage')).toBe(false);
|
||||
});
|
||||
|
||||
test('exported constants are stable', () => {
|
||||
expect(DEFAULT_COLUMN_NAME).toBe('embedding');
|
||||
expect(ALLOWED_COLUMN_TYPES.has('vector')).toBe(true);
|
||||
expect(ALLOWED_COLUMN_TYPES.has('halfvec')).toBe(true);
|
||||
expect(MAX_DIMENSIONS).toBe(8192);
|
||||
expect(COLUMN_NAME_REGEX.test('embedding_voyage')).toBe(true);
|
||||
expect(COLUMN_NAME_REGEX.test('Embedding')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('codex /ship #1 — prototype-pollution-safe registry', () => {
|
||||
test('resolver rejects "constructor" even though regex accepts it', () => {
|
||||
// The regex `^[a-z_][a-z0-9_]*$` matches "constructor" — but the
|
||||
// registry uses Object.create(null) + Object.hasOwn so Object's
|
||||
// inherited members don't masquerade as registered columns.
|
||||
expect(() =>
|
||||
resolveEmbeddingColumn({ embeddingColumn: 'constructor' }, cfg()),
|
||||
).toThrow(EmbeddingColumnNotRegisteredError);
|
||||
});
|
||||
|
||||
test('resolver rejects other inherited names (toString, hasOwnProperty)', () => {
|
||||
for (const name of ['tostring', 'hasownproperty', 'isprototypeof', 'valueof']) {
|
||||
expect(() =>
|
||||
resolveEmbeddingColumn({ embeddingColumn: name }, cfg()),
|
||||
).toThrow(EmbeddingColumnNotRegisteredError);
|
||||
}
|
||||
});
|
||||
|
||||
test('getEmbeddingColumnRegistry returns a null-prototype object', () => {
|
||||
const reg = getEmbeddingColumnRegistry(cfg());
|
||||
// No Object.prototype inheritance — direct prototype access returns null.
|
||||
expect(Object.getPrototypeOf(reg)).toBeNull();
|
||||
// Inherited properties are genuinely absent.
|
||||
expect((reg as any).constructor).toBeUndefined();
|
||||
expect((reg as any).toString).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('codex /ship #2 — descriptor passthrough validates', () => {
|
||||
test('passthrough re-validates name regex', () => {
|
||||
const bad: ResolvedColumn = {
|
||||
name: 'embedding"; DROP TABLE pages; --',
|
||||
type: 'vector',
|
||||
dimensions: 1536,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
};
|
||||
expect(() =>
|
||||
resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()),
|
||||
).toThrow(EmbeddingColumnNotRegisteredError);
|
||||
});
|
||||
|
||||
test('passthrough re-validates type field (rejects unknown)', () => {
|
||||
const bad = {
|
||||
name: 'embedding_voyage',
|
||||
type: 'jsonb',
|
||||
dimensions: 1024,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
} as unknown as ResolvedColumn;
|
||||
expect(() =>
|
||||
resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()),
|
||||
).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('passthrough re-validates dimensions field (rejects out-of-range)', () => {
|
||||
const bad: ResolvedColumn = {
|
||||
name: 'embedding_voyage',
|
||||
type: 'vector',
|
||||
dimensions: -5,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
};
|
||||
expect(() =>
|
||||
resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()),
|
||||
).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('passthrough re-validates dimensions field (rejects SQL-shaped string)', () => {
|
||||
const bad = {
|
||||
name: 'embedding_voyage',
|
||||
type: 'halfvec',
|
||||
dimensions: '1); DROP TABLE pages; --',
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
} as unknown as ResolvedColumn;
|
||||
expect(() =>
|
||||
resolveEmbeddingColumn({ embeddingColumn: bad }, cfg()),
|
||||
).toThrow(EmbeddingColumnConfigError);
|
||||
});
|
||||
|
||||
test('valid descriptor passes through unchanged', () => {
|
||||
const good: ResolvedColumn = {
|
||||
name: 'embedding_ze',
|
||||
type: 'halfvec',
|
||||
dimensions: 2560,
|
||||
embeddingModel: 'zeroentropyai:zembed-1',
|
||||
};
|
||||
expect(resolveEmbeddingColumn({ embeddingColumn: good }, cfg())).toEqual(good);
|
||||
});
|
||||
});
|
||||
|
||||
describe('codex /ship #4 — isCacheSafe (embedding-space-based skip)', () => {
|
||||
test('default name + matching dim + matching model → safe', () => {
|
||||
const r: ResolvedColumn = {
|
||||
name: 'embedding',
|
||||
type: 'vector',
|
||||
dimensions: 1536,
|
||||
embeddingModel: 'openai:text-embedding-3-large',
|
||||
};
|
||||
expect(isCacheSafe(r, cfg())).toBe(true);
|
||||
});
|
||||
|
||||
test('non-default name → unsafe', () => {
|
||||
const r: ResolvedColumn = {
|
||||
name: 'embedding_voyage',
|
||||
type: 'vector',
|
||||
dimensions: 1024,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
};
|
||||
expect(isCacheSafe(r, cfg())).toBe(false);
|
||||
});
|
||||
|
||||
test('default name BUT overridden to different dim → unsafe', () => {
|
||||
// User overrode the `embedding` builtin to point at a 1024-dim Voyage
|
||||
// column. Name is still 'embedding' but the cache table is sized for
|
||||
// 1536d (or whatever the brain's cfg dim was at init). UNSAFE.
|
||||
const r: ResolvedColumn = {
|
||||
name: 'embedding',
|
||||
type: 'vector',
|
||||
dimensions: 1024,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
};
|
||||
expect(isCacheSafe(r, cfg({ embedding_dimensions: 1536 }))).toBe(false);
|
||||
});
|
||||
|
||||
test('default name BUT overridden to different model (same dim) → unsafe', () => {
|
||||
// Different model = different embedding space even at the same dim.
|
||||
// OpenAI 1536d vectors are NOT interchangeable with Cohere/Voyage 1536d.
|
||||
const r: ResolvedColumn = {
|
||||
name: 'embedding',
|
||||
type: 'vector',
|
||||
dimensions: 1536,
|
||||
embeddingModel: 'voyage:voyage-3-large',
|
||||
};
|
||||
expect(
|
||||
isCacheSafe(
|
||||
r,
|
||||
cfg({
|
||||
embedding_dimensions: 1536,
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
}),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
test('zero-config brain (cfg has no embedding_dimensions/model) → defaults match → safe', () => {
|
||||
const r: ResolvedColumn = {
|
||||
name: 'embedding',
|
||||
type: 'vector',
|
||||
dimensions: 1536,
|
||||
embeddingModel: 'openai:text-embedding-3-large',
|
||||
};
|
||||
expect(isCacheSafe(r, cfg())).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -43,7 +43,7 @@ function baseKnobs(): ResolvedSearchKnobs {
|
||||
}
|
||||
|
||||
describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
test('version is 3 (1→2 v0.35.0.0 reranker; 2→3 v0.35.6.0 floor_ratio)', () => {
|
||||
test('version is 3 (1→2 v0.35.0.0 reranker; 2→3 v0.35.6.0 floor_ratio + v0.36 embedding-column)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(3);
|
||||
});
|
||||
|
||||
@@ -150,4 +150,44 @@ describe('append-only convention (CDX2-F13)', () => {
|
||||
expect(rrIdx).toBeGreaterThan(0);
|
||||
expect(rrIdx).toBeGreaterThan(limIdx);
|
||||
});
|
||||
|
||||
test('v=3 additions: col= and prov= appear AFTER the reranker block', async () => {
|
||||
// v0.36 D8: cache-key contamination across embedding columns + providers.
|
||||
// The two new tokens must sit at the bottom of parts[] so existing v=2
|
||||
// hashes can only differ in those positions — keeping the append-only
|
||||
// chain auditable for future v=4 readers.
|
||||
const src = await Bun.file(
|
||||
new URL('../../src/core/search/mode.ts', import.meta.url),
|
||||
).text();
|
||||
const rrtIdx = src.indexOf('rrt=${knobs.reranker_timeout_ms');
|
||||
const colIdx = src.indexOf('col=${ctx?.embeddingColumn');
|
||||
const provIdx = src.indexOf('prov=${ctx?.embeddingModel');
|
||||
expect(rrtIdx).toBeGreaterThan(0);
|
||||
expect(colIdx).toBeGreaterThan(rrtIdx);
|
||||
expect(provIdx).toBeGreaterThan(colIdx);
|
||||
});
|
||||
|
||||
test('v=3 fields participate: column flip changes the hash', () => {
|
||||
const k = baseKnobs();
|
||||
const defaultCol = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'openai:text-embedding-3-large' });
|
||||
const voyageCol = knobsHash(k, { embeddingColumn: 'embedding_voyage', embeddingModel: 'voyage:voyage-3-large' });
|
||||
expect(defaultCol).not.toBe(voyageCol);
|
||||
});
|
||||
|
||||
test('v=3 fields participate: same column + different provider → different hash', () => {
|
||||
const k = baseKnobs();
|
||||
const a = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'openai:text-embedding-3-large' });
|
||||
const b = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'openai:text-embedding-3-small' });
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
test('v=3 fields fall back to embedding/default when ctx undefined', () => {
|
||||
// Backward-compat: callers that don't know the column (e.g. telemetry
|
||||
// helpers) should still produce a stable hash matching the default
|
||||
// 'embedding' + 'default' provider pair.
|
||||
const k = baseKnobs();
|
||||
const bare = knobsHash(k);
|
||||
const explicit = knobsHash(k, { embeddingColumn: 'embedding', embeddingModel: 'default' });
|
||||
expect(bare).toBe(explicit);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user