1 Commits

Include renames

Author	SHA1	Message	Date
3 people	1d5f69fe7a	v0.36.3.0 feat: dynamic embedding column selection for search (#1164) ... * feat: migration v68 — eval_candidates.embedding_column Schema migration ALTERs eval_candidates to add a nullable embedding_column TEXT column. Per-row capture metadata so `gbrain eval replay` reproduces the same column the capture ran against (D16 / CDX-10). NULL-tolerant: pre-v0.36 rows fall back to current default. Renumbered v67→v68 because master claimed v67 for facts_typed_claim_columns during this branch's lifetime. PGLite parity via sqlFor.pglite — same ALTER IF NOT EXISTS. * feat: dynamic embedding column — core (resolver, types, gateway, engines) The read-path foundation for routing search through any populated embedding column, not just OpenAI 1536. src/core/search/embedding-column.ts (new) is the canonical seam. Single source of truth for column → provider/dim/type lookup. Validates registry keys via regex (/^[a-z_][a-z0-9_]*$/), uses Object.create(null) + Object.hasOwn so 'constructor' and other inherited names can't masquerade as registered columns. Identifier-quoting on SQL interpolation as defense in depth. src/core/types.ts widens SearchOpts.embeddingColumn to accept ResolvedColumn descriptors at the engine boundary; adds EmbeddingColumnConfig + ResolvedColumn exports. src/core/config.ts merges embedding_columns + search_embedding_column from the DB plane via loadConfigWithEngine, mirroring the existing embedding_multimodal_model pattern. Handles the no-file case so env-only Postgres installs see DB-plane overrides (codex /ship #3). src/core/ai/gateway.ts: embedQuery(text, opts) + embed(texts, opts) accept embeddingModel + dimensions overrides. isAvailable(touchpoint, modelOverride?) so hybrid asks 'is the active column's provider reachable?' not 'is the global default reachable?' (CDX-4 / D10). Engines: searchVector accepts ResolvedColumn descriptors via normalizeEngineColumn; engine code is config-free and unit-testable. getEmbeddingsByChunkIds(ids, column?) so cosineReScore hydrates from the active column instead of always 'embedding' (CDX-3 / D9). Identifier-quoting belt at the SQL boundary. src/core/eval-capture.ts threads embedding_column from hybridSearch meta into the persisted capture row. * feat: dynamic embedding column — integration (hybrid, ops, doctor) Wires the resolver into hybridSearch, the query op, doctor, and the config command. src/core/search/hybrid.ts: resolves the column once at the boundary, threads the descriptor into engine calls, routes embedQuery through the resolved column's provider/dims, and calls isCacheSafe (not isDefaultColumn) for cache skip so user overrides of the 'embedding' builtin can't leak across vector spaces (CDX-4). cosineReScore now hydrates from the active column. src/core/search/mode.ts: KNOBS_HASH_VERSION 2→3, append-only new fields col= and prov= alongside floor_ratio. Cache rows from different columns or providers now sit in different keyspaces — cross-column contamination impossible. src/core/operations.ts: query op accepts embedding_column param for per-call A/B benchmarking. search op (keyword-only) deliberately does NOT (CDX-9 / D15) — would be silent UX. src/commands/doctor.ts: new embedding_column_registry check. Batch format_type probe (D13) catches dim drift that information_schema.columns.udt_name can't. Batch pg_indexes probe (D5) warns on missing HNSW. Coverage % on active column, gates at <90% (D14), short-circuits on empty brains (codex /ship #5). src/commands/config.ts: validates embedding_columns JSON shape at set time, runs the coverage gate when setting search_embedding_column, uses Object.hasOwn for the registry lookup. src/commands/eval-replay.ts: replay re-runs queries against the captured embedding_column so post-flip-config replays don't surface as false-positive regressions. * test: dynamic embedding column — unit + e2e coverage 50 unit cases for the resolver (resolution chain, registry merge, validation, prototype pollution, descriptor passthrough, isCacheSafe, normalizeEngineColumn). 8 gateway override cases — embeddingModel + dimensions flow into providerOptions, isAvailable(touchpoint, override) routes to the right recipe, unknown models throw clean. 4 cosineReScore + 6 ops + 5 knobs-hash + 7 mode + 9 PGLite E2E + 7 Postgres E2E + 5 eval-replay column metadata. Postgres E2E (gated on DATABASE_URL) covers halfvec(2560) end-to-end on real pgvector, EXPLAIN-visible HNSW index on the alternate column, format_type-based dim drift catch, and the <90% coverage gate. Pins every codex /ship fix: prototype-pollution rejection ('constructor' as column name), descriptor passthrough validation (rejects SQL-shaped strings in dimensions), isCacheSafe semantics (space-based, not name-based). Total: 141 new + extended cases, all green. * chore: bump version and changelog (v0.36.3.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync to v0.36.3.0 Add CLAUDE.md key-files entry for src/core/search/embedding-column.ts. Annotate hybrid.ts, gateway.ts, doctor.ts, and migrate.ts entries with v0.36.3.0 wave changes (ResolvedColumn threading, embedQuery model override, embedding_column_registry check, migration v68). Document knobs_hash v=2 → v=3 bump under the Search Mode section. Regenerate llms-full.txt from the updated CLAUDE.md so the auto-checked bundle matches source (build-llms.test.ts CI guard). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): two CI failures from v0.36.3.0 1. test/loadConfig-merge.test.ts: update the 'returns null when base config is null' contract test. Pre-v0.36 the function returned null for null base; the codex /ship #3 fix changed that to synthesize a minimal `{ engine: 'postgres' }` so env-only installs see DB-plane overrides. Test now pins the new contract + adds a round-trip case asserting the merge actually surfaces `embedding_columns` / `search_embedding_column` set via gbrain config set on a null base. 2. test/schema-bootstrap-coverage.test.ts was failing because eval_candidates.embedding_column (added by migration v68) wasn't covered by applyForwardReferenceBootstrap. Fix: add the column to PGLITE_SCHEMA_SQL's eval_candidates CREATE TABLE definition (and src/schema.sql for parity) so fresh installs get it natively. The coverage test's third tier (schemaCreateTableCols) now finds it. Regenerated schema-embedded.ts via bun run build:schema. Schema-blob path is cleaner than COLUMN_EXEMPTIONS — fresh installs skip the migration entirely; upgrade installs still run v68. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 21:26:12 -07:00