diff --git a/CHANGELOG.md b/CHANGELOG.md index ecfe09b0e..e3f1ecaa0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,23 @@ All notable changes to GBrain will be documented in this file. +## [0.42.34.0] - 2026-06-07 + +**Relationship questions now get relationship answers.** Ask "who invested in widget-co", "who introduced me to alice-example", or "what connects fund-a and fund-b" and gbrain resolves the named entity and walks its typed-edge graph (`invested_in`, `works_at`, `founded`, `attended`, `advises`, …) to surface the answer — even when no single page mentions both sides. Until now the graph only re-ranked results that keyword/vector search had already found; a relationship that lived purely in the edges (an investor whose page never names the company) was invisible. It now enters retrieval as a first-class candidate. + +This is on by default in the `balanced` and `tokenmax` search modes, a pure no-op for non-relational questions and for brains with no typed edges, and off in `conservative`. On a benchmark of relationship queries whose answers are unreachable by content similarity, recall@10 goes from near-zero to over 75%. The traversal is deterministic (same query + brain → same answer), stays within a single source (it never crosses a mounted-brain boundary), excludes noisy body-text "mentions" edges by default, and is depth- and fan-out-bounded so a popular hub entity can't blow up a query. + +### Added +- **Typed-edge relational retrieval** (`search.relational_retrieval`, on for balanced/tokenmax). Relational questions resolve their seed entity and traverse the typed-edge graph, injecting edge-derived answers as a fourth fusion arm alongside keyword + vector. Relation vocabulary is schema-pack-extensible: a pack that defines its own link types can declare the query phrases that retrieve them. The `query` operation gains a `relational` flag (omit for the smart default; pass `false` to force lexical/vector-only). Results carry `--explain` attribution ("surfaced via invested_in from widget-co") and, for "what connects A and B", the connecting path. +- **`relationalFanout` engine method** (PGLite + Postgres, in lockstep). Seed-array typed-edge fan-out aggregating to ranked nodes (shortest hop, edge richness, connecting path, canonical chunk), source-scoped and deterministic. +- **`gbrain eval retrieval-quality --ab-relational`**: A/Bs the arm off vs on over a question set and reports the recall@10 lift + latency. The retrieval-quality harness gains recall@k / recall@10 metrics. + +### Fixed +- **Cross-source result collapse.** The search fusion/dedup key now carries `source_id`, so two pages that share a slug across mounted brains no longer merge into one result. The semantic query cache is likewise scoped per source-set, so a federated search can't be served a single-source cached result. + +### To take advantage of v0.42.34.0 +- Just ask relationship questions in natural language — the arm is on by default in balanced/tokenmax. To turn it off: `gbrain config set search.relational_retrieval false`. +- One-time cache note: this release advances the search-cache key version (a relational-on result must not be served to a relational-off lookup), so the first query after upgrade re-runs instead of hitting a stale cache row. No action needed; it self-heals on first use. ## [0.42.33.0] - 2026-06-07 **`gbrain sync` will never delete a repo it didn't create.** If a source was registered with a `remote_url` but its `local_path` pointed at a working tree you manage yourself (not a gbrain-managed clone), a failed or degraded code sync could remove that directory and re-clone over it. Sync now re-clones **only** clones gbrain actually created — identified by an ownership marker, or by gbrain's own clone location for clones made before this release. Anything else, including your live working tree, is treated as read-only: indexed, never deleted. On an unowned path, sync aborts loudly **before touching the filesystem** and tells you how to fix the source registration. Thanks to @zaqwery for the report. diff --git a/CLAUDE.md b/CLAUDE.md index cbdb58c52..6645fe69a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -149,6 +149,7 @@ project resolves through `src/core/search/mode.ts`. | `intentWeighting` | true | true | true | | `tokenBudget` | **4000** | **12000** | **off** | | `expansion` (LLM multi-query) | false | false | **true** | +| `relationalRetrieval` | false | **true** | **true** | | `searchLimit` default | 10 | 25 | 50 | **Cost anchors (downstream agent input cost — gbrain itself is rounding error).** @@ -207,6 +208,19 @@ written against `embedding` (1536d OpenAI). Existing v=2 rows become unreachable on first re-query (one-time miss spike on upgrade); `mode.ts:KNOBS_HASH_VERSION` is the single source of truth. +**v0.42.34.0 knobs_hash v=9 → v=10.** Folds the `relationalRetrieval` knob + +depth into the cache key so a relational-on result set can't be served to a +relational-off lookup (same contamination class as graph_signals). One-time +miss spike on upgrade. + +**Relational retrieval (v0.42.34.0).** `relationalRetrieval` (on for +balanced/tokenmax) adds a fourth recall arm: a relational query ("who invested +in X", "what connects A and B") resolves its seed entity and walks the typed-edge +graph (`src/core/search/relational-recall.ts` + `relational-intent.ts`, +`engine.relationalFanout`), injecting edge-derived answers into RRF. Within-source, +deterministic, mentions-excluded by default, pure no-op for non-relational queries. +The `query` op's `relational` flag forces it on/off per call. + **Three CLI surfaces:** gbrain search modes # what is running, with per-knob attribution diff --git a/VERSION b/VERSION index 88025242a..5f3886e1f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.33.0 \ No newline at end of file +0.42.34.0 diff --git a/docs/architecture/RETRIEVAL.md b/docs/architecture/RETRIEVAL.md index 82d0e2a98..4b2550b8c 100644 --- a/docs/architecture/RETRIEVAL.md +++ b/docs/architecture/RETRIEVAL.md @@ -123,6 +123,7 @@ expansion (if enabled) hybrid search: ├── vector (HNSW on chunk embeddings) ├── keyword (BM25 via tsvector) + ├── relational (v0.42.34.0: typed-edge recall arm — relational queries only) ├── source-aware re-rank (CASE in SQL) └── RRF fusion → top 30 │ diff --git a/llms-full.txt b/llms-full.txt index 018b531f1..8b3579bdc 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -297,6 +297,7 @@ project resolves through `src/core/search/mode.ts`. | `intentWeighting` | true | true | true | | `tokenBudget` | **4000** | **12000** | **off** | | `expansion` (LLM multi-query) | false | false | **true** | +| `relationalRetrieval` | false | **true** | **true** | | `searchLimit` default | 10 | 25 | 50 | **Cost anchors (downstream agent input cost — gbrain itself is rounding error).** @@ -355,6 +356,19 @@ written against `embedding` (1536d OpenAI). Existing v=2 rows become unreachable on first re-query (one-time miss spike on upgrade); `mode.ts:KNOBS_HASH_VERSION` is the single source of truth. +**v0.42.34.0 knobs_hash v=9 → v=10.** Folds the `relationalRetrieval` knob + +depth into the cache key so a relational-on result set can't be served to a +relational-off lookup (same contamination class as graph_signals). One-time +miss spike on upgrade. + +**Relational retrieval (v0.42.34.0).** `relationalRetrieval` (on for +balanced/tokenmax) adds a fourth recall arm: a relational query ("who invested +in X", "what connects A and B") resolves its seed entity and walks the typed-edge +graph (`src/core/search/relational-recall.ts` + `relational-intent.ts`, +`engine.relationalFanout`), injecting edge-derived answers into RRF. Within-source, +deterministic, mentions-excluded by default, pure no-op for non-relational queries. +The `query` op's `relational` flag forces it on/off per call. + **Three CLI surfaces:** gbrain search modes # what is running, with per-knob attribution diff --git a/package.json b/package.json index b962e4d6a..7d6f87318 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.33.0" + "version": "0.42.34.0" } diff --git a/src/commands/eval-retrieval-quality.ts b/src/commands/eval-retrieval-quality.ts index 175a1e7a3..7d48dc10f 100644 --- a/src/commands/eval-retrieval-quality.ts +++ b/src/commands/eval-retrieval-quality.ts @@ -20,12 +20,13 @@ import { export async function runEvalRetrievalQuality(engine: BrainEngine, args: string[]): Promise { const json = args.includes('--json'); + const abRelational = args.includes('--ab-relational'); const sourceIdx = args.indexOf('--source'); const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined; const fixture = args.find(a => !a.startsWith('--') && a !== sourceId); if (!fixture) { - console.error('Usage: gbrain eval retrieval-quality [--json] [--source ]'); + console.error('Usage: gbrain eval retrieval-quality [--json] [--source ] [--ab-relational]'); process.exit(2); } @@ -47,6 +48,48 @@ export async function runEvalRetrievalQuality(engine: BrainEngine, args: string[ return results.map(r => r.slug); }; + // v0.43 — A/B the relational recall arm (off vs on) over the same questions + // in a fixed mode (expansion off, skipCache-equivalent via bare hybridSearch) + // so the delta is purely the arm. Headline: graph-relationship recall@10 lift. + if (abRelational) { + const mk = (relationalRetrieval: boolean): SearchFn => async (q) => { + const results = await hybridSearch(engine, q, { + limit: 10, relationalRetrieval, expansion: false, + ...(sourceId ? { sourceId } : {}), + }); + return results.map(r => r.slug); + }; + const t0 = Date.now(); + const off = await runRetrievalQuality(questions, mk(false)); + const tMid = Date.now(); + const on = await runRetrievalQuality(questions, mk(true)); + const tEnd = Date.now(); + const fam = (r: typeof off, f: string) => r.families.find(x => x.family === f); + const rel = { off: fam(off, 'graph-relationship'), on: fam(on, 'graph-relationship') }; + const payload = { + schema_version: 1 as const, + ab: 'relational', + graph_relationship: { + n: rel.on?.n ?? 0, + recall_at_10: { off: rel.off?.recall_at_10 ?? 0, on: rel.on?.recall_at_10 ?? 0, + delta: (rel.on?.recall_at_10 ?? 0) - (rel.off?.recall_at_10 ?? 0) }, + hit_at_3: { off: rel.off?.hit_at_3 ?? 0, on: rel.on?.hit_at_3 ?? 0 }, + }, + latency_ms: { off_total: tMid - t0, on_total: tEnd - tMid }, + families: { off: off.families, on: on.families }, + }; + if (json) { + console.log(JSON.stringify(payload, null, 2)); + } else { + console.log(`Relational A/B — graph-relationship n=${payload.graph_relationship.n}`); + const g = payload.graph_relationship; + console.log(` recall@10: off=${(g.recall_at_10.off * 100).toFixed(0)}% on=${(g.recall_at_10.on * 100).toFixed(0)}% Δ=+${(g.recall_at_10.delta * 100).toFixed(0)}pp`); + console.log(` Hit@3: off=${(g.hit_at_3.off * 100).toFixed(0)}% on=${(g.hit_at_3.on * 100).toFixed(0)}%`); + console.log(` latency: off=${payload.latency_ms.off_total}ms on=${payload.latency_ms.on_total}ms (arm adds ${payload.latency_ms.on_total - payload.latency_ms.off_total}ms over ${payload.graph_relationship.n} queries)`); + } + process.exit(0); + } + const report = await runRetrievalQuality(questions, searchFn); const gate = evaluateGate(report); diff --git a/src/commands/search.ts b/src/commands/search.ts index b2738a6b1..971010979 100644 --- a/src/commands/search.ts +++ b/src/commands/search.ts @@ -71,6 +71,9 @@ const KNOB_DESCRIPTIONS: Record = { // v0.42.3.0 autocut autocut: 'Score-discontinuity result-sizing (cuts at the rerank-score cliff; no-op without a reranker)', autocut_jump: 'Autocut sensitivity: min normalized score gap that counts as a cliff (0..1, 0.20 default)', + // v0.43 relational recall + relationalRetrieval: 'Typed-edge relational recall arm (relational queries walk the graph; no-op otherwise)', + relational_retrieval_depth: 'Max hops for relational traversal (1..3, 2 default)', }; interface SearchModesReport { diff --git a/src/core/engine.ts b/src/core/engine.ts index a75e30110..0f285d498 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -2,7 +2,7 @@ import type { Page, PageInput, PageFilters, GetPageOpts, Chunk, ChunkInput, StaleChunkRow, StalePageRow, SearchResult, SearchOpts, - Link, GraphNode, GraphPath, + Link, GraphNode, GraphPath, RelationalFanoutRow, RelationalFanoutOpts, TimelineEntry, TimelineInput, TimelineOpts, RawData, PageVersion, @@ -1197,6 +1197,30 @@ export interface BrainEngine { slug: string, opts?: { depth?: number; linkType?: string; direction?: 'in' | 'out' | 'both'; sourceId?: string; sourceIds?: string[] }, ): Promise; + /** + * Typed-edge relational fan-out for the relational recall arm (v0.43). + * + * Generalizes traversePaths to a SEED ARRAY and aggregates to ranked NODES + * (not edges): for each reached page, the shortest hop from any seed, a + * connection-richness count, the edge types it was reached by, the shortest + * connecting slug path, and the page's canonical (lowest-ordinal) chunk id. + * + * Invariants: + * - WITHIN-SOURCE: a walk never crosses a source boundary (each branch + * stays in its seed's own source), even when multiple sources are in + * scope. Cross-source edge traversal is a separate, security-reviewed + * feature. + * - `link_source='mentions'` edges are EXCLUDED by default (noisy + the + * densest fan-out source); opt in via `includeMentions`. + * - `deleted_at IS NULL` at seed, every neighbor, and every returned node. + * - Deterministic ordering: hop ASC, edge_count DESC, source_id, slug. + * Must move in lockstep with the PGLite implementation + * (test/e2e/engine-parity.test.ts). + */ + relationalFanout( + seeds: string[], + opts?: RelationalFanoutOpts, + ): Promise; /** * For a list of slugs, return how many inbound links each has. * Used by hybrid search backlink boost. Single SQL query, not N+1. diff --git a/src/core/operations.ts b/src/core/operations.ts index 56b0eb801..8739a2232 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -1431,6 +1431,11 @@ const query: Operation = { " TRUE is redundant in default mode (it's already on); it only matters to override a brain whose config turned autocut off.\n" + "Safe by construction: never returns empty when there are matches, only applies to the first page (omit when paginating), and is a no-op when no reranker scored the results (so it can't cut on an untrustworthy signal). Distinct from `adaptive_return`: autocut cuts on the score cliff; adaptive_return caps by question intent. Leave both unset for the smart default.", }, + relational: { + type: 'boolean', + description: + "v0.43 — relational recall arm. SMART DEFAULT (on in balanced/tokenmax). When the question is about a RELATIONSHIP ('who invested in widget-co', 'who introduced me to alice', 'what connects fund-a and fund-b'), the brain resolves the named entity and walks its typed-edge graph (invested_in, works_at, founded, …), surfacing the answer even when no passage mentions both sides. Pure no-op for non-relational questions. Pass FALSE to force lexical/vector-only retrieval (e.g. debugging why a graph answer appeared). You almost never set this.", + }, }, handler: async (ctx, p) => { const startedAt = Date.now(); @@ -1528,6 +1533,8 @@ const query: Operation = { // v0.42.3.0 — autocut ceiling override. Omitted = smart default (ON in // reranked modes). `false` forces the full top-K. autocut: typeof p.autocut === 'boolean' ? (p.autocut as boolean) : undefined, + // v0.43 — relational recall override. Omitted = smart default (mode bundle). + relationalRetrieval: typeof p.relational === 'boolean' ? (p.relational as boolean) : undefined, }); const latency_ms = Date.now() - startedAt; diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 9df57522a..ca727cd5f 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -2859,6 +2859,90 @@ export class PGLiteEngine implements BrainEngine { return result; } + async relationalFanout( + seeds: string[], + opts?: import('./types.ts').RelationalFanoutOpts, + ): Promise { + if (!seeds || seeds.length === 0) return []; + const depth = Math.min(Math.max(1, opts?.depth ?? 2), 3); + const direction = opts?.direction ?? 'both'; + const limit = Math.min(Math.max(1, opts?.limit ?? 50), 200); + const types = opts?.linkTypes && opts.linkTypes.length > 0 ? opts.linkTypes : null; + + // $1=seeds, $2=depth, $3=limit; optional scope/type params appended. + const params: unknown[] = [seeds, depth, limit]; + const useSourceIds = opts?.sourceIds && opts.sourceIds.length > 0; + let seedScope = ''; + if (useSourceIds) { + params.push(opts!.sourceIds); + seedScope = `AND p.source_id = ANY($${params.length}::text[])`; + } else if (opts?.sourceId) { + params.push(opts.sourceId); + seedScope = `AND p.source_id = $${params.length}`; + } + let typeFilter = ''; + if (types) { + params.push(types); + typeFilter = `AND l.link_type = ANY($${params.length}::text[])`; + } + const mentionsFilter = opts?.includeMentions ? '' : `AND l.link_source IS DISTINCT FROM 'mentions'`; + + const recurStep = + direction === 'out' + ? `JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id` + : direction === 'in' + ? `JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id` + : `JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id) + JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END`; + + const sql = ` + WITH RECURSIVE walk AS ( + SELECT p.id, p.slug, p.source_id, 0::int AS depth, + ARRAY[p.id] AS visited, ARRAY[p.slug] AS path, + p.source_id AS seed_source, NULL::text AS last_link_type + FROM pages p + WHERE p.slug = ANY($1::text[]) ${seedScope} AND p.deleted_at IS NULL + UNION ALL + SELECT p2.id, p2.slug, p2.source_id, w.depth + 1, + w.visited || p2.id, w.path || p2.slug, + w.seed_source, l.link_type + FROM walk w + ${recurStep} + WHERE w.depth < $2 + AND NOT (p2.id = ANY(w.visited)) + AND p2.source_id = w.seed_source + AND p2.deleted_at IS NULL + ${mentionsFilter} + ${typeFilter} + ) + SELECT n.source_id, n.slug, + MIN(n.depth) AS hop, + COUNT(DISTINCT n.last_link_type) AS edge_count, + array_agg(DISTINCT n.last_link_type) + FILTER (WHERE n.last_link_type IS NOT NULL) AS via_link_types, + (array_agg(array_to_string(n.path, chr(9)) + ORDER BY n.depth ASC, array_length(n.path, 1) ASC))[1] AS path_str, + (SELECT cc.id FROM content_chunks cc + WHERE cc.page_id = n.id ORDER BY cc.chunk_index ASC LIMIT 1) AS canonical_chunk_id + FROM walk n + WHERE n.depth > 0 + GROUP BY n.source_id, n.slug, n.id + ORDER BY hop ASC, edge_count DESC, n.source_id ASC, n.slug ASC + LIMIT $3 + `; + + const { rows } = await this.db.query(sql, params); + return (rows as Record[]).map(r => ({ + source_id: r.source_id as string, + slug: r.slug as string, + hop: Number(r.hop), + edge_count: Number(r.edge_count), + via_link_types: Array.isArray(r.via_link_types) ? (r.via_link_types as string[]) : [], + path: r.path_str ? String(r.path_str).split('\t') : [], + canonical_chunk_id: r.canonical_chunk_id == null ? null : Number(r.canonical_chunk_id), + })); + } + async getBacklinkCounts(slugs: string[]): Promise> { const result = new Map(); if (slugs.length === 0) return result; diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 7066c6934..ba0218a61 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -2950,6 +2950,88 @@ export class PostgresEngine implements BrainEngine { return result; } + async relationalFanout( + seeds: string[], + opts?: import('./types.ts').RelationalFanoutOpts, + ): Promise { + if (!seeds || seeds.length === 0) return []; + const sql = this.sql; + const depth = Math.min(Math.max(1, opts?.depth ?? 2), 3); + const direction = opts?.direction ?? 'both'; + const limit = Math.min(Math.max(1, opts?.limit ?? 50), 200); + const types = opts?.linkTypes && opts.linkTypes.length > 0 ? opts.linkTypes : null; + + // Scope is applied to SEED selection only. Within-source traversal is + // enforced separately by `p2.source_id = w.seed_source` in the recursive + // step, so a walk can never cross a source boundary even when several + // sources are in scope. + const useSourceIds = opts?.sourceIds && opts.sourceIds.length > 0; + const seedScope = useSourceIds + ? sql`AND p.source_id = ANY(${opts!.sourceIds!}::text[])` + : opts?.sourceId + ? sql`AND p.source_id = ${opts.sourceId}` + : sql``; + const typeFilter = types ? sql`AND l.link_type = ANY(${types}::text[])` : sql``; + const mentionsFilter = opts?.includeMentions + ? sql`` + : sql`AND l.link_source IS DISTINCT FROM 'mentions'`; + + // Recursive step join differs by direction; everything else is shared. + const recurStep = + direction === 'out' + ? sql`JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id` + : direction === 'in' + ? sql`JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id` + : sql`JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id) + JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END`; + + const rows = await sql` + WITH RECURSIVE walk AS ( + SELECT p.id, p.slug, p.source_id, 0::int AS depth, + ARRAY[p.id] AS visited, ARRAY[p.slug] AS path, + p.source_id AS seed_source, NULL::text AS last_link_type + FROM pages p + WHERE p.slug = ANY(${seeds}::text[]) ${seedScope} AND p.deleted_at IS NULL + UNION ALL + SELECT p2.id, p2.slug, p2.source_id, w.depth + 1, + w.visited || p2.id, w.path || p2.slug, + w.seed_source, l.link_type + FROM walk w + ${recurStep} + WHERE w.depth < ${depth} + AND NOT (p2.id = ANY(w.visited)) + AND p2.source_id = w.seed_source + AND p2.deleted_at IS NULL + ${mentionsFilter} + ${typeFilter} + ) + SELECT n.source_id, n.slug, + MIN(n.depth) AS hop, + COUNT(DISTINCT n.last_link_type) AS edge_count, + array_agg(DISTINCT n.last_link_type) + FILTER (WHERE n.last_link_type IS NOT NULL) AS via_link_types, + (array_agg(array_to_string(n.path, chr(9)) + ORDER BY n.depth ASC, array_length(n.path, 1) ASC))[1] AS path_str, + (SELECT cc.id FROM content_chunks cc + WHERE cc.page_id = n.id ORDER BY cc.chunk_index ASC LIMIT 1) AS canonical_chunk_id + FROM walk n + WHERE n.depth > 0 + GROUP BY n.source_id, n.slug, n.id + ORDER BY hop ASC, edge_count DESC, n.source_id ASC, n.slug ASC + LIMIT ${limit} + `; + + return (rows as Record[]).map(r => ({ + source_id: r.source_id as string, + slug: r.slug as string, + hop: Number(r.hop), + edge_count: Number(r.edge_count), + via_link_types: Array.isArray(r.via_link_types) ? (r.via_link_types as string[]) : [], + path: r.path_str ? String(r.path_str).split('\t') : [], + canonical_chunk_id: r.canonical_chunk_id == null ? null : Number(r.canonical_chunk_id), + })); + } + async getBacklinkCounts(slugs: string[]): Promise> { const result = new Map(); if (slugs.length === 0) return result; diff --git a/src/core/search/hybrid.ts b/src/core/search/hybrid.ts index 179bdfb95..ee1b08c6e 100644 --- a/src/core/search/hybrid.ts +++ b/src/core/search/hybrid.ts @@ -23,6 +23,7 @@ import { type AdaptiveReturnDecision, } from './return-policy.ts'; import { applyAutocut, type AutocutDecision } from './autocut.ts'; +import { buildRelationalArm } from './relational-recall.ts'; import { loadConfigWithEngine } from '../config.ts'; import { dedupResults } from './dedup.ts'; import { applyReranker } from './rerank.ts'; @@ -664,6 +665,9 @@ export async function applyAliasHop( export interface HybridSearchOpts extends SearchOpts { expansion?: boolean; + /** v0.43 — observability sink for the relational recall arm (fired/no-op, + * kind, seeds resolved, candidates, errored). Best-effort. */ + onRelationalMeta?: (meta: import('./relational-recall.ts').RelationalArmMeta) => void; /** * T4/D5 — per-call search-mode selector (one of SEARCH_MODES). Selects the * whole mode bundle for this call, overriding the server-configured mode. @@ -806,6 +810,11 @@ export async function hybridSearch( // Non-boolean AutocutInput shapes (Partial) aren't a v1 per-call surface, // so only the boolean toggle threads here. autocut: typeof opts?.autocut === 'boolean' ? opts.autocut : undefined, + // v0.43 — relational recall per-call thread-through. Per-call wins over + // config override wins over mode bundle; without this the A/B eval gate + // would be a no-op (both branches resolve to the same mode default). + relationalRetrieval: opts?.relationalRetrieval, + relational_retrieval_depth: opts?.relationalRetrievalDepth, }, }); @@ -972,6 +981,23 @@ export async function hybridSearch( titleBoost: resolvedMode.title_boost, }; + // v0.43 — build the relational recall arm ONCE here, before any return + // path, so typed-edge answers contribute on ALL THREE paths: the + // no-embedding-provider path, the embed-failed keyword fallback, and the + // main RRF path. Parsed from the original query (deterministic); empty for + // non-relational queries → pure no-op. (Modality gate lives on the main + // path; the parser only matches text-shaped relational queries anyway.) + let relationalList: SearchResult[] = []; + if (resolvedMode.relationalRetrieval) { + relationalList = await buildRelationalArm(engine, query, { + sourceId: opts?.sourceId, + sourceIds: opts?.sourceIds, + depth: resolvedMode.relational_retrieval_depth, + limit: opts?.limit ?? resolvedMode.searchLimit, + onMeta: opts?.onRelationalMeta, + }); + } + // Skip vector search entirely if the gateway has no embedding provider configured (Codex C3). // v0.36 (D10): ask "is the RESOLVED column's provider reachable?" rather // than "is the global default reachable?" — otherwise an unreachable @@ -980,13 +1006,24 @@ export async function hybridSearch( const { isAvailable } = await import('../ai/gateway.ts'); const providerProbe = resolvedCol.embeddingModel || undefined; if (!isAvailable('embedding', providerProbe)) { - if (keywordResults.length > 0) { - await runPostFusionStages(engine, keywordResults, postFusionOpts); - keywordResults.sort((a, b) => b.score - a.score); + // v0.43 — fuse the relational arm with keyword so typed-edge answers + // survive on the no-embedding-provider path (the relational win is most + // valuable exactly when vector is unavailable). + let noEmbedResults = keywordResults; + if (relationalList.length > 0) { + const fk = opts?.rrfK ?? RRF_K; + noEmbedResults = rrfFusionWeighted( + [{ list: keywordResults, k: fk }, { list: relationalList, k: fk }], + detailResolved !== 'high', + ); + } + if (noEmbedResults.length > 0) { + await runPostFusionStages(engine, noEmbedResults, postFusionOpts); + noEmbedResults.sort((a, b) => b.score - a.score); } // T3/T4 — alias hop + evidence stamp even without an embedding provider // (the named-thing fix is most valuable exactly when vector is unavailable). - const noEmbedHopped = await applyAliasHop(engine, dedupResults(keywordResults), query, { + const noEmbedHopped = await applyAliasHop(engine, dedupResults(noEmbedResults), query, { sourceId: opts?.sourceId, sourceIds: opts?.sourceIds, }); @@ -1204,11 +1241,21 @@ export async function hybridSearch( // v0.29.1 codex pass-2 #4: this is the third return path. Apply // post-fusion stages here too — without it, salience='on' silently // does nothing on embed failures. - if (keywordResults.length > 0) { - await runPostFusionStages(engine, keywordResults, postFusionOpts); - keywordResults.sort((a, b) => b.score - a.score); + // v0.43: fuse the relational arm with keyword via RRF so typed-edge + // answers survive even when vector is unavailable. + let fallbackResults = keywordResults; + if (relationalList.length > 0) { + const fk = opts?.rrfK ?? RRF_K; + fallbackResults = rrfFusionWeighted( + [{ list: keywordResults, k: fk }, { list: relationalList, k: fk }], + detail !== 'high', + ); } - const kwHopped = await applyAliasHop(engine, dedupResults(keywordResults), query, { + if (fallbackResults.length > 0) { + await runPostFusionStages(engine, fallbackResults, postFusionOpts); + fallbackResults.sort((a, b) => b.score - a.score); + } + const kwHopped = await applyAliasHop(engine, dedupResults(fallbackResults), query, { sourceId: opts?.sourceId, sourceIds: opts?.sourceIds, }); @@ -1266,6 +1313,16 @@ export async function hybridSearch( ...vectorLists.map(list => ({ list, k: vectorK })), { list: keywordResults, k: keywordK }, ]; + + // v0.43 — relational recall arm (fourth RRF arm), built above so it also + // contributes on the keyword-only fallback path. Neutral weight (baseRrfK): + // competes evenly with keyword/vector, not dominating. Empty for + // non-relational queries → pure no-op. Rides every downstream stage (cosine + // re-score, post-fusion boosts, dedup, reranker, autocut, token budget). + if (relationalList.length > 0 && effectiveModality !== 'image') { + allLists.push({ list: relationalList, k: baseRrfK }); + } + let fused = rrfFusionWeighted(allLists, detail !== 'high'); // Cosine re-scoring before dedup so semantically better chunks survive. @@ -1511,6 +1568,11 @@ export async function hybridSearchCached( // this, an `autocut:false` (full top-K) call could be served a trimmed // autocut-on cache row, or vice versa. autocut: typeof opts?.autocut === 'boolean' ? opts.autocut : undefined, + // v0.43 — relational recall per-call thread-through. Per-call wins over + // config override wins over mode bundle; without this the A/B eval gate + // would be a no-op (both branches resolve to the same mode default). + relationalRetrieval: opts?.relationalRetrieval, + relational_retrieval_depth: opts?.relationalRetrievalDepth, }, }); // v0.36 (D8 / CDX-2 + codex /ship #4): resolve column for the cache @@ -1607,7 +1669,7 @@ export async function hybridSearchCached( } if (!skipCache && queryEmbedding && cacheStatus !== 'disabled') { - const hit = await cache.lookup(queryEmbedding, { sourceId: opts?.sourceId, knobsHash: cacheKnobsHash }); + const hit = await cache.lookup(queryEmbedding, { sourceId: cacheScopeKey(opts), knobsHash: cacheKnobsHash }); if (hit.hit && hit.results) { cacheStatus = 'hit'; cacheSimilarity = hit.similarity; @@ -1708,7 +1770,7 @@ export async function hybridSearchCached( ) { trackCacheWrite( cache - .store(query, queryEmbedding, results, finalMeta, { sourceId: opts?.sourceId, knobsHash: cacheKnobsHash }) + .store(query, queryEmbedding, results, finalMeta, { sourceId: cacheScopeKey(opts), knobsHash: cacheKnobsHash }) .catch(() => { /* swallow */ }), ); } @@ -1716,6 +1778,42 @@ export async function hybridSearchCached( return budgeted; } +/** + * RRF/dedup identity for a result row, at chunk granularity. + * + * Includes `source_id` so two same-slug pages in different federated sources + * don't collapse into one fusion entry (the same composite-key discipline + * `dedup.ts:pageKey` already uses at page granularity). Pre-fix the key was + * `slug:chunk_id`, which silently merged cross-source rows and let a + * synthetic chunkless row (chunk_id null) key on a text prefix; the + * `(source_id, slug, chunk_id)` shape is collision-safe for both. + */ +function rrfKey(r: SearchResult): string { + const source = r.source_id ?? 'default'; + return `${source}:${r.slug}:${r.chunk_id ?? r.chunk_text.slice(0, 50)}`; +} + +/** + * Canonical query-cache scope key. + * + * The semantic cache stores results keyed by `(scope, query, knobs_hash)`. + * A federated search (`sourceIds`) reads a different graph than a + * single-source one, so the two must never share a cache row. Pre-fix the + * cache only saw scalar `sourceId`; a federated query fell through to + * `'default'` and could cross-serve an unrelated scope. + * + * - federated (sourceIds set) → `__set__:` + sorted, comma-joined ids + * (order-independent; two different source-sets get distinct keys) + * - scalar sourceId → the id itself (single-source unchanged) + * - unscoped → `'default'` (single-source brains unchanged) + */ +export function cacheScopeKey(opts?: { sourceId?: string; sourceIds?: string[] }): string { + if (opts?.sourceIds && opts.sourceIds.length > 0) { + return '__set__:' + [...opts.sourceIds].sort().join(','); + } + return opts?.sourceId ?? 'default'; +} + /** * v0.32.x search-lite — weighted RRF. Each list contributes with its own * effective k value, which lets intent weighting bias keyword vs vector @@ -1731,7 +1829,7 @@ export function rrfFusionWeighted( for (const { list, k } of lists) { for (let rank = 0; rank < list.length; rank++) { const r = list[rank]; - const key = `${r.slug}:${r.chunk_id ?? r.chunk_text.slice(0, 50)}`; + const key = rrfKey(r); const existing = scores.get(key); const rrfScore = 1 / (k + rank); @@ -1771,7 +1869,7 @@ export function rrfFusion(lists: SearchResult[][], k: number, applyBoost = true) for (const list of lists) { for (let rank = 0; rank < list.length; rank++) { const r = list[rank]; - const key = `${r.slug}:${r.chunk_id ?? r.chunk_text.slice(0, 50)}`; + const key = rrfKey(r); const existing = scores.get(key); const rrfScore = 1 / (k + rank); diff --git a/src/core/search/mode.ts b/src/core/search/mode.ts index 9fe7eb9b3..d5a8198e6 100644 --- a/src/core/search/mode.ts +++ b/src/core/search/mode.ts @@ -261,6 +261,18 @@ export interface ModeBundle { * run. Override: `search.autocut_jump` config → mode bundle. */ autocut_jump: number; + /** + * v0.43 — relational recall arm. When on, a relational query ("who invested + * in widget-co", "what connects fund-a and fund-b") resolves its seed + * entity and walks the typed-edge graph, injecting edge-derived candidates + * as a fourth RRF arm. Pure no-op for non-relational queries. Default OFF + * for conservative; ON for balanced/tokenmax. Override path: per-call + * SearchOpts.relationalRetrieval → `search.relational_retrieval` config → + * mode bundle. See src/core/search/relational-recall.ts. + */ + relationalRetrieval: boolean; + /** v0.43 — max hops for relational traversal. Default 2, hard-capped at 3. */ + relational_retrieval_depth: number; } /** @@ -309,6 +321,10 @@ export const MODE_BUNDLES: Readonly>> = // v0.42.3.0 — autocut OFF: conservative has no reranker, so no trustworthy // cliff signal exists (autocut would no-op). Explicit for clarity. autocut: false, + // v0.43 — relational recall OFF for conservative (cost-sensitive tier, + // matches graph_signals posture). Power users opt in per-call. + relationalRetrieval: false, + relational_retrieval_depth: 2, autocut_jump: 0.2, }), balanced: Object.freeze({ @@ -363,6 +379,10 @@ export const MODE_BUNDLES: Readonly>> = contextual_retrieval_disabled: false, // v0.42.3.0 — autocut ON (reranker fires; cliff signal is trustworthy). autocut: true, + // v0.43 — relational recall ON (contingent on the no-regression gate; + // ships default-false everywhere if the gate flags any regression). + relationalRetrieval: true, + relational_retrieval_depth: 2, autocut_jump: 0.2, }), tokenmax: Object.freeze({ @@ -411,6 +431,9 @@ export const MODE_BUNDLES: Readonly>> = contextual_retrieval_disabled: false, // v0.42.3.0 — autocut ON. autocut: true, + // v0.43 — relational recall ON for tokenmax (max-recall tier). + relationalRetrieval: true, + relational_retrieval_depth: 2, autocut_jump: 0.2, }), }); @@ -462,6 +485,9 @@ export interface SearchKeyOverrides { contextual_retrieval_disabled?: boolean; // v0.42.3.0 — autocut overrides. autocut?: boolean; + // v0.43 — relational recall overrides. + relationalRetrieval?: boolean; + relational_retrieval_depth?: number; autocut_jump?: number; } @@ -509,6 +535,9 @@ export interface SearchPerCallOpts { // numeric per-call knob threaded through the bundle. autocut?: boolean; autocut_jump?: number; + // v0.43 — relational recall per-call overrides. + relationalRetrieval?: boolean; + relational_retrieval_depth?: number; } /** @@ -601,6 +630,9 @@ export function resolveSearchMode(input: ResolveSearchModeInput): ResolvedSearch // v0.42.3.0 — autocut resolved via the same pick chain. autocut: pick('autocut'), autocut_jump: pick('autocut_jump'), + // v0.43 — relational recall resolved via the same pick chain. + relationalRetrieval: pick('relationalRetrieval'), + relational_retrieval_depth: pick('relational_retrieval_depth'), resolved_mode, mode_valid: valid, }; @@ -706,7 +738,7 @@ export function attributeKnob( // to take effect immediately (one-time global cache cold-miss on upgrade; refills // within cache.ttl_seconds). Same cache-key-contamination convention as the // autocut / title_boost / graph_signals bumps above. -export const KNOBS_HASH_VERSION = 9; +export const KNOBS_HASH_VERSION = 10; /** * v0.36 (D8 / CDX-2) — second-arg context for the cache key. The @@ -814,6 +846,14 @@ export function knobsHash( // etc.) so a partial-knobs caller (tests passing a minimal literal) can't // crash the hash. Typed callers always carry the field. `acj=${(knobs.autocut_jump ?? 0.2).toFixed(2)}`, + // v=10 additions (v0.43, append-only): relational recall arm. A + // relational-on write (edge-seeded result set) must NOT be served to a + // relational-off lookup — same contamination class as graph_signals. The + // depth changes the candidate set too, so it folds in as well. ONE-TIME + // cold-miss on upgrade as v=9 rows become unreachable; pinned by + // test/model-pricing.test.ts-style drift guards and the mode tests. + `rel=${knobs.relationalRetrieval ? 1 : 0}`, + `reld=${knobs.relational_retrieval_depth ?? 2}`, ]; const h = createHash('sha256'); h.update(parts.join('|')); @@ -981,6 +1021,17 @@ export function loadOverridesFromConfig( if (Number.isFinite(n) && n > 0 && n <= 1) out.autocut_jump = n; } + // v0.43 — relational recall arm. + const rel = get('search.relational_retrieval'); + if (rel !== undefined) { + out.relationalRetrieval = rel === '1' || rel.toLowerCase() === 'true'; + } + const reld = get('search.relational_retrieval_depth'); + if (reld !== undefined) { + const n = parseInt(reld, 10); + if (Number.isFinite(n) && n >= 1 && n <= 3) out.relational_retrieval_depth = n; + } + return out; } @@ -1019,6 +1070,9 @@ export const SEARCH_MODE_CONFIG_KEYS: ReadonlyArray = Object.freeze([ 'search.contextual_retrieval_disabled', // v0.42.3.0 autocut 'search.autocut', + // v0.43 relational recall + 'search.relational_retrieval', + 'search.relational_retrieval_depth', 'search.autocut_jump', ]); diff --git a/src/core/search/relational-intent.ts b/src/core/search/relational-intent.ts new file mode 100644 index 000000000..cccf820de --- /dev/null +++ b/src/core/search/relational-intent.ts @@ -0,0 +1,243 @@ +/** + * Relational-query parser (typed-edge retrieval, v0.43). + * + * Detects queries whose answer is a RELATIONSHIP (an edge between entities) + * rather than a passage — "who invested in widget-co", "who at acme works on + * payments", "who introduced me to alice", "what connects fund-a and fund-b". + * The relational recall arm uses the parse to resolve seed entities and walk + * the typed-edge graph. + * + * Pure module. No DB, no LLM, no async. Detection is regex-only and + * deterministic, parsed from the ORIGINAL query (never the LLM-expanded + * variant) so the recall arm stays bit-for-bit reproducible. + * + * Precision-first (D4): this returns a CANDIDATE. The arm fires only when a + * resolvable seed entity is also found (seed resolution lives in + * relational-recall.ts). Patterns require the relation phrase and the entity + * to be adjacent, so "who invested TIME in learning Rust" does not match the + * "who invested in " pattern. + * + * Vocabulary (D2): the default bank covers the common archetypes; a schema + * pack can extend it with `extraVerbs`. Every emitted link_type is validated + * against KNOWN_LINK_TYPES so the query side can't drift from what ingest + * actually produces (see link-extraction.ts:inferLinkType). intro/connects + * traverse type-agnostically (linkTypes = null) because gbrain has no + * `introduced`/`knows` edge — any edge touching the seed is the signal. + * + * ReDoS: seed captures are length-bounded (`.{1,80}?`) and every pattern is + * anchored, so there is no catastrophic-backtracking surface. + * + * Tested in test/relational-intent.test.ts. + */ + +export type RelationalKind = 'who_rel' | 'who_at' | 'connects' | 'intro'; +export type RelationDirection = 'in' | 'out' | 'both'; + +export interface RelationalQuery { + /** Which archetype matched. */ + kind: RelationalKind; + /** Raw entity phrases to resolve, in query order. 1 for most, 2 for connects. */ + seeds: string[]; + /** Typed edges to traverse, or null for type-agnostic traversal. */ + linkTypes: string[] | null; + /** Traversal direction from the seed. */ + direction: RelationDirection; + /** The matched relation phrase, for telemetry / --explain. */ + relationPhrase: string; +} + +/** Schema-pack vocab extension (D2=B). */ +export interface RelationVerbSpec { + /** A regex-source alternation of phrasings, e.g. `acquired|bought`. */ + verb: string; + /** Edges this verb maps to. MUST be a subset of KNOWN_LINK_TYPES. */ + linkTypes: string[]; + /** Direction from the seed entity named after the verb. */ + direction: RelationDirection; +} + +export interface RelationVocab { + extraVerbs?: RelationVerbSpec[]; +} + +/** + * Link types ingest can actually produce (link-extraction.ts + frontmatter + * map + schema packs). The query parser may only emit a SUBSET of these, so a + * relation phrase can never traverse an edge type that ingest never writes. + * `validateVocab` enforces this for pack-supplied verbs. + */ +export const KNOWN_LINK_TYPES: ReadonlySet = new Set([ + 'founded', + 'invested_in', + 'advises', + 'works_at', + 'attended', + 'yc_partner', + 'led_round', + 'mentions', + 'image_of', + 'discussed_in', + 'source', + 'related_to', + 'wikilink_basename', +]); + +// Seeds that are pronouns / generic nouns, not entities. If a pattern's seed +// cleans down to one of these, the parse is rejected (precision-first). +const STOPWORD_SEEDS: ReadonlySet = new Set([ + 'it', 'that', 'this', 'them', 'these', 'those', 'here', 'there', + 'everyone', 'anyone', 'someone', 'anybody', 'somebody', 'people', + 'things', 'us', 'me', 'him', 'her', 'you', 'who', 'what', 'which', +]); + +interface CompiledPattern { + re: RegExp; + kind: RelationalKind; + linkTypes: string[] | null; + direction: RelationDirection; + /** Number of seed capture groups (1, or 2 for connects). */ + seedGroups: 1 | 2; +} + +// Bounded seed capture: 1–80 chars, lazy, so the trailing anchor decides the +// boundary without catastrophic backtracking. +const SEED = '(.{1,80}?)'; + +// ── who_rel verb bank: "who " → traverse INTO the seed ── +// Each entry is explicit (linkTypes inline) so there is no second lookup. +const WHO_REL_VERBS: Array<{ verb: string; linkTypes: string[]; direction: RelationDirection }> = [ + { verb: 'invested in|invests in|funded|backed|backs|led the round in|led the seed in|led the series [a-z] in', linkTypes: ['invested_in', 'led_round'], direction: 'in' }, + { verb: 'founded|co-?founded|started', linkTypes: ['founded'], direction: 'in' }, + { verb: 'advises|advised', linkTypes: ['advises'], direction: 'in' }, + { verb: 'works at|worked at|works for', linkTypes: ['works_at'], direction: 'in' }, + { verb: 'attended', linkTypes: ['attended'], direction: 'in' }, +]; + +function buildPatterns(vocab?: RelationVocab): CompiledPattern[] { + const patterns: CompiledPattern[] = []; + + // connects — two seeds, type-agnostic. Most specific, checked first. + patterns.push({ + re: new RegExp( + `\\b(?:what|which)\\s+(?:companies?|people|things|entities|deals?)?\\s*(?:connects?|links?|ties? together|is (?:the )?(?:connection|link|relationship) between)\\s+${SEED}\\s+(?:and|&)\\s+${SEED}\\s*\\??$`, + 'i', + ), + kind: 'connects', linkTypes: null, direction: 'both', seedGroups: 2, + }); + patterns.push({ + re: new RegExp( + `\\bhow\\s+(?:are|is|do|does)\\s+${SEED}\\s+(?:and|&)\\s+${SEED}\\s+(?:connected|related|linked|associated)\\b`, + 'i', + ), + kind: 'connects', linkTypes: null, direction: 'both', seedGroups: 2, + }); + + // intro — type-agnostic walk around the named person (no `introduced` edge). + patterns.push({ + re: new RegExp( + `\\bwho\\s+(?:introduced|connected|referred)\\s+(?:me|us|him|her|them)\\s+to\\s+${SEED}\\s*\\??$`, + 'i', + ), + kind: 'intro', linkTypes: null, direction: 'both', seedGroups: 1, + }); + + // who_at — entity in the middle: "who at acme works on payments". + patterns.push({ + re: new RegExp( + `\\bwho\\s+(?:at|from|in)\\s+${SEED}\\s+(?:works? on|works?|leads?|runs?|builds?|owns?|handles?|manages?)\\b`, + 'i', + ), + kind: 'who_at', linkTypes: ['works_at'], direction: 'in', seedGroups: 1, + }); + + // who_rel — "who ". + for (const v of WHO_REL_VERBS) { + patterns.push({ + re: new RegExp(`\\bwho\\s+(?:${v.verb})\\s+${SEED}\\s*\\??$`, 'i'), + kind: 'who_rel', linkTypes: v.linkTypes, direction: v.direction, seedGroups: 1, + }); + } + + // outgoing variants — "what did invest in", "where does work". + patterns.push({ + re: new RegExp( + `\\bwhat\\s+(?:companies?|startups?|deals?)?\\s*(?:has|have|did|does)?\\s*${SEED}\\s+(?:invest(?:ed)? in)\\b`, + 'i', + ), + kind: 'who_rel', linkTypes: ['invested_in', 'led_round'], direction: 'out', seedGroups: 1, + }); + patterns.push({ + re: new RegExp(`\\bwhere\\s+(?:does|did|has)\\s+${SEED}\\s+work\\b`, 'i'), + kind: 'who_rel', linkTypes: ['works_at'], direction: 'out', seedGroups: 1, + }); + + // schema-pack extensions: "who " for each extra verb. + for (const v of vocab?.extraVerbs ?? []) { + patterns.push({ + re: new RegExp(`\\bwho\\s+(?:${v.verb})\\s+${SEED}\\s*\\??$`, 'i'), + kind: 'who_rel', linkTypes: v.linkTypes, direction: v.direction, seedGroups: 1, + }); + } + + return patterns; +} + +/** Trim, drop a leading article and surrounding quotes, strip trailing `?`. */ +function cleanSeed(raw: string): string { + return raw + .trim() + .replace(/\?+$/, '') + .replace(/^["'`]|["'`]$/g, '') + .replace(/^(?:the|a|an)\s+/i, '') + .trim(); +} + +function validSeed(s: string): boolean { + if (s.length === 0 || s.length > 80) return false; + if (STOPWORD_SEEDS.has(s.toLowerCase())) return false; + return true; +} + +/** + * Validate that every link_type a vocab emits is one ingest can produce. + * Throws on an unknown type so a misconfigured schema pack fails loudly at + * load time rather than silently traversing an edge that never exists. + */ +export function validateVocab(vocab: RelationVocab): void { + for (const v of vocab.extraVerbs ?? []) { + for (const lt of v.linkTypes) { + if (!KNOWN_LINK_TYPES.has(lt)) { + throw new Error( + `relational vocab: unknown link_type "${lt}" for verb /${v.verb}/ — must be one of ${[...KNOWN_LINK_TYPES].join(', ')}`, + ); + } + } + } +} + +/** + * Parse a query into a RelationalQuery, or null if it isn't relational. + * First matching pattern wins (patterns are ordered specific → general). + */ +export function parseRelationalQuery(query: string, vocab?: RelationVocab): RelationalQuery | null { + if (!query || query.length > 512) return null; // bound work; real queries are short + const patterns = buildPatterns(vocab); + + for (const p of patterns) { + const m = p.re.exec(query); + if (!m) continue; + + if (p.seedGroups === 2) { + const a = cleanSeed(m[1] ?? ''); + const b = cleanSeed(m[2] ?? ''); + if (!validSeed(a) || !validSeed(b)) continue; + return { kind: p.kind, seeds: [a, b], linkTypes: p.linkTypes, direction: p.direction, relationPhrase: m[0].trim() }; + } + + const seed = cleanSeed(m[1] ?? ''); + if (!validSeed(seed)) continue; + return { kind: p.kind, seeds: [seed], linkTypes: p.linkTypes, direction: p.direction, relationPhrase: m[0].trim() }; + } + + return null; +} diff --git a/src/core/search/relational-recall.ts b/src/core/search/relational-recall.ts new file mode 100644 index 000000000..7a8850ba0 --- /dev/null +++ b/src/core/search/relational-recall.ts @@ -0,0 +1,232 @@ +/** + * Relational recall arm (typed-edge retrieval, v0.43). + * + * Turns a relational query into a ranked list of edge-derived candidates that + * hybridSearch injects as a FOURTH RRF arm (alongside keyword + vector), so a + * relationship answer competes for ranking instead of relying on lexical or + * vector similarity to surface it. + * + * Flow: parseRelationalQuery → resolve seed entity(ies) (scope-aware, + * confidence-gated) → engine.relationalFanout (within-source, deterministic) + * → batch-hydrate to SearchResult rows (reinforcing each page's REAL canonical + * chunk; page-level key for chunkless entity pages). + * + * Determinism: parses the ORIGINAL query (never an LLM-expanded variant); + * traversal + resolution are deterministic. Fail-open: any error returns an + * empty arm + an audit row, never breaking the search hot path. + * + * Federation (E2=A): resolves the seed in every in-scope source and fans out + * from each; traversal stays WITHIN each source (no cross-boundary edges in + * v1). Confidence gate (D3): a seed that only `fallback_slugify`-resolves is + * dropped, so the arm never traverses from an invented slug. The tier-2 + * resolution-margin gate is a filed TODO. + * + * Tested in test/relational-recall.test.ts. + */ + +import type { BrainEngine } from '../engine.ts'; +import type { SearchResult, PageType, RelationalFanoutRow } from '../types.ts'; +import { createAuditWriter } from '../audit/audit-writer.ts'; +import { resolveEntitySlugWithSource } from '../entities/resolve.ts'; +import { parseRelationalQuery, type RelationalQuery, type RelationVocab } from './relational-intent.ts'; + +export interface RelationalArmOpts { + sourceId?: string; + sourceIds?: string[]; + depth?: number; + limit?: number; + vocab?: RelationVocab; + onMeta?: (meta: RelationalArmMeta) => void; +} + +export interface RelationalArmMeta { + fired: boolean; + kind: RelationalQuery['kind'] | null; + seeds_resolved: number; + candidates: number; + errored: boolean; + duration_ms: number; +} + +interface RelationalFailureEvent { + ts: string; + error_summary: string; + query_kind: string; +} + +const failureWriter = createAuditWriter({ + featureName: 'relational-recall-failures', + errorLabel: 'gbrain', + errorTrailer: '; search continues', +}); + +/** Recent relational-arm fail-open events — consumed by doctor + search stats. */ +export function readRecentRelationalFailures(days = 7, now: Date = new Date()): RelationalFailureEvent[] { + return failureWriter.readRecent(days, now); +} + +function truncate(msg: string, max = 200): string { + return msg.length <= max ? msg : msg.slice(0, max - 1) + '…'; +} + +/** Sources to resolve a seed against. Federated → the set; scalar → [id]; + * unscoped/__all__ → ['default'] (single-source brains; multi-source + * enumeration under __all__ is a v1 limitation). */ +function scopeSources(opts: RelationalArmOpts): string[] { + if (opts.sourceIds && opts.sourceIds.length > 0) return opts.sourceIds; + if (opts.sourceId && opts.sourceId !== '__all__') return [opts.sourceId]; + return ['default']; +} + +/** Resolve a seed phrase to all in-scope (source_id, slug) pairs that + * resolve to a REAL page (confidence gate D3 tier-1: drop fallback_slugify). */ +async function resolveSeedScoped( + engine: BrainEngine, + sources: string[], + phrase: string, +): Promise> { + const out: Array<{ source_id: string; slug: string }> = []; + const seen = new Set(); + for (const sid of sources) { + const r = await resolveEntitySlugWithSource(engine, sid, phrase); + if (!r || r.source === 'fallback_slugify') continue; + const key = `${sid}:${r.slug}`; + if (seen.has(key)) continue; + seen.add(key); + out.push({ source_id: sid, slug: r.slug }); + } + return out; +} + +/** Batch-hydrate fanout rows into SearchResult rows in fanout (ranked) order. */ +async function hydrate( + engine: BrainEngine, + rows: RelationalFanoutRow[], + seedSlug: string, +): Promise { + if (rows.length === 0) return []; + const slugs = Array.from(new Set(rows.map(r => r.slug))); + const pageRows = await engine.executeRaw<{ + page_id: number; slug: string; source_id: string; title: string; type: string; synopsis: string | null; + }>( + `SELECT p.id AS page_id, p.slug, p.source_id, p.title, p.type, + LEFT(p.compiled_truth, 240) AS synopsis + FROM pages p + WHERE p.slug = ANY($1::text[]) AND p.deleted_at IS NULL`, + [slugs], + ); + const byKey = new Map(); + for (const pr of pageRows) byKey.set(`${pr.source_id}:${pr.slug}`, pr); + + const out: SearchResult[] = []; + for (const r of rows) { + const pr = byKey.get(`${r.source_id}:${r.slug}`); + if (!pr) continue; + out.push({ + slug: r.slug, + page_id: pr.page_id, + title: pr.title, + type: pr.type as PageType, + chunk_text: pr.synopsis ?? r.slug, + chunk_source: 'compiled_truth', + // E1: reinforce the page's REAL canonical chunk; F3: chunkless entity + // pages key page-level (chunk_id 0 → rrfKey `source:slug:0`, stable and + // collision-safe; SERIAL chunk ids start at 1 so 0 never aliases a real chunk). + chunk_id: r.canonical_chunk_id ?? 0, + chunk_index: 0, + score: 0, // rank-based: RRF derives score from list position + stale: false, + source_id: r.source_id, + relational_via_link_types: r.via_link_types, + relational_seed: seedSlug, + relational_hop: r.hop, + relational_path: r.path, + }); + } + return out; +} + +/** + * Build the relational recall arm. Returns an empty list (pure no-op) when the + * query isn't relational or no seed resolves. Never throws. + */ +export async function buildRelationalArm( + engine: BrainEngine, + query: string, + opts: RelationalArmOpts = {}, +): Promise { + const startedAt = Date.now(); + const meta: RelationalArmMeta = { + fired: false, kind: null, seeds_resolved: 0, candidates: 0, errored: false, duration_ms: 0, + }; + const finish = (list: SearchResult[]) => { + meta.candidates = list.length; + meta.duration_ms = Date.now() - startedAt; + opts.onMeta?.(meta); + return list; + }; + + const parsed = parseRelationalQuery(query, opts.vocab); + if (!parsed) return finish([]); + meta.kind = parsed.kind; + + try { + const sources = scopeSources(opts); + const fanoutOpts = { + linkTypes: parsed.linkTypes, + direction: parsed.direction, + depth: opts.depth, + limit: opts.limit, + }; + + if (parsed.kind === 'connects' && parsed.seeds.length === 2) { + // Resolve both endpoints; both must resolve or the arm no-ops. + const resA = await resolveSeedScoped(engine, sources, parsed.seeds[0]); + const resB = await resolveSeedScoped(engine, sources, parsed.seeds[1]); + if (resA.length === 0 || resB.length === 0) return finish([]); + meta.seeds_resolved = resA.length + resB.length; + + const perSource = (rs: typeof resA) => ({ + sourceId: rs.length === 1 ? rs[0].source_id : undefined, + sourceIds: rs.length > 1 ? Array.from(new Set(rs.map(x => x.source_id))) : undefined, + slugs: Array.from(new Set(rs.map(x => x.slug))), + }); + const a = perSource(resA); + const b = perSource(resB); + const fanA = await engine.relationalFanout(a.slugs, { ...fanoutOpts, sourceId: a.sourceId, sourceIds: a.sourceIds }); + const fanB = await engine.relationalFanout(b.slugs, { ...fanoutOpts, sourceId: b.sourceId, sourceIds: b.sourceIds }); + // Shared midpoints: nodes reachable from BOTH endpoints (exclude the + // endpoints themselves). Ordered by combined hop. + const bByKey = new Map(fanB.map(r => [`${r.source_id}:${r.slug}`, r] as const)); + const endpointSlugs = new Set([...a.slugs, ...b.slugs]); + const shared = fanA + .filter(r => bByKey.has(`${r.source_id}:${r.slug}`) && !endpointSlugs.has(r.slug)) + .map(r => ({ row: r, combined: r.hop + bByKey.get(`${r.source_id}:${r.slug}`)!.hop })) + .sort((x, y) => x.combined - y.combined || x.row.slug.localeCompare(y.row.slug)) + .map(x => x.row); + const list = await hydrate(engine, shared, parsed.seeds.join(' ↔ ')); + meta.fired = list.length > 0; + return finish(list); + } + + // who_rel / who_at / intro: single logical seed (may resolve in N sources). + const resolved = await resolveSeedScoped(engine, sources, parsed.seeds[0]); + if (resolved.length === 0) return finish([]); + meta.seeds_resolved = resolved.length; + const slugs = Array.from(new Set(resolved.map(r => r.slug))); + const srcIds = Array.from(new Set(resolved.map(r => r.source_id))); + const rows = await engine.relationalFanout(slugs, { + ...fanoutOpts, + sourceId: srcIds.length === 1 ? srcIds[0] : undefined, + sourceIds: srcIds.length > 1 ? srcIds : undefined, + }); + const list = await hydrate(engine, rows, resolved[0].slug); + meta.fired = list.length > 0; + return finish(list); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + failureWriter.log({ error_summary: truncate(msg), query_kind: parsed.kind }); + meta.errored = true; + return finish([]); + } +} diff --git a/src/core/types.ts b/src/core/types.ts index 2933c42a5..16488b5ff 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -719,6 +719,20 @@ export interface SearchResult { graph_session_demoted?: boolean; /** Slug prefix used for the session-diversification grouping. */ graph_session_prefix?: string; + /** + * v0.43 relational recall arm — set when this result was surfaced by + * typed-edge traversal (not lexical/vector). Drives `gbrain search + * --explain` attribution ("surfaced via invested_in edge from widget-co"). + * Absent for organic keyword/vector results. + */ + /** Edge types the result was reached by (e.g. ['invested_in']). */ + relational_via_link_types?: string[]; + /** The resolved seed entity slug the traversal started from. */ + relational_seed?: string; + /** Shortest hop distance from the seed (1 = direct neighbor). */ + relational_hop?: number; + /** Shortest connecting slug path seed→…→result (for "how I know this"). */ + relational_path?: string[]; /** * v0.40.4 full attribution (D12=A) — per-stage score deltas for the * `gbrain search --explain` formatter. Every boost stage stamps its @@ -1089,6 +1103,14 @@ export interface SearchOpts { * would resolve to the same mode default and the gate would be a no-op. */ graph_signals?: boolean; + /** + * v0.43 — relational recall arm per-call override. Per-call wins over the + * `search.relational_retrieval` config key wins over the mode bundle. Eval + * A/B gates need explicit per-call control or both branches resolve to the + * same mode default. `relationalRetrievalDepth` caps traversal hops (1..3). + */ + relationalRetrieval?: boolean; + relationalRetrievalDepth?: number; } /** @@ -1177,6 +1199,43 @@ export interface GraphPath { depth: number; } +/** + * One reached node from a typed-edge relational fan-out (v0.43). The recall + * arm hydrates these into SearchResult rows and injects them as a fourth RRF + * arm. Aggregated to the page level: `hop` is the shortest distance from any + * seed, `edge_count` is a connection-richness proxy (distinct edge types via + * which the node is reached), `via_link_types` names them, `path` is the + * shortest connecting slug chain (for --explain), and `canonical_chunk_id` is + * the page's lowest-ordinal chunk (null for frontmatter-only entity pages). + */ +export interface RelationalFanoutRow { + source_id: string; + slug: string; + hop: number; + edge_count: number; + via_link_types: string[]; + path: string[]; + canonical_chunk_id: number | null; +} + +/** Options for BrainEngine.relationalFanout. */ +export interface RelationalFanoutOpts { + /** Edge types to traverse; null/empty = type-agnostic. */ + linkTypes?: string[] | null; + /** Direction from each seed. Default 'both'. */ + direction?: 'in' | 'out' | 'both'; + /** Max hops. Default 2, hard-capped at 3. */ + depth?: number; + /** Include `link_source='mentions'` edges. Default false (typed edges only). */ + includeMentions?: boolean; + /** Single-source scope. */ + sourceId?: string; + /** Federated scope; traversal stays WITHIN each seed's own source. */ + sourceIds?: string[]; + /** Hard cap on returned candidate nodes. Default 50. */ + limit?: number; +} + // Timeline export interface TimelineEntry { id: number; diff --git a/src/eval/retrieval-quality/harness.ts b/src/eval/retrieval-quality/harness.ts index e39b81452..780bfb1f8 100644 --- a/src/eval/retrieval-quality/harness.ts +++ b/src/eval/retrieval-quality/harness.ts @@ -33,6 +33,15 @@ export interface NamedThingQuestion { /** slugs that must NOT appear in top-k (hard-negative family). */ forbidden?: string[]; notes?: string; + /** + * v0.43 relational metadata (graph-relationship family). Optional, typed + * (not buried in `notes`) so the relational parser/path/determinism tests + * can machine-check the parse. `seed` is the entity the answer relates to; + * `linkTypes` the edge(s) expected; `kind` the archetype. + */ + seed?: string; + linkTypes?: string[]; + kind?: 'who_rel' | 'who_at' | 'connects' | 'intro'; } /** Ranked slugs for a query, best-first. */ @@ -46,6 +55,11 @@ export interface QuestionResult { reciprocal_rank: number; // 0 if no relevant slug in results /** hard-negative only: true when NO forbidden slug appeared in top-3. */ negative_clean?: boolean; + /** v0.43 — fraction of relevant slugs found in top-K (K=3). 0 for hard-negative. */ + recall_at_k: number; + /** v0.43 — fraction of relevant slugs found in top-10. The relational + * headline metric (a relationship query often has several valid answers). */ + recall_at_10: number; } export interface FamilyReport { @@ -54,6 +68,9 @@ export interface FamilyReport { hit_at_1: number; // rate hit_at_3: number; // rate mrr: number; + /** v0.43 — mean recall@K / recall@10 across the family's questions. */ + recall_at_k: number; + recall_at_10: number; } export interface RetrievalQualityReport { @@ -71,14 +88,20 @@ export function scoreQuestion(q: NamedThingQuestion, ranked: string[]): Question const forbidden = new Set(q.forbidden ?? []); const topK = ranked.slice(0, K); const clean = !topK.some(s => forbidden.has(s)); - return { family: q.family, query: q.query, hit_at_1: clean, hit_at_3: clean, reciprocal_rank: clean ? 1 : 0, negative_clean: clean }; + return { family: q.family, query: q.query, hit_at_1: clean, hit_at_3: clean, reciprocal_rank: clean ? 1 : 0, negative_clean: clean, recall_at_k: 0, recall_at_10: 0 }; } const relevant = new Set(q.relevant ?? []); const firstRelevantIdx = ranked.findIndex(s => relevant.has(s)); const hit1 = firstRelevantIdx === 0; const hit3 = firstRelevantIdx >= 0 && firstRelevantIdx < K; const rr = firstRelevantIdx >= 0 ? 1 / (firstRelevantIdx + 1) : 0; - return { family: q.family, query: q.query, hit_at_1: hit1, hit_at_3: hit3, reciprocal_rank: rr }; + const recallAt = (k: number): number => { + if (relevant.size === 0) return 0; + const top = ranked.slice(0, k); + const found = top.filter(s => relevant.has(s)).length; + return found / relevant.size; + }; + return { family: q.family, query: q.query, hit_at_1: hit1, hit_at_3: hit3, reciprocal_rank: rr, recall_at_k: recallAt(K), recall_at_10: recallAt(10) }; } export async function runRetrievalQuality( @@ -106,6 +129,8 @@ export async function runRetrievalQuality( hit_at_1: n ? list.filter(r => r.hit_at_1).length / n : 0, hit_at_3: n ? list.filter(r => r.hit_at_3).length / n : 0, mrr: n ? list.reduce((s, r) => s + r.reciprocal_rank, 0) / n : 0, + recall_at_k: n ? list.reduce((s, r) => s + r.recall_at_k, 0) / n : 0, + recall_at_10: n ? list.reduce((s, r) => s + r.recall_at_10, 0) / n : 0, }); } families.sort((a, b) => a.family.localeCompare(b.family)); diff --git a/test/cache-scope-key.test.ts b/test/cache-scope-key.test.ts new file mode 100644 index 000000000..dac58c239 --- /dev/null +++ b/test/cache-scope-key.test.ts @@ -0,0 +1,42 @@ +/** + * Query-cache scope key (federation hardening). + * + * A federated search reads a different graph than a single-source one, so + * the semantic cache must key them apart. `cacheScopeKey` produces an + * order-independent key for federated scopes and leaves single-source + * brains on their existing key (scalar id or 'default'), so single-source + * cache hit-rate is unchanged. + */ + +import { describe, test, expect } from 'bun:test'; +import { cacheScopeKey } from '../src/core/search/hybrid.ts'; + +describe('cacheScopeKey', () => { + test('unscoped → default (single-source unchanged)', () => { + expect(cacheScopeKey(undefined)).toBe('default'); + expect(cacheScopeKey({})).toBe('default'); + }); + + test('scalar sourceId → itself (single-source unchanged)', () => { + expect(cacheScopeKey({ sourceId: 'host' })).toBe('host'); + }); + + test('federated sourceIds → order-independent set key', () => { + const k1 = cacheScopeKey({ sourceIds: ['team-b', 'team-a', 'host'] }); + const k2 = cacheScopeKey({ sourceIds: ['host', 'team-a', 'team-b'] }); + expect(k1).toBe(k2); // order does not matter + expect(k1).toBe('__set__:host,team-a,team-b'); + }); + + test('different source-sets do NOT share a key', () => { + const a = cacheScopeKey({ sourceIds: ['host', 'team-a'] }); + const b = cacheScopeKey({ sourceIds: ['host', 'team-b'] }); + expect(a).not.toBe(b); + }); + + test('federated set key is distinct from any single scalar key', () => { + const set = cacheScopeKey({ sourceIds: ['host'] }); + const scalar = cacheScopeKey({ sourceId: 'host' }); + expect(set).not.toBe(scalar); // a 1-element set still cannot serve a scalar read + }); +}); diff --git a/test/cross-modal-phase1.test.ts b/test/cross-modal-phase1.test.ts index e699c8789..eaa245540 100644 --- a/test/cross-modal-phase1.test.ts +++ b/test/cross-modal-phase1.test.ts @@ -136,14 +136,15 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { return resolveSearchMode({ mode: 'balanced' }); } - test('KNOBS_HASH_VERSION is 9 (cross-modal still appended; 8→9 archive-demote #1777)', () => { + test('KNOBS_HASH_VERSION is 10 (cross-modal still appended; 9→10 relational recall)', () => { // v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3 // with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) + // v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields. // v0.40.3.0 D8 bumps to v=5 (sequenced behind salem's v=4 graph-signals). // v0.41.22.0 (type-unification): 5→6 for alias_resolved post-fusion boost. // T2: 6→7 title_boost. v0.42.3.0: 7→8 autocut. issue #1777: 8→9 archive/ demote. - expect(KNOBS_HASH_VERSION).toBe(9); + // v0.43: 9→10 relational recall arm. + expect(KNOBS_HASH_VERSION).toBe(10); }); test('flipping unified_multimodal changes the hash', () => { diff --git a/test/e2e/engine-parity.test.ts b/test/e2e/engine-parity.test.ts index cce3e3ecc..d03c765c2 100644 --- a/test/e2e/engine-parity.test.ts +++ b/test/e2e/engine-parity.test.ts @@ -544,3 +544,71 @@ describeBoth('Engine parity — Postgres vs PGLite', () => { expect(pg.find((r) => r.slug === 'ep/ec-alice')!.inbound_count).toBe(2); }); }); + +// ── relationalFanout parity (v0.43) ───────────────────────────────────── +async function seedRelational(eng: BrainEngine) { + const pages: Array<[string, 'company' | 'person']> = [ + ['companies/ep-widget', 'company'], + ['companies/ep-other', 'company'], + ['people/ep-inv-a', 'person'], + ['people/ep-inv-b', 'person'], + ['people/ep-emp-c', 'person'], + ['people/ep-mentioner', 'person'], + ]; + for (const [slug, type] of pages) { + await eng.putPage(slug, { type, title: slug, compiled_truth: `${slug} body`, timeline: '' }); + } + await eng.upsertChunks('people/ep-inv-b', [{ + chunk_index: 0, chunk_text: 'b', chunk_source: 'compiled_truth', + embedding: basisEmbedding(2), token_count: 1, + }] satisfies ChunkInput[]); + await eng.addLink('people/ep-inv-a', 'companies/ep-widget', '', 'invested_in', 'manual'); + await eng.addLink('people/ep-inv-b', 'companies/ep-widget', '', 'invested_in', 'manual'); + await eng.addLink('people/ep-emp-c', 'companies/ep-widget', '', 'works_at', 'manual'); + await eng.addLink('people/ep-mentioner', 'companies/ep-widget', '', 'mentions', 'mentions'); + await eng.addLink('people/ep-inv-a', 'companies/ep-other', '', 'invested_in', 'manual'); +} + +describeBoth('Engine parity — relationalFanout', () => { + let pgEngine: BrainEngine; + let pgliteEngine: PGLiteEngine; + + beforeAll(async () => { + pgEngine = await setupDB(); + await seedRelational(pgEngine); + pgliteEngine = new PGLiteEngine(); + await pgliteEngine.connect({}); + await pgliteEngine.initSchema(); + await seedRelational(pgliteEngine); + }, 90_000); + + afterAll(async () => { + await pgliteEngine.disconnect(); + await teardownDB(); + }, 30_000); + + const shape = (rows: Awaited>) => + rows.map(r => `${r.source_id}:${r.slug}:${r.hop}:${r.edge_count}:${r.via_link_types.join(',')}:${r.path.join('>')}:${r.canonical_chunk_id ?? 'null'}`); + + test('typed-edge fan-out is identical across engines', async () => { + const opts = { direction: 'in' as const, linkTypes: ['invested_in'] }; + const pg = await pgEngine.relationalFanout(['companies/ep-widget'], opts); + const pglite = await pgliteEngine.relationalFanout(['companies/ep-widget'], opts); + expect(shape(pg)).toEqual(shape(pglite)); + expect(pg.map(r => r.slug).sort()).toEqual(['people/ep-inv-a', 'people/ep-inv-b']); + }); + + test('type-agnostic + mentions-exclusion identical across engines', async () => { + const pg = await pgEngine.relationalFanout(['companies/ep-widget'], { direction: 'in' }); + const pglite = await pgliteEngine.relationalFanout(['companies/ep-widget'], { direction: 'in' }); + expect(shape(pg)).toEqual(shape(pglite)); + expect(pg.map(r => r.slug)).not.toContain('people/ep-mentioner'); + }); + + test('connects (multi-seed, both) identical across engines', async () => { + const seeds = ['companies/ep-widget', 'companies/ep-other']; + const pg = await pgEngine.relationalFanout(seeds, { direction: 'both' }); + const pglite = await pgliteEngine.relationalFanout(seeds, { direction: 'both' }); + expect(shape(pg)).toEqual(shape(pglite)); + }); +}); diff --git a/test/fixtures/retrieval-quality/relational/corpus.ts b/test/fixtures/retrieval-quality/relational/corpus.ts new file mode 100644 index 000000000..df9388aed --- /dev/null +++ b/test/fixtures/retrieval-quality/relational/corpus.ts @@ -0,0 +1,195 @@ +/** + * Relational benchmark corpus (v0.43). + * + * A small entity graph whose relational answers are LEXICALLY UNRECOVERABLE: + * every page body is generic and NEVER names the entity it relates to. Only + * the typed edge connects them, so keyword + vector retrieval cannot surface + * the answer — isolating the contribution of the typed-edge recall arm. + * + * Canonical source for BOTH the seed loader and the question set, so the + * corpus and the gold answers can't drift. `relational.jsonl` is generated + * from RELATIONAL_QUESTIONS (a drift test pins them equal). + */ + +import type { BrainEngine } from '../../../../src/core/engine.ts'; +import type { ChunkInput } from '../../../../src/core/types.ts'; +import type { NamedThingQuestion } from '../../../../src/eval/retrieval-quality/harness.ts'; + +// edge maps: target entity → entities related to it via that edge type. +// invested_in / led_round point INTO the company (people/funds → company). +const INVESTMENTS: Record = { + 'companies/widget-co': ['people/alice-example', 'people/bob-example', 'funds/fund-a'], + 'companies/acme-co': ['people/carol-example', 'people/alice-example', 'funds/fund-b'], + 'companies/novapay': ['people/bob-example', 'funds/fund-a'], + 'companies/mindbridge': ['people/dave-example', 'people/carol-example'], + 'companies/helio': ['people/erin-example', 'funds/fund-b'], + 'companies/quanta': ['people/frank-example', 'people/alice-example'], + 'companies/zenith': ['people/grace-example', 'people/heidi-example'], + 'companies/orbital': ['people/ivan-example', 'people/alice-example'], +}; +const EMPLOYMENT: Record = { + 'companies/widget-co': ['people/erin-example', 'people/grace-example'], + 'companies/acme-co': ['people/dave-example', 'people/frank-example'], + 'companies/novapay': ['people/grace-example'], + 'companies/helio': ['people/heidi-example'], + 'companies/zenith': ['people/ivan-example'], +}; +const FOUNDED: Record = { + 'companies/mindbridge': ['people/carol-example'], + 'companies/quanta': ['people/frank-example'], + 'companies/widget-co': ['people/ivan-example'], + 'companies/orbital': ['people/grace-example'], +}; +const ADVISES: Record = { + 'companies/novapay': ['people/frank-example'], + 'companies/helio': ['people/alice-example'], +}; + +// Generic, cross-mention-free bodies keyed by slug prefix. +function bodyFor(slug: string): string { + if (slug.startsWith('companies/')) return 'A privately held company operating in its sector. Founded some years ago; details are sparse in this note.'; + if (slug.startsWith('funds/')) return 'An early-stage venture fund. Writes first checks; portfolio is not enumerated here.'; + return 'An individual in the network. Background and current focus are not described in this note.'; +} + +function titleFor(slug: string): string { + const tail = slug.split('/')[1]; + return tail.replace(/-/g, ' ').replace(/\b\w/g, c => c.toUpperCase()); +} + +/** Every distinct slug in the graph. */ +function allSlugs(): string[] { + const set = new Set(); + for (const map of [INVESTMENTS, EMPLOYMENT, FOUNDED, ADVISES]) { + for (const [company, members] of Object.entries(map)) { + set.add(company); + for (const m of members) set.add(m); + } + } + return [...set].sort(); +} + +function pageType(slug: string): 'company' | 'person' { + // funds + people render as person-ish entity pages; companies as company. + return slug.startsWith('companies/') ? 'company' : 'person'; +} + +// Deterministic per-slug basis embedding so pages are properly indexed +// (chunked) like a real brain. The query has no embedding in CI (no API key), +// so vector search can't connect entities anyway — but the pages must carry a +// chunk to be searchable at all. Body text stays generic + cross-mention-free. +// `dim` MUST match the schema's embedding column, which tracks the configured +// gateway default (1280 ZE / 1536 OpenAI) and can shift with shard order — +// so callers probe the real width via probeEmbeddingDim rather than hardcode. +function basisEmbedding(slug: string, dim: number): Float32Array { + let h = 0; + for (let i = 0; i < slug.length; i++) h = (h * 31 + slug.charCodeAt(i)) >>> 0; + const e = new Float32Array(dim); + e[h % dim] = 1.0; + return e; +} + +/** + * Probe the actual `content_chunks.embedding` column width. pgvector stores + * the dimension in `atttypmod` directly. Tests must size fixtures to this, not + * a hardcoded 1536 — the column tracks the gateway default and a prior test in + * the same shard can leave it at 1280 (ZE). Mirrors pglite-engine.test.ts. + */ +export async function probeEmbeddingDim(engine: BrainEngine): Promise { + const db = (engine as unknown as { db: { query: (sql: string) => Promise<{ rows: Array<{ atttypmod: number }> }> } }).db; + const r = await db.query( + `SELECT atttypmod FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'`, + ); + return r.rows[0].atttypmod; +} + +/** Seed the corpus into a fresh brain. Bodies never name related entities. */ +export async function seedRelationalCorpus(engine: BrainEngine): Promise { + const dim = await probeEmbeddingDim(engine); + for (const slug of allSlugs()) { + const body = bodyFor(slug); + await engine.putPage(slug, { + type: pageType(slug), + title: titleFor(slug), + compiled_truth: body, + timeline: '', + }); + await engine.upsertChunks(slug, [{ + chunk_index: 0, + chunk_text: body, + chunk_source: 'compiled_truth', + embedding: basisEmbedding(slug, dim), + token_count: body.split(/\s+/).length, + }] satisfies ChunkInput[]); + } + const addAll = async (map: Record, linkType: string) => { + for (const [company, members] of Object.entries(map)) { + for (const m of members) { + // edge points member → company (member invested_in / works_at / founded / advises company) + await engine.addLink(m, company, '', linkType, 'manual'); + } + } + }; + await addAll(INVESTMENTS, 'invested_in'); + await addAll(EMPLOYMENT, 'works_at'); + await addAll(FOUNDED, 'founded'); + await addAll(ADVISES, 'advises'); +} + +function bareName(slug: string): string { + return slug.split('/')[1]; +} + +/** Generate the gold question set from the same maps used to seed. */ +export function buildRelationalQuestions(): NamedThingQuestion[] { + const qs: NamedThingQuestion[] = []; + + for (const [company, investors] of Object.entries(INVESTMENTS)) { + qs.push({ + family: 'graph-relationship', kind: 'who_rel', seed: company, linkTypes: ['invested_in', 'led_round'], + query: `who invested in ${bareName(company)}`, relevant: investors, + }); + } + for (const [company, employees] of Object.entries(EMPLOYMENT)) { + qs.push({ + family: 'graph-relationship', kind: 'who_rel', seed: company, linkTypes: ['works_at'], + query: `who works at ${bareName(company)}`, relevant: employees, + }); + } + for (const [company, founders] of Object.entries(FOUNDED)) { + qs.push({ + family: 'graph-relationship', kind: 'who_rel', seed: company, linkTypes: ['founded'], + query: `who founded ${bareName(company)}`, relevant: founders, + }); + } + for (const [company, advisors] of Object.entries(ADVISES)) { + qs.push({ + family: 'graph-relationship', kind: 'who_rel', seed: company, linkTypes: ['advises'], + query: `who advises ${bareName(company)}`, relevant: advisors, + }); + } + + // connects: company pairs that share at least one related entity (any edge). + const relatedTo = (company: string): Set => { + const s = new Set(); + for (const map of [INVESTMENTS, EMPLOYMENT, FOUNDED, ADVISES]) for (const m of map[company] ?? []) s.add(m); + return s; + }; + const companies = Object.keys(INVESTMENTS); + for (let i = 0; i < companies.length; i++) { + for (let j = i + 1; j < companies.length; j++) { + const a = companies[i], b = companies[j]; + const shared = [...relatedTo(a)].filter(x => relatedTo(b).has(x)); + if (shared.length === 0) continue; + qs.push({ + family: 'graph-relationship', kind: 'connects', seed: `${a} ↔ ${b}`, linkTypes: undefined, + query: `what connects ${bareName(a)} and ${bareName(b)}`, relevant: shared.sort(), + }); + } + } + + return qs; +} + +export const RELATIONAL_QUESTIONS: NamedThingQuestion[] = buildRelationalQuestions(); diff --git a/test/fixtures/retrieval-quality/relational/relational.jsonl b/test/fixtures/retrieval-quality/relational/relational.jsonl new file mode 100644 index 000000000..06a4bf28f --- /dev/null +++ b/test/fixtures/retrieval-quality/relational/relational.jsonl @@ -0,0 +1,42 @@ +// Relational benchmark — graph-relationship family. GENERATED from +// test/fixtures/retrieval-quality/relational/corpus.ts (canonical source). +// Answers are lexically unrecoverable — only the typed edge connects +// query to answer. Regenerate: bun run scripts/_gen_relational_jsonl.ts +{"family":"graph-relationship","kind":"who_rel","seed":"companies/widget-co","linkTypes":["invested_in","led_round"],"query":"who invested in widget-co","relevant":["people/alice-example","people/bob-example","funds/fund-a"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/acme-co","linkTypes":["invested_in","led_round"],"query":"who invested in acme-co","relevant":["people/carol-example","people/alice-example","funds/fund-b"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/novapay","linkTypes":["invested_in","led_round"],"query":"who invested in novapay","relevant":["people/bob-example","funds/fund-a"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/mindbridge","linkTypes":["invested_in","led_round"],"query":"who invested in mindbridge","relevant":["people/dave-example","people/carol-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/helio","linkTypes":["invested_in","led_round"],"query":"who invested in helio","relevant":["people/erin-example","funds/fund-b"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/quanta","linkTypes":["invested_in","led_round"],"query":"who invested in quanta","relevant":["people/frank-example","people/alice-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/zenith","linkTypes":["invested_in","led_round"],"query":"who invested in zenith","relevant":["people/grace-example","people/heidi-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/orbital","linkTypes":["invested_in","led_round"],"query":"who invested in orbital","relevant":["people/ivan-example","people/alice-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/widget-co","linkTypes":["works_at"],"query":"who works at widget-co","relevant":["people/erin-example","people/grace-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/acme-co","linkTypes":["works_at"],"query":"who works at acme-co","relevant":["people/dave-example","people/frank-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/novapay","linkTypes":["works_at"],"query":"who works at novapay","relevant":["people/grace-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/helio","linkTypes":["works_at"],"query":"who works at helio","relevant":["people/heidi-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/zenith","linkTypes":["works_at"],"query":"who works at zenith","relevant":["people/ivan-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/mindbridge","linkTypes":["founded"],"query":"who founded mindbridge","relevant":["people/carol-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/quanta","linkTypes":["founded"],"query":"who founded quanta","relevant":["people/frank-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/widget-co","linkTypes":["founded"],"query":"who founded widget-co","relevant":["people/ivan-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/orbital","linkTypes":["founded"],"query":"who founded orbital","relevant":["people/grace-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/novapay","linkTypes":["advises"],"query":"who advises novapay","relevant":["people/frank-example"]} +{"family":"graph-relationship","kind":"who_rel","seed":"companies/helio","linkTypes":["advises"],"query":"who advises helio","relevant":["people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/widget-co ↔ companies/acme-co","query":"what connects widget-co and acme-co","relevant":["people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/widget-co ↔ companies/novapay","query":"what connects widget-co and novapay","relevant":["funds/fund-a","people/bob-example","people/grace-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/widget-co ↔ companies/helio","query":"what connects widget-co and helio","relevant":["people/alice-example","people/erin-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/widget-co ↔ companies/quanta","query":"what connects widget-co and quanta","relevant":["people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/widget-co ↔ companies/zenith","query":"what connects widget-co and zenith","relevant":["people/grace-example","people/ivan-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/widget-co ↔ companies/orbital","query":"what connects widget-co and orbital","relevant":["people/alice-example","people/grace-example","people/ivan-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/acme-co ↔ companies/novapay","query":"what connects acme-co and novapay","relevant":["people/frank-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/acme-co ↔ companies/mindbridge","query":"what connects acme-co and mindbridge","relevant":["people/carol-example","people/dave-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/acme-co ↔ companies/helio","query":"what connects acme-co and helio","relevant":["funds/fund-b","people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/acme-co ↔ companies/quanta","query":"what connects acme-co and quanta","relevant":["people/alice-example","people/frank-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/acme-co ↔ companies/orbital","query":"what connects acme-co and orbital","relevant":["people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/novapay ↔ companies/quanta","query":"what connects novapay and quanta","relevant":["people/frank-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/novapay ↔ companies/zenith","query":"what connects novapay and zenith","relevant":["people/grace-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/novapay ↔ companies/orbital","query":"what connects novapay and orbital","relevant":["people/grace-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/helio ↔ companies/quanta","query":"what connects helio and quanta","relevant":["people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/helio ↔ companies/zenith","query":"what connects helio and zenith","relevant":["people/heidi-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/helio ↔ companies/orbital","query":"what connects helio and orbital","relevant":["people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/quanta ↔ companies/orbital","query":"what connects quanta and orbital","relevant":["people/alice-example"]} +{"family":"graph-relationship","kind":"connects","seed":"companies/zenith ↔ companies/orbital","query":"what connects zenith and orbital","relevant":["people/grace-example","people/ivan-example"]} diff --git a/test/relational-ab.test.ts b/test/relational-ab.test.ts new file mode 100644 index 000000000..b1c747f32 --- /dev/null +++ b/test/relational-ab.test.ts @@ -0,0 +1,73 @@ +/** + * Relational A/B proof + no-regression gate (PGLite, default CI). + * + * Seeds the lexically-unrecoverable relational corpus, then runs the gold + * question set twice through bare hybridSearch — relational arm OFF vs ON — + * and asserts: + * 1. recall@10 on the graph-relationship family jumps materially with the + * arm on (the answers are unreachable by keyword/vector). + * 2. a non-relational content query returns the SAME results with the arm + * on vs off (the arm is a true no-op off-target — the regression gate). + * + * This is the headline proof artifact for the feature: relational queries + * that the corpus can't surface lexically jump from ~0 to high recall. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { hybridSearch } from '../src/core/search/hybrid.ts'; +import { runRetrievalQuality, parseQuestionsJsonl, type SearchFn } from '../src/eval/retrieval-quality/harness.ts'; +import { seedRelationalCorpus, RELATIONAL_QUESTIONS } from './fixtures/retrieval-quality/relational/corpus.ts'; +import { readFileSync } from 'fs'; +import { join } from 'path'; + +let eng: PGLiteEngine; + +beforeAll(async () => { + eng = new PGLiteEngine(); + await eng.connect({}); + await eng.initSchema(); + await seedRelationalCorpus(eng); +}, 60_000); + +afterAll(async () => { await eng.disconnect(); }); + +const searchFnWith = (relationalRetrieval: boolean): SearchFn => async (q) => { + const results = await hybridSearch(eng, q, { limit: 10, relationalRetrieval, expansion: false }); + return results.map(r => r.slug); +}; + +describe('relational A/B', () => { + test('corpus has a meaningful question set', () => { + expect(RELATIONAL_QUESTIONS.length).toBeGreaterThanOrEqual(30); + }); + + test('relational.jsonl matches the corpus module (no drift)', () => { + const path = join(import.meta.dir, 'fixtures/retrieval-quality/relational/relational.jsonl'); + const fromFile = parseQuestionsJsonl(readFileSync(path, 'utf8')); + expect(JSON.stringify(fromFile)).toBe(JSON.stringify(RELATIONAL_QUESTIONS)); + }); + + test('recall@10 lifts materially with the relational arm ON', async () => { + const off = await runRetrievalQuality(RELATIONAL_QUESTIONS, searchFnWith(false)); + const on = await runRetrievalQuality(RELATIONAL_QUESTIONS, searchFnWith(true)); + + const offFam = off.families.find(f => f.family === 'graph-relationship')!; + const onFam = on.families.find(f => f.family === 'graph-relationship')!; + + // Answers are lexically unrecoverable → baseline recall is near zero. + expect(offFam.recall_at_10).toBeLessThan(0.25); + // Typed-edge arm surfaces them → large lift. + expect(onFam.recall_at_10).toBeGreaterThan(0.75); + expect(onFam.recall_at_10 - offFam.recall_at_10).toBeGreaterThan(0.5); + // Hit@3 should also rise sharply. + expect(onFam.hit_at_3).toBeGreaterThan(offFam.hit_at_3); + }, 120_000); + + test('no-regression: non-relational query is identical arm-on vs arm-off', async () => { + const q = 'early-stage venture fund first checks'; + const off = await searchFnWith(false)(q); + const on = await searchFnWith(true)(q); + expect(on).toEqual(off); + }, 60_000); +}); diff --git a/test/relational-fanout.test.ts b/test/relational-fanout.test.ts new file mode 100644 index 000000000..69862ecda --- /dev/null +++ b/test/relational-fanout.test.ts @@ -0,0 +1,115 @@ +/** + * relationalFanout unit tests (PGLite, default CI). + * + * Exercises the typed-edge fan-out SQL directly so the engine method is + * covered even when the DATABASE_URL-gated parity test does not run. Pins: + * typed-edge filtering, mentions-excluded-by-default, deleted_at exclusion, + * hop/edge_count aggregation, canonical_chunk_id, multi-seed + connects, + * and determinism. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import type { ChunkInput } from '../src/core/types.ts'; +import { probeEmbeddingDim } from './fixtures/retrieval-quality/relational/corpus.ts'; + +function emb(idx: number, dim: number): Float32Array { + const e = new Float32Array(dim); + e[idx % dim] = 1.0; + return e; +} + +let eng: PGLiteEngine; +let DIM = 0; + +beforeAll(async () => { + eng = new PGLiteEngine(); + await eng.connect({}); + await eng.initSchema(); + DIM = await probeEmbeddingDim(eng); // match the schema's column width (1280 ZE / 1536 OpenAI) + + const pages: Array<[string, string, string]> = [ + ['companies/widget-co', 'company', 'Widget Co'], + ['companies/other-co', 'company', 'Other Co'], + ['people/investor-a', 'person', 'Investor A'], + ['people/investor-b', 'person', 'Investor B'], + ['people/employee-c', 'person', 'Employee C'], + ['people/mentioner', 'person', 'Mentioner'], + ['people/deleted-investor', 'person', 'Deleted Investor'], + ]; + for (const [slug, type, title] of pages) { + await eng.putPage(slug, { type: type as 'company' | 'person', title, compiled_truth: `${title} body`, timeline: '' }); + } + // investor-b carries a chunk (→ canonical_chunk_id non-null); investor-a stays chunkless. + const chunk: ChunkInput[] = [{ chunk_index: 0, chunk_text: 'b', chunk_source: 'compiled_truth', embedding: emb(2, DIM), token_count: 1 }]; + await eng.upsertChunks('people/investor-b', chunk); + + // Edges into widget-co. + await eng.addLink('people/investor-a', 'companies/widget-co', '', 'invested_in', 'manual'); + await eng.addLink('people/investor-b', 'companies/widget-co', '', 'invested_in', 'manual'); + await eng.addLink('people/employee-c', 'companies/widget-co', '', 'works_at', 'manual'); + await eng.addLink('people/mentioner', 'companies/widget-co', '', 'mentions', 'mentions'); + await eng.addLink('people/deleted-investor', 'companies/widget-co', '', 'invested_in', 'manual'); + // investor-a also invested in other-co → widget-co and other-co connect via investor-a. + await eng.addLink('people/investor-a', 'companies/other-co', '', 'invested_in', 'manual'); + + // Soft-delete one investor; it must never surface. + await eng.executeRaw(`UPDATE pages SET deleted_at = now() WHERE slug = $1`, ['people/deleted-investor']); +}, 60_000); + +afterAll(async () => { + await eng.disconnect(); +}); + +describe('relationalFanout', () => { + test('typed-edge: who invested in widget-co', async () => { + const rows = await eng.relationalFanout(['companies/widget-co'], { direction: 'in', linkTypes: ['invested_in'] }); + const slugs = rows.map(r => r.slug).sort(); + expect(slugs).toEqual(['people/investor-a', 'people/investor-b']); + for (const r of rows) { + expect(r.hop).toBe(1); + expect(r.via_link_types).toEqual(['invested_in']); + } + }); + + test('deleted pages are excluded', async () => { + const rows = await eng.relationalFanout(['companies/widget-co'], { direction: 'in', linkTypes: ['invested_in'] }); + expect(rows.map(r => r.slug)).not.toContain('people/deleted-investor'); + }); + + test('mentions excluded by default, included on opt-in', async () => { + const off = await eng.relationalFanout(['companies/widget-co'], { direction: 'in' }); + expect(off.map(r => r.slug)).not.toContain('people/mentioner'); + // type-agnostic also picks up the works_at neighbor + expect(off.map(r => r.slug).sort()).toEqual(['people/employee-c', 'people/investor-a', 'people/investor-b']); + + const on = await eng.relationalFanout(['companies/widget-co'], { direction: 'in', includeMentions: true }); + expect(on.map(r => r.slug)).toContain('people/mentioner'); + }); + + test('canonical_chunk_id: non-null for chunked page, null for chunkless', async () => { + const rows = await eng.relationalFanout(['companies/widget-co'], { direction: 'in', linkTypes: ['invested_in'] }); + const a = rows.find(r => r.slug === 'people/investor-a')!; + const b = rows.find(r => r.slug === 'people/investor-b')!; + expect(a.canonical_chunk_id).toBeNull(); + expect(b.canonical_chunk_id).not.toBeNull(); + }); + + test('connects: shared midpoint reachable from both seeds with a path', async () => { + const rows = await eng.relationalFanout(['companies/widget-co', 'companies/other-co'], { direction: 'both' }); + const a = rows.find(r => r.slug === 'people/investor-a'); + expect(a).toBeDefined(); + expect(a!.path.length).toBeGreaterThanOrEqual(2); // [seed, ..., node] + expect(a!.path[a!.path.length - 1]).toBe('people/investor-a'); + }); + + test('empty seeds → []', async () => { + expect(await eng.relationalFanout([])).toEqual([]); + }); + + test('deterministic: same call twice is byte-identical', async () => { + const r1 = await eng.relationalFanout(['companies/widget-co'], { direction: 'in' }); + const r2 = await eng.relationalFanout(['companies/widget-co'], { direction: 'in' }); + expect(JSON.stringify(r1)).toBe(JSON.stringify(r2)); + }); +}); diff --git a/test/relational-intent.test.ts b/test/relational-intent.test.ts new file mode 100644 index 000000000..813969bab --- /dev/null +++ b/test/relational-intent.test.ts @@ -0,0 +1,136 @@ +/** + * Relational-query parser unit tests. + * + * Covers the four archetypes, the no-match path, the precision-first + * false-positive guard, schema-pack vocab extension, subset validation, and + * the length bound (ReDoS surface). + */ + +import { describe, test, expect } from 'bun:test'; +import { + parseRelationalQuery, + validateVocab, + KNOWN_LINK_TYPES, + type RelationVocab, +} from '../src/core/search/relational-intent.ts'; + +describe('parseRelationalQuery — archetypes', () => { + test('who_rel: who invested in widget-co', () => { + const r = parseRelationalQuery('who invested in widget-co'); + expect(r).not.toBeNull(); + expect(r!.kind).toBe('who_rel'); + expect(r!.seeds).toEqual(['widget-co']); + expect(r!.linkTypes).toEqual(['invested_in', 'led_round']); + expect(r!.direction).toBe('in'); + }); + + test('who_rel: who founded acme (article stripped)', () => { + const r = parseRelationalQuery('who founded the acme corporation?'); + expect(r!.kind).toBe('who_rel'); + expect(r!.seeds).toEqual(['acme corporation']); + expect(r!.linkTypes).toEqual(['founded']); + }); + + test('who_at: who at acme works on payments', () => { + const r = parseRelationalQuery('who at acme works on payments'); + expect(r!.kind).toBe('who_at'); + expect(r!.seeds).toEqual(['acme']); + expect(r!.linkTypes).toEqual(['works_at']); + expect(r!.direction).toBe('in'); + }); + + test('intro: who introduced me to alice-example (type-agnostic)', () => { + const r = parseRelationalQuery('who introduced me to alice-example?'); + expect(r!.kind).toBe('intro'); + expect(r!.seeds).toEqual(['alice-example']); + expect(r!.linkTypes).toBeNull(); + expect(r!.direction).toBe('both'); + }); + + test('connects: what connects fund-a and fund-b (two seeds)', () => { + const r = parseRelationalQuery('what connects fund-a and fund-b'); + expect(r!.kind).toBe('connects'); + expect(r!.seeds).toEqual(['fund-a', 'fund-b']); + expect(r!.linkTypes).toBeNull(); + expect(r!.direction).toBe('both'); + }); + + test('connects: how are X and Y related', () => { + const r = parseRelationalQuery('how are widget-co and acme related'); + expect(r!.kind).toBe('connects'); + expect(r!.seeds).toEqual(['widget-co', 'acme']); + }); + + test('outgoing: what did alice invest in', () => { + const r = parseRelationalQuery('what did alice invest in'); + expect(r!.kind).toBe('who_rel'); + expect(r!.seeds).toEqual(['alice']); + expect(r!.direction).toBe('out'); + }); +}); + +describe('parseRelationalQuery — precision-first / no-match', () => { + test('non-relational content query → null', () => { + expect(parseRelationalQuery('what is the capital structure of a seed round')).toBeNull(); + expect(parseRelationalQuery('notes from the offsite')).toBeNull(); + expect(parseRelationalQuery('summarize the q3 board deck')).toBeNull(); + }); + + test('false-positive: "who invested TIME in learning Rust" does NOT match', () => { + // "invested time in" is not "invested in" — adjacency guard. + expect(parseRelationalQuery('who invested time in learning Rust')).toBeNull(); + }); + + test('pronoun / stopword seed is rejected', () => { + expect(parseRelationalQuery('who invested in it')).toBeNull(); + expect(parseRelationalQuery('who founded them?')).toBeNull(); + }); + + test('empty / overlong input → null', () => { + expect(parseRelationalQuery('')).toBeNull(); + expect(parseRelationalQuery('who invested in ' + 'x'.repeat(600))).toBeNull(); + }); +}); + +describe('schema-pack vocab extension (D2=B)', () => { + const vocab: RelationVocab = { + extraVerbs: [{ verb: 'related to|associated with', linkTypes: ['related_to'], direction: 'both' }], + }; + + test('extra verb extends detection', () => { + expect(parseRelationalQuery('who related to widget-co', vocab)!.linkTypes).toEqual(['related_to']); + // same query without the pack does NOT match the extra verb + expect(parseRelationalQuery('who related to widget-co')).toBeNull(); + }); + + test('validateVocab passes for known types', () => { + expect(() => validateVocab(vocab)).not.toThrow(); + }); + + test('validateVocab throws on unknown link_type', () => { + expect(() => + validateVocab({ extraVerbs: [{ verb: 'pwns', linkTypes: ['not_a_real_edge'], direction: 'in' }] }), + ).toThrow(/unknown link_type/); + }); +}); + +describe('default bank emits only known link types (no drift)', () => { + test('every parsed linkType is a subset of KNOWN_LINK_TYPES', () => { + const queries = [ + 'who invested in widget-co', + 'who founded acme', + 'who advises bob-example', + 'who works at acme', + 'who at acme leads payments', + 'what did alice invest in', + 'where does alice work', + ]; + for (const q of queries) { + const r = parseRelationalQuery(q); + if (!r || r.linkTypes === null) continue; + for (const lt of r.linkTypes) { + expect(KNOWN_LINK_TYPES.has(lt)).toBe(true); + } + } + }); +}); diff --git a/test/relational-recall.test.ts b/test/relational-recall.test.ts new file mode 100644 index 000000000..61eff1777 --- /dev/null +++ b/test/relational-recall.test.ts @@ -0,0 +1,72 @@ +/** + * Relational recall arm integration tests (PGLite, default CI). + * + * End-to-end through buildRelationalArm: parse → resolve seed → fanout → + * hydrate. Pins the lexically-unrecoverable win (the investor page never names + * the company; only the invested_in edge connects them), the non-relational + * no-op, attribution stamping, and fail-open. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { buildRelationalArm } from '../src/core/search/relational-recall.ts'; +import { probeEmbeddingDim } from './fixtures/retrieval-quality/relational/corpus.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let eng: PGLiteEngine; + +beforeAll(async () => { + eng = new PGLiteEngine(); + await eng.connect({}); + await eng.initSchema(); + const dim = await probeEmbeddingDim(eng); // match schema column width (1280 ZE / 1536 OpenAI) + + await eng.putPage('companies/widget-co', { type: 'company', title: 'Widget Co', compiled_truth: 'A payments company.', timeline: '' }); + // The investor's body deliberately NEVER mentions Widget Co — only the edge connects them. + await eng.putPage('people/alice-example', { type: 'person', title: 'Alice Example', compiled_truth: 'Alice is a seed-stage investor based in Lisbon.', timeline: '' }); + await eng.upsertChunks('people/alice-example', [{ + chunk_index: 0, chunk_text: 'Alice is a seed-stage investor based in Lisbon.', + chunk_source: 'compiled_truth', embedding: new Float32Array(dim), token_count: 8, + }] satisfies ChunkInput[]); + await eng.addLink('people/alice-example', 'companies/widget-co', '', 'invested_in', 'manual'); +}, 60_000); + +afterAll(async () => { await eng.disconnect(); }); + +describe('buildRelationalArm', () => { + test('surfaces the edge answer that lexical search would miss', async () => { + const list = await buildRelationalArm(eng, 'who invested in widget-co'); + const alice = list.find(r => r.slug === 'people/alice-example'); + expect(alice).toBeDefined(); + expect(alice!.relational_via_link_types).toEqual(['invested_in']); + expect(alice!.relational_hop).toBe(1); + expect(alice!.relational_seed).toBe('companies/widget-co'); + // chunk-backed page → reinforces a REAL chunk id (not synthetic 0). + expect(alice!.chunk_id).toBeGreaterThan(0); + }); + + test('non-relational query is a pure no-op', async () => { + const meta: { fired?: boolean } = {}; + const list = await buildRelationalArm(eng, 'summary of the payments roadmap', { onMeta: m => { meta.fired = m.fired; } }); + expect(list).toEqual([]); + expect(meta.fired).toBe(false); + }); + + test('unresolvable seed → no-op (never traverse from a guess)', async () => { + const list = await buildRelationalArm(eng, 'who invested in nonexistent-phantom-xyz'); + expect(list).toEqual([]); + }); + + test('fail-open: fanout error returns [] + errored meta, never throws', async () => { + const original = eng.relationalFanout.bind(eng); + let captured: { errored?: boolean } = {}; + eng.relationalFanout = async () => { throw new Error('boom'); }; + try { + const list = await buildRelationalArm(eng, 'who invested in widget-co', { onMeta: m => { captured = m; } }); + expect(list).toEqual([]); + expect(captured.errored).toBe(true); + } finally { + eng.relationalFanout = original; + } + }); +}); diff --git a/test/rrf-source-key.test.ts b/test/rrf-source-key.test.ts new file mode 100644 index 000000000..245682d25 --- /dev/null +++ b/test/rrf-source-key.test.ts @@ -0,0 +1,70 @@ +/** + * RRF fusion key is source-aware (federation hardening). + * + * Pre-fix the RRF/dedup key was `slug:chunk_id`, which collapsed two + * same-slug pages living in different federated sources into one fusion + * entry — a cross-source recall bug. The key is now + * `(source_id, slug, chunk_id)`, matching `dedup.ts:pageKey`'s composite + * discipline at chunk granularity. These tests pin that behavior for both + * `rrfFusion` and `rrfFusionWeighted` and confirm single-source ranking is + * unchanged. + */ + +import { describe, test, expect } from 'bun:test'; +import { rrfFusion, rrfFusionWeighted } from '../src/core/search/hybrid.ts'; +import type { SearchResult } from '../src/core/types.ts'; + +function makeResult(overrides: Partial = {}): SearchResult { + return { + slug: 'test-page', + page_id: 1, + title: 'Test', + type: 'concept', + chunk_text: 'unique chunk text', + chunk_source: 'timeline', // avoid compiled_truth 2x boost noise + chunk_id: 1, + chunk_index: 0, + score: 0.5, + stale: false, + ...overrides, + }; +} + +describe('RRF key is source-aware', () => { + test('same slug + chunk_id in DIFFERENT sources do NOT collapse', () => { + const a = makeResult({ slug: 'people/alice', chunk_id: 1, source_id: 'team-a', chunk_text: 'x' }); + const b = makeResult({ slug: 'people/alice', chunk_id: 1, source_id: 'team-b', chunk_text: 'y' }); + const out = rrfFusion([[a, b]], 60); + expect(out.length).toBe(2); + expect(new Set(out.map(r => r.source_id))).toEqual(new Set(['team-a', 'team-b'])); + }); + + test('same slug + chunk_id + SAME source DO collapse (reinforce across lists)', () => { + const a = makeResult({ slug: 'people/alice', chunk_id: 1, source_id: 'team-a', chunk_text: 'x' }); + const b = makeResult({ slug: 'people/alice', chunk_id: 1, source_id: 'team-a', chunk_text: 'x' }); + const out = rrfFusion([[a], [b]], 60); + expect(out.length).toBe(1); + }); + + test('single-source (no source_id → "default") ranking unchanged', () => { + const a = makeResult({ slug: 'a', chunk_id: 1, chunk_text: 'aaa' }); + const b = makeResult({ slug: 'b', chunk_id: 2, chunk_text: 'bbb' }); + const out = rrfFusion([[a, b]], 60); + expect(out.length).toBe(2); + expect(out[0].slug).toBe('a'); // rank-0 wins, as before the key change + }); + + test('chunkless rows (null chunk_id) stay distinct per source via text prefix', () => { + const a = makeResult({ slug: 'people/alice', chunk_id: undefined as unknown as number, source_id: 'team-a', chunk_text: 'alice page' }); + const b = makeResult({ slug: 'people/alice', chunk_id: undefined as unknown as number, source_id: 'team-b', chunk_text: 'alice page' }); + const out = rrfFusion([[a, b]], 60); + expect(out.length).toBe(2); + }); + + test('weighted RRF is also source-aware', () => { + const a = makeResult({ slug: 'people/alice', chunk_id: 1, source_id: 'team-a' }); + const b = makeResult({ slug: 'people/alice', chunk_id: 1, source_id: 'team-b' }); + const out = rrfFusionWeighted([{ list: [a, b], k: 60 }]); + expect(out.length).toBe(2); + }); +}); diff --git a/test/search-alias-resolved-boost.test.ts b/test/search-alias-resolved-boost.test.ts index 8c9eb1110..e67bb0300 100644 --- a/test/search-alias-resolved-boost.test.ts +++ b/test/search-alias-resolved-boost.test.ts @@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => { }); describe('KNOBS_HASH_VERSION', () => { - it('is 9 (8→9 archive-demote invalidates archive-excluded cache rows, #1777)', () => { - expect(KNOBS_HASH_VERSION).toBe(9); + it('is 10 (9→10 relational recall arm invalidates rel-off cache rows, v0.43)', () => { + expect(KNOBS_HASH_VERSION).toBe(10); }); }); diff --git a/test/search-mode.test.ts b/test/search-mode.test.ts index cec3ba4b1..c2a3516e1 100644 --- a/test/search-mode.test.ts +++ b/test/search-mode.test.ts @@ -77,6 +77,9 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => { // v0.42.3.0 — autocut OFF for conservative (no reranker). autocut: false, autocut_jump: 0.2, + // v0.43 — relational recall OFF for conservative. + relationalRetrieval: false, + relational_retrieval_depth: 2, }); }); @@ -106,6 +109,9 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => { // v0.42.3.0 — autocut ON. autocut: true, autocut_jump: 0.2, + // v0.43 — relational recall ON for balanced. + relationalRetrieval: true, + relational_retrieval_depth: 2, }); }); @@ -133,6 +139,9 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => { // v0.42.3.0 — autocut ON. autocut: true, autocut_jump: 0.2, + // v0.43 — relational recall ON for tokenmax. + relationalRetrieval: true, + relational_retrieval_depth: 2, }); }); @@ -392,7 +401,9 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => { // archive/ demote (search-exclude policy change isn't in the hash, so the // version bump is what invalidates archive-excluded cache rows). A query // must not be served from a cache row written before the policy change. - expect(KNOBS_HASH_VERSION).toBe(9); + // v0.43: bumped 9→10 for the relational recall arm (rel=/reld=) — a + // relational-on write must not be served to a relational-off lookup. + expect(KNOBS_HASH_VERSION).toBe(10); }); test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => { @@ -557,8 +568,8 @@ describe('v0.40.4 — graph_signals knob', () => { }); describe('v0.42.3.0 — autocut knobs', () => { - test('KNOBS_HASH_VERSION is 9 (8→9 archive-demote, issue #1777)', () => { - expect(KNOBS_HASH_VERSION).toBe(9); + test('KNOBS_HASH_VERSION is 10 (9→10 relational recall arm, v0.43)', () => { + expect(KNOBS_HASH_VERSION).toBe(10); }); test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => { @@ -621,3 +632,44 @@ describe('v0.42.3.0 — autocut knobs', () => { expect(attr.value).toBe(false); }); }); + +describe('v0.43 — relational recall knobs', () => { + test('bundle defaults: conservative off, balanced/tokenmax on; depth 2', () => { + expect(MODE_BUNDLES.conservative.relationalRetrieval).toBe(false); + expect(MODE_BUNDLES.balanced.relationalRetrieval).toBe(true); + expect(MODE_BUNDLES.tokenmax.relationalRetrieval).toBe(true); + for (const m of ['conservative', 'balanced', 'tokenmax'] as const) { + expect(MODE_BUNDLES[m].relational_retrieval_depth).toBe(2); + } + }); + + test('per-call relationalRetrieval:false overrides bundle/config true', () => { + // bundle default true (balanced); per-call false wins + expect(resolveSearchMode({ mode: 'balanced', perCall: { relationalRetrieval: false } }).relationalRetrieval).toBe(false); + // config override true on conservative (bundle false); then per-call false beats config + expect( + resolveSearchMode({ mode: 'conservative', overrides: { relationalRetrieval: true }, perCall: { relationalRetrieval: false } }).relationalRetrieval, + ).toBe(false); + // config override alone flips conservative on + expect(resolveSearchMode({ mode: 'conservative', overrides: { relationalRetrieval: true } }).relationalRetrieval).toBe(true); + }); + + test('loadOverridesFromConfig reads search.relational_retrieval(+_depth)', () => { + const ov = loadOverridesFromConfig({ 'search.relational_retrieval': 'true', 'search.relational_retrieval_depth': '3' }); + expect(ov.relationalRetrieval).toBe(true); + expect(ov.relational_retrieval_depth).toBe(3); + // out-of-range depth is ignored (falls through to bundle) + expect(loadOverridesFromConfig({ 'search.relational_retrieval_depth': '9' }).relational_retrieval_depth).toBeUndefined(); + }); + + test('SEARCH_MODE_CONFIG_KEYS includes the relational keys', () => { + expect(SEARCH_MODE_CONFIG_KEYS).toContain('search.relational_retrieval'); + expect(SEARCH_MODE_CONFIG_KEYS).toContain('search.relational_retrieval_depth'); + }); + + test('knobsHash: relational-on vs off differ (cache isolation)', () => { + const on = knobsHash(resolveSearchMode({ mode: 'balanced' })); // relational true + const off = knobsHash(resolveSearchMode({ mode: 'balanced', perCall: { relationalRetrieval: false } })); + expect(on).not.toBe(off); + }); +}); diff --git a/test/search/knobs-hash-reranker.test.ts b/test/search/knobs-hash-reranker.test.ts index fac2f23eb..509e98249 100644 --- a/test/search/knobs-hash-reranker.test.ts +++ b/test/search/knobs-hash-reranker.test.ts @@ -43,7 +43,7 @@ function baseKnobs(): ResolvedSearchKnobs { } describe('KNOBS_HASH_VERSION + version invariants', () => { - test('version is 9 (…; 6→7 title_boost; 7→8 autocut; 8→9 archive-demote #1777)', () => { + test('version is 10 (…; 7→8 autocut; 8→9 archive-demote #1777; 9→10 relational recall)', () => { // v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold // floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs // (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation). @@ -57,7 +57,8 @@ describe('KNOBS_HASH_VERSION + version invariants', () => { // cannot leak past the new stage. T2: 6→7 title_boost. v0.42.3.0: 7→8 // autocut. issue #1777: 8→9 archive/ demote (search-exclude policy change // isn't in the hash, so the bump invalidates archive-excluded cache rows). - expect(KNOBS_HASH_VERSION).toBe(9); + // v0.43: 9→10 relational recall arm (rel=/reld=). + expect(KNOBS_HASH_VERSION).toBe(10); }); test('hash is 16 hex chars regardless of reranker config', () => {