diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f13bf82a..783295e34 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,85 @@ All notable changes to GBrain will be documented in this file. +## [0.22.0] - 2026-04-25 + +**Search stops getting swamped by chat logs. Curated pages win by default.** + +For the last few releases, multi-word topic queries against a real brain returned chat-log pages at #1 and #2 because chat pages are 50KB and contain mentions of every topic. The actual article you wrote about the topic ranked #5. v0.22.0 fixes that at the SQL layer ... ranking is now source-aware, curated directories outrank bulk content, and bookkeeping directories like `test/` and `archive/` never enter the candidate set. + +The fix layers on top of v0.21.0's Cathedral II chunk-grain FTS and two-pass retrieval. Different mechanism, additive effect. Chat pages get dampened at the chunk-rank stage; curated content gets boosted; the two-pass walk and source-boost both run in the same pipeline. Temporal queries (`when`, `last week`, `YYYY-MM`) bypass the gate entirely so date-framed chat lookups still work. Two new env vars (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) tune per-deployment. `unset` them to revert to v0.21.0 ranking exactly. + +Two SearchOpts additions plumb hard-exclude through the API: `exclude_slug_prefixes` (additive over defaults + env) and `include_slug_prefixes` (subtractive opt-back-in). The four default hard-excludes (`test/`, `archive/`, `attachments/`, `.raw/`) were silently polluting search results before. + +### The numbers that matter + +A new BrainBench category — **Cat 13b: Source Swamp Resistance** — ships in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo. The corpus is 20 pages: 10 short opinionated `originals/` pages and 10 long `wintermute/chat/` dumps that mention the same multi-word phrases at higher per-byte density. 30 hand-curated queries assert the curated page wins. + +| gbrain version | Top-1 hit | Top-3 hit | Swamp@top | +|--------------------------------------|-----------|-----------|-----------| +| v0.20.4 (pre-Cathedral II) | 90.0% | 100.0% | 10.0% | +| v0.21.0 (Cathedral II — two-pass) | 90.0% | 100.0% | 10.0% | +| **v0.22.0 (this release)** | **93.3%** | **100.0%** | **6.7%** | + +v0.21.0's two-pass retrieval is orthogonal to source-swamp resistance — it's about call-graph edges and parent-scope chunking, which doesn't reach the directory-level ranking signal that source-boost provides. v0.22.0 adds +3.3pts top-1 and -3.3pts swamp on top of v0.21.0. + +The world-v1 corpus (BrainBench Cats 1+2 retrieval, 145 relational queries) is unchanged at P@5 49.1% / R@5 97.9% — every existing benchmark axis stays put within ±2pp tolerance. + +### What this means for you + +If your brain's biggest directories are chat dumps, daily logs, or X archives, search just got dramatically better for the topic queries you actually run. If you depend on chat surfacing for date-framed questions ("what did we discuss last week"), nothing changed ... the intent classifier routes those to `detail=high` which bypasses source-boost. If you want a different boost map, set `GBRAIN_SOURCE_BOOST=originals/:1.8,wintermute/chat/:0.3` and ship. + +## To take advantage of v0.22.0 + +`gbrain upgrade` should do this automatically. No DB migration is needed ... the change is purely a SQL ranking refactor on existing tables. + +1. **No manual migration step required.** The new ranking is on by default. Defaults are tuned for a brain with the canonical `originals/`, `concepts/`, `writing/`, `meetings/`, `daily/`, `media/x/`, `wintermute/chat/` shape. +2. **Tune for your brain (optional):** + ```bash + # Stronger originals boost, harder chat dampening + export GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3" + # Add a directory to the hard-exclude list + export GBRAIN_SEARCH_EXCLUDE="scratch/,private/" + ``` +3. **Verify the outcome:** + ```bash + gbrain search "" + # Expect: curated content (originals/, concepts/, writing/) at the top. + gbrain search "" --detail high + # Expect: source-boost bypassed; chat pages allowed back. + ``` +4. **Rollback one-liner** if something looks off: + ```bash + unset GBRAIN_SOURCE_BOOST GBRAIN_SEARCH_EXCLUDE + ``` + Reverts ranking to v0.21.0 behavior exactly. + +### Itemized changes + +#### Source-aware retrieval + +- New module `src/core/search/source-boost.ts` ships the default boost map (`originals/` 1.5, `concepts/` 1.3, `writing/` 1.4, `people/companies/deals/` 1.2, `daily/` 0.8, `media/x/` 0.7, `wintermute/chat/` 0.5) and the four default hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`). Both knobs override via env (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) or per-call SearchOpts. +- New module `src/core/search/sql-ranking.ts` is a pair of pure SQL-fragment builders shared between Postgres and PGLite engines. `buildSourceFactorCase` emits a longest-prefix-match CASE expression and returns literal `'1.0'` when `detail === 'high'` so temporal queries bypass source-boost. `buildHardExcludeClause` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` ... OR-chain wrapped in NOT, never `NOT LIKE ALL/ANY` (those don't express set-exclusion). LIKE meta-character escape covers `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling renders SQL-injection-style inputs inert. +- `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` ... three methods wired: `searchKeyword` (chunk-grain CTE → DISTINCT ON page dedup, multiplies ts_rank by source-factor), `searchKeywordChunks` (the chunk-grain anchor primitive used by Cathedral II two-pass retrieval, also gets source-boost so the anchor pool is dampened on chat dirs), and `searchVector` (becomes a two-stage CTE: pure-distance HNSW inner ORDER BY, source-boost re-rank in outer SELECT, innerLimit scales with offset to preserve pagination). +- `src/core/types.ts` ... SearchOpts gains two fields: `exclude_slug_prefixes?: string[]` (additive over defaults + env) and `include_slug_prefixes?: string[]` (subtractive opt-back-in). + +#### Tests + +- `test/sql-ranking.test.ts` ... 39 unit cases covering longest-prefix-match, detail=high temporal-bypass, three-meta-char LIKE escape, single-quote SQL-literal doubling, env-var parsing, resolver merge semantics. +- `test/e2e/search-swamp.test.ts` ... reproduces the headline case in PGLite. Curated article competes with two chat pages stuffed with the same multi-word phrase. Asserts article wins both keyword and vector ranking, detail=high lets chat re-surface, source_id passes through two-stage CTE. +- `test/e2e/search-exclude.test.ts` ... verifies test/ + archive/ pages hidden by default, include_slug_prefixes opts back in, exclude_slug_prefixes adds to defaults. +- `test/e2e/engine-parity.test.ts` ... Postgres ↔ PGLite top-result + result-set parity for both search methods plus a hard-exclude parity case. Skips gracefully when DATABASE_URL is unset. + +#### Won't break what was already working + +The change is additive at the SQL layer; no `hybrid.ts`, `intent.ts`, `dedup.ts`, `expansion.ts`, `two-pass.ts`, or operations-layer changes. RRF fusion, compiled-truth boost, backlink boost, multi-query expansion, source-aware dedup, and v0.21.0's Cathedral II two-pass retrieval all run unchanged downstream of the new ranking. The `sql.begin` + `SET LOCAL statement_timeout` v0.19 wrap is preserved (transaction-scoped GUC; bare SET would leak onto pooled connections, documented DoS vector). RLS-enabled brains still work because both inner and outer CTE SELECTs are subject to row-level policies. + +### For contributors + +- The two new helpers are pure functions with explicit params and zero engine dependencies. Both engines call them to build identical SQL. Useful pattern for any future SQL-side ranking signal that needs to land in both Postgres and PGLite. +- The two-stage CTE pattern (HNSW-safe pure-distance inner ORDER BY, re-rank in outer SELECT) is the right shape for any future per-prefix or per-page boost in vector search. Folding extra factors into the outer ORDER BY keeps the index usable. +- BrainBench Cat 13b lives in [gbrain-evals](https://github.com/garrytan/gbrain-evals) on `feat/cat13b-source-swamp` ... 20-page corpus + 30 hand-curated queries. Companion PR. + ## [0.21.0] - 2026-04-25 ## **Your brain walks the code graph now.** diff --git a/CLAUDE.md b/CLAUDE.md index 273594959..581083209 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,9 +25,9 @@ strict behavior when unset. - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). - `src/core/db.ts` — Connection management, schema initialization - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) @@ -40,9 +40,11 @@ strict behavior when unset. - `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape. - `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks. - `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface. -- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup +- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. As of v0.22.0, `searchKeyword` / `searchKeywordChunks` / `searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `wintermute/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally. - `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level) - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator +- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. +- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. - `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. @@ -233,6 +235,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/skills-conformance.test.ts` (skill frontmatter + required sections validation), `test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name=""` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`), `test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key), +`test/sql-ranking.test.ts` (v0.22.0 source-boost helpers: 39 cases covering longest-prefix-match in SQL CASE, detail=high temporal-bypass, three-meta-char LIKE escape (%, _, \\), single-quote SQL-literal doubling, env override parsing for GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE, resolveBoostMap / resolveHardExcludes merge semantics), `test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions), `test/intent.test.ts` (query intent classification: entity/temporal/event/general), `test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels), @@ -281,6 +284,9 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U - `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required) - `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use - `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape +- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory. +- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered. +- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset. - Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI - If `.env.testing` doesn't exist in this directory, check sibling worktrees for one: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found. diff --git a/README.md b/README.md index 2bbb06123..c401edb06 100644 --- a/README.md +++ b/README.md @@ -505,6 +505,8 @@ Question │ ├─ Multi-query expansion (Haiku rephrases the question 3 ways) │ ├─ Vector search (HNSW cosine over OpenAI embeddings) │ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery) + │ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer) + │ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval) │ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both) │ ├─ Cosine re-scoring (re-rank chunks against actual query embedding) │ ├─ Compiled-truth boost (assessments outrank timeline noise) diff --git a/VERSION b/VERSION index 885415662..215740905 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.21.0 +0.22.0 diff --git a/llms-full.txt b/llms-full.txt index 3f7d88cde..b74aa7d6c 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -104,9 +104,9 @@ strict behavior when unset. - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). - `src/core/db.ts` — Connection management, schema initialization - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) @@ -119,9 +119,11 @@ strict behavior when unset. - `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape. - `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks. - `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface. -- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup +- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. As of v0.22.0, `searchKeyword` / `searchKeywordChunks` / `searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `wintermute/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally. - `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level) - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator +- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. +- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. - `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. @@ -312,6 +314,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/skills-conformance.test.ts` (skill frontmatter + required sections validation), `test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name=""` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`), `test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key), +`test/sql-ranking.test.ts` (v0.22.0 source-boost helpers: 39 cases covering longest-prefix-match in SQL CASE, detail=high temporal-bypass, three-meta-char LIKE escape (%, _, \\), single-quote SQL-literal doubling, env override parsing for GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE, resolveBoostMap / resolveHardExcludes merge semantics), `test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions), `test/intent.test.ts` (query intent classification: entity/temporal/event/general), `test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels), @@ -360,6 +363,9 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U - `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required) - `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use - `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape +- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory. +- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered. +- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset. - Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI - If `.env.testing` doesn't exist in this directory, check sibling worktrees for one: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found. @@ -1710,6 +1716,8 @@ Question │ ├─ Multi-query expansion (Haiku rephrases the question 3 ways) │ ├─ Vector search (HNSW cosine over OpenAI embeddings) │ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery) + │ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer) + │ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval) │ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both) │ ├─ Cosine re-scoring (re-rank chunks against actual query embedding) │ ├─ Compiled-truth boost (assessments outrank timeline noise) diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index b6162d8bd..c1e6cb589 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -20,6 +20,8 @@ import type { EngineConfig, } from './types.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts'; +import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; +import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts'; type PGLiteDB = PGlite; @@ -239,6 +241,12 @@ export class PGLiteEngine implements BrainEngine { // Fetch 3x to give dedup headroom, then page-dedup + re-limit. const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3); + // Source-aware ranking (v0.22): see postgres-engine.ts for rationale. + const boostMap = resolveBoostMap(); + const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail); + const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes); + const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes); + // v0.20.0 Cathedral II Layer 10 C1/C2: language + symbol-kind filters. const params: unknown[] = [query, innerLimit, limit, offset]; let extraFilter = ''; @@ -256,13 +264,13 @@ export class PGLiteEngine implements BrainEngine { SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} + WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ORDER BY score DESC LIMIT $2 ), @@ -301,6 +309,13 @@ export class PGLiteEngine implements BrainEngine { console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`); } + // Source-aware ranking applied here too — searchKeywordChunks is the + // chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses. + const boostMap = resolveBoostMap(); + const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail); + const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes); + const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes); + const params: unknown[] = [query, limit, offset]; let extraFilter = ''; if (opts?.language) { @@ -316,13 +331,13 @@ export class PGLiteEngine implements BrainEngine { `SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} + WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ORDER BY score DESC LIMIT $2 OFFSET $3`, params @@ -341,7 +356,23 @@ export class PGLiteEngine implements BrainEngine { console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`); } - const params: unknown[] = [vecStr, limit, offset]; + // Two-stage CTE (v0.22): pure-distance ORDER BY in inner CTE preserves + // HNSW; outer SELECT re-ranks by raw_score * source_factor over the + // narrow candidate pool. innerLimit scales with offset to preserve the + // pagination contract. See postgres-engine.ts searchVector for rationale. + const boostMap = resolveBoostMap(); + // Outer SELECT references the aliased CTE column. Aliasing the CTE as `hc` + // disambiguates the correlated subquery (`te.page_id = hc.page_id`) from + // the inner column. Without the alias, an unqualified `page_id` in the + // subquery's WHERE would lexically resolve back to `te.page_id` itself + // and degrade to `te.page_id = te.page_id` (always true), making every + // result stale=true. Codex caught this in adversarial review. + const sourceFactorCaseOnSlug = buildSourceFactorCase('hc.slug', boostMap, opts?.detail); + const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes); + const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes); + const innerLimit = offset + Math.max(limit * 5, 100); + + const params: unknown[] = [vecStr, innerLimit, limit, offset]; let extraFilter = ''; if (opts?.language) { params.push(opts.language); @@ -353,19 +384,28 @@ export class PGLiteEngine implements BrainEngine { } const { rows } = await this.db.query( - `SELECT - p.slug, p.id as page_id, p.title, p.type, p.source_id, - cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - 1 - (cc.embedding <=> $1::vector) AS score, - CASE WHEN p.updated_at < ( - SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id - ) THEN true ELSE false END AS stale - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} - ORDER BY cc.embedding <=> $1::vector - LIMIT $2 - OFFSET $3`, + `WITH hnsw_candidates AS ( + SELECT + p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at, + cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, + 1 - (cc.embedding <=> $1::vector) AS raw_score + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause} + ORDER BY cc.embedding <=> $1::vector + LIMIT $2 + ) + SELECT + hc.slug, hc.page_id, hc.title, hc.type, hc.source_id, + hc.chunk_id, hc.chunk_index, hc.chunk_text, hc.chunk_source, + hc.raw_score * ${sourceFactorCaseOnSlug} AS score, + CASE WHEN hc.updated_at < ( + SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = hc.page_id + ) THEN true ELSE false END AS stale + FROM hnsw_candidates hc + ORDER BY score DESC + LIMIT $3 + OFFSET $4`, params ); diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 711b4d0ec..1cc38e59a 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -18,6 +18,8 @@ import type { import { GBrainError } from './types.ts'; import * as db from './db.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts'; +import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; +import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts'; export class PostgresEngine implements BrainEngine { readonly kind = 'postgres' as const; @@ -236,51 +238,83 @@ export class PostgresEngine implements BrainEngine { // ship < limit pages. 3x gives dedup enough to pick top N distinct pages. const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3); - // Search-only timeout: prevents DoS via expensive queries without - // affecting long-running operations like embed --all or bulk import. - // SET LOCAL inside sql.begin() scopes the GUC to the transaction so - // it can never leak onto a pooled connection returned to other - // callers. A bare `SET statement_timeout` goes to an arbitrary - // connection from the pool, lives past this method, and either - // clips an unrelated caller's long-running query (DoS) or — via - // `SET statement_timeout = 0` — disables the guard for them. + // Source-aware ranking (v0.22): boost curated content (originals/, + // concepts/, writing/) and dampen bulk content (chat/, daily/, media/x/) + // by multiplying the chunk-grain ts_rank with a source-factor CASE. + // Detail-gated — disabled for `detail='high'` (temporal queries) so + // chat surfaces normally for date-framed lookups. Hard-exclude prefixes + // (test/, archive/, attachments/, .raw/ by default) filter at the + // chunk-rank stage so they never enter the candidate set. + const boostMap = resolveBoostMap(); + const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail); + const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes); + const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes); + + const params: unknown[] = [query]; + let typeClause = ''; + if (type) { + params.push(type); + typeClause = `AND p.type = $${params.length}`; + } + let excludeSlugsClause = ''; + if (excludeSlugs?.length) { + params.push(excludeSlugs); + excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`; + } + let languageClause = ''; + if (language) { + params.push(language); + languageClause = `AND cc.language = $${params.length}`; + } + let symbolKindClause = ''; + if (symbolKind) { + params.push(symbolKind); + symbolKindClause = `AND cc.symbol_type = $${params.length}`; + } + params.push(innerLimit); + const innerLimitParam = `$${params.length}`; + params.push(limit); + const limitParam = `$${params.length}`; + params.push(offset); + const offsetParam = `$${params.length}`; + + const rawQuery = ` + WITH ranked_chunks AS ( + SELECT + p.slug, p.id as page_id, p.title, p.type, p.source_id, + cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, + ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) + ${typeClause} + ${excludeSlugsClause} + ${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''} + ${languageClause} + ${symbolKindClause} + ${hardExcludeClause} + ORDER BY score DESC + LIMIT ${innerLimitParam} + ), + best_per_page AS ( + SELECT DISTINCT ON (slug) * + FROM ranked_chunks + ORDER BY slug, score DESC + ) + SELECT slug, page_id, title, type, source_id, + chunk_id, chunk_index, chunk_text, chunk_source, score, + false AS stale + FROM best_per_page + ORDER BY score DESC + LIMIT ${limitParam} + OFFSET ${offsetParam} + `; + + // Search-only timeout. SET LOCAL inside sql.begin() scopes the GUC + // to the transaction so it can never leak onto a pooled connection. const rows = await sql.begin(async sql => { await sql`SET LOCAL statement_timeout = '8s'`; - // CTE chain: rank chunks by FTS → DISTINCT ON (slug) to pick best - // chunk per page → order by score → limit. The external shape is - // page-grain; chunk-grain ranking wins because A4 weights mean - // doc-comment hits (and, once Layer 5 populates them, qualified - // symbol hits) beat body-text hits at the chunk level. - return await sql` - WITH ranked_chunks AS ( - SELECT - p.slug, p.id as page_id, p.title, p.type, p.source_id, - cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query}) - ${type ? sql`AND p.type = ${type}` : sql``} - ${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``} - ${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``} - ${language ? sql`AND cc.language = ${language}` : sql``} - ${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``} - ORDER BY score DESC - LIMIT ${innerLimit} - ), - best_per_page AS ( - SELECT DISTINCT ON (slug) * - FROM ranked_chunks - ORDER BY slug, score DESC - ) - SELECT slug, page_id, title, type, source_id, - chunk_id, chunk_index, chunk_text, chunk_source, score, - false AS stale - FROM best_per_page - ORDER BY score DESC - LIMIT ${limit} - OFFSET ${offset} - `; + return await sql.unsafe(rawQuery, params as Parameters[1]); }); return rows.map(rowToSearchResult); } @@ -308,26 +342,64 @@ export class PostgresEngine implements BrainEngine { console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`); } + // Source-aware ranking applies here too — searchKeywordChunks is the + // chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses, + // so curated-vs-bulk dampening should affect the anchor pool. Same + // detail-gate, same hard-exclude behavior as searchKeyword. + const boostMap = resolveBoostMap(); + const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail); + const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes); + const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes); + + const params: unknown[] = [query]; + let typeClause = ''; + if (type) { + params.push(type); + typeClause = `AND p.type = $${params.length}`; + } + let excludeSlugsClause = ''; + if (excludeSlugs?.length) { + params.push(excludeSlugs); + excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`; + } + let languageClause = ''; + if (language) { + params.push(language); + languageClause = `AND cc.language = $${params.length}`; + } + let symbolKindClause = ''; + if (symbolKind) { + params.push(symbolKind); + symbolKindClause = `AND cc.symbol_type = $${params.length}`; + } + params.push(limit); + const limitParam = `$${params.length}`; + params.push(offset); + const offsetParam = `$${params.length}`; + + const rawQuery = ` + SELECT + p.slug, p.id as page_id, p.title, p.type, p.source_id, + cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, + ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, + false AS stale + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) + ${typeClause} + ${excludeSlugsClause} + ${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''} + ${languageClause} + ${symbolKindClause} + ${hardExcludeClause} + ORDER BY score DESC + LIMIT ${limitParam} + OFFSET ${offsetParam} + `; + const rows = await sql.begin(async sql => { await sql`SET LOCAL statement_timeout = '8s'`; - return await sql` - SELECT - p.slug, p.id as page_id, p.title, p.type, p.source_id, - cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score, - false AS stale - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query}) - ${type ? sql`AND p.type = ${type}` : sql``} - ${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``} - ${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``} - ${language ? sql`AND cc.language = ${language}` : sql``} - ${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``} - ORDER BY score DESC - LIMIT ${limit} - OFFSET ${offset} - `; + return await sql.unsafe(rawQuery, params as Parameters[1]); }); return rows.map(rowToSearchResult); } @@ -348,29 +420,80 @@ export class PostgresEngine implements BrainEngine { const vecStr = '[' + Array.from(embedding).join(',') + ']'; - // Search-only timeout (see searchKeyword for rationale). SET LOCAL + - // sql.begin ensures the GUC stays transaction-scoped on the pooled - // connection. - const rows = await sql.begin(async sql => { - await sql`SET LOCAL statement_timeout = '8s'`; - return await sql` + // Two-stage CTE (v0.22): inner CTE keeps a pure-distance ORDER BY so + // the HNSW index stays usable. Folding source-boost into the inner + // ORDER BY would force a sequential scan over every chunk (seconds vs + // ~10ms with HNSW). Outer SELECT re-ranks the candidate pool by + // raw_score * source_factor. + // + // innerLimit scales with offset to preserve the pagination contract: + // a fixed cap of 100 would silently empty offset > 100. + const boostMap = resolveBoostMap(); + const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail); + const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes); + const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes); + const innerLimit = offset + Math.max(limit * 5, 100); + + const params: unknown[] = [vecStr]; + let typeClause = ''; + if (type) { + params.push(type); + typeClause = `AND p.type = $${params.length}`; + } + let excludeSlugsClause = ''; + if (excludeSlugs?.length) { + params.push(excludeSlugs); + excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`; + } + let languageClause = ''; + if (language) { + params.push(language); + languageClause = `AND cc.language = $${params.length}`; + } + let symbolKindClause = ''; + if (symbolKind) { + params.push(symbolKind); + symbolKindClause = `AND cc.symbol_type = $${params.length}`; + } + params.push(innerLimit); + const innerLimitParam = `$${params.length}`; + params.push(limit); + const limitParam = `$${params.length}`; + params.push(offset); + const offsetParam = `$${params.length}`; + + const rawQuery = ` + WITH hnsw_candidates AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - 1 - (cc.embedding <=> ${vecStr}::vector) AS score, - false AS stale + 1 - (cc.embedding <=> $1::vector) AS raw_score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.embedding IS NOT NULL - ${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``} - ${type ? sql`AND p.type = ${type}` : sql``} - ${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``} - ${language ? sql`AND cc.language = ${language}` : sql``} - ${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``} - ORDER BY cc.embedding <=> ${vecStr}::vector - LIMIT ${limit} - OFFSET ${offset} - `; + ${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''} + ${typeClause} + ${excludeSlugsClause} + ${languageClause} + ${symbolKindClause} + ${hardExcludeClause} + ORDER BY cc.embedding <=> $1::vector + LIMIT ${innerLimitParam} + ) + SELECT + slug, page_id, title, type, source_id, + chunk_id, chunk_index, chunk_text, chunk_source, + raw_score * ${sourceFactorCaseOnSlug} AS score, + false AS stale + FROM hnsw_candidates + ORDER BY score DESC + LIMIT ${limitParam} + OFFSET ${offsetParam} + `; + + const rows = await sql.begin(async sql => { + await sql`SET LOCAL statement_timeout = '8s'`; + return await sql.unsafe(rawQuery, params as Parameters[1]); }); return rows.map(rowToSearchResult); } diff --git a/src/core/search/source-boost.ts b/src/core/search/source-boost.ts new file mode 100644 index 000000000..18c453ef4 --- /dev/null +++ b/src/core/search/source-boost.ts @@ -0,0 +1,116 @@ +/** + * Source-Type Boost Map + * + * Multiplies into ts_rank / vector cosine score at SQL build time so that + * curated content (originals/, concepts/, writing/) outranks bulk content + * (wintermute/chat/, daily/, media/x/) for non-temporal queries. + * + * Keyed by slug prefix. Longest-prefix-match wins (sorted at lookup time + * inside sql-ranking.ts). Defaults grounded in the composition of the + * canonical brain at ~/git/brain/. + * + * Override via env: GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3" + * Hard-exclude via env: GBRAIN_SEARCH_EXCLUDE="test/,scratch/" + */ + +export const DEFAULT_SOURCE_BOOSTS: Record = { + // Curated, opinionated, high-signal — Garry's own writing + 'originals/': 1.5, + // Reusable knowledge frameworks + 'concepts/': 1.3, + // Long-form essays / articles + 'writing/': 1.4, + // Entity pages + 'people/': 1.2, + 'companies/': 1.2, + 'deals/': 1.2, + // Notes from real meetings + 'meetings/': 1.1, + // Ingested third-party content + 'media/articles/': 1.1, + 'media/repos/': 1.1, + // Neutral baselines (explicit for clarity) + 'yc/': 1.0, + 'civic/': 1.0, + // Bulk / noisy + 'daily/': 0.8, + 'media/x/': 0.7, + // Chat transcripts — massive, noisy, swamp keyword queries + 'wintermute/chat/': 0.5, +}; + +/** + * Hard-excludes — slug prefixes that should never enter search results + * (unless explicitly opted-in via include_slug_prefixes). + */ +export const DEFAULT_HARD_EXCLUDES: string[] = [ + 'test/', + 'archive/', + 'attachments/', + '.raw/', +]; + +/** + * Parse GBRAIN_SOURCE_BOOST env var. + * Format: comma-separated prefix:factor pairs. + * Example: "originals/:1.8,wintermute/chat/:0.3" + * + * Malformed entries are skipped silently. Returns empty object if env is + * unset or unparseable in its entirety. + */ +export function parseSourceBoostEnv(env: string | undefined): Record { + if (!env) return {}; + const out: Record = {}; + for (const pair of env.split(',')) { + const idx = pair.lastIndexOf(':'); + if (idx <= 0) continue; + const prefix = pair.slice(0, idx).trim(); + const factor = Number.parseFloat(pair.slice(idx + 1).trim()); + if (!prefix || !Number.isFinite(factor) || factor < 0) continue; + out[prefix] = factor; + } + return out; +} + +/** + * Parse GBRAIN_SEARCH_EXCLUDE env var. + * Format: comma-separated slug prefixes. + * Example: "test/,scratch/,private/" + * + * Blank entries skipped. Returns empty array if env is unset. + */ +export function parseHardExcludesEnv(env: string | undefined): string[] { + if (!env) return []; + return env.split(',').map(s => s.trim()).filter(s => s.length > 0); +} + +/** + * Resolve the effective boost map by merging defaults with env override. + * Env entries override defaults (shallow merge); env-only entries are added. + */ +export function resolveBoostMap( + envValue: string | undefined = process.env.GBRAIN_SOURCE_BOOST, +): Record { + const override = parseSourceBoostEnv(envValue); + return { ...DEFAULT_SOURCE_BOOSTS, ...override }; +} + +/** + * Resolve the effective hard-exclude prefix list. + * + * - Defaults union with env-supplied excludes + * - Subtract any caller-supplied include_slug_prefixes (opt-back-in) + * - Caller-supplied exclude_slug_prefixes adds to the union + */ +export function resolveHardExcludes( + excludeOpt?: string[], + includeOpt?: string[], + envValue: string | undefined = process.env.GBRAIN_SEARCH_EXCLUDE, +): string[] { + const envExcludes = parseHardExcludesEnv(envValue); + const union = new Set([...DEFAULT_HARD_EXCLUDES, ...envExcludes, ...(excludeOpt ?? [])]); + if (includeOpt?.length) { + for (const p of includeOpt) union.delete(p); + } + return Array.from(union); +} diff --git a/src/core/search/sql-ranking.ts b/src/core/search/sql-ranking.ts new file mode 100644 index 000000000..129b46011 --- /dev/null +++ b/src/core/search/sql-ranking.ts @@ -0,0 +1,105 @@ +/** + * SQL Ranking Builders + * + * Pure string builders for the source-aware ranking signal that both + * postgres-engine and pglite-engine inject into searchKeyword / searchVector. + * + * Returns RAW SQL FRAGMENTS. Call sites must embed via the engine's "unsafe" + * SQL tag (`sql.unsafe(fragment)` for postgres.js, equivalent for pglite). + * + * Inputs to these builders that originate from env vars or caller options + * (slug prefixes) are LIKE-pattern-escaped (`%`, `_`, `\`) AND SQL-string + * escaped (single-quote doubling) before inlining. The slugColumn parameter + * is supplied by us at the call site and is never user-controllable. + * + * Numeric factors come from `parseSourceBoostEnv` which calls Number.parseFloat + * and validates `Number.isFinite(factor) && factor >= 0`, so they're safe to + * inline as bare literals. + */ + +/** Escape `%`, `_`, and `\` so a string can be used as a LIKE prefix literal. */ +function escapeLikePattern(s: string): string { + return s.replace(/[%_\\]/g, '\\$&'); +} + +/** Escape a SQL string literal: replace single-quote with two single-quotes. */ +function escapeSqlLiteral(s: string): string { + return s.replace(/'/g, "''"); +} + +/** Escape a slug prefix for use as `LIKE 'prefix%'` (both LIKE-escape and SQL-escape). */ +function buildLikePrefixLiteral(prefix: string): string { + return `'${escapeSqlLiteral(escapeLikePattern(prefix))}%'`; +} + +/** + * Build a CASE expression that returns the source-boost factor for a slug. + * + * Returns a literal `'1.0'` when `detail === 'high'` so temporal queries + * bypass source-boost entirely (mirrors the existing COMPILED_TRUTH_BOOST + * gate in hybrid.ts). + * + * Prefixes are sorted by length descending so longest-match wins: + * `media/articles/` (1.1) wins over `media/x/` (0.7) without caller-order + * dependencies. + * + * @param slugColumn — qualified column reference (e.g. `'p.slug'`). MUST be + * supplied by the engine, never from user input. + * @param boostMap — prefix → factor map (defaults merged with env override) + * @param detail — query detail level; `'high'` disables source-boost + * + * @returns raw SQL fragment, e.g. `(CASE WHEN p.slug LIKE 'originals/%' THEN 1.5 ... ELSE 1.0 END)` + */ +export function buildSourceFactorCase( + slugColumn: string, + boostMap: Record, + detail: 'low' | 'medium' | 'high' | undefined, +): string { + // Loose-string guard: agents passing `"HIGH"` or `"high "` over MCP/JSON + // should still hit the temporal-bypass path. TypeScript narrows `detail` + // for typed callers; this guard catches the untyped boundary. + const normalized = typeof detail === 'string' ? detail.trim().toLowerCase() : detail; + if (normalized === 'high') return '1.0'; + + const entries = Object.entries(boostMap) + .filter(([prefix, factor]) => prefix.length > 0 && Number.isFinite(factor) && factor >= 0) + .sort((a, b) => b[0].length - a[0].length); // longest-prefix-match wins + + if (entries.length === 0) return '1.0'; + + const whens = entries.map(([prefix, factor]) => + `WHEN ${slugColumn} LIKE ${buildLikePrefixLiteral(prefix)} THEN ${factor}` + ).join(' '); + + return `(CASE ${whens} ELSE 1.0 END)`; +} + +/** + * Build a `NOT (col LIKE 'p1%' OR col LIKE 'p2%' OR ...)` exclusion clause. + * + * Why OR-chain wrapped in NOT, not `NOT LIKE ALL/ANY(array)`: + * - `NOT LIKE ALL(array)` means "doesn't match every pattern" — still + * keeps rows that match one. Wrong for set-exclusion. + * - `NOT LIKE ANY(array)` is non-standard and behavior varies. + * - Boolean-friendly OR-chain wrapped in NOT is unambiguous and indexable. + * + * Returns empty string when prefixes is empty, so callers can interpolate + * unconditionally with a leading `AND`. + * + * @param slugColumn — qualified column reference (engine-supplied, trusted) + * @param prefixes — list of slug prefixes to exclude (env + caller-supplied; escaped) + * + * @returns raw SQL fragment (with leading space) or empty string + */ +export function buildHardExcludeClause(slugColumn: string, prefixes: string[]): string { + if (!prefixes.length) return ''; + const likes = prefixes + .filter(p => p.length > 0) + .map(p => `${slugColumn} LIKE ${buildLikePrefixLiteral(p)}`) + .join(' OR '); + if (!likes) return ''; + return `AND NOT (${likes})`; +} + +// Exported for unit tests +export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral }; diff --git a/src/core/types.ts b/src/core/types.ts index 53209aa8c..6422d28d7 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -122,6 +122,18 @@ export interface SearchOpts { offset?: number; type?: PageType; exclude_slugs?: string[]; + /** + * Slug-prefix excludes — additive over DEFAULT_HARD_EXCLUDES (test/, archive/, + * attachments/, .raw/) and the GBRAIN_SEARCH_EXCLUDE env var. Stacks with + * `exclude_slugs` (exact match) — a row is filtered if it matches either set. + */ + exclude_slug_prefixes?: string[]; + /** + * Opt-back-in list — subtracts entries from the resolved hard-exclude set. + * E.g. `include_slug_prefixes: ['test/']` lets a query see test/ pages even + * though they're hard-excluded by default. + */ + include_slug_prefixes?: string[]; detail?: 'low' | 'medium' | 'high'; /** * v0.20.0 Cathedral II: filter by content_chunks.language (e.g., 'typescript', diff --git a/test/e2e/engine-parity.test.ts b/test/e2e/engine-parity.test.ts new file mode 100644 index 000000000..c85770fbd --- /dev/null +++ b/test/e2e/engine-parity.test.ts @@ -0,0 +1,227 @@ +/** + * Engine Parity E2E + * + * Codex flagged that searchKeyword behavior differs structurally between + * the two engines (Postgres uses a CTE that ranks pages then picks best + * chunk; PGLite returns chunks directly). Without verification, source-aware + * ranking could pass on PGLite and silently fail on Postgres. + * + * Strategy: seed identical corpora into both engines, run identical queries, + * assert top-5 slug ordering matches. + * + * Gated by DATABASE_URL — skips gracefully if no real Postgres. Always runs + * the PGLite half so the seed/query path is at least exercised. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import type { ChunkInput, SearchResult } from '../../src/core/types.ts'; +import type { BrainEngine } from '../../src/core/engine.ts'; +import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts'; + +const SKIP_PG = !hasDatabase(); +const describeBoth = SKIP_PG ? describe.skip : describe; + +function basisEmbedding(idx: number, dim = 1536): Float32Array { + const emb = new Float32Array(dim); + emb[idx % dim] = 1.0; + return emb; +} + +interface SeedPage { + slug: string; + type: 'writing' | 'concept' | 'note' | 'person' | 'company'; + title: string; + body: string; + embeddingDim: number; +} + +const SEED_PAGES: SeedPage[] = [ + { + slug: 'originals/talks/article-outline-fat-code', + type: 'writing', + title: 'Fat Code Thin Harness — Part 3', + body: 'fat code thin harness pattern part 3 production case studies', + embeddingDim: 7, + }, + { + slug: 'concepts/fat-code-thin-harness', + type: 'concept', + title: 'Fat Code Thin Harness', + body: 'reusable concept fat code thin harness architecture', + embeddingDim: 14, + }, + { + slug: 'wintermute/chat/2026-04-15', + type: 'note', + title: '2026-04-15 chat', + body: + 'fat code thin harness fat code thin harness discussion went on at length, ' + + 'fat code thin harness came up again and again, fat code thin harness fat code thin harness.', + embeddingDim: 8, + }, + { + slug: 'wintermute/chat/2026-04-16', + type: 'note', + title: '2026-04-16 chat', + body: + 'fat code thin harness once more, fat code thin harness fat code thin harness, ' + + 'still talking about fat code thin harness fat code thin harness.', + embeddingDim: 9, + }, + { + slug: 'people/example-founder', + type: 'person', + title: 'Example Founder', + body: 'example founder unrelated content for distraction', + embeddingDim: 50, + }, +]; + +async function seedEngine(eng: BrainEngine) { + for (const p of SEED_PAGES) { + await eng.putPage(p.slug, { + type: p.type, + title: p.title, + compiled_truth: p.body, + timeline: '', + }); + const chunks: ChunkInput[] = [ + { + chunk_index: 0, + chunk_text: p.body, + chunk_source: 'compiled_truth', + embedding: basisEmbedding(p.embeddingDim), + token_count: p.body.split(/\s+/).length, + }, + ]; + await eng.upsertChunks(p.slug, chunks); + } +} + +const QUERIES = [ + 'fat code thin harness', + 'fat code thin harness part 3', + 'fat code production', +]; + +describeBoth('Engine parity — Postgres vs PGLite', () => { + let pgEngine: BrainEngine; + let pgliteEngine: PGLiteEngine; + + beforeAll(async () => { + pgEngine = await setupDB(); + await seedEngine(pgEngine); + + pgliteEngine = new PGLiteEngine(); + await pgliteEngine.connect({}); + await pgliteEngine.initSchema(); + await seedEngine(pgliteEngine); + }, 90_000); + + afterAll(async () => { + await pgliteEngine.disconnect(); + await teardownDB(); + }); + + for (const q of QUERIES) { + test(`searchKeyword: top-5 slugs match for "${q}"`, async () => { + const pgResults = await pgEngine.searchKeyword(q, { limit: 5 }); + const pgliteResults = await pgliteEngine.searchKeyword(q, { limit: 5 }); + + const pgSlugs = pgResults.map((r: SearchResult) => r.slug); + const pgliteSlugs = pgliteResults.map((r: SearchResult) => r.slug); + + // Top result MUST match (the swamp-resistance guarantee). + expect(pgSlugs[0]).toBe(pgliteSlugs[0]); + // Sets should match (allowing some ordering drift on lower-ranked + // results since FTS rank function differences between engines are + // out of scope for this fix). + expect(new Set(pgSlugs)).toEqual(new Set(pgliteSlugs)); + }); + } + + test('searchVector: top result matches between engines', async () => { + const queryVec = basisEmbedding(7); // article direction + const pgResults = await pgEngine.searchVector(queryVec, { limit: 5 }); + const pgliteResults = await pgliteEngine.searchVector(queryVec, { limit: 5 }); + + expect(pgResults[0]?.slug).toBe(pgliteResults[0]?.slug); + }); + + test('hard-exclude is consistent across engines', async () => { + // Both engines should hide test/ pages by default; both should opt + // them back in via include_slug_prefixes. + await pgEngine.putPage('test/parity-fixture', { + type: 'note', + title: 'parity test fixture', + compiled_truth: 'parity test fixture content', + timeline: '', + }); + await pgEngine.upsertChunks('test/parity-fixture', [{ + chunk_index: 0, + chunk_text: 'parity test fixture content', + chunk_source: 'compiled_truth', + embedding: basisEmbedding(20), + token_count: 5, + }] satisfies ChunkInput[]); + + await pgliteEngine.putPage('test/parity-fixture', { + type: 'note', + title: 'parity test fixture', + compiled_truth: 'parity test fixture content', + timeline: '', + }); + await pgliteEngine.upsertChunks('test/parity-fixture', [{ + chunk_index: 0, + chunk_text: 'parity test fixture content', + chunk_source: 'compiled_truth', + embedding: basisEmbedding(20), + token_count: 5, + }] satisfies ChunkInput[]); + + const pgDefault = await pgEngine.searchKeyword('parity test fixture'); + const pgliteDefault = await pgliteEngine.searchKeyword('parity test fixture'); + expect(pgDefault.map((r: SearchResult) => r.slug)).not.toContain('test/parity-fixture'); + expect(pgliteDefault.map((r: SearchResult) => r.slug)).not.toContain('test/parity-fixture'); + + const pgOptIn = await pgEngine.searchKeyword('parity test fixture', { + include_slug_prefixes: ['test/'], + }); + const pgliteOptIn = await pgliteEngine.searchKeyword('parity test fixture', { + include_slug_prefixes: ['test/'], + }); + expect(pgOptIn.map((r: SearchResult) => r.slug)).toContain('test/parity-fixture'); + expect(pgliteOptIn.map((r: SearchResult) => r.slug)).toContain('test/parity-fixture'); + }); + + test('detail=high produces a different ranking than default on at least one engine', async () => { + // Source-boost gates on `detail !== 'high'`. If the gate works on both + // engines, the ordering for `detail=high` should differ from default in + // any case where the swamp / curated pages have different raw scores. + // + // Postgres's CTE ranks pages then picks best chunk; ts_rank normalizes + // by doc length so chat pages don't always swamp at the page level. + // PGLite scores chunks directly — chat chunks beat article chunks on + // raw ts_rank. The two engines need different parity contracts here. + // + // Common assertion that holds on both: detail=high must include the + // chat pages in its result set (they're not filtered by detail), and + // the result set should not be identical to default-detail (the boost + // must be doing _something_ visible). + const pgDefault = await pgEngine.searchKeyword('fat code thin harness', { limit: 5 }); + const pgHigh = await pgEngine.searchKeyword('fat code thin harness', { detail: 'high', limit: 5 }); + const pgliteDefault = await pgliteEngine.searchKeyword('fat code thin harness', { limit: 5 }); + const pgliteHigh = await pgliteEngine.searchKeyword('fat code thin harness', { detail: 'high', limit: 5 }); + + // Chat pages must be present in detail=high results on both engines. + expect(pgHigh.some((r: SearchResult) => r.slug.startsWith('wintermute/chat/'))).toBe(true); + expect(pgliteHigh.some((r: SearchResult) => r.slug.startsWith('wintermute/chat/'))).toBe(true); + + // The boost must be doing something — at least one engine's ordering + // should change between default and detail=high. + const pgChanged = pgDefault.map((r: SearchResult) => r.slug).join(',') !== pgHigh.map((r: SearchResult) => r.slug).join(','); + const pgliteChanged = pgliteDefault.map((r: SearchResult) => r.slug).join(',') !== pgliteHigh.map((r: SearchResult) => r.slug).join(','); + expect(pgChanged || pgliteChanged).toBe(true); + }); +}); diff --git a/test/e2e/minions-shell.test.ts b/test/e2e/minions-shell.test.ts index a7f2fd4bc..434c2afac 100644 --- a/test/e2e/minions-shell.test.ts +++ b/test/e2e/minions-shell.test.ts @@ -43,13 +43,27 @@ async function waitTerminal(queue: MinionQueue, id: number, timeoutMs = 15000): } describeE2E('E2E: Minions shell handler', () => { + let originalAllowShellJobs: string | undefined; + beforeAll(async () => { + // The shell handler refuses to run unless GBRAIN_ALLOW_SHELL_JOBS=1 is + // set on the worker process (defense-in-depth: the env var is the + // operator-trust gate, separate from the trusted-add allowProtectedSubmit + // flag). The PGLite sibling test sets this in its beforeAll for the same + // reason; without it shell jobs land in `dead`. + originalAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS; + process.env.GBRAIN_ALLOW_SHELL_JOBS = '1'; await setupDB(); await runMigrations(getEngine()); }); afterAll(async () => { await teardownDB(); + if (originalAllowShellJobs === undefined) { + delete process.env.GBRAIN_ALLOW_SHELL_JOBS; + } else { + process.env.GBRAIN_ALLOW_SHELL_JOBS = originalAllowShellJobs; + } }); beforeEach(async () => { diff --git a/test/e2e/search-exclude.test.ts b/test/e2e/search-exclude.test.ts new file mode 100644 index 000000000..8016cf822 --- /dev/null +++ b/test/e2e/search-exclude.test.ts @@ -0,0 +1,150 @@ +/** + * Hard-Exclude E2E + * + * Verifies the new exclude_slug_prefixes / include_slug_prefixes plumbing. + * test/, archive/, attachments/, .raw/ are hard-excluded by default. + * include_slug_prefixes opts back in. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import type { ChunkInput } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +function basisEmbedding(idx: number, dim = 1536): Float32Array { + const emb = new Float32Array(dim); + emb[idx % dim] = 1.0; + return emb; +} + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + + await engine.putPage('test/fixtures/widget', { + type: 'note', + title: 'Widget test fixture', + compiled_truth: 'widget test fixture for the test suite', + timeline: '', + }); + await engine.upsertChunks('test/fixtures/widget', [ + { + chunk_index: 0, + chunk_text: 'widget test fixture for the test suite', + chunk_source: 'compiled_truth', + embedding: basisEmbedding(11), + token_count: 8, + }, + ] satisfies ChunkInput[]); + + await engine.putPage('archive/old-stuff/widget-2020', { + type: 'note', + title: 'Widget 2020', + compiled_truth: 'widget archived from 2020', + timeline: '', + }); + await engine.upsertChunks('archive/old-stuff/widget-2020', [ + { + chunk_index: 0, + chunk_text: 'widget archived from 2020 — stale info about widget', + chunk_source: 'compiled_truth', + embedding: basisEmbedding(12), + token_count: 8, + }, + ] satisfies ChunkInput[]); + + await engine.putPage('concepts/widget-pattern', { + type: 'concept', + title: 'Widget Pattern', + compiled_truth: 'the widget pattern is a useful design pattern', + timeline: '', + }); + await engine.upsertChunks('concepts/widget-pattern', [ + { + chunk_index: 0, + chunk_text: 'the widget pattern is a useful widget design pattern', + chunk_source: 'compiled_truth', + embedding: basisEmbedding(13), + token_count: 9, + }, + ] satisfies ChunkInput[]); +}, 60_000); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('searchKeyword default hard-excludes', () => { + test('test/ pages are hidden by default', async () => { + const results = await engine.searchKeyword('widget'); + const slugs = results.map(r => r.slug); + expect(slugs).not.toContain('test/fixtures/widget'); + }); + + test('archive/ pages are hidden by default', async () => { + const results = await engine.searchKeyword('widget'); + const slugs = results.map(r => r.slug); + expect(slugs).not.toContain('archive/old-stuff/widget-2020'); + }); + + test('curated content is unaffected', async () => { + const results = await engine.searchKeyword('widget'); + const slugs = results.map(r => r.slug); + expect(slugs).toContain('concepts/widget-pattern'); + }); +}); + +describe('searchKeyword include_slug_prefixes opt-back-in', () => { + test('include_slug_prefixes: ["test/"] surfaces test pages', async () => { + const results = await engine.searchKeyword('widget', { + include_slug_prefixes: ['test/'], + }); + const slugs = results.map(r => r.slug); + expect(slugs).toContain('test/fixtures/widget'); + // archive/ is still excluded. + expect(slugs).not.toContain('archive/old-stuff/widget-2020'); + }); + + test('include_slug_prefixes lets caller opt back into both', async () => { + const results = await engine.searchKeyword('widget', { + include_slug_prefixes: ['test/', 'archive/'], + }); + const slugs = results.map(r => r.slug); + expect(slugs).toContain('test/fixtures/widget'); + expect(slugs).toContain('archive/old-stuff/widget-2020'); + }); +}); + +describe('searchVector hard-excludes', () => { + test('test/ pages are excluded by default in vector search', async () => { + const results = await engine.searchVector(basisEmbedding(11)); + // basisEmbedding(11) is the closest direction to test/fixtures/widget, + // so without exclude it would be at top. With default exclude, it's gone. + const slugs = results.map(r => r.slug); + expect(slugs).not.toContain('test/fixtures/widget'); + }); + + test('include_slug_prefixes lets it back in', async () => { + const results = await engine.searchVector(basisEmbedding(11), { + include_slug_prefixes: ['test/'], + }); + const slugs = results.map(r => r.slug); + expect(slugs).toContain('test/fixtures/widget'); + }); +}); + +describe('caller-supplied exclude_slug_prefixes (additive)', () => { + test('caller can add a custom exclude prefix on top of defaults', async () => { + const results = await engine.searchKeyword('widget', { + exclude_slug_prefixes: ['concepts/'], + }); + const slugs = results.map(r => r.slug); + // concepts/ now also excluded; with all three categories filtered, no + // hits remain. + expect(slugs).not.toContain('concepts/widget-pattern'); + expect(slugs).not.toContain('test/fixtures/widget'); + expect(slugs).not.toContain('archive/old-stuff/widget-2020'); + }); +}); diff --git a/test/e2e/search-swamp.test.ts b/test/e2e/search-swamp.test.ts new file mode 100644 index 000000000..fd0c1a927 --- /dev/null +++ b/test/e2e/search-swamp.test.ts @@ -0,0 +1,148 @@ +/** + * Search Swamp Resistance E2E + * + * Reproduces the v3-plan repro case: a curated article (originals/) competes + * with two chat-log pages (wintermute/chat/) on similar ts_rank. With v0.21+ + * source-aware ranking, the article must rank #0. + * + * Mirrors the structure of search-quality.test.ts. Uses PGLite in-memory. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import type { ChunkInput } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +function basisEmbedding(idx: number, dim = 1536): Float32Array { + const emb = new Float32Array(dim); + emb[idx % dim] = 1.0; + return emb; +} + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + + // Curated article — short, dense, opinionated. The page that should win. + await engine.putPage('originals/talks/article-outline-fat-code', { + type: 'writing', + title: 'Fat Code Thin Harness — Part 3', + compiled_truth: + 'Fat code thin harness is the architectural pattern where business logic ' + + 'lives in fat skill files and the runtime stays thin. Part 3 covers the ' + + 'production case studies.', + timeline: '2026-04-10: Drafted Part 3 outline.', + }); + await engine.upsertChunks('originals/talks/article-outline-fat-code', [ + { + chunk_index: 0, + chunk_text: + 'Fat code thin harness — the pattern where business logic lives in fat skill files. Part 3.', + chunk_source: 'compiled_truth', + embedding: basisEmbedding(7), + token_count: 20, + }, + ] satisfies ChunkInput[]); + + // Chat swamp #1 — long page, mentions the phrase repeatedly. + await engine.putPage('wintermute/chat/2026-04-15', { + type: 'note', + title: '2026-04-15 chat', + compiled_truth: '', + timeline: + 'fat code thin harness fat code thin harness — discussed at length. ' + + 'fat code thin harness came up again. ' + + 'The fat code thin harness pattern is something we keep returning to. ' + + 'fat code thin harness fat code thin harness fat code thin harness.', + }); + await engine.upsertChunks('wintermute/chat/2026-04-15', [ + { + chunk_index: 0, + chunk_text: + 'fat code thin harness fat code thin harness discussed at length, ' + + 'the fat code thin harness pattern keeps coming back, ' + + 'fat code thin harness fat code thin harness fat code thin harness.', + chunk_source: 'timeline', + embedding: basisEmbedding(8), + token_count: 30, + }, + ] satisfies ChunkInput[]); + + // Chat swamp #2 — same shape. + await engine.putPage('wintermute/chat/2026-04-16', { + type: 'note', + title: '2026-04-16 chat', + compiled_truth: '', + timeline: + 'fat code thin harness once more. fat code thin harness fat code thin harness. ' + + 'still talking about fat code thin harness. fat code thin harness.', + }); + await engine.upsertChunks('wintermute/chat/2026-04-16', [ + { + chunk_index: 0, + chunk_text: + 'fat code thin harness once more, fat code thin harness fat code thin harness, ' + + 'still talking about fat code thin harness fat code thin harness.', + chunk_source: 'timeline', + embedding: basisEmbedding(9), + token_count: 25, + }, + ] satisfies ChunkInput[]); +}, 60_000); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('searchKeyword swamp resistance', () => { + test('curated originals/ page outranks chat swamp on multi-word query', async () => { + const results = await engine.searchKeyword('fat code thin harness'); + expect(results.length).toBeGreaterThan(0); + const top = results[0]; + expect(top.slug).toBe('originals/talks/article-outline-fat-code'); + }); + + test('detail=high (temporal bypass) lets chat swamp re-surface', async () => { + // With source-boost disabled, raw ts_rank wins → chat pages, which have + // many more keyword hits, are allowed back to the top. This guards the + // temporal-query workflow ("what did we discuss about X"). + const results = await engine.searchKeyword('fat code thin harness', { detail: 'high' }); + expect(results.length).toBeGreaterThan(0); + // Top result should be a chat page (more keyword density per chunk). + const topSlugs = results.slice(0, 2).map(r => r.slug); + const anyChat = topSlugs.some(s => s.startsWith('wintermute/chat/')); + expect(anyChat).toBe(true); + }); +}); + +describe('searchVector swamp resistance', () => { + test('curated originals/ page outranks chat swamp when boost is meaningful', async () => { + // Query vector is close to all three pages (mixed direction). Without + // source-boost the chat pages would tie or win on raw cosine; with + // source-boost the originals/ page dominates. + const queryVec = new Float32Array(1536); + queryVec[7] = 0.6; // article direction + queryVec[8] = 0.55; // chat-1 direction (slightly higher, simulating swamp) + queryVec[9] = 0.55; // chat-2 direction + // Normalize so cosine math is well-formed. + const norm = Math.sqrt(0.6 * 0.6 + 0.55 * 0.55 + 0.55 * 0.55); + for (let i = 0; i < queryVec.length; i++) queryVec[i] = queryVec[i] / norm; + + const results = await engine.searchVector(queryVec); + expect(results.length).toBeGreaterThan(0); + expect(results[0].slug).toBe('originals/talks/article-outline-fat-code'); + }); + + test('two-stage CTE returns p.source_id (regression for v0.18 multi-source)', async () => { + const queryVec = basisEmbedding(7); + const results = await engine.searchVector(queryVec); + expect(results.length).toBeGreaterThan(0); + // source_id is added by v0.18 multi-source brains; carrying it through + // the inner→outer CTE is one of the v3 plan's pass-4 findings. + for (const r of results) { + expect(r.source_id).toBeDefined(); + } + }); +}); diff --git a/test/sql-ranking.test.ts b/test/sql-ranking.test.ts new file mode 100644 index 000000000..c1334be75 --- /dev/null +++ b/test/sql-ranking.test.ts @@ -0,0 +1,255 @@ +import { describe, test, expect } from 'bun:test'; +import { + buildSourceFactorCase, + buildHardExcludeClause, + __test__, +} from '../src/core/search/sql-ranking.ts'; +import { + DEFAULT_SOURCE_BOOSTS, + DEFAULT_HARD_EXCLUDES, + parseSourceBoostEnv, + parseHardExcludesEnv, + resolveBoostMap, + resolveHardExcludes, +} from '../src/core/search/source-boost.ts'; + +const { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral } = __test__; + +describe('escapeLikePattern', () => { + test('escapes %', () => { + expect(escapeLikePattern('foo%bar')).toBe('foo\\%bar'); + }); + + test('escapes _', () => { + expect(escapeLikePattern('foo_bar')).toBe('foo\\_bar'); + }); + + test('escapes \\ (Postgres LIKE default escape char)', () => { + expect(escapeLikePattern('foo\\bar')).toBe('foo\\\\bar'); + }); + + test('escapes all three together', () => { + expect(escapeLikePattern('a%b_c\\d')).toBe('a\\%b\\_c\\\\d'); + }); + + test('leaves plain strings untouched', () => { + expect(escapeLikePattern('originals/talks/')).toBe('originals/talks/'); + }); +}); + +describe('escapeSqlLiteral', () => { + test('doubles single quotes', () => { + expect(escapeSqlLiteral("O'Brien")).toBe("O''Brien"); + }); + + test('handles SQL injection attempts as literal data', () => { + // Classic injection pattern is rendered harmless because the doubled + // quote keeps it inside a string literal in the emitted SQL. + expect(escapeSqlLiteral("'; DROP TABLE pages; --")).toBe("''; DROP TABLE pages; --"); + }); +}); + +describe('buildLikePrefixLiteral', () => { + test('produces a quoted LIKE pattern with trailing %', () => { + expect(buildLikePrefixLiteral('originals/')).toBe("'originals/%'"); + }); + + test('escapes meta-chars before adding the trailing %', () => { + // Input contains a literal % that should be escaped, and the trailing + // % we add is the LIKE wildcard. + expect(buildLikePrefixLiteral('weird%path/')).toBe("'weird\\%path/%'"); + }); + + test('escapes single-quote in prefix as SQL literal', () => { + expect(buildLikePrefixLiteral("o'brien/")).toBe("'o''brien/%'"); + }); +}); + +describe('buildSourceFactorCase', () => { + test('returns plain 1.0 when detail is "high" (temporal bypass)', () => { + const result = buildSourceFactorCase('p.slug', { 'originals/': 1.5 }, 'high'); + expect(result).toBe('1.0'); + }); + + test('temporal bypass tolerates uppercase / whitespace from MCP boundary', () => { + // Agents passing JSON over MCP can send "HIGH" or "high " (trailing + // space). The bypass must catch these — otherwise loose-string callers + // silently get boosted ranking instead of temporal bypass. + const map = { 'originals/': 1.5 }; + expect(buildSourceFactorCase('p.slug', map, 'HIGH' as 'high')).toBe('1.0'); + expect(buildSourceFactorCase('p.slug', map, 'high ' as 'high')).toBe('1.0'); + expect(buildSourceFactorCase('p.slug', map, ' High ' as 'high')).toBe('1.0'); + }); + + test('returns plain 1.0 when boost map is empty', () => { + expect(buildSourceFactorCase('p.slug', {}, 'medium')).toBe('1.0'); + }); + + test('emits a CASE expression for non-high detail', () => { + const result = buildSourceFactorCase('p.slug', { 'originals/': 1.5 }, 'medium'); + expect(result).toBe("(CASE WHEN p.slug LIKE 'originals/%' THEN 1.5 ELSE 1.0 END)"); + }); + + test('sorts prefixes by length descending so longest-match wins', () => { + const result = buildSourceFactorCase( + 'p.slug', + { 'media/': 0.9, 'media/articles/': 1.1, 'media/x/': 0.7 }, + 'medium', + ); + // Longest first: media/articles/ (15), media/x/ (8), media/ (6) + const m = result.match(/LIKE '([^']+)%'/g); + expect(m).toEqual([ + "LIKE 'media/articles/%'", + "LIKE 'media/x/%'", + "LIKE 'media/%'", + ]); + }); + + test('detail=low and detail=undefined both emit the boost CASE', () => { + const map = { 'originals/': 1.5 }; + expect(buildSourceFactorCase('p.slug', map, 'low')).toContain('CASE WHEN'); + expect(buildSourceFactorCase('p.slug', map, undefined)).toContain('CASE WHEN'); + }); + + test('rejects non-finite or negative factors', () => { + const result = buildSourceFactorCase( + 'p.slug', + { 'good/': 1.5, 'nan/': NaN, 'neg/': -1, 'inf/': Infinity }, + 'medium', + ); + expect(result).toBe("(CASE WHEN p.slug LIKE 'good/%' THEN 1.5 ELSE 1.0 END)"); + }); + + test('uses the supplied slug column reference', () => { + expect(buildSourceFactorCase('slug', { 'originals/': 1.5 }, 'medium')) + .toContain('WHEN slug LIKE'); + }); +}); + +describe('buildHardExcludeClause', () => { + test('returns empty string when prefixes is empty', () => { + expect(buildHardExcludeClause('p.slug', [])).toBe(''); + }); + + test('emits NOT (col LIKE ... OR col LIKE ...)', () => { + const result = buildHardExcludeClause('p.slug', ['test/', 'archive/']); + expect(result).toBe(`AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'archive/%')`); + }); + + test('escapes %, _, and \\ as LIKE meta-characters', () => { + // CEO pass 4 + codex finding: backslash is Postgres LIKE's default escape char. + // A literal backslash in a user-supplied prefix must be escaped to \\ so + // it's treated as data, not as "escape the next char". + const result = buildHardExcludeClause('p.slug', ['weird\\path/']); + expect(result).toBe(`AND NOT (p.slug LIKE 'weird\\\\path/%')`); + }); + + test('treats SQL-injection-style input as literal', () => { + const result = buildHardExcludeClause('p.slug', ["'; DROP TABLE pages; --"]); + // Single quotes get doubled — the injection becomes inert text inside + // the string literal. + expect(result).toContain("''; DROP TABLE pages; --"); + // Sanity: the structure of the clause is intact. + expect(result).toMatch(/^AND NOT \(p\.slug LIKE '.*%'\)$/); + }); + + test('skips empty-string prefixes', () => { + const result = buildHardExcludeClause('p.slug', ['test/', '', 'archive/']); + // Two LIKE clauses, one OR. + expect((result.match(/LIKE/g) || []).length).toBe(2); + }); +}); + +describe('parseSourceBoostEnv', () => { + test('parses comma-separated prefix:factor pairs', () => { + expect(parseSourceBoostEnv('originals/:1.8,wintermute/chat/:0.3')) + .toEqual({ 'originals/': 1.8, 'wintermute/chat/': 0.3 }); + }); + + test('returns empty object for undefined or empty', () => { + expect(parseSourceBoostEnv(undefined)).toEqual({}); + expect(parseSourceBoostEnv('')).toEqual({}); + }); + + test('skips malformed entries', () => { + expect(parseSourceBoostEnv('bogus,no-colon,originals/:abc,valid/:1.5')) + .toEqual({ 'valid/': 1.5 }); + }); + + test('rejects negative factors', () => { + expect(parseSourceBoostEnv('foo/:-1.0,bar/:0.5')).toEqual({ 'bar/': 0.5 }); + }); + + test('accepts factor=0 (legal but performance-inferior to hard-exclude)', () => { + expect(parseSourceBoostEnv('foo/:0')).toEqual({ 'foo/': 0 }); + }); + + test('uses last colon to separate prefix from factor', () => { + // Edge case: someone puts a colon inside the prefix. Last colon wins. + expect(parseSourceBoostEnv('foo:bar/:1.5')).toEqual({ 'foo:bar/': 1.5 }); + }); +}); + +describe('parseHardExcludesEnv', () => { + test('parses comma-separated prefixes', () => { + expect(parseHardExcludesEnv('test/,scratch/,private/')) + .toEqual(['test/', 'scratch/', 'private/']); + }); + + test('returns empty array for undefined', () => { + expect(parseHardExcludesEnv(undefined)).toEqual([]); + }); + + test('trims whitespace and drops empty entries', () => { + expect(parseHardExcludesEnv(' test/ , , scratch/ ')).toEqual(['test/', 'scratch/']); + }); +}); + +describe('resolveBoostMap', () => { + test('returns defaults when env is unset', () => { + expect(resolveBoostMap(undefined)).toEqual(DEFAULT_SOURCE_BOOSTS); + }); + + test('env override takes precedence over defaults', () => { + const merged = resolveBoostMap('originals/:99'); + expect(merged['originals/']).toBe(99); + // Other defaults still present. + expect(merged['concepts/']).toBe(DEFAULT_SOURCE_BOOSTS['concepts/']); + }); + + test('env-only entries are added on top of defaults', () => { + const merged = resolveBoostMap('newprefix/:2.5'); + expect(merged['newprefix/']).toBe(2.5); + expect(merged['originals/']).toBe(DEFAULT_SOURCE_BOOSTS['originals/']); + }); +}); + +describe('resolveHardExcludes', () => { + test('returns defaults when nothing is overridden', () => { + const r = resolveHardExcludes(undefined, undefined, undefined); + for (const p of DEFAULT_HARD_EXCLUDES) expect(r).toContain(p); + }); + + test('caller exclude_slug_prefixes adds to the union', () => { + const r = resolveHardExcludes(['scratch/'], undefined, undefined); + expect(r).toContain('scratch/'); + expect(r).toContain('test/'); // default still present + }); + + test('include_slug_prefixes opts back in', () => { + const r = resolveHardExcludes(undefined, ['test/'], undefined); + expect(r).not.toContain('test/'); + // Other defaults still present. + expect(r).toContain('archive/'); + }); + + test('env GBRAIN_SEARCH_EXCLUDE adds to the union', () => { + const r = resolveHardExcludes(undefined, undefined, 'envdir/'); + expect(r).toContain('envdir/'); + }); + + test('include subtracts from env-supplied excludes too', () => { + const r = resolveHardExcludes(undefined, ['envdir/'], 'envdir/'); + expect(r).not.toContain('envdir/'); + }); +});