mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.22.0 feat: source-aware search ranking — curated pages win, swamp dampened (#439)
* feat(search): add exclude_slug_prefixes + include_slug_prefixes to SearchOpts The two new fields plumb prefix-based hard-exclude through the search API. exclude_slug_prefixes is additive over the engine's default hard-exclude set (test/, archive/, attachments/, .raw/) and the GBRAIN_SEARCH_EXCLUDE env var. include_slug_prefixes subtracts entries from the resolved set so callers can opt back into directories that are hidden by default. Stand-alone change — no engine wiring yet (lands in subsequent commits). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(search): source-boost + SQL ranking helpers (no engine wiring yet) Two new modules + unit tests. Pure functions, zero engine dependencies. source-boost.ts: - DEFAULT_SOURCE_BOOSTS map (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5, etc.) — grounded in the composition of the canonical brain. - DEFAULT_HARD_EXCLUDES = ['test/', 'archive/', 'attachments/', '.raw/']. - GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE env-var parsers, malformed entries skipped silently. - resolveBoostMap / resolveHardExcludes merge defaults + env + caller opts. sql-ranking.ts: - buildSourceFactorCase emits a CASE expression for the source factor. Returns literal '1.0' when detail==='high' so temporal queries bypass source-boost (matches the COMPILED_TRUTH_BOOST gate in hybrid.ts). Prefixes sorted by length desc so longest-match wins. - buildHardExcludeClause emits NOT (col LIKE 'p1%' OR col LIKE 'p2%'). NOT a NOT LIKE ALL/ANY array — those quantifiers don't express set-exclusion correctly for multi-pattern LIKE. - LIKE meta-character escape covers all three: %, _, AND \. Backslash coverage matters because it's Postgres LIKE's default escape char — a literal backslash in a user env prefix would otherwise be interpreted as 'escape the next char' and silently match wrong rows. - SQL string literals get single-quote doubling so injection-style inputs render as inert text inside the quoted string. 39 unit tests cover escape behavior, longest-prefix-match, detail-gate bypass, malformed env, factor=0 (legal), negative-factor rejection, SQL-injection-as-literal, and resolver merge semantics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(search): E2E coverage for source-boost, hard-exclude, engine parity search-swamp.test.ts: reproduces the v3-plan headline case. Seeds a curated originals/talks/article-outline-fat-code page against two wintermute/chat/ pages stuffed with 'fat code thin harness' repetitions. Asserts the article wins both keyword and vector ranking, and that detail=high lets the chat swamp re-surface (temporal-query workflow preserved). Also asserts source_id passes through the two-stage CTE. search-exclude.test.ts: verifies test/ + archive/ pages are hidden by default, that include_slug_prefixes opts back in, and that exclude_slug_prefixes adds to defaults. engine-parity.test.ts: codex flagged that searchKeyword's structural behavior differs between engines (Postgres ranks pages then picks best chunk; PGLite returns chunks directly). Without parity coverage the fix could pass on PGLite and silently fail on Postgres. Seeds identical corpus into both engines, runs identical queries, asserts top-result + result-set match. Includes a vector-search parity case and a hard-exclude parity case. Skips gracefully when DATABASE_URL is unset, per the CLAUDE.md E2E lifecycle pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(search): wire source-boost into v0.21.0 chunk-grain searchKeyword + searchKeywordChunks + two-stage searchVector Layers source-aware ranking on top of v0.21.0's Cathedral II chunk-grain FTS architecture, in both Postgres and PGLite engines. postgres-engine.ts: - searchKeyword (chunk-grain CTE → DISTINCT ON page dedup): the inner ranked_chunks CTE multiplies ts_rank by the source-factor CASE expression, hard-exclude prefixes (test/, archive/, attachments/, .raw/ by default + env + caller) become a NOT-LIKE OR-chain on the WHERE clause, language/symbol-kind filters preserved. - searchKeywordChunks (chunk-grain anchor primitive used by two-pass Layer 7): same source-boost treatment so the anchor pool that feeds two-pass retrieval is also dampened on chat/daily/x dirs. - searchVector becomes a two-stage CTE: inner CTE keeps pure HNSW ORDER BY (folding source-boost into it would force a sequential scan over every chunk), outer SELECT re-ranks by raw_score × source-factor. innerLimit scales with offset to preserve pagination contract. p.source_id passes through inner→outer for v0.18 multi-source callers. - All three methods stay inside sql.begin + SET LOCAL statement_timeout from v0.19+ (transaction-scoped GUC; bare SET leaks onto pooled connections, documented DoS vector). pglite-engine.ts: mirrors the same three methods. Same SQL shape, same source-factor + hard-exclude. Two-stage CTE also lifts stale-flag computation into the outer SELECT (it referenced p.updated_at which now lives only inside the inner CTE). Detail-gate (`detail !== 'high'`) inherited from buildSourceFactorCase ... temporal queries bypass source-boost so chat surfaces normally for date-framed lookups. Same gate pattern as the existing COMPILED_TRUTH_BOOST in hybrid.ts. Tests: 142 pass across pglite-engine, postgres-engine, sql-ranking, search-swamp E2E, search-exclude E2E. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.22.0 (rebased onto v0.21.0 master) CHANGELOG: new v0.22.0 entry above v0.21.0 (Cathedral II). Headline positions v0.22.0 as additive on top of v0.21.0's two-pass retrieval ... different mechanism, +3.3pts top-1 / -3.3pts swamp on the new Cat 13b benchmark in the sibling gbrain-evals repo. CLAUDE.md: - postgres-engine.ts entry mentions all three updated methods (searchKeyword, searchKeywordChunks, searchVector) and the two-stage CTE for searchVector specifically. - pglite-engine.ts entry parallels the Postgres notes. - src/core/search/ entry calls out source-aware ranking + hard-exclude defaults + detail-gate parity with COMPILED_TRUTH_BOOST. - Added entries for src/core/search/source-boost.ts and src/core/search/sql-ranking.ts in the Key Files section. - Added test/sql-ranking.test.ts and the three new E2E test files (search-swamp, search-exclude, engine-parity) to the test listings. README.md: SEARCH PIPELINE diagram in the "many strategies in concert" section gains two lines for source-aware ranking and hard-exclude filtering. VERSION: 0.21.0 → 0.22.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tests): typecheck + Postgres minions-shell env-var setup Two test fixes uncovered while running the full bun run test + E2E suite at zero defects. test/e2e/engine-parity.test.ts: BrainEngine was being imported from src/core/types.ts but it's actually exported from src/core/engine.ts; the import was silently working under bare `bun test` but failing typecheck. Fixed the import path and annotated 6 implicit-any SearchResult callbacks. (No behavior change ... typecheck only.) test/e2e/minions-shell.test.ts: the Postgres minions-shell test was missing the `GBRAIN_ALLOW_SHELL_JOBS=1` env-var setup that the PGLite sibling test in test/e2e/minions-shell-pglite.test.ts already has. Without it the shell handler short-circuits and the job lands in `dead`, not `completed`. The env var is the operator-trust gate for the shell handler ... separate from the trusted-add allowProtectedSubmit flag. Adding the same beforeAll/afterAll setup-and-restore pattern from the PGLite sibling brings the test to green. Both bugs were latent on master ... bare `bun test` skipped the typecheck and the minions-shell E2E was a pre-existing flake (documented as such in earlier branch summary). Verified: full unit suite 2714 pass / 0 fail (`bun run test`), full E2E suite 225 pass / 0 fail across 24 files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate llms-full.txt for v0.22.0 doc updates Picks up the v0.22.0 entries added to CLAUDE.md (source-boost.ts, sql-ranking.ts, three new E2E test files, postgres/pglite engine search-method updates). The build-llms.test.ts regen-drift guard was failing because the committed bundle didn't match the current generator output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(search): adversarial review fixes — detail loose-string + PGLite CTE alias Two FIXABLE findings from /ship's adversarial subagent pass: 1. **buildSourceFactorCase: tolerate loose-string `detail` over the MCP boundary.** TypeScript narrows the typed callers, but agents passing JSON across MCP can send `"HIGH"` (uppercase) or `"high "` (trailing space). Before this change, those values silently fell through the `detail === 'high'` strict-equality check and got boosted ranking instead of the temporal bypass — the opposite of what the agent asked for. Now the gate normalizes `String(detail).trim().toLowerCase()` before comparing. Three new test cases cover `"HIGH"`, `"high "`, and `" High "`. 2. **PGLite searchVector: alias the hnsw_candidates CTE as `hc` and qualify the correlated subquery.** The prior shape had `WHERE te.page_id = page_id` in the staleness subquery — unqualified `page_id` resolved by lexical-scope fallback to `hnsw_candidates.page_id`, but if the inner column is ever renamed or the parser changes, it would silently bind to `te.page_id` itself (always true) and every result returns `stale=true`. Aliasing the CTE as `hc` and qualifying both `hc.page_id` and `hc.slug` (via building the source-factor CASE with `'hc.slug'`) eliminates the ambiguity. Postgres `searchVector` was already safe — it uses `false AS stale` (no correlated subquery) — so no symmetric change needed there. Three INVESTIGATE findings deferred: - HNSW + hard-exclude planner behavior on real Postgres (needs EXPLAIN on a 50K+ chunk Supabase corpus, not reproducible on PGLite) - searchKeywordChunks pagination pool growth (would change the v0.21.0 contract; inherits the original Cathedral II shape) - resolveBoostMap re-reads process.env per call (cheap, intentional — enables mid-process env reload for tuning) Verified: 137 pass / 0 fail across sql-ranking + pglite-engine + search-swamp + search-exclude tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
f718c595b3
commit
172b55ba9d
@@ -2,6 +2,85 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.22.0] - 2026-04-25
|
||||
|
||||
**Search stops getting swamped by chat logs. Curated pages win by default.**
|
||||
|
||||
For the last few releases, multi-word topic queries against a real brain returned chat-log pages at #1 and #2 because chat pages are 50KB and contain mentions of every topic. The actual article you wrote about the topic ranked #5. v0.22.0 fixes that at the SQL layer ... ranking is now source-aware, curated directories outrank bulk content, and bookkeeping directories like `test/` and `archive/` never enter the candidate set.
|
||||
|
||||
The fix layers on top of v0.21.0's Cathedral II chunk-grain FTS and two-pass retrieval. Different mechanism, additive effect. Chat pages get dampened at the chunk-rank stage; curated content gets boosted; the two-pass walk and source-boost both run in the same pipeline. Temporal queries (`when`, `last week`, `YYYY-MM`) bypass the gate entirely so date-framed chat lookups still work. Two new env vars (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) tune per-deployment. `unset` them to revert to v0.21.0 ranking exactly.
|
||||
|
||||
Two SearchOpts additions plumb hard-exclude through the API: `exclude_slug_prefixes` (additive over defaults + env) and `include_slug_prefixes` (subtractive opt-back-in). The four default hard-excludes (`test/`, `archive/`, `attachments/`, `.raw/`) were silently polluting search results before.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
A new BrainBench category — **Cat 13b: Source Swamp Resistance** — ships in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo. The corpus is 20 pages: 10 short opinionated `originals/` pages and 10 long `wintermute/chat/` dumps that mention the same multi-word phrases at higher per-byte density. 30 hand-curated queries assert the curated page wins.
|
||||
|
||||
| gbrain version | Top-1 hit | Top-3 hit | Swamp@top |
|
||||
|--------------------------------------|-----------|-----------|-----------|
|
||||
| v0.20.4 (pre-Cathedral II) | 90.0% | 100.0% | 10.0% |
|
||||
| v0.21.0 (Cathedral II — two-pass) | 90.0% | 100.0% | 10.0% |
|
||||
| **v0.22.0 (this release)** | **93.3%** | **100.0%** | **6.7%** |
|
||||
|
||||
v0.21.0's two-pass retrieval is orthogonal to source-swamp resistance — it's about call-graph edges and parent-scope chunking, which doesn't reach the directory-level ranking signal that source-boost provides. v0.22.0 adds +3.3pts top-1 and -3.3pts swamp on top of v0.21.0.
|
||||
|
||||
The world-v1 corpus (BrainBench Cats 1+2 retrieval, 145 relational queries) is unchanged at P@5 49.1% / R@5 97.9% — every existing benchmark axis stays put within ±2pp tolerance.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If your brain's biggest directories are chat dumps, daily logs, or X archives, search just got dramatically better for the topic queries you actually run. If you depend on chat surfacing for date-framed questions ("what did we discuss last week"), nothing changed ... the intent classifier routes those to `detail=high` which bypasses source-boost. If you want a different boost map, set `GBRAIN_SOURCE_BOOST=originals/:1.8,wintermute/chat/:0.3` and ship.
|
||||
|
||||
## To take advantage of v0.22.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. No DB migration is needed ... the change is purely a SQL ranking refactor on existing tables.
|
||||
|
||||
1. **No manual migration step required.** The new ranking is on by default. Defaults are tuned for a brain with the canonical `originals/`, `concepts/`, `writing/`, `meetings/`, `daily/`, `media/x/`, `wintermute/chat/` shape.
|
||||
2. **Tune for your brain (optional):**
|
||||
```bash
|
||||
# Stronger originals boost, harder chat dampening
|
||||
export GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3"
|
||||
# Add a directory to the hard-exclude list
|
||||
export GBRAIN_SEARCH_EXCLUDE="scratch/,private/"
|
||||
```
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain search "<a multi-word topic phrase from your brain>"
|
||||
# Expect: curated content (originals/, concepts/, writing/) at the top.
|
||||
gbrain search "<phrase>" --detail high
|
||||
# Expect: source-boost bypassed; chat pages allowed back.
|
||||
```
|
||||
4. **Rollback one-liner** if something looks off:
|
||||
```bash
|
||||
unset GBRAIN_SOURCE_BOOST GBRAIN_SEARCH_EXCLUDE
|
||||
```
|
||||
Reverts ranking to v0.21.0 behavior exactly.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Source-aware retrieval
|
||||
|
||||
- New module `src/core/search/source-boost.ts` ships the default boost map (`originals/` 1.5, `concepts/` 1.3, `writing/` 1.4, `people/companies/deals/` 1.2, `daily/` 0.8, `media/x/` 0.7, `wintermute/chat/` 0.5) and the four default hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`). Both knobs override via env (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) or per-call SearchOpts.
|
||||
- New module `src/core/search/sql-ranking.ts` is a pair of pure SQL-fragment builders shared between Postgres and PGLite engines. `buildSourceFactorCase` emits a longest-prefix-match CASE expression and returns literal `'1.0'` when `detail === 'high'` so temporal queries bypass source-boost. `buildHardExcludeClause` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` ... OR-chain wrapped in NOT, never `NOT LIKE ALL/ANY` (those don't express set-exclusion). LIKE meta-character escape covers `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling renders SQL-injection-style inputs inert.
|
||||
- `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` ... three methods wired: `searchKeyword` (chunk-grain CTE → DISTINCT ON page dedup, multiplies ts_rank by source-factor), `searchKeywordChunks` (the chunk-grain anchor primitive used by Cathedral II two-pass retrieval, also gets source-boost so the anchor pool is dampened on chat dirs), and `searchVector` (becomes a two-stage CTE: pure-distance HNSW inner ORDER BY, source-boost re-rank in outer SELECT, innerLimit scales with offset to preserve pagination).
|
||||
- `src/core/types.ts` ... SearchOpts gains two fields: `exclude_slug_prefixes?: string[]` (additive over defaults + env) and `include_slug_prefixes?: string[]` (subtractive opt-back-in).
|
||||
|
||||
#### Tests
|
||||
|
||||
- `test/sql-ranking.test.ts` ... 39 unit cases covering longest-prefix-match, detail=high temporal-bypass, three-meta-char LIKE escape, single-quote SQL-literal doubling, env-var parsing, resolver merge semantics.
|
||||
- `test/e2e/search-swamp.test.ts` ... reproduces the headline case in PGLite. Curated article competes with two chat pages stuffed with the same multi-word phrase. Asserts article wins both keyword and vector ranking, detail=high lets chat re-surface, source_id passes through two-stage CTE.
|
||||
- `test/e2e/search-exclude.test.ts` ... verifies test/ + archive/ pages hidden by default, include_slug_prefixes opts back in, exclude_slug_prefixes adds to defaults.
|
||||
- `test/e2e/engine-parity.test.ts` ... Postgres ↔ PGLite top-result + result-set parity for both search methods plus a hard-exclude parity case. Skips gracefully when DATABASE_URL is unset.
|
||||
|
||||
#### Won't break what was already working
|
||||
|
||||
The change is additive at the SQL layer; no `hybrid.ts`, `intent.ts`, `dedup.ts`, `expansion.ts`, `two-pass.ts`, or operations-layer changes. RRF fusion, compiled-truth boost, backlink boost, multi-query expansion, source-aware dedup, and v0.21.0's Cathedral II two-pass retrieval all run unchanged downstream of the new ranking. The `sql.begin` + `SET LOCAL statement_timeout` v0.19 wrap is preserved (transaction-scoped GUC; bare SET would leak onto pooled connections, documented DoS vector). RLS-enabled brains still work because both inner and outer CTE SELECTs are subject to row-level policies.
|
||||
|
||||
### For contributors
|
||||
|
||||
- The two new helpers are pure functions with explicit params and zero engine dependencies. Both engines call them to build identical SQL. Useful pattern for any future SQL-side ranking signal that needs to land in both Postgres and PGLite.
|
||||
- The two-stage CTE pattern (HNSW-safe pure-distance inner ORDER BY, re-rank in outer SELECT) is the right shape for any future per-prefix or per-page boost in vector search. Folding extra factors into the outer ORDER BY keeps the index usable.
|
||||
- BrainBench Cat 13b lives in [gbrain-evals](https://github.com/garrytan/gbrain-evals) on `feat/cat13b-source-swamp` ... 20-page corpus + 30 hand-curated queries. Companion PR.
|
||||
|
||||
## [0.21.0] - 2026-04-25
|
||||
|
||||
## **Your brain walks the code graph now.**
|
||||
|
||||
@@ -25,9 +25,9 @@ strict behavior when unset.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
@@ -40,9 +40,11 @@ strict behavior when unset.
|
||||
- `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. As of v0.22.0, `searchKeyword` / `searchKeywordChunks` / `searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `wintermute/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally.
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level)
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
@@ -233,6 +235,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/sql-ranking.test.ts` (v0.22.0 source-boost helpers: 39 cases covering longest-prefix-match in SQL CASE, detail=high temporal-bypass, three-meta-char LIKE escape (%, _, \\), single-quote SQL-literal doubling, env override parsing for GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE, resolveBoostMap / resolveHardExcludes merge semantics),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
`test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels),
|
||||
@@ -281,6 +284,9 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered.
|
||||
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
|
||||
@@ -505,6 +505,8 @@ Question
|
||||
│ ├─ Multi-query expansion (Haiku rephrases the question 3 ways)
|
||||
│ ├─ Vector search (HNSW cosine over OpenAI embeddings)
|
||||
│ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery)
|
||||
│ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer)
|
||||
│ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval)
|
||||
│ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both)
|
||||
│ ├─ Cosine re-scoring (re-rank chunks against actual query embedding)
|
||||
│ ├─ Compiled-truth boost (assessments outrank timeline noise)
|
||||
|
||||
+11
-3
@@ -104,9 +104,9 @@ strict behavior when unset.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
@@ -119,9 +119,11 @@ strict behavior when unset.
|
||||
- `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. As of v0.22.0, `searchKeyword` / `searchKeywordChunks` / `searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `wintermute/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally.
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level)
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
@@ -312,6 +314,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/sql-ranking.test.ts` (v0.22.0 source-boost helpers: 39 cases covering longest-prefix-match in SQL CASE, detail=high temporal-bypass, three-meta-char LIKE escape (%, _, \\), single-quote SQL-literal doubling, env override parsing for GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE, resolveBoostMap / resolveHardExcludes merge semantics),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
`test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels),
|
||||
@@ -360,6 +363,9 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered.
|
||||
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
@@ -1710,6 +1716,8 @@ Question
|
||||
│ ├─ Multi-query expansion (Haiku rephrases the question 3 ways)
|
||||
│ ├─ Vector search (HNSW cosine over OpenAI embeddings)
|
||||
│ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery)
|
||||
│ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer)
|
||||
│ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval)
|
||||
│ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both)
|
||||
│ ├─ Cosine re-scoring (re-rank chunks against actual query embedding)
|
||||
│ ├─ Compiled-truth boost (assessments outrank timeline noise)
|
||||
|
||||
+58
-18
@@ -20,6 +20,8 @@ import type {
|
||||
EngineConfig,
|
||||
} from './types.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
|
||||
|
||||
type PGLiteDB = PGlite;
|
||||
|
||||
@@ -239,6 +241,12 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// Fetch 3x to give dedup headroom, then page-dedup + re-limit.
|
||||
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
|
||||
|
||||
// Source-aware ranking (v0.22): see postgres-engine.ts for rationale.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
// v0.20.0 Cathedral II Layer 10 C1/C2: language + symbol-kind filters.
|
||||
const params: unknown[] = [query, innerLimit, limit, offset];
|
||||
let extraFilter = '';
|
||||
@@ -256,13 +264,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2
|
||||
),
|
||||
@@ -301,6 +309,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
// Source-aware ranking applied here too — searchKeywordChunks is the
|
||||
// chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
const params: unknown[] = [query, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.language) {
|
||||
@@ -316,13 +331,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2 OFFSET $3`,
|
||||
params
|
||||
@@ -341,7 +356,23 @@ export class PGLiteEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
const params: unknown[] = [vecStr, limit, offset];
|
||||
// Two-stage CTE (v0.22): pure-distance ORDER BY in inner CTE preserves
|
||||
// HNSW; outer SELECT re-ranks by raw_score * source_factor over the
|
||||
// narrow candidate pool. innerLimit scales with offset to preserve the
|
||||
// pagination contract. See postgres-engine.ts searchVector for rationale.
|
||||
const boostMap = resolveBoostMap();
|
||||
// Outer SELECT references the aliased CTE column. Aliasing the CTE as `hc`
|
||||
// disambiguates the correlated subquery (`te.page_id = hc.page_id`) from
|
||||
// the inner column. Without the alias, an unqualified `page_id` in the
|
||||
// subquery's WHERE would lexically resolve back to `te.page_id` itself
|
||||
// and degrade to `te.page_id = te.page_id` (always true), making every
|
||||
// result stale=true. Codex caught this in adversarial review.
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('hc.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const innerLimit = offset + Math.max(limit * 5, 100);
|
||||
|
||||
const params: unknown[] = [vecStr, innerLimit, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.language) {
|
||||
params.push(opts.language);
|
||||
@@ -353,19 +384,28 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.embedding <=> $1::vector) AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT $2
|
||||
OFFSET $3`,
|
||||
`WITH hnsw_candidates AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.embedding <=> $1::vector) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT $2
|
||||
)
|
||||
SELECT
|
||||
hc.slug, hc.page_id, hc.title, hc.type, hc.source_id,
|
||||
hc.chunk_id, hc.chunk_index, hc.chunk_text, hc.chunk_source,
|
||||
hc.raw_score * ${sourceFactorCaseOnSlug} AS score,
|
||||
CASE WHEN hc.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = hc.page_id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM hnsw_candidates hc
|
||||
ORDER BY score DESC
|
||||
LIMIT $3
|
||||
OFFSET $4`,
|
||||
params
|
||||
);
|
||||
|
||||
|
||||
+201
-78
@@ -18,6 +18,8 @@ import type {
|
||||
import { GBrainError } from './types.ts';
|
||||
import * as db from './db.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
|
||||
|
||||
export class PostgresEngine implements BrainEngine {
|
||||
readonly kind = 'postgres' as const;
|
||||
@@ -236,51 +238,83 @@ export class PostgresEngine implements BrainEngine {
|
||||
// ship < limit pages. 3x gives dedup enough to pick top N distinct pages.
|
||||
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
|
||||
|
||||
// Search-only timeout: prevents DoS via expensive queries without
|
||||
// affecting long-running operations like embed --all or bulk import.
|
||||
// SET LOCAL inside sql.begin() scopes the GUC to the transaction so
|
||||
// it can never leak onto a pooled connection returned to other
|
||||
// callers. A bare `SET statement_timeout` goes to an arbitrary
|
||||
// connection from the pool, lives past this method, and either
|
||||
// clips an unrelated caller's long-running query (DoS) or — via
|
||||
// `SET statement_timeout = 0` — disables the guard for them.
|
||||
// Source-aware ranking (v0.22): boost curated content (originals/,
|
||||
// concepts/, writing/) and dampen bulk content (chat/, daily/, media/x/)
|
||||
// by multiplying the chunk-grain ts_rank with a source-factor CASE.
|
||||
// Detail-gated — disabled for `detail='high'` (temporal queries) so
|
||||
// chat surfaces normally for date-framed lookups. Hard-exclude prefixes
|
||||
// (test/, archive/, attachments/, .raw/ by default) filter at the
|
||||
// chunk-rank stage so they never enter the candidate set.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
const params: unknown[] = [query];
|
||||
let typeClause = '';
|
||||
if (type) {
|
||||
params.push(type);
|
||||
typeClause = `AND p.type = $${params.length}`;
|
||||
}
|
||||
let excludeSlugsClause = '';
|
||||
if (excludeSlugs?.length) {
|
||||
params.push(excludeSlugs);
|
||||
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
let languageClause = '';
|
||||
if (language) {
|
||||
params.push(language);
|
||||
languageClause = `AND cc.language = $${params.length}`;
|
||||
}
|
||||
let symbolKindClause = '';
|
||||
if (symbolKind) {
|
||||
params.push(symbolKind);
|
||||
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
params.push(innerLimit);
|
||||
const innerLimitParam = `$${params.length}`;
|
||||
params.push(limit);
|
||||
const limitParam = `$${params.length}`;
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
const rawQuery = `
|
||||
WITH ranked_chunks AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${innerLimitParam}
|
||||
),
|
||||
best_per_page AS (
|
||||
SELECT DISTINCT ON (slug) *
|
||||
FROM ranked_chunks
|
||||
ORDER BY slug, score DESC
|
||||
)
|
||||
SELECT slug, page_id, title, type, source_id,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source, score,
|
||||
false AS stale
|
||||
FROM best_per_page
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limitParam}
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
// Search-only timeout. SET LOCAL inside sql.begin() scopes the GUC
|
||||
// to the transaction so it can never leak onto a pooled connection.
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
// CTE chain: rank chunks by FTS → DISTINCT ON (slug) to pick best
|
||||
// chunk per page → order by score → limit. The external shape is
|
||||
// page-grain; chunk-grain ranking wins because A4 weights mean
|
||||
// doc-comment hits (and, once Layer 5 populates them, qualified
|
||||
// symbol hits) beat body-text hits at the chunk level.
|
||||
return await sql`
|
||||
WITH ranked_chunks AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query})
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${innerLimit}
|
||||
),
|
||||
best_per_page AS (
|
||||
SELECT DISTINCT ON (slug) *
|
||||
FROM ranked_chunks
|
||||
ORDER BY slug, score DESC
|
||||
)
|
||||
SELECT slug, page_id, title, type, source_id,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source, score,
|
||||
false AS stale
|
||||
FROM best_per_page
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
@@ -308,26 +342,64 @@ export class PostgresEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
// Source-aware ranking applies here too — searchKeywordChunks is the
|
||||
// chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses,
|
||||
// so curated-vs-bulk dampening should affect the anchor pool. Same
|
||||
// detail-gate, same hard-exclude behavior as searchKeyword.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
const params: unknown[] = [query];
|
||||
let typeClause = '';
|
||||
if (type) {
|
||||
params.push(type);
|
||||
typeClause = `AND p.type = $${params.length}`;
|
||||
}
|
||||
let excludeSlugsClause = '';
|
||||
if (excludeSlugs?.length) {
|
||||
params.push(excludeSlugs);
|
||||
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
let languageClause = '';
|
||||
if (language) {
|
||||
params.push(language);
|
||||
languageClause = `AND cc.language = $${params.length}`;
|
||||
}
|
||||
let symbolKindClause = '';
|
||||
if (symbolKind) {
|
||||
params.push(symbolKind);
|
||||
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
params.push(limit);
|
||||
const limitParam = `$${params.length}`;
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
const rawQuery = `
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limitParam}
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql`
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query})
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
@@ -348,29 +420,80 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
const vecStr = '[' + Array.from(embedding).join(',') + ']';
|
||||
|
||||
// Search-only timeout (see searchKeyword for rationale). SET LOCAL +
|
||||
// sql.begin ensures the GUC stays transaction-scoped on the pooled
|
||||
// connection.
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql`
|
||||
// Two-stage CTE (v0.22): inner CTE keeps a pure-distance ORDER BY so
|
||||
// the HNSW index stays usable. Folding source-boost into the inner
|
||||
// ORDER BY would force a sequential scan over every chunk (seconds vs
|
||||
// ~10ms with HNSW). Outer SELECT re-ranks the candidate pool by
|
||||
// raw_score * source_factor.
|
||||
//
|
||||
// innerLimit scales with offset to preserve the pagination contract:
|
||||
// a fixed cap of 100 would silently empty offset > 100.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const innerLimit = offset + Math.max(limit * 5, 100);
|
||||
|
||||
const params: unknown[] = [vecStr];
|
||||
let typeClause = '';
|
||||
if (type) {
|
||||
params.push(type);
|
||||
typeClause = `AND p.type = $${params.length}`;
|
||||
}
|
||||
let excludeSlugsClause = '';
|
||||
if (excludeSlugs?.length) {
|
||||
params.push(excludeSlugs);
|
||||
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
let languageClause = '';
|
||||
if (language) {
|
||||
params.push(language);
|
||||
languageClause = `AND cc.language = $${params.length}`;
|
||||
}
|
||||
let symbolKindClause = '';
|
||||
if (symbolKind) {
|
||||
params.push(symbolKind);
|
||||
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
params.push(innerLimit);
|
||||
const innerLimitParam = `$${params.length}`;
|
||||
params.push(limit);
|
||||
const limitParam = `$${params.length}`;
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
const rawQuery = `
|
||||
WITH hnsw_candidates AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.embedding <=> ${vecStr}::vector) AS score,
|
||||
false AS stale
|
||||
1 - (cc.embedding <=> $1::vector) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY cc.embedding <=> ${vecStr}::vector
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT ${innerLimitParam}
|
||||
)
|
||||
SELECT
|
||||
slug, page_id, title, type, source_id,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source,
|
||||
raw_score * ${sourceFactorCaseOnSlug} AS score,
|
||||
false AS stale
|
||||
FROM hnsw_candidates
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limitParam}
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
/**
|
||||
* Source-Type Boost Map
|
||||
*
|
||||
* Multiplies into ts_rank / vector cosine score at SQL build time so that
|
||||
* curated content (originals/, concepts/, writing/) outranks bulk content
|
||||
* (wintermute/chat/, daily/, media/x/) for non-temporal queries.
|
||||
*
|
||||
* Keyed by slug prefix. Longest-prefix-match wins (sorted at lookup time
|
||||
* inside sql-ranking.ts). Defaults grounded in the composition of the
|
||||
* canonical brain at ~/git/brain/.
|
||||
*
|
||||
* Override via env: GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3"
|
||||
* Hard-exclude via env: GBRAIN_SEARCH_EXCLUDE="test/,scratch/"
|
||||
*/
|
||||
|
||||
export const DEFAULT_SOURCE_BOOSTS: Record<string, number> = {
|
||||
// Curated, opinionated, high-signal — Garry's own writing
|
||||
'originals/': 1.5,
|
||||
// Reusable knowledge frameworks
|
||||
'concepts/': 1.3,
|
||||
// Long-form essays / articles
|
||||
'writing/': 1.4,
|
||||
// Entity pages
|
||||
'people/': 1.2,
|
||||
'companies/': 1.2,
|
||||
'deals/': 1.2,
|
||||
// Notes from real meetings
|
||||
'meetings/': 1.1,
|
||||
// Ingested third-party content
|
||||
'media/articles/': 1.1,
|
||||
'media/repos/': 1.1,
|
||||
// Neutral baselines (explicit for clarity)
|
||||
'yc/': 1.0,
|
||||
'civic/': 1.0,
|
||||
// Bulk / noisy
|
||||
'daily/': 0.8,
|
||||
'media/x/': 0.7,
|
||||
// Chat transcripts — massive, noisy, swamp keyword queries
|
||||
'wintermute/chat/': 0.5,
|
||||
};
|
||||
|
||||
/**
|
||||
* Hard-excludes — slug prefixes that should never enter search results
|
||||
* (unless explicitly opted-in via include_slug_prefixes).
|
||||
*/
|
||||
export const DEFAULT_HARD_EXCLUDES: string[] = [
|
||||
'test/',
|
||||
'archive/',
|
||||
'attachments/',
|
||||
'.raw/',
|
||||
];
|
||||
|
||||
/**
|
||||
* Parse GBRAIN_SOURCE_BOOST env var.
|
||||
* Format: comma-separated prefix:factor pairs.
|
||||
* Example: "originals/:1.8,wintermute/chat/:0.3"
|
||||
*
|
||||
* Malformed entries are skipped silently. Returns empty object if env is
|
||||
* unset or unparseable in its entirety.
|
||||
*/
|
||||
export function parseSourceBoostEnv(env: string | undefined): Record<string, number> {
|
||||
if (!env) return {};
|
||||
const out: Record<string, number> = {};
|
||||
for (const pair of env.split(',')) {
|
||||
const idx = pair.lastIndexOf(':');
|
||||
if (idx <= 0) continue;
|
||||
const prefix = pair.slice(0, idx).trim();
|
||||
const factor = Number.parseFloat(pair.slice(idx + 1).trim());
|
||||
if (!prefix || !Number.isFinite(factor) || factor < 0) continue;
|
||||
out[prefix] = factor;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse GBRAIN_SEARCH_EXCLUDE env var.
|
||||
* Format: comma-separated slug prefixes.
|
||||
* Example: "test/,scratch/,private/"
|
||||
*
|
||||
* Blank entries skipped. Returns empty array if env is unset.
|
||||
*/
|
||||
export function parseHardExcludesEnv(env: string | undefined): string[] {
|
||||
if (!env) return [];
|
||||
return env.split(',').map(s => s.trim()).filter(s => s.length > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the effective boost map by merging defaults with env override.
|
||||
* Env entries override defaults (shallow merge); env-only entries are added.
|
||||
*/
|
||||
export function resolveBoostMap(
|
||||
envValue: string | undefined = process.env.GBRAIN_SOURCE_BOOST,
|
||||
): Record<string, number> {
|
||||
const override = parseSourceBoostEnv(envValue);
|
||||
return { ...DEFAULT_SOURCE_BOOSTS, ...override };
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the effective hard-exclude prefix list.
|
||||
*
|
||||
* - Defaults union with env-supplied excludes
|
||||
* - Subtract any caller-supplied include_slug_prefixes (opt-back-in)
|
||||
* - Caller-supplied exclude_slug_prefixes adds to the union
|
||||
*/
|
||||
export function resolveHardExcludes(
|
||||
excludeOpt?: string[],
|
||||
includeOpt?: string[],
|
||||
envValue: string | undefined = process.env.GBRAIN_SEARCH_EXCLUDE,
|
||||
): string[] {
|
||||
const envExcludes = parseHardExcludesEnv(envValue);
|
||||
const union = new Set<string>([...DEFAULT_HARD_EXCLUDES, ...envExcludes, ...(excludeOpt ?? [])]);
|
||||
if (includeOpt?.length) {
|
||||
for (const p of includeOpt) union.delete(p);
|
||||
}
|
||||
return Array.from(union);
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* SQL Ranking Builders
|
||||
*
|
||||
* Pure string builders for the source-aware ranking signal that both
|
||||
* postgres-engine and pglite-engine inject into searchKeyword / searchVector.
|
||||
*
|
||||
* Returns RAW SQL FRAGMENTS. Call sites must embed via the engine's "unsafe"
|
||||
* SQL tag (`sql.unsafe(fragment)` for postgres.js, equivalent for pglite).
|
||||
*
|
||||
* Inputs to these builders that originate from env vars or caller options
|
||||
* (slug prefixes) are LIKE-pattern-escaped (`%`, `_`, `\`) AND SQL-string
|
||||
* escaped (single-quote doubling) before inlining. The slugColumn parameter
|
||||
* is supplied by us at the call site and is never user-controllable.
|
||||
*
|
||||
* Numeric factors come from `parseSourceBoostEnv` which calls Number.parseFloat
|
||||
* and validates `Number.isFinite(factor) && factor >= 0`, so they're safe to
|
||||
* inline as bare literals.
|
||||
*/
|
||||
|
||||
/** Escape `%`, `_`, and `\` so a string can be used as a LIKE prefix literal. */
|
||||
function escapeLikePattern(s: string): string {
|
||||
return s.replace(/[%_\\]/g, '\\$&');
|
||||
}
|
||||
|
||||
/** Escape a SQL string literal: replace single-quote with two single-quotes. */
|
||||
function escapeSqlLiteral(s: string): string {
|
||||
return s.replace(/'/g, "''");
|
||||
}
|
||||
|
||||
/** Escape a slug prefix for use as `LIKE 'prefix%'` (both LIKE-escape and SQL-escape). */
|
||||
function buildLikePrefixLiteral(prefix: string): string {
|
||||
return `'${escapeSqlLiteral(escapeLikePattern(prefix))}%'`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a CASE expression that returns the source-boost factor for a slug.
|
||||
*
|
||||
* Returns a literal `'1.0'` when `detail === 'high'` so temporal queries
|
||||
* bypass source-boost entirely (mirrors the existing COMPILED_TRUTH_BOOST
|
||||
* gate in hybrid.ts).
|
||||
*
|
||||
* Prefixes are sorted by length descending so longest-match wins:
|
||||
* `media/articles/` (1.1) wins over `media/x/` (0.7) without caller-order
|
||||
* dependencies.
|
||||
*
|
||||
* @param slugColumn — qualified column reference (e.g. `'p.slug'`). MUST be
|
||||
* supplied by the engine, never from user input.
|
||||
* @param boostMap — prefix → factor map (defaults merged with env override)
|
||||
* @param detail — query detail level; `'high'` disables source-boost
|
||||
*
|
||||
* @returns raw SQL fragment, e.g. `(CASE WHEN p.slug LIKE 'originals/%' THEN 1.5 ... ELSE 1.0 END)`
|
||||
*/
|
||||
export function buildSourceFactorCase(
|
||||
slugColumn: string,
|
||||
boostMap: Record<string, number>,
|
||||
detail: 'low' | 'medium' | 'high' | undefined,
|
||||
): string {
|
||||
// Loose-string guard: agents passing `"HIGH"` or `"high "` over MCP/JSON
|
||||
// should still hit the temporal-bypass path. TypeScript narrows `detail`
|
||||
// for typed callers; this guard catches the untyped boundary.
|
||||
const normalized = typeof detail === 'string' ? detail.trim().toLowerCase() : detail;
|
||||
if (normalized === 'high') return '1.0';
|
||||
|
||||
const entries = Object.entries(boostMap)
|
||||
.filter(([prefix, factor]) => prefix.length > 0 && Number.isFinite(factor) && factor >= 0)
|
||||
.sort((a, b) => b[0].length - a[0].length); // longest-prefix-match wins
|
||||
|
||||
if (entries.length === 0) return '1.0';
|
||||
|
||||
const whens = entries.map(([prefix, factor]) =>
|
||||
`WHEN ${slugColumn} LIKE ${buildLikePrefixLiteral(prefix)} THEN ${factor}`
|
||||
).join(' ');
|
||||
|
||||
return `(CASE ${whens} ELSE 1.0 END)`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `NOT (col LIKE 'p1%' OR col LIKE 'p2%' OR ...)` exclusion clause.
|
||||
*
|
||||
* Why OR-chain wrapped in NOT, not `NOT LIKE ALL/ANY(array)`:
|
||||
* - `NOT LIKE ALL(array)` means "doesn't match every pattern" — still
|
||||
* keeps rows that match one. Wrong for set-exclusion.
|
||||
* - `NOT LIKE ANY(array)` is non-standard and behavior varies.
|
||||
* - Boolean-friendly OR-chain wrapped in NOT is unambiguous and indexable.
|
||||
*
|
||||
* Returns empty string when prefixes is empty, so callers can interpolate
|
||||
* unconditionally with a leading `AND`.
|
||||
*
|
||||
* @param slugColumn — qualified column reference (engine-supplied, trusted)
|
||||
* @param prefixes — list of slug prefixes to exclude (env + caller-supplied; escaped)
|
||||
*
|
||||
* @returns raw SQL fragment (with leading space) or empty string
|
||||
*/
|
||||
export function buildHardExcludeClause(slugColumn: string, prefixes: string[]): string {
|
||||
if (!prefixes.length) return '';
|
||||
const likes = prefixes
|
||||
.filter(p => p.length > 0)
|
||||
.map(p => `${slugColumn} LIKE ${buildLikePrefixLiteral(p)}`)
|
||||
.join(' OR ');
|
||||
if (!likes) return '';
|
||||
return `AND NOT (${likes})`;
|
||||
}
|
||||
|
||||
// Exported for unit tests
|
||||
export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral };
|
||||
@@ -122,6 +122,18 @@ export interface SearchOpts {
|
||||
offset?: number;
|
||||
type?: PageType;
|
||||
exclude_slugs?: string[];
|
||||
/**
|
||||
* Slug-prefix excludes — additive over DEFAULT_HARD_EXCLUDES (test/, archive/,
|
||||
* attachments/, .raw/) and the GBRAIN_SEARCH_EXCLUDE env var. Stacks with
|
||||
* `exclude_slugs` (exact match) — a row is filtered if it matches either set.
|
||||
*/
|
||||
exclude_slug_prefixes?: string[];
|
||||
/**
|
||||
* Opt-back-in list — subtracts entries from the resolved hard-exclude set.
|
||||
* E.g. `include_slug_prefixes: ['test/']` lets a query see test/ pages even
|
||||
* though they're hard-excluded by default.
|
||||
*/
|
||||
include_slug_prefixes?: string[];
|
||||
detail?: 'low' | 'medium' | 'high';
|
||||
/**
|
||||
* v0.20.0 Cathedral II: filter by content_chunks.language (e.g., 'typescript',
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
/**
|
||||
* Engine Parity E2E
|
||||
*
|
||||
* Codex flagged that searchKeyword behavior differs structurally between
|
||||
* the two engines (Postgres uses a CTE that ranks pages then picks best
|
||||
* chunk; PGLite returns chunks directly). Without verification, source-aware
|
||||
* ranking could pass on PGLite and silently fail on Postgres.
|
||||
*
|
||||
* Strategy: seed identical corpora into both engines, run identical queries,
|
||||
* assert top-5 slug ordering matches.
|
||||
*
|
||||
* Gated by DATABASE_URL — skips gracefully if no real Postgres. Always runs
|
||||
* the PGLite half so the seed/query path is at least exercised.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { ChunkInput, SearchResult } from '../../src/core/types.ts';
|
||||
import type { BrainEngine } from '../../src/core/engine.ts';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
|
||||
const SKIP_PG = !hasDatabase();
|
||||
const describeBoth = SKIP_PG ? describe.skip : describe;
|
||||
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
interface SeedPage {
|
||||
slug: string;
|
||||
type: 'writing' | 'concept' | 'note' | 'person' | 'company';
|
||||
title: string;
|
||||
body: string;
|
||||
embeddingDim: number;
|
||||
}
|
||||
|
||||
const SEED_PAGES: SeedPage[] = [
|
||||
{
|
||||
slug: 'originals/talks/article-outline-fat-code',
|
||||
type: 'writing',
|
||||
title: 'Fat Code Thin Harness — Part 3',
|
||||
body: 'fat code thin harness pattern part 3 production case studies',
|
||||
embeddingDim: 7,
|
||||
},
|
||||
{
|
||||
slug: 'concepts/fat-code-thin-harness',
|
||||
type: 'concept',
|
||||
title: 'Fat Code Thin Harness',
|
||||
body: 'reusable concept fat code thin harness architecture',
|
||||
embeddingDim: 14,
|
||||
},
|
||||
{
|
||||
slug: 'wintermute/chat/2026-04-15',
|
||||
type: 'note',
|
||||
title: '2026-04-15 chat',
|
||||
body:
|
||||
'fat code thin harness fat code thin harness discussion went on at length, ' +
|
||||
'fat code thin harness came up again and again, fat code thin harness fat code thin harness.',
|
||||
embeddingDim: 8,
|
||||
},
|
||||
{
|
||||
slug: 'wintermute/chat/2026-04-16',
|
||||
type: 'note',
|
||||
title: '2026-04-16 chat',
|
||||
body:
|
||||
'fat code thin harness once more, fat code thin harness fat code thin harness, ' +
|
||||
'still talking about fat code thin harness fat code thin harness.',
|
||||
embeddingDim: 9,
|
||||
},
|
||||
{
|
||||
slug: 'people/example-founder',
|
||||
type: 'person',
|
||||
title: 'Example Founder',
|
||||
body: 'example founder unrelated content for distraction',
|
||||
embeddingDim: 50,
|
||||
},
|
||||
];
|
||||
|
||||
async function seedEngine(eng: BrainEngine) {
|
||||
for (const p of SEED_PAGES) {
|
||||
await eng.putPage(p.slug, {
|
||||
type: p.type,
|
||||
title: p.title,
|
||||
compiled_truth: p.body,
|
||||
timeline: '',
|
||||
});
|
||||
const chunks: ChunkInput[] = [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: p.body,
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(p.embeddingDim),
|
||||
token_count: p.body.split(/\s+/).length,
|
||||
},
|
||||
];
|
||||
await eng.upsertChunks(p.slug, chunks);
|
||||
}
|
||||
}
|
||||
|
||||
const QUERIES = [
|
||||
'fat code thin harness',
|
||||
'fat code thin harness part 3',
|
||||
'fat code production',
|
||||
];
|
||||
|
||||
describeBoth('Engine parity — Postgres vs PGLite', () => {
|
||||
let pgEngine: BrainEngine;
|
||||
let pgliteEngine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
pgEngine = await setupDB();
|
||||
await seedEngine(pgEngine);
|
||||
|
||||
pgliteEngine = new PGLiteEngine();
|
||||
await pgliteEngine.connect({});
|
||||
await pgliteEngine.initSchema();
|
||||
await seedEngine(pgliteEngine);
|
||||
}, 90_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await pgliteEngine.disconnect();
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
for (const q of QUERIES) {
|
||||
test(`searchKeyword: top-5 slugs match for "${q}"`, async () => {
|
||||
const pgResults = await pgEngine.searchKeyword(q, { limit: 5 });
|
||||
const pgliteResults = await pgliteEngine.searchKeyword(q, { limit: 5 });
|
||||
|
||||
const pgSlugs = pgResults.map((r: SearchResult) => r.slug);
|
||||
const pgliteSlugs = pgliteResults.map((r: SearchResult) => r.slug);
|
||||
|
||||
// Top result MUST match (the swamp-resistance guarantee).
|
||||
expect(pgSlugs[0]).toBe(pgliteSlugs[0]);
|
||||
// Sets should match (allowing some ordering drift on lower-ranked
|
||||
// results since FTS rank function differences between engines are
|
||||
// out of scope for this fix).
|
||||
expect(new Set(pgSlugs)).toEqual(new Set(pgliteSlugs));
|
||||
});
|
||||
}
|
||||
|
||||
test('searchVector: top result matches between engines', async () => {
|
||||
const queryVec = basisEmbedding(7); // article direction
|
||||
const pgResults = await pgEngine.searchVector(queryVec, { limit: 5 });
|
||||
const pgliteResults = await pgliteEngine.searchVector(queryVec, { limit: 5 });
|
||||
|
||||
expect(pgResults[0]?.slug).toBe(pgliteResults[0]?.slug);
|
||||
});
|
||||
|
||||
test('hard-exclude is consistent across engines', async () => {
|
||||
// Both engines should hide test/ pages by default; both should opt
|
||||
// them back in via include_slug_prefixes.
|
||||
await pgEngine.putPage('test/parity-fixture', {
|
||||
type: 'note',
|
||||
title: 'parity test fixture',
|
||||
compiled_truth: 'parity test fixture content',
|
||||
timeline: '',
|
||||
});
|
||||
await pgEngine.upsertChunks('test/parity-fixture', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'parity test fixture content',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(20),
|
||||
token_count: 5,
|
||||
}] satisfies ChunkInput[]);
|
||||
|
||||
await pgliteEngine.putPage('test/parity-fixture', {
|
||||
type: 'note',
|
||||
title: 'parity test fixture',
|
||||
compiled_truth: 'parity test fixture content',
|
||||
timeline: '',
|
||||
});
|
||||
await pgliteEngine.upsertChunks('test/parity-fixture', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'parity test fixture content',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(20),
|
||||
token_count: 5,
|
||||
}] satisfies ChunkInput[]);
|
||||
|
||||
const pgDefault = await pgEngine.searchKeyword('parity test fixture');
|
||||
const pgliteDefault = await pgliteEngine.searchKeyword('parity test fixture');
|
||||
expect(pgDefault.map((r: SearchResult) => r.slug)).not.toContain('test/parity-fixture');
|
||||
expect(pgliteDefault.map((r: SearchResult) => r.slug)).not.toContain('test/parity-fixture');
|
||||
|
||||
const pgOptIn = await pgEngine.searchKeyword('parity test fixture', {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
const pgliteOptIn = await pgliteEngine.searchKeyword('parity test fixture', {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
expect(pgOptIn.map((r: SearchResult) => r.slug)).toContain('test/parity-fixture');
|
||||
expect(pgliteOptIn.map((r: SearchResult) => r.slug)).toContain('test/parity-fixture');
|
||||
});
|
||||
|
||||
test('detail=high produces a different ranking than default on at least one engine', async () => {
|
||||
// Source-boost gates on `detail !== 'high'`. If the gate works on both
|
||||
// engines, the ordering for `detail=high` should differ from default in
|
||||
// any case where the swamp / curated pages have different raw scores.
|
||||
//
|
||||
// Postgres's CTE ranks pages then picks best chunk; ts_rank normalizes
|
||||
// by doc length so chat pages don't always swamp at the page level.
|
||||
// PGLite scores chunks directly — chat chunks beat article chunks on
|
||||
// raw ts_rank. The two engines need different parity contracts here.
|
||||
//
|
||||
// Common assertion that holds on both: detail=high must include the
|
||||
// chat pages in its result set (they're not filtered by detail), and
|
||||
// the result set should not be identical to default-detail (the boost
|
||||
// must be doing _something_ visible).
|
||||
const pgDefault = await pgEngine.searchKeyword('fat code thin harness', { limit: 5 });
|
||||
const pgHigh = await pgEngine.searchKeyword('fat code thin harness', { detail: 'high', limit: 5 });
|
||||
const pgliteDefault = await pgliteEngine.searchKeyword('fat code thin harness', { limit: 5 });
|
||||
const pgliteHigh = await pgliteEngine.searchKeyword('fat code thin harness', { detail: 'high', limit: 5 });
|
||||
|
||||
// Chat pages must be present in detail=high results on both engines.
|
||||
expect(pgHigh.some((r: SearchResult) => r.slug.startsWith('wintermute/chat/'))).toBe(true);
|
||||
expect(pgliteHigh.some((r: SearchResult) => r.slug.startsWith('wintermute/chat/'))).toBe(true);
|
||||
|
||||
// The boost must be doing something — at least one engine's ordering
|
||||
// should change between default and detail=high.
|
||||
const pgChanged = pgDefault.map((r: SearchResult) => r.slug).join(',') !== pgHigh.map((r: SearchResult) => r.slug).join(',');
|
||||
const pgliteChanged = pgliteDefault.map((r: SearchResult) => r.slug).join(',') !== pgliteHigh.map((r: SearchResult) => r.slug).join(',');
|
||||
expect(pgChanged || pgliteChanged).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -43,13 +43,27 @@ async function waitTerminal(queue: MinionQueue, id: number, timeoutMs = 15000):
|
||||
}
|
||||
|
||||
describeE2E('E2E: Minions shell handler', () => {
|
||||
let originalAllowShellJobs: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
// The shell handler refuses to run unless GBRAIN_ALLOW_SHELL_JOBS=1 is
|
||||
// set on the worker process (defense-in-depth: the env var is the
|
||||
// operator-trust gate, separate from the trusted-add allowProtectedSubmit
|
||||
// flag). The PGLite sibling test sets this in its beforeAll for the same
|
||||
// reason; without it shell jobs land in `dead`.
|
||||
originalAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = '1';
|
||||
await setupDB();
|
||||
await runMigrations(getEngine());
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
if (originalAllowShellJobs === undefined) {
|
||||
delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
} else {
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = originalAllowShellJobs;
|
||||
}
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
/**
|
||||
* Hard-Exclude E2E
|
||||
*
|
||||
* Verifies the new exclude_slug_prefixes / include_slug_prefixes plumbing.
|
||||
* test/, archive/, attachments/, .raw/ are hard-excluded by default.
|
||||
* include_slug_prefixes opts back in.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { ChunkInput } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
await engine.putPage('test/fixtures/widget', {
|
||||
type: 'note',
|
||||
title: 'Widget test fixture',
|
||||
compiled_truth: 'widget test fixture for the test suite',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('test/fixtures/widget', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'widget test fixture for the test suite',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(11),
|
||||
token_count: 8,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
await engine.putPage('archive/old-stuff/widget-2020', {
|
||||
type: 'note',
|
||||
title: 'Widget 2020',
|
||||
compiled_truth: 'widget archived from 2020',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('archive/old-stuff/widget-2020', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'widget archived from 2020 — stale info about widget',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(12),
|
||||
token_count: 8,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
await engine.putPage('concepts/widget-pattern', {
|
||||
type: 'concept',
|
||||
title: 'Widget Pattern',
|
||||
compiled_truth: 'the widget pattern is a useful design pattern',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('concepts/widget-pattern', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'the widget pattern is a useful widget design pattern',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(13),
|
||||
token_count: 9,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('searchKeyword default hard-excludes', () => {
|
||||
test('test/ pages are hidden by default', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
});
|
||||
|
||||
test('archive/ pages are hidden by default', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
|
||||
test('curated content is unaffected', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('concepts/widget-pattern');
|
||||
});
|
||||
});
|
||||
|
||||
describe('searchKeyword include_slug_prefixes opt-back-in', () => {
|
||||
test('include_slug_prefixes: ["test/"] surfaces test pages', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
// archive/ is still excluded.
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
|
||||
test('include_slug_prefixes lets caller opt back into both', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
include_slug_prefixes: ['test/', 'archive/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
expect(slugs).toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
});
|
||||
|
||||
describe('searchVector hard-excludes', () => {
|
||||
test('test/ pages are excluded by default in vector search', async () => {
|
||||
const results = await engine.searchVector(basisEmbedding(11));
|
||||
// basisEmbedding(11) is the closest direction to test/fixtures/widget,
|
||||
// so without exclude it would be at top. With default exclude, it's gone.
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
});
|
||||
|
||||
test('include_slug_prefixes lets it back in', async () => {
|
||||
const results = await engine.searchVector(basisEmbedding(11), {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
});
|
||||
});
|
||||
|
||||
describe('caller-supplied exclude_slug_prefixes (additive)', () => {
|
||||
test('caller can add a custom exclude prefix on top of defaults', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
exclude_slug_prefixes: ['concepts/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
// concepts/ now also excluded; with all three categories filtered, no
|
||||
// hits remain.
|
||||
expect(slugs).not.toContain('concepts/widget-pattern');
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,148 @@
|
||||
/**
|
||||
* Search Swamp Resistance E2E
|
||||
*
|
||||
* Reproduces the v3-plan repro case: a curated article (originals/) competes
|
||||
* with two chat-log pages (wintermute/chat/) on similar ts_rank. With v0.21+
|
||||
* source-aware ranking, the article must rank #0.
|
||||
*
|
||||
* Mirrors the structure of search-quality.test.ts. Uses PGLite in-memory.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { ChunkInput } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Curated article — short, dense, opinionated. The page that should win.
|
||||
await engine.putPage('originals/talks/article-outline-fat-code', {
|
||||
type: 'writing',
|
||||
title: 'Fat Code Thin Harness — Part 3',
|
||||
compiled_truth:
|
||||
'Fat code thin harness is the architectural pattern where business logic ' +
|
||||
'lives in fat skill files and the runtime stays thin. Part 3 covers the ' +
|
||||
'production case studies.',
|
||||
timeline: '2026-04-10: Drafted Part 3 outline.',
|
||||
});
|
||||
await engine.upsertChunks('originals/talks/article-outline-fat-code', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text:
|
||||
'Fat code thin harness — the pattern where business logic lives in fat skill files. Part 3.',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(7),
|
||||
token_count: 20,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
// Chat swamp #1 — long page, mentions the phrase repeatedly.
|
||||
await engine.putPage('wintermute/chat/2026-04-15', {
|
||||
type: 'note',
|
||||
title: '2026-04-15 chat',
|
||||
compiled_truth: '',
|
||||
timeline:
|
||||
'fat code thin harness fat code thin harness — discussed at length. ' +
|
||||
'fat code thin harness came up again. ' +
|
||||
'The fat code thin harness pattern is something we keep returning to. ' +
|
||||
'fat code thin harness fat code thin harness fat code thin harness.',
|
||||
});
|
||||
await engine.upsertChunks('wintermute/chat/2026-04-15', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text:
|
||||
'fat code thin harness fat code thin harness discussed at length, ' +
|
||||
'the fat code thin harness pattern keeps coming back, ' +
|
||||
'fat code thin harness fat code thin harness fat code thin harness.',
|
||||
chunk_source: 'timeline',
|
||||
embedding: basisEmbedding(8),
|
||||
token_count: 30,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
// Chat swamp #2 — same shape.
|
||||
await engine.putPage('wintermute/chat/2026-04-16', {
|
||||
type: 'note',
|
||||
title: '2026-04-16 chat',
|
||||
compiled_truth: '',
|
||||
timeline:
|
||||
'fat code thin harness once more. fat code thin harness fat code thin harness. ' +
|
||||
'still talking about fat code thin harness. fat code thin harness.',
|
||||
});
|
||||
await engine.upsertChunks('wintermute/chat/2026-04-16', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text:
|
||||
'fat code thin harness once more, fat code thin harness fat code thin harness, ' +
|
||||
'still talking about fat code thin harness fat code thin harness.',
|
||||
chunk_source: 'timeline',
|
||||
embedding: basisEmbedding(9),
|
||||
token_count: 25,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('searchKeyword swamp resistance', () => {
|
||||
test('curated originals/ page outranks chat swamp on multi-word query', async () => {
|
||||
const results = await engine.searchKeyword('fat code thin harness');
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
const top = results[0];
|
||||
expect(top.slug).toBe('originals/talks/article-outline-fat-code');
|
||||
});
|
||||
|
||||
test('detail=high (temporal bypass) lets chat swamp re-surface', async () => {
|
||||
// With source-boost disabled, raw ts_rank wins → chat pages, which have
|
||||
// many more keyword hits, are allowed back to the top. This guards the
|
||||
// temporal-query workflow ("what did we discuss about X").
|
||||
const results = await engine.searchKeyword('fat code thin harness', { detail: 'high' });
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// Top result should be a chat page (more keyword density per chunk).
|
||||
const topSlugs = results.slice(0, 2).map(r => r.slug);
|
||||
const anyChat = topSlugs.some(s => s.startsWith('wintermute/chat/'));
|
||||
expect(anyChat).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('searchVector swamp resistance', () => {
|
||||
test('curated originals/ page outranks chat swamp when boost is meaningful', async () => {
|
||||
// Query vector is close to all three pages (mixed direction). Without
|
||||
// source-boost the chat pages would tie or win on raw cosine; with
|
||||
// source-boost the originals/ page dominates.
|
||||
const queryVec = new Float32Array(1536);
|
||||
queryVec[7] = 0.6; // article direction
|
||||
queryVec[8] = 0.55; // chat-1 direction (slightly higher, simulating swamp)
|
||||
queryVec[9] = 0.55; // chat-2 direction
|
||||
// Normalize so cosine math is well-formed.
|
||||
const norm = Math.sqrt(0.6 * 0.6 + 0.55 * 0.55 + 0.55 * 0.55);
|
||||
for (let i = 0; i < queryVec.length; i++) queryVec[i] = queryVec[i] / norm;
|
||||
|
||||
const results = await engine.searchVector(queryVec);
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
expect(results[0].slug).toBe('originals/talks/article-outline-fat-code');
|
||||
});
|
||||
|
||||
test('two-stage CTE returns p.source_id (regression for v0.18 multi-source)', async () => {
|
||||
const queryVec = basisEmbedding(7);
|
||||
const results = await engine.searchVector(queryVec);
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// source_id is added by v0.18 multi-source brains; carrying it through
|
||||
// the inner→outer CTE is one of the v3 plan's pass-4 findings.
|
||||
for (const r of results) {
|
||||
expect(r.source_id).toBeDefined();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,255 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
buildSourceFactorCase,
|
||||
buildHardExcludeClause,
|
||||
__test__,
|
||||
} from '../src/core/search/sql-ranking.ts';
|
||||
import {
|
||||
DEFAULT_SOURCE_BOOSTS,
|
||||
DEFAULT_HARD_EXCLUDES,
|
||||
parseSourceBoostEnv,
|
||||
parseHardExcludesEnv,
|
||||
resolveBoostMap,
|
||||
resolveHardExcludes,
|
||||
} from '../src/core/search/source-boost.ts';
|
||||
|
||||
const { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral } = __test__;
|
||||
|
||||
describe('escapeLikePattern', () => {
|
||||
test('escapes %', () => {
|
||||
expect(escapeLikePattern('foo%bar')).toBe('foo\\%bar');
|
||||
});
|
||||
|
||||
test('escapes _', () => {
|
||||
expect(escapeLikePattern('foo_bar')).toBe('foo\\_bar');
|
||||
});
|
||||
|
||||
test('escapes \\ (Postgres LIKE default escape char)', () => {
|
||||
expect(escapeLikePattern('foo\\bar')).toBe('foo\\\\bar');
|
||||
});
|
||||
|
||||
test('escapes all three together', () => {
|
||||
expect(escapeLikePattern('a%b_c\\d')).toBe('a\\%b\\_c\\\\d');
|
||||
});
|
||||
|
||||
test('leaves plain strings untouched', () => {
|
||||
expect(escapeLikePattern('originals/talks/')).toBe('originals/talks/');
|
||||
});
|
||||
});
|
||||
|
||||
describe('escapeSqlLiteral', () => {
|
||||
test('doubles single quotes', () => {
|
||||
expect(escapeSqlLiteral("O'Brien")).toBe("O''Brien");
|
||||
});
|
||||
|
||||
test('handles SQL injection attempts as literal data', () => {
|
||||
// Classic injection pattern is rendered harmless because the doubled
|
||||
// quote keeps it inside a string literal in the emitted SQL.
|
||||
expect(escapeSqlLiteral("'; DROP TABLE pages; --")).toBe("''; DROP TABLE pages; --");
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildLikePrefixLiteral', () => {
|
||||
test('produces a quoted LIKE pattern with trailing %', () => {
|
||||
expect(buildLikePrefixLiteral('originals/')).toBe("'originals/%'");
|
||||
});
|
||||
|
||||
test('escapes meta-chars before adding the trailing %', () => {
|
||||
// Input contains a literal % that should be escaped, and the trailing
|
||||
// % we add is the LIKE wildcard.
|
||||
expect(buildLikePrefixLiteral('weird%path/')).toBe("'weird\\%path/%'");
|
||||
});
|
||||
|
||||
test('escapes single-quote in prefix as SQL literal', () => {
|
||||
expect(buildLikePrefixLiteral("o'brien/")).toBe("'o''brien/%'");
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildSourceFactorCase', () => {
|
||||
test('returns plain 1.0 when detail is "high" (temporal bypass)', () => {
|
||||
const result = buildSourceFactorCase('p.slug', { 'originals/': 1.5 }, 'high');
|
||||
expect(result).toBe('1.0');
|
||||
});
|
||||
|
||||
test('temporal bypass tolerates uppercase / whitespace from MCP boundary', () => {
|
||||
// Agents passing JSON over MCP can send "HIGH" or "high " (trailing
|
||||
// space). The bypass must catch these — otherwise loose-string callers
|
||||
// silently get boosted ranking instead of temporal bypass.
|
||||
const map = { 'originals/': 1.5 };
|
||||
expect(buildSourceFactorCase('p.slug', map, 'HIGH' as 'high')).toBe('1.0');
|
||||
expect(buildSourceFactorCase('p.slug', map, 'high ' as 'high')).toBe('1.0');
|
||||
expect(buildSourceFactorCase('p.slug', map, ' High ' as 'high')).toBe('1.0');
|
||||
});
|
||||
|
||||
test('returns plain 1.0 when boost map is empty', () => {
|
||||
expect(buildSourceFactorCase('p.slug', {}, 'medium')).toBe('1.0');
|
||||
});
|
||||
|
||||
test('emits a CASE expression for non-high detail', () => {
|
||||
const result = buildSourceFactorCase('p.slug', { 'originals/': 1.5 }, 'medium');
|
||||
expect(result).toBe("(CASE WHEN p.slug LIKE 'originals/%' THEN 1.5 ELSE 1.0 END)");
|
||||
});
|
||||
|
||||
test('sorts prefixes by length descending so longest-match wins', () => {
|
||||
const result = buildSourceFactorCase(
|
||||
'p.slug',
|
||||
{ 'media/': 0.9, 'media/articles/': 1.1, 'media/x/': 0.7 },
|
||||
'medium',
|
||||
);
|
||||
// Longest first: media/articles/ (15), media/x/ (8), media/ (6)
|
||||
const m = result.match(/LIKE '([^']+)%'/g);
|
||||
expect(m).toEqual([
|
||||
"LIKE 'media/articles/%'",
|
||||
"LIKE 'media/x/%'",
|
||||
"LIKE 'media/%'",
|
||||
]);
|
||||
});
|
||||
|
||||
test('detail=low and detail=undefined both emit the boost CASE', () => {
|
||||
const map = { 'originals/': 1.5 };
|
||||
expect(buildSourceFactorCase('p.slug', map, 'low')).toContain('CASE WHEN');
|
||||
expect(buildSourceFactorCase('p.slug', map, undefined)).toContain('CASE WHEN');
|
||||
});
|
||||
|
||||
test('rejects non-finite or negative factors', () => {
|
||||
const result = buildSourceFactorCase(
|
||||
'p.slug',
|
||||
{ 'good/': 1.5, 'nan/': NaN, 'neg/': -1, 'inf/': Infinity },
|
||||
'medium',
|
||||
);
|
||||
expect(result).toBe("(CASE WHEN p.slug LIKE 'good/%' THEN 1.5 ELSE 1.0 END)");
|
||||
});
|
||||
|
||||
test('uses the supplied slug column reference', () => {
|
||||
expect(buildSourceFactorCase('slug', { 'originals/': 1.5 }, 'medium'))
|
||||
.toContain('WHEN slug LIKE');
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildHardExcludeClause', () => {
|
||||
test('returns empty string when prefixes is empty', () => {
|
||||
expect(buildHardExcludeClause('p.slug', [])).toBe('');
|
||||
});
|
||||
|
||||
test('emits NOT (col LIKE ... OR col LIKE ...)', () => {
|
||||
const result = buildHardExcludeClause('p.slug', ['test/', 'archive/']);
|
||||
expect(result).toBe(`AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'archive/%')`);
|
||||
});
|
||||
|
||||
test('escapes %, _, and \\ as LIKE meta-characters', () => {
|
||||
// CEO pass 4 + codex finding: backslash is Postgres LIKE's default escape char.
|
||||
// A literal backslash in a user-supplied prefix must be escaped to \\ so
|
||||
// it's treated as data, not as "escape the next char".
|
||||
const result = buildHardExcludeClause('p.slug', ['weird\\path/']);
|
||||
expect(result).toBe(`AND NOT (p.slug LIKE 'weird\\\\path/%')`);
|
||||
});
|
||||
|
||||
test('treats SQL-injection-style input as literal', () => {
|
||||
const result = buildHardExcludeClause('p.slug', ["'; DROP TABLE pages; --"]);
|
||||
// Single quotes get doubled — the injection becomes inert text inside
|
||||
// the string literal.
|
||||
expect(result).toContain("''; DROP TABLE pages; --");
|
||||
// Sanity: the structure of the clause is intact.
|
||||
expect(result).toMatch(/^AND NOT \(p\.slug LIKE '.*%'\)$/);
|
||||
});
|
||||
|
||||
test('skips empty-string prefixes', () => {
|
||||
const result = buildHardExcludeClause('p.slug', ['test/', '', 'archive/']);
|
||||
// Two LIKE clauses, one OR.
|
||||
expect((result.match(/LIKE/g) || []).length).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseSourceBoostEnv', () => {
|
||||
test('parses comma-separated prefix:factor pairs', () => {
|
||||
expect(parseSourceBoostEnv('originals/:1.8,wintermute/chat/:0.3'))
|
||||
.toEqual({ 'originals/': 1.8, 'wintermute/chat/': 0.3 });
|
||||
});
|
||||
|
||||
test('returns empty object for undefined or empty', () => {
|
||||
expect(parseSourceBoostEnv(undefined)).toEqual({});
|
||||
expect(parseSourceBoostEnv('')).toEqual({});
|
||||
});
|
||||
|
||||
test('skips malformed entries', () => {
|
||||
expect(parseSourceBoostEnv('bogus,no-colon,originals/:abc,valid/:1.5'))
|
||||
.toEqual({ 'valid/': 1.5 });
|
||||
});
|
||||
|
||||
test('rejects negative factors', () => {
|
||||
expect(parseSourceBoostEnv('foo/:-1.0,bar/:0.5')).toEqual({ 'bar/': 0.5 });
|
||||
});
|
||||
|
||||
test('accepts factor=0 (legal but performance-inferior to hard-exclude)', () => {
|
||||
expect(parseSourceBoostEnv('foo/:0')).toEqual({ 'foo/': 0 });
|
||||
});
|
||||
|
||||
test('uses last colon to separate prefix from factor', () => {
|
||||
// Edge case: someone puts a colon inside the prefix. Last colon wins.
|
||||
expect(parseSourceBoostEnv('foo:bar/:1.5')).toEqual({ 'foo:bar/': 1.5 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseHardExcludesEnv', () => {
|
||||
test('parses comma-separated prefixes', () => {
|
||||
expect(parseHardExcludesEnv('test/,scratch/,private/'))
|
||||
.toEqual(['test/', 'scratch/', 'private/']);
|
||||
});
|
||||
|
||||
test('returns empty array for undefined', () => {
|
||||
expect(parseHardExcludesEnv(undefined)).toEqual([]);
|
||||
});
|
||||
|
||||
test('trims whitespace and drops empty entries', () => {
|
||||
expect(parseHardExcludesEnv(' test/ , , scratch/ ')).toEqual(['test/', 'scratch/']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveBoostMap', () => {
|
||||
test('returns defaults when env is unset', () => {
|
||||
expect(resolveBoostMap(undefined)).toEqual(DEFAULT_SOURCE_BOOSTS);
|
||||
});
|
||||
|
||||
test('env override takes precedence over defaults', () => {
|
||||
const merged = resolveBoostMap('originals/:99');
|
||||
expect(merged['originals/']).toBe(99);
|
||||
// Other defaults still present.
|
||||
expect(merged['concepts/']).toBe(DEFAULT_SOURCE_BOOSTS['concepts/']);
|
||||
});
|
||||
|
||||
test('env-only entries are added on top of defaults', () => {
|
||||
const merged = resolveBoostMap('newprefix/:2.5');
|
||||
expect(merged['newprefix/']).toBe(2.5);
|
||||
expect(merged['originals/']).toBe(DEFAULT_SOURCE_BOOSTS['originals/']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveHardExcludes', () => {
|
||||
test('returns defaults when nothing is overridden', () => {
|
||||
const r = resolveHardExcludes(undefined, undefined, undefined);
|
||||
for (const p of DEFAULT_HARD_EXCLUDES) expect(r).toContain(p);
|
||||
});
|
||||
|
||||
test('caller exclude_slug_prefixes adds to the union', () => {
|
||||
const r = resolveHardExcludes(['scratch/'], undefined, undefined);
|
||||
expect(r).toContain('scratch/');
|
||||
expect(r).toContain('test/'); // default still present
|
||||
});
|
||||
|
||||
test('include_slug_prefixes opts back in', () => {
|
||||
const r = resolveHardExcludes(undefined, ['test/'], undefined);
|
||||
expect(r).not.toContain('test/');
|
||||
// Other defaults still present.
|
||||
expect(r).toContain('archive/');
|
||||
});
|
||||
|
||||
test('env GBRAIN_SEARCH_EXCLUDE adds to the union', () => {
|
||||
const r = resolveHardExcludes(undefined, undefined, 'envdir/');
|
||||
expect(r).toContain('envdir/');
|
||||
});
|
||||
|
||||
test('include subtracts from env-supplied excludes too', () => {
|
||||
const r = resolveHardExcludes(undefined, ['envdir/'], 'envdir/');
|
||||
expect(r).not.toContain('envdir/');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user