feat(search): configurable FTS language + reindex command (lands #580/#581/#582)

Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r
(#580 env-var language for query+write side, #581 migration backfill,
#582 gbrain reindex-search-vector), rebased onto current master with the
security review's required fixes applied:

- Migration renumbered v116 -> v123 (master's v116 was already claimed by
  code_edges_source_backfill; master is at v122).
- Restored the v120/#1647 search_path hardening: all four CREATE OR
  REPLACE trigger-function bodies (migration handler + reindex command)
  now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE
  resets proconfig and would otherwise strip the hardening on upgrade.
- reindex-search-vector: shared progress reporter (stderr phases
  reindex_search_vector.pages/.chunks), id-keyset batched backfill
  (5000 rows/UPDATE) instead of single whole-table statements, and
  --json no longer bypasses the --yes/TTY confirmation gate.
- Allowlist validation regex + injection tests kept exactly as authored.
- Stale v33/v116 comments swept; docs/guides/multi-language-fts.md
  written (README referenced it but no PR added it); llms bundles
  regenerated via bun run build:llms.
- Trilogy tests quarantined as *.serial.test.ts (env mutation, per
  check-test-isolation).

Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese
produces portuguese-stemmed vectors with search_path pinned; the reindex
command retokenizes an english brain to portuguese in place.

Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sinabina
2026-07-17 14:45:33 -07:00
co-authored by Rafael Reis Claude Fable 5
parent 54a8070640
commit 10c24d76c3
13 changed files with 972 additions and 9 deletions
+18
View File
@@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
```
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
+2
View File
@@ -36,6 +36,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/post-upgrade-reembed.ts` — Pure functions backing the `gbrain upgrade` chunker-bump cost prompt. `computeReembedEstimate(engine, model)` queries real SQL (`COUNT(*)` + `COALESCE(SUM(LENGTH(compiled_truth)) + SUM(LENGTH(timeline)), 0)`) on `pages WHERE chunker_version < MARKDOWN_CHUNKER_VERSION`. `formatReembedPrompt(est, graceSeconds)` is the stderr-line formatter. `runPostUpgradeReembedPrompt(engine, model, opts)` orchestrates the 10-second Ctrl-C window; TTY-only wait (non-TTY auto-proceeds for CI / cron); `GBRAIN_NO_REEMBED=1` bails with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait.
- `src/commands/reindex.ts``gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--no-embed] [--repo PATH]`. Walks `pages WHERE page_kind = 'markdown' AND chunker_version < MARKDOWN_CHUNKER_VERSION` in 100-row batches ordered by id. Rows with non-null `source_path` re-import via `importFromFile`; rows without fall back to `importFromContent`. Both paths pass `forceRechunk: true` to bypass `importFromContent`'s `content_hash` short-circuit — without it the chunker version bump never reaches pages whose source content hasn't changed, AND the stripFactsFence privacy strip never applies to pre-strip chunks. Idempotent — partial-completion re-runs pick up via id-ordered batches. Wired into `src/commands/upgrade.ts:runPostUpgrade` after `apply-migrations`. The DB-only fallback (no source file on disk) does NOT pass body-only `compiled_truth` to `importFromContent` (that path re-parses with EMPTY frontmatter and OVERWRITES the page's real frontmatter/title/timeline); it `getPage`+`getTags`, reconstructs FULL markdown via `serializeMarkdown(frontmatter, compiled_truth, timeline, {type, title, tags})`, and re-imports THAT so re-chunking a DB-only page preserves everything while bumping `chunker_version`. Pinned by `test/reindex-preserve-tags.test.ts`.
- `src/commands/reindex-code.ts``gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). `spend.posture=tokenmax` OR an explicit `--max-cost off`/`unlimited` makes the gate informational and proceeds (#2139); `--max-cost off` also disables the runtime BudgetTracker cap. Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`.
- `src/core/fts-language.ts` — Single source for the Postgres text-search configuration name used by FTS. `getFtsLanguage()` resolves `GBRAIN_FTS_LANGUAGE` (default `english`), validates against `/^[a-z][a-z0-9_]*$/` (tsvector config names can't be bound as parameters, so the value is interpolated into raw SQL — the allowlist regex is the injection guard; invalid values warn once and fall back to `english`), and caches on first read (`resetFtsLanguageCache()` is test-only). Consumed by both engines' `searchKeyword`/`searchKeywordChunks` (`websearch_to_tsquery` query side), the `configurable_fts_language` migration, and `reindex-search-vector` (write-side trigger functions). Pinned by `test/fts-language.serial.test.ts` + `test/fts-language-migration.serial.test.ts` (includes the `'; DROP TABLE pages; --` injection cases).
- `src/commands/reindex-search-vector.ts``gbrain reindex-search-vector [--dry-run] [--yes] [--json]`. Escape hatch for changing `GBRAIN_FTS_LANGUAGE` after the `configurable_fts_language` migration has run (the migration shows applied and is skipped): recreates `update_page_search_vector` + `update_chunk_search_vector` with the configured language — bodies mirror the migration's and KEEP the `SET search_path = pg_catalog, public` hardening (CREATE OR REPLACE resets proconfig) — then backfills `pages` (UPDATE-to-self re-fires the trigger) and `content_chunks` (direct vector recompute) in id-keyset batches of `BACKFILL_BATCH_SIZE` (5000) via `UPDATE … WHERE id IN (SELECT … LIMIT n) RETURNING id`, streaming phases `reindex_search_vector.pages`/`.chunks` through the shared progress reporter (stderr). Confirmation gate: `--yes`, or an interactive TTY [y/N]; `--json` does NOT bypass the gate (non-TTY without `--yes` refuses with a ConfirmationRequired envelope, exit 2). Idempotent. Pinned by `test/reindex-search-vector.serial.test.ts`.
- `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path.
- `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`.
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column).
+97
View File
@@ -0,0 +1,97 @@
# Multi-language full-text search
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE`
environment variable. Default: `english`.
## How it works
Postgres text-search configurations control stemming and stop-word removal.
`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on
both sides of the search:
- **Query side** — `websearch_to_tsquery('<lang>', $query)` in both engines
(Postgres and PGLite).
- **Write side** — the `update_page_search_vector` and
`update_chunk_search_vector` trigger functions that populate
`pages.search_vector` and `content_chunks.search_vector`.
The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever
interpolated into SQL (tsvector functions don't accept parameterized config
names). Invalid values fall back to `english` with a warning.
## Built-in languages
Set the env var to any configuration your Postgres instance ships:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
export GBRAIN_FTS_LANGUAGE=spanish
export GBRAIN_FTS_LANGUAGE=german
```
List what's available:
```sql
SELECT cfgname FROM pg_ts_config;
```
PGLite (the embedded default engine) ships the same built-in snowball
configurations as stock Postgres.
## First install vs. changing language later
On first install (or upgrade), the `configurable_fts_language` schema
migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with
that language. After the migration has run, changing the env var alone does
NOT retokenize existing rows — the migration shows as applied and is skipped.
Use the explicit command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview: language + row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command recreates both trigger functions under the new language and
backfills every existing `pages` and `content_chunks` row in batches,
streaming progress to stderr. It is idempotent: re-running with the same
language produces identical vectors. `--json` prints a machine-readable
result envelope but still requires `--yes` (or an interactive confirm).
## Recipe: accent-insensitive Portuguese (`pt_br`)
Brazilian Portuguese content often mixes accented and unaccented spellings
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
the `unaccent` extension, then stems with the portuguese snowball dictionary:
```sql
CREATE EXTENSION IF NOT EXISTS unaccent;
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
ALTER TEXT SEARCH CONFIGURATION pt_br
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, portuguese_stem;
```
Then point GBrain at it:
```bash
export GBRAIN_FTS_LANGUAGE=pt_br
gbrain reindex-search-vector --yes
```
Note: custom configurations require a real Postgres instance (e.g. the
Supabase engine). The config must exist BEFORE the migration or the reindex
command runs, or Postgres will reject the trigger recreation with
`text search configuration "pt_br" does not exist`.
## Caveats
- One language per brain: the setting is global to the database, not
per-source. Mixed-language brains should pick the dominant language (the
vector-search arm is language-agnostic and covers the rest).
- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that
writes to the brain (CLI shells, MCP server, cron jobs) — a writer without
the env var tokenizes new rows in `english` until the next reindex.
+18
View File
@@ -1752,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
```
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
+13 -1
View File
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -2001,6 +2001,15 @@ async function handleCliOnly(command: string, args: string[]) {
await runReindexCodeCli(engine, args);
break;
}
case 'reindex-search-vector': {
// Explicit recreate of FTS trigger functions + batched backfill,
// honoring GBRAIN_FTS_LANGUAGE. Use after changing the language
// env var on a brain that already ran the configurable_fts_language
// migration.
const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts');
await runReindexSearchVectorCli(engine, args);
break;
}
case 'reindex-frontmatter': {
// v0.29.1: recovery / explicit-rebuild path for pages.effective_date.
// Mirror of reindex-code shape. Wraps the shared library function in
@@ -2336,6 +2345,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II)
query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0)
reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0)
reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0)
reindex-search-vector [--dry-run] [--yes] [--json]
Recreate FTS triggers + backfill under
$GBRAIN_FTS_LANGUAGE (default 'english')
sync --strategy code Sync code files into the brain
JOBS (Minions)
+277
View File
@@ -0,0 +1,277 @@
/**
* `gbrain reindex-search-vector` — recreate FTS trigger functions and
* backfill existing rows under the language configured via
* GBRAIN_FTS_LANGUAGE.
*
* Why this command exists: schema migration v123 (configurable_fts_language)
* stamps the trigger functions with the configured language at first apply.
* After that, changing the env var has no effect on the write side because
* v123 already shows as "applied" — the migrations runner will skip it.
* This command is the documented escape hatch: it re-runs the same
* recreate-and-backfill logic v123 uses, gated on an explicit user
* action so the operation is intentional and visible (writes touch
* every row in pages and content_chunks).
*
* Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces
* the same trigger function bodies and the same tokenized vectors.
*
* Flags:
* --dry-run Show what would happen, exit 0 without touching DB.
* --yes Skip interactive [y/N]. Required for non-TTY (including --json).
* --json Machine-readable result envelope. Does NOT imply --yes.
*
* Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE)
* so a large brain never holds one giant row lock, and streams progress
* through the shared reporter (stderr; stdout stays clean for --json).
*
* Cost: trigger recreate is sub-millisecond. Backfill is one tsvector
* rebuild per page + per chunk. On a 20K-page brain with 80K chunks,
* expect ~5-15s depending on Postgres CPU and content size.
*/
import type { BrainEngine } from '../core/engine.ts';
import { getFtsLanguage } from '../core/fts-language.ts';
import { createInterface } from 'readline';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
export interface ReindexSearchVectorOpts {
dryRun?: boolean;
yes?: boolean;
json?: boolean;
}
export interface ReindexSearchVectorResult {
status: 'ok' | 'dry_run' | 'cancelled';
language: string;
pagesUpdated: number;
chunksUpdated: number;
triggersRecreated: number;
durationMs: number;
}
interface CountRow {
pages: number;
chunks: number;
}
/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */
export const BACKFILL_BATCH_SIZE = 5000;
/**
* Keyset-batched UPDATE: applies `setClause` to `table` rows where
* search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking
* the shared progress reporter after each batch. Terminates when a batch
* returns fewer rows than the batch size (or none).
*/
async function batchedBackfill(
engine: BrainEngine,
table: 'pages' | 'content_chunks',
setClause: string,
tick: (n: number) => void
): Promise<void> {
let cursor = 0;
for (;;) {
const rows = await engine.executeRaw<{ id: number }>(`
UPDATE ${table} SET ${setClause}
WHERE id IN (
SELECT id FROM ${table}
WHERE search_vector IS NOT NULL AND id > ${cursor}
ORDER BY id
LIMIT ${BACKFILL_BATCH_SIZE}
)
RETURNING id
`);
if (rows.length === 0) break;
tick(rows.length);
cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor);
if (rows.length < BACKFILL_BATCH_SIZE) break;
}
}
/**
* Programmatic entrypoint — takes a typed opts object. Used by tests and
* future internal callers. The CLI wrapper is `runReindexSearchVectorCli`
* defined at the bottom of this file.
*/
export async function runReindexSearchVector(
engine: BrainEngine,
opts: ReindexSearchVectorOpts
): Promise<ReindexSearchVectorResult> {
const lang = getFtsLanguage();
const startedAt = Date.now();
// Inventory: how many rows will the backfill touch?
const counts = await engine.executeRaw<CountRow>(
`SELECT
(SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages,
(SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks`
);
const pagesCount = counts[0]?.pages ?? 0;
const chunksCount = counts[0]?.chunks ?? 0;
if (opts.dryRun) {
const result: ReindexSearchVectorResult = {
status: 'dry_run',
language: lang,
pagesUpdated: pagesCount,
chunksUpdated: chunksCount,
triggersRecreated: 0,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`);
console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`);
console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`);
}
return result;
}
// Confirm unless --yes. --json does NOT bypass the gate — a machine
// caller must pass --yes explicitly (mirrors reindex-code, #1784).
if (!opts.yes) {
if (!process.stdin.isTTY) {
if (opts.json) {
console.log(JSON.stringify({
error: {
class: 'ConfirmationRequired',
code: 'reindex_requires_yes',
message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`,
hint: 'Pass --yes to proceed, or --dry-run to preview.',
},
language: lang,
pages: pagesCount,
chunks: chunksCount,
}));
} else {
console.error('Refusing to run without --yes in non-TTY environment.');
}
process.exit(2);
}
const rl = createInterface({ input: process.stdin, output: process.stdout });
const answer = await new Promise<string>(resolve => {
rl.question(
`Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `,
resolve
);
});
rl.close();
if (!/^y(es)?$/i.test(answer.trim())) {
const result: ReindexSearchVectorResult = {
status: 'cancelled',
language: lang,
pagesUpdated: 0,
chunksUpdated: 0,
triggersRecreated: 0,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log('Cancelled.');
}
return result;
}
}
// Recreate trigger functions. The strings are intentionally identical to
// the v123 migration body — keeping them in lockstep is the contract.
// `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening:
// CREATE OR REPLACE resets proconfig, so omitting it here would strip the
// hardening from every brain that runs this command.
const recreatePagesFn = `
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
DECLARE
timeline_text TEXT;
BEGIN
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
NEW.search_vector :=
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
const recreateChunksFn = `
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
BEGIN
NEW.search_vector :=
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
await engine.executeRaw(recreatePagesFn);
await engine.executeRaw(recreateChunksFn);
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
// Backfill: UPDATE-to-self forces the pages trigger to re-fire
// (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a
// direct vector compute since the column itself is what we want.
progress.start('reindex_search_vector.pages', pagesCount);
await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n));
progress.finish();
progress.start('reindex_search_vector.chunks', chunksCount);
await batchedBackfill(
engine,
'content_chunks',
`search_vector =
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`,
n => progress.tick(n)
);
progress.finish();
const result: ReindexSearchVectorResult = {
status: 'ok',
language: lang,
pagesUpdated: pagesCount,
chunksUpdated: chunksCount,
triggersRecreated: 2,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`✅ Recreated 2 trigger functions with language='${lang}'`);
console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`);
}
return result;
}
/**
* CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector.
* Matches the style of `reindex-code`: --dry-run, --yes/-y, --json.
*
* Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes.
*/
export async function runReindexSearchVectorCli(
engine: BrainEngine,
args: string[]
): Promise<void> {
const dryRun = args.includes('--dry-run');
const yes = args.includes('--yes') || args.includes('-y');
const json = args.includes('--json');
await runReindexSearchVector(engine, { dryRun, yes, json });
}
+69
View File
@@ -0,0 +1,69 @@
/**
* Full-text search language configuration.
*
* Postgres tsvector/tsquery require a text search configuration name (e.g.
* 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded
* 'english' across engines and trigger functions, which broke search
* quality for non-English brains (no stemming, no stop-word removal).
*
* This helper centralizes the choice. Default stays 'english' for backward
* compatibility — only users who set GBRAIN_FTS_LANGUAGE see different
* behavior.
*
* Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent +
* portuguese stemmer) are supported as long as the configuration exists
* in the target Postgres instance. See docs/guides/multi-language-fts.md
* for setup instructions.
*
* Validation: only allow lowercase letters, digits, and underscores. This
* prevents SQL injection when the value is interpolated into queries
* (Postgres tsvector functions don't accept parameterized config names —
* they must be literals or identifiers).
*/
const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/;
const DEFAULT_LANGUAGE = 'english';
let cachedLanguage: string | null = null;
/**
* Returns the configured Postgres text search configuration name.
*
* Resolution order:
* 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid)
* 2. 'english' (default — preserves existing behavior)
*
* The return value is safe to interpolate directly into SQL because it
* passes the VALID_CONFIG_NAME guard. If validation fails, falls back to
* the default and emits a one-time warning.
*
* Cached on first call; reset with `resetFtsLanguageCache()` (test only).
*/
export function getFtsLanguage(): string {
if (cachedLanguage !== null) return cachedLanguage;
const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim();
if (!raw) {
cachedLanguage = DEFAULT_LANGUAGE;
return cachedLanguage;
}
if (!VALID_CONFIG_NAME.test(raw)) {
console.warn(
`[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` +
`Falling back to '${DEFAULT_LANGUAGE}'.`
);
cachedLanguage = DEFAULT_LANGUAGE;
return cachedLanguage;
}
cachedLanguage = raw;
return cachedLanguage;
}
/**
* Resets the cached language. Tests only — don't use in production code.
*/
export function resetFtsLanguageCache(): void {
cachedLanguage = null;
}
+91
View File
@@ -1,5 +1,6 @@
import type { BrainEngine } from './engine.ts';
import { slugifyPath } from './sync.ts';
import { getFtsLanguage } from './fts-language.ts';
/**
* Schema migrations run automatically on initSchema().
@@ -5505,6 +5506,96 @@ export const MIGRATIONS: Migration[] = [
WHERE dimension IS NOT NULL;
`,
},
{
version: 123,
name: 'configurable_fts_language',
// Recreate the two search_vector trigger functions using the language
// configured via GBRAIN_FTS_LANGUAGE (default 'english'). Idempotent:
// CREATE OR REPLACE swaps the function body atomically; no trigger
// recreation needed since the trigger references the function by name.
//
// Why a handler instead of a static SQL string: Postgres tsvector
// functions don't accept parameterized config names — the language
// must be a literal in the SQL. getFtsLanguage() validates the value
// (lowercase letters/digits/underscores only) before interpolation.
//
// Function bodies mirror schema.sql / pglite-schema.ts exactly —
// INCLUDING the `SET search_path = pg_catalog, public` hardening from
// v120/#1647 (CREATE OR REPLACE resets proconfig, so omitting it here
// would silently strip the hardening on every upgraded brain). Only
// the text-search config name is parameterized. Keep all copies in
// sync when the trigger logic changes.
//
// Backfill: after recreating the functions, re-tokenize existing rows
// under the new language. Skipped when the configured language is
// 'english' (trigger output identical — re-tokenizing is wasted I/O).
// To change language after this migration has run, use
// `gbrain reindex-search-vector`.
sql: '',
handler: async (engine) => {
const lang = getFtsLanguage();
const recreatePagesFn = `
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
DECLARE
timeline_text TEXT;
BEGIN
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
NEW.search_vector :=
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
const recreateChunksFn = `
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
BEGIN
NEW.search_vector :=
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
await engine.executeRaw(recreatePagesFn);
await engine.executeRaw(recreateChunksFn);
if (lang === 'english') {
console.log(` v123: trigger functions recreated with language='english' (default — no backfill needed)`);
return;
}
// Backfill existing rows under the new tokenizer. UPDATE-to-same-value
// re-fires the pages trigger; chunks are rewritten directly with the
// same expression as the trigger.
await engine.executeRaw(`
UPDATE pages SET id = id
WHERE search_vector IS NOT NULL;
`);
await engine.executeRaw(`
UPDATE content_chunks
SET search_vector =
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')
WHERE search_vector IS NOT NULL;
`);
console.log(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows`);
},
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+12 -4
View File
@@ -24,6 +24,7 @@ import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
import { getFtsLanguage } from './fts-language.ts';
import type {
Page, PageInput, PageFilters, PageType,
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
@@ -1625,20 +1626,24 @@ export class PGLiteEngine implements BrainEngine {
extraFilter += ` AND p.source_id = $${params.length}`;
}
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const { rows } = await this.db.query(
`WITH ranked AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
) THEN true ELSE false END AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
-- v0.27.1: hide image rows from default text-keyword search so
-- OCR text doesn't drown text-page hits. Image-similarity queries
-- run a separate vector path on embedding_image.
@@ -1857,20 +1862,23 @@ export class PGLiteEngine implements BrainEngine {
}
// visibilityClause already declared above (v0.32.7: hoisted so CJK branch can reuse).
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const { rows } = await this.db.query(
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
) THEN true ELSE false END AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
ORDER BY score DESC
LIMIT $2 OFFSET $3`,
params
+11 -4
View File
@@ -34,6 +34,7 @@ import {
COLUMN_NAME_REGEX,
EmbeddingColumnNotRegisteredError,
} from './search/embedding-column.ts';
import { getFtsLanguage } from './fts-language.ts';
import type {
Page, PageInput, PageFilters, PageType,
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
@@ -1653,6 +1654,9 @@ export class PostgresEngine implements BrainEngine {
// column lookup. NOT bypassed by detail=high — soft-delete is a contract,
// not a temporal preference.
const visibilityClause = buildVisibilityClause('p', 's');
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const rawQuery = `
WITH ranked_chunks AS (
@@ -1660,11 +1664,11 @@ export class PostgresEngine implements BrainEngine {
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
${typeClause}
${typesClause}
${excludeSlugsClause}
@@ -1792,18 +1796,21 @@ export class PostgresEngine implements BrainEngine {
// v0.26.5: visibility filter for searchKeywordChunks (anchor primitive).
const visibilityClause = buildVisibilityClause('p', 's');
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const rawQuery = `
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
false AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
${typeClause}
${typesClause}
${excludeSlugsClause}
+119
View File
@@ -0,0 +1,119 @@
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import type { BrainEngine } from '../src/core/engine.ts';
import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts';
import { resetFtsLanguageCache } from '../src/core/fts-language.ts';
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
const originalLang = process.env[ENV_KEY];
beforeEach(() => {
delete process.env[ENV_KEY];
resetFtsLanguageCache();
});
afterEach(() => {
delete process.env[ENV_KEY];
if (originalLang !== undefined) process.env[ENV_KEY] = originalLang;
resetFtsLanguageCache();
});
describe('configurable_fts_language migration', () => {
test('migration is registered', () => {
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
expect(ftsMig).toBeDefined();
expect(ftsMig?.version).toBeGreaterThan(115);
});
test('fts migration is the latest migration', () => {
expect(MIGRATIONS.find(m => m.name === 'configurable_fts_language')?.version).toBe(LATEST_VERSION);
});
test('ftsMig uses handler (not static SQL) because language interpolation is dynamic', () => {
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
expect(ftsMig?.sql).toBe('');
expect(ftsMig?.handler).toBeTypeOf('function');
});
test('ftsMig handler is async', () => {
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
// Async function check: the constructor name is 'AsyncFunction'
expect(ftsMig?.handler?.constructor.name).toBe('AsyncFunction');
});
test('migration handler issues recreate-function calls (smoke check via mock engine)', async () => {
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
const calls: string[] = [];
const mockEngine = {
executeRaw: async (sql: string) => {
calls.push(sql);
return [];
},
} as unknown as BrainEngine;
process.env[ENV_KEY] = 'english';
resetFtsLanguageCache();
await ftsMig?.handler?.(mockEngine);
// Default 'english' \u2014 no backfill, only 2 CREATE OR REPLACE calls.
expect(calls.length).toBe(2);
expect(calls[0]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector');
expect(calls[0]).toContain("to_tsvector('english'");
expect(calls[1]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector');
expect(calls[1]).toContain("to_tsvector('english'");
// v120/#1647 hardening must survive the CREATE OR REPLACE (which resets
// proconfig): both recreated bodies pin search_path.
expect(calls[0]).toContain('SET search_path = pg_catalog, public');
expect(calls[1]).toContain('SET search_path = pg_catalog, public');
});
test('non-english language triggers backfill', async () => {
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
const calls: string[] = [];
const mockEngine = {
executeRaw: async (sql: string) => {
calls.push(sql);
return [];
},
} as unknown as BrainEngine;
process.env[ENV_KEY] = 'pt_br';
resetFtsLanguageCache();
await ftsMig?.handler?.(mockEngine);
// pt_br \u2014 2 CREATE + 2 backfill UPDATEs = 4 calls
expect(calls.length).toBe(4);
expect(calls[0]).toContain("to_tsvector('pt_br'");
expect(calls[1]).toContain("to_tsvector('pt_br'");
expect(calls[2]).toMatch(/UPDATE pages/);
expect(calls[3]).toContain("to_tsvector('pt_br'");
expect(calls[3]).toMatch(/UPDATE content_chunks/);
});
test('invalid language falls back to english (no SQL injection)', async () => {
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
const calls: string[] = [];
const mockEngine = {
executeRaw: async (sql: string) => {
calls.push(sql);
return [];
},
} as unknown as BrainEngine;
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
resetFtsLanguageCache();
await ftsMig?.handler?.(mockEngine);
// Falls back to english: 2 CREATE OR REPLACE only, no DROP TABLE in any SQL.
expect(calls.length).toBe(2);
for (const sql of calls) {
expect(sql).not.toContain('DROP TABLE');
expect(sql).toContain("to_tsvector('english'");
}
});
});
+93
View File
@@ -0,0 +1,93 @@
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { getFtsLanguage, resetFtsLanguageCache } from '../src/core/fts-language.ts';
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
beforeEach(() => {
delete process.env[ENV_KEY];
resetFtsLanguageCache();
});
afterEach(() => {
delete process.env[ENV_KEY];
resetFtsLanguageCache();
});
describe('getFtsLanguage', () => {
test('defaults to english when env is unset', () => {
expect(getFtsLanguage()).toBe('english');
});
test('defaults to english when env is empty string', () => {
process.env[ENV_KEY] = '';
expect(getFtsLanguage()).toBe('english');
});
test('defaults to english when env is whitespace', () => {
process.env[ENV_KEY] = ' ';
expect(getFtsLanguage()).toBe('english');
});
test('reads valid pt_br config', () => {
process.env[ENV_KEY] = 'pt_br';
expect(getFtsLanguage()).toBe('pt_br');
});
test('reads valid simple language name', () => {
process.env[ENV_KEY] = 'spanish';
expect(getFtsLanguage()).toBe('spanish');
});
test('reads name with underscores and digits', () => {
process.env[ENV_KEY] = 'custom_lang_v2';
expect(getFtsLanguage()).toBe('custom_lang_v2');
});
test('rejects names with quotes (SQL injection guard)', () => {
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
expect(getFtsLanguage()).toBe('english');
});
test('rejects names with spaces', () => {
process.env[ENV_KEY] = 'pt br';
expect(getFtsLanguage()).toBe('english');
});
test('rejects names with hyphens', () => {
process.env[ENV_KEY] = 'pt-br';
expect(getFtsLanguage()).toBe('english');
});
test('rejects names starting with digit', () => {
process.env[ENV_KEY] = '1lang';
expect(getFtsLanguage()).toBe('english');
});
test('rejects uppercase (Postgres config names are lowercase)', () => {
process.env[ENV_KEY] = 'English';
expect(getFtsLanguage()).toBe('english');
});
test('caches after first read', () => {
process.env[ENV_KEY] = 'pt_br';
expect(getFtsLanguage()).toBe('pt_br');
// Mutate env after first read \u2014 cached value wins.
process.env[ENV_KEY] = 'spanish';
expect(getFtsLanguage()).toBe('pt_br');
});
test('resetFtsLanguageCache clears cache', () => {
process.env[ENV_KEY] = 'pt_br';
expect(getFtsLanguage()).toBe('pt_br');
resetFtsLanguageCache();
process.env[ENV_KEY] = 'spanish';
expect(getFtsLanguage()).toBe('spanish');
});
test('trims surrounding whitespace from valid value', () => {
process.env[ENV_KEY] = ' pt_br ';
expect(getFtsLanguage()).toBe('pt_br');
});
});
+152
View File
@@ -0,0 +1,152 @@
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import type { BrainEngine } from '../src/core/engine.ts';
import { runReindexSearchVector } from '../src/commands/reindex-search-vector.ts';
import { resetFtsLanguageCache } from '../src/core/fts-language.ts';
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
const originalLang = process.env[ENV_KEY];
interface MockState {
calls: string[];
rowsToReturn: { pages: number; chunks: number };
}
function makeMockEngine(state: MockState): BrainEngine {
return {
executeRaw: async (sql: string) => {
state.calls.push(sql);
// Inventory query — return the configured counts
if (sql.includes('SELECT') && sql.includes('FROM pages WHERE search_vector')) {
return [{ pages: state.rowsToReturn.pages, chunks: state.rowsToReturn.chunks }];
}
return [];
},
} as unknown as BrainEngine;
}
beforeEach(() => {
delete process.env[ENV_KEY];
resetFtsLanguageCache();
});
afterEach(() => {
delete process.env[ENV_KEY];
if (originalLang !== undefined) process.env[ENV_KEY] = originalLang;
resetFtsLanguageCache();
});
describe('runReindexSearchVector', () => {
test('--dry-run does not issue any DDL or backfill SQL', async () => {
const state: MockState = { calls: [], rowsToReturn: { pages: 100, chunks: 500 } };
const engine = makeMockEngine(state);
process.env[ENV_KEY] = 'pt_br';
resetFtsLanguageCache();
const result = await runReindexSearchVector(engine, { dryRun: true, json: true });
expect(result.status).toBe('dry_run');
expect(result.language).toBe('pt_br');
expect(result.pagesUpdated).toBe(100);
expect(result.chunksUpdated).toBe(500);
expect(result.triggersRecreated).toBe(0);
// Only the inventory query — no CREATE OR REPLACE, no UPDATE.
expect(state.calls.length).toBe(1);
expect(state.calls[0]).toContain('SELECT');
expect(state.calls[0]).not.toContain('CREATE OR REPLACE');
expect(state.calls[0]).not.toContain('UPDATE');
});
test('--yes recreates triggers + backfills with configured language', async () => {
const state: MockState = { calls: [], rowsToReturn: { pages: 50, chunks: 200 } };
const engine = makeMockEngine(state);
process.env[ENV_KEY] = 'pt_br';
resetFtsLanguageCache();
const result = await runReindexSearchVector(engine, { yes: true, json: true });
expect(result.status).toBe('ok');
expect(result.language).toBe('pt_br');
expect(result.triggersRecreated).toBe(2);
expect(result.pagesUpdated).toBe(50);
expect(result.chunksUpdated).toBe(200);
// 1 inventory + 2 CREATE + 2 backfill batches (mock returns no rows, so
// the keyset loop terminates after the first batch per table) = 5 calls
expect(state.calls.length).toBe(5);
expect(state.calls[1]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector');
expect(state.calls[1]).toContain("to_tsvector('pt_br'");
expect(state.calls[2]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector');
expect(state.calls[2]).toContain("to_tsvector('pt_br'");
expect(state.calls[3]).toMatch(/UPDATE pages/);
expect(state.calls[4]).toMatch(/UPDATE content_chunks/);
expect(state.calls[4]).toContain("to_tsvector('pt_br'");
// v120/#1647 hardening must survive the CREATE OR REPLACE (which resets
// proconfig): both recreated bodies pin search_path.
expect(state.calls[1]).toContain('SET search_path = pg_catalog, public');
expect(state.calls[2]).toContain('SET search_path = pg_catalog, public');
});
test('default english language still recreates + backfills (no shortcut here)', async () => {
// Note: unlike the configurable_fts_language migration, the CLI command
// intentionally backfills even for english. The user explicitly asked for
// it, so we honor it. The migration skips backfill for english because it
// auto-runs on first apply.
const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } };
const engine = makeMockEngine(state);
const result = await runReindexSearchVector(engine, { yes: true, json: true });
expect(result.status).toBe('ok');
expect(result.language).toBe('english');
expect(state.calls.length).toBe(5);
// Trigger recreates (calls 1, 2) and chunks backfill (call 4) embed the
// language literal. Pages backfill (call 3) is UPDATE-to-self that
// re-fires the trigger, so the language literal lives in the trigger
// function body — not in the UPDATE statement.
expect(state.calls[1]).toContain("'english'");
expect(state.calls[2]).toContain("'english'");
expect(state.calls[3]).toMatch(/UPDATE pages/);
expect(state.calls[4]).toContain("'english'");
});
test('SQL injection attempt falls back to english', async () => {
const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } };
const engine = makeMockEngine(state);
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
resetFtsLanguageCache();
const result = await runReindexSearchVector(engine, { yes: true, json: true });
expect(result.language).toBe('english');
for (const sql of state.calls) {
expect(sql).not.toContain('DROP TABLE');
}
});
test('empty inventory still completes successfully', async () => {
const state: MockState = { calls: [], rowsToReturn: { pages: 0, chunks: 0 } };
const engine = makeMockEngine(state);
const result = await runReindexSearchVector(engine, { yes: true, json: true });
expect(result.status).toBe('ok');
expect(result.pagesUpdated).toBe(0);
expect(result.chunksUpdated).toBe(0);
expect(result.triggersRecreated).toBe(2);
});
test('result includes durationMs', async () => {
const state: MockState = { calls: [], rowsToReturn: { pages: 1, chunks: 1 } };
const engine = makeMockEngine(state);
const result = await runReindexSearchVector(engine, { yes: true, json: true });
expect(typeof result.durationMs).toBe('number');
expect(result.durationMs).toBeGreaterThanOrEqual(0);
});
});