diff --git a/README.md b/README.md index 9807cadb8..1f87d5883 100644 --- a/README.md +++ b/README.md @@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. +**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer +export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer +export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese) +``` + +List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +gbrain reindex-search-vector --dry-run # preview row counts +gbrain reindex-search-vector --yes # recreate triggers + backfill +``` + +The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe. + **43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. **Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index b40e6a45f..69339f35b 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -36,6 +36,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/post-upgrade-reembed.ts` — Pure functions backing the `gbrain upgrade` chunker-bump cost prompt. `computeReembedEstimate(engine, model)` queries real SQL (`COUNT(*)` + `COALESCE(SUM(LENGTH(compiled_truth)) + SUM(LENGTH(timeline)), 0)`) on `pages WHERE chunker_version < MARKDOWN_CHUNKER_VERSION`. `formatReembedPrompt(est, graceSeconds)` is the stderr-line formatter. `runPostUpgradeReembedPrompt(engine, model, opts)` orchestrates the 10-second Ctrl-C window; TTY-only wait (non-TTY auto-proceeds for CI / cron); `GBRAIN_NO_REEMBED=1` bails with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait. - `src/commands/reindex.ts` — `gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--no-embed] [--repo PATH]`. Walks `pages WHERE page_kind = 'markdown' AND chunker_version < MARKDOWN_CHUNKER_VERSION` in 100-row batches ordered by id. Rows with non-null `source_path` re-import via `importFromFile`; rows without fall back to `importFromContent`. Both paths pass `forceRechunk: true` to bypass `importFromContent`'s `content_hash` short-circuit — without it the chunker version bump never reaches pages whose source content hasn't changed, AND the stripFactsFence privacy strip never applies to pre-strip chunks. Idempotent — partial-completion re-runs pick up via id-ordered batches. Wired into `src/commands/upgrade.ts:runPostUpgrade` after `apply-migrations`. The DB-only fallback (no source file on disk) does NOT pass body-only `compiled_truth` to `importFromContent` (that path re-parses with EMPTY frontmatter and OVERWRITES the page's real frontmatter/title/timeline); it `getPage`+`getTags`, reconstructs FULL markdown via `serializeMarkdown(frontmatter, compiled_truth, timeline, {type, title, tags})`, and re-imports THAT so re-chunking a DB-only page preserves everything while bumping `chunker_version`. Pinned by `test/reindex-preserve-tags.test.ts`. - `src/commands/reindex-code.ts` — `gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). `spend.posture=tokenmax` OR an explicit `--max-cost off`/`unlimited` makes the gate informational and proceeds (#2139); `--max-cost off` also disables the runtime BudgetTracker cap. Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`. +- `src/core/fts-language.ts` — Single source for the Postgres text-search configuration name used by FTS. `getFtsLanguage()` resolves `GBRAIN_FTS_LANGUAGE` (default `english`), validates against `/^[a-z][a-z0-9_]*$/` (tsvector config names can't be bound as parameters, so the value is interpolated into raw SQL — the allowlist regex is the injection guard; invalid values warn once and fall back to `english`), and caches on first read (`resetFtsLanguageCache()` is test-only). Consumed by both engines' `searchKeyword`/`searchKeywordChunks` (`websearch_to_tsquery` query side), the `configurable_fts_language` migration, and `reindex-search-vector` (write-side trigger functions). Pinned by `test/fts-language.serial.test.ts` + `test/fts-language-migration.serial.test.ts` (includes the `'; DROP TABLE pages; --` injection cases). +- `src/commands/reindex-search-vector.ts` — `gbrain reindex-search-vector [--dry-run] [--yes] [--json]`. Escape hatch for changing `GBRAIN_FTS_LANGUAGE` after the `configurable_fts_language` migration has run (the migration shows applied and is skipped): recreates `update_page_search_vector` + `update_chunk_search_vector` with the configured language — bodies mirror the migration's and KEEP the `SET search_path = pg_catalog, public` hardening (CREATE OR REPLACE resets proconfig) — then backfills `pages` (UPDATE-to-self re-fires the trigger) and `content_chunks` (direct vector recompute) in id-keyset batches of `BACKFILL_BATCH_SIZE` (5000) via `UPDATE … WHERE id IN (SELECT … LIMIT n) RETURNING id`, streaming phases `reindex_search_vector.pages`/`.chunks` through the shared progress reporter (stderr). Confirmation gate: `--yes`, or an interactive TTY [y/N]; `--json` does NOT bypass the gate (non-TTY without `--yes` refuses with a ConfirmationRequired envelope, exit 2). Idempotent. Pinned by `test/reindex-search-vector.serial.test.ts`. - `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path. - `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column). diff --git a/docs/guides/multi-language-fts.md b/docs/guides/multi-language-fts.md new file mode 100644 index 000000000..1e5a2fe03 --- /dev/null +++ b/docs/guides/multi-language-fts.md @@ -0,0 +1,97 @@ +# Multi-language full-text search + +GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery). +The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE` +environment variable. Default: `english`. + +## How it works + +Postgres text-search configurations control stemming and stop-word removal. +`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on +both sides of the search: + +- **Query side** — `websearch_to_tsquery('', $query)` in both engines + (Postgres and PGLite). +- **Write side** — the `update_page_search_vector` and + `update_chunk_search_vector` trigger functions that populate + `pages.search_vector` and `content_chunks.search_vector`. + +The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever +interpolated into SQL (tsvector functions don't accept parameterized config +names). Invalid values fall back to `english` with a warning. + +## Built-in languages + +Set the env var to any configuration your Postgres instance ships: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +export GBRAIN_FTS_LANGUAGE=spanish +export GBRAIN_FTS_LANGUAGE=german +``` + +List what's available: + +```sql +SELECT cfgname FROM pg_ts_config; +``` + +PGLite (the embedded default engine) ships the same built-in snowball +configurations as stock Postgres. + +## First install vs. changing language later + +On first install (or upgrade), the `configurable_fts_language` schema +migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with +that language. After the migration has run, changing the env var alone does +NOT retokenize existing rows — the migration shows as applied and is skipped. +Use the explicit command: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +gbrain reindex-search-vector --dry-run # preview: language + row counts +gbrain reindex-search-vector --yes # recreate triggers + backfill +``` + +The command recreates both trigger functions under the new language and +backfills every existing `pages` and `content_chunks` row in batches, +streaming progress to stderr. It is idempotent: re-running with the same +language produces identical vectors. `--json` prints a machine-readable +result envelope but still requires `--yes` (or an interactive confirm). + +## Recipe: accent-insensitive Portuguese (`pt_br`) + +Brazilian Portuguese content often mixes accented and unaccented spellings +("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via +the `unaccent` extension, then stems with the portuguese snowball dictionary: + +```sql +CREATE EXTENSION IF NOT EXISTS unaccent; + +CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese); + +ALTER TEXT SEARCH CONFIGURATION pt_br + ALTER MAPPING FOR hword, hword_part, word + WITH unaccent, portuguese_stem; +``` + +Then point GBrain at it: + +```bash +export GBRAIN_FTS_LANGUAGE=pt_br +gbrain reindex-search-vector --yes +``` + +Note: custom configurations require a real Postgres instance (e.g. the +Supabase engine). The config must exist BEFORE the migration or the reindex +command runs, or Postgres will reject the trigger recreation with +`text search configuration "pt_br" does not exist`. + +## Caveats + +- One language per brain: the setting is global to the database, not + per-source. Mixed-language brains should pick the dominant language (the + vector-search arm is language-agnostic and covers the rest). +- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that + writes to the brain (CLI shells, MCP server, cron jobs) — a writer without + the env var tokenizes new rows in `english` until the next reindex. diff --git a/llms-full.txt b/llms-full.txt index 1a9710038..389b5ccfe 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1752,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. +**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer +export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer +export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese) +``` + +List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +gbrain reindex-search-vector --dry-run # preview row counts +gbrain reindex-search-vector --yes # recreate triggers + backfill +``` + +The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe. + **43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. **Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). diff --git a/src/cli.ts b/src/cli.ts index 3e14c39f9..53622bb05 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown { } // CLI-only commands that bypass the operation layer -export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch']); +export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -2001,6 +2001,15 @@ async function handleCliOnly(command: string, args: string[]) { await runReindexCodeCli(engine, args); break; } + case 'reindex-search-vector': { + // Explicit recreate of FTS trigger functions + batched backfill, + // honoring GBRAIN_FTS_LANGUAGE. Use after changing the language + // env var on a brain that already ran the configurable_fts_language + // migration. + const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts'); + await runReindexSearchVectorCli(engine, args); + break; + } case 'reindex-frontmatter': { // v0.29.1: recovery / explicit-rebuild path for pages.effective_date. // Mirror of reindex-code shape. Wraps the shared library function in @@ -2336,6 +2345,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) query --symbol-kind Filter to symbol type (function|class|method|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) + reindex-search-vector [--dry-run] [--yes] [--json] + Recreate FTS triggers + backfill under + $GBRAIN_FTS_LANGUAGE (default 'english') sync --strategy code Sync code files into the brain JOBS (Minions) diff --git a/src/commands/reindex-search-vector.ts b/src/commands/reindex-search-vector.ts new file mode 100644 index 000000000..59524511e --- /dev/null +++ b/src/commands/reindex-search-vector.ts @@ -0,0 +1,277 @@ +/** + * `gbrain reindex-search-vector` — recreate FTS trigger functions and + * backfill existing rows under the language configured via + * GBRAIN_FTS_LANGUAGE. + * + * Why this command exists: schema migration v123 (configurable_fts_language) + * stamps the trigger functions with the configured language at first apply. + * After that, changing the env var has no effect on the write side because + * v123 already shows as "applied" — the migrations runner will skip it. + * This command is the documented escape hatch: it re-runs the same + * recreate-and-backfill logic v123 uses, gated on an explicit user + * action so the operation is intentional and visible (writes touch + * every row in pages and content_chunks). + * + * Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces + * the same trigger function bodies and the same tokenized vectors. + * + * Flags: + * --dry-run Show what would happen, exit 0 without touching DB. + * --yes Skip interactive [y/N]. Required for non-TTY (including --json). + * --json Machine-readable result envelope. Does NOT imply --yes. + * + * Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE) + * so a large brain never holds one giant row lock, and streams progress + * through the shared reporter (stderr; stdout stays clean for --json). + * + * Cost: trigger recreate is sub-millisecond. Backfill is one tsvector + * rebuild per page + per chunk. On a 20K-page brain with 80K chunks, + * expect ~5-15s depending on Postgres CPU and content size. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { getFtsLanguage } from '../core/fts-language.ts'; +import { createInterface } from 'readline'; +import { createProgress } from '../core/progress.ts'; +import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; + +export interface ReindexSearchVectorOpts { + dryRun?: boolean; + yes?: boolean; + json?: boolean; +} + +export interface ReindexSearchVectorResult { + status: 'ok' | 'dry_run' | 'cancelled'; + language: string; + pagesUpdated: number; + chunksUpdated: number; + triggersRecreated: number; + durationMs: number; +} + +interface CountRow { + pages: number; + chunks: number; +} + +/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */ +export const BACKFILL_BATCH_SIZE = 5000; + +/** + * Keyset-batched UPDATE: applies `setClause` to `table` rows where + * search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking + * the shared progress reporter after each batch. Terminates when a batch + * returns fewer rows than the batch size (or none). + */ +async function batchedBackfill( + engine: BrainEngine, + table: 'pages' | 'content_chunks', + setClause: string, + tick: (n: number) => void +): Promise { + let cursor = 0; + for (;;) { + const rows = await engine.executeRaw<{ id: number }>(` + UPDATE ${table} SET ${setClause} + WHERE id IN ( + SELECT id FROM ${table} + WHERE search_vector IS NOT NULL AND id > ${cursor} + ORDER BY id + LIMIT ${BACKFILL_BATCH_SIZE} + ) + RETURNING id + `); + if (rows.length === 0) break; + tick(rows.length); + cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor); + if (rows.length < BACKFILL_BATCH_SIZE) break; + } +} + +/** + * Programmatic entrypoint — takes a typed opts object. Used by tests and + * future internal callers. The CLI wrapper is `runReindexSearchVectorCli` + * defined at the bottom of this file. + */ +export async function runReindexSearchVector( + engine: BrainEngine, + opts: ReindexSearchVectorOpts +): Promise { + const lang = getFtsLanguage(); + const startedAt = Date.now(); + + // Inventory: how many rows will the backfill touch? + const counts = await engine.executeRaw( + `SELECT + (SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages, + (SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks` + ); + const pagesCount = counts[0]?.pages ?? 0; + const chunksCount = counts[0]?.chunks ?? 0; + + if (opts.dryRun) { + const result: ReindexSearchVectorResult = { + status: 'dry_run', + language: lang, + pagesUpdated: pagesCount, + chunksUpdated: chunksCount, + triggersRecreated: 0, + durationMs: Date.now() - startedAt, + }; + if (opts.json) { + console.log(JSON.stringify(result, null, 2)); + } else { + console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`); + console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`); + console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`); + } + return result; + } + + // Confirm unless --yes. --json does NOT bypass the gate — a machine + // caller must pass --yes explicitly (mirrors reindex-code, #1784). + if (!opts.yes) { + if (!process.stdin.isTTY) { + if (opts.json) { + console.log(JSON.stringify({ + error: { + class: 'ConfirmationRequired', + code: 'reindex_requires_yes', + message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`, + hint: 'Pass --yes to proceed, or --dry-run to preview.', + }, + language: lang, + pages: pagesCount, + chunks: chunksCount, + })); + } else { + console.error('Refusing to run without --yes in non-TTY environment.'); + } + process.exit(2); + } + + const rl = createInterface({ input: process.stdin, output: process.stdout }); + const answer = await new Promise(resolve => { + rl.question( + `Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `, + resolve + ); + }); + rl.close(); + + if (!/^y(es)?$/i.test(answer.trim())) { + const result: ReindexSearchVectorResult = { + status: 'cancelled', + language: lang, + pagesUpdated: 0, + chunksUpdated: 0, + triggersRecreated: 0, + durationMs: Date.now() - startedAt, + }; + if (opts.json) { + console.log(JSON.stringify(result, null, 2)); + } else { + console.log('Cancelled.'); + } + return result; + } + } + + // Recreate trigger functions. The strings are intentionally identical to + // the v123 migration body — keeping them in lockstep is the contract. + // `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening: + // CREATE OR REPLACE resets proconfig, so omitting it here would strip the + // hardening from every brain that runs this command. + const recreatePagesFn = ` + CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$ + DECLARE + timeline_text TEXT; + BEGIN + SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '') + INTO timeline_text + FROM timeline_entries + WHERE page_id = NEW.id; + + NEW.search_vector := + setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') || + setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') || + setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') || + setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C'); + + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + const recreateChunksFn = ` + CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$ + BEGIN + NEW.search_vector := + setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B'); + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + await engine.executeRaw(recreatePagesFn); + await engine.executeRaw(recreateChunksFn); + + const progress = createProgress(cliOptsToProgressOptions(getCliOptions())); + + // Backfill: UPDATE-to-self forces the pages trigger to re-fire + // (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a + // direct vector compute since the column itself is what we want. + progress.start('reindex_search_vector.pages', pagesCount); + await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n)); + progress.finish(); + + progress.start('reindex_search_vector.chunks', chunksCount); + await batchedBackfill( + engine, + 'content_chunks', + `search_vector = + setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`, + n => progress.tick(n) + ); + progress.finish(); + + const result: ReindexSearchVectorResult = { + status: 'ok', + language: lang, + pagesUpdated: pagesCount, + chunksUpdated: chunksCount, + triggersRecreated: 2, + durationMs: Date.now() - startedAt, + }; + + if (opts.json) { + console.log(JSON.stringify(result, null, 2)); + } else { + console.log(`✅ Recreated 2 trigger functions with language='${lang}'`); + console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`); + } + + return result; +} + +/** + * CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector. + * Matches the style of `reindex-code`: --dry-run, --yes/-y, --json. + * + * Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes. + */ +export async function runReindexSearchVectorCli( + engine: BrainEngine, + args: string[] +): Promise { + const dryRun = args.includes('--dry-run'); + const yes = args.includes('--yes') || args.includes('-y'); + const json = args.includes('--json'); + + await runReindexSearchVector(engine, { dryRun, yes, json }); +} diff --git a/src/core/fts-language.ts b/src/core/fts-language.ts new file mode 100644 index 000000000..569c0410f --- /dev/null +++ b/src/core/fts-language.ts @@ -0,0 +1,69 @@ +/** + * Full-text search language configuration. + * + * Postgres tsvector/tsquery require a text search configuration name (e.g. + * 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded + * 'english' across engines and trigger functions, which broke search + * quality for non-English brains (no stemming, no stop-word removal). + * + * This helper centralizes the choice. Default stays 'english' for backward + * compatibility — only users who set GBRAIN_FTS_LANGUAGE see different + * behavior. + * + * Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent + + * portuguese stemmer) are supported as long as the configuration exists + * in the target Postgres instance. See docs/guides/multi-language-fts.md + * for setup instructions. + * + * Validation: only allow lowercase letters, digits, and underscores. This + * prevents SQL injection when the value is interpolated into queries + * (Postgres tsvector functions don't accept parameterized config names — + * they must be literals or identifiers). + */ + +const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/; +const DEFAULT_LANGUAGE = 'english'; + +let cachedLanguage: string | null = null; + +/** + * Returns the configured Postgres text search configuration name. + * + * Resolution order: + * 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid) + * 2. 'english' (default — preserves existing behavior) + * + * The return value is safe to interpolate directly into SQL because it + * passes the VALID_CONFIG_NAME guard. If validation fails, falls back to + * the default and emits a one-time warning. + * + * Cached on first call; reset with `resetFtsLanguageCache()` (test only). + */ +export function getFtsLanguage(): string { + if (cachedLanguage !== null) return cachedLanguage; + + const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim(); + if (!raw) { + cachedLanguage = DEFAULT_LANGUAGE; + return cachedLanguage; + } + + if (!VALID_CONFIG_NAME.test(raw)) { + console.warn( + `[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` + + `Falling back to '${DEFAULT_LANGUAGE}'.` + ); + cachedLanguage = DEFAULT_LANGUAGE; + return cachedLanguage; + } + + cachedLanguage = raw; + return cachedLanguage; +} + +/** + * Resets the cached language. Tests only — don't use in production code. + */ +export function resetFtsLanguageCache(): void { + cachedLanguage = null; +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index c9c3a1581..7b95a5767 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -1,5 +1,6 @@ import type { BrainEngine } from './engine.ts'; import { slugifyPath } from './sync.ts'; +import { getFtsLanguage } from './fts-language.ts'; /** * Schema migrations — run automatically on initSchema(). @@ -5505,6 +5506,96 @@ export const MIGRATIONS: Migration[] = [ WHERE dimension IS NOT NULL; `, }, + { + version: 123, + name: 'configurable_fts_language', + // Recreate the two search_vector trigger functions using the language + // configured via GBRAIN_FTS_LANGUAGE (default 'english'). Idempotent: + // CREATE OR REPLACE swaps the function body atomically; no trigger + // recreation needed since the trigger references the function by name. + // + // Why a handler instead of a static SQL string: Postgres tsvector + // functions don't accept parameterized config names — the language + // must be a literal in the SQL. getFtsLanguage() validates the value + // (lowercase letters/digits/underscores only) before interpolation. + // + // Function bodies mirror schema.sql / pglite-schema.ts exactly — + // INCLUDING the `SET search_path = pg_catalog, public` hardening from + // v120/#1647 (CREATE OR REPLACE resets proconfig, so omitting it here + // would silently strip the hardening on every upgraded brain). Only + // the text-search config name is parameterized. Keep all copies in + // sync when the trigger logic changes. + // + // Backfill: after recreating the functions, re-tokenize existing rows + // under the new language. Skipped when the configured language is + // 'english' (trigger output identical — re-tokenizing is wasted I/O). + // To change language after this migration has run, use + // `gbrain reindex-search-vector`. + sql: '', + handler: async (engine) => { + const lang = getFtsLanguage(); + + const recreatePagesFn = ` + CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$ + DECLARE + timeline_text TEXT; + BEGIN + SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '') + INTO timeline_text + FROM timeline_entries + WHERE page_id = NEW.id; + + NEW.search_vector := + setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') || + setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') || + setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') || + setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C'); + + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + const recreateChunksFn = ` + CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$ + BEGIN + NEW.search_vector := + setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B'); + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + await engine.executeRaw(recreatePagesFn); + await engine.executeRaw(recreateChunksFn); + + if (lang === 'english') { + console.log(` v123: trigger functions recreated with language='english' (default — no backfill needed)`); + return; + } + + // Backfill existing rows under the new tokenizer. UPDATE-to-same-value + // re-fires the pages trigger; chunks are rewritten directly with the + // same expression as the trigger. + await engine.executeRaw(` + UPDATE pages SET id = id + WHERE search_vector IS NOT NULL; + `); + + await engine.executeRaw(` + UPDATE content_chunks + SET search_vector = + setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B') + WHERE search_vector IS NOT NULL; + `); + + console.log(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows`); + }, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index d30738f24..8239ec354 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -24,6 +24,7 @@ import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts'; import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; import { DELETE_BATCH_SIZE } from './engine-constants.ts'; import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts'; +import { getFtsLanguage } from './fts-language.ts'; import type { Page, PageInput, PageFilters, PageType, Chunk, ChunkInput, StaleChunkRow, StalePageRow, @@ -1625,20 +1626,24 @@ export class PGLiteEngine implements BrainEngine { extraFilter += ` AND p.source_id = $${params.length}`; } + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); + const { rows } = await this.db.query( `WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} -- v0.27.1: hide image rows from default text-keyword search so -- OCR text doesn't drown text-page hits. Image-similarity queries -- run a separate vector path on embedding_image. @@ -1857,20 +1862,23 @@ export class PGLiteEngine implements BrainEngine { } // visibilityClause already declared above (v0.32.7: hoisted so CJK branch can reuse). + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); const { rows } = await this.db.query( `SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} ORDER BY score DESC LIMIT $2 OFFSET $3`, params diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 2fe57e517..4fdaaf3ca 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -34,6 +34,7 @@ import { COLUMN_NAME_REGEX, EmbeddingColumnNotRegisteredError, } from './search/embedding-column.ts'; +import { getFtsLanguage } from './fts-language.ts'; import type { Page, PageInput, PageFilters, PageType, Chunk, ChunkInput, StaleChunkRow, StalePageRow, @@ -1653,6 +1654,9 @@ export class PostgresEngine implements BrainEngine { // column lookup. NOT bypassed by detail=high — soft-delete is a contract, // not a temporal preference. const visibilityClause = buildVisibilityClause('p', 's'); + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); const rawQuery = ` WITH ranked_chunks AS ( @@ -1660,11 +1664,11 @@ export class PostgresEngine implements BrainEngine { p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${typeClause} ${typesClause} ${excludeSlugsClause} @@ -1792,18 +1796,21 @@ export class PostgresEngine implements BrainEngine { // v0.26.5: visibility filter for searchKeywordChunks (anchor primitive). const visibilityClause = buildVisibilityClause('p', 's'); + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); const rawQuery = ` SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score, false AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${typeClause} ${typesClause} ${excludeSlugsClause} diff --git a/test/fts-language-migration.serial.test.ts b/test/fts-language-migration.serial.test.ts new file mode 100644 index 000000000..da7841da0 --- /dev/null +++ b/test/fts-language-migration.serial.test.ts @@ -0,0 +1,119 @@ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import type { BrainEngine } from '../src/core/engine.ts'; +import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts'; +import { resetFtsLanguageCache } from '../src/core/fts-language.ts'; + +const ENV_KEY = 'GBRAIN_FTS_LANGUAGE'; +const originalLang = process.env[ENV_KEY]; + +beforeEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +afterEach(() => { + delete process.env[ENV_KEY]; + if (originalLang !== undefined) process.env[ENV_KEY] = originalLang; + resetFtsLanguageCache(); +}); + +describe('configurable_fts_language migration', () => { + test('migration is registered', () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + expect(ftsMig).toBeDefined(); + expect(ftsMig?.version).toBeGreaterThan(115); + }); + + test('fts migration is the latest migration', () => { + expect(MIGRATIONS.find(m => m.name === 'configurable_fts_language')?.version).toBe(LATEST_VERSION); + }); + + test('ftsMig uses handler (not static SQL) because language interpolation is dynamic', () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + expect(ftsMig?.sql).toBe(''); + expect(ftsMig?.handler).toBeTypeOf('function'); + }); + + test('ftsMig handler is async', () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + // Async function check: the constructor name is 'AsyncFunction' + expect(ftsMig?.handler?.constructor.name).toBe('AsyncFunction'); + }); + + test('migration handler issues recreate-function calls (smoke check via mock engine)', async () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + const calls: string[] = []; + + const mockEngine = { + executeRaw: async (sql: string) => { + calls.push(sql); + return []; + }, + } as unknown as BrainEngine; + + process.env[ENV_KEY] = 'english'; + resetFtsLanguageCache(); + + await ftsMig?.handler?.(mockEngine); + + // Default 'english' \u2014 no backfill, only 2 CREATE OR REPLACE calls. + expect(calls.length).toBe(2); + expect(calls[0]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector'); + expect(calls[0]).toContain("to_tsvector('english'"); + expect(calls[1]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector'); + expect(calls[1]).toContain("to_tsvector('english'"); + // v120/#1647 hardening must survive the CREATE OR REPLACE (which resets + // proconfig): both recreated bodies pin search_path. + expect(calls[0]).toContain('SET search_path = pg_catalog, public'); + expect(calls[1]).toContain('SET search_path = pg_catalog, public'); + }); + + test('non-english language triggers backfill', async () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + const calls: string[] = []; + + const mockEngine = { + executeRaw: async (sql: string) => { + calls.push(sql); + return []; + }, + } as unknown as BrainEngine; + + process.env[ENV_KEY] = 'pt_br'; + resetFtsLanguageCache(); + + await ftsMig?.handler?.(mockEngine); + + // pt_br \u2014 2 CREATE + 2 backfill UPDATEs = 4 calls + expect(calls.length).toBe(4); + expect(calls[0]).toContain("to_tsvector('pt_br'"); + expect(calls[1]).toContain("to_tsvector('pt_br'"); + expect(calls[2]).toMatch(/UPDATE pages/); + expect(calls[3]).toContain("to_tsvector('pt_br'"); + expect(calls[3]).toMatch(/UPDATE content_chunks/); + }); + + test('invalid language falls back to english (no SQL injection)', async () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + const calls: string[] = []; + + const mockEngine = { + executeRaw: async (sql: string) => { + calls.push(sql); + return []; + }, + } as unknown as BrainEngine; + + process.env[ENV_KEY] = "english'; DROP TABLE pages; --"; + resetFtsLanguageCache(); + + await ftsMig?.handler?.(mockEngine); + + // Falls back to english: 2 CREATE OR REPLACE only, no DROP TABLE in any SQL. + expect(calls.length).toBe(2); + for (const sql of calls) { + expect(sql).not.toContain('DROP TABLE'); + expect(sql).toContain("to_tsvector('english'"); + } + }); +}); diff --git a/test/fts-language.serial.test.ts b/test/fts-language.serial.test.ts new file mode 100644 index 000000000..ea6c4cfcb --- /dev/null +++ b/test/fts-language.serial.test.ts @@ -0,0 +1,93 @@ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import { getFtsLanguage, resetFtsLanguageCache } from '../src/core/fts-language.ts'; + +const ENV_KEY = 'GBRAIN_FTS_LANGUAGE'; + +beforeEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +afterEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +describe('getFtsLanguage', () => { + test('defaults to english when env is unset', () => { + expect(getFtsLanguage()).toBe('english'); + }); + + test('defaults to english when env is empty string', () => { + process.env[ENV_KEY] = ''; + expect(getFtsLanguage()).toBe('english'); + }); + + test('defaults to english when env is whitespace', () => { + process.env[ENV_KEY] = ' '; + expect(getFtsLanguage()).toBe('english'); + }); + + test('reads valid pt_br config', () => { + process.env[ENV_KEY] = 'pt_br'; + expect(getFtsLanguage()).toBe('pt_br'); + }); + + test('reads valid simple language name', () => { + process.env[ENV_KEY] = 'spanish'; + expect(getFtsLanguage()).toBe('spanish'); + }); + + test('reads name with underscores and digits', () => { + process.env[ENV_KEY] = 'custom_lang_v2'; + expect(getFtsLanguage()).toBe('custom_lang_v2'); + }); + + test('rejects names with quotes (SQL injection guard)', () => { + process.env[ENV_KEY] = "english'; DROP TABLE pages; --"; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects names with spaces', () => { + process.env[ENV_KEY] = 'pt br'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects names with hyphens', () => { + process.env[ENV_KEY] = 'pt-br'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects names starting with digit', () => { + process.env[ENV_KEY] = '1lang'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects uppercase (Postgres config names are lowercase)', () => { + process.env[ENV_KEY] = 'English'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('caches after first read', () => { + process.env[ENV_KEY] = 'pt_br'; + expect(getFtsLanguage()).toBe('pt_br'); + + // Mutate env after first read \u2014 cached value wins. + process.env[ENV_KEY] = 'spanish'; + expect(getFtsLanguage()).toBe('pt_br'); + }); + + test('resetFtsLanguageCache clears cache', () => { + process.env[ENV_KEY] = 'pt_br'; + expect(getFtsLanguage()).toBe('pt_br'); + + resetFtsLanguageCache(); + process.env[ENV_KEY] = 'spanish'; + expect(getFtsLanguage()).toBe('spanish'); + }); + + test('trims surrounding whitespace from valid value', () => { + process.env[ENV_KEY] = ' pt_br '; + expect(getFtsLanguage()).toBe('pt_br'); + }); +}); diff --git a/test/reindex-search-vector.serial.test.ts b/test/reindex-search-vector.serial.test.ts new file mode 100644 index 000000000..e8dd3a1a9 --- /dev/null +++ b/test/reindex-search-vector.serial.test.ts @@ -0,0 +1,152 @@ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import type { BrainEngine } from '../src/core/engine.ts'; +import { runReindexSearchVector } from '../src/commands/reindex-search-vector.ts'; +import { resetFtsLanguageCache } from '../src/core/fts-language.ts'; + +const ENV_KEY = 'GBRAIN_FTS_LANGUAGE'; +const originalLang = process.env[ENV_KEY]; + +interface MockState { + calls: string[]; + rowsToReturn: { pages: number; chunks: number }; +} + +function makeMockEngine(state: MockState): BrainEngine { + return { + executeRaw: async (sql: string) => { + state.calls.push(sql); + // Inventory query — return the configured counts + if (sql.includes('SELECT') && sql.includes('FROM pages WHERE search_vector')) { + return [{ pages: state.rowsToReturn.pages, chunks: state.rowsToReturn.chunks }]; + } + return []; + }, + } as unknown as BrainEngine; +} + +beforeEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +afterEach(() => { + delete process.env[ENV_KEY]; + if (originalLang !== undefined) process.env[ENV_KEY] = originalLang; + resetFtsLanguageCache(); +}); + +describe('runReindexSearchVector', () => { + test('--dry-run does not issue any DDL or backfill SQL', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 100, chunks: 500 } }; + const engine = makeMockEngine(state); + + process.env[ENV_KEY] = 'pt_br'; + resetFtsLanguageCache(); + + const result = await runReindexSearchVector(engine, { dryRun: true, json: true }); + + expect(result.status).toBe('dry_run'); + expect(result.language).toBe('pt_br'); + expect(result.pagesUpdated).toBe(100); + expect(result.chunksUpdated).toBe(500); + expect(result.triggersRecreated).toBe(0); + + // Only the inventory query — no CREATE OR REPLACE, no UPDATE. + expect(state.calls.length).toBe(1); + expect(state.calls[0]).toContain('SELECT'); + expect(state.calls[0]).not.toContain('CREATE OR REPLACE'); + expect(state.calls[0]).not.toContain('UPDATE'); + }); + + test('--yes recreates triggers + backfills with configured language', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 50, chunks: 200 } }; + const engine = makeMockEngine(state); + + process.env[ENV_KEY] = 'pt_br'; + resetFtsLanguageCache(); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.status).toBe('ok'); + expect(result.language).toBe('pt_br'); + expect(result.triggersRecreated).toBe(2); + expect(result.pagesUpdated).toBe(50); + expect(result.chunksUpdated).toBe(200); + + // 1 inventory + 2 CREATE + 2 backfill batches (mock returns no rows, so + // the keyset loop terminates after the first batch per table) = 5 calls + expect(state.calls.length).toBe(5); + expect(state.calls[1]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector'); + expect(state.calls[1]).toContain("to_tsvector('pt_br'"); + expect(state.calls[2]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector'); + expect(state.calls[2]).toContain("to_tsvector('pt_br'"); + expect(state.calls[3]).toMatch(/UPDATE pages/); + expect(state.calls[4]).toMatch(/UPDATE content_chunks/); + expect(state.calls[4]).toContain("to_tsvector('pt_br'"); + // v120/#1647 hardening must survive the CREATE OR REPLACE (which resets + // proconfig): both recreated bodies pin search_path. + expect(state.calls[1]).toContain('SET search_path = pg_catalog, public'); + expect(state.calls[2]).toContain('SET search_path = pg_catalog, public'); + }); + + test('default english language still recreates + backfills (no shortcut here)', async () => { + // Note: unlike the configurable_fts_language migration, the CLI command + // intentionally backfills even for english. The user explicitly asked for + // it, so we honor it. The migration skips backfill for english because it + // auto-runs on first apply. + const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } }; + const engine = makeMockEngine(state); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.status).toBe('ok'); + expect(result.language).toBe('english'); + expect(state.calls.length).toBe(5); + + // Trigger recreates (calls 1, 2) and chunks backfill (call 4) embed the + // language literal. Pages backfill (call 3) is UPDATE-to-self that + // re-fires the trigger, so the language literal lives in the trigger + // function body — not in the UPDATE statement. + expect(state.calls[1]).toContain("'english'"); + expect(state.calls[2]).toContain("'english'"); + expect(state.calls[3]).toMatch(/UPDATE pages/); + expect(state.calls[4]).toContain("'english'"); + }); + + test('SQL injection attempt falls back to english', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } }; + const engine = makeMockEngine(state); + + process.env[ENV_KEY] = "english'; DROP TABLE pages; --"; + resetFtsLanguageCache(); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.language).toBe('english'); + for (const sql of state.calls) { + expect(sql).not.toContain('DROP TABLE'); + } + }); + + test('empty inventory still completes successfully', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 0, chunks: 0 } }; + const engine = makeMockEngine(state); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.status).toBe('ok'); + expect(result.pagesUpdated).toBe(0); + expect(result.chunksUpdated).toBe(0); + expect(result.triggersRecreated).toBe(2); + }); + + test('result includes durationMs', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 1, chunks: 1 } }; + const engine = makeMockEngine(state); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(typeof result.durationMs).toBe('number'); + expect(result.durationMs).toBeGreaterThanOrEqual(0); + }); +});