mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
feat(search): configurable FTS language + reindex command (lands #580/#581/#582)
Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r (#580 env-var language for query+write side, #581 migration backfill, #582 gbrain reindex-search-vector), rebased onto current master with the security review's required fixes applied: - Migration renumbered v116 -> v123 (master's v116 was already claimed by code_edges_source_backfill; master is at v122). - Restored the v120/#1647 search_path hardening: all four CREATE OR REPLACE trigger-function bodies (migration handler + reindex command) now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE resets proconfig and would otherwise strip the hardening on upgrade. - reindex-search-vector: shared progress reporter (stderr phases reindex_search_vector.pages/.chunks), id-keyset batched backfill (5000 rows/UPDATE) instead of single whole-table statements, and --json no longer bypasses the --yes/TTY confirmation gate. - Allowlist validation regex + injection tests kept exactly as authored. - Stale v33/v116 comments swept; docs/guides/multi-language-fts.md written (README referenced it but no PR added it); llms bundles regenerated via bun run build:llms. - Trilogy tests quarantined as *.serial.test.ts (env mutation, per check-test-isolation). Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese produces portuguese-stemmed vectors with search_path pinned; the reindex command retokenizes an english brain to portuguese in place. Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Rafael Reis
Claude Fable 5
parent
54a8070640
commit
10c24d76c3
@@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
|
||||
```
|
||||
|
||||
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
@@ -36,6 +36,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/post-upgrade-reembed.ts` — Pure functions backing the `gbrain upgrade` chunker-bump cost prompt. `computeReembedEstimate(engine, model)` queries real SQL (`COUNT(*)` + `COALESCE(SUM(LENGTH(compiled_truth)) + SUM(LENGTH(timeline)), 0)`) on `pages WHERE chunker_version < MARKDOWN_CHUNKER_VERSION`. `formatReembedPrompt(est, graceSeconds)` is the stderr-line formatter. `runPostUpgradeReembedPrompt(engine, model, opts)` orchestrates the 10-second Ctrl-C window; TTY-only wait (non-TTY auto-proceeds for CI / cron); `GBRAIN_NO_REEMBED=1` bails with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait.
|
||||
- `src/commands/reindex.ts` — `gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--no-embed] [--repo PATH]`. Walks `pages WHERE page_kind = 'markdown' AND chunker_version < MARKDOWN_CHUNKER_VERSION` in 100-row batches ordered by id. Rows with non-null `source_path` re-import via `importFromFile`; rows without fall back to `importFromContent`. Both paths pass `forceRechunk: true` to bypass `importFromContent`'s `content_hash` short-circuit — without it the chunker version bump never reaches pages whose source content hasn't changed, AND the stripFactsFence privacy strip never applies to pre-strip chunks. Idempotent — partial-completion re-runs pick up via id-ordered batches. Wired into `src/commands/upgrade.ts:runPostUpgrade` after `apply-migrations`. The DB-only fallback (no source file on disk) does NOT pass body-only `compiled_truth` to `importFromContent` (that path re-parses with EMPTY frontmatter and OVERWRITES the page's real frontmatter/title/timeline); it `getPage`+`getTags`, reconstructs FULL markdown via `serializeMarkdown(frontmatter, compiled_truth, timeline, {type, title, tags})`, and re-imports THAT so re-chunking a DB-only page preserves everything while bumping `chunker_version`. Pinned by `test/reindex-preserve-tags.test.ts`.
|
||||
- `src/commands/reindex-code.ts` — `gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). `spend.posture=tokenmax` OR an explicit `--max-cost off`/`unlimited` makes the gate informational and proceeds (#2139); `--max-cost off` also disables the runtime BudgetTracker cap. Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`.
|
||||
- `src/core/fts-language.ts` — Single source for the Postgres text-search configuration name used by FTS. `getFtsLanguage()` resolves `GBRAIN_FTS_LANGUAGE` (default `english`), validates against `/^[a-z][a-z0-9_]*$/` (tsvector config names can't be bound as parameters, so the value is interpolated into raw SQL — the allowlist regex is the injection guard; invalid values warn once and fall back to `english`), and caches on first read (`resetFtsLanguageCache()` is test-only). Consumed by both engines' `searchKeyword`/`searchKeywordChunks` (`websearch_to_tsquery` query side), the `configurable_fts_language` migration, and `reindex-search-vector` (write-side trigger functions). Pinned by `test/fts-language.serial.test.ts` + `test/fts-language-migration.serial.test.ts` (includes the `'; DROP TABLE pages; --` injection cases).
|
||||
- `src/commands/reindex-search-vector.ts` — `gbrain reindex-search-vector [--dry-run] [--yes] [--json]`. Escape hatch for changing `GBRAIN_FTS_LANGUAGE` after the `configurable_fts_language` migration has run (the migration shows applied and is skipped): recreates `update_page_search_vector` + `update_chunk_search_vector` with the configured language — bodies mirror the migration's and KEEP the `SET search_path = pg_catalog, public` hardening (CREATE OR REPLACE resets proconfig) — then backfills `pages` (UPDATE-to-self re-fires the trigger) and `content_chunks` (direct vector recompute) in id-keyset batches of `BACKFILL_BATCH_SIZE` (5000) via `UPDATE … WHERE id IN (SELECT … LIMIT n) RETURNING id`, streaming phases `reindex_search_vector.pages`/`.chunks` through the shared progress reporter (stderr). Confirmation gate: `--yes`, or an interactive TTY [y/N]; `--json` does NOT bypass the gate (non-TTY without `--yes` refuses with a ConfirmationRequired envelope, exit 2). Idempotent. Pinned by `test/reindex-search-vector.serial.test.ts`.
|
||||
- `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path.
|
||||
- `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column).
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
# Multi-language full-text search
|
||||
|
||||
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
|
||||
The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE`
|
||||
environment variable. Default: `english`.
|
||||
|
||||
## How it works
|
||||
|
||||
Postgres text-search configurations control stemming and stop-word removal.
|
||||
`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on
|
||||
both sides of the search:
|
||||
|
||||
- **Query side** — `websearch_to_tsquery('<lang>', $query)` in both engines
|
||||
(Postgres and PGLite).
|
||||
- **Write side** — the `update_page_search_vector` and
|
||||
`update_chunk_search_vector` trigger functions that populate
|
||||
`pages.search_vector` and `content_chunks.search_vector`.
|
||||
|
||||
The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever
|
||||
interpolated into SQL (tsvector functions don't accept parameterized config
|
||||
names). Invalid values fall back to `english` with a warning.
|
||||
|
||||
## Built-in languages
|
||||
|
||||
Set the env var to any configuration your Postgres instance ships:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
export GBRAIN_FTS_LANGUAGE=spanish
|
||||
export GBRAIN_FTS_LANGUAGE=german
|
||||
```
|
||||
|
||||
List what's available:
|
||||
|
||||
```sql
|
||||
SELECT cfgname FROM pg_ts_config;
|
||||
```
|
||||
|
||||
PGLite (the embedded default engine) ships the same built-in snowball
|
||||
configurations as stock Postgres.
|
||||
|
||||
## First install vs. changing language later
|
||||
|
||||
On first install (or upgrade), the `configurable_fts_language` schema
|
||||
migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with
|
||||
that language. After the migration has run, changing the env var alone does
|
||||
NOT retokenize existing rows — the migration shows as applied and is skipped.
|
||||
Use the explicit command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview: language + row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command recreates both trigger functions under the new language and
|
||||
backfills every existing `pages` and `content_chunks` row in batches,
|
||||
streaming progress to stderr. It is idempotent: re-running with the same
|
||||
language produces identical vectors. `--json` prints a machine-readable
|
||||
result envelope but still requires `--yes` (or an interactive confirm).
|
||||
|
||||
## Recipe: accent-insensitive Portuguese (`pt_br`)
|
||||
|
||||
Brazilian Portuguese content often mixes accented and unaccented spellings
|
||||
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
|
||||
the `unaccent` extension, then stems with the portuguese snowball dictionary:
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS unaccent;
|
||||
|
||||
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
|
||||
|
||||
ALTER TEXT SEARCH CONFIGURATION pt_br
|
||||
ALTER MAPPING FOR hword, hword_part, word
|
||||
WITH unaccent, portuguese_stem;
|
||||
```
|
||||
|
||||
Then point GBrain at it:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br
|
||||
gbrain reindex-search-vector --yes
|
||||
```
|
||||
|
||||
Note: custom configurations require a real Postgres instance (e.g. the
|
||||
Supabase engine). The config must exist BEFORE the migration or the reindex
|
||||
command runs, or Postgres will reject the trigger recreation with
|
||||
`text search configuration "pt_br" does not exist`.
|
||||
|
||||
## Caveats
|
||||
|
||||
- One language per brain: the setting is global to the database, not
|
||||
per-source. Mixed-language brains should pick the dominant language (the
|
||||
vector-search arm is language-agnostic and covers the rest).
|
||||
- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that
|
||||
writes to the brain (CLI shells, MCP server, cron jobs) — a writer without
|
||||
the env var tokenizes new rows in `english` until the next reindex.
|
||||
@@ -1752,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
|
||||
```
|
||||
|
||||
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
+13
-1
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch']);
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -2001,6 +2001,15 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runReindexCodeCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reindex-search-vector': {
|
||||
// Explicit recreate of FTS trigger functions + batched backfill,
|
||||
// honoring GBRAIN_FTS_LANGUAGE. Use after changing the language
|
||||
// env var on a brain that already ran the configurable_fts_language
|
||||
// migration.
|
||||
const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts');
|
||||
await runReindexSearchVectorCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reindex-frontmatter': {
|
||||
// v0.29.1: recovery / explicit-rebuild path for pages.effective_date.
|
||||
// Mirror of reindex-code shape. Wraps the shared library function in
|
||||
@@ -2336,6 +2345,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II)
|
||||
query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0)
|
||||
reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0)
|
||||
reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0)
|
||||
reindex-search-vector [--dry-run] [--yes] [--json]
|
||||
Recreate FTS triggers + backfill under
|
||||
$GBRAIN_FTS_LANGUAGE (default 'english')
|
||||
sync --strategy code Sync code files into the brain
|
||||
|
||||
JOBS (Minions)
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
/**
|
||||
* `gbrain reindex-search-vector` — recreate FTS trigger functions and
|
||||
* backfill existing rows under the language configured via
|
||||
* GBRAIN_FTS_LANGUAGE.
|
||||
*
|
||||
* Why this command exists: schema migration v123 (configurable_fts_language)
|
||||
* stamps the trigger functions with the configured language at first apply.
|
||||
* After that, changing the env var has no effect on the write side because
|
||||
* v123 already shows as "applied" — the migrations runner will skip it.
|
||||
* This command is the documented escape hatch: it re-runs the same
|
||||
* recreate-and-backfill logic v123 uses, gated on an explicit user
|
||||
* action so the operation is intentional and visible (writes touch
|
||||
* every row in pages and content_chunks).
|
||||
*
|
||||
* Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces
|
||||
* the same trigger function bodies and the same tokenized vectors.
|
||||
*
|
||||
* Flags:
|
||||
* --dry-run Show what would happen, exit 0 without touching DB.
|
||||
* --yes Skip interactive [y/N]. Required for non-TTY (including --json).
|
||||
* --json Machine-readable result envelope. Does NOT imply --yes.
|
||||
*
|
||||
* Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE)
|
||||
* so a large brain never holds one giant row lock, and streams progress
|
||||
* through the shared reporter (stderr; stdout stays clean for --json).
|
||||
*
|
||||
* Cost: trigger recreate is sub-millisecond. Backfill is one tsvector
|
||||
* rebuild per page + per chunk. On a 20K-page brain with 80K chunks,
|
||||
* expect ~5-15s depending on Postgres CPU and content size.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { getFtsLanguage } from '../core/fts-language.ts';
|
||||
import { createInterface } from 'readline';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export interface ReindexSearchVectorOpts {
|
||||
dryRun?: boolean;
|
||||
yes?: boolean;
|
||||
json?: boolean;
|
||||
}
|
||||
|
||||
export interface ReindexSearchVectorResult {
|
||||
status: 'ok' | 'dry_run' | 'cancelled';
|
||||
language: string;
|
||||
pagesUpdated: number;
|
||||
chunksUpdated: number;
|
||||
triggersRecreated: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
interface CountRow {
|
||||
pages: number;
|
||||
chunks: number;
|
||||
}
|
||||
|
||||
/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */
|
||||
export const BACKFILL_BATCH_SIZE = 5000;
|
||||
|
||||
/**
|
||||
* Keyset-batched UPDATE: applies `setClause` to `table` rows where
|
||||
* search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking
|
||||
* the shared progress reporter after each batch. Terminates when a batch
|
||||
* returns fewer rows than the batch size (or none).
|
||||
*/
|
||||
async function batchedBackfill(
|
||||
engine: BrainEngine,
|
||||
table: 'pages' | 'content_chunks',
|
||||
setClause: string,
|
||||
tick: (n: number) => void
|
||||
): Promise<void> {
|
||||
let cursor = 0;
|
||||
for (;;) {
|
||||
const rows = await engine.executeRaw<{ id: number }>(`
|
||||
UPDATE ${table} SET ${setClause}
|
||||
WHERE id IN (
|
||||
SELECT id FROM ${table}
|
||||
WHERE search_vector IS NOT NULL AND id > ${cursor}
|
||||
ORDER BY id
|
||||
LIMIT ${BACKFILL_BATCH_SIZE}
|
||||
)
|
||||
RETURNING id
|
||||
`);
|
||||
if (rows.length === 0) break;
|
||||
tick(rows.length);
|
||||
cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor);
|
||||
if (rows.length < BACKFILL_BATCH_SIZE) break;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Programmatic entrypoint — takes a typed opts object. Used by tests and
|
||||
* future internal callers. The CLI wrapper is `runReindexSearchVectorCli`
|
||||
* defined at the bottom of this file.
|
||||
*/
|
||||
export async function runReindexSearchVector(
|
||||
engine: BrainEngine,
|
||||
opts: ReindexSearchVectorOpts
|
||||
): Promise<ReindexSearchVectorResult> {
|
||||
const lang = getFtsLanguage();
|
||||
const startedAt = Date.now();
|
||||
|
||||
// Inventory: how many rows will the backfill touch?
|
||||
const counts = await engine.executeRaw<CountRow>(
|
||||
`SELECT
|
||||
(SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages,
|
||||
(SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks`
|
||||
);
|
||||
const pagesCount = counts[0]?.pages ?? 0;
|
||||
const chunksCount = counts[0]?.chunks ?? 0;
|
||||
|
||||
if (opts.dryRun) {
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'dry_run',
|
||||
language: lang,
|
||||
pagesUpdated: pagesCount,
|
||||
chunksUpdated: chunksCount,
|
||||
triggersRecreated: 0,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`);
|
||||
console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`);
|
||||
console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Confirm unless --yes. --json does NOT bypass the gate — a machine
|
||||
// caller must pass --yes explicitly (mirrors reindex-code, #1784).
|
||||
if (!opts.yes) {
|
||||
if (!process.stdin.isTTY) {
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify({
|
||||
error: {
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'reindex_requires_yes',
|
||||
message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to preview.',
|
||||
},
|
||||
language: lang,
|
||||
pages: pagesCount,
|
||||
chunks: chunksCount,
|
||||
}));
|
||||
} else {
|
||||
console.error('Refusing to run without --yes in non-TTY environment.');
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
||||
const answer = await new Promise<string>(resolve => {
|
||||
rl.question(
|
||||
`Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `,
|
||||
resolve
|
||||
);
|
||||
});
|
||||
rl.close();
|
||||
|
||||
if (!/^y(es)?$/i.test(answer.trim())) {
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'cancelled',
|
||||
language: lang,
|
||||
pagesUpdated: 0,
|
||||
chunksUpdated: 0,
|
||||
triggersRecreated: 0,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log('Cancelled.');
|
||||
}
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
// Recreate trigger functions. The strings are intentionally identical to
|
||||
// the v123 migration body — keeping them in lockstep is the contract.
|
||||
// `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening:
|
||||
// CREATE OR REPLACE resets proconfig, so omitting it here would strip the
|
||||
// hardening from every brain that runs this command.
|
||||
const recreatePagesFn = `
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
const recreateChunksFn = `
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
await engine.executeRaw(recreatePagesFn);
|
||||
await engine.executeRaw(recreateChunksFn);
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// Backfill: UPDATE-to-self forces the pages trigger to re-fire
|
||||
// (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a
|
||||
// direct vector compute since the column itself is what we want.
|
||||
progress.start('reindex_search_vector.pages', pagesCount);
|
||||
await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n));
|
||||
progress.finish();
|
||||
|
||||
progress.start('reindex_search_vector.chunks', chunksCount);
|
||||
await batchedBackfill(
|
||||
engine,
|
||||
'content_chunks',
|
||||
`search_vector =
|
||||
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`,
|
||||
n => progress.tick(n)
|
||||
);
|
||||
progress.finish();
|
||||
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'ok',
|
||||
language: lang,
|
||||
pagesUpdated: pagesCount,
|
||||
chunksUpdated: chunksCount,
|
||||
triggersRecreated: 2,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`✅ Recreated 2 trigger functions with language='${lang}'`);
|
||||
console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector.
|
||||
* Matches the style of `reindex-code`: --dry-run, --yes/-y, --json.
|
||||
*
|
||||
* Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes.
|
||||
*/
|
||||
export async function runReindexSearchVectorCli(
|
||||
engine: BrainEngine,
|
||||
args: string[]
|
||||
): Promise<void> {
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const yes = args.includes('--yes') || args.includes('-y');
|
||||
const json = args.includes('--json');
|
||||
|
||||
await runReindexSearchVector(engine, { dryRun, yes, json });
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
/**
|
||||
* Full-text search language configuration.
|
||||
*
|
||||
* Postgres tsvector/tsquery require a text search configuration name (e.g.
|
||||
* 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded
|
||||
* 'english' across engines and trigger functions, which broke search
|
||||
* quality for non-English brains (no stemming, no stop-word removal).
|
||||
*
|
||||
* This helper centralizes the choice. Default stays 'english' for backward
|
||||
* compatibility — only users who set GBRAIN_FTS_LANGUAGE see different
|
||||
* behavior.
|
||||
*
|
||||
* Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent +
|
||||
* portuguese stemmer) are supported as long as the configuration exists
|
||||
* in the target Postgres instance. See docs/guides/multi-language-fts.md
|
||||
* for setup instructions.
|
||||
*
|
||||
* Validation: only allow lowercase letters, digits, and underscores. This
|
||||
* prevents SQL injection when the value is interpolated into queries
|
||||
* (Postgres tsvector functions don't accept parameterized config names —
|
||||
* they must be literals or identifiers).
|
||||
*/
|
||||
|
||||
const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/;
|
||||
const DEFAULT_LANGUAGE = 'english';
|
||||
|
||||
let cachedLanguage: string | null = null;
|
||||
|
||||
/**
|
||||
* Returns the configured Postgres text search configuration name.
|
||||
*
|
||||
* Resolution order:
|
||||
* 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid)
|
||||
* 2. 'english' (default — preserves existing behavior)
|
||||
*
|
||||
* The return value is safe to interpolate directly into SQL because it
|
||||
* passes the VALID_CONFIG_NAME guard. If validation fails, falls back to
|
||||
* the default and emits a one-time warning.
|
||||
*
|
||||
* Cached on first call; reset with `resetFtsLanguageCache()` (test only).
|
||||
*/
|
||||
export function getFtsLanguage(): string {
|
||||
if (cachedLanguage !== null) return cachedLanguage;
|
||||
|
||||
const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim();
|
||||
if (!raw) {
|
||||
cachedLanguage = DEFAULT_LANGUAGE;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
if (!VALID_CONFIG_NAME.test(raw)) {
|
||||
console.warn(
|
||||
`[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` +
|
||||
`Falling back to '${DEFAULT_LANGUAGE}'.`
|
||||
);
|
||||
cachedLanguage = DEFAULT_LANGUAGE;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
cachedLanguage = raw;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resets the cached language. Tests only — don't use in production code.
|
||||
*/
|
||||
export function resetFtsLanguageCache(): void {
|
||||
cachedLanguage = null;
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
|
||||
/**
|
||||
* Schema migrations — run automatically on initSchema().
|
||||
@@ -5505,6 +5506,96 @@ export const MIGRATIONS: Migration[] = [
|
||||
WHERE dimension IS NOT NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 123,
|
||||
name: 'configurable_fts_language',
|
||||
// Recreate the two search_vector trigger functions using the language
|
||||
// configured via GBRAIN_FTS_LANGUAGE (default 'english'). Idempotent:
|
||||
// CREATE OR REPLACE swaps the function body atomically; no trigger
|
||||
// recreation needed since the trigger references the function by name.
|
||||
//
|
||||
// Why a handler instead of a static SQL string: Postgres tsvector
|
||||
// functions don't accept parameterized config names — the language
|
||||
// must be a literal in the SQL. getFtsLanguage() validates the value
|
||||
// (lowercase letters/digits/underscores only) before interpolation.
|
||||
//
|
||||
// Function bodies mirror schema.sql / pglite-schema.ts exactly —
|
||||
// INCLUDING the `SET search_path = pg_catalog, public` hardening from
|
||||
// v120/#1647 (CREATE OR REPLACE resets proconfig, so omitting it here
|
||||
// would silently strip the hardening on every upgraded brain). Only
|
||||
// the text-search config name is parameterized. Keep all copies in
|
||||
// sync when the trigger logic changes.
|
||||
//
|
||||
// Backfill: after recreating the functions, re-tokenize existing rows
|
||||
// under the new language. Skipped when the configured language is
|
||||
// 'english' (trigger output identical — re-tokenizing is wasted I/O).
|
||||
// To change language after this migration has run, use
|
||||
// `gbrain reindex-search-vector`.
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
const lang = getFtsLanguage();
|
||||
|
||||
const recreatePagesFn = `
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
const recreateChunksFn = `
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
await engine.executeRaw(recreatePagesFn);
|
||||
await engine.executeRaw(recreateChunksFn);
|
||||
|
||||
if (lang === 'english') {
|
||||
console.log(` v123: trigger functions recreated with language='english' (default — no backfill needed)`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Backfill existing rows under the new tokenizer. UPDATE-to-same-value
|
||||
// re-fires the pages trigger; chunks are rewritten directly with the
|
||||
// same expression as the trigger.
|
||||
await engine.executeRaw(`
|
||||
UPDATE pages SET id = id
|
||||
WHERE search_vector IS NOT NULL;
|
||||
`);
|
||||
|
||||
await engine.executeRaw(`
|
||||
UPDATE content_chunks
|
||||
SET search_vector =
|
||||
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')
|
||||
WHERE search_vector IS NOT NULL;
|
||||
`);
|
||||
|
||||
console.log(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows`);
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -24,6 +24,7 @@ import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
|
||||
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
|
||||
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
|
||||
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
|
||||
@@ -1625,20 +1626,24 @@ export class PGLiteEngine implements BrainEngine {
|
||||
extraFilter += ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH ranked AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
-- v0.27.1: hide image rows from default text-keyword search so
|
||||
-- OCR text doesn't drown text-page hits. Image-similarity queries
|
||||
-- run a separate vector path on embedding_image.
|
||||
@@ -1857,20 +1862,23 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// visibilityClause already declared above (v0.32.7: hoisted so CJK branch can reuse).
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2 OFFSET $3`,
|
||||
params
|
||||
|
||||
@@ -34,6 +34,7 @@ import {
|
||||
COLUMN_NAME_REGEX,
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from './search/embedding-column.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
|
||||
@@ -1653,6 +1654,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
// column lookup. NOT bypassed by detail=high — soft-delete is a contract,
|
||||
// not a temporal preference.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const rawQuery = `
|
||||
WITH ranked_chunks AS (
|
||||
@@ -1660,11 +1664,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
|
||||
${typeClause}
|
||||
${typesClause}
|
||||
${excludeSlugsClause}
|
||||
@@ -1792,18 +1796,21 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
// v0.26.5: visibility filter for searchKeywordChunks (anchor primitive).
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const rawQuery = `
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
|
||||
${typeClause}
|
||||
${typesClause}
|
||||
${excludeSlugsClause}
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
import { resetFtsLanguageCache } from '../src/core/fts-language.ts';
|
||||
|
||||
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
|
||||
const originalLang = process.env[ENV_KEY];
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
if (originalLang !== undefined) process.env[ENV_KEY] = originalLang;
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
describe('configurable_fts_language migration', () => {
|
||||
test('migration is registered', () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
expect(ftsMig).toBeDefined();
|
||||
expect(ftsMig?.version).toBeGreaterThan(115);
|
||||
});
|
||||
|
||||
test('fts migration is the latest migration', () => {
|
||||
expect(MIGRATIONS.find(m => m.name === 'configurable_fts_language')?.version).toBe(LATEST_VERSION);
|
||||
});
|
||||
|
||||
test('ftsMig uses handler (not static SQL) because language interpolation is dynamic', () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
expect(ftsMig?.sql).toBe('');
|
||||
expect(ftsMig?.handler).toBeTypeOf('function');
|
||||
});
|
||||
|
||||
test('ftsMig handler is async', () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
// Async function check: the constructor name is 'AsyncFunction'
|
||||
expect(ftsMig?.handler?.constructor.name).toBe('AsyncFunction');
|
||||
});
|
||||
|
||||
test('migration handler issues recreate-function calls (smoke check via mock engine)', async () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
const calls: string[] = [];
|
||||
|
||||
const mockEngine = {
|
||||
executeRaw: async (sql: string) => {
|
||||
calls.push(sql);
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
process.env[ENV_KEY] = 'english';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
await ftsMig?.handler?.(mockEngine);
|
||||
|
||||
// Default 'english' \u2014 no backfill, only 2 CREATE OR REPLACE calls.
|
||||
expect(calls.length).toBe(2);
|
||||
expect(calls[0]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector');
|
||||
expect(calls[0]).toContain("to_tsvector('english'");
|
||||
expect(calls[1]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector');
|
||||
expect(calls[1]).toContain("to_tsvector('english'");
|
||||
// v120/#1647 hardening must survive the CREATE OR REPLACE (which resets
|
||||
// proconfig): both recreated bodies pin search_path.
|
||||
expect(calls[0]).toContain('SET search_path = pg_catalog, public');
|
||||
expect(calls[1]).toContain('SET search_path = pg_catalog, public');
|
||||
});
|
||||
|
||||
test('non-english language triggers backfill', async () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
const calls: string[] = [];
|
||||
|
||||
const mockEngine = {
|
||||
executeRaw: async (sql: string) => {
|
||||
calls.push(sql);
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
await ftsMig?.handler?.(mockEngine);
|
||||
|
||||
// pt_br \u2014 2 CREATE + 2 backfill UPDATEs = 4 calls
|
||||
expect(calls.length).toBe(4);
|
||||
expect(calls[0]).toContain("to_tsvector('pt_br'");
|
||||
expect(calls[1]).toContain("to_tsvector('pt_br'");
|
||||
expect(calls[2]).toMatch(/UPDATE pages/);
|
||||
expect(calls[3]).toContain("to_tsvector('pt_br'");
|
||||
expect(calls[3]).toMatch(/UPDATE content_chunks/);
|
||||
});
|
||||
|
||||
test('invalid language falls back to english (no SQL injection)', async () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
const calls: string[] = [];
|
||||
|
||||
const mockEngine = {
|
||||
executeRaw: async (sql: string) => {
|
||||
calls.push(sql);
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
|
||||
resetFtsLanguageCache();
|
||||
|
||||
await ftsMig?.handler?.(mockEngine);
|
||||
|
||||
// Falls back to english: 2 CREATE OR REPLACE only, no DROP TABLE in any SQL.
|
||||
expect(calls.length).toBe(2);
|
||||
for (const sql of calls) {
|
||||
expect(sql).not.toContain('DROP TABLE');
|
||||
expect(sql).toContain("to_tsvector('english'");
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,93 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { getFtsLanguage, resetFtsLanguageCache } from '../src/core/fts-language.ts';
|
||||
|
||||
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
describe('getFtsLanguage', () => {
|
||||
test('defaults to english when env is unset', () => {
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('defaults to english when env is empty string', () => {
|
||||
process.env[ENV_KEY] = '';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('defaults to english when env is whitespace', () => {
|
||||
process.env[ENV_KEY] = ' ';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('reads valid pt_br config', () => {
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
});
|
||||
|
||||
test('reads valid simple language name', () => {
|
||||
process.env[ENV_KEY] = 'spanish';
|
||||
expect(getFtsLanguage()).toBe('spanish');
|
||||
});
|
||||
|
||||
test('reads name with underscores and digits', () => {
|
||||
process.env[ENV_KEY] = 'custom_lang_v2';
|
||||
expect(getFtsLanguage()).toBe('custom_lang_v2');
|
||||
});
|
||||
|
||||
test('rejects names with quotes (SQL injection guard)', () => {
|
||||
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects names with spaces', () => {
|
||||
process.env[ENV_KEY] = 'pt br';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects names with hyphens', () => {
|
||||
process.env[ENV_KEY] = 'pt-br';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects names starting with digit', () => {
|
||||
process.env[ENV_KEY] = '1lang';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects uppercase (Postgres config names are lowercase)', () => {
|
||||
process.env[ENV_KEY] = 'English';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('caches after first read', () => {
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
|
||||
// Mutate env after first read \u2014 cached value wins.
|
||||
process.env[ENV_KEY] = 'spanish';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
});
|
||||
|
||||
test('resetFtsLanguageCache clears cache', () => {
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
|
||||
resetFtsLanguageCache();
|
||||
process.env[ENV_KEY] = 'spanish';
|
||||
expect(getFtsLanguage()).toBe('spanish');
|
||||
});
|
||||
|
||||
test('trims surrounding whitespace from valid value', () => {
|
||||
process.env[ENV_KEY] = ' pt_br ';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,152 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { runReindexSearchVector } from '../src/commands/reindex-search-vector.ts';
|
||||
import { resetFtsLanguageCache } from '../src/core/fts-language.ts';
|
||||
|
||||
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
|
||||
const originalLang = process.env[ENV_KEY];
|
||||
|
||||
interface MockState {
|
||||
calls: string[];
|
||||
rowsToReturn: { pages: number; chunks: number };
|
||||
}
|
||||
|
||||
function makeMockEngine(state: MockState): BrainEngine {
|
||||
return {
|
||||
executeRaw: async (sql: string) => {
|
||||
state.calls.push(sql);
|
||||
// Inventory query — return the configured counts
|
||||
if (sql.includes('SELECT') && sql.includes('FROM pages WHERE search_vector')) {
|
||||
return [{ pages: state.rowsToReturn.pages, chunks: state.rowsToReturn.chunks }];
|
||||
}
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
if (originalLang !== undefined) process.env[ENV_KEY] = originalLang;
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
describe('runReindexSearchVector', () => {
|
||||
test('--dry-run does not issue any DDL or backfill SQL', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 100, chunks: 500 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
const result = await runReindexSearchVector(engine, { dryRun: true, json: true });
|
||||
|
||||
expect(result.status).toBe('dry_run');
|
||||
expect(result.language).toBe('pt_br');
|
||||
expect(result.pagesUpdated).toBe(100);
|
||||
expect(result.chunksUpdated).toBe(500);
|
||||
expect(result.triggersRecreated).toBe(0);
|
||||
|
||||
// Only the inventory query — no CREATE OR REPLACE, no UPDATE.
|
||||
expect(state.calls.length).toBe(1);
|
||||
expect(state.calls[0]).toContain('SELECT');
|
||||
expect(state.calls[0]).not.toContain('CREATE OR REPLACE');
|
||||
expect(state.calls[0]).not.toContain('UPDATE');
|
||||
});
|
||||
|
||||
test('--yes recreates triggers + backfills with configured language', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 50, chunks: 200 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.language).toBe('pt_br');
|
||||
expect(result.triggersRecreated).toBe(2);
|
||||
expect(result.pagesUpdated).toBe(50);
|
||||
expect(result.chunksUpdated).toBe(200);
|
||||
|
||||
// 1 inventory + 2 CREATE + 2 backfill batches (mock returns no rows, so
|
||||
// the keyset loop terminates after the first batch per table) = 5 calls
|
||||
expect(state.calls.length).toBe(5);
|
||||
expect(state.calls[1]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector');
|
||||
expect(state.calls[1]).toContain("to_tsvector('pt_br'");
|
||||
expect(state.calls[2]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector');
|
||||
expect(state.calls[2]).toContain("to_tsvector('pt_br'");
|
||||
expect(state.calls[3]).toMatch(/UPDATE pages/);
|
||||
expect(state.calls[4]).toMatch(/UPDATE content_chunks/);
|
||||
expect(state.calls[4]).toContain("to_tsvector('pt_br'");
|
||||
// v120/#1647 hardening must survive the CREATE OR REPLACE (which resets
|
||||
// proconfig): both recreated bodies pin search_path.
|
||||
expect(state.calls[1]).toContain('SET search_path = pg_catalog, public');
|
||||
expect(state.calls[2]).toContain('SET search_path = pg_catalog, public');
|
||||
});
|
||||
|
||||
test('default english language still recreates + backfills (no shortcut here)', async () => {
|
||||
// Note: unlike the configurable_fts_language migration, the CLI command
|
||||
// intentionally backfills even for english. The user explicitly asked for
|
||||
// it, so we honor it. The migration skips backfill for english because it
|
||||
// auto-runs on first apply.
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.language).toBe('english');
|
||||
expect(state.calls.length).toBe(5);
|
||||
|
||||
// Trigger recreates (calls 1, 2) and chunks backfill (call 4) embed the
|
||||
// language literal. Pages backfill (call 3) is UPDATE-to-self that
|
||||
// re-fires the trigger, so the language literal lives in the trigger
|
||||
// function body — not in the UPDATE statement.
|
||||
expect(state.calls[1]).toContain("'english'");
|
||||
expect(state.calls[2]).toContain("'english'");
|
||||
expect(state.calls[3]).toMatch(/UPDATE pages/);
|
||||
expect(state.calls[4]).toContain("'english'");
|
||||
});
|
||||
|
||||
test('SQL injection attempt falls back to english', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
|
||||
resetFtsLanguageCache();
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.language).toBe('english');
|
||||
for (const sql of state.calls) {
|
||||
expect(sql).not.toContain('DROP TABLE');
|
||||
}
|
||||
});
|
||||
|
||||
test('empty inventory still completes successfully', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 0, chunks: 0 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.pagesUpdated).toBe(0);
|
||||
expect(result.chunksUpdated).toBe(0);
|
||||
expect(result.triggersRecreated).toBe(2);
|
||||
});
|
||||
|
||||
test('result includes durationMs', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 1, chunks: 1 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(typeof result.durationMs).toBe('number');
|
||||
expect(result.durationMs).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user