Files
gbrain/src/core/fts-language.ts
T
f8d11f67a3 feat(search): configurable FTS language + reindex command (lands #580/#581/#582) (#2941)
Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r
(#580 env-var language for query+write side, #581 migration backfill,
#582 gbrain reindex-search-vector), rebased onto current master with the
security review's required fixes applied:

- Migration renumbered v116 -> v123 (master's v116 was already claimed by
  code_edges_source_backfill; master is at v122).
- Restored the v120/#1647 search_path hardening: all four CREATE OR
  REPLACE trigger-function bodies (migration handler + reindex command)
  now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE
  resets proconfig and would otherwise strip the hardening on upgrade.
- reindex-search-vector: shared progress reporter (stderr phases
  reindex_search_vector.pages/.chunks), id-keyset batched backfill
  (5000 rows/UPDATE) instead of single whole-table statements, and
  --json no longer bypasses the --yes/TTY confirmation gate.
- Allowlist validation regex + injection tests kept exactly as authored.
- Stale v33/v116 comments swept; docs/guides/multi-language-fts.md
  written (README referenced it but no PR added it); llms bundles
  regenerated via bun run build:llms.
- Trilogy tests quarantined as *.serial.test.ts (env mutation, per
  check-test-isolation).

Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese
produces portuguese-stemmed vectors with search_path pinned; the reindex
command retokenizes an english brain to portuguese in place.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:23:47 -07:00

70 lines
2.3 KiB
TypeScript

/**
* Full-text search language configuration.
*
* Postgres tsvector/tsquery require a text search configuration name (e.g.
* 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded
* 'english' across engines and trigger functions, which broke search
* quality for non-English brains (no stemming, no stop-word removal).
*
* This helper centralizes the choice. Default stays 'english' for backward
* compatibility — only users who set GBRAIN_FTS_LANGUAGE see different
* behavior.
*
* Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent +
* portuguese stemmer) are supported as long as the configuration exists
* in the target Postgres instance. See docs/guides/multi-language-fts.md
* for setup instructions.
*
* Validation: only allow lowercase letters, digits, and underscores. This
* prevents SQL injection when the value is interpolated into queries
* (Postgres tsvector functions don't accept parameterized config names —
* they must be literals or identifiers).
*/
const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/;
const DEFAULT_LANGUAGE = 'english';
let cachedLanguage: string | null = null;
/**
* Returns the configured Postgres text search configuration name.
*
* Resolution order:
* 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid)
* 2. 'english' (default — preserves existing behavior)
*
* The return value is safe to interpolate directly into SQL because it
* passes the VALID_CONFIG_NAME guard. If validation fails, falls back to
* the default and emits a one-time warning.
*
* Cached on first call; reset with `resetFtsLanguageCache()` (test only).
*/
export function getFtsLanguage(): string {
if (cachedLanguage !== null) return cachedLanguage;
const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim();
if (!raw) {
cachedLanguage = DEFAULT_LANGUAGE;
return cachedLanguage;
}
if (!VALID_CONFIG_NAME.test(raw)) {
console.warn(
`[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` +
`Falling back to '${DEFAULT_LANGUAGE}'.`
);
cachedLanguage = DEFAULT_LANGUAGE;
return cachedLanguage;
}
cachedLanguage = raw;
return cachedLanguage;
}
/**
* Resets the cached language. Tests only — don't use in production code.
*/
export function resetFtsLanguageCache(): void {
cachedLanguage = null;
}