mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-29 19:01:39 +00:00
Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r (#580 env-var language for query+write side, #581 migration backfill, #582 gbrain reindex-search-vector), rebased onto current master with the security review's required fixes applied: - Migration renumbered v116 -> v123 (master's v116 was already claimed by code_edges_source_backfill; master is at v122). - Restored the v120/#1647 search_path hardening: all four CREATE OR REPLACE trigger-function bodies (migration handler + reindex command) now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE resets proconfig and would otherwise strip the hardening on upgrade. - reindex-search-vector: shared progress reporter (stderr phases reindex_search_vector.pages/.chunks), id-keyset batched backfill (5000 rows/UPDATE) instead of single whole-table statements, and --json no longer bypasses the --yes/TTY confirmation gate. - Allowlist validation regex + injection tests kept exactly as authored. - Stale v33/v116 comments swept; docs/guides/multi-language-fts.md written (README referenced it but no PR added it); llms bundles regenerated via bun run build:llms. - Trilogy tests quarantined as *.serial.test.ts (env mutation, per check-test-isolation). Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese produces portuguese-stemmed vectors with search_path pinned; the reindex command retokenizes an english brain to portuguese in place. Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
70 lines
2.3 KiB
TypeScript
70 lines
2.3 KiB
TypeScript
/**
|
|
* Full-text search language configuration.
|
|
*
|
|
* Postgres tsvector/tsquery require a text search configuration name (e.g.
|
|
* 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded
|
|
* 'english' across engines and trigger functions, which broke search
|
|
* quality for non-English brains (no stemming, no stop-word removal).
|
|
*
|
|
* This helper centralizes the choice. Default stays 'english' for backward
|
|
* compatibility — only users who set GBRAIN_FTS_LANGUAGE see different
|
|
* behavior.
|
|
*
|
|
* Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent +
|
|
* portuguese stemmer) are supported as long as the configuration exists
|
|
* in the target Postgres instance. See docs/guides/multi-language-fts.md
|
|
* for setup instructions.
|
|
*
|
|
* Validation: only allow lowercase letters, digits, and underscores. This
|
|
* prevents SQL injection when the value is interpolated into queries
|
|
* (Postgres tsvector functions don't accept parameterized config names —
|
|
* they must be literals or identifiers).
|
|
*/
|
|
|
|
const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/;
|
|
const DEFAULT_LANGUAGE = 'english';
|
|
|
|
let cachedLanguage: string | null = null;
|
|
|
|
/**
|
|
* Returns the configured Postgres text search configuration name.
|
|
*
|
|
* Resolution order:
|
|
* 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid)
|
|
* 2. 'english' (default — preserves existing behavior)
|
|
*
|
|
* The return value is safe to interpolate directly into SQL because it
|
|
* passes the VALID_CONFIG_NAME guard. If validation fails, falls back to
|
|
* the default and emits a one-time warning.
|
|
*
|
|
* Cached on first call; reset with `resetFtsLanguageCache()` (test only).
|
|
*/
|
|
export function getFtsLanguage(): string {
|
|
if (cachedLanguage !== null) return cachedLanguage;
|
|
|
|
const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim();
|
|
if (!raw) {
|
|
cachedLanguage = DEFAULT_LANGUAGE;
|
|
return cachedLanguage;
|
|
}
|
|
|
|
if (!VALID_CONFIG_NAME.test(raw)) {
|
|
console.warn(
|
|
`[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` +
|
|
`Falling back to '${DEFAULT_LANGUAGE}'.`
|
|
);
|
|
cachedLanguage = DEFAULT_LANGUAGE;
|
|
return cachedLanguage;
|
|
}
|
|
|
|
cachedLanguage = raw;
|
|
return cachedLanguage;
|
|
}
|
|
|
|
/**
|
|
* Resets the cached language. Tests only — don't use in production code.
|
|
*/
|
|
export function resetFtsLanguageCache(): void {
|
|
cachedLanguage = null;
|
|
}
|