v0.41.37.0 fix: critical fix wave — reindex tag wipe, grandfather hang, Windows migration spawn, sync ReDoS (#1621 #1581 #1605 #1569) (#1665)

* fix(reindex): add-only tag reconciliation + DB-only re-chunk preserves frontmatter (#1621)

reindex --markdown and re-import no longer wipe DB-side enrichment tags.
Tag reconciliation is now ADD-ONLY (import-file.ts): re-import adds current
frontmatter tags and never deletes, so auto/dream/signal-detector tags survive.
The reindex DB-only fallback reconstructs full markdown via serializeMarkdown
so re-chunking a page with no on-disk source preserves frontmatter/title/timeline.

* fix(migrations): v0.13.1 grandfather chunked, source-safe, soft-delete-filtered (#1581)

phaseCGrandfather rewritten from a per-page getPage+putPage loop (which hung
70+ min on an 82K-page PGLite brain) to a chunked bulk SQL pass keyed on
pages.id (NOT slug — slug isn't globally unique), filtering deleted_at IS NULL,
with a batched rollback log carrying source identity.

* fix(migrations): run schema phases in-process to fix Windows getaddrinfo ENOTFOUND (#1605)

The 9 'gbrain init --migrate-only' execSync spawns died on Windows+bun+Supabase
(child DNS resolution). runMigrateOnlyCore (extracted from initMigrateOnly) runs
the schema bring-up in-process for all engines, unblocking schema_version
advancement. Includes async-call-site audit, a wall-clock guard, and a
runGbrainSubprocess stderr-capture wrapper for the remaining backfill spawns.

* fix(sync): ReDoS hardening + diagnostics for schema-pack regexes (#1569)

Input-length cap in runRegexBounded + route the unbounded link-inference path
through it (closes the only no-timeout ReDoS hole); star-height lint rule warns
on nested-quantifier patterns; --no-schema-pack sync escape hatch; GBRAIN_SYNC_TRACE
per-file begin heartbeat; PGLite serve/sync concurrency doc. Defensive hardening +
diagnostics — the deterministic ~3100-file wedge root cause remains open (no repro).

* docs(todos): file v0.41.37.0 fix-wave follow-ups (#1621/#1605/#1569)

* chore: bump version and changelog (v0.41.37.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: sync README + CLAUDE.md for v0.41.37.0 critical fix wave

Add reindex add-only tag reconciliation (#1621), v0.13.1 grandfather +
Windows in-process migration (#1581/#1605), and schema-pack ReDoS
hardening + sync --no-schema-pack / GBRAIN_SYNC_TRACE triage (#1569)
to CLAUDE.md key-files annotations and README Troubleshooting.
Regenerated llms-full.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): bump llms-full.txt budget 700KB→750KB (CLAUDE.md crossed 700KB after master merge)

The build-llms size-budget test failed: llms-full.txt is 703,244 bytes after the
v0.41.37.0 key-files annotations merged on top of master's v0.41.34/35/36 CLAUDE.md
additions. Matches the v0.41.9.0 precedent (600→700); the single-fetch bundle still
fits comfortably in modern long-context models.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-30 14:56:26 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent ca13f40820
commit 6f26d5e4df
36 changed files with 1201 additions and 232 deletions
+15 -32
View File
@@ -515,44 +515,27 @@ async function resolveChatByEnv(out: ResolvedAIOptions): Promise<void> {
* clobbering the user's chosen engine.
*/
async function initMigrateOnly(opts: { jsonOutput: boolean }) {
const config = loadConfig();
if (!config) {
const msg = 'No brain configured. Run `gbrain init` (interactive) or `gbrain init --pglite` / `gbrain init --supabase` first.';
// v0.41.37.0 #1605: delegate to the shared runMigrateOnlyCore so the CLI path
// and the in-process migration-orchestrator path can't drift (single source
// of truth for configureGateway-before-initSchema + the schema bring-up).
const { runMigrateOnlyCore, MigrateOnlyError } = await import('./migrations/in-process.ts');
try {
const result = await runMigrateOnlyCore();
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'error', reason: 'no_config', message: msg }));
console.log(JSON.stringify({ status: 'success', engine: result.engine, mode: 'migrate-only' }));
} else {
console.log(`Schema up to date (engine: ${result.engine}).`);
}
} catch (e) {
const isNoConfig = e instanceof MigrateOnlyError && e.message.startsWith('No brain configured');
const msg = e instanceof Error ? e.message : String(e);
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'error', reason: isNoConfig ? 'no_config' : 'migrate_failed', message: msg }));
} else {
console.error(msg);
}
process.exit(1);
}
// B.3: configureGateway BEFORE initSchema even on the migrate-only path,
// so a schema bump on a brain whose file config is missing the embedding
// fields doesn't fall through to stale hardcoded fallbacks. Reads
// existing config (which loadConfig already merged with env) and
// propagates it into the gateway.
const { configureGateway: configureGw } = await import('../core/ai/gateway.ts');
configureGw({
embedding_model: config.embedding_model,
embedding_dimensions: config.embedding_dimensions,
expansion_model: config.expansion_model,
chat_model: config.chat_model,
env: { ...process.env },
});
const engine = await createEngine(toEngineConfig(config));
try {
await engine.connect(toEngineConfig(config));
await engine.initSchema();
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: config.engine, mode: 'migrate-only' }));
} else {
console.log(`Schema up to date (engine: ${config.engine}).`);
}
}
/**
+139
View File
@@ -0,0 +1,139 @@
/**
* In-process migration helpers (v0.41.37.0 #1605).
*
* Why this exists: migration schema phases used to shell out to a child
* `gbrain init --migrate-only` via `execSync`. On Windows + bun + Supabase
* pooler, the spawned CHILD process dies with `getaddrinfo ENOTFOUND` before it
* can connect — even though the PARENT connects fine and `env: process.env` is
* passed. It is a bun-on-Windows child-process DNS-resolution failure, not an
* env-propagation bug. The only robust fix is to not spawn at all: run the
* schema bring-up IN-PROCESS. The PGLite path at v0_11_0.ts already proved the
* pattern; this generalizes it to every engine + every schema phase.
*
* `runMigrateOnlyCore` is the single source of truth for "bring schema to head"
* — `init.ts:initMigrateOnly` (the `gbrain init --migrate-only` CLI path) and
* the migration orchestrators both call it, so the configureGateway-before-
* initSchema fix can't drift between them.
*
* `runGbrainSubprocess` is the diagnostic wrapper for the REMAINING (non-schema)
* gbrain-subprocess spawns (extract/repair/stats). It captures child stderr and
* folds it into the thrown error so a Windows failure shows the real
* `getaddrinfo ENOTFOUND` line instead of the bare `Command failed: ...`.
*/
import { execSync } from 'child_process';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
/** Default wall-clock guard for in-process initSchema. Matches the 600s cap
* the old `execSync('gbrain init --migrate-only', { timeout: 600_000 })` used,
* so a hung schema bring-up surfaces as a phase failure instead of wedging
* the whole cascade. */
export const MIGRATE_ONLY_TIMEOUT_MS = 600_000;
/** Large stderr buffer for captured subprocess output. `execSync`'s default
* ~1MB maxBuffer overflows on long backfills (extract/repair) and turns a
* successful run into a spurious failure. */
const SUBPROCESS_MAX_BUFFER = 64 * 1024 * 1024;
export interface MigrateOnlyResult {
/** The engine kind that was brought to head ('pglite' | 'postgres'). */
engine: string;
}
export class MigrateOnlyError extends Error {
constructor(message: string) {
super(message);
this.name = 'MigrateOnlyError';
}
}
/**
* Bring the configured brain's schema to head, in-process. Mirrors what
* `gbrain init --migrate-only` did via subprocess: configureGateway →
* createEngine → connect → initSchema → disconnect. Idempotent (initSchema is
* a no-op when already at head). Throws `MigrateOnlyError` on no-config or
* timeout so callers report a failed phase rather than hanging.
*/
export async function runMigrateOnlyCore(opts?: { timeoutMs?: number }): Promise<MigrateOnlyResult> {
const config = loadConfig();
if (!config) {
throw new MigrateOnlyError(
'No brain configured. Run `gbrain init` (interactive) or `gbrain init --pglite` / `gbrain init --supabase` first.',
);
}
// configureGateway BEFORE initSchema (init.ts B.3): a schema bump on a brain
// whose file config is missing embedding fields must not fall through to
// stale hardcoded fallbacks. loadConfig already merged env; propagate it.
const { configureGateway } = await import('../../core/ai/gateway.ts');
configureGateway({
embedding_model: config.embedding_model,
embedding_dimensions: config.embedding_dimensions,
expansion_model: config.expansion_model,
chat_model: config.chat_model,
env: { ...process.env },
});
const timeoutMs = opts?.timeoutMs ?? MIGRATE_ONLY_TIMEOUT_MS;
const engine = await createEngine(toEngineConfig(config));
try {
await engine.connect(toEngineConfig(config));
await withTimeout(
engine.initSchema(),
timeoutMs,
`schema init timed out after ${Math.round(timeoutMs / 1000)}s`,
);
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
return { engine: config.engine };
}
/**
* Run a `gbrain ...` subcommand as a subprocess, capturing child stderr so a
* failure surfaces the real reason. Used for the non-schema backfill phases
* (extract/repair/stats) that aren't yet in-process. On Windows these may still
* fail with `getaddrinfo ENOTFOUND`, but the operator now sees WHY instead of a
* bare `Command failed`. Returns captured stdout (utf-8) on success.
*
* Note: stderr is piped (captured), so gbrain progress lines (which go to
* stderr) are not shown live during these phases — acceptable for a one-shot
* `apply-migrations` run; the failure reason matters more than live progress.
*/
export function runGbrainSubprocess(cmd: string, opts?: { timeoutMs?: number }): string {
try {
const out = execSync(cmd, {
stdio: ['inherit', 'pipe', 'pipe'],
timeout: opts?.timeoutMs ?? MIGRATE_ONLY_TIMEOUT_MS,
env: process.env,
maxBuffer: SUBPROCESS_MAX_BUFFER,
encoding: 'utf-8',
});
return typeof out === 'string' ? out : '';
} catch (e: unknown) {
const err = e as { message?: string; stderr?: Buffer | string };
const stderrRaw = err?.stderr
? (Buffer.isBuffer(err.stderr) ? err.stderr.toString('utf-8') : String(err.stderr))
: '';
const tail = stderrRaw.split('\n').filter(Boolean).slice(-10).join('\n');
const base = err?.message ?? String(e);
throw new Error(tail ? `${base}\n--- child stderr (tail) ---\n${tail}` : base);
}
}
/** Reject `p` if it doesn't settle within `ms`. The original promise keeps
* running (best-effort) but the caller sees a clear timeout error. */
async function withTimeout<T>(p: Promise<T>, ms: number, message: string): Promise<T> {
let timer: ReturnType<typeof setTimeout> | undefined;
const timeout = new Promise<never>((_, reject) => {
timer = setTimeout(() => reject(new MigrateOnlyError(message)), ms);
});
try {
return await Promise.race([p, timeout]);
} finally {
if (timer) clearTimeout(timer);
}
}
+8 -23
View File
@@ -23,7 +23,6 @@
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, lstatSync, statSync, realpathSync } from 'fs';
import { join, resolve, dirname } from 'path';
import { execSync } from 'child_process';
import { childGlobalFlags } from '../../core/cli-options.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { savePreferences, loadPreferences } from '../../core/preferences.ts';
// Bug 3 — appendCompletedMigration moved to the runner (apply-migrations.ts).
@@ -61,28 +60,14 @@ export interface PendingHostWorkEntry {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
// v0.36.x #1100: route PGLite through an in-process schema apply rather
// than `execSync('gbrain init --migrate-only')`. The subprocess inherits
// HOME and tries to acquire the same file lock the parent process is
// holding (or briefly released and the on-disk artifact has not finished
// settling), which deadlocks until the 30s lock timeout fires. The
// structural fix is to not spawn a subprocess for work the parent can
// do directly — Postgres tolerates concurrent connections, so the
// legacy execSync path stays for Postgres callers.
const { loadConfig, toEngineConfig } = await import('../../core/config.ts');
const cfg = loadConfig();
if (cfg?.engine === 'pglite') {
const { createEngine } = await import('../../core/engine-factory.ts');
const eng = await createEngine(toEngineConfig(cfg));
try {
await eng.connect(toEngineConfig(cfg));
await eng.initSchema();
} finally {
try { await eng.disconnect(); } catch { /* best-effort */ }
}
return { name: 'schema', status: 'complete' };
}
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
// v0.41.37.0 #1605: bring schema to head IN-PROCESS for every engine. Was an
// a `gbrain init --migrate-only` subprocess subprocess for Postgres (died with
// `getaddrinfo ENOTFOUND` on Windows+bun+Supabase-pooler before it could
// connect) plus a PGLite-only in-process branch (which separately
// deadlocked on the file lock, #1100). runMigrateOnlyCore is the single
// in-process path for both engines.
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
+7 -5
View File
@@ -31,19 +31,21 @@
*/
import { execSync } from 'child_process';
import { runGbrainSubprocess } from './in-process.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
// 10-minute budget. Migrations v8/v9 dedup with helper-index should be sub-second
// even on 80K-duplicate brains, but the outer wall-clock cap shouldn't be the
// failure mode (the prior 60s ceiling tripped Garry's production upgrade).
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -93,7 +95,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
// --source db is idempotent: the UNIQUE constraint on
// (from_page_id, to_page_id, link_type) and ON CONFLICT DO NOTHING
// make re-runs cheap. Empty brains return 0/0 quickly.
execSync('gbrain extract links --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
runGbrainSubprocess('gbrain extract links --source db' + childGlobalFlags(), { timeoutMs: 600_000 });
return { name: 'backfill_links', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -104,7 +106,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
function phaseDBackfillTimeline(opts: OrchestratorOpts): OrchestratorPhaseResult {
if (opts.dryRun) return { name: 'backfill_timeline', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain extract timeline --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
runGbrainSubprocess('gbrain extract timeline --source db' + childGlobalFlags(), { timeoutMs: 600_000 });
return { name: 'backfill_timeline', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -188,7 +190,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
// A. Schema
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') {
return finalizeResult(phases, 'failed');
+6 -4
View File
@@ -21,18 +21,20 @@
*/
import { execSync } from 'child_process';
import { runGbrainSubprocess } from './in-process.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
// Propagate global progress flags so the child shows the same mode the
// parent orchestrator is running in.
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -46,7 +48,7 @@ function phaseBRepair(opts: OrchestratorOpts): OrchestratorPhaseResult {
if (opts.dryRun) return { name: 'jsonb_repair', status: 'skipped', detail: 'dry-run' };
try {
// stdio: 'inherit' — child's stderr progress streams straight through.
execSync('gbrain repair-jsonb' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
runGbrainSubprocess('gbrain repair-jsonb' + childGlobalFlags(), { timeoutMs: 600_000 });
return { name: 'jsonb_repair', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -94,7 +96,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalizeResult(phases, 'failed');
+6 -8
View File
@@ -26,6 +26,7 @@
*/
import { execSync } from 'child_process';
import { runGbrainSubprocess } from './in-process.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts). The
// orchestrator returns its result and the runner persists it.
@@ -44,10 +45,11 @@ import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhase
// upgrade mid-migration. The shim is already the canonical wrapper; trust
// it. Regression guarded by test/migrations-v0_13_0.test.ts.
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -64,11 +66,7 @@ function phaseBBackfill(opts: OrchestratorOpts): OrchestratorPhaseResult {
// `--include-frontmatter` is the v0.13 flag that enables the canonical
// frontmatter link extractor. Default-OFF in the CLI for back-compat;
// the migration explicitly opts in because this is the canonical backfill.
execSync('gbrain extract links --source db --include-frontmatter', {
stdio: 'inherit',
timeout: 1_800_000, // 30 min hard cap; typical 2-5 min on 46K pages
env: process.env,
});
runGbrainSubprocess('gbrain extract links --source db --include-frontmatter', { timeoutMs: 1_800_000 });
return { name: 'frontmatter_backfill', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -116,7 +114,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalizeResult(phases, 'failed');
+102 -81
View File
@@ -20,13 +20,14 @@
* frontmatter snapshot. Roll back by re-applying those snapshots via
* `gbrain apply-migrations --rollback v0.13.0` (future CLI; not in scope).
*
* Scale: on a 30K-page brain, ~15s on Postgres, ~30s on PGLite. Batched in
* chunks of 100 with a commit per batch so interruption losses are bounded.
* Scale (v0.41.37.0 #1581): chunked bulk SQL — one id-snapshot SELECT + N
* (SELECT-for-rollback + UPDATE) statements of CHUNK_SIZE rows each. Completes
* in ~1-2s even on an 82K-page PGLite brain. The prior per-page
* getPage+putPage loop hung CPU-bound for 70+ min on that brain (#1581).
*
* Snapshot-slugs rule: reads engine.getAllSlugs() upfront into an in-memory
* Set before iterating. Prior learning [listpages-pagination-mutation]: any
* batch write that mutates updated_at during OFFSET pagination is unstable.
* getAllSlugs returns a full snapshot that isn't invalidated by our writes.
* Snapshot rule: the affected id set is read once via SQL up front. It isn't
* invalidated by our writes because each UPDATE flips its rows out of the
* GRANDFATHER_WHERE predicate (idempotent + resumable).
*
* Safety: does NOT call saveConfig. Prior learning [gbrain-init-default-pglite-flip]:
* bare `gbrain init` defaults to PGLite and overwrites Postgres config.
@@ -46,7 +47,6 @@ import type { BrainEngine } from '../../core/engine.ts';
// Lazy: GBRAIN_HOME may be set after module load.
const getRollbackDir = () => gbrainPath('migrations');
const getRollbackFile = () => join(getRollbackDir(), 'v0_13_1-rollback.jsonl');
const BATCH_SIZE = 100;
// ---------------------------------------------------------------------------
// Phase A — connect (no config write)
@@ -75,30 +75,35 @@ async function phaseAConnect(opts: OrchestratorOpts): Promise<{ result: Orchestr
}
}
// ---------------------------------------------------------------------------
// Phase B — snapshot slugs upfront
// ---------------------------------------------------------------------------
async function phaseBSnapshot(engine: BrainEngine): Promise<{ result: OrchestratorPhaseResult; slugs: string[] }> {
try {
const slugSet = await engine.getAllSlugs();
const slugs = [...slugSet].sort();
return {
result: { name: 'snapshot', status: 'complete', detail: `${slugs.length} slugs` },
slugs,
};
} catch (e) {
return {
result: { name: 'snapshot', status: 'failed', detail: e instanceof Error ? e.message : String(e) },
slugs: [],
};
}
}
// ---------------------------------------------------------------------------
// Phase C — grandfather: add validate:false where absent
//
// v0.41.37.0 #1581: rewritten from a per-page getPage+putPage loop (which hung
// CPU-bound for 70+ min on an 82K-page PGLite brain) to a CHUNKED bulk SQL
// pass. Three correctness properties (the per-page loop and a naive single
// bulk UPDATE both got these wrong):
// 1. Keyed on pages.id (globally unique PK), NOT slug. `slug` uniqueness is
// (source_id, slug) — a slug-batched UPDATE would mutate same-slug pages
// across other sources and produce ambiguous rollback rows.
// 2. Filters `deleted_at IS NULL` — the old getPage path hid soft-deleted
// rows; a raw UPDATE must not grandfather tombstones.
// 3. Chunked in CHUNK_SIZE batches so lock-hold stays bounded (the
// DELETE_BATCH_SIZE convention) instead of one giant transaction.
// The rollback log carries source identity ({id, slug, source_id,
// pre_frontmatter}) so a rollback is unambiguous across sources.
// ---------------------------------------------------------------------------
// Filter for pages still needing the grandfather flag. Literal `?` jsonb
// existence operator (matches src/core/embed-skip.ts convention); 'validate'
// is a hardcoded literal, no injection surface. COALESCE for null-frontmatter
// safety. deleted_at IS NULL skips soft-deleted tombstones.
const GRANDFATHER_WHERE =
"NOT (COALESCE(frontmatter, '{}'::jsonb) ? 'validate') AND deleted_at IS NULL";
// Per-chunk row count. Bounded lock-hold + write-amplification per statement
// (same rationale as engine-constants.ts DELETE_BATCH_SIZE).
const CHUNK_SIZE = 1000;
interface GrandfatherResult {
touched: number;
skipped: number;
@@ -106,60 +111,69 @@ interface GrandfatherResult {
failures: string[];
}
async function phaseCGrandfather(
// Exported for direct hermetic testing against a PGLite engine (the config /
// loadConfig flow is exercised separately). Internal helper otherwise.
export async function phaseCGrandfather(
engine: BrainEngine,
slugs: string[],
opts: OrchestratorOpts,
): Promise<{ result: OrchestratorPhaseResult; detail: GrandfatherResult }> {
ensureRollbackDir();
const gf: GrandfatherResult = { touched: 0, skipped: 0, failed: 0, failures: [] };
for (let i = 0; i < slugs.length; i += BATCH_SIZE) {
const batch = slugs.slice(i, i + BATCH_SIZE);
for (const slug of batch) {
try {
if (opts.dryRun) {
const rows = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages WHERE ${GRANDFATHER_WHERE}`,
);
const c = rows[0]?.count ?? 0;
gf.touched = typeof c === 'string' ? parseInt(c, 10) : Number(c);
return {
result: { name: 'grandfather', status: 'complete', detail: `would touch ${gf.touched} (dry-run)` },
detail: gf,
};
}
ensureRollbackDir();
// Snapshot the affected id set up front (ids only — cheap, no frontmatter
// blobs in memory). The snapshot isn't invalidated by our writes because
// each UPDATE flips the rows out of the GRANDFATHER_WHERE predicate.
const idRows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM pages WHERE ${GRANDFATHER_WHERE} ORDER BY id`,
);
const ids = idRows.map(r => Number(r.id));
for (let i = 0; i < ids.length; i += CHUNK_SIZE) {
const chunk = ids.slice(i, i + CHUNK_SIZE);
try {
const page = await engine.getPage(slug);
if (!page) { gf.skipped++; continue; }
// Rollback log BEFORE mutation: one SELECT per chunk (bounded memory),
// one appendFileSync per chunk. Carries source_id so rollback is
// unambiguous across same-slug-different-source pages.
const snap = await engine.executeRaw<{
id: number; slug: string; source_id: string | null; frontmatter: Record<string, unknown> | null;
}>(
'SELECT id, slug, source_id, frontmatter FROM pages WHERE id = ANY($1::int[])',
[chunk],
);
appendRollbackBatch(snap);
// Idempotency: skip if frontmatter already has a `validate` key
// (whether true, false, or any other value). We don't flip existing
// explicit settings.
if (page.frontmatter && Object.prototype.hasOwnProperty.call(page.frontmatter, 'validate')) {
gf.skipped++;
continue;
}
if (opts.dryRun) {
gf.touched++;
continue;
}
// Rollback log BEFORE mutation, so a crash mid-write still lets us
// revert. Append-only, one line per page, newline-terminated.
appendRollbackEntry({
slug,
pre_frontmatter: page.frontmatter ?? {},
});
const nextFrontmatter = { ...(page.frontmatter ?? {}), validate: false };
await engine.putPage(slug, {
type: page.type,
title: page.title,
compiled_truth: page.compiled_truth,
timeline: page.timeline,
frontmatter: nextFrontmatter,
});
gf.touched++;
await engine.executeRaw(
`UPDATE pages SET frontmatter = jsonb_set(COALESCE(frontmatter, '{}'::jsonb), '{validate}', 'false'::jsonb) ` +
'WHERE id = ANY($1::int[])',
[chunk],
);
gf.touched += chunk.length;
} catch (e) {
gf.failed++;
gf.failed += chunk.length;
const msg = e instanceof Error ? e.message : String(e);
gf.failures.push(`${slug}: ${msg.slice(0, 100)}`);
gf.failures.push(`chunk@${i}: ${msg.slice(0, 100)}`);
}
}
} catch (e) {
gf.failed += 1;
gf.failures.push(`grandfather: ${e instanceof Error ? e.message : String(e)}`.slice(0, 120));
}
const status: OrchestratorPhaseResult['status'] =
gf.failed > 0 ? 'failed' : 'complete';
const status: OrchestratorPhaseResult['status'] = gf.failed > 0 ? 'failed' : 'complete';
const detailStr = `touched=${gf.touched} skipped=${gf.skipped} failed=${gf.failed}`;
return {
result: { name: 'grandfather', status, detail: detailStr },
@@ -215,13 +229,10 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
try {
const { result: snapRes, slugs } = await phaseBSnapshot(engine);
phases.push(snapRes);
if (snapRes.status !== 'complete') {
return { version: '0.13.1', status: 'failed', phases };
}
const { result: gfRes, detail: gfDetail } = await phaseCGrandfather(engine, slugs, opts);
// v0.41.37.0 #1581: phaseBSnapshot (getAllSlugs) is gone — the chunked bulk
// pass filters via SQL (GRANDFATHER_WHERE), so we no longer materialize a
// full slug list in JS.
const { result: gfRes, detail: gfDetail } = await phaseCGrandfather(engine, opts);
phases.push(gfRes);
filesRewritten = gfDetail.touched;
@@ -255,13 +266,23 @@ function ensureRollbackDir(): void {
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
}
function appendRollbackEntry(entry: { slug: string; pre_frontmatter: Record<string, unknown> }): void {
const line = JSON.stringify({
// v0.41.37.0 #1581: batch rollback writer. One appendFileSync per chunk, bounded
// memory. Each line carries id + slug + source_id so a rollback is unambiguous
// across same-slug-different-source pages (pages.slug is not globally unique).
function appendRollbackBatch(
rows: ReadonlyArray<{ id: number; slug: string; source_id: string | null; frontmatter: Record<string, unknown> | null }>,
): void {
if (rows.length === 0) return;
const ts = new Date().toISOString();
const lines = rows.map(r => JSON.stringify({
migration: 'v0.13.0',
timestamp: new Date().toISOString(),
...entry,
}) + '\n';
appendFileSync(getRollbackFile(), line, 'utf-8');
timestamp: ts,
id: r.id,
slug: r.slug,
source_id: r.source_id ?? 'default',
pre_frontmatter: r.frontmatter ?? {},
})).join('\n') + '\n';
appendFileSync(getRollbackFile(), lines, 'utf-8');
}
// ---------------------------------------------------------------------------
+4 -4
View File
@@ -18,7 +18,6 @@
* C. Record — append completed.jsonl.
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
@@ -28,10 +27,11 @@ const REQUIRED_TABLES = ['subagent_messages', 'subagent_tool_executions', 'subag
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -88,7 +88,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalize(phases, 'failed');
+4 -4
View File
@@ -19,7 +19,6 @@
* Idempotent: safe to re-run on partial state.
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
@@ -27,10 +26,11 @@ import { createEngine } from '../../core/engine-factory.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -174,7 +174,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalize(phases, 'failed');
+4 -4
View File
@@ -16,15 +16,15 @@
* fires on upgrade, because doctor + connectEngine never call initSchema().
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -36,7 +36,7 @@ function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
const phases: OrchestratorPhaseResult[] = [];
phases.push(phaseASchema(opts));
phases.push(await phaseASchema(opts));
const anyFailed = phases.some(p => p.status === 'failed');
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
+4 -9
View File
@@ -25,20 +25,15 @@
* All phases are idempotent and safe to re-run.
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
stdio: 'inherit',
timeout: 600_000,
env: process.env,
});
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -102,7 +97,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalizeResult(phases, 'failed');
+4 -9
View File
@@ -18,21 +18,16 @@
* D. Record — handled by the runner.
*/
import { execSync } from 'child_process';
import type { BrainEngine } from '../../core/engine.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
stdio: 'inherit',
timeout: 600_000, // 10 min — duplicate-heavy installs can be slow
env: process.env,
});
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
return { name: 'schema', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
@@ -129,7 +124,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalize(phases, 'failed');
+19 -2
View File
@@ -24,6 +24,7 @@
import type { BrainEngine } from '../core/engine.ts';
import { MARKDOWN_CHUNKER_VERSION } from '../core/chunkers/recursive.ts';
import { importFromContent, importFromFile } from '../core/import-file.ts';
import { serializeMarkdown } from '../core/markdown.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { existsSync } from 'fs';
@@ -220,8 +221,24 @@ export async function runReindex(engine: BrainEngine, args: string[]): Promise<R
return;
}
}
// Fallback path: re-chunk stored compiled_truth in place.
await importFromContent(engine, row.slug, row.compiled_truth, {
// No source file on disk (DB-only page, or repo not available) —
// re-chunk from the stored page. v0.41.37.0 #1621: reconstruct the
// FULL markdown (frontmatter + body + timeline) via serializeMarkdown
// and re-import THAT, instead of passing body-only `compiled_truth`
// to importFromContent. The body-only path re-parsed with empty
// frontmatter and OVERWROTE the page's real frontmatter / title /
// timeline (codex catch). The round-trip preserves everything while
// still re-chunking + bumping chunker_version.
const page = await engine.getPage(row.slug, { sourceId: row.source_id });
if (!page) { skipped++; return; }
const tags = await engine.getTags(row.slug, { sourceId: row.source_id });
const fullMarkdown = serializeMarkdown(
page.frontmatter ?? {},
page.compiled_truth ?? row.compiled_truth,
page.timeline ?? '',
{ type: page.type, title: page.title, tags },
);
await importFromContent(engine, row.slug, fullMarkdown, {
sourceId: row.source_id,
noEmbed: !!opts.noEmbed,
forceRechunk: true,
+35 -2
View File
@@ -226,6 +226,15 @@ export interface SyncOpts {
skipFailed?: boolean;
/** Bug 9 — re-attempt unacknowledged failures explicitly (CLI --retry-failed). */
retryFailed?: boolean;
/**
* v0.41.37.0 #1569 — skip loading the active schema pack during sync. When set,
* `loadActivePack` is not called, so no user-supplied pack page-type regex
* (markdown.ts subtype path_pattern) runs during import. Pages fall back to
* legacy prefix typing. Escape hatch for completing a sync when a suspect
* pack regex is the suspected cause of a wedge; re-run extraction later.
* Threaded through performSync AND syncOneSource so `sync --all` honors it.
*/
noSchemaPack?: boolean;
/**
* v0.18.0 Step 5 — sync a specific named source. When set, sync reads
* local_path + last_commit from the sources table (not the global
@@ -961,6 +970,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// failure falls through to legacy inferType (parity preserved).
let syncActivePack: { page_types: ReadonlyArray<{ name: string; path_prefixes: ReadonlyArray<string> }> } | undefined;
try {
// v0.41.37.0 #1569: --no-schema-pack escape hatch. Skip pack load entirely so
// no user-supplied pack regex (markdown.ts subtype path_pattern) runs during
// sync; pages fall back to legacy prefix typing.
if (opts.noSchemaPack) {
serr('[sync] --no-schema-pack: skipping schema pack; pages use legacy prefix typing');
throw new Error('schema-pack-skipped');
}
const { loadActivePack } = await import('../core/schema-pack/load-active.ts');
const { loadConfig } = await import('../core/config.ts');
const resolved = await loadActivePack({
@@ -1589,6 +1605,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
progress.tick(1, `skip:${path}`);
return;
}
// v0.41.37.0 #1569: per-file BEGIN heartbeat, emitted BEFORE importFile so a
// hang names the stalling file (the progress.tick below only fires AFTER
// importFile returns — useless when one file wedges). Off by default
// (GBRAIN_SYNC_TRACE=1) to avoid a line per file on huge brains. serr is
// source-prefix-aware, so under --workers>1 / --all the stuck file is the
// begin-line with no matching completion in the in-flight set.
if (process.env.GBRAIN_SYNC_TRACE) serr(`[sync] begin import: ${path}`);
try {
// v0.18.0+ multi-source: thread `opts.sourceId` so per-page tx writes
// (putPage / getTags / addTag / removeTag / deleteChunks / upsertChunks
@@ -2078,6 +2101,12 @@ Options:
--watch Re-sync continuously on an interval.
--interval N Watch-mode interval in seconds (default 60).
--no-pull Skip 'git pull' before the sync (useful for tests).
--no-schema-pack Skip loading the active schema pack (no per-file pack
regex runs; pages use legacy prefix typing). Escape
hatch if a suspect pack regex is wedging sync.
PGLite is single-writer: stop 'gbrain serve' before a
large sync (see docs/architecture/serve-sync-concurrency.md).
GBRAIN_SYNC_TRACE=1 names the file being imported (hang triage).
--all Sync every registered source instead of just the
default (multi-source brains).
--parallel N (with --all) Run up to N sources concurrently.
@@ -2113,6 +2142,7 @@ See also:
const noEmbed = args.includes('--no-embed');
const skipFailed = args.includes('--skip-failed');
const retryFailed = args.includes('--retry-failed');
const noSchemaPack = args.includes('--no-schema-pack'); // v0.41.37.0 #1569
const syncAll = args.includes('--all');
const jsonOut = args.includes('--json');
const yesFlag = args.includes('--yes');
@@ -2540,7 +2570,7 @@ See also:
repoPath: src.local_path!,
dryRun, full, noPull,
noEmbed: effectiveNoEmbed,
skipFailed, retryFailed,
skipFailed, retryFailed, noSchemaPack,
sourceId: src.id,
strategy: cfg.strategy,
concurrency,
@@ -2741,7 +2771,7 @@ See also:
: undefined;
singleSourceTimer?.unref?.();
const opts: SyncOpts = {
repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId,
repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, noSchemaPack, sourceId,
strategy: strategyArg, concurrency,
signal: singleSourceController?.signal,
};
@@ -2901,6 +2931,8 @@ export async function syncOneSource(
skipFailed: boolean;
retryFailed: boolean;
concurrency: number | undefined;
/** v0.41.37.0 #1569: propagate --no-schema-pack into every per-source sync. */
noSchemaPack?: boolean;
},
): Promise<{ result: SyncResult; log: string }> {
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
@@ -2913,6 +2945,7 @@ export async function syncOneSource(
noEmbed: shared.noEmbed,
skipFailed: shared.skipFailed,
retryFailed: shared.retryFailed,
noSchemaPack: shared.noSchemaPack,
sourceId: src.id,
strategy: cfg.strategy,
concurrency: shared.concurrency,
+19 -6
View File
@@ -669,12 +669,25 @@ export async function importFromContent(
);
}
// Tag reconciliation: remove stale, add current
const existingTags = await tx.getTags(slug, txOpts);
const newTags = new Set(parsed.tags);
for (const old of existingTags) {
if (!newTags.has(old)) await tx.removeTag(slug, old, txOpts);
}
// Tag reconciliation: ADD-ONLY (v0.41.37.0 #1621).
//
// We deliberately do NOT delete existing tags here. The `tags` table has
// no provenance column, and frontmatter tags are stripped from the stored
// `pages.frontmatter` (markdown.ts:118) — so at re-import time we cannot
// distinguish a frontmatter-origin tag from a DB-side enrichment tag
// (auto-tag / dream synthesize / signal-detector writes to the same
// table). The pre-v0.41.37.0 "delete every existing tag not in the current
// frontmatter" logic wiped ALL enrichment tags on every re-import — most
// visibly under `gbrain reindex --markdown` (#1621), which re-imports every
// page with forceRechunk. reindex is a re-chunk/re-embed op; it must not
// destroy tags.
//
// Trade-off (accepted): removing a tag from a page's frontmatter no longer
// removes it from the DB on the next sync. That staleness is minor (tags
// are additive metadata) and far preferable to silently losing enrichment
// tags. Frontmatter-tag REMOVAL would require a `tag_source` provenance
// column (deferred — see TODOS.md #1621-followup). addTag is idempotent
// (ON CONFLICT DO NOTHING), so re-adding existing tags is a no-op.
for (const tag of parsed.tags) {
await tx.addTag(slug, tag, txOpts);
}
+9 -5
View File
@@ -39,7 +39,7 @@
// universe.
import type { SchemaPackManifest } from './manifest-v1.ts';
import { PageRegexBudget } from './redos-guard.ts';
import { PageRegexBudget, runRegexBounded } from './redos-guard.ts';
/**
* Try to resolve a link verb from the active pack's declared
@@ -80,12 +80,16 @@ export function inferLinkTypeFromPack(
}
if (match !== null) return lt.name;
} else {
// No budget provided (test contexts) — run the regex directly.
// No budget provided (test contexts) — still route through the bounded
// executor so the v0.41.37.0 #1569 input-length cap + vm timeout apply.
// Previously this ran `new RegExp(pattern).test(context)` UNBOUNDED, the
// one ReDoS hole with no timeout. runRegexBounded throws on
// timeout/oversize/malformed → skip and continue (degrade to mentions).
try {
if (new RegExp(pattern).test(context)) return lt.name;
if (runRegexBounded(pattern, context) !== null) return lt.name;
} catch {
// Malformed pattern — skip and continue. Pack validation
// should have caught this at load.
// Timed out, oversize input, or malformed pattern — skip and continue.
// Pack validation + the star-height lint rule surface bad patterns.
}
}
}
+29
View File
@@ -320,6 +320,34 @@ export const mutationCountAnomaly: LintRule = (manifest, opts) => {
// Aggregator
// ────────────────────────────────────────────────────────────────────────
// v0.41.37.0 #1569: advisory ReDoS pre-screen for pack inference regexes.
// Flags the classic nested-quantifier shapes ((a+)+, (a*)*, (a+)*, (\w+)+)
// that cause catastrophic backtracking. WARNING, not error: a hard reject
// would disable the whole pack on upgrade (pages fall back to legacy typing).
// The runtime input-length cap (MAX_REGEX_INPUT_CHARS in redos-guard.ts) is
// the actual safety net; this rule tells the author to fix the pattern.
// Heuristic: an inner group containing a +/* quantifier, wrapped by an outer
// +/* quantifier. Catches the common ReDoS class, not every possible one.
const NESTED_QUANTIFIER_RE = /\([^()]*[+*][^()]*\)\s*[+*]/;
export const linkRegexCatastrophicBacktrack: LintRule = (manifest) => {
const issues: LintIssue[] = [];
for (const lt of manifest.link_types) {
const pattern = lt.inference?.regex;
if (!pattern) continue;
if (NESTED_QUANTIFIER_RE.test(pattern)) {
issues.push({
rule: 'link_regex_catastrophic_backtrack',
severity: 'warning',
message: `link_type '${lt.name}' inference.regex '${pattern}' has a nested quantifier (e.g. (a+)+) that can cause catastrophic backtracking (ReDoS)`,
pack: manifest.name,
link: lt.name,
hint: `rewrite without nested quantifiers (e.g. (a+)+ → a+). The runtime caps input length, but the pattern stays O(2^n) on adversarial input`,
});
}
}
return issues;
};
/** All rules. File-plane callers can compose a subset via FILE_PLANE_RULES. */
export const ALL_LINT_RULES: ReadonlyArray<{ name: string; rule: LintRule; planeAware: boolean }> = [
{ name: 'alias_shadows_type', rule: aliasShadowsType, planeAware: false },
@@ -331,6 +359,7 @@ export const ALL_LINT_RULES: ReadonlyArray<{ name: string; rule: LintRule; plane
{ name: 'expert_routing_without_prefix', rule: expertRoutingWithoutPrefix, planeAware: false },
{ name: 'prefix_collision', rule: prefixCollision, planeAware: false },
{ name: 'prefix_strict_subset_overlap', rule: prefixStrictSubsetOverlap, planeAware: false },
{ name: 'link_regex_catastrophic_backtrack', rule: linkRegexCatastrophicBacktrack, planeAware: false },
{ name: 'extractable_empty_corpus', rule: extractableEmptyCorpus, planeAware: true },
{ name: 'mutation_count_anomaly', rule: mutationCountAnomaly, planeAware: true },
];
+29
View File
@@ -37,6 +37,28 @@ import { runInContext, createContext } from 'node:vm';
export const LINK_EXTRACTION_TOTAL_BUDGET_MS = 500 as const;
export const PER_REGEX_TIMEOUT_MS = 50 as const;
// v0.41.37.0 #1569: hard input-length cap. Catastrophic backtracking needs a
// long input to blow up; a pack regex run against a multi-KB body is the blast
// radius. Capping the input length removes it cheaply — a link-extraction
// `context` is normally a sentence or short paragraph, so 64KB is generous.
// Over the cap, the regex is skipped (degrade-to-mentions) without even
// entering the vm. This is the real runtime safety net (the star-height lint
// rule is advisory). Env-overridable for power users with huge contexts.
export const MAX_REGEX_INPUT_CHARS = (() => {
const raw = process.env.GBRAIN_MAX_REGEX_INPUT_CHARS;
const n = raw ? parseInt(raw, 10) : NaN;
return Number.isFinite(n) && n > 0 ? n : 64_000;
})();
/** Tagged error thrown when input exceeds MAX_REGEX_INPUT_CHARS. Treated as
* degrade-to-mentions by `PageRegexBudget.runBounded` (counts against budget). */
export class RegexInputTooLargeError extends Error {
constructor(public readonly length: number) {
super(`regex input ${length} chars exceeds cap ${MAX_REGEX_INPUT_CHARS}`);
this.name = 'RegexInputTooLargeError';
}
}
export class RegexTimeoutError extends Error {
readonly verb: string;
readonly pattern: string;
@@ -123,6 +145,13 @@ export function runRegexBounded(
text: string,
timeoutMs: number = PER_REGEX_TIMEOUT_MS,
): RegExpMatchArray | null {
// v0.41.37.0 #1569: input-length cap BEFORE the vm. Over the cap, skip the
// regex entirely (the surrounding budget treats the throw as degrade). This
// is the primary ReDoS safety net — catastrophic backtracking can't blow up
// on input it never sees.
if (text.length > MAX_REGEX_INPUT_CHARS) {
throw new RegexInputTooLargeError(text.length);
}
// Create a fresh context so the pack's regex can't leak state across
// runs. Pass pattern + text as primitives only.
const ctx = createContext({ pattern, text });