v0.41.37.0 fix: critical fix wave — reindex tag wipe, grandfather hang, Windows migration spawn, sync ReDoS (#1621 #1581 #1605 #1569) (#1665)

* fix(reindex): add-only tag reconciliation + DB-only re-chunk preserves frontmatter (#1621)

reindex --markdown and re-import no longer wipe DB-side enrichment tags.
Tag reconciliation is now ADD-ONLY (import-file.ts): re-import adds current
frontmatter tags and never deletes, so auto/dream/signal-detector tags survive.
The reindex DB-only fallback reconstructs full markdown via serializeMarkdown
so re-chunking a page with no on-disk source preserves frontmatter/title/timeline.

* fix(migrations): v0.13.1 grandfather chunked, source-safe, soft-delete-filtered (#1581)

phaseCGrandfather rewritten from a per-page getPage+putPage loop (which hung
70+ min on an 82K-page PGLite brain) to a chunked bulk SQL pass keyed on
pages.id (NOT slug — slug isn't globally unique), filtering deleted_at IS NULL,
with a batched rollback log carrying source identity.

* fix(migrations): run schema phases in-process to fix Windows getaddrinfo ENOTFOUND (#1605)

The 9 'gbrain init --migrate-only' execSync spawns died on Windows+bun+Supabase
(child DNS resolution). runMigrateOnlyCore (extracted from initMigrateOnly) runs
the schema bring-up in-process for all engines, unblocking schema_version
advancement. Includes async-call-site audit, a wall-clock guard, and a
runGbrainSubprocess stderr-capture wrapper for the remaining backfill spawns.

* fix(sync): ReDoS hardening + diagnostics for schema-pack regexes (#1569)

Input-length cap in runRegexBounded + route the unbounded link-inference path
through it (closes the only no-timeout ReDoS hole); star-height lint rule warns
on nested-quantifier patterns; --no-schema-pack sync escape hatch; GBRAIN_SYNC_TRACE
per-file begin heartbeat; PGLite serve/sync concurrency doc. Defensive hardening +
diagnostics — the deterministic ~3100-file wedge root cause remains open (no repro).

* docs(todos): file v0.41.37.0 fix-wave follow-ups (#1621/#1605/#1569)

* chore: bump version and changelog (v0.41.37.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: sync README + CLAUDE.md for v0.41.37.0 critical fix wave

Add reindex add-only tag reconciliation (#1621), v0.13.1 grandfather +
Windows in-process migration (#1581/#1605), and schema-pack ReDoS
hardening + sync --no-schema-pack / GBRAIN_SYNC_TRACE triage (#1569)
to CLAUDE.md key-files annotations and README Troubleshooting.
Regenerated llms-full.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): bump llms-full.txt budget 700KB→750KB (CLAUDE.md crossed 700KB after master merge)

The build-llms size-budget test failed: llms-full.txt is 703,244 bytes after the
v0.41.37.0 key-files annotations merged on top of master's v0.41.34/35/36 CLAUDE.md
additions. Matches the v0.41.9.0 precedent (600→700); the single-fetch bundle still
fits comfortably in modern long-context models.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-30 14:56:26 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent ca13f40820
commit 6f26d5e4df
36 changed files with 1201 additions and 232 deletions
+117
View File
@@ -2,6 +2,110 @@
All notable changes to GBrain will be documented in this file.
## [0.41.37.0] - 2026-05-30
**A four-bug critical fix wave: your tags stop disappearing on reindex, a
migration that hung forever on big brains now finishes in seconds, Windows
users stuck mid-upgrade get unstuck, and a class of sync-time regex blowups is
capped.**
If you run `gbrain reindex --markdown` (or a cron that does), it was quietly
wiping tags. Specifically: any tag that wasn't written in the page's `.md`
frontmatter — the ones your dream cycle, signal-detector, or auto-tagging added
straight to the database — got deleted on every reindex. One reporter watched
their distinct-tag count fall from ~848 to ~100. That's fixed: reindex and
re-import are now **add-only**. They add whatever tags are in the frontmatter
and never delete, so database-side tags survive. (The trade-off: removing a tag
from frontmatter no longer removes it from the DB on the next sync — additive
metadata, low harm, and a provenance-column fix is filed for later.)
If you're on a large brain and `gbrain apply-migrations` ever hung, the v0.13.1
"grandfather" step was the culprit. On an 82K-page brain it pinned a CPU core
for over an hour after ~4,200 pages and never finished. It walked every page one
at a time. It now does the same work as a handful of bulk SQL statements and
finishes in a second or two. It's also now safe on multi-source brains (it keys
on the page's unique id, not its slug) and leaves soft-deleted pages alone.
If you're on Windows with a Supabase brain and got stuck at an old
`schema_version` while the binary kept upgrading, this one's for you. The
migration step shelled out to a child `gbrain init --migrate-only` process, and
on Windows+bun+Supabase that child died with `getaddrinfo ENOTFOUND` before it
could connect — even though the parent connected fine. The schema bring-up now
runs in-process (no child process), so the upgrade actually advances. The
remaining data-backfill steps that still shell out now surface the real error
instead of a bare "Command failed," so the next Windows issue is diagnosable.
And if `gbrain sync` ever pinned CPU on a brain with a custom schema pack, we
capped the blast radius: a pack's link-inference regex now has a hard
input-length limit (catastrophic backtracking needs a long input; we don't give
it one), and `gbrain schema lint` warns when a pack regex has the classic
exponential shape like `(a+)+`. New escape hatch: `gbrain sync --no-schema-pack`
completes a sync without running any pack regex. New diagnostic:
`GBRAIN_SYNC_TRACE=1 gbrain sync` prints the file being imported so a hang names
the culprit. (Honest scope: this is hardening plus diagnostics. The specific
deterministic wedge one reporter hit at ~3100 files on a 56K-file brain is not
root-caused yet — their repro sample is the next step.)
### What you might need to do
- **Lost tags from a past reindex?** They aren't recoverable from frontmatter
(they were never there). Re-run whatever enrichment originally created them.
Going forward, reindex won't wipe them.
- **Stuck mid-migration (Windows or a hung large-brain upgrade)?**
`gbrain upgrade` then `gbrain apply-migrations --yes` should now complete.
Verify with `gbrain doctor`.
- **Sync wedging on a schema-pack brain?** `gbrain sync --no-schema-pack` to get
unblocked, then `gbrain schema lint` to find the bad regex.
### Itemized changes
- **#1621 — reindex/sync no longer wipe DB-side tags.** Tag reconciliation in
`src/core/import-file.ts` is now add-only (adds current frontmatter tags via
the idempotent `ON CONFLICT DO NOTHING` `addTag`, never deletes). The
`gbrain reindex --markdown` DB-only fallback (pages with no on-disk source)
reconstructs full markdown via `serializeMarkdown(page)` before re-importing,
so re-chunking preserves frontmatter, title, and timeline instead of
overwriting them from body-only content.
- **#1581 — v0.13.1 grandfather migration no longer hangs.** `phaseCGrandfather`
rewritten from a per-page `getPage`+`putPage` loop to a chunked bulk SQL pass
keyed on `pages.id` (slug is not globally unique — a slug-keyed UPDATE would
cross-contaminate same-slug pages in other sources), filtering
`deleted_at IS NULL`, with a batched rollback log carrying `{id, slug,
source_id, pre_frontmatter}`.
- **#1605 — Windows migration upgrade unblocked.** New
`src/commands/migrations/in-process.ts` exports `runMigrateOnlyCore` (extracted
from `init.ts`'s `initMigrateOnly`, single source of truth so the
configure-gateway-before-initSchema fix can't drift). The 9
`gbrain init --migrate-only` schema-phase spawns across the migration
orchestrators now run in-process for every engine, with a wall-clock guard.
Remaining `extract`/`repair` spawns route through `runGbrainSubprocess`, which
captures child stderr (large `maxBuffer`) and folds it into the failure detail.
- **#1569 — schema-pack ReDoS hardening + sync diagnostics.** Input-length cap
(`MAX_REGEX_INPUT_CHARS`, default 64K, env-overridable) in
`redos-guard.ts:runRegexBounded`; the previously-unbounded
`link-inference.ts` path now routes through it. New `link_regex_catastrophic_backtrack`
lint rule (warning) flags nested-quantifier patterns. New `gbrain sync
--no-schema-pack` flag (threaded through the `--all` per-source path). New
`GBRAIN_SYNC_TRACE=1` per-file begin heartbeat. New
`docs/architecture/serve-sync-concurrency.md`.
### For contributors
- 8 new/updated test files (28 new cases): `test/reindex-preserve-tags.test.ts`,
`test/migrations-v0_13_1-grandfather.test.ts` (multi-source + soft-delete +
large-N), `test/migration-in-process.serial.test.ts`,
`test/redos-hardening.test.ts`, plus updates to `import-file`,
`migrations-v0_13_0`, `migrations-v0_16_0`, `schema-pack-lint-rules`, and
`fix-wave-structural` tests for the changed contracts.
- `phaseASchema` in the 9 migration orchestrators is now `async` and awaited at
every call site (a structural source-grep guard in
`migration-in-process.serial.test.ts` bans reintroducing the
`execSync('gbrain init --migrate-only')` spawn).
## To take advantage of v0.41.37.0
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain
doctor` warns about a partial migration:
## [0.41.36.0] - 2026-05-30
**Your MCP clients can now see and use your agent's skills. Point Codex desktop,
@@ -224,6 +328,19 @@ columns) automatically. If `gbrain doctor` warns about a partial migration:
```bash
gbrain apply-migrations --yes
```
2. **No agent action is required** — these are bug fixes; the mechanical schema
side is handled by the orchestrator.
3. **Verify the outcome:**
```bash
gbrain doctor
gbrain stats
```
On a brain that previously hung on v0.13.1, the cascade should now complete in
seconds. On Windows + Supabase, `schema_version` should reach head.
4. **If any step fails or numbers look wrong,** file an issue at
https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`
and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
2. **Backfill aliases for pages you already have** (one time):
```bash
gbrain reindex --aliases
+6 -5
View File
File diff suppressed because one or more lines are too long
+47
View File
@@ -340,6 +340,53 @@ anthropic/claude-sonnet-4-6 --max-cost 5` failed with
matched the colon form. Both shapes work now. No config change, no
schema migration — `gbrain upgrade` is the whole fix.
**`gbrain reindex --markdown` wiped your auto/dream/signal-detector
tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and
`reindex --markdown` now ADD current frontmatter tags and never delete,
so enrichment tags written to the DB (auto-tag, dream synthesize,
signal-detector) survive a re-chunk. The reindex DB-only fallback also
reconstructs the full markdown (frontmatter + body + timeline) before
re-chunking, so a page with no on-disk source keeps its frontmatter,
title, and timeline instead of getting overwritten with empty
frontmatter. Trade-off: removing a tag from a page's frontmatter no
longer removes it from the DB on the next sync (frontmatter-tag removal
needs a provenance column, deferred). (Closes #1621.)
**`gbrain sync` wedges on a large brain (no progress, high CPU)?**
v0.41.37.0 ships three things. First, name the stalling file:
```bash
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
```
The last `[sync] begin import: <path>` line with no following completion
is the file being processed when the hang hit. Second, if you suspect a
schema-pack `inference.regex` with catastrophic backtracking, complete
the sync with the pack disabled and re-run extraction later:
```bash
gbrain sync --no-schema-pack --no-pull --no-embed --yes
```
`gbrain schema lint` now warns on the classic nested-quantifier ReDoS
shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps
inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`).
Third, on a PGLite brain, stop `gbrain serve` before a large sync —
PGLite is single-writer and a live MCP server contends for the write
lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md)
for the full triage. (Closes #1569.)
**`gbrain init --migrate-only` / a schema migration fails on Windows
with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up
phases in-process instead of spawning a child `gbrain init
--migrate-only` per phase. The spawned child died on
Windows + bun + Supabase pooler with a DNS-resolution failure even
though the parent connected fine; running in-process removes the spawn
entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an
82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL
pass (keyed on the page PK, soft-delete-filtered, source-safe) that
completes in ~1-2 seconds. (Closes #1605, #1581.)
## Docs
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
+12
View File
@@ -1,5 +1,17 @@
# TODOS
## v0.41.37.0 critical-fix-wave follow-ups (v0.42+)
Filed from the v0.41.37.0 wave (#1621 tag-wipe, #1581 grandfather hang,
#1605 Windows migration spawn, #1569 sync ReDoS hardening). Each item was
deliberately scoped out of the wave (see plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-greedy-quiche.md`).
- [ ] **#1621-followup: tag_source provenance column for frontmatter-tag REMOVAL.** The wave shipped ADD-ONLY tag reconciliation (`src/core/import-file.ts`) — re-import never deletes tags, so DB-side enrichment tags survive. Trade-off: removing a tag from a page's frontmatter no longer removes it from the DB. To restore removal-on-edit without wiping enrichment tags, add a `tags.tag_source` column (migration, both engines), stamp `'frontmatter'` on import-path tags, and reconcile by deleting only `tag_source='frontmatter'` tags absent from the new frontmatter (enrichment/backfilled tags default NULL = preserved, so no enrichment-write-site enumeration needed). Priority: P3 (additive-metadata staleness is low-harm).
- [ ] **#1605-followup: convert migration backfill-phase spawns to in-process.** v0.41.37.0 made the 9 schema phases (`gbrain init --migrate-only`) run in-process via `runMigrateOnlyCore`, which unblocks `schema_version` advancement on Windows+bun+Supabase. The remaining non-schema spawns (`extract links/timeline`, `repair-jsonb`) still shell out via `runGbrainSubprocess` — they now surface child stderr (so a Windows failure is diagnosable) but still fail on Windows. Convert them to in-process calls (the extract/repair command functions are callable with an engine) so Windows brains complete data backfill, not just schema. Sites: `src/commands/migrations/v0_12_0.ts` (extract), `v0_12_2.ts` (repair), `v0_13_0.ts` (extract). Priority: P2.
- [ ] **#1569-followup: root-cause the 56K-file sync wedge with the reporter's repro.** v0.41.37.0 shipped ReDoS hardening (input-length cap + star-height lint + `--no-schema-pack` escape) + diagnostics (`GBRAIN_SYNC_TRACE=1` begin-heartbeat + PGLite serve/sync concurrency doc), but did NOT root-cause the deterministic wedge at ~3100 files — the reporter's redos-guard hypothesis didn't hold (it's not on the sync path). Get the reporter's sample files (`/tmp/gbrain-hang-sample.txt`, `/tmp/gbrain-prewedge-sample.txt`), reproduce, and pin the resume-mode deep-recursion pre-import phase (prime suspect: the walk/diff/checkpoint path). Priority: P1 once a repro exists; tracked on the #1569 thread.
## MCP skillpack distribution — PR2 (v0.41.37+)
Filed from the v0.41.36.0 skill-catalog wave (`list_skills` / `get_skill`).
+1 -1
View File
@@ -1 +1 @@
0.41.36.0
0.41.37.0
@@ -0,0 +1,54 @@
# `gbrain serve` ↔ `gbrain sync` concurrency (PGLite)
**Short version: on a PGLite brain, stop `gbrain serve` before a large sync.**
## Why
PGLite is a single-writer embedded Postgres (WASM). A running `gbrain serve`
(stdio or HTTP MCP) holds an open PGLite connection on the brain's data
directory. `gbrain sync` needs to write to that same data directory. The two
contend for PGLite's single-writer connection / write-lock — **this is NOT the
`gbrain-sync` advisory lock** (that's a separate, DB-row coordination lock for
two concurrent *syncs*). Confusing the two sends you debugging the wrong surface.
Symptoms of serve↔sync contention on PGLite:
- `gbrain sync` blocks acquiring the PGLite write lock, or makes very slow
progress, while a `gbrain serve` process is alive on the same brain.
- Killing stale `gbrain serve` MCP processes frees the lock and sync proceeds.
## What to do
1. Stop any `gbrain serve` process for this brain before a large sync:
```bash
pkill -f 'gbrain serve' # or stop your MCP client / Claude Desktop / Cursor
gbrain sync --no-pull --no-embed --yes
```
2. Restart `gbrain serve` after the sync completes.
This contention does **not** apply to the Postgres engine — Postgres tolerates
concurrent connections, so `serve` and `sync` can run simultaneously there.
## Diagnosing a sync hang
If a sync wedges (no progress, high CPU), re-run with the per-file begin trace
so the stalling file is named:
```bash
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
```
The last `[sync] begin import: <path>` line with no following completion is the
file being processed when the hang occurred. Under `--workers >1` / `--all`,
the stuck file is in the set of begin-lines without a matching completion.
If you suspect a schema-pack regex is the cause (a pack with a
catastrophic-backtracking `inference.regex`), complete the sync with the pack
disabled and re-run extraction afterward:
```bash
gbrain sync --no-schema-pack --no-pull --no-embed --yes
```
`gbrain schema lint` flags the classic nested-quantifier ReDoS shapes
(`(a+)+`, `(a*)*`, …) in pack regexes as warnings.
+53 -5
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -141,5 +141,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.41.36.0"
"version": "0.41.37.0"
}
+6 -5
View File
@@ -255,9 +255,10 @@ export const INLINE_TIPS = [
"`gbrain upgrade` runs post-upgrade + apply-migrations.",
];
// Target ~700KB so llms-full.txt fits in ~175k-token contexts with room to spare.
// Bumped from 600KB in v0.41.9.0 — CLAUDE.md grew past 600KB after the wave's
// new-file annotations + Conductor branch-name iron-rule landed; the bundle
// still fits comfortably in modern long-context models.
// Target so llms-full.txt fits in ~190k-token contexts with room to spare.
// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB in v0.41.37.0 — CLAUDE.md
// crossed 700KB once the critical-fix-wave key-files annotations merged on top
// of master's v0.41.34/35/36 additions (703KB). Still fits comfortably in
// modern long-context models.
// Generator prints a WARN if exceeded; ship with includeInFull=false exclusions.
export const FULL_SIZE_BUDGET = 700_000;
export const FULL_SIZE_BUDGET = 750_000;
+15 -32
View File
@@ -515,44 +515,27 @@ async function resolveChatByEnv(out: ResolvedAIOptions): Promise<void> {
* clobbering the user's chosen engine.
*/
async function initMigrateOnly(opts: { jsonOutput: boolean }) {
const config = loadConfig();
if (!config) {
const msg = 'No brain configured. Run `gbrain init` (interactive) or `gbrain init --pglite` / `gbrain init --supabase` first.';
// v0.41.37.0 #1605: delegate to the shared runMigrateOnlyCore so the CLI path
// and the in-process migration-orchestrator path can't drift (single source
// of truth for configureGateway-before-initSchema + the schema bring-up).
const { runMigrateOnlyCore, MigrateOnlyError } = await import('./migrations/in-process.ts');
try {
const result = await runMigrateOnlyCore();
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'error', reason: 'no_config', message: msg }));
console.log(JSON.stringify({ status: 'success', engine: result.engine, mode: 'migrate-only' }));
} else {
console.log(`Schema up to date (engine: ${result.engine}).`);
}
} catch (e) {
const isNoConfig = e instanceof MigrateOnlyError && e.message.startsWith('No brain configured');
const msg = e instanceof Error ? e.message : String(e);
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'error', reason: isNoConfig ? 'no_config' : 'migrate_failed', message: msg }));
} else {
console.error(msg);
}
process.exit(1);
}
// B.3: configureGateway BEFORE initSchema even on the migrate-only path,
// so a schema bump on a brain whose file config is missing the embedding
// fields doesn't fall through to stale hardcoded fallbacks. Reads
// existing config (which loadConfig already merged with env) and
// propagates it into the gateway.
const { configureGateway: configureGw } = await import('../core/ai/gateway.ts');
configureGw({
embedding_model: config.embedding_model,
embedding_dimensions: config.embedding_dimensions,
expansion_model: config.expansion_model,
chat_model: config.chat_model,
env: { ...process.env },
});
const engine = await createEngine(toEngineConfig(config));
try {
await engine.connect(toEngineConfig(config));
await engine.initSchema();
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: config.engine, mode: 'migrate-only' }));
} else {
console.log(`Schema up to date (engine: ${config.engine}).`);
}
}
/**
+139
View File
@@ -0,0 +1,139 @@
/**
* In-process migration helpers (v0.41.37.0 #1605).
*
* Why this exists: migration schema phases used to shell out to a child
* `gbrain init --migrate-only` via `execSync`. On Windows + bun + Supabase
* pooler, the spawned CHILD process dies with `getaddrinfo ENOTFOUND` before it
* can connect — even though the PARENT connects fine and `env: process.env` is
* passed. It is a bun-on-Windows child-process DNS-resolution failure, not an
* env-propagation bug. The only robust fix is to not spawn at all: run the
* schema bring-up IN-PROCESS. The PGLite path at v0_11_0.ts already proved the
* pattern; this generalizes it to every engine + every schema phase.
*
* `runMigrateOnlyCore` is the single source of truth for "bring schema to head"
* — `init.ts:initMigrateOnly` (the `gbrain init --migrate-only` CLI path) and
* the migration orchestrators both call it, so the configureGateway-before-
* initSchema fix can't drift between them.
*
* `runGbrainSubprocess` is the diagnostic wrapper for the REMAINING (non-schema)
* gbrain-subprocess spawns (extract/repair/stats). It captures child stderr and
* folds it into the thrown error so a Windows failure shows the real
* `getaddrinfo ENOTFOUND` line instead of the bare `Command failed: ...`.
*/
import { execSync } from 'child_process';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
/** Default wall-clock guard for in-process initSchema. Matches the 600s cap
* the old `execSync('gbrain init --migrate-only', { timeout: 600_000 })` used,
* so a hung schema bring-up surfaces as a phase failure instead of wedging
* the whole cascade. */
export const MIGRATE_ONLY_TIMEOUT_MS = 600_000;
/** Large stderr buffer for captured subprocess output. `execSync`'s default
* ~1MB maxBuffer overflows on long backfills (extract/repair) and turns a
* successful run into a spurious failure. */
const SUBPROCESS_MAX_BUFFER = 64 * 1024 * 1024;
export interface MigrateOnlyResult {
/** The engine kind that was brought to head ('pglite' | 'postgres'). */
engine: string;
}
export class MigrateOnlyError extends Error {
constructor(message: string) {
super(message);
this.name = 'MigrateOnlyError';
}
}
/**
* Bring the configured brain's schema to head, in-process. Mirrors what
* `gbrain init --migrate-only` did via subprocess: configureGateway →
* createEngine → connect → initSchema → disconnect. Idempotent (initSchema is
* a no-op when already at head). Throws `MigrateOnlyError` on no-config or
* timeout so callers report a failed phase rather than hanging.
*/
export async function runMigrateOnlyCore(opts?: { timeoutMs?: number }): Promise<MigrateOnlyResult> {
const config = loadConfig();
if (!config) {
throw new MigrateOnlyError(
'No brain configured. Run `gbrain init` (interactive) or `gbrain init --pglite` / `gbrain init --supabase` first.',
);
}
// configureGateway BEFORE initSchema (init.ts B.3): a schema bump on a brain
// whose file config is missing embedding fields must not fall through to
// stale hardcoded fallbacks. loadConfig already merged env; propagate it.
const { configureGateway } = await import('../../core/ai/gateway.ts');
configureGateway({
embedding_model: config.embedding_model,
embedding_dimensions: config.embedding_dimensions,
expansion_model: config.expansion_model,
chat_model: config.chat_model,
env: { ...process.env },
});
const timeoutMs = opts?.timeoutMs ?? MIGRATE_ONLY_TIMEOUT_MS;
const engine = await createEngine(toEngineConfig(config));
try {
await engine.connect(toEngineConfig(config));
await withTimeout(
engine.initSchema(),
timeoutMs,
`schema init timed out after ${Math.round(timeoutMs / 1000)}s`,
);
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
return { engine: config.engine };
}
/**
* Run a `gbrain ...` subcommand as a subprocess, capturing child stderr so a
* failure surfaces the real reason. Used for the non-schema backfill phases
* (extract/repair/stats) that aren't yet in-process. On Windows these may still
* fail with `getaddrinfo ENOTFOUND`, but the operator now sees WHY instead of a
* bare `Command failed`. Returns captured stdout (utf-8) on success.
*
* Note: stderr is piped (captured), so gbrain progress lines (which go to
* stderr) are not shown live during these phases — acceptable for a one-shot
* `apply-migrations` run; the failure reason matters more than live progress.
*/
export function runGbrainSubprocess(cmd: string, opts?: { timeoutMs?: number }): string {
try {
const out = execSync(cmd, {
stdio: ['inherit', 'pipe', 'pipe'],
timeout: opts?.timeoutMs ?? MIGRATE_ONLY_TIMEOUT_MS,
env: process.env,
maxBuffer: SUBPROCESS_MAX_BUFFER,
encoding: 'utf-8',
});
return typeof out === 'string' ? out : '';
} catch (e: unknown) {
const err = e as { message?: string; stderr?: Buffer | string };
const stderrRaw = err?.stderr
? (Buffer.isBuffer(err.stderr) ? err.stderr.toString('utf-8') : String(err.stderr))
: '';
const tail = stderrRaw.split('\n').filter(Boolean).slice(-10).join('\n');
const base = err?.message ?? String(e);
throw new Error(tail ? `${base}\n--- child stderr (tail) ---\n${tail}` : base);
}
}
/** Reject `p` if it doesn't settle within `ms`. The original promise keeps
* running (best-effort) but the caller sees a clear timeout error. */
async function withTimeout<T>(p: Promise<T>, ms: number, message: string): Promise<T> {
let timer: ReturnType<typeof setTimeout> | undefined;
const timeout = new Promise<never>((_, reject) => {
timer = setTimeout(() => reject(new MigrateOnlyError(message)), ms);
});
try {
return await Promise.race([p, timeout]);
} finally {
if (timer) clearTimeout(timer);
}
}
+8 -23
View File
@@ -23,7 +23,6 @@
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, lstatSync, statSync, realpathSync } from 'fs';
import { join, resolve, dirname } from 'path';
import { execSync } from 'child_process';
import { childGlobalFlags } from '../../core/cli-options.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { savePreferences, loadPreferences } from '../../core/preferences.ts';
// Bug 3 — appendCompletedMigration moved to the runner (apply-migrations.ts).
@@ -61,28 +60,14 @@ export interface PendingHostWorkEntry {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
// v0.36.x #1100: route PGLite through an in-process schema apply rather
// than `execSync('gbrain init --migrate-only')`. The subprocess inherits
// HOME and tries to acquire the same file lock the parent process is
// holding (or briefly released and the on-disk artifact has not finished
// settling), which deadlocks until the 30s lock timeout fires. The
// structural fix is to not spawn a subprocess for work the parent can
// do directly — Postgres tolerates concurrent connections, so the
// legacy execSync path stays for Postgres callers.
const { loadConfig, toEngineConfig } = await import('../../core/config.ts');
const cfg = loadConfig();
if (cfg?.engine === 'pglite') {
const { createEngine } = await import('../../core/engine-factory.ts');
const eng = await createEngine(toEngineConfig(cfg));
try {
await eng.connect(toEngineConfig(cfg));
await eng.initSchema();
} finally {
try { await eng.disconnect(); } catch { /* best-effort */ }
}
return { name: 'schema', status: 'complete' };
}
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
// v0.41.37.0 #1605: bring schema to head IN-PROCESS for every engine. Was an
// a `gbrain init --migrate-only` subprocess subprocess for Postgres (died with
// `getaddrinfo ENOTFOUND` on Windows+bun+Supabase-pooler before it could
// connect) plus a PGLite-only in-process branch (which separately
// deadlocked on the file lock, #1100). runMigrateOnlyCore is the single
// in-process path for both engines.
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
+7 -5
View File
@@ -31,19 +31,21 @@
*/
import { execSync } from 'child_process';
import { runGbrainSubprocess } from './in-process.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
// 10-minute budget. Migrations v8/v9 dedup with helper-index should be sub-second
// even on 80K-duplicate brains, but the outer wall-clock cap shouldn't be the
// failure mode (the prior 60s ceiling tripped Garry's production upgrade).
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -93,7 +95,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
// --source db is idempotent: the UNIQUE constraint on
// (from_page_id, to_page_id, link_type) and ON CONFLICT DO NOTHING
// make re-runs cheap. Empty brains return 0/0 quickly.
execSync('gbrain extract links --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
runGbrainSubprocess('gbrain extract links --source db' + childGlobalFlags(), { timeoutMs: 600_000 });
return { name: 'backfill_links', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -104,7 +106,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
function phaseDBackfillTimeline(opts: OrchestratorOpts): OrchestratorPhaseResult {
if (opts.dryRun) return { name: 'backfill_timeline', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain extract timeline --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
runGbrainSubprocess('gbrain extract timeline --source db' + childGlobalFlags(), { timeoutMs: 600_000 });
return { name: 'backfill_timeline', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -188,7 +190,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
// A. Schema
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') {
return finalizeResult(phases, 'failed');
+6 -4
View File
@@ -21,18 +21,20 @@
*/
import { execSync } from 'child_process';
import { runGbrainSubprocess } from './in-process.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
// Propagate global progress flags so the child shows the same mode the
// parent orchestrator is running in.
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -46,7 +48,7 @@ function phaseBRepair(opts: OrchestratorOpts): OrchestratorPhaseResult {
if (opts.dryRun) return { name: 'jsonb_repair', status: 'skipped', detail: 'dry-run' };
try {
// stdio: 'inherit' — child's stderr progress streams straight through.
execSync('gbrain repair-jsonb' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
runGbrainSubprocess('gbrain repair-jsonb' + childGlobalFlags(), { timeoutMs: 600_000 });
return { name: 'jsonb_repair', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -94,7 +96,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalizeResult(phases, 'failed');
+6 -8
View File
@@ -26,6 +26,7 @@
*/
import { execSync } from 'child_process';
import { runGbrainSubprocess } from './in-process.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts). The
// orchestrator returns its result and the runner persists it.
@@ -44,10 +45,11 @@ import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhase
// upgrade mid-migration. The shim is already the canonical wrapper; trust
// it. Regression guarded by test/migrations-v0_13_0.test.ts.
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -64,11 +66,7 @@ function phaseBBackfill(opts: OrchestratorOpts): OrchestratorPhaseResult {
// `--include-frontmatter` is the v0.13 flag that enables the canonical
// frontmatter link extractor. Default-OFF in the CLI for back-compat;
// the migration explicitly opts in because this is the canonical backfill.
execSync('gbrain extract links --source db --include-frontmatter', {
stdio: 'inherit',
timeout: 1_800_000, // 30 min hard cap; typical 2-5 min on 46K pages
env: process.env,
});
runGbrainSubprocess('gbrain extract links --source db --include-frontmatter', { timeoutMs: 1_800_000 });
return { name: 'frontmatter_backfill', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -116,7 +114,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalizeResult(phases, 'failed');
+102 -81
View File
@@ -20,13 +20,14 @@
* frontmatter snapshot. Roll back by re-applying those snapshots via
* `gbrain apply-migrations --rollback v0.13.0` (future CLI; not in scope).
*
* Scale: on a 30K-page brain, ~15s on Postgres, ~30s on PGLite. Batched in
* chunks of 100 with a commit per batch so interruption losses are bounded.
* Scale (v0.41.37.0 #1581): chunked bulk SQL — one id-snapshot SELECT + N
* (SELECT-for-rollback + UPDATE) statements of CHUNK_SIZE rows each. Completes
* in ~1-2s even on an 82K-page PGLite brain. The prior per-page
* getPage+putPage loop hung CPU-bound for 70+ min on that brain (#1581).
*
* Snapshot-slugs rule: reads engine.getAllSlugs() upfront into an in-memory
* Set before iterating. Prior learning [listpages-pagination-mutation]: any
* batch write that mutates updated_at during OFFSET pagination is unstable.
* getAllSlugs returns a full snapshot that isn't invalidated by our writes.
* Snapshot rule: the affected id set is read once via SQL up front. It isn't
* invalidated by our writes because each UPDATE flips its rows out of the
* GRANDFATHER_WHERE predicate (idempotent + resumable).
*
* Safety: does NOT call saveConfig. Prior learning [gbrain-init-default-pglite-flip]:
* bare `gbrain init` defaults to PGLite and overwrites Postgres config.
@@ -46,7 +47,6 @@ import type { BrainEngine } from '../../core/engine.ts';
// Lazy: GBRAIN_HOME may be set after module load.
const getRollbackDir = () => gbrainPath('migrations');
const getRollbackFile = () => join(getRollbackDir(), 'v0_13_1-rollback.jsonl');
const BATCH_SIZE = 100;
// ---------------------------------------------------------------------------
// Phase A — connect (no config write)
@@ -75,30 +75,35 @@ async function phaseAConnect(opts: OrchestratorOpts): Promise<{ result: Orchestr
}
}
// ---------------------------------------------------------------------------
// Phase B — snapshot slugs upfront
// ---------------------------------------------------------------------------
async function phaseBSnapshot(engine: BrainEngine): Promise<{ result: OrchestratorPhaseResult; slugs: string[] }> {
try {
const slugSet = await engine.getAllSlugs();
const slugs = [...slugSet].sort();
return {
result: { name: 'snapshot', status: 'complete', detail: `${slugs.length} slugs` },
slugs,
};
} catch (e) {
return {
result: { name: 'snapshot', status: 'failed', detail: e instanceof Error ? e.message : String(e) },
slugs: [],
};
}
}
// ---------------------------------------------------------------------------
// Phase C — grandfather: add validate:false where absent
//
// v0.41.37.0 #1581: rewritten from a per-page getPage+putPage loop (which hung
// CPU-bound for 70+ min on an 82K-page PGLite brain) to a CHUNKED bulk SQL
// pass. Three correctness properties (the per-page loop and a naive single
// bulk UPDATE both got these wrong):
// 1. Keyed on pages.id (globally unique PK), NOT slug. `slug` uniqueness is
// (source_id, slug) — a slug-batched UPDATE would mutate same-slug pages
// across other sources and produce ambiguous rollback rows.
// 2. Filters `deleted_at IS NULL` — the old getPage path hid soft-deleted
// rows; a raw UPDATE must not grandfather tombstones.
// 3. Chunked in CHUNK_SIZE batches so lock-hold stays bounded (the
// DELETE_BATCH_SIZE convention) instead of one giant transaction.
// The rollback log carries source identity ({id, slug, source_id,
// pre_frontmatter}) so a rollback is unambiguous across sources.
// ---------------------------------------------------------------------------
// Filter for pages still needing the grandfather flag. Literal `?` jsonb
// existence operator (matches src/core/embed-skip.ts convention); 'validate'
// is a hardcoded literal, no injection surface. COALESCE for null-frontmatter
// safety. deleted_at IS NULL skips soft-deleted tombstones.
const GRANDFATHER_WHERE =
"NOT (COALESCE(frontmatter, '{}'::jsonb) ? 'validate') AND deleted_at IS NULL";
// Per-chunk row count. Bounded lock-hold + write-amplification per statement
// (same rationale as engine-constants.ts DELETE_BATCH_SIZE).
const CHUNK_SIZE = 1000;
interface GrandfatherResult {
touched: number;
skipped: number;
@@ -106,60 +111,69 @@ interface GrandfatherResult {
failures: string[];
}
async function phaseCGrandfather(
// Exported for direct hermetic testing against a PGLite engine (the config /
// loadConfig flow is exercised separately). Internal helper otherwise.
export async function phaseCGrandfather(
engine: BrainEngine,
slugs: string[],
opts: OrchestratorOpts,
): Promise<{ result: OrchestratorPhaseResult; detail: GrandfatherResult }> {
ensureRollbackDir();
const gf: GrandfatherResult = { touched: 0, skipped: 0, failed: 0, failures: [] };
for (let i = 0; i < slugs.length; i += BATCH_SIZE) {
const batch = slugs.slice(i, i + BATCH_SIZE);
for (const slug of batch) {
try {
if (opts.dryRun) {
const rows = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages WHERE ${GRANDFATHER_WHERE}`,
);
const c = rows[0]?.count ?? 0;
gf.touched = typeof c === 'string' ? parseInt(c, 10) : Number(c);
return {
result: { name: 'grandfather', status: 'complete', detail: `would touch ${gf.touched} (dry-run)` },
detail: gf,
};
}
ensureRollbackDir();
// Snapshot the affected id set up front (ids only — cheap, no frontmatter
// blobs in memory). The snapshot isn't invalidated by our writes because
// each UPDATE flips the rows out of the GRANDFATHER_WHERE predicate.
const idRows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM pages WHERE ${GRANDFATHER_WHERE} ORDER BY id`,
);
const ids = idRows.map(r => Number(r.id));
for (let i = 0; i < ids.length; i += CHUNK_SIZE) {
const chunk = ids.slice(i, i + CHUNK_SIZE);
try {
const page = await engine.getPage(slug);
if (!page) { gf.skipped++; continue; }
// Rollback log BEFORE mutation: one SELECT per chunk (bounded memory),
// one appendFileSync per chunk. Carries source_id so rollback is
// unambiguous across same-slug-different-source pages.
const snap = await engine.executeRaw<{
id: number; slug: string; source_id: string | null; frontmatter: Record<string, unknown> | null;
}>(
'SELECT id, slug, source_id, frontmatter FROM pages WHERE id = ANY($1::int[])',
[chunk],
);
appendRollbackBatch(snap);
// Idempotency: skip if frontmatter already has a `validate` key
// (whether true, false, or any other value). We don't flip existing
// explicit settings.
if (page.frontmatter && Object.prototype.hasOwnProperty.call(page.frontmatter, 'validate')) {
gf.skipped++;
continue;
}
if (opts.dryRun) {
gf.touched++;
continue;
}
// Rollback log BEFORE mutation, so a crash mid-write still lets us
// revert. Append-only, one line per page, newline-terminated.
appendRollbackEntry({
slug,
pre_frontmatter: page.frontmatter ?? {},
});
const nextFrontmatter = { ...(page.frontmatter ?? {}), validate: false };
await engine.putPage(slug, {
type: page.type,
title: page.title,
compiled_truth: page.compiled_truth,
timeline: page.timeline,
frontmatter: nextFrontmatter,
});
gf.touched++;
await engine.executeRaw(
`UPDATE pages SET frontmatter = jsonb_set(COALESCE(frontmatter, '{}'::jsonb), '{validate}', 'false'::jsonb) ` +
'WHERE id = ANY($1::int[])',
[chunk],
);
gf.touched += chunk.length;
} catch (e) {
gf.failed++;
gf.failed += chunk.length;
const msg = e instanceof Error ? e.message : String(e);
gf.failures.push(`${slug}: ${msg.slice(0, 100)}`);
gf.failures.push(`chunk@${i}: ${msg.slice(0, 100)}`);
}
}
} catch (e) {
gf.failed += 1;
gf.failures.push(`grandfather: ${e instanceof Error ? e.message : String(e)}`.slice(0, 120));
}
const status: OrchestratorPhaseResult['status'] =
gf.failed > 0 ? 'failed' : 'complete';
const status: OrchestratorPhaseResult['status'] = gf.failed > 0 ? 'failed' : 'complete';
const detailStr = `touched=${gf.touched} skipped=${gf.skipped} failed=${gf.failed}`;
return {
result: { name: 'grandfather', status, detail: detailStr },
@@ -215,13 +229,10 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
try {
const { result: snapRes, slugs } = await phaseBSnapshot(engine);
phases.push(snapRes);
if (snapRes.status !== 'complete') {
return { version: '0.13.1', status: 'failed', phases };
}
const { result: gfRes, detail: gfDetail } = await phaseCGrandfather(engine, slugs, opts);
// v0.41.37.0 #1581: phaseBSnapshot (getAllSlugs) is gone — the chunked bulk
// pass filters via SQL (GRANDFATHER_WHERE), so we no longer materialize a
// full slug list in JS.
const { result: gfRes, detail: gfDetail } = await phaseCGrandfather(engine, opts);
phases.push(gfRes);
filesRewritten = gfDetail.touched;
@@ -255,13 +266,23 @@ function ensureRollbackDir(): void {
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
}
function appendRollbackEntry(entry: { slug: string; pre_frontmatter: Record<string, unknown> }): void {
const line = JSON.stringify({
// v0.41.37.0 #1581: batch rollback writer. One appendFileSync per chunk, bounded
// memory. Each line carries id + slug + source_id so a rollback is unambiguous
// across same-slug-different-source pages (pages.slug is not globally unique).
function appendRollbackBatch(
rows: ReadonlyArray<{ id: number; slug: string; source_id: string | null; frontmatter: Record<string, unknown> | null }>,
): void {
if (rows.length === 0) return;
const ts = new Date().toISOString();
const lines = rows.map(r => JSON.stringify({
migration: 'v0.13.0',
timestamp: new Date().toISOString(),
...entry,
}) + '\n';
appendFileSync(getRollbackFile(), line, 'utf-8');
timestamp: ts,
id: r.id,
slug: r.slug,
source_id: r.source_id ?? 'default',
pre_frontmatter: r.frontmatter ?? {},
})).join('\n') + '\n';
appendFileSync(getRollbackFile(), lines, 'utf-8');
}
// ---------------------------------------------------------------------------
+4 -4
View File
@@ -18,7 +18,6 @@
* C. Record — append completed.jsonl.
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
@@ -28,10 +27,11 @@ const REQUIRED_TABLES = ['subagent_messages', 'subagent_tool_executions', 'subag
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -88,7 +88,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalize(phases, 'failed');
+4 -4
View File
@@ -19,7 +19,6 @@
* Idempotent: safe to re-run on partial state.
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
@@ -27,10 +26,11 @@ import { createEngine } from '../../core/engine-factory.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -174,7 +174,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalize(phases, 'failed');
+4 -4
View File
@@ -16,15 +16,15 @@
* fires on upgrade, because doctor + connectEngine never call initSchema().
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -36,7 +36,7 @@ function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
const phases: OrchestratorPhaseResult[] = [];
phases.push(phaseASchema(opts));
phases.push(await phaseASchema(opts));
const anyFailed = phases.some(p => p.status === 'failed');
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
+4 -9
View File
@@ -25,20 +25,15 @@
* All phases are idempotent and safe to re-run.
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
stdio: 'inherit',
timeout: 600_000,
env: process.env,
});
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -102,7 +97,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalizeResult(phases, 'failed');
+4 -9
View File
@@ -18,21 +18,16 @@
* D. Record — handled by the runner.
*/
import { execSync } from 'child_process';
import type { BrainEngine } from '../../core/engine.ts';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
stdio: 'inherit',
timeout: 600_000, // 10 min — duplicate-heavy installs can be slow
env: process.env,
});
const { runMigrateOnlyCore } = await import('./in-process.ts');
await runMigrateOnlyCore();
return { name: 'schema', status: 'complete' };
} catch (e) {
return { name: 'schema', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
@@ -129,7 +124,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
const a = await phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalize(phases, 'failed');
+19 -2
View File
@@ -24,6 +24,7 @@
import type { BrainEngine } from '../core/engine.ts';
import { MARKDOWN_CHUNKER_VERSION } from '../core/chunkers/recursive.ts';
import { importFromContent, importFromFile } from '../core/import-file.ts';
import { serializeMarkdown } from '../core/markdown.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { existsSync } from 'fs';
@@ -220,8 +221,24 @@ export async function runReindex(engine: BrainEngine, args: string[]): Promise<R
return;
}
}
// Fallback path: re-chunk stored compiled_truth in place.
await importFromContent(engine, row.slug, row.compiled_truth, {
// No source file on disk (DB-only page, or repo not available) —
// re-chunk from the stored page. v0.41.37.0 #1621: reconstruct the
// FULL markdown (frontmatter + body + timeline) via serializeMarkdown
// and re-import THAT, instead of passing body-only `compiled_truth`
// to importFromContent. The body-only path re-parsed with empty
// frontmatter and OVERWROTE the page's real frontmatter / title /
// timeline (codex catch). The round-trip preserves everything while
// still re-chunking + bumping chunker_version.
const page = await engine.getPage(row.slug, { sourceId: row.source_id });
if (!page) { skipped++; return; }
const tags = await engine.getTags(row.slug, { sourceId: row.source_id });
const fullMarkdown = serializeMarkdown(
page.frontmatter ?? {},
page.compiled_truth ?? row.compiled_truth,
page.timeline ?? '',
{ type: page.type, title: page.title, tags },
);
await importFromContent(engine, row.slug, fullMarkdown, {
sourceId: row.source_id,
noEmbed: !!opts.noEmbed,
forceRechunk: true,
+35 -2
View File
@@ -226,6 +226,15 @@ export interface SyncOpts {
skipFailed?: boolean;
/** Bug 9 — re-attempt unacknowledged failures explicitly (CLI --retry-failed). */
retryFailed?: boolean;
/**
* v0.41.37.0 #1569 — skip loading the active schema pack during sync. When set,
* `loadActivePack` is not called, so no user-supplied pack page-type regex
* (markdown.ts subtype path_pattern) runs during import. Pages fall back to
* legacy prefix typing. Escape hatch for completing a sync when a suspect
* pack regex is the suspected cause of a wedge; re-run extraction later.
* Threaded through performSync AND syncOneSource so `sync --all` honors it.
*/
noSchemaPack?: boolean;
/**
* v0.18.0 Step 5 — sync a specific named source. When set, sync reads
* local_path + last_commit from the sources table (not the global
@@ -961,6 +970,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// failure falls through to legacy inferType (parity preserved).
let syncActivePack: { page_types: ReadonlyArray<{ name: string; path_prefixes: ReadonlyArray<string> }> } | undefined;
try {
// v0.41.37.0 #1569: --no-schema-pack escape hatch. Skip pack load entirely so
// no user-supplied pack regex (markdown.ts subtype path_pattern) runs during
// sync; pages fall back to legacy prefix typing.
if (opts.noSchemaPack) {
serr('[sync] --no-schema-pack: skipping schema pack; pages use legacy prefix typing');
throw new Error('schema-pack-skipped');
}
const { loadActivePack } = await import('../core/schema-pack/load-active.ts');
const { loadConfig } = await import('../core/config.ts');
const resolved = await loadActivePack({
@@ -1589,6 +1605,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
progress.tick(1, `skip:${path}`);
return;
}
// v0.41.37.0 #1569: per-file BEGIN heartbeat, emitted BEFORE importFile so a
// hang names the stalling file (the progress.tick below only fires AFTER
// importFile returns — useless when one file wedges). Off by default
// (GBRAIN_SYNC_TRACE=1) to avoid a line per file on huge brains. serr is
// source-prefix-aware, so under --workers>1 / --all the stuck file is the
// begin-line with no matching completion in the in-flight set.
if (process.env.GBRAIN_SYNC_TRACE) serr(`[sync] begin import: ${path}`);
try {
// v0.18.0+ multi-source: thread `opts.sourceId` so per-page tx writes
// (putPage / getTags / addTag / removeTag / deleteChunks / upsertChunks
@@ -2078,6 +2101,12 @@ Options:
--watch Re-sync continuously on an interval.
--interval N Watch-mode interval in seconds (default 60).
--no-pull Skip 'git pull' before the sync (useful for tests).
--no-schema-pack Skip loading the active schema pack (no per-file pack
regex runs; pages use legacy prefix typing). Escape
hatch if a suspect pack regex is wedging sync.
PGLite is single-writer: stop 'gbrain serve' before a
large sync (see docs/architecture/serve-sync-concurrency.md).
GBRAIN_SYNC_TRACE=1 names the file being imported (hang triage).
--all Sync every registered source instead of just the
default (multi-source brains).
--parallel N (with --all) Run up to N sources concurrently.
@@ -2113,6 +2142,7 @@ See also:
const noEmbed = args.includes('--no-embed');
const skipFailed = args.includes('--skip-failed');
const retryFailed = args.includes('--retry-failed');
const noSchemaPack = args.includes('--no-schema-pack'); // v0.41.37.0 #1569
const syncAll = args.includes('--all');
const jsonOut = args.includes('--json');
const yesFlag = args.includes('--yes');
@@ -2540,7 +2570,7 @@ See also:
repoPath: src.local_path!,
dryRun, full, noPull,
noEmbed: effectiveNoEmbed,
skipFailed, retryFailed,
skipFailed, retryFailed, noSchemaPack,
sourceId: src.id,
strategy: cfg.strategy,
concurrency,
@@ -2741,7 +2771,7 @@ See also:
: undefined;
singleSourceTimer?.unref?.();
const opts: SyncOpts = {
repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId,
repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, noSchemaPack, sourceId,
strategy: strategyArg, concurrency,
signal: singleSourceController?.signal,
};
@@ -2901,6 +2931,8 @@ export async function syncOneSource(
skipFailed: boolean;
retryFailed: boolean;
concurrency: number | undefined;
/** v0.41.37.0 #1569: propagate --no-schema-pack into every per-source sync. */
noSchemaPack?: boolean;
},
): Promise<{ result: SyncResult; log: string }> {
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
@@ -2913,6 +2945,7 @@ export async function syncOneSource(
noEmbed: shared.noEmbed,
skipFailed: shared.skipFailed,
retryFailed: shared.retryFailed,
noSchemaPack: shared.noSchemaPack,
sourceId: src.id,
strategy: cfg.strategy,
concurrency: shared.concurrency,
+19 -6
View File
@@ -669,12 +669,25 @@ export async function importFromContent(
);
}
// Tag reconciliation: remove stale, add current
const existingTags = await tx.getTags(slug, txOpts);
const newTags = new Set(parsed.tags);
for (const old of existingTags) {
if (!newTags.has(old)) await tx.removeTag(slug, old, txOpts);
}
// Tag reconciliation: ADD-ONLY (v0.41.37.0 #1621).
//
// We deliberately do NOT delete existing tags here. The `tags` table has
// no provenance column, and frontmatter tags are stripped from the stored
// `pages.frontmatter` (markdown.ts:118) — so at re-import time we cannot
// distinguish a frontmatter-origin tag from a DB-side enrichment tag
// (auto-tag / dream synthesize / signal-detector writes to the same
// table). The pre-v0.41.37.0 "delete every existing tag not in the current
// frontmatter" logic wiped ALL enrichment tags on every re-import — most
// visibly under `gbrain reindex --markdown` (#1621), which re-imports every
// page with forceRechunk. reindex is a re-chunk/re-embed op; it must not
// destroy tags.
//
// Trade-off (accepted): removing a tag from a page's frontmatter no longer
// removes it from the DB on the next sync. That staleness is minor (tags
// are additive metadata) and far preferable to silently losing enrichment
// tags. Frontmatter-tag REMOVAL would require a `tag_source` provenance
// column (deferred — see TODOS.md #1621-followup). addTag is idempotent
// (ON CONFLICT DO NOTHING), so re-adding existing tags is a no-op.
for (const tag of parsed.tags) {
await tx.addTag(slug, tag, txOpts);
}
+9 -5
View File
@@ -39,7 +39,7 @@
// universe.
import type { SchemaPackManifest } from './manifest-v1.ts';
import { PageRegexBudget } from './redos-guard.ts';
import { PageRegexBudget, runRegexBounded } from './redos-guard.ts';
/**
* Try to resolve a link verb from the active pack's declared
@@ -80,12 +80,16 @@ export function inferLinkTypeFromPack(
}
if (match !== null) return lt.name;
} else {
// No budget provided (test contexts) — run the regex directly.
// No budget provided (test contexts) — still route through the bounded
// executor so the v0.41.37.0 #1569 input-length cap + vm timeout apply.
// Previously this ran `new RegExp(pattern).test(context)` UNBOUNDED, the
// one ReDoS hole with no timeout. runRegexBounded throws on
// timeout/oversize/malformed → skip and continue (degrade to mentions).
try {
if (new RegExp(pattern).test(context)) return lt.name;
if (runRegexBounded(pattern, context) !== null) return lt.name;
} catch {
// Malformed pattern — skip and continue. Pack validation
// should have caught this at load.
// Timed out, oversize input, or malformed pattern — skip and continue.
// Pack validation + the star-height lint rule surface bad patterns.
}
}
}
+29
View File
@@ -320,6 +320,34 @@ export const mutationCountAnomaly: LintRule = (manifest, opts) => {
// Aggregator
// ────────────────────────────────────────────────────────────────────────
// v0.41.37.0 #1569: advisory ReDoS pre-screen for pack inference regexes.
// Flags the classic nested-quantifier shapes ((a+)+, (a*)*, (a+)*, (\w+)+)
// that cause catastrophic backtracking. WARNING, not error: a hard reject
// would disable the whole pack on upgrade (pages fall back to legacy typing).
// The runtime input-length cap (MAX_REGEX_INPUT_CHARS in redos-guard.ts) is
// the actual safety net; this rule tells the author to fix the pattern.
// Heuristic: an inner group containing a +/* quantifier, wrapped by an outer
// +/* quantifier. Catches the common ReDoS class, not every possible one.
const NESTED_QUANTIFIER_RE = /\([^()]*[+*][^()]*\)\s*[+*]/;
export const linkRegexCatastrophicBacktrack: LintRule = (manifest) => {
const issues: LintIssue[] = [];
for (const lt of manifest.link_types) {
const pattern = lt.inference?.regex;
if (!pattern) continue;
if (NESTED_QUANTIFIER_RE.test(pattern)) {
issues.push({
rule: 'link_regex_catastrophic_backtrack',
severity: 'warning',
message: `link_type '${lt.name}' inference.regex '${pattern}' has a nested quantifier (e.g. (a+)+) that can cause catastrophic backtracking (ReDoS)`,
pack: manifest.name,
link: lt.name,
hint: `rewrite without nested quantifiers (e.g. (a+)+ → a+). The runtime caps input length, but the pattern stays O(2^n) on adversarial input`,
});
}
}
return issues;
};
/** All rules. File-plane callers can compose a subset via FILE_PLANE_RULES. */
export const ALL_LINT_RULES: ReadonlyArray<{ name: string; rule: LintRule; planeAware: boolean }> = [
{ name: 'alias_shadows_type', rule: aliasShadowsType, planeAware: false },
@@ -331,6 +359,7 @@ export const ALL_LINT_RULES: ReadonlyArray<{ name: string; rule: LintRule; plane
{ name: 'expert_routing_without_prefix', rule: expertRoutingWithoutPrefix, planeAware: false },
{ name: 'prefix_collision', rule: prefixCollision, planeAware: false },
{ name: 'prefix_strict_subset_overlap', rule: prefixStrictSubsetOverlap, planeAware: false },
{ name: 'link_regex_catastrophic_backtrack', rule: linkRegexCatastrophicBacktrack, planeAware: false },
{ name: 'extractable_empty_corpus', rule: extractableEmptyCorpus, planeAware: true },
{ name: 'mutation_count_anomaly', rule: mutationCountAnomaly, planeAware: true },
];
+29
View File
@@ -37,6 +37,28 @@ import { runInContext, createContext } from 'node:vm';
export const LINK_EXTRACTION_TOTAL_BUDGET_MS = 500 as const;
export const PER_REGEX_TIMEOUT_MS = 50 as const;
// v0.41.37.0 #1569: hard input-length cap. Catastrophic backtracking needs a
// long input to blow up; a pack regex run against a multi-KB body is the blast
// radius. Capping the input length removes it cheaply — a link-extraction
// `context` is normally a sentence or short paragraph, so 64KB is generous.
// Over the cap, the regex is skipped (degrade-to-mentions) without even
// entering the vm. This is the real runtime safety net (the star-height lint
// rule is advisory). Env-overridable for power users with huge contexts.
export const MAX_REGEX_INPUT_CHARS = (() => {
const raw = process.env.GBRAIN_MAX_REGEX_INPUT_CHARS;
const n = raw ? parseInt(raw, 10) : NaN;
return Number.isFinite(n) && n > 0 ? n : 64_000;
})();
/** Tagged error thrown when input exceeds MAX_REGEX_INPUT_CHARS. Treated as
* degrade-to-mentions by `PageRegexBudget.runBounded` (counts against budget). */
export class RegexInputTooLargeError extends Error {
constructor(public readonly length: number) {
super(`regex input ${length} chars exceeds cap ${MAX_REGEX_INPUT_CHARS}`);
this.name = 'RegexInputTooLargeError';
}
}
export class RegexTimeoutError extends Error {
readonly verb: string;
readonly pattern: string;
@@ -123,6 +145,13 @@ export function runRegexBounded(
text: string,
timeoutMs: number = PER_REGEX_TIMEOUT_MS,
): RegExpMatchArray | null {
// v0.41.37.0 #1569: input-length cap BEFORE the vm. Over the cap, skip the
// regex entirely (the surrounding budget treats the throw as degrade). This
// is the primary ReDoS safety net — catastrophic backtracking can't blow up
// on input it never sees.
if (text.length > MAX_REGEX_INPUT_CHARS) {
throw new RegexInputTooLargeError(text.length);
}
// Create a fresh context so the pack's regex can't leak state across
// runs. Pass pattern + text as primitives only.
const ctx = createContext({ pattern, text });
+9 -4
View File
@@ -95,11 +95,16 @@ describe('v0.36.1.x #1077 — admin register-client supports PKCE public clients
});
});
describe('v0.36.1.x #1100 — PGLite v0.11.0 phaseASchema routes in-process', () => {
test('phaseASchema branches on pglite and calls initSchema directly', () => {
describe('v0.41.37.0 #1605 — v0.11.0 phaseASchema routes in-process for ALL engines', () => {
test('phaseASchema calls runMigrateOnlyCore (in-process) + is awaited', () => {
// Supersedes #1100's PGLite-only in-process branch. v0.41.37.0 #1605 routes
// EVERY engine through runMigrateOnlyCore (no execSync subprocess at all),
// which is strictly stronger: PGLite still never subprocesses, AND the
// Windows+Postgres getaddrinfo-ENOTFOUND spawn bug is closed too.
// The eng.initSchema() call moved into src/commands/migrations/in-process.ts.
const src = readFileSync('src/commands/migrations/v0_11_0.ts', 'utf8');
expect(src).toMatch(/cfg\?\.engine\s*===\s*'pglite'/);
expect(src).toMatch(/eng\.initSchema\(\)/);
expect(src).toContain('runMigrateOnlyCore()');
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
expect(src).toMatch(/await\s+phaseASchema/);
});
+9 -3
View File
@@ -217,7 +217,11 @@ Same content.
expect(putCall).toBeUndefined();
});
test('reconciles tags: removes old, adds new', async () => {
test('reconciles tags: ADD-ONLY, never removes (v0.41.37.0 #1621)', async () => {
// Add-only reconciliation: re-import adds current frontmatter tags and
// NEVER deletes — so DB-side enrichment tags (here simulated as 'old-tag')
// survive. The pre-#1621 behavior deleted every existing tag not in the
// current frontmatter, wiping enrichment tags on every re-import/reindex.
const filePath = join(TMP, 'retag.md');
writeFileSync(filePath, `---
type: concept
@@ -239,9 +243,11 @@ Content here.
const removeCalls = calls.filter((c: any) => c.method === 'removeTag');
const addCalls = calls.filter((c: any) => c.method === 'addTag');
expect(removeCalls.length).toBe(1);
expect(removeCalls[0].args[1]).toBe('old-tag');
// No removals — 'old-tag' (enrichment) is preserved.
expect(removeCalls.length).toBe(0);
// Both current frontmatter tags are added (idempotent via ON CONFLICT).
expect(addCalls.length).toBe(2);
expect(addCalls.map((c: any) => c.args[1]).sort()).toEqual(['kept-tag', 'new-tag']);
});
test('chunks compiled_truth and timeline separately', async () => {
+112
View File
@@ -0,0 +1,112 @@
// v0.41.37.0 #1605 — migration schema phases run IN-PROCESS (was a
// `gbrain init --migrate-only` subprocess that died with getaddrinfo ENOTFOUND
// on Windows+bun+Supabase). runMigrateOnlyCore is the single in-process path;
// runGbrainSubprocess captures child stderr for the remaining backfill spawns.
import { describe, test, expect } from 'bun:test';
import { tmpdir } from 'os';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, readdirSync } from 'fs';
import { join } from 'path';
import { withEnv } from './helpers/with-env.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import {
runMigrateOnlyCore,
runGbrainSubprocess,
MigrateOnlyError,
} from '../src/commands/migrations/in-process.ts';
const MIGRATION_DIR = join(import.meta.dir, '..', 'src', 'commands', 'migrations');
const SCHEMA_PHASE_FILES = [
'v0_11_0', 'v0_12_0', 'v0_12_2', 'v0_13_0', 'v0_16_0',
'v0_18_0', 'v0_18_1', 'v0_21_0', 'v0_29_1',
];
describe('#1605 runMigrateOnlyCore (in-process schema)', () => {
test('brings a fresh PGLite brain to head without spawning', async () => {
const home = mkdtempSync(join(tmpdir(), 'mip-'));
const dataDir = join(home, 'data');
mkdirSync(join(home, '.gbrain'), { recursive: true });
writeFileSync(
join(home, '.gbrain', 'config.json'),
JSON.stringify({ engine: 'pglite', database_path: dataDir }),
);
const result = await withEnv(
{ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined },
() => runMigrateOnlyCore(),
);
expect(result.engine).toBe('pglite');
// Verify schema landed: reconnect a fresh engine to the same data dir and
// confirm a core table exists (proves initSchema ran in-process).
const verify = new PGLiteEngine();
await verify.connect({ database_path: dataDir });
try {
const rows = await verify.executeRaw<{ t: string | null }>(
"SELECT to_regclass('public.pages')::text AS t",
);
expect(rows[0]?.t).toBe('pages');
} finally {
await verify.disconnect();
}
});
test('throws MigrateOnlyError when no brain is configured', async () => {
const home = mkdtempSync(join(tmpdir(), 'mip-noconf-'));
await expect(
withEnv(
{ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined },
() => runMigrateOnlyCore(),
),
).rejects.toBeInstanceOf(MigrateOnlyError);
});
});
describe('#1605 runGbrainSubprocess (stderr capture)', () => {
test('folds child stderr into the thrown error', () => {
let msg = '';
try {
runGbrainSubprocess("sh -c 'echo BOOM_STDERR 1>&2; exit 1'");
} catch (e) {
msg = e instanceof Error ? e.message : String(e);
}
expect(msg).toContain('BOOM_STDERR');
});
test('returns child stdout on success', () => {
const out = runGbrainSubprocess("sh -c 'echo hello-stdout'");
expect(out).toContain('hello-stdout');
});
});
describe('#1605 structural guard: schema phases are in-process', () => {
test('no schema phase still execSyncs `gbrain init --migrate-only`', () => {
for (const f of SCHEMA_PHASE_FILES) {
const src = readFileSync(join(MIGRATION_DIR, `${f}.ts`), 'utf-8');
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
}
});
test('every schema phase calls runMigrateOnlyCore + is awaited', () => {
for (const f of SCHEMA_PHASE_FILES) {
const src = readFileSync(join(MIGRATION_DIR, `${f}.ts`), 'utf-8');
expect(src).toContain('runMigrateOnlyCore()');
// phaseASchema must be async + awaited at its call site.
expect(src).toContain('async function phaseASchema');
const awaited = src.includes('await phaseASchema(opts)') ||
src.includes('push(await phaseASchema(opts))');
expect(awaited).toBe(true);
}
});
test('NO migration orchestrator anywhere spawns `gbrain init --migrate-only`', () => {
// All-files invariant (not just the 9): the subprocess spawn is the
// Windows-ENOTFOUND bug class. Other files (v0_22_4, v0_28_0, v0_31_0,
// v0_14_0, v0_32_2) define an in-process phaseASchema that never spawned —
// those are fine. We only ban the spawn literal.
const files = readdirSync(MIGRATION_DIR).filter(n => /^v\d/.test(n) && n.endsWith('.ts'));
for (const n of files) {
const src = readFileSync(join(MIGRATION_DIR, n), 'utf-8');
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
}
});
});
+9 -5
View File
@@ -59,12 +59,16 @@ describe('v0.13.0 — Frontmatter relationship indexing migration', () => {
expect(src).not.toMatch(/\$\{GBRAIN\}/);
});
test('phase commands invoke bare `gbrain` shell-out (Bug 1 fix)', () => {
test('phases use in-process schema + bare `gbrain` subprocess (v0.41.37.0 #1605)', () => {
const src = readFileSync(SRC_PATH, 'utf-8');
// All three phases shell out to bare `gbrain` so the canonical shim
// on PATH wins. This is the shape v0_12_0 has always used.
expect(src).toContain("execSync('gbrain init --migrate-only'");
expect(src).toContain("execSync('gbrain extract links --source db --include-frontmatter'");
// Schema bring-up is now IN-PROCESS (was execSync('gbrain init
// --migrate-only'), which died with getaddrinfo ENOTFOUND on Windows).
expect(src).toContain('runMigrateOnlyCore()');
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
// Backfill extract goes through the stderr-capturing wrapper (still bare
// `gbrain` so the canonical shim on PATH wins).
expect(src).toContain("runGbrainSubprocess('gbrain extract links --source db --include-frontmatter'");
// Stats readback still shells out (reads stdout); bare gbrain.
expect(src).toContain("execSync('gbrain call get_stats'");
});
+144
View File
@@ -0,0 +1,144 @@
// v0.41.37.0 #1581 — chunked, source-safe, soft-delete-filtered grandfather pass.
//
// The pre-#1581 per-page getPage+putPage loop hung CPU-bound for 70+ min on an
// 82K-page PGLite brain. The rewrite is a chunked bulk SQL pass keyed on
// pages.id (NOT slug — slug isn't globally unique), filtering deleted_at IS NULL.
// These tests drive phaseCGrandfather directly against a real PGLite engine.
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { tmpdir } from 'os';
import { mkdtempSync, existsSync, readFileSync } from 'fs';
import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { withEnv } from './helpers/with-env.ts';
import { phaseCGrandfather } from '../src/commands/migrations/v0_13_1.ts';
import type { OrchestratorOpts } from '../src/commands/migrations/types.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
const ensureSource = (id: string) =>
engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT DO NOTHING', [id]);
const seed = (slug: string, frontmatter: Record<string, unknown>, sourceId = 'default') =>
engine.putPage(slug, {
type: 'concept', title: slug, compiled_truth: 'body', timeline: '', frontmatter,
}, { sourceId });
const fmOf = async (slug: string, sourceId = 'default') => {
const p = await engine.getPage(slug, { sourceId });
return p?.frontmatter ?? null;
};
// Run the phase with an isolated GBRAIN_HOME so the rollback log lands in a
// tempdir (withEnv keeps R1 test-isolation compliance — no raw process.env writes).
async function gf(home: string, opts: Partial<OrchestratorOpts> = {}) {
const full: OrchestratorOpts = { yes: true, dryRun: false, noAutopilotInstall: true, ...opts };
return withEnv({ GBRAIN_HOME: home }, () => phaseCGrandfather(engine, full));
}
describe('#1581 phaseCGrandfather (chunked, source-safe)', () => {
test('absent validate key → validate:false', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
await seed('a', { foo: 1 });
const { result } = await gf(home);
expect(result.status).toBe('complete');
expect((await fmOf('a'))?.validate).toBe(false);
});
test('explicit validate (true/false/null) is NOT modified', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
await seed('t', { validate: true });
await seed('f', { validate: false });
await seed('n', { validate: null });
await gf(home);
expect((await fmOf('t'))?.validate).toBe(true);
expect((await fmOf('f'))?.validate).toBe(false);
// null value still counts as "key present" → left untouched.
expect((await fmOf('n'))?.validate ?? 'WAS_NULL').toBe('WAS_NULL');
});
test('soft-deleted pages are NOT grandfathered', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
await seed('live', { foo: 1 });
await seed('dead', { foo: 1 });
await engine.softDeletePage('dead');
await gf(home);
expect((await fmOf('live'))?.validate).toBe(false);
const rows = await engine.executeRaw<{ fm: Record<string, unknown> }>(
"SELECT frontmatter AS fm FROM pages WHERE slug = 'dead'",
);
expect(Object.prototype.hasOwnProperty.call(rows[0]?.fm ?? {}, 'validate')).toBe(false);
});
test('multi-source duplicate slugs: both grandfathered, no cross-contamination', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
await ensureSource('src2');
await seed('people/dup', { src: 'a' }, 'default');
await seed('people/dup', { src: 'b' }, 'src2');
await gf(home);
expect((await fmOf('people/dup', 'default'))?.validate).toBe(false);
expect((await fmOf('people/dup', 'src2'))?.validate).toBe(false);
expect((await fmOf('people/dup', 'default'))?.src).toBe('a');
expect((await fmOf('people/dup', 'src2'))?.src).toBe('b');
});
test('rollback log carries source identity', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
await ensureSource('src2');
await seed('people/dup', { src: 'a' }, 'default');
await seed('people/dup', { src: 'b' }, 'src2');
await gf(home);
// configDir() appends '.gbrain' to GBRAIN_HOME.
const logPath = join(home, '.gbrain', 'migrations', 'v0_13_1-rollback.jsonl');
expect(existsSync(logPath)).toBe(true);
const lines = readFileSync(logPath, 'utf-8').trim().split('\n').map(l => JSON.parse(l));
expect(lines.map(l => l.source_id).sort()).toEqual(['default', 'src2']);
expect(lines.every(l => typeof l.id === 'number' && l.slug === 'people/dup')).toBe(true);
});
test('re-run is a no-op (idempotent)', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
await seed('a', { foo: 1 });
const first = await gf(home);
expect(first.detail.touched).toBe(1);
const second = await gf(home);
expect(second.detail.touched).toBe(0);
});
test('dry-run counts without mutating', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
await seed('a', { foo: 1 });
const { detail } = await gf(home, { dryRun: true });
expect(detail.touched).toBe(1);
expect((await fmOf('a'))?.validate ?? 'UNSET').toBe('UNSET');
});
test('large-N (1200 pages) completes via chunked pass (hang regression)', async () => {
const home = mkdtempSync(join(tmpdir(), 'gf-'));
for (let i = 0; i < 1200; i++) await seed(`bulk/${i}`, { i });
const t0 = Date.now();
const { result, detail } = await gf(home);
expect(result.status).toBe('complete');
expect(detail.touched).toBe(1200);
expect((await fmOf('bulk/0'))?.validate).toBe(false);
expect((await fmOf('bulk/1199'))?.validate).toBe(false);
expect(Date.now() - t0).toBeLessThan(30_000);
});
});
+3 -2
View File
@@ -54,8 +54,9 @@ describe('v0.16.0 migration', () => {
]);
});
test('phaseASchema skips on dry-run', () => {
const r = __testing.phaseASchema({ dryRun: true, yes: true, noAutopilotInstall: true });
test('phaseASchema skips on dry-run', async () => {
// v0.41.37.0 #1605: phaseASchema is now async (in-process runMigrateOnlyCore).
const r = await __testing.phaseASchema({ dryRun: true, yes: true, noAutopilotInstall: true });
expect(r.status).toBe('skipped');
expect(r.detail).toBe('dry-run');
});
+96
View File
@@ -0,0 +1,96 @@
// v0.41.37.0 #1569 — ReDoS hardening + diagnostics for schema-pack regexes.
//
// The real exposure was link-inference.ts:85 running `new RegExp(pattern)
// .test(context)` UNBOUNDED when no PageRegexBudget was passed. Fix: an input-
// length cap (the runtime safety net) + routing that path through the bounded
// executor + an advisory star-height lint rule.
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'fs';
import { join } from 'path';
import {
runRegexBounded,
PageRegexBudget,
RegexInputTooLargeError,
MAX_REGEX_INPUT_CHARS,
} from '../src/core/schema-pack/redos-guard.ts';
import { inferLinkTypeFromPack } from '../src/core/schema-pack/link-inference.ts';
import { linkRegexCatastrophicBacktrack } from '../src/core/schema-pack/lint-rules.ts';
import type { SchemaPackManifest } from '../src/core/schema-pack/manifest-v1.ts';
describe('#1569 input-length cap', () => {
test('runRegexBounded throws RegexInputTooLargeError over the cap', () => {
const big = 'a'.repeat(MAX_REGEX_INPUT_CHARS + 1);
expect(() => runRegexBounded('a', big)).toThrow(RegexInputTooLargeError);
});
test('runRegexBounded works normally under the cap', () => {
const m = runRegexBounded('wor', 'hello world');
expect(m).not.toBeNull();
expect(runRegexBounded('zzz', 'hello world')).toBeNull();
});
test('PageRegexBudget.runBounded degrades (null) on oversize input', () => {
const budget = new PageRegexBudget();
const big = 'a'.repeat(MAX_REGEX_INPUT_CHARS + 1);
expect(budget.runBounded('verb', 'a', big)).toBeNull();
});
test('inferLinkTypeFromPack (no budget) does NOT run regex unbounded on huge input', () => {
// Pre-#1569 this ran new RegExp().test() with no length cap → ReDoS risk.
// A catastrophic pattern + a long input must NOT hang; the cap skips it.
const pack = {
link_types: [{ name: 'founded', inference: { regex: '(a+)+$' } }],
} as unknown as Pick<SchemaPackManifest, 'link_types'>;
const huge = 'a'.repeat(MAX_REGEX_INPUT_CHARS + 100) + '!';
const t0 = Date.now();
const result = inferLinkTypeFromPack(pack, 'company', huge);
// Skipped via the cap → no match, and fast (no catastrophic backtrack).
expect(result).toBeNull();
expect(Date.now() - t0).toBeLessThan(2_000);
});
});
describe('#1569 star-height lint rule', () => {
const mk = (regex: string): SchemaPackManifest =>
({ name: 'testpack', page_types: [], link_types: [{ name: 'founded', inference: { regex } }] }) as unknown as SchemaPackManifest;
test('flags classic nested-quantifier shapes as warnings', () => {
for (const bad of ['(a+)+', '(a*)*', '(a+)*', '(\\w+)+$', '(.*)+']) {
const issues = linkRegexCatastrophicBacktrack(mk(bad)) as ReturnType<typeof linkRegexCatastrophicBacktrack> & any[];
expect(issues.length).toBe(1);
expect(issues[0].severity).toBe('warning');
expect(issues[0].rule).toBe('link_regex_catastrophic_backtrack');
expect(issues[0].link).toBe('founded');
}
});
test('does NOT flag benign patterns', () => {
for (const ok of ['a+', '(abc)+', '(a|b)+', 'founded\\s+\\w+', '[a-z]+@[a-z]+']) {
const issues = linkRegexCatastrophicBacktrack(mk(ok)) as any[];
expect(issues.length).toBe(0);
}
});
test('no regex → no issue', () => {
const manifest = { name: 'p', page_types: [], link_types: [{ name: 'x' }] } as unknown as SchemaPackManifest;
expect((linkRegexCatastrophicBacktrack(manifest) as any[]).length).toBe(0);
});
});
describe('#1569 --no-schema-pack + heartbeat wiring (structural)', () => {
const SYNC = readFileSync(join(import.meta.dir, '..', 'src', 'commands', 'sync.ts'), 'utf-8');
test('SyncOpts carries noSchemaPack and it gates loadActivePack', () => {
expect(SYNC).toContain('noSchemaPack?: boolean');
expect(SYNC).toContain("args.includes('--no-schema-pack')");
expect(SYNC).toContain('if (opts.noSchemaPack)');
});
test('begin heartbeat fires before importFile (GBRAIN_SYNC_TRACE)', () => {
const beginIdx = SYNC.indexOf('begin import:');
const importIdx = SYNC.indexOf('importFile(eng, filePath, path');
expect(beginIdx).toBeGreaterThan(0);
expect(importIdx).toBeGreaterThan(0);
expect(beginIdx).toBeLessThan(importIdx);
});
});
+75
View File
@@ -0,0 +1,75 @@
// v0.41.37.0 #1621 — reindex / re-import must NOT wipe DB-side enrichment tags.
//
// Root cause: import-file.ts reconciled tags from frontmatter only via
// DELETE-then-INSERT. The tags table has no provenance column and frontmatter
// tags are stripped from stored frontmatter (markdown.ts:118), so every
// re-import (notably `gbrain reindex --markdown`, which re-imports with
// forceRechunk) deleted all enrichment / dream / signal-detector tags.
//
// Fix: ADD-ONLY reconciliation. Re-import adds current frontmatter tags and
// never deletes. Accepted trade-off: removing a frontmatter tag no longer
// removes it from the DB (additive metadata; far better than wiping enrichment).
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { importFromContent } from '../src/core/import-file.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
const page = (tags: string[]) =>
`---\ntype: concept\ntitle: Xavi\ntags: [${tags.join(', ')}]\n---\nBody text for the page.\n`;
describe('#1621 add-only tag reconciliation', () => {
test('re-import (reindex) preserves DB-side enrichment tags', async () => {
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
// Simulate DB-side enrichment writing a tag not present in frontmatter.
await engine.addTag('people/xavi', 'enrichment-tag');
// Re-import the same content (what reindex --markdown does).
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
const tags = await engine.getTags('people/xavi');
expect(tags.sort()).toEqual(['enrichment-tag', 'founder', 'yc']);
});
test('frontmatter tags are still added on import', async () => {
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
const tags = await engine.getTags('people/xavi');
expect(tags.sort()).toEqual(['founder', 'yc']);
});
test('add-only: removing a frontmatter tag does NOT remove it (accepted trade-off)', async () => {
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
await engine.addTag('people/xavi', 'enrichment-tag');
// User removes "yc" from frontmatter and re-imports.
await importFromContent(engine, 'people/xavi', page(['founder']), { forceRechunk: true, noEmbed: true });
const tags = await engine.getTags('people/xavi');
// "yc" lingers (add-only); enrichment-tag preserved; founder present.
expect(tags.sort()).toEqual(['enrichment-tag', 'founder', 'yc']);
});
test('adding a new frontmatter tag on re-import works (idempotent add)', async () => {
await importFromContent(engine, 'people/xavi', page(['founder']), { forceRechunk: true, noEmbed: true });
await engine.addTag('people/xavi', 'enrichment-tag');
await importFromContent(engine, 'people/xavi', page(['founder', 'growth']), { forceRechunk: true, noEmbed: true });
const tags = await engine.getTags('people/xavi');
expect(tags.sort()).toEqual(['enrichment-tag', 'founder', 'growth']);
});
});
+4 -3
View File
@@ -332,12 +332,13 @@ describe('runAllLintRules — composition', () => {
});
describe('rule registry shape', () => {
it('ALL_LINT_RULES contains 11 rules', () => {
expect(ALL_LINT_RULES.length).toBe(11);
it('ALL_LINT_RULES contains 12 rules', () => {
// v0.41.37.0 #1569 added link_regex_catastrophic_backtrack (file-plane).
expect(ALL_LINT_RULES.length).toBe(12);
});
it('FILE_PLANE_LINT_RULES excludes the 2 DB-aware rules', () => {
expect(FILE_PLANE_LINT_RULES.length).toBe(9);
expect(FILE_PLANE_LINT_RULES.length).toBe(10);
expect(FILE_PLANE_LINT_RULES.every((r) => !r.planeAware)).toBe(true);
});