mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.41.37.0 fix: critical fix wave — reindex tag wipe, grandfather hang, Windows migration spawn, sync ReDoS (#1621 #1581 #1605 #1569) (#1665)
* fix(reindex): add-only tag reconciliation + DB-only re-chunk preserves frontmatter (#1621) reindex --markdown and re-import no longer wipe DB-side enrichment tags. Tag reconciliation is now ADD-ONLY (import-file.ts): re-import adds current frontmatter tags and never deletes, so auto/dream/signal-detector tags survive. The reindex DB-only fallback reconstructs full markdown via serializeMarkdown so re-chunking a page with no on-disk source preserves frontmatter/title/timeline. * fix(migrations): v0.13.1 grandfather chunked, source-safe, soft-delete-filtered (#1581) phaseCGrandfather rewritten from a per-page getPage+putPage loop (which hung 70+ min on an 82K-page PGLite brain) to a chunked bulk SQL pass keyed on pages.id (NOT slug — slug isn't globally unique), filtering deleted_at IS NULL, with a batched rollback log carrying source identity. * fix(migrations): run schema phases in-process to fix Windows getaddrinfo ENOTFOUND (#1605) The 9 'gbrain init --migrate-only' execSync spawns died on Windows+bun+Supabase (child DNS resolution). runMigrateOnlyCore (extracted from initMigrateOnly) runs the schema bring-up in-process for all engines, unblocking schema_version advancement. Includes async-call-site audit, a wall-clock guard, and a runGbrainSubprocess stderr-capture wrapper for the remaining backfill spawns. * fix(sync): ReDoS hardening + diagnostics for schema-pack regexes (#1569) Input-length cap in runRegexBounded + route the unbounded link-inference path through it (closes the only no-timeout ReDoS hole); star-height lint rule warns on nested-quantifier patterns; --no-schema-pack sync escape hatch; GBRAIN_SYNC_TRACE per-file begin heartbeat; PGLite serve/sync concurrency doc. Defensive hardening + diagnostics — the deterministic ~3100-file wedge root cause remains open (no repro). * docs(todos): file v0.41.37.0 fix-wave follow-ups (#1621/#1605/#1569) * chore: bump version and changelog (v0.41.37.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync README + CLAUDE.md for v0.41.37.0 critical fix wave Add reindex add-only tag reconciliation (#1621), v0.13.1 grandfather + Windows in-process migration (#1581/#1605), and schema-pack ReDoS hardening + sync --no-schema-pack / GBRAIN_SYNC_TRACE triage (#1569) to CLAUDE.md key-files annotations and README Troubleshooting. Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): bump llms-full.txt budget 700KB→750KB (CLAUDE.md crossed 700KB after master merge) The build-llms size-budget test failed: llms-full.txt is 703,244 bytes after the v0.41.37.0 key-files annotations merged on top of master's v0.41.34/35/36 CLAUDE.md additions. Matches the v0.41.9.0 precedent (600→700); the single-fetch bundle still fits comfortably in modern long-context models. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
ca13f40820
commit
6f26d5e4df
+117
@@ -2,6 +2,110 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.37.0] - 2026-05-30
|
||||
|
||||
**A four-bug critical fix wave: your tags stop disappearing on reindex, a
|
||||
migration that hung forever on big brains now finishes in seconds, Windows
|
||||
users stuck mid-upgrade get unstuck, and a class of sync-time regex blowups is
|
||||
capped.**
|
||||
|
||||
If you run `gbrain reindex --markdown` (or a cron that does), it was quietly
|
||||
wiping tags. Specifically: any tag that wasn't written in the page's `.md`
|
||||
frontmatter — the ones your dream cycle, signal-detector, or auto-tagging added
|
||||
straight to the database — got deleted on every reindex. One reporter watched
|
||||
their distinct-tag count fall from ~848 to ~100. That's fixed: reindex and
|
||||
re-import are now **add-only**. They add whatever tags are in the frontmatter
|
||||
and never delete, so database-side tags survive. (The trade-off: removing a tag
|
||||
from frontmatter no longer removes it from the DB on the next sync — additive
|
||||
metadata, low harm, and a provenance-column fix is filed for later.)
|
||||
|
||||
If you're on a large brain and `gbrain apply-migrations` ever hung, the v0.13.1
|
||||
"grandfather" step was the culprit. On an 82K-page brain it pinned a CPU core
|
||||
for over an hour after ~4,200 pages and never finished. It walked every page one
|
||||
at a time. It now does the same work as a handful of bulk SQL statements and
|
||||
finishes in a second or two. It's also now safe on multi-source brains (it keys
|
||||
on the page's unique id, not its slug) and leaves soft-deleted pages alone.
|
||||
|
||||
If you're on Windows with a Supabase brain and got stuck at an old
|
||||
`schema_version` while the binary kept upgrading, this one's for you. The
|
||||
migration step shelled out to a child `gbrain init --migrate-only` process, and
|
||||
on Windows+bun+Supabase that child died with `getaddrinfo ENOTFOUND` before it
|
||||
could connect — even though the parent connected fine. The schema bring-up now
|
||||
runs in-process (no child process), so the upgrade actually advances. The
|
||||
remaining data-backfill steps that still shell out now surface the real error
|
||||
instead of a bare "Command failed," so the next Windows issue is diagnosable.
|
||||
|
||||
And if `gbrain sync` ever pinned CPU on a brain with a custom schema pack, we
|
||||
capped the blast radius: a pack's link-inference regex now has a hard
|
||||
input-length limit (catastrophic backtracking needs a long input; we don't give
|
||||
it one), and `gbrain schema lint` warns when a pack regex has the classic
|
||||
exponential shape like `(a+)+`. New escape hatch: `gbrain sync --no-schema-pack`
|
||||
completes a sync without running any pack regex. New diagnostic:
|
||||
`GBRAIN_SYNC_TRACE=1 gbrain sync` prints the file being imported so a hang names
|
||||
the culprit. (Honest scope: this is hardening plus diagnostics. The specific
|
||||
deterministic wedge one reporter hit at ~3100 files on a 56K-file brain is not
|
||||
root-caused yet — their repro sample is the next step.)
|
||||
|
||||
### What you might need to do
|
||||
|
||||
- **Lost tags from a past reindex?** They aren't recoverable from frontmatter
|
||||
(they were never there). Re-run whatever enrichment originally created them.
|
||||
Going forward, reindex won't wipe them.
|
||||
- **Stuck mid-migration (Windows or a hung large-brain upgrade)?**
|
||||
`gbrain upgrade` then `gbrain apply-migrations --yes` should now complete.
|
||||
Verify with `gbrain doctor`.
|
||||
- **Sync wedging on a schema-pack brain?** `gbrain sync --no-schema-pack` to get
|
||||
unblocked, then `gbrain schema lint` to find the bad regex.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **#1621 — reindex/sync no longer wipe DB-side tags.** Tag reconciliation in
|
||||
`src/core/import-file.ts` is now add-only (adds current frontmatter tags via
|
||||
the idempotent `ON CONFLICT DO NOTHING` `addTag`, never deletes). The
|
||||
`gbrain reindex --markdown` DB-only fallback (pages with no on-disk source)
|
||||
reconstructs full markdown via `serializeMarkdown(page)` before re-importing,
|
||||
so re-chunking preserves frontmatter, title, and timeline instead of
|
||||
overwriting them from body-only content.
|
||||
- **#1581 — v0.13.1 grandfather migration no longer hangs.** `phaseCGrandfather`
|
||||
rewritten from a per-page `getPage`+`putPage` loop to a chunked bulk SQL pass
|
||||
keyed on `pages.id` (slug is not globally unique — a slug-keyed UPDATE would
|
||||
cross-contaminate same-slug pages in other sources), filtering
|
||||
`deleted_at IS NULL`, with a batched rollback log carrying `{id, slug,
|
||||
source_id, pre_frontmatter}`.
|
||||
- **#1605 — Windows migration upgrade unblocked.** New
|
||||
`src/commands/migrations/in-process.ts` exports `runMigrateOnlyCore` (extracted
|
||||
from `init.ts`'s `initMigrateOnly`, single source of truth so the
|
||||
configure-gateway-before-initSchema fix can't drift). The 9
|
||||
`gbrain init --migrate-only` schema-phase spawns across the migration
|
||||
orchestrators now run in-process for every engine, with a wall-clock guard.
|
||||
Remaining `extract`/`repair` spawns route through `runGbrainSubprocess`, which
|
||||
captures child stderr (large `maxBuffer`) and folds it into the failure detail.
|
||||
- **#1569 — schema-pack ReDoS hardening + sync diagnostics.** Input-length cap
|
||||
(`MAX_REGEX_INPUT_CHARS`, default 64K, env-overridable) in
|
||||
`redos-guard.ts:runRegexBounded`; the previously-unbounded
|
||||
`link-inference.ts` path now routes through it. New `link_regex_catastrophic_backtrack`
|
||||
lint rule (warning) flags nested-quantifier patterns. New `gbrain sync
|
||||
--no-schema-pack` flag (threaded through the `--all` per-source path). New
|
||||
`GBRAIN_SYNC_TRACE=1` per-file begin heartbeat. New
|
||||
`docs/architecture/serve-sync-concurrency.md`.
|
||||
|
||||
### For contributors
|
||||
|
||||
- 8 new/updated test files (28 new cases): `test/reindex-preserve-tags.test.ts`,
|
||||
`test/migrations-v0_13_1-grandfather.test.ts` (multi-source + soft-delete +
|
||||
large-N), `test/migration-in-process.serial.test.ts`,
|
||||
`test/redos-hardening.test.ts`, plus updates to `import-file`,
|
||||
`migrations-v0_13_0`, `migrations-v0_16_0`, `schema-pack-lint-rules`, and
|
||||
`fix-wave-structural` tests for the changed contracts.
|
||||
- `phaseASchema` in the 9 migration orchestrators is now `async` and awaited at
|
||||
every call site (a structural source-grep guard in
|
||||
`migration-in-process.serial.test.ts` bans reintroducing the
|
||||
`execSync('gbrain init --migrate-only')` spawn).
|
||||
|
||||
## To take advantage of v0.41.37.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain
|
||||
doctor` warns about a partial migration:
|
||||
## [0.41.36.0] - 2026-05-30
|
||||
|
||||
**Your MCP clients can now see and use your agent's skills. Point Codex desktop,
|
||||
@@ -224,6 +328,19 @@ columns) automatically. If `gbrain doctor` warns about a partial migration:
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **No agent action is required** — these are bug fixes; the mechanical schema
|
||||
side is handled by the orchestrator.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
gbrain stats
|
||||
```
|
||||
On a brain that previously hung on v0.13.1, the cascade should now complete in
|
||||
seconds. On Windows + Supabase, `schema_version` should reach head.
|
||||
4. **If any step fails or numbers look wrong,** file an issue at
|
||||
https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`
|
||||
and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
2. **Backfill aliases for pages you already have** (one time):
|
||||
```bash
|
||||
gbrain reindex --aliases
|
||||
|
||||
@@ -340,6 +340,53 @@ anthropic/claude-sonnet-4-6 --max-cost 5` failed with
|
||||
matched the colon form. Both shapes work now. No config change, no
|
||||
schema migration — `gbrain upgrade` is the whole fix.
|
||||
|
||||
**`gbrain reindex --markdown` wiped your auto/dream/signal-detector
|
||||
tags?** v0.41.37.0 makes tag reconciliation add-only. Re-import and
|
||||
`reindex --markdown` now ADD current frontmatter tags and never delete,
|
||||
so enrichment tags written to the DB (auto-tag, dream synthesize,
|
||||
signal-detector) survive a re-chunk. The reindex DB-only fallback also
|
||||
reconstructs the full markdown (frontmatter + body + timeline) before
|
||||
re-chunking, so a page with no on-disk source keeps its frontmatter,
|
||||
title, and timeline instead of getting overwritten with empty
|
||||
frontmatter. Trade-off: removing a tag from a page's frontmatter no
|
||||
longer removes it from the DB on the next sync (frontmatter-tag removal
|
||||
needs a provenance column, deferred). (Closes #1621.)
|
||||
|
||||
**`gbrain sync` wedges on a large brain (no progress, high CPU)?**
|
||||
v0.41.37.0 ships three things. First, name the stalling file:
|
||||
|
||||
```bash
|
||||
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
|
||||
```
|
||||
|
||||
The last `[sync] begin import: <path>` line with no following completion
|
||||
is the file being processed when the hang hit. Second, if you suspect a
|
||||
schema-pack `inference.regex` with catastrophic backtracking, complete
|
||||
the sync with the pack disabled and re-run extraction later:
|
||||
|
||||
```bash
|
||||
gbrain sync --no-schema-pack --no-pull --no-embed --yes
|
||||
```
|
||||
|
||||
`gbrain schema lint` now warns on the classic nested-quantifier ReDoS
|
||||
shapes (`(a+)+`, `(a*)*`, …) in pack regexes, and the runtime caps
|
||||
inference-regex input length (override via `GBRAIN_MAX_REGEX_INPUT_CHARS`).
|
||||
Third, on a PGLite brain, stop `gbrain serve` before a large sync —
|
||||
PGLite is single-writer and a live MCP server contends for the write
|
||||
lock. See [`docs/architecture/serve-sync-concurrency.md`](docs/architecture/serve-sync-concurrency.md)
|
||||
for the full triage. (Closes #1569.)
|
||||
|
||||
**`gbrain init --migrate-only` / a schema migration fails on Windows
|
||||
with `getaddrinfo ENOTFOUND`?** v0.41.37.0 runs the 9 schema-bring-up
|
||||
phases in-process instead of spawning a child `gbrain init
|
||||
--migrate-only` per phase. The spawned child died on
|
||||
Windows + bun + Supabase pooler with a DNS-resolution failure even
|
||||
though the parent connected fine; running in-process removes the spawn
|
||||
entirely. The v0.13.1 grandfather migration that hung 70+ minutes on an
|
||||
82K-page PGLite brain is also fixed — it now runs as a chunked bulk SQL
|
||||
pass (keyed on the page PK, soft-delete-filtered, source-safe) that
|
||||
completes in ~1-2 seconds. (Closes #1605, #1581.)
|
||||
|
||||
## Docs
|
||||
|
||||
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# TODOS
|
||||
|
||||
## v0.41.37.0 critical-fix-wave follow-ups (v0.42+)
|
||||
|
||||
Filed from the v0.41.37.0 wave (#1621 tag-wipe, #1581 grandfather hang,
|
||||
#1605 Windows migration spawn, #1569 sync ReDoS hardening). Each item was
|
||||
deliberately scoped out of the wave (see plan + GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-greedy-quiche.md`).
|
||||
|
||||
- [ ] **#1621-followup: tag_source provenance column for frontmatter-tag REMOVAL.** The wave shipped ADD-ONLY tag reconciliation (`src/core/import-file.ts`) — re-import never deletes tags, so DB-side enrichment tags survive. Trade-off: removing a tag from a page's frontmatter no longer removes it from the DB. To restore removal-on-edit without wiping enrichment tags, add a `tags.tag_source` column (migration, both engines), stamp `'frontmatter'` on import-path tags, and reconcile by deleting only `tag_source='frontmatter'` tags absent from the new frontmatter (enrichment/backfilled tags default NULL = preserved, so no enrichment-write-site enumeration needed). Priority: P3 (additive-metadata staleness is low-harm).
|
||||
|
||||
- [ ] **#1605-followup: convert migration backfill-phase spawns to in-process.** v0.41.37.0 made the 9 schema phases (`gbrain init --migrate-only`) run in-process via `runMigrateOnlyCore`, which unblocks `schema_version` advancement on Windows+bun+Supabase. The remaining non-schema spawns (`extract links/timeline`, `repair-jsonb`) still shell out via `runGbrainSubprocess` — they now surface child stderr (so a Windows failure is diagnosable) but still fail on Windows. Convert them to in-process calls (the extract/repair command functions are callable with an engine) so Windows brains complete data backfill, not just schema. Sites: `src/commands/migrations/v0_12_0.ts` (extract), `v0_12_2.ts` (repair), `v0_13_0.ts` (extract). Priority: P2.
|
||||
|
||||
- [ ] **#1569-followup: root-cause the 56K-file sync wedge with the reporter's repro.** v0.41.37.0 shipped ReDoS hardening (input-length cap + star-height lint + `--no-schema-pack` escape) + diagnostics (`GBRAIN_SYNC_TRACE=1` begin-heartbeat + PGLite serve/sync concurrency doc), but did NOT root-cause the deterministic wedge at ~3100 files — the reporter's redos-guard hypothesis didn't hold (it's not on the sync path). Get the reporter's sample files (`/tmp/gbrain-hang-sample.txt`, `/tmp/gbrain-prewedge-sample.txt`), reproduce, and pin the resume-mode deep-recursion pre-import phase (prime suspect: the walk/diff/checkpoint path). Priority: P1 once a repro exists; tracked on the #1569 thread.
|
||||
## MCP skillpack distribution — PR2 (v0.41.37+)
|
||||
|
||||
Filed from the v0.41.36.0 skill-catalog wave (`list_skills` / `get_skill`).
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# `gbrain serve` ↔ `gbrain sync` concurrency (PGLite)
|
||||
|
||||
**Short version: on a PGLite brain, stop `gbrain serve` before a large sync.**
|
||||
|
||||
## Why
|
||||
|
||||
PGLite is a single-writer embedded Postgres (WASM). A running `gbrain serve`
|
||||
(stdio or HTTP MCP) holds an open PGLite connection on the brain's data
|
||||
directory. `gbrain sync` needs to write to that same data directory. The two
|
||||
contend for PGLite's single-writer connection / write-lock — **this is NOT the
|
||||
`gbrain-sync` advisory lock** (that's a separate, DB-row coordination lock for
|
||||
two concurrent *syncs*). Confusing the two sends you debugging the wrong surface.
|
||||
|
||||
Symptoms of serve↔sync contention on PGLite:
|
||||
|
||||
- `gbrain sync` blocks acquiring the PGLite write lock, or makes very slow
|
||||
progress, while a `gbrain serve` process is alive on the same brain.
|
||||
- Killing stale `gbrain serve` MCP processes frees the lock and sync proceeds.
|
||||
|
||||
## What to do
|
||||
|
||||
1. Stop any `gbrain serve` process for this brain before a large sync:
|
||||
```bash
|
||||
pkill -f 'gbrain serve' # or stop your MCP client / Claude Desktop / Cursor
|
||||
gbrain sync --no-pull --no-embed --yes
|
||||
```
|
||||
2. Restart `gbrain serve` after the sync completes.
|
||||
|
||||
This contention does **not** apply to the Postgres engine — Postgres tolerates
|
||||
concurrent connections, so `serve` and `sync` can run simultaneously there.
|
||||
|
||||
## Diagnosing a sync hang
|
||||
|
||||
If a sync wedges (no progress, high CPU), re-run with the per-file begin trace
|
||||
so the stalling file is named:
|
||||
|
||||
```bash
|
||||
GBRAIN_SYNC_TRACE=1 gbrain sync --no-pull --no-embed --yes
|
||||
```
|
||||
|
||||
The last `[sync] begin import: <path>` line with no following completion is the
|
||||
file being processed when the hang occurred. Under `--workers >1` / `--all`,
|
||||
the stuck file is in the set of begin-lines without a matching completion.
|
||||
|
||||
If you suspect a schema-pack regex is the cause (a pack with a
|
||||
catastrophic-backtracking `inference.regex`), complete the sync with the pack
|
||||
disabled and re-run extraction afterward:
|
||||
|
||||
```bash
|
||||
gbrain sync --no-schema-pack --no-pull --no-embed --yes
|
||||
```
|
||||
|
||||
`gbrain schema lint` flags the classic nested-quantifier ReDoS shapes
|
||||
(`(a+)+`, `(a*)*`, …) in pack regexes as warnings.
|
||||
+53
-5
File diff suppressed because one or more lines are too long
+1
-1
@@ -141,5 +141,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.41.36.0"
|
||||
"version": "0.41.37.0"
|
||||
}
|
||||
|
||||
@@ -255,9 +255,10 @@ export const INLINE_TIPS = [
|
||||
"`gbrain upgrade` runs post-upgrade + apply-migrations.",
|
||||
];
|
||||
|
||||
// Target ~700KB so llms-full.txt fits in ~175k-token contexts with room to spare.
|
||||
// Bumped from 600KB in v0.41.9.0 — CLAUDE.md grew past 600KB after the wave's
|
||||
// new-file annotations + Conductor branch-name iron-rule landed; the bundle
|
||||
// still fits comfortably in modern long-context models.
|
||||
// Target so llms-full.txt fits in ~190k-token contexts with room to spare.
|
||||
// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB in v0.41.37.0 — CLAUDE.md
|
||||
// crossed 700KB once the critical-fix-wave key-files annotations merged on top
|
||||
// of master's v0.41.34/35/36 additions (703KB). Still fits comfortably in
|
||||
// modern long-context models.
|
||||
// Generator prints a WARN if exceeded; ship with includeInFull=false exclusions.
|
||||
export const FULL_SIZE_BUDGET = 700_000;
|
||||
export const FULL_SIZE_BUDGET = 750_000;
|
||||
|
||||
+15
-32
@@ -515,44 +515,27 @@ async function resolveChatByEnv(out: ResolvedAIOptions): Promise<void> {
|
||||
* clobbering the user's chosen engine.
|
||||
*/
|
||||
async function initMigrateOnly(opts: { jsonOutput: boolean }) {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
const msg = 'No brain configured. Run `gbrain init` (interactive) or `gbrain init --pglite` / `gbrain init --supabase` first.';
|
||||
// v0.41.37.0 #1605: delegate to the shared runMigrateOnlyCore so the CLI path
|
||||
// and the in-process migration-orchestrator path can't drift (single source
|
||||
// of truth for configureGateway-before-initSchema + the schema bring-up).
|
||||
const { runMigrateOnlyCore, MigrateOnlyError } = await import('./migrations/in-process.ts');
|
||||
try {
|
||||
const result = await runMigrateOnlyCore();
|
||||
if (opts.jsonOutput) {
|
||||
console.log(JSON.stringify({ status: 'error', reason: 'no_config', message: msg }));
|
||||
console.log(JSON.stringify({ status: 'success', engine: result.engine, mode: 'migrate-only' }));
|
||||
} else {
|
||||
console.log(`Schema up to date (engine: ${result.engine}).`);
|
||||
}
|
||||
} catch (e) {
|
||||
const isNoConfig = e instanceof MigrateOnlyError && e.message.startsWith('No brain configured');
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (opts.jsonOutput) {
|
||||
console.log(JSON.stringify({ status: 'error', reason: isNoConfig ? 'no_config' : 'migrate_failed', message: msg }));
|
||||
} else {
|
||||
console.error(msg);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// B.3: configureGateway BEFORE initSchema even on the migrate-only path,
|
||||
// so a schema bump on a brain whose file config is missing the embedding
|
||||
// fields doesn't fall through to stale hardcoded fallbacks. Reads
|
||||
// existing config (which loadConfig already merged with env) and
|
||||
// propagates it into the gateway.
|
||||
const { configureGateway: configureGw } = await import('../core/ai/gateway.ts');
|
||||
configureGw({
|
||||
embedding_model: config.embedding_model,
|
||||
embedding_dimensions: config.embedding_dimensions,
|
||||
expansion_model: config.expansion_model,
|
||||
chat_model: config.chat_model,
|
||||
env: { ...process.env },
|
||||
});
|
||||
|
||||
const engine = await createEngine(toEngineConfig(config));
|
||||
try {
|
||||
await engine.connect(toEngineConfig(config));
|
||||
await engine.initSchema();
|
||||
} finally {
|
||||
try { await engine.disconnect(); } catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
console.log(JSON.stringify({ status: 'success', engine: config.engine, mode: 'migrate-only' }));
|
||||
} else {
|
||||
console.log(`Schema up to date (engine: ${config.engine}).`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
/**
|
||||
* In-process migration helpers (v0.41.37.0 #1605).
|
||||
*
|
||||
* Why this exists: migration schema phases used to shell out to a child
|
||||
* `gbrain init --migrate-only` via `execSync`. On Windows + bun + Supabase
|
||||
* pooler, the spawned CHILD process dies with `getaddrinfo ENOTFOUND` before it
|
||||
* can connect — even though the PARENT connects fine and `env: process.env` is
|
||||
* passed. It is a bun-on-Windows child-process DNS-resolution failure, not an
|
||||
* env-propagation bug. The only robust fix is to not spawn at all: run the
|
||||
* schema bring-up IN-PROCESS. The PGLite path at v0_11_0.ts already proved the
|
||||
* pattern; this generalizes it to every engine + every schema phase.
|
||||
*
|
||||
* `runMigrateOnlyCore` is the single source of truth for "bring schema to head"
|
||||
* — `init.ts:initMigrateOnly` (the `gbrain init --migrate-only` CLI path) and
|
||||
* the migration orchestrators both call it, so the configureGateway-before-
|
||||
* initSchema fix can't drift between them.
|
||||
*
|
||||
* `runGbrainSubprocess` is the diagnostic wrapper for the REMAINING (non-schema)
|
||||
* gbrain-subprocess spawns (extract/repair/stats). It captures child stderr and
|
||||
* folds it into the thrown error so a Windows failure shows the real
|
||||
* `getaddrinfo ENOTFOUND` line instead of the bare `Command failed: ...`.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
|
||||
import { loadConfig, toEngineConfig } from '../../core/config.ts';
|
||||
import { createEngine } from '../../core/engine-factory.ts';
|
||||
|
||||
/** Default wall-clock guard for in-process initSchema. Matches the 600s cap
|
||||
* the old `execSync('gbrain init --migrate-only', { timeout: 600_000 })` used,
|
||||
* so a hung schema bring-up surfaces as a phase failure instead of wedging
|
||||
* the whole cascade. */
|
||||
export const MIGRATE_ONLY_TIMEOUT_MS = 600_000;
|
||||
|
||||
/** Large stderr buffer for captured subprocess output. `execSync`'s default
|
||||
* ~1MB maxBuffer overflows on long backfills (extract/repair) and turns a
|
||||
* successful run into a spurious failure. */
|
||||
const SUBPROCESS_MAX_BUFFER = 64 * 1024 * 1024;
|
||||
|
||||
export interface MigrateOnlyResult {
|
||||
/** The engine kind that was brought to head ('pglite' | 'postgres'). */
|
||||
engine: string;
|
||||
}
|
||||
|
||||
export class MigrateOnlyError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'MigrateOnlyError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Bring the configured brain's schema to head, in-process. Mirrors what
|
||||
* `gbrain init --migrate-only` did via subprocess: configureGateway →
|
||||
* createEngine → connect → initSchema → disconnect. Idempotent (initSchema is
|
||||
* a no-op when already at head). Throws `MigrateOnlyError` on no-config or
|
||||
* timeout so callers report a failed phase rather than hanging.
|
||||
*/
|
||||
export async function runMigrateOnlyCore(opts?: { timeoutMs?: number }): Promise<MigrateOnlyResult> {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new MigrateOnlyError(
|
||||
'No brain configured. Run `gbrain init` (interactive) or `gbrain init --pglite` / `gbrain init --supabase` first.',
|
||||
);
|
||||
}
|
||||
|
||||
// configureGateway BEFORE initSchema (init.ts B.3): a schema bump on a brain
|
||||
// whose file config is missing embedding fields must not fall through to
|
||||
// stale hardcoded fallbacks. loadConfig already merged env; propagate it.
|
||||
const { configureGateway } = await import('../../core/ai/gateway.ts');
|
||||
configureGateway({
|
||||
embedding_model: config.embedding_model,
|
||||
embedding_dimensions: config.embedding_dimensions,
|
||||
expansion_model: config.expansion_model,
|
||||
chat_model: config.chat_model,
|
||||
env: { ...process.env },
|
||||
});
|
||||
|
||||
const timeoutMs = opts?.timeoutMs ?? MIGRATE_ONLY_TIMEOUT_MS;
|
||||
const engine = await createEngine(toEngineConfig(config));
|
||||
try {
|
||||
await engine.connect(toEngineConfig(config));
|
||||
await withTimeout(
|
||||
engine.initSchema(),
|
||||
timeoutMs,
|
||||
`schema init timed out after ${Math.round(timeoutMs / 1000)}s`,
|
||||
);
|
||||
} finally {
|
||||
try { await engine.disconnect(); } catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
return { engine: config.engine };
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a `gbrain ...` subcommand as a subprocess, capturing child stderr so a
|
||||
* failure surfaces the real reason. Used for the non-schema backfill phases
|
||||
* (extract/repair/stats) that aren't yet in-process. On Windows these may still
|
||||
* fail with `getaddrinfo ENOTFOUND`, but the operator now sees WHY instead of a
|
||||
* bare `Command failed`. Returns captured stdout (utf-8) on success.
|
||||
*
|
||||
* Note: stderr is piped (captured), so gbrain progress lines (which go to
|
||||
* stderr) are not shown live during these phases — acceptable for a one-shot
|
||||
* `apply-migrations` run; the failure reason matters more than live progress.
|
||||
*/
|
||||
export function runGbrainSubprocess(cmd: string, opts?: { timeoutMs?: number }): string {
|
||||
try {
|
||||
const out = execSync(cmd, {
|
||||
stdio: ['inherit', 'pipe', 'pipe'],
|
||||
timeout: opts?.timeoutMs ?? MIGRATE_ONLY_TIMEOUT_MS,
|
||||
env: process.env,
|
||||
maxBuffer: SUBPROCESS_MAX_BUFFER,
|
||||
encoding: 'utf-8',
|
||||
});
|
||||
return typeof out === 'string' ? out : '';
|
||||
} catch (e: unknown) {
|
||||
const err = e as { message?: string; stderr?: Buffer | string };
|
||||
const stderrRaw = err?.stderr
|
||||
? (Buffer.isBuffer(err.stderr) ? err.stderr.toString('utf-8') : String(err.stderr))
|
||||
: '';
|
||||
const tail = stderrRaw.split('\n').filter(Boolean).slice(-10).join('\n');
|
||||
const base = err?.message ?? String(e);
|
||||
throw new Error(tail ? `${base}\n--- child stderr (tail) ---\n${tail}` : base);
|
||||
}
|
||||
}
|
||||
|
||||
/** Reject `p` if it doesn't settle within `ms`. The original promise keeps
|
||||
* running (best-effort) but the caller sees a clear timeout error. */
|
||||
async function withTimeout<T>(p: Promise<T>, ms: number, message: string): Promise<T> {
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const timeout = new Promise<never>((_, reject) => {
|
||||
timer = setTimeout(() => reject(new MigrateOnlyError(message)), ms);
|
||||
});
|
||||
try {
|
||||
return await Promise.race([p, timeout]);
|
||||
} finally {
|
||||
if (timer) clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
@@ -23,7 +23,6 @@
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, lstatSync, statSync, realpathSync } from 'fs';
|
||||
import { join, resolve, dirname } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { savePreferences, loadPreferences } from '../../core/preferences.ts';
|
||||
// Bug 3 — appendCompletedMigration moved to the runner (apply-migrations.ts).
|
||||
@@ -61,28 +60,14 @@ export interface PendingHostWorkEntry {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// v0.36.x #1100: route PGLite through an in-process schema apply rather
|
||||
// than `execSync('gbrain init --migrate-only')`. The subprocess inherits
|
||||
// HOME and tries to acquire the same file lock the parent process is
|
||||
// holding (or briefly released and the on-disk artifact has not finished
|
||||
// settling), which deadlocks until the 30s lock timeout fires. The
|
||||
// structural fix is to not spawn a subprocess for work the parent can
|
||||
// do directly — Postgres tolerates concurrent connections, so the
|
||||
// legacy execSync path stays for Postgres callers.
|
||||
const { loadConfig, toEngineConfig } = await import('../../core/config.ts');
|
||||
const cfg = loadConfig();
|
||||
if (cfg?.engine === 'pglite') {
|
||||
const { createEngine } = await import('../../core/engine-factory.ts');
|
||||
const eng = await createEngine(toEngineConfig(cfg));
|
||||
try {
|
||||
await eng.connect(toEngineConfig(cfg));
|
||||
await eng.initSchema();
|
||||
} finally {
|
||||
try { await eng.disconnect(); } catch { /* best-effort */ }
|
||||
}
|
||||
return { name: 'schema', status: 'complete' };
|
||||
}
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
// v0.41.37.0 #1605: bring schema to head IN-PROCESS for every engine. Was an
|
||||
// a `gbrain init --migrate-only` subprocess subprocess for Postgres (died with
|
||||
// `getaddrinfo ENOTFOUND` on Windows+bun+Supabase-pooler before it could
|
||||
// connect) plus a PGLite-only in-process branch (which separately
|
||||
// deadlocked on the file lock, #1100). runMigrateOnlyCore is the single
|
||||
// in-process path for both engines.
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
|
||||
@@ -31,19 +31,21 @@
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import { runGbrainSubprocess } from './in-process.ts';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// 10-minute budget. Migrations v8/v9 dedup with helper-index should be sub-second
|
||||
// even on 80K-duplicate brains, but the outer wall-clock cap shouldn't be the
|
||||
// failure mode (the prior 60s ceiling tripped Garry's production upgrade).
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -93,7 +95,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
// --source db is idempotent: the UNIQUE constraint on
|
||||
// (from_page_id, to_page_id, link_type) and ON CONFLICT DO NOTHING
|
||||
// make re-runs cheap. Empty brains return 0/0 quickly.
|
||||
execSync('gbrain extract links --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
runGbrainSubprocess('gbrain extract links --source db' + childGlobalFlags(), { timeoutMs: 600_000 });
|
||||
return { name: 'backfill_links', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -104,7 +106,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
function phaseDBackfillTimeline(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'backfill_timeline', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain extract timeline --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
runGbrainSubprocess('gbrain extract timeline --source db' + childGlobalFlags(), { timeoutMs: 600_000 });
|
||||
return { name: 'backfill_timeline', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -188,7 +190,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
// A. Schema
|
||||
const a = phaseASchema(opts);
|
||||
const a = await phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') {
|
||||
return finalizeResult(phases, 'failed');
|
||||
|
||||
@@ -21,18 +21,20 @@
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import { runGbrainSubprocess } from './in-process.ts';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// Propagate global progress flags so the child shows the same mode the
|
||||
// parent orchestrator is running in.
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -46,7 +48,7 @@ function phaseBRepair(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'jsonb_repair', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// stdio: 'inherit' — child's stderr progress streams straight through.
|
||||
execSync('gbrain repair-jsonb' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
runGbrainSubprocess('gbrain repair-jsonb' + childGlobalFlags(), { timeoutMs: 600_000 });
|
||||
return { name: 'jsonb_repair', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -94,7 +96,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
const a = await phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import { runGbrainSubprocess } from './in-process.ts';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts). The
|
||||
// orchestrator returns its result and the runner persists it.
|
||||
@@ -44,10 +45,11 @@ import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhase
|
||||
// upgrade mid-migration. The shim is already the canonical wrapper; trust
|
||||
// it. Regression guarded by test/migrations-v0_13_0.test.ts.
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -64,11 +66,7 @@ function phaseBBackfill(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
// `--include-frontmatter` is the v0.13 flag that enables the canonical
|
||||
// frontmatter link extractor. Default-OFF in the CLI for back-compat;
|
||||
// the migration explicitly opts in because this is the canonical backfill.
|
||||
execSync('gbrain extract links --source db --include-frontmatter', {
|
||||
stdio: 'inherit',
|
||||
timeout: 1_800_000, // 30 min hard cap; typical 2-5 min on 46K pages
|
||||
env: process.env,
|
||||
});
|
||||
runGbrainSubprocess('gbrain extract links --source db --include-frontmatter', { timeoutMs: 1_800_000 });
|
||||
return { name: 'frontmatter_backfill', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -116,7 +114,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
const a = await phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
|
||||
@@ -20,13 +20,14 @@
|
||||
* frontmatter snapshot. Roll back by re-applying those snapshots via
|
||||
* `gbrain apply-migrations --rollback v0.13.0` (future CLI; not in scope).
|
||||
*
|
||||
* Scale: on a 30K-page brain, ~15s on Postgres, ~30s on PGLite. Batched in
|
||||
* chunks of 100 with a commit per batch so interruption losses are bounded.
|
||||
* Scale (v0.41.37.0 #1581): chunked bulk SQL — one id-snapshot SELECT + N
|
||||
* (SELECT-for-rollback + UPDATE) statements of CHUNK_SIZE rows each. Completes
|
||||
* in ~1-2s even on an 82K-page PGLite brain. The prior per-page
|
||||
* getPage+putPage loop hung CPU-bound for 70+ min on that brain (#1581).
|
||||
*
|
||||
* Snapshot-slugs rule: reads engine.getAllSlugs() upfront into an in-memory
|
||||
* Set before iterating. Prior learning [listpages-pagination-mutation]: any
|
||||
* batch write that mutates updated_at during OFFSET pagination is unstable.
|
||||
* getAllSlugs returns a full snapshot that isn't invalidated by our writes.
|
||||
* Snapshot rule: the affected id set is read once via SQL up front. It isn't
|
||||
* invalidated by our writes because each UPDATE flips its rows out of the
|
||||
* GRANDFATHER_WHERE predicate (idempotent + resumable).
|
||||
*
|
||||
* Safety: does NOT call saveConfig. Prior learning [gbrain-init-default-pglite-flip]:
|
||||
* bare `gbrain init` defaults to PGLite and overwrites Postgres config.
|
||||
@@ -46,7 +47,6 @@ import type { BrainEngine } from '../../core/engine.ts';
|
||||
// Lazy: GBRAIN_HOME may be set after module load.
|
||||
const getRollbackDir = () => gbrainPath('migrations');
|
||||
const getRollbackFile = () => join(getRollbackDir(), 'v0_13_1-rollback.jsonl');
|
||||
const BATCH_SIZE = 100;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase A — connect (no config write)
|
||||
@@ -75,30 +75,35 @@ async function phaseAConnect(opts: OrchestratorOpts): Promise<{ result: Orchestr
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase B — snapshot slugs upfront
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function phaseBSnapshot(engine: BrainEngine): Promise<{ result: OrchestratorPhaseResult; slugs: string[] }> {
|
||||
try {
|
||||
const slugSet = await engine.getAllSlugs();
|
||||
const slugs = [...slugSet].sort();
|
||||
return {
|
||||
result: { name: 'snapshot', status: 'complete', detail: `${slugs.length} slugs` },
|
||||
slugs,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
result: { name: 'snapshot', status: 'failed', detail: e instanceof Error ? e.message : String(e) },
|
||||
slugs: [],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase C — grandfather: add validate:false where absent
|
||||
//
|
||||
// v0.41.37.0 #1581: rewritten from a per-page getPage+putPage loop (which hung
|
||||
// CPU-bound for 70+ min on an 82K-page PGLite brain) to a CHUNKED bulk SQL
|
||||
// pass. Three correctness properties (the per-page loop and a naive single
|
||||
// bulk UPDATE both got these wrong):
|
||||
// 1. Keyed on pages.id (globally unique PK), NOT slug. `slug` uniqueness is
|
||||
// (source_id, slug) — a slug-batched UPDATE would mutate same-slug pages
|
||||
// across other sources and produce ambiguous rollback rows.
|
||||
// 2. Filters `deleted_at IS NULL` — the old getPage path hid soft-deleted
|
||||
// rows; a raw UPDATE must not grandfather tombstones.
|
||||
// 3. Chunked in CHUNK_SIZE batches so lock-hold stays bounded (the
|
||||
// DELETE_BATCH_SIZE convention) instead of one giant transaction.
|
||||
// The rollback log carries source identity ({id, slug, source_id,
|
||||
// pre_frontmatter}) so a rollback is unambiguous across sources.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Filter for pages still needing the grandfather flag. Literal `?` jsonb
|
||||
// existence operator (matches src/core/embed-skip.ts convention); 'validate'
|
||||
// is a hardcoded literal, no injection surface. COALESCE for null-frontmatter
|
||||
// safety. deleted_at IS NULL skips soft-deleted tombstones.
|
||||
const GRANDFATHER_WHERE =
|
||||
"NOT (COALESCE(frontmatter, '{}'::jsonb) ? 'validate') AND deleted_at IS NULL";
|
||||
|
||||
// Per-chunk row count. Bounded lock-hold + write-amplification per statement
|
||||
// (same rationale as engine-constants.ts DELETE_BATCH_SIZE).
|
||||
const CHUNK_SIZE = 1000;
|
||||
|
||||
interface GrandfatherResult {
|
||||
touched: number;
|
||||
skipped: number;
|
||||
@@ -106,60 +111,69 @@ interface GrandfatherResult {
|
||||
failures: string[];
|
||||
}
|
||||
|
||||
async function phaseCGrandfather(
|
||||
// Exported for direct hermetic testing against a PGLite engine (the config /
|
||||
// loadConfig flow is exercised separately). Internal helper otherwise.
|
||||
export async function phaseCGrandfather(
|
||||
engine: BrainEngine,
|
||||
slugs: string[],
|
||||
opts: OrchestratorOpts,
|
||||
): Promise<{ result: OrchestratorPhaseResult; detail: GrandfatherResult }> {
|
||||
ensureRollbackDir();
|
||||
const gf: GrandfatherResult = { touched: 0, skipped: 0, failed: 0, failures: [] };
|
||||
|
||||
for (let i = 0; i < slugs.length; i += BATCH_SIZE) {
|
||||
const batch = slugs.slice(i, i + BATCH_SIZE);
|
||||
for (const slug of batch) {
|
||||
try {
|
||||
if (opts.dryRun) {
|
||||
const rows = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM pages WHERE ${GRANDFATHER_WHERE}`,
|
||||
);
|
||||
const c = rows[0]?.count ?? 0;
|
||||
gf.touched = typeof c === 'string' ? parseInt(c, 10) : Number(c);
|
||||
return {
|
||||
result: { name: 'grandfather', status: 'complete', detail: `would touch ${gf.touched} (dry-run)` },
|
||||
detail: gf,
|
||||
};
|
||||
}
|
||||
|
||||
ensureRollbackDir();
|
||||
|
||||
// Snapshot the affected id set up front (ids only — cheap, no frontmatter
|
||||
// blobs in memory). The snapshot isn't invalidated by our writes because
|
||||
// each UPDATE flips the rows out of the GRANDFATHER_WHERE predicate.
|
||||
const idRows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE ${GRANDFATHER_WHERE} ORDER BY id`,
|
||||
);
|
||||
const ids = idRows.map(r => Number(r.id));
|
||||
|
||||
for (let i = 0; i < ids.length; i += CHUNK_SIZE) {
|
||||
const chunk = ids.slice(i, i + CHUNK_SIZE);
|
||||
try {
|
||||
const page = await engine.getPage(slug);
|
||||
if (!page) { gf.skipped++; continue; }
|
||||
// Rollback log BEFORE mutation: one SELECT per chunk (bounded memory),
|
||||
// one appendFileSync per chunk. Carries source_id so rollback is
|
||||
// unambiguous across same-slug-different-source pages.
|
||||
const snap = await engine.executeRaw<{
|
||||
id: number; slug: string; source_id: string | null; frontmatter: Record<string, unknown> | null;
|
||||
}>(
|
||||
'SELECT id, slug, source_id, frontmatter FROM pages WHERE id = ANY($1::int[])',
|
||||
[chunk],
|
||||
);
|
||||
appendRollbackBatch(snap);
|
||||
|
||||
// Idempotency: skip if frontmatter already has a `validate` key
|
||||
// (whether true, false, or any other value). We don't flip existing
|
||||
// explicit settings.
|
||||
if (page.frontmatter && Object.prototype.hasOwnProperty.call(page.frontmatter, 'validate')) {
|
||||
gf.skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (opts.dryRun) {
|
||||
gf.touched++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Rollback log BEFORE mutation, so a crash mid-write still lets us
|
||||
// revert. Append-only, one line per page, newline-terminated.
|
||||
appendRollbackEntry({
|
||||
slug,
|
||||
pre_frontmatter: page.frontmatter ?? {},
|
||||
});
|
||||
|
||||
const nextFrontmatter = { ...(page.frontmatter ?? {}), validate: false };
|
||||
await engine.putPage(slug, {
|
||||
type: page.type,
|
||||
title: page.title,
|
||||
compiled_truth: page.compiled_truth,
|
||||
timeline: page.timeline,
|
||||
frontmatter: nextFrontmatter,
|
||||
});
|
||||
gf.touched++;
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET frontmatter = jsonb_set(COALESCE(frontmatter, '{}'::jsonb), '{validate}', 'false'::jsonb) ` +
|
||||
'WHERE id = ANY($1::int[])',
|
||||
[chunk],
|
||||
);
|
||||
gf.touched += chunk.length;
|
||||
} catch (e) {
|
||||
gf.failed++;
|
||||
gf.failed += chunk.length;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
gf.failures.push(`${slug}: ${msg.slice(0, 100)}`);
|
||||
gf.failures.push(`chunk@${i}: ${msg.slice(0, 100)}`);
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
gf.failed += 1;
|
||||
gf.failures.push(`grandfather: ${e instanceof Error ? e.message : String(e)}`.slice(0, 120));
|
||||
}
|
||||
|
||||
const status: OrchestratorPhaseResult['status'] =
|
||||
gf.failed > 0 ? 'failed' : 'complete';
|
||||
const status: OrchestratorPhaseResult['status'] = gf.failed > 0 ? 'failed' : 'complete';
|
||||
const detailStr = `touched=${gf.touched} skipped=${gf.skipped} failed=${gf.failed}`;
|
||||
return {
|
||||
result: { name: 'grandfather', status, detail: detailStr },
|
||||
@@ -215,13 +229,10 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
}
|
||||
|
||||
try {
|
||||
const { result: snapRes, slugs } = await phaseBSnapshot(engine);
|
||||
phases.push(snapRes);
|
||||
if (snapRes.status !== 'complete') {
|
||||
return { version: '0.13.1', status: 'failed', phases };
|
||||
}
|
||||
|
||||
const { result: gfRes, detail: gfDetail } = await phaseCGrandfather(engine, slugs, opts);
|
||||
// v0.41.37.0 #1581: phaseBSnapshot (getAllSlugs) is gone — the chunked bulk
|
||||
// pass filters via SQL (GRANDFATHER_WHERE), so we no longer materialize a
|
||||
// full slug list in JS.
|
||||
const { result: gfRes, detail: gfDetail } = await phaseCGrandfather(engine, opts);
|
||||
phases.push(gfRes);
|
||||
filesRewritten = gfDetail.touched;
|
||||
|
||||
@@ -255,13 +266,23 @@ function ensureRollbackDir(): void {
|
||||
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
|
||||
}
|
||||
|
||||
function appendRollbackEntry(entry: { slug: string; pre_frontmatter: Record<string, unknown> }): void {
|
||||
const line = JSON.stringify({
|
||||
// v0.41.37.0 #1581: batch rollback writer. One appendFileSync per chunk, bounded
|
||||
// memory. Each line carries id + slug + source_id so a rollback is unambiguous
|
||||
// across same-slug-different-source pages (pages.slug is not globally unique).
|
||||
function appendRollbackBatch(
|
||||
rows: ReadonlyArray<{ id: number; slug: string; source_id: string | null; frontmatter: Record<string, unknown> | null }>,
|
||||
): void {
|
||||
if (rows.length === 0) return;
|
||||
const ts = new Date().toISOString();
|
||||
const lines = rows.map(r => JSON.stringify({
|
||||
migration: 'v0.13.0',
|
||||
timestamp: new Date().toISOString(),
|
||||
...entry,
|
||||
}) + '\n';
|
||||
appendFileSync(getRollbackFile(), line, 'utf-8');
|
||||
timestamp: ts,
|
||||
id: r.id,
|
||||
slug: r.slug,
|
||||
source_id: r.source_id ?? 'default',
|
||||
pre_frontmatter: r.frontmatter ?? {},
|
||||
})).join('\n') + '\n';
|
||||
appendFileSync(getRollbackFile(), lines, 'utf-8');
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -18,7 +18,6 @@
|
||||
* C. Record — append completed.jsonl.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
import { loadConfig, toEngineConfig } from '../../core/config.ts';
|
||||
@@ -28,10 +27,11 @@ const REQUIRED_TABLES = ['subagent_messages', 'subagent_tool_executions', 'subag
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -88,7 +88,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
const a = await phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalize(phases, 'failed');
|
||||
|
||||
|
||||
@@ -19,7 +19,6 @@
|
||||
* Idempotent: safe to re-run on partial state.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
import { loadConfig, toEngineConfig } from '../../core/config.ts';
|
||||
@@ -27,10 +26,11 @@ import { createEngine } from '../../core/engine-factory.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -174,7 +174,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
const a = await phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalize(phases, 'failed');
|
||||
|
||||
|
||||
@@ -16,15 +16,15 @@
|
||||
* fires on upgrade, because doctor + connectEngine never call initSchema().
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -36,7 +36,7 @@ function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
phases.push(phaseASchema(opts));
|
||||
phases.push(await phaseASchema(opts));
|
||||
|
||||
const anyFailed = phases.some(p => p.status === 'failed');
|
||||
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
|
||||
|
||||
@@ -25,20 +25,15 @@
|
||||
* All phases are idempotent and safe to re-run.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
|
||||
stdio: 'inherit',
|
||||
timeout: 600_000,
|
||||
env: process.env,
|
||||
});
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -102,7 +97,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
const a = await phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
|
||||
@@ -18,21 +18,16 @@
|
||||
* D. Record — handled by the runner.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { BrainEngine } from '../../core/engine.ts';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
async function phaseASchema(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
|
||||
stdio: 'inherit',
|
||||
timeout: 600_000, // 10 min — duplicate-heavy installs can be slow
|
||||
env: process.env,
|
||||
});
|
||||
const { runMigrateOnlyCore } = await import('./in-process.ts');
|
||||
await runMigrateOnlyCore();
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
return { name: 'schema', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
|
||||
@@ -129,7 +124,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
const a = await phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalize(phases, 'failed');
|
||||
|
||||
|
||||
+19
-2
@@ -24,6 +24,7 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { MARKDOWN_CHUNKER_VERSION } from '../core/chunkers/recursive.ts';
|
||||
import { importFromContent, importFromFile } from '../core/import-file.ts';
|
||||
import { serializeMarkdown } from '../core/markdown.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { existsSync } from 'fs';
|
||||
@@ -220,8 +221,24 @@ export async function runReindex(engine: BrainEngine, args: string[]): Promise<R
|
||||
return;
|
||||
}
|
||||
}
|
||||
// Fallback path: re-chunk stored compiled_truth in place.
|
||||
await importFromContent(engine, row.slug, row.compiled_truth, {
|
||||
// No source file on disk (DB-only page, or repo not available) —
|
||||
// re-chunk from the stored page. v0.41.37.0 #1621: reconstruct the
|
||||
// FULL markdown (frontmatter + body + timeline) via serializeMarkdown
|
||||
// and re-import THAT, instead of passing body-only `compiled_truth`
|
||||
// to importFromContent. The body-only path re-parsed with empty
|
||||
// frontmatter and OVERWROTE the page's real frontmatter / title /
|
||||
// timeline (codex catch). The round-trip preserves everything while
|
||||
// still re-chunking + bumping chunker_version.
|
||||
const page = await engine.getPage(row.slug, { sourceId: row.source_id });
|
||||
if (!page) { skipped++; return; }
|
||||
const tags = await engine.getTags(row.slug, { sourceId: row.source_id });
|
||||
const fullMarkdown = serializeMarkdown(
|
||||
page.frontmatter ?? {},
|
||||
page.compiled_truth ?? row.compiled_truth,
|
||||
page.timeline ?? '',
|
||||
{ type: page.type, title: page.title, tags },
|
||||
);
|
||||
await importFromContent(engine, row.slug, fullMarkdown, {
|
||||
sourceId: row.source_id,
|
||||
noEmbed: !!opts.noEmbed,
|
||||
forceRechunk: true,
|
||||
|
||||
+35
-2
@@ -226,6 +226,15 @@ export interface SyncOpts {
|
||||
skipFailed?: boolean;
|
||||
/** Bug 9 — re-attempt unacknowledged failures explicitly (CLI --retry-failed). */
|
||||
retryFailed?: boolean;
|
||||
/**
|
||||
* v0.41.37.0 #1569 — skip loading the active schema pack during sync. When set,
|
||||
* `loadActivePack` is not called, so no user-supplied pack page-type regex
|
||||
* (markdown.ts subtype path_pattern) runs during import. Pages fall back to
|
||||
* legacy prefix typing. Escape hatch for completing a sync when a suspect
|
||||
* pack regex is the suspected cause of a wedge; re-run extraction later.
|
||||
* Threaded through performSync AND syncOneSource so `sync --all` honors it.
|
||||
*/
|
||||
noSchemaPack?: boolean;
|
||||
/**
|
||||
* v0.18.0 Step 5 — sync a specific named source. When set, sync reads
|
||||
* local_path + last_commit from the sources table (not the global
|
||||
@@ -961,6 +970,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// failure falls through to legacy inferType (parity preserved).
|
||||
let syncActivePack: { page_types: ReadonlyArray<{ name: string; path_prefixes: ReadonlyArray<string> }> } | undefined;
|
||||
try {
|
||||
// v0.41.37.0 #1569: --no-schema-pack escape hatch. Skip pack load entirely so
|
||||
// no user-supplied pack regex (markdown.ts subtype path_pattern) runs during
|
||||
// sync; pages fall back to legacy prefix typing.
|
||||
if (opts.noSchemaPack) {
|
||||
serr('[sync] --no-schema-pack: skipping schema pack; pages use legacy prefix typing');
|
||||
throw new Error('schema-pack-skipped');
|
||||
}
|
||||
const { loadActivePack } = await import('../core/schema-pack/load-active.ts');
|
||||
const { loadConfig } = await import('../core/config.ts');
|
||||
const resolved = await loadActivePack({
|
||||
@@ -1589,6 +1605,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
// v0.41.37.0 #1569: per-file BEGIN heartbeat, emitted BEFORE importFile so a
|
||||
// hang names the stalling file (the progress.tick below only fires AFTER
|
||||
// importFile returns — useless when one file wedges). Off by default
|
||||
// (GBRAIN_SYNC_TRACE=1) to avoid a line per file on huge brains. serr is
|
||||
// source-prefix-aware, so under --workers>1 / --all the stuck file is the
|
||||
// begin-line with no matching completion in the in-flight set.
|
||||
if (process.env.GBRAIN_SYNC_TRACE) serr(`[sync] begin import: ${path}`);
|
||||
try {
|
||||
// v0.18.0+ multi-source: thread `opts.sourceId` so per-page tx writes
|
||||
// (putPage / getTags / addTag / removeTag / deleteChunks / upsertChunks
|
||||
@@ -2078,6 +2101,12 @@ Options:
|
||||
--watch Re-sync continuously on an interval.
|
||||
--interval N Watch-mode interval in seconds (default 60).
|
||||
--no-pull Skip 'git pull' before the sync (useful for tests).
|
||||
--no-schema-pack Skip loading the active schema pack (no per-file pack
|
||||
regex runs; pages use legacy prefix typing). Escape
|
||||
hatch if a suspect pack regex is wedging sync.
|
||||
PGLite is single-writer: stop 'gbrain serve' before a
|
||||
large sync (see docs/architecture/serve-sync-concurrency.md).
|
||||
GBRAIN_SYNC_TRACE=1 names the file being imported (hang triage).
|
||||
--all Sync every registered source instead of just the
|
||||
default (multi-source brains).
|
||||
--parallel N (with --all) Run up to N sources concurrently.
|
||||
@@ -2113,6 +2142,7 @@ See also:
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
const skipFailed = args.includes('--skip-failed');
|
||||
const retryFailed = args.includes('--retry-failed');
|
||||
const noSchemaPack = args.includes('--no-schema-pack'); // v0.41.37.0 #1569
|
||||
const syncAll = args.includes('--all');
|
||||
const jsonOut = args.includes('--json');
|
||||
const yesFlag = args.includes('--yes');
|
||||
@@ -2540,7 +2570,7 @@ See also:
|
||||
repoPath: src.local_path!,
|
||||
dryRun, full, noPull,
|
||||
noEmbed: effectiveNoEmbed,
|
||||
skipFailed, retryFailed,
|
||||
skipFailed, retryFailed, noSchemaPack,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
concurrency,
|
||||
@@ -2741,7 +2771,7 @@ See also:
|
||||
: undefined;
|
||||
singleSourceTimer?.unref?.();
|
||||
const opts: SyncOpts = {
|
||||
repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId,
|
||||
repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, noSchemaPack, sourceId,
|
||||
strategy: strategyArg, concurrency,
|
||||
signal: singleSourceController?.signal,
|
||||
};
|
||||
@@ -2901,6 +2931,8 @@ export async function syncOneSource(
|
||||
skipFailed: boolean;
|
||||
retryFailed: boolean;
|
||||
concurrency: number | undefined;
|
||||
/** v0.41.37.0 #1569: propagate --no-schema-pack into every per-source sync. */
|
||||
noSchemaPack?: boolean;
|
||||
},
|
||||
): Promise<{ result: SyncResult; log: string }> {
|
||||
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
|
||||
@@ -2913,6 +2945,7 @@ export async function syncOneSource(
|
||||
noEmbed: shared.noEmbed,
|
||||
skipFailed: shared.skipFailed,
|
||||
retryFailed: shared.retryFailed,
|
||||
noSchemaPack: shared.noSchemaPack,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
concurrency: shared.concurrency,
|
||||
|
||||
+19
-6
@@ -669,12 +669,25 @@ export async function importFromContent(
|
||||
);
|
||||
}
|
||||
|
||||
// Tag reconciliation: remove stale, add current
|
||||
const existingTags = await tx.getTags(slug, txOpts);
|
||||
const newTags = new Set(parsed.tags);
|
||||
for (const old of existingTags) {
|
||||
if (!newTags.has(old)) await tx.removeTag(slug, old, txOpts);
|
||||
}
|
||||
// Tag reconciliation: ADD-ONLY (v0.41.37.0 #1621).
|
||||
//
|
||||
// We deliberately do NOT delete existing tags here. The `tags` table has
|
||||
// no provenance column, and frontmatter tags are stripped from the stored
|
||||
// `pages.frontmatter` (markdown.ts:118) — so at re-import time we cannot
|
||||
// distinguish a frontmatter-origin tag from a DB-side enrichment tag
|
||||
// (auto-tag / dream synthesize / signal-detector writes to the same
|
||||
// table). The pre-v0.41.37.0 "delete every existing tag not in the current
|
||||
// frontmatter" logic wiped ALL enrichment tags on every re-import — most
|
||||
// visibly under `gbrain reindex --markdown` (#1621), which re-imports every
|
||||
// page with forceRechunk. reindex is a re-chunk/re-embed op; it must not
|
||||
// destroy tags.
|
||||
//
|
||||
// Trade-off (accepted): removing a tag from a page's frontmatter no longer
|
||||
// removes it from the DB on the next sync. That staleness is minor (tags
|
||||
// are additive metadata) and far preferable to silently losing enrichment
|
||||
// tags. Frontmatter-tag REMOVAL would require a `tag_source` provenance
|
||||
// column (deferred — see TODOS.md #1621-followup). addTag is idempotent
|
||||
// (ON CONFLICT DO NOTHING), so re-adding existing tags is a no-op.
|
||||
for (const tag of parsed.tags) {
|
||||
await tx.addTag(slug, tag, txOpts);
|
||||
}
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
// universe.
|
||||
|
||||
import type { SchemaPackManifest } from './manifest-v1.ts';
|
||||
import { PageRegexBudget } from './redos-guard.ts';
|
||||
import { PageRegexBudget, runRegexBounded } from './redos-guard.ts';
|
||||
|
||||
/**
|
||||
* Try to resolve a link verb from the active pack's declared
|
||||
@@ -80,12 +80,16 @@ export function inferLinkTypeFromPack(
|
||||
}
|
||||
if (match !== null) return lt.name;
|
||||
} else {
|
||||
// No budget provided (test contexts) — run the regex directly.
|
||||
// No budget provided (test contexts) — still route through the bounded
|
||||
// executor so the v0.41.37.0 #1569 input-length cap + vm timeout apply.
|
||||
// Previously this ran `new RegExp(pattern).test(context)` UNBOUNDED, the
|
||||
// one ReDoS hole with no timeout. runRegexBounded throws on
|
||||
// timeout/oversize/malformed → skip and continue (degrade to mentions).
|
||||
try {
|
||||
if (new RegExp(pattern).test(context)) return lt.name;
|
||||
if (runRegexBounded(pattern, context) !== null) return lt.name;
|
||||
} catch {
|
||||
// Malformed pattern — skip and continue. Pack validation
|
||||
// should have caught this at load.
|
||||
// Timed out, oversize input, or malformed pattern — skip and continue.
|
||||
// Pack validation + the star-height lint rule surface bad patterns.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -320,6 +320,34 @@ export const mutationCountAnomaly: LintRule = (manifest, opts) => {
|
||||
// Aggregator
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// v0.41.37.0 #1569: advisory ReDoS pre-screen for pack inference regexes.
|
||||
// Flags the classic nested-quantifier shapes ((a+)+, (a*)*, (a+)*, (\w+)+)
|
||||
// that cause catastrophic backtracking. WARNING, not error: a hard reject
|
||||
// would disable the whole pack on upgrade (pages fall back to legacy typing).
|
||||
// The runtime input-length cap (MAX_REGEX_INPUT_CHARS in redos-guard.ts) is
|
||||
// the actual safety net; this rule tells the author to fix the pattern.
|
||||
// Heuristic: an inner group containing a +/* quantifier, wrapped by an outer
|
||||
// +/* quantifier. Catches the common ReDoS class, not every possible one.
|
||||
const NESTED_QUANTIFIER_RE = /\([^()]*[+*][^()]*\)\s*[+*]/;
|
||||
export const linkRegexCatastrophicBacktrack: LintRule = (manifest) => {
|
||||
const issues: LintIssue[] = [];
|
||||
for (const lt of manifest.link_types) {
|
||||
const pattern = lt.inference?.regex;
|
||||
if (!pattern) continue;
|
||||
if (NESTED_QUANTIFIER_RE.test(pattern)) {
|
||||
issues.push({
|
||||
rule: 'link_regex_catastrophic_backtrack',
|
||||
severity: 'warning',
|
||||
message: `link_type '${lt.name}' inference.regex '${pattern}' has a nested quantifier (e.g. (a+)+) that can cause catastrophic backtracking (ReDoS)`,
|
||||
pack: manifest.name,
|
||||
link: lt.name,
|
||||
hint: `rewrite without nested quantifiers (e.g. (a+)+ → a+). The runtime caps input length, but the pattern stays O(2^n) on adversarial input`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return issues;
|
||||
};
|
||||
|
||||
/** All rules. File-plane callers can compose a subset via FILE_PLANE_RULES. */
|
||||
export const ALL_LINT_RULES: ReadonlyArray<{ name: string; rule: LintRule; planeAware: boolean }> = [
|
||||
{ name: 'alias_shadows_type', rule: aliasShadowsType, planeAware: false },
|
||||
@@ -331,6 +359,7 @@ export const ALL_LINT_RULES: ReadonlyArray<{ name: string; rule: LintRule; plane
|
||||
{ name: 'expert_routing_without_prefix', rule: expertRoutingWithoutPrefix, planeAware: false },
|
||||
{ name: 'prefix_collision', rule: prefixCollision, planeAware: false },
|
||||
{ name: 'prefix_strict_subset_overlap', rule: prefixStrictSubsetOverlap, planeAware: false },
|
||||
{ name: 'link_regex_catastrophic_backtrack', rule: linkRegexCatastrophicBacktrack, planeAware: false },
|
||||
{ name: 'extractable_empty_corpus', rule: extractableEmptyCorpus, planeAware: true },
|
||||
{ name: 'mutation_count_anomaly', rule: mutationCountAnomaly, planeAware: true },
|
||||
];
|
||||
|
||||
@@ -37,6 +37,28 @@ import { runInContext, createContext } from 'node:vm';
|
||||
export const LINK_EXTRACTION_TOTAL_BUDGET_MS = 500 as const;
|
||||
export const PER_REGEX_TIMEOUT_MS = 50 as const;
|
||||
|
||||
// v0.41.37.0 #1569: hard input-length cap. Catastrophic backtracking needs a
|
||||
// long input to blow up; a pack regex run against a multi-KB body is the blast
|
||||
// radius. Capping the input length removes it cheaply — a link-extraction
|
||||
// `context` is normally a sentence or short paragraph, so 64KB is generous.
|
||||
// Over the cap, the regex is skipped (degrade-to-mentions) without even
|
||||
// entering the vm. This is the real runtime safety net (the star-height lint
|
||||
// rule is advisory). Env-overridable for power users with huge contexts.
|
||||
export const MAX_REGEX_INPUT_CHARS = (() => {
|
||||
const raw = process.env.GBRAIN_MAX_REGEX_INPUT_CHARS;
|
||||
const n = raw ? parseInt(raw, 10) : NaN;
|
||||
return Number.isFinite(n) && n > 0 ? n : 64_000;
|
||||
})();
|
||||
|
||||
/** Tagged error thrown when input exceeds MAX_REGEX_INPUT_CHARS. Treated as
|
||||
* degrade-to-mentions by `PageRegexBudget.runBounded` (counts against budget). */
|
||||
export class RegexInputTooLargeError extends Error {
|
||||
constructor(public readonly length: number) {
|
||||
super(`regex input ${length} chars exceeds cap ${MAX_REGEX_INPUT_CHARS}`);
|
||||
this.name = 'RegexInputTooLargeError';
|
||||
}
|
||||
}
|
||||
|
||||
export class RegexTimeoutError extends Error {
|
||||
readonly verb: string;
|
||||
readonly pattern: string;
|
||||
@@ -123,6 +145,13 @@ export function runRegexBounded(
|
||||
text: string,
|
||||
timeoutMs: number = PER_REGEX_TIMEOUT_MS,
|
||||
): RegExpMatchArray | null {
|
||||
// v0.41.37.0 #1569: input-length cap BEFORE the vm. Over the cap, skip the
|
||||
// regex entirely (the surrounding budget treats the throw as degrade). This
|
||||
// is the primary ReDoS safety net — catastrophic backtracking can't blow up
|
||||
// on input it never sees.
|
||||
if (text.length > MAX_REGEX_INPUT_CHARS) {
|
||||
throw new RegexInputTooLargeError(text.length);
|
||||
}
|
||||
// Create a fresh context so the pack's regex can't leak state across
|
||||
// runs. Pass pattern + text as primitives only.
|
||||
const ctx = createContext({ pattern, text });
|
||||
|
||||
@@ -95,11 +95,16 @@ describe('v0.36.1.x #1077 — admin register-client supports PKCE public clients
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.36.1.x #1100 — PGLite v0.11.0 phaseASchema routes in-process', () => {
|
||||
test('phaseASchema branches on pglite and calls initSchema directly', () => {
|
||||
describe('v0.41.37.0 #1605 — v0.11.0 phaseASchema routes in-process for ALL engines', () => {
|
||||
test('phaseASchema calls runMigrateOnlyCore (in-process) + is awaited', () => {
|
||||
// Supersedes #1100's PGLite-only in-process branch. v0.41.37.0 #1605 routes
|
||||
// EVERY engine through runMigrateOnlyCore (no execSync subprocess at all),
|
||||
// which is strictly stronger: PGLite still never subprocesses, AND the
|
||||
// Windows+Postgres getaddrinfo-ENOTFOUND spawn bug is closed too.
|
||||
// The eng.initSchema() call moved into src/commands/migrations/in-process.ts.
|
||||
const src = readFileSync('src/commands/migrations/v0_11_0.ts', 'utf8');
|
||||
expect(src).toMatch(/cfg\?\.engine\s*===\s*'pglite'/);
|
||||
expect(src).toMatch(/eng\.initSchema\(\)/);
|
||||
expect(src).toContain('runMigrateOnlyCore()');
|
||||
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
|
||||
expect(src).toMatch(/await\s+phaseASchema/);
|
||||
});
|
||||
|
||||
|
||||
@@ -217,7 +217,11 @@ Same content.
|
||||
expect(putCall).toBeUndefined();
|
||||
});
|
||||
|
||||
test('reconciles tags: removes old, adds new', async () => {
|
||||
test('reconciles tags: ADD-ONLY, never removes (v0.41.37.0 #1621)', async () => {
|
||||
// Add-only reconciliation: re-import adds current frontmatter tags and
|
||||
// NEVER deletes — so DB-side enrichment tags (here simulated as 'old-tag')
|
||||
// survive. The pre-#1621 behavior deleted every existing tag not in the
|
||||
// current frontmatter, wiping enrichment tags on every re-import/reindex.
|
||||
const filePath = join(TMP, 'retag.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
@@ -239,9 +243,11 @@ Content here.
|
||||
const removeCalls = calls.filter((c: any) => c.method === 'removeTag');
|
||||
const addCalls = calls.filter((c: any) => c.method === 'addTag');
|
||||
|
||||
expect(removeCalls.length).toBe(1);
|
||||
expect(removeCalls[0].args[1]).toBe('old-tag');
|
||||
// No removals — 'old-tag' (enrichment) is preserved.
|
||||
expect(removeCalls.length).toBe(0);
|
||||
// Both current frontmatter tags are added (idempotent via ON CONFLICT).
|
||||
expect(addCalls.length).toBe(2);
|
||||
expect(addCalls.map((c: any) => c.args[1]).sort()).toEqual(['kept-tag', 'new-tag']);
|
||||
});
|
||||
|
||||
test('chunks compiled_truth and timeline separately', async () => {
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
// v0.41.37.0 #1605 — migration schema phases run IN-PROCESS (was a
|
||||
// `gbrain init --migrate-only` subprocess that died with getaddrinfo ENOTFOUND
|
||||
// on Windows+bun+Supabase). runMigrateOnlyCore is the single in-process path;
|
||||
// runGbrainSubprocess captures child stderr for the remaining backfill spawns.
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { tmpdir } from 'os';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, readdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
runMigrateOnlyCore,
|
||||
runGbrainSubprocess,
|
||||
MigrateOnlyError,
|
||||
} from '../src/commands/migrations/in-process.ts';
|
||||
|
||||
const MIGRATION_DIR = join(import.meta.dir, '..', 'src', 'commands', 'migrations');
|
||||
const SCHEMA_PHASE_FILES = [
|
||||
'v0_11_0', 'v0_12_0', 'v0_12_2', 'v0_13_0', 'v0_16_0',
|
||||
'v0_18_0', 'v0_18_1', 'v0_21_0', 'v0_29_1',
|
||||
];
|
||||
|
||||
describe('#1605 runMigrateOnlyCore (in-process schema)', () => {
|
||||
test('brings a fresh PGLite brain to head without spawning', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'mip-'));
|
||||
const dataDir = join(home, 'data');
|
||||
mkdirSync(join(home, '.gbrain'), { recursive: true });
|
||||
writeFileSync(
|
||||
join(home, '.gbrain', 'config.json'),
|
||||
JSON.stringify({ engine: 'pglite', database_path: dataDir }),
|
||||
);
|
||||
|
||||
const result = await withEnv(
|
||||
{ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined },
|
||||
() => runMigrateOnlyCore(),
|
||||
);
|
||||
expect(result.engine).toBe('pglite');
|
||||
|
||||
// Verify schema landed: reconnect a fresh engine to the same data dir and
|
||||
// confirm a core table exists (proves initSchema ran in-process).
|
||||
const verify = new PGLiteEngine();
|
||||
await verify.connect({ database_path: dataDir });
|
||||
try {
|
||||
const rows = await verify.executeRaw<{ t: string | null }>(
|
||||
"SELECT to_regclass('public.pages')::text AS t",
|
||||
);
|
||||
expect(rows[0]?.t).toBe('pages');
|
||||
} finally {
|
||||
await verify.disconnect();
|
||||
}
|
||||
});
|
||||
|
||||
test('throws MigrateOnlyError when no brain is configured', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'mip-noconf-'));
|
||||
await expect(
|
||||
withEnv(
|
||||
{ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined },
|
||||
() => runMigrateOnlyCore(),
|
||||
),
|
||||
).rejects.toBeInstanceOf(MigrateOnlyError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1605 runGbrainSubprocess (stderr capture)', () => {
|
||||
test('folds child stderr into the thrown error', () => {
|
||||
let msg = '';
|
||||
try {
|
||||
runGbrainSubprocess("sh -c 'echo BOOM_STDERR 1>&2; exit 1'");
|
||||
} catch (e) {
|
||||
msg = e instanceof Error ? e.message : String(e);
|
||||
}
|
||||
expect(msg).toContain('BOOM_STDERR');
|
||||
});
|
||||
|
||||
test('returns child stdout on success', () => {
|
||||
const out = runGbrainSubprocess("sh -c 'echo hello-stdout'");
|
||||
expect(out).toContain('hello-stdout');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1605 structural guard: schema phases are in-process', () => {
|
||||
test('no schema phase still execSyncs `gbrain init --migrate-only`', () => {
|
||||
for (const f of SCHEMA_PHASE_FILES) {
|
||||
const src = readFileSync(join(MIGRATION_DIR, `${f}.ts`), 'utf-8');
|
||||
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
|
||||
}
|
||||
});
|
||||
|
||||
test('every schema phase calls runMigrateOnlyCore + is awaited', () => {
|
||||
for (const f of SCHEMA_PHASE_FILES) {
|
||||
const src = readFileSync(join(MIGRATION_DIR, `${f}.ts`), 'utf-8');
|
||||
expect(src).toContain('runMigrateOnlyCore()');
|
||||
// phaseASchema must be async + awaited at its call site.
|
||||
expect(src).toContain('async function phaseASchema');
|
||||
const awaited = src.includes('await phaseASchema(opts)') ||
|
||||
src.includes('push(await phaseASchema(opts))');
|
||||
expect(awaited).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('NO migration orchestrator anywhere spawns `gbrain init --migrate-only`', () => {
|
||||
// All-files invariant (not just the 9): the subprocess spawn is the
|
||||
// Windows-ENOTFOUND bug class. Other files (v0_22_4, v0_28_0, v0_31_0,
|
||||
// v0_14_0, v0_32_2) define an in-process phaseASchema that never spawned —
|
||||
// those are fine. We only ban the spawn literal.
|
||||
const files = readdirSync(MIGRATION_DIR).filter(n => /^v\d/.test(n) && n.endsWith('.ts'));
|
||||
for (const n of files) {
|
||||
const src = readFileSync(join(MIGRATION_DIR, n), 'utf-8');
|
||||
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -59,12 +59,16 @@ describe('v0.13.0 — Frontmatter relationship indexing migration', () => {
|
||||
expect(src).not.toMatch(/\$\{GBRAIN\}/);
|
||||
});
|
||||
|
||||
test('phase commands invoke bare `gbrain` shell-out (Bug 1 fix)', () => {
|
||||
test('phases use in-process schema + bare `gbrain` subprocess (v0.41.37.0 #1605)', () => {
|
||||
const src = readFileSync(SRC_PATH, 'utf-8');
|
||||
// All three phases shell out to bare `gbrain` so the canonical shim
|
||||
// on PATH wins. This is the shape v0_12_0 has always used.
|
||||
expect(src).toContain("execSync('gbrain init --migrate-only'");
|
||||
expect(src).toContain("execSync('gbrain extract links --source db --include-frontmatter'");
|
||||
// Schema bring-up is now IN-PROCESS (was execSync('gbrain init
|
||||
// --migrate-only'), which died with getaddrinfo ENOTFOUND on Windows).
|
||||
expect(src).toContain('runMigrateOnlyCore()');
|
||||
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
|
||||
// Backfill extract goes through the stderr-capturing wrapper (still bare
|
||||
// `gbrain` so the canonical shim on PATH wins).
|
||||
expect(src).toContain("runGbrainSubprocess('gbrain extract links --source db --include-frontmatter'");
|
||||
// Stats readback still shells out (reads stdout); bare gbrain.
|
||||
expect(src).toContain("execSync('gbrain call get_stats'");
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
// v0.41.37.0 #1581 — chunked, source-safe, soft-delete-filtered grandfather pass.
|
||||
//
|
||||
// The pre-#1581 per-page getPage+putPage loop hung CPU-bound for 70+ min on an
|
||||
// 82K-page PGLite brain. The rewrite is a chunked bulk SQL pass keyed on
|
||||
// pages.id (NOT slug — slug isn't globally unique), filtering deleted_at IS NULL.
|
||||
// These tests drive phaseCGrandfather directly against a real PGLite engine.
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { tmpdir } from 'os';
|
||||
import { mkdtempSync, existsSync, readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { phaseCGrandfather } from '../src/commands/migrations/v0_13_1.ts';
|
||||
import type { OrchestratorOpts } from '../src/commands/migrations/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
const ensureSource = (id: string) =>
|
||||
engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT DO NOTHING', [id]);
|
||||
|
||||
const seed = (slug: string, frontmatter: Record<string, unknown>, sourceId = 'default') =>
|
||||
engine.putPage(slug, {
|
||||
type: 'concept', title: slug, compiled_truth: 'body', timeline: '', frontmatter,
|
||||
}, { sourceId });
|
||||
|
||||
const fmOf = async (slug: string, sourceId = 'default') => {
|
||||
const p = await engine.getPage(slug, { sourceId });
|
||||
return p?.frontmatter ?? null;
|
||||
};
|
||||
|
||||
// Run the phase with an isolated GBRAIN_HOME so the rollback log lands in a
|
||||
// tempdir (withEnv keeps R1 test-isolation compliance — no raw process.env writes).
|
||||
async function gf(home: string, opts: Partial<OrchestratorOpts> = {}) {
|
||||
const full: OrchestratorOpts = { yes: true, dryRun: false, noAutopilotInstall: true, ...opts };
|
||||
return withEnv({ GBRAIN_HOME: home }, () => phaseCGrandfather(engine, full));
|
||||
}
|
||||
|
||||
describe('#1581 phaseCGrandfather (chunked, source-safe)', () => {
|
||||
test('absent validate key → validate:false', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
await seed('a', { foo: 1 });
|
||||
const { result } = await gf(home);
|
||||
expect(result.status).toBe('complete');
|
||||
expect((await fmOf('a'))?.validate).toBe(false);
|
||||
});
|
||||
|
||||
test('explicit validate (true/false/null) is NOT modified', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
await seed('t', { validate: true });
|
||||
await seed('f', { validate: false });
|
||||
await seed('n', { validate: null });
|
||||
await gf(home);
|
||||
expect((await fmOf('t'))?.validate).toBe(true);
|
||||
expect((await fmOf('f'))?.validate).toBe(false);
|
||||
// null value still counts as "key present" → left untouched.
|
||||
expect((await fmOf('n'))?.validate ?? 'WAS_NULL').toBe('WAS_NULL');
|
||||
});
|
||||
|
||||
test('soft-deleted pages are NOT grandfathered', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
await seed('live', { foo: 1 });
|
||||
await seed('dead', { foo: 1 });
|
||||
await engine.softDeletePage('dead');
|
||||
await gf(home);
|
||||
|
||||
expect((await fmOf('live'))?.validate).toBe(false);
|
||||
const rows = await engine.executeRaw<{ fm: Record<string, unknown> }>(
|
||||
"SELECT frontmatter AS fm FROM pages WHERE slug = 'dead'",
|
||||
);
|
||||
expect(Object.prototype.hasOwnProperty.call(rows[0]?.fm ?? {}, 'validate')).toBe(false);
|
||||
});
|
||||
|
||||
test('multi-source duplicate slugs: both grandfathered, no cross-contamination', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
await ensureSource('src2');
|
||||
await seed('people/dup', { src: 'a' }, 'default');
|
||||
await seed('people/dup', { src: 'b' }, 'src2');
|
||||
await gf(home);
|
||||
|
||||
expect((await fmOf('people/dup', 'default'))?.validate).toBe(false);
|
||||
expect((await fmOf('people/dup', 'src2'))?.validate).toBe(false);
|
||||
expect((await fmOf('people/dup', 'default'))?.src).toBe('a');
|
||||
expect((await fmOf('people/dup', 'src2'))?.src).toBe('b');
|
||||
});
|
||||
|
||||
test('rollback log carries source identity', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
await ensureSource('src2');
|
||||
await seed('people/dup', { src: 'a' }, 'default');
|
||||
await seed('people/dup', { src: 'b' }, 'src2');
|
||||
await gf(home);
|
||||
|
||||
// configDir() appends '.gbrain' to GBRAIN_HOME.
|
||||
const logPath = join(home, '.gbrain', 'migrations', 'v0_13_1-rollback.jsonl');
|
||||
expect(existsSync(logPath)).toBe(true);
|
||||
const lines = readFileSync(logPath, 'utf-8').trim().split('\n').map(l => JSON.parse(l));
|
||||
expect(lines.map(l => l.source_id).sort()).toEqual(['default', 'src2']);
|
||||
expect(lines.every(l => typeof l.id === 'number' && l.slug === 'people/dup')).toBe(true);
|
||||
});
|
||||
|
||||
test('re-run is a no-op (idempotent)', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
await seed('a', { foo: 1 });
|
||||
const first = await gf(home);
|
||||
expect(first.detail.touched).toBe(1);
|
||||
const second = await gf(home);
|
||||
expect(second.detail.touched).toBe(0);
|
||||
});
|
||||
|
||||
test('dry-run counts without mutating', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
await seed('a', { foo: 1 });
|
||||
const { detail } = await gf(home, { dryRun: true });
|
||||
expect(detail.touched).toBe(1);
|
||||
expect((await fmOf('a'))?.validate ?? 'UNSET').toBe('UNSET');
|
||||
});
|
||||
|
||||
test('large-N (1200 pages) completes via chunked pass (hang regression)', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gf-'));
|
||||
for (let i = 0; i < 1200; i++) await seed(`bulk/${i}`, { i });
|
||||
const t0 = Date.now();
|
||||
const { result, detail } = await gf(home);
|
||||
expect(result.status).toBe('complete');
|
||||
expect(detail.touched).toBe(1200);
|
||||
expect((await fmOf('bulk/0'))?.validate).toBe(false);
|
||||
expect((await fmOf('bulk/1199'))?.validate).toBe(false);
|
||||
expect(Date.now() - t0).toBeLessThan(30_000);
|
||||
});
|
||||
});
|
||||
@@ -54,8 +54,9 @@ describe('v0.16.0 migration', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
test('phaseASchema skips on dry-run', () => {
|
||||
const r = __testing.phaseASchema({ dryRun: true, yes: true, noAutopilotInstall: true });
|
||||
test('phaseASchema skips on dry-run', async () => {
|
||||
// v0.41.37.0 #1605: phaseASchema is now async (in-process runMigrateOnlyCore).
|
||||
const r = await __testing.phaseASchema({ dryRun: true, yes: true, noAutopilotInstall: true });
|
||||
expect(r.status).toBe('skipped');
|
||||
expect(r.detail).toBe('dry-run');
|
||||
});
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
// v0.41.37.0 #1569 — ReDoS hardening + diagnostics for schema-pack regexes.
|
||||
//
|
||||
// The real exposure was link-inference.ts:85 running `new RegExp(pattern)
|
||||
// .test(context)` UNBOUNDED when no PageRegexBudget was passed. Fix: an input-
|
||||
// length cap (the runtime safety net) + routing that path through the bounded
|
||||
// executor + an advisory star-height lint rule.
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import {
|
||||
runRegexBounded,
|
||||
PageRegexBudget,
|
||||
RegexInputTooLargeError,
|
||||
MAX_REGEX_INPUT_CHARS,
|
||||
} from '../src/core/schema-pack/redos-guard.ts';
|
||||
import { inferLinkTypeFromPack } from '../src/core/schema-pack/link-inference.ts';
|
||||
import { linkRegexCatastrophicBacktrack } from '../src/core/schema-pack/lint-rules.ts';
|
||||
import type { SchemaPackManifest } from '../src/core/schema-pack/manifest-v1.ts';
|
||||
|
||||
describe('#1569 input-length cap', () => {
|
||||
test('runRegexBounded throws RegexInputTooLargeError over the cap', () => {
|
||||
const big = 'a'.repeat(MAX_REGEX_INPUT_CHARS + 1);
|
||||
expect(() => runRegexBounded('a', big)).toThrow(RegexInputTooLargeError);
|
||||
});
|
||||
|
||||
test('runRegexBounded works normally under the cap', () => {
|
||||
const m = runRegexBounded('wor', 'hello world');
|
||||
expect(m).not.toBeNull();
|
||||
expect(runRegexBounded('zzz', 'hello world')).toBeNull();
|
||||
});
|
||||
|
||||
test('PageRegexBudget.runBounded degrades (null) on oversize input', () => {
|
||||
const budget = new PageRegexBudget();
|
||||
const big = 'a'.repeat(MAX_REGEX_INPUT_CHARS + 1);
|
||||
expect(budget.runBounded('verb', 'a', big)).toBeNull();
|
||||
});
|
||||
|
||||
test('inferLinkTypeFromPack (no budget) does NOT run regex unbounded on huge input', () => {
|
||||
// Pre-#1569 this ran new RegExp().test() with no length cap → ReDoS risk.
|
||||
// A catastrophic pattern + a long input must NOT hang; the cap skips it.
|
||||
const pack = {
|
||||
link_types: [{ name: 'founded', inference: { regex: '(a+)+$' } }],
|
||||
} as unknown as Pick<SchemaPackManifest, 'link_types'>;
|
||||
const huge = 'a'.repeat(MAX_REGEX_INPUT_CHARS + 100) + '!';
|
||||
const t0 = Date.now();
|
||||
const result = inferLinkTypeFromPack(pack, 'company', huge);
|
||||
// Skipped via the cap → no match, and fast (no catastrophic backtrack).
|
||||
expect(result).toBeNull();
|
||||
expect(Date.now() - t0).toBeLessThan(2_000);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1569 star-height lint rule', () => {
|
||||
const mk = (regex: string): SchemaPackManifest =>
|
||||
({ name: 'testpack', page_types: [], link_types: [{ name: 'founded', inference: { regex } }] }) as unknown as SchemaPackManifest;
|
||||
|
||||
test('flags classic nested-quantifier shapes as warnings', () => {
|
||||
for (const bad of ['(a+)+', '(a*)*', '(a+)*', '(\\w+)+$', '(.*)+']) {
|
||||
const issues = linkRegexCatastrophicBacktrack(mk(bad)) as ReturnType<typeof linkRegexCatastrophicBacktrack> & any[];
|
||||
expect(issues.length).toBe(1);
|
||||
expect(issues[0].severity).toBe('warning');
|
||||
expect(issues[0].rule).toBe('link_regex_catastrophic_backtrack');
|
||||
expect(issues[0].link).toBe('founded');
|
||||
}
|
||||
});
|
||||
|
||||
test('does NOT flag benign patterns', () => {
|
||||
for (const ok of ['a+', '(abc)+', '(a|b)+', 'founded\\s+\\w+', '[a-z]+@[a-z]+']) {
|
||||
const issues = linkRegexCatastrophicBacktrack(mk(ok)) as any[];
|
||||
expect(issues.length).toBe(0);
|
||||
}
|
||||
});
|
||||
|
||||
test('no regex → no issue', () => {
|
||||
const manifest = { name: 'p', page_types: [], link_types: [{ name: 'x' }] } as unknown as SchemaPackManifest;
|
||||
expect((linkRegexCatastrophicBacktrack(manifest) as any[]).length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1569 --no-schema-pack + heartbeat wiring (structural)', () => {
|
||||
const SYNC = readFileSync(join(import.meta.dir, '..', 'src', 'commands', 'sync.ts'), 'utf-8');
|
||||
|
||||
test('SyncOpts carries noSchemaPack and it gates loadActivePack', () => {
|
||||
expect(SYNC).toContain('noSchemaPack?: boolean');
|
||||
expect(SYNC).toContain("args.includes('--no-schema-pack')");
|
||||
expect(SYNC).toContain('if (opts.noSchemaPack)');
|
||||
});
|
||||
|
||||
test('begin heartbeat fires before importFile (GBRAIN_SYNC_TRACE)', () => {
|
||||
const beginIdx = SYNC.indexOf('begin import:');
|
||||
const importIdx = SYNC.indexOf('importFile(eng, filePath, path');
|
||||
expect(beginIdx).toBeGreaterThan(0);
|
||||
expect(importIdx).toBeGreaterThan(0);
|
||||
expect(beginIdx).toBeLessThan(importIdx);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,75 @@
|
||||
// v0.41.37.0 #1621 — reindex / re-import must NOT wipe DB-side enrichment tags.
|
||||
//
|
||||
// Root cause: import-file.ts reconciled tags from frontmatter only via
|
||||
// DELETE-then-INSERT. The tags table has no provenance column and frontmatter
|
||||
// tags are stripped from stored frontmatter (markdown.ts:118), so every
|
||||
// re-import (notably `gbrain reindex --markdown`, which re-imports with
|
||||
// forceRechunk) deleted all enrichment / dream / signal-detector tags.
|
||||
//
|
||||
// Fix: ADD-ONLY reconciliation. Re-import adds current frontmatter tags and
|
||||
// never deletes. Accepted trade-off: removing a frontmatter tag no longer
|
||||
// removes it from the DB (additive metadata; far better than wiping enrichment).
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { importFromContent } from '../src/core/import-file.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
const page = (tags: string[]) =>
|
||||
`---\ntype: concept\ntitle: Xavi\ntags: [${tags.join(', ')}]\n---\nBody text for the page.\n`;
|
||||
|
||||
describe('#1621 add-only tag reconciliation', () => {
|
||||
test('re-import (reindex) preserves DB-side enrichment tags', async () => {
|
||||
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
|
||||
// Simulate DB-side enrichment writing a tag not present in frontmatter.
|
||||
await engine.addTag('people/xavi', 'enrichment-tag');
|
||||
|
||||
// Re-import the same content (what reindex --markdown does).
|
||||
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
|
||||
|
||||
const tags = await engine.getTags('people/xavi');
|
||||
expect(tags.sort()).toEqual(['enrichment-tag', 'founder', 'yc']);
|
||||
});
|
||||
|
||||
test('frontmatter tags are still added on import', async () => {
|
||||
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
|
||||
const tags = await engine.getTags('people/xavi');
|
||||
expect(tags.sort()).toEqual(['founder', 'yc']);
|
||||
});
|
||||
|
||||
test('add-only: removing a frontmatter tag does NOT remove it (accepted trade-off)', async () => {
|
||||
await importFromContent(engine, 'people/xavi', page(['founder', 'yc']), { forceRechunk: true, noEmbed: true });
|
||||
await engine.addTag('people/xavi', 'enrichment-tag');
|
||||
|
||||
// User removes "yc" from frontmatter and re-imports.
|
||||
await importFromContent(engine, 'people/xavi', page(['founder']), { forceRechunk: true, noEmbed: true });
|
||||
|
||||
const tags = await engine.getTags('people/xavi');
|
||||
// "yc" lingers (add-only); enrichment-tag preserved; founder present.
|
||||
expect(tags.sort()).toEqual(['enrichment-tag', 'founder', 'yc']);
|
||||
});
|
||||
|
||||
test('adding a new frontmatter tag on re-import works (idempotent add)', async () => {
|
||||
await importFromContent(engine, 'people/xavi', page(['founder']), { forceRechunk: true, noEmbed: true });
|
||||
await engine.addTag('people/xavi', 'enrichment-tag');
|
||||
await importFromContent(engine, 'people/xavi', page(['founder', 'growth']), { forceRechunk: true, noEmbed: true });
|
||||
|
||||
const tags = await engine.getTags('people/xavi');
|
||||
expect(tags.sort()).toEqual(['enrichment-tag', 'founder', 'growth']);
|
||||
});
|
||||
});
|
||||
@@ -332,12 +332,13 @@ describe('runAllLintRules — composition', () => {
|
||||
});
|
||||
|
||||
describe('rule registry shape', () => {
|
||||
it('ALL_LINT_RULES contains 11 rules', () => {
|
||||
expect(ALL_LINT_RULES.length).toBe(11);
|
||||
it('ALL_LINT_RULES contains 12 rules', () => {
|
||||
// v0.41.37.0 #1569 added link_regex_catastrophic_backtrack (file-plane).
|
||||
expect(ALL_LINT_RULES.length).toBe(12);
|
||||
});
|
||||
|
||||
it('FILE_PLANE_LINT_RULES excludes the 2 DB-aware rules', () => {
|
||||
expect(FILE_PLANE_LINT_RULES.length).toBe(9);
|
||||
expect(FILE_PLANE_LINT_RULES.length).toBe(10);
|
||||
expect(FILE_PLANE_LINT_RULES.every((r) => !r.planeAware)).toBe(true);
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user