From bde11bb18f66f7a1eb3938edae411c339a4b75db Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 3 Jun 2026 08:00:57 -0700 Subject: [PATCH] =?UTF-8?q?v0.42.18.0=20fix:=20sync=20orphan-pileup=20watc?= =?UTF-8?q?hdog=20(#1633)=20+=20links-lag=20=C2=B5s=20stamp=20(#1768)=20(#?= =?UTF-8?q?1807)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(extract): links_extraction_lag never clears on Postgres (#1768) Stamp the full-microsecond updated_at (via to_char ... AT TIME ZONE UTC) instead of the millisecond-truncated JS Date, so links_extracted_at equals the DB updated_at exactly and the staleness predicate clears. Stamp SQL unchanged: version-arm backdating still works, D4 preserved, CDX-1 strengthened. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(core): out-of-band hard-deadline watchdog primitive (#1633) Bun eval-Worker that SIGTERM->grace->SIGKILLs its own process from a separate OS thread, so a sync whose main event loop is starved (ReDoS spin) still dies. Signals SELF (no PID-reuse footgun). Empirically validated on Bun 1.3.13. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(sync): arm hard-deadline watchdog + graceful SIGINT cancel (#1633) cli.ts installs the watchdog before connectEngine (bounds connect hangs); resolveSyncHardDeadline + composeAbortSignals in sync.ts; SIGINT graceful cancel on single-source + --all; withRefreshingLock timer unref'd. Non-TTY default 3600s makes cron orphan-pileup structurally impossible. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(claude): annotate process-watchdog + #1768/#1633 fixes Co-Authored-By: Claude Opus 4.8 (1M context) * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) * chore: rebump v0.42.13.0 -> v0.42.18.0 (queue collision) Sibling workspaces claimed v0.42.13-v0.42.17; advance this branch's slot. VERSION + package.json + CHANGELOG header + CLAUDE.md annotations + llms bundles. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(key-files): current-state phrasing for #1633/#1768 entries (fix check:doc-history) The doc-history guard bans the bolded **v0.X release-clause marker in reference docs (history belongs in CHANGELOG + git). Rewrote the extract.ts/sync.ts additions as current-state prose and de-versioned the process-watchdog entry. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 25 ++++ VERSION | 2 +- docs/architecture/KEY_FILES.md | 5 +- package.json | 2 +- src/cli.ts | 34 ++++++ src/commands/extract.ts | 7 +- src/commands/sync.ts | 104 +++++++++++++++- src/core/db-lock.ts | 4 + src/core/pglite-engine.ts | 5 +- src/core/postgres-engine.ts | 6 +- src/core/process-watchdog.ts | 172 +++++++++++++++++++++++++++ src/core/types.ts | 9 ++ src/core/utils.ts | 7 ++ test/extract-stale.test.ts | 31 +++++ test/fixtures/watchdog-harness.ts | 33 +++++ test/process-watchdog.serial.test.ts | 67 +++++++++++ test/process-watchdog.test.ts | 61 ++++++++++ test/sync-hard-deadline.test.ts | 90 ++++++++++++++ 18 files changed, 655 insertions(+), 9 deletions(-) create mode 100644 src/core/process-watchdog.ts create mode 100644 test/fixtures/watchdog-harness.ts create mode 100644 test/process-watchdog.serial.test.ts create mode 100644 test/process-watchdog.test.ts create mode 100644 test/sync-hard-deadline.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index cab3b2725..ab7b8f8b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,31 @@ All notable changes to GBrain will be documented in this file. +## [0.42.18.0] - 2026-06-03 + +**A scheduled `gbrain sync` can no longer spin forever and pile up dead processes, and `gbrain doctor` stops showing "100% of pages need link extraction" right after you ran the thing that's supposed to fix it.** + +Two unrelated bugs, both reported from real Postgres/Supabase brains, both fixed here. + +The first one is the scary one. A `gbrain sync --source ` fired from cron could get stuck in a busy loop, peg a CPU core, and ignore `Ctrl-C` and `kill` (only `kill -9` stopped it). When the cron parent exited, the stuck sync was left orphaned, and the next cron tick spawned another. One reporter woke up to 13 of them, 24+ hours old, thrashing a 16 GB Mac mini down to 121 MB free. The root cause: when a sync spins on synchronous work, it starves its own event loop, so the SIGTERM handler and `--timeout` that gbrain already had could never actually run. + +The fix is a watchdog that runs on a separate OS thread and kills the process from outside the starved loop. On a non-interactive run (cron), gbrain now arms a hard deadline by default (1 hour), sends SIGTERM at the deadline for a clean exit, and SIGKILL shortly after if the process is too wedged to respond. A runaway sync now dies on its own instead of piling up. Sync is resumable, so a deadline-hit run just picks up next tick. + +- **Default on for cron, off for you at the keyboard.** Interactive (TTY) syncs stay unbounded. Tune the cron deadline with `GBRAIN_SYNC_MAX_RUNTIME_SECONDS=N`, set a one-off with `gbrain sync --hard-deadline 600`, or opt out with `--no-hard-deadline`. `gbrain sync --source x --timeout 300` now also arms the hard backstop automatically. +- **`Ctrl-C` is clean now too.** Hitting Ctrl-C during a long sync returns a partial result and releases the lock through the normal path, instead of a hard cut that could leave the sync lock stuck until its TTL expired. +- **The spin itself isn't root-caused yet** (it needs a live reproduction; the leading suspect is a pathological regex in a schema pack's link rules, already partly mitigated). The watchdog makes the *symptom* impossible. A `[sync-watchdog]` heartbeat line in your logs, plus the existing `[gbrain phase]` lines, will pinpoint where the next one hangs. + +The second fix: on Postgres, `gbrain doctor`'s `links_extraction_lag` check was permanently stuck at 100%. You'd run `gbrain extract --stale`, it would stamp every page as extracted, and the check would still say every page needs extraction. The stamp was being truncated to millisecond precision while the database kept microseconds, so "last extracted" always looked a hair older than "last updated." Now the stamp carries full microsecond precision and the check clears the moment extraction runs. (Postgres-only; the health score stops being dragged down by a check that could never pass.) + +## To take advantage of v0.42.18.0 + +`gbrain upgrade` is all that's required — both fixes are automatic. Two things worth knowing: + +1. **Scheduled (non-interactive) syncs now have a 1-hour hard deadline by default.** If you run a legitimately long sync from cron (e.g. a first import of a very large brain), raise it: `GBRAIN_SYNC_MAX_RUNTIME_SECONDS=14400 gbrain sync ...` (4h), pass `gbrain sync --hard-deadline `, or opt out with `--no-hard-deadline`. Interactive runs at your keyboard are unbounded as before. +2. **Verify the link-extraction fix (Postgres):** `gbrain extract --stale` then `gbrain doctor` — the `links_extraction_lag` check should now read near 0% and stay there on a re-run (it was stuck at 100%). + +### For contributors +- New reusable primitive `src/core/process-watchdog.ts` (Bun worker-thread self-kill, `eval:true` so it survives `bun --compile`); `resolveSyncHardDeadline` + `composeAbortSignals` in `sync.ts`; watchdog armed in `cli.ts` before `connectEngine` (bounds connect-phase hangs). #1768 fix threads a full-µs `updated_at_iso` (projected via `to_char(... AT TIME ZONE 'UTC', '…US"Z"')`) into `StalePageRow`; the `markPagesExtractedBatch` SQL is unchanged so the version-arm / CDX-1 tests stay green. New tests: `test/process-watchdog.test.ts` (+ `.serial`), `test/sync-hard-deadline.test.ts`, and a deterministic µs regression in `test/extract-stale.test.ts`. Eng review + Codex outside-voice both cleared the plan (Codex empirically validated the worker-self-kill on Bun 1.3.13). ## [0.42.17.0] - 2026-06-03 **A huge `gbrain sync` can no longer get stuck forever losing all its progress when it's killed partway through.** If your brain suddenly grows by tens of thousands of pages (say a background process is enriching one page per commit, all night long), the next sync has a giant backlog to import. If that sync gets killed before it finishes — a session timeout, a laptop sleep, anything — it used to throw away **everything** it had done and start over from zero. The next hour the backlog was even bigger, so it got killed again, and again. It could never catch up. This release makes sync **resumable**: a killed sync banks what it imported, and the next run picks up where it left off. It converges. diff --git a/VERSION b/VERSION index 343912cea..2ff0d0465 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.17.0 \ No newline at end of file +0.42.18.0 \ No newline at end of file diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 1a6c07e32..a0aeb9753 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -144,6 +144,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/dry-fix.ts` — `gbrain doctor --fix` engine. `autoFixDryViolations(fixes, {dryRun})` rewrites inlined rules to `> **Convention:** see [path](path).` callouts via three shape-aware expanders (bullet / blockquote / paragraph). Five guards: working-tree-dirty (`getWorkingTreeStatus()` 3-state `'clean' | 'dirty' | 'not_a_repo'`), no-git-backup, inside-code-fence, already-delegated (40-line proximity, consistent with detector), ambiguous-multi-match, block-is-callout. `execFileSync` array args (no shell, no injection surface). EOF newline preserved. Safety primitives are in `src/core/skill-fix-gates.ts` (back-compat re-exports preserved). `MISSING_RULE_PATTERNS` INSERT pattern type lives alongside REPLACE patterns — same auto-fix entry point + git-safety gates, but places a canonical callout at a target offset (`after-h1-paragraph` only). First INSERT pattern is `brain_first`, auto-inserting `> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) for the lookup chain (search → query → get_page → external).` on any flagged SKILL.md whose verdict is `missing_brain_first`. Idempotent — re-runs detect the existing callout and skip. - `src/core/backoff.ts` — Adaptive load-aware throttling: CPU/memory checks, exponential backoff, active hours multiplier. - `src/core/retry.ts` — canonical retry primitive for transient connection errors. Exports `withRetry(fn, opts)` execution wrapper + `BULK_RETRY_OPTS` constant (`{maxRetries:3, delayMs:1000, delayMaxMs:10000, jitter:'decorrelated'}`, tuned for Supabase Supavisor's 5-10s circuit-breaker recovery) + `BATCH_AUDIT_SITES` typed const (closed enum of every audit-emission site) + `resolveBulkRetryOpts(env)` (reads `GBRAIN_BULK_MAX_RETRIES`/`GBRAIN_BULK_RETRY_BASE_MS`/`GBRAIN_BULK_RETRY_MAX_MS` with `>=0` validation, throws on bad input with a paste-ready hint) + `abortableSleep(ms, signal?)` + `RetryAbortError` (tagged error for clean shutdown) + `computeNextDelay()` (pure-fn for 3 jitter modes: `'none'`, `'full'`, `'decorrelated'`). The execution wrapper is consumed by `postgres-engine.ts` + `pglite-engine.ts` batch primitives (`addLinksBatch` / `addTimelineEntriesBatch` / `upsertChunks`) so every caller inherits retry as part of the data-primitive's contract. CI guard `scripts/check-no-double-retry.sh` fails the build on `withRetry(...engine.batch...)` patterns (prevents 3×3=9 retry amplification); `scripts/check-batch-audit-site.sh` validates every string-literal `auditSite: '...'` against the closed `BATCH_AUDIT_SITES` enum. Decorrelated jitter (AWS-style: `uniform(base, prevDelay*3)` capped at `delayMaxMs`) — `'full'` jitter allowed near-zero retries that re-hit the recovering breaker. `WithRetryOpts` has an optional `reconnect?: () => Promise` callback awaited in the catch branch AFTER `isRetryableConnError` classification but BEFORE the inter-attempt sleep — lets engine-level callers rebuild a dead pool/singleton between attempts. `PostgresEngine.batchRetry` injects `() => this.reconnect()` (the race-safe `_reconnecting` guard kicks in). Fail-loud: a reconnect throw PROPAGATES as the new error, replacing the symptomatic "No database connection". `onRetry` callbacks are awaited (sync arrows work identically; async callbacks correctly delay the sleep). Pinned by `test/core/retry.test.ts` (37 cases), `test/core/retry-stress.slow.test.ts` (5 cases, 100 batches × 30% blip rate, asserts zero row loss), `test/core/retry-reconnect.test.ts` (5 cases), `test/e2e/db-singleton-shared-recovery.test.ts` (3 DB-gated cases). +- `src/core/process-watchdog.ts` (#1633) — out-of-band hard-deadline killer for `gbrain sync`. A spinning sync (synchronous catastrophic-regex / ReDoS in pack link-inference) STARVES the main event loop, so the existing SIGTERM handler (`process-cleanup.ts`), `--timeout` `setTimeout`, and abort-flag checks can't fire — the process becomes unkillable-by-SIGTERM and, under cron, orphans pile up for 24h+ (the reported incident). `installProcessWatchdog({deadlineMs, graceMs?, label?, heartbeatMs?, onWarn?}): WatchdogHandle` spawns a Bun `worker_threads` Worker via `new Worker(code, {eval: true, workerData})` — its own OS thread + event loop fires even while main is in an unyielding sync loop. At `deadlineMs` it `process.kill(process.pid, 'SIGTERM')` (clean-shutdown chance if responsive); at `deadlineMs+graceMs` `process.kill(process.pid, 'SIGKILL')` (uncatchable — guaranteed death under starvation). Signaling SELF has NO PID-reuse footgun (current PID never reused while alive — the reason the detached-child-watches-parent design was rejected). `eval: true` bakes the worker body into the `bun build --compile` binary with no separate-file embedding. Empirically validated on Bun 1.3.13 (worker timer + SIGKILL killed a `while(true){}`-starved process). `handle.dispose()` (clean-exit `finally`) `worker.terminate()`s it; `unref()`'d so it never keeps the process alive. Pure `watchdogDecision(elapsedMs, deadlineMs, graceMs) → 'wait'|'sigterm'|'sigkill'` extracted for unit tests. Optional `heartbeatMs` emits periodic `[