v0.34.3.0 fix: supervisor treats code=0 watchdog exits as crashes (#1003)

* fix: supervisor treats code=0 watchdog exits as crashes

The RSS watchdog triggers gracefulShutdown() which exits with code 0.
The supervisor was counting ALL exits < 5min as crashes, including
clean code=0 exits. After 10 watchdog-triggered restarts (typical with
a 96K-page brain where autopilot inflates RSS), the supervisor gave up
with max_crashes_exceeded.

Fix: code=0 exits reset crashCount to 0 and restart immediately with
no backoff. Only code≠0 exits count toward the crash limit.

Root cause: process.memoryUsage().rss reports 7GB during autopilot
sync on large repos (possibly shared page inflation from git mmap).
The 4096MB threshold triggers on every cycle. This is a separate
issue (RSS measurement accuracy) but the supervisor should handle
clean exits regardless.

* fix: use RssAnon instead of VmRSS for watchdog threshold

process.memoryUsage().rss returns VmRSS which includes file-backed
mmap'd pages. On repos with large git packfiles (96K+ pages), git
operations inflate VmRSS to 7GB+ while actual heap usage is ~100MB.
The kernel reclaims these pages under memory pressure — they're cache.

Replace with /proc/self/status RssAnon + RssShmem which measures only
anonymous pages (heap, stack, anonymous mmap). This is the memory that
actually matters for OOM risk.

Falls back to process.memoryUsage().rss on non-Linux.

Before: watchdog triggers every autopilot cycle (7GB VmRSS > 4GB threshold)
After:  watchdog only triggers on real memory growth (~100MB << 4GB threshold)

Related: #1002 (supervisor crash-count fix for the same symptom)

* refactor(minions): extract ChildWorkerSupervisor with D1/D2 amendments

MinionSupervisor and src/commands/autopilot.ts each owned a separate
spawn-and-respawn loop. PR #1003 fixed the supervisor's crash-counter
bug (counting code=0 watchdog drains as crashes) but the autopilot
loop has the same bug class. Worse, the as-shipped #1003 fix reset
crashCount=0 on every code=0 exit, which lost the "flapping worker"
signal in mixed-exit sequences.

Extract the shared spawn loop into ChildWorkerSupervisor so both
consumers compose one tested core. The new class bakes in two
amendments resolved during plan-eng-review:

D1 (lastExitCode track): code=0 exits no longer touch crashCount.
They emit ms:0 backoff and restart immediately, but the counter
survives across them. A worker alternating exit 1 / exit 0 / exit 1
correctly trips max_crashes; a worker drained 100 times by the
watchdog stays at crashCount=0 and runs forever (also correct).

D2 (clean-restart budget): on platforms where the watchdog measures
VmRSS instead of RssAnon (macOS, kernel <4.5, restricted containers),
a perpetually over-threshold worker could clean-exit in a tight loop
with no observability. New `cleanRestartBudget` option (default 10
clean restarts per 60s window) emits a `health_warn` and applies
backoff once exceeded.

The supervisor now delegates spawn/respawn/backoff to the inner
class and maps ChildSupervisorEvent → existing SupervisorEvent
emit() channel so JSONL audit consumers see byte-compatible output.
PID lock, signal handlers, health check, and process.exit on
max-crashes stay in MinionSupervisor (those are standalone-daemon
concerns the autopilot composer doesn't need).

Tests: 6 new ChildWorkerSupervisor cases (D1 classifier, interleaved
exits, stable-run + clean-exit interaction, D2 budget tripping, per-
instance config isolation, event shape regression). Existing supervisor
tests updated to use exit-1 workers where they previously relied on
clean-exit-as-crash semantics; their assertions (env plumbing, PID
lock, audit shape) are unaffected.

Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(autopilot): compose ChildWorkerSupervisor instead of inline spawn loop

src/commands/autopilot.ts:165-197 used to have its own spawn-and-
respawn loop separate from MinionSupervisor's. It hardcoded
maxCrashes=5, fixed 10s backoff, and counted every exit (including
code=0) toward the crash limit. Codex flagged this during plan-eng
review: the parallel implementation had the same bug class fixed
in #1003, just on a different code path. Anyone running
`gbrain autopilot` as a long-running daemon (instead of
`gbrain jobs supervisor`) would hit it.

Replace the inline `startWorker` + `child.on('exit')` block with
a ChildWorkerSupervisor instance. Drops the parallel `crashCount`,
`lastWorkerStartTime`, and `STABLE_RUN_RESET_MS` state. The
ChildWorkerSupervisor's D1 lastExitCode track + D2 clean-restart
budget apply to autopilot for free.

Shutdown now drains via the supervisor's killChild + awaitChildExit
typed surface instead of reaching into `workerProc` directly. The
onMaxCrashesExceeded callback routes through autopilot's existing
shutdown('max_crashes') path so the lockfile gets cleaned up
(pre-refactor, the inline loop called process.exit(1) directly and
bypassed the cleanup).

Regression coverage in test/autopilot-supervisor-wiring.test.ts:
static-shape grep guards for `--max-rss 2048`, `maxCrashes: 5`,
the shutdown-via-callback wiring, and absence of the legacy inline
names (startWorker, workerProc, crashCount, lastWorkerStartTime,
STABLE_RUN_RESET_MS).

Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(worker): parse RssAnon as field-presence + soften OOM docstring

Two follow-ups to the RssAnon watchdog fix (b81c598f), both surfaced
during plan-eng-review by Codex.

M1: getAccurateRss() used `if (anonKb > 0) return ...` to decide
whether to use the /proc/self/status reading or fall back to
process.memoryUsage().rss. That conflated "RssAnon field missing"
(old kernel, non-Linux) with "RssAnon field present but zero" (a
near-empty worker process whose only memory is shmem). The legitimate
shmem-only worker case fell through to VmRSS even though /proc had a
valid reading.

Fix: split the pure parser (parseRssFromProcStatus) into a separate
exported function that checks field presence via regex match, not
value comparison. Returns null only when the field text doesn't
match `^RssAnon:\s+(\d+)` AND `^RssShmem:\s+(\d+)`. Both fields
present + both zero is now a valid reading of 0 bytes.

M2: the docstring claimed RssAnon + RssShmem was "the memory that
actually matters for OOM risk." Codex pushed back: this is correct
for per-process leak detection but NOT a full container-OOM metric,
because cgroup memory pressure includes page cache. Soften to
"non-file-backed resident memory used for per-process leak
detection" and call out the cgroup caveat explicitly.

getAccurateRss now takes an optional readStatus function for
testability. Production callers use the default; tests inject
canned status text to cover the M1 regression and the fallback paths
without mocking the filesystem.

Tests: 11 cases covering parseRssFromProcStatus (normal, M1 regression
with anon=0 + shmem>0, both-zero, missing fields, malformed values,
shmem-only) and getAccurateRss (injected reader, ENOENT fallback,
old-kernel fallback, malformed-value fallback).

Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(minions): awaitChildExit short-circuits when child already exited

Pre-fix, awaitChildExit registered `child.once('exit', ...)` without
checking whether the child had already terminated. If the child drained
between killChild('SIGTERM') and awaitChildExit() — common on fast
SIGTERM responders — Node's 'exit' event had already fired, the late
listener never resolved, and the caller waited out the full timeout.
On the supervisor's clean shutdown path that's a 35-second hang on
every quick child.

Probe `child.exitCode` and `child.signalCode` first; resolve
immediately when either is non-null. Sub-second clean shutdown
restored.

Pre-existing in the legacy supervisor.ts shape (same bug pattern),
but since the refactor consolidates child-process management into one
class, fix the pattern at the new seam.

Regression test in test/child-worker-supervisor.test.ts: run one full
spawn cycle, then call awaitChildExit on the already-finished cycle
and assert it returns in under 200ms (well under any test timeout).

Surfaced during pre-landing /review on the fix wave.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.34.3.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md key-files entries for v0.34.3.0

Reflects the ChildWorkerSupervisor extraction shipped in this branch:

- Add new entry for src/core/minions/child-worker-supervisor.ts
  covering D1 lastExitCode classifier, D2 clean-restart budget, the
  awaitChildExit short-circuit, and test pinning at
  test/child-worker-supervisor.test.ts
- Update src/core/minions/supervisor.ts entry to note the spawn-loop
  extraction into the shared core + the byte-compatible event-shape
  mapping that preserves JSONL audit consumers
- Update src/commands/autopilot.ts entry to note the parallel-
  supervisor elimination + the shutdown-via-callback wiring
- Update src/core/minions/worker.ts entry with the new RssAnon /
  getAccurateRss exports + the M1 field-presence parser fix

Regenerated llms-full.txt to match (per project rule: every CLAUDE.md
edit must be followed by bun run build:llms).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-14 21:06:42 -07:00
committed by GitHub
co-authored by Wintermute Claude Opus 4.7
parent 3325b405bb
commit 668254b05e
13 changed files with 1345 additions and 233 deletions
+42
View File
@@ -2,6 +2,48 @@
All notable changes to GBrain will be documented in this file.
## [0.34.3.0] - 2026-05-14
**Supervisor stops crashing every 24h on large brains. Watchdog stops false-firing on git mmap.**
**One shared spawn-loop replaces two duplicate copies that drifted into the same bug class.**
The Minions supervisor was committing suicide every 24h on a 96K-page brain. Every autopilot cycle, `process.memoryUsage().rss` reported ~7GB because VmRSS counts file-backed mmap pages and git holds packfiles open. The 2GB RSS watchdog fired, the worker drained cleanly, exited code=0, and the supervisor counted that clean exit as a crash. 10 clean drains in 24h tripped `max_crashes_exceeded` and the supervisor process gave up. External cron restarted it. Loop repeated.
This release fixes the chain at every layer. The worker now reads RssAnon + RssShmem from `/proc/self/status` on Linux (the actual heap, ~100MB during sync on the same brain) and falls back to VmRSS only on macOS / restricted containers / kernels older than 4.5. The supervisor's exit classifier no longer collapses watchdog drains and real crashes into the same `code=0 vs code≠0` distinction. Clean exits leave `crashCount` untouched (so a worker alternating real-crash + watchdog still trips max_crashes), but they restart immediately with no backoff. A new `cleanRestartBudget` (default 10 restarts per 60s) catches the macOS fallback case where the watchdog still false-fires every cycle.
Underneath that, the supervisor and `gbrain autopilot` no longer maintain two parallel spawn-and-respawn loops with different bug classes. Both now compose a shared `ChildWorkerSupervisor` core. Fix the spawn loop once, both consumers benefit.
### What ships in v0.34.3.0
- **Worker RSS now reads RssAnon + RssShmem on Linux.** `process.memoryUsage().rss` returns VmRSS which includes file-backed mmap pages, so git packfiles inflated the reading to 7GB on large brains. The new helper reads `/proc/self/status` for the non-file-backed pages — the metric a leak watchdog actually wants. Fallback to VmRSS on macOS / kernel <4.5 / missing `/proc`.
- **Supervisor preserves flap detection across mixed exits.** code=0 exits no longer reset `crashCount` to 0. A worker that alternates real crashes with watchdog drains still trips `max_crashes_exceeded`; a worker that only ever drains cleanly runs forever.
- **Clean-restart budget caps the macOS-fallback tight-loop.** If 10 code=0 exits happen inside a 60s window, the supervisor emits `health_warn` with reason `clean_restart_budget_exceeded` and applies backoff. Operators get observability instead of a silent restart loop.
- **`ChildWorkerSupervisor` consolidates the spawn loop.** New module at `src/core/minions/child-worker-supervisor.ts` is the single source of truth for child-process supervision. `MinionSupervisor` and `gbrain autopilot` both compose it. No more parallel-implementation drift.
- **Parser field-presence fix.** `RssAnon: 0` with `RssShmem: 512` (shmem-only workers) now parses correctly. Pre-fix it fell through to VmRSS.
- **`awaitChildExit` short-circuit.** Fast SIGTERM responders no longer cause a 35-second hang on clean shutdown — the method now probes `child.exitCode` before registering an exit listener.
### Itemized changes
- `src/core/minions/worker.ts``getAccurateRss()` exported with injectable status reader; `parseRssFromProcStatus()` exported for unit tests. Docstring softened to call out the cgroup-OOM caveat.
- `src/core/minions/child-worker-supervisor.ts` — new file. Pure spawn-and-respawn core. No PID file, no signal handlers, no `process.exit`, no health check. Emits lifecycle events via injected callback.
- `src/core/minions/supervisor.ts` — refactored to compose `ChildWorkerSupervisor`. The pre-shipped reset-to-0 hunk is gone; D1 lastExitCode tracking + D2 budget live in the shared core. PID lock, signal handlers, health check, and `process.exit` on max-crashes stay.
- `src/commands/autopilot.ts` — replaced inline spawn-and-respawn loop (lines 165-197) with `ChildWorkerSupervisor` composition. `onMaxCrashesExceeded` routes through `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor it called `process.exit(1)` directly and bypassed cleanup).
- Tests: `test/child-worker-supervisor.test.ts` (NEW, 7 cases) covers D1 classifier + D2 budget + the awaitChildExit short-circuit regression. `test/worker-rss.test.ts` (NEW, 11 cases) covers the M1 parser fix and fallback paths. `test/autopilot-supervisor-wiring.test.ts` (NEW, 6 cases) static-shape regression guards. `test/supervisor.test.ts` updated to exit-1 in tests that previously relied on clean-exit-as-crash semantics.
### For contributors
- The `ChildWorkerSupervisor` class is the seam to compose against when adding a new long-running child-process supervisor surface. Don't roll a third spawn-and-respawn loop — compose this one.
- The `_now` injection point on `ChildWorkerSupervisor` lets tests fake the clock without `mock.module`. The stable-run reset behavior is pinned by a deterministic test that fakes a 6-minute clean exit between two real crashes.
## To take advantage of v0.34.3.0
`gbrain upgrade` carries the whole wave automatically. No migrations. No config flips. If your supervisor was crashing every 24h, watch the logs for 24h after the upgrade and confirm:
- `worker_exited` events have `likely_cause: 'clean_exit'` (not `unknown` or `runtime_error`)
- `backoff` events have `reason: 'clean_exit'` and `ms: 0`
- Zero `max_crashes_exceeded` events
If `health_warn` with `reason: 'clean_restart_budget_exceeded'` starts firing on your deployment, that's the new D2 budget telling you the watchdog is firing too often. On Linux that means the worker really is leaking; investigate. On macOS that's the VmRSS-fallback fire pattern — set a higher `--max-rss` threshold to suppress.
## [0.34.2.0] - 2026-05-14
**Path-based import checkpoint. The newest files in a partial import no longer disappear.**
+4 -3
View File
@@ -130,8 +130,9 @@ strict behavior when unset.
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`).
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`). **v0.34.3.0:** RSS watchdog metric switched to non-file-backed pages on Linux. New exports `parseRssFromProcStatus(status)` (pure parser, exported for unit tests) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5). The default `getRss` injected into `WorkerOpts` is now `getAccurateRss` instead of `process.memoryUsage().rss`. Closes the prod incident where VmRSS inflated to 7GB on a 96K-page brain (file-backed git packfile mmaps) while heap stayed at ~100MB; the watchdog was firing every autopilot cycle. M1 parser fix uses field-presence regex checks so `RssAnon: 0 + RssShmem: 512` (shmem-only worker case) parses correctly instead of falling through to VmRSS. Pinned by `test/worker-rss.test.ts` (11 cases).
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests. **v0.34.3.0:** spawn-and-respawn loop extracted into the shared `ChildWorkerSupervisor` core (see entry below). MinionSupervisor now composes the inner class via `runSuperviseLoop()``new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` shapes back through the existing `emit()` SupervisorEvent channel — JSONL audit consumers see byte-compatible output across the rename. PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor (standalone-daemon concerns). The pre-shipped reset-to-0-on-code=0 hunk that originally fixed the prod crash-counter incident is gone; the same fix lives in the shared core under the D1 amendment (code=0 leaves `crashCount` untouched, so a worker alternating real crashes + watchdog drains still trips `max_crashes`). D2 `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop by emitting `health_warn { reason: 'clean_restart_budget_exceeded' }` plus backoff after the threshold trips. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of reaching into `this.child` directly. Pinned by `test/supervisor.test.ts` (16 cases; existing tests that previously relied on clean-exit-as-crash semantics now use exit-1 workers since clean exits no longer count) and `test/supervisor-tini.test.ts`.
- `src/core/minions/child-worker-supervisor.ts` (v0.34.3.0) — shared spawn-and-respawn core extracted from `MinionSupervisor` so it can be reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon). Pre-v0.34.3.0 the two consumers maintained parallel spawn loops that drifted into the same bug class — Codex caught it during plan-eng-review on PR #1003. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void` callback so each composer routes to its own log/audit channel. **D1 exit classifier:** `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences — a worker that alternates `exit 1 / exit 0 / exit 1 / exit 0` correctly trips `max_crashes` after 10 real crashes regardless of intervening clean exits). `code != 0` follows the existing `runDuration > stableRunResetMs ? 1 : ++crashCount` rule. **D2 clean-restart budget:** sliding window tracks code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s) before the next spawn. Caps the worst-case tight-loop on macOS / restricted containers / kernel <4.5 where the worker's RSS watchdog falls back to VmRSS. Public read-only accessors `childAlive`, `inBackoff`, `crashCount` for composer health checks; `killChild(signal)` + `awaitChildExit(timeoutMs)` for shutdown paths. `awaitChildExit` short-circuits when `child.exitCode !== null || child.signalCode !== null` (regression caught in pre-landing /review: pre-fix, fast-SIGTERM responders caused a 35-second shutdown hang because the late `once('exit', ...)` listener never fired). Test hooks: `_backoffFloorMs` skips the real backoff curve, `_now` injects a fake clock. Pinned by `test/child-worker-supervisor.test.ts` (7 cases: D1 classifier with code=0 not counted, interleaved exits still trip max_crashes, stable-run + clean-exit interaction across faked 6-minute run, D2 budget triggers backoff + health_warn, budget config is per-instance, awaitChildExit short-circuit, event-shape regression). Plan that produced the design lives at `~/.claude/plans/this-is-a-real-sleepy-sketch.md`.
- `src/core/minions/spawn-helpers.ts` (v0.28.1) — pure `detectTini()` + `buildSpawnInvocation()` helpers consumed by both `supervisor.ts` and `autopilot.ts`. Resolves the DRY violation between the two spawn sites and makes the tini wrapping testable without `mock.module()` (rule R2 of `scripts/check-test-isolation.sh`). `detectTini()` calls `execFileSync('which', ['tini'])` with explicit `env: process.env` so Bun sees runtime PATH mutations (the env-snapshot bug fix). `buildSpawnInvocation(tiniPath, cmd, args)` returns `{cmd, args}` with tini prepended when present, or the bare invocation otherwise. Pinned by `test/spawn-helpers.test.ts` (5 cases) and `test/supervisor-tini.test.ts` (4 cases).
- `src/core/minions/types.ts``MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
@@ -152,7 +153,7 @@ strict behavior when unset.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts``gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts``gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts``gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart).
- `src/commands/autopilot.ts``gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart). **v0.34.3.0:** inline spawn-and-respawn loop replaced with a `ChildWorkerSupervisor` instance. Drops `crashCount`, `lastWorkerStartTime`, `STABLE_RUN_RESET_MS`, `startWorker`, and the inline `child.on('exit')` block — all consolidated into the shared core. `--max-rss 2048` and `maxCrashes: 5` preserved from the legacy loop. `onMaxCrashesExceeded` now routes through autopilot's own `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor the inline loop called `process.exit(1)` directly and bypassed cleanup). `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of `workerProc.kill()`. Pinned by `test/autopilot-supervisor-wiring.test.ts` (6 static-shape regression guards: composes ChildWorkerSupervisor not the legacy inline names, `--max-rss 2048` in argv, `maxCrashes: 5` literal, shutdown-via-callback wiring, no workerProc reference). Closes the parallel-supervisor bug class Codex flagged during plan-eng-review.
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. **v0.34.1.0 (#870):** stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'`. Gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`, similar) pipe the JSON-RPC handshake then close their stdin half; pre-fix this killed the server before the first tool call landed. Signal handlers (SIGTERM / SIGINT / SIGHUP) and the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the runtime guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`.
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP transports. Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1/F2/F3 drift bugs in the original v0.22.5 HTTP transport. **v0.26.9 (F8):** adds `summarizeMcpParams(opName, params)` — privacy-preserving redactor for `mcp_request_log` and the admin SSE feed. Returns `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. Intersects submitted top-level keys against the operation's declared `params` allow-list (declared keys preserved as a sorted array for debug visibility; unknown keys counted but never named, closing the attacker-controlled-key-name leak). Byte counts bucketed up to nearest 1KB so an attacker can't binary-search secret-content sizes via repeated probes. Operators on a personal laptop who want raw payload visibility opt back in with `gbrain serve --http --log-full-params` (loud stderr warning at startup). Canonical helper — new logging code paths route through it rather than `JSON.stringify(params)`.
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter. `buildDefaultLimiters()` returns the two-bucket pipeline: pre-auth IP (30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap bounds memory under attacker-controlled key growth.
+1 -1
View File
@@ -1 +1 @@
0.34.2.0
0.34.3.0
+4 -3
View File
@@ -238,8 +238,9 @@ strict behavior when unset.
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`).
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`). **v0.34.3.0:** RSS watchdog metric switched to non-file-backed pages on Linux. New exports `parseRssFromProcStatus(status)` (pure parser, exported for unit tests) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5). The default `getRss` injected into `WorkerOpts` is now `getAccurateRss` instead of `process.memoryUsage().rss`. Closes the prod incident where VmRSS inflated to 7GB on a 96K-page brain (file-backed git packfile mmaps) while heap stayed at ~100MB; the watchdog was firing every autopilot cycle. M1 parser fix uses field-presence regex checks so `RssAnon: 0 + RssShmem: 512` (shmem-only worker case) parses correctly instead of falling through to VmRSS. Pinned by `test/worker-rss.test.ts` (11 cases).
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests. **v0.34.3.0:** spawn-and-respawn loop extracted into the shared `ChildWorkerSupervisor` core (see entry below). MinionSupervisor now composes the inner class via `runSuperviseLoop()` → `new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` shapes back through the existing `emit()` SupervisorEvent channel — JSONL audit consumers see byte-compatible output across the rename. PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor (standalone-daemon concerns). The pre-shipped reset-to-0-on-code=0 hunk that originally fixed the prod crash-counter incident is gone; the same fix lives in the shared core under the D1 amendment (code=0 leaves `crashCount` untouched, so a worker alternating real crashes + watchdog drains still trips `max_crashes`). D2 `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop by emitting `health_warn { reason: 'clean_restart_budget_exceeded' }` plus backoff after the threshold trips. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of reaching into `this.child` directly. Pinned by `test/supervisor.test.ts` (16 cases; existing tests that previously relied on clean-exit-as-crash semantics now use exit-1 workers since clean exits no longer count) and `test/supervisor-tini.test.ts`.
- `src/core/minions/child-worker-supervisor.ts` (v0.34.3.0) — shared spawn-and-respawn core extracted from `MinionSupervisor` so it can be reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon). Pre-v0.34.3.0 the two consumers maintained parallel spawn loops that drifted into the same bug class — Codex caught it during plan-eng-review on PR #1003. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void` callback so each composer routes to its own log/audit channel. **D1 exit classifier:** `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences — a worker that alternates `exit 1 / exit 0 / exit 1 / exit 0` correctly trips `max_crashes` after 10 real crashes regardless of intervening clean exits). `code != 0` follows the existing `runDuration > stableRunResetMs ? 1 : ++crashCount` rule. **D2 clean-restart budget:** sliding window tracks code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s) before the next spawn. Caps the worst-case tight-loop on macOS / restricted containers / kernel <4.5 where the worker's RSS watchdog falls back to VmRSS. Public read-only accessors `childAlive`, `inBackoff`, `crashCount` for composer health checks; `killChild(signal)` + `awaitChildExit(timeoutMs)` for shutdown paths. `awaitChildExit` short-circuits when `child.exitCode !== null || child.signalCode !== null` (regression caught in pre-landing /review: pre-fix, fast-SIGTERM responders caused a 35-second shutdown hang because the late `once('exit', ...)` listener never fired). Test hooks: `_backoffFloorMs` skips the real backoff curve, `_now` injects a fake clock. Pinned by `test/child-worker-supervisor.test.ts` (7 cases: D1 classifier with code=0 not counted, interleaved exits still trip max_crashes, stable-run + clean-exit interaction across faked 6-minute run, D2 budget triggers backoff + health_warn, budget config is per-instance, awaitChildExit short-circuit, event-shape regression). Plan that produced the design lives at `~/.claude/plans/this-is-a-real-sleepy-sketch.md`.
- `src/core/minions/spawn-helpers.ts` (v0.28.1) — pure `detectTini()` + `buildSpawnInvocation()` helpers consumed by both `supervisor.ts` and `autopilot.ts`. Resolves the DRY violation between the two spawn sites and makes the tini wrapping testable without `mock.module()` (rule R2 of `scripts/check-test-isolation.sh`). `detectTini()` calls `execFileSync('which', ['tini'])` with explicit `env: process.env` so Bun sees runtime PATH mutations (the env-snapshot bug fix). `buildSpawnInvocation(tiniPath, cmd, args)` returns `{cmd, args}` with tini prepended when present, or the bare invocation otherwise. Pinned by `test/spawn-helpers.test.ts` (5 cases) and `test/supervisor-tini.test.ts` (4 cases).
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
@@ -260,7 +261,7 @@ strict behavior when unset.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart).
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart). **v0.34.3.0:** inline spawn-and-respawn loop replaced with a `ChildWorkerSupervisor` instance. Drops `crashCount`, `lastWorkerStartTime`, `STABLE_RUN_RESET_MS`, `startWorker`, and the inline `child.on('exit')` block — all consolidated into the shared core. `--max-rss 2048` and `maxCrashes: 5` preserved from the legacy loop. `onMaxCrashesExceeded` now routes through autopilot's own `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor the inline loop called `process.exit(1)` directly and bypassed cleanup). `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of `workerProc.kill()`. Pinned by `test/autopilot-supervisor-wiring.test.ts` (6 static-shape regression guards: composes ChildWorkerSupervisor not the legacy inline names, `--max-rss 2048` in argv, `maxCrashes: 5` literal, shutdown-via-callback wiring, no workerProc reference). Closes the parallel-supervisor bug class Codex flagged during plan-eng-review.
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. **v0.34.1.0 (#870):** stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'`. Gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`, similar) pipe the JSON-RPC handshake then close their stdin half; pre-fix this killed the server before the first tool call landed. Signal handlers (SIGTERM / SIGINT / SIGHUP) and the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the runtime guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`.
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP transports. Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1/F2/F3 drift bugs in the original v0.22.5 HTTP transport. **v0.26.9 (F8):** adds `summarizeMcpParams(opName, params)` — privacy-preserving redactor for `mcp_request_log` and the admin SSE feed. Returns `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. Intersects submitted top-level keys against the operation's declared `params` allow-list (declared keys preserved as a sorted array for debug visibility; unknown keys counted but never named, closing the attacker-controlled-key-name leak). Byte counts bucketed up to nearest 1KB so an attacker can't binary-search secret-content sizes via repeated probes. Operators on a personal laptop who want raw payload visibility opt back in with `gbrain serve --http --log-full-params` (loud stderr warning at startup). Canonical helper — new logging code paths route through it rather than `JSON.stringify(params)`.
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter. `buildDefaultLimiters()` returns the two-bucket pipeline: pre-auth IP (30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap bounds memory under attacker-controlled key growth.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.34.2.0",
"version": "0.34.3.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
+61 -55
View File
@@ -19,11 +19,11 @@
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync } from 'fs';
import { join } from 'path';
import { execSync, spawn, type ChildProcess } from 'child_process';
import { execSync } from 'child_process';
import type { BrainEngine } from '../core/engine.ts';
import { loadPreferences } from '../core/preferences.ts';
import { loadConfig } from '../core/config.ts';
import { detectTini, buildSpawnInvocation } from '../core/minions/spawn-helpers.ts';
import { ChildWorkerSupervisor } from '../core/minions/child-worker-supervisor.ts';
function parseArg(args: string[], flag: string): string | undefined {
const idx = args.indexOf(flag);
@@ -146,55 +146,64 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
const spawnManagedWorker = useMinionsDispatch && !noWorker;
let stopping = false;
let workerProc: ChildProcess | null = null;
let crashCount = 0;
let lastWorkerStartTime = 0;
// Stable-run reset window (matches MinionSupervisor.ts:471-476 pattern). If the
// worker ran > 5min before exit, treat as a fresh cycle (crashCount=1) so the
// RSS watchdog firing hourly does NOT trip autopilot's give-up threshold after
// ~5 hours of healthy uptime.
const STABLE_RUN_RESET_MS = 5 * 60 * 1000;
let childSupervisor: ChildWorkerSupervisor | null = null;
if (spawnManagedWorker) {
const cliPath = resolveGbrainCliPath();
// Resolve tini once at startup — not per respawn — to avoid shelling out
// every time the worker restarts. Reaps zombie children from shell jobs
// and embed batches that outlive a watchdog-killed worker.
const tiniPath = detectTini();
const startWorker = () => {
// Inject the RSS watchdog default (2048 MB) for the autopilot-supervised
// worker. Bare `gbrain jobs work` has no default; the supervisor and
// autopilot are the production paths that opt in.
const args = ['jobs', 'work', '--max-rss', '2048'];
const { cmd: spawnCmd, args: spawnArgs } = buildSpawnInvocation(tiniPath, cliPath, args);
const child = spawn(spawnCmd, spawnArgs, { stdio: 'inherit', env: process.env });
workerProc = child;
lastWorkerStartTime = Date.now();
console.log(`[autopilot] Minions worker spawned (pid: ${child.pid}, watchdog: 2048MB${tiniPath ? ', tini: active' : ''})`);
child.on('exit', (code) => {
workerProc = null;
if (stopping) return;
const runDuration = Date.now() - lastWorkerStartTime;
if (runDuration > STABLE_RUN_RESET_MS) {
// Stable run — forgive prior crash history. A watchdog-driven hourly
// exit (the production path post-fix) lands here every time.
crashCount = 1;
} else {
crashCount++;
// Inject the RSS watchdog default (2048 MB) for the autopilot-supervised
// worker. Bare `gbrain jobs work` has no default; the supervisor and
// autopilot are the production paths that opt in.
childSupervisor = new ChildWorkerSupervisor({
cliPath,
args: ['jobs', 'work', '--max-rss', '2048'],
// process.env clone; autopilot doesn't gate shell jobs the way the
// standalone supervisor does (autopilot is the operator-trust path).
env: { ...process.env },
maxCrashes: 5,
isStopping: () => stopping,
onMaxCrashesExceeded: (count, max) => {
console.error(`[autopilot] ${count}/${max} consecutive worker crashes, giving up.`);
void shutdown('max_crashes');
},
onEvent: (event) => {
// Route ChildWorkerSupervisor events to autopilot's stderr log.
// Matches the prior console output shape so operators reading
// existing logs see the same lines.
if (event.kind === 'worker_spawned') {
console.log(
`[autopilot] Minions worker spawned (pid: ${event.pid}, watchdog: 2048MB${event.tini ? ', tini: active' : ''})`,
);
} else if (event.kind === 'worker_spawn_failed') {
console.error(
`[autopilot] worker spawn failed (${event.phase}): ${event.error}${event.errnoCode ? ` (code=${event.errnoCode})` : ''}`,
);
} else if (event.kind === 'worker_exited') {
console.error(
`[autopilot] worker exited code=${event.code} signal=${event.signal} after ${event.runDurationMs}ms, crashCount=${event.crashCount}, cause=${event.likelyCause}`,
);
} else if (event.kind === 'backoff') {
if (event.reason === 'budget_exceeded') {
console.error(
`[autopilot] clean-restart budget exceeded; backing off ${event.ms}ms before next spawn`,
);
} else if (event.reason === 'crash') {
console.error(
`[autopilot] crash backoff ${event.ms}ms (crashCount=${event.crashCount})`,
);
}
// reason='clean_exit' with ms:0 is the steady-state watchdog drain;
// logging every iteration would be noisy. Keep silent (the
// worker_exited line already covers the user-visible signal).
} else if (event.kind === 'health_warn') {
console.error(
`[autopilot] health_warn: ${event.reason} count=${event.count} window=${event.windowMs}ms`,
);
}
if (crashCount >= 5) {
console.error(`[autopilot] 5 consecutive worker crashes (run ${runDuration}ms), giving up.`);
process.exit(1);
}
console.error(`[autopilot] worker exited code=${code} after ${runDuration}ms, restart #${crashCount} in 10s`);
setTimeout(startWorker, 10_000);
});
};
startWorker();
},
});
// Fire-and-forget; runs alongside the dispatch loop. shutdown() drives
// the child-supervisor's isStopping accessor + drain.
void childSupervisor.run();
} else if (!useMinionsDispatch) {
const why = mode === 'off'
? 'minion_mode=off'
@@ -215,14 +224,11 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
if (stopping) return;
stopping = true;
console.log(`Autopilot stopping (${sig}).`);
if (workerProc) {
try { workerProc.kill('SIGTERM'); } catch { /* already dead */ }
await Promise.race([
new Promise<void>(r => workerProc!.once('exit', () => r())),
new Promise<void>(r => setTimeout(() => r(), 35_000)),
]);
if (workerProc && !workerProc.killed) {
try { workerProc.kill('SIGKILL'); } catch { /* already dead */ }
if (childSupervisor) {
childSupervisor.killChild('SIGTERM');
await childSupervisor.awaitChildExit(35_000);
if (childSupervisor.childAlive) {
childSupervisor.killChild('SIGKILL');
}
}
try { unlinkSync(lockPath); } catch { /* already gone */ }
+411
View File
@@ -0,0 +1,411 @@
/**
* ChildWorkerSupervisor — Pure spawn-and-respawn core for child workers.
*
* Extracted from MinionSupervisor (src/core/minions/supervisor.ts) so it can
* be reused by both MinionSupervisor (standalone `gbrain jobs supervisor`
* daemon) and the autopilot command (src/commands/autopilot.ts). Pre-fix
* those two had separate spawn loops with different crash-counting bugs;
* this class is the single source of truth.
*
* RESPONSIBILITIES:
* - Spawn the child process (with optional tini wrapper for zombie reaping).
* - Await exit, classify the exit code, decide whether to respawn.
* - Track crash count and trip max_crashes for real failures (code != 0).
* - Track clean-restart budget for code=0 exits and apply backoff when
* the watchdog-drain rate exceeds the budget (D2 in plan).
* - Emit lifecycle events via injected callback.
*
* NON-RESPONSIBILITIES (these stay in the composing class):
* - PID file locking.
* - Signal handlers (SIGTERM/SIGINT).
* - process.exit() — composer decides what to do on max_crashes.
* - Health checks (DB probing, queue depth).
* - Audit-log writing (composer's onEvent decides where it lands).
*
* EXIT CLASSIFIER (post-D1/D2):
*
* code === 0 -> crashCount UNCHANGED (preserves flap detection across
* mixed exit sequences). Record clean-restart timestamp.
* If clean-restart budget exceeded -> emit health_warn +
* apply backoff. Else emit ms:0 backoff (immediate restart).
*
* code !== 0 AND runDuration > stableRunResetMs -> crashCount = 1
* (stable run forgives prior crash history).
*
* code !== 0 AND runDuration <= stableRunResetMs -> crashCount++
* (escalating exponential backoff).
*/
import { spawn, type ChildProcess } from 'child_process';
import { buildSpawnInvocation, detectTini } from './spawn-helpers.ts';
import { calculateBackoffMs } from './supervisor.ts';
export type ChildSupervisorEvent =
| { kind: 'worker_spawned'; pid: number; tini: boolean }
| { kind: 'worker_spawn_failed'; error: string; phase: 'sync' | 'async'; errnoCode?: string }
| {
kind: 'worker_exited';
code: number | null;
signal: NodeJS.Signals | null;
runDurationMs: number;
likelyCause: string;
crashCount: number;
}
| {
kind: 'backoff';
ms: number;
crashCount: number;
reason: 'clean_exit' | 'crash' | 'budget_exceeded';
}
| {
kind: 'health_warn';
reason: 'clean_restart_budget_exceeded';
count: number;
windowMs: number;
};
export interface ChildWorkerSupervisorOpts {
/** Path to the gbrain CLI binary. */
cliPath: string;
/** Worker argv after cliPath (e.g. ['jobs', 'work', '--max-rss', '2048']). */
args: string[];
/** Child env. Defaults to a clone of process.env. */
env?: NodeJS.ProcessEnv;
/** Give up after this many consecutive code != 0 exits. */
maxCrashes: number;
/** Stable-run reset window: code != 0 after this duration resets crashCount to 1. Default 5 min. */
stableRunResetMs?: number;
/**
* D2 clean-restart budget. Caps the rate of code=0 restarts so the
* supervisor cannot tight-loop when the worker exits cleanly forever
* (e.g. macOS RSS fallback path always over-threshold). When the count
* of clean restarts inside `cleanRestartWindowMs` exceeds this number,
* emit `health_warn` and apply `cleanRestartBudgetBackoffMs` before
* the next spawn. Default 10.
*/
cleanRestartBudget?: number;
/** Sliding-window size for budget tracking. Default 60 seconds. */
cleanRestartWindowMs?: number;
/** Backoff applied when budget is exceeded. Default 1 second. */
cleanRestartBudgetBackoffMs?: number;
/**
* Test-only override: minimum backoff in ms between child respawns.
* Tests pass `1` to make crash-loops finish in < 1s. Not exposed via CLI.
* @internal
*/
_backoffFloorMs?: number;
/** Lifecycle event callback. Composer routes these to its own log/audit channels. */
onEvent: (event: ChildSupervisorEvent) => void;
/** Called when crashCount reaches maxCrashes. Composer decides what to do (process.exit, shutdown, etc.). */
onMaxCrashesExceeded: (count: number, max: number) => void;
/** Accessor for the composer's stopping flag; loop exits when this returns true. */
isStopping: () => boolean;
/** Test seed for the clean-restart window. Defaults to Date.now. @internal */
_now?: () => number;
}
const DEFAULTS = {
stableRunResetMs: 5 * 60 * 1000,
cleanRestartBudget: 10,
cleanRestartWindowMs: 60_000,
cleanRestartBudgetBackoffMs: 1_000,
} as const;
export class ChildWorkerSupervisor {
private readonly opts: ChildWorkerSupervisorOpts;
private readonly tiniPath: string;
private _crashCount = 0;
private _lastExitCode: number | null = null;
private _cleanRestartTimestamps: number[] = [];
private _child: ChildProcess | null = null;
private _inBackoff = false;
private _lastStartTime = 0;
constructor(opts: ChildWorkerSupervisorOpts) {
this.opts = opts;
this.tiniPath = detectTini();
}
/** Read-only state surfaces for the composing class's health checks. */
get childAlive(): boolean {
return this._child !== null && this._child.exitCode === null;
}
get inBackoff(): boolean {
return this._inBackoff;
}
get crashCount(): number {
return this._crashCount;
}
/** Whether tini was detected at construction. Used by tests + worker_spawned event payload. */
get isTiniDetected(): boolean {
return this.tiniPath !== '';
}
/**
* Send a signal to the live child (no-op if none). Used by composers'
* shutdown paths. Idempotent — `kill('SIGTERM')` on a dead child is a no-op.
*/
killChild(signal: NodeJS.Signals): void {
if (this._child && !this._child.killed) {
try {
this._child.kill(signal);
} catch {
/* already dead */
}
}
}
/**
* Wait for the current child to exit, bounded by `timeoutMs`. No-op if no
* child is running. Used by composers' graceful-shutdown drains.
*
* Handles the already-exited case: if the child terminated between
* `killChild('SIGTERM')` and this call (common on fast SIGTERM
* responders), Node's `'exit'` event has already fired and a late
* `once('exit', ...)` listener would never resolve. We probe
* `child.exitCode !== null` first and short-circuit so clean shutdown
* is sub-second instead of waiting out the full `timeoutMs`.
*/
awaitChildExit(timeoutMs: number): Promise<void> {
if (!this._child) return Promise.resolve();
const child = this._child;
// Already exited? `exitCode` becomes non-null once Node has seen the
// child terminate. `signalCode` is the symmetric flag for kill-signal
// termination — checked too so a SIGKILLed child also short-circuits.
if (child.exitCode !== null || child.signalCode !== null) {
return Promise.resolve();
}
return new Promise<void>((resolve) => {
let settled = false;
const onExit = () => {
if (settled) return;
settled = true;
resolve();
};
child.once('exit', onExit);
setTimeout(() => {
if (settled) return;
settled = true;
child.removeListener('exit', onExit);
resolve();
}, timeoutMs);
});
}
/**
* Run the spawn-and-respawn loop. Resolves when:
* 1. composer.isStopping() returns true, OR
* 2. crashCount reaches maxCrashes (after firing onMaxCrashesExceeded).
*/
async run(): Promise<void> {
while (!this.opts.isStopping() && this._crashCount < this.opts.maxCrashes) {
await this.spawnOnce();
if (this.opts.isStopping()) return;
if (this._crashCount >= this.opts.maxCrashes) {
this.opts.onMaxCrashesExceeded(this._crashCount, this.opts.maxCrashes);
return;
}
await this.applyBackoff();
}
}
/** Single spawn lifecycle: spawn -> await exit -> classify. */
private spawnOnce(): Promise<void> {
return new Promise<void>((resolve) => {
if (this.opts.isStopping()) {
resolve();
return;
}
const env = this.opts.env ?? { ...process.env };
this._lastStartTime = this.now();
const { cmd: spawnCmd, args: spawnArgs } = buildSpawnInvocation(
this.tiniPath,
this.opts.cliPath,
this.opts.args,
);
let child: ChildProcess;
try {
child = spawn(spawnCmd, spawnArgs, {
stdio: 'inherit',
env,
});
} catch (err: unknown) {
// Synchronous spawn error (e.g. invalid cliPath shape). Count as a crash.
this.opts.onEvent({
kind: 'worker_spawn_failed',
error: err instanceof Error ? err.message : String(err),
phase: 'sync',
});
this._crashCount++;
this._lastExitCode = null;
resolve();
return;
}
this._child = child;
this.opts.onEvent({
kind: 'worker_spawned',
pid: child.pid ?? -1,
tini: this.tiniPath !== '',
});
// Async spawn errors (ENOENT, EACCES). Node fires 'error' first, then
// 'exit' with code=null. We log the error; the 'exit' handler increments
// crashCount as usual so the restart loop bounds permanent misconfigs
// via max_crashes.
child.on('error', (err) => {
this.opts.onEvent({
kind: 'worker_spawn_failed',
error: err.message,
phase: 'async',
errnoCode: (err as NodeJS.ErrnoException).code,
});
});
child.on('exit', (code, signal) => {
this._child = null;
if (this.opts.isStopping()) {
resolve();
return;
}
const runDuration = this.now() - this._lastStartTime;
// D1: code=0 is a clean exit (watchdog drain, graceful stop, etc.).
// Don't touch crashCount — preserves flap detection across mixed
// exit sequences. D2: record the clean-restart timestamp for budget
// tracking and prune entries outside the sliding window.
this._lastExitCode = code;
if (code === 0) {
const nowMs = this.now();
this._cleanRestartTimestamps.push(nowMs);
const windowMs = this.opts.cleanRestartWindowMs ?? DEFAULTS.cleanRestartWindowMs;
const cutoff = nowMs - windowMs;
this._cleanRestartTimestamps = this._cleanRestartTimestamps.filter(
(t) => t > cutoff,
);
} else {
const resetMs = this.opts.stableRunResetMs ?? DEFAULTS.stableRunResetMs;
if (runDuration > resetMs) {
// Stable-run reset: forgive prior crash history.
this._crashCount = 1;
} else {
this._crashCount++;
}
}
// Likely-cause heuristic, kept verbatim from MinionSupervisor.
let likelyCause: string;
if (signal === 'SIGKILL') {
likelyCause = 'oom_or_external_kill';
} else if (signal === 'SIGTERM') {
likelyCause = 'graceful_shutdown';
} else if (code === 1) {
likelyCause = 'runtime_error';
} else if (code === 0) {
likelyCause = 'clean_exit';
} else {
likelyCause = 'unknown';
}
this.opts.onEvent({
kind: 'worker_exited',
code: code ?? null,
signal: signal ?? null,
runDurationMs: runDuration,
likelyCause,
crashCount: this._crashCount,
});
resolve();
});
});
}
/** Compute and apply backoff based on the most recent exit classifier. */
private async applyBackoff(): Promise<void> {
if (this._lastExitCode === 0) {
// D2: check the clean-restart budget. If exceeded, emit health_warn
// and apply a fixed cooldown so the next spawn isn't instant. This
// bounds the worst case on platforms where Diff 2's RssAnon helper
// falls back to VmRSS (macOS, kernel <4.5, restricted containers).
const budget = this.opts.cleanRestartBudget ?? DEFAULTS.cleanRestartBudget;
const windowMs = this.opts.cleanRestartWindowMs ?? DEFAULTS.cleanRestartWindowMs;
if (this._cleanRestartTimestamps.length > budget) {
this.opts.onEvent({
kind: 'health_warn',
reason: 'clean_restart_budget_exceeded',
count: this._cleanRestartTimestamps.length,
windowMs,
});
const cooldown =
this.opts._backoffFloorMs !== undefined
? this.opts._backoffFloorMs
: this.opts.cleanRestartBudgetBackoffMs ?? DEFAULTS.cleanRestartBudgetBackoffMs;
this.opts.onEvent({
kind: 'backoff',
ms: Math.round(cooldown),
crashCount: this._crashCount,
reason: 'budget_exceeded',
});
this._inBackoff = true;
try {
await this.sleep(cooldown);
} finally {
this._inBackoff = false;
}
return;
}
// Within budget — immediate restart.
this.opts.onEvent({
kind: 'backoff',
ms: 0,
crashCount: this._crashCount,
reason: 'clean_exit',
});
return;
}
// code != 0: exponential backoff scaled by crashCount-1 (retry-attempt
// index). On first crash: crashCount=1, exponent=0 -> 1s. After stable-
// run reset: crashCount=1 again -> 1s fresh cycle.
const backoff =
this.opts._backoffFloorMs !== undefined
? this.opts._backoffFloorMs
: calculateBackoffMs(this._crashCount - 1);
this.opts.onEvent({
kind: 'backoff',
ms: Math.round(backoff),
crashCount: this._crashCount,
reason: 'crash',
});
this._inBackoff = true;
try {
await this.sleep(backoff);
} finally {
this._inBackoff = false;
}
}
private sleep(ms: number): Promise<void> {
return new Promise<void>((resolve) => setTimeout(resolve, ms));
}
private now(): number {
return this.opts._now ? this.opts._now() : Date.now();
}
}
+119 -164
View File
@@ -26,8 +26,11 @@
* 3 PID file unwritable (permission / path error)
*/
import { spawn, type ChildProcess } from 'child_process';
import { detectTini, buildSpawnInvocation } from './spawn-helpers.ts';
import { detectTini } from './spawn-helpers.ts';
import {
ChildWorkerSupervisor,
type ChildSupervisorEvent,
} from './child-worker-supervisor.ts';
import {
closeSync,
existsSync,
@@ -35,7 +38,6 @@ import {
openSync,
readFileSync,
unlinkSync,
writeFileSync,
writeSync,
} from 'fs';
import { dirname } from 'path';
@@ -136,13 +138,15 @@ export const ExitCodes = {
export class MinionSupervisor {
private opts: SupervisorOpts;
private engine: BrainEngine;
private child: ChildProcess | null = null;
private crashCount = 0;
private lastStartTime = 0;
/**
* Inner spawn-and-respawn core. Created lazily in `start()` so options
* passed via DEFAULTS merge are visible. Stays null when `stopping` is
* tripped before `start()` runs (test edge case).
*/
private childSupervisor: ChildWorkerSupervisor | null = null;
/** Path to tini binary for zombie reaping, or empty string when absent. */
private readonly tiniPath: string;
private stopping = false;
private inBackoff = false;
private healthInFlight = false;
private healthTimer: ReturnType<typeof setInterval> | null = null;
private exitListener: (() => void) | null = null;
@@ -158,7 +162,8 @@ export class MinionSupervisor {
// Detect tini for zombie reaping. Resolved once at construction so we
// don't shell out on every respawn. Belt-and-suspenders with the
// SIGCHLD handler in cli.ts — tini catches children spawned by native
// addons that bypass the JS event loop.
// addons that bypass the JS event loop. ChildWorkerSupervisor detects
// independently; both calls hit the same `which tini` lookup.
this.tiniPath = detectTini();
}
@@ -276,14 +281,12 @@ export class MinionSupervisor {
this.healthTimer = null;
}
if (this.child) {
try { this.child.kill('SIGTERM'); } catch { /* already dead */ }
await Promise.race([
new Promise<void>(r => this.child!.once('exit', () => r())),
new Promise<void>(r => setTimeout(() => r(), 35_000)),
]);
if (this.child && !this.child.killed) {
try { this.child.kill('SIGKILL'); } catch { /* already dead */ }
if (this.childSupervisor) {
this.childSupervisor.killChild('SIGTERM');
await this.childSupervisor.awaitChildExit(35_000);
// If the child is still up after the 35s drain window, escalate.
if (this.childSupervisor.childAlive) {
this.childSupervisor.killChild('SIGKILL');
}
}
@@ -392,166 +395,116 @@ export class MinionSupervisor {
}
}
/** Run the supervise loop: spawn child, await exit, backoff+retry or give up. */
/**
* Run the supervise loop. Constructs a ChildWorkerSupervisor with the
* D1 lastExitCode classifier + D2 clean-restart budget baked in, then
* defers spawn/respawn/backoff to it. Maps the inner ChildSupervisorEvent
* stream to MinionSupervisor's existing SupervisorEvent emit() channel
* so JSONL audit consumers see byte-compatible output.
*/
private async runSuperviseLoop(): Promise<void> {
while (!this.stopping && this.crashCount < this.opts.maxCrashes) {
await this.spawnOnce();
if (this.stopping) return;
if (this.crashCount >= this.opts.maxCrashes) {
this.emit('max_crashes_exceeded', {
crash_count: this.crashCount,
max_crashes: this.opts.maxCrashes,
});
await this.shutdown('max_crashes', ExitCodes.MAX_CRASHES);
return;
}
// crashCount - 1 is the retry-attempt index (0-based exponent for backoff math).
// On first crash: crashCount=1, backoff exponent=0 → 1s.
// After stable-run reset: crashCount=1 again → 1s fresh cycle.
// Test-only: _backoffFloorMs short-circuits to a fixed tiny value so integration
// tests can exercise crash loops in < 1s without waiting for the real curve.
const backoff = this.opts._backoffFloorMs !== undefined
? this.opts._backoffFloorMs
: calculateBackoffMs(this.crashCount - 1);
this.emit('backoff', { ms: Math.round(backoff), crash_count: this.crashCount });
this.inBackoff = true;
try {
await new Promise<void>(r => setTimeout(r, backoff));
} finally {
this.inBackoff = false;
}
const workerArgs = [
'jobs', 'work',
'--concurrency', String(this.opts.concurrency),
'--queue', this.opts.queue,
];
if (this.opts.maxRssMb > 0) {
workerArgs.push('--max-rss', String(this.opts.maxRssMb));
}
// Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS:
// inherit only when caller opts in, otherwise strip from the clone.
const env: Record<string, string | undefined> = { ...process.env };
if (this.opts.allowShellJobs) {
env.GBRAIN_ALLOW_SHELL_JOBS = '1';
} else {
delete env.GBRAIN_ALLOW_SHELL_JOBS;
}
// Signal to the child worker that it's running under a supervisor.
// The worker's self-health-check (DB probes, stall detection) is
// redundant when the supervisor already provides these — setting
// this env var causes the worker to skip its own health timer.
env.GBRAIN_SUPERVISED = '1';
this.childSupervisor = new ChildWorkerSupervisor({
cliPath: this.opts.cliPath,
args: workerArgs,
env,
maxCrashes: this.opts.maxCrashes,
_backoffFloorMs: this.opts._backoffFloorMs,
isStopping: () => this.stopping,
onMaxCrashesExceeded: (count, max) => {
this.emit('max_crashes_exceeded', {
crash_count: count,
max_crashes: max,
});
void this.shutdown('max_crashes', ExitCodes.MAX_CRASHES);
},
onEvent: (event) => this.relayChildEvent(event),
});
await this.childSupervisor.run();
}
/** Spawn the worker child once and await its exit. Updates `this.crashCount`. */
private spawnOnce(): Promise<void> {
return new Promise<void>((resolve) => {
if (this.stopping) { resolve(); return; }
const args = [
'jobs', 'work',
'--concurrency', String(this.opts.concurrency),
'--queue', this.opts.queue,
];
if (this.opts.maxRssMb > 0) {
args.push('--max-rss', String(this.opts.maxRssMb));
}
// Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS:
// inherit only when caller opts in, otherwise strip from the clone.
const env: Record<string, string | undefined> = { ...process.env };
if (this.opts.allowShellJobs) {
env.GBRAIN_ALLOW_SHELL_JOBS = '1';
} else {
delete env.GBRAIN_ALLOW_SHELL_JOBS;
}
// Signal to the child worker that it's running under a supervisor.
// The worker's self-health-check (DB probes, stall detection) is
// redundant when the supervisor already provides these — setting
// this env var causes the worker to skip its own health timer.
env.GBRAIN_SUPERVISED = '1';
this.lastStartTime = Date.now();
// Wrap with tini when available — reaps zombie children that the
// SIGCHLD handler in cli.ts might miss (native addons, edge cases).
const { cmd: spawnCmd, args: spawnArgs } = buildSpawnInvocation(
this.tiniPath,
this.opts.cliPath,
args,
);
let child: ChildProcess;
try {
child = spawn(spawnCmd, spawnArgs, {
stdio: 'inherit',
env,
/**
* Map ChildSupervisorEvent (the inner core's emission shape) to the
* existing SupervisorEvent emit channel. The wire shape of audit JSONL
* is unchanged — same event names, same field names, same payload
* coverage. The `reason='budget_exceeded'` backoff variant is a new
* additive field; pre-existing consumers ignore unknown fields.
*/
private relayChildEvent(event: ChildSupervisorEvent): void {
switch (event.kind) {
case 'worker_spawned':
this.emit('worker_spawned', {
pid: event.pid >= 0 ? event.pid : undefined,
cli_path: this.opts.cliPath,
...(event.tini ? { tini: true } : {}),
});
} catch (err: unknown) {
// Synchronous spawn error (e.g., invalid cliPath shape). Count as a crash.
return;
case 'worker_spawn_failed':
this.emit('worker_spawn_failed', {
cli_path: this.opts.cliPath,
error: err instanceof Error ? err.message : String(err),
phase: 'sync',
error: event.error,
phase: event.phase,
...(event.errnoCode ? { code: event.errnoCode } : {}),
});
return;
case 'worker_exited': {
const exitReason = event.signal
? `signal ${event.signal}`
: `code ${event.code ?? 'null'}`;
this.emit('worker_exited', {
code: event.code,
signal: event.signal,
reason: exitReason,
likely_cause: event.likelyCause,
crash_count: event.crashCount,
max_crashes: this.opts.maxCrashes,
run_duration_ms: event.runDurationMs,
});
this.crashCount++;
resolve();
return;
}
this.child = child;
this.emit('worker_spawned', {
pid: child.pid,
cli_path: this.opts.cliPath,
...(this.tiniPath ? { tini: true } : {}),
});
// Async spawn errors (ENOENT, EACCES after the fork/exec). Node fires
// 'error' first, then 'exit' with code=null. We log the error; the
// 'exit' handler increments crashCount as usual so the restart loop
// continues (max-crashes bounds this for permanent misconfigs).
child.on('error', (err) => {
this.emit('worker_spawn_failed', {
cli_path: this.opts.cliPath,
error: err.message,
code: (err as NodeJS.ErrnoException).code ?? 'unknown',
phase: 'async',
case 'backoff':
this.emit('backoff', {
ms: event.ms,
crash_count: event.crashCount,
reason: event.reason,
});
});
return;
child.on('exit', (code, signal) => {
this.child = null;
if (this.stopping) {
resolve();
return;
}
// Stable-run reset: if the worker ran > 5min before crashing, we forgive
// prior crash history and treat this as the first crash of a new cycle
// (crashCount = 1, so backoff math uses retry-index 0 = 1s).
const runDuration = Date.now() - this.lastStartTime;
if (runDuration > 5 * 60 * 1000) {
this.crashCount = 1;
} else {
this.crashCount++;
}
const exitReason = signal ? `signal ${signal}` : `code ${code ?? 'null'}`;
// Classify the likely cause for easier debugging
let likelyCause: string;
if (signal === 'SIGKILL') {
likelyCause = 'oom_or_external_kill';
} else if (signal === 'SIGTERM') {
likelyCause = 'graceful_shutdown';
} else if (code === 1) {
likelyCause = 'runtime_error';
} else if (code === 0) {
likelyCause = 'clean_exit';
} else {
likelyCause = 'unknown';
}
this.emit('worker_exited', {
code: code ?? null,
signal: signal ?? null,
reason: exitReason,
likely_cause: likelyCause,
crash_count: this.crashCount,
max_crashes: this.opts.maxCrashes,
run_duration_ms: runDuration,
case 'health_warn':
this.emit('health_warn', {
reason: event.reason,
count: event.count,
window_ms: event.windowMs,
queue: this.opts.queue,
});
resolve();
});
});
return;
}
}
/**
@@ -620,8 +573,10 @@ export class MinionSupervisor {
// F4: suppress "worker not alive" warn while we're in the expected
// null-child window (crash-exit → backoff-sleep → next-spawn).
const workerAlive = this.child != null && this.child.exitCode === null;
if (!workerAlive && !this.stopping && !this.inBackoff) {
const cs = this.childSupervisor;
const workerAlive = cs !== null && cs.childAlive;
const inBackoff = cs !== null && cs.inBackoff;
if (!workerAlive && !this.stopping && !inBackoff) {
this.emit('health_warn', {
reason: 'worker_not_alive',
queue: this.opts.queue,
+64 -1
View File
@@ -22,6 +22,69 @@ import { calculateBackoff } from './backoff.ts';
import { randomUUID } from 'crypto';
import { EventEmitter } from 'events';
import { evaluateQuietHours, type QuietHoursConfig } from './quiet-hours.ts';
import { readFileSync } from 'fs';
/**
* Pure parser for /proc/self/status RSS fields. Returns bytes of
* RssAnon + RssShmem when either field is present, or null when the
* status text is from a kernel that doesn't expose those fields
* (kernels older than 4.5) or the values are malformed. Exported so
* the test suite can unit-test the field-presence + malformed-value
* edge cases without mocking the filesystem.
*
* M1 fix: field-presence check, not value-presence check. The earlier
* `if (anonKb > 0)` form conflated "field exists with value 0" with
* "field missing", which mis-routed to VmRSS fallback in the legitimate
* shmem-only worker case (RssAnon: 0 + RssShmem: 512).
*/
export function parseRssFromProcStatus(status: string): number | null {
const anonMatch = status.match(/^RssAnon:\s+(\d+)/m);
const shmemMatch = status.match(/^RssShmem:\s+(\d+)/m);
if (anonMatch === null && shmemMatch === null) {
return null;
}
const anonKb = parseInt(anonMatch?.[1] ?? '0', 10);
const shmemKb = parseInt(shmemMatch?.[1] ?? '0', 10);
if (isNaN(anonKb) || isNaN(shmemKb)) {
return null;
}
return (anonKb + shmemKb) * 1024; // bytes
}
/**
* Read accurate RSS from /proc/self/status (RssAnon + RssShmem).
*
* `process.memoryUsage().rss` returns VmRSS which includes file-backed mmap'd
* pages (e.g. git packfiles). On a 96K-page brain repo, git operations can
* inflate VmRSS to 7GB+ while actual heap usage is ~100MB. The kernel reclaims
* file-backed pages under memory pressure — they're cache, not real usage.
*
* RssAnon = anonymous pages (heap, stack, anonymous mmap). RssShmem = shared
* anonymous pages (IPC, tmpfs). Their sum is the non-file-backed resident
* memory used for **per-process leak detection** — exactly the metric a leak
* watchdog wants. It is NOT a full container-OOM metric: cgroup memory
* pressure includes page cache, so a sibling container holding the page
* cache hot can OOM us even at low anon+shmem. Use cgroup-aware monitoring
* for that scenario; this helper is for the worker's own leak guard.
*
* Falls back to process.memoryUsage().rss on non-Linux, missing /proc, or
* kernels older than 4.5 that don't expose RssAnon/RssShmem.
*
* `readStatus` is injectable for tests — production callers use the default,
* which reads `/proc/self/status`.
*/
export function getAccurateRss(
readStatus: () => string = () => readFileSync('/proc/self/status', 'utf8'),
): number {
try {
const status = readStatus();
const parsed = parseRssFromProcStatus(status);
if (parsed !== null) return parsed;
} catch {
// Non-Linux or /proc unavailable
}
return process.memoryUsage().rss;
}
/** Reason payload emitted with `'unhealthy'` when self-health-check trips.
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit. */
@@ -93,7 +156,7 @@ export class MinionWorker extends EventEmitter {
maxStalledCount: opts?.maxStalledCount ?? 1,
pollInterval: opts?.pollInterval ?? 5000,
maxRssMb: opts?.maxRssMb ?? 0,
getRss: opts?.getRss ?? (() => process.memoryUsage().rss),
getRss: opts?.getRss ?? getAccurateRss,
rssCheckInterval: opts?.rssCheckInterval ?? 60000,
healthCheckInterval: opts?.healthCheckInterval ?? 60000,
stallWarnAfterMs: opts?.stallWarnAfterMs ?? 5 * 60_000,
+84
View File
@@ -0,0 +1,84 @@
/**
* Regression guards for the autopilot↔ChildWorkerSupervisor wiring.
*
* autopilot.ts:148+ used to have its own inline spawn-and-respawn loop
* (separate from MinionSupervisor's). It had the same bug class fixed
* in #1003 but on a parallel implementation. The fix wave refactored
* autopilot to use the shared ChildWorkerSupervisor core, eliminating
* the parallel-supervisor footgun.
*
* Because the autopilot spawn path is deep inside `runAutopilot()` and
* gated by `spawnManagedWorker` (which needs a Postgres engine), a full
* integration test would require a DATABASE_URL fixture. Instead, these
* static-shape regressions read the source file and pin the load-bearing
* constants:
*
* - `--max-rss 2048` is passed to the worker (incident-driving default)
* - `maxCrashes: 5` matches the prior `crashCount >= 5` give-up rule
* - The autopilot composes ChildWorkerSupervisor (not the legacy
* inline `child.on('exit')` loop)
* - The shutdown path drains via the supervisor's killChild/awaitChildExit
*
* If any of these regress in a future refactor, the test fails with a
* pointer at autopilot.ts.
*/
import { describe, expect, it } from 'bun:test';
import { readFileSync } from 'fs';
import { join } from 'path';
const AUTOPILOT_SRC = readFileSync(
join(import.meta.dir, '..', 'src', 'commands', 'autopilot.ts'),
'utf8',
);
describe('autopilot.ts ↔ ChildWorkerSupervisor wiring', () => {
it('imports ChildWorkerSupervisor from the shared core', () => {
expect(AUTOPILOT_SRC).toContain(
"import { ChildWorkerSupervisor } from '../core/minions/child-worker-supervisor.ts';",
);
});
it('does not retain the legacy inline spawn loop (`startWorker` + child.on)', () => {
// The old code at autopilot.ts:165-197 defined `startWorker` and called
// `child.on('exit', ...)` directly. The refactor must drop those names.
expect(AUTOPILOT_SRC).not.toContain('const startWorker');
expect(AUTOPILOT_SRC).not.toContain("child.on('exit'");
// Also drop the parallel crash-tracking state that lived in autopilot.
expect(AUTOPILOT_SRC).not.toContain('let crashCount');
expect(AUTOPILOT_SRC).not.toContain('let lastWorkerStartTime');
expect(AUTOPILOT_SRC).not.toContain('STABLE_RUN_RESET_MS');
});
it("constructs ChildWorkerSupervisor with --max-rss 2048", () => {
// The worker spawn args must include both flag tokens in argv order.
// This is the incident-driving default; changing it without a deliberate
// decision would regress the workaround for VmRSS inflation.
expect(AUTOPILOT_SRC).toContain("'--max-rss', '2048'");
expect(AUTOPILOT_SRC).toContain("'jobs', 'work'");
});
it("constructs ChildWorkerSupervisor with maxCrashes: 5", () => {
// Matches the legacy `crashCount >= 5` give-up rule from the inline
// loop. The shared core uses this to decide when to fire
// onMaxCrashesExceeded → autopilot's shutdown('max_crashes').
expect(AUTOPILOT_SRC).toMatch(/maxCrashes:\s*5\b/);
});
it('routes onMaxCrashesExceeded to autopilot.shutdown (not process.exit directly)', () => {
// Pre-refactor: autopilot.ts called process.exit(1) directly when the
// crash counter tripped, bypassing its own dispatch-loop cleanup and
// lockfile removal. Post-refactor: the callback routes through
// shutdown('max_crashes') so cleanup runs.
expect(AUTOPILOT_SRC).toMatch(/onMaxCrashesExceeded:[\s\S]{0,300}shutdown\('max_crashes'\)/);
});
it('shutdown drains via supervisor.killChild + awaitChildExit (not workerProc.kill)', () => {
// The legacy shutdown reached into `workerProc` directly. Post-refactor
// those calls go through the supervisor's typed surface, which lets the
// class encapsulate the kill/drain sequence.
expect(AUTOPILOT_SRC).toContain("childSupervisor.killChild('SIGTERM')");
expect(AUTOPILOT_SRC).toContain('childSupervisor.awaitChildExit(35_000)');
expect(AUTOPILOT_SRC).not.toContain('workerProc');
});
});
+410
View File
@@ -0,0 +1,410 @@
/**
* Tests for the shared spawn-and-respawn core used by MinionSupervisor
* and src/commands/autopilot.ts. Pins the D1 lastExitCode-track behavior
* and the D2 clean-restart-budget gate so future refactors can't silently
* regress the supervisor crash-count incident this wave fixes.
*
* Strategy: each test writes a small shell script to disk that exits with
* a chosen code after an optional sleep. The class spawns the script as
* the "worker" and we assert on the event stream the class emits.
*/
import { describe, it, expect, afterEach } from 'bun:test';
import { chmodSync, mkdirSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import {
ChildWorkerSupervisor,
type ChildSupervisorEvent,
} from '../src/core/minions/child-worker-supervisor.ts';
interface Harness {
workerScript: string;
cleanup: () => void;
}
function makeHarness(name: string, body: string): Harness {
const root = join(tmpdir(), `gbrain-cws-test-${name}-${process.pid}-${Date.now()}`);
mkdirSync(root, { recursive: true });
const workerScript = join(root, 'worker.sh');
writeFileSync(workerScript, `#!/bin/sh\n${body}\n`, 'utf8');
chmodSync(workerScript, 0o755);
return {
workerScript,
cleanup: () => {
try {
rmSync(root, { recursive: true, force: true });
} catch {
/* noop */
}
},
};
}
interface RunResult {
events: ChildSupervisorEvent[];
maxCrashesFired: { count: number; max: number } | null;
}
async function runUntilTerminal(
h: Harness,
overrides: Partial<{
maxCrashes: number;
_backoffFloorMs: number;
cleanRestartBudget: number;
cleanRestartWindowMs: number;
cleanRestartBudgetBackoffMs: number;
stableRunResetMs: number;
_now: () => number;
stopAfterEvents: number; // safety net so a buggy test can't hang
}>,
): Promise<RunResult> {
const events: ChildSupervisorEvent[] = [];
let stopping = false;
let maxCrashesFired: { count: number; max: number } | null = null;
const stopAfter = overrides.stopAfterEvents ?? 200;
const sup = new ChildWorkerSupervisor({
cliPath: h.workerScript,
args: [],
maxCrashes: overrides.maxCrashes ?? 3,
_backoffFloorMs: overrides._backoffFloorMs ?? 5,
cleanRestartBudget: overrides.cleanRestartBudget,
cleanRestartWindowMs: overrides.cleanRestartWindowMs,
cleanRestartBudgetBackoffMs: overrides.cleanRestartBudgetBackoffMs,
stableRunResetMs: overrides.stableRunResetMs,
_now: overrides._now,
isStopping: () => stopping,
onMaxCrashesExceeded: (count, max) => {
maxCrashesFired = { count, max };
stopping = true;
},
onEvent: (event) => {
events.push(event);
if (events.length >= stopAfter) {
stopping = true;
}
},
});
await sup.run();
return { events, maxCrashesFired };
}
afterEach(() => {
/* per-test harness.cleanup() runs in finally blocks below */
});
describe('ChildWorkerSupervisor', () => {
describe('D1 — code=0 exit classifier', () => {
it('code=0 worker exit does not count as crash; restarts immediately', async () => {
const h = makeHarness('clean-exits', 'exit 0');
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3,
stopAfterEvents: 30, // ~10 spawn/exit/backoff trios
});
expect(res.maxCrashesFired).toBeNull();
const exits = res.events.filter((e) => e.kind === 'worker_exited');
expect(exits.length).toBeGreaterThanOrEqual(3);
for (const e of exits) {
if (e.kind === 'worker_exited') {
expect(e.code).toBe(0);
expect(e.likelyCause).toBe('clean_exit');
// crashCount stays at 0 across every clean exit
expect(e.crashCount).toBe(0);
}
}
const backoffs = res.events.filter((e) => e.kind === 'backoff');
expect(backoffs.length).toBeGreaterThanOrEqual(1);
// Within the default 10-restart budget, all backoffs are ms:0 / clean_exit
for (const e of backoffs) {
if (e.kind === 'backoff') {
// Once we cross the 10-restart budget the reason flips to
// budget_exceeded, but until then they're all clean_exit ms:0.
if (e.reason === 'clean_exit') {
expect(e.ms).toBe(0);
expect(e.crashCount).toBe(0);
}
}
}
} finally {
h.cleanup();
}
});
it('interleaved code=0 and code!=0 exits still trip max_crashes', async () => {
// Worker alternates: each invocation increments a counter file and
// exits 1 on odd hits, 0 on even hits (so exit-sequence is 1,0,1,0,1).
const h = makeHarness(
'interleaved',
`
COUNTER_FILE="$(dirname "$0")/counter"
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
COUNT=$(cat "$COUNTER_FILE")
NEXT=$((COUNT + 1))
echo "$NEXT" > "$COUNTER_FILE"
# Odd-indexed runs (#1, #3, #5...) exit 1; even-indexed exit 0.
if [ $((NEXT % 2)) -eq 1 ]; then exit 1; else exit 0; fi
`,
);
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3,
_backoffFloorMs: 5,
stopAfterEvents: 200,
});
expect(res.maxCrashesFired).not.toBeNull();
// 3 code!=0 exits → max_crashes=3
expect(res.maxCrashesFired!.count).toBe(3);
const exits = res.events.filter((e) => e.kind === 'worker_exited');
// Should be exactly 5 exits: 1, 0, 1, 0, 1 — then max fires.
const codes = exits
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
.map((e) => e.code);
expect(codes).toEqual([1, 0, 1, 0, 1]);
const backoffs = res.events
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff');
// Backoffs only fire between iterations 1-4 (not after the 5th, since
// the loop bails out via onMaxCrashesExceeded before applyBackoff).
// Even-index exits (code=0, indices 1+3) → reason='clean_exit'.
// Odd-index exits (code=1, indices 0+2) → reason='crash'.
const reasons = backoffs.map((e) => e.reason);
expect(reasons).toEqual(['crash', 'clean_exit', 'crash', 'clean_exit']);
} finally {
h.cleanup();
}
});
it('code=0 after stable 5min+ run does not reset crashCount', async () => {
// Sequence (4 runs total): exit 1 → exit 0 (6 min, "stable") → exit 1 →
// exit 1. crashCount progression: 1, 1 (unchanged across the long
// clean exit), 2, 3 — last one trips max_crashes=3.
const h = makeHarness(
'stable-clean-no-reset',
`
COUNTER_FILE="$(dirname "$0")/counter"
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
COUNT=$(cat "$COUNTER_FILE")
NEXT=$((COUNT + 1))
echo "$NEXT" > "$COUNTER_FILE"
case $NEXT in
1) exit 1 ;;
2) exit 0 ;;
3) exit 1 ;;
4) exit 1 ;;
*) exit 0 ;;
esac
`,
);
try {
// Fake clock — each spawnOnce reads now() twice (start + exit) and
// applyBackoff may read once more. Run 2 sees a 6-minute duration
// (stable-run reset would fire IF the exit were code!=0 — we assert
// it does NOT fire when the exit is clean).
const SIX_MIN = 6 * 60_000;
const timestamps = [
0, // run 1 start
1_000, // run 1 exit (+1s) → crashCount 1
1_000, // run 2 start
1_000 + SIX_MIN, // run 2 exit (+6min) → code=0, stays at 1
1_000 + SIX_MIN, // run 3 start
1_000 + SIX_MIN + 1_000, // run 3 exit (+1s) → crashCount 2
1_000 + SIX_MIN + 1_000, // run 4 start
1_000 + SIX_MIN + 2_000, // run 4 exit (+1s) → crashCount 3, trips max
];
let idx = 0;
const last = timestamps[timestamps.length - 1];
const fakeNow = () => {
if (idx < timestamps.length) {
return timestamps[idx++];
}
return last + (idx++ - timestamps.length + 1) * 100;
};
const res = await runUntilTerminal(h, {
maxCrashes: 3,
_backoffFloorMs: 5,
_now: fakeNow,
stopAfterEvents: 200,
});
expect(res.maxCrashesFired).not.toBeNull();
expect(res.maxCrashesFired!.count).toBe(3);
const exits = res.events
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
.map((e) => ({ code: e.code, crashCount: e.crashCount, runDurationMs: e.runDurationMs }));
expect(exits.length).toBeGreaterThanOrEqual(4);
expect(exits[0]).toMatchObject({ code: 1, crashCount: 1 });
expect(exits[1]).toMatchObject({ code: 0, crashCount: 1 }); // D1: unchanged
expect(exits[2]).toMatchObject({ code: 1, crashCount: 2 });
expect(exits[3]).toMatchObject({ code: 1, crashCount: 3 });
// Run 2 ran 6min, but because exit code was 0 the stable-run reset
// branch did NOT fire — crashCount stayed at 1. This is the core
// D1 invariant: clean exits never reset crashCount, even stable ones.
expect(exits[1].runDurationMs).toBe(SIX_MIN);
} finally {
h.cleanup();
}
});
});
describe('D2 — clean-restart budget', () => {
it('budget exceeded triggers health_warn + budget_exceeded backoff', async () => {
// Tight budget of 2 so we trip it on the 3rd clean exit.
const h = makeHarness('budget-trip', 'exit 0');
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3, // never trips because code=0 doesn't increment
_backoffFloorMs: 5,
cleanRestartBudget: 2,
cleanRestartWindowMs: 60_000,
cleanRestartBudgetBackoffMs: 10,
stopAfterEvents: 25,
});
const healthWarns = res.events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> => e.kind === 'health_warn',
);
// Once tripped, every subsequent clean exit re-fires health_warn
// (the sliding window stays full at our test rate).
expect(healthWarns.length).toBeGreaterThan(0);
for (const w of healthWarns) {
expect(w.reason).toBe('clean_restart_budget_exceeded');
expect(w.windowMs).toBe(60_000);
expect(w.count).toBeGreaterThan(2);
}
const backoffReasons = res.events
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff')
.map((e) => e.reason);
// First 2 exits are within budget → reason='clean_exit'.
// From the 3rd exit onward → reason='budget_exceeded'.
expect(backoffReasons.slice(0, 2)).toEqual(['clean_exit', 'clean_exit']);
expect(backoffReasons.slice(2).every((r) => r === 'budget_exceeded')).toBe(true);
} finally {
h.cleanup();
}
});
it('budget config is per-instance (no module-level state leakage)', async () => {
// Run instance A with budget=2 and instance B with budget=5. Each
// tracks its own sliding window; A trips faster than B.
const hA = makeHarness('budget-a', 'exit 0');
const hB = makeHarness('budget-b', 'exit 0');
try {
const resA = await runUntilTerminal(hA, {
maxCrashes: 99,
_backoffFloorMs: 5,
cleanRestartBudget: 2,
cleanRestartBudgetBackoffMs: 5,
stopAfterEvents: 12,
});
const resB = await runUntilTerminal(hB, {
maxCrashes: 99,
_backoffFloorMs: 5,
cleanRestartBudget: 5,
cleanRestartBudgetBackoffMs: 5,
stopAfterEvents: 18,
});
const firstTripA = resA.events.findIndex(
(e) => e.kind === 'health_warn',
);
const firstTripB = resB.events.findIndex(
(e) => e.kind === 'health_warn',
);
expect(firstTripA).toBeGreaterThan(-1);
expect(firstTripB).toBeGreaterThan(-1);
// B's budget is more generous → its first health_warn appears later
// in the event stream (after more spawn/exit pairs).
expect(firstTripB).toBeGreaterThan(firstTripA);
} finally {
hA.cleanup();
hB.cleanup();
}
});
});
describe('awaitChildExit short-circuit (P2 review fix)', () => {
// Regression: pre-fix the method registered child.once('exit', ...) AFTER
// child.exitCode was already populated, so a child that drained quickly
// between killChild('SIGTERM') and awaitChildExit() would never resolve
// and the caller waited out the full timeout. Fix probes exitCode +
// signalCode first and short-circuits.
it('resolves immediately when the child has already exited', async () => {
const h = makeHarness('await-already-exited', 'exit 0');
try {
// Spin up a supervisor; drive it for ONE spawn cycle and then stop.
const events: ChildSupervisorEvent[] = [];
let stopping = false;
const sup = new ChildWorkerSupervisor({
cliPath: h.workerScript,
args: [],
maxCrashes: 1,
_backoffFloorMs: 1,
isStopping: () => stopping,
onMaxCrashesExceeded: () => { stopping = true; },
onEvent: (e) => {
events.push(e);
if (e.kind === 'worker_exited') stopping = true;
},
});
await sup.run();
// After run() returns, the child has exited; awaitChildExit on an
// already-finished cycle MUST resolve in well under the timeout.
const start = Date.now();
await sup.awaitChildExit(5_000);
const elapsed = Date.now() - start;
expect(elapsed).toBeLessThan(200);
} finally {
h.cleanup();
}
});
});
describe('event shape', () => {
it('worker_spawned + worker_exited fire on every cycle with consistent shape', async () => {
const h = makeHarness('shape', 'exit 0');
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3,
_backoffFloorMs: 5,
stopAfterEvents: 9, // 3 spawn-exit-backoff triples
});
const spawned = res.events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_spawned' }> => e.kind === 'worker_spawned',
);
const exited = res.events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited',
);
expect(spawned.length).toBeGreaterThanOrEqual(2);
expect(exited.length).toBe(spawned.length);
for (const s of spawned) {
expect(typeof s.pid).toBe('number');
expect(s.pid).toBeGreaterThan(0);
expect(typeof s.tini).toBe('boolean');
}
for (const e of exited) {
expect(e.code).toBe(0);
expect(e.signal).toBeNull();
expect(typeof e.runDurationMs).toBe('number');
expect(e.likelyCause).toBe('clean_exit');
}
} finally {
h.cleanup();
}
});
});
});
+19 -5
View File
@@ -282,7 +282,12 @@ describe('MinionSupervisor', () => {
const outFile = join(tmpdir(), `gbrain-sup-env-${process.pid}-${Date.now()}.txt`);
try { unlinkSync(outFile); } catch { /* may not exist */ }
const h = makeHarness('env-strip-outfile', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
// Worker writes env to OUT_FILE then exits 1. exit=1 is required (not
// exit=0) because post-D1/D2 (v0.33) clean exits don't count toward
// crashCount — the supervisor would respawn forever. The test's
// assertion is on the OUT_FILE contents (env plumbing), not the
// exit code, so any non-zero code that trips SUP_MAX_CRASHES=1 works.
const h = makeHarness('env-strip-outfile', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 1`);
try {
const sup = spawnSupervisor(h, {
@@ -308,7 +313,9 @@ describe('MinionSupervisor', () => {
const outFile = join(tmpdir(), `gbrain-sup-env-ok-${process.pid}-${Date.now()}.txt`);
try { unlinkSync(outFile); } catch { /* may not exist */ }
const h = makeHarness('env-pass-on-opt-in', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
// Worker exits 1 (not 0) so SUP_MAX_CRASHES=1 actually trips. See
// the comment on the env-strip test above for the v0.33 rationale.
const h = makeHarness('env-pass-on-opt-in', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 1`);
try {
const sup = spawnSupervisor(h, {
@@ -333,7 +340,9 @@ describe('MinionSupervisor', () => {
const outFile = join(tmpdir(), `gbrain-sup-supervised-${process.pid}-${Date.now()}.txt`);
try { unlinkSync(outFile); } catch { /* may not exist */ }
const h = makeHarness('supervised-env', `printf '%s\n' "\${GBRAIN_SUPERVISED-UNSET}" > "$OUT_FILE" ; exit 0`);
// exit 1 required post-D1/D2 to trip SUP_MAX_CRASHES=1; clean exits
// no longer count toward the crash limit.
const h = makeHarness('supervised-env', `printf '%s\n' "\${GBRAIN_SUPERVISED-UNSET}" > "$OUT_FILE" ; exit 1`);
try {
const sup = spawnSupervisor(h, {
@@ -372,7 +381,11 @@ describe('MinionSupervisor', () => {
// window before max-crashes fires. We assert the basic completion path
// and let CI's wall-clock detect any pathological CPU spike.
it('completes a normal supervise lifecycle with healthInterval=0', async () => {
const h = makeHarness('health-interval-zero', 'exit 0');
// exit 1 (not exit 0) because post-D1/D2 (v0.33) clean exits don't
// count toward max_crashes — a code=0 worker would respawn forever.
// The test's purpose is regression coverage that healthInterval=0
// disables the timer; the exit code doesn't matter to that assertion.
const h = makeHarness('health-interval-zero', 'exit 1');
try {
const sup = spawnSupervisor(h, {
@@ -409,7 +422,8 @@ describe('MinionSupervisor', () => {
// Worker logs its argv to OUT_FILE so the test can assert --max-rss 2048
// landed there. spawnOnce in supervisor.ts builds:
// ['jobs', 'work', '--concurrency', '1', '--queue', 'default', '--max-rss', '2048']
const h = makeHarness('maxrss-default', `printf '%s\\n' "$*" > "$OUT_FILE" ; exit 0`);
// exit 1 required post-D1/D2: code=0 workers respawn forever.
const h = makeHarness('maxrss-default', `printf '%s\\n' "$*" > "$OUT_FILE" ; exit 1`);
try {
const sup = spawnSupervisor(h, {
+125
View File
@@ -0,0 +1,125 @@
/**
* Tests for the per-process RSS reader used by the worker's leak watchdog.
*
* `process.memoryUsage().rss` returns VmRSS which counts file-backed mmap
* pages (e.g. git packfiles). On a 96K-page brain repo, git operations
* inflate VmRSS to 7GB+ while heap is ~100MB — the kernel will reclaim
* those pages under pressure so they should not count toward the watchdog
* threshold. The fix reads `/proc/self/status` for RssAnon + RssShmem
* (the non-file-backed pages) on Linux and falls back to VmRSS elsewhere.
*
* These tests pin the parser shape against the M1 regression Codex
* surfaced during eng review: the prior `if (anonKb > 0)` form conflated
* "field missing" with "field present but zero", which broke the
* shmem-only worker case.
*/
import { describe, it, expect } from 'bun:test';
import { getAccurateRss, parseRssFromProcStatus } from '../src/core/minions/worker.ts';
describe('parseRssFromProcStatus', () => {
it('parses normal RssAnon + RssShmem to bytes', () => {
const status = [
'Name:\tworker',
'VmRSS:\t1024 kB',
'RssAnon:\t1024 kB',
'RssShmem:\t0 kB',
'VmSize:\t8192 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(1024 * 1024);
});
it('M1 regression: RssAnon:0 + RssShmem>0 returns shmem-only bytes (not null)', () => {
// The pre-fix code did `if (anonKb > 0) return ...` which would treat
// this case as "field missing" and fall through to VmRSS. The fix uses
// field-presence checks so anon=0 + shmem=512 yields 512 KiB → 524_288.
const status = [
'RssAnon:\t0 kB',
'RssShmem:\t512 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(512 * 1024);
});
it('returns null when neither RssAnon nor RssShmem is present (old kernel)', () => {
// Linux kernels older than 4.5 don't expose these fields. Returning
// null signals "fall back to VmRSS" to the caller.
const status = [
'Name:\tworker',
'VmRSS:\t1024 kB',
'VmSize:\t8192 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBeNull();
});
it('treats non-numeric fields as absent (regex matches digits only)', () => {
// The `\d+` regex won't match "notanumber", so RssAnon is treated as
// absent and the result reflects only the well-formed RssShmem field.
// This is the right behavior: a corrupt RssAnon line shouldn't
// poison an otherwise valid RssShmem reading.
const status = [
'RssAnon:\tnotanumber kB',
'RssShmem:\t0 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(0);
});
it('returns null when ALL fields are non-numeric', () => {
const status = [
'RssAnon:\tnotanumber kB',
'RssShmem:\talsobad kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBeNull();
});
it('returns sum when only RssShmem is present (anon missing)', () => {
// Symmetric to the regression: if only RssShmem is exposed for some
// reason, we should still use it. Treats anonKb default as 0.
const status = [
'RssShmem:\t256 kB',
'VmRSS:\t9999 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(256 * 1024);
});
it('M1 regression: explicit RssAnon:0 + RssShmem:0 returns 0 (not null)', () => {
// Both fields present, both zero is a valid reading for a near-empty
// worker process. Should be treated as 0 bytes, NOT a missing reading.
const status = [
'RssAnon:\t0 kB',
'RssShmem:\t0 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(0);
});
});
describe('getAccurateRss', () => {
it('uses the parsed value when readStatus returns a valid /proc/self/status', () => {
const fakeStatus = 'RssAnon:\t2048 kB\nRssShmem:\t0 kB\n';
expect(getAccurateRss(() => fakeStatus)).toBe(2048 * 1024);
});
it('falls back to process.memoryUsage().rss when readStatus throws (non-Linux)', () => {
const result = getAccurateRss(() => {
throw Object.assign(new Error('ENOENT: no such file'), { code: 'ENOENT' });
});
// We can't assert the exact RSS value cross-platform, but it MUST be
// a positive integer (process.memoryUsage().rss is always set).
expect(result).toBeGreaterThan(0);
expect(Number.isInteger(result)).toBe(true);
});
it('falls back when status text has no RssAnon/RssShmem fields (old kernel)', () => {
const status = 'VmRSS:\t1024 kB\nVmSize:\t8192 kB\n';
const result = getAccurateRss(() => status);
// Falls back to process.memoryUsage().rss — a positive integer.
expect(result).toBeGreaterThan(0);
expect(Number.isInteger(result)).toBe(true);
});
it('falls back on malformed values rather than returning NaN', () => {
const status = 'RssAnon:\tNaN kB\n';
const result = getAccurateRss(() => status);
expect(Number.isNaN(result)).toBe(false);
expect(result).toBeGreaterThan(0);
});
});