mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.34.3.0 fix: supervisor treats code=0 watchdog exits as crashes (#1003)
* fix: supervisor treats code=0 watchdog exits as crashes
The RSS watchdog triggers gracefulShutdown() which exits with code 0.
The supervisor was counting ALL exits < 5min as crashes, including
clean code=0 exits. After 10 watchdog-triggered restarts (typical with
a 96K-page brain where autopilot inflates RSS), the supervisor gave up
with max_crashes_exceeded.
Fix: code=0 exits reset crashCount to 0 and restart immediately with
no backoff. Only code≠0 exits count toward the crash limit.
Root cause: process.memoryUsage().rss reports 7GB during autopilot
sync on large repos (possibly shared page inflation from git mmap).
The 4096MB threshold triggers on every cycle. This is a separate
issue (RSS measurement accuracy) but the supervisor should handle
clean exits regardless.
* fix: use RssAnon instead of VmRSS for watchdog threshold
process.memoryUsage().rss returns VmRSS which includes file-backed
mmap'd pages. On repos with large git packfiles (96K+ pages), git
operations inflate VmRSS to 7GB+ while actual heap usage is ~100MB.
The kernel reclaims these pages under memory pressure — they're cache.
Replace with /proc/self/status RssAnon + RssShmem which measures only
anonymous pages (heap, stack, anonymous mmap). This is the memory that
actually matters for OOM risk.
Falls back to process.memoryUsage().rss on non-Linux.
Before: watchdog triggers every autopilot cycle (7GB VmRSS > 4GB threshold)
After: watchdog only triggers on real memory growth (~100MB << 4GB threshold)
Related: #1002 (supervisor crash-count fix for the same symptom)
* refactor(minions): extract ChildWorkerSupervisor with D1/D2 amendments
MinionSupervisor and src/commands/autopilot.ts each owned a separate
spawn-and-respawn loop. PR #1003 fixed the supervisor's crash-counter
bug (counting code=0 watchdog drains as crashes) but the autopilot
loop has the same bug class. Worse, the as-shipped #1003 fix reset
crashCount=0 on every code=0 exit, which lost the "flapping worker"
signal in mixed-exit sequences.
Extract the shared spawn loop into ChildWorkerSupervisor so both
consumers compose one tested core. The new class bakes in two
amendments resolved during plan-eng-review:
D1 (lastExitCode track): code=0 exits no longer touch crashCount.
They emit ms:0 backoff and restart immediately, but the counter
survives across them. A worker alternating exit 1 / exit 0 / exit 1
correctly trips max_crashes; a worker drained 100 times by the
watchdog stays at crashCount=0 and runs forever (also correct).
D2 (clean-restart budget): on platforms where the watchdog measures
VmRSS instead of RssAnon (macOS, kernel <4.5, restricted containers),
a perpetually over-threshold worker could clean-exit in a tight loop
with no observability. New `cleanRestartBudget` option (default 10
clean restarts per 60s window) emits a `health_warn` and applies
backoff once exceeded.
The supervisor now delegates spawn/respawn/backoff to the inner
class and maps ChildSupervisorEvent → existing SupervisorEvent
emit() channel so JSONL audit consumers see byte-compatible output.
PID lock, signal handlers, health check, and process.exit on
max-crashes stay in MinionSupervisor (those are standalone-daemon
concerns the autopilot composer doesn't need).
Tests: 6 new ChildWorkerSupervisor cases (D1 classifier, interleaved
exits, stable-run + clean-exit interaction, D2 budget tripping, per-
instance config isolation, event shape regression). Existing supervisor
tests updated to use exit-1 workers where they previously relied on
clean-exit-as-crash semantics; their assertions (env plumbing, PID
lock, audit shape) are unaffected.
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(autopilot): compose ChildWorkerSupervisor instead of inline spawn loop
src/commands/autopilot.ts:165-197 used to have its own spawn-and-
respawn loop separate from MinionSupervisor's. It hardcoded
maxCrashes=5, fixed 10s backoff, and counted every exit (including
code=0) toward the crash limit. Codex flagged this during plan-eng
review: the parallel implementation had the same bug class fixed
in #1003, just on a different code path. Anyone running
`gbrain autopilot` as a long-running daemon (instead of
`gbrain jobs supervisor`) would hit it.
Replace the inline `startWorker` + `child.on('exit')` block with
a ChildWorkerSupervisor instance. Drops the parallel `crashCount`,
`lastWorkerStartTime`, and `STABLE_RUN_RESET_MS` state. The
ChildWorkerSupervisor's D1 lastExitCode track + D2 clean-restart
budget apply to autopilot for free.
Shutdown now drains via the supervisor's killChild + awaitChildExit
typed surface instead of reaching into `workerProc` directly. The
onMaxCrashesExceeded callback routes through autopilot's existing
shutdown('max_crashes') path so the lockfile gets cleaned up
(pre-refactor, the inline loop called process.exit(1) directly and
bypassed the cleanup).
Regression coverage in test/autopilot-supervisor-wiring.test.ts:
static-shape grep guards for `--max-rss 2048`, `maxCrashes: 5`,
the shutdown-via-callback wiring, and absence of the legacy inline
names (startWorker, workerProc, crashCount, lastWorkerStartTime,
STABLE_RUN_RESET_MS).
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): parse RssAnon as field-presence + soften OOM docstring
Two follow-ups to the RssAnon watchdog fix (b81c598f), both surfaced
during plan-eng-review by Codex.
M1: getAccurateRss() used `if (anonKb > 0) return ...` to decide
whether to use the /proc/self/status reading or fall back to
process.memoryUsage().rss. That conflated "RssAnon field missing"
(old kernel, non-Linux) with "RssAnon field present but zero" (a
near-empty worker process whose only memory is shmem). The legitimate
shmem-only worker case fell through to VmRSS even though /proc had a
valid reading.
Fix: split the pure parser (parseRssFromProcStatus) into a separate
exported function that checks field presence via regex match, not
value comparison. Returns null only when the field text doesn't
match `^RssAnon:\s+(\d+)` AND `^RssShmem:\s+(\d+)`. Both fields
present + both zero is now a valid reading of 0 bytes.
M2: the docstring claimed RssAnon + RssShmem was "the memory that
actually matters for OOM risk." Codex pushed back: this is correct
for per-process leak detection but NOT a full container-OOM metric,
because cgroup memory pressure includes page cache. Soften to
"non-file-backed resident memory used for per-process leak
detection" and call out the cgroup caveat explicitly.
getAccurateRss now takes an optional readStatus function for
testability. Production callers use the default; tests inject
canned status text to cover the M1 regression and the fallback paths
without mocking the filesystem.
Tests: 11 cases covering parseRssFromProcStatus (normal, M1 regression
with anon=0 + shmem>0, both-zero, missing fields, malformed values,
shmem-only) and getAccurateRss (injected reader, ENOENT fallback,
old-kernel fallback, malformed-value fallback).
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(minions): awaitChildExit short-circuits when child already exited
Pre-fix, awaitChildExit registered `child.once('exit', ...)` without
checking whether the child had already terminated. If the child drained
between killChild('SIGTERM') and awaitChildExit() — common on fast
SIGTERM responders — Node's 'exit' event had already fired, the late
listener never resolved, and the caller waited out the full timeout.
On the supervisor's clean shutdown path that's a 35-second hang on
every quick child.
Probe `child.exitCode` and `child.signalCode` first; resolve
immediately when either is non-null. Sub-second clean shutdown
restored.
Pre-existing in the legacy supervisor.ts shape (same bug pattern),
but since the refactor consolidates child-process management into one
class, fix the pattern at the new seam.
Regression test in test/child-worker-supervisor.test.ts: run one full
spawn cycle, then call awaitChildExit on the already-finished cycle
and assert it returns in under 200ms (well under any test timeout).
Surfaced during pre-landing /review on the fix wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.34.3.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: update CLAUDE.md key-files entries for v0.34.3.0
Reflects the ChildWorkerSupervisor extraction shipped in this branch:
- Add new entry for src/core/minions/child-worker-supervisor.ts
covering D1 lastExitCode classifier, D2 clean-restart budget, the
awaitChildExit short-circuit, and test pinning at
test/child-worker-supervisor.test.ts
- Update src/core/minions/supervisor.ts entry to note the spawn-loop
extraction into the shared core + the byte-compatible event-shape
mapping that preserves JSONL audit consumers
- Update src/commands/autopilot.ts entry to note the parallel-
supervisor elimination + the shutdown-via-callback wiring
- Update src/core/minions/worker.ts entry with the new RssAnon /
getAccurateRss exports + the M1 field-presence parser fix
Regenerated llms-full.txt to match (per project rule: every CLAUDE.md
edit must be followed by bun run build:llms).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Wintermute
Claude Opus 4.7
parent
3325b405bb
commit
668254b05e
@@ -2,6 +2,48 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.34.3.0] - 2026-05-14
|
||||
|
||||
**Supervisor stops crashing every 24h on large brains. Watchdog stops false-firing on git mmap.**
|
||||
**One shared spawn-loop replaces two duplicate copies that drifted into the same bug class.**
|
||||
|
||||
The Minions supervisor was committing suicide every 24h on a 96K-page brain. Every autopilot cycle, `process.memoryUsage().rss` reported ~7GB because VmRSS counts file-backed mmap pages and git holds packfiles open. The 2GB RSS watchdog fired, the worker drained cleanly, exited code=0, and the supervisor counted that clean exit as a crash. 10 clean drains in 24h tripped `max_crashes_exceeded` and the supervisor process gave up. External cron restarted it. Loop repeated.
|
||||
|
||||
This release fixes the chain at every layer. The worker now reads RssAnon + RssShmem from `/proc/self/status` on Linux (the actual heap, ~100MB during sync on the same brain) and falls back to VmRSS only on macOS / restricted containers / kernels older than 4.5. The supervisor's exit classifier no longer collapses watchdog drains and real crashes into the same `code=0 vs code≠0` distinction. Clean exits leave `crashCount` untouched (so a worker alternating real-crash + watchdog still trips max_crashes), but they restart immediately with no backoff. A new `cleanRestartBudget` (default 10 restarts per 60s) catches the macOS fallback case where the watchdog still false-fires every cycle.
|
||||
|
||||
Underneath that, the supervisor and `gbrain autopilot` no longer maintain two parallel spawn-and-respawn loops with different bug classes. Both now compose a shared `ChildWorkerSupervisor` core. Fix the spawn loop once, both consumers benefit.
|
||||
|
||||
### What ships in v0.34.3.0
|
||||
|
||||
- **Worker RSS now reads RssAnon + RssShmem on Linux.** `process.memoryUsage().rss` returns VmRSS which includes file-backed mmap pages, so git packfiles inflated the reading to 7GB on large brains. The new helper reads `/proc/self/status` for the non-file-backed pages — the metric a leak watchdog actually wants. Fallback to VmRSS on macOS / kernel <4.5 / missing `/proc`.
|
||||
- **Supervisor preserves flap detection across mixed exits.** code=0 exits no longer reset `crashCount` to 0. A worker that alternates real crashes with watchdog drains still trips `max_crashes_exceeded`; a worker that only ever drains cleanly runs forever.
|
||||
- **Clean-restart budget caps the macOS-fallback tight-loop.** If 10 code=0 exits happen inside a 60s window, the supervisor emits `health_warn` with reason `clean_restart_budget_exceeded` and applies backoff. Operators get observability instead of a silent restart loop.
|
||||
- **`ChildWorkerSupervisor` consolidates the spawn loop.** New module at `src/core/minions/child-worker-supervisor.ts` is the single source of truth for child-process supervision. `MinionSupervisor` and `gbrain autopilot` both compose it. No more parallel-implementation drift.
|
||||
- **Parser field-presence fix.** `RssAnon: 0` with `RssShmem: 512` (shmem-only workers) now parses correctly. Pre-fix it fell through to VmRSS.
|
||||
- **`awaitChildExit` short-circuit.** Fast SIGTERM responders no longer cause a 35-second hang on clean shutdown — the method now probes `child.exitCode` before registering an exit listener.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/core/minions/worker.ts` — `getAccurateRss()` exported with injectable status reader; `parseRssFromProcStatus()` exported for unit tests. Docstring softened to call out the cgroup-OOM caveat.
|
||||
- `src/core/minions/child-worker-supervisor.ts` — new file. Pure spawn-and-respawn core. No PID file, no signal handlers, no `process.exit`, no health check. Emits lifecycle events via injected callback.
|
||||
- `src/core/minions/supervisor.ts` — refactored to compose `ChildWorkerSupervisor`. The pre-shipped reset-to-0 hunk is gone; D1 lastExitCode tracking + D2 budget live in the shared core. PID lock, signal handlers, health check, and `process.exit` on max-crashes stay.
|
||||
- `src/commands/autopilot.ts` — replaced inline spawn-and-respawn loop (lines 165-197) with `ChildWorkerSupervisor` composition. `onMaxCrashesExceeded` routes through `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor it called `process.exit(1)` directly and bypassed cleanup).
|
||||
- Tests: `test/child-worker-supervisor.test.ts` (NEW, 7 cases) covers D1 classifier + D2 budget + the awaitChildExit short-circuit regression. `test/worker-rss.test.ts` (NEW, 11 cases) covers the M1 parser fix and fallback paths. `test/autopilot-supervisor-wiring.test.ts` (NEW, 6 cases) static-shape regression guards. `test/supervisor.test.ts` updated to exit-1 in tests that previously relied on clean-exit-as-crash semantics.
|
||||
|
||||
### For contributors
|
||||
|
||||
- The `ChildWorkerSupervisor` class is the seam to compose against when adding a new long-running child-process supervisor surface. Don't roll a third spawn-and-respawn loop — compose this one.
|
||||
- The `_now` injection point on `ChildWorkerSupervisor` lets tests fake the clock without `mock.module`. The stable-run reset behavior is pinned by a deterministic test that fakes a 6-minute clean exit between two real crashes.
|
||||
|
||||
## To take advantage of v0.34.3.0
|
||||
|
||||
`gbrain upgrade` carries the whole wave automatically. No migrations. No config flips. If your supervisor was crashing every 24h, watch the logs for 24h after the upgrade and confirm:
|
||||
|
||||
- `worker_exited` events have `likely_cause: 'clean_exit'` (not `unknown` or `runtime_error`)
|
||||
- `backoff` events have `reason: 'clean_exit'` and `ms: 0`
|
||||
- Zero `max_crashes_exceeded` events
|
||||
|
||||
If `health_warn` with `reason: 'clean_restart_budget_exceeded'` starts firing on your deployment, that's the new D2 budget telling you the watchdog is firing too often. On Linux that means the worker really is leaking; investigate. On macOS that's the VmRSS-fallback fire pattern — set a higher `--max-rss` threshold to suppress.
|
||||
## [0.34.2.0] - 2026-05-14
|
||||
|
||||
**Path-based import checkpoint. The newest files in a partial import no longer disappear.**
|
||||
|
||||
@@ -130,8 +130,9 @@ strict behavior when unset.
|
||||
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`). **v0.34.3.0:** RSS watchdog metric switched to non-file-backed pages on Linux. New exports `parseRssFromProcStatus(status)` (pure parser, exported for unit tests) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5). The default `getRss` injected into `WorkerOpts` is now `getAccurateRss` instead of `process.memoryUsage().rss`. Closes the prod incident where VmRSS inflated to 7GB on a 96K-page brain (file-backed git packfile mmaps) while heap stayed at ~100MB; the watchdog was firing every autopilot cycle. M1 parser fix uses field-presence regex checks so `RssAnon: 0 + RssShmem: 512` (shmem-only worker case) parses correctly instead of falling through to VmRSS. Pinned by `test/worker-rss.test.ts` (11 cases).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests. **v0.34.3.0:** spawn-and-respawn loop extracted into the shared `ChildWorkerSupervisor` core (see entry below). MinionSupervisor now composes the inner class via `runSuperviseLoop()` → `new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` shapes back through the existing `emit()` SupervisorEvent channel — JSONL audit consumers see byte-compatible output across the rename. PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor (standalone-daemon concerns). The pre-shipped reset-to-0-on-code=0 hunk that originally fixed the prod crash-counter incident is gone; the same fix lives in the shared core under the D1 amendment (code=0 leaves `crashCount` untouched, so a worker alternating real crashes + watchdog drains still trips `max_crashes`). D2 `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop by emitting `health_warn { reason: 'clean_restart_budget_exceeded' }` plus backoff after the threshold trips. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of reaching into `this.child` directly. Pinned by `test/supervisor.test.ts` (16 cases; existing tests that previously relied on clean-exit-as-crash semantics now use exit-1 workers since clean exits no longer count) and `test/supervisor-tini.test.ts`.
|
||||
- `src/core/minions/child-worker-supervisor.ts` (v0.34.3.0) — shared spawn-and-respawn core extracted from `MinionSupervisor` so it can be reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon). Pre-v0.34.3.0 the two consumers maintained parallel spawn loops that drifted into the same bug class — Codex caught it during plan-eng-review on PR #1003. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void` callback so each composer routes to its own log/audit channel. **D1 exit classifier:** `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences — a worker that alternates `exit 1 / exit 0 / exit 1 / exit 0` correctly trips `max_crashes` after 10 real crashes regardless of intervening clean exits). `code != 0` follows the existing `runDuration > stableRunResetMs ? 1 : ++crashCount` rule. **D2 clean-restart budget:** sliding window tracks code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s) before the next spawn. Caps the worst-case tight-loop on macOS / restricted containers / kernel <4.5 where the worker's RSS watchdog falls back to VmRSS. Public read-only accessors `childAlive`, `inBackoff`, `crashCount` for composer health checks; `killChild(signal)` + `awaitChildExit(timeoutMs)` for shutdown paths. `awaitChildExit` short-circuits when `child.exitCode !== null || child.signalCode !== null` (regression caught in pre-landing /review: pre-fix, fast-SIGTERM responders caused a 35-second shutdown hang because the late `once('exit', ...)` listener never fired). Test hooks: `_backoffFloorMs` skips the real backoff curve, `_now` injects a fake clock. Pinned by `test/child-worker-supervisor.test.ts` (7 cases: D1 classifier with code=0 not counted, interleaved exits still trip max_crashes, stable-run + clean-exit interaction across faked 6-minute run, D2 budget triggers backoff + health_warn, budget config is per-instance, awaitChildExit short-circuit, event-shape regression). Plan that produced the design lives at `~/.claude/plans/this-is-a-real-sleepy-sketch.md`.
|
||||
- `src/core/minions/spawn-helpers.ts` (v0.28.1) — pure `detectTini()` + `buildSpawnInvocation()` helpers consumed by both `supervisor.ts` and `autopilot.ts`. Resolves the DRY violation between the two spawn sites and makes the tini wrapping testable without `mock.module()` (rule R2 of `scripts/check-test-isolation.sh`). `detectTini()` calls `execFileSync('which', ['tini'])` with explicit `env: process.env` so Bun sees runtime PATH mutations (the env-snapshot bug fix). `buildSpawnInvocation(tiniPath, cmd, args)` returns `{cmd, args}` with tini prepended when present, or the bare invocation otherwise. Pinned by `test/spawn-helpers.test.ts` (5 cases) and `test/supervisor-tini.test.ts` (4 cases).
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
@@ -152,7 +153,7 @@ strict behavior when unset.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart).
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart). **v0.34.3.0:** inline spawn-and-respawn loop replaced with a `ChildWorkerSupervisor` instance. Drops `crashCount`, `lastWorkerStartTime`, `STABLE_RUN_RESET_MS`, `startWorker`, and the inline `child.on('exit')` block — all consolidated into the shared core. `--max-rss 2048` and `maxCrashes: 5` preserved from the legacy loop. `onMaxCrashesExceeded` now routes through autopilot's own `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor the inline loop called `process.exit(1)` directly and bypassed cleanup). `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of `workerProc.kill()`. Pinned by `test/autopilot-supervisor-wiring.test.ts` (6 static-shape regression guards: composes ChildWorkerSupervisor not the legacy inline names, `--max-rss 2048` in argv, `maxCrashes: 5` literal, shutdown-via-callback wiring, no workerProc reference). Closes the parallel-supervisor bug class Codex flagged during plan-eng-review.
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. **v0.34.1.0 (#870):** stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'`. Gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`, similar) pipe the JSON-RPC handshake then close their stdin half; pre-fix this killed the server before the first tool call landed. Signal handlers (SIGTERM / SIGINT / SIGHUP) and the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the runtime guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`.
|
||||
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP transports. Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1/F2/F3 drift bugs in the original v0.22.5 HTTP transport. **v0.26.9 (F8):** adds `summarizeMcpParams(opName, params)` — privacy-preserving redactor for `mcp_request_log` and the admin SSE feed. Returns `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. Intersects submitted top-level keys against the operation's declared `params` allow-list (declared keys preserved as a sorted array for debug visibility; unknown keys counted but never named, closing the attacker-controlled-key-name leak). Byte counts bucketed up to nearest 1KB so an attacker can't binary-search secret-content sizes via repeated probes. Operators on a personal laptop who want raw payload visibility opt back in with `gbrain serve --http --log-full-params` (loud stderr warning at startup). Canonical helper — new logging code paths route through it rather than `JSON.stringify(params)`.
|
||||
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter. `buildDefaultLimiters()` returns the two-bucket pipeline: pre-auth IP (30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap bounds memory under attacker-controlled key growth.
|
||||
|
||||
+4
-3
@@ -238,8 +238,9 @@ strict behavior when unset.
|
||||
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver). **v0.28.1 engine-ownership invariant:** `start()` no longer calls `engine.disconnect()` on shutdown — that was a leaky abstraction (the worker disconnected an engine it didn't own). The CLI handler in `src/commands/jobs.ts case 'work'` now owns engine lifecycle via try/finally with loud error logging on disconnect failure. Pinned by `test/worker-shutdown-disconnect.test.ts` asserting the inverse (`disconnectSpy).not.toHaveBeenCalled()`). **v0.34.3.0:** RSS watchdog metric switched to non-file-backed pages on Linux. New exports `parseRssFromProcStatus(status)` (pure parser, exported for unit tests) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5). The default `getRss` injected into `WorkerOpts` is now `getAccurateRss` instead of `process.memoryUsage().rss`. Closes the prod incident where VmRSS inflated to 7GB on a 96K-page brain (file-backed git packfile mmaps) while heap stayed at ~100MB; the watchdog was firing every autopilot cycle. M1 parser fix uses field-presence regex checks so `RssAnon: 0 + RssShmem: 512` (shmem-only worker case) parses correctly instead of falling through to VmRSS. Pinned by `test/worker-rss.test.ts` (11 cases).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. **v0.28.1:** consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping that the in-process SIGCHLD reaper can't reach). Exposes `isTiniDetected` read-only accessor for tests. **v0.34.3.0:** spawn-and-respawn loop extracted into the shared `ChildWorkerSupervisor` core (see entry below). MinionSupervisor now composes the inner class via `runSuperviseLoop()` → `new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` shapes back through the existing `emit()` SupervisorEvent channel — JSONL audit consumers see byte-compatible output across the rename. PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor (standalone-daemon concerns). The pre-shipped reset-to-0-on-code=0 hunk that originally fixed the prod crash-counter incident is gone; the same fix lives in the shared core under the D1 amendment (code=0 leaves `crashCount` untouched, so a worker alternating real crashes + watchdog drains still trips `max_crashes`). D2 `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop by emitting `health_warn { reason: 'clean_restart_budget_exceeded' }` plus backoff after the threshold trips. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of reaching into `this.child` directly. Pinned by `test/supervisor.test.ts` (16 cases; existing tests that previously relied on clean-exit-as-crash semantics now use exit-1 workers since clean exits no longer count) and `test/supervisor-tini.test.ts`.
|
||||
- `src/core/minions/child-worker-supervisor.ts` (v0.34.3.0) — shared spawn-and-respawn core extracted from `MinionSupervisor` so it can be reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon). Pre-v0.34.3.0 the two consumers maintained parallel spawn loops that drifted into the same bug class — Codex caught it during plan-eng-review on PR #1003. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void` callback so each composer routes to its own log/audit channel. **D1 exit classifier:** `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences — a worker that alternates `exit 1 / exit 0 / exit 1 / exit 0` correctly trips `max_crashes` after 10 real crashes regardless of intervening clean exits). `code != 0` follows the existing `runDuration > stableRunResetMs ? 1 : ++crashCount` rule. **D2 clean-restart budget:** sliding window tracks code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s) before the next spawn. Caps the worst-case tight-loop on macOS / restricted containers / kernel <4.5 where the worker's RSS watchdog falls back to VmRSS. Public read-only accessors `childAlive`, `inBackoff`, `crashCount` for composer health checks; `killChild(signal)` + `awaitChildExit(timeoutMs)` for shutdown paths. `awaitChildExit` short-circuits when `child.exitCode !== null || child.signalCode !== null` (regression caught in pre-landing /review: pre-fix, fast-SIGTERM responders caused a 35-second shutdown hang because the late `once('exit', ...)` listener never fired). Test hooks: `_backoffFloorMs` skips the real backoff curve, `_now` injects a fake clock. Pinned by `test/child-worker-supervisor.test.ts` (7 cases: D1 classifier with code=0 not counted, interleaved exits still trip max_crashes, stable-run + clean-exit interaction across faked 6-minute run, D2 budget triggers backoff + health_warn, budget config is per-instance, awaitChildExit short-circuit, event-shape regression). Plan that produced the design lives at `~/.claude/plans/this-is-a-real-sleepy-sketch.md`.
|
||||
- `src/core/minions/spawn-helpers.ts` (v0.28.1) — pure `detectTini()` + `buildSpawnInvocation()` helpers consumed by both `supervisor.ts` and `autopilot.ts`. Resolves the DRY violation between the two spawn sites and makes the tini wrapping testable without `mock.module()` (rule R2 of `scripts/check-test-isolation.sh`). `detectTini()` calls `execFileSync('which', ['tini'])` with explicit `env: process.env` so Bun sees runtime PATH mutations (the env-snapshot bug fix). `buildSpawnInvocation(tiniPath, cmd, args)` returns `{cmd, args}` with tini prepended when present, or the bare invocation otherwise. Pinned by `test/spawn-helpers.test.ts` (5 cases) and `test/supervisor-tini.test.ts` (4 cases).
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
@@ -260,7 +261,7 @@ strict behavior when unset.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart).
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart). **v0.34.3.0:** inline spawn-and-respawn loop replaced with a `ChildWorkerSupervisor` instance. Drops `crashCount`, `lastWorkerStartTime`, `STABLE_RUN_RESET_MS`, `startWorker`, and the inline `child.on('exit')` block — all consolidated into the shared core. `--max-rss 2048` and `maxCrashes: 5` preserved from the legacy loop. `onMaxCrashesExceeded` now routes through autopilot's own `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor the inline loop called `process.exit(1)` directly and bypassed cleanup). `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of `workerProc.kill()`. Pinned by `test/autopilot-supervisor-wiring.test.ts` (6 static-shape regression guards: composes ChildWorkerSupervisor not the legacy inline names, `--max-rss 2048` in argv, `maxCrashes: 5` literal, shutdown-via-callback wiring, no workerProc reference). Closes the parallel-supervisor bug class Codex flagged during plan-eng-review.
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. **v0.34.1.0 (#870):** stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'`. Gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`, similar) pipe the JSON-RPC handshake then close their stdin half; pre-fix this killed the server before the first tool call landed. Signal handlers (SIGTERM / SIGINT / SIGHUP) and the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the runtime guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`.
|
||||
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP transports. Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1/F2/F3 drift bugs in the original v0.22.5 HTTP transport. **v0.26.9 (F8):** adds `summarizeMcpParams(opName, params)` — privacy-preserving redactor for `mcp_request_log` and the admin SSE feed. Returns `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. Intersects submitted top-level keys against the operation's declared `params` allow-list (declared keys preserved as a sorted array for debug visibility; unknown keys counted but never named, closing the attacker-controlled-key-name leak). Byte counts bucketed up to nearest 1KB so an attacker can't binary-search secret-content sizes via repeated probes. Operators on a personal laptop who want raw payload visibility opt back in with `gbrain serve --http --log-full-params` (loud stderr warning at startup). Canonical helper — new logging code paths route through it rather than `JSON.stringify(params)`.
|
||||
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter. `buildDefaultLimiters()` returns the two-bucket pipeline: pre-auth IP (30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap bounds memory under attacker-controlled key growth.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.34.2.0",
|
||||
"version": "0.34.3.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+61
-55
@@ -19,11 +19,11 @@
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { execSync, spawn, type ChildProcess } from 'child_process';
|
||||
import { execSync } from 'child_process';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadPreferences } from '../core/preferences.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { detectTini, buildSpawnInvocation } from '../core/minions/spawn-helpers.ts';
|
||||
import { ChildWorkerSupervisor } from '../core/minions/child-worker-supervisor.ts';
|
||||
|
||||
function parseArg(args: string[], flag: string): string | undefined {
|
||||
const idx = args.indexOf(flag);
|
||||
@@ -146,55 +146,64 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
const spawnManagedWorker = useMinionsDispatch && !noWorker;
|
||||
|
||||
let stopping = false;
|
||||
let workerProc: ChildProcess | null = null;
|
||||
let crashCount = 0;
|
||||
let lastWorkerStartTime = 0;
|
||||
|
||||
// Stable-run reset window (matches MinionSupervisor.ts:471-476 pattern). If the
|
||||
// worker ran > 5min before exit, treat as a fresh cycle (crashCount=1) so the
|
||||
// RSS watchdog firing hourly does NOT trip autopilot's give-up threshold after
|
||||
// ~5 hours of healthy uptime.
|
||||
const STABLE_RUN_RESET_MS = 5 * 60 * 1000;
|
||||
let childSupervisor: ChildWorkerSupervisor | null = null;
|
||||
|
||||
if (spawnManagedWorker) {
|
||||
const cliPath = resolveGbrainCliPath();
|
||||
// Resolve tini once at startup — not per respawn — to avoid shelling out
|
||||
// every time the worker restarts. Reaps zombie children from shell jobs
|
||||
// and embed batches that outlive a watchdog-killed worker.
|
||||
const tiniPath = detectTini();
|
||||
const startWorker = () => {
|
||||
// Inject the RSS watchdog default (2048 MB) for the autopilot-supervised
|
||||
// worker. Bare `gbrain jobs work` has no default; the supervisor and
|
||||
// autopilot are the production paths that opt in.
|
||||
const args = ['jobs', 'work', '--max-rss', '2048'];
|
||||
|
||||
const { cmd: spawnCmd, args: spawnArgs } = buildSpawnInvocation(tiniPath, cliPath, args);
|
||||
|
||||
const child = spawn(spawnCmd, spawnArgs, { stdio: 'inherit', env: process.env });
|
||||
workerProc = child;
|
||||
lastWorkerStartTime = Date.now();
|
||||
console.log(`[autopilot] Minions worker spawned (pid: ${child.pid}, watchdog: 2048MB${tiniPath ? ', tini: active' : ''})`);
|
||||
|
||||
child.on('exit', (code) => {
|
||||
workerProc = null;
|
||||
if (stopping) return;
|
||||
const runDuration = Date.now() - lastWorkerStartTime;
|
||||
if (runDuration > STABLE_RUN_RESET_MS) {
|
||||
// Stable run — forgive prior crash history. A watchdog-driven hourly
|
||||
// exit (the production path post-fix) lands here every time.
|
||||
crashCount = 1;
|
||||
} else {
|
||||
crashCount++;
|
||||
// Inject the RSS watchdog default (2048 MB) for the autopilot-supervised
|
||||
// worker. Bare `gbrain jobs work` has no default; the supervisor and
|
||||
// autopilot are the production paths that opt in.
|
||||
childSupervisor = new ChildWorkerSupervisor({
|
||||
cliPath,
|
||||
args: ['jobs', 'work', '--max-rss', '2048'],
|
||||
// process.env clone; autopilot doesn't gate shell jobs the way the
|
||||
// standalone supervisor does (autopilot is the operator-trust path).
|
||||
env: { ...process.env },
|
||||
maxCrashes: 5,
|
||||
isStopping: () => stopping,
|
||||
onMaxCrashesExceeded: (count, max) => {
|
||||
console.error(`[autopilot] ${count}/${max} consecutive worker crashes, giving up.`);
|
||||
void shutdown('max_crashes');
|
||||
},
|
||||
onEvent: (event) => {
|
||||
// Route ChildWorkerSupervisor events to autopilot's stderr log.
|
||||
// Matches the prior console output shape so operators reading
|
||||
// existing logs see the same lines.
|
||||
if (event.kind === 'worker_spawned') {
|
||||
console.log(
|
||||
`[autopilot] Minions worker spawned (pid: ${event.pid}, watchdog: 2048MB${event.tini ? ', tini: active' : ''})`,
|
||||
);
|
||||
} else if (event.kind === 'worker_spawn_failed') {
|
||||
console.error(
|
||||
`[autopilot] worker spawn failed (${event.phase}): ${event.error}${event.errnoCode ? ` (code=${event.errnoCode})` : ''}`,
|
||||
);
|
||||
} else if (event.kind === 'worker_exited') {
|
||||
console.error(
|
||||
`[autopilot] worker exited code=${event.code} signal=${event.signal} after ${event.runDurationMs}ms, crashCount=${event.crashCount}, cause=${event.likelyCause}`,
|
||||
);
|
||||
} else if (event.kind === 'backoff') {
|
||||
if (event.reason === 'budget_exceeded') {
|
||||
console.error(
|
||||
`[autopilot] clean-restart budget exceeded; backing off ${event.ms}ms before next spawn`,
|
||||
);
|
||||
} else if (event.reason === 'crash') {
|
||||
console.error(
|
||||
`[autopilot] crash backoff ${event.ms}ms (crashCount=${event.crashCount})`,
|
||||
);
|
||||
}
|
||||
// reason='clean_exit' with ms:0 is the steady-state watchdog drain;
|
||||
// logging every iteration would be noisy. Keep silent (the
|
||||
// worker_exited line already covers the user-visible signal).
|
||||
} else if (event.kind === 'health_warn') {
|
||||
console.error(
|
||||
`[autopilot] health_warn: ${event.reason} count=${event.count} window=${event.windowMs}ms`,
|
||||
);
|
||||
}
|
||||
if (crashCount >= 5) {
|
||||
console.error(`[autopilot] 5 consecutive worker crashes (run ${runDuration}ms), giving up.`);
|
||||
process.exit(1);
|
||||
}
|
||||
console.error(`[autopilot] worker exited code=${code} after ${runDuration}ms, restart #${crashCount} in 10s`);
|
||||
setTimeout(startWorker, 10_000);
|
||||
});
|
||||
};
|
||||
startWorker();
|
||||
},
|
||||
});
|
||||
// Fire-and-forget; runs alongside the dispatch loop. shutdown() drives
|
||||
// the child-supervisor's isStopping accessor + drain.
|
||||
void childSupervisor.run();
|
||||
} else if (!useMinionsDispatch) {
|
||||
const why = mode === 'off'
|
||||
? 'minion_mode=off'
|
||||
@@ -215,14 +224,11 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
if (stopping) return;
|
||||
stopping = true;
|
||||
console.log(`Autopilot stopping (${sig}).`);
|
||||
if (workerProc) {
|
||||
try { workerProc.kill('SIGTERM'); } catch { /* already dead */ }
|
||||
await Promise.race([
|
||||
new Promise<void>(r => workerProc!.once('exit', () => r())),
|
||||
new Promise<void>(r => setTimeout(() => r(), 35_000)),
|
||||
]);
|
||||
if (workerProc && !workerProc.killed) {
|
||||
try { workerProc.kill('SIGKILL'); } catch { /* already dead */ }
|
||||
if (childSupervisor) {
|
||||
childSupervisor.killChild('SIGTERM');
|
||||
await childSupervisor.awaitChildExit(35_000);
|
||||
if (childSupervisor.childAlive) {
|
||||
childSupervisor.killChild('SIGKILL');
|
||||
}
|
||||
}
|
||||
try { unlinkSync(lockPath); } catch { /* already gone */ }
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
/**
|
||||
* ChildWorkerSupervisor — Pure spawn-and-respawn core for child workers.
|
||||
*
|
||||
* Extracted from MinionSupervisor (src/core/minions/supervisor.ts) so it can
|
||||
* be reused by both MinionSupervisor (standalone `gbrain jobs supervisor`
|
||||
* daemon) and the autopilot command (src/commands/autopilot.ts). Pre-fix
|
||||
* those two had separate spawn loops with different crash-counting bugs;
|
||||
* this class is the single source of truth.
|
||||
*
|
||||
* RESPONSIBILITIES:
|
||||
* - Spawn the child process (with optional tini wrapper for zombie reaping).
|
||||
* - Await exit, classify the exit code, decide whether to respawn.
|
||||
* - Track crash count and trip max_crashes for real failures (code != 0).
|
||||
* - Track clean-restart budget for code=0 exits and apply backoff when
|
||||
* the watchdog-drain rate exceeds the budget (D2 in plan).
|
||||
* - Emit lifecycle events via injected callback.
|
||||
*
|
||||
* NON-RESPONSIBILITIES (these stay in the composing class):
|
||||
* - PID file locking.
|
||||
* - Signal handlers (SIGTERM/SIGINT).
|
||||
* - process.exit() — composer decides what to do on max_crashes.
|
||||
* - Health checks (DB probing, queue depth).
|
||||
* - Audit-log writing (composer's onEvent decides where it lands).
|
||||
*
|
||||
* EXIT CLASSIFIER (post-D1/D2):
|
||||
*
|
||||
* code === 0 -> crashCount UNCHANGED (preserves flap detection across
|
||||
* mixed exit sequences). Record clean-restart timestamp.
|
||||
* If clean-restart budget exceeded -> emit health_warn +
|
||||
* apply backoff. Else emit ms:0 backoff (immediate restart).
|
||||
*
|
||||
* code !== 0 AND runDuration > stableRunResetMs -> crashCount = 1
|
||||
* (stable run forgives prior crash history).
|
||||
*
|
||||
* code !== 0 AND runDuration <= stableRunResetMs -> crashCount++
|
||||
* (escalating exponential backoff).
|
||||
*/
|
||||
|
||||
import { spawn, type ChildProcess } from 'child_process';
|
||||
import { buildSpawnInvocation, detectTini } from './spawn-helpers.ts';
|
||||
import { calculateBackoffMs } from './supervisor.ts';
|
||||
|
||||
export type ChildSupervisorEvent =
|
||||
| { kind: 'worker_spawned'; pid: number; tini: boolean }
|
||||
| { kind: 'worker_spawn_failed'; error: string; phase: 'sync' | 'async'; errnoCode?: string }
|
||||
| {
|
||||
kind: 'worker_exited';
|
||||
code: number | null;
|
||||
signal: NodeJS.Signals | null;
|
||||
runDurationMs: number;
|
||||
likelyCause: string;
|
||||
crashCount: number;
|
||||
}
|
||||
| {
|
||||
kind: 'backoff';
|
||||
ms: number;
|
||||
crashCount: number;
|
||||
reason: 'clean_exit' | 'crash' | 'budget_exceeded';
|
||||
}
|
||||
| {
|
||||
kind: 'health_warn';
|
||||
reason: 'clean_restart_budget_exceeded';
|
||||
count: number;
|
||||
windowMs: number;
|
||||
};
|
||||
|
||||
export interface ChildWorkerSupervisorOpts {
|
||||
/** Path to the gbrain CLI binary. */
|
||||
cliPath: string;
|
||||
/** Worker argv after cliPath (e.g. ['jobs', 'work', '--max-rss', '2048']). */
|
||||
args: string[];
|
||||
/** Child env. Defaults to a clone of process.env. */
|
||||
env?: NodeJS.ProcessEnv;
|
||||
/** Give up after this many consecutive code != 0 exits. */
|
||||
maxCrashes: number;
|
||||
/** Stable-run reset window: code != 0 after this duration resets crashCount to 1. Default 5 min. */
|
||||
stableRunResetMs?: number;
|
||||
|
||||
/**
|
||||
* D2 clean-restart budget. Caps the rate of code=0 restarts so the
|
||||
* supervisor cannot tight-loop when the worker exits cleanly forever
|
||||
* (e.g. macOS RSS fallback path always over-threshold). When the count
|
||||
* of clean restarts inside `cleanRestartWindowMs` exceeds this number,
|
||||
* emit `health_warn` and apply `cleanRestartBudgetBackoffMs` before
|
||||
* the next spawn. Default 10.
|
||||
*/
|
||||
cleanRestartBudget?: number;
|
||||
/** Sliding-window size for budget tracking. Default 60 seconds. */
|
||||
cleanRestartWindowMs?: number;
|
||||
/** Backoff applied when budget is exceeded. Default 1 second. */
|
||||
cleanRestartBudgetBackoffMs?: number;
|
||||
|
||||
/**
|
||||
* Test-only override: minimum backoff in ms between child respawns.
|
||||
* Tests pass `1` to make crash-loops finish in < 1s. Not exposed via CLI.
|
||||
* @internal
|
||||
*/
|
||||
_backoffFloorMs?: number;
|
||||
|
||||
/** Lifecycle event callback. Composer routes these to its own log/audit channels. */
|
||||
onEvent: (event: ChildSupervisorEvent) => void;
|
||||
/** Called when crashCount reaches maxCrashes. Composer decides what to do (process.exit, shutdown, etc.). */
|
||||
onMaxCrashesExceeded: (count: number, max: number) => void;
|
||||
/** Accessor for the composer's stopping flag; loop exits when this returns true. */
|
||||
isStopping: () => boolean;
|
||||
|
||||
/** Test seed for the clean-restart window. Defaults to Date.now. @internal */
|
||||
_now?: () => number;
|
||||
}
|
||||
|
||||
const DEFAULTS = {
|
||||
stableRunResetMs: 5 * 60 * 1000,
|
||||
cleanRestartBudget: 10,
|
||||
cleanRestartWindowMs: 60_000,
|
||||
cleanRestartBudgetBackoffMs: 1_000,
|
||||
} as const;
|
||||
|
||||
export class ChildWorkerSupervisor {
|
||||
private readonly opts: ChildWorkerSupervisorOpts;
|
||||
private readonly tiniPath: string;
|
||||
private _crashCount = 0;
|
||||
private _lastExitCode: number | null = null;
|
||||
private _cleanRestartTimestamps: number[] = [];
|
||||
private _child: ChildProcess | null = null;
|
||||
private _inBackoff = false;
|
||||
private _lastStartTime = 0;
|
||||
|
||||
constructor(opts: ChildWorkerSupervisorOpts) {
|
||||
this.opts = opts;
|
||||
this.tiniPath = detectTini();
|
||||
}
|
||||
|
||||
/** Read-only state surfaces for the composing class's health checks. */
|
||||
get childAlive(): boolean {
|
||||
return this._child !== null && this._child.exitCode === null;
|
||||
}
|
||||
get inBackoff(): boolean {
|
||||
return this._inBackoff;
|
||||
}
|
||||
get crashCount(): number {
|
||||
return this._crashCount;
|
||||
}
|
||||
/** Whether tini was detected at construction. Used by tests + worker_spawned event payload. */
|
||||
get isTiniDetected(): boolean {
|
||||
return this.tiniPath !== '';
|
||||
}
|
||||
|
||||
/**
|
||||
* Send a signal to the live child (no-op if none). Used by composers'
|
||||
* shutdown paths. Idempotent — `kill('SIGTERM')` on a dead child is a no-op.
|
||||
*/
|
||||
killChild(signal: NodeJS.Signals): void {
|
||||
if (this._child && !this._child.killed) {
|
||||
try {
|
||||
this._child.kill(signal);
|
||||
} catch {
|
||||
/* already dead */
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for the current child to exit, bounded by `timeoutMs`. No-op if no
|
||||
* child is running. Used by composers' graceful-shutdown drains.
|
||||
*
|
||||
* Handles the already-exited case: if the child terminated between
|
||||
* `killChild('SIGTERM')` and this call (common on fast SIGTERM
|
||||
* responders), Node's `'exit'` event has already fired and a late
|
||||
* `once('exit', ...)` listener would never resolve. We probe
|
||||
* `child.exitCode !== null` first and short-circuit so clean shutdown
|
||||
* is sub-second instead of waiting out the full `timeoutMs`.
|
||||
*/
|
||||
awaitChildExit(timeoutMs: number): Promise<void> {
|
||||
if (!this._child) return Promise.resolve();
|
||||
const child = this._child;
|
||||
// Already exited? `exitCode` becomes non-null once Node has seen the
|
||||
// child terminate. `signalCode` is the symmetric flag for kill-signal
|
||||
// termination — checked too so a SIGKILLed child also short-circuits.
|
||||
if (child.exitCode !== null || child.signalCode !== null) {
|
||||
return Promise.resolve();
|
||||
}
|
||||
return new Promise<void>((resolve) => {
|
||||
let settled = false;
|
||||
const onExit = () => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
resolve();
|
||||
};
|
||||
child.once('exit', onExit);
|
||||
setTimeout(() => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
child.removeListener('exit', onExit);
|
||||
resolve();
|
||||
}, timeoutMs);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the spawn-and-respawn loop. Resolves when:
|
||||
* 1. composer.isStopping() returns true, OR
|
||||
* 2. crashCount reaches maxCrashes (after firing onMaxCrashesExceeded).
|
||||
*/
|
||||
async run(): Promise<void> {
|
||||
while (!this.opts.isStopping() && this._crashCount < this.opts.maxCrashes) {
|
||||
await this.spawnOnce();
|
||||
|
||||
if (this.opts.isStopping()) return;
|
||||
|
||||
if (this._crashCount >= this.opts.maxCrashes) {
|
||||
this.opts.onMaxCrashesExceeded(this._crashCount, this.opts.maxCrashes);
|
||||
return;
|
||||
}
|
||||
|
||||
await this.applyBackoff();
|
||||
}
|
||||
}
|
||||
|
||||
/** Single spawn lifecycle: spawn -> await exit -> classify. */
|
||||
private spawnOnce(): Promise<void> {
|
||||
return new Promise<void>((resolve) => {
|
||||
if (this.opts.isStopping()) {
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
|
||||
const env = this.opts.env ?? { ...process.env };
|
||||
this._lastStartTime = this.now();
|
||||
|
||||
const { cmd: spawnCmd, args: spawnArgs } = buildSpawnInvocation(
|
||||
this.tiniPath,
|
||||
this.opts.cliPath,
|
||||
this.opts.args,
|
||||
);
|
||||
|
||||
let child: ChildProcess;
|
||||
try {
|
||||
child = spawn(spawnCmd, spawnArgs, {
|
||||
stdio: 'inherit',
|
||||
env,
|
||||
});
|
||||
} catch (err: unknown) {
|
||||
// Synchronous spawn error (e.g. invalid cliPath shape). Count as a crash.
|
||||
this.opts.onEvent({
|
||||
kind: 'worker_spawn_failed',
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
phase: 'sync',
|
||||
});
|
||||
this._crashCount++;
|
||||
this._lastExitCode = null;
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
|
||||
this._child = child;
|
||||
|
||||
this.opts.onEvent({
|
||||
kind: 'worker_spawned',
|
||||
pid: child.pid ?? -1,
|
||||
tini: this.tiniPath !== '',
|
||||
});
|
||||
|
||||
// Async spawn errors (ENOENT, EACCES). Node fires 'error' first, then
|
||||
// 'exit' with code=null. We log the error; the 'exit' handler increments
|
||||
// crashCount as usual so the restart loop bounds permanent misconfigs
|
||||
// via max_crashes.
|
||||
child.on('error', (err) => {
|
||||
this.opts.onEvent({
|
||||
kind: 'worker_spawn_failed',
|
||||
error: err.message,
|
||||
phase: 'async',
|
||||
errnoCode: (err as NodeJS.ErrnoException).code,
|
||||
});
|
||||
});
|
||||
|
||||
child.on('exit', (code, signal) => {
|
||||
this._child = null;
|
||||
|
||||
if (this.opts.isStopping()) {
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
|
||||
const runDuration = this.now() - this._lastStartTime;
|
||||
|
||||
// D1: code=0 is a clean exit (watchdog drain, graceful stop, etc.).
|
||||
// Don't touch crashCount — preserves flap detection across mixed
|
||||
// exit sequences. D2: record the clean-restart timestamp for budget
|
||||
// tracking and prune entries outside the sliding window.
|
||||
this._lastExitCode = code;
|
||||
if (code === 0) {
|
||||
const nowMs = this.now();
|
||||
this._cleanRestartTimestamps.push(nowMs);
|
||||
const windowMs = this.opts.cleanRestartWindowMs ?? DEFAULTS.cleanRestartWindowMs;
|
||||
const cutoff = nowMs - windowMs;
|
||||
this._cleanRestartTimestamps = this._cleanRestartTimestamps.filter(
|
||||
(t) => t > cutoff,
|
||||
);
|
||||
} else {
|
||||
const resetMs = this.opts.stableRunResetMs ?? DEFAULTS.stableRunResetMs;
|
||||
if (runDuration > resetMs) {
|
||||
// Stable-run reset: forgive prior crash history.
|
||||
this._crashCount = 1;
|
||||
} else {
|
||||
this._crashCount++;
|
||||
}
|
||||
}
|
||||
|
||||
// Likely-cause heuristic, kept verbatim from MinionSupervisor.
|
||||
let likelyCause: string;
|
||||
if (signal === 'SIGKILL') {
|
||||
likelyCause = 'oom_or_external_kill';
|
||||
} else if (signal === 'SIGTERM') {
|
||||
likelyCause = 'graceful_shutdown';
|
||||
} else if (code === 1) {
|
||||
likelyCause = 'runtime_error';
|
||||
} else if (code === 0) {
|
||||
likelyCause = 'clean_exit';
|
||||
} else {
|
||||
likelyCause = 'unknown';
|
||||
}
|
||||
|
||||
this.opts.onEvent({
|
||||
kind: 'worker_exited',
|
||||
code: code ?? null,
|
||||
signal: signal ?? null,
|
||||
runDurationMs: runDuration,
|
||||
likelyCause,
|
||||
crashCount: this._crashCount,
|
||||
});
|
||||
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/** Compute and apply backoff based on the most recent exit classifier. */
|
||||
private async applyBackoff(): Promise<void> {
|
||||
if (this._lastExitCode === 0) {
|
||||
// D2: check the clean-restart budget. If exceeded, emit health_warn
|
||||
// and apply a fixed cooldown so the next spawn isn't instant. This
|
||||
// bounds the worst case on platforms where Diff 2's RssAnon helper
|
||||
// falls back to VmRSS (macOS, kernel <4.5, restricted containers).
|
||||
const budget = this.opts.cleanRestartBudget ?? DEFAULTS.cleanRestartBudget;
|
||||
const windowMs = this.opts.cleanRestartWindowMs ?? DEFAULTS.cleanRestartWindowMs;
|
||||
if (this._cleanRestartTimestamps.length > budget) {
|
||||
this.opts.onEvent({
|
||||
kind: 'health_warn',
|
||||
reason: 'clean_restart_budget_exceeded',
|
||||
count: this._cleanRestartTimestamps.length,
|
||||
windowMs,
|
||||
});
|
||||
const cooldown =
|
||||
this.opts._backoffFloorMs !== undefined
|
||||
? this.opts._backoffFloorMs
|
||||
: this.opts.cleanRestartBudgetBackoffMs ?? DEFAULTS.cleanRestartBudgetBackoffMs;
|
||||
this.opts.onEvent({
|
||||
kind: 'backoff',
|
||||
ms: Math.round(cooldown),
|
||||
crashCount: this._crashCount,
|
||||
reason: 'budget_exceeded',
|
||||
});
|
||||
this._inBackoff = true;
|
||||
try {
|
||||
await this.sleep(cooldown);
|
||||
} finally {
|
||||
this._inBackoff = false;
|
||||
}
|
||||
return;
|
||||
}
|
||||
// Within budget — immediate restart.
|
||||
this.opts.onEvent({
|
||||
kind: 'backoff',
|
||||
ms: 0,
|
||||
crashCount: this._crashCount,
|
||||
reason: 'clean_exit',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// code != 0: exponential backoff scaled by crashCount-1 (retry-attempt
|
||||
// index). On first crash: crashCount=1, exponent=0 -> 1s. After stable-
|
||||
// run reset: crashCount=1 again -> 1s fresh cycle.
|
||||
const backoff =
|
||||
this.opts._backoffFloorMs !== undefined
|
||||
? this.opts._backoffFloorMs
|
||||
: calculateBackoffMs(this._crashCount - 1);
|
||||
|
||||
this.opts.onEvent({
|
||||
kind: 'backoff',
|
||||
ms: Math.round(backoff),
|
||||
crashCount: this._crashCount,
|
||||
reason: 'crash',
|
||||
});
|
||||
|
||||
this._inBackoff = true;
|
||||
try {
|
||||
await this.sleep(backoff);
|
||||
} finally {
|
||||
this._inBackoff = false;
|
||||
}
|
||||
}
|
||||
|
||||
private sleep(ms: number): Promise<void> {
|
||||
return new Promise<void>((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
private now(): number {
|
||||
return this.opts._now ? this.opts._now() : Date.now();
|
||||
}
|
||||
}
|
||||
+119
-164
@@ -26,8 +26,11 @@
|
||||
* 3 PID file unwritable (permission / path error)
|
||||
*/
|
||||
|
||||
import { spawn, type ChildProcess } from 'child_process';
|
||||
import { detectTini, buildSpawnInvocation } from './spawn-helpers.ts';
|
||||
import { detectTini } from './spawn-helpers.ts';
|
||||
import {
|
||||
ChildWorkerSupervisor,
|
||||
type ChildSupervisorEvent,
|
||||
} from './child-worker-supervisor.ts';
|
||||
import {
|
||||
closeSync,
|
||||
existsSync,
|
||||
@@ -35,7 +38,6 @@ import {
|
||||
openSync,
|
||||
readFileSync,
|
||||
unlinkSync,
|
||||
writeFileSync,
|
||||
writeSync,
|
||||
} from 'fs';
|
||||
import { dirname } from 'path';
|
||||
@@ -136,13 +138,15 @@ export const ExitCodes = {
|
||||
export class MinionSupervisor {
|
||||
private opts: SupervisorOpts;
|
||||
private engine: BrainEngine;
|
||||
private child: ChildProcess | null = null;
|
||||
private crashCount = 0;
|
||||
private lastStartTime = 0;
|
||||
/**
|
||||
* Inner spawn-and-respawn core. Created lazily in `start()` so options
|
||||
* passed via DEFAULTS merge are visible. Stays null when `stopping` is
|
||||
* tripped before `start()` runs (test edge case).
|
||||
*/
|
||||
private childSupervisor: ChildWorkerSupervisor | null = null;
|
||||
/** Path to tini binary for zombie reaping, or empty string when absent. */
|
||||
private readonly tiniPath: string;
|
||||
private stopping = false;
|
||||
private inBackoff = false;
|
||||
private healthInFlight = false;
|
||||
private healthTimer: ReturnType<typeof setInterval> | null = null;
|
||||
private exitListener: (() => void) | null = null;
|
||||
@@ -158,7 +162,8 @@ export class MinionSupervisor {
|
||||
// Detect tini for zombie reaping. Resolved once at construction so we
|
||||
// don't shell out on every respawn. Belt-and-suspenders with the
|
||||
// SIGCHLD handler in cli.ts — tini catches children spawned by native
|
||||
// addons that bypass the JS event loop.
|
||||
// addons that bypass the JS event loop. ChildWorkerSupervisor detects
|
||||
// independently; both calls hit the same `which tini` lookup.
|
||||
this.tiniPath = detectTini();
|
||||
}
|
||||
|
||||
@@ -276,14 +281,12 @@ export class MinionSupervisor {
|
||||
this.healthTimer = null;
|
||||
}
|
||||
|
||||
if (this.child) {
|
||||
try { this.child.kill('SIGTERM'); } catch { /* already dead */ }
|
||||
await Promise.race([
|
||||
new Promise<void>(r => this.child!.once('exit', () => r())),
|
||||
new Promise<void>(r => setTimeout(() => r(), 35_000)),
|
||||
]);
|
||||
if (this.child && !this.child.killed) {
|
||||
try { this.child.kill('SIGKILL'); } catch { /* already dead */ }
|
||||
if (this.childSupervisor) {
|
||||
this.childSupervisor.killChild('SIGTERM');
|
||||
await this.childSupervisor.awaitChildExit(35_000);
|
||||
// If the child is still up after the 35s drain window, escalate.
|
||||
if (this.childSupervisor.childAlive) {
|
||||
this.childSupervisor.killChild('SIGKILL');
|
||||
}
|
||||
}
|
||||
|
||||
@@ -392,166 +395,116 @@ export class MinionSupervisor {
|
||||
}
|
||||
}
|
||||
|
||||
/** Run the supervise loop: spawn child, await exit, backoff+retry or give up. */
|
||||
/**
|
||||
* Run the supervise loop. Constructs a ChildWorkerSupervisor with the
|
||||
* D1 lastExitCode classifier + D2 clean-restart budget baked in, then
|
||||
* defers spawn/respawn/backoff to it. Maps the inner ChildSupervisorEvent
|
||||
* stream to MinionSupervisor's existing SupervisorEvent emit() channel
|
||||
* so JSONL audit consumers see byte-compatible output.
|
||||
*/
|
||||
private async runSuperviseLoop(): Promise<void> {
|
||||
while (!this.stopping && this.crashCount < this.opts.maxCrashes) {
|
||||
await this.spawnOnce();
|
||||
|
||||
if (this.stopping) return;
|
||||
|
||||
if (this.crashCount >= this.opts.maxCrashes) {
|
||||
this.emit('max_crashes_exceeded', {
|
||||
crash_count: this.crashCount,
|
||||
max_crashes: this.opts.maxCrashes,
|
||||
});
|
||||
await this.shutdown('max_crashes', ExitCodes.MAX_CRASHES);
|
||||
return;
|
||||
}
|
||||
|
||||
// crashCount - 1 is the retry-attempt index (0-based exponent for backoff math).
|
||||
// On first crash: crashCount=1, backoff exponent=0 → 1s.
|
||||
// After stable-run reset: crashCount=1 again → 1s fresh cycle.
|
||||
// Test-only: _backoffFloorMs short-circuits to a fixed tiny value so integration
|
||||
// tests can exercise crash loops in < 1s without waiting for the real curve.
|
||||
const backoff = this.opts._backoffFloorMs !== undefined
|
||||
? this.opts._backoffFloorMs
|
||||
: calculateBackoffMs(this.crashCount - 1);
|
||||
|
||||
this.emit('backoff', { ms: Math.round(backoff), crash_count: this.crashCount });
|
||||
|
||||
this.inBackoff = true;
|
||||
try {
|
||||
await new Promise<void>(r => setTimeout(r, backoff));
|
||||
} finally {
|
||||
this.inBackoff = false;
|
||||
}
|
||||
const workerArgs = [
|
||||
'jobs', 'work',
|
||||
'--concurrency', String(this.opts.concurrency),
|
||||
'--queue', this.opts.queue,
|
||||
];
|
||||
if (this.opts.maxRssMb > 0) {
|
||||
workerArgs.push('--max-rss', String(this.opts.maxRssMb));
|
||||
}
|
||||
|
||||
// Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS:
|
||||
// inherit only when caller opts in, otherwise strip from the clone.
|
||||
const env: Record<string, string | undefined> = { ...process.env };
|
||||
if (this.opts.allowShellJobs) {
|
||||
env.GBRAIN_ALLOW_SHELL_JOBS = '1';
|
||||
} else {
|
||||
delete env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
}
|
||||
// Signal to the child worker that it's running under a supervisor.
|
||||
// The worker's self-health-check (DB probes, stall detection) is
|
||||
// redundant when the supervisor already provides these — setting
|
||||
// this env var causes the worker to skip its own health timer.
|
||||
env.GBRAIN_SUPERVISED = '1';
|
||||
|
||||
this.childSupervisor = new ChildWorkerSupervisor({
|
||||
cliPath: this.opts.cliPath,
|
||||
args: workerArgs,
|
||||
env,
|
||||
maxCrashes: this.opts.maxCrashes,
|
||||
_backoffFloorMs: this.opts._backoffFloorMs,
|
||||
isStopping: () => this.stopping,
|
||||
onMaxCrashesExceeded: (count, max) => {
|
||||
this.emit('max_crashes_exceeded', {
|
||||
crash_count: count,
|
||||
max_crashes: max,
|
||||
});
|
||||
void this.shutdown('max_crashes', ExitCodes.MAX_CRASHES);
|
||||
},
|
||||
onEvent: (event) => this.relayChildEvent(event),
|
||||
});
|
||||
|
||||
await this.childSupervisor.run();
|
||||
}
|
||||
|
||||
/** Spawn the worker child once and await its exit. Updates `this.crashCount`. */
|
||||
private spawnOnce(): Promise<void> {
|
||||
return new Promise<void>((resolve) => {
|
||||
if (this.stopping) { resolve(); return; }
|
||||
|
||||
const args = [
|
||||
'jobs', 'work',
|
||||
'--concurrency', String(this.opts.concurrency),
|
||||
'--queue', this.opts.queue,
|
||||
];
|
||||
if (this.opts.maxRssMb > 0) {
|
||||
args.push('--max-rss', String(this.opts.maxRssMb));
|
||||
}
|
||||
|
||||
// Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS:
|
||||
// inherit only when caller opts in, otherwise strip from the clone.
|
||||
const env: Record<string, string | undefined> = { ...process.env };
|
||||
if (this.opts.allowShellJobs) {
|
||||
env.GBRAIN_ALLOW_SHELL_JOBS = '1';
|
||||
} else {
|
||||
delete env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
}
|
||||
// Signal to the child worker that it's running under a supervisor.
|
||||
// The worker's self-health-check (DB probes, stall detection) is
|
||||
// redundant when the supervisor already provides these — setting
|
||||
// this env var causes the worker to skip its own health timer.
|
||||
env.GBRAIN_SUPERVISED = '1';
|
||||
|
||||
this.lastStartTime = Date.now();
|
||||
|
||||
// Wrap with tini when available — reaps zombie children that the
|
||||
// SIGCHLD handler in cli.ts might miss (native addons, edge cases).
|
||||
const { cmd: spawnCmd, args: spawnArgs } = buildSpawnInvocation(
|
||||
this.tiniPath,
|
||||
this.opts.cliPath,
|
||||
args,
|
||||
);
|
||||
|
||||
let child: ChildProcess;
|
||||
try {
|
||||
child = spawn(spawnCmd, spawnArgs, {
|
||||
stdio: 'inherit',
|
||||
env,
|
||||
/**
|
||||
* Map ChildSupervisorEvent (the inner core's emission shape) to the
|
||||
* existing SupervisorEvent emit channel. The wire shape of audit JSONL
|
||||
* is unchanged — same event names, same field names, same payload
|
||||
* coverage. The `reason='budget_exceeded'` backoff variant is a new
|
||||
* additive field; pre-existing consumers ignore unknown fields.
|
||||
*/
|
||||
private relayChildEvent(event: ChildSupervisorEvent): void {
|
||||
switch (event.kind) {
|
||||
case 'worker_spawned':
|
||||
this.emit('worker_spawned', {
|
||||
pid: event.pid >= 0 ? event.pid : undefined,
|
||||
cli_path: this.opts.cliPath,
|
||||
...(event.tini ? { tini: true } : {}),
|
||||
});
|
||||
} catch (err: unknown) {
|
||||
// Synchronous spawn error (e.g., invalid cliPath shape). Count as a crash.
|
||||
return;
|
||||
|
||||
case 'worker_spawn_failed':
|
||||
this.emit('worker_spawn_failed', {
|
||||
cli_path: this.opts.cliPath,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
phase: 'sync',
|
||||
error: event.error,
|
||||
phase: event.phase,
|
||||
...(event.errnoCode ? { code: event.errnoCode } : {}),
|
||||
});
|
||||
return;
|
||||
|
||||
case 'worker_exited': {
|
||||
const exitReason = event.signal
|
||||
? `signal ${event.signal}`
|
||||
: `code ${event.code ?? 'null'}`;
|
||||
this.emit('worker_exited', {
|
||||
code: event.code,
|
||||
signal: event.signal,
|
||||
reason: exitReason,
|
||||
likely_cause: event.likelyCause,
|
||||
crash_count: event.crashCount,
|
||||
max_crashes: this.opts.maxCrashes,
|
||||
run_duration_ms: event.runDurationMs,
|
||||
});
|
||||
this.crashCount++;
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
|
||||
this.child = child;
|
||||
|
||||
this.emit('worker_spawned', {
|
||||
pid: child.pid,
|
||||
cli_path: this.opts.cliPath,
|
||||
...(this.tiniPath ? { tini: true } : {}),
|
||||
});
|
||||
|
||||
// Async spawn errors (ENOENT, EACCES after the fork/exec). Node fires
|
||||
// 'error' first, then 'exit' with code=null. We log the error; the
|
||||
// 'exit' handler increments crashCount as usual so the restart loop
|
||||
// continues (max-crashes bounds this for permanent misconfigs).
|
||||
child.on('error', (err) => {
|
||||
this.emit('worker_spawn_failed', {
|
||||
cli_path: this.opts.cliPath,
|
||||
error: err.message,
|
||||
code: (err as NodeJS.ErrnoException).code ?? 'unknown',
|
||||
phase: 'async',
|
||||
case 'backoff':
|
||||
this.emit('backoff', {
|
||||
ms: event.ms,
|
||||
crash_count: event.crashCount,
|
||||
reason: event.reason,
|
||||
});
|
||||
});
|
||||
return;
|
||||
|
||||
child.on('exit', (code, signal) => {
|
||||
this.child = null;
|
||||
|
||||
if (this.stopping) {
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
|
||||
// Stable-run reset: if the worker ran > 5min before crashing, we forgive
|
||||
// prior crash history and treat this as the first crash of a new cycle
|
||||
// (crashCount = 1, so backoff math uses retry-index 0 = 1s).
|
||||
const runDuration = Date.now() - this.lastStartTime;
|
||||
if (runDuration > 5 * 60 * 1000) {
|
||||
this.crashCount = 1;
|
||||
} else {
|
||||
this.crashCount++;
|
||||
}
|
||||
|
||||
const exitReason = signal ? `signal ${signal}` : `code ${code ?? 'null'}`;
|
||||
|
||||
// Classify the likely cause for easier debugging
|
||||
let likelyCause: string;
|
||||
if (signal === 'SIGKILL') {
|
||||
likelyCause = 'oom_or_external_kill';
|
||||
} else if (signal === 'SIGTERM') {
|
||||
likelyCause = 'graceful_shutdown';
|
||||
} else if (code === 1) {
|
||||
likelyCause = 'runtime_error';
|
||||
} else if (code === 0) {
|
||||
likelyCause = 'clean_exit';
|
||||
} else {
|
||||
likelyCause = 'unknown';
|
||||
}
|
||||
|
||||
this.emit('worker_exited', {
|
||||
code: code ?? null,
|
||||
signal: signal ?? null,
|
||||
reason: exitReason,
|
||||
likely_cause: likelyCause,
|
||||
crash_count: this.crashCount,
|
||||
max_crashes: this.opts.maxCrashes,
|
||||
run_duration_ms: runDuration,
|
||||
case 'health_warn':
|
||||
this.emit('health_warn', {
|
||||
reason: event.reason,
|
||||
count: event.count,
|
||||
window_ms: event.windowMs,
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -620,8 +573,10 @@ export class MinionSupervisor {
|
||||
|
||||
// F4: suppress "worker not alive" warn while we're in the expected
|
||||
// null-child window (crash-exit → backoff-sleep → next-spawn).
|
||||
const workerAlive = this.child != null && this.child.exitCode === null;
|
||||
if (!workerAlive && !this.stopping && !this.inBackoff) {
|
||||
const cs = this.childSupervisor;
|
||||
const workerAlive = cs !== null && cs.childAlive;
|
||||
const inBackoff = cs !== null && cs.inBackoff;
|
||||
if (!workerAlive && !this.stopping && !inBackoff) {
|
||||
this.emit('health_warn', {
|
||||
reason: 'worker_not_alive',
|
||||
queue: this.opts.queue,
|
||||
|
||||
@@ -22,6 +22,69 @@ import { calculateBackoff } from './backoff.ts';
|
||||
import { randomUUID } from 'crypto';
|
||||
import { EventEmitter } from 'events';
|
||||
import { evaluateQuietHours, type QuietHoursConfig } from './quiet-hours.ts';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
/**
|
||||
* Pure parser for /proc/self/status RSS fields. Returns bytes of
|
||||
* RssAnon + RssShmem when either field is present, or null when the
|
||||
* status text is from a kernel that doesn't expose those fields
|
||||
* (kernels older than 4.5) or the values are malformed. Exported so
|
||||
* the test suite can unit-test the field-presence + malformed-value
|
||||
* edge cases without mocking the filesystem.
|
||||
*
|
||||
* M1 fix: field-presence check, not value-presence check. The earlier
|
||||
* `if (anonKb > 0)` form conflated "field exists with value 0" with
|
||||
* "field missing", which mis-routed to VmRSS fallback in the legitimate
|
||||
* shmem-only worker case (RssAnon: 0 + RssShmem: 512).
|
||||
*/
|
||||
export function parseRssFromProcStatus(status: string): number | null {
|
||||
const anonMatch = status.match(/^RssAnon:\s+(\d+)/m);
|
||||
const shmemMatch = status.match(/^RssShmem:\s+(\d+)/m);
|
||||
if (anonMatch === null && shmemMatch === null) {
|
||||
return null;
|
||||
}
|
||||
const anonKb = parseInt(anonMatch?.[1] ?? '0', 10);
|
||||
const shmemKb = parseInt(shmemMatch?.[1] ?? '0', 10);
|
||||
if (isNaN(anonKb) || isNaN(shmemKb)) {
|
||||
return null;
|
||||
}
|
||||
return (anonKb + shmemKb) * 1024; // bytes
|
||||
}
|
||||
|
||||
/**
|
||||
* Read accurate RSS from /proc/self/status (RssAnon + RssShmem).
|
||||
*
|
||||
* `process.memoryUsage().rss` returns VmRSS which includes file-backed mmap'd
|
||||
* pages (e.g. git packfiles). On a 96K-page brain repo, git operations can
|
||||
* inflate VmRSS to 7GB+ while actual heap usage is ~100MB. The kernel reclaims
|
||||
* file-backed pages under memory pressure — they're cache, not real usage.
|
||||
*
|
||||
* RssAnon = anonymous pages (heap, stack, anonymous mmap). RssShmem = shared
|
||||
* anonymous pages (IPC, tmpfs). Their sum is the non-file-backed resident
|
||||
* memory used for **per-process leak detection** — exactly the metric a leak
|
||||
* watchdog wants. It is NOT a full container-OOM metric: cgroup memory
|
||||
* pressure includes page cache, so a sibling container holding the page
|
||||
* cache hot can OOM us even at low anon+shmem. Use cgroup-aware monitoring
|
||||
* for that scenario; this helper is for the worker's own leak guard.
|
||||
*
|
||||
* Falls back to process.memoryUsage().rss on non-Linux, missing /proc, or
|
||||
* kernels older than 4.5 that don't expose RssAnon/RssShmem.
|
||||
*
|
||||
* `readStatus` is injectable for tests — production callers use the default,
|
||||
* which reads `/proc/self/status`.
|
||||
*/
|
||||
export function getAccurateRss(
|
||||
readStatus: () => string = () => readFileSync('/proc/self/status', 'utf8'),
|
||||
): number {
|
||||
try {
|
||||
const status = readStatus();
|
||||
const parsed = parseRssFromProcStatus(status);
|
||||
if (parsed !== null) return parsed;
|
||||
} catch {
|
||||
// Non-Linux or /proc unavailable
|
||||
}
|
||||
return process.memoryUsage().rss;
|
||||
}
|
||||
|
||||
/** Reason payload emitted with `'unhealthy'` when self-health-check trips.
|
||||
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit. */
|
||||
@@ -93,7 +156,7 @@ export class MinionWorker extends EventEmitter {
|
||||
maxStalledCount: opts?.maxStalledCount ?? 1,
|
||||
pollInterval: opts?.pollInterval ?? 5000,
|
||||
maxRssMb: opts?.maxRssMb ?? 0,
|
||||
getRss: opts?.getRss ?? (() => process.memoryUsage().rss),
|
||||
getRss: opts?.getRss ?? getAccurateRss,
|
||||
rssCheckInterval: opts?.rssCheckInterval ?? 60000,
|
||||
healthCheckInterval: opts?.healthCheckInterval ?? 60000,
|
||||
stallWarnAfterMs: opts?.stallWarnAfterMs ?? 5 * 60_000,
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* Regression guards for the autopilot↔ChildWorkerSupervisor wiring.
|
||||
*
|
||||
* autopilot.ts:148+ used to have its own inline spawn-and-respawn loop
|
||||
* (separate from MinionSupervisor's). It had the same bug class fixed
|
||||
* in #1003 but on a parallel implementation. The fix wave refactored
|
||||
* autopilot to use the shared ChildWorkerSupervisor core, eliminating
|
||||
* the parallel-supervisor footgun.
|
||||
*
|
||||
* Because the autopilot spawn path is deep inside `runAutopilot()` and
|
||||
* gated by `spawnManagedWorker` (which needs a Postgres engine), a full
|
||||
* integration test would require a DATABASE_URL fixture. Instead, these
|
||||
* static-shape regressions read the source file and pin the load-bearing
|
||||
* constants:
|
||||
*
|
||||
* - `--max-rss 2048` is passed to the worker (incident-driving default)
|
||||
* - `maxCrashes: 5` matches the prior `crashCount >= 5` give-up rule
|
||||
* - The autopilot composes ChildWorkerSupervisor (not the legacy
|
||||
* inline `child.on('exit')` loop)
|
||||
* - The shutdown path drains via the supervisor's killChild/awaitChildExit
|
||||
*
|
||||
* If any of these regress in a future refactor, the test fails with a
|
||||
* pointer at autopilot.ts.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
const AUTOPILOT_SRC = readFileSync(
|
||||
join(import.meta.dir, '..', 'src', 'commands', 'autopilot.ts'),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
describe('autopilot.ts ↔ ChildWorkerSupervisor wiring', () => {
|
||||
it('imports ChildWorkerSupervisor from the shared core', () => {
|
||||
expect(AUTOPILOT_SRC).toContain(
|
||||
"import { ChildWorkerSupervisor } from '../core/minions/child-worker-supervisor.ts';",
|
||||
);
|
||||
});
|
||||
|
||||
it('does not retain the legacy inline spawn loop (`startWorker` + child.on)', () => {
|
||||
// The old code at autopilot.ts:165-197 defined `startWorker` and called
|
||||
// `child.on('exit', ...)` directly. The refactor must drop those names.
|
||||
expect(AUTOPILOT_SRC).not.toContain('const startWorker');
|
||||
expect(AUTOPILOT_SRC).not.toContain("child.on('exit'");
|
||||
// Also drop the parallel crash-tracking state that lived in autopilot.
|
||||
expect(AUTOPILOT_SRC).not.toContain('let crashCount');
|
||||
expect(AUTOPILOT_SRC).not.toContain('let lastWorkerStartTime');
|
||||
expect(AUTOPILOT_SRC).not.toContain('STABLE_RUN_RESET_MS');
|
||||
});
|
||||
|
||||
it("constructs ChildWorkerSupervisor with --max-rss 2048", () => {
|
||||
// The worker spawn args must include both flag tokens in argv order.
|
||||
// This is the incident-driving default; changing it without a deliberate
|
||||
// decision would regress the workaround for VmRSS inflation.
|
||||
expect(AUTOPILOT_SRC).toContain("'--max-rss', '2048'");
|
||||
expect(AUTOPILOT_SRC).toContain("'jobs', 'work'");
|
||||
});
|
||||
|
||||
it("constructs ChildWorkerSupervisor with maxCrashes: 5", () => {
|
||||
// Matches the legacy `crashCount >= 5` give-up rule from the inline
|
||||
// loop. The shared core uses this to decide when to fire
|
||||
// onMaxCrashesExceeded → autopilot's shutdown('max_crashes').
|
||||
expect(AUTOPILOT_SRC).toMatch(/maxCrashes:\s*5\b/);
|
||||
});
|
||||
|
||||
it('routes onMaxCrashesExceeded to autopilot.shutdown (not process.exit directly)', () => {
|
||||
// Pre-refactor: autopilot.ts called process.exit(1) directly when the
|
||||
// crash counter tripped, bypassing its own dispatch-loop cleanup and
|
||||
// lockfile removal. Post-refactor: the callback routes through
|
||||
// shutdown('max_crashes') so cleanup runs.
|
||||
expect(AUTOPILOT_SRC).toMatch(/onMaxCrashesExceeded:[\s\S]{0,300}shutdown\('max_crashes'\)/);
|
||||
});
|
||||
|
||||
it('shutdown drains via supervisor.killChild + awaitChildExit (not workerProc.kill)', () => {
|
||||
// The legacy shutdown reached into `workerProc` directly. Post-refactor
|
||||
// those calls go through the supervisor's typed surface, which lets the
|
||||
// class encapsulate the kill/drain sequence.
|
||||
expect(AUTOPILOT_SRC).toContain("childSupervisor.killChild('SIGTERM')");
|
||||
expect(AUTOPILOT_SRC).toContain('childSupervisor.awaitChildExit(35_000)');
|
||||
expect(AUTOPILOT_SRC).not.toContain('workerProc');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,410 @@
|
||||
/**
|
||||
* Tests for the shared spawn-and-respawn core used by MinionSupervisor
|
||||
* and src/commands/autopilot.ts. Pins the D1 lastExitCode-track behavior
|
||||
* and the D2 clean-restart-budget gate so future refactors can't silently
|
||||
* regress the supervisor crash-count incident this wave fixes.
|
||||
*
|
||||
* Strategy: each test writes a small shell script to disk that exits with
|
||||
* a chosen code after an optional sleep. The class spawns the script as
|
||||
* the "worker" and we assert on the event stream the class emits.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, afterEach } from 'bun:test';
|
||||
import { chmodSync, mkdirSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import {
|
||||
ChildWorkerSupervisor,
|
||||
type ChildSupervisorEvent,
|
||||
} from '../src/core/minions/child-worker-supervisor.ts';
|
||||
|
||||
interface Harness {
|
||||
workerScript: string;
|
||||
cleanup: () => void;
|
||||
}
|
||||
|
||||
function makeHarness(name: string, body: string): Harness {
|
||||
const root = join(tmpdir(), `gbrain-cws-test-${name}-${process.pid}-${Date.now()}`);
|
||||
mkdirSync(root, { recursive: true });
|
||||
const workerScript = join(root, 'worker.sh');
|
||||
writeFileSync(workerScript, `#!/bin/sh\n${body}\n`, 'utf8');
|
||||
chmodSync(workerScript, 0o755);
|
||||
return {
|
||||
workerScript,
|
||||
cleanup: () => {
|
||||
try {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
} catch {
|
||||
/* noop */
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
interface RunResult {
|
||||
events: ChildSupervisorEvent[];
|
||||
maxCrashesFired: { count: number; max: number } | null;
|
||||
}
|
||||
|
||||
async function runUntilTerminal(
|
||||
h: Harness,
|
||||
overrides: Partial<{
|
||||
maxCrashes: number;
|
||||
_backoffFloorMs: number;
|
||||
cleanRestartBudget: number;
|
||||
cleanRestartWindowMs: number;
|
||||
cleanRestartBudgetBackoffMs: number;
|
||||
stableRunResetMs: number;
|
||||
_now: () => number;
|
||||
stopAfterEvents: number; // safety net so a buggy test can't hang
|
||||
}>,
|
||||
): Promise<RunResult> {
|
||||
const events: ChildSupervisorEvent[] = [];
|
||||
let stopping = false;
|
||||
let maxCrashesFired: { count: number; max: number } | null = null;
|
||||
const stopAfter = overrides.stopAfterEvents ?? 200;
|
||||
|
||||
const sup = new ChildWorkerSupervisor({
|
||||
cliPath: h.workerScript,
|
||||
args: [],
|
||||
maxCrashes: overrides.maxCrashes ?? 3,
|
||||
_backoffFloorMs: overrides._backoffFloorMs ?? 5,
|
||||
cleanRestartBudget: overrides.cleanRestartBudget,
|
||||
cleanRestartWindowMs: overrides.cleanRestartWindowMs,
|
||||
cleanRestartBudgetBackoffMs: overrides.cleanRestartBudgetBackoffMs,
|
||||
stableRunResetMs: overrides.stableRunResetMs,
|
||||
_now: overrides._now,
|
||||
isStopping: () => stopping,
|
||||
onMaxCrashesExceeded: (count, max) => {
|
||||
maxCrashesFired = { count, max };
|
||||
stopping = true;
|
||||
},
|
||||
onEvent: (event) => {
|
||||
events.push(event);
|
||||
if (events.length >= stopAfter) {
|
||||
stopping = true;
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
await sup.run();
|
||||
return { events, maxCrashesFired };
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
/* per-test harness.cleanup() runs in finally blocks below */
|
||||
});
|
||||
|
||||
describe('ChildWorkerSupervisor', () => {
|
||||
describe('D1 — code=0 exit classifier', () => {
|
||||
it('code=0 worker exit does not count as crash; restarts immediately', async () => {
|
||||
const h = makeHarness('clean-exits', 'exit 0');
|
||||
try {
|
||||
const res = await runUntilTerminal(h, {
|
||||
maxCrashes: 3,
|
||||
stopAfterEvents: 30, // ~10 spawn/exit/backoff trios
|
||||
});
|
||||
expect(res.maxCrashesFired).toBeNull();
|
||||
|
||||
const exits = res.events.filter((e) => e.kind === 'worker_exited');
|
||||
expect(exits.length).toBeGreaterThanOrEqual(3);
|
||||
for (const e of exits) {
|
||||
if (e.kind === 'worker_exited') {
|
||||
expect(e.code).toBe(0);
|
||||
expect(e.likelyCause).toBe('clean_exit');
|
||||
// crashCount stays at 0 across every clean exit
|
||||
expect(e.crashCount).toBe(0);
|
||||
}
|
||||
}
|
||||
|
||||
const backoffs = res.events.filter((e) => e.kind === 'backoff');
|
||||
expect(backoffs.length).toBeGreaterThanOrEqual(1);
|
||||
// Within the default 10-restart budget, all backoffs are ms:0 / clean_exit
|
||||
for (const e of backoffs) {
|
||||
if (e.kind === 'backoff') {
|
||||
// Once we cross the 10-restart budget the reason flips to
|
||||
// budget_exceeded, but until then they're all clean_exit ms:0.
|
||||
if (e.reason === 'clean_exit') {
|
||||
expect(e.ms).toBe(0);
|
||||
expect(e.crashCount).toBe(0);
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it('interleaved code=0 and code!=0 exits still trip max_crashes', async () => {
|
||||
// Worker alternates: each invocation increments a counter file and
|
||||
// exits 1 on odd hits, 0 on even hits (so exit-sequence is 1,0,1,0,1).
|
||||
const h = makeHarness(
|
||||
'interleaved',
|
||||
`
|
||||
COUNTER_FILE="$(dirname "$0")/counter"
|
||||
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
|
||||
COUNT=$(cat "$COUNTER_FILE")
|
||||
NEXT=$((COUNT + 1))
|
||||
echo "$NEXT" > "$COUNTER_FILE"
|
||||
# Odd-indexed runs (#1, #3, #5...) exit 1; even-indexed exit 0.
|
||||
if [ $((NEXT % 2)) -eq 1 ]; then exit 1; else exit 0; fi
|
||||
`,
|
||||
);
|
||||
try {
|
||||
const res = await runUntilTerminal(h, {
|
||||
maxCrashes: 3,
|
||||
_backoffFloorMs: 5,
|
||||
stopAfterEvents: 200,
|
||||
});
|
||||
|
||||
expect(res.maxCrashesFired).not.toBeNull();
|
||||
// 3 code!=0 exits → max_crashes=3
|
||||
expect(res.maxCrashesFired!.count).toBe(3);
|
||||
|
||||
const exits = res.events.filter((e) => e.kind === 'worker_exited');
|
||||
// Should be exactly 5 exits: 1, 0, 1, 0, 1 — then max fires.
|
||||
const codes = exits
|
||||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
|
||||
.map((e) => e.code);
|
||||
expect(codes).toEqual([1, 0, 1, 0, 1]);
|
||||
|
||||
const backoffs = res.events
|
||||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff');
|
||||
// Backoffs only fire between iterations 1-4 (not after the 5th, since
|
||||
// the loop bails out via onMaxCrashesExceeded before applyBackoff).
|
||||
// Even-index exits (code=0, indices 1+3) → reason='clean_exit'.
|
||||
// Odd-index exits (code=1, indices 0+2) → reason='crash'.
|
||||
const reasons = backoffs.map((e) => e.reason);
|
||||
expect(reasons).toEqual(['crash', 'clean_exit', 'crash', 'clean_exit']);
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it('code=0 after stable 5min+ run does not reset crashCount', async () => {
|
||||
// Sequence (4 runs total): exit 1 → exit 0 (6 min, "stable") → exit 1 →
|
||||
// exit 1. crashCount progression: 1, 1 (unchanged across the long
|
||||
// clean exit), 2, 3 — last one trips max_crashes=3.
|
||||
const h = makeHarness(
|
||||
'stable-clean-no-reset',
|
||||
`
|
||||
COUNTER_FILE="$(dirname "$0")/counter"
|
||||
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
|
||||
COUNT=$(cat "$COUNTER_FILE")
|
||||
NEXT=$((COUNT + 1))
|
||||
echo "$NEXT" > "$COUNTER_FILE"
|
||||
case $NEXT in
|
||||
1) exit 1 ;;
|
||||
2) exit 0 ;;
|
||||
3) exit 1 ;;
|
||||
4) exit 1 ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
`,
|
||||
);
|
||||
try {
|
||||
// Fake clock — each spawnOnce reads now() twice (start + exit) and
|
||||
// applyBackoff may read once more. Run 2 sees a 6-minute duration
|
||||
// (stable-run reset would fire IF the exit were code!=0 — we assert
|
||||
// it does NOT fire when the exit is clean).
|
||||
const SIX_MIN = 6 * 60_000;
|
||||
const timestamps = [
|
||||
0, // run 1 start
|
||||
1_000, // run 1 exit (+1s) → crashCount 1
|
||||
1_000, // run 2 start
|
||||
1_000 + SIX_MIN, // run 2 exit (+6min) → code=0, stays at 1
|
||||
1_000 + SIX_MIN, // run 3 start
|
||||
1_000 + SIX_MIN + 1_000, // run 3 exit (+1s) → crashCount 2
|
||||
1_000 + SIX_MIN + 1_000, // run 4 start
|
||||
1_000 + SIX_MIN + 2_000, // run 4 exit (+1s) → crashCount 3, trips max
|
||||
];
|
||||
let idx = 0;
|
||||
const last = timestamps[timestamps.length - 1];
|
||||
const fakeNow = () => {
|
||||
if (idx < timestamps.length) {
|
||||
return timestamps[idx++];
|
||||
}
|
||||
return last + (idx++ - timestamps.length + 1) * 100;
|
||||
};
|
||||
|
||||
const res = await runUntilTerminal(h, {
|
||||
maxCrashes: 3,
|
||||
_backoffFloorMs: 5,
|
||||
_now: fakeNow,
|
||||
stopAfterEvents: 200,
|
||||
});
|
||||
|
||||
expect(res.maxCrashesFired).not.toBeNull();
|
||||
expect(res.maxCrashesFired!.count).toBe(3);
|
||||
|
||||
const exits = res.events
|
||||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
|
||||
.map((e) => ({ code: e.code, crashCount: e.crashCount, runDurationMs: e.runDurationMs }));
|
||||
|
||||
expect(exits.length).toBeGreaterThanOrEqual(4);
|
||||
expect(exits[0]).toMatchObject({ code: 1, crashCount: 1 });
|
||||
expect(exits[1]).toMatchObject({ code: 0, crashCount: 1 }); // D1: unchanged
|
||||
expect(exits[2]).toMatchObject({ code: 1, crashCount: 2 });
|
||||
expect(exits[3]).toMatchObject({ code: 1, crashCount: 3 });
|
||||
// Run 2 ran 6min, but because exit code was 0 the stable-run reset
|
||||
// branch did NOT fire — crashCount stayed at 1. This is the core
|
||||
// D1 invariant: clean exits never reset crashCount, even stable ones.
|
||||
expect(exits[1].runDurationMs).toBe(SIX_MIN);
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('D2 — clean-restart budget', () => {
|
||||
it('budget exceeded triggers health_warn + budget_exceeded backoff', async () => {
|
||||
// Tight budget of 2 so we trip it on the 3rd clean exit.
|
||||
const h = makeHarness('budget-trip', 'exit 0');
|
||||
try {
|
||||
const res = await runUntilTerminal(h, {
|
||||
maxCrashes: 3, // never trips because code=0 doesn't increment
|
||||
_backoffFloorMs: 5,
|
||||
cleanRestartBudget: 2,
|
||||
cleanRestartWindowMs: 60_000,
|
||||
cleanRestartBudgetBackoffMs: 10,
|
||||
stopAfterEvents: 25,
|
||||
});
|
||||
|
||||
const healthWarns = res.events.filter(
|
||||
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> => e.kind === 'health_warn',
|
||||
);
|
||||
// Once tripped, every subsequent clean exit re-fires health_warn
|
||||
// (the sliding window stays full at our test rate).
|
||||
expect(healthWarns.length).toBeGreaterThan(0);
|
||||
for (const w of healthWarns) {
|
||||
expect(w.reason).toBe('clean_restart_budget_exceeded');
|
||||
expect(w.windowMs).toBe(60_000);
|
||||
expect(w.count).toBeGreaterThan(2);
|
||||
}
|
||||
|
||||
const backoffReasons = res.events
|
||||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff')
|
||||
.map((e) => e.reason);
|
||||
// First 2 exits are within budget → reason='clean_exit'.
|
||||
// From the 3rd exit onward → reason='budget_exceeded'.
|
||||
expect(backoffReasons.slice(0, 2)).toEqual(['clean_exit', 'clean_exit']);
|
||||
expect(backoffReasons.slice(2).every((r) => r === 'budget_exceeded')).toBe(true);
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it('budget config is per-instance (no module-level state leakage)', async () => {
|
||||
// Run instance A with budget=2 and instance B with budget=5. Each
|
||||
// tracks its own sliding window; A trips faster than B.
|
||||
const hA = makeHarness('budget-a', 'exit 0');
|
||||
const hB = makeHarness('budget-b', 'exit 0');
|
||||
try {
|
||||
const resA = await runUntilTerminal(hA, {
|
||||
maxCrashes: 99,
|
||||
_backoffFloorMs: 5,
|
||||
cleanRestartBudget: 2,
|
||||
cleanRestartBudgetBackoffMs: 5,
|
||||
stopAfterEvents: 12,
|
||||
});
|
||||
const resB = await runUntilTerminal(hB, {
|
||||
maxCrashes: 99,
|
||||
_backoffFloorMs: 5,
|
||||
cleanRestartBudget: 5,
|
||||
cleanRestartBudgetBackoffMs: 5,
|
||||
stopAfterEvents: 18,
|
||||
});
|
||||
|
||||
const firstTripA = resA.events.findIndex(
|
||||
(e) => e.kind === 'health_warn',
|
||||
);
|
||||
const firstTripB = resB.events.findIndex(
|
||||
(e) => e.kind === 'health_warn',
|
||||
);
|
||||
|
||||
expect(firstTripA).toBeGreaterThan(-1);
|
||||
expect(firstTripB).toBeGreaterThan(-1);
|
||||
// B's budget is more generous → its first health_warn appears later
|
||||
// in the event stream (after more spawn/exit pairs).
|
||||
expect(firstTripB).toBeGreaterThan(firstTripA);
|
||||
} finally {
|
||||
hA.cleanup();
|
||||
hB.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('awaitChildExit short-circuit (P2 review fix)', () => {
|
||||
// Regression: pre-fix the method registered child.once('exit', ...) AFTER
|
||||
// child.exitCode was already populated, so a child that drained quickly
|
||||
// between killChild('SIGTERM') and awaitChildExit() would never resolve
|
||||
// and the caller waited out the full timeout. Fix probes exitCode +
|
||||
// signalCode first and short-circuits.
|
||||
it('resolves immediately when the child has already exited', async () => {
|
||||
const h = makeHarness('await-already-exited', 'exit 0');
|
||||
try {
|
||||
// Spin up a supervisor; drive it for ONE spawn cycle and then stop.
|
||||
const events: ChildSupervisorEvent[] = [];
|
||||
let stopping = false;
|
||||
const sup = new ChildWorkerSupervisor({
|
||||
cliPath: h.workerScript,
|
||||
args: [],
|
||||
maxCrashes: 1,
|
||||
_backoffFloorMs: 1,
|
||||
isStopping: () => stopping,
|
||||
onMaxCrashesExceeded: () => { stopping = true; },
|
||||
onEvent: (e) => {
|
||||
events.push(e);
|
||||
if (e.kind === 'worker_exited') stopping = true;
|
||||
},
|
||||
});
|
||||
await sup.run();
|
||||
// After run() returns, the child has exited; awaitChildExit on an
|
||||
// already-finished cycle MUST resolve in well under the timeout.
|
||||
const start = Date.now();
|
||||
await sup.awaitChildExit(5_000);
|
||||
const elapsed = Date.now() - start;
|
||||
expect(elapsed).toBeLessThan(200);
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('event shape', () => {
|
||||
it('worker_spawned + worker_exited fire on every cycle with consistent shape', async () => {
|
||||
const h = makeHarness('shape', 'exit 0');
|
||||
try {
|
||||
const res = await runUntilTerminal(h, {
|
||||
maxCrashes: 3,
|
||||
_backoffFloorMs: 5,
|
||||
stopAfterEvents: 9, // 3 spawn-exit-backoff triples
|
||||
});
|
||||
|
||||
const spawned = res.events.filter(
|
||||
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_spawned' }> => e.kind === 'worker_spawned',
|
||||
);
|
||||
const exited = res.events.filter(
|
||||
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited',
|
||||
);
|
||||
|
||||
expect(spawned.length).toBeGreaterThanOrEqual(2);
|
||||
expect(exited.length).toBe(spawned.length);
|
||||
|
||||
for (const s of spawned) {
|
||||
expect(typeof s.pid).toBe('number');
|
||||
expect(s.pid).toBeGreaterThan(0);
|
||||
expect(typeof s.tini).toBe('boolean');
|
||||
}
|
||||
for (const e of exited) {
|
||||
expect(e.code).toBe(0);
|
||||
expect(e.signal).toBeNull();
|
||||
expect(typeof e.runDurationMs).toBe('number');
|
||||
expect(e.likelyCause).toBe('clean_exit');
|
||||
}
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
+19
-5
@@ -282,7 +282,12 @@ describe('MinionSupervisor', () => {
|
||||
const outFile = join(tmpdir(), `gbrain-sup-env-${process.pid}-${Date.now()}.txt`);
|
||||
try { unlinkSync(outFile); } catch { /* may not exist */ }
|
||||
|
||||
const h = makeHarness('env-strip-outfile', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
|
||||
// Worker writes env to OUT_FILE then exits 1. exit=1 is required (not
|
||||
// exit=0) because post-D1/D2 (v0.33) clean exits don't count toward
|
||||
// crashCount — the supervisor would respawn forever. The test's
|
||||
// assertion is on the OUT_FILE contents (env plumbing), not the
|
||||
// exit code, so any non-zero code that trips SUP_MAX_CRASHES=1 works.
|
||||
const h = makeHarness('env-strip-outfile', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 1`);
|
||||
|
||||
try {
|
||||
const sup = spawnSupervisor(h, {
|
||||
@@ -308,7 +313,9 @@ describe('MinionSupervisor', () => {
|
||||
const outFile = join(tmpdir(), `gbrain-sup-env-ok-${process.pid}-${Date.now()}.txt`);
|
||||
try { unlinkSync(outFile); } catch { /* may not exist */ }
|
||||
|
||||
const h = makeHarness('env-pass-on-opt-in', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
|
||||
// Worker exits 1 (not 0) so SUP_MAX_CRASHES=1 actually trips. See
|
||||
// the comment on the env-strip test above for the v0.33 rationale.
|
||||
const h = makeHarness('env-pass-on-opt-in', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 1`);
|
||||
|
||||
try {
|
||||
const sup = spawnSupervisor(h, {
|
||||
@@ -333,7 +340,9 @@ describe('MinionSupervisor', () => {
|
||||
const outFile = join(tmpdir(), `gbrain-sup-supervised-${process.pid}-${Date.now()}.txt`);
|
||||
try { unlinkSync(outFile); } catch { /* may not exist */ }
|
||||
|
||||
const h = makeHarness('supervised-env', `printf '%s\n' "\${GBRAIN_SUPERVISED-UNSET}" > "$OUT_FILE" ; exit 0`);
|
||||
// exit 1 required post-D1/D2 to trip SUP_MAX_CRASHES=1; clean exits
|
||||
// no longer count toward the crash limit.
|
||||
const h = makeHarness('supervised-env', `printf '%s\n' "\${GBRAIN_SUPERVISED-UNSET}" > "$OUT_FILE" ; exit 1`);
|
||||
|
||||
try {
|
||||
const sup = spawnSupervisor(h, {
|
||||
@@ -372,7 +381,11 @@ describe('MinionSupervisor', () => {
|
||||
// window before max-crashes fires. We assert the basic completion path
|
||||
// and let CI's wall-clock detect any pathological CPU spike.
|
||||
it('completes a normal supervise lifecycle with healthInterval=0', async () => {
|
||||
const h = makeHarness('health-interval-zero', 'exit 0');
|
||||
// exit 1 (not exit 0) because post-D1/D2 (v0.33) clean exits don't
|
||||
// count toward max_crashes — a code=0 worker would respawn forever.
|
||||
// The test's purpose is regression coverage that healthInterval=0
|
||||
// disables the timer; the exit code doesn't matter to that assertion.
|
||||
const h = makeHarness('health-interval-zero', 'exit 1');
|
||||
|
||||
try {
|
||||
const sup = spawnSupervisor(h, {
|
||||
@@ -409,7 +422,8 @@ describe('MinionSupervisor', () => {
|
||||
// Worker logs its argv to OUT_FILE so the test can assert --max-rss 2048
|
||||
// landed there. spawnOnce in supervisor.ts builds:
|
||||
// ['jobs', 'work', '--concurrency', '1', '--queue', 'default', '--max-rss', '2048']
|
||||
const h = makeHarness('maxrss-default', `printf '%s\\n' "$*" > "$OUT_FILE" ; exit 0`);
|
||||
// exit 1 required post-D1/D2: code=0 workers respawn forever.
|
||||
const h = makeHarness('maxrss-default', `printf '%s\\n' "$*" > "$OUT_FILE" ; exit 1`);
|
||||
|
||||
try {
|
||||
const sup = spawnSupervisor(h, {
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
/**
|
||||
* Tests for the per-process RSS reader used by the worker's leak watchdog.
|
||||
*
|
||||
* `process.memoryUsage().rss` returns VmRSS which counts file-backed mmap
|
||||
* pages (e.g. git packfiles). On a 96K-page brain repo, git operations
|
||||
* inflate VmRSS to 7GB+ while heap is ~100MB — the kernel will reclaim
|
||||
* those pages under pressure so they should not count toward the watchdog
|
||||
* threshold. The fix reads `/proc/self/status` for RssAnon + RssShmem
|
||||
* (the non-file-backed pages) on Linux and falls back to VmRSS elsewhere.
|
||||
*
|
||||
* These tests pin the parser shape against the M1 regression Codex
|
||||
* surfaced during eng review: the prior `if (anonKb > 0)` form conflated
|
||||
* "field missing" with "field present but zero", which broke the
|
||||
* shmem-only worker case.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'bun:test';
|
||||
import { getAccurateRss, parseRssFromProcStatus } from '../src/core/minions/worker.ts';
|
||||
|
||||
describe('parseRssFromProcStatus', () => {
|
||||
it('parses normal RssAnon + RssShmem to bytes', () => {
|
||||
const status = [
|
||||
'Name:\tworker',
|
||||
'VmRSS:\t1024 kB',
|
||||
'RssAnon:\t1024 kB',
|
||||
'RssShmem:\t0 kB',
|
||||
'VmSize:\t8192 kB',
|
||||
].join('\n');
|
||||
expect(parseRssFromProcStatus(status)).toBe(1024 * 1024);
|
||||
});
|
||||
|
||||
it('M1 regression: RssAnon:0 + RssShmem>0 returns shmem-only bytes (not null)', () => {
|
||||
// The pre-fix code did `if (anonKb > 0) return ...` which would treat
|
||||
// this case as "field missing" and fall through to VmRSS. The fix uses
|
||||
// field-presence checks so anon=0 + shmem=512 yields 512 KiB → 524_288.
|
||||
const status = [
|
||||
'RssAnon:\t0 kB',
|
||||
'RssShmem:\t512 kB',
|
||||
].join('\n');
|
||||
expect(parseRssFromProcStatus(status)).toBe(512 * 1024);
|
||||
});
|
||||
|
||||
it('returns null when neither RssAnon nor RssShmem is present (old kernel)', () => {
|
||||
// Linux kernels older than 4.5 don't expose these fields. Returning
|
||||
// null signals "fall back to VmRSS" to the caller.
|
||||
const status = [
|
||||
'Name:\tworker',
|
||||
'VmRSS:\t1024 kB',
|
||||
'VmSize:\t8192 kB',
|
||||
].join('\n');
|
||||
expect(parseRssFromProcStatus(status)).toBeNull();
|
||||
});
|
||||
|
||||
it('treats non-numeric fields as absent (regex matches digits only)', () => {
|
||||
// The `\d+` regex won't match "notanumber", so RssAnon is treated as
|
||||
// absent and the result reflects only the well-formed RssShmem field.
|
||||
// This is the right behavior: a corrupt RssAnon line shouldn't
|
||||
// poison an otherwise valid RssShmem reading.
|
||||
const status = [
|
||||
'RssAnon:\tnotanumber kB',
|
||||
'RssShmem:\t0 kB',
|
||||
].join('\n');
|
||||
expect(parseRssFromProcStatus(status)).toBe(0);
|
||||
});
|
||||
|
||||
it('returns null when ALL fields are non-numeric', () => {
|
||||
const status = [
|
||||
'RssAnon:\tnotanumber kB',
|
||||
'RssShmem:\talsobad kB',
|
||||
].join('\n');
|
||||
expect(parseRssFromProcStatus(status)).toBeNull();
|
||||
});
|
||||
|
||||
it('returns sum when only RssShmem is present (anon missing)', () => {
|
||||
// Symmetric to the regression: if only RssShmem is exposed for some
|
||||
// reason, we should still use it. Treats anonKb default as 0.
|
||||
const status = [
|
||||
'RssShmem:\t256 kB',
|
||||
'VmRSS:\t9999 kB',
|
||||
].join('\n');
|
||||
expect(parseRssFromProcStatus(status)).toBe(256 * 1024);
|
||||
});
|
||||
|
||||
it('M1 regression: explicit RssAnon:0 + RssShmem:0 returns 0 (not null)', () => {
|
||||
// Both fields present, both zero is a valid reading for a near-empty
|
||||
// worker process. Should be treated as 0 bytes, NOT a missing reading.
|
||||
const status = [
|
||||
'RssAnon:\t0 kB',
|
||||
'RssShmem:\t0 kB',
|
||||
].join('\n');
|
||||
expect(parseRssFromProcStatus(status)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('getAccurateRss', () => {
|
||||
it('uses the parsed value when readStatus returns a valid /proc/self/status', () => {
|
||||
const fakeStatus = 'RssAnon:\t2048 kB\nRssShmem:\t0 kB\n';
|
||||
expect(getAccurateRss(() => fakeStatus)).toBe(2048 * 1024);
|
||||
});
|
||||
|
||||
it('falls back to process.memoryUsage().rss when readStatus throws (non-Linux)', () => {
|
||||
const result = getAccurateRss(() => {
|
||||
throw Object.assign(new Error('ENOENT: no such file'), { code: 'ENOENT' });
|
||||
});
|
||||
// We can't assert the exact RSS value cross-platform, but it MUST be
|
||||
// a positive integer (process.memoryUsage().rss is always set).
|
||||
expect(result).toBeGreaterThan(0);
|
||||
expect(Number.isInteger(result)).toBe(true);
|
||||
});
|
||||
|
||||
it('falls back when status text has no RssAnon/RssShmem fields (old kernel)', () => {
|
||||
const status = 'VmRSS:\t1024 kB\nVmSize:\t8192 kB\n';
|
||||
const result = getAccurateRss(() => status);
|
||||
// Falls back to process.memoryUsage().rss — a positive integer.
|
||||
expect(result).toBeGreaterThan(0);
|
||||
expect(Number.isInteger(result)).toBe(true);
|
||||
});
|
||||
|
||||
it('falls back on malformed values rather than returning NaN', () => {
|
||||
const status = 'RssAnon:\tNaN kB\n';
|
||||
const result = getAccurateRss(() => status);
|
||||
expect(Number.isNaN(result)).toBe(false);
|
||||
expect(result).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user