mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
* fix: supervisor treats code=0 watchdog exits as crashes
The RSS watchdog triggers gracefulShutdown() which exits with code 0.
The supervisor was counting ALL exits < 5min as crashes, including
clean code=0 exits. After 10 watchdog-triggered restarts (typical with
a 96K-page brain where autopilot inflates RSS), the supervisor gave up
with max_crashes_exceeded.
Fix: code=0 exits reset crashCount to 0 and restart immediately with
no backoff. Only code≠0 exits count toward the crash limit.
Root cause: process.memoryUsage().rss reports 7GB during autopilot
sync on large repos (possibly shared page inflation from git mmap).
The 4096MB threshold triggers on every cycle. This is a separate
issue (RSS measurement accuracy) but the supervisor should handle
clean exits regardless.
* fix: use RssAnon instead of VmRSS for watchdog threshold
process.memoryUsage().rss returns VmRSS which includes file-backed
mmap'd pages. On repos with large git packfiles (96K+ pages), git
operations inflate VmRSS to 7GB+ while actual heap usage is ~100MB.
The kernel reclaims these pages under memory pressure — they're cache.
Replace with /proc/self/status RssAnon + RssShmem which measures only
anonymous pages (heap, stack, anonymous mmap). This is the memory that
actually matters for OOM risk.
Falls back to process.memoryUsage().rss on non-Linux.
Before: watchdog triggers every autopilot cycle (7GB VmRSS > 4GB threshold)
After: watchdog only triggers on real memory growth (~100MB << 4GB threshold)
Related: #1002 (supervisor crash-count fix for the same symptom)
* refactor(minions): extract ChildWorkerSupervisor with D1/D2 amendments
MinionSupervisor and src/commands/autopilot.ts each owned a separate
spawn-and-respawn loop. PR #1003 fixed the supervisor's crash-counter
bug (counting code=0 watchdog drains as crashes) but the autopilot
loop has the same bug class. Worse, the as-shipped #1003 fix reset
crashCount=0 on every code=0 exit, which lost the "flapping worker"
signal in mixed-exit sequences.
Extract the shared spawn loop into ChildWorkerSupervisor so both
consumers compose one tested core. The new class bakes in two
amendments resolved during plan-eng-review:
D1 (lastExitCode track): code=0 exits no longer touch crashCount.
They emit ms:0 backoff and restart immediately, but the counter
survives across them. A worker alternating exit 1 / exit 0 / exit 1
correctly trips max_crashes; a worker drained 100 times by the
watchdog stays at crashCount=0 and runs forever (also correct).
D2 (clean-restart budget): on platforms where the watchdog measures
VmRSS instead of RssAnon (macOS, kernel <4.5, restricted containers),
a perpetually over-threshold worker could clean-exit in a tight loop
with no observability. New `cleanRestartBudget` option (default 10
clean restarts per 60s window) emits a `health_warn` and applies
backoff once exceeded.
The supervisor now delegates spawn/respawn/backoff to the inner
class and maps ChildSupervisorEvent → existing SupervisorEvent
emit() channel so JSONL audit consumers see byte-compatible output.
PID lock, signal handlers, health check, and process.exit on
max-crashes stay in MinionSupervisor (those are standalone-daemon
concerns the autopilot composer doesn't need).
Tests: 6 new ChildWorkerSupervisor cases (D1 classifier, interleaved
exits, stable-run + clean-exit interaction, D2 budget tripping, per-
instance config isolation, event shape regression). Existing supervisor
tests updated to use exit-1 workers where they previously relied on
clean-exit-as-crash semantics; their assertions (env plumbing, PID
lock, audit shape) are unaffected.
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(autopilot): compose ChildWorkerSupervisor instead of inline spawn loop
src/commands/autopilot.ts:165-197 used to have its own spawn-and-
respawn loop separate from MinionSupervisor's. It hardcoded
maxCrashes=5, fixed 10s backoff, and counted every exit (including
code=0) toward the crash limit. Codex flagged this during plan-eng
review: the parallel implementation had the same bug class fixed
in #1003, just on a different code path. Anyone running
`gbrain autopilot` as a long-running daemon (instead of
`gbrain jobs supervisor`) would hit it.
Replace the inline `startWorker` + `child.on('exit')` block with
a ChildWorkerSupervisor instance. Drops the parallel `crashCount`,
`lastWorkerStartTime`, and `STABLE_RUN_RESET_MS` state. The
ChildWorkerSupervisor's D1 lastExitCode track + D2 clean-restart
budget apply to autopilot for free.
Shutdown now drains via the supervisor's killChild + awaitChildExit
typed surface instead of reaching into `workerProc` directly. The
onMaxCrashesExceeded callback routes through autopilot's existing
shutdown('max_crashes') path so the lockfile gets cleaned up
(pre-refactor, the inline loop called process.exit(1) directly and
bypassed the cleanup).
Regression coverage in test/autopilot-supervisor-wiring.test.ts:
static-shape grep guards for `--max-rss 2048`, `maxCrashes: 5`,
the shutdown-via-callback wiring, and absence of the legacy inline
names (startWorker, workerProc, crashCount, lastWorkerStartTime,
STABLE_RUN_RESET_MS).
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): parse RssAnon as field-presence + soften OOM docstring
Two follow-ups to the RssAnon watchdog fix (b81c598f), both surfaced
during plan-eng-review by Codex.
M1: getAccurateRss() used `if (anonKb > 0) return ...` to decide
whether to use the /proc/self/status reading or fall back to
process.memoryUsage().rss. That conflated "RssAnon field missing"
(old kernel, non-Linux) with "RssAnon field present but zero" (a
near-empty worker process whose only memory is shmem). The legitimate
shmem-only worker case fell through to VmRSS even though /proc had a
valid reading.
Fix: split the pure parser (parseRssFromProcStatus) into a separate
exported function that checks field presence via regex match, not
value comparison. Returns null only when the field text doesn't
match `^RssAnon:\s+(\d+)` AND `^RssShmem:\s+(\d+)`. Both fields
present + both zero is now a valid reading of 0 bytes.
M2: the docstring claimed RssAnon + RssShmem was "the memory that
actually matters for OOM risk." Codex pushed back: this is correct
for per-process leak detection but NOT a full container-OOM metric,
because cgroup memory pressure includes page cache. Soften to
"non-file-backed resident memory used for per-process leak
detection" and call out the cgroup caveat explicitly.
getAccurateRss now takes an optional readStatus function for
testability. Production callers use the default; tests inject
canned status text to cover the M1 regression and the fallback paths
without mocking the filesystem.
Tests: 11 cases covering parseRssFromProcStatus (normal, M1 regression
with anon=0 + shmem>0, both-zero, missing fields, malformed values,
shmem-only) and getAccurateRss (injected reader, ENOENT fallback,
old-kernel fallback, malformed-value fallback).
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(minions): awaitChildExit short-circuits when child already exited
Pre-fix, awaitChildExit registered `child.once('exit', ...)` without
checking whether the child had already terminated. If the child drained
between killChild('SIGTERM') and awaitChildExit() — common on fast
SIGTERM responders — Node's 'exit' event had already fired, the late
listener never resolved, and the caller waited out the full timeout.
On the supervisor's clean shutdown path that's a 35-second hang on
every quick child.
Probe `child.exitCode` and `child.signalCode` first; resolve
immediately when either is non-null. Sub-second clean shutdown
restored.
Pre-existing in the legacy supervisor.ts shape (same bug pattern),
but since the refactor consolidates child-process management into one
class, fix the pattern at the new seam.
Regression test in test/child-worker-supervisor.test.ts: run one full
spawn cycle, then call awaitChildExit on the already-finished cycle
and assert it returns in under 200ms (well under any test timeout).
Surfaced during pre-landing /review on the fix wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.34.3.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: update CLAUDE.md key-files entries for v0.34.3.0
Reflects the ChildWorkerSupervisor extraction shipped in this branch:
- Add new entry for src/core/minions/child-worker-supervisor.ts
covering D1 lastExitCode classifier, D2 clean-restart budget, the
awaitChildExit short-circuit, and test pinning at
test/child-worker-supervisor.test.ts
- Update src/core/minions/supervisor.ts entry to note the spawn-loop
extraction into the shared core + the byte-compatible event-shape
mapping that preserves JSONL audit consumers
- Update src/commands/autopilot.ts entry to note the parallel-
supervisor elimination + the shutdown-via-callback wiring
- Update src/core/minions/worker.ts entry with the new RssAnon /
getAccurateRss exports + the M1 field-presence parser fix
Regenerated llms-full.txt to match (per project rule: every CLAUDE.md
edit must be followed by bun run build:llms).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
126 lines
4.8 KiB
TypeScript
126 lines
4.8 KiB
TypeScript
/**
|
|
* Tests for the per-process RSS reader used by the worker's leak watchdog.
|
|
*
|
|
* `process.memoryUsage().rss` returns VmRSS which counts file-backed mmap
|
|
* pages (e.g. git packfiles). On a 96K-page brain repo, git operations
|
|
* inflate VmRSS to 7GB+ while heap is ~100MB — the kernel will reclaim
|
|
* those pages under pressure so they should not count toward the watchdog
|
|
* threshold. The fix reads `/proc/self/status` for RssAnon + RssShmem
|
|
* (the non-file-backed pages) on Linux and falls back to VmRSS elsewhere.
|
|
*
|
|
* These tests pin the parser shape against the M1 regression Codex
|
|
* surfaced during eng review: the prior `if (anonKb > 0)` form conflated
|
|
* "field missing" with "field present but zero", which broke the
|
|
* shmem-only worker case.
|
|
*/
|
|
|
|
import { describe, it, expect } from 'bun:test';
|
|
import { getAccurateRss, parseRssFromProcStatus } from '../src/core/minions/worker.ts';
|
|
|
|
describe('parseRssFromProcStatus', () => {
|
|
it('parses normal RssAnon + RssShmem to bytes', () => {
|
|
const status = [
|
|
'Name:\tworker',
|
|
'VmRSS:\t1024 kB',
|
|
'RssAnon:\t1024 kB',
|
|
'RssShmem:\t0 kB',
|
|
'VmSize:\t8192 kB',
|
|
].join('\n');
|
|
expect(parseRssFromProcStatus(status)).toBe(1024 * 1024);
|
|
});
|
|
|
|
it('M1 regression: RssAnon:0 + RssShmem>0 returns shmem-only bytes (not null)', () => {
|
|
// The pre-fix code did `if (anonKb > 0) return ...` which would treat
|
|
// this case as "field missing" and fall through to VmRSS. The fix uses
|
|
// field-presence checks so anon=0 + shmem=512 yields 512 KiB → 524_288.
|
|
const status = [
|
|
'RssAnon:\t0 kB',
|
|
'RssShmem:\t512 kB',
|
|
].join('\n');
|
|
expect(parseRssFromProcStatus(status)).toBe(512 * 1024);
|
|
});
|
|
|
|
it('returns null when neither RssAnon nor RssShmem is present (old kernel)', () => {
|
|
// Linux kernels older than 4.5 don't expose these fields. Returning
|
|
// null signals "fall back to VmRSS" to the caller.
|
|
const status = [
|
|
'Name:\tworker',
|
|
'VmRSS:\t1024 kB',
|
|
'VmSize:\t8192 kB',
|
|
].join('\n');
|
|
expect(parseRssFromProcStatus(status)).toBeNull();
|
|
});
|
|
|
|
it('treats non-numeric fields as absent (regex matches digits only)', () => {
|
|
// The `\d+` regex won't match "notanumber", so RssAnon is treated as
|
|
// absent and the result reflects only the well-formed RssShmem field.
|
|
// This is the right behavior: a corrupt RssAnon line shouldn't
|
|
// poison an otherwise valid RssShmem reading.
|
|
const status = [
|
|
'RssAnon:\tnotanumber kB',
|
|
'RssShmem:\t0 kB',
|
|
].join('\n');
|
|
expect(parseRssFromProcStatus(status)).toBe(0);
|
|
});
|
|
|
|
it('returns null when ALL fields are non-numeric', () => {
|
|
const status = [
|
|
'RssAnon:\tnotanumber kB',
|
|
'RssShmem:\talsobad kB',
|
|
].join('\n');
|
|
expect(parseRssFromProcStatus(status)).toBeNull();
|
|
});
|
|
|
|
it('returns sum when only RssShmem is present (anon missing)', () => {
|
|
// Symmetric to the regression: if only RssShmem is exposed for some
|
|
// reason, we should still use it. Treats anonKb default as 0.
|
|
const status = [
|
|
'RssShmem:\t256 kB',
|
|
'VmRSS:\t9999 kB',
|
|
].join('\n');
|
|
expect(parseRssFromProcStatus(status)).toBe(256 * 1024);
|
|
});
|
|
|
|
it('M1 regression: explicit RssAnon:0 + RssShmem:0 returns 0 (not null)', () => {
|
|
// Both fields present, both zero is a valid reading for a near-empty
|
|
// worker process. Should be treated as 0 bytes, NOT a missing reading.
|
|
const status = [
|
|
'RssAnon:\t0 kB',
|
|
'RssShmem:\t0 kB',
|
|
].join('\n');
|
|
expect(parseRssFromProcStatus(status)).toBe(0);
|
|
});
|
|
});
|
|
|
|
describe('getAccurateRss', () => {
|
|
it('uses the parsed value when readStatus returns a valid /proc/self/status', () => {
|
|
const fakeStatus = 'RssAnon:\t2048 kB\nRssShmem:\t0 kB\n';
|
|
expect(getAccurateRss(() => fakeStatus)).toBe(2048 * 1024);
|
|
});
|
|
|
|
it('falls back to process.memoryUsage().rss when readStatus throws (non-Linux)', () => {
|
|
const result = getAccurateRss(() => {
|
|
throw Object.assign(new Error('ENOENT: no such file'), { code: 'ENOENT' });
|
|
});
|
|
// We can't assert the exact RSS value cross-platform, but it MUST be
|
|
// a positive integer (process.memoryUsage().rss is always set).
|
|
expect(result).toBeGreaterThan(0);
|
|
expect(Number.isInteger(result)).toBe(true);
|
|
});
|
|
|
|
it('falls back when status text has no RssAnon/RssShmem fields (old kernel)', () => {
|
|
const status = 'VmRSS:\t1024 kB\nVmSize:\t8192 kB\n';
|
|
const result = getAccurateRss(() => status);
|
|
// Falls back to process.memoryUsage().rss — a positive integer.
|
|
expect(result).toBeGreaterThan(0);
|
|
expect(Number.isInteger(result)).toBe(true);
|
|
});
|
|
|
|
it('falls back on malformed values rather than returning NaN', () => {
|
|
const status = 'RssAnon:\tNaN kB\n';
|
|
const result = getAccurateRss(() => status);
|
|
expect(Number.isNaN(result)).toBe(false);
|
|
expect(result).toBeGreaterThan(0);
|
|
});
|
|
});
|