Files
gbrain/test/worker-rss.test.ts
T
668254b05e v0.34.3.0 fix: supervisor treats code=0 watchdog exits as crashes (#1003)
* fix: supervisor treats code=0 watchdog exits as crashes

The RSS watchdog triggers gracefulShutdown() which exits with code 0.
The supervisor was counting ALL exits < 5min as crashes, including
clean code=0 exits. After 10 watchdog-triggered restarts (typical with
a 96K-page brain where autopilot inflates RSS), the supervisor gave up
with max_crashes_exceeded.

Fix: code=0 exits reset crashCount to 0 and restart immediately with
no backoff. Only code≠0 exits count toward the crash limit.

Root cause: process.memoryUsage().rss reports 7GB during autopilot
sync on large repos (possibly shared page inflation from git mmap).
The 4096MB threshold triggers on every cycle. This is a separate
issue (RSS measurement accuracy) but the supervisor should handle
clean exits regardless.

* fix: use RssAnon instead of VmRSS for watchdog threshold

process.memoryUsage().rss returns VmRSS which includes file-backed
mmap'd pages. On repos with large git packfiles (96K+ pages), git
operations inflate VmRSS to 7GB+ while actual heap usage is ~100MB.
The kernel reclaims these pages under memory pressure — they're cache.

Replace with /proc/self/status RssAnon + RssShmem which measures only
anonymous pages (heap, stack, anonymous mmap). This is the memory that
actually matters for OOM risk.

Falls back to process.memoryUsage().rss on non-Linux.

Before: watchdog triggers every autopilot cycle (7GB VmRSS > 4GB threshold)
After:  watchdog only triggers on real memory growth (~100MB << 4GB threshold)

Related: #1002 (supervisor crash-count fix for the same symptom)

* refactor(minions): extract ChildWorkerSupervisor with D1/D2 amendments

MinionSupervisor and src/commands/autopilot.ts each owned a separate
spawn-and-respawn loop. PR #1003 fixed the supervisor's crash-counter
bug (counting code=0 watchdog drains as crashes) but the autopilot
loop has the same bug class. Worse, the as-shipped #1003 fix reset
crashCount=0 on every code=0 exit, which lost the "flapping worker"
signal in mixed-exit sequences.

Extract the shared spawn loop into ChildWorkerSupervisor so both
consumers compose one tested core. The new class bakes in two
amendments resolved during plan-eng-review:

D1 (lastExitCode track): code=0 exits no longer touch crashCount.
They emit ms:0 backoff and restart immediately, but the counter
survives across them. A worker alternating exit 1 / exit 0 / exit 1
correctly trips max_crashes; a worker drained 100 times by the
watchdog stays at crashCount=0 and runs forever (also correct).

D2 (clean-restart budget): on platforms where the watchdog measures
VmRSS instead of RssAnon (macOS, kernel <4.5, restricted containers),
a perpetually over-threshold worker could clean-exit in a tight loop
with no observability. New `cleanRestartBudget` option (default 10
clean restarts per 60s window) emits a `health_warn` and applies
backoff once exceeded.

The supervisor now delegates spawn/respawn/backoff to the inner
class and maps ChildSupervisorEvent → existing SupervisorEvent
emit() channel so JSONL audit consumers see byte-compatible output.
PID lock, signal handlers, health check, and process.exit on
max-crashes stay in MinionSupervisor (those are standalone-daemon
concerns the autopilot composer doesn't need).

Tests: 6 new ChildWorkerSupervisor cases (D1 classifier, interleaved
exits, stable-run + clean-exit interaction, D2 budget tripping, per-
instance config isolation, event shape regression). Existing supervisor
tests updated to use exit-1 workers where they previously relied on
clean-exit-as-crash semantics; their assertions (env plumbing, PID
lock, audit shape) are unaffected.

Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(autopilot): compose ChildWorkerSupervisor instead of inline spawn loop

src/commands/autopilot.ts:165-197 used to have its own spawn-and-
respawn loop separate from MinionSupervisor's. It hardcoded
maxCrashes=5, fixed 10s backoff, and counted every exit (including
code=0) toward the crash limit. Codex flagged this during plan-eng
review: the parallel implementation had the same bug class fixed
in #1003, just on a different code path. Anyone running
`gbrain autopilot` as a long-running daemon (instead of
`gbrain jobs supervisor`) would hit it.

Replace the inline `startWorker` + `child.on('exit')` block with
a ChildWorkerSupervisor instance. Drops the parallel `crashCount`,
`lastWorkerStartTime`, and `STABLE_RUN_RESET_MS` state. The
ChildWorkerSupervisor's D1 lastExitCode track + D2 clean-restart
budget apply to autopilot for free.

Shutdown now drains via the supervisor's killChild + awaitChildExit
typed surface instead of reaching into `workerProc` directly. The
onMaxCrashesExceeded callback routes through autopilot's existing
shutdown('max_crashes') path so the lockfile gets cleaned up
(pre-refactor, the inline loop called process.exit(1) directly and
bypassed the cleanup).

Regression coverage in test/autopilot-supervisor-wiring.test.ts:
static-shape grep guards for `--max-rss 2048`, `maxCrashes: 5`,
the shutdown-via-callback wiring, and absence of the legacy inline
names (startWorker, workerProc, crashCount, lastWorkerStartTime,
STABLE_RUN_RESET_MS).

Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(worker): parse RssAnon as field-presence + soften OOM docstring

Two follow-ups to the RssAnon watchdog fix (b81c598f), both surfaced
during plan-eng-review by Codex.

M1: getAccurateRss() used `if (anonKb > 0) return ...` to decide
whether to use the /proc/self/status reading or fall back to
process.memoryUsage().rss. That conflated "RssAnon field missing"
(old kernel, non-Linux) with "RssAnon field present but zero" (a
near-empty worker process whose only memory is shmem). The legitimate
shmem-only worker case fell through to VmRSS even though /proc had a
valid reading.

Fix: split the pure parser (parseRssFromProcStatus) into a separate
exported function that checks field presence via regex match, not
value comparison. Returns null only when the field text doesn't
match `^RssAnon:\s+(\d+)` AND `^RssShmem:\s+(\d+)`. Both fields
present + both zero is now a valid reading of 0 bytes.

M2: the docstring claimed RssAnon + RssShmem was "the memory that
actually matters for OOM risk." Codex pushed back: this is correct
for per-process leak detection but NOT a full container-OOM metric,
because cgroup memory pressure includes page cache. Soften to
"non-file-backed resident memory used for per-process leak
detection" and call out the cgroup caveat explicitly.

getAccurateRss now takes an optional readStatus function for
testability. Production callers use the default; tests inject
canned status text to cover the M1 regression and the fallback paths
without mocking the filesystem.

Tests: 11 cases covering parseRssFromProcStatus (normal, M1 regression
with anon=0 + shmem>0, both-zero, missing fields, malformed values,
shmem-only) and getAccurateRss (injected reader, ENOENT fallback,
old-kernel fallback, malformed-value fallback).

Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(minions): awaitChildExit short-circuits when child already exited

Pre-fix, awaitChildExit registered `child.once('exit', ...)` without
checking whether the child had already terminated. If the child drained
between killChild('SIGTERM') and awaitChildExit() — common on fast
SIGTERM responders — Node's 'exit' event had already fired, the late
listener never resolved, and the caller waited out the full timeout.
On the supervisor's clean shutdown path that's a 35-second hang on
every quick child.

Probe `child.exitCode` and `child.signalCode` first; resolve
immediately when either is non-null. Sub-second clean shutdown
restored.

Pre-existing in the legacy supervisor.ts shape (same bug pattern),
but since the refactor consolidates child-process management into one
class, fix the pattern at the new seam.

Regression test in test/child-worker-supervisor.test.ts: run one full
spawn cycle, then call awaitChildExit on the already-finished cycle
and assert it returns in under 200ms (well under any test timeout).

Surfaced during pre-landing /review on the fix wave.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.34.3.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md key-files entries for v0.34.3.0

Reflects the ChildWorkerSupervisor extraction shipped in this branch:

- Add new entry for src/core/minions/child-worker-supervisor.ts
  covering D1 lastExitCode classifier, D2 clean-restart budget, the
  awaitChildExit short-circuit, and test pinning at
  test/child-worker-supervisor.test.ts
- Update src/core/minions/supervisor.ts entry to note the spawn-loop
  extraction into the shared core + the byte-compatible event-shape
  mapping that preserves JSONL audit consumers
- Update src/commands/autopilot.ts entry to note the parallel-
  supervisor elimination + the shutdown-via-callback wiring
- Update src/core/minions/worker.ts entry with the new RssAnon /
  getAccurateRss exports + the M1 field-presence parser fix

Regenerated llms-full.txt to match (per project rule: every CLAUDE.md
edit must be followed by bun run build:llms).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 21:06:42 -07:00

126 lines
4.8 KiB
TypeScript

/**
* Tests for the per-process RSS reader used by the worker's leak watchdog.
*
* `process.memoryUsage().rss` returns VmRSS which counts file-backed mmap
* pages (e.g. git packfiles). On a 96K-page brain repo, git operations
* inflate VmRSS to 7GB+ while heap is ~100MB — the kernel will reclaim
* those pages under pressure so they should not count toward the watchdog
* threshold. The fix reads `/proc/self/status` for RssAnon + RssShmem
* (the non-file-backed pages) on Linux and falls back to VmRSS elsewhere.
*
* These tests pin the parser shape against the M1 regression Codex
* surfaced during eng review: the prior `if (anonKb > 0)` form conflated
* "field missing" with "field present but zero", which broke the
* shmem-only worker case.
*/
import { describe, it, expect } from 'bun:test';
import { getAccurateRss, parseRssFromProcStatus } from '../src/core/minions/worker.ts';
describe('parseRssFromProcStatus', () => {
it('parses normal RssAnon + RssShmem to bytes', () => {
const status = [
'Name:\tworker',
'VmRSS:\t1024 kB',
'RssAnon:\t1024 kB',
'RssShmem:\t0 kB',
'VmSize:\t8192 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(1024 * 1024);
});
it('M1 regression: RssAnon:0 + RssShmem>0 returns shmem-only bytes (not null)', () => {
// The pre-fix code did `if (anonKb > 0) return ...` which would treat
// this case as "field missing" and fall through to VmRSS. The fix uses
// field-presence checks so anon=0 + shmem=512 yields 512 KiB → 524_288.
const status = [
'RssAnon:\t0 kB',
'RssShmem:\t512 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(512 * 1024);
});
it('returns null when neither RssAnon nor RssShmem is present (old kernel)', () => {
// Linux kernels older than 4.5 don't expose these fields. Returning
// null signals "fall back to VmRSS" to the caller.
const status = [
'Name:\tworker',
'VmRSS:\t1024 kB',
'VmSize:\t8192 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBeNull();
});
it('treats non-numeric fields as absent (regex matches digits only)', () => {
// The `\d+` regex won't match "notanumber", so RssAnon is treated as
// absent and the result reflects only the well-formed RssShmem field.
// This is the right behavior: a corrupt RssAnon line shouldn't
// poison an otherwise valid RssShmem reading.
const status = [
'RssAnon:\tnotanumber kB',
'RssShmem:\t0 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(0);
});
it('returns null when ALL fields are non-numeric', () => {
const status = [
'RssAnon:\tnotanumber kB',
'RssShmem:\talsobad kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBeNull();
});
it('returns sum when only RssShmem is present (anon missing)', () => {
// Symmetric to the regression: if only RssShmem is exposed for some
// reason, we should still use it. Treats anonKb default as 0.
const status = [
'RssShmem:\t256 kB',
'VmRSS:\t9999 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(256 * 1024);
});
it('M1 regression: explicit RssAnon:0 + RssShmem:0 returns 0 (not null)', () => {
// Both fields present, both zero is a valid reading for a near-empty
// worker process. Should be treated as 0 bytes, NOT a missing reading.
const status = [
'RssAnon:\t0 kB',
'RssShmem:\t0 kB',
].join('\n');
expect(parseRssFromProcStatus(status)).toBe(0);
});
});
describe('getAccurateRss', () => {
it('uses the parsed value when readStatus returns a valid /proc/self/status', () => {
const fakeStatus = 'RssAnon:\t2048 kB\nRssShmem:\t0 kB\n';
expect(getAccurateRss(() => fakeStatus)).toBe(2048 * 1024);
});
it('falls back to process.memoryUsage().rss when readStatus throws (non-Linux)', () => {
const result = getAccurateRss(() => {
throw Object.assign(new Error('ENOENT: no such file'), { code: 'ENOENT' });
});
// We can't assert the exact RSS value cross-platform, but it MUST be
// a positive integer (process.memoryUsage().rss is always set).
expect(result).toBeGreaterThan(0);
expect(Number.isInteger(result)).toBe(true);
});
it('falls back when status text has no RssAnon/RssShmem fields (old kernel)', () => {
const status = 'VmRSS:\t1024 kB\nVmSize:\t8192 kB\n';
const result = getAccurateRss(() => status);
// Falls back to process.memoryUsage().rss — a positive integer.
expect(result).toBeGreaterThan(0);
expect(Number.isInteger(result)).toBe(true);
});
it('falls back on malformed values rather than returning NaN', () => {
const status = 'RssAnon:\tNaN kB\n';
const result = getAccurateRss(() => status);
expect(Number.isNaN(result)).toBe(false);
expect(result).toBeGreaterThan(0);
});
});