mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
fix: supervisor treats code=0 watchdog exits as crashes
The RSS watchdog triggers gracefulShutdown() which exits with code 0. The supervisor was counting ALL exits < 5min as crashes, including clean code=0 exits. After 10 watchdog-triggered restarts (typical with a 96K-page brain where autopilot inflates RSS), the supervisor gave up with max_crashes_exceeded. Fix: code=0 exits reset crashCount to 0 and restart immediately with no backoff. Only code≠0 exits count toward the crash limit. Root cause: process.memoryUsage().rss reports 7GB during autopilot sync on large repos (possibly shared page inflation from git mmap). The 4096MB threshold triggers on every cycle. This is a separate issue (RSS measurement accuracy) but the supervisor should handle clean exits regardless.
This commit is contained in:
@@ -408,11 +408,17 @@ export class MinionSupervisor {
|
||||
return;
|
||||
}
|
||||
|
||||
// Clean exits (crashCount=0) restart immediately with no backoff.
|
||||
// crashCount - 1 is the retry-attempt index (0-based exponent for backoff math).
|
||||
// On first crash: crashCount=1, backoff exponent=0 → 1s.
|
||||
// After stable-run reset: crashCount=1 again → 1s fresh cycle.
|
||||
// Test-only: _backoffFloorMs short-circuits to a fixed tiny value so integration
|
||||
// tests can exercise crash loops in < 1s without waiting for the real curve.
|
||||
if (this.crashCount === 0) {
|
||||
// Clean exit — restart immediately, no backoff
|
||||
this.emit('backoff', { ms: 0, crash_count: 0 });
|
||||
continue;
|
||||
}
|
||||
const backoff = this.opts._backoffFloorMs !== undefined
|
||||
? this.opts._backoffFloorMs
|
||||
: calculateBackoffMs(this.crashCount - 1);
|
||||
@@ -513,11 +519,17 @@ export class MinionSupervisor {
|
||||
return;
|
||||
}
|
||||
|
||||
// Classify exit: code=0 is a clean exit (e.g. watchdog RSS drain),
|
||||
// not a crash. Only increment crashCount for actual failures (code≠0).
|
||||
// Stable-run reset: if the worker ran > 5min before crashing, we forgive
|
||||
// prior crash history and treat this as the first crash of a new cycle
|
||||
// (crashCount = 1, so backoff math uses retry-index 0 = 1s).
|
||||
const runDuration = Date.now() - this.lastStartTime;
|
||||
if (runDuration > 5 * 60 * 1000) {
|
||||
if (code === 0) {
|
||||
// Clean exit (watchdog drain, graceful stop). Don't count as crash.
|
||||
// Reset crash counter since the worker was healthy.
|
||||
this.crashCount = 0;
|
||||
} else if (runDuration > 5 * 60 * 1000) {
|
||||
this.crashCount = 1;
|
||||
} else {
|
||||
this.crashCount++;
|
||||
|
||||
Reference in New Issue
Block a user