Production worker freezes silently every few hours. RSS climbs 68 MB → ~15 GB
over ~7 hours, the worker stops claiming jobs but never crashes (no OOM, no
SIGSEGV), the cron keeps enqueuing autopilot-cycle jobs every 5 minutes into a
queue nobody is draining, and within 2-3 hours the queue piles up to 28+
waiting jobs. Shell jobs in flight when the worker froze hit max_stalled and
dead-letter, producing 18% shell-job failure rate over 24h.
Three in-repo defenses close the cascade end-to-end while the underlying
memory leak gets investigated separately:
1. RSS watchdog (worker.ts): per-job AND 60s periodic check; on trip fires
shutdownAbort + per-job aborts BEFORE stop(), so shell handlers run their
SIGTERM→5s→SIGKILL cleanup and cooperative handlers bail instead of
eating the 30s drain. Closes the zombie-shell-children gap. Default 2048
MB on supervisor; bare `gbrain jobs work` stays opt-in to preserve large
embed/import working sets.
2. connectWithRetry (db.ts + cli.ts): wraps engine.connect() default-on,
3 attempts with 1s/2s/4s backoff. 5-pattern transient-error matcher
(auth failed, connection refused, db starting, terminated, ECONNRESET);
permanent errors do NOT retry. Operators can opt out per-call via
--no-retry-connect or GBRAIN_NO_RETRY_CONNECT=1. Fixes PgBouncer cold-
start auth races on autopilot/dream/jobs daemons.
3. autopilot-cycle backpressure: queue.add now passes maxWaiting:1 (1 active
+ 1 waiting; coalesce 3rd+). Combined with idempotency_key, cross-slot
pile-ups are bounded. Autopilot's worker spawn loop also gets the
supervisor's stable-run reset pattern (5min uptime → reset crash count)
so hourly watchdog exits don't trip the 5-crash give-up threshold.
Reviewed via /plan-eng-review (5 arch + 1 test issue, all resolved) and
/codex (6 additional findings B1-B6 surfaced real bugs the eng review
missed; all resolved Codex's way). 11 new tests across watchdog (5 cases
including the production-freeze-regression scenario where zero jobs ever
complete), connectWithRetry (6 cases), and supervisor argv (1 case).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
