mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 19:49:14 +00:00
c78c3d0135
Production worker freezes silently every few hours. RSS climbs 68 MB → ~15 GB over ~7 hours, the worker stops claiming jobs but never crashes (no OOM, no SIGSEGV), the cron keeps enqueuing autopilot-cycle jobs every 5 minutes into a queue nobody is draining, and within 2-3 hours the queue piles up to 28+ waiting jobs. Shell jobs in flight when the worker froze hit max_stalled and dead-letter, producing 18% shell-job failure rate over 24h. Three in-repo defenses close the cascade end-to-end while the underlying memory leak gets investigated separately: 1. RSS watchdog (worker.ts): per-job AND 60s periodic check; on trip fires shutdownAbort + per-job aborts BEFORE stop(), so shell handlers run their SIGTERM→5s→SIGKILL cleanup and cooperative handlers bail instead of eating the 30s drain. Closes the zombie-shell-children gap. Default 2048 MB on supervisor; bare `gbrain jobs work` stays opt-in to preserve large embed/import working sets. 2. connectWithRetry (db.ts + cli.ts): wraps engine.connect() default-on, 3 attempts with 1s/2s/4s backoff. 5-pattern transient-error matcher (auth failed, connection refused, db starting, terminated, ECONNRESET); permanent errors do NOT retry. Operators can opt out per-call via --no-retry-connect or GBRAIN_NO_RETRY_CONNECT=1. Fixes PgBouncer cold- start auth races on autopilot/dream/jobs daemons. 3. autopilot-cycle backpressure: queue.add now passes maxWaiting:1 (1 active + 1 waiting; coalesce 3rd+). Combined with idempotency_key, cross-slot pile-ups are bounded. Autopilot's worker spawn loop also gets the supervisor's stable-run reset pattern (5min uptime → reset crash count) so hourly watchdog exits don't trip the 5-crash give-up threshold. Reviewed via /plan-eng-review (5 arch + 1 test issue, all resolved) and /codex (6 additional findings B1-B6 surfaced real bugs the eng review missed; all resolved Codex's way). 11 new tests across watchdog (5 cases including the production-freeze-regression scenario where zero jobs ever complete), connectWithRetry (6 cases), and supervisor argv (1 case). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>