mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
a7b79b66d4
Engine-level retry primitive that closes the v0.41.17 production incident
where ~3,000 wiki links + timeline entries were silently lost per dream
cycle on a 16K-page brain. Supavisor's circuit-breaker takes 5-10s to
recover; the prior single-500ms-retry shape couldn't survive it.
ARCHITECTURE
============
Retry becomes a data-primitive contract, not a caller responsibility.
postgres-engine.ts + pglite-engine.ts now self-retry inside addLinksBatch,
addTimelineEntriesBatch, and upsertChunks. Every caller — current AND
future — inherits retry-for-free. CI lint guard `scripts/check-no-double-retry.sh`
fails the build if anyone re-wraps an engine batch method (preventing
3×3=9 retry amplification on incomplete reverts).
CODEX-HARDENED DEFAULTS
=======================
BULK_RETRY_OPTS = {maxRetries:3, delayMs:1000, delayMaxMs:10000,
jitter:'decorrelated'}. Total worst-case wait ≈12s covers full Supavisor
recovery window. Decorrelated jitter (AWS-style uniform(base, prevDelay*3)
capped at maxDelay) replaces 'full' which allowed near-zero retries that
re-hit the still-recovering breaker.
AbortSignal threading from MinionWorker.shutdownAbort.signal through
engine method opts → withRetry → abortableSleep. SIGTERM aborts sleeping
retries instead of blocking deploys for up to delayMaxMs.
OBSERVABILITY
=============
`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl` records every retry event
(success-after-blip AND exhausted-retries). Built on the v0.40.4.0
audit-writer cathedral. Privacy posture: never logs slugs / page IDs /
content (mirrors shell-audit.ts).
`gbrain doctor` learns `batch_retry_health` check. Reads last 24h
(not 7d — codex H-9: avoid permanent noise from one historical blip).
Thresholds: ok (zero or <3 same-site), warn (>=3 same-site OR >=5
cross-site), fail (>=20 sustained breaker). Surfaces bad GBRAIN_BULK_*
env at startup (codex M-10). Corrupt-JSONL tolerant.
30-day audit pruning hooked into the dream cycle's purge phase (codex H-8
— implements the 'pruning convention' for real).
OPERATOR TUNING
===============
GBRAIN_BULK_MAX_RETRIES (int >= 0; 0 disables retries for debugging)
GBRAIN_BULK_RETRY_BASE_MS (int > 0)
GBRAIN_BULK_RETRY_MAX_MS (int >= base)
Bad values throw GBrainError with paste-ready fix hints at doctor startup,
not at first-retry mid-cycle.
VERIFICATION
============
- bun run verify: 28/28 checks green (includes 2 new lint guards:
check-no-double-retry, check-batch-audit-site)
- bun run test: 11453 pass / 1 pre-existing flake (schema-cli.test.ts —
confirmed by running on clean master, NOT introduced by this wave)
- bun run test:slow: 40/40 including new test/core/retry-stress.slow.test.ts
(100 batches × 30% blip rate × decorrelated jitter, zero row loss)
- bunx tsc --noEmit: 0 errors
REVIEWS
=======
- CEO review (SELECTIVE EXPANSION): 4 cherry-picks proposed, 4 accepted
- Eng review (2 passes): 10 findings, 0 critical gaps, architectural
pivot from per-site to engine-level wrap
- Codex independent review: 23 findings; 10 critical/high absorbed
(decorrelated jitter, 12s backoff window, AbortSignal, idempotency
proof, backfill unification, typed audit-site enum, doctor expiry
thresholds, audit pruning, env validation at doctor startup)
PR #1523 closed and absorbed (@garrytan-agents original extract.ts fix
preserved via co-author trailer; 5 test cases moved to test/core/retry.test.ts
with assertions adjusted for the v0.41.19.0 BULK_RETRY_OPTS defaults).
Co-authored-by: garrytan-agents <noreply@anthropic.com>
64 lines
2.8 KiB
Bash
Executable File
64 lines
2.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# v0.41.18.0 — CI guard against double-retry hazard.
|
||
#
|
||
# Engine batch methods (addLinksBatch / addTimelineEntriesBatch /
|
||
# upsertChunks) self-retry via withRetry(BULK_RETRY_OPTS) inside the engine
|
||
# implementation. Wrapping them ALSO at the call site produces 3×3=9 retry
|
||
# attempts under failure, amplifying load on a recovering circuit breaker
|
||
# and worsening the very incident the wave was designed to fix.
|
||
#
|
||
# This script greps src/ for the pattern and fails the build if found.
|
||
# Catches the migration-ordering hazard from the v0.41.18.0 eng review (D6)
|
||
# AND prevents future refactors from re-introducing the bug class.
|
||
#
|
||
# Usage: scripts/check-no-double-retry.sh
|
||
# Exit: 0 when no matches, 1 when matches found.
|
||
|
||
set -euo pipefail
|
||
|
||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||
cd "$ROOT"
|
||
|
||
# Match: withRetry(...) wrapping any of the 3 engine batch methods.
|
||
# The greedy `.*` between `withRetry(` and `engine.` covers both the
|
||
# arrow-fn form and any direct invocation. (gbrain-allow-direct-insert: doc comment)
|
||
# Multi-line wraps are caught by `grep -E` per file (line-wise) for the
|
||
# common single-line case; multi-line wraps still get caught by a separate
|
||
# multi-line pass below.
|
||
PATTERN='withRetry\([^)]*engine\.(addLinksBatch|addTimelineEntriesBatch|upsertChunks)'
|
||
|
||
# Single-line scan (covers ~95% of real cases).
|
||
if grep -rEn "$PATTERN" src/ --include='*.ts' 2>/dev/null; then
|
||
echo
|
||
echo "ERROR: Found withRetry(...engine.{addLinksBatch|addTimelineEntriesBatch|upsertChunks})"
|
||
echo " pattern in src/."
|
||
echo
|
||
echo " Engine batch methods self-retry via withRetry(BULK_RETRY_OPTS) in"
|
||
echo " postgres-engine.ts + pglite-engine.ts. Wrapping AGAIN at the call site"
|
||
echo " produces 3×3=9 retry attempts under failure, amplifying load on a"
|
||
echo " recovering circuit breaker."
|
||
echo
|
||
echo " Fix: delete the outer withRetry wrap. Pass auditSite as a kwarg:"
|
||
echo " await engine.addLinks(batch, { auditSite: 'extract.links_inc' }); // example"
|
||
echo
|
||
echo " Audit JSONL records the retries silently at "
|
||
echo " ~/.gbrain/audit/batch-retry-YYYY-Www.jsonl; check"
|
||
echo " \`gbrain doctor\` for the batch_retry_health surface."
|
||
exit 1
|
||
fi
|
||
|
||
# Multi-line scan: a withRetry( on one line and the engine call on the next
|
||
# few. Bounded to 3-line window so we don't flag distant unrelated calls.
|
||
# Uses pcregrep if available, else falls back to a simple awk window.
|
||
if command -v pcregrep >/dev/null 2>&1; then
|
||
if pcregrep -r -M -n --include='\.ts$' \
|
||
'withRetry\([^)]*\n\s*\(?[^)]*=>\s*engine\.(addLinksBatch|addTimelineEntriesBatch|upsertChunks)' \
|
||
src/ 2>/dev/null; then
|
||
echo
|
||
echo "ERROR: Multi-line withRetry(...engine.batch...) wrap found in src/. See above."
|
||
exit 1
|
||
fi
|
||
fi
|
||
|
||
echo "OK: no withRetry(...engine.batch...) double-retry patterns in src/"
|