mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
a7b79b66d4
Engine-level retry primitive that closes the v0.41.17 production incident
where ~3,000 wiki links + timeline entries were silently lost per dream
cycle on a 16K-page brain. Supavisor's circuit-breaker takes 5-10s to
recover; the prior single-500ms-retry shape couldn't survive it.
ARCHITECTURE
============
Retry becomes a data-primitive contract, not a caller responsibility.
postgres-engine.ts + pglite-engine.ts now self-retry inside addLinksBatch,
addTimelineEntriesBatch, and upsertChunks. Every caller — current AND
future — inherits retry-for-free. CI lint guard `scripts/check-no-double-retry.sh`
fails the build if anyone re-wraps an engine batch method (preventing
3×3=9 retry amplification on incomplete reverts).
CODEX-HARDENED DEFAULTS
=======================
BULK_RETRY_OPTS = {maxRetries:3, delayMs:1000, delayMaxMs:10000,
jitter:'decorrelated'}. Total worst-case wait ≈12s covers full Supavisor
recovery window. Decorrelated jitter (AWS-style uniform(base, prevDelay*3)
capped at maxDelay) replaces 'full' which allowed near-zero retries that
re-hit the still-recovering breaker.
AbortSignal threading from MinionWorker.shutdownAbort.signal through
engine method opts → withRetry → abortableSleep. SIGTERM aborts sleeping
retries instead of blocking deploys for up to delayMaxMs.
OBSERVABILITY
=============
`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl` records every retry event
(success-after-blip AND exhausted-retries). Built on the v0.40.4.0
audit-writer cathedral. Privacy posture: never logs slugs / page IDs /
content (mirrors shell-audit.ts).
`gbrain doctor` learns `batch_retry_health` check. Reads last 24h
(not 7d — codex H-9: avoid permanent noise from one historical blip).
Thresholds: ok (zero or <3 same-site), warn (>=3 same-site OR >=5
cross-site), fail (>=20 sustained breaker). Surfaces bad GBRAIN_BULK_*
env at startup (codex M-10). Corrupt-JSONL tolerant.
30-day audit pruning hooked into the dream cycle's purge phase (codex H-8
— implements the 'pruning convention' for real).
OPERATOR TUNING
===============
GBRAIN_BULK_MAX_RETRIES (int >= 0; 0 disables retries for debugging)
GBRAIN_BULK_RETRY_BASE_MS (int > 0)
GBRAIN_BULK_RETRY_MAX_MS (int >= base)
Bad values throw GBrainError with paste-ready fix hints at doctor startup,
not at first-retry mid-cycle.
VERIFICATION
============
- bun run verify: 28/28 checks green (includes 2 new lint guards:
check-no-double-retry, check-batch-audit-site)
- bun run test: 11453 pass / 1 pre-existing flake (schema-cli.test.ts —
confirmed by running on clean master, NOT introduced by this wave)
- bun run test:slow: 40/40 including new test/core/retry-stress.slow.test.ts
(100 batches × 30% blip rate × decorrelated jitter, zero row loss)
- bunx tsc --noEmit: 0 errors
REVIEWS
=======
- CEO review (SELECTIVE EXPANSION): 4 cherry-picks proposed, 4 accepted
- Eng review (2 passes): 10 findings, 0 critical gaps, architectural
pivot from per-site to engine-level wrap
- Codex independent review: 23 findings; 10 critical/high absorbed
(decorrelated jitter, 12s backoff window, AbortSignal, idempotency
proof, backfill unification, typed audit-site enum, doctor expiry
thresholds, audit pruning, env validation at doctor startup)
PR #1523 closed and absorbed (@garrytan-agents original extract.ts fix
preserved via co-author trailer; 5 test cases moved to test/core/retry.test.ts
with assertions adjusted for the v0.41.19.0 BULK_RETRY_OPTS defaults).
Co-authored-by: garrytan-agents <noreply@anthropic.com>
80 lines
2.7 KiB
Bash
Executable File
80 lines
2.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# v0.41.18.0 — CI guard against batch-audit-site typo drift (codex H-7).
|
|
#
|
|
# auditSite labels flow from call sites into the batch-retry audit JSONL
|
|
# and from there into `gbrain doctor`'s batch_retry_health check. A typo
|
|
# like `'extract.lnks_inc'` doesn't break compilation (TypeScript narrows
|
|
# string literals only via the BatchAuditSite type, but external string
|
|
# values escape this — e.g. config, environment, dynamic dispatch).
|
|
#
|
|
# This script extracts every string-literal `auditSite: '...'` value from
|
|
# src/ and validates it appears in the BATCH_AUDIT_SITES const list in
|
|
# src/core/retry.ts. Fails the build on mismatch.
|
|
#
|
|
# Usage: scripts/check-batch-audit-site.sh
|
|
# Exit: 0 when every literal matches the enum, 1 otherwise.
|
|
|
|
set -euo pipefail
|
|
|
|
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
|
cd "$ROOT"
|
|
|
|
RETRY_FILE="src/core/retry.ts"
|
|
if [ ! -f "$RETRY_FILE" ]; then
|
|
echo "ERROR: $RETRY_FILE missing — cannot validate audit sites."
|
|
exit 1
|
|
fi
|
|
|
|
# Extract every entry inside BATCH_AUDIT_SITES = [ ... ] as const.
|
|
# Strips quotes + trailing commas + whitespace. Strict awk window between
|
|
# the array open and closing `] as const`.
|
|
KNOWN_SITES=$(awk '
|
|
/BATCH_AUDIT_SITES = \[/ { capture = 1; next }
|
|
capture && /\] as const/ { capture = 0; exit }
|
|
capture {
|
|
# Pull out '\''xyz'\'' or "xyz" string literals on the line.
|
|
while (match($0, /['\''"]([^'\''"]+)['\''"]/)) {
|
|
print substr($0, RSTART + 1, RLENGTH - 2)
|
|
$0 = substr($0, RSTART + RLENGTH)
|
|
}
|
|
}
|
|
' "$RETRY_FILE" | sort -u)
|
|
|
|
if [ -z "$KNOWN_SITES" ]; then
|
|
echo "ERROR: Could not extract BATCH_AUDIT_SITES from $RETRY_FILE."
|
|
exit 1
|
|
fi
|
|
|
|
# Extract every `auditSite: '...'` literal from src/ (excluding retry.ts
|
|
# itself which contains the enum definition, and test files which are
|
|
# allowed to use synthetic sites for assertion scaffolding).
|
|
USED_SITES=$(
|
|
grep -rEh "auditSite:[[:space:]]*['\"][^'\"]+['\"]" src/ \
|
|
--include='*.ts' \
|
|
--exclude-dir=core/audit \
|
|
--exclude='retry.ts' \
|
|
| sed -E "s/.*auditSite:[[:space:]]*['\"]([^'\"]+)['\"].*/\1/" \
|
|
| sort -u
|
|
)
|
|
|
|
if [ -z "$USED_SITES" ]; then
|
|
echo "OK: no auditSite literals found in src/ (engines use defaults)"
|
|
exit 0
|
|
fi
|
|
|
|
UNKNOWN_SITES=$(comm -23 <(echo "$USED_SITES") <(echo "$KNOWN_SITES") || true)
|
|
|
|
if [ -n "$UNKNOWN_SITES" ]; then
|
|
echo "ERROR: Unknown auditSite literal(s) found in src/:"
|
|
echo "$UNKNOWN_SITES" | sed 's/^/ /'
|
|
echo
|
|
echo "Fix: add the value to BATCH_AUDIT_SITES in src/core/retry.ts."
|
|
echo " The enum is the closed list of known sites."
|
|
echo
|
|
echo "Known sites:"
|
|
echo "$KNOWN_SITES" | sed 's/^/ /'
|
|
exit 1
|
|
fi
|
|
|
|
echo "OK: all auditSite literals match BATCH_AUDIT_SITES enum"
|