admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a7b79b66d49efea02efcad30f43e5b4d281b224e
gbrain/package.json
T
Garry TanGitHubgarrytan-agents
 a7b79b66d4 feat: v0.41.19.0 Supavisor Retry Cathedral (#1537) 
Engine-level retry primitive that closes the v0.41.17 production incident
where ~3,000 wiki links + timeline entries were silently lost per dream
cycle on a 16K-page brain. Supavisor's circuit-breaker takes 5-10s to
recover; the prior single-500ms-retry shape couldn't survive it.

ARCHITECTURE
============

Retry becomes a data-primitive contract, not a caller responsibility.
postgres-engine.ts + pglite-engine.ts now self-retry inside addLinksBatch,
addTimelineEntriesBatch, and upsertChunks. Every caller — current AND
future — inherits retry-for-free. CI lint guard `scripts/check-no-double-retry.sh`
fails the build if anyone re-wraps an engine batch method (preventing
3×3=9 retry amplification on incomplete reverts).

CODEX-HARDENED DEFAULTS
=======================

BULK_RETRY_OPTS = {maxRetries:3, delayMs:1000, delayMaxMs:10000,
jitter:'decorrelated'}. Total worst-case wait ≈12s covers full Supavisor
recovery window. Decorrelated jitter (AWS-style uniform(base, prevDelay*3)
capped at maxDelay) replaces 'full' which allowed near-zero retries that
re-hit the still-recovering breaker.

AbortSignal threading from MinionWorker.shutdownAbort.signal through
engine method opts → withRetry → abortableSleep. SIGTERM aborts sleeping
retries instead of blocking deploys for up to delayMaxMs.

OBSERVABILITY
=============

`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl` records every retry event
(success-after-blip AND exhausted-retries). Built on the v0.40.4.0
audit-writer cathedral. Privacy posture: never logs slugs / page IDs /
content (mirrors shell-audit.ts).

`gbrain doctor` learns `batch_retry_health` check. Reads last 24h
(not 7d — codex H-9: avoid permanent noise from one historical blip).
Thresholds: ok (zero or <3 same-site), warn (>=3 same-site OR >=5
cross-site), fail (>=20 sustained breaker). Surfaces bad GBRAIN_BULK_*
env at startup (codex M-10). Corrupt-JSONL tolerant.

30-day audit pruning hooked into the dream cycle's purge phase (codex H-8
— implements the 'pruning convention' for real).

OPERATOR TUNING
===============

GBRAIN_BULK_MAX_RETRIES (int >= 0; 0 disables retries for debugging)
GBRAIN_BULK_RETRY_BASE_MS (int > 0)
GBRAIN_BULK_RETRY_MAX_MS (int >= base)

Bad values throw GBrainError with paste-ready fix hints at doctor startup,
not at first-retry mid-cycle.

VERIFICATION
============

- bun run verify: 28/28 checks green (includes 2 new lint guards:
  check-no-double-retry, check-batch-audit-site)
- bun run test: 11453 pass / 1 pre-existing flake (schema-cli.test.ts —
  confirmed by running on clean master, NOT introduced by this wave)
- bun run test:slow: 40/40 including new test/core/retry-stress.slow.test.ts
  (100 batches × 30% blip rate × decorrelated jitter, zero row loss)
- bunx tsc --noEmit: 0 errors

REVIEWS
=======

- CEO review (SELECTIVE EXPANSION): 4 cherry-picks proposed, 4 accepted
- Eng review (2 passes): 10 findings, 0 critical gaps, architectural
  pivot from per-site to engine-level wrap
- Codex independent review: 23 findings; 10 critical/high absorbed
  (decorrelated jitter, 12s backoff window, AbortSignal, idempotency
  proof, backfill unification, typed audit-site enum, doctor expiry
  thresholds, audit pruning, env validation at doctor startup)

PR #1523 closed and absorbed (@garrytan-agents original extract.ts fix
preserved via co-author trailer; 5 test cases moved to test/core/retry.test.ts
with assertions adjusted for the v0.41.19.0 BULK_RETRY_OPTS defaults).

Co-authored-by: garrytan-agents <noreply@anthropic.com>
2026-05-26 23:15:44 -07:00

145 lines
7.3 KiB
JSON
Raw Blame History

{
"name": "gbrain",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
"bin": {
"gbrain": "src/cli.ts"
},
"exports": {
".": "./src/core/index.ts",
"./engine": "./src/core/engine.ts",
"./types": "./src/core/types.ts",
"./operations": "./src/core/operations.ts",
"./minions": "./src/core/minions/index.ts",
"./engine-factory": "./src/core/engine-factory.ts",
"./pglite-engine": "./src/core/pglite-engine.ts",
"./link-extraction": "./src/core/link-extraction.ts",
"./import-file": "./src/core/import-file.ts",
"./transcription": "./src/core/transcription.ts",
"./embedding": "./src/core/embedding.ts",
"./config": "./src/core/config.ts",
"./markdown": "./src/core/markdown.ts",
"./backoff": "./src/core/backoff.ts",
"./search/hybrid": "./src/core/search/hybrid.ts",
"./search/expansion": "./src/core/search/expansion.ts",
"./ai/gateway": "./src/core/ai/gateway.ts",
"./extract": "./src/commands/extract.ts",
"./ingestion": "./src/core/ingestion/index.ts",
"./ingestion/test-harness": "./src/core/ingestion/test-harness.ts"
},
"scripts": {
"dev": "bun run src/cli.ts",
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
"build:admin": "cd admin && bun run build && cd .. && bun run scripts/build-admin-embedded.ts",
"build:admin-embedded": "bun run scripts/build-admin-embedded.ts",
"build:schema": "bash scripts/build-schema.sh",
"build:llms": "bun run scripts/build-llms.ts",
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "bash scripts/run-unit-parallel.sh",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bash scripts/run-verify-parallel.sh",
"check:source-config-leak": "scripts/check-source-config-leak.sh",
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
"check:system-of-record": "scripts/check-system-of-record.sh",
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
"check:cli-exec": "scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"test:slow": "bash scripts/run-slow-tests.sh",
"test:heavy": "bash scripts/run-heavy.sh",
"test:profile": "bash scripts/profile-tests.sh",
"test:serial": "bash scripts/run-serial-tests.sh",
"ci:local": "bash scripts/ci-local.sh",
"ci:local:diff": "bash scripts/ci-local.sh --diff",
"ci:select-e2e": "bun run scripts/select-e2e.ts",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:no-double-retry": "scripts/check-no-double-retry.sh",
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
"check:source-id-projection": "scripts/check-source-id-projection.sh",
"check:privacy": "scripts/check-privacy.sh",
"check:proposal-pii": "scripts/check-proposal-pii.sh",
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
"check:test-names": "scripts/check-test-real-names.sh",
"check:progress": "scripts/check-progress-to-stdout.sh",
"check:exports-count": "scripts/check-exports-count.sh",
"check:admin-build": "scripts/check-admin-build.sh",
"check:admin-embedded": "scripts/check-admin-embedded.sh",
"check:test-isolation": "scripts/check-test-isolation.sh",
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
"prepublish:clawhub": "bun run build:all",
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
},
"openclaw": {
"compat": {
"pluginApi": ">=2026.4.0"
},
"extensions": [
"./src/openclaw-context-engine.ts"
]
},
"dependencies": {
"@ai-sdk/anthropic": "^3.0.71",
"@ai-sdk/google": "^3.0.64",
"@ai-sdk/openai": "^3.0.53",
"@ai-sdk/openai-compatible": "^2.0.41",
"@anthropic-ai/sdk": "^0.30.0",
"@aws-sdk/client-s3": "^3.1028.0",
"@dqbd/tiktoken": "^1.0.22",
"@electric-sql/pglite": "0.4.3",
"@jsquash/avif": "^2.1.1",
"@jsquash/png": "^3.1.1",
"@modelcontextprotocol/sdk": "1.29.0",
"ai": "^6.0.168",
"chokidar": "^4.0.3",
"cookie-parser": "^1.4.7",
"cors": "^2.8.5",
"eventsource-parser": "^3.0.8",
"exifr": "^7.1.3",
"express": "^5.1.0",
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.14.2",
"marked": "^18.0.0",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
"postgres": "^3.4.0",
"tree-sitter-wasms": "0.1.13",
"web-tree-sitter": "0.22.6",
"zod": "^4.3.6"
},
"devDependencies": {
"@types/bun": "latest",
"@types/cookie-parser": "^1.4.7",
"@types/cors": "^2.8.19",
"@types/express": "^5.0.6",
"@types/js-yaml": "^3.12.10",
"bun-types": "^1.3.13",
"fast-check": "^4.8.0",
"typescript": "^5.6.0"
},
"trustedDependencies": [
"@electric-sql/pglite"
],
"engines": {
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.41.19.0"
}
Reference in New Issue View Git Blame Copy Permalink