mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
Garry TanandClaude Opus 4.6
7e98739a6b
Final wiring of BrainBench v1 Complete. all.ts now orchestrates the full
Cat catalog (1-12) via a mix of subprocess dispatch (Cats 1, 2, 3, 4, 6,
7, 10, 11, 12 — standalone runners with CLI entry points) and
programmatic invocation (Cats 5, 8, 9 — require runtime inputs that
can't come via CLI flags). Subprocess Cats run concurrently under a
p-limit(2) bound to cap peak memory around ~800MB (two PGLite instances
at ~400MB each).
Cats 5/8/9 show as "programmatic" in the report with a one-line
reference to their `runCatN({...})` harness API. They're deliberately
skipped from the master runner because their inputs (claim catalog,
probe catalog, scenario catalog, pre-seeded agent state, evidence
pagesBySlug) are task-specific and assembled at the caller.
**eval/runner/all.ts** — rewritten:
- CATEGORIES is a tagged union of SubprocessCategory | ProgrammaticCategory
- runCatSubprocess spawns Bun with pipe'd stdout/stderr, 10-min timeout
per Cat (124 exit + SIGTERM on timeout; no hung subprocesses)
- runConcurrently is a bounded worker pool preserving input order
- buildReport emits the full markdown with per-Cat elapsed times,
migration-noise filter, and a separate programmatic-only section
- Honors BRAINBENCH_N (1/5/10 for smoke/iteration/published),
BRAINBENCH_CONCURRENCY (default 2),
BRAINBENCH_LLM_CONCURRENCY (default 4, consumed by llm-budget)
**eval/runner/llm-budget.ts** — shared LLM rate-limit semaphore. A full
N=10 published scorecard makes ~900 Anthropic calls (150 Cat 8/9 probes
× N=10 + 100 Cat 5 claims × N=10). Without coordination, concurrent
adapters trigger 429s on per-minute limits.
- LlmBudget class: acquireSlot/releaseSlot + withLlmSlot(fn) wrapper
that releases on success AND throw (try/finally)
- getDefaultLlmBudget() singleton reads BRAINBENCH_LLM_CONCURRENCY,
falls back to 4 on missing/garbage values
- capacity enforced ≥1 (rejects 0/negative)
- Double-release is a no-op (guards against upstream double-call bugs)
- Active + waiting counts exposed for observability / tests
**package.json** scripts:
- eval:brainbench — default N=5 iteration
- eval:brainbench:smoke — N=1 for fast iteration
- eval:brainbench:published — N=10 for committed baselines
- eval:cat6 / eval:cat11 — individual new subprocess Cats
**Tests (24):** CATEGORIES catalog enforces the exact Cat-number partition
(subprocess: 1,2,3,4,6,7,10,11,12; programmatic: 5,8,9). runConcurrently
respects the cap (observable via peak in-flight counter), preserves input
order under non-uniform delays, handles empty input. LlmBudget enforces
capacity, releases on throw, honors env var, rejects 0/negative.
buildReport filters migration noise, counts passed/failed/programmatic
correctly, includes every Cat + programmatic-only section.
Full eval suite now: 314 pass, 0 fail (15 test files).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
67 lines
2.8 KiB
JSON
67 lines
2.8 KiB
JSON
{
|
|
"name": "gbrain",
|
|
"version": "0.15.0",
|
|
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
|
"type": "module",
|
|
"main": "src/core/index.ts",
|
|
"bin": {
|
|
"gbrain": "src/cli.ts"
|
|
},
|
|
"exports": {
|
|
".": "./src/core/index.ts",
|
|
"./engine": "./src/core/engine.ts",
|
|
"./types": "./src/core/types.ts",
|
|
"./operations": "./src/core/operations.ts",
|
|
"./minions": "./src/core/minions/index.ts",
|
|
"./engine-factory": "./src/core/engine-factory.ts"
|
|
},
|
|
"scripts": {
|
|
"dev": "bun run src/cli.ts",
|
|
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
|
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
|
"build:schema": "bash scripts/build-schema.sh",
|
|
"test": "scripts/check-jsonb-pattern.sh && bun test",
|
|
"test:e2e": "bun test test/e2e/",
|
|
"test:eval": "bun test eval/runner/queries/validator.test.ts eval/runner/adapters/ eval/generators/world-html.test.ts test/eval/",
|
|
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
|
"eval:run": "bun eval/runner/multi-adapter.ts",
|
|
"eval:run:dev": "BRAINBENCH_N=1 bun eval/runner/multi-adapter.ts",
|
|
"eval:world:view": "bun eval/cli/world-view.ts",
|
|
"eval:world:render": "bun eval/cli/world-view.ts --no-open",
|
|
"eval:query:validate": "bun eval/cli/query-validate.ts",
|
|
"eval:query:new": "bun eval/cli/query-new.ts",
|
|
"eval:type-accuracy": "bun eval/runner/type-accuracy.ts",
|
|
"eval:generate-amara-life": "bun eval/generators/amara-life-gen.ts",
|
|
"eval:generate-amara-life:dry": "bun eval/generators/amara-life-gen.ts --dry-run",
|
|
"eval:brainbench": "bun eval/runner/all.ts",
|
|
"eval:brainbench:smoke": "BRAINBENCH_N=1 BRAINBENCH_CONCURRENCY=2 bun eval/runner/all.ts",
|
|
"eval:brainbench:published": "BRAINBENCH_N=10 BRAINBENCH_CONCURRENCY=2 bun eval/runner/all.ts",
|
|
"eval:cat6": "bun eval/runner/cat6-prose-scale.ts",
|
|
"eval:cat11": "bun eval/runner/cat11-multimodal.ts",
|
|
"postinstall": "gbrain --version >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive 2>/dev/null || true",
|
|
"prepublish:clawhub": "bun run build:all",
|
|
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
|
},
|
|
"openclaw": {
|
|
"compat": {
|
|
"pluginApi": ">=2026.4.0"
|
|
}
|
|
},
|
|
"dependencies": {
|
|
"@anthropic-ai/sdk": "^0.30.0",
|
|
"@aws-sdk/client-s3": "^3.1028.0",
|
|
"@electric-sql/pglite": "^0.4.4",
|
|
"@modelcontextprotocol/sdk": "^1.0.0",
|
|
"gray-matter": "^4.0.3",
|
|
"marked": "^18.0.0",
|
|
"openai": "^4.0.0",
|
|
"pgvector": "^0.2.0",
|
|
"postgres": "^3.4.0"
|
|
},
|
|
"devDependencies": {
|
|
"@types/bun": "latest",
|
|
"pdf-parse": "^2.4.5"
|
|
},
|
|
"license": "MIT"
|
|
}
|