94aaf7e396
* feat(eval-longmemeval): --by-type flag + question field + resume-replace
Per-question JSONL row gains `question`, `question_type`, and (when
ground truth is available) `recall_hit` — additive fields that existing
consumers (LongMemEval's `evaluate_qa.py`) ignore. New `--by-type` flag
emits a `{kind:"by_type_summary", recall_by_type, aggregate}` line at
the end of the output, resume-safe: rebuilt from existing rows so the
final aggregate covers cumulative resumed questions, prior summary at
the tail replaced rather than appended. New `--by-type-floor F` exits
non-zero per breached question_type. Empty-bucket guard emits null rate
not NaN. Exports `buildByTypeSummary` + `emitByTypeSummary` +
`seedRecallByTypeFromFile` for unit testing.
* feat(eval-cross-modal): --batch flag + semaphore + DI seam
Adds `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT]
[--yes]` to the existing eval cross-modal command. Mutually exclusive
with --task. Reads LongMemEval-shape JSONL output, filters by_type_summary
rows automatically, fans out via a new `runWithLimit<T>` semaphore
primitive (default --concurrent 3 x 3 model slots = 9 simultaneous calls;
below tier-1 rate limits on all 3 providers). Pre-flight cost estimate
refuses past --max-usd (default $5) unless --yes. Per-question receipts
written to a per-batch tempdir + deleted at end of run so
~/.gbrain/eval-receipts/ stays clean; summary receipt inlines verdicts.
Exit precedence (new batch-level policy, not inherited from aggregate.ts):
ERROR > FAIL > INCONCLUSIVE > PASS — any per-question runtime error exits 2.
New `runEvalCrossModal(args, opts?: {runEval?})` DI seam mirrors the
existing eval-longmemeval pattern. Tests pass a stub runEval so unit tests
don't need API keys; gateway availability check is also skipped when
opts.runEval is provided. Pinned by 17 cases.
* test: hermetic qrels retrieval gate against synthetic basis-vector corpus
Adds test/eval-replay-gate.test.ts as a unit-shard test (NOT under
test/e2e/ — the unit-shard CI matrix runs every PR via bun test;
test/e2e/ is fixed-file). Seeds a PGLite engine with synthetic
placeholder-name pages whose embeddings are basis vectors (same pattern
as test/e2e/search-quality.test.ts:23-28) so retrieval is hermetic — no
API keys, no DATABASE_URL, fully deterministic.
The qrels fixture at test/fixtures/eval-baselines/qrels-search.json has
12 hand-curated queries; each maps to a ranked list of relevant slugs +
`first_relevant_slug` (expected top-1). For each query, the gate asserts
`top1_match_rate >= 0.80` AND `recall_at_10 >= 0.85`. Env-overridable
floors via GBRAIN_REPLAY_GATE_TOP1_FLOOR / GBRAIN_REPLAY_GATE_RECALL_FLOOR
through withEnv(). Gate-fire prints per-query HIT/miss + recall to stderr.
When ranking changes intentionally move expected slugs, edit
qrels-search.json directly with a 'Why:' line in the commit body —
documented in docs/eval-bench.md.
scripts/check-test-real-names.sh allowlist gains 6 entries for the
privacy-grep regression guard inside the test, which must literally
spell the names it forbids to assert they're NOT in the fixture (same
meta-rule exception as skillpack-harvest privacy tests).
* feat(autopilot): opt-in nightly cross-modal quality probe + doctor check
Composes `gbrain eval longmemeval --by-type` + `gbrain eval cross-modal
--batch` into a 24h-cadenced quality check. Default DISABLED — opt-in via
`gbrain config set autopilot.nightly_quality_probe.enabled true` so new
users don't discover background API spend.
src/core/cycle/nightly-quality-probe.ts ships the phase implementation
with a full NightlyProbeDeps DI surface (isEnabled, hasEmbeddingProvider,
resolveMaxUsd, resolveRepoRoot, runLongMemEval, runCrossModalBatch, now)
so tests stub every external effect — no PGLite, no real LLM calls.
Pure `shouldRunNightly(now, recentEvents, windowMs?)` rate-limit fn.
src/core/audit-quality-probe.ts is the ISO-week-rotated JSONL writer
(mirrors audit-slug-fallback.ts; honors GBRAIN_AUDIT_DIR). One event per
run: outcome (pass/fail/inconclusive/error/budget_exceeded/rate_limited/
no_embedding_key), exit code, pass/fail/error counts, est_cost_usd,
fixture_sha8.
src/commands/doctor.ts gains a `nightly_quality_probe_health` check:
SKIPPED with paste-ready enable command when disabled; OK with timestamp
when all PASS in last 7 days; WARN with per-outcome counts when any
FAIL/ERROR/BUDGET_EXCEEDED. Extracted as pure
`computeNightlyQualityProbeHealthCheck(probeEnabled, events)` for
unit testing.
test/fixtures/longmemeval-nightly.jsonl is a 10-question placeholder
dataset (synthetic names only) distinct from the existing 5-question
mini fixture so the probe has consistent regression signal.
Real expected cost: ~$0.35/night = ~$10.50/month. Worst-case at
default $5 cap: $150/month.
Pinned by 21 cases in test/nightly-quality-probe.test.ts covering the
rate-limit pure function, every outcome branch, and all 7 branches of
the doctor check.
Autopilot scheduler wiring deferred to v0.41+ — the phase is callable
in isolation today (via the DI surface); cycle-loop dispatcher
integration filed in TODOS.md as a follow-up.
* docs: document Track D eval surfaces + file v0.41+ follow-up TODOs
docs/eval-bench.md gains a 'v0.40.1.0 Track D — Eval infrastructure'
section covering: --by-type usage + resume-replace semantics, the
hermetic qrels gate workflow + 'Why:' commit-body refresh convention,
--batch end-to-end with cost-bound + concurrency knobs, and the opt-in
nightly probe enable workflow + cost ceiling.
TODOS.md files two follow-ups:
- v0.41+: contributor-mode CI capture for BrainBench-Real replay gate
(the deferred original Task 2 design — replay against real captured
queries is more valuable than synthetic qrels long-term, but needs CI
secret + nightly capture pipeline + commit automation; deferred to a
dedicated wave)
- v0.41+: wire the nightly quality probe into autopilot scheduling
(phase callable in isolation today; cycle-loop dispatcher integration
is a ~3-hour follow-up)
CLAUDE.md Key Files annotations extended for the four lanes:
eval-longmemeval gains the --by-type description, eval-cross-modal
gains the --batch + DI seam description, new entries for the qrels
gate test + the nightly probe + audit-quality-probe writer.
* chore: bump version and changelog (v0.40.1.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(eval): close 4 codex-flagged eval-integrity bugs
Codex adversarial review on the Track D wave found 4 real ways the new
eval-gate code could silently bypass its gates. Each fix below either
counts what was previously dropped, fails fast on a parser edge case,
or enforces a gate that was previously skipped on an early-return path.
CDX-1: cross-modal --batch silently dropped failed/corrupt LongMemEval
rows. `gbrain eval longmemeval` emits {error:..., hypothesis:''} when
runOneQuestion throws; the batch reader's missing-field skip threw those
rows away, shrinking the denominator. A green eval on a subset is now
impossible:
- eval-longmemeval.ts: error rows now carry `question` + `question_type`
so the batch consumer can identify them as upstream failures, not
skip them as malformed.
- eval-cross-modal.ts: readBatchRows now returns {rows, upstream_errors,
malformed_count}. Upstream errors fold into per_question with verdict
'upstream_error'. BatchSummary gains `upstream_error_count` and
`malformed_count`. ERROR exit precedence widens to include both, so
any upstream failure exits 2.
CDX-2: --limit 0 was a direct CI bypass — zero-row check fired before
slicing, then the empty result fell through to verdict='pass'. Fixed
with a hard `limit >= 1` check.
CDX-3: --resume-from + --by-type-floor was a real gate skip. When a
prior run had every question answered, the early "nothing to do" return
fired BEFORE summary emission and floor enforcement. Now the no-op
resume path still seeds recallByType from the existing file, emits the
by_type_summary at the tail, and runs the floor gate.
CDX-5: doctor nightly_quality_probe_health only flagged fail / error /
budget_exceeded as warn. no_embedding_key / rate_limited / inconclusive
were silently reported as PASS — hiding misconfigurations and queue
backpressure. The bad-event filter is now `outcome !== 'pass'`, and the
counts string surfaces every bucket so the operator sees exactly what
went wrong.
scripts/check-privacy.sh: adds test/eval-replay-gate.test.ts to the
allowlist (the qrels test's privacy-grep regression guard literally
names what it forbids, same meta-rule exception as the existing
test/recency-decay.test.ts + skillpack-harvest allowlist entries).
Pinned by 8 new regression cases across eval-longmemeval (CDX-3),
eval-cross-modal-batch (CDX-1 + CDX-2), and nightly-quality-probe
(CDX-5). 76 Track D tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GBrain
Your AI agent is smart but forgetful. GBrain gives it a brain.
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: 146,646 pages, 24,585 people, 5,339 companies, 66 cron jobs running autonomously. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.
The brain wires itself. Every page write extracts entity references and creates typed links (attended, works_at, invested_in, founded, advises) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side: gbrain lands P@5 49.1%, R@5 97.9% on a 240-page Opus-generated rich-prose corpus, beating its graph-disabled variant by +31.4 points P@5 and ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling gbrain-evals repo.
New default in v0.36.2.0: ZeroEntropy for both embedding (zembed-1 at 1280d via Matryoshka) and reranker (zerank-2). On a real-corpus benchmark vs OpenAI and Voyage: 2.2× faster (442ms vs OpenAI 973ms), 2.6× cheaper at regular pricing ($0.05/M vs OpenAI $0.13), wins 11 of 20 queries head-to-head, reshuffles 60% of top-1 results when used as a second-pass reranker. Bring your own key from zeroentropy.dev, or switch to OpenAI/Voyage at install time via gbrain init --pglite --embedding-model <provider:model> --embedding-dimensions <N> — your choice is sticky. To switch an existing brain, run gbrain reinit-pglite --embedding-model <provider:model> --embedding-dimensions <N> (PGLite) or follow the SQL recipe in docs/embedding-migrations.md (Postgres). gbrain config set embedding_model is refused as of v0.37.11.0 because the schema column has to resize too.
GBrain is those patterns, generalized. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself. One command does the loop you used to run by hand: gbrain doctor --remediate --yes --target-score 90 --max-usd 5. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. gbrain doctor --remediation-plan --json previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (reindex, repair-jsonb, orphans, integrity, purge, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New --background flag on gbrain embed submits the job and exits with job_id=N for shell composition.
New in v0.35.7 — Temporal trajectory + founder scorecard. Author typed metric assertions in the ## Facts fence (mrr=50000, arr=2000000, team_size=12) and gbrain stores them as first-class typed columns. gbrain eval trajectory companies/acme-example prints the chronological history with regressions auto-flagged inline. gbrain founder scorecard companies/acme-example rolls up claim accuracy, consistency, growth direction, and red flags into a stable schema_version: 1 JSON contract. New MCP op find_trajectory exposes the same data to agents (read scope, visibility-filtered for remote callers). The consolidate cycle phase now writes valid_until on chronologically-superseded facts AND uses semantic upsert on (page_id, claim, since_date) — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions).
~30 minutes to a fully working brain. Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
LLMs: fetch
llms.txtfor the documentation map, orllms-full.txtfor the same map with core docs inlined in one fetch. Agents: start withAGENTS.md(orCLAUDE.mdif you're Claude Code).
Install
GBrain runs in three shapes. Pick the one that matches how you use AI agents today.
Run with your agent platform
Already using OpenClaw or Hermes? GBrain installs as a skillpack scaffold into your agent's workspace.
gbrain init --pglite
gbrain skillpack scaffold --all # or: scaffold <name> per skill
That's it. Your agent picks up 43 skills (signal detection, brain-ops, ingest, enrich, citation-fixer, daily-task-manager, cron-scheduler, eval framework, and 35 more). Routing lives in skills/RESOLVER.md — the agent reads it once per request, picks the right skill, executes. Scaffolded skills are first-class members of your agent repo — you own them, edit freely; gbrain skillpack reference <name> diffs your copy against gbrain's bundle when you want to pull upstream improvements. (The legacy gbrain skillpack install managed-block model was retired in v0.36.0.0; run gbrain skillpack migrate-fence once if you're upgrading from an older release.)
CLI standalone
Use gbrain from any shell, no agent platform required.
bun install -g github:garrytan/gbrain
gbrain init --pglite # 2 seconds; no server, no Docker
gbrain doctor # verify health
Then point any MCP-aware client (Claude Code, Cursor, Windsurf) at it, or use it from your shell:
gbrain search "who works at acme AI?"
gbrain query "what did bob invest in this quarter?"
gbrain graph-query people/garry-tan --depth 2
Detailed setup paths (Postgres at scale, Supabase, thin-client mode) live in docs/INSTALL.md.
MCP server (any MCP client)
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
# at /admin, SSE activity feed at /admin/events
Per-client guides (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) live under docs/mcp/. HTTP server supports DCR-style client registration, scope-gated access (read/write/admin), and built-in rate limiting.
How to get data in (v0.38+)
One command, local or hosted, synchronous receipt:
gbrain capture "the thought I want to remember"
gbrain capture --file ./notes/today.md
echo "from a pipe" | gbrain capture --stdin
SLUG=$(gbrain capture "..." --quiet)
The page lands in the DB AND on disk in one move (the v0.38 put_page
write-through plumbing). Default slug inbox/YYYY-MM-DD-<hash8> so
captures cluster in a predictable triage location. On thin-client installs
the verb routes through MCP to the server — same command, same UX.
For webhook ingestion (Zapier / IFTTT / Apple Shortcuts):
curl -X POST https://your-brain/ingest \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: text/markdown" \
-d "# a thought from a Shortcut"
For mobile capture, the inbox folder source picks up anything dropped into
~/.gbrain/inbox/ from iOS Shortcuts / AirDrop / Drafts / Finder.
Third-party skillpacks can ship custom ingestion sources (Granola, Linear,
voice, OCR) against the versioned IngestionSource contract at
gbrain/ingestion. See docs/skillpack-anatomy.md.
What it does (the loop)
signal → search → respond → write → auto-link → sync
(every (brain-first (informed (page + (typed edges (cron
message) retrieval) by context) timeline) + backlinks) keeps fresh)
- Signal detector runs on every message your agent receives. Captures ideas, entity mentions, time-sensitive todos, names, links.
- Brain-first lookup before any external API call. The cheapest, fastest, most personal information source you have.
- Auto-link fires on every page write. No LLM calls; pure pattern matching on
[[wiki/people/bob]]style references. New entity → new page stub → graph grows. - Cron-driven enrichment runs while you sleep: dedup people pages, fix citations, score salience, find contradictions, prep tomorrow's tasks.
The whole loop is described in docs/architecture/topologies.md with diagrams.
Capabilities
Hybrid search. Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (conservative, balanced, tokenmax) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in docs/eval/SEARCH_MODE_METHODOLOGY.md. Default: balanced with ZeroEntropy reranker on.
Self-wiring knowledge graph. Every put_page extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (attended, works_at, invested_in, founded, advises, mentions, …). Multi-hop traversal via gbrain graph-query. The graph is what produces the +31.4 P@5 lift over vector-only RAG.
Job queue (Minions). BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
43 curated skills. Routing lives in skills/RESOLVER.md. Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
Eval framework. gbrain eval longmemeval runs the public LongMemEval benchmark against your hybrid retrieval. gbrain eval export + gbrain eval replay capture real queries and replay them against code changes (set GBRAIN_CONTRIBUTOR_MODE=1). gbrain eval cross-modal cross-checks an output against the task using three different-provider frontier models. Full methodology in docs/eval/SEARCH_MODE_METHODOLOGY.md.
Brain consistency. gbrain eval suspected-contradictions samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.
Integrations
Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in recipes/ and is discoverable via gbrain integrations list.
- Voice: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe:
recipes/twilio-voice-brain.md. - Email + calendar: webhook handlers that route to brain signals.
docs/integrations/meeting-webhooks.md. - Embedding providers: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in
docs/integrations/embedding-providers.md. - Credential gateway: vault-aware secret distribution.
docs/integrations/credential-gateway.md. - MCP clients: every major MCP client is supported.
docs/mcp/per-client setup.
Architecture
Two engines, one contract. PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first BrainEngine interface in src/core/engine.ts defines ~47 operations both engines implement; CLI and MCP server are generated from one source.
Brain repo is the system of record. Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in docs/architecture/topologies.md.
Two organizational axes (brain ⊥ source). A brain is a database (your personal brain, a team mount you joined). A source is a repo inside that brain (wiki, gstack, an essay, a knowledge base). Routing lives in .gbrain-source dotfiles and resolves via a documented 6-tier precedence chain. Full diagrams in docs/architecture/brains-and-sources.md.
Why the graph matters. Vector search returns chunks that are semantically close. The graph returns chunks that are factually connected. Hybrid search pulls from both; auto-linking on every write keeps the graph fresh. Deep dive: docs/architecture/RETRIEVAL.md.
Troubleshooting
gbrain import fails with expected N dimensions, not M? Run gbrain doctor. It will print the exact gbrain config set ... or gbrain retrieval-upgrade command to repair the mismatch. You should not need to delete ~/.gbrain. As of v0.37, fresh gbrain init --pglite auto-detects your embedding provider from API keys in your environment — set OPENAI_API_KEY (or ZEROENTROPY_API_KEY / VOYAGE_API_KEY) before running init, or pass --embedding-model <provider>:<model> explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass --no-embedding to defer setup until runtime. See docs/integrations/embedding-providers.md for the full provider matrix and docs/operations/headless-install.md for Docker/CI sequencing.
Docs
docs/INSTALL.md— every install path, end to enddocs/architecture/— system design, topologies, retrieval theorydocs/guides/— how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion)docs/integrations/— connecting external data sources (voice, email, calendar, embedding providers)docs/mcp/— per-client MCP setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)docs/eval/— eval framework, metric glossary, methodologydocs/ethos/— philosophy (thin harness, fat skills, markdown as recipes, origin story)AGENTS.md— entry point for non-Claude agentsCLAUDE.md— entry point for Claude Code (deep operating context)CONTRIBUTING.md— contributor guide, test discipline, eval-capture modeSECURITY.md— OAuth threat model, hardening defaults
Contributing
Run bun run test for the fast loop, bun run verify for the pre-push gate, bun run ci:local to run the full Docker-backed CI stack locally. Detailed test discipline in CONTRIBUTING.md.
Community PRs are batched into release waves rather than merged one-by-one — see the "PR wave workflow" section in CLAUDE.md. Contributor attribution stays attached via Co-Authored-By: trailers. We credit every accepted contribution in CHANGELOG.md.
If you find a bug or want a feature: open an issue first. Quick fixes (typo, doc bug, obvious regression) can go straight to a PR. Anything touching schema, retrieval ranking, MCP protocol, or the security boundary needs a design discussion in the issue first.
License + credit
MIT. Built by Garry Tan to run his OpenClaw and Hermes deployments — the production brain behind his actual AI agents.
Origin story: docs/ethos/ORIGIN.md.
Community PR contributors are credited in CHANGELOG.md per release. ZeroEntropy (@zeroentropy) for the embedding + reranker stack that became the v0.36.2.0 default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.