mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
d4211f4176
* feat(skillopt): wire held-out gate, honest receipts, ENFORCE + ablation opts Wire the F11 held-out gate into the orchestrator at checkpoint acceptance (runHeldOutGate was dead code); parse + thread --held-out through CLI, batch, fleet, background job, and the run_skillopt MCP op. Populate the real receipt.baseline_sel_score (was hardcoded 0) and add a final-test eval (test_score + baseline_test_score) via a shared scoreSkillOnTasks primitive. Fix the --no-mutate proposed.md write (was a stub) and enforce maxRuntimeMin. D16 ENFORCE in core mutation policy (assertBundledMutationHeldOut): mutating a bundled skill in place requires a non-empty (>=5), benchmark-disjoint held-out set or hard-refuses. Add three eval-internal ablation opts (reflectMode, disableValidationGate, optimizerMode='one-shot-rewrite') recorded in the receipt + audit; ROLLOUT_SUCCESS_THRESHOLD named constant. Security: run_skillopt MCP op validates skill_name (kebab-only) and confines caller-supplied benchmark/held-out paths to the skills dir for remote callers. * test(skillopt): held-out gate, ENFORCE, one-shot rewrite, runtime + receipt honesty New test/skillopt/rollout.test.ts (rollout had zero coverage). Held-out ENFORCE unit cases + one-shot-rewrite fence handling (whole-response unwrap, embedded-fence preserved, error path). E2E: F11 held-out BLOCKS/ALLOWS, bundled no-mutate write, reflectMode/disableValidationGate/optimizerMode, maxRuntimeMin abort, receipt baseline/test-score honesty, held-out/benchmark disjointness, D2 no-DB-pollution. * chore: bump version and changelog (v0.42.9.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0 Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and the tutorial's bundled-skill step: mutating a bundled skill in place now requires --allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update the receipt contract to the honest baseline/test-score fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): AI SDK v6 toolLoop compat — multi-turn tool calls work again The ai@6.x bump tightened ModelMessage + tool-schema validation, which silently broke every multi-turn tool loop. Both `gbrain skillopt` rollouts and production background `subagent` jobs route through `chat()`/`toolLoop` and crashed the moment the model called a tool ("messages do not match the ModelMessage[] schema" / "schema is not a function"). Surfaced end-to-end by the SkillOpt real-LLM eval. Three fixes: - chat(): wrap tool defs with the SDK's `jsonSchema()` helper instead of a bare `{jsonSchema}` object (v6 asSchema() treated the bare object as a thunk and threw). - chat(): new exported pure `toModelMessages()` converts gbrain's provider-neutral ChatMessage[] into v6 ModelMessage[] — tool results ride a dedicated `role:'tool'` message with structured `{type,value}` output; null output preserved as json null. Load-bearing for the production subagent path, not just skillopt. - rollout.ts: replace the inline params→schema mapper (dropped `items` on array params) with the shared `paramDefToSchema` single source of truth. Pinned by test/gateway-model-messages.test.ts (8 cases). Folds into the open v0.42.9.0 PR (#1759) — these complete the eval-readiness wave by making skillopt actually run against a live model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skillopt): budget no-pricing for Haiku silently scored every rollout 0 Surfaced by the SkillOpt real-LLM eval (Track B). Two coupled bugs that made a budget-capped Haiku run report a vacuous "0/N" measurement in ~2ms with zero LLM calls — indistinguishable from a real deficient-skill score: 1. Claude Haiku 4.5's canonical dateless id (`claude-haiku-4-5`) was missing from anthropic-pricing.ts (only the dated `-20251001` was present). With `--max-cost` set, BudgetTracker.reserve() threw no_pricing on the FIRST chat() of every rollout. Added the dateless entry (sonnet already had its dateless form). 2. runValidationGate swallowed that BUDGET_EXHAUSTED error — runWithLimit settled it as {ok:false}, which the gate turned into median:0. A pricing/cap crash became a fake score. The gate now scans settled results for isMustAbortError() and re-throws so the caller aborts loudly; ordinary (non-abort) rollout errors still fail-open to 0 (judge-hiccup posture kept). Pinned by test/skillopt/validate-gate-abort.test.ts (3 cases). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): llms-full.txt over size budget — drop what-schemas-unlock from full bundle The toolLoop + budget bug-fix annotations grew CLAUDE.md, pushing llms-full.txt to 756KB over the 750KB FULL_SIZE_BUDGET (the `build-llms > size budget` test failed, failing the `test` CI job). CLAUDE.md stays inlined by design (it's the point of the one-fetch bundle), so per the budget comment's own guidance ("ship with includeInFull=false exclusions") this excludes docs/what-schemas-unlock.md (15.4KB value-explainer, not load-bearing operational reference) from llms-full.txt; it stays linked in llms.txt. Bundle now 740KB with ~9KB headroom. No budget bump — 750KB is near the ~190k-token-context fit ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ci): re-admit policy docs into ci-cache-hash before doc relocation docs/**/*.md is deny-listed from the CI cache hash (test-irrelevant). The CLAUDE.md restructure moves test/release POLICY into docs/TESTING.md + docs/RELEASING.md, which DO carry contracts the test suite reads. Without re-admitting them, a policy-only edit would produce the same cache hash and skip the test shard that runs the build-llms + doc-history guards (false-pass). Adds an ALLOW_PATTERNS re-admit step after the deny, scoped to the named policy docs (not a blanket docs un-deny). Lands FIRST, before any doc moves. Pinned by 3 new cases in test/scripts/ci-cache-hash.test.ts: TESTING.md + RELEASING.md edits MUST change the hash; docs/guide.md still must not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate Key files / thin-client / Testing out of CLAUDE.md (verbatim) CLAUDE.md had grown to 592KB / ~147k tokens auto-loaded every session (~77% of the llms-full.txt single-fetch bundle). The per-file index was append-only by mandate. This is the exact thin-dispatcher-vs-fat-blob anti-pattern gbrain exists to fix, so CLAUDE.md becomes a thin orientation + resolver that points at on-demand docs. This commit is the VERBATIM move (content-preserving — the next commit compresses): - docs/architecture/KEY_FILES.md <- ## Key files + the calibration key-files cluster + Schema Cathedral v3 impl detail - docs/architecture/thin-client.md <- ## Thin-client routing - docs/TESTING.md <- ## Testing - ## Commands DROPPED (18 'added in vX.Y' history blocks; current surface is gbrain 0.41.38.0 -- personal knowledge brain USAGE gbrain <command> [options] SETUP init [--pglite|--supabase|--url] Create brain (PGLite default, no server) migrate --to <supabase|pglite> Transfer brain between engines upgrade Self-update check-update [--json] Check for new versions doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings) integrations [subcommand] Manage integration recipes (senses + reflexes) PAGES get <slug> Read a page put <slug> [< file.md] Write/update a page delete <slug> Delete a page list [--type T] [--tag T] [-n N] List pages SEARCH search <query> Keyword search (tsvector) query <question> [--no-expand] Hybrid search (RRF + expansion) ask <question> [--no-expand] Alias for query IMPORT/EXPORT import <dir> [--no-embed] Import markdown directory sync [--repo <path>] [flags] Git-to-brain incremental sync sync --watch [--interval N] Continuous sync (loops until stopped) sync --install-cron Install persistent sync daemon export [--dir ./out/] Export to markdown export --restore-only [--repo <p>] Restore missing supabase-only files [--type T] [--slug-prefix S] With optional filters FILES files list [slug] List stored files files upload <file> --page <slug> Upload file to storage files upload-raw <file> --page <s> Smart upload (size routing + .redirect.yaml) files signed-url <path> Generate signed URL (1-hour) files sync <dir> Bulk upload directory files verify Verify all uploads EMBEDDINGS embed [<slug>|--all|--stale] Generate/refresh embeddings LINKS link <from> <to> [--type T] Create typed link unlink <from> <to> Remove link backlinks <slug> Incoming links graph <slug> [--depth N] Traverse link graph (returns nodes) graph-query <slug> [--type T] Edge-based traversal with type/direction filters [--depth N] [--direction in|out|both] TAGS tags <slug> List tags tag <slug> <tag> Add tag untag <slug> <tag> Remove tag TIMELINE timeline [<slug>] View timeline timeline-add <slug> <date> <text> Add timeline entry TOOLS extract <links|timeline|all> Extract links/timeline (idempotent) [--source fs|db] fs (default) walks .md files; db iterates engine pages [--dir <brain>] brain dir for fs source [--type T] [--since DATE] filters (db source) [--dry-run] [--json] publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256) check-backlinks <check|fix> [dir] Find/fix missing back-links across brain lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter orphans [--json] [--count] Find pages with no inbound wikilinks salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type) transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only) dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly). See also: autopilot --install (continuous daemon). check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY) report --type <name> --content ... Save timestamped report to brain/reports/ BRAIN (capture / ideate / explore — v0.37/v0.38) capture [content] [--file PATH] Single entrypoint for getting content into the brain [--stdin] [--slug s] [--type t] Inline content / file / stdin; writes to inbox/ by default [--source ID] [--quiet|--json] Multi-source brains: route to a non-default source brainstorm <question> [--json] Bisociation idea generator (hybrid search + far-set + judge) [--save|--no-save] [--limit N] lsd <question> [--json] Lateral Synaptic Drift: inverted-judge brainstorm [--save|--no-save] [--limit N] rewarding far-from-obvious + axiomatic inversions SOURCES (multi-repo / multi-brain) sources list Show registered sources sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki') sources remove <id> Remove a source + its pages sync --all Sync all sources with a local_path sync --source <id> Sync one specific source repos ... DEPRECATED alias for 'sources' (v0.19.0) CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) code-def <symbol> [--lang l] Find the definition of a symbol across code pages code-refs <symbol> [--lang l] Find all references to a symbol (JSON-first) code-callers <symbol> Who calls this symbol? (v0.20.0 A1) code-callees <symbol> What does this symbol call? (v0.20.0 A1) query <q> --lang <l> Filter hybrid search to one language (v0.20.0) query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) sync --strategy code Sync code files into the brain JOBS (Minions) jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run] jobs list [--status S] [--limit N] List jobs jobs get <id> Job details + history jobs cancel <id> Cancel job jobs retry <id> Re-queue failed/dead job jobs prune [--older-than 30d] Clean old jobs jobs stats Job health dashboard jobs work [--queue Q] Start worker daemon (Postgres only) ADMIN stats Brain statistics health Brain health dashboard history <slug> Page version history revert <slug> <version-id> Revert to version features [--json] [--auto-fix] Scan usage + recommend unused features autopilot [--repo] [--interval N] Self-maintaining brain daemon config [show|get|set] <key> [val] Brain config storage status [--repo <path>] Storage tier status and health [--json] (git-tracked vs supabase-only) serve MCP server (stdio) serve --http [--port N] HTTP MCP server with OAuth 2.1 --token-ttl N Access token TTL in seconds (default: 3600) --enable-dcr Enable Dynamic Client Registration --public-url URL Public issuer URL (required behind proxy/tunnel) call <tool> '<json>' Raw tool invocation version Version info --tools-json Tool discovery (JSON) Run gbrain <command> --help for command-specific help. + the per-command KEY_FILES entries; content stays in git) CLAUDE.md gains: a Reference map (resolver), a Maintaining section (the anti-disease rule), and a Cross-cutting invariants subsection under Architecture so the must-never-violate rules (trust fail-closed, sourceScopeOpts isolation, JSONB trap, engine parity, contract-first, migrations, multi-source) still auto-load after the index moved out. Result: CLAUDE.md 592KB -> 61KB; llms-full.txt 740KB -> 210KB (new docs link-only until compressed). build-llms drift + budget test green; verify 29/29 green. The pre-move content is recoverable at git show <this^>:CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): compress relocated docs to current-state + add recurrence guard Compresses the verbatim-relocated reference docs from append-only release-history to current-state-only (the disease cure), then makes recurrence structurally impossible via a CI guard. Compression (fan-out subagents + adversarial verify, audited mechanically): - KEY_FILES.md 453KB -> 356KB; TESTING.md 42KB -> 38KB; thin-client.md already clean. - 393/393 entries preserved; every src/test/scripts path from the verbatim original survives (mechanical comm-check); zero bolded **v0. markers remain. - Conservative ratio (~22%) because the content is invariant-dense — correctness over brevity. Dropped: **vX.Y.Z (#NNN):** clauses, codex/review tags, contributor credits, PR-numbers-as-ids, pre-fix/then/was-now history deltas. Kept: every exported symbol, invariant, and Pinned-by reference. Verbatim original recoverable at git show <relocation-commit>:docs/architecture/KEY_FILES.md. Recurrence guard (scripts/check-key-files-current-state.sh, wired into verify + check:all): - HARD: bans the bolded **v0.<digit> marker in the reference docs (scoped — plain 'as of pgvector 0.7' prose is fine, no false positives). - HARD: CLAUDE.md size cap (90KB; currently 61KB) — the structural backstop. - Pinned by test/scripts/check-key-files-current-state.test.ts (7 cases). Content contracts (test/build-llms.test.ts, +5 cases per codex outside-voice): CLAUDE.md keeps inline ship IRON RULES (version format, document-release, never-hand-roll); AGENTS.md keeps its boot order; llms indexes the new docs; KEY_FILES stays link-only (not inlined). Privacy: scrubbed the relocated 'wintermute/chat/' source-boost examples + the literal harvest-lint regex to generic placeholders (legitimate in allowlisted CLAUDE.md; genericized for the new public docs per the privacy rule). Reverts the284c50a4band-aid: re-inlines docs/what-schemas-unlock.md now that the restructure freed ~530KB of bundle headroom (llms-full.txt 740KB -> 225KB). verify 30/30 green (incl. new check:doc-history). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate verbose release process to docs/RELEASING.md The highest-/ship-risk commit (isolated so it can revert alone). Moves the verbose release + contributor procedure out of CLAUDE.md, keeping every ship-critical IRON RULE inline so /ship + /document-release (which read CLAUDE.md) cannot regress. Moved to docs/RELEASING.md: pre-ship test requirements; the CHANGELOG-branch-scoped + CHANGELOG voice + release-summary template; the 'To take advantage of vX' block spec; version migrations + migration-is-canonical; schema state tracking; GitHub Actions SHA maintenance; PR-descriptions-cover-the-branch; community-PR-wave; checking-out-PRs-from-garrytan-agents. Kept INLINE in CLAUDE.md (ship-critical IRON RULES — do NOT move): - the Version-locations table (5-file sync) + the 3-line consistency audit - Conductor branch=workspace - Post-ship /document-release (MANDATORY) - Privacy + Responsible-disclosure rules (Privacy also anchors the check-privacy allowlist — the only place allowed to name the fork) - PR-title-version-first - never-hand-roll-ship (Skill routing) Plus a new ## Releasing pointer ('Before any ship, read docs/RELEASING.md in full') and a resolver row. CLAUDE.md 61KB -> 39KB (592KB -> 39KB overall, 93% cut; ~9k tokens auto-loaded vs ~147k). CLAUDE.md size-gate tightened 90KB -> 60KB. The content-contract tests pin that the inline IRON RULES (MAJOR.MINOR.PATCH.MICRO, document-release, hand-roll ship) did NOT move out. The moved ranges carry no banned fork name, so RELEASING.md needs no privacy allowlist entry. verify 30/30; bundle 225KB -> 204KB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(changelog): note CLAUDE.md restructure in v0.42.9.0 The CLAUDE.md thin-resolver restructure (592KB → 39KB) rides in this release; record it under the existing v0.42.9.0 For-contributors section. No version bump — v0.42.9.0 is unreleased and already allocated to this PR. * fix(ci): ci-cache-hash re-admit matched a literal \t, a no-op on GNU grep The policy-doc re-admit (75992b77) put `\t` inline in the ALLOW patterns passed to `grep -E`. BSD grep (macOS local) treats `\t` as a tab so it worked locally; GNU grep (Ubuntu CI) treats it as literal `t`, so nothing re-admitted and docs/TESTING.md / docs/RELEASING.md stayed deny-listed — the two policy-doc tests failed on CI shard 6 (1097 pass / 2 fail). Build ALLOW_RE with `printf '\t(%s)'` so the tab is a real byte, identical in construction to DENY_RE (line 117), which the CI log shows matches correctly on GNU grep. End-to-end: editing docs/TESTING.md now flips the hash; a normal docs/*.md add still does not (deny stays scoped). * fix(skillopt): feed the scorer's success criteria to the optimizer Surfaced by the SkillOpt real-LLM eval (Track B). The reflect step was shown only a pass/fail score and the agent transcript — never WHAT the benchmark judge rewards. On a skill judged by structure (e.g. "must include a Confidence: line") the optimizer proposed plausible-but-off edits ("close with a synthesis") that never satisfied the literal check; every candidate scored 0 on D_sel, the validation gate rejected them all, and the skill text never changed (optimized === baseline === 0). Fix: render each benchmark Judge (rule checks / llm rubric / qrels) into plain-English criteria via new exported describeJudge / describeJudges, and thread them into the reflect prompt (a SUCCESS CRITERIA block) for both the loop reflect calls and the one-shot-rewrite path. The orchestrator computes the distinct criteria across train+sel+test once. The optimizer system prompt now instructs it to satisfy the criteria through genuine content, never empty keywords — reward-hacking stays defended by the independent held-out gate (cat32 confirms the gate catches a keyword-stuffing hack). End-to-end this took a deficient skill from 0.00 to 1.00 on a held-out set it never trained on. Pinned by test/skillopt/reflect.test.ts (describeJudge per kind, describeJudges dedup, criteria present/absent in the prompt). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tutorials
Step-by-step walkthroughs that take you from zero to a working outcome. Concrete commands, real numbers, no abstraction-first jargon. Each tutorial assumes no prior GBrain knowledge.
Shipped
- Set up your personal AI agent + brain from zero — the canonical solo install. Two GitHub repos, a Telegram bot, AlphaClaw on Render, OpenClaw + GBrain + Supabase. End-to-end in about 2 hours; about $100 to $150 a month sustained. The full-stack install I'd run today.
- Set up GBrain as your company brain — federated, multi-user, OAuth-scoped institutional memory for a 10-50 person team. Three sources (shared / customers / internal-only), per-user scope, first synthesized query as a teammate. About 90 minutes end-to-end, about $5 in API calls for the demo, under $100 a month sustained for a 25-person company.
- Auto-improve a skill with
gbrain skillopt— treat aSKILL.mdas the trainable parameter of a frozen agent. Write your first benchmark from scratch (the part everyone gets stuck on), preview the cost, run the optimizer, read accepted vs no_improvement vs aborted, and accept a measurably better skill. About 20 minutes, about $1 in API calls. Reference:../guides/skillopt.md. - Give your coding agent a memory: GBrain + Claude Code / Codex — the two-funnel walkthrough for coding-agent users. Path A: connect Claude Code / Codex to a brain you already run (OpenClaw, Hermes, any
gbrain serve --http). Path B: start from nothing with a 2-second local PGLite brain. Both end with the brain-first protocol you paste intoCLAUDE.md/AGENTS.mdand the four habits (brain-first lookup, ambient capture, briefing-from-your-brain, whoknows) that make it worth it. About 10 minutes.
In progress
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
-
Set up GBrain for VC dealflow — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows
gbrain whoknows,gbrain find_trajectory, andgbrain founder scorecardon real workflows. -
Migrate your existing vault into GBrain — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through
gbrain schema detect→suggest→review-candidatesso the brain learns your shape instead of forcing you to learn its. -
Index your codebase as a code brain — for developers. Initialize a brain in a code repo, swap to
voyage-code-3for embeddings, usegbrain code-def/gbrain code-refs/gbrain code-callersto navigate the codebase semantically from any MCP-aware editor. -
Run GBrain fully local with Ollama or llama.cpp — for privacy-first deployments. No cloud calls, no API keys, no telemetry. Trades some retrieval quality for full local control. Useful for regulated industries, air-gapped environments, or just paranoia.
-
Set up the dream cycle — the overnight enrichment daemon that makes the brain self-maintaining. Fixes citations, dedupes people pages, surfaces contradictions, generates founder scorecards on the schedule you configure. The piece that turns a static knowledge base into a brain that gets smarter while you sleep.
Want to write one?
Tutorials follow the Diataxis tutorial pattern: learning-oriented, walks a learner from zero to a working result in one session, every step produces a visible change. If you've used GBrain for something interesting and want to write the walkthrough, the existing company-brain.md is the model. Open a PR.
Related documentation
- Reference:
docs/architecture/— system design, topologies, retrieval theory - How-to:
docs/guides/— task-oriented runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion). Highlight: scaling skills past 300 — the three-tier architecture for agents that have outgrown the always-loaded skill manifest. - Integrations:
docs/integrations/— connecting external data sources (voice, email, calendar, embedding providers) - MCP setup:
docs/mcp/— per-client setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) - Install paths:
docs/INSTALL.md— every install path, end to end