ff32fcaa78 v0.41.25.0 perf(sync): batched deletes + global page-generation clock (supersedes #1538) (#1566)
* feat(engine): add deletePages + resolveSlugsByPaths to BrainEngine (v0.41.21.0 T1)

Two new REQUIRED methods on the BrainEngine interface, implemented on
both Postgres and PGLite engines. Closes the per-file N+1 query pattern
that PR #1538 batched on Postgres only.

  deletePages(slugs: string[], opts: { sourceId: string }): Promise<string[]>
    — Single SQL round-trip:
        DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2
        RETURNING slug
    — Returns slugs ACTUALLY DELETED (D6, codex CDX-8) so callers can
      filter pagesAffected to exclude phantom slugs (paths in the
      deletion list but with no DB row).
    — Single-batch primitive: caller chunks input to DELETE_BATCH_SIZE.
      Throws if input exceeds the cap.
    — sourceId is REQUIRED at the type level (D5, codex CDX-10).
      Asymmetric with single-row deletePage which keeps the optional
      'default' fallback for back-compat. v0.42+ TODO to tighten.

  resolveSlugsByPaths(paths, opts): Promise<Map<path, slug>>
    — Batch path → slug lookup. Single SQL round-trip:
        SELECT slug, source_path FROM pages
         WHERE source_path = ANY($1::text[]) AND source_id = $2
    — Missing paths absent from the Map (caller falls back to
      path-derived slug, same contract as resolveSlugByPathOrSourcePath).
    — Empty input short-circuits to empty Map (no SQL).

  src/core/engine-constants.ts (NEW)
    — Single source of truth for DELETE_BATCH_SIZE = 500.
    — Both engines import; no engine-from-engine coupling.
    — Lives outside engine.ts (the interface module) to avoid circular
      imports.

Also updates the deletePage JSDoc (CDX-11): drops the misleading
"hard delete is admin-only" framing. `gbrain sync` hard-deletes on
every run that sees a deleted file; not admin-only.

Co-Authored-By: garrytan-agents <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(sync): batched delete + rename + DRY refactor (v0.41.21.0 T2/T3/T4)

Replaces the per-file delete loop (sync.ts:1241-1257) and per-file
rename slug-resolve (sync.ts:1263-1295) with interleaved per-batch
flows using engine.resolveSlugsByPaths + engine.deletePages. Also
refactors resolveSlugByPathOrSourcePath (sync.ts:267) to delegate to
the new batch helper when sourceId is set — one owner of the SQL +
fallback semantics (D8).

ROUND-TRIP COUNTS (73K-delete commit):
  pre-fix:   73,000 SELECTs + 73,000 DELETEs = 146,000 (~5 hours)
  post-fix:     146 SELECTs +     146 DELETEs =     292 (~2 minutes)

Headline win: a single commit deleting 73K files no longer jams the
sync pipeline for hours, no longer cascades staleness across every
other source on the brain.

Shape (T2 delete loop, per the plan's ASCII diagram):

  filtered.deleted (73K paths)
        │
        ▼
   slice into batches of DELETE_BATCH_SIZE (500)
        │
        ▼  for each batch:
   abort-check ──► partial('timeout')
        │
        ▼
   engine.resolveSlugsByPaths(batch, {sourceId})  ◀── 1 SQL round-trip
        │
        ▼
   slugs = batch.map(path => map.get(path)
                  ?? resolveSlugForPath(path))    ◀── pure-JS fallback
        │                                              for frontmatter-
        ▼                                              fallback slugs
   try {
     deleted = engine.deletePages(slugs, opts)    ◀── 1 SQL round-trip
     pagesAffected.push(...deleted)               ◀── D6 confirmed only
   } catch {
     // D7 decompose: per-slug deletePage,
     // unrecoverable failures → failedFiles
   }

Per-batch try-catch (D7) decomposes batch DELETE failures to per-slug
deletePage so a transient blip on batch 73 doesn't lose 500 deletes —
it self-heals to one-at-a-time for that batch only. Unrecoverable
per-slug failures land in failedFiles (matching the existing
import-loop pattern at sync.ts:~1350). failedFiles declaration
hoisted above the delete loop so both delete decompose and import
loops feed the same sync-bookmark gate.

T4 rename loop: pre-resolves all `from` slugs in batches via
resolveSlugsByPaths BEFORE iterating. Per-file updateSlug + importFile
calls stay (those are inherently per-file). The try/catch around
updateSlug for slug-doesn't-exist preserves verbatim.

T3 DRY refactor: resolveSlugByPathOrSourcePath delegates to
resolveSlugsByPaths via a single-element array when sourceId is set.
When sourceId is undefined (legacy unscoped callers), falls back to
the original executeRaw shape — the batch engine surface requires
sourceId per D5 (multi-source-bug-class defense).

Atomicity coarsening (D3): each batch is one transaction. A mid-batch
abort or connection failure rolls back up to DELETE_BATCH_SIZE - 1
successful deletes from the in-flight batch. Sync is idempotent so
the next run picks them up via git diff regenerating the deletion
list. Documented at the call site + in the deletePages JSDoc.

Co-Authored-By: garrytan-agents <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(schema): global page-generation clock + statement-level trigger (v0.41.21.0 T5)

Migration v104: page_generation_clock_and_statement_trigger.

The pre-v0.41.21.0 query-cache Layer 1 bookmark read MAX(generation) FROM
pages to detect "writes happened since cache-store". Two bugs in that
contract — independent of any sync work, surfaced by codex
outside-voice on the /plan-eng-review pass:

  1. The row-level bump_page_generation_trg (migration v91) sets
     NEW.generation = OLD.generation + 1 on UPDATE. Updating a NON-MAX
     page didn't advance MAX(generation). Cache silently served stale
     for any UPDATE-to-non-max page. (CDX-2)
  2. The trigger is BEFORE INSERT OR UPDATE — DELETE doesn't fire it
     at all. Even an AFTER DELETE wouldn't move MAX (surviving rows
     are untouched). (CDX-1)

Fix: single-row page_generation_clock counter, bumped per-statement
(FOR EACH STATEMENT — per-row would turn a 73K-row batch DELETE into
73K UPDATEs on the same counter, recreating the bottleneck this PR
fixes elsewhere — codex CDX-4). Layer 1 reads the clock value
directly (T6, separate commit). Per-row pages.generation stays for
Layer 2 (per-page snapshot via jsonb_each + LEFT JOIN pages) which
doesn't care about MAX, only per-page advancement.

Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache
rows stored under the old MAX semantics aren't all instantly
invalidated on upgrade. Their max_generation_at_store stamp compares
cleanly against the seeded clock; future writes bump the clock and
the bookmark fires correctly.

  CREATE TABLE page_generation_clock (
    id    INTEGER PRIMARY KEY CHECK (id = 1),
    value BIGINT  NOT NULL DEFAULT 0
  );
  CREATE TRIGGER bump_page_generation_clock_trg
    AFTER INSERT OR UPDATE OR DELETE ON pages
    FOR EACH STATEMENT
    EXECUTE FUNCTION bump_page_generation_clock_fn();

Mirror in src/core/pglite-schema.ts so fresh PGLite installs get the
table + trigger via SCHEMA_SQL replay. The forward-reference bootstrap
probe doesn't need an entry: page_generation_clock is created directly
by SCHEMA_SQL (no separate index or FK references it), so the
schema-bootstrap-coverage gate is satisfied as-is.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cache): move Layer 1 to global clock + invalidate empty snapshots (v0.41.21.0 T6)

Closes the silent stale-cache bug class that's been live in master
since the bookmark feature shipped. Pre-fix, gbrain search would
silently serve stale cached results in three independent scenarios:

  1. UPDATE to a non-max-generation page (CDX-2) — the row-level
     trigger advanced per-page generation but didn't move
     MAX(generation), so the bookmark passed.
  2. DELETE of any page (CDX-1) — the trigger didn't fire at all,
     and even an AFTER DELETE wouldn't move MAX.
  3. Empty-result cache row + subsequent matching INSERT (CDX-6 /
     D20) — page_generations = '{}'::jsonb was "vacuously valid" via
     Layer 2, surviving any clock bump.

Fix:

  buildPageGenerationsSnapshot (store path)
    — Replaces the SELECT MAX(generation) FROM pages reads at
      cache-write time with SELECT value FROM page_generation_clock
      WHERE id = 1.
    — Empty pageIds path: only need the clock value (D20 contract).
    — Combined non-empty path: per-page generation (Layer 2 substrate)
      + clock value, both folded in one round trip via UNION ALL.

  CACHE_GATE_WHERE_CLAUSE (lookup path)
    — Layer 1 reads page_generation_clock.value (single-row O(1)
      lookup, faster than the pre-fix MAX(generation) backward index
      scan).
    — Layer 2 stricter: requires page_generations <> '{}'::jsonb AND
      the per-page check (not OR with the vacuously-valid `= '{}'`
      shortcut). Empty snapshots can no longer survive a Layer 1 miss.

  validateCacheRowAgainstPages (pure validator)
    — Layer 2 returns false for empty snapshots when Layer 1 fails.
    — Documented contract change.

Backward compat: pre-v0.40.3.0 cache rows have
max_generation_at_store = 0 AND page_generations = '{}'::jsonb. On a
populated brain, Layer 1 fails (clock > 0). Layer 2 is now stricter
so legacy rows invalidate once on first post-upgrade lookup, then the
cache fills back correctly. Acceptable one-time miss spike;
post-upgrade cache is structurally sound.

The clock seed (COALESCE(MAX(pages.generation), 0)) from migration
v104 keeps NON-empty legacy rows passing Layer 1 until the next
write — they don't all invalidate at once.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: cover v0.41.21.0 delete-batch + global clock + cache contract (T7+T8+T9)

Tests for every behavior the v0.41.21.0 wave introduces or changes.

New test files:

  test/sync-delete-batch.test.ts (PGLite hermetic)
    — engine.deletePages: empty input short-circuit, returns confirmed
      slugs (D6), multi-source isolation, cascade integrity (chunks +
      links cleared via FK), rejects oversized input.
    — engine.resolveSlugsByPaths: empty input, present + missing rows,
      D10 exotic-filename substrate (🌟.md / ทดสอบ.md / عربي.md),
      source isolation.
    — D13 pagesAffected filter: 100 deletable + 10 ghost paths →
      deletePages returns 100 (regression-pin: pre-fix would return
      all 110 via D6's pre-RETURNING shape).

  test/sync-delete-batch.slow.test.ts (.slow suffix keeps it out of
  the fast loop)
    — 10K-page batched delete completes in <5s on PGLite. Measured
      277ms on dev hardware (18x under the gate); pins the headline
      perf promise.

  test/sync-rename-batch.test.ts (PGLite hermetic)
    — 500-rename batch slug-resolve in 1 round-trip (exactly at
      DELETE_BATCH_SIZE boundary).
    — Frontmatter-fallback rename: exotic source_paths resolve via
      the batch SELECT.
    — Mixed present + missing: partial Map (missing → caller falls
      back to path-derived).

  test/page-generation-counter.test.ts (PGLite hermetic)
    — Statement-level trigger fires once per INSERT statement (raw
      SQL — NOT putPage, which uses ON CONFLICT DO UPDATE and bumps
      by 2 in PG semantics).
    — Statement-level trigger fires once per UPDATE statement.
    — Headline contract: batch DELETE bumps clock by 1, NOT by row
      count (25-row batch → +1).
    — CDX-1 regression: DELETE of non-max page bumps clock.
    — CDX-2 regression: UPDATE of non-max page bumps clock (raw SQL).
    — D14 end-to-end: clock advances after batch DELETE → cache rows
      stamped at the prior clock value are now stale by Layer 1.
    — CDX-6/D20: empty-result cache + INSERT matching page → clock
      advances (Layer 1 fires).
    — Documents the PG quirk: putPage's INSERT...ON CONFLICT DO UPDATE
      bumps clock by 2 (both INSERT and UPDATE triggers fire).

Test-helper update:

  test/helpers/reset-pglite.ts
    — Added page_generation_clock to PRESERVE_TABLES so the seeded
      single-row counter survives resetPgliteState between tests
      (same treatment as schema_version). Production never truncates.

Existing test contract inversions (CDX-6 / D20 fix):

  test/query-cache-gate.test.ts
    — Pre-v0.41.21.0 "vacuously valid for legacy empty snapshot"
      assertion inverted: empty snapshot now invalidates when Layer 1
      fires. Add positive CDX-6 regression test (empty-result + INSERT
      matching page).
    — SQL shape regression: page_generation_clock in Layer 1 (negative
      regression guard: MAX(generation) FROM pages MUST be gone).
    — Empty-snapshot reject guard:
      `qc.page_generations <> '{}'::jsonb` present; the old
      `qc.page_generations = '{}'::jsonb OR` shortcut MUST be gone.

  test/e2e/cache-gate-pglite.test.ts
    — Pre-v0.41.21.0 "legacy row serves vacuously" test inverted:
      legacy rows now invalidate on first clock advance post-upgrade.
    — CDX-1 regression: DELETE bumps clock → cached query for
      surviving pages invalidates.
    — CDX-2 regression: UPDATE-to-non-max-page bumps clock → cache
      invalidates.
    — CDX-11 comment fix: drop misleading "hard delete is admin-only"
      framing; gbrain sync hard-deletes on every run.

Engine parity extension:

  test/e2e/engine-parity.test.ts
    — deletePages parity: same input set, both engines return same
      string[] of confirmed-deleted slugs (D6).
    — resolveSlugsByPaths parity: same Map on both engines.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): v0.41.21.0 — batched sync deletes + global page-generation clock (T10)

VERSION bump (0.41.18.0 → 0.41.21.0; master is at 0.41.20.0 so
next free slot per the queue allocator). CHANGELOG entry with the
ELI10 lead per CLAUDE.md voice rules. CLAUDE.md annotations on
engine.ts, postgres-engine.ts, pglite-engine.ts, sync.ts, and
query-cache-gate.ts plus a new entry for engine-constants.ts.
llms-full.txt regenerated to match CLAUDE.md (per CLAUDE.md
mandatory rule).

Co-Authored-By: garrytan-agents <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* dx(test-runner): heartbeat shows real progress instead of 0p 0f

Bun's default test reporter doesn't print per-test markers — only a
single shard-end summary block when you pass it a file list. The
existing heartbeat tried to count `^[[:space:]]+✓` lines as a live
pass-count proxy, but bun never emits them in the multi-file mode this
runner uses, so every mid-run heartbeat showed `0p 0f` for the entire
12-20 minute wallclock. Users (and agents polling the runner) couldn't
distinguish "still bootstrapping" from "wedged" from "almost done."

Fix: parse three complementary real-time signals instead.

  1. Total files this shard was assigned — parsed from the
     `[unit-shard N/M] running X files` banner that run-unit-shard.sh
     echoes before invoking bun test. Available from second 1.

  2. PGLite initSchema() count — proxy for "test files started so far."
     Each PGLite-using test file's beforeAll triggers one initSchema(),
     which logs `Schema version 1 → 106 (101 migration(s) pending)`.
     Undercounts because not every test file opens a PGLite engine
     (covers ~30-60% of files in practice), but it's the only real-time
     progress signal bun's default reporter leaves in the log. The
     output uses a `~` prefix to convey "approximate count."

  3. Log size in KB — strictly monotonic liveness signal that works
     even when the PGLite count is still 0 (early-shard startup before
     the first initSchema fires).

  4. Per-shard elapsed time — formatted as MmSSs.

New mid-run heartbeat line:

  [heartbeat]  [s1: ~62/190f 476KB 12m31s] [s2: ~63/190f 513KB 12m31s] ...

When a shard finishes, the heartbeat upgrades to its final summary
including pass/fail counts from bun's end-of-shard summary block:

  [heartbeat]  [s1: done ✓ 2807p 0f] [s2: done ✓ 2784p 0f] ...

Portability: BSD awk on macOS doesn't support `match($0, /re/, arr)`
with the array sink — that's a gawk extension. The total-files parser
uses sed instead so the runner stays portable to the default Mac toolchain.

Helpers are pure functions and unit-testable in isolation: pass a log
file path, get the parsed number. No mocking. No bun runtime required.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): rebump v0.41.23.0 → v0.41.25.0

Per user request — skip v0.41.23.0 / v0.41.24.0 slots to land at
v0.41.25.0. Master is at v0.41.22.1, no version-trio collision.

Touches VERSION, package.json, CHANGELOG header, CLAUDE.md
annotations, src/core/engine-constants.ts header, src/core/migrate.ts
migration v106 comment, regenerated llms-full.txt + llms.txt.

Migration version (v106) and CDX1-6 trigger semantics unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(schema): reorder migration_impact_log AFTER minion_jobs (CI green)

Pre-existing bug in master's SCHEMA_SQL ordering, surfaced by CI on
this PR but lived silently on master since v0.41.18.0.

migration_impact_log declares `job_id BIGINT REFERENCES minion_jobs(id)`,
but its CREATE TABLE was at line 658 while minion_jobs's CREATE TABLE
was at line 778. On any fresh-install initSchema() the FK target
didn't exist yet:

  psql:/tmp/schema.sql:672: ERROR: relation "minion_jobs" does not exist

postgres-js's `unsafe()` aborts the multi-statement batch on the first
error response, so every CREATE TABLE after migration_impact_log
(including minion_jobs itself) never ran. Every subsequent CLI
subprocess that opened a connection then crashed with
`relation "minion_jobs" does not exist` on its first query.

Why master CI sometimes passed: the per-shard advisory lock + the
test setup's `engine.initSchema()` second pass (which runs the
migrations array) would eventually create minion_jobs via the v5
`minion_jobs_table` migration. From there migration_impact_log would
land via migration v103 with its FK resolving correctly. But CLI
subprocesses spawned by mechanical.test.ts's Parallel Import block
open their OWN connections and run a fresh `engine.connect() →
initSchema()` — that path runs SCHEMA_SQL FIRST and aborted at the
same forward-reference error before the migrations array could repair.

Fix: relocate the migration_impact_log CREATE TABLE + its two indexes
to AFTER the minion_jobs CREATE TABLE block (lines ~865), keeping
the rest of the schema layout intact. PGLite schema (pglite-schema.ts)
already had the correct ordering — only Postgres SCHEMA_SQL needed
the move.

Verified: fresh-DB local repro that previously failed 31/34 tests
with `relation minion_jobs does not exist` now passes 78/78 in
test/e2e/mechanical.test.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <noreply@anthropic.com>
2026-05-27 08:50:53 -07:00

GBrain

Search gives you raw pages. GBrain gives you the answer. It's the brain layer your AI agent has been missing — the only one that does synthesis, graph traversal, and gap analysis in one box.

I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: 146,646 pages, 24,585 people, 5,339 companies, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.

And now it works as a company brain too. Each person on the team gets their own slice of the brain, scoped by login. When you query, you only see what you're allowed to see — never another person's notes, never another team's data. We fuzz-tested this across every way you can read the brain (search, list, lookup, multi-source reads) and got zero leaks. Drop GBrain in as your team's shared institutional memory — the company-brain shape YC just put on its Request for Startups. If you're building in that space, you might as well build on this. Tutorial: set up GBrain as your company brain →

Lots of personal-knowledge systems give you keyword matching and grep in a box. GBrain does that, and adds two things nobody else ships together:

  • A synthesis layer that gives you the actual answer. Synthesized, well-cited prose across people, companies, deals, and ideas. Not "here are 10 chunks that mention your query"; an actual answer with citations and an explicit note on what the brain doesn't know yet. The gap analysis is the part that changes how you use the brain.
  • A self-wiring knowledge graph. Every page write extracts entity refs and creates typed edges (attended, works_at, invested_in, founded, advises) with zero LLM calls. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked: P@5 49.1%, R@5 97.9% on a 240-page Opus-generated rich-prose corpus, +31.4 points P@5 over its graph-disabled variant and over ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling gbrain-evals repo.

The point of building a 100K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.

It's easier to ship a daemon that runs 24/7 to ingest, enrich, and consolidate than it is to keep an agent in chat working hard. GBrain is that daemon, generalized. Install in 30 minutes. Your agent does the work. As my personal agent gets smarter, so does yours.

~30 minutes to a fully working brain. Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.

LLMs: fetch llms.txt for the documentation map, or llms-full.txt for the same map with core docs inlined in one fetch. Agents: start with AGENTS.md (or CLAUDE.md if you're Claude Code).

What this looks like

Say you have a meeting with Alice tomorrow. You want to walk in remembering what she works on, when you last talked, and what's still open between you. Here's what you'd type, and what you'd get back.

You ask:

"What do I need to know before my meeting with Alice tomorrow?"

Most personal-knowledge tools give you back a list of pages. Something like:

1. people/alice — Alice runs engineering at Acme...
2. meetings/2026-03-15-alice-q1 — Q1 product review with Alice...
3. meetings/2026-01-08-acme-kickoff — Kickoff meeting with Acme team...
4. customers/acme — Acme is a series-B fintech we work with...
5. notes/2026-04-22 — Quick chat with Alice about pricing...

Five pages you now have to open and read yourself to actually prepare. The tool found the right material, but it didn't do the work.

GBrain gives you back the answer, with sources:

Alice runs engineering at Acme (a series-B fintech). You last spoke
on April 22 in a quick pricing chat. Three things are still open
from that conversation:

1. She owes you the security review for the new tier
   (deadline was May 1; no update since).
2. You committed to pricing for a 500-seat tier
   (you sent it April 25; no response yet).
3. She mentioned they're hiring a CISO; you said you'd intro
   someone from your network.

Heads up: nothing's been added to the brain about Alice or Acme
since April 22, six weeks ago. She may have replied through email
or Slack DM, channels the brain doesn't see. Worth asking her to
catch up before assuming any of this is still current.

Every claim has a source page behind it. The "heads up" at the end tells you what the brain doesn't know yet, so you can ask Alice about it directly instead of being surprised. The brain just did your meeting prep.

This is the difference between a search engine and a brain. Search finds the pages. The brain reads them for you and writes the answer.

Install

GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.

If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:

Then paste this into your agent:

Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md

The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.

Never set up an AI agent platform before? The personal-brain tutorial walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.

Install it into your existing agent

Already running Codex, Claude Code, Cursor, or another coding agent? Paste the same instruction in:

Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md

This works in any agent that can read files over HTTPS and execute shell commands. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.

CLI standalone (no agent)

bun install -g github:garrytan/gbrain
gbrain init --pglite     # 2 seconds; no server, no Docker
gbrain doctor            # verify health
gbrain import ~/notes/   # index your markdown
gbrain query "what themes show up across my notes?"

Postgres-at-scale, Supabase, and thin-client setup paths live in docs/INSTALL.md.

Connect GBrain to your AI client (MCP)

GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:

  • Claude Code — one command: claude mcp add gbrain -- gbrain serve. Zero server, zero tunnel.
  • Cursor / Windsurf / any stdio MCP client — same shape, add {"command": "gbrain", "args": ["serve"]} to your MCP config.
  • Claude Desktop (Cowork) — Settings → Integrations → add the URL of your HTTP server. Remote only; the local claude_desktop_config.json does not work for remote servers.
  • Claude Cowork (team plan) — org Owner adds the connector under Organization Settings → Connectors.
  • Perplexity Computer — Settings → Connectors → add the URL + bearer token. Pro subscription required.
  • ChatGPT — uses OAuth 2.1 with PKCE (the hard requirement). Register a chatgpt client from the admin dashboard with grant type authorization_code.

For the HTTP server itself:

gbrain serve              # stdio MCP (local subprocess; for Claude Code, Cursor, Windsurf)
gbrain serve --http       # HTTP MCP with OAuth 2.1 + admin dashboard at /admin
                          # (required for Claude Desktop, Cowork, Perplexity, ChatGPT)

The HTTP server includes DCR-style client registration, scope-gated access (read / write / admin), and rate limiting. Deployment guides (ngrok, Railway, Fly.io) live under docs/mcp/.

Two ways to query your brain

Raw retrieval (what most personal-knowledge tools ship) and a synthesis layer that gives you an actual answer. They serve different jobs.

# raw retrieval: top pages by hybrid score, fast, no LLM cost
gbrain search "who's working on AI agents at portfolio companies?"

# brain layer: synthesized answer with citations and gap analysis
gbrain think "who's working on AI agents at portfolio companies?"

gbrain search returns the top retrieved pages, ranked by hybrid scoring (vector + keyword + RRF + source-tier boost + reranker). Use it when you want raw material to skim: agent context windows, citation lookups, finding a specific quote.

gbrain think runs the same retrieval, then composes a synthesized answer across the results with explicit citations to the source pages AND an honest note on what the brain doesn't know yet. The gap analysis is the differentiator: the answer tells you when a page is stale, when a claim is uncited, when two pages contradict each other, when there's a hole you should fill.

Why it compounds. Pair the brain layer with find_trajectory and you get answers like "how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here": well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 100K-page brain is worth the effort.

gbrain agent run "..." exposes the same surface to a sub-agent through the Minions queue, with crash-safe two-phase persistence. Same answers, durable.

How to get data in

One command, local or hosted, synchronous receipt:

gbrain capture "the thought I want to remember"
gbrain capture --file ./notes/today.md
echo "from a pipe" | gbrain capture --stdin
SLUG=$(gbrain capture "..." --quiet)

The page lands in the database and on disk in one move. Default slug inbox/YYYY-MM-DD-<hash8> so captures cluster in a predictable triage location. On thin-client installs the verb routes through MCP to the server: same command, same UX.

For webhook ingestion (Zapier / IFTTT / Apple Shortcuts):

curl -X POST https://your-brain/ingest \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/markdown" \
  -d "# a thought from a Shortcut"

For mobile capture, the inbox folder source picks up anything dropped into ~/.gbrain/inbox/ from iOS Shortcuts / AirDrop / Drafts / Finder.

Third-party skillpacks can ship custom ingestion sources (Granola, Linear, voice, OCR) against the versioned IngestionSource contract at gbrain/ingestion. See docs/skillpack-anatomy.md.

Your brain's shape (schema packs)

Most personal-knowledge tools force one fixed layout: their idea of "notes" + "people" + "tags." Drop a Notion export or your own years-old Obsidian vault on top, and the agent doesn't know what a Projects/ folder means or whether Reading/ is people or sources.

gbrain doesn't have a fixed layout. It ships with bundled schema packs and lets you author your own when none fit:

  • gbrain-base-v2 (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + note catch-all): person, company, media, tweet, social-digest, analysis, atom, concept, source, deal, email, slack, writing, project, note. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
  • gbrain-base (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via gbrain onboard --check --explaingbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'.
  • gbrain-recommended — extends gbrain-base with the 13 additional directories from docs/GBRAIN_RECOMMENDED_SCHEMA.md (source, place, trip, conversation, personal, civic, project, etc.). Activate with gbrain schema use gbrain-recommended.
  • Your own packgbrain schema detect clusters your actual filesystem into proposed types, gbrain schema suggest runs an LLM pass over them, and gbrain schema review-candidates --apply promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares migration_from: so existing brains can opt in): see docs/architecture/pack-upgrade-mechanism.md.
gbrain schema active                # which pack is running, which tier set it
gbrain schema list                  # bundled + installed packs
gbrain schema detect                # propose types matching your filesystem
gbrain schema suggest               # LLM-refined proposals on top of detect
gbrain schema review-candidates     # human gate: promote / rename / ignore
gbrain schema use my-pack           # activate

The active pack threads through every read + write path: parseMarkdown infers page type from the pack's path prefixes; whoknows scopes expert routing to types declared expert_routing: true; extract_facts runs only on extractable: true types; the search cache folds the pack name + version into its key so cross-pack contamination is structurally impossible. Switch packs and the brain re-interprets itself; switch back and nothing's lost.

Seven-tier resolution chain (per-call flag → env var → per-source DB key → brain-wide DB key → gbrain.yml~/.gbrain/config.jsongbrain-base default). Full reference + authoring guide: docs/architecture/schema-packs.md.

Tutorials

Step-by-step walkthroughs for getting the most out of GBrain. Each one takes you from zero to a working outcome, with concrete commands and real numbers.

More walkthroughs in progress: connecting an existing agent (Claude Code, Cursor, OpenClaw, Hermes) to a GBrain memory layer; setting up GBrain for VC dealflow with founder scorecards and meeting prep; migrating an existing Notion or Obsidian vault; indexing a codebase as a queryable code brain. Full tutorial index: docs/tutorials/.

Want to see a tutorial that isn't here yet? Open an issue describing the workflow you want documented.

What it does (the loop)

  signal   →   search   →   respond   →   write   →   auto-link   →   sync
  (every    (brain-first  (informed     (page +    (typed edges     (cron
  message)  retrieval)    by context)   timeline)  + backlinks)     keeps fresh)
  • Signal detector runs on every message your agent receives. Captures ideas, entity mentions, time-sensitive todos, names, links.
  • Brain-first lookup before any external API call. The cheapest, fastest, most personal information source you have.
  • Auto-link fires on every page write. No LLM calls; pure pattern matching on [[wiki/people/bob]] style references. New entity → new page stub → graph grows.
  • Cron-driven enrichment runs while you sleep: dedup people pages, fix citations, score salience, find contradictions, prep tomorrow's tasks.

The whole loop is described in docs/architecture/topologies.md with diagrams.

Capabilities

Hybrid search. Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (conservative, balanced, tokenmax) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in docs/eval/SEARCH_MODE_METHODOLOGY.md. Default: balanced with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run gbrain search "<query>" --explain to see per-stage attribution: base score, every boost that fired, what it multiplied. gbrain doctor ships a graph_signals_coverage check; gbrain search stats shows fire counts and failure breakdowns.

Self-wiring knowledge graph. Every put_page extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (attended, works_at, invested_in, founded, advises, mentions, …). Multi-hop traversal via gbrain graph-query. The graph is what produces the +31.4 P@5 lift over vector-only RAG.

Job queue (Minions). BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.

43 curated skills. Routing lives in skills/RESOLVER.md. Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.

Eval framework. gbrain eval longmemeval runs the public LongMemEval benchmark against your hybrid retrieval. gbrain eval export + gbrain eval replay capture real queries and replay them against code changes (set GBRAIN_CONTRIBUTOR_MODE=1). gbrain eval cross-modal cross-checks an output against the task using three different-provider frontier models. Full methodology in docs/eval/SEARCH_MODE_METHODOLOGY.md.

Brain consistency. gbrain eval suspected-contradictions samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.

Agent-authored schema (v0.40.7.0). Your brain has a shape — what page types exist (person, meeting, paper, case, lab-result), what they link to (attended, authored, prescribed-by), what facts get extracted automatically. The default ships with 22 universal types, but your brain's actual shape is not the default shape. Agents can now evolve that shape on your behalf via 14 gbrain schema CLI verbs + a batched MCP op (schema_apply_mutations, admin scope, NOT localOnly so remote agents reach it over HTTPS). Atomic file locks, audit log with the agent's identity, chunked UPDATE backfill in 1000-row batches that never wedge concurrent writers. The brain stops being a pile of notes and becomes something with structure. Why it matters: docs/what-schemas-unlock.md — 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator). 5-minute walkthrough: docs/schema-author-tutorial.md. Agent skill: skills/schema-author/SKILL.md.

Integrations

Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in recipes/ and is discoverable via gbrain integrations list.

  • Voice: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: recipes/twilio-voice-brain.md.
  • Email + calendar: webhook handlers that route to brain signals. docs/integrations/meeting-webhooks.md.
  • Embedding providers: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in docs/integrations/embedding-providers.md.
  • Rerankers: ZeroEntropy zerank-2 hosted (default in tokenmax mode) plus the v0.40.6.1 llama-server-reranker recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same gateway.rerank() seam. Setup walkthrough in docs/ai-providers/llama-server-reranker.md.
  • Credential gateway: vault-aware secret distribution. docs/integrations/credential-gateway.md.
  • MCP clients: every major MCP client is supported. docs/mcp/ per-client setup.

Architecture

Two engines, one contract. PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first BrainEngine interface in src/core/engine.ts defines ~47 operations both engines implement; CLI and MCP server are generated from one source.

Brain repo is the system of record. Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in docs/architecture/topologies.md.

Two organizational axes (brain ⊥ source). A brain is a database (your personal brain, a team mount you joined). A source is a repo inside that brain (wiki, gstack, an essay, a knowledge base). Routing lives in .gbrain-source dotfiles and resolves via a documented 6-tier precedence chain. Full diagrams in docs/architecture/brains-and-sources.md.

Why the graph matters. Vector search returns chunks that are semantically close. The graph returns chunks that are factually connected. Hybrid search pulls from both; auto-linking on every write keeps the graph fresh. Deep dive: docs/architecture/RETRIEVAL.md.

Troubleshooting

gbrain import fails with expected N dimensions, not M? Run gbrain doctor. It will print the exact gbrain config set ... or gbrain retrieval-upgrade command to repair the mismatch. You should not need to delete ~/.gbrain. Fresh gbrain init --pglite auto-detects your embedding provider from API keys in your environment: set OPENAI_API_KEY (or ZEROENTROPY_API_KEY / VOYAGE_API_KEY) before running init, or pass --embedding-model <provider>:<model> explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass --no-embedding to defer setup until runtime. See docs/integrations/embedding-providers.md for the full provider matrix and docs/operations/headless-install.md for Docker/CI sequencing.

Hourly cron sync keeps timing out on a federated brain? v0.41.13.0 ships two flags + a recommended pattern. Switch your cron to a per-source loop with shell timeout(1) doing the OS-level kill and gbrain self-terminating gracefully half-a-minute earlier:

gbrain sync --break-lock --all --max-age 1800
for src in $(gbrain sources list --json | jq -r '.[].id'); do
  timeout 600 gbrain sync --source "$src" --timeout 540 || true
done

When --timeout fires mid-import, gbrain sync exits 0 with status partial and last_commit UNCHANGED — the next run re-walks the same diff and content_hash short-circuits already-imported files. The --max-age 1800 first command self-heals any wedged-but-alive locks left by a hung previous run, using the v98 last_refreshed_at semantic (NOT acquired_at) so healthy long-running holders are safe by construction. See the v0.41.13.0 entry in CHANGELOG.md for the honest scope notes (extract + embed phases run to completion; 30-min rollout window for --max-age post-migration v98; full-sync triggers deferred to v0.42+).

Dream cycle silently losing wiki links on Supabase? v0.41.19.0 fixes the bug class structurally. The engine now self-retries every bulk batch write (addLinksBatch / addTimelineEntriesBatch / upsertChunks) on Supavisor pooler blips, with a 12s worst-case wait that covers the full 5-10s circuit-breaker recovery window. gbrain doctor surfaces incidents via the new batch_retry_health check (reads the last 24h of ~/.gbrain/audit/batch-retry-YYYY-Www.jsonl). To tune for an unusually slow pooler:

# Defaults: 3 retries, base 1s, max 10s, decorrelated jitter.
# Override per operator without a release:
export GBRAIN_BULK_MAX_RETRIES=5       # int >= 0; 0 disables retries
export GBRAIN_BULK_RETRY_BASE_MS=2000  # int > 0
export GBRAIN_BULK_RETRY_MAX_MS=15000  # int >= base

Bad values surface at gbrain doctor startup with a paste-ready fix (not at first-retry mid-cycle). PGLite-only installs pay zero cost — the retry wrap is engine-level, but PGLite has no pooler so retries never fire in practice.

gbrain brainstorm returning judge_failed: true with 0 scored ideas? v0.41.21.0 closes the two bugs that caused it. The judge hard-coded a 4K-token output cap; for any run past ~40 ideas the call truncated mid-JSON and the parser threw. Same release closes a slash- form pricing miss: gbrain brainstorm --judge-model anthropic/claude-sonnet-4-6 --max-cost 5 failed with BudgetExhausted reason=no_pricing because every pricing site only matched the colon form. Both shapes work now. No config change, no schema migration — gbrain upgrade is the whole fix.

Docs

  • docs/INSTALL.md — every install path, end to end
  • docs/what-schemas-unlock.md — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0)
  • docs/schema-author-tutorial.md — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via gbrain whoknows
  • docs/architecture/ — system design, topologies, retrieval theory
  • docs/guides/ — how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion)
  • docs/integrations/ — connecting external data sources (voice, email, calendar, embedding providers)
  • docs/mcp/ — per-client MCP setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)
  • docs/eval/ — eval framework, metric glossary, methodology
  • docs/ethos/ — philosophy (thin harness, fat skills, markdown as recipes, origin story)
  • AGENTS.md — entry point for non-Claude agents
  • CLAUDE.md — entry point for Claude Code (deep operating context)
  • CONTRIBUTING.md — contributor guide, test discipline, eval-capture mode
  • SECURITY.md — OAuth threat model, hardening defaults

Contributing

Run bun run test for the fast loop, bun run verify for the pre-push gate, bun run ci:local to run the full Docker-backed CI stack locally. Detailed test discipline in CONTRIBUTING.md.

Community PRs are batched into release waves rather than merged one-by-one — see the "PR wave workflow" section in CLAUDE.md. Contributor attribution stays attached via Co-Authored-By: trailers. We credit every accepted contribution in CHANGELOG.md.

If you find a bug or want a feature: open an issue first. Quick fixes (typo, doc bug, obvious regression) can go straight to a PR. Anything touching schema, retrieval ranking, MCP protocol, or the security boundary needs a design discussion in the issue first.

License + credit

MIT. I built GBrain to run my OpenClaw and Hermes deployments — the production brain behind my AI agents.

Origin story: docs/ethos/ORIGIN.md.

Community PR contributors are credited in CHANGELOG.md per release. ZeroEntropy (@zeroentropy) for the embedding + reranker stack that ships as the default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.

S
Description
No description provided
Readme MIT
160 MiB
Languages
TypeScript 97.4%
Shell 1.2%
JavaScript 0.9%
PLpgSQL 0.4%