dfa15ba22b v0.41.10.0 feat: orphan reduction via --by-mention + UTF-16 surrogate-pair fix (#1442)
* fix(synthesize): UTF-16 surrogate-safe hard-split in chunker

Part A of v0.42.0.0 fix wave: lifts surrogate-pair-safe slicing from
src/core/eval-contradictions/judge.ts into a new shared module
src/core/text-safe.ts. The dream-cycle chunker findBoundary tier-3
fallback (synthesize.ts) previously hard-split at maxChars, orphaning
a high surrogate when the boundary landed inside emoji / non-BMP CJK /
mathematical alphanumerics. Resulting chunks were not byte-identical
to the source content, which broke the v0.30.2 D9 stable-chunk-identity
invariant — the per-chunk idempotency key drifted across retries on
transcripts containing 4-byte UTF-8 characters near a hard-split.

Five agent-authored PRs (#1378-#1382) each independently introduced a
narrow safeSliceEnd helper that handled ONE of the three correctness
cases (high+low pair straddle) but missed the AT-low-surrogate case
that fires when a boundary lands inside a complete pair. The shared
text-safe.ts module exports both truncateUtf8 (the verbatim sliced
string, for judge.ts) and safeSplitIndex (the boundary index, for
chunker hot path), each covering all three cases.

Co-authored credit: @garrytan-agents for surfacing the fix in PRs
#1378-#1382 (closed in favor of consolidated design doc #1409).

* New: src/core/text-safe.ts (truncateUtf8 + safeSplitIndex helpers).
* New: test/text-safe.test.ts (18 cases, all 3 surrogate cases plus
  boundary-after-pair conservative back-up per codex CK16).
* refactor(judge): import truncateUtf8 from text-safe; re-export for
  back-compat. Existing 32 judge tests pass unchanged.
* fix(synthesize): findBoundary tier-3 routes through safeSplitIndex.
  3 new surrogate-safety cases in test/cycle-synthesize-chunker.test.ts
  (emoji at boundary, non-BMP CJK at boundary, determinism + joined
  chunks reconstruct source byte-identical across 5 fuzzed hashes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(schema): widen link_source CHECK to include 'mentions' (v95)

Part B of v0.42.0.0: link_source enum widening to admit a fourth
provenance channel for auto-linked body-text mentions from the
upcoming `gbrain extract links --by-mention` command.

Codex outside-voice review on the v0.42.0.0 plan caught that the
existing link_source CHECK is a hard wall (src/schema.sql:356) —
my earlier draft claimed "no schema migration needed; link_source
is free-form TEXT." Wrong. The CHECK admits only NULL OR
('markdown', 'frontmatter', 'manual'); attempting to insert
link_source='mentions' would have raised a constraint violation
on every auto-link write. Migration v95 widens the CHECK to admit
'mentions' alongside the three existing values.

Mentions are intentionally a separate provenance from markdown
(human-authored links) so the backlink-count SQL in postgres-engine
+ pglite-engine can filter `WHERE link_source != 'mentions'` for
search ranking (D12). Mentions still count toward orphan-ratio and
graph traversal — distinct semantics from the three human-authored
sources, modeled cleanly on the dedicated CHECK value.

* src/schema.sql: widened CHECK with provenance comment.
* src/core/pglite-schema.ts: same widening (PGLite engine parity).
* src/core/schema-embedded.ts: regenerated via `bun run build:schema`.
* src/core/migrate.ts: new migration v95
  `links_link_source_check_includes_mentions` with both Postgres
  and PGLite branches. DROP IF EXISTS + ADD CONSTRAINT pattern so
  re-applying the migration is a no-op (idempotent).
* test/schema-migrate-link-source-mentions.test.ts (NEW, 7 cases):
  registration shape, SQL shape (all 4 values present + DROP IF
  EXISTS pattern), PGLite branch present, post-migration insert
  succeeds, CHECK still rejects unknown values (widening did not
  nullify the gate), idempotent re-application via runMigration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(orphans): expose getOrphansData alias as canonical pure data fn (D1)

D1 from /plan-eng-review for v0.42.0.0: doctor's upcoming orphan_ratio
check needs the SAME exclusion logic as `gbrain orphans` so the two
surfaces cannot disagree on what counts as an orphan. The existing
findOrphans() was already the pure data fn — this commit just makes
that contract explicit via the getOrphansData alias and pins it with
an IRON RULE regression test.

* src/commands/orphans.ts: export const getOrphansData = findOrphans
  (alias, same function reference). Documents the v0.42.0.0 contract
  in findOrphans' docstring.
* test/orphans-pure-fn.test.ts (NEW, 12 cases):
  - getOrphansData === findOrphans (same reference).
  - findOrphans + getOrphansData deep-equal output.
  - includePseudo branch toggles excluded count.
  - CLI --json output deep-equals findOrphans (IRON RULE — catches
    drift if anyone adds CLI-side post-filtering).
  - CLI --count matches total_orphans (with and without --include-pseudo).
  - shouldExclude regression: pseudo-pages, auto-suffix, raw segment,
    deny-prefixes, first-segment exclusions all fire correctly;
    regular slugs are NOT excluded.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(engine): filter mentions out of backlink-count for search ranking (D12)

D12 from /plan-eng-review for v0.42.0.0: codex outside-voice review
caught that engine.getBacklinkCounts had NO link_source filter — so
every link counted equally toward backlink-boost in hybridSearch.
Running `gbrain extract links --by-mention` (migration #1 of #1409)
would silently shift search ranking globally on first run, boosting
popular-mention pages over intentional-backlink pages.

Add `AND l.link_source IS DISTINCT FROM 'mentions'` to the LEFT JOIN
in both engines. `IS DISTINCT FROM` is NULL-safe per the
[sql-neq-misses-null-drift] memory: a naive `!= 'mentions'` would
silently drop legacy pre-v0.13 rows where link_source IS NULL (because
NULL != 'mentions' evaluates to NULL not TRUE in SQL three-valued
logic). The IS DISTINCT FROM form treats NULL as a distinct value so
legacy rows still count toward backlinks — the only rows filtered are
the explicitly mention-derived ones from v0.42.0.0+.

Mentions still count toward:
  - orphan-ratio (the whole point — `findOrphans` runs against `links`
    with no source filter, so an auto-linked page is no longer an orphan)
  - graph traversal (`traverseGraph` walks all link_source values)
  - graph adjacency (`getAdjacencyBoosts` includes mentions in the
    induced subgraph counts)

Mentions are filtered ONLY from:
  - `getBacklinkCounts` (this commit) — the input to hybridSearch's
    backlink_boost stage

* src/core/postgres-engine.ts: AND clause on the LEFT JOIN.
* src/core/pglite-engine.ts: same change for engine parity.
* test/backlink-count-mention-filter.test.ts (NEW, 6 cases):
  - 10 markdown + 0 mention → count = 10
  - 0 markdown + 50 mention → count = 0
  - 10 markdown + 50 mention → count = 10
  - NULL link_source legacy rows still count (IS DISTINCT FROM semantics)
  - mixed (markdown + frontmatter + manual + mentions) → only mentions filtered
  - uninitialized slug returns 0

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(by-mention): pure mention scanner with gazetteer + guards (D2/D6/D12/D13)

Net new module powering migration #1 of #1409 (orphan reduction).
buildGazetteer queries entity-typed pages (hardcoded D2 filter:
person/company/organization/entity, pack-aware deferred to TODO-1) and
produces a token-Map lookup keyed by lowercase first-token. findMentionedEntities
is a pure function that scans body text against the gazetteer, applies
maximal-munch matching (longest entry wins at each offset), self-link
guard (D13), cross-source guard, and per-page first-mention-only cap
(1 link per source→target pair regardless of how many body mentions).

Token-Map + multi-word phrase pass per D6 — no new deps, no regex
alternation (pathological perf at 5K patterns), no Aho-Corasick (dep
tax not justified at this scale). At each token offset, lookup in
Map<lowercase, GazetteerEntry[]> is O(1); multi-word entries validate
subsequent tokens. Bucket pre-sorted longest-first so the first valid
entry IS the maximal-munch winner.

Ignore-list semantics per CK12: built-in ambiguous tokens (Apple,
Amazon, Square, Stripe, Box, Meta, Target, Oracle) suppressed at
gazetteer-build time ONLY when no corresponding entity page exists.
If the user has explicitly created companies/apple, gazetteer
presence wins — ignore list does NOT override user intent.

Min-name-length filter at 4 chars kills false-positive 2-3-char names
(AI, YC, X, IBM). Codex CK13 noted this trade-off will under-deliver
on 3-char real entities; pack-aware follow-up (TODO-1) can let users
opt 3-char entity types in deliberately.

Code-block stripping via existing stripCodeBlocks() from
link-extraction.ts. CK8 fix: stripCodeBlocks was internal-only; this
commit exports it so by-mention.ts can reuse without rolling its own
fenced/inline code parser.

* src/core/by-mention.ts (NEW, 240 LOC):
  - LINKABLE_ENTITY_TYPES const (hardcoded D2 type filter).
  - GazetteerEntry + Gazetteer + Mention types.
  - buildGazetteer(engine, opts) — engine-backed, hardcoded type filter,
    ignore-list at build time per CK12, sort buckets longest-first.
  - findMentionedEntities(text, gazetteer, opts) — pure, maximal-munch,
    guards (self-link/cross-source/first-mention-cap), code-block strip.
* src/core/link-extraction.ts: export stripCodeBlocks (CK8 fix).
* test/by-mention.test.ts (NEW, 22 cases):
  - All 20 plan-mandated cases.
  - Plus extraIgnore user-override case + LINKABLE_ENTITY_TYPES contract pin.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(extract): --by-mention auto-link entity mentions (migration #1 of #1409)

Wires the v0.42.0.0 mention scanner into 'gbrain extract links'. Mode
dispatch: when --by-mention is set, runs ONLY the new mention pass
(skips default link/frontmatter extract) so the two surfaces don't
conflict mid-run. The default extract path is unchanged.

Flag plumbing:
* --by-mention: opts into the mention pass. Mode dispatch.
* --source fs --by-mention rejected with paste-ready --source db
  fix-hint (D7: gazetteer needs the engine; FS-walk + DB-gazetteer is
  incoherent).
* timeline --by-mention rejected (mentions are a links-pass concern).
* --source-id scopes the page WALK; gazetteer remains brain-wide
  (cross-source guard in findMentionedEntities suppresses scanning
  pages in source A from auto-linking entities in source B).
* --since DATE filters the walk to recently-modified pages.
* --type filter applies (rarely useful; included for parity).
* --dry-run prints add_link action lines without writing; --json
  emits one JSON line per dry-run action.

extractMentionsFromDb function:
* buildGazetteer once per run via hardcoded type filter (D2).
* Walks pages via engine.listAllPageRefs (DB-source only).
* Reads body as compiled_truth || '\n\n' || COALESCE(timeline, '')
  per D3 — separator-joined so an end-of-compiled token doesn't
  merge with a start-of-timeline token into a false phrase match.
* findMentionedEntities returns Mention[] with self-link guard (D13)
  + cross-source guard + first-mention-only cap baked in.
* addLinksBatch with link_source='mentions' — distinct provenance
  channel that backlink-count filters out for search ranking (D12).
* Empty-gazetteer no-op with informative message (no entity pages =
  nothing to scan).

* src/commands/extract.ts: --by-mention flag + mode dispatch + FS
  rejection + extractMentionsFromDb function (~120 LOC).
* test/extract-by-mention.test.ts (NEW, 12 cases):
  end-to-end happy path, idempotency, --dry-run no writes, --json
  output shape, --source-id scoping, --source fs rejection with
  fix-hint, timeline rejection, mode dispatch (no markdown rows when
  --by-mention), coexistence of markdown + mention link_source on
  same (from,to) pair via ON CONFLICT key, schema migration
  verification (link_source='mentions' insert succeeds), empty-brain
  no-op, cross-source guard (team-b post → default acme = no link).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): orphan_ratio check on local + thin-client surfaces (D5/D11)

D5/D11 from /plan-eng-review for v0.42.0.0: surface orphan-page count
in 'gbrain doctor' so users discover the new --by-mention fix without
having to know the feature exists. Two surfaces because thin-client
installs (gbrain init --mcp-only) route to runRemoteDoctor entirely —
adding the check to runDoctor only would miss every brain-server
consumer (codex CK5 caught this exactly during outside-voice review).

Local surface (src/commands/doctor.ts):
* Inserts as check '9b' right after graph_coverage.
* Consumes getOrphansData() — the canonical pure data fn from T5 —
  so doctor and 'gbrain orphans --count' cannot disagree on the ratio.
* Vacuous gate at < 100 entity pages (small brains naturally show
  high orphan ratio; not actionable signal).
* warn > 0.5, fail > 0.8; both states recommend
  'gbrain extract links --by-mention' as the fix.

Thin-client surface (src/core/doctor-remote.ts):
* New exported runOrphanRatioCheck function. Mirrors local logic
  but routes through find_orphans MCP op (existing v0.12.3 op,
  scope: read — even minimal-scope thin-clients can call it).
* Operator-pointing hint: 'Ask the brain operator at <url> to run
  gbrain extract links --by-mention'. Thin-client users can't run
  the fix against a brain they don't host (v0.31.1 bug class).
* Network failure fall-back: returns informational ok with
  network_error detail, NOT fail — earlier mcp_smoke catches
  genuine unreachable; orphan_ratio is informational only.
* Skippable via the existing skipScopeProbe flag so hermetic
  fixtures that don't implement find_orphans on /mcp don't hang.

Wiring in --by-mention extract.ts integration test (fix-up):
CliOptions field is `progressInterval` not `progressIntervalMs`,
and `timeoutMs: null` is required. Pre-existing tsc error
surfaced when typechecking the new doctor changes.

* test/doctor-orphan-ratio.test.ts (NEW, 10 cases):
  - <100 entity pages → vacuous ok
  - 100+ entities + low ratio (20%) → ok
  - high ratio (70%) → warn with fix-hint
  - very high ratio (90%) → fail with urgency fix-hint
  - zero entity pages → vacuous ok
  - JSON envelope contains orphan_ratio check
  - Thin-client: network failure → informational ok with detail
  - Cross-surface parity: source greps verify orphan_ratio name and
    fix command appear in BOTH doctor.ts and doctor-remote.ts; local
    hint is self-fix, thin-client hint asks the operator.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): orphan-reduction end-to-end with cross-surface count parity

Pins the v0.42.0.0 design-doc claim shape — "material reduction in
orphan pages via --by-mention" — without committing to a specific %
(per TODO-4=C decision to soften the 88%->_30% promise into a
"material reduction, exact figure TBD via post-merge measurement on
representative brain").

3 e2e cases via hermetic PGLite:
* Seed 20 entities + 5 content pages mentioning 15 → assert orphan
  count drops by >=10 after --by-mention (material delta).
* Cross-check the D1 single-source contract end-to-end:
  gbrain orphans --count, getOrphansData() pure fn, and the doctor
  JSON orphan_ratio message all reflect the same numerator. If a
  future change makes them disagree, this fires.
* Re-run idempotency: second --by-mention invocation produces 0 new
  mention rows AND the first run actually created some (sanity gate
  so a no-op pass doesn't trivially satisfy the idempotency test).

* test/e2e/orphan-reduction.test.ts (NEW, 3 cases, hermetic PGLite,
  no DATABASE_URL needed).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* release: v0.41.10.0 — orphan reduction via --by-mention + surrogate-pair fix

Bumps VERSION + package.json to 0.41.10.0 (next available slot in the
v0.41.x queue after master moved to v0.41.4.0). Minor bump scope: new
CLI flag (`gbrain extract links --by-mention`), new schema migration
v95, new doctor check `orphan_ratio`, new public src/core/text-safe.ts
module, new src/core/by-mention.ts module, new link_source enum value
with ranking-filter semantic.

CHANGELOG entry follows the v0.41.x voice rules: ELI10 lead, To take
advantage block with paste-ready commands, How to turn it on, What
you'd see, Promise calibration (softens design-doc 88%->_30% claim
per codex CK13), What to watch for, Itemized changes split into Part
A (surrogate-pair fix) + Part B (auto-link --by-mention) + Follow-ups
(TODO-1 through TODO-4). Credits @garrytan-agents for the underlying
PR work (#1378-#1382 closed in favor of design doc #1409).

TODOS.md gets four new follow-up entries (pack-aware gazetteer,
cycle integration, MCP op, post-merge measurement).

System-of-record annotation: the addLinksBatch call in
extractMentionsFromDb carries `gbrain-allow-direct-insert` per the
canonical reconcile-layer write pattern.

3-line audit: VERSION + package.json + CHANGELOG top all on 0.41.10.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 14:56:38 -07:00

GBrain

Search gives you raw pages. GBrain gives you the answer. It's the brain layer your AI agent has been missing — the only one that does synthesis, graph traversal, and gap analysis in one box.

I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: 146,646 pages, 24,585 people, 5,339 companies, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.

And now it works as a company brain too. Each person on the team gets their own slice of the brain, scoped by login. When you query, you only see what you're allowed to see — never another person's notes, never another team's data. We fuzz-tested this across every way you can read the brain (search, list, lookup, multi-source reads) and got zero leaks. Drop GBrain in as your team's shared institutional memory — the company-brain shape YC just put on its Request for Startups. If you're building in that space, you might as well build on this. Tutorial: set up GBrain as your company brain →

Lots of personal-knowledge systems give you keyword matching and grep in a box. GBrain does that, and adds two things nobody else ships together:

  • A synthesis layer that gives you the actual answer. Synthesized, well-cited prose across people, companies, deals, and ideas. Not "here are 10 chunks that mention your query"; an actual answer with citations and an explicit note on what the brain doesn't know yet. The gap analysis is the part that changes how you use the brain.
  • A self-wiring knowledge graph. Every page write extracts entity refs and creates typed edges (attended, works_at, invested_in, founded, advises) with zero LLM calls. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked: P@5 49.1%, R@5 97.9% on a 240-page Opus-generated rich-prose corpus, +31.4 points P@5 over its graph-disabled variant and over ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling gbrain-evals repo.

The point of building a 100K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.

It's easier to ship a daemon that runs 24/7 to ingest, enrich, and consolidate than it is to keep an agent in chat working hard. GBrain is that daemon, generalized. Install in 30 minutes. Your agent does the work. As my personal agent gets smarter, so does yours.

~30 minutes to a fully working brain. Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.

LLMs: fetch llms.txt for the documentation map, or llms-full.txt for the same map with core docs inlined in one fetch. Agents: start with AGENTS.md (or CLAUDE.md if you're Claude Code).

What this looks like

Say you have a meeting with Alice tomorrow. You want to walk in remembering what she works on, when you last talked, and what's still open between you. Here's what you'd type, and what you'd get back.

You ask:

"What do I need to know before my meeting with Alice tomorrow?"

Most personal-knowledge tools give you back a list of pages. Something like:

1. people/alice — Alice runs engineering at Acme...
2. meetings/2026-03-15-alice-q1 — Q1 product review with Alice...
3. meetings/2026-01-08-acme-kickoff — Kickoff meeting with Acme team...
4. customers/acme — Acme is a series-B fintech we work with...
5. notes/2026-04-22 — Quick chat with Alice about pricing...

Five pages you now have to open and read yourself to actually prepare. The tool found the right material, but it didn't do the work.

GBrain gives you back the answer, with sources:

Alice runs engineering at Acme (a series-B fintech). You last spoke
on April 22 in a quick pricing chat. Three things are still open
from that conversation:

1. She owes you the security review for the new tier
   (deadline was May 1; no update since).
2. You committed to pricing for a 500-seat tier
   (you sent it April 25; no response yet).
3. She mentioned they're hiring a CISO; you said you'd intro
   someone from your network.

Heads up: nothing's been added to the brain about Alice or Acme
since April 22, six weeks ago. She may have replied through email
or Slack DM, channels the brain doesn't see. Worth asking her to
catch up before assuming any of this is still current.

Every claim has a source page behind it. The "heads up" at the end tells you what the brain doesn't know yet, so you can ask Alice about it directly instead of being surprised. The brain just did your meeting prep.

This is the difference between a search engine and a brain. Search finds the pages. The brain reads them for you and writes the answer.

Install

GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.

If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:

Then paste this into your agent:

Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md

The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.

Never set up an AI agent platform before? The personal-brain tutorial walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.

Install it into your existing agent

Already running Codex, Claude Code, Cursor, or another coding agent? Paste the same instruction in:

Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md

This works in any agent that can read files over HTTPS and execute shell commands. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.

CLI standalone (no agent)

bun install -g github:garrytan/gbrain
gbrain init --pglite     # 2 seconds; no server, no Docker
gbrain doctor            # verify health
gbrain import ~/notes/   # index your markdown
gbrain query "what themes show up across my notes?"

Postgres-at-scale, Supabase, and thin-client setup paths live in docs/INSTALL.md.

Connect GBrain to your AI client (MCP)

GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:

  • Claude Code — one command: claude mcp add gbrain -- gbrain serve. Zero server, zero tunnel.
  • Cursor / Windsurf / any stdio MCP client — same shape, add {"command": "gbrain", "args": ["serve"]} to your MCP config.
  • Claude Desktop (Cowork) — Settings → Integrations → add the URL of your HTTP server. Remote only; the local claude_desktop_config.json does not work for remote servers.
  • Claude Cowork (team plan) — org Owner adds the connector under Organization Settings → Connectors.
  • Perplexity Computer — Settings → Connectors → add the URL + bearer token. Pro subscription required.
  • ChatGPT — uses OAuth 2.1 with PKCE (the hard requirement). Register a chatgpt client from the admin dashboard with grant type authorization_code.

For the HTTP server itself:

gbrain serve              # stdio MCP (local subprocess; for Claude Code, Cursor, Windsurf)
gbrain serve --http       # HTTP MCP with OAuth 2.1 + admin dashboard at /admin
                          # (required for Claude Desktop, Cowork, Perplexity, ChatGPT)

The HTTP server includes DCR-style client registration, scope-gated access (read / write / admin), and rate limiting. Deployment guides (ngrok, Railway, Fly.io) live under docs/mcp/.

Two ways to query your brain

Raw retrieval (what most personal-knowledge tools ship) and a synthesis layer that gives you an actual answer. They serve different jobs.

# raw retrieval: top pages by hybrid score, fast, no LLM cost
gbrain search "who's working on AI agents at portfolio companies?"

# brain layer: synthesized answer with citations and gap analysis
gbrain think "who's working on AI agents at portfolio companies?"

gbrain search returns the top retrieved pages, ranked by hybrid scoring (vector + keyword + RRF + source-tier boost + reranker). Use it when you want raw material to skim: agent context windows, citation lookups, finding a specific quote.

gbrain think runs the same retrieval, then composes a synthesized answer across the results with explicit citations to the source pages AND an honest note on what the brain doesn't know yet. The gap analysis is the differentiator: the answer tells you when a page is stale, when a claim is uncited, when two pages contradict each other, when there's a hole you should fill.

Why it compounds. Pair the brain layer with find_trajectory and you get answers like "how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here": well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 100K-page brain is worth the effort.

gbrain agent run "..." exposes the same surface to a sub-agent through the Minions queue, with crash-safe two-phase persistence. Same answers, durable.

How to get data in

One command, local or hosted, synchronous receipt:

gbrain capture "the thought I want to remember"
gbrain capture --file ./notes/today.md
echo "from a pipe" | gbrain capture --stdin
SLUG=$(gbrain capture "..." --quiet)

The page lands in the database and on disk in one move. Default slug inbox/YYYY-MM-DD-<hash8> so captures cluster in a predictable triage location. On thin-client installs the verb routes through MCP to the server: same command, same UX.

For webhook ingestion (Zapier / IFTTT / Apple Shortcuts):

curl -X POST https://your-brain/ingest \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/markdown" \
  -d "# a thought from a Shortcut"

For mobile capture, the inbox folder source picks up anything dropped into ~/.gbrain/inbox/ from iOS Shortcuts / AirDrop / Drafts / Finder.

Third-party skillpacks can ship custom ingestion sources (Granola, Linear, voice, OCR) against the versioned IngestionSource contract at gbrain/ingestion. See docs/skillpack-anatomy.md.

Your brain's shape (schema packs)

Most personal-knowledge tools force one fixed layout: their idea of "notes" + "people" + "tags." Drop a Notion export or your own years-old Obsidian vault on top, and the agent doesn't know what a Projects/ folder means or whether Reading/ is people or sources.

gbrain doesn't have a fixed layout. It ships with two bundled schema packs and lets you author your own when neither fits:

  • gbrain-base (default) — the layout my production brain uses: people/, companies/, concepts/, meetings/, deal/, daily/, originals/, writing/, etc. Zero config. Drop a brain that fits this shape and everything works.
  • gbrain-recommended — extends gbrain-base with the 13 additional directories from docs/GBRAIN_RECOMMENDED_SCHEMA.md (source, place, trip, conversation, personal, civic, project, etc.). Activate with gbrain schema use gbrain-recommended.
  • Your own packgbrain schema detect clusters your actual filesystem into proposed types, gbrain schema suggest runs an LLM pass over them, and gbrain schema review-candidates --apply promotes the ones you like. Three commands and the brain knows your shape.
gbrain schema active                # which pack is running, which tier set it
gbrain schema list                  # bundled + installed packs
gbrain schema detect                # propose types matching your filesystem
gbrain schema suggest               # LLM-refined proposals on top of detect
gbrain schema review-candidates     # human gate: promote / rename / ignore
gbrain schema use my-pack           # activate

The active pack threads through every read + write path: parseMarkdown infers page type from the pack's path prefixes; whoknows scopes expert routing to types declared expert_routing: true; extract_facts runs only on extractable: true types; the search cache folds the pack name + version into its key so cross-pack contamination is structurally impossible. Switch packs and the brain re-interprets itself; switch back and nothing's lost.

Seven-tier resolution chain (per-call flag → env var → per-source DB key → brain-wide DB key → gbrain.yml~/.gbrain/config.jsongbrain-base default). Full reference + authoring guide: docs/architecture/schema-packs.md.

Tutorials

Step-by-step walkthroughs for getting the most out of GBrain. Each one takes you from zero to a working outcome, with concrete commands and real numbers.

More walkthroughs in progress: connecting an existing agent (Claude Code, Cursor, OpenClaw, Hermes) to a GBrain memory layer; setting up GBrain for VC dealflow with founder scorecards and meeting prep; migrating an existing Notion or Obsidian vault; indexing a codebase as a queryable code brain. Full tutorial index: docs/tutorials/.

Want to see a tutorial that isn't here yet? Open an issue describing the workflow you want documented.

What it does (the loop)

  signal   →   search   →   respond   →   write   →   auto-link   →   sync
  (every    (brain-first  (informed     (page +    (typed edges     (cron
  message)  retrieval)    by context)   timeline)  + backlinks)     keeps fresh)
  • Signal detector runs on every message your agent receives. Captures ideas, entity mentions, time-sensitive todos, names, links.
  • Brain-first lookup before any external API call. The cheapest, fastest, most personal information source you have.
  • Auto-link fires on every page write. No LLM calls; pure pattern matching on [[wiki/people/bob]] style references. New entity → new page stub → graph grows.
  • Cron-driven enrichment runs while you sleep: dedup people pages, fix citations, score salience, find contradictions, prep tomorrow's tasks.

The whole loop is described in docs/architecture/topologies.md with diagrams.

Capabilities

Hybrid search. Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (conservative, balanced, tokenmax) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in docs/eval/SEARCH_MODE_METHODOLOGY.md. Default: balanced with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run gbrain search "<query>" --explain to see per-stage attribution: base score, every boost that fired, what it multiplied. gbrain doctor ships a graph_signals_coverage check; gbrain search stats shows fire counts and failure breakdowns.

Self-wiring knowledge graph. Every put_page extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (attended, works_at, invested_in, founded, advises, mentions, …). Multi-hop traversal via gbrain graph-query. The graph is what produces the +31.4 P@5 lift over vector-only RAG.

Job queue (Minions). BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.

43 curated skills. Routing lives in skills/RESOLVER.md. Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.

Eval framework. gbrain eval longmemeval runs the public LongMemEval benchmark against your hybrid retrieval. gbrain eval export + gbrain eval replay capture real queries and replay them against code changes (set GBRAIN_CONTRIBUTOR_MODE=1). gbrain eval cross-modal cross-checks an output against the task using three different-provider frontier models. Full methodology in docs/eval/SEARCH_MODE_METHODOLOGY.md.

Brain consistency. gbrain eval suspected-contradictions samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.

Agent-authored schema (v0.40.7.0). Your brain has a shape — what page types exist (person, meeting, paper, case, lab-result), what they link to (attended, authored, prescribed-by), what facts get extracted automatically. The default ships with 22 universal types, but your brain's actual shape is not the default shape. Agents can now evolve that shape on your behalf via 14 gbrain schema CLI verbs + a batched MCP op (schema_apply_mutations, admin scope, NOT localOnly so remote agents reach it over HTTPS). Atomic file locks, audit log with the agent's identity, chunked UPDATE backfill in 1000-row batches that never wedge concurrent writers. The brain stops being a pile of notes and becomes something with structure. Why it matters: docs/what-schemas-unlock.md — 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator). 5-minute walkthrough: docs/schema-author-tutorial.md. Agent skill: skills/schema-author/SKILL.md.

Integrations

Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in recipes/ and is discoverable via gbrain integrations list.

  • Voice: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: recipes/twilio-voice-brain.md.
  • Email + calendar: webhook handlers that route to brain signals. docs/integrations/meeting-webhooks.md.
  • Embedding providers: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in docs/integrations/embedding-providers.md.
  • Rerankers: ZeroEntropy zerank-2 hosted (default in tokenmax mode) plus the v0.40.6.1 llama-server-reranker recipe for fully-local cross-encoder rerank via llama.cpp — runs Qwen3-Reranker or self-hosted ZeroEntropy weights against the same gateway.rerank() seam. Setup walkthrough in docs/ai-providers/llama-server-reranker.md.
  • Credential gateway: vault-aware secret distribution. docs/integrations/credential-gateway.md.
  • MCP clients: every major MCP client is supported. docs/mcp/ per-client setup.

Architecture

Two engines, one contract. PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first BrainEngine interface in src/core/engine.ts defines ~47 operations both engines implement; CLI and MCP server are generated from one source.

Brain repo is the system of record. Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in docs/architecture/topologies.md.

Two organizational axes (brain ⊥ source). A brain is a database (your personal brain, a team mount you joined). A source is a repo inside that brain (wiki, gstack, an essay, a knowledge base). Routing lives in .gbrain-source dotfiles and resolves via a documented 6-tier precedence chain. Full diagrams in docs/architecture/brains-and-sources.md.

Why the graph matters. Vector search returns chunks that are semantically close. The graph returns chunks that are factually connected. Hybrid search pulls from both; auto-linking on every write keeps the graph fresh. Deep dive: docs/architecture/RETRIEVAL.md.

Troubleshooting

gbrain import fails with expected N dimensions, not M? Run gbrain doctor. It will print the exact gbrain config set ... or gbrain retrieval-upgrade command to repair the mismatch. You should not need to delete ~/.gbrain. Fresh gbrain init --pglite auto-detects your embedding provider from API keys in your environment: set OPENAI_API_KEY (or ZEROENTROPY_API_KEY / VOYAGE_API_KEY) before running init, or pass --embedding-model <provider>:<model> explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass --no-embedding to defer setup until runtime. See docs/integrations/embedding-providers.md for the full provider matrix and docs/operations/headless-install.md for Docker/CI sequencing.

Docs

  • docs/INSTALL.md — every install path, end to end
  • docs/what-schemas-unlock.md — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0)
  • docs/schema-author-tutorial.md — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via gbrain whoknows
  • docs/architecture/ — system design, topologies, retrieval theory
  • docs/guides/ — how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion)
  • docs/integrations/ — connecting external data sources (voice, email, calendar, embedding providers)
  • docs/mcp/ — per-client MCP setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)
  • docs/eval/ — eval framework, metric glossary, methodology
  • docs/ethos/ — philosophy (thin harness, fat skills, markdown as recipes, origin story)
  • AGENTS.md — entry point for non-Claude agents
  • CLAUDE.md — entry point for Claude Code (deep operating context)
  • CONTRIBUTING.md — contributor guide, test discipline, eval-capture mode
  • SECURITY.md — OAuth threat model, hardening defaults

Contributing

Run bun run test for the fast loop, bun run verify for the pre-push gate, bun run ci:local to run the full Docker-backed CI stack locally. Detailed test discipline in CONTRIBUTING.md.

Community PRs are batched into release waves rather than merged one-by-one — see the "PR wave workflow" section in CLAUDE.md. Contributor attribution stays attached via Co-Authored-By: trailers. We credit every accepted contribution in CHANGELOG.md.

If you find a bug or want a feature: open an issue first. Quick fixes (typo, doc bug, obvious regression) can go straight to a PR. Anything touching schema, retrieval ranking, MCP protocol, or the security boundary needs a design discussion in the issue first.

License + credit

MIT. I built GBrain to run my OpenClaw and Hermes deployments — the production brain behind my AI agents.

Origin story: docs/ethos/ORIGIN.md.

Community PR contributors are credited in CHANGELOG.md per release. ZeroEntropy (@zeroentropy) for the embedding + reranker stack that ships as the default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.

S
Description
No description provided
Readme MIT
160 MiB
Languages
TypeScript 97.4%
Shell 1.2%
JavaScript 0.9%
PLpgSQL 0.4%