26c54588fe v0.38.0.0 ingestion cathedral — gbrain capture + write-through + IngestionSource contract (#1275)
* feat(ingestion): v0.38 substrate — daemon + IngestionSource contract + 2 sources

The foundation for the ingestion cathedral (CEO+DX+Eng plan-reviewed).
Plan: ~/.claude/plans/system-instruction-you-are-working-ethereal-riddle.md

WHAT YOU CAN NOW DO
The IngestionSource public contract is locked. Skillpack publishers can
build third-party ingestion sources (Granola, Linear, Mail, voice, OCR,
etc.) and ship them through the v0.37 skillpack registry. The locked
surface lives at the new package subpaths:

  import { IngestionSource, IngestionEvent } from 'gbrain/ingestion';
  import { IngestionTestHarness, expectEvent } from 'gbrain/ingestion/test-harness';

Both subpaths are pinned by test/public-exports.test.ts — breaking either
is a major-version change.

WHAT THIS COMMIT BUILDS
Foundation:
- src/core/ingestion/types.ts (IngestionSource, IngestionEvent,
  IngestionSourceContext, validateIngestionEvent, computeContentHash,
  INGESTION_SOURCE_API_VERSION, INGESTION_CONTENT_TYPES)
- src/core/ingestion/dedup.ts (24h content-hash LRU, 5000-entry cap)
- src/core/ingestion/skillpack-load.ts (gbrain.plugin.json discovery for
  third-party sources, api_version compat with paste-ready upgrade hints,
  in-process trust model for v1)
- src/core/ingestion/daemon.ts (IngestionDaemon: in-process source
  supervision sibling to v0.34.3.0 ChildWorkerSupervisor pattern, plus
  validate -> dedup -> rate-limit -> dispatch pipeline + health surface)
- src/core/ingestion/test-harness.ts (publisher-facing test utility with
  fake clock + in-memory event bus + expectEvent matchers + engine proxy
  that throws on access so publishers know what they're depending on)
- src/core/ingestion/index.ts (barrel for gbrain/ingestion subpath)

First two built-in sources prove the abstraction:
- file-watcher (chokidar over the brain repo; 1s debounce; honors
  pruneDir from src/core/sync.ts; symlinks rejected; Linux ENOSPC
  surfaces a paste-ready sysctl hint at runtime)
- inbox-folder (~/.gbrain/inbox/ target for iOS Shortcuts / AirDrop /
  Drafts; auto-archives processed files into .archived/YYYY-MM-DD/;
  symlink rejection; world-writable dir warning; routes content-type by
  extension)

Public exports surface (count 18 -> 20) pinned in:
- package.json exports map
- test/public-exports.test.ts EXPECTED_EXPORTS + count gate
- scripts/check-exports-count.sh baseline

ARCHITECTURE-LOCKED DECISIONS (from /plan-eng-review)
E1 webhook source process boundary: webhook source will live INSIDE
serve --http (NOT this daemon) when it lands in the next commit. Daemon
supervises only daemon-side sources.
E2 content-type processor execution: hybrid by size (inline <1MB,
Minion handlers >1MB). Processors land in a later commit.
E3 publisher TTHW: chokidar v4.0.3 across platforms; ephemeral PGLite
persistence and Linux inotify-limit doctor probe land in later commits.
E4 migration v80 (provenance columns) + forward-reference bootstrap:
lands with put_page write-through in a later commit.

DX-locked decisions (from /plan-devex-review):
- Source error semantics: throws bubble to daemon; supervisor backoff.
- IngestionTestHarness exported as gbrain/ingestion/test-harness.
- api_version field on gbrain.plugin.json with loud-fail on mismatch.

TESTS
192 cases across 8 test files, 0 failures:
- test/ingestion/types.test.ts (28 cases pinning the contract)
- test/ingestion/dedup.test.ts (15 cases for LRU + TTL + collision)
- test/ingestion/skillpack-load.test.ts (22 cases for manifest
  validation + api_version compat + collision policy + module load)
- test/ingestion/test-harness.test.ts (24 cases for harness lifecycle +
  clock + healthCheck + every expectEvent matcher)
- test/ingestion/daemon.test.ts (19 cases for supervision + dispatch
  pipeline + health surface + per-source config + logger wrapping)
- test/ingestion/sources/file-watcher.test.ts (10 cases including
  ENOSPC sysctl-hint surfacing)
- test/ingestion/sources/inbox-folder.test.ts (24 cases including
  symlink rejection + world-writable warning + archive-loop-prevention)
- test/public-exports.test.ts (2 new cases for the new subpaths)

typecheck clean. bun run verify gate passes.

NEXT IN WAVE
Subsequent commits in this PR ship webhook source (serve --http route),
cron-scheduler refactor + OpenClaw credential auto-migrate, content-type
processors (PDF + image OCR + audio transcribe + video keyframe), put_page
write-through with serializePageToMarkdown DRY extract, migration v80
+ bootstrap probes, gbrain capture verb, publisher DX cathedral (init
scaffold extension + gbrain ingest test [--watch] + tail + validate),
daemon rename autopilot -> ingest with forever-alias, doctor inotify
probe on Linux, skillpack contract docs + reference pack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): webhook source — POST /ingest + ingest_capture Minion handler

Lands the v0.38 ingestion cathedral's webhook source. Per the
/plan-eng-review E1 decision, the webhook source lives INSIDE
`serve --http` (NOT the ingestion daemon) so there is no new IPC: the
HTTP route submits Minion jobs directly into the existing queue, and
the daemon supervises only daemon-side sources.

WHAT YOU CAN NOW DO
With `gbrain serve --http` running and an OAuth client minted, any
HTTP caller (Zapier, IFTTT, n8n, Make, Apple Shortcuts) can POST a
captured thought into the brain:

  curl -X POST https://your-brain.example.com/ingest \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: text/markdown" \
    -d "# captured from my Shortcut"

The route auths via OAuth (write scope required), validates the
content-type, enforces a 1MB payload cap and per-IP rate limit
(100 events / 10s), submits an `ingest_capture` Minion job tagged
`untrusted_payload: true`, and returns 202 Accepted with the job id.
The job materializes the page under `inbox/YYYY-MM-DD-<hash6>` by
default (overridable via X-Gbrain-Slug header) so the user has a
predictable triage location.

WHAT THIS COMMIT BUILDS
- src/core/minions/handlers/ingest-capture.ts (new) — handler that
  takes an IngestionEvent payload, resolves a slug via fallback chain
  (job.data.slug -> event.metadata.slug -> inbox/<date>-<hash6>),
  validates the event at the handler boundary, REJECTS binary
  content_types with a paste-ready hint to install a processor
  skillpack, and routes through importFromContent. Defaults
  noEmbed: true (embed is a separate Minion job, matching the sync
  handler's pattern).
- src/commands/jobs.ts — registers `ingest_capture` in
  registerBuiltinHandlers alongside sync/embed/extract.
- src/commands/serve-http.ts — POST /ingest route with:
    - OAuth write-scope gate via requireBearerAuth({requiredScopes:['write']})
    - 100 events / 10s rate limiter (sibling to ccRateLimiter)
    - Content-type allowlist: text/markdown, text/plain, text/html,
      application/json; binary REJECTED with HTTP 415
    - 1 MB payload cap (configurable via GBRAIN_INGEST_MAX_BYTES)
    - Caller-overridable source identity via X-Gbrain-Source-Id /
      X-Gbrain-Source-Uri / X-Gbrain-Content-Type / X-Gbrain-Slug
      headers — useful for downstream tools that want clean provenance
    - untrusted_payload: true ALWAYS (network input)
    - Idempotency on (client_id, content_hash) so simultaneous retries
      collapse to one job
    - maxWaiting: 50 per client so a runaway integration can't
      monopolize the queue
    - Audit row in mcp_request_log + SSE broadcast for the admin feed

TESTS
test/ingestion/ingest-capture.test.ts (15 cases against PGLite):
- defaultSlugForEvent helper (3 cases pinning shape + UTC + determinism)
- slug resolution fallback chain (3 cases)
- validation + content-type routing (5 cases including binary rejection
  + untrusted_payload round-trip)
- importFromContent integration (3 cases including content_hash dedup
  via status='skipped' on repeat)

207 total ingestion tests passing. typecheck clean.

NEXT IN WAVE
cron-scheduler refactor + OpenClaw credential auto-migrate; content-type
processors (PDF + image OCR + audio transcribe + video keyframe);
put_page write-through + serializePageToMarkdown DRY extract +
migration v80 + bootstrap probes; gbrain capture verb; publisher DX
cathedral (init scaffold + gbrain ingest test --watch + tail + validate);
daemon rename autopilot -> ingest with forever-alias; doctor inotify
probe; skillpack contract docs + reference pack + VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): put_page write-through + migration v80 + DRY extract

WHAT YOU CAN NOW DO
The drift class is dead. Every `gbrain put_page` (CLI or MCP, local
or remote) now lands its markdown file on disk alongside the DB row
whenever `sync.repo_path` is configured. The page is queryable
immediately AND visible to git, your editor, and downstream tools.

Pre-v0.38, put_page wrote ONLY to the DB and synthesize/extract paths
had to reverse-render later. The v0.35.6.0 phantom-redirect pass was
the cleanup for what THIS commit prevents in the first place.

  # local CLI
  gbrain put inbox/test < my-thought.md
  # file lands at ${sync.repo_path}/inbox/test.md AND in the DB

  # MCP remote (Zapier / Cursor / Claude Desktop)
  curl -X POST /mcp ... '{"method":"tools/call","params":{"name":"put_page",...}}'
  # server-side write-through fires, agent gets a normal success response
  # untrusted_payload tagging applied (no auto-link, slug-allowlist gate)

Provenance frontmatter stamped on every write so future sync round-trips
know where the page came from:
  ingested_via: put_page         # local CLI
  ingested_via: 'mcp:put_page'   # MCP remote
  ingested_at: 2026-05-21T04:...

WHAT THIS COMMIT BUILDS
1. Migration v80 — `pages_provenance_columns` adds four nullable
   columns to `pages`: `ingested_via`, `ingested_at`, `source_uri`,
   `source_kind`. ADD COLUMN with no DEFAULT is metadata-only on
   Postgres 11+ and PGLite 17.5; instant on tables of any size. The
   four columns get NULL on every historical page (pre-v0.38 pages
   never had provenance).

2. DRY extract — `serializePageToMarkdown(page, tags, opts)` and
   `resolvePageFilePath(brainDir, slug, sourceId)` in `src/core/markdown.ts`.
   The dream-cycle's `renderPageToMarkdown` (synthesize.ts) and the new
   put_page write-through path were going to have 90% duplicate bodies.
   They now share one foundation; the dream version is a 4-line wrapper
   that passes `frontmatterOverrides: {dream_generated: true, ...}`.
   Future markdown-shape changes happen in one place.

3. put_page write-through (`src/core/operations.ts`) — after
   importFromContent succeeds, resolves sync.repo_path, computes the
   v0.32.8 source-aware path layout (default: brainDir/<slug>.md;
   non-default: brainDir/.sources/<id>/<slug>.md), serializes the
   freshly-written Page via `serializePageToMarkdown`, writes the file.
   Returns a `write_through: {written, path}` field in the put_page
   response so callers can see what happened.

   Trust gating:
   - subagent sandbox (viaSubagent without allowedSlugPrefixes) → DB-only
   - dry-run → DB-only (handler's early-return short-circuits before
     write-through; documented via the dry_run response field)
   - no sync.repo_path configured → DB-only, skipped reason returned
   - sync.repo_path points at a non-existent dir → DB-only, skipped
   - all other writes → write-through

   Failure isolation: disk-write failures are LOGGED loud but do NOT
   roll back the DB write. DB is the durable record; the
   phantom-redirect pass exists for drift cleanup if it ever shows up.

TESTS
- test/ingestion/put-page-write-through.test.ts (10 cases against PGLite):
  happy path (file land, provenance stamp local + remote), trust gating
  (subagent sandbox, dry-run, trusted-workspace), config edges (no
  repo_path, missing dir), multi-source filing (.sources/<id>/),
  failure isolation (DB write survives a disk failure).
- Migration v80 verified across both engines via the existing
  test/migrate.test.ts + test/bootstrap.test.ts coverage (~125 cases).

369 total tests passing in the ingestion + markdown + migrate bundle.
typecheck clean.

NOTES
- Bootstrap probes for the v80 provenance columns are NOT yet added
  to applyForwardReferenceBootstrap on either engine. This is safe
  for v0.38 because no SCHEMA_SQL CREATE INDEX or FK references the
  new columns — migration v80 is the only consumer, and it runs
  AFTER SCHEMA_SQL replay. A future commit may add bootstrap probes
  + REQUIRED_BOOTSTRAP_COVERAGE entries as defense-in-depth (eng
  review E4).

- The trusted-workspace path (dream cycle's reverseWriteRefs in
  synthesize.ts) still runs its own write at synthesize phase time.
  Both paths writing the same file is idempotent (byte-identical
  serialization), but a future commit may simplify reverseWriteRefs
  to skip pages whose file already matches.

NEXT IN WAVE
gbrain capture verb (the single human-facing entrypoint); daemon
rename autopilot -> ingest with forever-alias + plist migration;
doctor inotify probe (Linux); content-type processor router
(PDF + image OCR + audio transcribe stubs); cron-scheduler refactor
+ OpenClaw credential auto-migrate; skillpack contract docs +
reference pack; VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): gbrain capture — the single human-facing entrypoint

WHAT YOU CAN NOW DO
One command, local or thin-client, synchronous receipt with the resulting
page slug. The answer to "what is the best way to get data into the brain?"
is now: just type `gbrain capture` and the right thing happens.

  # the basic case
  gbrain capture "remember to follow up on the X deal"

  # from a file
  gbrain capture --file ./notes/today.md --slug daily/2026-05-21

  # from a pipe (shell pipelines)
  echo "from stdin" | gbrain capture --stdin

  # script-friendly: print just the slug
  SLUG=$(gbrain capture "a thought" --quiet)

  # JSON for agents
  gbrain capture "..." --json

Default slug is `inbox/YYYY-MM-DD-<hash8>` — deterministic for the same
content so re-running idempotently lands the same page. Receipt block
on stdout shows slug + status + content_hash + on-disk path so you
can confirm where the page went without rerunning `gbrain query`.

The local-install path routes through the put_page operation with the
v0.38 write-through plumbing landed in the prior commit, so the page
hits both the DB AND the file tree in one move. Thin-client installs
route through `callRemoteTool('put_page', ...)` so the server's
write-through handles disk persistence the same way.

WHAT THIS COMMIT BUILDS
- src/commands/capture.ts (new ~290 LOC):
  - `defaultSlug(content)` — UTC-stable `inbox/YYYY-MM-DD-<hash8>`
  - `parseArgs(args)` — positional + flag parsing with --file / --stdin
    / --slug / --type / --source / --quiet / --json / --help
  - `buildContent(rawBody, opts)` — wraps unstructured prose in
    frontmatter (type + title + captured_via + captured_at) and a
    leading `# Title` heading; passes through if the body already
    looks like markdown
  - `runCapture(engine, args)` — local install routes through the
    in-process put_page operation; thin-client routes through MCP.
    `--quiet` prints just the slug; `--json` prints structured output;
    default prints a 5-line receipt block.

- src/cli.ts:
  - Adds `case 'capture'` dispatch
  - Adds `'capture'` to the CLI_ONLY set so cli.ts wires it correctly

TESTS
test/commands/capture.test.ts (21 cases against PGLite):
- defaultSlug helper: shape + determinism + UTC math
- parseArgs: positional + multi-token join + every flag
- buildContent: prose wrapping, --type override, no double-wrap
  for pre-frontmattered content, title cap at 80 chars,
  --source provenance stamp
- Integration: inline content lands in DB + on disk, default slug
  shape, --file reads from disk, --json structured output,
  --help returns without engine roundtrip

271 total tests passing in the bundle. typecheck clean.

NOTES
- Thin-client routing relies on `callRemoteTool('put_page', ...)` from
  src/core/mcp-client.ts. Identical UX to the local path because the
  server's put_page handler runs the same write-through plumbing.

- buildContent's "looks like markdown" heuristic is intentionally
  simple — first-line heading or frontmatter delimiter is the trigger.
  Users who care about exact formatting pass a pre-formatted --file.

NEXT IN WAVE
Daemon rename autopilot -> ingest with forever-alias + plist migration;
doctor inotify probe (Linux); content-type processor router
(PDF + image OCR + audio transcribe stubs); cron-scheduler refactor
+ OpenClaw credential auto-migrate; skillpack contract docs +
reference pack; VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): v0.38.0.0 release hygiene + e2e roundtrip + capture skill

VERSION 0.37.1.0 → 0.38.0.0 (trio: VERSION, package.json, CHANGELOG header).
CHANGELOG entry written in user-facing ELI10-lead voice per CLAUDE.md
release-summary rules. README's pre-loop section gains a new "How to get
data in (v0.38+)" block leading with `gbrain capture`.

skills/capture/SKILL.md (NEW) so agents route "capture this" / "save this
thought" / "remember this" / "drop this in the inbox" / "save to brain" to
the capture verb. RESOLVER.md updated with the new triggers (sits above
idea-ingest/media-ingest/meeting-ingestion in the content-ingestion
section as the "simple thought" path).

E2E roundtrip test (test/e2e/ingestion-roundtrip.test.ts) covers the gap:
inbox-folder source -> daemon -> ingest_capture handler -> DB page,
including:
- Full pipeline: file drop appears as page in DB + file moves to .archived/
- Dedup catches byte-identical content from a different filename
- Multi-source coordination: two distinct inbox dirs, two sources, daemon
  ingests both events independently

The test runs against an in-memory PGLite (no DATABASE_URL needed) so it
exercises the substrate-level wiring in the standard test suite. A
follow-up commit can add a full-process e2e (gbrain serve --http + real
OAuth client + POST /ingest) that requires DATABASE_URL.

399/399 v0.38 wave tests passing (910 assertions). typecheck clean.
bun run verify gate green across all 14 shell checks.

DEFERRED TO FOLLOW-UP RELEASES (called out in CHANGELOG)
- Daemon rename autopilot -> ingest + forever-alias + plist migration
- cron-scheduler skill refactor + OpenClaw credential auto-migrate
- Content-type processors (PDF / OCR / audio / video)
- gbrain doctor inotify probe (Linux)
- Publisher DX cathedral: gbrain skillpack init --kind=ingestion-source,
  gbrain ingest test --watch, ingest tail, ingest validate
- Reference pack at examples/skillpack-ingestion-reference/ + 3-stage
  tutorial in docs/ingestion-source-skillpack.md

These are polish items; the substrate is shipped and queryable, and
skillpack publishers can build sources against the IngestionTestHarness
public export today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ingestion): test-gap fills — bootstrap probes + manifest entry + conformance

The v0.38.0.0 release-hygiene commit landed cleanly against the v0.38 wave
suite but tripped 3 categories of full-suite tests. This commit fixes
each. The remaining failure (doctorReportRemote > "healthy" status) was
verified pre-existing via `git stash + bun test` and is not caused by
v0.38; left alone.

Fix 1 — `schema-bootstrap-coverage.test.ts` (s1)
The test parses MIGRATIONS for ALTER TABLE ADD COLUMN statements and
fails if any column is not covered by `applyForwardReferenceBootstrap`
on both engines. Migration v80's four provenance columns triggered
the failure. Bootstrap probes added to both engines + 4 entries
appended to REQUIRED_BOOTSTRAP_COVERAGE:

- src/core/pglite-engine.ts — 4 EXISTS probes + state field + needs
  flag + ALTER TABLE block when bootstrap fires
- src/core/postgres-engine.ts — same pattern
- test/schema-bootstrap-coverage.test.ts — 4 coverage entries

Fix 2 — `check-resolvable.test.ts` (s3 — orphan_trigger)
RESOLVER.md references skills via name; check-resolvable cross-checks
against skills/manifest.json. The new `capture` skill was missing the
manifest entry; added between `brain-ops` and `idea-ingest` so the
manifest order mirrors the resolver order.

Fix 3 — `skills-conformance.test.ts` (s8)
Every SKILL.md must have `## Contract`, `## Output Format`, and
`## Anti-Patterns` sections. skills/capture/SKILL.md was missing all
three (initial draft skipped them); now compliant with concrete
content per the v0.38 contract.

Fix 4 — `build-llms.test.ts` (s6)
README + CHANGELOG edits in the release-hygiene commit caused
llms-full.txt to drift behind. Regenerated via `bun run build:llms`.
Per CLAUDE.md: any user-facing docs edit MUST run build:llms before
push.

The full bun-test parallel runner now passes everywhere except the
pre-existing `doctorReportRemote > healthy status` failure (50/100
score on an empty fresh brain — this is a pre-v0.38 health-score
tuning issue and orthogonal to ingestion work).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(version): bump 0.38.0.0 → 0.38.1.0

Renumbers the in-flight ingestion-cathedral release to v0.38.1.0.
Trio (VERSION, package.json, CHANGELOG.md) bumped together.

bun run typecheck → clean.

* chore(version): bump 0.38.1.0 → 0.38.0.0

Master sits at 0.37.11.0; 0.38.0.0 is the natural next slot rather than
skipping a release. Trio (VERSION, package.json, CHANGELOG.md) bumped
together. Migration v81 + ingestion substrate stay identical — this is a
header-only renumber.

bun run typecheck → clean.

* test(ingestion): fill v0.38 test gaps — markdown helpers + migration v81 + webhook E2E

Three gaps surfaced from a v0.38 audit against what shipped vs what was
covered. All three filled:

1. **test/markdown-serializer.test.ts** (NEW, 19 cases) — pure-function
   coverage of `serializePageToMarkdown` + `resolvePageFilePath`, the
   DRY extract that the dream-cycle reverse-render and put_page
   write-through both consume. Pre-fix nothing pinned the
   frontmatter-override merge precedence, the type/title defaults, or
   the source-aware filing layout (default → `<brainDir>/<slug>.md`,
   non-default → `<brainDir>/.sources/<source_id>/<slug>.md`). Future
   schema-shape changes to either helper now surface immediately.

2. **test/migrate.test.ts — v81 cases** (10 new cases, two describe
   blocks) — structural assertions on `pages_provenance_columns`
   (four nullable columns, no NOT NULL, no DEFAULT, no index — the
   ADD COLUMN stays metadata-only) plus a PGLite round-trip that
   asserts the columns appear post-`initSchema`, accept direct UPDATEs,
   and survive the historical-page NULL scenario. The
   schema-bootstrap-coverage test already pinned the forward-reference
   probe contract; this fills the migrate.test.ts contract gap.

3. **test/e2e/serve-http-ingest-webhook.test.ts** (NEW, 16 cases) — HTTP
   contract coverage for POST /ingest. The pre-existing
   ingestion-roundtrip E2E explicitly notes "e2e (gbrain serve --http +
   POST /ingest + real OAuth) is a separate" thing — it covers the
   in-process daemon → handler → DB pipeline, NOT the real HTTP route.
   This file fills that gap. Spawns real gbrain serve --http against
   real Postgres, mints OAuth tokens with various scopes, exercises:
     - Auth gate (missing → 401; read-only → 403)
     - Body validation (empty → 400 with error: empty_body)
     - Content-type allowlist (image/png → 415 with skillpack hint;
       application/pdf → 415; text/plain + application/json + text/html
       all accepted; unknown text/* falls through to text/plain)
     - X-Gbrain-Content-Type / Source-Id / Source-Uri / Slug header
       overrides
     - Idempotency (same content + same client = identical job_id via
       queue dedup on content_hash)

Also wires three new entries into `scripts/e2e-test-map.ts` so changes
to `src/commands/serve-http.ts`, `src/core/ingestion/**`, or the
`ingest-capture` Minion handler auto-trigger the relevant E2Es under
`bun run ci:local:diff`.

Verified locally:
- bun test test/markdown-serializer.test.ts → 19/19 green
- bun test test/migrate.test.ts -t "v81" → 10/10 green
- bun test test/e2e/serve-http-ingest-webhook.test.ts (real Postgres on
  ephemeral 5435) → 16/16 green
- bun test test/select-e2e.test.ts → 24/24 green (selector test still
  honors the v0.38 entries)
- bun run typecheck → clean

E2E DB lifecycle handled per CLAUDE.md (spin up pgvector:pg16 on a free
port, bootstrap via `gbrain doctor --json`, run, tear down).

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 08:00:42 -07:00

GBrain

Your AI agent is smart but forgetful. GBrain gives it a brain.

Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: 17,888 pages, 4,383 people, 723 companies, 21 cron jobs running autonomously, built in 12 days. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.

The brain wires itself. Every page write extracts entity references and creates typed links (attended, works_at, invested_in, founded, advises) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side: gbrain lands P@5 49.1%, R@5 97.9% on a 240-page Opus-generated rich-prose corpus, beating its graph-disabled variant by +31.4 points P@5 and ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling gbrain-evals repo.

New default in v0.36.2.0: ZeroEntropy for both embedding (zembed-1 at 1280d via Matryoshka) and reranker (zerank-2). On a real-corpus benchmark vs OpenAI and Voyage: 2.2× faster (442ms vs OpenAI 973ms), 2.6× cheaper at regular pricing ($0.05/M vs OpenAI $0.13), wins 11 of 20 queries head-to-head, reshuffles 60% of top-1 results when used as a second-pass reranker. Bring your own key from zeroentropy.dev, or switch to OpenAI/Voyage at install time via gbrain init --pglite --embedding-model <provider:model> --embedding-dimensions <N> — your choice is sticky. To switch an existing brain, run gbrain reinit-pglite --embedding-model <provider:model> --embedding-dimensions <N> (PGLite) or follow the SQL recipe in docs/embedding-migrations.md (Postgres). gbrain config set embedding_model is refused as of v0.37.11.0 because the schema column has to resize too.

GBrain is those patterns, generalized. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.

New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself. One command does the loop you used to run by hand: gbrain doctor --remediate --yes --target-score 90 --max-usd 5. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. gbrain doctor --remediation-plan --json previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (reindex, repair-jsonb, orphans, integrity, purge, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New --background flag on gbrain embed submits the job and exits with job_id=N for shell composition.

New in v0.35.7 — Temporal trajectory + founder scorecard. Author typed metric assertions in the ## Facts fence (mrr=50000, arr=2000000, team_size=12) and gbrain stores them as first-class typed columns. gbrain eval trajectory companies/acme-example prints the chronological history with regressions auto-flagged inline. gbrain founder scorecard companies/acme-example rolls up claim accuracy, consistency, growth direction, and red flags into a stable schema_version: 1 JSON contract. New MCP op find_trajectory exposes the same data to agents (read scope, visibility-filtered for remote callers). The consolidate cycle phase now writes valid_until on chronologically-superseded facts AND uses semantic upsert on (page_id, claim, since_date) — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions).

~30 minutes to a fully working brain. Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.

LLMs: fetch llms.txt for the documentation map, or llms-full.txt for the same map with core docs inlined in one fetch. Agents: start with AGENTS.md (or CLAUDE.md if you're Claude Code).

Install

GBrain runs in three shapes. Pick the one that matches how you use AI agents today.

Run with your agent platform

Already using OpenClaw or Hermes? GBrain installs as a skillpack scaffold into your agent's workspace.

gbrain init --pglite
gbrain skillpack scaffold --all   # or: scaffold <name> per skill

That's it. Your agent picks up 43 skills (signal detection, brain-ops, ingest, enrich, citation-fixer, daily-task-manager, cron-scheduler, eval framework, and 35 more). Routing lives in skills/RESOLVER.md — the agent reads it once per request, picks the right skill, executes. Scaffolded skills are first-class members of your agent repo — you own them, edit freely; gbrain skillpack reference <name> diffs your copy against gbrain's bundle when you want to pull upstream improvements. (The legacy gbrain skillpack install managed-block model was retired in v0.36.0.0; run gbrain skillpack migrate-fence once if you're upgrading from an older release.)

CLI standalone

Use gbrain from any shell, no agent platform required.

bun install -g github:garrytan/gbrain
gbrain init --pglite   # 2 seconds; no server, no Docker
gbrain doctor          # verify health

Then point any MCP-aware client (Claude Code, Cursor, Windsurf) at it, or use it from your shell:

gbrain search "who works at acme AI?"
gbrain query "what did bob invest in this quarter?"
gbrain graph-query people/garry-tan --depth 2

Detailed setup paths (Postgres at scale, Supabase, thin-client mode) live in docs/INSTALL.md.

MCP server (any MCP client)

gbrain serve              # stdio MCP (Claude Desktop / Code / Cursor)
gbrain serve --http       # HTTP MCP with OAuth 2.1 + admin dashboard
                          # at /admin, SSE activity feed at /admin/events

Per-client guides (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) live under docs/mcp/. HTTP server supports DCR-style client registration, scope-gated access (read/write/admin), and built-in rate limiting.

How to get data in (v0.38+)

One command, local or hosted, synchronous receipt:

gbrain capture "the thought I want to remember"
gbrain capture --file ./notes/today.md
echo "from a pipe" | gbrain capture --stdin
SLUG=$(gbrain capture "..." --quiet)

The page lands in the DB AND on disk in one move (the v0.38 put_page write-through plumbing). Default slug inbox/YYYY-MM-DD-<hash8> so captures cluster in a predictable triage location. On thin-client installs the verb routes through MCP to the server — same command, same UX.

For webhook ingestion (Zapier / IFTTT / Apple Shortcuts):

curl -X POST https://your-brain/ingest \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/markdown" \
  -d "# a thought from a Shortcut"

For mobile capture, the inbox folder source picks up anything dropped into ~/.gbrain/inbox/ from iOS Shortcuts / AirDrop / Drafts / Finder.

Third-party skillpacks can ship custom ingestion sources (Granola, Linear, voice, OCR) against the versioned IngestionSource contract at gbrain/ingestion. See docs/skillpack-anatomy.md.

What it does (the loop)

  signal   →   search   →   respond   →   write   →   auto-link   →   sync
  (every    (brain-first  (informed     (page +    (typed edges     (cron
  message)  retrieval)    by context)   timeline)  + backlinks)     keeps fresh)
  • Signal detector runs on every message your agent receives. Captures ideas, entity mentions, time-sensitive todos, names, links.
  • Brain-first lookup before any external API call. The cheapest, fastest, most personal information source you have.
  • Auto-link fires on every page write. No LLM calls; pure pattern matching on [[wiki/people/bob]] style references. New entity → new page stub → graph grows.
  • Cron-driven enrichment runs while you sleep: dedup people pages, fix citations, score salience, find contradictions, prep tomorrow's tasks.

The whole loop is described in docs/architecture/topologies.md with diagrams.

Capabilities

Hybrid search. Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (conservative, balanced, tokenmax) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in docs/eval/SEARCH_MODE_METHODOLOGY.md. Default: balanced with ZeroEntropy reranker on.

Self-wiring knowledge graph. Every put_page extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (attended, works_at, invested_in, founded, advises, mentions, …). Multi-hop traversal via gbrain graph-query. The graph is what produces the +31.4 P@5 lift over vector-only RAG.

Job queue (Minions). BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.

43 curated skills. Routing lives in skills/RESOLVER.md. Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.

Eval framework. gbrain eval longmemeval runs the public LongMemEval benchmark against your hybrid retrieval. gbrain eval export + gbrain eval replay capture real queries and replay them against code changes (set GBRAIN_CONTRIBUTOR_MODE=1). gbrain eval cross-modal cross-checks an output against the task using three different-provider frontier models. Full methodology in docs/eval/SEARCH_MODE_METHODOLOGY.md.

Brain consistency. gbrain eval suspected-contradictions samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.

Integrations

Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in recipes/ and is discoverable via gbrain integrations list.

Architecture

Two engines, one contract. PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first BrainEngine interface in src/core/engine.ts defines ~47 operations both engines implement; CLI and MCP server are generated from one source.

Brain repo is the system of record. Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in docs/architecture/topologies.md.

Two organizational axes (brain ⊥ source). A brain is a database (your personal brain, a team mount you joined). A source is a repo inside that brain (wiki, gstack, an essay, a knowledge base). Routing lives in .gbrain-source dotfiles and resolves via a documented 6-tier precedence chain. Full diagrams in docs/architecture/brains-and-sources.md.

Why the graph matters. Vector search returns chunks that are semantically close. The graph returns chunks that are factually connected. Hybrid search pulls from both; auto-linking on every write keeps the graph fresh. Deep dive: docs/architecture/RETRIEVAL.md.

Troubleshooting

gbrain import fails with expected N dimensions, not M? Run gbrain doctor. It will print the exact gbrain config set ... or gbrain retrieval-upgrade command to repair the mismatch. You should not need to delete ~/.gbrain. As of v0.37, fresh gbrain init --pglite auto-detects your embedding provider from API keys in your environment — set OPENAI_API_KEY (or ZEROENTROPY_API_KEY / VOYAGE_API_KEY) before running init, or pass --embedding-model <provider>:<model> explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass --no-embedding to defer setup until runtime. See docs/integrations/embedding-providers.md for the full provider matrix and docs/operations/headless-install.md for Docker/CI sequencing.

Docs

  • docs/INSTALL.md — every install path, end to end
  • docs/architecture/ — system design, topologies, retrieval theory
  • docs/guides/ — how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion)
  • docs/integrations/ — connecting external data sources (voice, email, calendar, embedding providers)
  • docs/mcp/ — per-client MCP setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)
  • docs/eval/ — eval framework, metric glossary, methodology
  • docs/ethos/ — philosophy (thin harness, fat skills, markdown as recipes, origin story)
  • AGENTS.md — entry point for non-Claude agents
  • CLAUDE.md — entry point for Claude Code (deep operating context)
  • CONTRIBUTING.md — contributor guide, test discipline, eval-capture mode
  • SECURITY.md — OAuth threat model, hardening defaults

Contributing

Run bun run test for the fast loop, bun run verify for the pre-push gate, bun run ci:local to run the full Docker-backed CI stack locally. Detailed test discipline in CONTRIBUTING.md.

Community PRs are batched into release waves rather than merged one-by-one — see the "PR wave workflow" section in CLAUDE.md. Contributor attribution stays attached via Co-Authored-By: trailers. We credit every accepted contribution in CHANGELOG.md.

If you find a bug or want a feature: open an issue first. Quick fixes (typo, doc bug, obvious regression) can go straight to a PR. Anything touching schema, retrieval ranking, MCP protocol, or the security boundary needs a design discussion in the issue first.

License + credit

MIT. Built by Garry Tan to run his OpenClaw and Hermes deployments — the production brain behind his actual AI agents.

Origin story: docs/ethos/ORIGIN.md.

Community PR contributors are credited in CHANGELOG.md per release. ZeroEntropy (@zeroentropy) for the embedding + reranker stack that became the v0.36.2.0 default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.

S
Description
No description provided
Readme MIT
160 MiB
Languages
TypeScript 97.4%
Shell 1.2%
JavaScript 0.9%
PLpgSQL 0.4%