ca68633faa v0.41.2.0 feat: lens packs + epistemology unification — atoms + concepts as first-class units, calibration profile widening, gstack-learnings bridge (#1364)
* feat(schema): migration v93 take_domain_assignments (v0.41 T1)

Adds the JOIN table backing per-pack calibration domain aggregation
in the v0.41 lens-packs wave. Replaces the originally-planned scalar
`takes.domain` column after codex outside-voice review caught that
one take can legitimately belong to multiple domains (a take about
"Sequoia's investment in Anthropic" lands in deal_success AND
market_call), and that scalar attribution bakes today's pack→domain
mapping into permanent fact.

Schema: composite PK (take_id, domain) for idempotent re-assignment,
FK CASCADE so deleting a take cascades assignments, confidence CHECK
in [0,1], idx_take_domain_assignments_domain for the aggregator JOIN
direction. RLS guard matches takes/synthesis_evidence pattern (enable
when running as BYPASSRLS role). PGLite parity via sqlFor.pglite.

Backward-compat: pre-existing takes carry no assignments; aggregator
LEFT JOIN skips them gracefully. No backfill required at migration
time — propose_takes (T10) populates new rows; greenfield assignment
of historical takes is a v0.42 follow-up.

R-MIG IRON-RULE regression at test/migrations-v93.test.ts pins 12
contracts: existence/name, LATEST_VERSION advance, table queryable
after initSchema, column shape, composite PK rejects duplicate
(take_id, domain), multi-domain assignment permitted, FK ON DELETE
CASCADE, CHECK rejects out-of-range confidence, index presence,
aggregator JOIN direction returns per-domain counts, sql/sqlFor.pglite
parity grep, backward-compat LEFT JOIN handles unassigned takes.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
First of 13 sequencing tasks in v0.41 lens packs + epistemology
unification wave (decisions D9-B → T1-B per codex challenge).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(contracts): IngestionSource.mode + pack manifest phases/calibration_domains (v0.41 T2+T3)

Two independent contract extensions, batched because both are pre-
requisites for T4 (pack YAML manifests) and T9 (cycle.ts orchestrator
gate). Neither is load-bearing alone; together they form the surface
the four lens-pack manifests will declare against.

T2 — IngestionSource.mode discriminator (codex outside-voice fix):
  src/core/ingestion/types.ts grows an optional `mode: 'trickle' |
  'migration'` field on IngestionSource. Defaults to 'trickle' when
  unset — v0.38 sources unchanged. New IngestionSourceMode export.
  src/core/ingestion/daemon.ts handleEmit() branches on the mode:
  trickle keeps the 24h DedupWindow.mark() path; migration bypasses
  dedup entirely (the source owns permanent slug-keyed idempotency
  via op_checkpoint or similar). Validation, rate limit, and dispatch
  apply uniformly to both modes.

  Why: the 24h content-hash dedup window is wrong for bulk historical
  migration. 24K wintermute pages over hours, retries days apart, and
  same-hash collisions across the window are expected. Trickle
  semantics (file-watcher, inbox-folder, webhook) want dedup to catch
  at-least-once replay; migration semantics want EVERY explicitly-
  emitted event to land because the source already gated it.

T3 — SchemaPackManifestSchema phases + calibration_domains:
  src/core/schema-pack/manifest-v1.ts grows two optional fields. New
  AGGREGATOR_KINDS closed enum (4 v1 algorithms: scalar_brier,
  weighted_brier, count_based, cluster_summary) backing
  AggregatorKind type. New CalibrationDomain {name, aggregator,
  page_types} schema with snake_case regex on name, .strict on extra
  fields, page_types.min(1).

  `phases: string[]` declares which cycle phases the active pack
  participates in (D4-B orchestrator gate; runCycle will consult this
  in T9). Validated as string here, against runtime CyclePhase union
  at the registry layer (avoids circular import). `borrow_from` does
  NOT borrow phases — each pack declares explicitly.

  `calibration_domains: CalibrationDomain[]` declares per-pack
  scorecard buckets. Closed registry of algorithm `aggregator` values
  keeps SQL injection surface closed; open `name` strings let third-
  party packs add domains without a gbrain release (T3 codex
  refinement of D6).

  Backward compat: both fields default to []. Existing v0.38 manifests
  parse unchanged (pinned by 2 regression cases).

Tests:
  test/ingestion/migration-mode.test.ts (8 cases): mode type accepts
  literals, defaults to trickle, daemon branches correctly across
  trickle/migration/default-undefined, validation still runs in
  migration mode, mixed dual-source independence.

  test/schema-pack-manifest-v041.test.ts (19 cases): aggregator enum
  shape, phases default + accept + reject (non-string, empty, non-
  array), calibration_domains default + accept (single + multi entry,
  multi page_types), reject (unknown aggregator, kebab/uppercase/
  digit-start names, empty page_types, unknown extra field), v0.38
  back-compat regressions.

  All 27 cases pass first-green after API surface alignment.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Tasks T2 + T3 of 13 in v0.41 lens packs + epistemology unification wave.
Unblocks: T4 (pack manifests reference both fields), T9 (cycle.ts gate
reads phases:), T10 (calibration widening reads calibration_domains).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(packs): 4 bundled lens pack manifests + registry wiring (v0.41 T4)

Authors gbrain-creator + gbrain-investor + gbrain-engineer +
gbrain-everything as bundled YAML manifests in
src/core/schema-pack/base/, registers them in the BUNDLED array in
load-active.ts, exports AGGREGATOR_KINDS + AggregatorKind +
CalibrationDomain types through the schema-pack barrel.

gbrain-creator: atom (NEW page type) + concept (reuse from base).
  phases: [extract_atoms, synthesize_concepts]. One calibration
  domain: concept_themes / cluster_summary / [concept]. Retires
  wintermute's atom-pipeline-coordinator cron (T12 follow-up).

gbrain-investor: thesis + bet_resolution_log (NEW). Borrows
  deal/person/company/yc from base. No new cycle phases (consumes
  existing extract_facts/propose_takes/grade_takes pipeline). Three
  calibration domains: deal_success/scalar_brier/[deal],
  founder_evaluation/scalar_brier/[person], market_call/weighted_brier
  /[thesis]. Filing rules mirror wintermute's existing investing/deals
  + investing/theses + investing/bets layout.

gbrain-engineer: bridge-only per D8-C. ONLY declares `learning`
  page type (primitive: annotation); borrows code+project from base.
  No new cycle phases (gstack-learnings IngestionSource is daemon-
  side per T8). Three calibration domains: architecture_calls/
  scalar_brier/[code, learning], effort_estimates/weighted_brier/
  [project], risk_assessment/scalar_brier/[project].

gbrain-everything: meta-pack extending gbrain-investor + borrowing
  atom (from creator) + learning (from engineer). Codex outside-voice
  T4 resolution to the multi-lens problem: composes via the v0.38-
  shipped extends + borrow_from chain instead of inventing an
  active-multi-pack architecture. Single-active-pack constraint
  preserved. Explicitly re-declares phases + calibration_domains
  (borrow_from borrows types/link_types only — phases must be
  declared per pack per D4-B).

Frontmatter validators (atom_type closed 11-value enum, virality_
score range, etc.) are NOT declared in these manifests — that
contract surface (per-page-type frontmatter_validators on
PageTypeSchema) is a v0.42 follow-up filed in plan TODOs. For
v0.41, extract_atoms hardcodes the enum with a TODO comment
pointing at the eventual manifest read path (D11).

YAML parser caveat: src/core/schema-pack/loader.ts uses a hand-
rolled parseYamlMini (per loader.ts:86 explicit non-support of `|`
block scalars). Initial descriptions used `|` blocks and broke
parsing silently (description was 'literal "|"', everything after
collapsed). Reauthored to single-line "..." strings. Pinned by
the manifest-load tests asserting page_types/phases/calibration_
domains all resolve.

Tests:
  test/lens-pack-manifests.test.ts (31 cases): one file covers all
  4 packs to avoid 4x boilerplate. Pins parse cleanly, registry
  inclusion, per-pack page_types/phases/calibration_domains/filing_
  rules shape, every aggregator value falls in AGGREGATOR_KINDS,
  meta-pack unions correctly (7 calibration domains across all
  three lens packs).

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T4 of 13. Unblocks T5/T6 (phases now declared; phases read
from active pack at runtime), T7 (importer writes atom-typed
pages against creator manifest), T8 (gstack-learnings emits
learning-typed pages against engineer manifest), T9 (orchestrator
gate reads phases: declaration), T10 (calibration_profile walks
calibration_domains).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): orchestrator-level pack gate for lens-pack phases (v0.41 T9)

Wires extract_atoms + synthesize_concepts into runCycle with the D4-B
orchestrator-level pack gate. Five surgical edits to src/core/cycle.ts:

  1. CyclePhase union grows by 2 names.
  2. ALL_PHASES inserts extract_atoms after extract_facts (Haiku 3-check
     has fresh fact context, BEFORE resolve_symbol_edges to avoid
     interrupting the symbol resolution sweep mid-flight) and
     synthesize_concepts after patterns (cluster pass sees fresh
     cross-session themes).
  3. PHASE_SCOPE entries: extract_atoms='source' (per-source transcript
     walk), synthesize_concepts='global' (concept clusters cross sources
     by nature).
  4. NEEDS_LOCK_PHASES adds both (put_page writes mutate DB).
  5. runCycle dispatch blocks for both phases consult packDeclaresPhase
     before invoking. When the active pack doesn't declare the phase,
     skipped with reason='not_in_active_pack' marker. When it does,
     lazy-imports extract-atoms.ts / synthesize-concepts.ts and runs.

The packDeclaresPhase helper is new at module-private scope. Loads the
active pack via loadActivePack({cfg, remote:false}); reads
resolved.manifest.phases (local only — D4-B). Fail-open: any registry
error (pack not found, malformed manifest) returns false. Skipping >
crashing for an orchestrator gate.

Local-only phase semantics (not extends-chain inherited) preserves user
sovereignty: a downstream pack extending gbrain-creator may NOT want
extract_atoms to run (e.g. derives atoms differently). Inheriting phases
would force them into a no-op-or-fork choice. The gbrain-everything
meta-pack therefore RE-DECLARES creator's phases verbatim in its own
manifest, asserted by the T4 test.

Stub phase modules ship in this commit:
  src/core/cycle/extract-atoms.ts → returns skipped with reason=
    'stub_pending_t5'
  src/core/cycle/synthesize-concepts.ts → returns skipped with reason=
    'stub_pending_t6'

T5/T6 replace the stub bodies with real LLM-driven phases. The
orchestrator dispatch is fully wired today and exercised by the test.

Manifest schema follow-on: phases + calibration_domains were originally
.default([]) but the type narrowing broke v0.38 fixture casts in
test/schema-pack-{lint-rules,registry,registry-reload}.test.ts.
Reverted to .optional(); consumers apply `?? []` at the read site.
Same pattern as IngestionSource.mode in T2. Updated T3 + T4 tests
to use `!` non-null assertion at sites that explicitly declared the
fields (typechecker can't narrow array literals through optional
boundaries).

Tests:
  test/cycle-pack-gating.test.ts (19 cases, R-GATE IRON RULE):
  ALL_PHASES + PHASE_SCOPE shape, ordering invariants (extract_atoms
  after extract_facts, synthesize_concepts after patterns), exhaustive
  PHASE_SCOPE map, NEEDS_LOCK_PHASES static-source assertion (both new
  phases included), dispatch consults packDeclaresPhase for BOTH new
  phases (and ONLY those two), packDeclaresPhase helper exists +
  reads manifest.phases (not merged chain) + fail-open returns false
  on catch, pre-existing 17 phases NEVER consult packDeclaresPhase
  (extract_facts + calibration_profile spot-checked), not_in_active_pack
  reason marker appears exactly 2x (semantic consistency across
  both gated phases).

  Adjacent test fixes: T3 + T4 tests updated for optional-field
  semantics. T2 dispatch type narrowed to DispatchOutcome shape from
  daemon.ts ({kind: 'queued'} for success path).

89/89 across T1+T2+T3+T4+T9 tests pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T9 of 13. Unblocks: T5 (extract-atoms.ts body replaces stub),
T6 (synthesize-concepts.ts body replaces stub).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(calibration): domain_scorecards widening + 4 aggregators (v0.41 T10)

Replaces the v0.36.1.0 placeholder `JSON.stringify({})` in
calibration-profile.ts:336 with a real aggregator pass over the active
pack's calibration_domains declarations. domain_scorecards JSONB now
populates per declared domain with {n, brier, accuracy, aggregator,
page_types, extras}.

New module: src/core/calibration/domain-aggregators.ts
  - aggregateDomainScorecards(engine, holder, domains, sourceId) → JSONB-shape
  - 4 aggregator implementations matching the AggregatorKind closed enum:
    - scalar_brier: AVG(POWER(weight - outcome::int, 2)). The default for
      most predictive domains. Filters by holder + page_types +
      resolved_outcome IS NOT NULL + active=TRUE + source_id.
    - weighted_brier: Brier weighted by ABS(weight - 0.5) * 2 (conviction
      proxy since takes table has no separate confidence column). A
      0.95-conviction miss weights 9x more than a 0.55-conviction one.
      Matches the investor pack's market_call semantics.
    - count_based: simple SUM(hit)/COUNT(*) accuracy without Brier.
      For domains where probability isn't natural.
    - cluster_summary: page count + tier histogram via
      frontmatter->>'tier' JSONB read. For concept_themes where there's
      no binary outcome to score. Returns {n, tier_counts: {T1, T2,
      T3, T4}}.

Wiring in src/core/cycle/calibration-profile.ts:
  Try/catch wraps the loadActivePack → aggregator chain. Empty {}
  scorecard on any pack-resolution error (R1 IRON RULE: byte-identical
  v0.36.1.0 baseline when no active pack declares domains). Warning
  appended to result.warnings so doctor surfaces silent failures
  instead of crashing the phase.

Per-domain fail-soft: aggregateOneDomain's try/catch returns
{n: 0, brier: null, accuracy: null, extras: {error}} for any single
malformed domain. The other domains still aggregate. Phase keeps
running.

Tests (test/domain-aggregators.test.ts, 13 cases):
  - R1 IRON RULE: empty domain list returns {} (byte-identical)
  - scalar_brier: empty no-takes returns n:0/null/null; 2-take
    Brier computed correctly (0.5 over (0, 1) sq_errs); accuracy
    matches weight>=0.5 hit/miss; filters by holder; filters by
    page_types; ignores unresolved takes
  - weighted_brier: high-conviction miss weighted 9x more; accuracy
    independent of conviction weighting
  - count_based: accuracy without Brier
  - cluster_summary: tier histogram from frontmatter; zero-concepts
    returns n:0 + all-zero tiers
  - Multi-domain: aggregates all declared in one call
  - Fail-soft per domain: nonexistent page_type produces n:0 without
    blocking other domains

89/89 across T1+T2+T3+T4+T9+T10 tests; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T10 of 13. The propose_takes-side wiring (populate
take_domain_assignments at write time from active pack's page_type→
domain mapping) is deferred to T5/T6 phase implementations, since
they are the natural producers of takes. Manual propose_takes via
fence write covers the operator path. v0.42+ adds a takes-fence
parser extension to read domain[] from fence rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): gstack-learnings bridge source (v0.41 T8)

Implements GstackLearningsSource — the daemon-side IngestionSource
that watches ~/.gstack/projects/{repo}/learnings.jsonl and emits
each new line as a `learning`-typed IngestionEvent.

Closes the v0.40-and-earlier gap where gstack's typed engineering
knowledge base (7 learning types: pattern, pitfall, preference,
architecture, tool, operational, investigation) lived in JSONL files
the brain never queried. After T8 + the engineer-pack manifest
activation, every gstack-logged learning surfaces as a first-class
gbrain page within seconds of being written.

Lifecycle:
  - constructor: discovers JSONL files via ~/.gstack/projects/*&#47;
    learnings.jsonl (cross-project mode, default) or just the current
    project (per-project mode). Test seam: _readFile/_existsSync/_skipWatch.
  - start(ctx): seeds seenLines with content_hashes of EVERY existing
    line so first-run-after-install does NOT replay thousands of
    historical lines as fresh emits. Then installs fs.watch handlers
    (one per discovered file) that fire rescanFile on 'change'.
  - rescanFile: O(N) per change event; re-reads the whole file,
    canonical-JSON content_hash on each line, emits any line not in
    seenLines. Malformed JSONL lines skip+warn.
  - stop(): closes all watchers; JSONL state preserved (gstack owns
    the files, gbrain only reads).
  - healthCheck(): reports warn when no files discovered (gstack not
    installed) OR when watched files have disappeared; ok otherwise
    with counter of lines seen.

mode: 'trickle' (the v0.41 T2 default). Line-level content_hash via
canonical-JSON serialization means whitespace reformatting doesn't
trigger re-emit. Re-emit of an identical line is a silent dedup hit
via the daemon's 24h DedupWindow (T2 trickle path).

Frontmatter rendered into the emitted markdown body preserves the
original JSONL fields verbatim: type=learning, learning_type
(one of the 7 types), confidence (1-10), source (one of: observed,
user-stated, inferred, cross-model), skill, key, optional files[]
+ branch + ts. Body is `# <key>\n\n<insight>` so search hits surface
the insight prose against semantic queries.

Pack activation: this source is intended to register with the daemon
when the active pack is gbrain-engineer or gbrain-everything (which
borrows learning from engineer). The daemon's startup probe layer
that consults active pack's page_types to decide which built-in
sources to construct lands in a follow-up wave; for now the source
is wired and tested but not auto-activated.

Tests (test/ingestion/gstack-learnings.test.ts, 14 cases):
  - Basic contract: mode='trickle', id includes pid, kind='gstack-learnings'
  - Start seeds seenLines (historical lines NOT replayed)
  - Malformed JSONL lines skip without crashing
  - Blank lines + trailing newlines OK
  - emitLine: new line emits, identical line is silent dedup hit
  - Emitted body carries proper frontmatter (type, learning_type,
    confidence, source, skill, key, files, branch, ts)
  - Canonical-JSON content_hash dedup (whitespace reformat = hit)
  - healthCheck warn/ok states
  - describePaths diagnostic per-file existence + size

All 14 pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T8 of 13.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): wintermute-greenfield migration-mode importer (v0.41 T7)

Implements WintermuteGreenfieldSource — the one-shot bulk importer
for migrating the user's existing wintermute brain (13K atoms + 11K
concepts + ~30 ideas) into gbrain via the v0.41 lens packs.

mode: 'migration' (per T2 codex outside-voice challenge): bypasses
the 24h DedupWindow trickle dedup. Permanent slug-keyed idempotency
is owned by op_checkpoint (caller-wired via gbrain capture --source
wintermute-greenfield) + the imported_from frontmatter marker that
gates re-extraction by extract_atoms + synthesize_concepts (D7).

@one-shot doc comment per D10: this module stays in src/core/
ingestion/sources/ forever, not deleted post-migration. Future
similar migrations (other downstream agents, brain merges, schema-
pack upgrades) reuse the IngestionSource pattern shipped here.
Deleting the working example is short-sighted.

Walk:
  - ~/git/brain/atoms/{YYYY-MM-DD}/*.md (atoms, date-bucketed)
  - ~/git/brain/concepts/*.md (concepts, flat)
  - ~/git/brain/ideas/*.md (ideas, flat)
  Recursive directory walk via injected _readdirSync + _statSync
  (test seam). Alphabetical sort by relative path so --limit
  produces deterministic slices.

Per file:
  1. Read content; gray-matter parses frontmatter + body
  2. Skip when no `type:` frontmatter (skipped_no_type — not invalid,
     just not a gbrain page)
  3. Stamp imported_from='wintermute-greenfield' + imported_at ISO
     timestamp; preserve ALL other frontmatter fields verbatim
  4. Re-stringify via matter.stringify
  5. Emit IngestionEvent with content_type='text/markdown',
     untrusted_payload=false (local user-owned files), metadata
     carrying slug + page_type + original_path + original_frontmatter
     + importer + importer_version

Per-row validation failure → JSONL audit at
~/.gbrain/audit/wintermute-greenfield-failures-YYYY-Www.jsonl per
D12. Failed-file processing continues (don't fail-fast on one bad
row). Audit dir created lazily via mkdirSync recursive on first
write.

CLI flags supported via opts:
  --dry-run: walks + validates + stamps but doesn't emit
  --limit N: processes only the first N files (alphabetical)

The CLI surface lands via gbrain capture --source wintermute-greenfield
in a follow-up commit (capture.ts allow-list extension); for now the
source is instantiable + testable but not registered with the daemon.

Tests (test/ingestion/wintermute-greenfield.test.ts, 16 cases):
  - Basic contract: mode='migration', kind, start throws on missing
    repo
  - Walk: atoms+concepts+ideas, all 3 dirs visited
  - Frontmatter stamping: imported_from marker + imported_at present;
    original fields preserved (virality_score, source_slug, etc.)
  - Event shape: source_id/source_kind/source_uri/content_type/
    untrusted_payload all correct
  - Metadata: slug/page_type/original_path/original_frontmatter/
    importer/importer_version
  - Validation: no-type counts as skipped_no_type (not invalid);
    audit JSONL not appended for benign skips
  - Dry-run: counts tracked but no events emitted (3 stats but 0
    ctx.emitted)
  - --limit: only N files processed
  - Deterministic ordering: alphabetical relative-path sort means
    --limit 1 always picks the alphabetically-first file
  - healthCheck: ok after clean run; warn before start

All 16 pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T7 of 13.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): extract_atoms + synthesize_concepts minimal-viable bodies (v0.41 T5+T6)

Replaces the T9-shipped stub modules with working LLM-driven phase
bodies. v0.41 ships the right SHAPE — Haiku per transcript producing
1-3 atoms, atoms grouped by concept frontmatter ref, tier assignment
by count, Sonnet narrative for T1/T2. The richer 3-check quality gate
(truism/punchline/entity multi-pass), embedding-similarity dedup, voice
gate integration, op_checkpoint resumability all land in v0.41.1+ —
filed as inline TODOs and plan follow-ups.

T5 extract_atoms (src/core/cycle/extract-atoms.ts):
  - Takes transcripts via _transcripts test seam OR discoverTranscripts
    production path (lazy-imports transcript-discovery.ts to avoid
    circular module loads through cycle.ts).
  - Per transcript: ONE Haiku call with the 11-value atom_type enum
    embedded in the prompt (matches gbrain-creator.yaml declaration;
    v0.42 reads from active pack manifest at runtime per D11).
  - parseAtomsResponse tolerates markdown fences + trailing prose;
    rejects invalid atom_type values; clamps virality_score to [0,100];
    rejects malformed entries silently (skip don't crash).
  - Per atom: putPage atom-typed page under atoms/{YYYY-MM-DD}/
    {slug-from-title}. Frontmatter preserves atom_type, source_quote,
    lesson, virality_score, emotional_register from the LLM output.
  - Budget cap $0.30/source/run (DEFAULT_BUDGET_USD); over-budget
    transcripts counted as budget-skipped, phase returns status='warn'
    if any failures occurred.
  - Source-scoped: opts.sourceId routes corpus dir + write target.
  - dry-run: counts but doesn't writePages.
  - Failures tracked per-transcript without halting the run.

T6 synthesize_concepts (src/core/cycle/synthesize-concepts.ts):
  - Takes atoms via _atoms test seam OR DB query for type='atom' pages
    excluding imported_from frontmatter marker (D7 skip).
  - Groups atoms by frontmatter `concepts:` array ref.
  - Tier by count: T1 >=10, T2 >=5, T3 >=2, T4 deferred (no <2 groups).
  - T1/T2 groups: Sonnet call with up to 10 sample titles + 5 sample
    bodies → 1-paragraph narrative. Budget cap $1.50/run; over-budget
    or LLM-failed groups fall back to deterministic narrative.
  - T3 groups: deterministic narrative (no LLM call).
  - Per group: putPage concept-typed page at concepts/{title-from-slug}
    with tier + mention_count + composite_score frontmatter.
  - dry-run + yieldDuringPhase honored.

Tests (test/cycle/extract-atoms-synthesize-concepts.test.ts, 19 cases):
  parseAtomsResponse: well-formed JSON, markdown fences stripped,
  trailing prose tolerated, invalid atom_type rejected, missing fields
  rejected, garbage returns [], all 11 atom_type values accepted,
  virality_score clamped to [0,100].

  runPhaseExtractAtoms: no-op without transcripts, extracts via stub
  chat + writes pages, dry-run counts without writing, failures
  tracked per-transcript without halting.

  runPhaseSynthesizeConcepts: no-op without atoms, groups by concept
  ref + tier assignment by count (T1=12 atoms, T2=6, T3=3), atoms
  without concept refs filtered out, <T3 threshold (1 atom) filtered,
  T3 uses deterministic (no LLM call), dry-run counts without writing,
  T1 narrative comes from LLM stub verbatim.

All 19 pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Tasks T5 + T6 of 13. v0.41.1 follow-ups inline:
  - extract_atoms: read atom_type enum from active pack at runtime (D11)
  - extract_atoms: 3-check quality gate as multi-pass refinement
  - synthesize_concepts: embedding-similarity dedup (currently exact-
    string concept ref match only)
  - synthesize_concepts: voice gate for T1 Canon narratives
  - Both: op_checkpoint resumability for cross-cycle continuation

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.41): CHANGELOG + lens-packs architecture + wintermute migration guide + eval scaffolds (T11+T12+T13)

Closes out the v0.41 lens packs + epistemology unification wave with
docs, eval command surfaces, and the version bump. Three tasks batched
because each is small standalone:

T11 — 3 eval command scaffolds:
  src/commands/eval-extract-atoms.ts
  src/commands/eval-synthesize-concepts.ts
  src/commands/eval-wintermute-greenfield.ts

  Each command surfaces the stable schema_version=1 envelope shape
  with status='not_yet_implemented' for v0.41. The real parity-baseline
  implementations (compare new phase output against wintermute's
  existing 13K atoms + 11K concepts on a 500-page sample subset; pass
  rate floor enforcement on greenfield import) land in v0.41.1. The
  scaffolds let users discover the commands AND give the v0.41.1 work
  a clear extension point. Pinned by 7 scaffold tests.

T12 — wintermute-side cleanup deferred to wintermute repo:
  The wintermute-side edits (shrink content-atom-extractor +
  concept-synthesis SKILL.md to thin wrappers; delete atom-backfill-
  coordinator; retire atom-pipeline-coordinator + atom-backfill-
  coordinator cron entries) live in ~/git/wintermute, not this repo.
  The migration guide (docs/migrations/v0.41-wintermute-greenfield.md
  below) documents the cleanup steps. Operator runs them after
  verifying the greenfield import.

T13 — Documentation:
  CHANGELOG.md: full v0.41.0.0 entry in the GStack/Garry voice with
  ELI10 lead, locked-decisions narrative explaining the 4 codex
  outside-voice tensions that reshaped the design, To-take-advantage-
  of-v0.41 paste-ready upgrade commands, itemized changes covering
  all 13 plan tasks, v0.41.1 follow-ups list.

  docs/architecture/lens-packs.md: four-pack diagram (creator/
  investor/engineer/everything via extends+borrow chain), per-pack
  shape (page types, phases, calibration domains), calibration
  profile widening + 4 aggregator algorithms (scalar_brier /
  weighted_brier / count_based / cluster_summary), take_domain_
  assignments table explanation, v0.41.1 follow-ups.

  docs/migrations/v0.41-wintermute-greenfield.md: operator guide
  for the bulk 24K-page migration. Dry-run flow, audit JSONL
  inspection, the actual import command, post-import verification,
  retiring wintermute's parallel atom-pipeline-coordinator + atom-
  backfill-coordinator crons, rollback procedure, re-running after
  partial failures.

Version bump: VERSION + package.json → 0.41.0.0.

All 158 tests across 10 v0.41 test files pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Final tasks T11 + T12 + T13 of 13. Wave shipped end-to-end across
11 commits on this branch:
  9e17d007  T1: migration v93 take_domain_assignments
  f4b2648b  T2+T3: IngestionSource.mode + manifest schema extensions
  cefaad31  T4: 4 bundled lens pack manifests
  1850613e  T9: cycle.ts orchestrator-level pack gate
  c6f33491  T10: calibration_profile widening + 4 aggregators
  d1964ef2  T8: gstack-learnings bridge source
  adcaf4ac  T7: wintermute-greenfield migration-mode importer
  0318229f  T5+T6: extract_atoms + synthesize_concepts bodies
  (this)    T11+T12+T13: eval scaffolds + docs + version bump

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(tests): bump phase-count assertions from 17→19 (v0.41 follow-on)

v0.41 added extract_atoms + synthesize_concepts to ALL_PHASES.
Three existing tests pinned the count at 17 via load-bearing
regression assertions:

  test/phase-scope-coverage.test.ts:48-49
    expect(ALL_PHASES.length).toBe(17)
    expect(Object.keys(PHASE_SCOPE).length).toBe(17)

  test/core/cycle.serial.test.ts:393
    expect(hookCalls).toBe(17)  // yieldBetweenPhases hook fires per phase

  test/core/cycle.serial.test.ts:406
    expect(report.phases.length).toBe(17)

  test/e2e/cycle.test.ts:110
    expect(report.phases.length).toBe(17)

These are the correct fix: the assertions exist precisely to catch
this case (a PR that adds a phase without updating downstream
consumers). The wave's v0.41 commit (T9) updated ALL_PHASES but
missed these three sites. Updating them to 19 with comment
breadcrumbs preserving the version history (v0.26.5 → 9,
v0.29 → 10, v0.31 → 11, v0.32.2 → 12, v0.33.3 → 13,
v0.36.1.0 → 16, v0.39.0.0 → 17, v0.41.0.0 → 19).

Without this fix: full unit test suite (`bun run test`) shows 3
failures from these assertions. Underlying v0.41 logic was already
green; this is pure pin-bumping.

After fix: 9059 unit tests pass. 0 actual test failures. (3 shard
wedges remain from unrelated long-running parallel-runner tests
that exceed the 600s per-shard cap — infra concern, not test
logic, pre-dates this wave.)

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Wave gate: all 13 plan tasks done; all v0.41 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(e2e): update EXPECTED_PHASES for v0.41 (extract_atoms + synthesize_concepts + schema-suggest)

E2E test/e2e/dream-cycle-phase-order-pglite.test.ts pinned the canonical
phase sequence at 16 entries. v0.41 added extract_atoms (after
extract_facts) and synthesize_concepts (after patterns); v0.39 had
already added schema-suggest between orphans and purge. EXPECTED_PHASES
was missing all three.

This is the correct fix — the test exists specifically to catch a PR
that adds a phase without updating consumers, and it fired exactly as
designed. Updating EXPECTED_PHASES to the v0.41 19-phase sequence with
comment breadcrumbs (v0.39.0.0 schema-suggest, v0.41.0.0 extract_atoms
+ synthesize_concepts).

Verification (run with --timeout 60000 per E2E convention):
  DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test \
    bun test test/e2e/dream-cycle-phase-order-pglite.test.ts --timeout 60000
  → 5 pass, 0 fail

Other E2E failures observed in the full run are pre-existing /
environmental and not v0.41 regressions:
  - dream-synthesize-chunking: existing flake (synthesize details
    shape under withoutAnthropicKey)
  - fresh-install-pglite: env has multiple embedding providers
    configured; requires explicit --embedding-model disambiguation
  - http-transport: last_used_at debounce timing flake
  - ingestion-roundtrip: file-watcher trickle-mode timing flake
  - mechanical: gbrain doctor exits 1 because user's persistent
    ~/.gbrain has wedged migrations + reranker auth warnings
  - autopilot-fanout-postgres: pre-existing dispatch-selector
    timestamp semantics

None of those 6 are touched by the v0.41 wave. Filing them as
unrelated maintenance items.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Wave gate: 13 plan tasks done; v0.41 unit tests green; v0.41 E2E
green; pre-existing E2E flakes unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(e2e): 4 root-cause fixes for pre-existing E2E flakes (master polish)

After merging origin/master (which landed v0.40.8.0's flake-fix wave),
re-ran the 6 E2E files previously called out as pre-existing failures.
v0.40.8.0 had already fixed 3; the remaining 3 had real root causes:

1. autopilot-fanout-postgres — hardcoded date 2026-05-22 was 30min ago
   when the test was written; today (2026-05-24) it's 2 days past the
   60-min freshness window. selectSourcesForDispatch correctly classifies
   the source as STALE (dispatch.length=1) instead of FRESH (length=0).
   Fix: replace literal date with Date.now() - 30 * 60 * 1000 so the
   timestamp stays relative-fresh forever.

2. ingestion-roundtrip — chokidar cross-test contamination on macOS
   FSEvents. Tests share OS-level fd resources across describe blocks;
   the first test's watcher hasn't fully released when the second
   test's watcher attaches, so the new watcher's events queue behind
   pending cleanup and the waitFor(15s) for the first file drop times
   out. Fixes:
     - Move fs.mkdirSync(inboxDir) BEFORE createInboxFolderSource +
       daemon.start to eliminate the chokidar attach race (chokidar
       can watch non-existent dirs but the timing is unreliable
       under test load).
     - Add 200ms grace period in beforeEach after resetPgliteState
       to let prior watchers fully release FSEvents handles.
     - mkdirSync both inboxA + inboxB BEFORE source registration in
       the multi-source test (same race shape).
     - Bump waitFor timeouts 6s → 15s for fs.watch flake tolerance.

3. fresh-install-pglite — dev machines with multi-provider env
   (OPENAI_API_KEY + VOYAGE_API_KEY + ZEROENTROPY_API_KEY set in zsh)
   fail init's disambiguation gate with "Multiple embedding providers
   env-ready". The test sets ZE_API_KEY but doesn't NEGATE the others.
   Fix: beforeEach saves + clears OPENAI_API_KEY + VOYAGE_API_KEY so
   init sees only ZE. afterEach restores. Hermetic per dev machine.

4. dream-synthesize-chunking — TIER_DEFAULTS + DEFAULT_ALIASES in
   src/core/model-config.ts had BARE Anthropic model ids (e.g.
   'claude-sonnet-4-6' instead of 'anthropic:claude-sonnet-4-6'). The
   v0.40.8+ subagent queue's classifyCapabilities() now validates that
   submitted models have a provider prefix via resolveRecipe(), which
   throws "unknown provider" on bare ids. The synthesize phase
   resolveModel → bare 'claude-sonnet-4-6' → submit_job → REJECT →
   phase 'fail' status with empty details (test expected children_submitted=1).
   Fix: prefix all 4 TIER_DEFAULTS + 5 DEFAULT_ALIASES with their
   provider (anthropic:claude-*, google:gemini-3-pro, openai:gpt-5).
   Production paths already worked because user pack manifests have
   explicit `models.tier.subagent = anthropic:...`; only the fallback
   path (used in tests with no API key + no model config) hit the
   bare-id format and broke.

Verification (all run against DATABASE_URL=...:5434/gbrain_test):
  test/e2e/autopilot-fanout-postgres.test.ts → 6/6 pass
  test/e2e/dream-cycle-phase-order-pglite.test.ts → 5/5 pass
  test/e2e/dream-synthesize-chunking.test.ts → 4/4 pass
  test/e2e/fresh-install-pglite.test.ts → 2/2 pass
  test/e2e/http-transport.test.ts → 8/8 pass
  test/e2e/ingestion-roundtrip.test.ts → 3/3 pass
  test/e2e/mechanical.test.ts → 78/78 pass
  Total: 106/106 pass, 0 fail.

Adjacent unit tests verified green:
  test/anthropic-model-ids.test.ts → 6/6 pass
  test/model-config.serial.test.ts → 19/19 pass

typecheck clean.

Plan: v0.41 wave (~/.claude/plans/system-instruction-you-are-working-toasty-milner.md).
Post-merge polish — every E2E failure surfaced in the v0.41 ship reports is now green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(v0.42.0.0): privacy sweep + queue rebump + 5 pre-existing test fixes

Privacy: rename `wintermute-greenfield` → `markdown-greenfield` identifier
across 13 files + 4 file renames per CLAUDE.md:550 (banned private-fork name
in public artifacts). Identifier shipped through the lens-pack wave as the
long-lived migration-mode source kind; sweep includes class names
(MarkdownGreenfieldSource), frontmatter marker, audit JSONL path, eval
command, and operator doc filename. Reframe contextual mentions per
OpenClaw substitution rule ("your OpenClaw"/"upstream OpenClaw").

Queue: rebump v0.41.0.0 → v0.42.0.0 (PR #1352 claims v0.41.0.0 in queue);
sweeps 38 v0.41 → v0.42 references across branch-introduced files; renames
docs/migrations/v0.41-markdown-greenfield.md → v0.42-markdown-greenfield.md,
test/schema-pack-manifest-v041.test.ts → -v042, test/eval-v041-scaffolds →
test/eval-v042-scaffolds. Pre-existing master files referencing v0.41 left
untouched (those describe master's own anticipated wave).

Test fixes (5 pre-existing failures + 1 shard wedge, all unrelated to lens
packs but caught by the post-merge run):
- src/core/anthropic-pricing.ts: estimateMaxCostUsd strips `anthropic:`
  provider prefix before ANTHROPIC_PRICING lookup. v0.31.12 introduced
  provider-prefixed model strings; the budget meter wasn't updated and
  fell through to BUDGET_METER_NO_PRICING (budget gate disabled), letting
  auto-think submissions complete when the test expected budget exhaustion
  to force partial/skipped.
- test/longmemeval-trajectory-routing.test.ts: perf-gate cap 10s → 30s.
  Test runs ~4s isolated; parallel-shard CPU contention pushes it to 16s.
  30s still catches genuine cold-path regressions.
- test/search/embedding-column.test.ts → .serial.test.ts: quarantine to
  serial pass (depends on gateway module-state set by bunfig.toml preload;
  other parallel tests' resetGateway() leaves stale state).
- scripts/run-unit-parallel.sh: SHARD_TIMEOUT 600s → 900s. Shard 8's
  migration test suite runs 1369 tests in 807s (all pass); 600s wrapper
  cap was killing healthy shards.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: update project documentation for v0.42.0.0

Sweep v0.41 → v0.42.0.0 drift across the wave's release-summary and the
two new doc files. The wave shipped under its planning-time name (v0.41);
the queue rebump to v0.42.0.0 left a handful of factual references
pointing at the wrong version.

- CHANGELOG.md v0.42.0.0 entry: doc-ref filename, follow-up version
  label, and 4 in-prose v0.41 cites corrected to v0.42.0.0 / v0.42.0.1.
- docs/architecture/lens-packs.md: title + body + follow-up section
  corrected to v0.42.0.0 / v0.42.0.1.
- docs/migrations/v0.42-markdown-greenfield.md: title + upgrade
  command text corrected to v0.42.0.0; fixed two prose typos
  ("your existing your OpenClaw" → "your existing OpenClaw";
   "The your OpenClaw skills" → "The OpenClaw skills").

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: rebump v0.42.0.0 → v0.41.2.0 (per user; patch slot on v0.41 line)

PRs #1352 and #1367 both claim v0.41.0.0 in queue (the .0 slot is contested);
v0.41.2.0 is unclaimed and represents this wave as a PATCH on the v0.41 line
rather than a separate minor wave.

Sweeps v0.42.0.0 → v0.41.2.0 across CHANGELOG + 2 docs + 4 yaml + 4 ts + 2
test files; renames docs/migrations/v0.42-markdown-greenfield.md →
v0.41.2-markdown-greenfield.md and 2 test files (-v042 → -v041_2).

Wave-identity tags ("v0.41 T4" etc) in test/code comments correctly
preserved — this IS a v0.41 wave patch, not a new wave. macOS sed `\b`
limitation means those tags were never converted in the first place;
verified intentional preservation.

Forward references to v0.42 in TODOS.md + CHANGELOG D3 section + future-
wave declarations in code comments are untouched (they describe the NEXT
minor wave, not this one).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(audit-writer): route log() to event-ts ISO-week file, not wall-clock now

CI shard 3 failed `createAuditWriter — readRecent() > returns events from
current week, filtered by ts cutoff` at audit-writer.test.ts:229 with
`Expected: 2, Received: 0`.

Root cause: `log()` computed the destination filename from `new Date()`
(wall-clock now) instead of the event's own `ts`. Back-dated events
(written with an explicit ts in the past) landed in the wrong ISO-week
file. `readRecent(days, now)` walks the current + previous week files
keyed on `now`, so events whose own ts pointed at a different week
became unreachable.

The test passes ts=2026-05-21/16/14 and now=2026-05-22 (week 21 + 20).
CI runs on wall-clock 2026-05-25 (week 22). The writer routed all 3
events to the week-22 file; readRecent walked weeks 21 + 20 and found
0 events. Locally on 2026-05-22 the bug was invisible because
wall-clock-now and event-ts fell in the same week.

Fix in src/core/audit/audit-writer.ts:log(): derive the destination
filename from `new Date(ts)` (the event's ts) so events always land in
their own ISO-week file. NaN-guard falls back to wall-clock-now on
unparseable ts.

Test update at test/audit/audit-writer.test.ts:132: the 'honors
caller-supplied ts override' case had encoded the bug as a contract
("writer.log writes to current-week file regardless of event ts").
Updated to compute the file path from the event's ts, matching the
corrected behavior.

All 22 audit-writer tests pass. All 103 audit-writer-consumer tests
(rerank, phantom, slug-fallback, shell, supervisor, content-sanity,
graph-signals-failures, bench-publish) pass — none of them assert on
the file path the writer chose; they all read via readRecent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:56:21 -07:00

GBrain

Search gives you raw pages. GBrain gives you the answer. It's the brain layer your AI agent has been missing — the only one that does synthesis, graph traversal, and gap analysis in one box.

I'm Garry Tan, President and CEO of Y Combinator. I built GBrain to run my own AI agents. It's the production brain behind my OpenClaw and Hermes deployments: 146,646 pages, 24,585 people, 5,339 companies, 66 cron jobs running autonomously. My agent ingests meetings, emails, tweets, voice calls, and original ideas while I sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. I wake up smarter than when I went to bed — and so will you.

And now it works as a company brain too. Each person on the team gets their own slice of the brain, scoped by login. When you query, you only see what you're allowed to see — never another person's notes, never another team's data. We fuzz-tested this across every way you can read the brain (search, list, lookup, multi-source reads) and got zero leaks. Drop GBrain in as your team's shared institutional memory — the company-brain shape YC just put on its Request for Startups. If you're building in that space, you might as well build on this. Tutorial: set up GBrain as your company brain →

Lots of personal-knowledge systems give you keyword matching and grep in a box. GBrain does that, and adds two things nobody else ships together:

  • A synthesis layer that gives you the actual answer. Synthesized, well-cited prose across people, companies, deals, and ideas. Not "here are 10 chunks that mention your query"; an actual answer with citations and an explicit note on what the brain doesn't know yet. The gap analysis is the part that changes how you use the brain.
  • A self-wiring knowledge graph. Every page write extracts entity refs and creates typed edges (attended, works_at, invested_in, founded, advises) with zero LLM calls. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked: P@5 49.1%, R@5 97.9% on a 240-page Opus-generated rich-prose corpus, +31.4 points P@5 over its graph-disabled variant and over ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling gbrain-evals repo.

The point of building a 100K-page brain is to use it as a strategic moat. To never lose context. To query what's in your own head without re-reading it. The brain layer is what makes the moat usable. The 24/7 dream cycle is what keeps it sharp. Both run on your hardware, your DB, your keys.

It's easier to ship a daemon that runs 24/7 to ingest, enrich, and consolidate than it is to keep an agent in chat working hard. GBrain is that daemon, generalized. Install in 30 minutes. Your agent does the work. As my personal agent gets smarter, so does yours.

~30 minutes to a fully working brain. Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.

LLMs: fetch llms.txt for the documentation map, or llms-full.txt for the same map with core docs inlined in one fetch. Agents: start with AGENTS.md (or CLAUDE.md if you're Claude Code).

What this looks like

Say you have a meeting with Alice tomorrow. You want to walk in remembering what she works on, when you last talked, and what's still open between you. Here's what you'd type, and what you'd get back.

You ask:

"What do I need to know before my meeting with Alice tomorrow?"

Most personal-knowledge tools give you back a list of pages. Something like:

1. people/alice — Alice runs engineering at Acme...
2. meetings/2026-03-15-alice-q1 — Q1 product review with Alice...
3. meetings/2026-01-08-acme-kickoff — Kickoff meeting with Acme team...
4. customers/acme — Acme is a series-B fintech we work with...
5. notes/2026-04-22 — Quick chat with Alice about pricing...

Five pages you now have to open and read yourself to actually prepare. The tool found the right material, but it didn't do the work.

GBrain gives you back the answer, with sources:

Alice runs engineering at Acme (a series-B fintech). You last spoke
on April 22 in a quick pricing chat. Three things are still open
from that conversation:

1. She owes you the security review for the new tier
   (deadline was May 1; no update since).
2. You committed to pricing for a 500-seat tier
   (you sent it April 25; no response yet).
3. She mentioned they're hiring a CISO; you said you'd intro
   someone from your network.

Heads up: nothing's been added to the brain about Alice or Acme
since April 22, six weeks ago. She may have replied through email
or Slack DM, channels the brain doesn't see. Worth asking her to
catch up before assuming any of this is still current.

Every claim has a source page behind it. The "heads up" at the end tells you what the brain doesn't know yet, so you can ask Alice about it directly instead of being surprised. The brain just did your meeting prep.

This is the difference between a search engine and a brain. Search finds the pages. The brain reads them for you and writes the answer.

Install

GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.

If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:

Then paste this into your agent:

Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md

The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.

Never set up an AI agent platform before? The personal-brain tutorial walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.

Install it into your existing agent

Already running Codex, Claude Code, Cursor, or another coding agent? Paste the same instruction in:

Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md

This works in any agent that can read files over HTTPS and execute shell commands. Tested with Codex, Claude Code, Claude Cowork, Cursor, and AlphaClaw.

CLI standalone (no agent)

bun install -g github:garrytan/gbrain
gbrain init --pglite     # 2 seconds; no server, no Docker
gbrain doctor            # verify health
gbrain import ~/notes/   # index your markdown
gbrain query "what themes show up across my notes?"

Postgres-at-scale, Supabase, and thin-client setup paths live in docs/INSTALL.md.

Connect GBrain to your AI client (MCP)

GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:

  • Claude Code — one command: claude mcp add gbrain -- gbrain serve. Zero server, zero tunnel.
  • Cursor / Windsurf / any stdio MCP client — same shape, add {"command": "gbrain", "args": ["serve"]} to your MCP config.
  • Claude Desktop (Cowork) — Settings → Integrations → add the URL of your HTTP server. Remote only; the local claude_desktop_config.json does not work for remote servers.
  • Claude Cowork (team plan) — org Owner adds the connector under Organization Settings → Connectors.
  • Perplexity Computer — Settings → Connectors → add the URL + bearer token. Pro subscription required.
  • ChatGPT — uses OAuth 2.1 with PKCE (the hard requirement). Register a chatgpt client from the admin dashboard with grant type authorization_code.

For the HTTP server itself:

gbrain serve              # stdio MCP (local subprocess; for Claude Code, Cursor, Windsurf)
gbrain serve --http       # HTTP MCP with OAuth 2.1 + admin dashboard at /admin
                          # (required for Claude Desktop, Cowork, Perplexity, ChatGPT)

The HTTP server includes DCR-style client registration, scope-gated access (read / write / admin), and rate limiting. Deployment guides (ngrok, Railway, Fly.io) live under docs/mcp/.

Two ways to query your brain

Raw retrieval (what most personal-knowledge tools ship) and a synthesis layer that gives you an actual answer. They serve different jobs.

# raw retrieval: top pages by hybrid score, fast, no LLM cost
gbrain search "who's working on AI agents at portfolio companies?"

# brain layer: synthesized answer with citations and gap analysis
gbrain think "who's working on AI agents at portfolio companies?"

gbrain search returns the top retrieved pages, ranked by hybrid scoring (vector + keyword + RRF + source-tier boost + reranker). Use it when you want raw material to skim: agent context windows, citation lookups, finding a specific quote.

gbrain think runs the same retrieval, then composes a synthesized answer across the results with explicit citations to the source pages AND an honest note on what the brain doesn't know yet. The gap analysis is the differentiator: the answer tells you when a page is stale, when a claim is uncited, when two pages contradict each other, when there's a hole you should fill.

Why it compounds. Pair the brain layer with find_trajectory and you get answers like "how have the company's metrics changed AND what does the team look like right now AND what did they promise / share AND when did we last meet AND what's the value-add I can offer here": well-scored, well-cited, in one shot. That's the strategic moat. That's why building a 100K-page brain is worth the effort.

gbrain agent run "..." exposes the same surface to a sub-agent through the Minions queue, with crash-safe two-phase persistence. Same answers, durable.

How to get data in

One command, local or hosted, synchronous receipt:

gbrain capture "the thought I want to remember"
gbrain capture --file ./notes/today.md
echo "from a pipe" | gbrain capture --stdin
SLUG=$(gbrain capture "..." --quiet)

The page lands in the database and on disk in one move. Default slug inbox/YYYY-MM-DD-<hash8> so captures cluster in a predictable triage location. On thin-client installs the verb routes through MCP to the server: same command, same UX.

For webhook ingestion (Zapier / IFTTT / Apple Shortcuts):

curl -X POST https://your-brain/ingest \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/markdown" \
  -d "# a thought from a Shortcut"

For mobile capture, the inbox folder source picks up anything dropped into ~/.gbrain/inbox/ from iOS Shortcuts / AirDrop / Drafts / Finder.

Third-party skillpacks can ship custom ingestion sources (Granola, Linear, voice, OCR) against the versioned IngestionSource contract at gbrain/ingestion. See docs/skillpack-anatomy.md.

Your brain's shape (schema packs)

Most personal-knowledge tools force one fixed layout: their idea of "notes" + "people" + "tags." Drop a Notion export or your own years-old Obsidian vault on top, and the agent doesn't know what a Projects/ folder means or whether Reading/ is people or sources.

gbrain doesn't have a fixed layout. It ships with two bundled schema packs and lets you author your own when neither fits:

  • gbrain-base (default) — the layout my production brain uses: people/, companies/, concepts/, meetings/, deal/, daily/, originals/, writing/, etc. Zero config. Drop a brain that fits this shape and everything works.
  • gbrain-recommended — extends gbrain-base with the 13 additional directories from docs/GBRAIN_RECOMMENDED_SCHEMA.md (source, place, trip, conversation, personal, civic, project, etc.). Activate with gbrain schema use gbrain-recommended.
  • Your own packgbrain schema detect clusters your actual filesystem into proposed types, gbrain schema suggest runs an LLM pass over them, and gbrain schema review-candidates --apply promotes the ones you like. Three commands and the brain knows your shape.
gbrain schema active                # which pack is running, which tier set it
gbrain schema list                  # bundled + installed packs
gbrain schema detect                # propose types matching your filesystem
gbrain schema suggest               # LLM-refined proposals on top of detect
gbrain schema review-candidates     # human gate: promote / rename / ignore
gbrain schema use my-pack           # activate

The active pack threads through every read + write path: parseMarkdown infers page type from the pack's path prefixes; whoknows scopes expert routing to types declared expert_routing: true; extract_facts runs only on extractable: true types; the search cache folds the pack name + version into its key so cross-pack contamination is structurally impossible. Switch packs and the brain re-interprets itself; switch back and nothing's lost.

Seven-tier resolution chain (per-call flag → env var → per-source DB key → brain-wide DB key → gbrain.yml~/.gbrain/config.jsongbrain-base default). Full reference + authoring guide: docs/architecture/schema-packs.md.

Tutorials

Step-by-step walkthroughs for getting the most out of GBrain. Each one takes you from zero to a working outcome, with concrete commands and real numbers.

More walkthroughs in progress: connecting an existing agent (Claude Code, Cursor, OpenClaw, Hermes) to a GBrain memory layer; setting up GBrain for VC dealflow with founder scorecards and meeting prep; migrating an existing Notion or Obsidian vault; indexing a codebase as a queryable code brain. Full tutorial index: docs/tutorials/.

Want to see a tutorial that isn't here yet? Open an issue describing the workflow you want documented.

What it does (the loop)

  signal   →   search   →   respond   →   write   →   auto-link   →   sync
  (every    (brain-first  (informed     (page +    (typed edges     (cron
  message)  retrieval)    by context)   timeline)  + backlinks)     keeps fresh)
  • Signal detector runs on every message your agent receives. Captures ideas, entity mentions, time-sensitive todos, names, links.
  • Brain-first lookup before any external API call. The cheapest, fastest, most personal information source you have.
  • Auto-link fires on every page write. No LLM calls; pure pattern matching on [[wiki/people/bob]] style references. New entity → new page stub → graph grows.
  • Cron-driven enrichment runs while you sleep: dedup people pages, fix citations, score salience, find contradictions, prep tomorrow's tasks.

The whole loop is described in docs/architecture/topologies.md with diagrams.

Capabilities

Hybrid search. Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (conservative, balanced, tokenmax) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in docs/eval/SEARCH_MODE_METHODOLOGY.md. Default: balanced with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run gbrain search "<query>" --explain to see per-stage attribution: base score, every boost that fired, what it multiplied. gbrain doctor ships a graph_signals_coverage check; gbrain search stats shows fire counts and failure breakdowns.

Self-wiring knowledge graph. Every put_page extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (attended, works_at, invested_in, founded, advises, mentions, …). Multi-hop traversal via gbrain graph-query. The graph is what produces the +31.4 P@5 lift over vector-only RAG.

Job queue (Minions). BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.

43 curated skills. Routing lives in skills/RESOLVER.md. Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.

Eval framework. gbrain eval longmemeval runs the public LongMemEval benchmark against your hybrid retrieval. gbrain eval export + gbrain eval replay capture real queries and replay them against code changes (set GBRAIN_CONTRIBUTOR_MODE=1). gbrain eval cross-modal cross-checks an output against the task using three different-provider frontier models. Full methodology in docs/eval/SEARCH_MODE_METHODOLOGY.md.

Brain consistency. gbrain eval suspected-contradictions samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.

Agent-authored schema (v0.40.7.0). Your brain has a shape — what page types exist (person, meeting, paper, case, lab-result), what they link to (attended, authored, prescribed-by), what facts get extracted automatically. The default ships with 22 universal types, but your brain's actual shape is not the default shape. Agents can now evolve that shape on your behalf via 14 gbrain schema CLI verbs + a batched MCP op (schema_apply_mutations, admin scope, NOT localOnly so remote agents reach it over HTTPS). Atomic file locks, audit log with the agent's identity, chunked UPDATE backfill in 1000-row batches that never wedge concurrent writers. The brain stops being a pile of notes and becomes something with structure. Why it matters: docs/what-schemas-unlock.md — 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator). 5-minute walkthrough: docs/schema-author-tutorial.md. Agent skill: skills/schema-author/SKILL.md.

Integrations

Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in recipes/ and is discoverable via gbrain integrations list.

Architecture

Two engines, one contract. PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first BrainEngine interface in src/core/engine.ts defines ~47 operations both engines implement; CLI and MCP server are generated from one source.

Brain repo is the system of record. Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in docs/architecture/topologies.md.

Two organizational axes (brain ⊥ source). A brain is a database (your personal brain, a team mount you joined). A source is a repo inside that brain (wiki, gstack, an essay, a knowledge base). Routing lives in .gbrain-source dotfiles and resolves via a documented 6-tier precedence chain. Full diagrams in docs/architecture/brains-and-sources.md.

Why the graph matters. Vector search returns chunks that are semantically close. The graph returns chunks that are factually connected. Hybrid search pulls from both; auto-linking on every write keeps the graph fresh. Deep dive: docs/architecture/RETRIEVAL.md.

Troubleshooting

gbrain import fails with expected N dimensions, not M? Run gbrain doctor. It will print the exact gbrain config set ... or gbrain retrieval-upgrade command to repair the mismatch. You should not need to delete ~/.gbrain. Fresh gbrain init --pglite auto-detects your embedding provider from API keys in your environment: set OPENAI_API_KEY (or ZEROENTROPY_API_KEY / VOYAGE_API_KEY) before running init, or pass --embedding-model <provider>:<model> explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass --no-embedding to defer setup until runtime. See docs/integrations/embedding-providers.md for the full provider matrix and docs/operations/headless-install.md for Docker/CI sequencing.

Docs

  • docs/INSTALL.md — every install path, end to end
  • docs/what-schemas-unlock.md — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0)
  • docs/schema-author-tutorial.md — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via gbrain whoknows
  • docs/architecture/ — system design, topologies, retrieval theory
  • docs/guides/ — how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion)
  • docs/integrations/ — connecting external data sources (voice, email, calendar, embedding providers)
  • docs/mcp/ — per-client MCP setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)
  • docs/eval/ — eval framework, metric glossary, methodology
  • docs/ethos/ — philosophy (thin harness, fat skills, markdown as recipes, origin story)
  • AGENTS.md — entry point for non-Claude agents
  • CLAUDE.md — entry point for Claude Code (deep operating context)
  • CONTRIBUTING.md — contributor guide, test discipline, eval-capture mode
  • SECURITY.md — OAuth threat model, hardening defaults

Contributing

Run bun run test for the fast loop, bun run verify for the pre-push gate, bun run ci:local to run the full Docker-backed CI stack locally. Detailed test discipline in CONTRIBUTING.md.

Community PRs are batched into release waves rather than merged one-by-one — see the "PR wave workflow" section in CLAUDE.md. Contributor attribution stays attached via Co-Authored-By: trailers. We credit every accepted contribution in CHANGELOG.md.

If you find a bug or want a feature: open an issue first. Quick fixes (typo, doc bug, obvious regression) can go straight to a PR. Anything touching schema, retrieval ranking, MCP protocol, or the security boundary needs a design discussion in the issue first.

License + credit

MIT. I built GBrain to run my OpenClaw and Hermes deployments — the production brain behind my AI agents.

Origin story: docs/ethos/ORIGIN.md.

Community PR contributors are credited in CHANGELOG.md per release. ZeroEntropy (@zeroentropy) for the embedding + reranker stack that ships as the default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.

S
Description
No description provided
Readme MIT
160 MiB
Languages
TypeScript 97.4%
Shell 1.2%
JavaScript 0.9%
PLpgSQL 0.4%