1666ec427e
* feat(source-id): canonical dependency-free source_id validator module
New src/core/source-id.ts consolidates the three regex sites that drifted
across the codebase (utils.ts permissive, sources-ops.ts strict,
source-resolver.ts strict). Exports:
- SOURCE_ID_RE: ^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$ (strict kebab,
1-32 chars, no underscores, alphanumeric boundaries)
- isValidSourceId(s): boolean — for silent-fallback tiers (dotfile,
brain_default config)
- assertValidSourceId(s): void, throws — for explicit-validation tiers
(explicit --source flag, GBRAIN_SOURCE env, cycleLockIdFor primitive)
Dependency-free by design (no engine imports), so both PGLite and
Postgres engines can pull it without circular-import risk. Replaces
the soon-to-be-removed local validators in utils.ts and sources-ops.ts;
preserves both call shapes (boolean + throwing) per the codex outside-voice
finding that resolver tiers need both.
19 unit tests covering valid ids, length boundary (32-char max), underscore
rejection, path-traversal shapes, edge hyphens, whitespace, non-ASCII,
non-string inputs, and TypeScript narrowing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(source-id): migrate three regex sites to canonical source-id.ts
Consolidates source_id validation through src/core/source-id.ts:
- src/core/utils.ts: validateSourceId is now a back-compat re-export of
assertValidSourceId. Regex TIGHTENS from permissive ^[a-z0-9_-]+$ to
the strict kebab. The path-safety boundary now matches what sources-ops
enforces at source creation time; no production source IDs break because
sources-ops always rejected underscored IDs at creation. Picks up the
blast-radius callers in cycle/patterns.ts and cycle/synthesize.ts
reverse-write paths.
- src/core/sources-ops.ts: deletes local SOURCE_ID_RE + validateSourceId;
imports isValidSourceId from source-id.ts. Keeps the thin SourceOpError-
wrapping validator so `gbrain sources add` keeps its user-facing error
envelope.
- src/core/source-resolver.ts: imports SOURCE_ID_RE + isValidSourceId from
source-id.ts. Per codex outside-voice P1-F, silent-fallback tiers
(dotfile read at tier 3, brain_default config at tier 5) use
isValidSourceId so an invalid dotfile/config value falls through to the
next resolver tier instead of throwing. Explicit + env tiers keep their
inline regex-test-and-throw shape because they need tailored error
messages.
Behaviour: validateSourceId('snake_id') NOW THROWS where pre-PR it accepted.
Documented as the intentional tightening; no existing IDs in production
contain underscores.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): per-source lock primitive + db-lock consolidation + PGLite ordering
Three intertwined cycle.ts changes that landed as one logical unit:
1) DELETE acquirePostgresLock + acquirePGLiteLock (~75 LOC of duplicated
UPSERT-with-TTL SQL). Replace with tryAcquireDbLock from
src/core/db-lock.ts, which was extracted in v0.22.13 and should have
been adopted here at that time. New acquireDbCycleLock(engine, sourceId)
is a 6-line adapter that keeps cycle.ts's LockHandle shape.
Deliberately uses tryAcquireDbLock NOT withRefreshingLock (codex r2 P0-A):
- tryAcquireDbLock returns null on busy → cycle returns
{status:'skipped', reason:'cycle_already_running'} (existing contract)
- withRefreshingLock throws → would convert busy cycles into failures
- withRefreshingLock's background timer would skip Minion job-lock
renewal (codex r2 P0-B) and add in-phase DB traffic on PGLite's
single connection (codex r2 P1-A)
2) Add cycleLockIdFor(sourceId?: string) primitive:
- undefined → 'gbrain-cycle' (legacy default, back-compat for autopilot
and every existing caller)
- valid kebab → 'gbrain-cycle:<source_id>' (per-source DB lock row)
- invalid → throws via assertValidSourceId (codex r2 P1-B defense-
in-depth at the primitive layer, since CycleOpts.sourceId is a new
direct API surface that becomes part of a DB lock ID AND a PGLite
file path component)
Add CycleOpts.sourceId; thread through to acquireDbCycleLock. Documents
that this only scopes the LOCK — embed/orphans/purge/etc remain
brain-global per PHASE_SCOPE.
3) PGLite file+DB ordering invariant (codex r2 P0-C + P0-D):
- PGLite engines acquire the GLOBAL file lock (cycle.lock, no source
suffix) BEFORE the per-source DB lock. PGLite's process-level
write-lock is the single-writer guard; per-source DB lock IDs
alone would let two PGLite cycles run concurrently.
- File lock release on DB acquisition failure (cleanup guarantee)
- Compose both handles into one LockHandle whose release() is
reverse-of-acquire (DB first, file last) so file lock isn't released
while DB lock is still live.
- Postgres engines skip the file lock entirely — per-source DB IDs
are the full granularity.
13 unit tests in test/cycle-lock-per-source.test.ts pin the back-compat
default, per-source ID shape, distinct-ID property, and the internal-
validation throws.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): PHASE_SCOPE taxonomy + doctor cycle_phase_scope check
Static documentation of each cycle phase's scope: 'source' (safe to
parallelize per source), 'global' (must serialize brain-wide), or
'mixed' (per-phase decomposition needed before parallelizing).
The PHASE_SCOPE record is the load-bearing input for any future
autopilot fan-out wave. It surfaces what codex round-1 P0-1 was
warning about: not all 14 cycle phases are source-scoped today.
embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile
walk brain-wide regardless of sourceId. Per-source cycle LOCKS (this
PR) let two cycles RUN concurrently, but global-scoped phases inside
each will still touch the same rows.
The taxonomy is documentation, not runtime enforcement (runtime
enforcement deferred per plan; filed as TODO).
New doctor check cycle_phase_scope renders the taxonomy as an
operator-facing message AND surfaces phase_scope_map under
Check.details for JSON consumers. Added optional Check.details field
to the doctor types — mirrors PhaseResult.details. Additive; no
schema_version bump.
11 unit tests across phase-scope-coverage + doctor-cycle-phase-scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(source-id): blast-radius regression for strict-regex callers
Pins the codex round-2 P1-D finding: utils.validateSourceId is also
used in cycle reverse-write paths at patterns.ts:263 and
synthesize.ts:909. Pre-PR they used the permissive regex; post-PR
they share the strict kebab regex with sources-ops creation-time
validation. Existing underscore IDs would fail at THOSE cycle sites,
not just at source add/remove.
Structural assertions guard against future drift:
- utils.ts validateSourceId === assertValidSourceId from source-id.ts
- patterns.ts + synthesize.ts both import validateSourceId from utils
- validation call precedes the join() at both reverse-write sites
- utils.ts no longer contains the inline ^[a-z0-9_-]+$ permissive regex
- utils.ts re-exports assertValidSourceId-as-validateSourceId from source-id.ts
9 cases pin the contract. IRON-RULE: source-text grep regressions land
on the offending refactor first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(engine): listAllSources + updateSourceConfig for per-source autopilot
Two lean engine-layer methods that the v0.38 per-source autopilot wave
consumes. Both have parity implementations on Postgres + PGLite.
listAllSources(opts?):
- Returns the bare SourceRow shape (id, name, local_path, last_sync_at,
config) without sources-ops.listSources's per-source page_count
enrichment (N+1 expensive; out of scope for hot-loop callers).
- `includeArchived` defaults false (matches sources-ops semantics).
- `localPathOnly` filters local_path IS NOT NULL so autopilot fan-out
doesn't dispatch jobs for pure-DB sources whose handler would fall
back to global sync.repo_path (codex r1 P1-4).
- Ordering: (id = 'default') DESC, id — same as sources-ops for
operator-output stability.
updateSourceConfig(sourceId, patch):
- Atomic JSONB merge via Postgres `config || $patch::jsonb` operator.
No read-modify-write race; same-key overwrites (no deep merge —
flat patches only, matches the v0.38 use case of last_full_cycle_at).
- Returns true when a row was updated, false when sourceId doesn't
exist (best-effort no-op; caller decides how to handle).
- Postgres: sql.json(patch) per the canonical pattern; PGLite:
JSON.stringify + ::jsonb cast on positional param.
New SourceRow type exported from engine.ts. Imported by both engine
impls. 11 integration tests in test/list-all-sources.test.ts cover
defaults, filters, JSONB round-trip, archived flag, and merge semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): write last_full_cycle_at to sources.config on per-source cycle exit
Closes codex round-1 P0-5 (write site for last_full_cycle_at was
unspecified pre-PR). runCycle's exit hook persists
{ last_full_cycle_at: '<ISO>' } to sources.config JSONB when a
successful per-source cycle completes. Autopilot's v0.38 per-source
fan-out gate reads this field next tick to decide whether to skip a
source (60-min freshness floor).
Conditions for write (all required):
- opts.sourceId is set — legacy callers without sourceId skip the
write (autopilot will keep working today via fallback path)
- engine is non-null — no-DB path skips
- status is 'ok' / 'clean' / 'partial' — failed/skipped cycles do NOT
mark a source as fresh (next cycle will redo work)
- dryRun is false — writes are out of scope
Best-effort: write failure logs a warning but does NOT change the
CycleReport status. The cycle already succeeded by the time we get
here; the cost of missing a stale write is one redundant cycle next
tick, not data loss.
5 PGLite integration tests cover all four gate conditions plus the
"timestamp advances on each successful run" property.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(jobs): autopilot-cycle handler honors source_id + pull + archive recheck
Threads the v0.38 per-source dispatch payload through the autopilot-cycle
handler at src/commands/jobs.ts:1146. Closes three codex round-1 findings:
- P0-2 / P1-B: validates job.data.source_id at handler entry via the
canonical source-id.ts isValidSourceId boolean check. Malformed
source_id from a queue replay dead-letters with a clear error
instead of reaching cycle code.
- P1-2: job.data.pull explicit boolean overrides the legacy hardcoded
`true`, so per-source dispatch for local-only sources can pass
pull: false (no git network round-trip for sources without remote_url).
Missing/undefined preserves the legacy true for back-compat with cron/
launchd callers that don't know about the new field.
- P1-5: archived-source recheck happens BEFORE runCycle is invoked
(cheap SELECT archived FROM sources WHERE id = $1). If the source was
archived between fan-out and worker claim, handler returns
{ status: 'skipped', reason: 'source_archived' } cleanly — no lock
acquired, no phases run, no last_full_cycle_at touched. Same skip
shape for source_not_found (deleted between dispatch and claim).
7 PGLite integration tests cover all five paths (legacy / valid /
not-found / archived / malformed-source_id / non-string source_id /
pull: false override).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(autopilot): per-source fan-out dispatch (the headline parallelism win)
The headline change of the v0.38 federated-sync wave. Replaces autopilot's
single-job-per-tick dispatch with per-source fan-out so a 5-source
federated brain refreshes in ~5min wall-clock instead of ~25min sequential.
src/commands/autopilot-fanout.ts (new) — pure-function dispatch helper:
- resolveFanoutMax(engine): PGLite=1 (codex P1-3 — preserves single-writer
invariant), Postgres=4, operator override via autopilot.fanout_max_per_tick
- readLastFullCycleAt(src): JSONB→Date with NULL/unparseable safety
- isSourceStale(src, now?, floorMin?): 60-min default freshness floor
- selectSourcesForDispatch(sources, fanoutMax): stale-only + oldest-first
+ alphabetical tiebreaker (deterministic for tests)
- dispatchPerSource(engine, queue, opts): the orchestrator
src/commands/autopilot.ts (modified): the existing shouldFullCycle branch
calls dispatchPerSource. Behavior preserved:
- Healthy + recent (60min floor) → sleep (unchanged)
- Targeted-plan path → unchanged (uses computeRecommendations)
- Full-cycle path → NOW fans out per-source rather than ONE job for default
Per-source dispatch shape:
- Idempotency key: `autopilot-cycle:<source_id>:<slot>` — two ticks for
the same source within one slot coalesce; different sources never collide
- pull: !!source.config.remote_url — remotes pull, local-only don't
- maxWaiting: 1 per submit — backpressure when worker can't drain
- Per-submit try/catch (codex E1 F1) — one source's failure doesn't
abort the tick; surfaces as fanout_submit_failed event
Fallback path: empty `sources` table (pre-v0.18 brain or fresh install
before `gbrain sources add`) falls back to the legacy single autopilot-
cycle job with no source_id, preserving today's single-source behavior.
JSON event stream extended:
- `dispatched` event gains source_id + mode='per_source' fields
- new `fanout_summary` event per tick with dispatched/skipped_fresh/
skipped_cap arrays so operators can see what the tick did
- new `fanout_cap_reached` event when sources overflow the cap
Caveat (intentional, codex r1 P0-1 scope): per-source LOCKS let two
cycles RUN concurrently, but several phases (embed, orphans, purge,
resolve_symbol_edges, grade_takes, calibration_profile) still walk the
brain globally inside each cycle. PHASE_SCOPE taxonomy from the prior
commit documents this. Genuine per-phase per-source isolation is the
deferred Phase 2 follow-up.
27 unit tests in test/autopilot-fanout.test.ts pin every branch (stale
gate, cap behavior, idempotency keys, legacy fallback, per-submit error
isolation, oldest-first sort, alphabetical tiebreaker).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): cycle_freshness check — sibling to sync_freshness
New check_cycle_freshness sibling to checkSyncFreshness. Where
sync_freshness reads sources.last_sync_at (one phase), this check
reads sources.config->>'last_full_cycle_at' — the canonical
"this whole cycle completed" timestamp the v0.38 runCycle exit hook
writes and the v0.38 autopilot fan-out gate reads.
Operator sees exactly what autopilot sees when deciding to skip a
source. Default thresholds tighter than sync_freshness (6h warn /
24h fail vs 24h/72h) because full-cycle staleness compounds: sync
stale → extract stale → embed stale → search returns stale results.
Env overrides:
- GBRAIN_CYCLE_FRESHNESS_WARN_HOURS (default 6)
- GBRAIN_CYCLE_FRESHNESS_FAIL_HOURS (default 24)
Edge cases covered (9 PGLite integration tests):
- empty (no federated sources) → ok
- last_full_cycle_at present + fresh → ok
- last_full_cycle_at present + warn window → warn
- last_full_cycle_at present + fail window → fail
- last_full_cycle_at NULL (never cycled) → fail
- mixed severity → highest wins
- future timestamp (clock skew) → warn
- unparseable timestamp → warn
- local_path NULL sources filtered (codex P1-4 parity)
Failure messages embed source.id so the printed fix command
`gbrain dream --source <id>` matches what the user copy-pastes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: fill gaps surfaced by post-Phase-2 audit (resolver silent-fallback + PGLite ordering)
Two test gaps identified by the test-coverage audit after Phase 1 + 2 shipped:
1. src/core/source-resolver.ts — the migration to isValidSourceId for
silent-fallback tiers (dotfile read, brain_default config) had no
direct test. Pre-PR these used inline regex; post-PR they use the
canonical isValidSourceId. Codex P1-F intent (silent fallback for
invalid input on tiers 3+5, throw on tiers 1+2) deserved an explicit
test.
test/source-resolver-silent-fallback.test.ts (12 cases):
- tier 3: valid dotfile honored; underscore/whitespace/uppercase
silently falls through to next tier
- tier 5: valid brain_default honored; underscore + 33+ char silently
falls through
- tier 1: valid explicit --source returns; underscore/whitespace
THROWS (contract distinction)
- tier 2: valid env GBRAIN_SOURCE returns; underscore THROWS
2. src/core/cycle.ts — the PGLite file+DB ordering invariant (codex r2
P0-C + P0-D) was implemented in Phase 1 (T5) but had no test pinning
the ordering / cleanup / per-source DB lock ID semantics.
test/cycle-pglite-lock-ordering.test.ts (6 cases):
- global file lock acquired during PGLite cycle
- cycle for source A then B serializes (file lock held in turn)
- DB-lock acquire failure releases file lock cleanly (no stranded state)
- engine=null path still uses file lock
- DB lock row uses per-source ID (gbrain-cycle:<source>) not legacy
- consecutive cycles can re-acquire both locks (release-on-exit works)
Plus .context/PHASE_3_ASSESSMENT.md (gitignored) documenting why each
Phase 3 DRY refactor item from the original plan is deferred: each item
on closer inspection is either a premature abstraction or was explicitly
rejected by the original author (BaseCyclePhase header comment).
18 new tests; 0 fails. typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: autopilot wiring static guard + Postgres parity e2e for fan-out
Closes the remaining test gaps identified by the post-Phase-2 audit:
test/autopilot-fanout-wiring.test.ts (5 cases) — static-shape regression
for autopilot.ts ↔ dispatchPerSource. The fan-out helper itself has 27
unit tests; this file pins the WIRING in autopilot.ts:
- imports dispatchPerSource + resolveFanoutMax
- calls dispatchPerSource inside the shouldFullCycle branch (not the
targeted-plan path)
- updates lastFullCycleAt after dispatch
- does NOT regress to the pre-PR single-job dispatch (regex-grep guard
against the legacy `autopilot-cycle:${slot}` idempotency-key shape
reappearing in autopilot.ts)
Same canonical static-shape pattern as test/autopilot-supervisor-wiring.test.ts.
test/e2e/list-all-sources-postgres.test.ts (10 cases) — Postgres parity
for engine.listAllSources + updateSourceConfig. The PGLite path has
unit-level coverage; the Postgres path has separate impls (sql.json
serialization, sql.count semantics) that could drift. Specifically pins:
- returns rows, filters archived/localPath correctly
- JSONB config parses to object (autopilot reads last_full_cycle_at)
- default source sorts first
- updateSourceConfig: not-found returns false, patch merges, same-key
overwrites, idempotent on repeat
- jsonb_typeof regression: round-trip stores real JSONB object, NOT
a JSON-encoded string (feedback_postgres_jsonb_double_encode class)
test/e2e/autopilot-fanout-postgres.test.ts (6 cases) — end-to-end
integration on Postgres:
- 3 sources fan out as 3 distinct jobs with per-source idempotency keys
- re-dispatch within same slot dedupes (idempotency-key coalesce)
- last_full_cycle_at < 60min ago sources are skipped by gate
- end-to-end: updateSourceConfig → listAllSources → selectSourcesForDispatch
correctly classifies fresh sources
- fan-out cap honored (5 sources, fanoutMax=2 → 2 dispatched)
- empty federated brain falls back to legacy single-job dispatch
21 new test cases. Brings the v0.38 wave coverage to 132 unit + 16 e2e.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(autopilot): drop maxWaiting from per-source submit (E2E found silent coalesce)
The Postgres E2E for fan-out surfaced two real bugs that the unit-stub
tests + PGLite tests couldn't catch:
1) **CRITICAL** — dispatchPerSource passed `maxWaiting: 1` to every
per-source queue.add. maxWaiting is per-(name, queue) — since all
per-source jobs share `name='autopilot-cycle'`, the second + third +
Nth source's submit silently coalesced into the FIRST source's
waiting job. Net result on a real worker: 1 job processed per tick,
not N. The entire fan-out feature was a silent no-op past the first
source.
Per-source idempotency_key (`autopilot-cycle:<source_id>:<slot>`)
already handles "two ticks for same source within slot" dedup, which
is the only thing maxWaiting was buying us. Dropping it fixes fan-out
without losing dedup.
New unit-stub regression test asserts maxWaiting is NOT in the per-
source submit opts so a future refactor that re-adds it gets caught
in 100x faster CI (test/autopilot-fanout.test.ts).
2) **postgres-engine.ts:updateSourceConfig** — initial impl used
sql.json() correctly but my mid-debug rewrite to executeRaw +
positional `$1::jsonb` produced JSONB STRING shape (not OBJECT)
because postgres-js double-encodes JS string params in unsafe mode.
`||` between JSONB object + JSONB string yields a JSONB ARRAY,
wiping every existing config key on update.
Same latent bug class exists at src/commands/sources.ts:482 (gbrain
sources federate/unfederate path); flagged for follow-up but
out-of-scope here.
Reverted to sql.json() inside the template tag (verified via direct
psql round-trip: jsonb_typeof = 'object'). Updated the e2e seed
helper to use sql.json() too — the executeRaw + JSON.stringify
pattern was producing string-shape JSONB at SEED time which made
the failure cascade harder to debug.
Coverage adds:
- test/e2e/list-all-sources-postgres.test.ts: 11 cases pin Postgres
parity for listAllSources + updateSourceConfig including jsonb_typeof
round-trip
- test/e2e/autopilot-fanout-postgres.test.ts: 6 cases end-to-end
including 3-source fan-out producing 3 distinct rows, idempotency
coalesce within slot, cap honored, legacy fallback path
- test/autopilot-fanout.test.ts: +1 regression guard on maxWaiting
This is the kind of bug that justifies the user's "fill test gaps then
run E2E" mandate. The unit tests + PGLite parity tests + typecheck all
passed cleanly; only the real-Postgres E2E found the coalesce.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): update multi-source-bug-class validateSourceId expectations for strict regex
The v0.32.8 test pinned the OLD permissive ^[a-z0-9_-]+$ behavior including
the underscored case 'jarvis_memory'. The v0.38 wave (this PR's E2 + codex
P1-D) tightens validateSourceId to the strict kebab regex shared with
sources-ops. 'jarvis_memory' now lives in the rejected set, not the
allowed set.
Updated the test to assert the new contract:
- Replaced 'jarvis_memory' allowed case with 'jarvis-memory' (kebab)
- Added 'a' (single-char) to allowed cases
- Added 'jarvis_memory', 'snake_case', '-leading', 'trailing-', and a
33-char string to the rejected cases — the v0.38 strict-regex additions
Comment explains the contract shift so future readers don't see the test
as flapping intent.
Found by running the main E2E suite — the test file is in the canonical
e2e set and would have failed CI otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* v0.39.3.0: per-source autopilot fan-out + cycle lock primitive + phase taxonomy
VERSION + CHANGELOG bump for the parallel-federated-sync wave (14 commits).
Headline change: federated brains refresh all sources in parallel via
per-source autopilot dispatch instead of one source per 5-min tick.
Five-source brain wall-clock: ~25min → ~5min.
Test infra adjustments for the v0.38 test-isolation lint that landed via
upstream master merge:
- test/source-resolver-silent-fallback.test.ts now uses withEnv() for
GBRAIN_SOURCE mutations (was direct process.env mutation)
- test/cycle-pglite-lock-ordering.test.ts → .serial.test.ts (the
file-wide GBRAIN_HOME setup needs quarantine from the parallel pool)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): bump cycle-abort handler-source-grep window 2000→6000
CI on c835f93f failed on 1 of 2005 tests: the v0.20.5
"autopilot-cycle handler contract" regression guard at
test/cycle-abort.test.ts:99 does a source-grep over the first 2000
chars after `worker.register('autopilot-cycle'` to assert `signal:
job.signal` appears.
The v0.38 wave (in c835f93f) added source_id validation + archive
recheck + pull-flag threading at the top of that handler, pushing the
runCycle({signal: job.signal}) call from ~chars 800 to ~chars 2100.
The slice cut off before reaching it, so the assertion failed even
though the code still propagates the signal correctly (line 1213).
Fix: bump the window to 6000 chars and add a comment explaining why.
The guard's intent is unchanged ("handler passes job.signal to
runCycle"); the window just needs to be wide enough to span any
reasonable handler. Also added an existence check on `handlerStart`
so a future refactor that renames the register call surfaces a
clearer error than `undefined.toContain`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(merge): add schema-suggest to PHASE_SCOPE taxonomy
The v0.39.1.0 master merge added a new 'schema-suggest' cycle phase.
PHASE_SCOPE is a TS Record<CyclePhase, PhaseScope> so the compiler
required an entry. Classified as 'source' — the phase accepts
sourceId and operates per-source via runSuggest().
* chore(version): renumber v0.39.3.0 → v0.39.2.0
Renumber to claim the next available slot after v0.39.1.0 (schema packs)
landed on master. No code changes.
* fix(test): bump PHASE_SCOPE count 16→17 for schema-suggest
The v0.39.1.0 master merge added the 17th cycle phase ('schema-suggest').
The previous commit added it to PHASE_SCOPE; this commit updates the
count-pin regression test to match.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GBrain
Your AI agent is smart but forgetful. GBrain gives it a brain.
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: 146,646 pages, 24,585 people, 5,339 companies, 66 cron jobs running autonomously. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.
The brain wires itself. Every page write extracts entity references and creates typed links (attended, works_at, invested_in, founded, advises) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side: gbrain lands P@5 49.1%, R@5 97.9% on a 240-page Opus-generated rich-prose corpus, beating its graph-disabled variant by +31.4 points P@5 and ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling gbrain-evals repo.
New default in v0.36.2.0: ZeroEntropy for both embedding (zembed-1 at 1280d via Matryoshka) and reranker (zerank-2). On a real-corpus benchmark vs OpenAI and Voyage: 2.2× faster (442ms vs OpenAI 973ms), 2.6× cheaper at regular pricing ($0.05/M vs OpenAI $0.13), wins 11 of 20 queries head-to-head, reshuffles 60% of top-1 results when used as a second-pass reranker. Bring your own key from zeroentropy.dev, or switch to OpenAI/Voyage at install time via gbrain init --pglite --embedding-model <provider:model> --embedding-dimensions <N> — your choice is sticky. To switch an existing brain, run gbrain reinit-pglite --embedding-model <provider:model> --embedding-dimensions <N> (PGLite) or follow the SQL recipe in docs/embedding-migrations.md (Postgres). gbrain config set embedding_model is refused as of v0.37.11.0 because the schema column has to resize too.
GBrain is those patterns, generalized. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself. One command does the loop you used to run by hand: gbrain doctor --remediate --yes --target-score 90 --max-usd 5. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. gbrain doctor --remediation-plan --json previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (reindex, repair-jsonb, orphans, integrity, purge, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New --background flag on gbrain embed submits the job and exits with job_id=N for shell composition.
New in v0.35.7 — Temporal trajectory + founder scorecard. Author typed metric assertions in the ## Facts fence (mrr=50000, arr=2000000, team_size=12) and gbrain stores them as first-class typed columns. gbrain eval trajectory companies/acme-example prints the chronological history with regressions auto-flagged inline. gbrain founder scorecard companies/acme-example rolls up claim accuracy, consistency, growth direction, and red flags into a stable schema_version: 1 JSON contract. New MCP op find_trajectory exposes the same data to agents (read scope, visibility-filtered for remote callers). The consolidate cycle phase now writes valid_until on chronologically-superseded facts AND uses semantic upsert on (page_id, claim, since_date) — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions).
~30 minutes to a fully working brain. Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
LLMs: fetch
llms.txtfor the documentation map, orllms-full.txtfor the same map with core docs inlined in one fetch. Agents: start withAGENTS.md(orCLAUDE.mdif you're Claude Code).
Install
GBrain runs in three shapes. Pick the one that matches how you use AI agents today.
Run with your agent platform
Already using OpenClaw or Hermes? GBrain installs as a skillpack scaffold into your agent's workspace.
gbrain init --pglite
gbrain skillpack scaffold --all # or: scaffold <name> per skill
That's it. Your agent picks up 43 skills (signal detection, brain-ops, ingest, enrich, citation-fixer, daily-task-manager, cron-scheduler, eval framework, and 35 more). Routing lives in skills/RESOLVER.md — the agent reads it once per request, picks the right skill, executes. Scaffolded skills are first-class members of your agent repo — you own them, edit freely; gbrain skillpack reference <name> diffs your copy against gbrain's bundle when you want to pull upstream improvements. (The legacy gbrain skillpack install managed-block model was retired in v0.36.0.0; run gbrain skillpack migrate-fence once if you're upgrading from an older release.)
CLI standalone
Use gbrain from any shell, no agent platform required.
bun install -g github:garrytan/gbrain
gbrain init --pglite # 2 seconds; no server, no Docker
gbrain doctor # verify health
Then point any MCP-aware client (Claude Code, Cursor, Windsurf) at it, or use it from your shell:
gbrain search "who works at acme AI?"
gbrain query "what did bob invest in this quarter?"
gbrain graph-query people/garry-tan --depth 2
Detailed setup paths (Postgres at scale, Supabase, thin-client mode) live in docs/INSTALL.md.
MCP server (any MCP client)
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
# at /admin, SSE activity feed at /admin/events
Per-client guides (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) live under docs/mcp/. HTTP server supports DCR-style client registration, scope-gated access (read/write/admin), and built-in rate limiting.
How to get data in (v0.38+)
One command, local or hosted, synchronous receipt:
gbrain capture "the thought I want to remember"
gbrain capture --file ./notes/today.md
echo "from a pipe" | gbrain capture --stdin
SLUG=$(gbrain capture "..." --quiet)
The page lands in the DB AND on disk in one move (the v0.38 put_page
write-through plumbing). Default slug inbox/YYYY-MM-DD-<hash8> so
captures cluster in a predictable triage location. On thin-client installs
the verb routes through MCP to the server — same command, same UX.
For webhook ingestion (Zapier / IFTTT / Apple Shortcuts):
curl -X POST https://your-brain/ingest \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: text/markdown" \
-d "# a thought from a Shortcut"
For mobile capture, the inbox folder source picks up anything dropped into
~/.gbrain/inbox/ from iOS Shortcuts / AirDrop / Drafts / Finder.
Third-party skillpacks can ship custom ingestion sources (Granola, Linear,
voice, OCR) against the versioned IngestionSource contract at
gbrain/ingestion. See docs/skillpack-anatomy.md.
What it does (the loop)
signal → search → respond → write → auto-link → sync
(every (brain-first (informed (page + (typed edges (cron
message) retrieval) by context) timeline) + backlinks) keeps fresh)
- Signal detector runs on every message your agent receives. Captures ideas, entity mentions, time-sensitive todos, names, links.
- Brain-first lookup before any external API call. The cheapest, fastest, most personal information source you have.
- Auto-link fires on every page write. No LLM calls; pure pattern matching on
[[wiki/people/bob]]style references. New entity → new page stub → graph grows. - Cron-driven enrichment runs while you sleep: dedup people pages, fix citations, score salience, find contradictions, prep tomorrow's tasks.
The whole loop is described in docs/architecture/topologies.md with diagrams.
Capabilities
Hybrid search. Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (conservative, balanced, tokenmax) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in docs/eval/SEARCH_MODE_METHODOLOGY.md. Default: balanced with ZeroEntropy reranker on.
Self-wiring knowledge graph. Every put_page extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (attended, works_at, invested_in, founded, advises, mentions, …). Multi-hop traversal via gbrain graph-query. The graph is what produces the +31.4 P@5 lift over vector-only RAG.
Job queue (Minions). BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
43 curated skills. Routing lives in skills/RESOLVER.md. Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
Eval framework. gbrain eval longmemeval runs the public LongMemEval benchmark against your hybrid retrieval. gbrain eval export + gbrain eval replay capture real queries and replay them against code changes (set GBRAIN_CONTRIBUTOR_MODE=1). gbrain eval cross-modal cross-checks an output against the task using three different-provider frontier models. Full methodology in docs/eval/SEARCH_MODE_METHODOLOGY.md.
Brain consistency. gbrain eval suspected-contradictions samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.
Integrations
Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in recipes/ and is discoverable via gbrain integrations list.
- Voice: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe:
recipes/twilio-voice-brain.md. - Email + calendar: webhook handlers that route to brain signals.
docs/integrations/meeting-webhooks.md. - Embedding providers: 16 recipes covering OpenAI (default fallback), OpenRouter, Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in
docs/integrations/embedding-providers.md. - Credential gateway: vault-aware secret distribution.
docs/integrations/credential-gateway.md. - MCP clients: every major MCP client is supported.
docs/mcp/per-client setup.
Architecture
Two engines, one contract. PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first BrainEngine interface in src/core/engine.ts defines ~47 operations both engines implement; CLI and MCP server are generated from one source.
Brain repo is the system of record. Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in docs/architecture/topologies.md.
Two organizational axes (brain ⊥ source). A brain is a database (your personal brain, a team mount you joined). A source is a repo inside that brain (wiki, gstack, an essay, a knowledge base). Routing lives in .gbrain-source dotfiles and resolves via a documented 6-tier precedence chain. Full diagrams in docs/architecture/brains-and-sources.md.
Why the graph matters. Vector search returns chunks that are semantically close. The graph returns chunks that are factually connected. Hybrid search pulls from both; auto-linking on every write keeps the graph fresh. Deep dive: docs/architecture/RETRIEVAL.md.
Troubleshooting
gbrain import fails with expected N dimensions, not M? Run gbrain doctor. It will print the exact gbrain config set ... or gbrain retrieval-upgrade command to repair the mismatch. You should not need to delete ~/.gbrain. As of v0.37, fresh gbrain init --pglite auto-detects your embedding provider from API keys in your environment — set OPENAI_API_KEY (or ZEROENTROPY_API_KEY / VOYAGE_API_KEY) before running init, or pass --embedding-model <provider>:<model> explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass --no-embedding to defer setup until runtime. See docs/integrations/embedding-providers.md for the full provider matrix and docs/operations/headless-install.md for Docker/CI sequencing.
Docs
docs/INSTALL.md— every install path, end to enddocs/architecture/— system design, topologies, retrieval theorydocs/guides/— how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion)docs/integrations/— connecting external data sources (voice, email, calendar, embedding providers)docs/mcp/— per-client MCP setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)docs/eval/— eval framework, metric glossary, methodologydocs/ethos/— philosophy (thin harness, fat skills, markdown as recipes, origin story)AGENTS.md— entry point for non-Claude agentsCLAUDE.md— entry point for Claude Code (deep operating context)CONTRIBUTING.md— contributor guide, test discipline, eval-capture modeSECURITY.md— OAuth threat model, hardening defaults
Contributing
Run bun run test for the fast loop, bun run verify for the pre-push gate, bun run ci:local to run the full Docker-backed CI stack locally. Detailed test discipline in CONTRIBUTING.md.
Community PRs are batched into release waves rather than merged one-by-one — see the "PR wave workflow" section in CLAUDE.md. Contributor attribution stays attached via Co-Authored-By: trailers. We credit every accepted contribution in CHANGELOG.md.
If you find a bug or want a feature: open an issue first. Quick fixes (typo, doc bug, obvious regression) can go straight to a PR. Anything touching schema, retrieval ranking, MCP protocol, or the security boundary needs a design discussion in the issue first.
License + credit
MIT. Built by Garry Tan to run his OpenClaw and Hermes deployments — the production brain behind his actual AI agents.
Origin story: docs/ethos/ORIGIN.md.
Community PR contributors are credited in CHANGELOG.md per release. ZeroEntropy (@zeroentropy) for the embedding + reranker stack that became the v0.36.2.0 default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.