mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
7ea92d602c
* feat(git): divergence-safe pull, push-probe, default-branch detection for brain durability Add GIT_ENV_AUTH + divergenceSafePull (skip-on-dirty, conflict-abort-clean, never-mid-rebase), detectDefaultBranch, pushProbe, and an env-gated GBRAIN_GIT_ALLOW_FILE_TRANSPORT escape hatch. Export GIT_ENV. pullRepo's --ff-only contract is unchanged. * feat(durability): brain-repo hardening core (hook, helper, cron, PAT, AGENTS rules) hardenBrainRepo/unhardenBrainRepo: local untracked post-commit hook + committed brain-commit-push.sh (one shared push-retry template), repo-scoped credential with existing-helper reuse, push-probe verify, active-resolver-file rules with taxonomy from _brain-filing-rules.json, minimal DB-free pull cron. PAT redaction via redactSecretsInText. * feat(sources): harden/pull/unharden commands + auto-harden on add --url sources harden/pull/unharden subcommands; --pat-file/--no-harden on add; auto-harden managed clones on add; unharden-before-remove. cli.ts pre-connect early-exit for DB-free 'sources pull --path' (the cron entry, never opens PGLite). * test(durability): unit + integration coverage for brain-repo durability git helpers, core harden/unharden, hook+helper E2E (real background push), cron generators. 41 tests across 4 files. * chore: bump version and changelog (v0.42.48.0) Brain-repo git durability: auto-harden a brain's working tree (local auto-push hook, committed commit-push helper, always-on agent rules, DB-free pull cron, repo-scoped credential, push-probe verify) the moment gbrain gets a PAT + URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sources): route harden exit code through setCliExitVerdict A raw process.exitCode write is zeroed by the owned-verdict flush-exit (#2084 PGLite-Emscripten pollution defense); cli-exit-verdict-pin guard caught it. Use setCliExitVerdict(3) so 'sources harden' actually reports needs-attention to cron/automation. * docs: document brain-repo durability (KEY_FILES + multi-source guide) KEY_FILES: extend git-remote.ts entry (divergenceSafePull, pushProbe, detectDefaultBranch, GIT_ENV_AUTH, GBRAIN_GIT_ALLOW_FILE_TRANSPORT) + add brain-repo-durability.ts/sources-harden.ts entry. multi-source-brains.md: add a Durability (auto-harden) how-to covering sources harden/pull/unharden, --pat-file, the guarantees, and the security posture. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
235 lines
8.8 KiB
Markdown
235 lines
8.8 KiB
Markdown
# Multi-source brains
|
|
|
|
**A single gbrain database can hold multiple knowledge repos.** Each one
|
|
is a `source`: a logical brain-within-the-brain with its own slug
|
|
namespace, its own sync state, and its own federation policy. The rest
|
|
of this guide walks the three canonical scenarios.
|
|
|
|
## The three scenarios
|
|
|
|
### 1. Unified knowledge recall (wiki + gstack)
|
|
|
|
You have a personal wiki and a `gstack` checkout. Both belong to you,
|
|
both are knowledge you want your agent to recall across. When you ask
|
|
"what did I learn about X?" you want the best hit whether it lives in
|
|
the wiki or in a gstack plan.
|
|
|
|
```bash
|
|
# Register the gstack source, federate so it joins cross-source search
|
|
gbrain sources add gstack --path ~/.gstack --federated
|
|
|
|
# Pin the directory so `gbrain sync` knows which source it's walking
|
|
cd ~/.gstack && gbrain sources attach gstack
|
|
|
|
# Initial sync
|
|
gbrain sync --source gstack
|
|
|
|
# Now `gbrain search "retry budgets"` returns hits from BOTH wiki and
|
|
# gstack. Each result includes source_id so the agent can cite properly.
|
|
```
|
|
|
|
Result: wiki pages and gstack plans are separate (different source_ids,
|
|
different slug namespaces) but share the search surface.
|
|
|
|
### 2. Purpose-separated brains (yc-media + garrys-list)
|
|
|
|
You run two completely different content pipelines on the same backend.
|
|
YC Media covers portfolio news and founder profiles. Garry's List is
|
|
personal writing. You explicitly DON'T want them mixed in search — YC
|
|
portfolio content leaking into essay searches is a bug, not a feature.
|
|
|
|
```bash
|
|
# Two sources, both isolated (federated=false)
|
|
gbrain sources add yc-media --path ~/yc-media --no-federated
|
|
gbrain sources add garrys-list --path ~/writing --no-federated
|
|
|
|
# Pin each checkout directory
|
|
(cd ~/yc-media && gbrain sources attach yc-media)
|
|
(cd ~/writing && gbrain sources attach garrys-list)
|
|
|
|
# Sync each independently
|
|
gbrain sync --source yc-media
|
|
gbrain sync --source garrys-list
|
|
```
|
|
|
|
Result: searching from neither directory returns the `default` source
|
|
(your main brain). Searching from inside `~/yc-media` returns only yc-
|
|
media hits. Searching from inside `~/writing` returns only garrys-list.
|
|
Federation is opt-in, not leaked.
|
|
|
|
To search across them explicitly on demand:
|
|
|
|
```bash
|
|
gbrain search "tech layoffs" --source yc-media,garrys-list
|
|
```
|
|
|
|
### 3. Mixed (wiki federated + sessions isolated)
|
|
|
|
Your main wiki is federated with a few trusted sources. Your session
|
|
transcripts (coming in v0.18) land in a separate isolated source so
|
|
they don't dominate every search result.
|
|
|
|
```bash
|
|
# Federated sources
|
|
gbrain sources add gstack --path ~/.gstack --federated
|
|
|
|
# Isolated source (future v0.18 — sessions use this shape today for ingest)
|
|
gbrain sources add sessions --path ~/.claude/sessions --no-federated
|
|
```
|
|
|
|
## Resolution priority
|
|
|
|
When any command needs to pick a source, gbrain walks this list (highest
|
|
first):
|
|
|
|
1. Explicit `--source <id>` flag.
|
|
2. `GBRAIN_SOURCE` environment variable.
|
|
3. `.gbrain-source` dotfile in CWD or any ancestor directory.
|
|
4. A registered source whose `local_path` contains the CWD (longest
|
|
prefix wins for nested checkouts).
|
|
5. The brain-level default set via `gbrain sources default <id>`.
|
|
6. The seeded `default` source.
|
|
|
|
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
|
|
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
|
|
the `gstack` source. Outside any registered directory with no env/dotfile
|
|
set, it writes to the default.
|
|
|
|
## Federation flag
|
|
|
|
Every source row stores `config.federated: boolean` in its JSONB config.
|
|
|
|
| Value | Meaning |
|
|
|-------|---------|
|
|
| `true` | Source participates in unqualified `gbrain search "X"` results. |
|
|
| `false` (default for new sources) | Source only searched when explicitly named via `--source <id>` or qualified citation. |
|
|
|
|
The seeded `default` source is `federated=true` so pre-v0.17 brains
|
|
behave exactly as before — every page appears in search.
|
|
|
|
Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
|
|
|
|
## Commands
|
|
|
|
Full subcommand reference:
|
|
|
|
```
|
|
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated]
|
|
Register a source. id: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
|
|
gbrain sources list [--json] List all sources with page counts + federation state.
|
|
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
|
|
Cascade-delete a source (pages, chunks, timeline).
|
|
gbrain sources rename <id> <new-name>
|
|
Change display name only; id is immutable.
|
|
gbrain sources default <id> Set the brain-level default.
|
|
gbrain sources attach <id> Write .gbrain-source in CWD (like kubectl context).
|
|
gbrain sources detach Remove .gbrain-source from CWD.
|
|
gbrain sources federate <id>
|
|
gbrain sources unfederate <id>
|
|
```
|
|
|
|
## Citation format for agents
|
|
|
|
When agents receive multi-source results they MUST cite pages in
|
|
`[source-id:slug]` form. Example:
|
|
|
|
> You told me about the distillation protocol — see [wiki:topics/ai]
|
|
> and [gstack:plans/multi-repo] for where this came from.
|
|
|
|
The citation key is `sources.id` (immutable). Renaming a source via
|
|
`gbrain sources rename` changes the display name only; existing
|
|
citations keep working.
|
|
|
|
## Writing to a specific source
|
|
|
|
```bash
|
|
# Pass --source explicitly
|
|
gbrain put-page topics/ai ... --source wiki
|
|
|
|
# Or rely on the dotfile / env / CWD match
|
|
cd ~/.gstack && gbrain put-page plans/multi-repo ...
|
|
# → source auto-resolves to gstack
|
|
```
|
|
|
|
Reads span federated sources by default. Writes require a resolved
|
|
source (explicit, inferred, or default). The resolver never picks a
|
|
source silently when ambiguous — it errors with a clear fix.
|
|
|
|
## Durability: keep a brain repo in sync (auto-harden)
|
|
|
|
A long-lived agent that writes to a knowledge-wiki git repo needs three
|
|
things to never lose work: pull before it edits, push every write, and not
|
|
go stale while it sits idle. `gbrain sources harden` installs all of that,
|
|
idempotently. The moment you add a brain repo with a token, it runs
|
|
automatically:
|
|
|
|
```bash
|
|
# Clone + register a GitHub repo, then auto-harden it for durability.
|
|
# Use a fine-grained PAT scoped to just this repo.
|
|
gbrain sources add wiki --url https://github.com/you/brain-wiki.git --pat-file ~/.secrets/wiki-pat
|
|
# → clones, then installs: local auto-push hook, scripts/brain-commit-push.sh,
|
|
# always-on durability rules in AGENTS.md/RESOLVER.md, a 30-min pull cron,
|
|
# and a repo-scoped credential. Verifies push works before declaring done.
|
|
|
|
# Run the same audit on an existing source any time (idempotent):
|
|
gbrain sources harden wiki --pat-file ~/.secrets/wiki-pat
|
|
|
|
# Pull on demand (the cron calls the --path form, which never opens the DB):
|
|
gbrain sources pull wiki
|
|
|
|
# Remove the durability scaffolding (also runs automatically on `sources remove`):
|
|
gbrain sources unharden wiki
|
|
```
|
|
|
|
What hardening guarantees:
|
|
|
|
- **Pull-first, conflict-safe.** Every pull is a divergence-safe rebase. A
|
|
dirty working tree is skipped (your in-progress edits are never touched); a
|
|
rebase conflict is aborted cleanly and flagged for attention, never left
|
|
half-applied.
|
|
- **Push is never deferred.** `scripts/brain-commit-push.sh "<msg>" <path>`
|
|
commits and pushes atomically and refuses to report success without a
|
|
confirmed push. The post-commit hook is a best-effort background fallback;
|
|
the helper is the guarantee.
|
|
- **No silent staleness.** A 30-minute background pull keeps an idle session
|
|
current. It runs DB-free, so it never contends with a live brain for the
|
|
PGLite single-writer lock.
|
|
|
|
Flags: `--no-cron` skips the scheduled pull, `--no-verify` skips the push
|
|
probe, `--dry-run` reports what would change, `--json` emits a machine
|
|
report, `--all` hardens every source with a remote (same-account only).
|
|
`--no-harden` on `sources add` opts out of auto-harden.
|
|
|
|
Security: the push automation is installed locally per machine (never
|
|
committed into the repo), the token is wired per-repo (an existing
|
|
credential helper is reused when present), and it never appears in the repo,
|
|
the remote URL, logs, or the JSON report. For a self-hosted git server
|
|
reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1`
|
|
(default is HTTPS-only).
|
|
|
|
## Upgrading an existing brain
|
|
|
|
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your
|
|
existing pages all move under `source_id='default'`. Behavior is
|
|
unchanged until you add a second source.
|
|
|
|
To add one:
|
|
|
|
```bash
|
|
gbrain sources add gstack --path ~/.gstack --federated
|
|
cd ~/.gstack && gbrain sources attach gstack && gbrain sync
|
|
```
|
|
|
|
Two commands. The existing default source is untouched.
|
|
|
|
## Not in v0.18.0
|
|
|
|
- Session transcript ingest (`.jsonl`, raised size cap, session
|
|
PageType) — v0.18.
|
|
- Per-source retention/TTL (`gbrain sources prune`) — v0.18.
|
|
- ACL enforcement via caller-identity — v0.17.1.
|
|
- `gbrain sources import-from-github <url>` one-shot bootstrap — patch
|
|
release after the core plumbing stabilizes.
|
|
|
|
All of these build on the `sources` primitive shipped here.
|