mirror of https://github.com/garrytan/gbrain.git synced 2026-07-28 14:59:47 +00:00

Files

T

2934c53c1d fix(sources): validate --path is a git repo with committed content at registration (#2707 ) (#2975 )

* fix(sources): validate --path is a git repo at registration time (#2707)

`sources add --path <dir>` accepted any existing non-git directory with
zero validation, deferring the failure to the first `gbrain sync` ("Not
inside a git repository: ..."). By the time that surfaces, the source
has been silently stale for however long nobody read the sync logs.

Add a registration-time check (git-remote.ts:isInsideGitRepo, mirroring
sync.ts's discoverGitRoot walk-up so subdir-of-git-repo sources still
pass) that rejects an existing-but-non-git --path directory with an
actionable error pointing at `git init && git add -A && git commit`.
Non-existent paths are unaffected (out of scope — different, pre-existing
failure mode) and `--force` opts out for callers who want to register
before git-init exists.

This is registration-time validation ONLY — it never auto-`git init`s
the directory, preserving the consent boundary #2967 established for
sync-time self-heal (a --path source is the user's own external
directory; gbrain must not mutate it without explicit ask).

Also documents the git requirement (docs/guides/multi-source-brains.md),
including the "files must be committed, not just present" gotcha and
that a stale/unreachable sync anchor already self-heals on plain
`gbrain sync` (verified manually against HEAD — no reset-anchor command
needed).

* fix(sources): require a committed HEAD + shell-quote remediation cmd (codex round 1)

Codex review round 1 on #2707 found two real gaps:

1. isInsideGitRepo alone accepts a `git init`ed-but-never-committed
   directory (rev-parse --show-toplevel succeeds with no HEAD), so
   registration would still pass a source that fails sync's own
   "No commits in repo ... Make at least one commit before syncing."
   Add hasGitCommits (git rev-parse HEAD) as a second required check.

2. The remediation command in the error message interpolated the raw
   path unquoted — spaces, $(), backticks, etc. would break or, worse,
   execute unintended shell syntax if pasted. POSIX single-quote it
   (mirrors src/commands/connect.ts:shellQuote; duplicated locally
   rather than imported, since commands/ depends on core/ not the
   reverse).

* fix(sources): require tracked content in HEAD, not just a resolvable HEAD (codex round 2)

Codex review round 2 P1: hasGitCommits (rev-parse HEAD) accepted a repo
with an empty commit (git commit --allow-empty) followed by untracked
files — HEAD resolves fine (to git's well-known empty-tree object), so
registration passed, but the first sync would "succeed" importing
nothing and then silently never notice the untracked files change. The
exact same gap applied to an untracked subdirectory of an otherwise-
real git repo (monorepo case).

Replace hasGitCommits with hasTrackedContent (`git ls-tree HEAD -- .`,
non-recursive — one entry is enough, no need to walk the whole
subtree). `-C path` + pathspec `.` scopes correctly to both a repo
toplevel and a subdirectory-of-a-repo source, and an empty tree lists
zero entries where a bare `rev-parse HEAD` would still succeed. Also
subsumes the "no commits at all" case hasGitCommits covered (ls-tree
on an unborn repo fails the same way), so this is one check instead
of two.

Updated the error copy and docs/guides/multi-source-brains.md to match
what's actually verified now.

* fix(sources): O(1)-output tree-emptiness probe, avoid maxBuffer overflow (codex round 3)

Codex review round 3 found the round-2 `git ls-tree HEAD -- .` listing
buffers the whole (non-recursive) tree — a real repo with ~17-20K
directly-tracked entries exceeds execFileSync's default 1 MiB
maxBuffer, throws ENOBUFS, and the catch-all incorrectly rejects a
perfectly valid registration.

Replace the listing with `git rev-parse --verify HEAD:./` (resolves
the tree object for `path` specifically, correct for both toplevel and
subdirectory sources same as before) compared against git's canonical
empty-tree SHA-1 (4b825dc6...) — a fixed ~40-byte read regardless of
how many entries the tree has, structurally immune to this class of
bug rather than just raising the threshold. Added a 300-file
regression test locking this in.

Declined a second round-3 finding (P1: reject a tree if ANY untracked
file exists anywhere under the path, not just when the tree is
entirely empty) — untracked files never being synced is standard,
existing git-source behavior throughout this codebase (identical for
--url managed clones), not a bug specific to this validation. Enforcing
zero-untracked-files at registration would reject ordinary repos with
gitignored build output, .DS_Store, editor swapfiles, etc. Out of
scope relative to what #2707 actually asks for (a directory with real,
committed content that will sync) and how every other git source in
this system already behaves.

* fix(sources): derive empty-tree OID per repo instead of hardcoding SHA-1 (codex round 4)

Codex review round 4 P2, confirmed by directly testing against a
`git init --object-format=sha256` repo: the hardcoded SHA-1 empty-tree
constant only matches SHA-1 repositories. An empty SHA-256 repo's real
empty-tree OID is a different (64-char) hash, so the SHA-1 comparison
silently mismatched and let an empty/untracked SHA-256 source through
— exactly the case this validation exists to catch.

Replace the constant with `emptyTreeOid()`: `git hash-object -t tree
--stdin < /dev/null` computed in the target repo's own context, so it
returns the correct empty-tree OID for whichever object format that
repo actually uses, without gbrain needing to know or care which one.
Added gated regression tests (git 2.29+ / --object-format=sha256,
test.skipIf on older git) for both the empty-repo-rejected and
real-content-registers-fine cases.

Converging here (4 review rounds; this is the last outstanding
finding from round 4, and round 4 raised only this one issue).

* fix(test): --force the incidental non-git second source in #1434 routing test (#2707)

CI caught a real regression from this PR's registration-time git
validation: test/sync-sole-non-default-routing.test.ts's "2+ non-default
sources" case registers a bare mkdtempSync temp dir (no git init) as a
second source purely to have 2 sources present — the directory's content
was never meant to be exercised, only its existence as a distinct
local_path. #2707's new validation correctly rejects that dir at
registration time, since nothing else in the test suite told it
otherwise.

--force is the right fix, not adding unnecessary git-init/commit
boilerplate to secondRepo: it documents that this specific registration
intentionally doesn't care about git-validity, matching what a real
caller opting into the legacy lenient behavior would do.

Verified: the specific test (3/3 pass), plus every other test file in
the repo using `sources add --path` (sources.test.ts, sources-ops.test.ts
already covered by the PR's own commits; repos-alias.test.ts,
sync-cost-gate.serial.test.ts — 11/11 pass, no similar fixture gap).
typecheck clean, verify 31/31.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-20 16:28:53 -07:00

11 KiB

Raw Permalink Blame History

Multi-source brains

A single gbrain database can hold multiple knowledge repos. Each one is a source: a logical brain-within-the-brain with its own slug namespace, its own sync state, and its own federation policy. The rest of this guide walks the three canonical scenarios.

The three scenarios

1. Unified knowledge recall (wiki + gstack)

You have a personal wiki and a gstack checkout. Both belong to you, both are knowledge you want your agent to recall across. When you ask "what did I learn about X?" you want the best hit whether it lives in the wiki or in a gstack plan.

# Register the gstack source, federate so it joins cross-source search
gbrain sources add gstack --path ~/.gstack --federated

# Pin the directory so `gbrain sync` knows which source it's walking
cd ~/.gstack && gbrain sources attach gstack

# Initial sync
gbrain sync --source gstack

# Now `gbrain search "retry budgets"` returns hits from BOTH wiki and
# gstack. Each result includes source_id so the agent can cite properly.

Result: wiki pages and gstack plans are separate (different source_ids, different slug namespaces) but share the search surface.

2. Purpose-separated brains (yc-media + garrys-list)

You run two completely different content pipelines on the same backend. YC Media covers portfolio news and founder profiles. Garry's List is personal writing. You explicitly DON'T want them mixed in search — YC portfolio content leaking into essay searches is a bug, not a feature.

# Two sources, both isolated (federated=false)
gbrain sources add yc-media --path ~/yc-media --no-federated
gbrain sources add garrys-list --path ~/writing --no-federated

# Pin each checkout directory
(cd ~/yc-media && gbrain sources attach yc-media)
(cd ~/writing && gbrain sources attach garrys-list)

# Sync each independently
gbrain sync --source yc-media
gbrain sync --source garrys-list

Result: searching from neither directory returns the default source (your main brain). Searching from inside ~/yc-media returns only yc- media hits. Searching from inside ~/writing returns only garrys-list. Federation is opt-in, not leaked.

To search across them explicitly on demand:

gbrain search "tech layoffs" --source yc-media,garrys-list

3. Mixed (wiki federated + sessions isolated)

Your main wiki is federated with a few trusted sources. Your session transcripts (coming in v0.18) land in a separate isolated source so they don't dominate every search result.

# Federated sources
gbrain sources add gstack --path ~/.gstack --federated

# Isolated source (future v0.18 — sessions use this shape today for ingest)
gbrain sources add sessions --path ~/.claude/sessions --no-federated

Resolution priority

When any command needs to pick a source, gbrain walks this list (highest first):

Explicit --source <id> flag.
GBRAIN_SOURCE environment variable.
.gbrain-source dotfile in CWD or any ancestor directory.
A registered source whose local_path contains the CWD (longest prefix wins for nested checkouts).
The brain-level default set via gbrain sources default <id>.
The seeded default source.

So inside ~/.gstack/plans/ on a brain that pinned gstack to ~/.gstack via .gbrain-source, gbrain put-page implicitly writes to the gstack source. Outside any registered directory with no env/dotfile set, it writes to the default.

Federation flag

Every source row stores config.federated: boolean in its JSONB config.

Value	Meaning
`true`	Source participates in unqualified `gbrain search "X"` results.
`false` (default for new sources)	Source only searched when explicitly named via `--source <id>` or qualified citation.

The seeded default source is federated=true so pre-v0.17 brains behave exactly as before — every page appears in search.

Flip later with gbrain sources federate <id> / unfederate <id>.

Commands

Full subcommand reference:

gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
                               Register a source. id: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
                               --path must be a git repo (or a subdirectory of one) — see
                               "The git requirement for --path sources" below. --force
                               skips that check to register before git-init exists.
gbrain sources list [--json]   List all sources with page counts + federation state.
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
                               Cascade-delete a source (pages, chunks, timeline).
gbrain sources rename <id> <new-name>
                               Change display name only; id is immutable.
gbrain sources default <id>    Set the brain-level default.
gbrain sources attach <id>     Write .gbrain-source in CWD (like kubectl context).
gbrain sources detach          Remove .gbrain-source from CWD.
gbrain sources federate <id>
gbrain sources unfederate <id>

The git requirement for --path sources

Every --path source must be a git repository (or live inside one — a subdirectory of a git repo works too) with at least one committed, tracked file under that path. gbrain sources add validates this at registration time and refuses a directory that doesn't qualify — no .git at all, a git init with no commit yet, or a commit made before git add — with an actionable error instead of silently registering a source that will fail (or worse, "succeed" while importing nothing) on its first gbrain sync. Fix it with:

git -C <path> init
git -C <path> add -A
git -C <path> commit -m "initial import"
gbrain sources add <id> --path <path>

Two details that are easy to miss:

Files must actually be committed, not just present. The sync walker reads files through git objects, so git init alone — even followed by an empty commit (git commit --allow-empty) — isn't enough. Registration checks for real tracked content (git ls-tree HEAD scoped to the path), not just a resolvable HEAD, so this footgun is caught immediately instead of surfacing later as a sync that imports nothing.
--force registers the source anyway, skipping the check. Use this if you're registering a path before an automated pipeline gets around to git init-ing it. GBrain never auto-git inits a --path source for you — it's your directory, not a gbrain-managed clone (same consent boundary as sync-time self-heal, which also never mutates a --path source without an explicit ask).

If sync ever reports a problem with the sync anchor (last_commit) — after a force-push, a history rewrite, or a from-scratch git init on a directory that was synced before — you do not need to reset anything by hand. gbrain sync detects an unreachable or non-ancestor anchor automatically and recovers: either a full reimport (anchor object missing) or a direct tree-to-tree diff against the orphaned bookmark (anchor present but rewritten), advancing the anchor to the new HEAD when it completes.

Citation format for agents

When agents receive multi-source results they MUST cite pages in [source-id:slug] form. Example:

You told me about the distillation protocol — see [wiki:topics/ai] and [gstack:plans/multi-repo] for where this came from.

The citation key is sources.id (immutable). Renaming a source via gbrain sources rename changes the display name only; existing citations keep working.

Writing to a specific source

# Pass --source explicitly
gbrain put-page topics/ai ... --source wiki

# Or rely on the dotfile / env / CWD match
cd ~/.gstack && gbrain put-page plans/multi-repo ...
# → source auto-resolves to gstack

Reads span federated sources by default. Writes require a resolved source (explicit, inferred, or default). The resolver never picks a source silently when ambiguous — it errors with a clear fix.

Durability: keep a brain repo in sync (auto-harden)

A long-lived agent that writes to a knowledge-wiki git repo needs three things to never lose work: pull before it edits, push every write, and not go stale while it sits idle. gbrain sources harden installs all of that, idempotently. The moment you add a brain repo with a token, it runs automatically:

# Clone + register a GitHub repo, then auto-harden it for durability.
# Use a fine-grained PAT scoped to just this repo.
gbrain sources add wiki --url https://github.com/you/brain-wiki.git --pat-file ~/.secrets/wiki-pat
#   → clones, then installs: local auto-push hook, scripts/brain-commit-push.sh,
#     always-on durability rules in AGENTS.md/RESOLVER.md, a 30-min pull cron,
#     and a repo-scoped credential. Verifies push works before declaring done.

# Run the same audit on an existing source any time (idempotent):
gbrain sources harden wiki --pat-file ~/.secrets/wiki-pat

# Pull on demand (the cron calls the --path form, which never opens the DB):
gbrain sources pull wiki

# Remove the durability scaffolding (also runs automatically on `sources remove`):
gbrain sources unharden wiki

What hardening guarantees:

Pull-first, conflict-safe. Every pull is a divergence-safe rebase. A dirty working tree is skipped (your in-progress edits are never touched); a rebase conflict is aborted cleanly and flagged for attention, never left half-applied.
Push is never deferred. scripts/brain-commit-push.sh "<msg>" <path> commits and pushes atomically and refuses to report success without a confirmed push. The post-commit hook is a best-effort background fallback; the helper is the guarantee.
No silent staleness. A 30-minute background pull keeps an idle session current. It runs DB-free, so it never contends with a live brain for the PGLite single-writer lock.

Flags: --no-cron skips the scheduled pull, --no-verify skips the push probe, --dry-run reports what would change, --json emits a machine report, --all hardens every source with a remote (same-account only). --no-harden on sources add opts out of auto-harden.

Security: the push automation is installed locally per machine (never committed into the repo), the token is wired per-repo (an existing credential helper is reused when present), and it never appears in the repo, the remote URL, logs, or the JSON report. For a self-hosted git server reachable only over a filesystem path, set GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1 (default is HTTPS-only).

Upgrading an existing brain

gbrain upgrade runs the v16 + v17 migrations automatically. Your existing pages all move under source_id='default'. Behavior is unchanged until you add a second source.

To add one:

gbrain sources add gstack --path ~/.gstack --federated
cd ~/.gstack && gbrain sources attach gstack && gbrain sync

Two commands. The existing default source is untouched.

Not in v0.18.0

Session transcript ingest (.jsonl, raised size cap, session PageType) — v0.18.
Per-source retention/TTL (gbrain sources prune) — v0.18.
ACL enforcement via caller-identity — v0.17.1.
gbrain sources import-from-github <url> one-shot bootstrap — patch release after the core plumbing stabilizes.

All of these build on the sources primitive shipped here.

11 KiB Raw Permalink Blame History