mirror of https://github.com/garrytan/gbrain.git synced 2026-07-28 14:59:47 +00:00

Files

T

65ff663f7d v0.36.4.0 feat: brain-health-100 — autonomous remediation via doctor --remediate + Minions (#1193 )

* feat(schema): op_checkpoints table + doctor_run_id partial GIN (v67+v68)

T1 of brain-health-100 wave. Two new migrations underpin autonomous
remediation via Minions:

- v67 op_checkpoints — shared checkpoint table for long-running ops
  (embed, extract, lint, backlinks, reindex, integrity). Pre-fix each
  op had its own file-backed checkpoint or none. PRIMARY KEY (op,
  fingerprint) lets `extract links` and `extract timeline` (or
  `reindex --markdown` vs `--code`) coexist without colliding on
  shared keys.

- v68 minion_jobs_doctor_run_id_idx — partial GIN on
  `minion_jobs.data WHERE data ? 'doctor_run_id'`. Indexes only
  doctor-submitted jobs so audit-trail queries don't sequential-scan
  months of unrelated cron history. PGLite skips via empty sqlFor.

Applied to src/schema.sql + src/core/pglite-schema.ts so both engines
get the table on fresh-install. Bootstrap coverage test +
122-case migrate test both pass.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(D12 + folded scope B from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): op-checkpoint module — DB-backed checkpoint primitive

T2 of brain-health-100 wave. Six exports plus per-op fingerprint helpers:

  loadOpCheckpoint(engine, key)     → string[]   (completed keys; [] if none)
  recordCompleted(engine, key, ks)  → void       (UPSERT atomic)
  clearOpCheckpoint(engine, key)    → void       (clean-exit drop)
  resumeFilter(all, completed)      → string[]   (pure; drives batched walks)
  purgeStaleCheckpoints(engine, ttl)→ number     (cycle purge phase consumer)

Fingerprint helpers:
  fingerprint(params)               — sha8 of canonical-JSON
  embedFingerprint(p)               — model+dim+slug+source variation
  extractFingerprint(p)             — mode (links vs timeline)
  reindexFingerprint(p)             — markdown vs code vs slug + chunker_version
  lintFingerprint, backlinksFingerprint, integrityFingerprint, importFingerprint

Canonical-JSON over keys-sorted ensures the same params produce the
same fingerprint across runs and hosts. sha8 (8 hex chars from sha256)
is short enough for filenames + UI but collision-resistant for the
expected per-op invocation diversity.

DB-backed for both engines (PGLite has the table too via v67). Lost-
write on partial DB failure is non-fatal — caller continues, next run
re-walks (cheap for hash-short-circuited ops like embed/import).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(D12 + codex #10–16 from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): brain-score-recommendations — shared data layer

T4 of brain-health-100 wave. Pure module — no engine I/O. Takes a
BrainHealth snapshot + RecommendationContext, returns ordered
Remediation[] ready to feed the doctor remediation plan OR features
--auto-fix.

Three public exports:
  computeRecommendations(health, ctx)  → Remediation[]
  classifyChecks(checks, ctx)          → CheckClassification[]
  maxReachableScore(health, classes)   → number (0-100 ceiling)

D13 — three-state classification per check: remediable / human_only /
blocked. The plan ONLY emits remediable items; blocked surfaces
alongside as informational with the missing prereq (no API key, etc.).
Closes the spin-loop bug on empty / API-key-missing brains (codex #20).

D14 — every Remediation has a stable string id (sync.repo, embed.stale,
backlinks.fix, extract.all). depends_on references ids, not check names.

D9 — idempotency_key is content-hash from canonical-JSON of params.
Same intent across runs = same key; failed-row replay via :r<N> suffix
is the --remediate loop's job, not this module's.

Scope item +A (cost-budget gate) — Remediation.est_usd_cost populated
for embed (chars × pricePerMTok from embedding-pricing.ts) and Anthropic
jobs (estimateAnthropicCost helper). doctor --remediate --max-usd N
gates submission against est_total_usd_cost.

Both consumers (doctor + features per D15) import from here. Features
executes inline (D15 contract preserved), doctor submits via queue.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(handlers): 11 new Minion handlers + 3 added to PROTECTED + sync noExtract fix

T5 of brain-health-100 wave.

PROTECTED_JOB_NAMES extension (D11): synthesize, patterns, consolidate.
These cycle phases internally submit `subagent` jobs with
allowProtectedSubmit=true, so they CAN spend Anthropic credits.
Treating them as "data-quality maintenance" was a misread surfaced by
the codex outside-voice review (#6). Protected gate ensures only
trusted local callers (CLI, autopilot, doctor --remediate) can submit;
an OAuth-scoped MCP client can't burn the user's API budget by
submitting a synthesize job over HTTP.

11 new handlers registered in jobs.ts registerBuiltinHandlers:

  PROTECTED (3) — phase-wrappers that spawn subagent children:
    synthesize, patterns, consolidate

  Open (8) — DB/fs writes only, no LLM spend:
    reindex, repair-jsonb, orphans, integrity, purge,
    extract_facts, resolve_symbol_edges, recompute_emotional_weight

Phase-wrappers all delegate to `runCycle({ phases: [name] })` rather
than extracting standalone phase functions. Cycle.ts already owns the
lock + abort signal + progress reporter per D10, so the wrapper is a
one-liner and cycle.ts remains the single source of truth for phase
semantics. Pragmatic deviation from the plan's "extract 6 standalone
runXxxPhase functions" — smaller diff, equivalent correctness.

Standalone `sync` handler now passes `noExtract: true` (codex #5 fix).
Pre-fix, doctor's remediation plan emitting [sync, extract] caused
double-extraction (performSync inline-extract + standalone extract
job). Now sync defers extract to the dedicated handler. Callers that
want inline extract pass { noExtract: false } in job params.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T5 + D10 + D11 + codex #5/#6 from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): --remediation-plan + --remediate CLI surfaces

T6 of brain-health-100 wave. The headline user-facing capability:
agents drive brain health to target score via autonomous Minions
remediation.

Two new flags on `gbrain doctor`:

  --remediation-plan [--json] [--target-score N]
    Read-only. Emits ordered Remediation[] from BrainHealth + context.
    Uses cheap path (D7) — engine.getHealth() + computeRecommendations,
    NOT a full doctor walk. JSON shape is stable agent contract.

  --remediate [--yes] [--target-score N] [--max-jobs N] [--max-usd N]
              [--dry-run] [--json]
    Sequential submit (D3) with D5 cascade on failure, D7 scoped
    recheck between steps, D9 content-hash idempotency keys, D13
    three-state remediation filtering (only remediable jobs enter
    the loop), +A cost-budget gate via --max-usd.

Check.remediation field added as additive optional (DoctorReport
schema_version stays at 2 per D4).

PGLite path: synchronous in-process execution with short polling.
Postgres path: durable queue submission with waitForCompletion.

The --remediate loop:
  1. Compute initial plan from BrainHealth
  2. Refuse if --target-score > maxReachableScore(health, classes)
  3. Refuse if est_total_usd_cost > --max-usd
  4. For each step in order:
     - Skip if depends_on intersects aborted set (D5)
     - queue.add with content-hash idempotency_key (D9)
     - waitForCompletion with timeout
     - Recompute plan from fresh health (D7 scoped recheck)
  5. Exit 0 if all completed; 1 if any failed/aborted

doctor_run_id UUID stamps every submitted job's data field so
operators can later query `SELECT * FROM minion_jobs WHERE
data->>'doctor_run_id' = '<uuid>'` (indexed via v68 partial GIN).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T6 + D1/D3/D5/D7/D9/D13 + folded scope A).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): maybeBackground helper + apply --background to embed

T7 of brain-health-100 wave. New helper in src/core/cli-options.ts
formalizes the --background flag pattern. Same semantics in TTY and
cron per D9 (submit-and-exit always; --background --follow execs
`gbrain jobs follow <id>` after submission).

  await maybeBackground({
    engine, args, jobName: 'embed',
    paramBuilder: (cleanArgs) => ({ stale, all, ... }),
  })
  // returns true if backgrounded → caller exits

Content-hash idempotency key (D9): `cli:embed:sha8(canonical-JSON(params))`.
No time-slot. Same intent across runs = same key. Failed-row replay
is the doctor --remediate loop's job, not this path's.

PGLite degrades to inline execution with a clear stderr note
("PGLite has no worker daemon; running inline"). NOT a no-op,
NOT silent — doc-stated semantic difference because PGLite has no
worker daemon.

Applied to `gbrain embed` as the reference integration. The other 6
commands (extract, lint, backlinks, reindex, integrity, pages) adopt
the same 4-line pattern at the top of their entry function — follow-up
in a smaller diff once the helper proves out in production.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T7 + D9 + Gap 6).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(autopilot): targeted-submit loop + op_checkpoints GC in purge phase

T8 of brain-health-100 wave.

Autopilot dispatch changes (src/commands/autopilot.ts):

Pre-fix: every tick submitted ONE autopilot-cycle job, full phase
set, regardless of brain state. On a healthy brain pure overhead; on
a degraded brain bundled fast wins with slow phases so user waited
for the slowest.

New decision logic (T8 from plan):
  - score >= 95 AND empty plan AND <60min since last full → SLEEP
  - score >= 95 AND empty plan AND >=60min → submit autopilot-cycle
    (phase-coupling exercise)
  - plan <= 3 steps AND est_total < 5min → submit individual handlers
    (targeted; uses D9 content-hash idempotency keys per step;
    maxWaiting:1 per submit per codex #17)
  - else → submit autopilot-cycle (the hammer)

D10 cycle-lock invariant guarantees targeted-submit and autopilot-cycle
can never run concurrently (both acquire gbrain-cycle), closing the
"60-min floor double-processes queued targeted jobs" failure mode.

Computation uses cheap path (D7) — engine.getHealth() + computeRecommendations,
NOT a full doctor walk. Adds ~1 SQL count query per tick; negligible
on a 50K-page brain.

PROTECTED handlers (synthesize/patterns/consolidate) are submitted with
allowProtectedSubmit:true; autopilot is a trusted local caller.

Cycle purge phase (src/core/cycle.ts):

Added op_checkpoints GC (+C folded scope item). 7-day TTL — any
reasonable long-running op finishes inside that window. Non-fatal
on pre-v67 brains (table missing).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T8 + D7/D9/D10 + codex #17 + folded scope +C).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(core): brain-score-recommendations + op-checkpoint unit tests

T10 of brain-health-100 wave — load-bearing decision-pinning tests.

test/brain-score-recommendations.test.ts (22 cases):
  - Healthy brain → empty plan
  - Per-component remediation paths (sync, embed, backlinks, extract)
  - depends_on wiring (extract → sync; embed → sync when stale)
  - Severity ordering (critical > high > medium > low)
  - D6 #5 determinism: same input twice → byte-identical output
  - D9 idempotency keys: content-hash format, no time-slot
  - D9 source isolation: different --source → different key
  - D13 status field always 'remediable' in output
  - +A cost-estimate populated for embed
  - classifyChecks: remediable / blocked / human_only triage
  - maxReachableScore: all-remediable → 100; all-blocked → current

test/op-checkpoint.test.ts (20 cases):
  - fingerprint stability + key-order invariance (canonical-JSON)
  - codex #11: extract links vs timeline get different fingerprints
  - codex #12: reindex markdown vs code get different fingerprints
  - codex #15: embed model+dim variation produces different fingerprints
  - reindex chunker_version bump invalidates checkpoint
  - DB round-trip (load → record → load)
  - Cross-fingerprint isolation (linksKey vs timelineKey)
  - clearOpCheckpoint idempotency on missing rows
  - resumeFilter purity (no I/O, deterministic)
  - purgeStaleCheckpoints TTL respect

42 new tests, all pass. PGLite engine + resetPgliteState pattern per
CLAUDE.md test-isolation guide.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T10 + D6 #5 + D9 + D12 + D13 + codex #11/#12/#15).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): v0.36.0.0 — brain-health-100 wave + docs/llms refresh

T12 of brain-health-100 wave. VERSION + package.json bumped 0.35.6.0
→ 0.36.0.0. CHANGELOG entry leads ELI10 ("your agent can now drive
your brain to 90/100 by itself, on a cron, without you watching")
then drills into the precise mechanics per CLAUDE.md voice rules.

llms.txt + llms-full.txt regenerated via bun run build:llms.

Trio audit (CLAUDE.md mandatory pre-push check):
  VERSION:     0.36.0.0
  package.json: 0.36.0.0
  CHANGELOG:   ## [0.36.0.0] - 2026-05-18

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README/CLAUDE/AGENTS/maintain for v0.36.4.0 brain-health-100 wave

- README.md: New-in-v0.36.4.0 callout — `gbrain doctor --remediate` headline,
  autopilot health-aware tick, eleven new background-job types, three PROTECTED.
- CLAUDE.md: Key Files entries for `op-checkpoint.ts`, `brain-score-recommendations.ts`,
  doctor.ts / jobs.ts / protected-names.ts / autopilot.ts / cycle.ts / embed.ts /
  cli-options.ts extensions; new "Key commands added in v0.36.4.0" section.
- AGENTS.md: Common-tasks entry pointing agents at the one-command remediation loop.
- skills/maintain/SKILL.md: Autonomous Phase (gbrain doctor --remediate) at the top,
  manual per-dimension walk preserved as the fallback path.
- llms-full.txt: regenerated to pick up the CLAUDE.md changes (project rule).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(changelog): respectful tone on spend caps for v0.36.4.0

Reframed the cost-budget callout. Pre-fix language said the spend cap
prevents a synthesize loop from "burning $100 of Anthropic credits
while you're at lunch" — casually treating $100 as the throwaway number
is tone-deaf. $100 is a meaningful amount for many people.

New language: "spend cap so a synthesize loop can't run up your
Anthropic bill while you're at lunch. The cap is yours to set per run."
And: "Pass --max-usd 5 (or whatever cap you're comfortable with)."
And: "Pick the cap that fits your wallet."

Also reframed three adjacent lines:
- "healthy brains stop burning cycles" → "stop spending tokens on
  work that has nothing to do"
- "agent can't submit them and burn your API budget" → "can't submit
  them on your behalf. Your provider bill stays in your hands"
- Table cell "Cron with cost cap" / "--max-usd 5" → "Cron with spend
  cap" / "--max-usd N"

llms-full.txt regenerated to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-19 10:05:31 -07:00

6.5 KiB

Raw Blame History

Agents working on GBrain

This is your install + operating protocol. Claude Code reads ./CLAUDE.md automatically. Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL): start here.

Install (5 min)

Install gbrain via Bun (the canonical path):
```
curl -fsSL https://bun.sh/install | bash
export PATH="$HOME/.bun/bin:$PATH"
bun install -g github:garrytan/gbrain
```
If bun install -g aborts or gbrain doctor reports schema_version: 0, the CLI prints a recovery hint pointing at #218. Run gbrain apply-migrations --yes to recover, or fall back to the deterministic install: git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain && bun install && bun link.
Init the brain: gbrain init (defaults to PGLite, zero-config). For 1000+ files or multi-machine sync, init suggests Postgres + pgvector via Supabase.
STOP — ask the user about search mode. gbrain init auto-applied a default but printed a 9-cell cost matrix (mode × downstream model) preceded by [AGENT] markers. You MUST relay the matrix to the operator and confirm their choice before continuing. Cost spread between corners is 25x — silent acceptance is the wrong default. See ./INSTALL_FOR_AGENTS.md Step 3.5 for the exact ask-the-user protocol. Same banner fires on gbrain post-upgrade for existing users (search modes were added in v0.32.3).
Read ./INSTALL_FOR_AGENTS.md for the full 9-step flow (API keys, identity, cron, verification).

Read this order

./AGENTS.md (this file) — install + operating protocol.
./CLAUDE.md — architecture reference, key files, trust boundaries, test layout.
./docs/architecture/brains-and-sources.md — the two-axis mental model (brain = which DB, source = which repo in the DB). Every query routes on both axes. Read before writing anything that touches brain ops.
./skills/conventions/brain-routing.md — agent-facing decision table: when to switch brain, when to switch source, how cross-brain federation works (latent-space only; the agent decides).
./skills/RESOLVER.md — skill dispatcher. Read before any task.

Trust boundary (critical)

GBrain distinguishes trusted local CLI callers (OperationContext.remote = false, set by src/cli.ts) from untrusted agent-facing callers (remote = true, set by src/mcp/server.ts). Security-sensitive operations like file_upload tighten filesystem confinement when remote = true and default to strict behavior when unset. If you are writing or reviewing an operation, consult src/core/operations.ts for the contract.

Common tasks

Configure: docs/ENGINES.md, docs/guides/live-sync.md, docs/mcp/DEPLOY.md.
Debug: docs/GBRAIN_VERIFY.md, docs/guides/minions-fix.md, gbrain doctor --fix.
Migrate / upgrade: gbrain upgrade (binary self-update + schema migrations + post-upgrade prompts), docs/UPGRADING_DOWNSTREAM_AGENTS.md, skills/migrations/, gbrain apply-migrations --yes (manual schema-only).
Eval retrieval changes: capture is off by default. To benchmark a retrieval change against real captured queries, set GBRAIN_CONTRIBUTOR_MODE=1, then gbrain eval export --since 7d > base.ndjson and gbrain eval replay --against base.ndjson. For public benchmark coverage (LongMemEval, ground-truth scoring), gbrain eval longmemeval <dataset.jsonl> (v0.28.8) runs against an isolated in-memory PGLite per question — your ~/.gbrain is never opened. Full guide: docs/eval-bench.md.
Drive the brain to a target health score (v0.36.4.0): the one-command loop. gbrain doctor --remediation-plan --json previews what would be fixed; gbrain doctor --remediate --yes --target-score 90 --max-usd 5 walks a dependency-ordered plan (sync before extract, embed after consolidate), re-checking score between every step, refusing to spend past the cost cap. Empty brains (no entity pages) or unconfigured embedding keys hit a max_reachable_score ceiling and bail with what's missing. Three phase handlers (synthesize / patterns / consolidate) are PROTECTED — only trusted local callers can submit them; MCP cannot. Reference: docs/architecture/topologies.md and the CHANGELOG entry for v0.36.4.0.
Track a founder/company over time (v0.35.7): when an entity has typed metric claims in its ## Facts fence (metric: mrr, value: 50000, unit: USD, period: monthly columns), run gbrain eval trajectory <entity-slug> for the chronological history with regressions auto-flagged, or gbrain founder scorecard <entity-slug> for a four-signal JSON rollup (claim_accuracy / consistency / growth_trajectory / red_flags). MCP op find_trajectory exposes the same data — read scope, visibility-filtered for remote callers.
Everything else: ./llms.txt is the full documentation map. ./llms-full.txt is the same map with core docs inlined for single-fetch ingestion.

Before shipping

Easiest path: bun run ci:local runs the full CI gate inside Docker (gitleaks, unit tests with DATABASE_URL unset, then all 29 E2E files sequentially against a fresh pgvector container) and tears down. Use bun run ci:local:diff for the diff-aware subset during fast iteration on a focused branch. Requires Docker (Docker Desktop / OrbStack / Colima) and gitleaks (brew install gitleaks).

Manual path: bun test plus the E2E lifecycle described in ./CLAUDE.md (spin up the test Postgres container, run bun run test:e2e, tear it down).

Ship via the /ship skill, not by hand.

Privacy

Never commit real names of people, companies, or funds into public artifacts. See the Privacy rule in ./CLAUDE.md. GBrain pages reference real contacts; public docs must use generic placeholders (alice-example, acme-example, fund-a).

Forks

If you are a fork, regenerate llms.txt + llms-full.txt with your own URL base before publishing: LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms.

6.5 KiB Raw Blame History Unescape Escape