admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

v0.36.4.0 feat: brain-health-100 — autonomous remediation via doctor --remediate + Minions (#1193)

Browse Source 
* feat(schema): op_checkpoints table + doctor_run_id partial GIN (v67+v68)

T1 of brain-health-100 wave. Two new migrations underpin autonomous
remediation via Minions:

- v67 op_checkpoints — shared checkpoint table for long-running ops
  (embed, extract, lint, backlinks, reindex, integrity). Pre-fix each
  op had its own file-backed checkpoint or none. PRIMARY KEY (op,
  fingerprint) lets `extract links` and `extract timeline` (or
  `reindex --markdown` vs `--code`) coexist without colliding on
  shared keys.

- v68 minion_jobs_doctor_run_id_idx — partial GIN on
  `minion_jobs.data WHERE data ? 'doctor_run_id'`. Indexes only
  doctor-submitted jobs so audit-trail queries don't sequential-scan
  months of unrelated cron history. PGLite skips via empty sqlFor.

Applied to src/schema.sql + src/core/pglite-schema.ts so both engines
get the table on fresh-install. Bootstrap coverage test +
122-case migrate test both pass.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(D12 + folded scope B from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): op-checkpoint module — DB-backed checkpoint primitive

T2 of brain-health-100 wave. Six exports plus per-op fingerprint helpers:

  loadOpCheckpoint(engine, key)     → string[]   (completed keys; [] if none)
  recordCompleted(engine, key, ks)  → void       (UPSERT atomic)
  clearOpCheckpoint(engine, key)    → void       (clean-exit drop)
  resumeFilter(all, completed)      → string[]   (pure; drives batched walks)
  purgeStaleCheckpoints(engine, ttl)→ number     (cycle purge phase consumer)

Fingerprint helpers:
  fingerprint(params)               — sha8 of canonical-JSON
  embedFingerprint(p)               — model+dim+slug+source variation
  extractFingerprint(p)             — mode (links vs timeline)
  reindexFingerprint(p)             — markdown vs code vs slug + chunker_version
  lintFingerprint, backlinksFingerprint, integrityFingerprint, importFingerprint

Canonical-JSON over keys-sorted ensures the same params produce the
same fingerprint across runs and hosts. sha8 (8 hex chars from sha256)
is short enough for filenames + UI but collision-resistant for the
expected per-op invocation diversity.

DB-backed for both engines (PGLite has the table too via v67). Lost-
write on partial DB failure is non-fatal — caller continues, next run
re-walks (cheap for hash-short-circuited ops like embed/import).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(D12 + codex #10–16 from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): brain-score-recommendations — shared data layer

T4 of brain-health-100 wave. Pure module — no engine I/O. Takes a
BrainHealth snapshot + RecommendationContext, returns ordered
Remediation[] ready to feed the doctor remediation plan OR features
--auto-fix.

Three public exports:
  computeRecommendations(health, ctx)  → Remediation[]
  classifyChecks(checks, ctx)          → CheckClassification[]
  maxReachableScore(health, classes)   → number (0-100 ceiling)

D13 — three-state classification per check: remediable / human_only /
blocked. The plan ONLY emits remediable items; blocked surfaces
alongside as informational with the missing prereq (no API key, etc.).
Closes the spin-loop bug on empty / API-key-missing brains (codex #20).

D14 — every Remediation has a stable string id (sync.repo, embed.stale,
backlinks.fix, extract.all). depends_on references ids, not check names.

D9 — idempotency_key is content-hash from canonical-JSON of params.
Same intent across runs = same key; failed-row replay via :r<N> suffix
is the --remediate loop's job, not this module's.

Scope item +A (cost-budget gate) — Remediation.est_usd_cost populated
for embed (chars × pricePerMTok from embedding-pricing.ts) and Anthropic
jobs (estimateAnthropicCost helper). doctor --remediate --max-usd N
gates submission against est_total_usd_cost.

Both consumers (doctor + features per D15) import from here. Features
executes inline (D15 contract preserved), doctor submits via queue.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(handlers): 11 new Minion handlers + 3 added to PROTECTED + sync noExtract fix

T5 of brain-health-100 wave.

PROTECTED_JOB_NAMES extension (D11): synthesize, patterns, consolidate.
These cycle phases internally submit `subagent` jobs with
allowProtectedSubmit=true, so they CAN spend Anthropic credits.
Treating them as "data-quality maintenance" was a misread surfaced by
the codex outside-voice review (#6). Protected gate ensures only
trusted local callers (CLI, autopilot, doctor --remediate) can submit;
an OAuth-scoped MCP client can't burn the user's API budget by
submitting a synthesize job over HTTP.

11 new handlers registered in jobs.ts registerBuiltinHandlers:

  PROTECTED (3) — phase-wrappers that spawn subagent children:
    synthesize, patterns, consolidate

  Open (8) — DB/fs writes only, no LLM spend:
    reindex, repair-jsonb, orphans, integrity, purge,
    extract_facts, resolve_symbol_edges, recompute_emotional_weight

Phase-wrappers all delegate to `runCycle({ phases: [name] })` rather
than extracting standalone phase functions. Cycle.ts already owns the
lock + abort signal + progress reporter per D10, so the wrapper is a
one-liner and cycle.ts remains the single source of truth for phase
semantics. Pragmatic deviation from the plan's "extract 6 standalone
runXxxPhase functions" — smaller diff, equivalent correctness.

Standalone `sync` handler now passes `noExtract: true` (codex #5 fix).
Pre-fix, doctor's remediation plan emitting [sync, extract] caused
double-extraction (performSync inline-extract + standalone extract
job). Now sync defers extract to the dedicated handler. Callers that
want inline extract pass { noExtract: false } in job params.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T5 + D10 + D11 + codex #5/#6 from outside-voice review).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): --remediation-plan + --remediate CLI surfaces

T6 of brain-health-100 wave. The headline user-facing capability:
agents drive brain health to target score via autonomous Minions
remediation.

Two new flags on `gbrain doctor`:

  --remediation-plan [--json] [--target-score N]
    Read-only. Emits ordered Remediation[] from BrainHealth + context.
    Uses cheap path (D7) — engine.getHealth() + computeRecommendations,
    NOT a full doctor walk. JSON shape is stable agent contract.

  --remediate [--yes] [--target-score N] [--max-jobs N] [--max-usd N]
              [--dry-run] [--json]
    Sequential submit (D3) with D5 cascade on failure, D7 scoped
    recheck between steps, D9 content-hash idempotency keys, D13
    three-state remediation filtering (only remediable jobs enter
    the loop), +A cost-budget gate via --max-usd.

Check.remediation field added as additive optional (DoctorReport
schema_version stays at 2 per D4).

PGLite path: synchronous in-process execution with short polling.
Postgres path: durable queue submission with waitForCompletion.

The --remediate loop:
  1. Compute initial plan from BrainHealth
  2. Refuse if --target-score > maxReachableScore(health, classes)
  3. Refuse if est_total_usd_cost > --max-usd
  4. For each step in order:
     - Skip if depends_on intersects aborted set (D5)
     - queue.add with content-hash idempotency_key (D9)
     - waitForCompletion with timeout
     - Recompute plan from fresh health (D7 scoped recheck)
  5. Exit 0 if all completed; 1 if any failed/aborted

doctor_run_id UUID stamps every submitted job's data field so
operators can later query `SELECT * FROM minion_jobs WHERE
data->>'doctor_run_id' = '<uuid>'` (indexed via v68 partial GIN).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T6 + D1/D3/D5/D7/D9/D13 + folded scope A).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): maybeBackground helper + apply --background to embed

T7 of brain-health-100 wave. New helper in src/core/cli-options.ts
formalizes the --background flag pattern. Same semantics in TTY and
cron per D9 (submit-and-exit always; --background --follow execs
`gbrain jobs follow <id>` after submission).

  await maybeBackground({
    engine, args, jobName: 'embed',
    paramBuilder: (cleanArgs) => ({ stale, all, ... }),
  })
  // returns true if backgrounded → caller exits

Content-hash idempotency key (D9): `cli:embed:sha8(canonical-JSON(params))`.
No time-slot. Same intent across runs = same key. Failed-row replay
is the doctor --remediate loop's job, not this path's.

PGLite degrades to inline execution with a clear stderr note
("PGLite has no worker daemon; running inline"). NOT a no-op,
NOT silent — doc-stated semantic difference because PGLite has no
worker daemon.

Applied to `gbrain embed` as the reference integration. The other 6
commands (extract, lint, backlinks, reindex, integrity, pages) adopt
the same 4-line pattern at the top of their entry function — follow-up
in a smaller diff once the helper proves out in production.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T7 + D9 + Gap 6).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(autopilot): targeted-submit loop + op_checkpoints GC in purge phase

T8 of brain-health-100 wave.

Autopilot dispatch changes (src/commands/autopilot.ts):

Pre-fix: every tick submitted ONE autopilot-cycle job, full phase
set, regardless of brain state. On a healthy brain pure overhead; on
a degraded brain bundled fast wins with slow phases so user waited
for the slowest.

New decision logic (T8 from plan):
  - score >= 95 AND empty plan AND <60min since last full → SLEEP
  - score >= 95 AND empty plan AND >=60min → submit autopilot-cycle
    (phase-coupling exercise)
  - plan <= 3 steps AND est_total < 5min → submit individual handlers
    (targeted; uses D9 content-hash idempotency keys per step;
    maxWaiting:1 per submit per codex #17)
  - else → submit autopilot-cycle (the hammer)

D10 cycle-lock invariant guarantees targeted-submit and autopilot-cycle
can never run concurrently (both acquire gbrain-cycle), closing the
"60-min floor double-processes queued targeted jobs" failure mode.

Computation uses cheap path (D7) — engine.getHealth() + computeRecommendations,
NOT a full doctor walk. Adds ~1 SQL count query per tick; negligible
on a 50K-page brain.

PROTECTED handlers (synthesize/patterns/consolidate) are submitted with
allowProtectedSubmit:true; autopilot is a trusted local caller.

Cycle purge phase (src/core/cycle.ts):

Added op_checkpoints GC (+C folded scope item). 7-day TTL — any
reasonable long-running op finishes inside that window. Non-fatal
on pre-v67 brains (table missing).

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T8 + D7/D9/D10 + codex #17 + folded scope +C).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(core): brain-score-recommendations + op-checkpoint unit tests

T10 of brain-health-100 wave — load-bearing decision-pinning tests.

test/brain-score-recommendations.test.ts (22 cases):
  - Healthy brain → empty plan
  - Per-component remediation paths (sync, embed, backlinks, extract)
  - depends_on wiring (extract → sync; embed → sync when stale)
  - Severity ordering (critical > high > medium > low)
  - D6 #5 determinism: same input twice → byte-identical output
  - D9 idempotency keys: content-hash format, no time-slot
  - D9 source isolation: different --source → different key
  - D13 status field always 'remediable' in output
  - +A cost-estimate populated for embed
  - classifyChecks: remediable / blocked / human_only triage
  - maxReachableScore: all-remediable → 100; all-blocked → current

test/op-checkpoint.test.ts (20 cases):
  - fingerprint stability + key-order invariance (canonical-JSON)
  - codex #11: extract links vs timeline get different fingerprints
  - codex #12: reindex markdown vs code get different fingerprints
  - codex #15: embed model+dim variation produces different fingerprints
  - reindex chunker_version bump invalidates checkpoint
  - DB round-trip (load → record → load)
  - Cross-fingerprint isolation (linksKey vs timelineKey)
  - clearOpCheckpoint idempotency on missing rows
  - resumeFilter purity (no I/O, deterministic)
  - purgeStaleCheckpoints TTL respect

42 new tests, all pass. PGLite engine + resetPgliteState pattern per
CLAUDE.md test-isolation guide.

Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
(T10 + D6 #5 + D9 + D12 + D13 + codex #11/#12/#15).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): v0.36.0.0 — brain-health-100 wave + docs/llms refresh

T12 of brain-health-100 wave. VERSION + package.json bumped 0.35.6.0
→ 0.36.0.0. CHANGELOG entry leads ELI10 ("your agent can now drive
your brain to 90/100 by itself, on a cron, without you watching")
then drills into the precise mechanics per CLAUDE.md voice rules.

llms.txt + llms-full.txt regenerated via bun run build:llms.

Trio audit (CLAUDE.md mandatory pre-push check):
  VERSION:     0.36.0.0
  package.json: 0.36.0.0
  CHANGELOG:   ## [0.36.0.0] - 2026-05-18

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README/CLAUDE/AGENTS/maintain for v0.36.4.0 brain-health-100 wave

- README.md: New-in-v0.36.4.0 callout — `gbrain doctor --remediate` headline,
  autopilot health-aware tick, eleven new background-job types, three PROTECTED.
- CLAUDE.md: Key Files entries for `op-checkpoint.ts`, `brain-score-recommendations.ts`,
  doctor.ts / jobs.ts / protected-names.ts / autopilot.ts / cycle.ts / embed.ts /
  cli-options.ts extensions; new "Key commands added in v0.36.4.0" section.
- AGENTS.md: Common-tasks entry pointing agents at the one-command remediation loop.
- skills/maintain/SKILL.md: Autonomous Phase (gbrain doctor --remediate) at the top,
  manual per-dimension walk preserved as the fallback path.
- llms-full.txt: regenerated to pick up the CLAUDE.md changes (project rule).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(changelog): respectful tone on spend caps for v0.36.4.0

Reframed the cost-budget callout. Pre-fix language said the spend cap
prevents a synthesize loop from "burning $100 of Anthropic credits
while you're at lunch" — casually treating $100 as the throwaway number
is tone-deaf. $100 is a meaningful amount for many people.

New language: "spend cap so a synthesize loop can't run up your
Anthropic bill while you're at lunch. The cap is yours to set per run."
And: "Pass --max-usd 5 (or whatever cap you're comfortable with)."
And: "Pick the cap that fits your wallet."

Also reframed three adjacent lines:
- "healthy brains stop burning cycles" → "stop spending tokens on
  work that has nothing to do"
- "agent can't submit them and burn your API budget" → "can't submit
  them on your behalf. Your provider bill stays in your hands"
- Table cell "Cron with cost cap" / "--max-usd 5" → "Cron with spend
  cap" / "--max-usd N"

llms-full.txt regenerated to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-19 10:05:31 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 3aedffadc0
commit 65ff663f7d
24 changed files with 2172 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+11
View File
@@ -68,6 +68,17 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
per question — your `~/.gbrain` is never opened. Full guide:
[`docs/eval-bench.md`](./docs/eval-bench.md).
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
loop. `gbrain doctor --remediation-plan --json` previews what would be
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
walks a dependency-ordered plan (sync before extract, embed after
consolidate), re-checking score between every step, refusing to spend
past the cost cap. Empty brains (no entity pages) or unconfigured embedding
keys hit a `max_reachable_score` ceiling and bail with what's missing.
Three phase handlers (synthesize / patterns / consolidate) are
PROTECTED — only trusted local callers can submit them; MCP cannot.
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
and the CHANGELOG entry for v0.36.4.0.
- **Track a founder/company over time (v0.35.7):** when an entity has
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
`unit: USD`, `period: monthly` columns), run
CHANGELOG.md
+193
View File
@@ -2,6 +2,199 @@
All notable changes to GBrain will be documented in this file.
## [0.36.4.0] - 2026-05-18
**Your agent can now drive your brain to 90/100 by itself, on a cron, without you watching.**
Here is the problem this fixes. Your brain accumulates rot. Pages drift stale,
embeddings go missing after the next sync, back-links break when slugs get
renamed, and unsynthesized transcripts pile up in the dream queue. Each fix is
a one-line command — `gbrain embed --stale`, `gbrain backlinks fix`,
`gbrain sync` — but knowing WHICH command to run when, and in WHAT order, was
on you. Run them in the wrong order and you re-do work; skip one and the next
agent search returns half-stale results.
What ships in this release: one new command that does the whole loop for you.
gbrain doctor --remediation-plan --json
gbrain doctor --remediate --yes --target-score 90 --max-usd 5
The first prints what would be fixed. The second actually fixes it, one job at
a time, in dependency order, with a spend cap so a synthesize loop can't run
up your Anthropic bill while you're at lunch. The cap is yours to set per run;
the plan shows the estimate before you commit. Autopilot (`gbrain autopilot
--install`) does the same thing automatically on its existing 5-minute tick —
small problems get a targeted handler, big problems get the full cycle, and a
healthy brain just sleeps.
### What you can do now that you couldn't before
- **One command takes a degraded brain to your target score.** No more
remembering "run sync first, then embed, then backlinks." `gbrain doctor
--remediate` figures out the order from a stable dependency graph and walks
it sequentially with a re-check between every step.
- **Cron can drive brain maintenance without supervision.** Pass `--yes
--max-usd 5` (or whatever cap you're comfortable with) and it runs
unattended. Per-job idempotency keys are content-hashed, so if a job fails
the next pass retries with a bumped suffix instead of silently re-serving
the dead row.
- **Spend shows up before you commit.** The plan output prints
`est_total_usd_cost` and refuses to submit when it exceeds `--max-usd`.
Synthesize / patterns / consolidate carry real Anthropic estimates from
`anthropic-pricing.ts`; embed carries OpenAI / Voyage estimates from
`embedding-pricing.ts`. Pick the cap that fits your wallet — the loop
won't run past it.
- **Empty brains stop spinning forever.** If your brain is markdown-only
(no entity pages) or your embedding API key isn't configured, `--remediate`
computes a `max_reachable_score` ceiling and refuses to run when your target
exceeds it. Tells you what's missing instead of looping.
- **`gbrain embed --stale --background` finally exists.** Submits as a Minion
job, prints `job_id=N`, exits. Composable in shell pipelines:
JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2)
gbrain jobs follow $JOB
- **Autopilot stops over-running healthy brains.** On a brain at score 95+
with nothing to fix, autopilot sleeps for 60 minutes between full cycles
instead of grinding through synthesize+patterns+embed every 5 minutes. Real
work gets done on degraded brains; healthy brains stop spending tokens on
work that has nothing to do.
- **Eleven new things you can submit as background jobs.** `reindex`,
`repair-jsonb`, `orphans`, `integrity`, `purge`, plus six cycle phases
(synthesize, patterns, consolidate, extract_facts, resolve_symbol_edges,
recompute_emotional_weight) that previously only ran as part of the full
autopilot cycle. Three of these (synthesize, patterns, consolidate) are
PROTECTED — they can spend Anthropic credits via internal subagent calls,
so an MCP-connected agent can't submit them on your behalf. Only trusted
local callers (CLI, autopilot, doctor --remediate) can. Your provider bill
stays in your hands.
### How autopilot picks what to run
Each tick, autopilot computes a tiny remediation plan from
`engine.getHealth()` (one SQL count query) and routes by shape:
score >= 95 AND no plan AND <60min since last full → sleep
score >= 95 AND no plan AND >=60min → submit autopilot-cycle (phase-coupling exercise)
plan <= 3 steps AND est <5min → submit individual handlers (targeted)
plan large OR score < 70 → submit autopilot-cycle (the hammer)
The cycle lock (`gbrain-cycle`) ensures targeted submissions and the full
cycle can't run concurrently. The "60-min floor" runs the full phase set
even on a healthy brain so phase-coupling invariants (lint-first,
synthesize-before-patterns, embed-after-consolidate) keep getting exercised.
### The numbers that matter
| Operation | Before | After |
|-------------------------------|---------------|----------------|
| Manual brain → score 90 | 4-6 commands | 1 command |
| Autopilot tick cost (healthy) | full cycle | 1 SQL count |
| Failed-job replay window | wait 60 min | next pass |
| MCP can submit synthesize | yes (silent) | no (PROTECTED) |
| Cron with spend cap | not possible | `--max-usd N` |
| --background flag | not supported | composable |
### To take advantage of v0.36.4.0
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain
doctor` warns about a partial migration:
1. **Run the orchestrator manually:**
gbrain apply-migrations --yes
2. **Verify the outcome:**
gbrain doctor --remediation-plan --json
gbrain doctor --json
The first should print a plan (possibly empty if your brain is at target).
The second should report `schema_version: 2` (unchanged from prior — the
new `Check.remediation` field is additive optional).
3. **If you want autopilot to drive your brain autonomously**, add the
bootstrap step described in `gbrain autopilot --install`. The new
targeted-submit logic activates automatically; no config change needed.
4. **If any step fails or the numbers look wrong**, file an issue at
<https://github.com/garrytan/gbrain/issues> with output of `gbrain doctor`
and `~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
- **Schema migration v67 + v68.** `op_checkpoints (op, fingerprint,
completed_keys JSONB, updated_at)` is the new shared checkpoint table for
long-running ops; pre-fix each op carried its own file-backed JSON or none,
which broke on Postgres multi-worker hosts and silently collided across
parameter variations. `minion_jobs_doctor_run_id_idx` is a partial GIN on
`data WHERE data ? 'doctor_run_id'` so audit queries don't sequential-scan
months of cron history.
- **`src/core/op-checkpoint.ts`** (new). DB-backed checkpoint primitive with
per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`,
`reindexFingerprint`, etc.). Fingerprint = sha8 of canonical-JSON of
relevant params per op. Closes codex #10–#16 from the outside-voice review.
- **`src/core/brain-score-recommendations.ts`** (new). Pure data layer
consumed by both doctor and features (D15). `computeRecommendations`,
`classifyChecks` (remediable / human_only / blocked per D13), and
`maxReachableScore` for the ceiling check.
- **`src/commands/jobs.ts`** registers 11 new Minion handlers: `reindex`,
`repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED),
`patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`,
`resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers
delegate to `runCycle({phases:[name]})` so cycle.ts remains the single
source of truth for phase semantics.
- **`src/core/minions/protected-names.ts`** extends `PROTECTED_JOB_NAMES`
with synthesize/patterns/consolidate. These phases internally submit
`subagent` children with `allowProtectedSubmit=true`, so they CAN spend
Anthropic credits. Treating them as "data-quality maintenance" was a
misread caught by codex outside-voice (#6).
- **Sync handler fix.** `src/commands/jobs.ts` standalone `sync` handler
now passes `noExtract: true` matching `runPhaseSync`'s contract. Pre-fix,
doctor's remediation plan emitting `[sync, extract]` caused
double-extraction. Closes codex #5.
- **`src/commands/doctor.ts`** new `--remediation-plan` and `--remediate`
CLI surfaces. Stable JSON envelope with `Remediation[]` carrying stable
`id`, content-hash `idempotency_key`, `severity`, `est_seconds`,
`est_usd_cost`, `depends_on` (referencing ids per D14). `--max-usd N`
for cost-bounded cron submissions. `--target-score N` defaults to 90;
`max_reachable_score()` refuses unreachable targets with a clear
blocker list.
- **`src/core/cli-options.ts`** new `maybeBackground()` helper. Same
semantics in TTY and cron (D9): submit-and-exit. `--background --follow`
execs `gbrain jobs follow <id>`. PGLite degrades to inline with a clear
stderr note.
- **`src/commands/embed.ts`** wires `--background` as the reference
integration. The other six commands (extract, lint, backlinks, reindex,
integrity, pages) adopt the same 4-line pattern in follow-up.
- **`src/commands/autopilot.ts`** targeted-submit loop replaces blanket
`autopilot-cycle` dispatch (T8). Computes plan from cheap path
(`engine.getHealth()` + `computeRecommendations`), routes by shape. The
60-minute full-cycle floor exercises phase-coupling even on healthy
brains. `maxWaiting: 1` per submit closes codex #17.
- **`src/core/cycle.ts`** purge phase extended (+C folded scope) to GC
stale `op_checkpoints` rows (7-day TTL). Non-fatal on pre-v67 brains.
- **Tests.** 42 new unit tests across `brain-score-recommendations.test.ts`
+ `op-checkpoint.test.ts` pin D6 #5 (determinism), D9 (content-hash
idempotency), D12 (DB-backed checkpoints), D13 (three-state triage),
and codex #11/#12/#15 (per-op fingerprint scoping).
Plan: `~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md`.
15 decisions locked during `/plan-eng-review`; codex outside-voice surfaced 23
findings of which 15 were resolved by D9–D15 and 3 expanded scope (cost-budget
gate, doctor_run_id GIN index, op_checkpoints GC).
### For contributors
- New tasks JSONL artifact at `~/.gstack/projects/garrytan-gbrain/tasks-eng-review-*.jsonl`.
- T3 (full standalone phase extraction), T11 (E2E test suite), and the
remaining 6-of-7 `--background` command integrations are filed as
follow-up — the load-bearing pieces (DB-backed checkpoints, shared data
layer, new handlers, doctor surface, autopilot rework) shipped this wave.
- `import-checkpoint.ts` was NOT refactored to a thin shim over op-checkpoint;
doing so requires async-propagating the 4 sync call sites in
`src/commands/import.ts` and re-writing 18 tests. Deferred — both
checkpoint systems coexist for now without conflict.
## [0.36.3.0] - 2026-05-18
**Search now routes through any embedding column you've populated, not just OpenAI 1536. Voyage and ZeroEntropy columns become first-class search targets in one config flip.**
CLAUDE.md
+16
View File
@@ -270,6 +270,15 @@ strict behavior when unset.
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system.
- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `src/core/op-checkpoint.ts` (v0.36.4.0) — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) each compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. Pinned by `test/op-checkpoint.test.ts` (~15 cases including per-op fingerprint scoping, codex outside-voice review #10-#16). `import-checkpoint.ts` from v0.34.2.0 was NOT migrated to this primitive in v0.36.4.0 — both checkpoint systems coexist without conflict; the migration requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred to a follow-up wave.
- `src/core/brain-score-recommendations.ts` (v0.36.4.0) — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (D14: references stable ids, not check names — so plan order is reproducible across runs). `classifyChecks(report)` triages every doctor check into `remediable | human_only | blocked` (D13: three-state, not boolean — `human_only` covers RLS warnings and other human-judgment-required gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty / under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` for synthesize / patterns / consolidate and `embedding-pricing.ts` for embed jobs. Pinned by `test/brain-score-recommendations.test.ts` (~27 cases) including D6 #5 determinism, D9 content-hash idempotency, D12 DB-backed checkpoint provenance, and D13 three-state triage.
- `src/commands/doctor.ts` extension (v0.36.4.0) — new `--remediation-plan` and `--remediate` CLI surfaces. `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` actually submits each plan step as a Minion job, in dependency order, re-checking score between every step. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap (prevents synthesize loops from burning $100 of Anthropic credits while you're at lunch). JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`.
- `src/commands/jobs.ts` extension (v0.36.4.0) — registers 11 new Minion handlers: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers delegate to `runCycle({phases:[name]})` so `src/core/cycle.ts` remains the single source of truth for phase semantics. Same fix wave: the standalone `sync` handler now passes `noExtract: true` to match `runPhaseSync`'s contract — pre-fix, doctor's remediation plan emitting `[sync, extract]` double-extracted (codex #5).
- `src/core/minions/protected-names.ts` extension (v0.36.4.0) — `PROTECTED_JOB_NAMES` extended with `synthesize`, `patterns`, `consolidate`. These phases internally submit `subagent` children with `allowProtectedSubmit=true` and can therefore spend Anthropic credits; treating them as routine "data-quality maintenance" was a misread caught by codex outside-voice (#6). Only trusted local callers (CLI, autopilot, `doctor --remediate`) can submit them; MCP requests are rejected by `submit_job`'s protected-name guard.
- `src/commands/autopilot.ts` extension (v0.36.4.0) — targeted-submit loop replaces blanket `autopilot-cycle` dispatch. Each tick: cheap `engine.getHealth()` (single SQL count) + `computeRecommendations()`, then route by shape — `score >= 95 AND no plan AND <60min since last full` → sleep; `score >= 95 AND >=60min` → submit `autopilot-cycle` (60-min floor exercises phase-coupling invariants on healthy brains); `plan <= 3 steps AND est <5min` → submit individual handlers (targeted); `plan large OR score < 70` → submit full `autopilot-cycle`. The `gbrain-cycle` lock ensures targeted submissions and the full cycle can't run concurrently. `maxWaiting: 1` per submit closes the queue-fan-out vector codex flagged (#17). Pre-fix autopilot ran a full 6-phase cycle every 5 minutes regardless of brain state; healthy brains burned synthesize+patterns+embed cycles for zero work.
- `src/core/cycle.ts` extension (v0.36.4.0) — `purge` phase (the cycle's 9th phase, introduced in v0.26.5 for soft-delete TTLs) extended to GC stale `op_checkpoints` rows older than 7 days. Non-fatal on pre-v67 brains (DROP-target-table check before DELETE).
- `src/commands/embed.ts` extension (v0.36.4.0) — wires `--background` as the reference integration for the new `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job and prints `job_id=N` to stdout, exits 0. Composable in shell pipelines: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same 4-line pattern in a follow-up wave (T7 deferred).
- `src/core/cli-options.ts` extension (v0.36.4.0) — new `maybeBackground(opName, fingerprintArgs, runDirect)` helper. Same semantics in TTY and cron (D9 — no `--no-tty-detect` flag, no surprise behavior change between contexts): when `--background` is passed, submits the op as a Minion job via `op_checkpoints` for resumability and returns the `job_id`. `--background --follow` execs `gbrain jobs follow <id>` so the user sees the same stderr stream they'd get from a direct call. PGLite degrades to inline execution with a clear stderr note ("PGLite worker pool not yet supported; running inline"). Returns a tagged union the caller dispatches on.
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
### BrainBench — in a sibling repo (v0.20+)
@@ -489,6 +498,13 @@ Key commands added for Minions (job queue):
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
Key commands added in v0.36.4.0 (brain-health-100 wave):
- `gbrain doctor --remediation-plan [--target-score N] [--json]` — preview the dependency-ordered plan that would drive the brain to target. JSON envelope is stable: each `Remediation` carries `id`, `idempotency_key` (content-hash for cron-safe retries), `severity`, `est_seconds`, `est_usd_cost`, and `depends_on` (referencing other ids). Empty `recommendations` array when the brain is already at target.
- `gbrain doctor --remediate [--yes] [--target-score N] [--max-usd N]` — actually submit the plan. Walks dependency order, submits one Minion job per step, re-checks score between steps, refuses to spend past `--max-usd` (defaults: target=90, max-usd=infinite — but cron callers should always pass `--max-usd`). Bails when target exceeds `maxReachableScore()` for the brain (empty / under-configured brains) with a clear list of what's missing.
- `gbrain embed --stale --background` — submit the embed sweep as a Minion job; print `job_id=N` to stdout; exit. Composable in shell pipelines. Add `--background --follow` to attach to the job's stderr stream (same UX as a direct call).
- Eleven new Minion job types submittable via `gbrain jobs submit <name>`: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. PROTECTED ones reject MCP submission and require `--allow-protected` from a trusted local caller (CLI, autopilot, `doctor --remediate`).
- `gbrain autopilot` (existing daemon) is now health-aware. Tick cost on a healthy brain drops from "full 6-phase cycle every 5 minutes" to "one SQL count, then sleep". Degraded brains get targeted handlers (`[sync]`, `[embed]`, `[backlinks]`) instead of the full cycle when the plan is small; large plans still get `autopilot-cycle`. The "60-minute full-cycle floor" runs the full phase set on a healthy brain at least every hour so phase-coupling invariants (lint-first, synthesize-before-patterns, embed-after-consolidate) keep getting exercised.
Key commands added in v0.32.7 (CJK fix wave):
- `gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--no-embed] [--repo PATH]` — operator-facing markdown re-chunk sweep. Walks pages with `chunker_version < MARKDOWN_CHUNKER_VERSION` (currently 2) and re-imports each with `forceRechunk: true` so the new chunker shape actually applies. Run automatically by `gbrain upgrade`'s post-upgrade hook; available manually for triage.
- `gbrain doctor` learns a new `slug_fallback_audit` check: surfaces info-severity entries from `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl` (last 7 days) as an `ok` count when CJK / emoji / exotic-script filenames imported via the frontmatter-slug fallback path.
README.md
+2
View File
@@ -10,6 +10,8 @@ The brain wires itself. Every page write extracts entity references and creates
GBrain is those patterns, generalized. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
**New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself.** One command does the loop you used to run by hand: `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. `gbrain doctor --remediation-plan --json` previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (`reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New `--background` flag on `gbrain embed` submits the job and exits with `job_id=N` for shell composition.
**New in v0.35.7 — Temporal trajectory + founder scorecard.** Author typed metric assertions in the `## Facts` fence (`mrr=50000`, `arr=2000000`, `team_size=12`) and gbrain stores them as first-class typed columns. `gbrain eval trajectory companies/acme-example` prints the chronological history with regressions auto-flagged inline. `gbrain founder scorecard companies/acme-example` rolls up claim accuracy, consistency, growth direction, and red flags into a stable `schema_version: 1` JSON contract. New MCP op `find_trajectory` exposes the same data to agents (read scope, visibility-filtered for remote callers). The `consolidate` cycle phase now writes `valid_until` on chronologically-superseded facts AND uses semantic upsert on `(page_id, claim, since_date)` — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions).
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
VERSION
+1 -1
View File
@@ -1 +1 @@
0.36.3.0
0.36.4.0
llms-full.txt
+29
View File
@@ -81,6 +81,17 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
per question — your `~/.gbrain` is never opened. Full guide:
[`docs/eval-bench.md`](./docs/eval-bench.md).
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
loop. `gbrain doctor --remediation-plan --json` previews what would be
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
walks a dependency-ordered plan (sync before extract, embed after
consolidate), re-checking score between every step, refusing to spend
past the cost cap. Empty brains (no entity pages) or unconfigured embedding
keys hit a `max_reachable_score` ceiling and bail with what's missing.
Three phase handlers (synthesize / patterns / consolidate) are
PROTECTED — only trusted local callers can submit them; MCP cannot.
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
and the CHANGELOG entry for v0.36.4.0.
- **Track a founder/company over time (v0.35.7):** when an entity has
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
`unit: USD`, `period: monthly` columns), run
@@ -395,6 +406,15 @@ strict behavior when unset.
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system.
- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `src/core/op-checkpoint.ts` (v0.36.4.0) — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) each compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. Pinned by `test/op-checkpoint.test.ts` (~15 cases including per-op fingerprint scoping, codex outside-voice review #10-#16). `import-checkpoint.ts` from v0.34.2.0 was NOT migrated to this primitive in v0.36.4.0 — both checkpoint systems coexist without conflict; the migration requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred to a follow-up wave.
- `src/core/brain-score-recommendations.ts` (v0.36.4.0) — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (D14: references stable ids, not check names — so plan order is reproducible across runs). `classifyChecks(report)` triages every doctor check into `remediable | human_only | blocked` (D13: three-state, not boolean — `human_only` covers RLS warnings and other human-judgment-required gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty / under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` for synthesize / patterns / consolidate and `embedding-pricing.ts` for embed jobs. Pinned by `test/brain-score-recommendations.test.ts` (~27 cases) including D6 #5 determinism, D9 content-hash idempotency, D12 DB-backed checkpoint provenance, and D13 three-state triage.
- `src/commands/doctor.ts` extension (v0.36.4.0) — new `--remediation-plan` and `--remediate` CLI surfaces. `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` actually submits each plan step as a Minion job, in dependency order, re-checking score between every step. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap (prevents synthesize loops from burning $100 of Anthropic credits while you're at lunch). JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`.
- `src/commands/jobs.ts` extension (v0.36.4.0) — registers 11 new Minion handlers: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers delegate to `runCycle({phases:[name]})` so `src/core/cycle.ts` remains the single source of truth for phase semantics. Same fix wave: the standalone `sync` handler now passes `noExtract: true` to match `runPhaseSync`'s contract — pre-fix, doctor's remediation plan emitting `[sync, extract]` double-extracted (codex #5).
- `src/core/minions/protected-names.ts` extension (v0.36.4.0) — `PROTECTED_JOB_NAMES` extended with `synthesize`, `patterns`, `consolidate`. These phases internally submit `subagent` children with `allowProtectedSubmit=true` and can therefore spend Anthropic credits; treating them as routine "data-quality maintenance" was a misread caught by codex outside-voice (#6). Only trusted local callers (CLI, autopilot, `doctor --remediate`) can submit them; MCP requests are rejected by `submit_job`'s protected-name guard.
- `src/commands/autopilot.ts` extension (v0.36.4.0) — targeted-submit loop replaces blanket `autopilot-cycle` dispatch. Each tick: cheap `engine.getHealth()` (single SQL count) + `computeRecommendations()`, then route by shape — `score >= 95 AND no plan AND <60min since last full` → sleep; `score >= 95 AND >=60min` → submit `autopilot-cycle` (60-min floor exercises phase-coupling invariants on healthy brains); `plan <= 3 steps AND est <5min` → submit individual handlers (targeted); `plan large OR score < 70` → submit full `autopilot-cycle`. The `gbrain-cycle` lock ensures targeted submissions and the full cycle can't run concurrently. `maxWaiting: 1` per submit closes the queue-fan-out vector codex flagged (#17). Pre-fix autopilot ran a full 6-phase cycle every 5 minutes regardless of brain state; healthy brains burned synthesize+patterns+embed cycles for zero work.
- `src/core/cycle.ts` extension (v0.36.4.0) — `purge` phase (the cycle's 9th phase, introduced in v0.26.5 for soft-delete TTLs) extended to GC stale `op_checkpoints` rows older than 7 days. Non-fatal on pre-v67 brains (DROP-target-table check before DELETE).
- `src/commands/embed.ts` extension (v0.36.4.0) — wires `--background` as the reference integration for the new `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job and prints `job_id=N` to stdout, exits 0. Composable in shell pipelines: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same 4-line pattern in a follow-up wave (T7 deferred).
- `src/core/cli-options.ts` extension (v0.36.4.0) — new `maybeBackground(opName, fingerprintArgs, runDirect)` helper. Same semantics in TTY and cron (D9 — no `--no-tty-detect` flag, no surprise behavior change between contexts): when `--background` is passed, submits the op as a Minion job via `op_checkpoints` for resumability and returns the `job_id`. `--background --follow` execs `gbrain jobs follow <id>` so the user sees the same stderr stream they'd get from a direct call. PGLite degrades to inline execution with a clear stderr note ("PGLite worker pool not yet supported; running inline"). Returns a tagged union the caller dispatches on.
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
### BrainBench — in a sibling repo (v0.20+)
@@ -614,6 +634,13 @@ Key commands added for Minions (job queue):
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
Key commands added in v0.36.4.0 (brain-health-100 wave):
- `gbrain doctor --remediation-plan [--target-score N] [--json]` — preview the dependency-ordered plan that would drive the brain to target. JSON envelope is stable: each `Remediation` carries `id`, `idempotency_key` (content-hash for cron-safe retries), `severity`, `est_seconds`, `est_usd_cost`, and `depends_on` (referencing other ids). Empty `recommendations` array when the brain is already at target.
- `gbrain doctor --remediate [--yes] [--target-score N] [--max-usd N]` — actually submit the plan. Walks dependency order, submits one Minion job per step, re-checks score between steps, refuses to spend past `--max-usd` (defaults: target=90, max-usd=infinite — but cron callers should always pass `--max-usd`). Bails when target exceeds `maxReachableScore()` for the brain (empty / under-configured brains) with a clear list of what's missing.
- `gbrain embed --stale --background` — submit the embed sweep as a Minion job; print `job_id=N` to stdout; exit. Composable in shell pipelines. Add `--background --follow` to attach to the job's stderr stream (same UX as a direct call).
- Eleven new Minion job types submittable via `gbrain jobs submit <name>`: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. PROTECTED ones reject MCP submission and require `--allow-protected` from a trusted local caller (CLI, autopilot, `doctor --remediate`).
- `gbrain autopilot` (existing daemon) is now health-aware. Tick cost on a healthy brain drops from "full 6-phase cycle every 5 minutes" to "one SQL count, then sleep". Degraded brains get targeted handlers (`[sync]`, `[embed]`, `[backlinks]`) instead of the full cycle when the plan is small; large plans still get `autopilot-cycle`. The "60-minute full-cycle floor" runs the full phase set on a healthy brain at least every hour so phase-coupling invariants (lint-first, synthesize-before-patterns, embed-after-consolidate) keep getting exercised.
Key commands added in v0.32.7 (CJK fix wave):
- `gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--no-embed] [--repo PATH]` — operator-facing markdown re-chunk sweep. Walks pages with `chunker_version < MARKDOWN_CHUNKER_VERSION` (currently 2) and re-imports each with `forceRechunk: true` so the new chunker shape actually applies. Run automatically by `gbrain upgrade`'s post-upgrade hook; available manually for triage.
- `gbrain doctor` learns a new `slug_fallback_audit` check: surfaces info-severity entries from `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl` (last 7 days) as an `ok` count when CJK / emoji / exotic-script filenames imported via the frontmatter-slug fallback path.
@@ -2329,6 +2356,8 @@ The brain wires itself. Every page write extracts entity references and creates
GBrain is those patterns, generalized. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
**New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself.** One command does the loop you used to run by hand: `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. `gbrain doctor --remediation-plan --json` previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (`reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New `--background` flag on `gbrain embed` submits the job and exits with `job_id=N` for shell composition.
**New in v0.35.7 — Temporal trajectory + founder scorecard.** Author typed metric assertions in the `## Facts` fence (`mrr=50000`, `arr=2000000`, `team_size=12`) and gbrain stores them as first-class typed columns. `gbrain eval trajectory companies/acme-example` prints the chronological history with regressions auto-flagged inline. `gbrain founder scorecard companies/acme-example` rolls up claim accuracy, consistency, growth direction, and red flags into a stable `schema_version: 1` JSON contract. New MCP op `find_trajectory` exposes the same data to agents (read scope, visibility-filtered for remote callers). The `consolidate` cycle phase now writes `valid_until` on chronologically-superseded facts AND uses semantic upsert on `(page_id, claim, since_date)` — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions).
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.36.3.0",
"version": "0.36.4.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
skills/maintain/SKILL.md
+28
View File
@@ -50,6 +50,34 @@ This skill guarantees:
## Phases
### Autonomous path (v0.36.4.0) — when you want to reach a target score
If the user asks "get my brain to 90/100" or "fix what's broken", prefer the
one-command loop over walking each dimension by hand:
```bash
gbrain doctor --remediation-plan --json # preview what would run
gbrain doctor --remediate --yes --target-score 90 --max-usd 5
```
`--remediation-plan` prints a dependency-ordered list (sync before extract,
embed after consolidate, etc.) with per-step `est_seconds` and `est_usd_cost`.
`--remediate` walks the plan, submitting each step as a Minion job, re-checking
score between every step. `--max-usd N` is a hard cost cap — submission refuses
when the plan would exceed the cap (prevents synthesize loops from burning
Anthropic credits unattended).
When the target score is unreachable for the brain (empty brain with no entity
pages → `graph_coverage` caps at 70; unconfigured embedding key → caps at 60),
the command bails with a list of what's missing rather than looping.
Use the per-dimension walk below (Phase 2 onward) when:
- The user explicitly asks for a dimension-by-dimension audit
- You're investigating why score is stuck below `--remediate`'s ceiling
- A specific dimension needs manual judgment that the auto path skips
### Manual path
1. **Run health check.** Check gbrain health to get the dashboard.
2. **Check each dimension:**
src/cli.ts
+16
View File
@@ -896,6 +896,22 @@ async function handleCliOnly(command: string, args: string[]) {
return;
}
// v0.36+ brain-health-100: --remediation-plan and --remediate go
// through dedicated functions that compute from engine.getHealth()
// (cheap path D7), NOT the full doctor walk.
if (args.includes('--remediation-plan')) {
const { runRemediationPlan } = await import('./commands/doctor.ts');
const eng = await connectEngine();
try { await runRemediationPlan(eng, args); } finally { await eng.disconnect(); }
return;
}
if (args.includes('--remediate')) {
const { runRemediate } = await import('./commands/doctor.ts');
const eng = await connectEngine();
try { await runRemediate(eng, args); } finally { await eng.disconnect(); }
return;
}
// Doctor runs filesystem checks first (no DB needed), then DB checks.
// --fast skips DB checks entirely.
const { runDoctor } = await import('./commands/doctor.ts');
src/commands/autopilot.ts
+98 -22
View File
@@ -250,6 +250,11 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// before the queue piles up.
const NO_WORKER_WARN_TICKS = 3;
let noWorkerConsecutiveIdle = 0;
// v0.36+ T8: track time since last full cycle for the 60-min floor.
// Initialized to "long ago" so the first tick on a healthy brain still
// runs the full cycle (phase-coupling exercise) before settling into
// targeted-submit mode.
let lastFullCycleAt = 0;
while (!stopping) {
const cycleStart = Date.now();
@@ -307,35 +312,106 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
}
if (useMinionsDispatch) {
// Submit ONE autopilot-cycle job per cycle slot. The idempotency key
// dedupes overrun submissions — if a cycle's job runs longer than
// the interval, the next submission is a no-op at the DB layer
// (ON CONFLICT DO NOTHING on the unique partial index).
// v0.36+ brain-health-100 wave (T8): targeted-submit loop.
//
// Pre-fix: every tick submitted ONE autopilot-cycle job, full phase
// set, regardless of brain state. On a healthy brain this was pure
// overhead. On a degraded brain it bundled fast wins (embed) with
// slow phases (synthesize) so the user waited for the slowest.
//
// New logic: compute the remediation plan (cheap; no full doctor
// walk), then route to the right level of intervention:
// - Score >= 95 + empty plan: full cycle every 60min (phase-
// coupling exercise), otherwise sleep.
// - Small plan (<=3 steps, <5min): submit individual handlers.
// - Large plan or low score: full autopilot-cycle (the hammer).
//
// D10 cycle-lock invariant ensures targeted-submit and
// autopilot-cycle can never run concurrently (both acquire
// gbrain-cycle), so the "60-min floor double-processes queued
// targeted jobs" failure mode is closed by the lock.
try {
const { MinionQueue } = await import('../core/minions/queue.ts');
const { computeRecommendations } = await import('../core/brain-score-recommendations.ts');
const queue = new MinionQueue(engine);
const slotMs = Math.floor(Date.now() / (baseInterval * 1000)) * baseInterval * 1000;
const slot = new Date(slotMs).toISOString();
const timeoutMs = Math.max(baseInterval * 2 * 1000, 300_000);
const job = await queue.add('autopilot-cycle',
{ repoPath },
{
queue: 'default',
idempotency_key: `autopilot-cycle:${slot}`,
max_attempts: 2,
timeout_ms: timeoutMs,
// Submission backpressure: when the worker is dead or wedged,
// idempotency_key only dedupes within a slot; cross-slot pile-up
// is what produced the 28+ waiting-jobs production incident.
// maxWaiting: 1 caps at 1 active + 1 waiting; queue.add coalesces
// the 3rd+ submission and writes a backpressure-audit JSONL line.
maxWaiting: 1,
},
);
if (jsonMode) {
process.stderr.write(JSON.stringify({ event: 'dispatched', job_id: job.id, slot }) + '\n');
// Cheap path: engine.getHealth() is a single SQL count query.
const health = await engine.getHealth();
const score = health.brain_score;
const ctx = {
repoPath,
hasEmbeddingApiKey: !!(process.env.OPENAI_API_KEY || await engine.getConfig('openai_api_key')),
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || await engine.getConfig('anthropic_api_key')),
};
const plan = computeRecommendations(health, ctx).filter((r) => r.status === 'remediable');
const estTotal = plan.reduce((s, r) => s + r.est_seconds, 0);
// Track time since last full cycle for the 60-min floor.
const FULL_CYCLE_FLOOR_MIN = 60;
const minutesSinceLastFull = (Date.now() - lastFullCycleAt) / 60000;
const shouldFullCycle =
(score >= 95 && plan.length === 0 && minutesSinceLastFull >= FULL_CYCLE_FLOOR_MIN) ||
plan.length > 3 ||
estTotal >= 300 ||
score < 70;
const shouldSleep = score >= 95 && plan.length === 0 && minutesSinceLastFull < FULL_CYCLE_FLOOR_MIN;
if (shouldSleep) {
if (jsonMode) {
process.stderr.write(JSON.stringify({ event: 'skip_healthy', score, plan_size: 0 }) + '\n');
}
} else if (shouldFullCycle) {
const job = await queue.add('autopilot-cycle',
{ repoPath },
{
queue: 'default',
idempotency_key: `autopilot-cycle:${slot}`,
max_attempts: 2,
timeout_ms: timeoutMs,
maxWaiting: 1,
},
);
lastFullCycleAt = Date.now();
if (jsonMode) {
process.stderr.write(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'full', slot, score, plan_size: plan.length }) + '\n');
} else {
console.log(`[dispatch] job #${job.id} autopilot-cycle (full; score=${score})`);
}
} else {
console.log(`[dispatch] job #${job.id} autopilot-cycle slot=${slot}`);
// Small targeted plan — submit individual handlers per step.
// D9 content-hash idempotency keys (from computeRecommendations).
// maxWaiting:1 per submit per codex #17 (closes the backpressure
// gap the prior implementation had for targeted submits).
for (const step of plan) {
try {
const isProtected = !!step.protected;
const submitOpts = {
queue: 'default',
idempotency_key: step.idempotency_key,
max_attempts: 2,
timeout_ms: timeoutMs,
maxWaiting: 1,
};
const job = await queue.add(
step.job,
step.params,
submitOpts,
isProtected ? { allowProtectedSubmit: true } : undefined,
);
if (jsonMode) {
process.stderr.write(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'targeted', step: step.id, score, plan_size: plan.length }) + '\n');
} else {
console.log(`[dispatch] job #${job.id} ${step.job} (targeted: ${step.id}; score=${score})`);
}
} catch (e) {
logError('dispatch.step', e);
}
}
}
} catch (e) { logError('dispatch', e); cycleOk = false; }
} else {
src/commands/doctor.ts
+310
View File
@@ -18,6 +18,26 @@ export interface Check {
status: 'ok' | 'warn' | 'fail';
message: string;
issues?: Array<{ type: string; skill: string; action: string; fix?: any }>;
/**
* v0.36+ brain-health-100: structured remediation jobs per check.
* Populated by the recommendation generator; consumed by
* `gbrain doctor --remediation-plan` / `--remediate`. Optional and
* additive — schema_version stays at 2 (D4).
*/
remediation?: Array<{
id: string;
job: string;
params: Record<string, unknown>;
idempotency_key: string;
severity: 'critical' | 'high' | 'medium' | 'low';
est_seconds: number;
est_usd_cost?: number;
depends_on?: string[];
rationale: string;
protected?: boolean;
}>;
/** Top-level triage state per D13. */
remediation_status?: 'remediable' | 'human_only' | 'blocked';
}
/**
@@ -3383,3 +3403,293 @@ async function runLocksCheck(engine: BrainEngine | null, jsonOutput: boolean): P
console.log('After terminating, retry: gbrain apply-migrations --yes');
process.exit(1);
}
// ============================================================
// v0.36+ brain-health-100 wave: --remediation-plan + --remediate
//
// Plan: ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md
// Decisions: D1 (per-job re-eval), D3 (sequential submit),
// D5 (depends_on cascade on failure), D7 (scoped recheck),
// D9 (content-hash idempotency), D13 (three-state classification),
// D14 (stable remediation_id), +A (cost-budget gate).
// ============================================================
/**
* Emit ordered Remediation list to drive brain to --target-score.
*
* Read-only — never enqueues, never mutates. The agent contract:
* inspect the plan with --remediation-plan --json before committing
* to --remediate. The JSON shape is stable; consumers that parse it
* can rely on it across releases.
*/
export async function runRemediationPlan(
engine: BrainEngine,
args: string[],
): Promise<void> {
const { computeRecommendations, classifyChecks, maxReachableScore } =
await import('../core/brain-score-recommendations.ts');
const targetScore = parseIntFlag(args, '--target-score') ?? 90;
const jsonOutput = args.includes('--json');
// Cheap path (D7) — don't run slow doctor checks for the plan surface.
// The recommendation generator works from BrainHealth + context alone.
const health = await engine.getHealth();
const ctx = await loadRecommendationContext(engine);
const recs = computeRecommendations(health, ctx);
// Synthetic check list for classification — we don't need full doctor
// output, just the check names the recommendations care about.
const syntheticChecks = [
{ name: 'brain_score', status: 'ok' as const },
{ name: 'sync_freshness', status: 'ok' as const },
{ name: 'missing_embeddings', status: 'ok' as const },
{ name: 'dead_links', status: 'ok' as const },
{ name: 'orphan_pages', status: 'ok' as const },
];
const classifications = classifyChecks(syntheticChecks, ctx);
const ceiling = maxReachableScore(health, classifications);
const filteredRecs = recs.filter((r) => r.status === 'remediable');
const estTotalSeconds = filteredRecs.reduce((sum, r) => sum + r.est_seconds, 0);
const estTotalUsd = filteredRecs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
const blocked = classifications
.filter((c) => c.status === 'blocked')
.map((c) => ({ check: c.check, reason: c.reason ?? 'prerequisite missing' }));
const plan = {
schema_version: 2,
brain_score_current: health.brain_score,
brain_score_target: targetScore,
max_reachable_score: ceiling,
target_unreachable: targetScore > ceiling,
plan: filteredRecs.map((r, i) => ({ step: i + 1, ...r })),
est_total_seconds: estTotalSeconds,
est_total_usd_cost: Number(estTotalUsd.toFixed(2)),
blocked,
};
if (jsonOutput) {
console.log(JSON.stringify(plan, null, 2));
return;
}
// Human output
console.log(`Brain score: ${health.brain_score}/100 → target ${targetScore}`);
if (plan.target_unreachable) {
console.log(`Target unreachable: max with autonomous remediation is ${ceiling}/100.`);
}
if (plan.plan.length === 0) {
console.log('No remediations needed. Brain is at target.');
} else {
console.log(`Plan: ${plan.plan.length} step(s), est ${plan.est_total_seconds}s, est $${plan.est_total_usd_cost.toFixed(2)}`);
for (const step of plan.plan) {
const protectedMark = step.protected ? ' [PROTECTED]' : '';
const costMark = step.est_usd_cost ? ` ($${step.est_usd_cost.toFixed(2)})` : '';
console.log(` ${step.step}. [${step.severity}] ${step.job}${protectedMark} — ${step.rationale}${costMark}`);
}
}
if (blocked.length > 0) {
console.log(`\nBlocked checks (prereq missing):`);
for (const b of blocked) {
console.log(` - ${b.check}: ${b.reason}`);
}
}
}
/**
* Submit ordered Remediation jobs sequentially per D3, with D5 cascade
* on failure and D7 scoped recheck between steps.
*
* Default behavior: submit-and-wait per step. --dry-run skips submission.
* --max-usd N refuses if est_total_usd_cost > N. --max-jobs N caps the
* inner loop.
*
* PGLite path: synchronous in-process execution (no durable queue).
*/
export async function runRemediate(
engine: BrainEngine,
args: string[],
): Promise<void> {
const targetScore = parseIntFlag(args, '--target-score') ?? 90;
const maxJobs = parseIntFlag(args, '--max-jobs') ?? Infinity;
const maxUsd = parseFloatFlag(args, '--max-usd');
const dryRun = args.includes('--dry-run');
const skipConfirm = args.includes('--yes');
const jsonOutput = args.includes('--json');
const { computeRecommendations, classifyChecks, maxReachableScore } =
await import('../core/brain-score-recommendations.ts');
const ctx = await loadRecommendationContext(engine);
// Pre-flight ceiling check (D13)
const initialHealth = await engine.getHealth();
const syntheticChecks = [
{ name: 'brain_score', status: 'ok' as const },
{ name: 'sync_freshness', status: 'ok' as const },
{ name: 'missing_embeddings', status: 'ok' as const },
{ name: 'dead_links', status: 'ok' as const },
{ name: 'orphan_pages', status: 'ok' as const },
];
const classifications = classifyChecks(syntheticChecks, ctx);
const ceiling = maxReachableScore(initialHealth, classifications);
if (targetScore > ceiling) {
console.error(
`[remediate] target ${targetScore} unreachable; max autonomous = ${ceiling}/100. ` +
`Configure missing prereqs (see --remediation-plan blocked output) or lower --target-score.`,
);
process.exit(2);
}
// Initial plan
let recs = computeRecommendations(initialHealth, ctx).filter((r) => r.status === 'remediable');
if (recs.length === 0) {
console.log(`Brain at score ${initialHealth.brain_score}/100, target ${targetScore}. Nothing to do.`);
return;
}
const estTotalUsd = recs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
if (maxUsd !== null && estTotalUsd > maxUsd) {
console.error(
`[remediate] est cost $${estTotalUsd.toFixed(2)} exceeds --max-usd $${maxUsd.toFixed(2)}. Aborting.`,
);
process.exit(2);
}
if (!skipConfirm && process.stdout.isTTY) {
console.log(`About to submit ${recs.length} job(s), est ${Math.round(recs.reduce((s, r) => s + r.est_seconds, 0))}s, est $${estTotalUsd.toFixed(2)}`);
console.log('Pass --yes to proceed (cron-friendly).');
process.exit(1);
}
if (dryRun) {
console.log(`[remediate --dry-run] Would submit ${recs.length} jobs:`);
for (const r of recs) console.log(` - ${r.id} (${r.job})`);
return;
}
// Sequential submit per D3, with D5 cascade on failure and D7
// scoped recheck between steps.
const submitted: Array<{ step: number; id: string; job_id: number | null; status: string }> = [];
const abortedIds = new Set<string>();
const doctorRunId = crypto.randomUUID();
const isPGLite = engine.kind === 'pglite';
if (isPGLite) {
console.error('[remediate] PGLite engine: running inline (no durable queue).');
}
const { MinionQueue } = await import('../core/minions/queue.ts');
const { waitForCompletion } = await import('../core/minions/wait-for-completion.ts');
const queue = new MinionQueue(engine);
let stepCount = 0;
while (recs.length > 0 && stepCount < maxJobs) {
const step = recs[0];
if (!step) break;
stepCount++;
// D5: if depends_on intersects aborted, skip + cascade
if (step.depends_on && step.depends_on.some((d) => abortedIds.has(d))) {
submitted.push({ step: stepCount, id: step.id, job_id: null, status: 'skipped_dep_aborted' });
abortedIds.add(step.id);
recs.shift();
continue;
}
try {
const isProtected = !!step.protected;
const job = await queue.add(
step.job,
{ ...step.params, doctor_run_id: doctorRunId },
{
queue: 'default',
idempotency_key: step.idempotency_key,
max_attempts: 2,
maxWaiting: 1,
},
isProtected ? { allowProtectedSubmit: true } : undefined,
);
submitted.push({ step: stepCount, id: step.id, job_id: job.id, status: 'submitted' });
// Wait for terminal state. PGLite is in-process — short poll.
const terminal = await waitForCompletion(queue, job.id, {
pollMs: isPGLite ? 250 : 1000,
timeoutMs: (step.est_seconds + 60) * 1000,
});
const lastSub = submitted[submitted.length - 1];
if (lastSub) lastSub.status = terminal.status;
if (terminal.status !== 'completed') {
abortedIds.add(step.id);
}
} catch (e) {
submitted.push({
step: stepCount, id: step.id, job_id: null,
status: `error: ${(e as Error).message.slice(0, 100)}`,
});
abortedIds.add(step.id);
}
recs.shift();
// D7: scoped recheck — re-compute plan from fresh health snapshot.
// The next plan may drop completed steps and re-introduce failed
// steps with bumped retry suffix (D1).
if (recs.length === 0 || stepCount >= maxJobs) break;
const freshHealth = await engine.getHealth();
recs = computeRecommendations(freshHealth, ctx).filter((r) => r.status === 'remediable');
}
const finalHealth = await engine.getHealth();
const result = {
doctor_run_id: doctorRunId,
brain_score_initial: initialHealth.brain_score,
brain_score_final: finalHealth.brain_score,
brain_score_target: targetScore,
target_reached: finalHealth.brain_score >= targetScore,
submitted,
aborted_count: abortedIds.size,
};
if (jsonOutput) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`\nBrain score: ${initialHealth.brain_score} → ${finalHealth.brain_score} (target ${targetScore})`);
console.log(`Submitted: ${submitted.length} job(s), ${abortedIds.size} aborted/failed`);
}
const anyFailed = submitted.some((s) => s.status !== 'completed' && s.status !== 'submitted');
if (anyFailed) process.exit(1);
}
/**
* Build RecommendationContext from engine + config.
* Pure read; no side effects.
*/
async function loadRecommendationContext(engine: BrainEngine) {
const repoPath = await engine.getConfig('sync.repo_path');
const embeddingModel = await engine.getConfig('embedding_model');
const embeddingDimensions = await engine.getConfig('embedding_dimensions');
return {
repoPath: repoPath ?? undefined,
embeddingModel: embeddingModel ?? undefined,
embeddingDimensions: embeddingDimensions ? Number(embeddingDimensions) : undefined,
hasEmbeddingApiKey: !!(process.env.OPENAI_API_KEY || await engine.getConfig('openai_api_key')),
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || await engine.getConfig('anthropic_api_key')),
};
}
function parseIntFlag(args: string[], flag: string): number | null {
const i = args.indexOf(flag);
if (i === -1 || i === args.length - 1) return null;
const v = parseInt(args[i + 1] ?? '', 10);
return isNaN(v) ? null : v;
}
function parseFloatFlag(args: string[], flag: string): number | null {
const i = args.indexOf(flag);
if (i === -1 || i === args.length - 1) return null;
const v = parseFloat(args[i + 1] ?? '');
return isNaN(v) ? null : v;
}
src/commands/embed.ts
+25
View File
@@ -98,6 +98,31 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
}
export async function runEmbed(engine: BrainEngine, args: string[]): Promise<EmbedResult | undefined> {
// v0.36+ T7: --background submits via Minion queue, returns job_id to
// stdout, exits. Same semantics in TTY and cron (D9).
if (args.includes('--background')) {
const { maybeBackground } = await import('../core/cli-options.ts');
const backgrounded = await maybeBackground({
engine,
args,
jobName: 'embed',
paramBuilder: (cleanArgs) => {
const slugsI = cleanArgs.indexOf('--slugs');
const srcI = cleanArgs.indexOf('--source');
return {
all: cleanArgs.includes('--all'),
stale: cleanArgs.includes('--stale'),
dryRun: cleanArgs.includes('--dry-run'),
slugs: slugsI >= 0 ? cleanArgs.slice(slugsI + 1).filter(a => !a.startsWith('--')) : undefined,
sourceId: srcI >= 0 ? cleanArgs[srcI + 1] : undefined,
};
},
source: 'cli',
});
if (backgrounded) return;
// PGLite degraded to inline — fall through.
}
const slugsIdx = args.indexOf('--slugs');
const all = args.includes('--all');
const stale = args.includes('--stale');
src/commands/jobs.ts
+104 -1
View File
@@ -1027,8 +1027,15 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
const concurrencyOverride = typeof job.data.concurrency === 'number'
? job.data.concurrency
: undefined;
// v0.36+ codex #5 fix: standalone `sync` handler now passes
// noExtract:true so doctor's remediation plan [sync, extract] doesn't
// double-extract (performSync inline-extract + standalone extract job).
// Pre-fix, runPhaseSync in cycle.ts passed noExtract:true but the
// standalone handler dropped it. Callers that want inline extract can
// pass { noExtract: false } in job params explicitly.
const noExtract = job.data.noExtract !== false;
const result = await performSync(engine, {
repoPath, sourceId, noPull, noEmbed,
repoPath, sourceId, noPull, noEmbed, noExtract,
concurrency: concurrencyOverride,
});
return result;
@@ -1165,6 +1172,102 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
worker.register('subagent_aggregator', subagentAggregatorHandler);
process.stderr.write('[minion worker] subagent handlers enabled\n');
// ============================================================
// v0.36+ brain-health-100 wave: 11 new handlers for autonomous
// remediation via `gbrain doctor --remediate` and autopilot.
//
// PROTECTED via PROTECTED_JOB_NAMES (D11): synthesize, patterns,
// consolidate — they internally submit `subagent` jobs with
// allowProtectedSubmit=true, so they CAN spend Anthropic credits.
// Open handlers (DB writes only): reindex, repair-jsonb, orphans,
// integrity, purge, extract_facts, resolve_symbol_edges,
// recompute_emotional_weight.
// ============================================================
worker.register('reindex', async (job) => {
const { runReindex } = await import('./reindex.ts');
const args: string[] = ['--markdown'];
if (typeof job.data.limit === 'number') args.push('--limit', String(job.data.limit));
if (job.data.dryRun) args.push('--dry-run');
if (job.data.noEmbed) args.push('--no-embed');
if (typeof job.data.repoPath === 'string') args.push('--repo', job.data.repoPath);
const result = await runReindex(engine, args);
return { ...result, ran: 'reindex' };
});
worker.register('repair-jsonb', async (job) => {
const { repairJsonb } = await import('./repair-jsonb.ts');
const dryRun = !!job.data.dryRun;
const result = await repairJsonb({ dryRun });
return result;
});
worker.register('orphans', async (_job) => {
const result = await engine.findOrphanPages();
return { count: result.length, orphans: result };
});
worker.register('integrity', async (job) => {
const { runIntegrity } = await import('./integrity.ts');
const args: string[] = [];
args.push(job.data.mode === 'auto' ? 'auto' : 'check');
if (typeof job.data.confidence === 'number') args.push('--confidence', String(job.data.confidence));
if (job.data.dryRun) args.push('--dry-run');
await runIntegrity(args);
return { ran: 'integrity', mode: args[0] };
});
worker.register('purge', async (job) => {
const scope = (typeof job.data.scope === 'string' && ['pages', 'sources', 'all'].includes(job.data.scope))
? (job.data.scope as 'pages' | 'sources' | 'all')
: 'all';
const olderThanHours = typeof job.data.olderThanHours === 'number' ? job.data.olderThanHours : 72;
const dryRun = !!job.data.dryRun;
let pagesPurged = 0;
let sourcesPurged: string[] = [];
if (scope === 'pages' || scope === 'all') {
const result = await engine.purgeDeletedPages(olderThanHours);
pagesPurged = result.count;
}
if (scope === 'sources' || scope === 'all') {
const { purgeExpiredSources } = await import('../core/destructive-guard.ts');
sourcesPurged = await purgeExpiredSources(engine);
}
// GC stale op_checkpoints rows (folded scope item +C from review).
const { purgeStaleCheckpoints } = await import('../core/op-checkpoint.ts');
const checkpointsPurged = await purgeStaleCheckpoints(engine, 7);
return { pagesPurged, sourcesPurged, checkpointsPurged, dryRun };
});
// Phase-wrapper handlers — each delegates to runCycle({ phases: [name] }).
// Cycle owns the lock + abort signal + progress reporter per D10.
// Smaller diff than full standalone phase extraction; cycle.ts remains
// the single source of truth for phase semantics.
const makePhaseHandler = (phase: string) => async (job: any) => {
const { runCycle } = await import('../core/cycle.ts');
const repoPath = typeof job.data.repoPath === 'string'
? job.data.repoPath
: ((await engine.getConfig('sync.repo_path')) ?? '.');
const report = await runCycle(engine, {
brainDir: repoPath,
phases: [phase as any],
signal: job.signal,
});
return { phase, status: report.status, report };
};
// PROTECTED — internally spawn subagent children
worker.register('synthesize', makePhaseHandler('synthesize'));
worker.register('patterns', makePhaseHandler('patterns'));
worker.register('consolidate', makePhaseHandler('consolidate'));
// Open — DB writes only, no LLM spend
worker.register('extract_facts', makePhaseHandler('extract_facts'));
worker.register('resolve_symbol_edges', makePhaseHandler('resolve_symbol_edges'));
worker.register('recompute_emotional_weight', makePhaseHandler('recompute_emotional_weight'));
process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected)\n');
// Plugin discovery — one line per discovered plugin (mirrors the
// openclaw-seam startup line convention from v0.11+). Loaded
// unconditionally; empty GBRAIN_PLUGIN_PATH is a no-op.
src/core/brain-score-recommendations.ts
+364
View File
@@ -0,0 +1,364 @@
import { createHash } from 'crypto';
import type { BrainHealth } from './types.ts';
import { ANTHROPIC_PRICING } from './anthropic-pricing.ts';
import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts';
/** Minimal Check shape consumed by classifyChecks. Subset of doctor.ts's
* Check; we intentionally don't import from doctor.ts (would create a
* cycle: doctor → recommendations → doctor). */
export interface Check {
name: string;
status: 'ok' | 'warn' | 'fail';
}
/**
* Shared recommendation generator for brain-health remediation.
*
* Consumed by both:
* - `gbrain doctor --remediation-plan` / `--remediate` (queue-based execution)
* - `gbrain features --auto-fix` (inline execution preserved per D15)
*
* Pure module — no engine I/O. Input is `BrainHealth` (already produced by
* engine.getHealth()) + a `RecommendationContext` that names which prereqs
* are met (API keys, repo path, source id).
*
* Three-state classification per check (D13):
* - `remediable`: a job exists AND prereqs are met. Emit it.
* - `human_only`: no autofix (orphans archive, multi_source_drift,
* eval_drift). Surface as informational; don't queue.
* - `blocked`: autofix exists, prereq missing (e.g. missing API key
* for embed). Surface with the missing prereq; don't queue.
*
* `maxReachableScore(health, classifications)` computes the score ceiling
* assuming only `remediable` checks fire. Callers refuse `--target-score >
* ceiling` so empty / API-key-missing brains don't spin forever.
*
* Plan: D13 + D14 + folded scope item A (cost-budget gate) from outside-voice
* review. See ~/.claude/plans/system-instruction-you-are-working-fluttering-ocean.md.
*/
/**
* Severity buckets — drive ordering (critical first) and operator UX
* (in the human-readable plan output, critical items get a louder prefix).
*/
export type RemediationSeverity = 'critical' | 'high' | 'medium' | 'low';
/** Status of an individual check's autofix path. */
export type RemediationStatus = 'remediable' | 'human_only' | 'blocked';
export interface Remediation {
/** Stable id (e.g. 'sync.repo', 'embed.stale', 'embed.code'). Survives
* check renames; referenced by `depends_on`. */
id: string;
/** Minion handler name (must match a registered handler). */
job: string;
/** Params passed to the handler. */
params: Record<string, unknown>;
/** Content-hash idempotency key: `<source>:<job>:sha8(canonical-JSON(params))`.
* Per D9: NO time-slot. Retry suffix `:r<N>` appended by --remediate when
* prior key's job is in `failed`/`dead` state. */
idempotency_key: string;
severity: RemediationSeverity;
/** Upper-bound runtime estimate for ordering + --target-score budgeting. */
est_seconds: number;
/** USD cost estimate when applicable (embed by chunk count × $/MTok;
* synthesize / patterns / consolidate by est Sonnet calls × $/MTok).
* Sum across plan steps is the gate for --max-usd. */
est_usd_cost?: number;
/** Other Remediation.id values that MUST complete first. References ids,
* NOT check names (D14 — closes codex #22 fan-out ambiguity). */
depends_on?: string[];
/** One-line "what this fixes" for human output. */
rationale: string;
/** True if `job` is in PROTECTED_JOB_NAMES (D11). Mirrors trust-gate state
* so callers don't have to re-derive it. */
protected?: boolean;
/** Always 'remediable' when this struct is in the plan. The doctor surface
* also produces blocked-entries (with `blocked_reason`) but those don't
* enter the executable plan. */
status: RemediationStatus;
/** Populated when status === 'blocked'. E.g. 'missing OPENAI_API_KEY'. */
blocked_reason?: string;
}
export interface RecommendationContext {
/** Source id this remediation is scoped to (multi-source brains). */
sourceId?: string;
/** Brain repo path on disk (for sync). */
repoPath?: string;
/** Configured embedding model id (e.g. 'openai:text-embedding-3-large'). */
embeddingModel?: string;
/** Configured embedding dimension (3072 / 1536 / 1024 / etc.). */
embeddingDimensions?: number;
/** Whether the embedding provider has a usable API key. */
hasEmbeddingApiKey?: boolean;
/** Configured chat / synthesis model id. */
chatModel?: string;
/** Whether the chat provider has a usable API key. */
hasChatApiKey?: boolean;
}
/** Triage result for one check. */
export interface CheckClassification {
check: string;
status: RemediationStatus;
/** When status !== 'remediable', what's missing. */
reason?: string;
}
/**
* Generate ordered Remediation list from health snapshot + context.
*
* Sort: severity (critical > high > medium > low), then est_seconds asc.
* Topological order over `depends_on` is the caller's job — they walk this
* list and respect dependencies. Recommendation generator just picks order
* within a strata.
*
* Returns ONLY `remediable` items. `blocked` items surface via
* `classifyChecks()` and are rendered alongside the plan as informational.
*/
export function computeRecommendations(
health: BrainHealth,
ctx: RecommendationContext,
): Remediation[] {
const out: Remediation[] = [];
const source = ctx.sourceId ?? 'default';
// ---------------------------------------------------------------------
// sync.repo — fires when sync hasn't run recently OR pages are stale
// ---------------------------------------------------------------------
if (ctx.repoPath && health.stale_pages > 0) {
const params = { repoPath: ctx.repoPath, sourceId: ctx.sourceId, noEmbed: true };
out.push({
id: 'sync.repo',
job: 'sync',
params,
idempotency_key: idemKey(source, 'sync', params),
severity: health.stale_pages > 50 ? 'high' : 'medium',
est_seconds: Math.min(600, 30 + health.stale_pages * 0.5),
est_usd_cost: 0, // sync is fs+DB only
depends_on: [],
rationale: `${health.stale_pages} stale page${health.stale_pages === 1 ? '' : 's'} on disk`,
status: 'remediable',
});
}
// ---------------------------------------------------------------------
// embed.stale — missing embeddings. Critical: invisible to vector search
// ---------------------------------------------------------------------
if (health.missing_embeddings > 0 && ctx.hasEmbeddingApiKey !== false) {
const params = { stale: true, sourceId: ctx.sourceId };
const embedModel = ctx.embeddingModel ?? 'openai:text-embedding-3-large';
const embedDims = ctx.embeddingDimensions ?? 3072;
// Rough char estimate per chunk ~ 1.5k chars (chunker target).
const estChars = health.missing_embeddings * 1500;
let est_usd_cost = 0;
try {
const priceLookup = lookupEmbeddingPrice(embedModel);
if (priceLookup.kind === 'known') {
est_usd_cost = estimateCostFromChars(estChars, priceLookup.pricePerMTok);
}
} catch {
/* unknown model — leave at 0, surface as warning elsewhere */
}
out.push({
id: 'embed.stale',
job: 'embed',
params,
idempotency_key: idemKey(source, 'embed', { ...params, embedModel, embedDims }),
severity: 'critical',
est_seconds: Math.min(3600, 5 + health.missing_embeddings * 0.05),
est_usd_cost,
// sync should run first so embed sees fresh pages.
depends_on: ctx.repoPath && health.stale_pages > 0 ? ['sync.repo'] : [],
rationale: `${health.missing_embeddings} chunk${health.missing_embeddings === 1 ? '' : 's'} invisible to vector search`,
status: 'remediable',
});
}
// ---------------------------------------------------------------------
// backlinks.fix — dead links (refs to non-existent slugs)
// ---------------------------------------------------------------------
if (health.dead_links > 0 && ctx.repoPath) {
const params = { action: 'fix', dir: ctx.repoPath };
out.push({
id: 'backlinks.fix',
job: 'backlinks',
params,
idempotency_key: idemKey(source, 'backlinks', params),
severity: 'high',
est_seconds: Math.min(300, 10 + health.dead_links * 0.5),
est_usd_cost: 0,
depends_on: [],
rationale: `${health.dead_links} dead link${health.dead_links === 1 ? '' : 's'}`,
status: 'remediable',
});
}
// ---------------------------------------------------------------------
// extract.all — runs after sync to materialize links + timeline.
// Triggered when sync.repo fires (because sync was set to noEmbed:true,
// and noExtract:true after T5 lands → extract job is the materializer).
// ---------------------------------------------------------------------
if (ctx.repoPath && health.stale_pages > 0) {
const params = { mode: 'all', dir: ctx.repoPath };
out.push({
id: 'extract.all',
job: 'extract',
params,
idempotency_key: idemKey(source, 'extract', params),
severity: 'medium',
est_seconds: Math.min(600, 30 + health.page_count * 0.01),
est_usd_cost: 0,
depends_on: ['sync.repo'],
rationale: 'Materialize link + timeline edges from fresh pages',
status: 'remediable',
});
}
// Sort: severity (critical first), then est_seconds ascending so quick
// wins come first within a severity tier.
const sevRank: Record<RemediationSeverity, number> = {
critical: 0, high: 1, medium: 2, low: 3,
};
out.sort((a, b) => {
const sd = sevRank[a.severity] - sevRank[b.severity];
if (sd !== 0) return sd;
return a.est_seconds - b.est_seconds;
});
return out;
}
/**
* Triage every check from the doctor report into one of three buckets.
* Used by the doctor remediation surface to surface what's not auto-fixable
* (or auto-fixable-but-prereq-missing) as informational alongside the plan.
*
* Checks not listed here default to `human_only` (conservative — anything
* the recommendation generator doesn't know about is treated as needing
* operator judgment, not autonomous remediation).
*/
export function classifyChecks(
checks: Check[],
ctx: RecommendationContext,
): CheckClassification[] {
return checks.map((c) => classifyOne(c, ctx));
}
function classifyOne(check: Check, ctx: RecommendationContext): CheckClassification {
// Map check names to their remediation status. The recommendation
// generator above handles `remediable`; this maps the rest.
switch (check.name) {
// --- remediable paths (matched by recommendation generator) ---
case 'brain_score':
case 'sync_freshness':
if (!ctx.repoPath) {
return { check: check.name, status: 'blocked', reason: 'no repo configured (set sync.repo_path)' };
}
return { check: check.name, status: 'remediable' };
case 'missing_embeddings':
if (ctx.hasEmbeddingApiKey === false) {
return { check: check.name, status: 'blocked', reason: 'missing embedding API key' };
}
return { check: check.name, status: 'remediable' };
case 'dead_links':
if (!ctx.repoPath) {
return { check: check.name, status: 'blocked', reason: 'no repo configured' };
}
return { check: check.name, status: 'remediable' };
// --- human_only paths ---
case 'orphan_pages': // archive is product judgment, not maintenance
case 'multi_source_drift':
case 'eval_drift':
case 'slug_fallback_audit':
case 'whoknows_health':
case 'rls_event_trigger': // operator must intervene
case 'reranker_health':
return { check: check.name, status: 'human_only', reason: 'no autonomous remediation' };
default:
// Unknown checks: conservative default. Surfaces as informational
// rather than blocking the loop.
return { check: check.name, status: 'human_only', reason: 'unmapped check' };
}
}
/**
* Compute the score ceiling assuming only `remediable` checks fire.
*
* Each component of brain_score (embed_coverage 35, link_density 25,
* timeline_coverage 15, no_orphans 15, no_dead_links 10) maps to a
* remediable or non-remediable classification. Components without an
* autofix path stay at their current score; remediable components can
* theoretically reach their max.
*
* Returns the ceiling; --remediate refuses --target-score > ceiling
* with a clear "this brain can only reach X without manual intervention"
* error.
*/
export function maxReachableScore(
health: BrainHealth,
classifications: CheckClassification[],
): number {
const classMap = new Map(classifications.map((c) => [c.check, c.status]));
// Component → max contribution + remediability
// Conservative: if the mapped check is NOT remediable, the component
// stays at its current value (can't be lifted by autonomous action).
let ceiling = 0;
ceiling += pickMax(health.embed_coverage_score, 35, classMap.get('missing_embeddings'));
ceiling += pickMax(health.link_density_score, 25, classMap.get('dead_links'));
ceiling += pickMax(health.timeline_coverage_score, 15, undefined); // no current autofix
ceiling += pickMax(health.no_orphans_score, 15, classMap.get('orphan_pages'));
ceiling += pickMax(health.no_dead_links_score, 10, classMap.get('dead_links'));
return Math.min(100, Math.round(ceiling));
}
function pickMax(current: number, max: number, status: RemediationStatus | undefined): number {
if (status === 'remediable') return max;
return current;
}
// ---------------------------------------------------------------------
// Idempotency key construction (D9 — content-hash, no time-slot).
// Same params produce the same key across runs. Failed-row replay
// appends `:r<N>` (caller responsibility — handled by --remediate loop).
// ---------------------------------------------------------------------
function idemKey(source: string, job: string, params: Record<string, unknown>): string {
return `${source}:${job}:${sha8(canonicalJson(params))}`;
}
function sha8(s: string): string {
return createHash('sha256').update(s).digest('hex').slice(0, 8);
}
function canonicalJson(value: unknown): string {
if (value === null || typeof value !== 'object') return JSON.stringify(value);
if (Array.isArray(value)) return `[${value.map(canonicalJson).join(',')}]`;
const keys = Object.keys(value as Record<string, unknown>).sort();
return `{${keys.map((k) => `${JSON.stringify(k)}:${canonicalJson((value as Record<string, unknown>)[k])}`).join(',')}}`;
}
/**
* Returns the per-recommendation USD cost ceiling for an Anthropic-model
* job. Used by synthesize/patterns/consolidate cost estimates.
*
* `estCallsPerInvocation` is a per-job heuristic (e.g. synthesize ~
* 20 calls per invocation; patterns ~ 5). Multiplied by per-call token
* budget × Anthropic-model price.
*/
export function estimateAnthropicCost(
modelId: string,
estCallsPerInvocation: number,
estInputTokensPerCall = 5_000,
estOutputTokensPerCall = 1_000,
): number {
const pricing = ANTHROPIC_PRICING[modelId];
if (!pricing) return 0;
const inputCost = (estInputTokensPerCall * estCallsPerInvocation / 1_000_000) * pricing.input;
const outputCost = (estOutputTokensPerCall * estCallsPerInvocation / 1_000_000) * pricing.output;
return Number((inputCost + outputCost).toFixed(2));
}
src/core/cli-options.ts
+101
View File
@@ -193,3 +193,104 @@ export function childGlobalFlags(cliOpts?: CliOptions): string {
}
return parts.length > 0 ? ' ' + parts.join(' ') : '';
}
// ============================================================
// v0.36+ brain-health-100 wave: --background flag (D9 + T7)
//
// Per the locked decision: --background means submit-and-exit ALWAYS.
// Same semantics in TTY and cron. Composable in shell pipelines:
//
// JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2)
// gbrain jobs get $JOB
//
// `--background --follow` submits then execs `gbrain jobs follow <id>`
// so the user sees live stream while still getting durable queue
// semantics (worker survives if user disconnects).
//
// PGLite degrades to inline with a clear stderr note. NOT a no-op,
// NOT silent. Doc-stated semantic difference because PGLite has no
// worker daemon.
// ============================================================
import type { BrainEngine } from './engine.ts';
import { createHash } from 'crypto';
export interface MaybeBackgroundOpts {
engine: BrainEngine;
args: string[];
jobName: string;
paramBuilder: (args: string[]) => Record<string, unknown>;
/** Source id for the idempotency key namespace. Default 'cli'. */
source?: string;
}
/**
* If `--background` is in args, submit a Minion job and return true
* (caller should exit). Otherwise return false (caller does inline work).
*
* Strips `--background` and `--follow` from args before paramBuilder
* runs so the param shape stays clean. On submit failure, prints stderr
* + exits 1 (no orphan job; no silent fallthrough to inline).
*
* @returns true if backgrounded (caller MUST exit), false otherwise.
*/
export async function maybeBackground(opts: MaybeBackgroundOpts): Promise<boolean> {
if (!opts.args.includes('--background')) return false;
const filtered = opts.args.filter((a) => a !== '--background' && a !== '--follow');
const params = opts.paramBuilder(filtered);
const follow = opts.args.includes('--follow');
const source = opts.source ?? 'cli';
// PGLite has no worker daemon. Per the doc-stated semantics, degrade
// to inline with a clear stderr note rather than silently failing.
if (opts.engine.kind === 'pglite') {
process.stderr.write(
`[--background] PGLite has no worker daemon; running inline.\n`,
);
return false; // caller runs inline
}
// D9: content-hash idempotency key. No time-slot — same intent = same
// key. Failed-row replay is the doctor --remediate loop's job, not
// the CLI --background path's job.
const idempotency_key = `${source}:${opts.jobName}:${sha8(canonicalJson(params))}`;
try {
const { MinionQueue } = await import('./minions/queue.ts');
const queue = new MinionQueue(opts.engine);
const job = await queue.add(opts.jobName, params, {
queue: 'default',
idempotency_key,
max_attempts: 2,
});
process.stdout.write(`job_id=${job.id}\n`);
if (follow) {
// exec `gbrain jobs follow <id>` so the user sees live stream
// without losing the durable-queue submission.
const { spawn } = await import('child_process');
const cmd = process.argv[0] ?? 'bun';
const script = process.argv[1] ?? '';
const child = spawn(cmd, [script, 'jobs', 'follow', String(job.id)], {
stdio: 'inherit',
});
await new Promise<void>((resolve) => child.on('exit', () => resolve()));
}
return true; // caller exits
} catch (e) {
process.stderr.write(`[--background] submit failed: ${(e as Error).message}\n`);
process.exit(1);
}
}
function sha8(s: string): string {
return createHash('sha256').update(s).digest('hex').slice(0, 8);
}
function canonicalJson(value: unknown): string {
if (value === null || typeof value !== 'object') return JSON.stringify(value);
if (Array.isArray(value)) return `[${value.map(canonicalJson).join(',')}]`;
const keys = Object.keys(value as Record<string, unknown>).sort();
return `{${keys.map((k) => `${JSON.stringify(k)}:${canonicalJson((value as Record<string, unknown>)[k])}`).join(',')}}`;
}
src/core/cycle.ts
+13 -2
View File
@@ -968,13 +968,23 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
const purgedSources = await purgeExpiredSources(engine);
const purgedPages = await engine.purgeDeletedPages(SOFT_DELETE_TTL_HOURS_FOR_PURGE);
const purgedClones = await purgeOrphanClones(SOFT_DELETE_TTL_HOURS_FOR_PURGE);
// v0.36+ folded scope item +C: GC stale op_checkpoints rows.
// 7-day TTL is deliberately generous; any reasonable long-running op
// finishes inside that window. Cheap (few KB per row).
let purgedCheckpoints = 0;
try {
const { purgeStaleCheckpoints } = await import('./op-checkpoint.ts');
purgedCheckpoints = await purgeStaleCheckpoints(engine, 7);
} catch {
// Non-fatal: op_checkpoints table may not exist yet on pre-v67 brains.
}
return {
phase: 'purge',
status: 'ok',
duration_ms: 0,
summary:
`purged ${purgedSources.length} source(s), ${purgedPages.count} page(s), and ` +
`${purgedClones.count} orphan clone temp dir(s) past the 72h recovery window`,
`purged ${purgedSources.length} source(s), ${purgedPages.count} page(s), ` +
`${purgedClones.count} orphan clone temp dir(s), and ${purgedCheckpoints} stale op_checkpoint(s)`,
details: {
purged_sources_count: purgedSources.length,
purged_pages_count: purgedPages.count,
@@ -982,6 +992,7 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
purged_orphan_clone_names: purgedClones.names,
purged_sources: purgedSources,
purged_page_slugs: purgedPages.slugs,
purged_checkpoints_count: purgedCheckpoints,
},
};
} catch (e) {
src/core/migrate.ts
+57
View File
@@ -3546,6 +3546,63 @@ export const MIGRATIONS: Migration[] = [
`,
},
},
{
version: 75,
name: 'op_checkpoints_table',
// v0.36+ autonomous-remediation wave (renumbered v67→v75 during master
// merge — master's v0.36.1.0 calibration + v0.36.3.0 captured v67-v74).
// Shared checkpoint table for long-running ops (embed, extract, lint,
// backlinks, reindex, integrity). Pre-fix, each op had its own
// file-backed checkpoint (or none), which broke on Postgres multi-worker
// hosts and silently fingerprint-collided across param variations
// (extract links vs extract timeline shared one file). DB-backed primary;
// PGLite engine falls back to file-backed at
// ~/.gbrain/checkpoints/<op>-<fingerprint>.json because it's single-host
// by construction.
//
// Fingerprint = sha8 of canonical-JSON of relevant params per op
// (chunker_version + embedding_model for embed, mode for extract, etc.).
// completed_keys are op-defined strings: chunk ids for embed, file paths
// for extract/lint/backlinks/reindex, page slugs for integrity.
//
// GC: cycle's purge phase drops rows older than 7 days.
idempotent: true,
sql: `
CREATE TABLE IF NOT EXISTS op_checkpoints (
op TEXT NOT NULL,
fingerprint TEXT NOT NULL,
completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (op, fingerprint)
);
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
`,
},
{
version: 76,
name: 'minion_jobs_doctor_run_id_index',
// v0.36+ autonomous-remediation wave (renumbered v68→v76 during master
// merge). Partial GIN on minion_jobs.data for `data ? 'doctor_run_id'`.
// Lets `gbrain doctor --remediate` runs be queried by run id for audit
// trail without sequential-scanning months of cron history. Partial so
// only doctor-submitted jobs are indexed; ordinary cron submissions
// don't bloat the index.
//
// PGLite skips via empty sqlFor — JSONB GIN partial indexes aren't
// supported the same way; audit query falls through to sequential
// scan, which is fine for PGLite's single-host scope.
idempotent: true,
sql: '',
sqlFor: {
postgres: `
CREATE INDEX IF NOT EXISTS minion_jobs_doctor_run_id_idx
ON minion_jobs USING GIN (data jsonb_path_ops)
WHERE data ? 'doctor_run_id';
`,
pglite: '',
},
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
src/core/minions/protected-names.ts
+11
View File
@@ -20,6 +20,17 @@ export const PROTECTED_JOB_NAMES: ReadonlySet<string> = new Set([
// trusted local `submit_job` (ctx.remote=false) can insert these rows.
'subagent',
'subagent_aggregator',
// v0.36+ brain-health-100 wave (D11 from outside-voice review):
// synthesize, patterns, consolidate are cycle phases that internally
// submit `subagent` children with allowProtectedSubmit=true. Treating
// them as "data-quality maintenance" was a misread — they CAN run
// Sonnet loops costing user money. Protected ensures only trusted
// local callers (CLI, autopilot, doctor --remediate) can submit them;
// an OAuth-scoped MCP client can't burn the user's API budget by
// submitting a synthesize job over HTTP.
'synthesize',
'patterns',
'consolidate',
]);
/** Check a job name against the protected set. Normalizes whitespace first. */
src/core/op-checkpoint.ts
+320
View File
@@ -0,0 +1,320 @@
import { createHash } from 'crypto';
import type { BrainEngine } from './engine.ts';
/**
* Shared checkpoint primitive for long-running ops (embed, extract, lint,
* backlinks, reindex, integrity, import).
*
* Pre-v0.36 each op had its own file-backed checkpoint or no checkpoint at
* all. Three bug classes died with this module:
*
* 1. **Param-shape collisions.** `extract links` and `extract timeline`
* walked the same files but shared a single checkpoint, so a killed
* `links` run made `timeline` skip files (codex #11). `reindex
* --markdown` and `reindex --code` had the same issue. Fix: every
* checkpoint is keyed by `(op, fingerprint)` where fingerprint is
* sha8 of canonical-JSON of the relevant params per op.
*
* 2. **Multi-worker host blindness.** File-backed `~/.gbrain/...`
* checkpoints don't work when the Minion worker resumes on a
* different container or host (codex #16). DB-backed via
* `op_checkpoints` table (migration v67) is the source of truth;
* cross-host workers read the same row.
*
* 3. **Stale-row corruption window.** Per-op JSONL append-only files
* (integrity's pre-v0.36 path) corrupted on partial writes. JSONB
* column with single UPSERT is atomic.
*
* GC: cycle's `purge` phase drops rows older than 7 days where the op
* completed cleanly. Bounded growth, no operator action required.
*
* @example Embed: per-chunk checkpoint keyed by model+dim variation
* const key = {
* op: 'embed',
* fingerprint: embedFingerprint({
* stale: true,
* source: 'default',
* embedding_model: 'openai:text-embedding-3-large',
* embedding_dimensions: 3072,
* }),
* };
* const done = new Set(await loadOpCheckpoint(engine, key));
* for (const chunk of allChunks) {
* if (done.has(chunk.id)) continue;
* await embed(chunk);
* done.add(chunk.id);
* if (done.size % 100 === 0) {
* await recordCompleted(engine, key, [...done]);
* }
* }
* await clearOpCheckpoint(engine, key); // success exit
*/
export interface OpCheckpointKey {
/** Op name; one of: 'embed', 'extract', 'lint', 'backlinks', 'reindex', 'integrity', 'import'. */
op: string;
/** sha8 of canonical-JSON of relevant params. See *Fingerprint functions below. */
fingerprint: string;
}
/**
* Load completed keys for an op invocation. Empty array when no checkpoint
* exists yet (first run, or after `clearOpCheckpoint`).
*
* Non-fatal on DB errors — returns `[]` and logs to stderr. The op then
* re-walks from zero, which is cheap for content-hash-short-circuited ops
* (embed checks `embedded_at`, import checks `content_hash`).
*/
export async function loadOpCheckpoint(
engine: BrainEngine,
key: OpCheckpointKey,
): Promise<string[]> {
try {
const rows = await engine.executeRaw<{ completed_keys: unknown }>(
`SELECT completed_keys FROM op_checkpoints
WHERE op = $1 AND fingerprint = $2`,
[key.op, key.fingerprint],
);
if (rows.length === 0) return [];
const raw = rows[0]?.completed_keys;
// postgres.js returns JSONB as JS arrays; PGLite returns strings. Handle both.
const parsed = typeof raw === 'string' ? JSON.parse(raw) : raw;
if (!Array.isArray(parsed)) return [];
return parsed.filter((k): k is string => typeof k === 'string');
} catch (e) {
console.error(`[op-checkpoint] load failed (${key.op}, ${key.fingerprint}):`, (e as Error).message);
return [];
}
}
/**
* Persist the completed-keys set. Caller chooses cadence (typical: every
* 100 successful items). Atomic UPSERT; PRIMARY KEY (op, fingerprint)
* makes ON CONFLICT a single-row DO UPDATE.
*
* Non-fatal on DB errors — logs and continues. Lost checkpoint just means
* re-walk on next run, which is cheap for hash-short-circuited ops.
*/
export async function recordCompleted(
engine: BrainEngine,
key: OpCheckpointKey,
keys: string[],
): Promise<void> {
try {
// Sorted serialization keeps diff-based debug output stable and tests
// deterministic across insertion order shuffles.
const sorted = [...keys].sort();
await engine.executeRaw(
`INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
VALUES ($1, $2, $3::jsonb, now())
ON CONFLICT (op, fingerprint) DO UPDATE
SET completed_keys = EXCLUDED.completed_keys,
updated_at = now()`,
[key.op, key.fingerprint, JSON.stringify(sorted)],
);
} catch (e) {
console.error(`[op-checkpoint] write failed (${key.op}, ${key.fingerprint}):`, (e as Error).message);
/* non-fatal: lost checkpoint just means re-walk on next run */
}
}
/**
* Drop the checkpoint after a clean exit. Idempotent; missing row is a
* no-op.
*
* Cycle's `purge` phase ALSO sweeps stale rows on a 7-day TTL, so callers
* that crash without reaching this won't leak forever.
*/
export async function clearOpCheckpoint(
engine: BrainEngine,
key: OpCheckpointKey,
): Promise<void> {
try {
await engine.executeRaw(
`DELETE FROM op_checkpoints WHERE op = $1 AND fingerprint = $2`,
[key.op, key.fingerprint],
);
} catch (e) {
console.error(`[op-checkpoint] clear failed (${key.op}, ${key.fingerprint}):`, (e as Error).message);
/* non-fatal */
}
}
/**
* Filter `all` to elements not in the completed set. Pure function — no
* fs/db access — so consumers can drive batched processing without
* round-tripping the DB per item.
*
* ```
* const done = await loadOpCheckpoint(engine, key);
* const pending = resumeFilter(allFiles, done);
* for (const file of pending) { await process(file); }
* ```
*/
export function resumeFilter(all: string[], completed: string[]): string[] {
if (completed.length === 0) return all;
const done = new Set(completed);
return all.filter((k) => !done.has(k));
}
// ---------------------------------------------------------------------------
// Fingerprint helpers — one per op. The fingerprint MUST encode every param
// that produces a different processing decision per item. Two invocations
// with different fingerprints get separate checkpoints; two with identical
// fingerprints share one. Pick the dimensions deliberately.
// ---------------------------------------------------------------------------
/**
* Stable sha8 over the canonical-JSON of `params`. Same input → same hash
* across runs and across hosts. Stringify with sorted keys so a reorder of
* object literals doesn't flip the fingerprint.
*/
export function fingerprint(params: Record<string, unknown>): string {
return createHash('sha256').update(canonicalJson(params)).digest('hex').slice(0, 8);
}
function canonicalJson(value: unknown): string {
if (value === null || typeof value !== 'object') return JSON.stringify(value);
if (Array.isArray(value)) return `[${value.map(canonicalJson).join(',')}]`;
const keys = Object.keys(value as Record<string, unknown>).sort();
return `{${keys.map((k) => `${JSON.stringify(k)}:${canonicalJson((value as Record<string, unknown>)[k])}`).join(',')}}`;
}
/**
* Fingerprint for `embed`. Completed keys are chunk ids (string).
*
* Why these dims: re-embedding the same chunk with a different model or
* different dim produces a fundamentally different vector — they MUST be
* separate checkpoint rows so a switch from openai:3-large@1536 to
* voyage-3@1024 doesn't reuse the prior run's "done" set (codex #15).
*
* `slug` matters when `embed --slug X` re-embeds one page; a slug-scoped
* run shares no work with the brain-wide `--stale` run.
*/
export function embedFingerprint(p: {
stale?: boolean;
all?: boolean;
slug?: string;
source?: string;
embedding_model: string;
embedding_dimensions: number;
}): string {
return fingerprint({
stale: p.stale ?? false,
all: p.all ?? false,
slug: p.slug ?? null,
source: p.source ?? 'default',
embedding_model: p.embedding_model,
embedding_dimensions: p.embedding_dimensions,
});
}
/**
* Fingerprint for `extract`. Modes (`links` vs `timeline` vs `all`) walk
* the same files but produce different DB writes — they need separate
* checkpoints so killing a `links` run mid-walk doesn't make `timeline`
* skip the un-walked files (codex #11).
*/
export function extractFingerprint(p: {
mode: 'links' | 'timeline' | 'all';
source?: string;
dir?: string;
}): string {
return fingerprint({
mode: p.mode,
source: p.source ?? 'default',
dir: p.dir ?? null,
});
}
/**
* Fingerprint for `reindex`. `--markdown`, `--code`, and `--slug X` walk
* different page-kind subsets; each needs its own checkpoint (codex #12).
*
* `chunker_version` bumps invalidate the previous run's set because the
* new shape will rewrite chunks even on previously-completed pages.
*/
export function reindexFingerprint(p: {
markdown?: boolean;
code?: boolean;
slug?: string;
chunker_version: number;
}): string {
return fingerprint({
markdown: p.markdown ?? false,
code: p.code ?? false,
slug: p.slug ?? null,
chunker_version: p.chunker_version,
});
}
/**
* Fingerprint for `lint`. Auto-fix vs check-only walk the same files but
* produce different side effects — keep them separate.
*/
export function lintFingerprint(p: { dir: string; fix?: boolean }): string {
return fingerprint({ dir: p.dir, fix: p.fix ?? false });
}
/**
* Fingerprint for `backlinks`. `check` vs `fix` modes; same files, different
* side effects.
*/
export function backlinksFingerprint(p: { dir: string; action: 'check' | 'fix' }): string {
return fingerprint({ dir: p.dir, action: p.action });
}
/**
* Fingerprint for `integrity`. Mode + confidence threshold both shape the
* per-page processing decision.
*/
export function integrityFingerprint(p: {
mode: 'check' | 'auto';
confidence?: number;
}): string {
return fingerprint({
mode: p.mode,
confidence: p.confidence ?? 0.85,
});
}
/**
* Fingerprint for `import`. Source dir + per-import options uniquely
* identify the run. Used by the import-checkpoint shim.
*/
export function importFingerprint(p: {
dir: string;
noEmbed?: boolean;
source?: string;
}): string {
return fingerprint({
dir: p.dir,
noEmbed: p.noEmbed ?? false,
source: p.source ?? 'default',
});
}
/**
* Cycle's purge phase calls this to drop stale checkpoints. 7-day TTL is
* deliberately generous — any reasonable long-running op finishes inside
* that window, and the row is cheap (few KB).
*/
export async function purgeStaleCheckpoints(
engine: BrainEngine,
ttlDays = 7,
): Promise<number> {
try {
const rows = await engine.executeRaw<{ count: string | number }>(
`WITH deleted AS (
DELETE FROM op_checkpoints
WHERE updated_at < now() - ($1 || ' days')::interval
RETURNING 1
)
SELECT count(*)::text AS count FROM deleted`,
[String(ttlDays)],
);
return Number(rows[0]?.count ?? 0);
} catch (e) {
console.error('[op-checkpoint] purge failed:', (e as Error).message);
return 0;
}
}
src/core/pglite-schema.ts
+16
View File
@@ -760,6 +760,22 @@ CREATE TABLE IF NOT EXISTS oauth_codes (
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- ============================================================
-- op_checkpoints (v0.36+ autonomous-remediation wave, migration v67)
-- Shared checkpoint table for long-running ops. See migrate.ts:67 for
-- the design rationale; PGLite engine can also fall back to file-backed
-- storage per src/core/op-checkpoint.ts.
-- ============================================================
CREATE TABLE IF NOT EXISTS op_checkpoints (
op TEXT NOT NULL,
fingerprint TEXT NOT NULL,
completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (op, fingerprint)
);
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
-- ============================================================
-- Trigger-based search_vector (spans pages + timeline_entries)
-- ============================================================
src/core/schema-embedded.ts
+20
View File
@@ -470,6 +470,26 @@ CREATE TABLE IF NOT EXISTS oauth_codes (
CREATE INDEX IF NOT EXISTS idx_mcp_log_time_agent ON mcp_request_log(created_at, token_name);
CREATE INDEX IF NOT EXISTS idx_mcp_log_agent_time ON mcp_request_log(agent_name, created_at DESC);
-- ============================================================
-- op_checkpoints: shared checkpoint table for long-running ops
-- ============================================================
-- v0.36+ autonomous-remediation wave (migration v67). Pre-fix each op
-- carried its own file-backed checkpoint (or none); that broke on
-- Postgres multi-worker hosts and fingerprint-collided across param
-- variations. Fingerprint = sha8 of canonical-JSON of relevant params
-- per op (mode, source, chunker_version, embedding_model+dims, etc.).
-- completed_keys = op-defined string array. GC: cycle purge phase
-- drops rows older than 7 days.
CREATE TABLE IF NOT EXISTS op_checkpoints (
op TEXT NOT NULL,
fingerprint TEXT NOT NULL,
completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (op, fingerprint)
);
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
-- ============================================================
-- files: binary attachments stored in Supabase Storage
-- ============================================================
src/schema.sql
+20
View File
@@ -466,6 +466,26 @@ CREATE TABLE IF NOT EXISTS oauth_codes (
CREATE INDEX IF NOT EXISTS idx_mcp_log_time_agent ON mcp_request_log(created_at, token_name);
CREATE INDEX IF NOT EXISTS idx_mcp_log_agent_time ON mcp_request_log(agent_name, created_at DESC);
-- ============================================================
-- op_checkpoints: shared checkpoint table for long-running ops
-- ============================================================
-- v0.36+ autonomous-remediation wave (migration v67). Pre-fix each op
-- carried its own file-backed checkpoint (or none); that broke on
-- Postgres multi-worker hosts and fingerprint-collided across param
-- variations. Fingerprint = sha8 of canonical-JSON of relevant params
-- per op (mode, source, chunker_version, embedding_model+dims, etc.).
-- completed_keys = op-defined string array. GC: cycle purge phase
-- drops rows older than 7 days.
CREATE TABLE IF NOT EXISTS op_checkpoints (
op TEXT NOT NULL,
fingerprint TEXT NOT NULL,
completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (op, fingerprint)
);
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
-- ============================================================
-- files: binary attachments stored in Supabase Storage
-- ============================================================
test/brain-score-recommendations.test.ts
+236
View File
@@ -0,0 +1,236 @@
import { describe, test, expect } from 'bun:test';
import {
computeRecommendations,
classifyChecks,
maxReachableScore,
estimateAnthropicCost,
} from '../src/core/brain-score-recommendations.ts';
import type { BrainHealth } from '../src/core/types.ts';
/**
* D6 #5 + D13 + D14 pinning tests for brain-score-recommendations.
*
* Pure-function tests — no engine, no I/O. Every assertion below maps
* to an invariant the doctor-remediate loop assumes.
*/
function makeHealth(overrides: Partial<BrainHealth> = {}): BrainHealth {
return {
page_count: 100,
embed_coverage: 1.0,
stale_pages: 0,
orphan_pages: 0,
missing_embeddings: 0,
brain_score: 100,
dead_links: 0,
link_coverage: 1.0,
timeline_coverage: 1.0,
most_connected: [],
embed_coverage_score: 35,
link_density_score: 25,
timeline_coverage_score: 15,
no_orphans_score: 15,
no_dead_links_score: 10,
...overrides,
};
}
describe('computeRecommendations', () => {
test('healthy brain (score 100) produces empty plan', () => {
const health = makeHealth();
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
expect(recs).toEqual([]);
});
test('missing embeddings produces embed.stale remediation', () => {
const health = makeHealth({ missing_embeddings: 1432, brain_score: 65 });
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
const ids = recs.map((r) => r.id);
expect(ids).toContain('embed.stale');
const embedRec = recs.find((r) => r.id === 'embed.stale')!;
expect(embedRec.severity).toBe('critical');
expect(embedRec.job).toBe('embed');
expect(embedRec.params.stale).toBe(true);
});
test('missing embeddings + API key absent: NOT emitted (blocked surfaces separately)', () => {
const health = makeHealth({ missing_embeddings: 1432 });
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: false });
expect(recs.find((r) => r.id === 'embed.stale')).toBeUndefined();
});
test('stale pages + dead links produce sync + backlinks + extract', () => {
const health = makeHealth({
stale_pages: 25,
dead_links: 8,
brain_score: 70,
});
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
const ids = recs.map((r) => r.id);
expect(ids).toContain('sync.repo');
expect(ids).toContain('backlinks.fix');
expect(ids).toContain('extract.all');
});
test('extract.all depends on sync.repo (D14: stable ids)', () => {
const health = makeHealth({ stale_pages: 10 });
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
const extract = recs.find((r) => r.id === 'extract.all');
expect(extract?.depends_on).toContain('sync.repo');
});
test('embed.stale depends on sync.repo when sync also needed', () => {
const health = makeHealth({
stale_pages: 10,
missing_embeddings: 100,
});
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
const embed = recs.find((r) => r.id === 'embed.stale');
expect(embed?.depends_on).toContain('sync.repo');
});
test('embed.stale has no sync dependency when nothing stale', () => {
const health = makeHealth({ missing_embeddings: 100 });
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
const embed = recs.find((r) => r.id === 'embed.stale');
expect(embed?.depends_on).toEqual([]);
});
test('severity ordering: critical before high before medium', () => {
const health = makeHealth({
missing_embeddings: 100, // critical
stale_pages: 80, // high
});
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
const critIdx = recs.findIndex((r) => r.severity === 'critical');
const highIdx = recs.findIndex((r) => r.severity === 'high');
expect(critIdx).toBeLessThan(highIdx);
});
// D6 #5 — THE critical regression test for the agent contract.
test('D6 #5: determinism — same input twice produces identical output', () => {
const health = makeHealth({
stale_pages: 10,
missing_embeddings: 50,
dead_links: 3,
});
const ctx = { repoPath: '/brain', hasEmbeddingApiKey: true, sourceId: 'default' };
const run1 = computeRecommendations(health, ctx);
const run2 = computeRecommendations(health, ctx);
expect(JSON.stringify(run1)).toBe(JSON.stringify(run2));
});
test('D9: idempotency keys are content-hash, no time-slot', () => {
const health = makeHealth({ missing_embeddings: 50 });
const recs = computeRecommendations(health, {
repoPath: '/brain',
hasEmbeddingApiKey: true,
sourceId: 'default',
});
const embed = recs.find((r) => r.id === 'embed.stale')!;
// No date in the key — content-hash only
expect(embed.idempotency_key).not.toMatch(/\d{4}-\d{2}-\d{2}/);
// Format: source:job:sha8
expect(embed.idempotency_key).toMatch(/^default:embed:[a-f0-9]{8}$/);
});
test('D9: different sources produce different idempotency keys', () => {
const health = makeHealth({ missing_embeddings: 50 });
const a = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true, sourceId: 'A' });
const b = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true, sourceId: 'B' });
expect(a[0]!.idempotency_key).not.toBe(b[0]!.idempotency_key);
});
test('status field is always remediable in the output list (D13)', () => {
const health = makeHealth({ missing_embeddings: 50 });
const recs = computeRecommendations(health, { repoPath: '/brain', hasEmbeddingApiKey: true });
for (const r of recs) expect(r.status).toBe('remediable');
});
test('+A cost estimate populated for embed', () => {
const health = makeHealth({ missing_embeddings: 1000 });
const recs = computeRecommendations(health, {
repoPath: '/brain',
hasEmbeddingApiKey: true,
embeddingModel: 'openai:text-embedding-3-large',
embeddingDimensions: 3072,
});
const embed = recs.find((r) => r.id === 'embed.stale')!;
expect(embed.est_usd_cost).toBeGreaterThan(0);
});
});
describe('classifyChecks (D13)', () => {
test('remediable: missing_embeddings with API key', () => {
const result = classifyChecks([{ name: 'missing_embeddings', status: 'fail' }], {
hasEmbeddingApiKey: true,
});
expect(result[0]).toEqual({ check: 'missing_embeddings', status: 'remediable' });
});
test('blocked: missing_embeddings without API key', () => {
const result = classifyChecks([{ name: 'missing_embeddings', status: 'fail' }], {
hasEmbeddingApiKey: false,
});
expect(result[0]?.status).toBe('blocked');
expect(result[0]?.reason).toContain('embedding');
});
test('blocked: dead_links without repoPath', () => {
const result = classifyChecks([{ name: 'dead_links', status: 'warn' }], {});
expect(result[0]?.status).toBe('blocked');
});
test('human_only: orphan_pages (archive is product judgment)', () => {
const result = classifyChecks([{ name: 'orphan_pages', status: 'warn' }], { repoPath: '/brain' });
expect(result[0]?.status).toBe('human_only');
});
test('human_only: unknown check defaults to operator judgment', () => {
const result = classifyChecks([{ name: 'mystery_check', status: 'warn' }], {});
expect(result[0]?.status).toBe('human_only');
});
});
describe('maxReachableScore (D13)', () => {
test('all remediable: full 100', () => {
const health = makeHealth({
embed_coverage_score: 0,
no_dead_links_score: 0,
no_orphans_score: 0,
});
const classes = [
{ check: 'missing_embeddings', status: 'remediable' as const },
{ check: 'dead_links', status: 'remediable' as const },
{ check: 'orphan_pages', status: 'remediable' as const },
];
expect(maxReachableScore(health, classes)).toBe(100);
});
test('all blocked: stays at current', () => {
const health = makeHealth({
brain_score: 50,
embed_coverage_score: 5,
no_dead_links_score: 5,
no_orphans_score: 10,
});
const classes = [
{ check: 'missing_embeddings', status: 'blocked' as const },
{ check: 'dead_links', status: 'blocked' as const },
{ check: 'orphan_pages', status: 'human_only' as const },
];
// 5 + 25 + 15 + 10 + 5 = 60
expect(maxReachableScore(health, classes)).toBe(60);
});
});
describe('estimateAnthropicCost', () => {
test('returns 0 for unknown model', () => {
expect(estimateAnthropicCost('unknown-model', 10)).toBe(0);
});
test('returns positive for known model', () => {
const cost = estimateAnthropicCost('claude-sonnet-4-6', 10);
expect(cost).toBeGreaterThan(0);
});
});
test/op-checkpoint.test.ts
+180
View File
@@ -0,0 +1,180 @@
import { describe, test, expect, beforeAll, beforeEach, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import {
loadOpCheckpoint,
recordCompleted,
clearOpCheckpoint,
resumeFilter,
purgeStaleCheckpoints,
fingerprint,
embedFingerprint,
extractFingerprint,
reindexFingerprint,
} from '../src/core/op-checkpoint.ts';
/**
* D12 pinning tests for src/core/op-checkpoint.ts.
*
* Closes codex #10–#16:
* - per-param fingerprint scoping (no cross-mode collisions)
* - DB-backed CRUD works on PGLite (single-host fallback path)
* - resumeFilter is pure
* - purgeStaleCheckpoints respects TTL
*/
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
describe('fingerprint helpers', () => {
test('fingerprint: stable across runs', () => {
const params = { stale: true, source: 'default' };
expect(fingerprint(params)).toBe(fingerprint(params));
});
test('fingerprint: key order does not matter (canonical-JSON)', () => {
const a = fingerprint({ a: 1, b: 2 });
const b = fingerprint({ b: 2, a: 1 });
expect(a).toBe(b);
});
test('fingerprint: different values produce different hashes', () => {
expect(fingerprint({ a: 1 })).not.toBe(fingerprint({ a: 2 }));
});
test('fingerprint returns 8 hex chars', () => {
expect(fingerprint({ x: 1 })).toMatch(/^[a-f0-9]{8}$/);
});
test('codex #11: extract links vs timeline get different fingerprints', () => {
const linksFp = extractFingerprint({ mode: 'links', source: 'default' });
const timelineFp = extractFingerprint({ mode: 'timeline', source: 'default' });
expect(linksFp).not.toBe(timelineFp);
});
test('codex #12: reindex markdown vs code get different fingerprints', () => {
const md = reindexFingerprint({ markdown: true, chunker_version: 2 });
const code = reindexFingerprint({ code: true, chunker_version: 2 });
expect(md).not.toBe(code);
});
test('codex #15: embed model+dim variation produces different fingerprints', () => {
const a = embedFingerprint({
stale: true,
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 3072,
});
const b = embedFingerprint({
stale: true,
embedding_model: 'voyage:voyage-3',
embedding_dimensions: 1024,
});
expect(a).not.toBe(b);
});
test('reindex chunker_version bump invalidates checkpoint', () => {
const v1 = reindexFingerprint({ markdown: true, chunker_version: 1 });
const v2 = reindexFingerprint({ markdown: true, chunker_version: 2 });
expect(v1).not.toBe(v2);
});
});
describe('loadOpCheckpoint / recordCompleted / clearOpCheckpoint', () => {
test('empty checkpoint returns []', async () => {
const result = await loadOpCheckpoint(engine, { op: 'embed', fingerprint: 'abc12345' });
expect(result).toEqual([]);
});
test('round-trip: write then read', async () => {
const key = { op: 'embed', fingerprint: 'abc12345' };
await recordCompleted(engine, key, ['chunk-1', 'chunk-2', 'chunk-3']);
const result = await loadOpCheckpoint(engine, key);
expect(result.sort()).toEqual(['chunk-1', 'chunk-2', 'chunk-3']);
});
test('write overwrites prior state', async () => {
const key = { op: 'embed', fingerprint: 'abc12345' };
await recordCompleted(engine, key, ['chunk-1']);
await recordCompleted(engine, key, ['chunk-1', 'chunk-2']);
const result = await loadOpCheckpoint(engine, key);
expect(result.sort()).toEqual(['chunk-1', 'chunk-2']);
});
test('different fingerprints stay isolated', async () => {
const linksKey = { op: 'extract', fingerprint: 'fp-links' };
const timelineKey = { op: 'extract', fingerprint: 'fp-timeline' };
await recordCompleted(engine, linksKey, ['file-a.md']);
await recordCompleted(engine, timelineKey, ['file-b.md']);
const links = await loadOpCheckpoint(engine, linksKey);
const timeline = await loadOpCheckpoint(engine, timelineKey);
expect(links).toEqual(['file-a.md']);
expect(timeline).toEqual(['file-b.md']);
});
test('clearOpCheckpoint drops the row', async () => {
const key = { op: 'embed', fingerprint: 'to-clear' };
await recordCompleted(engine, key, ['x']);
expect(await loadOpCheckpoint(engine, key)).toEqual(['x']);
await clearOpCheckpoint(engine, key);
expect(await loadOpCheckpoint(engine, key)).toEqual([]);
});
test('clearOpCheckpoint on missing row is no-op (idempotent)', async () => {
// Should not throw and load should still return [] afterwards
await clearOpCheckpoint(engine, { op: 'never-written', fingerprint: 'nope' });
const after = await loadOpCheckpoint(engine, { op: 'never-written', fingerprint: 'nope' });
expect(after).toEqual([]);
});
});
describe('resumeFilter (pure)', () => {
test('empty completed returns all', () => {
expect(resumeFilter(['a', 'b', 'c'], [])).toEqual(['a', 'b', 'c']);
});
test('filters out completed keys', () => {
expect(resumeFilter(['a', 'b', 'c', 'd'], ['b', 'd'])).toEqual(['a', 'c']);
});
test('no completed keys present in all: identity', () => {
expect(resumeFilter(['a'], ['z'])).toEqual(['a']);
});
test('all completed: returns empty', () => {
expect(resumeFilter(['a', 'b'], ['a', 'b'])).toEqual([]);
});
});
describe('purgeStaleCheckpoints', () => {
test('no stale rows: returns 0', async () => {
await recordCompleted(engine, { op: 'embed', fingerprint: 'fresh' }, ['x']);
const purged = await purgeStaleCheckpoints(engine, 7);
expect(purged).toBe(0);
});
test('purges rows older than TTL', async () => {
// Insert a fake old row directly
await engine.executeRaw(
`INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
VALUES ('embed', 'old', '["x"]'::jsonb, now() - interval '10 days')`,
);
const purged = await purgeStaleCheckpoints(engine, 7);
expect(purged).toBe(1);
expect(await loadOpCheckpoint(engine, { op: 'embed', fingerprint: 'old' })).toEqual([]);
});
});