mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
814258dda67945ffec9457a1e73980e947b7e462
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
5c49225e4b |
v0.42.45.0 feat(sync): delta-aware cost estimator — stop wedging the daily cron (#2139) (#2224)
* feat(core): shared computeSyncDelta + spend-posture module (#2139) sync-delta.ts: ONE implementation of "what changed since last_commit", consumed by both the sync executor and the inline cost estimator so the gate's dollar figure can't drift from what the sync imports. spend-posture.ts: spend.posture config + parseUsdLimit/formatUsdLimit off-switch parsing (off/unlimited/none → Infinity; undefined at the budget boundary so ledger rows never serialize null). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sync): delta-aware cost estimator + non-TTY auto-defer + per-source failure acks (#2139) The inline-embed cost gate was a ~400x phantom: it priced the entire tree whenever the working tree was dirty (always, on an active brain), then blocked the daily cron with exit 2. Now: - performSyncInner + the estimator both route through computeSyncDelta, so the estimate mirrors execution (fetch-first delta; dirty-but-caught-up tree → $0). - shouldBlockSync is posture-aware; non-TTY above floor AUTO-DEFERS embeds to capped backfill jobs (exit 0) instead of wedging — single shared runInlineCostGate on both --all and single-source paths. - --full prices delta + stale backlog (full sync sweeps it inline). - off/unlimited on the cost knobs; tokenmax bypasses the backfill cap (still ledgered) but never the cooldown. - --skip-failed/--retry-failed scoped per source; the D15 parallel refusal is lifted (the #1939 ledger is per-source + lock-serialized). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(config): register spend-control keys + validate spend.posture (#2139) Adds spend.posture + the five previously --force-only spend knobs to KNOWN_CONFIG_KEYS so `config set` accepts them directly (removes the archaeology the issue complained about), and rejects invalid spend.posture values at set time with a paste-ready hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reindex,enrich,onboard): spend.posture across the remaining cost gates (#2139) reindex-code: tokenmax makes the cost gate informational; --max-cost accepts off/unlimited. enrich + onboard --auto: tokenmax lifts the refuse-without-cap guardrail and runs UNCAPPED (spend still ledgered by BudgetTracker). Explicit --max-usd always wins over posture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cost-gate, delta estimator, spend-posture, off-switch coverage (#2139) New sync-delta + sync-cost-estimate unit suites; rewritten cost-gate serial tests (auto-defer instead of exit 2, posture, off-switch, format split, single-source); parseUsdLimit/posture-aware shouldBlockSync; backfill cap-off + tokenmax-bypass + cooldown-still-refuses; config known-key acceptance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spend-controls): single spend-control surface + ref-map + follow-up TODOs (#2139) New docs/operations/spend-controls.md (every gate, key, default, off switch, posture interaction); CLAUDE.md reference-map row; two P3 follow-up TODOs (measured chunk-count gating, per-source defer granularity). llms bundles regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(spend): SSRF-harden estimator fetch + complete off/uncapped across reindex/enrich/onboard (#2139) Ship-stage codex pre-landing review caught four P1s in the secondary cost gates: - The delta estimator's fetch-first ran `git fetch` through the plain git() helper, bypassing the GIT_SSRF_FLAGS + GIT_TERMINAL_PROMPT=0 hardening that real sync uses. Added `fetchRemote()` to git-remote.ts (same flags as pullRepo) and route the estimator through it — a cost preview / dry-run can no longer hit a remote through a less-protected path. - `reindex --max-cost off`, `enrich --max-usd off`, `onboard --auto --max-usd off` were parsed but didn't actually proceed/uncap. Now: explicit off (and spend.posture=tokenmax) proceed past the confirmation/missing-cap refusal AND run uncapped. enrich threads an Infinity sentinel mapped to "no BudgetTracker ceiling" (never raw Infinity → no null in audit rows); reindex/onboard use their native undefined=uncapped path. Spend still ledgered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.45.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(KEY_FILES): update sync/embedding/git-remote/reindex entries to post-#2139 truth document-release pass: the cost-gate entries described the pre-#2139 behavior (full-tree-ceiling estimator, --skip-failed-rejects-under-parallel, exit-2 confirmation gate). Updated to current truth — delta-aware estimator via the shared computeSyncDelta, per-source failure acks under parallel, non-TTY auto-defer (no exit 2), posture-aware shouldBlockSync. Added entries for the two new core modules (sync-delta.ts, spend-posture.ts) + fetchRemote on git-remote.ts + reindex --max-cost off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a81f7e05e8 |
v0.42.43.0 feat(context): push-based context (#2095) + teardown-exit hardening (#2084) (#2175)
* fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084)
Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering
sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the
bound raced past — keep Bun's event loop alive, so every `gbrain query` paid
a flat 10s tax exiting via the hard-deadline force-exit banner.
Three changes, one contract:
- flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the
command is not a daemon, exit deliberately — after stdout AND stderr drain
(writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a
blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the
regression class; pinned by a 256KB real-pipe subprocess test.
- drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites
(op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation
x3, ze-switch, dream, read-only timeout path). Drains the background-work
registry, then disconnect (best-effort), bounded by the 10s unref'd
hard-deadline — which is now armed around the TEARDOWN window only, not
before the op handler (the old placement would have force-killed any op
slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites
previously skipped the drain entirely and had no hang timer at all.
- Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing
paths (status, friction, claw-test, smoke-test, eval cross-modal /
takes-quality replay / conversation-parser / whoknows-thin, status-thin)
become process.exitCode + return so they flow through the drains and the
flush-exit. Pre-engine usage/parse/refusal exits stay as-is.
BrainRegistry.disconnectAll deliberately unchanged: zero production callers
in src/, per-engine disconnects already bounded, and the kernel reclaims
sockets on exit (src/core/timeout.ts doctrine).
DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084)
Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs
verified only against one production deployment — CI had no transaction-mode
pooler and could never see the class. Now it can:
- docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting
postgres-1, mirroring the production split-pool topology (direct :5432 +
pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext
password in the userlist) + IGNORE_STARTUP_PARAMETERS for the
statement_timeout/idle_in_transaction_session_timeout startup params
gbrain's client sets (the Supabase pooler whitelists the same).
- test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url
into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs),
then spawns the real CLI against the POOLED url and asserts: exit 0,
stdout intact (the #1959 truncation class), and NO
"did not return within 10000ms — force-exiting" banner (pre-#2084 it
printed on 100% of query-shaped ops on this topology). Class bound, not
exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL.
- scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL +
GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases.
Verified live: both tests green against pgbouncer 1.25.2 in transaction mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095)
One row per page the brain volunteers (op / reflex / watch channels).
"Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at
(the existing bumpLastRetrievedAt write-back is the open/cite signal), so
there is no second tracking path. session_id/turn are nullable
caller-supplied attribution; rationale is a deterministic template string,
never raw conversation text.
- Migration v116 (idempotent) + mirrors in src/schema.sql +
src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also
folds in pre-existing comment-only drift from the v114 links edits).
- src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE
multi-row parameterized INSERT — never per-row awaited round-trips) +
purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains).
- Dream cycle purge phase prunes stale events alongside op_checkpoints /
brainstorm checkpoints / batch-retry audit files.
- RLS on Postgres comes from the v35 auto_rls_on_create_table event
trigger (the same mechanism that covered v110 page_aliases and v115
op_checkpoint_paths); the volunteer Postgres e2e pins it.
- No ::jsonb anywhere; no bootstrap probe needed (nothing references the
table pre-creation; writers guard with try/catch).
Tests: v116 shape + columns + indexes + live insert/purge round-trip on
PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095)
- entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing
per-turn extractor across the last N turns (oldest→newest), merges by the
normalizeAlias form with occurrence/newest-turn/user-mention metadata, and
orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES
cap drops stale assistant chatter, not the entity the user just named.
Closes the filed assistant-introduced-entities recall TODO; true pronoun
coreference (never-named antecedents) stays out of scope.
- retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence +
matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6)
lives next to the arm definitions so identity and score can't drift.
Arm-2 provenance is classified in JS (codex D8 — the combined OR can't
report which predicate matched). Federated sourceIds[] scope (alias arm
loops per source; arm 2 uses source_id = ANY — no engine-interface
change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for
windowing): the legacy title-whole-word rule would suppress every entity
merely MENTIONED in a prior window turn, breaking the feature by
construction — slugs only enter context when a pointer/page was actually
surfaced. Default stays 'slug-and-title' for the window=1 legacy path.
- volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF,
unprefixed → one user turn), volunteerContext (zero-LLM: extract →
resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate →
cap 3/5; deterministic rationale strings, never raw conversation text),
and volunteerUsageStats (per-arm/channel precision from the
last_retrieved_at join, labeled approximate — 5-min throttle false
negatives, unrelated-read false positives; codex D9).
Tests: 35 green across volunteer-context (window parsing, pronoun follow-up
via assistant-introduced entity, confidence gating, slug-only suppression,
takes-fence privacy, multi-source scope, caps, stats join math) +
retrieval-reflex back-compat + resolve-ipc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095)
New read-scope op on the contract surface (CLI `gbrain volunteer-context`
with stdin → window, MCP tool for free): takes a rolling conversation
window, returns confidence-gated page pointers with rationales + synopses.
`window` is optional-unless-stats (validated in the handler, codex D9);
`stats: true` returns the volunteered-vs-used precision summary, labeled
APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both
bias the join). Source scope threads through sourceScopeOpts — federated
grants narrow the volunteer to the granted sources.
Event logging is fire-and-forget through a new `volunteer-events`
background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked
dangling promise set + bounded drain + snapshot-drop on timeout so a
long-lived process never accumulates ghosts). ONE batched INSERT per call,
drained on every exit path by the commit-1 drain hoist; failure never
fails the op (pinned by an injected failing-engine test).
cli formatResult renders both shapes (pointer lines with confidence/arm/
rationale; the stats summary with per-arm precision).
Tests: op contract surface, window-required validation, sink round-trip
with session_id/turn attribution, failing-engine fail-open, federated
grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e
proving the op + sink + stats join AND that context_volunteer_events has
RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): reflex consumes the rolling window + ambient-channel logging (#2095)
The default-on retrieval reflex now extracts entities from the last N turns
(retrieval_reflex_window_turns, default 4; env
GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy
current-turn-only behavior exactly). assemble() passes the recent
user/assistant turns (hard cap 12); the reflex slices to the configured
window. Assistant-introduced entities and "what did she invest in?"
follow-ups whose antecedent was NAMED in the window now surface pointers —
the issue's "zero agent-initiated queries" success criterion on the
ambient path.
Under windowing, suppression switches to slug-only (codex D7): the legacy
title-whole-word rule would suppress every entity merely MENTIONED in a
prior window turn, breaking the feature by construction. Slugs only enter
prior context when a pointer/page was actually surfaced, so
already-surfaced pages still suppress. The suppression mode flows through
all three resolver rungs (host opts, serve IPC request, direct Postgres).
Ambient-channel feedback (codex D11): the server-side resolver paths
(serve IPC + direct Postgres) log volunteered pointers with
channel: 'reflex' through the drained volunteer-events sink, so
`gbrain volunteer-context --stats` measures the default-on path where most
volunteering happens. Host-injected resolvers (no gbrain engine) can't
log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the
pointer cap are unchanged.
Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only
vs already-surfaced suppression; throwing resolver stays fail-open
(16 green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): gbrain watch — push transport over stdin (#2095)
The issue's headline: the brain volunteers pages as the conversation flows,
instead of waiting to be asked. `some-transcript-feed | gbrain watch` reads
turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed
lines are user turns), keeps a rolling window (--window-turns, default 4),
and streams confidence-gated pointers with rationales to stdout (--json for
JSONL). Session dedupe rides the core's slug-only suppression — a slug is
volunteered at most once per session. Events log on channel 'watch' with
session_id + turn through the drained sink.
Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an
interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF —
so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1
placeholder): when main() resolves the work is over, the CLI_ONLY finally
drains volunteer events via drainThenDisconnect, and the entrypoint
flush-exit ends the process. Keeping it in the daemon set would have made
the piped EOF path hang on lingering sockets — the exact #2084 class.
SIGINT closes the stream and flows through the same drain path instead of
killing mid-write. Per-turn resolution failures are fail-open (the stream
never dies on a transient DB error).
Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) +
THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP
op) + main --help entry.
Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling
window via assistant-introduced entity, session dedupe, --json shape with
turn attribution, channel-watch event rows, --min-confidence gate, CRLF/
blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch`
on a fresh PGLite brain exits 0 at EOF with no force-exit banner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave
- docs/architecture/KEY_FILES.md (current-state): context entries gain the
window extractor, arm provenance/confidence, suppression modes, volunteer
+ volunteer-events modules; background-work entry now lists FIVE sinks and
the drainThenDisconnect owner-disconnect contract; new entries for
src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
the confidence model, CLI usage, config keys, and the approximate-stats
caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
check, structured messages[] param); the #1981 entity-detection TODO
narrowed (window extraction covered assistant-introduced entities +
named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
#2084 wave.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): truncate context_volunteer_events in setupDB (#2095)
The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows
persisted across e2e runs on a reused database and poisoned count/stats
assertions in volunteer-context-postgres on the second run. No FK to pages
(slug join), so position before pages is for hygiene only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084)
Caught by the full unit suite: `gbrain apply-migrations` on PGLite started
exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM
backend's proc_exit status into process.exitCode (initdb at create-time,
the postmaster at close-time — `exitCode=status` in pglite's dist), and
the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around
create/close (a guarded attempt verified this). The pre-#2084 success path
never read process.exitCode, so the pollution was invisible; the new
deliberate flush-exit propagated it faithfully.
Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in
cli-force-exit.ts — every gbrain-owned exit-code assignment routes through
the setter (still mirrored to process.exitCode for outside readers), and
both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect
hard-deadline backstop) read the getter. Swept all assignment sites:
cli.ts (op error, friction, claw-test, smoke-test, eval runners, status,
import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot.
Also updates the v0.42.20 structural pins to the drainThenDisconnect shape
(ordering invariant asserted INSIDE the helper + >=8 helper call sites,
superseding the two-inline-pairs assertion).
Verified: apply-migrations spawn test green; `init --migrate-only` exits 0;
an errored op still exits 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: re-pin the teardown-arming invariant at its post-#2084 home
Master's v0.42.41.0 triage wave and the #2084 wave fixed the same
pre-armed-timer bug independently; the merge keeps #2084's shape (arming
inside the shared drainThenDisconnect helper, covering all 8 exit paths).
The structural pin now asserts the same invariant — no pre-try arming;
gated, unref'd, before-drain, cleared — at the helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: coverage for ambient reflex-channel logging + watch window/cap flags
Ship coverage audit (85%, gate PASS) named five gaps; the two substantive
cheap ones close here: the codex-D11 logChannel='reflex' path now has a
behavioral pin (events land on channel 'reflex' through the drained sink;
no logChannel → no events), and gbrain watch's --window-turns / --max-pages
flags are exercised (turn-1 attribution under window=1; cap to one page).
Remaining flagged-not-blocking: the wallclock-timeout branch (untestable
without >10s real-clock flake — same rationale as the arming pin),
formatResult's volunteer case (module-private), and the cycle purge wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT
formatResult exported for tests (same import-safety contract as cliAliases);
test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message,
and approximate stats summary. test/watch-command.test.ts gains a real
subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream,
assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under
the actual signal, not just the shared exit path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin
The merged-state suite caught it: doctor --fast --json reported FAIL but
exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes
(doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit
silently zeroes — getCliExitCode() deliberately never reads ambient
process.exitCode (the PGLite-Emscripten pollution defense), so any setter
that bypasses setCliExitCode reports success on failure.
Swept both sites and added the structural class pin: a test greps src/ for
raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that
introduces one fails loudly instead of lying about exit codes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.43.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: quarantine the watch SIGINT subprocess test to the serial lane
The parallel unit shards flake on concurrent CLI subprocess spawns (failed
at 7ms in-suite, green solo) — same isolation rationale as
apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: update project documentation for v0.42.43.0
Post-ship doc verification against the release diff (#2095 push-based
context + #2084 superset hardening), with a cross-model doc review:
- push-context.md: version tag corrected to v0.42.43.0; per-call knobs
now cover prior_context/days and watch's flag surface accurately;
feedback-log writes described as best-effort; synopsis fence-strip
described as unconditional.
- CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context
release reference corrected to v0.42.43.0.
- KEY_FILES.md: ci-local entry rewritten to current topology (4-shard
parallel default, four Postgres services, transaction-mode PgBouncer
+ GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts
dropped from the selector entry.
- TESTING.md: inventory entries for the new #2084 structural pins
(cli-exit-verdict-pin, cli-pipe-truncation), the push-context test
suite (volunteer-context, watch-command, watch-sigint.serial,
cli-format-volunteer), migrate v117 coverage, and the two new E2E
files (pgbouncer-teardown env gating, volunteer-context-postgres RLS
pin); check:all row corrected (not a superset of verify).
- AGENTS.md + RELEASING.md: ci:local descriptions updated to the
sharded + pooler topology.
- CHANGELOG (wording only, entry preserved): "retrieved" instead of
"opened" for the used-signal, pooler scoped to the local CI gate,
feedback log labeled best-effort.
- llms-config.ts: index the new push-context guide; bundles
regenerated (build:llms) and freshness test green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(test): correct the v116 reference — the table shipped as migration v117
* fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095)
Five specialist reviewers (testing/maintainability/security/performance/
data-migration) on the reconciled diff; every finding applied:
Performance: the alias arm now resolves all granted sources CONCURRENTLY
(a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources
cross-region, inside the reflex's 1.5s budget); watch's session dedupe is
O(1) Set membership instead of a monotonically growing priorContext string
(O(T²) over a long-lived session); getWindowTurns iterates from the tail
(per-turn cost no longer grows with session length); the resolver's
provenance maps fold into the existing candidate pass.
Security: volunteer_context clamps caller-supplied attribution at the trust
boundary — session_id capped at 256 chars (a read-scoped token could bank
~1MiB TEXT per request, retained 90 days), turn logged only when a safe
integer (a non-integer threw inside the batched INSERT and silently dropped
the whole batch). The privacy comments now state precisely what rationale
may contain (the matched entity's surface form — which by construction
resolved to an existing alias/title/slug — never free conversation text).
Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from
volunteer.ts and shared by watch/cli (the two surfaces can no longer
drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly
site for all three channels; watch's window default now honors the same
retrieval_reflex_window_turns config knob the reflex reads; the stale
pre-v116 comments swept to pre-v117.
Testing: the two flake-class CRITICALs fixed (pipe test asserts the
backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test
waits on watch's new machine-readable ready line instead of a fixed 15s
sleep — 2.5s and deterministic now); new coverage for the sink's timeout
branch + ghost-reference drop, watch per-turn fail-open, untrusted knob
clamps (min_confidence/max_pages/days), window-cap ordering (newest user
mention survives), serve-IPC suppression passthrough + channel=reflex
logging, windowTurnCount edge semantics, and structural pins for the sink
registration + cycle purge wiring. The exit-verdict pin's grep is now
operator/whitespace-tolerant.
Deferred with TODOs: resolver index shapes for the per-turn query;
batched first-prune after a long dream-cycle gap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp
Four red-team findings on the #2095 push-context surface:
- RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's
cap, so a recurring already-pushed entity burned cap slots every turn and
starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside
the pointer loop BEFORE the confidence gate and the cap.
- RT3 honest stats: reflex-channel event logging moved from inside the
resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires
only after the response write succeeds, and buildReflexAddition logs only
after the per-turn timeout admits the block. A block the client's 250ms
budget abandoned was never injected and no longer counts as volunteered.
(logChannel resolver opt removed; logDeliveredReflexPointers is the seam.)
- RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo
can't reintroduce the re-scan-everything-per-turn cost class.
- RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP,
push-context guide, TODO to route watch via serve IPC); host-resolver
suppression contract at ResolveEntitiesFn (TODO for a capability gate).
Tests: starvation guard (watch + volunteerContext unit), window clamp floor +
ceiling, delivery-side logging (helper writes channel=reflex through the
drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test
rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims
Three CI-only check failures, two root causes:
1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when
loadConfig() returned null (no config file AND no DATABASE_URL — a clean
CI shard with no brain). loadConfig drops its env→config mapping in that
case, so the documented escape hatch silently died and the window fell
back to 4 → windowed extraction widened when the test set window=1 →
prior-turn entity leaked. Fixed: read the env var directly in
windowTurnCount, mirroring reflexEnabled's direct process.env read. This
is a real product bug, not just a test artifact — any config-less host
using the env hatch was affected. Regression test pins it.
2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a
sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway
and never reset it. The legacy-embedding preload only restores the
OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign
config survives into the next file — and a file's beforeAll runs BEFORE
the preload's restoring beforeEach, so federation-health built a
vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new
test files reshuffled the deterministic file→shard assignment, exposing
this latent ordering bug. Hardened both victims to establish the gateway
state they assert (sync-cost-preview resets to the unconfigured fallback;
federation-health pins legacy 1536 before initSchema) so they're
order-independent. Verified against a simulated leaker run before them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(context): use withEnv() in the window env-hatch test (test-isolation guard)
The regression test added in
|
||
|
|
146a8f1eed |
v0.41.31.0 feat(embed): delta-aware sync --all cost gate + real stale-embedding semantics (#1632)
* fix(cost): embedding cost preview uses configured model rate, not hardcoded OpenAI
The sync --all cost gate computed spend from a hardcoded
EMBEDDING_COST_PER_1K_TOKENS = 0.00013 (OpenAI text-embedding-3-large)
and labeled the preview with the back-compat EMBEDDING_MODEL constant,
regardless of the actually-configured embedding model. A brain running a
cheaper model (e.g. zeroentropyai:zembed-1 @ $0.05/Mtok) saw a preview
that named the wrong provider and over-stated spend ~2.6x ($337 vs $130
on a 2.6B-token corpus).
estimateEmbeddingCostUsd now resolves the live model via the gateway and
prices it through embedding-pricing.ts (the existing per-provider:model
table), falling back to the OpenAI rate only when the gateway is
unconfigured (unit-test context) or the model is unknown. sync.ts surfaces
the real model name in the preview message and JSON.
Regression test pins model-aware pricing: openai 3-large vs zembed-1 must
produce materially different previews; collapsing both to the OpenAI number
fails the assertion.
* fix(cost): sync --all gate is informational when embed is deferred; delta-aware inline gate
Under federated_v2 (default), sync --all DEFERS embedding to per-source
embed-backfill jobs that already cap spend at $25/source/24h. The v0.20
cost gate predated that cap and fired ConfirmationRequired + exit 2 on
EVERY non-TTY sync --all without --yes, regardless of cost — blocking
nightly crons over already-synced corpora and forcing permanent --yes.
The gate is now mode-aware:
- Deferred embed (v2 default): print an FYI deferred notice (cap-aware,
"not charged by this sync") + the stale-chunk backlog estimate, and
NEVER exit 2. The backfill cap is the real money gate.
- Inline embed (v2 off, or --serial without --no-embed): keep the
blocking gate, but estimate the actual delta — full-tree ceiling for
changed sources (unchanged sources contribute 0 via the same git +
chunker_version "do work?" gate doctor/sync use) + stale backlog — and
block only when it exceeds the new configurable floor
sync.cost_gate_min_usd (default $0.50).
New pure helpers in embedding.ts (willEmbedSynchronously, shouldBlockSync)
keep the decision logic hermetically testable. New engine method
sumStaleChunkChars (both engines) prices the embedding backlog via
estimateCostFromChars. estimateSyncAllCost's per-source walk extracted to
estimateSourceTreeTokens (reused by the inline estimator).
Regressions pinned: R-1 deferred non-TTY never exit 2 (headline), R-2
inline above-floor still exit 2 (protection), plus the willEmbedSynchronously
/ shouldBlockSync matrix and sumStaleChunkChars engine + scope + embed_skip
coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(embed): real stale semantics — re-embed on model/dims swap (migration v108)
Pre-v0.41.30 "stale" meant only `embedding IS NULL`, so swapping the
embedding model or dimensions left the whole corpus silently embedded under
the OLD model — `embed --stale` ignored it and search quality quietly
degraded.
New `pages.embedding_signature` (TEXT, migration v108) stamps the embedding
provenance (`<provider:model>:<dims>`) whenever a page's chunks are embedded.
A later model/dims swap makes the stored signature differ from the current
one, which the embed paths now detect and re-embed.
GRANDFATHER (critical): the stale predicate is
`embedding IS NULL OR (embedding_signature IS NOT NULL AND <> $current)`
so a NULL signature is NEVER stale. After the migration every existing page
has NULL → none flagged → the next `embed --stale` does NOT re-embed the
whole corpus. Signatures are stamped going forward only.
Surface:
- countStaleChunks / sumStaleChunkChars gain an optional `signature` opt
that widens staleness (read-only; used by the dry-run preview + the
sync cost preview, which is now signature-aware).
- invalidateStaleSignatureEmbeddings(signature, sourceId?) NULLs the
embeddings of signature-mismatched pages so the EXISTING NULL-embedding
cursor (listStaleChunks, untouched) re-embeds them — keeps the keyset
pagination logic intact.
- setPageEmbeddingSignature stamps after a page's chunks land.
- Both embed loops wired: `gbrain embed --stale`/`--all` (embed.ts) and the
embed-backfill minion (embed-stale.ts) invalidate-then-stamp.
Migration v108 + bootstrap probe (both engines) + REQUIRED_BOOTSTRAP_COVERAGE
entry. Pinned by test/embedding-signature-stale.test.ts (R-4 grandfather,
mismatch detection, matching no-op, scoped invalidate, stamp) + the
bootstrap-coverage gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): surface embed-backfill job state in sources status + deferred notice
Under federated_v2, `sync --all` exits 0 and embedding lags behind in
embed-backfill jobs (subject to cooldown + the per-source 24h cap). Pre-fix
an operator had no signal those jobs were queued or lagging — the sync looked
"done" while embeddings trickled in later.
`gbrain sources status` now shows a BACKFILL column per source
(active(N)/queued(N)/idle) plus the last completion timestamp, read from
minion_jobs. The deferred-sync notice appends "N backfill job(s) queued" so a
cron operator sees work is enqueued, not lost. Both reads are best-effort —
a brain that never ran a worker (no minion_jobs table) reports idle/0 instead
of crashing the dashboard.
SyncStatusReportSource gains backfill_queued / backfill_active /
backfill_last_completed_at (additive; JSON envelope schema_version unchanged).
Pinned by a new case in test/e2e/sync-status-pglite.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add currentEmbeddingSignature to embedding.ts mocks + sync stale EXPECTED_PHASES
Commit 3 made embed.ts import currentEmbeddingSignature from embedding.ts.
Four tests mock.module the whole embedding.ts and omitted the new export, so
embed.ts (imported transitively) failed at load with "Export named
'currentEmbeddingSignature' not found". Add the export to each mock:
embed.serial.test.ts, e2e/cycle.test.ts, e2e/dream.test.ts,
e2e/dream-cycle-phase-order-pglite.test.ts.
Also sync the stale EXPECTED_PHASES in dream-cycle-phase-order-pglite.test.ts
to match cycle.ts ALL_PHASES — extract_atoms, synthesize_concepts, and
conversation_facts_backfill drifted in after the test was last touched
(v0.41.0.0) and were never added, so both phase-order assertions were failing
on the branch before this wave (confirmed against
|
||
|
|
f718c595b3 |
v0.21.0 feat: Code Cathedral II — call-graph edges, two-pass retrieval, parent-scope chunking (#422)
* feat: v0.18.0 baseline — code indexing + multi-repo (Layer 0)
Tree-sitter-based code chunker for TS/JS/Python/Ruby/Go. Splits code at
semantic boundaries (functions, classes, types, exports). Each chunk
includes a structured header for embedding context.
Multi-repo config: gbrain repos add/list/remove, gbrain sync --all.
Strategy-aware sync: markdown (default), code, or auto. New PageType
'code' for code file pages.
This is Layer 0 of the v0.18.0 code-indexing plan (see ~/.claude/plans
cathedral plan). Subsequent layers add: tests, bun --compile WASM
embedding + CI guard (A1), schema migrations v16 (pages.repo_name) +
v17 (content_chunks code metadata), per-repo sync bookmarks, runCycle
multi-repo, Chonkie chunker parity (E2a), incremental chunking (E2),
doc↔impl linking (E1), markdown fence extraction (E3), symbol navigation
commands (code-def, code-refs), cost preview, BrainBench code category,
CHANGELOG, migration file, docs.
Backward compatible: no config changes = existing behavior preserved.
* feat: v0.19.0 Layer 1 — tests for baseline + errors envelope + version bump
Adds the structured error envelope (src/core/errors.ts) that downstream
v0.19.0 commands (code-def, code-refs, sync --all cost preview,
importCodeFile) all hand back to agents. The envelope follows the v0.17.0
CycleReport.PhaseResult.error shape so agent-consumption stays consistent
across every gbrain surface.
Test coverage for Wintermute's baseline (added in Layer 0):
- test/errors.test.ts — envelope helper + GBrainError + serializeError
- test/multi-repo.test.ts — config CRUD, dedup, file permissions
- test/sync-strategy.test.ts — isSyncable strategy matrix + include/exclude
globs + slugifyCodePath + pathToSlug with pageKind
Bug fixes uncovered by the new tests:
- src/core/sync.ts: globToRegex handles `src/**/*.ts` matching `src/foo.ts`
(zero intermediate dirs). `**/` now compiles to `(?:.*/)?` instead of
`.*/`. Also `?` now matches only non-slash chars (was `.`).
- src/core/config.ts: configDir() respects GBRAIN_HOME env override so
tests can isolate ~/.gbrain/. Matches GBRAIN_AUDIT_DIR convention.
Bun's os.homedir() ignores $HOME on macOS, so we need an explicit
override variable.
Version bump: package.json 0.18.2 → 0.19.0. v0.18.0-2 were already
released (multi-source brains + RLS + migration hardening), so the next
free minor for code indexing is 0.19.0. Wintermute's baseline author
label of 0.16.4 had been stale since v0.17.0 shipped; no user-visible
regression from the jump.
Per the rebased cathedral plan: Wintermute's multi-repo.ts and repos
CLI are preserved at the baseline but will be superseded in Layer 4 by
the v0.18.0 sources system (src/core/source-resolver.ts,
src/commands/sources.ts). multi-repo tests stay valid for the baseline
and will be removed alongside the code they cover.
* feat: v0.19.0 Layer 2 — bun --compile WASM embedding + CI guard
The single highest-risk change in v0.19.0 code indexing. Before this, the
chunker loaded WASMs via `new URL('../../../node_modules/...', import.meta.url)`
which silently breaks in the compiled binary (no node_modules at runtime).
Users would see degraded chunking quality with no error, just fallback-
recursive chunks instead of real semantic chunks. Codex flagged this as
the #1 silent-failure mode.
Mechanics:
- `src/assets/wasm/tree-sitter.wasm` + 36 grammar WASMs committed to the
repo (50MB). Not a small check-in, but the alternative is a postinstall
script that runs before every dev bun run and fails fragile-ly on
network errors.
- `src/core/chunkers/code.ts` uses Bun's `import ... with { type: 'file' }`
import attribute. At runtime the imported value is a file path — the
actual repo path in dev, a bundler-synthesized path in the compiled
binary. The tree-sitter runtime's `Language.load(path)` reads it the
same way in both cases.
- Layer 2 keeps the 6-language support Wintermute shipped (TS/TSX/JS/Py/
Rb/Go). Layer 5 (E2a chunker parity) expands to all 36 bundled grammars.
- CHUNKER_VERSION=2 constant introduced. importCodeFile will fold this
into content_hash in Layer 3 so chunker-shape changes across releases
force clean re-chunks without the user needing `sync --force`.
CI guard — `scripts/check-wasm-embedded.sh` + `scripts/chunker-smoketest.ts`:
- Compiles a smoketest binary that calls chunkCodeText on a known TS
snippet.
- Asserts the output has `has_real_symbols: true`, a `[TypeScript]`
language tag, and the expected symbol name.
- If the chunker silently falls through to recursive chunks, the
assertions fail the build.
- Wired into `bun test` via package.json script pipeline. Also exposed
as `bun run check:wasm` for standalone invocation.
Verification:
- Dev: `bun -e '...'` smoke test returns 2 chunks with correct symbol
names in under 100ms.
- Compiled: `bash scripts/check-wasm-embedded.sh` passes end to end.
- Binary size: the gbrain binary grows from ~90MB to ~140MB, dominated
by the 50MB of grammar WASMs. Still well within normal for CLIs that
ship a language runtime.
* feat: v0.19.0 Layer 3 — schema migrations for page_kind + chunk code metadata
Adds two migrations to unblock C6/C7 (query --lang, code-def, code-refs)
and the orphans/auto-link branching in later layers.
v25 (pages_page_kind):
- ALTER TABLE pages ADD COLUMN page_kind TEXT NOT NULL DEFAULT 'markdown'
CHECK (page_kind IN ('markdown','code'))
- Postgres path uses ADD CONSTRAINT ... NOT VALID + VALIDATE CONSTRAINT
in a separate statement so tables with millions of pages don't hold a
write lock during the initial check. PGLite has no concurrent writers,
so its variant uses the simpler ALTER TABLE pattern.
- Existing rows carry DEFAULT 'markdown' — pre-v0.19 brains were
markdown-only by definition.
v26 (content_chunks_code_metadata):
- ALTER TABLE content_chunks ADD COLUMN language, symbol_name,
symbol_type, start_line, end_line (all nullable).
- Two partial indexes: idx_chunks_symbol_name WHERE symbol_name IS NOT
NULL, and idx_chunks_language WHERE language IS NOT NULL. Only code
chunks populate these columns, so partial indexes stay small even on
a 50K-chunk brain with mixed markdown+code.
- Markdown chunks leave all five columns NULL. Only importCodeFile
populates them, from the tree-sitter AST via chunkCodeText.
Wiring (both engines):
- PageInput gains `page_kind?: PageKind` ('markdown' | 'code'). Defaults
to 'markdown' when omitted so existing callers don't change. putPage
on both engines writes it through, with ON CONFLICT DO UPDATE updating
page_kind alongside the other fields.
- ChunkInput gains language, symbol_name, symbol_type, start_line,
end_line (all optional). upsertChunks on both engines writes them
through. Existing markdown call sites pass nothing and get NULLs —
zero behavior change for markdown pages.
importCodeFile updates:
- Sets page_kind='code' on the PageInput.
- Populates chunk metadata from the chunker's CodeChunk.metadata for
every chunk it persists. Columns line up 1:1 with the tree-sitter AST
output already produced by the chunker.
- Folds CHUNKER_VERSION=2 into content_hash so chunker shape changes
across releases force clean re-chunks without `sync --force`. The
hash was previously {title, type, content, lang} — now also
chunker_version.
Fresh-install path (src/schema.sql + pglite-schema.ts):
- Both include the page_kind column + CHECK constraint.
- Both include the five new content_chunks columns.
- Both ship the partial indexes so new brains have the same query
performance as migrated brains. Ran `bun run build:schema` to
regenerate src/core/schema-embedded.ts from schema.sql.
Naming: renamed our new Error subclass in src/core/errors.ts from
GBrainError to StructuredAgentError. The legacy GBrainError in
src/core/types.ts predates this change and has a different shape
(positional problem/cause/fix arguments) — keeping both under the same
name was inviting a year of import ambiguity. New v0.19.0 surfaces use
StructuredAgentError + the serializeError() helper.
Tests:
- test/migrations-v0_19_0.test.ts — 12 cases. Covers: MIGRATIONS array
shape (v25/v26 presence, NOT VALID pattern on Postgres, partial
index WHERE clauses), fresh-install schema (page_kind default, CHECK
constraint rejects invalid values, chunk metadata nullable), putPage
round-trip (markdown default + code explicit), upsertChunks
round-trip (code metadata preserved + markdown chunks leave NULLs).
- All 139 existing + new unit tests pass on PGLite (1.5 sec).
* feat: v0.19.0 Layer 4 — delete Wintermute's multi-repo, wire sources
Replaces Wintermute's short-lived repos abstraction with the v0.18.0
sources subsystem. Codex flagged this during plan review: v0.18.0's
sources table had already shipped the right shape (per-source
last_commit, federated search config, RLS-friendly) while Wintermute
coded against a ~/.gbrain/config.json repos array. Two systems solving
one problem.
Keep the surface, swap the backend:
- src/cli.ts: `gbrain repos` routes through runSources with a one-line
deprecation nudge on stderr. Scripts like `gbrain repos list` and
`gbrain repos add .` keep working against the sources table. Removed
the pre-engine-connect branch and added a case inside the
handleCliOnly switch so repos gets the DB connection it now needs.
- src/cli.ts help text: new SOURCES section replaces MULTI-REPO.
References the canonical `sources` commands with `repos` tagged
DEPRECATED.
sync --all — was iterating ~/.gbrain/config.json repos; now iterates
sources rows with local_path IS NOT NULL:
- Reads id, name, local_path, config jsonb via executeRaw.
- Honors config.syncEnabled=false (matching Wintermute's opt-out).
- Honors config.strategy for per-source markdown/code/auto filtering.
- Passes sourceId through to performSync so last_commit tracking lands
on the right sources row (was clobbering a global bookmark before).
Deletions:
- src/core/multi-repo.ts deleted (120 lines of config CRUD now handled
by sources table + RLS).
- src/commands/repos.ts deleted (121 lines of CLI parsing now handled
by src/commands/sources.ts).
- test/multi-repo.test.ts deleted (25 tests against the deleted module;
the schema-backed behavior is covered by test/sources.test.ts from
v0.18.0 + test/repos-alias.test.ts added here).
- src/core/config.ts: removed the `repos` field from GBrainConfig.
Legacy installs with `repos` in ~/.gbrain/config.json will see that
key ignored; no migration written because zero users are on that
path (Wintermute's commit never shipped on master).
Tests:
- test/repos-alias.test.ts — round-trips add/list/remove through
runSources to verify the alias path works. Also asserts the deleted
module is actually gone (catches accidental resurrection during
rebase conflicts).
- All 162 prior unit tests + 2 new = 164 pass on PGLite.
Codex's P0 #2 (per-repo sync state) and P0 #3 (slug collision) are
both resolved here — sources.last_commit scopes bookmarks per source,
and pages.slug uniqueness is (source_id, slug), which is what the
v0.18.0 schema already shipped.
* feat: v0.19.0 Layer 5 — Chonkie chunker parity (E2a)
Expands Wintermute's 6-language chunker to 29 languages, swaps the
heuristic tokenizer for the real thing, and adds small-sibling merging
so a file of 20 tiny const declarations doesn't produce 20 embedding
calls. This closes the Chonkie gap Garry called out in CEO review.
Language coverage — 6 → 29:
- Added grammars: rust, java, c_sharp, cpp, c, php, swift, kotlin,
scala, lua, elixir, elm, ocaml, dart, zig, solidity, bash, css,
html, vue, json, yaml, toml. All shipping in src/assets/wasm/
(committed in Layer 2). Bun's --compile bundles every import
attributes path, so the compiled binary carries every grammar.
- TOP_LEVEL_TYPES populated for the 11 most-used new languages
(rust, java, c_sharp, cpp, c, php, swift, kotlin, scala, lua,
elixir, bash, solidity) + the original 6. Tree-sitter loads the
grammar but the chunker falls through to recursive chunking when
TOP_LEVEL_TYPES isn't set — still correct output, just less
semantic. Every grammar ships with a working fallback.
- detectCodeLanguage extended for 29 extension families including
.mts/.cts (TypeScript), .cc/.hpp/.cxx (C++), .kt/.kts (Kotlin),
.scala/.sc (Scala), .ex/.exs (Elixir), etc.
- DISPLAY_LANG table lookup replaces the inline 6-entry map;
structured headers now read '[Rust]', '[C#]', '[PHP]' etc.
Accurate tokenizer:
- @dqbd/tiktoken with cl100k_base encoding (same encoder
text-embedding-3-large uses). Lazy-loaded on first call via
require() so dev and compiled binary share the init path.
- Falls back to the old len/4 heuristic only if the encoder fails
to initialize (vanishingly unlikely — keeps the chunker available
instead of throwing).
- Existing estimateTokens call sites (large-node threshold +
sub-range splitting + new merge pass) all now see real counts.
Real code is 2-3x more token-dense than prose; the old heuristic
systematically under-split so large functions sometimes exceeded
the embedding API's 8191-token hard cap.
Small-sibling merging:
- New mergeSmallSiblings post-pass runs on the chunk list after
tree-sitter extraction.
- Adjacent chunks under 40% of chunkSizeTokens get accumulated
into one merged chunk up to the full budget.
- Large chunks (functions, classes) pass through untouched.
- Merged chunks get symbolName=null, symbolType='merged',
startLine/endLine spanning the group. The header reads:
'[Lang] path:N-M merged (K siblings)' so retrieval can still
show coherent context.
- Mirrors Chonkie's CodeChunker._group_child_nodes() +
bisect_left accumulation. A Go file with 30 top-level imports +
5 functions no longer produces 30 separate import chunks.
CHUNKER_VERSION bumped 2 → 3:
- Any existing v0.18.x brain with code pages will re-chunk on next
sync because content_hash folds CHUNKER_VERSION in. Without the
bump, stale (2-3x token-off, non-merged) chunks would persist
forever until manual 'sync --force'.
CI guard + smoketest updates:
- scripts/chunker-smoketest.ts replaced the tiny hello/Foo/Id
fixture with a realistic TS snippet (calculateScore with branches
+ UserRegistry class) so at least one chunk has a concrete symbol
name — small-sibling merging would otherwise collapse the old
fixture and fail the assertion.
- scripts/check-wasm-embedded.sh assertions updated: check
has_symbol_names:true (at-least-one-real-symbol), still verify
[TypeScript] header and specifically the calculateScore symbol.
Tests — test/chunkers/code.test.ts (15 cases):
- CHUNKER_VERSION=3 shape assertion (guards silent re-chunking
across releases).
- detectCodeLanguage across 29 extensions + unknown + case-insensitive.
- chunkCodeText on TypeScript / Python / Rust / Go producing chunks
with correct language tag + symbol names.
- Fallback path for unsupported extension produces recursive-chunk
module-kind output.
- Small-sibling merging: 5 tiny consts → 1-2 chunks; big function
passes through untouched; merged chunk line range spans group.
- Structured header shape: starts with [Lang], contains file path,
line range, symbol name.
- Empty input returns empty array.
All 177 unit tests pass + CI guard on compiled binary passes.
* feat: v0.19.0 Layer 6 — incremental chunking + doc↔impl linking
Two expansions from the plan's E1 + E2. E3 (markdown fence extraction)
deferred to a follow-up PR — the feature surface is small and doesn't
block the main cathedral.
E1 — Design-doc ↔ implementation linking:
- New extractCodeRefs() in src/core/link-extraction.ts. Scans markdown
prose for references like 'src/core/sync.ts:42'. Anchored on a
prefix allowlist (src|lib|app|test|tests|scripts|docs|packages|
internal|cmd|examples) + the 39-extension code file list so random
phrases like 'foo/bar.js' don't generate false-positive edges. Dedups
by path (first occurrence wins).
- importFromContent writes bidirectional edges for every code ref
found in compiled_truth + timeline:
markdown_slug --[documents]--> code_slug
code_slug --[documented_by]--> markdown_slug
Both use link_source='markdown', origin_page_id=markdown_slug,
origin_field='compiled_truth' so runAutoLink reconciliation scopes
edges correctly.
- addLink's inner SELECT naturally drops edges to non-existent pages,
so a markdown guide imported before the code repo is synced writes
no edges — they'll land when the code arrives via A3 reverse-scan
(deferred to a follow-up since it only activates for users who sync
markdown and code in opposite order).
E2 — Incremental chunking:
- importCodeFile reads existing chunks via engine.getChunks(slug)
before embedding.
- Keys existing chunks by `${chunk_index}:${chunk_text}`. Any new
chunk that matches verbatim at the same index reuses the existing
embedding (chunk.embedding + token_count). Only new/changed chunks
go to embedBatch.
- Cost impact: a daily autopilot on a stable repo touches ~2-5% of
chunks on each run. E2 cuts OpenAI embedding spend by ~95% vs
naive full re-embed. Stated before (Codex A2 decision) and now
actually implemented.
- Uses chunk_index + chunk_text as the key (not symbol_fqn) because
the tree-sitter chunker already makes chunk_index semantic — it's
AST-order. A blank line at the top of a file shifts start_byte
for every chunk below but leaves chunk_text identical, so the
cache still hits.
- Fallback: when embedBatch throws (rate-limit, network, etc.) the
existing warn-but-continue behavior stays. Un-embedded chunks land
in the DB with NULL embedding; a later `embed --stale` will fix
them.
Tests (test/link-extraction-code-refs.test.ts, 10 cases):
- :line suffix capture.
- Prefix allowlist (11 directories).
- Extension recognition (39 extensions).
- Rejects paths outside allowlisted prefixes.
- Rejects non-code extensions.
- Dedup by path (first occurrence wins).
- Different paths coexist.
- Real-markdown integration: guide with 4 code refs (one with line
number) produces the right set of paths.
- Doesn't match URL-like strings (word-boundary behavior).
Tests (test/incremental-chunking.test.ts, 3 cases):
- Identical content re-import skips entirely (content_hash match).
- Editing ONE function in a 3-function file preserves the other two
chunks verbatim (same chunk_text in DB). Verifies the cache-hit
path actually works end-to-end on PGLite.
- Fresh-file import embeds all chunks (nothing to reuse).
All 189 unit tests pass on PGLite.
* feat: v0.19.0 Layer 7 — code-def + code-refs CLI surfaces
Delivers the magical-moment commands for v0.19.0 code indexing. These
are the agent-facing endpoints that turn 'brain-first lookup' from a
markdown-only Iron Law into something that covers code too.
gbrain code-def <symbol>:
- Queries content_chunks.symbol_name = $1 AND page_kind = 'code' AND
symbol_type IN (function, class, interface, type, enum, struct,
trait, module, contract, export statement).
- Orders by symbol_type rank (function first, then class, etc.) then
page slug then line number — deterministic across runs.
- --lang <language> filter narrows to a single language.
- --limit N caps results (default 20).
- Returns Array<{ slug, file, language, symbol_type, start_line,
end_line, snippet }> — the 7-field shape the agent persona needs.
gbrain code-refs <symbol>:
- Bypasses the standard searchKeyword path, which uses DISTINCT ON
(slug) to collapse results to one chunk per page. That collapse is
right for markdown search but wrong for code-refs — a single file
typically has many usage sites, each interesting to the agent.
- Direct ILIKE scan over content_chunks + JOIN pages WHERE page_kind
= 'code'. Word-boundary precision is a follow-up (would need
tsvector or regex); for v0.19.0 the substring heuristic is good
enough because symbol names are distinctive by design.
- Same --lang / --limit / --json flag surface as code-def.
- Returns Array<{ slug, file, language, symbol_name, symbol_type,
start_line, end_line, snippet }> — 8 fields (code-def + the
containing symbol_name).
Agent-DX doctrine (from DX review):
- Auto-JSON on pipe: both commands emit JSON when stdout is not a
TTY (gh-CLI convention). Explicit --json forces JSON on TTY;
--no-json forces human output even when piped.
- Structured error envelope: missing symbol argument returns
{ class: 'UsageError', code: '..._requires_symbol', hint: '...' }
serialized as JSON in non-TTY mode, plain message in TTY.
Catch-all DB error path uses serializeError() — no raw stack
traces leak to the agent.
Tests — test/code-def-refs.test.ts (10 cases):
- Seeds a fixture repo (two TS files with deliberately large symbols
to stay independent under small-sibling merging).
- findCodeDef:
- Resolves interface + function by name to the right file.
- Empty-symbol query returns [].
- Language filter narrows to typescript; python returns [].
- findCodeRefs:
- Finds multiple usage sites across files (both src/engine.ts
and src/sync.ts appear when searching for BrainEngine — this
is the DISTINCT ON bypass working).
- Deterministic ordering by slug + line number.
- Unknown symbol returns [].
- --limit caps result count.
- Snippets are <= 500 chars (the agent doesn't get flooded).
CLI wiring:
- Added 'code-def', 'code-refs' to CLI_ONLY.
- New switch cases in handleCliOnly call runCodeDef / runCodeRefs.
- Help text gains a CODE INDEXING (v0.19.0) section.
All 199 unit tests pass.
Deferred from Layer 7 per the cathedral plan:
- sync --all cost preview with TTY detection — requires folding the
tokenizer into the sync path. Pushed to a follow-up.
- query --lang filter — requires changes to src/core/search/*.ts.
Pushed to a follow-up.
* feat: v0.19.0 Layer 8 — BrainBench code category (E2E)
Retrieval-quality gate for v0.19.0 code indexing. Seeds a ~25-file
fictional corpus across 5 languages (TS, Python, Go, Rust, Java),
imports each via importCodeFile, and asserts code-def + code-refs
produce the expected shape. Runs against PGLite in-memory so no
OpenAI key or external Postgres is needed; reproducible on CI with
just Bun.
What the E2E covers:
- Corpus seeded: 25+ code pages, all page_kind='code'.
- code-def finds AuthService across multiple languages (≥2 of
TS/Rust/Java).
- code-def --lang typescript filters precisely (P@5=1.0 for
CacheService + typescript).
- code-refs surfaces multiple usage sites across files (the
DISTINCT ON bypass working in practice).
- code-refs over the shared "start" method across 5 languages
produces ≥3 language hits (ranking stability).
- Magical-moment assertion: code-refs completes in <500ms on a
25-file corpus (budget is 100ms; 500ms pad absorbs CI variance).
- MRR sanity: top result for exact symbol is the defining file.
- Edge cases: non-existent symbol returns [], not error. Language
filter with zero matches returns []. Re-import is idempotent.
Chunker retune:
- Small-sibling merge threshold dropped from 40% to 15% of
chunkSizeTokens. The 40% figure was collapsing 3-method classes
into 'merged' chunks, killing symbol_name lookups for the entire
class. 15% matches the original intent: merge truly tiny
declarations (const X = 1; import ... from ...;) while leaving
substantive symbols (functions, classes) independent. Verified
by the BrainBench test — AuthService is now its own chunk with
symbol_name='AuthService', so findCodeDef('AuthService') resolves.
- Unit test updated: 10 consts with a generous chunkSizeTokens=1000
still exercise the merge path.
Total v0.19.0 unit + E2E coverage: 91 tests across 9 new test
files, 357 assertions, all green.
* feat: v0.19.0 Layer 9 — release: CHANGELOG + migration + docs
Closes out the v0.19.0 cathedral. Total shipped across 10 layers:
- 91 new unit + E2E tests (9 new files, 357 assertions, all green)
- 2 schema migrations (v25 pages.page_kind + v26 content_chunks code metadata)
- 4 new CLI surfaces (repos [alias] + code-def + code-refs +
sources passthrough)
- 1 new core module (src/core/errors.ts)
- 36 tree-sitter grammar WASMs embedded via Bun --compile
- 1 CI guard preventing silent-chunker regression
- Wintermute's multi-repo replaced with v0.18.0 sources backend
CHANGELOG.md — release-summary section in the GStack/Garry voice per
CLAUDE.md "Release-summary template": bold two-line headline + lead
paragraph + "The numbers that matter" table + "What this means for
builders" + itemized changes + "To take advantage of v0.19.0" block.
No em dashes, no AI vocabulary, no banned phrases. Numbers are from
the v0.19.0 test-fixture benchmarks.
CLAUDE.md — four new file entries in the Key files section
(src/core/chunkers/ annotated with v0.19.0 additions, src/core/errors.ts,
src/assets/wasm/, src/commands/code-def.ts + code-refs.ts).
skills/migrations/v0.19.0.md — agent-readable migration walkthrough
per the v0.11.0 convention. Tells the agent what to do after
`gbrain upgrade` runs the orchestrator: verify schema v26, register a
code source via `gbrain sources add`, run `sync --source <id>`,
confirm `gbrain code-def` / `code-refs` both work. Notes the deprecated
`gbrain repos` alias for scripts that used Wintermute's baseline.
Flagged in pending-host-work.jsonl per the v0.11.0 convention so
headless agents surface the prompt.
VERSION — 0.18.2 → 0.19.0.
All 91 v0.19.0 tests + the CI guard pass.
* docs: v0.19.0 — add 4 deferred follow-ups to TODOS.md
Lands the four items the v0.19.0 cathedral explicitly scoped out but
that the /plan-ceo-review + /plan-devex-review + /plan-eng-review chain
identified as genuine follow-ups rather than abandoned ideas.
Items added under a new 'code-indexing (v0.19.0 follow-ups)' section:
- P1 — sync --all cost preview with TTY detection. Closes DX fix #1
from the /plan-devex-review pass: the agent persona can't respond
to stdin prompts. Non-TTY path must emit a parseable
ConfirmationRequired envelope; TTY path uses [y/N]. File refs:
src/commands/sync.ts:590, src/core/chunkers/code.ts estimateTokens,
src/core/errors.ts buildError.
- P2 — query --lang filter through src/core/search/*.ts. Column
ships in v0.19.0 (migration v26 + partial index); the query path
just needs to respect it. Keeps ranking honest when the user
knows the language. File refs: src/core/search/, pglite-engine
searchKeyword, test/e2e/code-indexing.test.ts language-filter
pattern.
- P2 — E3 markdown code-fence extraction. After parseMarkdown,
iterate marked's lexer tokens for { type: 'code', lang, text }
and chunk each through chunkCodeText with chunk_source='fenced_code'.
~40% of gbrain's brain is guides with substantial inline code —
this lands those fences as first-class TS/Python/Go chunks in
search instead of treating them as prose.
- P2 — A3 reverse-scan backfill for doc↔impl. Companion piece to
E1. Markdown-first → code-later import order currently loses edges
because addLink's JOIN drops them when the code page doesn't exist
yet. A3 makes importCodeFile scan existing markdown for
references to the new code path and backfill edges both
directions. Trade-off: per-file scan is expensive on first sync;
batch 'gbrain reconcile-links' is an alternative shape.
Each entry follows the CLAUDE.md TODOS format: What/Why/Pros/Cons/
Context with exact file refs/line numbers/Effort (S/M/L + human vs
CC)/Depends on. All four are purely additive on top of v0.19.0 —
nothing blocks.
* fix: pre-existing test infrastructure + typecheck drift
Three pre-existing conditions surfaced when running the full suite and
blocked a clean CI floor for Cathedral II work:
1. `bun run test` default 5s hook timeout fails under load. PGLite WASM
init can exceed 5s when many test files spin up instances in parallel.
The bunfig.toml `timeout = 60_000` key is honored by `bun test` but
does not propagate to beforeEach/afterEach hooks when `bun test` runs
behind `bun run typecheck` in the CI chain. Pass `--timeout=60000`
explicitly on the command line, where it covers both per-test and
per-hook timeouts.
Before: 2136 pass / 30 fail (on-branch baseline)
After: 2272 pass / 0 fail
All 30 failures were `beforeEach/afterEach hook timed out for this
test` → `TypeError: undefined is not an object (evaluating
'engine.disconnect')` — i.e. the hook never finished connecting
PGLite, so the engine variable was never assigned, so afterEach
tripped on `engine.disconnect()`. The new timeout gives PGLite
WASM init enough headroom under concurrent load.
2. `test/repos-alias.test.ts` references the deliberately-deleted
`src/core/multi-repo.ts` via a dynamic import inside a try/catch
(the test asserts the module is no longer importable at runtime).
TS 5.x module resolution flags this at typecheck time even inside
try/catch. Build the path at runtime (`'../src/core/' +
'multi-repo.ts'`) so TS's compile-time module resolution doesn't
fail on a path the test is EXPLICITLY verifying doesn't resolve.
3. `llms-full.txt` drifted from `bun run build:llms` output (earlier
CLAUDE.md updates in v0.19.0 never regenerated). `bun run build:llms`
now produces matching output.
Zero behavior changes to production code. Test infrastructure only.
* feat: v0.20.0 Cathedral II Layer 1 — Foundation schema migration
Layer 1 of 14 for the v0.20.0 "best code search in the world" cathedral.
Ships all Cathedral II DDL atomically so downstream layers have the
columns + tables + trigger they depend on. Schema-only; no consumer
behavior changes until Layer 5 (A1 edge extractor).
Reordered to Layer 1 after codex second-pass review (SP-4): previously
Layer 0b (chunk-grain FTS trigger) referenced columns added in the
former Layer 3 (Foundation), breaking bisectability. All schema DDL
now lands first; every subsequent layer's prerequisites exist.
### What this migration adds (one idempotent v27 transaction)
1. `content_chunks` gains 4 new columns:
- `parent_symbol_path TEXT[]` — scope chain for nested symbols (A3)
- `doc_comment TEXT` — extracted JSDoc/docstring (A4)
- `symbol_name_qualified TEXT` — 'Admin::UsersController#render' (A1)
- `search_vector TSVECTOR` — chunk-grain FTS (Layer 1b consumer)
All nullable; markdown chunks leave them NULL.
2. `sources.chunker_version TEXT` (SP-1 gate). Layer 10 will check this
against CURRENT_CHUNKER_VERSION and force a full sync walk on
mismatch, bypassing the git-HEAD up_to_date early-return that would
otherwise make a bare CHUNKER_VERSION bump a silent no-op.
3. `code_edges_chunk` — resolved call-graph + reference edges.
- `from_chunk_id` + `to_chunk_id` with FK CASCADE from content_chunks
- UNIQUE (from_chunk_id, to_chunk_id, edge_type) holds idempotency
- `source_id TEXT` matches `sources.id` actual type (codex F4 caught
the prior UUID typo)
- source scoping enforced in resolution logic, not the key, because
from_chunk_id → pages.source_id already determines it
4. `code_edges_symbol` — unresolved refs. Target symbol known by
qualified name; defining chunk not seen yet. Rows UNION with
code_edges_chunk on read (codex 1.3b); no promotion step (SP-7).
5. `update_chunk_search_vector` trigger — BEFORE INSERT/UPDATE OF
(chunk_text, doc_comment, symbol_name_qualified). Weights
doc_comment and symbol_name_qualified at 'A', chunk_text at 'B'.
Natural-language queries rank doc-comment hits above body text
(A4 intent, delivered via the trigger from day one even though
Layer 5 populates the doc_comment column).
### Engine interface + types
- `BrainEngine` gains 6 new methods for code edges, all stubbed in
both engines with explicit NotImplemented errors pointing at the
layer that will fill them (5, 7, or 1b):
addCodeEdges, deleteCodeEdgesForChunks, getCallersOf,
getCalleesOf, getEdgesByChunk, searchKeywordChunks
- `CodeEdgeInput`, `CodeEdgeResult` types added to src/core/types.ts
- `SearchOpts` extended with Cathedral II fields: language, symbolKind,
nearSymbol, walkDepth, sourceId (all optional; consumers wire in
Layer 5/7/10)
- `ChunkInput` extended with: parent_symbol_path, doc_comment,
symbol_name_qualified (populated by importCodeFile in Layer 5/6)
- `Chunk` read shape mirrors the added columns as optional fields
- `chunk_source` union widens to include 'fenced_code' for D2 fence
extraction (Layer 6 consumer)
### Tests
`test/migrations-v0_20_0.test.ts` — 17 structural assertions against
the v27 migration registry. Covers every column + table + index + the
trigger weight shape. E2E migration-application coverage lands in
`test/e2e/cathedral-ii.test.ts` alongside Layer 5.
### Status
- CEO + Eng + 2 codex passes CLEARED (see docs/designs/CODE_CATHEDRAL_II.md)
- 16 cross-model findings absorbed (7 codex pass 1 + 6 codex pass 2
+ 3 eng review)
- 13 more layers to go (0a → 14); see plan for full sequencing.
* feat: v0.20.0 Cathedral II Layer 2 (1a) — file-classifier widening + SP-5 slug dispatch
Codex F1: `sync.ts:35` v0.19.0 classified only 9 extensions as code.
Rust/Java/C#/C++/Swift/Kotlin/etc. never reached the chunker on a
normal repo sync, making v0.19.0's "29 languages" claim aspirational
on the read path. Layer 2 widens the classifier so every language the
chunker knows (~35 extensions) actually reaches it during sync.
### Changes
1. `src/core/sync.ts` CODE_EXTENSIONS widened from 9 to 35 extensions,
matching the chunker's detectCodeLanguage coverage: adds .rs, .java,
.cs, .cpp/.cc/.cxx/.hpp/.hxx/.hh, .c/.h, .php, .swift, .kt/.kts,
.scala/.sc, .lua, .ex/.exs, .elm, .ml/.mli, .dart, .zig, .sol,
.sh/.bash, .css, .html/.htm, .vue, .json, .yaml/.yml, .toml,
.mts/.cts.
2. `src/core/sync.ts` adds `resolveSlugForPath(path)` — SP-5 fix.
Before Cathedral II, sync delete/rename paths called
`pathToSlug(path)` with default pageKind='markdown'. For the 9-ext
classifier this was mostly fine (code files rare), but widening to
35 exts means Rust/Java/Ruby/etc. deletes and renames would mismatch
on slug shape (pathToSlug markdown-style vs slugifyCodePath
code-style). resolveSlugForPath dispatches on isCodeFilePath so
delete/rename always hit the right page. Used in `src/commands/sync.ts`
at the three slug-resolution sites (un-syncable delete, batch delete,
rename from/to).
3. `src/core/chunkers/code.ts` adds `setLanguageFallback(fn)` +
optional `content` arg to `detectCodeLanguage(path, content?)`.
Pre-wires the Magika fallback hook that Layer 9 (B2) will consume
for extension-less files (Dockerfile, Makefile, shell shebangs).
Null default → no behavior change today; Layer 9 sets it at bootstrap.
Fallback throws are swallowed (recursive chunker is always an
acceptable degradation).
### Tests
- `test/sync-classifier-widening.test.ts` — 20 cases covering the full
widened extension set, resolveSlugForPath dispatch, and the Magika
fallback hook contract (including throw-swallow and null-pass-through).
- `test/sync-strategy.test.ts` updated: `.json` is no longer rejected
(the chunker's language map includes JSON for structured-data
chunking). Test clarifies Cathedral II semantics; adds .svg + .zip
as non-code examples.
### CI result
2292 pass / 0 fail via `bun run test`, 388s wall time.
* feat: v0.20.0 Cathedral II Layer 3 (1b) — chunk-grain FTS with page-grain wrap
Codex F2 caught that v0.19.0's searchKeyword ranked via pages.search_vector,
so doc-comment content living on a chunk couldn't influence ranking and A2
two-pass retrieval had no way to find the best matching chunk. Layer 3
moves the FTS primitive to content_chunks.search_vector (the column +
trigger added in Layer 1/v27), dedups-to-best-chunk-per-page on return
so every external caller still sees the v0.19.0 page-grain contract
(SP-6), and exposes searchKeywordChunks as the raw chunk-grain primitive
A2 two-pass will consume (Layer 7).
### Backfill migration v28
Layer 1's trigger only fires on INSERT/UPDATE — rows inserted before v27
applied had NULL search_vector. v28 backfills every existing chunk with
the same weight shape the trigger uses (doc_comment + symbol_name_qualified
at weight A, chunk_text at B). Idempotent via `WHERE search_vector IS NULL`;
re-runs pick up only remaining NULL rows. ~2-3s on a 20K-chunk brain.
### searchKeyword rewrite (both engines)
CTE chain: rank chunks by cc.search_vector → DISTINCT ON (slug) picks
best chunk per page → order by score → limit. External shape identical
to v0.19.0: one row per matched page, score comes from the best chunk
on that page, chunk metadata attached. Zero breaking changes for
backlinks counting, enrichment-service.countMentions, list_pages, etc.
Inner fetch limit is 3x the requested page limit so dedup has enough
chunks to produce N distinct pages (a co-occurring-term cluster in one
page can't eat the result set).
Postgres keeps the SET LOCAL statement_timeout='8s' from v0.12.3 search
timeout scoping. PGLite gets the same CTE shape minus the transaction-
scoped GUC (PGLite has no pool).
### searchKeywordChunks (new internal primitive)
Same chunk-grain ranking WITHOUT dedup. Returns raw top-N chunks by
FTS score regardless of page. Used by A2 two-pass retrieval (Layer 7)
as its anchor-discovery primitive — two-pass wants top chunks, not
best-per-page. Most callers should prefer searchKeyword.
### Tests
- test/chunk-grain-fts.test.ts: 11 cases covering migration v28 shape,
page-grain external contract (dedup preserves invariants), chunk-grain
primitive (no dedup, score-ordered), and the doc-comment weight-A
precedence over body weight-B — the A4 ranking win validated today
even though Layer 5 is what populates doc_comment from AST.
- test/pglite-engine.test.ts existing "tsvector trigger populates
search_vector on insert" updated: v0.19.0 searched pages.search_vector
(built from title + compiled_truth) so two-word queries matching
non-chunk text worked. Cathedral II ranks chunks only — test updated
to search 'AI agents' which is in the chunk_text directly.
- test/migrations-v0_20_0.test.ts "v27 is highest" relaxed to
"v27 is the foundation migration; max >= 27" so later layers can
land migrations without breaking this assertion.
### CI result
2553 tests / 0 fail via `bun test --timeout=60000`, 422s wall time.
* feat: v0.20.0 Cathedral II Layer 4 (B1) — language manifest foundation
Consolidate the 29-way GRAMMAR_PATHS + parallel DISPLAY_LANG record into
a single LANGUAGE_MANIFEST keyed on SupportedCodeLanguage. Each entry is
a LanguageEntry with { displayName, embeddedPath?, lazyLoader? }.
### Why this matters for Cathedral II
Before: adding a language meant editing two maps (path + display name)
AND adding a new `import G_X from ...` at the top, for every new lang.
After: one manifest entry + one `with { type: 'file' }` import (embedded)
or one registerLanguage() call at boot (lazy). loadLanguage() consults
the manifest uniformly — it doesn't know or care whether a grammar is
embedded in the compiled binary or resolved from node_modules at runtime.
### The 3 extension points
- `embeddedPath` — Bun `with { type: 'file' }` asset. Ships with
`bun --compile` output; already in place for the 29 core grammars.
- `lazyLoader` — async function returning path or Uint8Array. Used at
first reference, then cached in `languageCache` like embedded grammars.
Forward-compat for v0.20.x+ full tree-sitter-wasms (~136 more langs).
- `registerLanguage(lang, entry)` / `unregisterLanguage(lang)` /
`listRegisteredLanguages()` — runtime registration hook. Layer 9
(B2 Magika) will wire detection for extensionless files through
this API. Dynamic registrations win over core manifest on conflict
so hot-fix overrides during a session work without restart.
### Behavior guarantees preserved
- All 29 v0.19.0 core grammars continue to ship embedded — no binary-size
growth, no runtime network dependency for the core set.
- `detectCodeLanguage` untouched; its output key still maps 1:1 through
LANGUAGE_MANIFEST.
- `displayLang()` now derived from the manifest. Chunk headers read
"[Python]" / "[TypeScript]" / "[Ruby]" just as before — one source of
truth, manifest-derived.
### Tests (test/language-manifest.test.ts, 8 cases)
- Manifest covers all 29 v0.19.0 languages (typescript/tsx/js/py/rb/go/
rust/java/c_sharp/cpp/c/php/swift/kotlin/scala/lua/elixir/elm/ocaml/
dart/zig/solidity/bash/css/html/vue/json/yaml/toml).
- registerLanguage does NOT invoke the lazy loader at registration time
(proves the loader fires at most on first chunkCodeText() call).
- Dynamic registrations override core manifest entries (hot-fix path).
- unregisterLanguage removes a dynamic entry and clears its parser cache.
- chunkCodeText still loads core grammars (TypeScript / Python / Ruby)
end-to-end; chunk headers use the manifest displayName ("[Python]",
not "[python]").
### What's NOT shipped here
Adding the additional ~136 languages from tree-sitter-wasms is
deliberate v0.20.x+ follow-up work. The manifest infrastructure is in
place; expanding coverage is now a data-only PR (one entry per language).
### CI result
2561 tests / 0 fail via `bun test --timeout=60000`, 425s wall time.
* feat: v0.20.0 Cathedral II Layer 8 D1 — sync --all cost preview + ConfirmationRequired envelope
Closes the v0.19.0 DX review's #1 pain point: "first sync surprise bill."
Before Cathedral II, `gbrain sync --all` on a fresh multi-source brain
could spin up tens of thousands of OpenAI embedding calls before anyone
saw a cost number. Agent callers (OpenClaw, Hermes, etc.) had no way
to gate the operation behind a spend check.
### Behavior
Before `sync --all` touches a single source, walk the working trees of
every registered source with `local_path`, sum tokens per file via the
same cl100k_base tokenizer text-embedding-3-large actually uses, and
compute a USD estimate. Gate on that:
- **TTY + !--json + !--yes** → interactive `[y/N]` prompt.
- **non-TTY OR --json OR piped** → emit `ConfirmationRequired` envelope
to stdout via the v0.18 `errorFor` builder, exit code 2. Reserves
exit 1 for runtime errors so agent callers can distinguish
"awaiting user call" from "something crashed."
- **--yes** → skip prompt entirely. Agent/CI path.
- **--dry-run** → print preview, exit 0 without syncing.
- **--no-embed** → skip the cost gate entirely (user already opted out
of OpenAI spend; they'll run `embed --stale` later).
### Preview shape
One stderr line or one JSON payload:
sync --all preview: <N> files across <M> source(s),
~<T> tokens, est. $<X> on text-embedding-3-large.
Conservative overestimate: full working-tree content, not just the
incremental diff. A source never embedded before WILL embed everything
on first sync; already-synced sources with small diffs get a ceiling,
not a floor. False-high bias is intentional — users never get
surprised by MORE cost than the preview claimed.
### Files
- `src/core/chunkers/code.ts`: `estimateTokens` now exported (was
module-private). Same cl100k_base tokenizer, just a public symbol.
- `src/core/embedding.ts`: add `EMBEDDING_COST_PER_1K_TOKENS = 0.00013`
+ `estimateEmbeddingCostUsd(tokens)`. Single source of truth for
cost math; every cost-preview surface reads this constant, so a
pricing change is a one-line edit.
- `src/commands/sync.ts`:
- new `estimateSyncAllCost(sources)` helper walks trees, sums
tokens per active source, returns breakdown.
- new `walkSyncableFiles(repo, cb, strategy)` recursive walker.
Honors the same `isSyncable` rules as the real sync so preview
and execution agree on scope. Skips hidden dirs, node_modules,
ops/, and files over 5MB. Best-effort file-read errors don't
block the preview.
- new `promptYesNo(question)` readline wrapper — resolves false
on non-'y' answer OR EOF.
- `--yes` and `--json` flags parsed at sync argv layer.
- cost preview runs before the per-source sync loop on `--all`,
gates via the TTY / --json / --yes / --dry-run matrix above.
### Tests
`test/sync-cost-preview.test.ts` (6 cases):
- EMBEDDING_COST_PER_1K_TOKENS pinned to $0.00013.
- `estimateEmbeddingCostUsd` scales linearly across 0 → 1M tokens.
- `estimateTokens` round-trips (empty → 0, short → <10, 100x text → >50x).
### CI result
2567 tests / 0 fail via `bun test --timeout=60000`, 424s wall time.
* feat: v0.20.0 Cathedral II Layer 8 D2 — markdown fence extraction
~40% of gbrain's brain is docs + guides + architecture notes with
substantial inline code. In v0.19.0 those fenced code blocks chunked as
prose, so querying "how do we handle errors in TypeScript" ranked
paragraphs ABOUT the import above the actual import example. D2 walks
the marked lexer tokens, extracts each recognized code fence, and
persists them as extra chunks on the parent markdown page with
`chunk_source='fenced_code'` and full code-metadata (language,
symbol_name, symbol_type, start/end line).
### Behavior
In `importFromContent`, after `parseMarkdown` returns compiled_truth,
we additionally run the text through `marked.lexer()` and walk for
`{ type: 'code', lang, text }` tokens. For each:
- Map the fence language tag (`ts`/`typescript`/`js`/...) to a
pseudo-path (`fence.ts`/`fence.js`/...) so `detectCodeLanguage`
picks the right grammar.
- Call `chunkCodeText(text, pseudoPath)` — one or more code chunks
depending on fence size. Tree-sitter-aware chunking means a big
TS fence splits at function boundaries, not character count.
- Persist each chunk with `chunk_source='fenced_code'`. Extends the
existing chunk_source enum; schema allows it via the TEXT column.
### Fence-bomb DOS guard
`MAX_FENCES_PER_PAGE = 100` by default, overridable via
`GBRAIN_MAX_FENCES_PER_PAGE` env var. A malicious markdown page with
10K ```ts blocks could otherwise force 10K embedding API calls.
Beyond the cap, remaining fences skip with a one-line console warn
so operators can see the event.
### Per-fence error isolation
Each fence runs through its own try/catch. One malformed fence (e.g.
marked lexer choking on edge-case markdown) doesn't abort the whole
page import — the other fences + the prose chunks from
compiled_truth all still land.
### Recognized fence tags (29 languages + 7 aliases)
ts/typescript, tsx, js/javascript, jsx, py/python, rb/ruby,
go/golang, rs/rust, java, c#/cs/csharp, cpp/c++, c, php, swift,
kt/kotlin, scala, lua, ex/elixir, elm, ml/ocaml, dart, zig,
sol/solidity, sh/bash/shell/zsh, css, html, vue, json, yaml/yml,
toml.
Unknown tag → skipped (no synthetic chunk, no crash). Missing tag
(```\n...\n```) → skipped. Empty body → skipped.
### Collateral fix
`rowToChunk` in src/core/utils.ts now maps the code-chunk metadata
columns (language, symbol_name, symbol_type, start_line, end_line)
+ the v0.20.0 Cathedral II additions (parent_symbol_path,
doc_comment, symbol_name_qualified) out of the DB. Pre-Cathedral II
the code columns were written via upsertChunks but never read back
— caught by the new fence test assertions.
### Tests (test/fence-extraction.test.ts, 7 cases)
- TS fence → language='typescript' chunk
- Python fence → language='python', chunk_text contains def
- Ruby fence → language='ruby'
- Unknown tag (```mermaid, ```unknown-xyz) → no fenced_code chunks
- Missing tag → no fenced_code chunks
- 3 fences on one page, mix of langs → 3+ fenced_code chunks
- Empty fence body → no chunks
### CI result
2574 tests / 0 fail via `bun test --timeout=60000`, 434s wall time.
* feat: v0.20.0 Cathedral II Layer 8 D3 — reconcile-links batch command
Closes the v0.19.0 Layer 6 doc↔impl order-dependency: when a markdown
guide imports BEFORE the code it cites (common — docs land first, code
sync runs second), the Layer 6 E1 forward-scan calls addLink but its
inner JOIN silently drops the edge because the code page doesn't exist
yet. The guide and the code eventually both exist in the brain, but
the edge never materialized.
### New CLI surface
gbrain reconcile-links [--dry-run] [--json]
Walks every markdown page, re-runs `extractCodeRefs` on
compiled_truth+timeline, and calls addLink(md, code, ..., 'documents')
+ reverse for each hit. ON CONFLICT DO NOTHING at the links table
makes the operation idempotent — existing edges stay, new edges land.
### Per-lang coverage via extractCodeRefs
Inherits the regex from `src/core/link-extraction.ts` which already
recognizes code paths for 29 extensions (ts/tsx/js/py/rb/go/rust/java/
c#/cpp/c/php/swift/kotlin/scala/lua/elixir/elm/ocaml/dart/zig/sol/sh/
css/html/vue/json/yaml/toml). Fence-extraction (D2) and classifier-
widening (Layer 2) keep this in sync with the chunker's actual reach.
### Why batch over per-import reverse-scan
Codex's two-pass review flagged per-import reverse-scan as O(N)
ILIKE/JOIN queries per code file imported — on a 47K-page brain first-
syncing 5K code files that's 5K ILIKE scans. A user-triggered batch
run on an already-synced brain is one walk, slug-indexed via addLink's
existing lookup. Same correctness, much faster.
### Behavior
- Dry-run: counts refs, attempts = 0, writes nothing.
- auto_link=false in config: returns status='auto_link_disabled' +
no-op. Users who disabled auto-linking on put_page don't want
reconcile-links silently re-populating edges either.
- Missing code target: counted as `edgesTargetsMissing`, not thrown.
The ref exists in the guide, but the code page hasn't been synced
yet. Re-run after the next code sync to materialize.
- Progress reporter: `reconcile_links.scan` phase, one tick per
markdown page, with rolling summary `guides/foo (+N refs)` per tick.
### Tests (test/reconcile-links.test.ts, 6 cases)
- Extracts code refs and creates bidirectional edges (guide→code +
code→guide).
- Idempotent: second run inserts zero new edges.
- Dry-run reports counts without writing.
- Markdown page with no code refs is a no-op.
- Respects auto_link=false.
- Missing code target is counted, not thrown.
### CI result
2580 tests / 0 fail via `bun test --timeout=60000`, 432s wall time.
* feat: v0.20.0 Cathedral II Layer 12 — CHUNKER_VERSION 3→4 + SP-1 gate
Codex's second-pass review caught that bumping CHUNKER_VERSION alone is a
silent no-op on an unchanged repo: performSync short-circuits at `up_to_date`
before reaching importCodeFile's content_hash check. Layer 12 adds a
sources.chunker_version gate that forces a full re-walk when the version
mismatches, regardless of git HEAD equality.
- CHUNKER_VERSION 3 → 4 (src/core/chunkers/code.ts:99), folded into
content_hash via v0.19.0 Layer 5 wiring — any bump forces clean re-chunks.
- src/commands/sync.ts: readChunkerVersion/writeChunkerVersion helpers;
version-mismatch gate runs BEFORE the up_to_date early-return and forces
a full walk; writeChunkerVersion called after every last_commit anchor.
- test/chunker-version-gate.test.ts: 3 pinning tests (constant value,
import stability, v27 migration shape).
- test/chunkers/code.test.ts: update v0.19.0 CHUNKER_VERSION=3 assertion
to Cathedral II v0.20.0 CHUNKER_VERSION=4.
Full CI: 2333 pass / 250 skip / 0 fail / 6155 expect() / 408s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 13 (E2) — reindex-code + migration orchestrator
Ships the user-facing explicit-backfill path. v0.19.0 → v0.20.0 brains get
CHUNKER_VERSION 3→4 rolled over automatically via Layer 12's gate on next
sync. Users who want the benefits NOW (before their next sync) run
`gbrain reindex-code --yes`.
- New src/commands/reindex-code.ts. runReindexCode(engine, opts) walks code
pages from the DB in batches of 100 (Finding 4.4 OOM protection), reads
compiled_truth + frontmatter.file, re-runs importCodeFile. --dry-run
reports cost + token count without importing. --force bypasses
importCodeFile's content_hash early-return. --source filters to one
sources row. Pages without frontmatter.file fail cleanly (counted, not
thrown). runReindexCodeCli parses argv, wires the D1 cost-preview gate
(TTY prompt or ConfirmationRequired envelope for non-TTY/JSON), delegates.
- src/core/import-file.ts: importCodeFile gains opts.force flag. When
true, skips the content_hash === hash early-return so a paranoid full
reindex always re-chunks + re-embeds even when content hasn't changed.
- src/cli.ts: register 'reindex-code' case + CLI_ONLY entry.
- src/commands/migrations/v0_20_0.ts: orchestrator with 3 phases
(schema → backfill_prompt → verify). Phase B prints the two backfill
choices directly (automatic via sync vs immediate via reindex-code).
Follows v0.12.2/v0.18.1 idempotent-resumable pattern.
- src/commands/migrations/index.ts: registers v0_20_0 after v0_18_1.
- skills/migrations/v0.20.0.md: agent-facing post-upgrade instructions.
- test/reindex-code.test.ts: 5 cases (count, dry-run, walk+failures,
empty brain, batch pagination).
- test/migration-orchestrator-v0_20_0.test.ts: 5 cases (registry wiring,
feature-pitch content, __testing exports, dry-run skips, is-latest).
- test/apply-migrations.test.ts: extend skippedFuture pins with 0.20.0.
Full CI: 2343 pass / 250 skip / 0 fail / 6193 expect() / 426s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 10 partial (C1 + C2) — query --lang / --symbol-kind
Ships the cheap half of the C tier: language + symbol-kind filters on
hybrid search. The content_chunks.language and content_chunks.symbol_type
columns have existed since v0.19.0 Layer 5 (code chunker populates both);
Layer 10 exposes them as filter flags on the 'query' operation.
The expensive half (C3 --near-symbol, C4 code-callers, C5 code-callees) is
blocked on Layer 5 A1 edge extractor — those need the code_edges_chunk +
code_edges_symbol tables populated. They ship in a follow-up.
- src/core/pglite-engine.ts: searchKeyword / searchKeywordChunks /
searchVector all accept opts.language + opts.symbolKind. Filters added
via parameterized $N indices; unknown values return zero results
(no false positives).
- src/core/postgres-engine.ts: same three methods, same filters, threaded
through the postgres.js sql-fragment pattern. Honors SET LOCAL
statement_timeout discipline.
- src/core/search/hybrid.ts: threads opts.language + opts.symbolKind into
per-engine searchOpts so filters fire at SQL level (not post-filtered
in-memory).
- src/core/operations.ts: query op params gain lang + symbol_kind entries.
Handler maps them into hybridSearch opts.language / opts.symbolKind.
- src/cli.ts: updated --help CODE INDEXING section to list the new flags
+ reconcile-links + reindex-code commands.
- test/search-lang-symbol-kind.test.ts: 9 cases (no filter, lang-only,
symbolKind-only, combined AND, searchKeywordChunks variant, unknown
lang/kind return zero, operation schema check).
Full CI: 2352 pass / 250 skip / 0 fail / 6216 expect() / 432s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 6 (A3) — parent-scope + nested-chunk emission
Ships the chunk-granularity change codex called out in the second-pass
review. Before Cathedral II, `export class BrainEngine { m1() {} m2() {} }`
emitted ONE chunk for the whole class. Retrieval returned the entire
class body for a symbol-specific query like "how does searchKeyword
work" — the agent had to re-read the whole thing. A3 extends the
chunker to emit each method as its own chunk carrying
`parentSymbolPath: ['BrainEngine']`, with a `(in BrainEngine)` suffix in
the header so the embedding captures scope context. The class-level
parent chunk still ships (slim body: declaration line + member digest)
so class-level queries still hit something.
Recursive expansion: Ruby `module Admin { class UsersController { def
render } }` emits 3 chunks — Admin (parent=[]), UsersController
(parent=[Admin]), render (parent=[Admin, UsersController]).
- src/core/chunkers/code.ts:
- CodeChunkMetadata gains `parentSymbolPath?: string[]`.
- NESTED_EMIT_CONFIG map per language (TS, TSX, JS, Python, Ruby,
Rust impl blocks, Java class/interface/record). Maps parent types
(class_declaration / class_definition / module / impl_item) to
child types (method / method_definition / function_definition /
singleton_method / constructor_declaration).
- findNestableParent unwraps TS export_statement to reach the inner
class_declaration — the export wrapper was a classic gotcha.
- emitNestedScoped: recursive, builds full parent-chain path, pushes
a slim scope-header chunk for each parent level + leaf chunks for
methods. Handles module → class → method chains.
- buildChunk emits "(in ClassName.method)" header suffix when
parentSymbolPath is non-empty.
- mergeSmallSiblings now bails on any file that has parent-scoped
chunks. Methods emitted by A3 are intentionally small and
individually addressable; merging them would erase the scope
context Layer 6 just established.
- src/core/import-file.ts: importCodeFile passes parent_symbol_path
from chunker metadata into ChunkInput so it lands in content_chunks.
- src/core/pglite-engine.ts + src/core/postgres-engine.ts: upsertChunks
extends the column list to persist parent_symbol_path (TEXT[]),
doc_comment (TEXT), symbol_name_qualified (TEXT). All three existed
as schema columns from Layer 1 but the writers weren't plumbed yet.
ON CONFLICT DO UPDATE includes all three so re-imports refresh
metadata correctly.
- test/parent-scope.test.ts: 9 cases covering TypeScript class method
expansion, Python class, Ruby module+class, top-level function
passthrough, and round-trip through upsertChunks to verify text[]
persistence.
Full CI: 2361 pass / 250 skip / 0 fail / 6270 expect() / 439s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 5 (A1) — edge extractor + qualified names (8 langs)
The 10x leap. v0.19.0 shipped symbol-column filtering and could find "the
definition of X"; v0.20.0 Layer 5 captures who CALLS X. Walk the tree-sitter
tree during chunking, harvest call-site edges, persist to code_edges_symbol
with the callee's short-name as to_symbol_qualified. `getCallersOf("helper")`
now returns every call site, ready for Layer 7 two-pass retrieval to expand
into structural neighbors.
Scope: precision 80, recall 99. We don't try to resolve receiver types at
capture time (obj.method() stores "method", not "ObjClass.method"). That
receiver-type inference is a future optimization; the edges are captured,
which is the whole point. Cross-file resolution is also deferred — all
Layer 5 edges land unresolved in code_edges_symbol.
Per-language shipped: TypeScript, TSX, JavaScript, Python, Ruby, Go, Rust,
Java. ~85% of real brain code. Other languages chunk normally, edges just
empty.
- src/core/chunkers/qualified-names.ts (new): per-language delimiter
conventions. Ruby `Admin::UsersController#render` (instance) vs Python
`admin.users.UsersController.render` vs Rust `users::UsersController::render`.
Unknown languages dot-join as fallback (never drop).
- src/core/chunkers/edge-extractor.ts (new): iterative AST walk (no
recursion — tree-sitter trees can be deep, stack overflow risk on
generated code). Per-language CALL_CONFIG maps node types to callee
field names. extractCalleeName unwraps member_expression, scoped_identifier,
field_expression to reach the innermost identifier. findChunkForOffset
maps a byte offset to the innermost chunk for from_chunk_id resolution.
- src/core/chunkers/code.ts: CodeChunkMetadata gains
symbolNameQualified. buildChunk folds in qualified-name from parents +
name. New chunkCodeTextFull API returns (chunks, edges); chunkCodeText
stays as back-compat wrapper.
- src/core/import-file.ts: call chunkCodeTextFull, build ChunkInput list
with symbol_name_qualified, after upsertChunks run findChunkForOffset
to map call-site byte offsets to resolved chunk IDs, call
deleteCodeEdgesForChunks (codex SP-2 inbound invalidation) then
addCodeEdges. Edge persistence is best-effort — failure logs a warn
but does not fail the import.
- src/core/pglite-engine.ts + src/core/postgres-engine.ts: implement the
5 stub methods. addCodeEdges splits resolved vs unresolved by
to_chunk_id presence, inserts with ON CONFLICT DO NOTHING. getCallersOf
/ getCalleesOf UNION code_edges_chunk + code_edges_symbol (codex 1.3b:
no promotion, UNION-on-read forever). getEdgesByChunk honors direction
{in, out, both}. deleteCodeEdgesForChunks wipes both tables in both
directions (codex SP-2).
- test/qualified-names.test.ts: 9 cases (TS/Ruby instance method/Python/
Rust/Java/unknown-lang fallback).
- test/edge-extractor.test.ts: 11 cases (per-language call capture +
findChunkForOffset mapping + unknown-language empty-list).
- test/code-edges.test.ts: 7 cases (addCodeEdges insert + idempotency,
getCallersOf short-name match, resolved path, getEdgesByChunk
direction filters, deleteCodeEdgesForChunks both-direction wipe).
Full CI: 2391 pass / 250 skip / 0 fail / 6308 expect() / 449s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 10 rest (C4 + C5) — code-callers / code-callees CLI
Exposes Layer 5's call-graph edges as user-facing agent commands. The
existing code-def / code-refs pair answers "where is X defined?" and
"where is X referenced?"; Layer 10 rest adds "who CALLS X?" and "what
does X CALL?" — the structural questions v0.19.0 couldn't answer.
Conventions follow the code-def / code-refs precedent:
- Auto-JSON on non-TTY (gh-CLI convention)
- StructuredAgentError envelope on usage / runtime failure
- Exit 2 on UsageError, exit 1 on runtime
- --all-sources to widen beyond the anchor's source; default source-scoped
- src/commands/code-callers.ts (new) — wraps engine.getCallersOf.
- src/commands/code-callees.ts (new) — wraps engine.getCalleesOf.
- src/cli.ts — register both cases, update CLI_ONLY list, update --help
CODE INDEXING section to list the two new commands.
- test/code-callers-cli.test.ts — 2 cases (module exports, callable).
The --near-symbol / --walk-depth flags on query ship with Layer 7
(A2 two-pass retrieval) in a follow-up layer commit.
Full CI: 2393 pass / 250 skip / 0 fail / 6310 expect() / 448s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 7 (A2) — two-pass structural retrieval
The capstone of the retrieval-side upgrade. Layer 5 captured edges at
chunk time; Layer 7 uses them. Given a query like "how does
searchKeyword handle N+1", standard hybrid search returns the function
body; A2 expansion additionally surfaces:
- the 3 functions that call it (1-hop)
- the 2 functions it calls (1-hop)
- the anchor set's neighbors' neighbors (2-hop, optional)
All ranked together with 1/(1+hop) score decay. One walk. Code-aware
brain, not RAG-over-code.
Default OFF per codex F5. Activation:
- `--walk-depth N` (1 or 2) walks N hops from the anchor set.
- `--near-symbol <qualified-name>` adds chunks matching the symbol's
qualified name as extra anchors, enabling "expand around this
specific symbol" without a keyword query.
Caps (codex F5):
- depth capped at 2 (max blast radius).
- neighbor cap 50 per hop (high-fan-out protection: console.log has
100k callers and should not flood the result set).
- per-page dedup cap lifts from 2 → min(10, walkDepth × 5) when
walking — structural neighbors from the same class are the point.
- src/core/search/two-pass.ts (new): expandAnchors walks
code_edges_chunk + code_edges_symbol, hydrating unresolved edges by
matching symbol_name_qualified on lookup. hydrateChunks fetches
SearchResult rows for expanded chunk IDs.
- src/core/search/hybrid.ts: gate the two-pass step on opts.walkDepth
> 0 OR opts.nearSymbol set. Expansion runs before dedup so neighbors
survive; dedup cap widens when walking. Best-effort — expansion
failure falls back to base hybrid retrieval.
- src/core/operations.ts: query op params gain near_symbol (string) +
walk_depth (number). Handler threads both into hybridSearch opts.
- test/two-pass.test.ts: 8 cases (walkDepth 0/1/2/5-clamp, nearSymbol
anchoring, hydrateChunks round-trip, operation schema).
Full CI: 2401 pass / 250 skip / 0 fail / 6332 expect() / 449s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 11 (E1) — BrainBench code sub-category tests
Pins the retrieval-quality behaviors Layer 5 and Layer 6 added, so any
accidental regression surfaces on CI rather than silently eroding search
quality.
Sub-categories:
- call_graph_recall — importCodeFile captures calls edges
end-to-end; getCallersOf + getCalleesOf round-trip through real
edge extraction; re-import idempotency via codex SP-2 per-chunk
invalidation.
- parent_scope_coverage — nested methods persist parent_symbol_path
through the upsertChunks path; qualified symbol names resolve
correctly for nested declarations.
doc_comment_matching is deferred: the chunk-grain FTS trigger from
Layer 1b already weights doc_comment 'A', but chunker doc_comment
extraction (A4 full implementation) is a follow-up. The column exists,
the ranking is ready — waiting on extraction.
type_signature_retrieval deferred with C6 to v0.20.1 per plan.
- test/cathedral-ii-brainbench.test.ts (new): 6 cases covering the
two sub-categories against real PGLite + importCodeFile.
Full CI: 2407 pass / 250 skip / 0 fail / 6345 expect() / 467s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.20.0 Cathedral II Layer 14 — release (CHANGELOG + TODOS + version bump)
The capstone commit. Ships v0.20.0 — Code Cathedral II — with a full
release-summary in CHANGELOG.md covering the 13 layers that landed
(Layer 9 / Magika deferred to v0.20.1 per plan risk gate), migration
guidance under "To take advantage of v0.20.0", and itemized changes
grouped by layer with real numbers.
- VERSION: 0.19.0 → 0.20.0
- package.json: 0.19.0 → 0.20.0
- CHANGELOG.md: new [0.20.0] entry with release-summary (two-line
bold headline, lead paragraph, numbers-that-matter table with
before/after delta, per-language call-capture table, "what this
means for builders" closer), "To take advantage of v0.20.0"
section with verify commands + issue-reporting template, and the
full itemized changes section grouped by layer (1 / 2 / 3 / 4 /
5 / 6 / 7 / 8 / 10 / 11 / 12 / 13 / 9-deferred). Credits 2 codex
passes + eng + ceo reviews — 16 cross-model findings absorbed.
- TODOS.md: retire the 4 v0.19.0 follow-ups (all landed in v0.20.0
Layer 8 + Layer 10). Add 4 new Cathedral II follow-ups:
- B2 Magika (Layer 9 deferred)
- A4 full doc_comment extraction at chunk time
- C6 code-signature
- Cross-file edge resolution (Layer 5 precision upgrade)
Full CI: 2407 pass / 250 skip / 0 fail / 6345 expect() / 465s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(import-file): tolerate missing pages in doc↔impl linking
importCodeFile / importFromContent's E1 doc↔impl forward-link path was
calling tx.addLink() expecting the pre-v0.18 silent-no-op behavior on
missing pages. Master tightened addLink in postgres-engine.ts to throw
when either endpoint is missing — which is correct for explicit callers,
but the doc↔impl case is intentionally order-agnostic: a guide that
cites src/core/sync.ts can land before the code repo syncs (and vice
versa).
Result on CI: 21 E2E tests failed in test/e2e/mechanical.test.ts because
the fixture corpus has prose pages citing code paths the corpus doesn't
include, so each importFromContent threw "addLink failed: page X or Y
not found" and aborted before downstream assertions could run.
Fix: wrap each tx.addLink call (forward + reverse edge) in try/catch.
Match the existing pattern in src/commands/extract.ts:547 and
src/core/operations.ts:453,470 — both run try { addLink } catch { skip }
for exactly this reason. Missing edges land later via
`gbrain reconcile-links` (Layer 8 D3), which forward-scans every
markdown page and idempotently inserts the edges that resolve.
Comment refresh: the old comment ("addLink's inner SELECT naturally
drops edges to non-existent pages") was true pre-v0.18; updated to
reflect the current throwing behavior + the reconcile-links recovery
path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test/migrate): bump v8/v9 dedup-regression budget 5s → 90s
The v8 (links_dedup) + v9 (timeline_dedup_index) regression tests time
the FULL `runMigrations` chain from version 7 → LATEST_VERSION. Their
5s budget was sized when the chain ended at v8/v9 themselves and v8 +
the helper-btree-index O(n log n) work were the dominant cost.
Cathedral II added v27 (TSVECTOR column + GIN index + plpgsql trigger
compile + 2 new tables w/ FK CASCADE) and v28 (UPDATE backfill of
search_vector). On PGLite WASM in CI, the full v7 → v28 chain now
takes ~30-40s — schema-creation overhead, not v8/v9 dedup itself.
Locally the chain ran in 2.75s; CI's container cold-start hit 33s.
The original O(n²) regression v8 had would have taken MINUTES on 1000
duplicate rows (the original incident was multi-minute, not multi-tens-
of-seconds). Bumping the budget to 90s preserves the regression gate
("if v8 reverts to O(n²), this test catches it because the run blows
past the budget by orders of magnitude") while accommodating Cathedral
II's longer schema chain.
CI: 33758ms (v8 test) + 33343ms (v9 test) → both under 90s. The 5s
assertion was failing them, not the test runner timeout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(migrate): v29 enables RLS on code_edges_chunk + code_edges_symbol
The two new tables added by v27 (Cathedral II foundation) shipped without
RLS enabled. The E2E test "RLS is enabled on every public table (no
hardcoded allowlist)" caught this — Supabase exposes the public schema
via PostgREST so any table without RLS is anon-readable. Same security
gap as the v0.18.1 RLS hardening pass that v24 closed for the original
10 gbrain-managed tables.
Three CI failures fixed by this migration:
1. "RLS is enabled on every public table" — direct fail on the new
tables.
2. "GBRAIN:RLS_EXEMPT comment with valid reason exempts a non-RLS
public table" — was failing because doctor saw the unrelated
code_edges tables ALSO un-RLS'd, so the exempt-comment fixture
wasn't the only no-RLS table and doctor stayed in fail status.
3. "gbrain doctor exits 0 on healthy DB" — same cause, doctor was
emitting a fail check for the missing-RLS tables on every healthy
run.
Pattern: matches v24 exactly. DO $$ block with BYPASSRLS guard so a
non-bypass session can't accidentally lock itself out of its own data;
RAISE EXCEPTION on guard fail leaves schema_version at the prior value
so the next initSchema retries. Postgres-only via sqlFor — PGLite
doesn't enforce RLS the same way and the E2E gate runs only against
real Postgres.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test/e2e): v24 self-heals — assert version >= 24, not exactly 24
Pre-existing test bug surfaced when the E2E job ran on the Cathedral II
branch (and would have surfaced on master too once anyone ran the Tier 1
Mechanical job). The test rolls schema_version back to 23, runs init,
then asserts the version becomes exactly '24'. The intent was to prove
v24 didn't crash on missing budget_* tables — not to pin a specific
final version.
But initSchema runs every pending migration. With v25 + v26 (v0.19.0)
and now v27 + v28 + v29 (v0.21.0 Cathedral II) shipped, init advances
schema_version to LATEST_VERSION (currently 29) regardless of where it
started. The exact-match `'24'` assertion has been wrong since v25
landed; only the lack of an E2E run on master CI hid it.
Fix: parse the final version as int and assert `>= 24`. Same intent
(prove v24 ran cleanly + didn't roll back), forward-compatible with
future schema growth.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(README): add "Using gbrain with GStack" — 5 code-search magical moments
Discoverability hint for engineering agents running on GStack. Cathedral
II (v0.21.0) shipped call-graph edges + two-pass retrieval, but a
GStack agent running /investigate or /review won't reach for them
unless someone tells it gbrain has these surfaces. The new subsection
slots between Remote MCP and the Skills index, lists the 5 commands
verbatim (code-callers, code-callees, code-def, code-refs, query
--near-symbol --walk-depth), and links to the v0.21.0 CHANGELOG entry
for context.
Tradeoff acknowledged: gbrain README serves both standalone and
agent-platform users, so the GStack section is kept tight (16 lines)
and slotted with the other agent-integration paths rather than at the
top.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: regenerate llms.txt + llms-full.txt for v0.21.0
The build-llms regen-drift guard caught that the committed llms files
were stale after the README "Using gbrain with GStack" addition + the
v0.21.0 CHANGELOG promotion. Running `bun run build:llms` rebuilds both
deterministically from llms-config.ts so the test passes.
No source content changed in this commit — just the generator output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garry@ycombinator.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|