mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(v0.34 pre-w0): add code-retrieval eval harness for v0.34 ship gate
Captures pre-v0.34 retrieval quality on the gbrain self-corpus before any
code-intel work lands, so the v0.34 ship gate (precision@5 +10pp OR
answered_rate +15pp on >=15/30 questions) measures real improvement
rather than an after-the-fact retuned baseline.
* src/eval/code-retrieval/harness.ts -- pure-function metrics (precision@k,
recall@k, top-1 stability, gate evaluator) + EvalRunReport types stable
across schema_version 1
* src/eval/code-retrieval/questions.json -- 30 questions across callers /
callees / definition / references / blast_radius / execution_flow /
cluster_membership kinds, expected_files captured against current
gbrain layout
* src/eval/code-retrieval/strategies.ts -- BaselineStrategy (hybridSearch)
+ WithCodeIntelStrategy stub (post-W3 fills in code_blast/code_flow/etc.)
* src/commands/eval-code-retrieval.ts -- gbrain eval code-retrieval CLI
with --baseline / --with-code-intel / --compare subcommands
* test/code-retrieval-harness.test.ts -- 26 unit tests across metrics,
loader, gate logic; no engine dependency
PRE-V0.34 BASELINE WORKFLOW:
gbrain eval code-retrieval --baseline --save /tmp/baseline-1.json
(run 3x for noise floor)
V0.34 SHIP GATE (after W3 lands):
gbrain eval code-retrieval --with-code-intel --save /tmp/v034.json
gbrain eval code-retrieval --compare /tmp/baseline-1.json /tmp/v034.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.34 W0a): source-routing leak across query + two-pass
Codex outside-voice review on the v0.34 plan caught two load-bearing
sites where sourceId was advertised but never applied — multi-source
brains silently cross-contaminated structural retrieval:
* operations.ts ~323 — `query` op handler called hybridSearch without
threading ctx.sourceId. Multi-source agents querying with a
--source flag got cross-source results.
* two-pass.ts:81 (nearSymbol lookup) and two-pass.ts:131 (unresolved
edge resolution) — TwoPassOpts.sourceId was declared and threaded
through hybridSearch's expandAnchors call, but the actual SQL ignored
it. The walk window crossed source boundaries every time.
Fix:
* `query` op now reads ctx.sourceId AND accepts a new `source_id`
param (with '__all__' as the explicit force-cross-source escape
hatch). Per-call param wins over ctx context.
* two-pass.ts both lookups join through pages.source_id when
opts.sourceId is set; omitted opts.sourceId preserves the legacy
cross-source contract for callers who want it.
Regression test: test/e2e/source-routing.test.ts seeds two sources
with the same `parseMarkdown` symbol + a cross-source caller edge.
Pins:
- nearSymbol + sourceId='source-a' returns ONLY source-a chunks
- nearSymbol + sourceId='source-b' returns ONLY source-b chunks
- nearSymbol with no sourceId still crosses sources (contract preserved)
- walk_depth=1 unresolved-edge resolution stays in source-a
PGLite in-memory, no DATABASE_URL needed. The fix proves out under
realistic structural retrieval not just a contrived unit test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.34 W0b): flip CLI source-scoping default to truly source-scoped
Codex outside-voice review (finding #7) caught that the v0.20.0
docstring claim "by default we only match the caller's source_id"
contradicted the implementation in code-callers.ts:54 + code-callees.ts:43:
allSources: allSources || !sourceId
The right side made `allSources` TRUE whenever `--source` was omitted,
INVERTING the documented default. Multi-source brains silently cross-
contaminated structural retrieval; `gbrain code-callers parseMarkdown`
on a brain with two repos returned callers from both even though the
docstring promised per-source scoping.
Fix:
* New canonical helper `resolveDefaultSource(engine)` in sources-ops.ts.
Contract per eng review D7:
- exactly 1 source registered → return its id (single-source brains,
the 80% case; --source flag is unnecessary friction there)
- 2+ sources → throw SourceResolutionError(multiple_sources_ambiguous)
with the list of valid ids
- 0 sources → throw SourceResolutionError(no_sources)
* code-callers.ts + code-callees.ts now resolve to the default source
when both --source AND --all-sources are absent. To get the pre-v0.34
cross-source behavior, callers must pass --all-sources explicitly.
* Same hint text on both commands. Pinned by test/e2e/cli-source-scoping-pglite.test.ts.
IRON RULE regression R2: docstring promise now holds. Multi-source brain
running `gbrain code-callers <symbol>` without --source gets a clear
error listing valid source ids instead of silent cross-resolution.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.34 W0c): within-file two-pass symbol resolver + edges_backfilled_at watermark
Codex's outside-voice review caught that the v0.20.0 graph stores BARE
callee tokens (`render`, `find`, `execute`) — not qualified names. Pre-v0.34
recursive blast/flow would alias every same-named function across classes.
W0c is the foundation that fixes this: resolve `code_edges_symbol` rows by
matching `to_symbol_qualified` against the SAME-FILE chunks'
`symbol_name_qualified`, then write the outcome to `edge_metadata`.
This commit is the resolver primitive + schema. The cycle-phase wiring
that calls it on every quick-cycle tick lands in the next commit.
Schema (v51 migration `edges_backfilled_at_v0_34`):
* `content_chunks.edges_backfilled_at TIMESTAMPTZ` — resume watermark.
Chunks where the column is NULL OR older than EDGE_EXTRACTOR_VERSION_TS
get re-walked next tick. SIGINT/OOM/sleep mid-backfill loses at most
one batch.
* Indexes per D11 from eng review:
- `idx_code_edges_symbol_resolver(source_id, to_symbol_qualified)` —
composite for the resolver's per-source lookup.
- `idx_content_chunks_symbol_lookup(page_id, symbol_name_qualified)`
WHERE `symbol_name_qualified IS NOT NULL` — file-batched candidate
fetch; also reused by W4-5 cluster recompute.
- `idx_content_chunks_edges_backfill(edges_backfilled_at)` WHERE
`edges_backfilled_at IS NULL` — fast unresumed-row scan.
Module (`src/core/chunkers/symbol-resolver.ts`):
* `resolveSymbolEdgesIncremental(engine, {sourceId, maxChunks?, onProgress?})`
walks stale chunks in 200-chunk batches. For each chunk, loads its
unresolved edges, finds same-page candidates by symbol_name_qualified,
and writes outcome to `edge_metadata`:
- exactly 1 candidate → `{resolved_chunk_id: <id>}`
- 2+ candidates → `{ambiguous: true, candidates: [...]}`
- 0 candidates → unchanged (cross-file; two-pass.ts handles those)
Each batch bumps `edges_backfilled_at = NOW()` for the chunks.
* `readEdgeResolution(metadata)` — public helper for downstream code
(two-pass.ts, code_blast op, eval-capture) to consume the resolver's
output without parsing JSON directly. Returns a tagged union.
* `EDGE_EXTRACTOR_VERSION_TS` exported constant — bump when extractor
shape changes and the next cycle re-walks all chunks.
Tests (5 E2E in test/e2e/symbol-resolver-pglite.test.ts, all PGLite,
no DATABASE_URL): unambiguous match, ambiguous multi-match, no match,
watermark advance + idempotency, source isolation (no cross-source
candidate leak).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.34 W0c): wire resolve_symbol_edges as a new cycle phase
W0c's symbol resolver lands as a 12th cycle phase between extract and
patterns. The autopilot's quick-cycle path (60s watchdog interval per
D2 from eng review) now resolves stale chunks incrementally so agents
see resolved edges within ~60s of writes rather than waiting on the
slow full-walk path.
* CyclePhase + ALL_PHASES + NEEDS_LOCK_PHASES extended with
'resolve_symbol_edges'. Position: between extract (which emits new
bare-token edges from sync diffs) and patterns (which reads the
graph). Acquires the cycle lock because it writes edge_metadata.
* CycleReport.totals adds edges_resolved + edges_ambiguous so doctor
and autopilot summaries surface the numbers.
* runPhaseResolveSymbolEdges walks every registered source via
listSources() + resolveSymbolEdgesIncremental(). Per-call cap is
BATCH_SIZE*10 = 2000 chunks so a single watchdog tick stays bounded
even on a 100K-chunk brain. Subsequent ticks pick up the leftovers
via the edges_backfilled_at watermark.
* Test count bumped from 11 → 12 phases in cycle.serial.test.ts and
cycle.test.ts (both pinned by the regression guards). Existing 28
cycle tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.34 W3): MCP-expose code_callers / code_callees / code_def / code_refs
Pre-v0.34 these four code-intelligence commands lived in CLI_ONLY at
cli.ts:30 — agents calling gbrain via MCP couldn't reach them and fell
through to text search. This commit ships the agent-facing MCP surface
for v0.34 against the existing v0.20+ tree-sitter call graph; recursive
blast/flow and clusters land in subsequent commits.
* `code_callers(symbol, [limit, source_id, all_sources])` — wraps
engine.getCallersOf. Reverse view of the A1 call graph.
* `code_callees(symbol, [limit, source_id, all_sources])` — wraps
engine.getCalleesOf. Forward view.
* `code_def(symbol, [limit, lang])` — wraps findCodeDef. Returns
definition sites with file/line/snippet.
* `code_refs(symbol, [limit, lang])` — wraps findCodeRefs. Returns
every reference (comments, strings, imports, call sites).
All four are scope:'read', source-scoped by default via ctx.sourceId
(W0a contract). Per-call source_id param wins over ctx; pass '__all__'
or all_sources=true to force cross-source.
* operations-descriptions.ts: 4 new constants per the eng review D10
finding — every description carries an inline example response so
agents don't burn first-call context discovering shape. Resolver-grade
wording ("BEFORE editing any function, run code_callers...") routes
plan-mode questions straight to the right op.
* SEARCH_DESCRIPTION gains a cross-link clause pointing at the four new
ops so agents stop falling through to text search for code-symbol
questions.
Tests (11 E2E in test/e2e/code-intel-mcp-ops-pglite.test.ts):
- All four ops registered + scope:read + description pinned by constant
- All four ops have required symbol param
- code_callers / code_callees return the documented envelope shape
- Source scoping honors ctx.sourceId
- all_sources=true / source_id='__all__' force cross-source
- code_def returns the def-site snippet
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(v0.33.0): agent-readable migration doc for the code-intel foundation
skills/migrations/v0.33.0.md gives existing-user upgrade guidance for the
v0.33.0 foundation pre-release (this branch's accumulated work toward
v0.34 Cathedral III):
* Source-routing fix (Codex #2) — query / two-pass now honor sourceId
* CLI source-scoping default flipped (Codex #7) — gbrain code-callers
defaults to source-scoped, --all-sources is the explicit opt-out
* MCP exposure of code-callers / code-callees / code-def / code-refs
with resolver-grade descriptions agents auto-route to
* Within-file symbol resolver runs as a new `resolve_symbol_edges`
cycle phase between extract and patterns
* Schema migration v51: edges_backfilled_at watermark + 3 composite/
partial indexes for the resolver hot path
* Verification commands the agent runs after `gbrain upgrade`
Bumps the existing-user migration ladder so the auto-update agent
(SKILLPACK Section 17) discovers + runs the v0.33.0 migration steps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(v0.33.0): bump VERSION + package.json + CHANGELOG
v0.33.0 ships the v0.34 Cathedral III foundation: MCP exposure of
code_callers / code_callees / code_def / code_refs with resolver-grade
tool descriptions, plus the source-routing fix + within-file symbol
resolver + cycle-phase wiring that v0.34's recursive blast/flow and
Leiden clusters will build on.
Full release notes in CHANGELOG.md. Trio in lockstep:
VERSION: 0.33.0
package.json: 0.33.0
CHANGELOG.md: ## [0.33.0] - 2026-05-11
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(v0.33.0): update dream-cycle phase-order assertions for resolve_symbol_edges
E2E test pinned the canonical phase sequence as a regression guard. The
v0.33.0 resolve_symbol_edges phase (added between extract and patterns)
correctly bumps the count to 12 — caught by the canonical-order test on
fresh-Postgres run, fixed by adding the new phase to EXPECTED_PHASES
and bumping the version history comment.
Both cycle.serial.test.ts and cycle.test.ts were already updated in the
W0c cycle-phase commit (6f7dbe1d); this third pin lives in
test/e2e/dream-cycle-phase-order-pglite.test.ts and was missed.
Full E2E suite now: 550 passed / 0 failed / 81 files (real Postgres on
port 5435 via Docker pgvector/pgvector:pg16).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(v0.33.3.0): rebump from v0.33.2.0 → v0.33.3.0
User asked to ship as v0.33.3.0 instead of v0.33.2.0. Single sweep:
* VERSION + package.json bumped to 0.33.3.0
* CHANGELOG header + body rewritten to v0.33.3
* skills/migrations/v0.33.0.md → skills/migrations/v0.33.3.0.md
(migration files use the version they ship FROM; renaming aligns with
the v0.21.0.md / v0.31.0.md convention in CLAUDE.md)
* Schema migration name edges_backfilled_at_v0_33_2 →
edges_backfilled_at_v0_33_3 in src/core/migrate.ts (also bumps the
in-code identifier so the registry name matches the version)
* All v0.33.2 comment references swept to v0.33.3 in cycle.ts,
operations.ts, operations-descriptions.ts, eval.ts, symbol-resolver.ts
+ cycle test phase-history comments
* llms.txt + llms-full.txt regenerated
Trio verified:
VERSION: 0.33.3.0
package.json: 0.33.3.0
CHANGELOG.md: ## [0.33.3.0] - 2026-05-12
bun run verify clean; 90 v0.33.3-touched tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
99 lines
3.6 KiB
TypeScript
99 lines
3.6 KiB
TypeScript
/**
|
|
* v0.34 W0b — CLI source-scoping default regression test.
|
|
*
|
|
* Pre-v0.34 (Codex finding #7): code-callers.ts:54 + code-callees.ts:43
|
|
* set `allSources: allSources || !sourceId`. Default behavior with the
|
|
* `--source` flag omitted was GLOBAL cross-source resolution — the
|
|
* opposite of what the docstring promised. Multi-source brains silently
|
|
* cross-resolved `Admin::UsersController#render` between repos.
|
|
*
|
|
* IRON RULE regression R2: existing caller without `--source` flag now
|
|
* defaults to source-scoped behavior. To get the pre-v0.34 cross-source
|
|
* default, the caller must pass `--all-sources` explicitly.
|
|
*
|
|
* This test bypasses CLI argv parsing and drives the resolveDefaultSource
|
|
* helper + the engine.getCallersOf path directly. The CLI handlers thread
|
|
* resolveDefaultSource() output to engine.getCallersOf, so this E2E pins
|
|
* the same contract end-to-end.
|
|
*
|
|
* PGLite in-memory, no DATABASE_URL.
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
|
import { resolveDefaultSource, SourceResolutionError } from '../../src/core/sources-ops.ts';
|
|
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
describe('v0.34 W0b — resolveDefaultSource resolution rule', () => {
|
|
test('single-source brain: returns the only source id', async () => {
|
|
await resetPgliteState(engine);
|
|
// After reset, the default 'default' source from schema bootstrap is
|
|
// the only one present. resolveDefaultSource returns it.
|
|
const id = await resolveDefaultSource(engine);
|
|
expect(id).toBe('default');
|
|
});
|
|
|
|
test('multi-source brain: throws with the list of valid ids', async () => {
|
|
await resetPgliteState(engine);
|
|
await engine.executeRaw(
|
|
`INSERT INTO sources (id, name, local_path, config, created_at)
|
|
VALUES ('repo-a', 'repo-a', '/fake/a', '{}'::jsonb, NOW())
|
|
ON CONFLICT (id) DO NOTHING`,
|
|
[],
|
|
);
|
|
await engine.executeRaw(
|
|
`INSERT INTO sources (id, name, local_path, config, created_at)
|
|
VALUES ('repo-b', 'repo-b', '/fake/b', '{}'::jsonb, NOW())
|
|
ON CONFLICT (id) DO NOTHING`,
|
|
[],
|
|
);
|
|
|
|
let caught: unknown = null;
|
|
try {
|
|
await resolveDefaultSource(engine);
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
expect(caught).toBeInstanceOf(SourceResolutionError);
|
|
if (caught instanceof SourceResolutionError) {
|
|
expect(caught.code).toBe('multiple_sources_ambiguous');
|
|
expect(caught.availableSources).toContain('default');
|
|
expect(caught.availableSources).toContain('repo-a');
|
|
expect(caught.availableSources).toContain('repo-b');
|
|
expect(caught.message).toContain('--source');
|
|
}
|
|
});
|
|
|
|
test('zero-sources brain: throws no_sources code', async () => {
|
|
await resetPgliteState(engine);
|
|
// resetPgliteState preserves the 'default' source from the schema
|
|
// bootstrap. Delete it to simulate a brain with no registered sources.
|
|
// FK from pages.source_id is ON DELETE CASCADE; resetPgliteState
|
|
// truncates pages first so this delete is safe.
|
|
await engine.executeRaw(`DELETE FROM sources WHERE id = 'default'`, []);
|
|
|
|
let caught: unknown = null;
|
|
try {
|
|
await resolveDefaultSource(engine);
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
expect(caught).toBeInstanceOf(SourceResolutionError);
|
|
if (caught instanceof SourceResolutionError) {
|
|
expect(caught.code).toBe('no_sources');
|
|
}
|
|
});
|
|
});
|