mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 19:49:14 +00:00
7d78bc593d136edbb93f41d39b2ffe29bc7e9147
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
dd1cc121d8 |
v0.35.3.1 feat(eval): temporal-aware contradiction probe + verdict enum (#1052)
* rfc: temporal axis for contradiction probe Field report on residual HIGH findings from gbrain eval suspected-contradictions and proposal for a 4-phase fix (Phase 1 = judge prompt + verdict enum is the recommended starting point). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): pass effective_date to judge prompt; bump PROMPT_VERSION Lane A1 of the temporal-contradiction-probe wave. Threads page-level effective_date through the search projection into the contradiction judge so the LLM can reason about supersession instead of treating every dated pair as a contradiction. Changes: - SearchResult interface adds optional effective_date + effective_date_source fields; rowToSearchResult populates them from the row data with date-only YYYY-MM-DD normalization (handles both postgres.js Date and PGLite string). - 8 SELECT projection sites (3 in postgres-engine, 5 in pglite-engine) now carry p.effective_date + p.effective_date_source through their inner CTEs and outer SELECTs so search results expose the field on both engines. - PairMember (eval-contradictions/types.ts) gets the two fields as required (string | null) so the type forces every constructor to think about temporal anchoring. Runner's searchResultToMember + takeToMember handle the normalization; takes inherit the chunk's page-level date. - buildJudgePrompt emits `Statement A (from: YYYY-MM-DD)` when effective_date is non-null, else `(date unknown)`. Prompt instructions explain the tag so the model knows what to do with it. - PROMPT_VERSION bumps '1' → '2'. Cache-key tuple shape unchanged; old rows miss naturally on first run against the new prompt. Test fixtures in 5 files updated to include the new required fields. All 205 eval-contradictions unit tests + 101 search-related tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): replace contradicts:boolean with verdict:enum (6 members) Lane A2 of the temporal-contradiction-probe wave. Expands the judge's classification vocabulary from a binary contradicts:bool to a six-member verdict enum so the probe can distinguish "this changed" from "this is wrong". Verdict taxonomy: no_contradiction — drop from findings contradiction — genuine conflict at same point in time temporal_supersession — newer claim updates/replaces older; not an error temporal_regression — metric/status went backwards over time (signal) temporal_evolution — legitimate change, neither supersession nor regression negation_artifact — judge misread an explicit negation Changes: - types.ts: Verdict union (6 members); Severity gains 'info'; ResolutionKind extended with temporal_supersede, flag_for_review, log_timeline_change; JudgeVerdict.contradicts → verdict; ContradictionFinding now carries verdict; ProbeReport adds queries_with_any_finding + verdict_breakdown (additive). - judge.ts: parseResolutionKind + parseVerdict guards; normalizeVerdict reads the new field and applies the C1 confidence floor only to verdict='contradiction' (the new verdicts are informational classifications, no floor). Prompt rubric rewritten to ask for verdict + extended severity scale. - severity-classify.ts: 'info' joins the rank with value 0; defaultSeverityForVerdict maps each verdict to its baseline severity (D7 — supersession=info, regression=high, etc.). parseSeverity gains a fallback param so consumers can override 'low' default. - auto-supersession.ts: classifyResolution + renderResolutionCommand handle the three new resolution kinds. Probe still NEVER auto-mutates — the new kinds render paste-ready commands or informational lines. - cache.ts: isJudgeVerdict shape check matches the new verdict field; old v1 rows fail the guard and treat as misses. - runner.ts: emit predicate at cache-hit and judge-success branches changes from `verdict.contradicts` to `verdict.verdict !== 'no_contradiction'`. Without this, the new verdicts vanish from the report. Added per-verdict tally + queriesWithAnyFinding alongside the strict queriesWithContradiction. - trends.ts: latest run verdict breakdown surfaces in the trend chart. Test fixtures updated across 8 test files. All 210 eval-contradictions unit tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): relax date-filter rule 3 when both sides dated Lane B of the temporal-contradiction-probe wave. The v1 date pre-filter skipped pairs whose chunk-text-extracted dates differed by >30 days as a cost-saving heuristic. That heuristic silently killed exactly the cases the new verdict taxonomy exists to surface — role transitions across years (e.g. a 2017 historical record vs. a 2025 current state), MRR claims years apart, status changes recorded over time. Lane A1+A2 made temporal supersession explicit and cheap to classify. The filter no longer needs to skip these pairs; the judge can label them. Changes: - date-filter.ts: shouldSkipForDateMismatch accepts optional effectiveDateA and effectiveDateB. When BOTH are non-null, returns skip=false with the new 'both_have_effective_date' reason — the judge will see the dates via the (from: YYYY-MM-DD) prompt tag from Lane A1. Other rules (same-paragraph dual-date override, missing-date fallback) preserved verbatim and still run first. - runner.ts: threads pair.{a,b}.effective_date into the date-filter call. Pairs that previously vanished into the skip bucket now reach the judge. Tests (R1 IRON RULE regression suite, 6 new cases): - both sides effective_date → not skipped - both sides effective_date overrides >30d chunk-text rule - rule 1 (same-paragraph dual-date) still wins over effective_date relaxation - rule 2 (missing chunk dates) still applies when effective_date partially present - undefined effective_dates fall through to v1 behavior (back-compat) - empty-string effective_date treated as missing (only real dates enable the relaxation) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): cost-estimate prompt + --budget-usd + Haiku routing Lane C of the temporal-contradiction-probe wave. Three layers of cost guardrail, all stacked: (a) cost-estimate prompt at probe-run-time. Before the runner spends any tokens after a PROMPT_VERSION change, eval-suspected-contradictions reads the most recent persisted prompt_version from eval_contradictions_runs and compares. When they differ: - TTY: prints an upper-bound estimate + Ctrl-C window (default 10s, override via GBRAIN_PROBE_PROMPT_GRACE_SECONDS). - non-TTY: prints the estimate + auto-proceeds (autopilot path). - --yes override or GBRAIN_NO_PROBE_PROMPT=1: skip entirely. Mirrors the v0.32.7 runPostUpgradeReembedPrompt pattern. (b) --budget-usd N hard cap (pre-existing; PreFlightBudgetError surfaces when the estimate alone exceeds the cap, and CostTracker halts the run mid-flight when cumulative cost exceeds it). Documented in the help text alongside (a). (c) Judge model now routes through resolveModel() with configKey 'models.eval.contradictions_judge', tier 'utility' (Haiku-class default), and env var GBRAIN_CONTRADICTIONS_JUDGE_MODEL. The legacy --judge CLI flag still wins as the highest-precedence override. Doctor's model touchpoint registry (src/commands/models.ts:50) carries the new key so `gbrain models` and `gbrain models doctor` surface it. Also in this lane: - CLI: --severity accepts 'info' (the new Severity member from Lane A2). - CLI: --severity output shows [verdict] tag alongside slug pairs so operators distinguish genuine contradictions from temporal classifications. - Human summary: prints the new queries_with_any_finding metric and the per-verdict breakdown table. - Help text: explains the cost-prompt + budget-cap + model-routing interactions in one paragraph. New tests (9 cases on the cost-prompt helper): - --yes override skips - GBRAIN_NO_PROBE_PROMPT=1 skips - prompt_version unchanged → skips - non-TTY auto-proceeds with stderr note - TTY proceeds after grace - TTY aborts on Ctrl-C - fresh brain (no prior runs) fires the prompt - GBRAIN_PROBE_PROMPT_GRACE_SECONDS override honored - estimate banner contains query count + judge model + dollar amount All 225 eval-contradictions tests + 25 model-config tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(eval): R4/R5/R6 IRON-RULE regressions for the verdict-enum wave Lane D of the temporal-contradiction-probe wave. The Lanes A1/A2/B/C lanes landed the behavior; this lane pins the regressions that protect the wave against future drift. R4 (runner emit predicate): five new tests, one per non-no_contradiction verdict, prove the runner.ts emit rule surfaces each one as a finding with the correct verdict tag, and that: - queries_with_contradiction (Wilson-CI denominator) ONLY counts verdict ='contradiction' — the strict metric is preserved - queries_with_any_finding counts every non-no_contradiction verdict - verdict_breakdown tallies correctly Plus one negative case: verdict='no_contradiction' produces zero findings. Without R4, a future runner refactor could collapse the new verdicts back to /dev/null and the report would silently shrink. R5 (cache key shape): direct shape assertion on buildCacheKey output. The key tuple is exactly 5 fields (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy). Adding a 6th field would silently break every operator's brain (no migration path). R6 (contradiction severity unchanged): four tests on normalizeVerdict pin the legacy semantics — judge-supplied severity wins (whether 'high' or 'low'), and on garbage severity input the fallback is 'medium' (per defaultSeverityForVerdict('contradiction')) NOT 'low'. The contradiction verdict's severity must never default to 'low', which would silently mask genuine conflicts as cosmetic naming issues. The temporal_regression case is included for parity (garbage → 'high' since regressions are real investor red flags). 236 eval-contradictions tests pass (211 + 6 R4 + 1 R5 + 4 R6 + 9 cost-prompt from Lane C). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ci): privacy lint for docs/proposals/*.md Captures the residual TODO from the temporal-contradiction-probe wave's plan: prevent the bug class where an RFC lands in docs/proposals/ with PII that should never appear in a public technical artifact. The original RFC had to be scrubbed at force-push time (Step 0); this lint catches the same patterns at CI time so the next one can't slip through. Sibling to scripts/check-privacy.sh: - check-privacy.sh: bans the literal "Wintermute" repo-wide. - check-proposal-pii.sh: focuses on docs/proposals/*.md and the OTHER PII classes — personal-relationship vocabulary, private repo refs. Design contract: the denylist names PATTERNS, not real people. Naming specific real names (deceased relatives, therapist first names, dealflow contacts) inside this script would leak PII into the repo just by appearing here. The structural patterns below catch the SURROUNDING vocabulary that always accompanies such content in personal RFC prose. Trade-off: a future RFC that names a real person without any contextual markers won't be caught — accepted as residual risk handled by human review. Patterns flagged in docs/proposals/*.md: - garrytan/brain (private repo reference) - trial separation, permanent separation - couples session, couples therapist - divorce attorney(s) - grandmother's funeral, aunt's funeral - wintermute (also caught by check-privacy.sh; listed here for proposal-scoped clarity) Bare common words (separation, funeral) are NOT banned — only the combined personal-context phrases. "Separation of concerns" and other software vocabulary survives. Wired into: - `bun run verify` (gates every push) - `bun run check:all` - `bun run check:proposal-pii` (standalone) Tests: 15 cases in test/scripts/check-proposal-pii.test.ts. - Each pattern flagged when present, plus exit-code + stderr signal. - Two negative cases (separation-of-concerns, funeral metaphor) prove the lint doesn't false-positive on legitimate software prose. - No-proposals-dir → exit 0 (not a failure). - Multi-hit case proves all patterns surface together with a summary count. - The two test fixtures that name "Wintermute" / "WINTERMUTE" as sentinel literals are allowlisted in check-test-real-names.sh per the same meta-rule-enforcement exception as check-privacy.sh itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(privacy): allowlist new privacy-guard files in check-privacy.sh check-privacy.sh bans the literal Wintermute repo-wide. The two new files from the v0.34 privacy lint (scripts/check-proposal-pii.sh and its test) necessarily name the token to do their job. Same meta-rule-enforcement exception as scripts/check-privacy.sh itself, scripts/check-test-real-names.sh, test/recency-decay.test.ts, and the existing entries — describing what the rule forbids requires naming it. Without this allowlist, `bun run verify` fails on check:privacy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.35.1.0) Temporal-contradiction-probe wave — Phase 1 of the RFC at docs/proposals/temporal-contradiction-probe.md. Headline: the contradiction probe now classifies pairs into a 6-member verdict enum (no_contradiction, contradiction, temporal_supersession, temporal_regression, temporal_evolution, negation_artifact) and sees the page-level effective_date for each chunk via a (from: YYYY-MM-DD) tag in the prompt. The pre-judge date filter no longer skips dated wide-gap pairs, so the role-transition class (e.g. a 2017 historical record vs. a 2025 current state) reaches the judge and gets classified as temporal_supersession instead of vanishing into the skip bucket. PROMPT_VERSION bumped 1 → 2 (cache fully invalidated). Three-layer cost guardrail: TTY-only cost-estimate prompt with Ctrl-C window, --budget-usd hard cap, Haiku-tier routing via new models.eval.contradictions_judge config key. Also adds a CI privacy lint (scripts/check-proposal-pii.sh) wired into bun run verify that catches PII patterns in docs/proposals/*.md so future RFCs can't ship with personal-context vocabulary the way this wave's source RFC did at draft time. Phases 2-4 deferred to follow-up RFCs per the plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
cdfc210e52 |
v0.34.0.0 feat: Cathedral III — recursive code intelligence + Leiden clusters + eval gate (#994)
* feat(v0.34 pre-w0): add code-retrieval eval harness for v0.34 ship gate Captures pre-v0.34 retrieval quality on the gbrain self-corpus before any code-intel work lands, so the v0.34 ship gate (precision@5 +10pp OR answered_rate +15pp on >=15/30 questions) measures real improvement rather than an after-the-fact retuned baseline. * src/eval/code-retrieval/harness.ts -- pure-function metrics (precision@k, recall@k, top-1 stability, gate evaluator) + EvalRunReport types stable across schema_version 1 * src/eval/code-retrieval/questions.json -- 30 questions across callers / callees / definition / references / blast_radius / execution_flow / cluster_membership kinds, expected_files captured against current gbrain layout * src/eval/code-retrieval/strategies.ts -- BaselineStrategy (hybridSearch) + WithCodeIntelStrategy stub (post-W3 fills in code_blast/code_flow/etc.) * src/commands/eval-code-retrieval.ts -- gbrain eval code-retrieval CLI with --baseline / --with-code-intel / --compare subcommands * test/code-retrieval-harness.test.ts -- 26 unit tests across metrics, loader, gate logic; no engine dependency PRE-V0.34 BASELINE WORKFLOW: gbrain eval code-retrieval --baseline --save /tmp/baseline-1.json (run 3x for noise floor) V0.34 SHIP GATE (after W3 lands): gbrain eval code-retrieval --with-code-intel --save /tmp/v034.json gbrain eval code-retrieval --compare /tmp/baseline-1.json /tmp/v034.json Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.34 W0a): source-routing leak across query + two-pass Codex outside-voice review on the v0.34 plan caught two load-bearing sites where sourceId was advertised but never applied — multi-source brains silently cross-contaminated structural retrieval: * operations.ts ~323 — `query` op handler called hybridSearch without threading ctx.sourceId. Multi-source agents querying with a --source flag got cross-source results. * two-pass.ts:81 (nearSymbol lookup) and two-pass.ts:131 (unresolved edge resolution) — TwoPassOpts.sourceId was declared and threaded through hybridSearch's expandAnchors call, but the actual SQL ignored it. The walk window crossed source boundaries every time. Fix: * `query` op now reads ctx.sourceId AND accepts a new `source_id` param (with '__all__' as the explicit force-cross-source escape hatch). Per-call param wins over ctx context. * two-pass.ts both lookups join through pages.source_id when opts.sourceId is set; omitted opts.sourceId preserves the legacy cross-source contract for callers who want it. Regression test: test/e2e/source-routing.test.ts seeds two sources with the same `parseMarkdown` symbol + a cross-source caller edge. Pins: - nearSymbol + sourceId='source-a' returns ONLY source-a chunks - nearSymbol + sourceId='source-b' returns ONLY source-b chunks - nearSymbol with no sourceId still crosses sources (contract preserved) - walk_depth=1 unresolved-edge resolution stays in source-a PGLite in-memory, no DATABASE_URL needed. The fix proves out under realistic structural retrieval not just a contrived unit test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.34 W0b): flip CLI source-scoping default to truly source-scoped Codex outside-voice review (finding #7) caught that the v0.20.0 docstring claim "by default we only match the caller's source_id" contradicted the implementation in code-callers.ts:54 + code-callees.ts:43: allSources: allSources || !sourceId The right side made `allSources` TRUE whenever `--source` was omitted, INVERTING the documented default. Multi-source brains silently cross- contaminated structural retrieval; `gbrain code-callers parseMarkdown` on a brain with two repos returned callers from both even though the docstring promised per-source scoping. Fix: * New canonical helper `resolveDefaultSource(engine)` in sources-ops.ts. Contract per eng review D7: - exactly 1 source registered → return its id (single-source brains, the 80% case; --source flag is unnecessary friction there) - 2+ sources → throw SourceResolutionError(multiple_sources_ambiguous) with the list of valid ids - 0 sources → throw SourceResolutionError(no_sources) * code-callers.ts + code-callees.ts now resolve to the default source when both --source AND --all-sources are absent. To get the pre-v0.34 cross-source behavior, callers must pass --all-sources explicitly. * Same hint text on both commands. Pinned by test/e2e/cli-source-scoping-pglite.test.ts. IRON RULE regression R2: docstring promise now holds. Multi-source brain running `gbrain code-callers <symbol>` without --source gets a clear error listing valid source ids instead of silent cross-resolution. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.34 W0c): within-file two-pass symbol resolver + edges_backfilled_at watermark Codex's outside-voice review caught that the v0.20.0 graph stores BARE callee tokens (`render`, `find`, `execute`) — not qualified names. Pre-v0.34 recursive blast/flow would alias every same-named function across classes. W0c is the foundation that fixes this: resolve `code_edges_symbol` rows by matching `to_symbol_qualified` against the SAME-FILE chunks' `symbol_name_qualified`, then write the outcome to `edge_metadata`. This commit is the resolver primitive + schema. The cycle-phase wiring that calls it on every quick-cycle tick lands in the next commit. Schema (v51 migration `edges_backfilled_at_v0_34`): * `content_chunks.edges_backfilled_at TIMESTAMPTZ` — resume watermark. Chunks where the column is NULL OR older than EDGE_EXTRACTOR_VERSION_TS get re-walked next tick. SIGINT/OOM/sleep mid-backfill loses at most one batch. * Indexes per D11 from eng review: - `idx_code_edges_symbol_resolver(source_id, to_symbol_qualified)` — composite for the resolver's per-source lookup. - `idx_content_chunks_symbol_lookup(page_id, symbol_name_qualified)` WHERE `symbol_name_qualified IS NOT NULL` — file-batched candidate fetch; also reused by W4-5 cluster recompute. - `idx_content_chunks_edges_backfill(edges_backfilled_at)` WHERE `edges_backfilled_at IS NULL` — fast unresumed-row scan. Module (`src/core/chunkers/symbol-resolver.ts`): * `resolveSymbolEdgesIncremental(engine, {sourceId, maxChunks?, onProgress?})` walks stale chunks in 200-chunk batches. For each chunk, loads its unresolved edges, finds same-page candidates by symbol_name_qualified, and writes outcome to `edge_metadata`: - exactly 1 candidate → `{resolved_chunk_id: <id>}` - 2+ candidates → `{ambiguous: true, candidates: [...]}` - 0 candidates → unchanged (cross-file; two-pass.ts handles those) Each batch bumps `edges_backfilled_at = NOW()` for the chunks. * `readEdgeResolution(metadata)` — public helper for downstream code (two-pass.ts, code_blast op, eval-capture) to consume the resolver's output without parsing JSON directly. Returns a tagged union. * `EDGE_EXTRACTOR_VERSION_TS` exported constant — bump when extractor shape changes and the next cycle re-walks all chunks. Tests (5 E2E in test/e2e/symbol-resolver-pglite.test.ts, all PGLite, no DATABASE_URL): unambiguous match, ambiguous multi-match, no match, watermark advance + idempotency, source isolation (no cross-source candidate leak). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.34 W0c): wire resolve_symbol_edges as a new cycle phase W0c's symbol resolver lands as a 12th cycle phase between extract and patterns. The autopilot's quick-cycle path (60s watchdog interval per D2 from eng review) now resolves stale chunks incrementally so agents see resolved edges within ~60s of writes rather than waiting on the slow full-walk path. * CyclePhase + ALL_PHASES + NEEDS_LOCK_PHASES extended with 'resolve_symbol_edges'. Position: between extract (which emits new bare-token edges from sync diffs) and patterns (which reads the graph). Acquires the cycle lock because it writes edge_metadata. * CycleReport.totals adds edges_resolved + edges_ambiguous so doctor and autopilot summaries surface the numbers. * runPhaseResolveSymbolEdges walks every registered source via listSources() + resolveSymbolEdgesIncremental(). Per-call cap is BATCH_SIZE*10 = 2000 chunks so a single watchdog tick stays bounded even on a 100K-chunk brain. Subsequent ticks pick up the leftovers via the edges_backfilled_at watermark. * Test count bumped from 11 → 12 phases in cycle.serial.test.ts and cycle.test.ts (both pinned by the regression guards). Existing 28 cycle tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.34 W3): MCP-expose code_callers / code_callees / code_def / code_refs Pre-v0.34 these four code-intelligence commands lived in CLI_ONLY at cli.ts:30 — agents calling gbrain via MCP couldn't reach them and fell through to text search. This commit ships the agent-facing MCP surface for v0.34 against the existing v0.20+ tree-sitter call graph; recursive blast/flow and clusters land in subsequent commits. * `code_callers(symbol, [limit, source_id, all_sources])` — wraps engine.getCallersOf. Reverse view of the A1 call graph. * `code_callees(symbol, [limit, source_id, all_sources])` — wraps engine.getCalleesOf. Forward view. * `code_def(symbol, [limit, lang])` — wraps findCodeDef. Returns definition sites with file/line/snippet. * `code_refs(symbol, [limit, lang])` — wraps findCodeRefs. Returns every reference (comments, strings, imports, call sites). All four are scope:'read', source-scoped by default via ctx.sourceId (W0a contract). Per-call source_id param wins over ctx; pass '__all__' or all_sources=true to force cross-source. * operations-descriptions.ts: 4 new constants per the eng review D10 finding — every description carries an inline example response so agents don't burn first-call context discovering shape. Resolver-grade wording ("BEFORE editing any function, run code_callers...") routes plan-mode questions straight to the right op. * SEARCH_DESCRIPTION gains a cross-link clause pointing at the four new ops so agents stop falling through to text search for code-symbol questions. Tests (11 E2E in test/e2e/code-intel-mcp-ops-pglite.test.ts): - All four ops registered + scope:read + description pinned by constant - All four ops have required symbol param - code_callers / code_callees return the documented envelope shape - Source scoping honors ctx.sourceId - all_sources=true / source_id='__all__' force cross-source - code_def returns the def-site snippet Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v0.33.0): agent-readable migration doc for the code-intel foundation skills/migrations/v0.33.0.md gives existing-user upgrade guidance for the v0.33.0 foundation pre-release (this branch's accumulated work toward v0.34 Cathedral III): * Source-routing fix (Codex #2) — query / two-pass now honor sourceId * CLI source-scoping default flipped (Codex #7) — gbrain code-callers defaults to source-scoped, --all-sources is the explicit opt-out * MCP exposure of code-callers / code-callees / code-def / code-refs with resolver-grade descriptions agents auto-route to * Within-file symbol resolver runs as a new `resolve_symbol_edges` cycle phase between extract and patterns * Schema migration v51: edges_backfilled_at watermark + 3 composite/ partial indexes for the resolver hot path * Verification commands the agent runs after `gbrain upgrade` Bumps the existing-user migration ladder so the auto-update agent (SKILLPACK Section 17) discovers + runs the v0.33.0 migration steps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(v0.33.0): bump VERSION + package.json + CHANGELOG v0.33.0 ships the v0.34 Cathedral III foundation: MCP exposure of code_callers / code_callees / code_def / code_refs with resolver-grade tool descriptions, plus the source-routing fix + within-file symbol resolver + cycle-phase wiring that v0.34's recursive blast/flow and Leiden clusters will build on. Full release notes in CHANGELOG.md. Trio in lockstep: VERSION: 0.33.0 package.json: 0.33.0 CHANGELOG.md: ## [0.33.0] - 2026-05-11 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.33.0): update dream-cycle phase-order assertions for resolve_symbol_edges E2E test pinned the canonical phase sequence as a regression guard. The v0.33.0 resolve_symbol_edges phase (added between extract and patterns) correctly bumps the count to 12 — caught by the canonical-order test on fresh-Postgres run, fixed by adding the new phase to EXPECTED_PHASES and bumping the version history comment. Both cycle.serial.test.ts and cycle.test.ts were already updated in the W0c cycle-phase commit ( |
||
|
|
9a5606af6d |
v0.32.6 feat: brain-consistency probe + doctor + MCP + dream-cycle wire-up (#901)
* feat(eval-contradictions): types + pure helpers for v0.33.0 probe Foundational module for the contradiction measurement probe (v0.33.0 plan). Pure, hermetic, no engine or LLM dependencies. Sets the wire contract for the rest of the implementation. - types.ts: schema_version + PROMPT_VERSION + TRUNCATION_POLICY constants, ProbeReport + ContradictionPair + JudgeVerdict + cache/run row shapes. - calibration.ts: Wilson 95% CI on the headline percentage with exact clamping at p=0 and p=1 (floating-point overshoot regression guard); small_sample_note when n<30. - judge-errors.ts: first-class typed error collector (Codex fix — bias guard for the silent-skip-on-throw decision); classifier maps to parse_fail/refusal/timeout/http_5xx/unknown. - severity-classify.ts: parseSeverity defaults to 'low' on garbage input; bucketBySeverity + buildHotPages (descending rank + tie-break by severity). - date-filter.ts: three-rule A1 pre-filter — same-paragraph-dual-date beats the separation rule (flip-flop case); missing dates falls through to the judge; only "both explicit AND >30d apart" actually skips. 51 hermetic tests across the four pure modules; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval-contradictions): schema migrations + engine methods (v0.33.0) Adds the persistent surface the contradiction probe needs: two new tables plus five BrainEngine methods, mirrored cleanly across PGLite + Postgres. Migrations v51 + v52 (idempotent on both engines): - eval_contradictions_cache: composite PK on (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy) per Codex outside-voice fix; verdict JSONB; expires_at-driven TTL. - eval_contradictions_runs: one row per probe run; Wilson CI bounds, judge-error totals, source-tier breakdown, full report_json. Engine methods (interface + 2 impls each): - listActiveTakesForPages(pageIds, opts): P1 batched per-page fetch. Single WHERE page_id = ANY($1) AND active = true; replaces the O(K) loop the probe would otherwise pay per query. - writeContradictionsRun(row): M5 time-series insert; idempotent on run_id via ON CONFLICT DO NOTHING. - loadContradictionsTrend(days): M5 history read, newest first. - getContradictionCacheEntry(key): P2 cache lookup; 5-component key includes prompt_version + truncation_policy. - putContradictionCacheEntry(opts): cache upsert with TTL refresh. - sweepContradictionCache(): periodic expired-row purge. JSONB writes use sql.json() on Postgres (matches existing eval_takes_quality + raw_data patterns; not the literal-template-tag pattern banned by scripts/check-jsonb-pattern.sh). PGLite uses $N::jsonb positional binds. 17 hermetic tests on PGLite cover P1 (4 cases: empty, grouped, supersede- excludes, holder-allow-list), M5 (5 cases: write+read, idempotent run_id, newest-first, days-window, JSONB round-trip), P2 (6 cases: miss, put-get, prompt-version differs, truncation differs, upsert refreshes, sweep deletes expired). Existing 109 migrate + bootstrap tests still green. Schema mirror in pglite-schema.ts; source.sql regenerated to schema-embedded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval-contradictions): cross-source + cost-tracker + cache wrappers Three pure-orchestration modules between the engine surface and the runner. Each is independently testable; the cache wrapper does hit the PGLite engine end-to-end since its job is to round-trip through P2. - cross-source.ts (M6): classifySlugTier maps a slug to curated/bulk/other using DEFAULT_SOURCE_BOOSTS (boost > 1.05 = curated, < 0.95 = bulk). buildSourceTierBreakdown produces the {curated_vs_curated, curated_vs_bulk, bulk_vs_bulk, other} counts; order-independent on the pair members. - cost-tracker.ts (A2 + P3): estimateUpperBoundCost for pre-flight refuse. CostTracker records judge calls (per-token-pricing per model) AND embedding calls (Codex P3 fix). Soft-ceiling semantics documented in the estimate_note string surfaced in the final report (Codex caveat: "hard ceiling" was overclaimed for token estimates). Anthropic + OpenAI pricing baked in; unknown models fall back to Haiku rates. - cache.ts (P2 wrapper): hashContent (sha256), buildCacheKey with lex-sorted (a, b) so verdicts are order-independent and key bakes in PROMPT_VERSION + TRUNCATION_POLICY (Codex outside-voice fix). JudgeCache class tracks hits/misses for the run report. Shape validation guards against corrupt rows: a cache row that doesn't parse as JudgeVerdict treats as a miss instead of crashing downstream. 40 hermetic tests across the three modules. Cache tests hit PGLite for real round-trip coverage of the new engine methods committed in C2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval-contradictions): judge + auto-supersession + fixture-redact Three modules that together turn an LLM into a contradiction probe and its output into actionable resolutions. - judge.ts: judgeContradiction() is the single LLM call. Query-conditioned prompt (Codex outside-voice fix — the judge sees what the user asked). Holder context for take pairs (C3). UTF-8-safe truncation at maxPairChars (default 1500, --max-pair-chars overridable; C4 wire-up). C1 double-enforcement: orchestrator filters contradicts:true with confidence < 0.7 to false regardless of prompt rules. parseJudgeJSON is a 3-strategy generic parser (direct → fence-strip → trailing-comma + quote + first-{} extraction) — we don't reuse parseModelJSON because that's shape-locked to cross-modal-eval's scores payload. Refusal detection via stopReason AND text-pattern fallback. chatFn injection for hermetic tests. - auto-supersession.ts (M7): proposeResolution classifies each pair into takes_supersede / dream_synthesize / takes_mark_debate / manual_review and emits a paste-ready CLI command. Judge's hint wins on cross-slug pairs (it has semantic context); structural fallback prefers dream_synthesize when either side is a curated entity slug (companies/, people/, deals/, projects/). pairToFinding merges a pair + verdict into a ContradictionFinding. - fixture-redact.ts (T2): privacy-redacted pass for the gold fixture build. Layers PII scrubber (v0.25.0 eval-capture-scrub) + slug rewrites (people/<name> → people/alice-example, deterministic per session) + capitalized firstname-lastname detection + monetary obfuscation (multiply revenues by session salt to preserve magnitude shape). isCleanForCommit is the pre-commit safety net: blocks if any raw name or email shape survives. Audit trail records every redaction made. 60 hermetic tests. Judge tests use direct chatFn stub (cleaner than module-level transport seam for one-shot wrapper). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval-contradictions): trends + runner orchestrator (v0.33.0) The heart of the probe — runner.ts ties every prior module together, trends.ts writes one row per run to eval_contradictions_runs and produces the trend chart for the CLI `trend` sub-subcommand. runner.ts: - Pair generation: cross-slug across top-K results (same-slug skipped) + intra-page chunk-vs-take via P1 batched listActiveTakesForPages. - A1 date pre-filter wired: pairs separated by >30 days skip without judge calls; same-paragraph-dual-date overrides separation rule (flip-flop case sees the judge). - A3 deterministic sampling: combined_score DESC, slug-lex tiebreaker, stable across re-runs. - A2 soft budget ceiling: pre-flight estimate refuses without --yes; mid-run cumulative cost stops the run and emits a partial report. - P2 cache integration: lookup before judge call, store after; hit/miss counters drive the cache stats block in the report. - C2 first-class judge_errors: every throw counted via the typed collector, surfaced in report.judge_errors with the no-silent-skip `note` field. - Wilson CI on the headline percentage; small_sample_note when n<30. - source_tier_breakdown + hot_pages aggregated across all findings. - AbortSignal propagation for cancellation mid-run. - PreFlightBudgetError exported as a discriminable rejection class. - Hermetic via judgeFn + searchFn dependency injection — runner tests stub both without ever touching the real gateway or hybridSearch. trends.ts: - writeRunRow flattens a ProbeReport into the eval_contradictions_runs row shape, including Wilson CI bounds + duration_ms. - loadTrend reads back as typed TrendRow[]. - renderTrendChart produces a fixed-width ASCII bar chart; empty input prints a friendly message naming the command to populate runs. 41 new hermetic tests on PGLite (15 trends, 26 runner). Full eval-contradictions suite at 194/194 across 13 files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval-contradictions): CLI + eval dispatch + mini fixture (v0.33.0) User-facing surface: `gbrain eval suspected-contradictions [run|trend|review]`. Engine-required sub-subcommand, dispatched via the existing eval.ts pattern (matches `replay`). Run mode: --queries-file FILE | --query "..." | --from-capture (mutually exclusive) --top-k N=5 --judge MODEL=claude-haiku-4-5 --limit N --budget-usd N (default $5 TTY / $1 non-TTY) --yes --output FILE --max-pair-chars N=1500 --sampling deterministic|score-first --no-cache --refresh-cache --json Trend mode: --days N=30 [--json] Review mode: --severity low|medium|high --since YYYY-MM-DD A4 wired: --from-capture detects empty eval_candidates and exits 2 with hint naming GBRAIN_CONTRIBUTOR_MODE=1 / eval.capture config key. Human summary on stderr always prints Wilson CI band, judge_errors counts broken out by class, cache hit-rate, source-tier breakdown, hot pages. Partial-report warning when mid-run budget cap fires. Run-row persistence (M5) writes to eval_contradictions_runs every successful run; subsequent `trend` and `review` invocations read from there. PreFlightBudgetError surfaces as exit 1 with the calculated estimate + cap in the message — operators see the exact number to pass to --budget-usd or override with --yes. TrendRow type extended with report_json so `review` can fetch the latest run's findings without a second query. test/fixtures/contradictions-mini.jsonl: 5 redacted queries for CLI smoke. Full eval-contradictions suite: 194 hermetic tests across 13 files. Real- brain CLI smoke covered by the E2E in commit 9. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval-contradictions): doctor + MCP + synthesize integrations (M1+M2+M3) Three thin wire-ups that turn the probe's output into action surfaces: M1 (doctor): src/commands/doctor.ts adds a `contradictions` check after the eval_capture check. Reads loadContradictionsTrend(7), surfaces the latest run's headline + severity breakdown + Wilson CI band + first 3 high-severity findings with paste-ready resolution commands. ok status when no runs exist or no findings; warn when high-severity > 0. Graceful skip when the table doesn't exist yet (pre-migration brain). M3 (MCP): src/core/operations.ts adds `find_contradictions` op (scope: read, NOT localOnly — agent-callable over HTTP MCP). Params: slug (substring match), severity (low|medium|high), limit. Reads loadContradictionsTrend(30), returns the latest run's findings filtered. NOT in the subagent allowlist by design — user-initiated only, not autonomous-action surface. New FIND_CONTRADICTIONS_DESCRIPTION constant in operations-descriptions.ts. M2 (synthesize): src/core/cycle/synthesize.ts pre-fetches the latest probe findings once at phase start (loadPriorContradictionsBlock helper) and threads up to 5 highest-severity items into buildSynthesisPrompt as an informational block. Subagent sees what to reconcile when writing compiled_truth to flagged slugs. Empty trend yields empty block (existing behavior unchanged on fresh installs). Try/catch around the engine call keeps synthesize robust even when the contradiction tables don't exist yet. 11 new hermetic tests for the MCP op (registry presence, scope, empty case, slug+severity+limit filters) and the M1/M2 data-shape contracts (end-to-end runDoctor coverage deferred to commit 9's E2E because doctor calls process.exit). Full eval-contradictions suite: 226/226 across 15 test files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval-contradictions): build-contradictions-fixture script (T2) Local-only operator script for building the privacy-redacted gold fixture used by the precision/recall test (deferred to v0.34 when probe data informs the labeling). Runs against the user's REAL brain via the local gbrain engine config; never auto-run in CI. Flow: 1. Read --queries-file (JSONL); spin up engine via loadConfig + toEngineConfig + createEngine + connectWithRetry. 2. Run the contradiction probe with --no-cache and a stubbed judgeFn that captures candidate pairs without spending tokens. 3. Interactive prompts (skipped under --non-interactive): for each candidate, the operator labels y/n/skip + severity + axis. 4. Apply the v0.33.0 fixture-redact passes (slug rewrite, name placeholders, monetary obfuscation, PII scrubber). 5. Pre-commit safety gate: every text field passes isCleanForCommit; anything that fails gets a [REDACT?] sentinel + an _operator_review marker on the JSONL line, and the script exits 1 so the operator can't accidentally commit unredacted output. Audit comment block at the top of the JSONL records every redaction the session made (slug→placeholder, name→placeholder, monetary multiplication) so reviewers can see what was changed. Usage: bun run scripts/build-contradictions-fixture.ts \\ --queries-file FILE.jsonl \\ [--top-k N] [--judge MODEL] [--max-pairs N] [--output PATH] \\ [--non-interactive] Output defaults to test/fixtures/contradictions-eval-gold.jsonl. Typecheck clean; redactor + isCleanForCommit guard tested separately in test/eval-contradictions-fixture-redact.test.ts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): real-Postgres E2E for contradiction probe (v0.33.0, T1) Required-on-DATABASE_URL E2E covering Postgres-specific behavior that PGLite can't exercise. Six surface areas, 12 cases total. All pass on fresh pgvector/pgvector:pg16: 1. Migrations v51 + v52 apply cleanly; both tables exist in information_schema; Wilson CI columns are REAL; composite PK on eval_contradictions_cache includes prompt_version + truncation_policy (Codex outside-voice fix pinned at the schema level). 2. JSONB round-trip on Postgres: writeContradictionsRun + loadTrend preserves nested object shapes (regression guard against the v0.12 double-encode bug class). Confirmed via jsonb_typeof = 'object', not 'string'. 3. P2 cache with real now(): lookup/upsert round-trip, expired rows hidden from lookup, sweepContradictionCache deletes them, and different prompt_version is a separate cache key. 4. M5 trend semantics: TIMESTAMPTZ ordering DESC is stable on real PG; days-window filter via ran_at >= cutoff correctly excludes/includes backdated rows. 5. find_contradictions MCP op end-to-end: empty case returns "No probe runs" note; populated case returns latest run findings with slug substring + severity filters applied. Verified locally against pgvector:pg16 on port 5434 — all 12 cases pass. Skips gracefully when DATABASE_URL is unset per gbrain E2E convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.33.0 feat: brain-consistency probe + doctor + MCP + dream-cycle wire-up VERSION 0.32.0 → 0.33.0. package.json + CHANGELOG.md + llms-full.txt synced. Headline: gbrain learns to detect its own integrity drift. - new command: gbrain eval suspected-contradictions [run|trend|review] - new MCP op: find_contradictions(slug?, severity?, limit?) - new doctor check: contradictions (paste-ready resolution commands) - new dream-cycle hook: synthesize reads prior contradictions per slug - new schema: v51 (eval_contradictions_cache) + v52 (eval_contradictions_runs) - 6 new engine methods (listActiveTakesForPages, write/load run, P2 cache trio) Codex outside-voice review folded in: - Command name "suspected-contradictions" (was "contradictions" — describes what the tool actually does, not what it pretends to evaluate) - judge_errors first-class output (not silent stderr — biased denominator) - prompt_version + truncation_policy in cache key (prompt edits cleanly invalidate prior verdicts) - Wilson 95% CI on headline % + small_sample_note when n<30 - Query-conditioned judge prompt (sees user's query, not just two chunks) - Deterministic sampling for prevalence metric (stable cache hit-rate) Decision criterion for the bigger swing (chunk-level revises field): Wilson CI lower-bound: <5% → source-boost + recency-decay + curated pages handle the load 5-15% → operator's call >15% → plan for v0.34+ New docs: - docs/contradictions.md (architecture, severity rubric, action criteria) - docs/eval-bench.md extended (nightly cadence + trend workflow) - skills/migrations/v0.33.0.md (post-upgrade agent instructions) Full test suite green at the cut: - 226 hermetic unit tests across 15 files (eval-contradictions-*) - 12 real-Postgres E2E (DATABASE_URL=...; verified locally on pgvector:pg16) - typecheck clean - build:llms regenerated and the test/build-llms.test.ts gate passes Plan reference: ~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regen llms-full.txt for v0.32.6 rename --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |