mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 06:23:01 +00:00
feat(eval): BrainBench core — adapters, metrics, harness, scoreboard, governance
Cross-harness memory conformance suite (Cathedral 2): one shared Reflex pipeline driven by three seam adapters (openclaw=production, claude-code + codex=contract with exported wire types), four metric suites (know-to-ask + false-fire, push P/R, write-back via the production pipeline, continuity), source-isolation gating at zero, intrusion diagnostics, sealed-gold loader with strict validation + corpus hash, fail-fast hermetic seeding, ONE PGLite per run with resetTables between fixtures, and the main-baseline compare gate (same-hash count-aware / corpus-bless modes, justification flow). Eleven new metrics registered in the glossary (doc regenerated).
This commit is contained in:
Garry Tan
@@ -168,6 +168,96 @@ Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-Engli
|
||||
|
||||
**Range:** 0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.
|
||||
|
||||
## BrainBench — Cross-Harness Memory Conformance
|
||||
|
||||
### Know-to-ask failure rate (BrainBench)
|
||||
|
||||
**Key:** `know_to_ask_failure_rate`
|
||||
|
||||
**Plain English:** Of the conversation turns where memory SHOULD have surfaced something unprompted, the fraction where nothing relevant was injected. This is the thesis failure mode every agent harness shares: the agent can't ask for what it doesn't know it forgot — the memory layer has to volunteer it.
|
||||
|
||||
**Range:** 0..1, LOWER is better. 0.15 means memory stayed silent on 15% of the turns where it had the answer.
|
||||
|
||||
### False-fire rate (BrainBench)
|
||||
|
||||
**Key:** `false_fire_rate`
|
||||
|
||||
**Plain English:** Of the turns where memory should have stayed SILENT, the fraction where it injected anyway. The anti-gaming companion to the know-to-ask rate — "always inject" would ace one and bomb the other. Silence beats noise.
|
||||
|
||||
**Range:** 0..1, LOWER is better.
|
||||
|
||||
### Push precision (BrainBench)
|
||||
|
||||
**Key:** `push_precision`
|
||||
|
||||
**Plain English:** Of everything the memory layer volunteered into context, what fraction was actually relevant to the turn? Micro-averaged over injected pointers, so a 3-pointer turn weighs three times a 1-pointer turn — the way a token budget experiences it.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Push recall (BrainBench)
|
||||
|
||||
**Key:** `push_recall`
|
||||
|
||||
**Plain English:** Of everything that SHOULD have been volunteered (the gold pointers), what fraction actually was? Pointer budgets cap this by design: a seam that may inject only 1 fragment cannot reach full recall on a 3-entity turn — that constraint is what the per-harness rows measure.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Write-back fidelity (BrainBench)
|
||||
|
||||
**Key:** `write_back_fidelity`
|
||||
|
||||
**Plain English:** Of the facts stated in a conversation, what fraction survived the PRODUCTION conversation→memory pipeline (segmentation, insertion, dedup) and are findable afterward with the right entity attached? Measures the write path users actually run, not a test-only insert.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Provenance accuracy (BrainBench)
|
||||
|
||||
**Key:** `provenance_accuracy`
|
||||
|
||||
**Plain English:** Of the facts that survived write-back, what fraction carry correct provenance — the right source tag, session id, and origin page? A fact you can't trace is a fact you can't trust, audit, or expire.
|
||||
|
||||
**Range:** 0..1, higher is better.
|
||||
|
||||
### Cross-session continuity rate (BrainBench)
|
||||
|
||||
**Key:** `continuity_rate`
|
||||
|
||||
**Plain English:** A decision is recorded in one harness's session; a DIFFERENT harness asks about it later on the same brain. What fraction of those decision probes were recalled — by pointer injection or stored-fact lookup? This is the continuity-that-survives-the-harness-hop moat, measured.
|
||||
|
||||
**Range:** 0..1, higher is better. Headline = mean over writer→reader pairs with different harnesses.
|
||||
|
||||
### Source-isolation violations (BrainBench)
|
||||
|
||||
**Key:** `source_isolation_violations`
|
||||
|
||||
**Plain English:** Count of injected pointers that belong to a source other than the active one. Cross-source leakage is gbrain's must-never-violate invariant (a missed source filter is a data leak), so this gates at ZERO — any baseline, any run.
|
||||
|
||||
**Range:** 0..n, count. MUST be 0; any value above 0 fails the gate.
|
||||
|
||||
### Average injected tokens per turn (BrainBench)
|
||||
|
||||
**Key:** `avg_injected_tokens`
|
||||
|
||||
**Plain English:** Estimated tokens of volunteered context per replayed turn (chars/4 heuristic). The intrusion-budget diagnostic: two seams with equal precision can differ 3x in how much context they spend to get it. Reported, not gated, until calibration data exists.
|
||||
|
||||
**Range:** 0..n tokens, judgment call — lower is cheaper, but starving the agent has its own cost. Non-gating.
|
||||
|
||||
### Extraction recall (BrainBench --llm)
|
||||
|
||||
**Key:** `extraction_recall`
|
||||
|
||||
**Plain English:** With the real LLM extractor running (instead of the deterministic gold extractor), what fraction of the gold facts did it actually extract and persist? Only scored in --llm runs — the hermetic CI gate never calls a model.
|
||||
|
||||
**Range:** 0..1, higher is better. Absent in deterministic runs.
|
||||
|
||||
### Extraction precision (BrainBench --llm)
|
||||
|
||||
**Key:** `extraction_precision`
|
||||
|
||||
**Plain English:** Of everything the real LLM extractor persisted, what fraction matches a gold fact? Low precision means the extractor invents or over-extracts — junk memory that pollutes future recall.
|
||||
|
||||
**Range:** 0..1, higher is better. Absent in deterministic runs.
|
||||
|
||||
---
|
||||
|
||||
## Coverage
|
||||
|
||||
@@ -147,6 +147,66 @@ export const METRIC_GLOSSARY: Readonly<Record<string, Readonly<MetricGlossEntry>
|
||||
eli10: 'The size of the largest score drop autocut found, as a fraction of the top result\'s score. A gap of 0.40 means the score fell by 40% of the top score at the steepest point. Autocut cuts there only when this clears the sensitivity threshold (autocut_jump, default 0.20).',
|
||||
range: '0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.',
|
||||
}),
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// BrainBench — cross-harness memory conformance suite
|
||||
// (`gbrain eval brainbench`; docs/eval/BRAINBENCH.md)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'know_to_ask_failure_rate': Object.freeze({
|
||||
industry_term: 'Know-to-ask failure rate (BrainBench)',
|
||||
eli10: 'Of the conversation turns where memory SHOULD have surfaced something unprompted, the fraction where nothing relevant was injected. This is the thesis failure mode every agent harness shares: the agent can\'t ask for what it doesn\'t know it forgot — the memory layer has to volunteer it.',
|
||||
range: '0..1, LOWER is better. 0.15 means memory stayed silent on 15% of the turns where it had the answer.',
|
||||
}),
|
||||
'false_fire_rate': Object.freeze({
|
||||
industry_term: 'False-fire rate (BrainBench)',
|
||||
eli10: 'Of the turns where memory should have stayed SILENT, the fraction where it injected anyway. The anti-gaming companion to the know-to-ask rate — "always inject" would ace one and bomb the other. Silence beats noise.',
|
||||
range: '0..1, LOWER is better.',
|
||||
}),
|
||||
'push_precision': Object.freeze({
|
||||
industry_term: 'Push precision (BrainBench)',
|
||||
eli10: 'Of everything the memory layer volunteered into context, what fraction was actually relevant to the turn? Micro-averaged over injected pointers, so a 3-pointer turn weighs three times a 1-pointer turn — the way a token budget experiences it.',
|
||||
range: '0..1, higher is better.',
|
||||
}),
|
||||
'push_recall': Object.freeze({
|
||||
industry_term: 'Push recall (BrainBench)',
|
||||
eli10: 'Of everything that SHOULD have been volunteered (the gold pointers), what fraction actually was? Pointer budgets cap this by design: a seam that may inject only 1 fragment cannot reach full recall on a 3-entity turn — that constraint is what the per-harness rows measure.',
|
||||
range: '0..1, higher is better.',
|
||||
}),
|
||||
'write_back_fidelity': Object.freeze({
|
||||
industry_term: 'Write-back fidelity (BrainBench)',
|
||||
eli10: 'Of the facts stated in a conversation, what fraction survived the PRODUCTION conversation→memory pipeline (segmentation, insertion, dedup) and are findable afterward with the right entity attached? Measures the write path users actually run, not a test-only insert.',
|
||||
range: '0..1, higher is better.',
|
||||
}),
|
||||
'provenance_accuracy': Object.freeze({
|
||||
industry_term: 'Provenance accuracy (BrainBench)',
|
||||
eli10: 'Of the facts that survived write-back, what fraction carry correct provenance — the right source tag, session id, and origin page? A fact you can\'t trace is a fact you can\'t trust, audit, or expire.',
|
||||
range: '0..1, higher is better.',
|
||||
}),
|
||||
'continuity_rate': Object.freeze({
|
||||
industry_term: 'Cross-session continuity rate (BrainBench)',
|
||||
eli10: 'A decision is recorded in one harness\'s session; a DIFFERENT harness asks about it later on the same brain. What fraction of those decision probes were recalled — by pointer injection or stored-fact lookup? This is the continuity-that-survives-the-harness-hop moat, measured.',
|
||||
range: '0..1, higher is better. Headline = mean over writer→reader pairs with different harnesses.',
|
||||
}),
|
||||
'source_isolation_violations': Object.freeze({
|
||||
industry_term: 'Source-isolation violations (BrainBench)',
|
||||
eli10: 'Count of injected pointers that belong to a source other than the active one. Cross-source leakage is gbrain\'s must-never-violate invariant (a missed source filter is a data leak), so this gates at ZERO — any baseline, any run.',
|
||||
range: '0..n, count. MUST be 0; any value above 0 fails the gate.',
|
||||
}),
|
||||
'avg_injected_tokens': Object.freeze({
|
||||
industry_term: 'Average injected tokens per turn (BrainBench)',
|
||||
eli10: 'Estimated tokens of volunteered context per replayed turn (chars/4 heuristic). The intrusion-budget diagnostic: two seams with equal precision can differ 3x in how much context they spend to get it. Reported, not gated, until calibration data exists.',
|
||||
range: '0..n tokens, judgment call — lower is cheaper, but starving the agent has its own cost. Non-gating.',
|
||||
}),
|
||||
'extraction_recall': Object.freeze({
|
||||
industry_term: 'Extraction recall (BrainBench --llm)',
|
||||
eli10: 'With the real LLM extractor running (instead of the deterministic gold extractor), what fraction of the gold facts did it actually extract and persist? Only scored in --llm runs — the hermetic CI gate never calls a model.',
|
||||
range: '0..1, higher is better. Absent in deterministic runs.',
|
||||
}),
|
||||
'extraction_precision': Object.freeze({
|
||||
industry_term: 'Extraction precision (BrainBench --llm)',
|
||||
eli10: 'Of everything the real LLM extractor persisted, what fraction matches a gold fact? Low precision means the extractor invents or over-extracts — junk memory that pollutes future recall.',
|
||||
range: '0..1, higher is better. Absent in deterministic runs.',
|
||||
}),
|
||||
});
|
||||
|
||||
/**
|
||||
@@ -225,6 +285,12 @@ export function renderMetricGlossaryMarkdown(): string {
|
||||
['Statistical-Significance Metrics', ['p_value', 'confidence_interval']],
|
||||
['Operational / Cost Metrics', ['cache_hit_rate', 'avg_results', 'avg_tokens', 'cost_per_query_usd', 'p99_latency_ms']],
|
||||
['Result-Sizing Metrics', ['autocut.signal', 'autocut.gap_ratio']],
|
||||
['BrainBench — Cross-Harness Memory Conformance', [
|
||||
'know_to_ask_failure_rate', 'false_fire_rate', 'push_precision', 'push_recall',
|
||||
'write_back_fidelity', 'provenance_accuracy', 'continuity_rate',
|
||||
'source_isolation_violations', 'avg_injected_tokens',
|
||||
'extraction_recall', 'extraction_precision',
|
||||
]],
|
||||
];
|
||||
|
||||
for (const [groupTitle, metrics] of groups) {
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
/**
|
||||
* BrainBench Claude Code adapter — seam: 'contract'.
|
||||
*
|
||||
* Defines the UserPromptSubmit hook contract a future integration PR
|
||||
* implements, and grades gbrain's memory primitives through it. The exported
|
||||
* wire types are THE contract: the real hook script will read
|
||||
* `UserPromptSubmitHookInput` JSON on stdin and write
|
||||
* `UserPromptSubmitHookOutput` JSON on stdout, so the bench is test-first for
|
||||
* the integration — when the real hook lands, this adapter swaps its in-process
|
||||
* transport for an exec of the hook script and flips seam to 'production' with
|
||||
* continuous bench numbers.
|
||||
*
|
||||
* Contract deltas vs the openclaw seam (the deltas ARE what the row measures):
|
||||
* - NO conversation memory: a hook sees only the current prompt, so
|
||||
* prior-context suppression is off → expect a higher re-injection rate.
|
||||
* - Smaller injection budget (2 pointers): hooks compete with everything
|
||||
* else feeding additionalContext.
|
||||
*
|
||||
* Every turn round-trips through JSON.stringify/parse of the wire shapes so a
|
||||
* contract-breaking change fails loudly here, not in a future integration.
|
||||
*/
|
||||
|
||||
import type { PGLiteEngine } from '../../../core/pglite-engine.ts';
|
||||
import type {
|
||||
AdapterFixtureView,
|
||||
HarnessAdapter,
|
||||
HarnessTurnResult,
|
||||
PublicTurn,
|
||||
} from '../types.ts';
|
||||
import { runReflexPipeline, toTurnResult } from './shared.ts';
|
||||
|
||||
/** stdin JSON the real UserPromptSubmit hook will receive. */
|
||||
export interface UserPromptSubmitHookInput {
|
||||
prompt: string;
|
||||
session_id: string;
|
||||
cwd: string;
|
||||
}
|
||||
|
||||
/** stdout JSON the real UserPromptSubmit hook will emit. */
|
||||
export interface UserPromptSubmitHookOutput {
|
||||
hookSpecificOutput: {
|
||||
hookEventName: 'UserPromptSubmit';
|
||||
additionalContext: string;
|
||||
};
|
||||
}
|
||||
|
||||
export const CLAUDE_CODE_MAX_POINTERS = 2;
|
||||
|
||||
export class ClaudeCodeAdapter implements HarnessAdapter {
|
||||
readonly name = 'claude-code' as const;
|
||||
readonly seam = 'contract' as const;
|
||||
|
||||
private engine: PGLiteEngine | null = null;
|
||||
private sourceId = 'default';
|
||||
private sessionId = '';
|
||||
|
||||
async beginConversation(engine: PGLiteEngine, fixture: AdapterFixtureView): Promise<void> {
|
||||
this.engine = engine;
|
||||
this.sourceId = fixture.active_source;
|
||||
this.sessionId = `brainbench-${fixture.fixture_id}`;
|
||||
}
|
||||
|
||||
async replayTurn(turn: PublicTurn, _priorContextText: string): Promise<HarnessTurnResult> {
|
||||
if (!this.engine) throw new Error('claude-code adapter: beginConversation not called');
|
||||
const started = performance.now();
|
||||
|
||||
// Serialize to the hook's stdin wire shape, then parse back — the
|
||||
// contract boundary is exercised on every turn.
|
||||
const wireIn: UserPromptSubmitHookInput = {
|
||||
prompt: turn.text,
|
||||
session_id: this.sessionId,
|
||||
cwd: '/',
|
||||
};
|
||||
const parsedIn = JSON.parse(JSON.stringify(wireIn)) as UserPromptSubmitHookInput;
|
||||
|
||||
const block = await runReflexPipeline(
|
||||
this.engine,
|
||||
this.sourceId,
|
||||
{ ...turn, text: parsedIn.prompt },
|
||||
'', // a hook has no prior-turn memory — deliberate contract delta
|
||||
{ maxPointers: CLAUDE_CODE_MAX_POINTERS, suppression: 'none' },
|
||||
);
|
||||
|
||||
const wireOut: UserPromptSubmitHookOutput = {
|
||||
hookSpecificOutput: {
|
||||
hookEventName: 'UserPromptSubmit',
|
||||
additionalContext: block?.text ?? '',
|
||||
},
|
||||
};
|
||||
const parsedOut = JSON.parse(JSON.stringify(wireOut)) as UserPromptSubmitHookOutput;
|
||||
const injected = parsedOut.hookSpecificOutput.additionalContext || null;
|
||||
|
||||
const latencyMs = performance.now() - started;
|
||||
return toTurnResult(block, injected, latencyMs);
|
||||
}
|
||||
|
||||
async endConversation(): Promise<void> {
|
||||
this.engine = null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* BrainBench Codex adapter — seam: 'contract'.
|
||||
*
|
||||
* Models the fragments integration shape: a STATIC entity-index preamble
|
||||
* (AGENTS.md-style — computed once at conversation start, slugs + titles only,
|
||||
* no per-turn awareness) plus AT MOST ONE per-turn fragment. Grades how much
|
||||
* push quality degrades when injection is mostly static.
|
||||
*
|
||||
* Scoring honesty: the static preamble is an INDEX (it names every page), so
|
||||
* its slugs deliberately do NOT count as injectedSlugs — counting them would
|
||||
* trivially game push_recall. Only the per-turn fragment is scored; the
|
||||
* preamble's size shows up once in turn 1's injectedTokens so the intrusion
|
||||
* diagnostics (decision 18) see its cost.
|
||||
*/
|
||||
|
||||
import type { PGLiteEngine } from '../../../core/pglite-engine.ts';
|
||||
import type {
|
||||
AdapterFixtureView,
|
||||
HarnessAdapter,
|
||||
HarnessTurnResult,
|
||||
PublicTurn,
|
||||
} from '../types.ts';
|
||||
import { estimateTokens, runReflexPipeline, toTurnResult } from './shared.ts';
|
||||
|
||||
export const CODEX_MAX_FRAGMENTS = 1;
|
||||
/** Preamble caps: keep the static index bounded like a real AGENTS.md section. */
|
||||
const PREAMBLE_MAX_PAGES = 50;
|
||||
|
||||
export class CodexAdapter implements HarnessAdapter {
|
||||
readonly name = 'codex' as const;
|
||||
readonly seam = 'contract' as const;
|
||||
|
||||
private engine: PGLiteEngine | null = null;
|
||||
private sourceId = 'default';
|
||||
private preamble = '';
|
||||
private firstTurn = true;
|
||||
|
||||
async beginConversation(engine: PGLiteEngine, fixture: AdapterFixtureView): Promise<void> {
|
||||
this.engine = engine;
|
||||
this.sourceId = fixture.active_source;
|
||||
this.firstTurn = true;
|
||||
this.preamble = await this.buildPreamble();
|
||||
}
|
||||
|
||||
/** Static entity index: titles + slugs in the active source, alphabetical. */
|
||||
private async buildPreamble(): Promise<string> {
|
||||
if (!this.engine) return '';
|
||||
try {
|
||||
const rows = await this.engine.executeRaw<{ slug: string; title: string }>(
|
||||
`SELECT slug, title FROM pages
|
||||
WHERE deleted_at IS NULL AND source_id = $1
|
||||
ORDER BY slug LIMIT $2`,
|
||||
[this.sourceId, PREAMBLE_MAX_PAGES],
|
||||
);
|
||||
if (!rows.length) return '';
|
||||
const lines = ['## Brain index', ...rows.map((r) => `- ${r.title} → \`${r.slug}\``)];
|
||||
return lines.join('\n');
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
async replayTurn(turn: PublicTurn, priorContextText: string): Promise<HarnessTurnResult> {
|
||||
if (!this.engine) throw new Error('codex adapter: beginConversation not called');
|
||||
const started = performance.now();
|
||||
const block = await runReflexPipeline(this.engine, this.sourceId, turn, priorContextText, {
|
||||
maxPointers: CODEX_MAX_FRAGMENTS,
|
||||
suppression: 'prior-context',
|
||||
});
|
||||
const latencyMs = performance.now() - started;
|
||||
|
||||
const fragment = block?.text ?? null;
|
||||
const result = toTurnResult(block, fragment, latencyMs);
|
||||
if (this.firstTurn) {
|
||||
// The static preamble's cost lands once, on the first turn, so intrusion
|
||||
// diagnostics see it without its slugs polluting push metrics.
|
||||
result.injectedTokens += estimateTokens(this.preamble);
|
||||
this.firstTurn = false;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async endConversation(): Promise<void> {
|
||||
this.engine = null;
|
||||
this.preamble = '';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* BrainBench OpenClaw adapter — seam: 'production'.
|
||||
*
|
||||
* Drives the exact pipeline the shipped OpenClaw context engine runs per turn
|
||||
* (src/core/context-engine.ts → buildReflexAddition → extractCandidates →
|
||||
* resolveEntitiesToPointers), with production defaults: 3-pointer budget,
|
||||
* prior-context suppression, markdown pointer-block wire shape. What this row
|
||||
* scores is what an OpenClaw user's reflex actually does.
|
||||
*/
|
||||
|
||||
import type { PGLiteEngine } from '../../../core/pglite-engine.ts';
|
||||
import { DEFAULT_MAX_POINTERS } from '../../../core/context/retrieval-reflex.ts';
|
||||
import type {
|
||||
AdapterFixtureView,
|
||||
HarnessAdapter,
|
||||
HarnessTurnResult,
|
||||
PublicTurn,
|
||||
} from '../types.ts';
|
||||
import { runReflexPipeline, toTurnResult } from './shared.ts';
|
||||
|
||||
export class OpenClawAdapter implements HarnessAdapter {
|
||||
readonly name = 'openclaw' as const;
|
||||
readonly seam = 'production' as const;
|
||||
|
||||
private engine: PGLiteEngine | null = null;
|
||||
private sourceId = 'default';
|
||||
|
||||
async beginConversation(engine: PGLiteEngine, fixture: AdapterFixtureView): Promise<void> {
|
||||
this.engine = engine;
|
||||
this.sourceId = fixture.active_source;
|
||||
}
|
||||
|
||||
async replayTurn(turn: PublicTurn, priorContextText: string): Promise<HarnessTurnResult> {
|
||||
if (!this.engine) throw new Error('openclaw adapter: beginConversation not called');
|
||||
const started = performance.now();
|
||||
const block = await runReflexPipeline(this.engine, this.sourceId, turn, priorContextText, {
|
||||
maxPointers: DEFAULT_MAX_POINTERS,
|
||||
suppression: 'prior-context',
|
||||
});
|
||||
const latencyMs = performance.now() - started;
|
||||
return toTurnResult(block, block?.text ?? null, latencyMs);
|
||||
}
|
||||
|
||||
async endConversation(): Promise<void> {
|
||||
this.engine = null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
/**
|
||||
* BrainBench shared Reflex pipeline (decision 13).
|
||||
*
|
||||
* ONE tested pipeline — extractCandidates → resolveEntitiesToPointers → slug
|
||||
* normalization — that all three adapters drive with declarative config. This
|
||||
* keeps cross-harness comparability STRUCTURAL: the primitives are identical
|
||||
* by construction; only the seam config (pointer budget, suppression mode,
|
||||
* wire shape) varies per adapter. A Reflex evolution lands here once and every
|
||||
* harness's score moves together.
|
||||
*
|
||||
* Tracks the production resolver ladder in src/core/context/reflex.ts —
|
||||
* alias-first via engine.resolveAliases, then title/slug-suffix. The
|
||||
* production orchestrator's loadConfig() gate, integration heartbeat, and
|
||||
* 1500ms timeout wrapper are deliberately NOT graded (disclosed in
|
||||
* docs/eval/BRAINBENCH.md).
|
||||
*/
|
||||
|
||||
import { extractCandidates } from '../../../core/context/entity-salience.ts';
|
||||
import {
|
||||
resolveEntitiesToPointers,
|
||||
type PointerBlock,
|
||||
} from '../../../core/context/retrieval-reflex.ts';
|
||||
import type { PGLiteEngine } from '../../../core/pglite-engine.ts';
|
||||
import type { HarnessTurnResult, PublicTurn } from '../types.ts';
|
||||
|
||||
export interface ReflexPipelineCfg {
|
||||
/** Pointer budget for this seam (openclaw 3, claude-code 2, codex 1). */
|
||||
maxPointers: number;
|
||||
/**
|
||||
* 'prior-context' — production suppression (pointers already seen in prior
|
||||
* turns are not re-injected). 'none' — the seam has no conversation memory
|
||||
* (the claude-code hook contract sees only the current prompt); the higher
|
||||
* re-injection rate that results is part of what the bench measures.
|
||||
*/
|
||||
suppression: 'prior-context' | 'none';
|
||||
}
|
||||
|
||||
/** chars/4 heuristic — intrusion diagnostics only, never a gate (decision 18). */
|
||||
export function estimateTokens(text: string | null): number {
|
||||
if (!text) return 0;
|
||||
return Math.ceil(text.length / 4);
|
||||
}
|
||||
|
||||
export async function runReflexPipeline(
|
||||
engine: PGLiteEngine,
|
||||
sourceId: string,
|
||||
turn: PublicTurn,
|
||||
priorContextText: string,
|
||||
cfg: ReflexPipelineCfg,
|
||||
): Promise<PointerBlock | null> {
|
||||
const candidates = extractCandidates(turn.text);
|
||||
if (!candidates.length) return null;
|
||||
return resolveEntitiesToPointers(engine, sourceId, candidates, {
|
||||
maxPointers: cfg.maxPointers,
|
||||
priorContextText: cfg.suppression === 'prior-context' ? priorContextText : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
/** Build the common turn-result shape from a pointer block + the seam's wire text. */
|
||||
export function toTurnResult(
|
||||
block: PointerBlock | null,
|
||||
wireText: string | null,
|
||||
latencyMs: number,
|
||||
): HarnessTurnResult {
|
||||
return {
|
||||
injectedText: wireText,
|
||||
injectedSlugs: block ? block.pointers.map((p) => p.slug) : [],
|
||||
pointers: block?.pointers ?? [],
|
||||
injectedTokens: estimateTokens(wireText),
|
||||
latencyMs,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,369 @@
|
||||
/**
|
||||
* BrainBench fixture loader + strict validator + corpus hash.
|
||||
*
|
||||
* Strictness is the seal: a fixture turn carrying a `gold` key (or any unknown
|
||||
* key) is a hard validation error, because the fixture file is the
|
||||
* adapter-visible surface — gold lives in the sealed gold dir only
|
||||
* (decision 22, gbrain-evals sealed-gold discipline).
|
||||
*
|
||||
* fixtures_hash covers BOTH fixture and gold files (sorted relative path +
|
||||
* content), so a gold-only edit invalidates baseline comparisons exactly like
|
||||
* a fixture edit (decision 4's same-hash vs corpus-bless modes key off it).
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readdir, readFile } from 'node:fs/promises';
|
||||
import { join } from 'node:path';
|
||||
import {
|
||||
ALL_SUITES,
|
||||
FIXTURE_SCHEMA_VERSION,
|
||||
type BrainBenchFixture,
|
||||
type BrainBenchSuite,
|
||||
type FixtureGold,
|
||||
type FixtureTurn,
|
||||
type LoadedCorpus,
|
||||
type LoadedFixture,
|
||||
} from './types.ts';
|
||||
|
||||
export class FixtureValidationError extends Error {
|
||||
constructor(
|
||||
public readonly file: string,
|
||||
message: string,
|
||||
) {
|
||||
super(`${file}: ${message}`);
|
||||
this.name = 'FixtureValidationError';
|
||||
}
|
||||
}
|
||||
|
||||
const FIXTURE_KEYS = new Set([
|
||||
'schema_version', 'fixture_id', 'suites', 'category', 'holdout', 'sources',
|
||||
'active_source', 'seed_pages', 'seed_facts', 'turns', 'continuity',
|
||||
]);
|
||||
const TURN_KEYS = new Set(['turn_id', 'role', 'text', 'ts']);
|
||||
const SEED_PAGE_KEYS = new Set(['slug', 'content', 'source_id']);
|
||||
const SEED_FACT_KEYS = new Set(['fact', 'entity_slug', 'source', 'source_session', 'source_id']);
|
||||
const GOLD_KEYS = new Set(['fixture_id', 'turns', 'continuity']);
|
||||
const TURN_GOLD_KEYS = new Set(['should_retrieve', 'gold_slugs', 'acceptable_slugs', 'gold_facts']);
|
||||
const GOLD_FACT_KEYS = new Set(['gist', 'fact', 'entity_slug', 'match_keywords', 'kind']);
|
||||
|
||||
function assertOnlyKeys(
|
||||
file: string,
|
||||
obj: Record<string, unknown>,
|
||||
allowed: Set<string>,
|
||||
where: string,
|
||||
): void {
|
||||
for (const k of Object.keys(obj)) {
|
||||
if (!allowed.has(k)) {
|
||||
const hint =
|
||||
k === 'gold'
|
||||
? ' — gold is SEALED: it belongs in the gold dir (<fixture_id>.gold.json), never inline'
|
||||
: '';
|
||||
throw new FixtureValidationError(file, `unknown key "${k}" in ${where}${hint}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function isStringArray(v: unknown): v is string[] {
|
||||
return Array.isArray(v) && v.every((x) => typeof x === 'string');
|
||||
}
|
||||
|
||||
export function validateFixture(file: string, raw: unknown): BrainBenchFixture {
|
||||
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
|
||||
throw new FixtureValidationError(file, 'fixture must be a JSON object');
|
||||
}
|
||||
const f = raw as Record<string, unknown>;
|
||||
assertOnlyKeys(file, f, FIXTURE_KEYS, 'fixture');
|
||||
|
||||
if (f.schema_version !== FIXTURE_SCHEMA_VERSION) {
|
||||
throw new FixtureValidationError(
|
||||
file,
|
||||
`schema_version must be ${FIXTURE_SCHEMA_VERSION} (got ${JSON.stringify(f.schema_version)})`,
|
||||
);
|
||||
}
|
||||
if (typeof f.fixture_id !== 'string' || !/^[a-z0-9][a-z0-9-]*$/.test(f.fixture_id)) {
|
||||
throw new FixtureValidationError(file, 'fixture_id must be a kebab-case string');
|
||||
}
|
||||
if (!isStringArray(f.suites) || f.suites.length === 0) {
|
||||
throw new FixtureValidationError(file, 'suites must be a non-empty string array');
|
||||
}
|
||||
for (const s of f.suites) {
|
||||
if (!(ALL_SUITES as readonly string[]).includes(s)) {
|
||||
throw new FixtureValidationError(file, `unknown suite "${s}" (valid: ${ALL_SUITES.join(', ')})`);
|
||||
}
|
||||
}
|
||||
if (f.holdout !== undefined && typeof f.holdout !== 'boolean') {
|
||||
throw new FixtureValidationError(file, 'holdout must be boolean');
|
||||
}
|
||||
if (f.sources !== undefined && !isStringArray(f.sources)) {
|
||||
throw new FixtureValidationError(file, 'sources must be a string array');
|
||||
}
|
||||
if (f.active_source !== undefined && typeof f.active_source !== 'string') {
|
||||
throw new FixtureValidationError(file, 'active_source must be a string');
|
||||
}
|
||||
|
||||
if (f.seed_pages !== undefined) {
|
||||
if (!Array.isArray(f.seed_pages)) throw new FixtureValidationError(file, 'seed_pages must be an array');
|
||||
for (const p of f.seed_pages as Array<Record<string, unknown>>) {
|
||||
assertOnlyKeys(file, p, SEED_PAGE_KEYS, 'seed_pages[]');
|
||||
if (typeof p.slug !== 'string' || !p.slug) throw new FixtureValidationError(file, 'seed_pages[].slug required');
|
||||
if (typeof p.content !== 'string' || !p.content) {
|
||||
throw new FixtureValidationError(file, `seed_pages[${p.slug}].content required`);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (f.seed_facts !== undefined) {
|
||||
if (!Array.isArray(f.seed_facts)) throw new FixtureValidationError(file, 'seed_facts must be an array');
|
||||
for (const sf of f.seed_facts as Array<Record<string, unknown>>) {
|
||||
assertOnlyKeys(file, sf, SEED_FACT_KEYS, 'seed_facts[]');
|
||||
if (typeof sf.fact !== 'string' || !sf.fact) throw new FixtureValidationError(file, 'seed_facts[].fact required');
|
||||
}
|
||||
}
|
||||
|
||||
if (!Array.isArray(f.turns) || f.turns.length === 0) {
|
||||
throw new FixtureValidationError(file, 'turns must be a non-empty array');
|
||||
}
|
||||
const seenTurnIds = new Set<number>();
|
||||
for (const t of f.turns as Array<Record<string, unknown>>) {
|
||||
assertOnlyKeys(file, t, TURN_KEYS, 'turns[]');
|
||||
if (typeof t.turn_id !== 'number' || !Number.isInteger(t.turn_id)) {
|
||||
throw new FixtureValidationError(file, 'turns[].turn_id must be an integer');
|
||||
}
|
||||
if (seenTurnIds.has(t.turn_id)) {
|
||||
throw new FixtureValidationError(file, `duplicate turn_id ${t.turn_id}`);
|
||||
}
|
||||
seenTurnIds.add(t.turn_id);
|
||||
if (t.role !== 'user' && t.role !== 'assistant') {
|
||||
throw new FixtureValidationError(file, `turns[${t.turn_id}].role must be user|assistant`);
|
||||
}
|
||||
if (typeof t.text !== 'string' || !t.text) {
|
||||
throw new FixtureValidationError(file, `turns[${t.turn_id}].text required`);
|
||||
}
|
||||
if (t.ts !== undefined && (typeof t.ts !== 'string' || Number.isNaN(Date.parse(t.ts)))) {
|
||||
throw new FixtureValidationError(file, `turns[${t.turn_id}].ts must be ISO 8601`);
|
||||
}
|
||||
}
|
||||
|
||||
if (f.continuity !== undefined) {
|
||||
const c = f.continuity as Record<string, unknown>;
|
||||
assertOnlyKeys(file, c, new Set(['pair_id', 'pair_role']), 'continuity');
|
||||
if (typeof c.pair_id !== 'string' || !c.pair_id) {
|
||||
throw new FixtureValidationError(file, 'continuity.pair_id required');
|
||||
}
|
||||
if (c.pair_role !== 'writer' && c.pair_role !== 'reader') {
|
||||
throw new FixtureValidationError(file, 'continuity.pair_role must be writer|reader');
|
||||
}
|
||||
}
|
||||
|
||||
// write-back fixtures (and continuity WRITERS, which run the write-back
|
||||
// pipeline to persist their decisions) need timestamps on every turn —
|
||||
// segmentation is time-based.
|
||||
const continuityRole = (f.continuity as Record<string, unknown> | undefined)?.pair_role;
|
||||
if ((f.suites as string[]).includes('write-back') || continuityRole === 'writer') {
|
||||
for (const t of f.turns as Array<Record<string, unknown>>) {
|
||||
if (t.ts === undefined) {
|
||||
throw new FixtureValidationError(
|
||||
file,
|
||||
`write-back fixture requires ts on every turn (missing on turn_id ${t.turn_id})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return raw as BrainBenchFixture;
|
||||
}
|
||||
|
||||
export function validateGold(file: string, raw: unknown, fixture: BrainBenchFixture): FixtureGold {
|
||||
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
|
||||
throw new FixtureValidationError(file, 'gold must be a JSON object');
|
||||
}
|
||||
const g = raw as Record<string, unknown>;
|
||||
assertOnlyKeys(file, g, GOLD_KEYS, 'gold');
|
||||
if (g.fixture_id !== fixture.fixture_id) {
|
||||
throw new FixtureValidationError(
|
||||
file,
|
||||
`gold.fixture_id "${String(g.fixture_id)}" does not match fixture "${fixture.fixture_id}"`,
|
||||
);
|
||||
}
|
||||
if (typeof g.turns !== 'object' || g.turns === null) {
|
||||
throw new FixtureValidationError(file, 'gold.turns must be an object keyed by turn_id');
|
||||
}
|
||||
const turnIds = new Set(fixture.turns.map((t: FixtureTurn) => String(t.turn_id)));
|
||||
for (const [key, val] of Object.entries(g.turns as Record<string, unknown>)) {
|
||||
if (!turnIds.has(key)) {
|
||||
throw new FixtureValidationError(file, `gold.turns["${key}"] has no matching fixture turn`);
|
||||
}
|
||||
const tg = val as Record<string, unknown>;
|
||||
assertOnlyKeys(file, tg, TURN_GOLD_KEYS, `gold.turns["${key}"]`);
|
||||
if (typeof tg.should_retrieve !== 'boolean') {
|
||||
throw new FixtureValidationError(file, `gold.turns["${key}"].should_retrieve must be boolean`);
|
||||
}
|
||||
if (tg.should_retrieve && !isStringArray(tg.gold_slugs)) {
|
||||
// gold_slugs may legitimately be absent on write-back-only turns; require
|
||||
// it only when the turn participates in retrieval suites.
|
||||
const retrievalSuites = fixture.suites.some(
|
||||
(s: BrainBenchSuite) => s === 'know-to-ask' || s === 'push',
|
||||
);
|
||||
if (retrievalSuites && !tg.gold_facts) {
|
||||
throw new FixtureValidationError(
|
||||
file,
|
||||
`gold.turns["${key}"].gold_slugs required when should_retrieve=true on a retrieval-suite fixture`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (tg.gold_slugs !== undefined && !isStringArray(tg.gold_slugs)) {
|
||||
throw new FixtureValidationError(file, `gold.turns["${key}"].gold_slugs must be a string array`);
|
||||
}
|
||||
if (tg.acceptable_slugs !== undefined && !isStringArray(tg.acceptable_slugs)) {
|
||||
throw new FixtureValidationError(file, `gold.turns["${key}"].acceptable_slugs must be a string array`);
|
||||
}
|
||||
if (tg.gold_facts !== undefined) {
|
||||
if (!Array.isArray(tg.gold_facts)) {
|
||||
throw new FixtureValidationError(file, `gold.turns["${key}"].gold_facts must be an array`);
|
||||
}
|
||||
for (const gf of tg.gold_facts as Array<Record<string, unknown>>) {
|
||||
assertOnlyKeys(file, gf, GOLD_FACT_KEYS, `gold.turns["${key}"].gold_facts[]`);
|
||||
if (typeof gf.gist !== 'string' || !gf.gist) {
|
||||
throw new FixtureValidationError(file, 'gold_facts[].gist required');
|
||||
}
|
||||
if (typeof gf.fact !== 'string' || !gf.fact) {
|
||||
throw new FixtureValidationError(file, 'gold_facts[].fact required');
|
||||
}
|
||||
if (gf.entity_slug !== null && typeof gf.entity_slug !== 'string') {
|
||||
throw new FixtureValidationError(file, 'gold_facts[].entity_slug must be string or null');
|
||||
}
|
||||
if (!isStringArray(gf.match_keywords) || gf.match_keywords.length === 0) {
|
||||
throw new FixtureValidationError(file, 'gold_facts[].match_keywords must be non-empty');
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (fixture.continuity) {
|
||||
const gc = g.continuity as Record<string, unknown> | undefined;
|
||||
if (fixture.continuity.pair_role === 'reader') {
|
||||
if (!gc) throw new FixtureValidationError(file, 'reader continuity fixture requires gold.continuity');
|
||||
}
|
||||
if (gc) {
|
||||
assertOnlyKeys(file, gc, new Set(['pair_id', 'decisions']), 'gold.continuity');
|
||||
if (gc.pair_id !== fixture.continuity.pair_id) {
|
||||
throw new FixtureValidationError(file, 'gold.continuity.pair_id mismatch');
|
||||
}
|
||||
if (!Array.isArray(gc.decisions) || gc.decisions.length === 0) {
|
||||
throw new FixtureValidationError(file, 'gold.continuity.decisions must be non-empty');
|
||||
}
|
||||
for (const d of gc.decisions as Array<Record<string, unknown>>) {
|
||||
assertOnlyKeys(file, d, new Set(['decision_id', 'expected_slugs', 'match_keywords']), 'decisions[]');
|
||||
if (typeof d.decision_id !== 'string') throw new FixtureValidationError(file, 'decisions[].decision_id required');
|
||||
if (!isStringArray(d.expected_slugs)) throw new FixtureValidationError(file, 'decisions[].expected_slugs required');
|
||||
if (!isStringArray(d.match_keywords)) throw new FixtureValidationError(file, 'decisions[].match_keywords required');
|
||||
}
|
||||
}
|
||||
} else if (g.continuity !== undefined) {
|
||||
throw new FixtureValidationError(file, 'gold.continuity present but fixture has no continuity block');
|
||||
}
|
||||
|
||||
return raw as FixtureGold;
|
||||
}
|
||||
|
||||
/**
|
||||
* Load + validate the corpus. Every *.fixture.json must have a matching
|
||||
* <fixture_id>.gold.json in goldDir; orphan gold files are an error too
|
||||
* (a renamed fixture must rename its gold).
|
||||
*/
|
||||
export async function loadCorpus(fixtureDir: string, goldDir: string): Promise<LoadedCorpus> {
|
||||
let fixtureFiles: string[];
|
||||
try {
|
||||
fixtureFiles = (await readdir(fixtureDir)).filter((f) => f.endsWith('.fixture.json')).sort();
|
||||
} catch (err) {
|
||||
throw new Error(`brainbench: cannot read fixtures dir ${fixtureDir}: ${(err as Error).message}`);
|
||||
}
|
||||
let goldFiles: string[];
|
||||
try {
|
||||
goldFiles = (await readdir(goldDir)).filter((f) => f.endsWith('.gold.json')).sort();
|
||||
} catch (err) {
|
||||
throw new Error(`brainbench: cannot read gold dir ${goldDir}: ${(err as Error).message}`);
|
||||
}
|
||||
|
||||
const hash = createHash('sha256');
|
||||
const fixtures: LoadedFixture[] = [];
|
||||
const goldByFixtureId = new Map<string, { raw: unknown; file: string }>();
|
||||
|
||||
for (const gf of goldFiles) {
|
||||
const path = join(goldDir, gf);
|
||||
const content = await readFile(path, 'utf-8');
|
||||
hash.update(`gold/${gf}\n`);
|
||||
hash.update(content);
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(content);
|
||||
} catch (err) {
|
||||
throw new FixtureValidationError(gf, `invalid JSON: ${(err as Error).message}`);
|
||||
}
|
||||
const id = (parsed as Record<string, unknown>)?.fixture_id;
|
||||
if (typeof id !== 'string') throw new FixtureValidationError(gf, 'gold.fixture_id required');
|
||||
if (goldByFixtureId.has(id)) throw new FixtureValidationError(gf, `duplicate gold for fixture_id ${id}`);
|
||||
goldByFixtureId.set(id, { raw: parsed, file: gf });
|
||||
}
|
||||
|
||||
const seenIds = new Set<string>();
|
||||
for (const ff of fixtureFiles) {
|
||||
const path = join(fixtureDir, ff);
|
||||
const content = await readFile(path, 'utf-8');
|
||||
hash.update(`fixtures/${ff}\n`);
|
||||
hash.update(content);
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(content);
|
||||
} catch (err) {
|
||||
throw new FixtureValidationError(ff, `invalid JSON: ${(err as Error).message}`);
|
||||
}
|
||||
const fixture = validateFixture(ff, parsed);
|
||||
if (seenIds.has(fixture.fixture_id)) {
|
||||
throw new FixtureValidationError(ff, `duplicate fixture_id ${fixture.fixture_id}`);
|
||||
}
|
||||
seenIds.add(fixture.fixture_id);
|
||||
const goldEntry = goldByFixtureId.get(fixture.fixture_id);
|
||||
if (!goldEntry) {
|
||||
throw new FixtureValidationError(ff, `no gold file for fixture_id ${fixture.fixture_id} in ${goldDir}`);
|
||||
}
|
||||
const gold = validateGold(goldEntry.file, goldEntry.raw, fixture);
|
||||
fixtures.push({ fixture, gold, path });
|
||||
}
|
||||
|
||||
for (const [id, entry] of goldByFixtureId) {
|
||||
if (!seenIds.has(id)) {
|
||||
throw new FixtureValidationError(entry.file, `orphan gold file: no fixture with fixture_id ${id}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Continuity pairing integrity: every pair_id has exactly one writer + one reader.
|
||||
const pairs = new Map<string, { writer?: string; reader?: string }>();
|
||||
for (const { fixture } of fixtures) {
|
||||
if (!fixture.continuity) continue;
|
||||
const p = pairs.get(fixture.continuity.pair_id) ?? {};
|
||||
const role = fixture.continuity.pair_role;
|
||||
if (p[role]) {
|
||||
throw new FixtureValidationError(
|
||||
fixture.fixture_id,
|
||||
`continuity pair ${fixture.continuity.pair_id} has two ${role}s`,
|
||||
);
|
||||
}
|
||||
p[role] = fixture.fixture_id;
|
||||
pairs.set(fixture.continuity.pair_id, p);
|
||||
}
|
||||
for (const [pairId, p] of pairs) {
|
||||
if (!p.writer || !p.reader) {
|
||||
throw new FixtureValidationError(
|
||||
pairId,
|
||||
`continuity pair ${pairId} incomplete (writer=${p.writer ?? '∅'}, reader=${p.reader ?? '∅'})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
fixtures,
|
||||
fixtures_hash: hash.digest('hex'),
|
||||
fixture_dir: fixtureDir,
|
||||
gold_dir: goldDir,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,401 @@
|
||||
/**
|
||||
* BrainBench orchestrator.
|
||||
*
|
||||
* Engine economy (eng-review D9): ONE in-memory PGLite for the whole run,
|
||||
* `resetTables()` between fixtures — the longmemeval lesson; per-fixture WASM
|
||||
* cold boots would blow the <2 min CI budget. Read-only suites (know-to-ask,
|
||||
* push) seed once and run ALL adapters against the same brain; mutating work
|
||||
* (write-back) runs after the replays, and the next fixture starts from a
|
||||
* reset. A sentinel-slug test pins that sharing leaks nothing.
|
||||
*
|
||||
* Continuity pairs (writer fixture → production write-back → reader fixture)
|
||||
* run per ordered (writerHarness ≠ readerHarness) pair on a shared brain;
|
||||
* scores land on the READER's cell. With a single requested harness the
|
||||
* writer==reader diagonal runs instead (disclosed, not the headline shape).
|
||||
*
|
||||
* Sealed gold: adapters only ever receive AdapterFixtureView + PublicTurn
|
||||
* (toPublicTurn picks fields; gold never crosses).
|
||||
*/
|
||||
|
||||
import { createBenchmarkBrain, resetTables } from '../longmemeval/harness.ts';
|
||||
import type { PGLiteEngine } from '../../core/pglite-engine.ts';
|
||||
import { ClaudeCodeAdapter } from './adapters/claude-code.ts';
|
||||
import { CodexAdapter } from './adapters/codex.ts';
|
||||
import { OpenClawAdapter } from './adapters/openclaw.ts';
|
||||
import { scoreKnowToAsk } from './metrics/know-to-ask.ts';
|
||||
import { scorePush } from './metrics/push.ts';
|
||||
import { runWriteBack, type WriteBackScore } from './metrics/write-back.ts';
|
||||
import { scoreContinuityPair } from './metrics/continuity.ts';
|
||||
import { SeedError, seedBrain, type SeedOutcome } from './seed.ts';
|
||||
import {
|
||||
toPublicTurn,
|
||||
type AdapterFixtureView,
|
||||
type BrainBenchSuite,
|
||||
type HarnessAdapter,
|
||||
type HarnessName,
|
||||
type LoadedCorpus,
|
||||
type LoadedFixture,
|
||||
type SuiteMetrics,
|
||||
type TurnRow,
|
||||
} from './types.ts';
|
||||
|
||||
export interface RunBrainBenchOpts {
|
||||
harnesses: HarnessName[];
|
||||
suites: BrainBenchSuite[];
|
||||
includeHoldout: boolean;
|
||||
/** Run the real LLM extractor for write-back (budget-guarded upstream). */
|
||||
llm: boolean;
|
||||
/** Spend cap for --llm mode (threaded to the extraction pipeline's tracker). */
|
||||
budgetUsd?: number;
|
||||
/** Progress note sink (CLI wires the shared stderr reporter). */
|
||||
onProgress?: (note: string) => void;
|
||||
}
|
||||
|
||||
export interface RunBrainBenchOutput {
|
||||
cells: SuiteMetrics[];
|
||||
turn_rows: TurnRow[];
|
||||
seed_failures: Array<{ fixture_id: string; error: string }>;
|
||||
fixtures_run: number;
|
||||
}
|
||||
|
||||
function makeAdapter(name: HarnessName): HarnessAdapter {
|
||||
switch (name) {
|
||||
case 'openclaw':
|
||||
return new OpenClawAdapter();
|
||||
case 'claude-code':
|
||||
return new ClaudeCodeAdapter();
|
||||
case 'codex':
|
||||
return new CodexAdapter();
|
||||
}
|
||||
}
|
||||
|
||||
const SEAM: Record<HarnessName, 'production' | 'contract'> = {
|
||||
openclaw: 'production',
|
||||
'claude-code': 'contract',
|
||||
codex: 'contract',
|
||||
};
|
||||
|
||||
function adapterView(lf: LoadedFixture): AdapterFixtureView {
|
||||
return {
|
||||
fixture_id: lf.fixture.fixture_id,
|
||||
active_source: lf.fixture.active_source ?? 'default',
|
||||
turns: lf.fixture.turns.map(toPublicTurn),
|
||||
};
|
||||
}
|
||||
|
||||
function crossSourceSlugs(
|
||||
injected: string[],
|
||||
slugSource: Map<string, Set<string>>,
|
||||
activeSource: string,
|
||||
): string[] {
|
||||
return injected.filter((s) => {
|
||||
const set = slugSource.get(s);
|
||||
return set !== undefined && !set.has(activeSource);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Replay every USER turn of a fixture through one adapter. Assistant turns
|
||||
* feed prior context only (the production reflex fires on user messages).
|
||||
* Returns one TurnRow per (user turn × suite) so per-suite scorers and
|
||||
* external re-scorers see self-contained rows.
|
||||
*/
|
||||
async function replayFixture(
|
||||
engine: PGLiteEngine,
|
||||
adapter: HarnessAdapter,
|
||||
lf: LoadedFixture,
|
||||
seed: SeedOutcome,
|
||||
rowSuites: BrainBenchSuite[],
|
||||
): Promise<TurnRow[]> {
|
||||
const view = adapterView(lf);
|
||||
const activeSource = view.active_source;
|
||||
await adapter.beginConversation(engine, view);
|
||||
const rows: TurnRow[] = [];
|
||||
let priorContext = '';
|
||||
try {
|
||||
for (const turn of view.turns) {
|
||||
if (turn.role !== 'user') {
|
||||
priorContext += `\n${turn.text}`;
|
||||
continue;
|
||||
}
|
||||
const result = await adapter.replayTurn(turn, priorContext);
|
||||
const gold = lf.gold.turns[String(turn.turn_id)] ?? null;
|
||||
for (const suite of rowSuites) {
|
||||
rows.push({
|
||||
fixture_id: lf.fixture.fixture_id,
|
||||
turn_id: turn.turn_id,
|
||||
harness: adapter.name,
|
||||
suite,
|
||||
injected_slugs: result.injectedSlugs,
|
||||
injected_tokens: result.injectedTokens,
|
||||
gold,
|
||||
cross_source_slugs: crossSourceSlugs(result.injectedSlugs, seed.slugSource, activeSource),
|
||||
latency_ms: Math.round(result.latencyMs * 1000) / 1000,
|
||||
});
|
||||
}
|
||||
priorContext += `\n${turn.text}`;
|
||||
if (result.injectedText) priorContext += `\n${result.injectedText}`;
|
||||
}
|
||||
} finally {
|
||||
await adapter.endConversation();
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
interface WriteBackAgg {
|
||||
gold_total: number;
|
||||
gold_failed: number;
|
||||
survived: number;
|
||||
provenance_ok: number;
|
||||
stored_rows: number;
|
||||
matched_any_gold: number;
|
||||
fixtures: string[];
|
||||
failed_items: string[];
|
||||
}
|
||||
|
||||
interface ContinuityAgg {
|
||||
gold_total: number;
|
||||
gold_failed: number;
|
||||
fixtures: string[];
|
||||
failed_items: string[];
|
||||
}
|
||||
|
||||
export async function runBrainBench(
|
||||
corpus: LoadedCorpus,
|
||||
opts: RunBrainBenchOpts,
|
||||
): Promise<RunBrainBenchOutput> {
|
||||
const progress = opts.onProgress ?? (() => {});
|
||||
const turnRows: TurnRow[] = [];
|
||||
const seedFailures: Array<{ fixture_id: string; error: string }> = [];
|
||||
|
||||
const wantedSuites = new Set(opts.suites);
|
||||
const eligible = corpus.fixtures.filter((lf) => {
|
||||
if (lf.fixture.holdout && !opts.includeHoldout) return false;
|
||||
return lf.fixture.suites.some((s) => wantedSuites.has(s));
|
||||
});
|
||||
|
||||
// Continuity pairs are orchestrated separately from regular fixtures.
|
||||
const pairFixtures = new Map<string, { writer?: LoadedFixture; reader?: LoadedFixture }>();
|
||||
const regular: LoadedFixture[] = [];
|
||||
for (const lf of eligible) {
|
||||
const cont = lf.fixture.continuity;
|
||||
if (cont && wantedSuites.has('continuity')) {
|
||||
const p = pairFixtures.get(cont.pair_id) ?? {};
|
||||
p[cont.pair_role] = lf;
|
||||
pairFixtures.set(cont.pair_id, p);
|
||||
// A writer that also declares know-to-ask/push runs as a regular fixture
|
||||
// too (its retrieval gold is independent of the pair).
|
||||
if (lf.fixture.suites.some((s) => s !== 'continuity' && s !== 'write-back' && wantedSuites.has(s))) {
|
||||
regular.push(lf);
|
||||
}
|
||||
} else {
|
||||
regular.push(lf);
|
||||
}
|
||||
}
|
||||
|
||||
const writeBackAgg: WriteBackAgg = {
|
||||
gold_total: 0, gold_failed: 0, survived: 0, provenance_ok: 0,
|
||||
stored_rows: 0, matched_any_gold: 0, fixtures: [], failed_items: [],
|
||||
};
|
||||
const continuityByReader = new Map<HarnessName, ContinuityAgg>();
|
||||
|
||||
const engine = await createBenchmarkBrain();
|
||||
let fixturesRun = 0;
|
||||
try {
|
||||
// ---- regular fixtures: seed once, replay all adapters, then mutate ----
|
||||
for (const lf of regular) {
|
||||
const id = lf.fixture.fixture_id;
|
||||
progress(`fixture ${id}`);
|
||||
await resetTables(engine);
|
||||
let seed: SeedOutcome;
|
||||
try {
|
||||
seed = await seedBrain(engine, lf.fixture);
|
||||
} catch (err) {
|
||||
if (err instanceof SeedError) {
|
||||
seedFailures.push({ fixture_id: id, error: err.message });
|
||||
continue;
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
fixturesRun++;
|
||||
|
||||
const retrievalSuites = lf.fixture.suites.filter(
|
||||
(s): s is BrainBenchSuite => (s === 'know-to-ask' || s === 'push') && wantedSuites.has(s),
|
||||
);
|
||||
if (retrievalSuites.length > 0) {
|
||||
for (const harness of opts.harnesses) {
|
||||
const adapter = makeAdapter(harness);
|
||||
const rows = await replayFixture(engine, adapter, lf, seed, retrievalSuites);
|
||||
turnRows.push(...rows);
|
||||
}
|
||||
}
|
||||
|
||||
if (lf.fixture.suites.includes('write-back') && wantedSuites.has('write-back')) {
|
||||
const score = await runWriteBack(engine, lf.fixture, lf.gold, {
|
||||
llm: opts.llm,
|
||||
budgetUsd: opts.budgetUsd,
|
||||
});
|
||||
accumulateWriteBack(writeBackAgg, id, score);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- continuity pairs ----
|
||||
const pairHarnesses: Array<[HarnessName, HarnessName]> = [];
|
||||
if (opts.harnesses.length === 1) {
|
||||
pairHarnesses.push([opts.harnesses[0], opts.harnesses[0]]);
|
||||
} else {
|
||||
for (const w of opts.harnesses) {
|
||||
for (const r of opts.harnesses) {
|
||||
if (w !== r) pairHarnesses.push([w, r]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const [pairId, pair] of pairFixtures) {
|
||||
if (!pair.writer || !pair.reader) continue; // loader validates; belt+suspenders
|
||||
const writer = pair.writer;
|
||||
const reader = pair.reader;
|
||||
const decisions = reader.gold.continuity?.decisions ?? [];
|
||||
if (!decisions.length) continue;
|
||||
|
||||
for (const [writerHarness, readerHarness] of pairHarnesses) {
|
||||
progress(`continuity ${pairId} ${writerHarness}→${readerHarness}`);
|
||||
await resetTables(engine);
|
||||
let writerSeed: SeedOutcome;
|
||||
let readerSeed: SeedOutcome;
|
||||
try {
|
||||
writerSeed = await seedBrain(engine, writer.fixture);
|
||||
readerSeed = await seedBrain(engine, reader.fixture);
|
||||
} catch (err) {
|
||||
if (err instanceof SeedError) {
|
||||
seedFailures.push({ fixture_id: `${pairId} (${writerHarness}→${readerHarness})`, error: err.message });
|
||||
continue;
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
|
||||
// Writer conversation replays through ITS harness (suppression state
|
||||
// realistic), then its decisions persist via the production pipeline.
|
||||
const writerAdapter = makeAdapter(writerHarness);
|
||||
await replayFixture(engine, writerAdapter, writer, writerSeed, []);
|
||||
await runWriteBack(engine, writer.fixture, writer.gold, {
|
||||
llm: opts.llm,
|
||||
budgetUsd: opts.budgetUsd,
|
||||
});
|
||||
|
||||
// Reader replays on the SAME brain through a different harness.
|
||||
const readerAdapter = makeAdapter(readerHarness);
|
||||
const readerRows = await replayFixture(engine, readerAdapter, reader, readerSeed, ['continuity']);
|
||||
turnRows.push(...readerRows);
|
||||
|
||||
const activeSource = reader.fixture.active_source ?? 'default';
|
||||
const score = await scoreContinuityPair(engine, activeSource, pairId, readerRows, decisions);
|
||||
|
||||
const agg = continuityByReader.get(readerHarness) ?? {
|
||||
gold_total: 0, gold_failed: 0, fixtures: [], failed_items: [],
|
||||
};
|
||||
agg.gold_total += score.gold_total;
|
||||
agg.gold_failed += score.gold_failed;
|
||||
if (!agg.fixtures.includes(reader.fixture.fixture_id)) agg.fixtures.push(reader.fixture.fixture_id);
|
||||
agg.failed_items.push(...score.failed_items.map((f) => `${f} [${writerHarness}→${readerHarness}]`));
|
||||
continuityByReader.set(readerHarness, agg);
|
||||
}
|
||||
fixturesRun += 2;
|
||||
}
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
|
||||
// ---- assemble cells ----
|
||||
const cells: SuiteMetrics[] = [];
|
||||
for (const harness of opts.harnesses) {
|
||||
for (const suite of opts.suites) {
|
||||
const cell = assembleCell(harness, suite, turnRows, writeBackAgg, continuityByReader);
|
||||
if (cell) cells.push(cell);
|
||||
}
|
||||
}
|
||||
|
||||
return { cells, turn_rows: turnRows, seed_failures: seedFailures, fixtures_run: fixturesRun };
|
||||
}
|
||||
|
||||
function accumulateWriteBack(agg: WriteBackAgg, fixtureId: string, score: WriteBackScore): void {
|
||||
agg.gold_total += score.gold_total;
|
||||
agg.gold_failed += score.gold_failed;
|
||||
agg.survived += score.survived;
|
||||
agg.provenance_ok += score.provenance_ok;
|
||||
agg.stored_rows += score.stored_rows;
|
||||
agg.fixtures.push(fixtureId);
|
||||
agg.failed_items.push(...score.failed_items);
|
||||
}
|
||||
|
||||
function round4(n: number): number {
|
||||
return Math.round(n * 10000) / 10000;
|
||||
}
|
||||
|
||||
function assembleCell(
|
||||
harness: HarnessName,
|
||||
suite: BrainBenchSuite,
|
||||
turnRows: TurnRow[],
|
||||
writeBackAgg: WriteBackAgg,
|
||||
continuityByReader: Map<HarnessName, ContinuityAgg>,
|
||||
): SuiteMetrics | null {
|
||||
if (suite === 'write-back') {
|
||||
if (writeBackAgg.fixtures.length === 0) return null;
|
||||
// The write path is gbrain's pipeline — identical for every harness seam
|
||||
// in v1, so each harness cell carries the same (once-computed) numbers.
|
||||
// When harness-specific write paths land, this is where they diverge.
|
||||
return {
|
||||
suite, harness, seam: SEAM[harness],
|
||||
gold_total: writeBackAgg.gold_total,
|
||||
gold_failed: writeBackAgg.gold_failed,
|
||||
metrics: {
|
||||
write_back_fidelity: round4(
|
||||
writeBackAgg.gold_total > 0 ? writeBackAgg.survived / writeBackAgg.gold_total : 1,
|
||||
),
|
||||
provenance_accuracy: round4(
|
||||
writeBackAgg.survived > 0 ? writeBackAgg.provenance_ok / writeBackAgg.survived : 1,
|
||||
),
|
||||
},
|
||||
fixtures: [...writeBackAgg.fixtures],
|
||||
};
|
||||
}
|
||||
|
||||
if (suite === 'continuity') {
|
||||
const agg = continuityByReader.get(harness);
|
||||
if (!agg) return null;
|
||||
const rows = turnRows.filter((r) => r.harness === harness && r.suite === 'continuity');
|
||||
return {
|
||||
suite, harness, seam: SEAM[harness],
|
||||
gold_total: agg.gold_total,
|
||||
gold_failed: agg.gold_failed,
|
||||
metrics: {
|
||||
continuity_rate: round4(
|
||||
agg.gold_total > 0 ? (agg.gold_total - agg.gold_failed) / agg.gold_total : 1,
|
||||
),
|
||||
source_isolation_violations: rows.reduce((n, r) => n + r.cross_source_slugs.length, 0),
|
||||
avg_injected_tokens: round4(avg(rows.map((r) => r.injected_tokens))),
|
||||
},
|
||||
fixtures: [...agg.fixtures],
|
||||
};
|
||||
}
|
||||
|
||||
const rows = turnRows.filter((r) => r.harness === harness && r.suite === suite);
|
||||
if (rows.length === 0) return null;
|
||||
const score = suite === 'know-to-ask' ? scoreKnowToAsk(rows) : scorePush(rows);
|
||||
const fixtures = [...new Set(rows.map((r) => r.fixture_id))];
|
||||
return {
|
||||
suite, harness, seam: SEAM[harness],
|
||||
gold_total: score.gold_total,
|
||||
gold_failed: score.gold_failed,
|
||||
metrics: {
|
||||
...Object.fromEntries(Object.entries(score.metrics).map(([k, v]) => [k, round4(v)])),
|
||||
source_isolation_violations: rows.reduce((n, r) => n + r.cross_source_slugs.length, 0),
|
||||
avg_injected_tokens: round4(avg(rows.map((r) => r.injected_tokens))),
|
||||
},
|
||||
fixtures,
|
||||
};
|
||||
}
|
||||
|
||||
function avg(ns: number[]): number {
|
||||
return ns.length === 0 ? 0 : ns.reduce((a, b) => a + b, 0) / ns.length;
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* BrainBench cross-session continuity scoring.
|
||||
*
|
||||
* A continuity pair runs writer fixture → (production write-back pipeline) →
|
||||
* reader fixture on the SAME brain, with writer and reader replayed through
|
||||
* DIFFERENT harness adapters and distinct session identities (decision 14's
|
||||
* session-boundary requirement is carried by the write path's per-page
|
||||
* source_session vs the reader's fresh conversation).
|
||||
*
|
||||
* A decision probe succeeds when the reader's replay either injected one of
|
||||
* the expected slugs, or the decision fact persisted by the writer is
|
||||
* recallable by keyword probe within the active source. Headline
|
||||
* continuity_rate per harness = mean over pairs where that harness READ.
|
||||
*/
|
||||
|
||||
import type { PGLiteEngine } from '../../../core/pglite-engine.ts';
|
||||
import type { ContinuityDecisionGold, TurnRow } from '../types.ts';
|
||||
|
||||
export interface ContinuityPairScore {
|
||||
gold_total: number;
|
||||
gold_failed: number;
|
||||
/** decision_id → hit. */
|
||||
hits: Record<string, boolean>;
|
||||
failed_items: string[];
|
||||
}
|
||||
|
||||
export async function scoreContinuityPair(
|
||||
engine: PGLiteEngine,
|
||||
activeSource: string,
|
||||
pairId: string,
|
||||
readerRows: TurnRow[],
|
||||
decisions: ContinuityDecisionGold[],
|
||||
): Promise<ContinuityPairScore> {
|
||||
const injected = new Set<string>();
|
||||
for (const row of readerRows) for (const s of row.injected_slugs) injected.add(s);
|
||||
|
||||
const hits: Record<string, boolean> = {};
|
||||
const failed: string[] = [];
|
||||
let failedCount = 0;
|
||||
|
||||
for (const d of decisions) {
|
||||
let hit = d.expected_slugs.some((slug) => injected.has(slug));
|
||||
if (!hit && d.match_keywords.length > 0) {
|
||||
hit = await factKeywordProbe(engine, activeSource, d.match_keywords);
|
||||
}
|
||||
hits[d.decision_id] = hit;
|
||||
if (!hit) {
|
||||
failedCount++;
|
||||
failed.push(`${pairId}/${d.decision_id} (decision not recalled)`);
|
||||
}
|
||||
}
|
||||
|
||||
return { gold_total: decisions.length, gold_failed: failedCount, hits, failed_items: failed };
|
||||
}
|
||||
|
||||
/** True when an active (non-expired) fact in the source contains every keyword. */
|
||||
export async function factKeywordProbe(
|
||||
engine: PGLiteEngine,
|
||||
sourceId: string,
|
||||
keywords: string[],
|
||||
): Promise<boolean> {
|
||||
const conds = keywords.map((_, i) => `fact ILIKE $${i + 2}`).join(' AND ');
|
||||
const params = [sourceId, ...keywords.map((kw) => `%${kw}%`)];
|
||||
const rows = await engine.executeRaw<{ one: number }>(
|
||||
`SELECT 1 AS one FROM facts
|
||||
WHERE source_id = $1 AND expired_at IS NULL AND ${conds}
|
||||
LIMIT 1`,
|
||||
params,
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* BrainBench know-to-ask scoring — pure over TurnRow[].
|
||||
*
|
||||
* know_to_ask_failure_rate (lower better): of turns where gold says the
|
||||
* memory layer SHOULD have surfaced something, the fraction where the
|
||||
* injection hit neither gold nor acceptable slugs. This grades the
|
||||
* deterministic injection decision — the mechanism gbrain ships at the seam
|
||||
* (decision 3); agent-LLM-in-the-loop replay is the pre-registered --live
|
||||
* extension.
|
||||
*
|
||||
* false_fire_rate (lower better, anti-gaming companion): of turns where gold
|
||||
* says STAY SILENT, the fraction where anything was injected. Without it,
|
||||
* "always inject" games the failure rate.
|
||||
*/
|
||||
|
||||
import type { TurnRow } from '../types.ts';
|
||||
|
||||
export interface KnowToAskScore {
|
||||
gold_total: number;
|
||||
gold_failed: number;
|
||||
metrics: Record<string, number>;
|
||||
/** `${fixture_id}#${turn_id}` of each failed gold item (breach reporting). */
|
||||
failed_items: string[];
|
||||
}
|
||||
|
||||
export function scoreKnowToAsk(rows: TurnRow[]): KnowToAskScore {
|
||||
let shouldRetrieve = 0;
|
||||
let missed = 0;
|
||||
let quiet = 0;
|
||||
let falseFires = 0;
|
||||
const failed: string[] = [];
|
||||
|
||||
for (const row of rows) {
|
||||
if (!row.gold) continue;
|
||||
const key = `${row.fixture_id}#${row.turn_id}`;
|
||||
if (row.gold.should_retrieve) {
|
||||
shouldRetrieve++;
|
||||
const ok = new Set([...(row.gold.gold_slugs ?? []), ...(row.gold.acceptable_slugs ?? [])]);
|
||||
const hit = row.injected_slugs.some((s) => ok.has(s));
|
||||
if (!hit) {
|
||||
missed++;
|
||||
failed.push(`${key} (missed retrieve)`);
|
||||
}
|
||||
} else {
|
||||
quiet++;
|
||||
if (row.injected_slugs.length > 0) {
|
||||
falseFires++;
|
||||
failed.push(`${key} (false fire: ${row.injected_slugs.join(',')})`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
gold_total: shouldRetrieve + quiet,
|
||||
gold_failed: missed + falseFires,
|
||||
metrics: {
|
||||
know_to_ask_failure_rate: shouldRetrieve > 0 ? missed / shouldRetrieve : 0,
|
||||
false_fire_rate: quiet > 0 ? falseFires / quiet : 0,
|
||||
},
|
||||
failed_items: failed,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
/**
|
||||
* BrainBench push precision/recall — pure over TurnRow[], micro-averaged.
|
||||
*
|
||||
* push_precision = Σ|injected ∩ (gold ∪ acceptable)| / Σ|injected|
|
||||
* over turns with non-empty injection. Acceptable slugs count for precision
|
||||
* (injecting them isn't noise) but not for recall (they're not required).
|
||||
*
|
||||
* push_recall = Σ|injected ∩ gold| / Σ|gold|
|
||||
* over should_retrieve turns.
|
||||
*
|
||||
* Micro-averaging (sum-based, not per-turn means) per the plan's formulas —
|
||||
* a 3-slug turn weighs three times a 1-slug turn, which is what a token
|
||||
* budget actually experiences.
|
||||
*/
|
||||
|
||||
import type { TurnRow } from '../types.ts';
|
||||
|
||||
export interface PushScore {
|
||||
gold_total: number;
|
||||
gold_failed: number;
|
||||
metrics: Record<string, number>;
|
||||
failed_items: string[];
|
||||
}
|
||||
|
||||
export function scorePush(rows: TurnRow[]): PushScore {
|
||||
let injectedTotal = 0;
|
||||
let injectedRelevant = 0;
|
||||
let goldTotal = 0;
|
||||
let goldHit = 0;
|
||||
const failed: string[] = [];
|
||||
|
||||
for (const row of rows) {
|
||||
if (!row.gold) continue;
|
||||
const gold = new Set(row.gold.gold_slugs ?? []);
|
||||
const acceptable = new Set([...gold, ...(row.gold.acceptable_slugs ?? [])]);
|
||||
|
||||
if (row.injected_slugs.length > 0) {
|
||||
injectedTotal += row.injected_slugs.length;
|
||||
injectedRelevant += row.injected_slugs.filter((s) => acceptable.has(s)).length;
|
||||
}
|
||||
if (row.gold.should_retrieve && gold.size > 0) {
|
||||
goldTotal += gold.size;
|
||||
const injected = new Set(row.injected_slugs);
|
||||
for (const slug of gold) {
|
||||
if (injected.has(slug)) {
|
||||
goldHit++;
|
||||
} else {
|
||||
failed.push(`${row.fixture_id}#${row.turn_id} (gold slug not pushed: ${slug})`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
gold_total: goldTotal,
|
||||
gold_failed: goldTotal - goldHit,
|
||||
metrics: {
|
||||
push_precision: injectedTotal > 0 ? injectedRelevant / injectedTotal : 1,
|
||||
push_recall: goldTotal > 0 ? goldHit / goldTotal : 1,
|
||||
},
|
||||
failed_items: failed,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
/**
|
||||
* BrainBench write-back fidelity — grades the PRODUCTION conversation→memory
|
||||
* pipeline (decision 15), not a bench-only path.
|
||||
*
|
||||
* Flow per fixture:
|
||||
* 1. Render the fixture's turns into a conversation page in the
|
||||
* imessage-slack line format the conversation-parser ships
|
||||
* (`**Speaker** (YYYY-MM-DD H:MM AM): text`), import it (noEmbed).
|
||||
* 2. Run `runExtractConversationFactsCore` over that page with an injected
|
||||
* GOLD extractor: for each segment, it emits exactly the gold facts whose
|
||||
* source turn text appears in the segment. Zero LLM calls; segmentation,
|
||||
* insertFacts batching, dedup, provenance stamping, terminal-audit rows
|
||||
* all execute the shipped code.
|
||||
* With opts.llm=true the injection is skipped — the real Haiku extractor
|
||||
* runs and extraction_recall / extraction_precision are additionally
|
||||
* scored against the gold keyword probes.
|
||||
* 3. Read back via the facts table and score:
|
||||
* write_back_fidelity — gold facts that survived (keyword probe) with
|
||||
* the right entity attribution
|
||||
* provenance_accuracy — surviving facts carrying correct
|
||||
* {source, source_session, source_markdown_slug}
|
||||
*/
|
||||
|
||||
import type { ExtractInput, ExtractedFact } from '../../../core/facts/extract.ts';
|
||||
import {
|
||||
PER_SEGMENT_SOURCE_PREFIX,
|
||||
TERMINAL_AUDIT_SOURCE,
|
||||
runExtractConversationFactsCore,
|
||||
} from '../../../commands/extract-conversation-facts.ts';
|
||||
import { importFromContent } from '../../../core/import-file.ts';
|
||||
import type { PGLiteEngine } from '../../../core/pglite-engine.ts';
|
||||
import type { BrainBenchFixture, FixtureGold, GoldFactSpec } from '../types.ts';
|
||||
import { SeedError } from '../seed.ts';
|
||||
|
||||
export interface WriteBackScore {
|
||||
gold_total: number;
|
||||
gold_failed: number;
|
||||
/** Raw counters so the harness can aggregate across fixtures (Σ-based). */
|
||||
survived: number;
|
||||
provenance_ok: number;
|
||||
/** Total non-audit rows stored (extraction_precision denominator in --llm mode). */
|
||||
stored_rows: number;
|
||||
metrics: Record<string, number>;
|
||||
failed_items: string[];
|
||||
}
|
||||
|
||||
/** Slug the rendered conversation page imports under. */
|
||||
export function conversationSlug(fixtureId: string): string {
|
||||
return `conversations/bench-${fixtureId}`;
|
||||
}
|
||||
|
||||
/** Render a fixture turn timestamp as the imessage-slack inline form (UTC). */
|
||||
function renderInlineTime(iso: string): string {
|
||||
const d = new Date(iso);
|
||||
const date = iso.slice(0, 10);
|
||||
let h = d.getUTCHours();
|
||||
const ampm = h >= 12 ? 'PM' : 'AM';
|
||||
h = h % 12;
|
||||
if (h === 0) h = 12;
|
||||
const mm = String(d.getUTCMinutes()).padStart(2, '0');
|
||||
return `(${date} ${h}:${mm} ${ampm})`;
|
||||
}
|
||||
|
||||
export function renderConversationPage(fixture: BrainBenchFixture): string {
|
||||
const lines: string[] = [
|
||||
'---',
|
||||
`title: Bench conversation ${fixture.fixture_id}`,
|
||||
'type: conversation',
|
||||
'---',
|
||||
'',
|
||||
];
|
||||
for (const turn of fixture.turns) {
|
||||
if (!turn.ts) {
|
||||
throw new Error(`write-back fixture ${fixture.fixture_id} turn ${turn.turn_id} missing ts`);
|
||||
}
|
||||
const speaker = turn.role === 'user' ? 'You' : 'Assistant';
|
||||
lines.push(`**${speaker}** ${renderInlineTime(turn.ts)}: ${turn.text}`);
|
||||
}
|
||||
lines.push('');
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/** All gold facts of a fixture, with the turn text that produces each. */
|
||||
function goldFactsWithSourceText(
|
||||
fixture: BrainBenchFixture,
|
||||
gold: FixtureGold,
|
||||
): Array<{ turnText: string; spec: GoldFactSpec }> {
|
||||
const out: Array<{ turnText: string; spec: GoldFactSpec }> = [];
|
||||
for (const turn of fixture.turns) {
|
||||
const tg = gold.turns[String(turn.turn_id)];
|
||||
if (!tg?.gold_facts) continue;
|
||||
for (const spec of tg.gold_facts) out.push({ turnText: turn.text, spec });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic gold extractor: emits the gold facts whose source turn text
|
||||
* appears verbatim inside the segment the production pipeline hands it.
|
||||
*/
|
||||
export function makeGoldExtractor(
|
||||
fixture: BrainBenchFixture,
|
||||
gold: FixtureGold,
|
||||
): (input: ExtractInput) => Promise<ExtractedFact[]> {
|
||||
const items = goldFactsWithSourceText(fixture, gold);
|
||||
return async (input: ExtractInput): Promise<ExtractedFact[]> => {
|
||||
const out: ExtractedFact[] = [];
|
||||
for (const { turnText, spec } of items) {
|
||||
if (!input.turnText.includes(turnText)) continue;
|
||||
out.push({
|
||||
fact: spec.fact,
|
||||
kind: spec.kind ?? 'fact',
|
||||
entity_slug: spec.entity_slug,
|
||||
source: input.source,
|
||||
confidence: 1.0,
|
||||
notability: 'medium',
|
||||
embedding: null,
|
||||
});
|
||||
}
|
||||
return out;
|
||||
};
|
||||
}
|
||||
|
||||
interface StoredFactRow {
|
||||
fact: string;
|
||||
entity_slug: string | null;
|
||||
source: string;
|
||||
source_session: string | null;
|
||||
source_markdown_slug: string | null;
|
||||
}
|
||||
|
||||
export async function runWriteBack(
|
||||
engine: PGLiteEngine,
|
||||
fixture: BrainBenchFixture,
|
||||
gold: FixtureGold,
|
||||
opts: { llm: boolean; budgetUsd?: number },
|
||||
): Promise<WriteBackScore> {
|
||||
const sourceId = fixture.active_source ?? 'default';
|
||||
const slug = conversationSlug(fixture.fixture_id);
|
||||
|
||||
const body = renderConversationPage(fixture);
|
||||
const imported = await importFromContent(engine, slug, body, { noEmbed: true, sourceId });
|
||||
if (imported.status !== 'imported') {
|
||||
throw new SeedError(
|
||||
fixture.fixture_id,
|
||||
slug,
|
||||
`conversation page import status=${imported.status}${imported.error ? ` (${imported.error})` : ''}`,
|
||||
);
|
||||
}
|
||||
|
||||
await runExtractConversationFactsCore(engine, {
|
||||
sourceId,
|
||||
slug,
|
||||
types: ['conversation'],
|
||||
overrideDisabled: true,
|
||||
sleepMs: 0,
|
||||
// Deterministic CI mode injects the gold extractor; --llm runs the real
|
||||
// one under the caller's budget cap (cost-guard pattern, decision 2).
|
||||
...(opts.llm
|
||||
? { maxCostUsd: opts.budgetUsd }
|
||||
: { extractor: makeGoldExtractor(fixture, gold) }),
|
||||
});
|
||||
|
||||
const stored = await engine.executeRaw<StoredFactRow>(
|
||||
`SELECT fact, entity_slug, source, source_session, source_markdown_slug
|
||||
FROM facts
|
||||
WHERE source_id = $1
|
||||
AND source_markdown_slug = $2
|
||||
AND source <> $3`,
|
||||
[sourceId, slug, TERMINAL_AUDIT_SOURCE],
|
||||
);
|
||||
|
||||
const expectedSession = `${PER_SEGMENT_SOURCE_PREFIX}:${slug}`;
|
||||
const goldItems = goldFactsWithSourceText(fixture, gold);
|
||||
|
||||
let survived = 0;
|
||||
let provenanceOk = 0;
|
||||
const failed: string[] = [];
|
||||
for (const { spec } of goldItems) {
|
||||
const match = stored.find(
|
||||
(row) =>
|
||||
spec.match_keywords.every((kw) => row.fact.toLowerCase().includes(kw.toLowerCase())) &&
|
||||
(spec.entity_slug === null || row.entity_slug === spec.entity_slug),
|
||||
);
|
||||
if (!match) {
|
||||
failed.push(`${fixture.fixture_id} (gold fact lost: ${spec.gist})`);
|
||||
continue;
|
||||
}
|
||||
survived++;
|
||||
if (
|
||||
match.source === PER_SEGMENT_SOURCE_PREFIX &&
|
||||
match.source_session === expectedSession &&
|
||||
match.source_markdown_slug === slug
|
||||
) {
|
||||
provenanceOk++;
|
||||
} else {
|
||||
failed.push(`${fixture.fixture_id} (provenance wrong on: ${spec.gist})`);
|
||||
}
|
||||
}
|
||||
|
||||
const metrics: Record<string, number> = {
|
||||
write_back_fidelity: goldItems.length > 0 ? survived / goldItems.length : 1,
|
||||
provenance_accuracy: survived > 0 ? provenanceOk / survived : 1,
|
||||
};
|
||||
|
||||
if (opts.llm) {
|
||||
// Extraction quality vs gold, only meaningful when the real extractor ran.
|
||||
const matchedAnyGold = stored.filter((row) =>
|
||||
goldItems.some(({ spec }) =>
|
||||
spec.match_keywords.every((kw) => row.fact.toLowerCase().includes(kw.toLowerCase())),
|
||||
),
|
||||
).length;
|
||||
metrics.extraction_recall = goldItems.length > 0 ? survived / goldItems.length : 1;
|
||||
metrics.extraction_precision = stored.length > 0 ? matchedAnyGold / stored.length : 1;
|
||||
}
|
||||
|
||||
// gold_failed counts BOTH lost facts and provenance failures — a fact that
|
||||
// survived with wrong provenance is a failed gold item for the count gate.
|
||||
return {
|
||||
gold_total: goldItems.length,
|
||||
gold_failed: goldItems.length - survived + (survived - provenanceOk),
|
||||
survived,
|
||||
provenance_ok: provenanceOk,
|
||||
stored_rows: stored.length,
|
||||
metrics,
|
||||
failed_items: failed,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
/**
|
||||
* BrainBench scoreboard: markdown render, canonical committed baseline, and
|
||||
* the compare gate with main-baseline governance (decisions 4, 8, 10).
|
||||
*
|
||||
* Gate semantics (deterministic corpus ⇒ any flip is a real behavior change):
|
||||
* same fixtures_hash → count-aware gate: any cell whose gold_failed rose,
|
||||
* any adverse gated-metric move, or ANY
|
||||
* source_isolation_violations > 0 is a breach.
|
||||
* different hash → corpus-bless mode: the committed baseline in the
|
||||
* working tree must EXACTLY match the current run
|
||||
* (the file can't lie); adverse moves vs main's
|
||||
* baseline additionally require a `justification`
|
||||
* string in the committed baseline.
|
||||
* --allow-regression → local/one-off escape hatch; reason recorded in the
|
||||
* outcome notes (and the run output).
|
||||
*
|
||||
* The committed baseline is diff-stable by construction: metrics rounded to 4
|
||||
* decimals, keys sorted, receipts excluded (decision 10).
|
||||
*/
|
||||
|
||||
import type {
|
||||
BrainBenchBaseline,
|
||||
BrainBenchResult,
|
||||
CompareOutcome,
|
||||
SuiteMetrics,
|
||||
} from './types.ts';
|
||||
|
||||
/** Gated metrics + their good direction. Anything absent is diagnostic-only. */
|
||||
export const GATED_METRICS: Readonly<Record<string, 'lower' | 'higher'>> = {
|
||||
know_to_ask_failure_rate: 'lower',
|
||||
false_fire_rate: 'lower',
|
||||
source_isolation_violations: 'lower',
|
||||
push_precision: 'higher',
|
||||
push_recall: 'higher',
|
||||
write_back_fidelity: 'higher',
|
||||
provenance_accuracy: 'higher',
|
||||
continuity_rate: 'higher',
|
||||
extraction_recall: 'higher',
|
||||
extraction_precision: 'higher',
|
||||
};
|
||||
|
||||
const BASELINE_SCHEMA_VERSION = 1;
|
||||
|
||||
function round4(n: number): number {
|
||||
return Math.round(n * 10000) / 10000;
|
||||
}
|
||||
|
||||
function cellKey(c: SuiteMetrics): string {
|
||||
return `${c.harness}/${c.suite}`;
|
||||
}
|
||||
|
||||
function sortedRecord<T>(entries: Array<[string, T]>): Record<string, T> {
|
||||
const out: Record<string, T> = {};
|
||||
for (const [k, v] of entries.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))) out[k] = v;
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Build the canonical, diff-stable committed-baseline shape from a run. */
|
||||
export function toCanonicalBaseline(
|
||||
result: Pick<BrainBenchResult, 'cells'> & { receipt: { fixtures_hash: string } },
|
||||
justification?: string,
|
||||
): BrainBenchBaseline {
|
||||
const cells: Array<[string, Record<string, number>]> = [];
|
||||
const counts: Array<[string, { gold_total: number; gold_failed: number }]> = [];
|
||||
for (const c of result.cells) {
|
||||
cells.push([
|
||||
cellKey(c),
|
||||
sortedRecord(Object.entries(c.metrics).map(([k, v]) => [k, round4(v)] as [string, number])),
|
||||
]);
|
||||
counts.push([cellKey(c), { gold_total: c.gold_total, gold_failed: c.gold_failed }]);
|
||||
}
|
||||
const baseline: BrainBenchBaseline = {
|
||||
schema_version: BASELINE_SCHEMA_VERSION,
|
||||
fixtures_hash: result.receipt.fixtures_hash,
|
||||
cells: sortedRecord(cells),
|
||||
counts: sortedRecord(counts),
|
||||
};
|
||||
if (justification) baseline.justification = justification;
|
||||
return baseline;
|
||||
}
|
||||
|
||||
/** Deterministic serialization for the committed file + bless-mode equality. */
|
||||
export function serializeBaseline(b: BrainBenchBaseline): string {
|
||||
// Field order is fixed by construction; metrics/cells already sorted.
|
||||
const ordered: Record<string, unknown> = {
|
||||
schema_version: b.schema_version,
|
||||
fixtures_hash: b.fixtures_hash,
|
||||
};
|
||||
if (b.justification) ordered.justification = b.justification;
|
||||
ordered.cells = b.cells;
|
||||
ordered.counts = b.counts;
|
||||
return JSON.stringify(ordered, null, 2) + '\n';
|
||||
}
|
||||
|
||||
export function parseBaseline(raw: string, file: string): BrainBenchBaseline {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch (err) {
|
||||
throw new Error(`brainbench baseline ${file}: invalid JSON (${(err as Error).message})`);
|
||||
}
|
||||
const b = parsed as Partial<BrainBenchBaseline>;
|
||||
if (b.schema_version !== BASELINE_SCHEMA_VERSION) {
|
||||
throw new Error(`brainbench baseline ${file}: schema_version must be ${BASELINE_SCHEMA_VERSION}`);
|
||||
}
|
||||
if (typeof b.fixtures_hash !== 'string' || !b.cells || !b.counts) {
|
||||
throw new Error(`brainbench baseline ${file}: missing fixtures_hash/cells/counts`);
|
||||
}
|
||||
return b as BrainBenchBaseline;
|
||||
}
|
||||
|
||||
export interface CompareOpts {
|
||||
/** Local escape hatch — reason recorded, breaches downgraded to notes. */
|
||||
allowRegression?: string;
|
||||
/**
|
||||
* Corpus-bless verification target: the committed baseline from the WORKING
|
||||
* TREE (decision 4). Required for bless mode to pass; must exactly match the
|
||||
* current run.
|
||||
*/
|
||||
committedBaseline?: BrainBenchBaseline | null;
|
||||
}
|
||||
|
||||
export function compareBaselines(
|
||||
current: BrainBenchBaseline,
|
||||
main: BrainBenchBaseline,
|
||||
opts: CompareOpts = {},
|
||||
): CompareOutcome {
|
||||
const breaches: CompareOutcome['breaches'] = [];
|
||||
const notes: string[] = [];
|
||||
|
||||
// Source isolation gates at zero regardless of what any baseline says.
|
||||
for (const [cell, metrics] of Object.entries(current.cells)) {
|
||||
const v = metrics.source_isolation_violations;
|
||||
if (v !== undefined && v > 0) {
|
||||
breaches.push({
|
||||
cell,
|
||||
metric: 'source_isolation_violations',
|
||||
baseline: 0,
|
||||
current: v,
|
||||
detail: 'cross-source injection — gates at zero (data-leak invariant)',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const sameHash = current.fixtures_hash === main.fixtures_hash;
|
||||
const mode: CompareOutcome['mode'] = sameHash ? 'same-hash' : 'corpus-bless';
|
||||
|
||||
// Adverse metric/count movement vs main's baseline (both modes; in bless
|
||||
// mode it's what the justification must answer for).
|
||||
for (const [cell, mainCounts] of Object.entries(main.counts)) {
|
||||
const cur = current.counts[cell];
|
||||
if (!cur) {
|
||||
breaches.push({
|
||||
cell,
|
||||
metric: 'gold_failed',
|
||||
baseline: mainCounts.gold_failed,
|
||||
current: NaN,
|
||||
detail: 'cell missing from current run (suite/harness coverage disappeared)',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
if (sameHash && cur.gold_failed > mainCounts.gold_failed) {
|
||||
breaches.push({
|
||||
cell,
|
||||
metric: 'gold_failed',
|
||||
baseline: mainCounts.gold_failed,
|
||||
current: cur.gold_failed,
|
||||
detail: `${cur.gold_failed - mainCounts.gold_failed} newly-failed gold item(s)`,
|
||||
});
|
||||
}
|
||||
}
|
||||
for (const [cell, mainMetrics] of Object.entries(main.cells)) {
|
||||
const curMetrics = current.cells[cell];
|
||||
if (!curMetrics) continue; // covered by the counts check above
|
||||
for (const [metric, direction] of Object.entries(GATED_METRICS)) {
|
||||
const was = mainMetrics[metric];
|
||||
const now = curMetrics[metric];
|
||||
if (was === undefined || now === undefined) continue;
|
||||
if (metric === 'source_isolation_violations') continue; // gated at zero above
|
||||
const adverse = direction === 'lower' ? now > was : now < was;
|
||||
if (adverse) {
|
||||
breaches.push({
|
||||
cell,
|
||||
metric,
|
||||
baseline: was,
|
||||
current: now,
|
||||
detail: `${direction === 'lower' ? 'rose' : 'fell'} ${round4(Math.abs(now - was))}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (mode === 'corpus-bless') {
|
||||
notes.push(
|
||||
`fixtures_hash changed (${main.fixtures_hash.slice(0, 12)} → ${current.fixtures_hash.slice(0, 12)}) — corpus-bless mode`,
|
||||
);
|
||||
const committed = opts.committedBaseline;
|
||||
if (!committed) {
|
||||
return {
|
||||
verdict: 'inconclusive',
|
||||
mode,
|
||||
breaches,
|
||||
notes: [
|
||||
...notes,
|
||||
'no committed baseline to verify — run `gbrain eval brainbench --update-baseline` and commit the result',
|
||||
],
|
||||
};
|
||||
}
|
||||
// The committed file must match the actual run (it can't lie).
|
||||
const runSer = serializeBaseline({ ...current, justification: committed.justification });
|
||||
const committedSer = serializeBaseline(committed);
|
||||
if (runSer !== committedSer) {
|
||||
return {
|
||||
verdict: 'inconclusive',
|
||||
mode,
|
||||
breaches,
|
||||
notes: [
|
||||
...notes,
|
||||
'committed baseline does not match this run — re-run `--update-baseline` and commit',
|
||||
],
|
||||
};
|
||||
}
|
||||
if (breaches.length > 0 && !committed.justification && !opts.allowRegression) {
|
||||
return {
|
||||
verdict: 'regression',
|
||||
mode,
|
||||
breaches,
|
||||
notes: [
|
||||
...notes,
|
||||
'metrics regressed vs main — add a `justification` to the committed baseline (the reviewable reason)',
|
||||
],
|
||||
};
|
||||
}
|
||||
if (breaches.length > 0) {
|
||||
notes.push(
|
||||
`regression blessed: ${committed.justification ?? opts.allowRegression ?? ''}`.trim(),
|
||||
);
|
||||
}
|
||||
return { verdict: 'pass', mode, breaches, notes };
|
||||
}
|
||||
|
||||
if (breaches.length > 0) {
|
||||
if (opts.allowRegression) {
|
||||
notes.push(`regression allowed: ${opts.allowRegression}`);
|
||||
return { verdict: 'pass', mode, breaches, notes };
|
||||
}
|
||||
return { verdict: 'regression', mode, breaches, notes };
|
||||
}
|
||||
return { verdict: 'pass', mode, breaches, notes };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Markdown scoreboard
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const METRIC_ORDER = [
|
||||
'know_to_ask_failure_rate',
|
||||
'false_fire_rate',
|
||||
'push_precision',
|
||||
'push_recall',
|
||||
'write_back_fidelity',
|
||||
'provenance_accuracy',
|
||||
'continuity_rate',
|
||||
'extraction_recall',
|
||||
'extraction_precision',
|
||||
'source_isolation_violations',
|
||||
'avg_injected_tokens',
|
||||
];
|
||||
|
||||
export function renderScoreboardMarkdown(
|
||||
result: BrainBenchResult,
|
||||
compare?: CompareOutcome | null,
|
||||
): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('# BrainBench scoreboard');
|
||||
lines.push('');
|
||||
lines.push(
|
||||
`fixtures \`${result.receipt.fixtures_hash.slice(0, 12)}\` · ${result.receipt.include_holdout ? 'holdout INCLUDED (published-run mode)' : 'holdout excluded (gate mode)'} · ${result.receipt.llm ? 'LLM extractor' : 'deterministic (hermetic)'}`,
|
||||
);
|
||||
lines.push('');
|
||||
lines.push('| harness | seam | suite | failed/gold | ' + METRIC_ORDER.map((m) => `\`${m}\``).join(' | ') + ' |');
|
||||
lines.push('|---|---|---|---|' + METRIC_ORDER.map(() => '---').join('|') + '|');
|
||||
for (const c of result.cells) {
|
||||
const vals = METRIC_ORDER.map((m) => (c.metrics[m] !== undefined ? String(c.metrics[m]) : '—'));
|
||||
lines.push(
|
||||
`| ${c.harness} | ${c.seam} | ${c.suite} | ${c.gold_failed}/${c.gold_total} | ${vals.join(' | ')} |`,
|
||||
);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push(
|
||||
'_seam: `production` rows exercise a shipped integration seam; `contract` rows grade the same gbrain primitives through a harness-shaped injection contract (see docs/eval/BRAINBENCH.md)._',
|
||||
);
|
||||
if (result.seed_failures.length > 0) {
|
||||
lines.push('');
|
||||
lines.push(`**SEED FAILURES (${result.seed_failures.length})** — run is invalid (exit 2):`);
|
||||
for (const f of result.seed_failures) lines.push(`- ${f.fixture_id}: ${f.error}`);
|
||||
}
|
||||
if (compare) {
|
||||
lines.push('');
|
||||
lines.push(`## Gate: ${compare.verdict.toUpperCase()} (${compare.mode})`);
|
||||
for (const n of compare.notes) lines.push(`- ${n}`);
|
||||
if (compare.breaches.length > 0) {
|
||||
lines.push('');
|
||||
lines.push('| cell | metric | main | current | detail |');
|
||||
lines.push('|---|---|---|---|---|');
|
||||
for (const b of compare.breaches) {
|
||||
lines.push(
|
||||
`| ${b.cell} | ${b.metric} | ${Number.isNaN(b.baseline) ? '—' : b.baseline} | ${Number.isNaN(b.current) ? '—' : b.current} | ${b.detail} |`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
lines.push('');
|
||||
return lines.join('\n');
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* BrainBench fixture seeding — fail-fast (decision 12).
|
||||
*
|
||||
* Hermetic: pages import with noEmbed (no gateway), facts insert with a NULL
|
||||
* embedding (the PGLite insertFact path stores NULL without touching any
|
||||
* provider). Keyword/alias-arm retrieval carries the bench in CI; the vector
|
||||
* path is a documented non-goal of hermetic mode (decision 2).
|
||||
*
|
||||
* `importFromContent` returns status 'imported' | 'skipped' | 'error' — never
|
||||
* 'success' (prior learning importFromContent-status-vocabulary). Anything
|
||||
* other than 'imported' means the fixture's brain is WRONG, and every metric
|
||||
* scored against it would be silent garbage — so seeding throws, the harness
|
||||
* marks the fixture seed_failed, and the run exits 2.
|
||||
*/
|
||||
|
||||
import { importFromContent } from '../../core/import-file.ts';
|
||||
import type { PGLiteEngine } from '../../core/pglite-engine.ts';
|
||||
import type { BrainBenchFixture } from './types.ts';
|
||||
|
||||
export class SeedError extends Error {
|
||||
constructor(
|
||||
public readonly fixtureId: string,
|
||||
public readonly slug: string,
|
||||
message: string,
|
||||
) {
|
||||
super(`seed failed for fixture ${fixtureId} at ${slug}: ${message}`);
|
||||
this.name = 'SeedError';
|
||||
}
|
||||
}
|
||||
|
||||
export interface SeedOutcome {
|
||||
/**
|
||||
* slug → set of source_ids the slug was seeded into (cross-source violation
|
||||
* detection, decision 14). A Set because multi-source fixtures deliberately
|
||||
* seed the SAME slug into two sources; a slug is a violation only when its
|
||||
* set does NOT include the fixture's active source.
|
||||
*/
|
||||
slugSource: Map<string, Set<string>>;
|
||||
pages: number;
|
||||
facts: number;
|
||||
}
|
||||
|
||||
const DEFAULT_SOURCE = 'default';
|
||||
|
||||
/** Idempotent: creates any non-default sources the fixture declares. */
|
||||
async function ensureSources(engine: PGLiteEngine, fixture: BrainBenchFixture): Promise<void> {
|
||||
const wanted = new Set<string>();
|
||||
for (const s of fixture.sources ?? []) wanted.add(s);
|
||||
if (fixture.active_source && fixture.active_source !== DEFAULT_SOURCE) {
|
||||
wanted.add(fixture.active_source);
|
||||
}
|
||||
for (const p of fixture.seed_pages ?? []) {
|
||||
if (p.source_id && p.source_id !== DEFAULT_SOURCE) wanted.add(p.source_id);
|
||||
}
|
||||
for (const f of fixture.seed_facts ?? []) {
|
||||
if (f.source_id && f.source_id !== DEFAULT_SOURCE) wanted.add(f.source_id);
|
||||
}
|
||||
for (const id of wanted) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ($1, $2, '{}'::jsonb)
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
[id, `bench source ${id}`],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export async function seedBrain(engine: PGLiteEngine, fixture: BrainBenchFixture): Promise<SeedOutcome> {
|
||||
await ensureSources(engine, fixture);
|
||||
|
||||
const slugSource = new Map<string, Set<string>>();
|
||||
let pages = 0;
|
||||
let facts = 0;
|
||||
|
||||
for (const page of fixture.seed_pages ?? []) {
|
||||
const sourceId = page.source_id ?? DEFAULT_SOURCE;
|
||||
let result;
|
||||
try {
|
||||
result = await importFromContent(engine, page.slug, page.content, {
|
||||
noEmbed: true,
|
||||
sourceId,
|
||||
});
|
||||
} catch (err) {
|
||||
throw new SeedError(fixture.fixture_id, page.slug, (err as Error).message);
|
||||
}
|
||||
if (result.status !== 'imported') {
|
||||
throw new SeedError(
|
||||
fixture.fixture_id,
|
||||
page.slug,
|
||||
`importFromContent status=${result.status}${result.error ? ` (${result.error})` : ''}`,
|
||||
);
|
||||
}
|
||||
const set = slugSource.get(page.slug) ?? new Set<string>();
|
||||
set.add(sourceId);
|
||||
slugSource.set(page.slug, set);
|
||||
pages++;
|
||||
}
|
||||
|
||||
for (const sf of fixture.seed_facts ?? []) {
|
||||
const sourceId = sf.source_id ?? DEFAULT_SOURCE;
|
||||
try {
|
||||
await engine.insertFact( // gbrain-allow-direct-insert: benchmark seeding into a throwaway in-memory brain — no fence exists, the fixture is the source of truth
|
||||
{
|
||||
fact: sf.fact,
|
||||
entity_slug: sf.entity_slug ?? null,
|
||||
source: sf.source ?? 'bench:seed',
|
||||
source_session: sf.source_session ?? null,
|
||||
// NULL embedding: hermetic mode never touches an embedding provider.
|
||||
embedding: null,
|
||||
},
|
||||
{ source_id: sourceId },
|
||||
);
|
||||
facts++;
|
||||
} catch (err) {
|
||||
throw new SeedError(
|
||||
fixture.fixture_id,
|
||||
sf.entity_slug ?? '(no-entity fact)',
|
||||
`insertFact failed: ${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return { slugSource, pages, facts };
|
||||
}
|
||||
@@ -0,0 +1,301 @@
|
||||
/**
|
||||
* BrainBench — cross-harness memory conformance suite (Cathedral 2).
|
||||
*
|
||||
* Type layer for the fixture corpus, harness adapters, and result documents.
|
||||
* The fixture + result shapes are PUBLISHED interchange formats (mirrored as
|
||||
* JSON Schemas in evals/brainbench/schema/) so foreign runners — notably the
|
||||
* sibling gbrain-evals repo — can drive `gbrain eval brainbench --fixtures DIR
|
||||
* --gold DIR --json --out FILE` against their own corpora. Breaking changes
|
||||
* bump FIXTURE_SCHEMA_VERSION / RESULT_SCHEMA_VERSION; additive-only within a
|
||||
* version.
|
||||
*
|
||||
* Sealed-gold discipline (gbrain-evals convention): fixture files carry ONLY
|
||||
* what an adapter may see (turns, seed content). Gold annotations live in a
|
||||
* separate gold dir, joined by the loader, and the harness hands adapters a
|
||||
* sanitized PublicTurn. A `gold` key inside a fixture turn is a VALIDATION
|
||||
* ERROR, not a convenience.
|
||||
*/
|
||||
|
||||
import type { ReflexPointer } from '../../core/context/retrieval-reflex.ts';
|
||||
import type { PGLiteEngine } from '../../core/pglite-engine.ts';
|
||||
|
||||
export const FIXTURE_SCHEMA_VERSION = 1;
|
||||
export const RESULT_SCHEMA_VERSION = 1;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Suites + harnesses
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export const ALL_SUITES = ['know-to-ask', 'push', 'write-back', 'continuity'] as const;
|
||||
export type BrainBenchSuite = (typeof ALL_SUITES)[number];
|
||||
|
||||
export const ALL_HARNESSES = ['openclaw', 'claude-code', 'codex'] as const;
|
||||
export type HarnessName = (typeof ALL_HARNESSES)[number];
|
||||
|
||||
/**
|
||||
* 'production' — exercises a shipped integration seam byte-for-byte.
|
||||
* 'contract' — grades gbrain primitives through a harness-shaped injection
|
||||
* contract a later PR will wire to the real harness. Printed on
|
||||
* every scoreboard row; see docs/eval/BRAINBENCH.md.
|
||||
*/
|
||||
export type SeamKind = 'production' | 'contract';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fixture (adapter-visible) shapes
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface SeedPage {
|
||||
slug: string;
|
||||
/** Full markdown content incl. frontmatter. Imported with noEmbed. */
|
||||
content: string;
|
||||
/** Which source this page seeds into. Default 'default'. */
|
||||
source_id?: string;
|
||||
}
|
||||
|
||||
export interface SeedFact {
|
||||
fact: string;
|
||||
entity_slug?: string | null;
|
||||
/** Provenance string; default 'bench:seed'. */
|
||||
source?: string;
|
||||
source_session?: string | null;
|
||||
source_id?: string;
|
||||
}
|
||||
|
||||
export interface FixtureTurn {
|
||||
turn_id: number;
|
||||
role: 'user' | 'assistant';
|
||||
text: string;
|
||||
/**
|
||||
* ISO timestamp. Required for write-back fixtures (the conversation page
|
||||
* rendering + segment splitting need real times); optional elsewhere.
|
||||
*/
|
||||
ts?: string;
|
||||
}
|
||||
|
||||
export interface BrainBenchFixture {
|
||||
schema_version: number;
|
||||
fixture_id: string;
|
||||
/** Which metric suites consume this fixture. */
|
||||
suites: BrainBenchSuite[];
|
||||
/** Generator category (kta-pos, kta-neg, push, write-back, continuity, multi-source, adversarial). */
|
||||
category?: string;
|
||||
/**
|
||||
* Excluded from the CI gate; scored only in published runs (--include-holdout).
|
||||
* Gaming resistance per decision 22.
|
||||
*/
|
||||
holdout?: boolean;
|
||||
/**
|
||||
* Extra source ids to create beyond 'default' (multi-source fixtures,
|
||||
* decision 14). Seed pages/facts route via their own source_id.
|
||||
*/
|
||||
sources?: string[];
|
||||
/** The source the conversation happens in. Default 'default'. */
|
||||
active_source?: string;
|
||||
seed_pages?: SeedPage[];
|
||||
seed_facts?: SeedFact[];
|
||||
turns: FixtureTurn[];
|
||||
/** Present on continuity fixtures only (pairing metadata, not gold). */
|
||||
continuity?: {
|
||||
pair_id: string;
|
||||
pair_role: 'writer' | 'reader';
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* What an adapter is allowed to see of a turn. Structurally sealed: built by
|
||||
* `toPublicTurn`, which picks exactly these fields — anything else (incl. a
|
||||
* smuggled `gold`) is dropped.
|
||||
*/
|
||||
export interface PublicTurn {
|
||||
turn_id: number;
|
||||
role: 'user' | 'assistant';
|
||||
text: string;
|
||||
ts?: string;
|
||||
}
|
||||
|
||||
export function toPublicTurn(turn: FixtureTurn): PublicTurn {
|
||||
const out: PublicTurn = { turn_id: turn.turn_id, role: turn.role, text: turn.text };
|
||||
if (turn.ts !== undefined) out.ts = turn.ts;
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Gold (sealed) shapes — evals/brainbench/gold/<fixture_id>.gold.json
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface GoldFactSpec {
|
||||
/** Human label for the fact ("pricing concern"). */
|
||||
gist: string;
|
||||
/** The exact fact text the gold extractor emits into the production pipeline. */
|
||||
fact: string;
|
||||
entity_slug: string | null;
|
||||
/** Keyword probe: every keyword must appear (case-insensitive) in the stored fact. */
|
||||
match_keywords: string[];
|
||||
kind?: 'event' | 'preference' | 'commitment' | 'belief' | 'fact';
|
||||
}
|
||||
|
||||
export interface TurnGold {
|
||||
should_retrieve: boolean;
|
||||
/** Slugs that SHOULD be injected (recall denominator). */
|
||||
gold_slugs?: string[];
|
||||
/** Additionally-acceptable slugs (count for precision, not required for recall). */
|
||||
acceptable_slugs?: string[];
|
||||
/** Write-back gold: facts this turn contributes (consumed via the gold extractor). */
|
||||
gold_facts?: GoldFactSpec[];
|
||||
}
|
||||
|
||||
export interface ContinuityDecisionGold {
|
||||
decision_id: string;
|
||||
/** Reader-side success: any of these slugs injected on the probe turn... */
|
||||
expected_slugs: string[];
|
||||
/** ...or a stored fact matching all keywords is recallable. */
|
||||
match_keywords: string[];
|
||||
}
|
||||
|
||||
export interface FixtureGold {
|
||||
fixture_id: string;
|
||||
/** Keyed by String(turn_id). Turns without an entry have no gold (assistant turns, filler). */
|
||||
turns: Record<string, TurnGold>;
|
||||
continuity?: {
|
||||
pair_id: string;
|
||||
decisions: ContinuityDecisionGold[];
|
||||
};
|
||||
}
|
||||
|
||||
/** Loader output: fixture joined with its gold. Internal to the harness — never crosses to adapters. */
|
||||
export interface LoadedFixture {
|
||||
fixture: BrainBenchFixture;
|
||||
gold: FixtureGold;
|
||||
/** Absolute path the fixture was loaded from (error reporting). */
|
||||
path: string;
|
||||
}
|
||||
|
||||
export interface LoadedCorpus {
|
||||
fixtures: LoadedFixture[];
|
||||
/** sha256 over sorted relative-path + content of every fixture AND gold file. */
|
||||
fixtures_hash: string;
|
||||
fixture_dir: string;
|
||||
gold_dir: string;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Adapter contract
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface HarnessTurnResult {
|
||||
/** The text the harness would inject this turn (null = stayed silent). */
|
||||
injectedText: string | null;
|
||||
/** Normalized slugs referenced by the injection — what metrics score. */
|
||||
injectedSlugs: string[];
|
||||
pointers: ReflexPointer[];
|
||||
/** Estimated tokens of injectedText (chars/4 heuristic; intrusion diagnostics). */
|
||||
injectedTokens: number;
|
||||
latencyMs: number;
|
||||
}
|
||||
|
||||
export interface HarnessAdapter {
|
||||
readonly name: HarnessName;
|
||||
readonly seam: SeamKind;
|
||||
/** Called once per (fixture, adapter) before any turn. */
|
||||
beginConversation(engine: PGLiteEngine, fixture: AdapterFixtureView): Promise<void>;
|
||||
/**
|
||||
* Replay one turn. `priorContextText` is the joined text of PRIOR turns +
|
||||
* prior injections — adapters whose seam has no conversation memory (e.g.
|
||||
* the claude-code hook contract) ignore it by config, and that delta is
|
||||
* part of what the bench measures.
|
||||
*/
|
||||
replayTurn(turn: PublicTurn, priorContextText: string): Promise<HarnessTurnResult>;
|
||||
endConversation(): Promise<void>;
|
||||
}
|
||||
|
||||
/** The slice of a fixture an adapter may see (no gold, no category metadata). */
|
||||
export interface AdapterFixtureView {
|
||||
fixture_id: string;
|
||||
active_source: string;
|
||||
turns: PublicTurn[];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Per-turn evaluation rows + metric outputs
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface TurnRow {
|
||||
fixture_id: string;
|
||||
turn_id: number;
|
||||
harness: HarnessName;
|
||||
suite: BrainBenchSuite;
|
||||
injected_slugs: string[];
|
||||
injected_tokens: number;
|
||||
gold: TurnGold | null;
|
||||
/** Slugs injected from a source other than the fixture's active source (decision 14). */
|
||||
cross_source_slugs: string[];
|
||||
latency_ms: number;
|
||||
}
|
||||
|
||||
/** One harness × suite cell of the scoreboard. Counts first; rates derived. */
|
||||
export interface SuiteMetrics {
|
||||
suite: BrainBenchSuite;
|
||||
harness: HarnessName;
|
||||
seam: SeamKind;
|
||||
/** Gold items evaluated / failed — the count-aware gate operates on these. */
|
||||
gold_total: number;
|
||||
gold_failed: number;
|
||||
/** Named metric values (registered in metric-glossary.ts). */
|
||||
metrics: Record<string, number>;
|
||||
/** Fixture ids that contributed (excludes holdout in gate mode). */
|
||||
fixtures: string[];
|
||||
}
|
||||
|
||||
export interface BrainBenchReceipt {
|
||||
result_schema_version: number;
|
||||
fixtures_hash: string;
|
||||
harness_sha: string;
|
||||
ts: string;
|
||||
cmd_args: string[];
|
||||
seed: number;
|
||||
include_holdout: boolean;
|
||||
llm: boolean;
|
||||
}
|
||||
|
||||
export interface BrainBenchResult {
|
||||
receipt: BrainBenchReceipt;
|
||||
cells: SuiteMetrics[];
|
||||
turn_rows: TurnRow[];
|
||||
/** Fixtures that failed to seed (decision 12) — run exits 2 when non-empty. */
|
||||
seed_failures: Array<{ fixture_id: string; error: string }>;
|
||||
_meta?: { metric_glossary: Record<string, unknown> };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Canonical committed baseline (decision 10) — diff-stable, receipts excluded
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface BrainBenchBaseline {
|
||||
schema_version: number;
|
||||
fixtures_hash: string;
|
||||
/**
|
||||
* Required when a regression vs the prior baseline is being blessed
|
||||
* (decision 4) — visible in the PR diff, review-enforced.
|
||||
*/
|
||||
justification?: string;
|
||||
/** `${harness}/${suite}` → metric name → value rounded to 4 decimals, keys sorted. */
|
||||
cells: Record<string, Record<string, number>>;
|
||||
/** `${harness}/${suite}` → { gold_total, gold_failed } for the count-aware gate. */
|
||||
counts: Record<string, { gold_total: number; gold_failed: number }>;
|
||||
}
|
||||
|
||||
/** Verdict of a compare run. Maps to exit codes 0 / 1 / 2. */
|
||||
export type CompareVerdict = 'pass' | 'regression' | 'inconclusive';
|
||||
|
||||
export interface CompareOutcome {
|
||||
verdict: CompareVerdict;
|
||||
mode: 'same-hash' | 'corpus-bless';
|
||||
breaches: Array<{
|
||||
cell: string;
|
||||
metric: string;
|
||||
baseline: number;
|
||||
current: number;
|
||||
detail: string;
|
||||
}>;
|
||||
notes: string[];
|
||||
}
|
||||
Reference in New Issue
Block a user