mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* v0.41.15.0 feat: conversation parser cathedral + progressive-batch primitive (closes #1461) Replaces PR #1461's single-format Telegram regex with a 12-pattern built-in registry covering iMessage/Slack, Telegram (×2), Discord (×2), WhatsApp (×2 locales), Signal, Matrix/Element, IRC (×2), Teams. Each pattern is hand-vetted from public format docs (signal-cli, DiscordChatExporter, Telegram Desktop, WhatsApp export docs, Element matrix-archive, irssi/weechat defaults); module-load validation runs test_positive[] + test_negative[] for every pattern at startup so a typo makes gbrain refuse to start. PR #1461 contributor's BRACKET_TIME_RX + cleanSpeaker survive verbatim as the `telegram-bracket` built-in pattern + DEFAULT_SPEAKER_CLEAN export. All 33 of their test cases pass against the new orchestrator. Three layers per page (orchestrator chooses): 1. Built-in pattern registry (zero-cost, deterministic) 2. User-declared simple_pattern via config (deferred to v0.42+) 3. Opt-IN LLM polish + fallback (privacy-first; chat content goes to Anthropic only when user explicitly enables) D18 priority scoring picks the highest-match-rate pattern across the first 10 lines (not first-wins) so overlapping formats don't silently mis-route. D5 multi_line per-pattern + D11 quick_reject prefix screen + D19 timezone_policy per-pattern complete the registry shape. Companion: src/core/progressive-batch/ primitive (rule of three satisfied across 12+ ad-hoc cost-prompt sites). Wintermute-inspired ramp shape (trial 10 → 100 → 500 → full with verification at each stage), productionized with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). D3 fail-closed budget gate: null tracker + null Policy.maxCostUsd → abort_cost_cap reason='no_budget_safety_net'. D20 discriminated Verifier union (output_count | idempotent_mutation | noop). extract-conversation-facts is the one proven consumer in v0.41.15.0; 9-site retrofit deferred to v0.41.16.0+ per TODOS.md. Codex outside-voice review absorbed 8 substantive findings: - Privacy posture (LLM polish/fallback flipped to opt-IN) - ReDoS theater (dropped arbitrary user regex; v0.42+ uses RE2) - LLM-inferred-regex persistence as silent-corruption machine - Pattern priority scoring across first 10 lines - Timezone policy on every PatternEntry - Verifier shape discriminated union - Behavior parity for sites that "jumped straight to full" - Real-corpus-redacted fixture gap (v0.42+ TODO) CI gates: - bun run check:conversation-parser (13 fixtures, --no-llm, deterministic) - bun run check:fixture-privacy (banned-token grep) Doctor surfaces 3 new checks: conversation_format_coverage, progressive_batch_audit_health, conversation_parser_probe_health. Tests: 198/198 across primitive + parser + LLM + nightly probe + eval CLI + debug CLI + doctor checks + migration v97 round-trip + E2E parser ↔ engine integration. Real bug caught + fixed during gap audit: IdempotentMutationVerifier was comparing absolute mutated-count vs per-stage expected (failed silently on stage 2+); now uses per-stage delta semantics matching OutputCountVerifier. Schema migration v97: conversation_parser_llm_cache table with (content_sha256, model_id, call_shape) composite key. NO inferred_patterns table (D17: silent-corruption machine). Plan + 23 decisions + codex outside-voice absorption at ~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md. Co-Authored-By: garrytan-agents (PR #1461) <noreply@github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(check-privacy): allowlist scripts/check-fixture-privacy.sh The new sibling privacy guard literally names the banned tokens in its BANNED_TOKENS array — same meta-exception that check-privacy.sh itself gets. Without this allowlist entry, bun run verify rejects the file post-merge because the banned name appears in the rule-definition script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: renumber v0.41.15.0 → v0.41.16.0 (queue drift) Mechanical rename across all surfaces: VERSION, package.json, CHANGELOG (header + body refs), CLAUDE.md, TODOS.md, src/core/ migrate.ts (migration v98 comment), all src/core/conversation-parser/* and src/core/progressive-batch/* file headers, all test/ headers, scripts/check-privacy.sh allowlist comment, llms-full.txt regenerated. Audit clean: VERSION + package.json + CHANGELOG header all show 0.41.16.0. verify 24/24, touched tests 179/179. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents (PR #1461) <noreply@github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
garrytan-agents
Claude Opus 4.7
garrytan-agents
Claude Opus 4.7
parent
cd8efee0ea
commit
f702ec053b
@@ -0,0 +1,197 @@
|
||||
/**
|
||||
* v0.41.16.0 — `gbrain conversation-parser` debug CLI tests.
|
||||
*
|
||||
* Pins:
|
||||
* - list-builtins prints all 12 patterns + accepts --json
|
||||
* - validate emits "deferred to v0.42+" notice (v1 has no compiler)
|
||||
* - scan errors without engine
|
||||
* - --help works
|
||||
*
|
||||
* Pure-function tests; no engine, no DB.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { runConversationParser } from '../src/commands/conversation-parser.ts';
|
||||
import { BUILTIN_PATTERNS } from '../src/core/conversation-parser/builtins.ts';
|
||||
|
||||
// Capture process.stdout.write + process.stderr.write + process.exit.
|
||||
function captureStdio() {
|
||||
const out: string[] = [];
|
||||
const err: string[] = [];
|
||||
let exitCode: number | null = null;
|
||||
const origOut = process.stdout.write.bind(process.stdout);
|
||||
const origErr = process.stderr.write.bind(process.stderr);
|
||||
const origExit = process.exit.bind(process);
|
||||
process.stdout.write = ((chunk: string | Uint8Array) => {
|
||||
out.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString());
|
||||
return true;
|
||||
}) as typeof process.stdout.write;
|
||||
process.stderr.write = ((chunk: string | Uint8Array) => {
|
||||
err.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString());
|
||||
return true;
|
||||
}) as typeof process.stderr.write;
|
||||
// process.exit throws to short-circuit; tests catch and inspect.
|
||||
process.exit = ((code?: number) => {
|
||||
exitCode = code ?? 0;
|
||||
throw new Error(`PROCESS_EXIT:${exitCode}`);
|
||||
}) as typeof process.exit;
|
||||
return {
|
||||
out,
|
||||
err,
|
||||
getExitCode: () => exitCode,
|
||||
restore: () => {
|
||||
process.stdout.write = origOut;
|
||||
process.stderr.write = origErr;
|
||||
process.exit = origExit;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('runConversationParser — help', () => {
|
||||
test('--help prints usage', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['--help']);
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
const text = cap.out.join('');
|
||||
expect(text).toContain('Usage: gbrain conversation-parser');
|
||||
expect(text).toContain('scan');
|
||||
expect(text).toContain('list-builtins');
|
||||
expect(text).toContain('validate');
|
||||
});
|
||||
|
||||
test('no subcommand prints help', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, []);
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
expect(cap.out.join('')).toContain('Usage: gbrain conversation-parser');
|
||||
});
|
||||
});
|
||||
|
||||
describe('runConversationParser — list-builtins', () => {
|
||||
test('human output includes all 12 pattern ids', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['list-builtins']);
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
const text = cap.out.join('');
|
||||
for (const pattern of BUILTIN_PATTERNS) {
|
||||
expect(text).toContain(pattern.id);
|
||||
}
|
||||
expect(text).toContain(`${BUILTIN_PATTERNS.length} built-in patterns`);
|
||||
});
|
||||
|
||||
test('--json output is stable schema', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['list-builtins', '--json']);
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
const json = JSON.parse(cap.out.join('').trim());
|
||||
expect(json.schema_version).toBe(1);
|
||||
expect(json.total).toBe(BUILTIN_PATTERNS.length);
|
||||
expect(json.patterns).toHaveLength(BUILTIN_PATTERNS.length);
|
||||
expect(json.patterns[0].id).toBe('imessage-slack');
|
||||
expect(json.patterns[0].date_source).toBeDefined();
|
||||
expect(json.patterns[0].time_format).toBeDefined();
|
||||
expect(json.patterns[0].timezone_policy).toBeDefined();
|
||||
expect(json.patterns[0].regex).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runConversationParser — validate', () => {
|
||||
test('emits deferred notice (v0.42+ scope)', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cv-parser-cli-'));
|
||||
const path = join(dir, 'pattern.json');
|
||||
writeFileSync(path, '{}');
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['validate', path]);
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
expect(cap.out.join('')).toContain('deferred to v0.42+');
|
||||
});
|
||||
|
||||
test('--json validate emits structured deferred envelope', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cv-parser-cli-'));
|
||||
const path = join(dir, 'pattern.json');
|
||||
writeFileSync(path, '{}');
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['validate', path, '--json']);
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
const json = JSON.parse(cap.out.join('').trim());
|
||||
expect(json.schema_version).toBe(1);
|
||||
expect(json.status).toBe('deferred');
|
||||
expect(json.todo_ref).toContain('v0.41.16.0');
|
||||
});
|
||||
|
||||
test('exits 2 on missing file path', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['validate']);
|
||||
} catch (e) {
|
||||
expect((e as Error).message).toBe('PROCESS_EXIT:2');
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
expect(cap.getExitCode()).toBe(2);
|
||||
});
|
||||
|
||||
test('exits 2 on nonexistent file', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, [
|
||||
'validate',
|
||||
'/nonexistent/path.json',
|
||||
]);
|
||||
} catch (e) {
|
||||
expect((e as Error).message).toBe('PROCESS_EXIT:2');
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
expect(cap.getExitCode()).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('runConversationParser — scan', () => {
|
||||
test('exits 2 when engine is null (scan requires brain)', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['scan', 'some-slug']);
|
||||
} catch (e) {
|
||||
expect((e as Error).message).toBe('PROCESS_EXIT:2');
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
expect(cap.getExitCode()).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('runConversationParser — unknown subcommand', () => {
|
||||
test('exits 2 with hint', async () => {
|
||||
const cap = captureStdio();
|
||||
try {
|
||||
await runConversationParser(null, ['bogus-cmd']);
|
||||
} catch (e) {
|
||||
expect((e as Error).message).toBe('PROCESS_EXIT:2');
|
||||
} finally {
|
||||
cap.restore();
|
||||
}
|
||||
expect(cap.err.join('')).toContain('unknown subcommand');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* v0.41.16.0 — Test helpers for conversation-parser unit tests.
|
||||
*
|
||||
* `makeChatResult(text)` builds a fully-typed `ChatResult` stub so
|
||||
* test transports satisfy the gateway's type contract without each
|
||||
* test repeating the empty fields.
|
||||
*/
|
||||
|
||||
import type { ChatResult } from '../../src/core/ai/gateway.ts';
|
||||
|
||||
export function makeChatResult(
|
||||
text: string,
|
||||
usage = { input_tokens: 10, output_tokens: 30 },
|
||||
): ChatResult {
|
||||
return {
|
||||
text,
|
||||
blocks: [{ type: 'text', text }],
|
||||
stopReason: 'end',
|
||||
usage: {
|
||||
input_tokens: usage.input_tokens,
|
||||
output_tokens: usage.output_tokens,
|
||||
cache_read_tokens: 0,
|
||||
cache_creation_tokens: 0,
|
||||
},
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
providerId: 'anthropic',
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
/**
|
||||
* v0.41.16.0 — LLM base unit tests.
|
||||
*
|
||||
* Hermetic via the `chatTransport` test seam. No real API calls.
|
||||
*
|
||||
* Pins:
|
||||
* - Cache hit doesn't re-call transport
|
||||
* - Provider unavailable returns null without calling transport
|
||||
* - Transport throw → fail-open null
|
||||
* - Parse failure → fail-open null
|
||||
* - parseLlmJson 4-strategy fallback
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeEach } from 'bun:test';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import {
|
||||
runLlmCall,
|
||||
parseLlmJson,
|
||||
_resetLlmCacheForTests,
|
||||
getLlmCacheStats,
|
||||
probeLlmAvailability,
|
||||
} from '../../src/core/conversation-parser/llm-base.ts';
|
||||
import { makeChatResult } from './helpers.ts';
|
||||
|
||||
beforeEach(() => {
|
||||
_resetLlmCacheForTests();
|
||||
});
|
||||
|
||||
describe('probeLlmAvailability', () => {
|
||||
test('returns null when ANTHROPIC_API_KEY is unset', async () => {
|
||||
await withEnv(
|
||||
{ ANTHROPIC_API_KEY: undefined as unknown as string },
|
||||
async () => {
|
||||
expect(probeLlmAvailability('claude-haiku-4-5')).toBeNull();
|
||||
expect(probeLlmAvailability('anthropic:claude-haiku-4-5')).toBeNull();
|
||||
},
|
||||
);
|
||||
});
|
||||
test('returns normalized model when key set', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
expect(probeLlmAvailability('claude-haiku-4-5')).toBe(
|
||||
'anthropic:claude-haiku-4-5',
|
||||
);
|
||||
});
|
||||
});
|
||||
test('returns null for unknown provider', async () => {
|
||||
expect(probeLlmAvailability('madeup-provider:foo-model')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runLlmCall — happy path', () => {
|
||||
test('calls transport, parses, caches', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
let calls = 0;
|
||||
const result = await runLlmCall<{ ok: boolean }>({
|
||||
shape: 'fallback',
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
content: 'hello',
|
||||
system: 'test',
|
||||
parse: (text) => parseLlmJson(text),
|
||||
chatTransport: async () => {
|
||||
calls++;
|
||||
return makeChatResult('{"ok": true}', { input_tokens: 1, output_tokens: 1 });
|
||||
},
|
||||
});
|
||||
expect(result).toEqual({ ok: true });
|
||||
expect(calls).toBe(1);
|
||||
expect(getLlmCacheStats().misses).toBe(1);
|
||||
});
|
||||
});
|
||||
test('cache hit on second call with same content', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
let calls = 0;
|
||||
const transport = async () => {
|
||||
calls++;
|
||||
return makeChatResult('{"ok": true}', { input_tokens: 1, output_tokens: 1 });
|
||||
};
|
||||
const opts = {
|
||||
shape: 'fallback' as const,
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
content: 'hello',
|
||||
system: 'test',
|
||||
parse: (t: string) => parseLlmJson<{ ok: boolean }>(t),
|
||||
chatTransport: transport,
|
||||
};
|
||||
await runLlmCall(opts);
|
||||
await runLlmCall(opts);
|
||||
expect(calls).toBe(1);
|
||||
expect(getLlmCacheStats().hits).toBe(1);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runLlmCall — fail-open paths', () => {
|
||||
test('provider unavailable returns null without calling transport', async () => {
|
||||
await withEnv(
|
||||
{ ANTHROPIC_API_KEY: undefined as unknown as string },
|
||||
async () => {
|
||||
let calls = 0;
|
||||
const result = await runLlmCall<unknown>({
|
||||
shape: 'fallback',
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
content: 'hello',
|
||||
system: 'test',
|
||||
parse: () => ({}),
|
||||
chatTransport: async () => {
|
||||
calls++;
|
||||
return makeChatResult('{}', { input_tokens: 1, output_tokens: 1 });
|
||||
},
|
||||
});
|
||||
expect(result).toBeNull();
|
||||
expect(calls).toBe(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
test('transport throw → fail-open null', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
const result = await runLlmCall<unknown>({
|
||||
shape: 'fallback',
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
content: 'hello',
|
||||
system: 'test',
|
||||
parse: () => ({}),
|
||||
chatTransport: async () => {
|
||||
throw new Error('network down');
|
||||
},
|
||||
});
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
test('parse failure → fail-open null, not cached', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
let calls = 0;
|
||||
const opts = {
|
||||
shape: 'fallback' as const,
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
content: 'hello',
|
||||
system: 'test',
|
||||
parse: () => null,
|
||||
chatTransport: async () => {
|
||||
calls++;
|
||||
return makeChatResult('garbage', { input_tokens: 1, output_tokens: 1 });
|
||||
},
|
||||
};
|
||||
const r1 = await runLlmCall(opts);
|
||||
const r2 = await runLlmCall(opts);
|
||||
expect(r1).toBeNull();
|
||||
expect(r2).toBeNull();
|
||||
// Both calls hit transport because parse-fail isn't cached.
|
||||
expect(calls).toBe(2);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseLlmJson — 4-strategy fallback', () => {
|
||||
test('direct parse object', () => {
|
||||
expect(parseLlmJson<{ a: number }>('{"a": 1}')).toEqual({ a: 1 });
|
||||
});
|
||||
test('strip ```json fences', () => {
|
||||
expect(parseLlmJson<{ a: number }>('```json\n{"a": 1}\n```')).toEqual({ a: 1 });
|
||||
});
|
||||
test('strip bare ``` fences', () => {
|
||||
expect(parseLlmJson<{ a: number }>('```\n{"a": 1}\n```')).toEqual({ a: 1 });
|
||||
});
|
||||
test('extract first {...} substring', () => {
|
||||
expect(parseLlmJson<{ a: number }>('Here is the output: {"a": 1} (done)')).toEqual({
|
||||
a: 1,
|
||||
});
|
||||
});
|
||||
test('array mode: direct parse', () => {
|
||||
expect(parseLlmJson<number[]>('[1,2,3]', { array: true })).toEqual([1, 2, 3]);
|
||||
});
|
||||
test('array mode: extract first [...] substring', () => {
|
||||
expect(parseLlmJson<number[]>('Output: [1,2,3] done', { array: true })).toEqual([
|
||||
1, 2, 3,
|
||||
]);
|
||||
});
|
||||
test('non-JSON returns null', () => {
|
||||
expect(parseLlmJson<unknown>('not json at all')).toBeNull();
|
||||
});
|
||||
test('empty string returns null', () => {
|
||||
expect(parseLlmJson<unknown>('')).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* v0.41.16.0 — LLM fallback unit tests.
|
||||
*
|
||||
* Hermetic via the `chatTransport` test seam.
|
||||
*
|
||||
* Pins:
|
||||
* - Happy path: parses LLM-returned JSON array
|
||||
* - Adversarial input: LLM returns [] → parser returns [] (skip page)
|
||||
* - Malformed LLM output → fail-open null
|
||||
* - Provider unavailable → null
|
||||
* - Cache hit: doesn't re-call
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeEach } from 'bun:test';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import { runLlmFallback } from '../../src/core/conversation-parser/llm-fallback.ts';
|
||||
import { _resetLlmCacheForTests } from '../../src/core/conversation-parser/llm-base.ts';
|
||||
import { makeChatResult } from './helpers.ts';
|
||||
|
||||
beforeEach(() => {
|
||||
_resetLlmCacheForTests();
|
||||
});
|
||||
|
||||
describe('runLlmFallback', () => {
|
||||
test('happy path: parses LLM JSON output', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
const result = await runLlmFallback({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'some-novel-chat-format',
|
||||
chatTransport: async () =>
|
||||
makeChatResult(
|
||||
JSON.stringify([
|
||||
{ speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'hello' },
|
||||
{ speaker: 'Bob', timestamp: '2024-03-15T18:38:00Z', text: 'world' },
|
||||
]),
|
||||
{ input_tokens: 10, output_tokens: 30 },
|
||||
),
|
||||
});
|
||||
expect(result).not.toBeNull();
|
||||
expect(result).toHaveLength(2);
|
||||
expect(result![0].speaker).toBe('Alice');
|
||||
expect(result![1].text).toBe('world');
|
||||
});
|
||||
});
|
||||
|
||||
test('adversarial input: LLM returns [] → empty array', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
const result = await runLlmFallback({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'This is just a recipe for cookies. Not a chat log.',
|
||||
chatTransport: async () => makeChatResult('[]', { input_tokens: 10, output_tokens: 1 }),
|
||||
});
|
||||
expect(result).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
test('malformed output → fail-open null', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
const result = await runLlmFallback({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'something',
|
||||
chatTransport: async () =>
|
||||
makeChatResult(
|
||||
'I think this might be a chat log but I am not sure.',
|
||||
{ input_tokens: 10, output_tokens: 12 },
|
||||
),
|
||||
});
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
test('provider unavailable: returns null without calling transport', async () => {
|
||||
await withEnv(
|
||||
{ ANTHROPIC_API_KEY: undefined as unknown as string },
|
||||
async () => {
|
||||
let calls = 0;
|
||||
const result = await runLlmFallback({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'whatever',
|
||||
chatTransport: async () => {
|
||||
calls++;
|
||||
return makeChatResult('[]', { input_tokens: 1, output_tokens: 1 });
|
||||
},
|
||||
});
|
||||
expect(result).toBeNull();
|
||||
expect(calls).toBe(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('cache hit: second call doesnt re-invoke transport', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
let calls = 0;
|
||||
const opts = {
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'stable-body-for-cache',
|
||||
chatTransport: async () => {
|
||||
calls++;
|
||||
return makeChatResult('[{"speaker":"A","timestamp":"2024-01-01T00:00:00Z","text":"x"}]', { input_tokens: 1, output_tokens: 1 });
|
||||
},
|
||||
};
|
||||
await runLlmFallback(opts);
|
||||
await runLlmFallback(opts);
|
||||
expect(calls).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
test('strips invalid items from array', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
const result = await runLlmFallback({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'something',
|
||||
chatTransport: async () =>
|
||||
makeChatResult(
|
||||
JSON.stringify([
|
||||
{ speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'good' },
|
||||
{ speaker: 42, timestamp: 'x', text: 'bad shape' }, // speaker not string
|
||||
{ timestamp: '2024-03-15T18:38:00Z', text: 'missing speaker' },
|
||||
{ speaker: 'Bob', timestamp: '2024-03-15T18:39:00Z', text: 'good2' },
|
||||
]),
|
||||
{ input_tokens: 10, output_tokens: 30 },
|
||||
),
|
||||
});
|
||||
expect(result).toHaveLength(2);
|
||||
expect(result![0].speaker).toBe('Alice');
|
||||
expect(result![1].speaker).toBe('Bob');
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,174 @@
|
||||
/**
|
||||
* v0.41.16.0 — LLM polish unit tests.
|
||||
*
|
||||
* Pins:
|
||||
* - applyPolish pure function: merge + drop + edits
|
||||
* - Headroom guard skip path
|
||||
* - Provider unavailable → input messages unchanged
|
||||
* - Cache hit doesn't re-call
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeEach } from 'bun:test';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import {
|
||||
runLlmPolish,
|
||||
applyPolish,
|
||||
} from '../../src/core/conversation-parser/llm-polish.ts';
|
||||
import { _resetLlmCacheForTests } from '../../src/core/conversation-parser/llm-base.ts';
|
||||
import { BudgetTracker } from '../../src/core/budget/budget-tracker.ts';
|
||||
import { withBudgetTracker } from '../../src/core/ai/gateway.ts';
|
||||
import type { MatchedMessage } from '../../src/core/conversation-parser/types.ts';
|
||||
import { makeChatResult } from './helpers.ts';
|
||||
|
||||
beforeEach(() => {
|
||||
_resetLlmCacheForTests();
|
||||
});
|
||||
|
||||
const SAMPLE: MatchedMessage[] = [
|
||||
{ speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'hello' },
|
||||
{ speaker: 'Alice', timestamp: '2024-03-15T18:37:30Z', text: 'continuation' },
|
||||
{ speaker: 'system', timestamp: '2024-03-15T18:38:00Z', text: 'Bob joined' },
|
||||
{ speaker: 'Bob', timestamp: '2024-03-15T18:39:00Z', text: 'world (edited)' },
|
||||
];
|
||||
|
||||
describe('applyPolish — pure function', () => {
|
||||
test('merge two indices into one', () => {
|
||||
const r = applyPolish(SAMPLE, {
|
||||
merge_indices: [[0, 1]],
|
||||
drop_indices: [],
|
||||
edits: [],
|
||||
});
|
||||
expect(r.messages).toHaveLength(3);
|
||||
expect(r.messages[0].text).toBe('hello\ncontinuation');
|
||||
expect(r.delta.merged).toBe(1);
|
||||
});
|
||||
|
||||
test('drop indices', () => {
|
||||
const r = applyPolish(SAMPLE, {
|
||||
merge_indices: [],
|
||||
drop_indices: [2],
|
||||
edits: [],
|
||||
});
|
||||
expect(r.messages).toHaveLength(3);
|
||||
expect(r.messages.find((m) => m.text === 'Bob joined')).toBeUndefined();
|
||||
expect(r.delta.dropped).toBe(1);
|
||||
});
|
||||
|
||||
test('edit speaker and text', () => {
|
||||
const r = applyPolish(SAMPLE, {
|
||||
merge_indices: [],
|
||||
drop_indices: [],
|
||||
edits: [
|
||||
{ index: 3, field: 'text', value: 'world' },
|
||||
],
|
||||
});
|
||||
expect(r.messages[3].text).toBe('world');
|
||||
expect(r.delta.edits).toBe(1);
|
||||
});
|
||||
|
||||
test('combined: drop system msg, merge two, edit edited marker', () => {
|
||||
const r = applyPolish(SAMPLE, {
|
||||
merge_indices: [[0, 1]],
|
||||
drop_indices: [2],
|
||||
edits: [{ index: 3, field: 'text', value: 'world' }],
|
||||
});
|
||||
expect(r.messages).toHaveLength(2);
|
||||
expect(r.messages[0].text).toBe('hello\ncontinuation');
|
||||
expect(r.messages[1].text).toBe('world');
|
||||
});
|
||||
|
||||
test('no-op: returns messages unchanged', () => {
|
||||
const r = applyPolish(SAMPLE, {
|
||||
merge_indices: [],
|
||||
drop_indices: [],
|
||||
edits: [],
|
||||
});
|
||||
expect(r.messages).toHaveLength(4);
|
||||
expect(r.delta).toEqual({ merged: 0, dropped: 0, edits: 0 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('runLlmPolish', () => {
|
||||
test('happy path: LLM returns polish ops, applied', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
const r = await runLlmPolish({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: '...full body...',
|
||||
messages: SAMPLE,
|
||||
patternId: 'imessage-slack',
|
||||
chatTransport: async () =>
|
||||
makeChatResult(
|
||||
JSON.stringify({
|
||||
merge_indices: [[0, 1]],
|
||||
drop_indices: [2],
|
||||
edits: [{ index: 3, field: 'text', value: 'world' }],
|
||||
}),
|
||||
{ input_tokens: 100, output_tokens: 50 },
|
||||
),
|
||||
});
|
||||
expect(r.skipped).toBeUndefined();
|
||||
expect(r.messages).toHaveLength(2);
|
||||
expect(r.delta.merged).toBe(1);
|
||||
expect(r.delta.dropped).toBe(1);
|
||||
expect(r.delta.edits).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
test('headroom guard: skips when tracker within $0.10 of cap', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
const t = new BudgetTracker({ label: 'unit', maxCostUsd: 0.05 });
|
||||
let calls = 0;
|
||||
await withBudgetTracker(t, async () => {
|
||||
const r = await runLlmPolish({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'b',
|
||||
messages: SAMPLE,
|
||||
patternId: 'imessage-slack',
|
||||
chatTransport: async () => {
|
||||
calls++;
|
||||
return makeChatResult('{"merge_indices":[],"drop_indices":[],"edits":[]}', { input_tokens: 1, output_tokens: 1 });
|
||||
},
|
||||
});
|
||||
expect(r.skipped).toBe('headroom');
|
||||
expect(r.messages).toBe(SAMPLE); // unchanged input
|
||||
});
|
||||
expect(calls).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('provider unavailable: returns input unchanged + skipped=provider', async () => {
|
||||
await withEnv(
|
||||
{ ANTHROPIC_API_KEY: undefined as unknown as string },
|
||||
async () => {
|
||||
const r = await runLlmPolish({
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'b',
|
||||
messages: SAMPLE,
|
||||
patternId: 'imessage-slack',
|
||||
chatTransport: async () => makeChatResult('{}', { input_tokens: 1, output_tokens: 1 }),
|
||||
});
|
||||
expect(r.skipped).toBe('provider');
|
||||
expect(r.messages).toEqual(SAMPLE);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('cache hit on same (body, patternId) pair', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
|
||||
let calls = 0;
|
||||
const opts = {
|
||||
modelStr: 'claude-haiku-4-5',
|
||||
body: 'stable',
|
||||
messages: SAMPLE,
|
||||
patternId: 'imessage-slack',
|
||||
chatTransport: async () => {
|
||||
calls++;
|
||||
return makeChatResult('{"merge_indices":[],"drop_indices":[],"edits":[]}', { input_tokens: 1, output_tokens: 1 });
|
||||
},
|
||||
};
|
||||
await runLlmPolish(opts);
|
||||
await runLlmPolish(opts);
|
||||
expect(calls).toBe(1);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* v0.41.16.0 — Nightly probe unit tests.
|
||||
*
|
||||
* Hermetic via NightlyProbeDeps injection. No real LLM calls.
|
||||
*
|
||||
* Pins:
|
||||
* - Disabled + mode=conservative → rate_limited (informational)
|
||||
* - Mode=tokenmax overrides disable
|
||||
* - 24h rate limit → rate_limited
|
||||
* - No LLM key → no_embedding_key
|
||||
* - Fixture path missing → fail
|
||||
* - Adversarial false positive → adversarial_false_positive
|
||||
* - All pass → pass
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import {
|
||||
runConversationParserNightlyProbe,
|
||||
type NightlyProbeDeps,
|
||||
} from '../../src/core/conversation-parser/nightly-probe.ts';
|
||||
|
||||
function tmpFixture(content: string): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'probe-'));
|
||||
const path = join(dir, 'fix.jsonl');
|
||||
writeFileSync(path, content);
|
||||
return path;
|
||||
}
|
||||
|
||||
const POSITIVE = `{"fixture_id":"f1","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): hi","expected_messages":1,"expected_participants":["Alice"]}`;
|
||||
const ADVERSARIAL = `{"fixture_id":"a1","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"just prose","expected_messages":0,"expected_participants":[]}`;
|
||||
|
||||
function baseDeps(overrides: Partial<NightlyProbeDeps> = {}): NightlyProbeDeps {
|
||||
return {
|
||||
isEnabled: () => true,
|
||||
searchMode: () => 'balanced',
|
||||
hasLlmKey: () => true,
|
||||
resolveFixturePath: () => tmpFixture(POSITIVE),
|
||||
resolveAdversarialPath: () => tmpFixture(ADVERSARIAL),
|
||||
now: () => new Date('2026-05-26T00:00:00Z'),
|
||||
shouldSkipForRateLimit: () => false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('runConversationParserNightlyProbe', () => {
|
||||
test('disabled + balanced mode → rate_limited (informational)', async () => {
|
||||
const r = await runConversationParserNightlyProbe(
|
||||
baseDeps({ isEnabled: () => false, searchMode: () => 'balanced' }),
|
||||
);
|
||||
expect(r.outcome).toBe('rate_limited');
|
||||
expect(r.reason).toContain('enabled=false');
|
||||
});
|
||||
|
||||
test('disabled but mode=tokenmax → proceeds (default-on per D10)', async () => {
|
||||
const r = await runConversationParserNightlyProbe(
|
||||
baseDeps({ isEnabled: () => false, searchMode: () => 'tokenmax' }),
|
||||
);
|
||||
expect(r.outcome).toBe('pass');
|
||||
});
|
||||
|
||||
test('24h rate limit → rate_limited', async () => {
|
||||
const r = await runConversationParserNightlyProbe(
|
||||
baseDeps({ shouldSkipForRateLimit: () => true }),
|
||||
);
|
||||
expect(r.outcome).toBe('rate_limited');
|
||||
expect(r.reason).toContain('24h');
|
||||
});
|
||||
|
||||
test('no LLM key → no_embedding_key', async () => {
|
||||
const r = await runConversationParserNightlyProbe(
|
||||
baseDeps({ hasLlmKey: () => false }),
|
||||
);
|
||||
expect(r.outcome).toBe('no_embedding_key');
|
||||
});
|
||||
|
||||
test('happy path: all pass', async () => {
|
||||
const r = await runConversationParserNightlyProbe(baseDeps());
|
||||
expect(r.outcome).toBe('pass');
|
||||
expect(r.fixtures_total).toBe(2);
|
||||
expect(r.fixtures_passed).toBe(2);
|
||||
expect(r.adversarial_false_positives).toBe(0);
|
||||
});
|
||||
|
||||
test('adversarial false positive → adversarial_false_positive', async () => {
|
||||
// Adversarial fixture that LOOKS like iMessage; the parser will
|
||||
// match it and the eval will flag as fp.
|
||||
const adversarialThatMatches = `{"fixture_id":"a-bad","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): I was supposed to be unparseable","expected_messages":0,"expected_participants":[]}`;
|
||||
const r = await runConversationParserNightlyProbe(
|
||||
baseDeps({
|
||||
resolveAdversarialPath: () => tmpFixture(adversarialThatMatches),
|
||||
}),
|
||||
);
|
||||
expect(r.outcome).toBe('adversarial_false_positive');
|
||||
expect(r.adversarial_false_positives).toBe(1);
|
||||
});
|
||||
|
||||
test('fixture path missing → fail', async () => {
|
||||
const r = await runConversationParserNightlyProbe(
|
||||
baseDeps({ resolveFixturePath: () => '/nonexistent/path.jsonl' }),
|
||||
);
|
||||
expect(r.outcome).toBe('fail');
|
||||
expect(r.reason).toContain('missing');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,348 @@
|
||||
/**
|
||||
* v0.41.16.0 — Conversation parser orchestrator tests.
|
||||
*
|
||||
* Covers:
|
||||
* - PR #1461's 6 telegram-bracket cases verbatim (REGRESSION pin)
|
||||
* - All 12 built-in patterns hit their test_positive samples
|
||||
* - Date derivation precedence (D8)
|
||||
* - Pattern priority scoring (D18) — overlap resolution
|
||||
* - Quick-reject fast path (D11)
|
||||
* - Multi-line continuation (D5)
|
||||
* - Disabled-builtin honored
|
||||
* - Timezone warning (D19) emitted when frontmatter timezone missing
|
||||
*
|
||||
* Pure-function tests; no PGLite, no LLM. The LLM polish/fallback
|
||||
* tests live in `llm-base.test.ts`, `llm-polish.test.ts`,
|
||||
* `llm-fallback.test.ts` (T4).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
parseConversation,
|
||||
deriveDateContext,
|
||||
applyPattern,
|
||||
scorePattern,
|
||||
} from '../../src/core/conversation-parser/parse.ts';
|
||||
import { BUILTIN_PATTERNS } from '../../src/core/conversation-parser/builtins.ts';
|
||||
import type { Page } from '../../src/core/types.ts';
|
||||
|
||||
// Helper to construct a minimal Page for date-derivation tests.
|
||||
function makePage(
|
||||
frontmatter: Record<string, unknown> = {},
|
||||
effective_date?: Date,
|
||||
): Page {
|
||||
return {
|
||||
id: 1,
|
||||
slug: 'test/page',
|
||||
type: 'conversation',
|
||||
title: 'Test',
|
||||
compiled_truth: '',
|
||||
timeline: '',
|
||||
frontmatter,
|
||||
content_hash: undefined,
|
||||
created_at: new Date(),
|
||||
updated_at: new Date(),
|
||||
effective_date: effective_date ?? null,
|
||||
} as Page;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// REGRESSION: PR #1461's 6 telegram-bracket cases verbatim
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('parseConversation — REGRESSION PR #1461 (telegram-bracket)', () => {
|
||||
test('bracket-time with 👤 emoji speaker prefix', () => {
|
||||
const body = '**[18:37] \u{1f464} G T:** hello world';
|
||||
const r = parseConversation(body, { fallbackDate: '2026-05-24' });
|
||||
expect(r.messages).toHaveLength(1);
|
||||
expect(r.messages[0].speaker).toBe('G T');
|
||||
expect(r.messages[0].text).toBe('hello world');
|
||||
expect(r.messages[0].timestamp).toBe('2026-05-24T18:37:00Z');
|
||||
expect(r.matched_pattern_id).toBe('telegram-bracket');
|
||||
});
|
||||
|
||||
test('bracket-time with 🤖 robot emoji', () => {
|
||||
const body = '**[06:00] \u{1f916} Zion:** On it.';
|
||||
const r = parseConversation(body, { fallbackDate: '2026-05-25' });
|
||||
expect(r.messages).toHaveLength(1);
|
||||
expect(r.messages[0].speaker).toBe('Zion');
|
||||
expect(r.messages[0].text).toBe('On it.');
|
||||
expect(r.messages[0].timestamp).toBe('2026-05-25T06:00:00Z');
|
||||
});
|
||||
|
||||
test('bracket-time multi-line continuation', () => {
|
||||
const body = [
|
||||
'**[09:00] \u{1f464} Alice Example:** first line',
|
||||
'second line of same message',
|
||||
'**[09:05] \u{1f464} Bob Example:** separate message',
|
||||
].join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2026-05-20' });
|
||||
expect(r.messages).toHaveLength(2);
|
||||
expect(r.messages[0].text).toBe('first line\nsecond line of same message');
|
||||
expect(r.messages[1].text).toBe('separate message');
|
||||
});
|
||||
|
||||
test('bracket-time falls back to 1970-01-01 without fallbackDate', () => {
|
||||
const body = '**[14:30] \u{1f464} Alice Example:** test';
|
||||
const r = parseConversation(body);
|
||||
expect(r.messages).toHaveLength(1);
|
||||
expect(r.messages[0].timestamp).toBe('1970-01-01T14:30:00Z');
|
||||
});
|
||||
|
||||
test('mixed formats in one body: iMessage + bracket-time', () => {
|
||||
const body = [
|
||||
'**Alice Example** (2024-03-15 9:00 AM): format 1',
|
||||
'**[10:30] \u{1f464} Bob Example:** format 2',
|
||||
].join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
|
||||
// D18 scoring picks the dominant pattern. Both have one hit; ties
|
||||
// resolve to declared priority (imessage-slack=0, telegram-bracket=1).
|
||||
// The imessage line matches; the telegram line becomes a
|
||||
// continuation. This is a known D18 tradeoff for very-mixed bodies.
|
||||
// In practice every chat export is homogeneous, so this is a
|
||||
// degenerate test case.
|
||||
expect(r.messages.length).toBeGreaterThanOrEqual(1);
|
||||
expect(r.matched_pattern_id).toBe('imessage-slack');
|
||||
});
|
||||
|
||||
test('bracket-time without emoji prefix', () => {
|
||||
const body = '**[22:15] Plain Name:** no emoji';
|
||||
const r = parseConversation(body, { fallbackDate: '2026-01-01' });
|
||||
expect(r.messages).toHaveLength(1);
|
||||
expect(r.messages[0].speaker).toBe('Plain Name');
|
||||
expect(r.messages[0].text).toBe('no emoji');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// All 12 built-ins must parse their test_positive samples
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('parseConversation — every built-in matches its test_positive sample', () => {
|
||||
for (const entry of BUILTIN_PATTERNS) {
|
||||
test(`pattern ${entry.id}: first test_positive parses`, () => {
|
||||
const body = entry.test_positive[0];
|
||||
// Multi-line patterns need a body line on the next line.
|
||||
const fullBody =
|
||||
entry.multi_line && entry.captures.text_group === 0
|
||||
? `${body}\nsome body text`
|
||||
: body;
|
||||
const r = parseConversation(fullBody, {
|
||||
fallbackDate: '2024-03-15',
|
||||
});
|
||||
// Either matches a message OR (for some multi-line patterns) the
|
||||
// first line is the anchor and the body is consumed as text.
|
||||
expect(r.messages.length).toBeGreaterThanOrEqual(1);
|
||||
expect(r.matched_pattern_id).toBe(entry.id);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Date derivation precedence (D8)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('deriveDateContext (D8 precedence chain)', () => {
|
||||
test('explicit fallbackDate wins', () => {
|
||||
const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15'));
|
||||
const ctx = deriveDateContext({ fallbackDate: '2025-12-25', page });
|
||||
expect(ctx.fallbackDate).toBe('2025-12-25');
|
||||
expect(ctx.source).toBe('explicit');
|
||||
});
|
||||
test('frontmatter.date wins over effective_date', () => {
|
||||
const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15'));
|
||||
const ctx = deriveDateContext({ page });
|
||||
expect(ctx.fallbackDate).toBe('2024-01-01');
|
||||
expect(ctx.source).toBe('frontmatter_date');
|
||||
});
|
||||
test('effective_date wins when no frontmatter.date', () => {
|
||||
const page = makePage({}, new Date('2023-06-15T00:00:00Z'));
|
||||
const ctx = deriveDateContext({ page });
|
||||
expect(ctx.fallbackDate).toBe('2023-06-15');
|
||||
expect(ctx.source).toBe('effective_date');
|
||||
});
|
||||
test('epoch_default when nothing set', () => {
|
||||
const ctx = deriveDateContext({});
|
||||
expect(ctx.fallbackDate).toBe('1970-01-01');
|
||||
expect(ctx.source).toBe('epoch_default');
|
||||
});
|
||||
test('frontmatter.timezone surfaces', () => {
|
||||
const page = makePage({
|
||||
date: '2024-01-01',
|
||||
timezone: 'America/Los_Angeles',
|
||||
});
|
||||
const ctx = deriveDateContext({ page });
|
||||
expect(ctx.timezone).toBe('America/Los_Angeles');
|
||||
});
|
||||
test('invalid frontmatter.date falls through', () => {
|
||||
const page = makePage({ date: 'not-a-date' });
|
||||
const ctx = deriveDateContext({ page });
|
||||
expect(ctx.source).toBe('epoch_default');
|
||||
});
|
||||
test('frontmatter.date slices full ISO to YYYY-MM-DD', () => {
|
||||
const page = makePage({ date: '2024-03-15T18:37:00Z' });
|
||||
const ctx = deriveDateContext({ page });
|
||||
expect(ctx.fallbackDate).toBe('2024-03-15');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pattern priority scoring (D18)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('scorePattern (D18 priority scoring)', () => {
|
||||
test('telegram-bracket scores 1.0 on a pure telegram body', () => {
|
||||
const body = [
|
||||
'**[18:37] \u{1f464} Alice:** one',
|
||||
'**[18:38] \u{1f464} Bob:** two',
|
||||
'**[18:39] \u{1f464} Alice:** three',
|
||||
].join('\n');
|
||||
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
|
||||
expect(scorePattern(body, tg)).toBe(1);
|
||||
});
|
||||
test('imessage-slack scores 0 on pure telegram body', () => {
|
||||
const body = [
|
||||
'**[18:37] \u{1f464} Alice:** one',
|
||||
'**[18:38] \u{1f464} Bob:** two',
|
||||
].join('\n');
|
||||
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
|
||||
expect(scorePattern(body, im)).toBe(0);
|
||||
});
|
||||
test('mixed body: D18 picks the higher-scoring pattern', () => {
|
||||
// 3 telegram lines + 1 imessage line. Telegram should win.
|
||||
const body = [
|
||||
'**[18:37] \u{1f464} Alice:** one',
|
||||
'**[18:38] \u{1f464} Bob:** two',
|
||||
'**[18:39] \u{1f464} Alice:** three',
|
||||
'**Charlie** (2024-03-15 9:00 AM): one imessage',
|
||||
].join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
|
||||
expect(r.matched_pattern_id).toBe('telegram-bracket');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Disabled-builtin honored
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('parseConversation — disabledBuiltinIds', () => {
|
||||
test('disabling top pattern falls through to next', () => {
|
||||
const body = '**[18:37] \u{1f464} Alice:** hello';
|
||||
const rDefault = parseConversation(body, { fallbackDate: '2024-03-15' });
|
||||
expect(rDefault.matched_pattern_id).toBe('telegram-bracket');
|
||||
const rDisabled = parseConversation(body, {
|
||||
fallbackDate: '2024-03-15',
|
||||
disabledBuiltinIds: ['telegram-bracket'],
|
||||
});
|
||||
// No other built-in matches this exact shape → no_match.
|
||||
expect(rDisabled.phase).toBe('no_match');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Multi-line continuation (D5)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('parseConversation — multi-line continuation (D5)', () => {
|
||||
test('iMessage continuation absorbs orphan lines', () => {
|
||||
const body = [
|
||||
'**Alice Example** (2024-03-15 9:00 AM): first line',
|
||||
'continuation line',
|
||||
'another continuation',
|
||||
'**Bob Example** (2024-03-15 9:05 AM): second message',
|
||||
].join('\n');
|
||||
const r = parseConversation(body);
|
||||
expect(r.messages).toHaveLength(2);
|
||||
expect(r.messages[0].text).toBe(
|
||||
'first line\ncontinuation line\nanother continuation',
|
||||
);
|
||||
expect(r.messages[1].text).toBe('second message');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Timezone warning (D19)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('parseConversation — timezone warning (D19)', () => {
|
||||
test('telegram-bracket emits warning when no timezone in frontmatter', () => {
|
||||
const body = '**[18:37] \u{1f464} Alice:** hello';
|
||||
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
|
||||
expect(r.timezone_warning).toBeDefined();
|
||||
expect(r.timezone_warning).toContain('telegram-bracket');
|
||||
expect(r.timezone_warning).toContain('UTC');
|
||||
});
|
||||
test('telegram-bracket does NOT warn when timezone is present', () => {
|
||||
const body = '**[18:37] \u{1f464} Alice:** hello';
|
||||
const page = makePage({
|
||||
date: '2024-03-15',
|
||||
timezone: 'America/Los_Angeles',
|
||||
});
|
||||
const r = parseConversation(body, { page });
|
||||
expect(r.timezone_warning).toBeUndefined();
|
||||
});
|
||||
test('imessage-slack does NOT warn (inline_utc policy)', () => {
|
||||
const body = '**Alice Example** (2024-03-15 9:00 AM): hello';
|
||||
const r = parseConversation(body);
|
||||
expect(r.timezone_warning).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Empty body + degenerate cases
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('parseConversation — degenerate inputs', () => {
|
||||
test('empty body returns no_match + empty messages', () => {
|
||||
expect(parseConversation('')).toEqual({
|
||||
messages: [],
|
||||
phase: 'no_match',
|
||||
});
|
||||
});
|
||||
test('non-conversational text returns no_match', () => {
|
||||
const r = parseConversation('This is just prose with no chat shape.');
|
||||
expect(r.phase).toBe('no_match');
|
||||
expect(r.messages).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// applyPattern — direct unit tests for the matcher
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('applyPattern — quick_reject fast path (D11)', () => {
|
||||
test('telegram quick_reject skips iMessage lines fast', () => {
|
||||
const body = '**Alice Example** (2024-03-15 9:00 AM): hello';
|
||||
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
|
||||
const r = applyPattern(body, tg, {
|
||||
fallbackDate: '2024-03-15',
|
||||
source: 'explicit',
|
||||
});
|
||||
// Quick_reject /^\*\*\[/ rejects '**Alice' (no `[`). Zero matches.
|
||||
expect(r).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// scorePattern boundary cases
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('scorePattern — boundary', () => {
|
||||
test('empty body scores 0', () => {
|
||||
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
|
||||
expect(scorePattern('', tg)).toBe(0);
|
||||
});
|
||||
test('only blank lines scores 0', () => {
|
||||
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
|
||||
expect(scorePattern('\n\n \n', tg)).toBe(0);
|
||||
});
|
||||
test('caps at SCORING_HEAD_LINES (10) lines', () => {
|
||||
// 100 telegram lines → still scores 1.0 because only first 10 sampled.
|
||||
const body = Array.from(
|
||||
{ length: 100 },
|
||||
(_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`,
|
||||
).join('\n');
|
||||
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
|
||||
expect(scorePattern(body, tg)).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,98 @@
|
||||
/**
|
||||
* v0.41.16.0 — Targeted assertions for the 3 new doctor checks.
|
||||
*
|
||||
* Spawns `bun src/cli.ts doctor --json --fast` as a subprocess and
|
||||
* parses the JSON envelope to verify:
|
||||
*
|
||||
* - conversation_format_coverage
|
||||
* - progressive_batch_audit_health
|
||||
* - conversation_parser_probe_health
|
||||
*
|
||||
* are present with stable shapes. The full doctor surface is covered
|
||||
* by test/doctor.test.ts; this file is a structural regression guard
|
||||
* for the 3 new v0.41.16.0 checks.
|
||||
*
|
||||
* Spawning the subprocess matches the actual user experience (`gbrain
|
||||
* doctor`) and avoids the in-process env/stdout-capture brittleness
|
||||
* that bit the original test draft.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
interface DoctorCheck {
|
||||
name: string;
|
||||
status: 'ok' | 'warn' | 'fail';
|
||||
message: string;
|
||||
}
|
||||
interface DoctorEnvelope {
|
||||
schema_version: number;
|
||||
status: 'healthy' | 'unhealthy';
|
||||
health_score: number;
|
||||
checks: DoctorCheck[];
|
||||
}
|
||||
|
||||
function runDoctor(): DoctorEnvelope {
|
||||
const result = spawnSync(
|
||||
process.execPath, // bun
|
||||
['src/cli.ts', 'doctor', '--json', '--fast'],
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
encoding: 'utf8',
|
||||
timeout: 60000,
|
||||
},
|
||||
);
|
||||
if (result.error) throw result.error;
|
||||
// Doctor's JSON envelope is the LAST line in stdout (CLI may print
|
||||
// banners on stderr; --json sends the envelope to stdout).
|
||||
const stdout = result.stdout ?? '';
|
||||
const lines = stdout.split('\n').filter((l) => l.trim().length > 0);
|
||||
const jsonLine = lines.reverse().find((l) => l.trim().startsWith('{'));
|
||||
if (!jsonLine) {
|
||||
throw new Error(
|
||||
`No JSON envelope found in doctor output. stdout=${stdout.slice(0, 500)} stderr=${(result.stderr ?? '').slice(0, 500)}`,
|
||||
);
|
||||
}
|
||||
return JSON.parse(jsonLine) as DoctorEnvelope;
|
||||
}
|
||||
|
||||
describe('doctor — v0.41.16.0 new checks emit', () => {
|
||||
test('all 3 new checks present in JSON envelope', () => {
|
||||
const env = runDoctor();
|
||||
const checkNames = env.checks.map((c) => c.name);
|
||||
// conversation_format_coverage may not appear in --fast mode (it
|
||||
// requires DB access); progressive_batch_audit_health and
|
||||
// conversation_parser_probe_health do not need DB.
|
||||
expect(checkNames).toContain('progressive_batch_audit_health');
|
||||
expect(checkNames).toContain('conversation_parser_probe_health');
|
||||
});
|
||||
|
||||
test('progressive_batch_audit_health shape', () => {
|
||||
const env = runDoctor();
|
||||
const check = env.checks.find(
|
||||
(c) => c.name === 'progressive_batch_audit_health',
|
||||
);
|
||||
expect(check).toBeDefined();
|
||||
expect(['ok', 'warn', 'fail']).toContain(check!.status);
|
||||
expect(typeof check!.message).toBe('string');
|
||||
expect(check!.message.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('conversation_parser_probe_health shape + opt-in hint', () => {
|
||||
const env = runDoctor();
|
||||
const check = env.checks.find(
|
||||
(c) => c.name === 'conversation_parser_probe_health',
|
||||
);
|
||||
expect(check).toBeDefined();
|
||||
expect(check!.status).toBe('ok');
|
||||
expect(check!.message).toContain('opt-in');
|
||||
expect(check!.message).toContain(
|
||||
'autopilot.conversation_parser_probe.enabled true',
|
||||
);
|
||||
});
|
||||
|
||||
test('schema_version is stable (2 at v0.41.16.0)', () => {
|
||||
const env = runDoctor();
|
||||
expect(env.schema_version).toBe(2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,163 @@
|
||||
/**
|
||||
* v0.41.16.0 — E2E test for the conversation parser cathedral against
|
||||
* a real PGLite brain.
|
||||
*
|
||||
* For each of the 12 built-in formats: seed a page through
|
||||
* `importFromContent`, run `parseConversation` against the body, assert
|
||||
* the parser identifies the correct pattern AND produces at least one
|
||||
* message AND the message timestamp lands in the expected date range.
|
||||
*
|
||||
* Per CLAUDE.md test-isolation R3+R4: uses the canonical PGLite block.
|
||||
* Hermetic (no DATABASE_URL needed); the in-memory PGLite is created
|
||||
* once per file and reset between tests.
|
||||
*
|
||||
* This is the integration test that proves the parser ↔ engine
|
||||
* interaction is correct for the dream cycle's
|
||||
* `conversation_facts_backfill` phase.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
||||
import {
|
||||
parseConversation,
|
||||
} from '../../src/core/conversation-parser/parse.ts';
|
||||
import { BUILTIN_PATTERNS } from '../../src/core/conversation-parser/builtins.ts';
|
||||
import { importFromContent } from '../../src/core/import-file.ts';
|
||||
import type { Page } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
/**
|
||||
* Build a conversation page body using the first test_positive sample
|
||||
* from the pattern's registry entry. Multi-line patterns get a body
|
||||
* line appended.
|
||||
*/
|
||||
function buildSampleBody(patternId: string): string {
|
||||
const pattern = BUILTIN_PATTERNS.find((p) => p.id === patternId);
|
||||
if (!pattern) throw new Error(`pattern ${patternId} not found`);
|
||||
// Repeat the positive sample 3 times with slight variations to
|
||||
// simulate a multi-message conversation.
|
||||
const samples = pattern.test_positive.slice(0, 2);
|
||||
if (pattern.multi_line && pattern.captures.text_group === 0) {
|
||||
// For patterns where text comes from the next line, append a body
|
||||
// line after each header.
|
||||
return samples.flatMap((s, i) => [s, `body line ${i + 1}`]).join('\n');
|
||||
}
|
||||
return samples.join('\n');
|
||||
}
|
||||
|
||||
describe('E2E: parser ↔ engine integration for every built-in', () => {
|
||||
for (const pattern of BUILTIN_PATTERNS) {
|
||||
test(`pattern=${pattern.id}: page imports + parser identifies pattern`, async () => {
|
||||
const slug = `conversations/test/${pattern.id}-sample`;
|
||||
const body = buildSampleBody(pattern.id);
|
||||
const frontmatter: Record<string, unknown> = {
|
||||
type: 'conversation',
|
||||
date: '2024-03-15',
|
||||
};
|
||||
if (pattern.timezone_policy === 'utc_assumed_with_warn') {
|
||||
// Some patterns warn without a timezone; provide one to avoid
|
||||
// stderr noise in the test output.
|
||||
frontmatter.timezone = 'America/Los_Angeles';
|
||||
}
|
||||
|
||||
// Seed the page through the canonical import path.
|
||||
const fullBody = `---\n${Object.entries(frontmatter)
|
||||
.map(([k, v]) => `${k}: ${typeof v === 'string' ? v : JSON.stringify(v)}`)
|
||||
.join('\n')}\n---\n\n${body}`;
|
||||
await importFromContent(engine, slug, fullBody, {
|
||||
sourceId: 'default',
|
||||
noEmbed: true,
|
||||
});
|
||||
|
||||
// Read the page back and parse it through the orchestrator
|
||||
// (mirrors what extract-conversation-facts.ts does in production).
|
||||
const page = (await engine.getPage(slug)) as Page;
|
||||
expect(page).not.toBeNull();
|
||||
|
||||
const bodyToParse = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim();
|
||||
const result = parseConversation(bodyToParse, { page });
|
||||
|
||||
expect(result.phase).toBe('regex_match');
|
||||
expect(result.matched_pattern_id).toBe(pattern.id);
|
||||
expect(result.messages.length).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// Every parsed message has a valid ISO timestamp.
|
||||
for (const msg of result.messages) {
|
||||
const ts = Date.parse(msg.timestamp);
|
||||
expect(Number.isFinite(ts)).toBe(true);
|
||||
// Pattern's timezone policy determines date range constraints.
|
||||
// For inline-date patterns the date is in the line; for
|
||||
// time-only patterns it's from frontmatter ('2024-03-15').
|
||||
if (pattern.date_source === 'frontmatter') {
|
||||
expect(msg.timestamp.slice(0, 10)).toBe('2024-03-15');
|
||||
}
|
||||
expect(msg.speaker.length).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe('E2E: parser handles unparseable bodies cleanly', () => {
|
||||
test('non-chat content returns no_match without crashing', async () => {
|
||||
const slug = 'conversations/test/not-a-chat';
|
||||
const fullBody = `---\ntype: conversation\ndate: 2024-03-15\n---\n\nThis is just prose. It has no chat structure at all. Just words that flow.`;
|
||||
await importFromContent(engine, slug, fullBody, {
|
||||
sourceId: 'default',
|
||||
noEmbed: true,
|
||||
});
|
||||
const page = (await engine.getPage(slug)) as Page;
|
||||
const bodyToParse = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim();
|
||||
const result = parseConversation(bodyToParse, { page });
|
||||
expect(result.phase).toBe('no_match');
|
||||
expect(result.messages).toEqual([]);
|
||||
});
|
||||
|
||||
test('empty body returns no_match', async () => {
|
||||
const slug = 'conversations/test/empty';
|
||||
const fullBody = `---\ntype: conversation\ndate: 2024-03-15\n---\n\n`;
|
||||
await importFromContent(engine, slug, fullBody, {
|
||||
sourceId: 'default',
|
||||
noEmbed: true,
|
||||
});
|
||||
const page = (await engine.getPage(slug)) as Page;
|
||||
const result = parseConversation(`${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(), { page });
|
||||
expect(result.phase).toBe('no_match');
|
||||
expect(result.messages).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E: D8 date derivation chain through Page', () => {
|
||||
test('frontmatter.date wins over epoch default', async () => {
|
||||
const slug = 'conversations/test/date-derive';
|
||||
const body = '**[18:37] 👤 Alice:** hi';
|
||||
const fullBody = `---\ntype: conversation\ndate: 2026-05-24\ntimezone: America/Los_Angeles\n---\n\n${body}`;
|
||||
await importFromContent(engine, slug, fullBody, {
|
||||
sourceId: 'default',
|
||||
noEmbed: true,
|
||||
});
|
||||
const page = (await engine.getPage(slug)) as Page;
|
||||
const result = parseConversation(
|
||||
`${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(),
|
||||
{ page },
|
||||
);
|
||||
expect(result.matched_pattern_id).toBe('telegram-bracket');
|
||||
expect(result.messages).toHaveLength(1);
|
||||
expect(result.messages[0].timestamp).toBe('2026-05-24T18:37:00Z');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,181 @@
|
||||
/**
|
||||
* v0.41.16.0 — `gbrain eval conversation-parser` CLI behavior tests.
|
||||
*
|
||||
* Covers argv parsing, exit codes (0/1/2), --json envelope shape,
|
||||
* --no-llm flag, --min-recall override, USAGE errors. Pure file I/O,
|
||||
* no DB, no API keys.
|
||||
*
|
||||
* Critical because `bun run check:conversation-parser` is wired into
|
||||
* `bun run verify` — a silent regression in exit-code handling would
|
||||
* make every PR's CI green even when the parser broke.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { runEvalConversationParser } from '../src/commands/eval-conversation-parser.ts';
|
||||
|
||||
function tmpFixture(content: string): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'eval-cli-'));
|
||||
const path = join(dir, 'fix.jsonl');
|
||||
writeFileSync(path, content);
|
||||
return path;
|
||||
}
|
||||
|
||||
const PASS = `{"fixture_id":"p1","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): hi","expected_messages":1,"expected_participants":["Alice"]}`;
|
||||
const FAIL_PATTERN_MISMATCH = `{"fixture_id":"f1","pattern":"telegram-bracket","frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): hi","expected_messages":1,"expected_participants":["Alice"]}`;
|
||||
|
||||
describe('runEvalConversationParser — exit codes', () => {
|
||||
test('exit 0 on all-pass fixture', async () => {
|
||||
const path = tmpFixture(PASS);
|
||||
const code = await runEvalConversationParser([path, '--no-llm']);
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
test('exit 1 on any failure', async () => {
|
||||
const path = tmpFixture(FAIL_PATTERN_MISMATCH);
|
||||
const code = await runEvalConversationParser([path, '--no-llm']);
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
test('exit 2 on missing fixture argument', async () => {
|
||||
const code = await runEvalConversationParser(['--no-llm']);
|
||||
expect(code).toBe(2);
|
||||
});
|
||||
|
||||
test('exit 2 on nonexistent fixture path', async () => {
|
||||
const code = await runEvalConversationParser([
|
||||
'/nonexistent/path.jsonl',
|
||||
'--no-llm',
|
||||
]);
|
||||
expect(code).toBe(2);
|
||||
});
|
||||
|
||||
test('exit 2 on malformed JSONL', async () => {
|
||||
const path = tmpFixture('NOT-JSON\n');
|
||||
const code = await runEvalConversationParser([path]);
|
||||
expect(code).toBe(2);
|
||||
});
|
||||
|
||||
test('exit 2 on empty fixture file', async () => {
|
||||
const path = tmpFixture('');
|
||||
const code = await runEvalConversationParser([path]);
|
||||
expect(code).toBe(2);
|
||||
});
|
||||
|
||||
test('exit 0 on help', async () => {
|
||||
const code = await runEvalConversationParser(['--help']);
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('runEvalConversationParser — flag parsing', () => {
|
||||
test('positional fixture path works without --fixtures', async () => {
|
||||
const path = tmpFixture(PASS);
|
||||
const code = await runEvalConversationParser([path, '--no-llm']);
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
test('--fixtures <path> form works', async () => {
|
||||
const path = tmpFixture(PASS);
|
||||
const code = await runEvalConversationParser([
|
||||
'--fixtures',
|
||||
path,
|
||||
'--no-llm',
|
||||
]);
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
test('--fixtures=<path> form works', async () => {
|
||||
const path = tmpFixture(PASS);
|
||||
const code = await runEvalConversationParser([
|
||||
`--fixtures=${path}`,
|
||||
'--no-llm',
|
||||
]);
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
test('--min-recall override (lower floor lets near-misses pass)', async () => {
|
||||
// A fixture where the parser only catches 0 of 1 expected messages.
|
||||
// Default --min-recall 0.9 would fail this; --min-recall 0.0 passes.
|
||||
const partial = `{"fixture_id":"pp","pattern":"telegram-bracket","frontmatter":{"date":"2024-03-15"},"body":"this is not a telegram line","expected_messages":1,"expected_participants":["Alice"]}`;
|
||||
const path = tmpFixture(partial);
|
||||
const defaultCode = await runEvalConversationParser([path, '--no-llm']);
|
||||
expect(defaultCode).toBe(1); // expected pattern not matched → fail
|
||||
// Lowering min-recall doesn't rescue a pattern mismatch — pattern
|
||||
// identity is checked separately. This pins that contract.
|
||||
const lowCode = await runEvalConversationParser([
|
||||
path,
|
||||
'--no-llm',
|
||||
'--min-recall',
|
||||
'0.0',
|
||||
]);
|
||||
expect(lowCode).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('runEvalConversationParser — --json envelope', () => {
|
||||
test('emits stable schema_version=1 + recall + pattern_coverage', async () => {
|
||||
const path = tmpFixture(PASS);
|
||||
const origWrite = process.stdout.write.bind(process.stdout);
|
||||
const captured: string[] = [];
|
||||
process.stdout.write = ((chunk: string | Uint8Array) => {
|
||||
captured.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString());
|
||||
return true;
|
||||
}) as typeof process.stdout.write;
|
||||
try {
|
||||
const code = await runEvalConversationParser([path, '--no-llm', '--json']);
|
||||
expect(code).toBe(0);
|
||||
} finally {
|
||||
process.stdout.write = origWrite;
|
||||
}
|
||||
const stdout = captured.join('');
|
||||
const json = JSON.parse(stdout.trim());
|
||||
expect(json.schema_version).toBe(1);
|
||||
expect(json.total_fixtures).toBe(1);
|
||||
expect(json.passed).toBe(1);
|
||||
expect(json.failed).toBe(0);
|
||||
expect(json.recall_mean).toBe(1);
|
||||
expect(json.participants_recall_mean).toBe(1);
|
||||
expect(json.pattern_coverage).toEqual({ 'imessage-slack': 1 });
|
||||
expect(json.fixtures).toHaveLength(1);
|
||||
expect(json.failed_fixture_ids).toEqual([]);
|
||||
});
|
||||
|
||||
test('--json on failure includes failed_fixture_ids', async () => {
|
||||
const path = tmpFixture(FAIL_PATTERN_MISMATCH);
|
||||
const origWrite = process.stdout.write.bind(process.stdout);
|
||||
const captured: string[] = [];
|
||||
process.stdout.write = ((chunk: string | Uint8Array) => {
|
||||
captured.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString());
|
||||
return true;
|
||||
}) as typeof process.stdout.write;
|
||||
try {
|
||||
const code = await runEvalConversationParser([path, '--no-llm', '--json']);
|
||||
expect(code).toBe(1);
|
||||
} finally {
|
||||
process.stdout.write = origWrite;
|
||||
}
|
||||
const json = JSON.parse(captured.join('').trim());
|
||||
expect(json.failed_fixture_ids).toContain('f1');
|
||||
});
|
||||
});
|
||||
|
||||
describe('runEvalConversationParser — adversarial fixture', () => {
|
||||
test('adversarial (pattern=null) with non-empty parse → fail', async () => {
|
||||
// Adversarial fixture whose body LOOKS like iMessage; parser will
|
||||
// match it; eval flags as adversarial false-positive.
|
||||
const advFp = `{"fixture_id":"adv-fp","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): I should not parse","expected_messages":0,"expected_participants":[]}`;
|
||||
const path = tmpFixture(advFp);
|
||||
const code = await runEvalConversationParser([path, '--no-llm']);
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
test('adversarial with truly unparseable body → pass', async () => {
|
||||
const advClean = `{"fixture_id":"adv-clean","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"This is just prose with no chat shape.","expected_messages":0,"expected_participants":[]}`;
|
||||
const path = tmpFixture(advClean);
|
||||
const code = await runEvalConversationParser([path, '--no-llm']);
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
{"fixture_id":"adversarial-recipe","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"Chocolate Chip Cookies\n\nIngredients:\n- 2 cups flour\n- 1 cup sugar\n- 1 cup chocolate chips\n\nInstructions:\nMix dry ingredients. Add wet. Bake at 375 for 12 minutes.","expected_messages":0,"expected_participants":[]}
|
||||
{"fixture_id":"adversarial-readme","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"# My Project\n\nA short description.\n\n## Install\n\n```\nnpm install my-project\n```\n\n## Usage\n\nImport the package and call the function.","expected_messages":0,"expected_participants":[]}
|
||||
{"fixture_id":"adversarial-code","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"function add(a, b) {\n return a + b;\n}\n\nconst result = add(1, 2);\nconsole.log(result);","expected_messages":0,"expected_participants":[]}
|
||||
{"fixture_id":"adversarial-lyrics","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"Twinkle twinkle little star\nHow I wonder what you are\nUp above the world so high\nLike a diamond in the sky","expected_messages":0,"expected_participants":[]}
|
||||
{"fixture_id":"adversarial-json","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"{\n \"key\": \"value\",\n \"num\": 42,\n \"list\": [1, 2, 3]\n}","expected_messages":0,"expected_participants":[]}
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
{"fixture_id":"imessage-001","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice Example** (2024-03-15 9:00 AM): morning\n**Bob Example** (2024-03-15 9:01 AM): hey there\n**Alice Example** (2024-03-15 9:02 AM): how are you\n**Bob Example** (2024-03-15 9:03 AM): good thanks\n**Alice Example** (2024-03-15 9:04 AM): you?","expected_messages":5,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"imessage-002","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Charlie Example** (2024-03-15 2:00 PM): afternoon\n**Charlie Example** (2024-03-15 2:01 PM): are you there?\n**Diana Example** (2024-03-15 2:05 PM): yes\n**Charlie Example** (2024-03-15 2:06 PM): great","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]}
|
||||
{"fixture_id":"telegram-bracket-001","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-24","timezone":"America/Los_Angeles"},"body":"**[18:37] 👤 Alice Example:** hello world\n**[18:38] 👤 Bob Example:** hey\n**[18:39] 👤 Alice Example:** how are you\n**[18:40] 👤 Bob Example:** good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"telegram-bracket-002","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-25","timezone":"America/Los_Angeles"},"body":"**[06:00] 🤖 Zion Bot:** On it.\n**[06:01] 👤 Charlie Example:** thanks\n**[06:02] 🤖 Zion Bot:** anything else?\n**[06:03] 👤 Charlie Example:** no good","expected_messages":4,"expected_participants":["Zion Bot","Charlie Example"]}
|
||||
{"fixture_id":"whatsapp-iso-001","pattern":"whatsapp-iso","frontmatter":{"date":"2024-03-15"},"body":"[15/03/24, 18:37:00] Alice Example: hello\n[15/03/24, 18:37:30] Bob Example: hey\n[15/03/24, 18:38:00] Alice Example: how are you\n[15/03/24, 18:39:00] Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"whatsapp-iso-002","pattern":"whatsapp-iso","frontmatter":{"date":"2024-01-01"},"body":"[01/01/24, 00:00:00] Charlie Example: happy new year\n[01/01/24, 00:00:30] Diana Example: same to you\n[01/01/24, 00:01:00] Charlie Example: any plans?\n[01/01/24, 00:02:00] Diana Example: sleep","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]}
|
||||
{"fixture_id":"whatsapp-us-001","pattern":"whatsapp-us","frontmatter":{"date":"2024-03-15"},"body":"3/15/24, 6:37 PM - Alice Example: hello\n3/15/24, 6:38 PM - Bob Example: hey\n3/15/24, 6:39 PM - Alice Example: how are you\n3/15/24, 6:40 PM - Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"whatsapp-us-002","pattern":"whatsapp-us","frontmatter":{"date":"2023-12-31"},"body":"12/31/23, 11:59 PM - Charlie Example: nye\n12/31/23, 11:59 PM - Diana Example: cheers","expected_messages":2,"expected_participants":["Charlie Example","Diana Example"]}
|
||||
{"fixture_id":"signal-export-001","pattern":"signal-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example (2024-03-15 18:37:00 UTC): hello\nBob Example (2024-03-15 18:37:30 UTC): hey there\nAlice Example (2024-03-15 18:38:00 UTC): how are you\nBob Example (2024-03-15 18:39:00 UTC): good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"irc-classic-001","pattern":"irc-classic","frontmatter":{"date":"2024-03-15"},"body":"<alice> hello world\n<bob> hey there\n<alice> how are you\n<bob> good thanks","expected_messages":4,"expected_participants":["alice","bob"]}
|
||||
{"fixture_id":"irc-weechat-001","pattern":"irc-weechat","frontmatter":{"date":"2024-03-15"},"body":"18:37 <alice> hello\n18:38 <bob> hey\n18:39 <alice> how are you\n18:40 <bob> good","expected_messages":4,"expected_participants":["alice","bob"]}
|
||||
{"fixture_id":"matrix-element-001","pattern":"matrix-element","frontmatter":{"date":"2024-03-15"},"body":"[18:37] @alice:matrix.org: hello world\n[18:38] @bob:example.org: hey\n[18:39] @alice:matrix.org: how are you\n[18:40] @bob:example.org: good","expected_messages":4,"expected_participants":["alice","bob"]}
|
||||
{"fixture_id":"teams-export-001","pattern":"teams-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example, 3/15/2024 6:37 PM: hello team\nBob Example, 3/15/2024 6:38 PM: hey\nAlice Example, 3/15/2024 6:39 PM: meeting at 4?\nBob Example, 3/15/2024 6:40 PM: works for me","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"fixture_id":"imessage-001","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice Example** (2024-03-15 9:00 AM): morning\n**Bob Example** (2024-03-15 9:01 AM): hey there\n**Alice Example** (2024-03-15 9:02 AM): how are you\n**Bob Example** (2024-03-15 9:03 AM): good thanks\n**Alice Example** (2024-03-15 9:04 AM): you?","expected_messages":5,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"imessage-002","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Charlie Example** (2024-03-15 2:00 PM): afternoon\n**Charlie Example** (2024-03-15 2:01 PM): are you there?\n**Diana Example** (2024-03-15 2:05 PM): yes\n**Charlie Example** (2024-03-15 2:06 PM): great","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"fixture_id":"irc-classic-001","pattern":"irc-classic","frontmatter":{"date":"2024-03-15"},"body":"<alice> hello world\n<bob> hey there\n<alice> how are you\n<bob> good thanks","expected_messages":4,"expected_participants":["alice","bob"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"fixture_id":"irc-weechat-001","pattern":"irc-weechat","frontmatter":{"date":"2024-03-15"},"body":"18:37 <alice> hello\n18:38 <bob> hey\n18:39 <alice> how are you\n18:40 <bob> good","expected_messages":4,"expected_participants":["alice","bob"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"fixture_id":"matrix-element-001","pattern":"matrix-element","frontmatter":{"date":"2024-03-15"},"body":"[18:37] @alice:matrix.org: hello world\n[18:38] @bob:example.org: hey\n[18:39] @alice:matrix.org: how are you\n[18:40] @bob:example.org: good","expected_messages":4,"expected_participants":["alice","bob"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"fixture_id":"signal-export-001","pattern":"signal-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example (2024-03-15 18:37:00 UTC): hello\nBob Example (2024-03-15 18:37:30 UTC): hey there\nAlice Example (2024-03-15 18:38:00 UTC): how are you\nBob Example (2024-03-15 18:39:00 UTC): good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"fixture_id":"teams-export-001","pattern":"teams-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example, 3/15/2024 6:37 PM: hello team\nBob Example, 3/15/2024 6:38 PM: hey\nAlice Example, 3/15/2024 6:39 PM: meeting at 4?\nBob Example, 3/15/2024 6:40 PM: works for me","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"fixture_id":"telegram-bracket-001","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-24","timezone":"America/Los_Angeles"},"body":"**[18:37] 👤 Alice Example:** hello world\n**[18:38] 👤 Bob Example:** hey\n**[18:39] 👤 Alice Example:** how are you\n**[18:40] 👤 Bob Example:** good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"telegram-bracket-002","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-25","timezone":"America/Los_Angeles"},"body":"**[06:00] 🤖 Zion Bot:** On it.\n**[06:01] 👤 Charlie Example:** thanks\n**[06:02] 🤖 Zion Bot:** anything else?\n**[06:03] 👤 Charlie Example:** no good","expected_messages":4,"expected_participants":["Zion Bot","Charlie Example"]}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"fixture_id":"whatsapp-iso-001","pattern":"whatsapp-iso","frontmatter":{"date":"2024-03-15"},"body":"[15/03/24, 18:37:00] Alice Example: hello\n[15/03/24, 18:37:30] Bob Example: hey\n[15/03/24, 18:38:00] Alice Example: how are you\n[15/03/24, 18:39:00] Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"whatsapp-iso-002","pattern":"whatsapp-iso","frontmatter":{"date":"2024-01-01"},"body":"[01/01/24, 00:00:00] Charlie Example: happy new year\n[01/01/24, 00:00:30] Diana Example: same to you\n[01/01/24, 00:01:00] Charlie Example: any plans?\n[01/01/24, 00:02:00] Diana Example: sleep","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"fixture_id":"whatsapp-us-001","pattern":"whatsapp-us","frontmatter":{"date":"2024-03-15"},"body":"3/15/24, 6:37 PM - Alice Example: hello\n3/15/24, 6:38 PM - Bob Example: hey\n3/15/24, 6:39 PM - Alice Example: how are you\n3/15/24, 6:40 PM - Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"whatsapp-us-002","pattern":"whatsapp-us","frontmatter":{"date":"2023-12-31"},"body":"12/31/23, 11:59 PM - Charlie Example: nye\n12/31/23, 11:59 PM - Diana Example: cheers","expected_messages":2,"expected_participants":["Charlie Example","Diana Example"]}
|
||||
@@ -0,0 +1,159 @@
|
||||
/**
|
||||
* v0.41.16.0 — Migration v99 round-trip test.
|
||||
*
|
||||
* Verifies the `conversation_parser_llm_cache` table:
|
||||
* - is created on schema init
|
||||
* - accepts inserts on (content_sha256, model_id, call_shape, value_json)
|
||||
* - rejects invalid call_shape via CHECK constraint
|
||||
* - ON CONFLICT DO NOTHING semantics (the llm-base.ts caller's contract)
|
||||
* - JSONB column round-trips a real object (no double-encode regression)
|
||||
* - composite primary key prevents duplicate (sha, model, shape)
|
||||
*
|
||||
* Hermetic via the canonical PGLite block from CLAUDE.md test-isolation
|
||||
* rules.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
describe('migration v99 — conversation_parser_llm_cache', () => {
|
||||
test('table exists after schema init', async () => {
|
||||
const rows = await engine.executeRaw<{ table_name: string }>(
|
||||
`SELECT table_name FROM information_schema.tables
|
||||
WHERE table_schema = 'public' AND table_name = 'conversation_parser_llm_cache'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
});
|
||||
|
||||
test('insert + select round-trip with polish call_shape', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO conversation_parser_llm_cache
|
||||
(content_sha256, model_id, call_shape, value_json)
|
||||
VALUES ($1, $2, $3, $4::jsonb)`,
|
||||
[
|
||||
'abc123',
|
||||
'anthropic:claude-haiku-4-5',
|
||||
'polish',
|
||||
JSON.stringify({ merge_indices: [], drop_indices: [], edits: [] }),
|
||||
],
|
||||
);
|
||||
const rows = await engine.executeRaw<{ value_json: unknown }>(
|
||||
`SELECT value_json FROM conversation_parser_llm_cache
|
||||
WHERE content_sha256 = $1 AND model_id = $2 AND call_shape = $3`,
|
||||
['abc123', 'anthropic:claude-haiku-4-5', 'polish'],
|
||||
);
|
||||
expect(rows).toHaveLength(1);
|
||||
// value_json should round-trip as a parsed object (not a JSON string).
|
||||
const val =
|
||||
typeof rows[0].value_json === 'string'
|
||||
? JSON.parse(rows[0].value_json)
|
||||
: rows[0].value_json;
|
||||
expect(val).toEqual({ merge_indices: [], drop_indices: [], edits: [] });
|
||||
});
|
||||
|
||||
test('insert + select round-trip with fallback call_shape', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO conversation_parser_llm_cache
|
||||
(content_sha256, model_id, call_shape, value_json)
|
||||
VALUES ($1, $2, $3, $4::jsonb)`,
|
||||
[
|
||||
'def456',
|
||||
'anthropic:claude-haiku-4-5',
|
||||
'fallback',
|
||||
JSON.stringify([
|
||||
{ speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'hi' },
|
||||
]),
|
||||
],
|
||||
);
|
||||
const rows = await engine.executeRaw<{ value_json: unknown }>(
|
||||
`SELECT value_json FROM conversation_parser_llm_cache
|
||||
WHERE content_sha256 = $1 AND call_shape = $2`,
|
||||
['def456', 'fallback'],
|
||||
);
|
||||
expect(rows).toHaveLength(1);
|
||||
const val =
|
||||
typeof rows[0].value_json === 'string'
|
||||
? JSON.parse(rows[0].value_json)
|
||||
: rows[0].value_json;
|
||||
expect(Array.isArray(val)).toBe(true);
|
||||
expect(val).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('CHECK constraint rejects invalid call_shape', async () => {
|
||||
let threw = false;
|
||||
try {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO conversation_parser_llm_cache
|
||||
(content_sha256, model_id, call_shape, value_json)
|
||||
VALUES ($1, $2, $3, $4::jsonb)`,
|
||||
['ghi789', 'anthropic:claude-haiku-4-5', 'INVALID_SHAPE', '{}'],
|
||||
);
|
||||
} catch {
|
||||
threw = true;
|
||||
}
|
||||
expect(threw).toBe(true);
|
||||
});
|
||||
|
||||
test('composite primary key prevents duplicate (sha, model, shape)', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO conversation_parser_llm_cache
|
||||
(content_sha256, model_id, call_shape, value_json)
|
||||
VALUES ($1, $2, $3, $4::jsonb)`,
|
||||
['dup1', 'anthropic:claude-haiku-4-5', 'polish', '{}'],
|
||||
);
|
||||
// ON CONFLICT DO NOTHING from llm-base.ts writeDbCache — should not throw.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO conversation_parser_llm_cache
|
||||
(content_sha256, model_id, call_shape, value_json)
|
||||
VALUES ($1, $2, $3, $4::jsonb)
|
||||
ON CONFLICT (content_sha256, model_id, call_shape) DO NOTHING`,
|
||||
['dup1', 'anthropic:claude-haiku-4-5', 'polish', '{"different":true}'],
|
||||
);
|
||||
// First write wins on conflict.
|
||||
const rows = await engine.executeRaw<{ value_json: unknown }>(
|
||||
`SELECT value_json FROM conversation_parser_llm_cache WHERE content_sha256 = 'dup1'`,
|
||||
);
|
||||
expect(rows).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('different call_shape on same (sha, model) coexists', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO conversation_parser_llm_cache
|
||||
(content_sha256, model_id, call_shape, value_json)
|
||||
VALUES ($1, $2, 'polish', $3::jsonb), ($1, $2, 'fallback', $3::jsonb)`,
|
||||
['co1', 'anthropic:claude-haiku-4-5', '{}'],
|
||||
);
|
||||
const rows = await engine.executeRaw<{ call_shape: string }>(
|
||||
`SELECT call_shape FROM conversation_parser_llm_cache WHERE content_sha256 = 'co1'`,
|
||||
);
|
||||
expect(rows).toHaveLength(2);
|
||||
const shapes = rows.map((r) => r.call_shape).sort();
|
||||
expect(shapes).toEqual(['fallback', 'polish']);
|
||||
});
|
||||
|
||||
test('created_at index supports time-based pruning queries', async () => {
|
||||
const rows = await engine.executeRaw<{ indexname: string }>(
|
||||
`SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'conversation_parser_llm_cache'
|
||||
AND indexname = 'idx_conversation_parser_llm_cache_created'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,625 @@
|
||||
/**
|
||||
* v0.41.16.0 — Progressive-batch primitive unit tests.
|
||||
*
|
||||
* Pins every verdict branch of runProgressiveBatch (D3 fail-closed
|
||||
* budget, D20 discriminated verifier shapes, D21 honest jump-to-full,
|
||||
* stage-slicing, ramp-stage interactive abort, cost projection).
|
||||
*
|
||||
* Hermetic: no PGLite, no Postgres, no real LLM. Every BudgetTracker
|
||||
* is constructed inline; the AsyncLocalStorage scope is set via
|
||||
* `withBudgetTracker` from gateway.ts. Audit JSONL writes go to
|
||||
* `GBRAIN_AUDIT_DIR=<tempdir>` to isolate from the user's real audit
|
||||
* dir. We use `withEnv` from test/helpers/with-env.ts per CLAUDE.md
|
||||
* R1 (test-isolation lint).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import {
|
||||
parseEnvStages,
|
||||
resolveStages,
|
||||
resolveCostCap,
|
||||
sliceIntoStages,
|
||||
awaitInteractiveAbort,
|
||||
runProgressiveBatch,
|
||||
} from '../../src/core/progressive-batch/orchestrator.ts';
|
||||
import { BudgetTracker } from '../../src/core/budget/budget-tracker.ts';
|
||||
import { withBudgetTracker } from '../../src/core/ai/gateway.ts';
|
||||
import type {
|
||||
Policy,
|
||||
StageReport,
|
||||
Verifier,
|
||||
} from '../../src/core/progressive-batch/types.ts';
|
||||
|
||||
// Helper: build a NoopVerifier with optional sampleQuality.
|
||||
function makeNoopVerifier(
|
||||
costPerItem = 0.001,
|
||||
sampleQuality?: () => Promise<{ ok: boolean; reasons?: string[] }>,
|
||||
): Verifier {
|
||||
return {
|
||||
kind: 'noop',
|
||||
costPerItem: () => costPerItem,
|
||||
sampleQuality,
|
||||
};
|
||||
}
|
||||
|
||||
// Helper: build an OutputCountVerifier backed by an in-memory counter.
|
||||
function makeOutputCountVerifier(opts: {
|
||||
initial?: number;
|
||||
perItemRows?: number;
|
||||
qualityOk?: boolean;
|
||||
qualityReasons?: string[];
|
||||
costPerItem?: number;
|
||||
}): { verifier: Verifier; bump: (n: number) => void } {
|
||||
let count = opts.initial ?? 0;
|
||||
const perItem = opts.perItemRows ?? 1;
|
||||
return {
|
||||
verifier: {
|
||||
kind: 'output_count',
|
||||
countBefore: async () => count,
|
||||
countAfter: async () => count,
|
||||
expectedDelta: (processed: number) => processed * perItem,
|
||||
sampleQuality: async () => ({
|
||||
ok: opts.qualityOk ?? true,
|
||||
reasons: opts.qualityReasons,
|
||||
}),
|
||||
costPerItem: () => opts.costPerItem ?? 0.001,
|
||||
},
|
||||
bump: (n: number) => {
|
||||
count += n;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Helper: build an IdempotentMutationVerifier backed by an in-memory counter.
|
||||
function makeIdempotentVerifier(opts: {
|
||||
qualityOk?: boolean;
|
||||
qualityReasons?: string[];
|
||||
mutationsPerItem?: number;
|
||||
costPerItem?: number;
|
||||
}): { verifier: Verifier; bump: (n: number) => void; resetForStage: () => void } {
|
||||
let mutations = 0;
|
||||
const perItem = opts.mutationsPerItem ?? 1;
|
||||
return {
|
||||
verifier: {
|
||||
kind: 'idempotent_mutation',
|
||||
mutatedCount: async () => mutations,
|
||||
expectedMutations: (processed: number) => processed * perItem,
|
||||
sampleQuality: async () => ({
|
||||
ok: opts.qualityOk ?? true,
|
||||
reasons: opts.qualityReasons,
|
||||
}),
|
||||
costPerItem: () => opts.costPerItem ?? 0.001,
|
||||
},
|
||||
bump: (n: number) => {
|
||||
mutations += n;
|
||||
},
|
||||
resetForStage: () => {
|
||||
mutations = 0;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Helper: collect stage reports the orchestrator emits.
|
||||
function collectReports(): {
|
||||
onStageReport: (r: StageReport) => void;
|
||||
reports: StageReport[];
|
||||
} {
|
||||
const reports: StageReport[] = [];
|
||||
return {
|
||||
onStageReport: (r) => {
|
||||
reports.push(r);
|
||||
},
|
||||
reports,
|
||||
};
|
||||
}
|
||||
|
||||
// Use a per-test tempdir for audit writes via GBRAIN_AUDIT_DIR.
|
||||
function makeAuditEnv(): Record<string, string> {
|
||||
return {
|
||||
GBRAIN_AUDIT_DIR: mkdtempSync(join(tmpdir(), 'pb-audit-')),
|
||||
GBRAIN_PROGRESSIVE_BATCH_AUTO: '1',
|
||||
GBRAIN_PROGRESSIVE_BATCH_DISABLED: undefined as unknown as string,
|
||||
GBRAIN_PROGRESSIVE_BATCH_STAGES: undefined as unknown as string,
|
||||
};
|
||||
}
|
||||
|
||||
describe('parseEnvStages', () => {
|
||||
test('null on unset', () => {
|
||||
expect(parseEnvStages(undefined)).toBeNull();
|
||||
});
|
||||
test('null on empty/whitespace', () => {
|
||||
expect(parseEnvStages('')).toBeNull();
|
||||
expect(parseEnvStages(' ')).toBeNull();
|
||||
});
|
||||
test('parses canonical', () => {
|
||||
expect(parseEnvStages('10,100,500')).toEqual([10, 100, 500]);
|
||||
});
|
||||
test('rejects non-int', () => {
|
||||
expect(parseEnvStages('10,foo,500')).toBeNull();
|
||||
});
|
||||
test('rejects zero/negative', () => {
|
||||
expect(parseEnvStages('0,100,500')).toBeNull();
|
||||
expect(parseEnvStages('-1,100,500')).toBeNull();
|
||||
});
|
||||
test('trims whitespace per entry', () => {
|
||||
expect(parseEnvStages(' 10 , 100 , 500 ')).toEqual([10, 100, 500]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveStages', () => {
|
||||
test('env override wins', async () => {
|
||||
await withEnv({ GBRAIN_PROGRESSIVE_BATCH_STAGES: '5,50,500' }, async () => {
|
||||
expect(resolveStages({ label: 't', stages: [10, 100, 500] })).toEqual([
|
||||
5, 50, 500,
|
||||
]);
|
||||
});
|
||||
});
|
||||
test('policy when env unset', async () => {
|
||||
await withEnv({ GBRAIN_PROGRESSIVE_BATCH_STAGES: undefined as unknown as string }, async () => {
|
||||
expect(resolveStages({ label: 't', stages: [7, 70, 700] })).toEqual([
|
||||
7, 70, 700,
|
||||
]);
|
||||
});
|
||||
});
|
||||
test('default when both unset', async () => {
|
||||
await withEnv({ GBRAIN_PROGRESSIVE_BATCH_STAGES: undefined as unknown as string }, async () => {
|
||||
expect(resolveStages({ label: 't' })).toEqual([10, 100, 500]);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveCostCap (D3 fail-closed)', () => {
|
||||
test('opt-out + maxCostUsd returns policy cap', () => {
|
||||
expect(
|
||||
resolveCostCap(
|
||||
{ label: 't', requireBudgetSafetyNet: false, maxCostUsd: 1 },
|
||||
null,
|
||||
),
|
||||
).toEqual({ capUsd: 1, source: 'policy' });
|
||||
});
|
||||
test('opt-out + no cap returns Infinity', () => {
|
||||
expect(
|
||||
resolveCostCap({ label: 't', requireBudgetSafetyNet: false }, null),
|
||||
).toEqual({ capUsd: Infinity, source: 'uncapped' });
|
||||
});
|
||||
test('no tracker + no policy cap = NULL (fail-closed)', () => {
|
||||
expect(resolveCostCap({ label: 't' }, null)).toBeNull();
|
||||
});
|
||||
test('policy cap + no tracker returns policy', () => {
|
||||
expect(resolveCostCap({ label: 't', maxCostUsd: 2 }, null)).toEqual({
|
||||
capUsd: 2,
|
||||
source: 'policy',
|
||||
});
|
||||
});
|
||||
test('tracker + no policy cap returns tracker headroom', () => {
|
||||
const t = new BudgetTracker({ label: 'test', maxCostUsd: 5 });
|
||||
const r = resolveCostCap({ label: 't' }, t);
|
||||
expect(r?.capUsd).toBeCloseTo(5);
|
||||
expect(r?.source).toBe('tracker');
|
||||
});
|
||||
test('both set: lower wins', () => {
|
||||
const t = new BudgetTracker({ label: 'test', maxCostUsd: 10 });
|
||||
// policy is tighter
|
||||
const r1 = resolveCostCap({ label: 't', maxCostUsd: 3 }, t);
|
||||
expect(r1?.capUsd).toBe(3);
|
||||
expect(r1?.source).toBe('policy');
|
||||
// tracker is tighter
|
||||
const r2 = resolveCostCap({ label: 't', maxCostUsd: 100 }, t);
|
||||
expect(r2?.capUsd).toBeCloseTo(10);
|
||||
expect(r2?.source).toBe('min');
|
||||
});
|
||||
});
|
||||
|
||||
describe('sliceIntoStages', () => {
|
||||
test('1000 items / canonical stages', () => {
|
||||
const items = Array.from({ length: 1000 }, (_, i) => i);
|
||||
const s = sliceIntoStages(items, [10, 100, 500]);
|
||||
expect(s.trial.length).toBe(10);
|
||||
expect(s.ramp_100.length).toBe(100);
|
||||
expect(s.ramp_500.length).toBe(500);
|
||||
expect(s.full.length).toBe(390);
|
||||
// Disjoint + ordered
|
||||
expect(s.trial[0]).toBe(0);
|
||||
expect(s.ramp_100[0]).toBe(10);
|
||||
expect(s.ramp_500[0]).toBe(110);
|
||||
expect(s.full[0]).toBe(610);
|
||||
});
|
||||
test('50 items: ramp_100 takes remaining; 500 + full empty', () => {
|
||||
const items = Array.from({ length: 50 }, (_, i) => i);
|
||||
const s = sliceIntoStages(items, [10, 100, 500]);
|
||||
expect(s.trial.length).toBe(10);
|
||||
expect(s.ramp_100.length).toBe(40);
|
||||
expect(s.ramp_500.length).toBe(0);
|
||||
expect(s.full.length).toBe(0);
|
||||
});
|
||||
test('5 items: only trial', () => {
|
||||
const items = Array.from({ length: 5 }, (_, i) => i);
|
||||
const s = sliceIntoStages(items, [10, 100, 500]);
|
||||
expect(s.trial.length).toBe(5);
|
||||
expect(s.ramp_100.length).toBe(0);
|
||||
expect(s.ramp_500.length).toBe(0);
|
||||
expect(s.full.length).toBe(0);
|
||||
});
|
||||
test('disabled (stages=[]) dumps everything to full', () => {
|
||||
const items = Array.from({ length: 100 }, (_, i) => i);
|
||||
const s = sliceIntoStages(items, []);
|
||||
expect(s.trial.length).toBe(0);
|
||||
expect(s.ramp_100.length).toBe(0);
|
||||
expect(s.ramp_500.length).toBe(0);
|
||||
expect(s.full.length).toBe(100);
|
||||
});
|
||||
});
|
||||
|
||||
describe('awaitInteractiveAbort', () => {
|
||||
test('ms=0 resolves false immediately', async () => {
|
||||
expect(await awaitInteractiveAbort(0)).toBe(false);
|
||||
});
|
||||
test('positive ms resolves false after timeout', async () => {
|
||||
const start = Date.now();
|
||||
const r = await awaitInteractiveAbort(50);
|
||||
expect(r).toBe(false);
|
||||
expect(Date.now() - start).toBeGreaterThanOrEqual(40);
|
||||
});
|
||||
});
|
||||
|
||||
describe('runProgressiveBatch — D3 fail-closed safety net', () => {
|
||||
test('NO tracker + NO Policy.maxCostUsd → abort_cost_cap reason=no_budget_safety_net', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { onStageReport, reports } = collectReports();
|
||||
const result = await runProgressiveBatch(
|
||||
[1, 2, 3],
|
||||
makeNoopVerifier(),
|
||||
{ label: 'd3-test', onStageReport },
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_cost_cap');
|
||||
expect(result.abortedAt?.reason).toBe('no_budget_safety_net');
|
||||
expect(result.itemsProcessed).toBe(0);
|
||||
expect(reports.length).toBe(1);
|
||||
expect(reports[0].verdict).toBe('abort_cost_cap');
|
||||
expect(reports[0].abortReason).toBe('no_budget_safety_net');
|
||||
});
|
||||
});
|
||||
test('Policy.requireBudgetSafetyNet=false bypasses the gate (Infinity cap)', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { onStageReport, reports } = collectReports();
|
||||
const result = await runProgressiveBatch(
|
||||
[1, 2, 3],
|
||||
makeNoopVerifier(),
|
||||
{
|
||||
label: 'optout',
|
||||
onStageReport,
|
||||
requireBudgetSafetyNet: false,
|
||||
},
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(3);
|
||||
// 3 items < trial(10) → only trial runs (the rest are empty
|
||||
// stages with no audit row).
|
||||
expect(reports.length).toBe(1);
|
||||
expect(reports[0].verdict).toBe('proceed');
|
||||
});
|
||||
});
|
||||
test('Tracker present + no Policy.maxCostUsd → tracker headroom wins', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const t = new BudgetTracker({ label: 'unit', maxCostUsd: 10 });
|
||||
await withBudgetTracker(t, async () => {
|
||||
const { onStageReport } = collectReports();
|
||||
const result = await runProgressiveBatch(
|
||||
[1, 2, 3],
|
||||
makeNoopVerifier(0.001),
|
||||
{ label: 'tracker-test', onStageReport },
|
||||
async (slice) => ({
|
||||
succeeded: slice.length,
|
||||
failed: 0,
|
||||
costUsd: 0.003,
|
||||
}),
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(3);
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runProgressiveBatch — verifier shapes (D20)', () => {
|
||||
test('OutputCountVerifier: matched delta proceeds', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { verifier, bump } = makeOutputCountVerifier({ perItemRows: 1 });
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 5 }, (_, i) => i),
|
||||
verifier,
|
||||
{ label: 'oc', maxCostUsd: 1 },
|
||||
async (slice) => {
|
||||
bump(slice.length);
|
||||
return { succeeded: slice.length, failed: 0, costUsd: 0 };
|
||||
},
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(5);
|
||||
});
|
||||
});
|
||||
test('OutputCountVerifier: zero delta → abort_count_mismatch', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { verifier } = makeOutputCountVerifier({ perItemRows: 1 });
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 10 }, (_, i) => i),
|
||||
verifier,
|
||||
{ label: 'oc-mismatch', maxCostUsd: 1 },
|
||||
async (slice) => ({
|
||||
// Runner reports success but verifier sees no new rows.
|
||||
succeeded: slice.length,
|
||||
failed: 0,
|
||||
costUsd: 0,
|
||||
}),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_count_mismatch');
|
||||
expect(result.abortedAt?.reason).toBe('count_delta_outside_band');
|
||||
});
|
||||
});
|
||||
test('IdempotentMutationVerifier: matched mutation count proceeds', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { verifier, bump } = makeIdempotentVerifier({});
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 5 }, (_, i) => i),
|
||||
verifier,
|
||||
{ label: 'im', maxCostUsd: 1 },
|
||||
async (slice) => {
|
||||
bump(slice.length);
|
||||
return { succeeded: slice.length, failed: 0, costUsd: 0 };
|
||||
},
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
});
|
||||
});
|
||||
test('NoopVerifier: only cost + error rate gating', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 5 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{ label: 'np', maxCostUsd: 1 },
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(5);
|
||||
});
|
||||
});
|
||||
test('NoopVerifier with sampleQuality returning not-ok → abort_data_quality', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const result = await runProgressiveBatch(
|
||||
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
|
||||
makeNoopVerifier(0.001, async () => ({
|
||||
ok: false,
|
||||
reasons: ['bad row 7'],
|
||||
})),
|
||||
{ label: 'np-bad-quality', maxCostUsd: 1 },
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_data_quality');
|
||||
expect(result.abortedAt?.reason).toBe('data_quality_sample_failed');
|
||||
const trial = result.stageReports[0];
|
||||
expect(trial.qualityReasons).toEqual(['bad row 7']);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runProgressiveBatch — error rate + cost cap gates', () => {
|
||||
test('error rate > maxErrorRate → abort_error_rate', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 10 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{ label: 'er', maxCostUsd: 1, maxErrorRate: 0.1 },
|
||||
async (slice) => ({
|
||||
// 50% fail rate, well over the 10% threshold.
|
||||
succeeded: Math.floor(slice.length / 2),
|
||||
failed: Math.ceil(slice.length / 2),
|
||||
costUsd: 0,
|
||||
}),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_error_rate');
|
||||
expect(result.abortedAt?.reason).toBe('error_rate_exceeded');
|
||||
});
|
||||
});
|
||||
test('cost projection > cap → abort_cost_cap', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 1000 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{ label: 'cc', maxCostUsd: 0.5 },
|
||||
async (slice) => ({
|
||||
// $0.01/item → 10 items = $0.10 in trial, projected
|
||||
// 1000-item run = $10.00 ≫ $0.50 cap.
|
||||
succeeded: slice.length,
|
||||
failed: 0,
|
||||
costUsd: slice.length * 0.01,
|
||||
}),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_cost_cap');
|
||||
expect(result.abortedAt?.reason).toBe('cost_projected_over_cap');
|
||||
// Trial stage should have processed 10; cumulative cost should be 0.10.
|
||||
expect(result.itemsProcessed).toBe(10);
|
||||
expect(result.totalCostUsd).toBeCloseTo(0.1);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runProgressiveBatch — D21 honest behavior preservation', () => {
|
||||
test('GBRAIN_PROGRESSIVE_BATCH_DISABLED=1 skips ramp; goes to full', async () => {
|
||||
await withEnv(
|
||||
{
|
||||
...makeAuditEnv(),
|
||||
GBRAIN_PROGRESSIVE_BATCH_DISABLED: '1',
|
||||
},
|
||||
async () => {
|
||||
const { onStageReport, reports } = collectReports();
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 100 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{ label: 'disabled', maxCostUsd: 1, onStageReport },
|
||||
async (slice) => ({
|
||||
succeeded: slice.length,
|
||||
failed: 0,
|
||||
costUsd: 0,
|
||||
}),
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(100);
|
||||
// Only the 'full' stage report should be emitted.
|
||||
expect(reports.length).toBe(1);
|
||||
expect(reports[0].stage).toBe('full');
|
||||
expect(reports[0].itemsInStage).toBe(100);
|
||||
},
|
||||
);
|
||||
});
|
||||
test('interactiveAbortMs=0 → no Ctrl-C wait between stages', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const start = Date.now();
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 120 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{ label: 'noramp', maxCostUsd: 1, interactiveAbortMs: 0 },
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
// Should run quickly without any 10s waits between stages.
|
||||
expect(Date.now() - start).toBeLessThan(500);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runProgressiveBatch — onStageReport caller abort', () => {
|
||||
test('caller returns {abort:true} after trial → abort_explicit', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
let calls = 0;
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 100 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{
|
||||
label: 'explicit-abort',
|
||||
maxCostUsd: 1,
|
||||
onStageReport: (r) => {
|
||||
calls++;
|
||||
if (r.stage === 'trial') return { abort: true };
|
||||
return undefined;
|
||||
},
|
||||
},
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_explicit');
|
||||
expect(result.abortedAt?.reason).toBe('caller_signaled_abort');
|
||||
// Trial completed but caller signaled abort.
|
||||
expect(result.itemsProcessed).toBe(10);
|
||||
expect(calls).toBe(1);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runProgressiveBatch — multi-stage success path', () => {
|
||||
test('1000 items: trial(10) → ramp_100 → ramp_500 → full(380); all stages reported', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { onStageReport, reports } = collectReports();
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 1000 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{ label: 'multi', maxCostUsd: 100, onStageReport },
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(1000);
|
||||
expect(result.stagesCompleted).toEqual(['trial', 'ramp_100', 'ramp_500', 'full']);
|
||||
// Four stage reports (one per stage; the full stage processes 380).
|
||||
expect(reports).toHaveLength(4);
|
||||
expect(reports[0].stage).toBe('trial');
|
||||
expect(reports[0].itemsInStage).toBe(10);
|
||||
expect(reports[1].stage).toBe('ramp_100');
|
||||
expect(reports[1].itemsInStage).toBe(100);
|
||||
expect(reports[2].stage).toBe('ramp_500');
|
||||
expect(reports[2].itemsInStage).toBe(500);
|
||||
expect(reports[3].stage).toBe('full');
|
||||
expect(reports[3].itemsInStage).toBe(390);
|
||||
// Cumulative processed reaches 1000 by the last stage.
|
||||
expect(reports[3].itemsProcessedCumulative).toBe(1000);
|
||||
// All verdicts are proceed.
|
||||
expect(reports.every((r) => r.verdict === 'proceed')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
test('IdempotentMutationVerifier: matched mutation count across stages', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { verifier, bump, resetForStage } = makeIdempotentVerifier({});
|
||||
void resetForStage;
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 25 }, (_, i) => i),
|
||||
verifier,
|
||||
{ label: 'multi-im', maxCostUsd: 1 },
|
||||
async (slice) => {
|
||||
bump(slice.length);
|
||||
return { succeeded: slice.length, failed: 0, costUsd: 0 };
|
||||
},
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(25);
|
||||
});
|
||||
});
|
||||
|
||||
test('IdempotentMutationVerifier: mutation count off → abort_mutation_mismatch', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { verifier } = makeIdempotentVerifier({});
|
||||
// Runner claims success but never bumps the mutation counter.
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 10 }, (_, i) => i),
|
||||
verifier,
|
||||
{ label: 'im-mm', maxCostUsd: 1 },
|
||||
async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_mutation_mismatch');
|
||||
expect(result.abortedAt?.reason).toBe('mutation_count_outside_band');
|
||||
});
|
||||
});
|
||||
|
||||
test('cumulative cost crosses cap mid-multi-stage → abort_cost_cap', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const result = await runProgressiveBatch(
|
||||
Array.from({ length: 200 }, (_, i) => i),
|
||||
makeNoopVerifier(),
|
||||
{ label: 'cc-mid', maxCostUsd: 0.05 },
|
||||
async (slice) => ({
|
||||
// Trial: 10 items × $0.001 = $0.01 cumulative; projected
|
||||
// 200 × $0.001 = $0.20 ≫ $0.05 cap.
|
||||
succeeded: slice.length,
|
||||
failed: 0,
|
||||
costUsd: slice.length * 0.001,
|
||||
}),
|
||||
);
|
||||
expect(result.abortedAt?.verdict).toBe('abort_cost_cap');
|
||||
expect(result.abortedAt?.reason).toBe('cost_projected_over_cap');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runProgressiveBatch — degenerate inputs', () => {
|
||||
test('empty item list runs no stages but reports the run happened', async () => {
|
||||
await withEnv(makeAuditEnv(), async () => {
|
||||
const { onStageReport, reports } = collectReports();
|
||||
const result = await runProgressiveBatch(
|
||||
[] as number[],
|
||||
makeNoopVerifier(),
|
||||
{ label: 'empty', maxCostUsd: 1, onStageReport },
|
||||
async () => ({ succeeded: 0, failed: 0, costUsd: 0 }),
|
||||
);
|
||||
expect(result.abortedAt).toBeUndefined();
|
||||
expect(result.itemsProcessed).toBe(0);
|
||||
// We DO emit one report for the zero-item full stage so audit
|
||||
// shows the run was attempted.
|
||||
expect(reports.length).toBe(1);
|
||||
expect(reports[0].stage).toBe('full');
|
||||
expect(reports[0].verdict).toBe('proceed');
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* v0.41.16.0 — retrofitWrap helper unit tests.
|
||||
*
|
||||
* Pins the thin wrapper used by 8 of the 9 v0.41.16.0 retrofits:
|
||||
* - Default opt-out of D3 safety net + zero ramp grace.
|
||||
* - Audit JSONL still writes.
|
||||
* - Cost projection rolls up correctly.
|
||||
* - Runner error counts surface.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import { retrofitWrap } from '../../src/core/progressive-batch/retrofit-wrap.ts';
|
||||
|
||||
function auditEnv(): Record<string, string> {
|
||||
return {
|
||||
GBRAIN_AUDIT_DIR: mkdtempSync(join(tmpdir(), 'rw-audit-')),
|
||||
GBRAIN_PROGRESSIVE_BATCH_AUTO: '1',
|
||||
};
|
||||
}
|
||||
|
||||
describe('retrofitWrap', () => {
|
||||
test('opt-out defaults: runs without BudgetTracker safety net', async () => {
|
||||
await withEnv(auditEnv(), async () => {
|
||||
const r = await retrofitWrap({
|
||||
label: 'unit-test',
|
||||
items: [1, 2, 3, 4, 5],
|
||||
runner: async (rows) => ({ succeeded: rows.length, failed: 0, costUsd: 0 }),
|
||||
});
|
||||
expect(r.abortedAt).toBeUndefined();
|
||||
expect(r.itemsProcessed).toBe(5);
|
||||
});
|
||||
});
|
||||
|
||||
test('passes per-item cost into projection', async () => {
|
||||
await withEnv(auditEnv(), async () => {
|
||||
const r = await retrofitWrap({
|
||||
label: 'cost-projection',
|
||||
items: Array.from({ length: 100 }, (_, i) => i),
|
||||
costPerItem: 0.001,
|
||||
runner: async (rows) => ({
|
||||
succeeded: rows.length,
|
||||
failed: 0,
|
||||
costUsd: rows.length * 0.001,
|
||||
}),
|
||||
});
|
||||
expect(r.abortedAt).toBeUndefined();
|
||||
expect(r.totalCostUsd).toBeCloseTo(0.1);
|
||||
});
|
||||
});
|
||||
|
||||
test('runner failures roll up into the totals', async () => {
|
||||
await withEnv(auditEnv(), async () => {
|
||||
const r = await retrofitWrap({
|
||||
label: 'failure-test',
|
||||
items: Array.from({ length: 10 }, (_, i) => i),
|
||||
runner: async (rows) => ({
|
||||
succeeded: 0,
|
||||
failed: rows.length,
|
||||
costUsd: 0,
|
||||
}),
|
||||
});
|
||||
// Default maxErrorRate=0.02 — 100% failure aborts at trial.
|
||||
expect(r.abortedAt?.verdict).toBe('abort_error_rate');
|
||||
});
|
||||
});
|
||||
|
||||
test('interactiveAbortMs=0 default → no grace period (cron-safe)', async () => {
|
||||
await withEnv(auditEnv(), async () => {
|
||||
const start = Date.now();
|
||||
const r = await retrofitWrap({
|
||||
label: 'cron-safe',
|
||||
items: Array.from({ length: 200 }, (_, i) => i),
|
||||
runner: async (rows) => ({ succeeded: rows.length, failed: 0, costUsd: 0 }),
|
||||
});
|
||||
expect(r.abortedAt).toBeUndefined();
|
||||
// Should sail through all 4 stages with no grace pause.
|
||||
expect(Date.now() - start).toBeLessThan(500);
|
||||
});
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user