mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(eval-longmemeval): RunOpts.engine seam for shared benchmark brain Adds optional `engine?: PGLiteEngine` field to RunOpts. When set, runEvalLongMemEval uses the caller-provided engine and skips the withBenchmarkBrain wrapper (no fresh PGLite create, no disconnect on exit). When unset, the production CLI path is unchanged: withBenchmarkBrain creates and disposes a fresh engine per invocation. Designed for the test seam that's about to land: one beforeAll-created brain shared across all 13 runEvalLongMemEval calls in test/eval-longmemeval-e2e.slow.test.ts, amortizing the ~1-3s PGLite cold-create cost. runOneQuestion already calls resetTables() as its first line so per-test isolation is preserved across the shared engine. Pure additive seam — every existing caller (CLI, current tests that already create engines via withBenchmarkBrain implicitly) keeps its current behavior because opts.engine defaults to undefined. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(test): split eval-longmemeval slow tests + share engine across e2e half The 884-line test/eval-longmemeval.slow.test.ts was the heaviest single file in CI at ~359s on the matrix. Split by runEvalLongMemEval usage: - test/eval-longmemeval.slow.test.ts (trimmed): 8 pure describes, 15 tests. Harness lifecycle, resetTables, schema-migration robustness, warm-create speed gate, adapter haystackToPages, source-boost guard, loadResumeSet, buildByTypeSummary. Local wall: 1.985s, projected CI ~42s. - test/eval-longmemeval-e2e.slow.test.ts (NEW): 8 e2e describes, 11 tests. Every describe that calls runEvalLongMemEval — 13 call sites total. Threads a single beforeAll-created PGLite via the v0.41.10 RunOpts.engine seam. Local wall: 9.33s (was 15.09s without sharing); projected CI ~196s (was ~317s). - test/helpers/longmemeval-stub.ts (NEW): shared makeStubClient + StubCall. Matches the existing test/helpers/ convention (with-env.ts, reset-pglite.ts). Single source of truth across the two split files. - scripts/test-weights.json: replaced 359087ms entry with TWO entries (42000ms pure, 196000ms e2e). Projected linearly from local wall-clock × 21 CI scaling factor. First post-merge CI run will refine via scripts/mine-shard-weights.ts. Test count is preserved: 15 pure + 11 e2e = 26, matches original file. No production code changes in this commit — only test reorganization + opt-in to the RunOpts.engine seam from the previous commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(test): bump matrix 6→10, dedicate two slow files, cache bun-install CI matrix wallclock: ~9 min → ~4.5 min. Three coordinated changes. 1. .github/workflows/test.yml matrix bumped from 6 → 10 shards. Per-shard total drops from 532s → 272s. Honest concurrency-budget call: total gated jobs go 13 → 18, so 2 concurrent PRs ≈ 36 queued, past the GH free-tier ~20 ceiling — single-PR runs unaffected, multi-PR days see queue pressure. Worth it for the 4-min CI saving. 2. Two slow files pulled out of the matrix and into their own dedicated jobs (sibling to verify, serial-tests): - slow-eval-longmemeval runs test/eval-longmemeval-e2e.slow.test.ts (~196s after the engine-sharing seam from the previous two commits). - slow-entity-resolve-perf runs test/entity-resolve-perf.slow.test.ts (~159s, single non-subdivisible perf test). The 60s default bun timeout is too tight for this file — bumped to 300000ms. scripts/test-shard.sh excludes both via -not -name clauses so the matrix sweep doesn't double-run them. Both new jobs wire into cache-write.needs and test-status.needs so CI gates on them. 3. actions/cache for ~/.bun/install/cache added to every job that runs bun install (test matrix, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf). Keyed on bun.lock hash. Saves ~15s per job on cache hit; first-PR push pays full cost, subsequent runs hit cache. Total CI wallclock now bounded by max(matrix ~4.5min, slow-eval ~3.3min, slow-entity-resolve-perf ~2.6min) = ~4.5 min. The matrix is back to being the floor; no single test file dominates a shard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.41.10.0 — CI wallclock 9min → 4.5min VERSION + package.json + CHANGELOG entry for the three preceding commits: feat(eval-longmemeval): RunOpts.engine seam for shared benchmark brain refactor(test): split eval-longmemeval slow tests + share engine across e2e half ci(test): bump matrix 6→10, dedicate two slow files, cache bun-install Net user-visible: CI 'Test' check finishes in ~4.5 min instead of ~9 min. Net contributor-visible: new RunOpts.engine seam on runEvalLongMemEval for benchmark suites that want to amortize PGLite cold-create across many calls. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): quarantine hybrid-meta + schema-pack-load-active to serial The 6→10 matrix shard bump in this branch re-shuffled file distribution across shard processes. Two pre-existing tests with hidden cross-file state dependencies surfaced as failures in CI run #77779498812/13: - test/hybrid-meta.test.ts shard 7: gateway state (configured by some other test in the same shard process) survived past the test's `delete process.env.OPENAI_API_KEY` call, so the early-return for expansion didn't fire and `expansion_applied` stayed true. - test/schema-pack-load-active.test.ts shard 8: the schema-pack module's test-injected locator state was left behind by an earlier file, so `loadActivePack` with the default config didn't fall through to the bundled gbrain-base path. Both files pass cleanly solo (verified). The pollution sources are unidentified — bun's reporter only printed 14 of 71 file headers per shard log, hiding the polluters. Rather than spelunk for the source, rename both files to *.serial.test.ts. The serial pass runs them at --max-concurrency=1 in a process that doesn't share state with the parallel matrix shards. Same-wave bookkeeping: - scripts/check-test-isolation.allowlist: drop test/hybrid-meta.test.ts entry (file is now serial, no longer R1-eligible). - scripts/test-weights.json: rename both weight entries to match the new filenames so future matrix LPT runs don't fall back to median. Companion to a7d029d0/2e1c269e/5a749acb of this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
103 lines
3.8 KiB
TypeScript
103 lines
3.8 KiB
TypeScript
/**
|
|
* hybridSearch meta-field accuracy (v0.25.0, callback-based API).
|
|
*
|
|
* v0.25.0 keeps hybridSearch's return as `Promise<SearchResult[]>` (so
|
|
* Cathedral II callers stay unchanged) and surfaces meta via an optional
|
|
* `onMeta` callback in HybridSearchOpts. Asserts the callback fires with
|
|
* accurate values:
|
|
* - vector_enabled=false when OPENAI_API_KEY missing (keyword-only path)
|
|
* - detail_resolved reflects auto-detect + caller override
|
|
* - expansion_applied only true when expandFn returned variants
|
|
*
|
|
* Uses PGLite in-memory + no embedding calls (vector path doesn't need
|
|
* real embeddings to test the meta flag since we control the env).
|
|
*/
|
|
|
|
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import { hybridSearch } from '../src/core/search/hybrid.ts';
|
|
import type { PageInput, HybridSearchMeta } from '../src/core/types.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
const savedKey = process.env.OPENAI_API_KEY;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
const page: PageInput = {
|
|
type: 'person',
|
|
title: 'Alice Example',
|
|
compiled_truth: 'Alice Example is a test person for hybrid-meta tests.',
|
|
};
|
|
await engine.putPage('people/alice-example', page);
|
|
});
|
|
|
|
afterAll(async () => {
|
|
if (savedKey === undefined) delete process.env.OPENAI_API_KEY;
|
|
else process.env.OPENAI_API_KEY = savedKey;
|
|
await engine.disconnect();
|
|
});
|
|
|
|
async function runWithMeta(query: string, opts: Parameters<typeof hybridSearch>[2] = {}): Promise<HybridSearchMeta | null> {
|
|
let captured: HybridSearchMeta | null = null;
|
|
await hybridSearch(engine, query, { ...opts, onMeta: (m) => { captured = m; } });
|
|
return captured;
|
|
}
|
|
|
|
describe('hybridSearch return shape (v0.25.0 keeps SearchResult[])', () => {
|
|
test('returns SearchResult[] (unchanged from Cathedral II contract)', async () => {
|
|
delete process.env.OPENAI_API_KEY;
|
|
const out = await hybridSearch(engine, 'alice');
|
|
expect(Array.isArray(out)).toBe(true);
|
|
});
|
|
});
|
|
|
|
describe('hybridSearch onMeta callback — vector_enabled', () => {
|
|
test('false when OPENAI_API_KEY is missing (keyword-only path)', async () => {
|
|
delete process.env.OPENAI_API_KEY;
|
|
const meta = await runWithMeta('alice');
|
|
expect(meta).not.toBeNull();
|
|
expect(meta!.vector_enabled).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe('hybridSearch onMeta callback — detail_resolved', () => {
|
|
test('passes through explicit detail override (caller specified "high")', async () => {
|
|
delete process.env.OPENAI_API_KEY;
|
|
const meta = await runWithMeta('alice', { detail: 'high' });
|
|
expect(meta!.detail_resolved).toBe('high');
|
|
});
|
|
|
|
test('detail_resolved reflects autoDetect output when caller omits detail', async () => {
|
|
delete process.env.OPENAI_API_KEY;
|
|
const meta = await runWithMeta('alice');
|
|
expect([null, 'low', 'medium', 'high']).toContain(meta!.detail_resolved);
|
|
});
|
|
});
|
|
|
|
describe('hybridSearch onMeta callback — expansion_applied', () => {
|
|
test('false when expansion flag is off', async () => {
|
|
delete process.env.OPENAI_API_KEY;
|
|
const meta = await runWithMeta('alice', { expansion: false });
|
|
expect(meta!.expansion_applied).toBe(false);
|
|
});
|
|
|
|
test('false when OPENAI_API_KEY missing (early-return short-circuits expansion)', async () => {
|
|
delete process.env.OPENAI_API_KEY;
|
|
const meta = await runWithMeta('alice', {
|
|
expansion: true,
|
|
expandFn: async () => ['alice', 'alice example', 'the person alice'],
|
|
});
|
|
expect(meta!.expansion_applied).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe('onMeta callback omitted', () => {
|
|
test('hybridSearch works without onMeta (existing Cathedral II callers unaffected)', async () => {
|
|
delete process.env.OPENAI_API_KEY;
|
|
const out = await hybridSearch(engine, 'alice');
|
|
expect(Array.isArray(out)).toBe(true);
|
|
});
|
|
});
|