mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(v0.33): add SearchOpts.types multi-type filter to searchHybrid Push the page-type filter into SQL via AND p.type = ANY(\$N::text[]) in both engines' searchKeyword + searchVector + searchKeywordChunks paths. Primary consumer is the upcoming gbrain whoknows command (filters to ['person','company']); the limit budget then goes to typed candidates instead of being eaten by note/transcript/article pages. Future entity-only search in v0.34+ reuses the parameter for free. AND-applies alongside the existing single-value type filter (callers can use either or both). HybridSearchOpts threads opts.types into the underlying searchOpts so hybridSearch callers get the SQL-level filter without any post-filter waste. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.33): whoknows core ranking function + 10 locked unit tests Implements ENG-D1's locked spec: score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience). raw_match comes from hybridSearch's RRF + source-boost-adjusted score; salience and recency boosts in hybridSearch are intentionally disabled so the formula applies on a clean signal. rankCandidates() is the pure function the eval grades against; findExperts() is the public entrypoint that wires hybrid search + batch salience/effective_date fetches; runWhoknows() is the CLI. Test/whoknows.test.ts covers the 10 ENG-D3 cases (zero results, negative recency floor, NaN salience neutral default, NaN match zeros gracefully, type preservation, --explain factor breakdown, top-K limit clamping, recency-floor extreme-days safety, alphabetical tie-break determinism, public-surface contract). Plus four sanity asserts (higher-match outranks, more-recent outranks, higher-salience outranks, all-zero candidate appears with score 0). Plus one factor decomposition assertion that pins the exact formula numerically. Plus a composite-key safety case (Codex F1). 22 expect calls across 16 tests. All passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.33): register find_experts MCP op + gbrain whoknows CLI Wires both surfaces per ENG-D5: MCP op = find_experts (matches find_anomalies naming convention; agent-facing); CLI command = gbrain whoknows (memorable, user-facing). One findExperts() core function backs both paths. The op is scope:'read', localOnly:false — accessible over HTTP MCP to read-scoped OAuth clients like the salience/anomalies family. Op handler validates non-empty topic and dispatches to the same findExperts() pure function the CLI uses. CLI dispatch in src/cli.ts:case 'whoknows' calls runWhoknows; thin- client routing happens inside runWhoknows via isThinClient(cfg) — remote MCP installs route through the v0.31.1 routing seam to callRemoteTool('find_experts', ...). FIND_EXPERTS_DESCRIPTION in operations-descriptions.ts mirrors the v0.29 redirect-hint style: leads with what the tool does, lists explicit user-intent triggers ("who should I talk to about X", "who knows about Y"), notes the type-filter behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.33): gbrain eval whoknows — two-layer eval gate (ENG-D2) Implements the locked spec: Layer 1 hand-labeled fixture (>=80% top-3 hit rate) is the primary ship-blocking gate; Layer 2 eval_candidates replay (>=0.4 mean set-Jaccard@3) is the regression gate that auto-skips when < 20 replay-eligible rows exist (CONTRIBUTOR_MODE sparseness fallback). Dispatch lands as `gbrain eval whoknows <fixture.jsonl>` sub-subcommand in src/commands/eval.ts (mirrors v0.25.0 export/prune/replay and v0.27.x cross-modal pattern). Exits 0/1/2 for pass/fail/usage so CI gates can consume. JSON output (--json) ships schema_version: 1 for stable consumer contract (mirrors v0.25.0 eval-replay.ts). Human output groups by layer + emits a per-miss diagnostic table so failures are self-debugging. Unit tests pin: - jaccardAtK math (7 cases — identical, disjoint, partial, k cutoff, empty-empty vacuous-stable, empty-vs-non-empty, Set dedup) - topKHit (7 cases — position 1, 3, 4, miss, multi-expected, empty actual, empty expected) - readFixture (6 cases — well-formed, comments/blanks, missing file, malformed JSON, missing required fields, non-string filter) - Locked thresholds (HIT_RATE=0.8, REGRESSION=0.4, MIN_REPLAY_ROWS=20) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.33): gbrain doctor adds whoknows_health check Per CEO-D7 (substrate-conditional v0.33 doctor check, but the fixture-presence sub-check ships in week 1 regardless — it's the "did you do the assignment?" signal). When the eval fixture is missing, empty, or undersized (< 5 rows), doctor warns with the exact path the user should populate. The check is intentionally lightweight: it does NOT run the eval itself or measure hit-rate regression. That's the job of `gbrain eval whoknows`, called from CI/ship time. This check is the cheap always-runs signal that surfaces in `gbrain doctor` and on the ship review dashboard. 5 unit cases pin the four-status behavior (missing/empty/undersized/ ok) plus the comment-and-blank-line filtering so users can comment out queries during iteration without breaking the row count. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.33): synthetic whoknows eval fixture + E2E quality gate test test/fixtures/whoknows-eval.jsonl ships as a 10-query placeholder demonstrating the schema. Comments document the assignment for end users: they replace these with their own real queries before shipping their gbrain install. The placeholder uses obviously- example slugs (wiki/people/example-alice, etc.) so nobody mistakes it for production data. test/e2e/whoknows.test.ts seeds a synthetic PGLite brain that matches the placeholder fixture, then runs findExperts on every fixture query and asserts >=80% top-3 hit rate per ENG-D2 quality gate. Also exercises the typeFilter (concept-decoy pages filtered out), empty-result graceful return, --explain factor breakdown, and top-K limit honoring. Basis-vector embeddings (no API key) follow the existing pattern from test/e2e/search-quality.test.ts. 5 test cases, 23 expect calls, all passing against PGLite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v0.33): VERSION bump + CHANGELOG + CLAUDE.md + llms regen Bumps VERSION 0.31.11 → 0.33.0 and package.json to match. CHANGELOG entry leads with the headline use ("ask gbrain who knows about X") and the locked ENG-D1 ranking formula. "Numbers that matter" replaced with a "what ships on which eval outcome" table — honest about the eval-gated trajectory rather than fabricating benchmarks before the release has been graded against a real brain. CLAUDE.md Key Files annotations added for src/commands/whoknows.ts, src/commands/eval-whoknows.ts, and test/fixtures/whoknows-eval.jsonl. src/core/search/hybrid.ts entry extended with the new types parameter documentation (push the type filter to SQL, no post-filter waste, AND-applies alongside the existing single-value type field). bun run build:llms ran the chaser; llms.txt + llms-full.txt regenerated to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.33): unit-test gap fill — engine typeFilter + find_experts op Two new files filling the gaps Garry called out: test/search-types-filter.test.ts — engine-level coverage on PGLite for the new SearchOpts.types filter. Asserts the SQL-clause behavior directly so a regression in the AND p.type = ANY(...) emission gets caught here with a tight assertion rather than as part of a longer findExperts pipeline. 9 cases across searchKeyword + searchVector + chunk-grain documentation. Documents the pre-existing PGLite parity gap (single-value `type` field is Postgres-only; `types` is the v0.33 multi-type filter that BOTH engines honor). test/find-experts-op.test.ts — MCP-op contract test for find_experts. Pins: - Registered in the operations array + operationsByName - scope: 'read', localOnly false (HTTP-MCP accessible per ENG-D5) - Documented params (topic / limit / explain) with correct types - cliHints.name === 'whoknows' (CLI surface bridge) - Non-trivial description that references the use case - Handler rejects empty / whitespace / missing topic with invalid_params - Handler returns array shape on valid topic - Handler honors limit param 11 op-contract cases + 9 engine-clause cases. All passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version to v0.33.1.0 Garry asked for v0.33.1 instead of v0.33.0 (queue collision with unrelated 0.33.0 work). 4-digit format: 0.33.1.0. CHANGELOG header and "To take advantage of" block updated. llms.txt regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.33.1.1): cliHints.positional on find_experts so CLI accepts <topic> Without `cliHints.positional: ['topic']`, the op-dispatch path in src/cli.ts couldn't parse `gbrain whoknows "ai agents"` and threw `invalid_params: topic is required`. Found while testing the v0.33.1.0 build against a real brain. The op handler validates topic; the CLI just needed to know the positional shape so the dispatcher could hand it through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.33.1.2): real-brain whoknows-eval fixture from VC intro network Replaces the synthetic 10-row placeholder with 10 real expertise-routing queries mined from Garry's actual brain via thin-client connection to Wintermute (v0.32.2). Source: reference/vc-intro-network ("Who Takes Intros from Garry") + adjacent routing context. All 15 unique expected person slugs verified against ~/git/brain/people/<slug>.md source markdown: people/amit-kumar Accel partner, 102 YC deals people/diana-hu YC GP people/elad-gil Angel, top-rated people/eric-vishria Benchmark, healthtech people/gokul-rajaram Angel, 57 YC deals people/joff-redfern Menlo Ventures, ex-CPO Atlassian people/jon-xu YC GP people/kristina-shen Chemistry, healthtech people/lachy-groom Angel, 43 YC deals people/lee-edwards Quiet Capital, 52 YC deals people/nick-shalek Ribbit Capital, fintech people/nina-achadian Index Ventures, 69 YC deals (note: slug uses 'achadian' not 'achadjian') people/parul-singh 645 Ventures people/rebecca-kaden USV people/trae-stephens Founders Fund, defense/deep-tech Eval cannot run yet against Wintermute thin-client: server is v0.32.2, find_experts MCP op was added in v0.33. Once Wintermute upgrades the eval will run end-to-end via the v0.31.1 thin-client routing seam. Local eval works once the brain is indexed with find_experts available. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.33.1.3): wire thin-client routing into eval-whoknows `gbrain eval whoknows` now works against a thin-client install. When isThinClient(cfg), each fixture query routes through the remote find_experts MCP op via callRemoteTool — same v0.31.1 routing seam runWhoknows already uses. Local mode unchanged: findExperts(engine, ...) called directly. Server prerequisite: the brain must be v0.33+ for find_experts to be registered. Wintermute (currently v0.32.2) gets it on next upgrade and then the eval runs end-to-end with zero client-side changes. Mechanics: - `WhoknowsFn` callable abstraction so the gates are impl-agnostic - runEvalWhoknows(engine: BrainEngine | null, args) — null engine allowed in thin-client mode - Regression gate auto-skips in thin-client mode (no DB access to eval_candidates; quality gate alone gates ship) - cli.ts adds a thin-client bypass before connectEngine for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern E2E test updated to use an inline synthetic fixture (the shipped fixture is real-brain data now, doesn't match the seeded test brain). Sanity-check the shipped fixture parses cleanly in a separate case. Tests: 25 unit cases (+2 for null-engine signature contract) + 6 E2E cases. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
198 lines
8.1 KiB
TypeScript
198 lines
8.1 KiB
TypeScript
import { describe, it, expect } from 'bun:test';
|
||
import { rankCandidates, runWhoknows, findExperts, type WhoknowsResult } from '../src/commands/whoknows.ts';
|
||
import type { PageType } from '../src/core/types.ts';
|
||
|
||
/**
|
||
* v0.33 whoknows — pure-function unit tests covering the 10 locked
|
||
* shadow-path cases from ENG-D3 plus a few obvious sanity asserts.
|
||
*
|
||
* The ranking spec (also documented in src/commands/whoknows.ts):
|
||
*
|
||
* score = log(1 + raw_match) // expertise (sub-linear)
|
||
* × max(0.1, exp(-days/180)) // recency (floored)
|
||
* × (0.5 + 0.5 × clamp(salience)) // salience (centered)
|
||
*
|
||
* These tests exercise rankCandidates (pure) and the CLI registration.
|
||
* Integration against a real brain lives in test/e2e/whoknows.test.ts.
|
||
*/
|
||
|
||
function input(
|
||
slug: string,
|
||
raw_match: number,
|
||
days: number | null,
|
||
salience: number | null,
|
||
type: PageType = 'person',
|
||
) {
|
||
return {
|
||
slug,
|
||
source_id: 'default',
|
||
title: slug,
|
||
type,
|
||
raw_match,
|
||
days_since_effective: days,
|
||
salience_raw: salience,
|
||
};
|
||
}
|
||
|
||
describe('whoknows / rankCandidates — locked shadow paths (ENG-D3)', () => {
|
||
// Case 1: zero hybrid-search results → empty array
|
||
it('returns empty array on empty input', () => {
|
||
expect(rankCandidates([])).toEqual([]);
|
||
});
|
||
|
||
// Case 2: negative recency input → floor activates, score stays valid
|
||
it('negative days_since_effective clamps to 0 (recency_decay = 1.0)', () => {
|
||
const ranked = rankCandidates([input('alice', 0.5, -10, 0.5)]);
|
||
expect(ranked[0].factors.recency_decay).toBeCloseTo(1.0, 5);
|
||
expect(Number.isFinite(ranked[0].score)).toBe(true);
|
||
});
|
||
|
||
// Case 3: NaN salience → defaults to neutral (0.5)
|
||
it('NaN salience defaults to neutral 0.5', () => {
|
||
const ranked = rankCandidates([input('bob', 0.5, 30, NaN)]);
|
||
expect(ranked[0].factors.salience).toBeCloseTo(0.5, 5);
|
||
expect(ranked[0].factors.salience_factor).toBeCloseTo(0.75, 5);
|
||
});
|
||
|
||
// Case 4: undefined / null match score → 0 expertise, score zeros gracefully
|
||
it('NaN raw_match → expertise=0; score zeros gracefully without NaN', () => {
|
||
const ranked = rankCandidates([input('carol', NaN, 30, 0.5)]);
|
||
expect(ranked[0].factors.expertise).toBe(0);
|
||
expect(ranked[0].score).toBe(0);
|
||
expect(Number.isFinite(ranked[0].score)).toBe(true);
|
||
});
|
||
|
||
// Case 5: person-type filter — verified at SQL level by SearchOpts.types.
|
||
// Here we assert rankCandidates preserves the type field passed in.
|
||
it('preserves page type in the result row (filter happens upstream at SQL)', () => {
|
||
const ranked = rankCandidates([
|
||
input('alice', 0.5, 30, 0.5, 'person'),
|
||
input('acme', 0.3, 30, 0.5, 'company'),
|
||
]);
|
||
expect(ranked.find((r) => r.slug === 'alice')?.type).toBe('person');
|
||
expect(ranked.find((r) => r.slug === 'acme')?.type).toBe('company');
|
||
});
|
||
|
||
// Case 6: --explain output includes all factor values
|
||
it('every result includes the full factor breakdown for --explain', () => {
|
||
const [row] = rankCandidates([input('alice', 0.5, 60, 0.4)]);
|
||
expect(row.factors).toBeDefined();
|
||
expect(typeof row.factors.expertise).toBe('number');
|
||
expect(typeof row.factors.recency_decay).toBe('number');
|
||
expect(typeof row.factors.recency_factor).toBe('number');
|
||
expect(typeof row.factors.salience).toBe('number');
|
||
expect(typeof row.factors.salience_factor).toBe('number');
|
||
expect(typeof row.factors.raw_match).toBe('number');
|
||
// days_since_effective may be null for cold-start; the shape is correct either way.
|
||
expect('days_since_effective' in row.factors).toBe(true);
|
||
});
|
||
|
||
// Case 7: top-K honors opts.limit; defaults to 5
|
||
it('top-K honors limit; defaults to 5; clamped to >= 1', () => {
|
||
const many = Array.from({ length: 12 }, (_, i) =>
|
||
input(`person-${String(i).padStart(2, '0')}`, 0.5 - i * 0.01, 30, 0.5),
|
||
);
|
||
expect(rankCandidates(many).length).toBe(5); // default
|
||
expect(rankCandidates(many, 3).length).toBe(3);
|
||
expect(rankCandidates(many, 100).length).toBe(12);
|
||
expect(rankCandidates(many, 0).length).toBe(1); // clamped to >= 1
|
||
});
|
||
|
||
// Case 8: recency floor (0.1) — extreme days never produces NaN/Infinity
|
||
it('extreme days_since_effective is floored, never produces NaN/Infinity', () => {
|
||
const ranked = rankCandidates([
|
||
input('ancient', 0.5, 365 * 100, 0.5), // 100 years
|
||
input('cold-start', 0.5, null, 0.5), // never updated
|
||
]);
|
||
for (const r of ranked) {
|
||
expect(Number.isFinite(r.score)).toBe(true);
|
||
expect(r.factors.recency_factor).toBeGreaterThanOrEqual(0.1);
|
||
}
|
||
// cold-start (null days) → recency_factor = floor (0.1)
|
||
const cold = ranked.find((r) => r.slug === 'cold-start')!;
|
||
expect(cold.factors.recency_factor).toBeCloseTo(0.1, 5);
|
||
});
|
||
|
||
// Case 9: stable ordering — same-score ties break by slug alphabetical
|
||
it('same-score ties break alphabetically by slug for determinism', () => {
|
||
const ranked = rankCandidates([
|
||
input('zoe', 0.5, 30, 0.5),
|
||
input('alice', 0.5, 30, 0.5),
|
||
input('bob', 0.5, 30, 0.5),
|
||
]);
|
||
expect(ranked.map((r) => r.slug)).toEqual(['alice', 'bob', 'zoe']);
|
||
});
|
||
|
||
// Case 10: contract shape — public exports exist and have expected types
|
||
it('public surface: rankCandidates / findExperts / runWhoknows are functions', () => {
|
||
expect(typeof rankCandidates).toBe('function');
|
||
expect(typeof findExperts).toBe('function');
|
||
expect(typeof runWhoknows).toBe('function');
|
||
});
|
||
});
|
||
|
||
describe('whoknows / rankCandidates — ranking sanity', () => {
|
||
it('higher raw_match outranks lower (with all else equal)', () => {
|
||
const ranked = rankCandidates([
|
||
input('low-match', 0.1, 30, 0.5),
|
||
input('high-match', 0.9, 30, 0.5),
|
||
]);
|
||
expect(ranked[0].slug).toBe('high-match');
|
||
});
|
||
|
||
it('more recent outranks older (with all else equal)', () => {
|
||
const ranked = rankCandidates([
|
||
input('old', 0.5, 365, 0.5),
|
||
input('recent', 0.5, 7, 0.5),
|
||
]);
|
||
expect(ranked[0].slug).toBe('recent');
|
||
});
|
||
|
||
it('higher salience outranks lower (with all else equal)', () => {
|
||
const ranked = rankCandidates([
|
||
input('low-salience', 0.5, 30, 0.1),
|
||
input('high-salience', 0.5, 30, 0.9),
|
||
]);
|
||
expect(ranked[0].slug).toBe('high-salience');
|
||
});
|
||
|
||
it('all-zero candidate scores 0 but still appears in the result set', () => {
|
||
const ranked = rankCandidates([input('flat', 0, 365 * 10, 0)]);
|
||
expect(ranked.length).toBe(1);
|
||
expect(ranked[0].score).toBe(0);
|
||
});
|
||
});
|
||
|
||
describe('whoknows / rankCandidates — composite key safety', () => {
|
||
it('preserves source_id on each result row', () => {
|
||
const ranked = rankCandidates([
|
||
{ slug: 'alice', source_id: 'srcA', title: 'Alice', type: 'person', raw_match: 0.5, days_since_effective: 30, salience_raw: 0.5 },
|
||
{ slug: 'alice', source_id: 'srcB', title: 'Alice B', type: 'person', raw_match: 0.6, days_since_effective: 30, salience_raw: 0.5 },
|
||
]);
|
||
// Both rows preserved with their source_ids — composite key intact.
|
||
expect(ranked.length).toBe(2);
|
||
const sources = new Set(ranked.map((r) => r.source_id));
|
||
expect(sources.has('srcA')).toBe(true);
|
||
expect(sources.has('srcB')).toBe(true);
|
||
});
|
||
});
|
||
|
||
describe('whoknows / rankCandidates — factor decomposition', () => {
|
||
it('returns the exact factor breakdown for a known input', () => {
|
||
// expertise = log(1 + 0.5) ≈ 0.405
|
||
// recency_decay = exp(-30/180) ≈ 0.846
|
||
// salience_factor = 0.5 + 0.5*0.5 = 0.75
|
||
// score ≈ 0.405 * 0.846 * 0.75 ≈ 0.257
|
||
const [row] = rankCandidates([input('alice', 0.5, 30, 0.5)]);
|
||
expect(row.factors.expertise).toBeCloseTo(Math.log1p(0.5), 5);
|
||
expect(row.factors.recency_decay).toBeCloseTo(Math.exp(-30 / 180), 5);
|
||
expect(row.factors.recency_factor).toBeCloseTo(Math.exp(-30 / 180), 5);
|
||
expect(row.factors.salience_factor).toBeCloseTo(0.75, 5);
|
||
expect(row.score).toBeCloseTo(Math.log1p(0.5) * Math.exp(-30 / 180) * 0.75, 5);
|
||
});
|
||
});
|
||
|
||
// Case-marker comment: the 10 ENG-D3 cases live above (1-10 in the
|
||
// "locked shadow paths" describe block). The additional describes cover
|
||
// ranking sanity and source-id safety beyond the locked minimum.
|