Files
gbrain/test/whoknows.test.ts
T
d71fcf6f65 v0.33.1.0 feat: eval-gated whoknows — expertise + relationship-proximity routing (#881)
* feat(v0.33): add SearchOpts.types multi-type filter to searchHybrid

Push the page-type filter into SQL via AND p.type = ANY(\$N::text[]) in
both engines' searchKeyword + searchVector + searchKeywordChunks paths.
Primary consumer is the upcoming gbrain whoknows command (filters to
['person','company']); the limit budget then goes to typed candidates
instead of being eaten by note/transcript/article pages. Future
entity-only search in v0.34+ reuses the parameter for free.

AND-applies alongside the existing single-value type filter (callers can
use either or both). HybridSearchOpts threads opts.types into the
underlying searchOpts so hybridSearch callers get the SQL-level filter
without any post-filter waste.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.33): whoknows core ranking function + 10 locked unit tests

Implements ENG-D1's locked spec: score = log(1 + raw_match) ×
max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience). raw_match comes
from hybridSearch's RRF + source-boost-adjusted score; salience and
recency boosts in hybridSearch are intentionally disabled so the
formula applies on a clean signal.

rankCandidates() is the pure function the eval grades against;
findExperts() is the public entrypoint that wires hybrid search +
batch salience/effective_date fetches; runWhoknows() is the CLI.

Test/whoknows.test.ts covers the 10 ENG-D3 cases (zero results,
negative recency floor, NaN salience neutral default, NaN match
zeros gracefully, type preservation, --explain factor breakdown,
top-K limit clamping, recency-floor extreme-days safety, alphabetical
tie-break determinism, public-surface contract). Plus four sanity
asserts (higher-match outranks, more-recent outranks, higher-salience
outranks, all-zero candidate appears with score 0). Plus one factor
decomposition assertion that pins the exact formula numerically.
Plus a composite-key safety case (Codex F1).

22 expect calls across 16 tests. All passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.33): register find_experts MCP op + gbrain whoknows CLI

Wires both surfaces per ENG-D5: MCP op = find_experts (matches
find_anomalies naming convention; agent-facing); CLI command =
gbrain whoknows (memorable, user-facing). One findExperts() core
function backs both paths.

The op is scope:'read', localOnly:false — accessible over HTTP MCP
to read-scoped OAuth clients like the salience/anomalies family.
Op handler validates non-empty topic and dispatches to the same
findExperts() pure function the CLI uses.

CLI dispatch in src/cli.ts:case 'whoknows' calls runWhoknows; thin-
client routing happens inside runWhoknows via isThinClient(cfg) —
remote MCP installs route through the v0.31.1 routing seam to
callRemoteTool('find_experts', ...).

FIND_EXPERTS_DESCRIPTION in operations-descriptions.ts mirrors the
v0.29 redirect-hint style: leads with what the tool does, lists
explicit user-intent triggers ("who should I talk to about X",
"who knows about Y"), notes the type-filter behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.33): gbrain eval whoknows — two-layer eval gate (ENG-D2)

Implements the locked spec: Layer 1 hand-labeled fixture (>=80% top-3
hit rate) is the primary ship-blocking gate; Layer 2 eval_candidates
replay (>=0.4 mean set-Jaccard@3) is the regression gate that
auto-skips when < 20 replay-eligible rows exist (CONTRIBUTOR_MODE
sparseness fallback).

Dispatch lands as `gbrain eval whoknows <fixture.jsonl>` sub-subcommand
in src/commands/eval.ts (mirrors v0.25.0 export/prune/replay and
v0.27.x cross-modal pattern). Exits 0/1/2 for pass/fail/usage so CI
gates can consume.

JSON output (--json) ships schema_version: 1 for stable consumer
contract (mirrors v0.25.0 eval-replay.ts). Human output groups by
layer + emits a per-miss diagnostic table so failures are
self-debugging.

Unit tests pin:
- jaccardAtK math (7 cases — identical, disjoint, partial, k cutoff,
  empty-empty vacuous-stable, empty-vs-non-empty, Set dedup)
- topKHit (7 cases — position 1, 3, 4, miss, multi-expected, empty
  actual, empty expected)
- readFixture (6 cases — well-formed, comments/blanks, missing file,
  malformed JSON, missing required fields, non-string filter)
- Locked thresholds (HIT_RATE=0.8, REGRESSION=0.4, MIN_REPLAY_ROWS=20)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.33): gbrain doctor adds whoknows_health check

Per CEO-D7 (substrate-conditional v0.33 doctor check, but the
fixture-presence sub-check ships in week 1 regardless — it's the
"did you do the assignment?" signal). When the eval fixture is
missing, empty, or undersized (< 5 rows), doctor warns with the
exact path the user should populate.

The check is intentionally lightweight: it does NOT run the eval
itself or measure hit-rate regression. That's the job of `gbrain
eval whoknows`, called from CI/ship time. This check is the cheap
always-runs signal that surfaces in `gbrain doctor` and on the
ship review dashboard.

5 unit cases pin the four-status behavior (missing/empty/undersized/
ok) plus the comment-and-blank-line filtering so users can comment
out queries during iteration without breaking the row count.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.33): synthetic whoknows eval fixture + E2E quality gate test

test/fixtures/whoknows-eval.jsonl ships as a 10-query placeholder
demonstrating the schema. Comments document the assignment for end
users: they replace these with their own real queries before
shipping their gbrain install. The placeholder uses obviously-
example slugs (wiki/people/example-alice, etc.) so nobody mistakes
it for production data.

test/e2e/whoknows.test.ts seeds a synthetic PGLite brain that
matches the placeholder fixture, then runs findExperts on every
fixture query and asserts >=80% top-3 hit rate per ENG-D2 quality
gate. Also exercises the typeFilter (concept-decoy pages filtered
out), empty-result graceful return, --explain factor breakdown, and
top-K limit honoring.

Basis-vector embeddings (no API key) follow the existing pattern from
test/e2e/search-quality.test.ts.

5 test cases, 23 expect calls, all passing against PGLite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.33): VERSION bump + CHANGELOG + CLAUDE.md + llms regen

Bumps VERSION 0.31.11 → 0.33.0 and package.json to match. CHANGELOG
entry leads with the headline use ("ask gbrain who knows about X")
and the locked ENG-D1 ranking formula. "Numbers that matter" replaced
with a "what ships on which eval outcome" table — honest about the
eval-gated trajectory rather than fabricating benchmarks before the
release has been graded against a real brain.

CLAUDE.md Key Files annotations added for src/commands/whoknows.ts,
src/commands/eval-whoknows.ts, and test/fixtures/whoknows-eval.jsonl.
src/core/search/hybrid.ts entry extended with the new types parameter
documentation (push the type filter to SQL, no post-filter waste,
AND-applies alongside the existing single-value type field).

bun run build:llms ran the chaser; llms.txt + llms-full.txt
regenerated to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(v0.33): unit-test gap fill — engine typeFilter + find_experts op

Two new files filling the gaps Garry called out:

test/search-types-filter.test.ts — engine-level coverage on PGLite for
the new SearchOpts.types filter. Asserts the SQL-clause behavior
directly so a regression in the AND p.type = ANY(...) emission gets
caught here with a tight assertion rather than as part of a longer
findExperts pipeline. 9 cases across searchKeyword + searchVector +
chunk-grain documentation. Documents the pre-existing PGLite parity
gap (single-value `type` field is Postgres-only; `types` is the v0.33
multi-type filter that BOTH engines honor).

test/find-experts-op.test.ts — MCP-op contract test for find_experts.
Pins:
- Registered in the operations array + operationsByName
- scope: 'read', localOnly false (HTTP-MCP accessible per ENG-D5)
- Documented params (topic / limit / explain) with correct types
- cliHints.name === 'whoknows' (CLI surface bridge)
- Non-trivial description that references the use case
- Handler rejects empty / whitespace / missing topic with invalid_params
- Handler returns array shape on valid topic
- Handler honors limit param

11 op-contract cases + 9 engine-clause cases. All passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version to v0.33.1.0

Garry asked for v0.33.1 instead of v0.33.0 (queue collision with
unrelated 0.33.0 work). 4-digit format: 0.33.1.0. CHANGELOG header
and "To take advantage of" block updated. llms.txt regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(v0.33.1.1): cliHints.positional on find_experts so CLI accepts <topic>

Without `cliHints.positional: ['topic']`, the op-dispatch path in
src/cli.ts couldn't parse `gbrain whoknows "ai agents"` and threw
`invalid_params: topic is required`. Found while testing the v0.33.1.0
build against a real brain. The op handler validates topic; the CLI
just needed to know the positional shape so the dispatcher could
hand it through.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(v0.33.1.2): real-brain whoknows-eval fixture from VC intro network

Replaces the synthetic 10-row placeholder with 10 real expertise-routing
queries mined from Garry's actual brain via thin-client connection to
Wintermute (v0.32.2). Source: reference/vc-intro-network ("Who Takes
Intros from Garry") + adjacent routing context. All 15 unique expected
person slugs verified against ~/git/brain/people/<slug>.md source
markdown:

  people/amit-kumar          Accel partner, 102 YC deals
  people/diana-hu            YC GP
  people/elad-gil            Angel, top-rated
  people/eric-vishria        Benchmark, healthtech
  people/gokul-rajaram       Angel, 57 YC deals
  people/joff-redfern        Menlo Ventures, ex-CPO Atlassian
  people/jon-xu              YC GP
  people/kristina-shen       Chemistry, healthtech
  people/lachy-groom         Angel, 43 YC deals
  people/lee-edwards         Quiet Capital, 52 YC deals
  people/nick-shalek         Ribbit Capital, fintech
  people/nina-achadian       Index Ventures, 69 YC deals (note: slug
                              uses 'achadian' not 'achadjian')
  people/parul-singh         645 Ventures
  people/rebecca-kaden       USV
  people/trae-stephens       Founders Fund, defense/deep-tech

Eval cannot run yet against Wintermute thin-client: server is v0.32.2,
find_experts MCP op was added in v0.33. Once Wintermute upgrades the
eval will run end-to-end via the v0.31.1 thin-client routing seam.
Local eval works once the brain is indexed with find_experts available.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.33.1.3): wire thin-client routing into eval-whoknows

`gbrain eval whoknows` now works against a thin-client install. When
isThinClient(cfg), each fixture query routes through the remote
find_experts MCP op via callRemoteTool — same v0.31.1 routing seam
runWhoknows already uses. Local mode unchanged: findExperts(engine, ...)
called directly.

Server prerequisite: the brain must be v0.33+ for find_experts to be
registered. Wintermute (currently v0.32.2) gets it on next upgrade and
then the eval runs end-to-end with zero client-side changes.

Mechanics:
- `WhoknowsFn` callable abstraction so the gates are impl-agnostic
- runEvalWhoknows(engine: BrainEngine | null, args) — null engine
  allowed in thin-client mode
- Regression gate auto-skips in thin-client mode (no DB access to
  eval_candidates; quality gate alone gates ship)
- cli.ts adds a thin-client bypass before connectEngine for
  `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB
  pattern

E2E test updated to use an inline synthetic fixture (the shipped
fixture is real-brain data now, doesn't match the seeded test brain).
Sanity-check the shipped fixture parses cleanly in a separate case.

Tests: 25 unit cases (+2 for null-engine signature contract) + 6 E2E
cases. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:33:29 -07:00

198 lines
8.1 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { describe, it, expect } from 'bun:test';
import { rankCandidates, runWhoknows, findExperts, type WhoknowsResult } from '../src/commands/whoknows.ts';
import type { PageType } from '../src/core/types.ts';
/**
* v0.33 whoknows — pure-function unit tests covering the 10 locked
* shadow-path cases from ENG-D3 plus a few obvious sanity asserts.
*
* The ranking spec (also documented in src/commands/whoknows.ts):
*
* score = log(1 + raw_match) // expertise (sub-linear)
* × max(0.1, exp(-days/180)) // recency (floored)
* × (0.5 + 0.5 × clamp(salience)) // salience (centered)
*
* These tests exercise rankCandidates (pure) and the CLI registration.
* Integration against a real brain lives in test/e2e/whoknows.test.ts.
*/
function input(
slug: string,
raw_match: number,
days: number | null,
salience: number | null,
type: PageType = 'person',
) {
return {
slug,
source_id: 'default',
title: slug,
type,
raw_match,
days_since_effective: days,
salience_raw: salience,
};
}
describe('whoknows / rankCandidates — locked shadow paths (ENG-D3)', () => {
// Case 1: zero hybrid-search results → empty array
it('returns empty array on empty input', () => {
expect(rankCandidates([])).toEqual([]);
});
// Case 2: negative recency input → floor activates, score stays valid
it('negative days_since_effective clamps to 0 (recency_decay = 1.0)', () => {
const ranked = rankCandidates([input('alice', 0.5, -10, 0.5)]);
expect(ranked[0].factors.recency_decay).toBeCloseTo(1.0, 5);
expect(Number.isFinite(ranked[0].score)).toBe(true);
});
// Case 3: NaN salience → defaults to neutral (0.5)
it('NaN salience defaults to neutral 0.5', () => {
const ranked = rankCandidates([input('bob', 0.5, 30, NaN)]);
expect(ranked[0].factors.salience).toBeCloseTo(0.5, 5);
expect(ranked[0].factors.salience_factor).toBeCloseTo(0.75, 5);
});
// Case 4: undefined / null match score → 0 expertise, score zeros gracefully
it('NaN raw_match → expertise=0; score zeros gracefully without NaN', () => {
const ranked = rankCandidates([input('carol', NaN, 30, 0.5)]);
expect(ranked[0].factors.expertise).toBe(0);
expect(ranked[0].score).toBe(0);
expect(Number.isFinite(ranked[0].score)).toBe(true);
});
// Case 5: person-type filter — verified at SQL level by SearchOpts.types.
// Here we assert rankCandidates preserves the type field passed in.
it('preserves page type in the result row (filter happens upstream at SQL)', () => {
const ranked = rankCandidates([
input('alice', 0.5, 30, 0.5, 'person'),
input('acme', 0.3, 30, 0.5, 'company'),
]);
expect(ranked.find((r) => r.slug === 'alice')?.type).toBe('person');
expect(ranked.find((r) => r.slug === 'acme')?.type).toBe('company');
});
// Case 6: --explain output includes all factor values
it('every result includes the full factor breakdown for --explain', () => {
const [row] = rankCandidates([input('alice', 0.5, 60, 0.4)]);
expect(row.factors).toBeDefined();
expect(typeof row.factors.expertise).toBe('number');
expect(typeof row.factors.recency_decay).toBe('number');
expect(typeof row.factors.recency_factor).toBe('number');
expect(typeof row.factors.salience).toBe('number');
expect(typeof row.factors.salience_factor).toBe('number');
expect(typeof row.factors.raw_match).toBe('number');
// days_since_effective may be null for cold-start; the shape is correct either way.
expect('days_since_effective' in row.factors).toBe(true);
});
// Case 7: top-K honors opts.limit; defaults to 5
it('top-K honors limit; defaults to 5; clamped to >= 1', () => {
const many = Array.from({ length: 12 }, (_, i) =>
input(`person-${String(i).padStart(2, '0')}`, 0.5 - i * 0.01, 30, 0.5),
);
expect(rankCandidates(many).length).toBe(5); // default
expect(rankCandidates(many, 3).length).toBe(3);
expect(rankCandidates(many, 100).length).toBe(12);
expect(rankCandidates(many, 0).length).toBe(1); // clamped to >= 1
});
// Case 8: recency floor (0.1) — extreme days never produces NaN/Infinity
it('extreme days_since_effective is floored, never produces NaN/Infinity', () => {
const ranked = rankCandidates([
input('ancient', 0.5, 365 * 100, 0.5), // 100 years
input('cold-start', 0.5, null, 0.5), // never updated
]);
for (const r of ranked) {
expect(Number.isFinite(r.score)).toBe(true);
expect(r.factors.recency_factor).toBeGreaterThanOrEqual(0.1);
}
// cold-start (null days) → recency_factor = floor (0.1)
const cold = ranked.find((r) => r.slug === 'cold-start')!;
expect(cold.factors.recency_factor).toBeCloseTo(0.1, 5);
});
// Case 9: stable ordering — same-score ties break by slug alphabetical
it('same-score ties break alphabetically by slug for determinism', () => {
const ranked = rankCandidates([
input('zoe', 0.5, 30, 0.5),
input('alice', 0.5, 30, 0.5),
input('bob', 0.5, 30, 0.5),
]);
expect(ranked.map((r) => r.slug)).toEqual(['alice', 'bob', 'zoe']);
});
// Case 10: contract shape — public exports exist and have expected types
it('public surface: rankCandidates / findExperts / runWhoknows are functions', () => {
expect(typeof rankCandidates).toBe('function');
expect(typeof findExperts).toBe('function');
expect(typeof runWhoknows).toBe('function');
});
});
describe('whoknows / rankCandidates — ranking sanity', () => {
it('higher raw_match outranks lower (with all else equal)', () => {
const ranked = rankCandidates([
input('low-match', 0.1, 30, 0.5),
input('high-match', 0.9, 30, 0.5),
]);
expect(ranked[0].slug).toBe('high-match');
});
it('more recent outranks older (with all else equal)', () => {
const ranked = rankCandidates([
input('old', 0.5, 365, 0.5),
input('recent', 0.5, 7, 0.5),
]);
expect(ranked[0].slug).toBe('recent');
});
it('higher salience outranks lower (with all else equal)', () => {
const ranked = rankCandidates([
input('low-salience', 0.5, 30, 0.1),
input('high-salience', 0.5, 30, 0.9),
]);
expect(ranked[0].slug).toBe('high-salience');
});
it('all-zero candidate scores 0 but still appears in the result set', () => {
const ranked = rankCandidates([input('flat', 0, 365 * 10, 0)]);
expect(ranked.length).toBe(1);
expect(ranked[0].score).toBe(0);
});
});
describe('whoknows / rankCandidates — composite key safety', () => {
it('preserves source_id on each result row', () => {
const ranked = rankCandidates([
{ slug: 'alice', source_id: 'srcA', title: 'Alice', type: 'person', raw_match: 0.5, days_since_effective: 30, salience_raw: 0.5 },
{ slug: 'alice', source_id: 'srcB', title: 'Alice B', type: 'person', raw_match: 0.6, days_since_effective: 30, salience_raw: 0.5 },
]);
// Both rows preserved with their source_ids — composite key intact.
expect(ranked.length).toBe(2);
const sources = new Set(ranked.map((r) => r.source_id));
expect(sources.has('srcA')).toBe(true);
expect(sources.has('srcB')).toBe(true);
});
});
describe('whoknows / rankCandidates — factor decomposition', () => {
it('returns the exact factor breakdown for a known input', () => {
// expertise = log(1 + 0.5) ≈ 0.405
// recency_decay = exp(-30/180) ≈ 0.846
// salience_factor = 0.5 + 0.5*0.5 = 0.75
// score ≈ 0.405 * 0.846 * 0.75 ≈ 0.257
const [row] = rankCandidates([input('alice', 0.5, 30, 0.5)]);
expect(row.factors.expertise).toBeCloseTo(Math.log1p(0.5), 5);
expect(row.factors.recency_decay).toBeCloseTo(Math.exp(-30 / 180), 5);
expect(row.factors.recency_factor).toBeCloseTo(Math.exp(-30 / 180), 5);
expect(row.factors.salience_factor).toBeCloseTo(0.75, 5);
expect(row.score).toBeCloseTo(Math.log1p(0.5) * Math.exp(-30 / 180) * 0.75, 5);
});
});
// Case-marker comment: the 10 ENG-D3 cases live above (1-10 in the
// "locked shadow paths" describe block). The additional describes cover
// ranking sanity and source-id safety beyond the locked minimum.