admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
82cc7fff90429ef05585705b3d91a74afeffb4ce
gbrain/test/benchmark-search-quality.ts
T
Garry TanGitHubClaude Opus 4.7Wintermute
 b8e0a0eada v0.29.0 + v0.29.1 feat: salience + anomaly detection — brain surfaces what's hot without being asked (#730) 
* v0.29 foundation: emotional_weight column + formula + anomaly stats

Migration v34 adds pages.emotional_weight REAL DEFAULT 0.0 (column-only,
no index — salience query orders by computed score, not raw weight).
Embedded DDL (schema.sql + pglite-schema.ts + schema-embedded.ts)
mirrors the column so fresh installs don't need migration replay.

types.ts gains: PageFilters.sort enum + PAGE_SORT_SQL whitelist (engines
hardcoded ORDER BY updated_at DESC; threading lands in the next commit);
SalienceOpts/SalienceResult, AnomaliesOpts/AnomalyResult,
EmotionalWeightInputRow/EmotionalWeightWriteRow contracts.

cycle/emotional-weight.ts: pure-function score in [0..1] from tags +
takes (anglocentric default seed list; user-overridable via config key
emotional_weight.high_tags). cycle/anomaly.ts: meanStddev + cohort
threshold helpers with zero-stddev fallback (count > mean + 1) so rare
cohorts don't produce NaN sigmas.

Test coverage: migrate v34 structural assertions + 14-case formula
unit + 13-case anomaly stats unit. Codex review fixes baked in:
formula clamped to [0,1]; per-take weight clamped to [0,1] before
averaging; zero-stddev fallback finite, never NaN.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 engine: batch emotional-weight methods + listPages sort

BrainEngine adds 4 methods, both engines implement:

- batchLoadEmotionalInputs(slugs?): CTE-shaped read with per-table
  pre-aggregates. A page with N tags + M takes never produces N×M rows
  (codex C4#4) — page_tags + page_takes CTEs aggregate independently,
  then LEFT JOIN to pages.

- setEmotionalWeightBatch(rows): UPDATE FROM unnest($1::text[],
  $2::text[], $3::real[]) composite-keyed on (slug, source_id). Multi-
  source brains can't fan out (codex C4#3) — pages.slug is unique only
  within source_id. Same shape that v0.18 link batches use.

- getRecentSalience: time boundary computed in JS, bound as TIMESTAMPTZ.
  SQL identical across engines (codex C5/D5 — avoids dialect drift on
  $1::interval binding which has zero current uses on PGLite).

- findAnomalies: tag + type cohort baselines via generate_series-
  densified daily-count CTEs (codex C4#6). Sparse-day rare cohorts get
  correct (mean, stddev) instead of biased upward by zero-omission.
  Year cohort deferred to v0.30.

listPages threads the new PageFilters.sort enum through both engines.
Was hardcoded ORDER BY updated_at DESC; now PAGE_SORT_SQL whitelist
maps the 4 enum values to literal SQL fragments — no injection surface.
postgres.js uses sql.unsafe; PGLite splices the fragment directly.

Regression tests (PGLite, no DATABASE_URL needed):

- multi-source-emotional-weight: same slug under two source_ids,
  setEmotionalWeightBatch on one of them, asserts the other survives
  untouched. Direct codex C4#3 guard.

- list-pages-regression (IRON RULE): old call shape (type, tag, limit)
  still returns updated_desc default; new sort=updated_asc reverses;
  sort=created_desc orders by created_at; sort=slug alphabetical;
  unsupported sort enum falls back to default (defense in depth).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 cycle: new recompute_emotional_weight phase

Adds a 9th cycle phase between extract and embed. Sees the union of
syncPagesAffected + synthesizeWrittenSlugs for incremental mode (so
synthesize-written pages get their weight computed too — codex C2 caught
that the prior plan threaded only sync). Full mode (no incremental
anchors) walks every page; users hit this path on first upgrade via
gbrain dream --phase recompute_emotional_weight.

Phase orchestrator (cycle/recompute-emotional-weight.ts) is two SQL
round-trips total regardless of brain size:
  1. batchLoadEmotionalInputs(slugs?) → per-page tag/take inputs.
  2. computeEmotionalWeight in memory (pure function).
  3. setEmotionalWeightBatch(rows) → composite-keyed UPDATE FROM unnest.

Empty affectedSlugs short-circuits (no DB read, no write). Dry-run
computes weights and reports the would-write count without touching
the DB. Engine throw bubbles into status:fail with code
RECOMPUTE_EMOTIONAL_WEIGHT_FAIL — cycle continues to the next phase.

Plumbing:
- CyclePhase type adds 'recompute_emotional_weight'.
- ALL_PHASES + NEEDS_LOCK_PHASES include it.
- CycleReport.totals adds pages_emotional_weight_recomputed (additive,
  schema_version stays "1").
- runCycle's totals rollup + status derivation honor the new field.
- synthesize.ts emits writtenSlugs in details so cycle.ts can union
  with syncPagesAffected for incremental backfill.

Tests: 7-case unit (fake-engine), 3-case PGLite e2e (full mode + dry-
run + ALL_PHASES position), 1000-page perf budget (<5s on PGLite).

Codex C2 → A: clean separation. Phase doesn't modify runExtractCore;
runs on its own seam after the existing 8 phases plus synthesize.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 ops: get_recent_salience + find_anomalies + get_recent_transcripts

Three new MCP operations + a transcripts library:

- get_recent_salience: pages ranked by emotional + activity salience.
  Subagent-allow-listed. params: days (default 14), limit (default 20,
  capped 100), slugPrefix (renamed from `kind` per codex C4#10 to
  avoid collision with PageKind/TakeKind).

- find_anomalies: cohort-level activity outliers (tag + type).
  Subagent-allow-listed. Year cohort deferred to v0.30.

- get_recent_transcripts: raw .txt transcripts from the dream-cycle
  corpus dirs. LOCAL-ONLY: rejects ctx.remote === true with
  permission_denied (codex C3). NOT in the subagent allow-list — all
  subagent calls run with remote=true, would always reject (footgun if
  visible). Cycle's synthesize phase calls discoverTranscripts
  directly, so subagents that need transcripts go through the library
  function, not the op.

Tool descriptions extracted to src/core/operations-descriptions.ts so
they're pinnable in tests and stable for the Tier-2 LLM routing eval.
Redirects on query/search/list_pages: personal/emotional questions
should reach the new ops, not semantic search. Anti-flattery hint on
query: "Do NOT assume words like crazy, notable, or big mean
impressive — they often mean difficult or emotionally charged."

list_pages gains updated_after (string ISO) and sort enum params,
surfacing the engine threading from the prior commit.

src/core/transcripts.ts: filesystem walk shared by the gated MCP op
and the (commit 5) CLI command. Reuses discoverTranscripts corpus-dir
resolution + isDreamOutput from cycle/transcript-discovery.ts. Trust
gate lives in the op handler, not the library — the library is
trusted by both the gated op and the local CLI.

Allow-list: 11 → 13 (add salience + anomalies; transcripts excluded
per codex C3, with a comment explaining why).

Tests: 21-case description pin (catches accidental edits that change
LLM-facing surface); 11-case transcripts unit covering trust gate,
mtime window, dream-output skip, summary truncation, no corpus_dir;
2-case salience type-contract smoke (full Garry-test fixture in commit
6's e2e suite).

Codex C1: routing-eval fixtures (skills/<x>/routing-eval.jsonl)
deliberately NOT shipped — routing-eval.ts is substring-match on
resolver triggers, not MCP tool routing. Real coverage lands as
test/e2e/salience-llm-routing.test.ts in commit 6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CLI: gbrain salience / anomalies / transcripts

Three new CLI commands wired into src/cli.ts dispatch + CLI_ONLY set +
help text:

- gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]
- gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]
- gbrain transcripts recent [--days N] [--full] [--json]

Each command file mirrors src/commands/orphans.ts shape: pure data fn
+ JSON formatter + human formatter. Calls into engine.getRecentSalience
/ findAnomalies (already shipped) and src/core/transcripts.ts.

salience and anomalies show ranked rows with per-cohort
mean/stddev/sigma. transcripts honors `--full` (caps at 100KB/file)
vs default summary (first non-empty line + ~250 chars). All three
emit JSON with --json for agent consumption.

`--kind` is accepted as a slug-prefix shorthand on `gbrain salience`
even though the underlying op param is `slugPrefix` (kept the CLI
flag short; the MCP-facing param uses the more-explicit name to
align with PageKind/TakeKind/slugPrefix vocabulary).

CLI_ONLY set in src/cli.ts gains the three new command names so
they don't get forwarded to MCP-only routing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 e2e: Garry-test fixtures + Postgres parity + LLM routing eval

PGLite e2e (no DATABASE_URL needed):

- salience-pglite: the Garry test. 7 wedding-tagged pages updated today
  + 100 background pages backdated across 30 days via raw SQL UPDATE
  (codex C4#7 — engine.putPage stamps updated_at = now(), so seeding
  via the engine alone can't reproduce historical recency windows).
  Asserts wedding pages outrank random-tag noise in the 7-day window;
  slugPrefix filter narrows correctly; days=0 boundary case; limit cap.

- anomalies-pglite: same fixture shape (7 wedding pages today, 100
  background backdated). findAnomalies with sigma=3 returns the
  wedding-tag cohort with sigma_observed > 3 vs near-zero baseline;
  page_slugs sample carries the wedding pages; date with no activity
  returns []; high sigma threshold suppresses borderline cohorts
  (zero-stddev fallback stays finite — no NaN sigma).

Postgres-gated e2e:

- engine-parity-salience: PGLite ↔ Postgres parity for getRecentSalience
  and findAnomalies. Same fixture into both engines; top-result and
  cohort-set match. Closes the v0.22.0-style parity gap for the new
  v0.29 SQL idioms (EXTRACT(EPOCH ...), generate_series, CTE chain).

Tier-2 LLM routing eval (ANTHROPIC_API_KEY-gated):

- salience-llm-routing: calls Claude with v0.29 tool descriptions and
  12 personal-query phrasings ("anything crazy lately", "what's been
  going on with me", etc.). Asserts the chosen tool is in the v0.29
  set, not query() / search(). ~$0.10 per CI run on Haiku. Tests the
  ACTUAL ship criterion — replaces the discarded fake-coverage
  routing-eval.jsonl fixtures (codex C1 → B).

This is the only test that proves the description edits drive routing.
Without it, we'd ship description changes and only learn from
production behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.0: ship-prep — VERSION + CHANGELOG + CLAUDE Key Files

VERSION + package.json bump 0.28.0 → 0.29.0.

CHANGELOG.md adds a v0.29.0 release-summary in the GStack/Garry voice
plus the "To take advantage of v0.29.0" block. Headline two-liner:
"The brain tells you what's hot without being asked. Salience +
anomaly detection ship. Search rewards hypotheses; salience surfaces
them." Numbers-that-matter table covers engine surface delta, MCP op
delta, allow-list delta, cycle-phase delta, schema migration, list_pages
param surface, and test count. Itemized changes section lists the
schema migration + new cycle phase + new MCP ops + redirect
descriptions + subagent allow-list rules + new tests + a contributor
note clarifying that routing-eval.ts is not the right surface for
testing MCP tool routing (use the Tier-2 LLM eval pattern instead).

CLAUDE.md Key Files updated for the v0.29 surface:

- src/core/engine.ts: notes the 4 new methods + PageFilters.sort threading.
- src/core/migrate.ts: v34 (pages_emotional_weight) entry.
- src/core/cycle.ts: 8 → 9 phases, recompute_emotional_weight inserted
  between patterns and embed; totals.pages_emotional_weight_recomputed.
- src/core/cycle/emotional-weight.ts (NEW): formula + override path.
- src/core/cycle/anomaly.ts (NEW): stats helpers + zero-stddev fallback.
- src/core/cycle/recompute-emotional-weight.ts (NEW): phase orchestrator.
- src/core/transcripts.ts (NEW): library shared by gated MCP op + CLI.
- src/core/operations-descriptions.ts (NEW): pinned tool descriptions.
- src/core/minions/tools/brain-allowlist.ts: 11 → 13 entries; comment
  on why get_recent_transcripts is excluded.
- src/commands/salience.ts / anomalies.ts / transcripts.ts (NEW): CLI surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1 feat: recency + salience as two orthogonal options on query op (#696)

* feat: recency boost for search (v0.27.0) — temporal intent auto-detection, date filters, configurable decay

New search pipeline stage: keyword + vector → RRF → cosine re-score → backlink boost → recency boost → dedup

- applyRecencyBoost: hyperbolic decay, two strengths (moderate 30-day halflife, aggressive 7-day halflife)
- Auto-enabled when intent.ts detects temporal/event queries (detail='high')
- Manual override via SearchOpts.recencyBoost (0/1/2)
- Date filtering: afterDate/beforeDate on all three search paths (keyword, keywordChunks, vector)
- getPageTimestamps on both Postgres and PGLite engines
- 15 tests passing (boost math + intent classification)

* v0.29.1 schema: pages.{effective_date, effective_date_source, import_filename, salience_touched_at} + expression index

Migration v38 adds 4 nullable columns to pages and an expression index on
COALESCE(effective_date, updated_at) to support the new since/until date
filters. All additive — no behavior change in the default search path; only
consulted when callers opt into the new salience='on' / recency='on' axes
or pass since/until.

  effective_date         — content date (event_date / date / published /
                           filename-date / fallback). Read by recency boost
                           and date-filter paths only. Auto-link doesn't
                           touch it (immune to updated_at churn).
  effective_date_source  — sentinel for the doctor's effective_date_health
                           check ('event_date' | 'date' | 'published' |
                           'filename' | 'fallback').
  import_filename        — basename without extension, captured at import.
                           Used for filename-date precedence on daily/,
                           meetings/. Older rows leave it NULL.
  salience_touched_at    — bumped by recompute_emotional_weight when
                           emotional_weight changes. Salience window uses
                           GREATEST(updated_at, salience_touched_at) so
                           newly-salient old pages enter the recent salience
                           query.

Index strategy: a partial index on effective_date alone wouldn't help the
COALESCE expression in since/until filters (planner can't use it for the
negative side). The expression index ((COALESCE(effective_date, updated_at)))
is what actually accelerates the filter.

Postgres uses CONCURRENTLY + v14-style pg_index.indisvalid pre-drop guard
for prior failed CONCURRENTLY runs; PGLite uses plain CREATE INDEX. Mirror
of v34's pattern.

src/schema.sql + src/core/pglite-schema.ts updated for fresh installs;
src/core/schema-embedded.ts regenerated via bun run build:schema.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: computeEffectiveDate helper + putPage integration

Pure helper computing a page's effective_date from frontmatter precedence:
  1. event_date (meeting/event pages)
  2. date (dated essays)
  3. published (writing/)
  4. filename-date (leading YYYY-MM-DD in basename)
  5. updated_at (fallback)
  6. created_at (last resort)

Per-prefix override: for daily/ and meetings/ slugs, filename-date jumps
to position 1 — the filename is the user's primary signal there.

Returns {date, source}. The source label powers the doctor's
effective_date_health check to detect "fell back to updated_at" rows that
look populated but are functionally a NULL.

Range validation: parsed value must be in [1990-01-01, NOW + 1 year].
Out-of-range values drop to the next chain element.

Wired into importFromContent + importFromFile. The put_page MCP op derives
filename from slug-tail when no caller-supplied filename is available.

putPage SQL on both engines extended to write the new columns. ON CONFLICT
uses COALESCE(EXCLUDED.x, pages.x) so callers that don't know about the
new columns (auto-link, code reindex) preserve existing values rather than
blanking them. SELECT projection extended to return them; rowToPage threads
them through.

21 unit tests covering: precedence chain default order, per-prefix override,
parse failure fall-through, range validation [1990, NOW+1y], parseDateLoose
shape variants. All pass; typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: backfill orchestrator + library function for existing pages

src/core/backfill-effective-date.ts is the shared library function. Walks
pages in keyset-paginated batches (id > last_id ORDER BY id LIMIT 1000),
runs computeEffectiveDate per row, UPDATEs effective_date +
effective_date_source. Resumable via the `backfill.effective_date.last_id`
checkpoint key in the config table — a killed process can re-run and pick
up without re-doing rows. Idempotent: a full re-walk produces the same
writes.

Postgres-only: SET LOCAL statement_timeout = '600s' per batch. Doesn't
refuse the migration on low session settings (codex pass-2 #16).

src/commands/migrations/v0_29_1.ts is the orchestrator (4 phases mirroring
v0_12_2). Phase A schema (gbrain init --migrate-only), Phase B backfill
(via the library function), Phase C verify (count NULL effective_date),
Phase D record (handled by runner). The library function is reusable from
the gbrain reindex-frontmatter CLI command in the next commit.

import_filename stays NULL for backfilled rows — pre-v0.29.1 imports
didn't capture it. computeEffectiveDate uses the slug-tail when filename
is NULL; daily/2024-03-15 backfilled gets effective_date from the slug.

Registered in src/commands/migrations/index.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: gbrain reindex-frontmatter CLI command

Recovery / explicit-rebuild path for pages.effective_date. Used when:
  - User edited frontmatter dates after import
  - Post-upgrade backfill orchestrator finished but the user wants to
    re-walk a subset (e.g. just meetings/) after fixing some frontmatter
  - Precedence rules change between releases

Thin wrapper over backfillEffectiveDate from commit 3 — same code path
the v0_29_1 orchestrator uses; one source of truth.

Flags mirror reindex-code:
  --source <id>      Scope to one sources row (placeholder; library
                     library doesn't filter by source today, tracked v0.30+)
  --slug-prefix P    Scope to slugs starting with P (e.g. 'meetings/')
  --dry-run          Print what WOULD change, no DB writes
  --yes              Skip confirmation prompt (required for non-TTY non-JSON)
  --json             Machine-readable result envelope
  --force            Re-apply even when computed value matches existing

Wired into src/cli.ts. CLI handles its own engine lifecycle (creates +
disconnects).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: recency-decay map + buildRecencyComponentSql (pure, unused)

src/core/search/recency-decay.ts mirrors source-boost.ts in shape but
drives RECENCY ONLY (per D9 codex resolution). Salience is a separate
orthogonal axis; this map does not feed it.

DEFAULT_RECENCY_DECAY: 10 generic prefixes (no fork-specific names).
  - concepts/      evergreen (halflifeDays=0)
  - originals/     180d × 0.5 (long-tail decay; new essays nudged)
  - writing/       365d × 0.4
  - daily/         14d × 1.5  (aggressive — freshness IS the signal)
  - meetings/      60d × 1.0
  - chat/          7d × 1.0
  - media/x/       7d × 1.5
  - media/articles/ 90d × 0.5
  - people/companies/ 365d × 0.3
  - deals/         180d × 0.5

DEFAULT_FALLBACK: 90d × 0.5 for unmatched slugs.

Override priority: defaults < gbrain.yml recency: < env (GBRAIN_RECENCY_DECAY)
< per-call SearchOpts.recency_decay.

parseRecencyDecayEnv format: comma-separated prefix:halflifeDays:coefficient
triples. Refuses LOUD on parse error (RecencyDecayParseError) — codex
pass-2 #M3 finding. No silent fallback like source-boost's parser.

parseRecencyDecayYaml takes already-parsed YAML; throws on bad shape.

buildRecencyComponentSql in sql-ranking.ts emits a CASE expression with
longest-prefix-first ordering, evergreen short-circuit (literal 0 when
halflifeDays=0 or coefficient=0), and EXTRACT(EPOCH ...) for non-zero
branches. Output: ((CASE WHEN p.slug LIKE 'daily/%' THEN 1.5 * 14.0 /
(14.0 + EXTRACT(EPOCH FROM (NOW() - <dateExpr>))/86400.0) ... END))

Typed NowExpr enum prevents SQL injection (codex pass-1 #5). Tests pass
{ kind: 'fixed', isoUtc } for deterministic output; production NOW().
The 'fixed' branch escapes single quotes via escapeSqlLiteral.

25 unit tests covering: env parser shape, env error cases, yaml parser
shape, merge precedence (defaults < yaml < env < caller), CASE longest-
prefix-first ordering, evergreen short-circuit, NowExpr fixed/now,
single-quote injection defense, empty decayMap fallback path, default
map composition (no fork names, concepts/ evergreen, daily/ aggressive).

Pure module. Zero consumers in this commit; commit 6 wires it into
getRecentSalience, commit 10 wires it into the post-fusion stage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: refactor getRecentSalience to consume buildRecencyComponentSql

Both engines (Postgres + PGLite) now build the salience formula's third
term via buildRecencyComponentSql instead of inlining 1.0 / (1 + days_old).
Parameters: empty decayMap + fallback { halflifeDays: 1, coefficient: 1.0 }.
Math expands to 1 * 1.0 / (1.0 + days_old) = 1 / (1 + days_old) — same
numeric output as v0.29.0.

This is a no-behavior-change refactor preparing for commit 7's recency_bias
param. recency_bias='flat' (default) reproduces v0.29.0 exactly; 'on'
swaps in DEFAULT_RECENCY_DECAY for per-prefix decay.

Single source of truth for the recency math: same builder feeds the
salience query AND (in commit 10) the post-fusion applyRecencyBoost stage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: get_recent_salience gains recency_bias param (default 'flat')

SalienceOpts.recency_bias: 'flat' | 'on' added; default 'flat' preserves
v0.29.0 ranking verbatim. Pass 'on' to opt into per-prefix decay map
(concepts/originals/writing/ evergreen; daily/, media/x/, chat/ aggressive
decay).

When recency_bias='on', the salience query reads
COALESCE(p.effective_date, p.updated_at) instead of bare p.updated_at, so
the recency component is immune to auto-link updated_at churn — old
concepts/ pages just-touched by auto-link don't suddenly look fresh.

Both engines (Postgres + PGLite) wire the param through. resolveRecencyDecayMap()
honors gbrain.yml + GBRAIN_RECENCY_DECAY env at runtime.

MCP op surface: get_recent_salience gains the param with a load-bearing
description teaching the agent when to use 'on' vs 'flat' (current state →
on; mattering across all time → flat).

No silent v0.29.0 behavior change — opt-in only (per D11 codex resolution).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: recompute_emotional_weight writes salience_touched_at; window picks up newly-salient pages

setEmotionalWeightBatch on both engines now bumps salience_touched_at to
NOW() ONLY when the new emotional_weight differs from the existing one
(IS DISTINCT FROM, NULL-safe). No-op writes (same weight) leave the
column alone — preserves "actual change" semantics.

getRecentSalience window changes from
  WHERE p.updated_at >= boundary
to
  WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= boundary

Closes codex pass-1 finding #4: pages whose emotional_weight just changed
in the dream cycle (because tags or takes shifted) but whose updated_at
is older than the salience window now correctly enter the recent-salience
results. Without this, "Garry just added a take to a 6-month-old page"
stayed invisible to get_recent_salience until the next content edit.

COALESCE(salience_touched_at, p.updated_at) handles pre-v0.29.1 rows
where salience_touched_at is NULL — they fall back to p.updated_at and
behave identically to v0.29.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: merge intent.ts → query-intent.ts; emit 3 suggestions per query

D1 + D4 + D6 + D8: single regex-pass classifier returning
{intent, suggestedDetail, suggestedSalience, suggestedRecency}.

intent + suggestedDetail are v0.29.0 behavior verbatim (legacy intent.ts
deleted; classifyQueryIntent + autoDetectDetail compat shims preserved).

NEW for v0.29.1 — two orthogonal recency-axis suggestions:

  suggestedSalience: 'off' | 'on' | 'strong'
  suggestedRecency:  'off' | 'on' | 'strong'

Resolution rules (per D6 narrow temporal-bound exception):
  - CANONICAL patterns (who is X / what is Y / code / graph) → both off
  - UNLESS an EXPLICIT_TEMPORAL_BOUND also matches (today / right now /
    this week / since X / last N days), in which case temporal-bound wins
  - STRONG_RECENCY (today / right now / this morning / just now) → strong
  - RECENCY_ON (latest / recent / this week / meeting prep / catch up
    / remind me / status update) → on
  - SALIENCE_ON (catch up / remind me / status update / prep me /
    what's going on / what matters) → on
  - default → off for both axes (v0.29.1 prime-directive: pure opt-in)

Salience and recency are TRULY orthogonal (per D9). A query like
"latest news on AI" → recency='on' but salience='off' (the user wants
fresh, not emotionally-weighted). "What's going on with widget-co" →
both on. "Who is X right now" → both 'strong'/'on' (temporal bound
beats canonical 'who is').

intent.ts deleted; test/intent.test.ts renamed → test/query-intent-legacy.test.ts
(unchanged behavior coverage). New test/query-intent.test.ts adds 21
cases covering all three axes' interactions: canonical wins on bare
'who is', temporal bound overrides, "catch me up" matches with up to 15
chars between, "today" → strong, intent vs recency independence.

Updated callers:
  - src/core/search/hybrid.ts (autoDetectDetail import)
  - test/recency-boost.test.ts (classifyQueryIntent import)
  - test/benchmark-search-quality.ts (autoDetectDetail import)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: applySalienceBoost + applyRecencyBoost + runPostFusionStages wrapper

D9 + codex pass-1 #2 + #3 + pass-2 #4: salience and recency are TRULY
ORTHOGONAL post-fusion stages, both running from ALL THREE hybridSearch
return paths (keyword-only, embed-failure-fallback, full-hybrid).

NEW src/core/search/hybrid.ts exports:
  - applySalienceBoost(results, scores, strength)
      score *= 1 + k * log(1 + score) where k = 0.15 (on) or 0.30 (strong)
      No time component. Pure mattering signal.
  - applyRecencyBoost(results, dates, strength, decayMap, fallback, nowMs?)
      Per-prefix decay factor: 1 + strengthMul * coefficient * halflife / (halflife + days_old)
      strengthMul: 1.0 (on) or 1.5 (strong)
      Evergreen prefixes (halflifeDays=0) skipped (factor 1.0).
      Pure recency signal. Independent of mattering.
  - runPostFusionStages(engine, results, opts)
      Wraps backlink + salience + recency. Called from EACH return path so
      keyless installs and embed failures get the same boost surface as
      the full hybrid path.

NEW engine methods (composite-keyed for multi-source isolation):
  - getEffectiveDates(refs: Array<{slug, source_id}>): Map<key, Date>
      Returns COALESCE(effective_date, updated_at, created_at). Key format:
      `${source_id}::${slug}`. Mirror of getBacklinkCounts shape.
  - getSalienceScores(refs: Array<{slug, source_id}>): Map<key, number>
      Returns emotional_weight × 5 + ln(1 + take_count). Composite key.

Deprecated (kept for back-compat through v0.29.x):
  - SearchOpts.afterDate / beforeDate (alias for since/until)
  - SearchOpts.recencyBoost: 0|1|2 (alias for recency: 'off'|'on'|'strong')
  - getPageTimestamps (use getEffectiveDates instead)

NEW SearchOpts fields:
  - salience: 'off' | 'on' | 'strong'
  - recency:  'off' | 'on' | 'strong'
  - since:    string (ISO-8601 or relative, replaces afterDate)
  - until:    string (replaces beforeDate)

Resolution: caller-explicit > legacy alias (recencyBoost) > heuristic
(classifyQuery's suggestedSalience / suggestedRecency).

Deleted: src/core/search/recency.ts (PR #618's, replaced) +
test/recency-boost.test.ts (its scope is replaced by query-intent.test.ts +
future post-fusion tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Wintermute <wintermute@garrytan.com>

* v0.29.1: query op gains salience + recency + since + until params; PGLite since/until parity

Combines commits 12 + 13 of the plan.

Query op surface (src/core/operations.ts):
  - salience: 'off' | 'on' | 'strong' (with load-bearing description)
  - recency:  'off' | 'on' | 'strong'
  - since:    string (ISO-8601 or relative; replaces deprecated afterDate)
  - until:    string (replaces deprecated beforeDate)

Tool descriptions teach the calling agent:
  - salience axis = mattering, no time component
  - recency axis = age decay, no mattering signal
  - omit either to let gbrain auto-detect from query text via classifyQuery

hybrid.ts maps since/until → afterDate/beforeDate at the engine call
boundary so PR #618's existing engine plumbing keeps working without
rename. Codex pass-1 #10 finding closed.

PGLite engine (codex pass-1 #10): since/until parity added to all three
search methods (searchKeyword, searchKeywordChunks, searchVector). SQL
filter against COALESCE(p.effective_date, p.updated_at, p.created_at)
so date filtering matches user content-date intent (a meeting was on
event_date, not when it got reimported). Filter is applied INSIDE the
HNSW inner CTE in searchVector so HNSW's candidate pool already
excludes out-of-range pages — preserves pagination contract.

This also closes existing cross-engine drift: pre-v0.29.1 Postgres had
afterDate/beforeDate from PR #618; PGLite had nothing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: migration v39 — eval_candidates capture columns for replay reproducibility

D11 codex pass-2 resolution: extend eval_candidates with 7 new nullable
columns so `gbrain eval replay` can reproduce captured runs of agent-explicit
salience + recency choices.

Without these columns, replays of the new axis params drift. The live
behavior depends on the resolved {salience, recency} values; v0.29.0's
schema doesn't capture them.

  as_of_ts            TIMESTAMPTZ  — brain's logical NOW at capture
                                     (replay uses this instead of wall-clock)
  salience_param      TEXT         — what the caller passed (NULL if omitted)
  recency_param       TEXT         — same
  salience_resolved   TEXT         — final value applied
  recency_resolved    TEXT         — same
  salience_source     TEXT         — 'caller' or 'auto_heuristic'
  recency_source      TEXT         — same

All nullable + additive. Pre-v0.29.1 rows stay valid. NDJSON
schema_version STAYS at 1 — consumers ignore unknown fields (codex
pass-1 #C2 dissolves; no cross-repo coordination needed).

ADD COLUMN with no DEFAULT is metadata-only on PG 11+ and PGLite —
instant on tables of any size.

src/schema.sql + src/core/pglite-schema.ts mirror the additions for fresh
installs; src/core/schema-embedded.ts regenerated. eval_capture.ts
populates the new fields in commit 16 (docs + ship).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: doctor checks — effective_date_health + salience_health

effective_date_health: sample-1000 scan detects three classes of
problems (codex pass-1 #5 resolution via the effective_date_source
sentinel column added in commit 1):

  fallback_with_fm_date  — page fell back to updated_at even though
                           frontmatter has parseable event_date / date /
                           published. The "wrong but populated" residual
                           that earlier review iterations missed.
  future_dated            — effective_date > NOW() + 1 year (corrupt
                            or typo'd century).
  pre_1990                — effective_date < 1990-01-01 (epoch math gone
                            wrong, bad parse).

Sample of last 1000 pages by default — fast on 200K-page brains. Fix
hint: gbrain reindex-frontmatter.

salience_health: detects pages with active takes whose emotional_weight
is still 0 (recompute_emotional_weight phase hasn't run since the
take landed). Reports the brain's non-zero emotional_weight count as
an informational baseline. Fix hint: gbrain dream --phase
recompute_emotional_weight.

Both checks gracefully skip on pre-v0.29.1 brains (column doesn't
exist → 42703) without surfacing as warnings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: docs + skills convention + CHANGELOG + version bump

- VERSION 0.29.0 → 0.29.1
- package.json version bump
- CHANGELOG.md: full release-summary + itemized + "To take advantage"
  block per the project's voice rules. Two-line headline + concrete
  pathology framing (existing callers unchanged; new axes opt-in;
  agent in charge per the prime directive).
- skills/conventions/salience-and-recency.md: agent-readable decision
  rules. "Current state → on. Canonical truth → off." plus the narrow
  temporal-bound exception. Cross-cutting convention propagates to
  brain skills via RESOLVER.md.
- skills/migrations/v0.29.1.md: agent-readable upgrade instructions.
  Verify steps + behavior-change reference + recovery commands.

The build-time tool-description generator from D2 (extract decision
tables from skills/conventions/salience-and-recency.md, embed into
operations.ts at build time) is deferred to a follow-up commit. The
tool descriptions on the query op + get_recent_salience are inline in
operations.ts for v0.29.1; the auto-gen + CI staleness gate land in
v0.29.2 if drift becomes a problem in practice.

148 unit tests pass across the v0.29.1 surface (effective-date,
recency-decay, query-intent, migrate, salience, recompute-emotional-weight).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Wintermute <wintermute@garrytan.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 master-rebase fixups: renumber + drift cleanup

- v0.29.1 migrations renumber v38/v39 → v41/v42 (master shipped takes_table at
  v37 + access_tokens_permissions at v38; v0.27.1 took v39). My v0.29.0
  emotional_weight slots in at v40; v0.29.1's pages_recency_columns lands at
  v41 and eval_candidates_recency_capture at v42.
- src/core/utils.ts comment refs updated v37 → v40 (emotional_weight) and
  v38 → v41 (effective_date/etc).
- test/brain-allowlist.test.ts: size assertion 11 → 13 + the new
  get_recent_salience / find_anomalies positive checks + the explicit
  get_recent_transcripts negative check (v0.29 added the salience pair to
  the allow-list; transcripts are deliberately excluded because all
  subagent calls have remote=true and the v0.29 trust gate rejects them —
  visibility would be a footgun).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CI fixups: privacy allow-list + cycle phase count + migration plan

Three CI test failures on PR #730, all caused by master-side state the
v0.29 cherry-picks didn't yet account for:

1. scripts/check-privacy.sh allow-lists test/recency-decay.test.ts
   The v0.29.1 recency-decay test asserts that DEFAULT_RECENCY_DECAY's
   keys do NOT include fork-specific path prefixes. Because the assertion
   has to name the banned tokens to assert their absence, the privacy
   guard flagged the literal occurrence. Same exception class as
   CHANGELOG.md, CLAUDE.md, and scripts/check-privacy.sh itself —
   meta-rule enforcement requires mentioning what the rule forbids.

2. test/core/cycle.serial.test.ts: 9 → 10 phases.
   The yieldBetweenPhases test was written for v0.26.5 (9 phases incl.
   purge). v0.29 added a 10th phase (recompute_emotional_weight)
   between patterns and embed; the test's expected hookCalls and
   report.phases.length needed bumping.

3. test/apply-migrations.test.ts: append '0.29.1' to skippedFuture lists.
   v0.29.1 added a new entry to src/commands/migrations/index.ts; the
   buildPlan test snapshots the exact ordered list of versions, so it
   needs the new entry in both the fresh-install case and the Codex H9
   regression case.

All three verified locally:
  - bash scripts/check-privacy.sh → exit 0
  - bun test test/apply-migrations.test.ts → 18/18 pass
  - bun test test/core/cycle.serial.test.ts → 28/28 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CI fixup: regenerate llms-full.txt to match CLAUDE.md state

build-llms test asserts the committed llms.txt + llms-full.txt match
what the generator produces from the current source tree. CLAUDE.md
got new v0.29 Key Files entries (recompute_emotional_weight phase,
emotional-weight formula, anomaly stats, transcripts library, salience
ops, etc.) without a corresponding regen. `bun run build:llms` brings
llms-full.txt back in sync; llms.txt is byte-for-byte identical so
only the larger inline bundle changed.

Verified locally: bun test test/build-llms.test.ts → 7/7 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 e2e: cover tool-surfaces + MCP dispatch path

Two gaps were uncovered when reviewing v0.29 coverage against the new
contracts the cherry-picks landed onto master.

1. test/v0_29-tool-surfaces.test.ts (unit, 9 cases)

   Existing tests pin the description constants module and the
   BRAIN_TOOL_ALLOWLIST set membership, but nothing checked the two
   filters that ACT on those constants:

   - serve-http.ts:745 filters operations by !op.localOnly to build the
     HTTP MCP tool list. Without a test, anyone removing `localOnly: true`
     from get_recent_transcripts would silently expose it to remote
     callers — defense-in-depth on top of the in-handler ctx.remote check
     would be the only guard. Now pinned: get_recent_transcripts is
     hidden, salience + anomalies stay visible.

   - buildBrainTools surfaces the v0.29 ops as `brain_get_recent_salience`
     and `brain_find_anomalies`, and EXCLUDES `brain_get_recent_transcripts`
     (codex C3 footgun gate — all subagent calls are remote=true, the op
     would always reject). Now pinned.

   Both filters are pure functions; no DB / engine.connect needed.

2. test/e2e/v0_29-mcp-dispatch-pglite.test.ts (e2e, 5 cases)

   Existing v0.29 e2e tests call engine methods directly. None went
   through the full dispatchToolCall pipeline that stdio MCP and HTTP
   MCP both use. The new file covers:

   - get_recent_salience returns ranked rows via dispatch (top result
     is the wedding-tagged page from the seeded fixture).
   - find_anomalies returns the AnomalyResult shape via dispatch.
   - get_recent_transcripts rejects with permission_denied when
     ctx.remote === true (the in-handler trust gate is the last line if
     localOnly ever drops).
   - get_recent_transcripts succeeds with ctx.remote === false (CLI
     path) and returns [] when no corpus dir is configured.
   - Unknown tool name returns the standard isError + "Unknown tool"
     envelope (regression guard for dispatch shape).

Verified locally — all 14 cases pass:
  bun test test/v0_29-tool-surfaces.test.ts                          → 9 pass
  bun test test/e2e/v0_29-mcp-dispatch-pglite.test.ts                → 5 pass

Re-ran the full v0.29 PGLite e2e suite to confirm no regressions:
  salience-pglite.test.ts                       5 pass
  anomalies-pglite.test.ts                      4 pass
  cycle-recompute-emotional-weight-pglite.test  3 pass
  list-pages-regression.test.ts                 6 pass
  multi-source-emotional-weight-pglite.test     4 pass
  backfill-perf-pglite.test.ts                  1 pass
  v0_29-mcp-dispatch-pglite.test.ts             5 pass
  -----
  Total: 28 pass / 0 fail
  Postgres parity test (DATABASE_URL gated)     7 skip (correct)
  LLM routing eval (ANTHROPIC_API_KEY gated)   12 skip (correct)
  bun run typecheck                             clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CI fixup: drop unused PGLiteEngine in tool-surfaces test

scripts/check-test-isolation.sh's R3 + R4 lints flagged the new
test/v0_29-tool-surfaces.test.ts for instantiating PGLiteEngine outside
a beforeAll() block (R3) and lacking the matching afterAll(disconnect)
(R4). The intent of those rules is to prevent engine leaks across the
shard process — every PGLiteEngine must follow the canonical
beforeAll(connect+initSchema) / afterAll(disconnect) pattern.

The fix here is upstream of the rule, not a workaround: this test never
needed an engine. buildBrainTools doesn't issue any SQL at registry-build
time — it only reads `engine.kind` for the put_page namespace-wrap
branch. A `{ kind: 'pglite' } as unknown as BrainEngine` fake-engine
literal keeps the test pure-function: no WASM cold-start, no connect
lifecycle, no test-isolation rule fired.

Verified locally:
  bash scripts/check-test-isolation.sh → OK (257 non-serial unit files)
  bun test test/v0_29-tool-surfaces.test.ts → 9 pass
  bun run typecheck → clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wintermute <wintermute@garrytan.com>
2026-05-07 21:52:58 -07:00

752 lines
58 KiB
TypeScript
Raw Blame History

/**
* Search Quality Benchmark — Rich benchmark with realistic overlap and noise.
*
* 30 pages, 60 chunks, 20 queries with graded relevance. Tests ranking quality
* in a brain with overlapping topics, multiple mentions, and temporal ambiguity.
*
* All data is fictional. No private information.
*
* Usage: bun run test/benchmark-search-quality.ts
*/
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { rrfFusion } from '../src/core/search/hybrid.ts';
import { dedupResults } from '../src/core/search/dedup.ts';
import { precisionAtK, recallAtK, mrr, ndcgAtK } from '../src/core/search/eval.ts';
import { autoDetectDetail } from '../src/core/search/query-intent.ts';
import type { SearchResult, ChunkInput } from '../src/core/types.ts';
const RRF_K = 60;
// ─── Embedding helpers ───────────────────────────────────────────
// Create embeddings with shared dimensions to simulate semantic overlap.
// Each "topic" gets a primary dimension. Related topics share secondary dimensions.
function topicEmbedding(topics: Record<number, number>, dim = 1536): Float32Array {
const emb = new Float32Array(dim);
for (const [idx, weight] of Object.entries(topics)) {
emb[Number(idx) % dim] = weight;
}
// Normalize
let mag = 0;
for (let i = 0; i < dim; i++) mag += emb[i] * emb[i];
mag = Math.sqrt(mag);
if (mag > 0) for (let i = 0; i < dim; i++) emb[i] /= mag;
return emb;
}
// Topic dimensions (semantic axes)
const T = {
AI: 0, FINTECH: 1, CRYPTO: 2, CLIMATE: 3, HEALTH: 4,
ENTERPRISE: 5, CONSUMER: 6, ROBOTICS: 7, EDUCATION: 8, BIOTECH: 9,
FOUNDER: 10, INVESTOR: 11, ENGINEER: 12, DESIGNER: 13,
MEETING: 20, ANNOUNCEMENT: 21, FUNDING: 22, LAUNCH: 23, HIRING: 24,
COMPILED: 30, TIMELINE: 31,
};
// ─── Test Data: 30 fictional pages ──────────────────────────────
interface TestPage {
slug: string;
type: 'person' | 'company' | 'concept';
title: string;
compiled_truth: string;
timeline: string;
chunks: ChunkInput[];
}
const PAGES: TestPage[] = [
// ── People (10) ──────────────────────────────────────────────
{
slug: 'people/alice-chen',
type: 'person',
title: 'Alice Chen',
compiled_truth: 'Alice Chen is the CEO of NovaPay, a fintech startup building instant cross-border payments for SMBs. Previously VP Engineering at Stripe. Deep expertise in payment rails and regulatory compliance.',
timeline: '2024-03-15: Met Alice at Fintech Forum. Discussed cross-border payment challenges in Southeast Asia. She mentioned NovaPay is expanding to Vietnam.\n2024-06-20: Coffee with Alice. NovaPay raised Series B. Hiring aggressively.',
chunks: [
{ chunk_index: 0, chunk_text: 'Alice Chen is the CEO of NovaPay, a fintech startup building instant cross-border payments for SMBs. Previously VP Engineering at Stripe. Deep expertise in payment rails and regulatory compliance.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.FINTECH]: 1, [T.FOUNDER]: 0.6, [T.ENTERPRISE]: 0.3}), token_count: 35 },
{ chunk_index: 1, chunk_text: '2024-03-15: Met Alice at Fintech Forum. Discussed cross-border payment challenges in Southeast Asia. NovaPay expanding to Vietnam. 2024-06-20: Coffee with Alice. NovaPay raised Series B. Hiring aggressively.', chunk_source: 'timeline', embedding: topicEmbedding({[T.FINTECH]: 0.5, [T.MEETING]: 0.8, [T.FUNDING]: 0.4}), token_count: 40 },
],
},
{
slug: 'people/bob-martinez',
type: 'person',
title: 'Bob Martinez',
compiled_truth: 'Bob Martinez is a partner at Green Horizon Ventures, focused on climate tech and clean energy investments. Board member at SolarGrid and WindFlow. Former McKinsey energy practice.',
timeline: '2024-04-10: Lunch with Bob. He is bullish on grid-scale battery storage. Mentioned a new fund for carbon capture.\n2024-08-05: Bob introduced me to the SolarGrid founder.',
chunks: [
{ chunk_index: 0, chunk_text: 'Bob Martinez is a partner at Green Horizon Ventures, focused on climate tech and clean energy investments. Board member at SolarGrid and WindFlow. Former McKinsey energy practice.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CLIMATE]: 1, [T.INVESTOR]: 0.7, [T.ENTERPRISE]: 0.2}), token_count: 32 },
{ chunk_index: 1, chunk_text: '2024-04-10: Lunch with Bob. Bullish on grid-scale battery storage. New fund for carbon capture. 2024-08-05: Bob introduced me to SolarGrid founder.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CLIMATE]: 0.5, [T.MEETING]: 0.8, [T.FUNDING]: 0.3}), token_count: 30 },
],
},
{
slug: 'people/carol-nakamura',
type: 'person',
title: 'Carol Nakamura',
compiled_truth: 'Carol Nakamura is CTO of MindBridge, an AI company building diagnostic tools for mental health professionals. PhD in computational neuroscience from MIT. Pioneer in applying transformer models to clinical psychology.',
timeline: '2024-02-28: Carol presented at AI Health Summit. MindBridge accuracy data is impressive, 94% concordance with clinical diagnosis.\n2024-07-12: Carol reached out about Series A. Looking for $15M.',
chunks: [
{ chunk_index: 0, chunk_text: 'Carol Nakamura is CTO of MindBridge, an AI company building diagnostic tools for mental health professionals. PhD in computational neuroscience from MIT. Pioneer in transformer models for clinical psychology.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.7, [T.HEALTH]: 0.8, [T.FOUNDER]: 0.4, [T.ENGINEER]: 0.3}), token_count: 35 },
{ chunk_index: 1, chunk_text: '2024-02-28: Carol presented at AI Health Summit. MindBridge 94% concordance with clinical diagnosis. 2024-07-12: Carol reached out about Series A, looking for $15M.', chunk_source: 'timeline', embedding: topicEmbedding({[T.AI]: 0.3, [T.HEALTH]: 0.4, [T.MEETING]: 0.6, [T.FUNDING]: 0.5}), token_count: 32 },
],
},
{
slug: 'people/david-okonkwo',
type: 'person',
title: 'David Okonkwo',
compiled_truth: 'David Okonkwo is founder of EduStack, an AI-powered adaptive learning platform. Previously taught CS at Stanford. Believes personalized education is the biggest unlocked market in tech.',
timeline: '2024-05-02: David demoed EduStack at demo day. The adaptive curriculum engine is genuinely novel.\n2024-09-18: David shipped v2 with real-time assessment. Growing 40% MoM in Nigeria.',
chunks: [
{ chunk_index: 0, chunk_text: 'David Okonkwo is founder of EduStack, an AI-powered adaptive learning platform. Previously taught CS at Stanford. Believes personalized education is the biggest unlocked market in tech.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.5, [T.EDUCATION]: 1, [T.FOUNDER]: 0.6}), token_count: 32 },
{ chunk_index: 1, chunk_text: '2024-05-02: David demoed EduStack at demo day. Adaptive curriculum engine is novel. 2024-09-18: David shipped v2 with real-time assessment. Growing 40% MoM in Nigeria.', chunk_source: 'timeline', embedding: topicEmbedding({[T.EDUCATION]: 0.5, [T.LAUNCH]: 0.7, [T.MEETING]: 0.4}), token_count: 35 },
],
},
{
slug: 'people/elena-volkov',
type: 'person',
title: 'Elena Volkov',
compiled_truth: 'Elena Volkov is co-founder of CryptoSafe, building institutional-grade custody for digital assets. Former security engineer at Google. Expert in HSM architecture and multi-party computation.',
timeline: '2024-01-20: Elena gave a talk on MPC wallets at ETH Denver. Very technical, very sharp.\n2024-06-15: CryptoSafe announced $30M Series A led by a16z crypto.',
chunks: [
{ chunk_index: 0, chunk_text: 'Elena Volkov is co-founder of CryptoSafe, building institutional-grade custody for digital assets. Former security engineer at Google. Expert in HSM architecture and multi-party computation.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CRYPTO]: 0.8, [T.ENTERPRISE]: 0.5, [T.ENGINEER]: 0.6, [T.FOUNDER]: 0.3}), token_count: 32 },
{ chunk_index: 1, chunk_text: '2024-01-20: Elena talk on MPC wallets at ETH Denver. Very technical. 2024-06-15: CryptoSafe announced $30M Series A led by a16z crypto.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CRYPTO]: 0.5, [T.ANNOUNCEMENT]: 0.6, [T.FUNDING]: 0.7}), token_count: 28 },
],
},
{
slug: 'people/frank-dubois',
type: 'person',
title: 'Frank Dubois',
compiled_truth: 'Frank Dubois is head of AI at RoboLogic, building autonomous warehouse robots. 15 years in robotics, previously at Boston Dynamics. Focused on manipulation in unstructured environments.',
timeline: '2024-03-22: Frank showed the latest RoboLogic demo. Picking irregular objects at 98% accuracy.\n2024-11-01: RoboLogic deployed at Amazon fulfillment center in Memphis.',
chunks: [
{ chunk_index: 0, chunk_text: 'Frank Dubois is head of AI at RoboLogic, building autonomous warehouse robots. 15 years in robotics, previously at Boston Dynamics. Focused on manipulation in unstructured environments.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.6, [T.ROBOTICS]: 1, [T.ENGINEER]: 0.5}), token_count: 30 },
{ chunk_index: 1, chunk_text: '2024-03-22: Frank showed RoboLogic demo. Picking irregular objects at 98% accuracy. 2024-11-01: RoboLogic deployed at Amazon fulfillment center in Memphis.', chunk_source: 'timeline', embedding: topicEmbedding({[T.ROBOTICS]: 0.6, [T.LAUNCH]: 0.7, [T.MEETING]: 0.3}), token_count: 28 },
],
},
{
slug: 'people/grace-lee',
type: 'person',
title: 'Grace Lee',
compiled_truth: 'Grace Lee is a designer and founder of PixelCraft, a design tool for AI-generated UI components. Former lead designer at Figma. Strong opinions on AI replacing mockups with working prototypes.',
timeline: '2024-04-30: Grace launched PixelCraft beta. 5000 signups in first week.\n2024-08-15: Grace hired 3 engineers from Vercel. PixelCraft growing fast.',
chunks: [
{ chunk_index: 0, chunk_text: 'Grace Lee is a designer and founder of PixelCraft, a design tool for AI-generated UI components. Former lead designer at Figma. Strong opinions on AI replacing mockups with working prototypes.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.5, [T.DESIGNER]: 0.9, [T.CONSUMER]: 0.4, [T.FOUNDER]: 0.5}), token_count: 34 },
{ chunk_index: 1, chunk_text: '2024-04-30: Grace launched PixelCraft beta. 5000 signups first week. 2024-08-15: Grace hired 3 engineers from Vercel. Growing fast.', chunk_source: 'timeline', embedding: topicEmbedding({[T.DESIGNER]: 0.3, [T.LAUNCH]: 0.8, [T.HIRING]: 0.5}), token_count: 25 },
],
},
{
slug: 'people/hiro-tanaka',
type: 'person',
title: 'Hiro Tanaka',
compiled_truth: 'Hiro Tanaka is CEO of GenomeAI, using large language models to predict protein folding for drug discovery. Previously research scientist at DeepMind. Published 40+ papers on computational biology.',
timeline: '2024-02-14: Hiro presented GenomeAI results at Bio conference. Beat AlphaFold on 3 benchmarks.\n2024-10-20: GenomeAI partnered with Pfizer for oncology drug discovery pipeline.',
chunks: [
{ chunk_index: 0, chunk_text: 'Hiro Tanaka is CEO of GenomeAI, using large language models to predict protein folding for drug discovery. Previously research scientist at DeepMind. Published 40+ papers on computational biology.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.7, [T.BIOTECH]: 0.9, [T.FOUNDER]: 0.4}), token_count: 34 },
{ chunk_index: 1, chunk_text: '2024-02-14: Hiro presented GenomeAI results. Beat AlphaFold on 3 benchmarks. 2024-10-20: GenomeAI partnered with Pfizer for oncology drug discovery.', chunk_source: 'timeline', embedding: topicEmbedding({[T.BIOTECH]: 0.6, [T.ANNOUNCEMENT]: 0.5, [T.MEETING]: 0.4}), token_count: 28 },
],
},
{
slug: 'people/iris-washington',
type: 'person',
title: 'Iris Washington',
compiled_truth: 'Iris Washington is VP of Product at CloudScale, an enterprise infrastructure company. Expert in developer experience and platform engineering. Previously PM at AWS Lambda team.',
timeline: '2024-05-18: Iris spoke at re:Invent about serverless at scale. Great talk on cold start optimization.\n2024-09-03: CloudScale acquired by Datadog for $2.1B.',
chunks: [
{ chunk_index: 0, chunk_text: 'Iris Washington is VP of Product at CloudScale, an enterprise infrastructure company. Expert in developer experience and platform engineering. Previously PM at AWS Lambda team.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.ENTERPRISE]: 0.9, [T.ENGINEER]: 0.5, [T.AI]: 0.2}), token_count: 30 },
{ chunk_index: 1, chunk_text: '2024-05-18: Iris spoke at re:Invent about serverless at scale. Cold start optimization. 2024-09-03: CloudScale acquired by Datadog for $2.1B.', chunk_source: 'timeline', embedding: topicEmbedding({[T.ENTERPRISE]: 0.4, [T.ANNOUNCEMENT]: 0.7, [T.MEETING]: 0.3}), token_count: 28 },
],
},
{
slug: 'people/james-park',
type: 'person',
title: 'James Park',
compiled_truth: 'James Park is a climate tech investor and founder of TerraFund. Focuses on hard tech: carbon capture, nuclear fusion, and sustainable materials. Believes climate is a $50T market by 2040.',
timeline: '2024-07-22: James announced TerraFund II, $500M for climate deep tech.\n2024-11-15: Met James at Climate Week. He invested in 3 fusion startups this year.',
chunks: [
{ chunk_index: 0, chunk_text: 'James Park is a climate tech investor and founder of TerraFund. Focuses on hard tech: carbon capture, nuclear fusion, sustainable materials. Climate is a $50T market by 2040.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CLIMATE]: 0.9, [T.INVESTOR]: 0.8, [T.FOUNDER]: 0.3}), token_count: 32 },
{ chunk_index: 1, chunk_text: '2024-07-22: James announced TerraFund II, $500M for climate deep tech. 2024-11-15: Met James at Climate Week. Invested in 3 fusion startups.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CLIMATE]: 0.5, [T.FUNDING]: 0.8, [T.MEETING]: 0.4}), token_count: 28 },
],
},
// ── Companies (10) ───────────────────────────────────────────
{
slug: 'companies/novapay',
type: 'company',
title: 'NovaPay',
compiled_truth: 'NovaPay builds instant cross-border payments for SMBs. Founded by Alice Chen (ex-Stripe). Series B stage, expanding across Southeast Asia. Regulatory-first approach differentiates from competitors.',
timeline: '2024-01-15: NovaPay launched in Thailand. 2024-06-20: Raised $45M Series B. 2024-09-01: Processed $1B in cross-border volume.',
chunks: [
{ chunk_index: 0, chunk_text: 'NovaPay builds instant cross-border payments for SMBs. Founded by Alice Chen (ex-Stripe). Series B stage, expanding across Southeast Asia. Regulatory-first approach differentiates.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.FINTECH]: 1, [T.ENTERPRISE]: 0.4}), token_count: 30 },
{ chunk_index: 1, chunk_text: '2024-01-15: NovaPay launched in Thailand. 2024-06-20: Raised $45M Series B. 2024-09-01: Processed $1B in cross-border volume.', chunk_source: 'timeline', embedding: topicEmbedding({[T.FINTECH]: 0.4, [T.LAUNCH]: 0.5, [T.FUNDING]: 0.6}), token_count: 25 },
],
},
{
slug: 'companies/mindbridge',
type: 'company',
title: 'MindBridge',
compiled_truth: 'MindBridge builds AI diagnostic tools for mental health. 94% concordance with clinical diagnosis. Used by 200+ clinics. Carol Nakamura (CTO) leads the technical vision.',
timeline: '2024-02-28: Presented at AI Health Summit. 2024-07-12: Series A fundraising, targeting $15M. 2024-10-01: FDA breakthrough device designation.',
chunks: [
{ chunk_index: 0, chunk_text: 'MindBridge builds AI diagnostic tools for mental health. 94% concordance with clinical diagnosis. Used by 200+ clinics. Carol Nakamura leads technical vision.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.6, [T.HEALTH]: 0.9, [T.ENTERPRISE]: 0.3}), token_count: 28 },
{ chunk_index: 1, chunk_text: '2024-02-28: AI Health Summit presentation. 2024-07-12: Series A targeting $15M. 2024-10-01: FDA breakthrough device designation.', chunk_source: 'timeline', embedding: topicEmbedding({[T.HEALTH]: 0.5, [T.FUNDING]: 0.5, [T.ANNOUNCEMENT]: 0.6}), token_count: 22 },
],
},
{
slug: 'companies/cryptosafe',
type: 'company',
title: 'CryptoSafe',
compiled_truth: 'CryptoSafe provides institutional-grade custody for digital assets using multi-party computation. Founded by Elena Volkov (ex-Google security). $30M Series A from a16z crypto.',
timeline: '2024-01-20: ETH Denver demo. 2024-06-15: $30M Series A announced. 2024-10-30: Onboarded first sovereign wealth fund client.',
chunks: [
{ chunk_index: 0, chunk_text: 'CryptoSafe provides institutional-grade custody for digital assets using multi-party computation. Founded by Elena Volkov. $30M Series A from a16z crypto.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CRYPTO]: 0.9, [T.ENTERPRISE]: 0.5, [T.FINTECH]: 0.3}), token_count: 26 },
{ chunk_index: 1, chunk_text: '2024-01-20: ETH Denver demo. 2024-06-15: $30M Series A. 2024-10-30: First sovereign wealth fund client.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CRYPTO]: 0.4, [T.FUNDING]: 0.7, [T.ANNOUNCEMENT]: 0.5}), token_count: 20 },
],
},
{
slug: 'companies/robologic',
type: 'company',
title: 'RoboLogic',
compiled_truth: 'RoboLogic builds autonomous warehouse robots for irregular object picking. 98% accuracy on unstructured items. Frank Dubois (head of AI) leads R&D. Deployed at major fulfillment centers.',
timeline: '2024-03-22: Demo day showing. 2024-11-01: Amazon fulfillment deployment in Memphis.',
chunks: [
{ chunk_index: 0, chunk_text: 'RoboLogic builds autonomous warehouse robots for irregular object picking. 98% accuracy. Frank Dubois leads R&D. Deployed at major fulfillment centers.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.ROBOTICS]: 0.9, [T.AI]: 0.6, [T.ENTERPRISE]: 0.4}), token_count: 25 },
{ chunk_index: 1, chunk_text: '2024-03-22: Demo day showing. 2024-11-01: Amazon fulfillment deployment in Memphis.', chunk_source: 'timeline', embedding: topicEmbedding({[T.ROBOTICS]: 0.4, [T.LAUNCH]: 0.8}), token_count: 15 },
],
},
{
slug: 'companies/edustack',
type: 'company',
title: 'EduStack',
compiled_truth: 'EduStack is an AI-powered adaptive learning platform. Personalizes curriculum in real-time based on student performance. Founded by David Okonkwo (ex-Stanford CS). Growing 40% MoM in Nigeria.',
timeline: '2024-05-02: Demo day presentation. 2024-09-18: V2 launch with real-time assessment. 2024-12-01: Expanded to Kenya and Ghana.',
chunks: [
{ chunk_index: 0, chunk_text: 'EduStack is an AI-powered adaptive learning platform. Personalizes curriculum in real-time. Founded by David Okonkwo. Growing 40% MoM in Nigeria.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.5, [T.EDUCATION]: 0.9, [T.CONSUMER]: 0.4}), token_count: 26 },
{ chunk_index: 1, chunk_text: '2024-05-02: Demo day. 2024-09-18: V2 with real-time assessment. 2024-12-01: Expanded to Kenya and Ghana.', chunk_source: 'timeline', embedding: topicEmbedding({[T.EDUCATION]: 0.4, [T.LAUNCH]: 0.7, [T.ANNOUNCEMENT]: 0.3}), token_count: 20 },
],
},
{
slug: 'companies/pixelcraft',
type: 'company', title: 'PixelCraft',
compiled_truth: 'PixelCraft is a design tool that generates working UI components from natural language. Founded by Grace Lee (ex-Figma). 5000 signups in first week of beta.',
timeline: '2024-04-30: Beta launch, 5000 signups. 2024-08-15: Hired 3 Vercel engineers.',
chunks: [
{ chunk_index: 0, chunk_text: 'PixelCraft generates working UI components from natural language. Founded by Grace Lee (ex-Figma). 5000 signups first week.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.6, [T.DESIGNER]: 0.8, [T.CONSUMER]: 0.5}), token_count: 22 },
{ chunk_index: 1, chunk_text: '2024-04-30: Beta launch, 5000 signups. 2024-08-15: Hired 3 Vercel engineers.', chunk_source: 'timeline', embedding: topicEmbedding({[T.LAUNCH]: 0.8, [T.HIRING]: 0.6}), token_count: 14 },
],
},
{
slug: 'companies/genomeai',
type: 'company', title: 'GenomeAI',
compiled_truth: 'GenomeAI uses LLMs to predict protein folding for drug discovery. Beat AlphaFold on 3 benchmarks. CEO Hiro Tanaka (ex-DeepMind). Partnered with Pfizer.',
timeline: '2024-02-14: Bio conference results. 2024-10-20: Pfizer partnership announced.',
chunks: [
{ chunk_index: 0, chunk_text: 'GenomeAI uses LLMs to predict protein folding for drug discovery. Beat AlphaFold on 3 benchmarks. CEO Hiro Tanaka. Pfizer partnership.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.7, [T.BIOTECH]: 0.9}), token_count: 24 },
{ chunk_index: 1, chunk_text: '2024-02-14: Bio conference, beat AlphaFold. 2024-10-20: Pfizer partnership for oncology.', chunk_source: 'timeline', embedding: topicEmbedding({[T.BIOTECH]: 0.5, [T.ANNOUNCEMENT]: 0.7}), token_count: 16 },
],
},
{
slug: 'companies/terrafund',
type: 'company', title: 'TerraFund',
compiled_truth: 'TerraFund is a $500M climate deep tech fund. Founded by James Park. Invests in carbon capture, nuclear fusion, and sustainable materials. Three fusion investments in 2024.',
timeline: '2024-07-22: TerraFund II announced at $500M. 2024-11-15: Climate Week panel.',
chunks: [
{ chunk_index: 0, chunk_text: 'TerraFund is a $500M climate deep tech fund. Founded by James Park. Carbon capture, nuclear fusion, sustainable materials.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CLIMATE]: 0.9, [T.INVESTOR]: 0.6, [T.FUNDING]: 0.3}), token_count: 22 },
{ chunk_index: 1, chunk_text: '2024-07-22: TerraFund II at $500M. 2024-11-15: Climate Week panel.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CLIMATE]: 0.4, [T.FUNDING]: 0.8, [T.ANNOUNCEMENT]: 0.5}), token_count: 14 },
],
},
{
slug: 'companies/cloudscale',
type: 'company', title: 'CloudScale',
compiled_truth: 'CloudScale is an enterprise infrastructure company focused on serverless at scale. Iris Washington is VP Product. Acquired by Datadog for $2.1B in 2024.',
timeline: '2024-05-18: re:Invent talk on cold starts. 2024-09-03: Datadog acquisition at $2.1B.',
chunks: [
{ chunk_index: 0, chunk_text: 'CloudScale is enterprise infrastructure for serverless at scale. VP Product Iris Washington. Acquired by Datadog for $2.1B.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.ENTERPRISE]: 0.9, [T.AI]: 0.2}), token_count: 22 },
{ chunk_index: 1, chunk_text: '2024-05-18: re:Invent cold start talk. 2024-09-03: Datadog acquired CloudScale for $2.1B.', chunk_source: 'timeline', embedding: topicEmbedding({[T.ENTERPRISE]: 0.3, [T.ANNOUNCEMENT]: 0.8}), token_count: 16 },
],
},
{
slug: 'companies/solargrid',
type: 'company', title: 'SolarGrid',
compiled_truth: 'SolarGrid builds distributed solar micro-grids for rural electrification. Bob Martinez is a board member. Operating in 12 African countries.',
timeline: '2024-08-05: Bob introduced the founder. 2024-12-10: SolarGrid hit 1M homes powered.',
chunks: [
{ chunk_index: 0, chunk_text: 'SolarGrid builds distributed solar micro-grids for rural electrification. Bob Martinez board member. Operating in 12 African countries.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CLIMATE]: 0.8, [T.ENTERPRISE]: 0.3}), token_count: 22 },
{ chunk_index: 1, chunk_text: '2024-08-05: Bob introduced founder. 2024-12-10: SolarGrid hit 1M homes powered.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CLIMATE]: 0.3, [T.ANNOUNCEMENT]: 0.5, [T.MEETING]: 0.4}), token_count: 14 },
],
},
// ── Concepts (10) ────────────────────────────────────────────
{
slug: 'concepts/ai-first-companies',
type: 'concept', title: 'AI-First Companies',
compiled_truth: 'AI-first companies embed machine learning into the core product loop, not as a feature bolt-on. Examples: MindBridge (diagnostics), EduStack (adaptive learning), PixelCraft (design). The common pattern is that AI IS the product, not AI-enhanced.',
timeline: '2024-03-01: Wrote first draft of AI-first thesis. 2024-09-15: Revisited after seeing 10 more examples.',
chunks: [
{ chunk_index: 0, chunk_text: 'AI-first companies embed machine learning into the core product loop. MindBridge, EduStack, PixelCraft. AI IS the product, not AI-enhanced.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 1, [T.FOUNDER]: 0.3, [T.ENTERPRISE]: 0.2, [T.CONSUMER]: 0.2}), token_count: 26 },
{ chunk_index: 1, chunk_text: '2024-03-01: First draft of AI-first thesis. 2024-09-15: Revisited after 10 more examples.', chunk_source: 'timeline', embedding: topicEmbedding({[T.AI]: 0.5, [T.TIMELINE]: 0.5}), token_count: 18 },
],
},
{
slug: 'concepts/climate-investing',
type: 'concept', title: 'Climate Tech Investment Thesis',
compiled_truth: 'Climate tech is a $50T market by 2040. Three waves: solar/wind (done), batteries/grid (now), carbon capture/fusion (next). TerraFund and Green Horizon are the key funds. Hard tech wins over software-only.',
timeline: '2024-04-10: Bob articulated the three-wave framework. 2024-11-15: James confirmed fusion timeline at Climate Week.',
chunks: [
{ chunk_index: 0, chunk_text: 'Climate tech is a $50T market by 2040. Three waves: solar/wind (done), batteries/grid (now), carbon capture/fusion (next). TerraFund and Green Horizon key funds.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CLIMATE]: 1, [T.INVESTOR]: 0.5, [T.FUNDING]: 0.3}), token_count: 30 },
{ chunk_index: 1, chunk_text: '2024-04-10: Bob three-wave framework. 2024-11-15: James confirmed fusion timeline at Climate Week.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CLIMATE]: 0.5, [T.MEETING]: 0.5, [T.INVESTOR]: 0.3}), token_count: 18 },
],
},
{
slug: 'concepts/fintech-rails',
type: 'concept', title: 'Payment Rails Infrastructure',
compiled_truth: 'Cross-border payments are still broken. SWIFT takes 3-5 days. NovaPay and similar startups are building real-time rails using local payment networks. Regulatory compliance is the moat, not technology.',
timeline: '2024-03-15: Alice explained regulatory-first approach at Fintech Forum.',
chunks: [
{ chunk_index: 0, chunk_text: 'Cross-border payments are still broken. SWIFT takes 3-5 days. NovaPay building real-time rails. Regulatory compliance is the moat, not technology.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.FINTECH]: 1, [T.ENTERPRISE]: 0.3}), token_count: 26 },
{ chunk_index: 1, chunk_text: '2024-03-15: Alice explained regulatory-first approach at Fintech Forum.', chunk_source: 'timeline', embedding: topicEmbedding({[T.FINTECH]: 0.4, [T.MEETING]: 0.6}), token_count: 12 },
],
},
{
slug: 'concepts/crypto-custody',
type: 'concept', title: 'Institutional Crypto Custody',
compiled_truth: 'Institutional adoption of crypto requires custody solutions that meet banking-grade security standards. MPC (multi-party computation) is the winning architecture. CryptoSafe is leading this space.',
timeline: '2024-01-20: Elena ETH Denver talk. 2024-10-30: First sovereign wealth fund using MPC custody.',
chunks: [
{ chunk_index: 0, chunk_text: 'Institutional crypto adoption requires banking-grade custody. MPC is the winning architecture. CryptoSafe leads.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.CRYPTO]: 0.9, [T.ENTERPRISE]: 0.5}), token_count: 18 },
{ chunk_index: 1, chunk_text: '2024-01-20: Elena ETH Denver talk on MPC. 2024-10-30: First sovereign wealth fund using MPC custody.', chunk_source: 'timeline', embedding: topicEmbedding({[T.CRYPTO]: 0.5, [T.ANNOUNCEMENT]: 0.5, [T.MEETING]: 0.3}), token_count: 18 },
],
},
{
slug: 'concepts/ai-health',
type: 'concept', title: 'AI in Healthcare',
compiled_truth: 'AI in healthcare is moving from research to deployment. MindBridge (mental health, 94% accuracy), GenomeAI (drug discovery, beat AlphaFold). FDA is creating new regulatory pathways for AI diagnostics.',
timeline: '2024-02-28: AI Health Summit. 2024-10-01: MindBridge FDA breakthrough designation.',
chunks: [
{ chunk_index: 0, chunk_text: 'AI in healthcare moving from research to deployment. MindBridge 94% accuracy, GenomeAI beat AlphaFold. FDA creating new AI diagnostic pathways.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.7, [T.HEALTH]: 0.8, [T.BIOTECH]: 0.4}), token_count: 26 },
{ chunk_index: 1, chunk_text: '2024-02-28: AI Health Summit. 2024-10-01: MindBridge FDA breakthrough.', chunk_source: 'timeline', embedding: topicEmbedding({[T.HEALTH]: 0.5, [T.ANNOUNCEMENT]: 0.5, [T.AI]: 0.3}), token_count: 12 },
],
},
{
slug: 'concepts/robotics-warehouse',
type: 'concept', title: 'Warehouse Automation',
compiled_truth: 'Warehouse robotics is moving from structured (conveyor belts, AGVs) to unstructured (picking irregular objects). RoboLogic at 98% accuracy. The bottleneck is manipulation, not navigation.',
timeline: '2024-03-22: RoboLogic demo. 2024-11-01: Amazon deployment validates the market.',
chunks: [
{ chunk_index: 0, chunk_text: 'Warehouse robotics moving from structured to unstructured picking. RoboLogic 98% accuracy. Bottleneck is manipulation, not navigation.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.ROBOTICS]: 0.9, [T.AI]: 0.5, [T.ENTERPRISE]: 0.3}), token_count: 22 },
{ chunk_index: 1, chunk_text: '2024-03-22: RoboLogic demo. 2024-11-01: Amazon deployment validates market.', chunk_source: 'timeline', embedding: topicEmbedding({[T.ROBOTICS]: 0.4, [T.LAUNCH]: 0.6}), token_count: 12 },
],
},
{
slug: 'concepts/ai-education',
type: 'concept', title: 'AI in Education',
compiled_truth: 'Personalized education at scale is now possible with AI. EduStack shows 40% MoM growth. The key insight: adaptive curriculum beats static textbooks because every student learns differently.',
timeline: '2024-05-02: David demo day. 2024-12-01: EduStack expanded to 3 African countries.',
chunks: [
{ chunk_index: 0, chunk_text: 'Personalized education at scale with AI. EduStack 40% MoM growth. Adaptive curriculum beats static textbooks.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.5, [T.EDUCATION]: 0.9, [T.CONSUMER]: 0.3}), token_count: 20 },
{ chunk_index: 1, chunk_text: '2024-05-02: David demo. 2024-12-01: EduStack to Kenya and Ghana.', chunk_source: 'timeline', embedding: topicEmbedding({[T.EDUCATION]: 0.4, [T.LAUNCH]: 0.5}), token_count: 12 },
],
},
{
slug: 'concepts/design-ai',
type: 'concept', title: 'AI-Powered Design Tools',
compiled_truth: 'AI is replacing the mockup-to-code pipeline. PixelCraft generates working components from descriptions. Grace Lee argues designers should think in systems, not screens. The next Figma is AI-native.',
timeline: '2024-04-30: PixelCraft beta launch validated the thesis.',
chunks: [
{ chunk_index: 0, chunk_text: 'AI replacing mockup-to-code pipeline. PixelCraft generates components from descriptions. Grace Lee: think in systems, not screens. Next Figma is AI-native.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.AI]: 0.6, [T.DESIGNER]: 0.9, [T.CONSUMER]: 0.3}), token_count: 28 },
{ chunk_index: 1, chunk_text: '2024-04-30: PixelCraft beta validated the thesis.', chunk_source: 'timeline', embedding: topicEmbedding({[T.DESIGNER]: 0.3, [T.LAUNCH]: 0.5}), token_count: 10 },
],
},
{
slug: 'concepts/acquisitions-2024',
type: 'concept', title: '2024 Notable Acquisitions',
compiled_truth: 'Datadog acquired CloudScale for $2.1B (serverless infrastructure). Signaling: infrastructure consolidation is accelerating. Platform companies are buying specialized tools.',
timeline: '2024-09-03: CloudScale acquisition announced. 2024-09-10: Market reacted positively, Datadog stock up 8%.',
chunks: [
{ chunk_index: 0, chunk_text: 'Datadog acquired CloudScale for $2.1B. Infrastructure consolidation accelerating. Platform companies buying specialized tools.', chunk_source: 'compiled_truth', embedding: topicEmbedding({[T.ENTERPRISE]: 0.7, [T.ANNOUNCEMENT]: 0.5}), token_count: 20 },
{ chunk_index: 1, chunk_text: '2024-09-03: CloudScale acquisition. 2024-09-10: Datadog stock up 8%.', chunk_source: 'timeline', embedding: topicEmbedding({[T.ENTERPRISE]: 0.3, [T.ANNOUNCEMENT]: 0.7}), token_count: 12 },
],
},
];
// ─── Benchmark Queries (20) ──────────────────────────────────────
interface BenchmarkQuery {
id: string;
query: string;
queryEmbedding: Float32Array;
relevant: string[];
grades?: Record<string, number>;
expectedSource: 'compiled_truth' | 'timeline';
description: string;
}
const QUERIES: BenchmarkQuery[] = [
// Entity lookups (should get compiled truth)
{ id: 'q01', query: 'Who is Alice Chen?', queryEmbedding: topicEmbedding({[T.FINTECH]: 0.8, [T.FOUNDER]: 0.5}), relevant: ['people/alice-chen', 'companies/novapay'], grades: {'people/alice-chen': 3, 'companies/novapay': 1}, expectedSource: 'compiled_truth', description: 'Person lookup: Alice Chen' },
{ id: 'q02', query: 'What does MindBridge do?', queryEmbedding: topicEmbedding({[T.AI]: 0.5, [T.HEALTH]: 0.8}), relevant: ['companies/mindbridge', 'people/carol-nakamura', 'concepts/ai-health'], grades: {'companies/mindbridge': 3, 'people/carol-nakamura': 2, 'concepts/ai-health': 1}, expectedSource: 'compiled_truth', description: 'Company lookup: MindBridge' },
{ id: 'q03', query: 'Tell me about climate tech investing', queryEmbedding: topicEmbedding({[T.CLIMATE]: 0.9, [T.INVESTOR]: 0.5}), relevant: ['concepts/climate-investing', 'people/bob-martinez', 'people/james-park', 'companies/terrafund'], grades: {'concepts/climate-investing': 3, 'people/james-park': 2, 'people/bob-martinez': 2, 'companies/terrafund': 1}, expectedSource: 'compiled_truth', description: 'Topic overview: climate investing' },
// Temporal queries (should get timeline)
{ id: 'q04', query: 'When did we last meet Alice?', queryEmbedding: topicEmbedding({[T.FINTECH]: 0.4, [T.MEETING]: 0.9}), relevant: ['people/alice-chen'], expectedSource: 'timeline', description: 'Temporal: last meeting with Alice' },
{ id: 'q05', query: 'Recent updates on GenomeAI', queryEmbedding: topicEmbedding({[T.BIOTECH]: 0.6, [T.ANNOUNCEMENT]: 0.5}), relevant: ['companies/genomeai', 'people/hiro-tanaka'], grades: {'companies/genomeai': 3, 'people/hiro-tanaka': 1}, expectedSource: 'timeline', description: 'Temporal: GenomeAI updates' },
{ id: 'q06', query: 'What happened with the CloudScale acquisition?', queryEmbedding: topicEmbedding({[T.ENTERPRISE]: 0.6, [T.ANNOUNCEMENT]: 0.8}), relevant: ['companies/cloudscale', 'concepts/acquisitions-2024', 'people/iris-washington'], grades: {'companies/cloudscale': 3, 'concepts/acquisitions-2024': 2, 'people/iris-washington': 1}, expectedSource: 'timeline', description: 'Event: CloudScale acquisition' },
// Cross-entity queries (tests relationship understanding)
{ id: 'q07', query: 'Alice Chen NovaPay cross-border payments', queryEmbedding: topicEmbedding({[T.FINTECH]: 0.9, [T.FOUNDER]: 0.3}), relevant: ['people/alice-chen', 'companies/novapay', 'concepts/fintech-rails'], grades: {'people/alice-chen': 2, 'companies/novapay': 3, 'concepts/fintech-rails': 2}, expectedSource: 'compiled_truth', description: 'Cross-entity: Alice + NovaPay' },
{ id: 'q08', query: 'Carol Nakamura MindBridge AI health', queryEmbedding: topicEmbedding({[T.AI]: 0.5, [T.HEALTH]: 0.7, [T.FOUNDER]: 0.3}), relevant: ['people/carol-nakamura', 'companies/mindbridge', 'concepts/ai-health'], grades: {'people/carol-nakamura': 2, 'companies/mindbridge': 2, 'concepts/ai-health': 2}, expectedSource: 'compiled_truth', description: 'Cross-entity: Carol + MindBridge' },
// Competitive/thematic queries (multiple relevant pages)
{ id: 'q09', query: 'AI companies building real products', queryEmbedding: topicEmbedding({[T.AI]: 0.9, [T.FOUNDER]: 0.3, [T.CONSUMER]: 0.2}), relevant: ['concepts/ai-first-companies', 'companies/mindbridge', 'companies/edustack', 'companies/pixelcraft', 'companies/genomeai'], grades: {'concepts/ai-first-companies': 3, 'companies/mindbridge': 2, 'companies/edustack': 2, 'companies/pixelcraft': 2, 'companies/genomeai': 2}, expectedSource: 'compiled_truth', description: 'Thematic: AI companies' },
{ id: 'q10', query: 'Who raised funding recently?', queryEmbedding: topicEmbedding({[T.FUNDING]: 0.9, [T.ANNOUNCEMENT]: 0.4}), relevant: ['companies/novapay', 'companies/cryptosafe', 'companies/terrafund', 'people/carol-nakamura'], grades: {'companies/novapay': 2, 'companies/cryptosafe': 2, 'companies/terrafund': 2, 'people/carol-nakamura': 1}, expectedSource: 'timeline', description: 'Temporal: recent funding rounds' },
// Hard disambiguation queries
{ id: 'q11', query: 'Bob and James climate investments', queryEmbedding: topicEmbedding({[T.CLIMATE]: 0.8, [T.INVESTOR]: 0.6}), relevant: ['people/bob-martinez', 'people/james-park', 'concepts/climate-investing', 'companies/terrafund'], grades: {'people/bob-martinez': 2, 'people/james-park': 2, 'concepts/climate-investing': 2, 'companies/terrafund': 1}, expectedSource: 'compiled_truth', description: 'Disambiguation: two climate investors' },
{ id: 'q12', query: 'AI replacing designers', queryEmbedding: topicEmbedding({[T.AI]: 0.6, [T.DESIGNER]: 0.8}), relevant: ['concepts/design-ai', 'companies/pixelcraft', 'people/grace-lee'], grades: {'concepts/design-ai': 3, 'companies/pixelcraft': 2, 'people/grace-lee': 2}, expectedSource: 'compiled_truth', description: 'Topic: AI and design' },
// Full context requests
{ id: 'q13', query: 'Give me everything on RoboLogic', queryEmbedding: topicEmbedding({[T.ROBOTICS]: 0.9, [T.AI]: 0.4}), relevant: ['companies/robologic', 'people/frank-dubois', 'concepts/robotics-warehouse'], grades: {'companies/robologic': 3, 'people/frank-dubois': 2, 'concepts/robotics-warehouse': 1}, expectedSource: 'timeline', description: 'Full context: RoboLogic' },
{ id: 'q14', query: 'Deep dive on crypto custody', queryEmbedding: topicEmbedding({[T.CRYPTO]: 0.9, [T.ENTERPRISE]: 0.4}), relevant: ['concepts/crypto-custody', 'companies/cryptosafe', 'people/elena-volkov'], grades: {'concepts/crypto-custody': 3, 'companies/cryptosafe': 2, 'people/elena-volkov': 2}, expectedSource: 'timeline', description: 'Full context: crypto custody' },
// Tricky queries that test boost vs natural
{ id: 'q15', query: 'Education technology Africa growth', queryEmbedding: topicEmbedding({[T.EDUCATION]: 0.8, [T.CONSUMER]: 0.3}), relevant: ['companies/edustack', 'people/david-okonkwo', 'concepts/ai-education'], grades: {'companies/edustack': 3, 'people/david-okonkwo': 2, 'concepts/ai-education': 2}, expectedSource: 'compiled_truth', description: 'Topic: edtech in Africa' },
{ id: 'q16', query: 'What launched this year?', queryEmbedding: topicEmbedding({[T.LAUNCH]: 0.9, [T.ANNOUNCEMENT]: 0.4}), relevant: ['companies/novapay', 'companies/pixelcraft', 'companies/edustack', 'companies/robologic'], grades: {'companies/pixelcraft': 2, 'companies/edustack': 2, 'companies/novapay': 2, 'companies/robologic': 2}, expectedSource: 'timeline', description: 'Temporal: 2024 launches' },
// Narrow expert queries
{ id: 'q17', query: 'MPC multi-party computation wallets', queryEmbedding: topicEmbedding({[T.CRYPTO]: 0.8, [T.ENGINEER]: 0.4}), relevant: ['people/elena-volkov', 'companies/cryptosafe', 'concepts/crypto-custody'], grades: {'people/elena-volkov': 3, 'companies/cryptosafe': 2, 'concepts/crypto-custody': 2}, expectedSource: 'compiled_truth', description: 'Expert: MPC wallets' },
{ id: 'q18', query: 'Protein folding drug discovery LLMs', queryEmbedding: topicEmbedding({[T.AI]: 0.6, [T.BIOTECH]: 0.9}), relevant: ['companies/genomeai', 'people/hiro-tanaka', 'concepts/ai-health'], grades: {'companies/genomeai': 3, 'people/hiro-tanaka': 2, 'concepts/ai-health': 1}, expectedSource: 'compiled_truth', description: 'Expert: protein folding AI' },
// Negative control
{ id: 'q19', query: 'quantum computing error correction', queryEmbedding: topicEmbedding({100: 1}), relevant: [], expectedSource: 'compiled_truth', description: 'Negative: no relevant pages' },
// Ambiguous query (could be entity OR temporal)
{ id: 'q20', query: 'EduStack Nigeria', queryEmbedding: topicEmbedding({[T.EDUCATION]: 0.7, [T.CONSUMER]: 0.3}), relevant: ['companies/edustack', 'people/david-okonkwo'], grades: {'companies/edustack': 3, 'people/david-okonkwo': 1}, expectedSource: 'compiled_truth', description: 'Ambiguous: EduStack in Nigeria' },
];
// ─── Benchmark Runner ────────────────────────────────────────────
interface RunResult {
queryId: string;
hits: SearchResult[];
// Page-level metrics (traditional IR)
precision1: number;
precision5: number;
recall5: number;
mrrScore: number;
ndcg5: number;
// Chunk-level metrics (what PR#64 actually improves)
sourceCorrect: boolean; // Is the top chunk the right source type?
chunksPerPage: number; // Avg chunks per unique page in results
compiledTruthFirst: number; // For entity queries: is compiled_truth the first chunk per page?
timelineAccessible: boolean; // Are timeline chunks present in results?
compiledTruthGuaranteed: boolean; // Does every page have at least 1 compiled_truth chunk?
uniquePages: number; // How many distinct pages appear
compiledTruthRatio: number; // What % of result chunks are compiled_truth
}
function analyzeRun(q: BenchmarkQuery, hits: SearchResult[]): RunResult {
const slugs = hits.map(r => r.slug);
const rel = new Set(q.relevant);
const grades = new Map(Object.entries(q.grades ?? Object.fromEntries(q.relevant.map(s => [s, 1]))));
// Page-level metrics
const uniqueSlugs = [...new Set(slugs)];
const chunksPerPage = uniqueSlugs.length > 0 ? hits.length / uniqueSlugs.length : 0;
// Chunk-source analysis per page
const byPage = new Map<string, SearchResult[]>();
for (const h of hits) {
const arr = byPage.get(h.slug) || [];
arr.push(h);
byPage.set(h.slug, arr);
}
// For entity queries: is the first chunk of each relevant page compiled_truth?
let ctFirstCount = 0, ctFirstTotal = 0;
for (const [slug, chunks] of byPage) {
if (rel.has(slug) && q.expectedSource === 'compiled_truth') {
ctFirstTotal++;
if (chunks[0]?.chunk_source === 'compiled_truth') ctFirstCount++;
}
}
// Compiled truth guarantee: does every page in results have at least 1 CT chunk?
let ctGuaranteed = true;
for (const [_, chunks] of byPage) {
if (!chunks.some(c => c.chunk_source === 'compiled_truth')) {
ctGuaranteed = false;
break;
}
}
const ctChunks = hits.filter(h => h.chunk_source === 'compiled_truth').length;
return {
queryId: q.id, hits,
precision1: precisionAtK(slugs, rel, 1),
precision5: precisionAtK(slugs, rel, 5),
recall5: recallAtK(slugs, rel, 5),
mrrScore: mrr(slugs, rel),
ndcg5: ndcgAtK(slugs, grades, 5),
sourceCorrect: hits.length > 0 ? hits[0].chunk_source === q.expectedSource : q.relevant.length === 0,
chunksPerPage,
compiledTruthFirst: ctFirstTotal > 0 ? ctFirstCount / ctFirstTotal : -1,
timelineAccessible: hits.some(h => h.chunk_source === 'timeline'),
compiledTruthGuaranteed: ctGuaranteed,
uniquePages: uniqueSlugs.length,
compiledTruthRatio: hits.length > 0 ? ctChunks / hits.length : 0,
};
}
async function runBenchmark(engine: PGLiteEngine, queries: BenchmarkQuery[], mode: 'baseline' | 'boost' | 'intent'): Promise<RunResult[]> {
const results: RunResult[] = [];
for (const q of queries) {
let detail: 'low' | 'medium' | 'high' | undefined;
let applyBoost = true;
if (mode === 'intent') {
detail = autoDetectDetail(q.query);
applyBoost = detail !== 'high';
} else if (mode === 'baseline') {
applyBoost = false;
}
const kw = await engine.searchKeyword(q.query, { limit: 20, detail });
const vec = await engine.searchVector(q.queryEmbedding, { limit: 20, detail });
const fused = mode === 'baseline'
? rrfFusionBaseline([vec, kw])
: rrfFusion([vec, kw], RRF_K, applyBoost);
const deduped = dedupResults(fused);
const top = deduped.slice(0, 10);
results.push(analyzeRun(q, top));
}
return results;
}
function rrfFusionBaseline(lists: SearchResult[][]): SearchResult[] {
const scores = new Map<string, { result: SearchResult; score: number }>();
for (const list of lists) {
for (let rank = 0; rank < list.length; rank++) {
const r = list[rank];
const key = `${r.slug}:${r.chunk_text.slice(0, 50)}`;
const existing = scores.get(key);
const s = 1 / (RRF_K + rank);
if (existing) existing.score += s;
else scores.set(key, { result: r, score: s });
}
}
return Array.from(scores.values()).sort((a, b) => b.score - a.score).map(({ result, score }) => ({ ...result, score }));
}
// ─── Output ──────────────────────────────────────────────────────
interface AggMetrics {
p1: number; p5: number; r5: number; mrr: number; ndcg: number;
srcAcc: number;
avgChunksPerPage: number;
ctFirstRate: number; // % of entity queries where compiled_truth is first per page
timelineRate: number; // % of temporal queries where timeline is accessible
ctGuaranteeRate: number; // % of queries where every page has a CT chunk
avgUniquePages: number;
avgCtRatio: number;
}
function aggregate(results: RunResult[], queries: BenchmarkQuery[]): AggMetrics {
const v = results.filter(r => queries.find(q => q.id === r.queryId)!.relevant.length > 0);
const entityQ = v.filter(r => queries.find(q => q.id === r.queryId)!.expectedSource === 'compiled_truth');
const temporalQ = v.filter(r => queries.find(q => q.id === r.queryId)!.expectedSource === 'timeline');
const ctFirstValid = entityQ.filter(r => r.compiledTruthFirst >= 0);
return {
p1: v.reduce((s, r) => s + r.precision1, 0) / v.length,
p5: v.reduce((s, r) => s + r.precision5, 0) / v.length,
r5: v.reduce((s, r) => s + r.recall5, 0) / v.length,
mrr: v.reduce((s, r) => s + r.mrrScore, 0) / v.length,
ndcg: v.reduce((s, r) => s + r.ndcg5, 0) / v.length,
srcAcc: v.filter(r => r.sourceCorrect).length / v.length,
avgChunksPerPage: v.reduce((s, r) => s + r.chunksPerPage, 0) / v.length,
ctFirstRate: ctFirstValid.length > 0 ? ctFirstValid.reduce((s, r) => s + r.compiledTruthFirst, 0) / ctFirstValid.length : 0,
timelineRate: temporalQ.length > 0 ? temporalQ.filter(r => r.timelineAccessible).length / temporalQ.length : 0,
ctGuaranteeRate: v.filter(r => r.compiledTruthGuaranteed).length / v.length,
avgUniquePages: v.reduce((s, r) => s + r.uniquePages, 0) / v.length,
avgCtRatio: v.reduce((s, r) => s + r.compiledTruthRatio, 0) / v.length,
};
}
function d(a: number, b: number): string {
const v = a - b;
return `${v >= 0 ? '+' : ''}${v.toFixed(3)}`;
}
function pct(v: number): string { return `${(v * 100).toFixed(1)}%`; }
// ─── Main ────────────────────────────────────────────────────────
async function main() {
const engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
for (const page of PAGES) {
await engine.putPage(page.slug, { type: page.type, title: page.title, compiled_truth: page.compiled_truth, timeline: page.timeline });
await engine.upsertChunks(page.slug, page.chunks);
}
console.log(`Seeded ${PAGES.length} pages, ${PAGES.reduce((s, p) => s + p.chunks.length, 0)} chunks`);
console.log(`Running ${QUERIES.length} queries x 3 configurations...\n`);
const baseline = await runBenchmark(engine, QUERIES, 'baseline');
const boosted = await runBenchmark(engine, QUERIES, 'boost');
const withIntent = await runBenchmark(engine, QUERIES, 'intent');
const bm = aggregate(baseline, QUERIES);
const am = aggregate(boosted, QUERIES);
const im = aggregate(withIntent, QUERIES);
const date = new Date().toISOString().split('T')[0];
const md: string[] = [];
md.push(`# Search Quality Benchmark: ${date}`);
md.push('');
md.push(`## Overview`);
md.push('');
md.push(`- **${PAGES.length} pages** (${PAGES.filter(p => p.type === 'person').length} people, ${PAGES.filter(p => p.type === 'company').length} companies, ${PAGES.filter(p => p.type === 'concept').length} concepts)`);
md.push(`- **${PAGES.reduce((s, p) => s + p.chunks.length, 0)} chunks** with overlapping semantic embeddings`);
md.push(`- **${QUERIES.length} queries** with graded relevance (1-3 grades, multiple relevant pages)`);
md.push(`- **3 configurations:** baseline, boost only, boost + intent classifier`);
md.push('');
md.push('All data is fictional. No private information. Embeddings use shared topic dimensions');
md.push('to simulate real semantic overlap (e.g., "AI" appears in health, education, design, robotics).');
md.push('');
md.push('Inspired by [Ramp Labs\' "Latent Briefing" paper](https://ramp.com) (April 2026).');
md.push('');
// ─── Traditional IR metrics ───────────────────────────────────
md.push('## Page-Level Retrieval (Traditional IR)');
md.push('');
md.push('*"Did we find the right page?"*');
md.push('');
md.push('| Metric | A. Baseline | B. Boost | C. Intent | B vs A | C vs A |');
md.push('|--------|-------------|----------|-----------|--------|--------|');
md.push(`| P@1 | ${bm.p1.toFixed(3)} | ${am.p1.toFixed(3)} | ${im.p1.toFixed(3)} | ${d(am.p1, bm.p1)} | ${d(im.p1, bm.p1)} |`);
md.push(`| P@5 | ${bm.p5.toFixed(3)} | ${am.p5.toFixed(3)} | ${im.p5.toFixed(3)} | ${d(am.p5, bm.p5)} | ${d(im.p5, bm.p5)} |`);
md.push(`| Recall@5 | ${bm.r5.toFixed(3)} | ${am.r5.toFixed(3)} | ${im.r5.toFixed(3)} | ${d(am.r5, bm.r5)} | ${d(im.r5, bm.r5)} |`);
md.push(`| MRR | ${bm.mrr.toFixed(3)} | ${am.mrr.toFixed(3)} | ${im.mrr.toFixed(3)} | ${d(am.mrr, bm.mrr)} | ${d(im.mrr, bm.mrr)} |`);
md.push(`| nDCG@5 | ${bm.ndcg.toFixed(3)} | ${am.ndcg.toFixed(3)} | ${im.ndcg.toFixed(3)} | ${d(am.ndcg, bm.ndcg)} | ${d(im.ndcg, bm.ndcg)} |`);
md.push('');
// ─── Chunk-level metrics (the real improvements) ──────────────
md.push('## Chunk-Level Quality (What PR#64 Actually Improves)');
md.push('');
md.push('*"Did we find the right CHUNK from the right page?"*');
md.push('');
md.push('| Metric | A. Baseline | B. Boost | C. Intent | B vs A | C vs A |');
md.push('|--------|-------------|----------|-----------|--------|--------|');
md.push(`| Source accuracy (top chunk = expected type) | ${pct(bm.srcAcc)} | ${pct(am.srcAcc)} | ${pct(im.srcAcc)} | ${d(am.srcAcc, bm.srcAcc)} | ${d(im.srcAcc, bm.srcAcc)} |`);
md.push(`| CT-first rate (entity Qs: CT chunk leads per page) | ${pct(bm.ctFirstRate)} | ${pct(am.ctFirstRate)} | ${pct(im.ctFirstRate)} | ${d(am.ctFirstRate, bm.ctFirstRate)} | ${d(im.ctFirstRate, bm.ctFirstRate)} |`);
md.push(`| Timeline accessible (temporal Qs: TL in results) | ${pct(bm.timelineRate)} | ${pct(am.timelineRate)} | ${pct(im.timelineRate)} | ${d(am.timelineRate, bm.timelineRate)} | ${d(im.timelineRate, bm.timelineRate)} |`);
md.push(`| CT guarantee (every page has a CT chunk) | ${pct(bm.ctGuaranteeRate)} | ${pct(am.ctGuaranteeRate)} | ${pct(im.ctGuaranteeRate)} | ${d(am.ctGuaranteeRate, bm.ctGuaranteeRate)} | ${d(im.ctGuaranteeRate, bm.ctGuaranteeRate)} |`);
md.push(`| Avg chunks per page in results | ${bm.avgChunksPerPage.toFixed(2)} | ${am.avgChunksPerPage.toFixed(2)} | ${im.avgChunksPerPage.toFixed(2)} | ${d(am.avgChunksPerPage, bm.avgChunksPerPage)} | ${d(im.avgChunksPerPage, bm.avgChunksPerPage)} |`);
md.push(`| Avg unique pages in top-10 | ${bm.avgUniquePages.toFixed(1)} | ${am.avgUniquePages.toFixed(1)} | ${im.avgUniquePages.toFixed(1)} | ${d(am.avgUniquePages, bm.avgUniquePages)} | ${d(im.avgUniquePages, bm.avgUniquePages)} |`);
md.push(`| Compiled truth ratio in results | ${pct(bm.avgCtRatio)} | ${pct(am.avgCtRatio)} | ${pct(im.avgCtRatio)} | ${d(am.avgCtRatio, bm.avgCtRatio)} | ${d(im.avgCtRatio, bm.avgCtRatio)} |`);
md.push('');
// ─── Per-query breakdown ──────────────────────────────────────
md.push('## Per-Query Detail');
md.push('');
md.push('| # | Query | Type | Detail | P@1 B/C | Src B→C | CT 1st B/C | Pages B/C |');
md.push('|---|-------|------|--------|---------|---------|------------|-----------|');
for (let i = 0; i < QUERIES.length; i++) {
const q = QUERIES[i];
if (q.relevant.length === 0) continue;
const b = baseline[i], c = withIntent[i];
const detail = autoDetectDetail(q.query) ?? 'med';
const srcB = b.hits[0]?.chunk_source?.slice(0, 4) ?? '-';
const srcC = c.hits[0]?.chunk_source?.slice(0, 4) ?? '-';
const exp = q.expectedSource.slice(0, 4);
const srcMatch = `${srcB}→${srcC} (${exp})`;
const ctB = b.compiledTruthFirst >= 0 ? pct(b.compiledTruthFirst) : 'n/a';
const ctC = c.compiledTruthFirst >= 0 ? pct(c.compiledTruthFirst) : 'n/a';
md.push(`| ${q.id} | ${q.description.slice(0, 38)} | ${q.expectedSource.slice(0,4)} | ${detail.slice(0,3)} | ${b.precision1.toFixed(0)}/${c.precision1.toFixed(0)} | ${srcMatch} | ${ctB}/${ctC} | ${b.uniquePages}/${c.uniquePages} |`);
}
md.push('');
// ─── Analysis ─────────────────────────────────────────────────
md.push('## Analysis');
md.push('');
const improvements: string[] = [];
const regressions: string[] = [];
if (im.srcAcc > bm.srcAcc) improvements.push(`Source accuracy: ${pct(bm.srcAcc)} → ${pct(im.srcAcc)}`);
if (im.srcAcc < bm.srcAcc) regressions.push(`Source accuracy: ${pct(bm.srcAcc)} → ${pct(im.srcAcc)}`);
if (im.ctFirstRate > bm.ctFirstRate) improvements.push(`CT-first rate: ${pct(bm.ctFirstRate)} → ${pct(im.ctFirstRate)}`);
if (im.ctGuaranteeRate > bm.ctGuaranteeRate) improvements.push(`CT guarantee: ${pct(bm.ctGuaranteeRate)} → ${pct(im.ctGuaranteeRate)}`);
if (im.timelineRate > bm.timelineRate) improvements.push(`Timeline accessible: ${pct(bm.timelineRate)} → ${pct(im.timelineRate)}`);
if (im.avgChunksPerPage > bm.avgChunksPerPage) improvements.push(`Chunks/page: ${bm.avgChunksPerPage.toFixed(2)} → ${im.avgChunksPerPage.toFixed(2)}`);
if (im.avgUniquePages > bm.avgUniquePages) improvements.push(`Unique pages: ${bm.avgUniquePages.toFixed(1)} → ${im.avgUniquePages.toFixed(1)}`);
if (improvements.length > 0) {
md.push('### Improvements (C vs A)');
for (const imp of improvements) md.push(`- ${imp}`);
md.push('');
}
if (regressions.length > 0) {
md.push('### Regressions (C vs A)');
for (const reg of regressions) md.push(`- ${reg}`);
md.push('');
}
if (improvements.length === 0 && regressions.length === 0) {
md.push('No chunk-level regressions or improvements detected in this run.');
md.push('');
}
// Boost-only damage report
md.push('### Boost-Only Damage Report (B vs A)');
md.push('');
md.push('The boost without the intent classifier causes these regressions:');
md.push('');
if (am.srcAcc < bm.srcAcc) md.push(`- Source accuracy drops: ${pct(bm.srcAcc)} → ${pct(am.srcAcc)} (${((am.srcAcc - bm.srcAcc) * 100).toFixed(1)}pp)`);
if (am.timelineRate < bm.timelineRate) md.push(`- Timeline accessibility drops: ${pct(bm.timelineRate)} → ${pct(am.timelineRate)}`);
if (am.p1 < bm.p1) md.push(`- P@1 drops: ${bm.p1.toFixed(3)} → ${am.p1.toFixed(3)}`);
md.push('');
md.push('The intent classifier recovers all of these by routing temporal/event queries to detail=high (no boost).');
md.push('');
md.push('## Methodology');
md.push('');
md.push('- **Engine:** PGLite (in-memory Postgres 17.5 via WASM)');
md.push('- **Embeddings:** Normalized topic vectors with shared dimensions (25 topic axes)');
md.push('- **Overlap:** Multiple pages share topics (e.g., 5 pages relevant for "AI companies")');
md.push('- **Graded relevance:** 1-3 grades per query (3 = primary, 1 = tangentially relevant)');
md.push('');
md.push('### Metrics explained');
md.push('');
md.push('**Page-level (traditional IR):** P@k, Recall@k, MRR, nDCG@5 measure "did we find the right page?"');
md.push('');
md.push('**Chunk-level (what matters for brain search):**');
md.push('- **Source accuracy:** Is the very first chunk the right TYPE for this query? Entity lookup → compiled truth. Temporal query → timeline.');
md.push('- **CT-first rate:** For entity queries, is compiled truth the FIRST chunk shown per page? (Not buried below timeline noise.)');
md.push('- **Timeline accessible:** For temporal queries, do timeline chunks actually appear in results? (Not filtered out by the boost.)');
md.push('- **CT guarantee:** Does every page in results have at least one compiled truth chunk? (Source-aware dedup.)');
md.push('- **Chunks/page:** How many chunks per page appear? More = richer context for the agent.');
md.push('- **Unique pages:** How many distinct pages in top-10? More = broader coverage.');
md.push('');
md.push('### Configurations');
md.push('- A. **Baseline:** RRF K=60, no normalization, no boost, text-prefix dedup key');
md.push('- B. **Boost only:** RRF normalized to 0-1, 2.0x compiled_truth boost, chunk_id dedup key, source-aware dedup');
md.push('- C. **Boost + Intent:** B + heuristic intent classifier auto-selects detail level. Entity queries get detail=low (CT only). Temporal/event queries get detail=high (no boost, natural ranking). General queries get default medium.');
const output = md.join('\n');
console.log(output);
// Note: this benchmark used to write to docs/benchmarks/{date}.md, but
// docs/benchmarks/ is now consolidated into BrainBench v1 (one file per
// dated benchmark run). Output goes to stdout only; redirect if you want
// to save it.
await engine.disconnect();
}
main().catch(console.error);
Reference in New Issue View Git Blame Copy Permalink