mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 03:12:32 +00:00
Garry TanandClaude Opus 4.7
633be384c9
Second external baseline for BrainBench. Pure cosine-similarity ranking using the SAME text-embedding-3-large model gbrain uses internally — apples-to-apples on the embedding layer so any gbrain lead reflects the graph + hybrid fusion, not a better embedder. Files: eval/runner/adapters/vector-only.ts ~130 LOC eval/runner/adapters/vector-only.test.ts 6 unit tests (cosine math) Design: - One vector per page (title + compiled_truth + timeline, capped 8K chars). - No chunking (intentional; chunked vector RAG would be EXT-2b later). - No keyword fallback (that's EXT-3 hybrid-without-graph). - Embeddings in batches of 50 via existing src/core/embedding.ts (retry+backoff). - Cost on 240 pages: ~$0.02/run. Three-adapter side-by-side on 240-page rich-prose corpus, 145 relational queries: | Adapter | P@5 | R@5 | Correct top-5 | |---------------|--------|--------|---------------| | gbrain-after | 49.1% | 97.9% | 248/261 | | ripgrep-bm25 | 17.1% | 62.4% | 124/261 | | vector-only | 10.8% | 40.7% | 78/261 | Interesting finding: vector-only scores WORSE than BM25 on relational queries like "Who invested in X?" — exact entity match matters more than semantic similarity for these templates. BM25 nails the entity-name term; vector-only returns topically-similar-but-not-mentioning pages. This is the known failure mode of pure-vector RAG on precise relational/identity queries. Real-world vector RAG systems always add keyword fallback; EXT-3 (hybrid-without-graph) will be that fairer comparator. gbrain's lead widens in vector-only comparison: +38.4 pts P@5, +57.2 pts R@5. The graph layer is doing the heavy lifting for relational traversal; pure vector RAG can't express "traverse 'attended' edges from this meeting page." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
52 lines
1.9 KiB
TypeScript
52 lines
1.9 KiB
TypeScript
import { describe, test, expect } from 'bun:test';
|
|
import { _cosine } from './vector-only.ts';
|
|
|
|
// Note: VectorOnlyAdapter.init/query require a live embedding API key.
|
|
// Those end-to-end tests live in a smoke-test class and gate on OPENAI_API_KEY.
|
|
// Here we unit-test the pure-function pieces.
|
|
|
|
describe('vector-only adapter (pure helpers)', () => {
|
|
test('cosine of identical vectors = 1.0', () => {
|
|
const a = new Float32Array([1, 2, 3, 4]);
|
|
const b = new Float32Array([1, 2, 3, 4]);
|
|
expect(_cosine(a, b)).toBeCloseTo(1.0, 6);
|
|
});
|
|
|
|
test('cosine of opposite vectors = -1.0', () => {
|
|
const a = new Float32Array([1, 2, 3]);
|
|
const b = new Float32Array([-1, -2, -3]);
|
|
expect(_cosine(a, b)).toBeCloseTo(-1.0, 6);
|
|
});
|
|
|
|
test('cosine of orthogonal vectors = 0', () => {
|
|
const a = new Float32Array([1, 0, 0]);
|
|
const b = new Float32Array([0, 1, 0]);
|
|
expect(_cosine(a, b)).toBeCloseTo(0.0, 6);
|
|
});
|
|
|
|
test('cosine handles zero vector by returning 0', () => {
|
|
const a = new Float32Array([0, 0, 0]);
|
|
const b = new Float32Array([1, 2, 3]);
|
|
expect(_cosine(a, b)).toBe(0);
|
|
});
|
|
|
|
test('cosine is scale-invariant', () => {
|
|
const a = new Float32Array([1, 2, 3]);
|
|
const b = new Float32Array([2, 4, 6]);
|
|
// Same direction, different magnitudes; cosine should still be 1.
|
|
expect(_cosine(a, b)).toBeCloseTo(1.0, 6);
|
|
});
|
|
|
|
test('cosine returns 0 on mismatched-length vectors at the tail', () => {
|
|
// Uses min(len) — shorter vector's dimensions are compared, longer's
|
|
// extras are implicitly dropped. Produces a sensible number even
|
|
// when upstream glue has a dim mismatch bug; helps fail-soft rather
|
|
// than crash the benchmark.
|
|
const a = new Float32Array([1, 1, 1]);
|
|
const b = new Float32Array([1, 1]);
|
|
const sim = _cosine(a, b);
|
|
expect(sim).toBeGreaterThan(0);
|
|
expect(sim).toBeLessThanOrEqual(1);
|
|
});
|
|
});
|