admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(TODOS): mark BrainBench Cats 5/6/8/9/11 + v0.10.5 inferLinkType as completed

Browse Source 
All five BrainBench categories shipped in v0.20.0 (to the gbrain-evals
sibling repo). v0.10.5 inferLinkType regex expansion shipped in-tree.

Remaining P1 BrainBench work: Cat 1+2 at full scale (2-3K pages) —
currently 240 pages in world-v1 corpus.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-23 22:47:50 -07:00
co-authored by Claude Opus 4.7
parent 6f2171984e
commit d035a72d02
1 changed files with 31 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TODOS.md
+31 -48
View File
@@ -11,44 +11,45 @@ Both checks shipped as real implementations, not just filed issues:
`DEFERRED[]` in `src/commands/check-resolvable.ts` is now empty — v0.19 shipped both deferred checks as working code paths, not as issue URLs. The export stays in place for future deferred checks.
## P1 (BrainBench v1.1 — categories deferred from PR #188)
### ~~BrainBench Cats 5/6/8/9/11 — shipped to sibling repo~~
**Completed:** v0.20.0 (2026-04-23)
### BrainBench Cat 5: Source Attribution / Provenance
**What:** Eval that gbrain correctly cites the right page when claiming fact F, and resolves source-conflict cases (3 sources disagree on $5M raise — which wins?). 200 queries across citation/provenance/conflict sub-categories on a 300-entity dataset with deliberately-conflicting sources.
All five previously-deferred BrainBench categories shipped as working runners
in the sibling repo [github.com/garrytan/gbrain-evals](https://github.com/garrytan/gbrain-evals):
**Why deferred from PR #188:** Needs ~$100-200 of Opus tokens to generate the conflict-graph dataset. v1 scope was procedural-only.
- **Cat 5 Provenance** — `eval/runner/cat5-provenance.ts` with dedicated `classify_claim` tool (3-way label: `supported | unsupported | over-generalized`)
- **Cat 6 Prose-scale auto-link precision** — `eval/runner/cat6-prose-scale.ts` (baseline-only) + `eval/runner/adversarial-injections.ts` (6 injection kinds)
- **Cat 8 Skill Compliance** — `eval/runner/cat8-skill-compliance.ts` (brain-first / back-link / citation-format / tier-escalation, deterministic from tool-bridge trace)
- **Cat 9 End-to-End Workflows** — `eval/runner/cat9-workflows.ts` (rubric-graded)
- **Cat 11 Multi-modal Ingestion** — `eval/runner/cat11-multimodal.ts` (PDF/audio/HTML)
**Threshold:** citation_recall > 90%, citation_precision > 85%, conflict_resolution > 70%.
Plus supporting infrastructure: agent adapter (Sonnet + 12 read + 3 dry_run tools),
structured-evidence Haiku judge contract, PublicPage/PublicQuery sealed qrels,
6-artifact flight-recorder, 6 portable JSON schemas for v1→v2 driver swap.
**Depends on:** Identity Resolution (Cat 3) shipped — uses same world generator pattern.
Scope pivot: originally planned for in-tree v1.1 delta; mid-PR pivoted to extract
the entire eval harness so gbrain users don't download the ~5MB corpus at install
time. BrainBench is now a public sibling benchmark; gbrain ships clean.
### BrainBench Cat 6: Auto-link Precision under Prose (at scale)
**What:** Cat 10 (Robustness/Adversarial) covered code-fence leak and false-positive substrings on 22 hand-crafted cases. v1.1 extends this to 500+ prose-heavy pages with realistic narrative noise. Tests link precision in the wild, not just edge cases.
### ~~v0.10.5: inferLinkType residuals (works_at, advises)~~
**Completed:** v0.20.0 (2026-04-23)
**Why deferred from PR #188:** Needs prose-heavy generated corpus (~$100-150 Opus). Existing 22-case eval already caught + fixed the code-fence leak bug.
`src/core/link-extraction.ts` — WORKS_AT_RE and ADVISES_RE expanded with
rank-prefixed engineer patterns ("senior/staff/principal/lead engineer at"),
discipline-prefixed ("backend/frontend/ML/security engineer at"), broader role
verbs ("manages engineering at", "running product at", "heads up X at"),
possessive time ("his/her/their time at"), role-noun forms ("tenure as",
"stint as", "role at"), advisory capacity phrasings, "as an advisor" forms,
and qualifier-specific advisors. New EMPLOYEE_ROLE_RE prior fires for
self-identified employees at the page level, biasing outbound company refs
toward works_at when per-edge verbs are absent. Precedence: investor > advisor
> employee. Existing tests in `test/link-extraction.test.ts` cover the new
patterns.
**Threshold:** link_precision > 95% on prose, type_accuracy > 80% on varied phrasing.
## P1 (BrainBench v1.1 — remaining categories)
### BrainBench Cat 8: Skill Behavior Compliance
**What:** Replays 100 inbound signals through a real LLM agent loop with gbrain skills loaded. Measures: brain-first lookup compliance, back-link iron-law adherence, citation format compliance, tier escalation correctness.
**Why deferred:** Needs real LLM API loop (~$2K total — most expensive single category).
**Threshold:** brain_first_compliance > 95%, back_link_compliance > 90%, citation_format > 95%.
### BrainBench Cat 9: End-to-End Workflows
**What:** 50 end-to-end scenarios across meeting ingestion, email-to-brain, daily-task-prep, briefing generation, sync cycle. Rubric-graded (10-15 criteria each).
**Why deferred:** Needs LLM agent loop (~$1K). Plus 50 hand-built rubrics.
**Threshold:** 80% scenario pass rate per workflow.
### BrainBench Cat 11: Multi-modal Ingestion
**What:** PDF/image/audio/video ingestion accuracy. 50 PDFs, 30 images, 20 audio files, 10 videos, 30 HTML pages. Per-modality recall and fidelity metrics.
**Why deferred:** Needs licensed real datasets (Common Voice for audio etc.). Dataset curation is the bulk of the work.
**Threshold:** PDF text fidelity > 95% (text-based) / > 80% (scanned), audio WER < 15%, entity_recall > 80% post-ingestion.
Cats 5/6/8/9/11 shipped to the sibling repo in v0.20.0 — see the Completed
section above. One remaining scope item:
### BrainBench Cat 1+2 at full scale
**What:** Existing benchmark-search-quality.ts (29 pages, 20 queries) and benchmark-graph-quality.ts (80 pages, 5 queries) currently pass at small scale. v1.1 extends both to 2-3K rich-prose pages generated via Opus to surface scale-dependent failures (tied keyword clusters, hub-node fan-out, prose-noise extraction precision).
@@ -68,24 +69,6 @@ company refs only). Per-type after fix: invested_in 91.7% (was 0%),
mentions 100%, attended 100%. works_at 58% and advises 41% are next
iteration's residuals.
### v0.10.5: inferLinkType residuals (works_at, advises)
**What:** After the v0.10.4 fix, two link types still under-perform on rich
prose. Drive these to >85% type accuracy in next iteration.
**works_at: 58% type accuracy.** Engineer/employee pages use varied phrasings
the regex doesn't catch ("spent some time at", "joined the team", narrative
"is currently at" without a verb). Approach: extend WORKS_AT_RE; consider
employee-role page prior similar to partner prior.
**advises: 41% type accuracy.** Advisor pages often describe board roles
without using the word "advisor" explicitly ("on Beta Health's board",
"joined Beta as a board member"). The v0.10.4 fix tightened ADVISES_RE to
require "advisor" rooting to avoid false positives from investors. Need
a tighter signal that distinguishes "advisor on board" from "investor on
board" — likely an advisor-role page prior plus verb-pattern combinations.
**Threshold:** Cat 2 rich-prose type accuracy > 92% (currently 88.5%).
### v0.10.4: gbrain alias resolution feature (driven by Cat 3)
**What:** Add an alias table to gbrain so "Sarah Chen" / "S. Chen" / "@schen" / "sarah.chen@example.com" resolve to one canonical entity. Schema: `aliases (id, slug, alias_text)` with a unique index. Search blends alias matches into hybrid scoring.