diff --git a/.github/PULL_REQUEST_TEMPLATE/tier5-queries.md b/.github/PULL_REQUEST_TEMPLATE/tier5-queries.md new file mode 100644 index 000000000..c75043597 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/tier5-queries.md @@ -0,0 +1,39 @@ + + +## Summary + +Submitting **N** Tier 5.5 queries for BrainBench. + +- Author handle: `@your-handle` +- File location: `eval/external-authors/your-handle/queries.json` +- Queries authored fresh (not copy-pasted from a model output) +- Slugs verified against `eval/data/world-v1/` (via `bun run eval:world:view`) + +## Checklist + +- [ ] `bun run eval:query:validate eval/external-authors/your-handle/queries.json` passes +- [ ] At least 20 queries +- [ ] Each query has either `gold.relevant` (with real slugs) or `gold.expected_abstention: true` +- [ ] Temporal queries have `as_of_date` set (`corpus-end` | `per-source` | ISO-8601) +- [ ] Phrasing is varied (not all the same template) +- [ ] `author` field matches my handle + +## Phrasing variety (optional self-audit) + +Tick the styles represented in your batch: + +- [ ] Full sentence questions +- [ ] Fragment-style ("crypto founder Goldman Sachs background") +- [ ] Comparison ("X vs Y") +- [ ] Follow-up ("And who else...") +- [ ] Imperative ("Pull up Alice Davis") +- [ ] Trait-based ("the demanding engineering leader") +- [ ] Abstention bait (answer is "not in corpus") + +## Notes to reviewer + +Anything worth flagging — ambiguous cases, corpus gaps you found, specific +phrasings you were uncertain about. diff --git a/.gitignore b/.gitignore index 229382f0c..fc91e5fa3 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,7 @@ supabase/.temp/ .claude/skills/ .idea eval/reports/ +eval/data/world-v1/world.html + +# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life) +eval/data/amara-life-v1/_cache/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 96359198a..559c33f45 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,67 @@ All notable changes to GBrain will be documented in this file. +## [0.20.0] - 2026-04-23 + +## **BrainBench moves out. gbrain gets its install surface back.** +## **The eval harness + 5MB fictional corpus now live in a sibling repo; gbrain exposes a clean public API they consume.** + +BrainBench is gbrain's benchmark harness. 10/12 Cats, 4-adapter scorecard, 418-item fictional corpus, 314 tests. Previously it lived inside this repo. Every `bun install` pulled down the eval tree, `docs/benchmarks/*.md` reports, `pdf-parse` devDep, and auxiliary test fixtures whether or not you ever ran a benchmark. For the 99% of gbrain users who want a knowledge-brain CLI, that's ~5MB of noise. + +v0.20 moves BrainBench to [github.com/garrytan/gbrain-evals](https://github.com/garrytan/gbrain-evals). gbrain stays the knowledge-brain CLI + library. `gbrain-evals` depends on gbrain via GitHub URL and consumes it through the public exports map. Same benchmarks, same scorecards, same Cat runners, same 418-item fictional amara-life corpus, just a separate install. Folks who don't care about evals never download them. Folks who do clone one extra repo. + +The clean separation also gives gbrain a first real public API surface. `package.json` adds 11 new subpath exports (`gbrain/engine`, `gbrain/pglite-engine`, `gbrain/search/hybrid`, `gbrain/link-extraction`, `gbrain/extract`, and so on) covering every gbrain internal the eval harness reaches into. Third-party tools (not just BrainBench) now have a stable contract to consume. Removing any of these exports is a breaking change going forward. + +### What moved where + +| Stays in gbrain | Moves to gbrain-evals | +|-----------------|----------------------| +| `src/` (CLI, MCP, engines, operations, skills runtime) | `eval/` (runners, adapters, generators, schemas, gold, cli) | +| `Page.type` enum including `email/slack/calendar-event/note/meeting` (useful for any ingested format, not just evals) | `test/eval/` (314 tests across 14 files) | +| `inferType()` heuristics for the new directory patterns | `docs/benchmarks/*.md` (all scorecards + regression reports) | +| Public exports map (11 new subpaths gbrain-evals consumes) | `pdf-parse` devDep (only eval/runner/loaders/pdf.ts used it) | +| `src/core/` test suite (1696 tests) | `eval:*` scripts (run from gbrain-evals now) | + +### What this means for you + +If you install gbrain via `git clone + bun install` or via npm/clawhub, you get a smaller, cleaner checkout. No eval corpus. No benchmark reports. No pdf-parse. `bun test` runs only gbrain's own test suite, not eval tests. + +If you want to run BrainBench: `git clone https://github.com/garrytan/gbrain-evals && cd gbrain-evals && bun install && bun run eval:run`. gbrain-evals fetches gbrain from GitHub via `"gbrain": "github:garrytan/gbrain#master"` so you always benchmark against the latest source. + +If you're a third-party library author importing gbrain internals: the new exports map is now your stable contract. Pin `gbrain/` imports against a version, not a file path. + +### Itemized changes + +**Extracted to [gbrain-evals](https://github.com/garrytan/gbrain-evals):** +- `eval/` ... schemas, runners, adapters, generators, queries, CLI tools, docs (CONTRIBUTING, RUNBOOK, CREDITS). +- `test/eval/` ... 14 test files, 314 tests covering schemas, sealed qrels, tool-bridge, agent adapter, judge, recorder, Cat 5/6/8/9/11, amara-life skeleton, adversarial-injections, pdf loader. +- `docs/benchmarks/` ... all scorecards and regression reports (4-adapter, v0.11 vs v0.12, Minions production/lab, tweet ingestion, knowledge runtime v0.13, BrainBench v1). +- `pdf-parse` devDep ... only consumed by `eval/runner/loaders/pdf.ts`. +- `eval:*` package.json scripts ... now live in gbrain-evals's `package.json` and run from there. + +**Kept in gbrain (useful beyond evals):** +- `Page.type` enum extensions in `src/core/types.ts`: `email | slack | calendar-event | note | meeting`. Any user ingesting an inbox dump, Slack export, iCal file, or meeting transcript benefits from first-class types. +- `inferType()` heuristics in `src/core/markdown.ts` for `/emails/`, `/slack/`, `/cal/`, `/notes/`, `/meetings/` directory patterns. +- 11 new public `exports` in `package.json`: `./pglite-engine`, `./link-extraction`, `./import-file`, `./transcription`, `./embedding`, `./config`, `./markdown`, `./backoff`, `./search/hybrid`, `./search/expansion`, `./extract`. These form gbrain's public-API contract for downstream consumers. + +**Docs synced:** +- `README.md` ... benchmark references now point at the gbrain-evals repo. +- `CLAUDE.md` ... BrainBench section replaced with a pointer to gbrain-evals + the list of public exports that consumers depend on. +- `src/commands/migrations/v0_12_0.ts` ... migration banner text references `github.com/garrytan/gbrain-evals` instead of a local `docs/benchmarks/*.md` path that no longer resolves. + +**Tests:** 1717 gbrain tests pass, 0 failures, 174 skipped (E2E requiring `DATABASE_URL`). The full eval suite (314 tests) moves with `gbrain-evals` and runs from there. + +### To take advantage of v0.20 + +For gbrain users: +1. `gbrain upgrade` ... no action required. The extraction is transparent. +2. If you previously ran `bun run eval:*` scripts from this repo: those scripts no longer exist here. `git clone https://github.com/garrytan/gbrain-evals && bun install` to get them. + +For gbrain-evals consumers: +1. Clone the sibling repo: `git clone https://github.com/garrytan/gbrain-evals` +2. `bun install && bun run eval:run` +3. Follow `gbrain-evals/eval/RUNBOOK.md` for full category runs and scorecard reproduction. + ## [0.19.1] - 2026-04-24 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 830f2af77..8e6bffe26 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,7 +44,14 @@ strict behavior when unset. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. - `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic. -- `src/commands/check-resolvable.ts` — Standalone CLI wrapper (v0.16.4) over `checkResolvable()`. Exports `parseFlags`, `resolveSkillsDir`, `DEFERRED`, `runCheckResolvable`. Exit rule: **1 on any issue (warnings OR errors)**, stricter than doctor's `ok` flag — honors README:259. Stable JSON envelope `{ok, skillsDir, report, autoFix, deferred, error, message}` — same shape on success and error paths. `--fix` path runs `autoFixDryViolations` BEFORE `checkResolvable` (same ordering as doctor). `deferred[]` array surfaces pending Checks 5 (trigger routing eval) and 6 (brain filing) with issue URLs. `scripts/skillify-check.ts` subprocess-calls `gbrain check-resolvable --json` (cached per process) and fails loud on binary-missing — no silent false-pass. +- `src/commands/check-resolvable.ts` — Standalone CLI wrapper (v0.16.4) over `checkResolvable()`. Exports `parseFlags`, `resolveSkillsDir`, `DEFERRED`, `runCheckResolvable`. Exit rule: **1 on any issue (warnings OR errors)**, stricter than doctor's `ok` flag — honors README:259. Stable JSON envelope `{ok, skillsDir, report, autoFix, deferred, error, message}` — same shape on success and error paths. `--fix` path runs `autoFixDryViolations` BEFORE `checkResolvable` (same ordering as doctor). `scripts/skillify-check.ts` subprocess-calls `gbrain check-resolvable --json` (cached per process) and fails loud on binary-missing — no silent false-pass. **v0.19:** AGENTS.md workspaces now resolve natively (see `src/core/resolver-filenames.ts`) — gbrain inspects the 107-skill OpenClaw deployment whether the routing file is `RESOLVER.md` or `AGENTS.md`. `DEFERRED[]` is empty — Checks 5 + 6 shipped as real code, not issue URLs. +- `src/core/resolver-filenames.ts` (v0.19) — central list of accepted routing filenames (`RESOLVER.md`, `AGENTS.md`). Shared by `findRepoRoot`, `check-resolvable`, and skillpack install so every code path walks the same fallback chain. +- `src/commands/skillify.ts` + `src/core/skillify/{generator,templates}.ts` (v0.19) — `gbrain skillify scaffold ` creates all stubs for a new skill in one command: SKILL.md, script, tests, routing-eval.jsonl, resolver entry, filing-rules pointer. `gbrain skillify check