mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
Garry TanandClaude Opus 4.6
e1e699f0a6
Codex fixes #1, #2, #3 from the plan's outside-voice review. Enforcement shifts from SOFT-VIA-TYPE-COMMENT to SOFT-VIA-SANITIZED-OBJECT. Hard enforcement via process isolation waits for BrainBench v2 Docker sandbox. **eval/runner/types.ts** additions: - `PublicPage = Pick<Page, 'slug' | 'type' | 'title' | 'compiled_truth' | 'timeline'>` — the exact 5 fields adapters should see. No _facts. No frontmatter (a known hiding spot for accidental gold leaks). - `sanitizePage(p: Page): PublicPage` — returns a NEW object with the 5 fields only. Cannot be bypassed by `(page as any)._facts` because the field does not exist on the sanitized object. - `PublicQuery = Omit<Query, 'gold'>` — strips the gold field. - `sanitizeQuery(q: Query): PublicQuery` — enumerates public fields explicitly (not spread+delete) so no prototype weirdness leaves gold reachable. **eval/runner/multi-adapter.ts** — scoreOneRun now calls sanitizePage / sanitizeQuery before passing to adapter.init / adapter.query. The scorer retains the full Query shape (including gold.relevant) for precision / recall computation. Adapter signatures unchanged — the sealing is at the OBJECT level, not the type level. This keeps existing adapters (ripgrep-bm25, vector-only, hybrid-nograph, gbrain-after) binary-compatible. Verified: no existing adapter reads q.gold or page._facts, so the change is safe without further adapter updates. **test/eval/sealed-qrels.test.ts** (17 tests): - sanitizePage strips _facts + frontmatter + arbitrary hidden keys - Output has exactly the 5 public keys (deep introspection) - Proxy tripwire simulates a malicious adapter: any access to _facts or gold throws `sealed-qrels violation` - sanitizeQuery retains optional fields (as_of_date, tags, author, acceptable_variants, known_failure_modes) but omits undefined ones - Honest documentation of the seal's limits: filesystem bypass and Proxy attacks would still work in v1; Docker isolation (v2) is the real enforcement Every existing eval test still passes (273 before + 17 sealed-qrels = 290). Total eval suite now: 290 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>