mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
Garry TanandClaude Opus 4.7
e2a5dc4a2b
Closes multiple Phase 2 items in one commit since they form a cohesive
package: query schema enforcement + new query tiers + per-query-set
statistical rigor.
Added:
eval/runner/queries/validator.ts — hand-rolled Query schema validator
eval/runner/queries/validator.test.ts — 24 unit tests, all pass
eval/runner/queries/tier5-fuzzy.ts — 30 hand-authored Tier 5 Fuzzy/Vibe queries
eval/runner/queries/tier5_5-synthetic.ts — 50 SYNTHETIC-labeled outsider-style queries (author: "synthetic-outsider-v1")
eval/runner/queries/index.ts — aggregator + validateAll()
Modified:
eval/runner/multi-adapter.ts — N=5 runs per adapter (BRAINBENCH_N override), page-order shuffle, mean±stddev reporting
Query validator (hand-rolled, no zod dep to match gbrain codebase style):
- Temporal verb regex enforces as_of_date (per eng pass 2 spec):
/\\b(is|was|were|current|now|at the time|during|as of|when did)\\b/i
- Validates tier enum, expected_output_type enum, gold shape per type
- gold.relevant must be non-empty slug[] for cited-source-pages queries
- abstention requires gold.expected_abstention === true
- externally-authored tier requires author field
- batch validation catches duplicate IDs
Tier 5 Fuzzy/Vibe (30 queries, hand-authored):
- Vague recall: "Someone who was a senior engineer at a biotech company..."
- Trait-based: "The engineer who pushed back on microservices"
- Cultural/epithet: "Who is known as a 'systems builder' in security?"
- Abstention bait: "Which Layer 1 project did the crypto guy leave?" (prose
mentions but never names; good systems abstain)
- Addresses Codex's circularity critique — vague queries where graph-heavy
systems shouldn't inherently win.
Tier 5.5 Synthetic Outsider (50 queries, AI-authored placeholder):
- Clearly labeled author: "synthetic-outsider-v1"
- Phrasing variety not in the 4 template families:
* fragment style ("crypto founder Goldman Sachs background")
* polite/natural ("Can you pull up what we have on...")
* comparison ("What is the difference between X and Y?")
* follow-up ("And who else advises Orbit Labs?")
* typos/misspellings ("adam lopez bioinformatcis")
* similarity ("Find me someone like Alice Davis...")
* imperative ("Pull up Alice Davis")
- Real Tier 5.5 from outside researchers supersedes synthetic via
PRs to eval/external-authors/ (docs ship in follow-up commit).
N=5 tolerance bands:
- Default N=5, override via BRAINBENCH_N env var (e.g. BRAINBENCH_N=1 for dev loops)
- Per-run seeded Fisher-Yates shuffle of page ingest order (LCG seed = run_idx+1)
- Surfaces order-dependent adapter bugs (tie-break-by-first-seen etc.)
- Reports mean ± sample-stddev per metric
- "stddev = 0" is honest signal that the adapter is deterministic, not a bug.
LLM-judge metrics (future) will naturally produce non-zero stddev.
Validation: all 80 Tier 5 + 5.5 queries pass validateAll(). 24 validator
unit tests pass.
Next commit: world.html contributor explorer (Phase 3).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
39 lines
1.4 KiB
TypeScript
39 lines
1.4 KiB
TypeScript
/**
|
|
* Aggregates all tier-5/5.5 query sets and exposes validator helpers.
|
|
*
|
|
* Usage:
|
|
* import { getAllTierQueries, validateAll } from './queries';
|
|
* const queries = getAllTierQueries();
|
|
* const result = validateAll(queries);
|
|
*/
|
|
|
|
import type { Query } from '../types.ts';
|
|
import { getTier5FuzzyQueries } from './tier5-fuzzy.ts';
|
|
import { getTier5_5SyntheticQueries } from './tier5_5-synthetic.ts';
|
|
import { validateQuerySet, formatIssues } from './validator.ts';
|
|
|
|
/** Tier 5 Fuzzy/Vibe (hand-authored by gstack maintainers). */
|
|
export { getTier5FuzzyQueries } from './tier5-fuzzy.ts';
|
|
|
|
/** Tier 5.5 externally-authored (SYNTHETIC placeholder; see CONTRIBUTING.md). */
|
|
export { getTier5_5SyntheticQueries } from './tier5_5-synthetic.ts';
|
|
|
|
export { validateQuery, validateQuerySet, formatIssues, TEMPORAL_VERBS } from './validator.ts';
|
|
export type { ValidationIssue, ValidationResult } from './validator.ts';
|
|
|
|
/** All Tier 5 + 5.5 queries concatenated. */
|
|
export function getAllTierQueries(): Query[] {
|
|
return [...getTier5FuzzyQueries(), ...getTier5_5SyntheticQueries()];
|
|
}
|
|
|
|
/** Validate the complete Tier-5 + 5.5 set (used by CI + eval:query:validate). */
|
|
export function validateAll(): { ok: boolean; count: number; report: string } {
|
|
const queries = getAllTierQueries();
|
|
const result = validateQuerySet(queries);
|
|
return {
|
|
ok: result.ok,
|
|
count: queries.length,
|
|
report: formatIssues(result),
|
|
};
|
|
}
|