mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.37.2.0: takes_resolution_consistency CHECK accepts 'unresolvable' (#1211)
* v0.36.1.1 hotfix: takes_resolution_consistency CHECK accepts 'unresolvable'
Unblocks production grading scripts that write the judge's 4th verdict
type. Before this fix, every quality='unresolvable' INSERT/UPDATE hit
a CHECK violation — 0 of 34 writes landed in a recent prod run.
Migration v74 widens BOTH:
- takes_resolution_consistency (table-level CHECK) — admits the
('unresolvable', NULL) pair alongside the existing 4 legal shapes
- resolved_quality column-level CHECK — drops the auto-generated
name from v37, re-adds as takes_resolved_quality_values with the
4-state enum
Backward compatible. Existing rows with quality IN (NULL, 'correct',
'incorrect', 'partial') all satisfy the new CHECKs unchanged.
TakesScorecard gains sibling fields unresolvable_count + unresolvable_rate;
the existing `resolved` field deliberately keeps its 3-state meaning
so historical scorecards compare apples-to-apples (T1c sibling-field
design from the eng review).
Pinned by:
- test/takes-resolution.test.ts — R1-R5 round-trip
- test/migrate.test.ts — v74 structural assertions + PGLite E2E
suite exercising all valid + invalid (quality, outcome) shapes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e/schema-drift): reset public schema in beforeAll to isolate from caller bootstrap state
Previously the test trusted caller-provided DATABASE_URL to point at a fresh
database. CLAUDE.md's E2E lifecycle prescribes 'gbrain doctor --json' as the
bootstrap step (needed by oauth-related tests for table creation), but doctor
configures the gateway and bakes the configured embedding model into
content_chunks.model DEFAULT during the initial CREATE TABLE.
On re-run, CREATE TABLE IF NOT EXISTS is a no-op and the bootstrapped default
sticks. PGLite (always fresh-in-memory) gets the unconfigured-gateway fallback
'text-embedding-3-large'. The test reported phantom drift:
pg.default="'zembed-1'::text" pglite.default="'text-embedding-3-large'::text"
Fix: DROP SCHEMA public CASCADE + CREATE SCHEMA public before pg.initSchema.
Resets every table/index/sequence/constraint added by prior tooling. The PGLite
side is already fresh-per-test by construction.
Verified order-independent:
- Fresh DB → 6/0 pass
- After 'gbrain doctor' bootstrap → 6/0 pass
- Full E2E suite (mechanical + schema-drift) → 84/0 pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(safety): gate schema-drift DROP SCHEMA + relax TakesScorecard interface
Two findings from Codex adversarial review on the v0.37.0.1 hotfix:
1. **DROP SCHEMA safety gate (P0).** test/e2e/schema-drift.test.ts had an
unguarded DROP SCHEMA public CASCADE. A developer running the E2E with
DATABASE_URL pointing at a real brain or staging DB would lose the entire
public schema. The fix: triple-check before destruction.
- Parse the DATABASE_URL hostname + db name
- Allow reset only when: explicit GBRAIN_TEST_DB=1 OR (localhost host AND
test-shaped db name like gbrain_test, *_test, test_*, *_e2e)
- Refuse otherwise with a loud paste-ready warning
- The test still proceeds (the parity check is the fail-safe — if the
caller already had a fresh DB, parity passes; if not, parity fails
LOUDLY instead of nuking their data)
Verified all three branches: localhost+gbrain_test resets (6/0 pass);
localhost+production_brain refuses + warns (6/0 pass against pre-existing
schema); GBRAIN_TEST_DB=1 override on production_brain name allows reset.
2. **TakesScorecard interface compat.** Making `unresolvable_count` +
`unresolvable_rate` required fields on the public TakesScorecard
interface broke downstream SDK consumers who construct scorecard
fixtures (gbrain-evals, custom engines). The hotfix shouldn't impose
a compile-break on hotfix users.
Fix: make both fields optional (`?: number` / `?: number | null`).
`finalizeScorecard` still always populates them, so all internal code
sees the real values. External fixtures that omit them compile cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(safety,ux): tighten DROP SCHEMA gate + surface unresolvable in scorecard CLI
Second codex adversarial pass on v0.37.0.1 surfaced two residual findings.
**P0 — Safety gate still bypassable.** First-pass safety gate used
`explicitOptIn || (isLocalhost && looksLikeTestDb)` — meaning
`GBRAIN_TEST_DB=1` bypassed BOTH the host check AND the db-name check.
Someone running the E2E with that env set against a production DATABASE_URL
would still nuke their schema. Codex re-flagged it as P0.
Tightened logic: `looksLikeTestDb && (isLocalhost || ciOptIn)`. The db-name
pattern is now the hard floor — `gbrain_test`, `*_test`, `test_*`, `*_e2e`.
GBRAIN_TEST_DB=1 only relaxes the localhost requirement (for CI service-name
hosts). Setting the env on a DATABASE_URL pointing at `production_data` is
explicitly refused with a paste-ready message naming the failed check.
Verified 3 ways:
- gbrain_test + localhost → resets (6/0 pass)
- production_data + GBRAIN_TEST_DB=1 → REFUSES with clear message
- foo_e2e + GBRAIN_TEST_DB=1 → resets (test-shaped name passes)
**P2 — gbrain takes scorecard hides the unresolvable signal.** Early-return
on `resolved === 0` was triggered before the new sibling fields rendered.
A brain with only `quality='unresolvable'` verdicts — the spec's whole
production case — printed "No resolved bets yet" and exited. The
unresolvable_rate field was unreachable from the human CLI unless the user
knew to pass `--json`.
Fix: gate the early-return on `resolved === 0 AND unresolvable_count === 0`.
Render `unresolvable` count + `unresolvable_rate` alongside `partial_rate`
when present. Threshold warn at 30% (mirrors PARTIAL_RATE_WARNING_THRESHOLD)
pointing at retrieval coverage, not prediction accuracy — the actionable
read for high-unresolvable brains.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: renumber v0.37.0.1 → v0.37.2.0 (v0.37.1.0 claimed by other PRs)
PRs #1214 and #1215 both claim v0.37.1.0; bumping past to the next free
slot. Migration v79 renamed `takes_unresolvable_quality_v0_37_0_1` →
`takes_unresolvable_quality_v0_37_2_0`. VERSION + package.json +
CHANGELOG + llms bundles + inline doc references all swept.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
39e14cd50e
commit
9a4ae0962e
@@ -138,6 +138,8 @@ describe('pickFallbackSlots', () => {
|
||||
accuracy: 0.4,
|
||||
brier: 0.32,
|
||||
partial_rate: 0,
|
||||
unresolvable_count: 0,
|
||||
unresolvable_rate: null,
|
||||
};
|
||||
expect(__testing.pickFallbackSlots(scorecard).direction).toBe('over-confident');
|
||||
});
|
||||
@@ -152,6 +154,8 @@ describe('pickFallbackSlots', () => {
|
||||
accuracy: 0.8,
|
||||
brier: 0.12,
|
||||
partial_rate: 0,
|
||||
unresolvable_count: 0,
|
||||
unresolvable_rate: null,
|
||||
};
|
||||
expect(__testing.pickFallbackSlots(scorecard).direction).toBe('mostly right');
|
||||
});
|
||||
@@ -166,6 +170,8 @@ describe('pickFallbackSlots', () => {
|
||||
accuracy: null,
|
||||
brier: null,
|
||||
partial_rate: null,
|
||||
unresolvable_count: 0,
|
||||
unresolvable_rate: null,
|
||||
};
|
||||
const out = __testing.pickFallbackSlots(scorecard);
|
||||
expect(out.nRight).toBe(0);
|
||||
@@ -184,6 +190,8 @@ const ENOUGH_RESOLVED_SCORECARD: TakesScorecard = {
|
||||
accuracy: 0.636,
|
||||
brier: 0.21,
|
||||
partial_rate: 0.083,
|
||||
unresolvable_count: 0,
|
||||
unresolvable_rate: null,
|
||||
};
|
||||
|
||||
describe('runPhaseCalibrationProfile — phase integration', () => {
|
||||
|
||||
@@ -75,11 +75,51 @@ describe.skipIf(skip)('schema drift: PGLite ↔ Postgres post-initSchema parity
|
||||
await pglite.connect({});
|
||||
await pglite.initSchema();
|
||||
|
||||
// Postgres side: connect to the test database, run the canonical initSchema.
|
||||
// The test container at `bun run ci:local` provides a fresh DB; outside that
|
||||
// path we rely on the caller having set DATABASE_URL to a fresh DB.
|
||||
// Postgres side: ensure the test database is FRESH before initSchema.
|
||||
// v0.37.2.0 fix: previously the test trusted the caller to pass a fresh
|
||||
// DATABASE_URL, but `gbrain doctor` (used by the CLAUDE.md E2E bootstrap
|
||||
// ritual) populates `content_chunks.model DEFAULT` from the configured
|
||||
// gateway model. On a re-run, `CREATE TABLE IF NOT EXISTS` is a no-op so
|
||||
// the stale default sticks while PGLite (always fresh-in-memory) gets the
|
||||
// engine fallback. That produced a phantom drift unrelated to schema
|
||||
// parity.
|
||||
//
|
||||
// SAFETY GATE (codex P0, tightened in v0.37.2.0): DROP SCHEMA public CASCADE is
|
||||
// destructive. The db name MUST always look test-shaped — no env-var override
|
||||
// bypasses that floor. GBRAIN_TEST_DB=1 only relaxes the localhost requirement
|
||||
// so CI environments where the host is a service name (e.g. "postgres") can
|
||||
// still reset. If the db name doesn't match the test pattern, nothing nukes it.
|
||||
pg = new PostgresEngine();
|
||||
await pg.connect({ database_url: DATABASE_URL! });
|
||||
const pgConnPre = (pg as any).sql;
|
||||
|
||||
const url = new URL(DATABASE_URL!);
|
||||
const dbName = url.pathname.replace(/^\//, '');
|
||||
const host = url.hostname;
|
||||
const isLocalhost = host === 'localhost' || host === '127.0.0.1' || host.endsWith('.local');
|
||||
// db-name pattern is the floor: gbrain_test, *_test, test_*, *_e2e.
|
||||
// Required REGARDLESS of any override — a production db named "production_data"
|
||||
// cannot be reset even with GBRAIN_TEST_DB=1.
|
||||
const looksLikeTestDb = /^(gbrain_test|.*_test|test_.*|.*_e2e)$/i.test(dbName);
|
||||
const ciOptIn = process.env.GBRAIN_TEST_DB === '1';
|
||||
// resetAllowed semantics: db name is test-shaped AND (localhost OR ci-opt-in).
|
||||
// Neither host nor env-var alone is sufficient.
|
||||
const resetAllowed = looksLikeTestDb && (isLocalhost || ciOptIn);
|
||||
|
||||
if (resetAllowed) {
|
||||
await pgConnPre.unsafe('DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public;');
|
||||
} else {
|
||||
// Surface a loud, paste-ready hint. The test will still try initSchema;
|
||||
// if the caller already had a fresh DB the parity check passes anyway.
|
||||
const reason = !looksLikeTestDb
|
||||
? `db name "${dbName}" doesn't match the test pattern (gbrain_test, *_test, test_*, *_e2e). ` +
|
||||
`GBRAIN_TEST_DB=1 does NOT override this — db name is the hard floor.`
|
||||
: `host="${host}" is non-local AND GBRAIN_TEST_DB=1 is not set. ` +
|
||||
`Set GBRAIN_TEST_DB=1 to allow non-local hosts (e.g. CI service names) — ` +
|
||||
`but only when the db name is already test-shaped.`;
|
||||
console.warn(`[schema-drift] Skipping DROP SCHEMA — ${reason}`);
|
||||
}
|
||||
|
||||
await pg.initSchema();
|
||||
|
||||
// Snapshot both. PGLite returns `{rows}`, postgres.js returns the array.
|
||||
|
||||
@@ -1610,3 +1610,189 @@ describe('resolveSessionTimeouts — env var overrides', () => {
|
||||
expect(Object.keys(t)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── v0.37.2.0 — v80 takes_unresolvable_quality_v0_37_2_0 ──────────────────
|
||||
//
|
||||
// Hotfix that unblocks the production grading script. Widens BOTH:
|
||||
// (a) the table-level takes_resolution_consistency CHECK to accept
|
||||
// quality='unresolvable' AND outcome=NULL
|
||||
// (b) the column-level CHECK on resolved_quality to allow 'unresolvable'
|
||||
// Structural assertions only — round-trip behavior is covered by E2E.
|
||||
// Renumbered v74→v79→v80 during successive master merges (autonomous-
|
||||
// remediation wave claimed v68-v78, then v0.37.1.0 claimed v79).
|
||||
|
||||
describe('migrate v80 — takes_unresolvable_quality_v0_37_2_0', () => {
|
||||
const v80 = MIGRATIONS.find(m => m.version === 80);
|
||||
|
||||
test('v80 entry exists with the documented name', () => {
|
||||
expect(v80).toBeDefined();
|
||||
expect(v80!.name).toBe('takes_unresolvable_quality_v0_37_2_0');
|
||||
});
|
||||
|
||||
test('v80 is marked idempotent so re-runs are safe', () => {
|
||||
expect(v80!.idempotent).toBe(true);
|
||||
});
|
||||
|
||||
test("v80 widens the column-level CHECK to include 'unresolvable'", () => {
|
||||
const sql = (v80!.sql ?? '').toLowerCase();
|
||||
// Drops both possible names (auto-generated + explicitly-named) so
|
||||
// pre-v80 brains converge regardless of which CHECK shape they had.
|
||||
expect(sql).toContain('drop constraint if exists takes_resolved_quality_check');
|
||||
expect(sql).toContain('drop constraint if exists takes_resolved_quality_values');
|
||||
// The new CHECK enumerates all four valid quality states.
|
||||
expect(sql).toContain('takes_resolved_quality_values');
|
||||
expect(sql).toMatch(/'correct'/);
|
||||
expect(sql).toMatch(/'incorrect'/);
|
||||
expect(sql).toMatch(/'partial'/);
|
||||
expect(sql).toMatch(/'unresolvable'/);
|
||||
});
|
||||
|
||||
test('v80 widens the table-level takes_resolution_consistency CHECK', () => {
|
||||
const sql = v80!.sql ?? '';
|
||||
expect(sql).toContain('DROP CONSTRAINT IF EXISTS takes_resolution_consistency');
|
||||
expect(sql).toContain('ADD CONSTRAINT takes_resolution_consistency CHECK');
|
||||
// The new (quality, outcome) row for unresolvable joins partial as
|
||||
// null-outcome. Pin the literal pair so regressions surface.
|
||||
expect(sql).toMatch(/resolved_quality\s*=\s*'unresolvable'\s+AND\s+resolved_outcome\s+IS\s+NULL/i);
|
||||
});
|
||||
|
||||
test('v80 keeps the existing four (quality, outcome) pairs intact', () => {
|
||||
const sql = v80!.sql ?? '';
|
||||
// Regression: shouldn't accidentally drop pre-existing legal states.
|
||||
expect(sql).toMatch(/resolved_quality\s+IS\s+NULL\s+AND\s+resolved_outcome\s+IS\s+NULL/i);
|
||||
expect(sql).toMatch(/resolved_quality\s*=\s*'correct'\s+AND\s+resolved_outcome\s*=\s*true/i);
|
||||
expect(sql).toMatch(/resolved_quality\s*=\s*'incorrect'\s+AND\s+resolved_outcome\s*=\s*false/i);
|
||||
expect(sql).toMatch(/resolved_quality\s*=\s*'partial'\s+AND\s+resolved_outcome\s+IS\s+NULL/i);
|
||||
});
|
||||
});
|
||||
|
||||
// E2E round-trip — runs against PGLite (no DATABASE_URL needed). Spins up a
|
||||
// fresh in-memory brain, applies all migrations through v80, then exercises
|
||||
// the regression checklist: R1 unresolvable persists, R2 pre-v80 (NULL,NULL)
|
||||
// rows survive, R3+R4 contradictory pairs still rejected, R5 the four legal
|
||||
// shapes all round-trip.
|
||||
describe('migrate v80 — CHECK widening end-to-end on PGLite', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function insertTake(rowNum: number): Promise<number> {
|
||||
// Need a page row to satisfy the FK before we can write a take.
|
||||
const slug = `wiki/people/v80-test-${rowNum}-${Math.random().toString(36).slice(2, 8)}`;
|
||||
const page = await engine.putPage(slug, {
|
||||
type: 'person',
|
||||
title: `v80 test row ${rowNum}`,
|
||||
compiled_truth: '',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
content_hash: `v80-${rowNum}-${Math.random()}`,
|
||||
});
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO takes (page_id, row_num, claim, kind, holder, weight, active)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)`,
|
||||
[page.id, rowNum, `claim ${rowNum}`, 'bet', 'garry', 0.5, true],
|
||||
);
|
||||
return page.id;
|
||||
}
|
||||
|
||||
async function tryResolve(
|
||||
pageId: number,
|
||||
rowNum: number,
|
||||
quality: string,
|
||||
outcome: boolean | null,
|
||||
): Promise<{ ok: true } | { ok: false; err: string }> {
|
||||
try {
|
||||
await engine.executeRaw(
|
||||
`UPDATE takes
|
||||
SET resolved_at = now(),
|
||||
resolved_quality = $1::text,
|
||||
resolved_outcome = $2,
|
||||
resolved_by = $3
|
||||
WHERE page_id = $4 AND row_num = $5`,
|
||||
[quality, outcome, 'gbrain:test', pageId, rowNum],
|
||||
);
|
||||
return { ok: true };
|
||||
} catch (err) {
|
||||
return { ok: false, err: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
test('R1: writing quality=unresolvable + outcome=NULL succeeds post-v80', async () => {
|
||||
const pageId = await insertTake(101);
|
||||
const result = await tryResolve(pageId, 101, 'unresolvable', null);
|
||||
expect(result.ok).toBe(true);
|
||||
|
||||
const row = (await engine.executeRaw<{ resolved_quality: string; resolved_outcome: boolean | null }>(
|
||||
`SELECT resolved_quality, resolved_outcome FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[pageId, 101],
|
||||
))[0];
|
||||
expect(row.resolved_quality).toBe('unresolvable');
|
||||
expect(row.resolved_outcome).toBeNull();
|
||||
});
|
||||
|
||||
test('R2: pre-v80 row with quality=NULL AND outcome=NULL survives widened CHECK', async () => {
|
||||
// After v80 ran on initSchema, inserting an unresolved take must still
|
||||
// succeed — the (NULL, NULL) case is still legal.
|
||||
const pageId = await insertTake(102);
|
||||
const row = (await engine.executeRaw<{ resolved_quality: string | null; resolved_outcome: boolean | null }>(
|
||||
`SELECT resolved_quality, resolved_outcome FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[pageId, 102],
|
||||
))[0];
|
||||
expect(row.resolved_quality).toBeNull();
|
||||
expect(row.resolved_outcome).toBeNull();
|
||||
});
|
||||
|
||||
test('R3 negative: quality=partial AND outcome=true STILL rejected', async () => {
|
||||
const pageId = await insertTake(103);
|
||||
const result = await tryResolve(pageId, 103, 'partial', true);
|
||||
expect(result.ok).toBe(false);
|
||||
expect((result as { ok: false; err: string }).err.toLowerCase()).toContain('check');
|
||||
});
|
||||
|
||||
test('R4 negative: quality=unresolvable AND outcome=true STILL rejected', async () => {
|
||||
// The widened CHECK admits unresolvable only with NULL outcome — same
|
||||
// shape as partial. A truthy outcome remains illegal.
|
||||
const pageId = await insertTake(104);
|
||||
const result = await tryResolve(pageId, 104, 'unresolvable', true);
|
||||
expect(result.ok).toBe(false);
|
||||
expect((result as { ok: false; err: string }).err.toLowerCase()).toContain('check');
|
||||
});
|
||||
|
||||
test('R4 negative: quality=unresolvable AND outcome=false STILL rejected', async () => {
|
||||
const pageId = await insertTake(105);
|
||||
const result = await tryResolve(pageId, 105, 'unresolvable', false);
|
||||
expect(result.ok).toBe(false);
|
||||
expect((result as { ok: false; err: string }).err.toLowerCase()).toContain('check');
|
||||
});
|
||||
|
||||
test('R5: getScorecard surfaces unresolvable_count + unresolvable_rate as siblings', async () => {
|
||||
// Seed three rows: one correct, one partial, one unresolvable. All under
|
||||
// the same holder so the scorecard groups them. unresolvable_rate must
|
||||
// come out as 1 / (2 + 1) since `resolved` stays 3-state (correct+partial).
|
||||
const pid1 = await insertTake(201);
|
||||
const pid2 = await insertTake(202);
|
||||
const pid3 = await insertTake(203);
|
||||
expect((await tryResolve(pid1, 201, 'correct', true)).ok).toBe(true);
|
||||
expect((await tryResolve(pid2, 202, 'partial', null)).ok).toBe(true);
|
||||
expect((await tryResolve(pid3, 203, 'unresolvable', null)).ok).toBe(true);
|
||||
|
||||
const scorecard = await engine.getScorecard({ holder: 'garry' }, undefined);
|
||||
// Three rows total, 2 in the 3-state subset, 1 unresolvable.
|
||||
expect(scorecard.resolved).toBeGreaterThanOrEqual(2);
|
||||
expect(scorecard.unresolvable_count).toBeGreaterThanOrEqual(1);
|
||||
// unresolvable_rate is computed against the 4-state denominator.
|
||||
expect(scorecard.unresolvable_rate).not.toBeNull();
|
||||
expect(scorecard.unresolvable_rate!).toBeGreaterThan(0);
|
||||
// The legacy fields keep their pre-v80 meaning.
|
||||
expect(typeof scorecard.accuracy).not.toBe('undefined');
|
||||
expect(typeof scorecard.partial_rate).not.toBe('undefined');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -32,6 +32,8 @@ function buildScorecard(opts: { resolved: number; brier: number | null }): Takes
|
||||
accuracy: 0.6,
|
||||
brier: opts.brier,
|
||||
partial_rate: 0,
|
||||
unresolvable_count: 0,
|
||||
unresolvable_rate: null,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -70,6 +70,26 @@ describe('deriveResolutionTuple', () => {
|
||||
deriveResolutionTuple({ resolvedBy: 'garry' })
|
||||
).toThrow(GBrainError);
|
||||
});
|
||||
|
||||
// v0.36.1.1 R1-class: 'unresolvable' joins partial as null-outcome.
|
||||
test('R1: quality=unresolvable → (unresolvable, null)', () => {
|
||||
expect(deriveResolutionTuple({ quality: 'unresolvable', resolvedBy: 'garry' })).toEqual({
|
||||
quality: 'unresolvable',
|
||||
outcome: null,
|
||||
});
|
||||
});
|
||||
|
||||
test('R4-helper: unresolvable + outcome=true is contradictory (throws)', () => {
|
||||
// Defense-in-depth: deriveResolutionTuple rejects contradictory inputs
|
||||
// before the schema CHECK fires. unresolvable requires null outcome —
|
||||
// same shape as partial.
|
||||
expect(() =>
|
||||
deriveResolutionTuple({ quality: 'unresolvable', outcome: true, resolvedBy: 'garry' })
|
||||
).toThrow(GBrainError);
|
||||
expect(() =>
|
||||
deriveResolutionTuple({ quality: 'unresolvable', outcome: false, resolvedBy: 'garry' })
|
||||
).toThrow(GBrainError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('finalizeScorecard (Brier math)', () => {
|
||||
@@ -86,6 +106,8 @@ describe('finalizeScorecard (Brier math)', () => {
|
||||
accuracy: null,
|
||||
brier: null,
|
||||
partial_rate: null,
|
||||
unresolvable_count: 0,
|
||||
unresolvable_rate: null,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -155,4 +177,65 @@ describe('finalizeScorecard (Brier math)', () => {
|
||||
});
|
||||
expect(card.partial_rate).toBe(0);
|
||||
});
|
||||
|
||||
// v0.36.1.1 R5 — sibling-field semantics. `resolved` MUST keep the v0.36.1.0
|
||||
// 3-state meaning so historical comparisons stay valid; unresolvable rows
|
||||
// live in the new sibling fields only.
|
||||
test('R5: unresolvable_count + unresolvable_rate populated; resolved/accuracy/partial_rate unchanged', () => {
|
||||
// 2 correct, 1 incorrect, 1 partial, 1 unresolvable. resolved=4 (3-state),
|
||||
// unresolvable_count=1. unresolvable_rate = 1 / (4 + 1) = 0.2.
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 5,
|
||||
resolved: 4,
|
||||
correct: 2,
|
||||
incorrect: 1,
|
||||
partial: 1,
|
||||
brier: 0.2,
|
||||
unresolvable_count: 1,
|
||||
});
|
||||
expect(card.resolved).toBe(4); // 3-state, NOT 5
|
||||
expect(card.unresolvable_count).toBe(1);
|
||||
expect(card.unresolvable_rate).toBeCloseTo(0.2, 5);
|
||||
// accuracy unchanged: correct / (correct + incorrect) = 2/3
|
||||
expect(card.accuracy).toBeCloseTo(2 / 3, 5);
|
||||
// partial_rate unchanged: partial / resolved (3-state denominator) = 1/4
|
||||
expect(card.partial_rate).toBe(0.25);
|
||||
// brier unchanged
|
||||
expect(card.brier).toBe(0.2);
|
||||
});
|
||||
|
||||
test('R5: legacy v0.36.1.0 ScorecardRowRaw (no unresolvable_count) backfills count=0; rate=0 when resolved>0', () => {
|
||||
// Engines pre-v0.36.1.1 won't return unresolvable_count in the row. The
|
||||
// optional `?` on ScorecardRowRaw + default-0 in finalizeScorecard means
|
||||
// existing brains keep working. unresolvable_rate matches partial_rate
|
||||
// semantics: 0 when there are resolved rows but no unresolvable ones.
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 3, resolved: 3, correct: 2, incorrect: 1, partial: 0, brier: 0.18,
|
||||
});
|
||||
expect(card.unresolvable_count).toBe(0);
|
||||
expect(card.unresolvable_rate).toBe(0);
|
||||
});
|
||||
|
||||
test('R5: empty brain (resolved=0 AND unresolvable_count=0) → unresolvable_rate=null', () => {
|
||||
// Only when BOTH counts are zero do we return null (avoid divide-by-zero
|
||||
// and signal "no data" rather than "0% unresolvable on no data").
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 0, resolved: 0, correct: 0, incorrect: 0, partial: 0, brier: null,
|
||||
});
|
||||
expect(card.unresolvable_count).toBe(0);
|
||||
expect(card.unresolvable_rate).toBeNull();
|
||||
});
|
||||
|
||||
test('R5: brain with only unresolvable rows surfaces unresolvable_rate=1.0', () => {
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 5, resolved: 0, correct: 0, incorrect: 0, partial: 0, brier: null,
|
||||
unresolvable_count: 5,
|
||||
});
|
||||
expect(card.resolved).toBe(0);
|
||||
expect(card.unresolvable_count).toBe(5);
|
||||
expect(card.unresolvable_rate).toBe(1.0);
|
||||
expect(card.accuracy).toBeNull();
|
||||
expect(card.brier).toBeNull();
|
||||
expect(card.partial_rate).toBeNull(); // resolved is 0 → null per existing contract
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user