Files
gbrain/src/core/takes-resolution.ts
T
9a4ae0962e v0.37.2.0: takes_resolution_consistency CHECK accepts 'unresolvable' (#1211)
* v0.36.1.1 hotfix: takes_resolution_consistency CHECK accepts 'unresolvable'

Unblocks production grading scripts that write the judge's 4th verdict
type. Before this fix, every quality='unresolvable' INSERT/UPDATE hit
a CHECK violation — 0 of 34 writes landed in a recent prod run.

Migration v74 widens BOTH:
  - takes_resolution_consistency (table-level CHECK) — admits the
    ('unresolvable', NULL) pair alongside the existing 4 legal shapes
  - resolved_quality column-level CHECK — drops the auto-generated
    name from v37, re-adds as takes_resolved_quality_values with the
    4-state enum

Backward compatible. Existing rows with quality IN (NULL, 'correct',
'incorrect', 'partial') all satisfy the new CHECKs unchanged.

TakesScorecard gains sibling fields unresolvable_count + unresolvable_rate;
the existing `resolved` field deliberately keeps its 3-state meaning
so historical scorecards compare apples-to-apples (T1c sibling-field
design from the eng review).

Pinned by:
  - test/takes-resolution.test.ts — R1-R5 round-trip
  - test/migrate.test.ts — v74 structural assertions + PGLite E2E
    suite exercising all valid + invalid (quality, outcome) shapes

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e/schema-drift): reset public schema in beforeAll to isolate from caller bootstrap state

Previously the test trusted caller-provided DATABASE_URL to point at a fresh
database. CLAUDE.md's E2E lifecycle prescribes 'gbrain doctor --json' as the
bootstrap step (needed by oauth-related tests for table creation), but doctor
configures the gateway and bakes the configured embedding model into
content_chunks.model DEFAULT during the initial CREATE TABLE.

On re-run, CREATE TABLE IF NOT EXISTS is a no-op and the bootstrapped default
sticks. PGLite (always fresh-in-memory) gets the unconfigured-gateway fallback
'text-embedding-3-large'. The test reported phantom drift:
  pg.default="'zembed-1'::text"  pglite.default="'text-embedding-3-large'::text"

Fix: DROP SCHEMA public CASCADE + CREATE SCHEMA public before pg.initSchema.
Resets every table/index/sequence/constraint added by prior tooling. The PGLite
side is already fresh-per-test by construction.

Verified order-independent:
  - Fresh DB → 6/0 pass
  - After 'gbrain doctor' bootstrap → 6/0 pass
  - Full E2E suite (mechanical + schema-drift) → 84/0 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(safety): gate schema-drift DROP SCHEMA + relax TakesScorecard interface

Two findings from Codex adversarial review on the v0.37.0.1 hotfix:

1. **DROP SCHEMA safety gate (P0).** test/e2e/schema-drift.test.ts had an
   unguarded DROP SCHEMA public CASCADE. A developer running the E2E with
   DATABASE_URL pointing at a real brain or staging DB would lose the entire
   public schema. The fix: triple-check before destruction.
   - Parse the DATABASE_URL hostname + db name
   - Allow reset only when: explicit GBRAIN_TEST_DB=1 OR (localhost host AND
     test-shaped db name like gbrain_test, *_test, test_*, *_e2e)
   - Refuse otherwise with a loud paste-ready warning
   - The test still proceeds (the parity check is the fail-safe — if the
     caller already had a fresh DB, parity passes; if not, parity fails
     LOUDLY instead of nuking their data)

   Verified all three branches: localhost+gbrain_test resets (6/0 pass);
   localhost+production_brain refuses + warns (6/0 pass against pre-existing
   schema); GBRAIN_TEST_DB=1 override on production_brain name allows reset.

2. **TakesScorecard interface compat.** Making `unresolvable_count` +
   `unresolvable_rate` required fields on the public TakesScorecard
   interface broke downstream SDK consumers who construct scorecard
   fixtures (gbrain-evals, custom engines). The hotfix shouldn't impose
   a compile-break on hotfix users.

   Fix: make both fields optional (`?: number` / `?: number | null`).
   `finalizeScorecard` still always populates them, so all internal code
   sees the real values. External fixtures that omit them compile cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(safety,ux): tighten DROP SCHEMA gate + surface unresolvable in scorecard CLI

Second codex adversarial pass on v0.37.0.1 surfaced two residual findings.

**P0 — Safety gate still bypassable.** First-pass safety gate used
`explicitOptIn || (isLocalhost && looksLikeTestDb)` — meaning
`GBRAIN_TEST_DB=1` bypassed BOTH the host check AND the db-name check.
Someone running the E2E with that env set against a production DATABASE_URL
would still nuke their schema. Codex re-flagged it as P0.

Tightened logic: `looksLikeTestDb && (isLocalhost || ciOptIn)`. The db-name
pattern is now the hard floor — `gbrain_test`, `*_test`, `test_*`, `*_e2e`.
GBRAIN_TEST_DB=1 only relaxes the localhost requirement (for CI service-name
hosts). Setting the env on a DATABASE_URL pointing at `production_data` is
explicitly refused with a paste-ready message naming the failed check.

Verified 3 ways:
  - gbrain_test + localhost → resets (6/0 pass)
  - production_data + GBRAIN_TEST_DB=1 → REFUSES with clear message
  - foo_e2e + GBRAIN_TEST_DB=1 → resets (test-shaped name passes)

**P2 — gbrain takes scorecard hides the unresolvable signal.** Early-return
on `resolved === 0` was triggered before the new sibling fields rendered.
A brain with only `quality='unresolvable'` verdicts — the spec's whole
production case — printed "No resolved bets yet" and exited. The
unresolvable_rate field was unreachable from the human CLI unless the user
knew to pass `--json`.

Fix: gate the early-return on `resolved === 0 AND unresolvable_count === 0`.
Render `unresolvable` count + `unresolvable_rate` alongside `partial_rate`
when present. Threshold warn at 30% (mirrors PARTIAL_RATE_WARNING_THRESHOLD)
pointing at retrieval coverage, not prediction accuracy — the actionable
read for high-unresolvable brains.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: renumber v0.37.0.1 → v0.37.2.0 (v0.37.1.0 claimed by other PRs)

PRs #1214 and #1215 both claim v0.37.1.0; bumping past to the next free
slot. Migration v79 renamed `takes_unresolvable_quality_v0_37_0_1` →
`takes_unresolvable_quality_v0_37_2_0`. VERSION + package.json +
CHANGELOG + llms bundles + inline doc references all swept.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 09:19:18 -07:00

121 lines
4.8 KiB
TypeScript

/**
* v0.30.0 (Slice A1): pure helpers for the resolution + scorecard layer.
* Shared between Postgres + PGLite engines so the math + the (quality, outcome)
* derivation are identical across backends.
*/
import { GBrainError } from './types.ts';
import type { TakeResolution, TakesScorecard } from './engine.ts';
/**
* Derive the (quality, outcome) tuple that gets written to the takes row.
* `quality` wins when both are set. Returns the tuple ready for an UPDATE.
*
* Throws TAKE_RESOLUTION_INVALID when neither field is set, or when the input
* combines fields that the schema CHECK constraint would reject (e.g.
* quality='partial' + outcome=true).
*
* The schema `takes_resolution_consistency` CHECK is defense-in-depth — this
* function is the first line, surfacing a clear CLI-friendly error before
* the row hits the DB.
*/
export function deriveResolutionTuple(
resolution: TakeResolution,
): { quality: 'correct' | 'incorrect' | 'partial' | 'unresolvable'; outcome: boolean | null } {
const { quality, outcome } = resolution;
if (quality === undefined && outcome === undefined) {
throw new GBrainError(
'TAKE_RESOLUTION_INVALID',
'resolveTake: must pass either `quality` (correct|incorrect|partial|unresolvable) or `outcome` (true|false)',
'use --quality on the CLI; --outcome is the back-compat alias and cannot express partial or unresolvable',
);
}
if (quality !== undefined) {
// Optional cross-check: when caller passed BOTH and they're inconsistent,
// surface the contradiction loudly instead of silently overwriting.
// 'unresolvable' (v0.36.1.1) is null-outcome, same as 'partial'.
if (outcome !== undefined) {
const expected = quality === 'correct' ? true : quality === 'incorrect' ? false : null;
if (expected !== outcome) {
throw new GBrainError(
'TAKE_RESOLUTION_INVALID',
`resolveTake: --quality=${quality} contradicts --outcome=${outcome}`,
'pass only one of --quality or --outcome; they cannot disagree',
);
}
}
return {
quality,
outcome: quality === 'correct' ? true : quality === 'incorrect' ? false : null,
};
}
// Back-compat path: only `outcome` was supplied (v0.28 callers). Boolean
// outcome can never derive 'unresolvable' — that requires explicit quality.
return {
quality: outcome ? 'correct' : 'incorrect',
outcome: outcome ?? null,
};
}
/** Raw aggregate row shape returned by both engines' getScorecard SQL. */
export interface ScorecardRowRaw {
total_bets: number;
resolved: number;
correct: number;
incorrect: number;
partial: number;
brier: number | null;
/** v0.36.1.1: count of rows where resolved_quality = 'unresolvable'. */
unresolvable_count?: number;
}
/**
* Finalize a scorecard from the raw aggregate row. Computes accuracy +
* partial_rate + unresolvable_rate; returns NULL for empty windows so CLI
* can render "no resolved bets yet" instead of NaN. Brier comes straight
* from SQL.
*
* Brier scope (D5 + D11): `partial` AND `unresolvable` (v0.36.1.1) rows
* are excluded from the Brier denominator entirely because neither is a
* binary outcome. The `partial_rate` and `unresolvable_rate` fields
* surface those non-binary signals separately so users see both the
* calibration math AND whether they're hedging into the unmeasured
* bucket (partial) or running into evidence gaps (unresolvable).
*
* v0.36.1.1 T1c — sibling-field semantics: `resolved` keeps its
* pre-v74 3-state meaning (correct+incorrect+partial) so historical
* scorecards compare apples-to-apples. `unresolvable_count` and
* `unresolvable_rate` are SIBLINGS, NOT folded into `resolved`.
*/
export function finalizeScorecard(raw: ScorecardRowRaw): TakesScorecard {
const correct = Number(raw.correct ?? 0);
const incorrect = Number(raw.incorrect ?? 0);
const partial = Number(raw.partial ?? 0);
const resolved = Number(raw.resolved ?? 0);
const totalBets = Number(raw.total_bets ?? 0);
const unresolvableCount = Number(raw.unresolvable_count ?? 0);
const binary = correct + incorrect;
const unresolvableDenom = resolved + unresolvableCount;
return {
total_bets: totalBets,
resolved,
correct,
incorrect,
partial,
accuracy: binary > 0 ? correct / binary : null,
brier: binary > 0 && raw.brier !== null && raw.brier !== undefined
? Number(raw.brier)
: null,
partial_rate: resolved > 0 ? partial / resolved : null,
unresolvable_count: unresolvableCount,
unresolvable_rate: unresolvableDenom > 0 ? unresolvableCount / unresolvableDenom : null,
};
}
/**
* Threshold above which scorecard CLI emits a warning that calibration may
* be optimistic (D11). 20% partial means 1 in 5 bets escaped the Brier
* denominator — the user is hedging into the unmeasured bucket.
*/
export const PARTIAL_RATE_WARNING_THRESHOLD = 0.20;