Files
gbrain/test
9a4ae0962e v0.37.2.0: takes_resolution_consistency CHECK accepts 'unresolvable' (#1211)
* v0.36.1.1 hotfix: takes_resolution_consistency CHECK accepts 'unresolvable'

Unblocks production grading scripts that write the judge's 4th verdict
type. Before this fix, every quality='unresolvable' INSERT/UPDATE hit
a CHECK violation — 0 of 34 writes landed in a recent prod run.

Migration v74 widens BOTH:
  - takes_resolution_consistency (table-level CHECK) — admits the
    ('unresolvable', NULL) pair alongside the existing 4 legal shapes
  - resolved_quality column-level CHECK — drops the auto-generated
    name from v37, re-adds as takes_resolved_quality_values with the
    4-state enum

Backward compatible. Existing rows with quality IN (NULL, 'correct',
'incorrect', 'partial') all satisfy the new CHECKs unchanged.

TakesScorecard gains sibling fields unresolvable_count + unresolvable_rate;
the existing `resolved` field deliberately keeps its 3-state meaning
so historical scorecards compare apples-to-apples (T1c sibling-field
design from the eng review).

Pinned by:
  - test/takes-resolution.test.ts — R1-R5 round-trip
  - test/migrate.test.ts — v74 structural assertions + PGLite E2E
    suite exercising all valid + invalid (quality, outcome) shapes

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e/schema-drift): reset public schema in beforeAll to isolate from caller bootstrap state

Previously the test trusted caller-provided DATABASE_URL to point at a fresh
database. CLAUDE.md's E2E lifecycle prescribes 'gbrain doctor --json' as the
bootstrap step (needed by oauth-related tests for table creation), but doctor
configures the gateway and bakes the configured embedding model into
content_chunks.model DEFAULT during the initial CREATE TABLE.

On re-run, CREATE TABLE IF NOT EXISTS is a no-op and the bootstrapped default
sticks. PGLite (always fresh-in-memory) gets the unconfigured-gateway fallback
'text-embedding-3-large'. The test reported phantom drift:
  pg.default="'zembed-1'::text"  pglite.default="'text-embedding-3-large'::text"

Fix: DROP SCHEMA public CASCADE + CREATE SCHEMA public before pg.initSchema.
Resets every table/index/sequence/constraint added by prior tooling. The PGLite
side is already fresh-per-test by construction.

Verified order-independent:
  - Fresh DB → 6/0 pass
  - After 'gbrain doctor' bootstrap → 6/0 pass
  - Full E2E suite (mechanical + schema-drift) → 84/0 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(safety): gate schema-drift DROP SCHEMA + relax TakesScorecard interface

Two findings from Codex adversarial review on the v0.37.0.1 hotfix:

1. **DROP SCHEMA safety gate (P0).** test/e2e/schema-drift.test.ts had an
   unguarded DROP SCHEMA public CASCADE. A developer running the E2E with
   DATABASE_URL pointing at a real brain or staging DB would lose the entire
   public schema. The fix: triple-check before destruction.
   - Parse the DATABASE_URL hostname + db name
   - Allow reset only when: explicit GBRAIN_TEST_DB=1 OR (localhost host AND
     test-shaped db name like gbrain_test, *_test, test_*, *_e2e)
   - Refuse otherwise with a loud paste-ready warning
   - The test still proceeds (the parity check is the fail-safe — if the
     caller already had a fresh DB, parity passes; if not, parity fails
     LOUDLY instead of nuking their data)

   Verified all three branches: localhost+gbrain_test resets (6/0 pass);
   localhost+production_brain refuses + warns (6/0 pass against pre-existing
   schema); GBRAIN_TEST_DB=1 override on production_brain name allows reset.

2. **TakesScorecard interface compat.** Making `unresolvable_count` +
   `unresolvable_rate` required fields on the public TakesScorecard
   interface broke downstream SDK consumers who construct scorecard
   fixtures (gbrain-evals, custom engines). The hotfix shouldn't impose
   a compile-break on hotfix users.

   Fix: make both fields optional (`?: number` / `?: number | null`).
   `finalizeScorecard` still always populates them, so all internal code
   sees the real values. External fixtures that omit them compile cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(safety,ux): tighten DROP SCHEMA gate + surface unresolvable in scorecard CLI

Second codex adversarial pass on v0.37.0.1 surfaced two residual findings.

**P0 — Safety gate still bypassable.** First-pass safety gate used
`explicitOptIn || (isLocalhost && looksLikeTestDb)` — meaning
`GBRAIN_TEST_DB=1` bypassed BOTH the host check AND the db-name check.
Someone running the E2E with that env set against a production DATABASE_URL
would still nuke their schema. Codex re-flagged it as P0.

Tightened logic: `looksLikeTestDb && (isLocalhost || ciOptIn)`. The db-name
pattern is now the hard floor — `gbrain_test`, `*_test`, `test_*`, `*_e2e`.
GBRAIN_TEST_DB=1 only relaxes the localhost requirement (for CI service-name
hosts). Setting the env on a DATABASE_URL pointing at `production_data` is
explicitly refused with a paste-ready message naming the failed check.

Verified 3 ways:
  - gbrain_test + localhost → resets (6/0 pass)
  - production_data + GBRAIN_TEST_DB=1 → REFUSES with clear message
  - foo_e2e + GBRAIN_TEST_DB=1 → resets (test-shaped name passes)

**P2 — gbrain takes scorecard hides the unresolvable signal.** Early-return
on `resolved === 0` was triggered before the new sibling fields rendered.
A brain with only `quality='unresolvable'` verdicts — the spec's whole
production case — printed "No resolved bets yet" and exited. The
unresolvable_rate field was unreachable from the human CLI unless the user
knew to pass `--json`.

Fix: gate the early-return on `resolved === 0 AND unresolvable_count === 0`.
Render `unresolvable` count + `unresolvable_rate` alongside `partial_rate`
when present. Threshold warn at 30% (mirrors PARTIAL_RATE_WARNING_THRESHOLD)
pointing at retrieval coverage, not prediction accuracy — the actionable
read for high-unresolvable brains.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: renumber v0.37.0.1 → v0.37.2.0 (v0.37.1.0 claimed by other PRs)

PRs #1214 and #1215 both claim v0.37.1.0; bumping past to the next free
slot. Migration v79 renamed `takes_unresolvable_quality_v0_37_0_1` →
`takes_unresolvable_quality_v0_37_2_0`. VERSION + package.json +
CHANGELOG + llms bundles + inline doc references all swept.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 09:19:18 -07:00
..