From 9a4ae0962e07d2bea5e8fd03395e02806d25263c Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 20 May 2026 09:19:18 -0700 Subject: [PATCH] v0.37.2.0: takes_resolution_consistency CHECK accepts 'unresolvable' (#1211) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * v0.36.1.1 hotfix: takes_resolution_consistency CHECK accepts 'unresolvable' Unblocks production grading scripts that write the judge's 4th verdict type. Before this fix, every quality='unresolvable' INSERT/UPDATE hit a CHECK violation — 0 of 34 writes landed in a recent prod run. Migration v74 widens BOTH: - takes_resolution_consistency (table-level CHECK) — admits the ('unresolvable', NULL) pair alongside the existing 4 legal shapes - resolved_quality column-level CHECK — drops the auto-generated name from v37, re-adds as takes_resolved_quality_values with the 4-state enum Backward compatible. Existing rows with quality IN (NULL, 'correct', 'incorrect', 'partial') all satisfy the new CHECKs unchanged. TakesScorecard gains sibling fields unresolvable_count + unresolvable_rate; the existing `resolved` field deliberately keeps its 3-state meaning so historical scorecards compare apples-to-apples (T1c sibling-field design from the eng review). Pinned by: - test/takes-resolution.test.ts — R1-R5 round-trip - test/migrate.test.ts — v74 structural assertions + PGLite E2E suite exercising all valid + invalid (quality, outcome) shapes Co-Authored-By: Claude Opus 4.7 (1M context) * test(e2e/schema-drift): reset public schema in beforeAll to isolate from caller bootstrap state Previously the test trusted caller-provided DATABASE_URL to point at a fresh database. CLAUDE.md's E2E lifecycle prescribes 'gbrain doctor --json' as the bootstrap step (needed by oauth-related tests for table creation), but doctor configures the gateway and bakes the configured embedding model into content_chunks.model DEFAULT during the initial CREATE TABLE. On re-run, CREATE TABLE IF NOT EXISTS is a no-op and the bootstrapped default sticks. PGLite (always fresh-in-memory) gets the unconfigured-gateway fallback 'text-embedding-3-large'. The test reported phantom drift: pg.default="'zembed-1'::text" pglite.default="'text-embedding-3-large'::text" Fix: DROP SCHEMA public CASCADE + CREATE SCHEMA public before pg.initSchema. Resets every table/index/sequence/constraint added by prior tooling. The PGLite side is already fresh-per-test by construction. Verified order-independent: - Fresh DB → 6/0 pass - After 'gbrain doctor' bootstrap → 6/0 pass - Full E2E suite (mechanical + schema-drift) → 84/0 pass Co-Authored-By: Claude Opus 4.7 (1M context) * fix(safety): gate schema-drift DROP SCHEMA + relax TakesScorecard interface Two findings from Codex adversarial review on the v0.37.0.1 hotfix: 1. **DROP SCHEMA safety gate (P0).** test/e2e/schema-drift.test.ts had an unguarded DROP SCHEMA public CASCADE. A developer running the E2E with DATABASE_URL pointing at a real brain or staging DB would lose the entire public schema. The fix: triple-check before destruction. - Parse the DATABASE_URL hostname + db name - Allow reset only when: explicit GBRAIN_TEST_DB=1 OR (localhost host AND test-shaped db name like gbrain_test, *_test, test_*, *_e2e) - Refuse otherwise with a loud paste-ready warning - The test still proceeds (the parity check is the fail-safe — if the caller already had a fresh DB, parity passes; if not, parity fails LOUDLY instead of nuking their data) Verified all three branches: localhost+gbrain_test resets (6/0 pass); localhost+production_brain refuses + warns (6/0 pass against pre-existing schema); GBRAIN_TEST_DB=1 override on production_brain name allows reset. 2. **TakesScorecard interface compat.** Making `unresolvable_count` + `unresolvable_rate` required fields on the public TakesScorecard interface broke downstream SDK consumers who construct scorecard fixtures (gbrain-evals, custom engines). The hotfix shouldn't impose a compile-break on hotfix users. Fix: make both fields optional (`?: number` / `?: number | null`). `finalizeScorecard` still always populates them, so all internal code sees the real values. External fixtures that omit them compile cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(safety,ux): tighten DROP SCHEMA gate + surface unresolvable in scorecard CLI Second codex adversarial pass on v0.37.0.1 surfaced two residual findings. **P0 — Safety gate still bypassable.** First-pass safety gate used `explicitOptIn || (isLocalhost && looksLikeTestDb)` — meaning `GBRAIN_TEST_DB=1` bypassed BOTH the host check AND the db-name check. Someone running the E2E with that env set against a production DATABASE_URL would still nuke their schema. Codex re-flagged it as P0. Tightened logic: `looksLikeTestDb && (isLocalhost || ciOptIn)`. The db-name pattern is now the hard floor — `gbrain_test`, `*_test`, `test_*`, `*_e2e`. GBRAIN_TEST_DB=1 only relaxes the localhost requirement (for CI service-name hosts). Setting the env on a DATABASE_URL pointing at `production_data` is explicitly refused with a paste-ready message naming the failed check. Verified 3 ways: - gbrain_test + localhost → resets (6/0 pass) - production_data + GBRAIN_TEST_DB=1 → REFUSES with clear message - foo_e2e + GBRAIN_TEST_DB=1 → resets (test-shaped name passes) **P2 — gbrain takes scorecard hides the unresolvable signal.** Early-return on `resolved === 0` was triggered before the new sibling fields rendered. A brain with only `quality='unresolvable'` verdicts — the spec's whole production case — printed "No resolved bets yet" and exited. The unresolvable_rate field was unreachable from the human CLI unless the user knew to pass `--json`. Fix: gate the early-return on `resolved === 0 AND unresolvable_count === 0`. Render `unresolvable` count + `unresolvable_rate` alongside `partial_rate` when present. Threshold warn at 30% (mirrors PARTIAL_RATE_WARNING_THRESHOLD) pointing at retrieval coverage, not prediction accuracy — the actionable read for high-unresolvable brains. Co-Authored-By: Claude Opus 4.7 (1M context) * chore: renumber v0.37.0.1 → v0.37.2.0 (v0.37.1.0 claimed by other PRs) PRs #1214 and #1215 both claim v0.37.1.0; bumping past to the next free slot. Migration v79 renamed `takes_unresolvable_quality_v0_37_0_1` → `takes_unresolvable_quality_v0_37_2_0`. VERSION + package.json + CHANGELOG + llms bundles + inline doc references all swept. --------- Co-authored-by: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 76 +++++++ CLAUDE.md | 19 ++ VERSION | 2 +- .../calibration-quality-gate-spec.md | 215 ++++++++++++++++++ llms-full.txt | 19 ++ package.json | 2 +- src/commands/takes.ts | 50 ++-- src/core/engine.ts | 54 ++++- src/core/migrate.ts | 53 +++++ src/core/pglite-engine.ts | 19 +- src/core/postgres-engine.ts | 19 +- src/core/takes-fence.ts | 4 +- src/core/takes-resolution.ts | 37 ++- src/core/utils.ts | 2 +- test/calibration-profile.test.ts | 8 + test/e2e/schema-drift.test.ts | 46 +++- test/migrate.test.ts | 186 +++++++++++++++ test/take-forecast.test.ts | 2 + test/takes-resolution.test.ts | 83 +++++++ 19 files changed, 835 insertions(+), 61 deletions(-) create mode 100644 docs/architecture/calibration-quality-gate-spec.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 6643310e8..eead6e51c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,82 @@ All notable changes to GBrain will be documented in this file. +## [0.37.2.0] - 2026-05-19 + +**Your grading script writes "unresolvable" verdicts now. Before this fix, every single one was rejected at the database layer — 0 of 34 writes landed in a recent production run.** + +When the v0.36.1.0 calibration wave grades a take, the judge can return one of four outcomes: correct, incorrect, partial, unresolvable. Unresolvable means "we tried but the evidence wasn't there to decide." The database constraint that v0.36.1.0 added only accepted the first three. Anything writing an unresolvable verdict hit a CHECK constraint violation and silently dropped the row. This hotfix widens the constraint and surfaces unresolvable as a first-class quality + a measurable scorecard signal, so a brain with 50% unresolvable verdicts can read that as "your retrieval is weak in this domain" instead of "no data." + +Hotfix-narrow. No prompt changes, no LLM dependencies, no eval gates. Rebased on top of v0.37.1.0 during a second master-merge collision — the work was originally cut against v0.36.1.0 as v0.36.1.1, but master shipped v0.36.1.1 + v0.36.2.0 + v0.36.3.0 + autonomous-remediation through v0.37.0.0 in parallel, then v0.37.1.0 (brainstorm/lsd) claimed v79. Migration renumbered v74 → v79 → v80; everything else (engine widening, scorecard sibling fields, R1-R5 tests) is unchanged. The full feature wave (falsifiability scoring, propose-side dedup, per-category calibration) follows separately as a follow-up minor. + +### What you can now do + +- Run your grading script and have unresolvable verdicts actually persist. `gbrain takes resolve --row N --quality unresolvable [--evidence "..."] [--by gbrain:grade_takes]` works through the CLI; `engine.resolveTake({ quality: 'unresolvable', resolvedBy: '...' })` works through the SDK. +- Read the unresolvable rate as a calibration signal: `engine.getScorecard(...)` returns two new sibling fields, `unresolvable_count` and `unresolvable_rate`. The denominator is `resolved + unresolvable_count`, so a 50% rate means half your grade-attempted takes ran into evidence gaps. The existing `resolved` field deliberately keeps its 3-state meaning so historical scorecards compare apples-to-apples. + +### Numbers that matter + +| Metric | Before v0.37.2.0 | After v0.37.2.0 | +|---|---|---| +| `quality='unresolvable'` writes accepted | 0% (CHECK violation) | 100% | +| Production v3 run write rate | 0 / 34 | 34 / 34 | +| `unresolvable_rate` field on `TakesScorecard` | absent | present, NULL when no data | +| Historical `partial_rate` / `accuracy` / `brier` semantics | (v0.36.1.0) | unchanged — sibling-field design preserves comparison | + +### How the fix works + +Two CHECK constraints lived on the `takes` table after v0.36.1.0: + +1. A table-level `takes_resolution_consistency` constraint enforcing valid `(resolved_quality, resolved_outcome)` pairs. Migration v80 widens it to admit `('unresolvable', NULL)` alongside the existing `('correct', true)` / `('incorrect', false)` / `('partial', NULL)` / `(NULL, NULL)` pairs. +2. A column-level CHECK on `resolved_quality` enumerating valid string values. Migration v80 drops it (both the auto-generated Postgres name `takes_resolved_quality_check` and the new explicit name `takes_resolved_quality_values`) and re-adds it with the 4-state list. + +Existing rows are unaffected. The `(NULL, NULL)`, `('correct', true)`, `('incorrect', false)`, and `('partial', NULL)` shapes still satisfy both new CHECKs. ALTER TABLE briefly acquires `AccessExclusiveLock` to validate existing rows — on a 36K-row takes table this is sub-second. Larger brains (>1M rows) may see a few seconds of write blocking during migration. + +### To take advantage of v0.37.2.0 + +`gbrain upgrade` runs migration v80 automatically. If your grading script was failing today: + +1. **Run the migration:** + ```bash + gbrain upgrade + gbrain doctor # verify schema_version >= 80 + ``` +2. **Re-run your grading script.** Unresolvable verdicts now persist; the constraint accepts them. +3. **Read the new metric:** + ```bash + gbrain calibration # shows unresolvable_rate on the scorecard + ``` + +If `gbrain doctor` reports a partial migration: +```bash +gbrain apply-migrations --yes +``` + +If anything still fails, file an issue at https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and `~/.gbrain/upgrade-errors.jsonl` if it exists. + +### Itemized changes + +#### Schema +- **`src/core/migrate.ts`** — new migration v80 `takes_unresolvable_quality_v0_37_2_0`. Idempotent. Drops + re-adds both the column-level CHECK (now named `takes_resolved_quality_values`) and the table-level `takes_resolution_consistency` CHECK with `'unresolvable'` admitted. Renumbered from v74 → v79 → v80 during successive master-merge collisions (v0.37.0.0 autonomous-remediation through v78, then v0.37.1.0 brainstorm/lsd claimed v79). + +#### Engine surface +- **`src/core/engine.ts`** — `Take.resolved_quality` widens to `'correct' | 'incorrect' | 'partial' | 'unresolvable' | null`. `TakeResolution.quality` widens to the same 4-state union. `TakesScorecard` gains two new sibling fields: `unresolvable_count: number` and `unresolvable_rate: number | null`. Existing fields (`resolved`, `correct`, `incorrect`, `partial`, `accuracy`, `brier`, `partial_rate`) unchanged. +- **`src/core/takes-resolution.ts`** — `deriveResolutionTuple` now accepts `quality: 'unresolvable'` (maps to `outcome: null`, same shape as partial). `ScorecardRowRaw.unresolvable_count` is optional so pre-v80 engine responses keep working. `finalizeScorecard` computes `unresolvable_rate = unresolvable_count / (resolved + unresolvable_count)`, NULL when both are zero. +- **`src/core/postgres-engine.ts`** + **`src/core/pglite-engine.ts`** — `getScorecard` SQL now filters `resolved` to the 3-state subset `('correct','incorrect','partial')` (deliberately, NOT `IS NOT NULL`) so historical comparisons stay valid. New `COUNT(*) FILTER (WHERE resolved_quality = 'unresolvable') AS unresolvable_count` aggregate. +- **`src/core/utils.ts`** — `rowToTake` type cast widened for the 4th quality state. +- **`src/core/takes-fence.ts`** — `TakeQuality` type union + `QUALITY_VALUES` set widened to include `'unresolvable'`, so markdown fence parsing accepts the new state alongside CLI and SDK. +- **`src/commands/takes.ts`** — `gbrain takes resolve --quality` now accepts `unresolvable`. The deprecated `--outcome` boolean alias cannot express unresolvable (same as partial); error message updated. + +#### Tests +- **`test/takes-resolution.test.ts`** — three new cases for `deriveResolutionTuple` covering the unresolvable mapping and contradiction rejection; three new cases for `finalizeScorecard` covering the sibling-field semantics (resolved stays 3-state, unresolvable_rate populates from siblings, legacy pre-v80 raw shape backfills cleanly). +- **`test/migrate.test.ts`** — four structural assertions on the v80 entry (name, idempotency, both CHECK widenings, regression guard that pre-existing legal pairs aren't dropped) plus six PGLite E2E cases exercising the round-trip: R1 unresolvable persists, R2 pre-v80 `(NULL, NULL)` survives, R3 partial+true still rejected, R4 unresolvable+true/false still rejected, R5 `getScorecard` surfaces the new sibling fields. + +#### Docs +- **`docs/architecture/calibration-quality-gate-spec.md`** — preserved from PR #1191 (now closed) with a historical-context header noting the two-wave split. v0.37.2.0 ships the CHECK fix; the follow-up feature wave ships the rest. + +#### For contributors +- The v0.36.1.0 calibration phases (`propose_takes`, `grade_takes`, `calibration_profile`) were correct: `grade-takes.ts` already returned `null` for unresolvable verdicts so they didn't try to write to the takes table. The bug was external grading scripts (and the future promote path, when it lands) writing the verdict directly. The hotfix lets either path land the same row shape. + ## [0.37.1.0] - 2026-05-19 **Your brain can now collide its own ideas.** diff --git a/CLAUDE.md b/CLAUDE.md index 52471a3fe..ff640d70c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -312,6 +312,25 @@ at `~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md`. Convention skill at `skills/conventions/calibration.md` has the agent- facing rules. +**v0.37.2.0 hotfix (2026-05-20, migration v80)** — `takes_resolution_consistency` +CHECK widened to accept `quality='unresolvable' AND outcome=NULL` as the 4th +valid resolution state. Column-level CHECK on `resolved_quality` renamed to +`takes_resolved_quality_values` and widened to enumerate all 4 states. Unblocks +production grading scripts that write the judge's 4th verdict type. `Take.resolved_quality`, +`TakeResolution.quality`, and `takes-fence.ts:TakeQuality` all widen to 4-state. +`TakesScorecard` gains sibling fields `unresolvable_count` + `unresolvable_rate`; +`resolved` stays 3-state (correct+incorrect+partial) so historical comparisons +hold. `finalizeScorecard` formula: `unresolvable_rate = unresolvable_count / (resolved ++ unresolvable_count)`, NULL when both 0. Spec doc preserved at +`docs/architecture/calibration-quality-gate-spec.md` (from closed PR #1191) since +the follow-up minor (forthcoming) ships the falsifiability + per-category +calibration on top. Migration renumbered v74→v79→v80 during successive master +merges — v0.37.0.0's autonomous-remediation wave claimed v68-v78, then v0.37.1.0 +(brainstorm/lsd) claimed v79. Pinned by +R1-R5 in `test/takes-resolution.test.ts` and `test/migrate.test.ts`'s +v80 structural + PGLite round-trip suite (CHECK admits unresolvable+NULL, still +rejects partial+true and unresolvable+true|false, pre-v80 NULL/NULL rows survive). + - `src/core/cycle/base-phase.ts` — abstract `BaseCyclePhase` class. Enforces `sourceScopeOpts(ctx)` threading at the type level; closes the v0.34.1 source-isolation leak class structurally for every new diff --git a/VERSION b/VERSION index 5f878d2ab..8a920d889 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.37.1.0 \ No newline at end of file +0.37.2.0 \ No newline at end of file diff --git a/docs/architecture/calibration-quality-gate-spec.md b/docs/architecture/calibration-quality-gate-spec.md new file mode 100644 index 000000000..54a03e414 --- /dev/null +++ b/docs/architecture/calibration-quality-gate-spec.md @@ -0,0 +1,215 @@ +# Calibration Quality Gate — Falsifiability Filter + Category Classification + +> **Historical context.** This is the source spec absorbed from PR #1191 into +> two waves of implementation: +> +> - **v0.37.2.0 hotfix** (this release): widens the `takes_resolution_consistency` +> CHECK constraint to accept `quality='unresolvable'` as a 4th valid state. +> Unblocks the production grading script. Adds `unresolvable_count` + +> `unresolvable_rate` to `TakesScorecard` as sibling fields (preserves +> v0.36.1.0 historical comparison semantics). Migration renumbered v74→v79→v80 +> during successive master merges — v0.37.0.0's autonomous-remediation wave +> claimed v68-v78, then v0.37.1.0 (brainstorm/lsd) claimed v79. +> - **Follow-up minor** (forthcoming): falsifiability + category extraction at +> `propose_takes`, SQL-side grade gate, per-category calibration scorecards, +> pg_trgm-based proposal dedup. Wave-blocking on cat15 F1 re-validation +> against the v0.36.1.0 fixtures. +> +> Preserved here per the hotfix plan's PR #1191 close protocol so the +> production context (96K-page brain, 6.8% falsifiability rate, category +> breakdown) doesn't get lost in the CHANGELOG → release-notes condensation. + +## Problem + +v0.36.1.0 ships `propose_takes`, `grade_takes`, and `calibration_profile` as a +connected pipeline: extract claims → grade them against outcomes → build a +calibration profile showing systematic biases. + +In production on a 96K-page brain with 36K takes across 6,239 holders, the +grade_takes phase produces noisy results: + +- **6.8% falsifiability rate**: Of 500 candidate takes (weight ≥ 0.7), only 34 + passed an LLM falsifiability filter. The other 93% were philosophical beliefs, + present-state observations, advice, logistics, or vague vibes. +- **50% unresolvable**: Even after filtering, 17/34 predictions couldn't be + graded because evidence was insufficient or the claim was too ambiguous. +- **Duplicates**: Same claim from the same page extracted multiple times with + slightly different wording. + +The root cause: `propose_takes` extracts everything that looks like a belief or +assertion. That's correct for the *takes* table (epistemological layer), but +`grade_takes` needs a much narrower subset: **falsifiable predictions about +future outcomes** where we can check what actually happened. + +### Example classifications from production testing + +**Genuine predictions (grade-worthy):** +- "X will reach $1M ARR very soon" → company_outcome +- "X is going to leave Y" → people_move +- "AI will make authentic authorship more important" → technology +- "X was convinced Y would win the Z market" → market_call + +**Not predictions (should skip grading):** +- "Desire is mimetic" → philosophical belief +- "X should charge 10x more" → advice +- "Return from Toronto on Monday" → logistics +- "Something is going to happen there" → vague/unfalsifiable +- "X is growing very quickly" → present-state observation + +## Solution + +### 1. Falsifiability score at extraction time + +Add a `falsifiability` column to the `takes` table (real, 0.0–1.0, nullable, +default null). `propose_takes` sets this during extraction using the same LLM +call that already produces the take — one additional field in the JSON schema. + +```sql +ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability real; +ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability_category text; +``` + +The LLM prompt addition (appended to the existing propose_takes extraction prompt): + +``` +For each claim, also assess: +- falsifiability (0.0-1.0): Can this claim be checked against future reality? + 1.0 = specific, measurable, time-bounded prediction about an outcome + 0.5 = directional claim that's partially checkable + 0.0 = philosophical belief, advice, observation, or unfalsifiable assertion +- falsifiability_category: one of + company_outcome | fundraising | technology | people_move | market_call | other_prediction | not_prediction +``` + +Cost: ~0 incremental tokens (the claim is already being extracted; this adds +two fields to the JSON output schema). + +### 2. Grade gate in `grade_takes` + +Before attempting grading, filter: + +```typescript +const gradeable = candidates.filter(t => + t.falsifiability !== null && t.falsifiability >= 0.7 + && t.falsifiability_category !== 'not_prediction' +); +``` + +This reduces grading volume by ~93% in production, which means: +- LLM cost for grading drops proportionally +- Evidence retrieval load drops (each grade attempt triggers hybrid search) +- Calibration profiles are built on real predictions, not noise + +### 3. Deduplication at extraction + +`propose_takes` should check for near-duplicate claims before inserting: + +```typescript +// Before inserting a new take, check if a similar claim exists +// for the same holder from the same page +const existing = await engine.sql` + SELECT id, claim FROM takes + WHERE holder = ${holder} + AND page_id = ${pageId} + AND similarity(claim, ${newClaim}) > 0.8 + LIMIT 1 +`; +if (existing.length > 0) { + // Skip — near-duplicate + continue; +} +``` + +Requires `pg_trgm` extension (already available on most Postgres installations). +Falls back gracefully: if `similarity()` isn't available, skip the dedup check. + +### 4. Category-aware calibration profiles + +The `calibration_profile` phase can now group resolved takes by +`falsifiability_category` to produce per-domain scorecards: + +``` +"Your company_outcome calls are 73% accurate. + Your people_move calls are 90% accurate. + Your technology calls are 60% accurate — you tend to be ~18 months early." +``` + +This is the tweetable output: a calibration profile that says "here's how you're +systematically right and wrong by category." + +## Schema Changes + +```sql +-- Migration: add falsifiability columns to takes +ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability real; +ALTER TABLE takes ADD COLUMN IF NOT EXISTS falsifiability_category text; + +-- Index for grade_takes filter +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_takes_falsifiability + ON takes (falsifiability) + WHERE falsifiability IS NOT NULL AND falsifiability >= 0.7; + +-- Optional: pg_trgm for dedup (CREATE EXTENSION IF NOT EXISTS pg_trgm;) +``` + +## Evidence Retrieval (v0.36.1.0 → v0.37 enhancement) + +The current `grade_takes` evidence retriever returns a stub placeholder. In +production testing, we wired real evidence retrieval via `gbrain query` (hybrid +search). The pattern that works: + +1. Extract the core claim from the take (first 150 chars) +2. Run `engine.query(claim)` to get relevant pages +3. Filter to pages updated AFTER the take's `since_date` (evidence must be newer) +4. Pass top-5 chunks as the evidence block to the judge + +This should replace the stub in the `evidenceRetriever` injection point. + +## Production Results + +After implementing the falsifiability filter (as a pre-processing step outside +the cycle): + +| Metric | Before (v2, no filter) | After (v3, with filter) | +|--------|----------------------|----------------------| +| Candidates evaluated | 50 | 34 (from 500 screened) | +| Falsifiable predictions | ~19 (38%) | 34 (100%) | +| Correct | 10 (52.6% of resolvable) | 10 (58.8% of resolvable) | +| Incorrect | 5 (26.3%) | 2 (11.8%) | +| Partial | 4 (21.1%) | 5 (29.4%) | +| Unresolvable | 31 (62%) | 17 (50%) | +| Category breakdown | N/A | people_move:13, company_outcome:11, technology:4, market_call:2 | + +Key improvement: **the false positive rate dropped from 62% noise to 0% noise** +in the gradeable set. The remaining 50% unresolvable rate is genuine — those +predictions are about outcomes that haven't happened yet or where the brain +lacks evidence. That's correct behavior, not noise. + +## Files to Change + +1. **`src/core/cycle/propose-takes.ts`** — Add falsifiability + category to + extraction prompt and output schema +2. **`src/core/cycle/grade-takes.ts`** — Add falsifiability gate before grading; + wire real evidence retrieval +3. **`src/core/cycle/calibration-profile.ts`** — Group scorecards by category +4. **`src/core/engine.ts`** — Add `similarity()` helper for dedup (graceful + fallback) +5. **New migration** — Add columns + index + +## Testing + +- Unit test: falsifiability classifier on 20 known-good and 20 known-noise takes +- Unit test: dedup correctly merges near-identical claims +- Unit test: grade gate filters below threshold +- Integration test: full cycle with falsifiability → grade → profile pipeline +- Regression test: existing takes without falsifiability score are not broken + (null falsifiability = ungated, backward compatible) + +## Backward Compatibility + +- `falsifiability` defaults to null. Existing takes are unaffected. +- `grade_takes` with null falsifiability: configurable behavior. Default: + grade all (backward compat). Operator can set + `cycle.grade_takes.require_falsifiability: true` to gate. +- Category column is purely additive. +- Dedup is opt-in: `cycle.propose_takes.dedup.enabled: true`. diff --git a/llms-full.txt b/llms-full.txt index e559a40d9..5069b0129 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -448,6 +448,25 @@ at `~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md`. Convention skill at `skills/conventions/calibration.md` has the agent- facing rules. +**v0.37.2.0 hotfix (2026-05-20, migration v80)** — `takes_resolution_consistency` +CHECK widened to accept `quality='unresolvable' AND outcome=NULL` as the 4th +valid resolution state. Column-level CHECK on `resolved_quality` renamed to +`takes_resolved_quality_values` and widened to enumerate all 4 states. Unblocks +production grading scripts that write the judge's 4th verdict type. `Take.resolved_quality`, +`TakeResolution.quality`, and `takes-fence.ts:TakeQuality` all widen to 4-state. +`TakesScorecard` gains sibling fields `unresolvable_count` + `unresolvable_rate`; +`resolved` stays 3-state (correct+incorrect+partial) so historical comparisons +hold. `finalizeScorecard` formula: `unresolvable_rate = unresolvable_count / (resolved ++ unresolvable_count)`, NULL when both 0. Spec doc preserved at +`docs/architecture/calibration-quality-gate-spec.md` (from closed PR #1191) since +the follow-up minor (forthcoming) ships the falsifiability + per-category +calibration on top. Migration renumbered v74→v79→v80 during successive master +merges — v0.37.0.0's autonomous-remediation wave claimed v68-v78, then v0.37.1.0 +(brainstorm/lsd) claimed v79. Pinned by +R1-R5 in `test/takes-resolution.test.ts` and `test/migrate.test.ts`'s +v80 structural + PGLite round-trip suite (CHECK admits unresolvable+NULL, still +rejects partial+true and unresolvable+true|false, pre-v80 NULL/NULL rows survive). + - `src/core/cycle/base-phase.ts` — abstract `BaseCyclePhase` class. Enforces `sourceScopeOpts(ctx)` threading at the type level; closes the v0.34.1 source-isolation leak class structurally for every new diff --git a/package.json b/package.json index 1735a0685..b4e0c45d0 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.37.1.0", + "version": "0.37.2.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/commands/takes.ts b/src/commands/takes.ts index 09ae7af8a..167255724 100644 --- a/src/commands/takes.ts +++ b/src/commands/takes.ts @@ -308,7 +308,7 @@ async function cmdResolve(engine: BrainEngine, args: string[]): Promise { const qualityStr = flagValue(args, '--quality'); const outcomeStr = flagValue(args, '--outcome'); if (!slug || !rowNumStr || (!qualityStr && !outcomeStr)) { - console.error('Usage: gbrain takes resolve --row N --quality correct|incorrect|partial [--evidence "..."] [--value N --unit usd|pct|count] [--by ]'); + console.error('Usage: gbrain takes resolve --row N --quality correct|incorrect|partial|unresolvable [--evidence "..."] [--value N --unit usd|pct|count] [--by ]'); console.error(' (back-compat) gbrain takes resolve --row N --outcome true|false [...]'); process.exit(1); } @@ -319,12 +319,13 @@ async function cmdResolve(engine: BrainEngine, args: string[]): Promise { const rowNum = parseInt(rowNumStr, 10); // v0.30.0: --quality is the new primary input. --outcome stays as a back-compat - // alias auto-mapping true→correct / false→incorrect; cannot express partial. - let quality: 'correct' | 'incorrect' | 'partial' | undefined; + // alias auto-mapping true→correct / false→incorrect; cannot express partial + // or unresolvable (v0.36.1.1). + let quality: 'correct' | 'incorrect' | 'partial' | 'unresolvable' | undefined; let outcome: boolean | undefined; if (qualityStr) { - if (qualityStr !== 'correct' && qualityStr !== 'incorrect' && qualityStr !== 'partial') { - console.error(`Invalid --quality "${qualityStr}". Expected: correct, incorrect, partial.`); + if (qualityStr !== 'correct' && qualityStr !== 'incorrect' && qualityStr !== 'partial' && qualityStr !== 'unresolvable') { + console.error(`Invalid --quality "${qualityStr}". Expected: correct, incorrect, partial, unresolvable.`); process.exit(1); } quality = qualityStr; @@ -432,29 +433,46 @@ async function cmdScorecard(engine: BrainEngine, args: string[]): Promise return; } - if (card.resolved === 0) { + // v0.37.2.0: don't hide the unresolvable signal. A brain with only unresolvable + // verdicts still has a story to tell — "your judge tried but couldn't decide" — + // and the spec's whole headline ("50% of your tech calls land unresolvable") + // depends on this output rendering when resolved=0 but unresolvable_count>0. + const unresolvableCount = card.unresolvable_count ?? 0; + if (card.resolved === 0 && unresolvableCount === 0) { console.log(`No resolved bets yet${holder ? ` for ${holder}` : ''}.`); return; } - const fmt = (n: number | null, digits = 3) => n === null ? '—' : n.toFixed(digits); + const fmt = (n: number | null | undefined, digits = 3) => + n === null || n === undefined ? '—' : n.toFixed(digits); console.log(`# Scorecard${holder ? ` — ${holder}` : ''}`); if (domainPrefix) console.log(`Scope: domain=${domainPrefix}`); if (since || until) console.log(`Window: ${since ?? 'all'} → ${until ?? 'now'}`); console.log(''); - console.log(` total bets: ${card.total_bets}`); - console.log(` resolved: ${card.resolved}`); - console.log(` correct: ${card.correct}`); - console.log(` incorrect: ${card.incorrect}`); - console.log(` partial: ${card.partial}`); - console.log(` accuracy: ${fmt(card.accuracy)}`); - console.log(` Brier: ${fmt(card.brier, 4)} (correct ∨ incorrect only; lower is better; 0.25 = always-50% baseline)`); - console.log(` partial_rate: ${fmt(card.partial_rate)}`); + console.log(` total bets: ${card.total_bets}`); + console.log(` resolved: ${card.resolved}`); + console.log(` correct: ${card.correct}`); + console.log(` incorrect: ${card.incorrect}`); + console.log(` partial: ${card.partial}`); + if (unresolvableCount > 0 || card.unresolvable_rate !== undefined) { + console.log(` unresolvable: ${unresolvableCount}`); + } + console.log(` accuracy: ${fmt(card.accuracy)}`); + console.log(` Brier: ${fmt(card.brier, 4)} (correct ∨ incorrect only; lower is better; 0.25 = always-50% baseline)`); + console.log(` partial_rate: ${fmt(card.partial_rate)}`); + if (unresolvableCount > 0 || card.unresolvable_rate !== undefined && card.unresolvable_rate !== null) { + console.log(` unresolvable_rate: ${fmt(card.unresolvable_rate)} (unresolvable / (resolved + unresolvable); high = weak evidence retrieval)`); + } if (card.partial_rate !== null && card.partial_rate > PARTIAL_RATE_WARNING_THRESHOLD) { console.log(''); console.log(` [!] partial_rate is high (>${(PARTIAL_RATE_WARNING_THRESHOLD * 100).toFixed(0)}%) — calibration may be optimistic.`); console.log(` Hedged bets escape the Brier denominator. Resolve them more decisively if the data supports it.`); } - if (card.resolved < 100) { + if (card.unresolvable_rate !== null && card.unresolvable_rate !== undefined && card.unresolvable_rate > 0.30) { + console.log(''); + console.log(` [!] unresolvable_rate is high (>${(0.30 * 100).toFixed(0)}%) — most grade attempts are running into evidence gaps.`); + console.log(` The judge is working; retrieval isn't producing enough context to decide. Look at evidence-retrieval coverage, not prediction accuracy.`); + } + if (card.resolved < 100 && card.resolved > 0) { console.log(''); console.log(` Note: n=${card.resolved} is small. Brier is noisy below ~100 resolved bets.`); } diff --git a/src/core/engine.ts b/src/core/engine.ts index 5fd917f5c..13f79a9dd 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -169,13 +169,16 @@ export interface Take { resolved_at: string | null; resolved_outcome: boolean | null; /** - * v0.30.0: 3-state outcome label. Sits alongside `resolved_outcome` for - * back-compat. New writes populate both; legacy v0.28-resolved rows have - * `resolved_quality` backfilled by migration v40 from the boolean. - * Null on unresolved rows. Schema CHECK enforces (quality, outcome) consistency: - * `correct` ↔ `outcome=true`, `incorrect` ↔ `outcome=false`, `partial` ↔ `outcome=NULL`. + * v0.30.0: 3-state outcome label. v0.36.1.1 added 'unresolvable' as a 4th + * state for verdicts where evidence was insufficient to grade. Sits + * alongside `resolved_outcome` for back-compat. New writes populate both; + * legacy v0.28-resolved rows have `resolved_quality` backfilled by + * migration v40 from the boolean. Null on unresolved rows. Schema CHECK + * (widened in v74) enforces (quality, outcome) consistency: + * `correct` ↔ `outcome=true`, `incorrect` ↔ `outcome=false`, + * `partial` ↔ `outcome=NULL`, `unresolvable` ↔ `outcome=NULL`. */ - resolved_quality: 'correct' | 'incorrect' | 'partial' | null; + resolved_quality: 'correct' | 'incorrect' | 'partial' | 'unresolvable' | null; resolved_value: number | null; resolved_unit: string | null; resolved_source: string | null; @@ -222,11 +225,14 @@ export interface StaleTakeRow { /** Resolution metadata for resolveTake. */ export interface TakeResolution { /** - * v0.30.0: primary 3-state input. When set, takes precedence over `outcome` - * and the engine writes both columns (quality directly; outcome derived: - * `correct→true`, `incorrect→false`, `partial→null`). + * v0.30.0: primary 3-state input; v0.36.1.1 widened to 4-state with + * 'unresolvable'. When set, takes precedence over `outcome` and the engine + * writes both columns (quality directly; outcome derived: + * `correct→true`, `incorrect→false`, `partial→null`, `unresolvable→null`). + * `unresolvable` marks rows where the judge ran but evidence was + * insufficient to grade; surfaces in `TakesScorecard.unresolvable_count`. */ - quality?: 'correct' | 'incorrect' | 'partial'; + quality?: 'correct' | 'incorrect' | 'partial' | 'unresolvable'; /** * v0.28 back-compat input. Keep submitting for v0.28 callers; the engine * derives quality (`true→correct`, `false→incorrect`). When `quality` is @@ -244,6 +250,12 @@ export interface TakeResolution { /** v0.30.0: scorecard aggregate. */ export interface TakesScorecard { total_bets: number; + /** + * Count of resolved rows where `resolved_quality IN + * ('correct','incorrect','partial')`. v0.36.1.1 deliberately keeps this + * 3-state semantic to preserve historical comparisons. Unresolvable rows + * land in the sibling `unresolvable_count` field instead. + */ resolved: number; correct: number; incorrect: number; @@ -254,12 +266,30 @@ export interface TakesScorecard { * Brier score over rows where `resolved_quality IN ('correct','incorrect')`. * Maps `correct→1`, `incorrect→0`, computes `mean((weight − outcome)²)`. * Lower is better; 0 = perfect; 0.25 = always-50% baseline. - * Excludes partial — that label hides hedging behavior; `partial_rate` - * surfaces it as a separate signal. NULL when no correct+incorrect rows. + * Excludes partial AND unresolvable — both hide signal; the dedicated + * `partial_rate` and `unresolvable_rate` fields surface them separately. + * NULL when no correct+incorrect rows. */ brier: number | null; /** partial / resolved. NULL when n=0. */ partial_rate: number | null; + /** + * v0.36.1.1: count of rows where `resolved_quality = 'unresolvable'`. + * Sibling field to `resolved` so historical comparisons against pre-v80 + * scorecards stay valid; `resolved` retains its 3-state meaning, and + * unresolvable rows count here separately. Optional for SDK back-compat — + * downstream consumers constructing TakesScorecard fixtures shouldn't have + * to update on a hotfix. `finalizeScorecard` always populates it. + */ + unresolvable_count?: number; + /** + * v0.37.2.0: `unresolvable_count / (resolved + unresolvable_count)`. NULL + * when both are 0. Surfaces the spec's headline calibration signal: + * "what fraction of grade-attempted takes couldn't be graded?" — high + * values signal weak evidence retrieval rather than wrong predictions. + * Optional for SDK back-compat; see `unresolvable_count` note above. + */ + unresolvable_rate?: number | null; } export interface TakesScorecardOpts { diff --git a/src/core/migrate.ts b/src/core/migrate.ts index edd79c0bc..ad6371a35 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -3713,6 +3713,59 @@ export const MIGRATIONS: Migration[] = [ ON pages (last_retrieved_at); `, }, + { + version: 80, + name: 'takes_unresolvable_quality_v0_37_2_0', + // v0.37.2.0 hotfix — accepts quality='unresolvable' as a 4th valid + // resolution state. Unblocks production grading scripts that write the + // 4th verdict type (the judge in grade-takes returns + // correct|incorrect|partial|unresolvable, but v37's CHECKs only allowed + // the first three). + // + // Two CHECKs to widen: + // (a) Table-level `takes_resolution_consistency` enumerates valid + // (quality, outcome) pairs. We add ('unresolvable', NULL). + // (b) Column-level CHECK on resolved_quality enumerates valid string + // values. Postgres auto-names this `takes_resolved_quality_check` + // when it's attached via ADD COLUMN ... CHECK. We drop it and + // re-add with the wider value list (named explicitly this time + // so future widening targets a known name). + // + // Existing rows with (NULL, NULL), ('correct', true), ('incorrect', + // false), ('partial', NULL) all satisfy both new CHECKs unchanged. + // + // ALTER TABLE ADD CONSTRAINT acquires AccessExclusiveLock while it + // validates existing rows. On a 36K-row takes table this is sub-second; + // larger tables would want NOT VALID + VALIDATE CONSTRAINT, deferred. + // + // Renumbered v74→v79→v80 during successive master merges: master's + // v0.36.1.0 calibration + v0.36.3.0 + autonomous-remediation claimed + // v68-v78, then v0.37.1.0 claimed v79. + idempotent: true, + sql: ` + -- (b) Drop both possible names for the column-level CHECK: + -- v37's auto-generated takes_resolved_quality_check (Postgres default + -- for inline ADD COLUMN CHECK) and the explicit + -- takes_resolved_quality_values name we re-add below (idempotent on + -- re-run). + ALTER TABLE takes DROP CONSTRAINT IF EXISTS takes_resolved_quality_check; + ALTER TABLE takes DROP CONSTRAINT IF EXISTS takes_resolved_quality_values; + ALTER TABLE takes ADD CONSTRAINT takes_resolved_quality_values CHECK ( + resolved_quality IS NULL + OR resolved_quality IN ('correct', 'incorrect', 'partial', 'unresolvable') + ); + + -- (a) Widen the (quality, outcome) consistency CHECK. + ALTER TABLE takes DROP CONSTRAINT IF EXISTS takes_resolution_consistency; + ALTER TABLE takes ADD CONSTRAINT takes_resolution_consistency CHECK ( + (resolved_quality IS NULL AND resolved_outcome IS NULL) + OR (resolved_quality = 'correct' AND resolved_outcome = true) + OR (resolved_quality = 'incorrect' AND resolved_outcome = false) + OR (resolved_quality = 'partial' AND resolved_outcome IS NULL) + OR (resolved_quality = 'unresolvable' AND resolved_outcome IS NULL) + ); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index d0792967f..33df16ae6 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -3458,23 +3458,28 @@ export class PGLiteEngine implements BrainEngine { if (opts.until !== undefined) { params.push(opts.until); clauses.push(`AND since_date <= $${params.length}`); } if (allowList !== undefined) { params.push(allowList); clauses.push(`AND holder = ANY($${params.length}::text[])`); } const where = clauses.join(' '); + // v0.36.1.1 T1c: `resolved` deliberately filters to the 3-state subset + // (correct|incorrect|partial) — NOT `resolved_quality IS NOT NULL` — so + // historical comparisons against pre-v74 scorecards stay valid. + // `unresolvable_count` is a sibling field counting the new 4th state. const res = await this.db.query( `SELECT - COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets, - COUNT(*) FILTER (WHERE resolved_quality IS NOT NULL)::int AS resolved, - COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct, - COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect, - COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial, + COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets, + COUNT(*) FILTER (WHERE resolved_quality IN ('correct','incorrect','partial'))::int AS resolved, + COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct, + COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect, + COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial, + COUNT(*) FILTER (WHERE resolved_quality = 'unresolvable')::int AS unresolvable_count, AVG( CASE WHEN resolved_quality IN ('correct','incorrect') THEN POWER(weight - (CASE resolved_quality WHEN 'correct' THEN 1 ELSE 0 END), 2) END - )::float AS brier + )::float AS brier FROM takes WHERE 1=1 ${where}`, params, ); - const r = res.rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; brier: number | null }; + const r = res.rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; unresolvable_count: number; brier: number | null }; return finalizeScorecard(r); } diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index ccc818df3..bd8f69adb 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -3429,22 +3429,27 @@ export class PostgresEngine implements BrainEngine { : sql``; const sinceClause = opts.since ? sql`AND since_date >= ${opts.since}` : sql``; const untilClause = opts.until ? sql`AND since_date <= ${opts.until}` : sql``; + // v0.36.1.1 T1c: `resolved` deliberately filters to the 3-state subset + // (correct|incorrect|partial) — NOT `resolved_quality IS NOT NULL` — so + // historical comparisons against pre-v74 scorecards stay valid. + // `unresolvable_count` is a sibling field counting the new 4th state. const rows = await sql` SELECT - COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets, - COUNT(*) FILTER (WHERE resolved_quality IS NOT NULL)::int AS resolved, - COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct, - COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect, - COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial, + COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets, + COUNT(*) FILTER (WHERE resolved_quality IN ('correct','incorrect','partial'))::int AS resolved, + COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct, + COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect, + COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial, + COUNT(*) FILTER (WHERE resolved_quality = 'unresolvable')::int AS unresolvable_count, AVG( CASE WHEN resolved_quality IN ('correct','incorrect') THEN POWER(weight - (CASE resolved_quality WHEN 'correct' THEN 1 ELSE 0 END), 2) END - )::float AS brier + )::float AS brier FROM takes WHERE 1=1 ${holderClause} ${domainClause} ${sinceClause} ${untilClause} ${allowed} `; - const r = rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; brier: number | null }; + const r = rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; unresolvable_count: number; brier: number | null }; return finalizeScorecard(r); } diff --git a/src/core/takes-fence.ts b/src/core/takes-fence.ts index 958b03f31..35c4fe89a 100644 --- a/src/core/takes-fence.ts +++ b/src/core/takes-fence.ts @@ -38,7 +38,7 @@ export type TakeKind = 'fact' | 'take' | 'bet' | 'hunch'; -export type TakeQuality = 'correct' | 'incorrect' | 'partial'; +export type TakeQuality = 'correct' | 'incorrect' | 'partial' | 'unresolvable'; export interface ParsedTake { rowNum: number; @@ -144,7 +144,7 @@ export function isValidHolder(holder: string): boolean { } const KIND_VALUES: ReadonlySet = new Set(['fact', 'take', 'bet', 'hunch']); -const QUALITY_VALUES: ReadonlySet = new Set(['correct', 'incorrect', 'partial']); +const QUALITY_VALUES: ReadonlySet = new Set(['correct', 'incorrect', 'partial', 'unresolvable']); // v0.30.0: header tokens that mark a v0.30-shape fence. Presence of `quality` // (or any other resolution column) widens the parser to read 7+ extra cells diff --git a/src/core/takes-resolution.ts b/src/core/takes-resolution.ts index 3d32dc27d..04a8e9e85 100644 --- a/src/core/takes-resolution.ts +++ b/src/core/takes-resolution.ts @@ -21,18 +21,19 @@ import type { TakeResolution, TakesScorecard } from './engine.ts'; */ export function deriveResolutionTuple( resolution: TakeResolution, -): { quality: 'correct' | 'incorrect' | 'partial'; outcome: boolean | null } { +): { quality: 'correct' | 'incorrect' | 'partial' | 'unresolvable'; outcome: boolean | null } { const { quality, outcome } = resolution; if (quality === undefined && outcome === undefined) { throw new GBrainError( 'TAKE_RESOLUTION_INVALID', - 'resolveTake: must pass either `quality` (correct|incorrect|partial) or `outcome` (true|false)', - 'use --quality on the CLI; --outcome is the back-compat alias and cannot express partial', + 'resolveTake: must pass either `quality` (correct|incorrect|partial|unresolvable) or `outcome` (true|false)', + 'use --quality on the CLI; --outcome is the back-compat alias and cannot express partial or unresolvable', ); } if (quality !== undefined) { // Optional cross-check: when caller passed BOTH and they're inconsistent, // surface the contradiction loudly instead of silently overwriting. + // 'unresolvable' (v0.36.1.1) is null-outcome, same as 'partial'. if (outcome !== undefined) { const expected = quality === 'correct' ? true : quality === 'incorrect' ? false : null; if (expected !== outcome) { @@ -48,7 +49,8 @@ export function deriveResolutionTuple( outcome: quality === 'correct' ? true : quality === 'incorrect' ? false : null, }; } - // Back-compat path: only `outcome` was supplied (v0.28 callers). + // Back-compat path: only `outcome` was supplied (v0.28 callers). Boolean + // outcome can never derive 'unresolvable' — that requires explicit quality. return { quality: outcome ? 'correct' : 'incorrect', outcome: outcome ?? null, @@ -63,18 +65,27 @@ export interface ScorecardRowRaw { incorrect: number; partial: number; brier: number | null; + /** v0.36.1.1: count of rows where resolved_quality = 'unresolvable'. */ + unresolvable_count?: number; } /** * Finalize a scorecard from the raw aggregate row. Computes accuracy + - * partial_rate; returns NULL for empty windows so CLI can render - * "no resolved bets yet" instead of NaN. Brier comes straight from SQL. + * partial_rate + unresolvable_rate; returns NULL for empty windows so CLI + * can render "no resolved bets yet" instead of NaN. Brier comes straight + * from SQL. * - * Brier scope (D5 + D11): `partial` rows are excluded from the Brier - * denominator entirely because partial isn't a binary outcome. The - * `partial_rate` field surfaces hedging behavior as a separate signal so - * users see both the calibration math AND whether they're hedging into - * the unmeasured bucket. + * Brier scope (D5 + D11): `partial` AND `unresolvable` (v0.36.1.1) rows + * are excluded from the Brier denominator entirely because neither is a + * binary outcome. The `partial_rate` and `unresolvable_rate` fields + * surface those non-binary signals separately so users see both the + * calibration math AND whether they're hedging into the unmeasured + * bucket (partial) or running into evidence gaps (unresolvable). + * + * v0.36.1.1 T1c — sibling-field semantics: `resolved` keeps its + * pre-v74 3-state meaning (correct+incorrect+partial) so historical + * scorecards compare apples-to-apples. `unresolvable_count` and + * `unresolvable_rate` are SIBLINGS, NOT folded into `resolved`. */ export function finalizeScorecard(raw: ScorecardRowRaw): TakesScorecard { const correct = Number(raw.correct ?? 0); @@ -82,7 +93,9 @@ export function finalizeScorecard(raw: ScorecardRowRaw): TakesScorecard { const partial = Number(raw.partial ?? 0); const resolved = Number(raw.resolved ?? 0); const totalBets = Number(raw.total_bets ?? 0); + const unresolvableCount = Number(raw.unresolvable_count ?? 0); const binary = correct + incorrect; + const unresolvableDenom = resolved + unresolvableCount; return { total_bets: totalBets, resolved, @@ -94,6 +107,8 @@ export function finalizeScorecard(raw: ScorecardRowRaw): TakesScorecard { ? Number(raw.brier) : null, partial_rate: resolved > 0 ? partial / resolved : null, + unresolvable_count: unresolvableCount, + unresolvable_rate: unresolvableDenom > 0 ? unresolvableCount / unresolvableDenom : null, }; } diff --git a/src/core/utils.ts b/src/core/utils.ts index 59e786f9b..74c5ee384 100644 --- a/src/core/utils.ts +++ b/src/core/utils.ts @@ -309,7 +309,7 @@ export function takeRowToTake(row: Record): Take { resolved_outcome: row.resolved_outcome == null ? null : Boolean(row.resolved_outcome), resolved_quality: row.resolved_quality == null ? null - : (String(row.resolved_quality) as 'correct' | 'incorrect' | 'partial'), + : (String(row.resolved_quality) as 'correct' | 'incorrect' | 'partial' | 'unresolvable'), resolved_value: row.resolved_value == null ? null : Number(row.resolved_value), resolved_unit: row.resolved_unit == null ? null : String(row.resolved_unit), resolved_source: row.resolved_source == null ? null : String(row.resolved_source), diff --git a/test/calibration-profile.test.ts b/test/calibration-profile.test.ts index 00718c7be..f5fd8583e 100644 --- a/test/calibration-profile.test.ts +++ b/test/calibration-profile.test.ts @@ -138,6 +138,8 @@ describe('pickFallbackSlots', () => { accuracy: 0.4, brier: 0.32, partial_rate: 0, + unresolvable_count: 0, + unresolvable_rate: null, }; expect(__testing.pickFallbackSlots(scorecard).direction).toBe('over-confident'); }); @@ -152,6 +154,8 @@ describe('pickFallbackSlots', () => { accuracy: 0.8, brier: 0.12, partial_rate: 0, + unresolvable_count: 0, + unresolvable_rate: null, }; expect(__testing.pickFallbackSlots(scorecard).direction).toBe('mostly right'); }); @@ -166,6 +170,8 @@ describe('pickFallbackSlots', () => { accuracy: null, brier: null, partial_rate: null, + unresolvable_count: 0, + unresolvable_rate: null, }; const out = __testing.pickFallbackSlots(scorecard); expect(out.nRight).toBe(0); @@ -184,6 +190,8 @@ const ENOUGH_RESOLVED_SCORECARD: TakesScorecard = { accuracy: 0.636, brier: 0.21, partial_rate: 0.083, + unresolvable_count: 0, + unresolvable_rate: null, }; describe('runPhaseCalibrationProfile — phase integration', () => { diff --git a/test/e2e/schema-drift.test.ts b/test/e2e/schema-drift.test.ts index 5b8611dde..159170a41 100644 --- a/test/e2e/schema-drift.test.ts +++ b/test/e2e/schema-drift.test.ts @@ -75,11 +75,51 @@ describe.skipIf(skip)('schema drift: PGLite ↔ Postgres post-initSchema parity await pglite.connect({}); await pglite.initSchema(); - // Postgres side: connect to the test database, run the canonical initSchema. - // The test container at `bun run ci:local` provides a fresh DB; outside that - // path we rely on the caller having set DATABASE_URL to a fresh DB. + // Postgres side: ensure the test database is FRESH before initSchema. + // v0.37.2.0 fix: previously the test trusted the caller to pass a fresh + // DATABASE_URL, but `gbrain doctor` (used by the CLAUDE.md E2E bootstrap + // ritual) populates `content_chunks.model DEFAULT` from the configured + // gateway model. On a re-run, `CREATE TABLE IF NOT EXISTS` is a no-op so + // the stale default sticks while PGLite (always fresh-in-memory) gets the + // engine fallback. That produced a phantom drift unrelated to schema + // parity. + // + // SAFETY GATE (codex P0, tightened in v0.37.2.0): DROP SCHEMA public CASCADE is + // destructive. The db name MUST always look test-shaped — no env-var override + // bypasses that floor. GBRAIN_TEST_DB=1 only relaxes the localhost requirement + // so CI environments where the host is a service name (e.g. "postgres") can + // still reset. If the db name doesn't match the test pattern, nothing nukes it. pg = new PostgresEngine(); await pg.connect({ database_url: DATABASE_URL! }); + const pgConnPre = (pg as any).sql; + + const url = new URL(DATABASE_URL!); + const dbName = url.pathname.replace(/^\//, ''); + const host = url.hostname; + const isLocalhost = host === 'localhost' || host === '127.0.0.1' || host.endsWith('.local'); + // db-name pattern is the floor: gbrain_test, *_test, test_*, *_e2e. + // Required REGARDLESS of any override — a production db named "production_data" + // cannot be reset even with GBRAIN_TEST_DB=1. + const looksLikeTestDb = /^(gbrain_test|.*_test|test_.*|.*_e2e)$/i.test(dbName); + const ciOptIn = process.env.GBRAIN_TEST_DB === '1'; + // resetAllowed semantics: db name is test-shaped AND (localhost OR ci-opt-in). + // Neither host nor env-var alone is sufficient. + const resetAllowed = looksLikeTestDb && (isLocalhost || ciOptIn); + + if (resetAllowed) { + await pgConnPre.unsafe('DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public;'); + } else { + // Surface a loud, paste-ready hint. The test will still try initSchema; + // if the caller already had a fresh DB the parity check passes anyway. + const reason = !looksLikeTestDb + ? `db name "${dbName}" doesn't match the test pattern (gbrain_test, *_test, test_*, *_e2e). ` + + `GBRAIN_TEST_DB=1 does NOT override this — db name is the hard floor.` + : `host="${host}" is non-local AND GBRAIN_TEST_DB=1 is not set. ` + + `Set GBRAIN_TEST_DB=1 to allow non-local hosts (e.g. CI service names) — ` + + `but only when the db name is already test-shaped.`; + console.warn(`[schema-drift] Skipping DROP SCHEMA — ${reason}`); + } + await pg.initSchema(); // Snapshot both. PGLite returns `{rows}`, postgres.js returns the array. diff --git a/test/migrate.test.ts b/test/migrate.test.ts index 89b6be99e..d41f28af7 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -1610,3 +1610,189 @@ describe('resolveSessionTimeouts — env var overrides', () => { expect(Object.keys(t)).toHaveLength(0); }); }); + +// ─── v0.37.2.0 — v80 takes_unresolvable_quality_v0_37_2_0 ────────────────── +// +// Hotfix that unblocks the production grading script. Widens BOTH: +// (a) the table-level takes_resolution_consistency CHECK to accept +// quality='unresolvable' AND outcome=NULL +// (b) the column-level CHECK on resolved_quality to allow 'unresolvable' +// Structural assertions only — round-trip behavior is covered by E2E. +// Renumbered v74→v79→v80 during successive master merges (autonomous- +// remediation wave claimed v68-v78, then v0.37.1.0 claimed v79). + +describe('migrate v80 — takes_unresolvable_quality_v0_37_2_0', () => { + const v80 = MIGRATIONS.find(m => m.version === 80); + + test('v80 entry exists with the documented name', () => { + expect(v80).toBeDefined(); + expect(v80!.name).toBe('takes_unresolvable_quality_v0_37_2_0'); + }); + + test('v80 is marked idempotent so re-runs are safe', () => { + expect(v80!.idempotent).toBe(true); + }); + + test("v80 widens the column-level CHECK to include 'unresolvable'", () => { + const sql = (v80!.sql ?? '').toLowerCase(); + // Drops both possible names (auto-generated + explicitly-named) so + // pre-v80 brains converge regardless of which CHECK shape they had. + expect(sql).toContain('drop constraint if exists takes_resolved_quality_check'); + expect(sql).toContain('drop constraint if exists takes_resolved_quality_values'); + // The new CHECK enumerates all four valid quality states. + expect(sql).toContain('takes_resolved_quality_values'); + expect(sql).toMatch(/'correct'/); + expect(sql).toMatch(/'incorrect'/); + expect(sql).toMatch(/'partial'/); + expect(sql).toMatch(/'unresolvable'/); + }); + + test('v80 widens the table-level takes_resolution_consistency CHECK', () => { + const sql = v80!.sql ?? ''; + expect(sql).toContain('DROP CONSTRAINT IF EXISTS takes_resolution_consistency'); + expect(sql).toContain('ADD CONSTRAINT takes_resolution_consistency CHECK'); + // The new (quality, outcome) row for unresolvable joins partial as + // null-outcome. Pin the literal pair so regressions surface. + expect(sql).toMatch(/resolved_quality\s*=\s*'unresolvable'\s+AND\s+resolved_outcome\s+IS\s+NULL/i); + }); + + test('v80 keeps the existing four (quality, outcome) pairs intact', () => { + const sql = v80!.sql ?? ''; + // Regression: shouldn't accidentally drop pre-existing legal states. + expect(sql).toMatch(/resolved_quality\s+IS\s+NULL\s+AND\s+resolved_outcome\s+IS\s+NULL/i); + expect(sql).toMatch(/resolved_quality\s*=\s*'correct'\s+AND\s+resolved_outcome\s*=\s*true/i); + expect(sql).toMatch(/resolved_quality\s*=\s*'incorrect'\s+AND\s+resolved_outcome\s*=\s*false/i); + expect(sql).toMatch(/resolved_quality\s*=\s*'partial'\s+AND\s+resolved_outcome\s+IS\s+NULL/i); + }); +}); + +// E2E round-trip — runs against PGLite (no DATABASE_URL needed). Spins up a +// fresh in-memory brain, applies all migrations through v80, then exercises +// the regression checklist: R1 unresolvable persists, R2 pre-v80 (NULL,NULL) +// rows survive, R3+R4 contradictory pairs still rejected, R5 the four legal +// shapes all round-trip. +describe('migrate v80 — CHECK widening end-to-end on PGLite', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + async function insertTake(rowNum: number): Promise { + // Need a page row to satisfy the FK before we can write a take. + const slug = `wiki/people/v80-test-${rowNum}-${Math.random().toString(36).slice(2, 8)}`; + const page = await engine.putPage(slug, { + type: 'person', + title: `v80 test row ${rowNum}`, + compiled_truth: '', + timeline: '', + frontmatter: {}, + content_hash: `v80-${rowNum}-${Math.random()}`, + }); + await engine.executeRaw( + `INSERT INTO takes (page_id, row_num, claim, kind, holder, weight, active) + VALUES ($1, $2, $3, $4, $5, $6, $7)`, + [page.id, rowNum, `claim ${rowNum}`, 'bet', 'garry', 0.5, true], + ); + return page.id; + } + + async function tryResolve( + pageId: number, + rowNum: number, + quality: string, + outcome: boolean | null, + ): Promise<{ ok: true } | { ok: false; err: string }> { + try { + await engine.executeRaw( + `UPDATE takes + SET resolved_at = now(), + resolved_quality = $1::text, + resolved_outcome = $2, + resolved_by = $3 + WHERE page_id = $4 AND row_num = $5`, + [quality, outcome, 'gbrain:test', pageId, rowNum], + ); + return { ok: true }; + } catch (err) { + return { ok: false, err: err instanceof Error ? err.message : String(err) }; + } + } + + test('R1: writing quality=unresolvable + outcome=NULL succeeds post-v80', async () => { + const pageId = await insertTake(101); + const result = await tryResolve(pageId, 101, 'unresolvable', null); + expect(result.ok).toBe(true); + + const row = (await engine.executeRaw<{ resolved_quality: string; resolved_outcome: boolean | null }>( + `SELECT resolved_quality, resolved_outcome FROM takes WHERE page_id = $1 AND row_num = $2`, + [pageId, 101], + ))[0]; + expect(row.resolved_quality).toBe('unresolvable'); + expect(row.resolved_outcome).toBeNull(); + }); + + test('R2: pre-v80 row with quality=NULL AND outcome=NULL survives widened CHECK', async () => { + // After v80 ran on initSchema, inserting an unresolved take must still + // succeed — the (NULL, NULL) case is still legal. + const pageId = await insertTake(102); + const row = (await engine.executeRaw<{ resolved_quality: string | null; resolved_outcome: boolean | null }>( + `SELECT resolved_quality, resolved_outcome FROM takes WHERE page_id = $1 AND row_num = $2`, + [pageId, 102], + ))[0]; + expect(row.resolved_quality).toBeNull(); + expect(row.resolved_outcome).toBeNull(); + }); + + test('R3 negative: quality=partial AND outcome=true STILL rejected', async () => { + const pageId = await insertTake(103); + const result = await tryResolve(pageId, 103, 'partial', true); + expect(result.ok).toBe(false); + expect((result as { ok: false; err: string }).err.toLowerCase()).toContain('check'); + }); + + test('R4 negative: quality=unresolvable AND outcome=true STILL rejected', async () => { + // The widened CHECK admits unresolvable only with NULL outcome — same + // shape as partial. A truthy outcome remains illegal. + const pageId = await insertTake(104); + const result = await tryResolve(pageId, 104, 'unresolvable', true); + expect(result.ok).toBe(false); + expect((result as { ok: false; err: string }).err.toLowerCase()).toContain('check'); + }); + + test('R4 negative: quality=unresolvable AND outcome=false STILL rejected', async () => { + const pageId = await insertTake(105); + const result = await tryResolve(pageId, 105, 'unresolvable', false); + expect(result.ok).toBe(false); + expect((result as { ok: false; err: string }).err.toLowerCase()).toContain('check'); + }); + + test('R5: getScorecard surfaces unresolvable_count + unresolvable_rate as siblings', async () => { + // Seed three rows: one correct, one partial, one unresolvable. All under + // the same holder so the scorecard groups them. unresolvable_rate must + // come out as 1 / (2 + 1) since `resolved` stays 3-state (correct+partial). + const pid1 = await insertTake(201); + const pid2 = await insertTake(202); + const pid3 = await insertTake(203); + expect((await tryResolve(pid1, 201, 'correct', true)).ok).toBe(true); + expect((await tryResolve(pid2, 202, 'partial', null)).ok).toBe(true); + expect((await tryResolve(pid3, 203, 'unresolvable', null)).ok).toBe(true); + + const scorecard = await engine.getScorecard({ holder: 'garry' }, undefined); + // Three rows total, 2 in the 3-state subset, 1 unresolvable. + expect(scorecard.resolved).toBeGreaterThanOrEqual(2); + expect(scorecard.unresolvable_count).toBeGreaterThanOrEqual(1); + // unresolvable_rate is computed against the 4-state denominator. + expect(scorecard.unresolvable_rate).not.toBeNull(); + expect(scorecard.unresolvable_rate!).toBeGreaterThan(0); + // The legacy fields keep their pre-v80 meaning. + expect(typeof scorecard.accuracy).not.toBe('undefined'); + expect(typeof scorecard.partial_rate).not.toBe('undefined'); + }); +}); diff --git a/test/take-forecast.test.ts b/test/take-forecast.test.ts index 49cfa25e7..6fe724d6f 100644 --- a/test/take-forecast.test.ts +++ b/test/take-forecast.test.ts @@ -32,6 +32,8 @@ function buildScorecard(opts: { resolved: number; brier: number | null }): Takes accuracy: 0.6, brier: opts.brier, partial_rate: 0, + unresolvable_count: 0, + unresolvable_rate: null, }; } diff --git a/test/takes-resolution.test.ts b/test/takes-resolution.test.ts index cabeb5ec7..5b8972492 100644 --- a/test/takes-resolution.test.ts +++ b/test/takes-resolution.test.ts @@ -70,6 +70,26 @@ describe('deriveResolutionTuple', () => { deriveResolutionTuple({ resolvedBy: 'garry' }) ).toThrow(GBrainError); }); + + // v0.36.1.1 R1-class: 'unresolvable' joins partial as null-outcome. + test('R1: quality=unresolvable → (unresolvable, null)', () => { + expect(deriveResolutionTuple({ quality: 'unresolvable', resolvedBy: 'garry' })).toEqual({ + quality: 'unresolvable', + outcome: null, + }); + }); + + test('R4-helper: unresolvable + outcome=true is contradictory (throws)', () => { + // Defense-in-depth: deriveResolutionTuple rejects contradictory inputs + // before the schema CHECK fires. unresolvable requires null outcome — + // same shape as partial. + expect(() => + deriveResolutionTuple({ quality: 'unresolvable', outcome: true, resolvedBy: 'garry' }) + ).toThrow(GBrainError); + expect(() => + deriveResolutionTuple({ quality: 'unresolvable', outcome: false, resolvedBy: 'garry' }) + ).toThrow(GBrainError); + }); }); describe('finalizeScorecard (Brier math)', () => { @@ -86,6 +106,8 @@ describe('finalizeScorecard (Brier math)', () => { accuracy: null, brier: null, partial_rate: null, + unresolvable_count: 0, + unresolvable_rate: null, }); }); @@ -155,4 +177,65 @@ describe('finalizeScorecard (Brier math)', () => { }); expect(card.partial_rate).toBe(0); }); + + // v0.36.1.1 R5 — sibling-field semantics. `resolved` MUST keep the v0.36.1.0 + // 3-state meaning so historical comparisons stay valid; unresolvable rows + // live in the new sibling fields only. + test('R5: unresolvable_count + unresolvable_rate populated; resolved/accuracy/partial_rate unchanged', () => { + // 2 correct, 1 incorrect, 1 partial, 1 unresolvable. resolved=4 (3-state), + // unresolvable_count=1. unresolvable_rate = 1 / (4 + 1) = 0.2. + const card = finalizeScorecard({ + total_bets: 5, + resolved: 4, + correct: 2, + incorrect: 1, + partial: 1, + brier: 0.2, + unresolvable_count: 1, + }); + expect(card.resolved).toBe(4); // 3-state, NOT 5 + expect(card.unresolvable_count).toBe(1); + expect(card.unresolvable_rate).toBeCloseTo(0.2, 5); + // accuracy unchanged: correct / (correct + incorrect) = 2/3 + expect(card.accuracy).toBeCloseTo(2 / 3, 5); + // partial_rate unchanged: partial / resolved (3-state denominator) = 1/4 + expect(card.partial_rate).toBe(0.25); + // brier unchanged + expect(card.brier).toBe(0.2); + }); + + test('R5: legacy v0.36.1.0 ScorecardRowRaw (no unresolvable_count) backfills count=0; rate=0 when resolved>0', () => { + // Engines pre-v0.36.1.1 won't return unresolvable_count in the row. The + // optional `?` on ScorecardRowRaw + default-0 in finalizeScorecard means + // existing brains keep working. unresolvable_rate matches partial_rate + // semantics: 0 when there are resolved rows but no unresolvable ones. + const card = finalizeScorecard({ + total_bets: 3, resolved: 3, correct: 2, incorrect: 1, partial: 0, brier: 0.18, + }); + expect(card.unresolvable_count).toBe(0); + expect(card.unresolvable_rate).toBe(0); + }); + + test('R5: empty brain (resolved=0 AND unresolvable_count=0) → unresolvable_rate=null', () => { + // Only when BOTH counts are zero do we return null (avoid divide-by-zero + // and signal "no data" rather than "0% unresolvable on no data"). + const card = finalizeScorecard({ + total_bets: 0, resolved: 0, correct: 0, incorrect: 0, partial: 0, brier: null, + }); + expect(card.unresolvable_count).toBe(0); + expect(card.unresolvable_rate).toBeNull(); + }); + + test('R5: brain with only unresolvable rows surfaces unresolvable_rate=1.0', () => { + const card = finalizeScorecard({ + total_bets: 5, resolved: 0, correct: 0, incorrect: 0, partial: 0, brier: null, + unresolvable_count: 5, + }); + expect(card.resolved).toBe(0); + expect(card.unresolvable_count).toBe(5); + expect(card.unresolvable_rate).toBe(1.0); + expect(card.accuracy).toBeNull(); + expect(card.brier).toBeNull(); + expect(card.partial_rate).toBeNull(); // resolved is 0 → null per existing contract + }); });