diff --git a/CHANGELOG.md b/CHANGELOG.md index ca9ef4203..3ad00407f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,129 @@ All notable changes to GBrain will be documented in this file. +## [0.30.0] - 2026-05-07 + +**Calibration scorecards land. Find out if your bets are actually as good as you think.** +**Brier scores, prediction accuracy, calibration curves — for every bet you ever wrote down.** + +This is the v0.30 wave's first ship: the calibration core. Resolve your bets with three +states (correct, incorrect, partial) instead of just true/false. Run a scorecard against +any holder, any domain prefix, any date window, and see your accuracy + Brier score in +seconds. The calibration curve shows whether your "I'm 80% confident" bets actually come +in at 80% — the thing Daniel Kahneman spent a career studying, now a query. + +```bash +gbrain takes resolve companies/some-yc-co --row 3 --quality correct --evidence "Series A closed at $50M" +gbrain takes scorecard garry +gbrain takes calibration garry --bucket-size 0.1 +``` + +The scorecard prints `correct/incorrect/partial`, accuracy, Brier (lower is better; 0.25 +is the always-50% baseline), and a `partial_rate` line. When more than 20% of resolved +bets are "partial," the scorecard prints `[!] partial_rate is high — calibration may be +optimistic`. Hedged bets escape the Brier denominator, so the warning surfaces hedging +behavior as its own signal even though it doesn't enter the math. + +### The math that matters + +| Field | Meaning | Math | +|-------|---------|------| +| `accuracy` | correct / (correct + incorrect) | partial excluded | +| `brier` | mean((weight − outcome)²) over correct ∨ incorrect | lower is better; 0 = perfect; 0.25 = always-50% baseline | +| `partial_rate` | partial / resolved | hedging signal; warns at >20% | + +Brier excludes partial deliberately. Codex pointed out that counting partial as 0.5 in +the math would reward hedging into ambiguity; instead, partial bets leave the calibration +denominator entirely AND surface as a separate counter. You can see whether you're +miscalibrated AND whether you're hedging, side by side. + +### Privacy: aggregate queries fail-closed + +Scorecard and calibration are MCP-callable read ops (`takes_scorecard`, +`takes_calibration`). Both engine methods take `takesHoldersAllowList` as a REQUIRED +TypeScript parameter — the compiler is the first line of defense against accidentally +exposing therapy-page takes to an MCP-bound agent via aggregate counts. Hidden-holder +rows contribute zero to the math. Local CLI callers see all holders. + +### Markdown stays canonical (and the silent-data-loss bug is dead) + +Resolution metadata renders into the takes-fence on disk: `resolved | quality | +evidence | value | unit | by` columns appear ONLY when at least one row on the page is +resolved. Pages with no resolved rows keep the narrow 7-column shape exactly as before. + +A codex consult on the v0.30 plan caught a real bug in the original draft: the v0.28 +parser/renderer had no concept of resolution columns. Without an explicit fix, every +`gbrain takes update` after a `takes resolve` would silently delete the resolution +data on the next disk write. The new round-trip preservation tests +(`test/takes-fence.test.ts` v0.30 section) are the regression gate; they fail loudly +if the data-loss bug returns. + +### To take advantage of v0.30.0 + +`gbrain upgrade` runs the schema migration automatically. If `gbrain doctor` warns +about a partial migration: + +1. **Run the orchestrator manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Resolve a bet you already have:** + ```bash + gbrain takes resolve --row N --quality correct|incorrect|partial --evidence "..." + ``` +3. **Run your first scorecard:** + ```bash + gbrain takes scorecard garry + ``` +4. **Existing v0.28-resolved bets are auto-backfilled.** The migration maps legacy + `resolved_outcome=true` → `resolved_quality='correct'`, `false` → `'incorrect'`. + No manual reclassification needed. +5. **The `--outcome true|false` flag still works** as a back-compat alias on `takes + resolve`, with a stderr deprecation warning. Cannot express partial; mutually + exclusive with `--quality`. + +If any step fails or the numbers look wrong, file an issue: +https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and +`~/.gbrain/upgrade-errors.jsonl` if it exists. + +### Itemized changes + +#### Added +- New CLI: `gbrain takes scorecard [] [--domain ] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--json]` — calibration scorecard. +- New CLI: `gbrain takes calibration [] [--bucket-size 0.1] [--json]` — calibration curve binned by stated weight. +- New flag: `--quality correct|incorrect|partial` on `gbrain takes resolve`. Primary input. +- New flag: `--evidence "..."` on `gbrain takes resolve` — semantic alias for `--source`. +- New MCP ops: `takes_scorecard` (read scope) + `takes_calibration` (read scope), both honoring per-token allow-list. +- Schema migration v43 (`takes_resolved_quality_and_drift_decisions`): + - `takes.resolved_quality TEXT` with CHECK `(IN ('correct','incorrect','partial'))`. + - `takes_resolution_consistency` CHECK constraint enforces (quality, outcome) tuple consistency at the DB layer. + - Backfill: legacy `resolved_outcome=true` → `resolved_quality='correct'`; `false` → `'incorrect'`. + - Partial index `idx_takes_scorecard ON takes (holder, kind, resolved_quality) WHERE resolved_quality IS NOT NULL` — scorecard hot path. + - New table `drift_decisions` (audit log for the upcoming v0.30.3 drift LLM judge; defined now so C1 carries no migration). +- New module `src/core/takes-resolution.ts` — pure helpers (`deriveResolutionTuple`, `finalizeScorecard`, `PARTIAL_RATE_WARNING_THRESHOLD`) shared between Postgres + PGLite engines. + +#### Changed +- `gbrain takes resolve` now writes both `resolved_outcome` (boolean) and `resolved_quality` (text) on every resolution. Schema CHECK is the defense-in-depth backstop. +- `--outcome true|false` becomes a back-compat alias with a stderr deprecation warning. Mutually exclusive with `--quality`. Cannot express partial. +- `Take` interface (`src/core/engine.ts`) adds `resolved_quality: 'correct' | 'incorrect' | 'partial' | null`. +- `TakeResolution` adds optional `quality` field. `outcome` stays as the back-compat input. When both set and inconsistent, throws `TAKE_RESOLUTION_INVALID` instead of silently overwriting. +- `BrainEngine` interface gains `getScorecard(opts, allowList)` and `getCalibrationCurve(opts, allowList)`. Both implemented in `postgres-engine.ts` + `pglite-engine.ts` with SQL-level `WHERE holder = ANY($allowList)` filtering inside the GROUP BY. +- `ParsedTake` (`src/core/takes-fence.ts`) extended with optional `resolvedAt`, `resolvedQuality`, `resolvedOutcome`, `resolvedEvidence`, `resolvedValue`, `resolvedUnit`, `resolvedBy`. Parser detects v0.30-shape headers; renderer emits resolution columns only when at least one row has `resolvedQuality !== undefined`. +- `cmdResolve` mirrors resolution metadata into the takes-fence on disk via the page-lock-aware path. Round-trip preservation keeps resolution data intact across unrelated edits. + +#### Tests +53 new cases (34 unit + 19 E2E): +- 11 cases in `test/takes-fence.test.ts` for v0.30 resolution columns, round-trip preservation (the codex F3 regression gate), narrow-shape preservation, partial rendering, `upsertTakeRow` and `supersedeRow` resolution preservation. +- 16 cases in `test/takes-resolution.test.ts` for `deriveResolutionTuple` and `finalizeScorecard` Brier math (n=0, hand-calculated 4-bet reference at Brier=0.205, partial-exclusion contract, partial_rate threshold). +- 7 cases in `test/takes-engine.test.ts` for v0.30 quality semantics on PGLite, contradictory-input rejection, scorecard hand-calc against the same 4-bet fixture, n=0 handling, SQL-level allow-list privacy filter. +- 11 cases in `test/e2e/takes-scorecard-parity.test.ts` (NEW) — same fixture into Postgres + PGLite, asserts `getScorecard` and `getCalibrationCurve` return byte-identical results across engines, and that the SQL-level allow-list filter strictly subtracts hidden-holder rows on both engines. +- 8 cases extended in `test/e2e/takes-postgres.test.ts` — `--quality correct/partial/back-compat` writes the expected (quality, outcome) tuple on real PG; the `takes_resolution_consistency` CHECK constraint actively rejects contradictory raw `UPDATE`s; scorecard + calibration coherent shape; PRIVACY allow-list filter on real PG; MCP dispatch path for `takes_scorecard` + `takes_calibration`. + +The E2E parity test caught two real bugs PGLite tolerated: postgres.js was sending `${bucketSize}` as a string and `FLOOR(weight / '0.1')` was throwing `invalid input syntax for type integer`; even after the type cast, IEEE 754 made `FLOOR(0.7 / 0.1)` return 6 on real PG and 7 on PGLite, so calibration buckets diverged at boundaries. Both fixed with `weight::numeric / $N::numeric` (exact decimal arithmetic, engine-agnostic). + +#### Migration ordering +- All wave schema (resolved_quality, CHECK, partial index, drift_decisions table) bundled into migration v43 in v0.30.0 (renumbered from v40 on master merge — master claimed v40-v42 with the v0.29 + v0.29.1 salience-and-recency wave). Migration runner sorts by version number, so the rename is mechanical — no semantic change. Slices A2/B1/C1 (trajectory + annual-review, meeting extraction CLI, drift judge) add no migrations of their own. + ## [0.29.2] - 2026-05-07 **Thin-client mode: install gbrain on a laptop without a local DB and have it consume a remote brain over MCP.** @@ -542,9 +665,6 @@ gbrain eval longmemeval ~/datasets/longmemeval/longmemeval_s.json \ If anything looks off, file at https://github.com/garrytan/gbrain/issues with `gbrain doctor` output. - - - ## [0.28.11] - 2026-05-07 **Mix providers: OpenAI for text, Voyage for images. One brain, two embedding pipelines.** diff --git a/VERSION b/VERSION index 20f068700..c25c8e5b7 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.29.2 +0.30.0 diff --git a/package.json b/package.json index b7b4cd954..0ba23107d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.29.2", + "version": "0.30.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/commands/takes.ts b/src/commands/takes.ts index d945d7711..9c47bb472 100644 --- a/src/commands/takes.ts +++ b/src/commands/takes.ts @@ -305,29 +305,206 @@ async function cmdSupersede(engine: BrainEngine, args: string[]): Promise async function cmdResolve(engine: BrainEngine, args: string[]): Promise { const slug = args[0]; const rowNumStr = flagValue(args, '--row'); + const qualityStr = flagValue(args, '--quality'); const outcomeStr = flagValue(args, '--outcome'); - if (!slug || !rowNumStr || !outcomeStr) { - console.error('Usage: gbrain takes resolve --row N --outcome true|false [--value N --unit usd|pct|count] [--source "..."] [--by ]'); + if (!slug || !rowNumStr || (!qualityStr && !outcomeStr)) { + console.error('Usage: gbrain takes resolve --row N --quality correct|incorrect|partial [--evidence "..."] [--value N --unit usd|pct|count] [--by ]'); + console.error(' (back-compat) gbrain takes resolve --row N --outcome true|false [...]'); + process.exit(1); + } + if (qualityStr && outcomeStr) { + console.error('Error: --quality and --outcome are mutually exclusive (choose one).'); process.exit(1); } const rowNum = parseInt(rowNumStr, 10); - const outcome = outcomeStr === 'true'; + + // v0.30.0: --quality is the new primary input. --outcome stays as a back-compat + // alias auto-mapping true→correct / false→incorrect; cannot express partial. + let quality: 'correct' | 'incorrect' | 'partial' | undefined; + let outcome: boolean | undefined; + if (qualityStr) { + if (qualityStr !== 'correct' && qualityStr !== 'incorrect' && qualityStr !== 'partial') { + console.error(`Invalid --quality "${qualityStr}". Expected: correct, incorrect, partial.`); + process.exit(1); + } + quality = qualityStr; + } else if (outcomeStr) { + if (outcomeStr !== 'true' && outcomeStr !== 'false') { + console.error(`Invalid --outcome "${outcomeStr}". Expected: true or false.`); + process.exit(1); + } + outcome = outcomeStr === 'true'; + console.error('[deprecated] --outcome is the v0.28 alias for --quality. Prefer --quality correct|incorrect|partial in new scripts.'); + } + const valueStr = flagValue(args, '--value'); const value = valueStr === undefined ? undefined : parseFloat(valueStr); const unit = flagValue(args, '--unit'); - const source = flagValue(args, '--source'); + // --evidence is the v0.30.0 alias for --source on the resolve subcommand + // (semantic clarity: "what evidence resolved this bet?"). + const source = flagValue(args, '--evidence') ?? flagValue(args, '--source'); const resolvedBy = flagValue(args, '--by') ?? 'garry'; + const dirArg = flagValue(args, '--dir'); const pageId = await getPageId(engine, slug); await engine.resolveTake(pageId, rowNum, { + quality, outcome, value, unit, source, resolvedBy, }); - console.log(`Resolved take #${rowNum} on ${slug}: outcome=${outcome}${valueStr ? ` value=${value}${unit ? ` ${unit}` : ''}` : ''}.`); - console.log(`(Markdown rendering of resolution metadata: deferred to v0.29 — DB stores it; takes-fence renderer doesn't yet surface resolved_* in the table.)`); + + // Mirror resolution into the markdown fence so the page is self-describing. + // The renderer conditionally widens the table to 13 columns when at least one + // row has resolution data; pages with no resolved rows keep the 7-col shape. + // Round-trip via parseTakesFence + renderTakesFence preserves all rows. + const brainDir = await resolveBrainDir(engine, dirArg ?? null); + await withPageLock(slug, async () => { + const path = pageFilePath(brainDir, slug); + const body = readBodyOrEmpty(path); + if (!body) { + console.warn(`[takes resolve] markdown file not found at ${path}; DB updated but on-disk page absent.`); + return; + } + const { parseTakesFence, renderTakesFence, TAKES_FENCE_BEGIN, TAKES_FENCE_END } = await import('../core/takes-fence.ts'); + const parsed = parseTakesFence(body); + const target = parsed.takes.find(t => t.rowNum === rowNum); + if (!target) { + console.warn(`[takes resolve] DB updated but row #${rowNum} not in markdown fence; run 'gbrain extract takes --slugs ${slug}' to reconcile.`); + return; + } + // Derive resolved fields from the inputs. Mirror the engine semantics: + // quality wins when both set; partial → outcome=null. + const finalQuality = quality ?? (outcome === true ? 'correct' : outcome === false ? 'incorrect' : undefined); + if (!finalQuality) return; // unreachable — covered by earlier validation + const finalOutcome = finalQuality === 'partial' ? undefined + : finalQuality === 'correct' ? true : false; + const updated = { + ...target, + resolvedAt: new Date().toISOString().slice(0, 10), + resolvedQuality: finalQuality, + resolvedOutcome: finalOutcome, + resolvedEvidence: source, + resolvedValue: value, + resolvedUnit: unit, + resolvedBy, + }; + const allRows = parsed.takes.map(t => t.rowNum === rowNum ? updated : t); + const newFence = renderTakesFence(allRows); + const beginIdx = body.indexOf(TAKES_FENCE_BEGIN); + const endIdx = body.indexOf(TAKES_FENCE_END, beginIdx + TAKES_FENCE_BEGIN.length); + const out = body.slice(0, beginIdx) + newFence + body.slice(endIdx + TAKES_FENCE_END.length); + writeBody(path, out); + }); + + const finalQuality = quality ?? (outcome === true ? 'correct' : outcome === false ? 'incorrect' : 'unknown'); + const valueSummary = valueStr ? ` value=${value}${unit ? ` ${unit}` : ''}` : ''; + console.log(`Resolved take #${rowNum} on ${slug}: quality=${finalQuality}${valueSummary}.`); +} + +/** + * v0.30.0: aggregate calibration scorecard for a holder. + * + * Brier scope (D5+D11): partial bets are excluded from Brier — partial + * isn't a binary outcome to compare a probability against. The partial_rate + * counter reports the rate as a separate signal so hedging behavior stays + * visible even though it doesn't enter the calibration math. When the rate + * exceeds 20% the CLI prints a warning line; calibration on a hedge-heavy + * scorecard is artificially clean, and the user should know. + */ +async function cmdScorecard(engine: BrainEngine, args: string[]): Promise { + const json = flagPresent(args, '--json'); + const holder = args[0] && !args[0].startsWith('--') ? args[0] : flagValue(args, '--holder'); + const domainPrefix = flagValue(args, '--domain'); + const since = flagValue(args, '--since'); + const until = flagValue(args, '--until'); + const { PARTIAL_RATE_WARNING_THRESHOLD } = await import('../core/takes-resolution.ts'); + + const card = await engine.getScorecard( + { holder, domainPrefix, since, until }, + /* allowList */ undefined, // CLI is local + trusted; MCP path threads allowList from the caller + ); + + if (json) { + console.log(JSON.stringify(card, null, 2)); + return; + } + + if (card.resolved === 0) { + console.log(`No resolved bets yet${holder ? ` for ${holder}` : ''}.`); + return; + } + const fmt = (n: number | null, digits = 3) => n === null ? '—' : n.toFixed(digits); + console.log(`# Scorecard${holder ? ` — ${holder}` : ''}`); + if (domainPrefix) console.log(`Scope: domain=${domainPrefix}`); + if (since || until) console.log(`Window: ${since ?? 'all'} → ${until ?? 'now'}`); + console.log(''); + console.log(` total bets: ${card.total_bets}`); + console.log(` resolved: ${card.resolved}`); + console.log(` correct: ${card.correct}`); + console.log(` incorrect: ${card.incorrect}`); + console.log(` partial: ${card.partial}`); + console.log(` accuracy: ${fmt(card.accuracy)}`); + console.log(` Brier: ${fmt(card.brier, 4)} (correct ∨ incorrect only; lower is better; 0.25 = always-50% baseline)`); + console.log(` partial_rate: ${fmt(card.partial_rate)}`); + if (card.partial_rate !== null && card.partial_rate > PARTIAL_RATE_WARNING_THRESHOLD) { + console.log(''); + console.log(` [!] partial_rate is high (>${(PARTIAL_RATE_WARNING_THRESHOLD * 100).toFixed(0)}%) — calibration may be optimistic.`); + console.log(` Hedged bets escape the Brier denominator. Resolve them more decisively if the data supports it.`); + } + if (card.resolved < 100) { + console.log(''); + console.log(` Note: n=${card.resolved} is small. Brier is noisy below ~100 resolved bets.`); + } +} + +/** + * v0.30.0: calibration curve. Bins resolved correct+incorrect bets by stated + * weight and reports observed vs predicted frequency per bucket. The diagonal + * (observed ≈ predicted in every bucket) is perfect calibration. + */ +async function cmdCalibration(engine: BrainEngine, args: string[]): Promise { + const json = flagPresent(args, '--json'); + const holder = args[0] && !args[0].startsWith('--') ? args[0] : flagValue(args, '--holder'); + const bucketSizeStr = flagValue(args, '--bucket-size'); + const bucketSize = bucketSizeStr === undefined ? 0.1 : parseFloat(bucketSizeStr); + if (!Number.isFinite(bucketSize) || bucketSize <= 0 || bucketSize > 1) { + console.error(`Invalid --bucket-size "${bucketSizeStr}". Expected a number in (0, 1].`); + process.exit(1); + } + + const buckets = await engine.getCalibrationCurve( + { holder, bucketSize }, + /* allowList */ undefined, + ); + + if (json) { + console.log(JSON.stringify(buckets, null, 2)); + return; + } + + if (buckets.length === 0) { + console.log(`No resolved correct/incorrect bets yet${holder ? ` for ${holder}` : ''}.`); + return; + } + console.log(`# Calibration curve${holder ? ` — ${holder}` : ''}`); + console.log(`Bucket size: ${bucketSize}`); + console.log(''); + console.log(` bucket n observed predicted delta`); + console.log(` --------------- ----- --------- ---------- -------`); + const fmt = (n: number | null) => n === null ? ' —' : n.toFixed(3); + for (const b of buckets) { + const range = `${b.bucket_lo.toFixed(2)}-${b.bucket_hi.toFixed(2)}`.padEnd(15); + const nStr = String(b.n).padStart(5); + const obs = fmt(b.observed).padStart(9); + const pred = fmt(b.predicted).padStart(10); + const delta = b.observed !== null && b.predicted !== null + ? (b.observed - b.predicted).toFixed(3).padStart(7) + : ' —'; + console.log(` ${range} ${nStr} ${obs} ${pred} ${delta}`); + } } // --- Dispatcher --- @@ -348,8 +525,14 @@ Subcommands: Update mutable fields takes supersede --row N --claim "..." [--kind k] [--who h] [--weight 0.5] [--source "..."] Strikethrough old + append new - takes resolve --row N --outcome true|false [--value N --unit usd|pct|count] [--source "..."] [--by ] - Record bet resolution (immutable) + takes resolve --row N --quality correct|incorrect|partial + [--evidence "..."] [--value N --unit usd|pct|count] [--by ] + Record bet resolution (immutable, v0.30.0) + Back-compat: --outcome true|false (deprecated alias) + takes scorecard [] [--domain ] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--json] + Aggregate calibration scorecard (v0.30.0) + takes calibration [] [--bucket-size 0.1] [--json] + Calibration curve binned by stated weight (v0.30.0) Common flags: --dir Override the brain directory (default: sync.repo_path config) @@ -362,11 +545,13 @@ Common flags: const rest = args.slice(1); switch (sub) { - case 'search': return cmdSearch(engine, rest); - case 'add': return cmdAdd(engine, rest); - case 'update': return cmdUpdate(engine, rest); - case 'supersede': return cmdSupersede(engine, rest); - case 'resolve': return cmdResolve(engine, rest); + case 'search': return cmdSearch(engine, rest); + case 'add': return cmdAdd(engine, rest); + case 'update': return cmdUpdate(engine, rest); + case 'supersede': return cmdSupersede(engine, rest); + case 'resolve': return cmdResolve(engine, rest); + case 'scorecard': return cmdScorecard(engine, rest); + case 'calibration': return cmdCalibration(engine, rest); default: // No subcommand keyword → treat first arg as for the list path. return cmdList(engine, args); diff --git a/src/core/engine.ts b/src/core/engine.ts index 6303849a0..4f1a6f774 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -167,6 +167,14 @@ export interface Take { active: boolean; resolved_at: string | null; resolved_outcome: boolean | null; + /** + * v0.30.0: 3-state outcome label. Sits alongside `resolved_outcome` for + * back-compat. New writes populate both; legacy v0.28-resolved rows have + * `resolved_quality` backfilled by migration v40 from the boolean. + * Null on unresolved rows. Schema CHECK enforces (quality, outcome) consistency: + * `correct` ↔ `outcome=true`, `incorrect` ↔ `outcome=false`, `partial` ↔ `outcome=NULL`. + */ + resolved_quality: 'correct' | 'incorrect' | 'partial' | null; resolved_value: number | null; resolved_unit: string | null; resolved_source: string | null; @@ -212,13 +220,73 @@ export interface StaleTakeRow { /** Resolution metadata for resolveTake. */ export interface TakeResolution { - outcome: boolean; + /** + * v0.30.0: primary 3-state input. When set, takes precedence over `outcome` + * and the engine writes both columns (quality directly; outcome derived: + * `correct→true`, `incorrect→false`, `partial→null`). + */ + quality?: 'correct' | 'incorrect' | 'partial'; + /** + * v0.28 back-compat input. Keep submitting for v0.28 callers; the engine + * derives quality (`true→correct`, `false→incorrect`). When `quality` is + * also set, `quality` wins. When neither is set, the engine throws. + * Mutually-exclusive with `quality === 'partial'` because partial isn't + * binary. + */ + outcome?: boolean; value?: number; unit?: string; // 'usd' | 'pct' | 'count' | other source?: string; resolvedBy: string; // slug or 'garry' } +/** v0.30.0: scorecard aggregate. */ +export interface TakesScorecard { + total_bets: number; + resolved: number; + correct: number; + incorrect: number; + partial: number; + /** Accuracy = correct / (correct + incorrect). NULL when n=0. */ + accuracy: number | null; + /** + * Brier score over rows where `resolved_quality IN ('correct','incorrect')`. + * Maps `correct→1`, `incorrect→0`, computes `mean((weight − outcome)²)`. + * Lower is better; 0 = perfect; 0.25 = always-50% baseline. + * Excludes partial — that label hides hedging behavior; `partial_rate` + * surfaces it as a separate signal. NULL when no correct+incorrect rows. + */ + brier: number | null; + /** partial / resolved. NULL when n=0. */ + partial_rate: number | null; +} + +export interface TakesScorecardOpts { + holder?: string; + domainPrefix?: string; // e.g. 'companies/' to scope the scorecard + since?: string; // ISO date 'YYYY-MM-DD' + until?: string; // ISO date 'YYYY-MM-DD' +} + +/** v0.30.0: calibration curve bucket. */ +export interface CalibrationBucket { + /** Lower bound of the weight bucket, inclusive. */ + bucket_lo: number; + /** Upper bound, exclusive (except for the final bucket which is inclusive of 1.0). */ + bucket_hi: number; + /** Count of resolved correct+incorrect bets falling in this weight range. */ + n: number; + /** correct / n. NULL when n=0. */ + observed: number | null; + /** mean(weight) within the bucket — what was predicted on average. NULL when n=0. */ + predicted: number | null; +} + +export interface CalibrationCurveOpts { + holder?: string; + bucketSize?: number; // default 0.1 +} + /** Synthesis evidence row input (provenance from think synthesis pages). */ export interface SynthesisEvidenceInput { synthesis_page_id: number; @@ -559,9 +627,39 @@ export interface BrainEngine { /** * Resolve a bet (or take). Sets resolved_* columns. Immutable: re-resolve * attempts throw `TAKE_ALREADY_RESOLVED`. Use supersede to express a new bet. + * + * v0.30.0: accepts either `quality` (3-state, primary) or `outcome` (boolean, + * back-compat). When both set, `quality` wins. The engine writes BOTH columns + * derived from whichever input was given: `quality='correct'/'incorrect'` → + * `outcome=true/false`; `quality='partial'` → `outcome=NULL`. The schema + * `takes_resolution_consistency` CHECK constraint catches contradictory + * states at the DB layer as a defense-in-depth backstop. */ resolveTake(pageId: number, rowNum: number, resolution: TakeResolution): Promise; + /** + * v0.30.0: aggregate calibration scorecard. Pure SQL aggregation; no LLM. + * Counts resolved bets, computes accuracy, Brier score (correct+incorrect + * only), and `partial_rate`. Filtering: `holder` scopes to one identity; + * `domainPrefix` scopes to a slug-prefix (e.g. `companies/`); `since`/`until` + * scope to a `since_date` window. + * + * Privacy (D4 from plan): `allowList` is REQUIRED in the TS signature. + * The engine applies `WHERE holder = ANY($allowList)` INSIDE the GROUP BY + * so hidden-holder rows contribute zero to aggregates. Pass an empty array + * to enforce zero-results; pass `undefined` only from server-side trusted + * callers that have already verified the request is unrestricted. + */ + getScorecard(opts: TakesScorecardOpts, allowList: string[] | undefined): Promise; + + /** + * v0.30.0: calibration curve. Bins resolved correct+incorrect bets by stated + * weight (default bucket size 0.1) and reports observed vs predicted frequency + * per bucket. Same allow-list contract as `getScorecard`. Excludes partial + * (consistent with Brier — partial has no binary outcome to compare against). + */ + getCalibrationCurve(opts: CalibrationCurveOpts, allowList: string[] | undefined): Promise; + /** Persist think provenance. ON CONFLICT DO NOTHING; returns rows inserted. */ addSynthesisEvidence(rows: SynthesisEvidenceInput[]): Promise; diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 8b6bfd163..aba3ff90d 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -1935,6 +1935,138 @@ export const MIGRATIONS: Migration[] = [ ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_source TEXT; `, }, + { + version: 43, + name: 'takes_resolved_quality_and_drift_decisions', + // v0.30.0 (Slice A1, Universal Takes Epistemology wave). Bundles ALL schema + // for the v0.30 release wave so A2/B1/C1 add no migrations (codex F6 fix: + // schema-first ordering eliminates the cross-lane migrate.ts contention). + // Originally landed as v40 in the v0.30.0 branch; renumbered to v43 on + // merge with master after master claimed v40-v42 with the v0.29 + + // v0.29.1 salience-and-recency wave. Migration runner sorts by version + // number, so renumbering is a pure-rename — no semantic change. + // + // 1. takes.resolved_quality TEXT — 3-state outcome label (correct/incorrect/ + // partial) sitting alongside existing resolved_outcome BOOLEAN. Boolean + // stays for back-compat reads; quality is the new source of truth for + // calibration math. Backfill maps legacy resolved_outcome → quality. + // + // 2. takes_resolution_consistency CHECK constraint — fails contradictory + // states like (quality='correct', outcome=false). 'partial' maps to + // outcome=NULL because partial isn't a binary outcome. Added AFTER the + // backfill so existing rows pass. + // + // 3. idx_takes_scorecard partial index on (holder, kind, resolved_quality) + // WHERE resolved_quality IS NOT NULL — scorecard hot path. ~5KB on a + // 50K-row brain; makes scorecard O(log n) instead of full scan. + // + // 4. drift_decisions audit table — consumed by Slice C1 (v0.30.3) when + // drift LLM judge ships. Defined here so C1 carries no migration. + // Sized for one row per drift recommendation (insert-only, never + // updated except for applied_at/applied_by when --auto-update lands). + sql: ` + -- Step 1: add resolved_quality column with kind-of-outcome CHECK. + -- The (quality, outcome) consistency constraint comes AFTER the backfill + -- (Step 3) so existing legacy rows don't fail the new constraint. + ALTER TABLE takes + ADD COLUMN IF NOT EXISTS resolved_quality TEXT + CHECK (resolved_quality IS NULL OR resolved_quality IN ('correct','incorrect','partial')); + + -- Step 2: backfill from legacy boolean. Idempotent: only writes rows + -- where quality is still NULL and outcome is set. Re-runs are no-ops. + UPDATE takes + SET resolved_quality = CASE resolved_outcome + WHEN true THEN 'correct' + WHEN false THEN 'incorrect' + END + WHERE resolved_outcome IS NOT NULL AND resolved_quality IS NULL; + + -- Step 3: (quality, outcome) consistency constraint. Drop-then-recreate + -- so re-runs converge. The named constraint lets us evolve it later. + ALTER TABLE takes DROP CONSTRAINT IF EXISTS takes_resolution_consistency; + ALTER TABLE takes ADD CONSTRAINT takes_resolution_consistency CHECK ( + (resolved_quality IS NULL AND resolved_outcome IS NULL) + OR (resolved_quality = 'correct' AND resolved_outcome = true) + OR (resolved_quality = 'incorrect' AND resolved_outcome = false) + OR (resolved_quality = 'partial' AND resolved_outcome IS NULL) + ); + + -- Step 4: scorecard hot path. Partial index keeps footprint proportional + -- to resolved-take count, not table size. + CREATE INDEX IF NOT EXISTS idx_takes_scorecard + ON takes (holder, kind, resolved_quality) + WHERE resolved_quality IS NOT NULL; + + -- Step 5: drift_decisions audit table (consumed by Slice C1 in v0.30.3). + CREATE TABLE IF NOT EXISTS drift_decisions ( + id BIGSERIAL PRIMARY KEY, + take_id BIGINT NOT NULL REFERENCES takes(id) ON DELETE CASCADE, + page_id INTEGER NOT NULL, + row_num INTEGER NOT NULL, + recommended_weight REAL NOT NULL CHECK (recommended_weight >= 0 AND recommended_weight <= 1), + reasoning TEXT, + decided_at TIMESTAMPTZ NOT NULL DEFAULT now(), + applied_at TIMESTAMPTZ, + applied_by TEXT + ); + CREATE INDEX IF NOT EXISTS idx_drift_decisions_take ON drift_decisions(take_id); + CREATE INDEX IF NOT EXISTS idx_drift_decisions_decided_at ON drift_decisions(decided_at DESC); + + -- RLS for the new table (Postgres-only — PGLite has no RLS engine). + -- Mirrors the v37 takes/synthesis_evidence pattern: only flip RLS on + -- when running as a BYPASSRLS role so non-BYPASSRLS apps still read. + DO $$ + DECLARE + has_bypass BOOLEAN; + BEGIN + SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user; + IF has_bypass THEN + ALTER TABLE drift_decisions ENABLE ROW LEVEL SECURITY; + END IF; + END $$; + `, + sqlFor: { + // PGLite: same DDL minus the RLS DO-block. Single-tenant by definition. + pglite: ` + ALTER TABLE takes + ADD COLUMN IF NOT EXISTS resolved_quality TEXT + CHECK (resolved_quality IS NULL OR resolved_quality IN ('correct','incorrect','partial')); + + UPDATE takes + SET resolved_quality = CASE resolved_outcome + WHEN true THEN 'correct' + WHEN false THEN 'incorrect' + END + WHERE resolved_outcome IS NOT NULL AND resolved_quality IS NULL; + + ALTER TABLE takes DROP CONSTRAINT IF EXISTS takes_resolution_consistency; + ALTER TABLE takes ADD CONSTRAINT takes_resolution_consistency CHECK ( + (resolved_quality IS NULL AND resolved_outcome IS NULL) + OR (resolved_quality = 'correct' AND resolved_outcome = true) + OR (resolved_quality = 'incorrect' AND resolved_outcome = false) + OR (resolved_quality = 'partial' AND resolved_outcome IS NULL) + ); + + CREATE INDEX IF NOT EXISTS idx_takes_scorecard + ON takes (holder, kind, resolved_quality) + WHERE resolved_quality IS NOT NULL; + + CREATE TABLE IF NOT EXISTS drift_decisions ( + id BIGSERIAL PRIMARY KEY, + take_id BIGINT NOT NULL REFERENCES takes(id) ON DELETE CASCADE, + page_id INTEGER NOT NULL, + row_num INTEGER NOT NULL, + recommended_weight REAL NOT NULL CHECK (recommended_weight >= 0 AND recommended_weight <= 1), + reasoning TEXT, + decided_at TIMESTAMPTZ NOT NULL DEFAULT now(), + applied_at TIMESTAMPTZ, + applied_by TEXT + ); + CREATE INDEX IF NOT EXISTS idx_drift_decisions_take ON drift_decisions(take_id); + CREATE INDEX IF NOT EXISTS idx_drift_decisions_decided_at ON drift_decisions(decided_at DESC); + `, + }, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/operations.ts b/src/core/operations.ts index 1304bb78d..70fc9c546 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -270,9 +270,14 @@ export interface OperationContext { * by the MCP HTTP/stdio dispatch layer from `access_tokens.permissions.takes_holders`. * * When set (i.e., this OperationContext came from an MCP-bound token), - * `takes_list`, `takes_search`, and `query` (when it returns takes) MUST - * apply `WHERE holder = ANY($takesHoldersAllowList)`. This is the - * server-side filter that backs the v0.28 visibility model. + * `takes_list`, `takes_search`, `takes_scorecard`, `takes_calibration`, + * and `query` (when it returns takes) MUST apply `WHERE holder = ANY($takesHoldersAllowList)`. + * This is the server-side filter that backs the v0.28+ visibility model. + * + * v0.30.0: aggregate ops (`takes_scorecard`, `takes_calibration`) require + * the allow-list as a TS-required engine method param (fail-closed by + * compiler). Hidden-holder rows contribute zero to aggregates. The CLI + * callers (local + trusted) leave it undefined. * * Default behavior when unset: local CLI callers see all holders. v0.28 * MCP dispatch sets it to `['world']` for tokens with no permissions row @@ -1023,6 +1028,62 @@ const takes_search: Operation = { cliHints: { name: 'takes-search', positional: ['query'] }, }; +/** + * v0.30.0 (Slice A1): aggregate calibration scorecard. Pure SQL aggregation. + * + * Privacy (D4 fail-closed): the engine method REQUIRES the takesHoldersAllowList + * param. The handler threads it from the OperationContext so MCP-bound callers + * see only their permitted holders' aggregate counts. Local CLI callers + * (ctx.takesHoldersAllowList=undefined) get the full scorecard. + */ +const takes_scorecard: Operation = { + name: 'takes_scorecard', + description: 'Calibration scorecard for resolved bets: counts, accuracy, Brier (correct ∨ incorrect only), partial_rate.', + scope: 'read', + params: { + holder: { type: 'string', description: 'Filter to this holder (world|garry|brain|)' }, + domain_prefix: { type: 'string', description: 'Slug prefix (e.g. companies/) to scope the scorecard' }, + since: { type: 'string', description: 'Window start (YYYY-MM-DD)' }, + until: { type: 'string', description: 'Window end (YYYY-MM-DD)' }, + }, + handler: async (ctx, p) => { + return ctx.engine.getScorecard( + { + holder: p.holder as string | undefined, + domainPrefix: p.domain_prefix as string | undefined, + since: p.since as string | undefined, + until: p.until as string | undefined, + }, + ctx.takesHoldersAllowList, + ); + }, + cliHints: { name: 'takes-scorecard' }, +}; + +/** + * v0.30.0 (Slice A1): calibration curve binned by stated weight. Pure SQL. + * Same allow-list contract as takes_scorecard. + */ +const takes_calibration: Operation = { + name: 'takes_calibration', + description: 'Calibration curve: resolved correct/incorrect bets binned by stated weight; observed vs predicted per bucket.', + scope: 'read', + params: { + holder: { type: 'string', description: 'Filter to this holder' }, + bucket_size: { type: 'number', description: 'Bucket width in (0,1]; default 0.1' }, + }, + handler: async (ctx, p) => { + return ctx.engine.getCalibrationCurve( + { + holder: p.holder as string | undefined, + bucketSize: p.bucket_size as number | undefined, + }, + ctx.takesHoldersAllowList, + ); + }, + cliHints: { name: 'takes-calibration' }, +}; + const think: Operation = { name: 'think', description: 'Multi-hop synthesis across pages + takes + graph. Pulls relevant evidence and produces a cited answer with conflict + gap analysis.', @@ -2191,6 +2252,8 @@ export const operations: Operation[] = [ find_orphans, // v0.28: Takes + think takes_list, takes_search, think, + // v0.30: calibration aggregates over takes + takes_scorecard, takes_calibration, // v0.28: whoami + scoped sources management whoami, sources_add, sources_list, sources_remove, sources_status, // v0.29: Salience + anomalies + recent transcripts diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 91fa1e18f..7b3d4c066 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -10,6 +10,7 @@ import type { FileSpec, FileRow, TakeBatchInput, Take, TakesListOpts, TakeHit, StaleTakeRow, TakeResolution, SynthesisEvidenceInput, + TakesScorecard, TakesScorecardOpts, CalibrationBucket, CalibrationCurveOpts, } from './engine.ts'; import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts'; import { runMigrations } from './migrate.ts'; @@ -32,6 +33,7 @@ import type { EmotionalWeightInputRow, EmotionalWeightWriteRow, } from './types.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, takeRowToTake } from './utils.ts'; +import { deriveResolutionTuple, finalizeScorecard } from './takes-resolution.ts'; import { GBrainError, PAGE_SORT_SQL } from './types.ts'; import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; @@ -1910,19 +1912,23 @@ export class PGLiteEngine implements BrainEngine { if (existing.resolved_at) { throw new GBrainError('TAKE_ALREADY_RESOLVED', `take ${pageId}#${rowNum} already resolved`, 'resolution is immutable; add a new take to record a new outcome'); } + // v0.30.0: derive (quality, outcome) tuple. quality wins when both set. + const { quality, outcome } = deriveResolutionTuple(resolution); await this.db.query( `UPDATE takes SET resolved_at = now(), - resolved_outcome = $3, - resolved_value = $4::real, - resolved_unit = $5::text, - resolved_source = $6::text, - resolved_by = $7, + resolved_quality = $3::text, + resolved_outcome = $4, + resolved_value = $5::real, + resolved_unit = $6::text, + resolved_source = $7::text, + resolved_by = $8, updated_at = now() WHERE page_id = $1 AND row_num = $2`, [ pageId, rowNum, - resolution.outcome, + quality, + outcome, resolution.value ?? null, resolution.unit ?? null, resolution.source ?? null, @@ -1931,6 +1937,88 @@ export class PGLiteEngine implements BrainEngine { ); } + /** + * v0.30.0: aggregate scorecard. SQL-level allow-list filter (D4 fail-closed). + * Hidden-holder rows contribute zero to aggregates. + */ + async getScorecard(opts: TakesScorecardOpts, allowList: string[] | undefined): Promise { + // Build the WHERE clause with positional params. PGLite (postgres-via-WASM) + // shares the SQL dialect with real Postgres so the math expressions match. + const params: unknown[] = []; + const clauses: string[] = []; + if (opts.holder !== undefined) { params.push(opts.holder); clauses.push(`AND holder = $${params.length}`); } + if (opts.domainPrefix !== undefined) { + params.push(opts.domainPrefix + '%'); + clauses.push(`AND EXISTS (SELECT 1 FROM pages p WHERE p.id = takes.page_id AND p.slug LIKE $${params.length})`); + } + if (opts.since !== undefined) { params.push(opts.since); clauses.push(`AND since_date >= $${params.length}`); } + if (opts.until !== undefined) { params.push(opts.until); clauses.push(`AND since_date <= $${params.length}`); } + if (allowList !== undefined) { params.push(allowList); clauses.push(`AND holder = ANY($${params.length}::text[])`); } + const where = clauses.join(' '); + const res = await this.db.query( + `SELECT + COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets, + COUNT(*) FILTER (WHERE resolved_quality IS NOT NULL)::int AS resolved, + COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct, + COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect, + COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial, + AVG( + CASE WHEN resolved_quality IN ('correct','incorrect') + THEN POWER(weight - (CASE resolved_quality WHEN 'correct' THEN 1 ELSE 0 END), 2) + END + )::float AS brier + FROM takes + WHERE 1=1 ${where}`, + params, + ); + const r = res.rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; brier: number | null }; + return finalizeScorecard(r); + } + + /** + * v0.30.0: calibration curve. Bins resolved correct/incorrect bets by stated weight. + */ + async getCalibrationCurve(opts: CalibrationCurveOpts, allowList: string[] | undefined): Promise { + const bucketSize = opts.bucketSize && opts.bucketSize > 0 && opts.bucketSize <= 1 ? opts.bucketSize : 0.1; + const maxIdx = Math.floor(1 / bucketSize) - 1; + const params: unknown[] = [bucketSize, maxIdx]; + const clauses: string[] = []; + if (opts.holder !== undefined) { params.push(opts.holder); clauses.push(`AND holder = $${params.length}`); } + if (allowList !== undefined) { params.push(allowList); clauses.push(`AND holder = ANY($${params.length}::text[])`); } + const where = clauses.join(' '); + // NUMERIC casts for exact decimal arithmetic — keeps PGLite + Postgres + // bucket boundaries identical at FP-edge weights (e.g. 0.7/0.1). + // See parity test in test/e2e/takes-scorecard-parity.test.ts. + const res = await this.db.query( + `WITH binned AS ( + SELECT + LEAST(FLOOR(weight::numeric / $1::numeric)::int, $2::int)::int AS bucket_idx, + weight, + (resolved_quality = 'correct')::int AS hit + FROM takes + WHERE resolved_quality IN ('correct','incorrect') + ${where} + ) + SELECT + (bucket_idx::numeric * $1::numeric)::float AS bucket_lo, + ((bucket_idx + 1)::numeric * $1::numeric)::float AS bucket_hi, + COUNT(*)::int AS n, + AVG(hit)::float AS observed, + AVG(weight)::float AS predicted + FROM binned + GROUP BY bucket_idx + ORDER BY bucket_idx`, + params, + ); + return (res.rows as { bucket_lo: number; bucket_hi: number; n: number; observed: number | null; predicted: number | null }[]).map(r => ({ + bucket_lo: r.bucket_lo, + bucket_hi: r.bucket_hi, + n: r.n, + observed: r.n > 0 ? r.observed : null, + predicted: r.n > 0 ? r.predicted : null, + })); + } + async addSynthesisEvidence(rowsIn: SynthesisEvidenceInput[]): Promise { if (rowsIn.length === 0) return 0; const synthesisIds = rowsIn.map(r => r.synthesis_page_id); diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 9f65803e1..f081579c0 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -7,8 +7,10 @@ import type { FileSpec, FileRow, TakeBatchInput, Take, TakesListOpts, TakeHit, StaleTakeRow, TakeResolution, SynthesisEvidenceInput, + TakesScorecard, TakesScorecardOpts, CalibrationBucket, CalibrationCurveOpts, } from './engine.ts'; import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts'; +import { deriveResolutionTuple, finalizeScorecard } from './takes-resolution.ts'; import { runMigrations } from './migrate.ts'; import { SCHEMA_SQL } from './schema-embedded.ts'; import { verifySchema } from './schema-verify.ts'; @@ -1953,10 +1955,14 @@ export class PostgresEngine implements BrainEngine { if ((existing as { resolved_at?: unknown }).resolved_at) { throw new GBrainError('TAKE_ALREADY_RESOLVED', `take ${pageId}#${rowNum} already resolved`, 'resolution is immutable; add a new take to record a new outcome'); } + // v0.30.0: derive (quality, outcome) tuple. quality wins when both set. + // Schema CHECK enforces consistency as a defense-in-depth backstop. + const { quality, outcome } = deriveResolutionTuple(resolution); await sql` UPDATE takes SET resolved_at = now(), - resolved_outcome = ${resolution.outcome}, + resolved_quality = ${quality}::text, + resolved_outcome = ${outcome}, resolved_value = ${resolution.value ?? null}::real, resolved_unit = ${resolution.unit ?? null}::text, resolved_source = ${resolution.source ?? null}::text, @@ -1966,6 +1972,89 @@ export class PostgresEngine implements BrainEngine { `; } + /** + * v0.30.0: aggregate scorecard. SQL-level allow-list filter (D4 fail-closed). + * Hidden-holder rows contribute zero to aggregates. NULL allowList means + * trusted caller (no filtering). Empty array → zero results. + */ + async getScorecard(opts: TakesScorecardOpts, allowList: string[] | undefined): Promise { + const sql = this.sql; + const allowed = allowList ? sql`AND holder = ANY(${allowList}::text[])` : sql``; + const holderClause = opts.holder ? sql`AND holder = ${opts.holder}` : sql``; + const domainClause = opts.domainPrefix + ? sql`AND EXISTS (SELECT 1 FROM pages p WHERE p.id = takes.page_id AND p.slug LIKE ${opts.domainPrefix + '%'})` + : sql``; + const sinceClause = opts.since ? sql`AND since_date >= ${opts.since}` : sql``; + const untilClause = opts.until ? sql`AND since_date <= ${opts.until}` : sql``; + const rows = await sql` + SELECT + COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets, + COUNT(*) FILTER (WHERE resolved_quality IS NOT NULL)::int AS resolved, + COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct, + COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect, + COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial, + AVG( + CASE WHEN resolved_quality IN ('correct','incorrect') + THEN POWER(weight - (CASE resolved_quality WHEN 'correct' THEN 1 ELSE 0 END), 2) + END + )::float AS brier + FROM takes + WHERE 1=1 ${holderClause} ${domainClause} ${sinceClause} ${untilClause} ${allowed} + `; + const r = rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; brier: number | null }; + return finalizeScorecard(r); + } + + /** + * v0.30.0: calibration curve. Bins resolved correct/incorrect bets by stated + * weight. Same allow-list contract as getScorecard. + * + * Real-Postgres-via-postgres.js sends scalar params as text by default, so + * `${bucketSize}` arrives as the string `'0.1'`. Without explicit `::float` + * casts the FLOOR/LEAST/multiplication contexts try to coerce text to int + * and bomb with `invalid input syntax for type integer: "0.1"`. PGLite is + * more permissive — caught at e2e parity by takes-scorecard-parity.test.ts. + */ + async getCalibrationCurve(opts: CalibrationCurveOpts, allowList: string[] | undefined): Promise { + const sql = this.sql; + const bucketSize = opts.bucketSize && opts.bucketSize > 0 && opts.bucketSize <= 1 ? opts.bucketSize : 0.1; + const maxIdx = Math.floor(1 / bucketSize) - 1; + const allowed = allowList ? sql`AND holder = ANY(${allowList}::text[])` : sql``; + const holderClause = opts.holder ? sql`AND holder = ${opts.holder}` : sql``; + // Bucketing uses NUMERIC for exact decimal arithmetic. Going through + // FLOAT introduces IEEE 754 rounding (e.g. 0.7/0.1 = 6.9999..., FLOOR=6 + // instead of the expected 7), which makes Postgres and PGLite diverge + // at bucket boundaries. NUMERIC is exact, so the bucket index is + // engine-agnostic and the parity test holds. + const rows = await sql` + WITH binned AS ( + SELECT + LEAST(FLOOR(weight::numeric / ${bucketSize}::numeric)::int, ${maxIdx}::int)::int AS bucket_idx, + weight, + (resolved_quality = 'correct')::int AS hit + FROM takes + WHERE resolved_quality IN ('correct','incorrect') + ${holderClause} ${allowed} + ) + SELECT + (bucket_idx::numeric * ${bucketSize}::numeric)::float AS bucket_lo, + ((bucket_idx + 1)::numeric * ${bucketSize}::numeric)::float AS bucket_hi, + COUNT(*)::int AS n, + AVG(hit)::float AS observed, + AVG(weight)::float AS predicted + FROM binned + GROUP BY bucket_idx + ORDER BY bucket_idx + `; + return (rows as unknown as { bucket_lo: number; bucket_hi: number; n: number; observed: number | null; predicted: number | null }[]).map(r => ({ + bucket_lo: r.bucket_lo, + bucket_hi: r.bucket_hi, + n: r.n, + observed: r.n > 0 ? r.observed : null, + predicted: r.n > 0 ? r.predicted : null, + })); + } + async addSynthesisEvidence(rowsIn: SynthesisEvidenceInput[]): Promise { if (rowsIn.length === 0) return 0; const sql = this.sql; diff --git a/src/core/takes-fence.ts b/src/core/takes-fence.ts index 682ca2e38..b0e124ca5 100644 --- a/src/core/takes-fence.ts +++ b/src/core/takes-fence.ts @@ -38,6 +38,8 @@ export type TakeKind = 'fact' | 'take' | 'bet' | 'hunch'; +export type TakeQuality = 'correct' | 'incorrect' | 'partial'; + export interface ParsedTake { rowNum: number; claim: string; // strikethrough markers stripped; inner text only @@ -48,6 +50,20 @@ export interface ParsedTake { untilDate?: string; source?: string; active: boolean; // false when claim was wrapped in ~~ ~~ + // v0.30.0 (Slice A1) resolution fields. Optional + always undefined on + // unresolved rows. The renderer emits the resolved/quality/evidence/value/ + // unit/by columns ONLY when at least one row on the page has resolvedQuality + // set; pages with no resolved rows keep their narrow 7-column shape. + // Round-trip preservation through cmdUpdate/cmdSupersede is the codex F3 + // safety net — without it, every update after a resolve silently deletes + // the resolution data on the next render. + resolvedAt?: string; // ISO timestamp 'YYYY-MM-DD' or full ISO + resolvedQuality?: TakeQuality; + resolvedOutcome?: boolean; // back-compat boolean; derivable from quality + resolvedEvidence?: string; // human note (alias for resolved_source) + resolvedValue?: number; + resolvedUnit?: string; + resolvedBy?: string; // slug or 'garry' } export interface ParseResult { @@ -60,6 +76,32 @@ export const TAKES_FENCE_BEGIN = ''; export const TAKES_FENCE_END = ''; const KIND_VALUES: ReadonlySet = new Set(['fact', 'take', 'bet', 'hunch']); +const QUALITY_VALUES: ReadonlySet = new Set(['correct', 'incorrect', 'partial']); + +// v0.30.0: header tokens that mark a v0.30-shape fence. Presence of `quality` +// (or any other resolution column) widens the parser to read 7+ extra cells +// per row. Missing tokens → v0.28 7-column shape, parsed exactly as before. +const RESOLUTION_HEADER_TOKENS = ['resolved', 'quality', 'evidence', 'value', 'unit', 'by'] as const; +type ResolutionColumn = typeof RESOLUTION_HEADER_TOKENS[number]; + +function parseQualityCell(raw: string): TakeQuality | undefined { + const trimmed = raw.trim().toLowerCase(); + if (!trimmed) return undefined; + if (QUALITY_VALUES.has(trimmed)) return trimmed as TakeQuality; + return undefined; +} + +function parseFloatCell(raw: string): number | undefined { + const trimmed = raw.trim(); + if (!trimmed) return undefined; + const n = parseFloat(trimmed); + return Number.isFinite(n) ? n : undefined; +} + +function parseStringCell(raw: string): string | undefined { + const trimmed = raw.trim(); + return trimmed ? trimmed : undefined; +} // Match a markdown table row's cell-stripped content. Allows surrounding // whitespace and tolerates trailing `|`. @@ -115,6 +157,9 @@ export function parseTakesFence(body: string): ParseResult { const lines = inner.split('\n'); const takes: ParsedTake[] = []; let sawHeader = false; + // Map from resolution column name → cell index in the row. Empty when the + // fence is a v0.28 7-column shape; populated when resolution columns appear. + const resolutionColIdx: Partial> = {}; const seenRowNums = new Set(); for (let i = 0; i < lines.length; i++) { @@ -124,11 +169,18 @@ export function parseTakesFence(body: string): ParseResult { if (!cells) continue; // Header row: `| # | claim | kind | who | weight | since | source |` + // (v0.28 7-column shape) OR with extra `| resolved | quality | evidence + // | value | unit | by |` columns appended (v0.30 13-column shape). if (!sawHeader) { - // Best-effort detection: header has 'claim' and 'kind' tokens. const lower = cells.map(c => c.toLowerCase()); if (lower.includes('claim') && lower.includes('kind')) { sawHeader = true; + // Detect v0.30 resolution columns. Columns are positional, but tolerate + // any subset (forward-compat: future schemas might add more). + for (const tok of RESOLUTION_HEADER_TOKENS) { + const idx = lower.indexOf(tok); + if (idx !== -1) resolutionColIdx[tok] = idx; + } continue; } // First content row before header — skip with warning. @@ -139,7 +191,7 @@ export function parseTakesFence(body: string): ParseResult { // Separator row (just dashes/colons) — skip. if (isSeparatorRow(cells)) continue; - // Expect 7 cells: row_num, claim, kind, holder, weight, since, source. + // Expect 7 cells minimum: row_num, claim, kind, holder, weight, since, source. if (cells.length < 6) { warnings.push(`TAKES_TABLE_MALFORMED: only ${cells.length} cells in row "${line.trim()}"`); continue; @@ -172,6 +224,26 @@ export function parseTakesFence(body: string): ParseResult { const { text: claimText, struck } = stripStrikethrough(claimRaw); const { since, until } = parseSinceCell(sinceRaw); + // v0.30 resolution columns. Only populated when the header contained the + // matching tokens AND the row has cells at those positions. + const cellAt = (col: ResolutionColumn): string | undefined => { + const idx = resolutionColIdx[col]; + if (idx === undefined) return undefined; + return idx < cells.length ? cells[idx] : undefined; + }; + const resolvedAt = cellAt('resolved'); + const qualityRaw = cellAt('quality'); + const evidenceRaw = cellAt('evidence'); + const valueRaw = cellAt('value'); + const unitRaw = cellAt('unit'); + const byRaw = cellAt('by'); + const resolvedQuality = qualityRaw !== undefined ? parseQualityCell(qualityRaw) : undefined; + // Derive resolvedOutcome from quality so the parsed shape is self-consistent + // for callers that read either field. + const resolvedOutcome = resolvedQuality === 'correct' ? true + : resolvedQuality === 'incorrect' ? false + : undefined; + takes.push({ rowNum, claim: claimText, @@ -182,6 +254,13 @@ export function parseTakesFence(body: string): ParseResult { untilDate: until, source: sourceRaw.trim() || undefined, active: !struck, + resolvedAt: resolvedAt ? parseStringCell(resolvedAt) : undefined, + resolvedQuality, + resolvedOutcome, + resolvedEvidence: evidenceRaw ? parseStringCell(evidenceRaw) : undefined, + resolvedValue: valueRaw ? parseFloatCell(valueRaw) : undefined, + resolvedUnit: unitRaw ? parseStringCell(unitRaw) : undefined, + resolvedBy: byRaw ? parseStringCell(byRaw) : undefined, }); } @@ -196,10 +275,30 @@ export function parseTakesFence(body: string): ParseResult { * Render a takes array back to a fenced markdown table. Round-trip safe * with parseTakesFence. Output uses tight column padding (one space per * side) — readable but not pretty-printed. + * + * v0.30.0 (Slice A1, codex F3 fix): conditional render of resolution + * columns. When ANY take in the array has `resolvedQuality !== undefined`, + * the renderer widens the table to 13 columns (`# | claim | kind | who | + * weight | since | source | resolved | quality | evidence | value | unit | + * by |`). Pages with no resolved rows keep the narrow 7-column shape + * exactly as v0.28 emitted. The parser tolerates both shapes. + * + * Round-trip preservation is the safety net for the silent-data-loss bug + * codex caught: every CLI that re-renders a fence (cmdUpdate, cmdSupersede, + * cmdAdd) must read existing rows via parseTakesFence and pass them + * through to renderTakesFence so resolution data on resolved rows survives + * unrelated edits to other rows on the same page. The + * round-trip-preservation test in test/takes-fence.test.ts is the + * regression gate. */ export function renderTakesFence(takes: ParsedTake[]): string { - const header = `| # | claim | kind | who | weight | since | source |`; - const separator = `|---|-------|------|-----|--------|-------|--------|`; + const hasAnyResolution = takes.some(t => t.resolvedQuality !== undefined); + const header = hasAnyResolution + ? `| # | claim | kind | who | weight | since | source | resolved | quality | evidence | value | unit | by |` + : `| # | claim | kind | who | weight | since | source |`; + const separator = hasAnyResolution + ? `|---|-------|------|-----|--------|-------|--------|----------|---------|----------|-------|------|----|` + : `|---|-------|------|-----|--------|-------|--------|`; const rows = takes.map(t => { const claimCell = t.active ? t.claim : `~~${t.claim}~~`; const sinceCell = t.untilDate ? `${t.sinceDate ?? ''} → ${t.untilDate}` : (t.sinceDate ?? ''); @@ -207,7 +306,17 @@ export function renderTakesFence(takes: ParsedTake[]): string { const source = t.source ?? ''; // Escape any pipes inside cells so the table doesn't break. const safe = (s: string) => s.replace(/\|/g, '\\|'); - return `| ${t.rowNum} | ${safe(claimCell)} | ${t.kind} | ${safe(t.holder)} | ${w} | ${safe(sinceCell)} | ${safe(source)} |`; + const baseCells = `| ${t.rowNum} | ${safe(claimCell)} | ${t.kind} | ${safe(t.holder)} | ${w} | ${safe(sinceCell)} | ${safe(source)} |`; + if (!hasAnyResolution) return baseCells; + // Resolution cells. Empty string for unresolved rows keeps the table + // visually clean; the parser treats empty cells as undefined fields. + const resolved = t.resolvedAt ? safe(t.resolvedAt) : ''; + const quality = t.resolvedQuality ?? ''; + const evidence = t.resolvedEvidence ? safe(t.resolvedEvidence) : ''; + const value = t.resolvedValue !== undefined ? formatWeight(t.resolvedValue) : ''; + const unit = t.resolvedUnit ? safe(t.resolvedUnit) : ''; + const by = t.resolvedBy ? safe(t.resolvedBy) : ''; + return `${baseCells} ${resolved} | ${quality} | ${evidence} | ${value} | ${unit} | ${by} |`; }); const inner = ['', header, separator, ...rows, ''].join('\n'); return `${TAKES_FENCE_BEGIN}${inner}${TAKES_FENCE_END}`; diff --git a/src/core/takes-resolution.ts b/src/core/takes-resolution.ts new file mode 100644 index 000000000..3d32dc27d --- /dev/null +++ b/src/core/takes-resolution.ts @@ -0,0 +1,105 @@ +/** + * v0.30.0 (Slice A1): pure helpers for the resolution + scorecard layer. + * Shared between Postgres + PGLite engines so the math + the (quality, outcome) + * derivation are identical across backends. + */ + +import { GBrainError } from './types.ts'; +import type { TakeResolution, TakesScorecard } from './engine.ts'; + +/** + * Derive the (quality, outcome) tuple that gets written to the takes row. + * `quality` wins when both are set. Returns the tuple ready for an UPDATE. + * + * Throws TAKE_RESOLUTION_INVALID when neither field is set, or when the input + * combines fields that the schema CHECK constraint would reject (e.g. + * quality='partial' + outcome=true). + * + * The schema `takes_resolution_consistency` CHECK is defense-in-depth — this + * function is the first line, surfacing a clear CLI-friendly error before + * the row hits the DB. + */ +export function deriveResolutionTuple( + resolution: TakeResolution, +): { quality: 'correct' | 'incorrect' | 'partial'; outcome: boolean | null } { + const { quality, outcome } = resolution; + if (quality === undefined && outcome === undefined) { + throw new GBrainError( + 'TAKE_RESOLUTION_INVALID', + 'resolveTake: must pass either `quality` (correct|incorrect|partial) or `outcome` (true|false)', + 'use --quality on the CLI; --outcome is the back-compat alias and cannot express partial', + ); + } + if (quality !== undefined) { + // Optional cross-check: when caller passed BOTH and they're inconsistent, + // surface the contradiction loudly instead of silently overwriting. + if (outcome !== undefined) { + const expected = quality === 'correct' ? true : quality === 'incorrect' ? false : null; + if (expected !== outcome) { + throw new GBrainError( + 'TAKE_RESOLUTION_INVALID', + `resolveTake: --quality=${quality} contradicts --outcome=${outcome}`, + 'pass only one of --quality or --outcome; they cannot disagree', + ); + } + } + return { + quality, + outcome: quality === 'correct' ? true : quality === 'incorrect' ? false : null, + }; + } + // Back-compat path: only `outcome` was supplied (v0.28 callers). + return { + quality: outcome ? 'correct' : 'incorrect', + outcome: outcome ?? null, + }; +} + +/** Raw aggregate row shape returned by both engines' getScorecard SQL. */ +export interface ScorecardRowRaw { + total_bets: number; + resolved: number; + correct: number; + incorrect: number; + partial: number; + brier: number | null; +} + +/** + * Finalize a scorecard from the raw aggregate row. Computes accuracy + + * partial_rate; returns NULL for empty windows so CLI can render + * "no resolved bets yet" instead of NaN. Brier comes straight from SQL. + * + * Brier scope (D5 + D11): `partial` rows are excluded from the Brier + * denominator entirely because partial isn't a binary outcome. The + * `partial_rate` field surfaces hedging behavior as a separate signal so + * users see both the calibration math AND whether they're hedging into + * the unmeasured bucket. + */ +export function finalizeScorecard(raw: ScorecardRowRaw): TakesScorecard { + const correct = Number(raw.correct ?? 0); + const incorrect = Number(raw.incorrect ?? 0); + const partial = Number(raw.partial ?? 0); + const resolved = Number(raw.resolved ?? 0); + const totalBets = Number(raw.total_bets ?? 0); + const binary = correct + incorrect; + return { + total_bets: totalBets, + resolved, + correct, + incorrect, + partial, + accuracy: binary > 0 ? correct / binary : null, + brier: binary > 0 && raw.brier !== null && raw.brier !== undefined + ? Number(raw.brier) + : null, + partial_rate: resolved > 0 ? partial / resolved : null, + }; +} + +/** + * Threshold above which scorecard CLI emits a warning that calibration may + * be optimistic (D11). 20% partial means 1 in 5 bets escaped the Brier + * denominator — the user is hedging into the unmeasured bucket. + */ +export const PARTIAL_RATE_WARNING_THRESHOLD = 0.20; diff --git a/src/core/utils.ts b/src/core/utils.ts index ab73dfe4b..c66f08a81 100644 --- a/src/core/utils.ts +++ b/src/core/utils.ts @@ -254,6 +254,9 @@ export function takeRowToTake(row: Record): Take { active: Boolean(row.active), resolved_at: isoOrNull(row.resolved_at), resolved_outcome: row.resolved_outcome == null ? null : Boolean(row.resolved_outcome), + resolved_quality: row.resolved_quality == null + ? null + : (String(row.resolved_quality) as 'correct' | 'incorrect' | 'partial'), resolved_value: row.resolved_value == null ? null : Number(row.resolved_value), resolved_unit: row.resolved_unit == null ? null : String(row.resolved_unit), resolved_source: row.resolved_source == null ? null : String(row.resolved_source), diff --git a/test/e2e/takes-postgres.test.ts b/test/e2e/takes-postgres.test.ts index 66069457a..6749ec338 100644 --- a/test/e2e/takes-postgres.test.ts +++ b/test/e2e/takes-postgres.test.ts @@ -228,3 +228,177 @@ d('v0.28 MCP allow-list — Postgres dispatch', () => { expect(env.remote_persisted_blocked).toBe(true); }); }); + +// ============================================================ +// v0.30.0 (Slice A1): 3-state quality + scorecard + calibration on real PG. +// Mirrors the unit-test invariants (PGLite) against postgres.js and the +// real CHECK constraint enforcement. +// ============================================================ +d('v0.30.0 takes resolve --quality on real Postgres', () => { + test('quality=correct writes both columns; CHECK constraint is enforced', async () => { + const engine = getEngine(); + await engine.addTakesBatch([ + { page_id: acmePageId, row_num: 50, claim: 'Series A correct', kind: 'bet', holder: 'garry', weight: 0.7 }, + ]); + await engine.resolveTake(acmePageId, 50, { quality: 'correct', resolvedBy: 'garry' }); + const rows = await engine.executeRaw<{ resolved_outcome: boolean | null; resolved_quality: string | null }>( + `SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`, + [acmePageId, 50], + ); + expect(rows[0].resolved_outcome).toBe(true); + expect(rows[0].resolved_quality).toBe('correct'); + }); + + test('quality=partial writes (partial, NULL) + CHECK accepts it', async () => { + const engine = getEngine(); + await engine.addTakesBatch([ + { page_id: acmePageId, row_num: 51, claim: 'partial scope bet', kind: 'bet', holder: 'garry', weight: 0.55 }, + ]); + await engine.resolveTake(acmePageId, 51, { quality: 'partial', resolvedBy: 'garry' }); + const rows = await engine.executeRaw<{ resolved_outcome: boolean | null; resolved_quality: string | null }>( + `SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`, + [acmePageId, 51], + ); + expect(rows[0].resolved_outcome).toBeNull(); + expect(rows[0].resolved_quality).toBe('partial'); + }); + + test('CHECK constraint rejects contradictory raw UPDATE', async () => { + const engine = getEngine(); + await engine.addTakesBatch([ + { page_id: acmePageId, row_num: 52, claim: 'unresolved bet', kind: 'bet', holder: 'garry', weight: 0.6 }, + ]); + // Bypass the engine method's deriveResolutionTuple guard and write a + // contradictory tuple directly. The schema CHECK should refuse. + await expect( + engine.executeRaw( + `UPDATE takes SET resolved_at = now(), resolved_outcome = true, resolved_quality = 'incorrect' WHERE page_id = $1 AND row_num = $2`, + [acmePageId, 52], + ), + ).rejects.toThrow(/takes_resolution_consistency|check constraint/i); + }); + + test('back-compat: resolveTake with outcome=true → quality=correct on real PG', async () => { + const engine = getEngine(); + await engine.addTakesBatch([ + { page_id: acmePageId, row_num: 53, claim: 'legacy v0.28 callers', kind: 'bet', holder: 'garry', weight: 0.7 }, + ]); + await engine.resolveTake(acmePageId, 53, { outcome: true, resolvedBy: 'garry' }); + const rows = await engine.executeRaw<{ resolved_outcome: boolean; resolved_quality: string }>( + `SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`, + [acmePageId, 53], + ); + expect(rows[0].resolved_outcome).toBe(true); + expect(rows[0].resolved_quality).toBe('correct'); + }); +}); + +d('v0.30.0 scorecard + calibration on real Postgres', () => { + // Note: this suite runs after the other Postgres takes suites in this + // file, so the takes table already has resolved data from the earlier + // tests. We don't need a fresh seed — we just verify the aggregate + // queries work end-to-end against postgres.js bind shapes. + test('getScorecard returns coherent shape against real PG', async () => { + const engine = getEngine(); + const card = await engine.getScorecard({ holder: 'garry' }, undefined); + expect(card.total_bets).toBeGreaterThan(0); + expect(card.resolved).toBeGreaterThanOrEqual(0); + expect(card.correct + card.incorrect + card.partial).toBe(card.resolved); + if (card.correct + card.incorrect > 0) { + expect(card.brier).not.toBeNull(); + expect(card.brier!).toBeGreaterThanOrEqual(0); + expect(card.brier!).toBeLessThanOrEqual(1); + } + }); + + test('getCalibrationCurve returns ordered buckets against real PG', async () => { + const engine = getEngine(); + const buckets = await engine.getCalibrationCurve({ holder: 'garry' }, undefined); + // Buckets must be ordered by bucket_lo ascending. + for (let i = 1; i < buckets.length; i++) { + expect(buckets[i].bucket_lo).toBeGreaterThanOrEqual(buckets[i - 1].bucket_lo); + } + // Every bucket has n > 0 (empty buckets aren't returned by GROUP BY). + for (const b of buckets) expect(b.n).toBeGreaterThan(0); + }); + + test('PRIVACY: scorecard SQL allow-list excludes hidden holders on real PG', async () => { + const engine = getEngine(); + // Seed a holder that the scorecard with allow-list ['garry'] should + // not see. Use a fresh page so the assertion is local and not noisy. + const harjPage = await engine.putPage('companies/scorecard-allowlist-fixture', { + title: 'Allow-list fixture', type: 'company', compiled_truth: '## Takes\n', + }); + await engine.addTakesBatch([ + { page_id: harjPage.id, row_num: 1, claim: 'g bet', kind: 'bet', holder: 'garry', weight: 0.7 }, + { page_id: harjPage.id, row_num: 2, claim: 'h bet', kind: 'bet', holder: 'harj-taggar', weight: 0.6 }, + ]); + await engine.resolveTake(harjPage.id, 1, { quality: 'correct', resolvedBy: 'garry' }); + await engine.resolveTake(harjPage.id, 2, { quality: 'incorrect', resolvedBy: 'harj-taggar' }); + + const garryOnly = await engine.getScorecard( + { domainPrefix: 'companies/scorecard-allowlist-fixture' }, + ['garry'], + ); + const trustedFull = await engine.getScorecard( + { domainPrefix: 'companies/scorecard-allowlist-fixture' }, + undefined, + ); + expect(garryOnly.resolved).toBe(1); + expect(garryOnly.correct).toBe(1); + expect(trustedFull.resolved).toBe(2); + // Allow-list strictly subtracts the harj row. + expect(trustedFull.resolved - garryOnly.resolved).toBe(1); + }); +}); + +// ============================================================ +// v0.30.0: MCP dispatch path for takes_scorecard + takes_calibration. +// ============================================================ +d('v0.30.0 MCP dispatch — Postgres', () => { + test('takes_scorecard via MCP returns correct counts with allow-list', async () => { + const engine = getEngine(); + const result = await dispatchToolCall(engine, 'takes_scorecard', { holder: 'garry' }, { + remote: true, + takesHoldersAllowList: ['garry'], + }); + expect(result.isError).toBeFalsy(); + const card = JSON.parse(result.content[0].text); + expect(card).toHaveProperty('correct'); + expect(card).toHaveProperty('incorrect'); + expect(card).toHaveProperty('partial'); + expect(card).toHaveProperty('brier'); + expect(card).toHaveProperty('partial_rate'); + }); + + test('takes_calibration via MCP returns bucket array with allow-list', async () => { + const engine = getEngine(); + const result = await dispatchToolCall(engine, 'takes_calibration', { holder: 'garry', bucket_size: 0.1 }, { + remote: true, + takesHoldersAllowList: ['garry'], + }); + expect(result.isError).toBeFalsy(); + const buckets = JSON.parse(result.content[0].text); + expect(Array.isArray(buckets)).toBe(true); + if (buckets.length > 0) { + expect(buckets[0]).toHaveProperty('bucket_lo'); + expect(buckets[0]).toHaveProperty('bucket_hi'); + expect(buckets[0]).toHaveProperty('n'); + expect(buckets[0]).toHaveProperty('observed'); + expect(buckets[0]).toHaveProperty('predicted'); + } + }); + + test('PRIVACY: takes_scorecard with allow-list ["world"] excludes garry rows', async () => { + const engine = getEngine(); + // 'world' has only fact-kind takes in the seed; bets are garry-only. + // Scorecard scoped to world should report zero resolved. + const result = await dispatchToolCall(engine, 'takes_scorecard', {}, { + remote: true, + takesHoldersAllowList: ['world'], + }); + const card = JSON.parse(result.content[0].text); + // No resolved bets exist with holder='world' in our seed. + expect(card.resolved).toBe(0); + }); +}); diff --git a/test/e2e/takes-scorecard-parity.test.ts b/test/e2e/takes-scorecard-parity.test.ts new file mode 100644 index 000000000..fad61c365 --- /dev/null +++ b/test/e2e/takes-scorecard-parity.test.ts @@ -0,0 +1,202 @@ +/** + * v0.30.0 (Slice A1) E2E: scorecard + calibration parity between Postgres + * and PGLite. Same fixture, same query, same numbers. + * + * Seeds 4 binary bets + 1 partial bet into both engines (mirrors the + * `takes-resolution.test.ts` 4-bet hand-calc reference: Brier=0.205). + * Asserts getScorecard returns byte-identical numeric output and + * getCalibrationCurve emits the same buckets. + * + * Privacy gate: also asserts the SQL-level allow-list filter (D4 + * fail-closed) returns identical results across engines — hidden-holder + * rows must contribute zero on both sides. + * + * Skips gracefully when DATABASE_URL is unset. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts'; +import type { BrainEngine } from '../../src/core/engine.ts'; +import type { TakesScorecardOpts, CalibrationCurveOpts } from '../../src/core/engine.ts'; + +const SKIP_PG = !hasDatabase(); +const d = SKIP_PG ? describe.skip : describe; + +let pglite: PGLiteEngine; +let pgEngine: BrainEngine; + +interface FixtureBet { + rowNum: number; + claim: string; + holder: string; + weight: number; + resolveAs?: 'correct' | 'incorrect' | 'partial'; +} + +// Same fixture used by takes-resolution.test.ts hand-calc. +// 4 binary garry bets at varied weights (Brier=0.205) + 1 partial garry + +// 1 binary harj bet so the allow-list assertion has signal. +const FIXTURE_BETS: FixtureBet[] = [ + { rowNum: 1, claim: 'b1', holder: 'garry', weight: 0.9, resolveAs: 'correct' }, + { rowNum: 2, claim: 'b2', holder: 'garry', weight: 0.6, resolveAs: 'correct' }, + { rowNum: 3, claim: 'b3', holder: 'garry', weight: 0.7, resolveAs: 'incorrect' }, + { rowNum: 4, claim: 'b4', holder: 'garry', weight: 0.4, resolveAs: 'incorrect' }, + { rowNum: 5, claim: 'b5', holder: 'garry', weight: 0.5, resolveAs: 'partial' }, + { rowNum: 6, claim: 'h1', holder: 'harj-taggar', weight: 0.8, resolveAs: 'correct' }, +]; + +const SCORECARD_PAGE = 'companies/scorecard-parity-fixture'; + +async function seedFixture(engine: BrainEngine): Promise { + const page = await engine.putPage(SCORECARD_PAGE, { + title: 'Scorecard parity fixture', + type: 'company' as const, + compiled_truth: '## Takes\n', + }); + await engine.addTakesBatch( + FIXTURE_BETS.map(b => ({ + page_id: page.id, + row_num: b.rowNum, + claim: b.claim, + kind: 'bet' as const, + holder: b.holder, + weight: b.weight, + })), + ); + for (const b of FIXTURE_BETS) { + if (b.resolveAs === undefined) continue; + await engine.resolveTake(page.id, b.rowNum, { + quality: b.resolveAs, + resolvedBy: b.holder, + }); + } + return page.id; +} + +beforeAll(async () => { + if (SKIP_PG) return; + // PGLite (in-memory, no DATABASE_URL needed; runs even on the skipped + // suite path so we'd still know if its seed broke). + pglite = new PGLiteEngine(); + await pglite.connect({}); + await pglite.initSchema(); + await seedFixture(pglite); + + // Real Postgres + await setupDB(); + pgEngine = getEngine(); + await seedFixture(pgEngine); +}); + +afterAll(async () => { + if (pglite) await pglite.disconnect(); + if (!SKIP_PG) await teardownDB(); +}); + +d('v0.30.0 e2e: scorecard parity (PG vs PGLite)', () => { + const queries: Array<{ name: string; opts: TakesScorecardOpts; allowList?: string[] }> = [ + { name: 'all garry, no allow-list', opts: { holder: 'garry', domainPrefix: SCORECARD_PAGE } }, + { name: 'all rows, no holder filter', opts: { domainPrefix: SCORECARD_PAGE } }, + { name: 'allow-list garry-only', opts: { domainPrefix: SCORECARD_PAGE }, allowList: ['garry'] }, + { name: 'allow-list world-only (zero match)', opts: { domainPrefix: SCORECARD_PAGE }, allowList: ['world'] }, + ]; + + for (const q of queries) { + test(`scorecard: ${q.name}`, async () => { + const pgCard = await pgEngine.getScorecard(q.opts, q.allowList); + const pgliteCard = await pglite.getScorecard(q.opts, q.allowList); + expect(pgCard.total_bets).toBe(pgliteCard.total_bets); + expect(pgCard.resolved).toBe(pgliteCard.resolved); + expect(pgCard.correct).toBe(pgliteCard.correct); + expect(pgCard.incorrect).toBe(pgliteCard.incorrect); + expect(pgCard.partial).toBe(pgliteCard.partial); + // Float fields: byte-equality is too strict due to engine-driver + // numeric coercion (Postgres returns string-coerced floats; PGLite + // returns JS numbers). Use 6-decimal closeness — well below any + // user-visible precision. + const closeOrBothNull = (a: number | null, b: number | null) => { + if (a === null && b === null) return; + expect(a).not.toBeNull(); + expect(b).not.toBeNull(); + expect(a).toBeCloseTo(b!, 6); + }; + closeOrBothNull(pgCard.accuracy, pgliteCard.accuracy); + closeOrBothNull(pgCard.brier, pgliteCard.brier); + closeOrBothNull(pgCard.partial_rate, pgliteCard.partial_rate); + }); + } + + test('Brier on the 4-bet hand-calc reference matches 0.205 on both engines', async () => { + // Filter to garry bets only — the harj bet would skew the Brier. + const opts: TakesScorecardOpts = { holder: 'garry', domainPrefix: SCORECARD_PAGE }; + const pgCard = await pgEngine.getScorecard(opts, undefined); + const pgliteCard = await pglite.getScorecard(opts, undefined); + expect(pgCard.brier).toBeCloseTo(0.205, 4); + expect(pgliteCard.brier).toBeCloseTo(0.205, 4); + // Also: partial_rate = 1/5 = 0.2 (4 binary + 1 partial = 5 resolved garry rows). + expect(pgCard.partial_rate).toBeCloseTo(0.2, 6); + expect(pgliteCard.partial_rate).toBeCloseTo(0.2, 6); + }); + + test('PRIVACY: allow-list ["garry"] gives same totals on both engines AND excludes harj', async () => { + const opts: TakesScorecardOpts = { domainPrefix: SCORECARD_PAGE }; + const pgGarry = await pgEngine.getScorecard(opts, ['garry']); + const pgliteGarry = await pglite.getScorecard(opts, ['garry']); + + // Both engines see exactly the 5 garry rows. + expect(pgGarry.resolved).toBe(5); + expect(pgliteGarry.resolved).toBe(5); + + // Without allow-list, both see all 6 resolved rows (5 garry + 1 harj). + const pgFull = await pgEngine.getScorecard(opts, undefined); + const pgliteFull = await pglite.getScorecard(opts, undefined); + expect(pgFull.resolved).toBe(6); + expect(pgliteFull.resolved).toBe(6); + + // Allow-list strictly subtracts (defense in depth: if SQL filter were + // post-filtered or applied wrong, this assertion catches it). + expect(pgFull.resolved - pgGarry.resolved).toBe(1); + expect(pgliteFull.resolved - pgliteGarry.resolved).toBe(1); + }); +}); + +d('v0.30.0 e2e: calibration curve parity (PG vs PGLite)', () => { + const queries: Array<{ name: string; opts: CalibrationCurveOpts; allowList?: string[] }> = [ + { name: 'garry-only, default bucket', opts: { holder: 'garry' } }, + { name: 'all-rows, default bucket', opts: {} }, + { name: 'allow-list garry-only', opts: {}, allowList: ['garry'] }, + ]; + for (const q of queries) { + test(`calibration: ${q.name}`, async () => { + const pgBuckets = await pgEngine.getCalibrationCurve(q.opts, q.allowList); + const pgliteBuckets = await pglite.getCalibrationCurve(q.opts, q.allowList); + expect(pgBuckets.length).toBe(pgliteBuckets.length); + for (let i = 0; i < pgBuckets.length; i++) { + expect(pgBuckets[i].bucket_lo).toBeCloseTo(pgliteBuckets[i].bucket_lo, 6); + expect(pgBuckets[i].bucket_hi).toBeCloseTo(pgliteBuckets[i].bucket_hi, 6); + expect(pgBuckets[i].n).toBe(pgliteBuckets[i].n); + if (pgBuckets[i].observed !== null) { + expect(pgBuckets[i].observed).toBeCloseTo(pgliteBuckets[i].observed!, 6); + } else { + expect(pgliteBuckets[i].observed).toBeNull(); + } + if (pgBuckets[i].predicted !== null) { + expect(pgBuckets[i].predicted).toBeCloseTo(pgliteBuckets[i].predicted!, 6); + } else { + expect(pgliteBuckets[i].predicted).toBeNull(); + } + } + }); + } + + test('partial bet (weight 0.5) is excluded from calibration curve on both engines', async () => { + const pgBuckets = await pgEngine.getCalibrationCurve({ holder: 'garry' }, undefined); + const pgliteBuckets = await pglite.getCalibrationCurve({ holder: 'garry' }, undefined); + const pgTotal = pgBuckets.reduce((s, b) => s + b.n, 0); + const pgliteTotal = pgliteBuckets.reduce((s, b) => s + b.n, 0); + // 4 binary garry rows; partial excluded. + expect(pgTotal).toBe(4); + expect(pgliteTotal).toBe(4); + }); +}); diff --git a/test/takes-engine.test.ts b/test/takes-engine.test.ts index e9ae78857..826cf41ae 100644 --- a/test/takes-engine.test.ts +++ b/test/takes-engine.test.ts @@ -162,6 +162,49 @@ describe('resolveTake + immutability', () => { ).rejects.toThrow(/TAKE_ALREADY_RESOLVED/); }); + // v0.30.0: 3-state quality input + back-compat outcome alias. + test('v0.30.0: resolve with --quality correct writes both columns', async () => { + await engine.addTakesBatch([ + { page_id: alicePageId, row_num: 11, claim: 'Will close Series A', kind: 'bet', holder: 'garry', weight: 0.7 }, + ]); + await engine.resolveTake(alicePageId, 11, { quality: 'correct', resolvedBy: 'garry' }); + const takes = await engine.listTakes({ page_id: alicePageId, resolved: true }); + const r = takes.find(t => t.row_num === 11)!; + expect(r.resolved_quality).toBe('correct'); + expect(r.resolved_outcome).toBe(true); + }); + + test('v0.30.0: resolve with --quality partial writes (partial, NULL)', async () => { + await engine.addTakesBatch([ + { page_id: alicePageId, row_num: 12, claim: 'Will reach $100M ARR', kind: 'bet', holder: 'garry', weight: 0.55 }, + ]); + await engine.resolveTake(alicePageId, 12, { quality: 'partial', resolvedBy: 'garry' }); + const takes = await engine.listTakes({ page_id: alicePageId, resolved: true }); + const r = takes.find(t => t.row_num === 12)!; + expect(r.resolved_quality).toBe('partial'); + expect(r.resolved_outcome).toBeNull(); + }); + + test('v0.30.0 (back-compat): outcome=true → quality=correct (legacy v0.28 callers)', async () => { + await engine.addTakesBatch([ + { page_id: alicePageId, row_num: 13, claim: 'Legacy bet', kind: 'bet', holder: 'garry', weight: 0.8 }, + ]); + await engine.resolveTake(alicePageId, 13, { outcome: true, resolvedBy: 'garry' }); + const takes = await engine.listTakes({ page_id: alicePageId, resolved: true }); + const r = takes.find(t => t.row_num === 13)!; + expect(r.resolved_quality).toBe('correct'); + expect(r.resolved_outcome).toBe(true); + }); + + test('v0.30.0: contradictory quality + outcome throws TAKE_RESOLUTION_INVALID', async () => { + await engine.addTakesBatch([ + { page_id: alicePageId, row_num: 14, claim: 'Conflicting input bet', kind: 'bet', holder: 'garry', weight: 0.5 }, + ]); + await expect( + engine.resolveTake(alicePageId, 14, { quality: 'correct', outcome: false, resolvedBy: 'garry' }), + ).rejects.toThrow(/TAKE_RESOLUTION_INVALID/); + }); + test('TAKE_RESOLVED_IMMUTABLE on supersede attempt of resolved bet', async () => { await expect( engine.supersedeTake(alicePageId, 10, { @@ -174,6 +217,122 @@ describe('resolveTake + immutability', () => { }); }); +// ============================================================ +// v0.30.0 (Slice A1): scorecard + calibration aggregates. +// ============================================================ +describe('v0.30.0 getScorecard', () => { + let scorePageId: number; + beforeAll(async () => { + // Fresh page so we control the resolved-bets population precisely. + const p = await engine.putPage('companies/scorecard-fixture', { + title: 'Scorecard fixture', + type: 'company' as const, + compiled_truth: '## Takes\n', + }); + scorePageId = p.id; + // 4 bets at varied weights, mixed outcomes — matches the unit-test + // hand-calc in takes-resolution.test.ts so we can sanity-check Brier. + await engine.addTakesBatch([ + { page_id: scorePageId, row_num: 1, claim: 'b1', kind: 'bet', holder: 'garry', weight: 0.9 }, + { page_id: scorePageId, row_num: 2, claim: 'b2', kind: 'bet', holder: 'garry', weight: 0.6 }, + { page_id: scorePageId, row_num: 3, claim: 'b3', kind: 'bet', holder: 'garry', weight: 0.7 }, + { page_id: scorePageId, row_num: 4, claim: 'b4', kind: 'bet', holder: 'garry', weight: 0.4 }, + { page_id: scorePageId, row_num: 5, claim: 'b5 partial', kind: 'bet', holder: 'garry', weight: 0.5 }, + ]); + await engine.resolveTake(scorePageId, 1, { quality: 'correct', resolvedBy: 'garry' }); + await engine.resolveTake(scorePageId, 2, { quality: 'correct', resolvedBy: 'garry' }); + await engine.resolveTake(scorePageId, 3, { quality: 'incorrect', resolvedBy: 'garry' }); + await engine.resolveTake(scorePageId, 4, { quality: 'incorrect', resolvedBy: 'garry' }); + await engine.resolveTake(scorePageId, 5, { quality: 'partial', resolvedBy: 'garry' }); + }); + + test('counts correct/incorrect/partial; accuracy excludes partial', async () => { + const card = await engine.getScorecard({ holder: 'garry', domainPrefix: 'companies/scorecard-fixture' }, undefined); + expect(card.correct).toBe(2); + expect(card.incorrect).toBe(2); + expect(card.partial).toBe(1); + expect(card.resolved).toBe(5); + expect(card.accuracy).toBeCloseTo(2 / 4, 5); // correct / (correct + incorrect) + expect(card.partial_rate).toBe(0.2); + }); + + test('Brier excludes partial: hand-calculated reference == 0.205', async () => { + const card = await engine.getScorecard({ holder: 'garry', domainPrefix: 'companies/scorecard-fixture' }, undefined); + // Per-row Brier (correct ∨ incorrect only): + // (0.9-1)^2 = 0.01 + // (0.6-1)^2 = 0.16 + // (0.7-0)^2 = 0.49 + // (0.4-0)^2 = 0.16 + // Mean: (0.01+0.16+0.49+0.16)/4 = 0.205 + expect(card.brier).toBeCloseTo(0.205, 4); + }); + + test('PRIVACY: SQL allow-list filter — hidden-holder rows contribute zero', async () => { + // Add a take from a different holder (e.g., harj). The scorecard with + // allow-list ['garry'] must NOT count that take in any aggregate. + const p = await engine.putPage('companies/allowlist-fixture', { + title: 'Allow-list fixture', + type: 'company' as const, + compiled_truth: '## Takes\n', + }); + await engine.addTakesBatch([ + { page_id: p.id, row_num: 1, claim: 'garry bet', kind: 'bet', holder: 'garry', weight: 0.7 }, + { page_id: p.id, row_num: 2, claim: 'harj bet', kind: 'bet', holder: 'harj-taggar', weight: 0.6 }, + ]); + await engine.resolveTake(p.id, 1, { quality: 'correct', resolvedBy: 'garry' }); + await engine.resolveTake(p.id, 2, { quality: 'incorrect', resolvedBy: 'harj-taggar' }); + + // Scoped to this page so we don't mix with the earlier fixture. + const allowedGarry = await engine.getScorecard({ domainPrefix: 'companies/allowlist-fixture' }, ['garry']); + expect(allowedGarry.correct).toBe(1); + expect(allowedGarry.incorrect).toBe(0); + expect(allowedGarry.resolved).toBe(1); + + const trustedFull = await engine.getScorecard({ domainPrefix: 'companies/allowlist-fixture' }, undefined); + expect(trustedFull.resolved).toBe(2); + }); + + test('n=0 scorecard does not divide by zero', async () => { + const card = await engine.getScorecard({ holder: 'nonexistent-holder' }, undefined); + expect(card.resolved).toBe(0); + expect(card.accuracy).toBeNull(); + expect(card.brier).toBeNull(); + expect(card.partial_rate).toBeNull(); + }); +}); + +describe('v0.30.0 getCalibrationCurve', () => { + test('bins resolved bets by stated weight; partial excluded; harj non-allowed contributes zero', async () => { + // The cross-suite state has accumulated several resolved bets; rather + // than couple to exact totals, assert the structural invariants: + // (1) all returned buckets contain garry rows only when allow-list is garry + // (2) the unfiltered count INCLUDES at least one harj row that the + // allow-listed call does NOT include + // (3) partial bets never appear (Brier excludes them by definition) + const garryAll = await engine.getCalibrationCurve({ holder: 'garry' }, undefined); + const totalGarry = garryAll.reduce((s, b) => s + b.n, 0); + expect(totalGarry).toBeGreaterThan(0); + for (const b of garryAll) { + // observed in [0, 1]; predicted in [0, 1) + if (b.observed !== null) { expect(b.observed).toBeGreaterThanOrEqual(0); expect(b.observed).toBeLessThanOrEqual(1); } + if (b.predicted !== null) { expect(b.predicted).toBeGreaterThanOrEqual(0); expect(b.predicted).toBeLessThan(1.001); } + } + }); + + test('PRIVACY: allow-list filter strictly subtracts harj rows', async () => { + // Without the allow-list (trusted caller) the curve sees harj's bets too. + const trustedAll = await engine.getCalibrationCurve({}, undefined); + const totalTrusted = trustedAll.reduce((s, b) => s + b.n, 0); + + const garryOnly = await engine.getCalibrationCurve({}, ['garry']); + const totalGarry = garryOnly.reduce((s, b) => s + b.n, 0); + + // Harj has at least one resolved binary bet from the allowlist fixture. + // The allow-list MUST drop it strictly: garry-only count < trusted count. + expect(totalGarry).toBeLessThan(totalTrusted); + }); +}); + describe('synthesis_evidence', () => { test('addSynthesisEvidence persists provenance and CASCADE deletes when take is removed', async () => { // Create a synthesis page diff --git a/test/takes-fence.test.ts b/test/takes-fence.test.ts index aeb87b5c3..f1b6145a8 100644 --- a/test/takes-fence.test.ts +++ b/test/takes-fence.test.ts @@ -236,3 +236,174 @@ describe('stripTakesFence', () => { expect(stripTakesFence(body)).toBe(body); }); }); + +// ============================================================ +// v0.30.0 (Slice A1): resolution columns + round-trip preservation. +// The round-trip preservation tests are the codex-F3 regression gate. +// Without these, every `gbrain takes update` after a resolve silently +// deletes the resolution data on the next render. +// ============================================================ + +describe('v0.30.0 resolution columns', () => { + const RESOLVED_BODY = `# Some Page\n\n## Takes\n\n${TAKES_FENCE_BEGIN} +| # | claim | kind | who | weight | since | source | resolved | quality | evidence | value | unit | by | +|---|-------|------|-----|--------|-------|--------|----------|---------|----------|-------|------|----| +| 1 | First bet | bet | garry | 0.7 | 2026-04 | OH | 2026-04-30 | correct | Series A closed | 50 | usd | garry | +| 2 | Pending bet | bet | garry | 0.6 | 2026-04 | OH | | | | | | | +${TAKES_FENCE_END}\n`; + + test('parses v0.30-shape fence: resolved row has resolution fields populated', () => { + const { takes, warnings } = parseTakesFence(RESOLVED_BODY); + expect(warnings).toEqual([]); + expect(takes).toHaveLength(2); + expect(takes[0]).toMatchObject({ + rowNum: 1, + resolvedAt: '2026-04-30', + resolvedQuality: 'correct', + resolvedOutcome: true, + resolvedEvidence: 'Series A closed', + resolvedValue: 50, + resolvedUnit: 'usd', + resolvedBy: 'garry', + }); + }); + + test('parses v0.30-shape fence: unresolved row has resolution fields undefined', () => { + const { takes } = parseTakesFence(RESOLVED_BODY); + expect(takes[1].resolvedQuality).toBeUndefined(); + expect(takes[1].resolvedOutcome).toBeUndefined(); + expect(takes[1].resolvedAt).toBeUndefined(); + }); + + test('renderer: page with no resolved rows keeps narrow 7-column shape', () => { + const { takes } = parseTakesFence(SAMPLE_BODY); + const rendered = renderTakesFence(takes); + expect(rendered).toContain('| # | claim | kind | who | weight | since | source |'); + expect(rendered).not.toContain('quality'); + expect(rendered).not.toContain('resolved'); + }); + + test('renderer: any resolved row triggers wide 13-column shape', () => { + const { takes } = parseTakesFence(SAMPLE_BODY); + // Mutate one row to have resolution data. + takes[0] = { + ...takes[0], + resolvedAt: '2026-05-01', + resolvedQuality: 'correct', + resolvedOutcome: true, + resolvedEvidence: 'verified', + }; + const rendered = renderTakesFence(takes); + expect(rendered).toContain('quality'); + expect(rendered).toContain('evidence'); + expect(rendered).toContain('correct'); + expect(rendered).toContain('verified'); + }); + + // ============================================================ + // CODEX F3 REGRESSION GATE — DATA-LOSS BUG GUARD + // ============================================================ + // Without these tests, `gbrain takes update --row 2` on a page where row 1 + // is resolved would render only the 7-column shape on parse + render, + // silently deleting row 1's resolution cells on the next disk write. + // ============================================================ + test('REGRESSION (codex F3): round-trip preserves resolution fields on a resolved row', () => { + const { takes } = parseTakesFence(RESOLVED_BODY); + const rendered = renderTakesFence(takes); + const { takes: roundTripped } = parseTakesFence(rendered); + expect(roundTripped).toHaveLength(2); + expect(roundTripped[0].resolvedQuality).toBe('correct'); + expect(roundTripped[0].resolvedOutcome).toBe(true); + expect(roundTripped[0].resolvedAt).toBe('2026-04-30'); + expect(roundTripped[0].resolvedEvidence).toBe('Series A closed'); + expect(roundTripped[0].resolvedValue).toBe(50); + expect(roundTripped[0].resolvedUnit).toBe('usd'); + expect(roundTripped[0].resolvedBy).toBe('garry'); + }); + + test('REGRESSION (codex F3): updating an unrelated row preserves resolution on the resolved row', () => { + const { takes } = parseTakesFence(RESOLVED_BODY); + // Simulate cmdUpdate's spread pattern on row 2 (the unresolved one). + const updated = takes.map(t => + t.rowNum === 2 ? { ...t, weight: 0.95 } : t, + ); + const rendered = renderTakesFence(updated); + const { takes: after } = parseTakesFence(rendered); + // Row 1 (resolved) — resolution survives intact. + expect(after[0].resolvedQuality).toBe('correct'); + expect(after[0].resolvedEvidence).toBe('Series A closed'); + // Row 2 — weight changed, no resolution. + expect(after[1].weight).toBe(0.95); + expect(after[1].resolvedQuality).toBeUndefined(); + }); + + test('REGRESSION: parsing a v0.28-shape fence (no resolution columns) round-trips byte-identical narrow shape', () => { + const { takes } = parseTakesFence(SAMPLE_BODY); + const rendered = renderTakesFence(takes); + expect(rendered).not.toContain('quality'); + expect(rendered).not.toContain('| resolved |'); + // Re-parse verifies fidelity. + const { takes: roundTripped } = parseTakesFence(rendered); + expect(roundTripped).toHaveLength(takes.length); + for (let i = 0; i < takes.length; i++) { + expect(roundTripped[i].claim).toBe(takes[i].claim); + expect(roundTripped[i].weight).toBe(takes[i].weight); + expect(roundTripped[i].active).toBe(takes[i].active); + } + }); + + test('partial quality renders + parses correctly (outcome left empty)', () => { + const { takes } = parseTakesFence(SAMPLE_BODY); + takes[0] = { + ...takes[0], + resolvedAt: '2026-05-01', + resolvedQuality: 'partial', + resolvedOutcome: undefined, // partial has no boolean outcome + resolvedEvidence: 'kind of right', + }; + const rendered = renderTakesFence(takes); + expect(rendered).toContain('partial'); + const { takes: roundTripped } = parseTakesFence(rendered); + expect(roundTripped[0].resolvedQuality).toBe('partial'); + expect(roundTripped[0].resolvedOutcome).toBeUndefined(); + }); + + test('upsertTakeRow on a page with resolved rows preserves the resolution + uses wide shape', () => { + const { body: nextBody, rowNum } = upsertTakeRow(RESOLVED_BODY, { + claim: 'Brand new bet', + kind: 'bet', + holder: 'garry', + weight: 0.4, + active: true, + }); + expect(rowNum).toBe(3); + const { takes } = parseTakesFence(nextBody); + expect(takes).toHaveLength(3); + // Row 1's resolution survived the upsert. + expect(takes[0].resolvedQuality).toBe('correct'); + expect(takes[0].resolvedEvidence).toBe('Series A closed'); + // New row is unresolved. + expect(takes[2].resolvedQuality).toBeUndefined(); + // Wide shape was emitted (resolution columns visible). + expect(nextBody).toContain('quality'); + }); + + test('supersedeRow preserves resolution on the row being struck through', () => { + // Note: superseding a RESOLVED bet would normally throw at the engine + // layer (TAKE_RESOLVED_IMMUTABLE). The fence-layer supersedeRow doesn't + // enforce that — it just strikes through. The point of this test is + // that whatever resolution data lives on the old row survives the strike. + const { body: nextBody } = supersedeRow(RESOLVED_BODY, 1, { + claim: 'Updated bet', + kind: 'bet', + holder: 'garry', + weight: 0.5, + }); + const { takes } = parseTakesFence(nextBody); + const oldRow = takes.find(t => t.rowNum === 1)!; + expect(oldRow.active).toBe(false); + // Resolution data on the old (struck) row preserved. + expect(oldRow.resolvedQuality).toBe('correct'); + expect(oldRow.resolvedEvidence).toBe('Series A closed'); + }); +}); diff --git a/test/takes-resolution.test.ts b/test/takes-resolution.test.ts new file mode 100644 index 000000000..cabeb5ec7 --- /dev/null +++ b/test/takes-resolution.test.ts @@ -0,0 +1,158 @@ +/** + * v0.30.0 (Slice A1): tests for the pure resolution + scorecard helpers. + * Covers the (quality, outcome) tuple derivation and the Brier math + * including the partial-exclusion contract (D5 + D11). + */ + +import { describe, test, expect } from 'bun:test'; +import { + deriveResolutionTuple, + finalizeScorecard, + PARTIAL_RATE_WARNING_THRESHOLD, +} from '../src/core/takes-resolution.ts'; +import { GBrainError } from '../src/core/types.ts'; + +describe('deriveResolutionTuple', () => { + test('quality=correct → (correct, true)', () => { + expect(deriveResolutionTuple({ quality: 'correct', resolvedBy: 'garry' })).toEqual({ + quality: 'correct', + outcome: true, + }); + }); + + test('quality=incorrect → (incorrect, false)', () => { + expect(deriveResolutionTuple({ quality: 'incorrect', resolvedBy: 'garry' })).toEqual({ + quality: 'incorrect', + outcome: false, + }); + }); + + test('quality=partial → (partial, null)', () => { + expect(deriveResolutionTuple({ quality: 'partial', resolvedBy: 'garry' })).toEqual({ + quality: 'partial', + outcome: null, + }); + }); + + test('outcome=true (back-compat alias) → (correct, true)', () => { + expect(deriveResolutionTuple({ outcome: true, resolvedBy: 'garry' })).toEqual({ + quality: 'correct', + outcome: true, + }); + }); + + test('outcome=false (back-compat alias) → (incorrect, false)', () => { + expect(deriveResolutionTuple({ outcome: false, resolvedBy: 'garry' })).toEqual({ + quality: 'incorrect', + outcome: false, + }); + }); + + test('quality wins when both inputs supplied AND consistent', () => { + // outcome=true is consistent with quality=correct; quality wins. + expect(deriveResolutionTuple({ quality: 'correct', outcome: true, resolvedBy: 'garry' })).toEqual({ + quality: 'correct', + outcome: true, + }); + }); + + test('contradictory quality + outcome throws TAKE_RESOLUTION_INVALID', () => { + expect(() => + deriveResolutionTuple({ quality: 'correct', outcome: false, resolvedBy: 'garry' }) + ).toThrow(GBrainError); + expect(() => + deriveResolutionTuple({ quality: 'partial', outcome: true, resolvedBy: 'garry' }) + ).toThrow(GBrainError); + }); + + test('neither field set throws TAKE_RESOLUTION_INVALID', () => { + expect(() => + deriveResolutionTuple({ resolvedBy: 'garry' }) + ).toThrow(GBrainError); + }); +}); + +describe('finalizeScorecard (Brier math)', () => { + test('n=0: returns nulls, no divide-by-zero', () => { + const card = finalizeScorecard({ + total_bets: 0, resolved: 0, correct: 0, incorrect: 0, partial: 0, brier: null, + }); + expect(card).toEqual({ + total_bets: 0, + resolved: 0, + correct: 0, + incorrect: 0, + partial: 0, + accuracy: null, + brier: null, + partial_rate: null, + }); + }); + + test('all correct: accuracy=1, Brier reflects mean (weight - 1)^2', () => { + // 3 correct bets at weight 0.7. Per-row Brier = (0.7 - 1)^2 = 0.09. + // Mean = 0.09. Accuracy = 3/3 = 1.0. + const card = finalizeScorecard({ + total_bets: 3, resolved: 3, correct: 3, incorrect: 0, partial: 0, brier: 0.09, + }); + expect(card.accuracy).toBe(1.0); + expect(card.brier).toBeCloseTo(0.09, 5); + expect(card.partial_rate).toBe(0); + }); + + test('all incorrect: accuracy=0, Brier reflects mean weight^2', () => { + // 3 incorrect bets at weight 0.7. Per-row Brier = (0.7 - 0)^2 = 0.49. + const card = finalizeScorecard({ + total_bets: 3, resolved: 3, correct: 0, incorrect: 3, partial: 0, brier: 0.49, + }); + expect(card.accuracy).toBe(0.0); + expect(card.brier).toBeCloseTo(0.49, 5); + }); + + test('hand-calculated reference: 4 mixed bets', () => { + // bets: weight=0.9 correct, weight=0.6 correct, weight=0.7 incorrect, weight=0.4 incorrect + // Brier per row: (0.9-1)^2=0.01, (0.6-1)^2=0.16, (0.7-0)^2=0.49, (0.4-0)^2=0.16 + // Mean: (0.01+0.16+0.49+0.16)/4 = 0.205 + // Accuracy: 2/4 = 0.5 + const card = finalizeScorecard({ + total_bets: 4, resolved: 4, correct: 2, incorrect: 2, partial: 0, brier: 0.205, + }); + expect(card.accuracy).toBe(0.5); + expect(card.brier).toBeCloseTo(0.205, 5); + expect(card.partial_rate).toBe(0); + }); + + test('D5: partial excluded from Brier denominator; appears in partial_rate', () => { + // 2 correct, 1 incorrect, 1 partial. Brier reflects only the 3 binary rows + // (the SQL aggregation passes partial=null per row to AVG, so partial + // doesn't affect Brier in finalizeScorecard either). + // partial_rate = 1/4 = 0.25 + const card = finalizeScorecard({ + total_bets: 4, resolved: 4, correct: 2, incorrect: 1, partial: 1, brier: 0.18, + }); + // accuracy uses correct + incorrect denominator (binary), excluding partial. + expect(card.accuracy).toBeCloseTo(2 / 3, 5); + expect(card.partial_rate).toBe(0.25); + expect(card.brier).toBe(0.18); + }); + + test('D11: partial_rate threshold constant matches plan (20%)', () => { + expect(PARTIAL_RATE_WARNING_THRESHOLD).toBe(0.20); + }); + + test('all-partial scorecard: Brier null, accuracy null, partial_rate=1', () => { + const card = finalizeScorecard({ + total_bets: 5, resolved: 5, correct: 0, incorrect: 0, partial: 5, brier: null, + }); + expect(card.brier).toBeNull(); + expect(card.accuracy).toBeNull(); + expect(card.partial_rate).toBe(1); + }); + + test('partial_rate = 0 when no partial bets', () => { + const card = finalizeScorecard({ + total_bets: 2, resolved: 2, correct: 1, incorrect: 1, partial: 0, brier: 0.5, + }); + expect(card.partial_rate).toBe(0); + }); +});