mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.30.0 feat: calibration scorecards (Slice A1 of v0.30 wave) (#731)
* feat(schema): migration v40 — takes_resolved_quality + drift_decisions Slice A1 of the v0.30 wave. Bundles all wave schema in one migration so A2/B1/C1 carry no schema of their own (codex F6 schema-first ordering). - takes.resolved_quality TEXT with CHECK (correct/incorrect/partial). - takes_resolution_consistency CHECK enforces (quality, outcome) tuple consistency at the DB layer. partial → outcome=NULL. - One-shot backfill maps legacy resolved_outcome → resolved_quality so v0.28 brains keep working with no manual reclassification. - idx_takes_scorecard partial index on (holder, kind, resolved_quality) WHERE resolved_quality IS NOT NULL — scorecard hot path. - drift_decisions audit table (consumed by Slice C1 in v0.30.3). - PGLite branch via sqlFor.pglite mirrors the same shape; RLS DO-block is Postgres-only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(takes-fence): extend ParsedTake + parser + conditional renderer (codex F3) A codex consult on the v0.30 plan caught a real bug: the v0.28 parser had no concept of resolution columns, so every cmdUpdate after a cmdResolve silently deleted resolution data on the next render. This commit kills that data-loss path. ParsedTake gains optional resolvedAt, resolvedQuality, resolvedOutcome, resolvedEvidence, resolvedValue, resolvedUnit, resolvedBy. parseTakesFence detects v0.30-shape headers and reads resolution cells when present; v0.28 7-column fences round-trip byte-identical. renderTakesFence emits the resolution columns ONLY when at least one row on the page has resolvedQuality set — pages with no resolved rows keep the narrow shape exactly as before. 11 new test cases including the round-trip preservation regression gate. Without those tests, the silent-delete bug returns the moment the parser shape drifts. Tests cover: parsing v0.30 + v0.28 shapes, conditional rendering, partial quality round-trip, upsertTakeRow + supersedeRow preservation when a page already has resolved rows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(engine): getScorecard + getCalibrationCurve + 3-state TakeResolution Adds the calibration aggregate methods on BrainEngine. Both engines implement them with SQL-level allow-list filtering inside the GROUP BY (D4 fail-closed): hidden-holder rows contribute zero to aggregates. TakeResolution gains optional `quality` (correct|incorrect|partial). When both quality and outcome are supplied AND inconsistent, the engine throws TAKE_RESOLUTION_INVALID rather than silently overwriting. resolveTake writes both columns: quality directly, outcome derived (correct→true, incorrect→false, partial→NULL). Schema CHECK is the defense-in-depth backstop. Brier scope (D5 + D11): the SQL aggregation excludes partial rows from the Brier denominator — partial isn't a binary outcome to compare a probability against. partial_rate is reported alongside as a separate counter so hedging behavior stays visible. The 20% threshold lives in src/core/takes-resolution.ts and the CLI surfaces it in v0.30.0's cmdScorecard. New module src/core/takes-resolution.ts holds shared pure helpers (deriveResolutionTuple, finalizeScorecard) consumed by both engines so the math stays identical across backends. takeRowToTake (utils.ts) reads resolved_quality through to the Take row shape. 23 new test cases: 16 for the helpers (Brier hand-calc against a 4-bet reference at 0.205, n=0 no-divide, contradictory-input rejection, partial-exclusion contract, threshold constant); 7 against PGLite for the engine path (3-state quality writes, contradictory throws, scorecard hand-calc, n=0, SQL-level allow-list privacy filter). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): gbrain takes resolve --quality, takes scorecard, takes calibration cmdResolve widened: --quality correct|incorrect|partial is the new primary input. --outcome true|false stays as a back-compat alias auto-mapping to quality, with a stderr deprecation warning on use. Mutually exclusive with --quality. --evidence is a semantic alias for --source on the resolve subcommand. cmdResolve mirrors resolution metadata into the takes-fence on disk via the page-lock-aware path. Round-trip preservation through parseTakesFence + renderTakesFence keeps resolution data intact across unrelated edits to other rows on the same page. Removes the v0.28 deferred-rendering warning. cmdScorecard prints `correct | incorrect | partial`, accuracy, Brier (correct ∨ incorrect only; lower is better; 0.25 = always-50% baseline), and partial_rate. When partial_rate > 20% the CLI prints "[!] partial_rate is high — calibration may be optimistic" so hedging behavior stays visible even though it doesn't enter the math (D11). Small-N note when resolved < 100. JSON output via --json. cmdCalibration bins resolved correct/incorrect bets by stated weight (--bucket-size, default 0.1) and prints observed vs predicted vs delta per bucket. Diagonal alignment = perfect calibration. Both new subcommands wire allow-list as undefined for local CLI callers (trusted); MCP path will thread it from access_tokens.permissions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(mcp): register takes_scorecard + takes_calibration ops Both ops are read-scope, MCP-callable, allow-list-honoring. Handlers thread ctx.takesHoldersAllowList into the engine method's required allowList parameter, which applies WHERE holder = ANY at SQL aggregation level (D4 fail-closed). Local CLI callers leave the allow-list undefined and see all holders. Updates the OperationContext.takesHoldersAllowList contract comment to list the new aggregate ops alongside takes_list, takes_search, query. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.0 release: calibration core (Slice A1 of v0.30 wave) VERSION + package.json bump. CHANGELOG entry covers the release-summary (headline + math table + privacy note + data-loss-bug-killed note), "## To take advantage of v0.30.0" upgrade path, and itemized changes. llms-full.txt regenerated to capture the v0.28.x annotations that had been merged but not yet rolled into the docs bundle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): scorecard + calibration parity on real Postgres + NUMERIC fix Adds end-to-end coverage for v0.30.0 (Slice A1) against real Postgres: - test/e2e/takes-scorecard-parity.test.ts (new): seeds the same 6-bet fixture (4 binary garry + 1 partial garry + 1 binary harj) into both Postgres and PGLite, asserts getScorecard + getCalibrationCurve return byte-identical results across engines, runs the 4-bet hand-calc Brier reference (0.205) on real PG, and verifies the SQL-level allow-list filter strictly subtracts hidden-holder rows on both engines. - test/e2e/takes-postgres.test.ts: extended with 8 v0.30 cases — quality semantics (correct/partial/back-compat) writes the expected (quality, outcome) tuple on real PG; the takes_resolution_consistency CHECK constraint actually fires on a contradictory raw UPDATE; getScorecard + getCalibrationCurve coherent shape + ordered-bucket invariants; PRIVACY allow-list filter on real PG; MCP dispatch path for takes_scorecard + takes_calibration with allow-list threading. While writing the parity test, the e2e harness caught a real bug PGLite tolerated: postgres.js sends scalar `${bucketSize}` params as text by default, so `FLOOR(weight / $N)` tried to coerce '0.1' to integer and threw `invalid input syntax for type integer: "0.1"`. The NUMERIC fix also kills a separate FP-precision divergence — `FLOOR(0.7 / 0.1)` returns 6 on real PG (IEEE 754 rounds 0.7/0.1 to 6.9999...) and 7 on PGLite. Both engines now bucket via `weight::numeric / $N::numeric` which is exact decimal arithmetic and engine-agnostic. This is the v0.30.0 wave's first cross-engine parity test. Same shape will guard A2's getTrajectory + getAnnualReview when those land. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(changelog): correct v40→v43 reference and expand v0.30.0 test note Two fixes to the v0.30.0 entry after the master merge renumbered the migration: - The "#### Added" bullet still said "Schema migration v40"; bumped to v43. - The "#### Tests" section only enumerated unit tests. The PR also ships 19 E2E cases (11 in takes-scorecard-parity, 8 extending takes-postgres) that exercise the calibration math against real Postgres and the PG↔PGLite engine parity. Added the count + a note about the two real bugs the parity test caught (postgres.js string-typed scalar params and IEEE 754 bucketing divergence) that PGLite tolerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
8392d434a9
commit
1399e519c0
+123
-3
@@ -2,6 +2,129 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.30.0] - 2026-05-07
|
||||
|
||||
**Calibration scorecards land. Find out if your bets are actually as good as you think.**
|
||||
**Brier scores, prediction accuracy, calibration curves — for every bet you ever wrote down.**
|
||||
|
||||
This is the v0.30 wave's first ship: the calibration core. Resolve your bets with three
|
||||
states (correct, incorrect, partial) instead of just true/false. Run a scorecard against
|
||||
any holder, any domain prefix, any date window, and see your accuracy + Brier score in
|
||||
seconds. The calibration curve shows whether your "I'm 80% confident" bets actually come
|
||||
in at 80% — the thing Daniel Kahneman spent a career studying, now a query.
|
||||
|
||||
```bash
|
||||
gbrain takes resolve companies/some-yc-co --row 3 --quality correct --evidence "Series A closed at $50M"
|
||||
gbrain takes scorecard garry
|
||||
gbrain takes calibration garry --bucket-size 0.1
|
||||
```
|
||||
|
||||
The scorecard prints `correct/incorrect/partial`, accuracy, Brier (lower is better; 0.25
|
||||
is the always-50% baseline), and a `partial_rate` line. When more than 20% of resolved
|
||||
bets are "partial," the scorecard prints `[!] partial_rate is high — calibration may be
|
||||
optimistic`. Hedged bets escape the Brier denominator, so the warning surfaces hedging
|
||||
behavior as its own signal even though it doesn't enter the math.
|
||||
|
||||
### The math that matters
|
||||
|
||||
| Field | Meaning | Math |
|
||||
|-------|---------|------|
|
||||
| `accuracy` | correct / (correct + incorrect) | partial excluded |
|
||||
| `brier` | mean((weight − outcome)²) over correct ∨ incorrect | lower is better; 0 = perfect; 0.25 = always-50% baseline |
|
||||
| `partial_rate` | partial / resolved | hedging signal; warns at >20% |
|
||||
|
||||
Brier excludes partial deliberately. Codex pointed out that counting partial as 0.5 in
|
||||
the math would reward hedging into ambiguity; instead, partial bets leave the calibration
|
||||
denominator entirely AND surface as a separate counter. You can see whether you're
|
||||
miscalibrated AND whether you're hedging, side by side.
|
||||
|
||||
### Privacy: aggregate queries fail-closed
|
||||
|
||||
Scorecard and calibration are MCP-callable read ops (`takes_scorecard`,
|
||||
`takes_calibration`). Both engine methods take `takesHoldersAllowList` as a REQUIRED
|
||||
TypeScript parameter — the compiler is the first line of defense against accidentally
|
||||
exposing therapy-page takes to an MCP-bound agent via aggregate counts. Hidden-holder
|
||||
rows contribute zero to the math. Local CLI callers see all holders.
|
||||
|
||||
### Markdown stays canonical (and the silent-data-loss bug is dead)
|
||||
|
||||
Resolution metadata renders into the takes-fence on disk: `resolved | quality |
|
||||
evidence | value | unit | by` columns appear ONLY when at least one row on the page is
|
||||
resolved. Pages with no resolved rows keep the narrow 7-column shape exactly as before.
|
||||
|
||||
A codex consult on the v0.30 plan caught a real bug in the original draft: the v0.28
|
||||
parser/renderer had no concept of resolution columns. Without an explicit fix, every
|
||||
`gbrain takes update` after a `takes resolve` would silently delete the resolution
|
||||
data on the next disk write. The new round-trip preservation tests
|
||||
(`test/takes-fence.test.ts` v0.30 section) are the regression gate; they fail loudly
|
||||
if the data-loss bug returns.
|
||||
|
||||
### To take advantage of v0.30.0
|
||||
|
||||
`gbrain upgrade` runs the schema migration automatically. If `gbrain doctor` warns
|
||||
about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Resolve a bet you already have:**
|
||||
```bash
|
||||
gbrain takes resolve <slug> --row N --quality correct|incorrect|partial --evidence "..."
|
||||
```
|
||||
3. **Run your first scorecard:**
|
||||
```bash
|
||||
gbrain takes scorecard garry
|
||||
```
|
||||
4. **Existing v0.28-resolved bets are auto-backfilled.** The migration maps legacy
|
||||
`resolved_outcome=true` → `resolved_quality='correct'`, `false` → `'incorrect'`.
|
||||
No manual reclassification needed.
|
||||
5. **The `--outcome true|false` flag still works** as a back-compat alias on `takes
|
||||
resolve`, with a stderr deprecation warning. Cannot express partial; mutually
|
||||
exclusive with `--quality`.
|
||||
|
||||
If any step fails or the numbers look wrong, file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and
|
||||
`~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
- New CLI: `gbrain takes scorecard [<holder>] [--domain <prefix>] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--json]` — calibration scorecard.
|
||||
- New CLI: `gbrain takes calibration [<holder>] [--bucket-size 0.1] [--json]` — calibration curve binned by stated weight.
|
||||
- New flag: `--quality correct|incorrect|partial` on `gbrain takes resolve`. Primary input.
|
||||
- New flag: `--evidence "..."` on `gbrain takes resolve` — semantic alias for `--source`.
|
||||
- New MCP ops: `takes_scorecard` (read scope) + `takes_calibration` (read scope), both honoring per-token allow-list.
|
||||
- Schema migration v43 (`takes_resolved_quality_and_drift_decisions`):
|
||||
- `takes.resolved_quality TEXT` with CHECK `(IN ('correct','incorrect','partial'))`.
|
||||
- `takes_resolution_consistency` CHECK constraint enforces (quality, outcome) tuple consistency at the DB layer.
|
||||
- Backfill: legacy `resolved_outcome=true` → `resolved_quality='correct'`; `false` → `'incorrect'`.
|
||||
- Partial index `idx_takes_scorecard ON takes (holder, kind, resolved_quality) WHERE resolved_quality IS NOT NULL` — scorecard hot path.
|
||||
- New table `drift_decisions` (audit log for the upcoming v0.30.3 drift LLM judge; defined now so C1 carries no migration).
|
||||
- New module `src/core/takes-resolution.ts` — pure helpers (`deriveResolutionTuple`, `finalizeScorecard`, `PARTIAL_RATE_WARNING_THRESHOLD`) shared between Postgres + PGLite engines.
|
||||
|
||||
#### Changed
|
||||
- `gbrain takes resolve` now writes both `resolved_outcome` (boolean) and `resolved_quality` (text) on every resolution. Schema CHECK is the defense-in-depth backstop.
|
||||
- `--outcome true|false` becomes a back-compat alias with a stderr deprecation warning. Mutually exclusive with `--quality`. Cannot express partial.
|
||||
- `Take` interface (`src/core/engine.ts`) adds `resolved_quality: 'correct' | 'incorrect' | 'partial' | null`.
|
||||
- `TakeResolution` adds optional `quality` field. `outcome` stays as the back-compat input. When both set and inconsistent, throws `TAKE_RESOLUTION_INVALID` instead of silently overwriting.
|
||||
- `BrainEngine` interface gains `getScorecard(opts, allowList)` and `getCalibrationCurve(opts, allowList)`. Both implemented in `postgres-engine.ts` + `pglite-engine.ts` with SQL-level `WHERE holder = ANY($allowList)` filtering inside the GROUP BY.
|
||||
- `ParsedTake` (`src/core/takes-fence.ts`) extended with optional `resolvedAt`, `resolvedQuality`, `resolvedOutcome`, `resolvedEvidence`, `resolvedValue`, `resolvedUnit`, `resolvedBy`. Parser detects v0.30-shape headers; renderer emits resolution columns only when at least one row has `resolvedQuality !== undefined`.
|
||||
- `cmdResolve` mirrors resolution metadata into the takes-fence on disk via the page-lock-aware path. Round-trip preservation keeps resolution data intact across unrelated edits.
|
||||
|
||||
#### Tests
|
||||
53 new cases (34 unit + 19 E2E):
|
||||
- 11 cases in `test/takes-fence.test.ts` for v0.30 resolution columns, round-trip preservation (the codex F3 regression gate), narrow-shape preservation, partial rendering, `upsertTakeRow` and `supersedeRow` resolution preservation.
|
||||
- 16 cases in `test/takes-resolution.test.ts` for `deriveResolutionTuple` and `finalizeScorecard` Brier math (n=0, hand-calculated 4-bet reference at Brier=0.205, partial-exclusion contract, partial_rate threshold).
|
||||
- 7 cases in `test/takes-engine.test.ts` for v0.30 quality semantics on PGLite, contradictory-input rejection, scorecard hand-calc against the same 4-bet fixture, n=0 handling, SQL-level allow-list privacy filter.
|
||||
- 11 cases in `test/e2e/takes-scorecard-parity.test.ts` (NEW) — same fixture into Postgres + PGLite, asserts `getScorecard` and `getCalibrationCurve` return byte-identical results across engines, and that the SQL-level allow-list filter strictly subtracts hidden-holder rows on both engines.
|
||||
- 8 cases extended in `test/e2e/takes-postgres.test.ts` — `--quality correct/partial/back-compat` writes the expected (quality, outcome) tuple on real PG; the `takes_resolution_consistency` CHECK constraint actively rejects contradictory raw `UPDATE`s; scorecard + calibration coherent shape; PRIVACY allow-list filter on real PG; MCP dispatch path for `takes_scorecard` + `takes_calibration`.
|
||||
|
||||
The E2E parity test caught two real bugs PGLite tolerated: postgres.js was sending `${bucketSize}` as a string and `FLOOR(weight / '0.1')` was throwing `invalid input syntax for type integer`; even after the type cast, IEEE 754 made `FLOOR(0.7 / 0.1)` return 6 on real PG and 7 on PGLite, so calibration buckets diverged at boundaries. Both fixed with `weight::numeric / $N::numeric` (exact decimal arithmetic, engine-agnostic).
|
||||
|
||||
#### Migration ordering
|
||||
- All wave schema (resolved_quality, CHECK, partial index, drift_decisions table) bundled into migration v43 in v0.30.0 (renumbered from v40 on master merge — master claimed v40-v42 with the v0.29 + v0.29.1 salience-and-recency wave). Migration runner sorts by version number, so the rename is mechanical — no semantic change. Slices A2/B1/C1 (trajectory + annual-review, meeting extraction CLI, drift judge) add no migrations of their own.
|
||||
|
||||
## [0.29.2] - 2026-05-07
|
||||
|
||||
**Thin-client mode: install gbrain on a laptop without a local DB and have it consume a remote brain over MCP.**
|
||||
@@ -542,9 +665,6 @@ gbrain eval longmemeval ~/datasets/longmemeval/longmemeval_s.json \
|
||||
If anything looks off, file at https://github.com/garrytan/gbrain/issues
|
||||
with `gbrain doctor` output.
|
||||
|
||||
|
||||
|
||||
|
||||
## [0.28.11] - 2026-05-07
|
||||
|
||||
**Mix providers: OpenAI for text, Voyage for images. One brain, two embedding pipelines.**
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.29.2",
|
||||
"version": "0.30.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+198
-13
@@ -305,29 +305,206 @@ async function cmdSupersede(engine: BrainEngine, args: string[]): Promise<void>
|
||||
async function cmdResolve(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const slug = args[0];
|
||||
const rowNumStr = flagValue(args, '--row');
|
||||
const qualityStr = flagValue(args, '--quality');
|
||||
const outcomeStr = flagValue(args, '--outcome');
|
||||
if (!slug || !rowNumStr || !outcomeStr) {
|
||||
console.error('Usage: gbrain takes resolve <slug> --row N --outcome true|false [--value N --unit usd|pct|count] [--source "..."] [--by <slug>]');
|
||||
if (!slug || !rowNumStr || (!qualityStr && !outcomeStr)) {
|
||||
console.error('Usage: gbrain takes resolve <slug> --row N --quality correct|incorrect|partial [--evidence "..."] [--value N --unit usd|pct|count] [--by <slug>]');
|
||||
console.error(' (back-compat) gbrain takes resolve <slug> --row N --outcome true|false [...]');
|
||||
process.exit(1);
|
||||
}
|
||||
if (qualityStr && outcomeStr) {
|
||||
console.error('Error: --quality and --outcome are mutually exclusive (choose one).');
|
||||
process.exit(1);
|
||||
}
|
||||
const rowNum = parseInt(rowNumStr, 10);
|
||||
const outcome = outcomeStr === 'true';
|
||||
|
||||
// v0.30.0: --quality is the new primary input. --outcome stays as a back-compat
|
||||
// alias auto-mapping true→correct / false→incorrect; cannot express partial.
|
||||
let quality: 'correct' | 'incorrect' | 'partial' | undefined;
|
||||
let outcome: boolean | undefined;
|
||||
if (qualityStr) {
|
||||
if (qualityStr !== 'correct' && qualityStr !== 'incorrect' && qualityStr !== 'partial') {
|
||||
console.error(`Invalid --quality "${qualityStr}". Expected: correct, incorrect, partial.`);
|
||||
process.exit(1);
|
||||
}
|
||||
quality = qualityStr;
|
||||
} else if (outcomeStr) {
|
||||
if (outcomeStr !== 'true' && outcomeStr !== 'false') {
|
||||
console.error(`Invalid --outcome "${outcomeStr}". Expected: true or false.`);
|
||||
process.exit(1);
|
||||
}
|
||||
outcome = outcomeStr === 'true';
|
||||
console.error('[deprecated] --outcome is the v0.28 alias for --quality. Prefer --quality correct|incorrect|partial in new scripts.');
|
||||
}
|
||||
|
||||
const valueStr = flagValue(args, '--value');
|
||||
const value = valueStr === undefined ? undefined : parseFloat(valueStr);
|
||||
const unit = flagValue(args, '--unit');
|
||||
const source = flagValue(args, '--source');
|
||||
// --evidence is the v0.30.0 alias for --source on the resolve subcommand
|
||||
// (semantic clarity: "what evidence resolved this bet?").
|
||||
const source = flagValue(args, '--evidence') ?? flagValue(args, '--source');
|
||||
const resolvedBy = flagValue(args, '--by') ?? 'garry';
|
||||
const dirArg = flagValue(args, '--dir');
|
||||
|
||||
const pageId = await getPageId(engine, slug);
|
||||
await engine.resolveTake(pageId, rowNum, {
|
||||
quality,
|
||||
outcome,
|
||||
value,
|
||||
unit,
|
||||
source,
|
||||
resolvedBy,
|
||||
});
|
||||
console.log(`Resolved take #${rowNum} on ${slug}: outcome=${outcome}${valueStr ? ` value=${value}${unit ? ` ${unit}` : ''}` : ''}.`);
|
||||
console.log(`(Markdown rendering of resolution metadata: deferred to v0.29 — DB stores it; takes-fence renderer doesn't yet surface resolved_* in the table.)`);
|
||||
|
||||
// Mirror resolution into the markdown fence so the page is self-describing.
|
||||
// The renderer conditionally widens the table to 13 columns when at least one
|
||||
// row has resolution data; pages with no resolved rows keep the 7-col shape.
|
||||
// Round-trip via parseTakesFence + renderTakesFence preserves all rows.
|
||||
const brainDir = await resolveBrainDir(engine, dirArg ?? null);
|
||||
await withPageLock(slug, async () => {
|
||||
const path = pageFilePath(brainDir, slug);
|
||||
const body = readBodyOrEmpty(path);
|
||||
if (!body) {
|
||||
console.warn(`[takes resolve] markdown file not found at ${path}; DB updated but on-disk page absent.`);
|
||||
return;
|
||||
}
|
||||
const { parseTakesFence, renderTakesFence, TAKES_FENCE_BEGIN, TAKES_FENCE_END } = await import('../core/takes-fence.ts');
|
||||
const parsed = parseTakesFence(body);
|
||||
const target = parsed.takes.find(t => t.rowNum === rowNum);
|
||||
if (!target) {
|
||||
console.warn(`[takes resolve] DB updated but row #${rowNum} not in markdown fence; run 'gbrain extract takes --slugs ${slug}' to reconcile.`);
|
||||
return;
|
||||
}
|
||||
// Derive resolved fields from the inputs. Mirror the engine semantics:
|
||||
// quality wins when both set; partial → outcome=null.
|
||||
const finalQuality = quality ?? (outcome === true ? 'correct' : outcome === false ? 'incorrect' : undefined);
|
||||
if (!finalQuality) return; // unreachable — covered by earlier validation
|
||||
const finalOutcome = finalQuality === 'partial' ? undefined
|
||||
: finalQuality === 'correct' ? true : false;
|
||||
const updated = {
|
||||
...target,
|
||||
resolvedAt: new Date().toISOString().slice(0, 10),
|
||||
resolvedQuality: finalQuality,
|
||||
resolvedOutcome: finalOutcome,
|
||||
resolvedEvidence: source,
|
||||
resolvedValue: value,
|
||||
resolvedUnit: unit,
|
||||
resolvedBy,
|
||||
};
|
||||
const allRows = parsed.takes.map(t => t.rowNum === rowNum ? updated : t);
|
||||
const newFence = renderTakesFence(allRows);
|
||||
const beginIdx = body.indexOf(TAKES_FENCE_BEGIN);
|
||||
const endIdx = body.indexOf(TAKES_FENCE_END, beginIdx + TAKES_FENCE_BEGIN.length);
|
||||
const out = body.slice(0, beginIdx) + newFence + body.slice(endIdx + TAKES_FENCE_END.length);
|
||||
writeBody(path, out);
|
||||
});
|
||||
|
||||
const finalQuality = quality ?? (outcome === true ? 'correct' : outcome === false ? 'incorrect' : 'unknown');
|
||||
const valueSummary = valueStr ? ` value=${value}${unit ? ` ${unit}` : ''}` : '';
|
||||
console.log(`Resolved take #${rowNum} on ${slug}: quality=${finalQuality}${valueSummary}.`);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.30.0: aggregate calibration scorecard for a holder.
|
||||
*
|
||||
* Brier scope (D5+D11): partial bets are excluded from Brier — partial
|
||||
* isn't a binary outcome to compare a probability against. The partial_rate
|
||||
* counter reports the rate as a separate signal so hedging behavior stays
|
||||
* visible even though it doesn't enter the calibration math. When the rate
|
||||
* exceeds 20% the CLI prints a warning line; calibration on a hedge-heavy
|
||||
* scorecard is artificially clean, and the user should know.
|
||||
*/
|
||||
async function cmdScorecard(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = flagPresent(args, '--json');
|
||||
const holder = args[0] && !args[0].startsWith('--') ? args[0] : flagValue(args, '--holder');
|
||||
const domainPrefix = flagValue(args, '--domain');
|
||||
const since = flagValue(args, '--since');
|
||||
const until = flagValue(args, '--until');
|
||||
const { PARTIAL_RATE_WARNING_THRESHOLD } = await import('../core/takes-resolution.ts');
|
||||
|
||||
const card = await engine.getScorecard(
|
||||
{ holder, domainPrefix, since, until },
|
||||
/* allowList */ undefined, // CLI is local + trusted; MCP path threads allowList from the caller
|
||||
);
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify(card, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (card.resolved === 0) {
|
||||
console.log(`No resolved bets yet${holder ? ` for ${holder}` : ''}.`);
|
||||
return;
|
||||
}
|
||||
const fmt = (n: number | null, digits = 3) => n === null ? '—' : n.toFixed(digits);
|
||||
console.log(`# Scorecard${holder ? ` — ${holder}` : ''}`);
|
||||
if (domainPrefix) console.log(`Scope: domain=${domainPrefix}`);
|
||||
if (since || until) console.log(`Window: ${since ?? 'all'} → ${until ?? 'now'}`);
|
||||
console.log('');
|
||||
console.log(` total bets: ${card.total_bets}`);
|
||||
console.log(` resolved: ${card.resolved}`);
|
||||
console.log(` correct: ${card.correct}`);
|
||||
console.log(` incorrect: ${card.incorrect}`);
|
||||
console.log(` partial: ${card.partial}`);
|
||||
console.log(` accuracy: ${fmt(card.accuracy)}`);
|
||||
console.log(` Brier: ${fmt(card.brier, 4)} (correct ∨ incorrect only; lower is better; 0.25 = always-50% baseline)`);
|
||||
console.log(` partial_rate: ${fmt(card.partial_rate)}`);
|
||||
if (card.partial_rate !== null && card.partial_rate > PARTIAL_RATE_WARNING_THRESHOLD) {
|
||||
console.log('');
|
||||
console.log(` [!] partial_rate is high (>${(PARTIAL_RATE_WARNING_THRESHOLD * 100).toFixed(0)}%) — calibration may be optimistic.`);
|
||||
console.log(` Hedged bets escape the Brier denominator. Resolve them more decisively if the data supports it.`);
|
||||
}
|
||||
if (card.resolved < 100) {
|
||||
console.log('');
|
||||
console.log(` Note: n=${card.resolved} is small. Brier is noisy below ~100 resolved bets.`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.30.0: calibration curve. Bins resolved correct+incorrect bets by stated
|
||||
* weight and reports observed vs predicted frequency per bucket. The diagonal
|
||||
* (observed ≈ predicted in every bucket) is perfect calibration.
|
||||
*/
|
||||
async function cmdCalibration(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = flagPresent(args, '--json');
|
||||
const holder = args[0] && !args[0].startsWith('--') ? args[0] : flagValue(args, '--holder');
|
||||
const bucketSizeStr = flagValue(args, '--bucket-size');
|
||||
const bucketSize = bucketSizeStr === undefined ? 0.1 : parseFloat(bucketSizeStr);
|
||||
if (!Number.isFinite(bucketSize) || bucketSize <= 0 || bucketSize > 1) {
|
||||
console.error(`Invalid --bucket-size "${bucketSizeStr}". Expected a number in (0, 1].`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const buckets = await engine.getCalibrationCurve(
|
||||
{ holder, bucketSize },
|
||||
/* allowList */ undefined,
|
||||
);
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify(buckets, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (buckets.length === 0) {
|
||||
console.log(`No resolved correct/incorrect bets yet${holder ? ` for ${holder}` : ''}.`);
|
||||
return;
|
||||
}
|
||||
console.log(`# Calibration curve${holder ? ` — ${holder}` : ''}`);
|
||||
console.log(`Bucket size: ${bucketSize}`);
|
||||
console.log('');
|
||||
console.log(` bucket n observed predicted delta`);
|
||||
console.log(` --------------- ----- --------- ---------- -------`);
|
||||
const fmt = (n: number | null) => n === null ? ' —' : n.toFixed(3);
|
||||
for (const b of buckets) {
|
||||
const range = `${b.bucket_lo.toFixed(2)}-${b.bucket_hi.toFixed(2)}`.padEnd(15);
|
||||
const nStr = String(b.n).padStart(5);
|
||||
const obs = fmt(b.observed).padStart(9);
|
||||
const pred = fmt(b.predicted).padStart(10);
|
||||
const delta = b.observed !== null && b.predicted !== null
|
||||
? (b.observed - b.predicted).toFixed(3).padStart(7)
|
||||
: ' —';
|
||||
console.log(` ${range} ${nStr} ${obs} ${pred} ${delta}`);
|
||||
}
|
||||
}
|
||||
|
||||
// --- Dispatcher ---
|
||||
@@ -348,8 +525,14 @@ Subcommands:
|
||||
Update mutable fields
|
||||
takes supersede <slug> --row N --claim "..." [--kind k] [--who h] [--weight 0.5] [--source "..."]
|
||||
Strikethrough old + append new
|
||||
takes resolve <slug> --row N --outcome true|false [--value N --unit usd|pct|count] [--source "..."] [--by <slug>]
|
||||
Record bet resolution (immutable)
|
||||
takes resolve <slug> --row N --quality correct|incorrect|partial
|
||||
[--evidence "..."] [--value N --unit usd|pct|count] [--by <slug>]
|
||||
Record bet resolution (immutable, v0.30.0)
|
||||
Back-compat: --outcome true|false (deprecated alias)
|
||||
takes scorecard [<holder>] [--domain <prefix>] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--json]
|
||||
Aggregate calibration scorecard (v0.30.0)
|
||||
takes calibration [<holder>] [--bucket-size 0.1] [--json]
|
||||
Calibration curve binned by stated weight (v0.30.0)
|
||||
|
||||
Common flags:
|
||||
--dir <path> Override the brain directory (default: sync.repo_path config)
|
||||
@@ -362,11 +545,13 @@ Common flags:
|
||||
const rest = args.slice(1);
|
||||
|
||||
switch (sub) {
|
||||
case 'search': return cmdSearch(engine, rest);
|
||||
case 'add': return cmdAdd(engine, rest);
|
||||
case 'update': return cmdUpdate(engine, rest);
|
||||
case 'supersede': return cmdSupersede(engine, rest);
|
||||
case 'resolve': return cmdResolve(engine, rest);
|
||||
case 'search': return cmdSearch(engine, rest);
|
||||
case 'add': return cmdAdd(engine, rest);
|
||||
case 'update': return cmdUpdate(engine, rest);
|
||||
case 'supersede': return cmdSupersede(engine, rest);
|
||||
case 'resolve': return cmdResolve(engine, rest);
|
||||
case 'scorecard': return cmdScorecard(engine, rest);
|
||||
case 'calibration': return cmdCalibration(engine, rest);
|
||||
default:
|
||||
// No subcommand keyword → treat first arg as <slug> for the list path.
|
||||
return cmdList(engine, args);
|
||||
|
||||
+99
-1
@@ -167,6 +167,14 @@ export interface Take {
|
||||
active: boolean;
|
||||
resolved_at: string | null;
|
||||
resolved_outcome: boolean | null;
|
||||
/**
|
||||
* v0.30.0: 3-state outcome label. Sits alongside `resolved_outcome` for
|
||||
* back-compat. New writes populate both; legacy v0.28-resolved rows have
|
||||
* `resolved_quality` backfilled by migration v40 from the boolean.
|
||||
* Null on unresolved rows. Schema CHECK enforces (quality, outcome) consistency:
|
||||
* `correct` ↔ `outcome=true`, `incorrect` ↔ `outcome=false`, `partial` ↔ `outcome=NULL`.
|
||||
*/
|
||||
resolved_quality: 'correct' | 'incorrect' | 'partial' | null;
|
||||
resolved_value: number | null;
|
||||
resolved_unit: string | null;
|
||||
resolved_source: string | null;
|
||||
@@ -212,13 +220,73 @@ export interface StaleTakeRow {
|
||||
|
||||
/** Resolution metadata for resolveTake. */
|
||||
export interface TakeResolution {
|
||||
outcome: boolean;
|
||||
/**
|
||||
* v0.30.0: primary 3-state input. When set, takes precedence over `outcome`
|
||||
* and the engine writes both columns (quality directly; outcome derived:
|
||||
* `correct→true`, `incorrect→false`, `partial→null`).
|
||||
*/
|
||||
quality?: 'correct' | 'incorrect' | 'partial';
|
||||
/**
|
||||
* v0.28 back-compat input. Keep submitting for v0.28 callers; the engine
|
||||
* derives quality (`true→correct`, `false→incorrect`). When `quality` is
|
||||
* also set, `quality` wins. When neither is set, the engine throws.
|
||||
* Mutually-exclusive with `quality === 'partial'` because partial isn't
|
||||
* binary.
|
||||
*/
|
||||
outcome?: boolean;
|
||||
value?: number;
|
||||
unit?: string; // 'usd' | 'pct' | 'count' | other
|
||||
source?: string;
|
||||
resolvedBy: string; // slug or 'garry'
|
||||
}
|
||||
|
||||
/** v0.30.0: scorecard aggregate. */
|
||||
export interface TakesScorecard {
|
||||
total_bets: number;
|
||||
resolved: number;
|
||||
correct: number;
|
||||
incorrect: number;
|
||||
partial: number;
|
||||
/** Accuracy = correct / (correct + incorrect). NULL when n=0. */
|
||||
accuracy: number | null;
|
||||
/**
|
||||
* Brier score over rows where `resolved_quality IN ('correct','incorrect')`.
|
||||
* Maps `correct→1`, `incorrect→0`, computes `mean((weight − outcome)²)`.
|
||||
* Lower is better; 0 = perfect; 0.25 = always-50% baseline.
|
||||
* Excludes partial — that label hides hedging behavior; `partial_rate`
|
||||
* surfaces it as a separate signal. NULL when no correct+incorrect rows.
|
||||
*/
|
||||
brier: number | null;
|
||||
/** partial / resolved. NULL when n=0. */
|
||||
partial_rate: number | null;
|
||||
}
|
||||
|
||||
export interface TakesScorecardOpts {
|
||||
holder?: string;
|
||||
domainPrefix?: string; // e.g. 'companies/' to scope the scorecard
|
||||
since?: string; // ISO date 'YYYY-MM-DD'
|
||||
until?: string; // ISO date 'YYYY-MM-DD'
|
||||
}
|
||||
|
||||
/** v0.30.0: calibration curve bucket. */
|
||||
export interface CalibrationBucket {
|
||||
/** Lower bound of the weight bucket, inclusive. */
|
||||
bucket_lo: number;
|
||||
/** Upper bound, exclusive (except for the final bucket which is inclusive of 1.0). */
|
||||
bucket_hi: number;
|
||||
/** Count of resolved correct+incorrect bets falling in this weight range. */
|
||||
n: number;
|
||||
/** correct / n. NULL when n=0. */
|
||||
observed: number | null;
|
||||
/** mean(weight) within the bucket — what was predicted on average. NULL when n=0. */
|
||||
predicted: number | null;
|
||||
}
|
||||
|
||||
export interface CalibrationCurveOpts {
|
||||
holder?: string;
|
||||
bucketSize?: number; // default 0.1
|
||||
}
|
||||
|
||||
/** Synthesis evidence row input (provenance from think synthesis pages). */
|
||||
export interface SynthesisEvidenceInput {
|
||||
synthesis_page_id: number;
|
||||
@@ -559,9 +627,39 @@ export interface BrainEngine {
|
||||
/**
|
||||
* Resolve a bet (or take). Sets resolved_* columns. Immutable: re-resolve
|
||||
* attempts throw `TAKE_ALREADY_RESOLVED`. Use supersede to express a new bet.
|
||||
*
|
||||
* v0.30.0: accepts either `quality` (3-state, primary) or `outcome` (boolean,
|
||||
* back-compat). When both set, `quality` wins. The engine writes BOTH columns
|
||||
* derived from whichever input was given: `quality='correct'/'incorrect'` →
|
||||
* `outcome=true/false`; `quality='partial'` → `outcome=NULL`. The schema
|
||||
* `takes_resolution_consistency` CHECK constraint catches contradictory
|
||||
* states at the DB layer as a defense-in-depth backstop.
|
||||
*/
|
||||
resolveTake(pageId: number, rowNum: number, resolution: TakeResolution): Promise<void>;
|
||||
|
||||
/**
|
||||
* v0.30.0: aggregate calibration scorecard. Pure SQL aggregation; no LLM.
|
||||
* Counts resolved bets, computes accuracy, Brier score (correct+incorrect
|
||||
* only), and `partial_rate`. Filtering: `holder` scopes to one identity;
|
||||
* `domainPrefix` scopes to a slug-prefix (e.g. `companies/`); `since`/`until`
|
||||
* scope to a `since_date` window.
|
||||
*
|
||||
* Privacy (D4 from plan): `allowList` is REQUIRED in the TS signature.
|
||||
* The engine applies `WHERE holder = ANY($allowList)` INSIDE the GROUP BY
|
||||
* so hidden-holder rows contribute zero to aggregates. Pass an empty array
|
||||
* to enforce zero-results; pass `undefined` only from server-side trusted
|
||||
* callers that have already verified the request is unrestricted.
|
||||
*/
|
||||
getScorecard(opts: TakesScorecardOpts, allowList: string[] | undefined): Promise<TakesScorecard>;
|
||||
|
||||
/**
|
||||
* v0.30.0: calibration curve. Bins resolved correct+incorrect bets by stated
|
||||
* weight (default bucket size 0.1) and reports observed vs predicted frequency
|
||||
* per bucket. Same allow-list contract as `getScorecard`. Excludes partial
|
||||
* (consistent with Brier — partial has no binary outcome to compare against).
|
||||
*/
|
||||
getCalibrationCurve(opts: CalibrationCurveOpts, allowList: string[] | undefined): Promise<CalibrationBucket[]>;
|
||||
|
||||
/** Persist think provenance. ON CONFLICT DO NOTHING; returns rows inserted. */
|
||||
addSynthesisEvidence(rows: SynthesisEvidenceInput[]): Promise<number>;
|
||||
|
||||
|
||||
@@ -1935,6 +1935,138 @@ export const MIGRATIONS: Migration[] = [
|
||||
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_source TEXT;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 43,
|
||||
name: 'takes_resolved_quality_and_drift_decisions',
|
||||
// v0.30.0 (Slice A1, Universal Takes Epistemology wave). Bundles ALL schema
|
||||
// for the v0.30 release wave so A2/B1/C1 add no migrations (codex F6 fix:
|
||||
// schema-first ordering eliminates the cross-lane migrate.ts contention).
|
||||
// Originally landed as v40 in the v0.30.0 branch; renumbered to v43 on
|
||||
// merge with master after master claimed v40-v42 with the v0.29 +
|
||||
// v0.29.1 salience-and-recency wave. Migration runner sorts by version
|
||||
// number, so renumbering is a pure-rename — no semantic change.
|
||||
//
|
||||
// 1. takes.resolved_quality TEXT — 3-state outcome label (correct/incorrect/
|
||||
// partial) sitting alongside existing resolved_outcome BOOLEAN. Boolean
|
||||
// stays for back-compat reads; quality is the new source of truth for
|
||||
// calibration math. Backfill maps legacy resolved_outcome → quality.
|
||||
//
|
||||
// 2. takes_resolution_consistency CHECK constraint — fails contradictory
|
||||
// states like (quality='correct', outcome=false). 'partial' maps to
|
||||
// outcome=NULL because partial isn't a binary outcome. Added AFTER the
|
||||
// backfill so existing rows pass.
|
||||
//
|
||||
// 3. idx_takes_scorecard partial index on (holder, kind, resolved_quality)
|
||||
// WHERE resolved_quality IS NOT NULL — scorecard hot path. ~5KB on a
|
||||
// 50K-row brain; makes scorecard O(log n) instead of full scan.
|
||||
//
|
||||
// 4. drift_decisions audit table — consumed by Slice C1 (v0.30.3) when
|
||||
// drift LLM judge ships. Defined here so C1 carries no migration.
|
||||
// Sized for one row per drift recommendation (insert-only, never
|
||||
// updated except for applied_at/applied_by when --auto-update lands).
|
||||
sql: `
|
||||
-- Step 1: add resolved_quality column with kind-of-outcome CHECK.
|
||||
-- The (quality, outcome) consistency constraint comes AFTER the backfill
|
||||
-- (Step 3) so existing legacy rows don't fail the new constraint.
|
||||
ALTER TABLE takes
|
||||
ADD COLUMN IF NOT EXISTS resolved_quality TEXT
|
||||
CHECK (resolved_quality IS NULL OR resolved_quality IN ('correct','incorrect','partial'));
|
||||
|
||||
-- Step 2: backfill from legacy boolean. Idempotent: only writes rows
|
||||
-- where quality is still NULL and outcome is set. Re-runs are no-ops.
|
||||
UPDATE takes
|
||||
SET resolved_quality = CASE resolved_outcome
|
||||
WHEN true THEN 'correct'
|
||||
WHEN false THEN 'incorrect'
|
||||
END
|
||||
WHERE resolved_outcome IS NOT NULL AND resolved_quality IS NULL;
|
||||
|
||||
-- Step 3: (quality, outcome) consistency constraint. Drop-then-recreate
|
||||
-- so re-runs converge. The named constraint lets us evolve it later.
|
||||
ALTER TABLE takes DROP CONSTRAINT IF EXISTS takes_resolution_consistency;
|
||||
ALTER TABLE takes ADD CONSTRAINT takes_resolution_consistency CHECK (
|
||||
(resolved_quality IS NULL AND resolved_outcome IS NULL)
|
||||
OR (resolved_quality = 'correct' AND resolved_outcome = true)
|
||||
OR (resolved_quality = 'incorrect' AND resolved_outcome = false)
|
||||
OR (resolved_quality = 'partial' AND resolved_outcome IS NULL)
|
||||
);
|
||||
|
||||
-- Step 4: scorecard hot path. Partial index keeps footprint proportional
|
||||
-- to resolved-take count, not table size.
|
||||
CREATE INDEX IF NOT EXISTS idx_takes_scorecard
|
||||
ON takes (holder, kind, resolved_quality)
|
||||
WHERE resolved_quality IS NOT NULL;
|
||||
|
||||
-- Step 5: drift_decisions audit table (consumed by Slice C1 in v0.30.3).
|
||||
CREATE TABLE IF NOT EXISTS drift_decisions (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
take_id BIGINT NOT NULL REFERENCES takes(id) ON DELETE CASCADE,
|
||||
page_id INTEGER NOT NULL,
|
||||
row_num INTEGER NOT NULL,
|
||||
recommended_weight REAL NOT NULL CHECK (recommended_weight >= 0 AND recommended_weight <= 1),
|
||||
reasoning TEXT,
|
||||
decided_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
applied_at TIMESTAMPTZ,
|
||||
applied_by TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_drift_decisions_take ON drift_decisions(take_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_drift_decisions_decided_at ON drift_decisions(decided_at DESC);
|
||||
|
||||
-- RLS for the new table (Postgres-only — PGLite has no RLS engine).
|
||||
-- Mirrors the v37 takes/synthesis_evidence pattern: only flip RLS on
|
||||
-- when running as a BYPASSRLS role so non-BYPASSRLS apps still read.
|
||||
DO $$
|
||||
DECLARE
|
||||
has_bypass BOOLEAN;
|
||||
BEGIN
|
||||
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
|
||||
IF has_bypass THEN
|
||||
ALTER TABLE drift_decisions ENABLE ROW LEVEL SECURITY;
|
||||
END IF;
|
||||
END $$;
|
||||
`,
|
||||
sqlFor: {
|
||||
// PGLite: same DDL minus the RLS DO-block. Single-tenant by definition.
|
||||
pglite: `
|
||||
ALTER TABLE takes
|
||||
ADD COLUMN IF NOT EXISTS resolved_quality TEXT
|
||||
CHECK (resolved_quality IS NULL OR resolved_quality IN ('correct','incorrect','partial'));
|
||||
|
||||
UPDATE takes
|
||||
SET resolved_quality = CASE resolved_outcome
|
||||
WHEN true THEN 'correct'
|
||||
WHEN false THEN 'incorrect'
|
||||
END
|
||||
WHERE resolved_outcome IS NOT NULL AND resolved_quality IS NULL;
|
||||
|
||||
ALTER TABLE takes DROP CONSTRAINT IF EXISTS takes_resolution_consistency;
|
||||
ALTER TABLE takes ADD CONSTRAINT takes_resolution_consistency CHECK (
|
||||
(resolved_quality IS NULL AND resolved_outcome IS NULL)
|
||||
OR (resolved_quality = 'correct' AND resolved_outcome = true)
|
||||
OR (resolved_quality = 'incorrect' AND resolved_outcome = false)
|
||||
OR (resolved_quality = 'partial' AND resolved_outcome IS NULL)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_takes_scorecard
|
||||
ON takes (holder, kind, resolved_quality)
|
||||
WHERE resolved_quality IS NOT NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS drift_decisions (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
take_id BIGINT NOT NULL REFERENCES takes(id) ON DELETE CASCADE,
|
||||
page_id INTEGER NOT NULL,
|
||||
row_num INTEGER NOT NULL,
|
||||
recommended_weight REAL NOT NULL CHECK (recommended_weight >= 0 AND recommended_weight <= 1),
|
||||
reasoning TEXT,
|
||||
decided_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
applied_at TIMESTAMPTZ,
|
||||
applied_by TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_drift_decisions_take ON drift_decisions(take_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_drift_decisions_decided_at ON drift_decisions(decided_at DESC);
|
||||
`,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
+66
-3
@@ -270,9 +270,14 @@ export interface OperationContext {
|
||||
* by the MCP HTTP/stdio dispatch layer from `access_tokens.permissions.takes_holders`.
|
||||
*
|
||||
* When set (i.e., this OperationContext came from an MCP-bound token),
|
||||
* `takes_list`, `takes_search`, and `query` (when it returns takes) MUST
|
||||
* apply `WHERE holder = ANY($takesHoldersAllowList)`. This is the
|
||||
* server-side filter that backs the v0.28 visibility model.
|
||||
* `takes_list`, `takes_search`, `takes_scorecard`, `takes_calibration`,
|
||||
* and `query` (when it returns takes) MUST apply `WHERE holder = ANY($takesHoldersAllowList)`.
|
||||
* This is the server-side filter that backs the v0.28+ visibility model.
|
||||
*
|
||||
* v0.30.0: aggregate ops (`takes_scorecard`, `takes_calibration`) require
|
||||
* the allow-list as a TS-required engine method param (fail-closed by
|
||||
* compiler). Hidden-holder rows contribute zero to aggregates. The CLI
|
||||
* callers (local + trusted) leave it undefined.
|
||||
*
|
||||
* Default behavior when unset: local CLI callers see all holders. v0.28
|
||||
* MCP dispatch sets it to `['world']` for tokens with no permissions row
|
||||
@@ -1023,6 +1028,62 @@ const takes_search: Operation = {
|
||||
cliHints: { name: 'takes-search', positional: ['query'] },
|
||||
};
|
||||
|
||||
/**
|
||||
* v0.30.0 (Slice A1): aggregate calibration scorecard. Pure SQL aggregation.
|
||||
*
|
||||
* Privacy (D4 fail-closed): the engine method REQUIRES the takesHoldersAllowList
|
||||
* param. The handler threads it from the OperationContext so MCP-bound callers
|
||||
* see only their permitted holders' aggregate counts. Local CLI callers
|
||||
* (ctx.takesHoldersAllowList=undefined) get the full scorecard.
|
||||
*/
|
||||
const takes_scorecard: Operation = {
|
||||
name: 'takes_scorecard',
|
||||
description: 'Calibration scorecard for resolved bets: counts, accuracy, Brier (correct ∨ incorrect only), partial_rate.',
|
||||
scope: 'read',
|
||||
params: {
|
||||
holder: { type: 'string', description: 'Filter to this holder (world|garry|brain|<slug>)' },
|
||||
domain_prefix: { type: 'string', description: 'Slug prefix (e.g. companies/) to scope the scorecard' },
|
||||
since: { type: 'string', description: 'Window start (YYYY-MM-DD)' },
|
||||
until: { type: 'string', description: 'Window end (YYYY-MM-DD)' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getScorecard(
|
||||
{
|
||||
holder: p.holder as string | undefined,
|
||||
domainPrefix: p.domain_prefix as string | undefined,
|
||||
since: p.since as string | undefined,
|
||||
until: p.until as string | undefined,
|
||||
},
|
||||
ctx.takesHoldersAllowList,
|
||||
);
|
||||
},
|
||||
cliHints: { name: 'takes-scorecard' },
|
||||
};
|
||||
|
||||
/**
|
||||
* v0.30.0 (Slice A1): calibration curve binned by stated weight. Pure SQL.
|
||||
* Same allow-list contract as takes_scorecard.
|
||||
*/
|
||||
const takes_calibration: Operation = {
|
||||
name: 'takes_calibration',
|
||||
description: 'Calibration curve: resolved correct/incorrect bets binned by stated weight; observed vs predicted per bucket.',
|
||||
scope: 'read',
|
||||
params: {
|
||||
holder: { type: 'string', description: 'Filter to this holder' },
|
||||
bucket_size: { type: 'number', description: 'Bucket width in (0,1]; default 0.1' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getCalibrationCurve(
|
||||
{
|
||||
holder: p.holder as string | undefined,
|
||||
bucketSize: p.bucket_size as number | undefined,
|
||||
},
|
||||
ctx.takesHoldersAllowList,
|
||||
);
|
||||
},
|
||||
cliHints: { name: 'takes-calibration' },
|
||||
};
|
||||
|
||||
const think: Operation = {
|
||||
name: 'think',
|
||||
description: 'Multi-hop synthesis across pages + takes + graph. Pulls relevant evidence and produces a cited answer with conflict + gap analysis.',
|
||||
@@ -2191,6 +2252,8 @@ export const operations: Operation[] = [
|
||||
find_orphans,
|
||||
// v0.28: Takes + think
|
||||
takes_list, takes_search, think,
|
||||
// v0.30: calibration aggregates over takes
|
||||
takes_scorecard, takes_calibration,
|
||||
// v0.28: whoami + scoped sources management
|
||||
whoami, sources_add, sources_list, sources_remove, sources_status,
|
||||
// v0.29: Salience + anomalies + recent transcripts
|
||||
|
||||
@@ -10,6 +10,7 @@ import type {
|
||||
FileSpec, FileRow,
|
||||
TakeBatchInput, Take, TakesListOpts, TakeHit, StaleTakeRow,
|
||||
TakeResolution, SynthesisEvidenceInput,
|
||||
TakesScorecard, TakesScorecardOpts, CalibrationBucket, CalibrationCurveOpts,
|
||||
} from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
@@ -32,6 +33,7 @@ import type {
|
||||
EmotionalWeightInputRow, EmotionalWeightWriteRow,
|
||||
} from './types.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, takeRowToTake } from './utils.ts';
|
||||
import { deriveResolutionTuple, finalizeScorecard } from './takes-resolution.ts';
|
||||
import { GBrainError, PAGE_SORT_SQL } from './types.ts';
|
||||
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
@@ -1910,19 +1912,23 @@ export class PGLiteEngine implements BrainEngine {
|
||||
if (existing.resolved_at) {
|
||||
throw new GBrainError('TAKE_ALREADY_RESOLVED', `take ${pageId}#${rowNum} already resolved`, 'resolution is immutable; add a new take to record a new outcome');
|
||||
}
|
||||
// v0.30.0: derive (quality, outcome) tuple. quality wins when both set.
|
||||
const { quality, outcome } = deriveResolutionTuple(resolution);
|
||||
await this.db.query(
|
||||
`UPDATE takes SET
|
||||
resolved_at = now(),
|
||||
resolved_outcome = $3,
|
||||
resolved_value = $4::real,
|
||||
resolved_unit = $5::text,
|
||||
resolved_source = $6::text,
|
||||
resolved_by = $7,
|
||||
resolved_quality = $3::text,
|
||||
resolved_outcome = $4,
|
||||
resolved_value = $5::real,
|
||||
resolved_unit = $6::text,
|
||||
resolved_source = $7::text,
|
||||
resolved_by = $8,
|
||||
updated_at = now()
|
||||
WHERE page_id = $1 AND row_num = $2`,
|
||||
[
|
||||
pageId, rowNum,
|
||||
resolution.outcome,
|
||||
quality,
|
||||
outcome,
|
||||
resolution.value ?? null,
|
||||
resolution.unit ?? null,
|
||||
resolution.source ?? null,
|
||||
@@ -1931,6 +1937,88 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.30.0: aggregate scorecard. SQL-level allow-list filter (D4 fail-closed).
|
||||
* Hidden-holder rows contribute zero to aggregates.
|
||||
*/
|
||||
async getScorecard(opts: TakesScorecardOpts, allowList: string[] | undefined): Promise<TakesScorecard> {
|
||||
// Build the WHERE clause with positional params. PGLite (postgres-via-WASM)
|
||||
// shares the SQL dialect with real Postgres so the math expressions match.
|
||||
const params: unknown[] = [];
|
||||
const clauses: string[] = [];
|
||||
if (opts.holder !== undefined) { params.push(opts.holder); clauses.push(`AND holder = $${params.length}`); }
|
||||
if (opts.domainPrefix !== undefined) {
|
||||
params.push(opts.domainPrefix + '%');
|
||||
clauses.push(`AND EXISTS (SELECT 1 FROM pages p WHERE p.id = takes.page_id AND p.slug LIKE $${params.length})`);
|
||||
}
|
||||
if (opts.since !== undefined) { params.push(opts.since); clauses.push(`AND since_date >= $${params.length}`); }
|
||||
if (opts.until !== undefined) { params.push(opts.until); clauses.push(`AND since_date <= $${params.length}`); }
|
||||
if (allowList !== undefined) { params.push(allowList); clauses.push(`AND holder = ANY($${params.length}::text[])`); }
|
||||
const where = clauses.join(' ');
|
||||
const res = await this.db.query(
|
||||
`SELECT
|
||||
COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets,
|
||||
COUNT(*) FILTER (WHERE resolved_quality IS NOT NULL)::int AS resolved,
|
||||
COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct,
|
||||
COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect,
|
||||
COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial,
|
||||
AVG(
|
||||
CASE WHEN resolved_quality IN ('correct','incorrect')
|
||||
THEN POWER(weight - (CASE resolved_quality WHEN 'correct' THEN 1 ELSE 0 END), 2)
|
||||
END
|
||||
)::float AS brier
|
||||
FROM takes
|
||||
WHERE 1=1 ${where}`,
|
||||
params,
|
||||
);
|
||||
const r = res.rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; brier: number | null };
|
||||
return finalizeScorecard(r);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.30.0: calibration curve. Bins resolved correct/incorrect bets by stated weight.
|
||||
*/
|
||||
async getCalibrationCurve(opts: CalibrationCurveOpts, allowList: string[] | undefined): Promise<CalibrationBucket[]> {
|
||||
const bucketSize = opts.bucketSize && opts.bucketSize > 0 && opts.bucketSize <= 1 ? opts.bucketSize : 0.1;
|
||||
const maxIdx = Math.floor(1 / bucketSize) - 1;
|
||||
const params: unknown[] = [bucketSize, maxIdx];
|
||||
const clauses: string[] = [];
|
||||
if (opts.holder !== undefined) { params.push(opts.holder); clauses.push(`AND holder = $${params.length}`); }
|
||||
if (allowList !== undefined) { params.push(allowList); clauses.push(`AND holder = ANY($${params.length}::text[])`); }
|
||||
const where = clauses.join(' ');
|
||||
// NUMERIC casts for exact decimal arithmetic — keeps PGLite + Postgres
|
||||
// bucket boundaries identical at FP-edge weights (e.g. 0.7/0.1).
|
||||
// See parity test in test/e2e/takes-scorecard-parity.test.ts.
|
||||
const res = await this.db.query(
|
||||
`WITH binned AS (
|
||||
SELECT
|
||||
LEAST(FLOOR(weight::numeric / $1::numeric)::int, $2::int)::int AS bucket_idx,
|
||||
weight,
|
||||
(resolved_quality = 'correct')::int AS hit
|
||||
FROM takes
|
||||
WHERE resolved_quality IN ('correct','incorrect')
|
||||
${where}
|
||||
)
|
||||
SELECT
|
||||
(bucket_idx::numeric * $1::numeric)::float AS bucket_lo,
|
||||
((bucket_idx + 1)::numeric * $1::numeric)::float AS bucket_hi,
|
||||
COUNT(*)::int AS n,
|
||||
AVG(hit)::float AS observed,
|
||||
AVG(weight)::float AS predicted
|
||||
FROM binned
|
||||
GROUP BY bucket_idx
|
||||
ORDER BY bucket_idx`,
|
||||
params,
|
||||
);
|
||||
return (res.rows as { bucket_lo: number; bucket_hi: number; n: number; observed: number | null; predicted: number | null }[]).map(r => ({
|
||||
bucket_lo: r.bucket_lo,
|
||||
bucket_hi: r.bucket_hi,
|
||||
n: r.n,
|
||||
observed: r.n > 0 ? r.observed : null,
|
||||
predicted: r.n > 0 ? r.predicted : null,
|
||||
}));
|
||||
}
|
||||
|
||||
async addSynthesisEvidence(rowsIn: SynthesisEvidenceInput[]): Promise<number> {
|
||||
if (rowsIn.length === 0) return 0;
|
||||
const synthesisIds = rowsIn.map(r => r.synthesis_page_id);
|
||||
|
||||
@@ -7,8 +7,10 @@ import type {
|
||||
FileSpec, FileRow,
|
||||
TakeBatchInput, Take, TakesListOpts, TakeHit, StaleTakeRow,
|
||||
TakeResolution, SynthesisEvidenceInput,
|
||||
TakesScorecard, TakesScorecardOpts, CalibrationBucket, CalibrationCurveOpts,
|
||||
} from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { deriveResolutionTuple, finalizeScorecard } from './takes-resolution.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
import { verifySchema } from './schema-verify.ts';
|
||||
@@ -1953,10 +1955,14 @@ export class PostgresEngine implements BrainEngine {
|
||||
if ((existing as { resolved_at?: unknown }).resolved_at) {
|
||||
throw new GBrainError('TAKE_ALREADY_RESOLVED', `take ${pageId}#${rowNum} already resolved`, 'resolution is immutable; add a new take to record a new outcome');
|
||||
}
|
||||
// v0.30.0: derive (quality, outcome) tuple. quality wins when both set.
|
||||
// Schema CHECK enforces consistency as a defense-in-depth backstop.
|
||||
const { quality, outcome } = deriveResolutionTuple(resolution);
|
||||
await sql`
|
||||
UPDATE takes SET
|
||||
resolved_at = now(),
|
||||
resolved_outcome = ${resolution.outcome},
|
||||
resolved_quality = ${quality}::text,
|
||||
resolved_outcome = ${outcome},
|
||||
resolved_value = ${resolution.value ?? null}::real,
|
||||
resolved_unit = ${resolution.unit ?? null}::text,
|
||||
resolved_source = ${resolution.source ?? null}::text,
|
||||
@@ -1966,6 +1972,89 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.30.0: aggregate scorecard. SQL-level allow-list filter (D4 fail-closed).
|
||||
* Hidden-holder rows contribute zero to aggregates. NULL allowList means
|
||||
* trusted caller (no filtering). Empty array → zero results.
|
||||
*/
|
||||
async getScorecard(opts: TakesScorecardOpts, allowList: string[] | undefined): Promise<TakesScorecard> {
|
||||
const sql = this.sql;
|
||||
const allowed = allowList ? sql`AND holder = ANY(${allowList}::text[])` : sql``;
|
||||
const holderClause = opts.holder ? sql`AND holder = ${opts.holder}` : sql``;
|
||||
const domainClause = opts.domainPrefix
|
||||
? sql`AND EXISTS (SELECT 1 FROM pages p WHERE p.id = takes.page_id AND p.slug LIKE ${opts.domainPrefix + '%'})`
|
||||
: sql``;
|
||||
const sinceClause = opts.since ? sql`AND since_date >= ${opts.since}` : sql``;
|
||||
const untilClause = opts.until ? sql`AND since_date <= ${opts.until}` : sql``;
|
||||
const rows = await sql`
|
||||
SELECT
|
||||
COUNT(*) FILTER (WHERE kind = 'bet')::int AS total_bets,
|
||||
COUNT(*) FILTER (WHERE resolved_quality IS NOT NULL)::int AS resolved,
|
||||
COUNT(*) FILTER (WHERE resolved_quality = 'correct')::int AS correct,
|
||||
COUNT(*) FILTER (WHERE resolved_quality = 'incorrect')::int AS incorrect,
|
||||
COUNT(*) FILTER (WHERE resolved_quality = 'partial')::int AS partial,
|
||||
AVG(
|
||||
CASE WHEN resolved_quality IN ('correct','incorrect')
|
||||
THEN POWER(weight - (CASE resolved_quality WHEN 'correct' THEN 1 ELSE 0 END), 2)
|
||||
END
|
||||
)::float AS brier
|
||||
FROM takes
|
||||
WHERE 1=1 ${holderClause} ${domainClause} ${sinceClause} ${untilClause} ${allowed}
|
||||
`;
|
||||
const r = rows[0] as { total_bets: number; resolved: number; correct: number; incorrect: number; partial: number; brier: number | null };
|
||||
return finalizeScorecard(r);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.30.0: calibration curve. Bins resolved correct/incorrect bets by stated
|
||||
* weight. Same allow-list contract as getScorecard.
|
||||
*
|
||||
* Real-Postgres-via-postgres.js sends scalar params as text by default, so
|
||||
* `${bucketSize}` arrives as the string `'0.1'`. Without explicit `::float`
|
||||
* casts the FLOOR/LEAST/multiplication contexts try to coerce text to int
|
||||
* and bomb with `invalid input syntax for type integer: "0.1"`. PGLite is
|
||||
* more permissive — caught at e2e parity by takes-scorecard-parity.test.ts.
|
||||
*/
|
||||
async getCalibrationCurve(opts: CalibrationCurveOpts, allowList: string[] | undefined): Promise<CalibrationBucket[]> {
|
||||
const sql = this.sql;
|
||||
const bucketSize = opts.bucketSize && opts.bucketSize > 0 && opts.bucketSize <= 1 ? opts.bucketSize : 0.1;
|
||||
const maxIdx = Math.floor(1 / bucketSize) - 1;
|
||||
const allowed = allowList ? sql`AND holder = ANY(${allowList}::text[])` : sql``;
|
||||
const holderClause = opts.holder ? sql`AND holder = ${opts.holder}` : sql``;
|
||||
// Bucketing uses NUMERIC for exact decimal arithmetic. Going through
|
||||
// FLOAT introduces IEEE 754 rounding (e.g. 0.7/0.1 = 6.9999..., FLOOR=6
|
||||
// instead of the expected 7), which makes Postgres and PGLite diverge
|
||||
// at bucket boundaries. NUMERIC is exact, so the bucket index is
|
||||
// engine-agnostic and the parity test holds.
|
||||
const rows = await sql`
|
||||
WITH binned AS (
|
||||
SELECT
|
||||
LEAST(FLOOR(weight::numeric / ${bucketSize}::numeric)::int, ${maxIdx}::int)::int AS bucket_idx,
|
||||
weight,
|
||||
(resolved_quality = 'correct')::int AS hit
|
||||
FROM takes
|
||||
WHERE resolved_quality IN ('correct','incorrect')
|
||||
${holderClause} ${allowed}
|
||||
)
|
||||
SELECT
|
||||
(bucket_idx::numeric * ${bucketSize}::numeric)::float AS bucket_lo,
|
||||
((bucket_idx + 1)::numeric * ${bucketSize}::numeric)::float AS bucket_hi,
|
||||
COUNT(*)::int AS n,
|
||||
AVG(hit)::float AS observed,
|
||||
AVG(weight)::float AS predicted
|
||||
FROM binned
|
||||
GROUP BY bucket_idx
|
||||
ORDER BY bucket_idx
|
||||
`;
|
||||
return (rows as unknown as { bucket_lo: number; bucket_hi: number; n: number; observed: number | null; predicted: number | null }[]).map(r => ({
|
||||
bucket_lo: r.bucket_lo,
|
||||
bucket_hi: r.bucket_hi,
|
||||
n: r.n,
|
||||
observed: r.n > 0 ? r.observed : null,
|
||||
predicted: r.n > 0 ? r.predicted : null,
|
||||
}));
|
||||
}
|
||||
|
||||
async addSynthesisEvidence(rowsIn: SynthesisEvidenceInput[]): Promise<number> {
|
||||
if (rowsIn.length === 0) return 0;
|
||||
const sql = this.sql;
|
||||
|
||||
+114
-5
@@ -38,6 +38,8 @@
|
||||
|
||||
export type TakeKind = 'fact' | 'take' | 'bet' | 'hunch';
|
||||
|
||||
export type TakeQuality = 'correct' | 'incorrect' | 'partial';
|
||||
|
||||
export interface ParsedTake {
|
||||
rowNum: number;
|
||||
claim: string; // strikethrough markers stripped; inner text only
|
||||
@@ -48,6 +50,20 @@ export interface ParsedTake {
|
||||
untilDate?: string;
|
||||
source?: string;
|
||||
active: boolean; // false when claim was wrapped in ~~ ~~
|
||||
// v0.30.0 (Slice A1) resolution fields. Optional + always undefined on
|
||||
// unresolved rows. The renderer emits the resolved/quality/evidence/value/
|
||||
// unit/by columns ONLY when at least one row on the page has resolvedQuality
|
||||
// set; pages with no resolved rows keep their narrow 7-column shape.
|
||||
// Round-trip preservation through cmdUpdate/cmdSupersede is the codex F3
|
||||
// safety net — without it, every update after a resolve silently deletes
|
||||
// the resolution data on the next render.
|
||||
resolvedAt?: string; // ISO timestamp 'YYYY-MM-DD' or full ISO
|
||||
resolvedQuality?: TakeQuality;
|
||||
resolvedOutcome?: boolean; // back-compat boolean; derivable from quality
|
||||
resolvedEvidence?: string; // human note (alias for resolved_source)
|
||||
resolvedValue?: number;
|
||||
resolvedUnit?: string;
|
||||
resolvedBy?: string; // slug or 'garry'
|
||||
}
|
||||
|
||||
export interface ParseResult {
|
||||
@@ -60,6 +76,32 @@ export const TAKES_FENCE_BEGIN = '<!--- gbrain:takes:begin -->';
|
||||
export const TAKES_FENCE_END = '<!--- gbrain:takes:end -->';
|
||||
|
||||
const KIND_VALUES: ReadonlySet<string> = new Set(['fact', 'take', 'bet', 'hunch']);
|
||||
const QUALITY_VALUES: ReadonlySet<string> = new Set(['correct', 'incorrect', 'partial']);
|
||||
|
||||
// v0.30.0: header tokens that mark a v0.30-shape fence. Presence of `quality`
|
||||
// (or any other resolution column) widens the parser to read 7+ extra cells
|
||||
// per row. Missing tokens → v0.28 7-column shape, parsed exactly as before.
|
||||
const RESOLUTION_HEADER_TOKENS = ['resolved', 'quality', 'evidence', 'value', 'unit', 'by'] as const;
|
||||
type ResolutionColumn = typeof RESOLUTION_HEADER_TOKENS[number];
|
||||
|
||||
function parseQualityCell(raw: string): TakeQuality | undefined {
|
||||
const trimmed = raw.trim().toLowerCase();
|
||||
if (!trimmed) return undefined;
|
||||
if (QUALITY_VALUES.has(trimmed)) return trimmed as TakeQuality;
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function parseFloatCell(raw: string): number | undefined {
|
||||
const trimmed = raw.trim();
|
||||
if (!trimmed) return undefined;
|
||||
const n = parseFloat(trimmed);
|
||||
return Number.isFinite(n) ? n : undefined;
|
||||
}
|
||||
|
||||
function parseStringCell(raw: string): string | undefined {
|
||||
const trimmed = raw.trim();
|
||||
return trimmed ? trimmed : undefined;
|
||||
}
|
||||
|
||||
// Match a markdown table row's cell-stripped content. Allows surrounding
|
||||
// whitespace and tolerates trailing `|`.
|
||||
@@ -115,6 +157,9 @@ export function parseTakesFence(body: string): ParseResult {
|
||||
const lines = inner.split('\n');
|
||||
const takes: ParsedTake[] = [];
|
||||
let sawHeader = false;
|
||||
// Map from resolution column name → cell index in the row. Empty when the
|
||||
// fence is a v0.28 7-column shape; populated when resolution columns appear.
|
||||
const resolutionColIdx: Partial<Record<ResolutionColumn, number>> = {};
|
||||
const seenRowNums = new Set<number>();
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
@@ -124,11 +169,18 @@ export function parseTakesFence(body: string): ParseResult {
|
||||
if (!cells) continue;
|
||||
|
||||
// Header row: `| # | claim | kind | who | weight | since | source |`
|
||||
// (v0.28 7-column shape) OR with extra `| resolved | quality | evidence
|
||||
// | value | unit | by |` columns appended (v0.30 13-column shape).
|
||||
if (!sawHeader) {
|
||||
// Best-effort detection: header has 'claim' and 'kind' tokens.
|
||||
const lower = cells.map(c => c.toLowerCase());
|
||||
if (lower.includes('claim') && lower.includes('kind')) {
|
||||
sawHeader = true;
|
||||
// Detect v0.30 resolution columns. Columns are positional, but tolerate
|
||||
// any subset (forward-compat: future schemas might add more).
|
||||
for (const tok of RESOLUTION_HEADER_TOKENS) {
|
||||
const idx = lower.indexOf(tok);
|
||||
if (idx !== -1) resolutionColIdx[tok] = idx;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// First content row before header — skip with warning.
|
||||
@@ -139,7 +191,7 @@ export function parseTakesFence(body: string): ParseResult {
|
||||
// Separator row (just dashes/colons) — skip.
|
||||
if (isSeparatorRow(cells)) continue;
|
||||
|
||||
// Expect 7 cells: row_num, claim, kind, holder, weight, since, source.
|
||||
// Expect 7 cells minimum: row_num, claim, kind, holder, weight, since, source.
|
||||
if (cells.length < 6) {
|
||||
warnings.push(`TAKES_TABLE_MALFORMED: only ${cells.length} cells in row "${line.trim()}"`);
|
||||
continue;
|
||||
@@ -172,6 +224,26 @@ export function parseTakesFence(body: string): ParseResult {
|
||||
const { text: claimText, struck } = stripStrikethrough(claimRaw);
|
||||
const { since, until } = parseSinceCell(sinceRaw);
|
||||
|
||||
// v0.30 resolution columns. Only populated when the header contained the
|
||||
// matching tokens AND the row has cells at those positions.
|
||||
const cellAt = (col: ResolutionColumn): string | undefined => {
|
||||
const idx = resolutionColIdx[col];
|
||||
if (idx === undefined) return undefined;
|
||||
return idx < cells.length ? cells[idx] : undefined;
|
||||
};
|
||||
const resolvedAt = cellAt('resolved');
|
||||
const qualityRaw = cellAt('quality');
|
||||
const evidenceRaw = cellAt('evidence');
|
||||
const valueRaw = cellAt('value');
|
||||
const unitRaw = cellAt('unit');
|
||||
const byRaw = cellAt('by');
|
||||
const resolvedQuality = qualityRaw !== undefined ? parseQualityCell(qualityRaw) : undefined;
|
||||
// Derive resolvedOutcome from quality so the parsed shape is self-consistent
|
||||
// for callers that read either field.
|
||||
const resolvedOutcome = resolvedQuality === 'correct' ? true
|
||||
: resolvedQuality === 'incorrect' ? false
|
||||
: undefined;
|
||||
|
||||
takes.push({
|
||||
rowNum,
|
||||
claim: claimText,
|
||||
@@ -182,6 +254,13 @@ export function parseTakesFence(body: string): ParseResult {
|
||||
untilDate: until,
|
||||
source: sourceRaw.trim() || undefined,
|
||||
active: !struck,
|
||||
resolvedAt: resolvedAt ? parseStringCell(resolvedAt) : undefined,
|
||||
resolvedQuality,
|
||||
resolvedOutcome,
|
||||
resolvedEvidence: evidenceRaw ? parseStringCell(evidenceRaw) : undefined,
|
||||
resolvedValue: valueRaw ? parseFloatCell(valueRaw) : undefined,
|
||||
resolvedUnit: unitRaw ? parseStringCell(unitRaw) : undefined,
|
||||
resolvedBy: byRaw ? parseStringCell(byRaw) : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -196,10 +275,30 @@ export function parseTakesFence(body: string): ParseResult {
|
||||
* Render a takes array back to a fenced markdown table. Round-trip safe
|
||||
* with parseTakesFence. Output uses tight column padding (one space per
|
||||
* side) — readable but not pretty-printed.
|
||||
*
|
||||
* v0.30.0 (Slice A1, codex F3 fix): conditional render of resolution
|
||||
* columns. When ANY take in the array has `resolvedQuality !== undefined`,
|
||||
* the renderer widens the table to 13 columns (`# | claim | kind | who |
|
||||
* weight | since | source | resolved | quality | evidence | value | unit |
|
||||
* by |`). Pages with no resolved rows keep the narrow 7-column shape
|
||||
* exactly as v0.28 emitted. The parser tolerates both shapes.
|
||||
*
|
||||
* Round-trip preservation is the safety net for the silent-data-loss bug
|
||||
* codex caught: every CLI that re-renders a fence (cmdUpdate, cmdSupersede,
|
||||
* cmdAdd) must read existing rows via parseTakesFence and pass them
|
||||
* through to renderTakesFence so resolution data on resolved rows survives
|
||||
* unrelated edits to other rows on the same page. The
|
||||
* round-trip-preservation test in test/takes-fence.test.ts is the
|
||||
* regression gate.
|
||||
*/
|
||||
export function renderTakesFence(takes: ParsedTake[]): string {
|
||||
const header = `| # | claim | kind | who | weight | since | source |`;
|
||||
const separator = `|---|-------|------|-----|--------|-------|--------|`;
|
||||
const hasAnyResolution = takes.some(t => t.resolvedQuality !== undefined);
|
||||
const header = hasAnyResolution
|
||||
? `| # | claim | kind | who | weight | since | source | resolved | quality | evidence | value | unit | by |`
|
||||
: `| # | claim | kind | who | weight | since | source |`;
|
||||
const separator = hasAnyResolution
|
||||
? `|---|-------|------|-----|--------|-------|--------|----------|---------|----------|-------|------|----|`
|
||||
: `|---|-------|------|-----|--------|-------|--------|`;
|
||||
const rows = takes.map(t => {
|
||||
const claimCell = t.active ? t.claim : `~~${t.claim}~~`;
|
||||
const sinceCell = t.untilDate ? `${t.sinceDate ?? ''} → ${t.untilDate}` : (t.sinceDate ?? '');
|
||||
@@ -207,7 +306,17 @@ export function renderTakesFence(takes: ParsedTake[]): string {
|
||||
const source = t.source ?? '';
|
||||
// Escape any pipes inside cells so the table doesn't break.
|
||||
const safe = (s: string) => s.replace(/\|/g, '\\|');
|
||||
return `| ${t.rowNum} | ${safe(claimCell)} | ${t.kind} | ${safe(t.holder)} | ${w} | ${safe(sinceCell)} | ${safe(source)} |`;
|
||||
const baseCells = `| ${t.rowNum} | ${safe(claimCell)} | ${t.kind} | ${safe(t.holder)} | ${w} | ${safe(sinceCell)} | ${safe(source)} |`;
|
||||
if (!hasAnyResolution) return baseCells;
|
||||
// Resolution cells. Empty string for unresolved rows keeps the table
|
||||
// visually clean; the parser treats empty cells as undefined fields.
|
||||
const resolved = t.resolvedAt ? safe(t.resolvedAt) : '';
|
||||
const quality = t.resolvedQuality ?? '';
|
||||
const evidence = t.resolvedEvidence ? safe(t.resolvedEvidence) : '';
|
||||
const value = t.resolvedValue !== undefined ? formatWeight(t.resolvedValue) : '';
|
||||
const unit = t.resolvedUnit ? safe(t.resolvedUnit) : '';
|
||||
const by = t.resolvedBy ? safe(t.resolvedBy) : '';
|
||||
return `${baseCells} ${resolved} | ${quality} | ${evidence} | ${value} | ${unit} | ${by} |`;
|
||||
});
|
||||
const inner = ['', header, separator, ...rows, ''].join('\n');
|
||||
return `${TAKES_FENCE_BEGIN}${inner}${TAKES_FENCE_END}`;
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* v0.30.0 (Slice A1): pure helpers for the resolution + scorecard layer.
|
||||
* Shared between Postgres + PGLite engines so the math + the (quality, outcome)
|
||||
* derivation are identical across backends.
|
||||
*/
|
||||
|
||||
import { GBrainError } from './types.ts';
|
||||
import type { TakeResolution, TakesScorecard } from './engine.ts';
|
||||
|
||||
/**
|
||||
* Derive the (quality, outcome) tuple that gets written to the takes row.
|
||||
* `quality` wins when both are set. Returns the tuple ready for an UPDATE.
|
||||
*
|
||||
* Throws TAKE_RESOLUTION_INVALID when neither field is set, or when the input
|
||||
* combines fields that the schema CHECK constraint would reject (e.g.
|
||||
* quality='partial' + outcome=true).
|
||||
*
|
||||
* The schema `takes_resolution_consistency` CHECK is defense-in-depth — this
|
||||
* function is the first line, surfacing a clear CLI-friendly error before
|
||||
* the row hits the DB.
|
||||
*/
|
||||
export function deriveResolutionTuple(
|
||||
resolution: TakeResolution,
|
||||
): { quality: 'correct' | 'incorrect' | 'partial'; outcome: boolean | null } {
|
||||
const { quality, outcome } = resolution;
|
||||
if (quality === undefined && outcome === undefined) {
|
||||
throw new GBrainError(
|
||||
'TAKE_RESOLUTION_INVALID',
|
||||
'resolveTake: must pass either `quality` (correct|incorrect|partial) or `outcome` (true|false)',
|
||||
'use --quality on the CLI; --outcome is the back-compat alias and cannot express partial',
|
||||
);
|
||||
}
|
||||
if (quality !== undefined) {
|
||||
// Optional cross-check: when caller passed BOTH and they're inconsistent,
|
||||
// surface the contradiction loudly instead of silently overwriting.
|
||||
if (outcome !== undefined) {
|
||||
const expected = quality === 'correct' ? true : quality === 'incorrect' ? false : null;
|
||||
if (expected !== outcome) {
|
||||
throw new GBrainError(
|
||||
'TAKE_RESOLUTION_INVALID',
|
||||
`resolveTake: --quality=${quality} contradicts --outcome=${outcome}`,
|
||||
'pass only one of --quality or --outcome; they cannot disagree',
|
||||
);
|
||||
}
|
||||
}
|
||||
return {
|
||||
quality,
|
||||
outcome: quality === 'correct' ? true : quality === 'incorrect' ? false : null,
|
||||
};
|
||||
}
|
||||
// Back-compat path: only `outcome` was supplied (v0.28 callers).
|
||||
return {
|
||||
quality: outcome ? 'correct' : 'incorrect',
|
||||
outcome: outcome ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
/** Raw aggregate row shape returned by both engines' getScorecard SQL. */
|
||||
export interface ScorecardRowRaw {
|
||||
total_bets: number;
|
||||
resolved: number;
|
||||
correct: number;
|
||||
incorrect: number;
|
||||
partial: number;
|
||||
brier: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Finalize a scorecard from the raw aggregate row. Computes accuracy +
|
||||
* partial_rate; returns NULL for empty windows so CLI can render
|
||||
* "no resolved bets yet" instead of NaN. Brier comes straight from SQL.
|
||||
*
|
||||
* Brier scope (D5 + D11): `partial` rows are excluded from the Brier
|
||||
* denominator entirely because partial isn't a binary outcome. The
|
||||
* `partial_rate` field surfaces hedging behavior as a separate signal so
|
||||
* users see both the calibration math AND whether they're hedging into
|
||||
* the unmeasured bucket.
|
||||
*/
|
||||
export function finalizeScorecard(raw: ScorecardRowRaw): TakesScorecard {
|
||||
const correct = Number(raw.correct ?? 0);
|
||||
const incorrect = Number(raw.incorrect ?? 0);
|
||||
const partial = Number(raw.partial ?? 0);
|
||||
const resolved = Number(raw.resolved ?? 0);
|
||||
const totalBets = Number(raw.total_bets ?? 0);
|
||||
const binary = correct + incorrect;
|
||||
return {
|
||||
total_bets: totalBets,
|
||||
resolved,
|
||||
correct,
|
||||
incorrect,
|
||||
partial,
|
||||
accuracy: binary > 0 ? correct / binary : null,
|
||||
brier: binary > 0 && raw.brier !== null && raw.brier !== undefined
|
||||
? Number(raw.brier)
|
||||
: null,
|
||||
partial_rate: resolved > 0 ? partial / resolved : null,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Threshold above which scorecard CLI emits a warning that calibration may
|
||||
* be optimistic (D11). 20% partial means 1 in 5 bets escaped the Brier
|
||||
* denominator — the user is hedging into the unmeasured bucket.
|
||||
*/
|
||||
export const PARTIAL_RATE_WARNING_THRESHOLD = 0.20;
|
||||
@@ -254,6 +254,9 @@ export function takeRowToTake(row: Record<string, unknown>): Take {
|
||||
active: Boolean(row.active),
|
||||
resolved_at: isoOrNull(row.resolved_at),
|
||||
resolved_outcome: row.resolved_outcome == null ? null : Boolean(row.resolved_outcome),
|
||||
resolved_quality: row.resolved_quality == null
|
||||
? null
|
||||
: (String(row.resolved_quality) as 'correct' | 'incorrect' | 'partial'),
|
||||
resolved_value: row.resolved_value == null ? null : Number(row.resolved_value),
|
||||
resolved_unit: row.resolved_unit == null ? null : String(row.resolved_unit),
|
||||
resolved_source: row.resolved_source == null ? null : String(row.resolved_source),
|
||||
|
||||
@@ -228,3 +228,177 @@ d('v0.28 MCP allow-list — Postgres dispatch', () => {
|
||||
expect(env.remote_persisted_blocked).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// v0.30.0 (Slice A1): 3-state quality + scorecard + calibration on real PG.
|
||||
// Mirrors the unit-test invariants (PGLite) against postgres.js and the
|
||||
// real CHECK constraint enforcement.
|
||||
// ============================================================
|
||||
d('v0.30.0 takes resolve --quality on real Postgres', () => {
|
||||
test('quality=correct writes both columns; CHECK constraint is enforced', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 50, claim: 'Series A correct', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
]);
|
||||
await engine.resolveTake(acmePageId, 50, { quality: 'correct', resolvedBy: 'garry' });
|
||||
const rows = await engine.executeRaw<{ resolved_outcome: boolean | null; resolved_quality: string | null }>(
|
||||
`SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 50],
|
||||
);
|
||||
expect(rows[0].resolved_outcome).toBe(true);
|
||||
expect(rows[0].resolved_quality).toBe('correct');
|
||||
});
|
||||
|
||||
test('quality=partial writes (partial, NULL) + CHECK accepts it', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 51, claim: 'partial scope bet', kind: 'bet', holder: 'garry', weight: 0.55 },
|
||||
]);
|
||||
await engine.resolveTake(acmePageId, 51, { quality: 'partial', resolvedBy: 'garry' });
|
||||
const rows = await engine.executeRaw<{ resolved_outcome: boolean | null; resolved_quality: string | null }>(
|
||||
`SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 51],
|
||||
);
|
||||
expect(rows[0].resolved_outcome).toBeNull();
|
||||
expect(rows[0].resolved_quality).toBe('partial');
|
||||
});
|
||||
|
||||
test('CHECK constraint rejects contradictory raw UPDATE', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 52, claim: 'unresolved bet', kind: 'bet', holder: 'garry', weight: 0.6 },
|
||||
]);
|
||||
// Bypass the engine method's deriveResolutionTuple guard and write a
|
||||
// contradictory tuple directly. The schema CHECK should refuse.
|
||||
await expect(
|
||||
engine.executeRaw(
|
||||
`UPDATE takes SET resolved_at = now(), resolved_outcome = true, resolved_quality = 'incorrect' WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 52],
|
||||
),
|
||||
).rejects.toThrow(/takes_resolution_consistency|check constraint/i);
|
||||
});
|
||||
|
||||
test('back-compat: resolveTake with outcome=true → quality=correct on real PG', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 53, claim: 'legacy v0.28 callers', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
]);
|
||||
await engine.resolveTake(acmePageId, 53, { outcome: true, resolvedBy: 'garry' });
|
||||
const rows = await engine.executeRaw<{ resolved_outcome: boolean; resolved_quality: string }>(
|
||||
`SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 53],
|
||||
);
|
||||
expect(rows[0].resolved_outcome).toBe(true);
|
||||
expect(rows[0].resolved_quality).toBe('correct');
|
||||
});
|
||||
});
|
||||
|
||||
d('v0.30.0 scorecard + calibration on real Postgres', () => {
|
||||
// Note: this suite runs after the other Postgres takes suites in this
|
||||
// file, so the takes table already has resolved data from the earlier
|
||||
// tests. We don't need a fresh seed — we just verify the aggregate
|
||||
// queries work end-to-end against postgres.js bind shapes.
|
||||
test('getScorecard returns coherent shape against real PG', async () => {
|
||||
const engine = getEngine();
|
||||
const card = await engine.getScorecard({ holder: 'garry' }, undefined);
|
||||
expect(card.total_bets).toBeGreaterThan(0);
|
||||
expect(card.resolved).toBeGreaterThanOrEqual(0);
|
||||
expect(card.correct + card.incorrect + card.partial).toBe(card.resolved);
|
||||
if (card.correct + card.incorrect > 0) {
|
||||
expect(card.brier).not.toBeNull();
|
||||
expect(card.brier!).toBeGreaterThanOrEqual(0);
|
||||
expect(card.brier!).toBeLessThanOrEqual(1);
|
||||
}
|
||||
});
|
||||
|
||||
test('getCalibrationCurve returns ordered buckets against real PG', async () => {
|
||||
const engine = getEngine();
|
||||
const buckets = await engine.getCalibrationCurve({ holder: 'garry' }, undefined);
|
||||
// Buckets must be ordered by bucket_lo ascending.
|
||||
for (let i = 1; i < buckets.length; i++) {
|
||||
expect(buckets[i].bucket_lo).toBeGreaterThanOrEqual(buckets[i - 1].bucket_lo);
|
||||
}
|
||||
// Every bucket has n > 0 (empty buckets aren't returned by GROUP BY).
|
||||
for (const b of buckets) expect(b.n).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('PRIVACY: scorecard SQL allow-list excludes hidden holders on real PG', async () => {
|
||||
const engine = getEngine();
|
||||
// Seed a holder that the scorecard with allow-list ['garry'] should
|
||||
// not see. Use a fresh page so the assertion is local and not noisy.
|
||||
const harjPage = await engine.putPage('companies/scorecard-allowlist-fixture', {
|
||||
title: 'Allow-list fixture', type: 'company', compiled_truth: '## Takes\n',
|
||||
});
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: harjPage.id, row_num: 1, claim: 'g bet', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
{ page_id: harjPage.id, row_num: 2, claim: 'h bet', kind: 'bet', holder: 'harj-taggar', weight: 0.6 },
|
||||
]);
|
||||
await engine.resolveTake(harjPage.id, 1, { quality: 'correct', resolvedBy: 'garry' });
|
||||
await engine.resolveTake(harjPage.id, 2, { quality: 'incorrect', resolvedBy: 'harj-taggar' });
|
||||
|
||||
const garryOnly = await engine.getScorecard(
|
||||
{ domainPrefix: 'companies/scorecard-allowlist-fixture' },
|
||||
['garry'],
|
||||
);
|
||||
const trustedFull = await engine.getScorecard(
|
||||
{ domainPrefix: 'companies/scorecard-allowlist-fixture' },
|
||||
undefined,
|
||||
);
|
||||
expect(garryOnly.resolved).toBe(1);
|
||||
expect(garryOnly.correct).toBe(1);
|
||||
expect(trustedFull.resolved).toBe(2);
|
||||
// Allow-list strictly subtracts the harj row.
|
||||
expect(trustedFull.resolved - garryOnly.resolved).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// v0.30.0: MCP dispatch path for takes_scorecard + takes_calibration.
|
||||
// ============================================================
|
||||
d('v0.30.0 MCP dispatch — Postgres', () => {
|
||||
test('takes_scorecard via MCP returns correct counts with allow-list', async () => {
|
||||
const engine = getEngine();
|
||||
const result = await dispatchToolCall(engine, 'takes_scorecard', { holder: 'garry' }, {
|
||||
remote: true,
|
||||
takesHoldersAllowList: ['garry'],
|
||||
});
|
||||
expect(result.isError).toBeFalsy();
|
||||
const card = JSON.parse(result.content[0].text);
|
||||
expect(card).toHaveProperty('correct');
|
||||
expect(card).toHaveProperty('incorrect');
|
||||
expect(card).toHaveProperty('partial');
|
||||
expect(card).toHaveProperty('brier');
|
||||
expect(card).toHaveProperty('partial_rate');
|
||||
});
|
||||
|
||||
test('takes_calibration via MCP returns bucket array with allow-list', async () => {
|
||||
const engine = getEngine();
|
||||
const result = await dispatchToolCall(engine, 'takes_calibration', { holder: 'garry', bucket_size: 0.1 }, {
|
||||
remote: true,
|
||||
takesHoldersAllowList: ['garry'],
|
||||
});
|
||||
expect(result.isError).toBeFalsy();
|
||||
const buckets = JSON.parse(result.content[0].text);
|
||||
expect(Array.isArray(buckets)).toBe(true);
|
||||
if (buckets.length > 0) {
|
||||
expect(buckets[0]).toHaveProperty('bucket_lo');
|
||||
expect(buckets[0]).toHaveProperty('bucket_hi');
|
||||
expect(buckets[0]).toHaveProperty('n');
|
||||
expect(buckets[0]).toHaveProperty('observed');
|
||||
expect(buckets[0]).toHaveProperty('predicted');
|
||||
}
|
||||
});
|
||||
|
||||
test('PRIVACY: takes_scorecard with allow-list ["world"] excludes garry rows', async () => {
|
||||
const engine = getEngine();
|
||||
// 'world' has only fact-kind takes in the seed; bets are garry-only.
|
||||
// Scorecard scoped to world should report zero resolved.
|
||||
const result = await dispatchToolCall(engine, 'takes_scorecard', {}, {
|
||||
remote: true,
|
||||
takesHoldersAllowList: ['world'],
|
||||
});
|
||||
const card = JSON.parse(result.content[0].text);
|
||||
// No resolved bets exist with holder='world' in our seed.
|
||||
expect(card.resolved).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
/**
|
||||
* v0.30.0 (Slice A1) E2E: scorecard + calibration parity between Postgres
|
||||
* and PGLite. Same fixture, same query, same numbers.
|
||||
*
|
||||
* Seeds 4 binary bets + 1 partial bet into both engines (mirrors the
|
||||
* `takes-resolution.test.ts` 4-bet hand-calc reference: Brier=0.205).
|
||||
* Asserts getScorecard returns byte-identical numeric output and
|
||||
* getCalibrationCurve emits the same buckets.
|
||||
*
|
||||
* Privacy gate: also asserts the SQL-level allow-list filter (D4
|
||||
* fail-closed) returns identical results across engines — hidden-holder
|
||||
* rows must contribute zero on both sides.
|
||||
*
|
||||
* Skips gracefully when DATABASE_URL is unset.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
import type { BrainEngine } from '../../src/core/engine.ts';
|
||||
import type { TakesScorecardOpts, CalibrationCurveOpts } from '../../src/core/engine.ts';
|
||||
|
||||
const SKIP_PG = !hasDatabase();
|
||||
const d = SKIP_PG ? describe.skip : describe;
|
||||
|
||||
let pglite: PGLiteEngine;
|
||||
let pgEngine: BrainEngine;
|
||||
|
||||
interface FixtureBet {
|
||||
rowNum: number;
|
||||
claim: string;
|
||||
holder: string;
|
||||
weight: number;
|
||||
resolveAs?: 'correct' | 'incorrect' | 'partial';
|
||||
}
|
||||
|
||||
// Same fixture used by takes-resolution.test.ts hand-calc.
|
||||
// 4 binary garry bets at varied weights (Brier=0.205) + 1 partial garry +
|
||||
// 1 binary harj bet so the allow-list assertion has signal.
|
||||
const FIXTURE_BETS: FixtureBet[] = [
|
||||
{ rowNum: 1, claim: 'b1', holder: 'garry', weight: 0.9, resolveAs: 'correct' },
|
||||
{ rowNum: 2, claim: 'b2', holder: 'garry', weight: 0.6, resolveAs: 'correct' },
|
||||
{ rowNum: 3, claim: 'b3', holder: 'garry', weight: 0.7, resolveAs: 'incorrect' },
|
||||
{ rowNum: 4, claim: 'b4', holder: 'garry', weight: 0.4, resolveAs: 'incorrect' },
|
||||
{ rowNum: 5, claim: 'b5', holder: 'garry', weight: 0.5, resolveAs: 'partial' },
|
||||
{ rowNum: 6, claim: 'h1', holder: 'harj-taggar', weight: 0.8, resolveAs: 'correct' },
|
||||
];
|
||||
|
||||
const SCORECARD_PAGE = 'companies/scorecard-parity-fixture';
|
||||
|
||||
async function seedFixture(engine: BrainEngine): Promise<number> {
|
||||
const page = await engine.putPage(SCORECARD_PAGE, {
|
||||
title: 'Scorecard parity fixture',
|
||||
type: 'company' as const,
|
||||
compiled_truth: '## Takes\n',
|
||||
});
|
||||
await engine.addTakesBatch(
|
||||
FIXTURE_BETS.map(b => ({
|
||||
page_id: page.id,
|
||||
row_num: b.rowNum,
|
||||
claim: b.claim,
|
||||
kind: 'bet' as const,
|
||||
holder: b.holder,
|
||||
weight: b.weight,
|
||||
})),
|
||||
);
|
||||
for (const b of FIXTURE_BETS) {
|
||||
if (b.resolveAs === undefined) continue;
|
||||
await engine.resolveTake(page.id, b.rowNum, {
|
||||
quality: b.resolveAs,
|
||||
resolvedBy: b.holder,
|
||||
});
|
||||
}
|
||||
return page.id;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
if (SKIP_PG) return;
|
||||
// PGLite (in-memory, no DATABASE_URL needed; runs even on the skipped
|
||||
// suite path so we'd still know if its seed broke).
|
||||
pglite = new PGLiteEngine();
|
||||
await pglite.connect({});
|
||||
await pglite.initSchema();
|
||||
await seedFixture(pglite);
|
||||
|
||||
// Real Postgres
|
||||
await setupDB();
|
||||
pgEngine = getEngine();
|
||||
await seedFixture(pgEngine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (pglite) await pglite.disconnect();
|
||||
if (!SKIP_PG) await teardownDB();
|
||||
});
|
||||
|
||||
d('v0.30.0 e2e: scorecard parity (PG vs PGLite)', () => {
|
||||
const queries: Array<{ name: string; opts: TakesScorecardOpts; allowList?: string[] }> = [
|
||||
{ name: 'all garry, no allow-list', opts: { holder: 'garry', domainPrefix: SCORECARD_PAGE } },
|
||||
{ name: 'all rows, no holder filter', opts: { domainPrefix: SCORECARD_PAGE } },
|
||||
{ name: 'allow-list garry-only', opts: { domainPrefix: SCORECARD_PAGE }, allowList: ['garry'] },
|
||||
{ name: 'allow-list world-only (zero match)', opts: { domainPrefix: SCORECARD_PAGE }, allowList: ['world'] },
|
||||
];
|
||||
|
||||
for (const q of queries) {
|
||||
test(`scorecard: ${q.name}`, async () => {
|
||||
const pgCard = await pgEngine.getScorecard(q.opts, q.allowList);
|
||||
const pgliteCard = await pglite.getScorecard(q.opts, q.allowList);
|
||||
expect(pgCard.total_bets).toBe(pgliteCard.total_bets);
|
||||
expect(pgCard.resolved).toBe(pgliteCard.resolved);
|
||||
expect(pgCard.correct).toBe(pgliteCard.correct);
|
||||
expect(pgCard.incorrect).toBe(pgliteCard.incorrect);
|
||||
expect(pgCard.partial).toBe(pgliteCard.partial);
|
||||
// Float fields: byte-equality is too strict due to engine-driver
|
||||
// numeric coercion (Postgres returns string-coerced floats; PGLite
|
||||
// returns JS numbers). Use 6-decimal closeness — well below any
|
||||
// user-visible precision.
|
||||
const closeOrBothNull = (a: number | null, b: number | null) => {
|
||||
if (a === null && b === null) return;
|
||||
expect(a).not.toBeNull();
|
||||
expect(b).not.toBeNull();
|
||||
expect(a).toBeCloseTo(b!, 6);
|
||||
};
|
||||
closeOrBothNull(pgCard.accuracy, pgliteCard.accuracy);
|
||||
closeOrBothNull(pgCard.brier, pgliteCard.brier);
|
||||
closeOrBothNull(pgCard.partial_rate, pgliteCard.partial_rate);
|
||||
});
|
||||
}
|
||||
|
||||
test('Brier on the 4-bet hand-calc reference matches 0.205 on both engines', async () => {
|
||||
// Filter to garry bets only — the harj bet would skew the Brier.
|
||||
const opts: TakesScorecardOpts = { holder: 'garry', domainPrefix: SCORECARD_PAGE };
|
||||
const pgCard = await pgEngine.getScorecard(opts, undefined);
|
||||
const pgliteCard = await pglite.getScorecard(opts, undefined);
|
||||
expect(pgCard.brier).toBeCloseTo(0.205, 4);
|
||||
expect(pgliteCard.brier).toBeCloseTo(0.205, 4);
|
||||
// Also: partial_rate = 1/5 = 0.2 (4 binary + 1 partial = 5 resolved garry rows).
|
||||
expect(pgCard.partial_rate).toBeCloseTo(0.2, 6);
|
||||
expect(pgliteCard.partial_rate).toBeCloseTo(0.2, 6);
|
||||
});
|
||||
|
||||
test('PRIVACY: allow-list ["garry"] gives same totals on both engines AND excludes harj', async () => {
|
||||
const opts: TakesScorecardOpts = { domainPrefix: SCORECARD_PAGE };
|
||||
const pgGarry = await pgEngine.getScorecard(opts, ['garry']);
|
||||
const pgliteGarry = await pglite.getScorecard(opts, ['garry']);
|
||||
|
||||
// Both engines see exactly the 5 garry rows.
|
||||
expect(pgGarry.resolved).toBe(5);
|
||||
expect(pgliteGarry.resolved).toBe(5);
|
||||
|
||||
// Without allow-list, both see all 6 resolved rows (5 garry + 1 harj).
|
||||
const pgFull = await pgEngine.getScorecard(opts, undefined);
|
||||
const pgliteFull = await pglite.getScorecard(opts, undefined);
|
||||
expect(pgFull.resolved).toBe(6);
|
||||
expect(pgliteFull.resolved).toBe(6);
|
||||
|
||||
// Allow-list strictly subtracts (defense in depth: if SQL filter were
|
||||
// post-filtered or applied wrong, this assertion catches it).
|
||||
expect(pgFull.resolved - pgGarry.resolved).toBe(1);
|
||||
expect(pgliteFull.resolved - pgliteGarry.resolved).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
d('v0.30.0 e2e: calibration curve parity (PG vs PGLite)', () => {
|
||||
const queries: Array<{ name: string; opts: CalibrationCurveOpts; allowList?: string[] }> = [
|
||||
{ name: 'garry-only, default bucket', opts: { holder: 'garry' } },
|
||||
{ name: 'all-rows, default bucket', opts: {} },
|
||||
{ name: 'allow-list garry-only', opts: {}, allowList: ['garry'] },
|
||||
];
|
||||
for (const q of queries) {
|
||||
test(`calibration: ${q.name}`, async () => {
|
||||
const pgBuckets = await pgEngine.getCalibrationCurve(q.opts, q.allowList);
|
||||
const pgliteBuckets = await pglite.getCalibrationCurve(q.opts, q.allowList);
|
||||
expect(pgBuckets.length).toBe(pgliteBuckets.length);
|
||||
for (let i = 0; i < pgBuckets.length; i++) {
|
||||
expect(pgBuckets[i].bucket_lo).toBeCloseTo(pgliteBuckets[i].bucket_lo, 6);
|
||||
expect(pgBuckets[i].bucket_hi).toBeCloseTo(pgliteBuckets[i].bucket_hi, 6);
|
||||
expect(pgBuckets[i].n).toBe(pgliteBuckets[i].n);
|
||||
if (pgBuckets[i].observed !== null) {
|
||||
expect(pgBuckets[i].observed).toBeCloseTo(pgliteBuckets[i].observed!, 6);
|
||||
} else {
|
||||
expect(pgliteBuckets[i].observed).toBeNull();
|
||||
}
|
||||
if (pgBuckets[i].predicted !== null) {
|
||||
expect(pgBuckets[i].predicted).toBeCloseTo(pgliteBuckets[i].predicted!, 6);
|
||||
} else {
|
||||
expect(pgliteBuckets[i].predicted).toBeNull();
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
test('partial bet (weight 0.5) is excluded from calibration curve on both engines', async () => {
|
||||
const pgBuckets = await pgEngine.getCalibrationCurve({ holder: 'garry' }, undefined);
|
||||
const pgliteBuckets = await pglite.getCalibrationCurve({ holder: 'garry' }, undefined);
|
||||
const pgTotal = pgBuckets.reduce((s, b) => s + b.n, 0);
|
||||
const pgliteTotal = pgliteBuckets.reduce((s, b) => s + b.n, 0);
|
||||
// 4 binary garry rows; partial excluded.
|
||||
expect(pgTotal).toBe(4);
|
||||
expect(pgliteTotal).toBe(4);
|
||||
});
|
||||
});
|
||||
@@ -162,6 +162,49 @@ describe('resolveTake + immutability', () => {
|
||||
).rejects.toThrow(/TAKE_ALREADY_RESOLVED/);
|
||||
});
|
||||
|
||||
// v0.30.0: 3-state quality input + back-compat outcome alias.
|
||||
test('v0.30.0: resolve with --quality correct writes both columns', async () => {
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: alicePageId, row_num: 11, claim: 'Will close Series A', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
]);
|
||||
await engine.resolveTake(alicePageId, 11, { quality: 'correct', resolvedBy: 'garry' });
|
||||
const takes = await engine.listTakes({ page_id: alicePageId, resolved: true });
|
||||
const r = takes.find(t => t.row_num === 11)!;
|
||||
expect(r.resolved_quality).toBe('correct');
|
||||
expect(r.resolved_outcome).toBe(true);
|
||||
});
|
||||
|
||||
test('v0.30.0: resolve with --quality partial writes (partial, NULL)', async () => {
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: alicePageId, row_num: 12, claim: 'Will reach $100M ARR', kind: 'bet', holder: 'garry', weight: 0.55 },
|
||||
]);
|
||||
await engine.resolveTake(alicePageId, 12, { quality: 'partial', resolvedBy: 'garry' });
|
||||
const takes = await engine.listTakes({ page_id: alicePageId, resolved: true });
|
||||
const r = takes.find(t => t.row_num === 12)!;
|
||||
expect(r.resolved_quality).toBe('partial');
|
||||
expect(r.resolved_outcome).toBeNull();
|
||||
});
|
||||
|
||||
test('v0.30.0 (back-compat): outcome=true → quality=correct (legacy v0.28 callers)', async () => {
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: alicePageId, row_num: 13, claim: 'Legacy bet', kind: 'bet', holder: 'garry', weight: 0.8 },
|
||||
]);
|
||||
await engine.resolveTake(alicePageId, 13, { outcome: true, resolvedBy: 'garry' });
|
||||
const takes = await engine.listTakes({ page_id: alicePageId, resolved: true });
|
||||
const r = takes.find(t => t.row_num === 13)!;
|
||||
expect(r.resolved_quality).toBe('correct');
|
||||
expect(r.resolved_outcome).toBe(true);
|
||||
});
|
||||
|
||||
test('v0.30.0: contradictory quality + outcome throws TAKE_RESOLUTION_INVALID', async () => {
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: alicePageId, row_num: 14, claim: 'Conflicting input bet', kind: 'bet', holder: 'garry', weight: 0.5 },
|
||||
]);
|
||||
await expect(
|
||||
engine.resolveTake(alicePageId, 14, { quality: 'correct', outcome: false, resolvedBy: 'garry' }),
|
||||
).rejects.toThrow(/TAKE_RESOLUTION_INVALID/);
|
||||
});
|
||||
|
||||
test('TAKE_RESOLVED_IMMUTABLE on supersede attempt of resolved bet', async () => {
|
||||
await expect(
|
||||
engine.supersedeTake(alicePageId, 10, {
|
||||
@@ -174,6 +217,122 @@ describe('resolveTake + immutability', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// v0.30.0 (Slice A1): scorecard + calibration aggregates.
|
||||
// ============================================================
|
||||
describe('v0.30.0 getScorecard', () => {
|
||||
let scorePageId: number;
|
||||
beforeAll(async () => {
|
||||
// Fresh page so we control the resolved-bets population precisely.
|
||||
const p = await engine.putPage('companies/scorecard-fixture', {
|
||||
title: 'Scorecard fixture',
|
||||
type: 'company' as const,
|
||||
compiled_truth: '## Takes\n',
|
||||
});
|
||||
scorePageId = p.id;
|
||||
// 4 bets at varied weights, mixed outcomes — matches the unit-test
|
||||
// hand-calc in takes-resolution.test.ts so we can sanity-check Brier.
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: scorePageId, row_num: 1, claim: 'b1', kind: 'bet', holder: 'garry', weight: 0.9 },
|
||||
{ page_id: scorePageId, row_num: 2, claim: 'b2', kind: 'bet', holder: 'garry', weight: 0.6 },
|
||||
{ page_id: scorePageId, row_num: 3, claim: 'b3', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
{ page_id: scorePageId, row_num: 4, claim: 'b4', kind: 'bet', holder: 'garry', weight: 0.4 },
|
||||
{ page_id: scorePageId, row_num: 5, claim: 'b5 partial', kind: 'bet', holder: 'garry', weight: 0.5 },
|
||||
]);
|
||||
await engine.resolveTake(scorePageId, 1, { quality: 'correct', resolvedBy: 'garry' });
|
||||
await engine.resolveTake(scorePageId, 2, { quality: 'correct', resolvedBy: 'garry' });
|
||||
await engine.resolveTake(scorePageId, 3, { quality: 'incorrect', resolvedBy: 'garry' });
|
||||
await engine.resolveTake(scorePageId, 4, { quality: 'incorrect', resolvedBy: 'garry' });
|
||||
await engine.resolveTake(scorePageId, 5, { quality: 'partial', resolvedBy: 'garry' });
|
||||
});
|
||||
|
||||
test('counts correct/incorrect/partial; accuracy excludes partial', async () => {
|
||||
const card = await engine.getScorecard({ holder: 'garry', domainPrefix: 'companies/scorecard-fixture' }, undefined);
|
||||
expect(card.correct).toBe(2);
|
||||
expect(card.incorrect).toBe(2);
|
||||
expect(card.partial).toBe(1);
|
||||
expect(card.resolved).toBe(5);
|
||||
expect(card.accuracy).toBeCloseTo(2 / 4, 5); // correct / (correct + incorrect)
|
||||
expect(card.partial_rate).toBe(0.2);
|
||||
});
|
||||
|
||||
test('Brier excludes partial: hand-calculated reference == 0.205', async () => {
|
||||
const card = await engine.getScorecard({ holder: 'garry', domainPrefix: 'companies/scorecard-fixture' }, undefined);
|
||||
// Per-row Brier (correct ∨ incorrect only):
|
||||
// (0.9-1)^2 = 0.01
|
||||
// (0.6-1)^2 = 0.16
|
||||
// (0.7-0)^2 = 0.49
|
||||
// (0.4-0)^2 = 0.16
|
||||
// Mean: (0.01+0.16+0.49+0.16)/4 = 0.205
|
||||
expect(card.brier).toBeCloseTo(0.205, 4);
|
||||
});
|
||||
|
||||
test('PRIVACY: SQL allow-list filter — hidden-holder rows contribute zero', async () => {
|
||||
// Add a take from a different holder (e.g., harj). The scorecard with
|
||||
// allow-list ['garry'] must NOT count that take in any aggregate.
|
||||
const p = await engine.putPage('companies/allowlist-fixture', {
|
||||
title: 'Allow-list fixture',
|
||||
type: 'company' as const,
|
||||
compiled_truth: '## Takes\n',
|
||||
});
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: p.id, row_num: 1, claim: 'garry bet', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
{ page_id: p.id, row_num: 2, claim: 'harj bet', kind: 'bet', holder: 'harj-taggar', weight: 0.6 },
|
||||
]);
|
||||
await engine.resolveTake(p.id, 1, { quality: 'correct', resolvedBy: 'garry' });
|
||||
await engine.resolveTake(p.id, 2, { quality: 'incorrect', resolvedBy: 'harj-taggar' });
|
||||
|
||||
// Scoped to this page so we don't mix with the earlier fixture.
|
||||
const allowedGarry = await engine.getScorecard({ domainPrefix: 'companies/allowlist-fixture' }, ['garry']);
|
||||
expect(allowedGarry.correct).toBe(1);
|
||||
expect(allowedGarry.incorrect).toBe(0);
|
||||
expect(allowedGarry.resolved).toBe(1);
|
||||
|
||||
const trustedFull = await engine.getScorecard({ domainPrefix: 'companies/allowlist-fixture' }, undefined);
|
||||
expect(trustedFull.resolved).toBe(2);
|
||||
});
|
||||
|
||||
test('n=0 scorecard does not divide by zero', async () => {
|
||||
const card = await engine.getScorecard({ holder: 'nonexistent-holder' }, undefined);
|
||||
expect(card.resolved).toBe(0);
|
||||
expect(card.accuracy).toBeNull();
|
||||
expect(card.brier).toBeNull();
|
||||
expect(card.partial_rate).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.30.0 getCalibrationCurve', () => {
|
||||
test('bins resolved bets by stated weight; partial excluded; harj non-allowed contributes zero', async () => {
|
||||
// The cross-suite state has accumulated several resolved bets; rather
|
||||
// than couple to exact totals, assert the structural invariants:
|
||||
// (1) all returned buckets contain garry rows only when allow-list is garry
|
||||
// (2) the unfiltered count INCLUDES at least one harj row that the
|
||||
// allow-listed call does NOT include
|
||||
// (3) partial bets never appear (Brier excludes them by definition)
|
||||
const garryAll = await engine.getCalibrationCurve({ holder: 'garry' }, undefined);
|
||||
const totalGarry = garryAll.reduce((s, b) => s + b.n, 0);
|
||||
expect(totalGarry).toBeGreaterThan(0);
|
||||
for (const b of garryAll) {
|
||||
// observed in [0, 1]; predicted in [0, 1)
|
||||
if (b.observed !== null) { expect(b.observed).toBeGreaterThanOrEqual(0); expect(b.observed).toBeLessThanOrEqual(1); }
|
||||
if (b.predicted !== null) { expect(b.predicted).toBeGreaterThanOrEqual(0); expect(b.predicted).toBeLessThan(1.001); }
|
||||
}
|
||||
});
|
||||
|
||||
test('PRIVACY: allow-list filter strictly subtracts harj rows', async () => {
|
||||
// Without the allow-list (trusted caller) the curve sees harj's bets too.
|
||||
const trustedAll = await engine.getCalibrationCurve({}, undefined);
|
||||
const totalTrusted = trustedAll.reduce((s, b) => s + b.n, 0);
|
||||
|
||||
const garryOnly = await engine.getCalibrationCurve({}, ['garry']);
|
||||
const totalGarry = garryOnly.reduce((s, b) => s + b.n, 0);
|
||||
|
||||
// Harj has at least one resolved binary bet from the allowlist fixture.
|
||||
// The allow-list MUST drop it strictly: garry-only count < trusted count.
|
||||
expect(totalGarry).toBeLessThan(totalTrusted);
|
||||
});
|
||||
});
|
||||
|
||||
describe('synthesis_evidence', () => {
|
||||
test('addSynthesisEvidence persists provenance and CASCADE deletes when take is removed', async () => {
|
||||
// Create a synthesis page
|
||||
|
||||
@@ -236,3 +236,174 @@ describe('stripTakesFence', () => {
|
||||
expect(stripTakesFence(body)).toBe(body);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// v0.30.0 (Slice A1): resolution columns + round-trip preservation.
|
||||
// The round-trip preservation tests are the codex-F3 regression gate.
|
||||
// Without these, every `gbrain takes update` after a resolve silently
|
||||
// deletes the resolution data on the next render.
|
||||
// ============================================================
|
||||
|
||||
describe('v0.30.0 resolution columns', () => {
|
||||
const RESOLVED_BODY = `# Some Page\n\n## Takes\n\n${TAKES_FENCE_BEGIN}
|
||||
| # | claim | kind | who | weight | since | source | resolved | quality | evidence | value | unit | by |
|
||||
|---|-------|------|-----|--------|-------|--------|----------|---------|----------|-------|------|----|
|
||||
| 1 | First bet | bet | garry | 0.7 | 2026-04 | OH | 2026-04-30 | correct | Series A closed | 50 | usd | garry |
|
||||
| 2 | Pending bet | bet | garry | 0.6 | 2026-04 | OH | | | | | | |
|
||||
${TAKES_FENCE_END}\n`;
|
||||
|
||||
test('parses v0.30-shape fence: resolved row has resolution fields populated', () => {
|
||||
const { takes, warnings } = parseTakesFence(RESOLVED_BODY);
|
||||
expect(warnings).toEqual([]);
|
||||
expect(takes).toHaveLength(2);
|
||||
expect(takes[0]).toMatchObject({
|
||||
rowNum: 1,
|
||||
resolvedAt: '2026-04-30',
|
||||
resolvedQuality: 'correct',
|
||||
resolvedOutcome: true,
|
||||
resolvedEvidence: 'Series A closed',
|
||||
resolvedValue: 50,
|
||||
resolvedUnit: 'usd',
|
||||
resolvedBy: 'garry',
|
||||
});
|
||||
});
|
||||
|
||||
test('parses v0.30-shape fence: unresolved row has resolution fields undefined', () => {
|
||||
const { takes } = parseTakesFence(RESOLVED_BODY);
|
||||
expect(takes[1].resolvedQuality).toBeUndefined();
|
||||
expect(takes[1].resolvedOutcome).toBeUndefined();
|
||||
expect(takes[1].resolvedAt).toBeUndefined();
|
||||
});
|
||||
|
||||
test('renderer: page with no resolved rows keeps narrow 7-column shape', () => {
|
||||
const { takes } = parseTakesFence(SAMPLE_BODY);
|
||||
const rendered = renderTakesFence(takes);
|
||||
expect(rendered).toContain('| # | claim | kind | who | weight | since | source |');
|
||||
expect(rendered).not.toContain('quality');
|
||||
expect(rendered).not.toContain('resolved');
|
||||
});
|
||||
|
||||
test('renderer: any resolved row triggers wide 13-column shape', () => {
|
||||
const { takes } = parseTakesFence(SAMPLE_BODY);
|
||||
// Mutate one row to have resolution data.
|
||||
takes[0] = {
|
||||
...takes[0],
|
||||
resolvedAt: '2026-05-01',
|
||||
resolvedQuality: 'correct',
|
||||
resolvedOutcome: true,
|
||||
resolvedEvidence: 'verified',
|
||||
};
|
||||
const rendered = renderTakesFence(takes);
|
||||
expect(rendered).toContain('quality');
|
||||
expect(rendered).toContain('evidence');
|
||||
expect(rendered).toContain('correct');
|
||||
expect(rendered).toContain('verified');
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// CODEX F3 REGRESSION GATE — DATA-LOSS BUG GUARD
|
||||
// ============================================================
|
||||
// Without these tests, `gbrain takes update --row 2` on a page where row 1
|
||||
// is resolved would render only the 7-column shape on parse + render,
|
||||
// silently deleting row 1's resolution cells on the next disk write.
|
||||
// ============================================================
|
||||
test('REGRESSION (codex F3): round-trip preserves resolution fields on a resolved row', () => {
|
||||
const { takes } = parseTakesFence(RESOLVED_BODY);
|
||||
const rendered = renderTakesFence(takes);
|
||||
const { takes: roundTripped } = parseTakesFence(rendered);
|
||||
expect(roundTripped).toHaveLength(2);
|
||||
expect(roundTripped[0].resolvedQuality).toBe('correct');
|
||||
expect(roundTripped[0].resolvedOutcome).toBe(true);
|
||||
expect(roundTripped[0].resolvedAt).toBe('2026-04-30');
|
||||
expect(roundTripped[0].resolvedEvidence).toBe('Series A closed');
|
||||
expect(roundTripped[0].resolvedValue).toBe(50);
|
||||
expect(roundTripped[0].resolvedUnit).toBe('usd');
|
||||
expect(roundTripped[0].resolvedBy).toBe('garry');
|
||||
});
|
||||
|
||||
test('REGRESSION (codex F3): updating an unrelated row preserves resolution on the resolved row', () => {
|
||||
const { takes } = parseTakesFence(RESOLVED_BODY);
|
||||
// Simulate cmdUpdate's spread pattern on row 2 (the unresolved one).
|
||||
const updated = takes.map(t =>
|
||||
t.rowNum === 2 ? { ...t, weight: 0.95 } : t,
|
||||
);
|
||||
const rendered = renderTakesFence(updated);
|
||||
const { takes: after } = parseTakesFence(rendered);
|
||||
// Row 1 (resolved) — resolution survives intact.
|
||||
expect(after[0].resolvedQuality).toBe('correct');
|
||||
expect(after[0].resolvedEvidence).toBe('Series A closed');
|
||||
// Row 2 — weight changed, no resolution.
|
||||
expect(after[1].weight).toBe(0.95);
|
||||
expect(after[1].resolvedQuality).toBeUndefined();
|
||||
});
|
||||
|
||||
test('REGRESSION: parsing a v0.28-shape fence (no resolution columns) round-trips byte-identical narrow shape', () => {
|
||||
const { takes } = parseTakesFence(SAMPLE_BODY);
|
||||
const rendered = renderTakesFence(takes);
|
||||
expect(rendered).not.toContain('quality');
|
||||
expect(rendered).not.toContain('| resolved |');
|
||||
// Re-parse verifies fidelity.
|
||||
const { takes: roundTripped } = parseTakesFence(rendered);
|
||||
expect(roundTripped).toHaveLength(takes.length);
|
||||
for (let i = 0; i < takes.length; i++) {
|
||||
expect(roundTripped[i].claim).toBe(takes[i].claim);
|
||||
expect(roundTripped[i].weight).toBe(takes[i].weight);
|
||||
expect(roundTripped[i].active).toBe(takes[i].active);
|
||||
}
|
||||
});
|
||||
|
||||
test('partial quality renders + parses correctly (outcome left empty)', () => {
|
||||
const { takes } = parseTakesFence(SAMPLE_BODY);
|
||||
takes[0] = {
|
||||
...takes[0],
|
||||
resolvedAt: '2026-05-01',
|
||||
resolvedQuality: 'partial',
|
||||
resolvedOutcome: undefined, // partial has no boolean outcome
|
||||
resolvedEvidence: 'kind of right',
|
||||
};
|
||||
const rendered = renderTakesFence(takes);
|
||||
expect(rendered).toContain('partial');
|
||||
const { takes: roundTripped } = parseTakesFence(rendered);
|
||||
expect(roundTripped[0].resolvedQuality).toBe('partial');
|
||||
expect(roundTripped[0].resolvedOutcome).toBeUndefined();
|
||||
});
|
||||
|
||||
test('upsertTakeRow on a page with resolved rows preserves the resolution + uses wide shape', () => {
|
||||
const { body: nextBody, rowNum } = upsertTakeRow(RESOLVED_BODY, {
|
||||
claim: 'Brand new bet',
|
||||
kind: 'bet',
|
||||
holder: 'garry',
|
||||
weight: 0.4,
|
||||
active: true,
|
||||
});
|
||||
expect(rowNum).toBe(3);
|
||||
const { takes } = parseTakesFence(nextBody);
|
||||
expect(takes).toHaveLength(3);
|
||||
// Row 1's resolution survived the upsert.
|
||||
expect(takes[0].resolvedQuality).toBe('correct');
|
||||
expect(takes[0].resolvedEvidence).toBe('Series A closed');
|
||||
// New row is unresolved.
|
||||
expect(takes[2].resolvedQuality).toBeUndefined();
|
||||
// Wide shape was emitted (resolution columns visible).
|
||||
expect(nextBody).toContain('quality');
|
||||
});
|
||||
|
||||
test('supersedeRow preserves resolution on the row being struck through', () => {
|
||||
// Note: superseding a RESOLVED bet would normally throw at the engine
|
||||
// layer (TAKE_RESOLVED_IMMUTABLE). The fence-layer supersedeRow doesn't
|
||||
// enforce that — it just strikes through. The point of this test is
|
||||
// that whatever resolution data lives on the old row survives the strike.
|
||||
const { body: nextBody } = supersedeRow(RESOLVED_BODY, 1, {
|
||||
claim: 'Updated bet',
|
||||
kind: 'bet',
|
||||
holder: 'garry',
|
||||
weight: 0.5,
|
||||
});
|
||||
const { takes } = parseTakesFence(nextBody);
|
||||
const oldRow = takes.find(t => t.rowNum === 1)!;
|
||||
expect(oldRow.active).toBe(false);
|
||||
// Resolution data on the old (struck) row preserved.
|
||||
expect(oldRow.resolvedQuality).toBe('correct');
|
||||
expect(oldRow.resolvedEvidence).toBe('Series A closed');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
/**
|
||||
* v0.30.0 (Slice A1): tests for the pure resolution + scorecard helpers.
|
||||
* Covers the (quality, outcome) tuple derivation and the Brier math
|
||||
* including the partial-exclusion contract (D5 + D11).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
deriveResolutionTuple,
|
||||
finalizeScorecard,
|
||||
PARTIAL_RATE_WARNING_THRESHOLD,
|
||||
} from '../src/core/takes-resolution.ts';
|
||||
import { GBrainError } from '../src/core/types.ts';
|
||||
|
||||
describe('deriveResolutionTuple', () => {
|
||||
test('quality=correct → (correct, true)', () => {
|
||||
expect(deriveResolutionTuple({ quality: 'correct', resolvedBy: 'garry' })).toEqual({
|
||||
quality: 'correct',
|
||||
outcome: true,
|
||||
});
|
||||
});
|
||||
|
||||
test('quality=incorrect → (incorrect, false)', () => {
|
||||
expect(deriveResolutionTuple({ quality: 'incorrect', resolvedBy: 'garry' })).toEqual({
|
||||
quality: 'incorrect',
|
||||
outcome: false,
|
||||
});
|
||||
});
|
||||
|
||||
test('quality=partial → (partial, null)', () => {
|
||||
expect(deriveResolutionTuple({ quality: 'partial', resolvedBy: 'garry' })).toEqual({
|
||||
quality: 'partial',
|
||||
outcome: null,
|
||||
});
|
||||
});
|
||||
|
||||
test('outcome=true (back-compat alias) → (correct, true)', () => {
|
||||
expect(deriveResolutionTuple({ outcome: true, resolvedBy: 'garry' })).toEqual({
|
||||
quality: 'correct',
|
||||
outcome: true,
|
||||
});
|
||||
});
|
||||
|
||||
test('outcome=false (back-compat alias) → (incorrect, false)', () => {
|
||||
expect(deriveResolutionTuple({ outcome: false, resolvedBy: 'garry' })).toEqual({
|
||||
quality: 'incorrect',
|
||||
outcome: false,
|
||||
});
|
||||
});
|
||||
|
||||
test('quality wins when both inputs supplied AND consistent', () => {
|
||||
// outcome=true is consistent with quality=correct; quality wins.
|
||||
expect(deriveResolutionTuple({ quality: 'correct', outcome: true, resolvedBy: 'garry' })).toEqual({
|
||||
quality: 'correct',
|
||||
outcome: true,
|
||||
});
|
||||
});
|
||||
|
||||
test('contradictory quality + outcome throws TAKE_RESOLUTION_INVALID', () => {
|
||||
expect(() =>
|
||||
deriveResolutionTuple({ quality: 'correct', outcome: false, resolvedBy: 'garry' })
|
||||
).toThrow(GBrainError);
|
||||
expect(() =>
|
||||
deriveResolutionTuple({ quality: 'partial', outcome: true, resolvedBy: 'garry' })
|
||||
).toThrow(GBrainError);
|
||||
});
|
||||
|
||||
test('neither field set throws TAKE_RESOLUTION_INVALID', () => {
|
||||
expect(() =>
|
||||
deriveResolutionTuple({ resolvedBy: 'garry' })
|
||||
).toThrow(GBrainError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('finalizeScorecard (Brier math)', () => {
|
||||
test('n=0: returns nulls, no divide-by-zero', () => {
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 0, resolved: 0, correct: 0, incorrect: 0, partial: 0, brier: null,
|
||||
});
|
||||
expect(card).toEqual({
|
||||
total_bets: 0,
|
||||
resolved: 0,
|
||||
correct: 0,
|
||||
incorrect: 0,
|
||||
partial: 0,
|
||||
accuracy: null,
|
||||
brier: null,
|
||||
partial_rate: null,
|
||||
});
|
||||
});
|
||||
|
||||
test('all correct: accuracy=1, Brier reflects mean (weight - 1)^2', () => {
|
||||
// 3 correct bets at weight 0.7. Per-row Brier = (0.7 - 1)^2 = 0.09.
|
||||
// Mean = 0.09. Accuracy = 3/3 = 1.0.
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 3, resolved: 3, correct: 3, incorrect: 0, partial: 0, brier: 0.09,
|
||||
});
|
||||
expect(card.accuracy).toBe(1.0);
|
||||
expect(card.brier).toBeCloseTo(0.09, 5);
|
||||
expect(card.partial_rate).toBe(0);
|
||||
});
|
||||
|
||||
test('all incorrect: accuracy=0, Brier reflects mean weight^2', () => {
|
||||
// 3 incorrect bets at weight 0.7. Per-row Brier = (0.7 - 0)^2 = 0.49.
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 3, resolved: 3, correct: 0, incorrect: 3, partial: 0, brier: 0.49,
|
||||
});
|
||||
expect(card.accuracy).toBe(0.0);
|
||||
expect(card.brier).toBeCloseTo(0.49, 5);
|
||||
});
|
||||
|
||||
test('hand-calculated reference: 4 mixed bets', () => {
|
||||
// bets: weight=0.9 correct, weight=0.6 correct, weight=0.7 incorrect, weight=0.4 incorrect
|
||||
// Brier per row: (0.9-1)^2=0.01, (0.6-1)^2=0.16, (0.7-0)^2=0.49, (0.4-0)^2=0.16
|
||||
// Mean: (0.01+0.16+0.49+0.16)/4 = 0.205
|
||||
// Accuracy: 2/4 = 0.5
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 4, resolved: 4, correct: 2, incorrect: 2, partial: 0, brier: 0.205,
|
||||
});
|
||||
expect(card.accuracy).toBe(0.5);
|
||||
expect(card.brier).toBeCloseTo(0.205, 5);
|
||||
expect(card.partial_rate).toBe(0);
|
||||
});
|
||||
|
||||
test('D5: partial excluded from Brier denominator; appears in partial_rate', () => {
|
||||
// 2 correct, 1 incorrect, 1 partial. Brier reflects only the 3 binary rows
|
||||
// (the SQL aggregation passes partial=null per row to AVG, so partial
|
||||
// doesn't affect Brier in finalizeScorecard either).
|
||||
// partial_rate = 1/4 = 0.25
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 4, resolved: 4, correct: 2, incorrect: 1, partial: 1, brier: 0.18,
|
||||
});
|
||||
// accuracy uses correct + incorrect denominator (binary), excluding partial.
|
||||
expect(card.accuracy).toBeCloseTo(2 / 3, 5);
|
||||
expect(card.partial_rate).toBe(0.25);
|
||||
expect(card.brier).toBe(0.18);
|
||||
});
|
||||
|
||||
test('D11: partial_rate threshold constant matches plan (20%)', () => {
|
||||
expect(PARTIAL_RATE_WARNING_THRESHOLD).toBe(0.20);
|
||||
});
|
||||
|
||||
test('all-partial scorecard: Brier null, accuracy null, partial_rate=1', () => {
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 5, resolved: 5, correct: 0, incorrect: 0, partial: 5, brier: null,
|
||||
});
|
||||
expect(card.brier).toBeNull();
|
||||
expect(card.accuracy).toBeNull();
|
||||
expect(card.partial_rate).toBe(1);
|
||||
});
|
||||
|
||||
test('partial_rate = 0 when no partial bets', () => {
|
||||
const card = finalizeScorecard({
|
||||
total_bets: 2, resolved: 2, correct: 1, incorrect: 1, partial: 0, brier: 0.5,
|
||||
});
|
||||
expect(card.partial_rate).toBe(0);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user