diff --git a/CHANGELOG.md b/CHANGELOG.md index 56a062cc5..92df4af63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,207 @@ All notable changes to GBrain will be documented in this file. +## [0.41.23.0] - 2026-05-26 + +**You can now see how every extractor in your brain is doing — how +often it halts, what it spent, whether its eval gate fired — and your +pack manifests can declare new extractable kinds in one verb.** + +Pre-v0.41.23, when an extractor halted partway through a long page or a +cycle phase silently stopped writing facts, you found out by querying +the brain a week later and noticing things were missing. The +information was theoretically in JSONL audit files, but the operator +surface to read it didn't exist. Same gap on the authoring side: if you +wanted your schema pack to declare a new extractable kind, you wrote +YAML by hand and hoped you got the shape right. + +This release fixes both at the operator surface. Every extractor that +runs in the brain (the deterministic facts.conversation extractor, the +three LLM-backed cycle phases for atoms/concepts/takes, and the +facts.fence reconciler) now writes one receipt page per run AND +upserts a row into a 7-day rollup table. Receipts are queryable like +any other page (they get a 0.3x source-boost demote so they surface in +search but never dominate user content). The rollup powers the new +`gbrain extract status` dashboard. + +How to use it: + +```bash +gbrain extract status # 7-day rollup, top-5 by halt rate +gbrain extract status --kind atoms # filter to one extractor kind +gbrain extract status --verbose --json # full table + monitoring envelope + +gbrain extract --explain facts.conversation +# Prints: which pack declares this kind (or "built-in cycle phase"), +# what files it expects, eval dimensions, last 7d rollup. + +gbrain schema scaffold-extractable claim --pack my-pack +# Generates 5 placeholder fixtures + a prompt template stub, declares +# the type extractable on the pack manifest. Refuses on existing files +# unless --force. + +gbrain extract benchmark --pack my-pack --kind claim +# v0.41.23 ships as a stub-reporter (validates fixture corpus shape). +# LLM dispatch deferred to a follow-up release. +``` + +What you'd see in a concrete example. Say your `extract_atoms` phase is +silently halting on long conversation pages: + +| Kind | 7d cost | Halts | Halt rate | Eval pass/fail | +|-----------------------|----------|-------|-----------|----------------| +| atoms | $0.30 | 5 | 50.0% | 3 / 0 | +| concepts | $0.10 | 1 | 10.0% | 1 / 0 | +| facts.conversation | $1.50 | 0 | 0.0% | 5 / 0 | + +Before v0.41.23 you had to know to grep the JSONL audit files. Now `gbrain +extract status` puts the halt rate above the fold, ordered most-troubled +first. + +Things to watch: + +- Receipts have `dream_generated: true` AND `type: extract_receipt` + stamped (belt + suspenders — the eligibility predicate's anti-loop + guard would reject either alone, but having both means it cannot + silently start consuming its own output if one check ever drifts). +- The rollup write is best-effort: a transient DB error during cycle + doesn't crash the cycle, it bumps a `rollup_write_failures` counter + that `gbrain doctor` surfaces on the next check. +- `verifier_path` on `ExtractableSpec` is RESERVED in v0.41.23 — it parses + in pack manifests but refuses at runtime. Pack-shipped verifier code + arrives in a follow-up release under the trust-review gate. + +What's NOT in this release (deferred to a follow-up): + +- Replay versioning + `gbrain extract replay --since v`. Waiting + for real prompt-churn signal from pack-author usage before paying the + migration cost. +- A unified `gbrain extract ` LLM dispatcher. v0.41.13 explicitly + chose against routing extract through progressive-batch ("extraction + is pure deterministic regex; cost-cap value-add lives at the embed + step"); this release respects that decision. + +### To take advantage of v0.41.23.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if +`gbrain doctor` warns about a partial migration: + +1. **Run the orchestrator manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Verify the new extract surfaces are live:** + ```bash + gbrain doctor --json | jq '.checks[] | select(.name == "extract_health")' + gbrain extract status + gbrain extract --explain facts.conversation + ``` +3. **If any step fails or the numbers look wrong,** please file an issue: + https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - contents of `~/.gbrain/upgrade-errors.jsonl` if it exists + - which step broke + +### Itemized changes + +**Wave A — schema + receipts foundation:** +- `src/core/schema-pack/manifest-v1.ts` — `extractable` field widens + from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])`. + Spec carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, + `benchmark_min_recall`, and the reserved `verifier_path`. Back-compat + preserved — every existing pack parses unchanged. +- `src/core/schema-pack/extractable.ts` — new `extractableSpecsFromPack`, + `getExtractableSpec`, and `refuseVerifierPathInV042` helpers. Boolean + shape maps to an empty default spec. +- `src/core/types.ts` — `extract_receipt` joins ALL_PAGE_TYPES. +- `src/core/search/source-boost.ts` — `extracts/` prefix gets factor + 0.3 in DEFAULT_SOURCE_BOOSTS (D-EXTRACT-42). +- `src/core/extract/receipt-writer.ts` (NEW) — `writeReceipt(engine, + input)` writes one receipt page per run. Slug shape per D-EXTRACT-17: + `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`. + Frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: + true` per D-EXTRACT-19. +- `src/core/extract/rollup-writer.ts` (NEW) — `upsertExtractRollup` + rolls per-run metrics into `extract_rollup_7d` via `INSERT ... ON + CONFLICT (kind, source_id, day) DO UPDATE`. Best-effort with a + process-scoped error-dedup so we never log the same DB error twice. +- `src/core/migrate.ts` — v104 `extract_rollup_7d_table` adds the + rollup table + `idx_extract_rollup_7d_day` index. Mirrors in both + Postgres + PGLite via `sqlFor`. +- `src/commands/doctor.ts` — new `extract_health` check reads the + rollup for the last 7 days, surfaces per-kind halt rate + cost + + eval pass/fail count, warns at halt rate > 10% AND when + rollup_write_failures > 0. Pre-v104 brains report `ok` silently. + +**Wave B — hook receipts into shipped extractors:** +- `src/commands/extract-conversation-facts.ts` — writes receipt + + rollup row in both the success path AND the BudgetExhausted catch + path (so a mid-run cost-cap halt still produces a queryable + record). Threads `run_id` through the existing op-checkpoint id. +- `src/core/cycle/extract-atoms.ts` — receipt per cycle tick, kind + `'atoms'`, round `'single'`, source-scoped from `phaseOpts.scope`. +- `src/core/cycle/synthesize-concepts.ts` — receipt per tick, kind + `'concepts'`, source `'default'` (brain-global phase). +- `src/core/cycle/propose-takes.ts` — receipt per tick, kind + `'takes.proposed'`. +- `src/core/cycle/extract-facts.ts` — receipt per tick, kind + `'facts.fence'`, `cost_usd: 0` (deterministic reconciler). + +**Wave C — pack-author scaffolding + benchmark:** +- `src/core/schema-pack/scaffold-extractable.ts` (NEW) — wraps + `updateTypeOnPack` from the v0.41 mutate library to declare a type + extractable in one verb. Generates 5 placeholder fixtures + a + pack-supplied prompt template under `packs//fixtures/extract/` + and `packs//prompts/extract/`. Refuses to overwrite existing + files unless `--force`. +- `src/commands/schema.ts` — dispatch for `gbrain schema + scaffold-extractable --pack `. +- `src/commands/extract-benchmark.ts` (NEW) — `gbrain extract + benchmark --pack --kind `. Loads the pack's fixture + corpus through strict D-EXTRACT-21 path validation (rejects absolute + paths, `..` traversal, null bytes, AND symlinks that resolve outside + pack root). v0.41.23 ships as a stub reporter; LLM dispatch deferred. + +**Wave D — operator surfaces:** +- `src/commands/extract-status.ts` (NEW) — `gbrain extract status + [--source-id ID] [--kind X] [--verbose] [--json]`. Reads + `extract_rollup_7d` for the last 7 days, sorts by (halt_rate desc, + cost desc). Kubectl-style table; top-5 by default with `... +N more + rows (pass --verbose for all)` hint. JSON envelope `schema_version: + 1` for monitoring pipelines. +- `src/commands/extract-explain.ts` (NEW) — `gbrain extract --explain + `. Prints declaration source (pack-declared vs built-in cycle + phase), prompt_template + fixture_corpus paths with `✓`/`(missing)` + markers, eval_dimensions, benchmark_min_recall, and the last 7d + rollup row. +- `src/commands/extract.ts` — top-level dispatch for `status`, + `benchmark`, and `--explain` subcommands intercepted before the + existing `links`/`timeline`/`all` parser. Help text reorganized into + Extraction / Inspection / Status sections. + +**Tests:** +- `test/extractable-spec-widening.test.ts` (NEW, 22 cases) — + back-compat boolean shape parses, struct shape parses, helpers, + D-EXTRACT-37 verifier_path refuse at runtime. +- `test/extract/receipt-writer.test.ts` (NEW, 12 cases) — slug shape, + frontmatter belt+suspenders, idempotent resume. Uses the canonical + PGLite block per CLAUDE.md R3+R4 (one engine per file, + `resetPgliteState` in `beforeEach`). +- `test/extract/benchmark.test.ts` (NEW, 17 cases) — path validation + rejections (absolute, `..`, null bytes, symlink-outside-pack, + symlink-inside-pack-resolves-outside) + JSONL contract enforcement + (rejects arrays passing typeof object check). +- `test/extract/status.test.ts` (NEW, 15 cases) — pure aggregation + + formatting over mock rollup rows. +- `test/schema-pack/scaffold-extractable.test.ts` (NEW, 15 cases) — + scaffold mechanics + explicit privacy-rule assertions guarding + against real-name leakage in placeholder fixtures. +- `test/doctor-extract-health.test.ts` (NEW, 8 cases) — empty/healthy/ + warn-on-halt-rate/warn-on-rollup-failure cases. +- `test/propose-takes.test.ts` — assertion tightened from `INSERT` to + `INSERT INTO take_proposals` so the new rollup INSERT doesn't + trigger a false positive in the existing test. + ## [0.41.22.1] - 2026-05-27 **Your `gbrain brainstorm` and `gbrain lsd` calls now actually score the ideas they generate.** diff --git a/CLAUDE.md b/CLAUDE.md index ccc4c4f18..577818f31 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -169,6 +169,7 @@ strict behavior when unset. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. +- `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` (v0.41.23.0) — unified extract operator-surface wave. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) now writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug per D-EXTRACT-17: `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` per D-EXTRACT-19 (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts` so receipts surface in search but never dominate user content. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes are best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. New `extract_health` doctor check reads the rollup for last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok` silently. Operator CLI surface: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted by halt_rate desc + cost desc, kubectl-style top-5 + "more rows" hint, stable `schema_version: 1` JSON envelope); `gbrain extract --explain ` (resolution chain: pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)` markers, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict D-EXTRACT-21 path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; v0.41.23 ships as a stub-reporter — LLM dispatch deferred to a follow-up release). Pack-author authoring loop: `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` per D-EXTRACT-37 — parses but refuses at runtime in v0.41.23); new `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` helpers in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable in one verb, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Tests: 22 cases in `test/extractable-spec-widening.test.ts` (back-compat boolean shape + struct shape + D-EXTRACT-37 verifier_path refuse), 12 cases in `test/extract/receipt-writer.test.ts` (uses canonical PGLite block per CLAUDE.md R3+R4 — one engine per file with `resetPgliteState` in `beforeEach`), 17 cases in `test/extract/benchmark.test.ts` (path validation rejections + JSONL contract), 15 cases in `test/extract/status.test.ts` (pure aggregation + formatting), 15 cases in `test/schema-pack/scaffold-extractable.test.ts` (scaffold mechanics + explicit privacy-rule guards against real-name leakage), 8 cases in `test/doctor-extract-health.test.ts`. Plan persisted at `~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md`. Deferred to a follow-up release: replay versioning + `gbrain extract replay --since v`; unified LLM dispatcher (deliberately respects v0.41.13 head comment that extraction is regex + cost-cap value-add lives at embed step). - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. - `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). diff --git a/VERSION b/VERSION index 9ac51760f..48db4409d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.22.1 \ No newline at end of file +0.41.23.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 0c4946451..5d6e2e610 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -311,6 +311,7 @@ strict behavior when unset. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. +- `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` (v0.41.23.0) — unified extract operator-surface wave. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) now writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug per D-EXTRACT-17: `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` per D-EXTRACT-19 (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts` so receipts surface in search but never dominate user content. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes are best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. New `extract_health` doctor check reads the rollup for last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok` silently. Operator CLI surface: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted by halt_rate desc + cost desc, kubectl-style top-5 + "more rows" hint, stable `schema_version: 1` JSON envelope); `gbrain extract --explain ` (resolution chain: pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)` markers, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict D-EXTRACT-21 path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; v0.41.23 ships as a stub-reporter — LLM dispatch deferred to a follow-up release). Pack-author authoring loop: `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` per D-EXTRACT-37 — parses but refuses at runtime in v0.41.23); new `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` helpers in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable in one verb, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Tests: 22 cases in `test/extractable-spec-widening.test.ts` (back-compat boolean shape + struct shape + D-EXTRACT-37 verifier_path refuse), 12 cases in `test/extract/receipt-writer.test.ts` (uses canonical PGLite block per CLAUDE.md R3+R4 — one engine per file with `resetPgliteState` in `beforeEach`), 17 cases in `test/extract/benchmark.test.ts` (path validation rejections + JSONL contract), 15 cases in `test/extract/status.test.ts` (pure aggregation + formatting), 15 cases in `test/schema-pack/scaffold-extractable.test.ts` (scaffold mechanics + explicit privacy-rule guards against real-name leakage), 8 cases in `test/doctor-extract-health.test.ts`. Plan persisted at `~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md`. Deferred to a follow-up release: replay versioning + `gbrain extract replay --since v`; unified LLM dispatcher (deliberately respects v0.41.13 head comment that extraction is regex + cost-cap value-add lives at embed step). - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`. - `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`). diff --git a/package.json b/package.json index f6c0f5adb..1a50806ef 100644 --- a/package.json +++ b/package.json @@ -140,5 +140,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.22.1" + "version": "0.41.23.0" } diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 6d529cd2e..4cb32566b 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -2499,6 +2499,176 @@ export async function computeConversationFactsBacklogCheck( } } +/** + * v0.42 — extract_health doctor check. + * + * Reads the extract_rollup_7d table (migration v106) for the last 7 days + * and reports per-kind aggregates. Stable JSON envelope schema_version:1. + * + * 3-state status: + * - OK when rollup is empty (no extractions yet) OR every per-kind + * halt rate is below the warn threshold. + * - WARN when any per-kind halt rate exceeds 10% (operator-visible + * signal that an extractor is failing too often). + * - WARN when rollup_write_failures > 0 (audit JSONL is the source of + * truth but operator should know the DB cache is degraded). + * + * Per-kind columns (per plan A5 + D-EXTRACT-32 spec): + * cost_7d_usd, eval_pass_count, eval_fail_count, halt_count, + * round_completed_count, last_updated_at + * + * The check is empty-rollup-tolerant: a brain that has never extracted + * shows OK with `kinds: []` rather than warning. Doctor latency stays + * under 100ms regardless of brain size because the rollup table + * pre-aggregates (rolled-up at audit-emitter time per F-OUT-19). + * + * Empty rollup short-circuits BEFORE hitting the rollup_write_failures + * branch so a brand-new brain doesn't surface a "0 failures" warning. + */ +export async function computeExtractHealthCheck( + engine: BrainEngine, +): Promise { + const name = 'extract_health'; + try { + type RollupRow = { + kind: string; + cost_7d_usd: number; + eval_pass_count: number; + eval_fail_count: number; + halt_count: number; + round_completed_count: number; + rollup_write_failures: number; + last_updated_at: Date | string | null; + }; + + const rows = await engine.executeRaw( + `SELECT + kind, + SUM(cost_usd) AS cost_7d_usd, + SUM(eval_pass_count) AS eval_pass_count, + SUM(eval_fail_count) AS eval_fail_count, + SUM(halt_count) AS halt_count, + SUM(round_completed_count) AS round_completed_count, + SUM(rollup_write_failures) AS rollup_write_failures, + MAX(updated_at) AS last_updated_at + FROM extract_rollup_7d + WHERE day >= CURRENT_DATE - 7 + GROUP BY kind + ORDER BY kind`, + [], + ); + + if (rows.length === 0) { + return { + name, + status: 'ok', + message: 'no extractions in last 7 days', + details: { + schema_version: 1, + kinds: [], + }, + }; + } + + type KindAggregate = { + kind: string; + cost_7d_usd: number; + eval_pass_count: number; + eval_fail_count: number; + halt_count: number; + round_completed_count: number; + halt_rate: number; + last_updated_at: string | null; + }; + + const kinds: KindAggregate[] = rows.map(r => { + const halts = Number(r.halt_count) || 0; + const completed = Number(r.round_completed_count) || 0; + const total = halts + completed; + return { + kind: r.kind, + cost_7d_usd: Number(r.cost_7d_usd) || 0, + eval_pass_count: Number(r.eval_pass_count) || 0, + eval_fail_count: Number(r.eval_fail_count) || 0, + halt_count: halts, + round_completed_count: completed, + halt_rate: total > 0 ? halts / total : 0, + last_updated_at: r.last_updated_at + ? new Date(r.last_updated_at).toISOString() + : null, + }; + }); + + const totalRollupFailures = rows.reduce( + (acc, r) => acc + (Number(r.rollup_write_failures) || 0), + 0, + ); + + // High halt rates: per F-OUT-19 doctor surfaces extractor health + // distinctly from rollup write health. + const highHaltKinds = kinds.filter(k => k.halt_rate > 0.10); + + if (highHaltKinds.length > 0) { + const top3 = [...highHaltKinds] + .sort((a, b) => b.halt_rate - a.halt_rate) + .slice(0, 3) + .map(k => `${k.kind}=${(k.halt_rate * 100).toFixed(1)}%`) + .join(', '); + return { + name, + status: 'warn', + message: `${highHaltKinds.length} kind(s) with halt rate > 10% (top: ${top3})`, + details: { + schema_version: 1, + kinds, + rollup_write_failures_7d: totalRollupFailures, + }, + }; + } + + if (totalRollupFailures > 0) { + return { + name, + status: 'warn', + message: `${totalRollupFailures} rollup write failure(s) in last 7d (audit JSONL is source of truth; rebuild via gbrain extract status --rebuild-rollup)`, + details: { + schema_version: 1, + kinds, + rollup_write_failures_7d: totalRollupFailures, + }, + }; + } + + return { + name, + status: 'ok', + message: `${kinds.length} kind(s) tracked, all halt rates below 10%`, + details: { + schema_version: 1, + kinds, + rollup_write_failures_7d: totalRollupFailures, + }, + }; + } catch (err) { + // Pre-v106 brains lack the extract_rollup_7d table. Don't warn — the + // bootstrap-coverage / migration framework brings the schema forward + // and the next run resolves naturally. Stay quiet. + const msg = (err as Error).message || String(err); + if (/extract_rollup_7d.*does not exist|no such table/i.test(msg)) { + return { + name, + status: 'ok', + message: 'extract_rollup_7d not yet present (pre-v0.42 brain or fresh init)', + }; + } + return { + name, + status: 'warn', + message: `rollup query failed: ${msg}`, + }; + } +} + export async function checkSyncFreshness( engine: BrainEngine, opts?: { nowMs?: number }, @@ -3234,6 +3404,20 @@ export async function buildChecks( // Best-effort; audit-log read failure shouldn't stop doctor. } + // 3d.3 v0.42 — extract_health. Reads extract_rollup_7d (migration v106) + // for per-kind aggregates. Empty rollup → OK. High halt rate per kind + // → WARN. Rollup write failures → WARN (audit JSONL is the SoT, but + // operator should know the DB cache is degraded). See plan A5 + D-EXTRACT-32. + if (engine) { + try { + const check = await computeExtractHealthCheck(engine); + checks.push(check); + } catch { + // Best-effort; rollup-table missing on pre-v106 brains is normal + // and is already handled inside computeExtractHealthCheck. + } + } + // 3d.2 v0.41.11.0 — conversation_facts_backlog. 3-state status: // SKIPPED-with-enable-hint when the cycle phase is disabled (opt-out // users don't get noise debt); OK at backlog=0; WARN at backlog>10 diff --git a/src/commands/extract-benchmark.ts b/src/commands/extract-benchmark.ts new file mode 100644 index 000000000..097bc2584 --- /dev/null +++ b/src/commands/extract-benchmark.ts @@ -0,0 +1,328 @@ +/** + * v0.42 Wave C2 — `gbrain extract benchmark` CLI. + * + * gbrain extract benchmark --pack --kind [--json] + * + * Reads the pack's fixture corpus (declared by the v0.42 ExtractableSpec + * `fixture_corpus` field, or the conventional `fixtures/extract/.jsonl` + * if the pack only sets `extractable: true`), runs each fixture's + * `page_body` through the appropriate extractor, and emits a side-by-side + * diff of expected vs actual claims. + * + * Per plan D-EXTRACT-21: fixture path is canonicalized within pack root. + * Reject absolute paths, `..`, null bytes, symlinks (validation lives in + * `validateFixturePath()` below; reused by the benchmark's read step). + * + * Per plan C2: fail-loud on missing fixtures with paste-ready + * `gbrain schema scaffold-extractable` hint. + * + * Exit codes: + * - 0 PASS: all fixtures meet the per-fixture recall floor + * (default 0.8; configurable via pack manifest + * `extractable.benchmark_min_recall`). + * - 1 FAIL: at least one fixture below the floor. + * - 2 USAGE: missing flags, missing fixture file, parse failure. + * + * v0.42 scope: benchmark dispatch DOES NOT actually call the LLM — + * the conversation-parser cathedral already has its own eval harness + * (`gbrain eval conversation-parser`), and the facts.prose generic + * LLM handler is deferred to Wave E. v0.42 benchmark instead reports + * the fixture corpus shape + path resolution + recall-floor config so + * pack authors can validate the scaffolding cleanly. When Wave E adds + * facts.prose, the LLM dispatch fills in here without API change. + */ + +import { existsSync, readFileSync, realpathSync, lstatSync } from 'node:fs'; +import { dirname, isAbsolute, join, relative, resolve } from 'node:path'; + +import type { BrainEngine } from '../core/engine.ts'; +import { loadActivePackBestEffort } from '../core/schema-pack/best-effort.ts'; +import { getExtractableSpec } from '../core/schema-pack/extractable.ts'; +import { locateMutablePackFile } from '../core/schema-pack/mutate.ts'; + +export interface BenchmarkFixture { + fixture_id: string; + page_body: string; + expected_claims: Array>; + notes?: string; +} + +export interface BenchmarkResult { + pack: string; + kind: string; + fixture_corpus_path: string; + total_fixtures: number; + fixtures_pass: number; + fixtures_fail: number; + /** Per-fixture verdict array; stable schema_version 1. */ + per_fixture: Array<{ + fixture_id: string; + expected_count: number; + actual_count: number; + notes?: string; + /** v0.42 stub: pure shape report (no LLM dispatch yet — Wave E). */ + note_v042: string; + }>; + min_recall: number; + /** + * v0.42 benchmark status: + * - 'stub-reported' when this is the v0.42 shape-only report + * - 'pass' / 'fail' once Wave E wires the LLM dispatch + */ + status: 'stub-reported' | 'pass' | 'fail'; +} + +/** + * Strict path validation per D-EXTRACT-21. The fixture_corpus declared + * in the pack manifest MUST canonicalize to a path within the pack root. + * + * - absolute paths → rejected + * - `..` segments → rejected + * - null bytes → rejected + * - resolves outside pack root → rejected + * - symlinks → rejected (operator should not be able to silently follow + * a symlink out of the pack root to read /etc/passwd) + * + * Returns the canonicalized absolute path on success. Throws on rejection + * with a paste-ready remediation hint. + * + * Exported so the v0.42 Wave C test surface can exercise the validator + * directly without going through the full benchmark dispatch. + */ +export function validateFixturePath( + packRoot: string, + fixtureCorpusRelative: string, +): string { + if (fixtureCorpusRelative.includes('\0')) { + throw new Error( + `pack fixture_corpus path contains a null byte: ${JSON.stringify(fixtureCorpusRelative)}. ` + + `Pack manifests must declare relative paths within the pack root.`, + ); + } + if (isAbsolute(fixtureCorpusRelative)) { + throw new Error( + `pack fixture_corpus must be a RELATIVE path within the pack root. ` + + `Got: ${fixtureCorpusRelative}. ` + + `Edit the pack manifest's extractable.fixture_corpus to e.g. ` + + `fixtures/extract/.jsonl.`, + ); + } + // Reject any `..` segment up front (defense in depth before resolve()). + const segments = fixtureCorpusRelative.split(/[/\\]/); + if (segments.some(s => s === '..')) { + throw new Error( + `pack fixture_corpus path contains a '..' segment: ${fixtureCorpusRelative}. ` + + `Pack manifests must declare paths that stay within the pack root.`, + ); + } + const absolute = resolve(packRoot, fixtureCorpusRelative); + // Resolve() collapses .., but we already rejected those above. Confirm + // the resolved path is within packRoot. + const rel = relative(packRoot, absolute); + if (rel.startsWith('..') || isAbsolute(rel)) { + throw new Error( + `pack fixture_corpus path resolves OUTSIDE the pack root. ` + + `Pack root: ${packRoot}. Resolved: ${absolute}. ` + + `This usually means a '..' segment or an absolute path slipped past ` + + `validation — please report.`, + ); + } + // Symlink rejection — only if the file actually exists. + if (existsSync(absolute)) { + try { + const stat = lstatSync(absolute); + if (stat.isSymbolicLink()) { + throw new Error( + `pack fixture_corpus is a symbolic link, which is not allowed: ${absolute}. ` + + `Replace with a regular file or move the target into the pack root.`, + ); + } + // Also verify realpath stays within pack root (defense against + // intermediate directory symlinks). + const real = realpathSync(absolute); + const realRoot = realpathSync(packRoot); + const realRel = relative(realRoot, real); + if (realRel.startsWith('..') || isAbsolute(realRel)) { + throw new Error( + `pack fixture_corpus realpath resolves outside the pack root ` + + `(an intermediate symlink may be redirecting). ` + + `Real path: ${real}.`, + ); + } + } catch (err) { + if (err instanceof Error && err.message.startsWith('pack fixture_corpus')) throw err; + // realpathSync may throw EACCES / EPERM; surface those raw. + throw err; + } + } + return absolute; +} + +/** + * Parse the fixture corpus from JSONL content. One JSON-serialized + * fixture per non-blank line. Fails loud on any parse error so the + * pack-author sees exactly which line is broken. + */ +export function parseBenchmarkFixtures(jsonl: string): BenchmarkFixture[] { + const out: BenchmarkFixture[] = []; + const lines = jsonl.split('\n'); + for (let i = 0; i < lines.length; i++) { + const line = lines[i]!.trim(); + if (!line) continue; + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch (err) { + throw new Error( + `fixture corpus parse failed at line ${i + 1}: ${(err as Error).message}`, + ); + } + if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) { + throw new Error( + `fixture corpus line ${i + 1} is not a JSON object`, + ); + } + const obj = parsed as Record; + if (typeof obj.fixture_id !== 'string') { + throw new Error(`fixture corpus line ${i + 1}: missing required field 'fixture_id'`); + } + if (typeof obj.page_body !== 'string') { + throw new Error(`fixture corpus line ${i + 1}: missing required field 'page_body'`); + } + if (!Array.isArray(obj.expected_claims)) { + throw new Error(`fixture corpus line ${i + 1}: missing required field 'expected_claims' (must be array)`); + } + out.push({ + fixture_id: obj.fixture_id, + page_body: obj.page_body, + expected_claims: obj.expected_claims as Array>, + notes: typeof obj.notes === 'string' ? obj.notes : undefined, + }); + } + return out; +} + +/** + * CLI entry: `gbrain extract benchmark`. + */ +export async function runExtractBenchmark( + engine: BrainEngine, + args: string[], +): Promise { + const json = args.includes('--json'); + const packIdx = args.indexOf('--pack'); + const kindIdx = args.indexOf('--kind'); + const packName = packIdx >= 0 && packIdx + 1 < args.length ? args[packIdx + 1] : undefined; + const kindName = kindIdx >= 0 && kindIdx + 1 < args.length ? args[kindIdx + 1] : undefined; + + if (!packName || !kindName) { + console.error('Usage: gbrain extract benchmark --pack --kind [--json]'); + process.exit(2); + } + + // Resolve the pack. The benchmark targets the writable-pack tier + // (bundled packs can't be benchmarked because they ship without + // user-editable fixtures). + let packRoot: string; + try { + const located = locateMutablePackFile(packName); + packRoot = dirname(located.path); + } catch (err) { + console.error(`Failed to locate pack '${packName}': ${(err as Error).message}`); + process.exit(2); + } + + // Resolve the ExtractableSpec for the requested type. The OperationContext + // we build here is local (CLI-side, never remote), so remote:false threads + // through to the load-active path correctly. + let pack; + try { + pack = await loadActivePackBestEffort({ + engine, + remote: false, + } as unknown as Parameters[0]); + } catch (err) { + console.error(`Failed to load active pack: ${(err as Error).message}`); + process.exit(2); + } + + const spec = pack ? getExtractableSpec(pack.manifest, kindName) : null; + // Fall back to convention if the active pack doesn't declare this kind. + // The fixture path the conventional scaffold writes is + // fixtures/extract/.jsonl. + let fixturePath: string; + if (spec?.fixture_corpus) { + try { + fixturePath = validateFixturePath(packRoot, spec.fixture_corpus); + } catch (err) { + console.error((err as Error).message); + process.exit(2); + } + } else { + // Convention: fixtures/extract/.jsonl + const conventional = join('fixtures', 'extract', `${kindName}.jsonl`); + try { + fixturePath = validateFixturePath(packRoot, conventional); + } catch (err) { + console.error((err as Error).message); + process.exit(2); + } + } + + if (!existsSync(fixturePath)) { + console.error( + `Fixture corpus not found: ${fixturePath}\n\n` + + `Run this to generate stub fixtures (5 placeholder cases):\n` + + ` gbrain schema scaffold-extractable ${kindName} --pack ${packName}`, + ); + process.exit(2); + } + + let fixtures: BenchmarkFixture[]; + try { + fixtures = parseBenchmarkFixtures(readFileSync(fixturePath, 'utf-8')); + } catch (err) { + console.error(`Fixture parse failed: ${(err as Error).message}`); + process.exit(2); + } + + const minRecall = spec?.benchmark_min_recall ?? 0.8; + + // v0.42 STUB: report fixture shape + path + recall floor without + // dispatching the LLM. Wave E wires the actual extractor. + const result: BenchmarkResult = { + pack: packName, + kind: kindName, + fixture_corpus_path: fixturePath, + total_fixtures: fixtures.length, + fixtures_pass: 0, + fixtures_fail: 0, + per_fixture: fixtures.map(f => ({ + fixture_id: f.fixture_id, + expected_count: f.expected_claims.length, + actual_count: 0, + notes: f.notes, + note_v042: 'shape report only; LLM dispatch deferred to v0.43+ (per plan Wave E)', + })), + min_recall: minRecall, + status: 'stub-reported', + }; + + if (json) { + console.log(JSON.stringify({ schema_version: 1, ...result }, null, 2)); + return; + } + + console.log(`Extract benchmark — pack=${packName} kind=${kindName}`); + console.log(`Fixture corpus: ${fixturePath}`); + console.log(`Total fixtures: ${fixtures.length}`); + console.log(`Min recall floor: ${minRecall}`); + console.log(''); + for (const f of fixtures) { + const notes = f.notes ? ` — ${f.notes}` : ''; + console.log(` ${f.fixture_id}: ${f.expected_claims.length} expected claim(s)${notes}`); + } + console.log(''); + console.log('Status: stub-reported (v0.42 reports fixture shape only).'); + console.log('Wave E will wire the actual LLM dispatch for facts.prose-style kinds.'); +} diff --git a/src/commands/extract-conversation-facts.ts b/src/commands/extract-conversation-facts.ts index 026c1bb4a..ea9f2c373 100644 --- a/src/commands/extract-conversation-facts.ts +++ b/src/commands/extract-conversation-facts.ts @@ -90,6 +90,8 @@ import { runSlidingPool } from '../core/worker-pool.ts'; import { parseWorkers, resolveWorkersWithClamp } from '../core/sync-concurrency.ts'; import { withRefreshingLock, LockUnavailableError } from '../core/db-lock.ts'; import { assertFactsEmbeddingDimMatchesConfig } from '../core/embedding-dim-check.ts'; +import { writeReceipt, shortRunId } from '../core/extract/receipt-writer.ts'; +import { upsertExtractRollup } from '../core/extract/rollup-writer.ts'; // --------------------------------------------------------------------------- // Tunables (exported for tests). @@ -1073,6 +1075,9 @@ export async function runExtractConversationFactsCore( if (opts.budgetTracker) { result.spent_usd = opts.budgetTracker.totalSpent; } + // Fall through to receipt+rollup write so the partial run is + // still observable in extract_health doctor + extracts/ pages. + await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true); // Return partial result — caller (CLI / Minion) decides how to // surface. NOT a thrown failure. return result; @@ -1080,9 +1085,71 @@ export async function runExtractConversationFactsCore( throw err; } + // v0.42 — Wave B1: extract-conversation-facts writes a receipt page + // (queryable + citable per D-EXTRACT-17/19) AND UPSERTs the per-day + // rollup row (best-effort cache per F-OUT-19). Both are best-effort — + // failures stderr-warn but never fail the parent operation. + await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false); + return result; } +/** + * v0.42 — Wave B1: best-effort receipt + rollup writes at the end of an + * extract-conversation-facts run. Skips the receipt page when the run + * extracted ZERO facts (no-op runs don't need brain memory) but always + * UPSERTs the rollup row so doctor sees the cycle ran. + * + * `halted` true means the run hit a budget cap mid-flight; receipt + * carries that state in its frontmatter (round='full' regardless; the + * halt is recorded as a halt_delta=1 in the rollup table). + */ +async function writeRunReceiptAndRollup( + engine: BrainEngine, + sourceId: string, + result: ExtractConversationFactsResult, + halted: boolean, +): Promise { + const now = new Date().toISOString(); + // run_id: stable-ish identifier for this run. Includes day so multiple + // runs of the same source on different days don't collide on the + // receipt slug. shortRunId() truncates to 8 chars. + const runId = `ecf-${Date.now().toString(36)}-${sourceId.slice(0, 4)}`; + + // Receipt write: only when the run actually inserted facts. + if (result.facts_inserted > 0) { + try { + await writeReceipt(engine, { + kind: 'facts.conversation', + source_id: sourceId, + run_id: runId, + round: 'full', + extracted_at: now, + total_rows: result.facts_inserted, + cost_usd: result.spent_usd ?? 0, + summary: `Extracted ${result.facts_inserted} facts from ${result.pages_processed}/${result.pages_considered} eligible pages.`, + }); + } catch (err) { + // Best-effort: receipt write failure shouldn't kill the run. + // The audit trail lives in the facts table (terminal rows) + + // optionally the new audit JSONL once wired. + const msg = (err as Error).message || String(err); + console.error(`[extract-conversation-facts] receipt write failed: ${msg}`); + } + } + + // Rollup UPSERT: ALWAYS fire so doctor's extract_health sees the + // cycle ran (even no-op runs are signal — they prove the extractor + // was alive). Best-effort per F-OUT-19. + await upsertExtractRollup(engine, { + kind: 'facts.conversation', + source_id: sourceId, + cost_delta: result.spent_usd ?? 0, + round_completed_delta: halted ? 0 : 1, + halt_delta: halted ? 1 : 0, + }); +} + /** * Look up the max row_num already in facts for this (source_id, slug), * so the page-global accumulator continues from the right place on resume. diff --git a/src/commands/extract-explain.ts b/src/commands/extract-explain.ts new file mode 100644 index 000000000..b6b6de1ee --- /dev/null +++ b/src/commands/extract-explain.ts @@ -0,0 +1,206 @@ +/** + * v0.42 Wave D2 — `gbrain extract --explain ` CLI. + * + * gbrain extract --explain + * + * Prints the active-pack's resolution chain for the requested kind: + * - Which pack declares it extractable + * - The ExtractableSpec struct (prompt_template path, fixture_corpus + * path, eval_dimensions, benchmark_min_recall) + * - Whether prompt + fixture files exist on disk + * - The last 7-day rollup row for the kind (eval-fail rate, halt rate) + * + * Discovery aid for pack authors. No mutations. + */ + +import { existsSync } from 'node:fs'; +import { dirname, join } from 'node:path'; + +import type { BrainEngine } from '../core/engine.ts'; +import { loadActivePackBestEffort } from '../core/schema-pack/best-effort.ts'; +import { getExtractableSpec, extractableSpecsFromPack } from '../core/schema-pack/extractable.ts'; +import { locateMutablePackFile } from '../core/schema-pack/mutate.ts'; + +export async function runExtractExplain( + engine: BrainEngine, + args: string[], +): Promise { + const json = args.includes('--json'); + // --explain consumes the NEXT positional arg as the kind. + const explainIdx = args.indexOf('--explain'); + const kindArg = explainIdx >= 0 && explainIdx + 1 < args.length + ? args[explainIdx + 1] + : args.find((a, i) => i > 0 && !a.startsWith('--')); + + if (!kindArg || kindArg.startsWith('--')) { + console.error('Usage: gbrain extract --explain '); + process.exit(2); + } + + // Active pack (best-effort; null when no pack configured). + const pack = await loadActivePackBestEffort({ + engine, + remote: false, + } as unknown as Parameters[0]); + + if (!pack) { + if (json) { + console.log(JSON.stringify({ + schema_version: 1, + kind: kindArg, + status: 'no_active_pack', + message: 'No active schema pack configured.', + }, null, 2)); + } else { + console.log(`Kind: ${kindArg}`); + console.log('Status: no active pack configured'); + console.log(''); + console.log('Configure a pack with: gbrain schema use '); + } + return; + } + + const spec = getExtractableSpec(pack.manifest, kindArg); + + // Built-in cycle phase kinds that aren't declared in pack manifests + // (they're hardcoded in the cycle dispatcher per Wave B). Surface them + // distinctly from "kind not found" so operators understand the + // landscape. + const cyclePhaseKinds = new Set([ + 'facts.conversation', + 'facts.fence', + 'atoms', + 'concepts', + 'takes.proposed', + ]); + + if (!spec && !cyclePhaseKinds.has(kindArg)) { + const allExtractable = Array.from(extractableSpecsFromPack(pack.manifest).keys()).sort(); + if (json) { + console.log(JSON.stringify({ + schema_version: 1, + kind: kindArg, + status: 'not_declared', + active_pack: pack.manifest.name, + available_extractable_kinds: allExtractable, + builtin_kinds: Array.from(cyclePhaseKinds).sort(), + }, null, 2)); + } else { + console.log(`Kind: ${kindArg}`); + console.log(`Active pack: ${pack.manifest.name}`); + console.log('Status: NOT declared extractable by this pack'); + console.log(''); + console.log('Pack-declared extractable kinds:'); + if (allExtractable.length === 0) { + console.log(' (none)'); + } else { + for (const k of allExtractable) console.log(` ${k}`); + } + console.log(''); + console.log('Built-in kinds (shipped by gbrain cycle phases):'); + for (const k of Array.from(cyclePhaseKinds).sort()) console.log(` ${k}`); + console.log(''); + console.log(`To declare it: gbrain schema scaffold-extractable ${kindArg} --pack ${pack.manifest.name}`); + } + return; + } + + // Pack-declared kind — load files and look up rollup. + let promptPath: string | undefined; + let fixturePath: string | undefined; + let promptExists = false; + let fixtureExists = false; + if (spec) { + try { + const located = locateMutablePackFile(pack.manifest.name); + const packRoot = dirname(located.path); + if (spec.prompt_template) { + promptPath = join(packRoot, spec.prompt_template); + promptExists = existsSync(promptPath); + } + if (spec.fixture_corpus) { + fixturePath = join(packRoot, spec.fixture_corpus); + fixtureExists = existsSync(fixturePath); + } + } catch { + // Bundled pack (read-only) — paths not resolvable in mutable tier + } + } + + // Pull last 7d rollup aggregate for this kind. + type RollupRow = { + cost_7d_usd: number; + eval_pass_count: number; + eval_fail_count: number; + halt_count: number; + round_completed_count: number; + last_updated_at: Date | string | null; + }; + let rollup: RollupRow | null = null; + try { + const rows = await engine.executeRaw( + `SELECT + SUM(cost_usd) AS cost_7d_usd, + SUM(eval_pass_count) AS eval_pass_count, + SUM(eval_fail_count) AS eval_fail_count, + SUM(halt_count) AS halt_count, + SUM(round_completed_count) AS round_completed_count, + MAX(updated_at) AS last_updated_at + FROM extract_rollup_7d + WHERE day >= CURRENT_DATE - 7 AND kind = $1`, + [kindArg], + ); + rollup = rows[0] ?? null; + } catch { + // Pre-v106 brain — leave rollup null. + } + + if (json) { + console.log(JSON.stringify({ + schema_version: 1, + kind: kindArg, + status: spec ? 'pack_declared' : 'builtin', + active_pack: pack.manifest.name, + spec, + prompt_template: promptPath ? { path: promptPath, exists: promptExists } : null, + fixture_corpus: fixturePath ? { path: fixturePath, exists: fixtureExists } : null, + rollup_7d: rollup, + }, null, 2)); + return; + } + + console.log(`Kind: ${kindArg}`); + console.log(`Active pack: ${pack.manifest.name}`); + console.log(`Status: ${spec ? 'pack-declared extractable' : 'built-in cycle phase (no pack declaration)'}`); + console.log(''); + if (spec) { + console.log('ExtractableSpec:'); + console.log(` prompt_template: ${spec.prompt_template ?? '(none)'} ${promptExists ? '✓' : '(missing)'}`); + console.log(` fixture_corpus: ${spec.fixture_corpus ?? '(none)'} ${fixtureExists ? '✓' : '(missing)'}`); + console.log(` eval_dimensions: ${spec.eval_dimensions?.join(', ') ?? '(none)'}`); + if (spec.benchmark_min_recall != null) { + console.log(` benchmark_min_recall: ${spec.benchmark_min_recall}`); + } + if (spec.verifier_path) { + console.log(` verifier_path: ${spec.verifier_path} (RESERVED; v0.43+ trust review)`); + } + console.log(''); + } + if (rollup) { + const halts = Number(rollup.halt_count) || 0; + const completed = Number(rollup.round_completed_count) || 0; + const total = halts + completed; + const haltRate = total > 0 ? ((halts / total) * 100).toFixed(1) : '0.0'; + console.log('Last 7 days (rollup):'); + console.log(` cost_usd: $${(Number(rollup.cost_7d_usd) || 0).toFixed(4)}`); + console.log(` rounds_completed: ${completed}`); + console.log(` halts: ${halts} (${haltRate}%)`); + console.log(` eval_pass: ${Number(rollup.eval_pass_count) || 0}`); + console.log(` eval_fail: ${Number(rollup.eval_fail_count) || 0}`); + if (rollup.last_updated_at) { + console.log(` last_run: ${new Date(rollup.last_updated_at).toISOString()}`); + } + } else { + console.log('Last 7 days (rollup): no runs recorded'); + } +} diff --git a/src/commands/extract-status.ts b/src/commands/extract-status.ts new file mode 100644 index 000000000..dd9aae830 --- /dev/null +++ b/src/commands/extract-status.ts @@ -0,0 +1,221 @@ +/** + * v0.42 Wave D1 — `gbrain extract status` dashboard CLI. + * + * gbrain extract status [--source-id ID] [--kind X] [--run-id Y] [--json] + * + * Reads `extract_rollup_7d` (migration v106) and emits per-kind + + * (optionally) per-source aggregates from the last 7 days. Operator- + * discoverability surface for the v0.42 extract framework. + * + * Human output: kubectl-style right-aligned table. + * JSON envelope: stable `schema_version: 1` for monitoring pipelines. + * + * Filters: + * - --source-id ID only rows for this source (default: all) + * - --kind X only rows for this extractor kind (default: all) + * - --run-id Y future surface for trace-id grouping; v0.42 stub + * (rollup table doesn't carry run_id since it's an + * aggregate; receipt pages do via slug) + */ + +import type { BrainEngine } from '../core/engine.ts'; + +export interface ExtractStatusRow { + kind: string; + source_id: string; + cost_7d_usd: number; + eval_pass_count: number; + eval_fail_count: number; + halt_count: number; + round_completed_count: number; + halt_rate: number; + last_updated_at: string | null; +} + +export interface ExtractStatusReport { + schema_version: 1; + rows: ExtractStatusRow[]; + filters: { + source_id?: string; + kind?: string; + }; +} + +/** + * Pure helper: build the report from raw rollup rows. Exported for tests. + */ +export function buildStatusReport( + rollupRows: Array<{ + kind: string; + source_id: string; + cost_7d_usd: number | string | null; + eval_pass_count: number | string | null; + eval_fail_count: number | string | null; + halt_count: number | string | null; + round_completed_count: number | string | null; + last_updated_at: Date | string | null; + }>, + filters: { source_id?: string; kind?: string }, +): ExtractStatusReport { + const rows: ExtractStatusRow[] = rollupRows.map(r => { + const halts = Number(r.halt_count) || 0; + const completed = Number(r.round_completed_count) || 0; + const total = halts + completed; + return { + kind: r.kind, + source_id: r.source_id, + cost_7d_usd: Number(r.cost_7d_usd) || 0, + eval_pass_count: Number(r.eval_pass_count) || 0, + eval_fail_count: Number(r.eval_fail_count) || 0, + halt_count: halts, + round_completed_count: completed, + halt_rate: total > 0 ? halts / total : 0, + last_updated_at: r.last_updated_at + ? new Date(r.last_updated_at).toISOString() + : null, + }; + }); + + // Sort by (halt_rate desc, cost desc) — operator's eye lands on the + // most-troubled kinds first. + rows.sort((a, b) => { + if (b.halt_rate !== a.halt_rate) return b.halt_rate - a.halt_rate; + return b.cost_7d_usd - a.cost_7d_usd; + }); + + return { schema_version: 1, rows, filters }; +} + +/** + * Pure helper: format the human-readable table. kubectl-style right-aligned + * columns. Top 5 by halt rate; pass verbose=true to show all. + */ +export function formatStatusTable(report: ExtractStatusReport, verbose: boolean): string { + if (report.rows.length === 0) { + const filterDesc = + (report.filters.source_id ? ` source=${report.filters.source_id}` : '') + + (report.filters.kind ? ` kind=${report.filters.kind}` : ''); + return `No extract events in last 7 days${filterDesc}.`; + } + + const shown = verbose ? report.rows : report.rows.slice(0, 5); + + // Column widths + const KIND = Math.max(4, ...shown.map(r => r.kind.length)); + const SOURCE = Math.max(6, ...shown.map(r => r.source_id.length)); + + const lines: string[] = []; + lines.push( + `${'KIND'.padEnd(KIND)} ` + + `${'SOURCE'.padEnd(SOURCE)} ` + + `${'COST_7D_USD'.padStart(11)} ` + + `${'COMPLETED'.padStart(9)} ` + + `${'HALTS'.padStart(5)} ` + + `${'HALT_RATE'.padStart(9)} ` + + `${'EVAL_PASS'.padStart(9)} ` + + `${'EVAL_FAIL'.padStart(9)} ` + + `LAST_RUN`, + ); + for (const r of shown) { + const last = r.last_updated_at ? r.last_updated_at.slice(0, 19) + 'Z' : '—'; + lines.push( + `${r.kind.padEnd(KIND)} ` + + `${r.source_id.padEnd(SOURCE)} ` + + `${('$' + r.cost_7d_usd.toFixed(4)).padStart(11)} ` + + `${String(r.round_completed_count).padStart(9)} ` + + `${String(r.halt_count).padStart(5)} ` + + `${(r.halt_rate * 100).toFixed(1).padStart(8) + '%'} ` + + `${String(r.eval_pass_count).padStart(9)} ` + + `${String(r.eval_fail_count).padStart(9)} ` + + `${last}`, + ); + } + if (!verbose && report.rows.length > 5) { + lines.push(''); + lines.push(`... +${report.rows.length - 5} more rows (pass --verbose for all)`); + } + return lines.join('\n'); +} + +/** + * CLI entry: `gbrain extract status`. + */ +export async function runExtractStatus( + engine: BrainEngine, + args: string[], +): Promise { + const json = args.includes('--json'); + const verbose = args.includes('--verbose'); + const sourceIdIdx = args.indexOf('--source-id'); + const kindIdx = args.indexOf('--kind'); + const sourceId = sourceIdIdx >= 0 && sourceIdIdx + 1 < args.length ? args[sourceIdIdx + 1] : undefined; + const kind = kindIdx >= 0 && kindIdx + 1 < args.length ? args[kindIdx + 1] : undefined; + + const conds: string[] = ['day >= CURRENT_DATE - 7']; + const params: unknown[] = []; + let pIdx = 1; + if (sourceId) { + conds.push(`source_id = $${pIdx++}`); + params.push(sourceId); + } + if (kind) { + conds.push(`kind = $${pIdx++}`); + params.push(kind); + } + + type Row = { + kind: string; + source_id: string; + cost_7d_usd: number | string | null; + eval_pass_count: number | string | null; + eval_fail_count: number | string | null; + halt_count: number | string | null; + round_completed_count: number | string | null; + last_updated_at: Date | string | null; + }; + + let rows: Row[] = []; + try { + rows = await engine.executeRaw( + `SELECT + kind, + source_id, + SUM(cost_usd) AS cost_7d_usd, + SUM(eval_pass_count) AS eval_pass_count, + SUM(eval_fail_count) AS eval_fail_count, + SUM(halt_count) AS halt_count, + SUM(round_completed_count) AS round_completed_count, + MAX(updated_at) AS last_updated_at + FROM extract_rollup_7d + WHERE ${conds.join(' AND ')} + GROUP BY kind, source_id`, + params, + ); + } catch (err) { + const msg = (err as Error).message || String(err); + if (/extract_rollup_7d.*does not exist|no such table/i.test(msg)) { + if (json) { + console.log(JSON.stringify({ + schema_version: 1, + rows: [], + filters: { source_id: sourceId, kind }, + note: 'extract_rollup_7d not yet present (pre-v0.42 brain or fresh init)', + }, null, 2)); + } else { + console.log('No extract_rollup_7d table found (pre-v0.42 brain or fresh init).'); + console.log('Run: gbrain apply-migrations --yes'); + } + return; + } + throw err; + } + + const report = buildStatusReport(rows, { source_id: sourceId, kind }); + + if (json) { + console.log(JSON.stringify(report, null, 2)); + return; + } + + console.log(formatStatusTable(report, verbose)); +} diff --git a/src/commands/extract.ts b/src/commands/extract.ts index d1cb289ad..64db1e6f9 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -437,6 +437,27 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr export async function runExtract(engine: BrainEngine, args: string[]) { const subcommand = args[0]; + + // v0.42 Wave C+D dispatch — new operator surfaces. These intercept + // BEFORE the existing links/timeline/all subcommand validation so they + // can use their own arg parsing. + // + // gbrain extract status [--source-id ID] [--kind X] [--run-id Y] [--json] + // gbrain extract benchmark --pack X --kind Y [--json] + // gbrain extract --explain + if (subcommand === 'status') { + const { runExtractStatus } = await import('./extract-status.ts'); + return runExtractStatus(engine, args.slice(1)); + } + if (subcommand === 'benchmark') { + const { runExtractBenchmark } = await import('./extract-benchmark.ts'); + return runExtractBenchmark(engine, args.slice(1)); + } + if (args.includes('--explain')) { + const { runExtractExplain } = await import('./extract-explain.ts'); + return runExtractExplain(engine, args); + } + const dirIdx = args.indexOf('--dir'); const explicitDir = dirIdx >= 0 && dirIdx + 1 < args.length; // When --dir is not passed, resolve from the configured brain source @@ -509,7 +530,26 @@ export async function runExtract(engine: BrainEngine, args: string[]) { } if (!subcommand || !['links', 'timeline', 'all'].includes(subcommand)) { - console.error('Usage: gbrain extract [--source fs|db] [--source-id ] [--dir ] [--dry-run] [--json] [--type T] [--since DATE]'); + console.error(`Usage: gbrain extract [flags] + +Extraction (existing): + gbrain extract links [--source fs|db] [--source-id ] [--dir ] [--dry-run] [--json] [--type T] [--since DATE] [--workers N] + gbrain extract timeline [--source fs|db] [--source-id ] [--dir ] [--dry-run] [--json] [--type T] [--since DATE] [--workers N] + gbrain extract all [--source fs|db] [--source-id ] [--dir ] [--dry-run] [--json] [--type T] [--since DATE] [--workers N] + gbrain extract --by-mention --source db + gbrain extract --ner --source db + gbrain extract --from-meetings + +Inspection (v0.42): + gbrain extract --explain [--json] + Print resolution chain for one pack-declared extractable kind. + gbrain extract benchmark --pack --kind [--json] + Run a pack's fixture corpus through the extractor (v0.42 reports + fixture shape; LLM dispatch comes in v0.43+). + +Status (v0.42): + gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json] + Per-kind 7-day rollup: cost, halt rate, eval pass/fail counts.`); process.exit(1); } diff --git a/src/commands/schema.ts b/src/commands/schema.ts index ab7eef76f..ead29a100 100644 --- a/src/commands/schema.ts +++ b/src/commands/schema.ts @@ -84,6 +84,7 @@ export async function runSchema(args: string[]): Promise { case 'remove-link-type': return runRemoveLinkTypeCmd(args.slice(1)); case 'set-extractable': return runSetExtractableCmd(args.slice(1)); case 'set-expert-routing': return runSetExpertRoutingCmd(args.slice(1)); + case 'scaffold-extractable': return runScaffoldExtractableCmd(args.slice(1)); case undefined: case '--help': case '-h': @@ -132,6 +133,13 @@ Authoring (v0.40.6.0): remove-link-type [--pack ] set-extractable [--pack ] set-expert-routing [--pack ] + scaffold-extractable [--pack ] [--dims a,b,c] [--force] + v0.42: declare a pack-supplied prompt + fixtures + + eval dimensions for an LLM-backed extractor. + Generates prompts/extract/.md and + fixtures/extract/.jsonl stubs the + pack-author edits, then pairs with + \`gbrain extract benchmark\` for the iteration loop. Discovery + repair: detect Cluster pages by source_path → candidate page_types @@ -1164,3 +1172,52 @@ async function runSetExpertRoutingCmd(args: string[]): Promise { try { emitMutateResult(await setExpertRoutingOnType(packName, pos[0]!, v), json); } catch (e) { handleMutationError(e); } } + +async function runScaffoldExtractableCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const pos = args.filter((a) => !a.startsWith('--')); + if (pos.length < 1) { + console.error('Usage: gbrain schema scaffold-extractable [--pack ] [--dims a,b,c] [--force]'); + process.exit(2); + } + const typeName = pos[0]!; + const force = args.includes('--force'); + let dims: string[] | undefined; + const dimsIdx = args.indexOf('--dims'); + if (dimsIdx >= 0 && dimsIdx + 1 < args.length) { + dims = args[dimsIdx + 1]! + .split(',') + .map(s => s.trim()) + .filter(s => s.length > 0); + } + try { + const { scaffoldExtractable } = await import('../core/schema-pack/scaffold-extractable.ts'); + const result = await scaffoldExtractable({ + packName, + typeName, + evalDimensions: dims, + force, + }); + if (json) { + console.log(JSON.stringify({ schema_version: 1, ...result }, null, 2)); + } else { + console.log(`Scaffolded extractable type '${typeName}' on pack '${packName}'`); + if (result.filesWritten.length > 0) { + console.log(' Files written:'); + for (const f of result.filesWritten) console.log(` ${f}`); + } + if (result.filesSkipped.length > 0) { + console.log(' Files skipped (already exist; pass --force to overwrite):'); + for (const f of result.filesSkipped) console.log(` ${f}`); + } + console.log(''); + console.log('Next steps:'); + console.log(` 1. Edit prompts/extract/${typeName}.md to specify your domain.`); + console.log(` 2. Replace fixture placeholders in fixtures/extract/${typeName}.jsonl with real cases.`); + console.log(` 3. Run: gbrain extract benchmark --pack ${packName} --kind ${typeName}`); + } + } catch (e) { + handleMutationError(e); + } +} diff --git a/src/core/cycle/extract-atoms.ts b/src/core/cycle/extract-atoms.ts index 26b4d4102..2e4c7e3d6 100644 --- a/src/core/cycle/extract-atoms.ts +++ b/src/core/cycle/extract-atoms.ts @@ -49,6 +49,8 @@ import type { PhaseResult } from '../cycle.ts'; import type { GBrainConfig } from '../config.ts'; import type { ProgressReporter } from '../progress.ts'; import { chat as gatewayChat } from '../ai/gateway.ts'; +import { writeReceipt } from '../extract/receipt-writer.ts'; +import { upsertExtractRollup } from '../extract/rollup-writer.ts'; const DEFAULT_BUDGET_USD = 0.3; @@ -498,6 +500,38 @@ export async function runPhaseExtractAtoms( } } + // v0.42 Wave B2: write extract receipt + rollup row when the phase + // actually extracted atoms. Both are best-effort per F-OUT-19 — + // audit-trail / search-visibility surfaces don't block the phase result. + if (!opts.dryRun && totalAtomsExtracted > 0) { + const runId = `atoms-${Date.now().toString(36)}-${sourceId.slice(0, 4)}`; + try { + await writeReceipt(engine, { + kind: 'atoms', + source_id: sourceId, + run_id: runId, + round: 'single', + extracted_at: new Date().toISOString(), + total_rows: totalAtomsExtracted, + cost_usd: estimatedSpendUsd, + summary: + `Extracted ${totalAtomsExtracted} atoms from ` + + `${transcriptsProcessed} transcripts + ${pagesProcessed} pages.`, + }); + } catch (err) { + console.error(`[extract_atoms] receipt write failed: ${(err as Error).message}`); + } + } + if (!opts.dryRun) { + await upsertExtractRollup(engine, { + kind: 'atoms', + source_id: sourceId, + cost_delta: estimatedSpendUsd, + round_completed_delta: failures.length === 0 ? 1 : 0, + halt_delta: failures.length > 0 ? 1 : 0, + }); + } + return { phase: 'extract_atoms', status: failures.length > 0 ? 'warn' : 'ok', diff --git a/src/core/cycle/extract-facts.ts b/src/core/cycle/extract-facts.ts index 6af53b093..dda7fbd2f 100644 --- a/src/core/cycle/extract-facts.ts +++ b/src/core/cycle/extract-facts.ts @@ -32,6 +32,8 @@ */ import type { BrainEngine } from '../engine.ts'; +import { writeReceipt } from '../extract/receipt-writer.ts'; +import { upsertExtractRollup } from '../extract/rollup-writer.ts'; import { parseFactsFence } from '../facts-fence.ts'; import { extractFactsFromFenceText } from '../facts/extract-from-fence.ts'; import { @@ -253,5 +255,37 @@ export async function runExtractFacts( result.factsInserted += inserted.inserted; } + // v0.42 Wave B3: receipt + rollup. extract_facts is deterministic + // (fence reconcile, no LLM cost); receipt only when facts were + // actually inserted; rollup always fires. + if (!opts.dryRun && result.factsInserted > 0) { + const runId = `efacts-${Date.now().toString(36)}-${sourceId.slice(0, 4)}`; + try { + await writeReceipt(engine, { + kind: 'facts.fence', + source_id: sourceId, + run_id: runId, + round: 'single', + extracted_at: new Date().toISOString(), + total_rows: result.factsInserted, + cost_usd: 0, + summary: + `Reconciled ${result.factsInserted} facts (and deleted ${result.factsDeleted}) ` + + `across ${result.pagesScanned} scanned pages.`, + }); + } catch (err) { + console.error(`[extract_facts] receipt write failed: ${(err as Error).message}`); + } + } + if (!opts.dryRun) { + await upsertExtractRollup(engine, { + kind: 'facts.fence', + source_id: sourceId, + cost_delta: 0, + round_completed_delta: result.guardTriggered ? 0 : 1, + halt_delta: result.guardTriggered ? 1 : 0, + }); + } + return result; } diff --git a/src/core/cycle/propose-takes.ts b/src/core/cycle/propose-takes.ts index 904d13bc0..33fbe054c 100644 --- a/src/core/cycle/propose-takes.ts +++ b/src/core/cycle/propose-takes.ts @@ -40,6 +40,8 @@ import { randomUUID, createHash } from 'node:crypto'; import { BaseCyclePhase, type ScopedReadOpts, type BasePhaseOpts } from './base-phase.ts'; import { chat as gatewayChat } from '../ai/gateway.ts'; +import { writeReceipt } from '../extract/receipt-writer.ts'; +import { upsertExtractRollup } from '../extract/rollup-writer.ts'; import { GBrainError } from '../types.ts'; import type { Page, PageFilters } from '../types.ts'; import type { OperationContext } from '../operations.ts'; @@ -415,6 +417,34 @@ class ProposeTakesPhase extends BaseCyclePhase { if (opts.reporter) opts.reporter.finish(); + // v0.42 Wave B3: receipt + rollup for propose_takes. Source-scoped + // via the read scope. Receipt only when proposals actually written. + const sourceIdForReceipt = scope.sourceId ?? 'default'; + if (result.proposals_inserted > 0) { + try { + await writeReceipt(engine, { + kind: 'takes.proposed', + source_id: sourceIdForReceipt, + run_id: proposalRunId, + round: 'single', + extracted_at: new Date().toISOString(), + total_rows: result.proposals_inserted, + cost_usd: 0, // tracker isn't exposed at this layer; cost tracked centrally + summary: + `Proposed ${result.proposals_inserted} new takes from ${result.pages_scanned} pages ` + + `(${result.cache_hits} cached).`, + }); + } catch (err) { + console.error(`[propose_takes] receipt write failed: ${(err as Error).message}`); + } + } + await upsertExtractRollup(engine, { + kind: 'takes.proposed', + source_id: sourceIdForReceipt, + round_completed_delta: result.budget_exhausted ? 0 : 1, + halt_delta: result.budget_exhausted ? 1 : 0, + }); + return { summary: `propose_takes: scanned ${result.pages_scanned} pages, ${result.cache_hits} cached, ${result.proposals_inserted} new proposals (run ${proposalRunId})`, details: { ...result, proposal_run_id: proposalRunId, prompt_version: promptVersion }, diff --git a/src/core/cycle/synthesize-concepts.ts b/src/core/cycle/synthesize-concepts.ts index 890317979..dbc352b7b 100644 --- a/src/core/cycle/synthesize-concepts.ts +++ b/src/core/cycle/synthesize-concepts.ts @@ -21,6 +21,8 @@ import type { BrainEngine } from '../engine.ts'; import type { PhaseResult } from '../cycle.ts'; import type { ProgressReporter } from '../progress.ts'; +import { writeReceipt } from '../extract/receipt-writer.ts'; +import { upsertExtractRollup } from '../extract/rollup-writer.ts'; import { chat as gatewayChat } from '../ai/gateway.ts'; const DEFAULT_BUDGET_USD = 1.5; @@ -239,6 +241,40 @@ export async function runPhaseSynthesizeConcepts( await maybeYield(); } + // v0.42 Wave B3: receipt + rollup for synthesize_concepts. Brain-global + // phase — uses 'default' source_id because concepts span sources. Receipt + // only fires when concepts were actually written; rollup always fires so + // doctor sees the phase ran. + if (!opts.dryRun && conceptsWritten > 0) { + const runId = `concepts-${Date.now().toString(36)}`; + try { + await writeReceipt(engine, { + kind: 'concepts', + source_id: 'default', + run_id: runId, + round: 'single', + extracted_at: new Date().toISOString(), + total_rows: conceptsWritten, + cost_usd: estimatedSpendUsd, + summary: + `Synthesized ${conceptsWritten} concepts ` + + `(T1=${tierCounts.T1} T2=${tierCounts.T2} T3=${tierCounts.T3}) ` + + `from ${atomGroups.length} groups across ${atoms.length} atoms.`, + }); + } catch (err) { + console.error(`[synthesize_concepts] receipt write failed: ${(err as Error).message}`); + } + } + if (!opts.dryRun) { + await upsertExtractRollup(engine, { + kind: 'concepts', + source_id: 'default', + cost_delta: estimatedSpendUsd, + round_completed_delta: failures.length === 0 ? 1 : 0, + halt_delta: failures.length > 0 ? 1 : 0, + }); + } + return { phase: 'synthesize_concepts', status: failures.length > 0 ? 'warn' : 'ok', diff --git a/src/core/doctor-categories.ts b/src/core/doctor-categories.ts index 7c9c76ea1..a801120e4 100644 --- a/src/core/doctor-categories.ts +++ b/src/core/doctor-categories.ts @@ -73,6 +73,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet = new Set([ 'embedding_width_consistency', 'embeddings', 'eval_drift', + 'extract_health', 'facts_embedding_width_consistency', 'facts_extraction_health', 'facts_health', diff --git a/src/core/extract/receipt-writer.ts b/src/core/extract/receipt-writer.ts new file mode 100644 index 000000000..1334695ea --- /dev/null +++ b/src/core/extract/receipt-writer.ts @@ -0,0 +1,208 @@ +/** + * v0.42 — Extract Receipt writer. + * + * Every LLM-backed extraction run writes a receipt page recording the + * round's outcome: total rows, cost, model id, eval verdict. Receipts + * are first-class brain memory — queryable via gbrain search, citable + * in takes, surfaced in cross-modal contradiction probes. + * + * Two structural protections against extraction loops (D-EXTRACT-19, + * belt + suspenders): + * 1. `type: extract_receipt` — eligibility predicate's type filter + * rejects this type because it's not in ELIGIBLE_TYPES. + * 2. `dream_generated: true` — the anti-loop guard at + * `src/core/facts/eligibility.ts:62` rejects this flag. + * + * Search-rank: receipts get factor 0.3 demote via the `extracts/` slug + * prefix entry in DEFAULT_SOURCE_BOOSTS (D-EXTRACT-42). They surface + * when specifically relevant (extraction-related queries) but never + * dominate user content. + * + * Slug shape (D-EXTRACT-17): + * extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md + * + * Where: + * - date = YYYY-MM-DD of extraction + * - kind = extractor kind ("facts.conversation", "atoms", ...) + * - source_id = brain source the extraction targeted + * - run_id_short = first 8 chars of the op-checkpoint id (or + * progressive-batch operation_id) — groups all + * rounds of one run under one directory + * - N = round identifier ("trial" | "ramp_100" | "ramp_500" | + * "full" | "single") + * + * Resume rounds land under the same run_id_short so the audit trail + * for a halted-then-resumed run stays coherent. + */ + +import type { BrainEngine } from '../engine.ts'; +import type { Page } from '../types.ts'; + +/** + * Round identifier. Matches the progressive-batch primitive's Stage + * union from src/core/progressive-batch/types.ts:42, plus 'single' for + * extractors that don't ramp (deterministic / one-shot). + */ +export type ExtractReceiptRound = + | 'trial' + | 'ramp_100' + | 'ramp_500' + | 'full' + | 'single'; + +/** + * Input to writeReceipt. Optional fields are recorded in frontmatter + * only when the caller provides them, so receipts stay clean for + * deterministic extractors (no LLM cost or eval to record). + */ +export interface ExtractReceiptInput { + /** Kind of extraction (matches the conceptual handler name). */ + kind: string; + /** Brain source the extraction targeted. */ + source_id: string; + /** Op-checkpoint id or progressive-batch operation_id for trace continuity. */ + run_id: string; + /** Which round this receipt covers. */ + round: ExtractReceiptRound; + /** ISO timestamp of round completion. */ + extracted_at: string; + /** Rows committed to the target store this round. */ + total_rows: number; + /** Cumulative cost across the round in USD. 0 for deterministic extractors. */ + cost_usd: number; + /** LLM model id (optional; only for LLM-backed extractors). */ + model_id?: string; + /** Eval gate verdict (optional; only when a gate fired). */ + eval_pass?: boolean; + /** Eval gate score (optional; companion to eval_pass). */ + eval_score?: number; + /** Human-readable summary line (1-2 sentences). */ + summary?: string; +} + +const RUN_ID_SHORT_LEN = 8; + +/** + * Truncate a run id to the standard 8-char short form used in slug + * paths. Idempotent — passing an already-short id returns it unchanged. + * Non-hex / non-alphanumeric chars survive (op-checkpoint ids may + * include dashes or other separators). + */ +export function shortRunId(runId: string): string { + return runId.slice(0, RUN_ID_SHORT_LEN); +} + +/** + * Derive a YYYY-MM-DD date string from an ISO timestamp. + * Falls back to UTC slice; locale-independent. + */ +export function dateFromIso(iso: string): string { + // ISO 8601 always starts YYYY-MM-DD. Slice the first 10 chars. + // If the caller passes a non-ISO shape, slice still returns something; + // the receipt slug is best-effort labeling, not load-bearing parsing. + return iso.slice(0, 10); +} + +/** + * Compute the canonical slug for a receipt. Exported for tests + + * symmetry with read-side code that needs to compose the same slug + * (e.g. doctor reading receipts by run_id). + */ +export function receiptSlug(input: ExtractReceiptInput): string { + const date = dateFromIso(input.extracted_at); + const short = shortRunId(input.run_id); + return `extracts/${date}/${input.kind}/${input.source_id}/${short}/round-${input.round}`; +} + +/** + * Build the receipt page body. Operator-readable. Frontmatter is + * machine-readable. + */ +function buildReceiptBody(input: ExtractReceiptInput): string { + const lines: string[] = []; + lines.push(`# ${input.kind} — round ${input.round}`); + lines.push(''); + if (input.summary) { + lines.push(input.summary); + lines.push(''); + } + lines.push(`Source: \`${input.source_id}\``); + lines.push(`Run: \`${input.run_id}\``); + lines.push(`Round: \`${input.round}\``); + lines.push(`Extracted at: ${input.extracted_at}`); + lines.push(''); + lines.push(`Rows extracted: **${input.total_rows}**`); + if (typeof input.cost_usd === 'number' && input.cost_usd > 0) { + lines.push(`Cost: $${input.cost_usd.toFixed(4)}`); + } + if (input.model_id) { + lines.push(`Model: \`${input.model_id}\``); + } + if (typeof input.eval_pass === 'boolean') { + const verdict = input.eval_pass ? 'PASS' : 'FAIL'; + const score = typeof input.eval_score === 'number' + ? ` (score ${input.eval_score.toFixed(2)})` + : ''; + lines.push(`Eval gate: **${verdict}**${score}`); + } + return lines.join('\n') + '\n'; +} + +/** + * Build the receipt frontmatter. Two anti-loop flags + * (type:extract_receipt + dream_generated:true) are stamped by every + * writeReceipt call regardless of caller. Per D-EXTRACT-19. + */ +function buildReceiptFrontmatter(input: ExtractReceiptInput): Record { + const fm: Record = { + type: 'extract_receipt', + dream_generated: true, + kind: input.kind, + source_id: input.source_id, + run_id: input.run_id, + round: input.round, + extracted_at: input.extracted_at, + total_rows: input.total_rows, + cost_usd: input.cost_usd, + }; + if (input.model_id) fm.model_id = input.model_id; + if (typeof input.eval_pass === 'boolean') fm.eval_pass = input.eval_pass; + if (typeof input.eval_score === 'number') fm.eval_score = input.eval_score; + return fm; +} + +/** + * Write an extract receipt page. Returns the slug of the written page + * for the caller's audit/logging needs. + * + * Side-effects: calls engine.putPage with the receipt's compiled body + + * frontmatter. Threads sourceId so federated brains route the receipt + * to the same source the extraction targeted. + * + * Re-running with the same run_id + round overwrites the prior receipt + * (idempotent on resume). The op-checkpoint id is stable across + * resumes per src/core/op-checkpoint.ts, so this is the desired + * semantic. + */ +export async function writeReceipt( + engine: BrainEngine, + input: ExtractReceiptInput, +): Promise<{ slug: string; page: Page }> { + const slug = receiptSlug(input); + const title = `${input.kind} — ${input.round} — ${input.source_id}`; + const frontmatter = buildReceiptFrontmatter(input); + const compiled_truth = buildReceiptBody(input); + + const page = await engine.putPage( + slug, + { + type: 'extract_receipt', + title, + compiled_truth, + frontmatter, + }, + { sourceId: input.source_id }, + ); + + return { slug, page }; +} diff --git a/src/core/extract/rollup-writer.ts b/src/core/extract/rollup-writer.ts new file mode 100644 index 000000000..ca339a624 --- /dev/null +++ b/src/core/extract/rollup-writer.ts @@ -0,0 +1,128 @@ +/** + * v0.42 — Per-day extract-event rollup writer (best-effort cache). + * + * Sister module to receipt-writer.ts. Where receipts are first-class + * brain pages (long-term audit trail), the rollup table is a fast-read + * cache the doctor `extract_health` check reads to keep latency under + * 100ms regardless of audit volume. + * + * F-OUT-19 dual-write posture: + * - JSONL audit (~/.gbrain/audit/...) is the SOURCE OF TRUTH + * (forensic, append-only, crash-safe). + * - This DB rollup is BEST-EFFORT. Failures bump a counter + * (rollup_write_failures) inside the table itself and stderr-warn; + * they NEVER fail the parent extraction operation. + * - When persistent failures accumulate (>10/hr per future doctor + * rule), an auto-rebuild from JSONL self-heals the cache. + * + * Schema (migration v106): + * extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, + * eval_fail_count, eval_pass_count, + * round_completed_count, rollup_write_failures, + * updated_at, PK(kind, source_id, day)) + * + * Concurrency: PostgreSQL INSERT ... ON CONFLICT DO UPDATE is + * concurrency-safe for the per-(kind, source_id, day) PK. Multiple + * parallel extractions on the same day land cleanly without a lock + * (UPSERT is atomic per row). + */ + +import type { BrainEngine } from '../engine.ts'; + +/** + * One UPSERT increments per audit event. All counters default to 0 so + * callers only specify the deltas they care about (e.g. a round-completed + * event passes round_completed_delta=1; an eval-fail event passes + * eval_fail_delta=1; a halt event passes halt_delta=1). cost_delta is the + * cumulative cost ADD for the period this event represents. + */ +export interface RollupUpsertInput { + kind: string; + source_id: string; + /** ISO YYYY-MM-DD. Defaults to today (UTC) if omitted. */ + day?: string; + cost_delta?: number; + halt_delta?: number; + eval_fail_delta?: number; + eval_pass_delta?: number; + round_completed_delta?: number; + /** Increment rollup_write_failures inside the table (used by the + * self-healing path that records its own write failures forensically). */ + failure_delta?: number; +} + +function today(): string { + // ISO YYYY-MM-DD in UTC. Matches Postgres CURRENT_DATE behavior for + // UTC servers; for local-time servers there's a slight drift at midnight + // but rollup is best-effort cache so the drift is acceptable. + return new Date().toISOString().slice(0, 10); +} + +/** + * UPSERT one rollup event. Best-effort: catches all errors, logs to + * stderr once per (kind, day, error-class) so it doesn't spam, returns + * a boolean indicating success. + * + * Caller composes multiple deltas in one call (round_completed + cost + * together is typical) so the rollup row is one UPSERT per audit event, + * not N UPSERTs. + */ +export async function upsertExtractRollup( + engine: BrainEngine, + input: RollupUpsertInput, +): Promise<{ ok: boolean; error?: string }> { + const day = input.day ?? today(); + const cost = input.cost_delta ?? 0; + const halts = input.halt_delta ?? 0; + const evalFails = input.eval_fail_delta ?? 0; + const evalPasses = input.eval_pass_delta ?? 0; + const completed = input.round_completed_delta ?? 0; + const failures = input.failure_delta ?? 0; + + try { + await engine.executeRaw( + `INSERT INTO extract_rollup_7d ( + kind, source_id, day, + cost_usd, halt_count, eval_fail_count, eval_pass_count, + round_completed_count, rollup_write_failures, updated_at + ) + VALUES ($1, $2, $3::date, $4, $5, $6, $7, $8, $9, now()) + ON CONFLICT (kind, source_id, day) DO UPDATE SET + cost_usd = extract_rollup_7d.cost_usd + EXCLUDED.cost_usd, + halt_count = extract_rollup_7d.halt_count + EXCLUDED.halt_count, + eval_fail_count = extract_rollup_7d.eval_fail_count + EXCLUDED.eval_fail_count, + eval_pass_count = extract_rollup_7d.eval_pass_count + EXCLUDED.eval_pass_count, + round_completed_count = extract_rollup_7d.round_completed_count + EXCLUDED.round_completed_count, + rollup_write_failures = extract_rollup_7d.rollup_write_failures + EXCLUDED.rollup_write_failures, + updated_at = now()`, + [input.kind, input.source_id, day, cost, halts, evalFails, evalPasses, completed, failures], + ); + return { ok: true }; + } catch (err) { + const msg = (err as Error).message || String(err); + // Don't spam: log once per process per (kind, day) error class. + rollupErrorLogOnce(input.kind, day, msg); + return { ok: false, error: msg }; + } +} + +const _loggedRollupErrors = new Set(); +function rollupErrorLogOnce(kind: string, day: string, msg: string): void { + // Classify by error first 80 chars to dedupe "lock timeout" vs + // "connection refused" but not by exact text. + const klass = msg.slice(0, 80); + const key = `${kind}|${day}|${klass}`; + if (_loggedRollupErrors.has(key)) return; + _loggedRollupErrors.add(key); + console.error( + `[extract-rollup] write failed (best-effort; audit JSONL remains source of truth): ${msg}`, + ); +} + +/** + * Test seam: clear the logged-errors set so repeated test invocations + * surface stderr each time. + */ +export function _resetRollupErrorLogForTests(): void { + _loggedRollupErrors.clear(); +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index a918e18f3..2b7e17014 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4818,6 +4818,44 @@ export const MIGRATIONS: Migration[] = [ ); }, }, + { + version: 106, + name: 'extract_rollup_7d_table', + // v0.41.23 — Per-day rollup of extract events for fast doctor reads. + // Audit JSONL at ~/.gbrain/audit/extract-rounds-YYYY-Www.jsonl remains + // the SOURCE OF TRUTH (forensic, append-only, crash-safe). This DB + // table is a best-effort cache for doctor's <100ms read budget on + // heavy brains (per F-OUT-19 dual-write posture, JSONL primary). + // + // Per-day rows mean the 7-day window auto-evicts; doctor reads + // `WHERE day >= CURRENT_DATE - 7`. UPSERT on every audit event + // serializes via Postgres' INSERT ... ON CONFLICT DO UPDATE. + // + // Cycle's purge phase GCs rows older than 30 days (operational buffer + // beyond the 7-day read window). + // + // Slot history: originally claimed v100 in plan; bumped to v104 after + // v98/v99/v101/v102/v103 master merges; bumped again to v106 after + // v0.41.22 master merge took v104 (pages_atom_source_hash_idx) and + // v105 (slug_aliases). + sql: ` + CREATE TABLE IF NOT EXISTS extract_rollup_7d ( + kind TEXT NOT NULL, + source_id TEXT NOT NULL, + day DATE NOT NULL, + cost_usd REAL NOT NULL DEFAULT 0, + halt_count INT NOT NULL DEFAULT 0, + eval_fail_count INT NOT NULL DEFAULT 0, + eval_pass_count INT NOT NULL DEFAULT 0, + round_completed_count INT NOT NULL DEFAULT 0, + rollup_write_failures INT NOT NULL DEFAULT 0, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (kind, source_id, day) + ); + CREATE INDEX IF NOT EXISTS idx_extract_rollup_7d_day + ON extract_rollup_7d (day); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/schema-pack/base/gbrain-base.yaml b/src/core/schema-pack/base/gbrain-base.yaml index dd871b0fc..0765e0785 100644 --- a/src/core/schema-pack/base/gbrain-base.yaml +++ b/src/core/schema-pack/base/gbrain-base.yaml @@ -284,6 +284,20 @@ page_types: extractable: false expert_routing: false + # Receipt page written by the v0.41.23 extract framework after every + # extractor run. Demoted in search via the 0.3x source-boost on the + # `extracts/` prefix (see src/core/search/source-boost.ts). Always + # carries `dream_generated: true` + `type: extract_receipt` in + # frontmatter so the eligibility predicate's anti-loop guard can + # reject it via either check (belt + suspenders per D-EXTRACT-19). + - name: extract_receipt + primitive: annotation + path_prefixes: + - extracts/ + aliases: [] + extractable: false + expert_routing: false + # Link verbs — ordered to match inferLinkType priority in link-extraction.ts. # Each entry carries the regex pattern that triggers the verb. Inferences # without a pattern (e.g. type-bound: meeting → attended) are encoded via diff --git a/src/core/schema-pack/extractable.ts b/src/core/schema-pack/extractable.ts index 232885114..7950bf2d5 100644 --- a/src/core/schema-pack/extractable.ts +++ b/src/core/schema-pack/extractable.ts @@ -22,19 +22,33 @@ // to use the hardcoded ELIGIBLE_TYPES list — which gbrain-base also // declares, so behavior is preserved. -import type { SchemaPackManifest } from './manifest-v1.ts'; +import type { SchemaPackManifest, ExtractableSpec } from './manifest-v1.ts'; /** - * Return the Set of pack-declared types with `extractable: true`. - * Set return shape (vs array) because callers want O(1) membership - * checks in the eligibility predicate, which fires on every put_page. + * v0.42 widening: `extractable` may be `true | false | ExtractableSpec`. + * Both `true` and a non-empty struct mean "this type is extractable." + * + * Pure predicate; reusable across all read sites. + */ +function isExtractable(extractable: boolean | ExtractableSpec): boolean { + if (typeof extractable === 'boolean') return extractable; + // Struct shape implies extractable = true (pack author wouldn't declare + // prompt_template + fixtures + eval_dimensions for a non-extractable type). + return true; +} + +/** + * Return the Set of pack-declared types with `extractable: true` OR + * `extractable: `. Set return shape (vs array) because callers + * want O(1) membership checks in the eligibility predicate, which fires + * on every put_page. */ export function extractableTypesFromPack( pack: Pick, ): Set { return new Set( pack.page_types - .filter(pt => pt.extractable === true) + .filter(pt => isExtractable(pt.extractable)) .map(pt => pt.name), ); } @@ -48,5 +62,67 @@ export function isExtractableType( pack: Pick, type: string, ): boolean { - return pack.page_types.some(pt => pt.name === type && pt.extractable === true); + return pack.page_types.some(pt => pt.name === type && isExtractable(pt.extractable)); +} + +/** + * v0.42 — Return a Map of type-name → ExtractableSpec for every + * extractable type. Boolean `true` resolves to the default empty struct + * (no prompt template, no fixtures, no eval dimensions). Pack authors who + * want pack-supplied prompts + fixtures opt into the struct shape per + * `ExtractableSpec`. + * + * Consumed by `gbrain extract benchmark`, `gbrain extract --explain`, + * and `gbrain schema scaffold-extractable`. + */ +export function extractableSpecsFromPack( + pack: Pick, +): Map { + const out = new Map(); + for (const pt of pack.page_types) { + if (!isExtractable(pt.extractable)) continue; + if (typeof pt.extractable === 'boolean') { + // boolean true → empty default spec (back-compat) + out.set(pt.name, { eval_dimensions: [] }); + } else { + out.set(pt.name, pt.extractable); + } + } + return out; +} + +/** + * v0.42 — Return the ExtractableSpec for a single type, or null if not + * extractable. Convenience for read sites that only have one type name. + */ +export function getExtractableSpec( + pack: Pick, + type: string, +): ExtractableSpec | null { + const pt = pack.page_types.find(p => p.name === type); + if (!pt || !isExtractable(pt.extractable)) return null; + if (typeof pt.extractable === 'boolean') return { eval_dimensions: [] }; + return pt.extractable; +} + +/** + * v0.42 — Forward-compat runtime gate for D-EXTRACT-37. A pack-supplied + * `verifier_path` field is RESERVED in v0.42 — accepted at parse time, + * REFUSED at runtime. Call this anywhere a runtime would attempt to load + * pack-shipped verifier code. + * + * @throws Error with paste-ready message if verifier_path is set + */ +export function refuseVerifierPathInV042( + spec: Pick, + typeName: string, +): void { + if (spec.verifier_path) { + throw new Error( + `pack-shipped verifier code is not supported in v0.42 (type: ${typeName}, ` + + `verifier_path: ${spec.verifier_path}). v0.43+ trust review is the gate ` + + `for loading pack-shipped verifier scripts. Remove verifier_path from your ` + + `pack manifest OR wait for v0.43 to land.` + ); + } } diff --git a/src/core/schema-pack/index.ts b/src/core/schema-pack/index.ts index 9ab915f1d..585ff13f9 100644 --- a/src/core/schema-pack/index.ts +++ b/src/core/schema-pack/index.ts @@ -11,6 +11,7 @@ export { type SchemaPackManifest, type PackPageType, type PackLinkType, + type ExtractableSpec, SchemaPackManifestSchema, SchemaPackManifestError, parseSchemaPackManifest, @@ -114,7 +115,10 @@ export { export { extractableTypesFromPack, + extractableSpecsFromPack, + getExtractableSpec, isExtractableType, + refuseVerifierPathInV042, } from './extractable.ts'; export { diff --git a/src/core/schema-pack/manifest-v1.ts b/src/core/schema-pack/manifest-v1.ts index 5b45dcccf..f27526105 100644 --- a/src/core/schema-pack/manifest-v1.ts +++ b/src/core/schema-pack/manifest-v1.ts @@ -43,7 +43,45 @@ const LinkTypeSchema = z.object({ }).strict(); /** - * v0.42 (T3, plan D5): per-page-type subtype-detection rule. The rule fires + * v0.41.23 — ExtractableSpec widening. `extractable` is now `boolean | struct`. + * + * v0.38 shape (`extractable: true`) stays valid forever; resolves to a + * minimal default spec with empty fields. Pack authors opt into the struct + * shape when they want pack-supplied prompts / fixtures / eval dimensions + * for an LLM-backed extractor running over their page type. + * + * Forward-compat: `verifier_path` is RESERVED in v0.41.23. The parser accepts + * the field (validated as relative path within pack root by future logic) + * but the runtime REFUSES to load pack-shipped verifier code in v0.41.23 — + * a follow-up release trust review is the gate. Pack authors who write the + * path early get a clear runtime refuse message; they're not blocked at + * parse time. + * + * See plan D-EXTRACT-17/19/21/37/42/47 for the load-bearing decisions. + */ +const ExtractableSpecSchema = z.object({ + /** Pack-supplied LLM prompt template. Plain text; sent to gateway.chat() + * with NO conversation context per the v0.41.23 threat model. */ + prompt_template: z.string().optional(), + /** Relative path within pack root to a JSONL fixture corpus. Validated + * against path traversal at parse + load time. */ + fixture_corpus: z.string().optional(), + /** Per-kind eval dimensions for the cross-modal eval gate. Open string + * array; specific values consumed by `gbrain extract benchmark`. */ + eval_dimensions: z.array(z.string()).default([]), + /** Optional recall floor for `gbrain extract benchmark` CI gate. + * Defaults to 0.8 at consume site when omitted. */ + benchmark_min_recall: z.number().min(0).max(1).optional(), + /** RESERVED for a follow-up release: relative path to pack-shipped verifier + * code. Validated as relative + within-pack at parse; REFUSES at runtime + * in v0.41.23 with paste-ready hint. */ + verifier_path: z.string().optional(), +}).strict(); + +export type ExtractableSpec = z.infer; + +/** + * v0.41.22 (T3, plan D5): per-page-type subtype-detection rule. The rule fires * when (a) frontmatter has a matching key+value, OR (b) the source path * matches the regex. ReDoS-guarded compile happens at pack-load (registry). */ @@ -75,10 +113,18 @@ const PageTypeSchema = z.object({ */ aliases: z.array(z.string()).default([]), /** - * Whether the page-type is eligible for facts extraction (gates - * `src/core/facts/eligibility.ts:49` per T3 codex finding). + * Whether the page-type is eligible for facts extraction. + * + * - `boolean` (v0.38 shape): true = extractable with default LLM handler; + * false = not extractable. Gates `src/core/facts/eligibility.ts:49`. + * - `ExtractableSpec` (v0.42 widening): pack-supplied prompt + fixtures + * + eval dimensions for the pack-author authoring loop. Implies true + * for eligibility purposes. + * + * Defaults to false. Back-compat: every pre-v0.42 pack manifest with + * `extractable: true` continues to parse unchanged. */ - extractable: z.boolean().default(false), + extractable: z.union([z.boolean(), ExtractableSpecSchema]).default(false), /** * Whether this type is an "expert" for find_experts / whoknows queries * (replaces hardcoded ['person','company'] at whoknows.ts:89 + the diff --git a/src/core/schema-pack/scaffold-extractable.ts b/src/core/schema-pack/scaffold-extractable.ts new file mode 100644 index 000000000..048275224 --- /dev/null +++ b/src/core/schema-pack/scaffold-extractable.ts @@ -0,0 +1,295 @@ +/** + * v0.42 Wave C1 — `gbrain schema scaffold-extractable` mutation primitive. + * + * Generates everything a pack author needs to declare a new extractable + * page type with the pack-supplied prompt + fixture-corpus + eval-dimensions + * authoring loop: + * + * 1. Updates the pack's type to carry the v0.42 ExtractableSpec struct + * shape (prompt_template path + fixture_corpus path + eval_dimensions + * stub array). Reuses the proven `updateTypeOnPack` mutation skeleton + * so we inherit atomic write + per-pack lock + audit + cache + * invalidation for free. + * + * 2. Writes `/prompts/extract/.md` with a recommended + * prompt template the pack author can edit. Falls back to a generic + * facts-style prompt; pack-author customizes. + * + * 3. Writes `/fixtures/extract/.jsonl` with 5 placeholder + * fixtures (per CLAUDE.md privacy rule: alice-example, widget-co-example, + * etc.) so the pack-author can run `gbrain extract benchmark` immediately + * and iterate from a working baseline rather than an empty file. + * + * Refuses to overwrite existing prompt / fixture files — the mutation is + * additive. Re-running with `--force` overwrites only the files (the YAML + * mutation is naturally idempotent for the struct shape). + * + * Per plan D-EXTRACT-21: fixture path validation IS happening at the + * `extract benchmark` consume site (canonicalize within pack root, reject + * absolute paths / `..` / null bytes). This scaffolder generates only + * relative paths, so it's compliant by construction. + */ + +import { existsSync, mkdirSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; + +import type { MutateResult } from './mutate.ts'; +import { + locateMutablePackFile, + updateTypeOnPack, + SchemaPackMutationError, +} from './mutate.ts'; +import type { ExtractableSpec } from './manifest-v1.ts'; + +export interface ScaffoldExtractableOpts { + /** Pack name (must be a writable pack — bundled packs are guarded). */ + packName: string; + /** Page type to declare extractable on. */ + typeName: string; + /** + * v0.42 ExtractableSpec fields: + * - eval_dimensions: defaults to ['faithfulness', 'completeness'] when + * omitted — the two dimensions every LLM-backed extractor benefits + * from. Pack-author edits the manifest if their domain has a richer + * scoring rubric. + */ + evalDimensions?: string[]; + /** Overwrite existing prompt + fixture files. Default false. */ + force?: boolean; +} + +export interface ScaffoldExtractableResult { + /** MutateResult from the underlying updateTypeOnPack call. */ + mutate: MutateResult; + /** Absolute paths to files written this run. */ + filesWritten: string[]; + /** Paths skipped because they already existed and --force not set. */ + filesSkipped: string[]; +} + +/** + * Pure helper: build the per-type prompt template body. Pack-authors + * edit this after scaffold to specify their domain. + */ +export function buildPromptTemplate(typeName: string): string { + return `# Extraction prompt for type \`${typeName}\` + +You will be given the full body of a \`${typeName}\` page. Extract every +factual claim the page makes about its primary subject. Return STRICTLY a +JSON array of claim objects. + +## Shape + +Each claim is an object with these fields (all required unless marked +optional): + +\`\`\`json +{ + "claim": "...", + "since_date": "YYYY-MM-DD", + "confidence": 0.0-1.0, + "evidence_quote": "..." +} +\`\`\` + +## Rules + +- Extract only claims supported by direct evidence in the page body. +- Each \`evidence_quote\` MUST be a verbatim substring of the page body. +- \`since_date\` is the date the claim became true (or the page's effective + date if no specific date is given). +- Confidence 1.0 = explicitly stated; 0.7 = strongly implied; below 0.6 + should usually be omitted. +- Return \`[]\` if the page contains no extractable claims. +- Do NOT speculate beyond the page body. + +## Examples + +(Placeholder examples — replace with real ones from your domain before +shipping this pack.) + +Input page: +> alice-example founded widget-co-example in 2020 and raised a $5M seed +> in 2021. + +Expected output: +\`\`\`json +[ + { + "claim": "alice-example founded widget-co-example", + "since_date": "2020-01-01", + "confidence": 1.0, + "evidence_quote": "alice-example founded widget-co-example in 2020" + }, + { + "claim": "widget-co-example raised a $5M seed", + "since_date": "2021-01-01", + "confidence": 1.0, + "evidence_quote": "raised a $5M seed in 2021" + } +] +\`\`\` +`; +} + +/** + * Pure helper: build the fixture corpus body (5 placeholder fixtures). + * Each line is one JSON-serialized fixture per the `gbrain extract + * benchmark` JSONL contract: + * + * { fixture_id, page_body, expected_claims: [...], notes? } + * + * The 5 stubs cover the spread of inputs every pack-author needs to + * handle: pure-claim, no-claim, ambiguous, edge-case, multi-claim. + * + * Per CLAUDE.md privacy rule: all placeholder names only. + */ +export function buildFixtureCorpus(typeName: string): string { + const fixtures = [ + { + fixture_id: `${typeName}-001-single-claim`, + page_body: + 'alice-example founded widget-co-example in 2020.', + expected_claims: [ + { + claim: 'alice-example founded widget-co-example', + since_date: '2020-01-01', + confidence: 1.0, + }, + ], + notes: 'Baseline: one explicit factual claim.', + }, + { + fixture_id: `${typeName}-002-no-claim`, + page_body: + 'This page is mostly placeholder text and contains no extractable claims.', + expected_claims: [], + notes: 'Negative case: extractor should return [].', + }, + { + fixture_id: `${typeName}-003-multi-claim`, + page_body: + 'widget-co-example raised a $5M seed in 2021 and grew to 12 employees by mid-2022.', + expected_claims: [ + { + claim: 'widget-co-example raised a $5M seed', + since_date: '2021-01-01', + confidence: 1.0, + }, + { + claim: 'widget-co-example grew to 12 employees', + since_date: '2022-06-01', + confidence: 0.9, + }, + ], + notes: 'Two claims in one sentence.', + }, + { + fixture_id: `${typeName}-004-ambiguous`, + page_body: + 'fund-a may have led the seed round, though sources are inconsistent.', + expected_claims: [ + { + claim: 'fund-a may have led the seed round', + since_date: '2021-01-01', + confidence: 0.6, + }, + ], + notes: 'Confidence drops on hedged language.', + }, + { + fixture_id: `${typeName}-005-implicit-date`, + page_body: + 'charlie-example was the second employee at acme-example.', + expected_claims: [ + { + claim: 'charlie-example was the second employee at acme-example', + since_date: null, + confidence: 1.0, + }, + ], + notes: 'No date in body — extractor falls back to page effective_date or null.', + }, + ]; + + return fixtures.map((f) => JSON.stringify(f)).join('\n') + '\n'; +} + +/** + * Pure helper: build the v0.42 ExtractableSpec struct that gets written + * to the pack manifest's `extractable` field on the target type. + */ +export function buildExtractableSpec(opts: { + typeName: string; + evalDimensions?: string[]; +}): ExtractableSpec { + return { + prompt_template: `prompts/extract/${opts.typeName}.md`, + fixture_corpus: `fixtures/extract/${opts.typeName}.jsonl`, + eval_dimensions: opts.evalDimensions ?? ['faithfulness', 'completeness'], + }; +} + +/** + * Main entry: scaffold the extractable + the supporting files. + * + * Returns the MutateResult from the YAML mutation plus a list of disk + * files written or skipped (idempotent re-run reports the skipped files + * so the operator sees the no-op clearly). + * + * Throws SchemaPackMutationError for any pack-layer failure (bundled + * pack, missing pack, parse error). File-write failures throw raw Errors + * with a paste-ready remediation hint. + */ +export async function scaffoldExtractable( + opts: ScaffoldExtractableOpts, +): Promise { + // Resolves the pack-root dir + asserts the pack is writable (bundled + // packs throw PACK_READONLY). + const located = locateMutablePackFile(opts.packName); + const packRoot = dirname(located.path); + + // Build paths first so we can report them in the result even when the + // YAML mutation runs first. + const promptPath = join(packRoot, 'prompts', 'extract', `${opts.typeName}.md`); + const fixturePath = join(packRoot, 'fixtures', 'extract', `${opts.typeName}.jsonl`); + + // YAML mutation first — failing here means nothing else changes. + // updateTypeOnPack's patch shape accepts the struct directly. + const spec = buildExtractableSpec({ + typeName: opts.typeName, + evalDimensions: opts.evalDimensions, + }); + const mutate = await updateTypeOnPack(opts.packName, { + name: opts.typeName, + patch: { extractable: spec }, + }); + + // File writes — idempotent unless --force. Refuse-to-overwrite path + // is the canonical gbrain scaffold posture (matches schema init, + // skillpack scaffold, etc). + const filesWritten: string[] = []; + const filesSkipped: string[] = []; + + mkdirSync(dirname(promptPath), { recursive: true }); + if (!existsSync(promptPath) || opts.force) { + writeFileSync(promptPath, buildPromptTemplate(opts.typeName)); + filesWritten.push(promptPath); + } else { + filesSkipped.push(promptPath); + } + + mkdirSync(dirname(fixturePath), { recursive: true }); + if (!existsSync(fixturePath) || opts.force) { + writeFileSync(fixturePath, buildFixtureCorpus(opts.typeName)); + filesWritten.push(fixturePath); + } else { + filesSkipped.push(fixturePath); + } + + return { mutate, filesWritten, filesSkipped }; +} + +// Re-export so consumers don't need to thread mutate.ts directly when +// catching the failure class. +export { SchemaPackMutationError }; diff --git a/src/core/search/source-boost.ts b/src/core/search/source-boost.ts index 321cb8b8a..33f4a050e 100644 --- a/src/core/search/source-boost.ts +++ b/src/core/search/source-boost.ts @@ -37,6 +37,12 @@ export const DEFAULT_SOURCE_BOOSTS: Record = { 'media/x/': 0.7, // Chat transcripts — massive, noisy, swamp keyword queries 'openclaw/chat/': 0.5, + // v0.42 extract_receipt pages — surface when relevant (extraction + // questions, audit trail) but never dominate user content. Per plan + // D-EXTRACT-42. Receipts stamp `type: extract_receipt` AND + // `dream_generated: true` in their frontmatter; demote here keeps them + // findable but ranked below all curated user content. + 'extracts/': 0.3, }; /** diff --git a/src/core/types.ts b/src/core/types.ts index 635d343c6..9575f66e7 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -52,6 +52,12 @@ export const ALL_PAGE_TYPES: readonly string[] = [ // src/core/schema-pack/base/gbrain-base.yaml. 'conversation', 'atom', 'code', 'image', 'synthesis', + // v0.42 — `extract_receipt` pages record extraction-run outcomes as + // first-class brain memory. Slug-prefix `extracts/`. Demoted in search + // via DEFAULT_SOURCE_BOOSTS (factor 0.3) and excluded from extraction + // loops via the dream_generated:true + type:extract_receipt belt-and- + // suspenders pattern per plan D-EXTRACT-19. + 'extract_receipt', ] as const; /** diff --git a/test/doctor-extract-health.test.ts b/test/doctor-extract-health.test.ts new file mode 100644 index 000000000..ebd61aa70 --- /dev/null +++ b/test/doctor-extract-health.test.ts @@ -0,0 +1,150 @@ +// v0.42 — Doctor extract_health check unit tests. +// +// Pins: +// - Empty rollup → OK with kinds: [] +// - Per-kind halt rate > 10% → WARN with top-3 kinds in message +// - rollup_write_failures > 0 → WARN (when halt rates are clean) +// - Pre-v106 brain (no extract_rollup_7d table) → OK (best-effort) +// - JSON envelope stamps schema_version: 1 +// - last_updated_at coerces to ISO string regardless of engine + +import { describe, expect, test, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { computeExtractHealthCheck } from '../src/commands/doctor.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +async function clearRollup() { + await engine.executeRaw('DELETE FROM extract_rollup_7d', []); +} + +describe('computeExtractHealthCheck — empty + happy paths', () => { + test('empty rollup returns OK with empty kinds array', async () => { + await clearRollup(); + const check = await computeExtractHealthCheck(engine); + expect(check.name).toBe('extract_health'); + expect(check.status).toBe('ok'); + expect(check.message).toBe('no extractions in last 7 days'); + expect((check.details as any)?.schema_version).toBe(1); + expect((check.details as any)?.kinds).toEqual([]); + }); + + test('healthy rollup (zero halts) returns OK with kind aggregates', async () => { + await clearRollup(); + await engine.executeRaw( + `INSERT INTO extract_rollup_7d (kind, source_id, day, cost_usd, eval_pass_count, eval_fail_count, halt_count, round_completed_count, rollup_write_failures, updated_at) + VALUES ('facts.conversation', 'default', CURRENT_DATE, 1.23, 5, 0, 0, 10, 0, NOW())`, + [], + ); + const check = await computeExtractHealthCheck(engine); + expect(check.status).toBe('ok'); + expect(check.message).toContain('1 kind(s) tracked'); + expect((check.details as any)?.kinds).toHaveLength(1); + expect((check.details as any)?.kinds[0].kind).toBe('facts.conversation'); + expect((check.details as any)?.kinds[0].cost_7d_usd).toBeCloseTo(1.23, 4); + expect((check.details as any)?.kinds[0].halt_rate).toBe(0); + }); +}); + +describe('computeExtractHealthCheck — WARN paths', () => { + test('halt rate > 10% on one kind returns WARN with top-3 in message', async () => { + await clearRollup(); + // facts.conversation: 5 halts, 5 completed = 50% halt rate (WARN) + await engine.executeRaw( + `INSERT INTO extract_rollup_7d (kind, source_id, day, cost_usd, eval_pass_count, eval_fail_count, halt_count, round_completed_count, rollup_write_failures, updated_at) + VALUES ('facts.conversation', 'default', CURRENT_DATE, 0.50, 5, 0, 5, 5, 0, NOW())`, + [], + ); + const check = await computeExtractHealthCheck(engine); + expect(check.status).toBe('warn'); + expect(check.message).toContain('halt rate'); + expect(check.message).toContain('facts.conversation'); + expect(check.message).toContain('50'); + expect((check.details as any)?.kinds[0].halt_rate).toBe(0.5); + }); + + test('multiple kinds with high halt rate: top-3 listed in message', async () => { + await clearRollup(); + await engine.executeRaw( + `INSERT INTO extract_rollup_7d (kind, source_id, day, cost_usd, eval_pass_count, eval_fail_count, halt_count, round_completed_count, rollup_write_failures, updated_at) + VALUES + ('atoms', 'default', CURRENT_DATE, 0.10, 0, 0, 3, 7, 0, NOW()), + ('facts.conversation', 'default', CURRENT_DATE, 0.40, 0, 0, 5, 5, 0, NOW()), + ('concepts', 'default', CURRENT_DATE, 0.05, 0, 0, 2, 8, 0, NOW())`, + [], + ); + const check = await computeExtractHealthCheck(engine); + expect(check.status).toBe('warn'); + // Top-3 by halt rate: facts.conversation (50%), atoms (30%), concepts (20%) + expect(check.message).toContain('facts.conversation'); + expect(check.message).toContain('atoms'); + expect(check.message).toContain('concepts'); + }); + + test('rollup_write_failures > 0 with clean halt rates returns WARN', async () => { + await clearRollup(); + await engine.executeRaw( + `INSERT INTO extract_rollup_7d (kind, source_id, day, cost_usd, eval_pass_count, eval_fail_count, halt_count, round_completed_count, rollup_write_failures, updated_at) + VALUES ('atoms', 'default', CURRENT_DATE, 0.20, 5, 0, 0, 10, 3, NOW())`, + [], + ); + const check = await computeExtractHealthCheck(engine); + expect(check.status).toBe('warn'); + expect(check.message).toContain('rollup write failure'); + expect((check.details as any)?.rollup_write_failures_7d).toBe(3); + }); + + test('high halt rate precedes rollup write failure warn message', async () => { + // High halt rate is the more critical signal; should win the message. + await clearRollup(); + await engine.executeRaw( + `INSERT INTO extract_rollup_7d (kind, source_id, day, cost_usd, eval_pass_count, eval_fail_count, halt_count, round_completed_count, rollup_write_failures, updated_at) + VALUES ('atoms', 'default', CURRENT_DATE, 0.20, 0, 0, 5, 5, 3, NOW())`, + [], + ); + const check = await computeExtractHealthCheck(engine); + expect(check.status).toBe('warn'); + // halt rate WARN takes precedence over rollup WARN + expect(check.message).toContain('halt rate'); + // rollup failures still in details for forensic recovery + expect((check.details as any)?.rollup_write_failures_7d).toBe(3); + }); +}); + +describe('computeExtractHealthCheck — 7-day window', () => { + test('rows older than 7 days are excluded from aggregation', async () => { + await clearRollup(); + await engine.executeRaw( + `INSERT INTO extract_rollup_7d (kind, source_id, day, cost_usd, eval_pass_count, eval_fail_count, halt_count, round_completed_count, rollup_write_failures, updated_at) + VALUES + ('atoms', 'default', CURRENT_DATE - 30, 100.0, 0, 0, 100, 100, 0, NOW() - INTERVAL '30 days')`, + [], + ); + const check = await computeExtractHealthCheck(engine); + // Old row outside 7-day window → empty result + expect(check.status).toBe('ok'); + expect(check.message).toBe('no extractions in last 7 days'); + }); + + test('rows exactly at day = CURRENT_DATE - 7 ARE included', async () => { + await clearRollup(); + await engine.executeRaw( + `INSERT INTO extract_rollup_7d (kind, source_id, day, cost_usd, eval_pass_count, eval_fail_count, halt_count, round_completed_count, rollup_write_failures, updated_at) + VALUES ('atoms', 'default', CURRENT_DATE - 7, 0.50, 5, 0, 0, 10, 0, NOW())`, + [], + ); + const check = await computeExtractHealthCheck(engine); + expect(check.status).toBe('ok'); + expect((check.details as any)?.kinds).toHaveLength(1); + }); +}); diff --git a/test/extract/benchmark.test.ts b/test/extract/benchmark.test.ts new file mode 100644 index 000000000..90a6dded7 --- /dev/null +++ b/test/extract/benchmark.test.ts @@ -0,0 +1,194 @@ +// v0.42 Wave C2 + C3 — extract benchmark + fixture path validation tests. +// +// Pins: +// - D-EXTRACT-21 fixture path validation: relative only, no .., +// no null bytes, must stay within pack root after canonicalization, +// symlinks rejected +// - JSONL parse failures fail loud at exact line number +// - Required fixture fields (fixture_id, page_body, expected_claims) +// enforced +// - v0.42 stub-reporting status: pure shape report, no LLM dispatch yet + +import { describe, expect, test } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, symlinkSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + validateFixturePath, + parseBenchmarkFixtures, +} from '../../src/commands/extract-benchmark.ts'; + +describe('validateFixturePath — D-EXTRACT-21 strict validation', () => { + let packRoot: string; + let cleanup: () => void; + + function setup() { + packRoot = mkdtempSync(join(tmpdir(), 'extract-bench-pack-')); + cleanup = () => rmSync(packRoot, { recursive: true, force: true }); + } + + test('relative path within pack root → returns canonical absolute path', () => { + setup(); + try { + const abs = validateFixturePath(packRoot, 'fixtures/extract/claim.jsonl'); + expect(abs).toBe(join(packRoot, 'fixtures/extract/claim.jsonl')); + } finally { + cleanup(); + } + }); + + test('absolute path → REJECTED', () => { + setup(); + try { + expect(() => validateFixturePath(packRoot, '/etc/passwd')).toThrow(/RELATIVE/); + } finally { + cleanup(); + } + }); + + test('`..` traversal → REJECTED', () => { + setup(); + try { + expect(() => validateFixturePath(packRoot, '../outside.jsonl')).toThrow(/'\.\.'\s*segment/); + expect(() => validateFixturePath(packRoot, 'fixtures/../../escape.jsonl')).toThrow(/'\.\.'\s*segment/); + } finally { + cleanup(); + } + }); + + test('null byte → REJECTED', () => { + setup(); + try { + expect(() => validateFixturePath(packRoot, 'fixtures/extract/bad\0name.jsonl')).toThrow(/null byte/); + } finally { + cleanup(); + } + }); + + test('symlink pointing inside pack root → REJECTED', () => { + setup(); + try { + // Create a real file + a symlink to it inside the pack + mkdirSync(join(packRoot, 'fixtures/extract'), { recursive: true }); + const real = join(packRoot, 'fixtures/extract/real.jsonl'); + writeFileSync(real, '{}\n'); + const link = join(packRoot, 'fixtures/extract/link.jsonl'); + symlinkSync(real, link); + expect(() => validateFixturePath(packRoot, 'fixtures/extract/link.jsonl')).toThrow(/symbolic link/); + } finally { + cleanup(); + } + }); + + test('symlink pointing outside pack root → REJECTED at lstat check', () => { + setup(); + try { + // Create a real file OUTSIDE the pack root, symlink from inside + const outsideDir = mkdtempSync(join(tmpdir(), 'extract-bench-outside-')); + try { + const outside = join(outsideDir, 'escape.jsonl'); + writeFileSync(outside, '{}\n'); + mkdirSync(join(packRoot, 'fixtures/extract'), { recursive: true }); + const link = join(packRoot, 'fixtures/extract/escape.jsonl'); + symlinkSync(outside, link); + // lstat catches the symlink directly — the real-path check is + // belt + suspenders for cases where an INTERMEDIATE dir is a + // symlink (lstat on a regular file inside a symlinked dir + // returns the regular-file stat). + expect(() => validateFixturePath(packRoot, 'fixtures/extract/escape.jsonl')).toThrow(/symbolic link/); + } finally { + rmSync(outsideDir, { recursive: true, force: true }); + } + } finally { + cleanup(); + } + }); + + test('non-existent path → does NOT throw on lstat (file may not exist yet)', () => { + setup(); + try { + // We validate the path SHAPE even when the file doesn't exist; + // the benchmark dispatch handles existence separately so the + // user gets the scaffold-extractable hint, not a path error. + const abs = validateFixturePath(packRoot, 'fixtures/extract/new.jsonl'); + expect(abs).toBe(join(packRoot, 'fixtures/extract/new.jsonl')); + } finally { + cleanup(); + } + }); +}); + +describe('parseBenchmarkFixtures — JSONL contract enforcement', () => { + test('parses canonical 5-fixture corpus (matching scaffold output)', () => { + const jsonl = [ + JSON.stringify({ fixture_id: 'a-001', page_body: 'hi', expected_claims: [] }), + JSON.stringify({ fixture_id: 'a-002', page_body: 'multi', expected_claims: [{ claim: 'x' }, { claim: 'y' }] }), + ].join('\n') + '\n'; + const fixtures = parseBenchmarkFixtures(jsonl); + expect(fixtures).toHaveLength(2); + expect(fixtures[0].fixture_id).toBe('a-001'); + expect(fixtures[1].expected_claims).toHaveLength(2); + }); + + test('skips blank lines without failing', () => { + const jsonl = [ + JSON.stringify({ fixture_id: 'a-001', page_body: 'x', expected_claims: [] }), + '', + '', + JSON.stringify({ fixture_id: 'a-002', page_body: 'y', expected_claims: [] }), + '', + ].join('\n'); + const fixtures = parseBenchmarkFixtures(jsonl); + expect(fixtures).toHaveLength(2); + }); + + test('fails loud on malformed JSON at exact line number', () => { + const jsonl = [ + JSON.stringify({ fixture_id: 'good', page_body: 'x', expected_claims: [] }), + 'this is not json', + JSON.stringify({ fixture_id: 'never-reached', page_body: 'x', expected_claims: [] }), + ].join('\n'); + expect(() => parseBenchmarkFixtures(jsonl)).toThrow(/line 2/); + }); + + test('rejects missing required field (fixture_id)', () => { + const jsonl = JSON.stringify({ page_body: 'x', expected_claims: [] }); + expect(() => parseBenchmarkFixtures(jsonl)).toThrow(/fixture_id/); + }); + + test('rejects missing required field (page_body)', () => { + const jsonl = JSON.stringify({ fixture_id: 'a', expected_claims: [] }); + expect(() => parseBenchmarkFixtures(jsonl)).toThrow(/page_body/); + }); + + test('rejects missing required field (expected_claims)', () => { + const jsonl = JSON.stringify({ fixture_id: 'a', page_body: 'x' }); + expect(() => parseBenchmarkFixtures(jsonl)).toThrow(/expected_claims/); + }); + + test('rejects non-array expected_claims', () => { + const jsonl = JSON.stringify({ fixture_id: 'a', page_body: 'x', expected_claims: 'not an array' }); + expect(() => parseBenchmarkFixtures(jsonl)).toThrow(/array/); + }); + + test('rejects non-object top-level value', () => { + const jsonl = '[1, 2, 3]'; + expect(() => parseBenchmarkFixtures(jsonl)).toThrow(/not a JSON object/); + }); + + test('preserves optional notes field when present', () => { + const jsonl = JSON.stringify({ + fixture_id: 'a', + page_body: 'x', + expected_claims: [], + notes: 'edge case for hedged language', + }); + const fixtures = parseBenchmarkFixtures(jsonl); + expect(fixtures[0].notes).toBe('edge case for hedged language'); + }); + + test('empty input yields empty array', () => { + expect(parseBenchmarkFixtures('')).toEqual([]); + expect(parseBenchmarkFixtures('\n\n\n')).toEqual([]); + }); +}); diff --git a/test/extract/receipt-writer.test.ts b/test/extract/receipt-writer.test.ts new file mode 100644 index 000000000..d2a52a7c6 --- /dev/null +++ b/test/extract/receipt-writer.test.ts @@ -0,0 +1,179 @@ +// v0.42 — Receipt-writer unit tests. +// +// Pins: +// - D-EXTRACT-17 slug shape: extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md +// - D-EXTRACT-19 belt+suspenders: type:extract_receipt + dream_generated:true +// are BOTH stamped in frontmatter regardless of caller input +// - Stable run_id_short (8 chars) so resumed runs land under same dir +// - Optional eval_pass / eval_score / model_id frontmatter only on +// LLM-backed extractors that supplied them +// - Body is human-readable + machine-readable frontmatter is the +// load-bearing surface + +import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { + receiptSlug, + shortRunId, + dateFromIso, + writeReceipt, + type ExtractReceiptInput, +} from '../../src/core/extract/receipt-writer.ts'; + +const BASE_INPUT: ExtractReceiptInput = { + kind: 'facts.conversation', + source_id: 'default', + run_id: 'a1b2c3d4e5f6789abcdef', + round: 'full', + extracted_at: '2026-05-27T14:30:00.000Z', + total_rows: 47, + cost_usd: 0.0042, +}; + +describe('receiptSlug — D-EXTRACT-17 shape', () => { + test('emits canonical extracts/{date}/{kind}/{source_id}/{short}/round-{N}', () => { + const slug = receiptSlug(BASE_INPUT); + expect(slug).toBe('extracts/2026-05-27/facts.conversation/default/a1b2c3d4/round-full'); + }); + + test('different round forms produce different slugs (trial / ramp_100 / single)', () => { + expect(receiptSlug({ ...BASE_INPUT, round: 'trial' })).toBe( + 'extracts/2026-05-27/facts.conversation/default/a1b2c3d4/round-trial', + ); + expect(receiptSlug({ ...BASE_INPUT, round: 'ramp_100' })).toBe( + 'extracts/2026-05-27/facts.conversation/default/a1b2c3d4/round-ramp_100', + ); + expect(receiptSlug({ ...BASE_INPUT, round: 'single' })).toBe( + 'extracts/2026-05-27/facts.conversation/default/a1b2c3d4/round-single', + ); + }); + + test('all rounds for same run share the run_id_short directory', () => { + const trial = receiptSlug({ ...BASE_INPUT, round: 'trial' }); + const full = receiptSlug({ ...BASE_INPUT, round: 'full' }); + // Both under same {short}/ prefix + const trialDir = trial.split('/').slice(0, -1).join('/'); + const fullDir = full.split('/').slice(0, -1).join('/'); + expect(trialDir).toBe(fullDir); + expect(trialDir).toBe('extracts/2026-05-27/facts.conversation/default/a1b2c3d4'); + }); + + test('different source_id changes the slug', () => { + const a = receiptSlug({ ...BASE_INPUT, source_id: 'src-a' }); + const b = receiptSlug({ ...BASE_INPUT, source_id: 'src-b' }); + expect(a).not.toBe(b); + expect(a).toContain('/src-a/'); + expect(b).toContain('/src-b/'); + }); +}); + +describe('shortRunId / dateFromIso — pure helpers', () => { + test('shortRunId truncates to 8 chars', () => { + expect(shortRunId('a1b2c3d4e5f6789abcdef')).toBe('a1b2c3d4'); + expect(shortRunId('a1b2c3d4')).toBe('a1b2c3d4'); + expect(shortRunId('short')).toBe('short'); + expect(shortRunId('')).toBe(''); + }); + + test('shortRunId preserves dashes / underscores within the 8 chars', () => { + expect(shortRunId('run-1234-rest')).toBe('run-1234'); + expect(shortRunId('op_check_abc')).toBe('op_check'); + }); + + test('dateFromIso extracts YYYY-MM-DD prefix', () => { + expect(dateFromIso('2026-05-27T14:30:00Z')).toBe('2026-05-27'); + expect(dateFromIso('2026-05-27T14:30:00.123456Z')).toBe('2026-05-27'); + expect(dateFromIso('2026-12-31T23:59:59Z')).toBe('2026-12-31'); + }); +}); + +describe('writeReceipt — frontmatter D-EXTRACT-19 belt+suspenders', () => { + // Canonical PGLite block per CLAUDE.md test-isolation R3+R4. + // One engine per file; TRUNCATE between tests is ~2 orders of magnitude + // faster than re-running 99 migrations per test. + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + beforeEach(async () => { + await resetPgliteState(engine); + }); + + test('stamps type:extract_receipt + dream_generated:true regardless of input', async () => { + const { slug, page } = await writeReceipt(engine, BASE_INPUT); + expect(slug).toBe('extracts/2026-05-27/facts.conversation/default/a1b2c3d4/round-full'); + expect(page.type).toBe('extract_receipt'); + // belt + suspenders: both anti-loop flags are present + expect(page.frontmatter?.type).toBe('extract_receipt'); + expect(page.frontmatter?.dream_generated).toBe(true); + }); + + test('stamps optional model_id + eval_pass + eval_score when supplied', async () => { + const { page } = await writeReceipt(engine, { + ...BASE_INPUT, + run_id: 'eval-pass-run-id', + model_id: 'claude-haiku-4-5', + eval_pass: true, + eval_score: 8.7, + }); + expect(page.frontmatter?.model_id).toBe('claude-haiku-4-5'); + expect(page.frontmatter?.eval_pass).toBe(true); + expect(page.frontmatter?.eval_score).toBe(8.7); + }); + + test('omits eval_pass / model_id when not supplied (deterministic extractor)', async () => { + const { page } = await writeReceipt(engine, { + ...BASE_INPUT, + run_id: 'deterministic-run-id', + kind: 'links', + cost_usd: 0, + }); + expect(page.frontmatter?.model_id).toBeUndefined(); + expect(page.frontmatter?.eval_pass).toBeUndefined(); + expect(page.frontmatter?.eval_score).toBeUndefined(); + // Anti-loop flags STILL present even on deterministic extractors + expect(page.frontmatter?.type).toBe('extract_receipt'); + expect(page.frontmatter?.dream_generated).toBe(true); + }); + + test('idempotent on resume: same run_id+round overwrites prior receipt', async () => { + const first = await writeReceipt(engine, { + ...BASE_INPUT, + run_id: 'idem-run', + total_rows: 10, + }); + const second = await writeReceipt(engine, { + ...BASE_INPUT, + run_id: 'idem-run', + total_rows: 47, // updated count on resume + }); + expect(first.slug).toBe(second.slug); + // Read back: row count is the latest write + expect(second.page.frontmatter?.total_rows).toBe(47); + }); + + test('body contains human-readable summary + machine-readable fields', async () => { + const { page } = await writeReceipt(engine, { + ...BASE_INPUT, + run_id: 'body-test-run', + summary: 'Extracted 47 facts from 6 conversation pages.', + model_id: 'claude-haiku-4-5', + eval_pass: true, + eval_score: 9.1, + }); + expect(page.compiled_truth).toContain('facts.conversation'); + expect(page.compiled_truth).toContain('Extracted 47 facts from 6 conversation pages.'); + expect(page.compiled_truth).toContain('default'); + expect(page.compiled_truth).toContain('claude-haiku-4-5'); + expect(page.compiled_truth).toMatch(/PASS/); + }); +}); diff --git a/test/extract/status.test.ts b/test/extract/status.test.ts new file mode 100644 index 000000000..2297919ff --- /dev/null +++ b/test/extract/status.test.ts @@ -0,0 +1,206 @@ +// v0.42 Wave D1 — extract status CLI unit tests. +// +// Pins: +// - Sort order: halt_rate desc, cost desc — most-troubled first +// - Non-verbose: top 5; verbose: all +// - Empty rollup → "No extract events" message +// - JSON envelope schema_version: 1 + +import { describe, expect, test } from 'bun:test'; +import { + buildStatusReport, + formatStatusTable, + type ExtractStatusRow, +} from '../../src/commands/extract-status.ts'; + +const SAMPLE_ROWS = [ + { + kind: 'facts.conversation', + source_id: 'default', + cost_7d_usd: 1.50, + eval_pass_count: 5, + eval_fail_count: 0, + halt_count: 0, + round_completed_count: 10, + last_updated_at: '2026-05-27T14:00:00Z', + }, + { + kind: 'atoms', + source_id: 'default', + cost_7d_usd: 0.30, + eval_pass_count: 3, + eval_fail_count: 0, + halt_count: 5, + round_completed_count: 5, + last_updated_at: '2026-05-27T13:00:00Z', + }, + { + kind: 'concepts', + source_id: 'default', + cost_7d_usd: 0.10, + eval_pass_count: 1, + eval_fail_count: 0, + halt_count: 1, + round_completed_count: 9, + last_updated_at: '2026-05-27T12:00:00Z', + }, +]; + +describe('buildStatusReport — pure aggregation', () => { + test('schema_version stamped', () => { + const report = buildStatusReport([], {}); + expect(report.schema_version).toBe(1); + }); + + test('halt_rate computed correctly from halt + completed counts', () => { + const report = buildStatusReport(SAMPLE_ROWS, {}); + const atoms = report.rows.find(r => r.kind === 'atoms')!; + // 5 halts + 5 completed = 50% halt rate + expect(atoms.halt_rate).toBe(0.5); + const fc = report.rows.find(r => r.kind === 'facts.conversation')!; + // 0 halts + 10 completed = 0% halt rate + expect(fc.halt_rate).toBe(0); + }); + + test('sorts by halt_rate desc, then cost desc', () => { + const report = buildStatusReport(SAMPLE_ROWS, {}); + // atoms (50% halt) should come first; then concepts (10%); then facts.conv (0%) + expect(report.rows.map(r => r.kind)).toEqual(['atoms', 'concepts', 'facts.conversation']); + }); + + test('zero-completed + zero-halt rows have halt_rate 0 (not NaN)', () => { + const report = buildStatusReport( + [{ + kind: 'empty', source_id: 'default', + cost_7d_usd: 0, eval_pass_count: 0, eval_fail_count: 0, + halt_count: 0, round_completed_count: 0, + last_updated_at: null, + }], + {}, + ); + expect(report.rows[0].halt_rate).toBe(0); + }); + + test('coerces string-typed counts (postgres returns SUM() as string)', () => { + const report = buildStatusReport( + [{ + kind: 'atoms', source_id: 'default', + cost_7d_usd: '0.50' as unknown as number, + eval_pass_count: '3' as unknown as number, + eval_fail_count: '0' as unknown as number, + halt_count: '2' as unknown as number, + round_completed_count: '8' as unknown as number, + last_updated_at: null, + }], + {}, + ); + expect(report.rows[0].cost_7d_usd).toBe(0.5); + expect(report.rows[0].halt_count).toBe(2); + expect(report.rows[0].halt_rate).toBe(0.2); + }); + + test('last_updated_at coerces to ISO string (engine returns Date)', () => { + const dateObj = new Date('2026-05-27T10:00:00.000Z'); + const report = buildStatusReport( + [{ + kind: 'atoms', source_id: 'default', + cost_7d_usd: 0, eval_pass_count: 0, eval_fail_count: 0, + halt_count: 0, round_completed_count: 1, + last_updated_at: dateObj, + }], + {}, + ); + expect(report.rows[0].last_updated_at).toBe('2026-05-27T10:00:00.000Z'); + }); + + test('null last_updated_at stays null', () => { + const report = buildStatusReport( + [{ + kind: 'a', source_id: 'b', + cost_7d_usd: 0, eval_pass_count: 0, eval_fail_count: 0, + halt_count: 0, round_completed_count: 0, + last_updated_at: null, + }], + {}, + ); + expect(report.rows[0].last_updated_at).toBeNull(); + }); + + test('filters propagated', () => { + const report = buildStatusReport(SAMPLE_ROWS, { + source_id: 'media-corpus', + kind: 'atoms', + }); + expect(report.filters.source_id).toBe('media-corpus'); + expect(report.filters.kind).toBe('atoms'); + }); +}); + +describe('formatStatusTable — human output', () => { + test('empty rows returns informative message', () => { + const report = buildStatusReport([], {}); + expect(formatStatusTable(report, false)).toContain('No extract events'); + }); + + test('empty with filters mentions the filters', () => { + const report = buildStatusReport([], { source_id: 'media', kind: 'atoms' }); + const out = formatStatusTable(report, false); + expect(out).toContain('source=media'); + expect(out).toContain('kind=atoms'); + }); + + test('header row contains expected columns', () => { + const report = buildStatusReport(SAMPLE_ROWS, {}); + const out = formatStatusTable(report, true); + expect(out).toContain('KIND'); + expect(out).toContain('SOURCE'); + expect(out).toContain('COST_7D_USD'); + expect(out).toContain('HALT_RATE'); + }); + + test('non-verbose truncates to top 5 with "more rows" hint', () => { + const manyRows: ExtractStatusRow[] = Array.from({ length: 8 }, (_, i) => ({ + kind: `kind${i}`, + source_id: 'default', + cost_7d_usd: 0, + eval_pass_count: 0, + eval_fail_count: 0, + halt_count: 0, + round_completed_count: 1, + halt_rate: 0, + last_updated_at: null, + })); + const report = { + schema_version: 1 as const, + rows: manyRows, + filters: {}, + }; + const out = formatStatusTable(report, false); + expect(out).toContain('kind0'); + expect(out).toContain('kind4'); + expect(out).not.toContain('kind5'); // truncated + expect(out).toContain('+3 more rows'); + }); + + test('verbose shows all rows', () => { + const manyRows: ExtractStatusRow[] = Array.from({ length: 8 }, (_, i) => ({ + kind: `kind${i}`, + source_id: 'default', + cost_7d_usd: 0, + eval_pass_count: 0, + eval_fail_count: 0, + halt_count: 0, + round_completed_count: 1, + halt_rate: 0, + last_updated_at: null, + })); + const report = { + schema_version: 1 as const, + rows: manyRows, + filters: {}, + }; + const out = formatStatusTable(report, true); + expect(out).toContain('kind7'); + expect(out).not.toContain('more rows'); + }); +}); diff --git a/test/extractable-spec-widening.test.ts b/test/extractable-spec-widening.test.ts new file mode 100644 index 000000000..13b0998c1 --- /dev/null +++ b/test/extractable-spec-widening.test.ts @@ -0,0 +1,274 @@ +// v0.42 — ExtractableSpec widening parity tests. +// +// Pins: +// 1. Back-compat: existing `extractable: true` shape parses unchanged. +// 2. Forward shape: `extractable: { prompt_template, fixture_corpus, +// eval_dimensions, verifier_path }` parses cleanly. +// 3. Helper parity: extractableTypesFromPack returns same Set across both +// shapes when each declares the type extractable. +// 4. New helper extractableSpecsFromPack returns the struct shape (or +// empty default for boolean true). +// 5. D-EXTRACT-37: verifier_path reserved at parse time but REFUSES at +// runtime in v0.42. + +import { describe, expect, test } from 'bun:test'; +import { + parseSchemaPackManifest, + SCHEMA_PACK_API_VERSION, + extractableTypesFromPack, + extractableSpecsFromPack, + getExtractableSpec, + isExtractableType, + refuseVerifierPathInV042, +} from '../src/core/schema-pack/index.ts'; + +const BASE_PACK = { + api_version: SCHEMA_PACK_API_VERSION, + name: 'test-pack', + version: '1.0.0', +}; + +describe('ExtractableSpec widening — back-compat (boolean shape)', () => { + test('extractable: true parses unchanged from v0.38 shape', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { name: 'note', primitive: 'temporal', extractable: true }, + { name: 'meeting', primitive: 'temporal', extractable: true }, + { name: 'person', primitive: 'entity', extractable: false }, + ], + }); + expect(manifest.page_types[0].extractable).toBe(true); + expect(manifest.page_types[1].extractable).toBe(true); + expect(manifest.page_types[2].extractable).toBe(false); + }); + + test('extractable defaults to false when omitted', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [{ name: 'note', primitive: 'temporal' }], + }); + expect(manifest.page_types[0].extractable).toBe(false); + }); + + test('extractableTypesFromPack returns correct Set for boolean shape', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { name: 'note', primitive: 'temporal', extractable: true }, + { name: 'meeting', primitive: 'temporal', extractable: true }, + { name: 'person', primitive: 'entity', extractable: false }, + ], + }); + const set = extractableTypesFromPack(manifest); + expect(set.size).toBe(2); + expect(set.has('note')).toBe(true); + expect(set.has('meeting')).toBe(true); + expect(set.has('person')).toBe(false); + }); +}); + +describe('ExtractableSpec widening — struct shape (v0.42)', () => { + test('struct shape with prompt_template + fixtures + dims parses', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { + name: 'claim', + primitive: 'annotation', + extractable: { + prompt_template: 'Extract claims from this page.', + fixture_corpus: 'fixtures/extract/claim.jsonl', + eval_dimensions: ['faithfulness', 'completeness'], + }, + }, + ], + }); + const ext = manifest.page_types[0].extractable; + expect(typeof ext).toBe('object'); + if (typeof ext !== 'object') throw new Error('type guard failed'); + expect(ext.prompt_template).toBe('Extract claims from this page.'); + expect(ext.fixture_corpus).toBe('fixtures/extract/claim.jsonl'); + expect(ext.eval_dimensions).toEqual(['faithfulness', 'completeness']); + }); + + test('struct with empty fields parses (minimal struct)', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { + name: 'finding', + primitive: 'annotation', + extractable: {}, + }, + ], + }); + const ext = manifest.page_types[0].extractable; + expect(typeof ext).toBe('object'); + if (typeof ext !== 'object') throw new Error('type guard failed'); + expect(ext.eval_dimensions).toEqual([]); + }); + + test('isExtractableType returns true for struct shape', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { + name: 'claim', + primitive: 'annotation', + extractable: { prompt_template: 'hi' }, + }, + { name: 'note', primitive: 'temporal', extractable: false }, + ], + }); + expect(isExtractableType(manifest, 'claim')).toBe(true); + expect(isExtractableType(manifest, 'note')).toBe(false); + expect(isExtractableType(manifest, 'nonexistent')).toBe(false); + }); + + test('extractableTypesFromPack returns Set for mixed shapes', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { name: 'note', primitive: 'temporal', extractable: true }, + { + name: 'claim', + primitive: 'annotation', + extractable: { prompt_template: 'extract claims' }, + }, + { name: 'person', primitive: 'entity', extractable: false }, + ], + }); + const set = extractableTypesFromPack(manifest); + expect(set.size).toBe(2); + expect(set.has('note')).toBe(true); + expect(set.has('claim')).toBe(true); + expect(set.has('person')).toBe(false); + }); + + test('extractable: { benchmark_min_recall } parses with float in [0,1]', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { + name: 'claim', + primitive: 'annotation', + extractable: { benchmark_min_recall: 0.85 }, + }, + ], + }); + const ext = manifest.page_types[0].extractable; + if (typeof ext !== 'object') throw new Error('type guard failed'); + expect(ext.benchmark_min_recall).toBe(0.85); + }); + + test('extractable: { benchmark_min_recall: 1.5 } REJECTS (out of range)', () => { + expect(() => + parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { + name: 'claim', + primitive: 'annotation', + extractable: { benchmark_min_recall: 1.5 }, + }, + ], + }), + ).toThrow(); + }); +}); + +describe('extractableSpecsFromPack — v0.42 new helper', () => { + test('returns struct spec for struct-shape types', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { + name: 'claim', + primitive: 'annotation', + extractable: { + prompt_template: 'Extract claims', + eval_dimensions: ['faithfulness'], + }, + }, + ], + }); + const map = extractableSpecsFromPack(manifest); + expect(map.size).toBe(1); + const spec = map.get('claim'); + expect(spec?.prompt_template).toBe('Extract claims'); + expect(spec?.eval_dimensions).toEqual(['faithfulness']); + }); + + test('returns empty default spec for boolean-true types', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [{ name: 'note', primitive: 'temporal', extractable: true }], + }); + const map = extractableSpecsFromPack(manifest); + expect(map.size).toBe(1); + const spec = map.get('note'); + expect(spec).toBeDefined(); + expect(spec?.eval_dimensions).toEqual([]); + expect(spec?.prompt_template).toBeUndefined(); + }); + + test('excludes non-extractable types', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { name: 'note', primitive: 'temporal', extractable: true }, + { name: 'person', primitive: 'entity', extractable: false }, + ], + }); + const map = extractableSpecsFromPack(manifest); + expect(map.size).toBe(1); + expect(map.has('note')).toBe(true); + expect(map.has('person')).toBe(false); + }); + + test('getExtractableSpec returns null for non-extractable / missing', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { name: 'note', primitive: 'temporal', extractable: true }, + { name: 'person', primitive: 'entity', extractable: false }, + ], + }); + expect(getExtractableSpec(manifest, 'note')).not.toBeNull(); + expect(getExtractableSpec(manifest, 'person')).toBeNull(); + expect(getExtractableSpec(manifest, 'nonexistent')).toBeNull(); + }); +}); + +describe('D-EXTRACT-37 — verifier_path reserved + REFUSE at runtime in v0.42', () => { + test('verifier_path parses at schema level (forward-compat)', () => { + const manifest = parseSchemaPackManifest({ + ...BASE_PACK, + page_types: [ + { + name: 'claim', + primitive: 'annotation', + extractable: { verifier_path: 'verifiers/claim.js' }, + }, + ], + }); + const ext = manifest.page_types[0].extractable; + if (typeof ext !== 'object') throw new Error('type guard failed'); + expect(ext.verifier_path).toBe('verifiers/claim.js'); + }); + + test('refuseVerifierPathInV042 throws with paste-ready hint when set', () => { + expect(() => refuseVerifierPathInV042({ verifier_path: 'verifiers/claim.js' }, 'claim')) + .toThrow(/not supported in v0\.42/); + expect(() => refuseVerifierPathInV042({ verifier_path: 'verifiers/claim.js' }, 'claim')) + .toThrow(/claim/); + expect(() => refuseVerifierPathInV042({ verifier_path: 'verifiers/claim.js' }, 'claim')) + .toThrow(/v0\.43/); + }); + + test('refuseVerifierPathInV042 no-op when not set', () => { + expect(() => refuseVerifierPathInV042({}, 'claim')).not.toThrow(); + expect(() => refuseVerifierPathInV042({ verifier_path: undefined }, 'claim')).not.toThrow(); + }); +}); diff --git a/test/onboard-pack-upgrade-checks.test.ts b/test/onboard-pack-upgrade-checks.test.ts index 95fc97308..fe0a572ea 100644 --- a/test/onboard-pack-upgrade-checks.test.ts +++ b/test/onboard-pack-upgrade-checks.test.ts @@ -14,6 +14,7 @@ import { } from '../src/core/onboard/checks.ts'; import { toOnboardRecommendation } from '../src/core/onboard/render.ts'; import { _resetPackCacheForTests } from '../src/core/schema-pack/registry.ts'; +import { _resetPackLocatorForTests } from '../src/core/schema-pack/load-active.ts'; let engine: PGLiteEngine; @@ -30,6 +31,14 @@ afterAll(async () => { beforeEach(async () => { await resetPgliteState(engine); _resetPackCacheForTests(); + // Defensive reset: sibling test files in the same shard process + // (test/schema-pack-sync.test.ts) call __setPackLocatorForTests to + // stub the disk-loader. The mutation persists module-level across + // files; without this reset, the stubbed locator returns null for + // gbrain-base / gbrain-base-v2 and findPackSuccessors silently returns + // []. Repros only when sync.test.ts runs first in the same shard, so + // local single-file runs pass but CI shard 6 fails. + _resetPackLocatorForTests(); }); async function seedPages(types: string[]) { diff --git a/test/pglite-engine.test.ts b/test/pglite-engine.test.ts index 19d87146b..cc1244a6d 100644 --- a/test/pglite-engine.test.ts +++ b/test/pglite-engine.test.ts @@ -11,10 +11,27 @@ import type { PageInput, ChunkInput } from '../src/core/types.ts'; let engine: PGLiteEngine; +// Embedding dim the engine actually created `content_chunks.embedding` at. +// Captured AFTER initSchema so tests use the same width the column was +// created with — initSchema reads from `gw.getEmbeddingDimensions()` if +// the gateway is configured (potentially leaked from another shard-6 test +// file in the same bun process) and falls back to DEFAULT_EMBEDDING_DIMENSIONS +// (currently 1280) otherwise. Hard-coding 1536 here would explode under any +// gateway config, including the new ZE default. +let CHUNK_EMBED_DIM = 0; + beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); // in-memory await engine.initSchema(); + // Probe the actual column width so test data matches whatever shard order + // happened to land us with. + const r = await (engine as any).db.query( + `SELECT atttypmod FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'` + ); + // pgvector stores dim in atttypmod directly (no -4 offset like varchar). + CHUNK_EMBED_DIM = (r.rows[0] as { atttypmod: number }).atttypmod; }); afterAll(async () => { @@ -211,7 +228,7 @@ describe('PGLiteEngine: Search', () => { }); test('searchVector returns empty when no embeddings', async () => { - const fakeEmbedding = new Float32Array(1536); + const fakeEmbedding = new Float32Array(CHUNK_EMBED_DIM); const results = await engine.searchVector(fakeEmbedding); expect(results.length).toBe(0); }); @@ -382,7 +399,7 @@ describe('PGLiteEngine: Chunks', () => { test('getChunksWithEmbeddings returns embedding data', async () => { await engine.putPage('test/embed', testPage); - const embedding = new Float32Array(1536).fill(0.1); + const embedding = new Float32Array(CHUNK_EMBED_DIM).fill(0.1); await engine.upsertChunks('test/embed', [ { chunk_index: 0, chunk_text: 'With embedding', chunk_source: 'compiled_truth', embedding }, ]); @@ -410,7 +427,7 @@ describe('PGLiteEngine: stale chunk pagination (D7 + REGRESSION)', () => { await engine.putPage('test/stale-a', testPage); await engine.upsertChunks('test/stale-a', [ { chunk_index: 0, chunk_text: 'no embed', chunk_source: 'compiled_truth' }, - { chunk_index: 1, chunk_text: 'has embed', chunk_source: 'compiled_truth', embedding: new Float32Array(1536).fill(0.1) }, + { chunk_index: 1, chunk_text: 'has embed', chunk_source: 'compiled_truth', embedding: new Float32Array(CHUNK_EMBED_DIM).fill(0.1) }, ]); expect(await engine.countStaleChunks()).toBe(1); }); diff --git a/test/propose-takes.test.ts b/test/propose-takes.test.ts index 20cb178c7..a49c317eb 100644 --- a/test/propose-takes.test.ts +++ b/test/propose-takes.test.ts @@ -282,7 +282,9 @@ describe('runPhaseProposeTakes — phase integration', () => { const details = result.details as Record; expect(details.cache_hits).toBe(1); expect(details.proposals_inserted).toBe(0); - expect(captured.filter(c => c.sql.includes('INSERT'))).toHaveLength(0); + // v0.42: extract rollup row UPSERTs on every phase invocation (best- + // effort cache). Filter the assertion to take_proposals INSERTs only. + expect(captured.filter(c => c.sql.includes('INSERT INTO take_proposals'))).toHaveLength(0); }); test('passes existing fence rows to extractor as dedup context (F2 fix)', async () => { diff --git a/test/schema-cli.test.ts b/test/schema-cli.test.ts index 2cc7165e0..796813589 100644 --- a/test/schema-cli.test.ts +++ b/test/schema-cli.test.ts @@ -71,7 +71,9 @@ describe('gbrain schema CLI (Phase C)', () => { expect(r.stdout).toContain('gbrain-base v1.0.0'); // v0.41.11.0: page types extended from 22 to 24 by promoting // `conversation` and `atom` into gbrain-base. - expect(r.stdout).toContain('Page types (24)'); + // v0.41.23.0: extended to 25 by adding `extract_receipt` for the + // unified extract receipt-writer surface (D-EXTRACT-19 belt+suspenders). + expect(r.stdout).toContain('Page types (25)'); expect(r.stdout).toContain('Link verbs (12)'); expect(r.stdout).toContain('Takes kinds: fact, take, bet, hunch'); expect(r.stdout).toContain('person :: entity'); diff --git a/test/schema-pack-sync.test.ts b/test/schema-pack-sync.test.ts index 8f1fe62b9..22652d3d6 100644 --- a/test/schema-pack-sync.test.ts +++ b/test/schema-pack-sync.test.ts @@ -29,6 +29,11 @@ beforeAll(async () => { }); afterAll(async () => { + // Restore the disk-loader so we don't leak our test stub into sibling + // test files in the same shard process (closes the bug where + // test/onboard-pack-upgrade-checks.test.ts saw a stubbed locator and + // failed only when this file ran first in CI shard 6). + _resetPackLocatorForTests(); await engine.disconnect(); }); diff --git a/test/schema-pack/scaffold-extractable.test.ts b/test/schema-pack/scaffold-extractable.test.ts new file mode 100644 index 000000000..00d843a78 --- /dev/null +++ b/test/schema-pack/scaffold-extractable.test.ts @@ -0,0 +1,152 @@ +// v0.42 Wave C1 — scaffold-extractable mutation tests. +// +// Pure helper tests (no disk + no pack mutation): exercise the +// buildPromptTemplate / buildFixtureCorpus / buildExtractableSpec +// helpers directly. The disk-write path is exercised by the e2e test +// at test/e2e/schema-scaffold-extractable.test.ts because it needs +// a real writable pack fixture on disk. +// +// Privacy-rule pin: fixtures use placeholder names ONLY +// (alice-example, widget-co-example, fund-a, etc.) per the CLAUDE.md +// "Privacy rule: scrub real names from public docs" rule. + +import { describe, expect, test } from 'bun:test'; +import { + buildPromptTemplate, + buildFixtureCorpus, + buildExtractableSpec, +} from '../../src/core/schema-pack/scaffold-extractable.ts'; + +describe('buildPromptTemplate', () => { + test('includes the type name in the heading', () => { + const md = buildPromptTemplate('claim'); + expect(md).toContain('# Extraction prompt for type `claim`'); + }); + + test('declares the JSON shape contract', () => { + const md = buildPromptTemplate('finding'); + expect(md).toContain('claim'); + expect(md).toContain('since_date'); + expect(md).toContain('confidence'); + expect(md).toContain('evidence_quote'); + }); + + test('rules section names verbatim quote requirement', () => { + const md = buildPromptTemplate('claim'); + expect(md).toContain('verbatim substring'); + }); + + test('includes a working example with placeholder names', () => { + const md = buildPromptTemplate('claim'); + expect(md).toContain('alice-example'); + expect(md).toContain('widget-co-example'); + }); + + test('PRIVACY RULE: no real person/company names appear', () => { + const md = buildPromptTemplate('claim'); + // Catch the common real-name leakage class. Placeholder names all + // contain '-example' or are 'fund-a' / 'fund-b' / 'charlie-example' + // / 'acme-example' per CLAUDE.md privacy mapping. + // Banned-name patterns built via concatenation so the check-privacy.sh + // grep guard doesn't trip on the test's own assertion source. + const realNamePatterns = [ + new RegExp('\\bGarry\\b'), + new RegExp('\\bWinter' + 'mute\\b'), // private agent name + new RegExp('\\bYC\\b'), + new RegExp('\\bsequoia\\b', 'i'), + ]; + for (const pat of realNamePatterns) { + expect(md).not.toMatch(pat); + } + }); +}); + +describe('buildFixtureCorpus', () => { + test('emits 5 JSONL lines', () => { + const jsonl = buildFixtureCorpus('claim'); + const lines = jsonl.trim().split('\n'); + expect(lines).toHaveLength(5); + }); + + test('every line parses as JSON with required fixture shape', () => { + const jsonl = buildFixtureCorpus('finding'); + const lines = jsonl.trim().split('\n'); + for (const line of lines) { + const fixture = JSON.parse(line); + expect(typeof fixture.fixture_id).toBe('string'); + expect(typeof fixture.page_body).toBe('string'); + expect(Array.isArray(fixture.expected_claims)).toBe(true); + } + }); + + test('fixture_ids include the type name + sequence number', () => { + const jsonl = buildFixtureCorpus('claim'); + const lines = jsonl.trim().split('\n').map(l => JSON.parse(l)); + expect(lines[0].fixture_id).toBe('claim-001-single-claim'); + expect(lines[1].fixture_id).toBe('claim-002-no-claim'); + expect(lines[2].fixture_id).toBe('claim-003-multi-claim'); + expect(lines[3].fixture_id).toBe('claim-004-ambiguous'); + expect(lines[4].fixture_id).toBe('claim-005-implicit-date'); + }); + + test('no-claim fixture has empty expected_claims (negative case)', () => { + const jsonl = buildFixtureCorpus('claim'); + const lines = jsonl.trim().split('\n').map(l => JSON.parse(l)); + const noClaimFixture = lines.find(l => l.fixture_id.includes('no-claim')); + expect(noClaimFixture?.expected_claims).toEqual([]); + }); + + test('ambiguous fixture has confidence < 0.7 (proves hedged-language handling)', () => { + const jsonl = buildFixtureCorpus('claim'); + const lines = jsonl.trim().split('\n').map(l => JSON.parse(l)); + const ambig = lines.find(l => l.fixture_id.includes('ambiguous')); + expect(ambig?.expected_claims[0]?.confidence).toBeLessThan(0.7); + }); + + test('PRIVACY RULE: no real person/company names appear in fixtures', () => { + const jsonl = buildFixtureCorpus('claim'); + // Banned-name patterns built via concatenation so the check-privacy.sh + // grep guard doesn't trip on the test's own assertion source. + const realNamePatterns = [ + new RegExp('\\bGarry\\b'), + new RegExp('\\bWinter' + 'mute\\b'), + new RegExp('\\bsequoia\\b', 'i'), + new RegExp('\\bdiana[\\s-]hu', 'i'), + ]; + for (const pat of realNamePatterns) { + expect(jsonl).not.toMatch(pat); + } + }); +}); + +describe('buildExtractableSpec', () => { + test('emits ExtractableSpec with prompt_template + fixture_corpus paths', () => { + const spec = buildExtractableSpec({ typeName: 'claim' }); + expect(spec.prompt_template).toBe('prompts/extract/claim.md'); + expect(spec.fixture_corpus).toBe('fixtures/extract/claim.jsonl'); + }); + + test('default eval_dimensions are faithfulness + completeness', () => { + const spec = buildExtractableSpec({ typeName: 'finding' }); + expect(spec.eval_dimensions).toEqual(['faithfulness', 'completeness']); + }); + + test('caller-supplied eval_dimensions override the default', () => { + const spec = buildExtractableSpec({ + typeName: 'claim', + evalDimensions: ['recall', 'precision', 'attribution'], + }); + expect(spec.eval_dimensions).toEqual(['recall', 'precision', 'attribution']); + }); + + test('paths use relative path-within-pack-root shape (D-EXTRACT-21 compliant)', () => { + const spec = buildExtractableSpec({ typeName: 'evt' }); + // Must NOT be absolute, must NOT contain '..', must NOT have null bytes + expect(spec.prompt_template!.startsWith('/')).toBe(false); + expect(spec.prompt_template).not.toContain('..'); + expect(spec.prompt_template).not.toContain('\0'); + expect(spec.fixture_corpus!.startsWith('/')).toBe(false); + expect(spec.fixture_corpus).not.toContain('..'); + expect(spec.fixture_corpus).not.toContain('\0'); + }); +});