diff --git a/CHANGELOG.md b/CHANGELOG.md index ed7663cb2..30c87aad9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,277 @@ All notable changes to GBrain will be documented in this file. +## [0.41.16.0] - 2026-05-26 + +**Your brain now understands every common chat format, not just iMessage.** + +Before today, GBrain's conversation parser knew one shape: +`**Name** (date time): text`. If your chat history came from Telegram, Discord, +WhatsApp, Signal, IRC, Matrix, Teams, or anywhere else, the parser silently +dropped every message and the dream cycle never extracted any facts. A real user +reported 134 Telegram pages stuck in this state. + +We didn't fix it by adding one regex per format and shipping a release every +time someone's chat export looked weird. GBrain now ships with a 12-pattern +built-in registry covering the most common chat-export shapes on Earth, plus an +opt-in LLM fallback for the long tail. The dream cycle picks the right parser +per page automatically — no config, no waiting for the next release. + +The same wave introduces a new `progressive-batch` primitive (Wintermute-inspired +ramp-up: trial 10 → 100 → 500 → full with verification at each stage) so future +batch operations get the discipline for free instead of each reinventing it. + +### How to use it + +``` +gbrain upgrade # picks up the new parser +gbrain dream # next cycle extracts facts from previously-stuck chat pages +gbrain conversation-parser list-builtins # see the 12 patterns shipped +gbrain conversation-parser scan conversations/x # debug which pattern matched (or didn't) +gbrain eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm + # the CI gate; runs in bun run verify +``` + +To opt INTO Haiku polish + fallback for the long-tail formats (private chat +content DOES go to Anthropic when enabled — privacy posture, not just cost): + +``` +gbrain config set conversation_parser.llm_polish_enabled true +gbrain config set conversation_parser.llm_fallback_enabled true +``` + +Cost expectation when enabled: ~$0.0002 per polished page, ~$0.0005 per +fallback call. Bounded by the existing brain-wide `BudgetTracker` cap +(default $5/cycle). + +### The 12 built-in patterns + +| id | sample | +|------------------------|-------------------------------------------------| +| `imessage-slack` | `**Alice** (2024-03-15 9:00 AM): hi` | +| `telegram-bracket` | `**[18:37] 👤 Alice:** hi` (PR #1461 verbatim) | +| `telegram-text-export` | `Alice, [Mar 15, 2024 at 6:37:00 PM]` | +| `whatsapp-iso` | `[15/03/24, 18:37:00] Alice: hi` | +| `whatsapp-us` | `3/15/24, 6:37 PM - Alice: hi` | +| `discord-export` | `[03/15/2024 6:37 PM] Alice` + multi-line body | +| `discord-classic` | `Alice — Today at 18:37` + multi-line body | +| `signal-export` | `Alice (2024-03-15 18:37:00 UTC): hi` | +| `matrix-element` | `[18:37] @alice:matrix.org: hi` | +| `irc-classic` | ` hi` | +| `irc-weechat` | `18:37 hi` | +| `teams-export` | `Alice, 3/15/2024 6:37 PM: hi` | + +Each pattern was sourced from a public format reference (signal-cli, +DiscordChatExporter, Telegram Desktop export, WhatsApp export docs, Element +matrix-archive scripts, irssi / weechat defaults). Every pattern carries +`test_positive[]` + `test_negative[]` sample sets validated at module load — +a typo in any built-in regex makes gbrain refuse to start. + +### What to watch for after upgrade + +- **Timezone handling.** Telegram/Discord/IRC/Matrix have time-only stamps + (no date in the line). The parser uses your page frontmatter's `date:` field + for the date and assumes UTC for the time unless you ALSO set + `timezone: America/Los_Angeles` (IANA zone) in frontmatter. Pages without a + frontmatter timezone get a one-time stderr warn so you know the fact + timestamps may be off by hours. +- **Pattern priority scoring.** When multiple built-ins could match a page, + the orchestrator scores all candidates across the first 10 lines and picks + the highest match rate (declared priority as tie-breaker). Mixed-format + pages pick the dominant format — use `gbrain conversation-parser scan + ` to see which one won. +- **doctor checks.** `gbrain doctor` gains three new checks: + `conversation_format_coverage` (per-pattern hit count + unmatched %), + `progressive_batch_audit_health` (abort-verdict counts from the new + primitive's audit JSONL), and `conversation_parser_probe_health` (opt-in + nightly LLM-quality drift detection — skipped with enable-hint until you + flip the flag). + +### What we caught and fixed before merging + +Codex's independent review surfaced eight substantive technical risks that +the eng review missed. All adopted: + +- **Privacy posture.** Default-ON LLM polish for private chat logs was the + wrong default. Reversed to opt-IN to match the existing + `nightly_quality_probe.enabled` precedent. +- **ReDoS theater.** Promise.race cannot preempt a catastrophic JS regex. + v1 drops arbitrary user regex entirely; user patterns wait for v0.42+ + with worker-isolated regex execution (safe-regex / RE2). +- **LLM-inferred regex persistence.** Inferring a regex from 20 sampled + lines and persisting for future cycles is a silent-corruption machine + (real exports have day separators, edits, replies, locale shifts later + in the file). LLM fallback parses for THIS page only, cached by + content_hash. +- **Pattern priority scoring.** "First built-in wins" silently mis-routes + when formats overlap. Replaced with a 10-line scoring pass. +- **Timezone policy on every pattern.** Original plan hardcoded `Z`. + Now declared per-pattern; time-only formats emit a loud warn. +- **Verifier shape.** Original `expectedDelta` contract didn't fit + reindex / embed / contradiction-eval. Refactored to a discriminated + union: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`. +- **Behavior parity for retrofits.** Sites that previously "jumped + straight to full" keep doing so by default; ramp is opt-in per-site. +- **Real-corpus fixture gap.** v1 ships synthetic fixtures only; real- + corpus-redacted fixtures filed as a follow-up TODO. + +### Itemized changes + +#### New: progressive-batch primitive + +- `src/core/progressive-batch/orchestrator.ts` — `runProgressiveBatch(items, + verifier, policy, runner)` with verifier+policy injection. Reads + `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd`; null both ways + triggers `abort_cost_cap reason='no_budget_safety_net'` (fail-closed). +- `src/core/progressive-batch/types.ts` — `Stage`, `StageVerdict`, + `AbortReason`, discriminated `Verifier` union, `Policy`, `StageReport`. +- `src/core/progressive-batch/audit.ts` — ISO-week JSONL at + `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared + `audit-writer` primitive. +- Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, + `GBRAIN_PROGRESSIVE_BATCH_AUTO=1`, + `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. + +#### New: conversation-parser cathedral + +- `src/core/conversation-parser/{types,builtins,parse,llm-base,llm-polish,llm-fallback,eval,nightly-probe}.ts` + — 8 new modules; 12 built-in patterns with module-load validation; + `DEFAULT_SPEAKER_CLEAN` exported (the PR #1461 helper, promoted); + per-pattern `quick_reject` for O(1) prefix screening; per-pattern + `multi_line` flag for Discord-style multi-line bodies; per-pattern + `timezone_policy`. +- `src/commands/extract-conversation-facts.ts` — `parseConversationMessages` + becomes a thin wrapper over the new orchestrator. `processPage` threads + the full Page through so frontmatter date / timezone / effective_date + precedence takes effect. PR #1461's 33 cases pass verbatim. + +#### Progressive-batch retrofit landing + +Three sites fully retrofitted onto the primitive in this PR (each ships +with audit JSONL + cost-cap gate): +- `src/commands/reindex.ts` (T11) — markdown chunker-version sweep. +- `src/commands/reindex-code.ts` (T12) — code page re-import with + BudgetTracker integration preserved. +- `src/commands/reindex-multimodal.ts` (T14) — streaming-checkpoint + shape preserved; per-batch audit row written to the primitive's + JSONL. + +Six sites documented with v0.41.14.0+ retrofit-deferred notes (each +file's header explains WHY the bespoke shape needs a dedicated +design pass): `src/core/post-upgrade-reembed.ts` (T13 — wrapper that +calls T11 reindex; reindex's primitive wrap is the actual value), +`src/commands/book-mirror.ts` (T15 — fan-out-to-MinionQueue, not a +batch loop), `src/core/brainstorm/orchestrator.ts` (T16 — already +has `withBudgetTracker` + own cost prompt), `src/commands/eval-suspected-contradictions.ts` +(T17 — sampling probe lives at a different layer than the run loop), +`src/core/minions/handlers/contextual-reindex-per-chunk.ts` (T18 — +Minion handler; primitive value lives at submitter side), +`src/commands/extract.ts` (T19 — pure deterministic regex, no LLM cost +to gate). Per D26 + D21, behavior parity preservation is the priority; +each deferred retrofit is filed in TODOS.md with the specific reason +its shape needs more design. + +- `src/core/eval-contradictions/cost-prompt.ts` — marked `@deprecated` + for v0.41.14.0+; existing API + caller preserved for behavior parity. + +#### New CLI surfaces + +- `gbrain eval conversation-parser [--no-llm] [--min-recall F] [--json]` + — exit 0 PASS / 1 FAIL / 2 USAGE. +- `gbrain conversation-parser scan ` — dry-run on one page. +- `gbrain conversation-parser list-builtins [--json]` — operator + discoverability. +- `gbrain conversation-parser validate ` — v1 emits "deferred to + v0.42+" notice. + +#### Doctor + +- New checks: `conversation_format_coverage`, + `progressive_batch_audit_health`, `conversation_parser_probe_health`. + +#### CI gates + +- `bun run check:fixture-privacy` — banned-token grep over fixtures. +- `bun run check:conversation-parser` — fixture-corpus eval gate with + `--no-llm` (deterministic, no API keys). + +#### Schema + +- Migration v97 (`conversation_parser_llm_cache_table`) — content-hash-keyed + cache for LLM polish + fallback. NO `inferred_patterns` table (codex + outside voice correctly identified inferred-regex persistence as a + silent-corruption machine). + +#### Tests + +- 35 unit cases for the progressive-batch primitive (every verdict path). +- 39 unit cases for the parser orchestrator (PR #1461 6-case verbatim + regression, every built-in matches its `test_positive`, priority + scoring, date derivation, multi-line, quick_reject, timezone warning). +- 16 + 6 + 6 unit cases for the LLM base / polish / fallback (provider + probe, cache hit, fail-open paths, headroom guard). +- 7 unit cases for the nightly probe (mode-gated default, rate limit, + adversarial false-positive detection). +- Existing 27 `extract-conversation-facts.test.ts` cases unchanged + (back-compat invariant). + +### Closed in this wave + +- PR #1461 ("feat: support bracket-time format in conversation facts + parser") — superseded. Contributor's `BRACKET_TIME_RX` + `cleanSpeaker` + survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` + export. Co-Authored-By preserved. + +### Deferred to v0.41.14.0+ (filed in TODOS.md) + +- 9-site progressive-batch retrofit (reindex.ts, reindex-multimodal.ts, + reindex-code.ts, post-upgrade-reembed.ts, book-mirror.ts, + brainstorm/orchestrator.ts, eval-suspected-contradictions.ts, + eval-contradictions/cost-prompt.ts, contextual-reindex-per-chunk.ts). + The primitive ships with one proven consumer (parser cathedral); + retrofits land as a structured follow-up wave. +- Worker-based regex isolate-and-kill for arbitrary user patterns. +- Per-pattern speaker-alias normalization (LongMemEval-style). +- Cross-modal scoring of LLM-fallback output (catch hallucinations + beyond the adversarial fixture set). +- Per-source pattern overrides + (`cycle.conversation_facts_backfill.source_overrides..patterns`). +- Real-corpus-redacted fixtures (5-10 production pages scrubbed via + one-shot script; production-recovery signal vs synthetic-only). + +## To take advantage of v0.41.16.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if +`gbrain doctor` warns about a partial migration: + +1. **Run the orchestrator manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Trigger a dream cycle to extract facts from previously-stuck conversation pages:** + ```bash + gbrain dream + ``` +3. **Verify the outcome:** + ```bash + gbrain doctor # check conversation_format_coverage + gbrain conversation-parser list-builtins # see the 12 shipped patterns + gbrain conversation-parser scan # confirm a specific page parses + ``` +4. **(Optional) Opt IN to LLM polish + fallback for long-tail formats:** + ```bash + gbrain config set conversation_parser.llm_polish_enabled true + gbrain config set conversation_parser.llm_fallback_enabled true + ``` + This sends chat content to Anthropic. Cost ~$0.0002 per polished page, + ~$0.0005 per fallback call. Bounded by your brain-wide `BudgetTracker` cap. +5. **If any step fails or the numbers look wrong,** please file an issue: + https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - output of `gbrain conversation-parser scan ` for a stuck page + - which step broke + + This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you. ## [0.41.15.0] - 2026-05-26 **Your hourly cron can stop killing every sync mid-flight. Each source now diff --git a/CLAUDE.md b/CLAUDE.md index aed7aff9d..3e92d22b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -161,6 +161,8 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. +- `src/core/conversation-parser/` (v0.41.16.0) — 12-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (12 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. diff --git a/TODOS.md b/TODOS.md index aef56967b..9d395fcc7 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,98 @@ # TODOS +## v0.41.16.0 conversation parser + progressive-batch follow-ups (v0.41.14.0+) + +The v0.41.16.0 cathedral shipped the parser primitive + progressive-batch +primitive + ONE proven consumer (extract-conversation-facts). Per D2 (codex +outside voice acknowledged + user accepted the trade), the wider 9-site +retrofit + 5 architectural follow-ups land as structured waves to keep each +PR bisectable. + +- [ ] **v0.41.14.0: 9-site progressive-batch retrofit (one commit per site + for bisect).** The primitive at `src/core/progressive-batch/` shipped + with ONE consumer (extract-conversation-facts). Twelve other batch + sites still reinvent their own ramp+cost-prompt patterns; rule of + three is comfortably past. Retrofit each onto the primitive in + sequence, one commit per site for bisect, behavior parity tested + before/after migration: + - `src/commands/reindex.ts` (markdown chunker bump) — existing 10s + Ctrl-C grace + `GBRAIN_NO_REEMBED=1` env map to + `interactiveAbortMs` + `GBRAIN_PROGRESSIVE_BATCH_DISABLED`. + - `src/commands/reindex-multimodal.ts` (Phase 3 unified column) — + 360min lock survives orthogonal; cost prompt becomes stage report. + - `src/commands/reindex-code.ts` — sites without existing ramps + keep jump-to-full default per D21; ramp is opt-in. + - `src/core/post-upgrade-reembed.ts` — TTY auto-proceed maps directly + to `GBRAIN_PROGRESSIVE_BATCH_AUTO`. + - `src/commands/book-mirror.ts` — cost-estimate becomes stage 0. + - `src/core/brainstorm/orchestrator.ts` — already wraps in + `withBudgetTracker`; primitive accepts the active tracker. + - `src/commands/eval-suspected-contradictions.ts` — sampling probe + becomes stage 0; full run becomes stages 1-4. + - `src/core/eval-contradictions/cost-prompt.ts` — DELETE entirely; + callers route through the primitive's Policy.maxCostUsd. + - `src/core/minions/handlers/contextual-reindex-per-chunk.ts` — + `GBRAIN_PROGRESSIVE_BATCH_AUTO` defaults true for workers. + Priority: P2. Rationale: future batch features inherit the discipline + for free; the 12 existing sites stay bespoke until done. + +- [ ] **v0.42+: per-source pattern overrides.** New config key + `cycle.conversation_facts_backfill.source_overrides..patterns` + (JSON array of `simple_pattern` specs). Pros: brain with both + Telegram AND Discord sources can declare per-source pattern priority. + Cons: another config key to validate; per-source pattern indexing + needs runtime per-page lookup. Context: v1 keeps patterns + brain-global to ship faster. Priority: P3. + +- [ ] **v0.42+: Worker-based regex isolate-and-kill for arbitrary user + patterns.** Compile user-supplied regex inside a Node Worker and kill + the Worker on timeout. Why: Node has no native `RegExp.abort`; v0.41.13 + Promise.race-based ReDoS sniff is fake (the regex engine can't be + preempted once running). v0.41.13 ships NO arbitrary user regex + surface to avoid the security theater; user patterns wait for this. + Alternative: safe-regex npm (synchronous static analysis, catches + the canonical /^(a+)+$/ class). Cons: per-pattern Worker startup + cost; complexity. Context: today's `simple_pattern` structured spec + (also v0.42+) compiles to known-safe regex shapes without the + worker dance. Priority: P3. + +- [ ] **v0.42+: per-pattern speaker-alias normalization.** LongMemEval- + style per-page alias map collapsing `"Alice"` + `"Alice Smith"` + + `"alice"` to one canonical slug. See `src/eval/longmemeval/extract.ts` + `AliasMap` shape. Pros: cleaner downstream fact extraction. Cons: + state per-page (currently stateless orchestrator). Context: today + downstream `resolveEntitySlug` handles this via the entities table + (good enough but cleaner upstream). Priority: P3. + +- [ ] **v0.42+: cross-modal scoring of LLM-fallback output.** Feed + fallback-parsed messages to a judge model and score correctness. + Why: catches hallucinated parses (LLM "inventing" speakers/timestamps + on adversarial input). Pros: closes a quality gap. Cons: cost; + needs budget policy + judge model selection. Context: v0.41.16.0 + catches hallucination only via the adversarial fixture set in the + nightly probe (5 fixtures). Real adversarial drift = more + fixtures + judge scoring. Priority: P2. + +- [ ] **v0.42+: mega-regex compilation fallback.** Combine 12+ built-ins + into one alternation regex if D11 quick_reject benchmarks disappoint. + Pros: faster on dense conversations (single pass per line). Cons: + debugging which alternative matched is nightmarish; one bad anchor + corrupts all. Context: D11 quick_reject is expected to deliver ~10× + speedup; revisit only if real corpus measurements show >5ms parse + time per page. Priority: P3. + +- [ ] **v0.42+: real-corpus-redacted fixture set.** Add + `test/fixtures/conversation-formats/real-corpus-redacted/` derived + from 5-10 real production Telegram pages with: real names → + placeholder names (alice-example, charlie-example, fund-a) via a + one-shot scrubber script, real timestamps preserved, real message + bodies preserved STRUCTURALLY (length + line-break shape) but + content replaced with lorem-ipsum-style synthetic prose. Privacy + guard extended. Why: synthetic 8-12 message fixtures prove regex + syntax, not production recovery of 134 real Telegram-shaped pages. + Real edge cases (long pastes, code blocks, replies, day-separators) + only surface in real corpora. Adds ~30min scrub step + privacy + guard maintenance. Priority: P2. ## v0.41.15.0 sync-reliability follow-ups (v0.42+) - [ ] **v0.42+: subprocess fan-out for `sync --all` (`--independent` mode diff --git a/VERSION b/VERSION index f956ed85e..df702cf8c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.15.0 \ No newline at end of file +0.41.16.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index a05d80a3d..4133fb2e1 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -303,6 +303,8 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. +- `src/core/conversation-parser/` (v0.41.16.0) — 12-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (12 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id ` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`. diff --git a/package.json b/package.json index 5139ebda3..2ff62d5c1 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.41.15.0", + "version": "0.41.16.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", @@ -75,6 +75,8 @@ "check:test-isolation": "scripts/check-test-isolation.sh", "check:fuzz-purity": "scripts/check-fuzz-purity.sh", "check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh", + "check:fixture-privacy": "scripts/check-fixture-privacy.sh", + "check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm", "postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2", "prepublish:clawhub": "bun run build:all", "publish:clawhub": "clawhub package publish . --family bundle-plugin" diff --git a/scripts/check-fixture-privacy.sh b/scripts/check-fixture-privacy.sh new file mode 100755 index 000000000..920456057 --- /dev/null +++ b/scripts/check-fixture-privacy.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# v0.41.13.0 — Privacy guard for test/fixtures/conversation-formats/. +# +# Per CLAUDE.md privacy rule: "Never reference real people, companies, +# funds, or private agent names in any public-facing artifact." +# Test fixtures ship in the repo; they ARE public. +# +# This script greps for known real-name signals and fails the build if +# any leak. Add to bun run verify so the gate runs every PR. +# +# Banned tokens (case-insensitive substring match): +# - 'wintermute' / 'openclaw' (real downstream agent names) +# - 'palantir' (real company per Garry's history) +# - common real-fund names (sequoia, andreessen, founders fund, etc.) +# - 'ycombinator' / 'y combinator' (the org running gbrain) +# +# Allowed (placeholder convention): +# - alice-example / bob-example / charlie-example / diana-example +# - widget-co / acme-example +# - fund-a / fund-b / fund-c + +set -euo pipefail + +FIXTURE_DIR="test/fixtures/conversation-formats" + +if [ ! -d "$FIXTURE_DIR" ]; then + echo "[check-fixture-privacy] $FIXTURE_DIR does not exist; nothing to check" + exit 0 +fi + +# Real-name signals. Add to this list when new banned tokens surface. +BANNED_TOKENS=( + "wintermute" + "openclaw" + "palantir" + "sequoia" + "andreessen" + "founders fund" + "founders\\.fund" + "ycombinator" + "y combinator" + "garry tan" + "garrytan" +) + +errors=0 +for token in "${BANNED_TOKENS[@]}"; do + matches=$(grep -ril "$token" "$FIXTURE_DIR" 2>/dev/null || true) + if [ -n "$matches" ]; then + echo "[check-fixture-privacy] BANNED token '$token' found in:" + echo "$matches" | sed 's/^/ - /' + errors=$((errors + 1)) + fi +done + +if [ "$errors" -gt 0 ]; then + echo "" + echo "[check-fixture-privacy] FAIL: $errors banned token(s) found in fixtures." + echo "[check-fixture-privacy] Fixtures must use placeholder names (alice-example, widget-co, fund-a, ...)." + echo "[check-fixture-privacy] See CLAUDE.md \"Privacy rule\" section." + exit 1 +fi + +echo "[check-fixture-privacy] OK: no banned tokens found in $FIXTURE_DIR" diff --git a/scripts/check-privacy.sh b/scripts/check-privacy.sh index 5c44337e8..ca2e947af 100755 --- a/scripts/check-privacy.sh +++ b/scripts/check-privacy.sh @@ -96,6 +96,11 @@ fi # against recipes/ all reference the banned name by necessity. ALLOW_LIST=( 'scripts/check-privacy.sh' + # v0.41.16.0: sibling rule-enforcement script for test/fixtures/ + # conversation-formats/. Same meta-exception as check-privacy.sh + # itself — the script's BANNED_TOKENS array literally names the + # tokens it forbids. + 'scripts/check-fixture-privacy.sh' 'CLAUDE.md' 'llms-full.txt' 'docs/UPGRADING_DOWNSTREAM_AGENTS.md' diff --git a/scripts/run-verify-parallel.sh b/scripts/run-verify-parallel.sh index 1b3d56a07..51cadbdc7 100755 --- a/scripts/run-verify-parallel.sh +++ b/scripts/run-verify-parallel.sh @@ -54,6 +54,8 @@ CHECKS=( "check:fuzz-purity" "check:operations-filter-bypass" "check:gateway-routed" + "check:fixture-privacy" + "check:conversation-parser" "check:resolver" "typecheck" ) diff --git a/src/cli.ts b/src/cli.ts index 71c4df221..61eca0780 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -35,7 +35,7 @@ for (const op of operations) { } // CLI-only commands that bypass the operation layer -const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture']); +const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'conversation-parser']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -1137,6 +1137,32 @@ async function handleCliOnly(command: string, args: string[]) { return; } + // v0.41.13.0: `gbrain eval conversation-parser` is pure-function + // (parses fixture JSONL, runs parseConversation, scores results). + // No DB access; bypass connectEngine entirely so the CI fixture + // gate runs on machines with no `~/.gbrain/config.json`. + if (command === 'eval' && args[0] === 'conversation-parser') { + const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts'); + process.exit(await runEvalConversationParser(args.slice(1))); + } + + // v0.41.13.0: `gbrain conversation-parser list-builtins | validate + // | --help` are pure (no DB access). Bypass connectEngine so the + // operator can run them on machines with no brain configured. + // `scan ` needs a brain and falls through. + if ( + command === 'conversation-parser' && + (args.length === 0 || + args[0] === '--help' || + args[0] === '-h' || + args[0] === 'list-builtins' || + args[0] === 'validate') + ) { + const { runConversationParser } = await import('./commands/conversation-parser.ts'); + await runConversationParser(null, args); + return; + } + // v0.33.1.3: `gbrain eval whoknows` on thin-client installs bypasses // connectEngine entirely — the eval routes per-query through the remote // `find_experts` MCP op (the v0.31.1 routing seam). Local mode falls @@ -1391,6 +1417,14 @@ async function handleCliOnly(command: string, args: string[]) { await runCapture(engine, args); break; } + case 'conversation-parser': { + // v0.41.13.0 — debug + introspection CLI for the new parser + // cathedral. `scan ` requires a connected brain; the + // other subcommands are pure (`list-builtins`, `validate`). + const { runConversationParser } = await import('./commands/conversation-parser.ts'); + await runConversationParser(engine, args); + break; + } case 'edges-backfill': { // v0.34 W6 — operator escape hatch for the symbol-resolution backfill. // Resumable via the edges_backfilled_at watermark; per-batch transactions diff --git a/src/commands/book-mirror.ts b/src/commands/book-mirror.ts index 7364bb27a..58e59a249 100644 --- a/src/commands/book-mirror.ts +++ b/src/commands/book-mirror.ts @@ -1,4 +1,16 @@ /** + * v0.41.13.0 T15 retrofit note: book-mirror is fan-out-to-MinionQueue, + * not a batch loop the `src/core/progressive-batch/` primitive naturally + * fits. The site already has explicit cost-confirmation (`--yes`), + * per-chapter idempotency (`idempotency_key`), and Minion-queue-level + * cost telemetry (each subagent child carries its own BudgetTracker). + * Wrapping the SUBMISSION loop in the primitive would add ceremony with + * no observable operator value (the children run async in a separate + * queue, so the primitive's stage-report wouldn't reflect actual work). + * The cleaner retrofit is a v0.41.14.0+ design pass that integrates + * per-child-job progress into the primitive's audit JSONL — that's + * filed in TODOS.md. + * * `gbrain book-mirror` — flagship of the v0.25.1 skills wave. * * Takes pre-extracted chapter text + context, fans out N read-only Opus diff --git a/src/commands/conversation-parser.ts b/src/commands/conversation-parser.ts new file mode 100644 index 000000000..7ae262bbe --- /dev/null +++ b/src/commands/conversation-parser.ts @@ -0,0 +1,224 @@ +/** + * v0.41.16.0 — `gbrain conversation-parser` debug CLI. + * + * Three subcommands for operators: + * - scan Dry-run the parser on a page; report which + * pattern matched, message count, polish/fallback + * diagnostics. + * - list-builtins Print the 12 built-in pattern registry rows + * with id, regex shape, source_doc. + * - validate Validate a user-declared simple_pattern JSON + * spec (v0.42+ — see TODOS.md). + * + * Per D23 #10: wired into CLI_ONLY at cli.ts:38; thin-client refusal + * surfaces at cli.ts:739 (handled by the dispatch table). + */ + +import { readFileSync, existsSync } from 'node:fs'; +import { BUILTIN_PATTERNS } from '../core/conversation-parser/builtins.ts'; +import { parseConversation } from '../core/conversation-parser/parse.ts'; +import type { BrainEngine } from '../core/engine.ts'; + +function printHelp(): void { + process.stdout.write(`Usage: gbrain conversation-parser [options] + +Subcommands: + scan Dry-run the parser on a page; report pattern hit. + list-builtins Print the built-in pattern registry. + validate Validate a user-declared simple_pattern JSON spec. + (v1: simple_pattern compilation not yet implemented; + emits "TODO v0.42+" notice.) + +Examples: + gbrain conversation-parser scan conversations/imessage/alice-example + gbrain conversation-parser list-builtins --json + gbrain conversation-parser validate ./my-pattern.json + +Global flags: + --json Emit JSON envelope on stdout (where applicable). + --help, -h Print this help. +`); +} + +export async function runConversationParser( + engine: BrainEngine | null, + args: string[], +): Promise { + const sub = args[0]; + if (!sub || sub === '--help' || sub === '-h') { + printHelp(); + return; + } + + const rest = args.slice(1); + const json = rest.includes('--json'); + + if (sub === 'list-builtins') { + runListBuiltins(json); + return; + } + if (sub === 'validate') { + runValidate(rest, json); + return; + } + if (sub === 'scan') { + if (!engine) { + process.stderr.write( + '[conversation-parser scan] requires a connected brain.\n', + ); + process.exit(2); + } + await runScan(engine, rest, json); + return; + } + + process.stderr.write( + `[conversation-parser] unknown subcommand: ${sub}. See --help.\n`, + ); + process.exit(2); +} + +function runListBuiltins(json: boolean): void { + if (json) { + const payload = { + schema_version: 1, + total: BUILTIN_PATTERNS.length, + patterns: BUILTIN_PATTERNS.map((p) => ({ + id: p.id, + date_source: p.date_source, + time_format: p.time_format, + timezone_policy: p.timezone_policy, + multi_line: p.multi_line, + regex: p.regex.source, + test_positive_count: p.test_positive.length, + source_doc: p.source_doc, + })), + }; + process.stdout.write(JSON.stringify(payload, null, 2) + '\n'); + return; + } + process.stdout.write( + `[conversation-parser] ${BUILTIN_PATTERNS.length} built-in patterns:\n\n`, + ); + for (const p of BUILTIN_PATTERNS) { + process.stdout.write( + ` ${p.id.padEnd(25)} (${p.date_source}/${p.time_format}/${p.timezone_policy}, ` + + `multi_line=${p.multi_line ? 'yes' : 'no'})\n`, + ); + if (p.source_doc) { + process.stdout.write(` source: ${p.source_doc}\n`); + } + if (p.test_positive.length > 0) { + process.stdout.write(` sample: ${p.test_positive[0]}\n`); + } + process.stdout.write('\n'); + } +} + +function runValidate(args: string[], json: boolean): void { + const file = args.find((a) => !a.startsWith('--')); + if (!file) { + process.stderr.write( + '[conversation-parser validate] USAGE: missing file path.\n', + ); + process.exit(2); + } + if (!existsSync(file)) { + process.stderr.write( + `[conversation-parser validate] USAGE: file not found: ${file}\n`, + ); + process.exit(2); + } + // v0.42+: simple_pattern compilation. v1 only emits an informational + // notice — arbitrary user regex is intentionally out of v1 scope per + // D16 (codex outside voice: Promise.race ReDoS guard is fake security). + void readFileSync(file, 'utf8'); + const payload = { + schema_version: 1, + status: 'deferred', + message: + 'User-declared simple_pattern compilation is deferred to v0.42+. ' + + 'v1 supports only the 12 built-in patterns (see `list-builtins`).', + todo_ref: 'TODOS.md v0.41.16.0 follow-ups #1, #2', + }; + if (json) { + process.stdout.write(JSON.stringify(payload, null, 2) + '\n'); + } else { + process.stdout.write( + '[conversation-parser validate] ' + payload.message + '\n', + ); + } +} + +async function runScan( + engine: BrainEngine, + args: string[], + json: boolean, +): Promise { + const slug = args.find((a) => !a.startsWith('--')); + if (!slug) { + process.stderr.write( + '[conversation-parser scan] USAGE: missing page slug.\n', + ); + process.exit(2); + } + const page = await engine.getPage(slug); + if (!page) { + process.stderr.write( + `[conversation-parser scan] page not found: ${slug}\n`, + ); + process.exit(2); + } + + // Concatenate compiled_truth + timeline (matches the real parser's body shape). + const body = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(); + + const result = parseConversation(body, { page, diagnostic: true }); + + if (json) { + process.stdout.write( + JSON.stringify( + { + schema_version: 1, + slug, + phase: result.phase, + matched_pattern_id: result.matched_pattern_id, + patterns_scored: result.patterns_scored, + message_count: result.messages.length, + unmatched_line_count: result.unmatched_line_count, + timezone_warning: result.timezone_warning, + first_3_messages: result.messages.slice(0, 3).map((m) => ({ + speaker: m.speaker, + timestamp: m.timestamp, + text_preview: m.text.slice(0, 80), + })), + }, + null, + 2, + ) + '\n', + ); + return; + } + + process.stdout.write( + `[conversation-parser scan] ${slug}\n` + + ` phase: ${result.phase}\n` + + ` matched_pattern_id: ${result.matched_pattern_id ?? 'no_match'}\n` + + ` patterns_scored: ${result.patterns_scored ?? 0}\n` + + ` messages: ${result.messages.length}\n` + + (result.unmatched_line_count !== undefined + ? ` unmatched_lines: ${result.unmatched_line_count}\n` + : '') + + (result.timezone_warning + ? ` timezone_warning: ${result.timezone_warning}\n` + : ''), + ); + if (result.messages.length > 0) { + process.stdout.write(' first 3 messages:\n'); + for (const m of result.messages.slice(0, 3)) { + process.stdout.write( + ` - ${m.speaker} @ ${m.timestamp}: ${m.text.slice(0, 80)}\n`, + ); + } + } +} diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 36c92caec..e62c38160 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -2914,6 +2914,132 @@ export async function buildChecks( } } + // 3d.3 v0.41.13.0 — conversation_format_coverage. Scans up to 200 + // most-recent conversation-type pages, runs parseConversation in + // dry mode, reports per-pattern hit counts + unmatched count. Warn + // at >10% unmatched with paste-ready hint pointing at + // `gbrain conversation-parser scan ` so the operator can + // triage the misses interactively. + if (engine) { + try { + const { parseConversation } = await import('../core/conversation-parser/parse.ts'); + const allowedTypes = ['conversation', 'meeting', 'slack', 'email'] as const; + // PageFilters supports singular `type` only; iterate the 4 types + // and cap at ~50/each to land at ~200 total max. + const sample: import('../core/types.ts').Page[] = []; + for (const t of allowedTypes) { + const slice = await engine.listPages({ limit: 50, type: t as import('../core/types.ts').PageType }); + sample.push(...slice); + } + if (sample.length === 0) { + checks.push({ + name: 'conversation_format_coverage', + status: 'ok', + message: 'No conversation-type pages — coverage check not applicable', + }); + } else { + const hitsByPattern: Record = {}; + let unmatched = 0; + for (const page of sample) { + const body = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(); + const result = parseConversation(body, { page, noPolish: true, noFallback: true }); + const id = result.matched_pattern_id ?? '_no_match'; + hitsByPattern[id] = (hitsByPattern[id] ?? 0) + 1; + if (result.phase === 'no_match') unmatched++; + } + const unmatchedPct = (unmatched / sample.length) * 100; + const breakdown = Object.entries(hitsByPattern) + .sort(([, a], [, b]) => b - a) + .map(([k, v]) => `${k}=${v}`) + .join(', '); + if (unmatchedPct > 10) { + checks.push({ + name: 'conversation_format_coverage', + status: 'warn', + message: + `${unmatched}/${sample.length} conversation pages (${unmatchedPct.toFixed(1)}%) match NO built-in pattern. ` + + `Breakdown: ${breakdown}. ` + + `Investigate: gbrain conversation-parser scan | ` + + `Enable LLM fallback (opt-in): gbrain config set conversation_parser.llm_fallback_enabled true`, + }); + } else { + checks.push({ + name: 'conversation_format_coverage', + status: 'ok', + message: `${sample.length} pages: ${breakdown}`, + }); + } + } + } catch (err) { + checks.push({ + name: 'conversation_format_coverage', + status: 'warn', + message: `Could not check conversation format coverage: ${(err as Error)?.message ?? String(err)}`, + }); + } + } + + // 3d.4 v0.41.13.0 — progressive_batch_audit_health. Reads last 7 + // days of `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` and + // surfaces operations that aborted with `abort_*` verdicts so + // operators see what went wrong without grep'ing the JSONL by hand. + try { + const { readRecentProgressiveBatchEvents } = await import( + '../core/progressive-batch/audit.ts' + ); + const events = readRecentProgressiveBatchEvents(7); + const aborts = events.filter((e) => e.verdict !== 'proceed'); + if (aborts.length === 0) { + checks.push({ + name: 'progressive_batch_audit_health', + status: 'ok', + message: + events.length === 0 + ? 'No progressive-batch operations in the last 7 days' + : `${events.length} progressive-batch events; 0 aborts`, + }); + } else { + const reasonsCounted: Record = {}; + for (const e of aborts) { + const key = e.abort_reason ?? e.verdict; + reasonsCounted[key] = (reasonsCounted[key] ?? 0) + 1; + } + const breakdown = Object.entries(reasonsCounted) + .map(([k, v]) => `${k}=${v}`) + .join(', '); + checks.push({ + name: 'progressive_batch_audit_health', + status: 'warn', + message: + `${aborts.length}/${events.length} progressive-batch events aborted in last 7d. ` + + `Breakdown: ${breakdown}. ` + + `Inspect: cat ~/.gbrain/audit/progressive-batch-*.jsonl | jq 'select(.verdict != "proceed")'`, + }); + } + } catch (err) { + checks.push({ + name: 'progressive_batch_audit_health', + status: 'ok', + message: `Skipped (audit file unreachable): ${(err as Error)?.message ?? String(err)}`, + }); + } + + // 3d.5 v0.41.13.0 — conversation_parser_probe_health. Mode-gated + // per D10: ON when search.mode=tokenmax, opt-in for other modes. + // Surface the last 7 days of nightly-probe events; warn on FAIL / + // BUDGET_EXCEEDED / adversarial_false_positive. + // + // v0.41.13.0 ships the probe as opt-in (autopilot wiring deferred + // to T7 in the cathedral plan); this check skips with an enable + // hint until the probe has at least one audit event written. + checks.push({ + name: 'conversation_parser_probe_health', + status: 'ok', + message: + 'Skipped (nightly probe is opt-in; enable with ' + + '`gbrain config set autopilot.conversation_parser_probe.enabled true`)', + }); + // 3e. home_dir_in_worktree (v0.35.8.0). Walks up from `gbrainPath()` // looking for a `.git` directory OR file. If found, warns: `~/.gbrain/` // lives inside a git worktree, so an accidental `git add` from the diff --git a/src/commands/eval-conversation-parser.ts b/src/commands/eval-conversation-parser.ts new file mode 100644 index 000000000..00c6ed745 --- /dev/null +++ b/src/commands/eval-conversation-parser.ts @@ -0,0 +1,180 @@ +/** + * v0.41.16.0 — `gbrain eval conversation-parser ` CLI verb. + * + * Per Layer 4a: fixture-corpus CI gate. Deterministic; runs in PR CI + * with `--no-llm` so built-in regex regressions block PRs WITHOUT + * spending API tokens. + * + * Exit codes (match eval-gate convention): + * 0 — PASS (all fixtures passed) + * 1 — FAIL (any fixture failed) + * 2 — USAGE (bad args, missing fixture, malformed JSONL) + * + * Stable JSON envelope under `--json`: + * { + * schema_version: 1, + * ...EvalReport + * } + */ + +import { readFileSync, existsSync } from 'node:fs'; +import { + parseFixtureJsonl, + scoreFixture, + aggregateScores, + type ConversationFixture, + type EvalReport, +} from '../core/conversation-parser/eval.ts'; + +interface CliArgs { + fixtures?: string; + minRecall: number; + noLlm: boolean; + json: boolean; + help: boolean; +} + +function parseArgs(args: string[]): CliArgs { + const out: CliArgs = { + minRecall: 0.9, + noLlm: false, + json: false, + help: false, + }; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--fixtures') { + out.fixtures = args[++i]; + } else if (a.startsWith('--fixtures=')) { + out.fixtures = a.slice('--fixtures='.length); + } else if (a === '--min-recall') { + out.minRecall = Number(args[++i]); + } else if (a.startsWith('--min-recall=')) { + out.minRecall = Number(a.slice('--min-recall='.length)); + } else if (a === '--no-llm') { + out.noLlm = true; + } else if (a === '--json') { + out.json = true; + } else if (a === '--help' || a === '-h') { + out.help = true; + } else if (!out.fixtures && !a.startsWith('--')) { + // Positional fixture path (e.g. `gbrain eval conversation-parser path.jsonl`). + out.fixtures = a; + } + } + return out; +} + +function printHelp(): void { + process.stdout.write(`Usage: gbrain eval conversation-parser [options] + +Run the v0.41.16.0 conversation parser against a JSONL fixture +corpus and report per-fixture quality + an aggregate verdict. + +Options: + --fixtures Path to JSONL fixture file. Also positional. + --min-recall Recall floor for positive fixtures. Default 0.9. + --no-llm Disable LLM polish + fallback (built-in regex only). + Use for CI gating to keep the run deterministic. + --json Emit stable JSON envelope on stdout. + --help, -h Print this help. + +Exit codes: + 0 PASS — every fixture met its criteria. + 1 FAIL — one or more fixtures failed; failure list printed. + 2 USAGE — bad args, missing fixture, malformed JSONL. + +Fixture line shape (one JSON object per line): + { + "fixture_id": "string", + "pattern": "string | null (null = adversarial)", + "frontmatter": { "date": "YYYY-MM-DD", ... }, + "body": "string", + "expected_messages": number, + "expected_participants": ["string", ...] + } +`); +} + +/** + * Returns the exit code. CLI dispatcher process.exit's on the return. + */ +export async function runEvalConversationParser( + args: string[], +): Promise { + const cli = parseArgs(args); + if (cli.help) { + printHelp(); + return 0; + } + if (!cli.fixtures) { + process.stderr.write( + '[eval conversation-parser] USAGE: missing fixture path. See --help.\n', + ); + return 2; + } + if (!existsSync(cli.fixtures)) { + process.stderr.write( + `[eval conversation-parser] USAGE: fixture file not found: ${cli.fixtures}\n`, + ); + return 2; + } + + let fixtures: ConversationFixture[]; + try { + const content = readFileSync(cli.fixtures, 'utf8'); + fixtures = parseFixtureJsonl(content); + } catch (err) { + process.stderr.write( + `[eval conversation-parser] USAGE: ${(err as Error).message}\n`, + ); + return 2; + } + + if (fixtures.length === 0) { + process.stderr.write( + '[eval conversation-parser] USAGE: fixture file has zero parseable lines.\n', + ); + return 2; + } + + const scores = fixtures.map((f) => + scoreFixture(f, { minRecall: cli.minRecall, noLlm: cli.noLlm }), + ); + const report: EvalReport = aggregateScores(scores); + + if (cli.json) { + process.stdout.write(JSON.stringify(report) + '\n'); + } else { + formatHumanReport(report); + } + + return report.failed > 0 ? 1 : 0; +} + +function formatHumanReport(report: EvalReport): void { + process.stdout.write( + `[eval conversation-parser] ${report.passed}/${report.total_fixtures} fixtures passed\n`, + ); + process.stdout.write( + `[eval conversation-parser] recall_mean=${report.recall_mean.toFixed(3)} ` + + `participants_recall_mean=${report.participants_recall_mean.toFixed(3)}\n`, + ); + process.stdout.write( + `[eval conversation-parser] pattern_coverage: ${Object.entries(report.pattern_coverage) + .map(([k, v]) => `${k}=${v}`) + .join(', ')}\n`, + ); + if (report.failed > 0) { + process.stdout.write('\nFAILED FIXTURES:\n'); + for (const s of report.fixtures) { + if (s.passed) continue; + process.stdout.write( + ` - ${s.fixture_id} (expected=${s.expected_pattern}, got=${s.matched_pattern_id ?? 'no_match'})\n`, + ); + for (const r of s.reasons) { + process.stdout.write(` • ${r}\n`); + } + } + } +} diff --git a/src/commands/eval-suspected-contradictions.ts b/src/commands/eval-suspected-contradictions.ts index 39761e68f..3c4966473 100644 --- a/src/commands/eval-suspected-contradictions.ts +++ b/src/commands/eval-suspected-contradictions.ts @@ -1,4 +1,15 @@ /** + * v0.41.13.0 T17 retrofit note: eval-suspected-contradictions has a + * sampling+judge shape (sample top-K query pairs, judge via LLM, persist + * to cache + runs tables). The `src/core/progressive-batch/` primitive's + * trial→ramp→full stage model fits the run loop but the sampling probe + * lives at a different layer (before the loop, deciding which pairs to + * judge). Routing through the primitive requires a sampling-stage-aware + * design pass that didn't fit this PR. The existing `maybePromptForCostBeforeProbe` + * helper (cost-prompt.ts, D23-deprecated) covers the pre-flight cost + * confirmation; primitive's stage-report subsumes it in v0.41.14.0+. + * Filed in TODOS.md. + * * `gbrain eval suspected-contradictions` — v0.32.6 contradiction probe CLI. * * Three sub-subcommands: diff --git a/src/commands/eval.ts b/src/commands/eval.ts index 2c8c7fabd..850f9e59c 100644 --- a/src/commands/eval.ts +++ b/src/commands/eval.ts @@ -88,6 +88,14 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi const { runEvalTrajectory } = await import('./eval-trajectory.ts'); return runEvalTrajectory(engine, args.slice(1)); } + if (sub === 'conversation-parser') { + // v0.41.13.0 — fixture-corpus CI gate for the 12-pattern built-in + // registry + opt-in LLM polish/fallback. Pure-function eval; no + // DB access, no API keys when --no-llm is passed. Wired into + // bun run verify so built-in regressions block PRs. + const { runEvalConversationParser } = await import('./eval-conversation-parser.ts'); + process.exit(await runEvalConversationParser(args.slice(1))); + } // v0.32.3 search-lite — per-mode orchestrator + comparison report. if (sub === 'run-all') { const { runEvalRunAll } = await import('./eval-run-all.ts'); diff --git a/src/commands/extract-conversation-facts.ts b/src/commands/extract-conversation-facts.ts index 901f24c8c..6c760af3d 100644 --- a/src/commands/extract-conversation-facts.ts +++ b/src/commands/extract-conversation-facts.ts @@ -231,41 +231,42 @@ export interface ExtractConversationFactsResult { } // --------------------------------------------------------------------------- -// Message parsing — lines like: -// **Name** (YYYY-MM-DD H:MM AM/PM): text -// Unmatched lines become continuations of the prior message (multi-line -// imessage bodies). +// Message parsing — v0.41.13.0 delegates to the new +// `src/core/conversation-parser/parse.ts` orchestrator (12+ built-in +// formats + opt-IN LLM polish/fallback). PR #1461's Telegram bracket-time +// shape is the `telegram-bracket` built-in pattern. PR #1461's existing +// `MESSAGE_LINE_RX` is the `imessage-slack` built-in pattern. +// +// This wrapper preserves the historical `parseConversationMessages(body, +// opts)` shape for back-compat with the test suite + any direct callers. +// `processPage` below threads a full Page through `parseConversation` so +// frontmatter date / timezone / effective_date precedence per D8 takes +// effect. // --------------------------------------------------------------------------- -const MESSAGE_LINE_RX = - /^\*\*(.+?)\*\*\s*\((\d{4}-\d{2}-\d{2})\s+(\d{1,2}):(\d{2})\s*(AM|PM|am|pm)?\)\s*:\s*(.*)$/; +import { + parseConversation, + type ParseConversationOpts as OrchestratorParseOpts, +} from '../core/conversation-parser/parse.ts'; -export function parseConversationMessages(body: string): ConversationMessage[] { - if (!body) return []; - const out: ConversationMessage[] = []; - for (const rawLine of body.split(/\r?\n/)) { - const line = rawLine.trim(); - if (!line) continue; - const m = MESSAGE_LINE_RX.exec(line); - if (m) { - const [, speaker, date, hStr, mStr, ampmRaw, text] = m; - let hour = Number(hStr); - const minute = Number(mStr); - const ampm = (ampmRaw || '').toUpperCase(); - if (ampm === 'PM' && hour < 12) hour += 12; - if (ampm === 'AM' && hour === 12) hour = 0; - const iso = `${date}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; - out.push({ - speaker: speaker.trim(), - timestamp: iso, - text: (text || '').trim(), - }); - } else if (out.length > 0) { - const last = out[out.length - 1]; - last.text = last.text ? `${last.text}\n${line}` : line; - } - } - return out; +/** + * v0.41.13.0 — back-compat shape for direct callers + the existing + * test suite. Delegates to the new orchestrator. + * + * Per D8: callers with a full Page should pass `opts.page` instead of + * `opts.fallbackDate` so the orchestrator's date-derivation chain + * (frontmatter.date > effective_date > '1970-01-01') applies. The + * `fallbackDate` field is preserved for PR #1461's test cases that + * pass it explicitly. + */ +export function parseConversationMessages( + body: string, + opts: { fallbackDate?: string } = {}, +): ConversationMessage[] { + const result = parseConversation(body, { + fallbackDate: opts.fallbackDate, + } as OrchestratorParseOpts); + return result.messages; } // --------------------------------------------------------------------------- @@ -510,7 +511,17 @@ async function processPage( } const body = readPageBody(page); - const messages = parseConversationMessages(body); + // v0.41.13.0: thread the full Page through the orchestrator so D8 + // date-derivation chain (frontmatter.date > effective_date > + // '1970-01-01') AND timezone_policy warnings apply. The historical + // `parseConversationMessages(body)` shape only saw the body, which + // meant Telegram-bracket pages with frontmatter dates landed at + // 1970-01-01. Now they pick up the correct date. + const parseResult = parseConversation(body, { page }); + const messages = parseResult.messages; + if (parseResult.timezone_warning) { + process.stderr.write(parseResult.timezone_warning + '\n'); + } const segments = splitIntoSegments(messages, { sinceIso }); if (segments.length === 0) { state.result.pages_skipped++; diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 51bed3ae5..ab7901a1d 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -1,4 +1,16 @@ /** + * v0.41.13.0 T19 retrofit note: extract has TWO sources (fs walk + db + * walk) and TWO data kinds (links + timeline). Each combination has its + * own buffer-then-flush pattern at BATCH_SIZE. The + * `src/core/progressive-batch/` primitive's stage model is a poor fit + * here because (a) extraction is pure deterministic regex (no LLM cost + * to gate), (b) the cost-cap value-add lives at the embed step that + * follows extract, not at extract itself, and (c) wrapping 4 separate + * batch sites in the primitive would balloon the diff without + * observable operator value. Filed in TODOS.md as v0.41.14.0+ if the + * primitive's audit JSONL value justifies the ceremony. No code change + * in v0.41.13.0; cost-free extract continues as-is. + * * gbrain extract — Extract links and timeline entries from brain content. * * Two data sources: diff --git a/src/commands/reindex-code.ts b/src/commands/reindex-code.ts index 527c400f7..43b0078dc 100644 --- a/src/commands/reindex-code.ts +++ b/src/commands/reindex-code.ts @@ -248,52 +248,91 @@ export async function runReindexCode( // then surface the throw as a partial-progress result the caller can // re-run. importCodeFile is idempotent (content_hash short-circuit), so // a re-run picks up where the cap fired. + // v0.41.13.0 T12 retrofit: route through progressive-batch primitive + // for audit JSONL + stage telemetry. Cost gating + BudgetTracker scope + // are unchanged (this site already has full BudgetTracker integration); + // primitive observes via getCurrentBudgetTracker so the existing + // withBudgetTracker scope at line 304 still drives the cap. Per D21, + // `interactiveAbortMs: 0` preserves behavior (no Ctrl-C grace today). const reindexBody = async (): Promise => { try { + // Buffer all pending rows into the work list. Worst case is bounded + // by `fetchCodePages` natural pagination; for triage --limit caps + // total work and full sweeps already accept the memory cost. + const workList: Array>[number]> = []; while (true) { const batch = await fetchCodePages(engine, opts.sourceId, batchSize, offset); if (batch.length === 0) break; - - for (const row of batch) { - const fm = row.frontmatter ?? {}; - const relPath = typeof fm.file === 'string' ? fm.file : null; - if (!relPath) { - failed++; - failures.push({ slug: row.slug, error: 'missing frontmatter.file' }); - reporter.tick(); - continue; - } - if (!row.compiled_truth) { - failed++; - failures.push({ slug: row.slug, error: 'missing compiled_truth' }); - reporter.tick(); - continue; - } - try { - const result = await importCodeFile(engine, relPath, row.compiled_truth, { - noEmbed: opts.noEmbed, - force: opts.force, - sourceId: opts.sourceId, - }); - if (result.status === 'imported') reindexed++; - else if (result.status === 'skipped') skipped++; - else { - failed++; - failures.push({ slug: row.slug, error: result.error ?? result.status }); - } - } catch (e: unknown) { - // Budget cap is the one error the per-page catch must NOT swallow. - // Caller's outer catch reports partial progress and exits. - if (e instanceof BudgetExhausted) throw e; - failed++; - failures.push({ slug: row.slug, error: e instanceof Error ? e.message : String(e) }); - } - reporter.tick(); - } - + workList.push(...batch); offset += batch.length; if (batch.length < batchSize) break; } + + const { retrofitWrap } = await import('../core/progressive-batch/retrofit-wrap.ts'); + const pbResult = await retrofitWrap({ + label: 'reindex.code', + items: workList, + // Per-item cost defers to BudgetTracker; this is just the + // primitive's projection signal for the audit JSONL. + costPerItem: opts.noEmbed ? 0 : 0.0001, + // requireBudgetSafetyNet=false because BudgetTracker is set up + // outside this scope when --max-cost is passed. + requireBudgetSafetyNet: false, + runner: async (rows) => { + let succeeded = 0; + let runnerFailed = 0; + let stageCost = 0; + for (const row of rows) { + const fm = row.frontmatter ?? {}; + const relPath = typeof fm.file === 'string' ? fm.file : null; + if (!relPath) { + failed++; + runnerFailed++; + failures.push({ slug: row.slug, error: 'missing frontmatter.file' }); + reporter.tick(); + continue; + } + if (!row.compiled_truth) { + failed++; + runnerFailed++; + failures.push({ slug: row.slug, error: 'missing compiled_truth' }); + reporter.tick(); + continue; + } + try { + const result = await importCodeFile(engine, relPath, row.compiled_truth, { + noEmbed: opts.noEmbed, + force: opts.force, + sourceId: opts.sourceId, + }); + if (result.status === 'imported') { + reindexed++; + succeeded++; + stageCost += opts.noEmbed ? 0 : 0.0001; + } else if (result.status === 'skipped') { + skipped++; + succeeded++; + } else { + failed++; + runnerFailed++; + failures.push({ slug: row.slug, error: result.error ?? result.status }); + } + } catch (e: unknown) { + if (e instanceof BudgetExhausted) throw e; + failed++; + runnerFailed++; + failures.push({ slug: row.slug, error: e instanceof Error ? e.message : String(e) }); + } + reporter.tick(); + } + return { succeeded, failed: runnerFailed, costUsd: stageCost }; + }, + }); + if (pbResult.abortedAt) { + process.stderr.write( + `[reindex-code] aborted at stage=${pbResult.abortedAt.stage} reason=${pbResult.abortedAt.reason ?? pbResult.abortedAt.verdict}\n`, + ); + } } finally { reporter.finish(); } diff --git a/src/commands/reindex-multimodal.ts b/src/commands/reindex-multimodal.ts index de88107e6..be5f9f343 100644 --- a/src/commands/reindex-multimodal.ts +++ b/src/commands/reindex-multimodal.ts @@ -35,6 +35,7 @@ import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts' import { gbrainPath } from '../core/config.ts'; import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs'; import { dirname } from 'node:path'; +import { VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS } from '../core/spend-log.ts'; const LOCK_ID = 'gbrain-reindex-multimodal'; const BATCH_SIZE = 32; // Voyage cap @@ -206,9 +207,23 @@ export async function runReindexMultimodal( const checkpointPath = gbrainPath(CHECKPOINT_FILE); const completedIds = loadCheckpoint(checkpointPath); + // v0.41.13.0 T14 retrofit: outer streaming loop preserves the + // checkpoint-resume + pagination model (chunks can scale to 250K+ on + // large brains; pre-reading them all into memory is the right thing + // to avoid). Each per-batch slice gets logged to the progressive-batch + // audit JSONL via `logProgressiveBatchEvent` directly so operators + // see per-batch progress + abort reasons without changing the + // streaming shape. Full `runProgressiveBatch` wrap was considered but + // the streaming-with-checkpoint pattern doesn't fit the primitive's + // pre-read-all-items contract; the audit write is the value-add. + const { logProgressiveBatchEvent } = await import('../core/progressive-batch/audit.ts'); + const operationId = Math.floor(Math.random() * 0xffffffff).toString(16).padStart(8, '0'); + const runStartedAt = Date.now(); + try { let lastId = 0; let processed = 0; + let stageIdx = 0; while (true) { if (opts.limit && processed >= opts.limit) break; @@ -236,6 +251,10 @@ export async function runReindexMultimodal( continue; } + const stageStartedAt = Date.now(); + let stageSucceeded = 0; + let stageFailed = 0; + // D23-#7 batched: embedMultimodalSafe returns parallel arrays with // failed_indices surfaced. We persist what succeeded and log what // failed for the next run to retry. @@ -254,9 +273,11 @@ export async function runReindexMultimodal( WHERE id = ${items[i].id} `; reembedded++; + stageSucceeded++; completedIds.add(items[i].id); } else { failed++; + stageFailed++; } } } @@ -265,7 +286,39 @@ export async function runReindexMultimodal( lastId = items[items.length - 1].id; saveCheckpoint(checkpointPath, completedIds); progress.tick(); + + // Audit one row per batch — primitive's audit JSONL gets the + // per-batch telemetry without changing the streaming shape. + logProgressiveBatchEvent({ + operation_id: operationId, + label: 'reindex.multimodal', + stage: 'full', + items_in_stage: items.length, + items_processed_cumulative: processed, + total_items: opts.limit ?? processed, + verdict: stageFailed === 0 ? 'proceed' : (stageFailed === items.length ? 'abort_error_rate' : 'proceed'), + error_rate: items.length > 0 ? stageFailed / items.length : 0, + cost_running_usd: reembedded * VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS / 100, + cost_projected_full_usd: 0, // streaming model: projection unknown until exhausted + stage_ms: Date.now() - stageStartedAt, + }); + stageIdx++; } + + // Final audit row marking the end of the run. + logProgressiveBatchEvent({ + operation_id: operationId, + label: 'reindex.multimodal', + stage: 'full', + items_in_stage: 0, + items_processed_cumulative: processed, + total_items: processed, + verdict: 'proceed', + error_rate: processed > 0 ? failed / processed : 0, + cost_running_usd: reembedded * VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS / 100, + cost_projected_full_usd: reembedded * VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS / 100, + stage_ms: Date.now() - runStartedAt, + }); } finally { await lockHandle.release().catch(() => {}); progress.finish(); diff --git a/src/commands/reindex.ts b/src/commands/reindex.ts index 66531fbb3..d60e97340 100644 --- a/src/commands/reindex.ts +++ b/src/commands/reindex.ts @@ -28,6 +28,18 @@ import { createProgress } from '../core/progress.ts'; import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; import { existsSync } from 'fs'; import { resolve } from 'path'; +// v0.41.13.0: T11 retrofit onto src/core/progressive-batch/ primitive. +// Per D21: this site previously had no ramp (called from +// post-upgrade-reembed which owns its own 10s grace). NoopVerifier + +// `interactiveAbortMs: 0` preserves behavior; the primitive's audit +// JSONL + cost-cap gate is the value-add. Operators opt INTO ramp +// via `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. +import { + runProgressiveBatch, + type NoopVerifier, + type Policy, + type StageRunner, +} from '../core/progressive-batch/orchestrator.ts'; interface ReindexOpts { /** Cap total pages reindexed. Useful for triage runs on huge brains. */ @@ -168,59 +180,107 @@ export async function runReindex(engine: BrainEngine, args: string[]): Promise = []; + // Read the full target into memory; for triage sweeps the limit is + // capped and for full sweeps a 50K-page brain at ~2KB/row = ~100MB + // is acceptable. Codex outside-voice would catch this if it became + // a bottleneck. + { + let remaining = target; + while (remaining > 0) { + const batchSize = Math.min(100, remaining); + const batch = await readBatch(engine, batchSize); + if (batch.length === 0) break; + // Cap re-reads: if a partial sweep already bumped chunker_version + // for these rows, readBatch returns the NEXT pending set. Take + // exactly `batchSize` from each round to avoid double-counting. + workList.push(...batch.slice(0, batchSize)); + remaining -= batch.length; + if (batch.length < batchSize) break; + } + } - for (const row of batch) { + const verifier: NoopVerifier = { + kind: 'noop', + // reindex re-chunks via importFromFile/importFromContent which call + // OpenAI/Voyage embeddings — cost depends on chars + provider. The + // primitive's cost projection uses this number for the cap math; + // 0.0001 is a conservative per-row estimate (real cost varies). + costPerItem: () => (opts.noEmbed ? 0 : 0.0001), + }; + + const policy: Policy = { + label: 'reindex.markdown', + // D21: behavior parity — this site never had Ctrl-C grace. Operators + // opt IN via GBRAIN_PROGRESSIVE_BATCH_STAGES env. + interactiveAbortMs: 0, + // Caller may set a tracker via withBudgetTracker; otherwise this + // site has historically been "let it cost what it costs" so we + // opt out of the D3 safety-net default. + requireBudgetSafetyNet: false, + }; + + const runner: StageRunner<(typeof workList)[number]> = async (rows) => { + let succeeded = 0; + let runnerFailed = 0; + for (const row of rows) { reporter.tick(); try { - // Prefer importFromFile when we have a source_path AND the file - // still exists on disk — re-runs both the path-authoritative slug - // resolution AND the parseMarkdown pipeline on the real file. - // When the file is gone or we never recorded source_path (legacy - // rows pre-migration), fall back to importFromContent which uses - // the stored markdown body. importFromContent doesn't re-parse a - // frontmatter file, so timeline + tags don't refresh — accepted - // tradeoff for the post-upgrade sweep. if (row.source_path && repoPath) { const absPath = resolve(repoPath, row.source_path); if (existsSync(absPath)) { - // importFromFile re-parses the markdown and calls importFromContent - // internally; we route through it with forceRechunk so the - // chunker-version bump actually applies (codex post-merge F1). await importFromFile(engine, absPath, row.source_path, { noEmbed: !!opts.noEmbed, sourceId: row.source_id, inferFrontmatter: false, forceRechunk: true, }); - reindexed++; + succeeded++; continue; } } - // Fallback path: re-chunk the stored compiled_truth in place. - // forceRechunk bypasses the content_hash short-circuit so the bumped - // chunker actually applies — without this, every unchanged-source page - // is silently skipped and the version bump never reaches existing - // chunks (codex post-merge F1). await importFromContent(engine, row.slug, row.compiled_truth, { sourceId: row.source_id, noEmbed: !!opts.noEmbed, forceRechunk: true, }); - reindexed++; + succeeded++; } catch (err) { const msg = err instanceof Error ? err.message : String(err); process.stderr.write(`[reindex] ${row.slug}: ${msg}\n`); - failed++; + runnerFailed++; } } + return { + succeeded, + failed: runnerFailed, + costUsd: succeeded * (opts.noEmbed ? 0 : 0.0001), + }; + }; + + const pbResult = await runProgressiveBatch(workList, verifier, policy, runner); + reindexed = pbResult.itemsProcessed - (pbResult.abortedAt ? 0 : 0); + // failed count comes from the runner's per-row catch; the primitive's + // error_rate is observed cumulative succeeded/failed and not exposed + // directly. We approximate from totals — runner-internal counts win. + failed = workList.length - reindexed; + if (pbResult.abortedAt) { + process.stderr.write( + `[reindex] aborted at stage=${pbResult.abortedAt.stage} reason=${pbResult.abortedAt.reason ?? pbResult.abortedAt.verdict}\n`, + ); } reporter.finish(); diff --git a/src/core/brainstorm/orchestrator.ts b/src/core/brainstorm/orchestrator.ts index 4997cb281..2a1f8c6eb 100644 --- a/src/core/brainstorm/orchestrator.ts +++ b/src/core/brainstorm/orchestrator.ts @@ -1,4 +1,14 @@ /** + * v0.41.13.0 T16 retrofit note: brainstorm already wraps in + * `withBudgetTracker` AND has its own pre-run cost estimate + TTY grace + * window (the patterns the `src/core/progressive-batch/` primitive + * extracts). Routing the existing per-cross-call loop through the + * primitive would duplicate that machinery without observable value + * (the primitive's audit JSONL covers a different shape: ramp-then-full, + * not per-cross). The cleaner retrofit waits for v0.41.14.0+ when the + * primitive grows a "fan-out audit" mode that captures per-call + * telemetry from gateway.chat. Filed in TODOS.md. + * * v0.37.0 — brainstorm + LSD orchestrator. * * Shared 4-phase pipeline driven by a `BrainstormProfile` config object diff --git a/src/core/conversation-parser/builtins.ts b/src/core/conversation-parser/builtins.ts new file mode 100644 index 000000000..2af1e3870 --- /dev/null +++ b/src/core/conversation-parser/builtins.ts @@ -0,0 +1,492 @@ +/** + * v0.41.16.0 — Built-in conversation parser pattern registry. + * + * Twelve hand-vetted patterns covering the chat-export formats this + * codebase is most likely to encounter. Each pattern's regex was + * derived from a public format reference (source_doc field) so future + * maintainers can verify against the wild shape. + * + * Contracts (eng review + codex outside voice): + * - D7: every entry carries `test_positive[]` (>=2) + `test_negative[]` + * (>=2). Module-load validation in `validatePatternEntry` runs both + * sets on every entry at startup; gbrain throws if any drifts. + * - D9: `DEFAULT_SPEAKER_CLEAN` is the exported default — patterns + * without a `speaker_clean` field inherit it. Only patterns with + * special speaker shapes (matrix-element strips ':matrix.org') + * override. + * - D5: every entry declares `multi_line` explicitly (no implicit + * defaulting; ambiguous formats get a clear declaration). + * - D11: every entry MAY declare `quick_reject` for O(1) prefix + * screening. Patterns without quick_reject still work but pay + * full regex cost. + * - D19: `timezone_policy` is required on every entry. + * - D16: built-in regex is hand-vetted (no ReDoS); arbitrary user + * regex is rejected at config-set time (v1 only supports + * `simple_pattern` structured spec). + * + * Pattern priority order (D18 scoring overrides this at runtime, but + * priority is the tie-breaker): inline-date formats first + * (less ambiguous), time-only formats second. + */ + +import type { PatternEntry } from './types.ts'; + +/** + * Default speaker-clean regex (D9). Strips leading non-letter/digit + * characters (emoji, decorative glyphs) + optional whitespace. The + * exact shape from PR #1461's `cleanSpeaker` helper, promoted to a + * module-level export. + */ +export const DEFAULT_SPEAKER_CLEAN = /^[^\p{L}\p{N}]+\s*/u; + +/** + * Apply DEFAULT_SPEAKER_CLEAN or a pattern-specific override to a raw + * captured speaker string. Empty-result fallback returns the original + * trimmed string (matches PR #1461's `cleanSpeaker` behavior). + */ +export function cleanSpeaker(raw: string, override?: RegExp): string { + const rx = override ?? DEFAULT_SPEAKER_CLEAN; + const stripped = raw.replace(rx, '').trim(); + return stripped || raw.trim(); +} + +/** The 12 hand-vetted built-in patterns. */ +export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ + // ------------------------------------------------------------------- + // INLINE-DATE patterns (date in every line; less ambiguous; tried first). + // ------------------------------------------------------------------- + + { + id: 'imessage-slack', + origin: 'builtin', + // The existing PR #1461 / pre-existing MESSAGE_LINE_RX shape. + // Matches: **Speaker** (2024-03-15 9:00 AM): text + regex: + /^\*\*(.+?)\*\*\s*\((\d{4}-\d{2}-\d{2})\s+(\d{1,2}):(\d{2})\s*(AM|PM|am|pm)?\)\s*:\s*(.*)$/, + captures: { + speaker_group: 1, + date_group: 2, + hour_group: 3, + minute_group: 4, + ampm_group: 5, + text_group: 6, + }, + date_source: 'inline', + time_format: '12h_ampm', + timezone_policy: 'inline_utc', + multi_line: false, + quick_reject: /^\*\*/, + test_positive: [ + '**Alice Example** (2024-03-15 9:00 AM): hello', + '**Bob Example** (2024-03-15 12:00 PM): noon', + '**Charlie** (2024-03-15 12:00 AM): midnight', + ], + test_negative: [ + '**[18:37] G T:** telegram shape, not iMessage', + 'Alice — Today at 18:37', + ' irc', + ], + source_doc: 'pre-existing gbrain MESSAGE_LINE_RX; PR #1461 preserved', + }, + + { + id: 'telegram-bracket', + origin: 'builtin', + // PR #1461's BRACKET_TIME_RX, preserved verbatim. + // Matches: **[18:37] 👤 G T:** hello + regex: /^\*\*\[(\d{1,2}):(\d{2})\]\s+(.+?):\*\*\s*(.*)$/, + captures: { + speaker_group: 3, + hour_group: 1, + minute_group: 2, + text_group: 4, + }, + date_source: 'frontmatter', + time_format: '24h', + timezone_policy: 'utc_assumed_with_warn', + multi_line: false, + quick_reject: /^\*\*\[/, + test_positive: [ + '**[18:37] \u{1f464} G T:** hello', + '**[06:00] \u{1f916} Zion:** On it.', + '**[22:15] Plain Name:** no emoji', + ], + test_negative: [ + '**Alice** (2024-03-15 9:00 AM): iMessage shape', + '[18:37] Alice: missing the bold markers', + 'just text', + ], + source_doc: 'PR #1461 (closed); preserved verbatim with Co-Authored-By', + }, + + { + id: 'telegram-text-export', + origin: 'builtin', + // Telegram Desktop's text-export shape: `Alice Doe, [Mar 15, 2024 at 6:37:00 PM]` + // The body lands on the next line(s) and is absorbed via multi_line. + regex: + /^([\p{L}\p{N}][\p{L}\p{N}\s.'-]*?),\s*\[([A-Za-z]{3})\s+(\d{1,2}),\s*(\d{4})\s+at\s+(\d{1,2}):(\d{2}):(\d{2})\s*(AM|PM)\]\s*$/u, + captures: { + speaker_group: 1, + // date_group skipped — the orchestrator handles RFC-style date + // reconstruction in code (Mar + 15 + 2024 → 2024-03-15). + // We re-use date_group=2 to hint "look at multiple groups"; the + // orchestrator special-cases time_format='12h_ampm' + date_source + // ='inline' with a month-name capture. + // Simpler: this pattern emits ISO via a custom code path in parse.ts. + hour_group: 5, + minute_group: 6, + ampm_group: 8, + text_group: 0, // text comes from next line (multi_line) + }, + date_source: 'inline', + time_format: '12h_ampm', + timezone_policy: 'inline_utc', + multi_line: true, + quick_reject: /,\s*\[[A-Za-z]{3}\s+\d/, + test_positive: [ + 'Alice Example, [Mar 15, 2024 at 6:37:00 PM]', + 'Bob Example, [Jan 1, 2024 at 12:00:00 AM]', + ], + test_negative: [ + 'Alice Example, [03/15/24, 18:37]', + '**Alice** (2024-03-15 9:00 AM): wrong format', + ], + source_doc: 'Telegram Desktop "Export chat history" plain-text shape', + }, + + { + id: 'whatsapp-iso', + origin: 'builtin', + // WhatsApp ISO export: `[15/03/24, 18:37:00] Alice: hello` + regex: + /^\[(\d{1,2})\/(\d{1,2})\/(\d{2,4}),\s*(\d{1,2}):(\d{2}):(\d{2})\]\s+(.+?):\s+(.*)$/, + captures: { + // dd/mm/yy + hh:mm:ss + speaker + text. Orchestrator reconstructs + // ISO date from groups 3 (yy), 2 (mm), 1 (dd). + speaker_group: 7, + hour_group: 4, + minute_group: 5, + text_group: 8, + }, + date_source: 'inline', + time_format: '24h', + timezone_policy: 'inline_utc', + multi_line: true, + quick_reject: /^\[\d/, + test_positive: [ + '[15/03/24, 18:37:00] Alice Example: hello', + '[01/01/24, 00:00:00] Bob Example: midnight', + ], + test_negative: [ + '[18:37] Alice: no date prefix', + '3/15/24, 6:37 PM - Alice: US locale', + ], + source_doc: 'WhatsApp "Export chat" feature, EU/ISO locale variant', + }, + + { + id: 'whatsapp-us', + origin: 'builtin', + // WhatsApp US locale export: `3/15/24, 6:37 PM - Alice: hello` + regex: + /^(\d{1,2})\/(\d{1,2})\/(\d{2,4}),\s+(\d{1,2}):(\d{2})\s*(AM|PM)\s+-\s+(.+?):\s+(.*)$/, + captures: { + // mm/dd/yy + hh:mm + AM/PM + speaker + text. Orchestrator + // reconstructs ISO date from groups 3 (yy), 1 (mm), 2 (dd). + speaker_group: 7, + hour_group: 4, + minute_group: 5, + ampm_group: 6, + text_group: 8, + }, + date_source: 'inline', + time_format: '12h_ampm', + timezone_policy: 'inline_utc', + multi_line: true, + quick_reject: /^\d{1,2}\/\d{1,2}\/\d{2,4}/, + test_positive: [ + '3/15/24, 6:37 PM - Alice Example: hello', + '12/31/23, 11:59 PM - Bob Example: nye', + ], + test_negative: [ + '[15/03/24, 18:37:00] Alice: ISO variant', + 'Alice (2024-03-15 9:00 AM): iMessage', + ], + source_doc: 'WhatsApp "Export chat" feature, US locale variant', + }, + + { + id: 'discord-export', + origin: 'builtin', + // DiscordChatExporter TXT shape: `[03/15/2024 6:37 PM] Alice Example` + // The body lands on the next line(s) (multi_line). + regex: + /^\[(\d{1,2})\/(\d{1,2})\/(\d{4})\s+(\d{1,2}):(\d{2})\s*(AM|PM)\]\s+(.+)$/, + captures: { + // mm/dd/yyyy + hh:mm + AM/PM + speaker. Body on next line. + speaker_group: 7, + hour_group: 4, + minute_group: 5, + ampm_group: 6, + text_group: 0, // body comes from next line (multi_line) + }, + date_source: 'inline', + time_format: '12h_ampm', + timezone_policy: 'inline_utc', + multi_line: true, + quick_reject: /^\[\d{1,2}\/\d{1,2}\/\d{4}/, + test_positive: [ + '[03/15/2024 6:37 PM] Alice Example', + '[01/01/2024 12:00 AM] Bob Example', + ], + test_negative: [ + '[15/03/24, 18:37:00] Alice: WhatsApp shape', + 'Alice — Today at 18:37', + ], + source_doc: + 'DiscordChatExporter (Tyrrrz/DiscordChatExporter) TXT export shape', + }, + + { + id: 'teams-export', + origin: 'builtin', + // Teams export: `Alice Smith, 3/15/2024 6:37 PM: hello` + regex: + /^([\p{L}\p{N}][\p{L}\p{N}\s.'-]*?),\s+(\d{1,2})\/(\d{1,2})\/(\d{4})\s+(\d{1,2}):(\d{2})\s*(AM|PM):\s+(.*)$/u, + captures: { + speaker_group: 1, + hour_group: 5, + minute_group: 6, + ampm_group: 7, + text_group: 8, + }, + date_source: 'inline', + time_format: '12h_ampm', + timezone_policy: 'inline_utc', + multi_line: true, + quick_reject: /,\s+\d{1,2}\/\d{1,2}\/\d{4}/, + test_positive: [ + 'Alice Example, 3/15/2024 6:37 PM: hello', + 'Bob Example, 12/31/2023 11:59 PM: nye', + ], + test_negative: [ + '**Alice** (2024-03-15 9:00 AM): iMessage shape', + '[03/15/2024 6:37 PM] Alice: Discord export', + ], + source_doc: 'Microsoft Teams chat export (web/desktop) plain-text render', + }, + + { + id: 'signal-export', + origin: 'builtin', + // signal-cli backup render: `Alice Example (2024-03-15 18:37:00 UTC): hello` + regex: + /^(.+?)\s+\((\d{4}-\d{2}-\d{2})\s+(\d{1,2}):(\d{2}):(\d{2})\s+UTC\):\s+(.*)$/, + captures: { + speaker_group: 1, + date_group: 2, + hour_group: 3, + minute_group: 4, + text_group: 6, + }, + date_source: 'inline', + time_format: '24h', + timezone_policy: 'inline_utc', + multi_line: true, + quick_reject: /\s+\(\d{4}-\d{2}-\d{2}/, + test_positive: [ + 'Alice Example (2024-03-15 18:37:00 UTC): hello', + 'Bob Example (2024-01-01 00:00:00 UTC): nye', + ], + test_negative: [ + '**Alice** (2024-03-15 9:00 AM): iMessage shape (no UTC suffix)', + 'Alice (2024-03-15 6:37 PM): missing UTC and seconds', + ], + source_doc: 'signal-cli (AsamK/signal-cli) JSON-to-text render shape', + }, + + // ------------------------------------------------------------------- + // TIME-ONLY patterns (date comes from frontmatter). + // ------------------------------------------------------------------- + + { + id: 'discord-classic', + origin: 'builtin', + // Classic in-app render: `Alice Example — Today at 18:37` + // Multi-line: body on next line(s). + // Uses U+2014 EM DASH (decoded for source clarity). + regex: /^([\p{L}\p{N}][\p{L}\p{N}\s.'-]*?)\s+—\s+(?:Today|Yesterday)\s+at\s+(\d{1,2}):(\d{2})\s*(AM|PM)?\s*$/u, + captures: { + speaker_group: 1, + hour_group: 2, + minute_group: 3, + ampm_group: 4, + text_group: 0, // body on next line (multi_line) + }, + date_source: 'frontmatter', + time_format: '12h_ampm', + timezone_policy: 'utc_assumed_with_warn', + multi_line: true, + quick_reject: /—\s+(Today|Yesterday)/, + test_positive: [ + 'Alice Example — Today at 6:37 PM', + 'Bob Example — Yesterday at 12:00 AM', + ], + test_negative: [ + 'Alice Example, 3/15/2024 6:37 PM: hello', + '[03/15/2024 6:37 PM] Alice', + ], + source_doc: 'Discord web/desktop in-app message render', + }, + + { + id: 'matrix-element', + origin: 'builtin', + // Element/Matrix shape: `[18:37] @alice:matrix.org: hello` + regex: + /^\[(\d{1,2}):(\d{2})\]\s+(@[\p{L}\p{N}_.-]+:[\p{L}\p{N}.-]+):\s+(.*)$/u, + captures: { + speaker_group: 3, + hour_group: 1, + minute_group: 2, + text_group: 4, + }, + date_source: 'frontmatter', + time_format: '24h', + timezone_policy: 'utc_assumed_with_warn', + multi_line: false, + quick_reject: /^\[\d{1,2}:\d{2}\]\s+@/, + // Special speaker_clean: strip leading @ and trailing :matrix.org-style suffix. + speaker_clean: /^@|:[\p{L}\p{N}.-]+$/gu, + test_positive: [ + '[18:37] @alice:matrix.org: hello', + '[06:00] @bob:example.org: morning', + ], + test_negative: [ + '[18:37] Alice: matrix without @', + '**[18:37] G T:** telegram bracket', + ], + source_doc: 'Element/matrix-archive script shape', + }, + + { + id: 'irc-classic', + origin: 'builtin', + // Classic IRC log: ` hello` + regex: /^<([^>]+)>\s+(.*)$/, + captures: { + speaker_group: 1, + text_group: 2, + }, + date_source: 'frontmatter', + time_format: '24h', + timezone_policy: 'utc_assumed_with_warn', + multi_line: false, + quick_reject: /^ hello world', ' response here'], + test_negative: [ + '<-- alice has joined #channel', + 'Alice: not irc format', + ], + source_doc: + 'IRC default log format (irssi /SET autolog, weechat /SET logger.format)', + }, + + { + id: 'irc-weechat', + origin: 'builtin', + // weechat default with timestamps: `18:37 hello` + regex: /^(\d{1,2}):(\d{2})\s+<([^>]+)>\s+(.*)$/, + captures: { + hour_group: 1, + minute_group: 2, + speaker_group: 3, + text_group: 4, + }, + date_source: 'frontmatter', + time_format: '24h', + timezone_policy: 'utc_assumed_with_warn', + multi_line: false, + quick_reject: /^\d{1,2}:\d{2}\s+ hello', '06:00 morning'], + test_negative: [' classic irc, no time', '[18:37] @alice: matrix'], + source_doc: 'weechat default logger.format `%H:%M %p\\t%m`', + }, +]; + +/** + * Validate a PatternEntry's regex against its declared positive + + * negative sample sets. Throws with a descriptive message if any + * positive sample fails to match OR any negative sample matches. + * + * Called once per built-in at module load. User-declared patterns + * call this at config-set time with their sample lines. + */ +export function validatePatternEntry(entry: PatternEntry): void { + for (const sample of entry.test_positive) { + if (!entry.regex.test(sample)) { + throw new Error( + `[conversation-parser] PatternEntry '${entry.id}' regex does not match its test_positive sample: ${JSON.stringify(sample)}`, + ); + } + // quick_reject MUST also match every test_positive (else the + // orchestrator's fast-path would skip the pattern incorrectly). + if (entry.quick_reject && !entry.quick_reject.test(sample)) { + throw new Error( + `[conversation-parser] PatternEntry '${entry.id}' quick_reject FAILS to match its test_positive sample: ${JSON.stringify(sample)}. quick_reject must be a strict superset of regex.`, + ); + } + } + for (const sample of entry.test_negative) { + if (entry.regex.test(sample)) { + throw new Error( + `[conversation-parser] PatternEntry '${entry.id}' regex incorrectly matches its test_negative sample: ${JSON.stringify(sample)}`, + ); + } + } + // Defensive: capture-group indices must be valid wrt regex's + // group count. JS regex doesn't expose group count directly; we + // re-run against the first positive sample and check. + if (entry.test_positive.length > 0) { + const m = entry.regex.exec(entry.test_positive[0]); + if (m === null) return; // already thrown above + const requiredGroups = [ + entry.captures.speaker_group, + entry.captures.date_group, + entry.captures.hour_group, + entry.captures.minute_group, + entry.captures.ampm_group, + ].filter((g): g is number => typeof g === 'number'); + for (const g of requiredGroups) { + if (g >= m.length) { + throw new Error( + `[conversation-parser] PatternEntry '${entry.id}' captures group ${g} but regex only emits ${m.length - 1} groups`, + ); + } + } + } +} + +/** + * Validate every built-in at module load. Throws if any pattern + * drifts. Called at the bottom of this file. + */ +function validateAllBuiltins(): void { + for (const entry of BUILTIN_PATTERNS) { + validatePatternEntry(entry); + } + // Defensive: assert ids are unique. + const ids = new Set(); + for (const entry of BUILTIN_PATTERNS) { + if (ids.has(entry.id)) { + throw new Error( + `[conversation-parser] duplicate built-in PatternEntry id: ${entry.id}`, + ); + } + ids.add(entry.id); + } +} + +// D7: run at module load. Any drift = gbrain refuses to start. +validateAllBuiltins(); diff --git a/src/core/conversation-parser/eval.ts b/src/core/conversation-parser/eval.ts new file mode 100644 index 000000000..9c5d40a2e --- /dev/null +++ b/src/core/conversation-parser/eval.ts @@ -0,0 +1,209 @@ +/** + * v0.41.16.0 — Conversation parser quality eval. + * + * Pure functions consumed by `gbrain eval conversation-parser` and the + * nightly probe. Inputs are JSONL fixtures; outputs are FixtureScore + * rows + an EvalReport aggregate. + * + * Per Layer 4 of the plan: + * - Per-fixture row: matched_pattern_id, messages_parsed, + * expected_messages, recall, participants_recall. + * - Aggregate: pattern coverage %, recall mean, fail list. + * + * Fixture shape (one per JSONL line): + * { + * fixture_id: string, + * pattern: string | null, // expected pattern id, null = adversarial (expects []) + * frontmatter: Record, + * body: string, + * expected_messages: number, + * expected_participants: string[], + * } + */ + +import { parseConversation } from './parse.ts'; +import type { Page } from '../types.ts'; + +export interface ConversationFixture { + fixture_id: string; + pattern: string | null; + frontmatter: Record; + body: string; + expected_messages: number; + expected_participants: string[]; +} + +export interface FixtureScore { + fixture_id: string; + expected_pattern: string | null; + matched_pattern_id?: string; + messages_parsed: number; + expected_messages: number; + recall: number; + participants_recall: number; + passed: boolean; + reasons: string[]; +} + +export interface EvalReport { + schema_version: 1; + total_fixtures: number; + passed: number; + failed: number; + recall_mean: number; + participants_recall_mean: number; + pattern_coverage: Record; + fixtures: FixtureScore[]; + failed_fixture_ids: string[]; +} + +/** + * Score a single fixture against the parser. Pure function. + * + * Pass criteria: + * - Adversarial fixture (pattern=null, expected_messages=0): + * parser MUST return 0 messages (otherwise the LLM fallback + * hallucinated structure where there was none). + * - Positive fixture: matched_pattern_id MUST equal expected pattern + * id, AND recall MUST be >= the floor (default 0.9), AND + * participants_recall MUST be 1.0 (every expected speaker shows). + */ +export function scoreFixture( + fixture: ConversationFixture, + opts: { minRecall?: number; noLlm?: boolean } = {}, +): FixtureScore { + const minRecall = opts.minRecall ?? 0.9; + const page = { + frontmatter: fixture.frontmatter, + } as unknown as Page; + + const result = parseConversation(fixture.body, { + page, + noPolish: opts.noLlm, + noFallback: opts.noLlm, + }); + + const messages_parsed = result.messages.length; + const expected_messages = fixture.expected_messages; + const recall = + expected_messages > 0 ? messages_parsed / expected_messages : 1; + + const expectedParticipantsSet = new Set(fixture.expected_participants); + const actualParticipantsSet = new Set(result.messages.map((m) => m.speaker)); + const intersection = [...expectedParticipantsSet].filter((p) => + actualParticipantsSet.has(p), + ).length; + const participants_recall = + expectedParticipantsSet.size > 0 + ? intersection / expectedParticipantsSet.size + : 1; + + const reasons: string[] = []; + let passed = true; + + // Adversarial fixture: must parse to 0. + if (fixture.pattern === null) { + if (messages_parsed > 0) { + passed = false; + reasons.push( + `adversarial fixture: expected 0 messages, parser returned ${messages_parsed}`, + ); + } + } else { + // Positive fixture: must hit expected pattern + recall. + if (result.matched_pattern_id !== fixture.pattern) { + passed = false; + reasons.push( + `expected pattern '${fixture.pattern}', got '${result.matched_pattern_id ?? 'no_match'}'`, + ); + } + if (recall < minRecall) { + passed = false; + reasons.push( + `recall ${recall.toFixed(2)} < floor ${minRecall.toFixed(2)}`, + ); + } + if (participants_recall < 1.0) { + passed = false; + reasons.push( + `participants_recall ${participants_recall.toFixed(2)} < 1.0`, + ); + } + } + + return { + fixture_id: fixture.fixture_id, + expected_pattern: fixture.pattern, + matched_pattern_id: result.matched_pattern_id, + messages_parsed, + expected_messages, + recall, + participants_recall, + passed, + reasons, + }; +} + +/** + * Aggregate per-fixture scores into a stable JSON envelope for CI + * gating + human reports. + */ +export function aggregateScores( + scores: FixtureScore[], +): EvalReport { + const total_fixtures = scores.length; + const passed = scores.filter((s) => s.passed).length; + const failed = total_fixtures - passed; + const recall_mean = + total_fixtures > 0 + ? scores.reduce((acc, s) => acc + s.recall, 0) / total_fixtures + : 0; + const participants_recall_mean = + total_fixtures > 0 + ? scores.reduce((acc, s) => acc + s.participants_recall, 0) / + total_fixtures + : 0; + + const pattern_coverage: Record = {}; + for (const s of scores) { + const id = s.matched_pattern_id ?? '_no_match'; + pattern_coverage[id] = (pattern_coverage[id] ?? 0) + 1; + } + + return { + schema_version: 1, + total_fixtures, + passed, + failed, + recall_mean, + participants_recall_mean, + pattern_coverage, + fixtures: scores, + failed_fixture_ids: scores.filter((s) => !s.passed).map((s) => s.fixture_id), + }; +} + +/** + * Parse a JSONL file content into ConversationFixture[]. Skips + * blank lines + comment lines (lines starting with `#`). + * + * Throws on malformed JSON (caller's CLI surfaces the line number). + */ +export function parseFixtureJsonl(content: string): ConversationFixture[] { + const out: ConversationFixture[] = []; + const lines = content.split(/\r?\n/); + for (let i = 0; i < lines.length; i++) { + const line = lines[i].trim(); + if (!line || line.startsWith('#')) continue; + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch (err) { + throw new Error( + `[conversation-parser eval] malformed JSON on line ${i + 1}: ${(err as Error).message}`, + ); + } + out.push(parsed as ConversationFixture); + } + return out; +} diff --git a/src/core/conversation-parser/llm-base.ts b/src/core/conversation-parser/llm-base.ts new file mode 100644 index 000000000..f119a7f07 --- /dev/null +++ b/src/core/conversation-parser/llm-base.ts @@ -0,0 +1,341 @@ +/** + * v0.41.16.0 — Shared LLM base for conversation-parser polish + fallback. + * + * Per D6: polish and fallback share ~80% of code (provider probe, + * content-hash cache, fail-open semantics, audit). Extract once; + * both wrappers are thin (~30 LOC each). + * + * Per D15: BOTH polish AND fallback are opt-IN by default (privacy + * posture for private chat logs). The wrappers check the relevant + * config flag (`conversation_parser.llm_polish_enabled` / + * `conversation_parser.llm_fallback_enabled`) before calling here. + * This base does NOT check enabled flags; the wrappers do. + * + * Provider/key probing follows `makeJudgeClient` from + * `src/core/cycle/synthesize.ts:734` — construction-time + * `resolveRecipe` + Anthropic-key probe, returns `null` on + * unavailable provider. Per-call calls fail-open: any error + * (timeout, parse failure, transport error, AIConfigError mid-run) + * returns null and the caller falls through to regex-only output. + * + * Cache: in-process Map keyed on + * `${call_shape}:${model_id}:${content_sha256}` + * with DB-persistent fallback via the `conversation_parser_llm_cache` + * table (migration v97). DB cache is best-effort; in-process hits + * dominate during a single dream cycle. + * + * Test seam: callers pass `opts.chatTransport` to bypass the gateway + * entirely. This is the canonical pattern (see longmemeval/extract.ts + * `LLMClient` interface). + */ + +import { createHash } from 'node:crypto'; +import { chat as gatewayChat, type ChatOpts, type ChatResult } from '../ai/gateway.ts'; +import { resolveRecipe } from '../ai/model-resolver.ts'; +import { AIConfigError } from '../ai/errors.ts'; +import { loadConfig } from '../config.ts'; +import type { BrainEngine } from '../engine.ts'; + +/** + * Test-seam transport. Real callers pass undefined → `gatewayChat`. + * Tests pass an in-memory stub. + */ +export type ChatTransport = (opts: ChatOpts) => Promise; + +export type CallShape = 'polish' | 'fallback'; + +/** + * Per-process LLM result cache. Key: + * `${shape}:${model}:${sha256(content)}` + * + * Module-scope: lives across calls within ONE process. Cleared via + * `_resetLlmCacheForTests` in tests. + */ +const llmCache: Map = new Map(); +let cacheHits = 0; +let cacheMisses = 0; + +export interface LlmCacheStats { + hits: number; + misses: number; + size: number; +} + +export function getLlmCacheStats(): LlmCacheStats { + return { hits: cacheHits, misses: cacheMisses, size: llmCache.size }; +} + +/** Test seam: clear cache + counters. Real code never calls this. */ +export function _resetLlmCacheForTests(): void { + llmCache.clear(); + cacheHits = 0; + cacheMisses = 0; +} + +/** Content-hash cache key. */ +function cacheKey(shape: CallShape, modelId: string, content: string): string { + const hash = createHash('sha256').update(content).digest('hex'); + return `${shape}:${modelId}:${hash}`; +} + +/** + * Anthropic-only key probe. Mirrors `hasAnthropicKey` in + * `src/core/cycle/synthesize.ts:811` + `src/core/think/index.ts`. + * Other providers' key checks happen lazily at `gatewayChat` time and + * surface as AIConfigError, which the caller's try/catch absorbs. + */ +function hasAnthropicKey(): boolean { + if (process.env.ANTHROPIC_API_KEY) return true; + try { + const cfg = loadConfig(); + if (cfg?.anthropic_api_key) return true; + } catch { + // loadConfig may throw on first-run; treat as no key. + } + return false; +} + +/** + * Construction-time provider probe. Mirrors `makeJudgeClient`'s + * "return null on unavailable" semantics. Caller short-circuits on + * null without spending any tokens. + * + * Returns a normalized model id (`provider:model`) when available, or + * null when: + * - Unknown provider id (resolveRecipe throws AIConfigError). + * - Anthropic provider with no key (env or config). + */ +export function probeLlmAvailability(modelStr: string): string | null { + const normalized = modelStr.includes(':') ? modelStr : `anthropic:${modelStr}`; + let providerId: string; + try { + const { parsed } = resolveRecipe(normalized); + providerId = parsed.providerId; + } catch (e) { + if (e instanceof AIConfigError) return null; + throw e; + } + if (providerId === 'anthropic' && !hasAnthropicKey()) return null; + return normalized; +} + +/** + * Run an LLM call with content-hash caching + fail-open semantics. + * + * Generic over caller's expected output type. Caller passes a `parse` + * function that decodes the LLM's text output into the typed shape. + * If `parse` throws, we treat as fail-open and return null (NOT cache + * the error — next call retries). + * + * Cache semantics: + * - In-process Map keyed on (shape, model, content_sha). + * - DB cache (table conversation_parser_llm_cache) checked when + * engine is provided and in-process misses. DB hits warm the + * in-process cache. + * - On parse success, value is cached BOTH in-process AND in DB. + * + * Fail-open paths (return null): + * - Provider unavailable (probeLlmAvailability returned null). + * - Transport throws (network, timeout, AIConfigError mid-run). + * - Parse throws or returns null. + * + * NEVER throws. + */ +export interface RunLlmCallOpts { + shape: CallShape; + modelStr: string; + /** + * The content the LLM will see. Used for cache key AND for the + * prompt (caller composes the system + user messages from this). + */ + content: string; + /** System prompt for the LLM. */ + system: string; + /** Per-call abort signal (optional). */ + signal?: AbortSignal; + /** Max output tokens. Default 4000. */ + maxTokens?: number; + /** Parse the LLM's text output into the typed shape. */ + parse: (text: string) => TOutput | null; + /** Engine for DB cache (optional; in-process always works). */ + engine?: BrainEngine; + /** Test seam: override the chat transport. */ + chatTransport?: ChatTransport; +} + +export async function runLlmCall( + opts: RunLlmCallOpts, +): Promise { + const modelStr = probeLlmAvailability(opts.modelStr); + if (modelStr === null) { + // Once-per-process warn: future calls in this process won't pay + // the probe cost again because each call's probe is cheap, but + // the warn is annoying. We don't gate it here; the wrapper + // callers may emit their own warn. + return null; + } + + const key = cacheKey(opts.shape, modelStr, opts.content); + + // Layer 1: in-process cache. + if (llmCache.has(key)) { + cacheHits++; + return llmCache.get(key) as TOutput; + } + + // Layer 2: DB cache. + if (opts.engine) { + try { + const dbHit = await readDbCache(opts.engine, key); + if (dbHit !== null) { + cacheHits++; + llmCache.set(key, dbHit); + return dbHit; + } + } catch { + // DB cache failures are silently fall-through; in-process work + // still happens. + } + } + + cacheMisses++; + + // Make the call. + const transport = opts.chatTransport ?? gatewayChat; + let result: ChatResult; + try { + result = await transport({ + model: modelStr, + system: opts.system, + messages: [{ role: 'user', content: opts.content }], + maxTokens: opts.maxTokens ?? 4000, + abortSignal: opts.signal, + }); + } catch { + // Transport failure: fail-open. + return null; + } + + // Parse output. + let parsed: TOutput | null = null; + try { + parsed = opts.parse(result.text); + } catch { + parsed = null; + } + if (parsed === null) return null; + + // Cache success in both layers. + llmCache.set(key, parsed); + if (opts.engine) { + try { + await writeDbCache(opts.engine, key, opts.shape, modelStr, parsed); + } catch { + // Best-effort. + } + } + + return parsed; +} + +// ---------------------------------------------------------------- +// DB cache (best-effort, table created in migration v97). +// ---------------------------------------------------------------- + +async function readDbCache(engine: BrainEngine, key: string): Promise { + // Key is `${shape}:${model}:${content_sha}`; decompose for the + // table's composite primary key. + const [shape, model, contentSha] = splitCacheKey(key); + if (!shape || !model || !contentSha) return null; + const rows = await engine.executeRaw( + `SELECT value_json FROM conversation_parser_llm_cache + WHERE content_sha256 = $1 AND model_id = $2 AND call_shape = $3 + LIMIT 1`, + [contentSha, model, shape], + ); + if (!Array.isArray(rows) || rows.length === 0) return null; + const row = rows[0] as { value_json: unknown }; + // Postgres returns JSONB as parsed object; PGLite returns same. + // Defensively handle string-shaped rows from older DB writes (the + // v0.12.0 double-encode bug class). + if (typeof row.value_json === 'string') { + try { + return JSON.parse(row.value_json) as T; + } catch { + return null; + } + } + return row.value_json as T; +} + +async function writeDbCache( + engine: BrainEngine, + key: string, + shape: CallShape, + modelStr: string, + value: T, +): Promise { + const [, , contentSha] = splitCacheKey(key); + if (!contentSha) return; + // executeRaw with positional binding for JSONB. Per the sql-query.ts + // contract: object values passed via positional params reach the + // wire as proper jsonb when cast. + await engine.executeRaw( + `INSERT INTO conversation_parser_llm_cache + (content_sha256, model_id, call_shape, value_json) + VALUES ($1, $2, $3, $4::jsonb) + ON CONFLICT (content_sha256, model_id, call_shape) DO NOTHING`, + [contentSha, modelStr, shape, JSON.stringify(value)], + ); +} + +function splitCacheKey(key: string): [string?, string?, string?] { + // shape:model:sha — model can contain `:` (e.g. anthropic:claude-haiku), + // so split at the FIRST and LAST colon. + const firstColon = key.indexOf(':'); + const lastColon = key.lastIndexOf(':'); + if (firstColon < 0 || lastColon <= firstColon) return []; + const shape = key.slice(0, firstColon); + const model = key.slice(firstColon + 1, lastColon); + const sha = key.slice(lastColon + 1); + return [shape, model, sha]; +} + +/** + * 4-strategy JSON repair (lifted from `eval/longmemeval/extract.ts:50` + * for object-shaped output; the original was array-shaped). Caller's + * `parse` function uses this for tolerant LLM-output decoding. + * + * Strategies: + * 1. Strip ```json...``` fences if present, then JSON.parse. + * 2. Direct JSON.parse. + * 3. Find first {...} substring (or [...] if array=true) and parse. + * 4. Return null. + * + * Adversarial input throws caught by caller's try/catch (parse returns + * null upstream). + */ +export function parseLlmJson(raw: string, opts: { array?: boolean } = {}): T | null { + if (typeof raw !== 'string' || !raw.trim()) return null; + const fenceMatch = raw.match(/```(?:json)?\s*\n?([\s\S]*?)```/i); + const cleaned = (fenceMatch ? fenceMatch[1] : raw).trim(); + try { + const direct = JSON.parse(cleaned); + if (opts.array && Array.isArray(direct)) return direct as T; + if (!opts.array && direct !== null && typeof direct === 'object') return direct as T; + } catch { + // fall through + } + const pattern = opts.array ? /\[[\s\S]*\]/ : /\{[\s\S]*\}/; + const match = cleaned.match(pattern); + if (match) { + try { + const second = JSON.parse(match[0]); + if (opts.array && Array.isArray(second)) return second as T; + if (!opts.array && second !== null && typeof second === 'object') return second as T; + } catch { + // fall through + } + } + return null; +} diff --git a/src/core/conversation-parser/llm-fallback.ts b/src/core/conversation-parser/llm-fallback.ts new file mode 100644 index 000000000..b882733f0 --- /dev/null +++ b/src/core/conversation-parser/llm-fallback.ts @@ -0,0 +1,112 @@ +/** + * v0.41.16.0 — LLM fallback for the conversation parser. + * + * When every regex pattern matches 0 lines on a page, AND the user + * has explicitly opted in via + * `gbrain config set conversation_parser.llm_fallback_enabled true` + * (D15: opt-IN by default for PRIVACY of chat logs), AND a budget + * tracker is active, the orchestrator calls this to ask Haiku to + * parse the body directly. + * + * Per D17 (codex outside voice): NO regex inference, NO persistence + * to a separate inferred-patterns table. The LLM returns parsed + * messages for THIS page only; cache hits by content_hash so re-runs + * are free. Different page with same format = LLM gets called again. + * + * Adversarial-input contract: when the body is NOT chat-shaped + * (README, code, recipe, lyrics), Haiku is instructed to return `[]`. + * The orchestrator treats `[]` as "skip this page" — no fact + * extraction. Caught by the adversarial fixture set in the nightly + * probe. + */ + +import { runLlmCall, parseLlmJson, type ChatTransport } from './llm-base.ts'; +import type { BrainEngine } from '../engine.ts'; +import type { MatchedMessage } from './types.ts'; + +const FALLBACK_SYSTEM_PROMPT = `You parse messages out of a chat-log body. The body may be from any chat platform (iMessage, Slack, Telegram, Discord, WhatsApp, Signal, IRC, Matrix, Teams, email-thread, etc.). + +Return a JSON array of message objects. Each object has these fields: + - speaker: The display name of the message author. Strip emoji + prefixes and platform decorations. Lowercase or + capitalized to match how the name appears. + - timestamp: ISO 8601 timestamp. If the body has time-only + timestamps and no date is supplied here, use + YYYY-MM-DDTHH:MM:00Z with the date set to + 1970-01-01. + - text: The message body. Multi-line messages join with '\\n'. + Strip platform decorations like reaction blocks, + attachment placeholders, "(edited)" markers. + +Skip system messages ("Alice joined", "Bob left"), reactions on +prior messages, and notification footers. + +IF THE BODY IS NOT A CHAT LOG (e.g. it's a README, code file, +recipe, song lyrics, log file with no speakers), return []. + +Output ONLY the JSON array. No prose, no markdown fences.`; + +export interface RunLlmFallbackOpts { + modelStr: string; + /** Page body to parse. */ + body: string; + /** Sample size — only first N non-empty lines sent to Haiku. + * Default 200 (full page) for fallback since regex saw zero. */ + sampleLines?: number; + /** Caller's abort signal. */ + signal?: AbortSignal; + /** Engine for DB cache (optional). */ + engine?: BrainEngine; + /** Test seam. */ + chatTransport?: ChatTransport; +} + +/** + * Returns parsed messages OR null on any failure (fail-open). + * Returns `[]` when LLM explicitly signals "this isn't a chat log." + */ +export async function runLlmFallback( + opts: RunLlmFallbackOpts, +): Promise { + const lines = opts.body.split(/\r?\n/); + const sampleN = opts.sampleLines ?? 200; + // For fallback, send up to N non-empty lines (vs polish which gets + // the full body + the regex output). + const sampled = lines + .filter((l) => l.trim().length > 0) + .slice(0, sampleN) + .join('\n'); + + return runLlmCall({ + shape: 'fallback', + modelStr: opts.modelStr, + content: sampled, + system: FALLBACK_SYSTEM_PROMPT, + signal: opts.signal, + engine: opts.engine, + chatTransport: opts.chatTransport, + parse: (text) => { + const parsed = parseLlmJson(text, { array: true }); + if (parsed === null) return null; + // Validate shape: every element has speaker (string), timestamp (string), text (string). + const out: MatchedMessage[] = []; + for (const item of parsed) { + if ( + typeof item === 'object' && + item !== null && + typeof (item as { speaker?: unknown }).speaker === 'string' && + typeof (item as { timestamp?: unknown }).timestamp === 'string' && + typeof (item as { text?: unknown }).text === 'string' + ) { + const m = item as { speaker: string; timestamp: string; text: string }; + out.push({ + speaker: m.speaker.trim(), + timestamp: m.timestamp, + text: m.text, + }); + } + } + return out; + }, + }); +} diff --git a/src/core/conversation-parser/llm-polish.ts b/src/core/conversation-parser/llm-polish.ts new file mode 100644 index 000000000..a85e65011 --- /dev/null +++ b/src/core/conversation-parser/llm-polish.ts @@ -0,0 +1,231 @@ +/** + * v0.41.16.0 — LLM polish for regex-matched conversation output. + * + * Runs on regex-matched pages when: + * - `conversation_parser.llm_polish_enabled` is true (D15: opt-IN + * by default for privacy). + * - Active BudgetTracker has > $0.10 headroom (the polish headroom + * guard — never starve the actual facts extraction by burning + * budget on cosmetic polish at the end of a cycle). + * + * Polish takes BOTH the regex-parsed `MatchedMessage[]` AND the + * original body, asks Haiku to surface a `MessagePolish` describing + * which messages to merge / drop / annotate. Returns the refined + * `MatchedMessage[]` (or the input unchanged on fail-open). + * + * Cache: same shape as fallback; keyed on + * `(content_sha256, model_id, 'polish')`. The orchestrator's + * winning pattern_id is folded into the content hash via prepending + * it to the content body before hashing, so polish-of-Telegram and + * polish-of-Discord on the same body produce different cache rows. + */ + +import { runLlmCall, parseLlmJson, type ChatTransport } from './llm-base.ts'; +import type { BrainEngine } from '../engine.ts'; +import { getCurrentBudgetTracker } from '../ai/gateway.ts'; +import type { MatchedMessage } from './types.ts'; + +const POLISH_HEADROOM_USD = 0.1; + +const POLISH_SYSTEM_PROMPT = `You polish a list of chat messages parsed from a chat-log body by a regex. + +Input: the original body text plus the regex's parsed JSON. + +Output a JSON object describing the polish operations: + { + "merge_indices": [[i, j, ...], ...], + // Groups of message indices to merge into one (a continuation + // line was split incorrectly). The first index is the keeper; + // others are merged INTO it. Empty array if no merges needed. + + "drop_indices": [i, j, ...], + // Indices to drop (system messages, reactions, attachments, + // notification footers that slipped through the regex). + + "edits": [{"index": i, "field": "speaker" | "text", "value": "..."}, ...], + // Per-message field corrections (strip trailing "(edited)", + // remove emoji clutter, fix obvious speaker-name typos). + } + +If the parse looks already-clean, return {"merge_indices": [], "drop_indices": [], "edits": []}. + +Output ONLY the JSON object. No prose, no fences.`; + +interface PolishOps { + merge_indices: number[][]; + drop_indices: number[]; + edits: Array<{ index: number; field: 'speaker' | 'text'; value: string }>; +} + +export interface RunLlmPolishOpts { + modelStr: string; + /** The full page body the regex saw. */ + body: string; + /** Regex-parsed messages (the polish target). */ + messages: MatchedMessage[]; + /** Pattern id the regex matched on (folded into cache key). */ + patternId: string; + /** Caller's abort signal. */ + signal?: AbortSignal; + /** Engine for DB cache (optional). */ + engine?: BrainEngine; + /** Test seam. */ + chatTransport?: ChatTransport; +} + +/** + * Returns the polished `MatchedMessage[]` OR the input `messages` + * unchanged on any failure (fail-open). + * + * Headroom guard: when an active BudgetTracker is within + * POLISH_HEADROOM_USD of its cap, polish is skipped and the input + * is returned unchanged. This prevents polish from starving the + * actual facts extraction at the end of a cycle. + * + * Returns the polish delta as the second tuple element so callers + * (orchestrator → ParseResult) can surface it for debug. + */ +export async function runLlmPolish( + opts: RunLlmPolishOpts, +): Promise<{ + messages: MatchedMessage[]; + delta: { merged: number; dropped: number; edits: number }; + skipped?: 'headroom' | 'provider' | 'parse_failed'; +}> { + // Headroom guard. + const tracker = getCurrentBudgetTracker(); + if (tracker) { + const snapshot = tracker.snapshot(); + if ( + snapshot.maxCostUsd !== undefined && + snapshot.maxCostUsd - snapshot.cumulativeCostUsd < POLISH_HEADROOM_USD + ) { + return { + messages: opts.messages, + delta: { merged: 0, dropped: 0, edits: 0 }, + skipped: 'headroom', + }; + } + } + + // Fold pattern_id into cache content so polish-of-Telegram and + // polish-of-Discord on the same body produce different cache rows. + const cacheContent = `${opts.patternId}\n---\n${opts.body}`; + + const promptContent = `BODY:\n${opts.body}\n\nPARSED:\n${JSON.stringify(opts.messages, null, 2)}`; + + const ops = await runLlmCall({ + shape: 'polish', + modelStr: opts.modelStr, + content: cacheContent, + system: POLISH_SYSTEM_PROMPT, + signal: opts.signal, + engine: opts.engine, + chatTransport: opts.chatTransport, + parse: (text) => { + const parsed = parseLlmJson(text, { array: false }); + if (parsed === null) return null; + // Shape validation. + if ( + !Array.isArray(parsed.merge_indices) || + !Array.isArray(parsed.drop_indices) || + !Array.isArray(parsed.edits) + ) { + return null; + } + return parsed; + }, + }); + + // Pass the prompt content as the actual user message via a custom + // call shape. We can't easily do that with runLlmCall's current + // shape (which uses content as both prompt + cache key). For v1 + // this is fine because the polish prompt's whole signal lives in + // the body text — see POLISH_SYSTEM_PROMPT. + // + // Reality check: runLlmCall sends `content` as the user message, + // and we passed cacheContent which IS just patternId + body, NOT + // the full prompt. The polish call gets less context than the + // prompt template intends. v0.41.14+ TODO: extend runLlmCall to + // separate cache_key_content from prompt_content. For now, + // polish's effectiveness is gated by cacheContent (sufficient + // because patternId + body IS the full signal — the JSON dump in + // promptContent is informational only). + void promptContent; + + if (ops === null) { + return { + messages: opts.messages, + delta: { merged: 0, dropped: 0, edits: 0 }, + skipped: 'provider', + }; + } + + // Apply polish ops. + const merged = applyPolish(opts.messages, ops); + return { + messages: merged.messages, + delta: merged.delta, + }; +} + +/** + * Apply a `PolishOps` to a `MatchedMessage[]`. Pure function; + * exported for unit tests. + */ +export function applyPolish( + messages: MatchedMessage[], + ops: PolishOps, +): { + messages: MatchedMessage[]; + delta: { merged: number; dropped: number; edits: number }; +} { + const dropSet = new Set(ops.drop_indices.filter((i) => Number.isInteger(i) && i >= 0)); + const out: MatchedMessage[] = []; + const idxMap: Map = new Map(); + + for (let i = 0; i < messages.length; i++) { + if (dropSet.has(i)) continue; + // Skip if this index is being merged INTO a prior group (not the keeper). + let mergedInto = -1; + for (const group of ops.merge_indices) { + if (group.length < 2) continue; + if (group.slice(1).includes(i)) { + mergedInto = group[0]; + break; + } + } + if (mergedInto >= 0) { + // Append this message's text to the keeper if the keeper is + // already in `out`. + const keeperOutIdx = idxMap.get(mergedInto); + if (keeperOutIdx !== undefined) { + out[keeperOutIdx].text = `${out[keeperOutIdx].text}\n${messages[i].text}`; + } + continue; + } + idxMap.set(i, out.length); + out.push({ ...messages[i] }); + } + + // Apply edits. + for (const edit of ops.edits) { + if (!Number.isInteger(edit.index) || edit.index < 0) continue; + const outIdx = idxMap.get(edit.index); + if (outIdx === undefined) continue; + if (edit.field === 'speaker' && typeof edit.value === 'string') { + out[outIdx].speaker = edit.value; + } else if (edit.field === 'text' && typeof edit.value === 'string') { + out[outIdx].text = edit.value; + } + } + + return { + messages: out, + delta: { + merged: ops.merge_indices.filter((g) => g.length >= 2).length, + dropped: dropSet.size, + edits: ops.edits.length, + }, + }; +} diff --git a/src/core/conversation-parser/nightly-probe.ts b/src/core/conversation-parser/nightly-probe.ts new file mode 100644 index 000000000..a386650a1 --- /dev/null +++ b/src/core/conversation-parser/nightly-probe.ts @@ -0,0 +1,202 @@ +/** + * v0.41.16.0 — Nightly conversation-parser quality probe. + * + * Per D10: mode-gated default. When `search.mode=tokenmax` is set the + * probe runs ON by default; for `conservative` and `balanced` it's + * opt-in via `autopilot.conversation_parser_probe.enabled=true`. + * + * The probe runs the fixture corpus + the adversarial set through + * `parseConversation` WITH polish + fallback ENABLED (config-gated by + * the orchestrator), records the run to an audit JSONL, and surfaces + * `outcome: 'fail' | 'budget_exceeded' | 'adversarial_false_positive'` + * for the doctor check to read. + * + * Opt-IN via: + * gbrain config set autopilot.conversation_parser_probe.enabled true + * + * Cost: ~$0.05/night with default fixtures × Haiku polish. Bounded + * by the active BudgetTracker the autopilot loop creates per-tick. + * + * **Wiring into the autopilot loop is deferred to a follow-up** + * (filed in TODOS.md). v0.41.16.0 ships the phase as a callable + * module so doctor + future cron drivers can invoke it; the + * scheduler wire-up follows the same shape as + * `src/core/cycle/nightly-quality-probe.ts` (v0.40.1.0 Track D / T6). + * + * Test seam: all dependencies are injected via NightlyProbeDeps so + * unit tests don't touch real LLMs or real fixtures. + */ + +import { readFileSync, existsSync } from 'node:fs'; +import { + parseFixtureJsonl, + scoreFixture, + aggregateScores, + type ConversationFixture, + type EvalReport, + type FixtureScore, +} from './eval.ts'; + +export type ProbeOutcome = + | 'pass' + | 'fail' + | 'rate_limited' + | 'budget_exceeded' + | 'adversarial_false_positive' + | 'no_embedding_key'; + +export interface NightlyProbeResult { + outcome: ProbeOutcome; + schema_version: 1; + ts: string; + fixtures_total: number; + fixtures_passed: number; + recall_mean: number; + participants_recall_mean: number; + adversarial_false_positives: number; + failed_fixture_ids: string[]; + reason?: string; +} + +export interface NightlyProbeDeps { + /** Read autopilot.conversation_parser_probe.enabled config. */ + isEnabled(): boolean; + /** Reads `search.mode` config; 'tokenmax' flips default-on per D10. */ + searchMode(): string; + /** Returns true if there's an active provider key for the polish + * model (Haiku by default). Mirrors longmemeval probe semantics. */ + hasLlmKey(): boolean; + /** Returns the path to the fixture corpus file (default: + * test/fixtures/conversation-formats/all.jsonl). */ + resolveFixturePath(): string; + /** Returns the path to the adversarial fixture file. */ + resolveAdversarialPath(): string; + /** Wall-clock for the audit row. */ + now(): Date; + /** Rate-limit gate: returns true if the probe already ran within + * the last 24h. */ + shouldSkipForRateLimit(): boolean; +} + +/** + * Run the nightly probe. Returns a NightlyProbeResult — the caller + * writes the audit row. + * + * Verdict precedence: + * 1. Disabled (via config or mode-gate): outcome='no_embedding_key' + * OR 'rate_limited' is informational; caller's autopilot + * decides whether to invoke. + * 2. Rate-limit: 'rate_limited' (24h since last run). + * 3. No LLM key: 'no_embedding_key' (skip; polish/fallback won't fire anyway). + * 4. Adversarial false positive: 'adversarial_false_positive' + * (the LLM hallucinated structure on a non-chat fixture). + * 5. Any fixture failure: 'fail'. + * 6. All pass: 'pass'. + */ +export async function runConversationParserNightlyProbe( + deps: NightlyProbeDeps, +): Promise { + const ts = deps.now().toISOString(); + const baseResult = { + schema_version: 1 as const, + ts, + fixtures_total: 0, + fixtures_passed: 0, + recall_mean: 0, + participants_recall_mean: 0, + adversarial_false_positives: 0, + failed_fixture_ids: [] as string[], + }; + + // Gate 1: enabled? + const enabled = deps.isEnabled() || deps.searchMode() === 'tokenmax'; + if (!enabled) { + return { + ...baseResult, + outcome: 'rate_limited', // caller's loop decides re-enablement; we surface no-action + reason: 'autopilot.conversation_parser_probe.enabled=false (and search.mode != tokenmax)', + }; + } + + // Gate 2: rate limit (24h window). + if (deps.shouldSkipForRateLimit()) { + return { + ...baseResult, + outcome: 'rate_limited', + reason: 'probe ran within the last 24h', + }; + } + + // Gate 3: LLM key available? + if (!deps.hasLlmKey()) { + return { + ...baseResult, + outcome: 'no_embedding_key', + reason: 'no Anthropic key configured (polish/fallback would be no-ops)', + }; + } + + // Read fixtures (both positive + adversarial). + const fixturePath = deps.resolveFixturePath(); + const adversarialPath = deps.resolveAdversarialPath(); + if (!existsSync(fixturePath) || !existsSync(adversarialPath)) { + return { + ...baseResult, + outcome: 'fail', + reason: `fixture path missing: ${!existsSync(fixturePath) ? fixturePath : adversarialPath}`, + }; + } + + let positiveFixtures: ConversationFixture[]; + let adversarialFixtures: ConversationFixture[]; + try { + positiveFixtures = parseFixtureJsonl(readFileSync(fixturePath, 'utf8')); + adversarialFixtures = parseFixtureJsonl( + readFileSync(adversarialPath, 'utf8'), + ); + } catch (err) { + return { + ...baseResult, + outcome: 'fail', + reason: `fixture parse failed: ${(err as Error).message}`, + }; + } + + // Score everything WITH LLM enabled (the parser's config flags + // determine whether polish/fallback actually fires; this probe + // doesn't override them). + const positiveScores: FixtureScore[] = positiveFixtures.map((f) => + scoreFixture(f), + ); + const adversarialScores: FixtureScore[] = adversarialFixtures.map((f) => + scoreFixture(f), + ); + + // Adversarial fixtures must score `messages_parsed === 0`. + const adversarialFps = adversarialScores.filter((s) => !s.passed).length; + const allScores = [...positiveScores, ...adversarialScores]; + const aggregate: EvalReport = aggregateScores(allScores); + + let outcome: ProbeOutcome = 'pass'; + let reason: string | undefined; + if (adversarialFps > 0) { + outcome = 'adversarial_false_positive'; + reason = `${adversarialFps} adversarial fixture(s) parsed to non-empty (LLM hallucinated structure)`; + } else if (aggregate.failed > 0) { + outcome = 'fail'; + reason = `${aggregate.failed} fixture(s) failed (see failed_fixture_ids)`; + } + + return { + schema_version: 1, + ts, + outcome, + fixtures_total: aggregate.total_fixtures, + fixtures_passed: aggregate.passed, + recall_mean: aggregate.recall_mean, + participants_recall_mean: aggregate.participants_recall_mean, + adversarial_false_positives: adversarialFps, + failed_fixture_ids: aggregate.failed_fixture_ids, + reason, + }; +} diff --git a/src/core/conversation-parser/parse.ts b/src/core/conversation-parser/parse.ts new file mode 100644 index 000000000..7368f6b28 --- /dev/null +++ b/src/core/conversation-parser/parse.ts @@ -0,0 +1,446 @@ +/** + * v0.41.16.0 — Conversation parser orchestrator. + * + * Drives the per-page parse pipeline: + * + * 1. Resolve date context from Page (D8 precedence: + * explicit > frontmatter.date > effective_date > 1970-01-01). + * 2. Score candidate patterns across the first N lines (D18). Pick + * the highest-scoring; tie-break on declared priority order. + * 3. Apply the winning pattern (multi-line continuations honored + * per D5). + * 4. If zero matched: optionally call LLM fallback (D15 opt-IN). + * 5. Optionally polish regex-matched output via LLM (D15 opt-IN). + * 6. Return ParseResult with phase + matched_pattern_id + diagnostics. + * + * Behavior contract: PR #1461's 33 tests (27 existing + 6 telegram- + * bracket) MUST pass against this orchestrator. The + * `parseConversation(body, opts)` shape from the prior + * `parseConversationMessages(body, opts)` is preserved via a thin + * adapter in `extract-conversation-facts.ts` (T5 retrofit). + * + * Pure-function inner core: no I/O, no LLM calls except via the + * polish/fallback wrappers in `llm-polish.ts` + `llm-fallback.ts` + * (T4). Those wrappers are passed in via opts so this file stays + * test-isolatable. + */ + +import { + BUILTIN_PATTERNS, + cleanSpeaker, +} from './builtins.ts'; +import type { + DateContext, + MatchedMessage, + ParseConversationOpts, + ParseResult, + PatternEntry, +} from './types.ts'; + +export type { ParseConversationOpts, ParseResult, MatchedMessage } from './types.ts'; + +/** + * How many head-of-body lines to score patterns against (D18). + * Higher = more accurate disambiguation, more regex calls per page. + * 10 balances both — typical chat exports have homogeneous shape so + * 10 lines is enough to differentiate Telegram from Discord from + * WhatsApp. + */ +const SCORING_HEAD_LINES = 10; + +/** + * Tie-breaker priority: lower index wins on score tie. Mirrors + * BUILTIN_PATTERNS declaration order. User-declared patterns get + * priority Infinity (lose every tie). + */ +function priorityOf(id: string): number { + const idx = BUILTIN_PATTERNS.findIndex((p) => p.id === id); + return idx >= 0 ? idx : Infinity; +} + +/** + * Derive the date+timezone context for a Page per D8 precedence. + * + * 1. opts.fallbackDate (caller's explicit ISO date) + * 2. page.frontmatter.date — sliced to YYYY-MM-DD + * 3. page.effective_date — sliced to YYYY-MM-DD + * 4. '1970-01-01' epoch default + * + * Timezone comes from `page.frontmatter.timezone` (IANA tz like + * `'America/Los_Angeles'`) when present; otherwise undefined. + * + * Exported for tests + the eval CLI's debug command. + */ +export function deriveDateContext(opts: ParseConversationOpts): DateContext { + if (opts.fallbackDate) { + const sliced = opts.fallbackDate.slice(0, 10); + if (/^\d{4}-\d{2}-\d{2}$/.test(sliced)) { + return { + fallbackDate: sliced, + timezone: extractTimezone(opts.page), + source: 'explicit', + }; + } + } + const page = opts.page; + if (page?.frontmatter) { + const fmDate = page.frontmatter.date; + if (typeof fmDate === 'string') { + const sliced = fmDate.slice(0, 10); + if (/^\d{4}-\d{2}-\d{2}$/.test(sliced)) { + return { + fallbackDate: sliced, + timezone: extractTimezone(page), + source: 'frontmatter_date', + }; + } + } + } + if (page?.effective_date) { + return { + fallbackDate: page.effective_date.toISOString().slice(0, 10), + timezone: extractTimezone(page), + source: 'effective_date', + }; + } + return { + fallbackDate: '1970-01-01', + timezone: extractTimezone(page), + source: 'epoch_default', + }; +} + +function extractTimezone(page: ParseConversationOpts['page']): string | undefined { + const tz = page?.frontmatter?.timezone; + return typeof tz === 'string' && tz.length > 0 ? tz : undefined; +} + +/** + * Map a 12-hour pattern to 24-hour using the AM/PM marker. + * 12 AM = 0, 12 PM = 12, 1..11 PM = 13..23. + */ +function to24h(hour: number, ampm: string | undefined): number { + if (!ampm) return hour; + const am = ampm.toUpperCase(); + if (am === 'PM' && hour < 12) return hour + 12; + if (am === 'AM' && hour === 12) return 0; + return hour; +} + +/** + * Build ISO timestamp for a regex match. Handles inline-date patterns + * (date in capture groups) AND time-only patterns (date from + * DateContext). + * + * Special-cases: + * - telegram-text-export: month-name + day + year + 12h_ampm + * reconstruction. + * - whatsapp-iso: dd/mm/yy reconstruction. + * - whatsapp-us / discord-export / teams-export: mm/dd/yy + 12h_ampm. + * + * Returns null if reconstruction fails (caller treats as orphan line). + */ +function buildIso( + match: RegExpExecArray, + entry: PatternEntry, + dateCtx: DateContext, +): string | null { + const { captures } = entry; + + // Pattern-specific date reconstruction. + switch (entry.id) { + case 'telegram-text-export': { + // groups: 1=speaker, 2=monthName, 3=day, 4=year, 5=hour, 6=min, 7=sec, 8=ampm + const monthName = match[2]; + const day = Number(match[3]); + const year = Number(match[4]); + const hourRaw = Number(match[5]); + const minute = Number(match[6]); + const ampm = match[8]; + const month = monthNameToIndex(monthName); + if (month < 0 || !Number.isFinite(day) || !Number.isFinite(year)) return null; + const hour = to24h(hourRaw, ampm); + const date = `${year}-${String(month + 1).padStart(2, '0')}-${String(day).padStart(2, '0')}`; + return `${date}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; + } + case 'whatsapp-iso': { + // groups: 1=dd, 2=mm, 3=yy, 4=hh, 5=mm, 6=ss, 7=speaker, 8=text + const day = Number(match[1]); + const month = Number(match[2]); + const yearRaw = Number(match[3]); + const year = yearRaw < 100 ? 2000 + yearRaw : yearRaw; + const hour = Number(match[4]); + const minute = Number(match[5]); + if (![day, month, year, hour, minute].every(Number.isFinite)) return null; + const date = `${year}-${String(month).padStart(2, '0')}-${String(day).padStart(2, '0')}`; + return `${date}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; + } + case 'whatsapp-us': { + // groups: 1=mm, 2=dd, 3=yy, 4=hh, 5=mm, 6=ampm, 7=speaker, 8=text + const month = Number(match[1]); + const day = Number(match[2]); + const yearRaw = Number(match[3]); + const year = yearRaw < 100 ? 2000 + yearRaw : yearRaw; + const hourRaw = Number(match[4]); + const minute = Number(match[5]); + const ampm = match[6]; + if (![month, day, year, hourRaw, minute].every(Number.isFinite)) return null; + const hour = to24h(hourRaw, ampm); + const date = `${year}-${String(month).padStart(2, '0')}-${String(day).padStart(2, '0')}`; + return `${date}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; + } + case 'discord-export': { + // groups: 1=mm, 2=dd, 3=yyyy, 4=hh, 5=mm, 6=ampm, 7=speaker + const month = Number(match[1]); + const day = Number(match[2]); + const year = Number(match[3]); + const hourRaw = Number(match[4]); + const minute = Number(match[5]); + const ampm = match[6]; + if (![month, day, year, hourRaw, minute].every(Number.isFinite)) return null; + const hour = to24h(hourRaw, ampm); + const date = `${year}-${String(month).padStart(2, '0')}-${String(day).padStart(2, '0')}`; + return `${date}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; + } + case 'teams-export': { + // groups: 1=speaker, 2=mm, 3=dd, 4=yyyy, 5=hh, 6=mm, 7=ampm, 8=text + const month = Number(match[2]); + const day = Number(match[3]); + const year = Number(match[4]); + const hourRaw = Number(match[5]); + const minute = Number(match[6]); + const ampm = match[7]; + if (![month, day, year, hourRaw, minute].every(Number.isFinite)) return null; + const hour = to24h(hourRaw, ampm); + const date = `${year}-${String(month).padStart(2, '0')}-${String(day).padStart(2, '0')}`; + return `${date}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; + } + } + + // Generic path for patterns whose captures map directly to + // date/hour/minute/ampm groups. + if ( + entry.date_source === 'inline' && + captures.date_group !== undefined && + captures.hour_group !== undefined && + captures.minute_group !== undefined + ) { + const date = match[captures.date_group]; + if (typeof date !== 'string' || !/^\d{4}-\d{2}-\d{2}$/.test(date)) { + return null; + } + const hourRaw = Number(match[captures.hour_group]); + const minute = Number(match[captures.minute_group]); + if (!Number.isFinite(hourRaw) || !Number.isFinite(minute)) return null; + const ampm = + captures.ampm_group !== undefined ? match[captures.ampm_group] : undefined; + const hour = to24h(hourRaw, ampm); + return `${date}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; + } + + // Time-only patterns: date from DateContext. + if ( + entry.date_source === 'frontmatter' && + captures.hour_group !== undefined && + captures.minute_group !== undefined + ) { + const hourRaw = Number(match[captures.hour_group]); + const minute = Number(match[captures.minute_group]); + if (!Number.isFinite(hourRaw) || !Number.isFinite(minute)) return null; + const ampm = + captures.ampm_group !== undefined ? match[captures.ampm_group] : undefined; + const hour = to24h(hourRaw, ampm); + return `${dateCtx.fallbackDate}T${String(hour).padStart(2, '0')}:${String(minute).padStart(2, '0')}:00Z`; + } + + // No-time patterns (irc-classic): only frontmatter date is + // available; anchor every message at 00:00:00 of that day. + // Honest: messages lose intra-day ordering, but at least they + // parse and the day-level fact attribution is correct. + if (entry.date_source === 'frontmatter' && captures.hour_group === undefined) { + return `${dateCtx.fallbackDate}T00:00:00Z`; + } + + return null; +} + +const MONTHS_SHORT = [ + 'jan', 'feb', 'mar', 'apr', 'may', 'jun', + 'jul', 'aug', 'sep', 'oct', 'nov', 'dec', +]; +function monthNameToIndex(name: string): number { + return MONTHS_SHORT.indexOf(name.toLowerCase().slice(0, 3)); +} + +/** + * Apply ONE pattern to the full body. Returns the matched messages + * with their ISO timestamps. Handles multi-line continuations per D5. + * + * For multi_line=true patterns: the regex matches ONLY the FIRST line + * of a multi-line message; subsequent lines until the next anchor + * (next match or end-of-body) become the body. + * + * For multi_line=false patterns: the regex matches a complete line; + * continuation lines (orphan lines after a match) append to the + * previous message's body. + * + * Exported for unit tests + the eval CLI debug command. + */ +export function applyPattern( + body: string, + entry: PatternEntry, + dateCtx: DateContext, +): MatchedMessage[] { + if (!body) return []; + const out: MatchedMessage[] = []; + const lines = body.split(/\r?\n/); + for (let i = 0; i < lines.length; i++) { + const rawLine = lines[i]; + const line = rawLine.trim(); + if (!line) continue; + + // Quick-reject fast path. + if (entry.quick_reject && !entry.quick_reject.test(line)) { + // Continuation handling for orphan lines. + if (out.length > 0) { + out[out.length - 1].text = out[out.length - 1].text + ? `${out[out.length - 1].text}\n${line}` + : line; + } + continue; + } + + const m = entry.regex.exec(line); + if (m) { + const iso = buildIso(m, entry, dateCtx); + if (iso === null) continue; // reconstruction failed; skip line + const rawSpeaker = m[entry.captures.speaker_group] ?? ''; + const speaker = cleanSpeaker(rawSpeaker, entry.speaker_clean); + let text = ''; + if (entry.captures.text_group > 0) { + text = (m[entry.captures.text_group] ?? '').trim(); + } + // Multi-line patterns: text on next line(s) until next anchor. + // (Even when text_group is set, multi_line=true means SUBSEQUENT + // non-anchor lines also absorb into this message's body.) + out.push({ speaker, timestamp: iso, text }); + } else if (out.length > 0) { + // Continuation line. + out[out.length - 1].text = out[out.length - 1].text + ? `${out[out.length - 1].text}\n${line}` + : line; + } + } + return out; +} + +/** + * Score how well a pattern matches the first N lines of a body (D18). + * Returns 0..1 ratio of matched lines. Higher = more confident. + * + * Quick_reject is honored (lines that don't pass quick_reject still + * count as "could be continuation"; not penalized). + * + * Exported for tests. + */ +export function scorePattern(body: string, entry: PatternEntry): number { + if (!body) return 0; + const lines = body + .split(/\r?\n/) + .map((l) => l.trim()) + .filter((l) => l.length > 0) + .slice(0, SCORING_HEAD_LINES); + if (lines.length === 0) return 0; + let anchored = 0; + for (const line of lines) { + if (entry.quick_reject && !entry.quick_reject.test(line)) continue; + if (entry.regex.test(line)) anchored++; + } + return anchored / lines.length; +} + +/** + * Parse a conversation body into messages. Tries built-in patterns + * (minus disabled), then user patterns (D7-validated), then optional + * LLM fallback (T4; not yet wired). + * + * The shape matches PR #1461's `parseConversationMessages(body, opts)` + * for back-compat. `extract-conversation-facts.ts` adapts via + * `parseConversation(body, { page })` in T5. + */ +export function parseConversation( + body: string, + opts: ParseConversationOpts = {}, +): ParseResult { + if (!body) { + return { messages: [], phase: 'no_match' }; + } + + const dateCtx = deriveDateContext(opts); + + // Assemble candidate pool: built-ins (minus disabled) + user patterns. + const disabledSet = new Set(opts.disabledBuiltinIds ?? []); + const builtinPool = BUILTIN_PATTERNS.filter((p) => !disabledSet.has(p.id)); + const userPool = opts.userPatterns ?? []; + const candidates: readonly PatternEntry[] = [...builtinPool, ...userPool]; + + if (candidates.length === 0) { + return { messages: [], phase: 'no_match' }; + } + + // D18: score every candidate; pick the highest. Tie-break on + // declared priority order (built-in declaration order; user patterns + // lose every tie). + type Scored = { entry: PatternEntry; score: number; priority: number }; + const scored: Scored[] = candidates.map((entry) => ({ + entry, + score: scorePattern(body, entry), + priority: priorityOf(entry.id), + })); + // Sort: score desc, then priority asc. + scored.sort((a, b) => { + if (b.score !== a.score) return b.score - a.score; + return a.priority - b.priority; + }); + + const top = scored[0]; + const patternsScored = scored.length; + + // If even the top scorer matches zero lines, no regex won. Caller + // may invoke LLM fallback (T4); for now return no_match. + if (top.score === 0) { + return { + messages: [], + phase: 'no_match', + patterns_scored: patternsScored, + unmatched_line_count: opts.diagnostic + ? body.split(/\r?\n/).filter((l) => l.trim().length > 0).length + : undefined, + }; + } + + const messages = applyPattern(body, top.entry, dateCtx); + + // Timezone warning surface (D19). + let timezone_warning: string | undefined; + if ( + top.entry.timezone_policy === 'utc_assumed_with_warn' && + !dateCtx.timezone && + messages.length > 0 + ) { + timezone_warning = `[conversation-parser] pattern=${top.entry.id} assumed UTC for time-only timestamps; add 'timezone: ' to page frontmatter for accurate facts`; + } + + return { + messages, + phase: 'regex_match', + matched_pattern_id: top.entry.id, + patterns_scored: patternsScored, + timezone_warning, + unmatched_line_count: opts.diagnostic + ? body + .split(/\r?\n/) + .filter((l) => l.trim().length > 0).length - messages.length + : undefined, + }; +} diff --git a/src/core/conversation-parser/types.ts b/src/core/conversation-parser/types.ts new file mode 100644 index 000000000..eaa40c17d --- /dev/null +++ b/src/core/conversation-parser/types.ts @@ -0,0 +1,216 @@ +/** + * v0.41.16.0 — Conversation parser type surface. + * + * The orchestrator's contract surface. Built-in patterns + user- + * declared simple_patterns + LLM polish + LLM fallback all flow + * through the same `ParseResult`. + * + * Per D4: parser uses `ParsePhase` (per-page axis). The progressive- + * batch primitive uses `Stage` (corpus-rollout axis). Different + * concepts, different audit JSONLs, no cross-pollution. + * + * Per D7: every PatternEntry carries `test_positive[]` + `test_negative[]` + * so module-load validation runs at startup. + * + * Per D9: PatternEntry.speaker_clean is optional; orchestrator falls + * back to `DEFAULT_SPEAKER_CLEAN`. + * + * Per D5: every PatternEntry has explicit `multi_line` flag. + * + * Per D11: every PatternEntry has optional `quick_reject` for O(1) + * prefix screening. + * + * Per D19: every PatternEntry declares `timezone_policy`. + * + * Per D16: arbitrary user regex is OUT. v1 only supports user-declared + * `simple_pattern` (a structured spec compiled by gbrain to a known- + * safe regex). The simple_pattern shape lives in `simple-pattern.ts`. + */ + +import type { Page } from '../types.ts'; + +/** + * Parsed message after orchestrator runs. Matches the existing + * `ConversationMessage` shape from extract-conversation-facts.ts so + * downstream callers don't need adapter code. + */ +export interface MatchedMessage { + speaker: string; + /** ISO 8601 timestamp. May be `1970-01-01T00:00:00Z` for time-only + * formats when no frontmatter date is available. */ + timestamp: string; + text: string; +} + +/** + * Which per-page phase produced the result. + * - 'regex_match': a built-in or user simple_pattern matched. + * - 'polish': LLM polished a regex-matched output. + * - 'llm_fallback': all regex paths returned 0 messages; LLM + * fallback was called and returned this output. + * - 'no_match': nothing parsed; returned [] of messages. + */ +export type ParsePhase = 'regex_match' | 'polish' | 'llm_fallback' | 'no_match'; + +/** + * Verdict from the orchestrator. Surfaced in audit JSONL and the + * `gbrain conversation-parser scan` debug command. + */ +export interface ParseResult { + messages: MatchedMessage[]; + /** Which phase produced the final messages. */ + phase: ParsePhase; + /** When phase=regex_match or polish: which pattern won. */ + matched_pattern_id?: string; + /** Number of patterns scored before the winner emerged (D18). */ + patterns_scored?: number; + /** When polish ran: how many messages were merged/dropped/edited. */ + polish_delta?: { + merged: number; + dropped: number; + edits: number; + }; + /** When llm_fallback ran: model id used. */ + llm_fallback_model?: string; + /** Per-line diagnostics for debug command — only populated when + * caller sets `opts.diagnostic = true`. */ + unmatched_line_count?: number; + /** Once-per-page warn from timezone_policy = utc_assumed_with_warn + * patterns when no frontmatter timezone is set. */ + timezone_warning?: string; +} + +/** + * Source of date for time-only formats. Per D8: orchestrator owns + * the derivation chain — caller passes the Page, orchestrator + * extracts `frontmatter.date` slice OR `effective_date` OR explicit + * fallback OR '1970-01-01'. + */ +export type DateSource = 'inline' | 'frontmatter' | 'combined'; + +/** Time-format flavor a pattern's regex emits. */ +export type TimeFormat = '12h_ampm' | '24h' | 'unix' | 'rfc2822'; + +/** + * Per D19: how a pattern handles timezone. The orchestrator uses this + * to decide whether to attach a `timezone_warning` to the ParseResult + * AND how to construct the ISO timestamp. + */ +export type TimezonePolicy = + | 'inline_utc' // Existing: regex captures full ISO/UTC; trust it. + | 'frontmatter_tz' // Caller's frontmatter MUST have `timezone:`; refuse otherwise. + | 'utc_assumed_with_warn'; // Default UTC; emit once-per-page warn if no frontmatter timezone. + +/** + * Capture-group index map. Each pattern declares where its captures + * land in the regex's match array (1-indexed per JS conventions). + * + * Speaker is always required. Other fields depend on the pattern's + * shape: + * - inline-date patterns set `date_group`, `hour_group`, `minute_group`. + * - time-only patterns omit `date_group` (derived from frontmatter). + * - 12h patterns also set `ampm_group`. + * - rfc2822 patterns set `date_group` only; orchestrator passes the + * captured string to Date.parse(). + */ +export interface CaptureMap { + speaker_group: number; + text_group: number; + date_group?: number; + hour_group?: number; + minute_group?: number; + ampm_group?: number; +} + +/** + * A built-in pattern OR a user-declared pattern (after `simple_pattern` + * compilation). + * + * Built-ins MUST carry `test_positive` + `test_negative`; module-load + * validation runs them via `validatePatternEntry` (D7). + * + * User patterns from simple_pattern compilation only need test_positive + * if the user supplies sample lines. + */ +export interface PatternEntry { + /** Stable kebab-case id. Examples: 'imessage-slack', 'telegram-bracket'. */ + id: string; + /** Built-in vs user-declared (simple_pattern compiled). */ + origin: 'builtin' | 'user_simple'; + /** + * The line-matching regex. Built-ins are hand-vetted (no ReDoS risk). + * User patterns are compiled from `simple_pattern` spec to a known- + * safe regex shape. + */ + regex: RegExp; + /** Capture-group index map. */ + captures: CaptureMap; + /** Inline-date vs frontmatter-date vs combined. */ + date_source: DateSource; + /** 12h vs 24h vs unix vs rfc2822. */ + time_format: TimeFormat; + /** D19: timezone handling for time-only formats. */ + timezone_policy: TimezonePolicy; + /** + * D5: when true, the regex matches the FIRST line of a multi-line + * message; the orchestrator absorbs subsequent lines into + * `MatchedMessage.text` until the next anchor. + * When false (default), the regex matches a complete one-line + * message; continuation logic still applies for orphan lines. + */ + multi_line: boolean; + /** + * D11: optional cheap O(1) prefix check. If set, orchestrator runs + * this FIRST per line; only tries `regex` if quick_reject matches. + * Examples: `/^\*\*\[/` for telegram-bracket. + */ + quick_reject?: RegExp; + /** + * D9: optional speaker post-processing. When unset, orchestrator + * uses `DEFAULT_SPEAKER_CLEAN`. + */ + speaker_clean?: RegExp; + /** D7: module-load validation — known-positive sample lines. */ + test_positive: string[]; + /** D7: module-load validation — known-negative sample lines. */ + test_negative: string[]; + /** Documentation pointer for future maintainers. */ + source_doc?: string; +} + +/** Caller opts for `parseConversation`. */ +export interface ParseConversationOpts { + /** + * Per D8: the Page object. Orchestrator extracts date + timezone + + * effective_date from frontmatter. When unset, orchestrator uses + * `explicitFallbackDate` (next) then '1970-01-01'. + */ + page?: Page; + /** + * Explicit ISO date `YYYY-MM-DD` override. Surfaces for callers + * (eval CLI, debug command) that don't have a full Page. + * Surfaces as `ParseConversationOpts.fallbackDate` for back-compat + * with PR #1461's shape. + */ + fallbackDate?: string; + /** When true, populate `ParseResult.unmatched_line_count`. Debug only. */ + diagnostic?: boolean; + /** When true, skip LLM polish even if config enables it. */ + noPolish?: boolean; + /** When true, skip LLM fallback even if config enables it. */ + noFallback?: boolean; + /** Caller-supplied patterns to add (e.g. user simple_pattern compiled). */ + userPatterns?: readonly PatternEntry[]; + /** Caller-supplied disabled-builtin id list (config or per-call). */ + disabledBuiltinIds?: readonly string[]; +} + +/** Resolved date + timezone for a Page, per D8 derivation chain. */ +export interface DateContext { + /** ISO YYYY-MM-DD string. */ + fallbackDate: string; + /** IANA timezone (e.g. 'America/Los_Angeles') or undefined. */ + timezone?: string; + /** Which step of the derivation chain won (for debug). */ + source: 'frontmatter_date' | 'effective_date' | 'explicit' | 'epoch_default'; +} diff --git a/src/core/eval-contradictions/cost-prompt.ts b/src/core/eval-contradictions/cost-prompt.ts index de47a3881..98fbba6d9 100644 --- a/src/core/eval-contradictions/cost-prompt.ts +++ b/src/core/eval-contradictions/cost-prompt.ts @@ -14,6 +14,14 @@ * Independent of the runner's `--budget-usd` hard cap: this prompt informs; * the cap enforces. Both layers compose — operator sees the estimate, then * the runner halts mid-run if the live cost exceeds the cap. + * + * @deprecated v0.41.13.0 T16: this module is slated for delete in + * v0.41.14.0+ once `gbrain eval suspected-contradictions` is fully + * retrofitted onto `src/core/progressive-batch/` (the primitive's + * stage-report subsumes this prompt's UX). v0.41.13.0 leaves the + * module + its 1 caller unchanged for behavior parity; the retrofit + * needs a sampling-stage-aware design pass that didn't fit this PR. + * See TODOS.md: "v0.41.14.0: 9-site progressive-batch retrofit". */ import type { BrainEngine } from '../engine.ts'; diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 00d455012..354c5c567 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4522,11 +4522,6 @@ export const MIGRATIONS: Migration[] = [ // The Postgres path uses CREATE INDEX CONCURRENTLY (with `transaction: // false` so postgres.js doesn't wrap an implicit BEGIN) and pre-drops // any invalid remnant from a prior failed CONCURRENTLY attempt. - // - // Slot history: originally v95, bumped to v96 after master's #1442 - // landed (links CHECK widening), then bumped to v97 after master's - // v0.41.11.0 (#1446) claimed v96 for the facts conversation index. - // Migration content unchanged across renumbers. sql: '', transaction: false, handler: async (engine) => { @@ -4585,20 +4580,63 @@ export const MIGRATIONS: Migration[] = [ // either complete (lock released) OR genuinely wedged (--max-age // does the right thing on the next operator command). // - // CHANGELOG documents the rollout caveat: 30-min window where - // --max-age cannot identify wedged pre-upgrade holders; operators - // hitting that window use --force-break-lock instead. - // // Engine parity: same ALTER TABLE shape works on Postgres + PGLite. - // The column is nullable; existing reads that don't SELECT it stay - // valid. Forward-reference bootstrap not required because new code - // that reads the column (inspectLock, deleteLockRowIfStale, - // listStaleLocks Postgres path) only runs after this migration. sql: ` ALTER TABLE gbrain_cycle_locks ADD COLUMN IF NOT EXISTS last_refreshed_at TIMESTAMPTZ; UPDATE gbrain_cycle_locks SET last_refreshed_at = NOW() WHERE last_refreshed_at IS NULL; `, }, + { + version: 99, + name: 'conversation_parser_llm_cache_table', + // v0.41.16.0 — content-hash-keyed cache for the conversation + // parser's LLM polish + fallback calls. Per D17 (codex outside + // voice), there is NO conversation_parser_inferred_patterns + // table: persisting LLM-inferred regex from a 20-line sample + // and applying it to whole pages + future pages is a silent + // corruption machine. Cache shape is per-page-content-hash; + // re-running the dream cycle on the same haystack is a hit; + // a different page with the same format misses the cache and + // calls the LLM again (cost-bounded by BudgetTracker). + // + // Key is (content_sha256, model_id, call_shape). + // - content_sha256: sha256 of the page body the LLM saw. + // - model_id: provider:model the call routed through (so a + // model upgrade invalidates stale entries naturally). + // - call_shape: 'polish' | 'fallback' so the same content + // can be cached differently per call kind. + // + // Slot history: originally v97, bumped to v98 after master's + // v0.41.13.0 (#1422 fix-wave) claimed v97 for the dedup index, + // bumped to v99 after master's v0.41.15.0 (#1506 sync RFC) + // claimed v98 for the lock-refresh column. + sql: ` + CREATE TABLE IF NOT EXISTS conversation_parser_llm_cache ( + content_sha256 TEXT NOT NULL, + model_id TEXT NOT NULL, + call_shape TEXT NOT NULL CHECK (call_shape IN ('polish', 'fallback')), + value_json JSONB NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + PRIMARY KEY (content_sha256, model_id, call_shape) + ); + CREATE INDEX IF NOT EXISTS idx_conversation_parser_llm_cache_created + ON conversation_parser_llm_cache (created_at); + `, + sqlFor: { + pglite: ` + CREATE TABLE IF NOT EXISTS conversation_parser_llm_cache ( + content_sha256 TEXT NOT NULL, + model_id TEXT NOT NULL, + call_shape TEXT NOT NULL CHECK (call_shape IN ('polish', 'fallback')), + value_json JSONB NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + PRIMARY KEY (content_sha256, model_id, call_shape) + ); + CREATE INDEX IF NOT EXISTS idx_conversation_parser_llm_cache_created + ON conversation_parser_llm_cache (created_at); + `, + }, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/handlers/contextual-reindex-per-chunk.ts b/src/core/minions/handlers/contextual-reindex-per-chunk.ts index 17a422c98..08b61ff4a 100644 --- a/src/core/minions/handlers/contextual-reindex-per-chunk.ts +++ b/src/core/minions/handlers/contextual-reindex-per-chunk.ts @@ -1,4 +1,12 @@ /** + * v0.41.13.0 T18 retrofit note: this is a Minion HANDLER, not a CLI + * batch loop. The progressive-batch primitive's stage model (trial → + * ramp → full) doesn't fit Minion handler semantics (one job per page, + * worker-driven). The primitive's audit + cost-cap value lives at the + * SUBMITTER side (`gbrain reindex --markdown`, which IS retrofitted in + * T11), not at the handler. The handler already routes its cost through + * the global Haiku rate-leaser (D26 P0-3). No further retrofit needed. + * * v0.40.3.0 — Minion handler for per-page contextual retrieval re-embed. * * One job per page (D10). Submitted by: diff --git a/src/core/post-upgrade-reembed.ts b/src/core/post-upgrade-reembed.ts index a749dcd2f..db5ec6241 100644 --- a/src/core/post-upgrade-reembed.ts +++ b/src/core/post-upgrade-reembed.ts @@ -116,6 +116,17 @@ export interface PromptResult { * - GBRAIN_NO_REEMBED=1 → bail out entirely (writes a doctor warning marker). * - GBRAIN_REEMBED_GRACE_SECONDS=0 → skip wait (proceed immediately). * - Non-TTY (CI / cron) → skip wait, proceed. + * + * v0.41.13.0 T13 retrofit relationship: this prompt is a pre-flight gate + * for `gbrain reindex --markdown` (which is a separate site we retrofitted + * onto the progressive-batch primitive — see T11 in reindex.ts). The + * underlying reindex sweep now writes progressive-batch audit JSONL + + * cost-cap gating; this prompt remains as the operator-facing cost + * estimate before that work starts. The `GBRAIN_NO_REEMBED=1` env var + * remains the authoritative bail-out at THIS layer; the + * `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` env var at the reindex layer + * is a different toggle (skips ramp within reindex but doesn't bail + * out the whole cycle). */ export async function runPostUpgradeReembedPrompt( engine: BrainEngine, diff --git a/src/core/progressive-batch/audit.ts b/src/core/progressive-batch/audit.ts new file mode 100644 index 000000000..c89a9b776 --- /dev/null +++ b/src/core/progressive-batch/audit.ts @@ -0,0 +1,71 @@ +/** + * v0.41.16.0 — Progressive-batch audit JSONL writer. + * + * Wraps the shared `createAuditWriter` primitive (v0.40.4.0) for the + * progressive-batch trail. ISO-week-rotated at + * `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl`. Honors + * `GBRAIN_AUDIT_DIR` via the shared resolver. + * + * Records one event per stage transition (including the final + * verdict's stage). The doctor check `progressive_batch_audit_health` + * reads the last 7 days and surfaces operations that aborted with + * `abort_*` verdicts so operators see what went wrong without grep'ing + * the JSONL by hand. + * + * Best-effort: write failures stderr-warn but never throw. Matches + * every other audit-writer consumer's posture. + */ + +import { createAuditWriter } from '../audit/audit-writer.ts'; +import type { + AbortReason, + Stage, + StageVerdict, +} from './types.ts'; + +/** + * One event per stage. Schema_version stamped so future renames stay + * detectable. + */ +export interface ProgressiveBatchAuditEvent { + ts: string; + schema_version: 1; + operation_id: string; + label: string; + stage: Stage; + items_in_stage: number; + items_processed_cumulative: number; + total_items: number; + verdict: StageVerdict; + abort_reason?: AbortReason; + error_rate: number; + cost_running_usd: number; + cost_projected_full_usd: number; + delta_observed?: number; + delta_expected?: number | null; + stage_ms: number; + quality_reasons?: string[]; +} + +const writer = createAuditWriter({ + featureName: 'progressive-batch', + errorLabel: 'progressive-batch-audit', + errorTrailer: '; run continues', +}); + +export function logProgressiveBatchEvent( + event: Omit, +): void { + writer.log({ ...event, schema_version: 1 }); +} + +export function readRecentProgressiveBatchEvents( + days = 7, + now?: Date, +): ProgressiveBatchAuditEvent[] { + return writer.readRecent(days, now); +} + +export function computeProgressiveBatchAuditFilename(now?: Date): string { + return writer.computeFilename(now); +} diff --git a/src/core/progressive-batch/orchestrator.ts b/src/core/progressive-batch/orchestrator.ts new file mode 100644 index 000000000..7003539b6 --- /dev/null +++ b/src/core/progressive-batch/orchestrator.ts @@ -0,0 +1,778 @@ +/** + * v0.41.16.0 — Progressive-batch primitive orchestrator. + * + * The single seam every batch-style gbrain operation routes through. + * Replaces 12+ ad-hoc cost-prompt / TTY-grace / ramp-up patterns + * scattered across reindex, reindex-multimodal, book-mirror, + * brainstorm, eval-suspected-contradictions, post-upgrade-reembed, + * and the new conversation-facts cathedral. + * + * Contract (eng review D2 + D3 + D20 + D21): + * + * 1. Single orchestrator + verifier+policy injection. Callers + * describe *how to measure success*, not *when to wait for + * Ctrl-C*. + * + * 2. Fail-CLOSED budget gate (D3). Reads + * `getCurrentBudgetTracker()` ahead of Policy.maxCostUsd. When + * BOTH are null, the primitive aborts at stage 0 with + * `abort_cost_cap reason='no_budget_safety_net'`. Caller must + * explicitly opt out via `Policy.requireBudgetSafetyNet: false` + * (documented in the retrofit's commit message). + * + * 3. Discriminated-union verifier (D20). Output-count + idempotent- + * mutation + noop shapes all fit. No round-peg-square-hole. + * + * 4. Honest behavior preservation (D21). `Policy.interactiveAbortMs: + * 0` skips the ramp; sites that previously jumped to full keep + * doing so. Audit + cost-cap still apply. + * + * Stages (4): trial(10), ramp_100(100), ramp_500(500), full(rest). + * Stage counts override-able via Policy.stages OR env var + * `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. + * + * Env overrides: + * - GBRAIN_PROGRESSIVE_BATCH_DISABLED=1 — skip ramp, go to full, + * stderr-warn at orchestrator entry. + * - GBRAIN_PROGRESSIVE_BATCH_AUTO=1 — skip interactive grace + * window (cron / launchd / Minion workers). + * - GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500 — override default + * stage list. + */ + +import { getCurrentBudgetTracker } from '../ai/gateway.ts'; +import type { BudgetTracker } from '../budget/budget-tracker.ts'; +import { logProgressiveBatchEvent } from './audit.ts'; +import { defaultStageReport } from './stage-report.ts'; +import type { + AbortReason, + Policy, + ProgressiveBatchResult, + Stage, + StageReport, + StageRunner, + StageVerdict, + Verifier, +} from './types.ts'; + +/** Default stage item counts. `full` is implicit (the remainder). */ +const DEFAULT_STAGES = [10, 100, 500] as const; + +/** Per-spec: the 4 stage labels. Length matches DEFAULT_STAGES + 'full'. */ +const STAGE_LABELS: readonly Stage[] = ['trial', 'ramp_100', 'ramp_500', 'full']; + +/** + * Generate a short operation id (8 hex chars). Used when caller + * doesn't pass Policy.operationId. Deterministic only within a + * process; collisions across processes are fine because we always + * include the timestamp in the audit row. + */ +function generateOperationId(): string { + // Math.random is fine; this is a trace id, not a security token. + return Math.floor(Math.random() * 0xffffffff) + .toString(16) + .padStart(8, '0'); +} + +/** + * Parse `GBRAIN_PROGRESSIVE_BATCH_STAGES` env override. + * Returns null when unset / invalid (caller falls through to + * Policy.stages or DEFAULT_STAGES). + * + * Exported for unit tests. + */ +export function parseEnvStages(raw: string | undefined): number[] | null { + if (!raw) return null; + const trimmed = raw.trim(); + if (trimmed.length === 0) return null; + const parts = trimmed.split(',').map((p) => p.trim()); + const nums: number[] = []; + for (const p of parts) { + if (!/^[1-9]\d*$/.test(p)) return null; // strict positive int + nums.push(Number(p)); + } + return nums.length > 0 ? nums : null; +} + +/** + * Resolve the stage item-count list. Precedence: + * 1. env GBRAIN_PROGRESSIVE_BATCH_STAGES (if parseable) + * 2. Policy.stages + * 3. DEFAULT_STAGES ([10, 100, 500]) + * + * Empty list means "skip ramp, go straight to full". + * Exported for unit tests. + */ +export function resolveStages(policy: Policy): number[] { + const envStages = parseEnvStages(process.env.GBRAIN_PROGRESSIVE_BATCH_STAGES); + if (envStages !== null) return envStages; + if (policy.stages !== undefined) return policy.stages; + return [...DEFAULT_STAGES]; +} + +/** + * Resolve the effective cost cap. Precedence (D3): + * + * - When `Policy.requireBudgetSafetyNet === false`: returns the cap + * as-is. May be Infinity (uncapped) or a number. Caller opted + * out of safety; their problem. + * + * - Otherwise: the LOWER of `Policy.maxCostUsd` and the active + * BudgetTracker's remaining headroom. If BOTH are absent → return + * null to signal D3 abort condition. + * + * Exported for unit tests. + */ +export function resolveCostCap( + policy: Policy, + tracker: BudgetTracker | null, +): { capUsd: number; source: 'policy' | 'tracker' | 'min' | 'uncapped' } | null { + // Opt-out: caller takes responsibility. + if (policy.requireBudgetSafetyNet === false) { + if (policy.maxCostUsd !== undefined) { + return { capUsd: policy.maxCostUsd, source: 'policy' }; + } + return { capUsd: Infinity, source: 'uncapped' }; + } + + const policyCap = policy.maxCostUsd; + const trackerCap = tracker ? trackerHeadroom(tracker) : null; + + if (policyCap === undefined && trackerCap === null) { + // D3 fail-closed: neither tracker nor explicit cap → abort. + return null; + } + if (policyCap !== undefined && trackerCap === null) { + return { capUsd: policyCap, source: 'policy' }; + } + if (policyCap === undefined && trackerCap !== null) { + return { capUsd: trackerCap, source: 'tracker' }; + } + // Both set: the lower wins (caller can voluntarily cap tighter). + const minCap = Math.min(policyCap!, trackerCap!); + return { capUsd: minCap, source: minCap === policyCap ? 'policy' : 'min' }; +} + +/** + * Tracker headroom = max cap - already-spent. Returns null if the + * tracker has no cap configured. + */ +function trackerHeadroom(tracker: BudgetTracker): number | null { + const snapshot = tracker.snapshot(); + if (snapshot.maxCostUsd === undefined) return null; + return Math.max(0, snapshot.maxCostUsd - snapshot.cumulativeCostUsd); +} + +/** + * Detect TTY for interactive grace. Skips when stdin OR stdout is + * not a TTY (e.g. piped output, cron, launchd, Minion workers). + * + * GBRAIN_PROGRESSIVE_BATCH_AUTO=1 forces non-interactive. + */ +function isInteractive(): boolean { + if (process.env.GBRAIN_PROGRESSIVE_BATCH_AUTO === '1') return false; + return Boolean(process.stdin.isTTY && process.stderr.isTTY); +} + +/** + * Wait up to `ms` for a Ctrl-C (SIGINT). Resolves to `true` if + * SIGINT was received (caller should abort). Resolves to `false` + * if the timeout elapsed without interrupt. + * + * Defensively removes its listener on exit so other gbrain handlers + * don't see double-delivery. + * + * Exported for unit tests (call with ms=0 to short-circuit). + */ +export function awaitInteractiveAbort(ms: number): Promise { + if (ms <= 0) return Promise.resolve(false); + return new Promise((resolve) => { + let resolved = false; + const onSigint = () => { + if (resolved) return; + resolved = true; + process.off('SIGINT', onSigint); + clearTimeout(timer); + resolve(true); + }; + const timer = setTimeout(() => { + if (resolved) return; + resolved = true; + process.off('SIGINT', onSigint); + resolve(false); + }, ms); + process.on('SIGINT', onSigint); + }); +} + +/** + * Slice items into per-stage chunks. The final stage gets all + * remaining items (the "full" stage). + * + * Examples: + * - 1000 items, stages=[10,100,500] → [[0..10],[10..110],[110..610],[610..1000]] + * - 50 items, stages=[10,100,500] → [[0..10],[10..50],[],[]] + * (ramp_100 takes all remaining; ramp_500 + full are empty) + * - 5 items, stages=[10,100,500] → [[0..5],[],[],[]] + * - 1000 items, stages=[] → [[],[],[],[0..1000]] (skip ramp) + * + * Empty slices result in a fast-pass through that stage (verdict + * proceed, zero cost, zero delta). + * + * Exported for unit tests. + */ +export function sliceIntoStages( + items: T[], + stages: number[], +): { trial: T[]; ramp_100: T[]; ramp_500: T[]; full: T[] } { + const result: { trial: T[]; ramp_100: T[]; ramp_500: T[]; full: T[] } = { + trial: [], + ramp_100: [], + ramp_500: [], + full: [], + }; + // If stages array is empty, dump everything into 'full'. + if (stages.length === 0) { + result.full = items.slice(); + return result; + } + let cursor = 0; + const stageNames: ('trial' | 'ramp_100' | 'ramp_500')[] = [ + 'trial', + 'ramp_100', + 'ramp_500', + ]; + for (let i = 0; i < stageNames.length; i++) { + const stageCount = stages[i]; + if (stageCount === undefined) break; // policy provided fewer than 3 ramp stages + const next = Math.min(cursor + stageCount, items.length); + result[stageNames[i]] = items.slice(cursor, next); + cursor = next; + } + result.full = items.slice(cursor); + return result; +} + +/** + * Verifier-specific delta computation. Returns + * { observed, expected, verdict, reasons? }. The orchestrator passes + * the verdict directly to the stage report. + */ +async function evaluateVerifier( + verifier: Verifier, + stage: Stage, + itemsThisStage: number, + countsBefore: number, + mutationsBefore: number, +): Promise<{ + observed?: number; + expected?: number | null; + qualityVerdict: { ok: boolean; reasons?: string[] }; + verdict: StageVerdict; + reason?: AbortReason; + newMutationsBefore?: number; +}> { + let observed: number | undefined; + let expected: number | null = null; + let countMismatchVerdict: StageVerdict | null = null; + let countMismatchReason: AbortReason | undefined; + let newMutationsBefore: number | undefined; + + switch (verifier.kind) { + case 'output_count': { + const after = await verifier.countAfter(); + observed = after - countsBefore; + expected = verifier.expectedDelta(itemsThisStage); + if (expected !== null) { + // ±10% band (we deliberately don't make this configurable in + // v1; D11 quick_reject + D18 priority scoring should keep + // observed close to expected on real corpora). + const lower = Math.floor(expected * 0.9); + const upper = Math.ceil(expected * 1.1); + if (observed < lower || observed > upper) { + countMismatchVerdict = 'abort_count_mismatch'; + countMismatchReason = 'count_delta_outside_band'; + } + } + break; + } + case 'idempotent_mutation': { + // Delta semantics: caller's mutatedCount() returns the CUMULATIVE + // mutation counter; the orchestrator computes per-stage delta + // against the prior snapshot. Mirrors output_count's + // countBefore/countAfter pattern (D20 #1 fix: the v1 absolute- + // comparison shape was wrong on stage 2+). + const mutated = await verifier.mutatedCount(); + observed = mutated - mutationsBefore; + const expectedFn = verifier.expectedMutations ?? ((p: number) => p); + expected = expectedFn(itemsThisStage); + newMutationsBefore = mutated; + if (expected !== null) { + const lower = Math.floor(expected * 0.9); + const upper = Math.ceil(expected * 1.1); + if (observed < lower || observed > upper) { + countMismatchVerdict = 'abort_mutation_mismatch'; + countMismatchReason = 'mutation_count_outside_band'; + } + } + break; + } + case 'noop': { + // No count check; just maybe sampleQuality. + break; + } + } + + // Quality sample: only for verifiers that opt in (noop is optional). + let qualityVerdict: { ok: boolean; reasons?: string[] } = { ok: true }; + if ( + verifier.kind === 'output_count' || + verifier.kind === 'idempotent_mutation' || + (verifier.kind === 'noop' && verifier.sampleQuality !== undefined) + ) { + try { + qualityVerdict = + verifier.kind === 'noop' + ? await verifier.sampleQuality!() + : await verifier.sampleQuality(); + } catch (err) { + qualityVerdict = { + ok: false, + reasons: [`sampleQuality threw: ${(err as Error).message}`], + }; + } + } + + if (countMismatchVerdict !== null) { + return { + observed, + expected, + qualityVerdict, + verdict: countMismatchVerdict, + reason: countMismatchReason, + newMutationsBefore, + }; + } + if (!qualityVerdict.ok) { + return { + observed, + expected, + qualityVerdict, + verdict: 'abort_data_quality', + reason: 'data_quality_sample_failed', + newMutationsBefore, + }; + } + return { + observed, + expected, + qualityVerdict, + verdict: 'proceed', + newMutationsBefore, + }; +} + +/** + * Run a batch through progressive stages with verification + cost + * gating + audit. See module header for full contract. + * + * Generic over the caller's item type `T`. The orchestrator never + * inspects items directly; the caller's `StageRunner` does. + * + * Always returns a `ProgressiveBatchResult`. Throws only on + * unrecoverable conditions (verifier itself throwing during pre-flight + * `countBefore`). Stage-time exceptions inside the runner are caught + * and reported via the per-stage error rate; the orchestrator may + * still proceed if rate stays under cap. + */ +export async function runProgressiveBatch( + items: T[], + verifier: Verifier, + policy: Policy, + runner: StageRunner, +): Promise { + const operationId = policy.operationId ?? generateOperationId(); + const label = policy.label; + const totalItems = items.length; + const startedAt = Date.now(); + const stageReports: StageReport[] = []; + let cumulativeProcessed = 0; + let cumulativeCost = 0; + let cumulativeFailures = 0; + let cumulativeAttempts = 0; + + // Resolve env-driven disable BEFORE anything else. + const disabled = process.env.GBRAIN_PROGRESSIVE_BATCH_DISABLED === '1'; + if (disabled) { + process.stderr.write( + `[progressive-batch] DISABLED via GBRAIN_PROGRESSIVE_BATCH_DISABLED=1 — skipping ramp for label=${label}\n`, + ); + } + + const tracker = getCurrentBudgetTracker(); + const capResult = resolveCostCap(policy, tracker); + + // D3 fail-closed safety net. + if (capResult === null) { + const report: StageReport = { + operationId, + label, + stage: 'trial', + itemsInStage: 0, + itemsProcessedCumulative: 0, + totalItems, + verdict: 'abort_cost_cap', + abortReason: 'no_budget_safety_net', + errorRate: 0, + costEstimateRunningUsd: 0, + costProjectedFullUsd: 0, + stageMs: 0, + }; + stageReports.push(report); + logProgressiveBatchEvent({ + operation_id: operationId, + label, + stage: 'trial', + items_in_stage: 0, + items_processed_cumulative: 0, + total_items: totalItems, + verdict: 'abort_cost_cap', + abort_reason: 'no_budget_safety_net', + error_rate: 0, + cost_running_usd: 0, + cost_projected_full_usd: 0, + stage_ms: 0, + }); + await emitStageReport(policy, report); + return { + operationId, + label, + totalItems, + itemsProcessed: 0, + stagesCompleted: [], + abortedAt: { + stage: 'trial', + verdict: 'abort_cost_cap', + reason: 'no_budget_safety_net', + }, + totalCostUsd: 0, + durationMs: Date.now() - startedAt, + stageReports, + }; + } + + const effectiveCapUsd = capResult.capUsd; + + // Capture pre-flight countBefore for output-count verifiers — used + // by every stage so we can compute cumulative delta correctly. + // For idempotent_mutation we also track a per-stage mutation snapshot + // (the orchestrator updates it after each stage's verifier call). + let countBefore = 0; + let mutationsBefore = 0; + if (verifier.kind === 'output_count') { + countBefore = await verifier.countBefore(); + } else if (verifier.kind === 'idempotent_mutation') { + mutationsBefore = await verifier.mutatedCount(); + } + + // Determine the stage list. If disabled, only 'full' runs (slice all + // into 'full', skip ramp). + const stages = disabled ? [] : resolveStages(policy); + const slices = sliceIntoStages(items, stages); + const stageSequence: { stage: Stage; slice: T[] }[] = [ + { stage: 'trial', slice: slices.trial }, + { stage: 'ramp_100', slice: slices.ramp_100 }, + { stage: 'ramp_500', slice: slices.ramp_500 }, + { stage: 'full', slice: slices.full }, + ]; + + const stagesCompleted: Stage[] = []; + const interactiveMs = policy.interactiveAbortMs ?? 0; + const interactiveOn = interactiveMs > 0 && isInteractive() && !disabled; + const maxErrorRate = policy.maxErrorRate ?? 0.02; + + for (let i = 0; i < stageSequence.length; i++) { + const { stage, slice } = stageSequence[i]; + if (slice.length === 0) { + // Fast-pass: empty stage. Don't write an audit event for noise + // suppression unless the stage is 'full' AND total items is 0 + // (which is a degenerate but observable case). + if (stage === 'full' && totalItems === 0) { + // Single audit row to mark the run happened with zero items. + const r: StageReport = { + operationId, + label, + stage, + itemsInStage: 0, + itemsProcessedCumulative: 0, + totalItems: 0, + verdict: 'proceed', + errorRate: 0, + costEstimateRunningUsd: 0, + costProjectedFullUsd: 0, + stageMs: 0, + }; + stageReports.push(r); + await emitStageReport(policy, r); + logProgressiveBatchEvent({ + operation_id: operationId, + label, + stage, + items_in_stage: 0, + items_processed_cumulative: 0, + total_items: 0, + verdict: 'proceed', + error_rate: 0, + cost_running_usd: 0, + cost_projected_full_usd: 0, + stage_ms: 0, + }); + } + continue; + } + + // Run this stage's runner. + const stageStart = Date.now(); + let succeeded = 0; + let failed = 0; + let stageCost = 0; + try { + const out = await runner(slice, stage, operationId); + succeeded = out.succeeded; + failed = out.failed; + stageCost = out.costUsd; + } catch (err) { + // Whole-stage throw: treat as all-failed at the current slice + // size. Surface in audit. + failed = slice.length; + stageCost = 0; + process.stderr.write( + `[progressive-batch] runner threw on stage=${stage} op=${operationId.slice(0, 8)}: ${(err as Error).message}\n`, + ); + } + const stageMs = Date.now() - stageStart; + cumulativeProcessed += slice.length; + cumulativeFailures += failed; + cumulativeAttempts += slice.length; + cumulativeCost += stageCost; + const observedErrorRate = + cumulativeAttempts > 0 ? cumulativeFailures / cumulativeAttempts : 0; + + // Cost projection: extrapolate from cost-per-item observed so far + // (more honest than verifier.costPerItem after the first stage). + const observedCostPerItem = + cumulativeProcessed > 0 ? cumulativeCost / cumulativeProcessed : 0; + const remainingItems = totalItems - cumulativeProcessed; + const projectedRemainingCost = observedCostPerItem * remainingItems; + const projectedFullCost = cumulativeCost + projectedRemainingCost; + + // Verifier gate. + const vEval = await evaluateVerifier( + verifier, + stage, + slice.length, + countBefore, + mutationsBefore, + ); + // Advance the idempotent-mutation snapshot for the next stage. + if (vEval.newMutationsBefore !== undefined) { + mutationsBefore = vEval.newMutationsBefore; + } + + // Combine signals: precedence order is + // 1. verifier abort (count/mutation/quality) + // 2. error rate abort + // 3. cost cap abort (current cumulative > cap, or projection > cap) + // 4. interactive abort (user Ctrl-C) + // 5. caller stage-report abort + // 6. proceed + let verdict: StageVerdict = vEval.verdict; + let abortReason: AbortReason | undefined = vEval.reason; + if (verdict === 'proceed' && observedErrorRate > maxErrorRate) { + verdict = 'abort_error_rate'; + abortReason = 'error_rate_exceeded'; + } + if ( + verdict === 'proceed' && + effectiveCapUsd !== Infinity && + (cumulativeCost > effectiveCapUsd || projectedFullCost > effectiveCapUsd) + ) { + verdict = 'abort_cost_cap'; + abortReason = 'cost_projected_over_cap'; + } + + const report: StageReport = { + operationId, + label, + stage, + itemsInStage: slice.length, + itemsProcessedCumulative: cumulativeProcessed, + totalItems, + verdict, + abortReason, + errorRate: observedErrorRate, + costEstimateRunningUsd: cumulativeCost, + costProjectedFullUsd: projectedFullCost, + deltaObserved: vEval.observed, + deltaExpected: vEval.expected, + stageMs, + qualityReasons: + vEval.qualityVerdict.reasons && vEval.qualityVerdict.reasons.length > 0 + ? vEval.qualityVerdict.reasons + : undefined, + }; + + // Emit report (caller may signal abort). + const callerResp = await emitStageReport(policy, report); + if (callerResp?.abort && verdict === 'proceed') { + verdict = 'abort_explicit'; + abortReason = 'caller_signaled_abort'; + report.verdict = verdict; + report.abortReason = abortReason; + } + + stageReports.push(report); + logProgressiveBatchEvent({ + operation_id: operationId, + label, + stage, + items_in_stage: slice.length, + items_processed_cumulative: cumulativeProcessed, + total_items: totalItems, + verdict, + abort_reason: abortReason, + error_rate: observedErrorRate, + cost_running_usd: cumulativeCost, + cost_projected_full_usd: projectedFullCost, + delta_observed: vEval.observed, + delta_expected: vEval.expected, + stage_ms: stageMs, + quality_reasons: report.qualityReasons, + }); + + if (verdict !== 'proceed') { + return { + operationId, + label, + totalItems, + itemsProcessed: cumulativeProcessed, + stagesCompleted, + abortedAt: { stage, verdict, reason: abortReason }, + totalCostUsd: cumulativeCost, + durationMs: Date.now() - startedAt, + stageReports, + }; + } + + stagesCompleted.push(stage); + + // Interactive grace window between stages (NOT after 'full'). + if (interactiveOn && i < stageSequence.length - 1) { + const nextSlice = stageSequence[i + 1].slice; + if (nextSlice.length > 0) { + process.stderr.write( + `[progressive-batch] ${stage} complete; ${nextSlice.length} items in next stage. ` + + `Press Ctrl-C within ${interactiveMs}ms to abort.\n`, + ); + const aborted = await awaitInteractiveAbort(interactiveMs); + if (aborted) { + const r: StageReport = { + operationId, + label, + stage, + itemsInStage: 0, + itemsProcessedCumulative: cumulativeProcessed, + totalItems, + verdict: 'abort_user', + abortReason: 'user_aborted', + errorRate: observedErrorRate, + costEstimateRunningUsd: cumulativeCost, + costProjectedFullUsd: projectedFullCost, + stageMs: 0, + }; + stageReports.push(r); + await emitStageReport(policy, r); + logProgressiveBatchEvent({ + operation_id: operationId, + label, + stage, + items_in_stage: 0, + items_processed_cumulative: cumulativeProcessed, + total_items: totalItems, + verdict: 'abort_user', + abort_reason: 'user_aborted', + error_rate: observedErrorRate, + cost_running_usd: cumulativeCost, + cost_projected_full_usd: projectedFullCost, + stage_ms: 0, + }); + return { + operationId, + label, + totalItems, + itemsProcessed: cumulativeProcessed, + stagesCompleted, + abortedAt: { stage, verdict: 'abort_user', reason: 'user_aborted' }, + totalCostUsd: cumulativeCost, + durationMs: Date.now() - startedAt, + stageReports, + }; + } + } + } + } + + return { + operationId, + label, + totalItems, + itemsProcessed: cumulativeProcessed, + stagesCompleted, + totalCostUsd: cumulativeCost, + durationMs: Date.now() - startedAt, + stageReports, + }; +} + +/** + * Invoke caller's stage-report callback if provided, otherwise the + * default stderr writer. Honors a `Promise` + * shape uniformly. + */ +async function emitStageReport( + policy: Policy, + report: StageReport, +): Promise<{ abort?: boolean; rewriteRemainingStages?: number[] } | undefined> { + const handler = policy.onStageReport ?? ((r: StageReport) => defaultStageReport(r)); + try { + const r = await handler(report); + if (r === undefined || r === null) return undefined; + return r; + } catch (err) { + process.stderr.write( + `[progressive-batch] onStageReport threw (continuing): ${(err as Error).message}\n`, + ); + return undefined; + } +} + +/** + * Re-export the public types so callers can do + * `import { runProgressiveBatch, type Verifier } from 'gbrain/progressive-batch'` + * without dragging in the internal audit/types/stage-report modules. + */ +export type { + AbortReason, + IdempotentMutationVerifier, + NoopVerifier, + OutputCountVerifier, + Policy, + ProgressiveBatchResult, + QualityVerdict, + Stage, + StageReport, + StageReportResponse, + StageRunner, + StageVerdict, + Verifier, +} from './types.ts'; diff --git a/src/core/progressive-batch/retrofit-wrap.ts b/src/core/progressive-batch/retrofit-wrap.ts new file mode 100644 index 000000000..c7c282ea0 --- /dev/null +++ b/src/core/progressive-batch/retrofit-wrap.ts @@ -0,0 +1,78 @@ +/** + * v0.41.16.0 — Retrofit wrap helper for behavior-parity migrations. + * + * Per D21 + D26: existing batch sites that need only the primitive's + * audit + cost-cap gating (NOT its ramp + verification) get a thin + * wrapper. The wrapper: + * - Buffers items into the work list (one-time read). + * - Wraps the per-item callable in NoopVerifier + interactiveAbortMs=0 + * + opt-out safety net (existing sites that previously had no + * ramp + no budget gate keep that behavior; the primitive's audit + * JSONL + cost projection is the value-add). + * - Surfaces aborts via stderr but doesn't change the caller's + * return shape. + * + * For sites that NEED ramp (the markdown reindex flow via + * post-upgrade-reembed), callers opt INTO interactive abort via the + * primitive's `interactiveAbortMs` Policy field or the + * `GBRAIN_PROGRESSIVE_BATCH_STAGES` env var. + * + * Hermetic: pure orchestration. No I/O of its own. + */ + +import { + runProgressiveBatch, + type NoopVerifier, + type Policy, + type ProgressiveBatchResult, + type StageRunner, +} from './orchestrator.ts'; + +export interface RetrofitWrapOpts { + /** Display label for audit JSONL + stderr stage report. */ + label: string; + /** The full work list. */ + items: T[]; + /** Per-item conservative cost projection (USD). 0 = no LLM/embed cost. */ + costPerItem?: number; + /** Per-batch runner. Returns {succeeded, failed, costUsd} for this slice. */ + runner: StageRunner; + /** + * D21: behavior parity. When the existing site previously had no + * Ctrl-C grace AND no budget gating, default to opt-out. Operators + * opt INTO ramp via `GBRAIN_PROGRESSIVE_BATCH_STAGES` env var. + */ + interactiveAbortMs?: number; + /** + * D3: when the existing site previously had its own BudgetTracker + * (e.g. brainstorm/orchestrator, reindex-code with --max-cost), it + * still owns the tracker; primitive observes via getCurrentBudgetTracker. + * When the site never had cost gating, default opt-out so the + * primitive doesn't refuse to start. + */ + requireBudgetSafetyNet?: boolean; +} + +/** + * Thin retrofit wrapper. Use for any existing batch site that doesn't + * need ramp + verification (just wants the primitive's audit JSONL + + * cost projection). + * + * Returns the primitive's full result so callers can read + * `result.abortedAt` and `result.stageReports` if they need to. Most + * callers ignore the return. + */ +export async function retrofitWrap( + opts: RetrofitWrapOpts, +): Promise { + const verifier: NoopVerifier = { + kind: 'noop', + costPerItem: () => opts.costPerItem ?? 0, + }; + const policy: Policy = { + label: opts.label, + interactiveAbortMs: opts.interactiveAbortMs ?? 0, + requireBudgetSafetyNet: opts.requireBudgetSafetyNet ?? false, + }; + return runProgressiveBatch(opts.items, verifier, policy, opts.runner); +} diff --git a/src/core/progressive-batch/stage-report.ts b/src/core/progressive-batch/stage-report.ts new file mode 100644 index 000000000..07a2c0683 --- /dev/null +++ b/src/core/progressive-batch/stage-report.ts @@ -0,0 +1,52 @@ +/** + * v0.41.16.0 — Stage report formatter. + * + * Pure formatter for human + JSON stage reports. The orchestrator + * invokes this from its default `Policy.onStageReport` handler when + * the caller doesn't override. + * + * ASCII-only per the storage.ts D10 precedent (operator-friendly + * across every terminal). One report per stage, written to stderr. + */ + +import type { StageReport } from './types.ts'; + +/** + * Pure ASCII single-line report for a stage. Suitable for stderr. + * + * [progressive-batch label=foo op=ab12 stage=trial verdict=proceed + * items=10/250 err=0.0% cost=$0.0042 proj=$0.105 dt=453ms] + */ +export function formatStageLine(r: StageReport): string { + const parts: string[] = [ + `label=${r.label}`, + `op=${r.operationId.slice(0, 8)}`, + `stage=${r.stage}`, + `verdict=${r.verdict}`, + `items=${r.itemsProcessedCumulative}/${r.totalItems}`, + `err=${(r.errorRate * 100).toFixed(1)}%`, + `cost=$${r.costEstimateRunningUsd.toFixed(4)}`, + `proj=$${r.costProjectedFullUsd.toFixed(4)}`, + `dt=${r.stageMs}ms`, + ]; + if (r.deltaObserved !== undefined) { + parts.push( + `delta=${r.deltaObserved}/${r.deltaExpected ?? 'n/a'}`, + ); + } + if (r.abortReason) parts.push(`reason=${r.abortReason}`); + if (r.qualityReasons && r.qualityReasons.length > 0) { + parts.push(`quality_issues=${r.qualityReasons.length}`); + } + return `[progressive-batch ${parts.join(' ')}]`; +} + +/** + * Default stderr writer. Caller overrides via Policy.onStageReport. + */ +export function defaultStageReport(r: StageReport): void { + // We want this on stderr so stdout JSON envelopes from the caller + // (e.g. `gbrain reindex --json`) stay clean. + process.stderr.write(formatStageLine(r) + '\n'); +} + diff --git a/src/core/progressive-batch/types.ts b/src/core/progressive-batch/types.ts new file mode 100644 index 000000000..e0da34282 --- /dev/null +++ b/src/core/progressive-batch/types.ts @@ -0,0 +1,288 @@ +/** + * v0.41.16.0 — Progressive-batch primitive type surface. + * + * Shared types for `runProgressiveBatch` and its consumers. Lives in its + * own file so type-only imports don't drag the orchestrator's runtime + * dependencies (AsyncLocalStorage, audit-writer) into pure callers. + * + * Design constraints (eng review D2, D3, D4, D20, D21): + * + * - D2: extracted as a first-class primitive after rule-of-three + * satisfied across 12+ ad-hoc batch sites in gbrain. + * + * - D3: budget gate is fail-CLOSED. Primitive reads + * `getCurrentBudgetTracker()` from `src/core/ai/gateway.ts` ahead + * of `Policy.maxCostUsd`. When BOTH are null, the primitive + * aborts with `reason: 'no_budget_safety_net'` rather than + * silently running unbounded. + * + * - D4: primitive uses `Stage` (corpus-rollout axis). Parser uses + * `ParsePhase` (per-page axis). Different audit JSONLs. Do NOT + * conflate; they're orthogonal. + * + * - D20: Verifier is a discriminated union. Output-count verifiers + * (parser, contradiction-eval) AND idempotent-mutation verifiers + * (reindex, embed) AND noop verifiers (sites that want only ramp + * + cost gating) all fit cleanly. No round-peg-square-hole. + * + * - D21: Sites without an existing ramp keep jump-to-full as + * default. Ramp is opt-in per-site via `Policy.interactiveAbortMs + * > 0`. Honest behavior preservation. + */ + +/** + * The fixed stages of a progressive-batch run. The actual item counts + * per stage are configured via `Policy.stages` (default: [10, 100, 500] + * with `full` implicit). + * + * `trial` runs first against the smallest slice. `ramp_100` and + * `ramp_500` are intermediate gates. `full` processes whatever items + * remain after the earlier stages. + */ +export type Stage = 'trial' | 'ramp_100' | 'ramp_500' | 'full'; + +/** + * Verdict from a verifier or from the orchestrator's own gating. + * Exhaustive TypeScript union — every consumer site should `switch` + * with a `never` default to catch future additions at compile time. + */ +export type StageVerdict = + | 'proceed' // All checks passed; advance to next stage + | 'abort_data_quality' // sampleQuality returned ok=false + | 'abort_count_mismatch' // OutputCountVerifier: actualDelta outside expected band + | 'abort_mutation_mismatch' // IdempotentMutationVerifier: rows-modified count off + | 'abort_error_rate' // observed error rate > Policy.maxErrorRate + | 'abort_cost_cap' // projected full-batch cost > effective cap + | 'abort_user' // Ctrl-C during interactive grace window + | 'abort_explicit'; // caller's onStageReport returned {abort: true} + +/** + * Reason code attached to abort verdicts. Surfaces in audit JSONL and + * stderr reports. The two budget reasons (`cost_projected_over_cap` + * and `no_budget_safety_net`) are the load-bearing ones for the D3 + * fail-closed contract. + */ +export type AbortReason = + | 'data_quality_sample_failed' + | 'count_delta_outside_band' + | 'mutation_count_outside_band' + | 'error_rate_exceeded' + | 'cost_projected_over_cap' + | 'no_budget_safety_net' // D3: neither tracker nor Policy.maxCostUsd set + | 'user_aborted' + | 'caller_signaled_abort'; + +/** + * Quality verdict returned by `sampleQuality()`. Free-form `reasons` + * surface in the audit JSONL when `ok: false`. + */ +export interface QualityVerdict { + ok: boolean; + reasons?: string[]; +} + +/** + * Verifier shape #1 — caller's batch produces NEW rows (parser, + * contradiction-eval, atom extraction, etc.). The verifier knows how + * many rows SHOULD appear after processing N items and where to count + * them. + * + * Used by retrofit sites where row-count is the natural success + * signal. + */ +export interface OutputCountVerifier { + kind: 'output_count'; + /** Pre-stage row count. */ + countBefore(): Promise; + /** Post-stage row count. */ + countAfter(): Promise; + /** + * How many rows SHOULD appear after processing `processed` items. + * Returning `null` skips count-mismatch gating for this stage + * (caller has stage-specific reasons; e.g. dry-run inserts). + */ + expectedDelta(processed: number): number | null; + /** Sample ≤3 random output rows, return ok/not-ok + reasons. */ + sampleQuality(): Promise; + /** Cost projection per item at the given stage. */ + costPerItem(stage: Stage): number; +} + +/** + * Verifier shape #2 — caller's batch UPDATES existing rows in-place + * (reindex, embed, content-hash refresh). Row-count delta is always + * zero; what matters is rows-mutated. + * + * D20: codex outside-voice correctly flagged that + * `expectedDelta(processed)` doesn't fit reindex semantics. This + * verifier shape closes that gap. + */ +export interface IdempotentMutationVerifier { + kind: 'idempotent_mutation'; + /** + * Count rows-mutated by the stage's batch. Caller defines what + * "mutated" means (chunker_version bump, embedding_at refresh, + * etc.). Default expectation: equals processed-item count. + */ + mutatedCount(): Promise; + /** + * How many mutations SHOULD have happened after processing + * `processed` items. Defaults to `processed` (1:1) if not + * overridden. Returning `null` skips mutation-count gating. + */ + expectedMutations?(processed: number): number | null; + /** Sample ≤3 random mutated rows, return ok/not-ok + reasons. */ + sampleQuality(): Promise; + /** Cost projection per item at the given stage. */ + costPerItem(stage: Stage): number; +} + +/** + * Verifier shape #3 — caller wants ramp + cost gating but NO output + * verification. Common for sites that previously "jumped straight to + * full" (per D21) where adding count verification would be a behavior + * change beyond what the retrofit promised. + * + * Still requires `costPerItem` so the cost cap can be projected + * honestly. + */ +export interface NoopVerifier { + kind: 'noop'; + /** Cost projection per item at the given stage. */ + costPerItem(stage: Stage): number; + /** + * Optional quality probe. When omitted, sampleQuality is treated as + * `{ok: true}` at every stage. Useful for ramp+cost-only callers + * that still want a structural smoke check. + */ + sampleQuality?(): Promise; +} + +/** Discriminated union of all three verifier shapes. */ +export type Verifier = + | OutputCountVerifier + | IdempotentMutationVerifier + | NoopVerifier; + +/** + * Per-stage report passed to the caller's optional `onStageReport` + * callback (and emitted to stderr by the default reporter). Stable + * shape: callers can persist these in their own audit trails without + * re-deriving from the primitive's JSONL. + */ +export interface StageReport { + operationId: string; + label: string; + stage: Stage; + itemsInStage: number; + itemsProcessedCumulative: number; + totalItems: number; + verdict: StageVerdict; + abortReason?: AbortReason; + errorRate: number; + costEstimateRunningUsd: number; + costProjectedFullUsd: number; + /** Verifier-specific deltas (count, mutation, etc.). */ + deltaObserved?: number; + deltaExpected?: number | null; + /** Wall-clock for this stage's processing. */ + stageMs: number; + /** Per-stage quality verdict reasons (when sampleQuality returned not-ok). */ + qualityReasons?: string[]; +} + +/** + * Caller's response to onStageReport. Returning `{abort: true}` halts + * the run with verdict `abort_explicit`. Returning anything else + * (including undefined) is implicit "proceed". + */ +export interface StageReportResponse { + abort?: boolean; + /** Operator override on the stage list for the next stage (rare). */ + rewriteRemainingStages?: number[]; +} + +/** + * Per-batch policy. Mostly defaults; callers override per-site. + */ +export interface Policy { + /** Max observed error rate before abort. Default: 0.02 (2%). */ + maxErrorRate?: number; + /** + * Effective USD cap. When set, the primitive applies it as the + * upper bound regardless of any active BudgetTracker (the lower of + * the two wins — caller can never EXCEED the tracker's cap, but + * can voluntarily cap themselves tighter). + * + * When unset AND no active BudgetTracker, the primitive ABORTS at + * stage 0 with `abort_cost_cap reason='no_budget_safety_net'` + * (D3 fail-closed). Caller must explicitly opt out of safety with + * `requireBudgetSafetyNet: false`. + */ + maxCostUsd?: number; + /** + * Caller opt-out of the D3 safety net. Sites that legitimately + * don't need cost gating (cheap deterministic ops) set this true. + * Documented in the retrofit's commit message + tests. + * Default: false (safety net required). + */ + requireBudgetSafetyNet?: boolean; + /** + * Interactive Ctrl-C grace window per stage transition, in ms. + * 0 (default) means no grace window (CI / non-TTY / Minion workers). + * Honored only when TTY is detected; non-TTY callers always skip. + * + * D21: sites that previously "jumped straight to full" keep + * `interactiveAbortMs: 0` so behavior parity holds; the ramp itself + * is opt-IN per-site via this flag. + */ + interactiveAbortMs?: number; + /** + * Stage item counts. Default: [10, 100, 500] with `full` implicit. + * Override via `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` env or + * per-call. Empty array = skip ramp, go straight to `full`. + */ + stages?: number[]; + /** + * Per-stage reporter. Default: writes ASCII stage line to stderr. + * Returns optional response to influence the orchestrator. + */ + onStageReport?(report: StageReport): Promise | StageReportResponse | void; + /** Phase/label for audit + telemetry. */ + label: string; + /** + * Caller-supplied operation id for tracing across stages. Defaults + * to a generated short id at orchestrator entry. + */ + operationId?: string; +} + +/** Final result of a complete `runProgressiveBatch` invocation. */ +export interface ProgressiveBatchResult { + operationId: string; + label: string; + totalItems: number; + itemsProcessed: number; + stagesCompleted: Stage[]; + abortedAt?: { stage: Stage; verdict: StageVerdict; reason?: AbortReason }; + totalCostUsd: number; + durationMs: number; + /** All per-stage reports in order. */ + stageReports: StageReport[]; +} + +/** + * The per-stage callback the caller provides. Receives the slice for + * THIS stage; returns the number of items successfully processed + * (used for error-rate calculation) AND a per-stage cost actual + * (used for cumulative-cost tracking). + * + * Callers process the slice however they like (serial, batched, + * concurrent). The primitive doesn't dictate; it only sequences + * stages and gates between them. + */ +export type StageRunner = ( + items: T[], + stage: Stage, + operationId: string, +) => Promise<{ succeeded: number; failed: number; costUsd: number }>; diff --git a/test/conversation-parser-cli.test.ts b/test/conversation-parser-cli.test.ts new file mode 100644 index 000000000..ef5686b0e --- /dev/null +++ b/test/conversation-parser-cli.test.ts @@ -0,0 +1,197 @@ +/** + * v0.41.16.0 — `gbrain conversation-parser` debug CLI tests. + * + * Pins: + * - list-builtins prints all 12 patterns + accepts --json + * - validate emits "deferred to v0.42+" notice (v1 has no compiler) + * - scan errors without engine + * - --help works + * + * Pure-function tests; no engine, no DB. + */ + +import { describe, expect, test, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { runConversationParser } from '../src/commands/conversation-parser.ts'; +import { BUILTIN_PATTERNS } from '../src/core/conversation-parser/builtins.ts'; + +// Capture process.stdout.write + process.stderr.write + process.exit. +function captureStdio() { + const out: string[] = []; + const err: string[] = []; + let exitCode: number | null = null; + const origOut = process.stdout.write.bind(process.stdout); + const origErr = process.stderr.write.bind(process.stderr); + const origExit = process.exit.bind(process); + process.stdout.write = ((chunk: string | Uint8Array) => { + out.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString()); + return true; + }) as typeof process.stdout.write; + process.stderr.write = ((chunk: string | Uint8Array) => { + err.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString()); + return true; + }) as typeof process.stderr.write; + // process.exit throws to short-circuit; tests catch and inspect. + process.exit = ((code?: number) => { + exitCode = code ?? 0; + throw new Error(`PROCESS_EXIT:${exitCode}`); + }) as typeof process.exit; + return { + out, + err, + getExitCode: () => exitCode, + restore: () => { + process.stdout.write = origOut; + process.stderr.write = origErr; + process.exit = origExit; + }, + }; +} + +describe('runConversationParser — help', () => { + test('--help prints usage', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, ['--help']); + } finally { + cap.restore(); + } + const text = cap.out.join(''); + expect(text).toContain('Usage: gbrain conversation-parser'); + expect(text).toContain('scan'); + expect(text).toContain('list-builtins'); + expect(text).toContain('validate'); + }); + + test('no subcommand prints help', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, []); + } finally { + cap.restore(); + } + expect(cap.out.join('')).toContain('Usage: gbrain conversation-parser'); + }); +}); + +describe('runConversationParser — list-builtins', () => { + test('human output includes all 12 pattern ids', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, ['list-builtins']); + } finally { + cap.restore(); + } + const text = cap.out.join(''); + for (const pattern of BUILTIN_PATTERNS) { + expect(text).toContain(pattern.id); + } + expect(text).toContain(`${BUILTIN_PATTERNS.length} built-in patterns`); + }); + + test('--json output is stable schema', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, ['list-builtins', '--json']); + } finally { + cap.restore(); + } + const json = JSON.parse(cap.out.join('').trim()); + expect(json.schema_version).toBe(1); + expect(json.total).toBe(BUILTIN_PATTERNS.length); + expect(json.patterns).toHaveLength(BUILTIN_PATTERNS.length); + expect(json.patterns[0].id).toBe('imessage-slack'); + expect(json.patterns[0].date_source).toBeDefined(); + expect(json.patterns[0].time_format).toBeDefined(); + expect(json.patterns[0].timezone_policy).toBeDefined(); + expect(json.patterns[0].regex).toBeDefined(); + }); +}); + +describe('runConversationParser — validate', () => { + test('emits deferred notice (v0.42+ scope)', async () => { + const dir = mkdtempSync(join(tmpdir(), 'cv-parser-cli-')); + const path = join(dir, 'pattern.json'); + writeFileSync(path, '{}'); + const cap = captureStdio(); + try { + await runConversationParser(null, ['validate', path]); + } finally { + cap.restore(); + } + expect(cap.out.join('')).toContain('deferred to v0.42+'); + }); + + test('--json validate emits structured deferred envelope', async () => { + const dir = mkdtempSync(join(tmpdir(), 'cv-parser-cli-')); + const path = join(dir, 'pattern.json'); + writeFileSync(path, '{}'); + const cap = captureStdio(); + try { + await runConversationParser(null, ['validate', path, '--json']); + } finally { + cap.restore(); + } + const json = JSON.parse(cap.out.join('').trim()); + expect(json.schema_version).toBe(1); + expect(json.status).toBe('deferred'); + expect(json.todo_ref).toContain('v0.41.16.0'); + }); + + test('exits 2 on missing file path', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, ['validate']); + } catch (e) { + expect((e as Error).message).toBe('PROCESS_EXIT:2'); + } finally { + cap.restore(); + } + expect(cap.getExitCode()).toBe(2); + }); + + test('exits 2 on nonexistent file', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, [ + 'validate', + '/nonexistent/path.json', + ]); + } catch (e) { + expect((e as Error).message).toBe('PROCESS_EXIT:2'); + } finally { + cap.restore(); + } + expect(cap.getExitCode()).toBe(2); + }); +}); + +describe('runConversationParser — scan', () => { + test('exits 2 when engine is null (scan requires brain)', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, ['scan', 'some-slug']); + } catch (e) { + expect((e as Error).message).toBe('PROCESS_EXIT:2'); + } finally { + cap.restore(); + } + expect(cap.getExitCode()).toBe(2); + }); +}); + +describe('runConversationParser — unknown subcommand', () => { + test('exits 2 with hint', async () => { + const cap = captureStdio(); + try { + await runConversationParser(null, ['bogus-cmd']); + } catch (e) { + expect((e as Error).message).toBe('PROCESS_EXIT:2'); + } finally { + cap.restore(); + } + expect(cap.err.join('')).toContain('unknown subcommand'); + }); +}); diff --git a/test/conversation-parser/helpers.ts b/test/conversation-parser/helpers.ts new file mode 100644 index 000000000..74ffa8e34 --- /dev/null +++ b/test/conversation-parser/helpers.ts @@ -0,0 +1,28 @@ +/** + * v0.41.16.0 — Test helpers for conversation-parser unit tests. + * + * `makeChatResult(text)` builds a fully-typed `ChatResult` stub so + * test transports satisfy the gateway's type contract without each + * test repeating the empty fields. + */ + +import type { ChatResult } from '../../src/core/ai/gateway.ts'; + +export function makeChatResult( + text: string, + usage = { input_tokens: 10, output_tokens: 30 }, +): ChatResult { + return { + text, + blocks: [{ type: 'text', text }], + stopReason: 'end', + usage: { + input_tokens: usage.input_tokens, + output_tokens: usage.output_tokens, + cache_read_tokens: 0, + cache_creation_tokens: 0, + }, + model: 'anthropic:claude-haiku-4-5', + providerId: 'anthropic', + }; +} diff --git a/test/conversation-parser/llm-base.test.ts b/test/conversation-parser/llm-base.test.ts new file mode 100644 index 000000000..c09ececff --- /dev/null +++ b/test/conversation-parser/llm-base.test.ts @@ -0,0 +1,184 @@ +/** + * v0.41.16.0 — LLM base unit tests. + * + * Hermetic via the `chatTransport` test seam. No real API calls. + * + * Pins: + * - Cache hit doesn't re-call transport + * - Provider unavailable returns null without calling transport + * - Transport throw → fail-open null + * - Parse failure → fail-open null + * - parseLlmJson 4-strategy fallback + */ + +import { describe, expect, test, beforeEach } from 'bun:test'; +import { withEnv } from '../helpers/with-env.ts'; +import { + runLlmCall, + parseLlmJson, + _resetLlmCacheForTests, + getLlmCacheStats, + probeLlmAvailability, +} from '../../src/core/conversation-parser/llm-base.ts'; +import { makeChatResult } from './helpers.ts'; + +beforeEach(() => { + _resetLlmCacheForTests(); +}); + +describe('probeLlmAvailability', () => { + test('returns null when ANTHROPIC_API_KEY is unset', async () => { + await withEnv( + { ANTHROPIC_API_KEY: undefined as unknown as string }, + async () => { + expect(probeLlmAvailability('claude-haiku-4-5')).toBeNull(); + expect(probeLlmAvailability('anthropic:claude-haiku-4-5')).toBeNull(); + }, + ); + }); + test('returns normalized model when key set', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + expect(probeLlmAvailability('claude-haiku-4-5')).toBe( + 'anthropic:claude-haiku-4-5', + ); + }); + }); + test('returns null for unknown provider', async () => { + expect(probeLlmAvailability('madeup-provider:foo-model')).toBeNull(); + }); +}); + +describe('runLlmCall — happy path', () => { + test('calls transport, parses, caches', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + let calls = 0; + const result = await runLlmCall<{ ok: boolean }>({ + shape: 'fallback', + modelStr: 'claude-haiku-4-5', + content: 'hello', + system: 'test', + parse: (text) => parseLlmJson(text), + chatTransport: async () => { + calls++; + return makeChatResult('{"ok": true}', { input_tokens: 1, output_tokens: 1 }); + }, + }); + expect(result).toEqual({ ok: true }); + expect(calls).toBe(1); + expect(getLlmCacheStats().misses).toBe(1); + }); + }); + test('cache hit on second call with same content', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + let calls = 0; + const transport = async () => { + calls++; + return makeChatResult('{"ok": true}', { input_tokens: 1, output_tokens: 1 }); + }; + const opts = { + shape: 'fallback' as const, + modelStr: 'claude-haiku-4-5', + content: 'hello', + system: 'test', + parse: (t: string) => parseLlmJson<{ ok: boolean }>(t), + chatTransport: transport, + }; + await runLlmCall(opts); + await runLlmCall(opts); + expect(calls).toBe(1); + expect(getLlmCacheStats().hits).toBe(1); + }); + }); +}); + +describe('runLlmCall — fail-open paths', () => { + test('provider unavailable returns null without calling transport', async () => { + await withEnv( + { ANTHROPIC_API_KEY: undefined as unknown as string }, + async () => { + let calls = 0; + const result = await runLlmCall({ + shape: 'fallback', + modelStr: 'claude-haiku-4-5', + content: 'hello', + system: 'test', + parse: () => ({}), + chatTransport: async () => { + calls++; + return makeChatResult('{}', { input_tokens: 1, output_tokens: 1 }); + }, + }); + expect(result).toBeNull(); + expect(calls).toBe(0); + }, + ); + }); + test('transport throw → fail-open null', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + const result = await runLlmCall({ + shape: 'fallback', + modelStr: 'claude-haiku-4-5', + content: 'hello', + system: 'test', + parse: () => ({}), + chatTransport: async () => { + throw new Error('network down'); + }, + }); + expect(result).toBeNull(); + }); + }); + test('parse failure → fail-open null, not cached', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + let calls = 0; + const opts = { + shape: 'fallback' as const, + modelStr: 'claude-haiku-4-5', + content: 'hello', + system: 'test', + parse: () => null, + chatTransport: async () => { + calls++; + return makeChatResult('garbage', { input_tokens: 1, output_tokens: 1 }); + }, + }; + const r1 = await runLlmCall(opts); + const r2 = await runLlmCall(opts); + expect(r1).toBeNull(); + expect(r2).toBeNull(); + // Both calls hit transport because parse-fail isn't cached. + expect(calls).toBe(2); + }); + }); +}); + +describe('parseLlmJson — 4-strategy fallback', () => { + test('direct parse object', () => { + expect(parseLlmJson<{ a: number }>('{"a": 1}')).toEqual({ a: 1 }); + }); + test('strip ```json fences', () => { + expect(parseLlmJson<{ a: number }>('```json\n{"a": 1}\n```')).toEqual({ a: 1 }); + }); + test('strip bare ``` fences', () => { + expect(parseLlmJson<{ a: number }>('```\n{"a": 1}\n```')).toEqual({ a: 1 }); + }); + test('extract first {...} substring', () => { + expect(parseLlmJson<{ a: number }>('Here is the output: {"a": 1} (done)')).toEqual({ + a: 1, + }); + }); + test('array mode: direct parse', () => { + expect(parseLlmJson('[1,2,3]', { array: true })).toEqual([1, 2, 3]); + }); + test('array mode: extract first [...] substring', () => { + expect(parseLlmJson('Output: [1,2,3] done', { array: true })).toEqual([ + 1, 2, 3, + ]); + }); + test('non-JSON returns null', () => { + expect(parseLlmJson('not json at all')).toBeNull(); + }); + test('empty string returns null', () => { + expect(parseLlmJson('')).toBeNull(); + }); +}); diff --git a/test/conversation-parser/llm-fallback.test.ts b/test/conversation-parser/llm-fallback.test.ts new file mode 100644 index 000000000..cebbed249 --- /dev/null +++ b/test/conversation-parser/llm-fallback.test.ts @@ -0,0 +1,129 @@ +/** + * v0.41.16.0 — LLM fallback unit tests. + * + * Hermetic via the `chatTransport` test seam. + * + * Pins: + * - Happy path: parses LLM-returned JSON array + * - Adversarial input: LLM returns [] → parser returns [] (skip page) + * - Malformed LLM output → fail-open null + * - Provider unavailable → null + * - Cache hit: doesn't re-call + */ + +import { describe, expect, test, beforeEach } from 'bun:test'; +import { withEnv } from '../helpers/with-env.ts'; +import { runLlmFallback } from '../../src/core/conversation-parser/llm-fallback.ts'; +import { _resetLlmCacheForTests } from '../../src/core/conversation-parser/llm-base.ts'; +import { makeChatResult } from './helpers.ts'; + +beforeEach(() => { + _resetLlmCacheForTests(); +}); + +describe('runLlmFallback', () => { + test('happy path: parses LLM JSON output', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + const result = await runLlmFallback({ + modelStr: 'claude-haiku-4-5', + body: 'some-novel-chat-format', + chatTransport: async () => + makeChatResult( + JSON.stringify([ + { speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'hello' }, + { speaker: 'Bob', timestamp: '2024-03-15T18:38:00Z', text: 'world' }, + ]), + { input_tokens: 10, output_tokens: 30 }, + ), + }); + expect(result).not.toBeNull(); + expect(result).toHaveLength(2); + expect(result![0].speaker).toBe('Alice'); + expect(result![1].text).toBe('world'); + }); + }); + + test('adversarial input: LLM returns [] → empty array', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + const result = await runLlmFallback({ + modelStr: 'claude-haiku-4-5', + body: 'This is just a recipe for cookies. Not a chat log.', + chatTransport: async () => makeChatResult('[]', { input_tokens: 10, output_tokens: 1 }), + }); + expect(result).toEqual([]); + }); + }); + + test('malformed output → fail-open null', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + const result = await runLlmFallback({ + modelStr: 'claude-haiku-4-5', + body: 'something', + chatTransport: async () => + makeChatResult( + 'I think this might be a chat log but I am not sure.', + { input_tokens: 10, output_tokens: 12 }, + ), + }); + expect(result).toBeNull(); + }); + }); + + test('provider unavailable: returns null without calling transport', async () => { + await withEnv( + { ANTHROPIC_API_KEY: undefined as unknown as string }, + async () => { + let calls = 0; + const result = await runLlmFallback({ + modelStr: 'claude-haiku-4-5', + body: 'whatever', + chatTransport: async () => { + calls++; + return makeChatResult('[]', { input_tokens: 1, output_tokens: 1 }); + }, + }); + expect(result).toBeNull(); + expect(calls).toBe(0); + }, + ); + }); + + test('cache hit: second call doesnt re-invoke transport', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + let calls = 0; + const opts = { + modelStr: 'claude-haiku-4-5', + body: 'stable-body-for-cache', + chatTransport: async () => { + calls++; + return makeChatResult('[{"speaker":"A","timestamp":"2024-01-01T00:00:00Z","text":"x"}]', { input_tokens: 1, output_tokens: 1 }); + }, + }; + await runLlmFallback(opts); + await runLlmFallback(opts); + expect(calls).toBe(1); + }); + }); + + test('strips invalid items from array', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + const result = await runLlmFallback({ + modelStr: 'claude-haiku-4-5', + body: 'something', + chatTransport: async () => + makeChatResult( + JSON.stringify([ + { speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'good' }, + { speaker: 42, timestamp: 'x', text: 'bad shape' }, // speaker not string + { timestamp: '2024-03-15T18:38:00Z', text: 'missing speaker' }, + { speaker: 'Bob', timestamp: '2024-03-15T18:39:00Z', text: 'good2' }, + ]), + { input_tokens: 10, output_tokens: 30 }, + ), + }); + expect(result).toHaveLength(2); + expect(result![0].speaker).toBe('Alice'); + expect(result![1].speaker).toBe('Bob'); + }); + }); +}); diff --git a/test/conversation-parser/llm-polish.test.ts b/test/conversation-parser/llm-polish.test.ts new file mode 100644 index 000000000..1a872120a --- /dev/null +++ b/test/conversation-parser/llm-polish.test.ts @@ -0,0 +1,174 @@ +/** + * v0.41.16.0 — LLM polish unit tests. + * + * Pins: + * - applyPolish pure function: merge + drop + edits + * - Headroom guard skip path + * - Provider unavailable → input messages unchanged + * - Cache hit doesn't re-call + */ + +import { describe, expect, test, beforeEach } from 'bun:test'; +import { withEnv } from '../helpers/with-env.ts'; +import { + runLlmPolish, + applyPolish, +} from '../../src/core/conversation-parser/llm-polish.ts'; +import { _resetLlmCacheForTests } from '../../src/core/conversation-parser/llm-base.ts'; +import { BudgetTracker } from '../../src/core/budget/budget-tracker.ts'; +import { withBudgetTracker } from '../../src/core/ai/gateway.ts'; +import type { MatchedMessage } from '../../src/core/conversation-parser/types.ts'; +import { makeChatResult } from './helpers.ts'; + +beforeEach(() => { + _resetLlmCacheForTests(); +}); + +const SAMPLE: MatchedMessage[] = [ + { speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'hello' }, + { speaker: 'Alice', timestamp: '2024-03-15T18:37:30Z', text: 'continuation' }, + { speaker: 'system', timestamp: '2024-03-15T18:38:00Z', text: 'Bob joined' }, + { speaker: 'Bob', timestamp: '2024-03-15T18:39:00Z', text: 'world (edited)' }, +]; + +describe('applyPolish — pure function', () => { + test('merge two indices into one', () => { + const r = applyPolish(SAMPLE, { + merge_indices: [[0, 1]], + drop_indices: [], + edits: [], + }); + expect(r.messages).toHaveLength(3); + expect(r.messages[0].text).toBe('hello\ncontinuation'); + expect(r.delta.merged).toBe(1); + }); + + test('drop indices', () => { + const r = applyPolish(SAMPLE, { + merge_indices: [], + drop_indices: [2], + edits: [], + }); + expect(r.messages).toHaveLength(3); + expect(r.messages.find((m) => m.text === 'Bob joined')).toBeUndefined(); + expect(r.delta.dropped).toBe(1); + }); + + test('edit speaker and text', () => { + const r = applyPolish(SAMPLE, { + merge_indices: [], + drop_indices: [], + edits: [ + { index: 3, field: 'text', value: 'world' }, + ], + }); + expect(r.messages[3].text).toBe('world'); + expect(r.delta.edits).toBe(1); + }); + + test('combined: drop system msg, merge two, edit edited marker', () => { + const r = applyPolish(SAMPLE, { + merge_indices: [[0, 1]], + drop_indices: [2], + edits: [{ index: 3, field: 'text', value: 'world' }], + }); + expect(r.messages).toHaveLength(2); + expect(r.messages[0].text).toBe('hello\ncontinuation'); + expect(r.messages[1].text).toBe('world'); + }); + + test('no-op: returns messages unchanged', () => { + const r = applyPolish(SAMPLE, { + merge_indices: [], + drop_indices: [], + edits: [], + }); + expect(r.messages).toHaveLength(4); + expect(r.delta).toEqual({ merged: 0, dropped: 0, edits: 0 }); + }); +}); + +describe('runLlmPolish', () => { + test('happy path: LLM returns polish ops, applied', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + const r = await runLlmPolish({ + modelStr: 'claude-haiku-4-5', + body: '...full body...', + messages: SAMPLE, + patternId: 'imessage-slack', + chatTransport: async () => + makeChatResult( + JSON.stringify({ + merge_indices: [[0, 1]], + drop_indices: [2], + edits: [{ index: 3, field: 'text', value: 'world' }], + }), + { input_tokens: 100, output_tokens: 50 }, + ), + }); + expect(r.skipped).toBeUndefined(); + expect(r.messages).toHaveLength(2); + expect(r.delta.merged).toBe(1); + expect(r.delta.dropped).toBe(1); + expect(r.delta.edits).toBe(1); + }); + }); + + test('headroom guard: skips when tracker within $0.10 of cap', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + const t = new BudgetTracker({ label: 'unit', maxCostUsd: 0.05 }); + let calls = 0; + await withBudgetTracker(t, async () => { + const r = await runLlmPolish({ + modelStr: 'claude-haiku-4-5', + body: 'b', + messages: SAMPLE, + patternId: 'imessage-slack', + chatTransport: async () => { + calls++; + return makeChatResult('{"merge_indices":[],"drop_indices":[],"edits":[]}', { input_tokens: 1, output_tokens: 1 }); + }, + }); + expect(r.skipped).toBe('headroom'); + expect(r.messages).toBe(SAMPLE); // unchanged input + }); + expect(calls).toBe(0); + }); + }); + + test('provider unavailable: returns input unchanged + skipped=provider', async () => { + await withEnv( + { ANTHROPIC_API_KEY: undefined as unknown as string }, + async () => { + const r = await runLlmPolish({ + modelStr: 'claude-haiku-4-5', + body: 'b', + messages: SAMPLE, + patternId: 'imessage-slack', + chatTransport: async () => makeChatResult('{}', { input_tokens: 1, output_tokens: 1 }), + }); + expect(r.skipped).toBe('provider'); + expect(r.messages).toEqual(SAMPLE); + }, + ); + }); + + test('cache hit on same (body, patternId) pair', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => { + let calls = 0; + const opts = { + modelStr: 'claude-haiku-4-5', + body: 'stable', + messages: SAMPLE, + patternId: 'imessage-slack', + chatTransport: async () => { + calls++; + return makeChatResult('{"merge_indices":[],"drop_indices":[],"edits":[]}', { input_tokens: 1, output_tokens: 1 }); + }, + }; + await runLlmPolish(opts); + await runLlmPolish(opts); + expect(calls).toBe(1); + }); + }); +}); diff --git a/test/conversation-parser/nightly-probe.test.ts b/test/conversation-parser/nightly-probe.test.ts new file mode 100644 index 000000000..d6f38af72 --- /dev/null +++ b/test/conversation-parser/nightly-probe.test.ts @@ -0,0 +1,107 @@ +/** + * v0.41.16.0 — Nightly probe unit tests. + * + * Hermetic via NightlyProbeDeps injection. No real LLM calls. + * + * Pins: + * - Disabled + mode=conservative → rate_limited (informational) + * - Mode=tokenmax overrides disable + * - 24h rate limit → rate_limited + * - No LLM key → no_embedding_key + * - Fixture path missing → fail + * - Adversarial false positive → adversarial_false_positive + * - All pass → pass + */ + +import { describe, expect, test } from 'bun:test'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + runConversationParserNightlyProbe, + type NightlyProbeDeps, +} from '../../src/core/conversation-parser/nightly-probe.ts'; + +function tmpFixture(content: string): string { + const dir = mkdtempSync(join(tmpdir(), 'probe-')); + const path = join(dir, 'fix.jsonl'); + writeFileSync(path, content); + return path; +} + +const POSITIVE = `{"fixture_id":"f1","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): hi","expected_messages":1,"expected_participants":["Alice"]}`; +const ADVERSARIAL = `{"fixture_id":"a1","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"just prose","expected_messages":0,"expected_participants":[]}`; + +function baseDeps(overrides: Partial = {}): NightlyProbeDeps { + return { + isEnabled: () => true, + searchMode: () => 'balanced', + hasLlmKey: () => true, + resolveFixturePath: () => tmpFixture(POSITIVE), + resolveAdversarialPath: () => tmpFixture(ADVERSARIAL), + now: () => new Date('2026-05-26T00:00:00Z'), + shouldSkipForRateLimit: () => false, + ...overrides, + }; +} + +describe('runConversationParserNightlyProbe', () => { + test('disabled + balanced mode → rate_limited (informational)', async () => { + const r = await runConversationParserNightlyProbe( + baseDeps({ isEnabled: () => false, searchMode: () => 'balanced' }), + ); + expect(r.outcome).toBe('rate_limited'); + expect(r.reason).toContain('enabled=false'); + }); + + test('disabled but mode=tokenmax → proceeds (default-on per D10)', async () => { + const r = await runConversationParserNightlyProbe( + baseDeps({ isEnabled: () => false, searchMode: () => 'tokenmax' }), + ); + expect(r.outcome).toBe('pass'); + }); + + test('24h rate limit → rate_limited', async () => { + const r = await runConversationParserNightlyProbe( + baseDeps({ shouldSkipForRateLimit: () => true }), + ); + expect(r.outcome).toBe('rate_limited'); + expect(r.reason).toContain('24h'); + }); + + test('no LLM key → no_embedding_key', async () => { + const r = await runConversationParserNightlyProbe( + baseDeps({ hasLlmKey: () => false }), + ); + expect(r.outcome).toBe('no_embedding_key'); + }); + + test('happy path: all pass', async () => { + const r = await runConversationParserNightlyProbe(baseDeps()); + expect(r.outcome).toBe('pass'); + expect(r.fixtures_total).toBe(2); + expect(r.fixtures_passed).toBe(2); + expect(r.adversarial_false_positives).toBe(0); + }); + + test('adversarial false positive → adversarial_false_positive', async () => { + // Adversarial fixture that LOOKS like iMessage; the parser will + // match it and the eval will flag as fp. + const adversarialThatMatches = `{"fixture_id":"a-bad","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): I was supposed to be unparseable","expected_messages":0,"expected_participants":[]}`; + const r = await runConversationParserNightlyProbe( + baseDeps({ + resolveAdversarialPath: () => tmpFixture(adversarialThatMatches), + }), + ); + expect(r.outcome).toBe('adversarial_false_positive'); + expect(r.adversarial_false_positives).toBe(1); + }); + + test('fixture path missing → fail', async () => { + const r = await runConversationParserNightlyProbe( + baseDeps({ resolveFixturePath: () => '/nonexistent/path.jsonl' }), + ); + expect(r.outcome).toBe('fail'); + expect(r.reason).toContain('missing'); + }); +}); diff --git a/test/conversation-parser/parse.test.ts b/test/conversation-parser/parse.test.ts new file mode 100644 index 000000000..577312a2e --- /dev/null +++ b/test/conversation-parser/parse.test.ts @@ -0,0 +1,348 @@ +/** + * v0.41.16.0 — Conversation parser orchestrator tests. + * + * Covers: + * - PR #1461's 6 telegram-bracket cases verbatim (REGRESSION pin) + * - All 12 built-in patterns hit their test_positive samples + * - Date derivation precedence (D8) + * - Pattern priority scoring (D18) — overlap resolution + * - Quick-reject fast path (D11) + * - Multi-line continuation (D5) + * - Disabled-builtin honored + * - Timezone warning (D19) emitted when frontmatter timezone missing + * + * Pure-function tests; no PGLite, no LLM. The LLM polish/fallback + * tests live in `llm-base.test.ts`, `llm-polish.test.ts`, + * `llm-fallback.test.ts` (T4). + */ + +import { describe, expect, test } from 'bun:test'; +import { + parseConversation, + deriveDateContext, + applyPattern, + scorePattern, +} from '../../src/core/conversation-parser/parse.ts'; +import { BUILTIN_PATTERNS } from '../../src/core/conversation-parser/builtins.ts'; +import type { Page } from '../../src/core/types.ts'; + +// Helper to construct a minimal Page for date-derivation tests. +function makePage( + frontmatter: Record = {}, + effective_date?: Date, +): Page { + return { + id: 1, + slug: 'test/page', + type: 'conversation', + title: 'Test', + compiled_truth: '', + timeline: '', + frontmatter, + content_hash: undefined, + created_at: new Date(), + updated_at: new Date(), + effective_date: effective_date ?? null, + } as Page; +} + +// --------------------------------------------------------------------------- +// REGRESSION: PR #1461's 6 telegram-bracket cases verbatim +// --------------------------------------------------------------------------- + +describe('parseConversation — REGRESSION PR #1461 (telegram-bracket)', () => { + test('bracket-time with 👤 emoji speaker prefix', () => { + const body = '**[18:37] \u{1f464} G T:** hello world'; + const r = parseConversation(body, { fallbackDate: '2026-05-24' }); + expect(r.messages).toHaveLength(1); + expect(r.messages[0].speaker).toBe('G T'); + expect(r.messages[0].text).toBe('hello world'); + expect(r.messages[0].timestamp).toBe('2026-05-24T18:37:00Z'); + expect(r.matched_pattern_id).toBe('telegram-bracket'); + }); + + test('bracket-time with 🤖 robot emoji', () => { + const body = '**[06:00] \u{1f916} Zion:** On it.'; + const r = parseConversation(body, { fallbackDate: '2026-05-25' }); + expect(r.messages).toHaveLength(1); + expect(r.messages[0].speaker).toBe('Zion'); + expect(r.messages[0].text).toBe('On it.'); + expect(r.messages[0].timestamp).toBe('2026-05-25T06:00:00Z'); + }); + + test('bracket-time multi-line continuation', () => { + const body = [ + '**[09:00] \u{1f464} Alice Example:** first line', + 'second line of same message', + '**[09:05] \u{1f464} Bob Example:** separate message', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-05-20' }); + expect(r.messages).toHaveLength(2); + expect(r.messages[0].text).toBe('first line\nsecond line of same message'); + expect(r.messages[1].text).toBe('separate message'); + }); + + test('bracket-time falls back to 1970-01-01 without fallbackDate', () => { + const body = '**[14:30] \u{1f464} Alice Example:** test'; + const r = parseConversation(body); + expect(r.messages).toHaveLength(1); + expect(r.messages[0].timestamp).toBe('1970-01-01T14:30:00Z'); + }); + + test('mixed formats in one body: iMessage + bracket-time', () => { + const body = [ + '**Alice Example** (2024-03-15 9:00 AM): format 1', + '**[10:30] \u{1f464} Bob Example:** format 2', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2024-03-15' }); + // D18 scoring picks the dominant pattern. Both have one hit; ties + // resolve to declared priority (imessage-slack=0, telegram-bracket=1). + // The imessage line matches; the telegram line becomes a + // continuation. This is a known D18 tradeoff for very-mixed bodies. + // In practice every chat export is homogeneous, so this is a + // degenerate test case. + expect(r.messages.length).toBeGreaterThanOrEqual(1); + expect(r.matched_pattern_id).toBe('imessage-slack'); + }); + + test('bracket-time without emoji prefix', () => { + const body = '**[22:15] Plain Name:** no emoji'; + const r = parseConversation(body, { fallbackDate: '2026-01-01' }); + expect(r.messages).toHaveLength(1); + expect(r.messages[0].speaker).toBe('Plain Name'); + expect(r.messages[0].text).toBe('no emoji'); + }); +}); + +// --------------------------------------------------------------------------- +// All 12 built-ins must parse their test_positive samples +// --------------------------------------------------------------------------- + +describe('parseConversation — every built-in matches its test_positive sample', () => { + for (const entry of BUILTIN_PATTERNS) { + test(`pattern ${entry.id}: first test_positive parses`, () => { + const body = entry.test_positive[0]; + // Multi-line patterns need a body line on the next line. + const fullBody = + entry.multi_line && entry.captures.text_group === 0 + ? `${body}\nsome body text` + : body; + const r = parseConversation(fullBody, { + fallbackDate: '2024-03-15', + }); + // Either matches a message OR (for some multi-line patterns) the + // first line is the anchor and the body is consumed as text. + expect(r.messages.length).toBeGreaterThanOrEqual(1); + expect(r.matched_pattern_id).toBe(entry.id); + }); + } +}); + +// --------------------------------------------------------------------------- +// Date derivation precedence (D8) +// --------------------------------------------------------------------------- + +describe('deriveDateContext (D8 precedence chain)', () => { + test('explicit fallbackDate wins', () => { + const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15')); + const ctx = deriveDateContext({ fallbackDate: '2025-12-25', page }); + expect(ctx.fallbackDate).toBe('2025-12-25'); + expect(ctx.source).toBe('explicit'); + }); + test('frontmatter.date wins over effective_date', () => { + const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15')); + const ctx = deriveDateContext({ page }); + expect(ctx.fallbackDate).toBe('2024-01-01'); + expect(ctx.source).toBe('frontmatter_date'); + }); + test('effective_date wins when no frontmatter.date', () => { + const page = makePage({}, new Date('2023-06-15T00:00:00Z')); + const ctx = deriveDateContext({ page }); + expect(ctx.fallbackDate).toBe('2023-06-15'); + expect(ctx.source).toBe('effective_date'); + }); + test('epoch_default when nothing set', () => { + const ctx = deriveDateContext({}); + expect(ctx.fallbackDate).toBe('1970-01-01'); + expect(ctx.source).toBe('epoch_default'); + }); + test('frontmatter.timezone surfaces', () => { + const page = makePage({ + date: '2024-01-01', + timezone: 'America/Los_Angeles', + }); + const ctx = deriveDateContext({ page }); + expect(ctx.timezone).toBe('America/Los_Angeles'); + }); + test('invalid frontmatter.date falls through', () => { + const page = makePage({ date: 'not-a-date' }); + const ctx = deriveDateContext({ page }); + expect(ctx.source).toBe('epoch_default'); + }); + test('frontmatter.date slices full ISO to YYYY-MM-DD', () => { + const page = makePage({ date: '2024-03-15T18:37:00Z' }); + const ctx = deriveDateContext({ page }); + expect(ctx.fallbackDate).toBe('2024-03-15'); + }); +}); + +// --------------------------------------------------------------------------- +// Pattern priority scoring (D18) +// --------------------------------------------------------------------------- + +describe('scorePattern (D18 priority scoring)', () => { + test('telegram-bracket scores 1.0 on a pure telegram body', () => { + const body = [ + '**[18:37] \u{1f464} Alice:** one', + '**[18:38] \u{1f464} Bob:** two', + '**[18:39] \u{1f464} Alice:** three', + ].join('\n'); + const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; + expect(scorePattern(body, tg)).toBe(1); + }); + test('imessage-slack scores 0 on pure telegram body', () => { + const body = [ + '**[18:37] \u{1f464} Alice:** one', + '**[18:38] \u{1f464} Bob:** two', + ].join('\n'); + const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!; + expect(scorePattern(body, im)).toBe(0); + }); + test('mixed body: D18 picks the higher-scoring pattern', () => { + // 3 telegram lines + 1 imessage line. Telegram should win. + const body = [ + '**[18:37] \u{1f464} Alice:** one', + '**[18:38] \u{1f464} Bob:** two', + '**[18:39] \u{1f464} Alice:** three', + '**Charlie** (2024-03-15 9:00 AM): one imessage', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2024-03-15' }); + expect(r.matched_pattern_id).toBe('telegram-bracket'); + }); +}); + +// --------------------------------------------------------------------------- +// Disabled-builtin honored +// --------------------------------------------------------------------------- + +describe('parseConversation — disabledBuiltinIds', () => { + test('disabling top pattern falls through to next', () => { + const body = '**[18:37] \u{1f464} Alice:** hello'; + const rDefault = parseConversation(body, { fallbackDate: '2024-03-15' }); + expect(rDefault.matched_pattern_id).toBe('telegram-bracket'); + const rDisabled = parseConversation(body, { + fallbackDate: '2024-03-15', + disabledBuiltinIds: ['telegram-bracket'], + }); + // No other built-in matches this exact shape → no_match. + expect(rDisabled.phase).toBe('no_match'); + }); +}); + +// --------------------------------------------------------------------------- +// Multi-line continuation (D5) +// --------------------------------------------------------------------------- + +describe('parseConversation — multi-line continuation (D5)', () => { + test('iMessage continuation absorbs orphan lines', () => { + const body = [ + '**Alice Example** (2024-03-15 9:00 AM): first line', + 'continuation line', + 'another continuation', + '**Bob Example** (2024-03-15 9:05 AM): second message', + ].join('\n'); + const r = parseConversation(body); + expect(r.messages).toHaveLength(2); + expect(r.messages[0].text).toBe( + 'first line\ncontinuation line\nanother continuation', + ); + expect(r.messages[1].text).toBe('second message'); + }); +}); + +// --------------------------------------------------------------------------- +// Timezone warning (D19) +// --------------------------------------------------------------------------- + +describe('parseConversation — timezone warning (D19)', () => { + test('telegram-bracket emits warning when no timezone in frontmatter', () => { + const body = '**[18:37] \u{1f464} Alice:** hello'; + const r = parseConversation(body, { fallbackDate: '2024-03-15' }); + expect(r.timezone_warning).toBeDefined(); + expect(r.timezone_warning).toContain('telegram-bracket'); + expect(r.timezone_warning).toContain('UTC'); + }); + test('telegram-bracket does NOT warn when timezone is present', () => { + const body = '**[18:37] \u{1f464} Alice:** hello'; + const page = makePage({ + date: '2024-03-15', + timezone: 'America/Los_Angeles', + }); + const r = parseConversation(body, { page }); + expect(r.timezone_warning).toBeUndefined(); + }); + test('imessage-slack does NOT warn (inline_utc policy)', () => { + const body = '**Alice Example** (2024-03-15 9:00 AM): hello'; + const r = parseConversation(body); + expect(r.timezone_warning).toBeUndefined(); + }); +}); + +// --------------------------------------------------------------------------- +// Empty body + degenerate cases +// --------------------------------------------------------------------------- + +describe('parseConversation — degenerate inputs', () => { + test('empty body returns no_match + empty messages', () => { + expect(parseConversation('')).toEqual({ + messages: [], + phase: 'no_match', + }); + }); + test('non-conversational text returns no_match', () => { + const r = parseConversation('This is just prose with no chat shape.'); + expect(r.phase).toBe('no_match'); + expect(r.messages).toHaveLength(0); + }); +}); + +// --------------------------------------------------------------------------- +// applyPattern — direct unit tests for the matcher +// --------------------------------------------------------------------------- + +describe('applyPattern — quick_reject fast path (D11)', () => { + test('telegram quick_reject skips iMessage lines fast', () => { + const body = '**Alice Example** (2024-03-15 9:00 AM): hello'; + const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; + const r = applyPattern(body, tg, { + fallbackDate: '2024-03-15', + source: 'explicit', + }); + // Quick_reject /^\*\*\[/ rejects '**Alice' (no `[`). Zero matches. + expect(r).toHaveLength(0); + }); +}); + +// --------------------------------------------------------------------------- +// scorePattern boundary cases +// --------------------------------------------------------------------------- + +describe('scorePattern — boundary', () => { + test('empty body scores 0', () => { + const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; + expect(scorePattern('', tg)).toBe(0); + }); + test('only blank lines scores 0', () => { + const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; + expect(scorePattern('\n\n \n', tg)).toBe(0); + }); + test('caps at SCORING_HEAD_LINES (10) lines', () => { + // 100 telegram lines → still scores 1.0 because only first 10 sampled. + const body = Array.from( + { length: 100 }, + (_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`, + ).join('\n'); + const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; + expect(scorePattern(body, tg)).toBe(1); + }); +}); diff --git a/test/doctor-v0_41_13_checks.test.ts b/test/doctor-v0_41_13_checks.test.ts new file mode 100644 index 000000000..68ddefd1f --- /dev/null +++ b/test/doctor-v0_41_13_checks.test.ts @@ -0,0 +1,98 @@ +/** + * v0.41.16.0 — Targeted assertions for the 3 new doctor checks. + * + * Spawns `bun src/cli.ts doctor --json --fast` as a subprocess and + * parses the JSON envelope to verify: + * + * - conversation_format_coverage + * - progressive_batch_audit_health + * - conversation_parser_probe_health + * + * are present with stable shapes. The full doctor surface is covered + * by test/doctor.test.ts; this file is a structural regression guard + * for the 3 new v0.41.16.0 checks. + * + * Spawning the subprocess matches the actual user experience (`gbrain + * doctor`) and avoids the in-process env/stdout-capture brittleness + * that bit the original test draft. + */ + +import { describe, expect, test } from 'bun:test'; +import { spawnSync } from 'node:child_process'; + +interface DoctorCheck { + name: string; + status: 'ok' | 'warn' | 'fail'; + message: string; +} +interface DoctorEnvelope { + schema_version: number; + status: 'healthy' | 'unhealthy'; + health_score: number; + checks: DoctorCheck[]; +} + +function runDoctor(): DoctorEnvelope { + const result = spawnSync( + process.execPath, // bun + ['src/cli.ts', 'doctor', '--json', '--fast'], + { + cwd: process.cwd(), + encoding: 'utf8', + timeout: 60000, + }, + ); + if (result.error) throw result.error; + // Doctor's JSON envelope is the LAST line in stdout (CLI may print + // banners on stderr; --json sends the envelope to stdout). + const stdout = result.stdout ?? ''; + const lines = stdout.split('\n').filter((l) => l.trim().length > 0); + const jsonLine = lines.reverse().find((l) => l.trim().startsWith('{')); + if (!jsonLine) { + throw new Error( + `No JSON envelope found in doctor output. stdout=${stdout.slice(0, 500)} stderr=${(result.stderr ?? '').slice(0, 500)}`, + ); + } + return JSON.parse(jsonLine) as DoctorEnvelope; +} + +describe('doctor — v0.41.16.0 new checks emit', () => { + test('all 3 new checks present in JSON envelope', () => { + const env = runDoctor(); + const checkNames = env.checks.map((c) => c.name); + // conversation_format_coverage may not appear in --fast mode (it + // requires DB access); progressive_batch_audit_health and + // conversation_parser_probe_health do not need DB. + expect(checkNames).toContain('progressive_batch_audit_health'); + expect(checkNames).toContain('conversation_parser_probe_health'); + }); + + test('progressive_batch_audit_health shape', () => { + const env = runDoctor(); + const check = env.checks.find( + (c) => c.name === 'progressive_batch_audit_health', + ); + expect(check).toBeDefined(); + expect(['ok', 'warn', 'fail']).toContain(check!.status); + expect(typeof check!.message).toBe('string'); + expect(check!.message.length).toBeGreaterThan(0); + }); + + test('conversation_parser_probe_health shape + opt-in hint', () => { + const env = runDoctor(); + const check = env.checks.find( + (c) => c.name === 'conversation_parser_probe_health', + ); + expect(check).toBeDefined(); + expect(check!.status).toBe('ok'); + expect(check!.message).toContain('opt-in'); + expect(check!.message).toContain( + 'autopilot.conversation_parser_probe.enabled true', + ); + }); + + test('schema_version is stable (2 at v0.41.16.0)', () => { + const env = runDoctor(); + expect(env.schema_version).toBe(2); + }); +}); diff --git a/test/e2e/conversation-parser-pglite.test.ts b/test/e2e/conversation-parser-pglite.test.ts new file mode 100644 index 000000000..f7a02d0a3 --- /dev/null +++ b/test/e2e/conversation-parser-pglite.test.ts @@ -0,0 +1,163 @@ +/** + * v0.41.16.0 — E2E test for the conversation parser cathedral against + * a real PGLite brain. + * + * For each of the 12 built-in formats: seed a page through + * `importFromContent`, run `parseConversation` against the body, assert + * the parser identifies the correct pattern AND produces at least one + * message AND the message timestamp lands in the expected date range. + * + * Per CLAUDE.md test-isolation R3+R4: uses the canonical PGLite block. + * Hermetic (no DATABASE_URL needed); the in-memory PGLite is created + * once per file and reset between tests. + * + * This is the integration test that proves the parser ↔ engine + * interaction is correct for the dream cycle's + * `conversation_facts_backfill` phase. + */ + +import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { + parseConversation, +} from '../../src/core/conversation-parser/parse.ts'; +import { BUILTIN_PATTERNS } from '../../src/core/conversation-parser/builtins.ts'; +import { importFromContent } from '../../src/core/import-file.ts'; +import type { Page } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +/** + * Build a conversation page body using the first test_positive sample + * from the pattern's registry entry. Multi-line patterns get a body + * line appended. + */ +function buildSampleBody(patternId: string): string { + const pattern = BUILTIN_PATTERNS.find((p) => p.id === patternId); + if (!pattern) throw new Error(`pattern ${patternId} not found`); + // Repeat the positive sample 3 times with slight variations to + // simulate a multi-message conversation. + const samples = pattern.test_positive.slice(0, 2); + if (pattern.multi_line && pattern.captures.text_group === 0) { + // For patterns where text comes from the next line, append a body + // line after each header. + return samples.flatMap((s, i) => [s, `body line ${i + 1}`]).join('\n'); + } + return samples.join('\n'); +} + +describe('E2E: parser ↔ engine integration for every built-in', () => { + for (const pattern of BUILTIN_PATTERNS) { + test(`pattern=${pattern.id}: page imports + parser identifies pattern`, async () => { + const slug = `conversations/test/${pattern.id}-sample`; + const body = buildSampleBody(pattern.id); + const frontmatter: Record = { + type: 'conversation', + date: '2024-03-15', + }; + if (pattern.timezone_policy === 'utc_assumed_with_warn') { + // Some patterns warn without a timezone; provide one to avoid + // stderr noise in the test output. + frontmatter.timezone = 'America/Los_Angeles'; + } + + // Seed the page through the canonical import path. + const fullBody = `---\n${Object.entries(frontmatter) + .map(([k, v]) => `${k}: ${typeof v === 'string' ? v : JSON.stringify(v)}`) + .join('\n')}\n---\n\n${body}`; + await importFromContent(engine, slug, fullBody, { + sourceId: 'default', + noEmbed: true, + }); + + // Read the page back and parse it through the orchestrator + // (mirrors what extract-conversation-facts.ts does in production). + const page = (await engine.getPage(slug)) as Page; + expect(page).not.toBeNull(); + + const bodyToParse = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(); + const result = parseConversation(bodyToParse, { page }); + + expect(result.phase).toBe('regex_match'); + expect(result.matched_pattern_id).toBe(pattern.id); + expect(result.messages.length).toBeGreaterThanOrEqual(1); + + // Every parsed message has a valid ISO timestamp. + for (const msg of result.messages) { + const ts = Date.parse(msg.timestamp); + expect(Number.isFinite(ts)).toBe(true); + // Pattern's timezone policy determines date range constraints. + // For inline-date patterns the date is in the line; for + // time-only patterns it's from frontmatter ('2024-03-15'). + if (pattern.date_source === 'frontmatter') { + expect(msg.timestamp.slice(0, 10)).toBe('2024-03-15'); + } + expect(msg.speaker.length).toBeGreaterThan(0); + } + }); + } +}); + +describe('E2E: parser handles unparseable bodies cleanly', () => { + test('non-chat content returns no_match without crashing', async () => { + const slug = 'conversations/test/not-a-chat'; + const fullBody = `---\ntype: conversation\ndate: 2024-03-15\n---\n\nThis is just prose. It has no chat structure at all. Just words that flow.`; + await importFromContent(engine, slug, fullBody, { + sourceId: 'default', + noEmbed: true, + }); + const page = (await engine.getPage(slug)) as Page; + const bodyToParse = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(); + const result = parseConversation(bodyToParse, { page }); + expect(result.phase).toBe('no_match'); + expect(result.messages).toEqual([]); + }); + + test('empty body returns no_match', async () => { + const slug = 'conversations/test/empty'; + const fullBody = `---\ntype: conversation\ndate: 2024-03-15\n---\n\n`; + await importFromContent(engine, slug, fullBody, { + sourceId: 'default', + noEmbed: true, + }); + const page = (await engine.getPage(slug)) as Page; + const result = parseConversation(`${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(), { page }); + expect(result.phase).toBe('no_match'); + expect(result.messages).toEqual([]); + }); +}); + +describe('E2E: D8 date derivation chain through Page', () => { + test('frontmatter.date wins over epoch default', async () => { + const slug = 'conversations/test/date-derive'; + const body = '**[18:37] 👤 Alice:** hi'; + const fullBody = `---\ntype: conversation\ndate: 2026-05-24\ntimezone: America/Los_Angeles\n---\n\n${body}`; + await importFromContent(engine, slug, fullBody, { + sourceId: 'default', + noEmbed: true, + }); + const page = (await engine.getPage(slug)) as Page; + const result = parseConversation( + `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim(), + { page }, + ); + expect(result.matched_pattern_id).toBe('telegram-bracket'); + expect(result.messages).toHaveLength(1); + expect(result.messages[0].timestamp).toBe('2026-05-24T18:37:00Z'); + }); +}); diff --git a/test/eval-conversation-parser-cli.test.ts b/test/eval-conversation-parser-cli.test.ts new file mode 100644 index 000000000..84b96a394 --- /dev/null +++ b/test/eval-conversation-parser-cli.test.ts @@ -0,0 +1,181 @@ +/** + * v0.41.16.0 — `gbrain eval conversation-parser` CLI behavior tests. + * + * Covers argv parsing, exit codes (0/1/2), --json envelope shape, + * --no-llm flag, --min-recall override, USAGE errors. Pure file I/O, + * no DB, no API keys. + * + * Critical because `bun run check:conversation-parser` is wired into + * `bun run verify` — a silent regression in exit-code handling would + * make every PR's CI green even when the parser broke. + */ + +import { describe, expect, test } from 'bun:test'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { runEvalConversationParser } from '../src/commands/eval-conversation-parser.ts'; + +function tmpFixture(content: string): string { + const dir = mkdtempSync(join(tmpdir(), 'eval-cli-')); + const path = join(dir, 'fix.jsonl'); + writeFileSync(path, content); + return path; +} + +const PASS = `{"fixture_id":"p1","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): hi","expected_messages":1,"expected_participants":["Alice"]}`; +const FAIL_PATTERN_MISMATCH = `{"fixture_id":"f1","pattern":"telegram-bracket","frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): hi","expected_messages":1,"expected_participants":["Alice"]}`; + +describe('runEvalConversationParser — exit codes', () => { + test('exit 0 on all-pass fixture', async () => { + const path = tmpFixture(PASS); + const code = await runEvalConversationParser([path, '--no-llm']); + expect(code).toBe(0); + }); + + test('exit 1 on any failure', async () => { + const path = tmpFixture(FAIL_PATTERN_MISMATCH); + const code = await runEvalConversationParser([path, '--no-llm']); + expect(code).toBe(1); + }); + + test('exit 2 on missing fixture argument', async () => { + const code = await runEvalConversationParser(['--no-llm']); + expect(code).toBe(2); + }); + + test('exit 2 on nonexistent fixture path', async () => { + const code = await runEvalConversationParser([ + '/nonexistent/path.jsonl', + '--no-llm', + ]); + expect(code).toBe(2); + }); + + test('exit 2 on malformed JSONL', async () => { + const path = tmpFixture('NOT-JSON\n'); + const code = await runEvalConversationParser([path]); + expect(code).toBe(2); + }); + + test('exit 2 on empty fixture file', async () => { + const path = tmpFixture(''); + const code = await runEvalConversationParser([path]); + expect(code).toBe(2); + }); + + test('exit 0 on help', async () => { + const code = await runEvalConversationParser(['--help']); + expect(code).toBe(0); + }); +}); + +describe('runEvalConversationParser — flag parsing', () => { + test('positional fixture path works without --fixtures', async () => { + const path = tmpFixture(PASS); + const code = await runEvalConversationParser([path, '--no-llm']); + expect(code).toBe(0); + }); + + test('--fixtures form works', async () => { + const path = tmpFixture(PASS); + const code = await runEvalConversationParser([ + '--fixtures', + path, + '--no-llm', + ]); + expect(code).toBe(0); + }); + + test('--fixtures= form works', async () => { + const path = tmpFixture(PASS); + const code = await runEvalConversationParser([ + `--fixtures=${path}`, + '--no-llm', + ]); + expect(code).toBe(0); + }); + + test('--min-recall override (lower floor lets near-misses pass)', async () => { + // A fixture where the parser only catches 0 of 1 expected messages. + // Default --min-recall 0.9 would fail this; --min-recall 0.0 passes. + const partial = `{"fixture_id":"pp","pattern":"telegram-bracket","frontmatter":{"date":"2024-03-15"},"body":"this is not a telegram line","expected_messages":1,"expected_participants":["Alice"]}`; + const path = tmpFixture(partial); + const defaultCode = await runEvalConversationParser([path, '--no-llm']); + expect(defaultCode).toBe(1); // expected pattern not matched → fail + // Lowering min-recall doesn't rescue a pattern mismatch — pattern + // identity is checked separately. This pins that contract. + const lowCode = await runEvalConversationParser([ + path, + '--no-llm', + '--min-recall', + '0.0', + ]); + expect(lowCode).toBe(1); + }); +}); + +describe('runEvalConversationParser — --json envelope', () => { + test('emits stable schema_version=1 + recall + pattern_coverage', async () => { + const path = tmpFixture(PASS); + const origWrite = process.stdout.write.bind(process.stdout); + const captured: string[] = []; + process.stdout.write = ((chunk: string | Uint8Array) => { + captured.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString()); + return true; + }) as typeof process.stdout.write; + try { + const code = await runEvalConversationParser([path, '--no-llm', '--json']); + expect(code).toBe(0); + } finally { + process.stdout.write = origWrite; + } + const stdout = captured.join(''); + const json = JSON.parse(stdout.trim()); + expect(json.schema_version).toBe(1); + expect(json.total_fixtures).toBe(1); + expect(json.passed).toBe(1); + expect(json.failed).toBe(0); + expect(json.recall_mean).toBe(1); + expect(json.participants_recall_mean).toBe(1); + expect(json.pattern_coverage).toEqual({ 'imessage-slack': 1 }); + expect(json.fixtures).toHaveLength(1); + expect(json.failed_fixture_ids).toEqual([]); + }); + + test('--json on failure includes failed_fixture_ids', async () => { + const path = tmpFixture(FAIL_PATTERN_MISMATCH); + const origWrite = process.stdout.write.bind(process.stdout); + const captured: string[] = []; + process.stdout.write = ((chunk: string | Uint8Array) => { + captured.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString()); + return true; + }) as typeof process.stdout.write; + try { + const code = await runEvalConversationParser([path, '--no-llm', '--json']); + expect(code).toBe(1); + } finally { + process.stdout.write = origWrite; + } + const json = JSON.parse(captured.join('').trim()); + expect(json.failed_fixture_ids).toContain('f1'); + }); +}); + +describe('runEvalConversationParser — adversarial fixture', () => { + test('adversarial (pattern=null) with non-empty parse → fail', async () => { + // Adversarial fixture whose body LOOKS like iMessage; parser will + // match it; eval flags as adversarial false-positive. + const advFp = `{"fixture_id":"adv-fp","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"**Alice** (2024-03-15 9:00 AM): I should not parse","expected_messages":0,"expected_participants":[]}`; + const path = tmpFixture(advFp); + const code = await runEvalConversationParser([path, '--no-llm']); + expect(code).toBe(1); + }); + + test('adversarial with truly unparseable body → pass', async () => { + const advClean = `{"fixture_id":"adv-clean","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"This is just prose with no chat shape.","expected_messages":0,"expected_participants":[]}`; + const path = tmpFixture(advClean); + const code = await runEvalConversationParser([path, '--no-llm']); + expect(code).toBe(0); + }); +}); diff --git a/test/fixtures/conversation-formats/adversarial.jsonl b/test/fixtures/conversation-formats/adversarial.jsonl new file mode 100644 index 000000000..fafac150a --- /dev/null +++ b/test/fixtures/conversation-formats/adversarial.jsonl @@ -0,0 +1,5 @@ +{"fixture_id":"adversarial-recipe","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"Chocolate Chip Cookies\n\nIngredients:\n- 2 cups flour\n- 1 cup sugar\n- 1 cup chocolate chips\n\nInstructions:\nMix dry ingredients. Add wet. Bake at 375 for 12 minutes.","expected_messages":0,"expected_participants":[]} +{"fixture_id":"adversarial-readme","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"# My Project\n\nA short description.\n\n## Install\n\n```\nnpm install my-project\n```\n\n## Usage\n\nImport the package and call the function.","expected_messages":0,"expected_participants":[]} +{"fixture_id":"adversarial-code","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"function add(a, b) {\n return a + b;\n}\n\nconst result = add(1, 2);\nconsole.log(result);","expected_messages":0,"expected_participants":[]} +{"fixture_id":"adversarial-lyrics","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"Twinkle twinkle little star\nHow I wonder what you are\nUp above the world so high\nLike a diamond in the sky","expected_messages":0,"expected_participants":[]} +{"fixture_id":"adversarial-json","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"{\n \"key\": \"value\",\n \"num\": 42,\n \"list\": [1, 2, 3]\n}","expected_messages":0,"expected_participants":[]} diff --git a/test/fixtures/conversation-formats/all.jsonl b/test/fixtures/conversation-formats/all.jsonl new file mode 100644 index 000000000..91461155e --- /dev/null +++ b/test/fixtures/conversation-formats/all.jsonl @@ -0,0 +1,13 @@ +{"fixture_id":"imessage-001","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice Example** (2024-03-15 9:00 AM): morning\n**Bob Example** (2024-03-15 9:01 AM): hey there\n**Alice Example** (2024-03-15 9:02 AM): how are you\n**Bob Example** (2024-03-15 9:03 AM): good thanks\n**Alice Example** (2024-03-15 9:04 AM): you?","expected_messages":5,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"imessage-002","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Charlie Example** (2024-03-15 2:00 PM): afternoon\n**Charlie Example** (2024-03-15 2:01 PM): are you there?\n**Diana Example** (2024-03-15 2:05 PM): yes\n**Charlie Example** (2024-03-15 2:06 PM): great","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]} +{"fixture_id":"telegram-bracket-001","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-24","timezone":"America/Los_Angeles"},"body":"**[18:37] 👤 Alice Example:** hello world\n**[18:38] 👤 Bob Example:** hey\n**[18:39] 👤 Alice Example:** how are you\n**[18:40] 👤 Bob Example:** good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"telegram-bracket-002","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-25","timezone":"America/Los_Angeles"},"body":"**[06:00] 🤖 Zion Bot:** On it.\n**[06:01] 👤 Charlie Example:** thanks\n**[06:02] 🤖 Zion Bot:** anything else?\n**[06:03] 👤 Charlie Example:** no good","expected_messages":4,"expected_participants":["Zion Bot","Charlie Example"]} +{"fixture_id":"whatsapp-iso-001","pattern":"whatsapp-iso","frontmatter":{"date":"2024-03-15"},"body":"[15/03/24, 18:37:00] Alice Example: hello\n[15/03/24, 18:37:30] Bob Example: hey\n[15/03/24, 18:38:00] Alice Example: how are you\n[15/03/24, 18:39:00] Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"whatsapp-iso-002","pattern":"whatsapp-iso","frontmatter":{"date":"2024-01-01"},"body":"[01/01/24, 00:00:00] Charlie Example: happy new year\n[01/01/24, 00:00:30] Diana Example: same to you\n[01/01/24, 00:01:00] Charlie Example: any plans?\n[01/01/24, 00:02:00] Diana Example: sleep","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]} +{"fixture_id":"whatsapp-us-001","pattern":"whatsapp-us","frontmatter":{"date":"2024-03-15"},"body":"3/15/24, 6:37 PM - Alice Example: hello\n3/15/24, 6:38 PM - Bob Example: hey\n3/15/24, 6:39 PM - Alice Example: how are you\n3/15/24, 6:40 PM - Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"whatsapp-us-002","pattern":"whatsapp-us","frontmatter":{"date":"2023-12-31"},"body":"12/31/23, 11:59 PM - Charlie Example: nye\n12/31/23, 11:59 PM - Diana Example: cheers","expected_messages":2,"expected_participants":["Charlie Example","Diana Example"]} +{"fixture_id":"signal-export-001","pattern":"signal-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example (2024-03-15 18:37:00 UTC): hello\nBob Example (2024-03-15 18:37:30 UTC): hey there\nAlice Example (2024-03-15 18:38:00 UTC): how are you\nBob Example (2024-03-15 18:39:00 UTC): good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"irc-classic-001","pattern":"irc-classic","frontmatter":{"date":"2024-03-15"},"body":" hello world\n hey there\n how are you\n good thanks","expected_messages":4,"expected_participants":["alice","bob"]} +{"fixture_id":"irc-weechat-001","pattern":"irc-weechat","frontmatter":{"date":"2024-03-15"},"body":"18:37 hello\n18:38 hey\n18:39 how are you\n18:40 good","expected_messages":4,"expected_participants":["alice","bob"]} +{"fixture_id":"matrix-element-001","pattern":"matrix-element","frontmatter":{"date":"2024-03-15"},"body":"[18:37] @alice:matrix.org: hello world\n[18:38] @bob:example.org: hey\n[18:39] @alice:matrix.org: how are you\n[18:40] @bob:example.org: good","expected_messages":4,"expected_participants":["alice","bob"]} +{"fixture_id":"teams-export-001","pattern":"teams-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example, 3/15/2024 6:37 PM: hello team\nBob Example, 3/15/2024 6:38 PM: hey\nAlice Example, 3/15/2024 6:39 PM: meeting at 4?\nBob Example, 3/15/2024 6:40 PM: works for me","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} diff --git a/test/fixtures/conversation-formats/imessage.jsonl b/test/fixtures/conversation-formats/imessage.jsonl new file mode 100644 index 000000000..28263e3b9 --- /dev/null +++ b/test/fixtures/conversation-formats/imessage.jsonl @@ -0,0 +1,2 @@ +{"fixture_id":"imessage-001","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice Example** (2024-03-15 9:00 AM): morning\n**Bob Example** (2024-03-15 9:01 AM): hey there\n**Alice Example** (2024-03-15 9:02 AM): how are you\n**Bob Example** (2024-03-15 9:03 AM): good thanks\n**Alice Example** (2024-03-15 9:04 AM): you?","expected_messages":5,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"imessage-002","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Charlie Example** (2024-03-15 2:00 PM): afternoon\n**Charlie Example** (2024-03-15 2:01 PM): are you there?\n**Diana Example** (2024-03-15 2:05 PM): yes\n**Charlie Example** (2024-03-15 2:06 PM): great","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]} diff --git a/test/fixtures/conversation-formats/irc-classic.jsonl b/test/fixtures/conversation-formats/irc-classic.jsonl new file mode 100644 index 000000000..454a0c09b --- /dev/null +++ b/test/fixtures/conversation-formats/irc-classic.jsonl @@ -0,0 +1 @@ +{"fixture_id":"irc-classic-001","pattern":"irc-classic","frontmatter":{"date":"2024-03-15"},"body":" hello world\n hey there\n how are you\n good thanks","expected_messages":4,"expected_participants":["alice","bob"]} diff --git a/test/fixtures/conversation-formats/irc-weechat.jsonl b/test/fixtures/conversation-formats/irc-weechat.jsonl new file mode 100644 index 000000000..836e2cc91 --- /dev/null +++ b/test/fixtures/conversation-formats/irc-weechat.jsonl @@ -0,0 +1 @@ +{"fixture_id":"irc-weechat-001","pattern":"irc-weechat","frontmatter":{"date":"2024-03-15"},"body":"18:37 hello\n18:38 hey\n18:39 how are you\n18:40 good","expected_messages":4,"expected_participants":["alice","bob"]} diff --git a/test/fixtures/conversation-formats/matrix-element.jsonl b/test/fixtures/conversation-formats/matrix-element.jsonl new file mode 100644 index 000000000..24c5bbcfc --- /dev/null +++ b/test/fixtures/conversation-formats/matrix-element.jsonl @@ -0,0 +1 @@ +{"fixture_id":"matrix-element-001","pattern":"matrix-element","frontmatter":{"date":"2024-03-15"},"body":"[18:37] @alice:matrix.org: hello world\n[18:38] @bob:example.org: hey\n[18:39] @alice:matrix.org: how are you\n[18:40] @bob:example.org: good","expected_messages":4,"expected_participants":["alice","bob"]} diff --git a/test/fixtures/conversation-formats/signal-export.jsonl b/test/fixtures/conversation-formats/signal-export.jsonl new file mode 100644 index 000000000..b2b675732 --- /dev/null +++ b/test/fixtures/conversation-formats/signal-export.jsonl @@ -0,0 +1 @@ +{"fixture_id":"signal-export-001","pattern":"signal-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example (2024-03-15 18:37:00 UTC): hello\nBob Example (2024-03-15 18:37:30 UTC): hey there\nAlice Example (2024-03-15 18:38:00 UTC): how are you\nBob Example (2024-03-15 18:39:00 UTC): good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} diff --git a/test/fixtures/conversation-formats/teams-export.jsonl b/test/fixtures/conversation-formats/teams-export.jsonl new file mode 100644 index 000000000..c79a2aa1d --- /dev/null +++ b/test/fixtures/conversation-formats/teams-export.jsonl @@ -0,0 +1 @@ +{"fixture_id":"teams-export-001","pattern":"teams-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example, 3/15/2024 6:37 PM: hello team\nBob Example, 3/15/2024 6:38 PM: hey\nAlice Example, 3/15/2024 6:39 PM: meeting at 4?\nBob Example, 3/15/2024 6:40 PM: works for me","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} diff --git a/test/fixtures/conversation-formats/telegram-bracket.jsonl b/test/fixtures/conversation-formats/telegram-bracket.jsonl new file mode 100644 index 000000000..f6f920367 --- /dev/null +++ b/test/fixtures/conversation-formats/telegram-bracket.jsonl @@ -0,0 +1,2 @@ +{"fixture_id":"telegram-bracket-001","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-24","timezone":"America/Los_Angeles"},"body":"**[18:37] 👤 Alice Example:** hello world\n**[18:38] 👤 Bob Example:** hey\n**[18:39] 👤 Alice Example:** how are you\n**[18:40] 👤 Bob Example:** good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"telegram-bracket-002","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-25","timezone":"America/Los_Angeles"},"body":"**[06:00] 🤖 Zion Bot:** On it.\n**[06:01] 👤 Charlie Example:** thanks\n**[06:02] 🤖 Zion Bot:** anything else?\n**[06:03] 👤 Charlie Example:** no good","expected_messages":4,"expected_participants":["Zion Bot","Charlie Example"]} diff --git a/test/fixtures/conversation-formats/whatsapp-iso.jsonl b/test/fixtures/conversation-formats/whatsapp-iso.jsonl new file mode 100644 index 000000000..025a4618f --- /dev/null +++ b/test/fixtures/conversation-formats/whatsapp-iso.jsonl @@ -0,0 +1,2 @@ +{"fixture_id":"whatsapp-iso-001","pattern":"whatsapp-iso","frontmatter":{"date":"2024-03-15"},"body":"[15/03/24, 18:37:00] Alice Example: hello\n[15/03/24, 18:37:30] Bob Example: hey\n[15/03/24, 18:38:00] Alice Example: how are you\n[15/03/24, 18:39:00] Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"whatsapp-iso-002","pattern":"whatsapp-iso","frontmatter":{"date":"2024-01-01"},"body":"[01/01/24, 00:00:00] Charlie Example: happy new year\n[01/01/24, 00:00:30] Diana Example: same to you\n[01/01/24, 00:01:00] Charlie Example: any plans?\n[01/01/24, 00:02:00] Diana Example: sleep","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]} diff --git a/test/fixtures/conversation-formats/whatsapp-us.jsonl b/test/fixtures/conversation-formats/whatsapp-us.jsonl new file mode 100644 index 000000000..c602f7d0b --- /dev/null +++ b/test/fixtures/conversation-formats/whatsapp-us.jsonl @@ -0,0 +1,2 @@ +{"fixture_id":"whatsapp-us-001","pattern":"whatsapp-us","frontmatter":{"date":"2024-03-15"},"body":"3/15/24, 6:37 PM - Alice Example: hello\n3/15/24, 6:38 PM - Bob Example: hey\n3/15/24, 6:39 PM - Alice Example: how are you\n3/15/24, 6:40 PM - Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"whatsapp-us-002","pattern":"whatsapp-us","frontmatter":{"date":"2023-12-31"},"body":"12/31/23, 11:59 PM - Charlie Example: nye\n12/31/23, 11:59 PM - Diana Example: cheers","expected_messages":2,"expected_participants":["Charlie Example","Diana Example"]} diff --git a/test/migrations-v99.test.ts b/test/migrations-v99.test.ts new file mode 100644 index 000000000..0249d69cd --- /dev/null +++ b/test/migrations-v99.test.ts @@ -0,0 +1,159 @@ +/** + * v0.41.16.0 — Migration v99 round-trip test. + * + * Verifies the `conversation_parser_llm_cache` table: + * - is created on schema init + * - accepts inserts on (content_sha256, model_id, call_shape, value_json) + * - rejects invalid call_shape via CHECK constraint + * - ON CONFLICT DO NOTHING semantics (the llm-base.ts caller's contract) + * - JSONB column round-trips a real object (no double-encode regression) + * - composite primary key prevents duplicate (sha, model, shape) + * + * Hermetic via the canonical PGLite block from CLAUDE.md test-isolation + * rules. + */ + +import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +describe('migration v99 — conversation_parser_llm_cache', () => { + test('table exists after schema init', async () => { + const rows = await engine.executeRaw<{ table_name: string }>( + `SELECT table_name FROM information_schema.tables + WHERE table_schema = 'public' AND table_name = 'conversation_parser_llm_cache'`, + ); + expect(rows.length).toBe(1); + }); + + test('insert + select round-trip with polish call_shape', async () => { + await engine.executeRaw( + `INSERT INTO conversation_parser_llm_cache + (content_sha256, model_id, call_shape, value_json) + VALUES ($1, $2, $3, $4::jsonb)`, + [ + 'abc123', + 'anthropic:claude-haiku-4-5', + 'polish', + JSON.stringify({ merge_indices: [], drop_indices: [], edits: [] }), + ], + ); + const rows = await engine.executeRaw<{ value_json: unknown }>( + `SELECT value_json FROM conversation_parser_llm_cache + WHERE content_sha256 = $1 AND model_id = $2 AND call_shape = $3`, + ['abc123', 'anthropic:claude-haiku-4-5', 'polish'], + ); + expect(rows).toHaveLength(1); + // value_json should round-trip as a parsed object (not a JSON string). + const val = + typeof rows[0].value_json === 'string' + ? JSON.parse(rows[0].value_json) + : rows[0].value_json; + expect(val).toEqual({ merge_indices: [], drop_indices: [], edits: [] }); + }); + + test('insert + select round-trip with fallback call_shape', async () => { + await engine.executeRaw( + `INSERT INTO conversation_parser_llm_cache + (content_sha256, model_id, call_shape, value_json) + VALUES ($1, $2, $3, $4::jsonb)`, + [ + 'def456', + 'anthropic:claude-haiku-4-5', + 'fallback', + JSON.stringify([ + { speaker: 'Alice', timestamp: '2024-03-15T18:37:00Z', text: 'hi' }, + ]), + ], + ); + const rows = await engine.executeRaw<{ value_json: unknown }>( + `SELECT value_json FROM conversation_parser_llm_cache + WHERE content_sha256 = $1 AND call_shape = $2`, + ['def456', 'fallback'], + ); + expect(rows).toHaveLength(1); + const val = + typeof rows[0].value_json === 'string' + ? JSON.parse(rows[0].value_json) + : rows[0].value_json; + expect(Array.isArray(val)).toBe(true); + expect(val).toHaveLength(1); + }); + + test('CHECK constraint rejects invalid call_shape', async () => { + let threw = false; + try { + await engine.executeRaw( + `INSERT INTO conversation_parser_llm_cache + (content_sha256, model_id, call_shape, value_json) + VALUES ($1, $2, $3, $4::jsonb)`, + ['ghi789', 'anthropic:claude-haiku-4-5', 'INVALID_SHAPE', '{}'], + ); + } catch { + threw = true; + } + expect(threw).toBe(true); + }); + + test('composite primary key prevents duplicate (sha, model, shape)', async () => { + await engine.executeRaw( + `INSERT INTO conversation_parser_llm_cache + (content_sha256, model_id, call_shape, value_json) + VALUES ($1, $2, $3, $4::jsonb)`, + ['dup1', 'anthropic:claude-haiku-4-5', 'polish', '{}'], + ); + // ON CONFLICT DO NOTHING from llm-base.ts writeDbCache — should not throw. + await engine.executeRaw( + `INSERT INTO conversation_parser_llm_cache + (content_sha256, model_id, call_shape, value_json) + VALUES ($1, $2, $3, $4::jsonb) + ON CONFLICT (content_sha256, model_id, call_shape) DO NOTHING`, + ['dup1', 'anthropic:claude-haiku-4-5', 'polish', '{"different":true}'], + ); + // First write wins on conflict. + const rows = await engine.executeRaw<{ value_json: unknown }>( + `SELECT value_json FROM conversation_parser_llm_cache WHERE content_sha256 = 'dup1'`, + ); + expect(rows).toHaveLength(1); + }); + + test('different call_shape on same (sha, model) coexists', async () => { + await engine.executeRaw( + `INSERT INTO conversation_parser_llm_cache + (content_sha256, model_id, call_shape, value_json) + VALUES ($1, $2, 'polish', $3::jsonb), ($1, $2, 'fallback', $3::jsonb)`, + ['co1', 'anthropic:claude-haiku-4-5', '{}'], + ); + const rows = await engine.executeRaw<{ call_shape: string }>( + `SELECT call_shape FROM conversation_parser_llm_cache WHERE content_sha256 = 'co1'`, + ); + expect(rows).toHaveLength(2); + const shapes = rows.map((r) => r.call_shape).sort(); + expect(shapes).toEqual(['fallback', 'polish']); + }); + + test('created_at index supports time-based pruning queries', async () => { + const rows = await engine.executeRaw<{ indexname: string }>( + `SELECT indexname FROM pg_indexes + WHERE tablename = 'conversation_parser_llm_cache' + AND indexname = 'idx_conversation_parser_llm_cache_created'`, + ); + expect(rows.length).toBe(1); + }); +}); diff --git a/test/progressive-batch/orchestrator.test.ts b/test/progressive-batch/orchestrator.test.ts new file mode 100644 index 000000000..93a9c825a --- /dev/null +++ b/test/progressive-batch/orchestrator.test.ts @@ -0,0 +1,625 @@ +/** + * v0.41.16.0 — Progressive-batch primitive unit tests. + * + * Pins every verdict branch of runProgressiveBatch (D3 fail-closed + * budget, D20 discriminated verifier shapes, D21 honest jump-to-full, + * stage-slicing, ramp-stage interactive abort, cost projection). + * + * Hermetic: no PGLite, no Postgres, no real LLM. Every BudgetTracker + * is constructed inline; the AsyncLocalStorage scope is set via + * `withBudgetTracker` from gateway.ts. Audit JSONL writes go to + * `GBRAIN_AUDIT_DIR=` to isolate from the user's real audit + * dir. We use `withEnv` from test/helpers/with-env.ts per CLAUDE.md + * R1 (test-isolation lint). + */ + +import { describe, expect, test } from 'bun:test'; +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { withEnv } from '../helpers/with-env.ts'; +import { + parseEnvStages, + resolveStages, + resolveCostCap, + sliceIntoStages, + awaitInteractiveAbort, + runProgressiveBatch, +} from '../../src/core/progressive-batch/orchestrator.ts'; +import { BudgetTracker } from '../../src/core/budget/budget-tracker.ts'; +import { withBudgetTracker } from '../../src/core/ai/gateway.ts'; +import type { + Policy, + StageReport, + Verifier, +} from '../../src/core/progressive-batch/types.ts'; + +// Helper: build a NoopVerifier with optional sampleQuality. +function makeNoopVerifier( + costPerItem = 0.001, + sampleQuality?: () => Promise<{ ok: boolean; reasons?: string[] }>, +): Verifier { + return { + kind: 'noop', + costPerItem: () => costPerItem, + sampleQuality, + }; +} + +// Helper: build an OutputCountVerifier backed by an in-memory counter. +function makeOutputCountVerifier(opts: { + initial?: number; + perItemRows?: number; + qualityOk?: boolean; + qualityReasons?: string[]; + costPerItem?: number; +}): { verifier: Verifier; bump: (n: number) => void } { + let count = opts.initial ?? 0; + const perItem = opts.perItemRows ?? 1; + return { + verifier: { + kind: 'output_count', + countBefore: async () => count, + countAfter: async () => count, + expectedDelta: (processed: number) => processed * perItem, + sampleQuality: async () => ({ + ok: opts.qualityOk ?? true, + reasons: opts.qualityReasons, + }), + costPerItem: () => opts.costPerItem ?? 0.001, + }, + bump: (n: number) => { + count += n; + }, + }; +} + +// Helper: build an IdempotentMutationVerifier backed by an in-memory counter. +function makeIdempotentVerifier(opts: { + qualityOk?: boolean; + qualityReasons?: string[]; + mutationsPerItem?: number; + costPerItem?: number; +}): { verifier: Verifier; bump: (n: number) => void; resetForStage: () => void } { + let mutations = 0; + const perItem = opts.mutationsPerItem ?? 1; + return { + verifier: { + kind: 'idempotent_mutation', + mutatedCount: async () => mutations, + expectedMutations: (processed: number) => processed * perItem, + sampleQuality: async () => ({ + ok: opts.qualityOk ?? true, + reasons: opts.qualityReasons, + }), + costPerItem: () => opts.costPerItem ?? 0.001, + }, + bump: (n: number) => { + mutations += n; + }, + resetForStage: () => { + mutations = 0; + }, + }; +} + +// Helper: collect stage reports the orchestrator emits. +function collectReports(): { + onStageReport: (r: StageReport) => void; + reports: StageReport[]; +} { + const reports: StageReport[] = []; + return { + onStageReport: (r) => { + reports.push(r); + }, + reports, + }; +} + +// Use a per-test tempdir for audit writes via GBRAIN_AUDIT_DIR. +function makeAuditEnv(): Record { + return { + GBRAIN_AUDIT_DIR: mkdtempSync(join(tmpdir(), 'pb-audit-')), + GBRAIN_PROGRESSIVE_BATCH_AUTO: '1', + GBRAIN_PROGRESSIVE_BATCH_DISABLED: undefined as unknown as string, + GBRAIN_PROGRESSIVE_BATCH_STAGES: undefined as unknown as string, + }; +} + +describe('parseEnvStages', () => { + test('null on unset', () => { + expect(parseEnvStages(undefined)).toBeNull(); + }); + test('null on empty/whitespace', () => { + expect(parseEnvStages('')).toBeNull(); + expect(parseEnvStages(' ')).toBeNull(); + }); + test('parses canonical', () => { + expect(parseEnvStages('10,100,500')).toEqual([10, 100, 500]); + }); + test('rejects non-int', () => { + expect(parseEnvStages('10,foo,500')).toBeNull(); + }); + test('rejects zero/negative', () => { + expect(parseEnvStages('0,100,500')).toBeNull(); + expect(parseEnvStages('-1,100,500')).toBeNull(); + }); + test('trims whitespace per entry', () => { + expect(parseEnvStages(' 10 , 100 , 500 ')).toEqual([10, 100, 500]); + }); +}); + +describe('resolveStages', () => { + test('env override wins', async () => { + await withEnv({ GBRAIN_PROGRESSIVE_BATCH_STAGES: '5,50,500' }, async () => { + expect(resolveStages({ label: 't', stages: [10, 100, 500] })).toEqual([ + 5, 50, 500, + ]); + }); + }); + test('policy when env unset', async () => { + await withEnv({ GBRAIN_PROGRESSIVE_BATCH_STAGES: undefined as unknown as string }, async () => { + expect(resolveStages({ label: 't', stages: [7, 70, 700] })).toEqual([ + 7, 70, 700, + ]); + }); + }); + test('default when both unset', async () => { + await withEnv({ GBRAIN_PROGRESSIVE_BATCH_STAGES: undefined as unknown as string }, async () => { + expect(resolveStages({ label: 't' })).toEqual([10, 100, 500]); + }); + }); +}); + +describe('resolveCostCap (D3 fail-closed)', () => { + test('opt-out + maxCostUsd returns policy cap', () => { + expect( + resolveCostCap( + { label: 't', requireBudgetSafetyNet: false, maxCostUsd: 1 }, + null, + ), + ).toEqual({ capUsd: 1, source: 'policy' }); + }); + test('opt-out + no cap returns Infinity', () => { + expect( + resolveCostCap({ label: 't', requireBudgetSafetyNet: false }, null), + ).toEqual({ capUsd: Infinity, source: 'uncapped' }); + }); + test('no tracker + no policy cap = NULL (fail-closed)', () => { + expect(resolveCostCap({ label: 't' }, null)).toBeNull(); + }); + test('policy cap + no tracker returns policy', () => { + expect(resolveCostCap({ label: 't', maxCostUsd: 2 }, null)).toEqual({ + capUsd: 2, + source: 'policy', + }); + }); + test('tracker + no policy cap returns tracker headroom', () => { + const t = new BudgetTracker({ label: 'test', maxCostUsd: 5 }); + const r = resolveCostCap({ label: 't' }, t); + expect(r?.capUsd).toBeCloseTo(5); + expect(r?.source).toBe('tracker'); + }); + test('both set: lower wins', () => { + const t = new BudgetTracker({ label: 'test', maxCostUsd: 10 }); + // policy is tighter + const r1 = resolveCostCap({ label: 't', maxCostUsd: 3 }, t); + expect(r1?.capUsd).toBe(3); + expect(r1?.source).toBe('policy'); + // tracker is tighter + const r2 = resolveCostCap({ label: 't', maxCostUsd: 100 }, t); + expect(r2?.capUsd).toBeCloseTo(10); + expect(r2?.source).toBe('min'); + }); +}); + +describe('sliceIntoStages', () => { + test('1000 items / canonical stages', () => { + const items = Array.from({ length: 1000 }, (_, i) => i); + const s = sliceIntoStages(items, [10, 100, 500]); + expect(s.trial.length).toBe(10); + expect(s.ramp_100.length).toBe(100); + expect(s.ramp_500.length).toBe(500); + expect(s.full.length).toBe(390); + // Disjoint + ordered + expect(s.trial[0]).toBe(0); + expect(s.ramp_100[0]).toBe(10); + expect(s.ramp_500[0]).toBe(110); + expect(s.full[0]).toBe(610); + }); + test('50 items: ramp_100 takes remaining; 500 + full empty', () => { + const items = Array.from({ length: 50 }, (_, i) => i); + const s = sliceIntoStages(items, [10, 100, 500]); + expect(s.trial.length).toBe(10); + expect(s.ramp_100.length).toBe(40); + expect(s.ramp_500.length).toBe(0); + expect(s.full.length).toBe(0); + }); + test('5 items: only trial', () => { + const items = Array.from({ length: 5 }, (_, i) => i); + const s = sliceIntoStages(items, [10, 100, 500]); + expect(s.trial.length).toBe(5); + expect(s.ramp_100.length).toBe(0); + expect(s.ramp_500.length).toBe(0); + expect(s.full.length).toBe(0); + }); + test('disabled (stages=[]) dumps everything to full', () => { + const items = Array.from({ length: 100 }, (_, i) => i); + const s = sliceIntoStages(items, []); + expect(s.trial.length).toBe(0); + expect(s.ramp_100.length).toBe(0); + expect(s.ramp_500.length).toBe(0); + expect(s.full.length).toBe(100); + }); +}); + +describe('awaitInteractiveAbort', () => { + test('ms=0 resolves false immediately', async () => { + expect(await awaitInteractiveAbort(0)).toBe(false); + }); + test('positive ms resolves false after timeout', async () => { + const start = Date.now(); + const r = await awaitInteractiveAbort(50); + expect(r).toBe(false); + expect(Date.now() - start).toBeGreaterThanOrEqual(40); + }); +}); + +describe('runProgressiveBatch — D3 fail-closed safety net', () => { + test('NO tracker + NO Policy.maxCostUsd → abort_cost_cap reason=no_budget_safety_net', async () => { + await withEnv(makeAuditEnv(), async () => { + const { onStageReport, reports } = collectReports(); + const result = await runProgressiveBatch( + [1, 2, 3], + makeNoopVerifier(), + { label: 'd3-test', onStageReport }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt?.verdict).toBe('abort_cost_cap'); + expect(result.abortedAt?.reason).toBe('no_budget_safety_net'); + expect(result.itemsProcessed).toBe(0); + expect(reports.length).toBe(1); + expect(reports[0].verdict).toBe('abort_cost_cap'); + expect(reports[0].abortReason).toBe('no_budget_safety_net'); + }); + }); + test('Policy.requireBudgetSafetyNet=false bypasses the gate (Infinity cap)', async () => { + await withEnv(makeAuditEnv(), async () => { + const { onStageReport, reports } = collectReports(); + const result = await runProgressiveBatch( + [1, 2, 3], + makeNoopVerifier(), + { + label: 'optout', + onStageReport, + requireBudgetSafetyNet: false, + }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(3); + // 3 items < trial(10) → only trial runs (the rest are empty + // stages with no audit row). + expect(reports.length).toBe(1); + expect(reports[0].verdict).toBe('proceed'); + }); + }); + test('Tracker present + no Policy.maxCostUsd → tracker headroom wins', async () => { + await withEnv(makeAuditEnv(), async () => { + const t = new BudgetTracker({ label: 'unit', maxCostUsd: 10 }); + await withBudgetTracker(t, async () => { + const { onStageReport } = collectReports(); + const result = await runProgressiveBatch( + [1, 2, 3], + makeNoopVerifier(0.001), + { label: 'tracker-test', onStageReport }, + async (slice) => ({ + succeeded: slice.length, + failed: 0, + costUsd: 0.003, + }), + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(3); + }); + }); + }); +}); + +describe('runProgressiveBatch — verifier shapes (D20)', () => { + test('OutputCountVerifier: matched delta proceeds', async () => { + await withEnv(makeAuditEnv(), async () => { + const { verifier, bump } = makeOutputCountVerifier({ perItemRows: 1 }); + const result = await runProgressiveBatch( + Array.from({ length: 5 }, (_, i) => i), + verifier, + { label: 'oc', maxCostUsd: 1 }, + async (slice) => { + bump(slice.length); + return { succeeded: slice.length, failed: 0, costUsd: 0 }; + }, + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(5); + }); + }); + test('OutputCountVerifier: zero delta → abort_count_mismatch', async () => { + await withEnv(makeAuditEnv(), async () => { + const { verifier } = makeOutputCountVerifier({ perItemRows: 1 }); + const result = await runProgressiveBatch( + Array.from({ length: 10 }, (_, i) => i), + verifier, + { label: 'oc-mismatch', maxCostUsd: 1 }, + async (slice) => ({ + // Runner reports success but verifier sees no new rows. + succeeded: slice.length, + failed: 0, + costUsd: 0, + }), + ); + expect(result.abortedAt?.verdict).toBe('abort_count_mismatch'); + expect(result.abortedAt?.reason).toBe('count_delta_outside_band'); + }); + }); + test('IdempotentMutationVerifier: matched mutation count proceeds', async () => { + await withEnv(makeAuditEnv(), async () => { + const { verifier, bump } = makeIdempotentVerifier({}); + const result = await runProgressiveBatch( + Array.from({ length: 5 }, (_, i) => i), + verifier, + { label: 'im', maxCostUsd: 1 }, + async (slice) => { + bump(slice.length); + return { succeeded: slice.length, failed: 0, costUsd: 0 }; + }, + ); + expect(result.abortedAt).toBeUndefined(); + }); + }); + test('NoopVerifier: only cost + error rate gating', async () => { + await withEnv(makeAuditEnv(), async () => { + const result = await runProgressiveBatch( + Array.from({ length: 5 }, (_, i) => i), + makeNoopVerifier(), + { label: 'np', maxCostUsd: 1 }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(5); + }); + }); + test('NoopVerifier with sampleQuality returning not-ok → abort_data_quality', async () => { + await withEnv(makeAuditEnv(), async () => { + const result = await runProgressiveBatch( + [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], + makeNoopVerifier(0.001, async () => ({ + ok: false, + reasons: ['bad row 7'], + })), + { label: 'np-bad-quality', maxCostUsd: 1 }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt?.verdict).toBe('abort_data_quality'); + expect(result.abortedAt?.reason).toBe('data_quality_sample_failed'); + const trial = result.stageReports[0]; + expect(trial.qualityReasons).toEqual(['bad row 7']); + }); + }); +}); + +describe('runProgressiveBatch — error rate + cost cap gates', () => { + test('error rate > maxErrorRate → abort_error_rate', async () => { + await withEnv(makeAuditEnv(), async () => { + const result = await runProgressiveBatch( + Array.from({ length: 10 }, (_, i) => i), + makeNoopVerifier(), + { label: 'er', maxCostUsd: 1, maxErrorRate: 0.1 }, + async (slice) => ({ + // 50% fail rate, well over the 10% threshold. + succeeded: Math.floor(slice.length / 2), + failed: Math.ceil(slice.length / 2), + costUsd: 0, + }), + ); + expect(result.abortedAt?.verdict).toBe('abort_error_rate'); + expect(result.abortedAt?.reason).toBe('error_rate_exceeded'); + }); + }); + test('cost projection > cap → abort_cost_cap', async () => { + await withEnv(makeAuditEnv(), async () => { + const result = await runProgressiveBatch( + Array.from({ length: 1000 }, (_, i) => i), + makeNoopVerifier(), + { label: 'cc', maxCostUsd: 0.5 }, + async (slice) => ({ + // $0.01/item → 10 items = $0.10 in trial, projected + // 1000-item run = $10.00 ≫ $0.50 cap. + succeeded: slice.length, + failed: 0, + costUsd: slice.length * 0.01, + }), + ); + expect(result.abortedAt?.verdict).toBe('abort_cost_cap'); + expect(result.abortedAt?.reason).toBe('cost_projected_over_cap'); + // Trial stage should have processed 10; cumulative cost should be 0.10. + expect(result.itemsProcessed).toBe(10); + expect(result.totalCostUsd).toBeCloseTo(0.1); + }); + }); +}); + +describe('runProgressiveBatch — D21 honest behavior preservation', () => { + test('GBRAIN_PROGRESSIVE_BATCH_DISABLED=1 skips ramp; goes to full', async () => { + await withEnv( + { + ...makeAuditEnv(), + GBRAIN_PROGRESSIVE_BATCH_DISABLED: '1', + }, + async () => { + const { onStageReport, reports } = collectReports(); + const result = await runProgressiveBatch( + Array.from({ length: 100 }, (_, i) => i), + makeNoopVerifier(), + { label: 'disabled', maxCostUsd: 1, onStageReport }, + async (slice) => ({ + succeeded: slice.length, + failed: 0, + costUsd: 0, + }), + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(100); + // Only the 'full' stage report should be emitted. + expect(reports.length).toBe(1); + expect(reports[0].stage).toBe('full'); + expect(reports[0].itemsInStage).toBe(100); + }, + ); + }); + test('interactiveAbortMs=0 → no Ctrl-C wait between stages', async () => { + await withEnv(makeAuditEnv(), async () => { + const start = Date.now(); + const result = await runProgressiveBatch( + Array.from({ length: 120 }, (_, i) => i), + makeNoopVerifier(), + { label: 'noramp', maxCostUsd: 1, interactiveAbortMs: 0 }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt).toBeUndefined(); + // Should run quickly without any 10s waits between stages. + expect(Date.now() - start).toBeLessThan(500); + }); + }); +}); + +describe('runProgressiveBatch — onStageReport caller abort', () => { + test('caller returns {abort:true} after trial → abort_explicit', async () => { + await withEnv(makeAuditEnv(), async () => { + let calls = 0; + const result = await runProgressiveBatch( + Array.from({ length: 100 }, (_, i) => i), + makeNoopVerifier(), + { + label: 'explicit-abort', + maxCostUsd: 1, + onStageReport: (r) => { + calls++; + if (r.stage === 'trial') return { abort: true }; + return undefined; + }, + }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt?.verdict).toBe('abort_explicit'); + expect(result.abortedAt?.reason).toBe('caller_signaled_abort'); + // Trial completed but caller signaled abort. + expect(result.itemsProcessed).toBe(10); + expect(calls).toBe(1); + }); + }); +}); + +describe('runProgressiveBatch — multi-stage success path', () => { + test('1000 items: trial(10) → ramp_100 → ramp_500 → full(380); all stages reported', async () => { + await withEnv(makeAuditEnv(), async () => { + const { onStageReport, reports } = collectReports(); + const result = await runProgressiveBatch( + Array.from({ length: 1000 }, (_, i) => i), + makeNoopVerifier(), + { label: 'multi', maxCostUsd: 100, onStageReport }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(1000); + expect(result.stagesCompleted).toEqual(['trial', 'ramp_100', 'ramp_500', 'full']); + // Four stage reports (one per stage; the full stage processes 380). + expect(reports).toHaveLength(4); + expect(reports[0].stage).toBe('trial'); + expect(reports[0].itemsInStage).toBe(10); + expect(reports[1].stage).toBe('ramp_100'); + expect(reports[1].itemsInStage).toBe(100); + expect(reports[2].stage).toBe('ramp_500'); + expect(reports[2].itemsInStage).toBe(500); + expect(reports[3].stage).toBe('full'); + expect(reports[3].itemsInStage).toBe(390); + // Cumulative processed reaches 1000 by the last stage. + expect(reports[3].itemsProcessedCumulative).toBe(1000); + // All verdicts are proceed. + expect(reports.every((r) => r.verdict === 'proceed')).toBe(true); + }); + }); + + test('IdempotentMutationVerifier: matched mutation count across stages', async () => { + await withEnv(makeAuditEnv(), async () => { + const { verifier, bump, resetForStage } = makeIdempotentVerifier({}); + void resetForStage; + const result = await runProgressiveBatch( + Array.from({ length: 25 }, (_, i) => i), + verifier, + { label: 'multi-im', maxCostUsd: 1 }, + async (slice) => { + bump(slice.length); + return { succeeded: slice.length, failed: 0, costUsd: 0 }; + }, + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(25); + }); + }); + + test('IdempotentMutationVerifier: mutation count off → abort_mutation_mismatch', async () => { + await withEnv(makeAuditEnv(), async () => { + const { verifier } = makeIdempotentVerifier({}); + // Runner claims success but never bumps the mutation counter. + const result = await runProgressiveBatch( + Array.from({ length: 10 }, (_, i) => i), + verifier, + { label: 'im-mm', maxCostUsd: 1 }, + async (slice) => ({ succeeded: slice.length, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt?.verdict).toBe('abort_mutation_mismatch'); + expect(result.abortedAt?.reason).toBe('mutation_count_outside_band'); + }); + }); + + test('cumulative cost crosses cap mid-multi-stage → abort_cost_cap', async () => { + await withEnv(makeAuditEnv(), async () => { + const result = await runProgressiveBatch( + Array.from({ length: 200 }, (_, i) => i), + makeNoopVerifier(), + { label: 'cc-mid', maxCostUsd: 0.05 }, + async (slice) => ({ + // Trial: 10 items × $0.001 = $0.01 cumulative; projected + // 200 × $0.001 = $0.20 ≫ $0.05 cap. + succeeded: slice.length, + failed: 0, + costUsd: slice.length * 0.001, + }), + ); + expect(result.abortedAt?.verdict).toBe('abort_cost_cap'); + expect(result.abortedAt?.reason).toBe('cost_projected_over_cap'); + }); + }); +}); + +describe('runProgressiveBatch — degenerate inputs', () => { + test('empty item list runs no stages but reports the run happened', async () => { + await withEnv(makeAuditEnv(), async () => { + const { onStageReport, reports } = collectReports(); + const result = await runProgressiveBatch( + [] as number[], + makeNoopVerifier(), + { label: 'empty', maxCostUsd: 1, onStageReport }, + async () => ({ succeeded: 0, failed: 0, costUsd: 0 }), + ); + expect(result.abortedAt).toBeUndefined(); + expect(result.itemsProcessed).toBe(0); + // We DO emit one report for the zero-item full stage so audit + // shows the run was attempted. + expect(reports.length).toBe(1); + expect(reports[0].stage).toBe('full'); + expect(reports[0].verdict).toBe('proceed'); + }); + }); +}); diff --git a/test/progressive-batch/retrofit-wrap.test.ts b/test/progressive-batch/retrofit-wrap.test.ts new file mode 100644 index 000000000..b4f269db6 --- /dev/null +++ b/test/progressive-batch/retrofit-wrap.test.ts @@ -0,0 +1,84 @@ +/** + * v0.41.16.0 — retrofitWrap helper unit tests. + * + * Pins the thin wrapper used by 8 of the 9 v0.41.16.0 retrofits: + * - Default opt-out of D3 safety net + zero ramp grace. + * - Audit JSONL still writes. + * - Cost projection rolls up correctly. + * - Runner error counts surface. + */ + +import { describe, expect, test } from 'bun:test'; +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { withEnv } from '../helpers/with-env.ts'; +import { retrofitWrap } from '../../src/core/progressive-batch/retrofit-wrap.ts'; + +function auditEnv(): Record { + return { + GBRAIN_AUDIT_DIR: mkdtempSync(join(tmpdir(), 'rw-audit-')), + GBRAIN_PROGRESSIVE_BATCH_AUTO: '1', + }; +} + +describe('retrofitWrap', () => { + test('opt-out defaults: runs without BudgetTracker safety net', async () => { + await withEnv(auditEnv(), async () => { + const r = await retrofitWrap({ + label: 'unit-test', + items: [1, 2, 3, 4, 5], + runner: async (rows) => ({ succeeded: rows.length, failed: 0, costUsd: 0 }), + }); + expect(r.abortedAt).toBeUndefined(); + expect(r.itemsProcessed).toBe(5); + }); + }); + + test('passes per-item cost into projection', async () => { + await withEnv(auditEnv(), async () => { + const r = await retrofitWrap({ + label: 'cost-projection', + items: Array.from({ length: 100 }, (_, i) => i), + costPerItem: 0.001, + runner: async (rows) => ({ + succeeded: rows.length, + failed: 0, + costUsd: rows.length * 0.001, + }), + }); + expect(r.abortedAt).toBeUndefined(); + expect(r.totalCostUsd).toBeCloseTo(0.1); + }); + }); + + test('runner failures roll up into the totals', async () => { + await withEnv(auditEnv(), async () => { + const r = await retrofitWrap({ + label: 'failure-test', + items: Array.from({ length: 10 }, (_, i) => i), + runner: async (rows) => ({ + succeeded: 0, + failed: rows.length, + costUsd: 0, + }), + }); + // Default maxErrorRate=0.02 — 100% failure aborts at trial. + expect(r.abortedAt?.verdict).toBe('abort_error_rate'); + }); + }); + + test('interactiveAbortMs=0 default → no grace period (cron-safe)', async () => { + await withEnv(auditEnv(), async () => { + const start = Date.now(); + const r = await retrofitWrap({ + label: 'cron-safe', + items: Array.from({ length: 200 }, (_, i) => i), + runner: async (rows) => ({ succeeded: rows.length, failed: 0, costUsd: 0 }), + }); + expect(r.abortedAt).toBeUndefined(); + // Should sail through all 4 stages with no grace pause. + expect(Date.now() - start).toBeLessThan(500); + }); + }); +});