diff --git a/CLAUDE.md b/CLAUDE.md index f1733dfe9..ea73092bd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -165,7 +165,7 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. -- `src/core/conversation-parser/` (v0.41.16.0) — 12-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (12 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/conversation-parser/` (v0.41.16.0, v0.41.21.0) — 13-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.21.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape wintermute's meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. diff --git a/llms-full.txt b/llms-full.txt index 28ed16193..b75f0d824 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -307,7 +307,7 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. -- `src/core/conversation-parser/` (v0.41.16.0) — 12-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (12 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/conversation-parser/` (v0.41.16.0, v0.41.21.0) — 13-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.21.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape wintermute's meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract.