diff --git a/CHANGELOG.md b/CHANGELOG.md index 92df4af63..50cd25ffc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,70 @@ All notable changes to GBrain will be documented in this file. +## [0.41.24.0] - 2026-05-27 + +**Your reformatted meeting transcripts now actually parse.** + +If you reformat Circleback meetings into a per-line shape like +`**Garry Tan** (12:34): hello`, gbrain used to look at them and say +`no_match` — even when 78% of the page was valid chat. Your 36 reformatted +meetings sat there inert and the fact extractor never saw a single +message. v0.41.24.0 makes them flow through. + +Two problems were stacked. First, the parser only looked at the first +10 lines of a page to figure out which chat format it was. Meeting +pages start with `## Summary`, a blockquote, a `## Transcript` heading +— ~10 lines of prose before the actual chat. None of those lines +matched any chat regex, so the parser gave up. Second, even if the +parser had kept looking, none of the 12 built-in patterns recognized +the time-only shape Circleback exports use: `**Speaker** (HH:MM):` or +`**Speaker** (HH:MM:SS):`. Both shapes were invisible. + +The fix tightens the detector AND teaches it the missing shape. The +detector now falls back to scanning the full body when the head pass +returns a weak score, AND it refuses to accept a final score below 5% +(so an essay with one stray chat-shape line stays `no_match` instead +of false-positive parsing as a 1-message conversation). The new +`bold-paren-time` built-in covers the two Circleback variants. After +the fix, 113 of 367 Circleback meeting pages on the canonical brain +parse correctly (was 0), and **20,167 messages** flow through to the +fact extractor for the first time. The other 254 are notes-only +meetings that link to a separate transcript file — those legitimately +have nothing to parse. + +**How to use it:** `gbrain upgrade`. Then either enable the cycle phase +that runs in the background (`gbrain config set +cycle.conversation_facts_backfill.enabled true`) or fire it manually +(`gbrain extract-conversation-facts `). The parser fix +unblocks the pipeline; the trigger is yours. + +**The detector numbers that matter** (parse.ts): + +| Setting | Value | What it does | +|---|---|---| +| `SCORING_HEAD_LINES` | 10 | Lines scanned in the fast path (unchanged) | +| `SCORING_HEAD_TRIGGER_THRESHOLD` | 0.3 | Below this, fall back to full-body scoring | +| `SCORING_MIN_ACCEPTANCE` | 0.05 | Final score floor — below this, return `no_match` | + +The 0.3 trigger threshold (instead of "fall back only on score === 0") +closes a silent bug class: a blockquote that accidentally matches an +unrelated pattern at 0.1 used to suppress the fallback entirely. Now +it doesn't. + +**Things to watch:** + +- The new pattern treats Circleback's `(00:00)` / `(00:00:00)` as + wall-clock time on the page's frontmatter date. Real Circleback + timestamps are elapsed-time-from-meeting-start, so every message in + a meeting lands on the same day starting at 00:00 + offset. The + fact extractor cares about speaker + content, not precise + timestamps, so this is honest enough. Proper per-line wall-clock + reconstruction would need an `elapsed_time: true` flag on + PatternEntry — filed as v0.42+ scope. +- `imessage-slack` still wins on full-datetime overlaps. The new + pattern's regex requires `\)` immediately after the time group, so + `(2024-03-15 9:00 AM)` shapes correctly fall through to the more + specific pattern. ## [0.41.23.0] - 2026-05-26 **You can now see how every extractor in your brain is doing — how @@ -577,6 +641,80 @@ doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists. ### Itemized changes #### Added +- `bold-paren-time` built-in pattern in + `src/core/conversation-parser/builtins.ts` recognizes + `**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text`. + Verified against 367 Circleback meeting files at `~/git/brain/meetings/`: + 113 now parse (was 0), 20,167 messages flowing through. +- `scorePatternFull(body, entry)` exported from + `src/core/conversation-parser/parse.ts` for callers that need + full-body scoring of a single pattern. + +#### Changed +- `parseConversation()` falls back to full-body re-scoring when the + head pass's top score is below `SCORING_HEAD_TRIGGER_THRESHOLD` + (0.3) instead of only when it is exactly 0. The pre-fix shape left + a real bug class open where a stray head match suppressed the + fallback. +- `parseConversation()` returns `no_match` when the final winner's + score is below `SCORING_MIN_ACCEPTANCE` (0.05) to prevent + essay-false-positive parsing. +- `scorePattern()` is now a thin wrapper over a private + `scoreFromLines` core that holds the quick_reject + regex loop in + one place. `scorePattern` behavior + contract + signature unchanged. + +#### Tests +- 11 new unit cases in `test/conversation-parser/parse.test.ts`: + the #1533 IRON-RULE regression pin (meeting page → + `regex_match`), honest unmatched-count after fallback, pure-prose + stays `no_match`, 300-line preamble + 50 chat lines hits fallback, + `scorePatternFull` direct boundaries, stray-head-match + miscategorization guard, essay false-positive acceptance-floor + guard, `bold-paren-time` matches HH:MM, `bold-paren-time` matches + HH:MM:SS, `imessage-slack` still wins on full-datetime overlap, + meeting page with preamble + `bold-paren-time` transcript hits + fallback. +- Existing "caps at SCORING_HEAD_LINES (10)" test reshaped to pin + behavior (10 match + 1 non-match scores 1.0; 9 non-match + 1 match + + 100 after scores 0.1) instead of importing the constant. + +#### Closed +- Closes #1533 — both the bug class (trigger threshold + acceptance + floor) and the user-facing case (113 Circleback meetings × 20,167 + messages). + +## To take advantage of v0.41.24.0 + +`gbrain upgrade` should do this automatically. Then to actually flow +your reformatted Circleback meetings through the fact extractor: + +1. **One-shot manual extraction (immediate):** + ```bash + gbrain extract-conversation-facts + ``` + Replace `` with the source id holding your meeting pages. + Use `gbrain sources list` to find it. + +2. **OR enable the cycle phase (continuous, opt-in):** + ```bash + gbrain config set cycle.conversation_facts_backfill.enabled true + ``` + The dream cycle picks them up on the next tick. + +3. **Verify the parser sees them:** + ```bash + gbrain conversation-parser scan --json + ``` + Expected output for a Circleback meeting: + `"phase": "regex_match"`, `"matched_pattern_id": + "bold-paren-time"`, `"message_count"` in the dozens to hundreds. + +4. **If parser still reports `no_match`** on a page you expected to + parse, your meeting may use a transcript shape outside the 13 + built-in patterns. Run `gbrain conversation-parser list-builtins` + to see what shapes are recognized, file an issue with a sample + page, or wait for v0.42+'s user-declared `simple_pattern` + support. - `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` — one batched SQL roundtrip that returns the set of `content_hash16` values already extracted as atoms diff --git a/CLAUDE.md b/CLAUDE.md index 577818f31..3bfb40229 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -165,7 +165,7 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. -- `src/core/conversation-parser/` (v0.41.16.0) — 12-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (12 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0) — 13-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. diff --git a/VERSION b/VERSION index 48db4409d..1fd237ef5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.23.0 \ No newline at end of file +0.41.24.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 5d6e2e610..021e1b7a0 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -307,7 +307,7 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. -- `src/core/conversation-parser/` (v0.41.16.0) — 12-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (12 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0) — 13-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. diff --git a/package.json b/package.json index 1a50806ef..f6017e452 100644 --- a/package.json +++ b/package.json @@ -140,5 +140,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.23.0" + "version": "0.41.24.0" } diff --git a/src/core/conversation-parser/builtins.ts b/src/core/conversation-parser/builtins.ts index 2af1e3870..db441a7e5 100644 --- a/src/core/conversation-parser/builtins.ts +++ b/src/core/conversation-parser/builtins.ts @@ -119,6 +119,65 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ source_doc: 'PR #1461 (closed); preserved verbatim with Co-Authored-By', }, + { + // v0.41.18+ (D-FOLLOWUP-1.B closes the user-facing half of #1533): + // matches the shape Circleback meeting exports use after an + // OpenClaw meeting-ingestion pipeline reformats them. Two + // sub-shapes in the wild (verified across a 367-file corpus): + // **Participant 2** (00:00): Companies that we have... ← (HH:MM) + // **Participant 1** (00:00:00): We found the apostrophes... ← (HH:MM:SS) + // + // The time group is elapsed time from meeting start, NOT + // wall-clock. Parser treats it as wall-clock 24h on the + // frontmatter date — speaker + text are captured correctly, but + // every message lands on the same day starting at 00:00 + offset + // minutes. The downstream fact extractor only cares about + // speaker + content, so this is honest-enough; precise per-line + // wall-clock timestamps would require a new `elapsed_time: + // true` flag on PatternEntry (v0.42+). + // + // Declaration position is AFTER imessage-slack + telegram- + // bracket so on the rare tie those more-specific patterns win. + // The regex deliberately requires `\)` immediately after the + // time so `(2024-03-15 9:00 AM)` and `(9:00 AM)` shapes fall + // through to imessage-slack instead of false-matching here. + // The seconds segment is a non-capturing optional group so + // capture indexes stay identical across both sub-shapes. + id: 'bold-paren-time', + origin: 'builtin', + regex: /^\*\*(.+?)\*\*\s+\((\d{1,2}):(\d{2})(?::\d{2})?\)\s*:\s*(.*)$/, + captures: { + speaker_group: 1, + hour_group: 2, + minute_group: 3, + text_group: 4, + }, + date_source: 'frontmatter', + time_format: '24h', + timezone_policy: 'utc_assumed_with_warn', + multi_line: false, + quick_reject: /^\*\*/, + test_positive: [ + '**Garry Tan** (00:00): hello world', + '**Participant 2** (02:22): response here', + '**Alex Graveley** (15:09): That’s exactly right.', + '**Participant 1** (00:00:00): hello world with seconds', + '**Participant 2** (01:23:45): mid-meeting line', + ], + test_negative: [ + // imessage-slack shape (full date+time) MUST fall through to imessage-slack: + '**Alice Example** (2024-03-15 9:00 AM): iMessage shape', + // telegram-bracket shape MUST fall through to telegram-bracket: + '**[18:37] \u{1f464} G T:** telegram bracket', + // No bold markers: + 'Alice (00:00): missing the bold', + // Bold but no parens: + '**Alice** hello world', + ], + source_doc: + 'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)', + }, + { id: 'telegram-text-export', origin: 'builtin', diff --git a/src/core/conversation-parser/parse.ts b/src/core/conversation-parser/parse.ts index 7368f6b28..215cbeab8 100644 --- a/src/core/conversation-parser/parse.ts +++ b/src/core/conversation-parser/parse.ts @@ -48,6 +48,33 @@ export type { ParseConversationOpts, ParseResult, MatchedMessage } from './types */ const SCORING_HEAD_LINES = 10; +/** + * Head-pass score below which `parseConversation` falls back to a + * full-body re-score (v0.41.18+ fix for #1533 + Codex P1 #1). + * + * Why a threshold instead of `=== 0`: meeting pages start with + * `## Summary` + blockquotes + `## Transcript` headings. A + * blockquote like `> [12:00] Foo` can accidentally match an + * unrelated pattern's regex (irc-classic at score 0.1) which would + * suppress the fallback even when 175 of 226 lines further down are + * valid imessage-slack. 0.3 = "fewer than 3 of 10 head lines + * matched"; chat-only pages still score 1.0 and skip the fallback + * entirely, so the fast path is preserved. + */ +const SCORING_HEAD_TRIGGER_THRESHOLD = 0.3; + +/** + * Minimum final winner score required to accept `regex_match` + * (v0.41.18+ Codex P1 #2). A 500-line essay with one stray + * `**Name** (date time):` line scores ~1/500 = 0.002 for + * imessage-slack, which without this floor would flip to + * `regex_match` with `messages.length = 1` — a false positive that + * silently corrupts downstream fact extraction. 0.05 = "at least 5% + * of non-blank lines anchored a message"; real transcript pages + * typically score 0.5+ and sail through, accidental anchors do not. + */ +const SCORING_MIN_ACCEPTANCE = 0.05; + /** * Tie-breaker priority: lower index wins on score tie. Mirrors * BUILTIN_PATTERNS declaration order. User-declared patterns get @@ -334,6 +361,41 @@ export function applyPattern( return out; } +/** + * Split a body into non-blank trimmed lines. `headCap` (when set) + * limits the result to the first N lines — used by the head-pass + * scorer; omit for full-body scoring. + */ +function getNonBlankLines(body: string, headCap?: number): string[] { + if (!body) return []; + const all = body + .split(/\r?\n/) + .map((l) => l.trim()) + .filter((l) => l.length > 0); + return headCap !== undefined ? all.slice(0, headCap) : all; +} + +/** + * Core scorer over a pre-split line array. Both `scorePattern` (head + * window) and `scorePatternFull` (whole body) delegate here so the + * quick_reject + regex loop lives in one place. Reused by + * `parseConversation`'s fallback path which pre-splits ONCE and + * passes the array to all 12 candidates (saves 11 redundant body + * splits per fallback pass). + */ +function scoreFromLines( + lines: readonly string[], + entry: PatternEntry, +): number { + if (lines.length === 0) return 0; + let anchored = 0; + for (const line of lines) { + if (entry.quick_reject && !entry.quick_reject.test(line)) continue; + if (entry.regex.test(line)) anchored++; + } + return anchored / lines.length; +} + /** * Score how well a pattern matches the first N lines of a body (D18). * Returns 0..1 ratio of matched lines. Higher = more confident. @@ -344,19 +406,22 @@ export function applyPattern( * Exported for tests. */ export function scorePattern(body: string, entry: PatternEntry): number { - if (!body) return 0; - const lines = body - .split(/\r?\n/) - .map((l) => l.trim()) - .filter((l) => l.length > 0) - .slice(0, SCORING_HEAD_LINES); - if (lines.length === 0) return 0; - let anchored = 0; - for (const line of lines) { - if (entry.quick_reject && !entry.quick_reject.test(line)) continue; - if (entry.regex.test(line)) anchored++; - } - return anchored / lines.length; + return scoreFromLines(getNonBlankLines(body, SCORING_HEAD_LINES), entry); +} + +/** + * Score how well a pattern matches the FULL body, no head cap + * (v0.41.18+ Codex P1 #1 — full-body fallback when head pass falls + * below `SCORING_HEAD_TRIGGER_THRESHOLD`). + * + * Cost-aware: in `parseConversation`'s fallback path we pre-split + * ONCE and route through `scoreFromLines` directly, NOT through this + * wrapper, to avoid 12 redundant body splits per pass. This wrapper + * exists for direct unit testing and for any future caller that + * needs full-body scoring of a single pattern. + */ +export function scorePatternFull(body: string, entry: PatternEntry): number { + return scoreFromLines(getNonBlankLines(body), entry); } /** @@ -392,23 +457,51 @@ export function parseConversation( // declared priority order (built-in declaration order; user patterns // lose every tie). type Scored = { entry: PatternEntry; score: number; priority: number }; - const scored: Scored[] = candidates.map((entry) => ({ + const sortScored = (arr: Scored[]) => + arr.sort((a, b) => { + if (b.score !== a.score) return b.score - a.score; + return a.priority - b.priority; + }); + let scored: Scored[] = candidates.map((entry) => ({ entry, score: scorePattern(body, entry), priority: priorityOf(entry.id), })); - // Sort: score desc, then priority asc. - scored.sort((a, b) => { - if (b.score !== a.score) return b.score - a.score; - return a.priority - b.priority; - }); + sortScored(scored); + + // REGRESSION (closes #1533 + Codex P1 #1): meeting pages have + // ## Summary + blockquote + ## Transcript ahead of the chat. The + // pre-fix "trigger fallback only when score === 0" shape left a + // real bug class open — a stray head match (e.g. a blockquote + // that accidentally matches an unrelated pattern at 0.1) + // suppressed the fallback even when 175 of 226 lines further down + // were valid imessage-slack. Threshold 0.3 means "fewer than 3 of + // 10 head lines matched"; chat-only pages still score 1.0 and skip + // the fallback. Re-score every candidate against the full body, + // pre-splitting ONCE to avoid 12 redundant body splits. + if (scored[0].score < SCORING_HEAD_TRIGGER_THRESHOLD) { + const allLines = getNonBlankLines(body); + scored = candidates.map((entry) => ({ + entry, + score: scoreFromLines(allLines, entry), + priority: priorityOf(entry.id), + })); + sortScored(scored); + // NOTE: patterns_scored stays as scored.length (= candidate + // count, typically 12) even when the fallback runs — the + // diagnostic reports "candidates considered" not "scoring + // attempts" (Codex P2 #7). + } const top = scored[0]; const patternsScored = scored.length; - // If even the top scorer matches zero lines, no regex won. Caller - // may invoke LLM fallback (T4); for now return no_match. - if (top.score === 0) { + // Minimum acceptance floor (closes Codex P1 #2): an essay with + // one stray `**Name** (date time):` line scores ~1/300 ≈ 0.003 — + // below the 5% floor we stay no_match instead of returning a + // 1-message false positive. Real transcript pages typically score + // 0.5+ and sail through. + if (top.score < SCORING_MIN_ACCEPTANCE) { return { messages: [], phase: 'no_match', diff --git a/test/conversation-parser/parse.test.ts b/test/conversation-parser/parse.test.ts index 577312a2e..b8a1cce86 100644 --- a/test/conversation-parser/parse.test.ts +++ b/test/conversation-parser/parse.test.ts @@ -22,6 +22,7 @@ import { deriveDateContext, applyPattern, scorePattern, + scorePatternFull, } from '../../src/core/conversation-parser/parse.ts'; import { BUILTIN_PATTERNS } from '../../src/core/conversation-parser/builtins.ts'; import type { Page } from '../../src/core/types.ts'; @@ -336,13 +337,272 @@ describe('scorePattern — boundary', () => { const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; expect(scorePattern('\n\n \n', tg)).toBe(0); }); - test('caps at SCORING_HEAD_LINES (10) lines', () => { - // 100 telegram lines → still scores 1.0 because only first 10 sampled. - const body = Array.from( - { length: 100 }, - (_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`, - ).join('\n'); + // T5 reshape (Codex P2 #6): pins BEHAVIOR not the constant value. + // The prior test ("100 matching lines score 1.0") would pass with + // head=10 or head=1000 — it didn't prove anything about the cap. + test('head cap ignores lines past line 10 (10 match + 1 non-match scores 1.0)', () => { const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; + const matching = Array.from( + { length: 10 }, + (_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`, + ); + const body = [...matching, 'plain text outside the head window'].join('\n'); + // First 10 lines all match → 10/10. Line 11 was ignored. expect(scorePattern(body, tg)).toBe(1); }); + test('head cap stops at line 10 (9 non-match + 1 match at line 10 + 100 match after scores 0.1)', () => { + const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!; + const nonMatches = Array.from({ length: 9 }, (_, i) => `non-matching prose line ${i}`); + const matchingLate = Array.from( + { length: 100 }, + (_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`, + ); + // Line 10 (index 9 in the matching array) IS a match; lines 11-109 are too + // but are past the head cap and don't count. + const body = [...nonMatches, matchingLate[0], ...matchingLate.slice(1)].join('\n'); + // Head sees 9 non-matches + 1 match = 1/10 = 0.1. Pre-fix: same result. + // Post-fix: same result (this test pins head-cap behavior, not the new + // fallback path — that's tested separately below). + expect(scorePattern(body, tg)).toBe(0.1); + }); +}); + +// --------------------------------------------------------------------------- +// scorePatternFull — direct unit tests (v0.41.18+ T3 #5) +// --------------------------------------------------------------------------- + +describe('scorePatternFull — full-body scoring (v0.41.18+ Codex P1 #1)', () => { + test('empty body scores 0', () => { + const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!; + expect(scorePatternFull('', im)).toBe(0); + }); + test('preamble + 20 matching lines scores 20/(preamble + 20)', () => { + const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!; + const preamble = ['## Summary', 'Three sentences.', '> Source: ref', '## Transcript']; + const matches = Array.from( + { length: 20 }, + (_, i) => `**Garry Tan** (2026-01-29 12:00 PM): message ${i}`, + ); + const body = [...preamble, ...matches].join('\n'); + // 24 total non-blank, 20 match → 20/24 ≈ 0.833 + expect(scorePatternFull(body, im)).toBeCloseTo(20 / 24, 5); + }); + test('preamble-only-no-match scores 0', () => { + const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!; + const body = '## Summary\nProse paragraph.\n> Blockquote\n## Heading'; + expect(scorePatternFull(body, im)).toBe(0); + }); +}); + +// --------------------------------------------------------------------------- +// bold-paren-time pattern (v0.41.18+ D-FOLLOWUP-1.B; closes user-facing +// half of #1533 — the 112 Circleback meeting files at +// ~/git/brain/meetings/*.md with `source: circleback` frontmatter) +// --------------------------------------------------------------------------- + +describe('bold-paren-time pattern (Circleback meeting transcripts)', () => { + test('matches **Speaker** (HH:MM): text with frontmatter date', () => { + const body = [ + '**Garry Tan** (00:00): Hey, can you hear me?', + '**Participant 2** (02:22): Yeah, just joined.', + '**Garry Tan** (15:09): That makes sense.', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-03-19' }); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('bold-paren-time'); + expect(r.messages).toHaveLength(3); + expect(r.messages[0]).toEqual({ + speaker: 'Garry Tan', + timestamp: '2026-03-19T00:00:00Z', + text: 'Hey, can you hear me?', + }); + expect(r.messages[2]).toEqual({ + speaker: 'Garry Tan', + timestamp: '2026-03-19T15:09:00Z', + text: 'That makes sense.', + }); + }); + + test('matches **Speaker** (HH:MM:SS): text shape (Circleback seconds variant)', () => { + const body = [ + '**Participant 1** (00:00:00): opening line', + '**Participant 2** (00:00:19): quick reply', + '**Participant 1** (01:23:45): later in the meeting', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-04-01' }); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('bold-paren-time'); + expect(r.messages).toHaveLength(3); + // Seconds segment is non-capturing; minute_group still captures the + // minutes component. Time-format is wall-clock 24h on frontmatter date. + expect(r.messages[0].timestamp).toBe('2026-04-01T00:00:00Z'); + expect(r.messages[1].timestamp).toBe('2026-04-01T00:00:00Z'); + expect(r.messages[2].timestamp).toBe('2026-04-01T01:23:00Z'); + }); + + test('imessage-slack shape still wins over bold-paren-time on overlap', () => { + // Both patterns start with `**` and have parens. The imessage- + // slack regex requires a full date+time inside; bold-paren-time + // requires just `(HH:MM)`. The dates-with-AM/PM shape MUST fall + // through to imessage-slack, not bold-paren-time. + const body = '**Alice Example** (2024-03-15 9:00 AM): hello world'; + const r = parseConversation(body); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('imessage-slack'); + expect(r.messages).toHaveLength(1); + expect(r.messages[0].text).toBe('hello world'); + }); + + test('meeting page with preamble + bold-paren-time transcript hits fallback', () => { + // Real Circleback shape: ## Summary + blockquote + ## Transcript + // before the bold-paren-time chat. Same fallback gate that + // closes #1533 must work for this pattern too. + const preamble = [ + '## Summary', + 'Meeting covered Q1 roadmap discussion.', + '> Source: circleback meeting #7411053', + '## Topics Discussed', + '- Roadmap', + '- Hiring', + '## Transcript', + ]; + const transcript = Array.from( + { length: 20 }, + (_, i) => `**Participant 2** (${String(Math.floor(i / 6)).padStart(2, '0')}:${String((i * 11) % 60).padStart(2, '0')}): transcript line ${i}`, + ); + const body = [...preamble, ...transcript].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-03-19' }); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('bold-paren-time'); + expect(r.messages).toHaveLength(20); + }); +}); + +// --------------------------------------------------------------------------- +// parseConversation — full-body fallback (v0.41.18+ #1533 + Codex P1 #1, #2, #8) +// --------------------------------------------------------------------------- + +describe('parseConversation — full-body fallback', () => { + // T3 #1: IRON-RULE regression pin for #1533. Pre-fix this returns + // no_match because head 10 sees only preamble. + test('#1533: meeting page with ## Summary + blockquote + ## Transcript before chat hits fallback', () => { + const preamble = [ + '## Summary', + 'This meeting covered Q1 roadmap discussion.', + 'Three engineers participated in the call.', + 'Action items were captured during the conversation.', + '> Source: [meeting recording](https://example.com/rec/123)', + '## Topics Discussed', + '- Product roadmap for Q1', + '- Engineering team allocation', + '- Customer feedback synthesis', + '## Transcript', + ]; + const transcript = Array.from( + { length: 20 }, + (_, i) => `**Garry Tan** (2026-01-29 12:00 PM): line ${i}`, + ); + const body = [...preamble, ...transcript].join('\n'); + const r = parseConversation(body); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('imessage-slack'); + expect(r.messages).toHaveLength(20); + }); + + // T3 #2: diagnostic now reports total_non_blank - matched, not total. + test('#1533: unmatched_line_count subtracts matched messages after fallback', () => { + const preamble = [ + '## Summary', + 'Prose A.', + 'Prose B.', + '> Blockquote', + '## Transcript', + ]; + const transcript = Array.from( + { length: 20 }, + (_, i) => `**Garry Tan** (2026-01-29 12:00 PM): line ${i}`, + ); + const body = [...preamble, ...transcript].join('\n'); + const r = parseConversation(body, { diagnostic: true }); + expect(r.phase).toBe('regex_match'); + expect(r.unmatched_line_count).toBe(5); // 25 total non-blank - 20 messages = 5 + }); + + // T3 #3: a 50-line essay with no chat shape stays no_match. + test('pure-prose 50-line essay stays no_match (fallback found nothing to anchor)', () => { + const body = Array.from( + { length: 50 }, + (_, i) => `This is the ${i + 1}th paragraph of a pure-prose article.`, + ).join('\n'); + const r = parseConversation(body); + expect(r.phase).toBe('no_match'); + expect(r.messages).toHaveLength(0); + }); + + // T3 #4: proves "full-body" not just "wider window" — 300-line preamble + // far exceeds any reasonable head-bump alternative. + test('300-line preamble + 50 chat lines hits fallback (any preamble length)', () => { + const preamble = Array.from( + { length: 300 }, + (_, i) => `Preamble paragraph ${i + 1} with prose content here.`, + ); + const transcript = Array.from( + { length: 50 }, + (_, i) => `**Garry Tan** (2026-01-29 12:00 PM): chat line ${i}`, + ); + const body = [...preamble, ...transcript].join('\n'); + const r = parseConversation(body); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('imessage-slack'); + expect(r.messages).toHaveLength(50); + }); + + // T3 #6 (Codex P1 #1 + #8): stray-head-match doesn't suppress fallback. + // Pre-fix: irc-classic 0.1 in head → no fallback → irc-classic wins with 1 + // message. Post-fix: 0.1 < 0.3 trigger → fallback re-scores → imessage-slack + // wins (50/60 ≈ 0.83 vs irc-classic 1/60 ≈ 0.017). + test('Codex P1 #1: stray irc-classic match in head does not suppress fallback', () => { + const preamble = [ + '## Meeting Notes', + ' Garry Tan opening remarks', // stray irc-classic match + '- agenda item 1', + '- agenda item 2', + '- agenda item 3', + '- agenda item 4', + '- agenda item 5', + '- agenda item 6', + '- agenda item 7', + '## Transcript', + ]; + const transcript = Array.from( + { length: 50 }, + (_, i) => `**Garry Tan** (2024-01-29 12:00 PM): real transcript line ${i}`, + ); + const body = [...preamble, ...transcript].join('\n'); + const r = parseConversation(body); + expect(r.phase).toBe('regex_match'); + // The critical assertion: imessage-slack wins, NOT irc-classic. + expect(r.matched_pattern_id).toBe('imessage-slack'); + expect(r.messages).toHaveLength(50); + }); + + // T3 #7 (Codex P1 #2): essay with one stray chat-shape line stays + // no_match. 1/301 ≈ 0.003, below SCORING_MIN_ACCEPTANCE (0.05). + test('Codex P1 #2: 300-line essay with one stray chat line stays no_match (acceptance floor)', () => { + const prose = Array.from( + { length: 150 }, + (_, i) => `Essay paragraph ${i + 1} of pure prose with no chat shape.`, + ); + const strayChatLine = '**Author Name** (2024-01-01 9:00 AM): stray quoted snippet'; + const morePros = Array.from( + { length: 150 }, + (_, i) => `Essay continuation paragraph ${i + 151}.`, + ); + const body = [...prose, strayChatLine, ...morePros].join('\n'); + const r = parseConversation(body); + // Pre-fix: regex_match with messages.length === 1. + // Post-fix: no_match because 1/301 < 0.05 acceptance floor. + expect(r.phase).toBe('no_match'); + expect(r.messages).toHaveLength(0); + }); });