diff --git a/CHANGELOG.md b/CHANGELOG.md index e98d11aff..f87e1a9d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,104 @@ All notable changes to GBrain will be documented in this file. +## [0.42.6.0] - 2026-06-01 + +**Most of your people and company pages are one-line stubs. `gbrain enrich --thin` +now develops them at scale using only what your brain already knows, no web +lookups, and every claim it writes is cited back to the note it came from.** + +Your brain is full of scattered knowledge about a person that never made it onto +their page: a meeting where they presented, a deal they led, another person's +page that mentions them, a fact you captured months ago. The stub page still +says "Stub page." `gbrain enrich --thin` finds the most-referenced stubs, pulls +together everything the brain knows about each one (search + backlinks + facts + +raw notes), and makes one grounded model call per page to consolidate it into a +real, cited dossier. When the brain doesn't know enough, it skips the page +instead of making things up. + +This is deliberately brain-internal. gbrain's own model tooling can only see your +brain (search, get_page, facts, backlinks), not the web or LinkedIn, so this +develops what you already have rather than researching new facts. Web research +stays the job of the agent-driven `enrich` skill. + +How to run it: + +```bash +# See what it would do + a cost estimate, no spend: +gbrain enrich --thin --dry-run --json + +# Develop the top 3 most-referenced stubs, cheap model, $0.50 cap: +gbrain enrich --thin --limit 3 --max-usd 0.50 --model anthropic:claude-haiku-4-5 +``` + +It's resumable (`--resume`), budget-capped (`--max-usd`, best-effort under +`--workers > 1`; pin `--workers 1` for a hard ceiling), and source-scoped +(`--source`). Enriched pages get `enriched_at` + `enriched_by` frontmatter so a +recency guard skips them on the next run, and the budget cap is the real money +gate. + +There's also an opt-in autopilot phase (`cycle.enrich_thin.enabled`, default OFF) +that trickles a few thin pages per source each cycle so the brain compounds over +time, with both per-source and brain-wide cost + walltime caps. + +What you'd see: a stub people page that read "Stub page." comes back as a +multi-section dossier with `[Source: meetings/2026-summit]` style citations on +each claim, and re-running confirms it's no longer selected. + +Things to know: untrusted note content can't break out of the model prompt (the +retrieved context is escaped, including the data-envelope delimiters), and +`enrich` is refused on thin-client / HTTP MCP installs because it spends model +budget and writes pages — run it on the host. + +## To take advantage of v0.42.6.0 + +`gbrain upgrade` brings the new command in. No migration is required for the core +feature (it reads existing pages + links + facts). To use it: + +1. **Preview first:** + ```bash + gbrain enrich --thin --dry-run --json + ``` +2. **Run a small, capped batch:** + ```bash + gbrain enrich --thin --limit 3 --max-usd 0.50 --model anthropic:claude-haiku-4-5 + ``` +3. **(Optional) turn on the autopilot trickle:** + ```bash + gbrain config set cycle.enrich_thin.enabled true + gbrain dream --phase enrich_thin --dry-run + ``` +4. **If anything looks wrong,** file an issue with `gbrain doctor` output: + https://github.com/garrytan/gbrain/issues + +### Itemized changes + +- **`gbrain enrich --thin`** — batch develops thin (stub) pages via brain-internal + grounded synthesis. Flags: `--order inbound-links|salience|updated`, + `--types person,company`, `--limit`, `--workers`, `--model`, `--max-usd`, + `--min-context`, `--reenrich-after`, `--source`, `--dry-run`, `--resume`, + `--background [--follow]`, `--json`, `--yes`. +- **New engine method `listEnrichCandidates`** (Postgres + PGLite parity) — one + source-aware SQL query: thin-filter + per-page source-correct inbound-link + count + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, returning a + lightweight projection (no page bodies) so ranking 100K stubs stays cheap. +- **Opt-in `enrich_thin` autopilot phase** (default OFF) with per-source and + brain-wide cost + walltime caps; develops `max_pages_per_tick` (default 3) per + source. +- **`--background`** fans out one Minion job per source (or one job with + `--source`); the per-source idempotency key carries the full run config so a + re-run with different flags enqueues new work instead of returning the old job. +- **Provenance:** enriched pages stamp `enriched_at` + `enriched_by: cli:enrich` + (survives `put_page` write-through); the recency guard reads `enriched_at`. +- **Prompt-injection hardening:** retrieved brain content is sanitized before it + enters the prompt, including neutralizing the `` data-envelope + delimiters so an untrusted note can't close the envelope and inject + instructions. +- **Budget honesty:** a final-call cost overage is now flagged on the result even + when the gateway swallowed the throw; the checkpoint is flushed on budget + exhaustion so a resume doesn't re-charge already-completed pages. +- Refused on thin-client / HTTP MCP installs (spends model budget + writes pages). + ## [0.42.2.0] - 2026-05-30 **One command now wires Claude Code, Codex, or Perplexity Computer to a remote diff --git a/CLAUDE.md b/CLAUDE.md index 412edb2b4..d08fd38de 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -183,6 +183,7 @@ strict behavior when unset. - `src/core/fail-improve.ts` — Deterministic-first, LLM-fallback loop with JSONL failure logging and auto-test generation - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling +- `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts` (v0.42.6.0, #1700) — `gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's own model tooling can only see brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich::')` → `getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1` per D5 — `runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). **codex-review fixes folded in:** `sanitizeContext` (thin.ts) neutralizes the `…` data-envelope delimiters (P1 injection escape, mirrors the `` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (P1, was a bare `enrich:${sid}` that returned stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (P1, via new read-only `BudgetTracker.cap` getter — no `gateway.ts` change); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge (P2). The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` (P2 fix) + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity). Plan: `~/.claude/plans/pick-up-this-context-composed-neumann.md`. - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. **v0.41.31.0:** every embed-write path now stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. - `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. diff --git a/VERSION b/VERSION index 4aed115c0..711ed3454 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.2.0 +0.42.6.0 diff --git a/llms-full.txt b/llms-full.txt index 666a51061..5bd94eae3 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -325,6 +325,7 @@ strict behavior when unset. - `src/core/fail-improve.ts` — Deterministic-first, LLM-fallback loop with JSONL failure logging and auto-test generation - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling +- `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts` (v0.42.6.0, #1700) — `gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's own model tooling can only see brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich::')` → `getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1` per D5 — `runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). **codex-review fixes folded in:** `sanitizeContext` (thin.ts) neutralizes the `…` data-envelope delimiters (P1 injection escape, mirrors the `` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (P1, was a bare `enrich:${sid}` that returned stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (P1, via new read-only `BudgetTracker.cap` getter — no `gateway.ts` change); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge (P2). The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` (P2 fix) + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity). Plan: `~/.claude/plans/pick-up-this-context-composed-neumann.md`. - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. **v0.41.31.0:** every embed-write path now stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. - `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. diff --git a/package.json b/package.json index b4cf523bb..2f9d2b8f6 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.2.0" + "version": "0.42.6.0" }