From 041d89babe7a81bcef163fd7cfaf6b65031c9668 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 29 May 2026 07:06:08 -0700 Subject: [PATCH] v0.41.29.0 feat(conversation-parser): bold-name-no-time builtin + fix(orphans): source-scoped orphan_ratio (supersedes #1613) (#1620) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(conversation-parser): add bold-name-no-time builtin (Circleback/Granola/Zoom, no timestamp) The 14th built-in pattern parses `**Speaker:** text` transcripts with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Every prior builtin required a time anchor, so this shape matched nothing: a production brain had 104 conversation pages + 3,423 eligible pages silently extracting zero facts. Messages anchor at T00:00:00Z of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same convention as irc-classic. Hardening beyond the original community proposal: - regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`: the colon-inside-bold (NOT declaration order) is what prevents shadowing bold-paren-time; the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest no_match instead of speaker="[18:37] Name". - new optional PatternEntry.score_full_body: `**Label:** text` is a common prose idiom, so a notes page with bold labels clustered in its first 10 lines scored 0.3 on the head pass (NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so the full-body fallback never fired) and cleared the 0.05 floor. parse.ts now recomputes the winner's score over the full body before the floor, so such a page drops to its true low density and stays no_match. - scrubbed pre-existing real names from bold-paren-time test_positive samples (privacy rule). Fixtures use placeholder names only. Pinned by new bold-name-no-time + clustered-head no_match cases in parse.test.ts and the eval corpus. Co-Authored-By: garrytan-agents Co-Authored-By: Claude Opus 4.8 (1M context) * fix(orphans): scope orphan_ratio + find_orphans by source; fix total_linkable denominator `gbrain doctor --source ` and `gbrain orphans --source ` now scope the orphan scan to that source instead of reporting brain-wide. Three fixes: - findOrphanPages(opts?: { sourceId?, sourceIds? }) on both engines scopes the CANDIDATE set (scalar `= $1` or federated `= ANY($1::text[])`). Inbound links from ANY source still count, so a page in source X linked FROM source Y is reachable and NOT an orphan of X (the deliberate, less-surprising definition). - corrected the total_linkable denominator in findOrphans: it now enumerates all live pages (scoped) and subtracts every excluded-by-slug page, not just excluded orphans. The old `total - excludedOrphans` left excluded NON-orphan pages (templates/, scratch/) with inbound links in the denominator, inflating it and suppressing warnings. Changes orphan_ratio output for every brain, in the accurate direction. - the find_orphans MCP op threads sourceScopeOpts(ctx), closing a cross-source read leak where a source-bound OAuth client saw brain-wide orphans (v0.34.1 source-isolation class). doctor uses an explicit `--source` flag parse (NOT resolveSourceWithTier, which would scope bare invocations to a default), and under explicit --source reports the ratio with a low-scale caveat below 100 entity pages instead of a vacuous "ok". Thin-client doctor --source orphan_ratio deferred (TODOS.md). Pinned by test/orphans-source-scope.test.ts (PGLite: scoping, cross-source inbound, denominator, find_orphans op scope) + a Postgres↔PGLite parity case in test/e2e/engine-parity.test.ts (scalar + federated binding). Co-Authored-By: Claude Opus 4.8 (1M context) * docs: v0.41.29.0 — bold-name-no-time + orphan source scoping VERSION + package.json → 0.41.29.0; CHANGELOG entry; CLAUDE.md conversation-parser (13→14 patterns) + orphans source-scoping notes; regenerated llms bundles; TODOS for thin-client doctor --source + check-test-real-names widening. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: garrytan-agents Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 131 +++++++++++ CLAUDE.md | 4 +- TODOS.md | 10 + VERSION | 2 +- llms-full.txt | 4 +- package.json | 2 +- src/commands/doctor.ts | 42 +++- src/commands/orphans.ts | 61 ++++- src/core/conversation-parser/builtins.ts | 77 ++++++- src/core/conversation-parser/parse.ts | 17 ++ src/core/conversation-parser/types.ts | 13 ++ src/core/engine.ts | 15 +- src/core/operations.ts | 11 +- src/core/pglite-engine.ts | 24 +- src/core/postgres-engine.ts | 18 +- test/conversation-parser/parse.test.ts | 115 +++++++++- test/e2e/engine-parity.test.ts | 49 ++++ .../conversation-formats/adversarial.jsonl | 1 + test/fixtures/conversation-formats/all.jsonl | 3 + .../bold-name-no-time.jsonl | 2 + test/orphans-source-scope.test.ts | 213 ++++++++++++++++++ 21 files changed, 780 insertions(+), 34 deletions(-) create mode 100644 test/fixtures/conversation-formats/bold-name-no-time.jsonl create mode 100644 test/orphans-source-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c3d12c4c..7fe07fe09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,137 @@ All notable changes to GBrain will be documented in this file. +## [0.41.29.0] - 2026-05-29 + +**Your meeting transcripts that look like `**Garry Tan:** ...` now actually +parse — the kind Circleback, Granola, and Zoom export with no timestamp on +each line. And `gbrain doctor --source ` finally scopes its orphan check +to that one source instead of your whole brain.** + +Two fixes in one release. + +**The transcript fix.** gbrain knows 13 chat formats, and every single one +needed a per-line timestamp to recognize a speaker. Modern meeting tools +don't put a time on every line — they emit plain `**Speaker Name:** what +they said`. With no time anchor, gbrain matched nothing: a real production +brain had 104 conversation pages plus 3,423 other eligible pages that +silently parsed to zero messages and extracted zero facts. The 14th built-in +pattern, `bold-name-no-time`, handles that shape. Each message anchors at +00:00:00 of the page's date (no fake wall-clock times are invented; line +order preserves the sequence). Nothing you do — it just works the next time +your cycle runs. + +How to check it's recognized: + +``` +gbrain conversation-parser list-builtins # bold-name-no-time is now listed +gbrain conversation-parser scan # see how one page parses +``` + +One thing we were careful about: that `**Label:** text` shape is also a +common way to write notes (`**Owner:** Alice`, `**Action:** ship it`). A +notes page is NOT a conversation, and gbrain won't treat it as one — even +when a few bold labels sit at the very top of the page. We score the whole +document's density, not just the first few lines, so a mostly-prose page +with scattered bold labels stays unparsed instead of turning into fake +"messages" that pollute your facts. + +**The orphan-check fix.** "Orphan ratio" is the fraction of your pages that +nothing links to — a health signal for `gbrain doctor`. On a brain with +multiple sources, `gbrain doctor --source dept-x` ignored the flag and +reported a brain-wide number. Now it scopes to the source you asked about, +and so does `gbrain orphans --source dept-x`: + +``` +gbrain doctor --source dept-x # orphan ratio for dept-x only +gbrain orphans --source dept-x # the orphan list for dept-x only +``` + +A page in dept-x that's linked from another source counts as reachable (not +an orphan) — the cross-source link still saves it. + +While we were in there we fixed two more things you'll feel: + +- The orphan ratio was reading lower than reality because junk pages + (templates, scratch, archives) that happen to be linked were padding the + denominator. The ratio is now honest, which means warnings that were + staying quiet will start showing up. This changes the number on every + brain, in the right direction. +- An agent connected over MCP and scoped to one source used to get the + orphan list for your *whole* brain. Now `find_orphans` respects the + agent's source scope, closing a cross-source read leak. + +### To take advantage of v0.41.29.0 + +`gbrain upgrade` handles this — there's no schema migration and no manual +step. After upgrading: + +1. **Conversation parsing** is automatic. Your next `gbrain dream` / + `gbrain extract conversation-facts` run will pick up `**Speaker:** text` + pages. To preview a specific page: + ```bash + gbrain conversation-parser scan + ``` +2. **Per-source orphan checks** work immediately: + ```bash + gbrain doctor --source + gbrain orphans --source + ``` +3. **If `gbrain doctor` shows a higher orphan ratio than before,** that's the + denominator fix surfacing real orphans that were previously hidden. Run + `gbrain extract links --by-mention` to auto-link entity mentions, then + `gbrain orphans` for the list. +4. **If anything looks wrong,** file an issue at + https://github.com/garrytan/gbrain/issues with `gbrain doctor` output. + +### Itemized changes + +**Conversation parser** +- New 14th built-in pattern `bold-name-no-time` in + `src/core/conversation-parser/builtins.ts` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`): + parses `**Speaker:** text` with no per-line timestamp (Circleback / Granola + / Zoom), anchored at `T00:00:00Z` of the frontmatter date. Declared after + the timestamped bold patterns; the colon-inside-bold regex (not declaration + order) is what prevents it from shadowing `bold-paren-time`. The `(?!\[)` + lookahead keeps it from mis-capturing telegram-bracket `**[18:37] Name:**` + lines as `speaker="[18:37] Name"`. +- New optional `PatternEntry.score_full_body` field + a full-body acceptance + recompute in `parse.ts`: broad patterns are scored on whole-document + density before the acceptance floor, so a prose notes page with bold labels + clustered in its first 10 lines no longer mis-parses as a conversation. +- Fixture coverage: `test/fixtures/conversation-formats/bold-name-no-time.jsonl` + + entries in `all.jsonl` and a clustered-head adversarial fixture in + `adversarial.jsonl` (all placeholder names). + +**Orphan source scoping** +- `BrainEngine.findOrphanPages(opts?: { sourceId?, sourceIds? })` (both + engines): scopes the candidate set to one source (scalar) or a federated + set (`= ANY(...)`). Inbound links from any source still count, so a + cross-source-linked page is correctly not an orphan. +- `gbrain orphans --source ` and `gbrain doctor --source ` (orphan_ratio + check) honor an explicit source. Bare invocations stay brain-wide. +- Corrected the `total_linkable` denominator in `findOrphans` so excluded + pages (templates/, scratch/, etc.) that have inbound links no longer + inflate it and suppress warnings. Changes orphan_ratio output for every + brain (in the accurate direction). +- The `find_orphans` MCP op now scopes by the caller's source + (`sourceScopeOpts(ctx)`), closing a cross-source read leak for source-bound + OAuth clients. +- Under explicit `--source`, `orphan_ratio` reports the ratio with a + low-scale caveat below 100 entity pages instead of returning a vacuous + "ok" that could hide a fully-orphaned small source. + +**Tests** +- `test/orphans-source-scope.test.ts` (PGLite): scoping, cross-source-inbound, + the denominator fix, and the `find_orphans` op-handler source scope. +- `test/e2e/engine-parity.test.ts`: Postgres↔PGLite parity for + `findOrphanPages` scalar + federated scoping. +- New `bold-name-no-time` + clustered-head regression cases in + `test/conversation-parser/parse.test.ts`. + +Incorporates the original `bold-name-no-time` pattern proposed in a community +PR; names scrubbed to placeholders per the project privacy rule. + ## [0.41.28.0] - 2026-05-27 **Your `gbrain dream` cycle stops losing rows when the database connection diff --git a/CLAUDE.md b/CLAUDE.md index 05df903ec..4b26413c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -173,7 +173,7 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. -- `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0) — 13-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. @@ -250,7 +250,7 @@ strict behavior when unset. - `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. - `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry ` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. -- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). +- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo] [--source ]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). **v0.41.29.0 source-scoping wave:** `findOrphans`/`getOrphansData` (the canonical pure fn shared with doctor's `orphan_ratio`) takes `{ sourceId?, sourceIds? }`; `BrainEngine.findOrphanPages(opts?)` (both engines) scopes ONLY the candidate side (`p.source_id = $1` scalar, or `= ANY($1::text[])` federated) while still counting inbound links from ANY source — a page in source X linked FROM source Y is reachable, so NOT an orphan of X (the deliberate, less-surprising definition; the stricter intra-source-only reading is rejected). `--source` is an explicit raw-flag parse (NOT `resolveSourceWithTier`, which would scope bare invocations to a default). Also corrected the `total_linkable` denominator: it now enumerates ALL live pages (scoped) and subtracts every excluded-by-slug page (templates/, scratch/, etc.) — pre-fix `total - excludedOrphans` left excluded NON-orphan pages with inbound links in the denominator, inflating it and suppressing warnings (changes orphan_ratio output for every brain, in the accurate direction). The `find_orphans` MCP op threads `sourceScopeOpts(ctx)` so a source-bound OAuth client no longer sees brain-wide orphans (closes a cross-source read leak in the v0.34.1 class). `gbrain doctor --source ` scopes `orphan_ratio` (entity-count gate + getOrphansData + messages) and, under explicit `--source` below 100 entity pages, reports the ratio with a low-scale caveat instead of a vacuous "ok". Thin-client `doctor --source` orphan_ratio remains brain-wide (TODO). Pinned by `test/orphans-source-scope.test.ts` (PGLite) + `test/e2e/engine-parity.test.ts` (Postgres↔PGLite scalar + federated parity). - `src/commands/salience.ts` (v0.29) — `gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]`: pages ranked by emotional + activity salience over a recency window. Mirrors orphans.ts shape (pure data fn + JSON formatter + human formatter). Calls `engine.getRecentSalience(opts)`. Score formula: `(emotional_weight × 5) + ln(1 + active_take_count) + 1/(1 + days_since_update)`. - `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. diff --git a/TODOS.md b/TODOS.md index 1b8a6311e..e8ce40ba6 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,15 @@ # TODOS +## v0.41.29.0 orphan source-scoping follow-ups (v0.42+) + +Filed from the v0.41.29.0 wave (bold-name-no-time pattern + orphan_ratio +source scoping). The Codex outside-voice review (F8) flagged two surfaces +the wave deliberately scoped out. + +- [ ] **v0.42+: thin-client `gbrain doctor --source` orphan_ratio scoping.** v0.41.29.0 scopes `orphan_ratio` to `--source` on the LOCAL doctor path (`buildChecks` in `src/commands/doctor.ts`) and closes the `find_orphans` MCP read leak via `sourceScopeOpts(ctx)`. The thin-client / remote doctor path (`src/core/doctor-remote.ts` `runRemoteDoctor`) is a separate code path that does not thread `--source`, so `gbrain doctor --source x` against a remote `gbrain serve --http` brain still reports brain-wide orphan_ratio. Thread the explicit `--source` into the remote doctor request + have the server-side check honor it. Priority: P3 (most users run doctor locally). + +- [ ] **v0.42+: widen `check-test-real-names.sh` BANNED_NAMES to catch real-name reintroduction in tests + src.** v0.41.29.0 scrubbed pre-existing real names (`Garry Tan`, `Alex Graveley`) from `bold-paren-time`'s `test_positive` (and the new `bold-name-no-time` samples), but no automated guard caught them: `check-test-real-names.sh` only scans `test/**` and its BANNED_NAMES list doesn't include `garry tan`; `check-fixture-privacy.sh` only scans `test/fixtures/conversation-formats/`. Add `garry tan` / `garrytan` (and consider extending the scan to `src/core/conversation-parser/builtins.ts` test samples) so future reintroductions fail CI. Priority: P3 (hardening). + ## v0.41.28.0 #1570 instrument-then-fix follow-ups (v0.41.28+ / v0.42+) Filed from the v0.41.28.0 plan-eng-review after the codex outside-voice diff --git a/VERSION b/VERSION index d59fd11cb..5625f3eba 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.28.0 \ No newline at end of file +0.41.29.0 diff --git a/llms-full.txt b/llms-full.txt index 120208615..0752d9c1d 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -315,7 +315,7 @@ strict behavior when unset. - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most. **v0.40.3.0:** all 16 `console.log` / `console.error` call sites migrated to `slog` / `serr` from `src/core/console-prefix.ts` so that when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (which the `gbrain sync --all` worker pool installs), every line carries the `[] ` prefix. Standalone `gbrain embed --stale` callers see identical output to v0.40.2 because slog/serr fall through to bare console fns outside the wrap. -- `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0) — 13-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/conversation-parser/` (v0.41.16.0, v0.41.24.0, v0.41.29.0) — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Replaces the single inline regex that PR #1461 was trying to extend with one more shape. **v0.41.29.0** added the 14th pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, declared at index 3 after `bold-paren-time`): parses `**Speaker:** text` with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Anchors every message at `T00:00:00Z` of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same no-time convention as `irc-classic`. The non-shadow guarantee is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest `no_match` instead of `speaker="[18:37] Name"`. Because `**Label:** text` is a common prose idiom, the pattern sets the new optional `PatternEntry.score_full_body: true`: `parse.ts` recomputes the winner's acceptance score over the FULL body (not just the 10-line head window) before the `SCORING_MIN_ACCEPTANCE` floor, so a notes page with bold labels clustered in its first 10 lines (head score 0.3, which is NOT `< SCORING_HEAD_TRIGGER_THRESHOLD` so the head fallback never fires) drops to its true low density and stays `no_match` instead of mis-parsing as a conversation. The same wave scrubbed the pre-existing real names in `bold-paren-time`'s `test_positive`. Pinned by the `bold-name-no-time` + clustered-head `no_match` cases in `test/conversation-parser/parse.test.ts` + `bold-name-no-time.jsonl` / `adversarial.jsonl` fixtures. **v0.41.24.0 (closes #1533)** added the 13th pattern `bold-paren-time` (`**Speaker** (HH:MM): text` and `**Speaker** (HH:MM:SS): text` — the time-only shape an OpenClaw meeting-ingestion pipeline produces from Circleback transcripts; date_source: frontmatter) AND tightened the parser's fallback gates in `parse.ts`: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that (closes the bug class where a stray head match at 0.1 — e.g. blockquote that accidentally matches an unrelated pattern — suppressed the fallback entirely); `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives (a 300-line essay with one stray chat-shape line scores ~0.003, below the floor, stays `no_match` instead of returning a 1-message regex_match). Both gates pinned by 11 new unit cases in `test/conversation-parser/parse.test.ts`. New exported `scorePatternFull(body, entry)` for direct full-body scoring; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` helpers DRY the quick_reject+regex loop. Empirical against all Circleback meetings in `~/git/brain/meetings/`: 113 of 367 parse correctly post-fix (was 0); 20,167 messages flow through to the fact extractor (was 0). The other 254 are notes-only meetings whose transcripts live in separate files. Plan + Codex absorption at `~/.claude/plans/system-instruction-you-are-working-starry-frost.md`. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (13 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup, so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as the PR #1461 helper promoted to a module-level default), `parse.ts` (orchestrator with D18 pattern-priority scoring across the first 10 lines + D8 date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + D5 multi-line continuation + D19 timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; both polish and fallback are thin wrappers per D6 DRY), `llm-polish.ts` (per D15 opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (per D15 opt-IN; per D17 NO regex inference + NO persistence — silent corruption avoided), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default per D10 — tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2 — wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` for debug, `list-builtins` for operator discoverability, `validate ` emits "deferred to v0.42+" notice). Doctor checks: `conversation_format_coverage` (per-pattern hit count + unmatched %), `progressive_batch_audit_health` (abort-verdict counts from last 7d), `conversation_parser_probe_health` (opt-in nightly probe state). Closes PR #1461 — contributor's `BRACKET_TIME_RX` + `cleanSpeaker` survive verbatim as the `telegram-bracket` built-in + `DEFAULT_SPEAKER_CLEAN` export, Co-Authored-By preserved. Plan + 23 decisions + codex outside-voice absorption at `~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md`. Pinned by 74 unit cases across `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case PR #1461 baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` (v0.41.16.0) — shared ramp-up + cost-cap + verification primitive extracted after rule-of-three was comfortably past (12+ ad-hoc cost-prompt patterns scattered across reindex, reindex-multimodal, book-mirror, brainstorm, eval-suspected-contradictions, post-upgrade-reembed, eval-contradictions/cost-prompt). Wintermute-inspired ramp shape (trial 10 → ramp 100 → ramp 500 → full with verification at each stage), productionized as a runtime primitive with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union per D20: `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` per D3 fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII stage-report formatter for the default `Policy.onStageReport` handler). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1` (skip ramp + stderr warn), `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500` (override). v0.41.16.0 ships with ONE proven consumer (the parser cathedral via extract-conversation-facts retrofit); the 9-site retrofit of `reindex.ts`, `reindex-multimodal.ts`, `reindex-code.ts`, `post-upgrade-reembed.ts`, `book-mirror.ts`, `brainstorm/orchestrator.ts`, `eval-suspected-contradictions.ts`, `eval-contradictions/cost-prompt.ts`, `contextual-reindex-per-chunk.ts` lands as the v0.41.14.0+ follow-up wave per D2's bisectability requirement (each retrofit gets its own commit; behavior parity tested before/after migration). Per D21: sites that previously "jumped straight to full" keep doing so by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by 35 unit cases in `test/progressive-batch/orchestrator.test.ts` covering every verdict path. - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` (v0.41.11.0) — bulk fact extraction for long-form conversation pages. The recall-miss class on long iMessage/Slack imports (60K+ msg history; a chunk that reads only "Locker 93 code 9494" has no topical anchor because "cabin" was established 50K messages earlier) gets fixed by walking conversation/meeting/slack/email pages, splitting into time-windowed segments (30-min gap or 30-msg cap), prepending a topical/temporal header ("Conversation between A and B from to "), and running through the existing `extractFactsFromTurn()` so the resulting anchor-rich facts surface in `gbrain search` (which already blends facts). Architecture decisions absorbed from 2 Codex outside-voice rounds + 2 eng review passes: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only per cycle.ts:131); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})` so worst case is 10×25MB=250MB resident vs PR #1406's unbounded; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide on segment 2); **page-level TERMINAL audit row** written to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS query matches the terminal row, NOT any-fact-for-slug, so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, core uses it as-is — nested `withBudgetTracker` REPLACES per gateway.ts:2011; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline** (iMessage importers put history in timeline half; PR's `compiled_truth ?? ''` would silently drop it); **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types` config as single source of truth; **fingerprint on sourceId only** so widening types config doesn't invalidate completed-page state; **string-encoded op-checkpoint** entries `"||"` for resume only (op_checkpoints stores string[] and is 7-day GC'd — durable audit is the facts table itself via terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job, catches + persists + marks `completed` with `result.budget_exhausted=true` — NOT a failure). The companion cycle phase `conversation_facts_backfill` (default OFF; `gbrain config set cycle.conversation_facts_backfill.enabled true` opts in) iterates `listSources(engine)` directly, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source invocation via `opts.budgetTracker` so the core doesn't nest-replace. Two-layer cost AND walltime protection: per-source caps (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide caps (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by 27 unit cases in `test/extract-conversation-facts.test.ts` (parse/segment/render/checkpoint/fingerprint/terminal-audit/row_num/F2-kill-switch). Schema migration v94 in `src/core/migrate.ts` adds a partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (v14 precedent: `transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite) so the doctor backlog query stays fast on million-fact brains. `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when `cycle.conversation_facts_backfill.enabled=false` — no opt-out noise debt per Eng-v2 C9; OK when caught up; WARN when >10 pages lack the terminal audit row, with paste-ready remediation step composable via `gbrain doctor --remediate`). `src/commands/sources.ts:runAudit` extended with `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}` so `gbrain sources audit ` previews cost before any run. Schema-pack (v0.41.11.0): `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable — atoms ARE the extracted form) into the base seed; `concept.extractable` flips true semantically (cosmetic on backstop path because backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51`, not pack extractable — the original concept-grandfather migration was solving a phantom, dropped per Codex T3). `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. Plan + all decisions persisted at `~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md` and the CEO plan at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id ` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract. @@ -392,7 +392,7 @@ strict behavior when unset. - `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. - `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry ` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. -- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). +- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo] [--source ]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). **v0.41.29.0 source-scoping wave:** `findOrphans`/`getOrphansData` (the canonical pure fn shared with doctor's `orphan_ratio`) takes `{ sourceId?, sourceIds? }`; `BrainEngine.findOrphanPages(opts?)` (both engines) scopes ONLY the candidate side (`p.source_id = $1` scalar, or `= ANY($1::text[])` federated) while still counting inbound links from ANY source — a page in source X linked FROM source Y is reachable, so NOT an orphan of X (the deliberate, less-surprising definition; the stricter intra-source-only reading is rejected). `--source` is an explicit raw-flag parse (NOT `resolveSourceWithTier`, which would scope bare invocations to a default). Also corrected the `total_linkable` denominator: it now enumerates ALL live pages (scoped) and subtracts every excluded-by-slug page (templates/, scratch/, etc.) — pre-fix `total - excludedOrphans` left excluded NON-orphan pages with inbound links in the denominator, inflating it and suppressing warnings (changes orphan_ratio output for every brain, in the accurate direction). The `find_orphans` MCP op threads `sourceScopeOpts(ctx)` so a source-bound OAuth client no longer sees brain-wide orphans (closes a cross-source read leak in the v0.34.1 class). `gbrain doctor --source ` scopes `orphan_ratio` (entity-count gate + getOrphansData + messages) and, under explicit `--source` below 100 entity pages, reports the ratio with a low-scale caveat instead of a vacuous "ok". Thin-client `doctor --source` orphan_ratio remains brain-wide (TODO). Pinned by `test/orphans-source-scope.test.ts` (PGLite) + `test/e2e/engine-parity.test.ts` (Postgres↔PGLite scalar + federated parity). - `src/commands/salience.ts` (v0.29) — `gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]`: pages ranked by emotional + activity salience over a recency window. Mirrors orphans.ts shape (pure data fn + JSON formatter + human formatter). Calls `engine.getRecentSalience(opts)`. Score formula: `(emotional_weight × 5) + ln(1 + active_take_count) + 1/(1 + days_since_update)`. - `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. diff --git a/package.json b/package.json index 36ba2bf88..63e002a35 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.28.0" + "version": "0.41.29.0" } diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 7c6117848..ec1085254 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -3080,6 +3080,19 @@ export async function buildChecks( // invoked later in the function. const scope: 'all' | 'brain' = args.includes('--scope=brain') ? 'brain' : 'all'; + // v0.41.29.0: explicit `--source ` scopes the `orphan_ratio` check to one + // source. EXPLICIT-ONLY by design — a raw flag parse, NOT resolveSourceWithTier. + // The tier resolver would pick a default source when `--source` is absent and + // silently scope a bare `gbrain doctor` to one source; we want bare doctor to + // stay brain-wide. Only `orphan_ratio` consumes this for now (other checks + // staying brain-wide is a separate, larger change — see TODOS.md). + let orphanRatioSourceId: string | undefined; + for (let i = 0; i < args.length; i++) { + if (args[i] === '--source' && i + 1 < args.length) { + orphanRatioSourceId = args[++i] || undefined; + } + } + const checks: Check[] = []; let autoFixReport: AutoFixReport | null = null; @@ -4543,22 +4556,39 @@ export async function buildChecks( // show high orphan ratio; not actionable signal). // Warn at >0.5; fail at >0.8. Both states recommend // `gbrain extract links --by-mention` as the fix. + // v0.41.29.0: explicit `--source ` scopes this check to one source + // (orphanRatioSourceId, parsed at the top of buildChecks). The entity-count + // gate + getOrphansData both scope to it; messages name the source. Bare + // doctor (no --source) stays brain-wide. progress.heartbeat('orphan_ratio'); try { const { getOrphansData } = await import('./orphans.ts'); + const srcId = orphanRatioSourceId; + const inSource = srcId ? ` in source '${srcId}'` : ''; const entityCount = (await engine.executeRaw<{ count: number }>( - "SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization') AND deleted_at IS NULL", + `SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization') AND deleted_at IS NULL${srcId ? ' AND source_id = $1' : ''}`, + srcId ? [srcId] : [], ))[0]?.count ?? 0; - if (entityCount < 100) { + // Brain-wide (no --source): <100 entities is vacuous — small brains + // naturally show a high orphan ratio; not actionable signal. Skip. + if (entityCount < 100 && !srcId) { checks.push({ name: 'orphan_ratio', status: 'ok', message: `Vacuous: ${entityCount} entity pages (<100). Orphan ratio not meaningful at this scale.`, }); } else { - const data = await getOrphansData(engine, { includePseudo: false }); + // F7 (Codex): under EXPLICIT --source, an operator deliberately asked + // about one source — answer it even below 100 entities, with a + // low-scale caveat, instead of swallowing a real per-source failure + // (e.g. 80 fully-orphaned entity pages) behind a vacuous "ok". + const data = await getOrphansData(engine, { includePseudo: false, sourceId: srcId }); const ratio = data.total_linkable > 0 ? data.total_orphans / data.total_linkable : 0; const pct = (ratio * 100).toFixed(0); + const caveat = + entityCount < 100 + ? ` — low scale (${entityCount} entity pages <100), interpret with caution` + : ''; const hint = 'Run: gbrain extract links --by-mention (auto-links entity mentions in body text). ' + 'Run gbrain orphans for the list.'; @@ -4566,19 +4596,19 @@ export async function buildChecks( checks.push({ name: 'orphan_ratio', status: 'fail', - message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`, + message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links)${caveat}. ${hint}`, }); } else if (ratio > 0.5) { checks.push({ name: 'orphan_ratio', status: 'warn', - message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`, + message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links)${caveat}. ${hint}`, }); } else { checks.push({ name: 'orphan_ratio', status: 'ok', - message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages)`, + message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages)${caveat}`, }); } } diff --git a/src/commands/orphans.ts b/src/commands/orphans.ts index cfd52633f..04765d05f 100644 --- a/src/commands/orphans.ts +++ b/src/commands/orphans.ts @@ -127,9 +127,16 @@ export async function queryOrphanPages( */ export async function findOrphans( engine: BrainEngine, - opts: { includePseudo?: boolean } = {}, + opts: { includePseudo?: boolean; sourceId?: string; sourceIds?: string[] } = {}, ): Promise { const includePseudo = !!opts.includePseudo; + // v0.41.29.0: `sourceId` (scalar, from `--source` + single-source MCP + // clients) or `sourceIds` (federated, from `allowedSources` MCP clients) + // scopes the candidate set. `sourceIds` wins when both set (mirrors + // sourceScopeOpts precedence). + const sourceId = opts.sourceId; + const sourceIds = + opts.sourceIds && opts.sourceIds.length > 0 ? opts.sourceIds : undefined; // The NOT EXISTS anti-join over pages × links can take seconds on 50K-page // brains. Heartbeat every second so agents see the scan is alive. Keyset // pagination was considered and rejected: without an index on @@ -140,16 +147,40 @@ export async function findOrphans( const stopHb = startHeartbeat(progress, 'scanning pages for missing inbound links…'); let allOrphans: { slug: string; title: string; domain: string | null }[]; let total: number; + let excludedAll: number; try { - allOrphans = await engine.findOrphanPages(); - // Count total pages in DB for the summary line - const stats = await engine.getStats(); - total = stats.page_count; + allOrphans = await engine.findOrphanPages( + sourceIds ? { sourceIds } : sourceId ? { sourceId } : undefined, + ); + // v0.41.29.0 (Codex F6): correct the `total_linkable` denominator. + // Enumerate ALL live pages (scoped) and count excluded-by-slug across + // the WHOLE set — not just among orphans. The old + // `total - excludedOrphans` left excluded NON-orphan pages (e.g. a + // `test/` page that HAS inbound links) in the denominator, inflating + // total_linkable and suppressing orphan warnings. `getAllSlugs` is NOT + // used here because it does not filter soft-deleted rows; `total` must + // match `findOrphanPages`'s `deleted_at IS NULL` candidate universe. + let scopeClause = ''; + const liveParams: unknown[] = []; + if (sourceIds) { + liveParams.push(sourceIds); + scopeClause = ` AND source_id = ANY($${liveParams.length}::text[])`; + } else if (sourceId) { + liveParams.push(sourceId); + scopeClause = ` AND source_id = $${liveParams.length}`; + } + const liveRows = await engine.executeRaw<{ slug: string }>( + `SELECT slug FROM pages WHERE deleted_at IS NULL${scopeClause}`, + liveParams, + ); + total = liveRows.length; + excludedAll = includePseudo + ? 0 + : liveRows.reduce((n, r) => n + (shouldExclude(r.slug) ? 1 : 0), 0); } finally { stopHb(); progress.finish(); } - const _totalPages = allOrphans.length; // pages with no inbound links (preserved for ref) const filtered = includePseudo ? allOrphans @@ -166,7 +197,10 @@ export async function findOrphans( return { orphans, total_orphans: orphans.length, - total_linkable: filtered.length + (total - allOrphans.length), + // v0.41.29.0 (Codex F6): denominator = live pages minus ALL excluded + // pages (orphan or not), so excluded pages with inbound links no longer + // inflate it. + total_linkable: total - excludedAll, total_pages: total, excluded, }; @@ -224,6 +258,16 @@ export async function runOrphans(engine: BrainEngine, args: string[]) { const json = args.includes('--json'); const count = args.includes('--count'); const includePseudo = args.includes('--include-pseudo'); + // v0.41.29.0: explicit `--source ` scopes the orphan scan to one + // source. Omitted → brain-wide (unchanged). Raw explicit-flag parse on + // purpose — NOT resolveSourceWithTier, which would pick a default source + // when the flag is absent and silently scope a bare `gbrain orphans`. + let sourceId: string | undefined; + for (let i = 0; i < args.length; i++) { + if (args[i] === '--source' && i + 1 < args.length) { + sourceId = args[++i] || undefined; + } + } if (args.includes('--help') || args.includes('-h')) { console.log(`Usage: gbrain orphans [options] @@ -234,6 +278,7 @@ Options: --json Output as JSON (for agent consumption) --count Output just the number of orphans --include-pseudo Include auto-generated and pseudo pages in results + --source Scope the scan to one brain source (default: brain-wide) --help, -h Show this help Output (default): grouped by domain, sorted alphabetically within each group @@ -242,7 +287,7 @@ Summary line: N orphans out of M linkable pages (K total; K-M excluded) return; } - const result = await findOrphans(engine, { includePseudo }); + const result = await findOrphans(engine, { includePseudo, sourceId }); if (count) { console.log(String(result.total_orphans)); diff --git a/src/core/conversation-parser/builtins.ts b/src/core/conversation-parser/builtins.ts index db441a7e5..b28ffba27 100644 --- a/src/core/conversation-parser/builtins.ts +++ b/src/core/conversation-parser/builtins.ts @@ -1,7 +1,7 @@ /** * v0.41.16.0 — Built-in conversation parser pattern registry. * - * Twelve hand-vetted patterns covering the chat-export formats this + * Fourteen hand-vetted patterns covering the chat-export formats this * codebase is most likely to encounter. Each pattern's regex was * derived from a public format reference (source_doc field) so future * maintainers can verify against the wild shape. @@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string { return stripped || raw.trim(); } -/** The 12 hand-vetted built-in patterns. */ +/** The 14 hand-vetted built-in patterns. */ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ // ------------------------------------------------------------------- // INLINE-DATE patterns (date in every line; less ambiguous; tried first). @@ -158,9 +158,9 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ multi_line: false, quick_reject: /^\*\*/, test_positive: [ - '**Garry Tan** (00:00): hello world', + '**Alice Example** (00:00): hello world', '**Participant 2** (02:22): response here', - '**Alex Graveley** (15:09): That’s exactly right.', + '**Bob Example** (15:09): That’s exactly right.', '**Participant 1** (00:00:00): hello world with seconds', '**Participant 2** (01:23:45): mid-meeting line', ], @@ -178,6 +178,75 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ 'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)', }, + { + // Modern meeting-transcription tools (Circleback, Granola, Zoom) + // emit `**Speaker Name:** message text` with NO per-line + // timestamp. Every other built-in requires a time anchor, so this + // shape scored ~0.002 (one stray line in a long page) and fell + // below SCORING_MIN_ACCEPTANCE — parsing to zero messages and + // extracting zero conversation-facts. This additive pattern fixes + // that: speaker is captured inside the bold markers (`**Name:**`), + // there is no time capture, and date_source='frontmatter' with + // hour_group undefined routes through parse.ts's no-time branch + // (same convention as irc-classic) — every message anchors at + // 00:00:00 of the page's frontmatter date. No wall-clock time is + // fabricated; intra-day ordering is preserved by line order. + // + // NON-SHADOW GUARANTEE (read carefully — the safety is in the + // REGEX, not the declaration order). parse.ts scores every + // candidate independently; declaration index is ONLY the + // tie-break. This pattern cannot steal `**Name** (time):` from + // bold-paren-time because its regex requires the colon INSIDE the + // bold markers (`**Name:**`), which the paren-time shape (colon + // OUTSIDE: `**Name** (time):`) never has. The `(?!\[)` lookahead + // additionally rejects telegram-bracket's `**[18:37] Name:**` + // shape so that disabling telegram-bracket yields an honest + // no_match instead of capturing speaker="[18:37] Name" at midnight. + // + // BROAD-REGEX GUARD (score_full_body): `**Label:** text` is a + // common prose idiom (`**Note:**`, `**Owner:**`). A notes page + // with a few bold labels clustered in its first 10 lines would + // score 0.3 on the head pass, skip the rescore, and clear the + // 0.05 floor. score_full_body forces full-body density scoring + // before acceptance so such a page falls to no_match. + id: 'bold-name-no-time', + origin: 'builtin', + // Matches: **Speaker Name:** message text (colon INSIDE bold, + // speaker must not start with `[` — see lookahead rationale above). + regex: /^\*\*(?!\[)(.+?):\*\*\s*(.*)$/, + captures: { + speaker_group: 1, + text_group: 2, + }, + date_source: 'frontmatter', + time_format: '24h', + timezone_policy: 'utc_assumed_with_warn', + multi_line: false, + quick_reject: /^\*\*/, + score_full_body: true, + test_positive: [ + '**Alice Example:** Okay, start on.', + '**Participant 2:** he tried to reset it remotely the other night.', + '**Bob Example:** That is exactly right.', + ], + test_negative: [ + // bold-paren-time shape (colon OUTSIDE bold) MUST fall through: + '**Alice** (00:00): text', + // imessage-slack shape MUST fall through: + '**Alice Example** (2024-03-15 9:00 AM): iMessage shape', + // Bold but no colon at all: + '**Alice** hello world', + // No bold markers: + 'Alice: plain no bold', + // telegram-bracket shape (timestamp INSIDE bold) MUST NOT match + // — the `(?!\[)` lookahead rejects it so disabling + // telegram-bracket yields no_match, not speaker="[18:37] Alice": + '**[18:37] \u{1f464} Alice:** hello', + ], + source_doc: + 'Circleback / Granola / Zoom meeting-transcript export shape: `**Speaker:** text` with no per-line timestamp', + }, + { id: 'telegram-text-export', origin: 'builtin', diff --git a/src/core/conversation-parser/parse.ts b/src/core/conversation-parser/parse.ts index 215cbeab8..2758571f6 100644 --- a/src/core/conversation-parser/parse.ts +++ b/src/core/conversation-parser/parse.ts @@ -479,6 +479,7 @@ export function parseConversation( // 10 head lines matched"; chat-only pages still score 1.0 and skip // the fallback. Re-score every candidate against the full body, // pre-splitting ONCE to avoid 12 redundant body splits. + let fullBodyScored = false; if (scored[0].score < SCORING_HEAD_TRIGGER_THRESHOLD) { const allLines = getNonBlankLines(body); scored = candidates.map((entry) => ({ @@ -487,6 +488,7 @@ export function parseConversation( priority: priorityOf(entry.id), })); sortScored(scored); + fullBodyScored = true; // NOTE: patterns_scored stays as scored.length (= candidate // count, typically 12) even when the fallback runs — the // diagnostic reports "candidates considered" not "scoring @@ -496,6 +498,21 @@ export function parseConversation( const top = scored[0]; const patternsScored = scored.length; + // v0.41.29.0 (Codex F1): broad no-time patterns (`bold-name-no-time`, + // regex matches any `**Label:** text`) can win the HEAD pass on a + // prose notes page whose first 10 lines happen to hold 3+ bold labels + // (head score 0.3, NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so the + // full-body fallback above never ran). 0.3 clears the 0.05 floor and + // the page mis-parses as a conversation. When the winner is flagged + // `score_full_body` and we have NOT already scored full-body, recompute + // its score over the whole document so the floor judges true density. + // We do NOT re-pick the winner: the head pick is already the best + // candidate; this only tightens its acceptance (a real transcript + // stays ~1.0; a 3/200-label notes page drops to ~0.015 → no_match). + if (top.entry.score_full_body && !fullBodyScored) { + top.score = scoreFromLines(getNonBlankLines(body), top.entry); + } + // Minimum acceptance floor (closes Codex P1 #2): an essay with // one stray `**Name** (date time):` line scores ~1/300 ≈ 0.003 — // below the 5% floor we stay no_match instead of returning a diff --git a/src/core/conversation-parser/types.ts b/src/core/conversation-parser/types.ts index eaa40c17d..f439f410a 100644 --- a/src/core/conversation-parser/types.ts +++ b/src/core/conversation-parser/types.ts @@ -170,6 +170,19 @@ export interface PatternEntry { * uses `DEFAULT_SPEAKER_CLEAN`. */ speaker_clean?: RegExp; + /** + * v0.41.29.0: when true, the orchestrator recomputes this pattern's + * acceptance score over the FULL body (not just the head window) + * before applying SCORING_MIN_ACCEPTANCE. Set on broad no-time + * patterns (`bold-name-no-time`) whose regex matches a common prose + * idiom (`**Label:** text`): without it, a notes page with a few bold + * labels CLUSTERED in its first 10 lines scores 0.3 on the head pass, + * skips the `< SCORING_HEAD_TRIGGER_THRESHOLD` rescore, and clears the + * 0.05 floor — mis-parsing prose as a conversation. Full-body scoring + * drops that page below the floor (3/200 ≈ 0.015 → no_match) while a + * real transcript stays ~1.0. + */ + score_full_body?: boolean; /** D7: module-load validation — known-positive sample lines. */ test_positive: string[]; /** D7: module-load validation — known-negative sample lines. */ diff --git a/src/core/engine.ts b/src/core/engine.ts index fa2f0d015..b9861d0d9 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -1168,12 +1168,23 @@ export interface BrainEngine { */ getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise>; /** - * Return every page with no inbound links (from any source). + * Return every page with no inbound links. * Domain comes from the frontmatter `domain` field (null if unset). * The caller filters pseudo-pages + derives display domain. * Used by `gbrain orphans` and `runCycle`'s orphan sweep phase. + * + * v0.41.29.0: scopes the CANDIDATE set to one source (`sourceId`, from + * `gbrain doctor/orphans --source` + single-source MCP clients) or a + * federated set (`sourceIds`, from `allowedSources` MCP clients). The + * inbound-link side is NOT scoped — a page in source X linked FROM + * source Y is genuinely reachable, so it is not an orphan of X. Omit + * `opts` for the brain-wide behavior (unchanged). When both are set, + * `sourceIds` wins (mirrors `sourceScopeOpts` precedence). */ - findOrphanPages(): Promise>; + findOrphanPages(opts?: { + sourceId?: string; + sourceIds?: string[]; + }): Promise>; // Tags /** diff --git a/src/core/operations.ts b/src/core/operations.ts index 2b23ffe00..d58fe1084 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -2756,7 +2756,16 @@ const find_orphans: Operation = { scope: 'read', handler: async (ctx, p) => { const { findOrphans } = await import('../commands/orphans.ts'); - return findOrphans(ctx.engine, { includePseudo: (p.include_pseudo as boolean) || false }); + // v0.41.29.0 (Codex F8): scope by the caller's source (ctx.sourceId / + // ctx.auth.allowedSources) via the canonical sourceScopeOpts ladder. + // Pre-fix, find_orphans returned brain-wide orphans regardless of a + // source-bound OAuth client's scope — a read leak in the v0.34.1 + // source-isolation class. Local CLI callers route through `gbrain + // orphans --source` instead (ctx.remote === false → empty scope here). + return findOrphans(ctx.engine, { + includePseudo: (p.include_pseudo as boolean) || false, + ...sourceScopeOpts(ctx), + }); }, cliHints: { name: 'orphans', hidden: true }, }; diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index d42a3c1e1..612d5de5f 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -2791,12 +2791,30 @@ export class PGLiteEngine implements BrainEngine { return out; } - async findOrphanPages(): Promise> { + async findOrphanPages(opts?: { + sourceId?: string; + sourceIds?: string[]; + }): Promise> { // Soft-delete filter on BOTH sides: // - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates // - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound // Without the link-source filter, a live page can hide from orphan results purely // because a soft-deleted page links to it. v0.26.5 invariant; codex C11. + // + // v0.41.29.0: scope ONLY the candidate side (`p.source_id`) when opts.sourceId + // is set. The inbound-link NOT EXISTS deliberately counts links from ANY source: + // a page in source X linked FROM source Y is reachable, so NOT an orphan of X. + // Do NOT add `src.source_id = p.source_id` here — that is the stricter + // intra-source-only definition we deliberately reject. + let sourceFilter = ''; + const params: unknown[] = []; + if (opts?.sourceIds && opts.sourceIds.length > 0) { + params.push(opts.sourceIds); + sourceFilter = `AND p.source_id = ANY($${params.length}::text[])`; + } else if (opts?.sourceId) { + params.push(opts.sourceId); + sourceFilter = `AND p.source_id = $${params.length}`; + } const { rows } = await this.db.query( `SELECT p.slug, @@ -2804,6 +2822,7 @@ export class PGLiteEngine implements BrainEngine { p.frontmatter->>'domain' AS domain FROM pages p WHERE p.deleted_at IS NULL + ${sourceFilter} AND NOT EXISTS ( SELECT 1 FROM links l @@ -2811,7 +2830,8 @@ export class PGLiteEngine implements BrainEngine { WHERE l.to_page_id = p.id AND src.deleted_at IS NULL ) - ORDER BY p.slug` + ORDER BY p.slug`, + params ); return rows as Array<{ slug: string; title: string; domain: string | null }>; } diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 89e0b6526..2fb90fd39 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -2857,13 +2857,28 @@ export class PostgresEngine implements BrainEngine { return out; } - async findOrphanPages(): Promise> { + async findOrphanPages(opts?: { + sourceId?: string; + sourceIds?: string[]; + }): Promise> { const sql = this.sql; // Soft-delete filter on BOTH sides: // - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates // - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound // Without the link-source filter, a live page can hide from orphan results purely // because a soft-deleted page links to it. v0.26.5 invariant; codex C11. + // + // v0.41.29.0: scope ONLY the candidate side (`p.source_id`) when opts.sourceId + // is set. The inbound-link NOT EXISTS deliberately counts links from ANY source: + // a page in source X linked FROM source Y is reachable, so NOT an orphan of X. + // Do NOT add `src.source_id = p.source_id` here — that would be the stricter + // intra-source-only definition we deliberately reject. + const sourceFilter = + opts?.sourceIds && opts.sourceIds.length > 0 + ? sql`AND p.source_id = ANY(${opts.sourceIds}::text[])` + : opts?.sourceId + ? sql`AND p.source_id = ${opts.sourceId}` + : sql``; const rows = await sql` SELECT p.slug, @@ -2871,6 +2886,7 @@ export class PostgresEngine implements BrainEngine { p.frontmatter->>'domain' AS domain FROM pages p WHERE p.deleted_at IS NULL + ${sourceFilter} AND NOT EXISTS ( SELECT 1 FROM links l diff --git a/test/conversation-parser/parse.test.ts b/test/conversation-parser/parse.test.ts index b8a1cce86..33cbf36fe 100644 --- a/test/conversation-parser/parse.test.ts +++ b/test/conversation-parser/parse.test.ts @@ -403,21 +403,21 @@ describe('scorePatternFull — full-body scoring (v0.41.18+ Codex P1 #1)', () => describe('bold-paren-time pattern (Circleback meeting transcripts)', () => { test('matches **Speaker** (HH:MM): text with frontmatter date', () => { const body = [ - '**Garry Tan** (00:00): Hey, can you hear me?', + '**Alice Example** (00:00): Hey, can you hear me?', '**Participant 2** (02:22): Yeah, just joined.', - '**Garry Tan** (15:09): That makes sense.', + '**Alice Example** (15:09): That makes sense.', ].join('\n'); const r = parseConversation(body, { fallbackDate: '2026-03-19' }); expect(r.phase).toBe('regex_match'); expect(r.matched_pattern_id).toBe('bold-paren-time'); expect(r.messages).toHaveLength(3); expect(r.messages[0]).toEqual({ - speaker: 'Garry Tan', + speaker: 'Alice Example', timestamp: '2026-03-19T00:00:00Z', text: 'Hey, can you hear me?', }); expect(r.messages[2]).toEqual({ - speaker: 'Garry Tan', + speaker: 'Alice Example', timestamp: '2026-03-19T15:09:00Z', text: 'That makes sense.', }); @@ -478,6 +478,113 @@ describe('bold-paren-time pattern (Circleback meeting transcripts)', () => { }); }); +// --------------------------------------------------------------------------- +// bold-name-no-time pattern (Circleback / Granola / Zoom transcripts with NO +// per-line timestamp — `**Speaker:** text`). Additive pattern; the colon +// inside the bold markers + the `(?!\[)` lookahead are what keep it from +// shadowing bold-paren-time / telegram-bracket (NOT declaration order). +// --------------------------------------------------------------------------- + +describe('bold-name-no-time pattern (Circleback/Granola/Zoom, no timestamp)', () => { + test('parses **Speaker:** text transcript with frontmatter date anchor', () => { + const body = [ + '**Alice Example:** Okay, start on. And then weirdly like zoom doesn’t...', + '**Participant 2:** he tried to reset it remotely the other night. Let me ask him.', + '**Alice Example:** I mean it’s really just like we need to get zoom to fix this.', + '**Participant 2:** Okay, let me.', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-05-28' }); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('bold-name-no-time'); + expect(r.messages).toHaveLength(4); + expect(r.messages[0]).toEqual({ + speaker: 'Alice Example', + timestamp: '2026-05-28T00:00:00Z', + text: 'Okay, start on. And then weirdly like zoom doesn’t...', + }); + expect(r.messages[1].speaker).toBe('Participant 2'); + expect(r.messages[1].text).toBe( + 'he tried to reset it remotely the other night. Let me ask him.', + ); + expect(r.messages[3]).toEqual({ + speaker: 'Participant 2', + timestamp: '2026-05-28T00:00:00Z', + text: 'Okay, let me.', + }); + // No-time pattern anchors at 00:00:00 of the frontmatter date + // (same convention as irc-classic). No wall-clock time fabricated. + }); + + test('scores above the 0.05 floor on a pure bold-name transcript (epoch default)', () => { + const body = [ + '**Alice Example:** line one', + '**Participant 2:** line two', + '**Alice Example:** line three', + ].join('\n'); + const r = parseConversation(body); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('bold-name-no-time'); + expect(r.messages).toHaveLength(3); + // No fallbackDate → epoch default, still anchors at 00:00:00. + expect(r.messages[0].timestamp).toBe('1970-01-01T00:00:00Z'); + }); + + // REGRESSION: must NOT shadow bold-paren-time. A `**Name** (00:00): text` + // line has its colon OUTSIDE the bold markers, so it must still parse via + // bold-paren-time (the safety is the regex, not declaration order). + test('REGRESSION: **Speaker** (HH:MM): text still matches bold-paren-time', () => { + const body = [ + '**Alice Example** (00:00): Hey, can you hear me?', + '**Participant 2** (02:22): Yeah, just joined.', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-03-19' }); + expect(r.phase).toBe('regex_match'); + expect(r.matched_pattern_id).toBe('bold-paren-time'); + expect(r.matched_pattern_id).not.toBe('bold-name-no-time'); + expect(r.messages).toHaveLength(2); + expect(r.messages[0].timestamp).toBe('2026-03-19T00:00:00Z'); + }); + + // REGRESSION: a pure telegram-bracket transcript has the colon inside the + // bold markers BUT the speaker starts with `[`, so the `(?!\[)` lookahead + // rejects it AND telegram-bracket (lower declaration index) wins anyway. + test('REGRESSION: telegram-bracket transcript still matches telegram-bracket', () => { + const body = [ + '**[18:37] \u{1f464} Alice:** one', + '**[18:38] \u{1f464} Bob:** two', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2024-03-15' }); + expect(r.matched_pattern_id).toBe('telegram-bracket'); + }); + + // F1 (Codex): the broad regex matches any `**Label:** text`. A prose notes + // page with bold labels CLUSTERED in its first 10 lines scores 0.3 on the + // head pass (3/10) — NOT < SCORING_HEAD_TRIGGER_THRESHOLD (0.3), so the + // full-body fallback never fires — and that 0.3 clears the 0.05 floor. + // Without score_full_body this page would mis-parse as a conversation. + // score_full_body recomputes the winner over the FULL body: 3 labels / + // 83 lines ≈ 0.036 < 0.05 → no_match. This is the exact bug Codex caught + // and the test the naive "0.05 floor protects us" assumption would fail. + test('F1: prose with bold labels clustered in the head returns no_match', () => { + const lines = [ + '**Attendees:** Alice Example, Bob Example, Participant 2', + '**Date:** 2026-05-28', + '**Goal:** decide on the Q3 roadmap and unblock the vendor migration', + ]; + // 80 plain prose lines (no bold-label shape) → 83 total, only 3 match. + // First 10 lines = 3 labels + 7 prose → head score 0.3 (skips rescore). + // Full body = 3/83 ≈ 0.036 < 0.05 floor. + for (let i = 0; i < 80; i++) { + lines.push( + `This is an ordinary prose sentence number ${i} describing the meeting in detail.`, + ); + } + const body = lines.join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-05-28' }); + expect(r.phase).toBe('no_match'); + }); +}); + // --------------------------------------------------------------------------- // parseConversation — full-body fallback (v0.41.18+ #1533 + Codex P1 #1, #2, #8) // --------------------------------------------------------------------------- diff --git a/test/e2e/engine-parity.test.ts b/test/e2e/engine-parity.test.ts index 549ce1371..70faefe85 100644 --- a/test/e2e/engine-parity.test.ts +++ b/test/e2e/engine-parity.test.ts @@ -363,4 +363,53 @@ describeBoth('Engine parity — Postgres vs PGLite', () => { expect(pgMap.get('wiki/rsp-missing.md')).toBeUndefined(); expect(pgliteMap.get('wiki/rsp-missing.md')).toBeUndefined(); }); + + // v0.41.29.0 — findOrphanPages source scoping parity. Real Postgres + // coverage for the postgres.js `sql` scalar fragment + `= ANY(${arr}::text[])` + // array binding (a documented footgun class — the jsonb double-encode saga). + // PGLite logic is pinned in test/orphans-source-scope.test.ts; this asserts + // the Postgres SQL produces the same scoped sets. Cross-source inbound + // (src-b → src-a) must NOT make the target an orphan of src-a (A2). + test('v0.41.29.0 findOrphanPages source scoping parity (scalar + federated)', async () => { + const srcSql = `INSERT INTO sources (id, name, config) VALUES ($1, $1, '{}'::jsonb) ON CONFLICT DO NOTHING`; + const pageSql = ` + INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ($1, $2, 'person', 't', 'b', '', '{}'::jsonb) + ON CONFLICT (source_id, slug) DO NOTHING + `; + for (const eng of [pgEngine, pgliteEngine]) { + await eng.executeRaw(srcSql, ['orphan-src-a']); + await eng.executeRaw(srcSql, ['orphan-src-b']); + await eng.executeRaw(pageSql, ['orphan-src-a', 'people/op-orphan-a']); + await eng.executeRaw(pageSql, ['orphan-src-a', 'people/op-target-a']); + await eng.executeRaw(pageSql, ['orphan-src-b', 'people/op-linker-b']); + // Cross-source inbound: src-b page → src-a target (A2). + await eng.addLink( + 'people/op-linker-b', 'people/op-target-a', '', 'mentions', 'markdown', + undefined, undefined, { fromSourceId: 'orphan-src-b', toSourceId: 'orphan-src-a' }, + ); + } + + const scoped = async (eng: BrainEngine, opts: { sourceId?: string; sourceIds?: string[] }) => + (await eng.findOrphanPages(opts)).map(r => r.slug).filter(s => s.startsWith('people/op-')).sort(); + + // Scalar scope to src-a: op-orphan-a is an orphan; op-target-a is saved + // by the cross-source inbound (A2). Parity on both engines. + const pgA = await scoped(pgEngine, { sourceId: 'orphan-src-a' }); + const pgliteA = await scoped(pgliteEngine, { sourceId: 'orphan-src-a' }); + expect(pgA).toEqual(['people/op-orphan-a']); + expect(pgliteA).toEqual(pgA); + + // Scalar scope to src-b. + const pgB = await scoped(pgEngine, { sourceId: 'orphan-src-b' }); + const pgliteB = await scoped(pgliteEngine, { sourceId: 'orphan-src-b' }); + expect(pgB).toEqual(['people/op-linker-b']); + expect(pgliteB).toEqual(pgB); + + // Federated array scope (= ANY binding) → union. + const pgFed = await scoped(pgEngine, { sourceIds: ['orphan-src-a', 'orphan-src-b'] }); + const pgliteFed = await scoped(pgliteEngine, { sourceIds: ['orphan-src-a', 'orphan-src-b'] }); + expect(pgFed).toEqual(['people/op-linker-b', 'people/op-orphan-a']); + expect(pgliteFed).toEqual(pgFed); + }); }); diff --git a/test/fixtures/conversation-formats/adversarial.jsonl b/test/fixtures/conversation-formats/adversarial.jsonl index fafac150a..2be6cc67a 100644 --- a/test/fixtures/conversation-formats/adversarial.jsonl +++ b/test/fixtures/conversation-formats/adversarial.jsonl @@ -3,3 +3,4 @@ {"fixture_id":"adversarial-code","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"function add(a, b) {\n return a + b;\n}\n\nconst result = add(1, 2);\nconsole.log(result);","expected_messages":0,"expected_participants":[]} {"fixture_id":"adversarial-lyrics","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"Twinkle twinkle little star\nHow I wonder what you are\nUp above the world so high\nLike a diamond in the sky","expected_messages":0,"expected_participants":[]} {"fixture_id":"adversarial-json","pattern":null,"frontmatter":{"date":"2024-03-15"},"body":"{\n \"key\": \"value\",\n \"num\": 42,\n \"list\": [1, 2, 3]\n}","expected_messages":0,"expected_participants":[]} +{"fixture_id":"adversarial-bold-labels-clustered-head","pattern":null,"frontmatter":{"date":"2026-05-28"},"body":"**Attendees:** Alice Example, Bob Example, Participant 2\n**Date:** 2026-05-28\n**Goal:** decide on the Q3 roadmap and unblock the vendor migration\nThis is an ordinary prose sentence number 0 describing the meeting in detail.\nThis is an ordinary prose sentence number 1 describing the meeting in detail.\nThis is an ordinary prose sentence number 2 describing the meeting in detail.\nThis is an ordinary prose sentence number 3 describing the meeting in detail.\nThis is an ordinary prose sentence number 4 describing the meeting in detail.\nThis is an ordinary prose sentence number 5 describing the meeting in detail.\nThis is an ordinary prose sentence number 6 describing the meeting in detail.\nThis is an ordinary prose sentence number 7 describing the meeting in detail.\nThis is an ordinary prose sentence number 8 describing the meeting in detail.\nThis is an ordinary prose sentence number 9 describing the meeting in detail.\nThis is an ordinary prose sentence number 10 describing the meeting in detail.\nThis is an ordinary prose sentence number 11 describing the meeting in detail.\nThis is an ordinary prose sentence number 12 describing the meeting in detail.\nThis is an ordinary prose sentence number 13 describing the meeting in detail.\nThis is an ordinary prose sentence number 14 describing the meeting in detail.\nThis is an ordinary prose sentence number 15 describing the meeting in detail.\nThis is an ordinary prose sentence number 16 describing the meeting in detail.\nThis is an ordinary prose sentence number 17 describing the meeting in detail.\nThis is an ordinary prose sentence number 18 describing the meeting in detail.\nThis is an ordinary prose sentence number 19 describing the meeting in detail.\nThis is an ordinary prose sentence number 20 describing the meeting in detail.\nThis is an ordinary prose sentence number 21 describing the meeting in detail.\nThis is an ordinary prose sentence number 22 describing the meeting in detail.\nThis is an ordinary prose sentence number 23 describing the meeting in detail.\nThis is an ordinary prose sentence number 24 describing the meeting in detail.\nThis is an ordinary prose sentence number 25 describing the meeting in detail.\nThis is an ordinary prose sentence number 26 describing the meeting in detail.\nThis is an ordinary prose sentence number 27 describing the meeting in detail.\nThis is an ordinary prose sentence number 28 describing the meeting in detail.\nThis is an ordinary prose sentence number 29 describing the meeting in detail.\nThis is an ordinary prose sentence number 30 describing the meeting in detail.\nThis is an ordinary prose sentence number 31 describing the meeting in detail.\nThis is an ordinary prose sentence number 32 describing the meeting in detail.\nThis is an ordinary prose sentence number 33 describing the meeting in detail.\nThis is an ordinary prose sentence number 34 describing the meeting in detail.\nThis is an ordinary prose sentence number 35 describing the meeting in detail.\nThis is an ordinary prose sentence number 36 describing the meeting in detail.\nThis is an ordinary prose sentence number 37 describing the meeting in detail.\nThis is an ordinary prose sentence number 38 describing the meeting in detail.\nThis is an ordinary prose sentence number 39 describing the meeting in detail.\nThis is an ordinary prose sentence number 40 describing the meeting in detail.\nThis is an ordinary prose sentence number 41 describing the meeting in detail.\nThis is an ordinary prose sentence number 42 describing the meeting in detail.\nThis is an ordinary prose sentence number 43 describing the meeting in detail.\nThis is an ordinary prose sentence number 44 describing the meeting in detail.\nThis is an ordinary prose sentence number 45 describing the meeting in detail.\nThis is an ordinary prose sentence number 46 describing the meeting in detail.\nThis is an ordinary prose sentence number 47 describing the meeting in detail.\nThis is an ordinary prose sentence number 48 describing the meeting in detail.\nThis is an ordinary prose sentence number 49 describing the meeting in detail.\nThis is an ordinary prose sentence number 50 describing the meeting in detail.\nThis is an ordinary prose sentence number 51 describing the meeting in detail.\nThis is an ordinary prose sentence number 52 describing the meeting in detail.\nThis is an ordinary prose sentence number 53 describing the meeting in detail.\nThis is an ordinary prose sentence number 54 describing the meeting in detail.\nThis is an ordinary prose sentence number 55 describing the meeting in detail.\nThis is an ordinary prose sentence number 56 describing the meeting in detail.\nThis is an ordinary prose sentence number 57 describing the meeting in detail.\nThis is an ordinary prose sentence number 58 describing the meeting in detail.\nThis is an ordinary prose sentence number 59 describing the meeting in detail.\nThis is an ordinary prose sentence number 60 describing the meeting in detail.\nThis is an ordinary prose sentence number 61 describing the meeting in detail.\nThis is an ordinary prose sentence number 62 describing the meeting in detail.\nThis is an ordinary prose sentence number 63 describing the meeting in detail.\nThis is an ordinary prose sentence number 64 describing the meeting in detail.\nThis is an ordinary prose sentence number 65 describing the meeting in detail.\nThis is an ordinary prose sentence number 66 describing the meeting in detail.\nThis is an ordinary prose sentence number 67 describing the meeting in detail.\nThis is an ordinary prose sentence number 68 describing the meeting in detail.\nThis is an ordinary prose sentence number 69 describing the meeting in detail.\nThis is an ordinary prose sentence number 70 describing the meeting in detail.\nThis is an ordinary prose sentence number 71 describing the meeting in detail.\nThis is an ordinary prose sentence number 72 describing the meeting in detail.\nThis is an ordinary prose sentence number 73 describing the meeting in detail.\nThis is an ordinary prose sentence number 74 describing the meeting in detail.\nThis is an ordinary prose sentence number 75 describing the meeting in detail.\nThis is an ordinary prose sentence number 76 describing the meeting in detail.\nThis is an ordinary prose sentence number 77 describing the meeting in detail.\nThis is an ordinary prose sentence number 78 describing the meeting in detail.\nThis is an ordinary prose sentence number 79 describing the meeting in detail.","expected_messages":0,"expected_participants":[]} diff --git a/test/fixtures/conversation-formats/all.jsonl b/test/fixtures/conversation-formats/all.jsonl index 91461155e..5127a65b5 100644 --- a/test/fixtures/conversation-formats/all.jsonl +++ b/test/fixtures/conversation-formats/all.jsonl @@ -11,3 +11,6 @@ {"fixture_id":"irc-weechat-001","pattern":"irc-weechat","frontmatter":{"date":"2024-03-15"},"body":"18:37 hello\n18:38 hey\n18:39 how are you\n18:40 good","expected_messages":4,"expected_participants":["alice","bob"]} {"fixture_id":"matrix-element-001","pattern":"matrix-element","frontmatter":{"date":"2024-03-15"},"body":"[18:37] @alice:matrix.org: hello world\n[18:38] @bob:example.org: hey\n[18:39] @alice:matrix.org: how are you\n[18:40] @bob:example.org: good","expected_messages":4,"expected_participants":["alice","bob"]} {"fixture_id":"teams-export-001","pattern":"teams-export","frontmatter":{"date":"2024-03-15"},"body":"Alice Example, 3/15/2024 6:37 PM: hello team\nBob Example, 3/15/2024 6:38 PM: hey\nAlice Example, 3/15/2024 6:39 PM: meeting at 4?\nBob Example, 3/15/2024 6:40 PM: works for me","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"bold-name-no-time-001","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** Okay, start on. And then weirdly like zoom doesn’t work.\n**Bob Example:** he tried to reset it remotely the other night. Let me ask him.\n**Alice Example:** I mean it’s really just like we need to get zoom to fix this.\n**Bob Example:** Okay, let me.","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"bold-name-no-time-002","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** can you hear me now\n**Participant 2:** yeah loud and clear\n**Alice Example:** great, let us start the review","expected_messages":3,"expected_participants":["Alice Example","Participant 2"]} +{"fixture_id":"adversarial-bold-labels-clustered-head","pattern":null,"frontmatter":{"date":"2026-05-28"},"body":"**Attendees:** Alice Example, Bob Example, Participant 2\n**Date:** 2026-05-28\n**Goal:** decide on the Q3 roadmap and unblock the vendor migration\nThis is an ordinary prose sentence number 0 describing the meeting in detail.\nThis is an ordinary prose sentence number 1 describing the meeting in detail.\nThis is an ordinary prose sentence number 2 describing the meeting in detail.\nThis is an ordinary prose sentence number 3 describing the meeting in detail.\nThis is an ordinary prose sentence number 4 describing the meeting in detail.\nThis is an ordinary prose sentence number 5 describing the meeting in detail.\nThis is an ordinary prose sentence number 6 describing the meeting in detail.\nThis is an ordinary prose sentence number 7 describing the meeting in detail.\nThis is an ordinary prose sentence number 8 describing the meeting in detail.\nThis is an ordinary prose sentence number 9 describing the meeting in detail.\nThis is an ordinary prose sentence number 10 describing the meeting in detail.\nThis is an ordinary prose sentence number 11 describing the meeting in detail.\nThis is an ordinary prose sentence number 12 describing the meeting in detail.\nThis is an ordinary prose sentence number 13 describing the meeting in detail.\nThis is an ordinary prose sentence number 14 describing the meeting in detail.\nThis is an ordinary prose sentence number 15 describing the meeting in detail.\nThis is an ordinary prose sentence number 16 describing the meeting in detail.\nThis is an ordinary prose sentence number 17 describing the meeting in detail.\nThis is an ordinary prose sentence number 18 describing the meeting in detail.\nThis is an ordinary prose sentence number 19 describing the meeting in detail.\nThis is an ordinary prose sentence number 20 describing the meeting in detail.\nThis is an ordinary prose sentence number 21 describing the meeting in detail.\nThis is an ordinary prose sentence number 22 describing the meeting in detail.\nThis is an ordinary prose sentence number 23 describing the meeting in detail.\nThis is an ordinary prose sentence number 24 describing the meeting in detail.\nThis is an ordinary prose sentence number 25 describing the meeting in detail.\nThis is an ordinary prose sentence number 26 describing the meeting in detail.\nThis is an ordinary prose sentence number 27 describing the meeting in detail.\nThis is an ordinary prose sentence number 28 describing the meeting in detail.\nThis is an ordinary prose sentence number 29 describing the meeting in detail.\nThis is an ordinary prose sentence number 30 describing the meeting in detail.\nThis is an ordinary prose sentence number 31 describing the meeting in detail.\nThis is an ordinary prose sentence number 32 describing the meeting in detail.\nThis is an ordinary prose sentence number 33 describing the meeting in detail.\nThis is an ordinary prose sentence number 34 describing the meeting in detail.\nThis is an ordinary prose sentence number 35 describing the meeting in detail.\nThis is an ordinary prose sentence number 36 describing the meeting in detail.\nThis is an ordinary prose sentence number 37 describing the meeting in detail.\nThis is an ordinary prose sentence number 38 describing the meeting in detail.\nThis is an ordinary prose sentence number 39 describing the meeting in detail.\nThis is an ordinary prose sentence number 40 describing the meeting in detail.\nThis is an ordinary prose sentence number 41 describing the meeting in detail.\nThis is an ordinary prose sentence number 42 describing the meeting in detail.\nThis is an ordinary prose sentence number 43 describing the meeting in detail.\nThis is an ordinary prose sentence number 44 describing the meeting in detail.\nThis is an ordinary prose sentence number 45 describing the meeting in detail.\nThis is an ordinary prose sentence number 46 describing the meeting in detail.\nThis is an ordinary prose sentence number 47 describing the meeting in detail.\nThis is an ordinary prose sentence number 48 describing the meeting in detail.\nThis is an ordinary prose sentence number 49 describing the meeting in detail.\nThis is an ordinary prose sentence number 50 describing the meeting in detail.\nThis is an ordinary prose sentence number 51 describing the meeting in detail.\nThis is an ordinary prose sentence number 52 describing the meeting in detail.\nThis is an ordinary prose sentence number 53 describing the meeting in detail.\nThis is an ordinary prose sentence number 54 describing the meeting in detail.\nThis is an ordinary prose sentence number 55 describing the meeting in detail.\nThis is an ordinary prose sentence number 56 describing the meeting in detail.\nThis is an ordinary prose sentence number 57 describing the meeting in detail.\nThis is an ordinary prose sentence number 58 describing the meeting in detail.\nThis is an ordinary prose sentence number 59 describing the meeting in detail.\nThis is an ordinary prose sentence number 60 describing the meeting in detail.\nThis is an ordinary prose sentence number 61 describing the meeting in detail.\nThis is an ordinary prose sentence number 62 describing the meeting in detail.\nThis is an ordinary prose sentence number 63 describing the meeting in detail.\nThis is an ordinary prose sentence number 64 describing the meeting in detail.\nThis is an ordinary prose sentence number 65 describing the meeting in detail.\nThis is an ordinary prose sentence number 66 describing the meeting in detail.\nThis is an ordinary prose sentence number 67 describing the meeting in detail.\nThis is an ordinary prose sentence number 68 describing the meeting in detail.\nThis is an ordinary prose sentence number 69 describing the meeting in detail.\nThis is an ordinary prose sentence number 70 describing the meeting in detail.\nThis is an ordinary prose sentence number 71 describing the meeting in detail.\nThis is an ordinary prose sentence number 72 describing the meeting in detail.\nThis is an ordinary prose sentence number 73 describing the meeting in detail.\nThis is an ordinary prose sentence number 74 describing the meeting in detail.\nThis is an ordinary prose sentence number 75 describing the meeting in detail.\nThis is an ordinary prose sentence number 76 describing the meeting in detail.\nThis is an ordinary prose sentence number 77 describing the meeting in detail.\nThis is an ordinary prose sentence number 78 describing the meeting in detail.\nThis is an ordinary prose sentence number 79 describing the meeting in detail.","expected_messages":0,"expected_participants":[]} diff --git a/test/fixtures/conversation-formats/bold-name-no-time.jsonl b/test/fixtures/conversation-formats/bold-name-no-time.jsonl new file mode 100644 index 000000000..3dfe9b818 --- /dev/null +++ b/test/fixtures/conversation-formats/bold-name-no-time.jsonl @@ -0,0 +1,2 @@ +{"fixture_id":"bold-name-no-time-001","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** Okay, start on. And then weirdly like zoom doesn’t work.\n**Bob Example:** he tried to reset it remotely the other night. Let me ask him.\n**Alice Example:** I mean it’s really just like we need to get zoom to fix this.\n**Bob Example:** Okay, let me.","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} +{"fixture_id":"bold-name-no-time-002","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** can you hear me now\n**Participant 2:** yeah loud and clear\n**Alice Example:** great, let us start the review","expected_messages":3,"expected_participants":["Alice Example","Participant 2"]} diff --git a/test/orphans-source-scope.test.ts b/test/orphans-source-scope.test.ts new file mode 100644 index 000000000..75ef21418 --- /dev/null +++ b/test/orphans-source-scope.test.ts @@ -0,0 +1,213 @@ +/** + * v0.41.29.0 — orphan source-scoping regression suite. + * + * Covers three Codex-flagged behaviors of the `--source` / find_orphans + * source-scoping wave, all at the layer the bugs live (engine + + * getOrphansData + the find_orphans op handler): + * + * - A2: candidate-only scoping. A page in source X linked FROM source Y + * is reachable, so it is NOT an orphan of X (cross-source inbound counts). + * - F6: total_linkable denominator. Excluded NON-orphan pages (e.g. a + * `templates/` page that HAS inbound links) must NOT inflate the + * denominator. Pre-fix the formula `total - excludedOrphans` left them in. + * - F8: the find_orphans MCP op handler scopes by ctx.sourceId AND + * ctx.auth.allowedSources (the v0.34.1 source-isolation read leak). + * + * Runs against PGLite in-memory (both engines share the SQL surface; the + * Postgres path is pinned separately in test/e2e/multi-source-bug-class). + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { findOrphans } from '../src/commands/orphans.ts'; + +let engine: PGLiteEngine; + +const page = (title: string) => ({ + type: 'person' as const, + title, + compiled_truth: `${title} body.`, + timeline: '', + frontmatter: {}, +}); + +// Every test in this file is READ-ONLY (findOrphanPages / getOrphansData / +// find_orphans op handler). Seed ONCE in beforeAll rather than reset+reseed +// per test — the canonical R3/R4 pattern (engine in beforeAll, disconnect in +// afterAll) is satisfied, and this cuts the file from ~6min to seconds. +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await resetPgliteState(engine); + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ('src-b', 'src-b', '{}'::jsonb) ON CONFLICT DO NOTHING`, + ); + + // --- default source --- + // alice-orphan: no inbound → orphan. + await engine.putPage('people/alice-orphan', page('Alice Orphan'), { sourceId: 'default' }); + // bob-linked: inbound from alice-orphan → NOT orphan. + await engine.putPage('people/bob-linked', page('Bob Linked'), { sourceId: 'default' }); + // cross-target: inbound ONLY from a src-b page → A2: NOT orphan of default. + await engine.putPage('people/cross-target', page('Cross Target'), { sourceId: 'default' }); + // templates/junk-linked: excluded prefix, HAS inbound → not orphan, and must + // be excluded from total_linkable (F6). + await engine.putPage('templates/junk-linked', page('Junk Template'), { sourceId: 'default' }); + + // --- src-b source --- + // zara-orphan: no inbound → orphan. + await engine.putPage('people/zara-orphan', page('Zara Orphan'), { sourceId: 'src-b' }); + // src-b-linker: links to default's cross-target; itself has no inbound → orphan. + await engine.putPage('people/src-b-linker', page('Src-B Linker'), { sourceId: 'src-b' }); + + // --- links --- + await engine.addLink( + 'people/alice-orphan', 'people/bob-linked', '', 'mentions', 'markdown', + undefined, undefined, { fromSourceId: 'default', toSourceId: 'default' }, + ); + await engine.addLink( + 'people/alice-orphan', 'templates/junk-linked', '', 'mentions', 'markdown', + undefined, undefined, { fromSourceId: 'default', toSourceId: 'default' }, + ); + // Cross-source inbound: src-b page → default page (A2). + await engine.addLink( + 'people/src-b-linker', 'people/cross-target', '', 'mentions', 'markdown', + undefined, undefined, { fromSourceId: 'src-b', toSourceId: 'default' }, + ); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('findOrphanPages — source scoping (A2)', () => { + test('scoped to default: alice-orphan is an orphan; cross-source-linked page is NOT', async () => { + const rows = await engine.findOrphanPages({ sourceId: 'default' }); + const slugs = rows.map(r => r.slug); + expect(slugs).toContain('people/alice-orphan'); + // A2: cross-target has inbound from src-b → reachable → NOT an orphan. + expect(slugs).not.toContain('people/cross-target'); + // bob-linked + templates/junk-linked have intra-source inbound. + expect(slugs).not.toContain('people/bob-linked'); + expect(slugs).not.toContain('templates/junk-linked'); + // src-b pages must not appear in a default-scoped scan. + expect(slugs).not.toContain('people/zara-orphan'); + expect(slugs).not.toContain('people/src-b-linker'); + }); + + test('scoped to src-b returns a different orphan set', async () => { + const rows = await engine.findOrphanPages({ sourceId: 'src-b' }); + const slugs = rows.map(r => r.slug).sort(); + expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']); + }); + + test('unscoped (brain-wide) returns the union', async () => { + const rows = await engine.findOrphanPages(); + const slugs = rows.map(r => r.slug).sort(); + expect(slugs).toEqual([ + 'people/alice-orphan', + 'people/src-b-linker', + 'people/zara-orphan', + ]); + }); + + test('federated sourceIds scopes to the union of the given sources', async () => { + const rows = await engine.findOrphanPages({ sourceIds: ['default', 'src-b'] }); + const slugs = rows.map(r => r.slug).sort(); + expect(slugs).toEqual([ + 'people/alice-orphan', + 'people/src-b-linker', + 'people/zara-orphan', + ]); + // Single-element federated array behaves like the scalar scope. + const onlyB = await engine.findOrphanPages({ sourceIds: ['src-b'] }); + expect(onlyB.map(r => r.slug).sort()).toEqual([ + 'people/src-b-linker', + 'people/zara-orphan', + ]); + }); +}); + +describe('getOrphansData — scoped totals + F6 denominator', () => { + test('default scope: counts differ from brain-wide; excluded non-orphan drops from total_linkable (F6)', async () => { + const data = await findOrphans(engine, { includePseudo: false, sourceId: 'default' }); + // 4 live default pages; 1 orphan (alice-orphan). + expect(data.total_pages).toBe(4); + expect(data.total_orphans).toBe(1); + expect(data.orphans.map(o => o.slug)).toEqual(['people/alice-orphan']); + // F6: templates/junk-linked is an EXCLUDED page that HAS an inbound link. + // It must NOT count in total_linkable. NEW denominator = total(4) - + // excludedAll(1) = 3. The pre-fix formula (total - excludedOrphans) left + // the excluded non-orphan in and returned 4. + expect(data.total_linkable).toBe(3); + expect(data.total_linkable).toBe(data.total_pages - 1); + }); + + test('src-b scope: distinct from default', async () => { + const data = await findOrphans(engine, { includePseudo: false, sourceId: 'src-b' }); + expect(data.total_pages).toBe(2); + expect(data.total_orphans).toBe(2); + expect(data.total_linkable).toBe(2); // no excluded pages in src-b + }); + + test('brain-wide F6: excluded non-orphan drops from total_linkable', async () => { + const data = await findOrphans(engine, { includePseudo: false }); + expect(data.total_pages).toBe(6); + expect(data.total_orphans).toBe(3); + // 6 live pages - 1 excluded (templates/junk-linked) = 5. + expect(data.total_linkable).toBe(5); + }); + + test('includePseudo keeps excluded pages in the denominator', async () => { + const data = await findOrphans(engine, { includePseudo: true, sourceId: 'default' }); + // No exclusion → denominator is the full live set. + expect(data.total_linkable).toBe(4); + }); +}); + +describe('find_orphans MCP op — F8 source-isolation scope', () => { + test('ctx.sourceId scopes results to that source only', async () => { + const { operations } = await import('../src/core/operations.ts'); + const op = operations.find(o => o.name === 'find_orphans'); + expect(op).toBeDefined(); + const ctx = { + engine, + config: { engine: 'pglite' as const }, + logger: { info: () => {}, warn: () => {}, error: () => {} }, + dryRun: false, + remote: true, + sourceId: 'src-b', + }; + const result = (await op!.handler(ctx as any, {})) as { orphans: { slug: string }[] }; + const slugs = result.orphans.map(o => o.slug).sort(); + expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']); + // Leak guard: default's orphan must NOT appear. + expect(slugs).not.toContain('people/alice-orphan'); + }); + + test('ctx.auth.allowedSources (federated) scopes to the allowed set', async () => { + const { operations } = await import('../src/core/operations.ts'); + const op = operations.find(o => o.name === 'find_orphans'); + const ctx = { + engine, + config: { engine: 'pglite' as const }, + logger: { info: () => {}, warn: () => {}, error: () => {} }, + dryRun: false, + remote: true, + sourceId: 'default', // scalar would scope to default-only... + auth: { + token: 'test', + clientId: 'test', + scopes: ['read'], + sourceId: 'default', + allowedSources: ['src-b'], // ...but the federated array wins. + }, + }; + const result = (await op!.handler(ctx as any, {})) as { orphans: { slug: string }[] }; + const slugs = result.orphans.map(o => o.slug).sort(); + expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']); + expect(slugs).not.toContain('people/alice-orphan'); + }); +});