mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.41.29.0 feat(conversation-parser): bold-name-no-time builtin + fix(orphans): source-scoped orphan_ratio (supersedes #1613) (#1620)
* feat(conversation-parser): add bold-name-no-time builtin (Circleback/Granola/Zoom, no timestamp) The 14th built-in pattern parses `**Speaker:** text` transcripts with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Every prior builtin required a time anchor, so this shape matched nothing: a production brain had 104 conversation pages + 3,423 eligible pages silently extracting zero facts. Messages anchor at T00:00:00Z of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same convention as irc-classic. Hardening beyond the original community proposal: - regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`: the colon-inside-bold (NOT declaration order) is what prevents shadowing bold-paren-time; the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest no_match instead of speaker="[18:37] Name". - new optional PatternEntry.score_full_body: `**Label:** text` is a common prose idiom, so a notes page with bold labels clustered in its first 10 lines scored 0.3 on the head pass (NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so the full-body fallback never fired) and cleared the 0.05 floor. parse.ts now recomputes the winner's score over the full body before the floor, so such a page drops to its true low density and stays no_match. - scrubbed pre-existing real names from bold-paren-time test_positive samples (privacy rule). Fixtures use placeholder names only. Pinned by new bold-name-no-time + clustered-head no_match cases in parse.test.ts and the eval corpus. Co-Authored-By: garrytan-agents <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(orphans): scope orphan_ratio + find_orphans by source; fix total_linkable denominator `gbrain doctor --source <id>` and `gbrain orphans --source <id>` now scope the orphan scan to that source instead of reporting brain-wide. Three fixes: - findOrphanPages(opts?: { sourceId?, sourceIds? }) on both engines scopes the CANDIDATE set (scalar `= $1` or federated `= ANY($1::text[])`). Inbound links from ANY source still count, so a page in source X linked FROM source Y is reachable and NOT an orphan of X (the deliberate, less-surprising definition). - corrected the total_linkable denominator in findOrphans: it now enumerates all live pages (scoped) and subtracts every excluded-by-slug page, not just excluded orphans. The old `total - excludedOrphans` left excluded NON-orphan pages (templates/, scratch/) with inbound links in the denominator, inflating it and suppressing warnings. Changes orphan_ratio output for every brain, in the accurate direction. - the find_orphans MCP op threads sourceScopeOpts(ctx), closing a cross-source read leak where a source-bound OAuth client saw brain-wide orphans (v0.34.1 source-isolation class). doctor uses an explicit `--source` flag parse (NOT resolveSourceWithTier, which would scope bare invocations to a default), and under explicit --source reports the ratio with a low-scale caveat below 100 entity pages instead of a vacuous "ok". Thin-client doctor --source orphan_ratio deferred (TODOS.md). Pinned by test/orphans-source-scope.test.ts (PGLite: scoping, cross-source inbound, denominator, find_orphans op scope) + a Postgres↔PGLite parity case in test/e2e/engine-parity.test.ts (scalar + federated binding). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: v0.41.29.0 — bold-name-no-time + orphan source scoping VERSION + package.json → 0.41.29.0; CHANGELOG entry; CLAUDE.md conversation-parser (13→14 patterns) + orphans source-scoping notes; regenerated llms bundles; TODOS for thin-client doctor --source + check-test-real-names widening. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
garrytan-agents
Claude Opus 4.8
garrytan-agents
Claude Opus 4.8
parent
ffac8ce0f4
commit
041d89babe
+131
@@ -2,6 +2,137 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.29.0] - 2026-05-29
|
||||
|
||||
**Your meeting transcripts that look like `**Garry Tan:** ...` now actually
|
||||
parse — the kind Circleback, Granola, and Zoom export with no timestamp on
|
||||
each line. And `gbrain doctor --source <id>` finally scopes its orphan check
|
||||
to that one source instead of your whole brain.**
|
||||
|
||||
Two fixes in one release.
|
||||
|
||||
**The transcript fix.** gbrain knows 13 chat formats, and every single one
|
||||
needed a per-line timestamp to recognize a speaker. Modern meeting tools
|
||||
don't put a time on every line — they emit plain `**Speaker Name:** what
|
||||
they said`. With no time anchor, gbrain matched nothing: a real production
|
||||
brain had 104 conversation pages plus 3,423 other eligible pages that
|
||||
silently parsed to zero messages and extracted zero facts. The 14th built-in
|
||||
pattern, `bold-name-no-time`, handles that shape. Each message anchors at
|
||||
00:00:00 of the page's date (no fake wall-clock times are invented; line
|
||||
order preserves the sequence). Nothing you do — it just works the next time
|
||||
your cycle runs.
|
||||
|
||||
How to check it's recognized:
|
||||
|
||||
```
|
||||
gbrain conversation-parser list-builtins # bold-name-no-time is now listed
|
||||
gbrain conversation-parser scan <slug> # see how one page parses
|
||||
```
|
||||
|
||||
One thing we were careful about: that `**Label:** text` shape is also a
|
||||
common way to write notes (`**Owner:** Alice`, `**Action:** ship it`). A
|
||||
notes page is NOT a conversation, and gbrain won't treat it as one — even
|
||||
when a few bold labels sit at the very top of the page. We score the whole
|
||||
document's density, not just the first few lines, so a mostly-prose page
|
||||
with scattered bold labels stays unparsed instead of turning into fake
|
||||
"messages" that pollute your facts.
|
||||
|
||||
**The orphan-check fix.** "Orphan ratio" is the fraction of your pages that
|
||||
nothing links to — a health signal for `gbrain doctor`. On a brain with
|
||||
multiple sources, `gbrain doctor --source dept-x` ignored the flag and
|
||||
reported a brain-wide number. Now it scopes to the source you asked about,
|
||||
and so does `gbrain orphans --source dept-x`:
|
||||
|
||||
```
|
||||
gbrain doctor --source dept-x # orphan ratio for dept-x only
|
||||
gbrain orphans --source dept-x # the orphan list for dept-x only
|
||||
```
|
||||
|
||||
A page in dept-x that's linked from another source counts as reachable (not
|
||||
an orphan) — the cross-source link still saves it.
|
||||
|
||||
While we were in there we fixed two more things you'll feel:
|
||||
|
||||
- The orphan ratio was reading lower than reality because junk pages
|
||||
(templates, scratch, archives) that happen to be linked were padding the
|
||||
denominator. The ratio is now honest, which means warnings that were
|
||||
staying quiet will start showing up. This changes the number on every
|
||||
brain, in the right direction.
|
||||
- An agent connected over MCP and scoped to one source used to get the
|
||||
orphan list for your *whole* brain. Now `find_orphans` respects the
|
||||
agent's source scope, closing a cross-source read leak.
|
||||
|
||||
### To take advantage of v0.41.29.0
|
||||
|
||||
`gbrain upgrade` handles this — there's no schema migration and no manual
|
||||
step. After upgrading:
|
||||
|
||||
1. **Conversation parsing** is automatic. Your next `gbrain dream` /
|
||||
`gbrain extract conversation-facts` run will pick up `**Speaker:** text`
|
||||
pages. To preview a specific page:
|
||||
```bash
|
||||
gbrain conversation-parser scan <slug>
|
||||
```
|
||||
2. **Per-source orphan checks** work immediately:
|
||||
```bash
|
||||
gbrain doctor --source <id>
|
||||
gbrain orphans --source <id>
|
||||
```
|
||||
3. **If `gbrain doctor` shows a higher orphan ratio than before,** that's the
|
||||
denominator fix surfacing real orphans that were previously hidden. Run
|
||||
`gbrain extract links --by-mention` to auto-link entity mentions, then
|
||||
`gbrain orphans` for the list.
|
||||
4. **If anything looks wrong,** file an issue at
|
||||
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Conversation parser**
|
||||
- New 14th built-in pattern `bold-name-no-time` in
|
||||
`src/core/conversation-parser/builtins.ts` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`):
|
||||
parses `**Speaker:** text` with no per-line timestamp (Circleback / Granola
|
||||
/ Zoom), anchored at `T00:00:00Z` of the frontmatter date. Declared after
|
||||
the timestamped bold patterns; the colon-inside-bold regex (not declaration
|
||||
order) is what prevents it from shadowing `bold-paren-time`. The `(?!\[)`
|
||||
lookahead keeps it from mis-capturing telegram-bracket `**[18:37] Name:**`
|
||||
lines as `speaker="[18:37] Name"`.
|
||||
- New optional `PatternEntry.score_full_body` field + a full-body acceptance
|
||||
recompute in `parse.ts`: broad patterns are scored on whole-document
|
||||
density before the acceptance floor, so a prose notes page with bold labels
|
||||
clustered in its first 10 lines no longer mis-parses as a conversation.
|
||||
- Fixture coverage: `test/fixtures/conversation-formats/bold-name-no-time.jsonl`
|
||||
+ entries in `all.jsonl` and a clustered-head adversarial fixture in
|
||||
`adversarial.jsonl` (all placeholder names).
|
||||
|
||||
**Orphan source scoping**
|
||||
- `BrainEngine.findOrphanPages(opts?: { sourceId?, sourceIds? })` (both
|
||||
engines): scopes the candidate set to one source (scalar) or a federated
|
||||
set (`= ANY(...)`). Inbound links from any source still count, so a
|
||||
cross-source-linked page is correctly not an orphan.
|
||||
- `gbrain orphans --source <id>` and `gbrain doctor --source <id>` (orphan_ratio
|
||||
check) honor an explicit source. Bare invocations stay brain-wide.
|
||||
- Corrected the `total_linkable` denominator in `findOrphans` so excluded
|
||||
pages (templates/, scratch/, etc.) that have inbound links no longer
|
||||
inflate it and suppress warnings. Changes orphan_ratio output for every
|
||||
brain (in the accurate direction).
|
||||
- The `find_orphans` MCP op now scopes by the caller's source
|
||||
(`sourceScopeOpts(ctx)`), closing a cross-source read leak for source-bound
|
||||
OAuth clients.
|
||||
- Under explicit `--source`, `orphan_ratio` reports the ratio with a
|
||||
low-scale caveat below 100 entity pages instead of returning a vacuous
|
||||
"ok" that could hide a fully-orphaned small source.
|
||||
|
||||
**Tests**
|
||||
- `test/orphans-source-scope.test.ts` (PGLite): scoping, cross-source-inbound,
|
||||
the denominator fix, and the `find_orphans` op-handler source scope.
|
||||
- `test/e2e/engine-parity.test.ts`: Postgres↔PGLite parity for
|
||||
`findOrphanPages` scalar + federated scoping.
|
||||
- New `bold-name-no-time` + clustered-head regression cases in
|
||||
`test/conversation-parser/parse.test.ts`.
|
||||
|
||||
Incorporates the original `bold-name-no-time` pattern proposed in a community
|
||||
PR; names scrubbed to placeholders per the project privacy rule.
|
||||
|
||||
## [0.41.28.0] - 2026-05-27
|
||||
|
||||
**Your `gbrain dream` cycle stops losing rows when the database connection
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
# TODOS
|
||||
|
||||
## v0.41.29.0 orphan source-scoping follow-ups (v0.42+)
|
||||
|
||||
Filed from the v0.41.29.0 wave (bold-name-no-time pattern + orphan_ratio
|
||||
source scoping). The Codex outside-voice review (F8) flagged two surfaces
|
||||
the wave deliberately scoped out.
|
||||
|
||||
- [ ] **v0.42+: thin-client `gbrain doctor --source` orphan_ratio scoping.** v0.41.29.0 scopes `orphan_ratio` to `--source` on the LOCAL doctor path (`buildChecks` in `src/commands/doctor.ts`) and closes the `find_orphans` MCP read leak via `sourceScopeOpts(ctx)`. The thin-client / remote doctor path (`src/core/doctor-remote.ts` `runRemoteDoctor`) is a separate code path that does not thread `--source`, so `gbrain doctor --source x` against a remote `gbrain serve --http` brain still reports brain-wide orphan_ratio. Thread the explicit `--source` into the remote doctor request + have the server-side check honor it. Priority: P3 (most users run doctor locally).
|
||||
|
||||
- [ ] **v0.42+: widen `check-test-real-names.sh` BANNED_NAMES to catch real-name reintroduction in tests + src.** v0.41.29.0 scrubbed pre-existing real names (`Garry Tan`, `Alex Graveley`) from `bold-paren-time`'s `test_positive` (and the new `bold-name-no-time` samples), but no automated guard caught them: `check-test-real-names.sh` only scans `test/**` and its BANNED_NAMES list doesn't include `garry tan`; `check-fixture-privacy.sh` only scans `test/fixtures/conversation-formats/`. Add `garry tan` / `garrytan` (and consider extending the scan to `src/core/conversation-parser/builtins.ts` test samples) so future reintroductions fail CI. Priority: P3 (hardening).
|
||||
|
||||
## v0.41.28.0 #1570 instrument-then-fix follow-ups (v0.41.28+ / v0.42+)
|
||||
|
||||
Filed from the v0.41.28.0 plan-eng-review after the codex outside-voice
|
||||
|
||||
+2
-2
File diff suppressed because one or more lines are too long
+1
-1
@@ -141,5 +141,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.41.28.0"
|
||||
"version": "0.41.29.0"
|
||||
}
|
||||
|
||||
+36
-6
@@ -3080,6 +3080,19 @@ export async function buildChecks(
|
||||
// invoked later in the function.
|
||||
const scope: 'all' | 'brain' = args.includes('--scope=brain') ? 'brain' : 'all';
|
||||
|
||||
// v0.41.29.0: explicit `--source <id>` scopes the `orphan_ratio` check to one
|
||||
// source. EXPLICIT-ONLY by design — a raw flag parse, NOT resolveSourceWithTier.
|
||||
// The tier resolver would pick a default source when `--source` is absent and
|
||||
// silently scope a bare `gbrain doctor` to one source; we want bare doctor to
|
||||
// stay brain-wide. Only `orphan_ratio` consumes this for now (other checks
|
||||
// staying brain-wide is a separate, larger change — see TODOS.md).
|
||||
let orphanRatioSourceId: string | undefined;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--source' && i + 1 < args.length) {
|
||||
orphanRatioSourceId = args[++i] || undefined;
|
||||
}
|
||||
}
|
||||
|
||||
const checks: Check[] = [];
|
||||
let autoFixReport: AutoFixReport | null = null;
|
||||
|
||||
@@ -4543,22 +4556,39 @@ export async function buildChecks(
|
||||
// show high orphan ratio; not actionable signal).
|
||||
// Warn at >0.5; fail at >0.8. Both states recommend
|
||||
// `gbrain extract links --by-mention` as the fix.
|
||||
// v0.41.29.0: explicit `--source <id>` scopes this check to one source
|
||||
// (orphanRatioSourceId, parsed at the top of buildChecks). The entity-count
|
||||
// gate + getOrphansData both scope to it; messages name the source. Bare
|
||||
// doctor (no --source) stays brain-wide.
|
||||
progress.heartbeat('orphan_ratio');
|
||||
try {
|
||||
const { getOrphansData } = await import('./orphans.ts');
|
||||
const srcId = orphanRatioSourceId;
|
||||
const inSource = srcId ? ` in source '${srcId}'` : '';
|
||||
const entityCount = (await engine.executeRaw<{ count: number }>(
|
||||
"SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization') AND deleted_at IS NULL",
|
||||
`SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization') AND deleted_at IS NULL${srcId ? ' AND source_id = $1' : ''}`,
|
||||
srcId ? [srcId] : [],
|
||||
))[0]?.count ?? 0;
|
||||
if (entityCount < 100) {
|
||||
// Brain-wide (no --source): <100 entities is vacuous — small brains
|
||||
// naturally show a high orphan ratio; not actionable signal. Skip.
|
||||
if (entityCount < 100 && !srcId) {
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'ok',
|
||||
message: `Vacuous: ${entityCount} entity pages (<100). Orphan ratio not meaningful at this scale.`,
|
||||
});
|
||||
} else {
|
||||
const data = await getOrphansData(engine, { includePseudo: false });
|
||||
// F7 (Codex): under EXPLICIT --source, an operator deliberately asked
|
||||
// about one source — answer it even below 100 entities, with a
|
||||
// low-scale caveat, instead of swallowing a real per-source failure
|
||||
// (e.g. 80 fully-orphaned entity pages) behind a vacuous "ok".
|
||||
const data = await getOrphansData(engine, { includePseudo: false, sourceId: srcId });
|
||||
const ratio = data.total_linkable > 0 ? data.total_orphans / data.total_linkable : 0;
|
||||
const pct = (ratio * 100).toFixed(0);
|
||||
const caveat =
|
||||
entityCount < 100
|
||||
? ` — low scale (${entityCount} entity pages <100), interpret with caution`
|
||||
: '';
|
||||
const hint =
|
||||
'Run: gbrain extract links --by-mention (auto-links entity mentions in body text). ' +
|
||||
'Run gbrain orphans for the list.';
|
||||
@@ -4566,19 +4596,19 @@ export async function buildChecks(
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'fail',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`,
|
||||
message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links)${caveat}. ${hint}`,
|
||||
});
|
||||
} else if (ratio > 0.5) {
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'warn',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`,
|
||||
message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links)${caveat}. ${hint}`,
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'ok',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages)`,
|
||||
message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages)${caveat}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
+53
-8
@@ -127,9 +127,16 @@ export async function queryOrphanPages(
|
||||
*/
|
||||
export async function findOrphans(
|
||||
engine: BrainEngine,
|
||||
opts: { includePseudo?: boolean } = {},
|
||||
opts: { includePseudo?: boolean; sourceId?: string; sourceIds?: string[] } = {},
|
||||
): Promise<OrphanResult> {
|
||||
const includePseudo = !!opts.includePseudo;
|
||||
// v0.41.29.0: `sourceId` (scalar, from `--source` + single-source MCP
|
||||
// clients) or `sourceIds` (federated, from `allowedSources` MCP clients)
|
||||
// scopes the candidate set. `sourceIds` wins when both set (mirrors
|
||||
// sourceScopeOpts precedence).
|
||||
const sourceId = opts.sourceId;
|
||||
const sourceIds =
|
||||
opts.sourceIds && opts.sourceIds.length > 0 ? opts.sourceIds : undefined;
|
||||
// The NOT EXISTS anti-join over pages × links can take seconds on 50K-page
|
||||
// brains. Heartbeat every second so agents see the scan is alive. Keyset
|
||||
// pagination was considered and rejected: without an index on
|
||||
@@ -140,16 +147,40 @@ export async function findOrphans(
|
||||
const stopHb = startHeartbeat(progress, 'scanning pages for missing inbound links…');
|
||||
let allOrphans: { slug: string; title: string; domain: string | null }[];
|
||||
let total: number;
|
||||
let excludedAll: number;
|
||||
try {
|
||||
allOrphans = await engine.findOrphanPages();
|
||||
// Count total pages in DB for the summary line
|
||||
const stats = await engine.getStats();
|
||||
total = stats.page_count;
|
||||
allOrphans = await engine.findOrphanPages(
|
||||
sourceIds ? { sourceIds } : sourceId ? { sourceId } : undefined,
|
||||
);
|
||||
// v0.41.29.0 (Codex F6): correct the `total_linkable` denominator.
|
||||
// Enumerate ALL live pages (scoped) and count excluded-by-slug across
|
||||
// the WHOLE set — not just among orphans. The old
|
||||
// `total - excludedOrphans` left excluded NON-orphan pages (e.g. a
|
||||
// `test/` page that HAS inbound links) in the denominator, inflating
|
||||
// total_linkable and suppressing orphan warnings. `getAllSlugs` is NOT
|
||||
// used here because it does not filter soft-deleted rows; `total` must
|
||||
// match `findOrphanPages`'s `deleted_at IS NULL` candidate universe.
|
||||
let scopeClause = '';
|
||||
const liveParams: unknown[] = [];
|
||||
if (sourceIds) {
|
||||
liveParams.push(sourceIds);
|
||||
scopeClause = ` AND source_id = ANY($${liveParams.length}::text[])`;
|
||||
} else if (sourceId) {
|
||||
liveParams.push(sourceId);
|
||||
scopeClause = ` AND source_id = $${liveParams.length}`;
|
||||
}
|
||||
const liveRows = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT slug FROM pages WHERE deleted_at IS NULL${scopeClause}`,
|
||||
liveParams,
|
||||
);
|
||||
total = liveRows.length;
|
||||
excludedAll = includePseudo
|
||||
? 0
|
||||
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug) ? 1 : 0), 0);
|
||||
} finally {
|
||||
stopHb();
|
||||
progress.finish();
|
||||
}
|
||||
const _totalPages = allOrphans.length; // pages with no inbound links (preserved for ref)
|
||||
|
||||
const filtered = includePseudo
|
||||
? allOrphans
|
||||
@@ -166,7 +197,10 @@ export async function findOrphans(
|
||||
return {
|
||||
orphans,
|
||||
total_orphans: orphans.length,
|
||||
total_linkable: filtered.length + (total - allOrphans.length),
|
||||
// v0.41.29.0 (Codex F6): denominator = live pages minus ALL excluded
|
||||
// pages (orphan or not), so excluded pages with inbound links no longer
|
||||
// inflate it.
|
||||
total_linkable: total - excludedAll,
|
||||
total_pages: total,
|
||||
excluded,
|
||||
};
|
||||
@@ -224,6 +258,16 @@ export async function runOrphans(engine: BrainEngine, args: string[]) {
|
||||
const json = args.includes('--json');
|
||||
const count = args.includes('--count');
|
||||
const includePseudo = args.includes('--include-pseudo');
|
||||
// v0.41.29.0: explicit `--source <id>` scopes the orphan scan to one
|
||||
// source. Omitted → brain-wide (unchanged). Raw explicit-flag parse on
|
||||
// purpose — NOT resolveSourceWithTier, which would pick a default source
|
||||
// when the flag is absent and silently scope a bare `gbrain orphans`.
|
||||
let sourceId: string | undefined;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--source' && i + 1 < args.length) {
|
||||
sourceId = args[++i] || undefined;
|
||||
}
|
||||
}
|
||||
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log(`Usage: gbrain orphans [options]
|
||||
@@ -234,6 +278,7 @@ Options:
|
||||
--json Output as JSON (for agent consumption)
|
||||
--count Output just the number of orphans
|
||||
--include-pseudo Include auto-generated and pseudo pages in results
|
||||
--source <id> Scope the scan to one brain source (default: brain-wide)
|
||||
--help, -h Show this help
|
||||
|
||||
Output (default): grouped by domain, sorted alphabetically within each group
|
||||
@@ -242,7 +287,7 @@ Summary line: N orphans out of M linkable pages (K total; K-M excluded)
|
||||
return;
|
||||
}
|
||||
|
||||
const result = await findOrphans(engine, { includePseudo });
|
||||
const result = await findOrphans(engine, { includePseudo, sourceId });
|
||||
|
||||
if (count) {
|
||||
console.log(String(result.total_orphans));
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
/**
|
||||
* v0.41.16.0 — Built-in conversation parser pattern registry.
|
||||
*
|
||||
* Twelve hand-vetted patterns covering the chat-export formats this
|
||||
* Fourteen hand-vetted patterns covering the chat-export formats this
|
||||
* codebase is most likely to encounter. Each pattern's regex was
|
||||
* derived from a public format reference (source_doc field) so future
|
||||
* maintainers can verify against the wild shape.
|
||||
@@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string {
|
||||
return stripped || raw.trim();
|
||||
}
|
||||
|
||||
/** The 12 hand-vetted built-in patterns. */
|
||||
/** The 14 hand-vetted built-in patterns. */
|
||||
export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
||||
// -------------------------------------------------------------------
|
||||
// INLINE-DATE patterns (date in every line; less ambiguous; tried first).
|
||||
@@ -158,9 +158,9 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
||||
multi_line: false,
|
||||
quick_reject: /^\*\*/,
|
||||
test_positive: [
|
||||
'**Garry Tan** (00:00): hello world',
|
||||
'**Alice Example** (00:00): hello world',
|
||||
'**Participant 2** (02:22): response here',
|
||||
'**Alex Graveley** (15:09): That’s exactly right.',
|
||||
'**Bob Example** (15:09): That’s exactly right.',
|
||||
'**Participant 1** (00:00:00): hello world with seconds',
|
||||
'**Participant 2** (01:23:45): mid-meeting line',
|
||||
],
|
||||
@@ -178,6 +178,75 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
||||
'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)',
|
||||
},
|
||||
|
||||
{
|
||||
// Modern meeting-transcription tools (Circleback, Granola, Zoom)
|
||||
// emit `**Speaker Name:** message text` with NO per-line
|
||||
// timestamp. Every other built-in requires a time anchor, so this
|
||||
// shape scored ~0.002 (one stray line in a long page) and fell
|
||||
// below SCORING_MIN_ACCEPTANCE — parsing to zero messages and
|
||||
// extracting zero conversation-facts. This additive pattern fixes
|
||||
// that: speaker is captured inside the bold markers (`**Name:**`),
|
||||
// there is no time capture, and date_source='frontmatter' with
|
||||
// hour_group undefined routes through parse.ts's no-time branch
|
||||
// (same convention as irc-classic) — every message anchors at
|
||||
// 00:00:00 of the page's frontmatter date. No wall-clock time is
|
||||
// fabricated; intra-day ordering is preserved by line order.
|
||||
//
|
||||
// NON-SHADOW GUARANTEE (read carefully — the safety is in the
|
||||
// REGEX, not the declaration order). parse.ts scores every
|
||||
// candidate independently; declaration index is ONLY the
|
||||
// tie-break. This pattern cannot steal `**Name** (time):` from
|
||||
// bold-paren-time because its regex requires the colon INSIDE the
|
||||
// bold markers (`**Name:**`), which the paren-time shape (colon
|
||||
// OUTSIDE: `**Name** (time):`) never has. The `(?!\[)` lookahead
|
||||
// additionally rejects telegram-bracket's `**[18:37] Name:**`
|
||||
// shape so that disabling telegram-bracket yields an honest
|
||||
// no_match instead of capturing speaker="[18:37] Name" at midnight.
|
||||
//
|
||||
// BROAD-REGEX GUARD (score_full_body): `**Label:** text` is a
|
||||
// common prose idiom (`**Note:**`, `**Owner:**`). A notes page
|
||||
// with a few bold labels clustered in its first 10 lines would
|
||||
// score 0.3 on the head pass, skip the rescore, and clear the
|
||||
// 0.05 floor. score_full_body forces full-body density scoring
|
||||
// before acceptance so such a page falls to no_match.
|
||||
id: 'bold-name-no-time',
|
||||
origin: 'builtin',
|
||||
// Matches: **Speaker Name:** message text (colon INSIDE bold,
|
||||
// speaker must not start with `[` — see lookahead rationale above).
|
||||
regex: /^\*\*(?!\[)(.+?):\*\*\s*(.*)$/,
|
||||
captures: {
|
||||
speaker_group: 1,
|
||||
text_group: 2,
|
||||
},
|
||||
date_source: 'frontmatter',
|
||||
time_format: '24h',
|
||||
timezone_policy: 'utc_assumed_with_warn',
|
||||
multi_line: false,
|
||||
quick_reject: /^\*\*/,
|
||||
score_full_body: true,
|
||||
test_positive: [
|
||||
'**Alice Example:** Okay, start on.',
|
||||
'**Participant 2:** he tried to reset it remotely the other night.',
|
||||
'**Bob Example:** That is exactly right.',
|
||||
],
|
||||
test_negative: [
|
||||
// bold-paren-time shape (colon OUTSIDE bold) MUST fall through:
|
||||
'**Alice** (00:00): text',
|
||||
// imessage-slack shape MUST fall through:
|
||||
'**Alice Example** (2024-03-15 9:00 AM): iMessage shape',
|
||||
// Bold but no colon at all:
|
||||
'**Alice** hello world',
|
||||
// No bold markers:
|
||||
'Alice: plain no bold',
|
||||
// telegram-bracket shape (timestamp INSIDE bold) MUST NOT match
|
||||
// — the `(?!\[)` lookahead rejects it so disabling
|
||||
// telegram-bracket yields no_match, not speaker="[18:37] Alice":
|
||||
'**[18:37] \u{1f464} Alice:** hello',
|
||||
],
|
||||
source_doc:
|
||||
'Circleback / Granola / Zoom meeting-transcript export shape: `**Speaker:** text` with no per-line timestamp',
|
||||
},
|
||||
|
||||
{
|
||||
id: 'telegram-text-export',
|
||||
origin: 'builtin',
|
||||
|
||||
@@ -479,6 +479,7 @@ export function parseConversation(
|
||||
// 10 head lines matched"; chat-only pages still score 1.0 and skip
|
||||
// the fallback. Re-score every candidate against the full body,
|
||||
// pre-splitting ONCE to avoid 12 redundant body splits.
|
||||
let fullBodyScored = false;
|
||||
if (scored[0].score < SCORING_HEAD_TRIGGER_THRESHOLD) {
|
||||
const allLines = getNonBlankLines(body);
|
||||
scored = candidates.map((entry) => ({
|
||||
@@ -487,6 +488,7 @@ export function parseConversation(
|
||||
priority: priorityOf(entry.id),
|
||||
}));
|
||||
sortScored(scored);
|
||||
fullBodyScored = true;
|
||||
// NOTE: patterns_scored stays as scored.length (= candidate
|
||||
// count, typically 12) even when the fallback runs — the
|
||||
// diagnostic reports "candidates considered" not "scoring
|
||||
@@ -496,6 +498,21 @@ export function parseConversation(
|
||||
const top = scored[0];
|
||||
const patternsScored = scored.length;
|
||||
|
||||
// v0.41.29.0 (Codex F1): broad no-time patterns (`bold-name-no-time`,
|
||||
// regex matches any `**Label:** text`) can win the HEAD pass on a
|
||||
// prose notes page whose first 10 lines happen to hold 3+ bold labels
|
||||
// (head score 0.3, NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so the
|
||||
// full-body fallback above never ran). 0.3 clears the 0.05 floor and
|
||||
// the page mis-parses as a conversation. When the winner is flagged
|
||||
// `score_full_body` and we have NOT already scored full-body, recompute
|
||||
// its score over the whole document so the floor judges true density.
|
||||
// We do NOT re-pick the winner: the head pick is already the best
|
||||
// candidate; this only tightens its acceptance (a real transcript
|
||||
// stays ~1.0; a 3/200-label notes page drops to ~0.015 → no_match).
|
||||
if (top.entry.score_full_body && !fullBodyScored) {
|
||||
top.score = scoreFromLines(getNonBlankLines(body), top.entry);
|
||||
}
|
||||
|
||||
// Minimum acceptance floor (closes Codex P1 #2): an essay with
|
||||
// one stray `**Name** (date time):` line scores ~1/300 ≈ 0.003 —
|
||||
// below the 5% floor we stay no_match instead of returning a
|
||||
|
||||
@@ -170,6 +170,19 @@ export interface PatternEntry {
|
||||
* uses `DEFAULT_SPEAKER_CLEAN`.
|
||||
*/
|
||||
speaker_clean?: RegExp;
|
||||
/**
|
||||
* v0.41.29.0: when true, the orchestrator recomputes this pattern's
|
||||
* acceptance score over the FULL body (not just the head window)
|
||||
* before applying SCORING_MIN_ACCEPTANCE. Set on broad no-time
|
||||
* patterns (`bold-name-no-time`) whose regex matches a common prose
|
||||
* idiom (`**Label:** text`): without it, a notes page with a few bold
|
||||
* labels CLUSTERED in its first 10 lines scores 0.3 on the head pass,
|
||||
* skips the `< SCORING_HEAD_TRIGGER_THRESHOLD` rescore, and clears the
|
||||
* 0.05 floor — mis-parsing prose as a conversation. Full-body scoring
|
||||
* drops that page below the floor (3/200 ≈ 0.015 → no_match) while a
|
||||
* real transcript stays ~1.0.
|
||||
*/
|
||||
score_full_body?: boolean;
|
||||
/** D7: module-load validation — known-positive sample lines. */
|
||||
test_positive: string[];
|
||||
/** D7: module-load validation — known-negative sample lines. */
|
||||
|
||||
+13
-2
@@ -1168,12 +1168,23 @@ export interface BrainEngine {
|
||||
*/
|
||||
getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, number>>;
|
||||
/**
|
||||
* Return every page with no inbound links (from any source).
|
||||
* Return every page with no inbound links.
|
||||
* Domain comes from the frontmatter `domain` field (null if unset).
|
||||
* The caller filters pseudo-pages + derives display domain.
|
||||
* Used by `gbrain orphans` and `runCycle`'s orphan sweep phase.
|
||||
*
|
||||
* v0.41.29.0: scopes the CANDIDATE set to one source (`sourceId`, from
|
||||
* `gbrain doctor/orphans --source` + single-source MCP clients) or a
|
||||
* federated set (`sourceIds`, from `allowedSources` MCP clients). The
|
||||
* inbound-link side is NOT scoped — a page in source X linked FROM
|
||||
* source Y is genuinely reachable, so it is not an orphan of X. Omit
|
||||
* `opts` for the brain-wide behavior (unchanged). When both are set,
|
||||
* `sourceIds` wins (mirrors `sourceScopeOpts` precedence).
|
||||
*/
|
||||
findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>>;
|
||||
findOrphanPages(opts?: {
|
||||
sourceId?: string;
|
||||
sourceIds?: string[];
|
||||
}): Promise<Array<{ slug: string; title: string; domain: string | null }>>;
|
||||
|
||||
// Tags
|
||||
/**
|
||||
|
||||
+10
-1
@@ -2756,7 +2756,16 @@ const find_orphans: Operation = {
|
||||
scope: 'read',
|
||||
handler: async (ctx, p) => {
|
||||
const { findOrphans } = await import('../commands/orphans.ts');
|
||||
return findOrphans(ctx.engine, { includePseudo: (p.include_pseudo as boolean) || false });
|
||||
// v0.41.29.0 (Codex F8): scope by the caller's source (ctx.sourceId /
|
||||
// ctx.auth.allowedSources) via the canonical sourceScopeOpts ladder.
|
||||
// Pre-fix, find_orphans returned brain-wide orphans regardless of a
|
||||
// source-bound OAuth client's scope — a read leak in the v0.34.1
|
||||
// source-isolation class. Local CLI callers route through `gbrain
|
||||
// orphans --source` instead (ctx.remote === false → empty scope here).
|
||||
return findOrphans(ctx.engine, {
|
||||
includePseudo: (p.include_pseudo as boolean) || false,
|
||||
...sourceScopeOpts(ctx),
|
||||
});
|
||||
},
|
||||
cliHints: { name: 'orphans', hidden: true },
|
||||
};
|
||||
|
||||
@@ -2791,12 +2791,30 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return out;
|
||||
}
|
||||
|
||||
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
async findOrphanPages(opts?: {
|
||||
sourceId?: string;
|
||||
sourceIds?: string[];
|
||||
}): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
// Soft-delete filter on BOTH sides:
|
||||
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
||||
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
||||
// Without the link-source filter, a live page can hide from orphan results purely
|
||||
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
||||
//
|
||||
// v0.41.29.0: scope ONLY the candidate side (`p.source_id`) when opts.sourceId
|
||||
// is set. The inbound-link NOT EXISTS deliberately counts links from ANY source:
|
||||
// a page in source X linked FROM source Y is reachable, so NOT an orphan of X.
|
||||
// Do NOT add `src.source_id = p.source_id` here — that is the stricter
|
||||
// intra-source-only definition we deliberately reject.
|
||||
let sourceFilter = '';
|
||||
const params: unknown[] = [];
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
params.push(opts.sourceIds);
|
||||
sourceFilter = `AND p.source_id = ANY($${params.length}::text[])`;
|
||||
} else if (opts?.sourceId) {
|
||||
params.push(opts.sourceId);
|
||||
sourceFilter = `AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug,
|
||||
@@ -2804,6 +2822,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
p.frontmatter->>'domain' AS domain
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
${sourceFilter}
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM links l
|
||||
@@ -2811,7 +2830,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
WHERE l.to_page_id = p.id
|
||||
AND src.deleted_at IS NULL
|
||||
)
|
||||
ORDER BY p.slug`
|
||||
ORDER BY p.slug`,
|
||||
params
|
||||
);
|
||||
return rows as Array<{ slug: string; title: string; domain: string | null }>;
|
||||
}
|
||||
|
||||
@@ -2857,13 +2857,28 @@ export class PostgresEngine implements BrainEngine {
|
||||
return out;
|
||||
}
|
||||
|
||||
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
async findOrphanPages(opts?: {
|
||||
sourceId?: string;
|
||||
sourceIds?: string[];
|
||||
}): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
const sql = this.sql;
|
||||
// Soft-delete filter on BOTH sides:
|
||||
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
||||
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
||||
// Without the link-source filter, a live page can hide from orphan results purely
|
||||
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
||||
//
|
||||
// v0.41.29.0: scope ONLY the candidate side (`p.source_id`) when opts.sourceId
|
||||
// is set. The inbound-link NOT EXISTS deliberately counts links from ANY source:
|
||||
// a page in source X linked FROM source Y is reachable, so NOT an orphan of X.
|
||||
// Do NOT add `src.source_id = p.source_id` here — that would be the stricter
|
||||
// intra-source-only definition we deliberately reject.
|
||||
const sourceFilter =
|
||||
opts?.sourceIds && opts.sourceIds.length > 0
|
||||
? sql`AND p.source_id = ANY(${opts.sourceIds}::text[])`
|
||||
: opts?.sourceId
|
||||
? sql`AND p.source_id = ${opts.sourceId}`
|
||||
: sql``;
|
||||
const rows = await sql`
|
||||
SELECT
|
||||
p.slug,
|
||||
@@ -2871,6 +2886,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
p.frontmatter->>'domain' AS domain
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
${sourceFilter}
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM links l
|
||||
|
||||
@@ -403,21 +403,21 @@ describe('scorePatternFull — full-body scoring (v0.41.18+ Codex P1 #1)', () =>
|
||||
describe('bold-paren-time pattern (Circleback meeting transcripts)', () => {
|
||||
test('matches **Speaker** (HH:MM): text with frontmatter date', () => {
|
||||
const body = [
|
||||
'**Garry Tan** (00:00): Hey, can you hear me?',
|
||||
'**Alice Example** (00:00): Hey, can you hear me?',
|
||||
'**Participant 2** (02:22): Yeah, just joined.',
|
||||
'**Garry Tan** (15:09): That makes sense.',
|
||||
'**Alice Example** (15:09): That makes sense.',
|
||||
].join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
|
||||
expect(r.phase).toBe('regex_match');
|
||||
expect(r.matched_pattern_id).toBe('bold-paren-time');
|
||||
expect(r.messages).toHaveLength(3);
|
||||
expect(r.messages[0]).toEqual({
|
||||
speaker: 'Garry Tan',
|
||||
speaker: 'Alice Example',
|
||||
timestamp: '2026-03-19T00:00:00Z',
|
||||
text: 'Hey, can you hear me?',
|
||||
});
|
||||
expect(r.messages[2]).toEqual({
|
||||
speaker: 'Garry Tan',
|
||||
speaker: 'Alice Example',
|
||||
timestamp: '2026-03-19T15:09:00Z',
|
||||
text: 'That makes sense.',
|
||||
});
|
||||
@@ -478,6 +478,113 @@ describe('bold-paren-time pattern (Circleback meeting transcripts)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// bold-name-no-time pattern (Circleback / Granola / Zoom transcripts with NO
|
||||
// per-line timestamp — `**Speaker:** text`). Additive pattern; the colon
|
||||
// inside the bold markers + the `(?!\[)` lookahead are what keep it from
|
||||
// shadowing bold-paren-time / telegram-bracket (NOT declaration order).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('bold-name-no-time pattern (Circleback/Granola/Zoom, no timestamp)', () => {
|
||||
test('parses **Speaker:** text transcript with frontmatter date anchor', () => {
|
||||
const body = [
|
||||
'**Alice Example:** Okay, start on. And then weirdly like zoom doesn’t...',
|
||||
'**Participant 2:** he tried to reset it remotely the other night. Let me ask him.',
|
||||
'**Alice Example:** I mean it’s really just like we need to get zoom to fix this.',
|
||||
'**Participant 2:** Okay, let me.',
|
||||
].join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
|
||||
expect(r.phase).toBe('regex_match');
|
||||
expect(r.matched_pattern_id).toBe('bold-name-no-time');
|
||||
expect(r.messages).toHaveLength(4);
|
||||
expect(r.messages[0]).toEqual({
|
||||
speaker: 'Alice Example',
|
||||
timestamp: '2026-05-28T00:00:00Z',
|
||||
text: 'Okay, start on. And then weirdly like zoom doesn’t...',
|
||||
});
|
||||
expect(r.messages[1].speaker).toBe('Participant 2');
|
||||
expect(r.messages[1].text).toBe(
|
||||
'he tried to reset it remotely the other night. Let me ask him.',
|
||||
);
|
||||
expect(r.messages[3]).toEqual({
|
||||
speaker: 'Participant 2',
|
||||
timestamp: '2026-05-28T00:00:00Z',
|
||||
text: 'Okay, let me.',
|
||||
});
|
||||
// No-time pattern anchors at 00:00:00 of the frontmatter date
|
||||
// (same convention as irc-classic). No wall-clock time fabricated.
|
||||
});
|
||||
|
||||
test('scores above the 0.05 floor on a pure bold-name transcript (epoch default)', () => {
|
||||
const body = [
|
||||
'**Alice Example:** line one',
|
||||
'**Participant 2:** line two',
|
||||
'**Alice Example:** line three',
|
||||
].join('\n');
|
||||
const r = parseConversation(body);
|
||||
expect(r.phase).toBe('regex_match');
|
||||
expect(r.matched_pattern_id).toBe('bold-name-no-time');
|
||||
expect(r.messages).toHaveLength(3);
|
||||
// No fallbackDate → epoch default, still anchors at 00:00:00.
|
||||
expect(r.messages[0].timestamp).toBe('1970-01-01T00:00:00Z');
|
||||
});
|
||||
|
||||
// REGRESSION: must NOT shadow bold-paren-time. A `**Name** (00:00): text`
|
||||
// line has its colon OUTSIDE the bold markers, so it must still parse via
|
||||
// bold-paren-time (the safety is the regex, not declaration order).
|
||||
test('REGRESSION: **Speaker** (HH:MM): text still matches bold-paren-time', () => {
|
||||
const body = [
|
||||
'**Alice Example** (00:00): Hey, can you hear me?',
|
||||
'**Participant 2** (02:22): Yeah, just joined.',
|
||||
].join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
|
||||
expect(r.phase).toBe('regex_match');
|
||||
expect(r.matched_pattern_id).toBe('bold-paren-time');
|
||||
expect(r.matched_pattern_id).not.toBe('bold-name-no-time');
|
||||
expect(r.messages).toHaveLength(2);
|
||||
expect(r.messages[0].timestamp).toBe('2026-03-19T00:00:00Z');
|
||||
});
|
||||
|
||||
// REGRESSION: a pure telegram-bracket transcript has the colon inside the
|
||||
// bold markers BUT the speaker starts with `[`, so the `(?!\[)` lookahead
|
||||
// rejects it AND telegram-bracket (lower declaration index) wins anyway.
|
||||
test('REGRESSION: telegram-bracket transcript still matches telegram-bracket', () => {
|
||||
const body = [
|
||||
'**[18:37] \u{1f464} Alice:** one',
|
||||
'**[18:38] \u{1f464} Bob:** two',
|
||||
].join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
|
||||
expect(r.matched_pattern_id).toBe('telegram-bracket');
|
||||
});
|
||||
|
||||
// F1 (Codex): the broad regex matches any `**Label:** text`. A prose notes
|
||||
// page with bold labels CLUSTERED in its first 10 lines scores 0.3 on the
|
||||
// head pass (3/10) — NOT < SCORING_HEAD_TRIGGER_THRESHOLD (0.3), so the
|
||||
// full-body fallback never fires — and that 0.3 clears the 0.05 floor.
|
||||
// Without score_full_body this page would mis-parse as a conversation.
|
||||
// score_full_body recomputes the winner over the FULL body: 3 labels /
|
||||
// 83 lines ≈ 0.036 < 0.05 → no_match. This is the exact bug Codex caught
|
||||
// and the test the naive "0.05 floor protects us" assumption would fail.
|
||||
test('F1: prose with bold labels clustered in the head returns no_match', () => {
|
||||
const lines = [
|
||||
'**Attendees:** Alice Example, Bob Example, Participant 2',
|
||||
'**Date:** 2026-05-28',
|
||||
'**Goal:** decide on the Q3 roadmap and unblock the vendor migration',
|
||||
];
|
||||
// 80 plain prose lines (no bold-label shape) → 83 total, only 3 match.
|
||||
// First 10 lines = 3 labels + 7 prose → head score 0.3 (skips rescore).
|
||||
// Full body = 3/83 ≈ 0.036 < 0.05 floor.
|
||||
for (let i = 0; i < 80; i++) {
|
||||
lines.push(
|
||||
`This is an ordinary prose sentence number ${i} describing the meeting in detail.`,
|
||||
);
|
||||
}
|
||||
const body = lines.join('\n');
|
||||
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
|
||||
expect(r.phase).toBe('no_match');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// parseConversation — full-body fallback (v0.41.18+ #1533 + Codex P1 #1, #2, #8)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -363,4 +363,53 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
|
||||
expect(pgMap.get('wiki/rsp-missing.md')).toBeUndefined();
|
||||
expect(pgliteMap.get('wiki/rsp-missing.md')).toBeUndefined();
|
||||
});
|
||||
|
||||
// v0.41.29.0 — findOrphanPages source scoping parity. Real Postgres
|
||||
// coverage for the postgres.js `sql` scalar fragment + `= ANY(${arr}::text[])`
|
||||
// array binding (a documented footgun class — the jsonb double-encode saga).
|
||||
// PGLite logic is pinned in test/orphans-source-scope.test.ts; this asserts
|
||||
// the Postgres SQL produces the same scoped sets. Cross-source inbound
|
||||
// (src-b → src-a) must NOT make the target an orphan of src-a (A2).
|
||||
test('v0.41.29.0 findOrphanPages source scoping parity (scalar + federated)', async () => {
|
||||
const srcSql = `INSERT INTO sources (id, name, config) VALUES ($1, $1, '{}'::jsonb) ON CONFLICT DO NOTHING`;
|
||||
const pageSql = `
|
||||
INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter)
|
||||
VALUES ($1, $2, 'person', 't', 'b', '', '{}'::jsonb)
|
||||
ON CONFLICT (source_id, slug) DO NOTHING
|
||||
`;
|
||||
for (const eng of [pgEngine, pgliteEngine]) {
|
||||
await eng.executeRaw(srcSql, ['orphan-src-a']);
|
||||
await eng.executeRaw(srcSql, ['orphan-src-b']);
|
||||
await eng.executeRaw(pageSql, ['orphan-src-a', 'people/op-orphan-a']);
|
||||
await eng.executeRaw(pageSql, ['orphan-src-a', 'people/op-target-a']);
|
||||
await eng.executeRaw(pageSql, ['orphan-src-b', 'people/op-linker-b']);
|
||||
// Cross-source inbound: src-b page → src-a target (A2).
|
||||
await eng.addLink(
|
||||
'people/op-linker-b', 'people/op-target-a', '', 'mentions', 'markdown',
|
||||
undefined, undefined, { fromSourceId: 'orphan-src-b', toSourceId: 'orphan-src-a' },
|
||||
);
|
||||
}
|
||||
|
||||
const scoped = async (eng: BrainEngine, opts: { sourceId?: string; sourceIds?: string[] }) =>
|
||||
(await eng.findOrphanPages(opts)).map(r => r.slug).filter(s => s.startsWith('people/op-')).sort();
|
||||
|
||||
// Scalar scope to src-a: op-orphan-a is an orphan; op-target-a is saved
|
||||
// by the cross-source inbound (A2). Parity on both engines.
|
||||
const pgA = await scoped(pgEngine, { sourceId: 'orphan-src-a' });
|
||||
const pgliteA = await scoped(pgliteEngine, { sourceId: 'orphan-src-a' });
|
||||
expect(pgA).toEqual(['people/op-orphan-a']);
|
||||
expect(pgliteA).toEqual(pgA);
|
||||
|
||||
// Scalar scope to src-b.
|
||||
const pgB = await scoped(pgEngine, { sourceId: 'orphan-src-b' });
|
||||
const pgliteB = await scoped(pgliteEngine, { sourceId: 'orphan-src-b' });
|
||||
expect(pgB).toEqual(['people/op-linker-b']);
|
||||
expect(pgliteB).toEqual(pgB);
|
||||
|
||||
// Federated array scope (= ANY binding) → union.
|
||||
const pgFed = await scoped(pgEngine, { sourceIds: ['orphan-src-a', 'orphan-src-b'] });
|
||||
const pgliteFed = await scoped(pgliteEngine, { sourceIds: ['orphan-src-a', 'orphan-src-b'] });
|
||||
expect(pgFed).toEqual(['people/op-linker-b', 'people/op-orphan-a']);
|
||||
expect(pgliteFed).toEqual(pgFed);
|
||||
});
|
||||
});
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,2 @@
|
||||
{"fixture_id":"bold-name-no-time-001","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** Okay, start on. And then weirdly like zoom doesn’t work.\n**Bob Example:** he tried to reset it remotely the other night. Let me ask him.\n**Alice Example:** I mean it’s really just like we need to get zoom to fix this.\n**Bob Example:** Okay, let me.","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||
{"fixture_id":"bold-name-no-time-002","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** can you hear me now\n**Participant 2:** yeah loud and clear\n**Alice Example:** great, let us start the review","expected_messages":3,"expected_participants":["Alice Example","Participant 2"]}
|
||||
@@ -0,0 +1,213 @@
|
||||
/**
|
||||
* v0.41.29.0 — orphan source-scoping regression suite.
|
||||
*
|
||||
* Covers three Codex-flagged behaviors of the `--source` / find_orphans
|
||||
* source-scoping wave, all at the layer the bugs live (engine +
|
||||
* getOrphansData + the find_orphans op handler):
|
||||
*
|
||||
* - A2: candidate-only scoping. A page in source X linked FROM source Y
|
||||
* is reachable, so it is NOT an orphan of X (cross-source inbound counts).
|
||||
* - F6: total_linkable denominator. Excluded NON-orphan pages (e.g. a
|
||||
* `templates/` page that HAS inbound links) must NOT inflate the
|
||||
* denominator. Pre-fix the formula `total - excludedOrphans` left them in.
|
||||
* - F8: the find_orphans MCP op handler scopes by ctx.sourceId AND
|
||||
* ctx.auth.allowedSources (the v0.34.1 source-isolation read leak).
|
||||
*
|
||||
* Runs against PGLite in-memory (both engines share the SQL surface; the
|
||||
* Postgres path is pinned separately in test/e2e/multi-source-bug-class).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { findOrphans } from '../src/commands/orphans.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
const page = (title: string) => ({
|
||||
type: 'person' as const,
|
||||
title,
|
||||
compiled_truth: `${title} body.`,
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
|
||||
// Every test in this file is READ-ONLY (findOrphanPages / getOrphansData /
|
||||
// find_orphans op handler). Seed ONCE in beforeAll rather than reset+reseed
|
||||
// per test — the canonical R3/R4 pattern (engine in beforeAll, disconnect in
|
||||
// afterAll) is satisfied, and this cuts the file from ~6min to seconds.
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
await resetPgliteState(engine);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('src-b', 'src-b', '{}'::jsonb) ON CONFLICT DO NOTHING`,
|
||||
);
|
||||
|
||||
// --- default source ---
|
||||
// alice-orphan: no inbound → orphan.
|
||||
await engine.putPage('people/alice-orphan', page('Alice Orphan'), { sourceId: 'default' });
|
||||
// bob-linked: inbound from alice-orphan → NOT orphan.
|
||||
await engine.putPage('people/bob-linked', page('Bob Linked'), { sourceId: 'default' });
|
||||
// cross-target: inbound ONLY from a src-b page → A2: NOT orphan of default.
|
||||
await engine.putPage('people/cross-target', page('Cross Target'), { sourceId: 'default' });
|
||||
// templates/junk-linked: excluded prefix, HAS inbound → not orphan, and must
|
||||
// be excluded from total_linkable (F6).
|
||||
await engine.putPage('templates/junk-linked', page('Junk Template'), { sourceId: 'default' });
|
||||
|
||||
// --- src-b source ---
|
||||
// zara-orphan: no inbound → orphan.
|
||||
await engine.putPage('people/zara-orphan', page('Zara Orphan'), { sourceId: 'src-b' });
|
||||
// src-b-linker: links to default's cross-target; itself has no inbound → orphan.
|
||||
await engine.putPage('people/src-b-linker', page('Src-B Linker'), { sourceId: 'src-b' });
|
||||
|
||||
// --- links ---
|
||||
await engine.addLink(
|
||||
'people/alice-orphan', 'people/bob-linked', '', 'mentions', 'markdown',
|
||||
undefined, undefined, { fromSourceId: 'default', toSourceId: 'default' },
|
||||
);
|
||||
await engine.addLink(
|
||||
'people/alice-orphan', 'templates/junk-linked', '', 'mentions', 'markdown',
|
||||
undefined, undefined, { fromSourceId: 'default', toSourceId: 'default' },
|
||||
);
|
||||
// Cross-source inbound: src-b page → default page (A2).
|
||||
await engine.addLink(
|
||||
'people/src-b-linker', 'people/cross-target', '', 'mentions', 'markdown',
|
||||
undefined, undefined, { fromSourceId: 'src-b', toSourceId: 'default' },
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('findOrphanPages — source scoping (A2)', () => {
|
||||
test('scoped to default: alice-orphan is an orphan; cross-source-linked page is NOT', async () => {
|
||||
const rows = await engine.findOrphanPages({ sourceId: 'default' });
|
||||
const slugs = rows.map(r => r.slug);
|
||||
expect(slugs).toContain('people/alice-orphan');
|
||||
// A2: cross-target has inbound from src-b → reachable → NOT an orphan.
|
||||
expect(slugs).not.toContain('people/cross-target');
|
||||
// bob-linked + templates/junk-linked have intra-source inbound.
|
||||
expect(slugs).not.toContain('people/bob-linked');
|
||||
expect(slugs).not.toContain('templates/junk-linked');
|
||||
// src-b pages must not appear in a default-scoped scan.
|
||||
expect(slugs).not.toContain('people/zara-orphan');
|
||||
expect(slugs).not.toContain('people/src-b-linker');
|
||||
});
|
||||
|
||||
test('scoped to src-b returns a different orphan set', async () => {
|
||||
const rows = await engine.findOrphanPages({ sourceId: 'src-b' });
|
||||
const slugs = rows.map(r => r.slug).sort();
|
||||
expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']);
|
||||
});
|
||||
|
||||
test('unscoped (brain-wide) returns the union', async () => {
|
||||
const rows = await engine.findOrphanPages();
|
||||
const slugs = rows.map(r => r.slug).sort();
|
||||
expect(slugs).toEqual([
|
||||
'people/alice-orphan',
|
||||
'people/src-b-linker',
|
||||
'people/zara-orphan',
|
||||
]);
|
||||
});
|
||||
|
||||
test('federated sourceIds scopes to the union of the given sources', async () => {
|
||||
const rows = await engine.findOrphanPages({ sourceIds: ['default', 'src-b'] });
|
||||
const slugs = rows.map(r => r.slug).sort();
|
||||
expect(slugs).toEqual([
|
||||
'people/alice-orphan',
|
||||
'people/src-b-linker',
|
||||
'people/zara-orphan',
|
||||
]);
|
||||
// Single-element federated array behaves like the scalar scope.
|
||||
const onlyB = await engine.findOrphanPages({ sourceIds: ['src-b'] });
|
||||
expect(onlyB.map(r => r.slug).sort()).toEqual([
|
||||
'people/src-b-linker',
|
||||
'people/zara-orphan',
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('getOrphansData — scoped totals + F6 denominator', () => {
|
||||
test('default scope: counts differ from brain-wide; excluded non-orphan drops from total_linkable (F6)', async () => {
|
||||
const data = await findOrphans(engine, { includePseudo: false, sourceId: 'default' });
|
||||
// 4 live default pages; 1 orphan (alice-orphan).
|
||||
expect(data.total_pages).toBe(4);
|
||||
expect(data.total_orphans).toBe(1);
|
||||
expect(data.orphans.map(o => o.slug)).toEqual(['people/alice-orphan']);
|
||||
// F6: templates/junk-linked is an EXCLUDED page that HAS an inbound link.
|
||||
// It must NOT count in total_linkable. NEW denominator = total(4) -
|
||||
// excludedAll(1) = 3. The pre-fix formula (total - excludedOrphans) left
|
||||
// the excluded non-orphan in and returned 4.
|
||||
expect(data.total_linkable).toBe(3);
|
||||
expect(data.total_linkable).toBe(data.total_pages - 1);
|
||||
});
|
||||
|
||||
test('src-b scope: distinct from default', async () => {
|
||||
const data = await findOrphans(engine, { includePseudo: false, sourceId: 'src-b' });
|
||||
expect(data.total_pages).toBe(2);
|
||||
expect(data.total_orphans).toBe(2);
|
||||
expect(data.total_linkable).toBe(2); // no excluded pages in src-b
|
||||
});
|
||||
|
||||
test('brain-wide F6: excluded non-orphan drops from total_linkable', async () => {
|
||||
const data = await findOrphans(engine, { includePseudo: false });
|
||||
expect(data.total_pages).toBe(6);
|
||||
expect(data.total_orphans).toBe(3);
|
||||
// 6 live pages - 1 excluded (templates/junk-linked) = 5.
|
||||
expect(data.total_linkable).toBe(5);
|
||||
});
|
||||
|
||||
test('includePseudo keeps excluded pages in the denominator', async () => {
|
||||
const data = await findOrphans(engine, { includePseudo: true, sourceId: 'default' });
|
||||
// No exclusion → denominator is the full live set.
|
||||
expect(data.total_linkable).toBe(4);
|
||||
});
|
||||
});
|
||||
|
||||
describe('find_orphans MCP op — F8 source-isolation scope', () => {
|
||||
test('ctx.sourceId scopes results to that source only', async () => {
|
||||
const { operations } = await import('../src/core/operations.ts');
|
||||
const op = operations.find(o => o.name === 'find_orphans');
|
||||
expect(op).toBeDefined();
|
||||
const ctx = {
|
||||
engine,
|
||||
config: { engine: 'pglite' as const },
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
sourceId: 'src-b',
|
||||
};
|
||||
const result = (await op!.handler(ctx as any, {})) as { orphans: { slug: string }[] };
|
||||
const slugs = result.orphans.map(o => o.slug).sort();
|
||||
expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']);
|
||||
// Leak guard: default's orphan must NOT appear.
|
||||
expect(slugs).not.toContain('people/alice-orphan');
|
||||
});
|
||||
|
||||
test('ctx.auth.allowedSources (federated) scopes to the allowed set', async () => {
|
||||
const { operations } = await import('../src/core/operations.ts');
|
||||
const op = operations.find(o => o.name === 'find_orphans');
|
||||
const ctx = {
|
||||
engine,
|
||||
config: { engine: 'pglite' as const },
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
sourceId: 'default', // scalar would scope to default-only...
|
||||
auth: {
|
||||
token: 'test',
|
||||
clientId: 'test',
|
||||
scopes: ['read'],
|
||||
sourceId: 'default',
|
||||
allowedSources: ['src-b'], // ...but the federated array wins.
|
||||
},
|
||||
};
|
||||
const result = (await op!.handler(ctx as any, {})) as { orphans: { slug: string }[] };
|
||||
const slugs = result.orphans.map(o => o.slug).sort();
|
||||
expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']);
|
||||
expect(slugs).not.toContain('people/alice-orphan');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user