mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.41.29.0 feat(conversation-parser): bold-name-no-time builtin + fix(orphans): source-scoped orphan_ratio (supersedes #1613) (#1620)
* feat(conversation-parser): add bold-name-no-time builtin (Circleback/Granola/Zoom, no timestamp) The 14th built-in pattern parses `**Speaker:** text` transcripts with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Every prior builtin required a time anchor, so this shape matched nothing: a production brain had 104 conversation pages + 3,423 eligible pages silently extracting zero facts. Messages anchor at T00:00:00Z of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same convention as irc-classic. Hardening beyond the original community proposal: - regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`: the colon-inside-bold (NOT declaration order) is what prevents shadowing bold-paren-time; the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest no_match instead of speaker="[18:37] Name". - new optional PatternEntry.score_full_body: `**Label:** text` is a common prose idiom, so a notes page with bold labels clustered in its first 10 lines scored 0.3 on the head pass (NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so the full-body fallback never fired) and cleared the 0.05 floor. parse.ts now recomputes the winner's score over the full body before the floor, so such a page drops to its true low density and stays no_match. - scrubbed pre-existing real names from bold-paren-time test_positive samples (privacy rule). Fixtures use placeholder names only. Pinned by new bold-name-no-time + clustered-head no_match cases in parse.test.ts and the eval corpus. Co-Authored-By: garrytan-agents <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(orphans): scope orphan_ratio + find_orphans by source; fix total_linkable denominator `gbrain doctor --source <id>` and `gbrain orphans --source <id>` now scope the orphan scan to that source instead of reporting brain-wide. Three fixes: - findOrphanPages(opts?: { sourceId?, sourceIds? }) on both engines scopes the CANDIDATE set (scalar `= $1` or federated `= ANY($1::text[])`). Inbound links from ANY source still count, so a page in source X linked FROM source Y is reachable and NOT an orphan of X (the deliberate, less-surprising definition). - corrected the total_linkable denominator in findOrphans: it now enumerates all live pages (scoped) and subtracts every excluded-by-slug page, not just excluded orphans. The old `total - excludedOrphans` left excluded NON-orphan pages (templates/, scratch/) with inbound links in the denominator, inflating it and suppressing warnings. Changes orphan_ratio output for every brain, in the accurate direction. - the find_orphans MCP op threads sourceScopeOpts(ctx), closing a cross-source read leak where a source-bound OAuth client saw brain-wide orphans (v0.34.1 source-isolation class). doctor uses an explicit `--source` flag parse (NOT resolveSourceWithTier, which would scope bare invocations to a default), and under explicit --source reports the ratio with a low-scale caveat below 100 entity pages instead of a vacuous "ok". Thin-client doctor --source orphan_ratio deferred (TODOS.md). Pinned by test/orphans-source-scope.test.ts (PGLite: scoping, cross-source inbound, denominator, find_orphans op scope) + a Postgres↔PGLite parity case in test/e2e/engine-parity.test.ts (scalar + federated binding). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: v0.41.29.0 — bold-name-no-time + orphan source scoping VERSION + package.json → 0.41.29.0; CHANGELOG entry; CLAUDE.md conversation-parser (13→14 patterns) + orphans source-scoping notes; regenerated llms bundles; TODOS for thin-client doctor --source + check-test-real-names widening. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
garrytan-agents
Claude Opus 4.8
parent
ffac8ce0f4
commit
041d89babe
+131
@@ -2,6 +2,137 @@
|
|||||||
|
|
||||||
All notable changes to GBrain will be documented in this file.
|
All notable changes to GBrain will be documented in this file.
|
||||||
|
|
||||||
|
## [0.41.29.0] - 2026-05-29
|
||||||
|
|
||||||
|
**Your meeting transcripts that look like `**Garry Tan:** ...` now actually
|
||||||
|
parse — the kind Circleback, Granola, and Zoom export with no timestamp on
|
||||||
|
each line. And `gbrain doctor --source <id>` finally scopes its orphan check
|
||||||
|
to that one source instead of your whole brain.**
|
||||||
|
|
||||||
|
Two fixes in one release.
|
||||||
|
|
||||||
|
**The transcript fix.** gbrain knows 13 chat formats, and every single one
|
||||||
|
needed a per-line timestamp to recognize a speaker. Modern meeting tools
|
||||||
|
don't put a time on every line — they emit plain `**Speaker Name:** what
|
||||||
|
they said`. With no time anchor, gbrain matched nothing: a real production
|
||||||
|
brain had 104 conversation pages plus 3,423 other eligible pages that
|
||||||
|
silently parsed to zero messages and extracted zero facts. The 14th built-in
|
||||||
|
pattern, `bold-name-no-time`, handles that shape. Each message anchors at
|
||||||
|
00:00:00 of the page's date (no fake wall-clock times are invented; line
|
||||||
|
order preserves the sequence). Nothing you do — it just works the next time
|
||||||
|
your cycle runs.
|
||||||
|
|
||||||
|
How to check it's recognized:
|
||||||
|
|
||||||
|
```
|
||||||
|
gbrain conversation-parser list-builtins # bold-name-no-time is now listed
|
||||||
|
gbrain conversation-parser scan <slug> # see how one page parses
|
||||||
|
```
|
||||||
|
|
||||||
|
One thing we were careful about: that `**Label:** text` shape is also a
|
||||||
|
common way to write notes (`**Owner:** Alice`, `**Action:** ship it`). A
|
||||||
|
notes page is NOT a conversation, and gbrain won't treat it as one — even
|
||||||
|
when a few bold labels sit at the very top of the page. We score the whole
|
||||||
|
document's density, not just the first few lines, so a mostly-prose page
|
||||||
|
with scattered bold labels stays unparsed instead of turning into fake
|
||||||
|
"messages" that pollute your facts.
|
||||||
|
|
||||||
|
**The orphan-check fix.** "Orphan ratio" is the fraction of your pages that
|
||||||
|
nothing links to — a health signal for `gbrain doctor`. On a brain with
|
||||||
|
multiple sources, `gbrain doctor --source dept-x` ignored the flag and
|
||||||
|
reported a brain-wide number. Now it scopes to the source you asked about,
|
||||||
|
and so does `gbrain orphans --source dept-x`:
|
||||||
|
|
||||||
|
```
|
||||||
|
gbrain doctor --source dept-x # orphan ratio for dept-x only
|
||||||
|
gbrain orphans --source dept-x # the orphan list for dept-x only
|
||||||
|
```
|
||||||
|
|
||||||
|
A page in dept-x that's linked from another source counts as reachable (not
|
||||||
|
an orphan) — the cross-source link still saves it.
|
||||||
|
|
||||||
|
While we were in there we fixed two more things you'll feel:
|
||||||
|
|
||||||
|
- The orphan ratio was reading lower than reality because junk pages
|
||||||
|
(templates, scratch, archives) that happen to be linked were padding the
|
||||||
|
denominator. The ratio is now honest, which means warnings that were
|
||||||
|
staying quiet will start showing up. This changes the number on every
|
||||||
|
brain, in the right direction.
|
||||||
|
- An agent connected over MCP and scoped to one source used to get the
|
||||||
|
orphan list for your *whole* brain. Now `find_orphans` respects the
|
||||||
|
agent's source scope, closing a cross-source read leak.
|
||||||
|
|
||||||
|
### To take advantage of v0.41.29.0
|
||||||
|
|
||||||
|
`gbrain upgrade` handles this — there's no schema migration and no manual
|
||||||
|
step. After upgrading:
|
||||||
|
|
||||||
|
1. **Conversation parsing** is automatic. Your next `gbrain dream` /
|
||||||
|
`gbrain extract conversation-facts` run will pick up `**Speaker:** text`
|
||||||
|
pages. To preview a specific page:
|
||||||
|
```bash
|
||||||
|
gbrain conversation-parser scan <slug>
|
||||||
|
```
|
||||||
|
2. **Per-source orphan checks** work immediately:
|
||||||
|
```bash
|
||||||
|
gbrain doctor --source <id>
|
||||||
|
gbrain orphans --source <id>
|
||||||
|
```
|
||||||
|
3. **If `gbrain doctor` shows a higher orphan ratio than before,** that's the
|
||||||
|
denominator fix surfacing real orphans that were previously hidden. Run
|
||||||
|
`gbrain extract links --by-mention` to auto-link entity mentions, then
|
||||||
|
`gbrain orphans` for the list.
|
||||||
|
4. **If anything looks wrong,** file an issue at
|
||||||
|
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output.
|
||||||
|
|
||||||
|
### Itemized changes
|
||||||
|
|
||||||
|
**Conversation parser**
|
||||||
|
- New 14th built-in pattern `bold-name-no-time` in
|
||||||
|
`src/core/conversation-parser/builtins.ts` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`):
|
||||||
|
parses `**Speaker:** text` with no per-line timestamp (Circleback / Granola
|
||||||
|
/ Zoom), anchored at `T00:00:00Z` of the frontmatter date. Declared after
|
||||||
|
the timestamped bold patterns; the colon-inside-bold regex (not declaration
|
||||||
|
order) is what prevents it from shadowing `bold-paren-time`. The `(?!\[)`
|
||||||
|
lookahead keeps it from mis-capturing telegram-bracket `**[18:37] Name:**`
|
||||||
|
lines as `speaker="[18:37] Name"`.
|
||||||
|
- New optional `PatternEntry.score_full_body` field + a full-body acceptance
|
||||||
|
recompute in `parse.ts`: broad patterns are scored on whole-document
|
||||||
|
density before the acceptance floor, so a prose notes page with bold labels
|
||||||
|
clustered in its first 10 lines no longer mis-parses as a conversation.
|
||||||
|
- Fixture coverage: `test/fixtures/conversation-formats/bold-name-no-time.jsonl`
|
||||||
|
+ entries in `all.jsonl` and a clustered-head adversarial fixture in
|
||||||
|
`adversarial.jsonl` (all placeholder names).
|
||||||
|
|
||||||
|
**Orphan source scoping**
|
||||||
|
- `BrainEngine.findOrphanPages(opts?: { sourceId?, sourceIds? })` (both
|
||||||
|
engines): scopes the candidate set to one source (scalar) or a federated
|
||||||
|
set (`= ANY(...)`). Inbound links from any source still count, so a
|
||||||
|
cross-source-linked page is correctly not an orphan.
|
||||||
|
- `gbrain orphans --source <id>` and `gbrain doctor --source <id>` (orphan_ratio
|
||||||
|
check) honor an explicit source. Bare invocations stay brain-wide.
|
||||||
|
- Corrected the `total_linkable` denominator in `findOrphans` so excluded
|
||||||
|
pages (templates/, scratch/, etc.) that have inbound links no longer
|
||||||
|
inflate it and suppress warnings. Changes orphan_ratio output for every
|
||||||
|
brain (in the accurate direction).
|
||||||
|
- The `find_orphans` MCP op now scopes by the caller's source
|
||||||
|
(`sourceScopeOpts(ctx)`), closing a cross-source read leak for source-bound
|
||||||
|
OAuth clients.
|
||||||
|
- Under explicit `--source`, `orphan_ratio` reports the ratio with a
|
||||||
|
low-scale caveat below 100 entity pages instead of returning a vacuous
|
||||||
|
"ok" that could hide a fully-orphaned small source.
|
||||||
|
|
||||||
|
**Tests**
|
||||||
|
- `test/orphans-source-scope.test.ts` (PGLite): scoping, cross-source-inbound,
|
||||||
|
the denominator fix, and the `find_orphans` op-handler source scope.
|
||||||
|
- `test/e2e/engine-parity.test.ts`: Postgres↔PGLite parity for
|
||||||
|
`findOrphanPages` scalar + federated scoping.
|
||||||
|
- New `bold-name-no-time` + clustered-head regression cases in
|
||||||
|
`test/conversation-parser/parse.test.ts`.
|
||||||
|
|
||||||
|
Incorporates the original `bold-name-no-time` pattern proposed in a community
|
||||||
|
PR; names scrubbed to placeholders per the project privacy rule.
|
||||||
|
|
||||||
## [0.41.28.0] - 2026-05-27
|
## [0.41.28.0] - 2026-05-27
|
||||||
|
|
||||||
**Your `gbrain dream` cycle stops losing rows when the database connection
|
**Your `gbrain dream` cycle stops losing rows when the database connection
|
||||||
|
|||||||
@@ -1,5 +1,15 @@
|
|||||||
# TODOS
|
# TODOS
|
||||||
|
|
||||||
|
## v0.41.29.0 orphan source-scoping follow-ups (v0.42+)
|
||||||
|
|
||||||
|
Filed from the v0.41.29.0 wave (bold-name-no-time pattern + orphan_ratio
|
||||||
|
source scoping). The Codex outside-voice review (F8) flagged two surfaces
|
||||||
|
the wave deliberately scoped out.
|
||||||
|
|
||||||
|
- [ ] **v0.42+: thin-client `gbrain doctor --source` orphan_ratio scoping.** v0.41.29.0 scopes `orphan_ratio` to `--source` on the LOCAL doctor path (`buildChecks` in `src/commands/doctor.ts`) and closes the `find_orphans` MCP read leak via `sourceScopeOpts(ctx)`. The thin-client / remote doctor path (`src/core/doctor-remote.ts` `runRemoteDoctor`) is a separate code path that does not thread `--source`, so `gbrain doctor --source x` against a remote `gbrain serve --http` brain still reports brain-wide orphan_ratio. Thread the explicit `--source` into the remote doctor request + have the server-side check honor it. Priority: P3 (most users run doctor locally).
|
||||||
|
|
||||||
|
- [ ] **v0.42+: widen `check-test-real-names.sh` BANNED_NAMES to catch real-name reintroduction in tests + src.** v0.41.29.0 scrubbed pre-existing real names (`Garry Tan`, `Alex Graveley`) from `bold-paren-time`'s `test_positive` (and the new `bold-name-no-time` samples), but no automated guard caught them: `check-test-real-names.sh` only scans `test/**` and its BANNED_NAMES list doesn't include `garry tan`; `check-fixture-privacy.sh` only scans `test/fixtures/conversation-formats/`. Add `garry tan` / `garrytan` (and consider extending the scan to `src/core/conversation-parser/builtins.ts` test samples) so future reintroductions fail CI. Priority: P3 (hardening).
|
||||||
|
|
||||||
## v0.41.28.0 #1570 instrument-then-fix follow-ups (v0.41.28+ / v0.42+)
|
## v0.41.28.0 #1570 instrument-then-fix follow-ups (v0.41.28+ / v0.42+)
|
||||||
|
|
||||||
Filed from the v0.41.28.0 plan-eng-review after the codex outside-voice
|
Filed from the v0.41.28.0 plan-eng-review after the codex outside-voice
|
||||||
|
|||||||
+2
-2
File diff suppressed because one or more lines are too long
+1
-1
@@ -141,5 +141,5 @@
|
|||||||
"bun": ">=1.3.10"
|
"bun": ">=1.3.10"
|
||||||
},
|
},
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"version": "0.41.28.0"
|
"version": "0.41.29.0"
|
||||||
}
|
}
|
||||||
|
|||||||
+36
-6
@@ -3080,6 +3080,19 @@ export async function buildChecks(
|
|||||||
// invoked later in the function.
|
// invoked later in the function.
|
||||||
const scope: 'all' | 'brain' = args.includes('--scope=brain') ? 'brain' : 'all';
|
const scope: 'all' | 'brain' = args.includes('--scope=brain') ? 'brain' : 'all';
|
||||||
|
|
||||||
|
// v0.41.29.0: explicit `--source <id>` scopes the `orphan_ratio` check to one
|
||||||
|
// source. EXPLICIT-ONLY by design — a raw flag parse, NOT resolveSourceWithTier.
|
||||||
|
// The tier resolver would pick a default source when `--source` is absent and
|
||||||
|
// silently scope a bare `gbrain doctor` to one source; we want bare doctor to
|
||||||
|
// stay brain-wide. Only `orphan_ratio` consumes this for now (other checks
|
||||||
|
// staying brain-wide is a separate, larger change — see TODOS.md).
|
||||||
|
let orphanRatioSourceId: string | undefined;
|
||||||
|
for (let i = 0; i < args.length; i++) {
|
||||||
|
if (args[i] === '--source' && i + 1 < args.length) {
|
||||||
|
orphanRatioSourceId = args[++i] || undefined;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const checks: Check[] = [];
|
const checks: Check[] = [];
|
||||||
let autoFixReport: AutoFixReport | null = null;
|
let autoFixReport: AutoFixReport | null = null;
|
||||||
|
|
||||||
@@ -4543,22 +4556,39 @@ export async function buildChecks(
|
|||||||
// show high orphan ratio; not actionable signal).
|
// show high orphan ratio; not actionable signal).
|
||||||
// Warn at >0.5; fail at >0.8. Both states recommend
|
// Warn at >0.5; fail at >0.8. Both states recommend
|
||||||
// `gbrain extract links --by-mention` as the fix.
|
// `gbrain extract links --by-mention` as the fix.
|
||||||
|
// v0.41.29.0: explicit `--source <id>` scopes this check to one source
|
||||||
|
// (orphanRatioSourceId, parsed at the top of buildChecks). The entity-count
|
||||||
|
// gate + getOrphansData both scope to it; messages name the source. Bare
|
||||||
|
// doctor (no --source) stays brain-wide.
|
||||||
progress.heartbeat('orphan_ratio');
|
progress.heartbeat('orphan_ratio');
|
||||||
try {
|
try {
|
||||||
const { getOrphansData } = await import('./orphans.ts');
|
const { getOrphansData } = await import('./orphans.ts');
|
||||||
|
const srcId = orphanRatioSourceId;
|
||||||
|
const inSource = srcId ? ` in source '${srcId}'` : '';
|
||||||
const entityCount = (await engine.executeRaw<{ count: number }>(
|
const entityCount = (await engine.executeRaw<{ count: number }>(
|
||||||
"SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization') AND deleted_at IS NULL",
|
`SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization') AND deleted_at IS NULL${srcId ? ' AND source_id = $1' : ''}`,
|
||||||
|
srcId ? [srcId] : [],
|
||||||
))[0]?.count ?? 0;
|
))[0]?.count ?? 0;
|
||||||
if (entityCount < 100) {
|
// Brain-wide (no --source): <100 entities is vacuous — small brains
|
||||||
|
// naturally show a high orphan ratio; not actionable signal. Skip.
|
||||||
|
if (entityCount < 100 && !srcId) {
|
||||||
checks.push({
|
checks.push({
|
||||||
name: 'orphan_ratio',
|
name: 'orphan_ratio',
|
||||||
status: 'ok',
|
status: 'ok',
|
||||||
message: `Vacuous: ${entityCount} entity pages (<100). Orphan ratio not meaningful at this scale.`,
|
message: `Vacuous: ${entityCount} entity pages (<100). Orphan ratio not meaningful at this scale.`,
|
||||||
});
|
});
|
||||||
} else {
|
} else {
|
||||||
const data = await getOrphansData(engine, { includePseudo: false });
|
// F7 (Codex): under EXPLICIT --source, an operator deliberately asked
|
||||||
|
// about one source — answer it even below 100 entities, with a
|
||||||
|
// low-scale caveat, instead of swallowing a real per-source failure
|
||||||
|
// (e.g. 80 fully-orphaned entity pages) behind a vacuous "ok".
|
||||||
|
const data = await getOrphansData(engine, { includePseudo: false, sourceId: srcId });
|
||||||
const ratio = data.total_linkable > 0 ? data.total_orphans / data.total_linkable : 0;
|
const ratio = data.total_linkable > 0 ? data.total_orphans / data.total_linkable : 0;
|
||||||
const pct = (ratio * 100).toFixed(0);
|
const pct = (ratio * 100).toFixed(0);
|
||||||
|
const caveat =
|
||||||
|
entityCount < 100
|
||||||
|
? ` — low scale (${entityCount} entity pages <100), interpret with caution`
|
||||||
|
: '';
|
||||||
const hint =
|
const hint =
|
||||||
'Run: gbrain extract links --by-mention (auto-links entity mentions in body text). ' +
|
'Run: gbrain extract links --by-mention (auto-links entity mentions in body text). ' +
|
||||||
'Run gbrain orphans for the list.';
|
'Run gbrain orphans for the list.';
|
||||||
@@ -4566,19 +4596,19 @@ export async function buildChecks(
|
|||||||
checks.push({
|
checks.push({
|
||||||
name: 'orphan_ratio',
|
name: 'orphan_ratio',
|
||||||
status: 'fail',
|
status: 'fail',
|
||||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`,
|
message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links)${caveat}. ${hint}`,
|
||||||
});
|
});
|
||||||
} else if (ratio > 0.5) {
|
} else if (ratio > 0.5) {
|
||||||
checks.push({
|
checks.push({
|
||||||
name: 'orphan_ratio',
|
name: 'orphan_ratio',
|
||||||
status: 'warn',
|
status: 'warn',
|
||||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`,
|
message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links)${caveat}. ${hint}`,
|
||||||
});
|
});
|
||||||
} else {
|
} else {
|
||||||
checks.push({
|
checks.push({
|
||||||
name: 'orphan_ratio',
|
name: 'orphan_ratio',
|
||||||
status: 'ok',
|
status: 'ok',
|
||||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages)`,
|
message: `Orphan ratio ${pct}%${inSource} (${data.total_orphans}/${data.total_linkable} linkable pages)${caveat}`,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+53
-8
@@ -127,9 +127,16 @@ export async function queryOrphanPages(
|
|||||||
*/
|
*/
|
||||||
export async function findOrphans(
|
export async function findOrphans(
|
||||||
engine: BrainEngine,
|
engine: BrainEngine,
|
||||||
opts: { includePseudo?: boolean } = {},
|
opts: { includePseudo?: boolean; sourceId?: string; sourceIds?: string[] } = {},
|
||||||
): Promise<OrphanResult> {
|
): Promise<OrphanResult> {
|
||||||
const includePseudo = !!opts.includePseudo;
|
const includePseudo = !!opts.includePseudo;
|
||||||
|
// v0.41.29.0: `sourceId` (scalar, from `--source` + single-source MCP
|
||||||
|
// clients) or `sourceIds` (federated, from `allowedSources` MCP clients)
|
||||||
|
// scopes the candidate set. `sourceIds` wins when both set (mirrors
|
||||||
|
// sourceScopeOpts precedence).
|
||||||
|
const sourceId = opts.sourceId;
|
||||||
|
const sourceIds =
|
||||||
|
opts.sourceIds && opts.sourceIds.length > 0 ? opts.sourceIds : undefined;
|
||||||
// The NOT EXISTS anti-join over pages × links can take seconds on 50K-page
|
// The NOT EXISTS anti-join over pages × links can take seconds on 50K-page
|
||||||
// brains. Heartbeat every second so agents see the scan is alive. Keyset
|
// brains. Heartbeat every second so agents see the scan is alive. Keyset
|
||||||
// pagination was considered and rejected: without an index on
|
// pagination was considered and rejected: without an index on
|
||||||
@@ -140,16 +147,40 @@ export async function findOrphans(
|
|||||||
const stopHb = startHeartbeat(progress, 'scanning pages for missing inbound links…');
|
const stopHb = startHeartbeat(progress, 'scanning pages for missing inbound links…');
|
||||||
let allOrphans: { slug: string; title: string; domain: string | null }[];
|
let allOrphans: { slug: string; title: string; domain: string | null }[];
|
||||||
let total: number;
|
let total: number;
|
||||||
|
let excludedAll: number;
|
||||||
try {
|
try {
|
||||||
allOrphans = await engine.findOrphanPages();
|
allOrphans = await engine.findOrphanPages(
|
||||||
// Count total pages in DB for the summary line
|
sourceIds ? { sourceIds } : sourceId ? { sourceId } : undefined,
|
||||||
const stats = await engine.getStats();
|
);
|
||||||
total = stats.page_count;
|
// v0.41.29.0 (Codex F6): correct the `total_linkable` denominator.
|
||||||
|
// Enumerate ALL live pages (scoped) and count excluded-by-slug across
|
||||||
|
// the WHOLE set — not just among orphans. The old
|
||||||
|
// `total - excludedOrphans` left excluded NON-orphan pages (e.g. a
|
||||||
|
// `test/` page that HAS inbound links) in the denominator, inflating
|
||||||
|
// total_linkable and suppressing orphan warnings. `getAllSlugs` is NOT
|
||||||
|
// used here because it does not filter soft-deleted rows; `total` must
|
||||||
|
// match `findOrphanPages`'s `deleted_at IS NULL` candidate universe.
|
||||||
|
let scopeClause = '';
|
||||||
|
const liveParams: unknown[] = [];
|
||||||
|
if (sourceIds) {
|
||||||
|
liveParams.push(sourceIds);
|
||||||
|
scopeClause = ` AND source_id = ANY($${liveParams.length}::text[])`;
|
||||||
|
} else if (sourceId) {
|
||||||
|
liveParams.push(sourceId);
|
||||||
|
scopeClause = ` AND source_id = $${liveParams.length}`;
|
||||||
|
}
|
||||||
|
const liveRows = await engine.executeRaw<{ slug: string }>(
|
||||||
|
`SELECT slug FROM pages WHERE deleted_at IS NULL${scopeClause}`,
|
||||||
|
liveParams,
|
||||||
|
);
|
||||||
|
total = liveRows.length;
|
||||||
|
excludedAll = includePseudo
|
||||||
|
? 0
|
||||||
|
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug) ? 1 : 0), 0);
|
||||||
} finally {
|
} finally {
|
||||||
stopHb();
|
stopHb();
|
||||||
progress.finish();
|
progress.finish();
|
||||||
}
|
}
|
||||||
const _totalPages = allOrphans.length; // pages with no inbound links (preserved for ref)
|
|
||||||
|
|
||||||
const filtered = includePseudo
|
const filtered = includePseudo
|
||||||
? allOrphans
|
? allOrphans
|
||||||
@@ -166,7 +197,10 @@ export async function findOrphans(
|
|||||||
return {
|
return {
|
||||||
orphans,
|
orphans,
|
||||||
total_orphans: orphans.length,
|
total_orphans: orphans.length,
|
||||||
total_linkable: filtered.length + (total - allOrphans.length),
|
// v0.41.29.0 (Codex F6): denominator = live pages minus ALL excluded
|
||||||
|
// pages (orphan or not), so excluded pages with inbound links no longer
|
||||||
|
// inflate it.
|
||||||
|
total_linkable: total - excludedAll,
|
||||||
total_pages: total,
|
total_pages: total,
|
||||||
excluded,
|
excluded,
|
||||||
};
|
};
|
||||||
@@ -224,6 +258,16 @@ export async function runOrphans(engine: BrainEngine, args: string[]) {
|
|||||||
const json = args.includes('--json');
|
const json = args.includes('--json');
|
||||||
const count = args.includes('--count');
|
const count = args.includes('--count');
|
||||||
const includePseudo = args.includes('--include-pseudo');
|
const includePseudo = args.includes('--include-pseudo');
|
||||||
|
// v0.41.29.0: explicit `--source <id>` scopes the orphan scan to one
|
||||||
|
// source. Omitted → brain-wide (unchanged). Raw explicit-flag parse on
|
||||||
|
// purpose — NOT resolveSourceWithTier, which would pick a default source
|
||||||
|
// when the flag is absent and silently scope a bare `gbrain orphans`.
|
||||||
|
let sourceId: string | undefined;
|
||||||
|
for (let i = 0; i < args.length; i++) {
|
||||||
|
if (args[i] === '--source' && i + 1 < args.length) {
|
||||||
|
sourceId = args[++i] || undefined;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (args.includes('--help') || args.includes('-h')) {
|
if (args.includes('--help') || args.includes('-h')) {
|
||||||
console.log(`Usage: gbrain orphans [options]
|
console.log(`Usage: gbrain orphans [options]
|
||||||
@@ -234,6 +278,7 @@ Options:
|
|||||||
--json Output as JSON (for agent consumption)
|
--json Output as JSON (for agent consumption)
|
||||||
--count Output just the number of orphans
|
--count Output just the number of orphans
|
||||||
--include-pseudo Include auto-generated and pseudo pages in results
|
--include-pseudo Include auto-generated and pseudo pages in results
|
||||||
|
--source <id> Scope the scan to one brain source (default: brain-wide)
|
||||||
--help, -h Show this help
|
--help, -h Show this help
|
||||||
|
|
||||||
Output (default): grouped by domain, sorted alphabetically within each group
|
Output (default): grouped by domain, sorted alphabetically within each group
|
||||||
@@ -242,7 +287,7 @@ Summary line: N orphans out of M linkable pages (K total; K-M excluded)
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const result = await findOrphans(engine, { includePseudo });
|
const result = await findOrphans(engine, { includePseudo, sourceId });
|
||||||
|
|
||||||
if (count) {
|
if (count) {
|
||||||
console.log(String(result.total_orphans));
|
console.log(String(result.total_orphans));
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
/**
|
/**
|
||||||
* v0.41.16.0 — Built-in conversation parser pattern registry.
|
* v0.41.16.0 — Built-in conversation parser pattern registry.
|
||||||
*
|
*
|
||||||
* Twelve hand-vetted patterns covering the chat-export formats this
|
* Fourteen hand-vetted patterns covering the chat-export formats this
|
||||||
* codebase is most likely to encounter. Each pattern's regex was
|
* codebase is most likely to encounter. Each pattern's regex was
|
||||||
* derived from a public format reference (source_doc field) so future
|
* derived from a public format reference (source_doc field) so future
|
||||||
* maintainers can verify against the wild shape.
|
* maintainers can verify against the wild shape.
|
||||||
@@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string {
|
|||||||
return stripped || raw.trim();
|
return stripped || raw.trim();
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The 12 hand-vetted built-in patterns. */
|
/** The 14 hand-vetted built-in patterns. */
|
||||||
export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
||||||
// -------------------------------------------------------------------
|
// -------------------------------------------------------------------
|
||||||
// INLINE-DATE patterns (date in every line; less ambiguous; tried first).
|
// INLINE-DATE patterns (date in every line; less ambiguous; tried first).
|
||||||
@@ -158,9 +158,9 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
|||||||
multi_line: false,
|
multi_line: false,
|
||||||
quick_reject: /^\*\*/,
|
quick_reject: /^\*\*/,
|
||||||
test_positive: [
|
test_positive: [
|
||||||
'**Garry Tan** (00:00): hello world',
|
'**Alice Example** (00:00): hello world',
|
||||||
'**Participant 2** (02:22): response here',
|
'**Participant 2** (02:22): response here',
|
||||||
'**Alex Graveley** (15:09): That’s exactly right.',
|
'**Bob Example** (15:09): That’s exactly right.',
|
||||||
'**Participant 1** (00:00:00): hello world with seconds',
|
'**Participant 1** (00:00:00): hello world with seconds',
|
||||||
'**Participant 2** (01:23:45): mid-meeting line',
|
'**Participant 2** (01:23:45): mid-meeting line',
|
||||||
],
|
],
|
||||||
@@ -178,6 +178,75 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
|||||||
'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)',
|
'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)',
|
||||||
},
|
},
|
||||||
|
|
||||||
|
{
|
||||||
|
// Modern meeting-transcription tools (Circleback, Granola, Zoom)
|
||||||
|
// emit `**Speaker Name:** message text` with NO per-line
|
||||||
|
// timestamp. Every other built-in requires a time anchor, so this
|
||||||
|
// shape scored ~0.002 (one stray line in a long page) and fell
|
||||||
|
// below SCORING_MIN_ACCEPTANCE — parsing to zero messages and
|
||||||
|
// extracting zero conversation-facts. This additive pattern fixes
|
||||||
|
// that: speaker is captured inside the bold markers (`**Name:**`),
|
||||||
|
// there is no time capture, and date_source='frontmatter' with
|
||||||
|
// hour_group undefined routes through parse.ts's no-time branch
|
||||||
|
// (same convention as irc-classic) — every message anchors at
|
||||||
|
// 00:00:00 of the page's frontmatter date. No wall-clock time is
|
||||||
|
// fabricated; intra-day ordering is preserved by line order.
|
||||||
|
//
|
||||||
|
// NON-SHADOW GUARANTEE (read carefully — the safety is in the
|
||||||
|
// REGEX, not the declaration order). parse.ts scores every
|
||||||
|
// candidate independently; declaration index is ONLY the
|
||||||
|
// tie-break. This pattern cannot steal `**Name** (time):` from
|
||||||
|
// bold-paren-time because its regex requires the colon INSIDE the
|
||||||
|
// bold markers (`**Name:**`), which the paren-time shape (colon
|
||||||
|
// OUTSIDE: `**Name** (time):`) never has. The `(?!\[)` lookahead
|
||||||
|
// additionally rejects telegram-bracket's `**[18:37] Name:**`
|
||||||
|
// shape so that disabling telegram-bracket yields an honest
|
||||||
|
// no_match instead of capturing speaker="[18:37] Name" at midnight.
|
||||||
|
//
|
||||||
|
// BROAD-REGEX GUARD (score_full_body): `**Label:** text` is a
|
||||||
|
// common prose idiom (`**Note:**`, `**Owner:**`). A notes page
|
||||||
|
// with a few bold labels clustered in its first 10 lines would
|
||||||
|
// score 0.3 on the head pass, skip the rescore, and clear the
|
||||||
|
// 0.05 floor. score_full_body forces full-body density scoring
|
||||||
|
// before acceptance so such a page falls to no_match.
|
||||||
|
id: 'bold-name-no-time',
|
||||||
|
origin: 'builtin',
|
||||||
|
// Matches: **Speaker Name:** message text (colon INSIDE bold,
|
||||||
|
// speaker must not start with `[` — see lookahead rationale above).
|
||||||
|
regex: /^\*\*(?!\[)(.+?):\*\*\s*(.*)$/,
|
||||||
|
captures: {
|
||||||
|
speaker_group: 1,
|
||||||
|
text_group: 2,
|
||||||
|
},
|
||||||
|
date_source: 'frontmatter',
|
||||||
|
time_format: '24h',
|
||||||
|
timezone_policy: 'utc_assumed_with_warn',
|
||||||
|
multi_line: false,
|
||||||
|
quick_reject: /^\*\*/,
|
||||||
|
score_full_body: true,
|
||||||
|
test_positive: [
|
||||||
|
'**Alice Example:** Okay, start on.',
|
||||||
|
'**Participant 2:** he tried to reset it remotely the other night.',
|
||||||
|
'**Bob Example:** That is exactly right.',
|
||||||
|
],
|
||||||
|
test_negative: [
|
||||||
|
// bold-paren-time shape (colon OUTSIDE bold) MUST fall through:
|
||||||
|
'**Alice** (00:00): text',
|
||||||
|
// imessage-slack shape MUST fall through:
|
||||||
|
'**Alice Example** (2024-03-15 9:00 AM): iMessage shape',
|
||||||
|
// Bold but no colon at all:
|
||||||
|
'**Alice** hello world',
|
||||||
|
// No bold markers:
|
||||||
|
'Alice: plain no bold',
|
||||||
|
// telegram-bracket shape (timestamp INSIDE bold) MUST NOT match
|
||||||
|
// — the `(?!\[)` lookahead rejects it so disabling
|
||||||
|
// telegram-bracket yields no_match, not speaker="[18:37] Alice":
|
||||||
|
'**[18:37] \u{1f464} Alice:** hello',
|
||||||
|
],
|
||||||
|
source_doc:
|
||||||
|
'Circleback / Granola / Zoom meeting-transcript export shape: `**Speaker:** text` with no per-line timestamp',
|
||||||
|
},
|
||||||
|
|
||||||
{
|
{
|
||||||
id: 'telegram-text-export',
|
id: 'telegram-text-export',
|
||||||
origin: 'builtin',
|
origin: 'builtin',
|
||||||
|
|||||||
@@ -479,6 +479,7 @@ export function parseConversation(
|
|||||||
// 10 head lines matched"; chat-only pages still score 1.0 and skip
|
// 10 head lines matched"; chat-only pages still score 1.0 and skip
|
||||||
// the fallback. Re-score every candidate against the full body,
|
// the fallback. Re-score every candidate against the full body,
|
||||||
// pre-splitting ONCE to avoid 12 redundant body splits.
|
// pre-splitting ONCE to avoid 12 redundant body splits.
|
||||||
|
let fullBodyScored = false;
|
||||||
if (scored[0].score < SCORING_HEAD_TRIGGER_THRESHOLD) {
|
if (scored[0].score < SCORING_HEAD_TRIGGER_THRESHOLD) {
|
||||||
const allLines = getNonBlankLines(body);
|
const allLines = getNonBlankLines(body);
|
||||||
scored = candidates.map((entry) => ({
|
scored = candidates.map((entry) => ({
|
||||||
@@ -487,6 +488,7 @@ export function parseConversation(
|
|||||||
priority: priorityOf(entry.id),
|
priority: priorityOf(entry.id),
|
||||||
}));
|
}));
|
||||||
sortScored(scored);
|
sortScored(scored);
|
||||||
|
fullBodyScored = true;
|
||||||
// NOTE: patterns_scored stays as scored.length (= candidate
|
// NOTE: patterns_scored stays as scored.length (= candidate
|
||||||
// count, typically 12) even when the fallback runs — the
|
// count, typically 12) even when the fallback runs — the
|
||||||
// diagnostic reports "candidates considered" not "scoring
|
// diagnostic reports "candidates considered" not "scoring
|
||||||
@@ -496,6 +498,21 @@ export function parseConversation(
|
|||||||
const top = scored[0];
|
const top = scored[0];
|
||||||
const patternsScored = scored.length;
|
const patternsScored = scored.length;
|
||||||
|
|
||||||
|
// v0.41.29.0 (Codex F1): broad no-time patterns (`bold-name-no-time`,
|
||||||
|
// regex matches any `**Label:** text`) can win the HEAD pass on a
|
||||||
|
// prose notes page whose first 10 lines happen to hold 3+ bold labels
|
||||||
|
// (head score 0.3, NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so the
|
||||||
|
// full-body fallback above never ran). 0.3 clears the 0.05 floor and
|
||||||
|
// the page mis-parses as a conversation. When the winner is flagged
|
||||||
|
// `score_full_body` and we have NOT already scored full-body, recompute
|
||||||
|
// its score over the whole document so the floor judges true density.
|
||||||
|
// We do NOT re-pick the winner: the head pick is already the best
|
||||||
|
// candidate; this only tightens its acceptance (a real transcript
|
||||||
|
// stays ~1.0; a 3/200-label notes page drops to ~0.015 → no_match).
|
||||||
|
if (top.entry.score_full_body && !fullBodyScored) {
|
||||||
|
top.score = scoreFromLines(getNonBlankLines(body), top.entry);
|
||||||
|
}
|
||||||
|
|
||||||
// Minimum acceptance floor (closes Codex P1 #2): an essay with
|
// Minimum acceptance floor (closes Codex P1 #2): an essay with
|
||||||
// one stray `**Name** (date time):` line scores ~1/300 ≈ 0.003 —
|
// one stray `**Name** (date time):` line scores ~1/300 ≈ 0.003 —
|
||||||
// below the 5% floor we stay no_match instead of returning a
|
// below the 5% floor we stay no_match instead of returning a
|
||||||
|
|||||||
@@ -170,6 +170,19 @@ export interface PatternEntry {
|
|||||||
* uses `DEFAULT_SPEAKER_CLEAN`.
|
* uses `DEFAULT_SPEAKER_CLEAN`.
|
||||||
*/
|
*/
|
||||||
speaker_clean?: RegExp;
|
speaker_clean?: RegExp;
|
||||||
|
/**
|
||||||
|
* v0.41.29.0: when true, the orchestrator recomputes this pattern's
|
||||||
|
* acceptance score over the FULL body (not just the head window)
|
||||||
|
* before applying SCORING_MIN_ACCEPTANCE. Set on broad no-time
|
||||||
|
* patterns (`bold-name-no-time`) whose regex matches a common prose
|
||||||
|
* idiom (`**Label:** text`): without it, a notes page with a few bold
|
||||||
|
* labels CLUSTERED in its first 10 lines scores 0.3 on the head pass,
|
||||||
|
* skips the `< SCORING_HEAD_TRIGGER_THRESHOLD` rescore, and clears the
|
||||||
|
* 0.05 floor — mis-parsing prose as a conversation. Full-body scoring
|
||||||
|
* drops that page below the floor (3/200 ≈ 0.015 → no_match) while a
|
||||||
|
* real transcript stays ~1.0.
|
||||||
|
*/
|
||||||
|
score_full_body?: boolean;
|
||||||
/** D7: module-load validation — known-positive sample lines. */
|
/** D7: module-load validation — known-positive sample lines. */
|
||||||
test_positive: string[];
|
test_positive: string[];
|
||||||
/** D7: module-load validation — known-negative sample lines. */
|
/** D7: module-load validation — known-negative sample lines. */
|
||||||
|
|||||||
+13
-2
@@ -1168,12 +1168,23 @@ export interface BrainEngine {
|
|||||||
*/
|
*/
|
||||||
getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, number>>;
|
getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, number>>;
|
||||||
/**
|
/**
|
||||||
* Return every page with no inbound links (from any source).
|
* Return every page with no inbound links.
|
||||||
* Domain comes from the frontmatter `domain` field (null if unset).
|
* Domain comes from the frontmatter `domain` field (null if unset).
|
||||||
* The caller filters pseudo-pages + derives display domain.
|
* The caller filters pseudo-pages + derives display domain.
|
||||||
* Used by `gbrain orphans` and `runCycle`'s orphan sweep phase.
|
* Used by `gbrain orphans` and `runCycle`'s orphan sweep phase.
|
||||||
|
*
|
||||||
|
* v0.41.29.0: scopes the CANDIDATE set to one source (`sourceId`, from
|
||||||
|
* `gbrain doctor/orphans --source` + single-source MCP clients) or a
|
||||||
|
* federated set (`sourceIds`, from `allowedSources` MCP clients). The
|
||||||
|
* inbound-link side is NOT scoped — a page in source X linked FROM
|
||||||
|
* source Y is genuinely reachable, so it is not an orphan of X. Omit
|
||||||
|
* `opts` for the brain-wide behavior (unchanged). When both are set,
|
||||||
|
* `sourceIds` wins (mirrors `sourceScopeOpts` precedence).
|
||||||
*/
|
*/
|
||||||
findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>>;
|
findOrphanPages(opts?: {
|
||||||
|
sourceId?: string;
|
||||||
|
sourceIds?: string[];
|
||||||
|
}): Promise<Array<{ slug: string; title: string; domain: string | null }>>;
|
||||||
|
|
||||||
// Tags
|
// Tags
|
||||||
/**
|
/**
|
||||||
|
|||||||
+10
-1
@@ -2756,7 +2756,16 @@ const find_orphans: Operation = {
|
|||||||
scope: 'read',
|
scope: 'read',
|
||||||
handler: async (ctx, p) => {
|
handler: async (ctx, p) => {
|
||||||
const { findOrphans } = await import('../commands/orphans.ts');
|
const { findOrphans } = await import('../commands/orphans.ts');
|
||||||
return findOrphans(ctx.engine, { includePseudo: (p.include_pseudo as boolean) || false });
|
// v0.41.29.0 (Codex F8): scope by the caller's source (ctx.sourceId /
|
||||||
|
// ctx.auth.allowedSources) via the canonical sourceScopeOpts ladder.
|
||||||
|
// Pre-fix, find_orphans returned brain-wide orphans regardless of a
|
||||||
|
// source-bound OAuth client's scope — a read leak in the v0.34.1
|
||||||
|
// source-isolation class. Local CLI callers route through `gbrain
|
||||||
|
// orphans --source` instead (ctx.remote === false → empty scope here).
|
||||||
|
return findOrphans(ctx.engine, {
|
||||||
|
includePseudo: (p.include_pseudo as boolean) || false,
|
||||||
|
...sourceScopeOpts(ctx),
|
||||||
|
});
|
||||||
},
|
},
|
||||||
cliHints: { name: 'orphans', hidden: true },
|
cliHints: { name: 'orphans', hidden: true },
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -2791,12 +2791,30 @@ export class PGLiteEngine implements BrainEngine {
|
|||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
|
|
||||||
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
async findOrphanPages(opts?: {
|
||||||
|
sourceId?: string;
|
||||||
|
sourceIds?: string[];
|
||||||
|
}): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||||
// Soft-delete filter on BOTH sides:
|
// Soft-delete filter on BOTH sides:
|
||||||
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
||||||
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
||||||
// Without the link-source filter, a live page can hide from orphan results purely
|
// Without the link-source filter, a live page can hide from orphan results purely
|
||||||
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
||||||
|
//
|
||||||
|
// v0.41.29.0: scope ONLY the candidate side (`p.source_id`) when opts.sourceId
|
||||||
|
// is set. The inbound-link NOT EXISTS deliberately counts links from ANY source:
|
||||||
|
// a page in source X linked FROM source Y is reachable, so NOT an orphan of X.
|
||||||
|
// Do NOT add `src.source_id = p.source_id` here — that is the stricter
|
||||||
|
// intra-source-only definition we deliberately reject.
|
||||||
|
let sourceFilter = '';
|
||||||
|
const params: unknown[] = [];
|
||||||
|
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||||
|
params.push(opts.sourceIds);
|
||||||
|
sourceFilter = `AND p.source_id = ANY($${params.length}::text[])`;
|
||||||
|
} else if (opts?.sourceId) {
|
||||||
|
params.push(opts.sourceId);
|
||||||
|
sourceFilter = `AND p.source_id = $${params.length}`;
|
||||||
|
}
|
||||||
const { rows } = await this.db.query(
|
const { rows } = await this.db.query(
|
||||||
`SELECT
|
`SELECT
|
||||||
p.slug,
|
p.slug,
|
||||||
@@ -2804,6 +2822,7 @@ export class PGLiteEngine implements BrainEngine {
|
|||||||
p.frontmatter->>'domain' AS domain
|
p.frontmatter->>'domain' AS domain
|
||||||
FROM pages p
|
FROM pages p
|
||||||
WHERE p.deleted_at IS NULL
|
WHERE p.deleted_at IS NULL
|
||||||
|
${sourceFilter}
|
||||||
AND NOT EXISTS (
|
AND NOT EXISTS (
|
||||||
SELECT 1
|
SELECT 1
|
||||||
FROM links l
|
FROM links l
|
||||||
@@ -2811,7 +2830,8 @@ export class PGLiteEngine implements BrainEngine {
|
|||||||
WHERE l.to_page_id = p.id
|
WHERE l.to_page_id = p.id
|
||||||
AND src.deleted_at IS NULL
|
AND src.deleted_at IS NULL
|
||||||
)
|
)
|
||||||
ORDER BY p.slug`
|
ORDER BY p.slug`,
|
||||||
|
params
|
||||||
);
|
);
|
||||||
return rows as Array<{ slug: string; title: string; domain: string | null }>;
|
return rows as Array<{ slug: string; title: string; domain: string | null }>;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2857,13 +2857,28 @@ export class PostgresEngine implements BrainEngine {
|
|||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
|
|
||||||
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
async findOrphanPages(opts?: {
|
||||||
|
sourceId?: string;
|
||||||
|
sourceIds?: string[];
|
||||||
|
}): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||||
const sql = this.sql;
|
const sql = this.sql;
|
||||||
// Soft-delete filter on BOTH sides:
|
// Soft-delete filter on BOTH sides:
|
||||||
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
||||||
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
||||||
// Without the link-source filter, a live page can hide from orphan results purely
|
// Without the link-source filter, a live page can hide from orphan results purely
|
||||||
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
||||||
|
//
|
||||||
|
// v0.41.29.0: scope ONLY the candidate side (`p.source_id`) when opts.sourceId
|
||||||
|
// is set. The inbound-link NOT EXISTS deliberately counts links from ANY source:
|
||||||
|
// a page in source X linked FROM source Y is reachable, so NOT an orphan of X.
|
||||||
|
// Do NOT add `src.source_id = p.source_id` here — that would be the stricter
|
||||||
|
// intra-source-only definition we deliberately reject.
|
||||||
|
const sourceFilter =
|
||||||
|
opts?.sourceIds && opts.sourceIds.length > 0
|
||||||
|
? sql`AND p.source_id = ANY(${opts.sourceIds}::text[])`
|
||||||
|
: opts?.sourceId
|
||||||
|
? sql`AND p.source_id = ${opts.sourceId}`
|
||||||
|
: sql``;
|
||||||
const rows = await sql`
|
const rows = await sql`
|
||||||
SELECT
|
SELECT
|
||||||
p.slug,
|
p.slug,
|
||||||
@@ -2871,6 +2886,7 @@ export class PostgresEngine implements BrainEngine {
|
|||||||
p.frontmatter->>'domain' AS domain
|
p.frontmatter->>'domain' AS domain
|
||||||
FROM pages p
|
FROM pages p
|
||||||
WHERE p.deleted_at IS NULL
|
WHERE p.deleted_at IS NULL
|
||||||
|
${sourceFilter}
|
||||||
AND NOT EXISTS (
|
AND NOT EXISTS (
|
||||||
SELECT 1
|
SELECT 1
|
||||||
FROM links l
|
FROM links l
|
||||||
|
|||||||
@@ -403,21 +403,21 @@ describe('scorePatternFull — full-body scoring (v0.41.18+ Codex P1 #1)', () =>
|
|||||||
describe('bold-paren-time pattern (Circleback meeting transcripts)', () => {
|
describe('bold-paren-time pattern (Circleback meeting transcripts)', () => {
|
||||||
test('matches **Speaker** (HH:MM): text with frontmatter date', () => {
|
test('matches **Speaker** (HH:MM): text with frontmatter date', () => {
|
||||||
const body = [
|
const body = [
|
||||||
'**Garry Tan** (00:00): Hey, can you hear me?',
|
'**Alice Example** (00:00): Hey, can you hear me?',
|
||||||
'**Participant 2** (02:22): Yeah, just joined.',
|
'**Participant 2** (02:22): Yeah, just joined.',
|
||||||
'**Garry Tan** (15:09): That makes sense.',
|
'**Alice Example** (15:09): That makes sense.',
|
||||||
].join('\n');
|
].join('\n');
|
||||||
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
|
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
|
||||||
expect(r.phase).toBe('regex_match');
|
expect(r.phase).toBe('regex_match');
|
||||||
expect(r.matched_pattern_id).toBe('bold-paren-time');
|
expect(r.matched_pattern_id).toBe('bold-paren-time');
|
||||||
expect(r.messages).toHaveLength(3);
|
expect(r.messages).toHaveLength(3);
|
||||||
expect(r.messages[0]).toEqual({
|
expect(r.messages[0]).toEqual({
|
||||||
speaker: 'Garry Tan',
|
speaker: 'Alice Example',
|
||||||
timestamp: '2026-03-19T00:00:00Z',
|
timestamp: '2026-03-19T00:00:00Z',
|
||||||
text: 'Hey, can you hear me?',
|
text: 'Hey, can you hear me?',
|
||||||
});
|
});
|
||||||
expect(r.messages[2]).toEqual({
|
expect(r.messages[2]).toEqual({
|
||||||
speaker: 'Garry Tan',
|
speaker: 'Alice Example',
|
||||||
timestamp: '2026-03-19T15:09:00Z',
|
timestamp: '2026-03-19T15:09:00Z',
|
||||||
text: 'That makes sense.',
|
text: 'That makes sense.',
|
||||||
});
|
});
|
||||||
@@ -478,6 +478,113 @@ describe('bold-paren-time pattern (Circleback meeting transcripts)', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// bold-name-no-time pattern (Circleback / Granola / Zoom transcripts with NO
|
||||||
|
// per-line timestamp — `**Speaker:** text`). Additive pattern; the colon
|
||||||
|
// inside the bold markers + the `(?!\[)` lookahead are what keep it from
|
||||||
|
// shadowing bold-paren-time / telegram-bracket (NOT declaration order).
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
describe('bold-name-no-time pattern (Circleback/Granola/Zoom, no timestamp)', () => {
|
||||||
|
test('parses **Speaker:** text transcript with frontmatter date anchor', () => {
|
||||||
|
const body = [
|
||||||
|
'**Alice Example:** Okay, start on. And then weirdly like zoom doesn’t...',
|
||||||
|
'**Participant 2:** he tried to reset it remotely the other night. Let me ask him.',
|
||||||
|
'**Alice Example:** I mean it’s really just like we need to get zoom to fix this.',
|
||||||
|
'**Participant 2:** Okay, let me.',
|
||||||
|
].join('\n');
|
||||||
|
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
|
||||||
|
expect(r.phase).toBe('regex_match');
|
||||||
|
expect(r.matched_pattern_id).toBe('bold-name-no-time');
|
||||||
|
expect(r.messages).toHaveLength(4);
|
||||||
|
expect(r.messages[0]).toEqual({
|
||||||
|
speaker: 'Alice Example',
|
||||||
|
timestamp: '2026-05-28T00:00:00Z',
|
||||||
|
text: 'Okay, start on. And then weirdly like zoom doesn’t...',
|
||||||
|
});
|
||||||
|
expect(r.messages[1].speaker).toBe('Participant 2');
|
||||||
|
expect(r.messages[1].text).toBe(
|
||||||
|
'he tried to reset it remotely the other night. Let me ask him.',
|
||||||
|
);
|
||||||
|
expect(r.messages[3]).toEqual({
|
||||||
|
speaker: 'Participant 2',
|
||||||
|
timestamp: '2026-05-28T00:00:00Z',
|
||||||
|
text: 'Okay, let me.',
|
||||||
|
});
|
||||||
|
// No-time pattern anchors at 00:00:00 of the frontmatter date
|
||||||
|
// (same convention as irc-classic). No wall-clock time fabricated.
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scores above the 0.05 floor on a pure bold-name transcript (epoch default)', () => {
|
||||||
|
const body = [
|
||||||
|
'**Alice Example:** line one',
|
||||||
|
'**Participant 2:** line two',
|
||||||
|
'**Alice Example:** line three',
|
||||||
|
].join('\n');
|
||||||
|
const r = parseConversation(body);
|
||||||
|
expect(r.phase).toBe('regex_match');
|
||||||
|
expect(r.matched_pattern_id).toBe('bold-name-no-time');
|
||||||
|
expect(r.messages).toHaveLength(3);
|
||||||
|
// No fallbackDate → epoch default, still anchors at 00:00:00.
|
||||||
|
expect(r.messages[0].timestamp).toBe('1970-01-01T00:00:00Z');
|
||||||
|
});
|
||||||
|
|
||||||
|
// REGRESSION: must NOT shadow bold-paren-time. A `**Name** (00:00): text`
|
||||||
|
// line has its colon OUTSIDE the bold markers, so it must still parse via
|
||||||
|
// bold-paren-time (the safety is the regex, not declaration order).
|
||||||
|
test('REGRESSION: **Speaker** (HH:MM): text still matches bold-paren-time', () => {
|
||||||
|
const body = [
|
||||||
|
'**Alice Example** (00:00): Hey, can you hear me?',
|
||||||
|
'**Participant 2** (02:22): Yeah, just joined.',
|
||||||
|
].join('\n');
|
||||||
|
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
|
||||||
|
expect(r.phase).toBe('regex_match');
|
||||||
|
expect(r.matched_pattern_id).toBe('bold-paren-time');
|
||||||
|
expect(r.matched_pattern_id).not.toBe('bold-name-no-time');
|
||||||
|
expect(r.messages).toHaveLength(2);
|
||||||
|
expect(r.messages[0].timestamp).toBe('2026-03-19T00:00:00Z');
|
||||||
|
});
|
||||||
|
|
||||||
|
// REGRESSION: a pure telegram-bracket transcript has the colon inside the
|
||||||
|
// bold markers BUT the speaker starts with `[`, so the `(?!\[)` lookahead
|
||||||
|
// rejects it AND telegram-bracket (lower declaration index) wins anyway.
|
||||||
|
test('REGRESSION: telegram-bracket transcript still matches telegram-bracket', () => {
|
||||||
|
const body = [
|
||||||
|
'**[18:37] \u{1f464} Alice:** one',
|
||||||
|
'**[18:38] \u{1f464} Bob:** two',
|
||||||
|
].join('\n');
|
||||||
|
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
|
||||||
|
expect(r.matched_pattern_id).toBe('telegram-bracket');
|
||||||
|
});
|
||||||
|
|
||||||
|
// F1 (Codex): the broad regex matches any `**Label:** text`. A prose notes
|
||||||
|
// page with bold labels CLUSTERED in its first 10 lines scores 0.3 on the
|
||||||
|
// head pass (3/10) — NOT < SCORING_HEAD_TRIGGER_THRESHOLD (0.3), so the
|
||||||
|
// full-body fallback never fires — and that 0.3 clears the 0.05 floor.
|
||||||
|
// Without score_full_body this page would mis-parse as a conversation.
|
||||||
|
// score_full_body recomputes the winner over the FULL body: 3 labels /
|
||||||
|
// 83 lines ≈ 0.036 < 0.05 → no_match. This is the exact bug Codex caught
|
||||||
|
// and the test the naive "0.05 floor protects us" assumption would fail.
|
||||||
|
test('F1: prose with bold labels clustered in the head returns no_match', () => {
|
||||||
|
const lines = [
|
||||||
|
'**Attendees:** Alice Example, Bob Example, Participant 2',
|
||||||
|
'**Date:** 2026-05-28',
|
||||||
|
'**Goal:** decide on the Q3 roadmap and unblock the vendor migration',
|
||||||
|
];
|
||||||
|
// 80 plain prose lines (no bold-label shape) → 83 total, only 3 match.
|
||||||
|
// First 10 lines = 3 labels + 7 prose → head score 0.3 (skips rescore).
|
||||||
|
// Full body = 3/83 ≈ 0.036 < 0.05 floor.
|
||||||
|
for (let i = 0; i < 80; i++) {
|
||||||
|
lines.push(
|
||||||
|
`This is an ordinary prose sentence number ${i} describing the meeting in detail.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const body = lines.join('\n');
|
||||||
|
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
|
||||||
|
expect(r.phase).toBe('no_match');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// parseConversation — full-body fallback (v0.41.18+ #1533 + Codex P1 #1, #2, #8)
|
// parseConversation — full-body fallback (v0.41.18+ #1533 + Codex P1 #1, #2, #8)
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -363,4 +363,53 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
|
|||||||
expect(pgMap.get('wiki/rsp-missing.md')).toBeUndefined();
|
expect(pgMap.get('wiki/rsp-missing.md')).toBeUndefined();
|
||||||
expect(pgliteMap.get('wiki/rsp-missing.md')).toBeUndefined();
|
expect(pgliteMap.get('wiki/rsp-missing.md')).toBeUndefined();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// v0.41.29.0 — findOrphanPages source scoping parity. Real Postgres
|
||||||
|
// coverage for the postgres.js `sql` scalar fragment + `= ANY(${arr}::text[])`
|
||||||
|
// array binding (a documented footgun class — the jsonb double-encode saga).
|
||||||
|
// PGLite logic is pinned in test/orphans-source-scope.test.ts; this asserts
|
||||||
|
// the Postgres SQL produces the same scoped sets. Cross-source inbound
|
||||||
|
// (src-b → src-a) must NOT make the target an orphan of src-a (A2).
|
||||||
|
test('v0.41.29.0 findOrphanPages source scoping parity (scalar + federated)', async () => {
|
||||||
|
const srcSql = `INSERT INTO sources (id, name, config) VALUES ($1, $1, '{}'::jsonb) ON CONFLICT DO NOTHING`;
|
||||||
|
const pageSql = `
|
||||||
|
INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter)
|
||||||
|
VALUES ($1, $2, 'person', 't', 'b', '', '{}'::jsonb)
|
||||||
|
ON CONFLICT (source_id, slug) DO NOTHING
|
||||||
|
`;
|
||||||
|
for (const eng of [pgEngine, pgliteEngine]) {
|
||||||
|
await eng.executeRaw(srcSql, ['orphan-src-a']);
|
||||||
|
await eng.executeRaw(srcSql, ['orphan-src-b']);
|
||||||
|
await eng.executeRaw(pageSql, ['orphan-src-a', 'people/op-orphan-a']);
|
||||||
|
await eng.executeRaw(pageSql, ['orphan-src-a', 'people/op-target-a']);
|
||||||
|
await eng.executeRaw(pageSql, ['orphan-src-b', 'people/op-linker-b']);
|
||||||
|
// Cross-source inbound: src-b page → src-a target (A2).
|
||||||
|
await eng.addLink(
|
||||||
|
'people/op-linker-b', 'people/op-target-a', '', 'mentions', 'markdown',
|
||||||
|
undefined, undefined, { fromSourceId: 'orphan-src-b', toSourceId: 'orphan-src-a' },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const scoped = async (eng: BrainEngine, opts: { sourceId?: string; sourceIds?: string[] }) =>
|
||||||
|
(await eng.findOrphanPages(opts)).map(r => r.slug).filter(s => s.startsWith('people/op-')).sort();
|
||||||
|
|
||||||
|
// Scalar scope to src-a: op-orphan-a is an orphan; op-target-a is saved
|
||||||
|
// by the cross-source inbound (A2). Parity on both engines.
|
||||||
|
const pgA = await scoped(pgEngine, { sourceId: 'orphan-src-a' });
|
||||||
|
const pgliteA = await scoped(pgliteEngine, { sourceId: 'orphan-src-a' });
|
||||||
|
expect(pgA).toEqual(['people/op-orphan-a']);
|
||||||
|
expect(pgliteA).toEqual(pgA);
|
||||||
|
|
||||||
|
// Scalar scope to src-b.
|
||||||
|
const pgB = await scoped(pgEngine, { sourceId: 'orphan-src-b' });
|
||||||
|
const pgliteB = await scoped(pgliteEngine, { sourceId: 'orphan-src-b' });
|
||||||
|
expect(pgB).toEqual(['people/op-linker-b']);
|
||||||
|
expect(pgliteB).toEqual(pgB);
|
||||||
|
|
||||||
|
// Federated array scope (= ANY binding) → union.
|
||||||
|
const pgFed = await scoped(pgEngine, { sourceIds: ['orphan-src-a', 'orphan-src-b'] });
|
||||||
|
const pgliteFed = await scoped(pgliteEngine, { sourceIds: ['orphan-src-a', 'orphan-src-b'] });
|
||||||
|
expect(pgFed).toEqual(['people/op-linker-b', 'people/op-orphan-a']);
|
||||||
|
expect(pgliteFed).toEqual(pgFed);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,2 @@
|
|||||||
|
{"fixture_id":"bold-name-no-time-001","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** Okay, start on. And then weirdly like zoom doesn’t work.\n**Bob Example:** he tried to reset it remotely the other night. Let me ask him.\n**Alice Example:** I mean it’s really just like we need to get zoom to fix this.\n**Bob Example:** Okay, let me.","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}
|
||||||
|
{"fixture_id":"bold-name-no-time-002","pattern":"bold-name-no-time","frontmatter":{"date":"2026-05-28"},"body":"**Alice Example:** can you hear me now\n**Participant 2:** yeah loud and clear\n**Alice Example:** great, let us start the review","expected_messages":3,"expected_participants":["Alice Example","Participant 2"]}
|
||||||
@@ -0,0 +1,213 @@
|
|||||||
|
/**
|
||||||
|
* v0.41.29.0 — orphan source-scoping regression suite.
|
||||||
|
*
|
||||||
|
* Covers three Codex-flagged behaviors of the `--source` / find_orphans
|
||||||
|
* source-scoping wave, all at the layer the bugs live (engine +
|
||||||
|
* getOrphansData + the find_orphans op handler):
|
||||||
|
*
|
||||||
|
* - A2: candidate-only scoping. A page in source X linked FROM source Y
|
||||||
|
* is reachable, so it is NOT an orphan of X (cross-source inbound counts).
|
||||||
|
* - F6: total_linkable denominator. Excluded NON-orphan pages (e.g. a
|
||||||
|
* `templates/` page that HAS inbound links) must NOT inflate the
|
||||||
|
* denominator. Pre-fix the formula `total - excludedOrphans` left them in.
|
||||||
|
* - F8: the find_orphans MCP op handler scopes by ctx.sourceId AND
|
||||||
|
* ctx.auth.allowedSources (the v0.34.1 source-isolation read leak).
|
||||||
|
*
|
||||||
|
* Runs against PGLite in-memory (both engines share the SQL surface; the
|
||||||
|
* Postgres path is pinned separately in test/e2e/multi-source-bug-class).
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||||
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||||
|
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||||
|
import { findOrphans } from '../src/commands/orphans.ts';
|
||||||
|
|
||||||
|
let engine: PGLiteEngine;
|
||||||
|
|
||||||
|
const page = (title: string) => ({
|
||||||
|
type: 'person' as const,
|
||||||
|
title,
|
||||||
|
compiled_truth: `${title} body.`,
|
||||||
|
timeline: '',
|
||||||
|
frontmatter: {},
|
||||||
|
});
|
||||||
|
|
||||||
|
// Every test in this file is READ-ONLY (findOrphanPages / getOrphansData /
|
||||||
|
// find_orphans op handler). Seed ONCE in beforeAll rather than reset+reseed
|
||||||
|
// per test — the canonical R3/R4 pattern (engine in beforeAll, disconnect in
|
||||||
|
// afterAll) is satisfied, and this cuts the file from ~6min to seconds.
|
||||||
|
beforeAll(async () => {
|
||||||
|
engine = new PGLiteEngine();
|
||||||
|
await engine.connect({});
|
||||||
|
await engine.initSchema();
|
||||||
|
await resetPgliteState(engine);
|
||||||
|
await engine.executeRaw(
|
||||||
|
`INSERT INTO sources (id, name, config) VALUES ('src-b', 'src-b', '{}'::jsonb) ON CONFLICT DO NOTHING`,
|
||||||
|
);
|
||||||
|
|
||||||
|
// --- default source ---
|
||||||
|
// alice-orphan: no inbound → orphan.
|
||||||
|
await engine.putPage('people/alice-orphan', page('Alice Orphan'), { sourceId: 'default' });
|
||||||
|
// bob-linked: inbound from alice-orphan → NOT orphan.
|
||||||
|
await engine.putPage('people/bob-linked', page('Bob Linked'), { sourceId: 'default' });
|
||||||
|
// cross-target: inbound ONLY from a src-b page → A2: NOT orphan of default.
|
||||||
|
await engine.putPage('people/cross-target', page('Cross Target'), { sourceId: 'default' });
|
||||||
|
// templates/junk-linked: excluded prefix, HAS inbound → not orphan, and must
|
||||||
|
// be excluded from total_linkable (F6).
|
||||||
|
await engine.putPage('templates/junk-linked', page('Junk Template'), { sourceId: 'default' });
|
||||||
|
|
||||||
|
// --- src-b source ---
|
||||||
|
// zara-orphan: no inbound → orphan.
|
||||||
|
await engine.putPage('people/zara-orphan', page('Zara Orphan'), { sourceId: 'src-b' });
|
||||||
|
// src-b-linker: links to default's cross-target; itself has no inbound → orphan.
|
||||||
|
await engine.putPage('people/src-b-linker', page('Src-B Linker'), { sourceId: 'src-b' });
|
||||||
|
|
||||||
|
// --- links ---
|
||||||
|
await engine.addLink(
|
||||||
|
'people/alice-orphan', 'people/bob-linked', '', 'mentions', 'markdown',
|
||||||
|
undefined, undefined, { fromSourceId: 'default', toSourceId: 'default' },
|
||||||
|
);
|
||||||
|
await engine.addLink(
|
||||||
|
'people/alice-orphan', 'templates/junk-linked', '', 'mentions', 'markdown',
|
||||||
|
undefined, undefined, { fromSourceId: 'default', toSourceId: 'default' },
|
||||||
|
);
|
||||||
|
// Cross-source inbound: src-b page → default page (A2).
|
||||||
|
await engine.addLink(
|
||||||
|
'people/src-b-linker', 'people/cross-target', '', 'mentions', 'markdown',
|
||||||
|
undefined, undefined, { fromSourceId: 'src-b', toSourceId: 'default' },
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterAll(async () => {
|
||||||
|
await engine.disconnect();
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('findOrphanPages — source scoping (A2)', () => {
|
||||||
|
test('scoped to default: alice-orphan is an orphan; cross-source-linked page is NOT', async () => {
|
||||||
|
const rows = await engine.findOrphanPages({ sourceId: 'default' });
|
||||||
|
const slugs = rows.map(r => r.slug);
|
||||||
|
expect(slugs).toContain('people/alice-orphan');
|
||||||
|
// A2: cross-target has inbound from src-b → reachable → NOT an orphan.
|
||||||
|
expect(slugs).not.toContain('people/cross-target');
|
||||||
|
// bob-linked + templates/junk-linked have intra-source inbound.
|
||||||
|
expect(slugs).not.toContain('people/bob-linked');
|
||||||
|
expect(slugs).not.toContain('templates/junk-linked');
|
||||||
|
// src-b pages must not appear in a default-scoped scan.
|
||||||
|
expect(slugs).not.toContain('people/zara-orphan');
|
||||||
|
expect(slugs).not.toContain('people/src-b-linker');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('scoped to src-b returns a different orphan set', async () => {
|
||||||
|
const rows = await engine.findOrphanPages({ sourceId: 'src-b' });
|
||||||
|
const slugs = rows.map(r => r.slug).sort();
|
||||||
|
expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('unscoped (brain-wide) returns the union', async () => {
|
||||||
|
const rows = await engine.findOrphanPages();
|
||||||
|
const slugs = rows.map(r => r.slug).sort();
|
||||||
|
expect(slugs).toEqual([
|
||||||
|
'people/alice-orphan',
|
||||||
|
'people/src-b-linker',
|
||||||
|
'people/zara-orphan',
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('federated sourceIds scopes to the union of the given sources', async () => {
|
||||||
|
const rows = await engine.findOrphanPages({ sourceIds: ['default', 'src-b'] });
|
||||||
|
const slugs = rows.map(r => r.slug).sort();
|
||||||
|
expect(slugs).toEqual([
|
||||||
|
'people/alice-orphan',
|
||||||
|
'people/src-b-linker',
|
||||||
|
'people/zara-orphan',
|
||||||
|
]);
|
||||||
|
// Single-element federated array behaves like the scalar scope.
|
||||||
|
const onlyB = await engine.findOrphanPages({ sourceIds: ['src-b'] });
|
||||||
|
expect(onlyB.map(r => r.slug).sort()).toEqual([
|
||||||
|
'people/src-b-linker',
|
||||||
|
'people/zara-orphan',
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('getOrphansData — scoped totals + F6 denominator', () => {
|
||||||
|
test('default scope: counts differ from brain-wide; excluded non-orphan drops from total_linkable (F6)', async () => {
|
||||||
|
const data = await findOrphans(engine, { includePseudo: false, sourceId: 'default' });
|
||||||
|
// 4 live default pages; 1 orphan (alice-orphan).
|
||||||
|
expect(data.total_pages).toBe(4);
|
||||||
|
expect(data.total_orphans).toBe(1);
|
||||||
|
expect(data.orphans.map(o => o.slug)).toEqual(['people/alice-orphan']);
|
||||||
|
// F6: templates/junk-linked is an EXCLUDED page that HAS an inbound link.
|
||||||
|
// It must NOT count in total_linkable. NEW denominator = total(4) -
|
||||||
|
// excludedAll(1) = 3. The pre-fix formula (total - excludedOrphans) left
|
||||||
|
// the excluded non-orphan in and returned 4.
|
||||||
|
expect(data.total_linkable).toBe(3);
|
||||||
|
expect(data.total_linkable).toBe(data.total_pages - 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('src-b scope: distinct from default', async () => {
|
||||||
|
const data = await findOrphans(engine, { includePseudo: false, sourceId: 'src-b' });
|
||||||
|
expect(data.total_pages).toBe(2);
|
||||||
|
expect(data.total_orphans).toBe(2);
|
||||||
|
expect(data.total_linkable).toBe(2); // no excluded pages in src-b
|
||||||
|
});
|
||||||
|
|
||||||
|
test('brain-wide F6: excluded non-orphan drops from total_linkable', async () => {
|
||||||
|
const data = await findOrphans(engine, { includePseudo: false });
|
||||||
|
expect(data.total_pages).toBe(6);
|
||||||
|
expect(data.total_orphans).toBe(3);
|
||||||
|
// 6 live pages - 1 excluded (templates/junk-linked) = 5.
|
||||||
|
expect(data.total_linkable).toBe(5);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('includePseudo keeps excluded pages in the denominator', async () => {
|
||||||
|
const data = await findOrphans(engine, { includePseudo: true, sourceId: 'default' });
|
||||||
|
// No exclusion → denominator is the full live set.
|
||||||
|
expect(data.total_linkable).toBe(4);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('find_orphans MCP op — F8 source-isolation scope', () => {
|
||||||
|
test('ctx.sourceId scopes results to that source only', async () => {
|
||||||
|
const { operations } = await import('../src/core/operations.ts');
|
||||||
|
const op = operations.find(o => o.name === 'find_orphans');
|
||||||
|
expect(op).toBeDefined();
|
||||||
|
const ctx = {
|
||||||
|
engine,
|
||||||
|
config: { engine: 'pglite' as const },
|
||||||
|
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||||
|
dryRun: false,
|
||||||
|
remote: true,
|
||||||
|
sourceId: 'src-b',
|
||||||
|
};
|
||||||
|
const result = (await op!.handler(ctx as any, {})) as { orphans: { slug: string }[] };
|
||||||
|
const slugs = result.orphans.map(o => o.slug).sort();
|
||||||
|
expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']);
|
||||||
|
// Leak guard: default's orphan must NOT appear.
|
||||||
|
expect(slugs).not.toContain('people/alice-orphan');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('ctx.auth.allowedSources (federated) scopes to the allowed set', async () => {
|
||||||
|
const { operations } = await import('../src/core/operations.ts');
|
||||||
|
const op = operations.find(o => o.name === 'find_orphans');
|
||||||
|
const ctx = {
|
||||||
|
engine,
|
||||||
|
config: { engine: 'pglite' as const },
|
||||||
|
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||||
|
dryRun: false,
|
||||||
|
remote: true,
|
||||||
|
sourceId: 'default', // scalar would scope to default-only...
|
||||||
|
auth: {
|
||||||
|
token: 'test',
|
||||||
|
clientId: 'test',
|
||||||
|
scopes: ['read'],
|
||||||
|
sourceId: 'default',
|
||||||
|
allowedSources: ['src-b'], // ...but the federated array wins.
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const result = (await op!.handler(ctx as any, {})) as { orphans: { slug: string }[] };
|
||||||
|
const slugs = result.orphans.map(o => o.slug).sort();
|
||||||
|
expect(slugs).toEqual(['people/src-b-linker', 'people/zara-orphan']);
|
||||||
|
expect(slugs).not.toContain('people/alice-orphan');
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user