Files
gbrain/test/e2e/global-basename-pglite.test.ts
T
f09f9177a9 v0.42.10.0 feat(extract): opt-in global-basename wikilink resolution (closes #972) (#1388)
* v0.40.8.2 fix(extract): opt-in global-basename wikilink resolution (#972)

Bare wikilinks like [[struktura]] that point at pages in another folder
were silently dropped from the graph. The issue reporter saw 71 wikilinks
in Obsidian render to 12 in gbrain (~83% lost). Symptoms downstream:
`gbrain graph` returns thin neighborhoods, `gbrain backlinks` undercounts.

This release adds an opt-in mode that resolves bare wikilinks by basename
match, covers all three resolver surfaces (FS-source extract, DB-source
extract, put_page auto-link), and emits one edge per match — no silent
winner on ambiguity. `gbrain doctor` surfaces a paste-ready enable hint
when ≥5 bare wikilinks would resolve under the new mode.

Enable with:
  gbrain config set link_resolution.global_basename true
  gbrain extract links

Default stays off. Existing brains see zero behavior change on upgrade.

Closes #972. Adapts PR #1233 from @rayers (regex shape + slug-tail index)
into a multi-match, opt-in form with FS-source coverage that the original
PR explicitly skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: document opt-in global-basename wikilink resolution (#972)

The #972 feature shipped with no user-facing docs — only CHANGELOG + CLAUDE.md.
Anyone migrating an Obsidian/Notion vault with bare [[name]] wikilinks couldn't
discover the link_resolution.global_basename flag unless gbrain doctor happened
to surface its hint.

- README "Self-wiring knowledge graph": one sentence on the opt-in mode for
  Obsidian-style cross-folder bare wikilinks + the doctor pre-check, linking to
  the install step.
- INSTALL_FOR_AGENTS Step 4.5 (Wire the Knowledge Graph): a dedicated agent-
  facing subsection — when bare [[name]] links need it, the enable command,
  re-running extract, the doctor opportunity hint, and the multi-match behavior.
- Regenerated llms-full.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): resolve aliased wikilinks by target slug, not display text

Codex outside-voice [P1]: `[[struktura|the project]]` resolved the basename
"the project" (the alias) instead of `struktura` (the target), because
extractPageLinks called resolveBasenameMatches(ref.name) and the doctor check
keyed basenameIndex.get(e.name). ref.name is the display alias (match[2]);
ref.slug is the wikilink target (match[1]).

- extractPageLinks resolves ref.slug; context excerpt locates ref.slug.
- doctor link_resolution_opportunity keys e.slug so its estimate matches
  what extraction actually resolves.
- Test: aliased wikilink calls resolveBasenameMatches with the target, never
  the display text.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): reconcile wikilink-resolved edges in put_page auto-link

Codex outside-voice [P1]: put_page's reconcilableOut filter excluded
link_source='wikilink-resolved', so a basename edge written by auto-link
survived after the bare wikilink was deleted from the page OR the
link_resolution.global_basename flag was turned off (the stale-removal loop
only iterates reconcilableOut). Add 'wikilink-resolved' to the reconcilable
set; manual edges still untouched.

Test: write page with [[struktura]] (flag on) → edge lands; re-put without
the wikilink → edge reconciled away.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): source-scope basename resolution (no cross-source edges)

Codex outside-voice [P1]: makeResolver.resolveBasenameMatches called
engine.getAllSlugs() unscoped, so a bare [[name]] could resolve to a
same-tail page in a DIFFERENT source and create a cross-source edge. The
engine exposes getAllSlugs({sourceId}) precisely to prevent this. #972 is
"global basename across folders," not "cross-source federation" — the
canonical gbrain multi-source bug class.

- makeResolver gains opts.sourceId; ensureBasenameIndex passes it to
  getAllSlugs (unscoped only when sourceId omitted — back-compat).
- runAutoLink (put_page) passes opts.sourceId; extractLinksFromDB passes
  sourceIdFilter. FS extract is already single-source (walks one dir).
- Tests: scoped index returns only the source's slugs (no cross-source);
  unscoped call stays brain-wide.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): FS-source basename edges carry link_source='wikilink-resolved'

The FS extract path is the issue's default repro (gbrain extract links with no
--source db). ExtractedLink had no link_source field, so FS basename edges
landed with the engine default ('markdown') instead of the 'wikilink-resolved'
provenance the DB / put_page paths set and the docs promise. The e2e FS test
only asserted link_type, so it was blind to this.

- ExtractedLink gains link_source?; extractLinksFromFile sets it to
  'wikilink-resolved' on basename edges (undefined for ordinary markdown).
- Carries through the addLinksBatch snapshots automatically (LinkBatchInput
  already has link_source); single-row addLink fallback now passes it too.
- e2e FS repro asserts link_source === 'wikilink-resolved'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(#972): one shared basename matcher across resolver/FS/doctor

Codex outside-voice [P2] DRY: three surfaces each hand-rolled a basename
matcher with divergent key sets — the doctor omitted the slugified key, so its
link_resolution_opportunity estimate undercounted what extraction resolves, and
the resolver returned matches in unsorted getAllSlugs bucket order.

New shared exports in link-extraction.ts: buildBasenameIndex(slugs) +
queryBasenameIndex(index, name) (keys raw/lower/slugified tail; stable sort
shorter-first then lexical) + normalizeBasename.

- makeResolver.resolveBasenameMatches → queryBasenameIndex (now stable-sorted).
- extract.ts resolveBasenameMatchesFromSlugs → delegates to the shared pair.
- doctor link_resolution_opportunity → shared builder/query (slugified key
  added; estimate now matches extraction).
- Test: doctor counts a slugified-only match ([[Fast Weigh]] → companies/fast-weigh).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): P2 cluster — masking, code-fence, self-link, dedup decision

Codex outside-voice P2 findings:
- P2a markdown-label masking: a wikilink inside a markdown-link label
  ([see [[acme]]](companies/acme.md)) spawned a stray generic basename ref.
  Pass-1 can't match the nested brackets, so a new MARKDOWN_LABEL_WIKILINK_RE
  masks those spans out of pass 2c. Inner [[acme]] is now inert.
- P2b FS code-fence: the FS path (extractMarkdownLinks on raw content) didn't
  strip code blocks like the DB path. extractLinksFromFile now scans
  stripCodeBlocks(content) so [[name]] inside a fence creates no FS edge.
- P2c self-link guard: a basename [[own-tail]] on its own page resolved back
  to itself. Dropped in both extractPageLinks and the FS path.
- P2d dedup: documented the decision to KEEP qualified + bare edges to the
  same target as separate rows (distinct provenance/audit trail).
- P2e: skipFrontmatter unresolved-contract tests added.

Tests: P2a inert-label, P2c self-link drop, P2b code-fence, P2e unresolved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(#972): bound the doctor link_resolution_opportunity scan

The check did listAllPageRefs() + a getPage() per page under a 60s budget.
On a large brain (the eng-review concern) it hit the budget every non-fast
doctor run and returned a perpetual partial, adding ~60s.

Now batch-loads the 1000 most-recent pages in ONE query
(ORDER BY id DESC LIMIT SAMPLE_LIMIT) and scans in memory, with the 60s cap
kept as a backstop. Mirrors the v0.40.9 sampling convention. The estimate
message names the bound when the brain exceeds the sample
("scanned the 1000 most-recent of N pages").

Test: source-grep pins the bounded query + the absence of the per-page
getPage walk.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(#972): reconcile stale version/migration references to v112 / 0.42.6.0

Merge churn left intermediate refs: schema.sql + schema-embedded.ts said
"migration v93", CLAUDE.md said "v0.41.32.0 / Migration v109", CHANGELOG said
"Migration v93". Reconciled all to migration v112 / shipping 0.42.6.0. The
CLAUDE.md annotation is also refreshed to describe the final behavior (shared
matcher, source-scoping, alias-by-target, stale-edge reconciliation, bounded
doctor scan) and credit @rayers + @ukd1. Regenerated schema-embedded + llms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#972): register doctor check category + bump llms budget to 800KB

Two full-suite gate failures from the re-sync:
- doctor-categories drift guard: the new `link_resolution_opportunity` check
  wasn't in any category set. Added to BRAIN_CHECK_NAMES (alongside
  graph_coverage / orphan_ratio — it's a graph-quality signal).
- build-llms size budget: the #972 Key Files annotation (landing with master's
  #1696/#1699 waves) pushed llms-full.txt past 750KB. Bumped FULL_SIZE_BUDGET
  750KB→800KB, the established "budget tracks CLAUDE.md's legitimate per-feature
  growth" pattern (600→700→750→800 across releases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-06-02 16:33:06 -07:00

283 lines
11 KiB
TypeScript

/**
* Issue #972 E2E — opt-in global-basename wikilink resolution.
*
* Reproduces the issue's exact repro inside an in-memory PGLite brain:
*
* /vault/projects/struktura.md ← a real page
* /vault/concepts/knowledge-graph.md ← contains `[[struktura]]`
*
* The bare wikilink `[[struktura]]` does NOT match WIKILINK_RE
* (DIR_PATTERN-gated) so it falls through to the new pass 2c.
*
* Three contracts:
* 1. Flag OFF (default) — extract emits ZERO basename edges (back-compat
* with pre-issue-#972 behavior: bare wikilinks outside DIR_PATTERN
* drop silently).
* 2. Flag ON — extract emits ONE edge per basename match, tagged
* `link_type: 'wikilink_basename'`.
* 3. Ambiguous basename (same name in two directories) — extract emits
* ONE edge per match. No silent winner-takes-all, no silent drop.
*
* Both DB-source AND FS-source paths covered. PGLite in-memory, no
* DATABASE_URL needed.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { runExtract } from '../../src/commands/extract.ts';
let engine: PGLiteEngine;
let brainDir: string;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
async function truncateAll() {
for (const t of [
'content_chunks', 'links', 'tags', 'raw_data',
'timeline_entries', 'page_versions', 'ingest_log', 'pages',
'config',
]) {
try { await (engine as any).db.exec(`DELETE FROM ${t}`); } catch { /* ok */ }
}
}
beforeEach(async () => {
await truncateAll();
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-issue-972-'));
}, 15_000);
function writeFile(rel: string, content: string) {
const full = join(brainDir, rel);
mkdirSync(join(full, '..'), { recursive: true });
writeFileSync(full, content);
}
// ─── FS-source path (the issue's repro) ─────────────────────────────────
describe('issue #972 — FS-source (gbrain extract links default)', () => {
test("repro: [[struktura]] in concepts/ resolves to projects/struktura when flag ON", async () => {
// Seed both pages in the DB (extract validates targetSlug exists)
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: 'A project page.', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This concept relates to [[struktura]].', timeline: '',
});
// Mirror to disk so the FS extractor sees the files
writeFile('projects/struktura.md',
'---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
writeFile('concepts/knowledge-graph.md',
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--dir', brainDir]);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
// Issue #972 (T1): FS-source basename edges carry the same provenance
// tag as DB / put_page, not the default 'markdown'.
expect(strk!.link_source).toBe('wikilink-resolved');
});
test('back-compat: flag OFF → ZERO basename edges from same repro', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This relates to [[struktura]].', timeline: '',
});
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
writeFile('concepts/knowledge-graph.md',
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis relates to [[struktura]].\n');
// Flag explicitly off (also the default)
await engine.setConfig('link_resolution.global_basename', 'false');
await runExtract(engine, ['links', '--dir', brainDir]);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
expect(outLinks.filter(l => l.link_type === 'wikilink_basename')).toEqual([]);
});
test('ambiguous basename → emits one edge per match (no silent winner)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('archive/struktura', {
type: 'concept' as any, title: 'Struktura (archived)',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[struktura]].', timeline: '',
});
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n');
writeFile('concepts/x.md',
'---\ntitle: X\ntype: concept\n---\n\nSee [[struktura]].\n');
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--dir', brainDir]);
const outLinks = await engine.getLinks('concepts/x');
const basenameLinks = outLinks
.filter(l => l.link_type === 'wikilink_basename')
.map(l => l.to_slug)
.sort();
expect(basenameLinks).toEqual(['archive/struktura', 'projects/struktura']);
});
});
// ─── DB-source path (gbrain extract links --source db) ──────────────────
describe('issue #972 — DB-source (gbrain extract links --source db)', () => {
test('flag ON → bare wikilink in compiled_truth resolves to basename match', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This relates to [[struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag OFF → no basename edges via DB path (back-compat)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This relates to [[struktura]].', timeline: '',
});
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
});
});
// ─── put_page auto-link path ────────────────────────────────────────────
//
// put_page accepts `content` as a full markdown document with frontmatter
// (not the engine's narrow Page shape). Auto-link runs INSIDE put_page,
// so the basename-resolution path picks up the flag from
// engine.getConfig() once per call.
const PUT_PAGE_MARKDOWN_WITH_WIKILINK = `---
title: Knowledge Graph
type: concept
---
This relates to [[struktura]].
`;
describe('issue #972 — put_page auto-link', () => {
test('newly-written page with bare wikilink → basename edge when flag ON', async () => {
// Need the target page to exist first (auto-link validates against the
// existing slug set).
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
const { operations } = await import('../../src/core/operations.ts');
const putPage = operations.find(op => op.name === 'put_page')!;
await putPage.handler(
{ engine, remote: false } as never,
{
slug: 'concepts/knowledge-graph',
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
},
);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag OFF → put_page does NOT emit basename edge (back-compat)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
// Default-off (no setConfig)
const { operations } = await import('../../src/core/operations.ts');
const putPage = operations.find(op => op.name === 'put_page')!;
await putPage.handler(
{ engine, remote: false } as never,
{
slug: 'concepts/knowledge-graph',
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
},
);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
});
test('stale basename edge is removed when the wikilink is deleted (codex #972)', async () => {
// Regression: wikilink-resolved edges must be reconcilable, else they
// survive after the bare wikilink is removed from the page body.
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
const { operations } = await import('../../src/core/operations.ts');
const putPage = operations.find(op => op.name === 'put_page')!;
// 1. Write the page WITH the wikilink → edge lands.
await putPage.handler({ engine, remote: false } as never, {
slug: 'concepts/knowledge-graph', content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
});
let outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeDefined();
// 2. Re-write the page WITHOUT the wikilink → edge must be reconciled away.
await putPage.handler({ engine, remote: false } as never, {
slug: 'concepts/knowledge-graph',
content: '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nNo links here anymore.\n',
});
outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
});
});