mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* v0.40.8.2 fix(extract): opt-in global-basename wikilink resolution (#972) Bare wikilinks like [[struktura]] that point at pages in another folder were silently dropped from the graph. The issue reporter saw 71 wikilinks in Obsidian render to 12 in gbrain (~83% lost). Symptoms downstream: `gbrain graph` returns thin neighborhoods, `gbrain backlinks` undercounts. This release adds an opt-in mode that resolves bare wikilinks by basename match, covers all three resolver surfaces (FS-source extract, DB-source extract, put_page auto-link), and emits one edge per match — no silent winner on ambiguity. `gbrain doctor` surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve under the new mode. Enable with: gbrain config set link_resolution.global_basename true gbrain extract links Default stays off. Existing brains see zero behavior change on upgrade. Closes #972. Adapts PR #1233 from @rayers (regex shape + slug-tail index) into a multi-match, opt-in form with FS-source coverage that the original PR explicitly skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: document opt-in global-basename wikilink resolution (#972) The #972 feature shipped with no user-facing docs — only CHANGELOG + CLAUDE.md. Anyone migrating an Obsidian/Notion vault with bare [[name]] wikilinks couldn't discover the link_resolution.global_basename flag unless gbrain doctor happened to surface its hint. - README "Self-wiring knowledge graph": one sentence on the opt-in mode for Obsidian-style cross-folder bare wikilinks + the doctor pre-check, linking to the install step. - INSTALL_FOR_AGENTS Step 4.5 (Wire the Knowledge Graph): a dedicated agent- facing subsection — when bare [[name]] links need it, the enable command, re-running extract, the doctor opportunity hint, and the multi-match behavior. - Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): resolve aliased wikilinks by target slug, not display text Codex outside-voice [P1]: `[[struktura|the project]]` resolved the basename "the project" (the alias) instead of `struktura` (the target), because extractPageLinks called resolveBasenameMatches(ref.name) and the doctor check keyed basenameIndex.get(e.name). ref.name is the display alias (match[2]); ref.slug is the wikilink target (match[1]). - extractPageLinks resolves ref.slug; context excerpt locates ref.slug. - doctor link_resolution_opportunity keys e.slug so its estimate matches what extraction actually resolves. - Test: aliased wikilink calls resolveBasenameMatches with the target, never the display text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): reconcile wikilink-resolved edges in put_page auto-link Codex outside-voice [P1]: put_page's reconcilableOut filter excluded link_source='wikilink-resolved', so a basename edge written by auto-link survived after the bare wikilink was deleted from the page OR the link_resolution.global_basename flag was turned off (the stale-removal loop only iterates reconcilableOut). Add 'wikilink-resolved' to the reconcilable set; manual edges still untouched. Test: write page with [[struktura]] (flag on) → edge lands; re-put without the wikilink → edge reconciled away. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): source-scope basename resolution (no cross-source edges) Codex outside-voice [P1]: makeResolver.resolveBasenameMatches called engine.getAllSlugs() unscoped, so a bare [[name]] could resolve to a same-tail page in a DIFFERENT source and create a cross-source edge. The engine exposes getAllSlugs({sourceId}) precisely to prevent this. #972 is "global basename across folders," not "cross-source federation" — the canonical gbrain multi-source bug class. - makeResolver gains opts.sourceId; ensureBasenameIndex passes it to getAllSlugs (unscoped only when sourceId omitted — back-compat). - runAutoLink (put_page) passes opts.sourceId; extractLinksFromDB passes sourceIdFilter. FS extract is already single-source (walks one dir). - Tests: scoped index returns only the source's slugs (no cross-source); unscoped call stays brain-wide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): FS-source basename edges carry link_source='wikilink-resolved' The FS extract path is the issue's default repro (gbrain extract links with no --source db). ExtractedLink had no link_source field, so FS basename edges landed with the engine default ('markdown') instead of the 'wikilink-resolved' provenance the DB / put_page paths set and the docs promise. The e2e FS test only asserted link_type, so it was blind to this. - ExtractedLink gains link_source?; extractLinksFromFile sets it to 'wikilink-resolved' on basename edges (undefined for ordinary markdown). - Carries through the addLinksBatch snapshots automatically (LinkBatchInput already has link_source); single-row addLink fallback now passes it too. - e2e FS repro asserts link_source === 'wikilink-resolved'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(#972): one shared basename matcher across resolver/FS/doctor Codex outside-voice [P2] DRY: three surfaces each hand-rolled a basename matcher with divergent key sets — the doctor omitted the slugified key, so its link_resolution_opportunity estimate undercounted what extraction resolves, and the resolver returned matches in unsorted getAllSlugs bucket order. New shared exports in link-extraction.ts: buildBasenameIndex(slugs) + queryBasenameIndex(index, name) (keys raw/lower/slugified tail; stable sort shorter-first then lexical) + normalizeBasename. - makeResolver.resolveBasenameMatches → queryBasenameIndex (now stable-sorted). - extract.ts resolveBasenameMatchesFromSlugs → delegates to the shared pair. - doctor link_resolution_opportunity → shared builder/query (slugified key added; estimate now matches extraction). - Test: doctor counts a slugified-only match ([[Fast Weigh]] → companies/fast-weigh). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): P2 cluster — masking, code-fence, self-link, dedup decision Codex outside-voice P2 findings: - P2a markdown-label masking: a wikilink inside a markdown-link label ([see [[acme]]](companies/acme.md)) spawned a stray generic basename ref. Pass-1 can't match the nested brackets, so a new MARKDOWN_LABEL_WIKILINK_RE masks those spans out of pass 2c. Inner [[acme]] is now inert. - P2b FS code-fence: the FS path (extractMarkdownLinks on raw content) didn't strip code blocks like the DB path. extractLinksFromFile now scans stripCodeBlocks(content) so [[name]] inside a fence creates no FS edge. - P2c self-link guard: a basename [[own-tail]] on its own page resolved back to itself. Dropped in both extractPageLinks and the FS path. - P2d dedup: documented the decision to KEEP qualified + bare edges to the same target as separate rows (distinct provenance/audit trail). - P2e: skipFrontmatter unresolved-contract tests added. Tests: P2a inert-label, P2c self-link drop, P2b code-fence, P2e unresolved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(#972): bound the doctor link_resolution_opportunity scan The check did listAllPageRefs() + a getPage() per page under a 60s budget. On a large brain (the eng-review concern) it hit the budget every non-fast doctor run and returned a perpetual partial, adding ~60s. Now batch-loads the 1000 most-recent pages in ONE query (ORDER BY id DESC LIMIT SAMPLE_LIMIT) and scans in memory, with the 60s cap kept as a backstop. Mirrors the v0.40.9 sampling convention. The estimate message names the bound when the brain exceeds the sample ("scanned the 1000 most-recent of N pages"). Test: source-grep pins the bounded query + the absence of the per-page getPage walk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(#972): reconcile stale version/migration references to v112 / 0.42.6.0 Merge churn left intermediate refs: schema.sql + schema-embedded.ts said "migration v93", CLAUDE.md said "v0.41.32.0 / Migration v109", CHANGELOG said "Migration v93". Reconciled all to migration v112 / shipping 0.42.6.0. The CLAUDE.md annotation is also refreshed to describe the final behavior (shared matcher, source-scoping, alias-by-target, stale-edge reconciliation, bounded doctor scan) and credit @rayers + @ukd1. Regenerated schema-embedded + llms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): register doctor check category + bump llms budget to 800KB Two full-suite gate failures from the re-sync: - doctor-categories drift guard: the new `link_resolution_opportunity` check wasn't in any category set. Added to BRAIN_CHECK_NAMES (alongside graph_coverage / orphan_ratio — it's a graph-quality signal). - build-llms size budget: the #972 Key Files annotation (landing with master's #1696/#1699 waves) pushed llms-full.txt past 750KB. Bumped FULL_SIZE_BUDGET 750KB→800KB, the established "budget tracks CLAUDE.md's legitimate per-feature growth" pattern (600→700→750→800 across releases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Garry Tan <garrytan@gmail.com>
283 lines
11 KiB
TypeScript
283 lines
11 KiB
TypeScript
/**
|
|
* Issue #972 E2E — opt-in global-basename wikilink resolution.
|
|
*
|
|
* Reproduces the issue's exact repro inside an in-memory PGLite brain:
|
|
*
|
|
* /vault/projects/struktura.md ← a real page
|
|
* /vault/concepts/knowledge-graph.md ← contains `[[struktura]]`
|
|
*
|
|
* The bare wikilink `[[struktura]]` does NOT match WIKILINK_RE
|
|
* (DIR_PATTERN-gated) so it falls through to the new pass 2c.
|
|
*
|
|
* Three contracts:
|
|
* 1. Flag OFF (default) — extract emits ZERO basename edges (back-compat
|
|
* with pre-issue-#972 behavior: bare wikilinks outside DIR_PATTERN
|
|
* drop silently).
|
|
* 2. Flag ON — extract emits ONE edge per basename match, tagged
|
|
* `link_type: 'wikilink_basename'`.
|
|
* 3. Ambiguous basename (same name in two directories) — extract emits
|
|
* ONE edge per match. No silent winner-takes-all, no silent drop.
|
|
*
|
|
* Both DB-source AND FS-source paths covered. PGLite in-memory, no
|
|
* DATABASE_URL needed.
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
|
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
|
|
import { join } from 'path';
|
|
import { tmpdir } from 'os';
|
|
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
|
import { runExtract } from '../../src/commands/extract.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
let brainDir: string;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({ engine: 'pglite' } as never);
|
|
await engine.initSchema();
|
|
}, 60_000);
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
async function truncateAll() {
|
|
for (const t of [
|
|
'content_chunks', 'links', 'tags', 'raw_data',
|
|
'timeline_entries', 'page_versions', 'ingest_log', 'pages',
|
|
'config',
|
|
]) {
|
|
try { await (engine as any).db.exec(`DELETE FROM ${t}`); } catch { /* ok */ }
|
|
}
|
|
}
|
|
|
|
beforeEach(async () => {
|
|
await truncateAll();
|
|
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-issue-972-'));
|
|
}, 15_000);
|
|
|
|
function writeFile(rel: string, content: string) {
|
|
const full = join(brainDir, rel);
|
|
mkdirSync(join(full, '..'), { recursive: true });
|
|
writeFileSync(full, content);
|
|
}
|
|
|
|
// ─── FS-source path (the issue's repro) ─────────────────────────────────
|
|
|
|
describe('issue #972 — FS-source (gbrain extract links default)', () => {
|
|
test("repro: [[struktura]] in concepts/ resolves to projects/struktura when flag ON", async () => {
|
|
// Seed both pages in the DB (extract validates targetSlug exists)
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura',
|
|
compiled_truth: 'A project page.', timeline: '',
|
|
});
|
|
await engine.putPage('concepts/knowledge-graph', {
|
|
type: 'concept', title: 'Knowledge Graph',
|
|
compiled_truth: 'This concept relates to [[struktura]].', timeline: '',
|
|
});
|
|
// Mirror to disk so the FS extractor sees the files
|
|
writeFile('projects/struktura.md',
|
|
'---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
|
|
writeFile('concepts/knowledge-graph.md',
|
|
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
|
|
|
|
await engine.setConfig('link_resolution.global_basename', 'true');
|
|
|
|
await runExtract(engine, ['links', '--dir', brainDir]);
|
|
|
|
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
|
|
expect(strk).toBeDefined();
|
|
expect(strk!.link_type).toBe('wikilink_basename');
|
|
// Issue #972 (T1): FS-source basename edges carry the same provenance
|
|
// tag as DB / put_page, not the default 'markdown'.
|
|
expect(strk!.link_source).toBe('wikilink-resolved');
|
|
});
|
|
|
|
test('back-compat: flag OFF → ZERO basename edges from same repro', async () => {
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura',
|
|
compiled_truth: '', timeline: '',
|
|
});
|
|
await engine.putPage('concepts/knowledge-graph', {
|
|
type: 'concept', title: 'Knowledge Graph',
|
|
compiled_truth: 'This relates to [[struktura]].', timeline: '',
|
|
});
|
|
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
|
|
writeFile('concepts/knowledge-graph.md',
|
|
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis relates to [[struktura]].\n');
|
|
|
|
// Flag explicitly off (also the default)
|
|
await engine.setConfig('link_resolution.global_basename', 'false');
|
|
|
|
await runExtract(engine, ['links', '--dir', brainDir]);
|
|
|
|
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
|
expect(outLinks.filter(l => l.link_type === 'wikilink_basename')).toEqual([]);
|
|
});
|
|
|
|
test('ambiguous basename → emits one edge per match (no silent winner)', async () => {
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura',
|
|
compiled_truth: '', timeline: '',
|
|
});
|
|
await engine.putPage('archive/struktura', {
|
|
type: 'concept' as any, title: 'Struktura (archived)',
|
|
compiled_truth: '', timeline: '',
|
|
});
|
|
await engine.putPage('concepts/x', {
|
|
type: 'concept', title: 'X',
|
|
compiled_truth: 'See [[struktura]].', timeline: '',
|
|
});
|
|
|
|
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
|
|
writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n');
|
|
writeFile('concepts/x.md',
|
|
'---\ntitle: X\ntype: concept\n---\n\nSee [[struktura]].\n');
|
|
|
|
await engine.setConfig('link_resolution.global_basename', 'true');
|
|
|
|
await runExtract(engine, ['links', '--dir', brainDir]);
|
|
|
|
const outLinks = await engine.getLinks('concepts/x');
|
|
const basenameLinks = outLinks
|
|
.filter(l => l.link_type === 'wikilink_basename')
|
|
.map(l => l.to_slug)
|
|
.sort();
|
|
expect(basenameLinks).toEqual(['archive/struktura', 'projects/struktura']);
|
|
});
|
|
});
|
|
|
|
// ─── DB-source path (gbrain extract links --source db) ──────────────────
|
|
|
|
describe('issue #972 — DB-source (gbrain extract links --source db)', () => {
|
|
test('flag ON → bare wikilink in compiled_truth resolves to basename match', async () => {
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura',
|
|
compiled_truth: '', timeline: '',
|
|
});
|
|
await engine.putPage('concepts/knowledge-graph', {
|
|
type: 'concept', title: 'Knowledge Graph',
|
|
compiled_truth: 'This relates to [[struktura]].', timeline: '',
|
|
});
|
|
await engine.setConfig('link_resolution.global_basename', 'true');
|
|
|
|
await runExtract(engine, ['links', '--source', 'db']);
|
|
|
|
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
|
|
expect(strk).toBeDefined();
|
|
expect(strk!.link_type).toBe('wikilink_basename');
|
|
});
|
|
|
|
test('flag OFF → no basename edges via DB path (back-compat)', async () => {
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura',
|
|
compiled_truth: '', timeline: '',
|
|
});
|
|
await engine.putPage('concepts/knowledge-graph', {
|
|
type: 'concept', title: 'Knowledge Graph',
|
|
compiled_truth: 'This relates to [[struktura]].', timeline: '',
|
|
});
|
|
|
|
await runExtract(engine, ['links', '--source', 'db']);
|
|
|
|
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
// ─── put_page auto-link path ────────────────────────────────────────────
|
|
//
|
|
// put_page accepts `content` as a full markdown document with frontmatter
|
|
// (not the engine's narrow Page shape). Auto-link runs INSIDE put_page,
|
|
// so the basename-resolution path picks up the flag from
|
|
// engine.getConfig() once per call.
|
|
|
|
const PUT_PAGE_MARKDOWN_WITH_WIKILINK = `---
|
|
title: Knowledge Graph
|
|
type: concept
|
|
---
|
|
|
|
This relates to [[struktura]].
|
|
`;
|
|
|
|
describe('issue #972 — put_page auto-link', () => {
|
|
test('newly-written page with bare wikilink → basename edge when flag ON', async () => {
|
|
// Need the target page to exist first (auto-link validates against the
|
|
// existing slug set).
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura',
|
|
compiled_truth: '', timeline: '',
|
|
});
|
|
|
|
await engine.setConfig('link_resolution.global_basename', 'true');
|
|
|
|
const { operations } = await import('../../src/core/operations.ts');
|
|
const putPage = operations.find(op => op.name === 'put_page')!;
|
|
await putPage.handler(
|
|
{ engine, remote: false } as never,
|
|
{
|
|
slug: 'concepts/knowledge-graph',
|
|
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
|
|
},
|
|
);
|
|
|
|
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
|
|
expect(strk).toBeDefined();
|
|
expect(strk!.link_type).toBe('wikilink_basename');
|
|
});
|
|
|
|
test('flag OFF → put_page does NOT emit basename edge (back-compat)', async () => {
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura',
|
|
compiled_truth: '', timeline: '',
|
|
});
|
|
|
|
// Default-off (no setConfig)
|
|
|
|
const { operations } = await import('../../src/core/operations.ts');
|
|
const putPage = operations.find(op => op.name === 'put_page')!;
|
|
await putPage.handler(
|
|
{ engine, remote: false } as never,
|
|
{
|
|
slug: 'concepts/knowledge-graph',
|
|
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
|
|
},
|
|
);
|
|
|
|
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
|
});
|
|
|
|
test('stale basename edge is removed when the wikilink is deleted (codex #972)', async () => {
|
|
// Regression: wikilink-resolved edges must be reconcilable, else they
|
|
// survive after the bare wikilink is removed from the page body.
|
|
await engine.putPage('projects/struktura', {
|
|
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
|
|
});
|
|
await engine.setConfig('link_resolution.global_basename', 'true');
|
|
|
|
const { operations } = await import('../../src/core/operations.ts');
|
|
const putPage = operations.find(op => op.name === 'put_page')!;
|
|
|
|
// 1. Write the page WITH the wikilink → edge lands.
|
|
await putPage.handler({ engine, remote: false } as never, {
|
|
slug: 'concepts/knowledge-graph', content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
|
|
});
|
|
let outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeDefined();
|
|
|
|
// 2. Re-write the page WITHOUT the wikilink → edge must be reconciled away.
|
|
await putPage.handler({ engine, remote: false } as never, {
|
|
slug: 'concepts/knowledge-graph',
|
|
content: '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nNo links here anymore.\n',
|
|
});
|
|
outLinks = await engine.getLinks('concepts/knowledge-graph');
|
|
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
|
});
|
|
});
|