mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
Takeover of #2978 (fork PR), rebased onto current master. Resolved a docs/architecture/KEY_FILES.md conflict (kept master's newer extract-conversation-facts entry, merged the PR's link-extraction additions) and regenerated llms.txt/llms-full.txt via bun run build:llms. - New regex passes 2a'/2b' match path-shaped wikilinks [[dir/.../name]] for any directory prefix, masked after whitelist passes (no double-emit) - Exact-slug resolution via optional SlugResolver.slugExists off the same source-scoped getAllSlugs snapshot; gate link_resolution.any_dir_exact_path defaults ON with env override - Misses ride the existing unresolved-report machinery (never silent) - Qualified [[src:path]] pins thread through resolution end-to-end; pins join the dedup key - #972 bare-name/basename machinery untouched Co-authored-by: gporterl <4203071+gporterl@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
gporterl
Claude Fable 5
gporterl
Claude Fable 5
parent
0612b0daa8
commit
28f372cece
@@ -163,11 +163,16 @@ and supports `--since YYYY-MM-DD` for incremental runs.
|
||||
|
||||
### Obsidian-style bare wikilinks (opt-in)
|
||||
|
||||
Path-qualified refs like `[[projects/struktura]]` or `[[janus/agents/drift-check]]`
|
||||
resolve out of the box for ANY directory — the target just has to exist as a page
|
||||
with that exact slug (`link_resolution.any_dir_exact_path`, on by default; misses
|
||||
land in the unresolved report instead of dropping silently).
|
||||
|
||||
If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]`
|
||||
wikilinks — where `[[struktura]]` written in one folder means the page that lives
|
||||
at `projects/struktura.md` in another — GBrain does NOT connect those by default.
|
||||
Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`,
|
||||
so a vault full of bare links shows up as a thin, broken graph. Turn on basename
|
||||
at `projects/struktura.md` in another — GBrain does NOT connect those by default:
|
||||
a bare name is fuzzy (basename match, possibly several winners), so a vault full
|
||||
of bare links shows up as a thin, broken graph until you opt in. Turn on basename
|
||||
resolution so the cross-folder links connect:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -254,7 +254,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. Path-qualified wikilinks (`[[any-dir/sub/name]]`) link to any existing page out of the box — exact slug match, any folder, misses surfaced in the unresolved report. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
+9
-4
@@ -1153,11 +1153,16 @@ and supports `--since YYYY-MM-DD` for incremental runs.
|
||||
|
||||
### Obsidian-style bare wikilinks (opt-in)
|
||||
|
||||
Path-qualified refs like `[[projects/struktura]]` or `[[janus/agents/drift-check]]`
|
||||
resolve out of the box for ANY directory — the target just has to exist as a page
|
||||
with that exact slug (`link_resolution.any_dir_exact_path`, on by default; misses
|
||||
land in the unresolved report instead of dropping silently).
|
||||
|
||||
If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]`
|
||||
wikilinks — where `[[struktura]]` written in one folder means the page that lives
|
||||
at `projects/struktura.md` in another — GBrain does NOT connect those by default.
|
||||
Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`,
|
||||
so a vault full of bare links shows up as a thin, broken graph. Turn on basename
|
||||
at `projects/struktura.md` in another — GBrain does NOT connect those by default:
|
||||
a bare name is fuzzy (basename match, possibly several winners), so a vault full
|
||||
of bare links shows up as a thin, broken graph until you opt in. Turn on basename
|
||||
resolution so the cross-folder links connect:
|
||||
|
||||
```bash
|
||||
@@ -1748,7 +1753,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. Path-qualified wikilinks (`[[any-dir/sub/name]]`) link to any existing page out of the box — exact slug match, any folder, misses surfaced in the unresolved report. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
|
||||
+88
-13
@@ -36,7 +36,8 @@ import type { PageType } from '../core/types.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import {
|
||||
extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver,
|
||||
extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS,
|
||||
extractFrontmatterLinks, isAnyDirExactPathEnabled, isGlobalBasenameEnabled,
|
||||
LINK_EXTRACTOR_VERSION_TS, type SlugResolver,
|
||||
WIKILINK_BASENAME_LINK_TYPE,
|
||||
buildBasenameIndex, queryBasenameIndex, stripCodeBlocks,
|
||||
type UnresolvedFrontmatterRef, type LinkCandidate,
|
||||
@@ -126,7 +127,15 @@ export function resolveCandidateSources(
|
||||
: (fromSources.includes('default') ? 'default' : fromSources[0]);
|
||||
const targetSources = slugToSources.get(c.targetSlug) ?? [];
|
||||
let toSourceId: string;
|
||||
if (targetSources.includes(fromSourceId)) {
|
||||
if (c.targetSourceId) {
|
||||
// Issue #1493 (codex P1): a qualified wikilink `[[src:slug]]` pins the
|
||||
// target source — that IS the qualification's meaning (v0.17.0). The pin
|
||||
// overrides the local-first/default fallback: the target must exist in
|
||||
// the pinned source or the candidate is skipped (never misdirected to a
|
||||
// same-slug page in the origin/default source).
|
||||
if (!targetSources.includes(c.targetSourceId)) return null;
|
||||
toSourceId = c.targetSourceId;
|
||||
} else if (targetSources.includes(fromSourceId)) {
|
||||
toSourceId = fromSourceId;
|
||||
} else if (targetSources.includes('default')) {
|
||||
toSourceId = 'default';
|
||||
@@ -179,6 +188,14 @@ interface ExtractResult {
|
||||
links_created: number;
|
||||
timeline_entries_created: number;
|
||||
pages_processed: number;
|
||||
/**
|
||||
* Issue #1493 (codex P2): unresolved refs (frontmatter misses +
|
||||
* wikilink exact-path misses) from the DB links pass. Present only when
|
||||
* non-empty so pre-existing JSON consumers see an unchanged shape on
|
||||
* clean runs. `--json` prints the whole ExtractResult, so agents see
|
||||
* the misses instead of only counts.
|
||||
*/
|
||||
unresolved_refs?: UnresolvedFrontmatterRef[];
|
||||
}
|
||||
|
||||
// --- Shared walker ---
|
||||
@@ -947,6 +964,9 @@ Status (v0.42):
|
||||
const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since, { includeFrontmatter, sourceIdFilter, stampWatermark: subcommand === 'all' });
|
||||
result.links_created = r.created;
|
||||
result.pages_processed = r.pages;
|
||||
// Issue #1493 (codex P2): thread unresolved refs into the JSON
|
||||
// result — --json used to suppress the miss report entirely.
|
||||
if (r.unresolved.length > 0) result.unresolved_refs = r.unresolved;
|
||||
}
|
||||
if (subcommand === 'timeline' || subcommand === 'all') {
|
||||
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
@@ -1381,6 +1401,9 @@ async function extractLinksFromDB(
|
||||
// Issue #972: opt-in global-basename wikilink resolution. Read once
|
||||
// per extract run; threaded into each extractPageLinks call.
|
||||
const globalBasename = await isGlobalBasenameEnabled(engine);
|
||||
// Issue #1493: exact-path resolution for wikilinks outside the DIR_PATTERN
|
||||
// whitelist. On by default; read once per run like globalBasename.
|
||||
const anyDirExactPath = await isAnyDirExactPathEnabled(engine);
|
||||
// v0.32.8: listAllPageRefs enumerates (slug, source_id) so we can thread
|
||||
// sourceId to getPage AND build a cross-source resolution map for link
|
||||
// disambiguation. Pre-fix used getAllSlugs() which collapsed
|
||||
@@ -1461,7 +1484,7 @@ async function extractLinksFromDB(
|
||||
// basename lookup; off by default for back-compat.
|
||||
const extracted = await extractPageLinks(
|
||||
slug, fullContent, page.frontmatter, page.type, resolver,
|
||||
{ skipFrontmatter: !includeFrontmatter, globalBasename },
|
||||
{ skipFrontmatter: !includeFrontmatter, globalBasename, anyDirExactPath },
|
||||
);
|
||||
unresolved.push(...extracted.unresolved);
|
||||
|
||||
@@ -1528,10 +1551,13 @@ async function extractLinksFromDB(
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
console.log(`Links: ${label} ${created} from ${processed} pages (db source)`);
|
||||
if (includeFrontmatter && unresolved.length > 0) {
|
||||
// Top-20 preview of unresolvable frontmatter names so the user can
|
||||
// see where the graph has holes (codex tension 6.4).
|
||||
console.log(`Unresolved frontmatter refs: ${unresolved.length} total`);
|
||||
if (unresolved.length > 0) {
|
||||
// Top-20 preview of unresolvable names so the user can see where the
|
||||
// graph has holes (codex tension 6.4). Pre-#1493 this was gated on
|
||||
// includeFrontmatter because frontmatter was the only unresolved
|
||||
// source; path-shaped wikilink misses (field='wikilink') now report
|
||||
// here too, so the gate is simply "anything unresolved".
|
||||
console.log(`Unresolved refs (frontmatter + wikilinks): ${unresolved.length} total`);
|
||||
const bucket = new Map<string, number>();
|
||||
for (const u of unresolved) {
|
||||
const key = `${u.field}:${u.name}`;
|
||||
@@ -1660,7 +1686,7 @@ async function extractStaleFromDB(
|
||||
sourceIdFilter?: string;
|
||||
catchUp: boolean;
|
||||
},
|
||||
): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number }> {
|
||||
): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number; unresolved?: UnresolvedFrontmatterRef[] }> {
|
||||
const { dryRun, jsonMode, includeFrontmatter, sourceIdFilter, catchUp } = opts;
|
||||
const versionTs = LINK_EXTRACTOR_VERSION_TS;
|
||||
|
||||
@@ -1684,9 +1710,39 @@ async function extractStaleFromDB(
|
||||
// Batch mode = pg_trgm + exact only, NO per-name search fallback. The
|
||||
// resolution map sees ALL sources so qualified cross-source wikilinks resolve
|
||||
// even when --source-id scopes the stale SCAN.
|
||||
const resolver = makeResolver(engine, { mode: 'batch' });
|
||||
const nullResolver = { resolve: async () => null as string | null };
|
||||
const activeResolver = includeFrontmatter ? resolver : nullResolver;
|
||||
// Issue #1493 (codex P2): always pass the REAL resolver and gate the
|
||||
// frontmatter pass via opts.skipFrontmatter — the same migration
|
||||
// extractLinksFromDB made for issue #972. The old nullResolver ternary
|
||||
// meant the default sweep (no --include-frontmatter) had no slugExists,
|
||||
// so any-dir exact-path misses were silently discarded by the endpoint
|
||||
// check instead of being verified + recorded as unresolved.
|
||||
//
|
||||
// Issue #1493 (codex P2, round 2): the resolver is scoped PER PAGE SOURCE,
|
||||
// not one unscoped instance for the sweep. An unscoped slugExists spans
|
||||
// all sources, so an alpha page's `[[janus/foo]]` whose target exists
|
||||
// only in beta passed verification, became a candidate, was rejected by
|
||||
// resolveCandidateSources (not in alpha ∪ default) — and the miss was
|
||||
// never recorded. Per-source resolvers give each page the SAME
|
||||
// resolution domain resolveCandidateSources applies (page source ∪
|
||||
// default). Cached per source_id: getAllSlugs runs at most twice per
|
||||
// distinct source across the whole sweep (scoped + default overlay),
|
||||
// matching the DB path's fetch discipline.
|
||||
const resolverBySource = new Map<string, SlugResolver>();
|
||||
function resolverForSource(sourceId: string): SlugResolver {
|
||||
let r = resolverBySource.get(sourceId);
|
||||
if (!r) {
|
||||
r = makeResolver(engine, { mode: 'batch', sourceId });
|
||||
resolverBySource.set(sourceId, r);
|
||||
}
|
||||
return r;
|
||||
}
|
||||
// Issue #1493: exact-path resolution for wikilinks outside the DIR_PATTERN
|
||||
// whitelist. Read once per sweep.
|
||||
const anyDirExactPath = await isAnyDirExactPathEnabled(engine);
|
||||
// Issue #1493 (codex P2): aggregate unresolved refs (wikilink misses +,
|
||||
// with --include-frontmatter, frontmatter misses) and surface them in the
|
||||
// sweep summary — consistent with the DB path.
|
||||
const unresolved: UnresolvedFrontmatterRef[] = [];
|
||||
const allRefs = await engine.listAllPageRefs();
|
||||
const allSlugs = new Set<string>();
|
||||
const slugToSources = new Map<string, string[]>();
|
||||
@@ -1718,8 +1774,10 @@ async function extractStaleFromDB(
|
||||
for (const page of rows) {
|
||||
const fullContent = page.compiled_truth + '\n' + page.timeline;
|
||||
const extracted = await extractPageLinks(
|
||||
page.slug, fullContent, page.frontmatter, page.type, activeResolver,
|
||||
page.slug, fullContent, page.frontmatter, page.type, resolverForSource(page.source_id),
|
||||
{ skipFrontmatter: !includeFrontmatter, anyDirExactPath },
|
||||
);
|
||||
unresolved.push(...extracted.unresolved);
|
||||
for (const c of extracted.candidates) {
|
||||
const r = resolveCandidateSources(c, page.slug, page.source_id, allSlugs, slugToSources);
|
||||
if (!r) continue;
|
||||
@@ -1772,6 +1830,20 @@ async function extractStaleFromDB(
|
||||
|
||||
if (!jsonMode) {
|
||||
console.log(`Extract --stale: ${linksCreated} link(s) + ${timelineCreated} timeline entr(ies) from ${pagesProcessed} page(s).`);
|
||||
// Issue #1493 (codex P2): surface unresolved refs (top-10 preview),
|
||||
// mirroring the DB path's summary — misses must not be silent.
|
||||
if (unresolved.length > 0) {
|
||||
console.log(`Unresolved refs (frontmatter + wikilinks): ${unresolved.length} total`);
|
||||
const bucket = new Map<string, number>();
|
||||
for (const u of unresolved) {
|
||||
const key = `${u.field}:${u.name}`;
|
||||
bucket.set(key, (bucket.get(key) || 0) + 1);
|
||||
}
|
||||
const top = Array.from(bucket.entries()).sort((a, b) => b[1] - a[1]).slice(0, 10);
|
||||
for (const [key, count] of top) {
|
||||
console.log(` ${count}× ${key}`);
|
||||
}
|
||||
}
|
||||
if (budgetHit && staleRemaining > 0) {
|
||||
console.log(`Time budget reached — ${staleRemaining} page(s) still stale. Re-run 'gbrain extract --stale' (or pass --catch-up) to continue.`);
|
||||
}
|
||||
@@ -1779,9 +1851,12 @@ async function extractStaleFromDB(
|
||||
process.stdout.write(JSON.stringify({
|
||||
action: 'extract_stale_done', links_created: linksCreated, timeline_created: timelineCreated,
|
||||
pages_processed: pagesProcessed, stale_remaining: staleRemaining, budget_hit: budgetHit,
|
||||
// Issue #1493 (codex P2): agents consuming --json must see the misses.
|
||||
unresolved_count: unresolved.length,
|
||||
...(unresolved.length > 0 ? { unresolved_refs: unresolved } : {}),
|
||||
}) + '\n');
|
||||
}
|
||||
return { linksCreated, timelineCreated, pagesProcessed, staleRemaining };
|
||||
return { linksCreated, timelineCreated, pagesProcessed, staleRemaining, unresolved };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -963,6 +963,9 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// Link resolution (issue #972)
|
||||
'link_resolution',
|
||||
'link_resolution.global_basename',
|
||||
// Issue #1493: any-dir exact-path wikilink resolution (default ON; this
|
||||
// key is the documented escape hatch, so `config set` must accept it).
|
||||
'link_resolution.any_dir_exact_path',
|
||||
// Spend controls (v0.42.42.0, issue #2139). Previously `--force`-only — the
|
||||
// operator had to discover these by reading source. Registered so `config
|
||||
// set` accepts them directly. See docs/operations/spend-controls.md.
|
||||
|
||||
+300
-32
@@ -28,7 +28,10 @@ import { ensureWellFormed } from './text-safe.ts';
|
||||
* OR updated_at > links_extracted_at`. It is an ISO-8601 string (NOT a number) —
|
||||
* the column is TIMESTAMPTZ and the predicate binds it as `::timestamptz`.
|
||||
*/
|
||||
export const LINK_EXTRACTOR_VERSION_TS = '2026-05-31T00:00:00Z';
|
||||
// Issue #1493: bumped for the any-dir exact-path wikilink pass so
|
||||
// already-stamped pages re-extract on the next `extract --stale` sweep
|
||||
// and gain their newly-resolvable path-shaped edges.
|
||||
export const LINK_EXTRACTOR_VERSION_TS = '2026-07-18T00:00:00Z';
|
||||
|
||||
// ─── Entity references ──────────────────────────────────────────
|
||||
|
||||
@@ -60,6 +63,16 @@ export interface EntityRef {
|
||||
* LinkCandidate per resolved match.
|
||||
*/
|
||||
needsResolution?: boolean;
|
||||
/**
|
||||
* Issue #1493 (generalized path resolution): set when the ref came from
|
||||
* the path-shaped any-directory wikilink passes (2a'/2b') — the target
|
||||
* is `dir/.../name` in canonical slug form but the top-level dir is NOT
|
||||
* in the DIR_PATTERN whitelist. The `slug` field IS the literal slug the
|
||||
* author wrote; extractPageLinks verifies it exists (exact match, via
|
||||
* `SlugResolver.slugExists`) before emitting a candidate, and records a
|
||||
* miss in the unresolved report instead of dropping it silently.
|
||||
*/
|
||||
exactPath?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -113,6 +126,39 @@ const WIKILINK_RE = new RegExp(
|
||||
'g',
|
||||
);
|
||||
|
||||
/**
|
||||
* Issue #1493 (generalized path resolution): one path segment in canonical
|
||||
* slug form. Conservative ASCII subset of sync.ts's SLUG_SEGMENT_PATTERN:
|
||||
* must START with [a-z0-9] (so `..`/`.hidden` relative-path segments never
|
||||
* match) and may continue with dots, underscores and hyphens (`janus-ui`,
|
||||
* `v1.0.0`, `foo_bar`). CJK slugs are deliberately NOT covered here — they
|
||||
* keep the pre-existing behavior (generic pass 2c / basename resolution).
|
||||
*/
|
||||
const SLUG_PATH_SEGMENT = '[a-z0-9][a-z0-9._-]*';
|
||||
|
||||
/**
|
||||
* Issue #1493: path-shaped wikilink `[[dir/name]]` / `[[dir/sub/name]]`
|
||||
* (>= 2 segments) with ANY directory prefix — NOT gated by DIR_PATTERN.
|
||||
* The whitelist silently dropped every wikilink whose target lives outside
|
||||
* the recognized entity dirs (`[[janus/agents/drift-check]]`), losing the
|
||||
* authored graph with no candidate and no unresolved report.
|
||||
*
|
||||
* Unlike the #972 bare-name pass (fuzzy by design, opt-in), a path-shaped
|
||||
* target is already an exact slug: it resolves by exact match against
|
||||
* existing pages, so it is safe to run by default
|
||||
* (`link_resolution.any_dir_exact_path`, default on).
|
||||
*
|
||||
* Runs AFTER WIKILINK_RE with its spans masked, so whitelisted dirs keep
|
||||
* their existing pass (2b) untouched; anything this matches has a
|
||||
* non-whitelisted top dir. Same alias/anchor tail shape as WIKILINK_RE.
|
||||
* The strict slug grammar means targets with uppercase or spaces do NOT
|
||||
* match — they fall through to the generic pass 2c as before.
|
||||
*/
|
||||
const WIKILINK_ANY_PATH_RE = new RegExp(
|
||||
`\\[\\[(${SLUG_PATH_SEGMENT}(?:\\/${SLUG_PATH_SEGMENT})+)(?:#[^|\\]]*?)?(?:\\|([^\\]]+?))?\\]\\]`,
|
||||
'g',
|
||||
);
|
||||
|
||||
/**
|
||||
* v0.17.0: qualified wikilink `[[source-id:dir/slug]]` or
|
||||
* `[[source-id:dir/slug|Display Text]]`. The source-id segment pins the
|
||||
@@ -130,6 +176,18 @@ const QUALIFIED_WIKILINK_RE = new RegExp(
|
||||
'g',
|
||||
);
|
||||
|
||||
/**
|
||||
* Issue #1493: qualified variant of WIKILINK_ANY_PATH_RE —
|
||||
* `[[source-id:dir/name]]` with a non-whitelisted dir. Previously these
|
||||
* were dropped by ALL passes: QUALIFIED_WIKILINK_RE requires DIR_PATTERN
|
||||
* and the generic pass 2c skips any token containing `:`. Runs after
|
||||
* QUALIFIED_WIKILINK_RE with its spans masked, same source-id grammar.
|
||||
*/
|
||||
const QUALIFIED_WIKILINK_ANY_PATH_RE = new RegExp(
|
||||
`\\[\\[([a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?):(${SLUG_PATH_SEGMENT}(?:\\/${SLUG_PATH_SEGMENT})+)(?:#[^|\\]]*?)?(?:\\|([^\\]]+?))?\\]\\]`,
|
||||
'g',
|
||||
);
|
||||
|
||||
/**
|
||||
* Issue #972: generic Obsidian-style wikilink — matches `[[anything]]`
|
||||
* with no DIR_PATTERN gate. Wiki/topic/learning pages frequently link
|
||||
@@ -318,6 +376,16 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
markdownRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// Issue #972 (codex [P2]) / #1493: markdown-link labels containing a
|
||||
// wikilink — `[see [[acme]]](companies/acme.md)` — are masked out of the
|
||||
// generalized (2a'/2b') and generic (2c) scans so the inner `[[...]]`
|
||||
// stays inert. Computed once up front; consumed by those passes below.
|
||||
const labelWikilinkRanges: Array<[number, number]> = [];
|
||||
const labelWlPattern = new RegExp(MARKDOWN_LABEL_WIKILINK_RE.source, MARKDOWN_LABEL_WIKILINK_RE.flags);
|
||||
while ((match = labelWlPattern.exec(stripped)) !== null) {
|
||||
labelWikilinkRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// 2a. v0.17.0 qualified wikilinks: [[source-id:path]] or [[source-id:path|Display]]
|
||||
// Must run BEFORE the unqualified pass or we'd double-emit. We also
|
||||
// mask out the matched spans so pass 2b can't grab them.
|
||||
@@ -335,10 +403,31 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
qualifiedRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// 2a'. Issue #1493: qualified path-shaped wikilinks outside DIR_PATTERN
|
||||
// ([[src:janus/foo]]). Masked by 2a's spans so whitelisted qualified
|
||||
// links are never re-emitted. Tagged `exactPath: true`; the target
|
||||
// may live in a different source, so existence is left to the
|
||||
// caller's cross-source resolution (same as whitelisted qualified
|
||||
// refs) — extractPageLinks does NOT slugExists-check qualified refs.
|
||||
const qualifiedPathRanges: Array<[number, number]> = [];
|
||||
const qualAnyMasked = maskRanges(stripped, [...qualifiedRanges, ...labelWikilinkRanges]);
|
||||
const qualAnyPattern = new RegExp(QUALIFIED_WIKILINK_ANY_PATH_RE.source, QUALIFIED_WIKILINK_ANY_PATH_RE.flags);
|
||||
while ((match = qualAnyPattern.exec(qualAnyMasked)) !== null) {
|
||||
const sourceId = match[1];
|
||||
let slug = match[2].trim();
|
||||
if (!slug) continue;
|
||||
if (slug.includes('://')) continue;
|
||||
if (slug.endsWith('.md')) slug = slug.slice(0, -3);
|
||||
const displayName = (match[3] || slug).trim();
|
||||
const dir = slug.split('/')[0];
|
||||
refs.push({ name: displayName, slug, dir, sourceId, exactPath: true });
|
||||
qualifiedPathRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// 2b. Unqualified Obsidian wikilinks: [[path]] or [[path|Display Text]]
|
||||
// Same shape rule: omit sourceId when unqualified.
|
||||
const unqualifiedRanges: Array<[number, number]> = [];
|
||||
const unmasked = maskRanges(stripped, qualifiedRanges);
|
||||
const unmasked = maskRanges(stripped, [...qualifiedRanges, ...qualifiedPathRanges]);
|
||||
const wikiPattern = new RegExp(WIKILINK_RE.source, WIKILINK_RE.flags);
|
||||
while ((match = wikiPattern.exec(unmasked)) !== null) {
|
||||
let slug = match[1].trim();
|
||||
@@ -351,24 +440,45 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
unqualifiedRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// 2b'. Issue #1493: path-shaped wikilinks with ANY directory prefix
|
||||
// ([[janus/agents/drift-check]], [[janus-ui/docs/tier2-agent-plan]]).
|
||||
// Masked by 2b's spans so whitelisted dirs keep their existing pass
|
||||
// untouched — anything matched here has a non-whitelisted top dir.
|
||||
// Tagged `exactPath: true` so extractPageLinks verifies the slug
|
||||
// exists (exact match) before emitting, and records misses in the
|
||||
// unresolved report instead of silently dropping them.
|
||||
const pathRanges: Array<[number, number]> = [];
|
||||
const anyPathMasked = maskRanges(
|
||||
stripped,
|
||||
[...markdownRanges, ...qualifiedRanges, ...qualifiedPathRanges, ...unqualifiedRanges, ...labelWikilinkRanges],
|
||||
);
|
||||
const anyPathPattern = new RegExp(WIKILINK_ANY_PATH_RE.source, WIKILINK_ANY_PATH_RE.flags);
|
||||
while ((match = anyPathPattern.exec(anyPathMasked)) !== null) {
|
||||
let slug = match[1].trim();
|
||||
if (!slug) continue;
|
||||
if (slug.includes('://')) continue;
|
||||
if (slug.endsWith('.md')) slug = slug.slice(0, -3);
|
||||
const displayName = (match[2] || slug).trim();
|
||||
const dir = slug.split('/')[0];
|
||||
refs.push({ name: displayName, slug, dir, exactPath: true });
|
||||
pathRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// 2c. Issue #972: generic `[[bare-name]]` wikilinks not gated by
|
||||
// DIR_PATTERN. Wiki/topic/learning content frequently uses these
|
||||
// where the bare text isn't a canonical brain slug. Tagged
|
||||
// `needsResolution: true` so the caller routes them through a
|
||||
// SlugResolver's `resolveBasenameMatches` before persisting.
|
||||
// Mask out 2a/2b ranges so we don't double-emit; skip qualified-
|
||||
// syntax tokens (contain `:`) that would be 2a's job. Issue #972
|
||||
// (codex [P2]): ALSO mask pass-1 markdown-link ranges so a wikilink
|
||||
// inside a markdown label — `[see [[acme]]](companies/acme.md)` —
|
||||
// doesn't spawn a stray generic basename ref from inside the label.
|
||||
const labelWikilinkRanges: Array<[number, number]> = [];
|
||||
const labelWlPattern = new RegExp(MARKDOWN_LABEL_WIKILINK_RE.source, MARKDOWN_LABEL_WIKILINK_RE.flags);
|
||||
while ((match = labelWlPattern.exec(stripped)) !== null) {
|
||||
labelWikilinkRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
// Mask out 2a/2a'/2b/2b' ranges so we don't double-emit; skip
|
||||
// qualified-syntax tokens (contain `:`) that would be 2a's job.
|
||||
// Issue #972 (codex [P2]): ALSO mask pass-1 markdown-link ranges
|
||||
// (via labelWikilinkRanges, computed above 2a) so a wikilink inside
|
||||
// a markdown label — `[see [[acme]]](companies/acme.md)` — doesn't
|
||||
// spawn a stray generic basename ref from inside the label.
|
||||
const genericMasked = maskRanges(
|
||||
stripped,
|
||||
[...markdownRanges, ...qualifiedRanges, ...unqualifiedRanges, ...labelWikilinkRanges],
|
||||
[...markdownRanges, ...qualifiedRanges, ...qualifiedPathRanges,
|
||||
...unqualifiedRanges, ...pathRanges, ...labelWikilinkRanges],
|
||||
);
|
||||
const genericPattern = new RegExp(WIKILINK_GENERIC_RE.source, WIKILINK_GENERIC_RE.flags);
|
||||
while ((match = genericPattern.exec(genericMasked)) !== null) {
|
||||
@@ -431,6 +541,18 @@ export interface LinkCandidate {
|
||||
originSlug?: string;
|
||||
/** Frontmatter field name (e.g. 'key_people'), for debug + unresolved report. */
|
||||
originField?: string;
|
||||
/**
|
||||
* Issue #1493 (codex P1): source pin from a qualified wikilink
|
||||
* `[[source-id:slug]]`, threaded from EntityRef.sourceId. Callers doing
|
||||
* cross-source resolution (resolveCandidateSources) must honor it — the
|
||||
* target must exist in THIS source — instead of applying the
|
||||
* local-first/default fallback used by unqualified refs. Absent =
|
||||
* unqualified. Pre-#1493 the pin was dropped at this layer for EVERY
|
||||
* qualified wikilink (whitelisted dirs included), so `[[b:slug]]`
|
||||
* written in source `a` either linked to a same-slug page in `a`/default
|
||||
* (misdirected) or vanished; now the pinned source wins.
|
||||
*/
|
||||
targetSourceId?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -468,9 +590,19 @@ export async function extractPageLinks(
|
||||
frontmatter: Record<string, unknown>,
|
||||
pageType: PageType,
|
||||
resolver: SlugResolver,
|
||||
opts: { globalBasename?: boolean; skipFrontmatter?: boolean } = {},
|
||||
opts: { globalBasename?: boolean; skipFrontmatter?: boolean; anyDirExactPath?: boolean } = {},
|
||||
): Promise<PageLinksResult> {
|
||||
const candidates: LinkCandidate[] = [];
|
||||
// Issue #1493: path-shaped wikilink targets that failed the exact-match
|
||||
// existence check. Surfaced through the same unresolved report as
|
||||
// frontmatter misses (field='wikilink') so the gap is visible in the
|
||||
// put_page auto_links response and the extract summary — never silently
|
||||
// dropped, never written as a ghost edge. Deduped by target within page.
|
||||
const wikilinkUnresolved: UnresolvedFrontmatterRef[] = [];
|
||||
const wikilinkUnresolvedSeen = new Set<string>();
|
||||
// Default ON: exact slug match can't false-positive the way basename
|
||||
// matching can, so unlike global_basename this needs no opt-in.
|
||||
const anyDirExactPath = opts.anyDirExactPath !== false;
|
||||
|
||||
// 1. Markdown entity refs.
|
||||
for (const ref of extractEntityRefs(content)) {
|
||||
@@ -506,6 +638,30 @@ export async function extractPageLinks(
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Issue #1493: path-shaped wikilink outside the DIR_PATTERN whitelist.
|
||||
// The written target IS a canonical slug — verify it exists (exact
|
||||
// match, no fuzzing) and then treat it exactly like a whitelisted
|
||||
// wikilink (verb-inferred type, linkSource 'markdown', reconcilable).
|
||||
if (ref.exactPath) {
|
||||
if (!anyDirExactPath) continue; // gated off → legacy silent drop
|
||||
// Qualified refs ([[src:janus/foo]]) pin a target in a possibly
|
||||
// different source; the resolver's snapshot is scoped to THIS source,
|
||||
// so existence is left to the caller's cross-source endpoint check —
|
||||
// the same dead-link protection whitelisted qualified refs get.
|
||||
if (ref.sourceId == null && typeof resolver.slugExists === 'function') {
|
||||
if (!(await resolver.slugExists(ref.slug))) {
|
||||
if (!wikilinkUnresolvedSeen.has(ref.slug)) {
|
||||
wikilinkUnresolvedSeen.add(ref.slug);
|
||||
wikilinkUnresolved.push({ field: 'wikilink', name: ref.slug });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
// Falls through to the shared emission below. When the resolver
|
||||
// lacks slugExists (nullResolver in extract --stale, synthetic test
|
||||
// resolvers), the candidate goes out unverified and the caller's
|
||||
// existing dead-link filter drops misses.
|
||||
}
|
||||
const idx = content.indexOf(ref.name);
|
||||
// Wider context window (240 chars vs original 80) catches verbs that
|
||||
// appear at sentence-or-paragraph distance from the slug — common in
|
||||
@@ -517,6 +673,10 @@ export async function extractPageLinks(
|
||||
linkType: inferLinkType(pageType, context, content, ref.slug),
|
||||
context,
|
||||
linkSource: 'markdown',
|
||||
// Issue #1493 (codex P1): carry the qualified-wikilink source pin
|
||||
// ([[src:slug]] → EntityRef.sourceId) so resolveCandidateSources can
|
||||
// honor it. Applies to whitelisted AND any-dir qualified refs alike.
|
||||
...(ref.sourceId ? { targetSourceId: ref.sourceId } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -524,14 +684,30 @@ export async function extractPageLinks(
|
||||
// Limited to the same entity directories ENTITY_REF_RE covers.
|
||||
// Code blocks are stripped first — slugs in code samples are not real refs.
|
||||
const strippedContent = stripCodeBlocks(content);
|
||||
// Issue #1493 (codex P1, tightened in round 2): mask qualified-wikilink
|
||||
// spans out of the bare-slug scan. The slug tail inside
|
||||
// `[[src:people/alice]]` double-matches here; pass 2a/2a' already emitted
|
||||
// it WITH its source pin, and since the pin is part of the dedup key the
|
||||
// unpinned duplicate would persist a second edge. Masking the exact
|
||||
// wikilink spans (instead of skipping any `:`-preceded match) keeps
|
||||
// legitimate prose like `Owner:people/alice` extracting as before.
|
||||
const qualifiedSpans: Array<[number, number]> = [];
|
||||
for (const re of [QUALIFIED_WIKILINK_RE, QUALIFIED_WIKILINK_ANY_PATH_RE]) {
|
||||
const p = new RegExp(re.source, re.flags);
|
||||
let qm: RegExpExecArray | null;
|
||||
while ((qm = p.exec(strippedContent)) !== null) {
|
||||
qualifiedSpans.push([qm.index, qm.index + qm[0].length]);
|
||||
}
|
||||
}
|
||||
const bareScan = maskRanges(strippedContent, qualifiedSpans);
|
||||
const bareRe = new RegExp(
|
||||
`\\b(${DIR_PATTERN}\\/[a-z0-9][a-z0-9/-]*[a-z0-9])\\b`,
|
||||
'g',
|
||||
);
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = bareRe.exec(strippedContent)) !== null) {
|
||||
while ((m = bareRe.exec(bareScan)) !== null) {
|
||||
// Skip matches that are part of a markdown link (already handled above).
|
||||
const charBefore = m.index > 0 ? strippedContent[m.index - 1] : '';
|
||||
const charBefore = m.index > 0 ? bareScan[m.index - 1] : '';
|
||||
if (charBefore === '/' || charBefore === '(') continue;
|
||||
const context = excerpt(strippedContent, m.index, 240);
|
||||
candidates.push({
|
||||
@@ -568,12 +744,17 @@ export async function extractPageLinks(
|
||||
const seen = new Set<string>();
|
||||
const result: LinkCandidate[] = [];
|
||||
for (const c of candidates) {
|
||||
const key = `${c.fromSlug ?? ''}\u0000${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? ''}`;
|
||||
// Issue #1493 (codex P1): targetSourceId participates in the key — a
|
||||
// pinned `[[b:slug]]` and an unqualified `[[slug]]` can resolve to
|
||||
// different (to_slug, to_source_id) rows, so they must not collapse.
|
||||
const key = `${c.fromSlug ?? ''}\u0000${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? ''}\u0000${c.targetSourceId ?? ''}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
result.push(c);
|
||||
}
|
||||
return { candidates: result, unresolved: fmUnresolved };
|
||||
// Issue #1493: wikilink misses ride alongside frontmatter misses in the
|
||||
// one unresolved report.
|
||||
return { candidates: result, unresolved: [...fmUnresolved, ...wikilinkUnresolved] };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -819,6 +1000,20 @@ export interface SlugResolver {
|
||||
* a single extract run (build the index once, reuse it).
|
||||
*/
|
||||
resolveBasenameMatches?(name: string): Promise<string[]>;
|
||||
/**
|
||||
* Issue #1493 (generalized path resolution): exact-match existence check
|
||||
* for a canonical slug. Used by extractPageLinks to verify path-shaped
|
||||
* wikilink targets outside the DIR_PATTERN whitelist before emitting a
|
||||
* candidate — a miss goes into the unresolved report, never a ghost edge.
|
||||
*
|
||||
* Optional like resolveBasenameMatches. When absent, extractPageLinks
|
||||
* emits the candidate unverified and relies on the caller's existing
|
||||
* dead-link protection (runAutoLink's allSlugs filter /
|
||||
* resolveCandidateSources' endpoint check) — same treatment whitelisted
|
||||
* refs get. Implementations should share the per-run slug snapshot with
|
||||
* the basename index (one getAllSlugs call, reused).
|
||||
*/
|
||||
slugExists?(slug: string): Promise<boolean>;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -899,30 +1094,63 @@ export function makeResolver(
|
||||
// index keys BOTH the raw tail and the slugified-tail so a wikilink
|
||||
// `[[Fast-Weigh]]` matches a page slugged `companies/fast-weigh` as
|
||||
// well as `notes/Fast-Weigh`.
|
||||
let basenameIndex: Map<string, string[]> | null = null;
|
||||
async function ensureBasenameIndex(): Promise<Map<string, string[]>> {
|
||||
if (basenameIndex !== null) return basenameIndex;
|
||||
const idx = new Map<string, string[]>();
|
||||
// Issue #1493: the raw slug snapshot backing BOTH the basename index and
|
||||
// the exact-match slugExists check. One getAllSlugs call per resolver
|
||||
// instance, shared by both consumers. Load failure (or an engine without
|
||||
// getAllSlugs) degrades to an empty set — resolveBasenameMatches finds
|
||||
// nothing and slugExists returns false. Never throws.
|
||||
let allSlugsSet: Set<string> | null = null;
|
||||
async function ensureAllSlugsSet(): Promise<Set<string>> {
|
||||
if (allSlugsSet !== null) return allSlugsSet;
|
||||
if (typeof engine.getAllSlugs !== 'function') {
|
||||
basenameIndex = idx;
|
||||
return idx;
|
||||
allSlugsSet = new Set();
|
||||
return allSlugsSet;
|
||||
}
|
||||
try {
|
||||
// Issue #972 (codex [P1]): scope the basename index to the resolver's
|
||||
// Issue #972 (codex [P1]): scope the slug snapshot to the resolver's
|
||||
// source. getAllSlugs({sourceId}) keeps wikilink resolution from
|
||||
// spanning unrelated sources — a bare [[name]] must NOT resolve to a
|
||||
// same-tail page in a DIFFERENT source and create a cross-source edge.
|
||||
// #972 is "global basename across folders," not "cross-source federation."
|
||||
const all = await engine.getAllSlugs(opts.sourceId ? { sourceId: opts.sourceId } : undefined);
|
||||
// Issue #972 (codex [P2] DRY): one shared index builder for all surfaces.
|
||||
basenameIndex = buildBasenameIndex(all);
|
||||
return basenameIndex;
|
||||
allSlugsSet = all instanceof Set ? all : new Set(all);
|
||||
} catch {
|
||||
// Index build failed — empty map → resolveBasenameMatches finds
|
||||
// nothing, resolver continues as if the flag was off. Never throw.
|
||||
allSlugsSet = new Set();
|
||||
}
|
||||
basenameIndex = idx;
|
||||
return idx;
|
||||
return allSlugsSet;
|
||||
}
|
||||
|
||||
let basenameIndex: Map<string, string[]> | null = null;
|
||||
async function ensureBasenameIndex(): Promise<Map<string, string[]>> {
|
||||
if (basenameIndex !== null) return basenameIndex;
|
||||
// Issue #972 (codex [P2] DRY): one shared index builder for all surfaces.
|
||||
basenameIndex = buildBasenameIndex(await ensureAllSlugsSet());
|
||||
return basenameIndex;
|
||||
}
|
||||
|
||||
// Issue #1493 (codex P1): slugExists domain = scoped source ∪ 'default'.
|
||||
// resolveCandidateSources resolves an unqualified target local-first,
|
||||
// THEN falls back to the default source — so a source-scoped extract of
|
||||
// source `a` must not pre-discard a candidate whose target lives only in
|
||||
// `default` (whitelisted refs bypass slugExists and survive to that
|
||||
// fallback; any-dir refs must survive the SAME resolution domain). The
|
||||
// basename index deliberately stays scoped (#972 [P1]: no cross-source
|
||||
// basename edges) — only exact-match existence gets the default overlay.
|
||||
let slugExistsDomain: Set<string> | null = null;
|
||||
async function ensureSlugExistsDomain(): Promise<Set<string>> {
|
||||
if (slugExistsDomain !== null) return slugExistsDomain;
|
||||
const scoped = await ensureAllSlugsSet();
|
||||
if (!opts.sourceId || opts.sourceId === 'default' || typeof engine.getAllSlugs !== 'function') {
|
||||
slugExistsDomain = scoped;
|
||||
return slugExistsDomain;
|
||||
}
|
||||
try {
|
||||
const def = await engine.getAllSlugs({ sourceId: 'default' });
|
||||
slugExistsDomain = new Set([...scoped, ...def]);
|
||||
} catch {
|
||||
slugExistsDomain = scoped;
|
||||
}
|
||||
return slugExistsDomain;
|
||||
}
|
||||
|
||||
return {
|
||||
@@ -932,6 +1160,14 @@ export function makeResolver(
|
||||
return queryBasenameIndex(await ensureBasenameIndex(), name);
|
||||
},
|
||||
|
||||
async slugExists(slug: string): Promise<boolean> {
|
||||
// Issue #1493: exact-match existence for path-shaped wikilink targets.
|
||||
// Scoped-source snapshot plus the default-source fallback overlay
|
||||
// (mirrors resolveCandidateSources' resolution domain); no fuzzing.
|
||||
if (!slug || typeof slug !== 'string') return false;
|
||||
return (await ensureSlugExistsDomain()).has(slug);
|
||||
},
|
||||
|
||||
async resolve(name: string, dirHint?: string | string[]): Promise<string | null> {
|
||||
if (!name || typeof name !== 'string') return null;
|
||||
const trimmed = name.trim();
|
||||
@@ -1252,3 +1488,35 @@ export async function isGlobalBasenameEnabled(engine: BrainEngine): Promise<bool
|
||||
const normalized = val.trim().toLowerCase();
|
||||
return ['1', 'true', 'yes', 'on'].includes(normalized);
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue #1493: read the `link_resolution.any_dir_exact_path` config flag.
|
||||
* Defaults to TRUE — unlike global_basename, exact-path resolution matches
|
||||
* the literal written slug against existing pages only, so it cannot
|
||||
* false-positive and needs no opt-in. The flag exists as an escape hatch
|
||||
* for brains that intentionally author `[[dir/name]]` prose that must
|
||||
* never link.
|
||||
*
|
||||
* When TRUE: path-shaped wikilinks like `[[janus/agents/drift-check]]`
|
||||
* whose top-level dir is NOT in the DIR_PATTERN whitelist emit a link
|
||||
* candidate when a page with that exact slug exists; misses are recorded
|
||||
* in the unresolved report (field='wikilink').
|
||||
*
|
||||
* Resolution order (highest → lowest), mirroring isGlobalBasenameEnabled:
|
||||
* 1. Env var `GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH=0` (operator override)
|
||||
* 2. DB plane via `engine.getConfig('link_resolution.any_dir_exact_path')`
|
||||
* 3. Default true
|
||||
*
|
||||
* Falsy spellings match isAutoLinkEnabled (default-on flags): 'false',
|
||||
* '0', 'no', 'off' — anything else stays on.
|
||||
*/
|
||||
export async function isAnyDirExactPathEnabled(engine: BrainEngine): Promise<boolean> {
|
||||
const OFF = ['false', '0', 'no', 'off'];
|
||||
const envVal = process.env.GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH;
|
||||
if (envVal != null) {
|
||||
return !OFF.includes(envVal.trim().toLowerCase());
|
||||
}
|
||||
const val = await engine.getConfig('link_resolution.any_dir_exact_path');
|
||||
if (val == null) return true;
|
||||
return !OFF.includes(val.trim().toLowerCase());
|
||||
}
|
||||
|
||||
+30
-4
@@ -16,7 +16,7 @@ import { expandQuery } from './search/expansion.ts';
|
||||
import { dedupResults } from './search/dedup.ts';
|
||||
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
|
||||
import type { HybridSearchMeta } from './types.ts';
|
||||
import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, isGlobalBasenameEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
|
||||
import { extractPageLinks, isAnyDirExactPathEnabled, isAutoLinkEnabled, isAutoTimelineEnabled, isGlobalBasenameEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
|
||||
import { isFactsBackstopEligible } from './facts/eligibility.ts';
|
||||
import { stripTakesFence } from './takes-fence.ts';
|
||||
import { stripFactsFence } from './facts-fence.ts';
|
||||
@@ -1157,9 +1157,12 @@ async function runAutoLink(
|
||||
const resolver = makeResolver(engine, { mode: 'live', sourceId: opts?.sourceId });
|
||||
// Issue #972: opt-in bare-wikilink basename resolution. Off by default.
|
||||
const globalBasename = await isGlobalBasenameEnabled(engine);
|
||||
// Issue #1493: exact-path resolution for wikilinks outside the DIR_PATTERN
|
||||
// whitelist. On by default (escape hatch: link_resolution.any_dir_exact_path).
|
||||
const anyDirExactPath = await isAnyDirExactPathEnabled(engine);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
slug, fullContent, parsed.frontmatter, parsed.type, resolver,
|
||||
{ globalBasename },
|
||||
{ globalBasename, anyDirExactPath },
|
||||
);
|
||||
|
||||
// Resolve which targets exist (skip refs to non-existent pages to avoid FK
|
||||
@@ -1167,9 +1170,32 @@ async function runAutoLink(
|
||||
// v0.31.8 (D12): scoped to the source when opts.sourceId is set so wikilink
|
||||
// resolution doesn't span unrelated sources.
|
||||
const allSlugs = await engine.getAllSlugs(sourceOpts);
|
||||
const valid = candidates.filter(c =>
|
||||
allSlugs.has(c.targetSlug) && (!c.fromSlug || allSlugs.has(c.fromSlug))
|
||||
// Issue #1493 (codex P1): put_page auto-link writes are single-source
|
||||
// (linkSourceOpts pins from/to to the page's own source), so a qualified
|
||||
// wikilink pinned to a DIFFERENT source cannot be honored here — skip it
|
||||
// rather than misdirect the edge to a same-slug page in this source.
|
||||
// Cross-source pinned edges are the extract paths' job
|
||||
// (resolveCandidateSources honors targetSourceId).
|
||||
const effectiveSourceId = opts?.sourceId ?? 'default';
|
||||
const accepted = candidates.filter(c =>
|
||||
allSlugs.has(c.targetSlug) && (!c.fromSlug || allSlugs.has(c.fromSlug)) &&
|
||||
(!c.targetSourceId || c.targetSourceId === effectiveSourceId)
|
||||
);
|
||||
// Issue #1493 (codex P2, round 2): within put_page's single-source write
|
||||
// scope, a pinned `[[alpha:people/alice]]` and an unpinned
|
||||
// `[[people/alice]]` on an alpha page resolve to the SAME edge row — the
|
||||
// pin survived extractPageLinks' within-page dedup (it's part of that
|
||||
// key) and the guard above (pin === this source). Re-dedupe by the
|
||||
// RESOLVED single-source edge key so addLink isn't called twice and
|
||||
// `created` isn't double-counted (the second insert hits ON CONFLICT,
|
||||
// leaving one row but two counts).
|
||||
const seenEdgeKeys = new Set<string>();
|
||||
const valid = accepted.filter(c => {
|
||||
const key = `${c.fromSlug ?? slug}\u0000${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? 'markdown'}`;
|
||||
if (seenEdgeKeys.has(key)) return false;
|
||||
seenEdgeKeys.add(key);
|
||||
return true;
|
||||
});
|
||||
|
||||
// Split candidates by direction. Outgoing (fromSlug === slug or unset) are
|
||||
// this page's own edges, reconciled against getLinks(slug). Incoming
|
||||
|
||||
@@ -264,4 +264,13 @@ describe('KNOWN_CONFIG_KEYS — documented enable commands must be registered',
|
||||
// chronicle.tz (chronicleTz) + future chronicle.* knobs.
|
||||
expect(KNOWN_CONFIG_KEY_PREFIXES.some(p => 'chronicle.tz'.startsWith(p))).toBe(true);
|
||||
});
|
||||
|
||||
test('link_resolution keys are registered (#972 opt-in + #1493 escape hatch)', async () => {
|
||||
const { KNOWN_CONFIG_KEYS } = await import('../src/core/config.ts');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('link_resolution.global_basename');
|
||||
// Issue #1493 (codex P2): the documented disable command is
|
||||
// `gbrain config set link_resolution.any_dir_exact_path false` — it
|
||||
// must work without --force.
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('link_resolution.any_dir_exact_path');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -249,3 +249,57 @@ describe('gbrain extract all --source db', () => {
|
||||
expect(entries.length).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Issue #1493 (codex P2): unresolved refs in the --json result ────
|
||||
|
||||
describe('extract links --source db --json: unresolved refs (#1493)', () => {
|
||||
beforeEach(truncateAll);
|
||||
|
||||
const conceptPage = (title: string, body = ''): PageInput => ({
|
||||
type: 'concept', title, compiled_truth: body, timeline: '',
|
||||
});
|
||||
|
||||
test('any-dir wikilink miss appears in the final JSON result', async () => {
|
||||
await engine.putPage('concepts/note', conceptPage('Note', 'See [[janus/never-existed]].'));
|
||||
|
||||
const logged: string[] = [];
|
||||
const origLog = console.log;
|
||||
console.log = (m: unknown) => { logged.push(String(m)); };
|
||||
try {
|
||||
await runExtract(engine, ['links', '--source', 'db', '--json']);
|
||||
} finally {
|
||||
console.log = origLog;
|
||||
}
|
||||
|
||||
const result = logged
|
||||
.filter(l => l.trim().startsWith('{'))
|
||||
.map(l => JSON.parse(l))
|
||||
.find(o => 'links_created' in o);
|
||||
expect(result).toBeDefined();
|
||||
expect(result.unresolved_refs).toContainEqual({ field: 'wikilink', name: 'janus/never-existed' });
|
||||
expect((await engine.getLinks('concepts/note')).length).toBe(0);
|
||||
});
|
||||
|
||||
test('clean run has no unresolved_refs key (shape back-compat)', async () => {
|
||||
await engine.putPage('janus/real', conceptPage('Real', 'target'));
|
||||
await engine.putPage('concepts/note', conceptPage('Note', 'See [[janus/real]].'));
|
||||
|
||||
const logged: string[] = [];
|
||||
const origLog = console.log;
|
||||
console.log = (m: unknown) => { logged.push(String(m)); };
|
||||
try {
|
||||
await runExtract(engine, ['links', '--source', 'db', '--json']);
|
||||
} finally {
|
||||
console.log = origLog;
|
||||
}
|
||||
|
||||
const result = logged
|
||||
.filter(l => l.trim().startsWith('{'))
|
||||
.map(l => JSON.parse(l))
|
||||
.find(o => 'links_created' in o);
|
||||
expect(result).toBeDefined();
|
||||
expect(result.unresolved_refs).toBeUndefined();
|
||||
// The any-dir link itself landed.
|
||||
expect((await engine.getLinks('concepts/note')).some((l: any) => l.to_slug === 'janus/real')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -123,3 +123,94 @@ describe('extract --source-id flag (#1204)', () => {
|
||||
expect(Number(linkRows[0]?.n ?? 0)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Issue #1493 (codex P1): qualified pin + scoped default fallback ──
|
||||
|
||||
describe('extract --source db: any-dir wikilinks across sources (#1493)', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ('alpha', 'alpha'), ('beta', 'beta')
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
});
|
||||
|
||||
async function runLinksDb(extra: string[] = []): Promise<void> {
|
||||
const origLog = console.log;
|
||||
console.log = () => {};
|
||||
try {
|
||||
await runExtract(engine, ['links', '--source', 'db', ...extra, '--json']);
|
||||
} finally {
|
||||
console.log = origLog;
|
||||
}
|
||||
}
|
||||
|
||||
test('qualified [[beta:janus/target]] links to the PINNED source', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES
|
||||
('concepts/note', 'alpha', 'concept', 'Note', 'See [[beta:janus/target]].', ''),
|
||||
('janus/target', 'beta', 'concept', 'Target', 'The target.', '')`,
|
||||
);
|
||||
await runLinksDb();
|
||||
const rows = await engine.executeRaw<{ to_slug: string; to_source: string }>(
|
||||
`SELECT t.slug AS to_slug, t.source_id AS to_source
|
||||
FROM links l JOIN pages t ON l.to_page_id = t.id`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].to_slug).toBe('janus/target');
|
||||
expect(rows[0].to_source).toBe('beta');
|
||||
});
|
||||
|
||||
test('pin miss is NOT misdirected to a same-slug page in origin/default', async () => {
|
||||
// janus/dup exists in alpha AND default — but the wikilink pins beta,
|
||||
// where it does not exist. No edge at all (skip beats misdirect).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES
|
||||
('concepts/note', 'alpha', 'concept', 'Note', 'See [[beta:janus/dup]].', ''),
|
||||
('janus/dup', 'alpha', 'concept', 'Dup A', 'x', ''),
|
||||
('janus/dup', 'default', 'concept', 'Dup D', 'x', '')`,
|
||||
);
|
||||
await runLinksDb();
|
||||
const rows = await engine.executeRaw<{ n: string }>(`SELECT COUNT(*)::text AS n FROM links`);
|
||||
expect(Number(rows[0]?.n ?? 0)).toBe(0);
|
||||
});
|
||||
|
||||
test('scoped --source-id run keeps the default-source fallback for unqualified any-dir links', async () => {
|
||||
// The target lives ONLY in the default source; the scoped slugExists
|
||||
// snapshot must include the default overlay so the candidate survives
|
||||
// to resolveCandidateSources' documented fallback (parity with
|
||||
// whitelisted refs, which bypass slugExists).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES
|
||||
('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/shared-doc]].', ''),
|
||||
('janus/shared-doc', 'default', 'concept', 'Shared', 'The doc.', '')`,
|
||||
);
|
||||
await runLinksDb(['--source-id', 'alpha']);
|
||||
const rows = await engine.executeRaw<{ to_slug: string; to_source: string; from_source: string }>(
|
||||
`SELECT t.slug AS to_slug, t.source_id AS to_source, f.source_id AS from_source
|
||||
FROM links l JOIN pages t ON l.to_page_id = t.id JOIN pages f ON l.from_page_id = f.id`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].from_source).toBe('alpha');
|
||||
expect(rows[0].to_slug).toBe('janus/shared-doc');
|
||||
expect(rows[0].to_source).toBe('default');
|
||||
});
|
||||
|
||||
test('scoped run still refuses cross-source (non-default) any-dir resolution', async () => {
|
||||
// Target exists only in beta. Scoped alpha run: slugExists domain is
|
||||
// alpha ∪ default → miss → recorded unresolved, no edge (a bare
|
||||
// unqualified link must not silently span unrelated sources).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES
|
||||
('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/foreign]].', ''),
|
||||
('janus/foreign', 'beta', 'concept', 'Foreign', 'x', '')`,
|
||||
);
|
||||
await runLinksDb(['--source-id', 'alpha']);
|
||||
const rows = await engine.executeRaw<{ n: string }>(`SELECT COUNT(*)::text AS n FROM links`);
|
||||
expect(Number(rows[0]?.n ?? 0)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
+134
-1
@@ -188,7 +188,8 @@ describe('gbrain extract --stale', () => {
|
||||
await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) advises [Acme](companies/acme).'));
|
||||
// Microsecond-precision updated_at, recent (after LINK_EXTRACTOR_VERSION_TS) so the
|
||||
// version arm doesn't fire — the edited arm is what must clear.
|
||||
await engine.executeRaw(`UPDATE pages SET updated_at = '2026-06-02 08:18:58.999166+00'`);
|
||||
// (#1493: fixture date advanced past the bumped watermark.)
|
||||
await engine.executeRaw(`UPDATE pages SET updated_at = '2026-08-02 08:18:58.999166+00'`);
|
||||
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(2);
|
||||
|
||||
await runExtract(engine, ['--stale']);
|
||||
@@ -290,3 +291,135 @@ describe('gbrain extract --stale', () => {
|
||||
expect(msg).toContain('DB-source only');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Issue #1493 (codex P2): any-dir links + unresolved surfacing ────
|
||||
|
||||
describe('extract --stale: any-dir exact-path wikilinks (#1493)', () => {
|
||||
test('creates the edge for an existing any-dir target', async () => {
|
||||
await engine.putPage('janus/real', { type: 'concept', title: 'Real', compiled_truth: 'The target.', timeline: '' });
|
||||
await engine.putPage('concepts/note', { type: 'concept', title: 'Note', compiled_truth: 'See [[janus/real]].', timeline: '' });
|
||||
|
||||
await runExtract(engine, ['--stale']);
|
||||
|
||||
const links = await engine.getLinks('concepts/note');
|
||||
expect(links.some(l => l.to_slug === 'janus/real')).toBe(true);
|
||||
});
|
||||
|
||||
test('misses are surfaced in the --json sweep summary, not silently dropped', async () => {
|
||||
await engine.putPage('concepts/note', {
|
||||
type: 'concept', title: 'Note',
|
||||
compiled_truth: 'See [[janus/never-existed]] and [[janus/also-gone]].', timeline: '',
|
||||
});
|
||||
|
||||
const lines: string[] = [];
|
||||
const originalWrite = process.stdout.write.bind(process.stdout);
|
||||
process.stdout.write = ((chunk: string | Uint8Array): boolean => {
|
||||
lines.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8'));
|
||||
return true;
|
||||
}) as any;
|
||||
try {
|
||||
await runExtract(engine, ['--stale', '--json']);
|
||||
} finally {
|
||||
process.stdout.write = originalWrite;
|
||||
}
|
||||
|
||||
const done = lines
|
||||
.filter(l => l.trim().startsWith('{'))
|
||||
.map(l => JSON.parse(l.trim()))
|
||||
.find(o => o.action === 'extract_stale_done');
|
||||
expect(done).toBeDefined();
|
||||
expect(done.unresolved_count).toBe(2);
|
||||
expect(done.unresolved_refs).toContainEqual({ field: 'wikilink', name: 'janus/never-existed' });
|
||||
expect(done.unresolved_refs).toContainEqual({ field: 'wikilink', name: 'janus/also-gone' });
|
||||
// And no ghost edges were written.
|
||||
expect((await engine.getLinks('concepts/note')).length).toBe(0);
|
||||
});
|
||||
|
||||
test('clean sweep reports unresolved_count 0 with no unresolved_refs key', async () => {
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
|
||||
const lines: string[] = [];
|
||||
const originalWrite = process.stdout.write.bind(process.stdout);
|
||||
process.stdout.write = ((chunk: string | Uint8Array): boolean => {
|
||||
lines.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8'));
|
||||
return true;
|
||||
}) as any;
|
||||
try {
|
||||
await runExtract(engine, ['--stale', '--json']);
|
||||
} finally {
|
||||
process.stdout.write = originalWrite;
|
||||
}
|
||||
|
||||
const done = lines
|
||||
.filter(l => l.trim().startsWith('{'))
|
||||
.map(l => JSON.parse(l.trim()))
|
||||
.find(o => o.action === 'extract_stale_done');
|
||||
expect(done).toBeDefined();
|
||||
expect(done.unresolved_count).toBe(0);
|
||||
expect(done.unresolved_refs).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Issue #1493 (codex P2, round 2): per-page-source resolution domain ──
|
||||
|
||||
describe('extract --stale: per-source resolution domain (#1493 round 2)', () => {
|
||||
test('alpha page linking a beta-only target records the miss (no silent drop, no cross-source edge)', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ('alpha', 'alpha'), ('beta', 'beta')
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES
|
||||
('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/foreign]].', ''),
|
||||
('janus/foreign', 'beta', 'concept', 'Foreign', 'x', '')`,
|
||||
);
|
||||
|
||||
const lines: string[] = [];
|
||||
const originalWrite = process.stdout.write.bind(process.stdout);
|
||||
process.stdout.write = ((chunk: string | Uint8Array): boolean => {
|
||||
lines.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8'));
|
||||
return true;
|
||||
}) as any;
|
||||
try {
|
||||
await runExtract(engine, ['--stale', '--json']);
|
||||
} finally {
|
||||
process.stdout.write = originalWrite;
|
||||
}
|
||||
|
||||
const done = lines
|
||||
.filter(l => l.trim().startsWith('{'))
|
||||
.map(l => JSON.parse(l.trim()))
|
||||
.find(o => o.action === 'extract_stale_done');
|
||||
expect(done).toBeDefined();
|
||||
// The alpha page's resolution domain is alpha ∪ default — the beta-only
|
||||
// target is a MISS and must be recorded, not silently dropped after
|
||||
// passing an unscoped existence check.
|
||||
expect(done.unresolved_refs ?? []).toContainEqual({ field: 'wikilink', name: 'janus/foreign' });
|
||||
const linkRows = await engine.executeRaw<{ n: string }>(`SELECT COUNT(*)::text AS n FROM links`);
|
||||
expect(Number(linkRows[0]?.n ?? 0)).toBe(0);
|
||||
});
|
||||
|
||||
test('alpha page linking a default-source target resolves via the default overlay', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ('alpha', 'alpha')
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES
|
||||
('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/shared]].', ''),
|
||||
('janus/shared', 'default', 'concept', 'Shared', 'x', '')`,
|
||||
);
|
||||
|
||||
await runExtract(engine, ['--stale']);
|
||||
|
||||
const rows = await engine.executeRaw<{ to_slug: string; to_source: string }>(
|
||||
`SELECT t.slug AS to_slug, t.source_id AS to_source
|
||||
FROM links l JOIN pages t ON l.to_page_id = t.id`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].to_slug).toBe('janus/shared');
|
||||
expect(rows[0].to_source).toBe('default');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,607 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
extractEntityRefs,
|
||||
extractPageLinks,
|
||||
isAnyDirExactPathEnabled,
|
||||
makeResolver,
|
||||
type LinkCandidate,
|
||||
type SlugResolver,
|
||||
} from '../src/core/link-extraction.ts';
|
||||
import { resolveCandidateSources } from '../src/commands/extract.ts';
|
||||
import { parseMarkdown } from '../src/core/markdown.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
// ─── Issue #1493: generalized path resolution ─────────────────────
|
||||
//
|
||||
// Path-shaped wikilinks ([[dir/name]], [[dir/sub/name]]) whose top-level
|
||||
// directory is NOT in the DIR_PATTERN whitelist used to be dropped
|
||||
// silently — no candidate, no unresolved report. They now extract via the
|
||||
// 2a'/2b' passes (tagged `exactPath: true`) and resolve by EXACT slug
|
||||
// match against existing pages. On by default
|
||||
// (link_resolution.any_dir_exact_path); misses are recorded in the
|
||||
// unresolved report as field='wikilink'.
|
||||
|
||||
// ─── extractEntityRefs — passes 2a'/2b' ───────────────────────────
|
||||
|
||||
describe('extractEntityRefs — any-dir path-shaped wikilinks (#1493)', () => {
|
||||
test('2-segment path outside the whitelist extracts with exactPath flag', () => {
|
||||
const refs = extractEntityRefs('See [[janus/foo]] for context.');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('janus/foo');
|
||||
expect(refs[0].dir).toBe('janus');
|
||||
expect(refs[0].exactPath).toBe(true);
|
||||
// NOT the #972 basename pass — that one carries needsResolution.
|
||||
expect(refs[0].needsResolution).toBeUndefined();
|
||||
});
|
||||
|
||||
test('deep nesting: [[janus/agents/drift-check]]', () => {
|
||||
const refs = extractEntityRefs('Owned by [[janus/agents/drift-check]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('janus/agents/drift-check');
|
||||
expect(refs[0].dir).toBe('janus');
|
||||
expect(refs[0].exactPath).toBe(true);
|
||||
});
|
||||
|
||||
test('hyphenated directory: [[janus-ui/docs/tier2-agent-plan]]', () => {
|
||||
const refs = extractEntityRefs('Plan: [[janus-ui/docs/tier2-agent-plan]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('janus-ui/docs/tier2-agent-plan');
|
||||
expect(refs[0].dir).toBe('janus-ui');
|
||||
});
|
||||
|
||||
test('alias form: [[janus/foo|Display]] keeps target, uses display name', () => {
|
||||
const refs = extractEntityRefs('See [[janus/foo|Display]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('janus/foo');
|
||||
expect(refs[0].name).toBe('Display');
|
||||
});
|
||||
|
||||
test('fragment stripped: [[janus/foo#section]]', () => {
|
||||
const refs = extractEntityRefs('Jump to [[janus/foo#section]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('janus/foo');
|
||||
});
|
||||
|
||||
test('.md suffix stripped: [[janus/foo.md]]', () => {
|
||||
const refs = extractEntityRefs('See [[janus/foo.md]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('janus/foo');
|
||||
});
|
||||
|
||||
test('dotted segments survive when not a .md suffix: [[notes/v1.0.0]]', () => {
|
||||
const refs = extractEntityRefs('Released [[notes/v1.0.0]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('notes/v1.0.0');
|
||||
});
|
||||
|
||||
test('qualified variant: [[src:janus/foo]] carries sourceId + exactPath', () => {
|
||||
const refs = extractEntityRefs('See [[src:janus/foo]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0]).toMatchObject({
|
||||
slug: 'janus/foo', dir: 'janus', sourceId: 'src', exactPath: true,
|
||||
});
|
||||
});
|
||||
|
||||
test('qualified whitelist links still go through the 2a pass unchanged', () => {
|
||||
// [[wiki:concepts/ai]] is 2a territory (whitelisted dir) — must not be
|
||||
// double-emitted by the new 2a' pass.
|
||||
const refs = extractEntityRefs('Ref: [[wiki:concepts/ai]]');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].sourceId).toBe('wiki');
|
||||
expect(refs[0].exactPath).toBeUndefined();
|
||||
});
|
||||
|
||||
test('wikilinks inside fenced and inline code are ignored', () => {
|
||||
const refs = extractEntityRefs(
|
||||
'```\n[[janus/in-fence]]\n```\nInline `[[janus/in-tick]]` too.\nReal: [[janus/real]].',
|
||||
);
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('janus/real');
|
||||
});
|
||||
|
||||
test('URLs are not matched', () => {
|
||||
const refs = extractEntityRefs('See [[https://example.com/foo]] and https://example.com/bar.');
|
||||
expect(refs.filter(r => r.exactPath)).toEqual([]);
|
||||
});
|
||||
|
||||
test('uppercase target is NOT path-shaped (slug grammar is lowercase) — falls to the generic pass', () => {
|
||||
// Engine slugs are lowercase (sync.ts slugifySegment). [[Janus/Foo]] is
|
||||
// not a canonical slug, so it keeps the pre-#1493 behavior: generic
|
||||
// pass 2c (needsResolution — basename resolution or silent drop).
|
||||
const refs = extractEntityRefs('See [[Janus/Foo]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].exactPath).toBeUndefined();
|
||||
expect(refs[0].needsResolution).toBe(true);
|
||||
expect(refs[0].slug).toBe('Janus/Foo');
|
||||
});
|
||||
|
||||
test('whitelisted dirs keep their existing pass (no exactPath tag)', () => {
|
||||
const refs = extractEntityRefs('See [[people/alice]] and [[janus/foo]].');
|
||||
const alice = refs.find(r => r.slug === 'people/alice');
|
||||
const janus = refs.find(r => r.slug === 'janus/foo');
|
||||
expect(alice).toBeDefined();
|
||||
expect(alice!.exactPath).toBeUndefined();
|
||||
expect(janus).toBeDefined();
|
||||
expect(janus!.exactPath).toBe(true);
|
||||
expect(refs.length).toBe(2); // no double emission
|
||||
});
|
||||
|
||||
test('bare-name #972 pass is unaffected', () => {
|
||||
const refs = extractEntityRefs('See [[struktura]] and [[janus/foo]].');
|
||||
const bare = refs.find(r => r.slug === 'struktura');
|
||||
expect(bare).toBeDefined();
|
||||
expect(bare!.needsResolution).toBe(true);
|
||||
expect(bare!.exactPath).toBeUndefined();
|
||||
});
|
||||
|
||||
test('a path wikilink inside a markdown-link label is inert (mirrors codex P2a)', () => {
|
||||
const refs = extractEntityRefs('[see [[janus/foo]]](companies/acme.md)');
|
||||
expect(refs.filter(r => r.exactPath)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── extractPageLinks — exact-slug resolution + unresolved report ──
|
||||
|
||||
/** Resolver fixture backed by an explicit slug set (mirrors makeResolver's slugExists). */
|
||||
function makeExistsResolver(slugs: string[]): SlugResolver {
|
||||
const set = new Set(slugs);
|
||||
return {
|
||||
resolve: async () => null,
|
||||
slugExists: async (slug: string) => set.has(slug),
|
||||
};
|
||||
}
|
||||
|
||||
describe('extractPageLinks — any-dir exact-path resolution (#1493)', () => {
|
||||
test('existing path target emits a candidate like a whitelisted wikilink', async () => {
|
||||
const resolver = makeExistsResolver(['janus/agents/drift-check']);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'Owned by [[janus/agents/drift-check]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('janus/agents/drift-check');
|
||||
// Same treatment as whitelisted refs: verb-inferred type + markdown
|
||||
// provenance (NOT the #972 wikilink_basename/wikilink-resolved pair).
|
||||
expect(candidates[0].linkType).toBe('mentions');
|
||||
expect(candidates[0].linkSource).toBe('markdown');
|
||||
expect(unresolved).toEqual([]);
|
||||
});
|
||||
|
||||
test('missing path target is RECORDED as unresolved, never a ghost edge', async () => {
|
||||
const resolver = makeExistsResolver([]);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'See [[janus/never-existed]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates).toEqual([]);
|
||||
expect(unresolved).toEqual([{ field: 'wikilink', name: 'janus/never-existed' }]);
|
||||
});
|
||||
|
||||
test('repeated misses of the same target dedup to one unresolved entry', async () => {
|
||||
const resolver = makeExistsResolver([]);
|
||||
const { unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'See [[janus/gone]] and again [[janus/gone]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(unresolved).toEqual([{ field: 'wikilink', name: 'janus/gone' }]);
|
||||
});
|
||||
|
||||
test('opts.anyDirExactPath false restores the legacy silent drop', async () => {
|
||||
const resolver = makeExistsResolver(['janus/foo']);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'See [[janus/foo]].',
|
||||
{}, 'concept', resolver, { anyDirExactPath: false },
|
||||
);
|
||||
expect(candidates).toEqual([]);
|
||||
expect(unresolved).toEqual([]);
|
||||
});
|
||||
|
||||
test('resolver without slugExists emits unverified (caller dead-link filter applies)', async () => {
|
||||
// extract --stale runs a nullResolver with no slugExists — candidates
|
||||
// must still come out so resolveCandidateSources can endpoint-check
|
||||
// them, the same protection whitelisted refs rely on.
|
||||
const resolver: SlugResolver = { resolve: async () => null };
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'See [[janus/foo]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('janus/foo');
|
||||
expect(unresolved).toEqual([]);
|
||||
});
|
||||
|
||||
test('qualified path ref skips slugExists (cross-source target) and emits', async () => {
|
||||
// [[src:janus/foo]] pins a target that may live in a DIFFERENT source;
|
||||
// the resolver's snapshot is scoped to this source, so existence is the
|
||||
// caller's cross-source endpoint check — even a resolver that says
|
||||
// "does not exist here" must not block it.
|
||||
const resolver = makeExistsResolver([]);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'See [[src:janus/foo]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('janus/foo');
|
||||
expect(unresolved).toEqual([]);
|
||||
});
|
||||
|
||||
test('alias + fragment + .md forms all resolve to the same page', async () => {
|
||||
const resolver = makeExistsResolver(['janus/foo']);
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x',
|
||||
'A [[janus/foo|Display]] B [[janus/foo#section]] C [[janus/foo.md]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
// Within-page dedup collapses the three mentions to one candidate.
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('janus/foo');
|
||||
});
|
||||
|
||||
test('whitelisted dirs behave exactly as before (no existence gate at this layer)', async () => {
|
||||
// Whitelisted refs are emitted unverified — callers filter. The #1493
|
||||
// pass must not change that, even with a resolver that would say no.
|
||||
const resolver = makeExistsResolver([]);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'See [[people/alice]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('people/alice');
|
||||
expect(unresolved).toEqual([]);
|
||||
});
|
||||
|
||||
test('#972 bare-name pass composes: basename resolution still works alongside', async () => {
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
slugExists: async (slug) => slug === 'janus/foo',
|
||||
resolveBasenameMatches: async (name) =>
|
||||
name === 'struktura' ? ['projects/struktura'] : [],
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'See [[janus/foo]] and [[struktura]].',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
const exact = candidates.find(c => c.targetSlug === 'janus/foo');
|
||||
const basename = candidates.find(c => c.targetSlug === 'projects/struktura');
|
||||
expect(exact).toBeDefined();
|
||||
expect(exact!.linkSource).toBe('markdown');
|
||||
expect(basename).toBeDefined();
|
||||
expect(basename!.linkType).toBe('wikilink_basename');
|
||||
expect(basename!.linkSource).toBe('wikilink-resolved');
|
||||
});
|
||||
|
||||
test('uppercase path target goes to the generic pass, not exact-path (documented behavior)', async () => {
|
||||
// [[Janus/Foo]] is not a canonical slug. With globalBasename off
|
||||
// (default) it drops silently exactly as before #1493 — it is NOT
|
||||
// recorded as a wikilink miss because it never entered the exact-path
|
||||
// pass. (The #2868 line of work covers slugify-before-match.)
|
||||
const resolver = makeExistsResolver(['janus/foo']);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
'concepts/x', 'See [[Janus/Foo]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates).toEqual([]);
|
||||
expect(unresolved).toEqual([]);
|
||||
});
|
||||
|
||||
test('frontmatter is parsed out before extraction ever sees it (put_page layering)', async () => {
|
||||
// put_page extracts over compiled_truth + timeline, which parseMarkdown
|
||||
// produces AFTER gray-matter removes the frontmatter block. A wikilink
|
||||
// inside frontmatter therefore never reaches the extractor as body text.
|
||||
const raw = [
|
||||
'---',
|
||||
'title: X',
|
||||
'related: "[[janus/from-frontmatter]]"',
|
||||
'---',
|
||||
'',
|
||||
'Body links [[janus/from-body]].',
|
||||
].join('\n');
|
||||
const parsed = parseMarkdown(raw, 'concepts/x.md');
|
||||
expect(parsed.compiled_truth).not.toContain('janus/from-frontmatter');
|
||||
const resolver = makeExistsResolver(['janus/from-body', 'janus/from-frontmatter']);
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', parsed.compiled_truth + '\n' + parsed.timeline,
|
||||
parsed.frontmatter, 'concept', resolver, { skipFrontmatter: true },
|
||||
);
|
||||
expect(candidates.map(c => c.targetSlug)).toEqual(['janus/from-body']);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── makeResolver.slugExists ──────────────────────────────────────
|
||||
|
||||
describe('makeResolver — slugExists (#1493)', () => {
|
||||
function makeFakeEngineWithSlugs(slugs: string[]): BrainEngine {
|
||||
let getAllCalls = 0;
|
||||
const engine = {
|
||||
async getPage() { return null; },
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
async getAllSlugs() {
|
||||
getAllCalls++;
|
||||
return new Set(slugs);
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
(engine as any)._counts = () => ({ getAllCalls });
|
||||
return engine;
|
||||
}
|
||||
|
||||
test('exact match hits, near-miss does not', async () => {
|
||||
const r = makeResolver(makeFakeEngineWithSlugs(['janus/agents/drift-check']));
|
||||
expect(await r.slugExists!('janus/agents/drift-check')).toBe(true);
|
||||
expect(await r.slugExists!('janus/agents/drift')).toBe(false);
|
||||
expect(await r.slugExists!('drift-check')).toBe(false);
|
||||
expect(await r.slugExists!('')).toBe(false);
|
||||
});
|
||||
|
||||
test('shares one getAllSlugs snapshot with resolveBasenameMatches', async () => {
|
||||
const engine = makeFakeEngineWithSlugs(['janus/foo', 'projects/struktura']);
|
||||
const r = makeResolver(engine);
|
||||
await r.slugExists!('janus/foo');
|
||||
await r.resolveBasenameMatches!('struktura');
|
||||
await r.slugExists!('janus/foo');
|
||||
expect((engine as any)._counts().getAllCalls).toBe(1);
|
||||
});
|
||||
|
||||
test('scopes the snapshot by sourceId (no cross-source exact match)', async () => {
|
||||
const bySource: Record<string, string[]> = {
|
||||
'src-a': ['janus/foo'],
|
||||
'src-b': ['janus/bar'],
|
||||
};
|
||||
const engine = {
|
||||
async getPage() { return null; },
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
async getAllSlugs(opts?: { sourceId?: string }) {
|
||||
const sid = opts?.sourceId;
|
||||
return new Set(sid ? (bySource[sid] ?? []) : Object.values(bySource).flat());
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
|
||||
expect(await r.slugExists!('janus/foo')).toBe(true);
|
||||
expect(await r.slugExists!('janus/bar')).toBe(false);
|
||||
});
|
||||
|
||||
test('degrades to false when getAllSlugs is missing', async () => {
|
||||
const engine = {
|
||||
async getPage() { return null; },
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
} as unknown as BrainEngine;
|
||||
const r = makeResolver(engine);
|
||||
expect(await r.slugExists!('janus/foo')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── isAnyDirExactPathEnabled ─────────────────────────────────────
|
||||
|
||||
describe('isAnyDirExactPathEnabled', () => {
|
||||
function makeFakeEngine(configMap: Map<string, string | null>): BrainEngine {
|
||||
return {
|
||||
getConfig: async (key: string) => configMap.get(key) ?? null,
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
test('null/undefined -> true (default ON — exact match cannot false-positive)', async () => {
|
||||
expect(await isAnyDirExactPathEnabled(makeFakeEngine(new Map()))).toBe(true);
|
||||
});
|
||||
|
||||
test('"false"/"0"/"no"/"off" -> false', async () => {
|
||||
for (const v of ['false', '0', 'no', 'off', ' False ']) {
|
||||
const engine = makeFakeEngine(new Map([['link_resolution.any_dir_exact_path', v]]));
|
||||
expect(await isAnyDirExactPathEnabled(engine)).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
test('"true" and garbage -> true (fail-safe to default, like auto_link)', async () => {
|
||||
for (const v of ['true', '1', 'garbage']) {
|
||||
const engine = makeFakeEngine(new Map([['link_resolution.any_dir_exact_path', v]]));
|
||||
expect(await isAnyDirExactPathEnabled(engine)).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('env var override wins over DB config', async () => {
|
||||
const engine = makeFakeEngine(new Map([['link_resolution.any_dir_exact_path', 'true']]));
|
||||
process.env.GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH = 'off';
|
||||
try {
|
||||
expect(await isAnyDirExactPathEnabled(engine)).toBe(false);
|
||||
} finally {
|
||||
delete process.env.GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Issue #1493 codex P1: qualified source pin threading ──────────
|
||||
|
||||
describe('extractPageLinks — qualified wikilink source pin (codex P1)', () => {
|
||||
test('qualified any-dir ref carries targetSourceId on the candidate', async () => {
|
||||
const resolver = makeExistsResolver([]);
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'See [[beta:janus/foo]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('janus/foo');
|
||||
expect(candidates[0].targetSourceId).toBe('beta');
|
||||
});
|
||||
|
||||
test('qualified WHITELIST ref carries targetSourceId identically', async () => {
|
||||
const resolver = makeExistsResolver([]);
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'See [[beta:people/alice]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('people/alice');
|
||||
expect(candidates[0].targetSourceId).toBe('beta');
|
||||
});
|
||||
|
||||
test('unqualified refs carry NO targetSourceId', async () => {
|
||||
const resolver = makeExistsResolver(['janus/foo']);
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'See [[janus/foo]] and [[people/alice]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
for (const c of candidates) {
|
||||
expect(c.targetSourceId).toBeUndefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('pinned + unpinned refs to the same target do NOT dedup-collapse', async () => {
|
||||
// [[beta:janus/foo]] and [[janus/foo]] resolve to different
|
||||
// (to_slug, to_source_id) rows — both candidates must survive.
|
||||
const resolver = makeExistsResolver(['janus/foo']);
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'Pinned [[beta:janus/foo]] and local [[janus/foo]].',
|
||||
{}, 'concept', resolver,
|
||||
);
|
||||
const toFoo = candidates.filter(c => c.targetSlug === 'janus/foo');
|
||||
expect(toFoo.length).toBe(2);
|
||||
expect(new Set(toFoo.map(c => c.targetSourceId))).toEqual(new Set(['beta', undefined]));
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveCandidateSources — honors the qualified source pin (codex P1)', () => {
|
||||
const mk = (over: Partial<LinkCandidate> = {}): LinkCandidate => ({
|
||||
targetSlug: 'janus/foo', linkType: 'mentions', context: 'ctx', linkSource: 'markdown', ...over,
|
||||
});
|
||||
|
||||
test('pinned candidate resolves to the pinned source', () => {
|
||||
const r = resolveCandidateSources(
|
||||
mk({ targetSourceId: 'beta' }), 'concepts/x', 'alpha',
|
||||
new Set(['concepts/x', 'janus/foo']),
|
||||
new Map([['concepts/x', ['alpha']], ['janus/foo', ['beta']]]),
|
||||
);
|
||||
expect(r).not.toBeNull();
|
||||
expect(r!.toSourceId).toBe('beta');
|
||||
expect(r!.fromSourceId).toBe('alpha');
|
||||
});
|
||||
|
||||
test('pin miss returns null even when origin/default HAS a same-slug page (no misdirection)', () => {
|
||||
const r = resolveCandidateSources(
|
||||
mk({ targetSourceId: 'beta' }), 'concepts/x', 'alpha',
|
||||
new Set(['concepts/x', 'janus/foo']),
|
||||
// Target exists in origin source AND default — but NOT in beta.
|
||||
new Map([['concepts/x', ['alpha']], ['janus/foo', ['alpha', 'default']]]),
|
||||
);
|
||||
expect(r).toBeNull();
|
||||
});
|
||||
|
||||
test('unpinned candidate keeps the local-first → default fallback', () => {
|
||||
const local = resolveCandidateSources(
|
||||
mk(), 'concepts/x', 'alpha',
|
||||
new Set(['concepts/x', 'janus/foo']),
|
||||
new Map([['concepts/x', ['alpha']], ['janus/foo', ['alpha']]]),
|
||||
);
|
||||
expect(local!.toSourceId).toBe('alpha');
|
||||
const viaDefault = resolveCandidateSources(
|
||||
mk(), 'concepts/x', 'alpha',
|
||||
new Set(['concepts/x', 'janus/foo']),
|
||||
new Map([['concepts/x', ['alpha']], ['janus/foo', ['default']]]),
|
||||
);
|
||||
expect(viaDefault!.toSourceId).toBe('default');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Issue #1493 codex P1: scoped slugExists includes default fallback ─
|
||||
|
||||
describe('makeResolver — slugExists default-source fallback domain (codex P1)', () => {
|
||||
function makeMultiSourceEngine(bySource: Record<string, string[]>): BrainEngine {
|
||||
const calls: Array<string | undefined> = [];
|
||||
const engine = {
|
||||
async getPage() { return null; },
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
async getAllSlugs(opts?: { sourceId?: string }) {
|
||||
calls.push(opts?.sourceId);
|
||||
const sid = opts?.sourceId;
|
||||
return new Set(sid ? (bySource[sid] ?? []) : Object.values(bySource).flat());
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
(engine as any)._calls = () => calls;
|
||||
return engine;
|
||||
}
|
||||
|
||||
test('scoped resolver sees default-source slugs (mirrors resolveCandidateSources fallback)', async () => {
|
||||
const engine = makeMultiSourceEngine({
|
||||
'src-a': ['janus/local'],
|
||||
'default': ['janus/shared-doc'],
|
||||
'src-b': ['janus/foreign'],
|
||||
});
|
||||
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
|
||||
expect(await r.slugExists!('janus/local')).toBe(true); // scoped source
|
||||
expect(await r.slugExists!('janus/shared-doc')).toBe(true); // default fallback
|
||||
expect(await r.slugExists!('janus/foreign')).toBe(false); // other source: NOT in domain
|
||||
});
|
||||
|
||||
test('basename index stays source-scoped (#972 P1 preserved — no default overlay)', async () => {
|
||||
const engine = makeMultiSourceEngine({
|
||||
'src-a': ['janus/local'],
|
||||
'default': ['projects/shared-doc'],
|
||||
});
|
||||
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
|
||||
// Basename lookup must NOT see the default source's pages.
|
||||
expect(await r.resolveBasenameMatches!('shared-doc')).toEqual([]);
|
||||
expect(await r.resolveBasenameMatches!('local')).toEqual(['janus/local']);
|
||||
});
|
||||
|
||||
test('unscoped and default-scoped resolvers do not double-fetch', async () => {
|
||||
const unscoped = makeMultiSourceEngine({ 'default': ['janus/foo'] });
|
||||
const r1 = makeResolver(unscoped, { mode: 'batch' });
|
||||
await r1.slugExists!('janus/foo');
|
||||
await r1.slugExists!('janus/foo');
|
||||
expect((unscoped as any)._calls()).toEqual([undefined]);
|
||||
|
||||
const defScoped = makeMultiSourceEngine({ 'default': ['janus/foo'] });
|
||||
const r2 = makeResolver(defScoped, { mode: 'batch', sourceId: 'default' });
|
||||
await r2.slugExists!('janus/foo');
|
||||
expect((defScoped as any)._calls()).toEqual(['default']);
|
||||
});
|
||||
|
||||
test('scoped resolver fetches exactly scoped + default snapshots', async () => {
|
||||
const engine = makeMultiSourceEngine({ 'src-a': [], 'default': ['janus/foo'] });
|
||||
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
|
||||
await r.slugExists!('janus/foo');
|
||||
await r.slugExists!('janus/foo');
|
||||
await r.resolveBasenameMatches!('foo');
|
||||
expect((engine as any)._calls().sort()).toEqual(['default', 'src-a']);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Issue #1493 codex round 2: bare-slug pass vs qualified spans ────
|
||||
|
||||
describe('extractPageLinks — bare-slug pass masks qualified spans only (codex P2 round 2)', () => {
|
||||
test('legit prose `Owner:people/alice` still emits a bare-slug candidate', async () => {
|
||||
// Regression: round 1 skipped ANY `:`-preceded bare slug, which also
|
||||
// dropped legitimate prose refs. Only actual qualified-wikilink spans
|
||||
// are masked now.
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'Owner:people/alice runs this.',
|
||||
{}, 'concept', makeExistsResolver([]),
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('people/alice');
|
||||
expect(candidates[0].targetSourceId).toBeUndefined();
|
||||
});
|
||||
|
||||
test('qualified wikilink still emits exactly ONE (pinned) candidate — no bare-slug duplicate', async () => {
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'See [[wiki:people/alice]].',
|
||||
{}, 'concept', makeExistsResolver([]),
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSourceId).toBe('wiki');
|
||||
});
|
||||
|
||||
test('prose colon ref and qualified wikilink coexist correctly', async () => {
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'Owner:people/bob maintains it. See [[wiki:people/alice]].',
|
||||
{}, 'concept', makeExistsResolver([]),
|
||||
);
|
||||
const bob = candidates.filter(c => c.targetSlug === 'people/bob');
|
||||
const alice = candidates.filter(c => c.targetSlug === 'people/alice');
|
||||
expect(bob.length).toBe(1);
|
||||
expect(bob[0].targetSourceId).toBeUndefined();
|
||||
expect(alice.length).toBe(1);
|
||||
expect(alice[0].targetSourceId).toBe('wiki');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,100 @@
|
||||
/**
|
||||
* Issue #1493 (codex P2, round 2) — put_page auto-link must not double-count
|
||||
* (or double-insert) when a page contains BOTH the pinned and unpinned form
|
||||
* of the same wikilink, e.g. `[[default:people/alice]]` and
|
||||
* `[[people/alice]]` on a default-source page. The pin is part of
|
||||
* extractPageLinks' within-page dedup key (correct for the extract paths,
|
||||
* where the two can resolve to different to_source_id rows), but inside
|
||||
* put_page's single-source write scope both resolve to the SAME edge row —
|
||||
* runAutoLink re-dedupes by the resolved edge key before writing/counting.
|
||||
*
|
||||
* Hermetic PGLite; embed transport stubbed (same pattern as
|
||||
* test/put-page-provenance.test.ts).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, beforeEach, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { operations } from '../src/core/operations.ts';
|
||||
import type { OperationContext } from '../src/core/operations.ts';
|
||||
import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';
|
||||
|
||||
const putPageOp = operations.find((o) => o.name === 'put_page')!;
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { ...process.env, OPENAI_API_KEY: process.env.OPENAI_API_KEY || 'sk-test-stub' },
|
||||
});
|
||||
__setEmbedTransportForTests(async ({ values }: any) => ({
|
||||
embeddings: values.map(() => new Array(1536).fill(0)),
|
||||
usage: { tokens: 0 },
|
||||
}) as any);
|
||||
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
for (const t of ['content_chunks', 'links', 'tags', 'timeline_entries', 'page_versions', 'pages']) {
|
||||
await engine.executeRaw(`DELETE FROM ${t}`, []);
|
||||
}
|
||||
});
|
||||
|
||||
function makeCtx(): OperationContext {
|
||||
return {
|
||||
engine,
|
||||
config: { engine: 'pglite' as const },
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
sourceId: 'default',
|
||||
};
|
||||
}
|
||||
|
||||
describe('put_page auto-link — pinned + unpinned same-target dedup (#1493)', () => {
|
||||
test('own-source pin + unpinned form create ONE edge and count created=1', async () => {
|
||||
const ctx = makeCtx();
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'people/alice',
|
||||
content: '---\ntype: person\ntitle: Alice\n---\n\nAlice.',
|
||||
});
|
||||
|
||||
const res = await putPageOp.handler(ctx, {
|
||||
slug: 'concepts/note',
|
||||
content: '---\ntype: concept\ntitle: Note\n---\n\nSee [[default:people/alice]] and [[people/alice]].',
|
||||
}) as { auto_links?: { created: number; errors: number } };
|
||||
|
||||
const links = (await engine.getLinks('concepts/note')).filter(l => l.to_slug === 'people/alice');
|
||||
expect(links.length).toBe(1);
|
||||
expect(res.auto_links).toBeDefined();
|
||||
expect(res.auto_links!.created).toBe(1);
|
||||
expect(res.auto_links!.errors).toBe(0);
|
||||
});
|
||||
|
||||
test('foreign-pinned form is skipped, unpinned form still links (no misdirection)', async () => {
|
||||
const ctx = makeCtx();
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'people/alice',
|
||||
content: '---\ntype: person\ntitle: Alice\n---\n\nAlice.',
|
||||
});
|
||||
|
||||
const res = await putPageOp.handler(ctx, {
|
||||
slug: 'concepts/note',
|
||||
content: '---\ntype: concept\ntitle: Note\n---\n\nSee [[somewhere-else:people/alice]] and [[people/alice]].',
|
||||
}) as { auto_links?: { created: number } };
|
||||
|
||||
const links = (await engine.getLinks('concepts/note')).filter(l => l.to_slug === 'people/alice');
|
||||
expect(links.length).toBe(1);
|
||||
expect(res.auto_links!.created).toBe(1);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user