diff --git a/CHANGELOG.md b/CHANGELOG.md index 8aa3c42a5..4c1ba6748 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,70 @@ All notable changes to GBrain will be documented in this file. +## [0.42.10.0] - 2026-06-02 + +**Wikilinks like `[[struktura]]` that point at pages in another folder finally connect.** Until now, if you wrote `[[struktura]]` in `concepts/knowledge-graph.md` and the actual page lived at `projects/struktura.md`, GBrain silently dropped the link from its graph. Obsidian users saw a dense web of connections in their vault and a thin, broken graph inside GBrain. The issue reporter had 71 wikilinks across 20 pages — GBrain captured 12. + +This release adds an **opt-in mode** that resolves bare wikilinks by basename match. Turn it on with one command and the missing edges show up. + +``` +gbrain config set link_resolution.global_basename true +gbrain extract links +``` + +`gbrain doctor` scans your brain on every run and tells you what you'd gain. If five or more bare wikilinks would resolve under the new mode, you'll see a paste-ready hint: + +``` +link_resolution_opportunity: 47 of 60 bare wikilinks (78%) would resolve to +24 distinct page(s) under global_basename mode. +Enable with: gbrain config set link_resolution.global_basename true +``` + +The mode covers every place wikilinks get resolved: `gbrain extract links` (both the filesystem-walk path that the issue reporter hit by default, and the database path that the autopilot cycle uses), and the auto-link step that runs every time an agent writes a page through `put_page`. Same flag, all three surfaces. + +### How it handles ambiguity + +If `[[struktura]]` matches two pages — say `projects/struktura.md` and `archive/struktura.md` — GBrain emits **one edge to each**, both tagged with a new `wikilink_basename` link type. No silent winner-takes-all, no arbitrary first-wins choice. You can audit and prune the dual-match cases via: + +``` +gbrain graph-query concepts/knowledge-graph --type wikilink_basename +``` + +This is a deliberate departure from the issue's original proposal (which suggested dropping ambiguous matches with a warning). The graph layer is supposed to capture relationships, not editorialize about them. + +### Why opt-in (and why this default is safe) + +The default stays off, so existing brains get zero behavior change on upgrade. The bug class this fix opens — a `[[review]]` accidentally linking to some `topics/review` page that exists for unrelated reasons — is real, and turning it on globally would change every existing user's graph in ways they didn't ask for. Doctor surfaces the opportunity; the user decides. + +For brains that flip the flag on, all newly extracted edges carry a distinct `link_source = 'wikilink-resolved'` provenance tag, so you can later distinguish basename-resolved edges from your hand-written markdown links. + +### What this means + +If you write in the Obsidian convention (flat-namespace, globally-unique basenames, bare `[[wikilinks]]` everywhere), your GBrain graph has been wrong. Flip the flag and run extract once — the missing edges show up immediately. + +If you don't use bare wikilinks, doctor will tell you so (`No bare wikilinks found`) and you can keep ignoring this knob. + +Closes https://github.com/garrytan/gbrain/issues/972. + +### Itemized changes + +- `src/core/link-extraction.ts` — three new exports: + - `WIKILINK_BASENAME_LINK_TYPE = 'wikilink_basename'` — the edge-type constant used by every path that emits basename-resolved edges. Adapted from PR #1233's regex (which was the kernel of the resolver-side fix). + - `isGlobalBasenameEnabled(engine)` — single source of truth for the + flag read. Order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB + config `link_resolution.global_basename` → default false. + - `WIKILINK_GENERIC_RE` — generic Obsidian-shape wikilink regex for the `[[bare-name]]` pass that falls outside `DIR_PATTERN`. `EntityRef` gains an optional `needsResolution` flag tagged onto refs from this pass so downstream callers know to route them through the resolver instead of treating them as canonical slugs. +- `SlugResolver` interface gains an optional `resolveBasenameMatches(name): Promise` method. Returns ALL slugs whose basename matches — multi-match by design. `makeResolver(engine)` implements this with a lazy-built basename → `string[]` index (single `engine.getAllSlugs()` call per resolver instance) covering raw + lowercase + slugified key variants. +- `extractPageLinks` gains two options. `opts.globalBasename` routes `needsResolution: true` refs through `resolveBasenameMatches`, emitting one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`. `opts.skipFrontmatter` replaces the pre-issue-#972 `nullResolver` pattern — callers can now pass the real resolver everywhere AND independently gate the frontmatter pass, which is necessary because the bare-wikilink path needs `resolveBasenameMatches` on the real resolver. +- `src/commands/extract.ts` — DB-source `extractLinksFromDB` and FS-source `extractLinksFromFile` / `extractLinksFromDir` / `extractLinksForSlugs` thread `globalBasename` (read once per extract run via `isGlobalBasenameEnabled`). FS-source adds two new exported helpers — `resolveBasenameMatchesFromSlugs(name, allSlugs): string[]` and `resolveSlugAll(fileDir, relTarget, allSlugs, opts): string[]`. `resolveSlug` stays single-match for back-compat; `resolveSlugAll` wraps it with the basename-fallback. Stable sort: shorter slug first, then lexical. +- `src/core/operations.ts:put_page` — auto-link path now reads the flag and passes `globalBasename` through to `extractPageLinks`. Newly-written pages get basename edges when the flag is on. +- `src/commands/doctor.ts` — new `link_resolution_opportunity` check. Full scan with a 60s budget, heartbeat-reported. Severity: skipped when flag already on; ok when no bare wikilinks; ok with explanation when bare wikilinks present but none match; **warn with paste-ready hint when ≥5 would resolve AND ≥20% of bare wikilinks have matches**. Threshold tuned to surface real opportunities without flagging brains that only have a handful of unresolvable refs. Wired into both `buildChecks` (local doctor) and `doctorReportRemote` (thin-client) for parity. +- Migration v113 (`links_link_source_widen_for_wikilink_basename`) widens the `links_link_source_check` constraint to accept `'wikilink-resolved'` as a valid `link_source`. Idempotent — DROP IF EXISTS + ADD CONSTRAINT. +- `src/schema.sql` + regenerated `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` carry the widened constraint for fresh installs so first-init brains start with the right shape. +- New CHECK constraint value adds `'wikilink-resolved'` alongside `'markdown'`, `'frontmatter'`, `'manual'`. +- `KNOWN_CONFIG_KEYS` (in `src/core/config.ts`) adds `'link_resolution'` and `'link_resolution.global_basename'` so `gbrain config set ...` accepts the new key without `--force`. +- Tests: 38 new cases pinning the contract. `test/link-extraction.test.ts` adds 17 cases covering `WIKILINK_GENERIC_RE` shape (anchor / display / strip / escape paths), the `extractEntityRefs` pass-2c no-double-emit invariant, `resolveBasenameMatches` multi-match + index-built-once + missing-`getAllSlugs` degradation, and the `extractPageLinks` opt routing under both flag states. `test/extract-fs.test.ts` adds 11 cases for the pure-function helpers (`resolveBasenameMatchesFromSlugs`, `resolveSlugAll`) plus 3 round-trip tests of the issue's exact repro inside a PGLite brain. `test/doctor.test.ts` adds 7 cases for the new doctor check (skip / ok / warn paths + the cross-surface wiring source-grep). `test/e2e/global-basename-pglite.test.ts` adds 7 end-to-end cases against an in-memory PGLite brain covering FS-source, DB-source, and put_page auto-link paths under both flag states. +- PR #1233 from @rayers contributed the kernel of the resolver-side approach (the generic wikilink regex + slug-tail index pattern). This PR keeps that mechanism, makes it opt-in via the new config flag, replaces the first-write-wins lookup with multi-match return, and extends the coverage to the FS-source path that the issue's repro actually hits. ## [0.42.8.0] - 2026-06-01 **Scraped junk stops landing in your brain as if it were real content, and when something looks off, your agent gets told instead of being left to guess.** @@ -1535,6 +1599,7 @@ behavior for an automated caller, set `gbrain config set search.mcp_keyword_only - A page represented by its weakest chunk in vector search (the incident). - Same-slug pages in different sources collapsing in per-page pooling and the alias hop (source-isolation). + ## [0.41.33.0] - 2026-05-29 **`gbrain search` and `query` can now return a tight, intent-sized set of @@ -1775,6 +1840,7 @@ doctor` warns about a partial migration: `test/doctor.test.ts` (T1 untracked-folders headline bug, T2 remote-never-shells-out trust boundary), `test/sync-all-parallel.test.ts` (column-path staleness). - **CI tooling** — `scripts/ship-remote-tests.sh` + `workflow_dispatch` on `test.yml`. + ## [0.41.31.0] - 2026-05-30 **Your nightly `gbrain sync --all` cron stops getting blocked. It used to diff --git a/CLAUDE.md b/CLAUDE.md index 213ec1227..7de9b210f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -221,7 +221,8 @@ strict behavior when unset. - `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit ` + `git rm` source-of-truth, oversize → split or accept). - `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines). - `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`. -- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. +- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **v0.42.10.0 (issue #972):** opt-in global-basename wikilink resolution. New `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in extractEntityRefs); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise`. The ONE shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical) — used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — **source-scoped** so a bare `[[name]]` never resolves to a same-tail page in a DIFFERENT source (no cross-source edges; #972 is across-folders, not cross-source federation). `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug` — the TARGET, not the display alias — emits one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the pre-#972 `nullResolver` ternary). All three surfaces tag provenance: FS edges carry `link_source='wikilink-resolved'` via the new `ExtractedLink.link_source` field; `put_page` auto-link includes `wikilink-resolved` in its reconcilable-edge set so stale basename edges are removed when the wikilink (or the flag) goes away. A wikilink inside a markdown-link label (`[see [[acme]]](path)`) is masked out of pass 2c; the FS path strips code fences before scanning. New exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). Multi-match by design. The mode is opt-in (default false); `gbrain doctor`'s `link_resolution_opportunity` check (bounded scan: 1000 most-recent pages in one query, not a per-page walk) surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% ratio. Migration v113 widens the `links_link_source_check` constraint to admit the `'wikilink-resolved'` provenance value (full union with master's `'mentions'`). Pinned by `test/link-extraction.test.ts` + `test/extract-fs.test.ts` + `test/doctor.test.ts` + `test/e2e/global-basename-pglite.test.ts` (incl. the issue's exact `mkdir -p /tmp/vault/{concepts,projects}` repro, alias-target resolution, cross-source non-leak, put_page stale-edge removal). PR #1233 from @rayers contributed the regex shape + slug-tail index pattern; PR #1388 from @ukd1 (Russell Smith) made it opt-in + multi-match + FS coverage; the v0.42.10.0 wave added the codex source-correctness fixes (alias-by-target, stale-edge reconciliation, source-scoping) + the shared matcher. +- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. - `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`. - `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`. - `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine ` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`. diff --git a/INSTALL_FOR_AGENTS.md b/INSTALL_FOR_AGENTS.md index 4974a0eb6..421d2ff09 100644 --- a/INSTALL_FOR_AGENTS.md +++ b/INSTALL_FOR_AGENTS.md @@ -161,6 +161,29 @@ After this step: If a user has a very large brain (>10K pages), `extract --source db` is idempotent and supports `--since YYYY-MM-DD` for incremental runs. +### Obsidian-style bare wikilinks (opt-in) + +If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]` +wikilinks — where `[[struktura]]` written in one folder means the page that lives +at `projects/struktura.md` in another — GBrain does NOT connect those by default. +Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`, +so a vault full of bare links shows up as a thin, broken graph. Turn on basename +resolution so the cross-folder links connect: + +```bash +gbrain config set link_resolution.global_basename true +gbrain extract links --source db # re-run so the new edges land +``` + +`gbrain doctor` surfaces a `link_resolution_opportunity` hint with the exact count +("47 of 60 bare wikilinks would resolve") so you know whether it's worth enabling +before you flip it. When a bare name matches more than one page (`[[struktura]]` → +both `projects/struktura` and `archive/struktura`), GBrain emits one edge to each +rather than guessing a winner — review and prune the duplicates with +`gbrain graph-query `. The mode is also honored on the filesystem-walk path +(`gbrain extract links` with no `--source db`) and by auto-link on every future +`put_page`. + ## Step 5: Load Skills If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace), diff --git a/README.md b/README.md index 91d66798a..c3ea8086c 100644 --- a/README.md +++ b/README.md @@ -254,7 +254,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. -**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. +**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. diff --git a/VERSION b/VERSION index 7fb2c43bc..19f6e33f9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.8.0 \ No newline at end of file +0.42.10.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 19dfcb4b4..5a2c087c2 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -363,7 +363,8 @@ strict behavior when unset. - `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit ` + `git rm` source-of-truth, oversize → split or accept). - `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines). - `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`. -- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. +- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **v0.42.10.0 (issue #972):** opt-in global-basename wikilink resolution. New `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in extractEntityRefs); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise`. The ONE shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical) — used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — **source-scoped** so a bare `[[name]]` never resolves to a same-tail page in a DIFFERENT source (no cross-source edges; #972 is across-folders, not cross-source federation). `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug` — the TARGET, not the display alias — emits one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the pre-#972 `nullResolver` ternary). All three surfaces tag provenance: FS edges carry `link_source='wikilink-resolved'` via the new `ExtractedLink.link_source` field; `put_page` auto-link includes `wikilink-resolved` in its reconcilable-edge set so stale basename edges are removed when the wikilink (or the flag) goes away. A wikilink inside a markdown-link label (`[see [[acme]]](path)`) is masked out of pass 2c; the FS path strips code fences before scanning. New exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). Multi-match by design. The mode is opt-in (default false); `gbrain doctor`'s `link_resolution_opportunity` check (bounded scan: 1000 most-recent pages in one query, not a per-page walk) surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% ratio. Migration v113 widens the `links_link_source_check` constraint to admit the `'wikilink-resolved'` provenance value (full union with master's `'mentions'`). Pinned by `test/link-extraction.test.ts` + `test/extract-fs.test.ts` + `test/doctor.test.ts` + `test/e2e/global-basename-pglite.test.ts` (incl. the issue's exact `mkdir -p /tmp/vault/{concepts,projects}` repro, alias-target resolution, cross-source non-leak, put_page stale-edge removal). PR #1233 from @rayers contributed the regex shape + slug-tail index pattern; PR #1388 from @ukd1 (Russell Smith) made it opt-in + multi-match + FS coverage; the v0.42.10.0 wave added the codex source-correctness fixes (alias-by-target, stale-edge reconciliation, source-scoping) + the shared matcher. +- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. - `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`. - `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`. - `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine ` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`. @@ -2396,6 +2397,29 @@ After this step: If a user has a very large brain (>10K pages), `extract --source db` is idempotent and supports `--since YYYY-MM-DD` for incremental runs. +### Obsidian-style bare wikilinks (opt-in) + +If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]` +wikilinks — where `[[struktura]]` written in one folder means the page that lives +at `projects/struktura.md` in another — GBrain does NOT connect those by default. +Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`, +so a vault full of bare links shows up as a thin, broken graph. Turn on basename +resolution so the cross-folder links connect: + +```bash +gbrain config set link_resolution.global_basename true +gbrain extract links --source db # re-run so the new edges land +``` + +`gbrain doctor` surfaces a `link_resolution_opportunity` hint with the exact count +("47 of 60 bare wikilinks would resolve") so you know whether it's worth enabling +before you flip it. When a bare name matches more than one page (`[[struktura]]` → +both `projects/struktura` and `archive/struktura`), GBrain emits one edge to each +rather than guessing a winner — review and prune the duplicates with +`gbrain graph-query `. The mode is also honored on the filesystem-walk path +(`gbrain extract links` with no `--source db`) and by auto-link on every future +`put_page`. + ## Step 5: Load Skills If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace), @@ -2967,7 +2991,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. -**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. +**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. diff --git a/package.json b/package.json index d3a449b94..6abf53c75 100644 --- a/package.json +++ b/package.json @@ -142,5 +142,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.8.0" + "version": "0.42.10.0" } diff --git a/scripts/llms-config.ts b/scripts/llms-config.ts index ee718679e..ba676dc02 100644 --- a/scripts/llms-config.ts +++ b/scripts/llms-config.ts @@ -255,12 +255,13 @@ export const INLINE_TIPS = [ "`gbrain upgrade` runs post-upgrade + apply-migrations.", ]; -// Target ~750KB so llms-full.txt fits in ~190k-token contexts with room to spare. -// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB once CLAUDE.md crossed 700KB: -// it's ~540KB (77% of the bundle) and grows ~5-15KB per release with each feature's -// Key Files annotation. Both master (v0.41.34-38 waves) and this branch (skillopt -// wave) independently hit the 700KB line and bumped to the same 750KB. CLAUDE.md is -// the whole point of the one-fetch bundle, so it stays inlined; the budget tracks -// its legitimate growth. Still fits comfortably in 200k+ context models. +// Target ~800KB so llms-full.txt fits in ~200k-token contexts with room to spare. +// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB once CLAUDE.md crossed 700KB, +// then 750KB→800KB in v0.42.10.0 when the #972 global-basename Key Files annotation +// (landing alongside master's #1696/#1699 waves) crossed the 750KB line. CLAUDE.md +// is ~540KB+ (the bulk of the bundle) and grows ~5-15KB per release with each +// feature's Key Files annotation. CLAUDE.md is the whole point of the one-fetch +// bundle, so it stays inlined; the budget tracks its legitimate growth. Still fits +// comfortably in 200k+ context models. // Generator prints a WARN if exceeded; ship with includeInFull=false exclusions. -export const FULL_SIZE_BUDGET = 750_000; +export const FULL_SIZE_BUDGET = 800_000; diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 633c360c6..6441ab3d3 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -27,6 +27,12 @@ import { gbrainPath } from '../core/config.ts'; import { dirname, isAbsolute, join, resolve as resolvePath } from 'path'; import { fileURLToPath } from 'url'; import { existsSync, readFileSync, readdirSync, statSync } from 'fs'; +import { + extractEntityRefs, + isGlobalBasenameEnabled, + buildBasenameIndex, + queryBasenameIndex, +} from '../core/link-extraction.ts'; import { isSourceUnchangedSinceSync } from '../core/git-head.ts'; // v0.41.32.0: remote staleness reads the stored newest_content_at column via // this pure comparator (no git subprocess on the HTTP MCP doctor path). @@ -728,6 +734,11 @@ export async function doctorReportRemote(engine: BrainEngine): Promise=5 would-resolve AND >=20% of bare + * wikilinks have matches). Skipped silently when the flag is already + * enabled (no signal to surface) or the brain is empty. + * + * Bounded scan: batch-loads the 1000 most-recent pages in one query (not a + * per-page getPage walk) with a 60s backstop. On DB error, downgrades to an + * informational `ok` so doctor never blocks on this check. + */ +export async function checkLinkResolutionOpportunity( + engine: BrainEngine, + progress?: ProgressReporter, +): Promise { + const name = 'link_resolution_opportunity'; + try { + if (await isGlobalBasenameEnabled(engine)) { + return { name, status: 'ok', message: 'global_basename mode already enabled' }; + } + const allSlugs = await engine.getAllSlugs(); + if (allSlugs.size === 0) { + return { name, status: 'ok', message: 'Brain is empty — nothing to scan' }; + } + // Build a basename → slug[] index ONCE for the entire scan via the shared + // builder (issue #972 codex [P2] DRY) — same key set (raw/lower/slugified) + // as extraction, so this estimate matches what extraction actually + // resolves. Pre-fix the doctor omitted the slugified key and undercounted. + const basenameIndex = buildBasenameIndex(allSlugs); + + let bareCount = 0; + let wouldResolveCount = 0; + const distinctTargets = new Set(); + + // Issue #972 (codex [P2] perf): batch-load the most-recent N pages in ONE + // query instead of listAllPageRefs() + a getPage() per page. The prior + // full N-page walk hit the 60s budget every run on large brains and + // returned a perpetual partial; this bounds the work to a fixed sample. + const SAMPLE_LIMIT = 1000; + const sampled = await engine.executeRaw<{ compiled_truth: string | null; timeline: string | null }>( + `SELECT compiled_truth, timeline FROM pages WHERE deleted_at IS NULL ORDER BY id DESC LIMIT ${SAMPLE_LIMIT}`, + ); + const totalPages = allSlugs.size; + const sampledNote = totalPages > SAMPLE_LIMIT + ? ` (scanned the ${SAMPLE_LIMIT} most-recent of ${totalPages} pages)` + : ''; + const deadline = Date.now() + 60_000; + const hb = progress ? startHeartbeat(progress, `scanning ${sampled.length} pages for bare wikilinks…`) : null; + try { + for (const row of sampled) { + if (Date.now() > deadline) break; // backstop; in-memory scan rarely hits it + const content = (row.compiled_truth ?? '') + '\n' + (row.timeline ?? ''); + for (const e of extractEntityRefs(content)) { + if (!e.needsResolution) continue; + bareCount++; + // Issue #972 (codex): match on the wikilink TARGET (e.slug), not + // the display alias (e.name), via the shared query so the doctor + // estimate equals what extraction actually resolves. + const matches = queryBasenameIndex(basenameIndex, e.slug); + if (matches.length > 0) { + wouldResolveCount++; + for (const m of matches) distinctTargets.add(m); + } + } + } + } finally { + hb?.(); + } + + if (bareCount === 0) { + return { name, status: 'ok', message: 'No bare wikilinks found' }; + } + if (wouldResolveCount === 0) { + return { + name, + status: 'ok', + message: `${bareCount} bare wikilink(s) found, but none have basename matches in the brain.`, + }; + } + const ratio = wouldResolveCount / bareCount; + if (wouldResolveCount >= 5 && ratio >= 0.20) { + const pct = Math.round(ratio * 100); + return { + name, + status: 'warn', + message: + `${wouldResolveCount} of ${bareCount} bare wikilinks (${pct}%) would resolve to ` + + `${distinctTargets.size} distinct page(s) under global_basename mode${sampledNote}. ` + + `Enable with: gbrain config set link_resolution.global_basename true`, + }; + } + const pct = Math.round(ratio * 100); + return { + name, + status: 'ok', + message: `${wouldResolveCount}/${bareCount} bare wikilinks (${pct}%) would resolve — below the 20% / 5-link threshold for surfacing a hint${sampledNote}.`, + }; + } catch (e) { + return { + name, + status: 'ok', + message: `Skipped (${e instanceof Error ? e.message : String(e)})`, + }; + } +} + export async function checkAbandonedThreads(engine: BrainEngine): Promise { try { const rows = await engine.executeRaw<{ count: number }>( @@ -6166,6 +6287,13 @@ export async function buildChecks( // v0.37.0 brainstorm_health — migration v79, track_retrieval, calibration cold-start. progress.heartbeat('brainstorm_health'); checks.push(await checkBrainstormHealth(engine)); + // issue #972 link_resolution_opportunity — full scan: count bare wikilinks + // that would resolve under global_basename mode. Surfaces a paste-ready + // enable hint when ≥5 hits AND ≥20% of bare wikilinks would resolve. + // Skipped silently when the flag is already enabled. Bounded by a 60s + // budget so a huge brain never wedges doctor on this check. + progress.heartbeat('link_resolution_opportunity'); + checks.push(await checkLinkResolutionOpportunity(engine, progress)); // v0.36.0.0 (A5): ZE embedding key health + schema/config width consistency. progress.heartbeat('ze_embedding_health'); checks.push(await checkZeEmbeddingHealth(engine)); diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 0f157c44d..04ab7e3d4 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -35,7 +35,9 @@ import type { PageType } from '../core/types.ts'; import { parseMarkdown } from '../core/markdown.ts'; import { extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver, - extractFrontmatterLinks, LINK_EXTRACTOR_VERSION_TS, + extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS, + WIKILINK_BASENAME_LINK_TYPE, + buildBasenameIndex, queryBasenameIndex, stripCodeBlocks, type UnresolvedFrontmatterRef, type LinkCandidate, } from '../core/link-extraction.ts'; import { createProgress } from '../core/progress.ts'; @@ -156,6 +158,11 @@ export interface ExtractedLink { to_slug: string; link_type: string; context: string; + // Issue #972: provenance for FS-source edges. Set to 'wikilink-resolved' + // on basename-matched bare wikilinks so the FS path tags them the same way + // the DB / put_page paths do. Undefined for ordinary markdown edges (the + // engine defaults those to 'markdown'). + link_source?: string; } export interface ExtractedTimelineEntry { @@ -273,6 +280,56 @@ export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set, +): string[] { + // Issue #972 (codex [P2] DRY): delegate to the shared matcher so the FS + // path keys + sorts identically to the resolver and doctor. (Per-call + // index build is O(N), the same cost as the prior inline scan.) + return queryBasenameIndex(buildBasenameIndex(allSlugs), name); +} + +/** + * Issue #972: multi-match variant of `resolveSlug`. Always tries the + * existing ancestor walk first (preserving the v0.10.1 behavior); on + * miss, falls back to basename lookup against `allSlugs` when + * `opts.globalBasename === true`. Returns an array so the caller emits + * one graph edge per matching page. + * + * Return shape: + * - Ancestor walk hits → `[ancestor_match]` (length 1) + * - Ancestor walk misses + globalBasename off → `[]` + * - Ancestor walk misses + globalBasename on + basename hits → all matches + * - Ancestor walk misses + globalBasename on + no basename hits → `[]` + */ +export function resolveSlugAll( + fileDir: string, relTarget: string, allSlugs: Set, + opts: { globalBasename?: boolean } = {}, +): string[] { + const direct = resolveSlug(fileDir, relTarget, allSlugs); + if (direct !== null) return [direct]; + if (!opts.globalBasename) return []; + // Strip .md suffix + dirname so `[[struktura]]` (relTarget=`struktura.md`) + // and `[[notes/struktura]]` (relTarget=`notes/struktura.md`) both query + // for the basename `struktura`. + const targetNoExt = relTarget.endsWith('.md') ? relTarget.slice(0, -3) : relTarget; + const basename = targetNoExt.includes('/') + ? targetNoExt.slice(targetNoExt.lastIndexOf('/') + 1) + : targetNoExt; + return resolveBasenameMatchesFromSlugs(basename, allSlugs); +} + /** * Directory-based link-type inference for the fs-source path. * @@ -319,20 +376,49 @@ function parseFrontmatterFromContent(content: string, relPath: string): Record, - opts?: { includeFrontmatter?: boolean }, + opts?: { includeFrontmatter?: boolean; globalBasename?: boolean }, ): Promise { const links: ExtractedLink[] = []; const slug = pathToSlug(relPath); const fileDir = dirname(relPath); const fm = parseFrontmatterFromContent(content, relPath); + // Issue #972: globalBasename routes bare `[[name]]` wikilinks through + // basename lookup against allSlugs when the ancestor walk fails. Off + // by default for back-compat with the v0.10.1 ancestor-only behavior. + const globalBasename = opts?.globalBasename ?? false; - for (const { name, relTarget } of extractMarkdownLinks(content)) { - const resolved = resolveSlug(fileDir, relTarget, allSlugs); - if (resolved !== null) { + // Issue #972 (codex [P2]): strip code fences before scanning so a + // `[[name]]` inside a code block doesn't create an FS edge. Mirrors the + // DB path, which goes through extractEntityRefs (which strips internally). + const scanContent = stripCodeBlocks(content); + + for (const { name, relTarget } of extractMarkdownLinks(scanContent)) { + const resolvedSlugs = resolveSlugAll(fileDir, relTarget, allSlugs, { globalBasename }); + if (resolvedSlugs.length === 0) continue; + // Single hit on the ancestor path → emit one edge with the inferred + // verb type. Multiple hits (only possible when globalBasename is on + // AND ancestor walk missed) → emit one edge per match, all tagged + // `wikilink_basename` so users can audit via `gbrain graph-query + // --type wikilink_basename`. + const isBasename = resolvedSlugs.length > 1 + || (globalBasename && resolvedSlugs.length === 1 + && resolveSlug(fileDir, relTarget, allSlugs) === null); + for (const target of resolvedSlugs) { + // Issue #972 (codex [P2]): drop a basename self-loop ([[own-tail]] on + // its own page resolving back to itself). + if (isBasename && target === slug) continue; links.push({ - from_slug: slug, to_slug: resolved, - link_type: inferTypeByDir(fileDir, dirname(resolved), fm), - context: `markdown link: [${name}]`, + from_slug: slug, + to_slug: target, + link_type: isBasename + ? WIKILINK_BASENAME_LINK_TYPE + : inferTypeByDir(fileDir, dirname(target), fm), + context: isBasename + ? `wikilink (basename match): [${name}]` + : `markdown link: [${name}]`, + // Issue #972: tag basename edges so the FS path matches DB/put_page + // provenance and migration v112's widened CHECK is exercised here too. + link_source: isBasename ? 'wikilink-resolved' : undefined, }); } } @@ -860,6 +946,9 @@ async function extractForSlugs( let timelineCreated = 0; let pagesProcessed = 0; + // Issue #972: read the basename flag once per extract run. + const globalBasename = await isGlobalBasenameEnabled(engine); + const linkBatch: LinkBatchInput[] = []; const timelineBatch: TimelineBatchInput[] = []; @@ -912,7 +1001,7 @@ async function extractForSlugs( const content = readFileSync(fullPath, 'utf-8'); if (doLinks) { - const links = await extractLinksFromFile(content, relPath, allSlugs); + const links = await extractLinksFromFile(content, relPath, allSlugs, { globalBasename }); for (const link of links) { if (dryRun) { if (!jsonMode) console.log(` ${link.from_slug} → ${link.to_slug} (${link.link_type})`); @@ -963,6 +1052,11 @@ async function extractLinksFromDir( const files = walkMarkdownFiles(brainDir); const allSlugs = new Set(files.map(f => pathToSlug(f.relPath))); + // Issue #972: read once before the walk so the per-file calls don't + // re-query the DB. globalBasename = true emits one edge per basename + // match for bare wikilinks like `[[struktura]]`. + const globalBasename = await isGlobalBasenameEnabled(engine); + // Progress stream on stderr (separate from the action-events --json writes // to stdout, which tests grep for). Rate-gated; respects global --quiet / // --progress-json flags. @@ -998,7 +1092,7 @@ async function extractLinksFromDir( onItem: async (file) => { try { const content = readFileSync(file.path, 'utf-8'); - const links = await extractLinksFromFile(content, file.relPath, allSlugs); + const links = await extractLinksFromFile(content, file.relPath, allSlugs, { globalBasename }); for (const link of links) { if (dryRunSeen) { const key = `${link.from_slug}::${link.to_slug}::${link.link_type}`; @@ -1107,14 +1201,16 @@ export async function extractLinksForSlugs( const linkOpts = opts?.sourceId ? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId, originSourceId: opts.sourceId } : undefined; + // Issue #972: same flag as the standalone extract path. + const globalBasename = await isGlobalBasenameEnabled(engine); let created = 0; for (const slug of slugs) { const filePath = join(repoPath, slug + '.md'); if (!existsSync(filePath)) continue; try { const content = readFileSync(filePath, 'utf-8'); - for (const link of await extractLinksFromFile(content, slug + '.md', allSlugs)) { - try { await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type, undefined, undefined, undefined, linkOpts); created++; } catch { /* skip */ } // gbrain-allow-direct-insert: gbrain extract single-row fallback when batch path declines a row + for (const link of await extractLinksFromFile(content, slug + '.md', allSlugs, { globalBasename })) { + try { await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type, link.link_source, undefined, undefined, linkOpts); created++; } catch { /* skip */ } // gbrain-allow-direct-insert: gbrain extract single-row fallback when batch path declines a row } } catch { /* skip */ } } @@ -1170,12 +1266,19 @@ async function extractLinksFromDB( // Batch resolver: pg_trgm + exact only, NO search fallback. Dodges the // N-thousand API call trap on 46K-page brains. Resolver has a per-run // cache so duplicate names (same person appearing on many pages) resolve - // once, not once per mention. - const resolver = makeResolver(engine, { mode: 'batch' }); + // once, not once per mention. Used for BOTH the frontmatter pass (gated + // by `includeFrontmatter` via `opts.skipFrontmatter` on extractPageLinks) + // AND the issue-#972 global-basename pass (gated by `globalBasename`). + // Replaces the pre-issue-#972 `nullResolver` ternary — that synthetic + // resolver lacked `resolveBasenameMatches`, so we always pass the real + // one and let extractPageLinks's opts gate which pass actually runs. + // Issue #972 (codex [P1]): scope basename resolution to the source being + // extracted so bare wikilinks don't resolve across unrelated sources. + const resolver = makeResolver(engine, { mode: 'batch', sourceId: sourceIdFilter }); const unresolved: UnresolvedFrontmatterRef[] = []; - const nullResolver = { - resolve: async () => null as string | null, - }; + // Issue #972: opt-in global-basename wikilink resolution. Read once + // per extract run; threaded into each extractPageLinks call. + const globalBasename = await isGlobalBasenameEnabled(engine); // v0.32.8: listAllPageRefs enumerates (slug, source_id) so we can thread // sourceId to getPage AND build a cross-source resolution map for link // disambiguation. Pre-fix used getAllSlugs() which collapsed @@ -1252,9 +1355,11 @@ async function extractLinksFromDB( // --include-frontmatter default OFF in v0.13 (codex tension 5, back-compat). // Migration orchestrator explicitly enables it for the one-time backfill; // user-invoked `gbrain extract links` stays outgoing-only. - const activeResolver = includeFrontmatter ? resolver : nullResolver; + // Issue #972: globalBasename routes bare `[[name]]` wikilinks through + // basename lookup; off by default for back-compat. const extracted = await extractPageLinks( - slug, fullContent, page.frontmatter, page.type, activeResolver, + slug, fullContent, page.frontmatter, page.type, resolver, + { skipFrontmatter: !includeFrontmatter, globalBasename }, ); unresolved.push(...extracted.unresolved); diff --git a/src/core/config.ts b/src/core/config.ts index bebc40b35..8e097cdda 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -736,6 +736,9 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ // Misc 'artifacts_sync_mode', 'cross_project_learnings', + // Link resolution (issue #972) + 'link_resolution', + 'link_resolution.global_basename', ]; /** diff --git a/src/core/doctor-categories.ts b/src/core/doctor-categories.ts index 999fdc8a4..81dfedd5b 100644 --- a/src/core/doctor-categories.ts +++ b/src/core/doctor-categories.ts @@ -85,6 +85,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet = new Set([ 'image_assets', 'integrity', 'jsonb_integrity', + 'link_resolution_opportunity', 'links_extraction_lag', 'markdown_body_completeness', 'nightly_quality_probe_health', diff --git a/src/core/link-extraction.ts b/src/core/link-extraction.ts index 684919191..caa7c4166 100644 --- a/src/core/link-extraction.ts +++ b/src/core/link-extraction.ts @@ -46,8 +46,32 @@ export interface EntityRef { * - sourceId null → 'unqualified' */ sourceId?: string | null; + /** + * Issue #972: set when the ref came from the generic `[[bare-name]]` + * pass (not gated by DIR_PATTERN). The `slug` field holds the literal + * wikilink text — callers MUST run it through a SlugResolver's + * `resolveBasenameMatches` (gated by `link_resolution.global_basename`) + * before persisting. Untagged refs already match a known entity dir + * (people/, companies/, etc.) and need no resolution. + * + * One ref tagged `needsResolution: true` may resolve to MULTIPLE target + * slugs when multiple pages share the basename. The caller emits one + * LinkCandidate per resolved match. + */ + needsResolution?: boolean; } +/** + * Issue #972: edge type stamped on graph rows produced by the + * global-basename wikilink resolution path. Distinct from the verb- + * inferred types (`mentions`, `works_at`, etc.) so users can audit or + * prune via `gbrain graph-query --type wikilink_basename`. + * + * All edges from this path also carry `linkSource: 'wikilink-resolved'` + * (the link-source provenance is the why; this edge type is the what). + */ +export const WIKILINK_BASENAME_LINK_TYPE = 'wikilink_basename'; + /** v0.17.0: how a link's target source was pinned at extraction time. */ export type LinkResolutionType = 'qualified' | 'unqualified'; @@ -105,6 +129,38 @@ const QUALIFIED_WIKILINK_RE = new RegExp( 'g', ); +/** + * Issue #972: generic Obsidian-style wikilink — matches `[[anything]]` + * with no DIR_PATTERN gate. Wiki/topic/learning pages frequently link + * via `[[bare-name]]` (`[[Fast-Weigh]]`, `[[2026-05-07-cost-plan]]`, + * `[[struktura]]`) where the literal text isn't a canonical slug; the + * resolver is responsible for matching it to the real page via + * basename uniqueness. + * + * Used AFTER QUALIFIED_WIKILINK_RE and WIKILINK_RE pick off their + * cases — the masked-ranges pass in extractEntityRefs prevents + * double-emission. Refs from this pass are tagged + * `needsResolution: true` so callers know to run them through a + * SlugResolver before the page-existence check. + * + * Regex shape is the union of the two existing wikilink regexes, + * minus the DIR_PATTERN gate. The character class disallows pipe, + * close-bracket, hash (anchor separator), newline, and open-bracket + * so we never match across line breaks or capture nested `[[...]]`. + * + * Adapted from PR #1233 (rayers). + */ +const WIKILINK_GENERIC_RE = /\[\[([^|\]#\n[]+?)(?:#[^|\]]*?)?(?:\|([^\]]+?))?\]\]/g; + +/** + * Issue #972 (codex [P2]): a markdown link whose LABEL contains a wikilink, + * e.g. `[see [[acme]]](companies/acme.md)`. Pass-1's ENTITY_REF_RE can't match + * the nested `]]` so it never spans this; without masking, the generic 2c pass + * would emit a stray basename ref for the inner `[[acme]]`. Mask the whole + * span out of the 2c scan so a wikilink inside a markdown label is inert. + */ +const MARKDOWN_LABEL_WIKILINK_RE = /\[[^\]\n]*\[\[[^\]\n]+\]\][^\]\n]*\]\([^)\n]+\)/g; + /** * Strip fenced code blocks (```...```) and inline code (`...`) from markdown, * replacing them with whitespace of equivalent length. Preserves byte offsets @@ -250,6 +306,7 @@ export function extractEntityRefs(content: string): EntityRef[] { // Markdown links have no source-qualification syntax — they're // always unqualified. Omit sourceId so the shape stays compatible // with pre-v0.17 consumers doing strict equality. + const markdownRanges: Array<[number, number]> = []; const mdPattern = new RegExp(ENTITY_REF_RE.source, ENTITY_REF_RE.flags); while ((match = mdPattern.exec(stripped)) !== null) { const name = match[1]; @@ -257,6 +314,7 @@ export function extractEntityRefs(content: string): EntityRef[] { const slug = fullPath; const dir = fullPath.split('/')[0]; refs.push({ name, slug, dir }); + markdownRanges.push([match.index, match.index + match[0].length]); } // 2a. v0.17.0 qualified wikilinks: [[source-id:path]] or [[source-id:path|Display]] @@ -278,6 +336,7 @@ export function extractEntityRefs(content: string): EntityRef[] { // 2b. Unqualified Obsidian wikilinks: [[path]] or [[path|Display Text]] // Same shape rule: omit sourceId when unqualified. + const unqualifiedRanges: Array<[number, number]> = []; const unmasked = maskRanges(stripped, qualifiedRanges); const wikiPattern = new RegExp(WIKILINK_RE.source, WIKILINK_RE.flags); while ((match = wikiPattern.exec(unmasked)) !== null) { @@ -288,6 +347,38 @@ export function extractEntityRefs(content: string): EntityRef[] { const displayName = (match[2] || slug).trim(); const dir = slug.split('/')[0]; refs.push({ name: displayName, slug, dir }); + unqualifiedRanges.push([match.index, match.index + match[0].length]); + } + + // 2c. Issue #972: generic `[[bare-name]]` wikilinks not gated by + // DIR_PATTERN. Wiki/topic/learning content frequently uses these + // where the bare text isn't a canonical brain slug. Tagged + // `needsResolution: true` so the caller routes them through a + // SlugResolver's `resolveBasenameMatches` before persisting. + // Mask out 2a/2b ranges so we don't double-emit; skip qualified- + // syntax tokens (contain `:`) that would be 2a's job. Issue #972 + // (codex [P2]): ALSO mask pass-1 markdown-link ranges so a wikilink + // inside a markdown label — `[see [[acme]]](companies/acme.md)` — + // doesn't spawn a stray generic basename ref from inside the label. + const labelWikilinkRanges: Array<[number, number]> = []; + const labelWlPattern = new RegExp(MARKDOWN_LABEL_WIKILINK_RE.source, MARKDOWN_LABEL_WIKILINK_RE.flags); + while ((match = labelWlPattern.exec(stripped)) !== null) { + labelWikilinkRanges.push([match.index, match.index + match[0].length]); + } + const genericMasked = maskRanges( + stripped, + [...markdownRanges, ...qualifiedRanges, ...unqualifiedRanges, ...labelWikilinkRanges], + ); + const genericPattern = new RegExp(WIKILINK_GENERIC_RE.source, WIKILINK_GENERIC_RE.flags); + while ((match = genericPattern.exec(genericMasked)) !== null) { + let slug = match[1].trim(); + if (!slug) continue; + if (slug.includes('://')) continue; + if (slug.includes(':')) continue; // qualified-syntax token; 2a owns these + if (slug.endsWith('.md')) slug = slug.slice(0, -3); + const displayName = (match[2] || slug).trim(); + const dir = slug.includes('/') ? slug.split('/')[0] : ''; + refs.push({ name: displayName, slug, dir, needsResolution: true }); } return refs; @@ -376,11 +467,44 @@ export async function extractPageLinks( frontmatter: Record, pageType: PageType, resolver: SlugResolver, + opts: { globalBasename?: boolean; skipFrontmatter?: boolean } = {}, ): Promise { const candidates: LinkCandidate[] = []; // 1. Markdown entity refs. for (const ref of extractEntityRefs(content)) { + // Issue #972: refs from the generic `[[bare-name]]` pass carry the + // literal wikilink text, not a real page slug. When global_basename + // mode is on AND the resolver implements basename lookup, resolve + // to every matching page and emit one candidate per match. When the + // flag is off (default), drop silently — back-compat with the + // pre-v0.40.8.2 behavior of dropping bare wikilinks outside + // DIR_PATTERN. + if (ref.needsResolution) { + if (!opts.globalBasename || typeof resolver.resolveBasenameMatches !== 'function') { + continue; + } + // Issue #972 (codex): resolve by the wikilink TARGET (ref.slug — the + // text inside `[[...]]` before any `|`), NOT the display alias + // (ref.name = match[2]). `[[struktura|the project]]` must resolve + // `struktura`, not "the project". The display text is for context only. + const matches = await resolver.resolveBasenameMatches(ref.slug); + if (matches.length === 0) continue; + const idx = content.indexOf(ref.slug); + const context = idx >= 0 ? excerpt(content, idx, 240) : ref.name; + for (const matched of matches) { + // Issue #972 (codex [P2]): a basename `[[own-tail]]` on its own page + // resolves back to itself — drop the self-loop. + if (matched === slug) continue; + candidates.push({ + targetSlug: matched, + linkType: WIKILINK_BASENAME_LINK_TYPE, + context, + linkSource: 'wikilink-resolved', + }); + } + continue; + } const idx = content.indexOf(ref.name); // Wider context window (240 chars vs original 80) catches verbs that // appear at sentence-or-paragraph distance from the slug — common in @@ -418,12 +542,28 @@ export async function extractPageLinks( } // 3. Frontmatter-derived edges (v0.13). Includes the legacy `source:` - // field along with the full field map. - const fm = await extractFrontmatterLinks(slug, pageType, frontmatter, resolver); - candidates.push(...fm.candidates); + // field along with the full field map. Caller can suppress via + // `opts.skipFrontmatter` — used by `extractLinksFromDB` to keep + // `--include-frontmatter` off semantics while still passing a real + // resolver (needed for `resolveBasenameMatches` in global-basename + // mode). Pre-issue-#972 this gating lived in the caller via a + // synthetic `nullResolver`; that pattern broke once the bare-wikilink + // path needed `resolveBasenameMatches` on the real resolver. + let fmUnresolved: UnresolvedFrontmatterRef[] = []; + if (!opts.skipFrontmatter) { + const fm = await extractFrontmatterLinks(slug, pageType, frontmatter, resolver); + candidates.push(...fm.candidates); + fmUnresolved = fm.unresolved; + } // Within-page dedup: same (fromSlug, targetSlug, linkType, linkSource) // collapses to one entry. First occurrence wins. + // Issue #972 (codex P2d, decided): a qualified `[[companies/acme]]` (typed + // markdown edge) and a bare `[[acme]]` (wikilink-resolved edge) to the SAME + // target are KEPT as separate rows — they carry different provenance + // (link_source) and link_type, and the audit trail (which kind of reference + // created the edge) is worth more than collapsing them. graph-query callers + // that want a unique target set dedup on to_slug themselves. const seen = new Set(); const result: LinkCandidate[] = []; for (const c of candidates) { @@ -432,7 +572,7 @@ export async function extractPageLinks( seen.add(key); result.push(c); } - return { candidates: result, unresolved: fm.unresolved }; + return { candidates: result, unresolved: fmUnresolved }; } /** Excerpt a window of `width` chars around `idx`, collapsed to one line. */ @@ -655,6 +795,67 @@ export interface SlugResolver { * extract/put_page summary so the user can see the gap. */ resolve(name: string, dirHint?: string | string[]): Promise; + /** + * Issue #972: return every slug whose basename (final `/`-segment, or + * the whole slug if it has no `/`) matches `name`. Multi-match by + * design — `[[struktura]]` referencing both `projects/struktura` and + * `archive/struktura` returns both; the caller emits one graph edge + * per result. Returns `[]` when no matches. + * + * Optional so existing SlugResolver consumers (e.g. synthetic + * frontmatter-only resolvers) keep working without code changes. + * Implementations should make this cheap to call repeatedly within + * a single extract run (build the index once, reuse it). + */ + resolveBasenameMatches?(name: string): Promise; +} + +/** + * Issue #972 (codex [P2] DRY): the ONE basename matcher. Before this, three + * surfaces (makeResolver, FS `resolveBasenameMatchesFromSlugs`, the doctor + * `link_resolution_opportunity` check) each hand-rolled their own key set + + * sort, and they drifted — the doctor omitted the slugified key, so its + * "N would resolve" estimate undercounted what extraction actually produces. + * All three now build/query through these two functions so they cannot drift. + * + * Keying: raw tail + lowercase tail + slugified tail. A slug's tail is its + * final `/`-segment (or the whole slug when it has no `/`). + */ +export function normalizeBasename(s: string): string { + return s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-'); +} + +/** Stable order: shorter slug first (likely closer to brain root), then lexical. */ +function basenameSort(a: string, b: string): number { + return (a.length - b.length) || a.localeCompare(b); +} + +/** Build a `key → slug[]` index over a slug collection. Keys: raw/lower/slugified tail. */ +export function buildBasenameIndex(slugs: Iterable): Map { + const idx = new Map(); + const addKey = (key: string, slug: string) => { + const existing = idx.get(key); + if (existing) { if (!existing.includes(slug)) existing.push(slug); } + else idx.set(key, [slug]); + }; + for (const slug of slugs) { + const tail = slug.includes('/') ? slug.slice(slug.lastIndexOf('/') + 1) : slug; + addKey(tail, slug); + const lower = tail.toLowerCase(); + if (lower !== tail) addKey(lower, slug); + const slugified = normalizeBasename(tail); + if (slugified && slugified !== tail && slugified !== lower) addKey(slugified, slug); + } + return idx; +} + +/** Look a name up in a basename index (raw → lower → slugified), stable-sorted. */ +export function queryBasenameIndex(idx: Map, name: string): string[] { + if (!name || typeof name !== 'string') return []; + const trimmed = name.trim(); + if (!trimmed) return []; + const hit = idx.get(trimmed) ?? idx.get(trimmed.toLowerCase()) ?? idx.get(normalizeBasename(trimmed)); + return hit ? [...hit].sort(basenameSort) : []; } /** @@ -674,13 +875,52 @@ export interface SlugResolver { */ export function makeResolver( engine: BrainEngine, - opts: { mode: 'batch' | 'live' } = { mode: 'live' }, + opts: { mode: 'batch' | 'live'; sourceId?: string } = { mode: 'live' }, ): SlugResolver { const cache = new Map(); const norm = (s: string) => s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-'); + // Issue #972: lazy-built basename → slug[] index for global-basename + // resolution. Built on first call to `resolveBasenameMatches`; reused + // for the rest of the resolver instance's lifetime. Cost is bounded + // (one `engine.getAllSlugs()` call per extract run / put_page). The + // index keys BOTH the raw tail and the slugified-tail so a wikilink + // `[[Fast-Weigh]]` matches a page slugged `companies/fast-weigh` as + // well as `notes/Fast-Weigh`. + let basenameIndex: Map | null = null; + async function ensureBasenameIndex(): Promise> { + if (basenameIndex !== null) return basenameIndex; + const idx = new Map(); + if (typeof engine.getAllSlugs !== 'function') { + basenameIndex = idx; + return idx; + } + try { + // Issue #972 (codex [P1]): scope the basename index to the resolver's + // source. getAllSlugs({sourceId}) keeps wikilink resolution from + // spanning unrelated sources — a bare [[name]] must NOT resolve to a + // same-tail page in a DIFFERENT source and create a cross-source edge. + // #972 is "global basename across folders," not "cross-source federation." + const all = await engine.getAllSlugs(opts.sourceId ? { sourceId: opts.sourceId } : undefined); + // Issue #972 (codex [P2] DRY): one shared index builder for all surfaces. + basenameIndex = buildBasenameIndex(all); + return basenameIndex; + } catch { + // Index build failed — empty map → resolveBasenameMatches finds + // nothing, resolver continues as if the flag was off. Never throw. + } + basenameIndex = idx; + return idx; + } + return { + async resolveBasenameMatches(name: string): Promise { + // Issue #972 (codex [P2] DRY): shared query so resolver + FS + doctor + // return the same matches in the same stable order. + return queryBasenameIndex(await ensureBasenameIndex(), name); + }, + async resolve(name: string, dirHint?: string | string[]): Promise { if (!name || typeof name !== 'string') return null; const trimmed = name.trim(); @@ -948,3 +1188,31 @@ export async function isAutoTimelineEnabled(engine: BrainEngine): Promise { + const envVal = process.env.GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME; + if (envVal != null) { + const normalized = envVal.trim().toLowerCase(); + return ['1', 'true', 'yes', 'on'].includes(normalized); + } + const val = await engine.getConfig('link_resolution.global_basename'); + if (val == null) return false; + const normalized = val.trim().toLowerCase(); + return ['1', 'true', 'yes', 'on'].includes(normalized); +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 18ac0d979..c351aa965 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -5088,6 +5088,38 @@ export const MIGRATIONS: Migration[] = [ } }, }, + { + version: 113, + name: 'links_link_source_widen_for_wikilink_basename', + // Issue #972: opt-in global-basename wikilink resolution (bare [[name]] + // resolved by slug tail) emits edges tagged + // `link_source = 'wikilink-resolved'`. Widen the CHECK to admit it. + // + // The FULL set is enumerated here — not just the new value — because + // DROP + re-ADD replaces the whole constraint. v95 + // (links_link_source_check_includes_mentions) added 'mentions'; since + // this migration runs AFTER v95, omitting 'mentions' would silently + // clobber that widening. Keep both branches in sync with src/schema.sql + // and src/core/pglite-schema.ts. + // + // Renumbered v93 → v109 → v110 → v112 → v113 across successive master + // merges (upstream claimed through v112 — pages_links_extracted_at — in + // the interim). Idempotent via DROP ... IF EXISTS, so it no-ops on + // installs that never created the constraint. + idempotent: true, + sql: ` + ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check; + ALTER TABLE links ADD CONSTRAINT links_link_source_check + CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')); + `, + sqlFor: { + pglite: ` + ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check; + ALTER TABLE links ADD CONSTRAINT links_link_source_check + CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')); + `, + }, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/operations.ts b/src/core/operations.ts index 5fa4c915c..574a363a6 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -16,7 +16,7 @@ import { expandQuery } from './search/expansion.ts'; import { dedupResults } from './search/dedup.ts'; import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts'; import type { HybridSearchMeta } from './types.ts'; -import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts'; +import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, isGlobalBasenameEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts'; import { isFactsBackstopEligible } from './facts/eligibility.ts'; import { stripTakesFence } from './takes-fence.ts'; import { stripFactsFence } from './facts-fence.ts'; @@ -991,9 +991,14 @@ async function runAutoLink( : {}; // Live-mode resolver: per-put throwaway cache, pg_trgm + optional search. - const resolver = makeResolver(engine, { mode: 'live' }); + // Issue #972 (codex [P1]): pass sourceId so basename resolution stays + // within this page's source — no cross-source basename edges. + const resolver = makeResolver(engine, { mode: 'live', sourceId: opts?.sourceId }); + // Issue #972: opt-in bare-wikilink basename resolution. Off by default. + const globalBasename = await isGlobalBasenameEnabled(engine); const { candidates, unresolved } = await extractPageLinks( slug, fullContent, parsed.frontmatter, parsed.type, resolver, + { globalBasename }, ); // Resolve which targets exist (skip refs to non-existent pages to avoid FK @@ -1040,10 +1045,17 @@ async function runAutoLink( l => l.link_source === 'frontmatter' && l.origin_slug === slug, ); - // Reconcilable outgoing edges: markdown + our own frontmatter edges. - // Manual edges (link_source='manual') are NEVER touched by reconciliation. + // Reconcilable outgoing edges: markdown + our own frontmatter edges + + // basename-resolved wikilinks (issue #972). Manual edges + // (link_source='manual') are NEVER touched by reconciliation. + // 'wikilink-resolved' MUST be reconcilable (codex outside-voice [P1]): + // auto-link writes these; if it weren't here, a basename edge would + // survive after the wikilink is deleted from the page OR the + // link_resolution.global_basename flag is turned off (out no longer + // includes it, so the stale-removal loop below must be allowed to drop it). const reconcilableOut = existingOut.filter( l => l.link_source === 'markdown' || l.link_source == null || + l.link_source === 'wikilink-resolved' || (l.link_source === 'frontmatter' && l.origin_slug === slug), ); diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index ae8e4ef98..39e2c7a28 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -258,7 +258,9 @@ CREATE TABLE IF NOT EXISTS links ( -- v0.41.18.0: 'mentions' added for auto-linked body-text mentions -- (gbrain extract links --by-mention). Filtered OUT of backlink-count -- for search ranking; only counts toward orphan-ratio + graph traversal. - link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')), + -- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename + -- wikilink resolution. See src/schema.sql for full rationale. + link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')), -- v0.41.18.0 (codex finding #12): nullable link_kind distinguishes -- "plain body mention" from "verb-pattern-derived typed link" within -- link_source='mentions'. See src/schema.sql for full rationale. diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 963257a44..b3be23bcc 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -411,8 +411,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to -- ============================================================ -- links: cross-references between pages -- ============================================================ --- Provenance model (v0.13): --- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL +-- Provenance model (v0.13, extended issue #972): +-- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL +-- 'wikilink-resolved' is the opt-in +-- (link_resolution.global_basename) basename-match +-- provenance — see issue #972 / migration v113. -- (NULL = legacy row written before v0.13; unknown source) -- origin_page_id — for link_source='frontmatter', the page whose YAML -- frontmatter created this edge; scopes reconciliation @@ -431,7 +434,9 @@ CREATE TABLE IF NOT EXISTS links ( -- v0.41.18.0: 'mentions' added for auto-linked body-text mentions -- (gbrain extract links --by-mention). Filtered OUT of backlink-count -- for search ranking; only counts toward orphan-ratio + graph traversal. - link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')), + -- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename + -- wikilink resolution (bare [[name]] resolved by slug tail). + link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')), -- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from -- "verb-pattern-derived typed link" within link_source='mentions'. -- Codex finding #12 design: keep link_source stable; add link_kind diff --git a/src/schema.sql b/src/schema.sql index 1a10af1fa..4facd491c 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -407,8 +407,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to -- ============================================================ -- links: cross-references between pages -- ============================================================ --- Provenance model (v0.13): --- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL +-- Provenance model (v0.13, extended issue #972): +-- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL +-- 'wikilink-resolved' is the opt-in +-- (link_resolution.global_basename) basename-match +-- provenance — see issue #972 / migration v113. -- (NULL = legacy row written before v0.13; unknown source) -- origin_page_id — for link_source='frontmatter', the page whose YAML -- frontmatter created this edge; scopes reconciliation @@ -427,7 +430,9 @@ CREATE TABLE IF NOT EXISTS links ( -- v0.41.18.0: 'mentions' added for auto-linked body-text mentions -- (gbrain extract links --by-mention). Filtered OUT of backlink-count -- for search ranking; only counts toward orphan-ratio + graph traversal. - link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')), + -- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename + -- wikilink resolution (bare [[name]] resolved by slug tail). + link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')), -- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from -- "verb-pattern-derived typed link" within link_source='mentions'. -- Codex finding #12 design: keep link_source stable; add link_kind diff --git a/test/doctor.test.ts b/test/doctor.test.ts index fc4010eae..fab1201d0 100644 --- a/test/doctor.test.ts +++ b/test/doctor.test.ts @@ -1159,6 +1159,164 @@ describe('v0.40.4 — graph_signals_coverage check', () => { }); }); +// ─── issue #972 — link_resolution_opportunity check ─────────────────────── + +describe('issue #972 — link_resolution_opportunity check', () => { + const { PGLiteEngine } = require('../src/core/pglite-engine.ts'); + const { checkLinkResolutionOpportunity } = require('../src/commands/doctor.ts'); + + let engine: any; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' }); + await engine.initSchema(); + }); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }); + + beforeEach(async () => { + await engine.executeRaw(`DELETE FROM links`); + await engine.executeRaw(`DELETE FROM pages`); + await engine.executeRaw( + `DELETE FROM config WHERE key = 'link_resolution.global_basename'`, + ); + }); + + test('skipped silently when flag is already enabled', async () => { + await engine.setConfig('link_resolution.global_basename', 'true'); + // Even with bare wikilinks that would resolve, the check returns ok + // because the user is already opted in — no hint to surface. + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/x', { + type: 'concept', title: 'X', + compiled_truth: 'See [[struktura]] and [[struktura]] and [[struktura]].', + timeline: '', + }); + const check = await checkLinkResolutionOpportunity(engine); + expect(check.status).toBe('ok'); + expect(check.message).toContain('already enabled'); + }); + + test('empty brain → ok with explanation', async () => { + const check = await checkLinkResolutionOpportunity(engine); + expect(check.status).toBe('ok'); + expect(check.message).toContain('empty'); + }); + + test('no bare wikilinks → ok', async () => { + await engine.putPage('people/alice', { + type: 'person', title: 'Alice', + compiled_truth: 'Met [Bob](people/bob).', timeline: '', + }); + await engine.putPage('people/bob', { + type: 'person', title: 'Bob', compiled_truth: '', timeline: '', + }); + const check = await checkLinkResolutionOpportunity(engine); + expect(check.status).toBe('ok'); + expect(check.message).toContain('No bare wikilinks'); + }); + + test('bare wikilinks present but none match → ok', async () => { + await engine.putPage('concepts/x', { + type: 'concept', title: 'X', + compiled_truth: 'See [[never-existed]] and [[also-not-here]].', + timeline: '', + }); + const check = await checkLinkResolutionOpportunity(engine); + expect(check.status).toBe('ok'); + expect(check.message).toContain('none have basename matches'); + }); + + test('≥5 would-resolve AND ≥20% ratio → warn with paste-ready hint', async () => { + // 5 distinct bare wikilinks, all resolving = 100% ratio. + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', compiled_truth: '', timeline: '', + }); + await engine.putPage('projects/eeva', { + type: 'project', title: 'Eeva', compiled_truth: '', timeline: '', + }); + await engine.putPage('companies/fast-weigh', { + type: 'company', title: 'Fast-Weigh', compiled_truth: '', timeline: '', + }); + await engine.putPage('projects/rosa', { + type: 'project', title: 'Rosa', compiled_truth: '', timeline: '', + }); + await engine.putPage('projects/dragon', { + type: 'project', title: 'Dragon', compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/wiki-index', { + type: 'concept', title: 'Wiki', + compiled_truth: 'See [[struktura]], [[eeva]], [[Fast-Weigh]], [[rosa]], [[dragon]] for context.', + timeline: '', + }); + const check = await checkLinkResolutionOpportunity(engine); + expect(check.status).toBe('warn'); + expect(check.message).toContain('5 of 5'); + expect(check.message).toContain('100%'); + expect(check.message).toContain( + 'gbrain config set link_resolution.global_basename true', + ); + }); + + test('<5 would-resolve → ok without warning (below threshold)', async () => { + // Only 2 bare wikilinks resolve → below the 5-link floor. + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', compiled_truth: '', timeline: '', + }); + await engine.putPage('projects/eeva', { + type: 'project', title: 'Eeva', compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/x', { + type: 'concept', title: 'X', + compiled_truth: 'See [[struktura]] and [[eeva]] for context.', + timeline: '', + }); + const check = await checkLinkResolutionOpportunity(engine); + expect(check.status).toBe('ok'); + expect(check.message).toContain('below the 20% / 5-link threshold'); + }); + + test('counts slugified-only matches (shared matcher, codex #972 DRY)', async () => { + // `[[Fast Weigh]]` (space) only resolves to `companies/fast-weigh` via the + // slugified key. Pre-consolidation the doctor index keyed raw+lower only + // and would have reported "none have basename matches"; the shared matcher + // adds the slugified key so the estimate matches what extraction resolves. + await engine.putPage('companies/fast-weigh', { + type: 'company', title: 'Fast-Weigh', compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/x', { + type: 'concept', title: 'X', + compiled_truth: 'See [[Fast Weigh]] for context.', timeline: '', + }); + const check = await checkLinkResolutionOpportunity(engine); + // 1/1 resolves (below the 5-link warn floor → ok) but it DID resolve. + expect(check.message).toContain('1/1'); + expect(check.message).toContain('would resolve'); + expect(check.message).not.toContain('none have basename matches'); + }); + + test('check is wired into runDoctor AND doctorReportRemote (source-grep)', async () => { + const source = await Bun.file( + new URL('../src/commands/doctor.ts', import.meta.url), + ).text(); + // Local buildChecks path. + expect(source).toMatch(/await checkLinkResolutionOpportunity\(engine, progress\)/); + // Thin-client doctorReportRemote path. + expect(source).toMatch(/await checkLinkResolutionOpportunity\(engine\)/); + // Heartbeat label registered. + expect(source).toContain("progress.heartbeat('link_resolution_opportunity')"); + // Issue #972 (T3): scan is bounded to a most-recent sample, not a full + // per-page getPage walk. + expect(source).toMatch(/ORDER BY id DESC LIMIT \$\{?SAMPLE_LIMIT\}?|ORDER BY id DESC LIMIT 1000/); + expect(source).not.toMatch(/await engine\.getPage\(ref\.slug/); + }); +}); + describe('v0.42 (#1699) — quarantined_pages + flagged_pages checks', () => { test('both checks are wired into buildChecks (source-grep)', async () => { const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text(); diff --git a/test/e2e/global-basename-pglite.test.ts b/test/e2e/global-basename-pglite.test.ts new file mode 100644 index 000000000..d3c767b46 --- /dev/null +++ b/test/e2e/global-basename-pglite.test.ts @@ -0,0 +1,282 @@ +/** + * Issue #972 E2E — opt-in global-basename wikilink resolution. + * + * Reproduces the issue's exact repro inside an in-memory PGLite brain: + * + * /vault/projects/struktura.md ← a real page + * /vault/concepts/knowledge-graph.md ← contains `[[struktura]]` + * + * The bare wikilink `[[struktura]]` does NOT match WIKILINK_RE + * (DIR_PATTERN-gated) so it falls through to the new pass 2c. + * + * Three contracts: + * 1. Flag OFF (default) — extract emits ZERO basename edges (back-compat + * with pre-issue-#972 behavior: bare wikilinks outside DIR_PATTERN + * drop silently). + * 2. Flag ON — extract emits ONE edge per basename match, tagged + * `link_type: 'wikilink_basename'`. + * 3. Ambiguous basename (same name in two directories) — extract emits + * ONE edge per match. No silent winner-takes-all, no silent drop. + * + * Both DB-source AND FS-source paths covered. PGLite in-memory, no + * DATABASE_URL needed. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { runExtract } from '../../src/commands/extract.ts'; + +let engine: PGLiteEngine; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); +}, 60_000); + +afterAll(async () => { + await engine.disconnect(); +}); + +async function truncateAll() { + for (const t of [ + 'content_chunks', 'links', 'tags', 'raw_data', + 'timeline_entries', 'page_versions', 'ingest_log', 'pages', + 'config', + ]) { + try { await (engine as any).db.exec(`DELETE FROM ${t}`); } catch { /* ok */ } + } +} + +beforeEach(async () => { + await truncateAll(); + brainDir = mkdtempSync(join(tmpdir(), 'gbrain-issue-972-')); +}, 15_000); + +function writeFile(rel: string, content: string) { + const full = join(brainDir, rel); + mkdirSync(join(full, '..'), { recursive: true }); + writeFileSync(full, content); +} + +// ─── FS-source path (the issue's repro) ───────────────────────────────── + +describe('issue #972 — FS-source (gbrain extract links default)', () => { + test("repro: [[struktura]] in concepts/ resolves to projects/struktura when flag ON", async () => { + // Seed both pages in the DB (extract validates targetSlug exists) + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', + compiled_truth: 'A project page.', timeline: '', + }); + await engine.putPage('concepts/knowledge-graph', { + type: 'concept', title: 'Knowledge Graph', + compiled_truth: 'This concept relates to [[struktura]].', timeline: '', + }); + // Mirror to disk so the FS extractor sees the files + writeFile('projects/struktura.md', + '---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n'); + writeFile('concepts/knowledge-graph.md', + '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n'); + + await engine.setConfig('link_resolution.global_basename', 'true'); + + await runExtract(engine, ['links', '--dir', brainDir]); + + const outLinks = await engine.getLinks('concepts/knowledge-graph'); + const strk = outLinks.find(l => l.to_slug === 'projects/struktura'); + expect(strk).toBeDefined(); + expect(strk!.link_type).toBe('wikilink_basename'); + // Issue #972 (T1): FS-source basename edges carry the same provenance + // tag as DB / put_page, not the default 'markdown'. + expect(strk!.link_source).toBe('wikilink-resolved'); + }); + + test('back-compat: flag OFF → ZERO basename edges from same repro', async () => { + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', + compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/knowledge-graph', { + type: 'concept', title: 'Knowledge Graph', + compiled_truth: 'This relates to [[struktura]].', timeline: '', + }); + writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n'); + writeFile('concepts/knowledge-graph.md', + '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis relates to [[struktura]].\n'); + + // Flag explicitly off (also the default) + await engine.setConfig('link_resolution.global_basename', 'false'); + + await runExtract(engine, ['links', '--dir', brainDir]); + + const outLinks = await engine.getLinks('concepts/knowledge-graph'); + expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined(); + expect(outLinks.filter(l => l.link_type === 'wikilink_basename')).toEqual([]); + }); + + test('ambiguous basename → emits one edge per match (no silent winner)', async () => { + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', + compiled_truth: '', timeline: '', + }); + await engine.putPage('archive/struktura', { + type: 'concept' as any, title: 'Struktura (archived)', + compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/x', { + type: 'concept', title: 'X', + compiled_truth: 'See [[struktura]].', timeline: '', + }); + + writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n'); + writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n'); + writeFile('concepts/x.md', + '---\ntitle: X\ntype: concept\n---\n\nSee [[struktura]].\n'); + + await engine.setConfig('link_resolution.global_basename', 'true'); + + await runExtract(engine, ['links', '--dir', brainDir]); + + const outLinks = await engine.getLinks('concepts/x'); + const basenameLinks = outLinks + .filter(l => l.link_type === 'wikilink_basename') + .map(l => l.to_slug) + .sort(); + expect(basenameLinks).toEqual(['archive/struktura', 'projects/struktura']); + }); +}); + +// ─── DB-source path (gbrain extract links --source db) ────────────────── + +describe('issue #972 — DB-source (gbrain extract links --source db)', () => { + test('flag ON → bare wikilink in compiled_truth resolves to basename match', async () => { + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', + compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/knowledge-graph', { + type: 'concept', title: 'Knowledge Graph', + compiled_truth: 'This relates to [[struktura]].', timeline: '', + }); + await engine.setConfig('link_resolution.global_basename', 'true'); + + await runExtract(engine, ['links', '--source', 'db']); + + const outLinks = await engine.getLinks('concepts/knowledge-graph'); + const strk = outLinks.find(l => l.to_slug === 'projects/struktura'); + expect(strk).toBeDefined(); + expect(strk!.link_type).toBe('wikilink_basename'); + }); + + test('flag OFF → no basename edges via DB path (back-compat)', async () => { + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', + compiled_truth: '', timeline: '', + }); + await engine.putPage('concepts/knowledge-graph', { + type: 'concept', title: 'Knowledge Graph', + compiled_truth: 'This relates to [[struktura]].', timeline: '', + }); + + await runExtract(engine, ['links', '--source', 'db']); + + const outLinks = await engine.getLinks('concepts/knowledge-graph'); + expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined(); + }); +}); + +// ─── put_page auto-link path ──────────────────────────────────────────── +// +// put_page accepts `content` as a full markdown document with frontmatter +// (not the engine's narrow Page shape). Auto-link runs INSIDE put_page, +// so the basename-resolution path picks up the flag from +// engine.getConfig() once per call. + +const PUT_PAGE_MARKDOWN_WITH_WIKILINK = `--- +title: Knowledge Graph +type: concept +--- + +This relates to [[struktura]]. +`; + +describe('issue #972 — put_page auto-link', () => { + test('newly-written page with bare wikilink → basename edge when flag ON', async () => { + // Need the target page to exist first (auto-link validates against the + // existing slug set). + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', + compiled_truth: '', timeline: '', + }); + + await engine.setConfig('link_resolution.global_basename', 'true'); + + const { operations } = await import('../../src/core/operations.ts'); + const putPage = operations.find(op => op.name === 'put_page')!; + await putPage.handler( + { engine, remote: false } as never, + { + slug: 'concepts/knowledge-graph', + content: PUT_PAGE_MARKDOWN_WITH_WIKILINK, + }, + ); + + const outLinks = await engine.getLinks('concepts/knowledge-graph'); + const strk = outLinks.find(l => l.to_slug === 'projects/struktura'); + expect(strk).toBeDefined(); + expect(strk!.link_type).toBe('wikilink_basename'); + }); + + test('flag OFF → put_page does NOT emit basename edge (back-compat)', async () => { + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', + compiled_truth: '', timeline: '', + }); + + // Default-off (no setConfig) + + const { operations } = await import('../../src/core/operations.ts'); + const putPage = operations.find(op => op.name === 'put_page')!; + await putPage.handler( + { engine, remote: false } as never, + { + slug: 'concepts/knowledge-graph', + content: PUT_PAGE_MARKDOWN_WITH_WIKILINK, + }, + ); + + const outLinks = await engine.getLinks('concepts/knowledge-graph'); + expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined(); + }); + + test('stale basename edge is removed when the wikilink is deleted (codex #972)', async () => { + // Regression: wikilink-resolved edges must be reconcilable, else they + // survive after the bare wikilink is removed from the page body. + await engine.putPage('projects/struktura', { + type: 'project', title: 'Struktura', compiled_truth: '', timeline: '', + }); + await engine.setConfig('link_resolution.global_basename', 'true'); + + const { operations } = await import('../../src/core/operations.ts'); + const putPage = operations.find(op => op.name === 'put_page')!; + + // 1. Write the page WITH the wikilink → edge lands. + await putPage.handler({ engine, remote: false } as never, { + slug: 'concepts/knowledge-graph', content: PUT_PAGE_MARKDOWN_WITH_WIKILINK, + }); + let outLinks = await engine.getLinks('concepts/knowledge-graph'); + expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeDefined(); + + // 2. Re-write the page WITHOUT the wikilink → edge must be reconciled away. + await putPage.handler({ engine, remote: false } as never, { + slug: 'concepts/knowledge-graph', + content: '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nNo links here anymore.\n', + }); + outLinks = await engine.getLinks('concepts/knowledge-graph'); + expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined(); + }); +}); diff --git a/test/extract-fs.test.ts b/test/extract-fs.test.ts index 4121e2889..adb4afdd4 100644 --- a/test/extract-fs.test.ts +++ b/test/extract-fs.test.ts @@ -246,3 +246,213 @@ describe('gbrain extract --dir default resolution', () => { } }); }); + +// ─── issue #972: pure-function tests for resolveSlugAll + helpers ──────── + +import { + resolveSlug, + resolveSlugAll, + resolveBasenameMatchesFromSlugs, + extractLinksFromFile, +} from '../src/commands/extract.ts'; + +describe('extractLinksFromFile — code-fence stripping (codex P2b)', () => { + test('a bare wikilink inside a code fence does NOT create an FS edge', async () => { + const allSlugs = new Set(['projects/struktura', 'concepts/x']); + const content = [ + '---', 'title: X', 'type: concept', '---', '', + 'Real ref: [[struktura]].', + '', + '```', 'code mentions [[struktura]] but must be ignored', '```', + ].join('\n'); + const links = await extractLinksFromFile(content, 'concepts/x.md', allSlugs, { globalBasename: true }); + const toStruktura = links.filter(l => l.to_slug === 'projects/struktura'); + // Exactly one edge — from the prose ref, NOT the fenced one. + expect(toStruktura.length).toBe(1); + expect(toStruktura[0].link_source).toBe('wikilink-resolved'); + }); +}); + +describe('resolveBasenameMatchesFromSlugs (pure)', () => { + test('returns ALL slugs whose tail matches', () => { + const all = new Set([ + 'projects/struktura', + 'archive/struktura', + 'notes/other', + ]); + const matches = resolveBasenameMatchesFromSlugs('struktura', all); + expect(matches.sort()).toEqual(['archive/struktura', 'projects/struktura']); + }); + + test('case-insensitive tail match', () => { + const all = new Set(['companies/fast-weigh']); + expect(resolveBasenameMatchesFromSlugs('Fast-Weigh', all)) + .toEqual(['companies/fast-weigh']); + expect(resolveBasenameMatchesFromSlugs('FAST-WEIGH', all)) + .toEqual(['companies/fast-weigh']); + }); + + test('slugified fallback (spaces → hyphens)', () => { + const all = new Set(['companies/fast-weigh']); + expect(resolveBasenameMatchesFromSlugs('Fast Weigh', all)) + .toEqual(['companies/fast-weigh']); + }); + + test('top-level slugs (no `/`) match by themselves', () => { + const all = new Set(['struktura', 'notes/struktura']); + const matches = resolveBasenameMatchesFromSlugs('struktura', all); + expect(matches.sort()).toEqual(['notes/struktura', 'struktura']); + }); + + test('no match returns []', () => { + const all = new Set(['projects/struktura']); + expect(resolveBasenameMatchesFromSlugs('never-existed', all)).toEqual([]); + }); + + test('empty/whitespace input returns []', () => { + const all = new Set(['projects/struktura']); + expect(resolveBasenameMatchesFromSlugs('', all)).toEqual([]); + expect(resolveBasenameMatchesFromSlugs(' ', all)).toEqual([]); + }); + + test('stable sort: shorter slug first, then lexical', () => { + const all = new Set([ + 'zzz/struktura', + 'projects/struktura', + 'archive/struktura', + 'a/struktura', + ]); + const matches = resolveBasenameMatchesFromSlugs('struktura', all); + // Lengths: a/struktura(11), zzz/struktura(13), archive/struktura(17), projects/struktura(18) + expect(matches).toEqual([ + 'a/struktura', + 'zzz/struktura', + 'archive/struktura', + 'projects/struktura', + ]); + }); +}); + +describe('resolveSlugAll', () => { + test('ancestor walk wins → single-element array, no basename fallback', () => { + // resolveSlug already finds notes/struktura via ancestor walk. + // resolveSlugAll must return only that one even though basename + // would find others. + const all = new Set([ + 'notes/struktura', + 'archive/struktura', + 'projects/struktura', + ]); + const out = resolveSlugAll('notes/sub', 'struktura.md', all, + { globalBasename: true }); + expect(out).toEqual(['notes/struktura']); + }); + + test('ancestor walk misses + globalBasename off → []', () => { + const all = new Set(['projects/struktura']); + const out = resolveSlugAll('concepts', 'struktura.md', all); + expect(out).toEqual([]); + }); + + test('ancestor walk misses + globalBasename on → all basename matches', () => { + const all = new Set([ + 'projects/struktura', + 'archive/struktura', + ]); + const out = resolveSlugAll('concepts', 'struktura.md', all, + { globalBasename: true }); + expect(out.sort()).toEqual(['archive/struktura', 'projects/struktura']); + }); + + test('zero basename matches when globalBasename on returns []', () => { + const all = new Set(['projects/struktura']); + const out = resolveSlugAll('concepts', 'phantom.md', all, + { globalBasename: true }); + expect(out).toEqual([]); + }); + + test('strips dirname from relTarget when applying basename lookup', () => { + // [[notes/struktura]] with no `notes` ancestor: ancestor walk strips + // `concepts` → tries `notes/struktura` which DOES exist → emits. + // This case verifies the basename fallback only fires when ancestor + // walk truly fails. (Sanity check on the fallback ordering.) + const all = new Set(['notes/struktura']); + expect(resolveSlugAll('concepts', 'notes/struktura.md', all)) + .toEqual(['notes/struktura']); + }); + + test('resolveSlug back-compat: existing single-match callers unaffected', () => { + const all = new Set(['notes/struktura']); + // The legacy resolveSlug must keep returning the string|null shape. + expect(resolveSlug('notes', 'struktura.md', all)).toBe('notes/struktura'); + expect(resolveSlug('concepts', 'phantom.md', all)).toBeNull(); + }); +}); + +describe('issue #972 repro: bare wikilinks resolve when flag is on', () => { + // End-to-end: reproduces the issue's exact repro inside a tempdir + + // PGLite, then asserts edge count under both flag states. + test('flag OFF → 0 edges (back-compat)', async () => { + await engine.putPage('projects/struktura', + { type: 'project', title: 'Struktura', compiled_truth: 'A project page.', timeline: '' }); + await engine.putPage('concepts/knowledge-graph', + { type: 'concept', title: 'Knowledge Graph', + compiled_truth: 'This concept relates to [[struktura]].', timeline: '' }); + + // Mirror to disk: the FS extractor walks files, not DB pages. + writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n'); + writeFile('concepts/knowledge-graph.md', + '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n'); + + // Ensure flag is off (default) + await engine.setConfig('link_resolution.global_basename', 'false'); + + await runExtract(engine, ['links', '--dir', brainDir]); + const links = await engine.getLinks('concepts/knowledge-graph'); + expect(links.find(l => l.to_slug === 'projects/struktura')).toBeUndefined(); + }); + + test('flag ON → 1 edge with wikilink_basename type', async () => { + await engine.putPage('projects/struktura', + { type: 'project', title: 'Struktura', compiled_truth: 'A project page.', timeline: '' }); + await engine.putPage('concepts/knowledge-graph', + { type: 'concept', title: 'Knowledge Graph', + compiled_truth: 'This concept relates to [[struktura]].', timeline: '' }); + + writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n'); + writeFile('concepts/knowledge-graph.md', + '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n'); + + await engine.setConfig('link_resolution.global_basename', 'true'); + + await runExtract(engine, ['links', '--dir', brainDir]); + const links = await engine.getLinks('concepts/knowledge-graph'); + const strk = links.find(l => l.to_slug === 'projects/struktura'); + expect(strk).toBeDefined(); + expect(strk!.link_type).toBe('wikilink_basename'); + }); + + test('flag ON + ambiguous basename → one edge per match', async () => { + await engine.putPage('projects/struktura', + { type: 'project', title: 'Struktura', compiled_truth: '', timeline: '' }); + await engine.putPage('archive/struktura', + { type: 'concept' as any, title: 'Struktura (archived)', + compiled_truth: '', timeline: '' }); + await engine.putPage('concepts/knowledge-graph', + { type: 'concept', title: 'Knowledge Graph', + compiled_truth: 'See [[struktura]].', timeline: '' }); + + writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n'); + writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n'); + writeFile('concepts/knowledge-graph.md', + '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nSee [[struktura]].\n'); + + await engine.setConfig('link_resolution.global_basename', 'true'); + + await runExtract(engine, ['links', '--dir', brainDir]); + const links = await engine.getLinks('concepts/knowledge-graph'); + const basenameLinks = links.filter(l => l.link_type === 'wikilink_basename'); + const targets = basenameLinks.map(l => l.to_slug).sort(); + expect(targets).toEqual(['archive/struktura', 'projects/struktura']); + }); +}); diff --git a/test/link-extraction.test.ts b/test/link-extraction.test.ts index 6829ffeca..56c4fa662 100644 --- a/test/link-extraction.test.ts +++ b/test/link-extraction.test.ts @@ -109,6 +109,92 @@ describe('extractEntityRefs', () => { expect(refs.length).toBe(1); expect(refs[0].dir).toBe('meetings'); }); + + // ─── issue #972: generic `[[bare-name]]` wikilinks (pass 2c) ───────────── + + test('tags bare wikilinks with needsResolution flag', () => { + const refs = extractEntityRefs( + 'See [[Fast-Weigh]] and [[2026-05-07-cost-plan-rosa-pilot]] for context.', + ); + expect(refs.length).toBe(2); + expect(refs.every(r => r.needsResolution === true)).toBe(true); + expect(refs.map(r => r.slug).sort()).toEqual([ + '2026-05-07-cost-plan-rosa-pilot', + 'Fast-Weigh', + ]); + // dir is empty string when the bare wikilink has no `/` + for (const r of refs) { + expect(r.dir).toBe(''); + } + }); + + test('does NOT double-emit when DIR_PATTERN wikilink also passes 2b', () => { + // [[people/alice]] matches 2b (DIR_PATTERN-gated). 2c must NOT emit + // a duplicate ref. [[Fast-Weigh]] only matches 2c (no DIR_PATTERN). + const refs = extractEntityRefs('See [[people/alice]] and [[Fast-Weigh]].'); + const aliceRefs = refs.filter(r => r.slug === 'people/alice'); + const wikiRefs = refs.filter(r => r.slug === 'Fast-Weigh'); + expect(aliceRefs.length).toBe(1); + expect(aliceRefs[0].needsResolution).toBeUndefined(); + expect(wikiRefs.length).toBe(1); + expect(wikiRefs[0].needsResolution).toBe(true); + }); + + test('skips qualified-syntax tokens (those belong to 2a)', () => { + // [[wiki:topics/ai]] looks like 2a's qualified shape — even though + // it wouldn't satisfy DIR_PATTERN, 2c must not claim it either + // (the leading `:` is the qualified-syntax tell). + const refs = extractEntityRefs('See [[wiki:topics/ai]] and [[bare-name]].'); + const bare = refs.find(r => r.slug === 'bare-name'); + expect(bare).toBeDefined(); + expect(bare!.needsResolution).toBe(true); + const wrongQualified = refs.filter( + r => r.slug.includes(':') && r.needsResolution === true, + ); + expect(wrongQualified.length).toBe(0); + }); + + test('a wikilink inside a markdown-link label is inert (codex P2a)', () => { + // `[see [[acme]]](companies/acme.md)` must NOT spawn a stray generic + // basename ref for the inner `[[acme]]`. Pass-1 can't match the nested + // brackets, so the label-wikilink span is masked out of pass 2c. + const refs = extractEntityRefs('[see [[acme]]](companies/acme.md)'); + expect(refs.filter(r => r.needsResolution)).toEqual([]); + // But an independent bare wikilink on the same line still emits. + const refs2 = extractEntityRefs('[Acme](companies/acme) and bare [[acme]] here.'); + expect(refs2.find(r => r.slug === 'companies/acme' && !r.needsResolution)).toBeDefined(); + expect(refs2.find(r => r.slug === 'acme' && r.needsResolution)).toBeDefined(); + }); + + test('strips .md suffix from bare wikilinks', () => { + const refs = extractEntityRefs('See [[struktura.md]] for context.'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('struktura'); + expect(refs[0].needsResolution).toBe(true); + }); + + test('extracts display name from [[slug|Display]] shape', () => { + const refs = extractEntityRefs('See [[struktura|The Project]] for details.'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('struktura'); + expect(refs[0].name).toBe('The Project'); + expect(refs[0].needsResolution).toBe(true); + }); + + test('strips #anchor from bare wikilinks', () => { + const refs = extractEntityRefs('Jump to [[notes#section-2]].'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('notes'); + expect(refs[0].needsResolution).toBe(true); + }); + + test('skips bare wikilinks inside fenced code blocks', () => { + const refs = extractEntityRefs( + '```\nThis is a code block with [[fake-link]] inside.\n```\nReal: [[real-link]].', + ); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('real-link'); + }); }); // ─── extractPageLinks ────────────────────────────────────────── @@ -181,6 +267,187 @@ describe('extractPageLinks', () => { const aliceLink = candidates.find(c => c.targetSlug === 'people/alice'); expect(aliceLink!.linkType).toBe('attended'); }); + + // ─── issue #972: bare wikilink → resolver.resolveBasenameMatches ───────── + + test('bare wikilink drops silently when globalBasename flag is OFF', async () => { + // Resolver that WOULD resolve, but we never reach it because the + // flag is off — this is the back-compat invariant. + const resolver: SlugResolver = { + resolve: async () => null, + resolveBasenameMatches: async () => ['projects/struktura'], + }; + const { candidates } = await extractPageLinks( + 'concepts/knowledge-graph', + 'This relates to [[struktura]].', + {}, 'concept', resolver, + // opts.globalBasename omitted (= false) + ); + expect(candidates.find(c => c.targetSlug === 'projects/struktura')).toBeUndefined(); + expect(candidates).toEqual([]); + }); + + test('bare wikilink emits one candidate per basename match when flag ON', async () => { + const resolver: SlugResolver = { + resolve: async () => null, + resolveBasenameMatches: async (name) => { + if (name === 'struktura') return ['projects/struktura', 'archive/struktura']; + return []; + }, + }; + const { candidates } = await extractPageLinks( + 'concepts/knowledge-graph', + 'This relates to [[struktura]].', + {}, 'concept', resolver, { globalBasename: true }, + ); + const targets = candidates.map(c => c.targetSlug).sort(); + expect(targets).toEqual(['archive/struktura', 'projects/struktura']); + // Both edges stamped with the new edge type + provenance. + for (const c of candidates) { + expect(c.linkType).toBe('wikilink_basename'); + expect(c.linkSource).toBe('wikilink-resolved'); + } + }); + + test('bare wikilink with single basename match emits one candidate', async () => { + const resolver: SlugResolver = { + resolve: async () => null, + resolveBasenameMatches: async (name) => + name === 'struktura' ? ['projects/struktura'] : [], + }; + const { candidates } = await extractPageLinks( + 'concepts/knowledge-graph', + 'See [[struktura]] for details.', + {}, 'concept', resolver, { globalBasename: true }, + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('projects/struktura'); + expect(candidates[0].linkType).toBe('wikilink_basename'); + }); + + test('basename self-link is dropped (codex P2c)', async () => { + // `[[struktura]]` on the page concepts/struktura resolves back to itself — + // the self-loop must be dropped. + const resolver: SlugResolver = { + resolve: async () => null, + resolveBasenameMatches: async (name) => + name === 'struktura' ? ['concepts/struktura', 'projects/struktura'] : [], + }; + const { candidates } = await extractPageLinks( + 'concepts/struktura', // the page being processed + 'See [[struktura]].', + {}, 'concept', resolver, { globalBasename: true }, + ); + // Only the OTHER match survives; no self-edge to concepts/struktura. + expect(candidates.map(c => c.targetSlug)).toEqual(['projects/struktura']); + }); + + test('aliased wikilink resolves the TARGET, not the display text (codex #972)', async () => { + // `[[struktura|the project]]` must resolve basename `struktura`, never + // the alias "the project". Regression for the codex-caught bug where + // extractPageLinks resolved ref.name (display) instead of ref.slug. + const seen: string[] = []; + const resolver: SlugResolver = { + resolve: async () => null, + resolveBasenameMatches: async (name) => { + seen.push(name); + return name === 'struktura' ? ['projects/struktura'] : []; + }, + }; + const { candidates } = await extractPageLinks( + 'concepts/knowledge-graph', + 'This relates to [[struktura|the project]].', + {}, 'concept', resolver, { globalBasename: true }, + ); + expect(seen).toContain('struktura'); + expect(seen).not.toContain('the project'); + expect(candidates.map(c => c.targetSlug)).toEqual(['projects/struktura']); + }); + + test('bare wikilink with zero basename matches drops silently (no dangling row)', async () => { + const resolver: SlugResolver = { + resolve: async () => null, + resolveBasenameMatches: async () => [], + }; + const { candidates } = await extractPageLinks( + 'concepts/x', 'Mention [[never-existed]].', + {}, 'concept', resolver, { globalBasename: true }, + ); + expect(candidates.find(c => c.targetSlug === 'never-existed')).toBeUndefined(); + expect(candidates).toEqual([]); + }); + + test('bare wikilink resolution does not interfere with DIR_PATTERN wikilinks', async () => { + // 2b refs (people/alice) take the verb-inferred type; + // 2c refs (struktura) take wikilink_basename. Same call. + const resolver: SlugResolver = { + resolve: async () => null, + resolveBasenameMatches: async (name) => + name === 'struktura' ? ['projects/struktura'] : [], + }; + const { candidates } = await extractPageLinks( + 'concepts/x', + '[[people/alice]] is the lead. The work is [[struktura]].', + {}, 'concept', resolver, { globalBasename: true }, + ); + const alice = candidates.find(c => c.targetSlug === 'people/alice'); + const strk = candidates.find(c => c.targetSlug === 'projects/struktura'); + expect(alice).toBeDefined(); + expect(alice!.linkType).not.toBe('wikilink_basename'); // verb-inferred + expect(strk).toBeDefined(); + expect(strk!.linkType).toBe('wikilink_basename'); + }); + + test('opts.skipFrontmatter suppresses the frontmatter pass', async () => { + // Real resolver shape that WOULD resolve frontmatter source: too, + // but skipFrontmatter blocks the path entirely. + const resolver: SlugResolver = { + resolve: async (name) => + name === 'meetings/2026-01-15' ? 'meetings/2026-01-15' : null, + }; + const fm = { source: 'meetings/2026-01-15' }; + const withFm = await extractPageLinks( + 'docs/x', 'plain content', fm, 'person', resolver, + { skipFrontmatter: false }, + ); + const withoutFm = await extractPageLinks( + 'docs/x', 'plain content', fm, 'person', resolver, + { skipFrontmatter: true }, + ); + expect(withFm.candidates.find(c => c.linkType === 'source')).toBeDefined(); + expect(withoutFm.candidates.find(c => c.linkType === 'source')).toBeUndefined(); + // Issue #972 (codex P2e): skipFrontmatter must return an empty unresolved + // list (the pass is skipped entirely), never undefined. + expect(withoutFm.unresolved).toEqual([]); + }); + + test('skipFrontmatter suppresses unresolved frontmatter refs too (codex P2e)', async () => { + // A frontmatter field the resolver CANNOT resolve normally populates + // `unresolved`; with skipFrontmatter the whole pass is gone so it's []. + const resolver: SlugResolver = { resolve: async () => null }; + const fm = { key_people: ['Nobody Known'] }; + const withFm = await extractPageLinks( + 'companies/acme', 'plain content', fm, 'company', resolver, + { skipFrontmatter: false }, + ); + const withoutFm = await extractPageLinks( + 'companies/acme', 'plain content', fm, 'company', resolver, + { skipFrontmatter: true }, + ); + expect(withFm.unresolved.length).toBeGreaterThan(0); // pass ran, ref unresolved + expect(withoutFm.unresolved).toEqual([]); // pass skipped + }); + + test('globalBasename does nothing when resolver lacks resolveBasenameMatches', async () => { + // The frontmatter-only synthetic resolver doesn't implement basename + // lookup. Make sure we don't blow up — just drop the bare ref. + const resolver: SlugResolver = { resolve: async () => null }; + const { candidates } = await extractPageLinks( + 'concepts/x', 'See [[struktura]].', + {}, 'concept', resolver, { globalBasename: true }, + ); + expect(candidates).toEqual([]); + }); }); // ─── inferLinkType ───────────────────────────────────────────── @@ -729,6 +996,149 @@ describe('makeResolver — fallback chain', () => { const r = makeResolver(engine, { mode: 'batch' }); expect(await r.resolve('Nonexistent Person', 'people')).toBeNull(); }); + + // ─── issue #972: resolveBasenameMatches ─────────────────────────────── + + // Extended fake engine that also implements `getAllSlugs` so + // resolveBasenameMatches has something to walk. + function makeFakeEngineWithSlugs(slugs: string[]): BrainEngine { + const lookup = new Set(slugs); + let getAllCalls = 0; + const engine = { + async getPage(slug: string) { + return lookup.has(slug) ? { slug } as any : null; + }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + async getAllSlugs() { + getAllCalls++; + return new Set(slugs); + }, + } as unknown as BrainEngine; + (engine as any)._counts = () => ({ getAllCalls }); + return engine; + } + + test('resolveBasenameMatches: exact tail hit returns the slug', async () => { + const engine = makeFakeEngineWithSlugs([ + 'projects/struktura', + 'people/alice', + ]); + const r = makeResolver(engine); + expect(await r.resolveBasenameMatches!('struktura')).toEqual(['projects/struktura']); + }); + + test('resolveBasenameMatches: multi-match returns ALL hits', async () => { + const engine = makeFakeEngineWithSlugs([ + 'projects/struktura', + 'archive/struktura', + 'notes/struktura', + ]); + const r = makeResolver(engine); + const out = await r.resolveBasenameMatches!('struktura'); + expect(out.sort()).toEqual([ + 'archive/struktura', + 'notes/struktura', + 'projects/struktura', + ]); + }); + + test('resolveBasenameMatches: case-insensitive fallback', async () => { + const engine = makeFakeEngineWithSlugs(['companies/fast-weigh']); + const r = makeResolver(engine); + // Raw `Fast-Weigh` does not match the lowercase tail, but the + // lowercased+slugified key does — both should hit. + expect(await r.resolveBasenameMatches!('fast-weigh')).toEqual(['companies/fast-weigh']); + expect(await r.resolveBasenameMatches!('Fast-Weigh')).toContain('companies/fast-weigh'); + }); + + test('resolveBasenameMatches: no matches returns []', async () => { + const engine = makeFakeEngineWithSlugs(['projects/struktura']); + const r = makeResolver(engine); + expect(await r.resolveBasenameMatches!('never-existed')).toEqual([]); + }); + + test('resolveBasenameMatches: scopes the index by sourceId (codex #972)', async () => { + // Regression: a bare [[struktura]] in source A must NOT resolve to a + // same-tail page in source B. makeResolver({sourceId}) must pass the + // scope to getAllSlugs so the index only contains the source's slugs. + let sawOpts: any; + const bySource: Record = { + 'src-a': ['projects/struktura'], + 'src-b': ['archive/struktura'], + }; + const engine = { + async getPage() { return null; }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + async getAllSlugs(opts?: { sourceId?: string }) { + sawOpts = opts; + const sid = opts?.sourceId; + return new Set(sid ? (bySource[sid] ?? []) : Object.values(bySource).flat()); + }, + } as unknown as BrainEngine; + const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' }); + const out = await r.resolveBasenameMatches!('struktura'); + expect(sawOpts).toEqual({ sourceId: 'src-a' }); + expect(out).toEqual(['projects/struktura']); // src-a only + expect(out).not.toContain('archive/struktura'); // no cross-source + }); + + test('resolveBasenameMatches: no sourceId stays brain-wide (back-compat)', async () => { + let sawOpts: any = 'unset'; + const engine = { + async getPage() { return null; }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + async getAllSlugs(opts?: { sourceId?: string }) { + sawOpts = opts; + return new Set(['projects/struktura', 'archive/struktura']); + }, + } as unknown as BrainEngine; + const r = makeResolver(engine, { mode: 'batch' }); + const out = await r.resolveBasenameMatches!('struktura'); + expect(sawOpts).toBeUndefined(); // unscoped call + expect(out.sort()).toEqual(['archive/struktura', 'projects/struktura']); + }); + + test('resolveBasenameMatches: empty input returns []', async () => { + const engine = makeFakeEngineWithSlugs(['projects/struktura']); + const r = makeResolver(engine); + expect(await r.resolveBasenameMatches!('')).toEqual([]); + expect(await r.resolveBasenameMatches!(' ')).toEqual([]); + }); + + test('resolveBasenameMatches: index built once, reused across calls', async () => { + const engine = makeFakeEngineWithSlugs([ + 'projects/struktura', + 'archive/struktura', + ]); + const r = makeResolver(engine); + await r.resolveBasenameMatches!('struktura'); + await r.resolveBasenameMatches!('struktura'); + await r.resolveBasenameMatches!('struktura'); + // Single getAllSlugs() call across three resolveBasenameMatches calls. + expect((engine as any)._counts().getAllCalls).toBe(1); + }); + + test('resolveBasenameMatches: degrades gracefully when getAllSlugs missing', async () => { + // Test seam for engines that don't implement getAllSlugs (legacy / mocks). + const engine = { + async getPage() { return null; }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + } as unknown as BrainEngine; + const r = makeResolver(engine); + expect(await r.resolveBasenameMatches!('struktura')).toEqual([]); + }); + + test('resolveBasenameMatches: handles top-level slugs (no `/`)', async () => { + const engine = makeFakeEngineWithSlugs(['struktura', 'notes/struktura']); + const r = makeResolver(engine); + // Both should match because basename of `struktura` is `struktura`. + const out = await r.resolveBasenameMatches!('struktura'); + expect(out.sort()).toEqual(['notes/struktura', 'struktura']); + }); }); describe('FRONTMATTER_LINK_MAP integrity', () => {