Merge remote-tracking branch 'origin/master' into garrytan/fix-issue-1777

# Conflicts:
#	src/commands/doctor.ts
This commit is contained in:
Garry Tan
2026-06-02 21:36:50 -07:00
22 changed files with 1794 additions and 63 deletions
+66
View File
@@ -2,6 +2,70 @@
All notable changes to GBrain will be documented in this file.
## [0.42.10.0] - 2026-06-02
**Wikilinks like `[[struktura]]` that point at pages in another folder finally connect.** Until now, if you wrote `[[struktura]]` in `concepts/knowledge-graph.md` and the actual page lived at `projects/struktura.md`, GBrain silently dropped the link from its graph. Obsidian users saw a dense web of connections in their vault and a thin, broken graph inside GBrain. The issue reporter had 71 wikilinks across 20 pages — GBrain captured 12.
This release adds an **opt-in mode** that resolves bare wikilinks by basename match. Turn it on with one command and the missing edges show up.
```
gbrain config set link_resolution.global_basename true
gbrain extract links
```
`gbrain doctor` scans your brain on every run and tells you what you'd gain. If five or more bare wikilinks would resolve under the new mode, you'll see a paste-ready hint:
```
link_resolution_opportunity: 47 of 60 bare wikilinks (78%) would resolve to
24 distinct page(s) under global_basename mode.
Enable with: gbrain config set link_resolution.global_basename true
```
The mode covers every place wikilinks get resolved: `gbrain extract links` (both the filesystem-walk path that the issue reporter hit by default, and the database path that the autopilot cycle uses), and the auto-link step that runs every time an agent writes a page through `put_page`. Same flag, all three surfaces.
### How it handles ambiguity
If `[[struktura]]` matches two pages — say `projects/struktura.md` and `archive/struktura.md` — GBrain emits **one edge to each**, both tagged with a new `wikilink_basename` link type. No silent winner-takes-all, no arbitrary first-wins choice. You can audit and prune the dual-match cases via:
```
gbrain graph-query concepts/knowledge-graph --type wikilink_basename
```
This is a deliberate departure from the issue's original proposal (which suggested dropping ambiguous matches with a warning). The graph layer is supposed to capture relationships, not editorialize about them.
### Why opt-in (and why this default is safe)
The default stays off, so existing brains get zero behavior change on upgrade. The bug class this fix opens — a `[[review]]` accidentally linking to some `topics/review` page that exists for unrelated reasons — is real, and turning it on globally would change every existing user's graph in ways they didn't ask for. Doctor surfaces the opportunity; the user decides.
For brains that flip the flag on, all newly extracted edges carry a distinct `link_source = 'wikilink-resolved'` provenance tag, so you can later distinguish basename-resolved edges from your hand-written markdown links.
### What this means
If you write in the Obsidian convention (flat-namespace, globally-unique basenames, bare `[[wikilinks]]` everywhere), your GBrain graph has been wrong. Flip the flag and run extract once — the missing edges show up immediately.
If you don't use bare wikilinks, doctor will tell you so (`No bare wikilinks found`) and you can keep ignoring this knob.
Closes https://github.com/garrytan/gbrain/issues/972.
### Itemized changes
- `src/core/link-extraction.ts` — three new exports:
- `WIKILINK_BASENAME_LINK_TYPE = 'wikilink_basename'` — the edge-type constant used by every path that emits basename-resolved edges. Adapted from PR #1233's regex (which was the kernel of the resolver-side fix).
- `isGlobalBasenameEnabled(engine)` — single source of truth for the
flag read. Order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB
config `link_resolution.global_basename` → default false.
- `WIKILINK_GENERIC_RE` — generic Obsidian-shape wikilink regex for the `[[bare-name]]` pass that falls outside `DIR_PATTERN`. `EntityRef` gains an optional `needsResolution` flag tagged onto refs from this pass so downstream callers know to route them through the resolver instead of treating them as canonical slugs.
- `SlugResolver` interface gains an optional `resolveBasenameMatches(name): Promise<string[]>` method. Returns ALL slugs whose basename matches — multi-match by design. `makeResolver(engine)` implements this with a lazy-built basename → `string[]` index (single `engine.getAllSlugs()` call per resolver instance) covering raw + lowercase + slugified key variants.
- `extractPageLinks` gains two options. `opts.globalBasename` routes `needsResolution: true` refs through `resolveBasenameMatches`, emitting one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`. `opts.skipFrontmatter` replaces the pre-issue-#972 `nullResolver` pattern — callers can now pass the real resolver everywhere AND independently gate the frontmatter pass, which is necessary because the bare-wikilink path needs `resolveBasenameMatches` on the real resolver.
- `src/commands/extract.ts` — DB-source `extractLinksFromDB` and FS-source `extractLinksFromFile` / `extractLinksFromDir` / `extractLinksForSlugs` thread `globalBasename` (read once per extract run via `isGlobalBasenameEnabled`). FS-source adds two new exported helpers — `resolveBasenameMatchesFromSlugs(name, allSlugs): string[]` and `resolveSlugAll(fileDir, relTarget, allSlugs, opts): string[]`. `resolveSlug` stays single-match for back-compat; `resolveSlugAll` wraps it with the basename-fallback. Stable sort: shorter slug first, then lexical.
- `src/core/operations.ts:put_page` — auto-link path now reads the flag and passes `globalBasename` through to `extractPageLinks`. Newly-written pages get basename edges when the flag is on.
- `src/commands/doctor.ts` — new `link_resolution_opportunity` check. Full scan with a 60s budget, heartbeat-reported. Severity: skipped when flag already on; ok when no bare wikilinks; ok with explanation when bare wikilinks present but none match; **warn with paste-ready hint when ≥5 would resolve AND ≥20% of bare wikilinks have matches**. Threshold tuned to surface real opportunities without flagging brains that only have a handful of unresolvable refs. Wired into both `buildChecks` (local doctor) and `doctorReportRemote` (thin-client) for parity.
- Migration v113 (`links_link_source_widen_for_wikilink_basename`) widens the `links_link_source_check` constraint to accept `'wikilink-resolved'` as a valid `link_source`. Idempotent — DROP IF EXISTS + ADD CONSTRAINT.
- `src/schema.sql` + regenerated `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` carry the widened constraint for fresh installs so first-init brains start with the right shape.
- New CHECK constraint value adds `'wikilink-resolved'` alongside `'markdown'`, `'frontmatter'`, `'manual'`.
- `KNOWN_CONFIG_KEYS` (in `src/core/config.ts`) adds `'link_resolution'` and `'link_resolution.global_basename'` so `gbrain config set ...` accepts the new key without `--force`.
- Tests: 38 new cases pinning the contract. `test/link-extraction.test.ts` adds 17 cases covering `WIKILINK_GENERIC_RE` shape (anchor / display / strip / escape paths), the `extractEntityRefs` pass-2c no-double-emit invariant, `resolveBasenameMatches` multi-match + index-built-once + missing-`getAllSlugs` degradation, and the `extractPageLinks` opt routing under both flag states. `test/extract-fs.test.ts` adds 11 cases for the pure-function helpers (`resolveBasenameMatchesFromSlugs`, `resolveSlugAll`) plus 3 round-trip tests of the issue's exact repro inside a PGLite brain. `test/doctor.test.ts` adds 7 cases for the new doctor check (skip / ok / warn paths + the cross-surface wiring source-grep). `test/e2e/global-basename-pglite.test.ts` adds 7 end-to-end cases against an in-memory PGLite brain covering FS-source, DB-source, and put_page auto-link paths under both flag states.
- PR #1233 from @rayers contributed the kernel of the resolver-side approach (the generic wikilink regex + slug-tail index pattern). This PR keeps that mechanism, makes it opt-in via the new config flag, replaces the first-write-wins lookup with multi-match return, and extends the coverage to the FS-source path that the issue's repro actually hits.
## [0.42.8.0] - 2026-06-01
**Scraped junk stops landing in your brain as if it were real content, and when something looks off, your agent gets told instead of being left to guess.**
@@ -1535,6 +1599,7 @@ behavior for an automated caller, set `gbrain config set search.mcp_keyword_only
- A page represented by its weakest chunk in vector search (the incident).
- Same-slug pages in different sources collapsing in per-page pooling and the
alias hop (source-isolation).
## [0.41.33.0] - 2026-05-29
**`gbrain search` and `query` can now return a tight, intent-sized set of
@@ -1775,6 +1840,7 @@ doctor` warns about a partial migration:
`test/doctor.test.ts` (T1 untracked-folders headline bug, T2 remote-never-shells-out trust
boundary), `test/sync-all-parallel.test.ts` (column-path staleness).
- **CI tooling**`scripts/ship-remote-tests.sh` + `workflow_dispatch` on `test.yml`.
## [0.41.31.0] - 2026-05-30
**Your nightly `gbrain sync --all` cron stops getting blocked. It used to
+2 -1
View File
@@ -221,7 +221,8 @@ strict behavior when unset.
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **v0.42.10.0 (issue #972):** opt-in global-basename wikilink resolution. New `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in extractEntityRefs); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise<string[]>`. The ONE shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical) — used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})`**source-scoped** so a bare `[[name]]` never resolves to a same-tail page in a DIFFERENT source (no cross-source edges; #972 is across-folders, not cross-source federation). `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug` — the TARGET, not the display alias — emits one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the pre-#972 `nullResolver` ternary). All three surfaces tag provenance: FS edges carry `link_source='wikilink-resolved'` via the new `ExtractedLink.link_source` field; `put_page` auto-link includes `wikilink-resolved` in its reconcilable-edge set so stale basename edges are removed when the wikilink (or the flag) goes away. A wikilink inside a markdown-link label (`[see [[acme]]](path)`) is masked out of pass 2c; the FS path strips code fences before scanning. New exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). Multi-match by design. The mode is opt-in (default false); `gbrain doctor`'s `link_resolution_opportunity` check (bounded scan: 1000 most-recent pages in one query, not a per-page walk) surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% ratio. Migration v113 widens the `links_link_source_check` constraint to admit the `'wikilink-resolved'` provenance value (full union with master's `'mentions'`). Pinned by `test/link-extraction.test.ts` + `test/extract-fs.test.ts` + `test/doctor.test.ts` + `test/e2e/global-basename-pglite.test.ts` (incl. the issue's exact `mkdir -p /tmp/vault/{concepts,projects}` repro, alias-target resolution, cross-source non-leak, put_page stale-edge removal). PR #1233 from @rayers contributed the regex shape + slug-tail index pattern; PR #1388 from @ukd1 (Russell Smith) made it opt-in + multi-match + FS coverage; the v0.42.10.0 wave added the codex source-correctness fixes (alias-by-target, stale-edge reconciliation, source-scoping) + the shared matcher.
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`.
- `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`.
- `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`.
- `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine <list|clear|scan>` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear <slug> [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`.
+23
View File
@@ -161,6 +161,29 @@ After this step:
If a user has a very large brain (>10K pages), `extract --source db` is idempotent
and supports `--since YYYY-MM-DD` for incremental runs.
### Obsidian-style bare wikilinks (opt-in)
If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]`
wikilinks — where `[[struktura]]` written in one folder means the page that lives
at `projects/struktura.md` in another — GBrain does NOT connect those by default.
Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`,
so a vault full of bare links shows up as a thin, broken graph. Turn on basename
resolution so the cross-folder links connect:
```bash
gbrain config set link_resolution.global_basename true
gbrain extract links --source db # re-run so the new edges land
```
`gbrain doctor` surfaces a `link_resolution_opportunity` hint with the exact count
("47 of 60 bare wikilinks would resolve") so you know whether it's worth enabling
before you flip it. When a bare name matches more than one page (`[[struktura]]`
both `projects/struktura` and `archive/struktura`), GBrain emits one edge to each
rather than guessing a winner — review and prune the duplicates with
`gbrain graph-query <slug>`. The mode is also honored on the filesystem-walk path
(`gbrain extract links` with no `--source db`) and by auto-link on every future
`put_page`.
## Step 5: Load Skills
If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace),
+1 -1
View File
@@ -254,7 +254,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG.
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
+1 -1
View File
@@ -1 +1 @@
0.42.8.0
0.42.10.0
+26 -2
View File
@@ -363,7 +363,8 @@ strict behavior when unset.
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **v0.42.10.0 (issue #972):** opt-in global-basename wikilink resolution. New `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in extractEntityRefs); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise<string[]>`. The ONE shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical) — used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — **source-scoped** so a bare `[[name]]` never resolves to a same-tail page in a DIFFERENT source (no cross-source edges; #972 is across-folders, not cross-source federation). `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug` — the TARGET, not the display alias — emits one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the pre-#972 `nullResolver` ternary). All three surfaces tag provenance: FS edges carry `link_source='wikilink-resolved'` via the new `ExtractedLink.link_source` field; `put_page` auto-link includes `wikilink-resolved` in its reconcilable-edge set so stale basename edges are removed when the wikilink (or the flag) goes away. A wikilink inside a markdown-link label (`[see [[acme]]](path)`) is masked out of pass 2c; the FS path strips code fences before scanning. New exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). Multi-match by design. The mode is opt-in (default false); `gbrain doctor`'s `link_resolution_opportunity` check (bounded scan: 1000 most-recent pages in one query, not a per-page walk) surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% ratio. Migration v113 widens the `links_link_source_check` constraint to admit the `'wikilink-resolved'` provenance value (full union with master's `'mentions'`). Pinned by `test/link-extraction.test.ts` + `test/extract-fs.test.ts` + `test/doctor.test.ts` + `test/e2e/global-basename-pglite.test.ts` (incl. the issue's exact `mkdir -p /tmp/vault/{concepts,projects}` repro, alias-target resolution, cross-source non-leak, put_page stale-edge removal). PR #1233 from @rayers contributed the regex shape + slug-tail index pattern; PR #1388 from @ukd1 (Russell Smith) made it opt-in + multi-match + FS coverage; the v0.42.10.0 wave added the codex source-correctness fixes (alias-by-target, stale-edge reconciliation, source-scoping) + the shared matcher.
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`.
- `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`.
- `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`.
- `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine <list|clear|scan>` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear <slug> [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`.
@@ -2396,6 +2397,29 @@ After this step:
If a user has a very large brain (>10K pages), `extract --source db` is idempotent
and supports `--since YYYY-MM-DD` for incremental runs.
### Obsidian-style bare wikilinks (opt-in)
If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]`
wikilinks — where `[[struktura]]` written in one folder means the page that lives
at `projects/struktura.md` in another — GBrain does NOT connect those by default.
Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`,
so a vault full of bare links shows up as a thin, broken graph. Turn on basename
resolution so the cross-folder links connect:
```bash
gbrain config set link_resolution.global_basename true
gbrain extract links --source db # re-run so the new edges land
```
`gbrain doctor` surfaces a `link_resolution_opportunity` hint with the exact count
("47 of 60 bare wikilinks would resolve") so you know whether it's worth enabling
before you flip it. When a bare name matches more than one page (`[[struktura]]` →
both `projects/struktura` and `archive/struktura`), GBrain emits one edge to each
rather than guessing a winner — review and prune the duplicates with
`gbrain graph-query <slug>`. The mode is also honored on the filesystem-walk path
(`gbrain extract links` with no `--source db`) and by auto-link on every future
`put_page`.
## Step 5: Load Skills
If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace),
@@ -2967,7 +2991,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG.
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
+1 -1
View File
@@ -142,5 +142,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.8.0"
"version": "0.42.10.0"
}
+9 -8
View File
@@ -255,12 +255,13 @@ export const INLINE_TIPS = [
"`gbrain upgrade` runs post-upgrade + apply-migrations.",
];
// Target ~750KB so llms-full.txt fits in ~190k-token contexts with room to spare.
// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB once CLAUDE.md crossed 700KB:
// it's ~540KB (77% of the bundle) and grows ~5-15KB per release with each feature's
// Key Files annotation. Both master (v0.41.34-38 waves) and this branch (skillopt
// wave) independently hit the 700KB line and bumped to the same 750KB. CLAUDE.md is
// the whole point of the one-fetch bundle, so it stays inlined; the budget tracks
// its legitimate growth. Still fits comfortably in 200k+ context models.
// Target ~800KB so llms-full.txt fits in ~200k-token contexts with room to spare.
// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB once CLAUDE.md crossed 700KB,
// then 750KB→800KB in v0.42.10.0 when the #972 global-basename Key Files annotation
// (landing alongside master's #1696/#1699 waves) crossed the 750KB line. CLAUDE.md
// is ~540KB+ (the bulk of the bundle) and grows ~5-15KB per release with each
// feature's Key Files annotation. CLAUDE.md is the whole point of the one-fetch
// bundle, so it stays inlined; the budget tracks its legitimate growth. Still fits
// comfortably in 200k+ context models.
// Generator prints a WARN if exceeded; ship with includeInFull=false exclusions.
export const FULL_SIZE_BUDGET = 750_000;
export const FULL_SIZE_BUDGET = 800_000;
+137 -14
View File
@@ -27,6 +27,12 @@ import { gbrainPath } from '../core/config.ts';
import { dirname, isAbsolute, join, resolve as resolvePath } from 'path';
import { fileURLToPath } from 'url';
import { existsSync, readFileSync, readdirSync, statSync } from 'fs';
import {
extractEntityRefs,
isGlobalBasenameEnabled,
buildBasenameIndex,
queryBasenameIndex,
} from '../core/link-extraction.ts';
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
// v0.41.32.0: remote staleness reads the stored newest_content_at column via
// this pure comparator (no git subprocess on the HTTP MCP doctor path).
@@ -739,6 +745,11 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
// remote/thin-client path.
checks.push(await checkHiddenBySearchPolicy(engine));
// 11a. issue #972 link_resolution_opportunity — same check the local
// doctor runs at the equivalent slot in buildChecks. Mirrored for
// thin-client parity so `gbrain remote doctor` sees the same hint.
checks.push(await checkLinkResolutionOpportunity(engine));
// 12. v0.40.5.0 Federated Sync v2 (T12) — federation_health:
// - Per-source lag, embed coverage, failed-job rate.
// - Single-source brain short-circuits to ok.
@@ -855,19 +866,16 @@ export async function checkContextualRetrievalCoverage(engine: BrainEngine): Pro
* modality + non-null embedding, so `EXISTS (content_chunks)` over-includes
* image-only / non-text pages. Tightening to the exact per-modality predicate
* would couple this check to search internals for a number nobody paginates on;
* the superset is the right operator signal (which subtrees are withheld, how
* big). The message says "chunked page(s)", never "searchable".
* the superset is the right operator signal. The message says "chunked page(s)".
*
* Status (CV-1a): pages hidden ONLY under DEFAULT excludes `ok` (those are
* intentional noise; warning would make every healthy brain look unhealthy).
* Pages hidden under a NON-default (env-supplied) prefix `warn` (the operator
* added that exclude; surfacing it is the signal they want). The message is
* agent-prescriptive: the consumer can move content out of the excluded prefix
* or pass `include_slug_prefixes` on the query.
* Status (CV-1a): pages hidden ONLY under DEFAULT excludes `ok` (intentional
* noise; warning would make every healthy brain look unhealthy). Pages hidden
* under a NON-default (env-supplied) prefix `warn`. The message is
* agent-prescriptive: move content out of the excluded prefix or pass
* `include_slug_prefixes` on the query.
*
* NOTE: this does NOT verify `archive/` pages are embedded/graphed after the
* #1777 fix `archive/` is no longer excluded, so it never appears here. Archive
* findability is verified by the search tests + `gbrain search` manually.
* #1777 fix `archive/` is no longer excluded, so it never appears here.
*/
export async function checkHiddenBySearchPolicy(engine: BrainEngine): Promise<Check> {
const name = 'hidden_by_search_policy';
@@ -879,7 +887,7 @@ export async function checkHiddenBySearchPolicy(engine: BrainEngine): Promise<Ch
// ONE query: COUNT(DISTINCT p.id) per prefix in a single pass. Prefixes are
// bound params, LIKE-escaped (env-supplied prefixes may contain %/_/\) with
// an explicit ESCAPE clause. The candidate gate is EXISTS(content_chunks);
// an explicit ESCAPE clause. Candidate gate is EXISTS(content_chunks);
// buildVisibilityClause mirrors search's page-level visibility (soft-delete,
// archived source, quarantine) and REQUIRES the `sources s` join.
const visibility = buildVisibilityClause('p', 's');
@@ -914,8 +922,6 @@ export async function checkHiddenBySearchPolicy(engine: BrainEngine): Promise<Ch
const counts: Record<string, number> = {};
for (const e of perPrefix) counts[e.prefix] = e.count;
const breakdown = perPrefix.map((e) => `${e.count} under '${e.prefix}'`).join(', ');
// Non-default (env/caller-supplied) prefix hiding pages → warn; the operator
// added that exclude on purpose and wants to see its effect.
const hasNonDefault = perPrefix.some((e) => !e.isDefault);
const guidance =
'If any hold content you want findable, move them out of the excluded ' +
@@ -935,6 +941,116 @@ export async function checkHiddenBySearchPolicy(engine: BrainEngine): Promise<Ch
}
}
/**
* Issue #972 link_resolution_opportunity check.
*
* Walks every page in the brain, scans for bare wikilinks
* (`[[struktura]]` outside DIR_PATTERN) that would resolve to at least
* one page under global-basename mode, and surfaces a paste-ready
* `gbrain config set link_resolution.global_basename true` hint when
* the count is meaningful (>=5 would-resolve AND >=20% of bare
* wikilinks have matches). Skipped silently when the flag is already
* enabled (no signal to surface) or the brain is empty.
*
* Bounded scan: batch-loads the 1000 most-recent pages in one query (not a
* per-page getPage walk) with a 60s backstop. On DB error, downgrades to an
* informational `ok` so doctor never blocks on this check.
*/
export async function checkLinkResolutionOpportunity(
engine: BrainEngine,
progress?: ProgressReporter,
): Promise<Check> {
const name = 'link_resolution_opportunity';
try {
if (await isGlobalBasenameEnabled(engine)) {
return { name, status: 'ok', message: 'global_basename mode already enabled' };
}
const allSlugs = await engine.getAllSlugs();
if (allSlugs.size === 0) {
return { name, status: 'ok', message: 'Brain is empty — nothing to scan' };
}
// Build a basename → slug[] index ONCE for the entire scan via the shared
// builder (issue #972 codex [P2] DRY) — same key set (raw/lower/slugified)
// as extraction, so this estimate matches what extraction actually
// resolves. Pre-fix the doctor omitted the slugified key and undercounted.
const basenameIndex = buildBasenameIndex(allSlugs);
let bareCount = 0;
let wouldResolveCount = 0;
const distinctTargets = new Set<string>();
// Issue #972 (codex [P2] perf): batch-load the most-recent N pages in ONE
// query instead of listAllPageRefs() + a getPage() per page. The prior
// full N-page walk hit the 60s budget every run on large brains and
// returned a perpetual partial; this bounds the work to a fixed sample.
const SAMPLE_LIMIT = 1000;
const sampled = await engine.executeRaw<{ compiled_truth: string | null; timeline: string | null }>(
`SELECT compiled_truth, timeline FROM pages WHERE deleted_at IS NULL ORDER BY id DESC LIMIT ${SAMPLE_LIMIT}`,
);
const totalPages = allSlugs.size;
const sampledNote = totalPages > SAMPLE_LIMIT
? ` (scanned the ${SAMPLE_LIMIT} most-recent of ${totalPages} pages)`
: '';
const deadline = Date.now() + 60_000;
const hb = progress ? startHeartbeat(progress, `scanning ${sampled.length} pages for bare wikilinks…`) : null;
try {
for (const row of sampled) {
if (Date.now() > deadline) break; // backstop; in-memory scan rarely hits it
const content = (row.compiled_truth ?? '') + '\n' + (row.timeline ?? '');
for (const e of extractEntityRefs(content)) {
if (!e.needsResolution) continue;
bareCount++;
// Issue #972 (codex): match on the wikilink TARGET (e.slug), not
// the display alias (e.name), via the shared query so the doctor
// estimate equals what extraction actually resolves.
const matches = queryBasenameIndex(basenameIndex, e.slug);
if (matches.length > 0) {
wouldResolveCount++;
for (const m of matches) distinctTargets.add(m);
}
}
}
} finally {
hb?.();
}
if (bareCount === 0) {
return { name, status: 'ok', message: 'No bare wikilinks found' };
}
if (wouldResolveCount === 0) {
return {
name,
status: 'ok',
message: `${bareCount} bare wikilink(s) found, but none have basename matches in the brain.`,
};
}
const ratio = wouldResolveCount / bareCount;
if (wouldResolveCount >= 5 && ratio >= 0.20) {
const pct = Math.round(ratio * 100);
return {
name,
status: 'warn',
message:
`${wouldResolveCount} of ${bareCount} bare wikilinks (${pct}%) would resolve to ` +
`${distinctTargets.size} distinct page(s) under global_basename mode${sampledNote}. ` +
`Enable with: gbrain config set link_resolution.global_basename true`,
};
}
const pct = Math.round(ratio * 100);
return {
name,
status: 'ok',
message: `${wouldResolveCount}/${bareCount} bare wikilinks (${pct}%) would resolve — below the 20% / 5-link threshold for surfacing a hint${sampledNote}.`,
};
} catch (e) {
return {
name,
status: 'ok',
message: `Skipped (${e instanceof Error ? e.message : String(e)})`,
};
}
}
export async function checkAbandonedThreads(engine: BrainEngine): Promise<Check> {
try {
const rows = await engine.executeRaw<{ count: number }>(
@@ -6257,7 +6373,7 @@ export async function buildChecks(
progress.heartbeat('search_mode');
checks.push(await checkSearchMode(engine));
// issue #1777 — hidden_by_search_policy: chunked pages withheld from default
// search by the hard-exclude prefix policy (search-policy adjacent).
// search by the hard-exclude prefix policy (audit the surviving excludes).
progress.heartbeat('hidden_by_search_policy');
checks.push(await checkHiddenBySearchPolicy(engine));
progress.heartbeat('eval_drift');
@@ -6276,6 +6392,13 @@ export async function buildChecks(
// v0.37.0 brainstorm_health — migration v79, track_retrieval, calibration cold-start.
progress.heartbeat('brainstorm_health');
checks.push(await checkBrainstormHealth(engine));
// issue #972 link_resolution_opportunity — full scan: count bare wikilinks
// that would resolve under global_basename mode. Surfaces a paste-ready
// enable hint when ≥5 hits AND ≥20% of bare wikilinks would resolve.
// Skipped silently when the flag is already enabled. Bounded by a 60s
// budget so a huge brain never wedges doctor on this check.
progress.heartbeat('link_resolution_opportunity');
checks.push(await checkLinkResolutionOpportunity(engine, progress));
// v0.36.0.0 (A5): ZE embedding key health + schema/config width consistency.
progress.heartbeat('ze_embedding_health');
checks.push(await checkZeEmbeddingHealth(engine));
+124 -19
View File
@@ -35,7 +35,9 @@ import type { PageType } from '../core/types.ts';
import { parseMarkdown } from '../core/markdown.ts';
import {
extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver,
extractFrontmatterLinks, LINK_EXTRACTOR_VERSION_TS,
extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS,
WIKILINK_BASENAME_LINK_TYPE,
buildBasenameIndex, queryBasenameIndex, stripCodeBlocks,
type UnresolvedFrontmatterRef, type LinkCandidate,
} from '../core/link-extraction.ts';
import { createProgress } from '../core/progress.ts';
@@ -156,6 +158,11 @@ export interface ExtractedLink {
to_slug: string;
link_type: string;
context: string;
// Issue #972: provenance for FS-source edges. Set to 'wikilink-resolved'
// on basename-matched bare wikilinks so the FS path tags them the same way
// the DB / put_page paths do. Undefined for ordinary markdown edges (the
// engine defaults those to 'markdown').
link_source?: string;
}
export interface ExtractedTimelineEntry {
@@ -273,6 +280,56 @@ export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set<st
return null;
}
/**
* Issue #972: return every slug whose basename matches `name` (the
* final path segment, with case-insensitive + slugified fallback keys).
* Pure-function variant of the resolver's `resolveBasenameMatches` that
* reads a pre-loaded Set directly — no engine call. Used by the
* FS-source path's `resolveSlugAll`.
*
* Matches are deterministically sorted (shortest-slug first, then
* lexical) so repeated runs over the same brain produce stable edges.
* Returns `[]` on empty input or no matches.
*/
export function resolveBasenameMatchesFromSlugs(
name: string, allSlugs: Set<string>,
): string[] {
// Issue #972 (codex [P2] DRY): delegate to the shared matcher so the FS
// path keys + sorts identically to the resolver and doctor. (Per-call
// index build is O(N), the same cost as the prior inline scan.)
return queryBasenameIndex(buildBasenameIndex(allSlugs), name);
}
/**
* Issue #972: multi-match variant of `resolveSlug`. Always tries the
* existing ancestor walk first (preserving the v0.10.1 behavior); on
* miss, falls back to basename lookup against `allSlugs` when
* `opts.globalBasename === true`. Returns an array so the caller emits
* one graph edge per matching page.
*
* Return shape:
* - Ancestor walk hits → `[ancestor_match]` (length 1)
* - Ancestor walk misses + globalBasename off → `[]`
* - Ancestor walk misses + globalBasename on + basename hits → all matches
* - Ancestor walk misses + globalBasename on + no basename hits → `[]`
*/
export function resolveSlugAll(
fileDir: string, relTarget: string, allSlugs: Set<string>,
opts: { globalBasename?: boolean } = {},
): string[] {
const direct = resolveSlug(fileDir, relTarget, allSlugs);
if (direct !== null) return [direct];
if (!opts.globalBasename) return [];
// Strip .md suffix + dirname so `[[struktura]]` (relTarget=`struktura.md`)
// and `[[notes/struktura]]` (relTarget=`notes/struktura.md`) both query
// for the basename `struktura`.
const targetNoExt = relTarget.endsWith('.md') ? relTarget.slice(0, -3) : relTarget;
const basename = targetNoExt.includes('/')
? targetNoExt.slice(targetNoExt.lastIndexOf('/') + 1)
: targetNoExt;
return resolveBasenameMatchesFromSlugs(basename, allSlugs);
}
/**
* Directory-based link-type inference for the fs-source path.
*
@@ -319,20 +376,49 @@ function parseFrontmatterFromContent(content: string, relPath: string): Record<s
*/
export async function extractLinksFromFile(
content: string, relPath: string, allSlugs: Set<string>,
opts?: { includeFrontmatter?: boolean },
opts?: { includeFrontmatter?: boolean; globalBasename?: boolean },
): Promise<ExtractedLink[]> {
const links: ExtractedLink[] = [];
const slug = pathToSlug(relPath);
const fileDir = dirname(relPath);
const fm = parseFrontmatterFromContent(content, relPath);
// Issue #972: globalBasename routes bare `[[name]]` wikilinks through
// basename lookup against allSlugs when the ancestor walk fails. Off
// by default for back-compat with the v0.10.1 ancestor-only behavior.
const globalBasename = opts?.globalBasename ?? false;
for (const { name, relTarget } of extractMarkdownLinks(content)) {
const resolved = resolveSlug(fileDir, relTarget, allSlugs);
if (resolved !== null) {
// Issue #972 (codex [P2]): strip code fences before scanning so a
// `[[name]]` inside a code block doesn't create an FS edge. Mirrors the
// DB path, which goes through extractEntityRefs (which strips internally).
const scanContent = stripCodeBlocks(content);
for (const { name, relTarget } of extractMarkdownLinks(scanContent)) {
const resolvedSlugs = resolveSlugAll(fileDir, relTarget, allSlugs, { globalBasename });
if (resolvedSlugs.length === 0) continue;
// Single hit on the ancestor path → emit one edge with the inferred
// verb type. Multiple hits (only possible when globalBasename is on
// AND ancestor walk missed) → emit one edge per match, all tagged
// `wikilink_basename` so users can audit via `gbrain graph-query
// <slug> --type wikilink_basename`.
const isBasename = resolvedSlugs.length > 1
|| (globalBasename && resolvedSlugs.length === 1
&& resolveSlug(fileDir, relTarget, allSlugs) === null);
for (const target of resolvedSlugs) {
// Issue #972 (codex [P2]): drop a basename self-loop ([[own-tail]] on
// its own page resolving back to itself).
if (isBasename && target === slug) continue;
links.push({
from_slug: slug, to_slug: resolved,
link_type: inferTypeByDir(fileDir, dirname(resolved), fm),
context: `markdown link: [${name}]`,
from_slug: slug,
to_slug: target,
link_type: isBasename
? WIKILINK_BASENAME_LINK_TYPE
: inferTypeByDir(fileDir, dirname(target), fm),
context: isBasename
? `wikilink (basename match): [${name}]`
: `markdown link: [${name}]`,
// Issue #972: tag basename edges so the FS path matches DB/put_page
// provenance and migration v112's widened CHECK is exercised here too.
link_source: isBasename ? 'wikilink-resolved' : undefined,
});
}
}
@@ -860,6 +946,9 @@ async function extractForSlugs(
let timelineCreated = 0;
let pagesProcessed = 0;
// Issue #972: read the basename flag once per extract run.
const globalBasename = await isGlobalBasenameEnabled(engine);
const linkBatch: LinkBatchInput[] = [];
const timelineBatch: TimelineBatchInput[] = [];
@@ -912,7 +1001,7 @@ async function extractForSlugs(
const content = readFileSync(fullPath, 'utf-8');
if (doLinks) {
const links = await extractLinksFromFile(content, relPath, allSlugs);
const links = await extractLinksFromFile(content, relPath, allSlugs, { globalBasename });
for (const link of links) {
if (dryRun) {
if (!jsonMode) console.log(` ${link.from_slug}${link.to_slug} (${link.link_type})`);
@@ -963,6 +1052,11 @@ async function extractLinksFromDir(
const files = walkMarkdownFiles(brainDir);
const allSlugs = new Set(files.map(f => pathToSlug(f.relPath)));
// Issue #972: read once before the walk so the per-file calls don't
// re-query the DB. globalBasename = true emits one edge per basename
// match for bare wikilinks like `[[struktura]]`.
const globalBasename = await isGlobalBasenameEnabled(engine);
// Progress stream on stderr (separate from the action-events --json writes
// to stdout, which tests grep for). Rate-gated; respects global --quiet /
// --progress-json flags.
@@ -998,7 +1092,7 @@ async function extractLinksFromDir(
onItem: async (file) => {
try {
const content = readFileSync(file.path, 'utf-8');
const links = await extractLinksFromFile(content, file.relPath, allSlugs);
const links = await extractLinksFromFile(content, file.relPath, allSlugs, { globalBasename });
for (const link of links) {
if (dryRunSeen) {
const key = `${link.from_slug}::${link.to_slug}::${link.link_type}`;
@@ -1107,14 +1201,16 @@ export async function extractLinksForSlugs(
const linkOpts = opts?.sourceId
? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId, originSourceId: opts.sourceId }
: undefined;
// Issue #972: same flag as the standalone extract path.
const globalBasename = await isGlobalBasenameEnabled(engine);
let created = 0;
for (const slug of slugs) {
const filePath = join(repoPath, slug + '.md');
if (!existsSync(filePath)) continue;
try {
const content = readFileSync(filePath, 'utf-8');
for (const link of await extractLinksFromFile(content, slug + '.md', allSlugs)) {
try { await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type, undefined, undefined, undefined, linkOpts); created++; } catch { /* skip */ } // gbrain-allow-direct-insert: gbrain extract single-row fallback when batch path declines a row
for (const link of await extractLinksFromFile(content, slug + '.md', allSlugs, { globalBasename })) {
try { await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type, link.link_source, undefined, undefined, linkOpts); created++; } catch { /* skip */ } // gbrain-allow-direct-insert: gbrain extract single-row fallback when batch path declines a row
}
} catch { /* skip */ }
}
@@ -1170,12 +1266,19 @@ async function extractLinksFromDB(
// Batch resolver: pg_trgm + exact only, NO search fallback. Dodges the
// N-thousand API call trap on 46K-page brains. Resolver has a per-run
// cache so duplicate names (same person appearing on many pages) resolve
// once, not once per mention.
const resolver = makeResolver(engine, { mode: 'batch' });
// once, not once per mention. Used for BOTH the frontmatter pass (gated
// by `includeFrontmatter` via `opts.skipFrontmatter` on extractPageLinks)
// AND the issue-#972 global-basename pass (gated by `globalBasename`).
// Replaces the pre-issue-#972 `nullResolver` ternary — that synthetic
// resolver lacked `resolveBasenameMatches`, so we always pass the real
// one and let extractPageLinks's opts gate which pass actually runs.
// Issue #972 (codex [P1]): scope basename resolution to the source being
// extracted so bare wikilinks don't resolve across unrelated sources.
const resolver = makeResolver(engine, { mode: 'batch', sourceId: sourceIdFilter });
const unresolved: UnresolvedFrontmatterRef[] = [];
const nullResolver = {
resolve: async () => null as string | null,
};
// Issue #972: opt-in global-basename wikilink resolution. Read once
// per extract run; threaded into each extractPageLinks call.
const globalBasename = await isGlobalBasenameEnabled(engine);
// v0.32.8: listAllPageRefs enumerates (slug, source_id) so we can thread
// sourceId to getPage AND build a cross-source resolution map for link
// disambiguation. Pre-fix used getAllSlugs() which collapsed
@@ -1252,9 +1355,11 @@ async function extractLinksFromDB(
// --include-frontmatter default OFF in v0.13 (codex tension 5, back-compat).
// Migration orchestrator explicitly enables it for the one-time backfill;
// user-invoked `gbrain extract links` stays outgoing-only.
const activeResolver = includeFrontmatter ? resolver : nullResolver;
// Issue #972: globalBasename routes bare `[[name]]` wikilinks through
// basename lookup; off by default for back-compat.
const extracted = await extractPageLinks(
slug, fullContent, page.frontmatter, page.type, activeResolver,
slug, fullContent, page.frontmatter, page.type, resolver,
{ skipFrontmatter: !includeFrontmatter, globalBasename },
);
unresolved.push(...extracted.unresolved);
+3
View File
@@ -736,6 +736,9 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
// Misc
'artifacts_sync_mode',
'cross_project_learnings',
// Link resolution (issue #972)
'link_resolution',
'link_resolution.global_basename',
];
/**
+1
View File
@@ -86,6 +86,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'image_assets',
'integrity',
'jsonb_integrity',
'link_resolution_opportunity',
'links_extraction_lag',
'markdown_body_completeness',
'nightly_quality_probe_health',
+273 -5
View File
@@ -46,8 +46,32 @@ export interface EntityRef {
* - sourceId null → 'unqualified'
*/
sourceId?: string | null;
/**
* Issue #972: set when the ref came from the generic `[[bare-name]]`
* pass (not gated by DIR_PATTERN). The `slug` field holds the literal
* wikilink text — callers MUST run it through a SlugResolver's
* `resolveBasenameMatches` (gated by `link_resolution.global_basename`)
* before persisting. Untagged refs already match a known entity dir
* (people/, companies/, etc.) and need no resolution.
*
* One ref tagged `needsResolution: true` may resolve to MULTIPLE target
* slugs when multiple pages share the basename. The caller emits one
* LinkCandidate per resolved match.
*/
needsResolution?: boolean;
}
/**
* Issue #972: edge type stamped on graph rows produced by the
* global-basename wikilink resolution path. Distinct from the verb-
* inferred types (`mentions`, `works_at`, etc.) so users can audit or
* prune via `gbrain graph-query <slug> --type wikilink_basename`.
*
* All edges from this path also carry `linkSource: 'wikilink-resolved'`
* (the link-source provenance is the why; this edge type is the what).
*/
export const WIKILINK_BASENAME_LINK_TYPE = 'wikilink_basename';
/** v0.17.0: how a link's target source was pinned at extraction time. */
export type LinkResolutionType = 'qualified' | 'unqualified';
@@ -105,6 +129,38 @@ const QUALIFIED_WIKILINK_RE = new RegExp(
'g',
);
/**
* Issue #972: generic Obsidian-style wikilink — matches `[[anything]]`
* with no DIR_PATTERN gate. Wiki/topic/learning pages frequently link
* via `[[bare-name]]` (`[[Fast-Weigh]]`, `[[2026-05-07-cost-plan]]`,
* `[[struktura]]`) where the literal text isn't a canonical slug; the
* resolver is responsible for matching it to the real page via
* basename uniqueness.
*
* Used AFTER QUALIFIED_WIKILINK_RE and WIKILINK_RE pick off their
* cases — the masked-ranges pass in extractEntityRefs prevents
* double-emission. Refs from this pass are tagged
* `needsResolution: true` so callers know to run them through a
* SlugResolver before the page-existence check.
*
* Regex shape is the union of the two existing wikilink regexes,
* minus the DIR_PATTERN gate. The character class disallows pipe,
* close-bracket, hash (anchor separator), newline, and open-bracket
* so we never match across line breaks or capture nested `[[...]]`.
*
* Adapted from PR #1233 (rayers).
*/
const WIKILINK_GENERIC_RE = /\[\[([^|\]#\n[]+?)(?:#[^|\]]*?)?(?:\|([^\]]+?))?\]\]/g;
/**
* Issue #972 (codex [P2]): a markdown link whose LABEL contains a wikilink,
* e.g. `[see [[acme]]](companies/acme.md)`. Pass-1's ENTITY_REF_RE can't match
* the nested `]]` so it never spans this; without masking, the generic 2c pass
* would emit a stray basename ref for the inner `[[acme]]`. Mask the whole
* span out of the 2c scan so a wikilink inside a markdown label is inert.
*/
const MARKDOWN_LABEL_WIKILINK_RE = /\[[^\]\n]*\[\[[^\]\n]+\]\][^\]\n]*\]\([^)\n]+\)/g;
/**
* Strip fenced code blocks (```...```) and inline code (`...`) from markdown,
* replacing them with whitespace of equivalent length. Preserves byte offsets
@@ -250,6 +306,7 @@ export function extractEntityRefs(content: string): EntityRef[] {
// Markdown links have no source-qualification syntax — they're
// always unqualified. Omit sourceId so the shape stays compatible
// with pre-v0.17 consumers doing strict equality.
const markdownRanges: Array<[number, number]> = [];
const mdPattern = new RegExp(ENTITY_REF_RE.source, ENTITY_REF_RE.flags);
while ((match = mdPattern.exec(stripped)) !== null) {
const name = match[1];
@@ -257,6 +314,7 @@ export function extractEntityRefs(content: string): EntityRef[] {
const slug = fullPath;
const dir = fullPath.split('/')[0];
refs.push({ name, slug, dir });
markdownRanges.push([match.index, match.index + match[0].length]);
}
// 2a. v0.17.0 qualified wikilinks: [[source-id:path]] or [[source-id:path|Display]]
@@ -278,6 +336,7 @@ export function extractEntityRefs(content: string): EntityRef[] {
// 2b. Unqualified Obsidian wikilinks: [[path]] or [[path|Display Text]]
// Same shape rule: omit sourceId when unqualified.
const unqualifiedRanges: Array<[number, number]> = [];
const unmasked = maskRanges(stripped, qualifiedRanges);
const wikiPattern = new RegExp(WIKILINK_RE.source, WIKILINK_RE.flags);
while ((match = wikiPattern.exec(unmasked)) !== null) {
@@ -288,6 +347,38 @@ export function extractEntityRefs(content: string): EntityRef[] {
const displayName = (match[2] || slug).trim();
const dir = slug.split('/')[0];
refs.push({ name: displayName, slug, dir });
unqualifiedRanges.push([match.index, match.index + match[0].length]);
}
// 2c. Issue #972: generic `[[bare-name]]` wikilinks not gated by
// DIR_PATTERN. Wiki/topic/learning content frequently uses these
// where the bare text isn't a canonical brain slug. Tagged
// `needsResolution: true` so the caller routes them through a
// SlugResolver's `resolveBasenameMatches` before persisting.
// Mask out 2a/2b ranges so we don't double-emit; skip qualified-
// syntax tokens (contain `:`) that would be 2a's job. Issue #972
// (codex [P2]): ALSO mask pass-1 markdown-link ranges so a wikilink
// inside a markdown label — `[see [[acme]]](companies/acme.md)` —
// doesn't spawn a stray generic basename ref from inside the label.
const labelWikilinkRanges: Array<[number, number]> = [];
const labelWlPattern = new RegExp(MARKDOWN_LABEL_WIKILINK_RE.source, MARKDOWN_LABEL_WIKILINK_RE.flags);
while ((match = labelWlPattern.exec(stripped)) !== null) {
labelWikilinkRanges.push([match.index, match.index + match[0].length]);
}
const genericMasked = maskRanges(
stripped,
[...markdownRanges, ...qualifiedRanges, ...unqualifiedRanges, ...labelWikilinkRanges],
);
const genericPattern = new RegExp(WIKILINK_GENERIC_RE.source, WIKILINK_GENERIC_RE.flags);
while ((match = genericPattern.exec(genericMasked)) !== null) {
let slug = match[1].trim();
if (!slug) continue;
if (slug.includes('://')) continue;
if (slug.includes(':')) continue; // qualified-syntax token; 2a owns these
if (slug.endsWith('.md')) slug = slug.slice(0, -3);
const displayName = (match[2] || slug).trim();
const dir = slug.includes('/') ? slug.split('/')[0] : '';
refs.push({ name: displayName, slug, dir, needsResolution: true });
}
return refs;
@@ -376,11 +467,44 @@ export async function extractPageLinks(
frontmatter: Record<string, unknown>,
pageType: PageType,
resolver: SlugResolver,
opts: { globalBasename?: boolean; skipFrontmatter?: boolean } = {},
): Promise<PageLinksResult> {
const candidates: LinkCandidate[] = [];
// 1. Markdown entity refs.
for (const ref of extractEntityRefs(content)) {
// Issue #972: refs from the generic `[[bare-name]]` pass carry the
// literal wikilink text, not a real page slug. When global_basename
// mode is on AND the resolver implements basename lookup, resolve
// to every matching page and emit one candidate per match. When the
// flag is off (default), drop silently — back-compat with the
// pre-v0.40.8.2 behavior of dropping bare wikilinks outside
// DIR_PATTERN.
if (ref.needsResolution) {
if (!opts.globalBasename || typeof resolver.resolveBasenameMatches !== 'function') {
continue;
}
// Issue #972 (codex): resolve by the wikilink TARGET (ref.slug — the
// text inside `[[...]]` before any `|`), NOT the display alias
// (ref.name = match[2]). `[[struktura|the project]]` must resolve
// `struktura`, not "the project". The display text is for context only.
const matches = await resolver.resolveBasenameMatches(ref.slug);
if (matches.length === 0) continue;
const idx = content.indexOf(ref.slug);
const context = idx >= 0 ? excerpt(content, idx, 240) : ref.name;
for (const matched of matches) {
// Issue #972 (codex [P2]): a basename `[[own-tail]]` on its own page
// resolves back to itself — drop the self-loop.
if (matched === slug) continue;
candidates.push({
targetSlug: matched,
linkType: WIKILINK_BASENAME_LINK_TYPE,
context,
linkSource: 'wikilink-resolved',
});
}
continue;
}
const idx = content.indexOf(ref.name);
// Wider context window (240 chars vs original 80) catches verbs that
// appear at sentence-or-paragraph distance from the slug — common in
@@ -418,12 +542,28 @@ export async function extractPageLinks(
}
// 3. Frontmatter-derived edges (v0.13). Includes the legacy `source:`
// field along with the full field map.
const fm = await extractFrontmatterLinks(slug, pageType, frontmatter, resolver);
candidates.push(...fm.candidates);
// field along with the full field map. Caller can suppress via
// `opts.skipFrontmatter` — used by `extractLinksFromDB` to keep
// `--include-frontmatter` off semantics while still passing a real
// resolver (needed for `resolveBasenameMatches` in global-basename
// mode). Pre-issue-#972 this gating lived in the caller via a
// synthetic `nullResolver`; that pattern broke once the bare-wikilink
// path needed `resolveBasenameMatches` on the real resolver.
let fmUnresolved: UnresolvedFrontmatterRef[] = [];
if (!opts.skipFrontmatter) {
const fm = await extractFrontmatterLinks(slug, pageType, frontmatter, resolver);
candidates.push(...fm.candidates);
fmUnresolved = fm.unresolved;
}
// Within-page dedup: same (fromSlug, targetSlug, linkType, linkSource)
// collapses to one entry. First occurrence wins.
// Issue #972 (codex P2d, decided): a qualified `[[companies/acme]]` (typed
// markdown edge) and a bare `[[acme]]` (wikilink-resolved edge) to the SAME
// target are KEPT as separate rows — they carry different provenance
// (link_source) and link_type, and the audit trail (which kind of reference
// created the edge) is worth more than collapsing them. graph-query callers
// that want a unique target set dedup on to_slug themselves.
const seen = new Set<string>();
const result: LinkCandidate[] = [];
for (const c of candidates) {
@@ -432,7 +572,7 @@ export async function extractPageLinks(
seen.add(key);
result.push(c);
}
return { candidates: result, unresolved: fm.unresolved };
return { candidates: result, unresolved: fmUnresolved };
}
/** Excerpt a window of `width` chars around `idx`, collapsed to one line. */
@@ -655,6 +795,67 @@ export interface SlugResolver {
* extract/put_page summary so the user can see the gap.
*/
resolve(name: string, dirHint?: string | string[]): Promise<string | null>;
/**
* Issue #972: return every slug whose basename (final `/`-segment, or
* the whole slug if it has no `/`) matches `name`. Multi-match by
* design — `[[struktura]]` referencing both `projects/struktura` and
* `archive/struktura` returns both; the caller emits one graph edge
* per result. Returns `[]` when no matches.
*
* Optional so existing SlugResolver consumers (e.g. synthetic
* frontmatter-only resolvers) keep working without code changes.
* Implementations should make this cheap to call repeatedly within
* a single extract run (build the index once, reuse it).
*/
resolveBasenameMatches?(name: string): Promise<string[]>;
}
/**
* Issue #972 (codex [P2] DRY): the ONE basename matcher. Before this, three
* surfaces (makeResolver, FS `resolveBasenameMatchesFromSlugs`, the doctor
* `link_resolution_opportunity` check) each hand-rolled their own key set +
* sort, and they drifted — the doctor omitted the slugified key, so its
* "N would resolve" estimate undercounted what extraction actually produces.
* All three now build/query through these two functions so they cannot drift.
*
* Keying: raw tail + lowercase tail + slugified tail. A slug's tail is its
* final `/`-segment (or the whole slug when it has no `/`).
*/
export function normalizeBasename(s: string): string {
return s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-');
}
/** Stable order: shorter slug first (likely closer to brain root), then lexical. */
function basenameSort(a: string, b: string): number {
return (a.length - b.length) || a.localeCompare(b);
}
/** Build a `key → slug[]` index over a slug collection. Keys: raw/lower/slugified tail. */
export function buildBasenameIndex(slugs: Iterable<string>): Map<string, string[]> {
const idx = new Map<string, string[]>();
const addKey = (key: string, slug: string) => {
const existing = idx.get(key);
if (existing) { if (!existing.includes(slug)) existing.push(slug); }
else idx.set(key, [slug]);
};
for (const slug of slugs) {
const tail = slug.includes('/') ? slug.slice(slug.lastIndexOf('/') + 1) : slug;
addKey(tail, slug);
const lower = tail.toLowerCase();
if (lower !== tail) addKey(lower, slug);
const slugified = normalizeBasename(tail);
if (slugified && slugified !== tail && slugified !== lower) addKey(slugified, slug);
}
return idx;
}
/** Look a name up in a basename index (raw → lower → slugified), stable-sorted. */
export function queryBasenameIndex(idx: Map<string, string[]>, name: string): string[] {
if (!name || typeof name !== 'string') return [];
const trimmed = name.trim();
if (!trimmed) return [];
const hit = idx.get(trimmed) ?? idx.get(trimmed.toLowerCase()) ?? idx.get(normalizeBasename(trimmed));
return hit ? [...hit].sort(basenameSort) : [];
}
/**
@@ -674,13 +875,52 @@ export interface SlugResolver {
*/
export function makeResolver(
engine: BrainEngine,
opts: { mode: 'batch' | 'live' } = { mode: 'live' },
opts: { mode: 'batch' | 'live'; sourceId?: string } = { mode: 'live' },
): SlugResolver {
const cache = new Map<string, string | null>();
const norm = (s: string) => s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-');
// Issue #972: lazy-built basename → slug[] index for global-basename
// resolution. Built on first call to `resolveBasenameMatches`; reused
// for the rest of the resolver instance's lifetime. Cost is bounded
// (one `engine.getAllSlugs()` call per extract run / put_page). The
// index keys BOTH the raw tail and the slugified-tail so a wikilink
// `[[Fast-Weigh]]` matches a page slugged `companies/fast-weigh` as
// well as `notes/Fast-Weigh`.
let basenameIndex: Map<string, string[]> | null = null;
async function ensureBasenameIndex(): Promise<Map<string, string[]>> {
if (basenameIndex !== null) return basenameIndex;
const idx = new Map<string, string[]>();
if (typeof engine.getAllSlugs !== 'function') {
basenameIndex = idx;
return idx;
}
try {
// Issue #972 (codex [P1]): scope the basename index to the resolver's
// source. getAllSlugs({sourceId}) keeps wikilink resolution from
// spanning unrelated sources — a bare [[name]] must NOT resolve to a
// same-tail page in a DIFFERENT source and create a cross-source edge.
// #972 is "global basename across folders," not "cross-source federation."
const all = await engine.getAllSlugs(opts.sourceId ? { sourceId: opts.sourceId } : undefined);
// Issue #972 (codex [P2] DRY): one shared index builder for all surfaces.
basenameIndex = buildBasenameIndex(all);
return basenameIndex;
} catch {
// Index build failed — empty map → resolveBasenameMatches finds
// nothing, resolver continues as if the flag was off. Never throw.
}
basenameIndex = idx;
return idx;
}
return {
async resolveBasenameMatches(name: string): Promise<string[]> {
// Issue #972 (codex [P2] DRY): shared query so resolver + FS + doctor
// return the same matches in the same stable order.
return queryBasenameIndex(await ensureBasenameIndex(), name);
},
async resolve(name: string, dirHint?: string | string[]): Promise<string | null> {
if (!name || typeof name !== 'string') return null;
const trimmed = name.trim();
@@ -948,3 +1188,31 @@ export async function isAutoTimelineEnabled(engine: BrainEngine): Promise<boolea
const normalized = val.trim().toLowerCase();
return !['false', '0', 'no', 'off'].includes(normalized);
}
/**
* Read the `link_resolution.global_basename` config flag. Defaults to
* FALSE (opt-in only; existing brains keep ancestor-walk resolution).
*
* When TRUE: bare wikilinks like `[[struktura]]` (no DIR_PATTERN prefix)
* that don't resolve via the existing path are matched against every
* page's basename. Each unique match emits a graph edge tagged
* `linkType: 'wikilink_basename'`. Multiple matches emit multiple edges.
*
* Resolution order (highest → lowest):
* 1. Env var `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME=1` (operator override)
* 2. DB plane via `engine.getConfig('link_resolution.global_basename')`
* 3. Default false
*
* Closes https://github.com/garrytan/gbrain/issues/972.
*/
export async function isGlobalBasenameEnabled(engine: BrainEngine): Promise<boolean> {
const envVal = process.env.GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME;
if (envVal != null) {
const normalized = envVal.trim().toLowerCase();
return ['1', 'true', 'yes', 'on'].includes(normalized);
}
const val = await engine.getConfig('link_resolution.global_basename');
if (val == null) return false;
const normalized = val.trim().toLowerCase();
return ['1', 'true', 'yes', 'on'].includes(normalized);
}
+32
View File
@@ -5088,6 +5088,38 @@ export const MIGRATIONS: Migration[] = [
}
},
},
{
version: 113,
name: 'links_link_source_widen_for_wikilink_basename',
// Issue #972: opt-in global-basename wikilink resolution (bare [[name]]
// resolved by slug tail) emits edges tagged
// `link_source = 'wikilink-resolved'`. Widen the CHECK to admit it.
//
// The FULL set is enumerated here — not just the new value — because
// DROP + re-ADD replaces the whole constraint. v95
// (links_link_source_check_includes_mentions) added 'mentions'; since
// this migration runs AFTER v95, omitting 'mentions' would silently
// clobber that widening. Keep both branches in sync with src/schema.sql
// and src/core/pglite-schema.ts.
//
// Renumbered v93 → v109 → v110 → v112 → v113 across successive master
// merges (upstream claimed through v112 — pages_links_extracted_at — in
// the interim). Idempotent via DROP ... IF EXISTS, so it no-ops on
// installs that never created the constraint.
idempotent: true,
sql: `
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check;
ALTER TABLE links ADD CONSTRAINT links_link_source_check
CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved'));
`,
sqlFor: {
pglite: `
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check;
ALTER TABLE links ADD CONSTRAINT links_link_source_check
CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved'));
`,
},
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+16 -4
View File
@@ -16,7 +16,7 @@ import { expandQuery } from './search/expansion.ts';
import { dedupResults } from './search/dedup.ts';
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
import type { HybridSearchMeta } from './types.ts';
import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, isGlobalBasenameEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
import { isFactsBackstopEligible } from './facts/eligibility.ts';
import { stripTakesFence } from './takes-fence.ts';
import { stripFactsFence } from './facts-fence.ts';
@@ -991,9 +991,14 @@ async function runAutoLink(
: {};
// Live-mode resolver: per-put throwaway cache, pg_trgm + optional search.
const resolver = makeResolver(engine, { mode: 'live' });
// Issue #972 (codex [P1]): pass sourceId so basename resolution stays
// within this page's source — no cross-source basename edges.
const resolver = makeResolver(engine, { mode: 'live', sourceId: opts?.sourceId });
// Issue #972: opt-in bare-wikilink basename resolution. Off by default.
const globalBasename = await isGlobalBasenameEnabled(engine);
const { candidates, unresolved } = await extractPageLinks(
slug, fullContent, parsed.frontmatter, parsed.type, resolver,
{ globalBasename },
);
// Resolve which targets exist (skip refs to non-existent pages to avoid FK
@@ -1040,10 +1045,17 @@ async function runAutoLink(
l => l.link_source === 'frontmatter' && l.origin_slug === slug,
);
// Reconcilable outgoing edges: markdown + our own frontmatter edges.
// Manual edges (link_source='manual') are NEVER touched by reconciliation.
// Reconcilable outgoing edges: markdown + our own frontmatter edges +
// basename-resolved wikilinks (issue #972). Manual edges
// (link_source='manual') are NEVER touched by reconciliation.
// 'wikilink-resolved' MUST be reconcilable (codex outside-voice [P1]):
// auto-link writes these; if it weren't here, a basename edge would
// survive after the wikilink is deleted from the page OR the
// link_resolution.global_basename flag is turned off (out no longer
// includes it, so the stale-removal loop below must be allowed to drop it).
const reconcilableOut = existingOut.filter(
l => l.link_source === 'markdown' || l.link_source == null ||
l.link_source === 'wikilink-resolved' ||
(l.link_source === 'frontmatter' && l.origin_slug === slug),
);
+3 -1
View File
@@ -258,7 +258,9 @@ CREATE TABLE IF NOT EXISTS links (
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
-- for search ranking; only counts toward orphan-ratio + graph traversal.
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
-- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename
-- wikilink resolution. See src/schema.sql for full rationale.
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')),
-- v0.41.18.0 (codex finding #12): nullable link_kind distinguishes
-- "plain body mention" from "verb-pattern-derived typed link" within
-- link_source='mentions'. See src/schema.sql for full rationale.
+8 -3
View File
@@ -411,8 +411,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
-- ============================================================
-- links: cross-references between pages
-- ============================================================
-- Provenance model (v0.13):
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
-- Provenance model (v0.13, extended issue #972):
-- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL
-- 'wikilink-resolved' is the opt-in
-- (link_resolution.global_basename) basename-match
-- provenance — see issue #972 / migration v113.
-- (NULL = legacy row written before v0.13; unknown source)
-- origin_page_id — for link_source='frontmatter', the page whose YAML
-- frontmatter created this edge; scopes reconciliation
@@ -431,7 +434,9 @@ CREATE TABLE IF NOT EXISTS links (
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
-- for search ranking; only counts toward orphan-ratio + graph traversal.
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
-- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename
-- wikilink resolution (bare [[name]] resolved by slug tail).
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')),
-- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from
-- "verb-pattern-derived typed link" within link_source='mentions'.
-- Codex finding #12 design: keep link_source stable; add link_kind
+8 -3
View File
@@ -407,8 +407,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
-- ============================================================
-- links: cross-references between pages
-- ============================================================
-- Provenance model (v0.13):
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
-- Provenance model (v0.13, extended issue #972):
-- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL
-- 'wikilink-resolved' is the opt-in
-- (link_resolution.global_basename) basename-match
-- provenance — see issue #972 / migration v113.
-- (NULL = legacy row written before v0.13; unknown source)
-- origin_page_id — for link_source='frontmatter', the page whose YAML
-- frontmatter created this edge; scopes reconciliation
@@ -427,7 +430,9 @@ CREATE TABLE IF NOT EXISTS links (
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
-- for search ranking; only counts toward orphan-ratio + graph traversal.
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
-- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename
-- wikilink resolution (bare [[name]] resolved by slug tail).
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')),
-- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from
-- "verb-pattern-derived typed link" within link_source='mentions'.
-- Codex finding #12 design: keep link_source stable; add link_kind
+158
View File
@@ -1159,6 +1159,164 @@ describe('v0.40.4 — graph_signals_coverage check', () => {
});
});
// ─── issue #972 — link_resolution_opportunity check ───────────────────────
describe('issue #972 — link_resolution_opportunity check', () => {
const { PGLiteEngine } = require('../src/core/pglite-engine.ts');
const { checkLinkResolutionOpportunity } = require('../src/commands/doctor.ts');
let engine: any;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' });
await engine.initSchema();
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
beforeEach(async () => {
await engine.executeRaw(`DELETE FROM links`);
await engine.executeRaw(`DELETE FROM pages`);
await engine.executeRaw(
`DELETE FROM config WHERE key = 'link_resolution.global_basename'`,
);
});
test('skipped silently when flag is already enabled', async () => {
await engine.setConfig('link_resolution.global_basename', 'true');
// Even with bare wikilinks that would resolve, the check returns ok
// because the user is already opted in — no hint to surface.
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[struktura]] and [[struktura]] and [[struktura]].',
timeline: '',
});
const check = await checkLinkResolutionOpportunity(engine);
expect(check.status).toBe('ok');
expect(check.message).toContain('already enabled');
});
test('empty brain → ok with explanation', async () => {
const check = await checkLinkResolutionOpportunity(engine);
expect(check.status).toBe('ok');
expect(check.message).toContain('empty');
});
test('no bare wikilinks → ok', async () => {
await engine.putPage('people/alice', {
type: 'person', title: 'Alice',
compiled_truth: 'Met [Bob](people/bob).', timeline: '',
});
await engine.putPage('people/bob', {
type: 'person', title: 'Bob', compiled_truth: '', timeline: '',
});
const check = await checkLinkResolutionOpportunity(engine);
expect(check.status).toBe('ok');
expect(check.message).toContain('No bare wikilinks');
});
test('bare wikilinks present but none match → ok', async () => {
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[never-existed]] and [[also-not-here]].',
timeline: '',
});
const check = await checkLinkResolutionOpportunity(engine);
expect(check.status).toBe('ok');
expect(check.message).toContain('none have basename matches');
});
test('≥5 would-resolve AND ≥20% ratio → warn with paste-ready hint', async () => {
// 5 distinct bare wikilinks, all resolving = 100% ratio.
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
});
await engine.putPage('projects/eeva', {
type: 'project', title: 'Eeva', compiled_truth: '', timeline: '',
});
await engine.putPage('companies/fast-weigh', {
type: 'company', title: 'Fast-Weigh', compiled_truth: '', timeline: '',
});
await engine.putPage('projects/rosa', {
type: 'project', title: 'Rosa', compiled_truth: '', timeline: '',
});
await engine.putPage('projects/dragon', {
type: 'project', title: 'Dragon', compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/wiki-index', {
type: 'concept', title: 'Wiki',
compiled_truth: 'See [[struktura]], [[eeva]], [[Fast-Weigh]], [[rosa]], [[dragon]] for context.',
timeline: '',
});
const check = await checkLinkResolutionOpportunity(engine);
expect(check.status).toBe('warn');
expect(check.message).toContain('5 of 5');
expect(check.message).toContain('100%');
expect(check.message).toContain(
'gbrain config set link_resolution.global_basename true',
);
});
test('<5 would-resolve → ok without warning (below threshold)', async () => {
// Only 2 bare wikilinks resolve → below the 5-link floor.
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
});
await engine.putPage('projects/eeva', {
type: 'project', title: 'Eeva', compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[struktura]] and [[eeva]] for context.',
timeline: '',
});
const check = await checkLinkResolutionOpportunity(engine);
expect(check.status).toBe('ok');
expect(check.message).toContain('below the 20% / 5-link threshold');
});
test('counts slugified-only matches (shared matcher, codex #972 DRY)', async () => {
// `[[Fast Weigh]]` (space) only resolves to `companies/fast-weigh` via the
// slugified key. Pre-consolidation the doctor index keyed raw+lower only
// and would have reported "none have basename matches"; the shared matcher
// adds the slugified key so the estimate matches what extraction resolves.
await engine.putPage('companies/fast-weigh', {
type: 'company', title: 'Fast-Weigh', compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[Fast Weigh]] for context.', timeline: '',
});
const check = await checkLinkResolutionOpportunity(engine);
// 1/1 resolves (below the 5-link warn floor → ok) but it DID resolve.
expect(check.message).toContain('1/1');
expect(check.message).toContain('would resolve');
expect(check.message).not.toContain('none have basename matches');
});
test('check is wired into runDoctor AND doctorReportRemote (source-grep)', async () => {
const source = await Bun.file(
new URL('../src/commands/doctor.ts', import.meta.url),
).text();
// Local buildChecks path.
expect(source).toMatch(/await checkLinkResolutionOpportunity\(engine, progress\)/);
// Thin-client doctorReportRemote path.
expect(source).toMatch(/await checkLinkResolutionOpportunity\(engine\)/);
// Heartbeat label registered.
expect(source).toContain("progress.heartbeat('link_resolution_opportunity')");
// Issue #972 (T3): scan is bounded to a most-recent sample, not a full
// per-page getPage walk.
expect(source).toMatch(/ORDER BY id DESC LIMIT \$\{?SAMPLE_LIMIT\}?|ORDER BY id DESC LIMIT 1000/);
expect(source).not.toMatch(/await engine\.getPage\(ref\.slug/);
});
});
describe('v0.42 (#1699) — quarantined_pages + flagged_pages checks', () => {
test('both checks are wired into buildChecks (source-grep)', async () => {
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
+282
View File
@@ -0,0 +1,282 @@
/**
* Issue #972 E2E — opt-in global-basename wikilink resolution.
*
* Reproduces the issue's exact repro inside an in-memory PGLite brain:
*
* /vault/projects/struktura.md ← a real page
* /vault/concepts/knowledge-graph.md ← contains `[[struktura]]`
*
* The bare wikilink `[[struktura]]` does NOT match WIKILINK_RE
* (DIR_PATTERN-gated) so it falls through to the new pass 2c.
*
* Three contracts:
* 1. Flag OFF (default) — extract emits ZERO basename edges (back-compat
* with pre-issue-#972 behavior: bare wikilinks outside DIR_PATTERN
* drop silently).
* 2. Flag ON — extract emits ONE edge per basename match, tagged
* `link_type: 'wikilink_basename'`.
* 3. Ambiguous basename (same name in two directories) — extract emits
* ONE edge per match. No silent winner-takes-all, no silent drop.
*
* Both DB-source AND FS-source paths covered. PGLite in-memory, no
* DATABASE_URL needed.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { runExtract } from '../../src/commands/extract.ts';
let engine: PGLiteEngine;
let brainDir: string;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
async function truncateAll() {
for (const t of [
'content_chunks', 'links', 'tags', 'raw_data',
'timeline_entries', 'page_versions', 'ingest_log', 'pages',
'config',
]) {
try { await (engine as any).db.exec(`DELETE FROM ${t}`); } catch { /* ok */ }
}
}
beforeEach(async () => {
await truncateAll();
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-issue-972-'));
}, 15_000);
function writeFile(rel: string, content: string) {
const full = join(brainDir, rel);
mkdirSync(join(full, '..'), { recursive: true });
writeFileSync(full, content);
}
// ─── FS-source path (the issue's repro) ─────────────────────────────────
describe('issue #972 — FS-source (gbrain extract links default)', () => {
test("repro: [[struktura]] in concepts/ resolves to projects/struktura when flag ON", async () => {
// Seed both pages in the DB (extract validates targetSlug exists)
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: 'A project page.', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This concept relates to [[struktura]].', timeline: '',
});
// Mirror to disk so the FS extractor sees the files
writeFile('projects/struktura.md',
'---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
writeFile('concepts/knowledge-graph.md',
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--dir', brainDir]);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
// Issue #972 (T1): FS-source basename edges carry the same provenance
// tag as DB / put_page, not the default 'markdown'.
expect(strk!.link_source).toBe('wikilink-resolved');
});
test('back-compat: flag OFF → ZERO basename edges from same repro', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This relates to [[struktura]].', timeline: '',
});
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
writeFile('concepts/knowledge-graph.md',
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis relates to [[struktura]].\n');
// Flag explicitly off (also the default)
await engine.setConfig('link_resolution.global_basename', 'false');
await runExtract(engine, ['links', '--dir', brainDir]);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
expect(outLinks.filter(l => l.link_type === 'wikilink_basename')).toEqual([]);
});
test('ambiguous basename → emits one edge per match (no silent winner)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('archive/struktura', {
type: 'concept' as any, title: 'Struktura (archived)',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[struktura]].', timeline: '',
});
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n');
writeFile('concepts/x.md',
'---\ntitle: X\ntype: concept\n---\n\nSee [[struktura]].\n');
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--dir', brainDir]);
const outLinks = await engine.getLinks('concepts/x');
const basenameLinks = outLinks
.filter(l => l.link_type === 'wikilink_basename')
.map(l => l.to_slug)
.sort();
expect(basenameLinks).toEqual(['archive/struktura', 'projects/struktura']);
});
});
// ─── DB-source path (gbrain extract links --source db) ──────────────────
describe('issue #972 — DB-source (gbrain extract links --source db)', () => {
test('flag ON → bare wikilink in compiled_truth resolves to basename match', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This relates to [[struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag OFF → no basename edges via DB path (back-compat)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This relates to [[struktura]].', timeline: '',
});
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
});
});
// ─── put_page auto-link path ────────────────────────────────────────────
//
// put_page accepts `content` as a full markdown document with frontmatter
// (not the engine's narrow Page shape). Auto-link runs INSIDE put_page,
// so the basename-resolution path picks up the flag from
// engine.getConfig() once per call.
const PUT_PAGE_MARKDOWN_WITH_WIKILINK = `---
title: Knowledge Graph
type: concept
---
This relates to [[struktura]].
`;
describe('issue #972 — put_page auto-link', () => {
test('newly-written page with bare wikilink → basename edge when flag ON', async () => {
// Need the target page to exist first (auto-link validates against the
// existing slug set).
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
const { operations } = await import('../../src/core/operations.ts');
const putPage = operations.find(op => op.name === 'put_page')!;
await putPage.handler(
{ engine, remote: false } as never,
{
slug: 'concepts/knowledge-graph',
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
},
);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag OFF → put_page does NOT emit basename edge (back-compat)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
compiled_truth: '', timeline: '',
});
// Default-off (no setConfig)
const { operations } = await import('../../src/core/operations.ts');
const putPage = operations.find(op => op.name === 'put_page')!;
await putPage.handler(
{ engine, remote: false } as never,
{
slug: 'concepts/knowledge-graph',
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
},
);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
});
test('stale basename edge is removed when the wikilink is deleted (codex #972)', async () => {
// Regression: wikilink-resolved edges must be reconcilable, else they
// survive after the bare wikilink is removed from the page body.
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
const { operations } = await import('../../src/core/operations.ts');
const putPage = operations.find(op => op.name === 'put_page')!;
// 1. Write the page WITH the wikilink → edge lands.
await putPage.handler({ engine, remote: false } as never, {
slug: 'concepts/knowledge-graph', content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
});
let outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeDefined();
// 2. Re-write the page WITHOUT the wikilink → edge must be reconciled away.
await putPage.handler({ engine, remote: false } as never, {
slug: 'concepts/knowledge-graph',
content: '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nNo links here anymore.\n',
});
outLinks = await engine.getLinks('concepts/knowledge-graph');
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
});
});
+210
View File
@@ -246,3 +246,213 @@ describe('gbrain extract --dir default resolution', () => {
}
});
});
// ─── issue #972: pure-function tests for resolveSlugAll + helpers ────────
import {
resolveSlug,
resolveSlugAll,
resolveBasenameMatchesFromSlugs,
extractLinksFromFile,
} from '../src/commands/extract.ts';
describe('extractLinksFromFile — code-fence stripping (codex P2b)', () => {
test('a bare wikilink inside a code fence does NOT create an FS edge', async () => {
const allSlugs = new Set(['projects/struktura', 'concepts/x']);
const content = [
'---', 'title: X', 'type: concept', '---', '',
'Real ref: [[struktura]].',
'',
'```', 'code mentions [[struktura]] but must be ignored', '```',
].join('\n');
const links = await extractLinksFromFile(content, 'concepts/x.md', allSlugs, { globalBasename: true });
const toStruktura = links.filter(l => l.to_slug === 'projects/struktura');
// Exactly one edge — from the prose ref, NOT the fenced one.
expect(toStruktura.length).toBe(1);
expect(toStruktura[0].link_source).toBe('wikilink-resolved');
});
});
describe('resolveBasenameMatchesFromSlugs (pure)', () => {
test('returns ALL slugs whose tail matches', () => {
const all = new Set([
'projects/struktura',
'archive/struktura',
'notes/other',
]);
const matches = resolveBasenameMatchesFromSlugs('struktura', all);
expect(matches.sort()).toEqual(['archive/struktura', 'projects/struktura']);
});
test('case-insensitive tail match', () => {
const all = new Set(['companies/fast-weigh']);
expect(resolveBasenameMatchesFromSlugs('Fast-Weigh', all))
.toEqual(['companies/fast-weigh']);
expect(resolveBasenameMatchesFromSlugs('FAST-WEIGH', all))
.toEqual(['companies/fast-weigh']);
});
test('slugified fallback (spaces → hyphens)', () => {
const all = new Set(['companies/fast-weigh']);
expect(resolveBasenameMatchesFromSlugs('Fast Weigh', all))
.toEqual(['companies/fast-weigh']);
});
test('top-level slugs (no `/`) match by themselves', () => {
const all = new Set(['struktura', 'notes/struktura']);
const matches = resolveBasenameMatchesFromSlugs('struktura', all);
expect(matches.sort()).toEqual(['notes/struktura', 'struktura']);
});
test('no match returns []', () => {
const all = new Set(['projects/struktura']);
expect(resolveBasenameMatchesFromSlugs('never-existed', all)).toEqual([]);
});
test('empty/whitespace input returns []', () => {
const all = new Set(['projects/struktura']);
expect(resolveBasenameMatchesFromSlugs('', all)).toEqual([]);
expect(resolveBasenameMatchesFromSlugs(' ', all)).toEqual([]);
});
test('stable sort: shorter slug first, then lexical', () => {
const all = new Set([
'zzz/struktura',
'projects/struktura',
'archive/struktura',
'a/struktura',
]);
const matches = resolveBasenameMatchesFromSlugs('struktura', all);
// Lengths: a/struktura(11), zzz/struktura(13), archive/struktura(17), projects/struktura(18)
expect(matches).toEqual([
'a/struktura',
'zzz/struktura',
'archive/struktura',
'projects/struktura',
]);
});
});
describe('resolveSlugAll', () => {
test('ancestor walk wins → single-element array, no basename fallback', () => {
// resolveSlug already finds notes/struktura via ancestor walk.
// resolveSlugAll must return only that one even though basename
// would find others.
const all = new Set([
'notes/struktura',
'archive/struktura',
'projects/struktura',
]);
const out = resolveSlugAll('notes/sub', 'struktura.md', all,
{ globalBasename: true });
expect(out).toEqual(['notes/struktura']);
});
test('ancestor walk misses + globalBasename off → []', () => {
const all = new Set(['projects/struktura']);
const out = resolveSlugAll('concepts', 'struktura.md', all);
expect(out).toEqual([]);
});
test('ancestor walk misses + globalBasename on → all basename matches', () => {
const all = new Set([
'projects/struktura',
'archive/struktura',
]);
const out = resolveSlugAll('concepts', 'struktura.md', all,
{ globalBasename: true });
expect(out.sort()).toEqual(['archive/struktura', 'projects/struktura']);
});
test('zero basename matches when globalBasename on returns []', () => {
const all = new Set(['projects/struktura']);
const out = resolveSlugAll('concepts', 'phantom.md', all,
{ globalBasename: true });
expect(out).toEqual([]);
});
test('strips dirname from relTarget when applying basename lookup', () => {
// [[notes/struktura]] with no `notes` ancestor: ancestor walk strips
// `concepts` → tries `notes/struktura` which DOES exist → emits.
// This case verifies the basename fallback only fires when ancestor
// walk truly fails. (Sanity check on the fallback ordering.)
const all = new Set(['notes/struktura']);
expect(resolveSlugAll('concepts', 'notes/struktura.md', all))
.toEqual(['notes/struktura']);
});
test('resolveSlug back-compat: existing single-match callers unaffected', () => {
const all = new Set(['notes/struktura']);
// The legacy resolveSlug must keep returning the string|null shape.
expect(resolveSlug('notes', 'struktura.md', all)).toBe('notes/struktura');
expect(resolveSlug('concepts', 'phantom.md', all)).toBeNull();
});
});
describe('issue #972 repro: bare wikilinks resolve when flag is on', () => {
// End-to-end: reproduces the issue's exact repro inside a tempdir +
// PGLite, then asserts edge count under both flag states.
test('flag OFF → 0 edges (back-compat)', async () => {
await engine.putPage('projects/struktura',
{ type: 'project', title: 'Struktura', compiled_truth: 'A project page.', timeline: '' });
await engine.putPage('concepts/knowledge-graph',
{ type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This concept relates to [[struktura]].', timeline: '' });
// Mirror to disk: the FS extractor walks files, not DB pages.
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
writeFile('concepts/knowledge-graph.md',
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
// Ensure flag is off (default)
await engine.setConfig('link_resolution.global_basename', 'false');
await runExtract(engine, ['links', '--dir', brainDir]);
const links = await engine.getLinks('concepts/knowledge-graph');
expect(links.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
});
test('flag ON → 1 edge with wikilink_basename type', async () => {
await engine.putPage('projects/struktura',
{ type: 'project', title: 'Struktura', compiled_truth: 'A project page.', timeline: '' });
await engine.putPage('concepts/knowledge-graph',
{ type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'This concept relates to [[struktura]].', timeline: '' });
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
writeFile('concepts/knowledge-graph.md',
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--dir', brainDir]);
const links = await engine.getLinks('concepts/knowledge-graph');
const strk = links.find(l => l.to_slug === 'projects/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag ON + ambiguous basename → one edge per match', async () => {
await engine.putPage('projects/struktura',
{ type: 'project', title: 'Struktura', compiled_truth: '', timeline: '' });
await engine.putPage('archive/struktura',
{ type: 'concept' as any, title: 'Struktura (archived)',
compiled_truth: '', timeline: '' });
await engine.putPage('concepts/knowledge-graph',
{ type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'See [[struktura]].', timeline: '' });
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n');
writeFile('concepts/knowledge-graph.md',
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nSee [[struktura]].\n');
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--dir', brainDir]);
const links = await engine.getLinks('concepts/knowledge-graph');
const basenameLinks = links.filter(l => l.link_type === 'wikilink_basename');
const targets = basenameLinks.map(l => l.to_slug).sort();
expect(targets).toEqual(['archive/struktura', 'projects/struktura']);
});
});
+410
View File
@@ -109,6 +109,92 @@ describe('extractEntityRefs', () => {
expect(refs.length).toBe(1);
expect(refs[0].dir).toBe('meetings');
});
// ─── issue #972: generic `[[bare-name]]` wikilinks (pass 2c) ─────────────
test('tags bare wikilinks with needsResolution flag', () => {
const refs = extractEntityRefs(
'See [[Fast-Weigh]] and [[2026-05-07-cost-plan-rosa-pilot]] for context.',
);
expect(refs.length).toBe(2);
expect(refs.every(r => r.needsResolution === true)).toBe(true);
expect(refs.map(r => r.slug).sort()).toEqual([
'2026-05-07-cost-plan-rosa-pilot',
'Fast-Weigh',
]);
// dir is empty string when the bare wikilink has no `/`
for (const r of refs) {
expect(r.dir).toBe('');
}
});
test('does NOT double-emit when DIR_PATTERN wikilink also passes 2b', () => {
// [[people/alice]] matches 2b (DIR_PATTERN-gated). 2c must NOT emit
// a duplicate ref. [[Fast-Weigh]] only matches 2c (no DIR_PATTERN).
const refs = extractEntityRefs('See [[people/alice]] and [[Fast-Weigh]].');
const aliceRefs = refs.filter(r => r.slug === 'people/alice');
const wikiRefs = refs.filter(r => r.slug === 'Fast-Weigh');
expect(aliceRefs.length).toBe(1);
expect(aliceRefs[0].needsResolution).toBeUndefined();
expect(wikiRefs.length).toBe(1);
expect(wikiRefs[0].needsResolution).toBe(true);
});
test('skips qualified-syntax tokens (those belong to 2a)', () => {
// [[wiki:topics/ai]] looks like 2a's qualified shape — even though
// it wouldn't satisfy DIR_PATTERN, 2c must not claim it either
// (the leading `:` is the qualified-syntax tell).
const refs = extractEntityRefs('See [[wiki:topics/ai]] and [[bare-name]].');
const bare = refs.find(r => r.slug === 'bare-name');
expect(bare).toBeDefined();
expect(bare!.needsResolution).toBe(true);
const wrongQualified = refs.filter(
r => r.slug.includes(':') && r.needsResolution === true,
);
expect(wrongQualified.length).toBe(0);
});
test('a wikilink inside a markdown-link label is inert (codex P2a)', () => {
// `[see [[acme]]](companies/acme.md)` must NOT spawn a stray generic
// basename ref for the inner `[[acme]]`. Pass-1 can't match the nested
// brackets, so the label-wikilink span is masked out of pass 2c.
const refs = extractEntityRefs('[see [[acme]]](companies/acme.md)');
expect(refs.filter(r => r.needsResolution)).toEqual([]);
// But an independent bare wikilink on the same line still emits.
const refs2 = extractEntityRefs('[Acme](companies/acme) and bare [[acme]] here.');
expect(refs2.find(r => r.slug === 'companies/acme' && !r.needsResolution)).toBeDefined();
expect(refs2.find(r => r.slug === 'acme' && r.needsResolution)).toBeDefined();
});
test('strips .md suffix from bare wikilinks', () => {
const refs = extractEntityRefs('See [[struktura.md]] for context.');
expect(refs.length).toBe(1);
expect(refs[0].slug).toBe('struktura');
expect(refs[0].needsResolution).toBe(true);
});
test('extracts display name from [[slug|Display]] shape', () => {
const refs = extractEntityRefs('See [[struktura|The Project]] for details.');
expect(refs.length).toBe(1);
expect(refs[0].slug).toBe('struktura');
expect(refs[0].name).toBe('The Project');
expect(refs[0].needsResolution).toBe(true);
});
test('strips #anchor from bare wikilinks', () => {
const refs = extractEntityRefs('Jump to [[notes#section-2]].');
expect(refs.length).toBe(1);
expect(refs[0].slug).toBe('notes');
expect(refs[0].needsResolution).toBe(true);
});
test('skips bare wikilinks inside fenced code blocks', () => {
const refs = extractEntityRefs(
'```\nThis is a code block with [[fake-link]] inside.\n```\nReal: [[real-link]].',
);
expect(refs.length).toBe(1);
expect(refs[0].slug).toBe('real-link');
});
});
// ─── extractPageLinks ──────────────────────────────────────────
@@ -181,6 +267,187 @@ describe('extractPageLinks', () => {
const aliceLink = candidates.find(c => c.targetSlug === 'people/alice');
expect(aliceLink!.linkType).toBe('attended');
});
// ─── issue #972: bare wikilink → resolver.resolveBasenameMatches ─────────
test('bare wikilink drops silently when globalBasename flag is OFF', async () => {
// Resolver that WOULD resolve, but we never reach it because the
// flag is off — this is the back-compat invariant.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async () => ['projects/struktura'],
};
const { candidates } = await extractPageLinks(
'concepts/knowledge-graph',
'This relates to [[struktura]].',
{}, 'concept', resolver,
// opts.globalBasename omitted (= false)
);
expect(candidates.find(c => c.targetSlug === 'projects/struktura')).toBeUndefined();
expect(candidates).toEqual([]);
});
test('bare wikilink emits one candidate per basename match when flag ON', async () => {
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) => {
if (name === 'struktura') return ['projects/struktura', 'archive/struktura'];
return [];
},
};
const { candidates } = await extractPageLinks(
'concepts/knowledge-graph',
'This relates to [[struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
const targets = candidates.map(c => c.targetSlug).sort();
expect(targets).toEqual(['archive/struktura', 'projects/struktura']);
// Both edges stamped with the new edge type + provenance.
for (const c of candidates) {
expect(c.linkType).toBe('wikilink_basename');
expect(c.linkSource).toBe('wikilink-resolved');
}
});
test('bare wikilink with single basename match emits one candidate', async () => {
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['projects/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/knowledge-graph',
'See [[struktura]] for details.',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.length).toBe(1);
expect(candidates[0].targetSlug).toBe('projects/struktura');
expect(candidates[0].linkType).toBe('wikilink_basename');
});
test('basename self-link is dropped (codex P2c)', async () => {
// `[[struktura]]` on the page concepts/struktura resolves back to itself —
// the self-loop must be dropped.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['concepts/struktura', 'projects/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/struktura', // the page being processed
'See [[struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
// Only the OTHER match survives; no self-edge to concepts/struktura.
expect(candidates.map(c => c.targetSlug)).toEqual(['projects/struktura']);
});
test('aliased wikilink resolves the TARGET, not the display text (codex #972)', async () => {
// `[[struktura|the project]]` must resolve basename `struktura`, never
// the alias "the project". Regression for the codex-caught bug where
// extractPageLinks resolved ref.name (display) instead of ref.slug.
const seen: string[] = [];
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) => {
seen.push(name);
return name === 'struktura' ? ['projects/struktura'] : [];
},
};
const { candidates } = await extractPageLinks(
'concepts/knowledge-graph',
'This relates to [[struktura|the project]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(seen).toContain('struktura');
expect(seen).not.toContain('the project');
expect(candidates.map(c => c.targetSlug)).toEqual(['projects/struktura']);
});
test('bare wikilink with zero basename matches drops silently (no dangling row)', async () => {
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async () => [],
};
const { candidates } = await extractPageLinks(
'concepts/x', 'Mention [[never-existed]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.find(c => c.targetSlug === 'never-existed')).toBeUndefined();
expect(candidates).toEqual([]);
});
test('bare wikilink resolution does not interfere with DIR_PATTERN wikilinks', async () => {
// 2b refs (people/alice) take the verb-inferred type;
// 2c refs (struktura) take wikilink_basename. Same call.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['projects/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/x',
'[[people/alice]] is the lead. The work is [[struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
const alice = candidates.find(c => c.targetSlug === 'people/alice');
const strk = candidates.find(c => c.targetSlug === 'projects/struktura');
expect(alice).toBeDefined();
expect(alice!.linkType).not.toBe('wikilink_basename'); // verb-inferred
expect(strk).toBeDefined();
expect(strk!.linkType).toBe('wikilink_basename');
});
test('opts.skipFrontmatter suppresses the frontmatter pass', async () => {
// Real resolver shape that WOULD resolve frontmatter source: too,
// but skipFrontmatter blocks the path entirely.
const resolver: SlugResolver = {
resolve: async (name) =>
name === 'meetings/2026-01-15' ? 'meetings/2026-01-15' : null,
};
const fm = { source: 'meetings/2026-01-15' };
const withFm = await extractPageLinks(
'docs/x', 'plain content', fm, 'person', resolver,
{ skipFrontmatter: false },
);
const withoutFm = await extractPageLinks(
'docs/x', 'plain content', fm, 'person', resolver,
{ skipFrontmatter: true },
);
expect(withFm.candidates.find(c => c.linkType === 'source')).toBeDefined();
expect(withoutFm.candidates.find(c => c.linkType === 'source')).toBeUndefined();
// Issue #972 (codex P2e): skipFrontmatter must return an empty unresolved
// list (the pass is skipped entirely), never undefined.
expect(withoutFm.unresolved).toEqual([]);
});
test('skipFrontmatter suppresses unresolved frontmatter refs too (codex P2e)', async () => {
// A frontmatter field the resolver CANNOT resolve normally populates
// `unresolved`; with skipFrontmatter the whole pass is gone so it's [].
const resolver: SlugResolver = { resolve: async () => null };
const fm = { key_people: ['Nobody Known'] };
const withFm = await extractPageLinks(
'companies/acme', 'plain content', fm, 'company', resolver,
{ skipFrontmatter: false },
);
const withoutFm = await extractPageLinks(
'companies/acme', 'plain content', fm, 'company', resolver,
{ skipFrontmatter: true },
);
expect(withFm.unresolved.length).toBeGreaterThan(0); // pass ran, ref unresolved
expect(withoutFm.unresolved).toEqual([]); // pass skipped
});
test('globalBasename does nothing when resolver lacks resolveBasenameMatches', async () => {
// The frontmatter-only synthetic resolver doesn't implement basename
// lookup. Make sure we don't blow up — just drop the bare ref.
const resolver: SlugResolver = { resolve: async () => null };
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates).toEqual([]);
});
});
// ─── inferLinkType ─────────────────────────────────────────────
@@ -729,6 +996,149 @@ describe('makeResolver — fallback chain', () => {
const r = makeResolver(engine, { mode: 'batch' });
expect(await r.resolve('Nonexistent Person', 'people')).toBeNull();
});
// ─── issue #972: resolveBasenameMatches ───────────────────────────────
// Extended fake engine that also implements `getAllSlugs` so
// resolveBasenameMatches has something to walk.
function makeFakeEngineWithSlugs(slugs: string[]): BrainEngine {
const lookup = new Set(slugs);
let getAllCalls = 0;
const engine = {
async getPage(slug: string) {
return lookup.has(slug) ? { slug } as any : null;
},
async findByTitleFuzzy() { return null; },
async searchKeyword() { return []; },
async getAllSlugs() {
getAllCalls++;
return new Set(slugs);
},
} as unknown as BrainEngine;
(engine as any)._counts = () => ({ getAllCalls });
return engine;
}
test('resolveBasenameMatches: exact tail hit returns the slug', async () => {
const engine = makeFakeEngineWithSlugs([
'projects/struktura',
'people/alice',
]);
const r = makeResolver(engine);
expect(await r.resolveBasenameMatches!('struktura')).toEqual(['projects/struktura']);
});
test('resolveBasenameMatches: multi-match returns ALL hits', async () => {
const engine = makeFakeEngineWithSlugs([
'projects/struktura',
'archive/struktura',
'notes/struktura',
]);
const r = makeResolver(engine);
const out = await r.resolveBasenameMatches!('struktura');
expect(out.sort()).toEqual([
'archive/struktura',
'notes/struktura',
'projects/struktura',
]);
});
test('resolveBasenameMatches: case-insensitive fallback', async () => {
const engine = makeFakeEngineWithSlugs(['companies/fast-weigh']);
const r = makeResolver(engine);
// Raw `Fast-Weigh` does not match the lowercase tail, but the
// lowercased+slugified key does — both should hit.
expect(await r.resolveBasenameMatches!('fast-weigh')).toEqual(['companies/fast-weigh']);
expect(await r.resolveBasenameMatches!('Fast-Weigh')).toContain('companies/fast-weigh');
});
test('resolveBasenameMatches: no matches returns []', async () => {
const engine = makeFakeEngineWithSlugs(['projects/struktura']);
const r = makeResolver(engine);
expect(await r.resolveBasenameMatches!('never-existed')).toEqual([]);
});
test('resolveBasenameMatches: scopes the index by sourceId (codex #972)', async () => {
// Regression: a bare [[struktura]] in source A must NOT resolve to a
// same-tail page in source B. makeResolver({sourceId}) must pass the
// scope to getAllSlugs so the index only contains the source's slugs.
let sawOpts: any;
const bySource: Record<string, string[]> = {
'src-a': ['projects/struktura'],
'src-b': ['archive/struktura'],
};
const engine = {
async getPage() { return null; },
async findByTitleFuzzy() { return null; },
async searchKeyword() { return []; },
async getAllSlugs(opts?: { sourceId?: string }) {
sawOpts = opts;
const sid = opts?.sourceId;
return new Set(sid ? (bySource[sid] ?? []) : Object.values(bySource).flat());
},
} as unknown as BrainEngine;
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
const out = await r.resolveBasenameMatches!('struktura');
expect(sawOpts).toEqual({ sourceId: 'src-a' });
expect(out).toEqual(['projects/struktura']); // src-a only
expect(out).not.toContain('archive/struktura'); // no cross-source
});
test('resolveBasenameMatches: no sourceId stays brain-wide (back-compat)', async () => {
let sawOpts: any = 'unset';
const engine = {
async getPage() { return null; },
async findByTitleFuzzy() { return null; },
async searchKeyword() { return []; },
async getAllSlugs(opts?: { sourceId?: string }) {
sawOpts = opts;
return new Set(['projects/struktura', 'archive/struktura']);
},
} as unknown as BrainEngine;
const r = makeResolver(engine, { mode: 'batch' });
const out = await r.resolveBasenameMatches!('struktura');
expect(sawOpts).toBeUndefined(); // unscoped call
expect(out.sort()).toEqual(['archive/struktura', 'projects/struktura']);
});
test('resolveBasenameMatches: empty input returns []', async () => {
const engine = makeFakeEngineWithSlugs(['projects/struktura']);
const r = makeResolver(engine);
expect(await r.resolveBasenameMatches!('')).toEqual([]);
expect(await r.resolveBasenameMatches!(' ')).toEqual([]);
});
test('resolveBasenameMatches: index built once, reused across calls', async () => {
const engine = makeFakeEngineWithSlugs([
'projects/struktura',
'archive/struktura',
]);
const r = makeResolver(engine);
await r.resolveBasenameMatches!('struktura');
await r.resolveBasenameMatches!('struktura');
await r.resolveBasenameMatches!('struktura');
// Single getAllSlugs() call across three resolveBasenameMatches calls.
expect((engine as any)._counts().getAllCalls).toBe(1);
});
test('resolveBasenameMatches: degrades gracefully when getAllSlugs missing', async () => {
// Test seam for engines that don't implement getAllSlugs (legacy / mocks).
const engine = {
async getPage() { return null; },
async findByTitleFuzzy() { return null; },
async searchKeyword() { return []; },
} as unknown as BrainEngine;
const r = makeResolver(engine);
expect(await r.resolveBasenameMatches!('struktura')).toEqual([]);
});
test('resolveBasenameMatches: handles top-level slugs (no `/`)', async () => {
const engine = makeFakeEngineWithSlugs(['struktura', 'notes/struktura']);
const r = makeResolver(engine);
// Both should match because basename of `struktura` is `struktura`.
const out = await r.resolveBasenameMatches!('struktura');
expect(out.sort()).toEqual(['notes/struktura', 'struktura']);
});
});
describe('FRONTMATTER_LINK_MAP integrity', () => {