mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 19:49:14 +00:00
Merge remote-tracking branch 'origin/master' into garrytan/santo-domingo-v4
This commit is contained in:
@@ -2,6 +2,70 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.10.0] - 2026-06-02
|
||||
|
||||
**Wikilinks like `[[struktura]]` that point at pages in another folder finally connect.** Until now, if you wrote `[[struktura]]` in `concepts/knowledge-graph.md` and the actual page lived at `projects/struktura.md`, GBrain silently dropped the link from its graph. Obsidian users saw a dense web of connections in their vault and a thin, broken graph inside GBrain. The issue reporter had 71 wikilinks across 20 pages — GBrain captured 12.
|
||||
|
||||
This release adds an **opt-in mode** that resolves bare wikilinks by basename match. Turn it on with one command and the missing edges show up.
|
||||
|
||||
```
|
||||
gbrain config set link_resolution.global_basename true
|
||||
gbrain extract links
|
||||
```
|
||||
|
||||
`gbrain doctor` scans your brain on every run and tells you what you'd gain. If five or more bare wikilinks would resolve under the new mode, you'll see a paste-ready hint:
|
||||
|
||||
```
|
||||
link_resolution_opportunity: 47 of 60 bare wikilinks (78%) would resolve to
|
||||
24 distinct page(s) under global_basename mode.
|
||||
Enable with: gbrain config set link_resolution.global_basename true
|
||||
```
|
||||
|
||||
The mode covers every place wikilinks get resolved: `gbrain extract links` (both the filesystem-walk path that the issue reporter hit by default, and the database path that the autopilot cycle uses), and the auto-link step that runs every time an agent writes a page through `put_page`. Same flag, all three surfaces.
|
||||
|
||||
### How it handles ambiguity
|
||||
|
||||
If `[[struktura]]` matches two pages — say `projects/struktura.md` and `archive/struktura.md` — GBrain emits **one edge to each**, both tagged with a new `wikilink_basename` link type. No silent winner-takes-all, no arbitrary first-wins choice. You can audit and prune the dual-match cases via:
|
||||
|
||||
```
|
||||
gbrain graph-query concepts/knowledge-graph --type wikilink_basename
|
||||
```
|
||||
|
||||
This is a deliberate departure from the issue's original proposal (which suggested dropping ambiguous matches with a warning). The graph layer is supposed to capture relationships, not editorialize about them.
|
||||
|
||||
### Why opt-in (and why this default is safe)
|
||||
|
||||
The default stays off, so existing brains get zero behavior change on upgrade. The bug class this fix opens — a `[[review]]` accidentally linking to some `topics/review` page that exists for unrelated reasons — is real, and turning it on globally would change every existing user's graph in ways they didn't ask for. Doctor surfaces the opportunity; the user decides.
|
||||
|
||||
For brains that flip the flag on, all newly extracted edges carry a distinct `link_source = 'wikilink-resolved'` provenance tag, so you can later distinguish basename-resolved edges from your hand-written markdown links.
|
||||
|
||||
### What this means
|
||||
|
||||
If you write in the Obsidian convention (flat-namespace, globally-unique basenames, bare `[[wikilinks]]` everywhere), your GBrain graph has been wrong. Flip the flag and run extract once — the missing edges show up immediately.
|
||||
|
||||
If you don't use bare wikilinks, doctor will tell you so (`No bare wikilinks found`) and you can keep ignoring this knob.
|
||||
|
||||
Closes https://github.com/garrytan/gbrain/issues/972.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/core/link-extraction.ts` — three new exports:
|
||||
- `WIKILINK_BASENAME_LINK_TYPE = 'wikilink_basename'` — the edge-type constant used by every path that emits basename-resolved edges. Adapted from PR #1233's regex (which was the kernel of the resolver-side fix).
|
||||
- `isGlobalBasenameEnabled(engine)` — single source of truth for the
|
||||
flag read. Order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB
|
||||
config `link_resolution.global_basename` → default false.
|
||||
- `WIKILINK_GENERIC_RE` — generic Obsidian-shape wikilink regex for the `[[bare-name]]` pass that falls outside `DIR_PATTERN`. `EntityRef` gains an optional `needsResolution` flag tagged onto refs from this pass so downstream callers know to route them through the resolver instead of treating them as canonical slugs.
|
||||
- `SlugResolver` interface gains an optional `resolveBasenameMatches(name): Promise<string[]>` method. Returns ALL slugs whose basename matches — multi-match by design. `makeResolver(engine)` implements this with a lazy-built basename → `string[]` index (single `engine.getAllSlugs()` call per resolver instance) covering raw + lowercase + slugified key variants.
|
||||
- `extractPageLinks` gains two options. `opts.globalBasename` routes `needsResolution: true` refs through `resolveBasenameMatches`, emitting one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`. `opts.skipFrontmatter` replaces the pre-issue-#972 `nullResolver` pattern — callers can now pass the real resolver everywhere AND independently gate the frontmatter pass, which is necessary because the bare-wikilink path needs `resolveBasenameMatches` on the real resolver.
|
||||
- `src/commands/extract.ts` — DB-source `extractLinksFromDB` and FS-source `extractLinksFromFile` / `extractLinksFromDir` / `extractLinksForSlugs` thread `globalBasename` (read once per extract run via `isGlobalBasenameEnabled`). FS-source adds two new exported helpers — `resolveBasenameMatchesFromSlugs(name, allSlugs): string[]` and `resolveSlugAll(fileDir, relTarget, allSlugs, opts): string[]`. `resolveSlug` stays single-match for back-compat; `resolveSlugAll` wraps it with the basename-fallback. Stable sort: shorter slug first, then lexical.
|
||||
- `src/core/operations.ts:put_page` — auto-link path now reads the flag and passes `globalBasename` through to `extractPageLinks`. Newly-written pages get basename edges when the flag is on.
|
||||
- `src/commands/doctor.ts` — new `link_resolution_opportunity` check. Full scan with a 60s budget, heartbeat-reported. Severity: skipped when flag already on; ok when no bare wikilinks; ok with explanation when bare wikilinks present but none match; **warn with paste-ready hint when ≥5 would resolve AND ≥20% of bare wikilinks have matches**. Threshold tuned to surface real opportunities without flagging brains that only have a handful of unresolvable refs. Wired into both `buildChecks` (local doctor) and `doctorReportRemote` (thin-client) for parity.
|
||||
- Migration v113 (`links_link_source_widen_for_wikilink_basename`) widens the `links_link_source_check` constraint to accept `'wikilink-resolved'` as a valid `link_source`. Idempotent — DROP IF EXISTS + ADD CONSTRAINT.
|
||||
- `src/schema.sql` + regenerated `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` carry the widened constraint for fresh installs so first-init brains start with the right shape.
|
||||
- New CHECK constraint value adds `'wikilink-resolved'` alongside `'markdown'`, `'frontmatter'`, `'manual'`.
|
||||
- `KNOWN_CONFIG_KEYS` (in `src/core/config.ts`) adds `'link_resolution'` and `'link_resolution.global_basename'` so `gbrain config set ...` accepts the new key without `--force`.
|
||||
- Tests: 38 new cases pinning the contract. `test/link-extraction.test.ts` adds 17 cases covering `WIKILINK_GENERIC_RE` shape (anchor / display / strip / escape paths), the `extractEntityRefs` pass-2c no-double-emit invariant, `resolveBasenameMatches` multi-match + index-built-once + missing-`getAllSlugs` degradation, and the `extractPageLinks` opt routing under both flag states. `test/extract-fs.test.ts` adds 11 cases for the pure-function helpers (`resolveBasenameMatchesFromSlugs`, `resolveSlugAll`) plus 3 round-trip tests of the issue's exact repro inside a PGLite brain. `test/doctor.test.ts` adds 7 cases for the new doctor check (skip / ok / warn paths + the cross-surface wiring source-grep). `test/e2e/global-basename-pglite.test.ts` adds 7 end-to-end cases against an in-memory PGLite brain covering FS-source, DB-source, and put_page auto-link paths under both flag states.
|
||||
- PR #1233 from @rayers contributed the kernel of the resolver-side approach (the generic wikilink regex + slug-tail index pattern). This PR keeps that mechanism, makes it opt-in via the new config flag, replaces the first-write-wins lookup with multi-match return, and extends the coverage to the FS-source path that the issue's repro actually hits.
|
||||
## [0.42.8.0] - 2026-06-01
|
||||
|
||||
**Scraped junk stops landing in your brain as if it were real content, and when something looks off, your agent gets told instead of being left to guess.**
|
||||
@@ -1535,6 +1599,7 @@ behavior for an automated caller, set `gbrain config set search.mcp_keyword_only
|
||||
- A page represented by its weakest chunk in vector search (the incident).
|
||||
- Same-slug pages in different sources collapsing in per-page pooling and the
|
||||
alias hop (source-isolation).
|
||||
|
||||
## [0.41.33.0] - 2026-05-29
|
||||
|
||||
**`gbrain search` and `query` can now return a tight, intent-sized set of
|
||||
@@ -1775,6 +1840,7 @@ doctor` warns about a partial migration:
|
||||
`test/doctor.test.ts` (T1 untracked-folders headline bug, T2 remote-never-shells-out trust
|
||||
boundary), `test/sync-all-parallel.test.ts` (column-path staleness).
|
||||
- **CI tooling** — `scripts/ship-remote-tests.sh` + `workflow_dispatch` on `test.yml`.
|
||||
|
||||
## [0.41.31.0] - 2026-05-30
|
||||
|
||||
**Your nightly `gbrain sync --all` cron stops getting blocked. It used to
|
||||
|
||||
@@ -222,7 +222,8 @@ strict behavior when unset.
|
||||
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
|
||||
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
|
||||
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **v0.42.10.0 (issue #972):** opt-in global-basename wikilink resolution. New `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in extractEntityRefs); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise<string[]>`. The ONE shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical) — used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — **source-scoped** so a bare `[[name]]` never resolves to a same-tail page in a DIFFERENT source (no cross-source edges; #972 is across-folders, not cross-source federation). `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug` — the TARGET, not the display alias — emits one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the pre-#972 `nullResolver` ternary). All three surfaces tag provenance: FS edges carry `link_source='wikilink-resolved'` via the new `ExtractedLink.link_source` field; `put_page` auto-link includes `wikilink-resolved` in its reconcilable-edge set so stale basename edges are removed when the wikilink (or the flag) goes away. A wikilink inside a markdown-link label (`[see [[acme]]](path)`) is masked out of pass 2c; the FS path strips code fences before scanning. New exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). Multi-match by design. The mode is opt-in (default false); `gbrain doctor`'s `link_resolution_opportunity` check (bounded scan: 1000 most-recent pages in one query, not a per-page walk) surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% ratio. Migration v113 widens the `links_link_source_check` constraint to admit the `'wikilink-resolved'` provenance value (full union with master's `'mentions'`). Pinned by `test/link-extraction.test.ts` + `test/extract-fs.test.ts` + `test/doctor.test.ts` + `test/e2e/global-basename-pglite.test.ts` (incl. the issue's exact `mkdir -p /tmp/vault/{concepts,projects}` repro, alias-target resolution, cross-source non-leak, put_page stale-edge removal). PR #1233 from @rayers contributed the regex shape + slug-tail index pattern; PR #1388 from @ukd1 (Russell Smith) made it opt-in + multi-match + FS coverage; the v0.42.10.0 wave added the codex source-correctness fixes (alias-by-target, stale-edge reconciliation, source-scoping) + the shared matcher.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`.
|
||||
- `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`.
|
||||
- `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`.
|
||||
- `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine <list|clear|scan>` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear <slug> [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`.
|
||||
|
||||
@@ -161,6 +161,29 @@ After this step:
|
||||
If a user has a very large brain (>10K pages), `extract --source db` is idempotent
|
||||
and supports `--since YYYY-MM-DD` for incremental runs.
|
||||
|
||||
### Obsidian-style bare wikilinks (opt-in)
|
||||
|
||||
If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]`
|
||||
wikilinks — where `[[struktura]]` written in one folder means the page that lives
|
||||
at `projects/struktura.md` in another — GBrain does NOT connect those by default.
|
||||
Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`,
|
||||
so a vault full of bare links shows up as a thin, broken graph. Turn on basename
|
||||
resolution so the cross-folder links connect:
|
||||
|
||||
```bash
|
||||
gbrain config set link_resolution.global_basename true
|
||||
gbrain extract links --source db # re-run so the new edges land
|
||||
```
|
||||
|
||||
`gbrain doctor` surfaces a `link_resolution_opportunity` hint with the exact count
|
||||
("47 of 60 bare wikilinks would resolve") so you know whether it's worth enabling
|
||||
before you flip it. When a bare name matches more than one page (`[[struktura]]` →
|
||||
both `projects/struktura` and `archive/struktura`), GBrain emits one edge to each
|
||||
rather than guessing a winner — review and prune the duplicates with
|
||||
`gbrain graph-query <slug>`. The mode is also honored on the filesystem-walk path
|
||||
(`gbrain extract links` with no `--source db`) and by auto-link on every future
|
||||
`put_page`.
|
||||
|
||||
## Step 5: Load Skills
|
||||
|
||||
If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace),
|
||||
|
||||
@@ -254,7 +254,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG.
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
|
||||
+26
-2
@@ -364,7 +364,8 @@ strict behavior when unset.
|
||||
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
|
||||
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
|
||||
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **v0.42.10.0 (issue #972):** opt-in global-basename wikilink resolution. New `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in extractEntityRefs); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise<string[]>`. The ONE shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical) — used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — **source-scoped** so a bare `[[name]]` never resolves to a same-tail page in a DIFFERENT source (no cross-source edges; #972 is across-folders, not cross-source federation). `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug` — the TARGET, not the display alias — emits one candidate per match tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the pre-#972 `nullResolver` ternary). All three surfaces tag provenance: FS edges carry `link_source='wikilink-resolved'` via the new `ExtractedLink.link_source` field; `put_page` auto-link includes `wikilink-resolved` in its reconcilable-edge set so stale basename edges are removed when the wikilink (or the flag) goes away. A wikilink inside a markdown-link label (`[see [[acme]]](path)`) is masked out of pass 2c; the FS path strips code fences before scanning. New exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). Multi-match by design. The mode is opt-in (default false); `gbrain doctor`'s `link_resolution_opportunity` check (bounded scan: 1000 most-recent pages in one query, not a per-page walk) surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% ratio. Migration v113 widens the `links_link_source_check` constraint to admit the `'wikilink-resolved'` provenance value (full union with master's `'mentions'`). Pinned by `test/link-extraction.test.ts` + `test/extract-fs.test.ts` + `test/doctor.test.ts` + `test/e2e/global-basename-pglite.test.ts` (incl. the issue's exact `mkdir -p /tmp/vault/{concepts,projects}` repro, alias-target resolution, cross-source non-leak, put_page stale-edge removal). PR #1233 from @rayers contributed the regex shape + slug-tail index pattern; PR #1388 from @ukd1 (Russell Smith) made it opt-in + multi-match + FS coverage; the v0.42.10.0 wave added the codex source-correctness fixes (alias-by-target, stale-edge reconciliation, source-scoping) + the shared matcher.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`.
|
||||
- `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`.
|
||||
- `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`.
|
||||
- `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine <list|clear|scan>` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear <slug> [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`.
|
||||
@@ -2397,6 +2398,29 @@ After this step:
|
||||
If a user has a very large brain (>10K pages), `extract --source db` is idempotent
|
||||
and supports `--since YYYY-MM-DD` for incremental runs.
|
||||
|
||||
### Obsidian-style bare wikilinks (opt-in)
|
||||
|
||||
If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]`
|
||||
wikilinks — where `[[struktura]]` written in one folder means the page that lives
|
||||
at `projects/struktura.md` in another — GBrain does NOT connect those by default.
|
||||
Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`,
|
||||
so a vault full of bare links shows up as a thin, broken graph. Turn on basename
|
||||
resolution so the cross-folder links connect:
|
||||
|
||||
```bash
|
||||
gbrain config set link_resolution.global_basename true
|
||||
gbrain extract links --source db # re-run so the new edges land
|
||||
```
|
||||
|
||||
`gbrain doctor` surfaces a `link_resolution_opportunity` hint with the exact count
|
||||
("47 of 60 bare wikilinks would resolve") so you know whether it's worth enabling
|
||||
before you flip it. When a bare name matches more than one page (`[[struktura]]` →
|
||||
both `projects/struktura` and `archive/struktura`), GBrain emits one edge to each
|
||||
rather than guessing a winner — review and prune the duplicates with
|
||||
`gbrain graph-query <slug>`. The mode is also honored on the filesystem-walk path
|
||||
(`gbrain extract links` with no `--source db`) and by auto-link on every future
|
||||
`put_page`.
|
||||
|
||||
## Step 5: Load Skills
|
||||
|
||||
If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace),
|
||||
@@ -2968,7 +2992,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "<query>" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "<query>" --target <slug>` traces which retrieval layer surfaces (or misses) a page.
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG.
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
|
||||
+1
-1
@@ -142,5 +142,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.8.0"
|
||||
"version": "0.42.10.0"
|
||||
}
|
||||
|
||||
@@ -255,12 +255,13 @@ export const INLINE_TIPS = [
|
||||
"`gbrain upgrade` runs post-upgrade + apply-migrations.",
|
||||
];
|
||||
|
||||
// Target ~750KB so llms-full.txt fits in ~190k-token contexts with room to spare.
|
||||
// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB once CLAUDE.md crossed 700KB:
|
||||
// it's ~540KB (77% of the bundle) and grows ~5-15KB per release with each feature's
|
||||
// Key Files annotation. Both master (v0.41.34-38 waves) and this branch (skillopt
|
||||
// wave) independently hit the 700KB line and bumped to the same 750KB. CLAUDE.md is
|
||||
// the whole point of the one-fetch bundle, so it stays inlined; the budget tracks
|
||||
// its legitimate growth. Still fits comfortably in 200k+ context models.
|
||||
// Target ~800KB so llms-full.txt fits in ~200k-token contexts with room to spare.
|
||||
// Bumped 600KB→700KB in v0.41.9.0, then 700KB→750KB once CLAUDE.md crossed 700KB,
|
||||
// then 750KB→800KB in v0.42.10.0 when the #972 global-basename Key Files annotation
|
||||
// (landing alongside master's #1696/#1699 waves) crossed the 750KB line. CLAUDE.md
|
||||
// is ~540KB+ (the bulk of the bundle) and grows ~5-15KB per release with each
|
||||
// feature's Key Files annotation. CLAUDE.md is the whole point of the one-fetch
|
||||
// bundle, so it stays inlined; the budget tracks its legitimate growth. Still fits
|
||||
// comfortably in 200k+ context models.
|
||||
// Generator prints a WARN if exceeded; ship with includeInFull=false exclusions.
|
||||
export const FULL_SIZE_BUDGET = 750_000;
|
||||
export const FULL_SIZE_BUDGET = 800_000;
|
||||
|
||||
@@ -27,6 +27,12 @@ import { gbrainPath } from '../core/config.ts';
|
||||
import { dirname, isAbsolute, join, resolve as resolvePath } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { existsSync, readFileSync, readdirSync, statSync } from 'fs';
|
||||
import {
|
||||
extractEntityRefs,
|
||||
isGlobalBasenameEnabled,
|
||||
buildBasenameIndex,
|
||||
queryBasenameIndex,
|
||||
} from '../core/link-extraction.ts';
|
||||
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
|
||||
// v0.41.32.0: remote staleness reads the stored newest_content_at column via
|
||||
// this pure comparator (no git subprocess on the HTTP MCP doctor path).
|
||||
@@ -728,6 +734,11 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// - synopsis-failures audit JSONL entries from the last 7 days
|
||||
checks.push(await checkContextualRetrievalCoverage(engine));
|
||||
|
||||
// 11a. issue #972 link_resolution_opportunity — same check the local
|
||||
// doctor runs at the equivalent slot in buildChecks. Mirrored for
|
||||
// thin-client parity so `gbrain remote doctor` sees the same hint.
|
||||
checks.push(await checkLinkResolutionOpportunity(engine));
|
||||
|
||||
// 12. v0.40.5.0 Federated Sync v2 (T12) — federation_health:
|
||||
// - Per-source lag, embed coverage, failed-job rate.
|
||||
// - Single-source brain short-circuits to ok.
|
||||
@@ -829,6 +840,116 @@ export async function checkContextualRetrievalCoverage(engine: BrainEngine): Pro
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue #972 — link_resolution_opportunity check.
|
||||
*
|
||||
* Walks every page in the brain, scans for bare wikilinks
|
||||
* (`[[struktura]]` outside DIR_PATTERN) that would resolve to at least
|
||||
* one page under global-basename mode, and surfaces a paste-ready
|
||||
* `gbrain config set link_resolution.global_basename true` hint when
|
||||
* the count is meaningful (>=5 would-resolve AND >=20% of bare
|
||||
* wikilinks have matches). Skipped silently when the flag is already
|
||||
* enabled (no signal to surface) or the brain is empty.
|
||||
*
|
||||
* Bounded scan: batch-loads the 1000 most-recent pages in one query (not a
|
||||
* per-page getPage walk) with a 60s backstop. On DB error, downgrades to an
|
||||
* informational `ok` so doctor never blocks on this check.
|
||||
*/
|
||||
export async function checkLinkResolutionOpportunity(
|
||||
engine: BrainEngine,
|
||||
progress?: ProgressReporter,
|
||||
): Promise<Check> {
|
||||
const name = 'link_resolution_opportunity';
|
||||
try {
|
||||
if (await isGlobalBasenameEnabled(engine)) {
|
||||
return { name, status: 'ok', message: 'global_basename mode already enabled' };
|
||||
}
|
||||
const allSlugs = await engine.getAllSlugs();
|
||||
if (allSlugs.size === 0) {
|
||||
return { name, status: 'ok', message: 'Brain is empty — nothing to scan' };
|
||||
}
|
||||
// Build a basename → slug[] index ONCE for the entire scan via the shared
|
||||
// builder (issue #972 codex [P2] DRY) — same key set (raw/lower/slugified)
|
||||
// as extraction, so this estimate matches what extraction actually
|
||||
// resolves. Pre-fix the doctor omitted the slugified key and undercounted.
|
||||
const basenameIndex = buildBasenameIndex(allSlugs);
|
||||
|
||||
let bareCount = 0;
|
||||
let wouldResolveCount = 0;
|
||||
const distinctTargets = new Set<string>();
|
||||
|
||||
// Issue #972 (codex [P2] perf): batch-load the most-recent N pages in ONE
|
||||
// query instead of listAllPageRefs() + a getPage() per page. The prior
|
||||
// full N-page walk hit the 60s budget every run on large brains and
|
||||
// returned a perpetual partial; this bounds the work to a fixed sample.
|
||||
const SAMPLE_LIMIT = 1000;
|
||||
const sampled = await engine.executeRaw<{ compiled_truth: string | null; timeline: string | null }>(
|
||||
`SELECT compiled_truth, timeline FROM pages WHERE deleted_at IS NULL ORDER BY id DESC LIMIT ${SAMPLE_LIMIT}`,
|
||||
);
|
||||
const totalPages = allSlugs.size;
|
||||
const sampledNote = totalPages > SAMPLE_LIMIT
|
||||
? ` (scanned the ${SAMPLE_LIMIT} most-recent of ${totalPages} pages)`
|
||||
: '';
|
||||
const deadline = Date.now() + 60_000;
|
||||
const hb = progress ? startHeartbeat(progress, `scanning ${sampled.length} pages for bare wikilinks…`) : null;
|
||||
try {
|
||||
for (const row of sampled) {
|
||||
if (Date.now() > deadline) break; // backstop; in-memory scan rarely hits it
|
||||
const content = (row.compiled_truth ?? '') + '\n' + (row.timeline ?? '');
|
||||
for (const e of extractEntityRefs(content)) {
|
||||
if (!e.needsResolution) continue;
|
||||
bareCount++;
|
||||
// Issue #972 (codex): match on the wikilink TARGET (e.slug), not
|
||||
// the display alias (e.name), via the shared query so the doctor
|
||||
// estimate equals what extraction actually resolves.
|
||||
const matches = queryBasenameIndex(basenameIndex, e.slug);
|
||||
if (matches.length > 0) {
|
||||
wouldResolveCount++;
|
||||
for (const m of matches) distinctTargets.add(m);
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
hb?.();
|
||||
}
|
||||
|
||||
if (bareCount === 0) {
|
||||
return { name, status: 'ok', message: 'No bare wikilinks found' };
|
||||
}
|
||||
if (wouldResolveCount === 0) {
|
||||
return {
|
||||
name,
|
||||
status: 'ok',
|
||||
message: `${bareCount} bare wikilink(s) found, but none have basename matches in the brain.`,
|
||||
};
|
||||
}
|
||||
const ratio = wouldResolveCount / bareCount;
|
||||
if (wouldResolveCount >= 5 && ratio >= 0.20) {
|
||||
const pct = Math.round(ratio * 100);
|
||||
return {
|
||||
name,
|
||||
status: 'warn',
|
||||
message:
|
||||
`${wouldResolveCount} of ${bareCount} bare wikilinks (${pct}%) would resolve to ` +
|
||||
`${distinctTargets.size} distinct page(s) under global_basename mode${sampledNote}. ` +
|
||||
`Enable with: gbrain config set link_resolution.global_basename true`,
|
||||
};
|
||||
}
|
||||
const pct = Math.round(ratio * 100);
|
||||
return {
|
||||
name,
|
||||
status: 'ok',
|
||||
message: `${wouldResolveCount}/${bareCount} bare wikilinks (${pct}%) would resolve — below the 20% / 5-link threshold for surfacing a hint${sampledNote}.`,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
name,
|
||||
status: 'ok',
|
||||
message: `Skipped (${e instanceof Error ? e.message : String(e)})`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkAbandonedThreads(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ count: number }>(
|
||||
@@ -6166,6 +6287,13 @@ export async function buildChecks(
|
||||
// v0.37.0 brainstorm_health — migration v79, track_retrieval, calibration cold-start.
|
||||
progress.heartbeat('brainstorm_health');
|
||||
checks.push(await checkBrainstormHealth(engine));
|
||||
// issue #972 link_resolution_opportunity — full scan: count bare wikilinks
|
||||
// that would resolve under global_basename mode. Surfaces a paste-ready
|
||||
// enable hint when ≥5 hits AND ≥20% of bare wikilinks would resolve.
|
||||
// Skipped silently when the flag is already enabled. Bounded by a 60s
|
||||
// budget so a huge brain never wedges doctor on this check.
|
||||
progress.heartbeat('link_resolution_opportunity');
|
||||
checks.push(await checkLinkResolutionOpportunity(engine, progress));
|
||||
// v0.36.0.0 (A5): ZE embedding key health + schema/config width consistency.
|
||||
progress.heartbeat('ze_embedding_health');
|
||||
checks.push(await checkZeEmbeddingHealth(engine));
|
||||
|
||||
+124
-19
@@ -35,7 +35,9 @@ import type { PageType } from '../core/types.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import {
|
||||
extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver,
|
||||
extractFrontmatterLinks, LINK_EXTRACTOR_VERSION_TS,
|
||||
extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS,
|
||||
WIKILINK_BASENAME_LINK_TYPE,
|
||||
buildBasenameIndex, queryBasenameIndex, stripCodeBlocks,
|
||||
type UnresolvedFrontmatterRef, type LinkCandidate,
|
||||
} from '../core/link-extraction.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
@@ -156,6 +158,11 @@ export interface ExtractedLink {
|
||||
to_slug: string;
|
||||
link_type: string;
|
||||
context: string;
|
||||
// Issue #972: provenance for FS-source edges. Set to 'wikilink-resolved'
|
||||
// on basename-matched bare wikilinks so the FS path tags them the same way
|
||||
// the DB / put_page paths do. Undefined for ordinary markdown edges (the
|
||||
// engine defaults those to 'markdown').
|
||||
link_source?: string;
|
||||
}
|
||||
|
||||
export interface ExtractedTimelineEntry {
|
||||
@@ -273,6 +280,56 @@ export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set<st
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue #972: return every slug whose basename matches `name` (the
|
||||
* final path segment, with case-insensitive + slugified fallback keys).
|
||||
* Pure-function variant of the resolver's `resolveBasenameMatches` that
|
||||
* reads a pre-loaded Set directly — no engine call. Used by the
|
||||
* FS-source path's `resolveSlugAll`.
|
||||
*
|
||||
* Matches are deterministically sorted (shortest-slug first, then
|
||||
* lexical) so repeated runs over the same brain produce stable edges.
|
||||
* Returns `[]` on empty input or no matches.
|
||||
*/
|
||||
export function resolveBasenameMatchesFromSlugs(
|
||||
name: string, allSlugs: Set<string>,
|
||||
): string[] {
|
||||
// Issue #972 (codex [P2] DRY): delegate to the shared matcher so the FS
|
||||
// path keys + sorts identically to the resolver and doctor. (Per-call
|
||||
// index build is O(N), the same cost as the prior inline scan.)
|
||||
return queryBasenameIndex(buildBasenameIndex(allSlugs), name);
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue #972: multi-match variant of `resolveSlug`. Always tries the
|
||||
* existing ancestor walk first (preserving the v0.10.1 behavior); on
|
||||
* miss, falls back to basename lookup against `allSlugs` when
|
||||
* `opts.globalBasename === true`. Returns an array so the caller emits
|
||||
* one graph edge per matching page.
|
||||
*
|
||||
* Return shape:
|
||||
* - Ancestor walk hits → `[ancestor_match]` (length 1)
|
||||
* - Ancestor walk misses + globalBasename off → `[]`
|
||||
* - Ancestor walk misses + globalBasename on + basename hits → all matches
|
||||
* - Ancestor walk misses + globalBasename on + no basename hits → `[]`
|
||||
*/
|
||||
export function resolveSlugAll(
|
||||
fileDir: string, relTarget: string, allSlugs: Set<string>,
|
||||
opts: { globalBasename?: boolean } = {},
|
||||
): string[] {
|
||||
const direct = resolveSlug(fileDir, relTarget, allSlugs);
|
||||
if (direct !== null) return [direct];
|
||||
if (!opts.globalBasename) return [];
|
||||
// Strip .md suffix + dirname so `[[struktura]]` (relTarget=`struktura.md`)
|
||||
// and `[[notes/struktura]]` (relTarget=`notes/struktura.md`) both query
|
||||
// for the basename `struktura`.
|
||||
const targetNoExt = relTarget.endsWith('.md') ? relTarget.slice(0, -3) : relTarget;
|
||||
const basename = targetNoExt.includes('/')
|
||||
? targetNoExt.slice(targetNoExt.lastIndexOf('/') + 1)
|
||||
: targetNoExt;
|
||||
return resolveBasenameMatchesFromSlugs(basename, allSlugs);
|
||||
}
|
||||
|
||||
/**
|
||||
* Directory-based link-type inference for the fs-source path.
|
||||
*
|
||||
@@ -319,20 +376,49 @@ function parseFrontmatterFromContent(content: string, relPath: string): Record<s
|
||||
*/
|
||||
export async function extractLinksFromFile(
|
||||
content: string, relPath: string, allSlugs: Set<string>,
|
||||
opts?: { includeFrontmatter?: boolean },
|
||||
opts?: { includeFrontmatter?: boolean; globalBasename?: boolean },
|
||||
): Promise<ExtractedLink[]> {
|
||||
const links: ExtractedLink[] = [];
|
||||
const slug = pathToSlug(relPath);
|
||||
const fileDir = dirname(relPath);
|
||||
const fm = parseFrontmatterFromContent(content, relPath);
|
||||
// Issue #972: globalBasename routes bare `[[name]]` wikilinks through
|
||||
// basename lookup against allSlugs when the ancestor walk fails. Off
|
||||
// by default for back-compat with the v0.10.1 ancestor-only behavior.
|
||||
const globalBasename = opts?.globalBasename ?? false;
|
||||
|
||||
for (const { name, relTarget } of extractMarkdownLinks(content)) {
|
||||
const resolved = resolveSlug(fileDir, relTarget, allSlugs);
|
||||
if (resolved !== null) {
|
||||
// Issue #972 (codex [P2]): strip code fences before scanning so a
|
||||
// `[[name]]` inside a code block doesn't create an FS edge. Mirrors the
|
||||
// DB path, which goes through extractEntityRefs (which strips internally).
|
||||
const scanContent = stripCodeBlocks(content);
|
||||
|
||||
for (const { name, relTarget } of extractMarkdownLinks(scanContent)) {
|
||||
const resolvedSlugs = resolveSlugAll(fileDir, relTarget, allSlugs, { globalBasename });
|
||||
if (resolvedSlugs.length === 0) continue;
|
||||
// Single hit on the ancestor path → emit one edge with the inferred
|
||||
// verb type. Multiple hits (only possible when globalBasename is on
|
||||
// AND ancestor walk missed) → emit one edge per match, all tagged
|
||||
// `wikilink_basename` so users can audit via `gbrain graph-query
|
||||
// <slug> --type wikilink_basename`.
|
||||
const isBasename = resolvedSlugs.length > 1
|
||||
|| (globalBasename && resolvedSlugs.length === 1
|
||||
&& resolveSlug(fileDir, relTarget, allSlugs) === null);
|
||||
for (const target of resolvedSlugs) {
|
||||
// Issue #972 (codex [P2]): drop a basename self-loop ([[own-tail]] on
|
||||
// its own page resolving back to itself).
|
||||
if (isBasename && target === slug) continue;
|
||||
links.push({
|
||||
from_slug: slug, to_slug: resolved,
|
||||
link_type: inferTypeByDir(fileDir, dirname(resolved), fm),
|
||||
context: `markdown link: [${name}]`,
|
||||
from_slug: slug,
|
||||
to_slug: target,
|
||||
link_type: isBasename
|
||||
? WIKILINK_BASENAME_LINK_TYPE
|
||||
: inferTypeByDir(fileDir, dirname(target), fm),
|
||||
context: isBasename
|
||||
? `wikilink (basename match): [${name}]`
|
||||
: `markdown link: [${name}]`,
|
||||
// Issue #972: tag basename edges so the FS path matches DB/put_page
|
||||
// provenance and migration v112's widened CHECK is exercised here too.
|
||||
link_source: isBasename ? 'wikilink-resolved' : undefined,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -860,6 +946,9 @@ async function extractForSlugs(
|
||||
let timelineCreated = 0;
|
||||
let pagesProcessed = 0;
|
||||
|
||||
// Issue #972: read the basename flag once per extract run.
|
||||
const globalBasename = await isGlobalBasenameEnabled(engine);
|
||||
|
||||
const linkBatch: LinkBatchInput[] = [];
|
||||
const timelineBatch: TimelineBatchInput[] = [];
|
||||
|
||||
@@ -912,7 +1001,7 @@ async function extractForSlugs(
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
|
||||
if (doLinks) {
|
||||
const links = await extractLinksFromFile(content, relPath, allSlugs);
|
||||
const links = await extractLinksFromFile(content, relPath, allSlugs, { globalBasename });
|
||||
for (const link of links) {
|
||||
if (dryRun) {
|
||||
if (!jsonMode) console.log(` ${link.from_slug} → ${link.to_slug} (${link.link_type})`);
|
||||
@@ -963,6 +1052,11 @@ async function extractLinksFromDir(
|
||||
const files = walkMarkdownFiles(brainDir);
|
||||
const allSlugs = new Set(files.map(f => pathToSlug(f.relPath)));
|
||||
|
||||
// Issue #972: read once before the walk so the per-file calls don't
|
||||
// re-query the DB. globalBasename = true emits one edge per basename
|
||||
// match for bare wikilinks like `[[struktura]]`.
|
||||
const globalBasename = await isGlobalBasenameEnabled(engine);
|
||||
|
||||
// Progress stream on stderr (separate from the action-events --json writes
|
||||
// to stdout, which tests grep for). Rate-gated; respects global --quiet /
|
||||
// --progress-json flags.
|
||||
@@ -998,7 +1092,7 @@ async function extractLinksFromDir(
|
||||
onItem: async (file) => {
|
||||
try {
|
||||
const content = readFileSync(file.path, 'utf-8');
|
||||
const links = await extractLinksFromFile(content, file.relPath, allSlugs);
|
||||
const links = await extractLinksFromFile(content, file.relPath, allSlugs, { globalBasename });
|
||||
for (const link of links) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${link.from_slug}::${link.to_slug}::${link.link_type}`;
|
||||
@@ -1107,14 +1201,16 @@ export async function extractLinksForSlugs(
|
||||
const linkOpts = opts?.sourceId
|
||||
? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId, originSourceId: opts.sourceId }
|
||||
: undefined;
|
||||
// Issue #972: same flag as the standalone extract path.
|
||||
const globalBasename = await isGlobalBasenameEnabled(engine);
|
||||
let created = 0;
|
||||
for (const slug of slugs) {
|
||||
const filePath = join(repoPath, slug + '.md');
|
||||
if (!existsSync(filePath)) continue;
|
||||
try {
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
for (const link of await extractLinksFromFile(content, slug + '.md', allSlugs)) {
|
||||
try { await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type, undefined, undefined, undefined, linkOpts); created++; } catch { /* skip */ } // gbrain-allow-direct-insert: gbrain extract single-row fallback when batch path declines a row
|
||||
for (const link of await extractLinksFromFile(content, slug + '.md', allSlugs, { globalBasename })) {
|
||||
try { await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type, link.link_source, undefined, undefined, linkOpts); created++; } catch { /* skip */ } // gbrain-allow-direct-insert: gbrain extract single-row fallback when batch path declines a row
|
||||
}
|
||||
} catch { /* skip */ }
|
||||
}
|
||||
@@ -1170,12 +1266,19 @@ async function extractLinksFromDB(
|
||||
// Batch resolver: pg_trgm + exact only, NO search fallback. Dodges the
|
||||
// N-thousand API call trap on 46K-page brains. Resolver has a per-run
|
||||
// cache so duplicate names (same person appearing on many pages) resolve
|
||||
// once, not once per mention.
|
||||
const resolver = makeResolver(engine, { mode: 'batch' });
|
||||
// once, not once per mention. Used for BOTH the frontmatter pass (gated
|
||||
// by `includeFrontmatter` via `opts.skipFrontmatter` on extractPageLinks)
|
||||
// AND the issue-#972 global-basename pass (gated by `globalBasename`).
|
||||
// Replaces the pre-issue-#972 `nullResolver` ternary — that synthetic
|
||||
// resolver lacked `resolveBasenameMatches`, so we always pass the real
|
||||
// one and let extractPageLinks's opts gate which pass actually runs.
|
||||
// Issue #972 (codex [P1]): scope basename resolution to the source being
|
||||
// extracted so bare wikilinks don't resolve across unrelated sources.
|
||||
const resolver = makeResolver(engine, { mode: 'batch', sourceId: sourceIdFilter });
|
||||
const unresolved: UnresolvedFrontmatterRef[] = [];
|
||||
const nullResolver = {
|
||||
resolve: async () => null as string | null,
|
||||
};
|
||||
// Issue #972: opt-in global-basename wikilink resolution. Read once
|
||||
// per extract run; threaded into each extractPageLinks call.
|
||||
const globalBasename = await isGlobalBasenameEnabled(engine);
|
||||
// v0.32.8: listAllPageRefs enumerates (slug, source_id) so we can thread
|
||||
// sourceId to getPage AND build a cross-source resolution map for link
|
||||
// disambiguation. Pre-fix used getAllSlugs() which collapsed
|
||||
@@ -1252,9 +1355,11 @@ async function extractLinksFromDB(
|
||||
// --include-frontmatter default OFF in v0.13 (codex tension 5, back-compat).
|
||||
// Migration orchestrator explicitly enables it for the one-time backfill;
|
||||
// user-invoked `gbrain extract links` stays outgoing-only.
|
||||
const activeResolver = includeFrontmatter ? resolver : nullResolver;
|
||||
// Issue #972: globalBasename routes bare `[[name]]` wikilinks through
|
||||
// basename lookup; off by default for back-compat.
|
||||
const extracted = await extractPageLinks(
|
||||
slug, fullContent, page.frontmatter, page.type, activeResolver,
|
||||
slug, fullContent, page.frontmatter, page.type, resolver,
|
||||
{ skipFrontmatter: !includeFrontmatter, globalBasename },
|
||||
);
|
||||
unresolved.push(...extracted.unresolved);
|
||||
|
||||
|
||||
@@ -736,6 +736,9 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// Misc
|
||||
'artifacts_sync_mode',
|
||||
'cross_project_learnings',
|
||||
// Link resolution (issue #972)
|
||||
'link_resolution',
|
||||
'link_resolution.global_basename',
|
||||
];
|
||||
|
||||
/**
|
||||
|
||||
@@ -85,6 +85,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
'image_assets',
|
||||
'integrity',
|
||||
'jsonb_integrity',
|
||||
'link_resolution_opportunity',
|
||||
'links_extraction_lag',
|
||||
'markdown_body_completeness',
|
||||
'nightly_quality_probe_health',
|
||||
|
||||
+273
-5
@@ -46,8 +46,32 @@ export interface EntityRef {
|
||||
* - sourceId null → 'unqualified'
|
||||
*/
|
||||
sourceId?: string | null;
|
||||
/**
|
||||
* Issue #972: set when the ref came from the generic `[[bare-name]]`
|
||||
* pass (not gated by DIR_PATTERN). The `slug` field holds the literal
|
||||
* wikilink text — callers MUST run it through a SlugResolver's
|
||||
* `resolveBasenameMatches` (gated by `link_resolution.global_basename`)
|
||||
* before persisting. Untagged refs already match a known entity dir
|
||||
* (people/, companies/, etc.) and need no resolution.
|
||||
*
|
||||
* One ref tagged `needsResolution: true` may resolve to MULTIPLE target
|
||||
* slugs when multiple pages share the basename. The caller emits one
|
||||
* LinkCandidate per resolved match.
|
||||
*/
|
||||
needsResolution?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue #972: edge type stamped on graph rows produced by the
|
||||
* global-basename wikilink resolution path. Distinct from the verb-
|
||||
* inferred types (`mentions`, `works_at`, etc.) so users can audit or
|
||||
* prune via `gbrain graph-query <slug> --type wikilink_basename`.
|
||||
*
|
||||
* All edges from this path also carry `linkSource: 'wikilink-resolved'`
|
||||
* (the link-source provenance is the why; this edge type is the what).
|
||||
*/
|
||||
export const WIKILINK_BASENAME_LINK_TYPE = 'wikilink_basename';
|
||||
|
||||
/** v0.17.0: how a link's target source was pinned at extraction time. */
|
||||
export type LinkResolutionType = 'qualified' | 'unqualified';
|
||||
|
||||
@@ -105,6 +129,38 @@ const QUALIFIED_WIKILINK_RE = new RegExp(
|
||||
'g',
|
||||
);
|
||||
|
||||
/**
|
||||
* Issue #972: generic Obsidian-style wikilink — matches `[[anything]]`
|
||||
* with no DIR_PATTERN gate. Wiki/topic/learning pages frequently link
|
||||
* via `[[bare-name]]` (`[[Fast-Weigh]]`, `[[2026-05-07-cost-plan]]`,
|
||||
* `[[struktura]]`) where the literal text isn't a canonical slug; the
|
||||
* resolver is responsible for matching it to the real page via
|
||||
* basename uniqueness.
|
||||
*
|
||||
* Used AFTER QUALIFIED_WIKILINK_RE and WIKILINK_RE pick off their
|
||||
* cases — the masked-ranges pass in extractEntityRefs prevents
|
||||
* double-emission. Refs from this pass are tagged
|
||||
* `needsResolution: true` so callers know to run them through a
|
||||
* SlugResolver before the page-existence check.
|
||||
*
|
||||
* Regex shape is the union of the two existing wikilink regexes,
|
||||
* minus the DIR_PATTERN gate. The character class disallows pipe,
|
||||
* close-bracket, hash (anchor separator), newline, and open-bracket
|
||||
* so we never match across line breaks or capture nested `[[...]]`.
|
||||
*
|
||||
* Adapted from PR #1233 (rayers).
|
||||
*/
|
||||
const WIKILINK_GENERIC_RE = /\[\[([^|\]#\n[]+?)(?:#[^|\]]*?)?(?:\|([^\]]+?))?\]\]/g;
|
||||
|
||||
/**
|
||||
* Issue #972 (codex [P2]): a markdown link whose LABEL contains a wikilink,
|
||||
* e.g. `[see [[acme]]](companies/acme.md)`. Pass-1's ENTITY_REF_RE can't match
|
||||
* the nested `]]` so it never spans this; without masking, the generic 2c pass
|
||||
* would emit a stray basename ref for the inner `[[acme]]`. Mask the whole
|
||||
* span out of the 2c scan so a wikilink inside a markdown label is inert.
|
||||
*/
|
||||
const MARKDOWN_LABEL_WIKILINK_RE = /\[[^\]\n]*\[\[[^\]\n]+\]\][^\]\n]*\]\([^)\n]+\)/g;
|
||||
|
||||
/**
|
||||
* Strip fenced code blocks (```...```) and inline code (`...`) from markdown,
|
||||
* replacing them with whitespace of equivalent length. Preserves byte offsets
|
||||
@@ -250,6 +306,7 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
// Markdown links have no source-qualification syntax — they're
|
||||
// always unqualified. Omit sourceId so the shape stays compatible
|
||||
// with pre-v0.17 consumers doing strict equality.
|
||||
const markdownRanges: Array<[number, number]> = [];
|
||||
const mdPattern = new RegExp(ENTITY_REF_RE.source, ENTITY_REF_RE.flags);
|
||||
while ((match = mdPattern.exec(stripped)) !== null) {
|
||||
const name = match[1];
|
||||
@@ -257,6 +314,7 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
const slug = fullPath;
|
||||
const dir = fullPath.split('/')[0];
|
||||
refs.push({ name, slug, dir });
|
||||
markdownRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// 2a. v0.17.0 qualified wikilinks: [[source-id:path]] or [[source-id:path|Display]]
|
||||
@@ -278,6 +336,7 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
|
||||
// 2b. Unqualified Obsidian wikilinks: [[path]] or [[path|Display Text]]
|
||||
// Same shape rule: omit sourceId when unqualified.
|
||||
const unqualifiedRanges: Array<[number, number]> = [];
|
||||
const unmasked = maskRanges(stripped, qualifiedRanges);
|
||||
const wikiPattern = new RegExp(WIKILINK_RE.source, WIKILINK_RE.flags);
|
||||
while ((match = wikiPattern.exec(unmasked)) !== null) {
|
||||
@@ -288,6 +347,38 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
const displayName = (match[2] || slug).trim();
|
||||
const dir = slug.split('/')[0];
|
||||
refs.push({ name: displayName, slug, dir });
|
||||
unqualifiedRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
|
||||
// 2c. Issue #972: generic `[[bare-name]]` wikilinks not gated by
|
||||
// DIR_PATTERN. Wiki/topic/learning content frequently uses these
|
||||
// where the bare text isn't a canonical brain slug. Tagged
|
||||
// `needsResolution: true` so the caller routes them through a
|
||||
// SlugResolver's `resolveBasenameMatches` before persisting.
|
||||
// Mask out 2a/2b ranges so we don't double-emit; skip qualified-
|
||||
// syntax tokens (contain `:`) that would be 2a's job. Issue #972
|
||||
// (codex [P2]): ALSO mask pass-1 markdown-link ranges so a wikilink
|
||||
// inside a markdown label — `[see [[acme]]](companies/acme.md)` —
|
||||
// doesn't spawn a stray generic basename ref from inside the label.
|
||||
const labelWikilinkRanges: Array<[number, number]> = [];
|
||||
const labelWlPattern = new RegExp(MARKDOWN_LABEL_WIKILINK_RE.source, MARKDOWN_LABEL_WIKILINK_RE.flags);
|
||||
while ((match = labelWlPattern.exec(stripped)) !== null) {
|
||||
labelWikilinkRanges.push([match.index, match.index + match[0].length]);
|
||||
}
|
||||
const genericMasked = maskRanges(
|
||||
stripped,
|
||||
[...markdownRanges, ...qualifiedRanges, ...unqualifiedRanges, ...labelWikilinkRanges],
|
||||
);
|
||||
const genericPattern = new RegExp(WIKILINK_GENERIC_RE.source, WIKILINK_GENERIC_RE.flags);
|
||||
while ((match = genericPattern.exec(genericMasked)) !== null) {
|
||||
let slug = match[1].trim();
|
||||
if (!slug) continue;
|
||||
if (slug.includes('://')) continue;
|
||||
if (slug.includes(':')) continue; // qualified-syntax token; 2a owns these
|
||||
if (slug.endsWith('.md')) slug = slug.slice(0, -3);
|
||||
const displayName = (match[2] || slug).trim();
|
||||
const dir = slug.includes('/') ? slug.split('/')[0] : '';
|
||||
refs.push({ name: displayName, slug, dir, needsResolution: true });
|
||||
}
|
||||
|
||||
return refs;
|
||||
@@ -376,11 +467,44 @@ export async function extractPageLinks(
|
||||
frontmatter: Record<string, unknown>,
|
||||
pageType: PageType,
|
||||
resolver: SlugResolver,
|
||||
opts: { globalBasename?: boolean; skipFrontmatter?: boolean } = {},
|
||||
): Promise<PageLinksResult> {
|
||||
const candidates: LinkCandidate[] = [];
|
||||
|
||||
// 1. Markdown entity refs.
|
||||
for (const ref of extractEntityRefs(content)) {
|
||||
// Issue #972: refs from the generic `[[bare-name]]` pass carry the
|
||||
// literal wikilink text, not a real page slug. When global_basename
|
||||
// mode is on AND the resolver implements basename lookup, resolve
|
||||
// to every matching page and emit one candidate per match. When the
|
||||
// flag is off (default), drop silently — back-compat with the
|
||||
// pre-v0.40.8.2 behavior of dropping bare wikilinks outside
|
||||
// DIR_PATTERN.
|
||||
if (ref.needsResolution) {
|
||||
if (!opts.globalBasename || typeof resolver.resolveBasenameMatches !== 'function') {
|
||||
continue;
|
||||
}
|
||||
// Issue #972 (codex): resolve by the wikilink TARGET (ref.slug — the
|
||||
// text inside `[[...]]` before any `|`), NOT the display alias
|
||||
// (ref.name = match[2]). `[[struktura|the project]]` must resolve
|
||||
// `struktura`, not "the project". The display text is for context only.
|
||||
const matches = await resolver.resolveBasenameMatches(ref.slug);
|
||||
if (matches.length === 0) continue;
|
||||
const idx = content.indexOf(ref.slug);
|
||||
const context = idx >= 0 ? excerpt(content, idx, 240) : ref.name;
|
||||
for (const matched of matches) {
|
||||
// Issue #972 (codex [P2]): a basename `[[own-tail]]` on its own page
|
||||
// resolves back to itself — drop the self-loop.
|
||||
if (matched === slug) continue;
|
||||
candidates.push({
|
||||
targetSlug: matched,
|
||||
linkType: WIKILINK_BASENAME_LINK_TYPE,
|
||||
context,
|
||||
linkSource: 'wikilink-resolved',
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const idx = content.indexOf(ref.name);
|
||||
// Wider context window (240 chars vs original 80) catches verbs that
|
||||
// appear at sentence-or-paragraph distance from the slug — common in
|
||||
@@ -418,12 +542,28 @@ export async function extractPageLinks(
|
||||
}
|
||||
|
||||
// 3. Frontmatter-derived edges (v0.13). Includes the legacy `source:`
|
||||
// field along with the full field map.
|
||||
const fm = await extractFrontmatterLinks(slug, pageType, frontmatter, resolver);
|
||||
candidates.push(...fm.candidates);
|
||||
// field along with the full field map. Caller can suppress via
|
||||
// `opts.skipFrontmatter` — used by `extractLinksFromDB` to keep
|
||||
// `--include-frontmatter` off semantics while still passing a real
|
||||
// resolver (needed for `resolveBasenameMatches` in global-basename
|
||||
// mode). Pre-issue-#972 this gating lived in the caller via a
|
||||
// synthetic `nullResolver`; that pattern broke once the bare-wikilink
|
||||
// path needed `resolveBasenameMatches` on the real resolver.
|
||||
let fmUnresolved: UnresolvedFrontmatterRef[] = [];
|
||||
if (!opts.skipFrontmatter) {
|
||||
const fm = await extractFrontmatterLinks(slug, pageType, frontmatter, resolver);
|
||||
candidates.push(...fm.candidates);
|
||||
fmUnresolved = fm.unresolved;
|
||||
}
|
||||
|
||||
// Within-page dedup: same (fromSlug, targetSlug, linkType, linkSource)
|
||||
// collapses to one entry. First occurrence wins.
|
||||
// Issue #972 (codex P2d, decided): a qualified `[[companies/acme]]` (typed
|
||||
// markdown edge) and a bare `[[acme]]` (wikilink-resolved edge) to the SAME
|
||||
// target are KEPT as separate rows — they carry different provenance
|
||||
// (link_source) and link_type, and the audit trail (which kind of reference
|
||||
// created the edge) is worth more than collapsing them. graph-query callers
|
||||
// that want a unique target set dedup on to_slug themselves.
|
||||
const seen = new Set<string>();
|
||||
const result: LinkCandidate[] = [];
|
||||
for (const c of candidates) {
|
||||
@@ -432,7 +572,7 @@ export async function extractPageLinks(
|
||||
seen.add(key);
|
||||
result.push(c);
|
||||
}
|
||||
return { candidates: result, unresolved: fm.unresolved };
|
||||
return { candidates: result, unresolved: fmUnresolved };
|
||||
}
|
||||
|
||||
/** Excerpt a window of `width` chars around `idx`, collapsed to one line. */
|
||||
@@ -655,6 +795,67 @@ export interface SlugResolver {
|
||||
* extract/put_page summary so the user can see the gap.
|
||||
*/
|
||||
resolve(name: string, dirHint?: string | string[]): Promise<string | null>;
|
||||
/**
|
||||
* Issue #972: return every slug whose basename (final `/`-segment, or
|
||||
* the whole slug if it has no `/`) matches `name`. Multi-match by
|
||||
* design — `[[struktura]]` referencing both `projects/struktura` and
|
||||
* `archive/struktura` returns both; the caller emits one graph edge
|
||||
* per result. Returns `[]` when no matches.
|
||||
*
|
||||
* Optional so existing SlugResolver consumers (e.g. synthetic
|
||||
* frontmatter-only resolvers) keep working without code changes.
|
||||
* Implementations should make this cheap to call repeatedly within
|
||||
* a single extract run (build the index once, reuse it).
|
||||
*/
|
||||
resolveBasenameMatches?(name: string): Promise<string[]>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue #972 (codex [P2] DRY): the ONE basename matcher. Before this, three
|
||||
* surfaces (makeResolver, FS `resolveBasenameMatchesFromSlugs`, the doctor
|
||||
* `link_resolution_opportunity` check) each hand-rolled their own key set +
|
||||
* sort, and they drifted — the doctor omitted the slugified key, so its
|
||||
* "N would resolve" estimate undercounted what extraction actually produces.
|
||||
* All three now build/query through these two functions so they cannot drift.
|
||||
*
|
||||
* Keying: raw tail + lowercase tail + slugified tail. A slug's tail is its
|
||||
* final `/`-segment (or the whole slug when it has no `/`).
|
||||
*/
|
||||
export function normalizeBasename(s: string): string {
|
||||
return s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-');
|
||||
}
|
||||
|
||||
/** Stable order: shorter slug first (likely closer to brain root), then lexical. */
|
||||
function basenameSort(a: string, b: string): number {
|
||||
return (a.length - b.length) || a.localeCompare(b);
|
||||
}
|
||||
|
||||
/** Build a `key → slug[]` index over a slug collection. Keys: raw/lower/slugified tail. */
|
||||
export function buildBasenameIndex(slugs: Iterable<string>): Map<string, string[]> {
|
||||
const idx = new Map<string, string[]>();
|
||||
const addKey = (key: string, slug: string) => {
|
||||
const existing = idx.get(key);
|
||||
if (existing) { if (!existing.includes(slug)) existing.push(slug); }
|
||||
else idx.set(key, [slug]);
|
||||
};
|
||||
for (const slug of slugs) {
|
||||
const tail = slug.includes('/') ? slug.slice(slug.lastIndexOf('/') + 1) : slug;
|
||||
addKey(tail, slug);
|
||||
const lower = tail.toLowerCase();
|
||||
if (lower !== tail) addKey(lower, slug);
|
||||
const slugified = normalizeBasename(tail);
|
||||
if (slugified && slugified !== tail && slugified !== lower) addKey(slugified, slug);
|
||||
}
|
||||
return idx;
|
||||
}
|
||||
|
||||
/** Look a name up in a basename index (raw → lower → slugified), stable-sorted. */
|
||||
export function queryBasenameIndex(idx: Map<string, string[]>, name: string): string[] {
|
||||
if (!name || typeof name !== 'string') return [];
|
||||
const trimmed = name.trim();
|
||||
if (!trimmed) return [];
|
||||
const hit = idx.get(trimmed) ?? idx.get(trimmed.toLowerCase()) ?? idx.get(normalizeBasename(trimmed));
|
||||
return hit ? [...hit].sort(basenameSort) : [];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -674,13 +875,52 @@ export interface SlugResolver {
|
||||
*/
|
||||
export function makeResolver(
|
||||
engine: BrainEngine,
|
||||
opts: { mode: 'batch' | 'live' } = { mode: 'live' },
|
||||
opts: { mode: 'batch' | 'live'; sourceId?: string } = { mode: 'live' },
|
||||
): SlugResolver {
|
||||
const cache = new Map<string, string | null>();
|
||||
|
||||
const norm = (s: string) => s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-');
|
||||
|
||||
// Issue #972: lazy-built basename → slug[] index for global-basename
|
||||
// resolution. Built on first call to `resolveBasenameMatches`; reused
|
||||
// for the rest of the resolver instance's lifetime. Cost is bounded
|
||||
// (one `engine.getAllSlugs()` call per extract run / put_page). The
|
||||
// index keys BOTH the raw tail and the slugified-tail so a wikilink
|
||||
// `[[Fast-Weigh]]` matches a page slugged `companies/fast-weigh` as
|
||||
// well as `notes/Fast-Weigh`.
|
||||
let basenameIndex: Map<string, string[]> | null = null;
|
||||
async function ensureBasenameIndex(): Promise<Map<string, string[]>> {
|
||||
if (basenameIndex !== null) return basenameIndex;
|
||||
const idx = new Map<string, string[]>();
|
||||
if (typeof engine.getAllSlugs !== 'function') {
|
||||
basenameIndex = idx;
|
||||
return idx;
|
||||
}
|
||||
try {
|
||||
// Issue #972 (codex [P1]): scope the basename index to the resolver's
|
||||
// source. getAllSlugs({sourceId}) keeps wikilink resolution from
|
||||
// spanning unrelated sources — a bare [[name]] must NOT resolve to a
|
||||
// same-tail page in a DIFFERENT source and create a cross-source edge.
|
||||
// #972 is "global basename across folders," not "cross-source federation."
|
||||
const all = await engine.getAllSlugs(opts.sourceId ? { sourceId: opts.sourceId } : undefined);
|
||||
// Issue #972 (codex [P2] DRY): one shared index builder for all surfaces.
|
||||
basenameIndex = buildBasenameIndex(all);
|
||||
return basenameIndex;
|
||||
} catch {
|
||||
// Index build failed — empty map → resolveBasenameMatches finds
|
||||
// nothing, resolver continues as if the flag was off. Never throw.
|
||||
}
|
||||
basenameIndex = idx;
|
||||
return idx;
|
||||
}
|
||||
|
||||
return {
|
||||
async resolveBasenameMatches(name: string): Promise<string[]> {
|
||||
// Issue #972 (codex [P2] DRY): shared query so resolver + FS + doctor
|
||||
// return the same matches in the same stable order.
|
||||
return queryBasenameIndex(await ensureBasenameIndex(), name);
|
||||
},
|
||||
|
||||
async resolve(name: string, dirHint?: string | string[]): Promise<string | null> {
|
||||
if (!name || typeof name !== 'string') return null;
|
||||
const trimmed = name.trim();
|
||||
@@ -948,3 +1188,31 @@ export async function isAutoTimelineEnabled(engine: BrainEngine): Promise<boolea
|
||||
const normalized = val.trim().toLowerCase();
|
||||
return !['false', '0', 'no', 'off'].includes(normalized);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the `link_resolution.global_basename` config flag. Defaults to
|
||||
* FALSE (opt-in only; existing brains keep ancestor-walk resolution).
|
||||
*
|
||||
* When TRUE: bare wikilinks like `[[struktura]]` (no DIR_PATTERN prefix)
|
||||
* that don't resolve via the existing path are matched against every
|
||||
* page's basename. Each unique match emits a graph edge tagged
|
||||
* `linkType: 'wikilink_basename'`. Multiple matches emit multiple edges.
|
||||
*
|
||||
* Resolution order (highest → lowest):
|
||||
* 1. Env var `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME=1` (operator override)
|
||||
* 2. DB plane via `engine.getConfig('link_resolution.global_basename')`
|
||||
* 3. Default false
|
||||
*
|
||||
* Closes https://github.com/garrytan/gbrain/issues/972.
|
||||
*/
|
||||
export async function isGlobalBasenameEnabled(engine: BrainEngine): Promise<boolean> {
|
||||
const envVal = process.env.GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME;
|
||||
if (envVal != null) {
|
||||
const normalized = envVal.trim().toLowerCase();
|
||||
return ['1', 'true', 'yes', 'on'].includes(normalized);
|
||||
}
|
||||
const val = await engine.getConfig('link_resolution.global_basename');
|
||||
if (val == null) return false;
|
||||
const normalized = val.trim().toLowerCase();
|
||||
return ['1', 'true', 'yes', 'on'].includes(normalized);
|
||||
}
|
||||
|
||||
@@ -5088,6 +5088,38 @@ export const MIGRATIONS: Migration[] = [
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 113,
|
||||
name: 'links_link_source_widen_for_wikilink_basename',
|
||||
// Issue #972: opt-in global-basename wikilink resolution (bare [[name]]
|
||||
// resolved by slug tail) emits edges tagged
|
||||
// `link_source = 'wikilink-resolved'`. Widen the CHECK to admit it.
|
||||
//
|
||||
// The FULL set is enumerated here — not just the new value — because
|
||||
// DROP + re-ADD replaces the whole constraint. v95
|
||||
// (links_link_source_check_includes_mentions) added 'mentions'; since
|
||||
// this migration runs AFTER v95, omitting 'mentions' would silently
|
||||
// clobber that widening. Keep both branches in sync with src/schema.sql
|
||||
// and src/core/pglite-schema.ts.
|
||||
//
|
||||
// Renumbered v93 → v109 → v110 → v112 → v113 across successive master
|
||||
// merges (upstream claimed through v112 — pages_links_extracted_at — in
|
||||
// the interim). Idempotent via DROP ... IF EXISTS, so it no-ops on
|
||||
// installs that never created the constraint.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check;
|
||||
ALTER TABLE links ADD CONSTRAINT links_link_source_check
|
||||
CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved'));
|
||||
`,
|
||||
sqlFor: {
|
||||
pglite: `
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check;
|
||||
ALTER TABLE links ADD CONSTRAINT links_link_source_check
|
||||
CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved'));
|
||||
`,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
+16
-4
@@ -16,7 +16,7 @@ import { expandQuery } from './search/expansion.ts';
|
||||
import { dedupResults } from './search/dedup.ts';
|
||||
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
|
||||
import type { HybridSearchMeta } from './types.ts';
|
||||
import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
|
||||
import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, isGlobalBasenameEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
|
||||
import { isFactsBackstopEligible } from './facts/eligibility.ts';
|
||||
import { stripTakesFence } from './takes-fence.ts';
|
||||
import { stripFactsFence } from './facts-fence.ts';
|
||||
@@ -991,9 +991,14 @@ async function runAutoLink(
|
||||
: {};
|
||||
|
||||
// Live-mode resolver: per-put throwaway cache, pg_trgm + optional search.
|
||||
const resolver = makeResolver(engine, { mode: 'live' });
|
||||
// Issue #972 (codex [P1]): pass sourceId so basename resolution stays
|
||||
// within this page's source — no cross-source basename edges.
|
||||
const resolver = makeResolver(engine, { mode: 'live', sourceId: opts?.sourceId });
|
||||
// Issue #972: opt-in bare-wikilink basename resolution. Off by default.
|
||||
const globalBasename = await isGlobalBasenameEnabled(engine);
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
slug, fullContent, parsed.frontmatter, parsed.type, resolver,
|
||||
{ globalBasename },
|
||||
);
|
||||
|
||||
// Resolve which targets exist (skip refs to non-existent pages to avoid FK
|
||||
@@ -1040,10 +1045,17 @@ async function runAutoLink(
|
||||
l => l.link_source === 'frontmatter' && l.origin_slug === slug,
|
||||
);
|
||||
|
||||
// Reconcilable outgoing edges: markdown + our own frontmatter edges.
|
||||
// Manual edges (link_source='manual') are NEVER touched by reconciliation.
|
||||
// Reconcilable outgoing edges: markdown + our own frontmatter edges +
|
||||
// basename-resolved wikilinks (issue #972). Manual edges
|
||||
// (link_source='manual') are NEVER touched by reconciliation.
|
||||
// 'wikilink-resolved' MUST be reconcilable (codex outside-voice [P1]):
|
||||
// auto-link writes these; if it weren't here, a basename edge would
|
||||
// survive after the wikilink is deleted from the page OR the
|
||||
// link_resolution.global_basename flag is turned off (out no longer
|
||||
// includes it, so the stale-removal loop below must be allowed to drop it).
|
||||
const reconcilableOut = existingOut.filter(
|
||||
l => l.link_source === 'markdown' || l.link_source == null ||
|
||||
l.link_source === 'wikilink-resolved' ||
|
||||
(l.link_source === 'frontmatter' && l.origin_slug === slug),
|
||||
);
|
||||
|
||||
|
||||
@@ -258,7 +258,9 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
-- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename
|
||||
-- wikilink resolution. See src/schema.sql for full rationale.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')),
|
||||
-- v0.41.18.0 (codex finding #12): nullable link_kind distinguishes
|
||||
-- "plain body mention" from "verb-pattern-derived typed link" within
|
||||
-- link_source='mentions'. See src/schema.sql for full rationale.
|
||||
|
||||
@@ -411,8 +411,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
-- ============================================================
|
||||
-- Provenance model (v0.13):
|
||||
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
|
||||
-- Provenance model (v0.13, extended issue #972):
|
||||
-- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL
|
||||
-- 'wikilink-resolved' is the opt-in
|
||||
-- (link_resolution.global_basename) basename-match
|
||||
-- provenance — see issue #972 / migration v113.
|
||||
-- (NULL = legacy row written before v0.13; unknown source)
|
||||
-- origin_page_id — for link_source='frontmatter', the page whose YAML
|
||||
-- frontmatter created this edge; scopes reconciliation
|
||||
@@ -431,7 +434,9 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
-- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename
|
||||
-- wikilink resolution (bare [[name]] resolved by slug tail).
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')),
|
||||
-- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from
|
||||
-- "verb-pattern-derived typed link" within link_source='mentions'.
|
||||
-- Codex finding #12 design: keep link_source stable; add link_kind
|
||||
|
||||
+8
-3
@@ -407,8 +407,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
-- ============================================================
|
||||
-- Provenance model (v0.13):
|
||||
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
|
||||
-- Provenance model (v0.13, extended issue #972):
|
||||
-- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL
|
||||
-- 'wikilink-resolved' is the opt-in
|
||||
-- (link_resolution.global_basename) basename-match
|
||||
-- provenance — see issue #972 / migration v113.
|
||||
-- (NULL = legacy row written before v0.13; unknown source)
|
||||
-- origin_page_id — for link_source='frontmatter', the page whose YAML
|
||||
-- frontmatter created this edge; scopes reconciliation
|
||||
@@ -427,7 +430,9 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
-- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename
|
||||
-- wikilink resolution (bare [[name]] resolved by slug tail).
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions', 'wikilink-resolved')),
|
||||
-- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from
|
||||
-- "verb-pattern-derived typed link" within link_source='mentions'.
|
||||
-- Codex finding #12 design: keep link_source stable; add link_kind
|
||||
|
||||
@@ -1159,6 +1159,164 @@ describe('v0.40.4 — graph_signals_coverage check', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─── issue #972 — link_resolution_opportunity check ───────────────────────
|
||||
|
||||
describe('issue #972 — link_resolution_opportunity check', () => {
|
||||
const { PGLiteEngine } = require('../src/core/pglite-engine.ts');
|
||||
const { checkLinkResolutionOpportunity } = require('../src/commands/doctor.ts');
|
||||
|
||||
let engine: any;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite' });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM links`);
|
||||
await engine.executeRaw(`DELETE FROM pages`);
|
||||
await engine.executeRaw(
|
||||
`DELETE FROM config WHERE key = 'link_resolution.global_basename'`,
|
||||
);
|
||||
});
|
||||
|
||||
test('skipped silently when flag is already enabled', async () => {
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
// Even with bare wikilinks that would resolve, the check returns ok
|
||||
// because the user is already opted in — no hint to surface.
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/x', {
|
||||
type: 'concept', title: 'X',
|
||||
compiled_truth: 'See [[struktura]] and [[struktura]] and [[struktura]].',
|
||||
timeline: '',
|
||||
});
|
||||
const check = await checkLinkResolutionOpportunity(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('already enabled');
|
||||
});
|
||||
|
||||
test('empty brain → ok with explanation', async () => {
|
||||
const check = await checkLinkResolutionOpportunity(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('empty');
|
||||
});
|
||||
|
||||
test('no bare wikilinks → ok', async () => {
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person', title: 'Alice',
|
||||
compiled_truth: 'Met [Bob](people/bob).', timeline: '',
|
||||
});
|
||||
await engine.putPage('people/bob', {
|
||||
type: 'person', title: 'Bob', compiled_truth: '', timeline: '',
|
||||
});
|
||||
const check = await checkLinkResolutionOpportunity(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('No bare wikilinks');
|
||||
});
|
||||
|
||||
test('bare wikilinks present but none match → ok', async () => {
|
||||
await engine.putPage('concepts/x', {
|
||||
type: 'concept', title: 'X',
|
||||
compiled_truth: 'See [[never-existed]] and [[also-not-here]].',
|
||||
timeline: '',
|
||||
});
|
||||
const check = await checkLinkResolutionOpportunity(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('none have basename matches');
|
||||
});
|
||||
|
||||
test('≥5 would-resolve AND ≥20% ratio → warn with paste-ready hint', async () => {
|
||||
// 5 distinct bare wikilinks, all resolving = 100% ratio.
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('projects/eeva', {
|
||||
type: 'project', title: 'Eeva', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('companies/fast-weigh', {
|
||||
type: 'company', title: 'Fast-Weigh', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('projects/rosa', {
|
||||
type: 'project', title: 'Rosa', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('projects/dragon', {
|
||||
type: 'project', title: 'Dragon', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/wiki-index', {
|
||||
type: 'concept', title: 'Wiki',
|
||||
compiled_truth: 'See [[struktura]], [[eeva]], [[Fast-Weigh]], [[rosa]], [[dragon]] for context.',
|
||||
timeline: '',
|
||||
});
|
||||
const check = await checkLinkResolutionOpportunity(engine);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toContain('5 of 5');
|
||||
expect(check.message).toContain('100%');
|
||||
expect(check.message).toContain(
|
||||
'gbrain config set link_resolution.global_basename true',
|
||||
);
|
||||
});
|
||||
|
||||
test('<5 would-resolve → ok without warning (below threshold)', async () => {
|
||||
// Only 2 bare wikilinks resolve → below the 5-link floor.
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('projects/eeva', {
|
||||
type: 'project', title: 'Eeva', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/x', {
|
||||
type: 'concept', title: 'X',
|
||||
compiled_truth: 'See [[struktura]] and [[eeva]] for context.',
|
||||
timeline: '',
|
||||
});
|
||||
const check = await checkLinkResolutionOpportunity(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('below the 20% / 5-link threshold');
|
||||
});
|
||||
|
||||
test('counts slugified-only matches (shared matcher, codex #972 DRY)', async () => {
|
||||
// `[[Fast Weigh]]` (space) only resolves to `companies/fast-weigh` via the
|
||||
// slugified key. Pre-consolidation the doctor index keyed raw+lower only
|
||||
// and would have reported "none have basename matches"; the shared matcher
|
||||
// adds the slugified key so the estimate matches what extraction resolves.
|
||||
await engine.putPage('companies/fast-weigh', {
|
||||
type: 'company', title: 'Fast-Weigh', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/x', {
|
||||
type: 'concept', title: 'X',
|
||||
compiled_truth: 'See [[Fast Weigh]] for context.', timeline: '',
|
||||
});
|
||||
const check = await checkLinkResolutionOpportunity(engine);
|
||||
// 1/1 resolves (below the 5-link warn floor → ok) but it DID resolve.
|
||||
expect(check.message).toContain('1/1');
|
||||
expect(check.message).toContain('would resolve');
|
||||
expect(check.message).not.toContain('none have basename matches');
|
||||
});
|
||||
|
||||
test('check is wired into runDoctor AND doctorReportRemote (source-grep)', async () => {
|
||||
const source = await Bun.file(
|
||||
new URL('../src/commands/doctor.ts', import.meta.url),
|
||||
).text();
|
||||
// Local buildChecks path.
|
||||
expect(source).toMatch(/await checkLinkResolutionOpportunity\(engine, progress\)/);
|
||||
// Thin-client doctorReportRemote path.
|
||||
expect(source).toMatch(/await checkLinkResolutionOpportunity\(engine\)/);
|
||||
// Heartbeat label registered.
|
||||
expect(source).toContain("progress.heartbeat('link_resolution_opportunity')");
|
||||
// Issue #972 (T3): scan is bounded to a most-recent sample, not a full
|
||||
// per-page getPage walk.
|
||||
expect(source).toMatch(/ORDER BY id DESC LIMIT \$\{?SAMPLE_LIMIT\}?|ORDER BY id DESC LIMIT 1000/);
|
||||
expect(source).not.toMatch(/await engine\.getPage\(ref\.slug/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.42 (#1699) — quarantined_pages + flagged_pages checks', () => {
|
||||
test('both checks are wired into buildChecks (source-grep)', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
|
||||
|
||||
@@ -0,0 +1,282 @@
|
||||
/**
|
||||
* Issue #972 E2E — opt-in global-basename wikilink resolution.
|
||||
*
|
||||
* Reproduces the issue's exact repro inside an in-memory PGLite brain:
|
||||
*
|
||||
* /vault/projects/struktura.md ← a real page
|
||||
* /vault/concepts/knowledge-graph.md ← contains `[[struktura]]`
|
||||
*
|
||||
* The bare wikilink `[[struktura]]` does NOT match WIKILINK_RE
|
||||
* (DIR_PATTERN-gated) so it falls through to the new pass 2c.
|
||||
*
|
||||
* Three contracts:
|
||||
* 1. Flag OFF (default) — extract emits ZERO basename edges (back-compat
|
||||
* with pre-issue-#972 behavior: bare wikilinks outside DIR_PATTERN
|
||||
* drop silently).
|
||||
* 2. Flag ON — extract emits ONE edge per basename match, tagged
|
||||
* `link_type: 'wikilink_basename'`.
|
||||
* 3. Ambiguous basename (same name in two directories) — extract emits
|
||||
* ONE edge per match. No silent winner-takes-all, no silent drop.
|
||||
*
|
||||
* Both DB-source AND FS-source paths covered. PGLite in-memory, no
|
||||
* DATABASE_URL needed.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { runExtract } from '../../src/commands/extract.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let brainDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite' } as never);
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function truncateAll() {
|
||||
for (const t of [
|
||||
'content_chunks', 'links', 'tags', 'raw_data',
|
||||
'timeline_entries', 'page_versions', 'ingest_log', 'pages',
|
||||
'config',
|
||||
]) {
|
||||
try { await (engine as any).db.exec(`DELETE FROM ${t}`); } catch { /* ok */ }
|
||||
}
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-issue-972-'));
|
||||
}, 15_000);
|
||||
|
||||
function writeFile(rel: string, content: string) {
|
||||
const full = join(brainDir, rel);
|
||||
mkdirSync(join(full, '..'), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
|
||||
// ─── FS-source path (the issue's repro) ─────────────────────────────────
|
||||
|
||||
describe('issue #972 — FS-source (gbrain extract links default)', () => {
|
||||
test("repro: [[struktura]] in concepts/ resolves to projects/struktura when flag ON", async () => {
|
||||
// Seed both pages in the DB (extract validates targetSlug exists)
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura',
|
||||
compiled_truth: 'A project page.', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/knowledge-graph', {
|
||||
type: 'concept', title: 'Knowledge Graph',
|
||||
compiled_truth: 'This concept relates to [[struktura]].', timeline: '',
|
||||
});
|
||||
// Mirror to disk so the FS extractor sees the files
|
||||
writeFile('projects/struktura.md',
|
||||
'---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
|
||||
writeFile('concepts/knowledge-graph.md',
|
||||
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
|
||||
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
|
||||
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
|
||||
expect(strk).toBeDefined();
|
||||
expect(strk!.link_type).toBe('wikilink_basename');
|
||||
// Issue #972 (T1): FS-source basename edges carry the same provenance
|
||||
// tag as DB / put_page, not the default 'markdown'.
|
||||
expect(strk!.link_source).toBe('wikilink-resolved');
|
||||
});
|
||||
|
||||
test('back-compat: flag OFF → ZERO basename edges from same repro', async () => {
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura',
|
||||
compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/knowledge-graph', {
|
||||
type: 'concept', title: 'Knowledge Graph',
|
||||
compiled_truth: 'This relates to [[struktura]].', timeline: '',
|
||||
});
|
||||
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
|
||||
writeFile('concepts/knowledge-graph.md',
|
||||
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis relates to [[struktura]].\n');
|
||||
|
||||
// Flag explicitly off (also the default)
|
||||
await engine.setConfig('link_resolution.global_basename', 'false');
|
||||
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
|
||||
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
||||
expect(outLinks.filter(l => l.link_type === 'wikilink_basename')).toEqual([]);
|
||||
});
|
||||
|
||||
test('ambiguous basename → emits one edge per match (no silent winner)', async () => {
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura',
|
||||
compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('archive/struktura', {
|
||||
type: 'concept' as any, title: 'Struktura (archived)',
|
||||
compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/x', {
|
||||
type: 'concept', title: 'X',
|
||||
compiled_truth: 'See [[struktura]].', timeline: '',
|
||||
});
|
||||
|
||||
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
|
||||
writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n');
|
||||
writeFile('concepts/x.md',
|
||||
'---\ntitle: X\ntype: concept\n---\n\nSee [[struktura]].\n');
|
||||
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
|
||||
const outLinks = await engine.getLinks('concepts/x');
|
||||
const basenameLinks = outLinks
|
||||
.filter(l => l.link_type === 'wikilink_basename')
|
||||
.map(l => l.to_slug)
|
||||
.sort();
|
||||
expect(basenameLinks).toEqual(['archive/struktura', 'projects/struktura']);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── DB-source path (gbrain extract links --source db) ──────────────────
|
||||
|
||||
describe('issue #972 — DB-source (gbrain extract links --source db)', () => {
|
||||
test('flag ON → bare wikilink in compiled_truth resolves to basename match', async () => {
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura',
|
||||
compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/knowledge-graph', {
|
||||
type: 'concept', title: 'Knowledge Graph',
|
||||
compiled_truth: 'This relates to [[struktura]].', timeline: '',
|
||||
});
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
|
||||
await runExtract(engine, ['links', '--source', 'db']);
|
||||
|
||||
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
|
||||
expect(strk).toBeDefined();
|
||||
expect(strk!.link_type).toBe('wikilink_basename');
|
||||
});
|
||||
|
||||
test('flag OFF → no basename edges via DB path (back-compat)', async () => {
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura',
|
||||
compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.putPage('concepts/knowledge-graph', {
|
||||
type: 'concept', title: 'Knowledge Graph',
|
||||
compiled_truth: 'This relates to [[struktura]].', timeline: '',
|
||||
});
|
||||
|
||||
await runExtract(engine, ['links', '--source', 'db']);
|
||||
|
||||
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// ─── put_page auto-link path ────────────────────────────────────────────
|
||||
//
|
||||
// put_page accepts `content` as a full markdown document with frontmatter
|
||||
// (not the engine's narrow Page shape). Auto-link runs INSIDE put_page,
|
||||
// so the basename-resolution path picks up the flag from
|
||||
// engine.getConfig() once per call.
|
||||
|
||||
const PUT_PAGE_MARKDOWN_WITH_WIKILINK = `---
|
||||
title: Knowledge Graph
|
||||
type: concept
|
||||
---
|
||||
|
||||
This relates to [[struktura]].
|
||||
`;
|
||||
|
||||
describe('issue #972 — put_page auto-link', () => {
|
||||
test('newly-written page with bare wikilink → basename edge when flag ON', async () => {
|
||||
// Need the target page to exist first (auto-link validates against the
|
||||
// existing slug set).
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura',
|
||||
compiled_truth: '', timeline: '',
|
||||
});
|
||||
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
|
||||
const { operations } = await import('../../src/core/operations.ts');
|
||||
const putPage = operations.find(op => op.name === 'put_page')!;
|
||||
await putPage.handler(
|
||||
{ engine, remote: false } as never,
|
||||
{
|
||||
slug: 'concepts/knowledge-graph',
|
||||
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
|
||||
},
|
||||
);
|
||||
|
||||
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
const strk = outLinks.find(l => l.to_slug === 'projects/struktura');
|
||||
expect(strk).toBeDefined();
|
||||
expect(strk!.link_type).toBe('wikilink_basename');
|
||||
});
|
||||
|
||||
test('flag OFF → put_page does NOT emit basename edge (back-compat)', async () => {
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura',
|
||||
compiled_truth: '', timeline: '',
|
||||
});
|
||||
|
||||
// Default-off (no setConfig)
|
||||
|
||||
const { operations } = await import('../../src/core/operations.ts');
|
||||
const putPage = operations.find(op => op.name === 'put_page')!;
|
||||
await putPage.handler(
|
||||
{ engine, remote: false } as never,
|
||||
{
|
||||
slug: 'concepts/knowledge-graph',
|
||||
content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
|
||||
},
|
||||
);
|
||||
|
||||
const outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('stale basename edge is removed when the wikilink is deleted (codex #972)', async () => {
|
||||
// Regression: wikilink-resolved edges must be reconcilable, else they
|
||||
// survive after the bare wikilink is removed from the page body.
|
||||
await engine.putPage('projects/struktura', {
|
||||
type: 'project', title: 'Struktura', compiled_truth: '', timeline: '',
|
||||
});
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
|
||||
const { operations } = await import('../../src/core/operations.ts');
|
||||
const putPage = operations.find(op => op.name === 'put_page')!;
|
||||
|
||||
// 1. Write the page WITH the wikilink → edge lands.
|
||||
await putPage.handler({ engine, remote: false } as never, {
|
||||
slug: 'concepts/knowledge-graph', content: PUT_PAGE_MARKDOWN_WITH_WIKILINK,
|
||||
});
|
||||
let outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeDefined();
|
||||
|
||||
// 2. Re-write the page WITHOUT the wikilink → edge must be reconciled away.
|
||||
await putPage.handler({ engine, remote: false } as never, {
|
||||
slug: 'concepts/knowledge-graph',
|
||||
content: '---\ntitle: Knowledge Graph\ntype: concept\n---\n\nNo links here anymore.\n',
|
||||
});
|
||||
outLinks = await engine.getLinks('concepts/knowledge-graph');
|
||||
expect(outLinks.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -246,3 +246,213 @@ describe('gbrain extract --dir default resolution', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─── issue #972: pure-function tests for resolveSlugAll + helpers ────────
|
||||
|
||||
import {
|
||||
resolveSlug,
|
||||
resolveSlugAll,
|
||||
resolveBasenameMatchesFromSlugs,
|
||||
extractLinksFromFile,
|
||||
} from '../src/commands/extract.ts';
|
||||
|
||||
describe('extractLinksFromFile — code-fence stripping (codex P2b)', () => {
|
||||
test('a bare wikilink inside a code fence does NOT create an FS edge', async () => {
|
||||
const allSlugs = new Set(['projects/struktura', 'concepts/x']);
|
||||
const content = [
|
||||
'---', 'title: X', 'type: concept', '---', '',
|
||||
'Real ref: [[struktura]].',
|
||||
'',
|
||||
'```', 'code mentions [[struktura]] but must be ignored', '```',
|
||||
].join('\n');
|
||||
const links = await extractLinksFromFile(content, 'concepts/x.md', allSlugs, { globalBasename: true });
|
||||
const toStruktura = links.filter(l => l.to_slug === 'projects/struktura');
|
||||
// Exactly one edge — from the prose ref, NOT the fenced one.
|
||||
expect(toStruktura.length).toBe(1);
|
||||
expect(toStruktura[0].link_source).toBe('wikilink-resolved');
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveBasenameMatchesFromSlugs (pure)', () => {
|
||||
test('returns ALL slugs whose tail matches', () => {
|
||||
const all = new Set([
|
||||
'projects/struktura',
|
||||
'archive/struktura',
|
||||
'notes/other',
|
||||
]);
|
||||
const matches = resolveBasenameMatchesFromSlugs('struktura', all);
|
||||
expect(matches.sort()).toEqual(['archive/struktura', 'projects/struktura']);
|
||||
});
|
||||
|
||||
test('case-insensitive tail match', () => {
|
||||
const all = new Set(['companies/fast-weigh']);
|
||||
expect(resolveBasenameMatchesFromSlugs('Fast-Weigh', all))
|
||||
.toEqual(['companies/fast-weigh']);
|
||||
expect(resolveBasenameMatchesFromSlugs('FAST-WEIGH', all))
|
||||
.toEqual(['companies/fast-weigh']);
|
||||
});
|
||||
|
||||
test('slugified fallback (spaces → hyphens)', () => {
|
||||
const all = new Set(['companies/fast-weigh']);
|
||||
expect(resolveBasenameMatchesFromSlugs('Fast Weigh', all))
|
||||
.toEqual(['companies/fast-weigh']);
|
||||
});
|
||||
|
||||
test('top-level slugs (no `/`) match by themselves', () => {
|
||||
const all = new Set(['struktura', 'notes/struktura']);
|
||||
const matches = resolveBasenameMatchesFromSlugs('struktura', all);
|
||||
expect(matches.sort()).toEqual(['notes/struktura', 'struktura']);
|
||||
});
|
||||
|
||||
test('no match returns []', () => {
|
||||
const all = new Set(['projects/struktura']);
|
||||
expect(resolveBasenameMatchesFromSlugs('never-existed', all)).toEqual([]);
|
||||
});
|
||||
|
||||
test('empty/whitespace input returns []', () => {
|
||||
const all = new Set(['projects/struktura']);
|
||||
expect(resolveBasenameMatchesFromSlugs('', all)).toEqual([]);
|
||||
expect(resolveBasenameMatchesFromSlugs(' ', all)).toEqual([]);
|
||||
});
|
||||
|
||||
test('stable sort: shorter slug first, then lexical', () => {
|
||||
const all = new Set([
|
||||
'zzz/struktura',
|
||||
'projects/struktura',
|
||||
'archive/struktura',
|
||||
'a/struktura',
|
||||
]);
|
||||
const matches = resolveBasenameMatchesFromSlugs('struktura', all);
|
||||
// Lengths: a/struktura(11), zzz/struktura(13), archive/struktura(17), projects/struktura(18)
|
||||
expect(matches).toEqual([
|
||||
'a/struktura',
|
||||
'zzz/struktura',
|
||||
'archive/struktura',
|
||||
'projects/struktura',
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveSlugAll', () => {
|
||||
test('ancestor walk wins → single-element array, no basename fallback', () => {
|
||||
// resolveSlug already finds notes/struktura via ancestor walk.
|
||||
// resolveSlugAll must return only that one even though basename
|
||||
// would find others.
|
||||
const all = new Set([
|
||||
'notes/struktura',
|
||||
'archive/struktura',
|
||||
'projects/struktura',
|
||||
]);
|
||||
const out = resolveSlugAll('notes/sub', 'struktura.md', all,
|
||||
{ globalBasename: true });
|
||||
expect(out).toEqual(['notes/struktura']);
|
||||
});
|
||||
|
||||
test('ancestor walk misses + globalBasename off → []', () => {
|
||||
const all = new Set(['projects/struktura']);
|
||||
const out = resolveSlugAll('concepts', 'struktura.md', all);
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('ancestor walk misses + globalBasename on → all basename matches', () => {
|
||||
const all = new Set([
|
||||
'projects/struktura',
|
||||
'archive/struktura',
|
||||
]);
|
||||
const out = resolveSlugAll('concepts', 'struktura.md', all,
|
||||
{ globalBasename: true });
|
||||
expect(out.sort()).toEqual(['archive/struktura', 'projects/struktura']);
|
||||
});
|
||||
|
||||
test('zero basename matches when globalBasename on returns []', () => {
|
||||
const all = new Set(['projects/struktura']);
|
||||
const out = resolveSlugAll('concepts', 'phantom.md', all,
|
||||
{ globalBasename: true });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('strips dirname from relTarget when applying basename lookup', () => {
|
||||
// [[notes/struktura]] with no `notes` ancestor: ancestor walk strips
|
||||
// `concepts` → tries `notes/struktura` which DOES exist → emits.
|
||||
// This case verifies the basename fallback only fires when ancestor
|
||||
// walk truly fails. (Sanity check on the fallback ordering.)
|
||||
const all = new Set(['notes/struktura']);
|
||||
expect(resolveSlugAll('concepts', 'notes/struktura.md', all))
|
||||
.toEqual(['notes/struktura']);
|
||||
});
|
||||
|
||||
test('resolveSlug back-compat: existing single-match callers unaffected', () => {
|
||||
const all = new Set(['notes/struktura']);
|
||||
// The legacy resolveSlug must keep returning the string|null shape.
|
||||
expect(resolveSlug('notes', 'struktura.md', all)).toBe('notes/struktura');
|
||||
expect(resolveSlug('concepts', 'phantom.md', all)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('issue #972 repro: bare wikilinks resolve when flag is on', () => {
|
||||
// End-to-end: reproduces the issue's exact repro inside a tempdir +
|
||||
// PGLite, then asserts edge count under both flag states.
|
||||
test('flag OFF → 0 edges (back-compat)', async () => {
|
||||
await engine.putPage('projects/struktura',
|
||||
{ type: 'project', title: 'Struktura', compiled_truth: 'A project page.', timeline: '' });
|
||||
await engine.putPage('concepts/knowledge-graph',
|
||||
{ type: 'concept', title: 'Knowledge Graph',
|
||||
compiled_truth: 'This concept relates to [[struktura]].', timeline: '' });
|
||||
|
||||
// Mirror to disk: the FS extractor walks files, not DB pages.
|
||||
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
|
||||
writeFile('concepts/knowledge-graph.md',
|
||||
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
|
||||
|
||||
// Ensure flag is off (default)
|
||||
await engine.setConfig('link_resolution.global_basename', 'false');
|
||||
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const links = await engine.getLinks('concepts/knowledge-graph');
|
||||
expect(links.find(l => l.to_slug === 'projects/struktura')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('flag ON → 1 edge with wikilink_basename type', async () => {
|
||||
await engine.putPage('projects/struktura',
|
||||
{ type: 'project', title: 'Struktura', compiled_truth: 'A project page.', timeline: '' });
|
||||
await engine.putPage('concepts/knowledge-graph',
|
||||
{ type: 'concept', title: 'Knowledge Graph',
|
||||
compiled_truth: 'This concept relates to [[struktura]].', timeline: '' });
|
||||
|
||||
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n\nA project page.\n');
|
||||
writeFile('concepts/knowledge-graph.md',
|
||||
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nThis concept relates to [[struktura]].\n');
|
||||
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const links = await engine.getLinks('concepts/knowledge-graph');
|
||||
const strk = links.find(l => l.to_slug === 'projects/struktura');
|
||||
expect(strk).toBeDefined();
|
||||
expect(strk!.link_type).toBe('wikilink_basename');
|
||||
});
|
||||
|
||||
test('flag ON + ambiguous basename → one edge per match', async () => {
|
||||
await engine.putPage('projects/struktura',
|
||||
{ type: 'project', title: 'Struktura', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('archive/struktura',
|
||||
{ type: 'concept' as any, title: 'Struktura (archived)',
|
||||
compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('concepts/knowledge-graph',
|
||||
{ type: 'concept', title: 'Knowledge Graph',
|
||||
compiled_truth: 'See [[struktura]].', timeline: '' });
|
||||
|
||||
writeFile('projects/struktura.md', '---\ntitle: Struktura\ntype: project\n---\n');
|
||||
writeFile('archive/struktura.md', '---\ntitle: Struktura\ntype: concept\n---\n');
|
||||
writeFile('concepts/knowledge-graph.md',
|
||||
'---\ntitle: Knowledge Graph\ntype: concept\n---\n\nSee [[struktura]].\n');
|
||||
|
||||
await engine.setConfig('link_resolution.global_basename', 'true');
|
||||
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const links = await engine.getLinks('concepts/knowledge-graph');
|
||||
const basenameLinks = links.filter(l => l.link_type === 'wikilink_basename');
|
||||
const targets = basenameLinks.map(l => l.to_slug).sort();
|
||||
expect(targets).toEqual(['archive/struktura', 'projects/struktura']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -109,6 +109,92 @@ describe('extractEntityRefs', () => {
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].dir).toBe('meetings');
|
||||
});
|
||||
|
||||
// ─── issue #972: generic `[[bare-name]]` wikilinks (pass 2c) ─────────────
|
||||
|
||||
test('tags bare wikilinks with needsResolution flag', () => {
|
||||
const refs = extractEntityRefs(
|
||||
'See [[Fast-Weigh]] and [[2026-05-07-cost-plan-rosa-pilot]] for context.',
|
||||
);
|
||||
expect(refs.length).toBe(2);
|
||||
expect(refs.every(r => r.needsResolution === true)).toBe(true);
|
||||
expect(refs.map(r => r.slug).sort()).toEqual([
|
||||
'2026-05-07-cost-plan-rosa-pilot',
|
||||
'Fast-Weigh',
|
||||
]);
|
||||
// dir is empty string when the bare wikilink has no `/`
|
||||
for (const r of refs) {
|
||||
expect(r.dir).toBe('');
|
||||
}
|
||||
});
|
||||
|
||||
test('does NOT double-emit when DIR_PATTERN wikilink also passes 2b', () => {
|
||||
// [[people/alice]] matches 2b (DIR_PATTERN-gated). 2c must NOT emit
|
||||
// a duplicate ref. [[Fast-Weigh]] only matches 2c (no DIR_PATTERN).
|
||||
const refs = extractEntityRefs('See [[people/alice]] and [[Fast-Weigh]].');
|
||||
const aliceRefs = refs.filter(r => r.slug === 'people/alice');
|
||||
const wikiRefs = refs.filter(r => r.slug === 'Fast-Weigh');
|
||||
expect(aliceRefs.length).toBe(1);
|
||||
expect(aliceRefs[0].needsResolution).toBeUndefined();
|
||||
expect(wikiRefs.length).toBe(1);
|
||||
expect(wikiRefs[0].needsResolution).toBe(true);
|
||||
});
|
||||
|
||||
test('skips qualified-syntax tokens (those belong to 2a)', () => {
|
||||
// [[wiki:topics/ai]] looks like 2a's qualified shape — even though
|
||||
// it wouldn't satisfy DIR_PATTERN, 2c must not claim it either
|
||||
// (the leading `:` is the qualified-syntax tell).
|
||||
const refs = extractEntityRefs('See [[wiki:topics/ai]] and [[bare-name]].');
|
||||
const bare = refs.find(r => r.slug === 'bare-name');
|
||||
expect(bare).toBeDefined();
|
||||
expect(bare!.needsResolution).toBe(true);
|
||||
const wrongQualified = refs.filter(
|
||||
r => r.slug.includes(':') && r.needsResolution === true,
|
||||
);
|
||||
expect(wrongQualified.length).toBe(0);
|
||||
});
|
||||
|
||||
test('a wikilink inside a markdown-link label is inert (codex P2a)', () => {
|
||||
// `[see [[acme]]](companies/acme.md)` must NOT spawn a stray generic
|
||||
// basename ref for the inner `[[acme]]`. Pass-1 can't match the nested
|
||||
// brackets, so the label-wikilink span is masked out of pass 2c.
|
||||
const refs = extractEntityRefs('[see [[acme]]](companies/acme.md)');
|
||||
expect(refs.filter(r => r.needsResolution)).toEqual([]);
|
||||
// But an independent bare wikilink on the same line still emits.
|
||||
const refs2 = extractEntityRefs('[Acme](companies/acme) and bare [[acme]] here.');
|
||||
expect(refs2.find(r => r.slug === 'companies/acme' && !r.needsResolution)).toBeDefined();
|
||||
expect(refs2.find(r => r.slug === 'acme' && r.needsResolution)).toBeDefined();
|
||||
});
|
||||
|
||||
test('strips .md suffix from bare wikilinks', () => {
|
||||
const refs = extractEntityRefs('See [[struktura.md]] for context.');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('struktura');
|
||||
expect(refs[0].needsResolution).toBe(true);
|
||||
});
|
||||
|
||||
test('extracts display name from [[slug|Display]] shape', () => {
|
||||
const refs = extractEntityRefs('See [[struktura|The Project]] for details.');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('struktura');
|
||||
expect(refs[0].name).toBe('The Project');
|
||||
expect(refs[0].needsResolution).toBe(true);
|
||||
});
|
||||
|
||||
test('strips #anchor from bare wikilinks', () => {
|
||||
const refs = extractEntityRefs('Jump to [[notes#section-2]].');
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('notes');
|
||||
expect(refs[0].needsResolution).toBe(true);
|
||||
});
|
||||
|
||||
test('skips bare wikilinks inside fenced code blocks', () => {
|
||||
const refs = extractEntityRefs(
|
||||
'```\nThis is a code block with [[fake-link]] inside.\n```\nReal: [[real-link]].',
|
||||
);
|
||||
expect(refs.length).toBe(1);
|
||||
expect(refs[0].slug).toBe('real-link');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── extractPageLinks ──────────────────────────────────────────
|
||||
@@ -181,6 +267,187 @@ describe('extractPageLinks', () => {
|
||||
const aliceLink = candidates.find(c => c.targetSlug === 'people/alice');
|
||||
expect(aliceLink!.linkType).toBe('attended');
|
||||
});
|
||||
|
||||
// ─── issue #972: bare wikilink → resolver.resolveBasenameMatches ─────────
|
||||
|
||||
test('bare wikilink drops silently when globalBasename flag is OFF', async () => {
|
||||
// Resolver that WOULD resolve, but we never reach it because the
|
||||
// flag is off — this is the back-compat invariant.
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
resolveBasenameMatches: async () => ['projects/struktura'],
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/knowledge-graph',
|
||||
'This relates to [[struktura]].',
|
||||
{}, 'concept', resolver,
|
||||
// opts.globalBasename omitted (= false)
|
||||
);
|
||||
expect(candidates.find(c => c.targetSlug === 'projects/struktura')).toBeUndefined();
|
||||
expect(candidates).toEqual([]);
|
||||
});
|
||||
|
||||
test('bare wikilink emits one candidate per basename match when flag ON', async () => {
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
resolveBasenameMatches: async (name) => {
|
||||
if (name === 'struktura') return ['projects/struktura', 'archive/struktura'];
|
||||
return [];
|
||||
},
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/knowledge-graph',
|
||||
'This relates to [[struktura]].',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
const targets = candidates.map(c => c.targetSlug).sort();
|
||||
expect(targets).toEqual(['archive/struktura', 'projects/struktura']);
|
||||
// Both edges stamped with the new edge type + provenance.
|
||||
for (const c of candidates) {
|
||||
expect(c.linkType).toBe('wikilink_basename');
|
||||
expect(c.linkSource).toBe('wikilink-resolved');
|
||||
}
|
||||
});
|
||||
|
||||
test('bare wikilink with single basename match emits one candidate', async () => {
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
resolveBasenameMatches: async (name) =>
|
||||
name === 'struktura' ? ['projects/struktura'] : [],
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/knowledge-graph',
|
||||
'See [[struktura]] for details.',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
expect(candidates.length).toBe(1);
|
||||
expect(candidates[0].targetSlug).toBe('projects/struktura');
|
||||
expect(candidates[0].linkType).toBe('wikilink_basename');
|
||||
});
|
||||
|
||||
test('basename self-link is dropped (codex P2c)', async () => {
|
||||
// `[[struktura]]` on the page concepts/struktura resolves back to itself —
|
||||
// the self-loop must be dropped.
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
resolveBasenameMatches: async (name) =>
|
||||
name === 'struktura' ? ['concepts/struktura', 'projects/struktura'] : [],
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/struktura', // the page being processed
|
||||
'See [[struktura]].',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
// Only the OTHER match survives; no self-edge to concepts/struktura.
|
||||
expect(candidates.map(c => c.targetSlug)).toEqual(['projects/struktura']);
|
||||
});
|
||||
|
||||
test('aliased wikilink resolves the TARGET, not the display text (codex #972)', async () => {
|
||||
// `[[struktura|the project]]` must resolve basename `struktura`, never
|
||||
// the alias "the project". Regression for the codex-caught bug where
|
||||
// extractPageLinks resolved ref.name (display) instead of ref.slug.
|
||||
const seen: string[] = [];
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
resolveBasenameMatches: async (name) => {
|
||||
seen.push(name);
|
||||
return name === 'struktura' ? ['projects/struktura'] : [];
|
||||
},
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/knowledge-graph',
|
||||
'This relates to [[struktura|the project]].',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
expect(seen).toContain('struktura');
|
||||
expect(seen).not.toContain('the project');
|
||||
expect(candidates.map(c => c.targetSlug)).toEqual(['projects/struktura']);
|
||||
});
|
||||
|
||||
test('bare wikilink with zero basename matches drops silently (no dangling row)', async () => {
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
resolveBasenameMatches: async () => [],
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'Mention [[never-existed]].',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
expect(candidates.find(c => c.targetSlug === 'never-existed')).toBeUndefined();
|
||||
expect(candidates).toEqual([]);
|
||||
});
|
||||
|
||||
test('bare wikilink resolution does not interfere with DIR_PATTERN wikilinks', async () => {
|
||||
// 2b refs (people/alice) take the verb-inferred type;
|
||||
// 2c refs (struktura) take wikilink_basename. Same call.
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async () => null,
|
||||
resolveBasenameMatches: async (name) =>
|
||||
name === 'struktura' ? ['projects/struktura'] : [],
|
||||
};
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x',
|
||||
'[[people/alice]] is the lead. The work is [[struktura]].',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
const alice = candidates.find(c => c.targetSlug === 'people/alice');
|
||||
const strk = candidates.find(c => c.targetSlug === 'projects/struktura');
|
||||
expect(alice).toBeDefined();
|
||||
expect(alice!.linkType).not.toBe('wikilink_basename'); // verb-inferred
|
||||
expect(strk).toBeDefined();
|
||||
expect(strk!.linkType).toBe('wikilink_basename');
|
||||
});
|
||||
|
||||
test('opts.skipFrontmatter suppresses the frontmatter pass', async () => {
|
||||
// Real resolver shape that WOULD resolve frontmatter source: too,
|
||||
// but skipFrontmatter blocks the path entirely.
|
||||
const resolver: SlugResolver = {
|
||||
resolve: async (name) =>
|
||||
name === 'meetings/2026-01-15' ? 'meetings/2026-01-15' : null,
|
||||
};
|
||||
const fm = { source: 'meetings/2026-01-15' };
|
||||
const withFm = await extractPageLinks(
|
||||
'docs/x', 'plain content', fm, 'person', resolver,
|
||||
{ skipFrontmatter: false },
|
||||
);
|
||||
const withoutFm = await extractPageLinks(
|
||||
'docs/x', 'plain content', fm, 'person', resolver,
|
||||
{ skipFrontmatter: true },
|
||||
);
|
||||
expect(withFm.candidates.find(c => c.linkType === 'source')).toBeDefined();
|
||||
expect(withoutFm.candidates.find(c => c.linkType === 'source')).toBeUndefined();
|
||||
// Issue #972 (codex P2e): skipFrontmatter must return an empty unresolved
|
||||
// list (the pass is skipped entirely), never undefined.
|
||||
expect(withoutFm.unresolved).toEqual([]);
|
||||
});
|
||||
|
||||
test('skipFrontmatter suppresses unresolved frontmatter refs too (codex P2e)', async () => {
|
||||
// A frontmatter field the resolver CANNOT resolve normally populates
|
||||
// `unresolved`; with skipFrontmatter the whole pass is gone so it's [].
|
||||
const resolver: SlugResolver = { resolve: async () => null };
|
||||
const fm = { key_people: ['Nobody Known'] };
|
||||
const withFm = await extractPageLinks(
|
||||
'companies/acme', 'plain content', fm, 'company', resolver,
|
||||
{ skipFrontmatter: false },
|
||||
);
|
||||
const withoutFm = await extractPageLinks(
|
||||
'companies/acme', 'plain content', fm, 'company', resolver,
|
||||
{ skipFrontmatter: true },
|
||||
);
|
||||
expect(withFm.unresolved.length).toBeGreaterThan(0); // pass ran, ref unresolved
|
||||
expect(withoutFm.unresolved).toEqual([]); // pass skipped
|
||||
});
|
||||
|
||||
test('globalBasename does nothing when resolver lacks resolveBasenameMatches', async () => {
|
||||
// The frontmatter-only synthetic resolver doesn't implement basename
|
||||
// lookup. Make sure we don't blow up — just drop the bare ref.
|
||||
const resolver: SlugResolver = { resolve: async () => null };
|
||||
const { candidates } = await extractPageLinks(
|
||||
'concepts/x', 'See [[struktura]].',
|
||||
{}, 'concept', resolver, { globalBasename: true },
|
||||
);
|
||||
expect(candidates).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── inferLinkType ─────────────────────────────────────────────
|
||||
@@ -729,6 +996,149 @@ describe('makeResolver — fallback chain', () => {
|
||||
const r = makeResolver(engine, { mode: 'batch' });
|
||||
expect(await r.resolve('Nonexistent Person', 'people')).toBeNull();
|
||||
});
|
||||
|
||||
// ─── issue #972: resolveBasenameMatches ───────────────────────────────
|
||||
|
||||
// Extended fake engine that also implements `getAllSlugs` so
|
||||
// resolveBasenameMatches has something to walk.
|
||||
function makeFakeEngineWithSlugs(slugs: string[]): BrainEngine {
|
||||
const lookup = new Set(slugs);
|
||||
let getAllCalls = 0;
|
||||
const engine = {
|
||||
async getPage(slug: string) {
|
||||
return lookup.has(slug) ? { slug } as any : null;
|
||||
},
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
async getAllSlugs() {
|
||||
getAllCalls++;
|
||||
return new Set(slugs);
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
(engine as any)._counts = () => ({ getAllCalls });
|
||||
return engine;
|
||||
}
|
||||
|
||||
test('resolveBasenameMatches: exact tail hit returns the slug', async () => {
|
||||
const engine = makeFakeEngineWithSlugs([
|
||||
'projects/struktura',
|
||||
'people/alice',
|
||||
]);
|
||||
const r = makeResolver(engine);
|
||||
expect(await r.resolveBasenameMatches!('struktura')).toEqual(['projects/struktura']);
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: multi-match returns ALL hits', async () => {
|
||||
const engine = makeFakeEngineWithSlugs([
|
||||
'projects/struktura',
|
||||
'archive/struktura',
|
||||
'notes/struktura',
|
||||
]);
|
||||
const r = makeResolver(engine);
|
||||
const out = await r.resolveBasenameMatches!('struktura');
|
||||
expect(out.sort()).toEqual([
|
||||
'archive/struktura',
|
||||
'notes/struktura',
|
||||
'projects/struktura',
|
||||
]);
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: case-insensitive fallback', async () => {
|
||||
const engine = makeFakeEngineWithSlugs(['companies/fast-weigh']);
|
||||
const r = makeResolver(engine);
|
||||
// Raw `Fast-Weigh` does not match the lowercase tail, but the
|
||||
// lowercased+slugified key does — both should hit.
|
||||
expect(await r.resolveBasenameMatches!('fast-weigh')).toEqual(['companies/fast-weigh']);
|
||||
expect(await r.resolveBasenameMatches!('Fast-Weigh')).toContain('companies/fast-weigh');
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: no matches returns []', async () => {
|
||||
const engine = makeFakeEngineWithSlugs(['projects/struktura']);
|
||||
const r = makeResolver(engine);
|
||||
expect(await r.resolveBasenameMatches!('never-existed')).toEqual([]);
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: scopes the index by sourceId (codex #972)', async () => {
|
||||
// Regression: a bare [[struktura]] in source A must NOT resolve to a
|
||||
// same-tail page in source B. makeResolver({sourceId}) must pass the
|
||||
// scope to getAllSlugs so the index only contains the source's slugs.
|
||||
let sawOpts: any;
|
||||
const bySource: Record<string, string[]> = {
|
||||
'src-a': ['projects/struktura'],
|
||||
'src-b': ['archive/struktura'],
|
||||
};
|
||||
const engine = {
|
||||
async getPage() { return null; },
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
async getAllSlugs(opts?: { sourceId?: string }) {
|
||||
sawOpts = opts;
|
||||
const sid = opts?.sourceId;
|
||||
return new Set(sid ? (bySource[sid] ?? []) : Object.values(bySource).flat());
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
|
||||
const out = await r.resolveBasenameMatches!('struktura');
|
||||
expect(sawOpts).toEqual({ sourceId: 'src-a' });
|
||||
expect(out).toEqual(['projects/struktura']); // src-a only
|
||||
expect(out).not.toContain('archive/struktura'); // no cross-source
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: no sourceId stays brain-wide (back-compat)', async () => {
|
||||
let sawOpts: any = 'unset';
|
||||
const engine = {
|
||||
async getPage() { return null; },
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
async getAllSlugs(opts?: { sourceId?: string }) {
|
||||
sawOpts = opts;
|
||||
return new Set(['projects/struktura', 'archive/struktura']);
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
const r = makeResolver(engine, { mode: 'batch' });
|
||||
const out = await r.resolveBasenameMatches!('struktura');
|
||||
expect(sawOpts).toBeUndefined(); // unscoped call
|
||||
expect(out.sort()).toEqual(['archive/struktura', 'projects/struktura']);
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: empty input returns []', async () => {
|
||||
const engine = makeFakeEngineWithSlugs(['projects/struktura']);
|
||||
const r = makeResolver(engine);
|
||||
expect(await r.resolveBasenameMatches!('')).toEqual([]);
|
||||
expect(await r.resolveBasenameMatches!(' ')).toEqual([]);
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: index built once, reused across calls', async () => {
|
||||
const engine = makeFakeEngineWithSlugs([
|
||||
'projects/struktura',
|
||||
'archive/struktura',
|
||||
]);
|
||||
const r = makeResolver(engine);
|
||||
await r.resolveBasenameMatches!('struktura');
|
||||
await r.resolveBasenameMatches!('struktura');
|
||||
await r.resolveBasenameMatches!('struktura');
|
||||
// Single getAllSlugs() call across three resolveBasenameMatches calls.
|
||||
expect((engine as any)._counts().getAllCalls).toBe(1);
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: degrades gracefully when getAllSlugs missing', async () => {
|
||||
// Test seam for engines that don't implement getAllSlugs (legacy / mocks).
|
||||
const engine = {
|
||||
async getPage() { return null; },
|
||||
async findByTitleFuzzy() { return null; },
|
||||
async searchKeyword() { return []; },
|
||||
} as unknown as BrainEngine;
|
||||
const r = makeResolver(engine);
|
||||
expect(await r.resolveBasenameMatches!('struktura')).toEqual([]);
|
||||
});
|
||||
|
||||
test('resolveBasenameMatches: handles top-level slugs (no `/`)', async () => {
|
||||
const engine = makeFakeEngineWithSlugs(['struktura', 'notes/struktura']);
|
||||
const r = makeResolver(engine);
|
||||
// Both should match because basename of `struktura` is `struktura`.
|
||||
const out = await r.resolveBasenameMatches!('struktura');
|
||||
expect(out.sort()).toEqual(['notes/struktura', 'struktura']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('FRONTMATTER_LINK_MAP integrity', () => {
|
||||
|
||||
Reference in New Issue
Block a user