diff --git a/INSTALL_FOR_AGENTS.md b/INSTALL_FOR_AGENTS.md index 421d2ff09..e0f24d187 100644 --- a/INSTALL_FOR_AGENTS.md +++ b/INSTALL_FOR_AGENTS.md @@ -163,11 +163,16 @@ and supports `--since YYYY-MM-DD` for incremental runs. ### Obsidian-style bare wikilinks (opt-in) +Path-qualified refs like `[[projects/struktura]]` or `[[janus/agents/drift-check]]` +resolve out of the box for ANY directory — the target just has to exist as a page +with that exact slug (`link_resolution.any_dir_exact_path`, on by default; misses +land in the unresolved report instead of dropping silently). + If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]` wikilinks — where `[[struktura]]` written in one folder means the page that lives -at `projects/struktura.md` in another — GBrain does NOT connect those by default. -Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`, -so a vault full of bare links shows up as a thin, broken graph. Turn on basename +at `projects/struktura.md` in another — GBrain does NOT connect those by default: +a bare name is fuzzy (basename match, possibly several winners), so a vault full +of bare links shows up as a thin, broken graph until you opt in. Turn on basename resolution so the cross-folder links connect: ```bash diff --git a/README.md b/README.md index 1f87d5883..de861fa79 100644 --- a/README.md +++ b/README.md @@ -254,7 +254,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. -**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). +**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. Path-qualified wikilinks (`[[any-dir/sub/name]]`) link to any existing page out of the box — exact slug match, any folder, misses surfaced in the unresolved report. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index c102a7ef5..ff8aff81b 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -193,7 +193,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/conversation-parser/` — 15-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (15 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, index 3 after `bold-paren-time`) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path). - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and runs through `extractFactsFromTurn()` so anchor-rich facts surface in `gbrain search`. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide); **page-level TERMINAL audit row** to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS matches the terminal row so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, used as-is — nested `withBudgetTracker` REPLACES; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline**; **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`) with CLI default reading `cycle.conversation_facts_backfill.types`; **fingerprint on sourceId only**; **string-encoded op-checkpoint** entries `"||"` for resume (durable audit is the facts terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job catches + persists + marks `completed` with `result.budget_exhausted=true`). The companion cycle phase `conversation_facts_backfill` (default OFF) iterates `listSources(engine)`, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source call. Two-layer cost AND walltime caps: per-source (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by `test/extract-conversation-facts.test.ts` (27 cases). Migration v94 adds partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (`transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite). `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when disabled; OK when caught up; WARN when >10 pages lack the terminal row, with paste-ready `gbrain doctor --remediate` step). `src/commands/sources.ts:runAudit` adds `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Schema-pack `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable) into the base seed; backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51` not pack extractable. `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. -- `src/core/link-extraction.ts` — shared library for the graph layer. `extractEntityRefs` (canonical) matches `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks; `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled`. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **Opt-in global-basename wikilink resolution** (issue #972, default off): `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in `extractEntityRefs`); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise` (multi-match by design — emits one edge per matching page). The single shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical), used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — source-scoped so a bare `[[name]]` never resolves to a same-tail page in a different source. `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug`, emits candidates tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the old `nullResolver` ternary). All three surfaces (FS extract, DB extract, `put_page` auto-link) tag provenance with `link_source='wikilink-resolved'`; `put_page` includes it in its reconcilable-edge set so stale basename edges are removed when the wikilink or the flag goes away. Exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). `gbrain doctor`'s `link_resolution_opportunity` check surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% match. Migration v113 widens `links_link_source_check` to admit `'wikilink-resolved'`; v114 (#1941) then opens it to any kebab-case provenance (`^[a-z][a-z0-9]*(-[a-z0-9]+)*$`, ≤64 chars) so external derivers register their own tag (e.g. `citation-graph`) without a migration. `LINK_EXTRACTOR_VERSION_TS` also lives here (bump like `CHUNKER_VERSION` to invalidate prior extract-stale stamps). Pinned by `test/link-extraction.test.ts`, `test/extract-fs.test.ts`, `test/doctor.test.ts`, `test/e2e/global-basename-pglite.test.ts`. +- `src/core/link-extraction.ts` — shared library for the graph layer. `extractEntityRefs` (canonical) matches `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks; `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled`. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **Opt-in global-basename wikilink resolution** (issue #972, default off): `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in `extractEntityRefs`); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise` (multi-match by design — emits one edge per matching page). The single shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical), used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — source-scoped so a bare `[[name]]` never resolves to a same-tail page in a different source. `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug`, emits candidates tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the old `nullResolver` ternary). All three surfaces (FS extract, DB extract, `put_page` auto-link) tag provenance with `link_source='wikilink-resolved'`; `put_page` includes it in its reconcilable-edge set so stale basename edges are removed when the wikilink or the flag goes away. Exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). `gbrain doctor`'s `link_resolution_opportunity` check surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% match. Migration v113 widens `links_link_source_check` to admit `'wikilink-resolved'`; v114 (#1941) then opens it to any kebab-case provenance (`^[a-z][a-z0-9]*(-[a-z0-9]+)*$`, ≤64 chars) so external derivers register their own tag (e.g. `citation-graph`) without a migration. **Generalized path resolution** (issue #1493, default ON): passes `2a'`/`2b'` (`QUALIFIED_WIKILINK_ANY_PATH_RE` / `WIKILINK_ANY_PATH_RE`) match path-shaped wikilinks `[[dir/.../name]]` for ANY directory prefix — not just `DIR_PATTERN` — tagged `EntityRef.exactPath: true`. `extractPageLinks` (opts.anyDirExactPath, default true; config `link_resolution.any_dir_exact_path` via `isAnyDirExactPathEnabled`, env `GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH`) verifies the literal slug exists via the optional `SlugResolver.slugExists` (exact match against the same source-scoped `getAllSlugs` snapshot the basename index uses) and emits the candidate exactly like a whitelisted wikilink (verb-inferred type, `link_source='markdown'`, reconcilable); a miss is recorded in the unresolved report as `{field: 'wikilink'}` — never a ghost edge, never a silent drop. Resolvers without `slugExists` (the stale-sweep nullResolver) emit unverified and rely on the callers' endpoint checks. Targets with uppercase/spaces are not canonical slugs and keep the pre-existing generic-pass behavior. `LINK_EXTRACTOR_VERSION_TS` also lives here (bump like `CHUNKER_VERSION` to invalidate prior extract-stale stamps). Pinned by `test/link-extraction.test.ts`, `test/link-extraction-any-dir.test.ts`, `test/extract-fs.test.ts`, `test/doctor.test.ts`, `test/e2e/global-basename-pglite.test.ts`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). No in-memory dedup pre-load — candidates buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, `created` counter returns real rows inserted. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id ` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id ] [--catch-up] [--dry-run] [--json]` branch (`extractStaleFromDB`) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three new `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush: link/timeline flush throws propagate and abort the batch; stamp LAST so a crash leaves pages unstamped and they re-extract idempotently (`addLinksBatch` ON CONFLICT DO NOTHING + timeline dedup). Race fix: `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Schema parity in schema.sql + pglite-schema.ts + schema-embedded.ts + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` gains `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` for `pagesAffected` at the inline-extract call site, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` adds `'extract.stale'` to `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` adds `links_extraction_lag` to `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp regression + crash-contract), `test/sync-inline-extract-stamps.serial.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite regression in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` — unified extract operator surface. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts`. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. `extract_health` doctor check reads last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok`. CLI: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted halt_rate desc + cost desc, top-5 + "more rows" hint, stable `schema_version: 1`); `gbrain extract --explain ` (resolution chain pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)`, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; ships as a stub-reporter). `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` — parses but refuses at runtime); `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Pinned by `test/extractable-spec-widening.test.ts` (22), `test/extract/receipt-writer.test.ts` (12, canonical PGLite block R3+R4), `test/extract/benchmark.test.ts` (17), `test/extract/status.test.ts` (15), `test/schema-pack/scaffold-extractable.test.ts` (15, privacy guards), `test/doctor-extract-health.test.ts` (8). - `src/commands/import.ts` — `gbrain import [--source-id ]`: page import with the path-set checkpoint. `--source-id ` routes pages to the named source (resolved via `resolveSourceWithTier()` at the boundary; consistent across `import`, `extract`, `graph-query`, `sources current`). Pinned by `test/import-source-id.test.ts`. diff --git a/llms-full.txt b/llms-full.txt index 4de2b11e4..0c64f312e 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1153,11 +1153,16 @@ and supports `--since YYYY-MM-DD` for incremental runs. ### Obsidian-style bare wikilinks (opt-in) +Path-qualified refs like `[[projects/struktura]]` or `[[janus/agents/drift-check]]` +resolve out of the box for ANY directory — the target just has to exist as a page +with that exact slug (`link_resolution.any_dir_exact_path`, on by default; misses +land in the unresolved report instead of dropping silently). + If the user imported an Obsidian or Notion vault that uses **bare** `[[note-name]]` wikilinks — where `[[struktura]]` written in one folder means the page that lives -at `projects/struktura.md` in another — GBrain does NOT connect those by default. -Out of the box it only resolves path-qualified refs like `[[projects/struktura]]`, -so a vault full of bare links shows up as a thin, broken graph. Turn on basename +at `projects/struktura.md` in another — GBrain does NOT connect those by default: +a bare name is fuzzy (basename match, possibly several winners), so a vault full +of bare links shows up as a thin, broken graph until you opt in. Turn on basename resolution so the cross-folder links connect: ```bash @@ -1748,7 +1753,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. -**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). +**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. Path-qualified wikilinks (`[[any-dir/sub/name]]`) link to any existing page out of the box — exact slug match, any folder, misses surfaced in the unresolved report. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph). **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 21eeaaef5..8cef69fa5 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -36,7 +36,8 @@ import type { PageType } from '../core/types.ts'; import { parseMarkdown } from '../core/markdown.ts'; import { extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver, - extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS, + extractFrontmatterLinks, isAnyDirExactPathEnabled, isGlobalBasenameEnabled, + LINK_EXTRACTOR_VERSION_TS, type SlugResolver, WIKILINK_BASENAME_LINK_TYPE, buildBasenameIndex, queryBasenameIndex, stripCodeBlocks, type UnresolvedFrontmatterRef, type LinkCandidate, @@ -126,7 +127,15 @@ export function resolveCandidateSources( : (fromSources.includes('default') ? 'default' : fromSources[0]); const targetSources = slugToSources.get(c.targetSlug) ?? []; let toSourceId: string; - if (targetSources.includes(fromSourceId)) { + if (c.targetSourceId) { + // Issue #1493 (codex P1): a qualified wikilink `[[src:slug]]` pins the + // target source — that IS the qualification's meaning (v0.17.0). The pin + // overrides the local-first/default fallback: the target must exist in + // the pinned source or the candidate is skipped (never misdirected to a + // same-slug page in the origin/default source). + if (!targetSources.includes(c.targetSourceId)) return null; + toSourceId = c.targetSourceId; + } else if (targetSources.includes(fromSourceId)) { toSourceId = fromSourceId; } else if (targetSources.includes('default')) { toSourceId = 'default'; @@ -179,6 +188,14 @@ interface ExtractResult { links_created: number; timeline_entries_created: number; pages_processed: number; + /** + * Issue #1493 (codex P2): unresolved refs (frontmatter misses + + * wikilink exact-path misses) from the DB links pass. Present only when + * non-empty so pre-existing JSON consumers see an unchanged shape on + * clean runs. `--json` prints the whole ExtractResult, so agents see + * the misses instead of only counts. + */ + unresolved_refs?: UnresolvedFrontmatterRef[]; } // --- Shared walker --- @@ -947,6 +964,9 @@ Status (v0.42): const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since, { includeFrontmatter, sourceIdFilter, stampWatermark: subcommand === 'all' }); result.links_created = r.created; result.pages_processed = r.pages; + // Issue #1493 (codex P2): thread unresolved refs into the JSON + // result — --json used to suppress the miss report entirely. + if (r.unresolved.length > 0) result.unresolved_refs = r.unresolved; } if (subcommand === 'timeline' || subcommand === 'all') { const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter }); @@ -1381,6 +1401,9 @@ async function extractLinksFromDB( // Issue #972: opt-in global-basename wikilink resolution. Read once // per extract run; threaded into each extractPageLinks call. const globalBasename = await isGlobalBasenameEnabled(engine); + // Issue #1493: exact-path resolution for wikilinks outside the DIR_PATTERN + // whitelist. On by default; read once per run like globalBasename. + const anyDirExactPath = await isAnyDirExactPathEnabled(engine); // v0.32.8: listAllPageRefs enumerates (slug, source_id) so we can thread // sourceId to getPage AND build a cross-source resolution map for link // disambiguation. Pre-fix used getAllSlugs() which collapsed @@ -1461,7 +1484,7 @@ async function extractLinksFromDB( // basename lookup; off by default for back-compat. const extracted = await extractPageLinks( slug, fullContent, page.frontmatter, page.type, resolver, - { skipFrontmatter: !includeFrontmatter, globalBasename }, + { skipFrontmatter: !includeFrontmatter, globalBasename, anyDirExactPath }, ); unresolved.push(...extracted.unresolved); @@ -1528,10 +1551,13 @@ async function extractLinksFromDB( if (!jsonMode) { const label = dryRun ? '(dry run) would create' : 'created'; console.log(`Links: ${label} ${created} from ${processed} pages (db source)`); - if (includeFrontmatter && unresolved.length > 0) { - // Top-20 preview of unresolvable frontmatter names so the user can - // see where the graph has holes (codex tension 6.4). - console.log(`Unresolved frontmatter refs: ${unresolved.length} total`); + if (unresolved.length > 0) { + // Top-20 preview of unresolvable names so the user can see where the + // graph has holes (codex tension 6.4). Pre-#1493 this was gated on + // includeFrontmatter because frontmatter was the only unresolved + // source; path-shaped wikilink misses (field='wikilink') now report + // here too, so the gate is simply "anything unresolved". + console.log(`Unresolved refs (frontmatter + wikilinks): ${unresolved.length} total`); const bucket = new Map(); for (const u of unresolved) { const key = `${u.field}:${u.name}`; @@ -1660,7 +1686,7 @@ async function extractStaleFromDB( sourceIdFilter?: string; catchUp: boolean; }, -): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number }> { +): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number; unresolved?: UnresolvedFrontmatterRef[] }> { const { dryRun, jsonMode, includeFrontmatter, sourceIdFilter, catchUp } = opts; const versionTs = LINK_EXTRACTOR_VERSION_TS; @@ -1684,9 +1710,39 @@ async function extractStaleFromDB( // Batch mode = pg_trgm + exact only, NO per-name search fallback. The // resolution map sees ALL sources so qualified cross-source wikilinks resolve // even when --source-id scopes the stale SCAN. - const resolver = makeResolver(engine, { mode: 'batch' }); - const nullResolver = { resolve: async () => null as string | null }; - const activeResolver = includeFrontmatter ? resolver : nullResolver; + // Issue #1493 (codex P2): always pass the REAL resolver and gate the + // frontmatter pass via opts.skipFrontmatter — the same migration + // extractLinksFromDB made for issue #972. The old nullResolver ternary + // meant the default sweep (no --include-frontmatter) had no slugExists, + // so any-dir exact-path misses were silently discarded by the endpoint + // check instead of being verified + recorded as unresolved. + // + // Issue #1493 (codex P2, round 2): the resolver is scoped PER PAGE SOURCE, + // not one unscoped instance for the sweep. An unscoped slugExists spans + // all sources, so an alpha page's `[[janus/foo]]` whose target exists + // only in beta passed verification, became a candidate, was rejected by + // resolveCandidateSources (not in alpha ∪ default) — and the miss was + // never recorded. Per-source resolvers give each page the SAME + // resolution domain resolveCandidateSources applies (page source ∪ + // default). Cached per source_id: getAllSlugs runs at most twice per + // distinct source across the whole sweep (scoped + default overlay), + // matching the DB path's fetch discipline. + const resolverBySource = new Map(); + function resolverForSource(sourceId: string): SlugResolver { + let r = resolverBySource.get(sourceId); + if (!r) { + r = makeResolver(engine, { mode: 'batch', sourceId }); + resolverBySource.set(sourceId, r); + } + return r; + } + // Issue #1493: exact-path resolution for wikilinks outside the DIR_PATTERN + // whitelist. Read once per sweep. + const anyDirExactPath = await isAnyDirExactPathEnabled(engine); + // Issue #1493 (codex P2): aggregate unresolved refs (wikilink misses +, + // with --include-frontmatter, frontmatter misses) and surface them in the + // sweep summary — consistent with the DB path. + const unresolved: UnresolvedFrontmatterRef[] = []; const allRefs = await engine.listAllPageRefs(); const allSlugs = new Set(); const slugToSources = new Map(); @@ -1718,8 +1774,10 @@ async function extractStaleFromDB( for (const page of rows) { const fullContent = page.compiled_truth + '\n' + page.timeline; const extracted = await extractPageLinks( - page.slug, fullContent, page.frontmatter, page.type, activeResolver, + page.slug, fullContent, page.frontmatter, page.type, resolverForSource(page.source_id), + { skipFrontmatter: !includeFrontmatter, anyDirExactPath }, ); + unresolved.push(...extracted.unresolved); for (const c of extracted.candidates) { const r = resolveCandidateSources(c, page.slug, page.source_id, allSlugs, slugToSources); if (!r) continue; @@ -1772,6 +1830,20 @@ async function extractStaleFromDB( if (!jsonMode) { console.log(`Extract --stale: ${linksCreated} link(s) + ${timelineCreated} timeline entr(ies) from ${pagesProcessed} page(s).`); + // Issue #1493 (codex P2): surface unresolved refs (top-10 preview), + // mirroring the DB path's summary — misses must not be silent. + if (unresolved.length > 0) { + console.log(`Unresolved refs (frontmatter + wikilinks): ${unresolved.length} total`); + const bucket = new Map(); + for (const u of unresolved) { + const key = `${u.field}:${u.name}`; + bucket.set(key, (bucket.get(key) || 0) + 1); + } + const top = Array.from(bucket.entries()).sort((a, b) => b[1] - a[1]).slice(0, 10); + for (const [key, count] of top) { + console.log(` ${count}× ${key}`); + } + } if (budgetHit && staleRemaining > 0) { console.log(`Time budget reached — ${staleRemaining} page(s) still stale. Re-run 'gbrain extract --stale' (or pass --catch-up) to continue.`); } @@ -1779,9 +1851,12 @@ async function extractStaleFromDB( process.stdout.write(JSON.stringify({ action: 'extract_stale_done', links_created: linksCreated, timeline_created: timelineCreated, pages_processed: pagesProcessed, stale_remaining: staleRemaining, budget_hit: budgetHit, + // Issue #1493 (codex P2): agents consuming --json must see the misses. + unresolved_count: unresolved.length, + ...(unresolved.length > 0 ? { unresolved_refs: unresolved } : {}), }) + '\n'); } - return { linksCreated, timelineCreated, pagesProcessed, staleRemaining }; + return { linksCreated, timelineCreated, pagesProcessed, staleRemaining, unresolved }; } /** diff --git a/src/core/config.ts b/src/core/config.ts index 4ca00cdc9..9b146123d 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -963,6 +963,9 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ // Link resolution (issue #972) 'link_resolution', 'link_resolution.global_basename', + // Issue #1493: any-dir exact-path wikilink resolution (default ON; this + // key is the documented escape hatch, so `config set` must accept it). + 'link_resolution.any_dir_exact_path', // Spend controls (v0.42.42.0, issue #2139). Previously `--force`-only — the // operator had to discover these by reading source. Registered so `config // set` accepts them directly. See docs/operations/spend-controls.md. diff --git a/src/core/link-extraction.ts b/src/core/link-extraction.ts index c0fc2644a..23ffa72e5 100644 --- a/src/core/link-extraction.ts +++ b/src/core/link-extraction.ts @@ -28,7 +28,10 @@ import { ensureWellFormed } from './text-safe.ts'; * OR updated_at > links_extracted_at`. It is an ISO-8601 string (NOT a number) — * the column is TIMESTAMPTZ and the predicate binds it as `::timestamptz`. */ -export const LINK_EXTRACTOR_VERSION_TS = '2026-05-31T00:00:00Z'; +// Issue #1493: bumped for the any-dir exact-path wikilink pass so +// already-stamped pages re-extract on the next `extract --stale` sweep +// and gain their newly-resolvable path-shaped edges. +export const LINK_EXTRACTOR_VERSION_TS = '2026-07-18T00:00:00Z'; // ─── Entity references ────────────────────────────────────────── @@ -60,6 +63,16 @@ export interface EntityRef { * LinkCandidate per resolved match. */ needsResolution?: boolean; + /** + * Issue #1493 (generalized path resolution): set when the ref came from + * the path-shaped any-directory wikilink passes (2a'/2b') — the target + * is `dir/.../name` in canonical slug form but the top-level dir is NOT + * in the DIR_PATTERN whitelist. The `slug` field IS the literal slug the + * author wrote; extractPageLinks verifies it exists (exact match, via + * `SlugResolver.slugExists`) before emitting a candidate, and records a + * miss in the unresolved report instead of dropping it silently. + */ + exactPath?: boolean; } /** @@ -113,6 +126,39 @@ const WIKILINK_RE = new RegExp( 'g', ); +/** + * Issue #1493 (generalized path resolution): one path segment in canonical + * slug form. Conservative ASCII subset of sync.ts's SLUG_SEGMENT_PATTERN: + * must START with [a-z0-9] (so `..`/`.hidden` relative-path segments never + * match) and may continue with dots, underscores and hyphens (`janus-ui`, + * `v1.0.0`, `foo_bar`). CJK slugs are deliberately NOT covered here — they + * keep the pre-existing behavior (generic pass 2c / basename resolution). + */ +const SLUG_PATH_SEGMENT = '[a-z0-9][a-z0-9._-]*'; + +/** + * Issue #1493: path-shaped wikilink `[[dir/name]]` / `[[dir/sub/name]]` + * (>= 2 segments) with ANY directory prefix — NOT gated by DIR_PATTERN. + * The whitelist silently dropped every wikilink whose target lives outside + * the recognized entity dirs (`[[janus/agents/drift-check]]`), losing the + * authored graph with no candidate and no unresolved report. + * + * Unlike the #972 bare-name pass (fuzzy by design, opt-in), a path-shaped + * target is already an exact slug: it resolves by exact match against + * existing pages, so it is safe to run by default + * (`link_resolution.any_dir_exact_path`, default on). + * + * Runs AFTER WIKILINK_RE with its spans masked, so whitelisted dirs keep + * their existing pass (2b) untouched; anything this matches has a + * non-whitelisted top dir. Same alias/anchor tail shape as WIKILINK_RE. + * The strict slug grammar means targets with uppercase or spaces do NOT + * match — they fall through to the generic pass 2c as before. + */ +const WIKILINK_ANY_PATH_RE = new RegExp( + `\\[\\[(${SLUG_PATH_SEGMENT}(?:\\/${SLUG_PATH_SEGMENT})+)(?:#[^|\\]]*?)?(?:\\|([^\\]]+?))?\\]\\]`, + 'g', +); + /** * v0.17.0: qualified wikilink `[[source-id:dir/slug]]` or * `[[source-id:dir/slug|Display Text]]`. The source-id segment pins the @@ -130,6 +176,18 @@ const QUALIFIED_WIKILINK_RE = new RegExp( 'g', ); +/** + * Issue #1493: qualified variant of WIKILINK_ANY_PATH_RE — + * `[[source-id:dir/name]]` with a non-whitelisted dir. Previously these + * were dropped by ALL passes: QUALIFIED_WIKILINK_RE requires DIR_PATTERN + * and the generic pass 2c skips any token containing `:`. Runs after + * QUALIFIED_WIKILINK_RE with its spans masked, same source-id grammar. + */ +const QUALIFIED_WIKILINK_ANY_PATH_RE = new RegExp( + `\\[\\[([a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?):(${SLUG_PATH_SEGMENT}(?:\\/${SLUG_PATH_SEGMENT})+)(?:#[^|\\]]*?)?(?:\\|([^\\]]+?))?\\]\\]`, + 'g', +); + /** * Issue #972: generic Obsidian-style wikilink — matches `[[anything]]` * with no DIR_PATTERN gate. Wiki/topic/learning pages frequently link @@ -318,6 +376,16 @@ export function extractEntityRefs(content: string): EntityRef[] { markdownRanges.push([match.index, match.index + match[0].length]); } + // Issue #972 (codex [P2]) / #1493: markdown-link labels containing a + // wikilink — `[see [[acme]]](companies/acme.md)` — are masked out of the + // generalized (2a'/2b') and generic (2c) scans so the inner `[[...]]` + // stays inert. Computed once up front; consumed by those passes below. + const labelWikilinkRanges: Array<[number, number]> = []; + const labelWlPattern = new RegExp(MARKDOWN_LABEL_WIKILINK_RE.source, MARKDOWN_LABEL_WIKILINK_RE.flags); + while ((match = labelWlPattern.exec(stripped)) !== null) { + labelWikilinkRanges.push([match.index, match.index + match[0].length]); + } + // 2a. v0.17.0 qualified wikilinks: [[source-id:path]] or [[source-id:path|Display]] // Must run BEFORE the unqualified pass or we'd double-emit. We also // mask out the matched spans so pass 2b can't grab them. @@ -335,10 +403,31 @@ export function extractEntityRefs(content: string): EntityRef[] { qualifiedRanges.push([match.index, match.index + match[0].length]); } + // 2a'. Issue #1493: qualified path-shaped wikilinks outside DIR_PATTERN + // ([[src:janus/foo]]). Masked by 2a's spans so whitelisted qualified + // links are never re-emitted. Tagged `exactPath: true`; the target + // may live in a different source, so existence is left to the + // caller's cross-source resolution (same as whitelisted qualified + // refs) — extractPageLinks does NOT slugExists-check qualified refs. + const qualifiedPathRanges: Array<[number, number]> = []; + const qualAnyMasked = maskRanges(stripped, [...qualifiedRanges, ...labelWikilinkRanges]); + const qualAnyPattern = new RegExp(QUALIFIED_WIKILINK_ANY_PATH_RE.source, QUALIFIED_WIKILINK_ANY_PATH_RE.flags); + while ((match = qualAnyPattern.exec(qualAnyMasked)) !== null) { + const sourceId = match[1]; + let slug = match[2].trim(); + if (!slug) continue; + if (slug.includes('://')) continue; + if (slug.endsWith('.md')) slug = slug.slice(0, -3); + const displayName = (match[3] || slug).trim(); + const dir = slug.split('/')[0]; + refs.push({ name: displayName, slug, dir, sourceId, exactPath: true }); + qualifiedPathRanges.push([match.index, match.index + match[0].length]); + } + // 2b. Unqualified Obsidian wikilinks: [[path]] or [[path|Display Text]] // Same shape rule: omit sourceId when unqualified. const unqualifiedRanges: Array<[number, number]> = []; - const unmasked = maskRanges(stripped, qualifiedRanges); + const unmasked = maskRanges(stripped, [...qualifiedRanges, ...qualifiedPathRanges]); const wikiPattern = new RegExp(WIKILINK_RE.source, WIKILINK_RE.flags); while ((match = wikiPattern.exec(unmasked)) !== null) { let slug = match[1].trim(); @@ -351,24 +440,45 @@ export function extractEntityRefs(content: string): EntityRef[] { unqualifiedRanges.push([match.index, match.index + match[0].length]); } + // 2b'. Issue #1493: path-shaped wikilinks with ANY directory prefix + // ([[janus/agents/drift-check]], [[janus-ui/docs/tier2-agent-plan]]). + // Masked by 2b's spans so whitelisted dirs keep their existing pass + // untouched — anything matched here has a non-whitelisted top dir. + // Tagged `exactPath: true` so extractPageLinks verifies the slug + // exists (exact match) before emitting, and records misses in the + // unresolved report instead of silently dropping them. + const pathRanges: Array<[number, number]> = []; + const anyPathMasked = maskRanges( + stripped, + [...markdownRanges, ...qualifiedRanges, ...qualifiedPathRanges, ...unqualifiedRanges, ...labelWikilinkRanges], + ); + const anyPathPattern = new RegExp(WIKILINK_ANY_PATH_RE.source, WIKILINK_ANY_PATH_RE.flags); + while ((match = anyPathPattern.exec(anyPathMasked)) !== null) { + let slug = match[1].trim(); + if (!slug) continue; + if (slug.includes('://')) continue; + if (slug.endsWith('.md')) slug = slug.slice(0, -3); + const displayName = (match[2] || slug).trim(); + const dir = slug.split('/')[0]; + refs.push({ name: displayName, slug, dir, exactPath: true }); + pathRanges.push([match.index, match.index + match[0].length]); + } + // 2c. Issue #972: generic `[[bare-name]]` wikilinks not gated by // DIR_PATTERN. Wiki/topic/learning content frequently uses these // where the bare text isn't a canonical brain slug. Tagged // `needsResolution: true` so the caller routes them through a // SlugResolver's `resolveBasenameMatches` before persisting. - // Mask out 2a/2b ranges so we don't double-emit; skip qualified- - // syntax tokens (contain `:`) that would be 2a's job. Issue #972 - // (codex [P2]): ALSO mask pass-1 markdown-link ranges so a wikilink - // inside a markdown label — `[see [[acme]]](companies/acme.md)` — - // doesn't spawn a stray generic basename ref from inside the label. - const labelWikilinkRanges: Array<[number, number]> = []; - const labelWlPattern = new RegExp(MARKDOWN_LABEL_WIKILINK_RE.source, MARKDOWN_LABEL_WIKILINK_RE.flags); - while ((match = labelWlPattern.exec(stripped)) !== null) { - labelWikilinkRanges.push([match.index, match.index + match[0].length]); - } + // Mask out 2a/2a'/2b/2b' ranges so we don't double-emit; skip + // qualified-syntax tokens (contain `:`) that would be 2a's job. + // Issue #972 (codex [P2]): ALSO mask pass-1 markdown-link ranges + // (via labelWikilinkRanges, computed above 2a) so a wikilink inside + // a markdown label — `[see [[acme]]](companies/acme.md)` — doesn't + // spawn a stray generic basename ref from inside the label. const genericMasked = maskRanges( stripped, - [...markdownRanges, ...qualifiedRanges, ...unqualifiedRanges, ...labelWikilinkRanges], + [...markdownRanges, ...qualifiedRanges, ...qualifiedPathRanges, + ...unqualifiedRanges, ...pathRanges, ...labelWikilinkRanges], ); const genericPattern = new RegExp(WIKILINK_GENERIC_RE.source, WIKILINK_GENERIC_RE.flags); while ((match = genericPattern.exec(genericMasked)) !== null) { @@ -431,6 +541,18 @@ export interface LinkCandidate { originSlug?: string; /** Frontmatter field name (e.g. 'key_people'), for debug + unresolved report. */ originField?: string; + /** + * Issue #1493 (codex P1): source pin from a qualified wikilink + * `[[source-id:slug]]`, threaded from EntityRef.sourceId. Callers doing + * cross-source resolution (resolveCandidateSources) must honor it — the + * target must exist in THIS source — instead of applying the + * local-first/default fallback used by unqualified refs. Absent = + * unqualified. Pre-#1493 the pin was dropped at this layer for EVERY + * qualified wikilink (whitelisted dirs included), so `[[b:slug]]` + * written in source `a` either linked to a same-slug page in `a`/default + * (misdirected) or vanished; now the pinned source wins. + */ + targetSourceId?: string; } /** @@ -468,9 +590,19 @@ export async function extractPageLinks( frontmatter: Record, pageType: PageType, resolver: SlugResolver, - opts: { globalBasename?: boolean; skipFrontmatter?: boolean } = {}, + opts: { globalBasename?: boolean; skipFrontmatter?: boolean; anyDirExactPath?: boolean } = {}, ): Promise { const candidates: LinkCandidate[] = []; + // Issue #1493: path-shaped wikilink targets that failed the exact-match + // existence check. Surfaced through the same unresolved report as + // frontmatter misses (field='wikilink') so the gap is visible in the + // put_page auto_links response and the extract summary — never silently + // dropped, never written as a ghost edge. Deduped by target within page. + const wikilinkUnresolved: UnresolvedFrontmatterRef[] = []; + const wikilinkUnresolvedSeen = new Set(); + // Default ON: exact slug match can't false-positive the way basename + // matching can, so unlike global_basename this needs no opt-in. + const anyDirExactPath = opts.anyDirExactPath !== false; // 1. Markdown entity refs. for (const ref of extractEntityRefs(content)) { @@ -506,6 +638,30 @@ export async function extractPageLinks( } continue; } + // Issue #1493: path-shaped wikilink outside the DIR_PATTERN whitelist. + // The written target IS a canonical slug — verify it exists (exact + // match, no fuzzing) and then treat it exactly like a whitelisted + // wikilink (verb-inferred type, linkSource 'markdown', reconcilable). + if (ref.exactPath) { + if (!anyDirExactPath) continue; // gated off → legacy silent drop + // Qualified refs ([[src:janus/foo]]) pin a target in a possibly + // different source; the resolver's snapshot is scoped to THIS source, + // so existence is left to the caller's cross-source endpoint check — + // the same dead-link protection whitelisted qualified refs get. + if (ref.sourceId == null && typeof resolver.slugExists === 'function') { + if (!(await resolver.slugExists(ref.slug))) { + if (!wikilinkUnresolvedSeen.has(ref.slug)) { + wikilinkUnresolvedSeen.add(ref.slug); + wikilinkUnresolved.push({ field: 'wikilink', name: ref.slug }); + } + continue; + } + } + // Falls through to the shared emission below. When the resolver + // lacks slugExists (nullResolver in extract --stale, synthetic test + // resolvers), the candidate goes out unverified and the caller's + // existing dead-link filter drops misses. + } const idx = content.indexOf(ref.name); // Wider context window (240 chars vs original 80) catches verbs that // appear at sentence-or-paragraph distance from the slug — common in @@ -517,6 +673,10 @@ export async function extractPageLinks( linkType: inferLinkType(pageType, context, content, ref.slug), context, linkSource: 'markdown', + // Issue #1493 (codex P1): carry the qualified-wikilink source pin + // ([[src:slug]] → EntityRef.sourceId) so resolveCandidateSources can + // honor it. Applies to whitelisted AND any-dir qualified refs alike. + ...(ref.sourceId ? { targetSourceId: ref.sourceId } : {}), }); } @@ -524,14 +684,30 @@ export async function extractPageLinks( // Limited to the same entity directories ENTITY_REF_RE covers. // Code blocks are stripped first — slugs in code samples are not real refs. const strippedContent = stripCodeBlocks(content); + // Issue #1493 (codex P1, tightened in round 2): mask qualified-wikilink + // spans out of the bare-slug scan. The slug tail inside + // `[[src:people/alice]]` double-matches here; pass 2a/2a' already emitted + // it WITH its source pin, and since the pin is part of the dedup key the + // unpinned duplicate would persist a second edge. Masking the exact + // wikilink spans (instead of skipping any `:`-preceded match) keeps + // legitimate prose like `Owner:people/alice` extracting as before. + const qualifiedSpans: Array<[number, number]> = []; + for (const re of [QUALIFIED_WIKILINK_RE, QUALIFIED_WIKILINK_ANY_PATH_RE]) { + const p = new RegExp(re.source, re.flags); + let qm: RegExpExecArray | null; + while ((qm = p.exec(strippedContent)) !== null) { + qualifiedSpans.push([qm.index, qm.index + qm[0].length]); + } + } + const bareScan = maskRanges(strippedContent, qualifiedSpans); const bareRe = new RegExp( `\\b(${DIR_PATTERN}\\/[a-z0-9][a-z0-9/-]*[a-z0-9])\\b`, 'g', ); let m: RegExpExecArray | null; - while ((m = bareRe.exec(strippedContent)) !== null) { + while ((m = bareRe.exec(bareScan)) !== null) { // Skip matches that are part of a markdown link (already handled above). - const charBefore = m.index > 0 ? strippedContent[m.index - 1] : ''; + const charBefore = m.index > 0 ? bareScan[m.index - 1] : ''; if (charBefore === '/' || charBefore === '(') continue; const context = excerpt(strippedContent, m.index, 240); candidates.push({ @@ -568,12 +744,17 @@ export async function extractPageLinks( const seen = new Set(); const result: LinkCandidate[] = []; for (const c of candidates) { - const key = `${c.fromSlug ?? ''}\u0000${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? ''}`; + // Issue #1493 (codex P1): targetSourceId participates in the key — a + // pinned `[[b:slug]]` and an unqualified `[[slug]]` can resolve to + // different (to_slug, to_source_id) rows, so they must not collapse. + const key = `${c.fromSlug ?? ''}\u0000${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? ''}\u0000${c.targetSourceId ?? ''}`; if (seen.has(key)) continue; seen.add(key); result.push(c); } - return { candidates: result, unresolved: fmUnresolved }; + // Issue #1493: wikilink misses ride alongside frontmatter misses in the + // one unresolved report. + return { candidates: result, unresolved: [...fmUnresolved, ...wikilinkUnresolved] }; } /** @@ -819,6 +1000,20 @@ export interface SlugResolver { * a single extract run (build the index once, reuse it). */ resolveBasenameMatches?(name: string): Promise; + /** + * Issue #1493 (generalized path resolution): exact-match existence check + * for a canonical slug. Used by extractPageLinks to verify path-shaped + * wikilink targets outside the DIR_PATTERN whitelist before emitting a + * candidate — a miss goes into the unresolved report, never a ghost edge. + * + * Optional like resolveBasenameMatches. When absent, extractPageLinks + * emits the candidate unverified and relies on the caller's existing + * dead-link protection (runAutoLink's allSlugs filter / + * resolveCandidateSources' endpoint check) — same treatment whitelisted + * refs get. Implementations should share the per-run slug snapshot with + * the basename index (one getAllSlugs call, reused). + */ + slugExists?(slug: string): Promise; } /** @@ -899,30 +1094,63 @@ export function makeResolver( // index keys BOTH the raw tail and the slugified-tail so a wikilink // `[[Fast-Weigh]]` matches a page slugged `companies/fast-weigh` as // well as `notes/Fast-Weigh`. - let basenameIndex: Map | null = null; - async function ensureBasenameIndex(): Promise> { - if (basenameIndex !== null) return basenameIndex; - const idx = new Map(); + // Issue #1493: the raw slug snapshot backing BOTH the basename index and + // the exact-match slugExists check. One getAllSlugs call per resolver + // instance, shared by both consumers. Load failure (or an engine without + // getAllSlugs) degrades to an empty set — resolveBasenameMatches finds + // nothing and slugExists returns false. Never throws. + let allSlugsSet: Set | null = null; + async function ensureAllSlugsSet(): Promise> { + if (allSlugsSet !== null) return allSlugsSet; if (typeof engine.getAllSlugs !== 'function') { - basenameIndex = idx; - return idx; + allSlugsSet = new Set(); + return allSlugsSet; } try { - // Issue #972 (codex [P1]): scope the basename index to the resolver's + // Issue #972 (codex [P1]): scope the slug snapshot to the resolver's // source. getAllSlugs({sourceId}) keeps wikilink resolution from // spanning unrelated sources — a bare [[name]] must NOT resolve to a // same-tail page in a DIFFERENT source and create a cross-source edge. // #972 is "global basename across folders," not "cross-source federation." const all = await engine.getAllSlugs(opts.sourceId ? { sourceId: opts.sourceId } : undefined); - // Issue #972 (codex [P2] DRY): one shared index builder for all surfaces. - basenameIndex = buildBasenameIndex(all); - return basenameIndex; + allSlugsSet = all instanceof Set ? all : new Set(all); } catch { - // Index build failed — empty map → resolveBasenameMatches finds - // nothing, resolver continues as if the flag was off. Never throw. + allSlugsSet = new Set(); } - basenameIndex = idx; - return idx; + return allSlugsSet; + } + + let basenameIndex: Map | null = null; + async function ensureBasenameIndex(): Promise> { + if (basenameIndex !== null) return basenameIndex; + // Issue #972 (codex [P2] DRY): one shared index builder for all surfaces. + basenameIndex = buildBasenameIndex(await ensureAllSlugsSet()); + return basenameIndex; + } + + // Issue #1493 (codex P1): slugExists domain = scoped source ∪ 'default'. + // resolveCandidateSources resolves an unqualified target local-first, + // THEN falls back to the default source — so a source-scoped extract of + // source `a` must not pre-discard a candidate whose target lives only in + // `default` (whitelisted refs bypass slugExists and survive to that + // fallback; any-dir refs must survive the SAME resolution domain). The + // basename index deliberately stays scoped (#972 [P1]: no cross-source + // basename edges) — only exact-match existence gets the default overlay. + let slugExistsDomain: Set | null = null; + async function ensureSlugExistsDomain(): Promise> { + if (slugExistsDomain !== null) return slugExistsDomain; + const scoped = await ensureAllSlugsSet(); + if (!opts.sourceId || opts.sourceId === 'default' || typeof engine.getAllSlugs !== 'function') { + slugExistsDomain = scoped; + return slugExistsDomain; + } + try { + const def = await engine.getAllSlugs({ sourceId: 'default' }); + slugExistsDomain = new Set([...scoped, ...def]); + } catch { + slugExistsDomain = scoped; + } + return slugExistsDomain; } return { @@ -932,6 +1160,14 @@ export function makeResolver( return queryBasenameIndex(await ensureBasenameIndex(), name); }, + async slugExists(slug: string): Promise { + // Issue #1493: exact-match existence for path-shaped wikilink targets. + // Scoped-source snapshot plus the default-source fallback overlay + // (mirrors resolveCandidateSources' resolution domain); no fuzzing. + if (!slug || typeof slug !== 'string') return false; + return (await ensureSlugExistsDomain()).has(slug); + }, + async resolve(name: string, dirHint?: string | string[]): Promise { if (!name || typeof name !== 'string') return null; const trimmed = name.trim(); @@ -1252,3 +1488,35 @@ export async function isGlobalBasenameEnabled(engine: BrainEngine): Promise { + const OFF = ['false', '0', 'no', 'off']; + const envVal = process.env.GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH; + if (envVal != null) { + return !OFF.includes(envVal.trim().toLowerCase()); + } + const val = await engine.getConfig('link_resolution.any_dir_exact_path'); + if (val == null) return true; + return !OFF.includes(val.trim().toLowerCase()); +} diff --git a/src/core/operations.ts b/src/core/operations.ts index ab2d4cbf1..242c3dc1e 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -16,7 +16,7 @@ import { expandQuery } from './search/expansion.ts'; import { dedupResults } from './search/dedup.ts'; import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts'; import type { HybridSearchMeta } from './types.ts'; -import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, isGlobalBasenameEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts'; +import { extractPageLinks, isAnyDirExactPathEnabled, isAutoLinkEnabled, isAutoTimelineEnabled, isGlobalBasenameEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts'; import { isFactsBackstopEligible } from './facts/eligibility.ts'; import { stripTakesFence } from './takes-fence.ts'; import { stripFactsFence } from './facts-fence.ts'; @@ -1157,9 +1157,12 @@ async function runAutoLink( const resolver = makeResolver(engine, { mode: 'live', sourceId: opts?.sourceId }); // Issue #972: opt-in bare-wikilink basename resolution. Off by default. const globalBasename = await isGlobalBasenameEnabled(engine); + // Issue #1493: exact-path resolution for wikilinks outside the DIR_PATTERN + // whitelist. On by default (escape hatch: link_resolution.any_dir_exact_path). + const anyDirExactPath = await isAnyDirExactPathEnabled(engine); const { candidates, unresolved } = await extractPageLinks( slug, fullContent, parsed.frontmatter, parsed.type, resolver, - { globalBasename }, + { globalBasename, anyDirExactPath }, ); // Resolve which targets exist (skip refs to non-existent pages to avoid FK @@ -1167,9 +1170,32 @@ async function runAutoLink( // v0.31.8 (D12): scoped to the source when opts.sourceId is set so wikilink // resolution doesn't span unrelated sources. const allSlugs = await engine.getAllSlugs(sourceOpts); - const valid = candidates.filter(c => - allSlugs.has(c.targetSlug) && (!c.fromSlug || allSlugs.has(c.fromSlug)) + // Issue #1493 (codex P1): put_page auto-link writes are single-source + // (linkSourceOpts pins from/to to the page's own source), so a qualified + // wikilink pinned to a DIFFERENT source cannot be honored here — skip it + // rather than misdirect the edge to a same-slug page in this source. + // Cross-source pinned edges are the extract paths' job + // (resolveCandidateSources honors targetSourceId). + const effectiveSourceId = opts?.sourceId ?? 'default'; + const accepted = candidates.filter(c => + allSlugs.has(c.targetSlug) && (!c.fromSlug || allSlugs.has(c.fromSlug)) && + (!c.targetSourceId || c.targetSourceId === effectiveSourceId) ); + // Issue #1493 (codex P2, round 2): within put_page's single-source write + // scope, a pinned `[[alpha:people/alice]]` and an unpinned + // `[[people/alice]]` on an alpha page resolve to the SAME edge row — the + // pin survived extractPageLinks' within-page dedup (it's part of that + // key) and the guard above (pin === this source). Re-dedupe by the + // RESOLVED single-source edge key so addLink isn't called twice and + // `created` isn't double-counted (the second insert hits ON CONFLICT, + // leaving one row but two counts). + const seenEdgeKeys = new Set(); + const valid = accepted.filter(c => { + const key = `${c.fromSlug ?? slug}\u0000${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? 'markdown'}`; + if (seenEdgeKeys.has(key)) return false; + seenEdgeKeys.add(key); + return true; + }); // Split candidates by direction. Outgoing (fromSlug === slug or unset) are // this page's own edges, reconciled against getLinks(slug). Incoming diff --git a/test/config.test.ts b/test/config.test.ts index cdfcbf116..991ea8f84 100644 --- a/test/config.test.ts +++ b/test/config.test.ts @@ -264,4 +264,13 @@ describe('KNOWN_CONFIG_KEYS — documented enable commands must be registered', // chronicle.tz (chronicleTz) + future chronicle.* knobs. expect(KNOWN_CONFIG_KEY_PREFIXES.some(p => 'chronicle.tz'.startsWith(p))).toBe(true); }); + + test('link_resolution keys are registered (#972 opt-in + #1493 escape hatch)', async () => { + const { KNOWN_CONFIG_KEYS } = await import('../src/core/config.ts'); + expect(KNOWN_CONFIG_KEYS).toContain('link_resolution.global_basename'); + // Issue #1493 (codex P2): the documented disable command is + // `gbrain config set link_resolution.any_dir_exact_path false` — it + // must work without --force. + expect(KNOWN_CONFIG_KEYS).toContain('link_resolution.any_dir_exact_path'); + }); }); diff --git a/test/extract-db.test.ts b/test/extract-db.test.ts index 97a9f8761..f06754d2c 100644 --- a/test/extract-db.test.ts +++ b/test/extract-db.test.ts @@ -249,3 +249,57 @@ describe('gbrain extract all --source db', () => { expect(entries.length).toBe(1); }); }); + +// ─── Issue #1493 (codex P2): unresolved refs in the --json result ──── + +describe('extract links --source db --json: unresolved refs (#1493)', () => { + beforeEach(truncateAll); + + const conceptPage = (title: string, body = ''): PageInput => ({ + type: 'concept', title, compiled_truth: body, timeline: '', + }); + + test('any-dir wikilink miss appears in the final JSON result', async () => { + await engine.putPage('concepts/note', conceptPage('Note', 'See [[janus/never-existed]].')); + + const logged: string[] = []; + const origLog = console.log; + console.log = (m: unknown) => { logged.push(String(m)); }; + try { + await runExtract(engine, ['links', '--source', 'db', '--json']); + } finally { + console.log = origLog; + } + + const result = logged + .filter(l => l.trim().startsWith('{')) + .map(l => JSON.parse(l)) + .find(o => 'links_created' in o); + expect(result).toBeDefined(); + expect(result.unresolved_refs).toContainEqual({ field: 'wikilink', name: 'janus/never-existed' }); + expect((await engine.getLinks('concepts/note')).length).toBe(0); + }); + + test('clean run has no unresolved_refs key (shape back-compat)', async () => { + await engine.putPage('janus/real', conceptPage('Real', 'target')); + await engine.putPage('concepts/note', conceptPage('Note', 'See [[janus/real]].')); + + const logged: string[] = []; + const origLog = console.log; + console.log = (m: unknown) => { logged.push(String(m)); }; + try { + await runExtract(engine, ['links', '--source', 'db', '--json']); + } finally { + console.log = origLog; + } + + const result = logged + .filter(l => l.trim().startsWith('{')) + .map(l => JSON.parse(l)) + .find(o => 'links_created' in o); + expect(result).toBeDefined(); + expect(result.unresolved_refs).toBeUndefined(); + // The any-dir link itself landed. + expect((await engine.getLinks('concepts/note')).some((l: any) => l.to_slug === 'janus/real')).toBe(true); + }); +}); diff --git a/test/extract-source-aware.test.ts b/test/extract-source-aware.test.ts index ee79fb035..9b18f9cdf 100644 --- a/test/extract-source-aware.test.ts +++ b/test/extract-source-aware.test.ts @@ -123,3 +123,94 @@ describe('extract --source-id flag (#1204)', () => { expect(Number(linkRows[0]?.n ?? 0)).toBe(0); }); }); + +// ─── Issue #1493 (codex P1): qualified pin + scoped default fallback ── + +describe('extract --source db: any-dir wikilinks across sources (#1493)', () => { + beforeEach(async () => { + await truncateAll(); + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ('alpha', 'alpha'), ('beta', 'beta') + ON CONFLICT (id) DO NOTHING`, + ); + }); + + async function runLinksDb(extra: string[] = []): Promise { + const origLog = console.log; + console.log = () => {}; + try { + await runExtract(engine, ['links', '--source', 'db', ...extra, '--json']); + } finally { + console.log = origLog; + } + } + + test('qualified [[beta:janus/target]] links to the PINNED source', async () => { + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES + ('concepts/note', 'alpha', 'concept', 'Note', 'See [[beta:janus/target]].', ''), + ('janus/target', 'beta', 'concept', 'Target', 'The target.', '')`, + ); + await runLinksDb(); + const rows = await engine.executeRaw<{ to_slug: string; to_source: string }>( + `SELECT t.slug AS to_slug, t.source_id AS to_source + FROM links l JOIN pages t ON l.to_page_id = t.id`, + ); + expect(rows.length).toBe(1); + expect(rows[0].to_slug).toBe('janus/target'); + expect(rows[0].to_source).toBe('beta'); + }); + + test('pin miss is NOT misdirected to a same-slug page in origin/default', async () => { + // janus/dup exists in alpha AND default — but the wikilink pins beta, + // where it does not exist. No edge at all (skip beats misdirect). + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES + ('concepts/note', 'alpha', 'concept', 'Note', 'See [[beta:janus/dup]].', ''), + ('janus/dup', 'alpha', 'concept', 'Dup A', 'x', ''), + ('janus/dup', 'default', 'concept', 'Dup D', 'x', '')`, + ); + await runLinksDb(); + const rows = await engine.executeRaw<{ n: string }>(`SELECT COUNT(*)::text AS n FROM links`); + expect(Number(rows[0]?.n ?? 0)).toBe(0); + }); + + test('scoped --source-id run keeps the default-source fallback for unqualified any-dir links', async () => { + // The target lives ONLY in the default source; the scoped slugExists + // snapshot must include the default overlay so the candidate survives + // to resolveCandidateSources' documented fallback (parity with + // whitelisted refs, which bypass slugExists). + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES + ('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/shared-doc]].', ''), + ('janus/shared-doc', 'default', 'concept', 'Shared', 'The doc.', '')`, + ); + await runLinksDb(['--source-id', 'alpha']); + const rows = await engine.executeRaw<{ to_slug: string; to_source: string; from_source: string }>( + `SELECT t.slug AS to_slug, t.source_id AS to_source, f.source_id AS from_source + FROM links l JOIN pages t ON l.to_page_id = t.id JOIN pages f ON l.from_page_id = f.id`, + ); + expect(rows.length).toBe(1); + expect(rows[0].from_source).toBe('alpha'); + expect(rows[0].to_slug).toBe('janus/shared-doc'); + expect(rows[0].to_source).toBe('default'); + }); + + test('scoped run still refuses cross-source (non-default) any-dir resolution', async () => { + // Target exists only in beta. Scoped alpha run: slugExists domain is + // alpha ∪ default → miss → recorded unresolved, no edge (a bare + // unqualified link must not silently span unrelated sources). + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES + ('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/foreign]].', ''), + ('janus/foreign', 'beta', 'concept', 'Foreign', 'x', '')`, + ); + await runLinksDb(['--source-id', 'alpha']); + const rows = await engine.executeRaw<{ n: string }>(`SELECT COUNT(*)::text AS n FROM links`); + expect(Number(rows[0]?.n ?? 0)).toBe(0); + }); +}); diff --git a/test/extract-stale.test.ts b/test/extract-stale.test.ts index a725e309b..fa1d5f84a 100644 --- a/test/extract-stale.test.ts +++ b/test/extract-stale.test.ts @@ -188,7 +188,8 @@ describe('gbrain extract --stale', () => { await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) advises [Acme](companies/acme).')); // Microsecond-precision updated_at, recent (after LINK_EXTRACTOR_VERSION_TS) so the // version arm doesn't fire — the edited arm is what must clear. - await engine.executeRaw(`UPDATE pages SET updated_at = '2026-06-02 08:18:58.999166+00'`); + // (#1493: fixture date advanced past the bumped watermark.) + await engine.executeRaw(`UPDATE pages SET updated_at = '2026-08-02 08:18:58.999166+00'`); expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(2); await runExtract(engine, ['--stale']); @@ -290,3 +291,135 @@ describe('gbrain extract --stale', () => { expect(msg).toContain('DB-source only'); }); }); + +// ─── Issue #1493 (codex P2): any-dir links + unresolved surfacing ──── + +describe('extract --stale: any-dir exact-path wikilinks (#1493)', () => { + test('creates the edge for an existing any-dir target', async () => { + await engine.putPage('janus/real', { type: 'concept', title: 'Real', compiled_truth: 'The target.', timeline: '' }); + await engine.putPage('concepts/note', { type: 'concept', title: 'Note', compiled_truth: 'See [[janus/real]].', timeline: '' }); + + await runExtract(engine, ['--stale']); + + const links = await engine.getLinks('concepts/note'); + expect(links.some(l => l.to_slug === 'janus/real')).toBe(true); + }); + + test('misses are surfaced in the --json sweep summary, not silently dropped', async () => { + await engine.putPage('concepts/note', { + type: 'concept', title: 'Note', + compiled_truth: 'See [[janus/never-existed]] and [[janus/also-gone]].', timeline: '', + }); + + const lines: string[] = []; + const originalWrite = process.stdout.write.bind(process.stdout); + process.stdout.write = ((chunk: string | Uint8Array): boolean => { + lines.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8')); + return true; + }) as any; + try { + await runExtract(engine, ['--stale', '--json']); + } finally { + process.stdout.write = originalWrite; + } + + const done = lines + .filter(l => l.trim().startsWith('{')) + .map(l => JSON.parse(l.trim())) + .find(o => o.action === 'extract_stale_done'); + expect(done).toBeDefined(); + expect(done.unresolved_count).toBe(2); + expect(done.unresolved_refs).toContainEqual({ field: 'wikilink', name: 'janus/never-existed' }); + expect(done.unresolved_refs).toContainEqual({ field: 'wikilink', name: 'janus/also-gone' }); + // And no ghost edges were written. + expect((await engine.getLinks('concepts/note')).length).toBe(0); + }); + + test('clean sweep reports unresolved_count 0 with no unresolved_refs key', async () => { + await engine.putPage('people/alice', personPage('Alice')); + + const lines: string[] = []; + const originalWrite = process.stdout.write.bind(process.stdout); + process.stdout.write = ((chunk: string | Uint8Array): boolean => { + lines.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8')); + return true; + }) as any; + try { + await runExtract(engine, ['--stale', '--json']); + } finally { + process.stdout.write = originalWrite; + } + + const done = lines + .filter(l => l.trim().startsWith('{')) + .map(l => JSON.parse(l.trim())) + .find(o => o.action === 'extract_stale_done'); + expect(done).toBeDefined(); + expect(done.unresolved_count).toBe(0); + expect(done.unresolved_refs).toBeUndefined(); + }); +}); + +// ─── Issue #1493 (codex P2, round 2): per-page-source resolution domain ── + +describe('extract --stale: per-source resolution domain (#1493 round 2)', () => { + test('alpha page linking a beta-only target records the miss (no silent drop, no cross-source edge)', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ('alpha', 'alpha'), ('beta', 'beta') + ON CONFLICT (id) DO NOTHING`, + ); + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES + ('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/foreign]].', ''), + ('janus/foreign', 'beta', 'concept', 'Foreign', 'x', '')`, + ); + + const lines: string[] = []; + const originalWrite = process.stdout.write.bind(process.stdout); + process.stdout.write = ((chunk: string | Uint8Array): boolean => { + lines.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8')); + return true; + }) as any; + try { + await runExtract(engine, ['--stale', '--json']); + } finally { + process.stdout.write = originalWrite; + } + + const done = lines + .filter(l => l.trim().startsWith('{')) + .map(l => JSON.parse(l.trim())) + .find(o => o.action === 'extract_stale_done'); + expect(done).toBeDefined(); + // The alpha page's resolution domain is alpha ∪ default — the beta-only + // target is a MISS and must be recorded, not silently dropped after + // passing an unscoped existence check. + expect(done.unresolved_refs ?? []).toContainEqual({ field: 'wikilink', name: 'janus/foreign' }); + const linkRows = await engine.executeRaw<{ n: string }>(`SELECT COUNT(*)::text AS n FROM links`); + expect(Number(linkRows[0]?.n ?? 0)).toBe(0); + }); + + test('alpha page linking a default-source target resolves via the default overlay', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ('alpha', 'alpha') + ON CONFLICT (id) DO NOTHING`, + ); + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES + ('concepts/note', 'alpha', 'concept', 'Note', 'See [[janus/shared]].', ''), + ('janus/shared', 'default', 'concept', 'Shared', 'x', '')`, + ); + + await runExtract(engine, ['--stale']); + + const rows = await engine.executeRaw<{ to_slug: string; to_source: string }>( + `SELECT t.slug AS to_slug, t.source_id AS to_source + FROM links l JOIN pages t ON l.to_page_id = t.id`, + ); + expect(rows.length).toBe(1); + expect(rows[0].to_slug).toBe('janus/shared'); + expect(rows[0].to_source).toBe('default'); + }); +}); diff --git a/test/link-extraction-any-dir.test.ts b/test/link-extraction-any-dir.test.ts new file mode 100644 index 000000000..70b4b0e10 --- /dev/null +++ b/test/link-extraction-any-dir.test.ts @@ -0,0 +1,607 @@ +import { describe, test, expect } from 'bun:test'; +import { + extractEntityRefs, + extractPageLinks, + isAnyDirExactPathEnabled, + makeResolver, + type LinkCandidate, + type SlugResolver, +} from '../src/core/link-extraction.ts'; +import { resolveCandidateSources } from '../src/commands/extract.ts'; +import { parseMarkdown } from '../src/core/markdown.ts'; +import type { BrainEngine } from '../src/core/engine.ts'; + +// ─── Issue #1493: generalized path resolution ───────────────────── +// +// Path-shaped wikilinks ([[dir/name]], [[dir/sub/name]]) whose top-level +// directory is NOT in the DIR_PATTERN whitelist used to be dropped +// silently — no candidate, no unresolved report. They now extract via the +// 2a'/2b' passes (tagged `exactPath: true`) and resolve by EXACT slug +// match against existing pages. On by default +// (link_resolution.any_dir_exact_path); misses are recorded in the +// unresolved report as field='wikilink'. + +// ─── extractEntityRefs — passes 2a'/2b' ─────────────────────────── + +describe('extractEntityRefs — any-dir path-shaped wikilinks (#1493)', () => { + test('2-segment path outside the whitelist extracts with exactPath flag', () => { + const refs = extractEntityRefs('See [[janus/foo]] for context.'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('janus/foo'); + expect(refs[0].dir).toBe('janus'); + expect(refs[0].exactPath).toBe(true); + // NOT the #972 basename pass — that one carries needsResolution. + expect(refs[0].needsResolution).toBeUndefined(); + }); + + test('deep nesting: [[janus/agents/drift-check]]', () => { + const refs = extractEntityRefs('Owned by [[janus/agents/drift-check]].'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('janus/agents/drift-check'); + expect(refs[0].dir).toBe('janus'); + expect(refs[0].exactPath).toBe(true); + }); + + test('hyphenated directory: [[janus-ui/docs/tier2-agent-plan]]', () => { + const refs = extractEntityRefs('Plan: [[janus-ui/docs/tier2-agent-plan]].'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('janus-ui/docs/tier2-agent-plan'); + expect(refs[0].dir).toBe('janus-ui'); + }); + + test('alias form: [[janus/foo|Display]] keeps target, uses display name', () => { + const refs = extractEntityRefs('See [[janus/foo|Display]].'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('janus/foo'); + expect(refs[0].name).toBe('Display'); + }); + + test('fragment stripped: [[janus/foo#section]]', () => { + const refs = extractEntityRefs('Jump to [[janus/foo#section]].'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('janus/foo'); + }); + + test('.md suffix stripped: [[janus/foo.md]]', () => { + const refs = extractEntityRefs('See [[janus/foo.md]].'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('janus/foo'); + }); + + test('dotted segments survive when not a .md suffix: [[notes/v1.0.0]]', () => { + const refs = extractEntityRefs('Released [[notes/v1.0.0]].'); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('notes/v1.0.0'); + }); + + test('qualified variant: [[src:janus/foo]] carries sourceId + exactPath', () => { + const refs = extractEntityRefs('See [[src:janus/foo]].'); + expect(refs.length).toBe(1); + expect(refs[0]).toMatchObject({ + slug: 'janus/foo', dir: 'janus', sourceId: 'src', exactPath: true, + }); + }); + + test('qualified whitelist links still go through the 2a pass unchanged', () => { + // [[wiki:concepts/ai]] is 2a territory (whitelisted dir) — must not be + // double-emitted by the new 2a' pass. + const refs = extractEntityRefs('Ref: [[wiki:concepts/ai]]'); + expect(refs.length).toBe(1); + expect(refs[0].sourceId).toBe('wiki'); + expect(refs[0].exactPath).toBeUndefined(); + }); + + test('wikilinks inside fenced and inline code are ignored', () => { + const refs = extractEntityRefs( + '```\n[[janus/in-fence]]\n```\nInline `[[janus/in-tick]]` too.\nReal: [[janus/real]].', + ); + expect(refs.length).toBe(1); + expect(refs[0].slug).toBe('janus/real'); + }); + + test('URLs are not matched', () => { + const refs = extractEntityRefs('See [[https://example.com/foo]] and https://example.com/bar.'); + expect(refs.filter(r => r.exactPath)).toEqual([]); + }); + + test('uppercase target is NOT path-shaped (slug grammar is lowercase) — falls to the generic pass', () => { + // Engine slugs are lowercase (sync.ts slugifySegment). [[Janus/Foo]] is + // not a canonical slug, so it keeps the pre-#1493 behavior: generic + // pass 2c (needsResolution — basename resolution or silent drop). + const refs = extractEntityRefs('See [[Janus/Foo]].'); + expect(refs.length).toBe(1); + expect(refs[0].exactPath).toBeUndefined(); + expect(refs[0].needsResolution).toBe(true); + expect(refs[0].slug).toBe('Janus/Foo'); + }); + + test('whitelisted dirs keep their existing pass (no exactPath tag)', () => { + const refs = extractEntityRefs('See [[people/alice]] and [[janus/foo]].'); + const alice = refs.find(r => r.slug === 'people/alice'); + const janus = refs.find(r => r.slug === 'janus/foo'); + expect(alice).toBeDefined(); + expect(alice!.exactPath).toBeUndefined(); + expect(janus).toBeDefined(); + expect(janus!.exactPath).toBe(true); + expect(refs.length).toBe(2); // no double emission + }); + + test('bare-name #972 pass is unaffected', () => { + const refs = extractEntityRefs('See [[struktura]] and [[janus/foo]].'); + const bare = refs.find(r => r.slug === 'struktura'); + expect(bare).toBeDefined(); + expect(bare!.needsResolution).toBe(true); + expect(bare!.exactPath).toBeUndefined(); + }); + + test('a path wikilink inside a markdown-link label is inert (mirrors codex P2a)', () => { + const refs = extractEntityRefs('[see [[janus/foo]]](companies/acme.md)'); + expect(refs.filter(r => r.exactPath)).toEqual([]); + }); +}); + +// ─── extractPageLinks — exact-slug resolution + unresolved report ── + +/** Resolver fixture backed by an explicit slug set (mirrors makeResolver's slugExists). */ +function makeExistsResolver(slugs: string[]): SlugResolver { + const set = new Set(slugs); + return { + resolve: async () => null, + slugExists: async (slug: string) => set.has(slug), + }; +} + +describe('extractPageLinks — any-dir exact-path resolution (#1493)', () => { + test('existing path target emits a candidate like a whitelisted wikilink', async () => { + const resolver = makeExistsResolver(['janus/agents/drift-check']); + const { candidates, unresolved } = await extractPageLinks( + 'concepts/x', 'Owned by [[janus/agents/drift-check]].', + {}, 'concept', resolver, + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('janus/agents/drift-check'); + // Same treatment as whitelisted refs: verb-inferred type + markdown + // provenance (NOT the #972 wikilink_basename/wikilink-resolved pair). + expect(candidates[0].linkType).toBe('mentions'); + expect(candidates[0].linkSource).toBe('markdown'); + expect(unresolved).toEqual([]); + }); + + test('missing path target is RECORDED as unresolved, never a ghost edge', async () => { + const resolver = makeExistsResolver([]); + const { candidates, unresolved } = await extractPageLinks( + 'concepts/x', 'See [[janus/never-existed]].', + {}, 'concept', resolver, + ); + expect(candidates).toEqual([]); + expect(unresolved).toEqual([{ field: 'wikilink', name: 'janus/never-existed' }]); + }); + + test('repeated misses of the same target dedup to one unresolved entry', async () => { + const resolver = makeExistsResolver([]); + const { unresolved } = await extractPageLinks( + 'concepts/x', 'See [[janus/gone]] and again [[janus/gone]].', + {}, 'concept', resolver, + ); + expect(unresolved).toEqual([{ field: 'wikilink', name: 'janus/gone' }]); + }); + + test('opts.anyDirExactPath false restores the legacy silent drop', async () => { + const resolver = makeExistsResolver(['janus/foo']); + const { candidates, unresolved } = await extractPageLinks( + 'concepts/x', 'See [[janus/foo]].', + {}, 'concept', resolver, { anyDirExactPath: false }, + ); + expect(candidates).toEqual([]); + expect(unresolved).toEqual([]); + }); + + test('resolver without slugExists emits unverified (caller dead-link filter applies)', async () => { + // extract --stale runs a nullResolver with no slugExists — candidates + // must still come out so resolveCandidateSources can endpoint-check + // them, the same protection whitelisted refs rely on. + const resolver: SlugResolver = { resolve: async () => null }; + const { candidates, unresolved } = await extractPageLinks( + 'concepts/x', 'See [[janus/foo]].', + {}, 'concept', resolver, + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('janus/foo'); + expect(unresolved).toEqual([]); + }); + + test('qualified path ref skips slugExists (cross-source target) and emits', async () => { + // [[src:janus/foo]] pins a target that may live in a DIFFERENT source; + // the resolver's snapshot is scoped to this source, so existence is the + // caller's cross-source endpoint check — even a resolver that says + // "does not exist here" must not block it. + const resolver = makeExistsResolver([]); + const { candidates, unresolved } = await extractPageLinks( + 'concepts/x', 'See [[src:janus/foo]].', + {}, 'concept', resolver, + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('janus/foo'); + expect(unresolved).toEqual([]); + }); + + test('alias + fragment + .md forms all resolve to the same page', async () => { + const resolver = makeExistsResolver(['janus/foo']); + const { candidates } = await extractPageLinks( + 'concepts/x', + 'A [[janus/foo|Display]] B [[janus/foo#section]] C [[janus/foo.md]].', + {}, 'concept', resolver, + ); + // Within-page dedup collapses the three mentions to one candidate. + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('janus/foo'); + }); + + test('whitelisted dirs behave exactly as before (no existence gate at this layer)', async () => { + // Whitelisted refs are emitted unverified — callers filter. The #1493 + // pass must not change that, even with a resolver that would say no. + const resolver = makeExistsResolver([]); + const { candidates, unresolved } = await extractPageLinks( + 'concepts/x', 'See [[people/alice]].', + {}, 'concept', resolver, + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('people/alice'); + expect(unresolved).toEqual([]); + }); + + test('#972 bare-name pass composes: basename resolution still works alongside', async () => { + const resolver: SlugResolver = { + resolve: async () => null, + slugExists: async (slug) => slug === 'janus/foo', + resolveBasenameMatches: async (name) => + name === 'struktura' ? ['projects/struktura'] : [], + }; + const { candidates } = await extractPageLinks( + 'concepts/x', 'See [[janus/foo]] and [[struktura]].', + {}, 'concept', resolver, { globalBasename: true }, + ); + const exact = candidates.find(c => c.targetSlug === 'janus/foo'); + const basename = candidates.find(c => c.targetSlug === 'projects/struktura'); + expect(exact).toBeDefined(); + expect(exact!.linkSource).toBe('markdown'); + expect(basename).toBeDefined(); + expect(basename!.linkType).toBe('wikilink_basename'); + expect(basename!.linkSource).toBe('wikilink-resolved'); + }); + + test('uppercase path target goes to the generic pass, not exact-path (documented behavior)', async () => { + // [[Janus/Foo]] is not a canonical slug. With globalBasename off + // (default) it drops silently exactly as before #1493 — it is NOT + // recorded as a wikilink miss because it never entered the exact-path + // pass. (The #2868 line of work covers slugify-before-match.) + const resolver = makeExistsResolver(['janus/foo']); + const { candidates, unresolved } = await extractPageLinks( + 'concepts/x', 'See [[Janus/Foo]].', + {}, 'concept', resolver, + ); + expect(candidates).toEqual([]); + expect(unresolved).toEqual([]); + }); + + test('frontmatter is parsed out before extraction ever sees it (put_page layering)', async () => { + // put_page extracts over compiled_truth + timeline, which parseMarkdown + // produces AFTER gray-matter removes the frontmatter block. A wikilink + // inside frontmatter therefore never reaches the extractor as body text. + const raw = [ + '---', + 'title: X', + 'related: "[[janus/from-frontmatter]]"', + '---', + '', + 'Body links [[janus/from-body]].', + ].join('\n'); + const parsed = parseMarkdown(raw, 'concepts/x.md'); + expect(parsed.compiled_truth).not.toContain('janus/from-frontmatter'); + const resolver = makeExistsResolver(['janus/from-body', 'janus/from-frontmatter']); + const { candidates } = await extractPageLinks( + 'concepts/x', parsed.compiled_truth + '\n' + parsed.timeline, + parsed.frontmatter, 'concept', resolver, { skipFrontmatter: true }, + ); + expect(candidates.map(c => c.targetSlug)).toEqual(['janus/from-body']); + }); +}); + +// ─── makeResolver.slugExists ────────────────────────────────────── + +describe('makeResolver — slugExists (#1493)', () => { + function makeFakeEngineWithSlugs(slugs: string[]): BrainEngine { + let getAllCalls = 0; + const engine = { + async getPage() { return null; }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + async getAllSlugs() { + getAllCalls++; + return new Set(slugs); + }, + } as unknown as BrainEngine; + (engine as any)._counts = () => ({ getAllCalls }); + return engine; + } + + test('exact match hits, near-miss does not', async () => { + const r = makeResolver(makeFakeEngineWithSlugs(['janus/agents/drift-check'])); + expect(await r.slugExists!('janus/agents/drift-check')).toBe(true); + expect(await r.slugExists!('janus/agents/drift')).toBe(false); + expect(await r.slugExists!('drift-check')).toBe(false); + expect(await r.slugExists!('')).toBe(false); + }); + + test('shares one getAllSlugs snapshot with resolveBasenameMatches', async () => { + const engine = makeFakeEngineWithSlugs(['janus/foo', 'projects/struktura']); + const r = makeResolver(engine); + await r.slugExists!('janus/foo'); + await r.resolveBasenameMatches!('struktura'); + await r.slugExists!('janus/foo'); + expect((engine as any)._counts().getAllCalls).toBe(1); + }); + + test('scopes the snapshot by sourceId (no cross-source exact match)', async () => { + const bySource: Record = { + 'src-a': ['janus/foo'], + 'src-b': ['janus/bar'], + }; + const engine = { + async getPage() { return null; }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + async getAllSlugs(opts?: { sourceId?: string }) { + const sid = opts?.sourceId; + return new Set(sid ? (bySource[sid] ?? []) : Object.values(bySource).flat()); + }, + } as unknown as BrainEngine; + const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' }); + expect(await r.slugExists!('janus/foo')).toBe(true); + expect(await r.slugExists!('janus/bar')).toBe(false); + }); + + test('degrades to false when getAllSlugs is missing', async () => { + const engine = { + async getPage() { return null; }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + } as unknown as BrainEngine; + const r = makeResolver(engine); + expect(await r.slugExists!('janus/foo')).toBe(false); + }); +}); + +// ─── isAnyDirExactPathEnabled ───────────────────────────────────── + +describe('isAnyDirExactPathEnabled', () => { + function makeFakeEngine(configMap: Map): BrainEngine { + return { + getConfig: async (key: string) => configMap.get(key) ?? null, + } as unknown as BrainEngine; + } + + test('null/undefined -> true (default ON — exact match cannot false-positive)', async () => { + expect(await isAnyDirExactPathEnabled(makeFakeEngine(new Map()))).toBe(true); + }); + + test('"false"/"0"/"no"/"off" -> false', async () => { + for (const v of ['false', '0', 'no', 'off', ' False ']) { + const engine = makeFakeEngine(new Map([['link_resolution.any_dir_exact_path', v]])); + expect(await isAnyDirExactPathEnabled(engine)).toBe(false); + } + }); + + test('"true" and garbage -> true (fail-safe to default, like auto_link)', async () => { + for (const v of ['true', '1', 'garbage']) { + const engine = makeFakeEngine(new Map([['link_resolution.any_dir_exact_path', v]])); + expect(await isAnyDirExactPathEnabled(engine)).toBe(true); + } + }); + + test('env var override wins over DB config', async () => { + const engine = makeFakeEngine(new Map([['link_resolution.any_dir_exact_path', 'true']])); + process.env.GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH = 'off'; + try { + expect(await isAnyDirExactPathEnabled(engine)).toBe(false); + } finally { + delete process.env.GBRAIN_LINK_RESOLUTION_ANY_DIR_EXACT_PATH; + } + }); +}); + +// ─── Issue #1493 codex P1: qualified source pin threading ────────── + +describe('extractPageLinks — qualified wikilink source pin (codex P1)', () => { + test('qualified any-dir ref carries targetSourceId on the candidate', async () => { + const resolver = makeExistsResolver([]); + const { candidates } = await extractPageLinks( + 'concepts/x', 'See [[beta:janus/foo]].', + {}, 'concept', resolver, + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('janus/foo'); + expect(candidates[0].targetSourceId).toBe('beta'); + }); + + test('qualified WHITELIST ref carries targetSourceId identically', async () => { + const resolver = makeExistsResolver([]); + const { candidates } = await extractPageLinks( + 'concepts/x', 'See [[beta:people/alice]].', + {}, 'concept', resolver, + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('people/alice'); + expect(candidates[0].targetSourceId).toBe('beta'); + }); + + test('unqualified refs carry NO targetSourceId', async () => { + const resolver = makeExistsResolver(['janus/foo']); + const { candidates } = await extractPageLinks( + 'concepts/x', 'See [[janus/foo]] and [[people/alice]].', + {}, 'concept', resolver, + ); + for (const c of candidates) { + expect(c.targetSourceId).toBeUndefined(); + } + }); + + test('pinned + unpinned refs to the same target do NOT dedup-collapse', async () => { + // [[beta:janus/foo]] and [[janus/foo]] resolve to different + // (to_slug, to_source_id) rows — both candidates must survive. + const resolver = makeExistsResolver(['janus/foo']); + const { candidates } = await extractPageLinks( + 'concepts/x', 'Pinned [[beta:janus/foo]] and local [[janus/foo]].', + {}, 'concept', resolver, + ); + const toFoo = candidates.filter(c => c.targetSlug === 'janus/foo'); + expect(toFoo.length).toBe(2); + expect(new Set(toFoo.map(c => c.targetSourceId))).toEqual(new Set(['beta', undefined])); + }); +}); + +describe('resolveCandidateSources — honors the qualified source pin (codex P1)', () => { + const mk = (over: Partial = {}): LinkCandidate => ({ + targetSlug: 'janus/foo', linkType: 'mentions', context: 'ctx', linkSource: 'markdown', ...over, + }); + + test('pinned candidate resolves to the pinned source', () => { + const r = resolveCandidateSources( + mk({ targetSourceId: 'beta' }), 'concepts/x', 'alpha', + new Set(['concepts/x', 'janus/foo']), + new Map([['concepts/x', ['alpha']], ['janus/foo', ['beta']]]), + ); + expect(r).not.toBeNull(); + expect(r!.toSourceId).toBe('beta'); + expect(r!.fromSourceId).toBe('alpha'); + }); + + test('pin miss returns null even when origin/default HAS a same-slug page (no misdirection)', () => { + const r = resolveCandidateSources( + mk({ targetSourceId: 'beta' }), 'concepts/x', 'alpha', + new Set(['concepts/x', 'janus/foo']), + // Target exists in origin source AND default — but NOT in beta. + new Map([['concepts/x', ['alpha']], ['janus/foo', ['alpha', 'default']]]), + ); + expect(r).toBeNull(); + }); + + test('unpinned candidate keeps the local-first → default fallback', () => { + const local = resolveCandidateSources( + mk(), 'concepts/x', 'alpha', + new Set(['concepts/x', 'janus/foo']), + new Map([['concepts/x', ['alpha']], ['janus/foo', ['alpha']]]), + ); + expect(local!.toSourceId).toBe('alpha'); + const viaDefault = resolveCandidateSources( + mk(), 'concepts/x', 'alpha', + new Set(['concepts/x', 'janus/foo']), + new Map([['concepts/x', ['alpha']], ['janus/foo', ['default']]]), + ); + expect(viaDefault!.toSourceId).toBe('default'); + }); +}); + +// ─── Issue #1493 codex P1: scoped slugExists includes default fallback ─ + +describe('makeResolver — slugExists default-source fallback domain (codex P1)', () => { + function makeMultiSourceEngine(bySource: Record): BrainEngine { + const calls: Array = []; + const engine = { + async getPage() { return null; }, + async findByTitleFuzzy() { return null; }, + async searchKeyword() { return []; }, + async getAllSlugs(opts?: { sourceId?: string }) { + calls.push(opts?.sourceId); + const sid = opts?.sourceId; + return new Set(sid ? (bySource[sid] ?? []) : Object.values(bySource).flat()); + }, + } as unknown as BrainEngine; + (engine as any)._calls = () => calls; + return engine; + } + + test('scoped resolver sees default-source slugs (mirrors resolveCandidateSources fallback)', async () => { + const engine = makeMultiSourceEngine({ + 'src-a': ['janus/local'], + 'default': ['janus/shared-doc'], + 'src-b': ['janus/foreign'], + }); + const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' }); + expect(await r.slugExists!('janus/local')).toBe(true); // scoped source + expect(await r.slugExists!('janus/shared-doc')).toBe(true); // default fallback + expect(await r.slugExists!('janus/foreign')).toBe(false); // other source: NOT in domain + }); + + test('basename index stays source-scoped (#972 P1 preserved — no default overlay)', async () => { + const engine = makeMultiSourceEngine({ + 'src-a': ['janus/local'], + 'default': ['projects/shared-doc'], + }); + const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' }); + // Basename lookup must NOT see the default source's pages. + expect(await r.resolveBasenameMatches!('shared-doc')).toEqual([]); + expect(await r.resolveBasenameMatches!('local')).toEqual(['janus/local']); + }); + + test('unscoped and default-scoped resolvers do not double-fetch', async () => { + const unscoped = makeMultiSourceEngine({ 'default': ['janus/foo'] }); + const r1 = makeResolver(unscoped, { mode: 'batch' }); + await r1.slugExists!('janus/foo'); + await r1.slugExists!('janus/foo'); + expect((unscoped as any)._calls()).toEqual([undefined]); + + const defScoped = makeMultiSourceEngine({ 'default': ['janus/foo'] }); + const r2 = makeResolver(defScoped, { mode: 'batch', sourceId: 'default' }); + await r2.slugExists!('janus/foo'); + expect((defScoped as any)._calls()).toEqual(['default']); + }); + + test('scoped resolver fetches exactly scoped + default snapshots', async () => { + const engine = makeMultiSourceEngine({ 'src-a': [], 'default': ['janus/foo'] }); + const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' }); + await r.slugExists!('janus/foo'); + await r.slugExists!('janus/foo'); + await r.resolveBasenameMatches!('foo'); + expect((engine as any)._calls().sort()).toEqual(['default', 'src-a']); + }); +}); + +// ─── Issue #1493 codex round 2: bare-slug pass vs qualified spans ──── + +describe('extractPageLinks — bare-slug pass masks qualified spans only (codex P2 round 2)', () => { + test('legit prose `Owner:people/alice` still emits a bare-slug candidate', async () => { + // Regression: round 1 skipped ANY `:`-preceded bare slug, which also + // dropped legitimate prose refs. Only actual qualified-wikilink spans + // are masked now. + const { candidates } = await extractPageLinks( + 'concepts/x', 'Owner:people/alice runs this.', + {}, 'concept', makeExistsResolver([]), + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSlug).toBe('people/alice'); + expect(candidates[0].targetSourceId).toBeUndefined(); + }); + + test('qualified wikilink still emits exactly ONE (pinned) candidate — no bare-slug duplicate', async () => { + const { candidates } = await extractPageLinks( + 'concepts/x', 'See [[wiki:people/alice]].', + {}, 'concept', makeExistsResolver([]), + ); + expect(candidates.length).toBe(1); + expect(candidates[0].targetSourceId).toBe('wiki'); + }); + + test('prose colon ref and qualified wikilink coexist correctly', async () => { + const { candidates } = await extractPageLinks( + 'concepts/x', 'Owner:people/bob maintains it. See [[wiki:people/alice]].', + {}, 'concept', makeExistsResolver([]), + ); + const bob = candidates.filter(c => c.targetSlug === 'people/bob'); + const alice = candidates.filter(c => c.targetSlug === 'people/alice'); + expect(bob.length).toBe(1); + expect(bob[0].targetSourceId).toBeUndefined(); + expect(alice.length).toBe(1); + expect(alice[0].targetSourceId).toBe('wiki'); + }); +}); diff --git a/test/put-page-autolink-pin-dedup.test.ts b/test/put-page-autolink-pin-dedup.test.ts new file mode 100644 index 000000000..87393bfd8 --- /dev/null +++ b/test/put-page-autolink-pin-dedup.test.ts @@ -0,0 +1,100 @@ +/** + * Issue #1493 (codex P2, round 2) — put_page auto-link must not double-count + * (or double-insert) when a page contains BOTH the pinned and unpinned form + * of the same wikilink, e.g. `[[default:people/alice]]` and + * `[[people/alice]]` on a default-source page. The pin is part of + * extractPageLinks' within-page dedup key (correct for the extract paths, + * where the two can resolve to different to_source_id rows), but inside + * put_page's single-source write scope both resolve to the SAME edge row — + * runAutoLink re-dedupes by the resolved edge key before writing/counting. + * + * Hermetic PGLite; embed transport stubbed (same pattern as + * test/put-page-provenance.test.ts). + */ + +import { describe, test, expect, beforeAll, beforeEach, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { operations } from '../src/core/operations.ts'; +import type { OperationContext } from '../src/core/operations.ts'; +import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts'; + +const putPageOp = operations.find((o) => o.name === 'put_page')!; + +let engine: PGLiteEngine; + +beforeAll(async () => { + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { ...process.env, OPENAI_API_KEY: process.env.OPENAI_API_KEY || 'sk-test-stub' }, + }); + __setEmbedTransportForTests(async ({ values }: any) => ({ + embeddings: values.map(() => new Array(1536).fill(0)), + usage: { tokens: 0 }, + }) as any); + + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); + __setEmbedTransportForTests(null); + resetGateway(); +}); + +beforeEach(async () => { + for (const t of ['content_chunks', 'links', 'tags', 'timeline_entries', 'page_versions', 'pages']) { + await engine.executeRaw(`DELETE FROM ${t}`, []); + } +}); + +function makeCtx(): OperationContext { + return { + engine, + config: { engine: 'pglite' as const }, + logger: { info: () => {}, warn: () => {}, error: () => {} }, + dryRun: false, + remote: false, + sourceId: 'default', + }; +} + +describe('put_page auto-link — pinned + unpinned same-target dedup (#1493)', () => { + test('own-source pin + unpinned form create ONE edge and count created=1', async () => { + const ctx = makeCtx(); + await putPageOp.handler(ctx, { + slug: 'people/alice', + content: '---\ntype: person\ntitle: Alice\n---\n\nAlice.', + }); + + const res = await putPageOp.handler(ctx, { + slug: 'concepts/note', + content: '---\ntype: concept\ntitle: Note\n---\n\nSee [[default:people/alice]] and [[people/alice]].', + }) as { auto_links?: { created: number; errors: number } }; + + const links = (await engine.getLinks('concepts/note')).filter(l => l.to_slug === 'people/alice'); + expect(links.length).toBe(1); + expect(res.auto_links).toBeDefined(); + expect(res.auto_links!.created).toBe(1); + expect(res.auto_links!.errors).toBe(0); + }); + + test('foreign-pinned form is skipped, unpinned form still links (no misdirection)', async () => { + const ctx = makeCtx(); + await putPageOp.handler(ctx, { + slug: 'people/alice', + content: '---\ntype: person\ntitle: Alice\n---\n\nAlice.', + }); + + const res = await putPageOp.handler(ctx, { + slug: 'concepts/note', + content: '---\ntype: concept\ntitle: Note\n---\n\nSee [[somewhere-else:people/alice]] and [[people/alice]].', + }) as { auto_links?: { created: number } }; + + const links = (await engine.getLinks('concepts/note')).filter(l => l.to_slug === 'people/alice'); + expect(links.length).toBe(1); + expect(res.auto_links!.created).toBe(1); + }); +});