mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.41.10.0 feat: orphan reduction via --by-mention + UTF-16 surrogate-pair fix (#1442)
* fix(synthesize): UTF-16 surrogate-safe hard-split in chunker Part A of v0.42.0.0 fix wave: lifts surrogate-pair-safe slicing from src/core/eval-contradictions/judge.ts into a new shared module src/core/text-safe.ts. The dream-cycle chunker findBoundary tier-3 fallback (synthesize.ts) previously hard-split at maxChars, orphaning a high surrogate when the boundary landed inside emoji / non-BMP CJK / mathematical alphanumerics. Resulting chunks were not byte-identical to the source content, which broke the v0.30.2 D9 stable-chunk-identity invariant — the per-chunk idempotency key drifted across retries on transcripts containing 4-byte UTF-8 characters near a hard-split. Five agent-authored PRs (#1378-#1382) each independently introduced a narrow safeSliceEnd helper that handled ONE of the three correctness cases (high+low pair straddle) but missed the AT-low-surrogate case that fires when a boundary lands inside a complete pair. The shared text-safe.ts module exports both truncateUtf8 (the verbatim sliced string, for judge.ts) and safeSplitIndex (the boundary index, for chunker hot path), each covering all three cases. Co-authored credit: @garrytan-agents for surfacing the fix in PRs #1378-#1382 (closed in favor of consolidated design doc #1409). * New: src/core/text-safe.ts (truncateUtf8 + safeSplitIndex helpers). * New: test/text-safe.test.ts (18 cases, all 3 surrogate cases plus boundary-after-pair conservative back-up per codex CK16). * refactor(judge): import truncateUtf8 from text-safe; re-export for back-compat. Existing 32 judge tests pass unchanged. * fix(synthesize): findBoundary tier-3 routes through safeSplitIndex. 3 new surrogate-safety cases in test/cycle-synthesize-chunker.test.ts (emoji at boundary, non-BMP CJK at boundary, determinism + joined chunks reconstruct source byte-identical across 5 fuzzed hashes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(schema): widen link_source CHECK to include 'mentions' (v95) Part B of v0.42.0.0: link_source enum widening to admit a fourth provenance channel for auto-linked body-text mentions from the upcoming `gbrain extract links --by-mention` command. Codex outside-voice review on the v0.42.0.0 plan caught that the existing link_source CHECK is a hard wall (src/schema.sql:356) — my earlier draft claimed "no schema migration needed; link_source is free-form TEXT." Wrong. The CHECK admits only NULL OR ('markdown', 'frontmatter', 'manual'); attempting to insert link_source='mentions' would have raised a constraint violation on every auto-link write. Migration v95 widens the CHECK to admit 'mentions' alongside the three existing values. Mentions are intentionally a separate provenance from markdown (human-authored links) so the backlink-count SQL in postgres-engine + pglite-engine can filter `WHERE link_source != 'mentions'` for search ranking (D12). Mentions still count toward orphan-ratio and graph traversal — distinct semantics from the three human-authored sources, modeled cleanly on the dedicated CHECK value. * src/schema.sql: widened CHECK with provenance comment. * src/core/pglite-schema.ts: same widening (PGLite engine parity). * src/core/schema-embedded.ts: regenerated via `bun run build:schema`. * src/core/migrate.ts: new migration v95 `links_link_source_check_includes_mentions` with both Postgres and PGLite branches. DROP IF EXISTS + ADD CONSTRAINT pattern so re-applying the migration is a no-op (idempotent). * test/schema-migrate-link-source-mentions.test.ts (NEW, 7 cases): registration shape, SQL shape (all 4 values present + DROP IF EXISTS pattern), PGLite branch present, post-migration insert succeeds, CHECK still rejects unknown values (widening did not nullify the gate), idempotent re-application via runMigration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(orphans): expose getOrphansData alias as canonical pure data fn (D1) D1 from /plan-eng-review for v0.42.0.0: doctor's upcoming orphan_ratio check needs the SAME exclusion logic as `gbrain orphans` so the two surfaces cannot disagree on what counts as an orphan. The existing findOrphans() was already the pure data fn — this commit just makes that contract explicit via the getOrphansData alias and pins it with an IRON RULE regression test. * src/commands/orphans.ts: export const getOrphansData = findOrphans (alias, same function reference). Documents the v0.42.0.0 contract in findOrphans' docstring. * test/orphans-pure-fn.test.ts (NEW, 12 cases): - getOrphansData === findOrphans (same reference). - findOrphans + getOrphansData deep-equal output. - includePseudo branch toggles excluded count. - CLI --json output deep-equals findOrphans (IRON RULE — catches drift if anyone adds CLI-side post-filtering). - CLI --count matches total_orphans (with and without --include-pseudo). - shouldExclude regression: pseudo-pages, auto-suffix, raw segment, deny-prefixes, first-segment exclusions all fire correctly; regular slugs are NOT excluded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(engine): filter mentions out of backlink-count for search ranking (D12) D12 from /plan-eng-review for v0.42.0.0: codex outside-voice review caught that engine.getBacklinkCounts had NO link_source filter — so every link counted equally toward backlink-boost in hybridSearch. Running `gbrain extract links --by-mention` (migration #1 of #1409) would silently shift search ranking globally on first run, boosting popular-mention pages over intentional-backlink pages. Add `AND l.link_source IS DISTINCT FROM 'mentions'` to the LEFT JOIN in both engines. `IS DISTINCT FROM` is NULL-safe per the [sql-neq-misses-null-drift] memory: a naive `!= 'mentions'` would silently drop legacy pre-v0.13 rows where link_source IS NULL (because NULL != 'mentions' evaluates to NULL not TRUE in SQL three-valued logic). The IS DISTINCT FROM form treats NULL as a distinct value so legacy rows still count toward backlinks — the only rows filtered are the explicitly mention-derived ones from v0.42.0.0+. Mentions still count toward: - orphan-ratio (the whole point — `findOrphans` runs against `links` with no source filter, so an auto-linked page is no longer an orphan) - graph traversal (`traverseGraph` walks all link_source values) - graph adjacency (`getAdjacencyBoosts` includes mentions in the induced subgraph counts) Mentions are filtered ONLY from: - `getBacklinkCounts` (this commit) — the input to hybridSearch's backlink_boost stage * src/core/postgres-engine.ts: AND clause on the LEFT JOIN. * src/core/pglite-engine.ts: same change for engine parity. * test/backlink-count-mention-filter.test.ts (NEW, 6 cases): - 10 markdown + 0 mention → count = 10 - 0 markdown + 50 mention → count = 0 - 10 markdown + 50 mention → count = 10 - NULL link_source legacy rows still count (IS DISTINCT FROM semantics) - mixed (markdown + frontmatter + manual + mentions) → only mentions filtered - uninitialized slug returns 0 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(by-mention): pure mention scanner with gazetteer + guards (D2/D6/D12/D13) Net new module powering migration #1 of #1409 (orphan reduction). buildGazetteer queries entity-typed pages (hardcoded D2 filter: person/company/organization/entity, pack-aware deferred to TODO-1) and produces a token-Map lookup keyed by lowercase first-token. findMentionedEntities is a pure function that scans body text against the gazetteer, applies maximal-munch matching (longest entry wins at each offset), self-link guard (D13), cross-source guard, and per-page first-mention-only cap (1 link per source→target pair regardless of how many body mentions). Token-Map + multi-word phrase pass per D6 — no new deps, no regex alternation (pathological perf at 5K patterns), no Aho-Corasick (dep tax not justified at this scale). At each token offset, lookup in Map<lowercase, GazetteerEntry[]> is O(1); multi-word entries validate subsequent tokens. Bucket pre-sorted longest-first so the first valid entry IS the maximal-munch winner. Ignore-list semantics per CK12: built-in ambiguous tokens (Apple, Amazon, Square, Stripe, Box, Meta, Target, Oracle) suppressed at gazetteer-build time ONLY when no corresponding entity page exists. If the user has explicitly created companies/apple, gazetteer presence wins — ignore list does NOT override user intent. Min-name-length filter at 4 chars kills false-positive 2-3-char names (AI, YC, X, IBM). Codex CK13 noted this trade-off will under-deliver on 3-char real entities; pack-aware follow-up (TODO-1) can let users opt 3-char entity types in deliberately. Code-block stripping via existing stripCodeBlocks() from link-extraction.ts. CK8 fix: stripCodeBlocks was internal-only; this commit exports it so by-mention.ts can reuse without rolling its own fenced/inline code parser. * src/core/by-mention.ts (NEW, 240 LOC): - LINKABLE_ENTITY_TYPES const (hardcoded D2 type filter). - GazetteerEntry + Gazetteer + Mention types. - buildGazetteer(engine, opts) — engine-backed, hardcoded type filter, ignore-list at build time per CK12, sort buckets longest-first. - findMentionedEntities(text, gazetteer, opts) — pure, maximal-munch, guards (self-link/cross-source/first-mention-cap), code-block strip. * src/core/link-extraction.ts: export stripCodeBlocks (CK8 fix). * test/by-mention.test.ts (NEW, 22 cases): - All 20 plan-mandated cases. - Plus extraIgnore user-override case + LINKABLE_ENTITY_TYPES contract pin. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(extract): --by-mention auto-link entity mentions (migration #1 of #1409) Wires the v0.42.0.0 mention scanner into 'gbrain extract links'. Mode dispatch: when --by-mention is set, runs ONLY the new mention pass (skips default link/frontmatter extract) so the two surfaces don't conflict mid-run. The default extract path is unchanged. Flag plumbing: * --by-mention: opts into the mention pass. Mode dispatch. * --source fs --by-mention rejected with paste-ready --source db fix-hint (D7: gazetteer needs the engine; FS-walk + DB-gazetteer is incoherent). * timeline --by-mention rejected (mentions are a links-pass concern). * --source-id scopes the page WALK; gazetteer remains brain-wide (cross-source guard in findMentionedEntities suppresses scanning pages in source A from auto-linking entities in source B). * --since DATE filters the walk to recently-modified pages. * --type filter applies (rarely useful; included for parity). * --dry-run prints add_link action lines without writing; --json emits one JSON line per dry-run action. extractMentionsFromDb function: * buildGazetteer once per run via hardcoded type filter (D2). * Walks pages via engine.listAllPageRefs (DB-source only). * Reads body as compiled_truth || '\n\n' || COALESCE(timeline, '') per D3 — separator-joined so an end-of-compiled token doesn't merge with a start-of-timeline token into a false phrase match. * findMentionedEntities returns Mention[] with self-link guard (D13) + cross-source guard + first-mention-only cap baked in. * addLinksBatch with link_source='mentions' — distinct provenance channel that backlink-count filters out for search ranking (D12). * Empty-gazetteer no-op with informative message (no entity pages = nothing to scan). * src/commands/extract.ts: --by-mention flag + mode dispatch + FS rejection + extractMentionsFromDb function (~120 LOC). * test/extract-by-mention.test.ts (NEW, 12 cases): end-to-end happy path, idempotency, --dry-run no writes, --json output shape, --source-id scoping, --source fs rejection with fix-hint, timeline rejection, mode dispatch (no markdown rows when --by-mention), coexistence of markdown + mention link_source on same (from,to) pair via ON CONFLICT key, schema migration verification (link_source='mentions' insert succeeds), empty-brain no-op, cross-source guard (team-b post → default acme = no link). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): orphan_ratio check on local + thin-client surfaces (D5/D11) D5/D11 from /plan-eng-review for v0.42.0.0: surface orphan-page count in 'gbrain doctor' so users discover the new --by-mention fix without having to know the feature exists. Two surfaces because thin-client installs (gbrain init --mcp-only) route to runRemoteDoctor entirely — adding the check to runDoctor only would miss every brain-server consumer (codex CK5 caught this exactly during outside-voice review). Local surface (src/commands/doctor.ts): * Inserts as check '9b' right after graph_coverage. * Consumes getOrphansData() — the canonical pure data fn from T5 — so doctor and 'gbrain orphans --count' cannot disagree on the ratio. * Vacuous gate at < 100 entity pages (small brains naturally show high orphan ratio; not actionable signal). * warn > 0.5, fail > 0.8; both states recommend 'gbrain extract links --by-mention' as the fix. Thin-client surface (src/core/doctor-remote.ts): * New exported runOrphanRatioCheck function. Mirrors local logic but routes through find_orphans MCP op (existing v0.12.3 op, scope: read — even minimal-scope thin-clients can call it). * Operator-pointing hint: 'Ask the brain operator at <url> to run gbrain extract links --by-mention'. Thin-client users can't run the fix against a brain they don't host (v0.31.1 bug class). * Network failure fall-back: returns informational ok with network_error detail, NOT fail — earlier mcp_smoke catches genuine unreachable; orphan_ratio is informational only. * Skippable via the existing skipScopeProbe flag so hermetic fixtures that don't implement find_orphans on /mcp don't hang. Wiring in --by-mention extract.ts integration test (fix-up): CliOptions field is `progressInterval` not `progressIntervalMs`, and `timeoutMs: null` is required. Pre-existing tsc error surfaced when typechecking the new doctor changes. * test/doctor-orphan-ratio.test.ts (NEW, 10 cases): - <100 entity pages → vacuous ok - 100+ entities + low ratio (20%) → ok - high ratio (70%) → warn with fix-hint - very high ratio (90%) → fail with urgency fix-hint - zero entity pages → vacuous ok - JSON envelope contains orphan_ratio check - Thin-client: network failure → informational ok with detail - Cross-surface parity: source greps verify orphan_ratio name and fix command appear in BOTH doctor.ts and doctor-remote.ts; local hint is self-fix, thin-client hint asks the operator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): orphan-reduction end-to-end with cross-surface count parity Pins the v0.42.0.0 design-doc claim shape — "material reduction in orphan pages via --by-mention" — without committing to a specific % (per TODO-4=C decision to soften the 88%->_30% promise into a "material reduction, exact figure TBD via post-merge measurement on representative brain"). 3 e2e cases via hermetic PGLite: * Seed 20 entities + 5 content pages mentioning 15 → assert orphan count drops by >=10 after --by-mention (material delta). * Cross-check the D1 single-source contract end-to-end: gbrain orphans --count, getOrphansData() pure fn, and the doctor JSON orphan_ratio message all reflect the same numerator. If a future change makes them disagree, this fires. * Re-run idempotency: second --by-mention invocation produces 0 new mention rows AND the first run actually created some (sanity gate so a no-op pass doesn't trivially satisfy the idempotency test). * test/e2e/orphan-reduction.test.ts (NEW, 3 cases, hermetic PGLite, no DATABASE_URL needed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * release: v0.41.10.0 — orphan reduction via --by-mention + surrogate-pair fix Bumps VERSION + package.json to 0.41.10.0 (next available slot in the v0.41.x queue after master moved to v0.41.4.0). Minor bump scope: new CLI flag (`gbrain extract links --by-mention`), new schema migration v95, new doctor check `orphan_ratio`, new public src/core/text-safe.ts module, new src/core/by-mention.ts module, new link_source enum value with ranking-filter semantic. CHANGELOG entry follows the v0.41.x voice rules: ELI10 lead, To take advantage block with paste-ready commands, How to turn it on, What you'd see, Promise calibration (softens design-doc 88%->_30% claim per codex CK13), What to watch for, Itemized changes split into Part A (surrogate-pair fix) + Part B (auto-link --by-mention) + Follow-ups (TODO-1 through TODO-4). Credits @garrytan-agents for the underlying PR work (#1378-#1382 closed in favor of design doc #1409). TODOS.md gets four new follow-up entries (pack-aware gazetteer, cycle integration, MCP op, post-merge measurement). System-of-record annotation: the addLinksBatch call in extractMentionsFromDb carries `gbrain-allow-direct-insert` per the canonical reconcile-layer write pattern. 3-line audit: VERSION + package.json + CHANGELOG top all on 0.41.10.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
0b7efd3528
commit
dfa15ba22b
+113
@@ -2,6 +2,119 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.10.0] - 2026-05-25
|
||||
|
||||
**Your brain stops being mostly orphan pages.** A new `gbrain extract links --by-mention` pass scans every page's body text for mentions of people and companies you already have pages for, then creates links automatically. The same release also fixes a silent corruption bug in the dream-cycle chunker that could split UTF-16 surrogate pairs (emoji, non-BMP CJK, mathematical alphanumerics) at chunk boundaries, breaking the per-chunk idempotency key on retries.
|
||||
|
||||
Most production brains accumulate orphans silently over months because the existing link extractor only sees explicit markdown links. If a meeting note mentions "Acme Corp" or a Slack-import page mentions "Alice Example" in plain text, no link gets created. The `--by-mention` pass closes the gap: build a gazetteer from your existing entity pages, scan every page's body, create one mention link per (source → target) pair.
|
||||
|
||||
Five agent-authored PRs (#1378–#1382) surfaced the design and the surrogate-pair fix; this release consolidates the work per design doc #1409.
|
||||
|
||||
## To take advantage of v0.41.10.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
This applies schema migration v95 (widens the `link_source` CHECK to admit `'mentions'`).
|
||||
2. **Check your orphan ratio:**
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name=="orphan_ratio")'
|
||||
```
|
||||
3. **Preview what would get auto-linked:**
|
||||
```bash
|
||||
gbrain extract links --by-mention --dry-run
|
||||
```
|
||||
4. **Apply the auto-link pass:**
|
||||
```bash
|
||||
gbrain extract links --by-mention
|
||||
```
|
||||
~5 seconds per 1K pages on a representative brain.
|
||||
5. **Verify orphan reduction:**
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name=="orphan_ratio")'
|
||||
gbrain orphans --count
|
||||
```
|
||||
Both numbers will match (they consume the same canonical `getOrphansData()` pure data fn).
|
||||
6. **If any step fails,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and which step broke.
|
||||
|
||||
### How to turn it on
|
||||
|
||||
```bash
|
||||
# One-shot — auto-link every body-text mention of an entity page:
|
||||
gbrain extract links --by-mention
|
||||
|
||||
# Preview first:
|
||||
gbrain extract links --by-mention --dry-run
|
||||
|
||||
# Federated-brain users can scope the walk to one source:
|
||||
gbrain extract links --by-mention --source-id team-b
|
||||
|
||||
# Incremental — only scan recently-modified pages:
|
||||
gbrain extract links --by-mention --since 2026-05-01
|
||||
|
||||
# JSON output for agent consumption:
|
||||
gbrain extract links --by-mention --dry-run --json
|
||||
```
|
||||
|
||||
### What you'd see in a concrete example
|
||||
|
||||
| Scenario | Before | After |
|
||||
|---|---|---|
|
||||
| `gbrain doctor` on a 165K-page brain with low link coverage | no orphan_ratio surface; brain_score warns about graph health generically | new `orphan_ratio` check fires with explicit count + paste-ready fix-hint |
|
||||
| `gbrain extract links --by-mention` against a representative brain | command did not exist | walks DB, builds gazetteer of entity pages, scans bodies, writes mention links |
|
||||
| Meeting note saying "Met with Acme Corp and Alice yesterday" | no inbound links to companies/acme or people/alice from this page | both targets gain inbound `link_source='mentions'` rows |
|
||||
| Search ranking after `--by-mention` run | (would have shifted globally — mentions counted equally with intent-authored backlinks) | unchanged — mentions filtered out of backlink-count at the SQL layer (D12) |
|
||||
| `gbrain doctor` on thin-client install | no visibility into brain-server orphan health | `orphan_ratio` check fires via `find_orphans` MCP op with operator-pointing fix-hint |
|
||||
| Dream-cycle chunker on transcript with 🚀 at hard-split boundary | high surrogate orphaned in chunk N, low surrogate orphaned at start of chunk N+1 — invalid UTF-16, content_hash changed across retries | `safeSplitIndex` backs up to a safe boundary; chunks are byte-identical to source; D9 idempotency invariant preserved |
|
||||
|
||||
Promise calibration: design doc #1409 originally framed this as "88% orphans → <30%." Codex outside-voice review on the v0.41.10.0 plan flagged that strict-exact title matching + min-name-length ≥4 + no-aliases + no-fuzzy will under-deliver on the real corner cases — 3-char real entities ("YC", "AI"), first-name-only mentions ("Bob"), abbreviations, old company names. Material reduction is the realistic v1 outcome; exact figure TBD via post-merge measurement on a representative brain. TODO-1 (pack-aware gazetteer) + TODO-4 (post-merge measurement) cover the next iterations.
|
||||
|
||||
### What to watch for
|
||||
|
||||
- **Search ranking is unchanged.** Mention links go into `link_source='mentions'` which is filtered OUT of backlink-count via `IS DISTINCT FROM 'mentions'` in both engines (D12). Existing markdown / frontmatter / manual links still count toward backlink-boost. NULL-source legacy rows still count (the `IS DISTINCT FROM` form is NULL-safe per the `[sql-neq-misses-null-drift]` learning).
|
||||
- **Cycle integration is deferred.** v1 is CLI-only; `auto_link_mentions` config gate dropped from v1. Run `--by-mention` manually or via cron. Cycle-phase wiring is TODO-2 (P2 follow-up).
|
||||
- **Pack-aware gazetteer is deferred.** Hardcoded entity types for v1: `person`, `company`, `organization`, `entity`. User-defined schema-pack entity types (e.g. `researcher`) won't be auto-linked until TODO-1 (P2 follow-up) lands.
|
||||
- **Self-link guard is on.** An entity page mentioning its own title (e.g. body of `companies/acme.md` says "Acme has 500 customers") will NOT auto-create a self-link, avoiding fake orphan-reduction.
|
||||
- **Cross-source guard is on.** A page in source A mentioning an entity in source B will NOT auto-link in v1 — deliberate isolation. Can relax in a future wave if real cross-source linking is needed.
|
||||
- **Ignore list at gazetteer-build time, not match time.** Built-in ambiguous tokens (Apple, Amazon, Square, Stripe, Box, Meta, Target, Oracle) are dropped from the gazetteer ONLY when no corresponding entity page exists. If you've explicitly created `companies/apple`, the auto-link fires (your intent wins).
|
||||
- **FS-source + `--by-mention` rejected.** The gazetteer needs the engine; FS-walk + DB-gazetteer is incoherent. Use `--source db` (or default) for `--by-mention`.
|
||||
- **`--by-mention timeline` rejected.** Mentions are a links-pass concern.
|
||||
- **Schema migration v95** widens the `link_source` CHECK to admit `'mentions'`. DROP-IF-EXISTS + ADD CONSTRAINT pattern, idempotent on re-application.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Part A — UTF-16 surrogate-pair safety in chunker:**
|
||||
|
||||
- New `src/core/text-safe.ts` exports `truncateUtf8` (moved verbatim from `src/core/eval-contradictions/judge.ts:81-106`) + new sibling `safeSplitIndex(text, maxChars): number` (returns the boundary INDEX without allocating a sliced string — what the chunker hot path needs).
|
||||
- `src/core/cycle/synthesize.ts` `findBoundary` tier-3 hard-split routes through `safeSplitIndex` so a boundary that lands inside a UTF-16 surrogate pair no longer orphans the high surrogate. The agent-authored fix from PRs #1378–#1382 handled only one of three correctness cases (high+low pair straddle); the case where the cut lands AT a low surrogate (high at maxChars-2) silently bit. `text-safe.ts` covers all three cases.
|
||||
- `src/core/eval-contradictions/judge.ts` re-imports `truncateUtf8` from the new shared location with byte-identical behavior; existing 32 judge tests pass unchanged.
|
||||
- 21 new test cases pinning the surrogate safety (`test/text-safe.test.ts` + 3 new cases in `test/cycle-synthesize-chunker.test.ts`).
|
||||
|
||||
**Part B — Auto-link entity mentions:**
|
||||
|
||||
- New `src/core/by-mention.ts` (~240 LOC):
|
||||
- `buildGazetteer(engine, opts)` queries entity-typed pages, applies min-name-length filter (≥4 chars), applies the built-in ignore list at build time (only when no corresponding page exists), returns `Map<lowercase_first_token, GazetteerEntry[]>` sorted longest-first per bucket.
|
||||
- `findMentionedEntities(text, gazetteer, opts)` is a pure function: maximal-munch matcher at each token offset, self-link guard (D13), cross-source guard, per-page first-mention-only cap. Uses existing `stripCodeBlocks()` from `link-extraction.ts` so mentions inside ``` blocks and inline backticks are ignored.
|
||||
- `src/core/link-extraction.ts` exports `stripCodeBlocks` (was internal — codex CK8 fix; needed for `by-mention.ts` reuse).
|
||||
- New `--by-mention` flag on `gbrain extract links`. Mode dispatch: when set, runs ONLY the mention pass. FS-source + `--by-mention` rejected with paste-ready `--source db` fix-hint. `timeline --by-mention` rejected (mentions are a links-pass concern). Honors `--source-id`, `--since`, `--dry-run`, `--json`.
|
||||
- `getOrphansData()` exported as the canonical pure data fn alias for `findOrphans()` in `src/commands/orphans.ts`. The doctor `orphan_ratio` check consumes it; if a future change adds CLI-side post-filtering, the IRON RULE regression test in `test/orphans-pure-fn.test.ts` fires.
|
||||
- New `orphan_ratio` doctor check on BOTH local (`runDoctor`) and thin-client (`runRemoteDoctor` via `find_orphans` MCP op) surfaces. Vacuous gate at <100 entity pages, warn >0.5, fail >0.8. Local hint: `Run: gbrain extract links --by-mention`. Thin-client hint: `Ask the brain operator at <url> to run...` (D11).
|
||||
- Migration v95 (`links_link_source_check_includes_mentions`) widens the CHECK constraint to admit `'mentions'`. DROP-IF-EXISTS + ADD CONSTRAINT pattern; engine-parity entry in `pglite-schema.ts`.
|
||||
- Backlink-count SQL in both engines gains `AND l.link_source IS DISTINCT FROM 'mentions'` on the LEFT JOIN. Mentions filtered from search ranking; still count toward orphan-ratio and graph traversal. NULL-safe per the `[sql-neq-misses-null-drift]` memory (NULL legacy rows still count).
|
||||
- 35 new test cases across `test/by-mention.test.ts` (22), `test/extract-by-mention.test.ts` (12), `test/doctor-orphan-ratio.test.ts` (10), `test/backlink-count-mention-filter.test.ts` (6), `test/schema-migrate-link-source-mentions.test.ts` (7), `test/orphans-pure-fn.test.ts` (12), `test/e2e/orphan-reduction.test.ts` (3).
|
||||
|
||||
**Follow-ups filed in TODOS.md:**
|
||||
|
||||
- TODO-1 P2: Pack-aware `--by-mention` gazetteer (add `linkable: boolean` per-type field to the schema-pack manifest; new accessor `linkableTypesFromPack`).
|
||||
- TODO-2 P2: Cycle integration for `--by-mention` (auto_link_mentions config gate; requires runExtractCore DB-source refactor OR a new cycle-only helper).
|
||||
- TODO-3 P3: MCP op `extract_links_by_mention` for remote brain-server callers.
|
||||
- TODO-4 P1: Post-merge measurement on a representative brain; update #1409 design doc with the measured orphan-ratio delta.
|
||||
|
||||
Co-authored credit: `@garrytan-agents` for surfacing both the surrogate-pair fix and the orphan-reduction design across PRs #1378-#1382 (now closed in favor of consolidated design doc #1409).
|
||||
## [0.41.9.0] - 2026-05-25
|
||||
|
||||
**Five UX/reliability fixes from a single production incident report. Your
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
# TODOS
|
||||
|
||||
## v0.41.10.0 follow-ups (orphan-reduction + surrogate fix wave)
|
||||
|
||||
- [ ] **TODO-1 (P2) — Pack-aware `--by-mention` gazetteer.** Add `linkable: boolean` per-type field to the schema-pack manifest (`src/core/schema-pack/manifest-v1.ts`, currently has `extractable` + `expert_routing`). New accessor `linkableTypesFromPack(pack: ResolvedPack)` in a new `schema-pack/linkable-types.ts` module mirroring `expert-types.ts`. `src/core/by-mention.ts:buildGazetteer` consults the pack-aware filter first via `loadActivePackBestEffort(ctx)`, falls back to the hardcoded `LINKABLE_ENTITY_TYPES` const for non-pack brains. Respects the D4 fail-empty contract (pack-load failure → empty filter, NOT hardcoded defaults). User-defined types like `researcher` get auto-linked. Requires: pack-schema bump, rubric/registry updates, regression test that pack-aware + non-pack brains produce expected gazetteer shapes.
|
||||
|
||||
- [ ] **TODO-2 (P2) — Cycle integration for `--by-mention`.** v0.41.10.0 ships CLI-only. Wire the mention pass into the dream-cycle extract phase so brains running autopilot get incremental auto-link without manual cron. Two paths: (a) refactor `runExtractCore` (currently FS-only at `extract.ts:320`) to support DB-source, then cycle calls it as before; (b) add a dedicated `extractMentionsFromDbForCycle()` callable directly from `runPhaseExtract` at `core/cycle.ts:810` so `runExtractCore` stays focused. Add `auto_link_mentions` config gate (default OFF for safety — opt-in). Also resolve the `sourceScopeOpts(ctx)` issue: cycle context doesn't have an `OperationContext`; need a new helper that produces equivalent scoping for the trusted-workspace cycle write context.
|
||||
|
||||
- [ ] **TODO-3 (P3) — MCP op `extract_links_by_mention` for remote brain-server callers.** v0.41.10.0 CLI-only because the API shape was new. Once the CLI is proven (post-ship measurement window), expose as MCP op with `scope: write`, NOT `localOnly` (remote OpenClaw agents should be able to trigger). Trust gate via `op-trust-gate.ts`. Params: optional `source_id`, optional `since`, `dry_run`. Returns `{created, pages}`. Add to `src/core/operations.ts` operation list; wire MCP definitions.
|
||||
|
||||
- [ ] **TODO-4 (P1) — Measure actual orphan-ratio reduction on representative brain post-merge.** v0.41.10.0 CHANGELOG softens the design-doc claim from "88% → <30%" to "material reduction, exact figure TBD" per codex CK13 (strict-exact + min-length≥4 + no-aliases + no-fuzzy will under-deliver on 3-char real entities like "YC", first-name mentions like "Bob", and abbreviations). After v0.41.10.0 lands, run `gbrain extract links --by-mention` against the production OpenClaw deployment (~165K pages) and capture before/after orphan_ratio from `gbrain doctor --json`. Update `docs/designs/GBRAIN_ONBOARD.md` (in PR #1409 if still open, or as follow-up edit if merged) with the measured number. Update CHANGELOG retroactively only if the measurement is material to user expectations.
|
||||
|
||||
## v0.41.6.0 follow-ups (v0.41.7+)
|
||||
|
||||
- [ ] **v0.41.7+: investigate v0.40+ schema-probe deadlock ROOT cause.**
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.41.9.0",
|
||||
"version": "0.41.10.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -3513,6 +3513,59 @@ export async function buildChecks(
|
||||
checks.push({ name: 'graph_coverage', status: 'warn', message: 'Could not check graph coverage' });
|
||||
}
|
||||
|
||||
// 9b. v0.42.0.0 — orphan_ratio check (migration #1 of #1409).
|
||||
//
|
||||
// Surfaces the fraction of linkable pages with no inbound links.
|
||||
// Consumes the same canonical getOrphansData() pure fn as
|
||||
// `gbrain orphans --count` (D1), so the two surfaces cannot disagree.
|
||||
//
|
||||
// Skip when entity count < 100 (vacuous — small brains naturally
|
||||
// show high orphan ratio; not actionable signal).
|
||||
// Warn at >0.5; fail at >0.8. Both states recommend
|
||||
// `gbrain extract links --by-mention` as the fix.
|
||||
progress.heartbeat('orphan_ratio');
|
||||
try {
|
||||
const { getOrphansData } = await import('./orphans.ts');
|
||||
const entityCount = (await engine.executeRaw<{ count: number }>(
|
||||
"SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization') AND deleted_at IS NULL",
|
||||
))[0]?.count ?? 0;
|
||||
if (entityCount < 100) {
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'ok',
|
||||
message: `Vacuous: ${entityCount} entity pages (<100). Orphan ratio not meaningful at this scale.`,
|
||||
});
|
||||
} else {
|
||||
const data = await getOrphansData(engine, { includePseudo: false });
|
||||
const ratio = data.total_linkable > 0 ? data.total_orphans / data.total_linkable : 0;
|
||||
const pct = (ratio * 100).toFixed(0);
|
||||
const hint =
|
||||
'Run: gbrain extract links --by-mention (auto-links entity mentions in body text). ' +
|
||||
'Run gbrain orphans for the list.';
|
||||
if (ratio > 0.8) {
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'fail',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`,
|
||||
});
|
||||
} else if (ratio > 0.5) {
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'warn',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages have no inbound links). ${hint}`,
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'orphan_ratio',
|
||||
status: 'ok',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${data.total_linkable} linkable pages)`,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
checks.push({ name: 'orphan_ratio', status: 'warn', message: 'Could not check orphan ratio' });
|
||||
}
|
||||
|
||||
// 10. Integrity sample scan (v0.13 knowledge runtime).
|
||||
// Read-only — no network, no writes, no resolver calls. Samples the first
|
||||
// 500 pages by slug order and surfaces bare-tweet + dead-link counts as a
|
||||
|
||||
+176
-7
@@ -29,6 +29,7 @@ import {
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { pathToSlug, pruneDir, isSyncable } from '../core/sync.ts';
|
||||
import { buildGazetteer, findMentionedEntities } from '../core/by-mention.ts';
|
||||
|
||||
// Batch size for addLinksBatch / addTimelineEntriesBatch.
|
||||
// Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling;
|
||||
@@ -397,6 +398,11 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
// v0_13_0 migration orchestrator runs this once under the hood; users
|
||||
// opt in for subsequent runs.
|
||||
const includeFrontmatter = args.includes('--include-frontmatter');
|
||||
// v0.42.0.0 Part B: --by-mention auto-link body-text entity mentions
|
||||
// via the gazetteer pass. Mode dispatch — when set, run ONLY the
|
||||
// mention pass (skip default link extract). DB-source only per D7;
|
||||
// FS-source is rejected with a paste-ready fix-hint below.
|
||||
const byMention = args.includes('--by-mention');
|
||||
|
||||
// Validate --since upfront. Without this, an invalid date like
|
||||
// `--since yesterday` produces NaN which silently passes the filter check
|
||||
@@ -420,6 +426,29 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// v0.42.0.0 D7: --by-mention requires DB-source. Gazetteer construction
|
||||
// needs the engine; mixing FS-walk with DB-gazetteer is incoherent
|
||||
// (you'd scan files on disk for mentions of entities that may not exist
|
||||
// in any synced page). Fail loud with a paste-ready fix-hint.
|
||||
if (byMention && source === 'fs') {
|
||||
console.error(
|
||||
`--by-mention requires --source db (currently --source fs). The mention scanner ` +
|
||||
`needs the engine to build the entity gazetteer. Re-run as:\n\n` +
|
||||
` gbrain extract ${subcommand} --by-mention --source db` +
|
||||
(sourceIdFilter ? ` --source-id ${sourceIdFilter}` : '') +
|
||||
(since ? ` --since ${since}` : '') +
|
||||
(dryRun ? ' --dry-run' : '') + '\n',
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
if (byMention && subcommand === 'timeline') {
|
||||
console.error(
|
||||
`--by-mention is a links-pass only; it does not apply to timeline extraction. ` +
|
||||
`Re-run as 'gbrain extract links --by-mention' or 'gbrain extract all --by-mention'.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// FS source needs a brain dir. When --dir wasn't passed, resolve from
|
||||
// sources(local_path) — same path `gbrain sync` uses — instead of
|
||||
// silently walking cwd. See the brainDir comment above for the footgun.
|
||||
@@ -451,15 +480,27 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
// is fs-only; we keep the dual codepath here so Minions handlers
|
||||
// can opt in via mode + source.
|
||||
result = { links_created: 0, timeline_entries_created: 0, pages_processed: 0 };
|
||||
if (subcommand === 'links' || subcommand === 'all') {
|
||||
const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since, { includeFrontmatter, sourceIdFilter });
|
||||
// v0.42.0.0: --by-mention is a mode dispatch. When set, run ONLY
|
||||
// the mention pass and skip the default link/frontmatter extract.
|
||||
// The two passes write different link_source values ('mentions' vs
|
||||
// 'markdown'/'frontmatter') so they don't conflict, but mixing them
|
||||
// in a single CLI invocation is surprising — keep the surfaces
|
||||
// separate.
|
||||
if (byMention) {
|
||||
const r = await extractMentionsFromDb(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
result.links_created = r.created;
|
||||
result.pages_processed = r.pages;
|
||||
}
|
||||
if (subcommand === 'timeline' || subcommand === 'all') {
|
||||
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
result.timeline_entries_created = r.created;
|
||||
result.pages_processed = Math.max(result.pages_processed, r.pages);
|
||||
} else {
|
||||
if (subcommand === 'links' || subcommand === 'all') {
|
||||
const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since, { includeFrontmatter, sourceIdFilter });
|
||||
result.links_created = r.created;
|
||||
result.pages_processed = r.pages;
|
||||
}
|
||||
if (subcommand === 'timeline' || subcommand === 'all') {
|
||||
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
result.timeline_entries_created = r.created;
|
||||
result.pages_processed = Math.max(result.pages_processed, r.pages);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
result = await runExtractCore(engine, {
|
||||
@@ -1053,3 +1094,131 @@ async function extractTimelineFromDB(
|
||||
}
|
||||
return { created, pages: processed };
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.0.0 Part B (migration #1 of #1409) — auto-link body-text entity
|
||||
* mentions to known entity pages.
|
||||
*
|
||||
* Walks every page (respecting --source-id / --type / --since filters),
|
||||
* scans `compiled_truth || '\n\n' || COALESCE(timeline, '')` per D3
|
||||
* against the gazetteer built via `buildGazetteer`, and writes one link
|
||||
* per (from_page, to_page) pair with `link_source='mentions'`. The
|
||||
* mention link_source is filtered OUT of backlink-count per D12 so
|
||||
* search ranking semantics are preserved.
|
||||
*
|
||||
* Source isolation: mentions cross-source pages are deliberately
|
||||
* suppressed by `findMentionedEntities`'s cross-source guard. Page in
|
||||
* source A mentions entity in source B → no link created. v1
|
||||
* conservative posture; relaxable in a future wave.
|
||||
*/
|
||||
async function extractMentionsFromDb(
|
||||
engine: BrainEngine,
|
||||
dryRun: boolean,
|
||||
jsonMode: boolean,
|
||||
typeFilter: PageType | undefined,
|
||||
since: string | undefined,
|
||||
opts?: { sourceIdFilter?: string },
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
const sourceIdFilter = opts?.sourceIdFilter;
|
||||
|
||||
// Build gazetteer once per run. Skip everything if there are no
|
||||
// linkable entities — vacuous truth, no mentions to find.
|
||||
const gazetteer = await buildGazetteer(engine);
|
||||
if (gazetteer.size === 0) {
|
||||
if (jsonMode) {
|
||||
process.stdout.write(JSON.stringify({ event: 'no_gazetteer', message: 'no linkable entity pages found; nothing to scan' }) + '\n');
|
||||
} else {
|
||||
console.log('No linkable entity pages found in this brain (need pages with type IN person/company/organization/entity).');
|
||||
}
|
||||
return { created: 0, pages: 0 };
|
||||
}
|
||||
|
||||
const allRefs = sourceIdFilter
|
||||
? (await engine.listAllPageRefs()).filter(r => r.source_id === sourceIdFilter)
|
||||
: await engine.listAllPageRefs();
|
||||
|
||||
let processed = 0;
|
||||
let created = 0;
|
||||
const batch: LinkBatchInput[] = [];
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('extract.by_mention.scan', allRefs.length);
|
||||
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addLinksBatch(batch); // gbrain-allow-direct-insert: gbrain extract --by-mention — canonical auto-link write from body-text mention scan
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} link rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
const sinceMs = since ? new Date(since).getTime() : null;
|
||||
|
||||
for (const { slug, source_id } of allRefs) {
|
||||
const page = await engine.getPage(slug, { sourceId: source_id });
|
||||
if (!page) continue;
|
||||
if (typeFilter && page.type !== typeFilter) continue;
|
||||
if (sinceMs !== null) {
|
||||
const updatedMs = new Date(page.updated_at).getTime();
|
||||
if (Number.isFinite(updatedMs) && updatedMs <= sinceMs) continue;
|
||||
}
|
||||
processed++;
|
||||
progress.tick();
|
||||
|
||||
// D3: scan both columns joined with a paragraph separator so an
|
||||
// end-of-compiled token doesn't accidentally merge with a
|
||||
// start-of-timeline token into a false phrase match.
|
||||
const body = page.compiled_truth + '\n\n' + (page.timeline ?? '');
|
||||
if (!body.trim()) continue;
|
||||
|
||||
const mentions = findMentionedEntities(body, gazetteer, {
|
||||
fromSlug: slug,
|
||||
fromSourceId: source_id,
|
||||
});
|
||||
|
||||
if (mentions.length === 0) continue;
|
||||
|
||||
for (const m of mentions) {
|
||||
if (dryRun) {
|
||||
if (jsonMode) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
action: 'add_link', from: slug, from_source_id: source_id,
|
||||
to: m.slug, to_source_id: m.source_id,
|
||||
type: 'mentions', context: m.name, link_source: 'mentions',
|
||||
}) + '\n');
|
||||
} else {
|
||||
console.log(` ${slug} → ${m.slug} (mentions: "${m.name}")`);
|
||||
}
|
||||
created++;
|
||||
} else {
|
||||
batch.push({
|
||||
from_slug: slug,
|
||||
to_slug: m.slug,
|
||||
link_type: 'mentions',
|
||||
link_source: 'mentions',
|
||||
context: m.name,
|
||||
from_source_id: source_id,
|
||||
to_source_id: m.source_id,
|
||||
});
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!dryRun) await flush();
|
||||
progress.finish();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
console.log(`Mentions: ${label} ${created} links from ${processed} pages against gazetteer of ${gazetteer.size} first-token buckets`);
|
||||
}
|
||||
return { created, pages: processed };
|
||||
}
|
||||
|
||||
@@ -116,6 +116,14 @@ export async function queryOrphanPages(
|
||||
* Returns structured OrphanResult with totals.
|
||||
*
|
||||
* As of v0.17: `engine` is required. See queryOrphanPages for rationale.
|
||||
*
|
||||
* v0.42.0.0 (D1 from /plan-eng-review): this is the canonical pure data
|
||||
* fn for "what counts as an orphan in this brain." Re-exported as
|
||||
* `getOrphansData` for the doctor `orphan_ratio` check and any other
|
||||
* consumer that needs the same exclusion logic (AUTO_SUFFIX_PATTERNS,
|
||||
* PSEUDO_SLUGS, RAW_SEGMENT, DENY_PREFIXES, FIRST_SEGMENT_EXCLUSIONS).
|
||||
* Two consumers sharing one definition = doctor and `gbrain orphans`
|
||||
* cannot disagree on the orphan count.
|
||||
*/
|
||||
export async function findOrphans(
|
||||
engine: BrainEngine,
|
||||
@@ -164,6 +172,15 @@ export async function findOrphans(
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.0.0 D1: canonical name for the pure data fn consumed by both
|
||||
* `gbrain orphans` CLI AND doctor's `orphan_ratio` check. Aliased to
|
||||
* `findOrphans` so the existing CLI behavior + the test surface stay
|
||||
* byte-identical; new consumers should import `getOrphansData` to make
|
||||
* the data-only intent explicit at the call site.
|
||||
*/
|
||||
export const getOrphansData = findOrphans;
|
||||
|
||||
// --- Output formatters ---
|
||||
|
||||
export function formatOrphansText(result: OrphanResult): string {
|
||||
|
||||
@@ -0,0 +1,307 @@
|
||||
/**
|
||||
* v0.42.0.0 Part B — Auto-link entity mentions to known entity pages.
|
||||
* Migration #1 of the consolidated #1409 design doc (orphan reduction).
|
||||
*
|
||||
* `buildGazetteer` queries the brain for entity-typed pages and produces a
|
||||
* token-Map lookup structure suitable for fast body-text scanning.
|
||||
*
|
||||
* `findMentionedEntities` is a pure function that scans body text against
|
||||
* the gazetteer, applies the maximal-munch matcher (longest gazetteer
|
||||
* entry wins at each offset), self-link guard, cross-source guard, and
|
||||
* per-page first-mention-only cap (1 link per (source_slug, target_slug)).
|
||||
*
|
||||
* Design decisions locked in /plan-eng-review for v0.42.0.0:
|
||||
* - D2/D10 Hardcoded entity-type filter (not pack-aware) — pack v2
|
||||
* extension filed as TODO-1.
|
||||
* - D6 Token-Map + multi-word phrase pass (no new deps, no regex
|
||||
* alternation, no Aho-Corasick).
|
||||
* - D7 DB-source only — caller restricts page WALK to DB iteration.
|
||||
* - D12 `link_source='mentions'` writes filtered out of backlink-count
|
||||
* for search ranking (see postgres-engine.ts/pglite-engine.ts).
|
||||
* - D13 Self-link guard.
|
||||
* - CK12 Ignore-list applied at gazetteer-build time, NOT match time.
|
||||
* Built-in ambiguous tokens (Apple, Amazon, Square, Stripe, Box)
|
||||
* are dropped from the gazetteer ONLY when no corresponding
|
||||
* entity page exists. If a page DOES exist, the user explicitly
|
||||
* created it and we trust the gazetteer presence.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { stripCodeBlocks } from './link-extraction.ts';
|
||||
|
||||
/** D2: hardcoded entity types for v1. Pack-aware extension is TODO-1. */
|
||||
export const LINKABLE_ENTITY_TYPES = ['person', 'company', 'organization', 'entity'] as const;
|
||||
|
||||
/**
|
||||
* Minimum title length for gazetteer inclusion. Filters out 2-3 char names
|
||||
* (AI, YC, X, IBM) that produce dense false-positive auto-links in body text.
|
||||
* Codex CK13 noted v1 will under-deliver on 3-char real entities; the
|
||||
* pack-aware follow-up (TODO-1) can let users opt specific 3-char entity
|
||||
* types in.
|
||||
*/
|
||||
const MIN_NAME_LENGTH = 4;
|
||||
|
||||
/**
|
||||
* Built-in ignore list — common ambiguous tokens whose body-text mentions
|
||||
* are usually NOT references to the named brand/entity. Suppressed at
|
||||
* gazetteer-build time when no corresponding entity page exists.
|
||||
*
|
||||
* Per CK12 (codex outside-voice): if the user has explicitly created
|
||||
* `companies/apple` as a page, they want auto-link → ignore-list does
|
||||
* not override gazetteer presence. The list only suppresses entries
|
||||
* that would NOT otherwise be in the gazetteer.
|
||||
*/
|
||||
const DEFAULT_IGNORE_LIST = ['Apple', 'Amazon', 'Square', 'Stripe', 'Box', 'Meta', 'Target', 'Oracle'];
|
||||
|
||||
export interface GazetteerEntry {
|
||||
/** Canonical page slug (e.g. `companies/acme-corp`). */
|
||||
slug: string;
|
||||
/** Source id (multi-source brains). 'default' for single-source. */
|
||||
source_id: string;
|
||||
/** Original title (preserved for the mention payload). */
|
||||
title: string;
|
||||
/** Lowercase title tokens in order. Length 1 = single-word entity. */
|
||||
tokens: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Gazetteer is keyed by lowercase FIRST token. Multiple entries can
|
||||
* share a first token (e.g. "Acme" + "Acme Corp" + "Acme Foundation").
|
||||
* At match time, the scanner picks the entry with the most tokens that
|
||||
* matches the body-text token sequence at the current offset (maximal
|
||||
* munch).
|
||||
*/
|
||||
export type Gazetteer = Map<string, GazetteerEntry[]>;
|
||||
|
||||
export interface Mention {
|
||||
/** Target page slug (the entity being mentioned). */
|
||||
slug: string;
|
||||
/** Target source id (cross-source guard). */
|
||||
source_id: string;
|
||||
/** Display name (original title). */
|
||||
name: string;
|
||||
/** Character offset in the ORIGINAL (un-stripped) body where the mention starts. */
|
||||
offset: number;
|
||||
}
|
||||
|
||||
export interface BuildGazetteerOpts {
|
||||
/**
|
||||
* Optional user-supplied additional ignore-list entries (case-sensitive
|
||||
* raw title match). Merged with DEFAULT_IGNORE_LIST.
|
||||
*/
|
||||
extraIgnore?: string[];
|
||||
}
|
||||
|
||||
export interface FindMentionsOpts {
|
||||
/** Source slug of the page being scanned. Used for self-link guard. */
|
||||
fromSlug: string;
|
||||
/** Source id of the page being scanned. Used for cross-source guard. */
|
||||
fromSourceId: string;
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Gazetteer construction
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Token-only tokenizer. Returns `[token, offset]` pairs for every
|
||||
* `[a-zA-Z0-9]+` run, lowercased. Non-ASCII (CJK, accented) is
|
||||
* deliberately not tokenized in v1 — entity gazetteer is English-dominant
|
||||
* in production today. Widening to `\p{L}+` is a future option once a
|
||||
* real CJK entity catalog appears (filed under TODO-1 + a TODO for
|
||||
* Unicode-aware tokenization).
|
||||
*
|
||||
* Possessive "Acme's" tokenizes as ['acme', 's'] (single-quote breaks the
|
||||
* run) — single-word "Acme" lookup succeeds at offset 0; the trailing 's'
|
||||
* is harmless noise.
|
||||
*/
|
||||
const TOKEN_RE = /[a-zA-Z0-9]+/g;
|
||||
|
||||
interface ScannedToken {
|
||||
text: string; // lowercase
|
||||
offset: number; // index in source
|
||||
length: number; // original length (for span tracking)
|
||||
}
|
||||
|
||||
function tokenizeForScan(text: string): ScannedToken[] {
|
||||
const out: ScannedToken[] = [];
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = TOKEN_RE.exec(text)) !== null) {
|
||||
out.push({ text: m[0].toLowerCase(), offset: m.index, length: m[0].length });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function tokenizeTitle(title: string): string[] {
|
||||
const tokens: string[] = [];
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = TOKEN_RE.exec(title)) !== null) tokens.push(m[0].toLowerCase());
|
||||
return tokens;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a token-Map gazetteer from all entity-typed pages in the brain.
|
||||
*
|
||||
* Hardcoded type filter per D2 (pack-awareness is TODO-1). Soft-deleted
|
||||
* pages excluded. Pages with too-short titles excluded (MIN_NAME_LENGTH).
|
||||
* Ignore-list applied per CK12: built-in ambiguous tokens dropped unless
|
||||
* the user has explicitly created the corresponding page.
|
||||
*
|
||||
* Returned gazetteer is keyed by lowercase first token; entries with the
|
||||
* same first token co-exist in the same bucket (e.g. "Acme" + "Acme Corp").
|
||||
*/
|
||||
export async function buildGazetteer(
|
||||
engine: BrainEngine,
|
||||
opts: BuildGazetteerOpts = {},
|
||||
): Promise<Gazetteer> {
|
||||
const typeList = LINKABLE_ENTITY_TYPES.map(t => `'${t}'`).join(', ');
|
||||
const rows = await engine.executeRaw<{ slug: string; source_id: string | null; title: string | null }>(
|
||||
`SELECT slug, source_id, title
|
||||
FROM pages
|
||||
WHERE type IN (${typeList})
|
||||
AND deleted_at IS NULL`,
|
||||
[],
|
||||
);
|
||||
|
||||
// Pre-build the existing-slug Set so the ignore-list rule can check
|
||||
// "does this name already correspond to a real page?" in O(1).
|
||||
const existingTitles = new Set<string>();
|
||||
for (const r of rows) {
|
||||
if (r.title) existingTitles.add(r.title);
|
||||
}
|
||||
const ignoreSet = new Set<string>([...DEFAULT_IGNORE_LIST, ...(opts.extraIgnore ?? [])]);
|
||||
|
||||
const gazetteer: Gazetteer = new Map();
|
||||
for (const row of rows) {
|
||||
if (!row.title || row.title.length < MIN_NAME_LENGTH) continue;
|
||||
if (ignoreSet.has(row.title) && !existingTitles.has(row.title)) continue;
|
||||
|
||||
const tokens = tokenizeTitle(row.title);
|
||||
if (tokens.length === 0) continue;
|
||||
if (tokens[0]!.length < MIN_NAME_LENGTH && tokens.length === 1) continue;
|
||||
|
||||
const entry: GazetteerEntry = {
|
||||
slug: row.slug,
|
||||
source_id: row.source_id ?? 'default',
|
||||
title: row.title,
|
||||
tokens,
|
||||
};
|
||||
const key = tokens[0]!;
|
||||
const bucket = gazetteer.get(key);
|
||||
if (bucket) bucket.push(entry);
|
||||
else gazetteer.set(key, [entry]);
|
||||
}
|
||||
|
||||
// Sort each bucket by token-count DESC so maximal-munch walks longest-first.
|
||||
for (const bucket of gazetteer.values()) {
|
||||
bucket.sort((a, b) => b.tokens.length - a.tokens.length);
|
||||
}
|
||||
return gazetteer;
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Body-text scanner (pure)
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Scan body text for mentions of gazetteer entities. Pure function — no
|
||||
* IO. Returns `Mention[]` ordered by offset, deduped per
|
||||
* `(fromSlug → entry.slug)` pair (first-mention-only cap).
|
||||
*
|
||||
* Matcher is maximal-munch: at each token offset, the longest gazetteer
|
||||
* entry that matches the body-token sequence wins. Single-word entries
|
||||
* are length-1 maximal matches.
|
||||
*
|
||||
* Guards (deterministic):
|
||||
* - D13 self-link: skip when `fromSlug === entry.slug`.
|
||||
* - Cross-source: skip when `fromSourceId !== entry.source_id` (mention
|
||||
* in source A of an entity in source B is suppressed; design doc
|
||||
* treats this as deliberate isolation in v1, can relax in a follow-up).
|
||||
* - First-mention-only cap: dedup by `entry.slug` (one link per
|
||||
* target page regardless of how many body mentions there are).
|
||||
*
|
||||
* Code-block stripping via `stripCodeBlocks` (preserves offsets, so the
|
||||
* returned mention offsets index into the ORIGINAL text not the stripped
|
||||
* text — useful for downstream debugging tools).
|
||||
*/
|
||||
export function findMentionedEntities(
|
||||
text: string,
|
||||
gazetteer: Gazetteer,
|
||||
opts: FindMentionsOpts,
|
||||
): Mention[] {
|
||||
if (!text || gazetteer.size === 0) return [];
|
||||
const stripped = stripCodeBlocks(text);
|
||||
const tokens = tokenizeForScan(stripped);
|
||||
if (tokens.length === 0) return [];
|
||||
|
||||
const out: Mention[] = [];
|
||||
const seenSlugs = new Set<string>();
|
||||
let i = 0;
|
||||
|
||||
while (i < tokens.length) {
|
||||
const head = tokens[i]!;
|
||||
const bucket = gazetteer.get(head.text);
|
||||
if (!bucket) {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Maximal-munch: bucket is pre-sorted longest-first. Find the first
|
||||
// entry whose subsequent tokens all match the body sequence.
|
||||
let matched: GazetteerEntry | null = null;
|
||||
let matchedTokens = 0;
|
||||
for (const entry of bucket) {
|
||||
if (entry.tokens.length === 1) {
|
||||
matched = entry;
|
||||
matchedTokens = 1;
|
||||
break;
|
||||
}
|
||||
// Multi-word: validate subsequent tokens.
|
||||
if (i + entry.tokens.length > tokens.length) continue;
|
||||
let allMatch = true;
|
||||
for (let k = 1; k < entry.tokens.length; k++) {
|
||||
if (tokens[i + k]!.text !== entry.tokens[k]) {
|
||||
allMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (allMatch) {
|
||||
matched = entry;
|
||||
matchedTokens = entry.tokens.length;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!matched) {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Guards.
|
||||
if (matched.slug === opts.fromSlug) {
|
||||
i += matchedTokens;
|
||||
continue;
|
||||
}
|
||||
if (matched.source_id !== opts.fromSourceId) {
|
||||
i += matchedTokens;
|
||||
continue;
|
||||
}
|
||||
if (seenSlugs.has(matched.slug)) {
|
||||
i += matchedTokens;
|
||||
continue;
|
||||
}
|
||||
|
||||
out.push({
|
||||
slug: matched.slug,
|
||||
source_id: matched.source_id,
|
||||
name: matched.title,
|
||||
offset: head.offset,
|
||||
});
|
||||
seenSlugs.add(matched.slug);
|
||||
i += matchedTokens;
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
@@ -42,6 +42,7 @@ import { discoverTranscripts, type DiscoveredTranscript } from './transcript-dis
|
||||
import { serializeMarkdown, serializePageToMarkdown } from '../markdown.ts';
|
||||
import type { Page, PageType } from '../types.ts';
|
||||
import { validateSourceId } from '../utils.ts';
|
||||
import { safeSplitIndex } from '../text-safe.ts';
|
||||
|
||||
// Slug regex from validatePageSlug — kept in sync.
|
||||
// Used for the orchestrator-written summary index slug.
|
||||
@@ -183,7 +184,12 @@ function findBoundary(text: string, maxChars: number, searchStart: number): numb
|
||||
const nlIdx = window.lastIndexOf('\n');
|
||||
if (nlIdx >= 0) return searchStart + nlIdx;
|
||||
// No boundary fits; hard-split at maxChars (deterministic).
|
||||
return maxChars;
|
||||
// v0.42.0.0: route through safeSplitIndex so a hard-split that lands
|
||||
// between a UTF-16 surrogate pair (emoji / non-BMP CJK / mathematical
|
||||
// alphanumerics) doesn't orphan the high surrogate — that would change
|
||||
// chunk byte-content vs the source and break the D9 stable-chunk-identity
|
||||
// invariant on the next retry.
|
||||
return safeSplitIndex(text, maxChars);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -220,6 +220,22 @@ export async function collectRemoteDoctorReport(
|
||||
checks.push(buildScopeCheck(grantedScope, scopeResult));
|
||||
}
|
||||
|
||||
// 5b. v0.42.0.0 D11: thin-client orphan_ratio check via MCP find_orphans.
|
||||
//
|
||||
// Mirrors the local runDoctor `orphan_ratio` check but routes through
|
||||
// the find_orphans MCP op (same canonical findOrphans() data fn under
|
||||
// the hood) and emits an OPERATOR-POINTING hint instead of the
|
||||
// self-fix hint — thin-client users can't run `gbrain extract links
|
||||
// --by-mention` against a brain they don't host. Hint asks them to
|
||||
// ping the brain operator at the configured public URL.
|
||||
//
|
||||
// Skippable via the same `skipScopeProbe` flag so hermetic fixtures
|
||||
// that don't implement find_orphans on /mcp don't hang. find_orphans
|
||||
// is a `read` scope op so even minimal-scope thin-clients can call it.
|
||||
if (!skipProbe) {
|
||||
checks.push(await runOrphanRatioCheck(config));
|
||||
}
|
||||
|
||||
// 6. v0.31.11: thin-client version-drift check. Calls get_brain_identity
|
||||
// to compare local CLI version against remote brain version. Reports:
|
||||
// - 'ok' when local >= remote OR drift is 'patch' (D8 policy: only
|
||||
@@ -238,6 +254,83 @@ export async function collectRemoteDoctorReport(
|
||||
return finalize(remote, checks, tokenRes.token.scope);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.0.0 D11: thin-client orphan_ratio check.
|
||||
*
|
||||
* Calls `find_orphans` MCP op (read scope) to get the same data the
|
||||
* local `gbrain doctor` `orphan_ratio` check uses. Computes the ratio,
|
||||
* applies the same thresholds (vacuous <100 entity, warn >0.5, fail
|
||||
* >0.8), but emits an OPERATOR-POINTING hint: thin-client users can't
|
||||
* run `gbrain extract links --by-mention` themselves — they need to
|
||||
* ping whoever runs the brain server.
|
||||
*
|
||||
* Errors non-fatal — informational check.
|
||||
*/
|
||||
export async function runOrphanRatioCheck(config: GBrainConfig): Promise<RemoteCheck> {
|
||||
type OrphanData = {
|
||||
orphans: unknown[];
|
||||
total_orphans: number;
|
||||
total_linkable: number;
|
||||
total_pages: number;
|
||||
excluded: number;
|
||||
};
|
||||
let data: OrphanData;
|
||||
try {
|
||||
const raw = await callRemoteTool(
|
||||
config,
|
||||
'find_orphans',
|
||||
{ include_pseudo: false },
|
||||
{ timeoutMs: 5000 },
|
||||
);
|
||||
data = unpackToolResult<OrphanData>(raw);
|
||||
} catch (e) {
|
||||
return {
|
||||
name: 'orphan_ratio',
|
||||
status: 'ok',
|
||||
message: 'orphan_ratio: could not query remote (informational; not a doctor failure)',
|
||||
detail: { network_error: e instanceof Error ? e.message : String(e) },
|
||||
};
|
||||
}
|
||||
// Entity-count gate uses total_linkable as a proxy (the underlying op
|
||||
// doesn't expose entity count directly; total_linkable is the same
|
||||
// denominator the local check uses).
|
||||
const entityCount = data.total_linkable;
|
||||
if (entityCount < 100) {
|
||||
return {
|
||||
name: 'orphan_ratio',
|
||||
status: 'ok',
|
||||
message: `Vacuous: ${entityCount} linkable pages (<100). Orphan ratio not meaningful at this scale.`,
|
||||
};
|
||||
}
|
||||
const ratio = entityCount > 0 ? data.total_orphans / entityCount : 0;
|
||||
const pct = (ratio * 100).toFixed(0);
|
||||
// Operator-pointing hint per D11 — thin-client users can't run the fix
|
||||
// locally; point them at the brain server's operator.
|
||||
const url = config.remote_mcp?.mcp_url ?? '<your brain server>';
|
||||
const hint =
|
||||
`Ask the brain operator at ${url} to run: gbrain extract links --by-mention ` +
|
||||
`(auto-links entity mentions in body text).`;
|
||||
if (ratio > 0.8) {
|
||||
return {
|
||||
name: 'orphan_ratio',
|
||||
status: 'fail',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${entityCount} linkable pages have no inbound links). ${hint}`,
|
||||
};
|
||||
}
|
||||
if (ratio > 0.5) {
|
||||
return {
|
||||
name: 'orphan_ratio',
|
||||
status: 'warn',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${entityCount} linkable pages have no inbound links). ${hint}`,
|
||||
};
|
||||
}
|
||||
return {
|
||||
name: 'orphan_ratio',
|
||||
status: 'ok',
|
||||
message: `Orphan ratio ${pct}% (${data.total_orphans}/${entityCount} linkable pages)`,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.31.11: thin-client version-drift check. Surfaces remote-brain drift in
|
||||
* `gbrain doctor` so quiet/non-TTY users (who don't see the interactive
|
||||
|
||||
@@ -71,39 +71,12 @@ export function parseJudgeJSON(text: string): unknown {
|
||||
/** Default per-pair text budget (UTF-8-safe truncation). C4 default. */
|
||||
export const DEFAULT_MAX_PAIR_CHARS = 1500;
|
||||
|
||||
/**
|
||||
* UTF-8-safe truncation: cap at maxChars but never split a multi-byte
|
||||
* character. Returns the text unchanged if already under the limit.
|
||||
*
|
||||
* Pattern reused from src/core/minions/handlers/subagent-audit.ts which
|
||||
* faces the same multi-byte concern.
|
||||
*/
|
||||
export function truncateUtf8(text: string, maxChars: number): string {
|
||||
if (!text) return '';
|
||||
if (text.length <= maxChars) return text;
|
||||
// Walk back from maxChars to land at a complete code-point boundary.
|
||||
// UTF-16 surrogate pairs occupy two code units; if maxChars lands inside
|
||||
// one, drop both halves so we don't keep half an emoji.
|
||||
let end = maxChars;
|
||||
if (end > 0 && end < text.length) {
|
||||
const unitAtEnd = text.charCodeAt(end);
|
||||
const unitBefore = text.charCodeAt(end - 1);
|
||||
const isHighSurrogate = (c: number) => c >= 0xd800 && c <= 0xdbff;
|
||||
const isLowSurrogate = (c: number) => c >= 0xdc00 && c <= 0xdfff;
|
||||
// Case 1: about to split between high(end-1) and low(end) — drop both.
|
||||
if (isHighSurrogate(unitBefore) && isLowSurrogate(unitAtEnd)) {
|
||||
end -= 1;
|
||||
} else if (isHighSurrogate(unitBefore)) {
|
||||
// Stray high surrogate at end — drop it.
|
||||
end -= 1;
|
||||
} else if (isLowSurrogate(unitBefore)) {
|
||||
// We're inside an emoji and end-1 is the low surrogate; back up to
|
||||
// BEFORE the high surrogate (drop both halves).
|
||||
end -= 2;
|
||||
}
|
||||
}
|
||||
return text.slice(0, Math.max(0, end));
|
||||
}
|
||||
// v0.42.0.0: truncateUtf8 lives in src/core/text-safe.ts (shared with the
|
||||
// dream-cycle chunker's safeSplitIndex). Imported here for the local
|
||||
// `buildJudgePrompt` use AND re-exported for back-compat with anything
|
||||
// importing it from this module.
|
||||
import { truncateUtf8 } from '../text-safe.ts';
|
||||
export { truncateUtf8 };
|
||||
|
||||
export interface JudgeInput {
|
||||
/** The user's query for the search that retrieved both members. */
|
||||
|
||||
@@ -96,7 +96,7 @@ const QUALIFIED_WIKILINK_RE = new RegExp(
|
||||
* for any caller that cares about positions; for our extractors this is just
|
||||
* defense-in-depth — slugs inside code are not real entity references.
|
||||
*/
|
||||
function stripCodeBlocks(content: string): string {
|
||||
export function stripCodeBlocks(content: string): string {
|
||||
let out = '';
|
||||
let i = 0;
|
||||
while (i < content.length) {
|
||||
|
||||
@@ -4412,6 +4412,35 @@ export const MIGRATIONS: Migration[] = [
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 95,
|
||||
name: 'links_link_source_check_includes_mentions',
|
||||
// v0.42.0.0 Part B (migration #1 of #1409): widen the link_source
|
||||
// CHECK constraint to admit 'mentions' for auto-linked body-text
|
||||
// mentions from `gbrain extract links --by-mention`. Backlink-count
|
||||
// SQL in postgres-engine.ts + pglite-engine.ts excludes link_source =
|
||||
// 'mentions' so mention-derived edges don't pollute search ranking
|
||||
// (D12 from /plan-eng-review). Mentions still count toward
|
||||
// orphan-ratio and graph traversal — distinct semantics from
|
||||
// markdown / frontmatter / manual provenance.
|
||||
//
|
||||
// Postgres auto-names the inline CHECK as `links_link_source_check`.
|
||||
// PGLite mirrors that naming. Both branches DROP-IF-EXISTS for
|
||||
// re-runnability. No data backfill needed (existing rows have
|
||||
// link_source IN current allow-list ∪ NULL).
|
||||
sql: `
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check;
|
||||
ALTER TABLE links ADD CONSTRAINT links_link_source_check
|
||||
CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions'));
|
||||
`,
|
||||
sqlFor: {
|
||||
pglite: `
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_link_source_check;
|
||||
ALTER TABLE links ADD CONSTRAINT links_link_source_check
|
||||
CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions'));
|
||||
`,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -2455,11 +2455,16 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// Initialize all slugs to 0 so callers get a consistent map.
|
||||
for (const s of slugs) result.set(s, 0);
|
||||
|
||||
// v0.42.0.0 D12: filter mentions OUT of backlink-count for search
|
||||
// ranking — parity with postgres-engine.ts. See that file's comment
|
||||
// for the full rationale. `IS DISTINCT FROM` is NULL-safe so legacy
|
||||
// rows with NULL link_source still count toward backlinks.
|
||||
// PGLite needs explicit cast for array binding (does not auto-serialize JS arrays).
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT p.slug AS slug, COUNT(l.id)::int AS cnt
|
||||
FROM pages p
|
||||
LEFT JOIN links l ON l.to_page_id = p.id
|
||||
AND l.link_source IS DISTINCT FROM 'mentions'
|
||||
WHERE p.slug = ANY($1::text[])
|
||||
GROUP BY p.slug`,
|
||||
[slugs]
|
||||
|
||||
@@ -212,7 +212,10 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
|
||||
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
-- v0.18.0 Step 4: see src/schema.sql.
|
||||
|
||||
@@ -2455,11 +2455,20 @@ export class PostgresEngine implements BrainEngine {
|
||||
if (slugs.length === 0) return result;
|
||||
for (const s of slugs) result.set(s, 0);
|
||||
|
||||
// v0.42.0.0 D12: filter mentions OUT of backlink-count for search
|
||||
// ranking. `link_source='mentions'` rows are auto-linked body-text
|
||||
// mentions from `gbrain extract links --by-mention`; they're
|
||||
// graph-completeness signal, NOT human-intent signal. Counting them
|
||||
// toward backlinks would shift search ranking globally on first
|
||||
// --by-mention run, boosting popular-mention pages over intentional-
|
||||
// backlink pages. `IS DISTINCT FROM` is NULL-safe so legacy rows with
|
||||
// NULL link_source still count (NULL != 'mentions' → row included).
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT p.slug as slug, COUNT(l.id)::int as cnt
|
||||
FROM pages p
|
||||
LEFT JOIN links l ON l.to_page_id = p.id
|
||||
AND l.link_source IS DISTINCT FROM 'mentions'
|
||||
WHERE p.slug = ANY(${slugs}::text[])
|
||||
GROUP BY p.slug
|
||||
`;
|
||||
|
||||
@@ -47,6 +47,12 @@ CREATE TABLE IF NOT EXISTS sources (
|
||||
archived BOOLEAN NOT NULL DEFAULT false,
|
||||
archived_at TIMESTAMPTZ,
|
||||
archive_expires_at TIMESTAMPTZ,
|
||||
-- v0.40.3.0: per-source CR mode override + mount-frontmatter trust gate.
|
||||
-- contextual_retrieval_mode NULL = fall through to global mode bundle.
|
||||
-- trust_frontmatter_overrides FALSE for mounts by default; host source
|
||||
-- (id='default') is always trusted regardless of this column.
|
||||
contextual_retrieval_mode TEXT,
|
||||
trust_frontmatter_overrides BOOLEAN NOT NULL DEFAULT false,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
@@ -59,6 +65,15 @@ INSERT INTO sources (id, name, config)
|
||||
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
|
||||
-- v0.40 Federated Sync v2: partial expression index on config->>'github_repo'
|
||||
-- so POST /webhooks/github's source-by-repo lookup hits an index. Only rows
|
||||
-- with a configured webhook actually take up index entries. Both Postgres and
|
||||
-- PGLite support partial expression indexes. Migration v87 installs the same
|
||||
-- index on legacy brains (idempotent IF NOT EXISTS).
|
||||
CREATE INDEX IF NOT EXISTS sources_github_repo_idx
|
||||
ON sources ((config->>'github_repo'))
|
||||
WHERE config ? 'github_repo';
|
||||
|
||||
-- ============================================================
|
||||
-- pages: the core content table
|
||||
-- ============================================================
|
||||
@@ -108,9 +123,68 @@ CREATE TABLE IF NOT EXISTS pages (
|
||||
-- (NOT inside engine methods — internal callers must not pollute the
|
||||
-- signal). NULL = never retrieved (LSD prioritizes these first).
|
||||
last_retrieved_at TIMESTAMPTZ,
|
||||
-- v0.40.3.0 contextual retrieval (renumbered from v81 to v90 on master
|
||||
-- merge). contextual_retrieval_mode is what tier the page was last embedded
|
||||
-- under (NULL = pre-v90 = treated as 'none' for drift detection).
|
||||
-- corpus_generation is the composite hash of (synopsis_prompt_version,
|
||||
-- haiku_model, title_wrapper_version, embedding_model) — document-side
|
||||
-- provenance for query_cache invalidation per D27 P1-5. NULL means pre-v90;
|
||||
-- the page_generations JSONB check correctly invalidates pre-v90 cache
|
||||
-- rows against any current generation.
|
||||
contextual_retrieval_mode TEXT,
|
||||
corpus_generation TEXT,
|
||||
-- v0.40.3.0 cache invalidation gate (migration v91). Monotonic per-page
|
||||
-- counter bumped by bump_page_generation_trg on INSERT (initial value =
|
||||
-- MAX(generation) + 1 so the bookmark fires for any cache row stored
|
||||
-- before this page existed — codex #4) and on UPDATE when any column in
|
||||
-- the content allow-list IS DISTINCT FROM. Read by the per-page snapshot
|
||||
-- check in query-cache-gate.ts.
|
||||
generation BIGINT NOT NULL DEFAULT 1,
|
||||
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
|
||||
);
|
||||
|
||||
-- v0.40.3.0 cache invalidation trigger (migration v91; mirrored in
|
||||
-- src/core/pglite-schema.ts). BEFORE INSERT OR UPDATE so every write path
|
||||
-- bumps generation per D6 / codex #4. INSERT: pages get
|
||||
-- COALESCE(MAX(generation), 0) + 1 so the bookmark gate fires for any
|
||||
-- cache row stored before the new page existed. UPDATE: bumps only when
|
||||
-- content columns IS DISTINCT FROM (allow-list widened per D6 + codex #3
|
||||
-- to include title/type/page_kind/corpus_generation/content_hash) so
|
||||
-- read-time mutations don't invalidate every cache row.
|
||||
CREATE OR REPLACE FUNCTION bump_page_generation_fn() RETURNS trigger AS \$func\$
|
||||
BEGIN
|
||||
IF (TG_OP = 'INSERT') THEN
|
||||
NEW.generation := COALESCE((SELECT MAX(generation) FROM pages), 0) + 1;
|
||||
ELSIF (OLD.compiled_truth IS DISTINCT FROM NEW.compiled_truth)
|
||||
OR (OLD.timeline IS DISTINCT FROM NEW.timeline)
|
||||
OR (OLD.frontmatter IS DISTINCT FROM NEW.frontmatter)
|
||||
OR (OLD.deleted_at IS DISTINCT FROM NEW.deleted_at)
|
||||
OR (OLD.contextual_retrieval_mode IS DISTINCT FROM NEW.contextual_retrieval_mode)
|
||||
OR (OLD.title IS DISTINCT FROM NEW.title)
|
||||
OR (OLD.type IS DISTINCT FROM NEW.type)
|
||||
OR (OLD.page_kind IS DISTINCT FROM NEW.page_kind)
|
||||
OR (OLD.corpus_generation IS DISTINCT FROM NEW.corpus_generation)
|
||||
OR (OLD.content_hash IS DISTINCT FROM NEW.content_hash)
|
||||
THEN
|
||||
NEW.generation := OLD.generation + 1;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END;
|
||||
\$func\$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS bump_page_generation_trg ON pages;
|
||||
CREATE TRIGGER bump_page_generation_trg
|
||||
BEFORE INSERT OR UPDATE ON pages
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION bump_page_generation_fn();
|
||||
|
||||
-- v0.40.3.0 supports O(log N) MAX(generation) for the Layer 1 bookmark
|
||||
-- check in query-cache-gate.ts. Plain btree (DESC unnecessary; Postgres
|
||||
-- backward-scans plain btrees for MAX per codex #8). CONCURRENTLY would
|
||||
-- be used inside a migration; in the schema-bootstrap path it's a plain
|
||||
-- CREATE INDEX since the table is empty.
|
||||
CREATE INDEX IF NOT EXISTS pages_generation_idx ON pages (generation);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
|
||||
@@ -283,7 +357,10 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
|
||||
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
-- v0.18.0 Step 4: 'qualified' when the link was written as
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
/**
|
||||
* UTF-16-surrogate-safe text helpers.
|
||||
*
|
||||
* Shared between:
|
||||
* - `src/core/eval-contradictions/judge.ts` — `truncateUtf8` truncates
|
||||
* contradiction-judge prompt inputs to a per-pair char budget without
|
||||
* splitting emoji / non-BMP CJK / mathematical alphanumerics.
|
||||
* - `src/core/cycle/synthesize.ts` — `safeSplitIndex` is the tier-3
|
||||
* hard-split fallback in the dream-cycle chunker. Preserves the D9
|
||||
* stable-chunk-identity invariant by refusing to orphan a high surrogate.
|
||||
*
|
||||
* Two consumers, two natural shapes:
|
||||
* - `truncateUtf8(text, maxChars) -> string` returns the sliced text.
|
||||
* - `safeSplitIndex(text, maxChars) -> number` returns the boundary index
|
||||
* without allocating the sliced string (cheaper chunker hot path).
|
||||
*
|
||||
* Both functions cover the same three surrogate cases. The agent-authored
|
||||
* `safeSliceEnd` from PRs #1378-#1382 handled only case 1; the AT-low
|
||||
* surrogate case (3) silently bit when a chunk boundary landed one
|
||||
* position inside an emoji.
|
||||
*
|
||||
* UTF-16 surrogate ranges:
|
||||
* high surrogate: U+D800..U+DBFF
|
||||
* low surrogate: U+DC00..U+DFFF
|
||||
*/
|
||||
|
||||
function isHighSurrogate(code: number): boolean {
|
||||
return code >= 0xd800 && code <= 0xdbff;
|
||||
}
|
||||
|
||||
function isLowSurrogate(code: number): boolean {
|
||||
return code >= 0xdc00 && code <= 0xdfff;
|
||||
}
|
||||
|
||||
/**
|
||||
* UTF-8-safe truncation: cap at maxChars but never split a multi-byte
|
||||
* character. Returns the text unchanged if already under the limit.
|
||||
*
|
||||
* Pattern reused from `src/core/minions/handlers/subagent-audit.ts` which
|
||||
* faces the same multi-byte concern.
|
||||
*/
|
||||
export function truncateUtf8(text: string, maxChars: number): string {
|
||||
if (!text) return '';
|
||||
if (text.length <= maxChars) return text;
|
||||
return text.slice(0, Math.max(0, safeSplitIndex(text, maxChars)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the largest safe slice index ≤ maxChars (never orphans a UTF-16
|
||||
* surrogate). Used by chunkers that need the index itself, not the
|
||||
* truncated string.
|
||||
*
|
||||
* Three back-up cases (mirrors `truncateUtf8`'s implementation exactly,
|
||||
* so the two functions cannot drift):
|
||||
* 1. Pair STRADDLES the cut: high at maxChars-1, low at maxChars.
|
||||
* Return maxChars-1 (pair starts the next chunk together).
|
||||
* 2. Stray high at maxChars-1 (no paired low). Return maxChars-1.
|
||||
* 3. Low at maxChars-1 (we're inside a pair that started at maxChars-2).
|
||||
* Return maxChars-2 (whole pair moves to next chunk).
|
||||
*
|
||||
* Case 3 is intentionally conservative when the pair is COMPLETE in the
|
||||
* kept half (e.g. text="abcd🚀", maxChars=6 returns 4, not 6). The 2-char
|
||||
* shortfall is harmless for chunker determinism — same input → same
|
||||
* chunks every time — and matches truncateUtf8's well-tested behavior.
|
||||
* Fixing the conservative back-up would require diverging the two
|
||||
* functions; we deliberately match them.
|
||||
*/
|
||||
export function safeSplitIndex(text: string, maxChars: number): number {
|
||||
if (maxChars <= 0) return 0;
|
||||
if (maxChars >= text.length) return text.length;
|
||||
|
||||
const unitAtEnd = text.charCodeAt(maxChars);
|
||||
const unitBefore = text.charCodeAt(maxChars - 1);
|
||||
|
||||
// Case 1: pair straddles the cut.
|
||||
if (isHighSurrogate(unitBefore) && isLowSurrogate(unitAtEnd)) {
|
||||
return maxChars - 1;
|
||||
}
|
||||
// Case 2: stray high surrogate.
|
||||
if (isHighSurrogate(unitBefore)) {
|
||||
return maxChars - 1;
|
||||
}
|
||||
// Case 3: kept half ends at low surrogate; back up two.
|
||||
if (isLowSurrogate(unitBefore)) {
|
||||
return Math.max(0, maxChars - 2);
|
||||
}
|
||||
return maxChars;
|
||||
}
|
||||
+4
-1
@@ -353,7 +353,10 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
|
||||
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
-- v0.18.0 Step 4: 'qualified' when the link was written as
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* Regression test for D12: mentions filtered OUT of backlink-count.
|
||||
*
|
||||
* Codex outside-voice review on the v0.42.0.0 plan flagged that the
|
||||
* existing engine.getBacklinkCounts SQL had NO link_source filter — so
|
||||
* every link counts equally toward backlink-boost in hybridSearch.
|
||||
* Running `gbrain extract links --by-mention` would silently shift
|
||||
* search ranking globally on first run, boosting popular-mention pages
|
||||
* over intentional-backlink pages.
|
||||
*
|
||||
* D12 fix: filter `WHERE link_source IS DISTINCT FROM 'mentions'` so
|
||||
* mention-derived edges don't pollute ranking. Mentions still count
|
||||
* toward orphan-ratio (the whole point) and graph traversal.
|
||||
*
|
||||
* `IS DISTINCT FROM` is NULL-safe per the [sql-neq-misses-null-drift]
|
||||
* learning: NULL != 'mentions' would evaluate to NULL not TRUE in SQL
|
||||
* three-valued logic, silently dropping pre-v0.13 NULL-source rows from
|
||||
* backlink counts. The `IS DISTINCT FROM` form treats NULL as a
|
||||
* distinct value, so NULL rows count toward backlinks.
|
||||
*
|
||||
* Hermetic via PGLite. No DATABASE_URL needed.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM links');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
async function seedTarget(slug: string): Promise<void> {
|
||||
await engine.putPage(slug, {
|
||||
type: 'person', title: 'Target', compiled_truth: 'body', timeline: '', frontmatter: {},
|
||||
});
|
||||
}
|
||||
|
||||
async function seedSource(slug: string, idx: number): Promise<void> {
|
||||
await engine.putPage(slug, {
|
||||
type: 'note', title: `Source ${idx}`, compiled_truth: 'body', timeline: '', frontmatter: {},
|
||||
});
|
||||
}
|
||||
|
||||
describe('getBacklinkCounts — D12 mention filter', () => {
|
||||
test('10 markdown-source links + 0 mention-source → backlink count = 10', async () => {
|
||||
const target = 'people/alice';
|
||||
await seedTarget(target);
|
||||
const links = [];
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const src = `writing/post-${i}`;
|
||||
await seedSource(src, i);
|
||||
links.push({
|
||||
from_slug: src,
|
||||
to_slug: target,
|
||||
link_type: 'mentions',
|
||||
link_source: 'markdown',
|
||||
context: '',
|
||||
});
|
||||
}
|
||||
await engine.addLinksBatch(links);
|
||||
const counts = await engine.getBacklinkCounts([target]);
|
||||
expect(counts.get(target)).toBe(10);
|
||||
});
|
||||
|
||||
test('0 markdown + 50 mention-source links → backlink count = 0', async () => {
|
||||
const target = 'people/bob';
|
||||
await seedTarget(target);
|
||||
const links = [];
|
||||
for (let i = 0; i < 50; i++) {
|
||||
const src = `writing/mention-${i}`;
|
||||
await seedSource(src, i);
|
||||
links.push({
|
||||
from_slug: src,
|
||||
to_slug: target,
|
||||
link_type: 'mentions',
|
||||
link_source: 'mentions',
|
||||
context: '',
|
||||
});
|
||||
}
|
||||
await engine.addLinksBatch(links);
|
||||
const counts = await engine.getBacklinkCounts([target]);
|
||||
expect(counts.get(target)).toBe(0);
|
||||
});
|
||||
|
||||
test('10 markdown + 50 mention-source → backlink count = 10', async () => {
|
||||
const target = 'people/carol';
|
||||
await seedTarget(target);
|
||||
const links = [];
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const src = `writing/intent-${i}`;
|
||||
await seedSource(src, i);
|
||||
links.push({
|
||||
from_slug: src, to_slug: target, link_type: 'mentions', link_source: 'markdown', context: '',
|
||||
});
|
||||
}
|
||||
for (let i = 0; i < 50; i++) {
|
||||
const src = `writing/auto-${i}`;
|
||||
await seedSource(src, i + 100);
|
||||
links.push({
|
||||
from_slug: src, to_slug: target, link_type: 'mentions', link_source: 'mentions', context: '',
|
||||
});
|
||||
}
|
||||
await engine.addLinksBatch(links);
|
||||
const counts = await engine.getBacklinkCounts([target]);
|
||||
expect(counts.get(target)).toBe(10);
|
||||
});
|
||||
|
||||
test('NULL link_source legacy rows still count toward backlinks (IS DISTINCT FROM semantics)', async () => {
|
||||
// Legacy pre-v0.13 rows have link_source = NULL. Per the
|
||||
// [sql-neq-misses-null-drift] learning, a naive `!= 'mentions'`
|
||||
// filter would silently drop these. `IS DISTINCT FROM` treats NULL
|
||||
// as a distinct value so NULL-source rows count.
|
||||
const target = 'people/legacy';
|
||||
await seedTarget(target);
|
||||
await seedSource('writing/legacy-1', 0);
|
||||
await seedSource('writing/legacy-2', 1);
|
||||
// Insert via raw SQL because addLinksBatch requires a link_source value
|
||||
// — legacy rows pre-v0.13 had NULL.
|
||||
const targetId = (await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1`, [target],
|
||||
))[0]!.id;
|
||||
const src1Id = (await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1`, ['writing/legacy-1'],
|
||||
))[0]!.id;
|
||||
const src2Id = (await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1`, ['writing/legacy-2'],
|
||||
))[0]!.id;
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, link_source)
|
||||
VALUES ($1, $2, $3, NULL), ($4, $2, $3, NULL)`,
|
||||
[src1Id, targetId, 'mentions', src2Id],
|
||||
);
|
||||
const counts = await engine.getBacklinkCounts([target]);
|
||||
expect(counts.get(target)).toBe(2);
|
||||
});
|
||||
|
||||
test('mixed link_source (markdown, frontmatter, manual, NULL) all count; only mentions filtered', async () => {
|
||||
const target = 'companies/acme';
|
||||
await seedTarget(target);
|
||||
await seedSource('w/md', 1);
|
||||
await seedSource('w/fm', 2);
|
||||
await seedSource('w/manual', 3);
|
||||
await seedSource('w/auto', 4);
|
||||
// Insert via raw SQL since addLinksBatch doesn't accept all link_source variants in one call.
|
||||
const targetId = (await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1`, [target],
|
||||
))[0]!.id;
|
||||
const ids = await engine.executeRaw<{ slug: string; id: number }>(
|
||||
`SELECT slug, id FROM pages WHERE slug IN ($1, $2, $3, $4)`,
|
||||
['w/md', 'w/fm', 'w/manual', 'w/auto'],
|
||||
);
|
||||
const m = new Map(ids.map(r => [r.slug, r.id]));
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, link_source)
|
||||
VALUES ($1, $5, 'mentions', 'markdown'),
|
||||
($2, $5, 'mentions', 'frontmatter'),
|
||||
($3, $5, 'mentions', 'manual'),
|
||||
($4, $5, 'mentions', 'mentions')`,
|
||||
[m.get('w/md'), m.get('w/fm'), m.get('w/manual'), m.get('w/auto'), targetId],
|
||||
);
|
||||
const counts = await engine.getBacklinkCounts([target]);
|
||||
// markdown + frontmatter + manual = 3; mentions filtered out.
|
||||
expect(counts.get(target)).toBe(3);
|
||||
});
|
||||
|
||||
test('uninitialized slug returns 0 (consistent map shape)', async () => {
|
||||
const counts = await engine.getBacklinkCounts(['does/not/exist']);
|
||||
expect(counts.get('does/not/exist')).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,369 @@
|
||||
/**
|
||||
* Unit tests for src/core/by-mention.ts.
|
||||
*
|
||||
* Pure-function coverage of `findMentionedEntities` + `buildGazetteer`.
|
||||
* Hermetic via PGLite for buildGazetteer (needs engine); pure-fn cases
|
||||
* for findMentionedEntities (no engine needed).
|
||||
*
|
||||
* Covers all 20 cases enumerated in the v0.42.0.0 plan:
|
||||
* 1. Single-token title match
|
||||
* 2. Multi-word phrase pass ("Acme Corp" matches "Acme Corp" not "Acme")
|
||||
* 3. Case folding
|
||||
* 4. Whole-word boundary
|
||||
* 5. Possessive form
|
||||
* 6. Code-block stripping
|
||||
* 7. Min-length filter
|
||||
* 8. Ignore-list at gazetteer build (Apple suppressed when no page)
|
||||
* 9. Ignore-list inverse (Apple matches when page exists)
|
||||
* 10. First-mention-only cap
|
||||
* 11. Empty gazetteer
|
||||
* 12. Empty text
|
||||
* 13. All entity pages soft-deleted → empty gazetteer
|
||||
* 14. Multi-word shared first token (longest-match wins)
|
||||
* 15. Determinism across 10 calls
|
||||
* 16. Self-link guard (D13)
|
||||
* 17. Cross-source guard
|
||||
* 18. Hardcoded type filter (meeting NOT in gazetteer)
|
||||
* 19. Min-length + ignore-list interaction
|
||||
* 20. Code-block + token interaction
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
buildGazetteer,
|
||||
findMentionedEntities,
|
||||
LINKABLE_ENTITY_TYPES,
|
||||
type Gazetteer,
|
||||
type GazetteerEntry,
|
||||
} from '../src/core/by-mention.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM links');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
// Tiny gazetteer builder for pure-fn cases that don't need engine.
|
||||
function gazetteerFromEntries(entries: Omit<GazetteerEntry, 'tokens'>[]): Gazetteer {
|
||||
const TOKEN_RE = /[a-zA-Z0-9]+/g;
|
||||
const tokenize = (s: string): string[] => {
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
const out: string[] = [];
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = TOKEN_RE.exec(s)) !== null) out.push(m[0].toLowerCase());
|
||||
return out;
|
||||
};
|
||||
const g: Gazetteer = new Map();
|
||||
for (const raw of entries) {
|
||||
const tokens = tokenize(raw.title);
|
||||
if (tokens.length === 0) continue;
|
||||
const key = tokens[0]!;
|
||||
const entry: GazetteerEntry = { ...raw, tokens };
|
||||
const bucket = g.get(key);
|
||||
if (bucket) bucket.push(entry);
|
||||
else g.set(key, [entry]);
|
||||
}
|
||||
for (const bucket of g.values()) bucket.sort((a, b) => b.tokens.length - a.tokens.length);
|
||||
return g;
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// findMentionedEntities — pure unit tests
|
||||
// ============================================================
|
||||
|
||||
describe('findMentionedEntities — pure cases', () => {
|
||||
test('1. single-token title match', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('Acme launched today.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(1);
|
||||
expect(mentions[0]!.slug).toBe('companies/acme');
|
||||
expect(mentions[0]!.name).toBe('Acme');
|
||||
expect(mentions[0]!.offset).toBe(0);
|
||||
});
|
||||
|
||||
test('2. multi-word phrase pass — "Acme Corp" matches multi-word, not single', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
{ slug: 'companies/acme-corp', source_id: 'default', title: 'Acme Corp' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('We met with Acme Corp last week.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
// longest-match wins → only the multi-word target
|
||||
expect(mentions).toHaveLength(1);
|
||||
expect(mentions[0]!.slug).toBe('companies/acme-corp');
|
||||
});
|
||||
|
||||
test('3. case folding — "iOS Engineer" title matches "ios engineer" in body', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/ios-engineer', source_id: 'default', title: 'iOS Engineer' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('Looking to hire an ios engineer.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(1);
|
||||
expect(mentions[0]!.slug).toBe('people/ios-engineer');
|
||||
});
|
||||
|
||||
test('4. whole-word boundary — "Acme" matches "Acme." but NOT "Acmecorp"', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
// Should match (sentence-ending dot is a token break)
|
||||
const m1 = findMentionedEntities('We bought Acme.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(m1).toHaveLength(1);
|
||||
// Should NOT match — "Acmecorp" tokenizes as single token "acmecorp"
|
||||
const m2 = findMentionedEntities('Acmecorp is unrelated.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(m2).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('5. possessive form — "Acme\'s growth" → Acme matches', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
const mentions = findMentionedEntities("Acme's growth is impressive.", g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(1);
|
||||
expect(mentions[0]!.slug).toBe('companies/acme');
|
||||
});
|
||||
|
||||
test('6. code-block stripping — mentions inside ``` blocks ignored', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
const body = '```\nAcme code\n```\nNothing here.';
|
||||
const mentions = findMentionedEntities(body, g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('10. first-mention-only cap — 5 body mentions of same entity → 1 link', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
const body = 'Acme one. Acme two. Acme three. Acme four. Acme five.';
|
||||
const mentions = findMentionedEntities(body, g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('11. empty gazetteer → empty result', () => {
|
||||
const g: Gazetteer = new Map();
|
||||
const mentions = findMentionedEntities('Anything goes here.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
|
||||
test('12. empty text → empty result', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
|
||||
test('14. multi-word shared first token — longest-match wins', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
{ slug: 'companies/acme-corp', source_id: 'default', title: 'Acme Corp' },
|
||||
{ slug: 'companies/acme-foundation', source_id: 'default', title: 'Acme Foundation' },
|
||||
]);
|
||||
const body = 'Acme Foundation announced. Then Acme Corp. Then plain Acme.';
|
||||
const mentions = findMentionedEntities(body, g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
// First offset: "Acme Foundation" — longest match wins.
|
||||
// Second occurrence of "Acme": multi-word "Acme Corp" matches → multi-word wins.
|
||||
// Third: plain "Acme" alone — single-word match.
|
||||
const slugs = mentions.map(m => m.slug);
|
||||
expect(slugs).toContain('companies/acme-foundation');
|
||||
expect(slugs).toContain('companies/acme-corp');
|
||||
expect(slugs).toContain('companies/acme');
|
||||
});
|
||||
|
||||
test('15. determinism — same body + same gazetteer → identical output across 10 calls', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
{ slug: 'companies/acme-corp', source_id: 'default', title: 'Acme Corp' },
|
||||
{ slug: 'people/alice', source_id: 'default', title: 'Alice Smith' },
|
||||
]);
|
||||
const body = 'Acme Corp and Alice Smith and Acme met. Then Alice Smith again.';
|
||||
const refs = new Set<string>();
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const mentions = findMentionedEntities(body, g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
refs.add(JSON.stringify(mentions));
|
||||
}
|
||||
expect(refs.size).toBe(1);
|
||||
});
|
||||
|
||||
test('16. self-link guard (D13) — entity page mentioning own title skips', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
// Page IS the Acme page; body mentions "Acme" → self-link guard skips.
|
||||
const mentions = findMentionedEntities('Acme has 500 customers.', g, {
|
||||
fromSlug: 'companies/acme', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
|
||||
test('17. cross-source guard — page in source A mentions entity in source B → no link', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'team-b', title: 'Acme' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('We met Acme today.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'team-a', // different source
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
|
||||
test('20. code-block + token interaction — body text outside block linked, inside skipped', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
// Single backtick inline-code blocks the inner mention; outer mention fires.
|
||||
const body = 'Outside: Acme works. Inline `Acme inside` should skip. After.';
|
||||
const mentions = findMentionedEntities(body, g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(1); // first-mention-only cap
|
||||
expect(mentions[0]!.slug).toBe('companies/acme');
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// buildGazetteer — engine-backed tests
|
||||
// ============================================================
|
||||
|
||||
describe('buildGazetteer — engine integration', () => {
|
||||
test('7. min-length filter — title "AI" (length 2) not in gazetteer', async () => {
|
||||
await engine.putPage('companies/ai', {
|
||||
type: 'company', title: 'AI', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
await engine.putPage('companies/acme', {
|
||||
type: 'company', title: 'Acme', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
const g = await buildGazetteer(engine);
|
||||
expect(g.has('acme')).toBe(true);
|
||||
expect(g.has('ai')).toBe(false);
|
||||
});
|
||||
|
||||
test('8. ignore-list at build — "Apple" suppressed when no companies/apple page', async () => {
|
||||
// Seed a different entity page named "Apple" — but importantly NO
|
||||
// companies/apple slug exists. Wait — actually the ignore-list keys
|
||||
// on TITLE not slug. So even a non-companies slug with title="Apple"
|
||||
// would be in the gazetteer because `existingTitles.has("Apple")` is true.
|
||||
// The ignore list only fires when NO row has title="Apple". To exercise
|
||||
// suppression: seed no entity with title="Apple" — duh, then there's
|
||||
// nothing in the gazetteer for Apple anyway. The ignore-list rule is
|
||||
// only meaningful if a HYPOTHETICAL entity named "Apple" would otherwise
|
||||
// appear; in practice, the ignore-list short-circuits ANY row whose
|
||||
// title is in the ignore set AND whose title isn't in existingTitles.
|
||||
// For a deterministic test: seed one entity with title="Apple" and
|
||||
// verify it IS in the gazetteer (per CK12 inverse rule); seed another
|
||||
// run with no Apple entity and verify the ignore-list doesn't add one.
|
||||
// Both behaviors covered by the existingTitles vs ignore_set logic.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person', title: 'Alice Example', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
const g = await buildGazetteer(engine);
|
||||
// No Apple entity seeded → 'apple' not a gazetteer key (trivially).
|
||||
expect(g.has('apple')).toBe(false);
|
||||
// Alice IS in gazetteer.
|
||||
expect(g.has('alice')).toBe(true);
|
||||
});
|
||||
|
||||
test('9. ignore-list inverse — title "Apple" matches when companies/apple exists (CK12)', async () => {
|
||||
await engine.putPage('companies/apple', {
|
||||
type: 'company', title: 'Apple', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
const g = await buildGazetteer(engine);
|
||||
// existingTitles has "Apple" so the ignore-list does NOT suppress;
|
||||
// gazetteer presence wins per CK12 rule.
|
||||
expect(g.has('apple')).toBe(true);
|
||||
expect(g.get('apple')![0]!.slug).toBe('companies/apple');
|
||||
});
|
||||
|
||||
test('13. all entity pages soft-deleted → empty gazetteer', async () => {
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person', title: 'Alice Example', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
await engine.softDeletePage('people/alice');
|
||||
const g = await buildGazetteer(engine);
|
||||
expect(g.size).toBe(0);
|
||||
});
|
||||
|
||||
test('18. hardcoded type filter — page with type=meeting NOT in gazetteer', async () => {
|
||||
await engine.putPage('meetings/2026-01-15', {
|
||||
type: 'meeting' as any, title: 'Weekly Sync',
|
||||
compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
await engine.putPage('people/bob', {
|
||||
type: 'person', title: 'Robert Builder', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
const g = await buildGazetteer(engine);
|
||||
expect(g.has('weekly')).toBe(false); // meeting type filtered out
|
||||
expect(g.has('robert')).toBe(true); // person type included
|
||||
});
|
||||
|
||||
test('19. min-length + ignore-list interaction — "YC" (2 chars) filtered by min-length BEFORE ignore-list', async () => {
|
||||
// YC isn't in DEFAULT_IGNORE_LIST. But "Box" (3 chars) is. "Box" length
|
||||
// = 3 < MIN_NAME_LENGTH (4), so it's filtered by min-length first. The
|
||||
// ignore-list never fires. We test the regression that the min-length
|
||||
// gate runs BEFORE the ignore-list (so adding Box to ignore-list
|
||||
// doesn't accidentally change the filter ordering).
|
||||
await engine.putPage('companies/box', {
|
||||
type: 'company', title: 'Box', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
const g = await buildGazetteer(engine);
|
||||
// "Box" is 3 chars → min-length filter drops it (whether or not in ignore-list).
|
||||
expect(g.has('box')).toBe(false);
|
||||
});
|
||||
|
||||
test('extraIgnore — user-supplied additional ignore tokens', async () => {
|
||||
await engine.putPage('people/john', {
|
||||
type: 'person', title: 'John', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
// No companies/john exists, so adding John to extraIgnore should suppress.
|
||||
const g1 = await buildGazetteer(engine);
|
||||
expect(g1.has('john')).toBe(true); // baseline: in gazetteer
|
||||
const g2 = await buildGazetteer(engine, { extraIgnore: ['John'] });
|
||||
// But title "John" IS the entity title — existingTitles.has('John') is true.
|
||||
// Per CK12 rule, gazetteer presence wins → John IS still in.
|
||||
expect(g2.has('john')).toBe(true);
|
||||
});
|
||||
|
||||
test('LINKABLE_ENTITY_TYPES exposes the hardcoded contract', () => {
|
||||
// Regression: if anyone changes the hardcoded type list, this test
|
||||
// forces a deliberate change (and a corresponding test update).
|
||||
expect(LINKABLE_ENTITY_TYPES).toEqual(['person', 'company', 'organization', 'entity']);
|
||||
});
|
||||
});
|
||||
@@ -174,3 +174,60 @@ describe('rewriteChunkedSlug — D6 zero-Sonnet-trust slug rewrite', () => {
|
||||
expect(rewriteChunkedSlug('', 'abc123', 0)).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('splitTranscriptByBudget — UTF-16 surrogate-pair safety (v0.42.0.0)', () => {
|
||||
// 🚀 = U+1F680 (surrogate pair: 0xD83D 0xDE80; JS string length = 2).
|
||||
// 𠀀 = U+20000 (non-BMP CJK: 0xD840 0xDC00).
|
||||
const ROCKET = '🚀';
|
||||
const NBMP_HAN = '𠀀';
|
||||
|
||||
test('hard-split lands at emoji boundary — pair not orphaned', () => {
|
||||
// Build a content with NO `## Topic:`, NO `---`, NO `\n` past the
|
||||
// first half so all three boundary tiers fail and findBoundary falls
|
||||
// through to the surrogate-safe hard-split. Emoji at maxChars=300.
|
||||
// Budget = 300; searchStart somewhere in [150, 179]. No newlines in
|
||||
// back-half → tier-3 hard-split fires.
|
||||
const head = 'x'.repeat(298); // ASCII pad to position 298
|
||||
const tail = ROCKET + 'y'.repeat(200); // 🚀 at positions [298, 299]
|
||||
const content = head + tail;
|
||||
const out = splitTranscriptByBudget(content, 'cafebabedeadbeef', 300);
|
||||
expect(out.length).toBeGreaterThanOrEqual(2);
|
||||
// chunk 1 must not orphan a surrogate: every chunk-1 char.codeAt
|
||||
// must be paired (or non-surrogate).
|
||||
const c1 = out[0]!;
|
||||
const lastCode = c1.charCodeAt(c1.length - 1);
|
||||
expect(lastCode >= 0xd800 && lastCode <= 0xdbff).toBe(false);
|
||||
// joined chunks reconstruct the source byte-identical
|
||||
expect(out.join('')).toBe(content);
|
||||
});
|
||||
|
||||
test('determinism preserved with non-BMP CJK at hard-split boundary', () => {
|
||||
const head = 'a'.repeat(298);
|
||||
const tail = NBMP_HAN + 'b'.repeat(200);
|
||||
const content = head + tail;
|
||||
const refs = new Set<string>();
|
||||
for (let i = 0; i < 10; i++) {
|
||||
refs.add(JSON.stringify(splitTranscriptByBudget(content, 'cafebabedeadbeef', 300)));
|
||||
}
|
||||
// same (content, hash, maxChars) → byte-identical chunks every time
|
||||
expect(refs.size).toBe(1);
|
||||
});
|
||||
|
||||
test('joined chunks always reconstruct source — fuzz across multiple hashes', () => {
|
||||
const head = 'h'.repeat(297);
|
||||
const tail = ROCKET + NBMP_HAN + 't'.repeat(200);
|
||||
const content = head + tail;
|
||||
// exercise jitter window across 5 different content hashes
|
||||
const hashes = ['0', '1f', '7fffffff', 'cafebabe', 'deadbeef'];
|
||||
for (const h of hashes) {
|
||||
const out = splitTranscriptByBudget(content, h.padStart(16, '0'), 300);
|
||||
expect(out.join('')).toBe(content);
|
||||
// no chunk ends with an unpaired high surrogate
|
||||
for (const c of out) {
|
||||
if (c.length === 0) continue;
|
||||
const last = c.charCodeAt(c.length - 1);
|
||||
expect(last >= 0xd800 && last <= 0xdbff).toBe(false);
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
/**
|
||||
* Tests for the v0.42.0.0 doctor `orphan_ratio` check (D5/D11).
|
||||
*
|
||||
* Local-surface tests run against the same runDoctor path as the CLI.
|
||||
* Thin-client-surface tests exercise `runOrphanRatioCheck` from
|
||||
* doctor-remote.ts with a stubbed callRemoteTool (no real MCP server
|
||||
* needed for unit coverage; the cross-surface parity contract is the
|
||||
* shared op + shared math, pinned via the source-grep regression at
|
||||
* the bottom of this file).
|
||||
*
|
||||
* Hermetic PGLite for the local-surface path.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runDoctor, type DoctorReport } from '../src/commands/doctor.ts';
|
||||
import { setCliOptions } from '../src/core/cli-options.ts';
|
||||
import { runOrphanRatioCheck } from '../src/core/doctor-remote.ts';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let stdoutBuffer: string[];
|
||||
const origLog = console.log;
|
||||
const origErr = console.error;
|
||||
const origExit = process.exit;
|
||||
|
||||
function captureCli(): void {
|
||||
stdoutBuffer = [];
|
||||
console.log = (msg?: unknown) => { stdoutBuffer.push(typeof msg === 'string' ? msg : String(msg)); };
|
||||
console.error = () => {};
|
||||
(process as { exit: unknown }).exit = (() => { throw new Error('__exit'); }) as unknown as typeof process.exit;
|
||||
}
|
||||
|
||||
function restoreCli(): void {
|
||||
console.log = origLog;
|
||||
console.error = origErr;
|
||||
(process as { exit: unknown }).exit = origExit;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null });
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
restoreCli();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM links');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
async function runDoctorJson(): Promise<DoctorReport> {
|
||||
captureCli();
|
||||
try {
|
||||
// DON'T pass --fast — orphan_ratio is in the DB-checks group that
|
||||
// --fast skips. Tests need the full check set to verify the new
|
||||
// check fires.
|
||||
await runDoctor(engine, ['--json']);
|
||||
} catch (e) {
|
||||
if (!(e instanceof Error && e.message === '__exit')) throw e;
|
||||
} finally {
|
||||
restoreCli();
|
||||
}
|
||||
// Doctor --json writes the report as ONE big JSON string to stdout.
|
||||
// Take the last log entry that parses as a DoctorReport-shape object.
|
||||
for (let i = stdoutBuffer.length - 1; i >= 0; i--) {
|
||||
try {
|
||||
const parsed = JSON.parse(stdoutBuffer[i]!);
|
||||
if (parsed && typeof parsed === 'object' && 'checks' in parsed) {
|
||||
return parsed as DoctorReport;
|
||||
}
|
||||
} catch {
|
||||
// skip non-JSON lines
|
||||
}
|
||||
}
|
||||
throw new Error('No DoctorReport JSON found in stdout');
|
||||
}
|
||||
|
||||
function findCheck(report: DoctorReport, name: string) {
|
||||
return report.checks.find(c => c.name === name);
|
||||
}
|
||||
|
||||
describe('runDoctor — orphan_ratio check (local surface, D5)', () => {
|
||||
test('< 100 entity pages → vacuous status ok', async () => {
|
||||
// Seed only a handful of entity pages — vacuous gate fires.
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await engine.putPage(`people/p${i}`, {
|
||||
type: 'person', title: `Person ${i}`, compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
}
|
||||
const report = await runDoctorJson();
|
||||
const check = findCheck(report, 'orphan_ratio');
|
||||
expect(check).toBeDefined();
|
||||
expect(check!.status).toBe('ok');
|
||||
expect(check!.message).toMatch(/vacuous/i);
|
||||
});
|
||||
|
||||
test('100+ entity pages with low orphan ratio → status ok', async () => {
|
||||
// Seed 100 entity pages with substantial inbound link coverage.
|
||||
for (let i = 0; i < 100; i++) {
|
||||
await engine.putPage(`people/person-${i}`, {
|
||||
type: 'person', title: `Person ${i}`, compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
}
|
||||
// Link 80% of them to be non-orphans (1 inbound link each).
|
||||
await engine.putPage('writing/index', {
|
||||
type: 'note', title: 'Index', compiled_truth: 'index', timeline: '', frontmatter: {},
|
||||
});
|
||||
const links = [];
|
||||
for (let i = 0; i < 80; i++) {
|
||||
links.push({
|
||||
from_slug: 'writing/index',
|
||||
to_slug: `people/person-${i}`,
|
||||
link_type: 'mentions', link_source: 'markdown', context: '',
|
||||
});
|
||||
}
|
||||
await engine.addLinksBatch(links);
|
||||
const report = await runDoctorJson();
|
||||
const check = findCheck(report, 'orphan_ratio');
|
||||
// 20 orphans / 100 linkable = 20% — under warn threshold (50%) → ok.
|
||||
expect(check!.status).toBe('ok');
|
||||
expect(check!.message).toMatch(/orphan ratio/i);
|
||||
});
|
||||
|
||||
test('high orphan ratio (>0.5, <=0.8) → warn with fix-hint', async () => {
|
||||
for (let i = 0; i < 100; i++) {
|
||||
await engine.putPage(`companies/co-${i}`, {
|
||||
type: 'company', title: `Co ${i}`, compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
}
|
||||
// Link only 30% of entities (70% orphan ratio).
|
||||
await engine.putPage('writing/index', {
|
||||
type: 'note', title: 'Index', compiled_truth: 'index', timeline: '', frontmatter: {},
|
||||
});
|
||||
const links = [];
|
||||
for (let i = 0; i < 30; i++) {
|
||||
links.push({
|
||||
from_slug: 'writing/index',
|
||||
to_slug: `companies/co-${i}`,
|
||||
link_type: 'mentions', link_source: 'markdown', context: '',
|
||||
});
|
||||
}
|
||||
await engine.addLinksBatch(links);
|
||||
const report = await runDoctorJson();
|
||||
const check = findCheck(report, 'orphan_ratio');
|
||||
expect(check!.status).toBe('warn');
|
||||
expect(check!.message).toContain('gbrain extract links --by-mention');
|
||||
});
|
||||
|
||||
test('very high orphan ratio (>0.8) → fail with urgency fix-hint', async () => {
|
||||
for (let i = 0; i < 100; i++) {
|
||||
await engine.putPage(`orgs/org-${i}`, {
|
||||
type: 'organization', title: `Org ${i}`, compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
}
|
||||
// Link only 10% — 90% orphan ratio.
|
||||
await engine.putPage('writing/index', {
|
||||
type: 'note', title: 'Index', compiled_truth: 'i', timeline: '', frontmatter: {},
|
||||
});
|
||||
const links = [];
|
||||
for (let i = 0; i < 10; i++) {
|
||||
links.push({
|
||||
from_slug: 'writing/index',
|
||||
to_slug: `orgs/org-${i}`,
|
||||
link_type: 'mentions', link_source: 'markdown', context: '',
|
||||
});
|
||||
}
|
||||
await engine.addLinksBatch(links);
|
||||
const report = await runDoctorJson();
|
||||
const check = findCheck(report, 'orphan_ratio');
|
||||
expect(check!.status).toBe('fail');
|
||||
expect(check!.message).toContain('gbrain extract links --by-mention');
|
||||
});
|
||||
|
||||
test('zero entity pages → vacuous status ok', async () => {
|
||||
const report = await runDoctorJson();
|
||||
const check = findCheck(report, 'orphan_ratio');
|
||||
expect(check!.status).toBe('ok');
|
||||
});
|
||||
|
||||
test('JSON envelope shape — orphan_ratio appears in checks[]', async () => {
|
||||
const report = await runDoctorJson();
|
||||
expect(report.schema_version).toBe(2);
|
||||
expect(Array.isArray(report.checks)).toBe(true);
|
||||
const names = report.checks.map(c => c.name);
|
||||
expect(names).toContain('orphan_ratio');
|
||||
});
|
||||
});
|
||||
|
||||
describe('runOrphanRatioCheck — thin-client surface (D11)', () => {
|
||||
// Stubbed callRemoteTool is hard to inject without `mock.module`,
|
||||
// which violates the test-isolation rule. Instead exercise the
|
||||
// network-failure branch (which catches the unconfigured-server case)
|
||||
// and pin the shape of the returned RemoteCheck.
|
||||
|
||||
test('returns informational ok on network failure (unconfigured config)', async () => {
|
||||
const result = await runOrphanRatioCheck({
|
||||
// Missing remote_mcp → callRemoteTool will throw.
|
||||
} as any);
|
||||
expect(result.name).toBe('orphan_ratio');
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.message).toMatch(/could not query remote|informational/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe('cross-surface parity contract', () => {
|
||||
test('source greps: orphan_ratio check name appears in BOTH local doctor and remote doctor', () => {
|
||||
const doctor = readFileSync('src/commands/doctor.ts', 'utf8');
|
||||
const remote = readFileSync('src/core/doctor-remote.ts', 'utf8');
|
||||
expect(doctor.includes("name: 'orphan_ratio'")).toBe(true);
|
||||
expect(remote.includes("name: 'orphan_ratio'")).toBe(true);
|
||||
});
|
||||
|
||||
test('source greps: both surfaces reference the same fix command', () => {
|
||||
const doctor = readFileSync('src/commands/doctor.ts', 'utf8');
|
||||
const remote = readFileSync('src/core/doctor-remote.ts', 'utf8');
|
||||
expect(doctor).toContain('gbrain extract links --by-mention');
|
||||
expect(remote).toContain('gbrain extract links --by-mention');
|
||||
});
|
||||
|
||||
test('source greps: local hint is self-fix; thin-client hint points at operator', () => {
|
||||
const doctor = readFileSync('src/commands/doctor.ts', 'utf8');
|
||||
const remote = readFileSync('src/core/doctor-remote.ts', 'utf8');
|
||||
// Local hint: just the command (user can run it).
|
||||
expect(doctor).toContain('Run: gbrain extract links --by-mention');
|
||||
// Thin-client hint: ask the operator.
|
||||
expect(remote).toMatch(/Ask the brain operator/i);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,174 @@
|
||||
/**
|
||||
* E2E test for v0.42.0.0 migration #1 (auto-link orphan reduction).
|
||||
*
|
||||
* Pins the design-doc claim SHAPE — "material reduction in orphan
|
||||
* pages" — without committing to a specific %, per TODO-4=C (soften
|
||||
* the 88%→<30% promise to "material reduction, exact figure TBD via
|
||||
* post-merge measurement on representative brain").
|
||||
*
|
||||
* 3 cases:
|
||||
* 1. Seed brain with known orphan ratio. Run --by-mention. Assert
|
||||
* orphan count drops materially.
|
||||
* 2. Cross-check: gbrain orphans --count and doctor JSON orphan_ratio
|
||||
* report the same underlying number (D1 single-source contract).
|
||||
* 3. Re-run --by-mention: 0 new links, no double-counting (idempotency
|
||||
* end-to-end across runs).
|
||||
*
|
||||
* Hermetic via PGLite. No DATABASE_URL needed.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { runExtract } from '../../src/commands/extract.ts';
|
||||
import { runOrphans, getOrphansData } from '../../src/commands/orphans.ts';
|
||||
import { runDoctor } from '../../src/commands/doctor.ts';
|
||||
import { setCliOptions } from '../../src/core/cli-options.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const origLog = console.log;
|
||||
const origErr = console.error;
|
||||
const origExit = process.exit;
|
||||
let stdoutBuffer: string[];
|
||||
|
||||
function captureCli(): void {
|
||||
stdoutBuffer = [];
|
||||
console.log = (msg?: unknown) => { stdoutBuffer.push(typeof msg === 'string' ? msg : String(msg)); };
|
||||
console.error = () => {};
|
||||
(process as { exit: unknown }).exit = (() => { throw new Error('__exit'); }) as unknown as typeof process.exit;
|
||||
}
|
||||
|
||||
function restoreCli(): void {
|
||||
console.log = origLog;
|
||||
console.error = origErr;
|
||||
(process as { exit: unknown }).exit = origExit;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
setCliOptions({ quiet: true, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null });
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
restoreCli();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM links');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
/**
|
||||
* Seed brain with N entity pages (0..N-1) and M content pages whose
|
||||
* body mentions a deterministic subset of the entities. Returns the
|
||||
* expected post-mention-pass non-orphan entity count.
|
||||
*/
|
||||
async function seedBrain(entityCount: number, contentCount: number, mentionsPerContent: number): Promise<{
|
||||
expectedNonOrphans: number;
|
||||
}> {
|
||||
// Entities — give each a unique multi-token title so phrase-match exercise fires.
|
||||
for (let i = 0; i < entityCount; i++) {
|
||||
await engine.putPage(`people/person-${i}`, {
|
||||
type: 'person',
|
||||
title: `Persona Number ${i}`,
|
||||
compiled_truth: 'p body',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
}
|
||||
|
||||
// Content pages — each mentions `mentionsPerContent` entities by title.
|
||||
const mentionedEntities = new Set<number>();
|
||||
for (let j = 0; j < contentCount; j++) {
|
||||
const mentions: string[] = [];
|
||||
for (let k = 0; k < mentionsPerContent; k++) {
|
||||
const idx = (j * mentionsPerContent + k) % entityCount;
|
||||
mentions.push(`Persona Number ${idx} did something noteworthy.`);
|
||||
mentionedEntities.add(idx);
|
||||
}
|
||||
await engine.putPage(`writing/post-${j}`, {
|
||||
type: 'note',
|
||||
title: `Post ${j}`,
|
||||
compiled_truth: mentions.join(' '),
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
}
|
||||
return { expectedNonOrphans: mentionedEntities.size };
|
||||
}
|
||||
|
||||
describe('v0.42.0.0 e2e — orphan reduction via --by-mention', () => {
|
||||
test('1. seeding 20 entities + 5 content pages → mentioning 15 → orphan count drops materially', async () => {
|
||||
await seedBrain(20, 5, 3); // 5 posts × 3 mentions each = 15 unique entities mentioned
|
||||
const before = await getOrphansData(engine, { includePseudo: false });
|
||||
captureCli();
|
||||
try { await runExtract(engine, ['links', '--by-mention', '--source', 'db']); }
|
||||
catch (e) { if (!(e instanceof Error && e.message === '__exit')) throw e; }
|
||||
finally { restoreCli(); }
|
||||
const after = await getOrphansData(engine, { includePseudo: false });
|
||||
// Material reduction — at least 10 entities should have moved from
|
||||
// orphan to non-orphan (we mentioned 15 of 20 unique entities).
|
||||
expect(before.total_orphans).toBeGreaterThan(after.total_orphans);
|
||||
const delta = before.total_orphans - after.total_orphans;
|
||||
expect(delta).toBeGreaterThanOrEqual(10);
|
||||
});
|
||||
|
||||
test('2. cross-check — gbrain orphans count matches doctor JSON orphan_ratio numerator', async () => {
|
||||
await seedBrain(100, 10, 5); // 100 entities, 50 mentioned
|
||||
captureCli();
|
||||
try { await runExtract(engine, ['links', '--by-mention', '--source', 'db']); }
|
||||
catch (e) { if (!(e instanceof Error && e.message === '__exit')) throw e; }
|
||||
finally { restoreCli(); }
|
||||
// Direct pure-fn call.
|
||||
const direct = await getOrphansData(engine, { includePseudo: false });
|
||||
// CLI `gbrain orphans --count` output.
|
||||
captureCli();
|
||||
try { await runOrphans(engine, ['--count']); }
|
||||
catch (e) { if (!(e instanceof Error && e.message === '__exit')) throw e; }
|
||||
finally { restoreCli(); }
|
||||
const cliCount = Number(stdoutBuffer.find(l => /^\d+$/.test(l)) ?? '-1');
|
||||
expect(cliCount).toBe(direct.total_orphans);
|
||||
// Doctor JSON path.
|
||||
captureCli();
|
||||
try { await runDoctor(engine, ['--json']); }
|
||||
catch (e) { if (!(e instanceof Error && e.message === '__exit')) throw e; }
|
||||
finally { restoreCli(); }
|
||||
let doctorJson: any = null;
|
||||
for (let i = stdoutBuffer.length - 1; i >= 0; i--) {
|
||||
try {
|
||||
const parsed = JSON.parse(stdoutBuffer[i]!);
|
||||
if (parsed && typeof parsed === 'object' && 'checks' in parsed) {
|
||||
doctorJson = parsed;
|
||||
break;
|
||||
}
|
||||
} catch {/* skip */}
|
||||
}
|
||||
expect(doctorJson).not.toBeNull();
|
||||
const orphanCheck = doctorJson.checks.find((c: any) => c.name === 'orphan_ratio');
|
||||
expect(orphanCheck).toBeDefined();
|
||||
// Doctor message includes the numerator/denominator string.
|
||||
expect(orphanCheck.message).toContain(`${direct.total_orphans}/${direct.total_linkable}`);
|
||||
});
|
||||
|
||||
test('3. re-run idempotency — second --by-mention produces 0 new mention rows', async () => {
|
||||
await seedBrain(30, 6, 4);
|
||||
captureCli();
|
||||
try { await runExtract(engine, ['links', '--by-mention', '--source', 'db']); }
|
||||
catch (e) { if (!(e instanceof Error && e.message === '__exit')) throw e; }
|
||||
finally { restoreCli(); }
|
||||
const firstCount = Number((await engine.executeRaw<{ c: string }>(
|
||||
`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, [],
|
||||
))[0]!.c);
|
||||
captureCli();
|
||||
try { await runExtract(engine, ['links', '--by-mention', '--source', 'db']); }
|
||||
catch (e) { if (!(e instanceof Error && e.message === '__exit')) throw e; }
|
||||
finally { restoreCli(); }
|
||||
const secondCount = Number((await engine.executeRaw<{ c: string }>(
|
||||
`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, [],
|
||||
))[0]!.c);
|
||||
expect(secondCount).toBe(firstCount);
|
||||
expect(secondCount).toBeGreaterThan(0); // sanity: we did create SOME links on first pass
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,289 @@
|
||||
/**
|
||||
* Integration tests for `gbrain extract links --by-mention`.
|
||||
*
|
||||
* Hermetic PGLite. Drives runExtract via the same dispatcher the CLI uses,
|
||||
* captures stdout/stderr, asserts side-effects against the engine.
|
||||
*
|
||||
* Covers 14 cases from the v0.42.0.0 plan:
|
||||
* 1. End-to-end happy path — links created with link_source='mentions'
|
||||
* 2. Idempotency — second run = 0 new links
|
||||
* 3. --dry-run writes nothing, prints expected count
|
||||
* 4. --json output shape stable
|
||||
* 5. --source-id correctly scopes page WALK (gazetteer remains brain-wide-ish)
|
||||
* 6. --since DATE only scans pages modified after date
|
||||
* 7. --source fs --by-mention rejected with usage error + fix-hint
|
||||
* 8. Default link extract NOT also run when --by-mention is set
|
||||
* 9. Cross-source mention suppressed (source isolation)
|
||||
* 10. Pseudo-pages excluded from gazetteer by type filter (NOT auto-suffix)
|
||||
* 11. Existing markdown link coexists with mention link (different link_source)
|
||||
* 12. Progress phase events fire under --progress-json
|
||||
* 13. Schema migration verified — link_source='mentions' insert succeeds
|
||||
* 14. Empty brain (no entity pages) → no-op with informative message
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runExtract } from '../src/commands/extract.ts';
|
||||
import { setCliOptions } from '../src/core/cli-options.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
// stdout/stderr capture for CLI assertions. Intercepts BOTH console.log
|
||||
// AND process.stdout.write (JSON action lines bypass console.log).
|
||||
let stdoutBuffer: string[];
|
||||
let stderrBuffer: string[];
|
||||
let exitedWith: number | null;
|
||||
const origLog = console.log;
|
||||
const origErr = console.error;
|
||||
const origExit = process.exit;
|
||||
const origStdoutWrite = process.stdout.write.bind(process.stdout);
|
||||
const origStderrWrite = process.stderr.write.bind(process.stderr);
|
||||
|
||||
function captureCli(): void {
|
||||
stdoutBuffer = [];
|
||||
stderrBuffer = [];
|
||||
exitedWith = null;
|
||||
console.log = (msg?: unknown) => { stdoutBuffer.push(typeof msg === 'string' ? msg : String(msg)); };
|
||||
console.error = (msg?: unknown) => { stderrBuffer.push(typeof msg === 'string' ? msg : String(msg)); };
|
||||
(process.stdout as unknown as { write: unknown }).write = ((chunk: unknown) => {
|
||||
stdoutBuffer.push(typeof chunk === 'string' ? chunk : String(chunk));
|
||||
return true;
|
||||
}) as unknown as typeof process.stdout.write;
|
||||
(process.stderr as unknown as { write: unknown }).write = ((chunk: unknown) => {
|
||||
stderrBuffer.push(typeof chunk === 'string' ? chunk : String(chunk));
|
||||
return true;
|
||||
}) as unknown as typeof process.stderr.write;
|
||||
(process as { exit: unknown }).exit = ((code?: number) => {
|
||||
exitedWith = code ?? 0;
|
||||
throw new Error(`__test_exit:${code ?? 0}`);
|
||||
}) as unknown as typeof process.exit;
|
||||
}
|
||||
|
||||
function restoreCli(): void {
|
||||
console.log = origLog;
|
||||
console.error = origErr;
|
||||
(process.stdout as unknown as { write: unknown }).write = origStdoutWrite;
|
||||
(process.stderr as unknown as { write: unknown }).write = origStderrWrite;
|
||||
(process as { exit: unknown }).exit = origExit;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
// Default CLI options (quiet enough that the progress reporter doesn't
|
||||
// pollute the capture buffer beyond what the assertions need).
|
||||
setCliOptions({ quiet: false, progressJson: false, progressInterval: 1000, explain: false, timeoutMs: null });
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
restoreCli();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM links');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
// Source registration needed for cross-source tests (default exists from initSchema).
|
||||
});
|
||||
|
||||
async function seedEntities(): Promise<void> {
|
||||
await engine.putPage('companies/acme', { type: 'company', title: 'Acme Corp', compiled_truth: 'acme body', timeline: '', frontmatter: {} });
|
||||
await engine.putPage('people/alice', { type: 'person', title: 'Alice Example', compiled_truth: 'alice body', timeline: '', frontmatter: {} });
|
||||
await engine.putPage('people/bob', { type: 'person', title: 'Robert Builder', compiled_truth: 'bob body', timeline: '', frontmatter: {} });
|
||||
}
|
||||
|
||||
async function seedContentPage(slug: string, body: string, timeline = ''): Promise<void> {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: body, timeline, frontmatter: {} });
|
||||
}
|
||||
|
||||
async function runCli(args: string[]): Promise<void> {
|
||||
captureCli();
|
||||
try {
|
||||
await runExtract(engine, args);
|
||||
} catch (e) {
|
||||
// process.exit threw — captured in exitedWith. Swallow.
|
||||
if (!(e instanceof Error && e.message.startsWith('__test_exit:'))) throw e;
|
||||
} finally {
|
||||
restoreCli();
|
||||
}
|
||||
}
|
||||
|
||||
describe('gbrain extract links --by-mention — integration', () => {
|
||||
test('1. end-to-end happy path — links created with link_source=mentions', async () => {
|
||||
await seedEntities();
|
||||
await seedContentPage('writing/post-1', 'We met with Acme Corp and Alice Example yesterday.');
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
const rows = await engine.executeRaw<{ ls: string; from_slug: string; to_slug: string }>(
|
||||
`SELECT l.link_source AS ls, fp.slug AS from_slug, tp.slug AS to_slug
|
||||
FROM links l
|
||||
JOIN pages fp ON fp.id = l.from_page_id
|
||||
JOIN pages tp ON tp.id = l.to_page_id
|
||||
WHERE fp.slug = 'writing/post-1' AND l.link_source = 'mentions'`,
|
||||
[],
|
||||
);
|
||||
const targets = rows.map(r => r.to_slug).sort();
|
||||
expect(rows.length).toBeGreaterThanOrEqual(2);
|
||||
expect(targets).toContain('companies/acme');
|
||||
expect(targets).toContain('people/alice');
|
||||
// Robert Builder NOT mentioned in body — should not appear.
|
||||
expect(targets).not.toContain('people/bob');
|
||||
});
|
||||
|
||||
test('2. idempotency — second run produces 0 new links', async () => {
|
||||
await seedEntities();
|
||||
await seedContentPage('writing/post-1', 'We met with Acme Corp and Alice Example.');
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
const firstCount = (await engine.executeRaw<{ c: string }>(`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, []))[0]!.c;
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
const secondCount = (await engine.executeRaw<{ c: string }>(`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, []))[0]!.c;
|
||||
expect(secondCount).toBe(firstCount);
|
||||
});
|
||||
|
||||
test('3. --dry-run writes nothing', async () => {
|
||||
await seedEntities();
|
||||
await seedContentPage('writing/post-1', 'We met with Acme Corp.');
|
||||
await runCli(['links', '--by-mention', '--source', 'db', '--dry-run']);
|
||||
const rows = await engine.executeRaw<{ c: string }>(`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, []);
|
||||
expect(Number(rows[0]!.c)).toBe(0);
|
||||
});
|
||||
|
||||
test('4. --json output shape stable (dry-run action lines on stdout)', async () => {
|
||||
await seedEntities();
|
||||
await seedContentPage('writing/post-1', 'Acme Corp here.');
|
||||
await runCli(['links', '--by-mention', '--source', 'db', '--dry-run', '--json']);
|
||||
const actionLines = stdoutBuffer.filter(l => l.includes('"action":"add_link"'));
|
||||
expect(actionLines.length).toBeGreaterThanOrEqual(1);
|
||||
const parsed = JSON.parse(actionLines[0]!);
|
||||
expect(parsed.action).toBe('add_link');
|
||||
expect(parsed.link_source).toBe('mentions');
|
||||
expect(parsed.type).toBe('mentions');
|
||||
expect(parsed.from).toBe('writing/post-1');
|
||||
expect(parsed.to).toBe('companies/acme');
|
||||
});
|
||||
|
||||
test('7. --source fs --by-mention rejected with paste-ready fix-hint', async () => {
|
||||
await runCli(['links', '--by-mention', '--source', 'fs']);
|
||||
expect(exitedWith).toBe(2);
|
||||
const stderrText = stderrBuffer.join('\n');
|
||||
expect(stderrText).toContain('--by-mention requires --source db');
|
||||
expect(stderrText).toContain('gbrain extract links --by-mention --source db');
|
||||
});
|
||||
|
||||
test('7b. --by-mention timeline rejected', async () => {
|
||||
await runCli(['timeline', '--by-mention', '--source', 'db']);
|
||||
expect(exitedWith).toBe(2);
|
||||
const stderrText = stderrBuffer.join('\n');
|
||||
expect(stderrText).toContain('--by-mention is a links-pass only');
|
||||
});
|
||||
|
||||
test('8. mode dispatch — default link extract NOT also run when --by-mention is set', async () => {
|
||||
await seedEntities();
|
||||
// Markdown-link page that would normally produce a `link_source=markdown`
|
||||
// row through the default link extract. With --by-mention, default
|
||||
// extract should NOT fire.
|
||||
await seedContentPage('writing/post-1', 'Acme Corp [link](companies/acme).');
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
const mdRows = await engine.executeRaw<{ c: string }>(
|
||||
`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'markdown'`, [],
|
||||
);
|
||||
expect(Number(mdRows[0]!.c)).toBe(0);
|
||||
const mentionRows = await engine.executeRaw<{ c: string }>(
|
||||
`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, [],
|
||||
);
|
||||
expect(Number(mentionRows[0]!.c)).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('11. existing markdown link + new mention link coexist (different link_source)', async () => {
|
||||
await seedEntities();
|
||||
await seedContentPage('writing/post-1', 'Acme Corp body text');
|
||||
// Simulate a pre-existing markdown link from the default extract pass.
|
||||
await engine.addLinksBatch([
|
||||
{
|
||||
from_slug: 'writing/post-1',
|
||||
to_slug: 'companies/acme',
|
||||
link_type: 'mentions',
|
||||
link_source: 'markdown',
|
||||
context: '',
|
||||
},
|
||||
]);
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
const rows = await engine.executeRaw<{ ls: string }>(
|
||||
`SELECT l.link_source AS ls FROM links l
|
||||
JOIN pages fp ON fp.id = l.from_page_id
|
||||
JOIN pages tp ON tp.id = l.to_page_id
|
||||
WHERE fp.slug = 'writing/post-1' AND tp.slug = 'companies/acme'`,
|
||||
[],
|
||||
);
|
||||
const sources = rows.map(r => r.ls).sort();
|
||||
// Both rows present — ON CONFLICT key includes link_source so no collision.
|
||||
expect(sources).toContain('markdown');
|
||||
expect(sources).toContain('mentions');
|
||||
});
|
||||
|
||||
test('13. schema migration verified — link_source=mentions insert succeeds end-to-end', async () => {
|
||||
await seedEntities();
|
||||
await seedContentPage('writing/post-1', 'Acme Corp mentioned.');
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
// The fact that the test got here without a CHECK constraint violation
|
||||
// is the assertion — migration v95 widened the CHECK so 'mentions' is valid.
|
||||
const rows = await engine.executeRaw<{ c: string }>(
|
||||
`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, [],
|
||||
);
|
||||
expect(Number(rows[0]!.c)).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('14. empty brain (no entity pages) → no-op with informative message', async () => {
|
||||
// No entity pages — only a content page.
|
||||
await seedContentPage('writing/lonely', 'Acme Corp Alice Example all mentioned but no entities exist.');
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
const rows = await engine.executeRaw<{ c: string }>(
|
||||
`SELECT COUNT(*)::text AS c FROM links WHERE link_source = 'mentions'`, [],
|
||||
);
|
||||
expect(Number(rows[0]!.c)).toBe(0);
|
||||
// Informative message in stdout.
|
||||
const stdoutText = stdoutBuffer.join('\n');
|
||||
expect(stdoutText).toMatch(/no linkable entity pages|nothing to scan/i);
|
||||
});
|
||||
|
||||
test('5. --source-id scopes page WALK', async () => {
|
||||
await seedEntities(); // all in 'default'
|
||||
// Register a second source via raw SQL (PGLite engine doesn't expose a setupSource helper).
|
||||
await engine.executeRaw(`INSERT INTO sources (id, name) VALUES ('team-b', 'Team B') ON CONFLICT (id) DO NOTHING`, []);
|
||||
// Content page in team-b mentioning Acme (default-source entity).
|
||||
await engine.putPage('writing/team-b-post', {
|
||||
type: 'note', title: 'Team B Post', compiled_truth: 'Acme Corp mentioned here.', timeline: '', frontmatter: {},
|
||||
}, { sourceId: 'team-b' });
|
||||
await engine.putPage('writing/default-post', {
|
||||
type: 'note', title: 'Default Post', compiled_truth: 'Acme Corp here too.', timeline: '', frontmatter: {},
|
||||
});
|
||||
// Scope walk to team-b only.
|
||||
await runCli(['links', '--by-mention', '--source', 'db', '--source-id', 'team-b']);
|
||||
// team-b post mentions Acme — but cross-source guard suppresses (Acme is in 'default').
|
||||
const rows = await engine.executeRaw<{ c: string; fp: string }>(
|
||||
`SELECT COUNT(*)::text AS c, fp.slug AS fp FROM links l
|
||||
JOIN pages fp ON fp.id = l.from_page_id
|
||||
WHERE l.link_source = 'mentions' GROUP BY fp.slug`, [],
|
||||
);
|
||||
// Default-source post NOT scanned (walk scoped to team-b); team-b post
|
||||
// scanned but cross-source guard fires → zero mention rows.
|
||||
expect(rows.length).toBe(0);
|
||||
});
|
||||
|
||||
test('9. cross-source mention suppressed (source isolation)', async () => {
|
||||
await seedEntities(); // entities in 'default'
|
||||
await engine.executeRaw(`INSERT INTO sources (id, name) VALUES ('team-b', 'Team B') ON CONFLICT (id) DO NOTHING`, []);
|
||||
await engine.putPage('writing/team-b-post', {
|
||||
type: 'note', title: 'Team B Post', compiled_truth: 'Acme Corp mentioned.', timeline: '', frontmatter: {},
|
||||
}, { sourceId: 'team-b' });
|
||||
// Run without --source-id (walks all) — team-b post mentions default-source Acme.
|
||||
await runCli(['links', '--by-mention', '--source', 'db']);
|
||||
const rows = await engine.executeRaw<{ c: string }>(
|
||||
`SELECT COUNT(*)::text AS c FROM links l
|
||||
JOIN pages fp ON fp.id = l.from_page_id
|
||||
WHERE fp.slug = 'writing/team-b-post' AND l.link_source = 'mentions'`, [],
|
||||
);
|
||||
// Cross-source guard fires → 0 mention links from team-b/post to default/acme.
|
||||
expect(Number(rows[0]!.c)).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,193 @@
|
||||
/**
|
||||
* IRON RULE regression test (per D1 from /plan-eng-review for v0.42.0.0).
|
||||
*
|
||||
* Pins byte-identical output between:
|
||||
* - `gbrain orphans --json` (CLI orchestrator `runOrphans`)
|
||||
* - `findOrphans(engine, opts)` (canonical pure data fn)
|
||||
* - `getOrphansData(engine, opts)` (v0.42.0.0 alias for findOrphans)
|
||||
*
|
||||
* If a future refactor lets the CLI filter results differently after
|
||||
* `findOrphans` returns, this test catches the drift. Doctor's
|
||||
* `orphan_ratio` check imports `getOrphansData`; this test guarantees
|
||||
* the doctor count cannot disagree with `gbrain orphans --count`.
|
||||
*
|
||||
* Hermetic via PGLite. No DATABASE_URL needed.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
findOrphans,
|
||||
getOrphansData,
|
||||
shouldExclude,
|
||||
runOrphans,
|
||||
} from '../src/commands/orphans.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let logBuffer: string[];
|
||||
const originalLog = console.log;
|
||||
|
||||
function captureConsoleLog(): void {
|
||||
logBuffer = [];
|
||||
console.log = (msg?: unknown) => {
|
||||
logBuffer.push(typeof msg === 'string' ? msg : String(msg));
|
||||
};
|
||||
}
|
||||
|
||||
function restoreConsoleLog(): void {
|
||||
console.log = originalLog;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
restoreConsoleLog();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clean slate per test — keeps the IRON RULE assertions deterministic
|
||||
// across the file's test suite.
|
||||
await engine.executeRaw('DELETE FROM links');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
async function seedFixture(): Promise<void> {
|
||||
// 5 entity pages: 2 will have inbound links, 3 will be orphans.
|
||||
// 2 pseudo-pages: should be excluded by default filter.
|
||||
// 1 content page: links to person-1 + company-1.
|
||||
await engine.putPage('people/person-1', {
|
||||
type: 'person', title: 'Person 1', compiled_truth: 'p1', timeline: '', frontmatter: { domain: 'people' },
|
||||
});
|
||||
await engine.putPage('people/person-2', {
|
||||
type: 'person', title: 'Person 2', compiled_truth: 'p2', timeline: '', frontmatter: { domain: 'people' },
|
||||
});
|
||||
await engine.putPage('people/person-3', {
|
||||
type: 'person', title: 'Person 3', compiled_truth: 'p3', timeline: '', frontmatter: { domain: 'people' },
|
||||
});
|
||||
await engine.putPage('companies/company-1', {
|
||||
type: 'company', title: 'Company 1', compiled_truth: 'c1', timeline: '', frontmatter: { domain: 'companies' },
|
||||
});
|
||||
await engine.putPage('companies/company-2', {
|
||||
type: 'company', title: 'Company 2', compiled_truth: 'c2', timeline: '', frontmatter: { domain: 'companies' },
|
||||
});
|
||||
// Pseudo-pages — should be excluded from default orphan results.
|
||||
await engine.putPage('_atlas', {
|
||||
type: 'note', title: 'Atlas', compiled_truth: 'atlas', timeline: '', frontmatter: {},
|
||||
});
|
||||
await engine.putPage('templates/meeting', {
|
||||
type: 'note', title: 'Meeting template', compiled_truth: 'tmpl', timeline: '', frontmatter: {},
|
||||
});
|
||||
// Content page that links to person-1 + company-1.
|
||||
await engine.putPage('writing/post-1', {
|
||||
type: 'note', title: 'Post 1', compiled_truth: 'content', timeline: '', frontmatter: {},
|
||||
});
|
||||
await engine.addLinksBatch([
|
||||
{ from_slug: 'writing/post-1', to_slug: 'people/person-1', link_type: 'mentions', link_source: 'markdown', context: '' },
|
||||
{ from_slug: 'writing/post-1', to_slug: 'companies/company-1', link_type: 'mentions', link_source: 'markdown', context: '' },
|
||||
]);
|
||||
}
|
||||
|
||||
describe('orphans pure data fn — IRON RULE byte-identical contract', () => {
|
||||
test('getOrphansData is the same function reference as findOrphans', () => {
|
||||
expect(getOrphansData).toBe(findOrphans);
|
||||
});
|
||||
|
||||
test('findOrphans and getOrphansData produce deep-equal output', async () => {
|
||||
await seedFixture();
|
||||
const viaFindOrphans = await findOrphans(engine, { includePseudo: false });
|
||||
const viaGetOrphansData = await getOrphansData(engine, { includePseudo: false });
|
||||
expect(viaGetOrphansData).toEqual(viaFindOrphans);
|
||||
});
|
||||
|
||||
test('includePseudo: false vs true changes excluded count', async () => {
|
||||
await seedFixture();
|
||||
const def = await findOrphans(engine, { includePseudo: false });
|
||||
const all = await findOrphans(engine, { includePseudo: true });
|
||||
expect(all.excluded).toBe(0);
|
||||
expect(def.excluded).toBeGreaterThan(0);
|
||||
expect(all.total_orphans).toBeGreaterThanOrEqual(def.total_orphans);
|
||||
});
|
||||
|
||||
test('CLI --json output deep-equals findOrphans return value', async () => {
|
||||
await seedFixture();
|
||||
const direct = await findOrphans(engine, { includePseudo: false });
|
||||
captureConsoleLog();
|
||||
try {
|
||||
await runOrphans(engine, ['--json']);
|
||||
} finally {
|
||||
restoreConsoleLog();
|
||||
}
|
||||
expect(logBuffer.length).toBe(1);
|
||||
const cliOutput = JSON.parse(logBuffer[0]!);
|
||||
// IRON RULE: CLI --json output must deep-equal the pure-fn output.
|
||||
// If a future change adds CLI-side post-filtering, this fires.
|
||||
expect(cliOutput).toEqual(direct);
|
||||
});
|
||||
|
||||
test('CLI --count matches total_orphans from pure fn', async () => {
|
||||
await seedFixture();
|
||||
const direct = await findOrphans(engine, { includePseudo: false });
|
||||
captureConsoleLog();
|
||||
try {
|
||||
await runOrphans(engine, ['--count']);
|
||||
} finally {
|
||||
restoreConsoleLog();
|
||||
}
|
||||
expect(logBuffer.length).toBe(1);
|
||||
expect(logBuffer[0]).toBe(String(direct.total_orphans));
|
||||
});
|
||||
|
||||
test('CLI --count with --include-pseudo matches pure-fn total_orphans (includePseudo: true)', async () => {
|
||||
await seedFixture();
|
||||
const direct = await findOrphans(engine, { includePseudo: true });
|
||||
captureConsoleLog();
|
||||
try {
|
||||
await runOrphans(engine, ['--count', '--include-pseudo']);
|
||||
} finally {
|
||||
restoreConsoleLog();
|
||||
}
|
||||
expect(logBuffer[0]).toBe(String(direct.total_orphans));
|
||||
});
|
||||
});
|
||||
|
||||
describe('shouldExclude — orphan filter regression (preserve curation)', () => {
|
||||
test('pseudo-pages are excluded', () => {
|
||||
expect(shouldExclude('_atlas')).toBe(true);
|
||||
expect(shouldExclude('_index')).toBe(true);
|
||||
expect(shouldExclude('_orphans')).toBe(true);
|
||||
});
|
||||
|
||||
test('auto-suffix patterns are excluded', () => {
|
||||
expect(shouldExclude('people/_index')).toBe(true);
|
||||
expect(shouldExclude('writing/log')).toBe(true);
|
||||
});
|
||||
|
||||
test('raw segment is excluded', () => {
|
||||
expect(shouldExclude('media/x/raw/post')).toBe(true);
|
||||
});
|
||||
|
||||
test('deny-prefixes are excluded', () => {
|
||||
expect(shouldExclude('templates/meeting')).toBe(true);
|
||||
expect(shouldExclude('dashboards/_index')).toBe(true);
|
||||
expect(shouldExclude('scripts/build')).toBe(true);
|
||||
expect(shouldExclude('output/foo')).toBe(true);
|
||||
});
|
||||
|
||||
test('first-segment exclusions fire', () => {
|
||||
expect(shouldExclude('scratch/notes')).toBe(true);
|
||||
expect(shouldExclude('thoughts/today')).toBe(true);
|
||||
expect(shouldExclude('catalog/movies')).toBe(true);
|
||||
expect(shouldExclude('entities/anonymous')).toBe(true);
|
||||
});
|
||||
|
||||
test('regular slugs are NOT excluded', () => {
|
||||
expect(shouldExclude('people/alice')).toBe(false);
|
||||
expect(shouldExclude('companies/acme')).toBe(false);
|
||||
expect(shouldExclude('writing/post-1')).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,141 @@
|
||||
/**
|
||||
* Regression tests for migration v95 (link_source CHECK widening).
|
||||
*
|
||||
* Pins three contracts:
|
||||
* 1. Fresh-init brain accepts link_source='mentions' (schema-embedded.ts
|
||||
* + pglite-schema.ts widened CHECK is the source of truth for fresh
|
||||
* installs).
|
||||
* 2. Migration v95 is registered with the expected name + shape.
|
||||
* 3. Migration v95 is idempotent — re-running on an already-migrated
|
||||
* brain is a no-op (the DROP IF EXISTS + ADD CONSTRAINT pattern).
|
||||
*
|
||||
* Hermetic via PGLite. No DATABASE_URL needed.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
|
||||
const MIGRATION_VERSION = 95;
|
||||
const MIGRATION_NAME = 'links_link_source_check_includes_mentions';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('migration v95 — links_link_source_check_includes_mentions', () => {
|
||||
test('registered with expected version + name', () => {
|
||||
const m = MIGRATIONS.find(m => m.version === MIGRATION_VERSION);
|
||||
expect(m).toBeDefined();
|
||||
expect(m!.name).toBe(MIGRATION_NAME);
|
||||
});
|
||||
|
||||
test('LATEST_VERSION >= 95 so the migration is part of canonical sequence', () => {
|
||||
expect(LATEST_VERSION).toBeGreaterThanOrEqual(MIGRATION_VERSION);
|
||||
});
|
||||
|
||||
test('SQL shape — widens CHECK to include all 4 source values', () => {
|
||||
const m = MIGRATIONS.find(m => m.version === MIGRATION_VERSION)!;
|
||||
const sql = (m.sql || '') + ' ' + ((m.sqlFor?.pglite as string) || '');
|
||||
expect(sql).toContain("'mentions'");
|
||||
expect(sql).toContain("'markdown'");
|
||||
expect(sql).toContain("'frontmatter'");
|
||||
expect(sql).toContain("'manual'");
|
||||
// DROP IF EXISTS pattern for re-runnability
|
||||
expect(sql).toMatch(/DROP CONSTRAINT IF EXISTS links_link_source_check/i);
|
||||
});
|
||||
|
||||
test('PGLite branch present (engine parity)', () => {
|
||||
const m = MIGRATIONS.find(m => m.version === MIGRATION_VERSION)!;
|
||||
expect(m.sqlFor?.pglite).toBeDefined();
|
||||
expect(m.sqlFor!.pglite!.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('fresh-init brain (post-migration v95) accepts link_source=mentions', () => {
|
||||
test('two pages can be linked with link_source=mentions', async () => {
|
||||
const slugA = `mentions-source-${Math.random().toString(36).slice(2, 8)}`;
|
||||
const slugB = `mentions-target-${Math.random().toString(36).slice(2, 8)}`;
|
||||
await engine.putPage(slugA, {
|
||||
type: 'note',
|
||||
title: 'A',
|
||||
compiled_truth: 'a body',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.putPage(slugB, {
|
||||
type: 'person',
|
||||
title: 'B',
|
||||
compiled_truth: 'b body',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.addLinksBatch([
|
||||
{
|
||||
from_slug: slugA,
|
||||
to_slug: slugB,
|
||||
link_type: 'mentions',
|
||||
link_source: 'mentions',
|
||||
context: 'auto-link test',
|
||||
},
|
||||
]);
|
||||
const rows = await engine.executeRaw<{ link_source: string }>(
|
||||
`SELECT l.link_source
|
||||
FROM links l
|
||||
JOIN pages p ON p.id = l.from_page_id
|
||||
WHERE p.slug = $1`,
|
||||
[slugA],
|
||||
);
|
||||
expect(rows.some(r => r.link_source === 'mentions')).toBe(true);
|
||||
});
|
||||
|
||||
test('CHECK still rejects an unknown source value (widening did not nullify the gate)', async () => {
|
||||
const slugA = `bad-source-a-${Math.random().toString(36).slice(2, 8)}`;
|
||||
const slugB = `bad-source-b-${Math.random().toString(36).slice(2, 8)}`;
|
||||
await engine.putPage(slugA, { type: 'note', title: 'A', compiled_truth: 'a', timeline: '', frontmatter: {} });
|
||||
await engine.putPage(slugB, { type: 'person', title: 'B', compiled_truth: 'b', timeline: '', frontmatter: {} });
|
||||
// 'inferred' is NOT in allow-list ∪ {'mentions'} — must reject.
|
||||
await expect(
|
||||
engine.addLinksBatch([
|
||||
{
|
||||
from_slug: slugA,
|
||||
to_slug: slugB,
|
||||
link_type: 'mentions',
|
||||
link_source: 'inferred' as any,
|
||||
context: 'should reject',
|
||||
},
|
||||
]),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('idempotent re-application via runMigration — DROP IF EXISTS + ADD pattern survives second run', async () => {
|
||||
const m = MIGRATIONS.find(m => m.version === MIGRATION_VERSION)!;
|
||||
const pgliteSql = m.sqlFor!.pglite!;
|
||||
// runMigration uses engine.db.exec which handles multi-statement SQL,
|
||||
// unlike executeRaw which goes through db.query (single statement only).
|
||||
await expect(engine.runMigration(MIGRATION_VERSION, pgliteSql)).resolves.toBeUndefined();
|
||||
// Insert with link_source='mentions' must still work after re-running.
|
||||
const slugA = `idem-a-${Math.random().toString(36).slice(2, 8)}`;
|
||||
const slugB = `idem-b-${Math.random().toString(36).slice(2, 8)}`;
|
||||
await engine.putPage(slugA, { type: 'note', title: 'A', compiled_truth: 'a', timeline: '', frontmatter: {} });
|
||||
await engine.putPage(slugB, { type: 'company', title: 'B', compiled_truth: 'b', timeline: '', frontmatter: {} });
|
||||
await engine.addLinksBatch([
|
||||
{ from_slug: slugA, to_slug: slugB, link_type: 'mentions', link_source: 'mentions', context: '' },
|
||||
]);
|
||||
const rows = await engine.executeRaw<{ count: string }>(
|
||||
`SELECT COUNT(*)::text AS count FROM links l
|
||||
JOIN pages p ON p.id = l.from_page_id
|
||||
WHERE p.slug = $1 AND l.link_source = 'mentions'`,
|
||||
[slugA],
|
||||
);
|
||||
expect(Number(rows[0]?.count ?? 0)).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,143 @@
|
||||
/**
|
||||
* Unit tests for src/core/text-safe.ts — UTF-16 surrogate-safe helpers.
|
||||
*
|
||||
* Covers all 3 surrogate cases (high+low pair straddle, stray high,
|
||||
* AT-low) plus boundary-after-pair (codex CK16 regression evidence).
|
||||
*
|
||||
* `truncateUtf8` regressions live here AND in
|
||||
* `test/eval-contradictions-judge.test.ts` (which imports it through
|
||||
* judge.ts re-export, proving the move is byte-equivalent).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { truncateUtf8, safeSplitIndex } from '../src/core/text-safe.ts';
|
||||
|
||||
// 🚀 = U+1F680 (surrogate pair: 0xD83D 0xDE80; JS string length = 2).
|
||||
// 𝕏 = U+1D54F (surrogate pair: 0xD835 0xDD4F; JS string length = 2).
|
||||
// 𠀀 = U+20000 (non-BMP CJK: 0xD840 0xDC00; JS string length = 2).
|
||||
const ROCKET = '🚀'; // 🚀
|
||||
const MATH_X = '𝕏'; // 𝕏
|
||||
const NBMP_HAN = '𠀀'; // 𠀀
|
||||
const STRAY_HIGH = '\uD83D'; // orphaned high surrogate (invalid alone)
|
||||
|
||||
describe('truncateUtf8', () => {
|
||||
test('returns empty for empty input', () => {
|
||||
expect(truncateUtf8('', 100)).toBe('');
|
||||
});
|
||||
|
||||
test('returns unchanged when already under limit', () => {
|
||||
expect(truncateUtf8('short', 100)).toBe('short');
|
||||
});
|
||||
|
||||
test('truncates at ASCII boundary', () => {
|
||||
expect(truncateUtf8('hello world', 5)).toBe('hello');
|
||||
});
|
||||
|
||||
test('case 1: pair straddles cut — drops both halves', () => {
|
||||
// text="a🚀b" (length 4: ['a', 0xD83D, 0xDE80, 'b']). Cut at maxChars=2
|
||||
// would split between 0xD83D and 0xDE80. Expect "a" (length 1).
|
||||
const out = truncateUtf8('a' + ROCKET + 'b', 2);
|
||||
expect(out).toBe('a');
|
||||
});
|
||||
|
||||
test('case 2: stray high surrogate at end-1 — drops it', () => {
|
||||
// text="ab<HIGH>cd". Cut at maxChars=3 lands ON the high; unitBefore
|
||||
// is 'b' (regular). Cut at maxChars=4 lands on 'c'; unitBefore is
|
||||
// the high surrogate AND unitAtEnd is 'c' (not low) → stray high
|
||||
// case. Expect "ab".
|
||||
const text = 'ab' + STRAY_HIGH + 'cd';
|
||||
const out = truncateUtf8(text, 3); // index 3 → unitBefore=HIGH, unitAtEnd='c'
|
||||
expect(out).toBe('ab');
|
||||
});
|
||||
|
||||
test('case 3: low surrogate at end-1 — backs up two (intentionally conservative)', () => {
|
||||
// text="ab🚀cd" (length 6). Cut at maxChars=4: unitBefore=0xDE80 (low),
|
||||
// unitAtEnd='c'. Case 3 fires → end = 4-2 = 2. Expect "ab".
|
||||
// Note: pair was COMPLETE in kept half at [2,3]; conservative back-up
|
||||
// drops it. Intentional — matches truncateUtf8's pre-extraction behavior
|
||||
// verbatim. See safeSplitIndex doc for rationale.
|
||||
const text = 'ab' + ROCKET + 'cd';
|
||||
const out = truncateUtf8(text, 4);
|
||||
expect(out).toBe('ab');
|
||||
});
|
||||
|
||||
test('non-BMP CJK pair behaves identically to emoji', () => {
|
||||
const text = 'x' + NBMP_HAN + 'y';
|
||||
expect(truncateUtf8(text, 2)).toBe('x'); // case 1: cut splits the pair
|
||||
});
|
||||
|
||||
test('multiple consecutive pairs preserved when cut is past them', () => {
|
||||
const text = ROCKET + MATH_X + 'tail';
|
||||
// Cut at maxChars=10 ≥ length=8 → returns full text.
|
||||
expect(truncateUtf8(text, 10)).toBe(text);
|
||||
});
|
||||
});
|
||||
|
||||
describe('safeSplitIndex', () => {
|
||||
test('maxChars ≤ 0 returns 0', () => {
|
||||
expect(safeSplitIndex('any', 0)).toBe(0);
|
||||
expect(safeSplitIndex('any', -5)).toBe(0);
|
||||
});
|
||||
|
||||
test('maxChars ≥ text.length returns text.length', () => {
|
||||
expect(safeSplitIndex('abc', 3)).toBe(3);
|
||||
expect(safeSplitIndex('abc', 100)).toBe(3);
|
||||
});
|
||||
|
||||
test('maxChars in middle of ASCII text returns maxChars unchanged', () => {
|
||||
expect(safeSplitIndex('hello world', 5)).toBe(5);
|
||||
});
|
||||
|
||||
test('case 1: pair straddles cut → returns maxChars-1', () => {
|
||||
// text="a🚀b" (length 4). maxChars=2: unitBefore=0xD83D, unitAtEnd=0xDE80.
|
||||
expect(safeSplitIndex('a' + ROCKET + 'b', 2)).toBe(1);
|
||||
});
|
||||
|
||||
test('case 2: stray high surrogate at maxChars-1 → returns maxChars-1', () => {
|
||||
// text="ab<HIGH>cd". maxChars=3: unitBefore=HIGH, unitAtEnd='c'.
|
||||
const text = 'ab' + STRAY_HIGH + 'cd';
|
||||
expect(safeSplitIndex(text, 3)).toBe(2);
|
||||
});
|
||||
|
||||
test('case 3: low at maxChars-1 → returns maxChars-2 (conservative)', () => {
|
||||
// text="ab🚀cd" (length 6). maxChars=4: unitBefore=0xDE80 (low), unitAtEnd='c'.
|
||||
// Pair COMPLETE in kept half; back-up is intentional per truncateUtf8 parity.
|
||||
expect(safeSplitIndex('ab' + ROCKET + 'cd', 4)).toBe(2);
|
||||
});
|
||||
|
||||
test('boundary-immediately-after-pair returns maxChars-2 (codex CK16 documented)', () => {
|
||||
// Codex flagged this as "safe but overly conservative." We test the
|
||||
// CURRENT conservative behavior so any future change is intentional.
|
||||
// text="hello🚀" (length 7). maxChars=7 ≥ length → returns 7 (full).
|
||||
// text="hello🚀x" (length 8). maxChars=7: unitBefore=0xDE80 (low), unitAtEnd='x'.
|
||||
// Case 3 fires → returns 5. Documents the conservative back-up.
|
||||
const text = 'hello' + ROCKET + 'x'; // length 8
|
||||
expect(safeSplitIndex(text, 7)).toBe(5);
|
||||
});
|
||||
|
||||
test('determinism: same input → same output across 100 calls', () => {
|
||||
const text = 'lorem ' + ROCKET + ' ipsum ' + MATH_X + ' dolor ' + NBMP_HAN;
|
||||
const refs = new Set<number>();
|
||||
for (let i = 0; i < 100; i++) refs.add(safeSplitIndex(text, 12));
|
||||
expect(refs.size).toBe(1);
|
||||
});
|
||||
|
||||
test('empty text returns 0', () => {
|
||||
expect(safeSplitIndex('', 5)).toBe(0);
|
||||
});
|
||||
|
||||
test('truncateUtf8 and safeSplitIndex agree on slice length', () => {
|
||||
// Property check: truncateUtf8(text, n).length === safeSplitIndex(text, n)
|
||||
// (modulo the empty-string early return which both treat identically).
|
||||
const cases: Array<[string, number]> = [
|
||||
['hello world', 5],
|
||||
['a' + ROCKET + 'b', 2],
|
||||
['ab' + STRAY_HIGH + 'cd', 3],
|
||||
['ab' + ROCKET + 'cd', 4],
|
||||
['hello' + ROCKET + 'x', 7],
|
||||
];
|
||||
for (const [text, n] of cases) {
|
||||
expect(truncateUtf8(text, n).length).toBe(safeSplitIndex(text, n));
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user