diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index ab0be6a12..4432bd9e5 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -311,7 +311,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `scripts/check-gateway-routed-no-direct-anthropic.sh` — CI guard that fails the build if `src/core/cycle/synthesize.ts` or `src/core/think/index.ts` reintroduces a runtime `new Anthropic()` constructor call or a value-shaped `import Anthropic from '@anthropic-ai/sdk'` import. Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay allowed for adapter types; comment lines (`//` or ` *` prefixes) are excluded so historical JSDoc doesn't false-fire. Mirrors `scripts/check-jsonb-pattern.sh`. Wired into `bun run verify` and `bun run check:all`. Extend `GUARDED_FILES` when migrating another file off direct SDK construction. - `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh. - `src/core/cycle/extract-facts.ts` — extract_facts cycle phase. Fence is canonical: per-page wipe (`deleteFactsForPage`) + reinsert from `parseFactsFence` + `extractFactsFromFenceText` + `engine.insertFacts`. #1928: the per-page wipe passes `excludeSourcePrefixes: ['cli:']` so conversation facts (written by `extract-conversation-facts`, on pages with NO `## Facts` fence to recreate them from) survive the reconcile instead of being deleted-with-nothing-to-reinsert. The destructive phase no longer inherits a failed sync's full-brain walk: `slugs: []` (a real incremental no-op) is distinguished from `slugs: undefined` (full-walk intent) by presence, not length. `runPhaseExtractFacts` (cycle.ts) surfaces a `warn` (`net_fact_deletion`) when the reconcile deletes at least `NET_DELETION_WARN_FLOOR` (50) more facts than it reinserts — the exact signature of the conversation-facts wipe, which previously read as a silent `ok`. Empty-fence guard refuses when legacy rows (`row_num IS NULL AND entity_slug IS NOT NULL`) pend backfill (status: warn, hint: `gbrain apply-migrations --yes`). A phantom-redirect pre-pass runs AFTER the legacy-row guard, BEFORE the main reconcile loop: when `opts.brainDir` is set, `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun)` walks unprefixed-slug pages capped by `GBRAIN_PHANTOM_REDIRECT_LIMIT` (default 50). The pass returns `touched_canonicals` — canonical slugs whose disk fence merged with phantom rows; `runExtractFacts` UNIONs them into the main reconcile slug set so canonical's DB facts derive from the merged fence in the same cycle (handles phantom-had-only-on-disk-fence). `ExtractFactsResult` gains six phantom fields: `phantomsScanned`, `phantomsRedirected`, `phantomsAmbiguous`, `phantomsSkippedDrift`, `phantomsLockBusy`, `phantomsMorePending`. Three bubble to `CycleReport.totals` (`phantoms_redirected`, `phantoms_ambiguous`, `phantoms_skipped_drift`). -- `src/core/entities/resolve.ts` — Free-form entity name → canonical slug resolution. `resolveEntitySlug(engine, source_id, raw)`: exact slug → fuzzy (pg_trgm @ 0.4 threshold) → bare-name prefix expansion (`people/-%` then `companies/-%`, `connection_count` correlated-subquery tiebreaker) → deterministic `slugify` fallback. Two helpers for the phantom-redirect pass: `resolvePhantomCanonical(engine, sourceId, phantomSlug)` SKIPS the exact-slug step (a phantom slug `'alice'` would exact-match itself and no-op the redirect); returns the canonical only when non-null AND contains `/`. `findPrefixCandidates(engine, sourceId, token)` is a standalone SQL query returning ALL candidates across `PREFIX_EXPANSION_DIRS` (hardcoded `['people', 'companies']`) via `slug LIKE ANY($N::text[])` over patterns `dir/token` + `dir/token-%`, cap of 10 ordered by `connection_count DESC, slug ASC` — NOT a wrapper around `tryPrefixExpansion` (that path returns per-dir top-1 and suppresses ambiguity by design). Pinned by `test/phantom-redirect.test.ts` (resolvePhantomCanonical 3 cases + findPrefixCandidates 6 cases incl. multi-dir ambiguity and the `people/aliceberg`-doesn't-match-`alice` false-positive guard). +- `src/core/entities/resolve.ts` — Free-form entity name → canonical slug resolution. `resolveEntitySlug(engine, source_id, raw)`: exact slug → unambiguous bare-name prefix expansion across `people/-%` + `companies/-%` → high-specificity fuzzy match for multi-token input (pg_trgm @ 0.7 threshold) → deterministic `slugify` holding fallback. Bare-name collisions never use popularity as confidence; shared-token company names below the threshold remain unresolved. Two helpers for the phantom-redirect pass: `resolvePhantomCanonical(engine, sourceId, phantomSlug)` SKIPS the exact-slug step (a phantom slug `'alice'` would exact-match itself and no-op the redirect); returns the canonical only when non-null AND contains `/`. `findPrefixCandidates(engine, sourceId, token)` is a standalone SQL query returning ALL candidates across `PREFIX_EXPANSION_DIRS` (hardcoded `['people', 'companies']`) via `slug LIKE ANY($N::text[])` over patterns `dir/token` + `dir/token-%`, cap of 10 ordered by `connection_count DESC, slug ASC`. Pinned by `test/entity-resolve.test.ts` (explicit, unique, ambiguous-person, and shared-token-company cases) plus `test/phantom-redirect.test.ts` (resolvePhantomCanonical 3 cases + findPrefixCandidates 6 cases incl. multi-dir ambiguity and the `people/aliceberg`-doesn't-match-`alice` false-positive guard). - `src/core/cycle/phantom-redirect.ts` — Phantom-redirect orchestrator. Exports `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun): Promise` (per-cycle wrapper acquiring the `gbrain-sync` writer lock once for the whole pass, 30s bounded retry, walks up to `GBRAIN_PHANTOM_REDIRECT_LIMIT` unprefixed phantoms) + `tryRedirectPhantom(engine, page, sourceId, brainDir, dryRun): Promise` + `stripFenceAndFrontmatterAndLeadingH1` (pure body-shape gate helper — strips facts fence incl. preceding `## Facts` heading and the leading H1; zero residue = phantom). Handler order: body-shape gate → `resolvePhantomCanonical` (bypasses exact-self-match) → `findPrefixCandidates` ambiguity check → `fenceDbDrift` bi-directional check → dry-run early exit → materialize canonical via `serializeMarkdown` if DB-only → append phantom fence rows to canonical's disk fence with `(claim, valid_from)` dedup-guard + row_num continuation → `engine.refreshPageBody` with SHA-256 content_hash recomputed via the import-file shape → `engine.migrateFactsToCanonical` (lossless) → `engine.rewriteLinks` (DB FK rewrite; wiki-link text rewrite is a documented follow-up) → `engine.softDeletePage` + `engine.deleteFactsForPage(phantom)` + `fs.unlinkSync(phantomPath)`. `RedirectResult.canonical` populated on `'redirected'` (incl. dry-run preview) so the caller builds `touched_canonicals`. Idempotent on re-run: phantom soft-deleted → predicate fails (`deleted_at IS NULL`); migrate UPDATE matches no rows; dedup-guard prevents double-append. - `src/core/facts/phantom-audit.ts` — JSONL audit at `${resolveAuditDir()}/phantoms-YYYY-Www.jsonl`. Pattern copy of `src/core/audit-slug-fallback.ts` (ISO-week rotation, honors `GBRAIN_AUDIT_DIR`). Exports `logPhantomEvent(record)` + `readRecentPhantomEvents(days)` + `computePhantomAuditFilename(now?)`. Records every outcome: `redirected | ambiguous | drift | no_canonical | not_phantom_has_residue | pass_skipped_lock_busy`. Best-effort writes — stderr warn on failure, never throws. Separate file from `stub-guard-audit.ts` (distinct consumer + lifecycle: stub-guard logs PREVENTIVE blocks; phantom-audit logs CLEANUP decisions, to be read by a future `phantoms_pending` doctor check). - `src/core/cycle/emotional-weight.ts` — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder `'garry'`). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; override via config `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`. diff --git a/src/core/entities/resolve.ts b/src/core/entities/resolve.ts index 89eb6ea16..5aed381f5 100644 --- a/src/core/entities/resolve.ts +++ b/src/core/entities/resolve.ts @@ -10,15 +10,15 @@ * Lives under `src/core/entities/` so signal-detector can reuse it for the * Sonnet pass too without circular import through facts/. * - * Prefix-expansion step lives between fuzzy match and slugify fallback. + * Bare-name prefix expansion lives before fuzzy match and slugify fallback. * Bare first names like "Alice" score too low on pg_trgm (short strings * have terrible trigram overlap), so without this step they fall through * to slugify("Alice") → "alice", which spawns a phantom `people/alice.md` * stub at brain root instead of resolving to the existing * `people/alice-example` page. The fix queries `slug LIKE 'people/X-%'` * (then `companies/X-%`) when fuzzy fails on a single-word bare name, and - * uses connection count (links + chunks) as the tiebreaker when multiple - * candidates match. + * resolves only a single candidate; collisions fall through to the guarded + * unprefixed holding path rather than guessing from connection count. */ import type { BrainEngine } from '../engine.ts'; @@ -29,9 +29,10 @@ import type { BrainEngine } from '../engine.ts'; * Resolution order: * 1. If `raw` is already a page slug shape (contains a "/" or matches an * exact pages.slug row in this source), return it untouched. - * 2. Try fuzzy match against pages.slug + pages.title within the source - * (case-insensitive). Pick the highest-trgm-score match if any. - * 3. Fall back to a deterministic slugify: lowercase-no-spaces with + * 2. Resolve a bare name only when prefix expansion finds one candidate. + * 3. For multi-token input, require a high-specificity fuzzy match against + * pages.slug + pages.title within the source (case-insensitive). + * 4. Fall back to a deterministic slugify: lowercase-no-spaces with * hyphen-collapse. NOT prefixed with a directory — caller decides * whether to prefix `people/`, `companies/`, etc. * @@ -54,21 +55,22 @@ export async function resolveEntitySlug( if (exact) return exact; } - // 2. Fuzzy match against existing pages within the source. Match either - // on slug fragment or on title. - const fuzzy = await tryFuzzyMatch(engine, source_id, trimmed); - if (fuzzy) return fuzzy; - - // 3. Prefix-expansion match: when the input looks like a bare first name + // 2. Prefix-expansion match: when the input looks like a bare first name // (no slash, no prefix, slugifies to a single short token), try // `people/-%` then `companies/-%`. Short bare names // score terribly on pg_trgm — similarity('alice', 'alice-example') - // is below the 0.4 threshold — so this is the layer that catches + // is below the fuzzy threshold — so this is the layer that catches // `"Alice"` → `people/alice-example` before we phantom-stub a bare // `people/alice.md`. if (isBareName(trimmed)) { - const expanded = await tryPrefixExpansion(engine, source_id, slugify(trimmed)); + const expanded = await tryUnambiguousPrefixExpansion(engine, source_id, slugify(trimmed)); if (expanded) return expanded; + } else { + // 3. Fuzzy match against existing pages within the source. Bare names + // deliberately skip this arm: a shared first name is not specific + // enough to choose one person by trigram score or popularity. + const fuzzy = await tryFuzzyMatch(engine, source_id, trimmed); + if (fuzzy) return fuzzy; } // 4. Fallback: deterministic slugify. @@ -131,12 +133,12 @@ export async function resolveEntitySlugWithSource( if (exact) return { slug: exact, source: 'exact_page' }; } - const fuzzy = await tryFuzzyMatch(engine, source_id, trimmed); - if (fuzzy) return { slug: fuzzy, source: 'fuzzy_match' }; - if (isBareName(trimmed)) { - const expanded = await tryPrefixExpansion(engine, source_id, slugify(trimmed)); + const expanded = await tryUnambiguousPrefixExpansion(engine, source_id, slugify(trimmed)); if (expanded) return { slug: expanded, source: 'fuzzy_match' }; + } else { + const fuzzy = await tryFuzzyMatch(engine, source_id, trimmed); + if (fuzzy) return { slug: fuzzy, source: 'fuzzy_match' }; } return { slug: slugify(trimmed), source: 'fallback_slugify' }; @@ -240,6 +242,15 @@ export async function findPrefixCandidates( } } +async function tryUnambiguousPrefixExpansion( + engine: BrainEngine, + source_id: string, + token: string, +): Promise { + const candidates = await findPrefixCandidates(engine, source_id, token); + return candidates.length === 1 ? candidates[0].slug : null; +} + /** * Look up pages whose slug starts with `/-` for each known * entity directory. When multiple candidates match within a directory, @@ -359,7 +370,11 @@ async function tryFuzzyMatch( LIMIT 3`, [source_id, lc, fragment], ); - if (rows.length > 0 && rows[0].score >= 0.4) return rows[0].slug; + // 0.4 confidently misattributes names that share only a generic company + // token (for example "Beacon Capital" → "Benton Capital"). Keep fuzzy + // typo tolerance, but require high-specificity overlap before writing a + // fact to an existing entity. + if (rows.length > 0 && rows[0].score >= 0.7) return rows[0].slug; } catch { // pg_trgm functions might not be available on every engine config; // fall through to slugify. diff --git a/test/entity-resolve.test.ts b/test/entity-resolve.test.ts index 8bd129fde..2b67fde09 100644 --- a/test/entity-resolve.test.ts +++ b/test/entity-resolve.test.ts @@ -29,8 +29,8 @@ beforeAll(async () => { // Seed test pages. Naming pattern: // - alice-example: single-match case (only people/alice-*) - // - bob-example vs bob-rosenstein: multi-match tiebreaker (bob-example wins on connections) - // - charlie-example vs charlie-bankcroft: multi-match tiebreaker (charlie-example wins on connections) + // - bob-example vs bob-rosenstein: ambiguous bare-name collision + // - charlie-example vs charlie-bankcroft: ambiguous bare-name collision // - dave-example: single-match case const pages = [ { slug: 'people/alice-example', title: 'Alice Example', type: 'person' }, @@ -41,6 +41,7 @@ beforeAll(async () => { { slug: 'people/dave-example', title: 'Dave Example', type: 'person' }, { slug: 'companies/stripe', title: 'Stripe', type: 'company' }, { slug: 'companies/stripe-atlas', title: 'Stripe Atlas', type: 'company' }, + { slug: 'companies/benton-capital', title: 'Benton Capital', type: 'company' }, ]; for (const p of pages) { @@ -113,14 +114,14 @@ describe('resolveEntitySlug — prefix expansion', () => { expect(result).toBe('people/alice-example'); }); - it('resolves "Bob" to people/bob-example (more connections)', async () => { + it('refuses to choose between people sharing the same bare first name', async () => { const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Bob'); - expect(result).toBe('people/bob-example'); + expect(result).toBe('bob'); }); - it('resolves "Charlie" to people/charlie-example (more connections)', async () => { + it('does not use connection count to turn bare-name ambiguity into confidence', async () => { const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Charlie'); - expect(result).toBe('people/charlie-example'); + expect(result).toBe('charlie'); }); it('resolves "Dave" to people/dave-example (single match)', async () => { @@ -145,11 +146,26 @@ describe('resolveEntitySlug — prefix expansion', () => { expect(result).toContain('alice-example'); }); + it('preserves a high-specificity multi-token typo match', async () => { + const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Alice Exampl'); + expect(result).toBe('people/alice-example'); + }); + it('hyphenated input does NOT trigger prefix expansion', async () => { const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'alice-example'); expect(result).toBe('people/alice-example'); }); + it('preserves an explicit full company-name match', async () => { + const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Benton Capital'); + expect(result).toBe('companies/benton-capital'); + }); + + it('refuses a company match supported mainly by a shared generic token', async () => { + const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Beacon Capital'); + expect(result).toBe('beacon-capital'); + }); + it('returns null for empty input', async () => { const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', ''); expect(result).toBeNull();