Files
gbrain/src/core/schema-pack/load-active.ts
T
5d42f3295e v0.41.22.0 feat: type-unification cathedral — 94 types → 15 canonical (closes #1479) (#1542)
* Merge branch 'master' into garrytan/type-taxonomy-unification

Resolve VERSION, package.json, CHANGELOG conflicts with v0.41.22.0
on top, preserving master's v0.41.19.0 entry below.

* feat: v0.41.22.0 type-unification cathedral — collapse 94 types to 15 (closes #1479)

Ships gbrain-base-v2 as the new install default (15 canonical types: 14
+ note catch-all) and the unify-types PROTECTED Minion handler that
runs the gbrain-base→v2 migration end-to-end on existing brains.

What this delivers:
- gbrain-base-v2.yaml standalone schema pack (no extends:) with 14
  canonical page_types + 9 cluster mapping_rules + catch-all sentinel
- 3 new schema-pack primitives: runRetypeCore (chunked UPDATE with
  legacy_type stamping), runPageToLinkCore (edge-shaped pages →
  link rows), runPageToAliasCore (concept-redirect → slug_aliases)
- rewriteLinksBatch for N-pair atomic FK rewrite
- Migration v104 slug_aliases table (forward-bootstrap probed on both
  engines for safe upgrade chain)
- New engine method resolveSlugWithAlias(slug, sourceOrSources) on
  both Postgres + PGLite with multi-source ambiguity warning
- inferTypeAndSubtypeFromPack overload + subtypes: + mapping_rules:
  + migration_from: schema-pack manifest extensions
- findPackSuccessors version-range walker (1.x / 1.0.x / exact match)
- expandTypeFilter for --type back-compat (D14): legacy aliases route
  through mapping_rules → canonical+subtype before the SQL filter fires
- 3 new onboard checks: pack_upgrade_available, type_proliferation,
  dangling_aliases (source-scoped per F12)
- unify-types Minion handler (PROTECTED, manual_only via render.ts
  allowlist per D17): retype-explicit → retype-catch-all →
  page-to-link → page-to-alias → final sync → active-pack flip
- alias_resolved 1.05x post-fusion search boost stage; KNOBS_HASH_VERSION
  bumped 5→6 (one-time cache miss on upgrade, self-healing in TTL)
- ELIGIBLE_TYPES for facts extraction extended with v2 canonicals
  (codex F-ELIGIBLE: blocker not v0.43 follow-up)

Tests: 79 new unit/integration cases + 3 E2E cases covering all 9
production clusters end-to-end. 124-case verification on the cache-key
+ build-llms fixes. KNOBS_HASH_VERSION assertions updated in 3 tests.

Plan: ~/.claude/plans/system-instruction-you-are-working-transient-elephant.md
(16 locked decisions D1-D17, 12 baseline fixes F7-F21 absorbed from
codex outside voice).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: CI verify failures — system-of-record allow-comment + schema-unify manifest registration

Two CI failures on PR #1542:

1. check:system-of-record flagged page-to-link.ts:207 addLinksBatch as
   a direct write to a derived table. The call IS the reconcile surface
   for page_to_link mapping_rules — it converts edge-shaped pages into
   canonical link rows under the PROTECTED unify-types Minion handler,
   source-scoped, atomic per-rule. Added the canonical
   `// gbrain-allow-direct-insert: <reason>` comment on the same line.

2. check:resolver emitted 11 orphan_trigger warnings for `schema-unify`
   because the skill was added to skills/RESOLVER.md without a
   corresponding entry in skills/manifest.json. Added the registration
   under the existing skills[] array.

bun run verify: 28/28 checks pass locally.

* fix: CI test failures — schema-unify conformance + eligibility regression

Six test failures across shards 2 + 10 on PR #1542:

1. resolver.test.ts: round-trip parser requires frontmatter triggers to
   be quoted (`- "..."` or `- '...'`). schema-unify shipped with bare
   YAML strings; quoted the 10 triggers to round-trip correctly.

2. skills-conformance.test.ts (×3): schema-unify SKILL.md was missing
   the required Contract, Anti-Patterns, and Output Format sections
   that every conformant skill must declare. Added all three:
   - Contract: inputs / outputs / side effects / failure modes
   - Anti-Patterns: 5 DON'Ts including the autopilot trust boundary
   - Output Format: per-phase stderr lines + celebration summary +
     JSON envelope shape

3. facts-eligibility.test.ts (×2): the v0.41.22 ELIGIBLE_TYPES
   expansion added `concept` to the eligible list, but the existing
   test suite pins concept as rejected (it's `extractable: true` in
   the schema pack but the v0.41.11 contract documented this as
   "cosmetic on the backstop path because backstop uses hardcoded
   ELIGIBLE_TYPES"). Removed `concept` from the expansion; other v2
   canonicals (media, tweet, atom, analysis) stay. Comment updated
   to document the deliberate omission.

All 6 failing tests now pass locally (370/370 across the 3 affected
files). bun run verify: 28/28 checks green.

* fix: harden findPackSuccessors test against shard pollution

CI shard 8 reported 1 fail (1.00ms — too fast for any real loadActivePack
file I/O) on `finds gbrain-base-v2 as successor of gbrain-base@1.0.0`.
Local triple-run passes 9/9 in isolation.

Root cause: the existing afterEach reset clears the module-level pack
cache AFTER each test, but the FIRST test in the file inherits whatever
state sibling files in the same bun shard process left behind. With
24+ schema-pack tests in shard 8 (mutate, mutate-audit, best-effort,
registry-reload, manifest-v041_2, etc.) running before this file, the
first test can read a poisoned cache.

Fix: add `beforeEach(_resetPackCacheForTests)`. Two-sided reset
guarantees clean state regardless of file ordering within the shard.

bun run verify: 28/28 checks pass.

* fix: quarantine two flaky tests to serial runner

CI shard 1 + shard 8 each surfaced one intermittent failure:

shard 1: buildBrainTools > execute() on put_page with valid namespace
shard 8: findPackSuccessors > finds gbrain-base-v2 as successor

Both pass cleanly in isolation. Both are concurrency races against
shared in-shard state:

- brain-allowlist.test.ts shares a singleton PGLiteEngine across 18
  tests with a beforeEach DELETE FROM pages. With max-concurrency=4,
  two put_page tests can interleave their TRUNCATE + write phases,
  so the auto-link/extract sub-steps inside put_page race against
  the sibling test's DELETE.
- schema-pack-find-pack-successors.test.ts reads bundled YAML packs
  via loadActivePack. The module-level pack cache is shared across
  parallel tests in the same shard; the previous beforeEach reset
  helped but didn't fully isolate against concurrent file reads
  under CI load.

Fix per CLAUDE.md test-isolation lint rule R2 (concurrency-fragile
files belong in the .serial.test.ts quarantine): rename both files
to *.serial.test.ts. Serial runner picks them up at max-concurrency=1.
49/49 serial files pass locally. 28/28 verify checks pass.

* fix: quarantine embed-stale test to serial runner

CI shard 9 reported 6 failures, all from the embedStaleForSource describe
block, all ~120-150ms each — classic shared-engine concurrency race shape.
Passes 7/7 locally in isolation.

Root cause: embed-stale.test.ts shares a singleton PGLiteEngine across 7
tests with beforeEach resetPgliteState. Under bun's max-concurrency=4 in
the parallel shard, two tests can interleave their TRUNCATE + seedPage +
upsertChunks + embedStaleForSource flow, so one test's stale-chunk count
sees another test's mid-flight writes.

Same fix as brain-allowlist.serial.test.ts and
schema-pack-find-pack-successors.serial.test.ts: rename to *.serial.test.ts
so the serial runner picks it up at max-concurrency=1.

bun run verify: 28/28 checks pass. 7/7 embed-stale tests pass via serial.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 07:01:28 -07:00

296 lines
12 KiB
TypeScript

// v0.38 active-pack loader — the boundary helper Phase B consumers call.
//
// Composes:
// 1. Resolution chain (registry.resolveActivePackName) — 7 tiers per D13
// 2. Pack manifest loading from disk:
// - Built-in 'gbrain-base' lives at src/core/schema-pack/base/gbrain-base.yaml
// - User packs live at ~/.gbrain/schema-packs/<name>/pack.yaml
// - Custom paths supported via `__setPackLocatorForTests` (test seam)
// 3. `extends` chain resolution (registry.resolvePack)
//
// Result: a `ResolvedPack` keyed by pack identity (sha8-stable). Cached
// in-process; cache invalidated by manifest content changes (sha8 mismatch).
//
// Trust gate: per-call schema_pack opt is honored ONLY when
// `ctx.remote === false`. Remote/MCP callers passing schema_pack get
// `permission_denied` BEFORE this is invoked — operations.ts handles the
// rejection at the dispatch layer. This helper assumes the input is
// already-trust-vetted.
//
// Test seam: `__setPackLocatorForTests` replaces the disk-loader so unit
// tests can drive the boundary helper with synthetic packs without
// writing to `~/.gbrain/schema-packs/`.
import { existsSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import type { GBrainConfig } from '../config.ts';
import { gbrainPath } from '../config.ts';
import type { SchemaPackManifest } from './manifest-v1.ts';
import { loadPackFromFile } from './loader.ts';
import {
resolveActivePackName,
resolvePack,
tryCachedPack,
UnknownPackError,
type ResolvedPack,
type ResolutionInput,
type ResolutionResult,
} from './registry.ts';
/**
* Inputs the caller (operations.ts handler / engine query path) provides.
* Most callers only need `cfg` + `remote`; thin-client + source-aware
* ops pass additional fields.
*/
export interface LoadActivePackInput {
/** Loaded GBrain config (file + env merged). Pass null for default-only resolution. */
cfg: GBrainConfig | null;
/** Tier-1 trust gate: false for CLI, true for MCP/OAuth callers. */
remote: boolean;
/** Tier-1 per-call opt. Honored only when remote=false. */
perCall?: string;
/** Tier-3 per-source query target. */
sourceId?: string;
/** Tier-3 per-source DB config map (from `gbrain config get` keyspace). */
perSourceDb?: ReadonlyMap<string, string>;
/** Tier-5 gbrain.yml schema.pack field (already parsed by storage-config). */
gbrainYml?: string;
/** Tier-4 brain-wide DB config (overrides tier 6 file-plane). */
dbConfig?: string;
}
/**
* Test seam — a function that maps a pack name to the file path on disk.
* Production wires this to the built-in + ~/.gbrain/schema-packs lookup.
* Tests inject a Map-backed locator.
*/
export type PackLocator = (name: string) => string | null;
let _packLocator: PackLocator = defaultPackLocator;
/**
* Replace the pack locator. Tests use this to inject synthetic packs
* without writing to ~/.gbrain. Always pair with `_resetPackLocatorForTests`
* in afterAll to avoid leaking across files.
*/
export function __setPackLocatorForTests(locator: PackLocator): void {
_packLocator = locator;
}
/** Reset to the default disk-backed locator. */
export function _resetPackLocatorForTests(): void {
_packLocator = defaultPackLocator;
}
/**
* Default pack locator: maps a pack name to its filesystem path.
* 'gbrain-base' → bundled src/core/schema-pack/base/gbrain-base.yaml
* other → ~/.gbrain/schema-packs/<name>/pack.yaml or pack.json
*
* Returns null when the pack is not found. Callers handle null by
* throwing UnknownPackError with a paste-ready install hint.
*/
function defaultPackLocator(name: string): string | null {
// v0.39 T8 — bundled packs registry. gbrain-base + gbrain-recommended
// ship in src/core/schema-pack/base/. Add a new entry here to bundle
// additional canonical packs.
//
// v0.41 T4 — lens packs join the bundle: creator (atoms + concepts +
// extract_atoms/synthesize_concepts phases), investor (theses + bet
// resolution + 3 calibration domains), engineer (gstack-learnings bridge
// + 3 calibration domains), everything (meta-pack stacking all three
// via extends + borrow_from). Each ships as a real YAML at base/<name>.yaml.
const BUNDLED: ReadonlyArray<string> = [
'gbrain-base',
'gbrain-recommended',
'gbrain-creator',
'gbrain-investor',
'gbrain-engineer',
'gbrain-everything',
// v0.42 type-unification: 15-type canonical successor to gbrain-base.
// Ships as install default (Lane E T17) + via gbrain onboard pack
// upgrade flow (the unify-types Minion handler).
'gbrain-base-v2',
];
if (BUNDLED.includes(name)) {
// Resolve bundled YAML relative to this source file. Works in both
// direct-bun execution and bun --compile binaries.
const here = dirname(fileURLToPath(import.meta.url));
const bundledPath = join(here, 'base', `${name}.yaml`);
if (existsSync(bundledPath)) return bundledPath;
// Repo-root fallback for tests running from a worktree where the
// module path doesn't resolve to the source tree.
const repoRootFallback = join(here, '..', '..', '..', 'src', 'core', 'schema-pack', 'base', `${name}.yaml`);
if (existsSync(repoRootFallback)) return repoRootFallback;
return null;
}
// User-installed pack at ~/.gbrain/schema-packs/<name>/pack.{yaml,json}
const baseDir = gbrainPath('schema-packs', name);
const candidates = ['pack.yaml', 'pack.yml', 'pack.json'];
for (const c of candidates) {
const candidate = join(baseDir, c);
if (existsSync(candidate)) return candidate;
}
return null;
}
/**
* Load + parse + validate a pack by name. Used by `resolvePack` to walk
* the extends chain. Throws UnknownPackError when the pack isn't on disk.
*/
async function loadPackManifestByName(name: string): Promise<SchemaPackManifest> {
const path = _packLocator(name);
if (!path) {
throw new UnknownPackError(name);
}
return loadPackFromFile(path);
}
/**
* The boundary helper. Resolves the active pack identity, loads the
* manifest from disk, walks the extends chain, builds the alias graph
* + closure hash, and returns the cached ResolvedPack.
*
* Throws:
* - UnknownPackError if the resolved pack name isn't on disk
* - ExtendsChainTooDeepError if the parent chain exceeds depth 8
* - AliasCycleError if the manifest contains a cycle in the alias graph
* - SchemaPackManifestError if any manifest fails validation
*/
export async function loadActivePack(input: LoadActivePackInput): Promise<ResolvedPack> {
const resolutionInput = buildResolutionInput(input);
const resolution: ResolutionResult = resolveActivePackName(resolutionInput);
// v0.40.6.0: TTL-gated cache fast path. Inside STAT_TTL_MS (default 1s)
// returns immediately (~10ns). Outside the window: stats files in the
// extends chain; cascade-invalidates and falls through on mtime change.
const cached = tryCachedPack(resolution.pack_name);
if (cached) return cached;
const manifest = await loadPackManifestByName(resolution.pack_name);
// Thread the locator so resolvePack can snapshot file paths + mtimes
// for the stat-TTL gate on subsequent calls (codex C6 + D11 + D13).
return await resolvePack(manifest, loadPackManifestByName, {
loadByPath: (name) => _packLocator(name),
});
}
/**
* Return the resolved pack NAME and source tier WITHOUT loading the
* manifest from disk. Used by `gbrain schema active` to surface
* provenance ("active pack: garry — source: gbrain.yml") without
* paying the load cost.
*/
export function resolveActivePackNameOnly(input: LoadActivePackInput): ResolutionResult {
return resolveActivePackName(buildResolutionInput(input));
}
/**
* v0.42 (T4, plan D7): enumerate packs whose `migration_from` declares a
* version range matching (packName, packVersion). Used by the
* `checkPackUpgradeAvailable` onboard check to surface "your brain is on
* gbrain-base@1.x; gbrain-base-v2@1.0.0 is available." Results sorted by
* successor version descending so the highest-available successor wins.
*
* Walks BUNDLED_PACK_NAMES + any installed pack under `~/.gbrain/schema-packs/`
* discoverable via the locator. Each candidate is loaded + parsed; load
* failures are logged-and-skipped per the D4 EMPTY FILTER contract — a
* corrupt pack on disk doesn't break the upgrade-available check for
* everyone else.
*
* Version-range matching supports:
* - exact literal: `1.0.0` matches `1.0.0` only
* - major wildcard: `1.x` matches `1.0.0`, `1.5.2`, etc.
* - minor wildcard: `1.0.x` matches `1.0.0`, `1.0.5`, etc.
*
* Returns empty array when no successors found (caller interprets as
* "brain already on latest").
*/
export async function findPackSuccessors(
packName: string,
packVersion: string,
): Promise<ResolvedPack[]> {
const { BUNDLED_PACK_NAMES } = await import('./mutate.ts');
const candidates: string[] = [];
for (const name of BUNDLED_PACK_NAMES) {
if (name !== packName) candidates.push(name);
}
// Walk ~/.gbrain/schema-packs/* via the locator. We can't enumerate
// directly without filesystem scan; defer to v0.43+ for installed-pack
// enumeration. Bundled packs alone cover v0.42's gbrain-base→v2 path.
const successors: ResolvedPack[] = [];
for (const candidateName of candidates) {
try {
const candidate = await loadActivePack({
cfg: null,
remote: false,
perCall: candidateName,
});
const mf = candidate.manifest.migration_from;
if (!mf) continue;
if (mf.pack !== packName) continue;
if (!_versionRangeMatches(packVersion, mf.version)) continue;
successors.push(candidate);
} catch {
// Log-and-skip per D4 EMPTY FILTER contract; corrupt pack on disk
// shouldn't break the upgrade-available check.
continue;
}
}
// Sort by successor version desc (so the newest successor wins when
// multiple match — uncommon today but defensive for v0.43+).
successors.sort((a, b) => _versionDescCompare(b.manifest.version, a.manifest.version));
return successors;
}
/**
* @internal exported for unit test seam (test/schema-pack-find-successors.test.ts).
* Matches version against a wildcard range: `1.x` / `1.0.x` / `1.0.0` literal.
*/
export function _versionRangeMatches(version: string, range: string): boolean {
// Exact match (no wildcards)
if (!range.includes('x') && !range.includes('*')) {
return version === range;
}
// Convert range like `1.x` or `1.0.x` to a regex
const rangeParts = range.split('.');
const versionParts = version.split('.');
if (rangeParts.length > versionParts.length) return false;
for (let i = 0; i < rangeParts.length; i++) {
const r = rangeParts[i];
if (r === 'x' || r === '*') continue;
if (r !== versionParts[i]) return false;
}
return true;
}
/**
* @internal Compare two semver strings (M.m.p). Negative if a < b; positive
* if a > b; zero if equal. Treats `0.41.2.0` (4-part) by truncating to
* 3-part because Zod schema validates `M.m.p` only.
*/
export function _versionDescCompare(a: string, b: string): number {
const ap = a.split('.').slice(0, 3).map(n => parseInt(n, 10));
const bp = b.split('.').slice(0, 3).map(n => parseInt(n, 10));
for (let i = 0; i < 3; i++) {
if ((ap[i] ?? 0) !== (bp[i] ?? 0)) return (ap[i] ?? 0) - (bp[i] ?? 0);
}
return 0;
}
function buildResolutionInput(input: LoadActivePackInput): ResolutionInput {
const envVar = process.env.GBRAIN_SCHEMA_PACK?.trim() || undefined;
// tier-6: ~/.gbrain/config.json schema_pack field
const homeConfig = input.cfg?.schema_pack?.trim() || undefined;
return {
perCall: input.perCall,
remote: input.remote,
perSourceDb: input.perSourceDb,
sourceId: input.sourceId,
envVar,
dbConfig: input.dbConfig,
gbrainYml: input.gbrainYml,
homeConfig,
};
}