mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* Merge branch 'master' into garrytan/type-taxonomy-unification Resolve VERSION, package.json, CHANGELOG conflicts with v0.41.22.0 on top, preserving master's v0.41.19.0 entry below. * feat: v0.41.22.0 type-unification cathedral — collapse 94 types to 15 (closes #1479) Ships gbrain-base-v2 as the new install default (15 canonical types: 14 + note catch-all) and the unify-types PROTECTED Minion handler that runs the gbrain-base→v2 migration end-to-end on existing brains. What this delivers: - gbrain-base-v2.yaml standalone schema pack (no extends:) with 14 canonical page_types + 9 cluster mapping_rules + catch-all sentinel - 3 new schema-pack primitives: runRetypeCore (chunked UPDATE with legacy_type stamping), runPageToLinkCore (edge-shaped pages → link rows), runPageToAliasCore (concept-redirect → slug_aliases) - rewriteLinksBatch for N-pair atomic FK rewrite - Migration v104 slug_aliases table (forward-bootstrap probed on both engines for safe upgrade chain) - New engine method resolveSlugWithAlias(slug, sourceOrSources) on both Postgres + PGLite with multi-source ambiguity warning - inferTypeAndSubtypeFromPack overload + subtypes: + mapping_rules: + migration_from: schema-pack manifest extensions - findPackSuccessors version-range walker (1.x / 1.0.x / exact match) - expandTypeFilter for --type back-compat (D14): legacy aliases route through mapping_rules → canonical+subtype before the SQL filter fires - 3 new onboard checks: pack_upgrade_available, type_proliferation, dangling_aliases (source-scoped per F12) - unify-types Minion handler (PROTECTED, manual_only via render.ts allowlist per D17): retype-explicit → retype-catch-all → page-to-link → page-to-alias → final sync → active-pack flip - alias_resolved 1.05x post-fusion search boost stage; KNOBS_HASH_VERSION bumped 5→6 (one-time cache miss on upgrade, self-healing in TTL) - ELIGIBLE_TYPES for facts extraction extended with v2 canonicals (codex F-ELIGIBLE: blocker not v0.43 follow-up) Tests: 79 new unit/integration cases + 3 E2E cases covering all 9 production clusters end-to-end. 124-case verification on the cache-key + build-llms fixes. KNOBS_HASH_VERSION assertions updated in 3 tests. Plan: ~/.claude/plans/system-instruction-you-are-working-transient-elephant.md (16 locked decisions D1-D17, 12 baseline fixes F7-F21 absorbed from codex outside voice). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: CI verify failures — system-of-record allow-comment + schema-unify manifest registration Two CI failures on PR #1542: 1. check:system-of-record flagged page-to-link.ts:207 addLinksBatch as a direct write to a derived table. The call IS the reconcile surface for page_to_link mapping_rules — it converts edge-shaped pages into canonical link rows under the PROTECTED unify-types Minion handler, source-scoped, atomic per-rule. Added the canonical `// gbrain-allow-direct-insert: <reason>` comment on the same line. 2. check:resolver emitted 11 orphan_trigger warnings for `schema-unify` because the skill was added to skills/RESOLVER.md without a corresponding entry in skills/manifest.json. Added the registration under the existing skills[] array. bun run verify: 28/28 checks pass locally. * fix: CI test failures — schema-unify conformance + eligibility regression Six test failures across shards 2 + 10 on PR #1542: 1. resolver.test.ts: round-trip parser requires frontmatter triggers to be quoted (`- "..."` or `- '...'`). schema-unify shipped with bare YAML strings; quoted the 10 triggers to round-trip correctly. 2. skills-conformance.test.ts (×3): schema-unify SKILL.md was missing the required Contract, Anti-Patterns, and Output Format sections that every conformant skill must declare. Added all three: - Contract: inputs / outputs / side effects / failure modes - Anti-Patterns: 5 DON'Ts including the autopilot trust boundary - Output Format: per-phase stderr lines + celebration summary + JSON envelope shape 3. facts-eligibility.test.ts (×2): the v0.41.22 ELIGIBLE_TYPES expansion added `concept` to the eligible list, but the existing test suite pins concept as rejected (it's `extractable: true` in the schema pack but the v0.41.11 contract documented this as "cosmetic on the backstop path because backstop uses hardcoded ELIGIBLE_TYPES"). Removed `concept` from the expansion; other v2 canonicals (media, tweet, atom, analysis) stay. Comment updated to document the deliberate omission. All 6 failing tests now pass locally (370/370 across the 3 affected files). bun run verify: 28/28 checks green. * fix: harden findPackSuccessors test against shard pollution CI shard 8 reported 1 fail (1.00ms — too fast for any real loadActivePack file I/O) on `finds gbrain-base-v2 as successor of gbrain-base@1.0.0`. Local triple-run passes 9/9 in isolation. Root cause: the existing afterEach reset clears the module-level pack cache AFTER each test, but the FIRST test in the file inherits whatever state sibling files in the same bun shard process left behind. With 24+ schema-pack tests in shard 8 (mutate, mutate-audit, best-effort, registry-reload, manifest-v041_2, etc.) running before this file, the first test can read a poisoned cache. Fix: add `beforeEach(_resetPackCacheForTests)`. Two-sided reset guarantees clean state regardless of file ordering within the shard. bun run verify: 28/28 checks pass. * fix: quarantine two flaky tests to serial runner CI shard 1 + shard 8 each surfaced one intermittent failure: shard 1: buildBrainTools > execute() on put_page with valid namespace shard 8: findPackSuccessors > finds gbrain-base-v2 as successor Both pass cleanly in isolation. Both are concurrency races against shared in-shard state: - brain-allowlist.test.ts shares a singleton PGLiteEngine across 18 tests with a beforeEach DELETE FROM pages. With max-concurrency=4, two put_page tests can interleave their TRUNCATE + write phases, so the auto-link/extract sub-steps inside put_page race against the sibling test's DELETE. - schema-pack-find-pack-successors.test.ts reads bundled YAML packs via loadActivePack. The module-level pack cache is shared across parallel tests in the same shard; the previous beforeEach reset helped but didn't fully isolate against concurrent file reads under CI load. Fix per CLAUDE.md test-isolation lint rule R2 (concurrency-fragile files belong in the .serial.test.ts quarantine): rename both files to *.serial.test.ts. Serial runner picks them up at max-concurrency=1. 49/49 serial files pass locally. 28/28 verify checks pass. * fix: quarantine embed-stale test to serial runner CI shard 9 reported 6 failures, all from the embedStaleForSource describe block, all ~120-150ms each — classic shared-engine concurrency race shape. Passes 7/7 locally in isolation. Root cause: embed-stale.test.ts shares a singleton PGLiteEngine across 7 tests with beforeEach resetPgliteState. Under bun's max-concurrency=4 in the parallel shard, two tests can interleave their TRUNCATE + seedPage + upsertChunks + embedStaleForSource flow, so one test's stale-chunk count sees another test's mid-flight writes. Same fix as brain-allowlist.serial.test.ts and schema-pack-find-pack-successors.serial.test.ts: rename to *.serial.test.ts so the serial runner picks it up at max-concurrency=1. bun run verify: 28/28 checks pass. 7/7 embed-stale tests pass via serial. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
661 lines
23 KiB
TypeScript
661 lines
23 KiB
TypeScript
import matter from 'gray-matter';
|
|
import { safeLoad as yamlSafeLoad } from 'js-yaml';
|
|
import type { Page, PageType } from './types.ts';
|
|
import { slugifyPath } from './sync.ts';
|
|
|
|
export type ParseValidationCode =
|
|
| 'MISSING_OPEN'
|
|
| 'MISSING_CLOSE'
|
|
| 'YAML_PARSE'
|
|
| 'SLUG_MISMATCH'
|
|
| 'NULL_BYTES'
|
|
| 'NESTED_QUOTES'
|
|
| 'EMPTY_FRONTMATTER';
|
|
|
|
export interface ParseValidationError {
|
|
code: ParseValidationCode;
|
|
message: string;
|
|
line?: number;
|
|
}
|
|
|
|
export interface ParseOpts {
|
|
/** When true, errors[] is populated. Existing callers unaffected. */
|
|
validate?: boolean;
|
|
/** When validate is true and frontmatter has a `slug:` field that doesn't
|
|
* match expectedSlug, emits SLUG_MISMATCH. */
|
|
expectedSlug?: string;
|
|
/**
|
|
* v0.39 T1.5 — active schema pack to drive type inference. When set,
|
|
* `inferType` uses the pack's `page_types[].path_prefixes` instead of
|
|
* the hardcoded gbrain-base table. When unset, falls back to the
|
|
* pre-v0.39 hardcoded behavior (preserves byte-for-byte parity gate
|
|
* `test/regressions/gbrain-base-equivalence.test.ts`).
|
|
*
|
|
* Callers thread this from `loadActivePack(ctx)` once per command —
|
|
* NEVER per file inside sync, per codex perf finding #7.
|
|
*/
|
|
activePack?: { page_types: ReadonlyArray<{ name: string; path_prefixes: ReadonlyArray<string> }> };
|
|
}
|
|
|
|
export interface ParsedMarkdown {
|
|
frontmatter: Record<string, unknown>;
|
|
compiled_truth: string;
|
|
timeline: string;
|
|
slug: string;
|
|
type: PageType;
|
|
title: string;
|
|
tags: string[];
|
|
/** Present iff opts.validate. Empty array means no errors. */
|
|
errors?: ParseValidationError[];
|
|
}
|
|
|
|
/**
|
|
* Parse a markdown file with YAML frontmatter into its components.
|
|
*
|
|
* Structure:
|
|
* ---
|
|
* type: concept
|
|
* title: Do Things That Don't Scale
|
|
* tags: [startups, growth]
|
|
* ---
|
|
* Compiled truth content here...
|
|
*
|
|
* <!-- timeline -->
|
|
* Timeline content here...
|
|
*
|
|
* The first --- pair is YAML frontmatter (handled by gray-matter).
|
|
* After frontmatter, the body is split at the first recognized timeline
|
|
* sentinel: `<!-- timeline -->` (preferred), `--- timeline ---` (decorated),
|
|
* or a plain `---` immediately preceding a `## Timeline` / `## History`
|
|
* heading (backward-compat for existing files). A bare `---` in body text
|
|
* is treated as a markdown horizontal rule, not a timeline separator.
|
|
*/
|
|
export function parseMarkdown(
|
|
content: string,
|
|
filePath?: string,
|
|
opts?: ParseOpts,
|
|
): ParsedMarkdown {
|
|
const errors: ParseValidationError[] = [];
|
|
|
|
// gray-matter is forgiving: it returns empty data + original content for
|
|
// pretty much any input. The validation surface below catches the cases
|
|
// it silently swallows. Validation only runs when opts.validate is true,
|
|
// so existing callers are unaffected.
|
|
let parsed: ReturnType<typeof matter> | null = null;
|
|
let yamlParseError: Error | null = null;
|
|
try {
|
|
parsed = matter(content);
|
|
} catch (e) {
|
|
yamlParseError = e as Error;
|
|
}
|
|
|
|
if (opts?.validate) {
|
|
collectValidationErrors(content, errors, {
|
|
yamlParseError,
|
|
expectedSlug: opts.expectedSlug,
|
|
parsedFrontmatter: parsed?.data ?? {},
|
|
});
|
|
}
|
|
|
|
// When YAML parsing failed (rare; gray-matter is forgiving), fall back to
|
|
// empty frontmatter + raw content as the body so non-validate callers still
|
|
// get a usable shape.
|
|
const frontmatter = (parsed?.data ?? {}) as Record<string, unknown>;
|
|
const body = parsed?.content ?? content;
|
|
|
|
const { compiled_truth, timeline } = splitBody(body);
|
|
|
|
const type = (frontmatter.type as string) || (
|
|
opts?.activePack ? inferTypeFromPack(filePath, opts.activePack) : inferType(filePath)
|
|
);
|
|
const title = (frontmatter.title as string) || inferTitle(filePath);
|
|
const tags = extractTags(frontmatter);
|
|
const slug = (frontmatter.slug as string) || inferSlug(filePath);
|
|
|
|
const cleanFrontmatter = { ...frontmatter };
|
|
delete cleanFrontmatter.type;
|
|
delete cleanFrontmatter.title;
|
|
delete cleanFrontmatter.tags;
|
|
delete cleanFrontmatter.slug;
|
|
|
|
const result: ParsedMarkdown = {
|
|
frontmatter: cleanFrontmatter,
|
|
compiled_truth: compiled_truth.trim(),
|
|
timeline: timeline.trim(),
|
|
slug,
|
|
type,
|
|
title,
|
|
tags,
|
|
};
|
|
if (opts?.validate) result.errors = errors;
|
|
return result;
|
|
}
|
|
|
|
/**
|
|
* Inspect raw content for the 7 frontmatter validation classes that gray-matter
|
|
* silently accepts. Mutates `errors` in place. The order of checks is
|
|
* deliberate: cheap byte-level checks first, then structural checks, then
|
|
* YAML-parse-dependent checks.
|
|
*/
|
|
function collectValidationErrors(
|
|
content: string,
|
|
errors: ParseValidationError[],
|
|
ctx: {
|
|
yamlParseError: Error | null;
|
|
expectedSlug?: string;
|
|
parsedFrontmatter: Record<string, unknown>;
|
|
},
|
|
): void {
|
|
// 1. NULL_BYTES — binary corruption indicator.
|
|
const nullIdx = content.indexOf('\x00');
|
|
if (nullIdx >= 0) {
|
|
const line = content.slice(0, nullIdx).split('\n').length;
|
|
errors.push({
|
|
code: 'NULL_BYTES',
|
|
message: 'Content contains null bytes (likely binary corruption)',
|
|
line,
|
|
});
|
|
}
|
|
|
|
// 2. MISSING_OPEN — first non-empty line must be `---`.
|
|
const lines = content.split('\n');
|
|
let firstNonEmpty = -1;
|
|
for (let i = 0; i < lines.length; i++) {
|
|
if (lines[i].trim().length > 0) {
|
|
firstNonEmpty = i;
|
|
break;
|
|
}
|
|
}
|
|
if (firstNonEmpty === -1) {
|
|
// Empty file: treat as MISSING_OPEN. Don't run other structural checks.
|
|
errors.push({
|
|
code: 'MISSING_OPEN',
|
|
message: 'File is empty or whitespace-only; expected frontmatter starting with ---',
|
|
line: 1,
|
|
});
|
|
return;
|
|
}
|
|
if (lines[firstNonEmpty].trim() !== '---') {
|
|
errors.push({
|
|
code: 'MISSING_OPEN',
|
|
message: 'Frontmatter must start with --- on the first non-empty line',
|
|
line: firstNonEmpty + 1,
|
|
});
|
|
// Without an opener we can't reason about MISSING_CLOSE / EMPTY_FRONTMATTER
|
|
// / NESTED_QUOTES inside frontmatter. Stop structural checks here.
|
|
return;
|
|
}
|
|
|
|
// 3. MISSING_CLOSE — find the next `---` after the opener. If a markdown
|
|
// heading appears before it, that's a strong signal the closing
|
|
// delimiter is missing (the heading was meant to be in the body).
|
|
let closeLine = -1;
|
|
let headingBeforeClose = -1;
|
|
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
|
|
const t = lines[i].trim();
|
|
if (t === '---') {
|
|
closeLine = i;
|
|
break;
|
|
}
|
|
if (/^#{1,6}\s/.test(t) && headingBeforeClose === -1) {
|
|
headingBeforeClose = i;
|
|
}
|
|
}
|
|
if (closeLine === -1) {
|
|
errors.push({
|
|
code: 'MISSING_CLOSE',
|
|
message:
|
|
headingBeforeClose >= 0
|
|
? `No closing --- before heading at line ${headingBeforeClose + 1}`
|
|
: 'No closing --- delimiter found',
|
|
line: headingBeforeClose >= 0 ? headingBeforeClose + 1 : firstNonEmpty + 1,
|
|
});
|
|
return;
|
|
}
|
|
if (headingBeforeClose >= 0 && headingBeforeClose < closeLine) {
|
|
errors.push({
|
|
code: 'MISSING_CLOSE',
|
|
message: `Heading at line ${headingBeforeClose + 1} found inside frontmatter zone (closing --- comes after)`,
|
|
line: headingBeforeClose + 1,
|
|
});
|
|
}
|
|
|
|
// 4. EMPTY_FRONTMATTER — open and close present but nothing meaningful between.
|
|
const fmBody = lines.slice(firstNonEmpty + 1, closeLine).join('\n').trim();
|
|
if (fmBody.length === 0) {
|
|
errors.push({
|
|
code: 'EMPTY_FRONTMATTER',
|
|
message: 'Frontmatter block is empty',
|
|
line: firstNonEmpty + 1,
|
|
});
|
|
}
|
|
|
|
// 5. NESTED_QUOTES — common breakage pattern: `title: "Name "Nick" Last"`.
|
|
// The heuristic: a frontmatter `key: value` line with 3+ unescaped
|
|
// double-quote characters is suspicious. But raw quote-counting is
|
|
// too dumb: a YAML flow sequence like `tags: ["yc", "w2025"]` has
|
|
// 4 unescaped `"` by design (valid), and a single-quoted scalar
|
|
// like `title: 'a: "b" "c"'` has literal inner `"` (also valid).
|
|
// Disambiguate by running js-yaml on just the value; only flag
|
|
// lines that genuinely fail to parse. The full-frontmatter YAML
|
|
// parse error is caught separately by check 6 (YAML_PARSE) below.
|
|
for (let i = firstNonEmpty + 1; i < closeLine; i++) {
|
|
const line = lines[i];
|
|
const m = line.match(/^\s*[A-Za-z_][\w-]*\s*:\s*(.*)$/);
|
|
if (!m) continue;
|
|
const value = m[1];
|
|
let count = 0;
|
|
for (let j = 0; j < value.length; j++) {
|
|
if (value[j] === '"' && (j === 0 || value[j - 1] !== '\\')) count++;
|
|
}
|
|
if (count < 3) continue;
|
|
|
|
// 3+ unescaped quotes — could be valid YAML (flow seq, single-quoted
|
|
// scalar with inner quotes, bare scalar with embedded quotes) or
|
|
// genuinely broken. Parse the value to disambiguate.
|
|
let isValidYaml = false;
|
|
try {
|
|
yamlSafeLoad(value);
|
|
isValidYaml = true;
|
|
} catch {
|
|
// YAML parse failed — line is genuinely broken
|
|
}
|
|
|
|
if (!isValidYaml) {
|
|
errors.push({
|
|
code: 'NESTED_QUOTES',
|
|
message: 'Nested double quotes in YAML value (use single quotes for the outer)',
|
|
line: i + 1,
|
|
});
|
|
}
|
|
}
|
|
|
|
// 6. YAML_PARSE — gray-matter threw.
|
|
if (ctx.yamlParseError) {
|
|
errors.push({
|
|
code: 'YAML_PARSE',
|
|
message: `YAML parse failed: ${ctx.yamlParseError.message}`,
|
|
line: firstNonEmpty + 1,
|
|
});
|
|
}
|
|
|
|
// 7. SLUG_MISMATCH — only when expectedSlug was provided and a slug field exists.
|
|
if (ctx.expectedSlug && typeof ctx.parsedFrontmatter.slug === 'string') {
|
|
const declared = ctx.parsedFrontmatter.slug as string;
|
|
if (declared !== ctx.expectedSlug) {
|
|
errors.push({
|
|
code: 'SLUG_MISMATCH',
|
|
message: `Frontmatter slug "${declared}" does not match path-derived slug "${ctx.expectedSlug}"`,
|
|
});
|
|
}
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Split body content at the first recognized timeline sentinel.
|
|
* Returns compiled_truth (before) and timeline (after).
|
|
*
|
|
* Recognized sentinels (in order of precedence):
|
|
* 1. `<!-- timeline -->` — preferred, unambiguous, what serializeMarkdown emits
|
|
* 2. `--- timeline ---` — decorated separator
|
|
* 3. `---` ONLY when the next non-empty line is `## Timeline` or `## History`
|
|
* (backward-compat fallback for older gbrain-written files)
|
|
*
|
|
* A plain `---` line is a markdown horizontal rule, NOT a timeline separator.
|
|
* Treating bare `---` as a separator caused 83% content truncation on wiki corpora.
|
|
*/
|
|
export function splitBody(body: string): { compiled_truth: string; timeline: string } {
|
|
const lines = body.split('\n');
|
|
const splitIndex = findTimelineSplitIndex(lines);
|
|
|
|
if (splitIndex === -1) {
|
|
return { compiled_truth: body, timeline: '' };
|
|
}
|
|
|
|
const compiled_truth = lines.slice(0, splitIndex).join('\n');
|
|
const timeline = lines.slice(splitIndex + 1).join('\n');
|
|
return { compiled_truth, timeline };
|
|
}
|
|
|
|
function findTimelineSplitIndex(lines: string[]): number {
|
|
for (let i = 0; i < lines.length; i++) {
|
|
const trimmed = lines[i].trim();
|
|
|
|
if (trimmed === '<!-- timeline -->' || trimmed === '<!--timeline-->') {
|
|
return i;
|
|
}
|
|
|
|
if (trimmed === '--- timeline ---' || /^---\s+timeline\s+---$/i.test(trimmed)) {
|
|
return i;
|
|
}
|
|
|
|
if (trimmed === '---') {
|
|
const beforeContent = lines.slice(0, i).join('\n').trim();
|
|
if (beforeContent.length === 0) continue;
|
|
|
|
for (let j = i + 1; j < lines.length; j++) {
|
|
const next = lines[j].trim();
|
|
if (next.length === 0) continue;
|
|
if (/^##\s+(timeline|history)\b/i.test(next)) return i;
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
return -1;
|
|
}
|
|
|
|
/**
|
|
* Serialize a page back to markdown format.
|
|
* Produces: frontmatter + compiled_truth + --- + timeline
|
|
*/
|
|
export function serializeMarkdown(
|
|
frontmatter: Record<string, unknown>,
|
|
compiled_truth: string,
|
|
timeline: string,
|
|
meta: { type: PageType; title: string; tags: string[] },
|
|
): string {
|
|
// Build full frontmatter including type, title, tags
|
|
const fullFrontmatter: Record<string, unknown> = {
|
|
type: meta.type,
|
|
title: meta.title,
|
|
...frontmatter,
|
|
};
|
|
if (meta.tags.length > 0) {
|
|
fullFrontmatter.tags = meta.tags;
|
|
}
|
|
|
|
const yamlContent = matter.stringify('', fullFrontmatter).trim();
|
|
|
|
let body = compiled_truth;
|
|
if (timeline) {
|
|
body += '\n\n<!-- timeline -->\n\n' + timeline;
|
|
}
|
|
|
|
return yamlContent + '\n\n' + body + '\n';
|
|
}
|
|
|
|
// v0.38 T7a (Phase B): inferType is now pack-aware.
|
|
//
|
|
// The original hardcoded behavior is preserved IDENTICALLY when the
|
|
// pack is gbrain-base (or when no pack is provided — back-compat
|
|
// callers). Schema packs can extend the path → type mapping by
|
|
// declaring `page_types[].path_prefixes` in their manifest; users
|
|
// who add `paper: { path_prefixes: [papers/, literature/] }` get
|
|
// papers/foo.md → type 'paper' without forking the engine.
|
|
//
|
|
// inferType (legacy sync wrapper) → calls inferTypeWithPrefixes with
|
|
// the GBRAIN_BASE_PATH_PREFIXES table below, which reproduces the
|
|
// pre-v0.38 hardcoded behavior byte-for-byte (parity-pinned by
|
|
// test/regressions/gbrain-base-equivalence.test.ts).
|
|
// inferTypeFromPack(filePath, manifest) → new primitive that walks
|
|
// the active pack's page_types[].path_prefixes. Priority: pack
|
|
// declarations are scanned in order they appear in the manifest;
|
|
// first match wins. Empty/unset prefixes do NOT match.
|
|
//
|
|
// Callers in async paths (import-file.ts, sync.ts, cycle phases) can
|
|
// adopt inferTypeFromPack(path, pack) to honor user-defined types.
|
|
// The bare inferType(path) call site remains for sync legacy paths
|
|
// and matches gbrain-base by construction.
|
|
|
|
/**
|
|
* Hardcoded path-prefix table mirroring pre-v0.38 inferType behavior.
|
|
* MUST stay in lockstep with the gbrain-base.yaml `path_prefixes` field
|
|
* (parity-pinned by test/regressions/gbrain-base-equivalence.test.ts).
|
|
* Order matters: wiki subtypes + writing scan first (stronger signal
|
|
* than ancestor directories).
|
|
*/
|
|
const GBRAIN_BASE_PATH_PREFIXES: ReadonlyArray<{ prefixes: string[]; type: PageType }> = [
|
|
{ prefixes: ['/writing/'], type: 'writing' },
|
|
{ prefixes: ['/wiki/analysis/'], type: 'analysis' },
|
|
{ prefixes: ['/wiki/guides/', '/wiki/guide/'], type: 'guide' },
|
|
{ prefixes: ['/wiki/hardware/'], type: 'hardware' },
|
|
{ prefixes: ['/wiki/architecture/'], type: 'architecture' },
|
|
{ prefixes: ['/wiki/concepts/', '/wiki/concept/'], type: 'concept' },
|
|
{ prefixes: ['/people/', '/person/'], type: 'person' },
|
|
{ prefixes: ['/companies/', '/company/'], type: 'company' },
|
|
{ prefixes: ['/deals/', '/deal/'], type: 'deal' },
|
|
{ prefixes: ['/yc/'], type: 'yc' },
|
|
{ prefixes: ['/civic/'], type: 'civic' },
|
|
{ prefixes: ['/projects/', '/project/'], type: 'project' },
|
|
{ prefixes: ['/sources/', '/source/'], type: 'source' },
|
|
{ prefixes: ['/media/'], type: 'media' },
|
|
{ prefixes: ['/emails/', '/email/'], type: 'email' },
|
|
{ prefixes: ['/slack/'], type: 'slack' },
|
|
{ prefixes: ['/cal/', '/calendar/'], type: 'calendar-event' },
|
|
{ prefixes: ['/notes/', '/note/'], type: 'note' },
|
|
{ prefixes: ['/meetings/', '/meeting/'], type: 'meeting' },
|
|
];
|
|
|
|
function inferType(filePath?: string): PageType {
|
|
return inferTypeWithPrefixes(filePath, GBRAIN_BASE_PATH_PREFIXES);
|
|
}
|
|
|
|
/**
|
|
* Pack-aware variant. Callers with access to the active pack pass it
|
|
* here to honor user-declared types. Empty `page_types` array (no
|
|
* `extends: null` pack) falls back to gbrain-base defaults.
|
|
*
|
|
* Algorithm: each pack page_type contributes its `path_prefixes` array
|
|
* in declaration order. First prefix that matches wins. Default
|
|
* 'concept' applies when nothing matches.
|
|
*
|
|
* Note on prefix shape: gbrain-base stores prefixes WITHOUT the
|
|
* leading `/` (e.g. `people/`). For matching, we lower-case the path
|
|
* with a leading `/` prepended (matches the original behavior) and
|
|
* test against `'/' + prefix` so `people/` matches `/people/` inside
|
|
* the full path.
|
|
*/
|
|
export function inferTypeFromPack(
|
|
filePath: string | undefined,
|
|
pack: { page_types: ReadonlyArray<{ name: string; path_prefixes: ReadonlyArray<string> }> },
|
|
): PageType {
|
|
if (!filePath) return 'concept';
|
|
// Empty pack → fall back to gbrain-base hardcoded defaults.
|
|
if (pack.page_types.length === 0) {
|
|
return inferTypeWithPrefixes(filePath, GBRAIN_BASE_PATH_PREFIXES);
|
|
}
|
|
const lower = ('/' + filePath).toLowerCase();
|
|
for (const pt of pack.page_types) {
|
|
for (const prefix of pt.path_prefixes) {
|
|
const needle = prefix.startsWith('/') ? prefix.toLowerCase() : '/' + prefix.toLowerCase();
|
|
if (lower.includes(needle)) {
|
|
return pt.name;
|
|
}
|
|
}
|
|
}
|
|
return 'concept';
|
|
}
|
|
|
|
/**
|
|
* v0.42 (T5, plan D5): pack-aware type+subtype inference. Same path-prefix
|
|
* resolution as `inferTypeFromPack` PLUS subtype detection from
|
|
* `pack.page_types[i].subtypes[]` (declared per type). Walks subtype rules
|
|
* AFTER the prefix match wins. ReDoS-guarded compile of `path_pattern`
|
|
* happens here (test/regex per call; acceptable on the ingest path).
|
|
*
|
|
* Subtype-rule resolution order:
|
|
* 1. frontmatter_field+frontmatter_value (exact match)
|
|
* 2. path_pattern (regex test against the lower-cased full path)
|
|
*
|
|
* Frontmatter rule wins when both match. Returns the FIRST matching
|
|
* subtype name; subtype declarations earlier in the pack's subtypes
|
|
* array take precedence.
|
|
*
|
|
* Back-compat: legacy `inferTypeFromPack(filePath, pack)` preserved
|
|
* unchanged for the ~17 call sites that don't yet need subtype info.
|
|
*/
|
|
export function inferTypeAndSubtypeFromPack(
|
|
filePath: string | undefined,
|
|
pack: { page_types: ReadonlyArray<{
|
|
name: string;
|
|
path_prefixes: ReadonlyArray<string>;
|
|
subtypes?: ReadonlyArray<{
|
|
name: string;
|
|
when: { path_pattern?: string; frontmatter_field?: string; frontmatter_value?: unknown };
|
|
}>;
|
|
}> },
|
|
frontmatter?: Record<string, unknown>,
|
|
): { type: PageType; subtype?: string } {
|
|
if (!filePath) return { type: 'concept' };
|
|
// Empty pack → legacy fallback; no subtype info available.
|
|
if (pack.page_types.length === 0) {
|
|
return { type: inferTypeWithPrefixes(filePath, GBRAIN_BASE_PATH_PREFIXES) };
|
|
}
|
|
const lower = ('/' + filePath).toLowerCase();
|
|
// Stage 1: prefix-match wins (same as inferTypeFromPack)
|
|
let matchedType: { name: string; subtypes?: ReadonlyArray<{ name: string; when: { path_pattern?: string; frontmatter_field?: string; frontmatter_value?: unknown } }> } | undefined;
|
|
outer: for (const pt of pack.page_types) {
|
|
for (const prefix of pt.path_prefixes) {
|
|
const needle = prefix.startsWith('/') ? prefix.toLowerCase() : '/' + prefix.toLowerCase();
|
|
if (lower.includes(needle)) {
|
|
matchedType = pt;
|
|
break outer;
|
|
}
|
|
}
|
|
}
|
|
if (!matchedType) return { type: 'concept' };
|
|
const typeName = matchedType.name as PageType;
|
|
// Stage 2: subtype rule resolution (if any declared)
|
|
const subtypes = matchedType.subtypes ?? [];
|
|
if (subtypes.length === 0) return { type: typeName };
|
|
for (const st of subtypes) {
|
|
// Frontmatter rule first
|
|
if (st.when.frontmatter_field !== undefined && frontmatter !== undefined) {
|
|
const value = frontmatter[st.when.frontmatter_field];
|
|
if (st.when.frontmatter_value !== undefined && value === st.when.frontmatter_value) {
|
|
return { type: typeName, subtype: st.name };
|
|
}
|
|
}
|
|
// Path pattern rule
|
|
if (st.when.path_pattern !== undefined) {
|
|
try {
|
|
const re = new RegExp(st.when.path_pattern);
|
|
if (re.test(filePath) || re.test(lower)) {
|
|
return { type: typeName, subtype: st.name };
|
|
}
|
|
} catch {
|
|
// Malformed regex — skip silently; pack-load validation should
|
|
// have caught this at parse time via redos-guard.
|
|
continue;
|
|
}
|
|
}
|
|
}
|
|
return { type: typeName };
|
|
}
|
|
|
|
function inferTypeWithPrefixes(
|
|
filePath: string | undefined,
|
|
table: ReadonlyArray<{ prefixes: ReadonlyArray<string>; type: PageType }>,
|
|
): PageType {
|
|
if (!filePath) return 'concept';
|
|
const lower = ('/' + filePath).toLowerCase();
|
|
for (const row of table) {
|
|
for (const p of row.prefixes) {
|
|
if (lower.includes(p)) return row.type;
|
|
}
|
|
}
|
|
return 'concept';
|
|
}
|
|
|
|
function inferTitle(filePath?: string): string {
|
|
if (!filePath) return 'Untitled';
|
|
|
|
// Extract filename without extension, convert dashes/underscores to spaces
|
|
const parts = filePath.split('/');
|
|
const filename = parts[parts.length - 1]?.replace(/\.md$/i, '') || 'Untitled';
|
|
return filename.replace(/[-_]/g, ' ').replace(/\b\w/g, c => c.toUpperCase());
|
|
}
|
|
|
|
function inferSlug(filePath?: string): string {
|
|
if (!filePath) return 'untitled';
|
|
return slugifyPath(filePath);
|
|
}
|
|
|
|
function extractTags(frontmatter: Record<string, unknown>): string[] {
|
|
const tags = frontmatter.tags;
|
|
if (!tags) return [];
|
|
if (Array.isArray(tags)) return tags.map(String);
|
|
if (typeof tags === 'string') return tags.split(',').map(t => t.trim()).filter(Boolean);
|
|
return [];
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Page -> markdown serialization helpers (v0.38 DRY extract per eng review)
|
|
//
|
|
// Pre-v0.38 the dream cycle's reverse-render at src/core/cycle/synthesize.ts
|
|
// and the planned v0.38 put_page write-through path were going to have
|
|
// near-identical 15-line bodies that differed only in their frontmatter
|
|
// stamps. This extract is the single source of truth.
|
|
// ---------------------------------------------------------------------------
|
|
|
|
import { join } from 'node:path';
|
|
|
|
/** Options for serializePageToMarkdown. */
|
|
export interface SerializePageOpts {
|
|
/** Frontmatter fields merged on top of page.frontmatter at render time.
|
|
* Use this to stamp provenance (`ingested_via: 'webhook'`), identity
|
|
* markers (`dream_generated: true`), or any caller-specific extra
|
|
* fields. Original page.frontmatter keys win unless explicitly
|
|
* overridden. */
|
|
frontmatterOverrides?: Record<string, unknown>;
|
|
}
|
|
|
|
/**
|
|
* Render a Page row to its canonical on-disk markdown form. Sibling to
|
|
* `serializeMarkdown` (which takes the underlying primitives); this version
|
|
* pulls everything from a `Page` object so callers don't have to destructure
|
|
* compiled_truth / timeline / tags / frontmatter at every site.
|
|
*
|
|
* - Frontmatter: starts from `page.frontmatter`, merged with optional
|
|
* `opts.frontmatterOverrides`. Useful for stamping `dream_generated`,
|
|
* `ingested_via`, etc.
|
|
* - Type / title: pulled from the Page columns; falls back to 'note' /
|
|
* empty string when absent.
|
|
* - Tags: passed separately so callers don't need to query engine.getTags
|
|
* if they already have them in hand.
|
|
*/
|
|
export function serializePageToMarkdown(
|
|
page: Page,
|
|
tags: string[],
|
|
opts: SerializePageOpts = {},
|
|
): string {
|
|
const frontmatter: Record<string, unknown> = {
|
|
...((page.frontmatter ?? {}) as Record<string, unknown>),
|
|
...(opts.frontmatterOverrides ?? {}),
|
|
};
|
|
return serializeMarkdown(
|
|
frontmatter,
|
|
page.compiled_truth ?? '',
|
|
page.timeline ?? '',
|
|
{
|
|
type: (page.type as PageType) ?? 'note',
|
|
title: page.title ?? '',
|
|
tags,
|
|
},
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Compute the on-disk path for a (brainDir, slug, source_id) tuple per
|
|
* the v0.32.8 multi-source filing layout:
|
|
* - Default source: `<brainDir>/<slug>.md`
|
|
* - Non-default source: `<brainDir>/.sources/<source_id>/<slug>.md`
|
|
*
|
|
* Shared by the dream-cycle reverse-render (`reverseWriteRefs` in
|
|
* synthesize.ts) and the v0.38 put_page write-through path so both
|
|
* sites compute the same path for the same row.
|
|
*
|
|
* NOTE: caller is responsible for validating `source_id` against path-
|
|
* traversal attacks via `validateSourceId` (src/core/utils.ts) BEFORE
|
|
* passing it here. This helper does the filename math only.
|
|
*/
|
|
export function resolvePageFilePath(
|
|
brainDir: string,
|
|
slug: string,
|
|
sourceId: string,
|
|
): string {
|
|
return sourceId === 'default'
|
|
? join(brainDir, `${slug}.md`)
|
|
: join(brainDir, '.sources', sourceId, `${slug}.md`);
|
|
}
|