mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* fix: dream --source/--source-id plumbs sourceId to runCycle (supersedes #1559) Closes the silent-no-op class where `gbrain dream --source <id>` ran the cycle but never wrote `last_full_cycle_at`, leaving `gbrain doctor`'s cycle_freshness check stuck red forever. Changes to src/commands/dream.ts: - DreamArgs.source field; parseArgs recognizes --source <id> AND the --source-id alias (matches v0.37.7.0 #1167 naming across import/extract/graph-query) - Argv validation: missing value → exit 2; repeated different values → exit 2; --source X --source-id Y conflict → exit 2; same-value repetition → accepted - --help short-circuit ordering preserved with IRON-RULE comment + structural test guard - runDream engine-null guard: --source requires a connected brain - runDream resolveSourceId → archived-source guard via fetchSource from src/core/sources-load.ts (single-row SELECT that projects archived + handles pre-v0.26.5 schema via isUndefinedColumnError) - Typed-error try/catch via isResolverUserError predicate: only swallows known resolver-user errors; TypeError / postgres errors propagate uncaught with stack trace so genuine programmer bugs aren't hidden behind operator-error UX - Forwarded sourceId to runCycle; existing v0.38 writeback at cycle.ts:1947-1967 now actually fires - --help text documents both flag names Tests: - test/dream-cli-flags.test.ts: structural assertions for new flags, help text, IRON-RULE comment guard, resolver/predicate wiring - test/dream.test.ts: 13 PGLite integration cases covering happy path (the regression that closes PR #1559), back-compat, alias equivalence, all argv edge cases, engine-null, archived, --help short-circuit ordering, T3 typed-error propagation, and D5 end-to-end dream→checkCycleFreshness column-name drift guard Plan + 11 decisions: ~/.claude/plans/system-instruction-you-are-working-starry-papert.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: judgeSignificance uses canonical safeSplitIndex (closes #1559/#1561 emoji crash) Closes the 2026-05-24 production SYNTH_PHASE_FAIL: 🤖 (U+1F916, surrogate pair U+D83E U+DD16) at offset 3999 in a long telegram transcript made the raw 4000-char slice produce a lone high surrogate; Anthropic's JSON parser rejected the payload with "no low surrogate in string"; the synthesize phase failed. Changes to src/core/cycle/synthesize.ts: - judgeSignificance head+tail slice routed through safeSplitIndex from src/core/text-safe.ts (already imported) - Did NOT introduce safeSliceEnd from PRs #1559+#1561 — that helper re-introduces the case-3 bug src/core/text-safe.ts:18-21 documents - Did NOT touch findBoundary — master already routes through safeSplitIndex per the v0.42.0.0 wave Tests in test/cycle-synthesize.test.ts: - New describe('judgeSignificance — UTF-16 safety') block - test.each over head boundaries (offsets 3998-4001) AND tail boundaries (offsets 3999-4002) for an 8001-char content with the robot emoji placed at each - Primary assertion: explicit unpaired-surrogate scan over the captured prompt (NOT JSON.stringify per codex C-11 — V8/JSCore do not throw on lone surrogates, so that assertion was weak) - Sub-8000 short-content branch case: no slicing, emoji passes through unchanged Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: expand error_page_title + add cloudflare_challenge_title (supersedes #1561) Closes the bug class where scraper error pages with titles like "Forbidden", "Access Denied", "Service Unavailable", "Robot Check", and "Just a moment..." were slipping through the ingest gate because the matcher only caught bare numeric codes (403/404/500...) and "page not found". 232+ pages observed (202+ from straylight- brain) were inflating page counts and tripping content_sanity_audit_recent on every doctor run. Changes to src/core/content-sanity.ts BUILT_IN_JUNK_PATTERNS: - Expanded error_page_title regex to also catch forbidden, access denied, service unavailable, robot check, verify you are human (case-insensitive, anchored — so long-form essays about these topics still ingest fine) - New cloudflare_challenge_title pattern with DISTINCT name from error_page_title (PR #1561 collapsed both into one name and lost audit signal — the new name preserves diagnosability in ~/.gbrain/audit/content-sanity-YYYY-Www.jsonl and doctor's content_sanity_audit_recent aggregation) - Dropped PR #1561's bare-`error` matcher — too aggressive on legitimate concept/taxonomy pages titled exactly "Error" Tests: - test/content-sanity.test.ts: pattern-count locked at 7, new matches via test.each, over-match regression guard (legitimate prose titled "How to Handle Access Denied Errors" / "Error Boundary in React" etc. must pass), audit-name distinctness pinned - test/import-file-content-sanity.test.ts: end-to-end ContentSanityBlockError via importFromContent for each new pattern family (D6 — assessor wiring coverage, not just regex) Out of scope, filed in TODOS.md as TODO-V13-C: gbrain pages audit-junk-titles legacy-cleanup command. Dropped from this PR per codex outside-voice tension (T1) for ship-and-validate- matchers-first discipline. The 200+ pre-existing scraper pages already in the DB will get the destructive-cleanup operator surface after ~1 week of production observation against this matcher. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump v0.41.23.0 + CHANGELOG + follow-up TODOs VERSION + package.json bump to 0.41.23.0. CHANGELOG voice: ELI10 lead naming the bug ("`gbrain dream --source <id>` finally counts as a cycle"), then per-fix detail, then a "To take advantage of v0.41.23.0" operator-action block and itemized changes. TODOS.md v0.41.23.x follow-ups: - TODO-V13-A (P2): --max-pages plumbing (PR #1559's flag, deferred because CycleOpts has no maxPages field today) - TODO-V13-B (P3): --source vs --source-id flag-name unification across all CLI commands - TODO-V13-C (P2): gbrain pages audit-junk-titles legacy cleanup (deferred for ~1 week of matcher production observation) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump v0.41.25.0 → v0.41.26.0 (leave headroom for in-flight PR) Master shipped v0.41.23.0 + v0.41.24.0 mid-review; this branch originally bumped to v0.41.25.0 post-merge. User flagged v0.41.26.0 to leave a slot open for another in-flight PR. No code changes; VERSION + package.json + CHANGELOG header + "To take advantage" section updated in lockstep. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1265 lines
53 KiB
TypeScript
1265 lines
53 KiB
TypeScript
/**
|
||
* Synthesize phase (v0.23) — conversation-to-brain pipeline.
|
||
*
|
||
* Reads transcripts from the configured corpus dir, runs a cheap Haiku
|
||
* "is this worth processing?" verdict (cached in `dream_verdicts`), then
|
||
* fans out one Sonnet subagent per worth-processing transcript with the
|
||
* trusted-workspace `allowed_slug_prefixes` list. After children resolve,
|
||
* the orchestrator queries `subagent_tool_executions` for the put_page
|
||
* slugs each child wrote (codex finding #2: NOT a time-windowed pages
|
||
* query — picks up unrelated writes), reverse-renders each new page from
|
||
* DB to disk, and writes a deterministic summary index.
|
||
*
|
||
* Hard guarantees:
|
||
* - Subagent never gets fs-write access. Orchestrator holds the dual-write.
|
||
* - Allow-list is sourced from `skills/_brain-filing-rules.json` (single
|
||
* source of truth) and threaded as handler data; PROTECTED_JOB_NAMES
|
||
* prevents MCP from submitting `subagent` jobs, so the field is trusted.
|
||
* - Cooldown via `dream.synthesize.last_completion_ts` config key —
|
||
* written ONLY on success (codex finding #5 deferral: no auto git commit
|
||
* in v1).
|
||
* - Idempotency via `dream:synth:<file_path>:<content_hash>` job key.
|
||
* - Edited transcripts produce slugs with content-hash suffix → no overwrite.
|
||
*
|
||
* NOT in v1:
|
||
* - git auto-commit / push (deferred to v1.1, codex finding #5).
|
||
* - Daily token budget cap (cooldown bounds spend at v1 scale).
|
||
*/
|
||
|
||
import type Anthropic from '@anthropic-ai/sdk';
|
||
import { readFileSync, existsSync, writeFileSync, mkdirSync } from 'node:fs';
|
||
import { chat as gatewayChat, type ChatResult } from '../ai/gateway.ts';
|
||
import { resolveRecipe } from '../ai/model-resolver.ts';
|
||
import { AIConfigError } from '../ai/errors.ts';
|
||
import { loadConfig } from '../config.ts';
|
||
import { join, dirname, isAbsolute, resolve } from 'node:path';
|
||
import type { BrainEngine } from '../engine.ts';
|
||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||
import { MinionQueue } from '../minions/queue.ts';
|
||
import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.ts';
|
||
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
|
||
import { discoverTranscripts, type DiscoveredTranscript } from './transcript-discovery.ts';
|
||
import { serializeMarkdown, serializePageToMarkdown } from '../markdown.ts';
|
||
import type { Page, PageType } from '../types.ts';
|
||
import { validateSourceId } from '../utils.ts';
|
||
import { safeSplitIndex } from '../text-safe.ts';
|
||
|
||
// Slug regex from validatePageSlug — kept in sync.
|
||
// Used for the orchestrator-written summary index slug.
|
||
const SUMMARY_SLUG_RE = /^[a-z0-9][a-z0-9\-]*(\/[a-z0-9][a-z0-9\-]*)*$/;
|
||
|
||
// ── Model context budget (D1, D5, D7, D9) ─────────────────────────────
|
||
|
||
/**
|
||
* Anthropic model id → input context window (tokens).
|
||
* Unknown id (non-Anthropic alias, custom string) → safe 200K-token fallback
|
||
* via `computeChunkCharBudget`. Codex finding #4: `resolveModel()` does not
|
||
* canonicalize to Anthropic-only; this map keys on the exact strings the
|
||
* resolver returns for known Anthropic aliases.
|
||
*/
|
||
const MODEL_CONTEXT_TOKENS: Record<string, number> = {
|
||
'claude-opus-4-7': 1_000_000,
|
||
'claude-opus-4-6': 1_000_000,
|
||
'claude-sonnet-4-6': 200_000,
|
||
'claude-sonnet-4-5': 200_000,
|
||
'claude-haiku-4-5-20251001': 200_000,
|
||
};
|
||
|
||
/** Token-to-char ratio. 3.5 matches PR #748; conservative for English text. */
|
||
const CHARS_PER_TOKEN = 3.5;
|
||
/** Reserve 10% of context window for system prompt + tool defs + output. */
|
||
const HEADROOM_RATIO = 0.9;
|
||
/** Floor on user-overridable max_prompt_tokens (matches PR #748 minimum). */
|
||
const MIN_PROMPT_TOKENS = 100_000;
|
||
/** Default chunk-count cap; operator-configurable via dream.synthesize.max_chunks_per_transcript. */
|
||
const DEFAULT_MAX_CHUNKS = 24;
|
||
/** Conservative default budget when model is unknown (200K × HEADROOM_RATIO). */
|
||
const UNKNOWN_MODEL_BUDGET_TOKENS = 180_000;
|
||
|
||
/**
|
||
* Compute per-chunk character budget for the resolved model + config override.
|
||
*
|
||
* Resolution:
|
||
* - configMaxPromptTokens (already floored at MIN_PROMPT_TOKENS) wins when set.
|
||
* - Else the model's MODEL_CONTEXT_TOKENS entry × HEADROOM_RATIO.
|
||
* - Else (non-Anthropic alias / custom id) UNKNOWN_MODEL_BUDGET_TOKENS, with
|
||
* a once-per-process stderr warning.
|
||
*
|
||
* D7 scope: this bounds the INITIAL prompt size only. Tool-loop turn-N
|
||
* accumulation is out of scope for v0.30.2 (terminal-error classification
|
||
* catches turn-N blowups; per-turn budget guard is a v0.31+ follow-up).
|
||
*/
|
||
function computeChunkCharBudget(
|
||
model: string,
|
||
configMaxPromptTokens: number | null,
|
||
): number {
|
||
if (configMaxPromptTokens !== null) {
|
||
return Math.floor(configMaxPromptTokens * CHARS_PER_TOKEN);
|
||
}
|
||
const ctx = MODEL_CONTEXT_TOKENS[model];
|
||
if (ctx === undefined) {
|
||
warnUnknownModelOnce(model);
|
||
return Math.floor(UNKNOWN_MODEL_BUDGET_TOKENS * CHARS_PER_TOKEN);
|
||
}
|
||
return Math.floor(ctx * HEADROOM_RATIO * CHARS_PER_TOKEN);
|
||
}
|
||
|
||
const _unknownModelWarned = new Set<string>();
|
||
function warnUnknownModelOnce(model: string): void {
|
||
if (_unknownModelWarned.has(model)) return;
|
||
_unknownModelWarned.add(model);
|
||
process.stderr.write(
|
||
`[dream] model "${model}" is not in MODEL_CONTEXT_TOKENS; ` +
|
||
`using ${UNKNOWN_MODEL_BUDGET_TOKENS}-token fallback budget. ` +
|
||
`Set dream.synthesize.max_prompt_tokens to override.\n`,
|
||
);
|
||
}
|
||
|
||
// ── Hash-deterministic transcript chunker (D9) ────────────────────────
|
||
|
||
/**
|
||
* Split content into chunks at most maxChars long, picking boundaries via a
|
||
* 3-tier ladder lifted from PR #748:
|
||
* 1. `## Topic:` separators (matches the daily-aggregated transcript shape)
|
||
* 2. `---` markdown HR markers
|
||
* 3. nearest `\n` newline
|
||
*
|
||
* D9 stable chunk identity: the back-half-of-budget search window is seeded
|
||
* with a deterministic offset derived from contentHash so the same
|
||
* (content, contentHash, maxChars) triple always produces identical chunks.
|
||
* Closes the partial-progress ambiguity: chunk 2 of a transcript that
|
||
* previously failed terminally produces byte-identical content on retry,
|
||
* so the per-chunk idempotency key is durable across runs.
|
||
*
|
||
* The hash-derived offset jitters the search start within
|
||
* [0.5×budget, 0.6×budget] so the back-half rule still holds.
|
||
*
|
||
* If no boundary fits, hard-split at maxChars (also deterministic in the
|
||
* inputs).
|
||
*
|
||
* Pure function. Tested by `test/cycle/synthesize-chunker.test.ts`.
|
||
*/
|
||
export function splitTranscriptByBudget(
|
||
content: string,
|
||
contentHash: string,
|
||
maxChars: number,
|
||
): string[] {
|
||
if (maxChars <= 0) {
|
||
throw new Error(`splitTranscriptByBudget: maxChars must be > 0, got ${maxChars}`);
|
||
}
|
||
if (content.length <= maxChars) return [content];
|
||
|
||
const hashInt = parseHashOffset(contentHash);
|
||
// Jitter window is the next 10% of budget after the 50% midpoint.
|
||
const jitterRange = Math.max(1, Math.floor(maxChars * 0.1));
|
||
const searchStart = Math.floor(maxChars * 0.5) + (hashInt % jitterRange);
|
||
|
||
const out: string[] = [];
|
||
let remaining = content;
|
||
while (remaining.length > maxChars) {
|
||
const split = findBoundary(remaining, maxChars, searchStart);
|
||
out.push(remaining.slice(0, split));
|
||
remaining = remaining.slice(split);
|
||
}
|
||
if (remaining.length > 0) out.push(remaining);
|
||
return out;
|
||
}
|
||
|
||
function parseHashOffset(contentHash: string): number {
|
||
// First 8 hex chars = 32 bits; plenty of entropy for the offset jitter.
|
||
const hex = contentHash.slice(0, 8);
|
||
const n = parseInt(hex, 16);
|
||
return Number.isFinite(n) && n >= 0 ? n : 0;
|
||
}
|
||
|
||
function findBoundary(text: string, maxChars: number, searchStart: number): number {
|
||
const window = text.slice(searchStart, maxChars);
|
||
// Tier 1: "\n## Topic:" — last occurrence inside the search window.
|
||
const topicIdx = window.lastIndexOf('\n## Topic:');
|
||
if (topicIdx >= 0) return searchStart + topicIdx;
|
||
// Tier 2: "\n---\n" markdown HR.
|
||
const hrIdx = window.lastIndexOf('\n---\n');
|
||
if (hrIdx >= 0) return searchStart + hrIdx;
|
||
// Tier 3: any newline.
|
||
const nlIdx = window.lastIndexOf('\n');
|
||
if (nlIdx >= 0) return searchStart + nlIdx;
|
||
// No boundary fits; hard-split at maxChars (deterministic).
|
||
// v0.42.0.0: route through safeSplitIndex so a hard-split that lands
|
||
// between a UTF-16 surrogate pair (emoji / non-BMP CJK / mathematical
|
||
// alphanumerics) doesn't orphan the high surrogate — that would change
|
||
// chunk byte-content vs the source and break the D9 stable-chunk-identity
|
||
// invariant on the next retry.
|
||
return safeSplitIndex(text, maxChars);
|
||
}
|
||
|
||
/**
|
||
* D6: orchestrator-side deterministic slug rewrite. Zero Sonnet trust.
|
||
*
|
||
* Expected shape from `buildSynthesisPrompt` for a chunked child is already
|
||
* `<base>-<hash6>-c<idx>`, but if Sonnet drops the chunk suffix this rewrite
|
||
* enforces uniqueness post-hoc. Same hash AND same chunk idx → idempotent.
|
||
*
|
||
* Pure function. Cases:
|
||
* - already correctly suffixed (`...-<hash6>-c<idx>`) → return unchanged.
|
||
* - bare hash suffix (`...-<hash6>`) → append `-c<idx>`.
|
||
* - some other shape → pass through (orchestrator can't safely guess
|
||
* where to inject the chunk index; e2e test pins this).
|
||
*/
|
||
export function rewriteChunkedSlug(slug: string, hash6: string, idx: number): string {
|
||
if (!slug) return slug;
|
||
const expected = `${hash6}-c${idx}`;
|
||
// Already correctly chunk-suffixed.
|
||
if (slug === expected) return slug;
|
||
if (slug.endsWith(`-${expected}`) || slug.endsWith(`/${expected}`)) return slug;
|
||
// Bare hash6 at end of last path segment: rewrite.
|
||
// Match either at start-of-slug, after a "/" path separator, or after a "-".
|
||
const re = new RegExp(`(^|[/-])${hash6}$`);
|
||
if (re.test(slug)) return `${slug}-c${idx}`;
|
||
// Unknown shape — pass through; collision risk is now bounded by Sonnet's
|
||
// per-chunk-prompt guidance and the existing slug-prefix allow-list.
|
||
return slug;
|
||
}
|
||
|
||
// ── Public entry ──────────────────────────────────────────────────────
|
||
|
||
export interface SynthesizePhaseOpts {
|
||
brainDir: string;
|
||
dryRun: boolean;
|
||
/** Generic in-cycle keepalive for cycle-lock TTL renewal during long waits. */
|
||
yieldDuringPhase?: () => Promise<void>;
|
||
/**
|
||
* Override the corpus directory and other tunables. Primarily for the
|
||
* `gbrain dream --input <file>` ad-hoc path; bypasses config reads.
|
||
*/
|
||
inputFile?: string;
|
||
date?: string;
|
||
from?: string;
|
||
to?: string;
|
||
/**
|
||
* Disable the self-consumption guard. Wired from the
|
||
* `--unsafe-bypass-dream-guard` CLI flag. NOT auto-applied for `--input`
|
||
* because that would allow any dream-generated page to silently re-enter
|
||
* the synthesize loop. Caller must opt in explicitly.
|
||
*/
|
||
bypassDreamGuard?: boolean;
|
||
}
|
||
|
||
export async function runPhaseSynthesize(
|
||
engine: BrainEngine,
|
||
opts: SynthesizePhaseOpts,
|
||
): Promise<PhaseResult> {
|
||
const start = Date.now();
|
||
// Normalize brainDir to an absolute path BEFORE any reverse-write. Without
|
||
// this, a relative or empty brainDir flows down to writeReversePages →
|
||
// `join(brainDir, '${slug}.md')` → relative path → resolves against cwd at
|
||
// writeFileSync time, spilling synthesize output into whatever directory
|
||
// the cycle ran from (e.g., `companies/novamind.md` at the repo root).
|
||
// Surfaced by the warm-narwhal wave when E2E test cleanup found orphan
|
||
// synthesize pages at repo root from a `runCycle({brainDir: '.'})` call
|
||
// chain. Throw on empty (silent cwd-resolution is worse than a loud
|
||
// failure); resolve if relative (`.` / `./brain` / `../sibling` all valid
|
||
// inputs but must canonicalize before the write).
|
||
if (!opts.brainDir || opts.brainDir.trim() === '') {
|
||
return failed(makeError('InternalError', 'BRAINDIR_EMPTY',
|
||
'opts.brainDir is empty; refusing to run synthesize. Pass an absolute path.'));
|
||
}
|
||
if (!isAbsolute(opts.brainDir)) {
|
||
opts.brainDir = resolve(opts.brainDir);
|
||
}
|
||
try {
|
||
const config = await loadSynthConfig(engine);
|
||
|
||
// Allow ad-hoc --input to run even when config is disabled.
|
||
if (!opts.inputFile && !config.corpusDir) {
|
||
return skipped('not_configured',
|
||
'dream.synthesize.session_corpus_dir is unset');
|
||
}
|
||
if (!opts.inputFile && !config.enabled) {
|
||
return skipped('not_configured',
|
||
'dream.synthesize.enabled is explicitly false');
|
||
}
|
||
|
||
// Cooldown check (skipped for explicit --input / --date / --from / --to runs).
|
||
const explicitTarget = opts.inputFile || opts.date || opts.from || opts.to;
|
||
if (!explicitTarget) {
|
||
const cooldown = await checkCooldown(engine, config.cooldownHours);
|
||
if (cooldown.active) {
|
||
return skipped('cooldown_active',
|
||
`synthesize cooled down until ${cooldown.expires_at} (${config.cooldownHours}h cooldown)`);
|
||
}
|
||
}
|
||
|
||
if (opts.bypassDreamGuard) {
|
||
process.stderr.write(
|
||
'[dream] WARNING: --unsafe-bypass-dream-guard set; self-consumption guard disabled. ' +
|
||
'Re-ingestion of dream output will incur Sonnet costs forever.\n',
|
||
);
|
||
}
|
||
|
||
// v0.32.6 M2: pre-fetch prior contradictions from the most recent probe
|
||
// run (if any). Surfaced as an informational block to the synthesize
|
||
// subagent so it knows which slugs it should reconcile if it writes to
|
||
// them. Best-effort — a probe that's never run is a normal early state.
|
||
const priorContradictionsBlock = await loadPriorContradictionsBlock(engine);
|
||
|
||
// Discover.
|
||
const transcripts = opts.inputFile
|
||
? loadAdHocTranscript(opts.inputFile, config.minChars, config.excludePatterns, opts.bypassDreamGuard)
|
||
: discoverTranscripts({
|
||
corpusDir: config.corpusDir!,
|
||
meetingTranscriptsDir: config.meetingTranscriptsDir ?? undefined,
|
||
minChars: config.minChars,
|
||
excludePatterns: config.excludePatterns,
|
||
date: opts.date,
|
||
from: opts.from,
|
||
to: opts.to,
|
||
bypassGuard: opts.bypassDreamGuard,
|
||
});
|
||
|
||
if (transcripts.length === 0) {
|
||
return ok('no transcripts to process', { transcripts_processed: 0, pages_written: 0 });
|
||
}
|
||
|
||
// Significance verdicts (cached in dream_verdicts; Haiku on miss).
|
||
const worthProcessing: DiscoveredTranscript[] = [];
|
||
const verdicts: Array<{ filePath: string; worth: boolean; reasons: string[]; cached: boolean }> = [];
|
||
// Provider-aware judge client routes through gateway.chat, so any
|
||
// configured provider works (Anthropic, DeepSeek, OpenRouter, Voyage,
|
||
// Ollama, llama-server, etc.). Returns null when the resolved verdict
|
||
// model has no reachable provider (legacy "no API key" branch preserved
|
||
// as the cheap pre-flight check).
|
||
const judge = makeJudgeClient(config.verdictModel);
|
||
for (const t of transcripts) {
|
||
const cached = await engine.getDreamVerdict(t.filePath, t.contentHash);
|
||
if (cached) {
|
||
verdicts.push({ filePath: t.filePath, worth: cached.worth_processing, reasons: cached.reasons, cached: true });
|
||
if (cached.worth_processing) worthProcessing.push(t);
|
||
continue;
|
||
}
|
||
if (!judge) {
|
||
// No configured provider for the verdict model — can't judge.
|
||
// Skip with explicit reason; don't crash phase.
|
||
verdicts.push({
|
||
filePath: t.filePath,
|
||
worth: false,
|
||
reasons: [`no configured provider for verdict model: ${config.verdictModel}`],
|
||
cached: false,
|
||
});
|
||
continue;
|
||
}
|
||
try {
|
||
const verdict = await judgeSignificance(judge, t, config.verdictModel);
|
||
await engine.putDreamVerdict(t.filePath, t.contentHash, verdict);
|
||
verdicts.push({ filePath: t.filePath, worth: verdict.worth_processing, reasons: verdict.reasons, cached: false });
|
||
if (verdict.worth_processing) worthProcessing.push(t);
|
||
} catch (e) {
|
||
// AIConfigError at chat time = provider auth/config went bad mid-run
|
||
// (revoked key, recipe misconfig surfacing at first real call). Skip
|
||
// this transcript with the gateway error message so the user sees the
|
||
// shape of the problem in `gbrain dream --phase synthesize --dry-run`.
|
||
if (e instanceof AIConfigError) {
|
||
verdicts.push({
|
||
filePath: t.filePath,
|
||
worth: false,
|
||
reasons: [`gateway error: ${e.message}`],
|
||
cached: false,
|
||
});
|
||
continue;
|
||
}
|
||
throw e;
|
||
}
|
||
}
|
||
|
||
// Dry-run stops here: significance filter ran (Haiku verdicts cached),
|
||
// but no Sonnet synthesis. Codex finding #8: --dry-run does NOT mean
|
||
// "zero LLM calls"; it means "skip Sonnet."
|
||
if (opts.dryRun) {
|
||
return ok(`dry-run: ${worthProcessing.length} of ${transcripts.length} transcripts would synthesize`, {
|
||
transcripts_discovered: transcripts.length,
|
||
transcripts_processed: 0,
|
||
pages_written: 0,
|
||
verdicts,
|
||
dryRun: true,
|
||
});
|
||
}
|
||
|
||
if (worthProcessing.length === 0) {
|
||
// Even with verdicts, the cooldown timestamp is updated only on a
|
||
// real successful run — not on "nothing worth processing." Lets a
|
||
// re-run pick up if a new transcript lands later.
|
||
return ok('all transcripts skipped by significance filter', {
|
||
transcripts_discovered: transcripts.length,
|
||
transcripts_processed: 0,
|
||
pages_written: 0,
|
||
verdicts,
|
||
});
|
||
}
|
||
|
||
// Fan-out: submit one subagent per worth-processing transcript (or one
|
||
// per chunk for transcripts that exceed the model's per-prompt budget).
|
||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||
if (allowedSlugPrefixes.length === 0) {
|
||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||
}
|
||
|
||
const queue = new MinionQueue(engine);
|
||
const childIds: number[] = [];
|
||
/** Map child job_id → chunk metadata for D6 orchestrator-side slug rewrite. */
|
||
const chunkInfo = new Map<number, { idx: number; hash6: string }>();
|
||
/** Skip reasons for the cycle report (D5 cap hits, D8 legacy-key skips). */
|
||
const skipReports: Array<{ filePath: string; reason: string }> = [];
|
||
|
||
const maxCharsPerChunk = computeChunkCharBudget(config.model, config.maxPromptTokens);
|
||
|
||
for (const t of worthProcessing) {
|
||
const hash16 = t.contentHash.slice(0, 16);
|
||
const hash6 = t.contentHash.slice(0, 6);
|
||
|
||
// D8: single→multi-chunk migration safety. If a completed legacy
|
||
// single-chunk job exists for this content_hash, treat as already-
|
||
// synthesized and skip. Prevents duplicate writes when a transcript
|
||
// that was previously single-chunk now multi-chunks (because budget
|
||
// shrank or model changed).
|
||
if (await hasLegacySingleChunkCompletion(engine, t.filePath, hash16)) {
|
||
skipReports.push({
|
||
filePath: t.filePath,
|
||
reason: 'already_synthesized_legacy_single_chunk',
|
||
});
|
||
continue;
|
||
}
|
||
|
||
const chunks = splitTranscriptByBudget(t.content, t.contentHash, maxCharsPerChunk);
|
||
|
||
// D5 cap hit: log + skip; do NOT write to dream_verdicts. Closes the
|
||
// poison-pill class — next cycle re-attempts under whatever budget
|
||
// is then current.
|
||
if (chunks.length > config.maxChunksPerTranscript) {
|
||
process.stderr.write(
|
||
`[dream] transcript ${t.basename} produced ${chunks.length} chunks at ` +
|
||
`${maxCharsPerChunk}-char budget (cap=${config.maxChunksPerTranscript}); skipping. ` +
|
||
`Increase dream.synthesize.max_chunks_per_transcript or use a larger-context model.\n`,
|
||
);
|
||
skipReports.push({
|
||
filePath: t.filePath,
|
||
reason: `oversize_after_split: ${chunks.length}/${config.maxChunksPerTranscript}`,
|
||
});
|
||
continue;
|
||
}
|
||
|
||
const isChunked = chunks.length > 1;
|
||
// queue.add subagent validator (classifyCapabilities → resolveRecipe)
|
||
// requires `provider:model`. resolveModel can return a bare id when
|
||
// TIER_DEFAULTS / DEFAULT_ALIASES carry a bare value; ensure the
|
||
// anthropic: prefix is present for known claude-* ids before passing
|
||
// to the queue. Non-anthropic providers must already declare a colon.
|
||
const subagentModel = config.model.includes(':')
|
||
? config.model
|
||
: config.model.toLowerCase().startsWith('claude-')
|
||
? `anthropic:${config.model}`
|
||
: config.model;
|
||
for (let i = 0; i < chunks.length; i++) {
|
||
const childData: SubagentHandlerData = {
|
||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock),
|
||
model: subagentModel,
|
||
max_turns: 30,
|
||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||
};
|
||
// Idempotency key parity:
|
||
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
|
||
// equivalent across versions; preserves dedup for unchanged
|
||
// transcripts on upgrade).
|
||
// - multi-chunk → `<legacy>:c<i>of<n>` per chunk; durable across
|
||
// runs because D9 splitTranscriptByBudget is hash-deterministic.
|
||
const idempotency_key = isChunked
|
||
? `dream:synth:${t.filePath}:${hash16}:c${i}of${chunks.length}`
|
||
: `dream:synth:${t.filePath}:${hash16}`;
|
||
const submitOpts: Partial<MinionJobInput> = {
|
||
max_stalled: 3,
|
||
on_child_fail: 'continue',
|
||
idempotency_key,
|
||
timeout_ms: 30 * 60 * 1000, // 30 min per chunk
|
||
};
|
||
const child = await queue.add(
|
||
'subagent',
|
||
childData as unknown as Record<string, unknown>,
|
||
submitOpts,
|
||
{ allowProtectedSubmit: true },
|
||
);
|
||
childIds.push(child.id);
|
||
if (isChunked) {
|
||
chunkInfo.set(child.id, { idx: i, hash6 });
|
||
}
|
||
}
|
||
}
|
||
|
||
// Wait for every child to reach a terminal state. Tick yieldDuringPhase
|
||
// every 5 min so the cycle lock TTL refreshes.
|
||
const childOutcomes: Array<{ jobId: number; status: string }> = [];
|
||
for (const jobId of childIds) {
|
||
try {
|
||
const job = await waitForCompletion(queue, jobId, {
|
||
timeoutMs: 35 * 60 * 1000,
|
||
pollMs: 5 * 1000,
|
||
});
|
||
childOutcomes.push({ jobId, status: job.status });
|
||
} catch (e) {
|
||
if (e instanceof TimeoutError) {
|
||
childOutcomes.push({ jobId, status: 'timeout' });
|
||
} else {
|
||
throw e;
|
||
}
|
||
}
|
||
// After each child terminal, give the cycle lock + worker job lock a chance.
|
||
if (opts.yieldDuringPhase) {
|
||
try { await opts.yieldDuringPhase(); } catch { /* best-effort */ }
|
||
}
|
||
}
|
||
|
||
// Collect slugs from put_page tool executions across the children
|
||
// (codex finding #2: deterministic provenance, NOT pages.updated_at).
|
||
// D6 orchestrator slug rewrite: chunkInfo drives post-hoc rewrite of
|
||
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
|
||
// even if Sonnet drops the chunk suffix.
|
||
// v0.32.8: refs carry source_id so reverseWriteRefs picks the correct
|
||
// (source, slug) row (currently always 'default' from subagent put_page).
|
||
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
|
||
|
||
// Dual-write: reverse-render each DB row → markdown file.
|
||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
|
||
|
||
// Summary index page (deterministic; orchestrator-written via direct
|
||
// engine.putPage so no allow-list path needed).
|
||
const summaryDate = opts.date ?? today();
|
||
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
|
||
// Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs.
|
||
const writtenSlugs = writtenRefs.map(r => r.slug);
|
||
if (SUMMARY_SLUG_RE.test(summarySlug)) {
|
||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
|
||
}
|
||
|
||
// Write completion timestamp ON SUCCESS only.
|
||
await engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString());
|
||
|
||
const ms = Date.now() - start;
|
||
const submittedTranscripts = worthProcessing.length - skipReports.length;
|
||
return ok(`${submittedTranscripts} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, {
|
||
transcripts_discovered: transcripts.length,
|
||
transcripts_processed: submittedTranscripts,
|
||
pages_written: writtenSlugs.length,
|
||
// v0.29: emit the slug list so the recompute_emotional_weight phase can
|
||
// union with sync's pagesAffected and recompute weights for every page
|
||
// synthesize wrote in this cycle.
|
||
written_slugs: writtenSlugs,
|
||
reverse_write_count: reverseWriteCount,
|
||
child_outcomes: childOutcomes,
|
||
// Children submitted (one per chunk for chunked transcripts; one per
|
||
// transcript for single-chunk). Differs from transcripts_processed
|
||
// when chunking is in play.
|
||
children_submitted: childIds.length,
|
||
// D5 cap hits + D8 legacy-key skips. Empty when nothing skipped.
|
||
skips: skipReports,
|
||
summary_slug: summarySlug,
|
||
verdicts,
|
||
});
|
||
} catch (e) {
|
||
return failed(makeError('InternalError', 'SYNTH_PHASE_FAIL',
|
||
e instanceof Error ? (e.message || 'synthesize phase threw') : String(e)));
|
||
}
|
||
}
|
||
|
||
// ── Config ────────────────────────────────────────────────────────────
|
||
|
||
interface SynthConfig {
|
||
enabled: boolean;
|
||
corpusDir: string | null;
|
||
meetingTranscriptsDir: string | null;
|
||
minChars: number;
|
||
excludePatterns: string[];
|
||
model: string;
|
||
verdictModel: string;
|
||
cooldownHours: number;
|
||
/**
|
||
* D1: Override the per-chunk token budget (model_context × HEADROOM_RATIO
|
||
* by default). Floor MIN_PROMPT_TOKENS, no upper cap (model context wins).
|
||
* Surface name follows PR #748: `dream.synthesize.max_prompt_tokens`.
|
||
* `null` means use the model-context lookup.
|
||
*/
|
||
maxPromptTokens: number | null;
|
||
/**
|
||
* D5/D10: Cap on chunks produced from a single transcript. On cap hit, the
|
||
* transcript is logged + skipped (NOT cached in dream_verdicts — closes the
|
||
* cache-poisoning class). Operator override:
|
||
* `dream.synthesize.max_chunks_per_transcript`.
|
||
*/
|
||
maxChunksPerTranscript: number;
|
||
}
|
||
|
||
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||
const enabledRaw = await engine.getConfig('dream.synthesize.enabled');
|
||
const corpusDir = await engine.getConfig('dream.synthesize.session_corpus_dir');
|
||
// v2: enabled defaults to true when corpus dir is configured, false otherwise.
|
||
// Explicit enabled=false still wins for pausing synthesis without removing corpus config.
|
||
const enabled = enabledRaw === 'false' ? false : (enabledRaw === 'true' || !!corpusDir);
|
||
const meetingTranscriptsDir = await engine.getConfig('dream.synthesize.meeting_transcripts_dir');
|
||
const minCharsStr = await engine.getConfig('dream.synthesize.min_chars');
|
||
const excludeStr = await engine.getConfig('dream.synthesize.exclude_patterns');
|
||
// v0.28: resolveModel() unifies CLI flag > new key > deprecated key > models.default > env > fallback
|
||
const { resolveModel } = await import('../model-config.ts');
|
||
const model = await resolveModel(engine, {
|
||
configKey: 'models.dream.synthesize',
|
||
deprecatedConfigKey: 'dream.synthesize.model',
|
||
tier: 'reasoning',
|
||
fallback: 'sonnet',
|
||
});
|
||
const verdictModel = await resolveModel(engine, {
|
||
configKey: 'models.dream.synthesize_verdict',
|
||
deprecatedConfigKey: 'dream.synthesize.verdict_model',
|
||
tier: 'utility',
|
||
fallback: 'haiku',
|
||
});
|
||
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
|
||
const maxPromptTokensStr = await engine.getConfig('dream.synthesize.max_prompt_tokens');
|
||
const maxChunksStr = await engine.getConfig('dream.synthesize.max_chunks_per_transcript');
|
||
|
||
let excludePatterns: string[] = ['medical', 'therapy'];
|
||
if (excludeStr) {
|
||
try {
|
||
const parsed = JSON.parse(excludeStr);
|
||
if (Array.isArray(parsed)) excludePatterns = parsed.filter(p => typeof p === 'string');
|
||
} catch { /* keep default */ }
|
||
}
|
||
|
||
// D1: max_prompt_tokens floored at MIN_PROMPT_TOKENS; null → use model lookup.
|
||
let maxPromptTokens: number | null = null;
|
||
if (maxPromptTokensStr) {
|
||
const parsed = parseInt(maxPromptTokensStr, 10);
|
||
if (Number.isFinite(parsed) && parsed > 0) {
|
||
maxPromptTokens = Math.max(MIN_PROMPT_TOKENS, parsed);
|
||
}
|
||
}
|
||
// D10: max_chunks default 24, floor 1.
|
||
let maxChunksPerTranscript = DEFAULT_MAX_CHUNKS;
|
||
if (maxChunksStr) {
|
||
const parsed = parseInt(maxChunksStr, 10);
|
||
if (Number.isFinite(parsed) && parsed >= 1) {
|
||
maxChunksPerTranscript = parsed;
|
||
}
|
||
}
|
||
|
||
return {
|
||
enabled,
|
||
corpusDir: corpusDir ?? null,
|
||
meetingTranscriptsDir: meetingTranscriptsDir ?? null,
|
||
minChars: minCharsStr ? Math.max(0, parseInt(minCharsStr, 10) || 2000) : 2000,
|
||
excludePatterns,
|
||
model,
|
||
verdictModel,
|
||
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
|
||
maxPromptTokens,
|
||
maxChunksPerTranscript,
|
||
};
|
||
}
|
||
|
||
async function checkCooldown(
|
||
engine: BrainEngine,
|
||
hours: number,
|
||
): Promise<{ active: boolean; expires_at?: string }> {
|
||
if (hours <= 0) return { active: false };
|
||
const last = await engine.getConfig('dream.synthesize.last_completion_ts');
|
||
if (!last) return { active: false };
|
||
const lastMs = Date.parse(last);
|
||
if (Number.isNaN(lastMs)) return { active: false };
|
||
const expiresMs = lastMs + hours * 60 * 60 * 1000;
|
||
if (Date.now() >= expiresMs) return { active: false };
|
||
return { active: true, expires_at: new Date(expiresMs).toISOString() };
|
||
}
|
||
|
||
// ── Allow-list source of truth ───────────────────────────────────────
|
||
|
||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||
// Search a few known locations relative to the binary / repo. The first
|
||
// hit wins; if none found, return [].
|
||
const candidates = [
|
||
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
|
||
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
|
||
];
|
||
for (const path of candidates) {
|
||
if (!existsSync(path)) continue;
|
||
try {
|
||
const raw = readFileSync(path, 'utf8');
|
||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||
return globs as string[];
|
||
}
|
||
} catch { /* try next */ }
|
||
}
|
||
return [];
|
||
}
|
||
|
||
// ── Significance judge (gateway-routed; provider-agnostic) ──────────────
|
||
//
|
||
// The JudgeClient interface is unchanged for test-seam stability — existing
|
||
// tests that pass a mock client to judgeSignificance keep working byte-
|
||
// identically. Only the construction path moved from `new Anthropic()` to
|
||
// `gateway.chat()` so any provider with a registered recipe (Anthropic,
|
||
// DeepSeek, OpenRouter, Voyage, Ollama, llama-server, etc.) is reachable
|
||
// via `gbrain config set models.dream.synthesize_verdict <provider>:<model>`.
|
||
//
|
||
// This mirrors v0.35.5.0's `tryBuildGatewayClient` in src/core/think/index.ts
|
||
// (which closed #952 for runThink). Same pattern, same trade-offs:
|
||
// construction-time provider/key probe returns null on a clear miss (cheap
|
||
// pre-flight), and the verdict loop wraps the actual chat call in try/catch
|
||
// for AIConfigError surfacing mid-run.
|
||
|
||
export interface JudgeClient {
|
||
create: (params: Anthropic.MessageCreateParamsNonStreaming) => Promise<Anthropic.Message>;
|
||
}
|
||
|
||
/**
|
||
* Build a gateway-routed JudgeClient for the resolved verdict model.
|
||
* Returns null when no chat provider is reachable for `verdictModel`:
|
||
* - Unknown provider id (resolveRecipe throws AIConfigError).
|
||
* - Anthropic provider with no key (env or config) — preserves the legacy
|
||
* "no ANTHROPIC_API_KEY" cheap-skip semantics.
|
||
* On null, the verdict loop short-circuits each transcript with an explicit
|
||
* "no configured provider" reason and continues the phase.
|
||
*
|
||
* For non-Anthropic providers (deepseek, openrouter, voyage, ollama,
|
||
* llama-server, ...), we delegate auth probing to the gateway's own
|
||
* recipe `auth_env.required` machinery — AIConfigError at gateway.chat()
|
||
* time is caught by the verdict loop and surfaced per-transcript.
|
||
*/
|
||
export function makeJudgeClient(verdictModel: string): JudgeClient | null {
|
||
// Normalize: ensure provider:model shape. resolveModel returns bare
|
||
// anthropic ids (e.g. `claude-haiku-4-5-20251001`); gateway.chat needs
|
||
// `anthropic:...`.
|
||
const modelStr = verdictModel.includes(':') ? verdictModel : `anthropic:${verdictModel}`;
|
||
|
||
// Availability probe: resolveRecipe throws AIConfigError on unknown provider.
|
||
let providerId: string;
|
||
try {
|
||
const { parsed } = resolveRecipe(modelStr);
|
||
providerId = parsed.providerId;
|
||
} catch (e) {
|
||
if (e instanceof AIConfigError) return null;
|
||
throw e;
|
||
}
|
||
|
||
// Anthropic key probe (legacy behavior preserved). Other providers'
|
||
// key checks happen lazily at chat call time and surface as
|
||
// AIConfigError, which the verdict loop catches per-transcript.
|
||
if (providerId === 'anthropic' && !hasAnthropicKey()) return null;
|
||
|
||
return {
|
||
create: async (params): Promise<Anthropic.Message> => {
|
||
// Map Anthropic.MessageCreateParamsNonStreaming → gateway.ChatOpts.
|
||
// `judgeSignificance` always sends string content + string system,
|
||
// and the adapter only TEXT-flattens the array-of-blocks shape —
|
||
// `tool_use`, `tool_result`, image, and other non-text blocks become
|
||
// empty strings. If a future caller wires tool-use or image content
|
||
// through this client, extend the mapping instead of relying on the
|
||
// current silent drop. Same pattern as think/index.ts:607-615.
|
||
const messages = params.messages.map(m => ({
|
||
role: m.role,
|
||
content: typeof m.content === 'string'
|
||
? m.content
|
||
: (Array.isArray(m.content)
|
||
? m.content.map(b => ('text' in b ? b.text : '')).join('')
|
||
: ''),
|
||
}));
|
||
const system = typeof params.system === 'string'
|
||
? params.system
|
||
: (Array.isArray(params.system)
|
||
? params.system.map(b => ('text' in b ? b.text : '')).join('')
|
||
: undefined);
|
||
|
||
const result: ChatResult = await gatewayChat({
|
||
model: modelStr,
|
||
system,
|
||
messages,
|
||
maxTokens: params.max_tokens,
|
||
});
|
||
|
||
// Map gateway.ChatResult → Anthropic.Message shape. judgeSignificance
|
||
// reads `.content[0].type === 'text'` and `.content[0].text`; other
|
||
// fields are best-effort for downstream telemetry parity.
|
||
return {
|
||
id: '',
|
||
type: 'message',
|
||
role: 'assistant',
|
||
model: modelStr,
|
||
content: [{ type: 'text', text: result.text }],
|
||
stop_reason: 'end_turn',
|
||
stop_sequence: null,
|
||
usage: {
|
||
input_tokens: result.usage.input_tokens,
|
||
output_tokens: result.usage.output_tokens,
|
||
},
|
||
} as unknown as Anthropic.Message;
|
||
},
|
||
};
|
||
}
|
||
|
||
/**
|
||
* Anthropic key availability probe. Reads BOTH env (`ANTHROPIC_API_KEY`)
|
||
* AND the gbrain config file (`anthropic_api_key` set via
|
||
* `gbrain config set`) so stdio MCP launches that don't inherit shell env
|
||
* keep working (mirrors `hasAnthropicKey()` in src/core/think/index.ts).
|
||
*/
|
||
function hasAnthropicKey(): boolean {
|
||
if (process.env.ANTHROPIC_API_KEY) return true;
|
||
try {
|
||
const cfg = loadConfig();
|
||
if (cfg?.anthropic_api_key) return true;
|
||
} catch {
|
||
// loadConfig may throw on first-run installs; treat as no key.
|
||
}
|
||
return false;
|
||
}
|
||
|
||
interface VerdictResult {
|
||
worth_processing: boolean;
|
||
reasons: string[];
|
||
}
|
||
|
||
export async function judgeSignificance(
|
||
client: JudgeClient,
|
||
t: DiscoveredTranscript,
|
||
verdictModel = 'claude-haiku-4-5-20251001',
|
||
): Promise<VerdictResult> {
|
||
// Truncate the transcript at 8K chars for cost control. Haiku's verdict
|
||
// doesn't need the full body; the opening + closing sections are usually
|
||
// representative of significance.
|
||
//
|
||
// v0.41.13 surrogate-safety (supersedes PRs #1559+#1561's safeSliceEnd
|
||
// helper; see text-safe.ts:18-21 module docstring for why that helper
|
||
// re-introduces the case-3 bug the canonical safeSplitIndex was written
|
||
// to fix). Routes head + tail slicing through safeSplitIndex so an emoji
|
||
// at offset 4000 (or length-4000) never produces a lone surrogate that
|
||
// Anthropic's JSON parser rejects ("no low surrogate in string", caught
|
||
// 2026-05-24 on telegram).
|
||
//
|
||
// Contract: this branch only runs when content.length > 8000, so
|
||
// length - 4000 > 4000 > 0 — safeSplitIndex never sees an out-of-range
|
||
// maxChars here. (Codex C-10 documented contract.)
|
||
let trimmed: string;
|
||
if (t.content.length > 8000) {
|
||
const headEnd = safeSplitIndex(t.content, 4000);
|
||
const tailStart = safeSplitIndex(t.content, t.content.length - 4000);
|
||
trimmed = t.content.slice(0, headEnd) + '\n[...truncated...]\n' + t.content.slice(tailStart);
|
||
} else {
|
||
trimmed = t.content;
|
||
}
|
||
|
||
const sys = `You judge whether a conversation transcript is worth synthesizing into a personal knowledge brain.
|
||
|
||
WORTH PROCESSING (return worth_processing=true):
|
||
- The user articulates a new idea, frame, mental model, or thesis
|
||
- The user reflects on themselves, names patterns, processes emotion
|
||
- The user discusses specific people, companies, or decisions in depth
|
||
- The user makes a strategic call worth remembering
|
||
|
||
NOT WORTH PROCESSING (return worth_processing=false):
|
||
- Routine ops ("check my email", "schedule X")
|
||
- Pure code debugging without user reflection
|
||
- Short message exchanges with no original thought
|
||
- Repetitive content the brain already has
|
||
|
||
Respond as JSON: {"worth_processing": <bool>, "reasons": ["<short>", "<short>"]}.
|
||
Two reasons max, one phrase each.`;
|
||
|
||
const msg = await client.create({
|
||
model: verdictModel,
|
||
max_tokens: 200,
|
||
system: sys,
|
||
messages: [{ role: 'user', content: `Transcript ${t.basename}:\n\n${trimmed}` }],
|
||
});
|
||
|
||
for (const block of msg.content) {
|
||
if (block.type === 'text') {
|
||
const text = block.text.trim();
|
||
const m = /\{[\s\S]*\}/.exec(text);
|
||
if (!m) continue;
|
||
try {
|
||
const parsed = JSON.parse(m[0]) as { worth_processing?: unknown; reasons?: unknown };
|
||
const worth = parsed.worth_processing === true;
|
||
const reasons = Array.isArray(parsed.reasons)
|
||
? parsed.reasons.filter((r): r is string => typeof r === 'string').slice(0, 4)
|
||
: [];
|
||
return { worth_processing: worth, reasons };
|
||
} catch { /* fall through */ }
|
||
}
|
||
}
|
||
// Couldn't parse — default to NOT processing (cheap fallback).
|
||
return { worth_processing: false, reasons: ['judge response unparseable'] };
|
||
}
|
||
|
||
// ── Subagent prompt ──────────────────────────────────────────────────
|
||
|
||
/**
|
||
* Build the prompt for one subagent. When `chunkTotal > 1`, the slug seed
|
||
* gains a `-c<idx>` suffix and the prompt names which chunk this is.
|
||
*
|
||
* D6 enforcement is orchestrator-side (rewriteChunkedSlug runs at slug-
|
||
* collection time). Sonnet still gets the chunked seed via the prompt's
|
||
* `USE THIS in slugs` rule for the happy path.
|
||
*/
|
||
/**
|
||
* v0.32.6 M2 — Load prior probe findings into an informational block.
|
||
* Returns '' if no probe runs exist or the engine doesn't know how (pre-v33
|
||
* brain that hasn't applied migrations). Best-effort and silent on failure.
|
||
*/
|
||
async function loadPriorContradictionsBlock(engine: BrainEngine): Promise<string> {
|
||
try {
|
||
const rows = await engine.loadContradictionsTrend(30);
|
||
if (!rows || rows.length === 0) return '';
|
||
const latest = rows[0];
|
||
const report = latest.report_json as Record<string, unknown> | null;
|
||
const perQuery = (report?.per_query as Array<{
|
||
contradictions: Array<{
|
||
severity: 'low' | 'medium' | 'high';
|
||
axis: string;
|
||
a: { slug: string };
|
||
b: { slug: string };
|
||
}>;
|
||
}> | undefined) ?? [];
|
||
const findings: Array<{ severity: string; axis: string; a: string; b: string }> = [];
|
||
for (const q of perQuery) {
|
||
for (const c of q.contradictions) {
|
||
findings.push({ severity: c.severity, axis: c.axis, a: c.a.slug, b: c.b.slug });
|
||
}
|
||
}
|
||
if (findings.length === 0) return '';
|
||
// Sort by severity DESC (high first); take top 5 to keep prompt bounded.
|
||
const rank: Record<string, number> = { high: 3, medium: 2, low: 1 };
|
||
findings.sort((x, y) => (rank[y.severity] ?? 0) - (rank[x.severity] ?? 0));
|
||
const top = findings.slice(0, 5);
|
||
const lines = top.map((f) => ` - [${f.severity}] ${f.a} vs ${f.b}${f.axis ? ' — ' + f.axis : ''}`);
|
||
return [
|
||
'',
|
||
'PRIOR DETECTED CONTRADICTIONS (latest probe run, severity DESC, top 5):',
|
||
...lines,
|
||
'',
|
||
'If your synthesis writes to any of these slugs, reconcile the contradiction',
|
||
'in the compiled_truth instead of recreating it. Either update to the newer/',
|
||
'correct value, mark the older claim as historical, or note the conflict',
|
||
'explicitly. Ignore findings irrelevant to what this transcript covers.',
|
||
].join('\n');
|
||
} catch {
|
||
return '';
|
||
}
|
||
}
|
||
|
||
function buildSynthesisPrompt(
|
||
t: DiscoveredTranscript,
|
||
chunkText: string,
|
||
chunkIdx: number,
|
||
chunkTotal: number,
|
||
priorContradictionsBlock = '',
|
||
): string {
|
||
const dateHint = t.inferredDate ?? today();
|
||
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
|
||
const isChunked = chunkTotal > 1;
|
||
const hashSuffix = isChunked
|
||
? `${t.contentHash.slice(0, 6)}-c${chunkIdx}`
|
||
: t.contentHash.slice(0, 6);
|
||
const chunkBanner = isChunked
|
||
? `\n- This is CHUNK ${chunkIdx + 1} of ${chunkTotal} from the same transcript. Different chunks process different sections; do not assume continuity with other chunks.`
|
||
: '';
|
||
const transcriptHeader = isChunked
|
||
? `${t.filePath} (chunk ${chunkIdx + 1}/${chunkTotal})`
|
||
: t.filePath;
|
||
return `You are synthesizing a conversation transcript into the user's personal knowledge brain.
|
||
|
||
CONTEXT
|
||
- Today's date: ${dateHint}
|
||
- Transcript hash suffix (USE THIS in slugs): ${hashSuffix}
|
||
- Source file basename: ${baseSlugSegment}${chunkBanner}${priorContradictionsBlock}
|
||
|
||
OUTPUT POLICY (ALL of these are required)
|
||
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
|
||
2. Cross-reference compulsively: every new page MUST contain at least one wikilink (e.g., \`[ref](people/jane-doe)\` or \`[[people/jane-doe]]\`) to existing brain content. Use the search tool to find existing pages first.
|
||
3. Do NOT write to any path outside the allow-list shown in the put_page schema.
|
||
4. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated segments. NO underscores, NO file extensions.
|
||
|
||
TASKS
|
||
A. Reflections (self-knowledge, pattern recognition, emotional processing):
|
||
slug: \`wiki/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||
|
||
B. Originals (new ideas, frames, theses, mental models):
|
||
slug: \`wiki/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||
|
||
C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries — your job is the reflection/original synthesis, NOT modifying existing person pages).
|
||
|
||
D. If nothing in this transcript meets the bar (significance filter already passed but the content is still routine), return without writing anything.
|
||
|
||
TRANSCRIPT (${transcriptHeader})
|
||
---
|
||
${chunkText}
|
||
---
|
||
|
||
When done, briefly list the slugs you wrote in your final message so the orchestrator can audit.`;
|
||
}
|
||
|
||
function sanitizeForSlug(s: string): string {
|
||
return s
|
||
.toLowerCase()
|
||
.replace(/[^a-z0-9]+/g, '-')
|
||
.replace(/^-+|-+$/g, '')
|
||
.slice(0, 60);
|
||
}
|
||
|
||
// ── Slug collection from child put_page calls (codex #2 + D6) ────────
|
||
|
||
/**
|
||
* D6 (orchestrator-side deterministic slug rewrite, zero Sonnet trust):
|
||
* two-stage path — raw fetch (no DISTINCT, preserves duplicate evidence) →
|
||
* in-memory chunk-suffix rewrite via `rewriteChunkedSlug` for chunked
|
||
* children → return distinct rewritten set.
|
||
*
|
||
* Closes Codex finding #2 ("collision detection via SELECT DISTINCT was
|
||
* fake"): we no longer need detection because the rewrite enforces
|
||
* uniqueness at slug-write time.
|
||
*
|
||
* `chunkInfo` maps child job_id → { chunk_index, hash6 }. Single-chunk
|
||
* children are absent from the map and pass through unchanged.
|
||
*/
|
||
async function collectChildPutPageSlugs(
|
||
engine: BrainEngine,
|
||
childIds: number[],
|
||
chunkInfo: Map<number, { idx: number; hash6: string }>,
|
||
): Promise<Array<{ slug: string; source_id: string }>> {
|
||
if (childIds.length === 0) return [];
|
||
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
|
||
// the orchestrator sees what each child wrote. COALESCE handles both
|
||
// properly-stored jsonb objects (input->>'slug') and double-encoded jsonb
|
||
// strings from pre-fix data ((input #>> '{}')::jsonb->>'slug').
|
||
//
|
||
// v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent
|
||
// put_page tool schema doesn't expose source_id (subagents are scoped to
|
||
// a single source); default to 'default' for the current dream-cycle
|
||
// product behavior. Threading the source_id through reverseWriteRefs
|
||
// guarantees getPage targets the correct (source, slug) row instead of
|
||
// the first DB match.
|
||
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
|
||
`SELECT job_id,
|
||
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
|
||
FROM subagent_tool_executions
|
||
WHERE job_id = ANY($1::int[])
|
||
AND tool_name = 'brain_put_page'
|
||
AND status = 'complete'`,
|
||
[childIds],
|
||
);
|
||
const rewritten = new Set<string>();
|
||
for (const r of rows) {
|
||
if (typeof r.slug !== 'string' || r.slug.length === 0) continue;
|
||
const ci = chunkInfo.get(r.job_id);
|
||
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
|
||
}
|
||
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' }));
|
||
}
|
||
|
||
/**
|
||
* D8: query for any `completed` legacy single-chunk job at the canonical
|
||
* idempotency key shape `dream:synth:<filePath>:<hash16>`. Used at fan-out
|
||
* time to detect transcripts that were synthesized under the pre-chunking
|
||
* code path; those should NOT be re-submitted under chunked keys.
|
||
*
|
||
* Reuses the existing `minion_jobs.idempotency_key` index — no schema
|
||
* additions. One indexed lookup per worth-processing transcript.
|
||
*/
|
||
async function hasLegacySingleChunkCompletion(
|
||
engine: BrainEngine,
|
||
filePath: string,
|
||
hash16: string,
|
||
): Promise<boolean> {
|
||
const legacyKey = `dream:synth:${filePath}:${hash16}`;
|
||
const rows = await engine.executeRaw<{ status: string }>(
|
||
`SELECT status
|
||
FROM minion_jobs
|
||
WHERE idempotency_key = $1
|
||
AND status = 'completed'
|
||
LIMIT 1`,
|
||
[legacyKey],
|
||
);
|
||
return rows.length > 0;
|
||
}
|
||
|
||
// ── Reverse-write DB rows → markdown files ───────────────────────────
|
||
|
||
async function reverseWriteRefs(
|
||
engine: BrainEngine,
|
||
brainDir: string,
|
||
refs: Array<{ slug: string; source_id: string }>,
|
||
): Promise<number> {
|
||
let count = 0;
|
||
for (const { slug, source_id } of refs) {
|
||
// v0.32.8 F6: validate source_id is filesystem-safe before any join().
|
||
validateSourceId(source_id);
|
||
const page = await engine.getPage(slug, { sourceId: source_id });
|
||
if (!page) continue;
|
||
const tags = await engine.getTags(slug, { sourceId: source_id });
|
||
try {
|
||
const md = renderPageToMarkdown(page, tags);
|
||
// v0.32.8 F6: non-default sources land at brainDir/.sources/<id>/<slug>.md
|
||
// so same-slug-different-source pages don't collide. Default-source
|
||
// pages stay at brainDir/<slug>.md so single-source brains see no change.
|
||
const filePath = source_id === 'default'
|
||
? join(brainDir, `${slug}.md`)
|
||
: join(brainDir, '.sources', source_id, `${slug}.md`);
|
||
mkdirSync(dirname(filePath), { recursive: true });
|
||
writeFileSync(filePath, md, 'utf8');
|
||
count++;
|
||
} catch (e) {
|
||
// Per-slug failures are non-fatal — phase continues.
|
||
const msg = e instanceof Error ? e.message : String(e);
|
||
process.stderr.write(`[dream] reverse-write ${slug}@${source_id} failed: ${msg}\n`);
|
||
}
|
||
}
|
||
return count;
|
||
}
|
||
|
||
/**
|
||
* Render a Page to markdown, stamping the dream-output identity marker into
|
||
* frontmatter. This stamp is the explicit identity surface checked by
|
||
* `isDreamOutput` in transcript-discovery.ts. Stamping at render time covers
|
||
* every reverse-write path (subagent reflections + originals + summary) with
|
||
* one funnel; the prior content-pattern guard could miss real output because
|
||
* `serializeMarkdown` does not embed the page slug in the body.
|
||
*/
|
||
export function renderPageToMarkdown(page: Page, tags: string[]): string {
|
||
// v0.38 DRY: the dream-output identity stamp (dream_generated +
|
||
// dream_cycle_date) is the ONLY thing that differs from the v0.38
|
||
// put_page write-through renderer. Both call the shared
|
||
// serializePageToMarkdown helper in markdown.ts; this wrapper passes
|
||
// the dream-specific overrides. Future markdown-shape changes happen
|
||
// in one place.
|
||
return serializePageToMarkdown(page, tags, {
|
||
frontmatterOverrides: {
|
||
dream_generated: true,
|
||
dream_cycle_date: today(),
|
||
},
|
||
});
|
||
}
|
||
|
||
// ── Summary index page ───────────────────────────────────────────────
|
||
|
||
async function writeSummaryPage(
|
||
engine: BrainEngine,
|
||
brainDir: string,
|
||
summarySlug: string,
|
||
summaryDate: string,
|
||
writtenSlugs: string[],
|
||
childOutcomes: Array<{ jobId: number; status: string }>,
|
||
): Promise<void> {
|
||
const completed = childOutcomes.filter(c => c.status === 'completed').length;
|
||
const failed = childOutcomes.length - completed;
|
||
|
||
const lines: string[] = [];
|
||
lines.push(`# Dream cycle ${summaryDate}`);
|
||
lines.push('');
|
||
lines.push(`**Children:** ${completed} completed, ${failed} failed/timeout.`);
|
||
lines.push(`**Pages written:** ${writtenSlugs.length}.`);
|
||
lines.push('');
|
||
if (writtenSlugs.length > 0) {
|
||
lines.push('## Pages');
|
||
lines.push('');
|
||
for (const s of writtenSlugs) {
|
||
lines.push(`- [[${s}]]`);
|
||
}
|
||
lines.push('');
|
||
}
|
||
|
||
const body = lines.join('\n');
|
||
// Stamp the dream-output identity marker into the summary's frontmatter.
|
||
// parseMarkdown below round-trips it into the DB-stored frontmatter, so the
|
||
// marker survives any later reverse-render of the summary page.
|
||
const fullMarkdown = serializeMarkdown(
|
||
{ dream_generated: true, dream_cycle_date: summaryDate } as Record<string, unknown>,
|
||
body,
|
||
'',
|
||
{ type: 'note' as string, title: `Dream cycle ${summaryDate}`, tags: ['dream-cycle'] },
|
||
);
|
||
|
||
// Direct engine.putPage — orchestrator write, no subagent context, no
|
||
// allow-list check (server-side viaSubagent=false). The summary slug is
|
||
// pre-validated against SUMMARY_SLUG_RE in the caller.
|
||
// Importing put_page via operations.ts would re-run namespace logic
|
||
// unnecessarily; we go straight to the engine.
|
||
const { parseMarkdown } = await import('../markdown.ts');
|
||
const parsed = parseMarkdown(fullMarkdown);
|
||
await engine.putPage(summarySlug, {
|
||
type: parsed.type,
|
||
title: parsed.title,
|
||
compiled_truth: parsed.compiled_truth,
|
||
timeline: parsed.timeline,
|
||
frontmatter: parsed.frontmatter,
|
||
});
|
||
|
||
// Also write to disk (orchestrator dual-write).
|
||
try {
|
||
const filePath = join(brainDir, `${summarySlug}.md`);
|
||
mkdirSync(dirname(filePath), { recursive: true });
|
||
writeFileSync(filePath, fullMarkdown, 'utf8');
|
||
} catch (e) {
|
||
const msg = e instanceof Error ? e.message : String(e);
|
||
process.stderr.write(`[dream] summary file-write failed: ${msg}\n`);
|
||
}
|
||
}
|
||
|
||
// ── Helpers ──────────────────────────────────────────────────────────
|
||
|
||
function loadAdHocTranscript(
|
||
filePath: string,
|
||
minChars: number,
|
||
excludePatterns: string[],
|
||
bypassGuard?: boolean,
|
||
): DiscoveredTranscript[] {
|
||
const { readSingleTranscript } = require('./transcript-discovery.ts') as typeof import('./transcript-discovery.ts');
|
||
const t = readSingleTranscript(filePath, { minChars, excludePatterns, bypassGuard });
|
||
return t ? [t] : [];
|
||
}
|
||
|
||
function today(): string {
|
||
return new Date().toISOString().slice(0, 10);
|
||
}
|
||
|
||
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
|
||
return { phase: 'synthesize', status: 'ok', duration_ms: 0, summary, details };
|
||
}
|
||
|
||
function skipped(reason: string, summary: string): PhaseResult {
|
||
return {
|
||
phase: 'synthesize',
|
||
status: 'skipped',
|
||
duration_ms: 0,
|
||
summary,
|
||
details: { reason },
|
||
};
|
||
}
|
||
|
||
function failed(error: PhaseError): PhaseResult {
|
||
return {
|
||
phase: 'synthesize',
|
||
status: 'fail',
|
||
duration_ms: 0,
|
||
summary: 'synthesize phase failed',
|
||
details: {},
|
||
error,
|
||
};
|
||
}
|
||
|
||
function makeError(cls: string, code: string, message: string, hint?: string): PhaseError {
|
||
return hint ? { class: cls, code, message, hint } : { class: cls, code, message };
|
||
}
|
||
|
||
// ── Test-only export ───────────────────────────────────────
|
||
// `__testing` re-exports otherwise-private helpers so unit tests can pin
|
||
// behavior at function granularity (e.g., #745 collectChildPutPageSlugs
|
||
// double-encoded jsonb regression). Not part of the runtime contract.
|
||
export const __testing = {
|
||
collectChildPutPageSlugs,
|
||
};
|