diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index f3ebec6ca..a17905db5 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -151,7 +151,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `test/helpers/cli-pty-runner.ts` — generic real-PTY harness (~470 lines) using pure `Bun.spawn({terminal:})` (Bun 1.3.10+; engines.bun pin in package.json). Generic primitives only, no plan-mode orchestrators. Exports `launchPty`, `resolveBinary`, `stripAnsi`, `parseNumberedOptions`, `optionsSignature`, `isNumberedOptionListVisible`, `isTrustDialogVisible`. Self-tests in `test/cli-pty-runner.test.ts` (24 cases). - `src/core/skillpack/{init-brain-pack,brain-pack-advisory,brain-pack-lint,brain-resident-locate,nag-state}.ts` (#2180) — brain-resident skillpacks. `manifest-v1.ts` gains optional `brain_resident` + `schema_pack` (additive). `runInitBrainPack` scaffolds a pack beside brain content (`brain_resident:true`, exact `gbrain_min_version`, 5-section machine-parseable README); `applyWritePlan` is factored out of `init-scaffold.ts` for the shared refuse-overwrite loop. `brain-pack-lint.lintBrainPackTools` validates each skill's `tools:` against the serving op set (E6 version-skew). Topology A: `src/commands/sources.ts` `runAdd` prints `brain-pack-advisory` to stderr after `opsAddSource`, fail-open; `nag-state.ts` (`~/.gbrain/skillpack-nag-state.json`) keys declines by `(source-repo brain_id, source_id, pack_name)` with pure `decideNagAction` (first/reminder/version-bump/ceiling) — declines count ONLY on CLI-interactive displays. Topology B: `brain-resident-locate.loadResidentPacksForServer` (source-scoped via `sourceScopeOpts`) backs the `list_brain_skillpack` op; `getResidentSkillDetail` backs `get_skill` `source_id`; `scaffold_spec` is the git source, never a server FS path. Tests: `test/skillpack-{init-brain-pack,nag-state,brain-resident-locate}.test.ts` + the brain-resident cases in `test/skillpack-manifest-v1.test.ts`. - `src/core/advisor/{types,run,render,recommended-set,history,apply,collect-*}.ts` + `src/commands/advisor.ts` (#2180) — `gbrain advisor`: read-only ranked actions from brain state. `run.runAdvisor` executes 8 hardcoded collectors (version [cache-only], migration, schema-pack, stalled-jobs [absent-table tolerant], usage-shape, setup-smells, uninstalled-brain-pack, uninstalled-bundled), each in its own try/catch; `rankFindings` orders critical>warn>info then collector order, caps the info tail, and drops `workspace_dependent` findings when `remote` (A1). `render.ts` is the shared `=`-bar renderer used by the advisor AND `post-install-advisory.ts` (generalized to a single current-state `recommended-set.RECOMMENDED`, `install`→`scaffold`). `history.ts` appends bounded `~/.gbrain/advisor-history.jsonl` (no DB migration) for since-last-run deltas; local-only. `apply.resolveApplyTarget` is the allowlist+injection guard for `commands/advisor.ts --apply ` (structured argv, never a shell; local-only). The `advisor` op (`operations.ts`) is read-scoped, NOT localOnly, gated by `mcp.publish_advisor` (config.ts; default off) and strictly read-only on remote. CLI wired in `cli.ts` (`CLI_ONLY` + dispatch). Bundled skill `skills/gbrain-advisor/` + weekly cron recipe. Tests: `test/advisor-{core,apply,op-gate,ranking-eval}.test.ts`. -- `src/core/chronicle/{eligibility,config,backstop,extract-events,ontology,narrative}.ts` + `src/eval/chronicle/harness.ts` + `src/commands/eval-chronicle.ts` (#2390) — Life Chronicle: the temporal spine. `eligibility.isChronicleEligible` decides which pages auto-emit events (meeting/conversation/calendar-event + directory rescue; diary and event pages NEVER eligible — privacy + anti-loop). `backstop.runChronicleBackstop` is the put_page hook body (fires ONLY on `status==='imported'` + the auto-link trust gate + the default-OFF `auto_chronicle` flag; enqueues a `chronicle_extract` minion job — LLM never runs on the write path). `extract-events.runChronicleExtract` is the job body: deterministic when/who, injectable judge (default = chat gateway), an ALL-or-nothing parse barrier (`isValidProposal` requires a real parseable date — a malformed batch writes NOTHING), then content-addressed `life/events/` pages + a `timeline_entries` projection via `engine.upsertEventProjection` (dedup `(event_page_id, date)`; idempotent re-runs). `ontology.ts` carries the deterministic pieces of the bi-temporal per-entity ontology that RIDES THE `facts` TABLE (migration v122 adds `dimension`/`value`/`value_hash`/`dim_status`): `valueHash` (normalized, timestamp-free → crash-retry idempotent), `normalizeDimension` (seed alias lexicon), `isNovelDimension` (novel → quarantined, excluded from resolution/context until confirmed). The engine methods (`mergeOntologyFact` — corroborate on same value, forward-supersede via `valid_until`+`superseded_by` on a new value, backdated conflicts kept + flagged; `getOntology` with `--asof` valid-time travel; `discoverOntologyDimensions`; `findOntologyConflicts` — currently-open rows only) live in BOTH engines; both engines are on the R8 `valid_until` write allow-list (engine-layer, `dimension IS NOT NULL` rows only). Chronicle reads (`getTimelineForDate`/`getSince`/`getLastSeen`/`getOnThisDay`) JOIN the depth page (`deleted_at IS NULL`), hide soft-deleted event projections at READ time, and order by event `effective_date` for intra-day sequence. Ops: `chronicle_day`/`chronicle_since`/`chronicle_last_seen`/`chronicle_on_this_day`/`ontology_*`/`volunteer_chronicle` (agent orientation via `src/core/context/chronicle-context.ts`)/`chronicle_backfill` (admin, localOnly). Diary privacy: diary-sourced ontology + conflict values redacted for `ctx.remote !== false` callers. Search: `applyChronicleTypeBoost` in `search/hybrid.ts` (bounded [1.0,1.25], fires only inside the `recency !== 'off'` post-fusion branch → non-temporal search bit-for-bit unchanged). Advisor collector `collect-chronicle.ts` (conflicts + coverage gap); doctor `chronicle_projection_health` (BRAIN category). Eval: `gbrain eval chronicle` — deterministic, own in-memory PGLite, 6 gold tasks (day order, last-seen, supersession, asof, conflict, isolation), exit 0 iff 6/6. Tests: `test/chronicle-*.test.ts`, `test/eval-chronicle.test.ts`. +- `src/core/chronicle/{eligibility,config,backstop,extract-events,ontology,narrative}.ts` + `src/eval/chronicle/harness.ts` + `src/commands/eval-chronicle.ts` (#2390) — Life Chronicle: the temporal spine. `eligibility.isChronicleEligible` decides which pages auto-emit events (meeting/conversation/calendar-event + directory rescue; diary and event pages NEVER eligible — privacy + anti-loop). `backstop.runChronicleBackstop` is the put_page hook body (fires ONLY on `status==='imported'` + the auto-link trust gate + the default-OFF `auto_chronicle` flag; enqueues a `chronicle_extract` minion job — LLM never runs on the write path). `extract-events.runChronicleExtract` is the job body: deterministic when/who, injectable judge (default = chat gateway; output cap 4000 tokens by default, operator override `chronicle.judge_max_tokens`), an ALL-or-nothing parse barrier (`isValidProposal` requires a real parseable date — a malformed batch writes NOTHING), then content-addressed `life/events/` pages + a `timeline_entries` projection via `engine.upsertEventProjection` (dedup `(event_page_id, date)`; idempotent re-runs). An unusable judge response is never recorded as `no_events` (#2606): a `stopReason: 'length'` truncation or a no-JSON-array response (`parseJudgeJson` returns `null` on parse failure; `[]` only for a legitimate empty array) surfaces as `status: 'skipped'` with reason `judge_truncated` / `judge_parse_failed`. `ontology.ts` carries the deterministic pieces of the bi-temporal per-entity ontology that RIDES THE `facts` TABLE (migration v122 adds `dimension`/`value`/`value_hash`/`dim_status`): `valueHash` (normalized, timestamp-free → crash-retry idempotent), `normalizeDimension` (seed alias lexicon), `isNovelDimension` (novel → quarantined, excluded from resolution/context until confirmed). The engine methods (`mergeOntologyFact` — corroborate on same value, forward-supersede via `valid_until`+`superseded_by` on a new value, backdated conflicts kept + flagged; `getOntology` with `--asof` valid-time travel; `discoverOntologyDimensions`; `findOntologyConflicts` — currently-open rows only) live in BOTH engines; both engines are on the R8 `valid_until` write allow-list (engine-layer, `dimension IS NOT NULL` rows only). Chronicle reads (`getTimelineForDate`/`getSince`/`getLastSeen`/`getOnThisDay`) JOIN the depth page (`deleted_at IS NULL`), hide soft-deleted event projections at READ time, and order by event `effective_date` for intra-day sequence. Ops: `chronicle_day`/`chronicle_since`/`chronicle_last_seen`/`chronicle_on_this_day`/`ontology_*`/`volunteer_chronicle` (agent orientation via `src/core/context/chronicle-context.ts`)/`chronicle_backfill` (admin, localOnly). Diary privacy: diary-sourced ontology + conflict values redacted for `ctx.remote !== false` callers. Search: `applyChronicleTypeBoost` in `search/hybrid.ts` (bounded [1.0,1.25], fires only inside the `recency !== 'off'` post-fusion branch → non-temporal search bit-for-bit unchanged). Advisor collector `collect-chronicle.ts` (conflicts + coverage gap); doctor `chronicle_projection_health` (BRAIN category). Eval: `gbrain eval chronicle` — deterministic, own in-memory PGLite, 6 gold tasks (day order, last-seen, supersession, asof, conflict, isolation), exit 0 iff 6/6. Tests: `test/chronicle-*.test.ts`, `test/eval-chronicle.test.ts`. - `src/core/skill-manifest.ts` — parser for `skill-manifest.json` records. Used by skillpack installer to detect drift between the shipped bundle and the user's local edits, so updates merge instead of overwriting. - `src/commands/routing-eval.ts` + `src/core/routing-eval.ts` — `gbrain routing-eval` catches user phrasings that route to the wrong skill. Reads `skills//routing-eval.jsonl` fixtures (`{intent, expected_skill, ambiguous_with?}`). Structural layer runs in `check-resolvable` by default (zero API cost). `--llm` is a placeholder for a future LLM tie-break layer; today it emits a stderr notice and runs structural only. Uses `autoDetectSkillsDirReadOnly` and the same multi-file resolver merge as `check-resolvable`, so on OpenClaw layouts (`skills/RESOLVER.md` + `../AGENTS.md`) all three commands see the same trigger index. RESOLVER.md rows carry the full frontmatter `triggers:` arrays so the structural matcher sees realistic phrasings; ambiguous-fixture annotations cover deliberate skill chains like `enrich → article-enrichment`. - `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json` — Check 6 of `check-resolvable`. Parses `writes_pages:` / `writes_to:` frontmatter on skills and audits their filing claims against the filing-rules JSON (error severity). Internal `parseFrontmatter` is a thin wrapper over the shared `src/core/skill-frontmatter.ts` parser so both filing-audit and skill-brain-first read the same shape (`tools?`, `triggers?`, `brain_first?: 'exempt'`, typed `brain_first_typo`) from one source of truth. @@ -301,15 +301,15 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/extract-conversation-facts.ts` extension — `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts::`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). `deleteOrphanFactsForPage(engine, sourceId, slug)` provides delete-orphans-first replay safety — wipes facts from a prior crashed run for this (sourceId, slug) before re-extracting, closing the "terminal audit row written after partial insertFacts failure" class. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Checkpoint state is a shared `cpMap: Map` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-workers.test.ts` + the existing extract-conversation-facts behavioral tests. - `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. -- `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. +- `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. `synthesize_concepts` writes concept pages through `importFromContent` (#2163: the same parse→chunk→embed pipeline put_page uses, with put_page's `isAvailable('embedding')` → `noEmbed` gate) so `concepts/` pages carry `content_chunks` + embeddings and are reachable by retrieval (where `source-boost.ts` weights them 1.3×). - `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. Below the valve, stale pages are partitioned by git history via exported `listEverCommittedPaths(repoPath)` (one `git log --all --no-renames --diff-filter=A --name-only` pass; null on non-git dirs → unchanged behavior): a stale path that EVER existed in history was genuinely deleted → reconciled; a path with NO history is DB-only write-through (never committed/pushed, e.g. lost to a fresh clone) → the page is KEPT and its markdown re-exported to the working tree via `writePageThrough`, with a stderr hint to commit it (#2426 — "absent from git" is the symptom of the missing write-through commit, not evidence the content is disposable). Pinned by `test/sync-reconcile-db-only.serial.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). `collectSyncableFiles`' shared emit filter `isCollectibleForWalker` applies the SAME segment-level `pruneDir` gate as incremental sync's `classifySync` — load-bearing for the `git ls-files` fast path, which enumerates tracked files under dot-dirs/vendored trees that the FS walk never descends into; without it `sync --full` imported (and resurrected soft-deleted) pages incremental sync excludes (#2607). Pinned by `test/import-git-fastpath-prune.test.ts`. - `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). -- `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. -- `src/core/cycle/synthesize.ts` — Synthesize phase: conversation-transcript-to-brain pipeline. Reads `dream.synthesize.session_corpus_dir`, runs a cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at`) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. `--dry-run` runs Haiku, skips Sonnet. Subagent never gets fs-write access. `renderPageToMarkdown` (exported) stamps `dream_generated: true` + `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the summary index — this marker is the explicit identity surface `isDreamOutput` checks in `transcript-discovery.ts`. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` takes a `verdictModel` param loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically; per-chunk char budget = `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token` (non-Anthropic ids fall back to a 180K-token safe default + once-per-process stderr warn); operator overrides `dream.synthesize.max_prompt_tokens` (floor 100K, wins) and `dream.synthesize.max_chunks_per_transcript` (default 24); per-chunk subagent job/wait timeouts are `dream.synthesize.subagent_timeout_ms` / `dream.synthesize.subagent_wait_timeout_ms` (defaults 30/35 min). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts` so raising the cap on next run re-attempts cleanly. Bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by terminal-error classification in `subagent.ts`. Verdict routing is gateway-routed: `makeJudgeClient(verdictModel)` (exported, replacing `makeHaikuClient()`) mirrors `tryBuildGatewayClient` in `src/core/think/index.ts` — a construction-time provider/key probe returns `null` on a clear miss (unknown provider id via `resolveRecipe` AIConfigError, or Anthropic provider with no key via `hasAnthropicKey()`). The verdict loop wraps `judgeSignificance` in try/catch for `AIConfigError` so mid-run provider failures surface as per-transcript `worth=false, reasons=['gateway error: ...']` instead of crashing the phase. Canonical config key `models.dream.synthesize_verdict` (per `PER_TASK_KEYS` in `src/core/model-config.ts`); `JudgeClient` signature preserved verbatim for test-seam stability; CI guard `scripts/check-gateway-routed-no-direct-anthropic.sh` prevents reintroducing `new Anthropic()` here or in `think/index.ts`. At the queue.add boundary (lines 395-404) a conditional `anthropic:` prefix is applied ONLY when the resolved model has no colon AND starts with `claude-` (because `resolveModel` returns bare ids from `TIER_DEFAULTS`/`DEFAULT_ALIASES` and the subagent validator requires `provider:model` form) — avoids changing the shared constants which would ripple across every `resolveModel` caller. Pinned by `test/cycle/synthesize-gateway-adapter.test.ts`, `test/e2e/dream-synthesize-pglite.test.ts` (gateway-adapter mid-run AIConfigError catch), `test/cycle/regression-pr-wave-r1-r2-r4.test.ts`. +- `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration — and for `synthesize` (#1586: threaded as `SynthesizePhaseOpts.sourceId` so synthesized pages land in the cycle's resolved source, not `'default'`) — so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. +- `src/core/cycle/synthesize.ts` — Synthesize phase: conversation-transcript-to-brain pipeline. Reads `dream.synthesize.session_corpus_dir`, runs a cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`; when `dream.synthesize.output_root` is set, `loadAllowedSlugPrefixes(outputRoot)` remaps the `wiki/`-rooted globs to the configured namespace — #2415 — and the same root drives the prompt slug templates; default 'wiki', validated against the slug grammar via the exported `loadOutputRoot`). The phase is source-scoped (#1586): cycle.ts threads `cycleSourceId` as `opts.sourceId` → each child's `SubagentHandlerData.source_id` → the subagent tool registry's `OperationContext.sourceId`, so put_page writes, collected refs, the summary page, and reverse-writes all target the cycle's resolved source ('default' when unscoped; reverse-writes for the cycle's own source land at `brainDir/.md`, foreign sources under `brainDir/.sources//`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at`) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. `--dry-run` runs Haiku, skips Sonnet. Subagent never gets fs-write access. `renderPageToMarkdown` (exported) stamps `dream_generated: true` + `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the summary index — this marker is the explicit identity surface `isDreamOutput` checks in `transcript-discovery.ts`. `stampDreamProvenance` (#2569) additionally persists the same marker into the `pages.frontmatter` JSONB row (merge via `executeRawJsonb`, raw object bound to `$N::jsonb`) for every child-written page BEFORE reverse-rendering, so generated pages are DB-queryable and a later put_page write-through (which re-renders from the DB row) can't erase the stamp. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` takes a `verdictModel` param loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically; per-chunk char budget = `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token` (non-Anthropic ids fall back to a 180K-token safe default + once-per-process stderr warn); operator overrides `dream.synthesize.max_prompt_tokens` (floor 100K, wins) and `dream.synthesize.max_chunks_per_transcript` (default 24); per-chunk subagent job/wait timeouts are `dream.synthesize.subagent_timeout_ms` / `dream.synthesize.subagent_wait_timeout_ms` (defaults 30/35 min). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts` so raising the cap on next run re-attempts cleanly. Bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by terminal-error classification in `subagent.ts`. Verdict routing is gateway-routed: `makeJudgeClient(verdictModel)` (exported, replacing `makeHaikuClient()`) mirrors `tryBuildGatewayClient` in `src/core/think/index.ts` — a construction-time provider/key probe returns `null` on a clear miss (unknown provider id via `resolveRecipe` AIConfigError, or Anthropic provider with no key via `hasAnthropicKey()`). The verdict loop wraps `judgeSignificance` in try/catch for `AIConfigError` so mid-run provider failures surface as per-transcript `worth=false, reasons=['gateway error: ...']` instead of crashing the phase. Canonical config key `models.dream.synthesize_verdict` (per `PER_TASK_KEYS` in `src/core/model-config.ts`); `JudgeClient` signature preserved verbatim for test-seam stability; CI guard `scripts/check-gateway-routed-no-direct-anthropic.sh` prevents reintroducing `new Anthropic()` here or in `think/index.ts`. At the queue.add boundary (lines 395-404) a conditional `anthropic:` prefix is applied ONLY when the resolved model has no colon AND starts with `claude-` (because `resolveModel` returns bare ids from `TIER_DEFAULTS`/`DEFAULT_ALIASES` and the subagent validator requires `provider:model` form) — avoids changing the shared constants which would ripple across every `resolveModel` caller. Pinned by `test/cycle/synthesize-gateway-adapter.test.ts`, `test/e2e/dream-synthesize-pglite.test.ts` (gateway-adapter mid-run AIConfigError catch), `test/cycle/regression-pr-wave-r1-r2-r4.test.ts`. - `scripts/check-gateway-routed-no-direct-anthropic.sh` — CI guard that fails the build if `src/core/cycle/synthesize.ts` or `src/core/think/index.ts` reintroduces a runtime `new Anthropic()` constructor call or a value-shaped `import Anthropic from '@anthropic-ai/sdk'` import. Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay allowed for adapter types; comment lines (`//` or ` *` prefixes) are excluded so historical JSDoc doesn't false-fire. Mirrors `scripts/check-jsonb-pattern.sh`. Wired into `bun run verify` and `bun run check:all`. Extend `GUARDED_FILES` when migrating another file off direct SDK construction. -- `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Subagent job/wait timeouts are config keys `dream.patterns.subagent_timeout_ms` / `dream.patterns.subagent_wait_timeout_ms` (defaults 30/35 min, mirroring the `dream.synthesize.*` pair). The phase status reflects the child outcome: non-`complete` outcome with zero writes → `fail` (error code `PATTERNS_CHILD_`); non-`complete` with partial writes → `warn`. Runs AFTER `extract` so the graph is fresh. +- `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize (imports `loadAllowedSlugPrefixes` + `loadOutputRoot` from synthesize.ts — #2415: the reflections lookup, prompt slug templates, and allow-list all honor `dream.synthesize.output_root`, default 'wiki'). Subagent job/wait timeouts are config keys `dream.patterns.subagent_timeout_ms` / `dream.patterns.subagent_wait_timeout_ms` (defaults 30/35 min, mirroring the `dream.synthesize.*` pair). The phase status reflects the child outcome: non-`complete` outcome with zero writes → `fail` (error code `PATTERNS_CHILD_`); non-`complete` with partial writes → `warn`. Runs AFTER `extract` so the graph is fresh. - `src/core/cycle/extract-facts.ts` — extract_facts cycle phase. Fence is canonical: per-page wipe (`deleteFactsForPage`) + reinsert from `parseFactsFence` + `extractFactsFromFenceText` + `engine.insertFacts`. #1928: the per-page wipe passes `excludeSourcePrefixes: ['cli:']` so conversation facts (written by `extract-conversation-facts`, on pages with NO `## Facts` fence to recreate them from) survive the reconcile instead of being deleted-with-nothing-to-reinsert. The destructive phase no longer inherits a failed sync's full-brain walk: `slugs: []` (a real incremental no-op) is distinguished from `slugs: undefined` (full-walk intent) by presence, not length. `runPhaseExtractFacts` (cycle.ts) surfaces a `warn` (`net_fact_deletion`) when the reconcile deletes at least `NET_DELETION_WARN_FLOOR` (50) more facts than it reinserts — the exact signature of the conversation-facts wipe, which previously read as a silent `ok`. Empty-fence guard refuses when legacy rows (`row_num IS NULL AND entity_slug IS NOT NULL`) pend backfill (status: warn, hint: `gbrain apply-migrations --yes`). A phantom-redirect pre-pass runs AFTER the legacy-row guard, BEFORE the main reconcile loop: when `opts.brainDir` is set, `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun)` walks unprefixed-slug pages capped by `GBRAIN_PHANTOM_REDIRECT_LIMIT` (default 50). The pass returns `touched_canonicals` — canonical slugs whose disk fence merged with phantom rows; `runExtractFacts` UNIONs them into the main reconcile slug set so canonical's DB facts derive from the merged fence in the same cycle (handles phantom-had-only-on-disk-fence). `ExtractFactsResult` gains six phantom fields: `phantomsScanned`, `phantomsRedirected`, `phantomsAmbiguous`, `phantomsSkippedDrift`, `phantomsLockBusy`, `phantomsMorePending`. Three bubble to `CycleReport.totals` (`phantoms_redirected`, `phantoms_ambiguous`, `phantoms_skipped_drift`). - `src/core/fence-shared.ts` — shared pipe-table primitives for the `## Takes` (`takes-fence.ts`) and `## Facts` (`facts-fence.ts`) fences: `parseRowCells`, `isSeparatorRow`, `stripStrikethrough`, `parseStringCell`, `escapeFenceCell`. `parseRowCells` is escape-aware: `\|` stays inside its cell and decodes back to a literal `|` (exact inverse of `escapeFenceCell`), while any other backslash passes through verbatim so existing fence text (e.g. Windows paths) is byte-stable across a render → parse cycle. This matters because the fence is canonical and reconcile is wipe-and-reinsert — a pipe inside a cell shifting the column layout would corrupt every row behind it on the next reconcile. Pinned by the escape cases in `test/facts-fence.test.ts` + the full render → parse → reconcile round-trip in `test/e2e/facts-fence-reconcile-postgres.test.ts`. - `src/core/entities/resolve.ts` — Free-form entity name → canonical slug resolution. `resolveEntitySlug(engine, source_id, raw)`: exact slug → unambiguous bare-name prefix expansion across `people/-%` + `companies/-%` → high-specificity fuzzy match for multi-token input (pg_trgm @ 0.7 threshold) → deterministic `slugify` holding fallback. Bare-name collisions never use popularity as confidence; shared-token company names below the threshold remain unresolved. Two helpers for the phantom-redirect pass: `resolvePhantomCanonical(engine, sourceId, phantomSlug)` SKIPS the exact-slug step (a phantom slug `'alice'` would exact-match itself and no-op the redirect); returns the canonical only when non-null AND contains `/`. `findPrefixCandidates(engine, sourceId, token)` is a standalone SQL query returning ALL candidates across `PREFIX_EXPANSION_DIRS` (hardcoded `['people', 'companies']`) via `slug LIKE ANY($N::text[])` over patterns `dir/token` + `dir/token-%`, cap of 10 ordered by `connection_count DESC, slug ASC`. Pinned by `test/entity-resolve.test.ts` (explicit, unique, ambiguous-person, and shared-token-company cases) plus `test/phantom-redirect.test.ts` (resolvePhantomCanonical 3 cases + findPrefixCandidates 6 cases incl. multi-dir ambiguity and the `people/aliceberg`-doesn't-match-`alice` false-positive guard). diff --git a/src/core/chronicle/extract-events.ts b/src/core/chronicle/extract-events.ts index 4354614ea..a1324118d 100644 --- a/src/core/chronicle/extract-events.ts +++ b/src/core/chronicle/extract-events.ts @@ -26,7 +26,17 @@ export interface ChronicleJudgeInput { effectiveDate: string | null; // depth page effective_date (deterministic when) attendees: string[]; // deterministic who from frontmatter } -export interface ChronicleJudgeResult { events: ChronicleEventProposal[] } +export interface ChronicleJudgeResult { + events: ChronicleEventProposal[]; + /** + * #2606 — distinct judge-failure signal so an unusable response is never + * recorded as a legitimate `no_events`: + * - 'truncated': the model hit the output-token cap (stopReason 'length'); + * the JSON array was cut mid-stream and must not be parsed as complete. + * - 'parse_failed': the model returned text but no valid JSON array. + */ + failure?: 'truncated' | 'parse_failed'; +} export type ChronicleJudge = (input: ChronicleJudgeInput) => Promise; export interface ChronicleExtractResult { @@ -126,6 +136,12 @@ export async function runChronicleExtract( return { slug: opts.slug, status: 'skipped', events_written: 0, reason: 'judge_error' }; } + // #2606: a truncated or unparseable judge response is a FAILURE, not an + // empty page. Record it as a distinct skipped reason so operators (and + // retries) can tell it apart from a genuine no_events. + if (result?.failure) { + return { slug: opts.slug, status: 'skipped', events_written: 0, reason: `judge_${result.failure}` }; + } const proposals = Array.isArray(result?.events) ? result.events : []; if (proposals.length === 0) return { slug: opts.slug, status: 'no_events', events_written: 0 }; // PARSE BARRIER — reject the WHOLE batch on any malformed proposal; no partial writes. @@ -167,11 +183,25 @@ const JUDGE_SYSTEM = `You segment a meeting/transcript page into discrete timeli Return ONLY a JSON array. Each element: {"when": ISO datetime or YYYY-MM-DD, "who": [entity slugs/names], "what": one-clause summary, "where": optional string, "kind": one of meeting|call|meal|solo|travel|work|commitment|decision|intro|conflict|milestone|event}. Prefer the page's known date for "when" when the text gives no explicit time. Use the provided attendee slugs for "who" when the text does not name participants. No prose, no markdown — just the JSON array.`; +/** + * #2606: default output-token cap for the judge. Raised from the original + * 1500 (which event-dense pages overflowed, silently truncating the JSON + * array). Override via `chronicle.judge_max_tokens`. + */ +const DEFAULT_JUDGE_MAX_TOKENS = 4000; + function defaultJudge(engine: BrainEngine): ChronicleJudge { return async (input) => { const { isAvailable, chat } = await import('../ai/gateway.ts'); if (!isAvailable('chat')) return { events: [] }; const body = (input.body || '').slice(0, 12_000); + // #2606: configurable cap so event-dense pages have headroom. + let maxTokens = DEFAULT_JUDGE_MAX_TOKENS; + const capRaw = await engine.getConfig('chronicle.judge_max_tokens').catch(() => null); + if (capRaw) { + const n = parseInt(capRaw, 10); + if (Number.isFinite(n) && n > 0) maxTokens = n; + } let text: string; try { const res = await chat({ @@ -183,32 +213,44 @@ function defaultJudge(engine: BrainEngine): ChronicleJudge { `${input.title}\n\n${body}\n\n\n` + `Known attendees: ${input.attendees.slice(0, 10).join(', ') || '(none)'}.\nExtract the events.`, }], - maxTokens: 1500, + maxTokens, }); if (res.stopReason === 'refusal' || res.stopReason === 'content_filter') return { events: [] }; + // #2606: output hit the token cap — the JSON array is cut mid-stream. + // Do NOT feed it to the parser as if complete; surface the truncation. + if (res.stopReason === 'length') return { events: [], failure: 'truncated' }; text = res.text; } catch (err) { if ((err as Error)?.name === 'AbortError') throw err; return { events: [] }; } const parsed = parseJudgeJson(text); + // #2606: non-empty model text with no parseable JSON array is a parse + // failure, distinct from the model legitimately answering `[]`. + if (parsed === null) return { events: [], failure: 'parse_failed' }; return { events: parsed }; }; } -/** Tolerant JSON-array extraction from a model response (mirrors facts parser). */ -export function parseJudgeJson(text: string): ChronicleEventProposal[] { - if (!text) return []; +/** + * Tolerant JSON-array extraction from a model response (mirrors facts parser). + * + * #2606: returns `null` on parse FAILURE (empty text, no `[...]` found, + * JSON.parse throw, non-array result) so callers can distinguish "the model + * said no events" (a legitimate `[]`) from "the response was unusable". + */ +export function parseJudgeJson(text: string): ChronicleEventProposal[] | null { + if (!text) return null; let s = text.trim(); const fence = s.match(/```(?:json)?\s*([\s\S]*?)```/i); if (fence) s = fence[1].trim(); const start = s.indexOf('['); const end = s.lastIndexOf(']'); - if (start === -1 || end === -1 || end < start) return []; + if (start === -1 || end === -1 || end < start) return null; try { const arr = JSON.parse(s.slice(start, end + 1)); - return Array.isArray(arr) ? arr : []; + return Array.isArray(arr) ? arr : null; } catch { - return []; + return null; } } diff --git a/src/core/config.ts b/src/core/config.ts index 89493e040..4ca00cdc9 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -915,6 +915,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ 'dream.synthesize.verdict_model', 'dream.synthesize.max_prompt_tokens', 'dream.synthesize.max_chunks_per_transcript', + // #2415: top-level namespace for synthesize/patterns output (default 'wiki'). + 'dream.synthesize.output_root', 'dream.synthesize.subagent_timeout_ms', 'dream.synthesize.subagent_wait_timeout_ms', 'dream.patterns.lookback_days', @@ -971,6 +973,10 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ // operator had to discover --force by reading source. Same class as the // spend-controls registration above. 'auto_chronicle', + // #2606: chronicle judge output-token cap (default 4000). Event-dense + // pages overflowed the old hardcoded 1500 and were misrecorded as + // no_events; the cap is now configurable and truncation is surfaced. + 'chronicle.judge_max_tokens', // Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate // consent reads this key, and enabling it is the documented path to // `gbrain takes extract --from-pages` — same unregistered-key class. diff --git a/src/core/cycle.ts b/src/core/cycle.ts index d7410868d..a52efcd89 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -1682,6 +1682,9 @@ export async function runCycle( from: opts.synthFrom, to: opts.synthTo, bypassDreamGuard: opts.synthBypassDreamGuard, + // #1586: scope synthesized writes to the cycle's resolved source + // (explicit --source wins, else derived from the checkout dir). + sourceId: cycleSourceId, })); result.duration_ms = duration_ms; phaseResults.push(result); diff --git a/src/core/cycle/patterns.ts b/src/core/cycle/patterns.ts index 4a39779e1..d96584dad 100644 --- a/src/core/cycle/patterns.ts +++ b/src/core/cycle/patterns.ts @@ -19,7 +19,7 @@ */ import { join, dirname } from 'node:path'; -import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs'; +import { mkdirSync, writeFileSync } from 'node:fs'; import type { BrainEngine } from '../engine.ts'; import type { PhaseResult, PhaseError } from '../cycle.ts'; import { MinionQueue } from '../minions/queue.ts'; @@ -27,6 +27,9 @@ import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion. import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts'; import { serializeMarkdown } from '../markdown.ts'; import type { Page, PageType } from '../types.ts'; +// #2415: allow-list + output-root resolution shared with the synthesize +// phase — both phases must agree on the configured namespace. +import { loadAllowedSlugPrefixes, loadOutputRoot } from './synthesize.ts'; import { probeChatModel } from '../ai/gateway.ts'; import { normalizeModelId } from '../model-id.ts'; @@ -49,7 +52,7 @@ export async function runPhasePatterns( } // Gather reflections within lookback window. - const reflections = await gatherReflections(engine, config.lookbackDays); + const reflections = await gatherReflections(engine, config.lookbackDays, config.outputRoot); if (reflections.length < config.minEvidence) { return skipped( 'insufficient_evidence', @@ -81,7 +84,7 @@ export async function runPhasePatterns( return skipped('no_provider', `pattern detection skipped: ${probe.detail}`); } - const allowedSlugPrefixes = await loadAllowedSlugPrefixes(); + const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot); if (allowedSlugPrefixes.length === 0) { return failed(makeError('InternalError', 'NO_ALLOWLIST', 'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs')); @@ -89,7 +92,7 @@ export async function runPhasePatterns( const queue = new MinionQueue(engine); const data: SubagentHandlerData = { - prompt: buildPatternsPrompt(reflections, config.minEvidence), + prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot), model: config.model, max_turns: 30, allowed_slug_prefixes: allowedSlugPrefixes, @@ -182,6 +185,8 @@ interface PatternsConfig { lookbackDays: number; minEvidence: number; model: string; + /** #2415: shared output namespace (dream.synthesize.output_root, default 'wiki'). */ + outputRoot: string; /** #1594-family: subagent job timeout, config `dream.patterns.subagent_timeout_ms`. */ subagentTimeoutMs: number; /** #1594-family: waitForCompletion timeout, config `dream.patterns.subagent_wait_timeout_ms`. */ @@ -216,6 +221,7 @@ async function loadPatternsConfig(engine: BrainEngine): Promise lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30, minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3, model, + outputRoot: await loadOutputRoot(engine), subagentTimeoutMs: await getNumberConfig( engine, 'dream.patterns.subagent_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS, ), @@ -236,16 +242,19 @@ interface ReflectionRef { async function gatherReflections( engine: BrainEngine, lookbackDays: number, + outputRoot = 'wiki', ): Promise { const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString(); + // #2415: reflections live under the configured output root (bound as a + // parameter; outputRoot is slug-grammar-validated by loadOutputRoot). const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>( `SELECT slug, title, compiled_truth FROM pages - WHERE slug LIKE 'wiki/personal/reflections/%' + WHERE slug LIKE $2 AND updated_at >= $1::timestamptz ORDER BY updated_at DESC LIMIT 100`, - [since], + [since, `${outputRoot}/personal/reflections/%`], ); return rows.map(r => ({ slug: r.slug, @@ -256,7 +265,7 @@ async function gatherReflections( // ── Prompt ──────────────────────────────────────────────────────────── -function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string { +function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number, outputRoot = 'wiki'): string { const today = new Date().toISOString().slice(0, 10); const corpus = reflections .map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`) @@ -266,15 +275,15 @@ function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): OUTPUT POLICY - Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections. -- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks). +- Each pattern page MUST cite the reflections that constitute its evidence (use [[${outputRoot}/personal/reflections/...]] wikilinks). - Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one. -- Pattern slug format: \`wiki/personal/patterns/\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date). +- Pattern slug format: \`${outputRoot}/personal/patterns/\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date). - A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics. DO NOT WRITE - A "patterns from today" digest (that's the dream-cycle-summaries page; not your job). - Patterns with <${minEvidence} reflections cited. -- Anything outside wiki/personal/patterns/. +- Anything outside ${outputRoot}/personal/patterns/. CONTEXT - Today: ${today} @@ -365,27 +374,6 @@ function renderPageToMarkdown(page: Page, tags: string[]): string { ); } -// ── Allow-list (shared with synthesize.ts) ─────────────────────────── - -async function loadAllowedSlugPrefixes(): Promise { - const candidates = [ - join(process.cwd(), 'skills', '_brain-filing-rules.json'), - join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'), - ]; - for (const path of candidates) { - if (!existsSync(path)) continue; - try { - const raw = readFileSync(path, 'utf8'); - const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } }; - const globs = parsed?.dream_synthesize_paths?.globs; - if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) { - return globs as string[]; - } - } catch { /* try next */ } - } - return []; -} - // ── Status helpers ─────────────────────────────────────────────────── function ok(summary: string, details: Record = {}): PhaseResult { diff --git a/src/core/cycle/synthesize-concepts.ts b/src/core/cycle/synthesize-concepts.ts index dbc352b7b..b88cc06d5 100644 --- a/src/core/cycle/synthesize-concepts.ts +++ b/src/core/cycle/synthesize-concepts.ts @@ -23,7 +23,13 @@ import type { PhaseResult } from '../cycle.ts'; import type { ProgressReporter } from '../progress.ts'; import { writeReceipt } from '../extract/receipt-writer.ts'; import { upsertExtractRollup } from '../extract/rollup-writer.ts'; -import { chat as gatewayChat } from '../ai/gateway.ts'; +import { chat as gatewayChat, isAvailable } from '../ai/gateway.ts'; +// #2163: concept pages route through importFromContent (the same +// parse→chunk→embed pipeline put_page uses) instead of a bare engine.putPage, +// so they land in the retrieval surface (content_chunks + embeddings) where +// source-boost's 1.3× 'concepts/' weighting can actually reach them. +import { importFromContent } from '../import-file.ts'; +import { serializeMarkdown } from '../markdown.ts'; const DEFAULT_BUDGET_USD = 1.5; const TIER_T1_MIN = 10; @@ -216,19 +222,23 @@ export async function runPhaseSynthesizeConcepts( if (!opts.dryRun) { const title = group.conceptSlug.split('/').pop() ?? group.conceptSlug; - await engine.putPage(`concepts/${title}`, { - title: title.replace(/-/g, ' '), - type: 'concept', - compiled_truth: narrative, - frontmatter: { - type: 'concept', + // #2163: serialize to markdown and import via the canonical pipeline so + // the page is chunked (+ embedded when a provider is configured) — + // mirrors put_page's isAvailable('embedding') → noEmbed gate. + const md = serializeMarkdown( + { tier: group.tier, mention_count: group.atomTitles.length, composite_score: group.atomTitles.length, synthesized_at: new Date().toISOString(), synthesized_by: 'synthesize_concepts-v0.41', }, - timeline: '', + narrative, + '', + { type: 'concept', title: title.replace(/-/g, ' '), tags: [] }, + ); + await importFromContent(engine, `concepts/${title}`, md, { + noEmbed: !isAvailable('embedding'), }); } conceptsWritten++; diff --git a/src/core/cycle/synthesize.ts b/src/core/cycle/synthesize.ts index 94564379e..59aad6630 100644 --- a/src/core/cycle/synthesize.ts +++ b/src/core/cycle/synthesize.ts @@ -244,6 +244,14 @@ export interface SynthesizePhaseOpts { * the synthesize loop. Caller must opt in explicitly. */ bypassDreamGuard?: boolean; + /** + * #1586: the cycle's resolved brain source (cycleSourceId from cycle.ts — + * explicit --source wins, else derived from the checkout dir). Threaded to + * every subagent child as `source_id` so put_page writes land in this + * source, and stamped onto collected refs so reverse-writes read the + * correct (source_id, slug) row. Unset → legacy 'default'. + */ + sourceId?: string; } export async function runPhaseSynthesize( @@ -399,7 +407,7 @@ export async function runPhaseSynthesize( // Fan-out: submit one subagent per worth-processing transcript (or one // per chunk for transcripts that exceed the model's per-prompt budget). - const allowedSlugPrefixes = await loadAllowedSlugPrefixes(); + const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot); if (allowedSlugPrefixes.length === 0) { return failed(makeError('InternalError', 'NO_ALLOWLIST', 'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs')); @@ -462,10 +470,13 @@ export async function runPhaseSynthesize( : config.model; for (let i = 0; i < chunks.length; i++) { const childData: SubagentHandlerData = { - prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock), + prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock, config.outputRoot), model: subagentModel, max_turns: 30, allowed_slug_prefixes: allowedSlugPrefixes, + // #1586: scope every child tool call to the cycle's resolved source + // so put_page writes land there instead of the hardcoded 'default'. + ...(opts.sourceId ? { source_id: opts.sourceId } : {}), }; // Idempotency key parity: // - single-chunk → legacy `dream:synth::` (byte- @@ -524,20 +535,29 @@ export async function runPhaseSynthesize( // bare-hash slugs to `-c` so chunked siblings can't collide // even if Sonnet drops the chunk suffix. // v0.32.8: refs carry source_id so reverseWriteRefs picks the correct - // (source, slug) row (currently always 'default' from subagent put_page). - const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo); + // (source, slug) row. #1586: refs are stamped with the cycle's resolved + // source (children write there via SubagentHandlerData.source_id). + const cycleSourceId = opts.sourceId ?? 'default'; + const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId); + + const summaryDate = opts.date ?? today(); + + // #2569: persist the dream-output identity marker into the DB frontmatter + // of every child-written page BEFORE reverse-rendering, so generated pages + // are queryable (`frontmatter->>'dream_generated'`) and a later put_page + // write-through (which re-renders from the DB row) can't erase the stamp. + await stampDreamProvenance(engine, writtenRefs, summaryDate); // Dual-write: reverse-render each DB row → markdown file. - const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs); + const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs, cycleSourceId); // Summary index page (deterministic; orchestrator-written via direct // engine.putPage so no allow-list path needed). - const summaryDate = opts.date ?? today(); const summarySlug = `dream-cycle-summaries/${summaryDate}`; // Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs. const writtenSlugs = writtenRefs.map(r => r.slug); if (SUMMARY_SLUG_RE.test(summarySlug)) { - await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes); + await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes, cycleSourceId); } // Write completion timestamp ON SUCCESS only. @@ -595,10 +615,29 @@ interface SynthConfig { * `dream.synthesize.max_chunks_per_transcript`. */ maxChunksPerTranscript: number; + /** + * #2415: top-level namespace for synthesized output (reflections, originals, + * patterns). Config key `dream.synthesize.output_root`; default 'wiki' — + * zero behavior change unless set. No trailing slash. Must satisfy the slug + * grammar; invalid values fall back to 'wiki' with a stderr warning. + */ + outputRoot: string; subagentTimeoutMs: number; subagentWaitTimeoutMs: number; } +/** #2415: shared output-root resolution (synthesize + patterns phases). */ +export async function loadOutputRoot(engine: BrainEngine): Promise { + const raw = await engine.getConfig('dream.synthesize.output_root'); + if (!raw) return 'wiki'; + const trimmed = raw.trim().replace(/^\/+|\/+$/g, ''); + if (SUMMARY_SLUG_RE.test(trimmed)) return trimmed; + process.stderr.write( + `[dream] dream.synthesize.output_root "${raw}" is not a valid slug prefix; falling back to "wiki".\n`, + ); + return 'wiki'; +} + async function loadSynthConfig(engine: BrainEngine): Promise { const enabledRaw = await engine.getConfig('dream.synthesize.enabled'); const corpusDir = await engine.getConfig('dream.synthesize.session_corpus_dir'); @@ -672,6 +711,7 @@ async function loadSynthConfig(engine: BrainEngine): Promise { cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12, maxPromptTokens, maxChunksPerTranscript, + outputRoot: await loadOutputRoot(engine), subagentTimeoutMs, subagentWaitTimeoutMs, }; @@ -704,7 +744,13 @@ async function checkCooldown( // ── Allow-list source of truth ─────────────────────────────────────── -async function loadAllowedSlugPrefixes(): Promise { +/** + * #2415: `outputRoot` remaps the canonical `wiki/`-rooted globs to the + * configured namespace (e.g. `notes/personal/reflections/*`). Default 'wiki' + * returns the globs verbatim. Shared by the patterns phase (imported there — + * the two phases must enforce the same allow-list). + */ +export async function loadAllowedSlugPrefixes(outputRoot = 'wiki'): Promise { // Search a few known locations relative to the binary / repo. The first // hit wins; if none found, return []. const candidates = [ @@ -718,7 +764,10 @@ async function loadAllowedSlugPrefixes(): Promise { const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } }; const globs = parsed?.dream_synthesize_paths?.globs; if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) { - return globs as string[]; + if (outputRoot === 'wiki') return globs as string[]; + return (globs as string[]).map(g => + g.startsWith('wiki/') ? `${outputRoot}/${g.slice('wiki/'.length)}` : g, + ); } } catch { /* try next */ } } @@ -966,6 +1015,7 @@ function buildSynthesisPrompt( chunkIdx: number, chunkTotal: number, priorContradictionsBlock = '', + outputRoot = 'wiki', ): string { const dateHint = t.inferredDate ?? today(); const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`; @@ -994,10 +1044,10 @@ OUTPUT POLICY (ALL of these are required) TASKS A. Reflections (self-knowledge, pattern recognition, emotional processing): - slug: \`wiki/personal/reflections/${dateHint}--${hashSuffix}\` + slug: \`${outputRoot}/personal/reflections/${dateHint}--${hashSuffix}\` B. Originals (new ideas, frames, theses, mental models): - slug: \`wiki/originals/ideas/${dateHint}--${hashSuffix}\` + slug: \`${outputRoot}/originals/ideas/${dateHint}--${hashSuffix}\` C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries — your job is the reflection/original synthesis, NOT modifying existing person pages). @@ -1038,6 +1088,7 @@ async function collectChildPutPageSlugs( engine: BrainEngine, childIds: number[], chunkInfo: Map, + sourceId = 'default', ): Promise> { if (childIds.length === 0) return []; // Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so @@ -1047,10 +1098,10 @@ async function collectChildPutPageSlugs( // // v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent // put_page tool schema doesn't expose source_id (subagents are scoped to - // a single source); default to 'default' for the current dream-cycle - // product behavior. Threading the source_id through reverseWriteRefs - // guarantees getPage targets the correct (source, slug) row instead of - // the first DB match. + // a single source). #1586: the orchestrator scopes each child to the + // cycle's resolved source via SubagentHandlerData.source_id, and stamps + // the SAME source here so reverseWriteRefs / provenance reads target the + // correct (source_id, slug) row. Unset → legacy 'default'. const rows = await engine.executeRaw<{ job_id: number; slug: string }>( `SELECT job_id, COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug @@ -1066,7 +1117,7 @@ async function collectChildPutPageSlugs( const ci = chunkInfo.get(r.job_id); rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug); } - return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' })); + return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId })); } /** @@ -1095,12 +1146,52 @@ async function hasLegacySingleChunkCompletion( return rows.length > 0; } +// ── Dream-provenance DB stamp (#2569) ──────────────────────────────── + +/** + * Persist the dream-output identity marker (`dream_generated: true` + + * `dream_cycle_date`) into the `pages.frontmatter` JSONB row for every page + * a synthesize child wrote. Render-time `frontmatterOverrides` alone only + * reach the markdown FILE — the DB row stayed unstamped, so DB consumers + * couldn't enumerate generated pages and a later put_page write-through + * (which re-renders from the DB row) silently erased the marker. + * + * Plain UPDATE through executeRawJsonb (raw object bound to $3::jsonb — + * never JSON.stringify into a ::jsonb cast; engine-parity safe, no new + * engine method). Best-effort per row: a stamp failure never kills the + * phase (the render-time override still covers the file). + */ +async function stampDreamProvenance( + engine: BrainEngine, + refs: Array<{ slug: string; source_id: string }>, + cycleDate: string, +): Promise { + if (refs.length === 0) return; + const { executeRawJsonb } = await import('../sql-query.ts'); + for (const { slug, source_id } of refs) { + try { + await executeRawJsonb( + engine, + `UPDATE pages + SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb + WHERE slug = $1 AND source_id = $2`, + [slug, source_id], + [{ dream_generated: true, dream_cycle_date: cycleDate }], + ); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + process.stderr.write(`[dream] provenance stamp ${slug}@${source_id} failed: ${msg}\n`); + } + } +} + // ── Reverse-write DB rows → markdown files ─────────────────────────── async function reverseWriteRefs( engine: BrainEngine, brainDir: string, refs: Array<{ slug: string; source_id: string }>, + nativeSourceId = 'default', ): Promise { let count = 0; for (const { slug, source_id } of refs) { @@ -1111,10 +1202,11 @@ async function reverseWriteRefs( const tags = await engine.getTags(slug, { sourceId: source_id }); try { const md = renderPageToMarkdown(page, tags); - // v0.32.8 F6: non-default sources land at brainDir/.sources//.md - // so same-slug-different-source pages don't collide. Default-source - // pages stay at brainDir/.md so single-source brains see no change. - const filePath = source_id === 'default' + // v0.32.8 F6: foreign-source pages land at brainDir/.sources//.md + // so same-slug-different-source pages don't collide. Pages belonging to + // the cycle's own source (#1586: brainDir IS that source's checkout — + // legacy 'default' when unscoped) stay at brainDir/.md. + const filePath = source_id === nativeSourceId ? join(brainDir, `${slug}.md`) : join(brainDir, '.sources', source_id, `${slug}.md`); mkdirSync(dirname(filePath), { recursive: true }); @@ -1161,6 +1253,7 @@ async function writeSummaryPage( summaryDate: string, writtenSlugs: string[], childOutcomes: Array<{ jobId: number; status: string }>, + sourceId = 'default', ): Promise { const completed = childOutcomes.filter(c => c.status === 'completed').length; const failed = childOutcomes.length - completed; @@ -1198,13 +1291,15 @@ async function writeSummaryPage( // unnecessarily; we go straight to the engine. const { parseMarkdown } = await import('../markdown.ts'); const parsed = parseMarkdown(fullMarkdown); + // #1586: summary lands in the cycle's resolved source too — otherwise the + // children live in the named source while the index drifts to 'default'. await engine.putPage(summarySlug, { type: parsed.type, title: parsed.title, compiled_truth: parsed.compiled_truth, timeline: parsed.timeline, frontmatter: parsed.frontmatter, - }); + }, { sourceId }); // Also write to disk (orchestrator dual-write). try { @@ -1269,4 +1364,7 @@ function makeError(cls: string, code: string, message: string, hint?: string): P // double-encoded jsonb regression). Not part of the runtime contract. export const __testing = { collectChildPutPageSlugs, + buildSynthesisPrompt, + stampDreamProvenance, + reverseWriteRefs, }; diff --git a/src/core/minions/handlers/subagent.ts b/src/core/minions/handlers/subagent.ts index 53ef86433..3852469ed 100644 --- a/src/core/minions/handlers/subagent.ts +++ b/src/core/minions/handlers/subagent.ts @@ -272,6 +272,8 @@ export function makeSubagentHandler(deps: SubagentDeps) { config, brainId: data.brain_id, allowedSlugPrefixes: data.allowed_slug_prefixes, + // #1586: cycle-resolved source scope for tool-call OperationContexts. + sourceId: data.source_id, }); const toolDefs = data.allowed_tools && data.allowed_tools.length > 0 ? filterAllowedTools(registry, data.allowed_tools) diff --git a/src/core/minions/tools/brain-allowlist.ts b/src/core/minions/tools/brain-allowlist.ts index ffcce65e3..2b1a0f195 100644 --- a/src/core/minions/tools/brain-allowlist.ts +++ b/src/core/minions/tools/brain-allowlist.ts @@ -27,6 +27,7 @@ import type { GBrainConfig } from '../../config.ts'; import { operations } from '../../operations.ts'; import type { Operation, OperationContext } from '../../operations.ts'; import { paramDefToSchema } from '../../../mcp/tool-defs.ts'; +import { validateSourceId } from '../../utils.ts'; import type { ToolCtx, ToolDef } from '../types.ts'; /** @@ -201,6 +202,13 @@ export interface BuildBrainToolsOpts { * SubagentHandlerData.allowed_slug_prefixes via the handler. */ allowedSlugPrefixes?: readonly string[]; + /** + * Brain source every tool-call OperationContext is scoped to (#1586). + * Trusted (flows from SubagentHandlerData.source_id, which only + * PROTECTED_JOB_NAMES-gated submitters can set); validated at build time. + * Unset → legacy 'default'. + */ + sourceId?: string; } interface OpContextDeps { @@ -211,6 +219,7 @@ interface OpContextDeps { signal?: AbortSignal; brainId?: string; allowedSlugPrefixes?: readonly string[]; + sourceId?: string; } function buildOpContext(deps: OpContextDeps): OperationContext { @@ -224,7 +233,8 @@ function buildOpContext(deps: OpContextDeps): OperationContext { }, dryRun: false, remote: true, // match MCP trust boundary for auto-link skip - sourceId: 'default', // v0.34 D4: required; subagent tools default to host source + // #1586: cycle-resolved source when provided; legacy host default else. + sourceId: deps.sourceId ?? 'default', jobId: deps.jobId, subagentId: deps.subagentId, viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace @@ -248,6 +258,11 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] { op => BRAIN_TOOL_ALLOWLIST.has(op.name) && filter.has(op.name), ); + // #1586: fail fast on a malformed source id before any tool executes + // (defense-in-depth — the seam is trusted, but the value round-trips + // through the job payload). + if (opts.sourceId !== undefined) validateSourceId(opts.sourceId); + return picked.map(op => { const schema = op.name === 'put_page' ? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes) @@ -277,6 +292,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] { signal: ctx.signal, brainId: opts.brainId, allowedSlugPrefixes: opts.allowedSlugPrefixes, + sourceId: opts.sourceId, }); const params = (input && typeof input === 'object') ? input as Record : {}; return op.handler(opCtx, params); diff --git a/src/core/minions/types.ts b/src/core/minions/types.ts index a24d5446d..73877bb99 100644 --- a/src/core/minions/types.ts +++ b/src/core/minions/types.ts @@ -455,6 +455,17 @@ export interface SubagentHandlerData { * and direct CLI submitters set it. */ allowed_slug_prefixes?: string[]; + /** + * Brain source the subagent's tool calls are scoped to (#1586). + * + * When set, every tool-call `OperationContext.sourceId` uses this value + * instead of the legacy 'default', so put_page writes land in the cycle's + * resolved source. Same trust story as `allowed_slug_prefixes`: + * PROTECTED_JOB_NAMES gates subagent submission, so only cycle.ts and + * direct CLI submitters can set it. Validated via `validateSourceId` at + * tool-registry build time. + */ + source_id?: string; /** * v0.41 Approach C: opt out of the auto-generated tool-usage preamble * that `buildSystemPrompt()` splices into `system`. Default behavior diff --git a/test/brain-allowlist.serial.test.ts b/test/brain-allowlist.serial.test.ts index 60e74bba9..7cfd5c2a7 100644 --- a/test/brain-allowlist.serial.test.ts +++ b/test/brain-allowlist.serial.test.ts @@ -146,6 +146,41 @@ describe('buildBrainTools', () => { ), ).rejects.toBeInstanceOf(OperationError); }); + + // #1586: sourceId threads through buildBrainTools → buildOpContext → + // put_page → importFromContent, so subagent writes land in the cycle's + // resolved source instead of the hardcoded 'default'. + test('execute() on put_page writes to the configured sourceId (#1586)', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config, archived, created_at) + VALUES ('mybrain', 'My Brain', '/tmp/mybrain', '{}'::jsonb, false, now()) + ON CONFLICT (id) DO NOTHING`, + ); + const tools = buildBrainTools({ + subagentId: 42, + engine, + config, + allowedSlugPrefixes: ['wiki/personal/reflections/*'], + sourceId: 'mybrain', + }); + const putPage = tools.find(t => t.name === 'brain_put_page'); + const ctx: ToolCtx = { engine, jobId: 1, remote: true }; + await putPage!.execute( + { slug: 'wiki/personal/reflections/2026-07-17-scoped', content: '---\ntitle: Scoped\n---\nbody' }, + ctx, + ); + const rows = await engine.executeRaw<{ source_id: string }>( + `SELECT source_id FROM pages WHERE slug = 'wiki/personal/reflections/2026-07-17-scoped'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].source_id).toBe('mybrain'); + }); + + test('buildBrainTools rejects a malformed sourceId at build time (#1586)', () => { + expect(() => + buildBrainTools({ subagentId: 1, engine, config, sourceId: '../evil' }), + ).toThrow(); + }); }); describe('filterAllowedTools', () => { diff --git a/test/chronicle-extract.test.ts b/test/chronicle-extract.test.ts index 404177af4..4152a568d 100644 --- a/test/chronicle-extract.test.ts +++ b/test/chronicle-extract.test.ts @@ -8,7 +8,7 @@ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { isChronicleEligible } from '../src/core/chronicle/eligibility.ts'; -import { runChronicleExtract, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts'; +import { runChronicleExtract, parseJudgeJson, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts'; import { runChronicleBackstop } from '../src/core/chronicle/backstop.ts'; let engine: PGLiteEngine; @@ -111,6 +111,44 @@ describe('runChronicleExtract', () => { const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: none }); expect(r.status).toBe('no_events'); }); + + // #2606: a truncated or unparseable judge response must NOT be recorded as + // a legitimate no_events — it gets a distinct skipped reason. + test('truncated judge output → skipped/judge_truncated, not no_events (#2606)', async () => { + const truncated: ChronicleJudge = async () => ({ events: [], failure: 'truncated' }); + const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: truncated }); + expect(r.status).toBe('skipped'); + expect(r.reason).toBe('judge_truncated'); + expect(await countEvents()).toBe(0); + }); + + test('unparseable judge output → skipped/judge_parse_failed (#2606)', async () => { + const parseFailed: ChronicleJudge = async () => ({ events: [], failure: 'parse_failed' }); + const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: parseFailed }); + expect(r.status).toBe('skipped'); + expect(r.reason).toBe('judge_parse_failed'); + }); +}); + +describe('parseJudgeJson failure signalling (#2606)', () => { + test('a legitimate empty array parses to []', () => { + expect(parseJudgeJson('[]')).toEqual([]); + expect(parseJudgeJson('```json\n[]\n```')).toEqual([]); + }); + + test('a valid array round-trips', () => { + const arr = parseJudgeJson('[{"when":"2026-06-18","who":[],"what":"x","kind":"meeting"}]'); + expect(Array.isArray(arr)).toBe(true); + expect(arr!.length).toBe(1); + }); + + test('empty / no-array / truncated / non-array responses return null', () => { + expect(parseJudgeJson('')).toBeNull(); + expect(parseJudgeJson('I found no events worth extracting.')).toBeNull(); + // Truncated mid-array (the maxTokens-cap shape from the issue). + expect(parseJudgeJson('[{"when":"2026-06-18","who":["a"],"what":"long ev')).toBeNull(); + expect(parseJudgeJson('{"events": 1}')).toBeNull(); + }); }); describe('runChronicleBackstop gating', () => { diff --git a/test/cycle-dream-output-root.test.ts b/test/cycle-dream-output-root.test.ts new file mode 100644 index 000000000..384a7a64d --- /dev/null +++ b/test/cycle-dream-output-root.test.ts @@ -0,0 +1,112 @@ +/** + * #2415 — configurable dream output namespace (`dream.synthesize.output_root`). + * + * The synthesize + patterns phases previously hardcoded `wiki/` in the + * subagent prompt slug templates, the patterns reflection lookup, and the + * trusted-workspace allow-list loaded from skills/_brain-filing-rules.json. + * This suite pins: + * - default 'wiki' → byte-identical prompt + verbatim filing-rule globs + * (zero behavior change unless the key is set); + * - a custom root remaps prompt slug templates and the allow-list globs; + * - loadOutputRoot validates against the slug grammar (bad values fall + * back to 'wiki'); + * - the patterns phase gathers reflections under the configured root. + */ +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { __testing, loadAllowedSlugPrefixes, loadOutputRoot } from '../src/core/cycle/synthesize.ts'; +import { runPhasePatterns } from '../src/core/cycle/patterns.ts'; +import type { DiscoveredTranscript } from '../src/core/cycle/transcript-discovery.ts'; + +const { buildSynthesisPrompt } = __testing; + +const transcript: DiscoveredTranscript = { + filePath: '/tmp/t.txt', + basename: 't', + content: 'User: hello world', + contentHash: 'abcdef0123456789', + inferredDate: '2026-07-17', +} as DiscoveredTranscript; + +describe('#2415: buildSynthesisPrompt output root', () => { + test('defaults to wiki/ slug templates', () => { + const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1); + expect(prompt).toContain('wiki/personal/reflections/2026-07-17-'); + expect(prompt).toContain('wiki/originals/ideas/2026-07-17-'); + }); + + test('custom root replaces wiki/ in both slug templates', () => { + const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1, '', 'notes'); + expect(prompt).toContain('notes/personal/reflections/2026-07-17-'); + expect(prompt).toContain('notes/originals/ideas/2026-07-17-'); + expect(prompt).not.toContain('wiki/personal/reflections/'); + expect(prompt).not.toContain('wiki/originals/ideas/'); + }); +}); + +describe('#2415: loadAllowedSlugPrefixes remap', () => { + // Runs from the repo root, so skills/_brain-filing-rules.json resolves. + test("default 'wiki' returns the filing-rule globs verbatim", async () => { + const globs = await loadAllowedSlugPrefixes(); + expect(globs).toContain('wiki/personal/reflections/*'); + expect(globs).toContain('dream-cycle-summaries/*'); + }); + + test('custom root remaps only wiki/-rooted globs', async () => { + const globs = await loadAllowedSlugPrefixes('notes'); + expect(globs).toContain('notes/personal/reflections/*'); + expect(globs).toContain('notes/originals/*'); + expect(globs).toContain('notes/personal/patterns/*'); + // Non-wiki globs pass through untouched. + expect(globs).toContain('dream-cycle-summaries/*'); + expect(globs.some(g => g.startsWith('wiki/'))).toBe(false); + }); +}); + +describe('#2415: loadOutputRoot validation + patterns gather scope', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('unset → wiki; trailing slash trimmed; invalid → wiki fallback', async () => { + expect(await loadOutputRoot(engine)).toBe('wiki'); + await engine.setConfig('dream.synthesize.output_root', 'notes/'); + expect(await loadOutputRoot(engine)).toBe('notes'); + await engine.setConfig('dream.synthesize.output_root', '../escape'); + expect(await loadOutputRoot(engine)).toBe('wiki'); + await engine.setConfig('dream.synthesize.output_root', 'Bad_Root'); + expect(await loadOutputRoot(engine)).toBe('wiki'); + }); + + test('patterns phase gathers reflections under the configured root', async () => { + await engine.setConfig('dream.synthesize.output_root', 'notes'); + for (let i = 0; i < 3; i++) { + await engine.putPage(`notes/personal/reflections/2026-07-17-r${i}`, { + type: 'note', + title: `R${i}`, + compiled_truth: `reflection ${i}`, + timeline: '', + frontmatter: {}, + }); + } + // A wiki/-rooted reflection must NOT be counted under the custom root. + await engine.putPage('wiki/personal/reflections/2026-07-17-old', { + type: 'note', + title: 'Old', + compiled_truth: 'legacy reflection', + timeline: '', + frontmatter: {}, + }); + const result = await runPhasePatterns(engine, { brainDir: '/tmp', dryRun: true }); + expect(result.status).toBe('ok'); + expect(result.details?.reflections_considered).toBe(3); + }); +}); diff --git a/test/cycle-patterns.test.ts b/test/cycle-patterns.test.ts index 1ea368d89..f50892348 100644 --- a/test/cycle-patterns.test.ts +++ b/test/cycle-patterns.test.ts @@ -74,8 +74,12 @@ describe('patterns phase wiring', () => { }); describe('patterns scope filter', () => { - test('filters reflections by slug LIKE wiki/personal/reflections/%', () => { - expect(patternsSrc).toContain("slug LIKE 'wiki/personal/reflections/%'"); + test('filters reflections by slug LIKE /personal/reflections/%', () => { + // #2415: the namespace root is configurable (dream.synthesize.output_root, + // default 'wiki') and bound as a parameter — the scope filter itself and + // the reflections sub-path stay pinned. + expect(patternsSrc).toContain('slug LIKE $2'); + expect(patternsSrc).toContain('/personal/reflections/%'); }); test('orders by updated_at DESC for recency-bias', () => { diff --git a/test/cycle-synthesize-slug-collection.test.ts b/test/cycle-synthesize-slug-collection.test.ts index bdbba6c4c..1ccbaa27e 100644 --- a/test/cycle-synthesize-slug-collection.test.ts +++ b/test/cycle-synthesize-slug-collection.test.ts @@ -21,7 +21,7 @@ import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { __testing } from '../src/core/cycle/synthesize.ts'; -const { collectChildPutPageSlugs } = __testing; +const { collectChildPutPageSlugs, stampDreamProvenance } = __testing; let engine: PGLiteEngine; @@ -103,4 +103,52 @@ describe('C6: collectChildPutPageSlugs survives double-encoded jsonb (#745)', () // Function silently drops rows whose slug resolves to null/empty. expect(refs.map((r: { slug: string }) => r.slug)).not.toContain('no-slug'); }); + + // #1586: refs are stamped with the cycle's resolved source, not a + // hardcoded 'default'. + test('stamps refs with the provided cycle sourceId (#1586)', async () => { + const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map(), 'mybrain'); + expect(refs.length).toBeGreaterThan(0); + for (const r of refs) expect(r.source_id).toBe('mybrain'); + }); + + test('defaults to source_id=default when no sourceId is passed (legacy)', async () => { + const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map()); + expect(refs.length).toBeGreaterThan(0); + for (const r of refs) expect(r.source_id).toBe('default'); + }); +}); + +describe('#2569: stampDreamProvenance persists the marker into DB frontmatter', () => { + test('merges dream_generated + dream_cycle_date into pages.frontmatter', async () => { + await engine.putPage('wiki/originals/ideas/2026-07-17-stamp-me-abc123', { + type: 'note', + title: 'Stamp me', + compiled_truth: 'body', + timeline: '', + frontmatter: { keep_me: 'yes' }, + }); + await stampDreamProvenance( + engine as any, + [{ slug: 'wiki/originals/ideas/2026-07-17-stamp-me-abc123', source_id: 'default' }], + '2026-07-17', + ); + const rows = await engine.executeRaw<{ fm: Record }>( + `SELECT frontmatter AS fm FROM pages WHERE slug = 'wiki/originals/ideas/2026-07-17-stamp-me-abc123'`, + ); + expect(rows.length).toBe(1); + const fm = rows[0].fm as Record; + // The stamp lands as real JSONB values (queryable via ->>), not a + // double-encoded string scalar. + expect(fm.dream_generated).toBe(true); + expect(fm.dream_cycle_date).toBe('2026-07-17'); + // Merge, not replace: pre-existing frontmatter keys survive. + expect(fm.keep_me).toBe('yes'); + }); + + test('is idempotent and never throws for a missing page', async () => { + const refs = [{ slug: 'wiki/originals/ideas/does-not-exist', source_id: 'default' }]; + await stampDreamProvenance(engine as any, refs, '2026-07-17'); // no throw + await stampDreamProvenance(engine as any, refs, '2026-07-17'); // idempotent + }); }); diff --git a/test/cycle/extract-atoms-synthesize-concepts.test.ts b/test/cycle/extract-atoms-synthesize-concepts.test.ts index a22874110..d14102495 100644 --- a/test/cycle/extract-atoms-synthesize-concepts.test.ts +++ b/test/cycle/extract-atoms-synthesize-concepts.test.ts @@ -316,4 +316,32 @@ describe('v0.41 T6: runPhaseSynthesizeConcepts via stubbed chat', () => { ); expect(rows[0].compiled_truth).toContain('Custom synthesized narrative'); }); + + // #2163: concept pages must enter the retrieval surface. The write routes + // through importFromContent (the same parse→chunk pipeline put_page uses), + // so content_chunks rows exist and source-boost's 1.3× 'concepts/' weight + // has something to boost. (Embeddings are skipped in this env — no + // provider — but chunks + search_vector land regardless.) + test('concept pages are chunked (#2163)', async () => { + const atoms = Array.from({ length: 12 }, (_, i) => ({ + slug: `c${i}`, + title: `Chunk atom ${i}`, + body: `Chunky body ${i}.`, + concept_refs: ['chunked-concept'], + })); + const chat = stubChat('A concept narrative long enough to produce at least one chunk.'); + await runPhaseSynthesizeConcepts(engine, { _atoms: atoms, _chat: chat }); + const rows = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n + FROM content_chunks c JOIN pages p ON p.id = c.page_id + WHERE p.slug = 'concepts/chunked-concept'`, + ); + expect(Number(rows[0].n)).toBeGreaterThan(0); + // Page metadata survives the importFromContent round-trip. + const page = await engine.executeRaw<{ type: string; fm: Record }>( + `SELECT type, frontmatter AS fm FROM pages WHERE slug = 'concepts/chunked-concept'`, + ); + expect(page[0].type).toBe('concept'); + expect((page[0].fm as Record).tier).toBe('T1'); + }); });