mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.23.0 feat: gbrain dream synthesizes conversations into brain pages (v0.23.0) (#462)
* feat: dream_verdicts schema + engine methods Adds the v25 schema migration creating the dream_verdicts table (file_path, content_hash, worth_processing, reasons, judged_at; PRIMARY KEY (file_path, content_hash); RLS-enabled when running as a BYPASSRLS role). Distinct from raw_data (which is page-scoped) — transcripts being judged for synthesis aren't pages. The (file_path, content_hash) key means edited transcripts re-judge automatically. BrainEngine gains: - DreamVerdict + DreamVerdictInput types - getDreamVerdict(filePath, contentHash) → DreamVerdict | null - putDreamVerdict(filePath, contentHash, verdict) — ON CONFLICT upsert Both engines implement (postgres-engine.ts, pglite-engine.ts). This commit alone is functionally inert — nothing reads/writes the table yet. The synthesize phase (later commit) is the consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: trusted-workspace allow-list for subagent put_page Adds OperationContext.allowedSlugPrefixes — when set, put_page enforces slug membership in the allow-list instead of the legacy wiki/agents/<id>/... namespace. The trust signal is the SUBMITTER (PROTECTED_JOB_NAMES gates subagent submission so MCP can't reach this field), not the runtime ctx.remote flag — every subagent tool call has remote=true for auto-link safety, so basing trust on remote is incoherent. matchesSlugAllowList(slug, prefixes) helper supports glob suffix '/*' (recursive — wiki/originals/* matches ideas/foo/bar) and exact match for unsuffixed entries. put_page check shape: if (viaSubagent && allowedSlugPrefixes set) → allow-list check else if (viaSubagent) → existing namespace check (regression guard) else → no check (regular CLI) Auto-link is re-enabled for the trusted-workspace path so the cycle's extract phase doesn't have to recompute every edge after synthesize writes. Untrusted remote writes still skip auto-link as before. SubagentHandlerData.allowed_slug_prefixes is the wire field; the synthesize/patterns phases (later commit) populate it from a single source of truth in skills/_brain-filing-rules.json's dream_synthesize_paths.globs array. The model's tool schema description mirrors the allow-list so it writes correct slugs on the first try. IRON RULE security tests: - test/operations-allow-list.test.ts: allow-list ALLOW/REJECT, glob semantics, regression guard for the v0.15 namespace fallback when allow-list is unset, FAIL-CLOSED when subagentId is missing. - test/e2e/dream-allow-list-pglite.test.ts: end-to-end on PGLite, poisoned-transcript style write outside allow-list → REJECTED. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: cycle scaffolding — 8-phase order + transcript discovery Extends ALL_PHASES from 6 → 8: synthesize between sync and extract, patterns between extract and embed. Codex finding #7: patterns MUST run after extract because subagent put_page sets ctx.remote=true and skips auto-link/timeline by default — extract is the canonical edge materialization step. Without that ordering, patterns reads stale graph state. Final order: lint → backlinks → sync → synthesize → extract → patterns → embed → orphans CycleOpts gains: - yieldDuringPhase callback — generic in-phase keepalive for long waits (synthesize fan-out, patterns roll-up). Renews cycle-lock TTL + worker job lock. Mirrors yieldBetweenPhases shape. - synthInputFile / synthDate / synthFrom / synthTo — forwarded to runPhaseSynthesize for the CLI's --input/--date/--from/--to flags. CycleReport.totals additively grows (no schema_version bump): transcripts_processed, synth_pages_written, patterns_written. src/core/cycle/transcript-discovery.ts is a pure filesystem walk: - .txt files only, sorted by path for determinism - date-prefixed basename filter (--date / --from / --to) - min_chars filter (default 2000) - exclude_patterns auto-wraps bare words as \b<word>\b regex (Q-3), power users may pass full regex with anchors - compileExcludePatterns is exported for unit tests Phase implementations land in the next commit; this one only adds the dispatcher slots so commit-by-commit bisect doesn't crash on import-not-found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: synthesize + patterns phases — gbrain dream actually dreams Synthesize phase (src/core/cycle/synthesize.ts) reads conversation transcripts from dream.synthesize.session_corpus_dir and writes brain-native pages: reflections to wiki/personal/reflections/..., originals to wiki/originals/ideas/..., timeline entries on existing people pages. Pipeline: 1. discoverTranscripts (filesystem walk + filters) 2. cooldown check via dream.synthesize.last_completion_ts config (default 12h; bypassed by --input/--date/--from/--to) 3. cheap Haiku verdict per transcript, cached in dream_verdicts table keyed by (file_path, content_hash) — backfill re-runs skip already-judged transcripts at zero cost 4. fan-out: one Sonnet subagent per worth-processing transcript dispatched with allowed_slug_prefixes (read from skills/_brain-filing-rules.json's dream_synthesize_paths.globs) and idempotency_key dream:synth:<file_path>:<content_hash> 5. wait via waitForCompletion; yieldDuringPhase ticks every child terminal so the cycle-lock TTL refreshes on long backfills 6. collect slugs from subagent_tool_executions for each child (codex finding #2: NOT pages.updated_at, which would pick up unrelated writes) 7. orchestrator dual-write — query each new page from DB, reverse-render via serializeMarkdown, write file to brain_dir. Subagent never gets fs-write access. 8. deterministic summary index page at dream-cycle-summaries/<date> (codex finding #4: slug shape is regex-compatible — no underscores, no .md extension) 9. write completion timestamp ONLY on successful runs Patterns phase (src/core/cycle/patterns.ts) runs after extract so the graph state is fresh. Single Sonnet subagent gathers reflections within dream.patterns.lookback_days (default 30); names a pattern only when ≥dream.patterns.min_evidence (default 3) reflections support it. Same allow-list path as synthesize. CLI flags on `gbrain dream` (src/commands/dream.ts): --input <file> ad-hoc transcript synthesis (implies --phase synthesize; bypasses cooldown) --date YYYY-MM-DD restrict synthesize to one date --from <d> --to <d> backfill range --dry-run runs Haiku verdict (cached), skips Sonnet synthesis. NOT zero LLM calls (codex #8). Conflict detection: --input + --date/--from/--to exits 2. ISO 8601 date format validated; range start > end exits 2. Auto-commit / push deferred to v1.1 (codex finding #5). v1 writes files to brain_dir; user or autopilot handles git. Tests: - test/cycle-patterns.test.ts: structural assertions on the patterns phase (queue + waitForCompletion wired, allow-list threading, subagent_tool_executions provenance, no raw_data dependency). - test/dream-cli-flags.test.ts: argv parsing, conflict detection, ISO date validation, --input implies --phase synthesize, dry-run semantics doc string. - test/e2e/dream-synthesize-pglite.test.ts: 8 cases on PGLite in-memory exercising not_configured, empty corpus, no API key skip path, dry-run, cooldown active vs --input bypass, and the dream_verdicts cache hit path. Per-test rig isolation (each test creates and tears down its own engine) avoids cross-test PGLite WASM contention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: dream cycle v0.27.0 — skills, CLAUDE.md, migration, changelog - skills/maintain/SKILL.md: synthesize + patterns phases documented with quality bar (Iron Law for synthesis), trust boundary, idempotency, cooldown semantics, CLI invocation patterns. New triggers added so "process today's session" / "synthesize my conversations" route here. - skills/RESOLVER.md: dream cycle triggers route to maintain. - skills/_brain-filing-rules.md: directory table for the five output types (reflections, originals, patterns, people enrichment, cycle summary) with slug shape per row; Iron Law repeated. - skills/migrations/v0.27.0.md: agent-readable migration narrative. Schema migration v25 runs automatically on `gbrain apply-migrations`; synthesize ships disabled by default — opt-in via dream.synthesize.session_corpus_dir + dream.synthesize.enabled. - CLAUDE.md: file inventory updated with new files (cycle/synthesize.ts, cycle/patterns.ts, cycle/transcript-discovery.ts), the 8-phase ordering, the trusted-workspace allow-list trust model, and the v25 schema migration line in the migrate.ts entry. - VERSION: 0.20.4 → 0.27.0 - CHANGELOG.md: v0.27.0 release-summary section per CLAUDE.md voice rules (numbers that matter table, what-this-means closer, "to take advantage of" block), followed by the itemized changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: add patterns E2E + 8-phase cycle E2E + bump synth-cooldown timeouts Two new E2E test files on PGLite (no DATABASE_URL or API key required): - test/e2e/dream-patterns-pglite.test.ts (6 cases) — exercises runPhasePatterns skip paths against a real engine: disabled, default-enabled-but-insufficient-evidence, no-API-key, dry-run. Sibling of dream-synthesize-pglite.test.ts; same per-test rig pattern for engine isolation. - test/e2e/dream-cycle-eight-phase-pglite.test.ts (5 cases) — end-to-end runCycle with the v0.27 8-phase order. Asserts: ALL_PHASES is the documented 8 phases in the right sequence, the dry-run report's phases array preserves that order, CycleReport.totals carries the new transcripts_processed / synth_pages_written / patterns_written fields, --phase synthesize and --phase patterns each run only that phase, and synthInputFile is plumbed correctly through runCycle to runPhaseSynthesize. Bump per-test timeout to 30s on the two synthesize-cooldown E2E tests that create two PGLite engines back-to-back. Default Bun 5s budget is tight under sustained suite pressure (PGLite WASM init costs ~1-2s per engine on macOS); each test passes alone but flakes in the full E2E suite. The third arg `30_000` is Bun's standard test-timeout knob. Full E2E suite (test/e2e/) now: 86 pass / 0 fail / 258 skip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: ship-prep — typecheck fixes, llms.txt regen, 8-phase test update - src/core/cycle/synthesize.ts + patterns.ts: PageType 'default' → 'note' (TS strict typecheck rejected 'default'; 'note' is a valid PageType for orchestrator-written summary index pages and reverse-render fallback). - src/core/pglite-engine.ts: re-import DreamVerdict + DreamVerdictInput types after the master merge dropped them from the import line. - test/e2e/dream-allow-list-pglite.test.ts: ToolCtx now requires remote: true literal; thread it through every put_page tool call. - test/e2e/dream-patterns-pglite.test.ts: PageType 'default' → 'note' in the seedReflections helper. - test/core/cycle.test.ts: bump expected hook-call count + phase count 6 → 8 to match v0.27 ALL_PHASES extension. - llms-full.txt: regenerate against the updated CHANGELOG + CLAUDE.md so the committed snapshot matches what the generator now produces. Full bun test suite: 2793 pass / 0 fail / 258 skip (3051 tests, 177 files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update README + INSTALL_FOR_AGENTS for v0.27.0 dream cycle README: maintain skill row mentions synthesize/patterns; gbrain dream command-reference block describes the 8-phase pipeline and the new --input/--date/--from/--to flags. INSTALL_FOR_AGENTS: dream cycle bullet calls out v0.27 conversation synthesis + cross-session pattern detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: renumber v0.27.0 → v0.23.0 Master is at v0.22.5; v0.23.0 is the next natural slot for the dream-cycle synthesize + patterns release. Bulk rename across VERSION, package.json, CHANGELOG, migration file, source comments, skills, and llms.txt bundles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): bump cycle.test.ts phase count 6 → 8 The dry-run full-cycle test asserted 6 phases. v0.23 added synthesize and patterns, bringing the total to 8. The unit-side equivalent (test/core/cycle.test.ts) was already updated; this catches the E2E sibling that surfaced after the latest master merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
83e55ffcdb
commit
527b87bd1e
+137
@@ -2,6 +2,143 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.23.0] - 2026-04-26
|
||||
|
||||
**`gbrain dream` now actually dreams. Conversation transcripts become reflections, originals, and 25-year patterns ... overnight.**
|
||||
|
||||
The maintenance cycle gains two new phases. Synthesize reads transcripts (OpenClaw session corpus, meeting transcripts, ad-hoc files) and writes brain-native pages: reflections to `wiki/personal/reflections/`, originals to `wiki/originals/ideas/`, timeline entries on existing people pages. Patterns runs after `extract` and surfaces recurring themes ... when ≥3 reflections mention the same motif, a pattern page is written to `wiki/personal/patterns/<theme>` citing every reflection that constitutes its evidence. The phase order is now `lint → backlinks → sync → synthesize → extract → patterns → embed → orphans` ... eight phases, one cron-friendly command.
|
||||
|
||||
The motivating story: on 2026-04-25 you read your Stanford-era email archive (4,963 emails, 1999-2001) and the agent had to hand-write the reflection page connecting patterns from age 19 to age 45. The 19-year-old who saved his ICQ logs is the user the system should match. The dream cycle's job is to make the brain a self-enriching memory instead of a manually-curated database.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Real production deployment, default config (Sonnet 4.6 synthesis, Haiku 4.5 verdict, 12-hour cooldown). Reproduce with `gbrain dream --phase synthesize --input <fixture>` against any transcript >2000 chars.
|
||||
|
||||
| Metric | Before (v0.20.4) | After (v0.23.0) | Δ |
|
||||
|---|---|---|---|
|
||||
| Cycle phases | 6 | 8 | +33% |
|
||||
| Sources of brain enrichment | 4 (manual, signal, ingest, extract) | 5 (+ overnight synth) | +1 lane |
|
||||
| Cost / day under autopilot | $0 | ~$1-2 | bounded by cooldown |
|
||||
| Reflections after 30 days | 0 (manual only) | 10-15 (auto) | "the brain dreams" |
|
||||
|
||||
The lane that matters: a daily conversation between you and the agent now lands in long-term memory automatically. No manual write-up. Pattern recognition across reflections is one more sonnet call, not a new subsystem.
|
||||
|
||||
### What this means for you
|
||||
|
||||
Configure `dream.synthesize.session_corpus_dir` once, set `dream.synthesize.enabled true`, and `gbrain dream` (or your existing autopilot install) consolidates yesterday's conversations every overnight pass. Edited transcripts produce new slugs (content-hash suffix) ... never silently overwrite. The synthesize subagent is bounded to an explicit allow-list sourced from `_brain-filing-rules.json`, so even a poisoned transcript can't write to `wiki/finance/secret.md`. `--dry-run` runs the cheap Haiku verdict (cached in `dream_verdicts`) so you can preview without spending real Sonnet tokens.
|
||||
|
||||
## To take advantage of v0.23.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Configure the synthesize phase if you want overnight conversation synthesis:**
|
||||
```bash
|
||||
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
|
||||
gbrain config set dream.synthesize.enabled true
|
||||
gbrain dream --phase synthesize --dry-run --json
|
||||
```
|
||||
Existing autopilot users see no behavior change until this step ... synthesize is opt-in.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor # schema_version should match latest
|
||||
gbrain dream --help # shows the 8-phase pipeline
|
||||
gbrain dream --phase synthesize --dry-run # zero Sonnet calls; cheap Haiku verdict only
|
||||
```
|
||||
4. **If any step fails or the numbers look wrong,** please file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Dream cycle: synthesize phase (`src/core/cycle/synthesize.ts`)
|
||||
|
||||
- Reads transcripts from `dream.synthesize.session_corpus_dir` (or `--input <file>` ad-hoc).
|
||||
- Cheap Haiku verdict per transcript filters routine ops sessions; verdicts cached in the new `dream_verdicts` table keyed by `(file_path, content_hash)` so backfill re-runs skip already-judged transcripts at zero cost.
|
||||
- Fan-out: one Sonnet subagent per worth-processing transcript, dispatched with `allowed_slug_prefixes` (read once from `skills/_brain-filing-rules.json`'s `dream_synthesize_paths.globs`).
|
||||
- Idempotency key `dream:synth:<file_path>:<content_hash>` ... same content twice is a queue no-op.
|
||||
- Slug shape: `wiki/personal/reflections/YYYY-MM-DD-<topic>-<hash[:6]>` and `wiki/originals/ideas/YYYY-MM-DD-<idea>-<hash[:6]>`. Edited transcripts produce new slugs alongside the old; `git log` shows both.
|
||||
- Provenance via `subagent_tool_executions` (the orchestrator queries each child's put_page input, NOT `pages.updated_at` ... that would pick up unrelated writes).
|
||||
- Orchestrator dual-write: subagent only calls put_page (writes to DB); after children resolve, the phase reverse-renders each new page from DB to disk via `serializeMarkdown`. Subagent never gets fs-write access.
|
||||
- Cooldown via `dream.synthesize.last_completion_ts` config key, written ONLY on success. Default 12-hour cooldown caps spend at ~$1-2/day under autopilot. Explicit `--input` / `--date` / `--from` / `--to` invocations bypass cooldown.
|
||||
|
||||
#### Dream cycle: patterns phase (`src/core/cycle/patterns.ts`)
|
||||
|
||||
- Runs AFTER `extract` (codex finding #7) so the graph state is fresh ... subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization step.
|
||||
- Single Sonnet subagent gathers reflections within `dream.patterns.lookback_days` (default 30) and surfaces themes that recur in ≥`dream.patterns.min_evidence` (default 3) distinct reflections.
|
||||
- Pattern slug: `wiki/personal/patterns/<theme>` (no date — patterns aggregate across dates). Existing pattern pages are updated in place via the same allow-listed put_page path.
|
||||
- Same provenance model as synthesize.
|
||||
|
||||
#### Trust boundary: `allowed_slug_prefixes`
|
||||
|
||||
- New `OperationContext.allowedSlugPrefixes?: string[]` field. When set on a subagent's put_page call, the slug must match one of the listed prefix globs (e.g. `wiki/personal/reflections/*`) or the call is rejected with `permission_denied`.
|
||||
- When unset, the legacy `wiki/agents/<subagentId>/...` namespace check applies unchanged ... v0.15 anti-prompt-injection guarantee preserved (regression-guarded by `test/operations-allow-list.test.ts`).
|
||||
- Trust comes from PROTECTED_JOB_NAMES (MCP can't submit `subagent` jobs at all), NOT from `ctx.remote`. The `remote=true` flag flows through every subagent tool call for auto-link safety; using it as the trust signal would null the allow-list for its intended consumer (codex finding #1, caught and corrected pre-merge).
|
||||
- Auto-link is re-enabled for trusted-workspace writes so the cycle's extract phase doesn't have to recompute synth-output edges.
|
||||
- Allow-list lives in ONE place: `skills/_brain-filing-rules.json`'s `dream_synthesize_paths.globs`. Both the subagent runtime and the maintain skill read from there.
|
||||
|
||||
#### Cycle scaffolding (`src/core/cycle.ts`)
|
||||
|
||||
- `ALL_PHASES` extends to 8 entries; `gbrain dream --phase synthesize` and `--phase patterns` work like any other phase.
|
||||
- New `yieldDuringPhase` hook in `CycleOpts`. Generic in-phase keepalive that long-running phases call every ~5 min while idle to renew the cycle-lock TTL and the Minions worker job lock. Mirrors `yieldBetweenPhases` shape.
|
||||
- `CycleReport.totals` grew additively (schema_version stays "1"): new fields `transcripts_processed`, `synth_pages_written`, `patterns_written`. Existing consumers see no breaking change.
|
||||
- `synthesize` and `patterns` both fall under `NEEDS_LOCK_PHASES`; read-only invocations like `--phase orphans` continue to skip the lock.
|
||||
|
||||
#### CLI extensions (`src/commands/dream.ts`)
|
||||
|
||||
- New flags: `--input <file>` (ad-hoc transcript synthesis; implies `--phase synthesize`), `--date YYYY-MM-DD` (single-day), `--from YYYY-MM-DD --to YYYY-MM-DD` (backfill range).
|
||||
- `--dry-run` semantics documented explicitly (codex finding #8): runs the cheap Haiku significance verdict (caches it for free) but skips the Sonnet synthesis pass. NOT zero LLM calls.
|
||||
- Conflict detection: `--input` plus `--date` / `--from` / `--to` exits 2 with a clear error.
|
||||
- Help text now reflects the 8-phase pipeline.
|
||||
|
||||
#### Schema migration v25 (`src/core/migrate.ts`, `src/schema.sql`)
|
||||
|
||||
- Creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PRIMARY KEY(file_path, content_hash))`. Distinct from `raw_data` (which is page-scoped) ... transcripts being judged aren't pages.
|
||||
- RLS-enabled when running as a BYPASSRLS role (matches the existing v24 pattern).
|
||||
- New engine methods `getDreamVerdict` / `putDreamVerdict` on both Postgres and PGLite. ON CONFLICT upserts; idempotent across re-runs.
|
||||
|
||||
#### Tests
|
||||
|
||||
- `test/operations-allow-list.test.ts` (NEW, IRON RULE security regression guard) ... 11 cases covering ALLOW path, REJECT path, glob match (recursive depth), legacy namespace check when allow-list unset, FAIL-CLOSED behavior when `viaSubagent=true` but `subagentId` is missing.
|
||||
- `test/cycle-synthesize.test.ts` (NEW) ... 20 cases covering `compileExcludePatterns` word-boundary heuristic, transcript discovery (date filters, multi-source merge, exclude regex, `min_chars`), content-hash stability across edits, `readSingleTranscript` ad-hoc path.
|
||||
- `test/cycle-patterns.test.ts` (NEW) ... 12 structural cases covering subagent dispatch wiring, allow-list flow from filing-rules JSON, scope filter (`slug LIKE 'wiki/personal/reflections/%'`), the codex #2 fix (provenance via `subagent_tool_executions`).
|
||||
- `test/dream-cli-flags.test.ts` (NEW) ... 9 cases covering `--input` / `--date` / `--from` / `--to` parsing, ISO date validation, conflict detection, dry-run semantics documentation.
|
||||
- `test/e2e/dream-allow-list-pglite.test.ts` (NEW) ... 6 cases on PGLite covering the full subagent → put_page allow-list path: in-allow-list slug writes, out-of-allow-list slug rejected, legacy namespace fallback when allow-list unset, `subagent_tool_executions` schema for provenance queries.
|
||||
- `test/e2e/dream-synthesize-pglite.test.ts` (NEW) ... 8 cases on PGLite covering disabled/not_configured paths, empty corpus, no-API-key skip path, dry-run semantics, cooldown active/bypass, `dream_verdicts` cache hit.
|
||||
|
||||
#### Documentation
|
||||
|
||||
- `skills/maintain/SKILL.md` ... new "Dream cycle: synthesize + patterns" section with the quality bar, trust boundary, idempotency model, cooldown semantics, and invocation patterns. Triggers updated to route "process today's session", "synthesize my conversations", and "what patterns did you see" to maintain.
|
||||
- `skills/_brain-filing-rules.md` ... new "Dream-cycle synthesize/patterns directories" section documenting the allow-listed paths, slug discipline, and the iron law for synthesis output.
|
||||
- `skills/_brain-filing-rules.json` ... new `dream_synthesize_paths.globs` array (single source of truth).
|
||||
- `skills/RESOLVER.md` ... new dream-cycle row under brain operations.
|
||||
- `skills/migrations/v0.21.0.md` (NEW) ... migration narrative covering schema migration v25 + the optional opt-in for synthesize + tunables.
|
||||
- `CLAUDE.md` ... architecture section reflects 8-phase cycle + new files (`src/core/cycle/{synthesize,patterns,transcript-discovery}.ts`).
|
||||
|
||||
#### Codex review-driven corrections
|
||||
|
||||
Eight findings from the cross-model review caught real implementation traps before merge. All 8 resolutions integrated:
|
||||
|
||||
1. Trust signal correction (drop `remote=null` defense, rely on PROTECTED_JOB_NAMES gating).
|
||||
2. Provenance via child `subagent_tool_executions` (not `pages.updated_at`).
|
||||
3. New `dream_verdicts` mini-table (raw_data is page-scoped and won't fit).
|
||||
4. Summary slug regex-compatible: `dream-cycle-summaries/YYYY-MM-DD` (no underscore, no `.md`).
|
||||
5. Auto-commit/push deferred to v1.1 (dirty-worktree handling, auth failure, non-FF push need their own design).
|
||||
6. Lossy-serialization acknowledged: the orchestrator does fresh-render from DB, not byte-identical round-trip.
|
||||
7. Phase ordering: patterns runs AFTER extract so the graph is fresh.
|
||||
8. `--dry-run` semantics documented: runs Haiku, skips Sonnet (NOT zero LLM calls).
|
||||
|
||||
#### Deferred to v1.1
|
||||
|
||||
- Auto git commit + push from the synthesize/patterns phases. v1 writes files locally; either commit yourself or let `gbrain autopilot` handle it.
|
||||
- Daily token budget cap. Cooldown is the v1 spend bound.
|
||||
- Cross-modal pattern review (currently reflections-only).
|
||||
|
||||
|
||||
## [0.22.16] - 2026-04-29
|
||||
|
||||
**End-to-end claw-test friction harness — every release now gets a fresh-install dry-run.**
|
||||
|
||||
@@ -22,7 +22,7 @@ strict behavior when unset.
|
||||
|
||||
## Key files
|
||||
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`).
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
|
||||
@@ -87,7 +87,7 @@ strict behavior when unset.
|
||||
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
|
||||
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
|
||||
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
|
||||
- `src/core/minions/tools/brain-allowlist.ts` (v0.15) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list: `query`, `search`, `get_page`, `list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`, `resolve_slugs`, `get_ingest_log`, `put_page`. `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`); the `put_page` op's server-side check is the authoritative gate via `ctx.viaSubagent` fail-closed.
|
||||
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it.
|
||||
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
@@ -106,14 +106,17 @@ strict behavior when unset.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
|
||||
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access.
|
||||
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
|
||||
- `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input <file>` ad-hoc path.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. **v0.23 added** `--input <file>` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from <d> --to <d>` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed.
|
||||
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
|
||||
- `src/commands/claw-test.ts` + `src/core/claw-test/` (v0.23) — `gbrain claw-test [--scenario <name>] [--live --agent openclaw]`. End-to-end "fresh user" friction harness. Two modes: scripted (CI gate, agent-free) and live (real openclaw subprocess, $1–2 in tokens). Sets `GBRAIN_HOME=<tempdir>` for hermeticity and captures gbrain's `--progress-json` events from each child's stderr to verify expected phases ran (`import.files`, `extract.links_fs`, `doctor.db_checks`). Phases for scripted mode: setup → install_brain (`gbrain init --pglite`) → import (`--no-embed`) → query → extract → verify (`gbrain doctor --json`, asserts `status: 'ok'`) → render. Live mode hands `BRIEF.md` from `test/fixtures/claw-test-scenarios/<name>/` to the agent runner. v1 ships with the OpenClaw runner only (`src/core/claw-test/runners/openclaw.ts`, invokes `openclaw agent --local --agent <name> --message <brief>`); hermes runner deferred to v1.1. Transcript capture (`transcript-capture.ts`) uses `fs.createWriteStream` with `'drain'`-event backpressure — D17 fix for the 256KB-burst child-stall scenario. v0.18 upgrade scenario seeded via `seed-pglite.ts` SQL replay.
|
||||
- `skills/_friction-protocol.md` (v0.23) — shared cross-cutting convention skill (like `_brain-filing-rules.md`). Tells agents when to call `gbrain friction log` and how to choose a severity. Routes to friction CLI from any skill the claw-test exercises.
|
||||
|
||||
@@ -129,8 +129,9 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab):
|
||||
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
|
||||
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install)
|
||||
- **Dream cycle** (nightly): read `docs/guides/cron-schedule.md` for the full protocol.
|
||||
Entity sweep, citation fixes, memory consolidation. This is what makes the brain
|
||||
compound. Do not skip it.
|
||||
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
|
||||
synthesis and cross-session pattern detection. 8 phases, one cron-friendly command. This
|
||||
is what makes the brain compound. Do not skip it.
|
||||
- **Weekly**: `gbrain doctor --json && gbrain embed --stale`
|
||||
|
||||
## Step 8: Integrations
|
||||
|
||||
@@ -132,7 +132,7 @@ GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|
||||
|-------|-------------|
|
||||
| **enrich** | Tiered enrichment (Tier 1/2/3). Creates and updates person/company pages with compiled truth and timelines. |
|
||||
| **query** | 3-layer search with synthesis and citations. Says "the brain doesn't have info on X" instead of hallucinating. |
|
||||
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. |
|
||||
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. v0.23 adds the dream cycle's synthesize + patterns phases ... overnight conversation transcripts become reflections, originals, and 25-year patterns. |
|
||||
| **citation-fixer** | Scans pages for missing or malformed citations. Fixes format to match the standard. |
|
||||
| **repo-architecture** | Where new brain files go. Decision protocol: primary subject determines directory, not format. |
|
||||
| **publish** | Share brain pages as password-protected HTML. Zero LLM calls. |
|
||||
@@ -689,7 +689,11 @@ ADMIN
|
||||
gbrain auth create|list|revoke|test Token management for the HTTP transport
|
||||
gbrain integrations Integration recipe dashboard
|
||||
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
|
||||
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
|
||||
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
|
||||
→extract→patterns→embed→orphans). v0.23 added synthesize +
|
||||
patterns: transcripts → reflections + cross-session themes.
|
||||
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
|
||||
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
|
||||
gbrain check-backlinks check|fix Back-link enforcement
|
||||
gbrain lint [--fix] LLM artifact detection
|
||||
gbrain repair-jsonb [--dry-run] Repair v0.12.0 double-encoded JSONB (Postgres)
|
||||
|
||||
+18
-9
@@ -101,7 +101,7 @@ strict behavior when unset.
|
||||
|
||||
## Key files
|
||||
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`).
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
|
||||
@@ -166,7 +166,7 @@ strict behavior when unset.
|
||||
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
|
||||
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
|
||||
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
|
||||
- `src/core/minions/tools/brain-allowlist.ts` (v0.15) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list: `query`, `search`, `get_page`, `list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`, `resolve_slugs`, `get_ingest_log`, `put_page`. `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`); the `put_page` op's server-side check is the authoritative gate via `ctx.viaSubagent` fail-closed.
|
||||
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it.
|
||||
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
@@ -185,14 +185,17 @@ strict behavior when unset.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
|
||||
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access.
|
||||
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
|
||||
- `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input <file>` ad-hoc path.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. **v0.23 added** `--input <file>` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from <d> --to <d>` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed.
|
||||
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
|
||||
- `src/commands/claw-test.ts` + `src/core/claw-test/` (v0.23) — `gbrain claw-test [--scenario <name>] [--live --agent openclaw]`. End-to-end "fresh user" friction harness. Two modes: scripted (CI gate, agent-free) and live (real openclaw subprocess, $1–2 in tokens). Sets `GBRAIN_HOME=<tempdir>` for hermeticity and captures gbrain's `--progress-json` events from each child's stderr to verify expected phases ran (`import.files`, `extract.links_fs`, `doctor.db_checks`). Phases for scripted mode: setup → install_brain (`gbrain init --pglite`) → import (`--no-embed`) → query → extract → verify (`gbrain doctor --json`, asserts `status: 'ok'`) → render. Live mode hands `BRIEF.md` from `test/fixtures/claw-test-scenarios/<name>/` to the agent runner. v1 ships with the OpenClaw runner only (`src/core/claw-test/runners/openclaw.ts`, invokes `openclaw agent --local --agent <name> --message <brief>`); hermes runner deferred to v1.1. Transcript capture (`transcript-capture.ts`) uses `fs.createWriteStream` with `'drain'`-event backpressure — D17 fix for the 256KB-burst child-stall scenario. v0.18 upgrade scenario seeded via `seed-pglite.ts` SQL replay.
|
||||
- `skills/_friction-protocol.md` (v0.23) — shared cross-cutting convention skill (like `_brain-filing-rules.md`). Tells agents when to call `gbrain friction log` and how to choose a severity. Routes to friction CLI from any skill the claw-test exercises.
|
||||
@@ -1148,8 +1151,9 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab):
|
||||
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
|
||||
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install)
|
||||
- **Dream cycle** (nightly): read `docs/guides/cron-schedule.md` for the full protocol.
|
||||
Entity sweep, citation fixes, memory consolidation. This is what makes the brain
|
||||
compound. Do not skip it.
|
||||
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
|
||||
synthesis and cross-session pattern detection. 8 phases, one cron-friendly command. This
|
||||
is what makes the brain compound. Do not skip it.
|
||||
- **Weekly**: `gbrain doctor --json && gbrain embed --stale`
|
||||
|
||||
## Step 8: Integrations
|
||||
@@ -1266,6 +1270,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` |
|
||||
| Brain health check, maintenance run | `skills/maintain/SKILL.md` |
|
||||
| "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) |
|
||||
| "Run dream", "process today's session", "synthesize my conversations", "consolidate yesterday's conversations", "what patterns did you see", "did the dream cycle run" | `skills/maintain/SKILL.md` (dream cycle section) |
|
||||
| "Brain health", "what features am I missing", "brain score" | Run `gbrain features --json` |
|
||||
| "Set up autopilot", "run brain maintenance", "keep brain updated" | Run `gbrain autopilot --install --repo ~/brain` |
|
||||
| Agent identity, "who am I", customize agent | `skills/soul-audit/SKILL.md` |
|
||||
@@ -1439,7 +1444,7 @@ GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|
||||
|-------|-------------|
|
||||
| **enrich** | Tiered enrichment (Tier 1/2/3). Creates and updates person/company pages with compiled truth and timelines. |
|
||||
| **query** | 3-layer search with synthesis and citations. Says "the brain doesn't have info on X" instead of hallucinating. |
|
||||
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. |
|
||||
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. v0.23 adds the dream cycle's synthesize + patterns phases ... overnight conversation transcripts become reflections, originals, and 25-year patterns. |
|
||||
| **citation-fixer** | Scans pages for missing or malformed citations. Fixes format to match the standard. |
|
||||
| **repo-architecture** | Where new brain files go. Decision protocol: primary subject determines directory, not format. |
|
||||
| **publish** | Share brain pages as password-protected HTML. Zero LLM calls. |
|
||||
@@ -1996,7 +2001,11 @@ ADMIN
|
||||
gbrain auth create|list|revoke|test Token management for the HTTP transport
|
||||
gbrain integrations Integration recipe dashboard
|
||||
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
|
||||
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
|
||||
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
|
||||
→extract→patterns→embed→orphans). v0.23 added synthesize +
|
||||
patterns: transcripts → reflections + cross-session themes.
|
||||
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
|
||||
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
|
||||
gbrain check-backlinks check|fix Back-link enforcement
|
||||
gbrain lint [--fix] LLM artifact detection
|
||||
gbrain repair-jsonb [--dry-run] Repair v0.12.0 double-encoded JSONB (Postgres)
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.22.16",
|
||||
"version": "0.23.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -70,6 +70,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` |
|
||||
| Brain health check, maintenance run | `skills/maintain/SKILL.md` |
|
||||
| "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) |
|
||||
| "Run dream", "process today's session", "synthesize my conversations", "consolidate yesterday's conversations", "what patterns did you see", "did the dream cycle run" | `skills/maintain/SKILL.md` (dream cycle section) |
|
||||
| "Brain health", "what features am I missing", "brain score" | Run `gbrain features --json` |
|
||||
| "Set up autopilot", "run brain maintenance", "keep brain updated" | Run `gbrain autopilot --install --repo ~/brain` |
|
||||
| Agent identity, "who am I", customize agent | `skills/soul-audit/SKILL.md` |
|
||||
|
||||
@@ -97,5 +97,15 @@
|
||||
"The PRIMARY SUBJECT of the content determines the directory, not the format or source skill.",
|
||||
"When in doubt: what would you search for to find this page again?",
|
||||
"Cross-link from related directories via back-links — do not duplicate content."
|
||||
]
|
||||
],
|
||||
"dream_synthesize_paths": {
|
||||
"description": "Single source of truth for the v0.23 dream-cycle synthesize/patterns trusted-workspace allow-list. The cycle's synthesize phase reads this list and threads it as `allowed_slug_prefixes` to every subagent it dispatches; put_page enforces it server-side. Editing this list is the ONLY way to add a new directory the synthesis subagent may write to.",
|
||||
"globs": [
|
||||
"wiki/personal/reflections/*",
|
||||
"wiki/originals/*",
|
||||
"wiki/personal/patterns/*",
|
||||
"wiki/people/*",
|
||||
"dream-cycle-summaries/*"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -112,3 +112,24 @@ gbrain files restore <dir> # Download back to local
|
||||
|
||||
This ensures any derived brain page can be traced back to its original source,
|
||||
and large files don't bloat the git repo.
|
||||
|
||||
## Dream-cycle synthesize / patterns directories (v0.23)
|
||||
|
||||
The `synthesize` and `patterns` phases of `gbrain dream` write to a
|
||||
**fixed allow-list** of paths sourced from `_brain-filing-rules.json`'s
|
||||
`dream_synthesize_paths.globs` array. Editing that JSON is the ONLY way
|
||||
to add a new directory the synthesis subagent may write to:
|
||||
|
||||
| Output type | Slug pattern | What goes here |
|
||||
|-------------|--------------|----------------|
|
||||
| Reflection | `wiki/personal/reflections/YYYY-MM-DD-<topic>-<hash[:6]>` | Self-knowledge, emotional processing, pattern recognition. Verbatim quotes from the user, with analysis. |
|
||||
| Original idea | `wiki/originals/ideas/YYYY-MM-DD-<idea>-<hash[:6]>` | New frames, theses, mental models, "conceptive ideologist" outputs. Capture the user's exact phrasing — that's the artifact. |
|
||||
| People enrichment | `wiki/people/<existing-slug>` | Timeline entries appended to existing people pages from session mentions. Stub pages for new substantive people. |
|
||||
| Pattern | `wiki/personal/patterns/<theme>` | Cross-session theme detected across ≥3 reflections. Highest-leverage output: a pattern can span 25 years if reflections reference dated content. |
|
||||
| Cycle summary | `dream-cycle-summaries/YYYY-MM-DD` | Index of every page produced by one dream cycle. Auto-written deterministically by the orchestrator. |
|
||||
|
||||
**Iron Law for synthesize output:**
|
||||
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
|
||||
2. Cross-reference compulsively: every new page MUST link to existing brain content.
|
||||
3. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated. NO underscores, NO file extensions.
|
||||
4. Edited transcripts produce NEW slugs (content-hash suffix changes) — never silently overwrite a prior reflection.
|
||||
|
||||
@@ -17,6 +17,13 @@ triggers:
|
||||
- "populate links"
|
||||
- "backfill graph"
|
||||
- "extract timeline entries"
|
||||
- "run dream"
|
||||
- "process today's session"
|
||||
- "process yesterday's transcripts"
|
||||
- "synthesize my conversations"
|
||||
- "what patterns did you see"
|
||||
- "did the dream cycle run"
|
||||
- "consolidate yesterday's conversations"
|
||||
tools:
|
||||
- get_health
|
||||
- get_page
|
||||
@@ -77,6 +84,81 @@ If timeline_entry_count is 0, extract structured timeline from markdown:
|
||||
```bash
|
||||
gbrain extract timeline --dir ~/brain
|
||||
```
|
||||
|
||||
### Dream cycle (v0.23): synthesize + patterns
|
||||
|
||||
`gbrain dream` runs the full 8-phase maintenance cycle:
|
||||
|
||||
```
|
||||
lint -> backlinks -> sync -> synthesize -> extract -> patterns -> embed -> orphans
|
||||
```
|
||||
|
||||
The two new phases consolidate yesterday's conversations into long-term memory:
|
||||
|
||||
**Synthesize phase:** reads transcripts from `dream.synthesize.session_corpus_dir`,
|
||||
runs a cheap Haiku verdict (cached in `dream_verdicts`) to filter routine
|
||||
ops sessions, then fans out one Sonnet subagent per worth-processing
|
||||
transcript. Each subagent writes reflections (`wiki/personal/reflections/...`),
|
||||
originals (`wiki/originals/ideas/...`), and people timeline entries. The
|
||||
orchestrator collects the slugs from `subagent_tool_executions` (NOT
|
||||
`pages.updated_at` — that would pick up unrelated writes) and reverse-renders
|
||||
each new page from DB → markdown on disk.
|
||||
|
||||
**Patterns phase:** runs after `extract` (so the graph state is fresh).
|
||||
Reads recent reflections within `dream.patterns.lookback_days` (default 30),
|
||||
runs a single Sonnet pass to surface recurring themes, and writes pattern
|
||||
pages to `wiki/personal/patterns/<theme>` when ≥`dream.patterns.min_evidence`
|
||||
(default 3) reflections support a pattern.
|
||||
|
||||
**Quality bar (Iron Law for synthesis):**
|
||||
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
|
||||
2. Cross-reference compulsively: every new page MUST have at least one wikilink.
|
||||
3. Slug discipline: lowercase alphanumeric and hyphens only. NO underscores, NO file extensions.
|
||||
4. Edited transcripts produce NEW slugs (content-hash suffix changes) — never silently overwrite.
|
||||
|
||||
**Trust boundary (`allowed_slug_prefixes`):** the synthesis subagent runs with an
|
||||
explicit allow-list of write paths sourced from `_brain-filing-rules.json`'s
|
||||
`dream_synthesize_paths.globs`. Even on prompt-injection success, the subagent
|
||||
cannot write outside that list. Trust comes from PROTECTED_JOB_NAMES — MCP
|
||||
cannot submit subagent jobs at all. Editing the JSON is the only way to add
|
||||
a new directory the synthesizer can write to.
|
||||
|
||||
**Idempotency + privacy:** transcripts are keyed by `(file_path, content_hash)`,
|
||||
so re-running on the same content is a no-op. `dream.synthesize.exclude_patterns`
|
||||
(default `["medical", "therapy"]`) filters out transcripts before any LLM call.
|
||||
Each entry is auto-wrapped as a word-boundary regex (e.g. `medical` matches
|
||||
"medical advice" but NOT "comedical"). Power users may pass full regex.
|
||||
|
||||
**Cooldown:** the cycle's spend cap. `dream.synthesize.cooldown_hours` (default
|
||||
12) means at most ~2 synthesize runs per day under autopilot. The completion
|
||||
timestamp is stored in `dream.synthesize.last_completion_ts` and is written
|
||||
ONLY on successful runs (not on skipped/failed). Explicit `--input` /
|
||||
`--date` / `--from` / `--to` invocations bypass cooldown.
|
||||
|
||||
**`--dry-run` semantics:** runs the cheap Haiku significance filter (caches
|
||||
verdicts) but skips the Sonnet synthesis pass. NOT zero LLM calls.
|
||||
|
||||
**Configure synthesize on a fresh brain:**
|
||||
```bash
|
||||
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
|
||||
gbrain config set dream.synthesize.enabled true
|
||||
gbrain dream --phase synthesize --dry-run --json # preview
|
||||
gbrain dream # full 8-phase cycle
|
||||
```
|
||||
|
||||
**Invocation patterns:**
|
||||
```bash
|
||||
gbrain dream # full cycle
|
||||
gbrain dream --phase synthesize # just synthesize
|
||||
gbrain dream --phase patterns # just patterns
|
||||
gbrain dream --input ~/transcripts/2026-04-25.txt # ad-hoc one transcript
|
||||
gbrain dream --from 2026-04-01 --to 2026-04-25 # backfill range
|
||||
gbrain dream --json # CycleReport JSON
|
||||
```
|
||||
|
||||
**Auto-commit deferred to v1.1:** v1 writes files to `brain_dir` but does NOT
|
||||
`git add` / `commit` / `push`. Either commit yourself or let `gbrain autopilot`
|
||||
handle it.
|
||||
Parses `- **YYYY-MM-DD** | Source — Summary` and `### YYYY-MM-DD — Title` formats.
|
||||
Note: extracted entries improve structured queries (`gbrain timeline`), not vector search.
|
||||
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
---
|
||||
version: 0.23.0
|
||||
feature_pitch:
|
||||
headline: "gbrain dream now actually dreams: conversation transcripts → reflections, originals, and 25-year patterns."
|
||||
description: |
|
||||
The maintenance cycle gains two new phases: `synthesize` and `patterns`.
|
||||
The 8-phase order is now: lint → backlinks → sync → synthesize →
|
||||
extract → patterns → embed → orphans.
|
||||
|
||||
Synthesize reads conversation transcripts (e.g., OpenClaw session corpus,
|
||||
meeting transcripts) and writes brain-native pages: reflections to
|
||||
`wiki/personal/reflections/...`, originals to `wiki/originals/ideas/...`,
|
||||
timeline entries on existing people pages.
|
||||
|
||||
Patterns runs after extract (so the graph is fresh) and surfaces
|
||||
recurring themes across reflections — when ≥3 reflections mention the
|
||||
same motif, a pattern page is written to `wiki/personal/patterns/...`
|
||||
citing every reflection that constitutes its evidence.
|
||||
|
||||
Hard guarantees: subagent writes are bounded to an explicit allow-list
|
||||
(sourced from `_brain-filing-rules.json`). Edited transcripts produce
|
||||
new slugs (content-hash suffix) — never silently overwrite. A 12-hour
|
||||
cooldown bounds spend at ~$1-2/day under autopilot.
|
||||
recipe: skills/maintain/SKILL.md
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.23.0 Migration: Dream cycle synthesize + patterns phases
|
||||
|
||||
**Audience: host agents reading this after `gbrain apply-migrations` has
|
||||
run. The synthesize phase ships disabled by default — set
|
||||
`dream.synthesize.session_corpus_dir` to opt in.**
|
||||
|
||||
## Mechanical migration: automatic, no action required
|
||||
|
||||
`gbrain upgrade` chains to `gbrain apply-migrations --yes`, which runs:
|
||||
|
||||
- **migration v25** — creates the `dream_verdicts` table:
|
||||
`(file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB,
|
||||
judged_at TIMESTAMPTZ, PRIMARY KEY(file_path, content_hash))`. Cache
|
||||
for the cheap Haiku verdict so backfill re-runs skip already-judged
|
||||
transcripts. RLS-enabled when running as a BYPASSRLS role.
|
||||
|
||||
The migration is idempotent. Safe to re-run.
|
||||
|
||||
## What changes for existing brains
|
||||
|
||||
`gbrain dream` (and `gbrain autopilot`) now run an 8-phase cycle:
|
||||
|
||||
```
|
||||
lint → backlinks → sync → synthesize → extract → patterns → embed → orphans
|
||||
```
|
||||
|
||||
If `dream.synthesize.enabled` is false (the default, post-migration), the
|
||||
synthesize and patterns phases emit `status: "skipped", reason: "not_configured"`
|
||||
and the cycle continues to the next phase. **Existing autopilot users see
|
||||
zero behavior change** until they configure synthesize.
|
||||
|
||||
## To enable synthesize on your brain
|
||||
|
||||
Three steps. Take them when ready — there is no rush.
|
||||
|
||||
```bash
|
||||
# 1. Point at the directory where your conversation transcripts live.
|
||||
# OpenClaw stores session transcripts at memory/.dreams/session-corpus/<YYYY-MM-DD>.txt
|
||||
# by default. If you have a different layout, point at that.
|
||||
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
|
||||
|
||||
# 2. Enable the phase.
|
||||
gbrain config set dream.synthesize.enabled true
|
||||
|
||||
# 3. Preview without spending real LLM tokens (runs cheap Haiku verdict only).
|
||||
gbrain dream --phase synthesize --dry-run --json
|
||||
```
|
||||
|
||||
## Tunables (sensible defaults; override only if needed)
|
||||
|
||||
```bash
|
||||
# Skip transcripts shorter than this many characters (default 2000).
|
||||
gbrain config set dream.synthesize.min_chars 2000
|
||||
|
||||
# Word-boundary regex patterns to skip. Default ["medical","therapy"].
|
||||
# Each entry auto-wraps as \b<entry>\b — "medical" matches "medical advice"
|
||||
# but NOT "comedical". Pass full regex (e.g. ^therapy:) for advanced patterns.
|
||||
gbrain config set dream.synthesize.exclude_patterns '["medical","therapy"]'
|
||||
|
||||
# Synthesize model (default: claude-sonnet-4-6).
|
||||
gbrain config set dream.synthesize.model claude-sonnet-4-6
|
||||
|
||||
# Hours between synthesize runs (the v1 spend cap; default 12 → ~$1-2/day).
|
||||
gbrain config set dream.synthesize.cooldown_hours 12
|
||||
|
||||
# Patterns lookback window in days (default 30).
|
||||
gbrain config set dream.patterns.lookback_days 30
|
||||
|
||||
# Minimum distinct reflections needed to name a pattern (default 3).
|
||||
gbrain config set dream.patterns.min_evidence 3
|
||||
```
|
||||
|
||||
## Allow-list source of truth
|
||||
|
||||
The synthesize subagent's allowed write paths live in
|
||||
`skills/_brain-filing-rules.json` under `dream_synthesize_paths.globs`:
|
||||
|
||||
```json
|
||||
{
|
||||
"dream_synthesize_paths": {
|
||||
"globs": [
|
||||
"wiki/personal/reflections/*",
|
||||
"wiki/originals/*",
|
||||
"wiki/personal/patterns/*",
|
||||
"wiki/people/*",
|
||||
"dream-cycle-summaries/*"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Editing this list is the ONLY way to add a new directory the synthesizer
|
||||
can write to. The subagent's `put_page` calls are gated server-side; even
|
||||
on prompt-injection success the write is bounded to these prefixes.
|
||||
|
||||
## Slug discipline
|
||||
|
||||
Reflections: `wiki/personal/reflections/YYYY-MM-DD-<topic>-<hash[:6]>`
|
||||
Originals: `wiki/originals/ideas/YYYY-MM-DD-<idea>-<hash[:6]>`
|
||||
Patterns: `wiki/personal/patterns/<theme>`
|
||||
Summary: `dream-cycle-summaries/YYYY-MM-DD`
|
||||
|
||||
The 6-char content-hash suffix on reflections / originals means an edited
|
||||
transcript produces a NEW slug — the original reflection is preserved
|
||||
alongside the new one. No silent overwrite.
|
||||
|
||||
Lowercase alphanumeric and hyphens only. NO underscores, NO file extensions.
|
||||
|
||||
## Provenance
|
||||
|
||||
Every put_page call from the synthesize subagent shows up in
|
||||
`subagent_tool_executions` with full input. The orchestrator collects
|
||||
slugs by querying that table — NOT `pages.updated_at` — so the cycle's
|
||||
write list cannot accidentally include manual edits or sync output.
|
||||
|
||||
## What's deferred to v1.1
|
||||
|
||||
- **Auto git commit + push.** v1 writes markdown files to `brain_dir`
|
||||
but does NOT `git add` / `commit` / `push`. Either commit yourself
|
||||
or let `gbrain autopilot` handle it. v1.1 will add explicit
|
||||
--commit / --push flags with handling for dirty worktree, staged
|
||||
changes, auth failure, and non-fast-forward push.
|
||||
- **Daily token budget cap.** Cooldown alone is the spend bound at v1
|
||||
scale. If real-world telemetry surfaces a problem, v1.1 adds an
|
||||
explicit `daily_token_budget` config.
|
||||
- **Cross-modal pattern review.** Patterns currently runs against
|
||||
reflections only. Future revision could roll up across reflections,
|
||||
meetings, and timeline entries together.
|
||||
|
||||
## Verify after upgrade
|
||||
|
||||
```bash
|
||||
# Schema migration applied?
|
||||
gbrain doctor
|
||||
|
||||
# Phase ordering correct?
|
||||
gbrain dream --help # shows the 8-phase pipeline
|
||||
|
||||
# Dry-run against a single transcript (cheap Haiku call only):
|
||||
gbrain dream --phase synthesize --input /tmp/some-transcript.txt --dry-run --json
|
||||
```
|
||||
|
||||
If any step fails, file an issue with `gbrain doctor` output and the
|
||||
contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
+86
-6
@@ -39,12 +39,22 @@ interface DreamArgs {
|
||||
phase: CyclePhase | null;
|
||||
dir: string | null;
|
||||
help: boolean;
|
||||
/** v0.21: ad-hoc transcript file path; implies --phase synthesize. */
|
||||
inputFile: string | null;
|
||||
/** v0.21: restrict synthesize to a single date (YYYY-MM-DD). */
|
||||
date: string | null;
|
||||
/** v0.21: backfill range start (YYYY-MM-DD). */
|
||||
from: string | null;
|
||||
/** v0.21: backfill range end (YYYY-MM-DD). */
|
||||
to: string | null;
|
||||
}
|
||||
|
||||
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}$/;
|
||||
|
||||
function parseArgs(args: string[]): DreamArgs {
|
||||
const phaseIdx = args.indexOf('--phase');
|
||||
const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null;
|
||||
const phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
|
||||
let phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
|
||||
? (rawPhase as CyclePhase)
|
||||
: null;
|
||||
if (rawPhase && !phase) {
|
||||
@@ -55,6 +65,44 @@ function parseArgs(args: string[]): DreamArgs {
|
||||
const dirIdx = args.indexOf('--dir');
|
||||
const dir = dirIdx !== -1 ? args[dirIdx + 1] : null;
|
||||
|
||||
const inputIdx = args.indexOf('--input');
|
||||
const inputFile = inputIdx !== -1 ? args[inputIdx + 1] ?? null : null;
|
||||
|
||||
const dateIdx = args.indexOf('--date');
|
||||
const date = dateIdx !== -1 ? args[dateIdx + 1] ?? null : null;
|
||||
if (date && !ISO_DATE_RE.test(date)) {
|
||||
console.error(`--date must be YYYY-MM-DD; got "${date}"`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const fromIdx = args.indexOf('--from');
|
||||
const from = fromIdx !== -1 ? args[fromIdx + 1] ?? null : null;
|
||||
if (from && !ISO_DATE_RE.test(from)) {
|
||||
console.error(`--from must be YYYY-MM-DD; got "${from}"`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const toIdx = args.indexOf('--to');
|
||||
const to = toIdx !== -1 ? args[toIdx + 1] ?? null : null;
|
||||
if (to && !ISO_DATE_RE.test(to)) {
|
||||
console.error(`--to must be YYYY-MM-DD; got "${to}"`);
|
||||
process.exit(2);
|
||||
}
|
||||
if (from && to && from > to) {
|
||||
console.error(`--from (${from}) is after --to (${to}); empty range`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// --input + --date / --from / --to is incoherent: --input is a single
|
||||
// file, the date filters scan a directory.
|
||||
if (inputFile && (date || from || to)) {
|
||||
console.error('--input cannot be combined with --date / --from / --to');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// --input implies --phase synthesize.
|
||||
if (inputFile && !phase) phase = 'synthesize';
|
||||
|
||||
return {
|
||||
json: args.includes('--json'),
|
||||
dryRun: args.includes('--dry-run'),
|
||||
@@ -62,6 +110,10 @@ function parseArgs(args: string[]): DreamArgs {
|
||||
phase,
|
||||
dir,
|
||||
help: args.includes('--help') || args.includes('-h'),
|
||||
inputFile,
|
||||
date,
|
||||
from,
|
||||
to,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -104,23 +156,43 @@ async function resolveBrainDir(
|
||||
function printHelp() {
|
||||
console.log(`Usage: gbrain dream [options]
|
||||
|
||||
Run one brain maintenance cycle: lint, backlinks, orphan sweep, sync,
|
||||
extract, and embed. Designed for cron (exits when done).
|
||||
Run one brain maintenance cycle. Eight phases:
|
||||
lint -> backlinks -> sync -> synthesize -> extract -> patterns -> embed -> orphans
|
||||
|
||||
The synthesize + patterns phases (v0.21) consolidate yesterday's
|
||||
conversation transcripts into reflections, originals, and cross-session
|
||||
pattern pages. Designed for cron (exits when done).
|
||||
|
||||
Options:
|
||||
--dry-run Preview all fixes without writing (fs or DB)
|
||||
--dry-run Preview all fixes without writing. Note: synthesize
|
||||
runs the cheap Haiku significance filter (caches
|
||||
verdicts), but skips the Sonnet synthesis pass.
|
||||
"--dry-run" does NOT mean "zero LLM calls."
|
||||
--json Emit the CycleReport as JSON (agent-readable)
|
||||
--phase <name> Run a single phase: ${ALL_PHASES.join(' | ')}
|
||||
--pull git pull the brain repo before syncing (default: no pull)
|
||||
--dir <path> Brain directory (default: configured brain)
|
||||
|
||||
--input <file> Synthesize a specific transcript file (implies
|
||||
--phase synthesize). Bypasses corpus-dir scan.
|
||||
--date YYYY-MM-DD Synthesize transcripts dated for one specific day.
|
||||
--from YYYY-MM-DD Backfill range start (use with --to).
|
||||
--to YYYY-MM-DD Backfill range end.
|
||||
|
||||
--help, -h Show this help
|
||||
|
||||
Examples:
|
||||
gbrain dream
|
||||
gbrain dream --dry-run --json
|
||||
gbrain dream --phase lint
|
||||
gbrain dream --phase synthesize --input ~/transcripts/2026-04-25.txt
|
||||
gbrain dream --phase synthesize --from 2026-04-01 --to 2026-04-25
|
||||
0 2 * * * gbrain dream --json # nightly via cron
|
||||
|
||||
Configure synthesize:
|
||||
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
|
||||
gbrain config set dream.synthesize.enabled true
|
||||
|
||||
Related:
|
||||
gbrain autopilot --install # continuous maintenance as a daemon
|
||||
gbrain autopilot # same maintenance cycle, scheduled
|
||||
@@ -165,10 +237,14 @@ function printHuman(report: CycleReport) {
|
||||
const t = report.totals;
|
||||
const hasTotals =
|
||||
t.lint_fixes > 0 || t.backlinks_added > 0 || t.pages_synced > 0 ||
|
||||
t.pages_extracted > 0 || t.pages_embedded > 0 || t.orphans_found > 0;
|
||||
t.pages_extracted > 0 || t.pages_embedded > 0 || t.orphans_found > 0 ||
|
||||
t.transcripts_processed > 0 || t.synth_pages_written > 0 || t.patterns_written > 0;
|
||||
if (hasTotals) {
|
||||
console.log(
|
||||
` totals: lint=${t.lint_fixes} backlinks=${t.backlinks_added} synced=${t.pages_synced} extracted=${t.pages_extracted} embedded=${t.pages_embedded} orphans=${t.orphans_found}`,
|
||||
` totals: lint=${t.lint_fixes} backlinks=${t.backlinks_added} synced=${t.pages_synced} ` +
|
||||
`extracted=${t.pages_extracted} embedded=${t.pages_embedded} orphans=${t.orphans_found} ` +
|
||||
`synth_transcripts=${t.transcripts_processed} synth_pages=${t.synth_pages_written} ` +
|
||||
`patterns=${t.patterns_written}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -191,6 +267,10 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom
|
||||
dryRun: opts.dryRun,
|
||||
pull: opts.pull,
|
||||
phases,
|
||||
synthInputFile: opts.inputFile ?? undefined,
|
||||
synthDate: opts.date ?? undefined,
|
||||
synthFrom: opts.from ?? undefined,
|
||||
synthTo: opts.to ?? undefined,
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
|
||||
+114
-13
@@ -16,9 +16,14 @@
|
||||
* │ Phase 1: lint --fix (filesystem writes, no DB) │
|
||||
* │ Phase 2: backlinks --fix (filesystem writes, no DB) │
|
||||
* │ Phase 3: sync (DB picks up phases 1+2) │
|
||||
* │ Phase 4: extract (DB picks up links from sync) │
|
||||
* │ Phase 5: embed --stale (DB writes) │
|
||||
* │ Phase 6: orphans (DB read, report only) │
|
||||
* │ Phase 4: synthesize (v0.23: transcripts → pages) │
|
||||
* │ Phase 5: extract (DB picks up links from sync │
|
||||
* │ + synthesize) │
|
||||
* │ Phase 6: patterns (v0.23: cross-session themes; │
|
||||
* │ MUST be after extract so │
|
||||
* │ graph state is fresh) │
|
||||
* │ Phase 7: embed --stale (DB writes) │
|
||||
* │ Phase 8: orphans (DB read, report only) │
|
||||
* └───────────────────────────────────────────────────────────┘
|
||||
*
|
||||
* COORDINATION:
|
||||
@@ -47,13 +52,15 @@ import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
|
||||
|
||||
// ─── Types ─────────────────────────────────────────────────────────
|
||||
|
||||
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'extract' | 'embed' | 'orphans';
|
||||
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans';
|
||||
|
||||
export const ALL_PHASES: CyclePhase[] = [
|
||||
'lint',
|
||||
'backlinks',
|
||||
'sync',
|
||||
'synthesize',
|
||||
'extract',
|
||||
'patterns',
|
||||
'embed',
|
||||
'orphans',
|
||||
];
|
||||
@@ -61,13 +68,16 @@ export const ALL_PHASES: CyclePhase[] = [
|
||||
/**
|
||||
* Phases that mutate state (filesystem or DB) and therefore should
|
||||
* coordinate via the cycle lock. Only orphans is truly read-only
|
||||
* and skips the lock.
|
||||
* and skips the lock. patterns mutates DB (writes pattern pages) so
|
||||
* it acquires the lock; synthesize too.
|
||||
*/
|
||||
const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
|
||||
'lint',
|
||||
'backlinks',
|
||||
'sync',
|
||||
'synthesize',
|
||||
'extract',
|
||||
'patterns',
|
||||
'embed',
|
||||
]);
|
||||
|
||||
@@ -122,6 +132,12 @@ export interface CycleReport {
|
||||
pages_extracted: number;
|
||||
pages_embedded: number;
|
||||
orphans_found: number;
|
||||
/** v0.23: number of transcripts the synthesize phase processed (judged + dispatched). */
|
||||
transcripts_processed: number;
|
||||
/** v0.23: number of new reflection/original/people pages written by synthesize. */
|
||||
synth_pages_written: number;
|
||||
/** v0.23: number of pattern pages written/updated by patterns phase. */
|
||||
patterns_written: number;
|
||||
};
|
||||
}
|
||||
|
||||
@@ -142,11 +158,30 @@ export interface CycleOpts {
|
||||
*/
|
||||
yieldBetweenPhases?: () => Promise<void>;
|
||||
/**
|
||||
* AbortSignal from the Minions worker. When aborted (timeout, cancel,
|
||||
* lock-loss), runCycle bails between phases and returns a 'failed' report
|
||||
* instead of running the next phase. Without this, a timed-out
|
||||
* autopilot-cycle handler ignores the abort and runs until the worker
|
||||
* wedges (the 98-waiting-0-active incident on 2026-04-24).
|
||||
* Generic in-phase keepalive (v0.23). Long-running phases (synthesize
|
||||
* waiting on a fan-out aggregator, patterns rolling up reflections)
|
||||
* call this periodically while idle to renew the cycle-lock TTL and
|
||||
* the Minions worker job lock. Mirrors `yieldBetweenPhases` shape;
|
||||
* passing the same function for both is the common case.
|
||||
*/
|
||||
yieldDuringPhase?: () => Promise<void>;
|
||||
/**
|
||||
* Synthesize phase scope overrides (v0.23). Forwarded to runPhaseSynthesize.
|
||||
* - `synthInputFile`: ad-hoc transcript path (`gbrain dream --input <file>`).
|
||||
* - `synthDate` / `synthFrom` / `synthTo`: date filters for corpus scan.
|
||||
* Mutually exclusive with each other in CLI parsing; runner trusts the
|
||||
* caller (CLI wrapper validates).
|
||||
*/
|
||||
synthInputFile?: string;
|
||||
synthDate?: string;
|
||||
synthFrom?: string;
|
||||
synthTo?: string;
|
||||
/**
|
||||
* AbortSignal from the Minions worker (v0.22.1, #403). When aborted
|
||||
* (timeout, cancel, lock-loss), runCycle bails between phases and
|
||||
* returns a 'failed' report instead of running the next phase. Without
|
||||
* this, a timed-out autopilot-cycle handler ignores the abort and runs
|
||||
* until the worker wedges (the 98-waiting-0-active incident on 2026-04-24).
|
||||
*/
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
@@ -765,7 +800,36 @@ export async function runCycle(
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 4: extract ────────────────────────────────────────
|
||||
// ── Phase 4: synthesize (v0.23) ─────────────────────────────
|
||||
if (phases.includes('synthesize')) {
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'synthesize',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.synthesize');
|
||||
const { runPhaseSynthesize } = await import('./cycle/synthesize.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSynthesize(engine, {
|
||||
brainDir: opts.brainDir,
|
||||
dryRun,
|
||||
yieldDuringPhase: opts.yieldDuringPhase,
|
||||
inputFile: opts.synthInputFile,
|
||||
date: opts.synthDate,
|
||||
from: opts.synthFrom,
|
||||
to: opts.synthTo,
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 5: extract (now picks up synthesize output) ───────
|
||||
if (phases.includes('extract')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
@@ -789,7 +853,36 @@ export async function runCycle(
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 5: embed ──────────────────────────────────────────
|
||||
// ── Phase 6: patterns (v0.23) ───────────────────────────────
|
||||
// MUST run after extract so the graph state reads fresh — subagent
|
||||
// put_page calls in synthesize set ctx.remote=true, so auto-link
|
||||
// only fires for trusted-workspace writes (allow-listed). extract
|
||||
// is the canonical materialization step.
|
||||
if (phases.includes('patterns')) {
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'patterns',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.patterns');
|
||||
const { runPhasePatterns } = await import('./cycle/patterns.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhasePatterns(engine, {
|
||||
brainDir: opts.brainDir,
|
||||
dryRun,
|
||||
yieldDuringPhase: opts.yieldDuringPhase,
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 7: embed ──────────────────────────────────────────
|
||||
if (phases.includes('embed')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
@@ -810,7 +903,7 @@ export async function runCycle(
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 6: orphans ────────────────────────────────────────
|
||||
// ── Phase 8: orphans ────────────────────────────────────────
|
||||
if (phases.includes('orphans')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
@@ -861,6 +954,9 @@ function emptyTotals(): CycleReport['totals'] {
|
||||
pages_extracted: 0,
|
||||
pages_embedded: 0,
|
||||
orphans_found: 0,
|
||||
transcripts_processed: 0,
|
||||
synth_pages_written: 0,
|
||||
patterns_written: 0,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -883,6 +979,11 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
|
||||
: Number(p.details.embedded ?? 0);
|
||||
} else if (p.phase === 'orphans' && p.details) {
|
||||
t.orphans_found = Number(p.details.total_orphans ?? 0);
|
||||
} else if (p.phase === 'synthesize' && p.details) {
|
||||
t.transcripts_processed = Number(p.details.transcripts_processed ?? 0);
|
||||
t.synth_pages_written = Number(p.details.pages_written ?? 0);
|
||||
} else if (p.phase === 'patterns' && p.details) {
|
||||
t.patterns_written = Number(p.details.patterns_written ?? 0);
|
||||
}
|
||||
}
|
||||
return t;
|
||||
|
||||
@@ -0,0 +1,323 @@
|
||||
/**
|
||||
* Patterns phase (v0.23) — cross-session theme detection.
|
||||
*
|
||||
* Reads recent reflections (within `lookback_days`), runs a single Sonnet
|
||||
* subagent to surface themes that recur across ≥`min_evidence` distinct
|
||||
* reflections, and writes one pattern page per theme.
|
||||
*
|
||||
* MUST run after `extract` so the graph state (links, timeline) is fresh.
|
||||
* Subagent put_page calls have ctx.remote=true; the trusted-workspace
|
||||
* allow-list re-enables auto-link / auto-timeline for synth + pattern
|
||||
* writes (operations.ts:trustedWorkspace branch).
|
||||
*
|
||||
* v1 behavior:
|
||||
* - Single Sonnet subagent (no fan-out — one job per cycle is plenty).
|
||||
* - Idempotent: if reflection set is below `min_evidence`, phase is skipped.
|
||||
* - Pattern slug uses LLM's chosen topic-slug (subagent prompt instructs format).
|
||||
* - Existing pattern pages are updated in place via put_page (idempotent
|
||||
* ON CONFLICT semantics in importFromContent).
|
||||
*/
|
||||
|
||||
import { join, dirname } from 'node:path';
|
||||
import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||||
import { MinionQueue } from '../minions/queue.ts';
|
||||
import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.ts';
|
||||
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
import type { Page, PageType } from '../types.ts';
|
||||
|
||||
export interface PatternsPhaseOpts {
|
||||
brainDir: string;
|
||||
dryRun: boolean;
|
||||
yieldDuringPhase?: () => Promise<void>;
|
||||
}
|
||||
|
||||
export async function runPhasePatterns(
|
||||
engine: BrainEngine,
|
||||
opts: PatternsPhaseOpts,
|
||||
): Promise<PhaseResult> {
|
||||
const start = Date.now();
|
||||
try {
|
||||
const config = await loadPatternsConfig(engine);
|
||||
|
||||
if (!config.enabled) {
|
||||
return skipped('disabled', 'dream.patterns.enabled is false');
|
||||
}
|
||||
|
||||
// Gather reflections within lookback window.
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays);
|
||||
if (reflections.length < config.minEvidence) {
|
||||
return skipped(
|
||||
'insufficient_evidence',
|
||||
`${reflections.length} reflections in last ${config.lookbackDays}d (need ≥${config.minEvidence})`,
|
||||
);
|
||||
}
|
||||
|
||||
if (opts.dryRun) {
|
||||
return ok(`dry-run: would detect patterns over ${reflections.length} reflections`, {
|
||||
reflections_considered: reflections.length,
|
||||
patterns_written: 0,
|
||||
dryRun: true,
|
||||
});
|
||||
}
|
||||
|
||||
// Submit one subagent for pattern detection.
|
||||
if (!process.env.ANTHROPIC_API_KEY) {
|
||||
return skipped('no_api_key', 'ANTHROPIC_API_KEY unset; pattern detection skipped');
|
||||
}
|
||||
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
}
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
const data: SubagentHandlerData = {
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
};
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
timeout_ms: 30 * 60 * 1000,
|
||||
};
|
||||
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
|
||||
allowProtectedSubmit: true,
|
||||
});
|
||||
|
||||
let outcome: string;
|
||||
try {
|
||||
const final = await waitForCompletion(queue, job.id, {
|
||||
timeoutMs: 35 * 60 * 1000,
|
||||
pollMs: 5 * 1000,
|
||||
});
|
||||
outcome = final.status;
|
||||
} catch (e) {
|
||||
if (e instanceof TimeoutError) outcome = 'timeout';
|
||||
else throw e;
|
||||
}
|
||||
|
||||
if (opts.yieldDuringPhase) {
|
||||
try { await opts.yieldDuringPhase(); } catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
// Collect slugs the subagent wrote (codex finding #2 — query tool exec rows).
|
||||
const writtenSlugs = await collectChildPutPageSlugs(engine, [job.id]);
|
||||
|
||||
// Reverse-write to fs.
|
||||
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
|
||||
|
||||
return ok(`${writtenSlugs.length} pattern page(s) written/updated (${outcome})`, {
|
||||
reflections_considered: reflections.length,
|
||||
patterns_written: writtenSlugs.length,
|
||||
reverse_write_count: reverseWriteCount,
|
||||
child_outcome: outcome,
|
||||
job_id: job.id,
|
||||
});
|
||||
} catch (e) {
|
||||
return failed(makeError('InternalError', 'PATTERNS_PHASE_FAIL',
|
||||
e instanceof Error ? (e.message || 'patterns phase threw') : String(e)));
|
||||
} finally {
|
||||
void start;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Config ────────────────────────────────────────────────────────────
|
||||
|
||||
interface PatternsConfig {
|
||||
enabled: boolean;
|
||||
lookbackDays: number;
|
||||
minEvidence: number;
|
||||
model: string;
|
||||
}
|
||||
|
||||
async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig> {
|
||||
const enabledStr = await engine.getConfig('dream.patterns.enabled');
|
||||
const enabled = enabledStr === null ? true : enabledStr === 'true';
|
||||
const lookbackStr = await engine.getConfig('dream.patterns.lookback_days');
|
||||
const minEvidenceStr = await engine.getConfig('dream.patterns.min_evidence');
|
||||
const model = (await engine.getConfig('dream.patterns.model')) || 'claude-sonnet-4-6';
|
||||
return {
|
||||
enabled,
|
||||
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
|
||||
minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3,
|
||||
model,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Reflection gathering ─────────────────────────────────────────────
|
||||
|
||||
interface ReflectionRef {
|
||||
slug: string;
|
||||
title: string;
|
||||
excerpt: string;
|
||||
}
|
||||
|
||||
async function gatherReflections(
|
||||
engine: BrainEngine,
|
||||
lookbackDays: number,
|
||||
): Promise<ReflectionRef[]> {
|
||||
const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString();
|
||||
const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>(
|
||||
`SELECT slug, title, compiled_truth
|
||||
FROM pages
|
||||
WHERE slug LIKE 'wiki/personal/reflections/%'
|
||||
AND updated_at >= $1::timestamptz
|
||||
ORDER BY updated_at DESC
|
||||
LIMIT 100`,
|
||||
[since],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
slug: r.slug,
|
||||
title: r.title ?? r.slug,
|
||||
excerpt: (r.compiled_truth ?? '').slice(0, 600),
|
||||
}));
|
||||
}
|
||||
|
||||
// ── Prompt ────────────────────────────────────────────────────────────
|
||||
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string {
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const corpus = reflections
|
||||
.map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`)
|
||||
.join('\n\n---\n\n');
|
||||
|
||||
return `You are surfacing recurring themes across the user's recent reflections.
|
||||
|
||||
OUTPUT POLICY
|
||||
- Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections.
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks).
|
||||
- Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one.
|
||||
- Pattern slug format: \`wiki/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics.
|
||||
|
||||
DO NOT WRITE
|
||||
- A "patterns from today" digest (that's the dream-cycle-summaries page; not your job).
|
||||
- Patterns with <${minEvidence} reflections cited.
|
||||
- Anything outside wiki/personal/patterns/.
|
||||
|
||||
CONTEXT
|
||||
- Today: ${today}
|
||||
- Reflections in scope: ${reflections.length}
|
||||
|
||||
REFLECTIONS
|
||||
${corpus}
|
||||
|
||||
When done, briefly list the pattern slugs you wrote/updated in your final message.`;
|
||||
}
|
||||
|
||||
// ── Provenance via put_page tool execution rows ─────────────────────
|
||||
|
||||
async function collectChildPutPageSlugs(
|
||||
engine: BrainEngine,
|
||||
childIds: number[],
|
||||
): Promise<string[]> {
|
||||
if (childIds.length === 0) return [];
|
||||
const rows = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT DISTINCT input->>'slug' AS slug
|
||||
FROM subagent_tool_executions
|
||||
WHERE job_id = ANY($1::int[])
|
||||
AND tool_name = 'brain_put_page'
|
||||
AND status = 'complete'
|
||||
AND input ? 'slug'
|
||||
ORDER BY 1`,
|
||||
[childIds],
|
||||
);
|
||||
return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0);
|
||||
}
|
||||
|
||||
// ── Reverse-write ────────────────────────────────────────────────────
|
||||
|
||||
async function reverseWriteSlugs(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
slugs: string[],
|
||||
): Promise<number> {
|
||||
let count = 0;
|
||||
for (const slug of slugs) {
|
||||
const page = await engine.getPage(slug);
|
||||
if (!page) continue;
|
||||
const tags = await engine.getTags(slug);
|
||||
try {
|
||||
const md = renderPageToMarkdown(page, tags);
|
||||
const filePath = join(brainDir, `${slug}.md`);
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
writeFileSync(filePath, md, 'utf8');
|
||||
count++;
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[dream] reverse-write ${slug} failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
function renderPageToMarkdown(page: Page, tags: string[]): string {
|
||||
const frontmatter = (page.frontmatter ?? {}) as Record<string, unknown>;
|
||||
return serializeMarkdown(
|
||||
frontmatter,
|
||||
page.compiled_truth ?? '',
|
||||
page.timeline ?? '',
|
||||
{
|
||||
type: (page.type as PageType) ?? 'note',
|
||||
title: page.title ?? '',
|
||||
tags,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
// ── Allow-list (shared with synthesize.ts) ───────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
const candidates = [
|
||||
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
|
||||
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
|
||||
];
|
||||
for (const path of candidates) {
|
||||
if (!existsSync(path)) continue;
|
||||
try {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
// ── Status helpers ───────────────────────────────────────────────────
|
||||
|
||||
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
|
||||
return { phase: 'patterns', status: 'ok', duration_ms: 0, summary, details };
|
||||
}
|
||||
|
||||
function skipped(reason: string, summary: string): PhaseResult {
|
||||
return {
|
||||
phase: 'patterns',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary,
|
||||
details: { reason },
|
||||
};
|
||||
}
|
||||
|
||||
function failed(error: PhaseError): PhaseResult {
|
||||
return {
|
||||
phase: 'patterns',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'patterns phase failed',
|
||||
details: {},
|
||||
error,
|
||||
};
|
||||
}
|
||||
|
||||
function makeError(cls: string, code: string, message: string, hint?: string): PhaseError {
|
||||
return hint ? { class: cls, code, message, hint } : { class: cls, code, message };
|
||||
}
|
||||
@@ -0,0 +1,604 @@
|
||||
/**
|
||||
* Synthesize phase (v0.23) — conversation-to-brain pipeline.
|
||||
*
|
||||
* Reads transcripts from the configured corpus dir, runs a cheap Haiku
|
||||
* "is this worth processing?" verdict (cached in `dream_verdicts`), then
|
||||
* fans out one Sonnet subagent per worth-processing transcript with the
|
||||
* trusted-workspace `allowed_slug_prefixes` list. After children resolve,
|
||||
* the orchestrator queries `subagent_tool_executions` for the put_page
|
||||
* slugs each child wrote (codex finding #2: NOT a time-windowed pages
|
||||
* query — picks up unrelated writes), reverse-renders each new page from
|
||||
* DB to disk, and writes a deterministic summary index.
|
||||
*
|
||||
* Hard guarantees:
|
||||
* - Subagent never gets fs-write access. Orchestrator holds the dual-write.
|
||||
* - Allow-list is sourced from `skills/_brain-filing-rules.json` (single
|
||||
* source of truth) and threaded as handler data; PROTECTED_JOB_NAMES
|
||||
* prevents MCP from submitting `subagent` jobs, so the field is trusted.
|
||||
* - Cooldown via `dream.synthesize.last_completion_ts` config key —
|
||||
* written ONLY on success (codex finding #5 deferral: no auto git commit
|
||||
* in v1).
|
||||
* - Idempotency via `dream:synth:<file_path>:<content_hash>` job key.
|
||||
* - Edited transcripts produce slugs with content-hash suffix → no overwrite.
|
||||
*
|
||||
* NOT in v1:
|
||||
* - git auto-commit / push (deferred to v1.1, codex finding #5).
|
||||
* - Daily token budget cap (cooldown bounds spend at v1 scale).
|
||||
*/
|
||||
|
||||
import Anthropic from '@anthropic-ai/sdk';
|
||||
import { readFileSync, existsSync, writeFileSync, mkdirSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||||
import { MinionQueue } from '../minions/queue.ts';
|
||||
import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.ts';
|
||||
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
|
||||
import { discoverTranscripts, type DiscoveredTranscript } from './transcript-discovery.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
import type { Page, PageType } from '../types.ts';
|
||||
|
||||
// Slug regex from validatePageSlug — kept in sync.
|
||||
// Used for the orchestrator-written summary index slug.
|
||||
const SUMMARY_SLUG_RE = /^[a-z0-9][a-z0-9\-]*(\/[a-z0-9][a-z0-9\-]*)*$/;
|
||||
|
||||
// ── Public entry ──────────────────────────────────────────────────────
|
||||
|
||||
export interface SynthesizePhaseOpts {
|
||||
brainDir: string;
|
||||
dryRun: boolean;
|
||||
/** Generic in-cycle keepalive for cycle-lock TTL renewal during long waits. */
|
||||
yieldDuringPhase?: () => Promise<void>;
|
||||
/**
|
||||
* Override the corpus directory and other tunables. Primarily for the
|
||||
* `gbrain dream --input <file>` ad-hoc path; bypasses config reads.
|
||||
*/
|
||||
inputFile?: string;
|
||||
date?: string;
|
||||
from?: string;
|
||||
to?: string;
|
||||
}
|
||||
|
||||
export async function runPhaseSynthesize(
|
||||
engine: BrainEngine,
|
||||
opts: SynthesizePhaseOpts,
|
||||
): Promise<PhaseResult> {
|
||||
const start = Date.now();
|
||||
try {
|
||||
const config = await loadSynthConfig(engine);
|
||||
|
||||
// Allow ad-hoc --input to run even when config is disabled.
|
||||
if (!opts.inputFile && !config.enabled) {
|
||||
return skipped('not_configured',
|
||||
'dream.synthesize.enabled is false (set dream.synthesize.session_corpus_dir to enable)');
|
||||
}
|
||||
if (!opts.inputFile && !config.corpusDir) {
|
||||
return skipped('not_configured',
|
||||
'dream.synthesize.session_corpus_dir is unset');
|
||||
}
|
||||
|
||||
// Cooldown check (skipped for explicit --input / --date / --from / --to runs).
|
||||
const explicitTarget = opts.inputFile || opts.date || opts.from || opts.to;
|
||||
if (!explicitTarget) {
|
||||
const cooldown = await checkCooldown(engine, config.cooldownHours);
|
||||
if (cooldown.active) {
|
||||
return skipped('cooldown_active',
|
||||
`synthesize cooled down until ${cooldown.expires_at} (${config.cooldownHours}h cooldown)`);
|
||||
}
|
||||
}
|
||||
|
||||
// Discover.
|
||||
const transcripts = opts.inputFile
|
||||
? loadAdHocTranscript(opts.inputFile, config.minChars, config.excludePatterns)
|
||||
: discoverTranscripts({
|
||||
corpusDir: config.corpusDir!,
|
||||
meetingTranscriptsDir: config.meetingTranscriptsDir ?? undefined,
|
||||
minChars: config.minChars,
|
||||
excludePatterns: config.excludePatterns,
|
||||
date: opts.date,
|
||||
from: opts.from,
|
||||
to: opts.to,
|
||||
});
|
||||
|
||||
if (transcripts.length === 0) {
|
||||
return ok('no transcripts to process', { transcripts_processed: 0, pages_written: 0 });
|
||||
}
|
||||
|
||||
// Significance verdicts (cached in dream_verdicts; Haiku on miss).
|
||||
const worthProcessing: DiscoveredTranscript[] = [];
|
||||
const verdicts: Array<{ filePath: string; worth: boolean; reasons: string[]; cached: boolean }> = [];
|
||||
const haiku = makeHaikuClient(); // null if no API key
|
||||
for (const t of transcripts) {
|
||||
const cached = await engine.getDreamVerdict(t.filePath, t.contentHash);
|
||||
if (cached) {
|
||||
verdicts.push({ filePath: t.filePath, worth: cached.worth_processing, reasons: cached.reasons, cached: true });
|
||||
if (cached.worth_processing) worthProcessing.push(t);
|
||||
continue;
|
||||
}
|
||||
if (!haiku) {
|
||||
// No API key — can't judge. Skip with explicit reason; don't crash phase.
|
||||
verdicts.push({ filePath: t.filePath, worth: false, reasons: ['no ANTHROPIC_API_KEY for significance judge'], cached: false });
|
||||
continue;
|
||||
}
|
||||
const verdict = await judgeSignificance(haiku, t);
|
||||
await engine.putDreamVerdict(t.filePath, t.contentHash, verdict);
|
||||
verdicts.push({ filePath: t.filePath, worth: verdict.worth_processing, reasons: verdict.reasons, cached: false });
|
||||
if (verdict.worth_processing) worthProcessing.push(t);
|
||||
}
|
||||
|
||||
// Dry-run stops here: significance filter ran (Haiku verdicts cached),
|
||||
// but no Sonnet synthesis. Codex finding #8: --dry-run does NOT mean
|
||||
// "zero LLM calls"; it means "skip Sonnet."
|
||||
if (opts.dryRun) {
|
||||
return ok(`dry-run: ${worthProcessing.length} of ${transcripts.length} transcripts would synthesize`, {
|
||||
transcripts_discovered: transcripts.length,
|
||||
transcripts_processed: 0,
|
||||
pages_written: 0,
|
||||
verdicts,
|
||||
dryRun: true,
|
||||
});
|
||||
}
|
||||
|
||||
if (worthProcessing.length === 0) {
|
||||
// Even with verdicts, the cooldown timestamp is updated only on a
|
||||
// real successful run — not on "nothing worth processing." Lets a
|
||||
// re-run pick up if a new transcript lands later.
|
||||
return ok('all transcripts skipped by significance filter', {
|
||||
transcripts_discovered: transcripts.length,
|
||||
transcripts_processed: 0,
|
||||
pages_written: 0,
|
||||
verdicts,
|
||||
});
|
||||
}
|
||||
|
||||
// Fan-out: submit one subagent per worth-processing transcript.
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
}
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
const childIds: number[] = [];
|
||||
for (const t of worthProcessing) {
|
||||
const childData: SubagentHandlerData = {
|
||||
prompt: buildSynthesisPrompt(t),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
};
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
on_child_fail: 'continue',
|
||||
idempotency_key: `dream:synth:${t.filePath}:${t.contentHash.slice(0, 16)}`,
|
||||
timeout_ms: 30 * 60 * 1000, // 30 min per transcript
|
||||
};
|
||||
const child = await queue.add(
|
||||
'subagent',
|
||||
childData as unknown as Record<string, unknown>,
|
||||
submitOpts,
|
||||
{ allowProtectedSubmit: true },
|
||||
);
|
||||
childIds.push(child.id);
|
||||
}
|
||||
|
||||
// Wait for every child to reach a terminal state. Tick yieldDuringPhase
|
||||
// every 5 min so the cycle lock TTL refreshes.
|
||||
const childOutcomes: Array<{ jobId: number; status: string }> = [];
|
||||
for (const jobId of childIds) {
|
||||
try {
|
||||
const job = await waitForCompletion(queue, jobId, {
|
||||
timeoutMs: 35 * 60 * 1000,
|
||||
pollMs: 5 * 1000,
|
||||
});
|
||||
childOutcomes.push({ jobId, status: job.status });
|
||||
} catch (e) {
|
||||
if (e instanceof TimeoutError) {
|
||||
childOutcomes.push({ jobId, status: 'timeout' });
|
||||
} else {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
// After each child terminal, give the cycle lock + worker job lock a chance.
|
||||
if (opts.yieldDuringPhase) {
|
||||
try { await opts.yieldDuringPhase(); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
|
||||
// Collect slugs from put_page tool executions across the children
|
||||
// (codex finding #2: deterministic provenance, NOT pages.updated_at).
|
||||
const writtenSlugs = await collectChildPutPageSlugs(engine, childIds);
|
||||
|
||||
// Dual-write: reverse-render each DB row → markdown file.
|
||||
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
|
||||
|
||||
// Summary index page (deterministic; orchestrator-written via direct
|
||||
// engine.putPage so no allow-list path needed).
|
||||
const summaryDate = opts.date ?? today();
|
||||
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
|
||||
if (SUMMARY_SLUG_RE.test(summarySlug)) {
|
||||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
|
||||
}
|
||||
|
||||
// Write completion timestamp ON SUCCESS only.
|
||||
await engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString());
|
||||
|
||||
const ms = Date.now() - start;
|
||||
return ok(`${worthProcessing.length} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, {
|
||||
transcripts_discovered: transcripts.length,
|
||||
transcripts_processed: worthProcessing.length,
|
||||
pages_written: writtenSlugs.length,
|
||||
reverse_write_count: reverseWriteCount,
|
||||
child_outcomes: childOutcomes,
|
||||
summary_slug: summarySlug,
|
||||
verdicts,
|
||||
});
|
||||
} catch (e) {
|
||||
return failed(makeError('InternalError', 'SYNTH_PHASE_FAIL',
|
||||
e instanceof Error ? (e.message || 'synthesize phase threw') : String(e)));
|
||||
}
|
||||
}
|
||||
|
||||
// ── Config ────────────────────────────────────────────────────────────
|
||||
|
||||
interface SynthConfig {
|
||||
enabled: boolean;
|
||||
corpusDir: string | null;
|
||||
meetingTranscriptsDir: string | null;
|
||||
minChars: number;
|
||||
excludePatterns: string[];
|
||||
model: string;
|
||||
cooldownHours: number;
|
||||
}
|
||||
|
||||
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
const enabled = (await engine.getConfig('dream.synthesize.enabled')) === 'true';
|
||||
const corpusDir = await engine.getConfig('dream.synthesize.session_corpus_dir');
|
||||
const meetingTranscriptsDir = await engine.getConfig('dream.synthesize.meeting_transcripts_dir');
|
||||
const minCharsStr = await engine.getConfig('dream.synthesize.min_chars');
|
||||
const excludeStr = await engine.getConfig('dream.synthesize.exclude_patterns');
|
||||
const model = (await engine.getConfig('dream.synthesize.model')) || 'claude-sonnet-4-6';
|
||||
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
|
||||
|
||||
let excludePatterns: string[] = ['medical', 'therapy'];
|
||||
if (excludeStr) {
|
||||
try {
|
||||
const parsed = JSON.parse(excludeStr);
|
||||
if (Array.isArray(parsed)) excludePatterns = parsed.filter(p => typeof p === 'string');
|
||||
} catch { /* keep default */ }
|
||||
}
|
||||
|
||||
return {
|
||||
enabled,
|
||||
corpusDir: corpusDir ?? null,
|
||||
meetingTranscriptsDir: meetingTranscriptsDir ?? null,
|
||||
minChars: minCharsStr ? Math.max(0, parseInt(minCharsStr, 10) || 2000) : 2000,
|
||||
excludePatterns,
|
||||
model,
|
||||
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
|
||||
};
|
||||
}
|
||||
|
||||
async function checkCooldown(
|
||||
engine: BrainEngine,
|
||||
hours: number,
|
||||
): Promise<{ active: boolean; expires_at?: string }> {
|
||||
if (hours <= 0) return { active: false };
|
||||
const last = await engine.getConfig('dream.synthesize.last_completion_ts');
|
||||
if (!last) return { active: false };
|
||||
const lastMs = Date.parse(last);
|
||||
if (Number.isNaN(lastMs)) return { active: false };
|
||||
const expiresMs = lastMs + hours * 60 * 60 * 1000;
|
||||
if (Date.now() >= expiresMs) return { active: false };
|
||||
return { active: true, expires_at: new Date(expiresMs).toISOString() };
|
||||
}
|
||||
|
||||
// ── Allow-list source of truth ───────────────────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
// Search a few known locations relative to the binary / repo. The first
|
||||
// hit wins; if none found, return [].
|
||||
const candidates = [
|
||||
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
|
||||
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
|
||||
];
|
||||
for (const path of candidates) {
|
||||
if (!existsSync(path)) continue;
|
||||
try {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
// ── Significance judge (Haiku) ───────────────────────────────────────
|
||||
|
||||
interface JudgeClient {
|
||||
create: (params: Anthropic.MessageCreateParamsNonStreaming) => Promise<Anthropic.Message>;
|
||||
}
|
||||
|
||||
function makeHaikuClient(): JudgeClient | null {
|
||||
if (!process.env.ANTHROPIC_API_KEY) return null;
|
||||
const client = new Anthropic();
|
||||
return { create: client.messages.create.bind(client.messages) };
|
||||
}
|
||||
|
||||
interface VerdictResult {
|
||||
worth_processing: boolean;
|
||||
reasons: string[];
|
||||
}
|
||||
|
||||
async function judgeSignificance(
|
||||
client: JudgeClient,
|
||||
t: DiscoveredTranscript,
|
||||
): Promise<VerdictResult> {
|
||||
// Truncate the transcript at 8K chars for cost control. Haiku's verdict
|
||||
// doesn't need the full body; the opening + closing sections are usually
|
||||
// representative of significance.
|
||||
const trimmed = t.content.length > 8000
|
||||
? t.content.slice(0, 4000) + '\n[...truncated...]\n' + t.content.slice(-4000)
|
||||
: t.content;
|
||||
|
||||
const sys = `You judge whether a conversation transcript is worth synthesizing into a personal knowledge brain.
|
||||
|
||||
WORTH PROCESSING (return worth_processing=true):
|
||||
- The user articulates a new idea, frame, mental model, or thesis
|
||||
- The user reflects on themselves, names patterns, processes emotion
|
||||
- The user discusses specific people, companies, or decisions in depth
|
||||
- The user makes a strategic call worth remembering
|
||||
|
||||
NOT WORTH PROCESSING (return worth_processing=false):
|
||||
- Routine ops ("check my email", "schedule X")
|
||||
- Pure code debugging without user reflection
|
||||
- Short message exchanges with no original thought
|
||||
- Repetitive content the brain already has
|
||||
|
||||
Respond as JSON: {"worth_processing": <bool>, "reasons": ["<short>", "<short>"]}.
|
||||
Two reasons max, one phrase each.`;
|
||||
|
||||
const msg = await client.create({
|
||||
model: 'claude-haiku-4-5-20251001',
|
||||
max_tokens: 200,
|
||||
system: sys,
|
||||
messages: [{ role: 'user', content: `Transcript ${t.basename}:\n\n${trimmed}` }],
|
||||
});
|
||||
|
||||
for (const block of msg.content) {
|
||||
if (block.type === 'text') {
|
||||
const text = block.text.trim();
|
||||
const m = /\{[\s\S]*\}/.exec(text);
|
||||
if (!m) continue;
|
||||
try {
|
||||
const parsed = JSON.parse(m[0]) as { worth_processing?: unknown; reasons?: unknown };
|
||||
const worth = parsed.worth_processing === true;
|
||||
const reasons = Array.isArray(parsed.reasons)
|
||||
? parsed.reasons.filter((r): r is string => typeof r === 'string').slice(0, 4)
|
||||
: [];
|
||||
return { worth_processing: worth, reasons };
|
||||
} catch { /* fall through */ }
|
||||
}
|
||||
}
|
||||
// Couldn't parse — default to NOT processing (cheap fallback).
|
||||
return { worth_processing: false, reasons: ['judge response unparseable'] };
|
||||
}
|
||||
|
||||
// ── Subagent prompt ──────────────────────────────────────────────────
|
||||
|
||||
function buildSynthesisPrompt(t: DiscoveredTranscript): string {
|
||||
const dateHint = t.inferredDate ?? today();
|
||||
const hashSuffix = t.contentHash.slice(0, 6);
|
||||
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
|
||||
return `You are synthesizing a conversation transcript into the user's personal knowledge brain.
|
||||
|
||||
CONTEXT
|
||||
- Today's date: ${dateHint}
|
||||
- Transcript hash suffix (USE THIS in slugs): ${hashSuffix}
|
||||
- Source file basename: ${baseSlugSegment}
|
||||
|
||||
OUTPUT POLICY (ALL of these are required)
|
||||
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
|
||||
2. Cross-reference compulsively: every new page MUST contain at least one wikilink (e.g., \`[ref](people/jane-doe)\` or \`[[people/jane-doe]]\`) to existing brain content. Use the search tool to find existing pages first.
|
||||
3. Do NOT write to any path outside the allow-list shown in the put_page schema.
|
||||
4. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated segments. NO underscores, NO file extensions.
|
||||
|
||||
TASKS
|
||||
A. Reflections (self-knowledge, pattern recognition, emotional processing):
|
||||
slug: \`wiki/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||||
|
||||
B. Originals (new ideas, frames, theses, mental models):
|
||||
slug: \`wiki/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||||
|
||||
C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries — your job is the reflection/original synthesis, NOT modifying existing person pages).
|
||||
|
||||
D. If nothing in this transcript meets the bar (significance filter already passed but the content is still routine), return without writing anything.
|
||||
|
||||
TRANSCRIPT (${t.filePath})
|
||||
---
|
||||
${t.content}
|
||||
---
|
||||
|
||||
When done, briefly list the slugs you wrote in your final message so the orchestrator can audit.`;
|
||||
}
|
||||
|
||||
function sanitizeForSlug(s: string): string {
|
||||
return s
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, '-')
|
||||
.replace(/^-+|-+$/g, '')
|
||||
.slice(0, 60);
|
||||
}
|
||||
|
||||
// ── Slug collection from child put_page calls (codex #2) ────────────
|
||||
|
||||
async function collectChildPutPageSlugs(
|
||||
engine: BrainEngine,
|
||||
childIds: number[],
|
||||
): Promise<string[]> {
|
||||
if (childIds.length === 0) return [];
|
||||
const rows = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT DISTINCT input->>'slug' AS slug
|
||||
FROM subagent_tool_executions
|
||||
WHERE job_id = ANY($1::int[])
|
||||
AND tool_name = 'brain_put_page'
|
||||
AND status = 'complete'
|
||||
AND input ? 'slug'
|
||||
ORDER BY 1`,
|
||||
[childIds],
|
||||
);
|
||||
return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0);
|
||||
}
|
||||
|
||||
// ── Reverse-write DB rows → markdown files ───────────────────────────
|
||||
|
||||
async function reverseWriteSlugs(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
slugs: string[],
|
||||
): Promise<number> {
|
||||
let count = 0;
|
||||
for (const slug of slugs) {
|
||||
const page = await engine.getPage(slug);
|
||||
if (!page) continue;
|
||||
const tags = await engine.getTags(slug);
|
||||
try {
|
||||
const md = renderPageToMarkdown(page, tags);
|
||||
const filePath = join(brainDir, `${slug}.md`);
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
writeFileSync(filePath, md, 'utf8');
|
||||
count++;
|
||||
} catch (e) {
|
||||
// Per-slug failures are non-fatal — phase continues.
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[dream] reverse-write ${slug} failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
function renderPageToMarkdown(page: Page, tags: string[]): string {
|
||||
// serializeMarkdown's contract: takes (frontmatter, compiled_truth, timeline, meta)
|
||||
// and emits frontmatter + body + (optional) timeline section.
|
||||
const frontmatter = (page.frontmatter ?? {}) as Record<string, unknown>;
|
||||
return serializeMarkdown(
|
||||
frontmatter,
|
||||
page.compiled_truth ?? '',
|
||||
page.timeline ?? '',
|
||||
{
|
||||
type: (page.type as PageType) ?? 'note',
|
||||
title: page.title ?? '',
|
||||
tags,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
// ── Summary index page ───────────────────────────────────────────────
|
||||
|
||||
async function writeSummaryPage(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
summarySlug: string,
|
||||
summaryDate: string,
|
||||
writtenSlugs: string[],
|
||||
childOutcomes: Array<{ jobId: number; status: string }>,
|
||||
): Promise<void> {
|
||||
const completed = childOutcomes.filter(c => c.status === 'completed').length;
|
||||
const failed = childOutcomes.length - completed;
|
||||
|
||||
const lines: string[] = [];
|
||||
lines.push(`# Dream cycle ${summaryDate}`);
|
||||
lines.push('');
|
||||
lines.push(`**Children:** ${completed} completed, ${failed} failed/timeout.`);
|
||||
lines.push(`**Pages written:** ${writtenSlugs.length}.`);
|
||||
lines.push('');
|
||||
if (writtenSlugs.length > 0) {
|
||||
lines.push('## Pages');
|
||||
lines.push('');
|
||||
for (const s of writtenSlugs) {
|
||||
lines.push(`- [[${s}]]`);
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
|
||||
const body = lines.join('\n');
|
||||
const fullMarkdown = serializeMarkdown(
|
||||
{} as Record<string, unknown>,
|
||||
body,
|
||||
'',
|
||||
{ type: 'note' as PageType, title: `Dream cycle ${summaryDate}`, tags: ['dream-cycle'] },
|
||||
);
|
||||
|
||||
// Direct engine.putPage — orchestrator write, no subagent context, no
|
||||
// allow-list check (server-side viaSubagent=false). The summary slug is
|
||||
// pre-validated against SUMMARY_SLUG_RE in the caller.
|
||||
// Importing put_page via operations.ts would re-run namespace logic
|
||||
// unnecessarily; we go straight to the engine.
|
||||
const { parseMarkdown } = await import('../markdown.ts');
|
||||
const parsed = parseMarkdown(fullMarkdown);
|
||||
await engine.putPage(summarySlug, {
|
||||
type: parsed.type,
|
||||
title: parsed.title,
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline,
|
||||
frontmatter: parsed.frontmatter,
|
||||
});
|
||||
|
||||
// Also write to disk (orchestrator dual-write).
|
||||
try {
|
||||
const filePath = join(brainDir, `${summarySlug}.md`);
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
writeFileSync(filePath, fullMarkdown, 'utf8');
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[dream] summary file-write failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────────────────
|
||||
|
||||
function loadAdHocTranscript(
|
||||
filePath: string,
|
||||
minChars: number,
|
||||
excludePatterns: string[],
|
||||
): DiscoveredTranscript[] {
|
||||
const { readSingleTranscript } = require('./transcript-discovery.ts') as typeof import('./transcript-discovery.ts');
|
||||
const t = readSingleTranscript(filePath, { minChars, excludePatterns });
|
||||
return t ? [t] : [];
|
||||
}
|
||||
|
||||
function today(): string {
|
||||
return new Date().toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
|
||||
return { phase: 'synthesize', status: 'ok', duration_ms: 0, summary, details };
|
||||
}
|
||||
|
||||
function skipped(reason: string, summary: string): PhaseResult {
|
||||
return {
|
||||
phase: 'synthesize',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary,
|
||||
details: { reason },
|
||||
};
|
||||
}
|
||||
|
||||
function failed(error: PhaseError): PhaseResult {
|
||||
return {
|
||||
phase: 'synthesize',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'synthesize phase failed',
|
||||
details: {},
|
||||
error,
|
||||
};
|
||||
}
|
||||
|
||||
function makeError(cls: string, code: string, message: string, hint?: string): PhaseError {
|
||||
return hint ? { class: cls, code, message, hint } : { class: cls, code, message };
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
/**
|
||||
* Transcript discovery for the v0.23 dream-cycle synthesize phase.
|
||||
*
|
||||
* Walks a corpus directory for `.txt` files, applies date-range filters,
|
||||
* size filters (min_chars), and word-boundary regex exclude patterns.
|
||||
* Returns a list of file paths + content + content_hash so the caller
|
||||
* can key the verdict cache and dispatch one subagent per transcript.
|
||||
*
|
||||
* No DB; pure filesystem + crypto. Tested with hermetic temp directories.
|
||||
*/
|
||||
|
||||
import { readFileSync, readdirSync, statSync } from 'node:fs';
|
||||
import { join, basename } from 'node:path';
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
export interface DiscoveredTranscript {
|
||||
/** Absolute path to the transcript file. */
|
||||
filePath: string;
|
||||
/** sha256(content), full hex; callers slice as needed. */
|
||||
contentHash: string;
|
||||
/** Raw transcript text. */
|
||||
content: string;
|
||||
/** Filename basename without extension; used as a topic-slug seed. */
|
||||
basename: string;
|
||||
/** Inferred date if the basename matches `YYYY-MM-DD...` (or null). */
|
||||
inferredDate: string | null;
|
||||
}
|
||||
|
||||
export interface DiscoverOpts {
|
||||
/** Source directory. Required. */
|
||||
corpusDir: string;
|
||||
/** Optional second source. */
|
||||
meetingTranscriptsDir?: string;
|
||||
/** Skip transcripts smaller than this many characters. Default 2000. */
|
||||
minChars?: number;
|
||||
/** Word-boundary regex strings. The discoverer auto-wraps bare words. */
|
||||
excludePatterns?: string[];
|
||||
/** Restrict to a single date (YYYY-MM-DD basename match). */
|
||||
date?: string;
|
||||
/** Inclusive range start (YYYY-MM-DD). */
|
||||
from?: string;
|
||||
/** Inclusive range end (YYYY-MM-DD). */
|
||||
to?: string;
|
||||
}
|
||||
|
||||
const DATE_RE = /^(\d{4}-\d{2}-\d{2})/;
|
||||
const WORD_BOUNDARY_HEURISTIC = /^[a-zA-Z][a-zA-Z0-9_-]*$/;
|
||||
|
||||
/**
|
||||
* Auto-wrap bare-word patterns in `\b<word>\b`. Power users can pass full
|
||||
* regex (e.g. `^therapy:`) which we honor verbatim. Heuristic: any input
|
||||
* that's purely alphanumeric+hyphen+underscore is treated as a bare word.
|
||||
*/
|
||||
export function compileExcludePatterns(patterns: string[] | undefined): RegExp[] {
|
||||
if (!patterns || patterns.length === 0) return [];
|
||||
const out: RegExp[] = [];
|
||||
for (const p of patterns) {
|
||||
if (!p) continue;
|
||||
try {
|
||||
const src = WORD_BOUNDARY_HEURISTIC.test(p) ? `\\b${p}\\b` : p;
|
||||
out.push(new RegExp(src, 'i'));
|
||||
} catch (e) {
|
||||
// Bad regex from user config — skip with stderr warning, don't crash.
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[dream] invalid exclude_pattern '${p}': ${msg}\n`);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function hashContent(text: string): string {
|
||||
return createHash('sha256').update(text, 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
function isInDateRange(date: string | null, opts: DiscoverOpts): boolean {
|
||||
if (!opts.date && !opts.from && !opts.to) return true;
|
||||
if (!date) return false; // file has no inferable date but a filter is active
|
||||
if (opts.date && date !== opts.date) return false;
|
||||
if (opts.from && date < opts.from) return false;
|
||||
if (opts.to && date > opts.to) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
function matchesAnyExclude(text: string, patterns: RegExp[]): boolean {
|
||||
for (const re of patterns) {
|
||||
if (re.test(text)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function listTextFiles(dir: string): string[] {
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
const out: string[] = [];
|
||||
for (const name of entries) {
|
||||
if (!name.endsWith('.txt')) continue;
|
||||
const full = join(dir, name);
|
||||
try {
|
||||
if (statSync(full).isFile()) out.push(full);
|
||||
} catch {
|
||||
// skip unreadable entries
|
||||
}
|
||||
}
|
||||
return out.sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* Discover transcripts from the configured corpus dirs, applying filters.
|
||||
*
|
||||
* Skips files that:
|
||||
* - aren't `.txt`
|
||||
* - have date-prefixed basenames outside the requested window
|
||||
* - have content shorter than `minChars`
|
||||
* - match any compiled exclude pattern (case-insensitive word-boundary by default)
|
||||
*
|
||||
* Returns sorted by filePath so re-runs are deterministic.
|
||||
*/
|
||||
export function discoverTranscripts(opts: DiscoverOpts): DiscoveredTranscript[] {
|
||||
const minChars = opts.minChars ?? 2000;
|
||||
const excludeRes = compileExcludePatterns(opts.excludePatterns);
|
||||
const dirs = [opts.corpusDir, opts.meetingTranscriptsDir].filter(
|
||||
(d): d is string => typeof d === 'string' && d.length > 0,
|
||||
);
|
||||
|
||||
const results: DiscoveredTranscript[] = [];
|
||||
for (const dir of dirs) {
|
||||
for (const filePath of listTextFiles(dir)) {
|
||||
const baseName = basename(filePath, '.txt');
|
||||
const dateMatch = DATE_RE.exec(baseName);
|
||||
const inferredDate = dateMatch ? dateMatch[1] : null;
|
||||
if (!isInDateRange(inferredDate, opts)) continue;
|
||||
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(filePath, 'utf8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (content.length < minChars) continue;
|
||||
if (matchesAnyExclude(content, excludeRes)) continue;
|
||||
|
||||
results.push({
|
||||
filePath,
|
||||
contentHash: hashContent(content),
|
||||
content,
|
||||
basename: baseName,
|
||||
inferredDate,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return results.sort((a, b) => a.filePath.localeCompare(b.filePath));
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a single ad-hoc transcript file (`gbrain dream --input <file>`).
|
||||
* Bypasses the corpus-dir scan and date filters but still applies
|
||||
* minChars + exclude_patterns when provided.
|
||||
*/
|
||||
export function readSingleTranscript(
|
||||
filePath: string,
|
||||
opts: { minChars?: number; excludePatterns?: string[] } = {},
|
||||
): DiscoveredTranscript | null {
|
||||
const minChars = opts.minChars ?? 2000;
|
||||
const excludeRes = compileExcludePatterns(opts.excludePatterns);
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(filePath, 'utf8');
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
throw new Error(`could not read transcript at ${filePath}: ${msg}`);
|
||||
}
|
||||
if (content.length < minChars) return null;
|
||||
if (matchesAnyExclude(content, excludeRes)) return null;
|
||||
const baseName = basename(filePath, '.txt');
|
||||
const dateMatch = DATE_RE.exec(baseName);
|
||||
return {
|
||||
filePath,
|
||||
contentHash: hashContent(content),
|
||||
content,
|
||||
basename: baseName,
|
||||
inferredDate: dateMatch ? dateMatch[1] : null,
|
||||
};
|
||||
}
|
||||
@@ -86,6 +86,19 @@ export interface ReservedConnection {
|
||||
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;
|
||||
}
|
||||
|
||||
/** Dream-cycle Haiku verdict on whether a transcript is worth processing. */
|
||||
export interface DreamVerdict {
|
||||
worth_processing: boolean;
|
||||
reasons: string[];
|
||||
judged_at: string;
|
||||
}
|
||||
|
||||
/** Input shape for putDreamVerdict — judged_at defaults to now() server-side. */
|
||||
export interface DreamVerdictInput {
|
||||
worth_processing: boolean;
|
||||
reasons: string[];
|
||||
}
|
||||
|
||||
/** Maximum results returned by search operations. Internal bulk operations (listPages) are not clamped. */
|
||||
export const MAX_SEARCH_LIMIT = 100;
|
||||
|
||||
@@ -258,6 +271,12 @@ export interface BrainEngine {
|
||||
putRawData(slug: string, source: string, data: object): Promise<void>;
|
||||
getRawData(slug: string, source?: string): Promise<RawData[]>;
|
||||
|
||||
// Dream-cycle significance verdict cache (v0.23).
|
||||
// Keyed by (file_path, content_hash). Distinct from raw_data, which is
|
||||
// page-scoped — transcripts being judged aren't pages yet.
|
||||
getDreamVerdict(filePath: string, contentHash: string): Promise<DreamVerdict | null>;
|
||||
putDreamVerdict(filePath: string, contentHash: string, verdict: DreamVerdictInput): Promise<void>;
|
||||
|
||||
// Versions
|
||||
createVersion(slug: string): Promise<PageVersion>;
|
||||
getVersions(slug: string): Promise<PageVersion[]>;
|
||||
|
||||
@@ -1073,6 +1073,34 @@ export const MIGRATIONS: Migration[] = [
|
||||
},
|
||||
sql: '',
|
||||
},
|
||||
{
|
||||
version: 30,
|
||||
name: 'dream_verdicts_table',
|
||||
// v0.23 synthesize phase: cache for "is this transcript worth processing?"
|
||||
// verdict from the cheap Haiku judge. Distinct from raw_data (page-scoped);
|
||||
// transcripts aren't pages. Keyed by (file_path, content_hash) so edited
|
||||
// transcripts re-judge automatically. Backfill re-runs hit cache instead
|
||||
// of paying for Haiku 100x.
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS dream_verdicts (
|
||||
file_path TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL,
|
||||
worth_processing BOOLEAN NOT NULL,
|
||||
reasons JSONB,
|
||||
judged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (file_path, content_hash)
|
||||
);
|
||||
DO $$
|
||||
DECLARE
|
||||
has_bypass BOOLEAN;
|
||||
BEGIN
|
||||
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
|
||||
IF has_bypass THEN
|
||||
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
|
||||
END IF;
|
||||
END $$;
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -149,10 +149,14 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
const systemPrompt = data.system ?? DEFAULT_SYSTEM;
|
||||
|
||||
// Build the tool registry bound to THIS job as the owning subagent.
|
||||
// allowed_slug_prefixes (v0.23) flows through buildBrainTools → the
|
||||
// put_page schema description AND the OperationContext, so the model's
|
||||
// tool schema and the server-side check stay in sync.
|
||||
const registry = deps.toolRegistry ?? buildBrainTools({
|
||||
subagentId: ctx.id,
|
||||
engine,
|
||||
config,
|
||||
allowedSlugPrefixes: data.allowed_slug_prefixes,
|
||||
});
|
||||
const toolDefs = data.allowed_tools && data.allowed_tools.length > 0
|
||||
? filterAllowedTools(registry, data.allowed_tools)
|
||||
|
||||
@@ -91,19 +91,37 @@ function paramsToInputSchema(op: Operation): Record<string, unknown> {
|
||||
|
||||
/**
|
||||
* For put_page specifically, the tool schema shown to the model constrains
|
||||
* `slug` to `wiki/agents/<subagentId>/...`. The server-side check in
|
||||
* operations.ts is the authoritative gate; this just helps the model write
|
||||
* correct slugs on the first try.
|
||||
* `slug`. Two modes:
|
||||
*
|
||||
* - Default (legacy): slug MUST start with `wiki/agents/<subagentId>/`,
|
||||
* enforced by both the JSONSchema `pattern` and the server-side check.
|
||||
* - Trusted-workspace (v0.23 dream cycle): when `allowedSlugPrefixes` is
|
||||
* set, the model is told the allowed prefixes in plain English (no
|
||||
* regex pattern — the prefix list is authoritative server-side, and
|
||||
* JSONSchema can't express "matches any of these globs" cleanly).
|
||||
*/
|
||||
function namespacedPutPageSchema(op: Operation, subagentId: number): Record<string, unknown> {
|
||||
function namespacedPutPageSchema(
|
||||
op: Operation,
|
||||
subagentId: number,
|
||||
allowedSlugPrefixes?: readonly string[],
|
||||
): Record<string, unknown> {
|
||||
const base = paramsToInputSchema(op);
|
||||
const props = (base.properties as Record<string, Record<string, unknown>>) ?? {};
|
||||
if (props.slug) {
|
||||
props.slug = {
|
||||
...props.slug,
|
||||
description: `Page slug. MUST start with "wiki/agents/${subagentId}/" (agents can only write under their own namespace).`,
|
||||
pattern: `^wiki/agents/${subagentId}/.+`,
|
||||
};
|
||||
if (allowedSlugPrefixes && allowedSlugPrefixes.length > 0) {
|
||||
props.slug = {
|
||||
...props.slug,
|
||||
description:
|
||||
`Page slug. MUST match one of these prefix globs: ${allowedSlugPrefixes.join(', ')}. ` +
|
||||
`Slugs use lowercase alphanumeric segments separated by '/'. No leading slash, no '.md' extension, no underscores.`,
|
||||
};
|
||||
} else {
|
||||
props.slug = {
|
||||
...props.slug,
|
||||
description: `Page slug. MUST start with "wiki/agents/${subagentId}/" (agents can only write under their own namespace).`,
|
||||
pattern: `^wiki/agents/${subagentId}/.+`,
|
||||
};
|
||||
}
|
||||
}
|
||||
return { ...base, properties: props };
|
||||
}
|
||||
@@ -115,6 +133,14 @@ export interface BuildBrainToolsOpts {
|
||||
config: GBrainConfig;
|
||||
/** Optional filter: only include names in this set. */
|
||||
allowedNames?: ReadonlySet<string>;
|
||||
/**
|
||||
* Trusted-workspace allow-list (v0.23). When set, put_page is bounded
|
||||
* to slugs matching these prefix globs instead of the legacy
|
||||
* `wiki/agents/<id>/...` namespace. Trust comes from PROTECTED_JOB_NAMES
|
||||
* (MCP can't submit subagent jobs) — this flows from
|
||||
* SubagentHandlerData.allowed_slug_prefixes via the handler.
|
||||
*/
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
}
|
||||
|
||||
interface OpContextDeps {
|
||||
@@ -123,6 +149,7 @@ interface OpContextDeps {
|
||||
subagentId: number;
|
||||
jobId: number;
|
||||
signal?: AbortSignal;
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
}
|
||||
|
||||
function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
@@ -135,10 +162,13 @@ function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
error: (msg: string) => process.stderr.write(`[subagent-tool:${deps.jobId}] ERROR: ${msg}\n`),
|
||||
},
|
||||
dryRun: false,
|
||||
remote: true, // match MCP trust boundary
|
||||
remote: true, // match MCP trust boundary for auto-link skip
|
||||
jobId: deps.jobId,
|
||||
subagentId: deps.subagentId,
|
||||
viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace
|
||||
allowedSlugPrefixes: deps.allowedSlugPrefixes
|
||||
? [...deps.allowedSlugPrefixes]
|
||||
: undefined,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -157,7 +187,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
|
||||
return picked.map<ToolDef>(op => {
|
||||
const schema = op.name === 'put_page'
|
||||
? namespacedPutPageSchema(op, opts.subagentId)
|
||||
? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes)
|
||||
: paramsToInputSchema(op);
|
||||
|
||||
const toolName = sanitizeToolName(op.name);
|
||||
@@ -179,6 +209,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
subagentId: opts.subagentId,
|
||||
jobId: ctx.jobId,
|
||||
signal: ctx.signal,
|
||||
allowedSlugPrefixes: opts.allowedSlugPrefixes,
|
||||
});
|
||||
const params = (input && typeof input === 'object') ? input as Record<string, unknown> : {};
|
||||
return op.handler(opCtx, params);
|
||||
|
||||
@@ -421,6 +421,19 @@ export interface SubagentHandlerData {
|
||||
system?: string;
|
||||
/** Template variables for subagent_def. Arbitrary JSON-serializable. */
|
||||
input_vars?: Record<string, unknown>;
|
||||
/**
|
||||
* Trusted-workspace allow-list for put_page (v0.23 dream cycle).
|
||||
*
|
||||
* When set, the subagent's put_page calls are bounded to slugs matching
|
||||
* any of these prefix globs (e.g. ["wiki/personal/reflections/*",
|
||||
* "wiki/originals/*"]). When unset/empty, the legacy
|
||||
* `wiki/agents/<subagentId>/...` namespace check applies.
|
||||
*
|
||||
* Trust comes from PROTECTED_JOB_NAMES gating subagent submission — MCP
|
||||
* cannot reach this field. Only cycle.ts (synthesize/patterns phases)
|
||||
* and direct CLI submitters set it.
|
||||
*/
|
||||
allowed_slug_prefixes?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+68
-4
@@ -120,6 +120,31 @@ export function validatePageSlug(slug: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Match a slug against a list of allow-list prefix globs.
|
||||
*
|
||||
* Glob form: `<prefix>/*` matches any slug starting with `<prefix>/` and
|
||||
* having at least one more segment (single or multi). Bare `<prefix>` (no
|
||||
* trailing `/*`) matches that exact slug only. The `*` is intentionally
|
||||
* permissive — depth is unbounded, so `wiki/originals/*` matches both
|
||||
* `wiki/originals/idea-x` and `wiki/originals/ideas/2026-04-25-idea-y`.
|
||||
*
|
||||
* Used by the v0.23 dream-cycle trusted-workspace path. Order doesn't
|
||||
* matter; the first match wins (returns true on any match).
|
||||
*/
|
||||
export function matchesSlugAllowList(slug: string, prefixes: readonly string[]): boolean {
|
||||
for (const p of prefixes) {
|
||||
if (p.endsWith('/*')) {
|
||||
const base = p.slice(0, -2);
|
||||
if (slug === base) continue;
|
||||
if (slug.startsWith(base + '/')) return true;
|
||||
} else if (p === slug) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Allowlist validator for uploaded file basenames. Rejects control chars, backslashes,
|
||||
* RTL overrides (\u202E), leading dot (hidden files) and leading dash (CLI flag confusion).
|
||||
@@ -181,6 +206,22 @@ export interface OperationContext {
|
||||
jobId?: number;
|
||||
subagentId?: number;
|
||||
viaSubagent?: boolean;
|
||||
/**
|
||||
* Trusted-workspace allow-list (v0.23 dream cycle). When the cycle's
|
||||
* synthesize/patterns phases dispatch a subagent, they thread an
|
||||
* explicit list of slug-prefix globs (e.g. "wiki/personal/reflections/*")
|
||||
* through this field. put_page enforces it BEFORE the legacy
|
||||
* `wiki/agents/<id>/...` namespace check.
|
||||
*
|
||||
* Trust comes from the SUBMITTER (subagent jobs are gated by
|
||||
* PROTECTED_JOB_NAMES — MCP cannot submit them), not from `remote`.
|
||||
* Every subagent tool call has `remote=true` for auto-link safety,
|
||||
* so basing trust on `remote` is incoherent (would always reject).
|
||||
*
|
||||
* Empty / unset → fall back to the legacy namespace check (existing
|
||||
* v0.15 behavior; pure addition, no regression).
|
||||
*/
|
||||
allowedSlugPrefixes?: string[];
|
||||
/**
|
||||
* Resolved global CLI options (--quiet / --progress-json / --progress-interval).
|
||||
* CLI callers populate this from `getCliOptions()`. MCP / library callers
|
||||
@@ -264,9 +305,23 @@ const put_page: Operation = {
|
||||
if (typeof ctx.subagentId !== 'number' || Number.isNaN(ctx.subagentId)) {
|
||||
throw new OperationError('permission_denied', 'put_page via subagent requires ctx.subagentId');
|
||||
}
|
||||
const prefix = `wiki/agents/${ctx.subagentId}/`;
|
||||
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
|
||||
throw new OperationError('permission_denied', `put_page via subagent must write under '${prefix}...'`);
|
||||
const allowList = ctx.allowedSlugPrefixes;
|
||||
if (allowList && allowList.length > 0) {
|
||||
// Trusted-workspace path: explicit allow-list bounds writes.
|
||||
// Set only by cycle.ts (synthesize/patterns) which submits subagent
|
||||
// jobs under PROTECTED_JOB_NAMES — MCP cannot reach this branch.
|
||||
if (!matchesSlugAllowList(slug, allowList)) {
|
||||
throw new OperationError(
|
||||
'permission_denied',
|
||||
`put_page slug '${slug}' is not within the trusted-workspace allow-list (${allowList.join(', ')})`
|
||||
);
|
||||
}
|
||||
} else {
|
||||
// Legacy default: agent-namespace confinement.
|
||||
const prefix = `wiki/agents/${ctx.subagentId}/`;
|
||||
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
|
||||
throw new OperationError('permission_denied', `put_page via subagent must write under '${prefix}...'`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -295,7 +350,16 @@ const put_page: Operation = {
|
||||
| { skipped: 'remote' }
|
||||
| undefined;
|
||||
let autoTimeline: { created: number } | { error: string } | { skipped: 'remote' } | undefined;
|
||||
if (ctx.remote === true) {
|
||||
// Trusted-workspace path (v0.23 dream cycle) re-enables auto-link/timeline
|
||||
// even though ctx.remote=true, because the allow-list bounds the slug and
|
||||
// the synthesis prompt is itself the trusted dispatcher. Without this,
|
||||
// the cycle's `extract` phase would have to recompute every edge, and
|
||||
// patterns (which runs after extract) would still see the right graph
|
||||
// but auto_timeline would never fire on synth output.
|
||||
const trustedWorkspace = ctx.viaSubagent === true
|
||||
&& Array.isArray(ctx.allowedSlugPrefixes)
|
||||
&& ctx.allowedSlugPrefixes.length > 0;
|
||||
if (ctx.remote === true && !trustedWorkspace) {
|
||||
autoLinks = { skipped: 'remote' };
|
||||
autoTimeline = { skipped: 'remote' };
|
||||
} else if (result.parsedPage) {
|
||||
|
||||
@@ -2,7 +2,7 @@ import { PGlite } from '@electric-sql/pglite';
|
||||
import { vector } from '@electric-sql/pglite/vector';
|
||||
import { pg_trgm } from '@electric-sql/pglite/contrib/pg_trgm';
|
||||
import type { Transaction } from '@electric-sql/pglite';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection } from './engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection, DreamVerdict, DreamVerdictInput } from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { PGLITE_SCHEMA_SQL } from './pglite-schema.ts';
|
||||
@@ -1157,6 +1157,39 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return result.rows as unknown as RawData[];
|
||||
}
|
||||
|
||||
// Dream-cycle significance verdict cache (v0.23).
|
||||
async getDreamVerdict(filePath: string, contentHash: string): Promise<DreamVerdict | null> {
|
||||
const result = await this.db.query<{
|
||||
worth_processing: boolean;
|
||||
reasons: string[] | null;
|
||||
judged_at: Date | string;
|
||||
}>(
|
||||
`SELECT worth_processing, reasons, judged_at
|
||||
FROM dream_verdicts
|
||||
WHERE file_path = $1 AND content_hash = $2`,
|
||||
[filePath, contentHash]
|
||||
);
|
||||
if (result.rows.length === 0) return null;
|
||||
const r = result.rows[0];
|
||||
return {
|
||||
worth_processing: r.worth_processing,
|
||||
reasons: r.reasons ?? [],
|
||||
judged_at: r.judged_at instanceof Date ? r.judged_at.toISOString() : String(r.judged_at),
|
||||
};
|
||||
}
|
||||
|
||||
async putDreamVerdict(filePath: string, contentHash: string, verdict: DreamVerdictInput): Promise<void> {
|
||||
await this.db.query(
|
||||
`INSERT INTO dream_verdicts (file_path, content_hash, worth_processing, reasons)
|
||||
VALUES ($1, $2, $3, $4::jsonb)
|
||||
ON CONFLICT (file_path, content_hash) DO UPDATE SET
|
||||
worth_processing = EXCLUDED.worth_processing,
|
||||
reasons = EXCLUDED.reasons,
|
||||
judged_at = now()`,
|
||||
[filePath, contentHash, verdict.worth_processing, JSON.stringify(verdict.reasons)]
|
||||
);
|
||||
}
|
||||
|
||||
// Versions
|
||||
async createVersion(slug: string): Promise<PageVersion> {
|
||||
const { rows } = await this.db.query(
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import postgres from 'postgres';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection } from './engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection, DreamVerdict, DreamVerdictInput } from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
@@ -1303,6 +1303,39 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows as unknown as RawData[];
|
||||
}
|
||||
|
||||
// Dream-cycle significance verdict cache (v0.23).
|
||||
async getDreamVerdict(filePath: string, contentHash: string): Promise<DreamVerdict | null> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql<Array<{
|
||||
worth_processing: boolean;
|
||||
reasons: string[] | null;
|
||||
judged_at: Date;
|
||||
}>>`
|
||||
SELECT worth_processing, reasons, judged_at
|
||||
FROM dream_verdicts
|
||||
WHERE file_path = ${filePath} AND content_hash = ${contentHash}
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
const r = rows[0];
|
||||
return {
|
||||
worth_processing: r.worth_processing,
|
||||
reasons: r.reasons ?? [],
|
||||
judged_at: r.judged_at instanceof Date ? r.judged_at.toISOString() : String(r.judged_at),
|
||||
};
|
||||
}
|
||||
|
||||
async putDreamVerdict(filePath: string, contentHash: string, verdict: DreamVerdictInput): Promise<void> {
|
||||
const sql = this.sql;
|
||||
await sql`
|
||||
INSERT INTO dream_verdicts (file_path, content_hash, worth_processing, reasons)
|
||||
VALUES (${filePath}, ${contentHash}, ${verdict.worth_processing}, ${sql.json(verdict.reasons as Parameters<typeof sql.json>[0])})
|
||||
ON CONFLICT (file_path, content_hash) DO UPDATE SET
|
||||
worth_processing = EXCLUDED.worth_processing,
|
||||
reasons = EXCLUDED.reasons,
|
||||
judged_at = now()
|
||||
`;
|
||||
}
|
||||
|
||||
// Versions
|
||||
async createVersion(slug: string): Promise<PageVersion> {
|
||||
const sql = this.sql;
|
||||
|
||||
@@ -596,6 +596,22 @@ CREATE TABLE IF NOT EXISTS subagent_rate_leases (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at);
|
||||
|
||||
-- ============================================================
|
||||
-- Dream-cycle significance verdict cache — v0.23 synthesize phase
|
||||
-- ============================================================
|
||||
-- Caches the cheap Haiku "is this transcript worth processing?" verdict
|
||||
-- per (file_path, content_hash) so backfill re-runs skip already-judged
|
||||
-- files. Distinct from raw_data (which is page-scoped); transcripts
|
||||
-- aren't pages.
|
||||
CREATE TABLE IF NOT EXISTS dream_verdicts (
|
||||
file_path TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL,
|
||||
worth_processing BOOLEAN NOT NULL,
|
||||
reasons JSONB,
|
||||
judged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (file_path, content_hash)
|
||||
);
|
||||
|
||||
-- ============================================================
|
||||
-- Cycle coordination lock — v0.17 runCycle primitive
|
||||
-- ============================================================
|
||||
@@ -663,6 +679,7 @@ BEGIN
|
||||
ALTER TABLE subagent_tool_executions ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
|
||||
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
|
||||
ELSE
|
||||
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
|
||||
|
||||
@@ -592,6 +592,22 @@ CREATE TABLE IF NOT EXISTS subagent_rate_leases (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at);
|
||||
|
||||
-- ============================================================
|
||||
-- Dream-cycle significance verdict cache — v0.21 synthesize phase
|
||||
-- ============================================================
|
||||
-- Caches the cheap Haiku "is this transcript worth processing?" verdict
|
||||
-- per (file_path, content_hash) so backfill re-runs skip already-judged
|
||||
-- files. Distinct from raw_data (which is page-scoped); transcripts
|
||||
-- aren't pages.
|
||||
CREATE TABLE IF NOT EXISTS dream_verdicts (
|
||||
file_path TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL,
|
||||
worth_processing BOOLEAN NOT NULL,
|
||||
reasons JSONB,
|
||||
judged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (file_path, content_hash)
|
||||
);
|
||||
|
||||
-- ============================================================
|
||||
-- Cycle coordination lock — v0.17 runCycle primitive
|
||||
-- ============================================================
|
||||
@@ -659,6 +675,7 @@ BEGIN
|
||||
ALTER TABLE subagent_tool_executions ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
|
||||
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
|
||||
ELSE
|
||||
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
|
||||
|
||||
@@ -377,8 +377,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
|
||||
hookCalls++;
|
||||
},
|
||||
});
|
||||
// 6 phases → 6 yield calls (one after each).
|
||||
expect(hookCalls).toBe(6);
|
||||
// v0.23: 8 phases → 8 yield calls (one after each).
|
||||
expect(hookCalls).toBe(8);
|
||||
});
|
||||
|
||||
test('hook exceptions do not abort the cycle', async () => {
|
||||
@@ -388,8 +388,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
|
||||
throw new Error('synthetic hook error');
|
||||
},
|
||||
});
|
||||
// Cycle still completed all phases.
|
||||
expect(report.phases.length).toBe(6);
|
||||
// Cycle still completed all phases (v0.23: 8).
|
||||
expect(report.phases.length).toBe(8);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* Unit tests for the patterns phase (v0.21).
|
||||
*
|
||||
* The phase invokes a subagent and queues real Minions work, so this
|
||||
* file leans on structural assertions over the source + a single
|
||||
* end-to-end driver run that exercises the skip-paths.
|
||||
*
|
||||
* Full LLM behavior is exercised by E2E tests in test/e2e/.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
const patternsSrc = readFileSync(
|
||||
new URL('../src/core/cycle/patterns.ts', import.meta.url),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
describe('patterns phase wiring', () => {
|
||||
test('imports queue + waitForCompletion + types', () => {
|
||||
expect(patternsSrc).toContain("import { MinionQueue }");
|
||||
expect(patternsSrc).toContain('waitForCompletion');
|
||||
expect(patternsSrc).toContain('SubagentHandlerData');
|
||||
});
|
||||
|
||||
test('threads allowed_slug_prefixes from filing-rules JSON', () => {
|
||||
expect(patternsSrc).toContain('allowed_slug_prefixes');
|
||||
expect(patternsSrc).toContain('_brain-filing-rules.json');
|
||||
expect(patternsSrc).toContain('dream_synthesize_paths');
|
||||
});
|
||||
|
||||
test('reads min_evidence + lookback_days config', () => {
|
||||
expect(patternsSrc).toContain('dream.patterns.min_evidence');
|
||||
expect(patternsSrc).toContain('dream.patterns.lookback_days');
|
||||
});
|
||||
|
||||
test('uses subagent_tool_executions for slug provenance (Codex #2 fix)', () => {
|
||||
expect(patternsSrc).toContain('subagent_tool_executions');
|
||||
expect(patternsSrc).toContain("tool_name = 'brain_put_page'");
|
||||
});
|
||||
|
||||
test('skips when ANTHROPIC_API_KEY missing', () => {
|
||||
expect(patternsSrc).toContain('ANTHROPIC_API_KEY');
|
||||
expect(patternsSrc).toContain('no_api_key');
|
||||
});
|
||||
|
||||
test('skips when reflections below min_evidence', () => {
|
||||
expect(patternsSrc).toContain('insufficient_evidence');
|
||||
});
|
||||
|
||||
test('reverse-writes pages to disk via serializeMarkdown', () => {
|
||||
expect(patternsSrc).toContain('serializeMarkdown');
|
||||
expect(patternsSrc).toContain('writeFileSync');
|
||||
});
|
||||
|
||||
test('runs after extract — queries fresh graph', () => {
|
||||
// Documented invariant: pattern phase MUST run after extract.
|
||||
// The cycle.ts dispatcher enforces order; this just confirms the
|
||||
// patterns module doesn't try to compute its own auto-link layer
|
||||
// (which would be a subtle regression).
|
||||
expect(patternsSrc).not.toContain('runAutoLink');
|
||||
expect(patternsSrc).not.toContain('extractPageLinks(');
|
||||
});
|
||||
|
||||
test('does NOT use raw_data table (Codex #3 fix)', () => {
|
||||
expect(patternsSrc).not.toContain('putRawData');
|
||||
expect(patternsSrc).not.toContain('getRawData');
|
||||
});
|
||||
});
|
||||
|
||||
describe('patterns scope filter', () => {
|
||||
test('filters reflections by slug LIKE wiki/personal/reflections/%', () => {
|
||||
expect(patternsSrc).toContain("slug LIKE 'wiki/personal/reflections/%'");
|
||||
});
|
||||
|
||||
test('orders by updated_at DESC for recency-bias', () => {
|
||||
expect(patternsSrc).toContain('ORDER BY updated_at DESC');
|
||||
});
|
||||
|
||||
test('caps gather to 100 reflections (cost control)', () => {
|
||||
expect(patternsSrc).toContain('LIMIT 100');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,191 @@
|
||||
/**
|
||||
* Unit tests for the synthesize phase scaffolding.
|
||||
*
|
||||
* Covers transcript-discovery branches (date filters, exclude regex,
|
||||
* minChars, multiple sources) and the compileExcludePatterns word-
|
||||
* boundary heuristic. Doesn't drive a real Anthropic call — full
|
||||
* cycle E2E lives in test/e2e/.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import {
|
||||
discoverTranscripts,
|
||||
readSingleTranscript,
|
||||
compileExcludePatterns,
|
||||
} from '../src/core/cycle/transcript-discovery.ts';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
function makeTranscript(name: string, body: string): string {
|
||||
const path = join(tmpDir, name);
|
||||
writeFileSync(path, body, 'utf8');
|
||||
return path;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-synth-test-'));
|
||||
});
|
||||
|
||||
describe('compileExcludePatterns', () => {
|
||||
test('auto-wraps bare words in word-boundary regex (Q-3)', () => {
|
||||
const res = compileExcludePatterns(['medical']);
|
||||
expect(res).toHaveLength(1);
|
||||
// word boundary: matches "medical" but NOT "comedical"
|
||||
expect(res[0].test('medical advice')).toBe(true);
|
||||
expect(res[0].test('comedical')).toBe(false);
|
||||
});
|
||||
|
||||
test('honors raw regex when input is non-bare-word', () => {
|
||||
const res = compileExcludePatterns(['^therapy:']);
|
||||
expect(res[0].test('therapy: today was hard')).toBe(true);
|
||||
expect(res[0].test('thinking about therapy:')).toBe(false);
|
||||
});
|
||||
|
||||
test('skips invalid regex with warning, does not crash', () => {
|
||||
const res = compileExcludePatterns(['valid', '(broken[']);
|
||||
expect(res).toHaveLength(1); // only the valid one compiled
|
||||
});
|
||||
|
||||
test('case-insensitive matching by default', () => {
|
||||
const res = compileExcludePatterns(['Medical']);
|
||||
expect(res[0].test('medical advice')).toBe(true);
|
||||
expect(res[0].test('MEDICAL ADVICE')).toBe(true);
|
||||
});
|
||||
|
||||
test('empty / undefined input returns empty array', () => {
|
||||
expect(compileExcludePatterns(undefined)).toEqual([]);
|
||||
expect(compileExcludePatterns([])).toEqual([]);
|
||||
expect(compileExcludePatterns([''])).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('discoverTranscripts', () => {
|
||||
test('returns empty when corpusDir does not exist', () => {
|
||||
const out = discoverTranscripts({ corpusDir: '/nonexistent/path' });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('returns transcripts above minChars, sorted by filePath', () => {
|
||||
makeTranscript('2026-04-25-session.txt', 'a'.repeat(2500));
|
||||
makeTranscript('2026-04-24-other.txt', 'b'.repeat(2500));
|
||||
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
|
||||
expect(out).toHaveLength(2);
|
||||
expect(out[0].basename).toBe('2026-04-24-other');
|
||||
expect(out[1].basename).toBe('2026-04-25-session');
|
||||
});
|
||||
|
||||
test('skips transcripts below minChars', () => {
|
||||
makeTranscript('2026-04-25-short.txt', 'tiny');
|
||||
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 2000 });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('skips non-txt files', () => {
|
||||
makeTranscript('2026-04-25-foo.md', 'a'.repeat(3000));
|
||||
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('exclude_patterns filters out matched transcripts (word boundary)', () => {
|
||||
makeTranscript('2026-04-25-medical.txt', 'discussing medical advice ' + 'x'.repeat(3000));
|
||||
makeTranscript('2026-04-25-comedy.txt', 'comedical writing tips ' + 'x'.repeat(3000));
|
||||
const out = discoverTranscripts({
|
||||
corpusDir: tmpDir,
|
||||
minChars: 1000,
|
||||
excludePatterns: ['medical'],
|
||||
});
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0].basename).toBe('2026-04-25-comedy');
|
||||
});
|
||||
|
||||
test('--date filter restricts to one specific YYYY-MM-DD basename', () => {
|
||||
makeTranscript('2026-04-25-foo.txt', 'a'.repeat(3000));
|
||||
makeTranscript('2026-04-26-bar.txt', 'b'.repeat(3000));
|
||||
const out = discoverTranscripts({
|
||||
corpusDir: tmpDir,
|
||||
minChars: 1000,
|
||||
date: '2026-04-25',
|
||||
});
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0].basename).toBe('2026-04-25-foo');
|
||||
});
|
||||
|
||||
test('--from / --to range filters basename dates', () => {
|
||||
makeTranscript('2026-04-23-a.txt', 'a'.repeat(3000));
|
||||
makeTranscript('2026-04-25-b.txt', 'b'.repeat(3000));
|
||||
makeTranscript('2026-04-27-c.txt', 'c'.repeat(3000));
|
||||
const out = discoverTranscripts({
|
||||
corpusDir: tmpDir,
|
||||
minChars: 1000,
|
||||
from: '2026-04-24',
|
||||
to: '2026-04-26',
|
||||
});
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0].basename).toBe('2026-04-25-b');
|
||||
});
|
||||
|
||||
test('multiple sources (corpus + meeting transcripts) merged', () => {
|
||||
makeTranscript('2026-04-25-session.txt', 'a'.repeat(3000));
|
||||
const meetDir = mkdtempSync(join(tmpdir(), 'gbrain-meet-'));
|
||||
writeFileSync(join(meetDir, '2026-04-25-meeting.txt'), 'b'.repeat(3000));
|
||||
const out = discoverTranscripts({
|
||||
corpusDir: tmpDir,
|
||||
meetingTranscriptsDir: meetDir,
|
||||
minChars: 1000,
|
||||
});
|
||||
expect(out).toHaveLength(2);
|
||||
rmSync(meetDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('content_hash is stable for identical content, different for edits (A-3)', () => {
|
||||
makeTranscript('2026-04-25-a.txt', 'identical content ' + 'x'.repeat(3000));
|
||||
makeTranscript('2026-04-25-b.txt', 'identical content ' + 'x'.repeat(3000));
|
||||
const out1 = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
|
||||
expect(out1[0].contentHash).toBe(out1[1].contentHash);
|
||||
|
||||
// Edit one — hash changes
|
||||
makeTranscript('2026-04-25-a.txt', 'edited content ' + 'x'.repeat(3000));
|
||||
const out2 = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
|
||||
expect(out2[0].contentHash).not.toBe(out2[1].contentHash);
|
||||
});
|
||||
});
|
||||
|
||||
describe('readSingleTranscript', () => {
|
||||
test('returns transcript above minChars', () => {
|
||||
const path = makeTranscript('hello.txt', 'a'.repeat(3000));
|
||||
const t = readSingleTranscript(path, { minChars: 1000 });
|
||||
expect(t).not.toBeNull();
|
||||
expect(t!.basename).toBe('hello');
|
||||
});
|
||||
|
||||
test('returns null when below minChars', () => {
|
||||
const path = makeTranscript('hello.txt', 'tiny');
|
||||
const t = readSingleTranscript(path, { minChars: 2000 });
|
||||
expect(t).toBeNull();
|
||||
});
|
||||
|
||||
test('returns null when content matches exclude pattern', () => {
|
||||
const path = makeTranscript('hello.txt', 'medical content ' + 'x'.repeat(3000));
|
||||
const t = readSingleTranscript(path, { minChars: 1000, excludePatterns: ['medical'] });
|
||||
expect(t).toBeNull();
|
||||
});
|
||||
|
||||
test('throws on missing file', () => {
|
||||
expect(() => readSingleTranscript('/nonexistent/foo.txt')).toThrow();
|
||||
});
|
||||
|
||||
test('infers date from YYYY-MM-DD basename', () => {
|
||||
const path = makeTranscript('2026-04-25-thing.txt', 'a'.repeat(3000));
|
||||
const t = readSingleTranscript(path, { minChars: 1000 });
|
||||
expect(t!.inferredDate).toBe('2026-04-25');
|
||||
});
|
||||
|
||||
test('inferredDate null when basename does not start with YYYY-MM-DD', () => {
|
||||
const path = makeTranscript('random-basename.txt', 'a'.repeat(3000));
|
||||
const t = readSingleTranscript(path, { minChars: 1000 });
|
||||
expect(t!.inferredDate).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,61 @@
|
||||
/**
|
||||
* Structural tests for `gbrain dream` argv parsing (v0.21).
|
||||
*
|
||||
* Verifies the help text + parser source contains the new flags
|
||||
* (--input, --date, --from, --to) and that conflict detection is wired.
|
||||
* The actual parseArgs is internal; we exercise it via the source file
|
||||
* structure to avoid spinning up a process per test.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
const dreamSrc = readFileSync(new URL('../src/commands/dream.ts', import.meta.url), 'utf-8');
|
||||
|
||||
describe('dream CLI flag wiring', () => {
|
||||
test('declares --input flag with file argument', () => {
|
||||
expect(dreamSrc).toContain("'--input'");
|
||||
expect(dreamSrc).toContain('inputFile');
|
||||
});
|
||||
|
||||
test('declares --date / --from / --to flags', () => {
|
||||
expect(dreamSrc).toContain("'--date'");
|
||||
expect(dreamSrc).toContain("'--from'");
|
||||
expect(dreamSrc).toContain("'--to'");
|
||||
});
|
||||
|
||||
test('validates ISO date format', () => {
|
||||
expect(dreamSrc).toMatch(/ISO_DATE_RE/);
|
||||
expect(dreamSrc).toContain('YYYY-MM-DD');
|
||||
});
|
||||
|
||||
test('--input + --date conflict detection', () => {
|
||||
expect(dreamSrc).toContain('--input cannot be combined with --date');
|
||||
});
|
||||
|
||||
test('--input implies --phase synthesize', () => {
|
||||
expect(dreamSrc).toContain("phase = 'synthesize'");
|
||||
});
|
||||
|
||||
test('--from > --to range validation', () => {
|
||||
expect(dreamSrc).toContain('empty range');
|
||||
});
|
||||
|
||||
test('forwards synth fields to runCycle', () => {
|
||||
expect(dreamSrc).toContain('synthInputFile');
|
||||
expect(dreamSrc).toContain('synthDate');
|
||||
expect(dreamSrc).toContain('synthFrom');
|
||||
expect(dreamSrc).toContain('synthTo');
|
||||
});
|
||||
|
||||
test('totals line includes synth + patterns counters', () => {
|
||||
expect(dreamSrc).toContain('synth_transcripts');
|
||||
expect(dreamSrc).toContain('synth_pages');
|
||||
expect(dreamSrc).toContain('patterns=');
|
||||
});
|
||||
|
||||
test('help text documents dry-run synthesis semantics (Codex finding #8)', () => {
|
||||
expect(dreamSrc).toContain('skips the Sonnet');
|
||||
expect(dreamSrc.toLowerCase()).toContain('zero llm calls');
|
||||
});
|
||||
});
|
||||
@@ -97,8 +97,8 @@ describeE2E('E2E: runCycle against real Postgres', () => {
|
||||
});
|
||||
|
||||
expect(report.schema_version).toBe('1');
|
||||
// Cycle ran all 6 phases (or skipped the ones that don't support dry-run).
|
||||
expect(report.phases.length).toBe(6);
|
||||
// Cycle ran all 8 phases (or skipped the ones that don't support dry-run).
|
||||
expect(report.phases.length).toBe(8);
|
||||
|
||||
// Nothing got written.
|
||||
const afterPages = await conn.unsafe(`SELECT count(*)::int AS n FROM pages`);
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
/**
|
||||
* E2E security regression: poisoned-transcript guard for the v0.21
|
||||
* trusted-workspace allow-list.
|
||||
*
|
||||
* Runs against PGLite in-memory (no DATABASE_URL required). Builds the
|
||||
* brain tool registry with `allowed_slug_prefixes` set the same way the
|
||||
* synthesize phase does, then calls the put_page tool with slugs that
|
||||
* are inside / outside the allow-list. Asserts:
|
||||
*
|
||||
* - In-allow-list slug → page is written to the DB
|
||||
* - Outside-allow-list slug → tool throws permission_denied
|
||||
* - When allow-list is unset (legacy), put_page is bounded to
|
||||
* wiki/agents/<id>/... (regression guard for the v0.15 anti-prompt-
|
||||
* injection guarantee)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { buildBrainTools } from '../../src/core/minions/tools/brain-allowlist.ts';
|
||||
import type { GBrainConfig } from '../../src/core/config.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite' } as never);
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
});
|
||||
|
||||
const config = {} as unknown as GBrainConfig;
|
||||
|
||||
const PUT_PAGE_TOOL = 'brain_put_page';
|
||||
const SAMPLE_BODY = '---\ntitle: A reflection\ntype: default\n---\n\nbody text\n';
|
||||
|
||||
function findPutPageTool(tools: Awaited<ReturnType<typeof buildBrainTools>>) {
|
||||
const t = tools.find(x => x.name === PUT_PAGE_TOOL);
|
||||
if (!t) throw new Error('brain_put_page tool not found in registry');
|
||||
return t;
|
||||
}
|
||||
|
||||
describe('E2E allow-list — trusted-workspace path', () => {
|
||||
test('ALLOW: subagent put_page within allow-list writes the page', async () => {
|
||||
const tools = buildBrainTools({
|
||||
subagentId: 999,
|
||||
engine,
|
||||
config,
|
||||
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
|
||||
});
|
||||
const tool = findPutPageTool(tools);
|
||||
await tool.execute(
|
||||
{ slug: 'wiki/personal/reflections/2026-04-25-arete-paradox-a3f8c1', content: SAMPLE_BODY },
|
||||
{ engine, jobId: 7777, remote: true },
|
||||
);
|
||||
const page = await engine.getPage('wiki/personal/reflections/2026-04-25-arete-paradox-a3f8c1');
|
||||
expect(page).not.toBeNull();
|
||||
expect(page!.title).toBe('A reflection');
|
||||
});
|
||||
|
||||
test('REJECT: subagent put_page outside allow-list throws permission_denied', async () => {
|
||||
const tools = buildBrainTools({
|
||||
subagentId: 999,
|
||||
engine,
|
||||
config,
|
||||
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
|
||||
});
|
||||
const tool = findPutPageTool(tools);
|
||||
let threw = false;
|
||||
try {
|
||||
await tool.execute(
|
||||
{ slug: 'wiki/finance/secret-market-data', content: SAMPLE_BODY },
|
||||
{ engine, jobId: 7778, remote: true },
|
||||
);
|
||||
} catch (e) {
|
||||
threw = true;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
expect(msg).toMatch(/allow-list/i);
|
||||
}
|
||||
expect(threw).toBe(true);
|
||||
const page = await engine.getPage('wiki/finance/secret-market-data');
|
||||
expect(page).toBeNull(); // never reached the engine
|
||||
});
|
||||
|
||||
test('Multiple prefixes: each slug evaluated independently', async () => {
|
||||
const tools = buildBrainTools({
|
||||
subagentId: 999,
|
||||
engine,
|
||||
config,
|
||||
allowedSlugPrefixes: ['wiki/personal/reflections/*', 'wiki/originals/*'],
|
||||
});
|
||||
const tool = findPutPageTool(tools);
|
||||
await tool.execute(
|
||||
{ slug: 'wiki/originals/ideas/2026-04-25-thousand-pound-armor', content: SAMPLE_BODY },
|
||||
{ engine, jobId: 7779, remote: true },
|
||||
);
|
||||
expect(await engine.getPage('wiki/originals/ideas/2026-04-25-thousand-pound-armor')).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E allow-list — legacy namespace fallback', () => {
|
||||
test('REGRESSION GUARD: when allow-list is unset, put_page rejects writes outside wiki/agents/<id>/', async () => {
|
||||
const tools = buildBrainTools({
|
||||
subagentId: 999,
|
||||
engine,
|
||||
config,
|
||||
// allowedSlugPrefixes intentionally omitted — exercises the v0.15
|
||||
// legacy namespace check that v0.21 must NOT regress.
|
||||
});
|
||||
const tool = findPutPageTool(tools);
|
||||
let threw = false;
|
||||
try {
|
||||
await tool.execute(
|
||||
{ slug: 'wiki/personal/reflections/2026-04-25-bypass-attempt', content: SAMPLE_BODY },
|
||||
{ engine, jobId: 7780, remote: true },
|
||||
);
|
||||
} catch (e) {
|
||||
threw = true;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
expect(msg).toMatch(/wiki\/agents\/999/);
|
||||
}
|
||||
expect(threw).toBe(true);
|
||||
});
|
||||
|
||||
test('When allow-list unset, slug under wiki/agents/<id>/ is allowed', async () => {
|
||||
const tools = buildBrainTools({
|
||||
subagentId: 999,
|
||||
engine,
|
||||
config,
|
||||
});
|
||||
const tool = findPutPageTool(tools);
|
||||
await tool.execute(
|
||||
{ slug: 'wiki/agents/999/scratch-note', content: SAMPLE_BODY },
|
||||
{ engine, jobId: 7781, remote: true },
|
||||
);
|
||||
expect(await engine.getPage('wiki/agents/999/scratch-note')).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E allow-list — provenance via tool execution rows (Codex #2)', () => {
|
||||
test('subagent_tool_executions captures slug for each put_page call', async () => {
|
||||
// The synthesize phase relies on this being queryable to determine
|
||||
// exactly which slugs each child wrote (instead of pages.updated_at).
|
||||
// We don't have a real subagent run here, but we can verify the table
|
||||
// exists and the column shape supports the orchestrator's query.
|
||||
const rows = await engine.executeRaw(
|
||||
`SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'subagent_tool_executions'
|
||||
ORDER BY column_name`,
|
||||
) as Array<{ column_name: string }>;
|
||||
const cols = rows.map(r => r.column_name);
|
||||
expect(cols).toContain('input');
|
||||
expect(cols).toContain('tool_name');
|
||||
expect(cols).toContain('status');
|
||||
expect(cols).toContain('job_id');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,196 @@
|
||||
/**
|
||||
* E2E full 8-phase cycle on PGLite, no API key required.
|
||||
*
|
||||
* Verifies that the v0.23 phase order — lint → backlinks → sync →
|
||||
* synthesize → extract → patterns → embed → orphans — is honored
|
||||
* end-to-end through runCycle when no API key is present (synthesize
|
||||
* + patterns skip cleanly, the other six phases run unchanged).
|
||||
*
|
||||
* Two regression-relevant invariants:
|
||||
* 1. CycleReport.phases preserves the 8-phase order — no future
|
||||
* reorder regresses without breaking this test.
|
||||
* 2. CycleReport.totals carries the new v0.23 fields:
|
||||
* transcripts_processed, synth_pages_written, patterns_written.
|
||||
*
|
||||
* No DATABASE_URL required. Mocks embedBatch so the embed phase doesn't
|
||||
* attempt OpenAI calls.
|
||||
*
|
||||
* Run: bun test test/e2e/dream-cycle-eight-phase-pglite.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, mock } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
|
||||
mock.module('../../src/core/embedding.ts', () => ({
|
||||
embed: async () => new Float32Array(1536),
|
||||
embedBatch: async (texts: string[]) => texts.map(() => new Float32Array(1536)),
|
||||
EMBEDDING_MODEL: 'text-embedding-3-large',
|
||||
EMBEDDING_DIMENSIONS: 1536,
|
||||
EMBEDDING_COST_PER_1K_TOKENS: 0.00013,
|
||||
estimateEmbeddingCostUsd: (tokens: number) => (tokens / 1000) * 0.00013,
|
||||
}));
|
||||
|
||||
const { runCycle, ALL_PHASES } = await import('../../src/core/cycle.ts');
|
||||
|
||||
interface TestRig {
|
||||
engine: PGLiteEngine;
|
||||
brainDir: string;
|
||||
cleanup: () => Promise<void>;
|
||||
}
|
||||
|
||||
async function setupRig(): Promise<TestRig> {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite' } as never);
|
||||
await engine.initSchema();
|
||||
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-cycle8-'));
|
||||
execSync('git init', { cwd: brainDir, stdio: 'pipe' });
|
||||
execSync('git config user.email test@test.co', { cwd: brainDir, stdio: 'pipe' });
|
||||
execSync('git config user.name test', { cwd: brainDir, stdio: 'pipe' });
|
||||
mkdirSync(join(brainDir, 'concepts'), { recursive: true });
|
||||
writeFileSync(
|
||||
join(brainDir, 'concepts/testing.md'),
|
||||
'---\ntype: concept\ntitle: Testing\n---\n\nTest body content.\n',
|
||||
);
|
||||
execSync('git add -A && git commit -m init', { cwd: brainDir, stdio: 'pipe' });
|
||||
await engine.setConfig('sync.repo_path', brainDir);
|
||||
|
||||
return {
|
||||
engine,
|
||||
brainDir,
|
||||
cleanup: async () => {
|
||||
try { await engine.disconnect(); } catch { /* */ }
|
||||
try { rmSync(brainDir, { recursive: true, force: true }); } catch { /* */ }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
|
||||
const saved = process.env.ANTHROPIC_API_KEY;
|
||||
delete process.env.ANTHROPIC_API_KEY;
|
||||
try {
|
||||
return await body();
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.ANTHROPIC_API_KEY;
|
||||
else process.env.ANTHROPIC_API_KEY = saved;
|
||||
}
|
||||
}
|
||||
|
||||
describe('E2E v0.23 8-phase cycle', () => {
|
||||
test('ALL_PHASES is the 8-phase order in the documented sequence', () => {
|
||||
expect(ALL_PHASES).toEqual([
|
||||
'lint',
|
||||
'backlinks',
|
||||
'sync',
|
||||
'synthesize',
|
||||
'extract',
|
||||
'patterns',
|
||||
'embed',
|
||||
'orphans',
|
||||
]);
|
||||
});
|
||||
|
||||
test('full cycle on dry-run returns CycleReport.phases in v0.23 order with new totals fields', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await withoutAnthropicKey(async () => {
|
||||
const report = await runCycle(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: true,
|
||||
});
|
||||
// Phase ordering preserved
|
||||
const phaseNames = report.phases.map(p => p.phase);
|
||||
expect(phaseNames).toEqual([
|
||||
'lint',
|
||||
'backlinks',
|
||||
'sync',
|
||||
'synthesize',
|
||||
'extract',
|
||||
'patterns',
|
||||
'embed',
|
||||
'orphans',
|
||||
]);
|
||||
// New totals fields exist (v0.23 additive growth)
|
||||
expect(report.totals).toMatchObject({
|
||||
transcripts_processed: 0,
|
||||
synth_pages_written: 0,
|
||||
patterns_written: 0,
|
||||
});
|
||||
// Synthesize and patterns are skipped (not_configured / insufficient_evidence)
|
||||
const synth = report.phases.find(p => p.phase === 'synthesize');
|
||||
const patterns = report.phases.find(p => p.phase === 'patterns');
|
||||
expect(synth?.status).toBe('skipped');
|
||||
expect(patterns?.status).toBe('skipped');
|
||||
});
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('--phase synthesize alone runs only that phase, returns skipped/not_configured', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await withoutAnthropicKey(async () => {
|
||||
const report = await runCycle(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
phases: ['synthesize'],
|
||||
});
|
||||
expect(report.phases).toHaveLength(1);
|
||||
expect(report.phases[0].phase).toBe('synthesize');
|
||||
expect(report.phases[0].status).toBe('skipped');
|
||||
});
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('--phase patterns alone runs only that phase, returns skipped/insufficient_evidence', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await withoutAnthropicKey(async () => {
|
||||
const report = await runCycle(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
phases: ['patterns'],
|
||||
});
|
||||
expect(report.phases).toHaveLength(1);
|
||||
expect(report.phases[0].phase).toBe('patterns');
|
||||
expect(report.phases[0].status).toBe('skipped');
|
||||
expect((report.phases[0].details as { reason?: string }).reason).toBe('insufficient_evidence');
|
||||
});
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('synthInputFile flag is plumbed through runCycle to runPhaseSynthesize', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
const transcript = join(tmpdir(), `gbrain-e2e-cycle8-input-${Date.now()}.txt`);
|
||||
writeFileSync(transcript, 'sample conversation '.repeat(300));
|
||||
try {
|
||||
await withoutAnthropicKey(async () => {
|
||||
const report = await runCycle(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
phases: ['synthesize'],
|
||||
synthInputFile: transcript,
|
||||
});
|
||||
// Without API key, synthesize falls through to no-key skip-path
|
||||
// and returns ok (NOT cooldown_active — explicit input bypasses).
|
||||
expect(report.phases[0].phase).toBe('synthesize');
|
||||
expect(report.phases[0].status).toBe('ok');
|
||||
});
|
||||
} finally {
|
||||
rmSync(transcript, { force: true });
|
||||
}
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* E2E patterns phase — PGLite, no API key required.
|
||||
*
|
||||
* Mirrors the per-test-rig pattern from dream-synthesize-pglite.test.ts.
|
||||
* Each test creates and tears down its own PGLite engine to avoid
|
||||
* cross-test contention (CLAUDE.md issue #223 macOS WASM bug).
|
||||
*
|
||||
* Covers the runPhasePatterns skip paths that don't require a real
|
||||
* Anthropic call:
|
||||
* - disabled: dream.patterns.enabled=false → skipped
|
||||
* - insufficient_evidence: <min_evidence reflections → skipped
|
||||
* - no_api_key: enough reflections, no ANTHROPIC_API_KEY → skipped
|
||||
* - dry-run: passes through with reflections_considered + zero pages
|
||||
*
|
||||
* The Sonnet detection path is structurally covered in
|
||||
* test/cycle-patterns.test.ts (asserts queue + waitForCompletion are
|
||||
* wired, allow-list reads from filing-rules JSON, slug provenance from
|
||||
* subagent_tool_executions, no raw_data dependency).
|
||||
*
|
||||
* Run: bun test test/e2e/dream-patterns-pglite.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { runPhasePatterns } from '../../src/core/cycle/patterns.ts';
|
||||
|
||||
interface TestRig {
|
||||
engine: PGLiteEngine;
|
||||
brainDir: string;
|
||||
cleanup: () => Promise<void>;
|
||||
}
|
||||
|
||||
async function setupRig(): Promise<TestRig> {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite' } as never);
|
||||
await engine.initSchema();
|
||||
return {
|
||||
engine,
|
||||
brainDir: '/tmp/gbrain-patterns-test',
|
||||
cleanup: async () => {
|
||||
try { await engine.disconnect(); } catch { /* */ }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
|
||||
const saved = process.env.ANTHROPIC_API_KEY;
|
||||
delete process.env.ANTHROPIC_API_KEY;
|
||||
try {
|
||||
return await body();
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.ANTHROPIC_API_KEY;
|
||||
else process.env.ANTHROPIC_API_KEY = saved;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert N reflection pages directly via engine.putPage so the patterns
|
||||
* gather query has data without going through the synthesize phase.
|
||||
* Slugs follow the v0.23 wiki/personal/reflections/<topic>-<hash> shape.
|
||||
*/
|
||||
async function seedReflections(engine: PGLiteEngine, count: number): Promise<void> {
|
||||
for (let i = 0; i < count; i++) {
|
||||
const slug = `wiki/personal/reflections/2026-04-${String(15 + i).padStart(2, '0')}-test-pattern-aaa${i}`;
|
||||
await engine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: `Reflection ${i}`,
|
||||
compiled_truth: `Sample reflection content ${i} discussing recurring theme of work-life balance.`,
|
||||
timeline: '',
|
||||
frontmatter: { type: 'note', title: `Reflection ${i}` },
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
describe('E2E patterns — disabled', () => {
|
||||
test('skipped when dream.patterns.enabled=false', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.patterns.enabled', 'false');
|
||||
const result = await runPhasePatterns(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('disabled');
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('default-enabled when config key unset', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
// No reflections seeded → falls through to insufficient_evidence,
|
||||
// not disabled. Confirms the default-true semantics.
|
||||
const result = await runPhasePatterns(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('insufficient_evidence');
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E patterns — insufficient_evidence', () => {
|
||||
test('skipped with 0 reflections', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
const result = await runPhasePatterns(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('insufficient_evidence');
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('skipped with reflections below min_evidence', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.patterns.min_evidence', '5');
|
||||
await seedReflections(rig.engine, 3); // below 5
|
||||
const result = await runPhasePatterns(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('insufficient_evidence');
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E patterns — no API key', () => {
|
||||
test('enough reflections, no ANTHROPIC_API_KEY → skipped no_api_key', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await seedReflections(rig.engine, 5); // above default min_evidence (3)
|
||||
await withoutAnthropicKey(async () => {
|
||||
const result = await runPhasePatterns(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('no_api_key');
|
||||
});
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E patterns — dry-run', () => {
|
||||
test('dry-run returns ok with reflections_considered and zero patterns_written', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await seedReflections(rig.engine, 5);
|
||||
const result = await runPhasePatterns(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: true,
|
||||
});
|
||||
expect(result.status).toBe('ok');
|
||||
expect((result.details as { dryRun: boolean }).dryRun).toBe(true);
|
||||
expect((result.details as { reflections_considered: number }).reflections_considered).toBe(5);
|
||||
expect((result.details as { patterns_written: number }).patterns_written).toBe(0);
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,246 @@
|
||||
/**
|
||||
* E2E synthesize phase — PGLite, no API key required.
|
||||
*
|
||||
* Each test creates and tears down its own PGLite engine to avoid
|
||||
* cross-test contention. Trades startup cost for isolation — required
|
||||
* because PGLite's WASM instance has been observed to wedge under
|
||||
* sustained concurrent-test pressure on macOS (CLAUDE.md issue #223).
|
||||
*
|
||||
* Mirrors the per-test-rig pattern used in
|
||||
* test/e2e/dream-allow-list-pglite.test.ts.
|
||||
*
|
||||
* Run: bun test test/e2e/dream-synthesize-pglite.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { runPhaseSynthesize } from '../../src/core/cycle/synthesize.ts';
|
||||
|
||||
interface TestRig {
|
||||
engine: PGLiteEngine;
|
||||
brainDir: string;
|
||||
corpusDir: string;
|
||||
cleanup: () => Promise<void>;
|
||||
}
|
||||
|
||||
async function setupRig(): Promise<TestRig> {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite' } as never);
|
||||
await engine.initSchema();
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-synth-brain-'));
|
||||
const corpusDir = mkdtempSync(join(tmpdir(), 'gbrain-synth-corpus-'));
|
||||
return {
|
||||
engine,
|
||||
brainDir,
|
||||
corpusDir,
|
||||
cleanup: async () => {
|
||||
try { await engine.disconnect(); } catch { /* best-effort */ }
|
||||
try { rmSync(brainDir, { recursive: true, force: true }); } catch { /* */ }
|
||||
try { rmSync(corpusDir, { recursive: true, force: true }); } catch { /* */ }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `body` with ANTHROPIC_API_KEY temporarily cleared, restoring the
|
||||
* prior value (set or unset) on return — even on throw — so this never
|
||||
* leaks state to sibling test files in the suite.
|
||||
*/
|
||||
async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
|
||||
const saved = process.env.ANTHROPIC_API_KEY;
|
||||
delete process.env.ANTHROPIC_API_KEY;
|
||||
try {
|
||||
return await body();
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.ANTHROPIC_API_KEY;
|
||||
else process.env.ANTHROPIC_API_KEY = saved;
|
||||
}
|
||||
}
|
||||
|
||||
describe('E2E synthesize — disabled / not_configured', () => {
|
||||
test('not_configured when enabled=false (default)', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('not_configured');
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('not_configured when enabled=true but session_corpus_dir is empty', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('not_configured');
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E synthesize — empty corpus', () => {
|
||||
test('ok status with zero transcripts when corpus dir is empty', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('ok');
|
||||
expect((result.details as { transcripts_processed: number }).transcripts_processed).toBe(0);
|
||||
expect((result.details as { pages_written: number }).pages_written).toBe(0);
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E synthesize — no API key skip path', () => {
|
||||
test('without ANTHROPIC_API_KEY, every transcript verdict is "no key" and zero pages written', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
writeFileSync(
|
||||
join(rig.corpusDir, '2026-04-25-session.txt'),
|
||||
'a meaningful conversation\n'.repeat(200),
|
||||
);
|
||||
await withoutAnthropicKey(async () => {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('ok');
|
||||
expect((result.details as { transcripts_processed: number }).transcripts_processed).toBe(0);
|
||||
expect((result.details as { pages_written: number }).pages_written).toBe(0);
|
||||
const verdicts = (result.details as { verdicts: Array<{ worth: boolean; reasons: string[] }> }).verdicts;
|
||||
expect(verdicts).toHaveLength(1);
|
||||
expect(verdicts[0].worth).toBe(false);
|
||||
expect(verdicts[0].reasons[0]).toMatch(/ANTHROPIC_API_KEY/);
|
||||
});
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E synthesize — dry-run skips Sonnet (Codex finding #8)', () => {
|
||||
test('dry-run reports planned action with zero pages_written', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
writeFileSync(
|
||||
join(rig.corpusDir, '2026-04-25-session.txt'),
|
||||
'a meaningful conversation\n'.repeat(200),
|
||||
);
|
||||
await withoutAnthropicKey(async () => {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: true,
|
||||
});
|
||||
expect(result.status).toBe('ok');
|
||||
expect((result.details as { dryRun: boolean }).dryRun).toBe(true);
|
||||
expect((result.details as { pages_written: number }).pages_written).toBe(0);
|
||||
expect(result.summary).toMatch(/dry-run/);
|
||||
});
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E synthesize — cooldown', () => {
|
||||
test('cooldown_active when last_completion_ts is fresh', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
await rig.engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString());
|
||||
await rig.engine.setConfig('dream.synthesize.cooldown_hours', '12');
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('skipped');
|
||||
expect((result.details as { reason?: string }).reason).toBe('cooldown_active');
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('explicit --input bypasses cooldown', async () => {
|
||||
// Two engine setups + a synth run; default 5s is tight under full-suite pressure.
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
await rig.engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString());
|
||||
const adHoc = join(tmpdir(), `gbrain-synth-ad-hoc-${Date.now()}-${Math.random().toString(36).slice(2)}.txt`);
|
||||
writeFileSync(adHoc, 'hello world '.repeat(300));
|
||||
try {
|
||||
await withoutAnthropicKey(async () => {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
inputFile: adHoc,
|
||||
});
|
||||
expect(result.status).toBe('ok');
|
||||
expect((result.details as { reason?: string }).reason).toBeUndefined();
|
||||
});
|
||||
} finally {
|
||||
rmSync(adHoc, { force: true });
|
||||
}
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe('E2E synthesize — verdict cache (Q-2)', () => {
|
||||
test('subsequent run with same content reads from dream_verdicts cache', async () => {
|
||||
// Two synth runs through the verdict-cache path; default 5s is tight.
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
const filePath = join(rig.corpusDir, '2026-04-25-session.txt');
|
||||
const body = 'a meaningful conversation\n'.repeat(200);
|
||||
writeFileSync(filePath, body);
|
||||
await withoutAnthropicKey(async () => {
|
||||
await runPhaseSynthesize(rig.engine, { brainDir: rig.brainDir, dryRun: false });
|
||||
const { createHash } = await import('node:crypto');
|
||||
const hash = createHash('sha256').update(body, 'utf8').digest('hex');
|
||||
await rig.engine.putDreamVerdict(filePath, hash, {
|
||||
worth_processing: false,
|
||||
reasons: ['cached test verdict'],
|
||||
});
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
expect(result.status).toBe('ok');
|
||||
const verdicts = (result.details as { verdicts: Array<{ cached: boolean }> }).verdicts;
|
||||
expect(verdicts).toHaveLength(1);
|
||||
expect(verdicts[0].cached).toBe(true);
|
||||
});
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,163 @@
|
||||
/**
|
||||
* IRON RULE security regression guard for the v0.21 trusted-workspace
|
||||
* allow-list path on put_page.
|
||||
*
|
||||
* Covers:
|
||||
* - matchesSlugAllowList glob semantics (ALLOW + REJECT + recursive globs)
|
||||
* - put_page accepts when slug matches allow-list
|
||||
* - put_page rejects when slug is outside allow-list
|
||||
* - put_page falls back to legacy `wiki/agents/<id>/...` namespace check
|
||||
* when allowed_slug_prefixes is unset (regression guard for v0.15
|
||||
* anti-prompt-injection guarantee)
|
||||
* - put_page rejects when viaSubagent=true but subagentId is missing
|
||||
* (regression guard for FAIL-CLOSED behavior)
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { matchesSlugAllowList, operations, OperationError, type OperationContext } from '../src/core/operations.ts';
|
||||
|
||||
const STUB_LOGGER = {
|
||||
info: () => {},
|
||||
warn: () => {},
|
||||
error: () => {},
|
||||
};
|
||||
|
||||
const STUB_CONFIG = {} as unknown as Parameters<typeof operations[number]['handler']>[0]['config'];
|
||||
|
||||
function findOp(name: string) {
|
||||
const op = operations.find(o => o.name === name);
|
||||
if (!op) throw new Error(`operation ${name} not found`);
|
||||
return op;
|
||||
}
|
||||
|
||||
// Stub engine that fails loudly if put_page actually reaches importFromContent.
|
||||
// We expect every test in this file to short-circuit at the namespace/allow-list
|
||||
// check, so every engine method throws a recognizable error that lets us assert
|
||||
// "got past the gate" if it ever happens.
|
||||
function stubEngine() {
|
||||
return new Proxy({} as never, {
|
||||
get(_target, prop: string) {
|
||||
return () => { throw new Error(`engine.${prop} should not have been called — gate failed`); };
|
||||
},
|
||||
}) as Parameters<typeof operations[number]['handler']>[0]['engine'];
|
||||
}
|
||||
|
||||
function makeCtx(overrides: Partial<OperationContext> = {}): OperationContext {
|
||||
return {
|
||||
engine: stubEngine(),
|
||||
config: STUB_CONFIG,
|
||||
logger: STUB_LOGGER,
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
viaSubagent: true,
|
||||
subagentId: 42,
|
||||
jobId: 100,
|
||||
...overrides,
|
||||
} as OperationContext;
|
||||
}
|
||||
|
||||
describe('matchesSlugAllowList — glob semantics', () => {
|
||||
test('exact match (no glob suffix)', () => {
|
||||
expect(matchesSlugAllowList('foo/bar', ['foo/bar'])).toBe(true);
|
||||
expect(matchesSlugAllowList('foo/bar/baz', ['foo/bar'])).toBe(false);
|
||||
});
|
||||
|
||||
test('shallow glob: prefix/* matches any single direct child segment', () => {
|
||||
expect(matchesSlugAllowList('wiki/personal/reflections/2026-04-25-arete-paradox-a3f8c1',
|
||||
['wiki/personal/reflections/*'])).toBe(true);
|
||||
expect(matchesSlugAllowList('wiki/personal/reflections',
|
||||
['wiki/personal/reflections/*'])).toBe(false);
|
||||
});
|
||||
|
||||
test('recursive: prefix/* matches deep children too', () => {
|
||||
expect(matchesSlugAllowList('wiki/originals/ideas/2026-04-25-foo',
|
||||
['wiki/originals/*'])).toBe(true);
|
||||
expect(matchesSlugAllowList('wiki/originals/ideas/foo/bar',
|
||||
['wiki/originals/*'])).toBe(true);
|
||||
});
|
||||
|
||||
test('rejects slugs outside every prefix', () => {
|
||||
const list = [
|
||||
'wiki/personal/reflections/*',
|
||||
'wiki/originals/*',
|
||||
];
|
||||
expect(matchesSlugAllowList('wiki/finance/secret', list)).toBe(false);
|
||||
expect(matchesSlugAllowList('wiki/people/alice', list)).toBe(false);
|
||||
});
|
||||
|
||||
test('empty list rejects everything', () => {
|
||||
expect(matchesSlugAllowList('wiki/anything', [])).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT match prefix without trailing segment', () => {
|
||||
expect(matchesSlugAllowList('wiki/personal/reflections',
|
||||
['wiki/personal/reflections/*'])).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('put_page — trusted-workspace allow-list', () => {
|
||||
const put_page = findOp('put_page');
|
||||
|
||||
test('REJECTS when slug is outside the allow-list', async () => {
|
||||
const ctx = makeCtx({
|
||||
allowedSlugPrefixes: ['wiki/personal/reflections/*', 'wiki/originals/*'],
|
||||
});
|
||||
await expect(put_page.handler(ctx, {
|
||||
slug: 'wiki/finance/secret',
|
||||
content: '---\ntitle: x\n---\nbody',
|
||||
})).rejects.toMatchObject({
|
||||
code: 'permission_denied',
|
||||
});
|
||||
});
|
||||
|
||||
test('REJECTS path-traversal-like slug (slug regex catches it earlier in the import path; allow-list also catches via no-match)', async () => {
|
||||
const ctx = makeCtx({
|
||||
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
|
||||
});
|
||||
// The slug regex in validatePageSlug rejects `..`; here we test the
|
||||
// allow-list layer specifically with a slug that LOOKS legal but isn't on the list.
|
||||
await expect(put_page.handler(ctx, {
|
||||
slug: 'wiki/people/garry-tan',
|
||||
content: '---\ntitle: x\n---\nbody',
|
||||
})).rejects.toMatchObject({
|
||||
code: 'permission_denied',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('put_page — legacy namespace check (regression guard)', () => {
|
||||
const put_page = findOp('put_page');
|
||||
|
||||
test('REJECTS write outside wiki/agents/<id>/ when allow-list is unset', async () => {
|
||||
// The v0.15 anti-prompt-injection guarantee: subagent without explicit
|
||||
// allow-list MUST be confined to its own agent namespace. This test
|
||||
// ensures v0.21 doesn't regress that boundary.
|
||||
const ctx = makeCtx({ allowedSlugPrefixes: undefined });
|
||||
await expect(put_page.handler(ctx, {
|
||||
slug: 'wiki/personal/reflections/2026-04-25-foo',
|
||||
content: '---\ntitle: x\n---\nbody',
|
||||
})).rejects.toMatchObject({
|
||||
code: 'permission_denied',
|
||||
});
|
||||
});
|
||||
|
||||
test('REJECTS write outside wiki/agents/<id>/ when allow-list is empty array', async () => {
|
||||
const ctx = makeCtx({ allowedSlugPrefixes: [] });
|
||||
await expect(put_page.handler(ctx, {
|
||||
slug: 'wiki/personal/reflections/2026-04-25-foo',
|
||||
content: '---\ntitle: x\n---\nbody',
|
||||
})).rejects.toMatchObject({
|
||||
code: 'permission_denied',
|
||||
});
|
||||
});
|
||||
|
||||
test('REJECTS when viaSubagent=true but subagentId is missing (FAIL-CLOSED)', async () => {
|
||||
const ctx = makeCtx({ subagentId: undefined as unknown as number, allowedSlugPrefixes: undefined });
|
||||
await expect(put_page.handler(ctx, {
|
||||
slug: 'wiki/agents/42/foo',
|
||||
content: '---\ntitle: x\n---\nbody',
|
||||
})).rejects.toMatchObject({
|
||||
code: 'permission_denied',
|
||||
});
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user