Files
gbrain/test/e2e/dream-allow-list-pglite.test.ts
T
527b87bd1e v0.23.0 feat: gbrain dream synthesizes conversations into brain pages (v0.23.0) (#462)
* feat: dream_verdicts schema + engine methods

Adds the v25 schema migration creating the dream_verdicts table
(file_path, content_hash, worth_processing, reasons, judged_at;
PRIMARY KEY (file_path, content_hash); RLS-enabled when running as
a BYPASSRLS role).

Distinct from raw_data (which is page-scoped) — transcripts being
judged for synthesis aren't pages. The (file_path, content_hash)
key means edited transcripts re-judge automatically.

BrainEngine gains:
- DreamVerdict + DreamVerdictInput types
- getDreamVerdict(filePath, contentHash) → DreamVerdict | null
- putDreamVerdict(filePath, contentHash, verdict) — ON CONFLICT upsert

Both engines implement (postgres-engine.ts, pglite-engine.ts).

This commit alone is functionally inert — nothing reads/writes the
table yet. The synthesize phase (later commit) is the consumer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: trusted-workspace allow-list for subagent put_page

Adds OperationContext.allowedSlugPrefixes — when set, put_page
enforces slug membership in the allow-list instead of the legacy
wiki/agents/<id>/... namespace. The trust signal is the SUBMITTER
(PROTECTED_JOB_NAMES gates subagent submission so MCP can't reach
this field), not the runtime ctx.remote flag — every subagent tool
call has remote=true for auto-link safety, so basing trust on
remote is incoherent.

matchesSlugAllowList(slug, prefixes) helper supports glob suffix
'/*' (recursive — wiki/originals/* matches ideas/foo/bar) and
exact match for unsuffixed entries.

put_page check shape:
  if (viaSubagent && allowedSlugPrefixes set) → allow-list check
  else if (viaSubagent) → existing namespace check (regression guard)
  else → no check (regular CLI)

Auto-link is re-enabled for the trusted-workspace path so the cycle's
extract phase doesn't have to recompute every edge after synthesize
writes. Untrusted remote writes still skip auto-link as before.

SubagentHandlerData.allowed_slug_prefixes is the wire field; the
synthesize/patterns phases (later commit) populate it from a single
source of truth in skills/_brain-filing-rules.json's
dream_synthesize_paths.globs array. The model's tool schema description
mirrors the allow-list so it writes correct slugs on the first try.

IRON RULE security tests:
- test/operations-allow-list.test.ts: allow-list ALLOW/REJECT, glob
  semantics, regression guard for the v0.15 namespace fallback when
  allow-list is unset, FAIL-CLOSED when subagentId is missing.
- test/e2e/dream-allow-list-pglite.test.ts: end-to-end on PGLite,
  poisoned-transcript style write outside allow-list → REJECTED.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: cycle scaffolding — 8-phase order + transcript discovery

Extends ALL_PHASES from 6 → 8: synthesize between sync and extract,
patterns between extract and embed. Codex finding #7: patterns MUST
run after extract because subagent put_page sets ctx.remote=true and
skips auto-link/timeline by default — extract is the canonical edge
materialization step. Without that ordering, patterns reads stale
graph state.

Final order:
  lint → backlinks → sync → synthesize → extract → patterns → embed → orphans

CycleOpts gains:
- yieldDuringPhase callback — generic in-phase keepalive for long
  waits (synthesize fan-out, patterns roll-up). Renews cycle-lock TTL
  + worker job lock. Mirrors yieldBetweenPhases shape.
- synthInputFile / synthDate / synthFrom / synthTo — forwarded to
  runPhaseSynthesize for the CLI's --input/--date/--from/--to flags.

CycleReport.totals additively grows (no schema_version bump):
  transcripts_processed, synth_pages_written, patterns_written.

src/core/cycle/transcript-discovery.ts is a pure filesystem walk:
- .txt files only, sorted by path for determinism
- date-prefixed basename filter (--date / --from / --to)
- min_chars filter (default 2000)
- exclude_patterns auto-wraps bare words as \b<word>\b regex (Q-3),
  power users may pass full regex with anchors
- compileExcludePatterns is exported for unit tests

Phase implementations land in the next commit; this one only adds
the dispatcher slots so commit-by-commit bisect doesn't crash on
import-not-found.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: synthesize + patterns phases — gbrain dream actually dreams

Synthesize phase (src/core/cycle/synthesize.ts) reads conversation
transcripts from dream.synthesize.session_corpus_dir and writes
brain-native pages: reflections to wiki/personal/reflections/...,
originals to wiki/originals/ideas/..., timeline entries on existing
people pages.

Pipeline:
  1. discoverTranscripts (filesystem walk + filters)
  2. cooldown check via dream.synthesize.last_completion_ts config
     (default 12h; bypassed by --input/--date/--from/--to)
  3. cheap Haiku verdict per transcript, cached in dream_verdicts
     table keyed by (file_path, content_hash) — backfill re-runs
     skip already-judged transcripts at zero cost
  4. fan-out: one Sonnet subagent per worth-processing transcript
     dispatched with allowed_slug_prefixes (read from
     skills/_brain-filing-rules.json's dream_synthesize_paths.globs)
     and idempotency_key dream:synth:<file_path>:<content_hash>
  5. wait via waitForCompletion; yieldDuringPhase ticks every child
     terminal so the cycle-lock TTL refreshes on long backfills
  6. collect slugs from subagent_tool_executions for each child
     (codex finding #2: NOT pages.updated_at, which would pick up
     unrelated writes)
  7. orchestrator dual-write — query each new page from DB,
     reverse-render via serializeMarkdown, write file to brain_dir.
     Subagent never gets fs-write access.
  8. deterministic summary index page at dream-cycle-summaries/<date>
     (codex finding #4: slug shape is regex-compatible — no
     underscores, no .md extension)
  9. write completion timestamp ONLY on successful runs

Patterns phase (src/core/cycle/patterns.ts) runs after extract so
the graph state is fresh. Single Sonnet subagent gathers reflections
within dream.patterns.lookback_days (default 30); names a pattern
only when ≥dream.patterns.min_evidence (default 3) reflections
support it. Same allow-list path as synthesize.

CLI flags on `gbrain dream` (src/commands/dream.ts):
  --input <file>      ad-hoc transcript synthesis (implies
                      --phase synthesize; bypasses cooldown)
  --date YYYY-MM-DD   restrict synthesize to one date
  --from <d> --to <d> backfill range
  --dry-run           runs Haiku verdict (cached), skips Sonnet
                      synthesis. NOT zero LLM calls (codex #8).

Conflict detection: --input + --date/--from/--to exits 2.
ISO 8601 date format validated; range start > end exits 2.

Auto-commit / push deferred to v1.1 (codex finding #5). v1 writes
files to brain_dir; user or autopilot handles git.

Tests:
- test/cycle-patterns.test.ts: structural assertions on the patterns
  phase (queue + waitForCompletion wired, allow-list threading,
  subagent_tool_executions provenance, no raw_data dependency).
- test/dream-cli-flags.test.ts: argv parsing, conflict detection,
  ISO date validation, --input implies --phase synthesize, dry-run
  semantics doc string.
- test/e2e/dream-synthesize-pglite.test.ts: 8 cases on PGLite
  in-memory exercising not_configured, empty corpus, no API key
  skip path, dry-run, cooldown active vs --input bypass, and the
  dream_verdicts cache hit path. Per-test rig isolation (each
  test creates and tears down its own engine) avoids
  cross-test PGLite WASM contention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: dream cycle v0.27.0 — skills, CLAUDE.md, migration, changelog

- skills/maintain/SKILL.md: synthesize + patterns phases documented
  with quality bar (Iron Law for synthesis), trust boundary, idempotency,
  cooldown semantics, CLI invocation patterns. New triggers added so
  "process today's session" / "synthesize my conversations" route here.
- skills/RESOLVER.md: dream cycle triggers route to maintain.
- skills/_brain-filing-rules.md: directory table for the five output
  types (reflections, originals, patterns, people enrichment, cycle
  summary) with slug shape per row; Iron Law repeated.
- skills/migrations/v0.27.0.md: agent-readable migration narrative.
  Schema migration v25 runs automatically on `gbrain apply-migrations`;
  synthesize ships disabled by default — opt-in via
  dream.synthesize.session_corpus_dir + dream.synthesize.enabled.
- CLAUDE.md: file inventory updated with new files (cycle/synthesize.ts,
  cycle/patterns.ts, cycle/transcript-discovery.ts), the 8-phase
  ordering, the trusted-workspace allow-list trust model, and the v25
  schema migration line in the migrate.ts entry.
- VERSION: 0.20.4 → 0.27.0
- CHANGELOG.md: v0.27.0 release-summary section per CLAUDE.md voice
  rules (numbers that matter table, what-this-means closer, "to take
  advantage of" block), followed by the itemized changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: add patterns E2E + 8-phase cycle E2E + bump synth-cooldown timeouts

Two new E2E test files on PGLite (no DATABASE_URL or API key required):

- test/e2e/dream-patterns-pglite.test.ts (6 cases) — exercises
  runPhasePatterns skip paths against a real engine: disabled,
  default-enabled-but-insufficient-evidence, no-API-key, dry-run.
  Sibling of dream-synthesize-pglite.test.ts; same per-test rig
  pattern for engine isolation.

- test/e2e/dream-cycle-eight-phase-pglite.test.ts (5 cases) —
  end-to-end runCycle with the v0.27 8-phase order. Asserts:
  ALL_PHASES is the documented 8 phases in the right sequence,
  the dry-run report's phases array preserves that order,
  CycleReport.totals carries the new transcripts_processed /
  synth_pages_written / patterns_written fields, --phase synthesize
  and --phase patterns each run only that phase, and synthInputFile
  is plumbed correctly through runCycle to runPhaseSynthesize.

Bump per-test timeout to 30s on the two synthesize-cooldown E2E
tests that create two PGLite engines back-to-back. Default Bun 5s
budget is tight under sustained suite pressure (PGLite WASM init
costs ~1-2s per engine on macOS); each test passes alone but flakes
in the full E2E suite. The third arg `30_000` is Bun's standard
test-timeout knob.

Full E2E suite (test/e2e/) now: 86 pass / 0 fail / 258 skip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: ship-prep — typecheck fixes, llms.txt regen, 8-phase test update

- src/core/cycle/synthesize.ts + patterns.ts: PageType 'default' → 'note'
  (TS strict typecheck rejected 'default'; 'note' is a valid PageType
  for orchestrator-written summary index pages and reverse-render fallback).
- src/core/pglite-engine.ts: re-import DreamVerdict + DreamVerdictInput
  types after the master merge dropped them from the import line.
- test/e2e/dream-allow-list-pglite.test.ts: ToolCtx now requires
  remote: true literal; thread it through every put_page tool call.
- test/e2e/dream-patterns-pglite.test.ts: PageType 'default' → 'note'
  in the seedReflections helper.
- test/core/cycle.test.ts: bump expected hook-call count + phase count
  6 → 8 to match v0.27 ALL_PHASES extension.
- llms-full.txt: regenerate against the updated CHANGELOG + CLAUDE.md
  so the committed snapshot matches what the generator now produces.

Full bun test suite: 2793 pass / 0 fail / 258 skip (3051 tests, 177 files).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README + INSTALL_FOR_AGENTS for v0.27.0 dream cycle

README: maintain skill row mentions synthesize/patterns; gbrain dream
command-reference block describes the 8-phase pipeline and the new
--input/--date/--from/--to flags.

INSTALL_FOR_AGENTS: dream cycle bullet calls out v0.27 conversation
synthesis + cross-session pattern detection.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: renumber v0.27.0 → v0.23.0

Master is at v0.22.5; v0.23.0 is the next natural slot for the dream-cycle
synthesize + patterns release. Bulk rename across VERSION, package.json,
CHANGELOG, migration file, source comments, skills, and llms.txt bundles.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): bump cycle.test.ts phase count 6 → 8

The dry-run full-cycle test asserted 6 phases. v0.23 added synthesize
and patterns, bringing the total to 8. The unit-side equivalent
(test/core/cycle.test.ts) was already updated; this catches the
E2E sibling that surfaced after the latest master merge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 01:23:29 -07:00

160 lines
5.7 KiB
TypeScript

/**
* E2E security regression: poisoned-transcript guard for the v0.21
* trusted-workspace allow-list.
*
* Runs against PGLite in-memory (no DATABASE_URL required). Builds the
* brain tool registry with `allowed_slug_prefixes` set the same way the
* synthesize phase does, then calls the put_page tool with slugs that
* are inside / outside the allow-list. Asserts:
*
* - In-allow-list slug → page is written to the DB
* - Outside-allow-list slug → tool throws permission_denied
* - When allow-list is unset (legacy), put_page is bounded to
* wiki/agents/<id>/... (regression guard for the v0.15 anti-prompt-
* injection guarantee)
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { buildBrainTools } from '../../src/core/minions/tools/brain-allowlist.ts';
import type { GBrainConfig } from '../../src/core/config.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
const config = {} as unknown as GBrainConfig;
const PUT_PAGE_TOOL = 'brain_put_page';
const SAMPLE_BODY = '---\ntitle: A reflection\ntype: default\n---\n\nbody text\n';
function findPutPageTool(tools: Awaited<ReturnType<typeof buildBrainTools>>) {
const t = tools.find(x => x.name === PUT_PAGE_TOOL);
if (!t) throw new Error('brain_put_page tool not found in registry');
return t;
}
describe('E2E allow-list — trusted-workspace path', () => {
test('ALLOW: subagent put_page within allow-list writes the page', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
});
const tool = findPutPageTool(tools);
await tool.execute(
{ slug: 'wiki/personal/reflections/2026-04-25-arete-paradox-a3f8c1', content: SAMPLE_BODY },
{ engine, jobId: 7777, remote: true },
);
const page = await engine.getPage('wiki/personal/reflections/2026-04-25-arete-paradox-a3f8c1');
expect(page).not.toBeNull();
expect(page!.title).toBe('A reflection');
});
test('REJECT: subagent put_page outside allow-list throws permission_denied', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
});
const tool = findPutPageTool(tools);
let threw = false;
try {
await tool.execute(
{ slug: 'wiki/finance/secret-market-data', content: SAMPLE_BODY },
{ engine, jobId: 7778, remote: true },
);
} catch (e) {
threw = true;
const msg = e instanceof Error ? e.message : String(e);
expect(msg).toMatch(/allow-list/i);
}
expect(threw).toBe(true);
const page = await engine.getPage('wiki/finance/secret-market-data');
expect(page).toBeNull(); // never reached the engine
});
test('Multiple prefixes: each slug evaluated independently', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
allowedSlugPrefixes: ['wiki/personal/reflections/*', 'wiki/originals/*'],
});
const tool = findPutPageTool(tools);
await tool.execute(
{ slug: 'wiki/originals/ideas/2026-04-25-thousand-pound-armor', content: SAMPLE_BODY },
{ engine, jobId: 7779, remote: true },
);
expect(await engine.getPage('wiki/originals/ideas/2026-04-25-thousand-pound-armor')).not.toBeNull();
});
});
describe('E2E allow-list — legacy namespace fallback', () => {
test('REGRESSION GUARD: when allow-list is unset, put_page rejects writes outside wiki/agents/<id>/', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
// allowedSlugPrefixes intentionally omitted — exercises the v0.15
// legacy namespace check that v0.21 must NOT regress.
});
const tool = findPutPageTool(tools);
let threw = false;
try {
await tool.execute(
{ slug: 'wiki/personal/reflections/2026-04-25-bypass-attempt', content: SAMPLE_BODY },
{ engine, jobId: 7780, remote: true },
);
} catch (e) {
threw = true;
const msg = e instanceof Error ? e.message : String(e);
expect(msg).toMatch(/wiki\/agents\/999/);
}
expect(threw).toBe(true);
});
test('When allow-list unset, slug under wiki/agents/<id>/ is allowed', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
});
const tool = findPutPageTool(tools);
await tool.execute(
{ slug: 'wiki/agents/999/scratch-note', content: SAMPLE_BODY },
{ engine, jobId: 7781, remote: true },
);
expect(await engine.getPage('wiki/agents/999/scratch-note')).not.toBeNull();
});
});
describe('E2E allow-list — provenance via tool execution rows (Codex #2)', () => {
test('subagent_tool_executions captures slug for each put_page call', async () => {
// The synthesize phase relies on this being queryable to determine
// exactly which slugs each child wrote (instead of pages.updated_at).
// We don't have a real subagent run here, but we can verify the table
// exists and the column shape supports the orchestrator's query.
const rows = await engine.executeRaw(
`SELECT column_name FROM information_schema.columns
WHERE table_name = 'subagent_tool_executions'
ORDER BY column_name`,
) as Array<{ column_name: string }>;
const cols = rows.map(r => r.column_name);
expect(cols).toContain('input');
expect(cols).toContain('tool_name');
expect(cols).toContain('status');
expect(cols).toContain('job_id');
});
});