mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 03:12:32 +00:00
c250a4d0142e569ab49dacd9bb123788fa546751
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
058fe69575 |
v0.26.7 test: isolation foundation (helpers + lint + quarantine) (#613)
* test: add withEnv helper + canonical PGLite block JSDoc withEnv(overrides, fn) saves prior values, runs the callback, restores via try/finally — including on throw. Handles delete via undefined override. Nested calls compose. Cross-test safe; explicitly NOT intra-file concurrent-safe (process.env is process-global). 7 unit cases covering sync, async, delete-key, delete-when-prior-unset, restore-on-throw, nested compose, multi-key atomic restore. reset-pglite.ts JSDoc extended with the canonical 4-line PGLite block (beforeAll create + afterAll disconnect + beforeEach reset). The lint script in the next commit enforces this exact shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: add check-test-isolation lint script + wire into verify Grep-based lint enforcing 4 rules on non-serial unit test files: R1: no process.env mutations (use withEnv() or rename to *.serial.test.ts) R2: no mock.module() (rename to *.serial.test.ts) R3: new PGLiteEngine( only inside beforeAll() context R4: PGLiteEngine creators must pair with afterAll{disconnect} Wired into 'bun run verify' and 'bun run check:all' (NOT 'bun run test' which is the parallel runner script with no pre-check chain). Matches the existing scripts/check-*.sh family shape (jsonb, progress, etc). 51 baseline violators captured in scripts/check-test-isolation.allowlist. List MUST shrink over time — entries removed by v0.26.8 (env sweep) and v0.26.9 (PGLite sweep). New files cannot be added. CLAUDE.md ## Testing section extended with R1-R4 rules table, the canonical 4-line PGLite block, withEnv pattern, and when-to-quarantine guidance. 16 fixture-driven test cases for the lint: clean, R1 (5 patterns + 1 negative), R2, R3 (top-level vs in-beforeAll), R4 (missing disconnect), *.serial.test.ts skip, test/e2e/ skip, allowlist (3 cases). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: quarantine cycle and embed mock.module test files Both files use mock.module(...) at top level — leaks across files in the same shard process. The check-test-isolation lint (R2) bans this pattern in non-serial files; quarantine is the escape hatch. Per v0.26.7 plan D5: prefer quarantine over DI on runCycle/runEmbed. Production signatures stay frozen; tests run at --max-concurrency=1 in the serial post-pass (the existing pattern shipped in v0.26.4 for brain-registry and reconcile-links). Quarantine count: 2 → 4. Cap raised to 10 informational per D15. Renames: test/core/cycle.test.ts → test/core/cycle.serial.test.ts test/embed.test.ts → test/embed.serial.test.ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.26.7) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: post-ship documentation sync for v0.26.7 - README.md "Contributing" line: point to bun run test + bun run verify (parallel fast loop) - CONTRIBUTING.md "Running tests": rewrite for the v0.26.4/v0.26.7 test surface (parallel runner, verify, slow/serial/e2e tiers) - CONTRIBUTING.md adds "Writing tests that survive the parallel loop" section: R1-R4 lint, canonical PGLite block, withEnv pattern, when to quarantine - llms-full.txt regenerated to pick up the README + CONTRIBUTING changes Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
0de9eb68ba |
v0.26.5 feat: destructive operation guard end-to-end (sources + pages + autopilot purge) (#600)
* feat(v0.26.5): destructive operation guard — impact preview, confirmation gate, soft-delete
Three-layer protection against accidental data loss:
1. **Impact preview**: Every destructive operation (sources remove, purge)
now shows a formatted preview of exactly what will be destroyed —
page count, chunk count, embedding count, file count — BEFORE acting.
2. **--confirm-destructive flag**: `--yes` alone is no longer sufficient
when a source has data. Must pass `--confirm-destructive` to proceed
with permanent deletion. Prevents scripted/reflexive destroys.
3. **Soft-delete with 72h TTL**: New `gbrain sources archive <id>`
hides a source from search and federation without destroying any data.
Data preserved for 72 hours. Restorable via `gbrain sources restore <id>`.
Expired archives purged via `gbrain sources purge`.
New subcommands:
- `gbrain sources archive <id>` — soft-delete (hide, preserve 72h)
- `gbrain sources restore <id>` — un-archive, re-federate
- `gbrain sources archived` — list soft-deleted sources + TTL
- `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete
Behavioral changes:
- `sources remove` with data now requires `--confirm-destructive` (not just `--yes`)
- `sources remove --dry-run` shows full impact preview without side effects
- Impact box format shows source name, id, and all cascade counts
New files:
- src/core/destructive-guard.ts — impact assessment, confirmation gate,
soft-delete/restore/purge logic, display formatters
* chore(release): v0.26.5 — destructive operation guard
Bump VERSION + package.json to 0.26.5 and add the v0.26.5 CHANGELOG entry
on top of the destructive-guard feature commit cherry-picked from PR #595.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.26.5): page-level soft-delete + autopilot purge + search visibility
Closes the destructive-guard posture across every gbrain destructive surface.
PR #595 cherry-pick covered the CLI source-remove path; this commit closes
the higher-velocity MCP `delete_page` agent footgun and the three internal
correctness gaps the CEO+Eng review surfaced:
- Gap 1: archived sources were not actually filtered from search. Now they
are, via `buildVisibilityClause` in `searchKeyword`/`searchKeywordChunks`/
`searchVector` for both engines.
- Gap 2: 72h TTL was honor-system. Now wired into a new autopilot `purge`
phase (9th in ALL_PHASES) that calls `purgeExpiredSources` + `engine.
purgeDeletedPages(72)`. Manual escape hatch: `gbrain pages purge-deleted`.
- Gap 3: zero tests for safety-critical code. ~30 cases now in
`test/destructive-guard.test.ts`, `test/pages-soft-delete.test.ts`, and
`test/sql-ranking.test.ts` covering the boundary truth table, JSONB→column
migration, soft-delete/restore/purge round-trip, multi-source isolation,
cascade verification, and the Q3 IRON-rule contract test.
Schema migration v33 (`destructive_guard_columns`): adds `pages.deleted_at`
+ partial purge index, promotes `archived` from `sources.config` JSONB to
real columns (`sources.archived BOOLEAN`, `archived_at`, `archive_expires_at`),
backfills any pre-v0.26.5 JSONB shape. Engine-aware: Postgres uses CREATE
INDEX CONCURRENTLY, PGLite uses plain CREATE INDEX. Forward-reference
bootstrap extended in both engines so pre-v0.26.5 brains don't crash on the
embedded-schema replay.
BrainEngine surface: new `softDeletePage` / `restorePage` /
`purgeDeletedPages` methods + `includeDeleted` flag on `getPage`/`listPages`.
MCP ops: `delete_page` rewired to soft-delete (description string updated);
new `restore_page` (scope: write) + `purge_deleted_pages` (scope: admin,
localOnly: true).
Q3 contract (eng-review lynchpin): `get_page(slug)` returns null for
soft-deleted by default; `get_page(slug, {include_deleted: true})` surfaces
the row with `deleted_at` populated. Same flag for `list_pages`. Mirrors
the search-filter contract end-to-end.
Issue 5 (eng-review): `archived` is now a real column on `sources`, not a
JSONB key. No reserved-key footgun. Faster filter. Visibility clause
compiles to a column lookup, not JSONB containment.
Verification:
- bun run typecheck: PASS
- bun run build:schema + bun run build:llms: regenerated
- targeted test runs: 90 pass / 0 fail across destructive-guard,
pages-soft-delete, sql-ranking, schema-bootstrap-coverage, build-llms
- full bun test: 16 pre-existing failures inherited from v0.26.2 (sync,
sync-parallel, queue-child-done, etc — already filed in TODOS.md as
"Fix 22 pre-existing test failures unrelated to OAuth")
CHANGELOG, CLAUDE.md (Key Files + Commands), TODOS.md updated. The plan
file at ~/.claude/plans/take-a-look-and-gentle-pine.md captures the full
review trail (CEO=C, Eng-Q3=A, Eng-Issue5=a, 8 defaults applied).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.26.5): CI fallout — getStats excludes soft-deleted; tests use --confirm-destructive
Two CI failures from the v0.26.5 ship:
1. **Tier 1 (Postgres E2E):** `E2E: Page CRUD > delete_page removes page and
others survive` failed because `delete_page` now soft-deletes (sets
deleted_at) but `getStats.page_count` was still counting all rows. The
test seeds 16 pages, deletes one, and asserts page_count is 15. Fix:
`getStats` now filters `WHERE deleted_at IS NULL` for page_count in both
engines. This matches the visibility-filter contract — soft-deleted pages
are hidden everywhere the user looks (search, get_page, list_pages, stats).
Chunks and links stay raw because they still occupy storage until the
autopilot purge phase runs.
2. **Test 2 (PGLite unit):** `multi-source-integration.test.ts:184` and
`e2e/multi-source.test.ts:274` called `runSources(engine, ['remove', X,
'--yes'])` against populated sources. v0.26.5's destructive guard rejects
`--yes` alone on populated sources and calls `process.exit(5)`, which
killed the bun test runner mid-suite (CI exit 5). Both test sites now
pass `--confirm-destructive` per the v0.26.5 contract.
Verification: 115/0 pass across destructive-guard, pages-soft-delete,
sql-ranking, schema-bootstrap-coverage, sources, repos-alias, and
multi-source-integration test files. typecheck PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): cycle phase count is 9 (v0.26.5 added `purge` phase)
CI failure: `runCycle — yieldBetweenPhases hook` tests asserted exactly 8
phases. v0.26.5 added the autopilot `purge` phase as the 9th, so:
- `test/core/cycle.test.ts:381` — `hookCalls` is now 9 (one yield per phase)
- `test/core/cycle.test.ts:392` — `report.phases.length` is now 9
- `test/e2e/cycle.test.ts:101` — same update for the dry-run E2E
The `purge` phase invocation was already visible in the failing log output:
the cycle ran 9 phases end-to-end; the test assertions hadn't been updated.
Verification: bun run typecheck PASS. cycle.test.ts: 28/0 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
3c032d79ec |
v0.26.0 feat: GBrain — MCP Keys OAuth 2.1 + HTTP server + admin dashboard (#358)
* feat: OAuth 2.1 schema tables + shared token utilities
Add oauth_clients, oauth_tokens, oauth_codes tables to both PGLite and
Postgres schemas. Migration v5 creates tables for existing databases.
PGLite now includes auth infrastructure (access_tokens, mcp_request_log,
OAuth tables) because `serve --http` makes it network-accessible.
Extract hashToken() and generateToken() to src/core/utils.ts for DRY
reuse across auth.ts and oauth-provider.ts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: GBrainOAuthProvider — MCP SDK OAuthServerProvider implementation
Implements OAuthServerProvider backed by raw SQL (PGLite or Postgres).
Supports client credentials, authorization code with PKCE, token refresh
with rotation, revocation, and legacy access_tokens fallback.
Key decisions from eng review:
- Uses raw SQL connection, not BrainEngine (OAuth is infrastructure)
- All tokens/secrets SHA-256 hashed before storage
- Legacy tokens grandfathered as read+write+admin
- sweepExpiredTokens() wrapped in try/catch (non-blocking startup)
- Client credentials: no refresh token per RFC 6749 4.4.3
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: scope + localOnly annotations on all 30 operations
Add AuthInfo, scope ('read'|'write'|'admin'), and localOnly fields to
Operation interface. Per-operation audit:
- 14 read ops, 9 write ops, 2 admin ops, 4 admin+localOnly ops
- sync_brain, file_upload, file_list, file_url: admin + localOnly
- Scope enforcement happens in serve-http.ts before handler dispatch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: HTTP MCP server with OAuth 2.1 + 27 OAuth tests
gbrain serve --http starts Express 5 server with:
- MCP SDK mcpAuthRouter (authorize, token, register, revoke endpoints)
- Custom client_credentials handler (SDK doesn't support CC grant)
- Bearer auth + scope enforcement on /mcp tool calls
- Admin dashboard auth via HTTP-only cookie + bootstrap token
- SSE live activity feed at /admin/events
- DCR default OFF (--enable-dcr to enable)
- Rate limiting on /token (50/15min)
- localOnly operations excluded from HTTP
CLI: gbrain serve --http [--port 3131] [--token-ttl 3600] [--enable-dcr]
Dependencies: express@5.2.1, express-rate-limit@7.5.1, cors@2.8.6
SDK pinned to exact 1.29.0 (was ^1.0.0)
27 new tests covering OAuth provider, scope enforcement, auth code flow,
refresh rotation, token revocation, legacy fallback, and sweep.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: React admin dashboard — 7 screens, dark theme, Krug-designed
Admin SPA at /admin with client-side routing (#login, #dashboard,
#agents, #log). Built with Vite + React, served from admin/dist/.
Screens:
- Login: one field, one button, zero happy talk
- Dashboard: metrics bar, SSE live activity feed, token health panel
- Agents: table with scopes/badges, + Register Agent button
- Register: modal form (name, scopes), 3 mindless choices
- Credentials: full-screen modal, copy buttons, download JSON, warning
- Request Log: paginated table (50/page), time-relative timestamps
- Agent Detail: slide-out drawer, config export tabs (Perplexity/Claude/JSON)
Design tokens: #0a0a0f bg, Inter + JetBrains Mono, 4-32px spacing.
Build: bun run build:admin (Vite, 65KB gzipped).
Admin API: /admin/api/register-client endpoint for dashboard registration.
SPA serving: Express static + index.html fallback for client-side routing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: add admin SPA lockfile
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v1.0.0.0)
Milestone release: multi-agent GBrain with OAuth 2.1, HTTP server,
and React admin dashboard. See CHANGELOG.md for details.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs: update project documentation for v1.0.0.0
Sync README, CLAUDE.md, and docs/mcp/ with the OAuth 2.1 + HTTP server
+ admin dashboard surface that shipped in v1.0.0.0.
- README.md: new "Remote MCP with OAuth 2.1" section covering
gbrain serve --http, admin dashboard, scoped operations, legacy
bearer fallback; add serve --http + auth notes to the commands
reference.
- CLAUDE.md: add src/commands/serve-http.ts, src/core/oauth-provider.ts,
admin/ directory as key files; document scope + localOnly additions
to Operation contract; add oauth.test.ts (27 cases) to the test list;
add v1.0.0 key-commands section clarifying that OAuth client
registration is via the /admin dashboard or SDK (no CLI subcommand).
- docs/mcp/DEPLOY.md: promote --http as the recommended remote path,
add OAuth 2.1 Setup section, list ChatGPT in supported clients,
remove the "not yet implemented" footer.
- docs/mcp/CHATGPT.md (new): unblocks the P0 TODO. Full ChatGPT
connector setup via OAuth 2.1 + PKCE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: wire gbrain auth subcommand with OAuth register-client
Previously auth.ts was a standalone script invoked via
`bun run src/commands/auth.ts`. CHANGELOG and README documented
`gbrain auth ...` commands that didn't actually work.
- Export `runAuth(args)` from auth.ts (keeps standalone entry intact
via `import.meta.url === file://${process.argv[1]}` check)
- Add `auth` to CLI_ONLY + dispatch in handleCliOnly
- New subcommand `gbrain auth register-client <name> [--grant-types]
[--scopes]` wraps GBrainOAuthProvider.registerClientManual
- Lazy DB check: only subcommands that need DATABASE_URL error out
Now the documented CLI flow works end to end:
gbrain auth register-client perplexity --grant-types client_credentials --scopes "read write"
gbrain serve --http --port 3131
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs: reflect wired gbrain auth register-client CLI
After /ship, the doc subagent wrote docs assuming `gbrain auth
register-client` did not exist (it said so explicitly in CLAUDE.md:184).
A follow-up commit (
|
||
|
|
527b87bd1e |
v0.23.0 feat: gbrain dream synthesizes conversations into brain pages (v0.23.0) (#462)
* feat: dream_verdicts schema + engine methods Adds the v25 schema migration creating the dream_verdicts table (file_path, content_hash, worth_processing, reasons, judged_at; PRIMARY KEY (file_path, content_hash); RLS-enabled when running as a BYPASSRLS role). Distinct from raw_data (which is page-scoped) — transcripts being judged for synthesis aren't pages. The (file_path, content_hash) key means edited transcripts re-judge automatically. BrainEngine gains: - DreamVerdict + DreamVerdictInput types - getDreamVerdict(filePath, contentHash) → DreamVerdict | null - putDreamVerdict(filePath, contentHash, verdict) — ON CONFLICT upsert Both engines implement (postgres-engine.ts, pglite-engine.ts). This commit alone is functionally inert — nothing reads/writes the table yet. The synthesize phase (later commit) is the consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: trusted-workspace allow-list for subagent put_page Adds OperationContext.allowedSlugPrefixes — when set, put_page enforces slug membership in the allow-list instead of the legacy wiki/agents/<id>/... namespace. The trust signal is the SUBMITTER (PROTECTED_JOB_NAMES gates subagent submission so MCP can't reach this field), not the runtime ctx.remote flag — every subagent tool call has remote=true for auto-link safety, so basing trust on remote is incoherent. matchesSlugAllowList(slug, prefixes) helper supports glob suffix '/*' (recursive — wiki/originals/* matches ideas/foo/bar) and exact match for unsuffixed entries. put_page check shape: if (viaSubagent && allowedSlugPrefixes set) → allow-list check else if (viaSubagent) → existing namespace check (regression guard) else → no check (regular CLI) Auto-link is re-enabled for the trusted-workspace path so the cycle's extract phase doesn't have to recompute every edge after synthesize writes. Untrusted remote writes still skip auto-link as before. SubagentHandlerData.allowed_slug_prefixes is the wire field; the synthesize/patterns phases (later commit) populate it from a single source of truth in skills/_brain-filing-rules.json's dream_synthesize_paths.globs array. The model's tool schema description mirrors the allow-list so it writes correct slugs on the first try. IRON RULE security tests: - test/operations-allow-list.test.ts: allow-list ALLOW/REJECT, glob semantics, regression guard for the v0.15 namespace fallback when allow-list is unset, FAIL-CLOSED when subagentId is missing. - test/e2e/dream-allow-list-pglite.test.ts: end-to-end on PGLite, poisoned-transcript style write outside allow-list → REJECTED. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: cycle scaffolding — 8-phase order + transcript discovery Extends ALL_PHASES from 6 → 8: synthesize between sync and extract, patterns between extract and embed. Codex finding #7: patterns MUST run after extract because subagent put_page sets ctx.remote=true and skips auto-link/timeline by default — extract is the canonical edge materialization step. Without that ordering, patterns reads stale graph state. Final order: lint → backlinks → sync → synthesize → extract → patterns → embed → orphans CycleOpts gains: - yieldDuringPhase callback — generic in-phase keepalive for long waits (synthesize fan-out, patterns roll-up). Renews cycle-lock TTL + worker job lock. Mirrors yieldBetweenPhases shape. - synthInputFile / synthDate / synthFrom / synthTo — forwarded to runPhaseSynthesize for the CLI's --input/--date/--from/--to flags. CycleReport.totals additively grows (no schema_version bump): transcripts_processed, synth_pages_written, patterns_written. src/core/cycle/transcript-discovery.ts is a pure filesystem walk: - .txt files only, sorted by path for determinism - date-prefixed basename filter (--date / --from / --to) - min_chars filter (default 2000) - exclude_patterns auto-wraps bare words as \b<word>\b regex (Q-3), power users may pass full regex with anchors - compileExcludePatterns is exported for unit tests Phase implementations land in the next commit; this one only adds the dispatcher slots so commit-by-commit bisect doesn't crash on import-not-found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: synthesize + patterns phases — gbrain dream actually dreams Synthesize phase (src/core/cycle/synthesize.ts) reads conversation transcripts from dream.synthesize.session_corpus_dir and writes brain-native pages: reflections to wiki/personal/reflections/..., originals to wiki/originals/ideas/..., timeline entries on existing people pages. Pipeline: 1. discoverTranscripts (filesystem walk + filters) 2. cooldown check via dream.synthesize.last_completion_ts config (default 12h; bypassed by --input/--date/--from/--to) 3. cheap Haiku verdict per transcript, cached in dream_verdicts table keyed by (file_path, content_hash) — backfill re-runs skip already-judged transcripts at zero cost 4. fan-out: one Sonnet subagent per worth-processing transcript dispatched with allowed_slug_prefixes (read from skills/_brain-filing-rules.json's dream_synthesize_paths.globs) and idempotency_key dream:synth:<file_path>:<content_hash> 5. wait via waitForCompletion; yieldDuringPhase ticks every child terminal so the cycle-lock TTL refreshes on long backfills 6. collect slugs from subagent_tool_executions for each child (codex finding #2: NOT pages.updated_at, which would pick up unrelated writes) 7. orchestrator dual-write — query each new page from DB, reverse-render via serializeMarkdown, write file to brain_dir. Subagent never gets fs-write access. 8. deterministic summary index page at dream-cycle-summaries/<date> (codex finding #4: slug shape is regex-compatible — no underscores, no .md extension) 9. write completion timestamp ONLY on successful runs Patterns phase (src/core/cycle/patterns.ts) runs after extract so the graph state is fresh. Single Sonnet subagent gathers reflections within dream.patterns.lookback_days (default 30); names a pattern only when ≥dream.patterns.min_evidence (default 3) reflections support it. Same allow-list path as synthesize. CLI flags on `gbrain dream` (src/commands/dream.ts): --input <file> ad-hoc transcript synthesis (implies --phase synthesize; bypasses cooldown) --date YYYY-MM-DD restrict synthesize to one date --from <d> --to <d> backfill range --dry-run runs Haiku verdict (cached), skips Sonnet synthesis. NOT zero LLM calls (codex #8). Conflict detection: --input + --date/--from/--to exits 2. ISO 8601 date format validated; range start > end exits 2. Auto-commit / push deferred to v1.1 (codex finding #5). v1 writes files to brain_dir; user or autopilot handles git. Tests: - test/cycle-patterns.test.ts: structural assertions on the patterns phase (queue + waitForCompletion wired, allow-list threading, subagent_tool_executions provenance, no raw_data dependency). - test/dream-cli-flags.test.ts: argv parsing, conflict detection, ISO date validation, --input implies --phase synthesize, dry-run semantics doc string. - test/e2e/dream-synthesize-pglite.test.ts: 8 cases on PGLite in-memory exercising not_configured, empty corpus, no API key skip path, dry-run, cooldown active vs --input bypass, and the dream_verdicts cache hit path. Per-test rig isolation (each test creates and tears down its own engine) avoids cross-test PGLite WASM contention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: dream cycle v0.27.0 — skills, CLAUDE.md, migration, changelog - skills/maintain/SKILL.md: synthesize + patterns phases documented with quality bar (Iron Law for synthesis), trust boundary, idempotency, cooldown semantics, CLI invocation patterns. New triggers added so "process today's session" / "synthesize my conversations" route here. - skills/RESOLVER.md: dream cycle triggers route to maintain. - skills/_brain-filing-rules.md: directory table for the five output types (reflections, originals, patterns, people enrichment, cycle summary) with slug shape per row; Iron Law repeated. - skills/migrations/v0.27.0.md: agent-readable migration narrative. Schema migration v25 runs automatically on `gbrain apply-migrations`; synthesize ships disabled by default — opt-in via dream.synthesize.session_corpus_dir + dream.synthesize.enabled. - CLAUDE.md: file inventory updated with new files (cycle/synthesize.ts, cycle/patterns.ts, cycle/transcript-discovery.ts), the 8-phase ordering, the trusted-workspace allow-list trust model, and the v25 schema migration line in the migrate.ts entry. - VERSION: 0.20.4 → 0.27.0 - CHANGELOG.md: v0.27.0 release-summary section per CLAUDE.md voice rules (numbers that matter table, what-this-means closer, "to take advantage of" block), followed by the itemized changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: add patterns E2E + 8-phase cycle E2E + bump synth-cooldown timeouts Two new E2E test files on PGLite (no DATABASE_URL or API key required): - test/e2e/dream-patterns-pglite.test.ts (6 cases) — exercises runPhasePatterns skip paths against a real engine: disabled, default-enabled-but-insufficient-evidence, no-API-key, dry-run. Sibling of dream-synthesize-pglite.test.ts; same per-test rig pattern for engine isolation. - test/e2e/dream-cycle-eight-phase-pglite.test.ts (5 cases) — end-to-end runCycle with the v0.27 8-phase order. Asserts: ALL_PHASES is the documented 8 phases in the right sequence, the dry-run report's phases array preserves that order, CycleReport.totals carries the new transcripts_processed / synth_pages_written / patterns_written fields, --phase synthesize and --phase patterns each run only that phase, and synthInputFile is plumbed correctly through runCycle to runPhaseSynthesize. Bump per-test timeout to 30s on the two synthesize-cooldown E2E tests that create two PGLite engines back-to-back. Default Bun 5s budget is tight under sustained suite pressure (PGLite WASM init costs ~1-2s per engine on macOS); each test passes alone but flakes in the full E2E suite. The third arg `30_000` is Bun's standard test-timeout knob. Full E2E suite (test/e2e/) now: 86 pass / 0 fail / 258 skip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: ship-prep — typecheck fixes, llms.txt regen, 8-phase test update - src/core/cycle/synthesize.ts + patterns.ts: PageType 'default' → 'note' (TS strict typecheck rejected 'default'; 'note' is a valid PageType for orchestrator-written summary index pages and reverse-render fallback). - src/core/pglite-engine.ts: re-import DreamVerdict + DreamVerdictInput types after the master merge dropped them from the import line. - test/e2e/dream-allow-list-pglite.test.ts: ToolCtx now requires remote: true literal; thread it through every put_page tool call. - test/e2e/dream-patterns-pglite.test.ts: PageType 'default' → 'note' in the seedReflections helper. - test/core/cycle.test.ts: bump expected hook-call count + phase count 6 → 8 to match v0.27 ALL_PHASES extension. - llms-full.txt: regenerate against the updated CHANGELOG + CLAUDE.md so the committed snapshot matches what the generator now produces. Full bun test suite: 2793 pass / 0 fail / 258 skip (3051 tests, 177 files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update README + INSTALL_FOR_AGENTS for v0.27.0 dream cycle README: maintain skill row mentions synthesize/patterns; gbrain dream command-reference block describes the 8-phase pipeline and the new --input/--date/--from/--to flags. INSTALL_FOR_AGENTS: dream cycle bullet calls out v0.27 conversation synthesis + cross-session pattern detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: renumber v0.27.0 → v0.23.0 Master is at v0.22.5; v0.23.0 is the next natural slot for the dream-cycle synthesize + patterns release. Bulk rename across VERSION, package.json, CHANGELOG, migration file, source comments, skills, and llms.txt bundles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): bump cycle.test.ts phase count 6 → 8 The dry-run full-cycle test asserted 6 phases. v0.23 added synthesize and patterns, bringing the total to 8. The unit-side equivalent (test/core/cycle.test.ts) was already updated; this catches the E2E sibling that surfaced after the latest master merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
e734937254 |
fix: pass sourceId in cycle sync phase to prevent full reimport (#475)
* fix: pass sourceId in cycle sync phase to prevent full reimport cycle.ts calls performSync without sourceId, so it always reads the global config.sync.last_commit key instead of the per-source sources.last_commit. When the global anchor gets garbage-collected (after a force push or rebase), sync falls back to a full reimport of all files — on a large brain this takes 30+ minutes and blocks the autopilot cycle. The fix resolves the source id from the brain directory by querying the sources table. When a matching source exists, sync reads the per-source anchor which is updated on every successful sync and stays in sync with the repo history. Falls back gracefully to the global config path for pre-v0.18 brains without a sources table. * v0.22.5: tests + version bump for sync-cycle-source-id fix Adds 6 regression tests in test/core/cycle.test.ts pinning the new resolveSourceForDir() helper added to src/core/cycle.ts in this PR: 1. Seeded sources row → performSync receives matching sourceId 2. No matching row → sourceId=undefined (falls through to global key) 3. Different brainDir than registered source → undefined (no cross-match) 4. sources table missing (very old brain) → catch returns undefined, sync still runs. Uses a fresh PGLiteEngine because initSchema() only re-runs PENDING migrations; DROP TABLE on the shared engine would leave it permanently degraded for every later test in the file. (Codex review caught this landmine.) 5. Multiple rows with same local_path → resolver returns one matching id (non-deterministic; SQL has no ORDER BY). Documents the contract for the v0.23 UNIQUE-constraint follow-up. 6. Empty-string id row → resolver propagates "" (defensive case Codex flagged: schema PK prevents NULL but '' can be inserted). Extends the performSync mock at line 51-65 to also capture sourceId. Bumps: - VERSION: 0.22.4 → 0.22.5 - package.json: 0.22.4 → 0.22.5 - CHANGELOG.md: new [0.22.5] entry following v0.22.4 voice (release summary + numbers table + behavior matrix + To-take-advantage block + itemized changes + for-contributors) - CLAUDE.md: annotates src/core/cycle.ts entry with v0.22.5 (#475) note - llms-full.txt: regenerated via bun run build:llms Test results: - Unit: 28 pass / 0 fail in test/core/cycle.test.ts (22 prior + 6 new) - Full unit suite: pass (exit 0) - E2E: 236 pass / 0 fail across 26 files Plan + codex outside-voice review at: ~/.claude/plans/whimsical-bubbling-goose.md Follow-up TODOs filed for v0.23: - Normalize brainDir + sources.local_path before SQL compare - Add UNIQUE index on sources.local_path - Narrow resolveSourceForDir's catch to PG 42P01 (undefined_table) - Add doctor check for config.sync.last_commit / sources divergence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: typecheck error in cycle.test.ts test 5 (sourceId regression) CI typecheck failed because `toContain()` on `string[]` rejects the `string | undefined` produced by `syncCalls.at(-1)?.sourceId`'s optional chain. Tests 1, 4, and 6 use `toBe()` which accepts `string | undefined` through its overload, but `toContain()` is stricter. Fix: pull the value into a typed variable, assert it's defined, then check membership. Makes the contract explicit ("resolver returned a defined sourceId, and it was one of the matching ids") instead of relying on a silent undefined → no-match-in-array assertion. Locally: - bun run typecheck: clean - bun test test/core/cycle.test.ts: 28 pass / 0 fail (75 expect calls) - All CI gate scripts: OK (jsonb, progress-to-stdout, wasm-embedded) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: add --timeout=60000 to E2E runner to prevent setupDB flake PR #475's Tier 1 (Mechanical) CI job hit a 5000.09ms beforeAll hook timeout in `E2E: Tags > (unnamed)`. Cause: scripts/run-e2e.sh invokes `bun test "$f"` without a --timeout flag, falling back to bun's 5s default. setupDB() does TRUNCATE CASCADE on ~30 tables, and on a CI runner under load that can exceed 5s. Match what the unit suite uses (--timeout=60000 in package.json's "test" script). Same 1m ceiling, no behavior change for healthy runs; just removes the artificial 5s floor on hooks. Verified locally: bun test --timeout=60000 test/e2e/mechanical.test.ts runs 78 pass / 0 fail in 27.99s against a fresh pgvector pg16 docker container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
e2961c04bd |
v0.22.1 autopilot fix wave — 5 prod hotfixes (#417, #403, #406, #363, #409) (#447)
* fix: propagate AbortSignal to runCycle + worker force-eviction safety net Root cause: autopilot-cycle handler called runCycle() without passing the job's AbortSignal. When the per-job timeout fired abort(), runCycle never checked it and kept grinding through extract (54,605 pages). The executeJob promise never resolved, inFlight never decremented, and the worker thought it was at capacity forever — 98 jobs piled up waiting with 0 active while a live worker sat idle. Three-layer fix: 1. CycleOpts.signal: new optional AbortSignal field. runCycle checks it between every phase via checkAborted(). A timed-out cycle now bails after the current phase completes instead of running all 6 phases. 2. autopilot-cycle handler: passes job.signal to runCycle so the abort actually propagates. 3. Worker safety net: 30s after the abort fires, if the handler still hasn't resolved, force-evict from inFlight and mark as dead in DB. This is the last-resort escape hatch for any handler that ignores AbortSignal — the worker resumes claiming new jobs instead of wedging forever. Incident: 2026-04-24, 98 waiting / 0 active / worker alive but idle. 143 existing minions tests pass unchanged. * test: abort signal propagation + worker recovery regression tests 16 new tests across 3 files covering the 2026-04-24 worker wedge: test/minions.test.ts (6 new, 149 total): - handler receiving abort signal exits cleanly - handler ignoring abort still gets signal delivered - worker claims new jobs after timeout (no wedge) ← key regression - checkAborted pattern: undefined/non-aborted/aborted signals test/cycle-abort.test.ts (7 new): - CycleOpts.signal type contract - runCycle accepts signal without error - runCycle bails on pre-aborted signal - runCycle bails mid-flight when signal fires between phases - Source-level guard: jobs.ts passes job.signal to runCycle - Source-level guard: worker.ts has force-eviction safety net - Source-level guard: cycle.ts has checkAborted between all 6 phases test/e2e/worker-abort-recovery.test.ts (3 new): - worker recovers from timed-out handler and processes next job - concurrency=2 processes parallel jobs during timeout - multiple sequential timeouts don't permanently wedge worker All 159 tests pass. * perf: incremental extract — only process slugs that sync touched The autopilot-cycle runs every 5 min. Its extract phase was doing a full filesystem walk of ALL markdown files (54K+) — twice (links + timeline). On a brain this size, extract alone exceeded the 600s job timeout, producing zero useful writes. Fix: sync already returns pagesAffected (the slugs it added/modified). Pipe that list through to extract. When provided, extract reads ONLY those files instead of walking the entire brain directory. - Add ExtractOpts.slugs for targeted extraction - Add extractForSlugs() — single-pass links + timeline for specific slugs - cycle.ts: capture sync's pagesAffected, pass to runPhaseExtract - If sync didn't run or failed, extract falls back to full walk (safe) - If pagesAffected is empty (nothing changed), extract returns instantly Expected improvement: 54K file reads → ~10-50 per cycle. The full walk is still available via CLI `gbrain extract` and on first-run. * fix: connection resilience for minion supervisor + worker Three fixes for the minion supervisor dying silently when PgBouncer rotates: 1. PostgresEngine: executeRaw retries once on connection-class errors (ECONNREFUSED, password auth failed, connection terminated, etc.) by tearing down the poisoned pool and creating a fresh one via reconnect(). Prevents cascading failures when Supabase bounces. 2. Supervisor: tracks consecutive health check failures. After 3 in a row, emits health_warn with reason=db_connection_degraded and attempts engine.reconnect() if available. Resets counter on success. 3. Supervisor: worker_exited events now include likely_cause field: SIGKILL → oom_or_external_kill, SIGTERM → graceful_shutdown, code=1 → runtime_error. Makes it trivial to distinguish OOM kills from connection deaths in logs. Tests: 23 new tests covering connection error detection, reconnect guard against concurrent reconnects, retry-once-not-infinite-loop, health failure tracking, and exit classification. * fix(db): set session timeouts on every connection to kill orphan backends Prevents the failure mode from #361: a single autopilot UPDATE on minion_jobs can leave a pooler backend in state='active'/ClientRead for 24h+, holding a RowExclusiveLock that blocks every subsequent ALTER TABLE minion_jobs. The stuck backend never times out on its own because Supabase Micro has no default idle_in_transaction_session_timeout and autovacuum can't reap sessions that hold active locks. Fix: deliver statement_timeout + idle_in_transaction_session_timeout as startup parameters via postgres.js's `connection` option, applied automatically on every new backend connection. Works correctly on both session-mode and transaction-mode PgBouncer poolers (startup params persist for the backend's lifetime, unlike SET commands which transaction-mode PgBouncer strips between transactions). Defaults chosen conservatively so they don't interfere with bulk work like multi-minute embed passes or CREATE INDEX on large pages tables: - statement_timeout: '5min' - idle_in_transaction_session_timeout: '2min' Each overridable per-GUC via env var (GBRAIN_STATEMENT_TIMEOUT, GBRAIN_IDLE_TX_TIMEOUT). Set any to '0' or 'off' to disable. client_connection_check_interval is the specific GUC that would kill the observed state='active'/ClientRead case, but it's Postgres 14+ and some managed poolers reject unknown startup parameters. Made it opt-in only via GBRAIN_CLIENT_CHECK_INTERVAL for users who know their Postgres supports it. Applied in both the module-level singleton connect (src/core/db.ts) and the per-engine-instance pool used by `gbrain jobs work` (src/core/postgres-engine.ts) via a shared resolveSessionTimeouts() helper. Tests: 5 new cases in migrate.test.ts covering defaults, env overrides, '0'/'off' disable, and multi-GUC disable. 39/39 pass (34 pre-existing + 5 new). Closes #361. Co-Authored-By: orendi84 <orendigergo@gmail.com> * fix(embed): server-side staleness filter for embed --stale (v0.20.5) embed --stale walked listPages + per-page getChunks (incl. vector(1536) embedding column) on every call, then client-side-filtered for chunks where embedding was missing. On a 1.5K-page brain at 100% coverage, ~76 MB pulled per call, all discarded. With autopilot firing every 5-10 min plus a 2h cron, this hit Supabase's 5 GB free-tier ceiling at 102 GB used (2058% over) twice in one week. Two new BrainEngine methods replace the page walk with a SQL-side filter: - countStaleChunks(): single SELECT count(*) WHERE embedding IS NULL. Pre-flight short-circuit; ~50 bytes wire when 0 stale. - listStaleChunks(): slug + chunk_index + chunk_text + chunk_source + model + token_count for stale rows only. Excludes the (NULL) embedding column. Bounded by LIMIT 100000 mirroring listPages. embedAll forks: staleOnly=true takes the new SQL-side path (embedAllStale); staleOnly=false (--all) keeps existing behavior verbatim. embedAllStale preserves non-stale chunks on partially-stale pages: it re-fetches existing chunks per stale slug and merges (embedding=undefined for non-stale → COALESCE preserves existing). Without the merge, the upsertChunks != ALL filter would delete non-stale chunks. Re-fetch cost is bounded by stale slug count; the autopilot common case (0 stale) never reaches this path. Predicate uses `embedding IS NULL`, not `embedded_at IS NULL`. The bulk- import path could leave embedded_at populated while embedding was NULL (see upsertChunks consistency fix below), so `embedding IS NULL` is the truth source for "this chunk needs an embedding". Also fixes the upsertChunks consistency bug in both engines: when chunk_text changes and no new embedding is supplied, embedding correctly clears to NULL but embedded_at kept its old timestamp. New behavior resets BOTH columns together, keeping write-time honesty. Wire-cost impact (measured against current behavior on a 1.5K-page brain): - 0 stale chunks (autopilot common case): ~76 MB → ~50 bytes (~1.5M× reduction) - 100 stale across 10 pages: ~76 MB → ~150 KB (~500× reduction) - 8K stale across 1.5K pages (cold start): ~76 MB → ~12 MB (~6× reduction) Tests: 4 new in test/embed.test.ts (zero-stale short-circuit; N-stale- across-M-pages with non-stale preservation; --stale dry-run; --all path byte-identical). Existing --stale tests updated for the new mock surface. Migration impact: none. embedded_at and embedding columns have been on content_chunks since schema inception. Co-Authored-By: atrevino47 <atbuster47@gmail.com> * chore(wave): post-merge tightening — drop executeRaw retry (D3) + gate noExtract (F2) - Drop #406's per-call executeRaw retry wrapper. The regex idempotence boundary is unsound (writable CTEs, side-effecting SELECTs). Recovery now happens at the supervisor level via 3-strikes-then-reconnect. - Update db.ts: setSessionDefaults becomes a back-compat no-op. resolveSessionTimeouts (from #363) is the source of truth, sending GUCs as startup parameters that survive PgBouncer transaction mode. Bumped idle_in_transaction default from 2min to 5min to match v0.21.0 posture. - Gate noExtract in cycle's runPhaseSync on whether extract phase is scheduled. Avoids silently dropping extraction when the user runs `gbrain dream --phase sync` (Codex F2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(db): rephrase docstring to avoid false-positive in test source-grep The migrate.test.ts structural check counts `SET idle_in_transaction_session_timeout` matches in source. The literal string in this docstring was tripping it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: backfill regression guards for #417, D3, F2 (Step 5) 15 new test cases across 3 files, ~250 LOC, all PGLite/in-memory: test/extract-incremental.test.ts (NEW, 8 cases for #417): - slugs: [] returns immediately (early-return) - slugs: undefined falls through to full-walk - slugs: [a, b] reads only those files - Slug whose file no longer exists is silently skipped - Mode filter (links) skips timeline extraction - dryRun: true does not invoke addLinksBatch / addTimelineEntriesBatch - BATCH_SIZE flush — >100 candidate links exercise mid-iteration flush - Full-slug-set resolution — link to file outside changed set still resolves test/core/cycle.test.ts (4 new cases for #417 + Codex F2): - cycle threads sync.pagesAffected into extract phase as the slugs argument - extract phase falls back to full walk when sync was skipped - F2 guard: full cycle (sync + extract) sets noExtract=true on sync - F2 guard: phases:[sync] only sets noExtract=false (no silent extract drop) test/connection-resilience.test.ts (3 new cases for D3): - PostgresEngine.executeRaw is a single-statement passthrough (no try/catch) - PostgresEngine.reconnect() still exists for supervisor-driven recovery - Supervisor still has the 3-strikes-then-reconnect path Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(wave): v0.21.1 release notes + 3 follow-up TODOs + CLAUDE.md updates CHANGELOG.md: segment-aware entry per CEO-review D1 — 'For everyone' section (#417 incremental extract, #403 cycle abort) leads, 'For Postgres / Supabase users' section (#406, #363, #409) follows. Production proof point as a sidebar, not the lead. TODOS.md: 3 follow-up items per Eng-review D6: 1. Caller-opt-in retry for executeRaw (D3 follow-up) 2. Replace walkMarkdownFiles with engine.getAllSlugs() (F1 follow-up) 3. err.code-based connection-error matching (B1 follow-up) CLAUDE.md: 6 file-reference updates for the wave's behavioral additions (postgres-engine, db, cycle, worker, supervisor, embed, extract). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(release): bump version 0.21.1 → 0.22.1 + document version locations User-explicit version override on /ship: ship as v0.22.1 (MINOR jump from master's 0.21.0) instead of the v0.21.1 PATCH the wave originally targeted. The wave bundles 5 production fixes which is meaningful enough to clear a MINOR version, even though the API surface is additive. Files updated to 0.22.1: - VERSION (single source of truth) - package.json (Bun/npm version) - CHANGELOG.md (release header + "To take advantage of v0.22.1" block) - TODOS.md (3 follow-up TODOs reference the version that filed them) - CLAUDE.md (Key Files annotations cite the release that introduced behavior) Also adds a "Version locations" section to CLAUDE.md documenting all five required files plus the auto-derived (bun.lock, llms-full.txt) and historical (skills/migrations/v*.md, src/commands/migrations/v*.ts, test/migrations-v*.test.ts) categories. Future /ship runs and the auto-update agent now have a canonical list of where versions live. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): unbreak CI typecheck — annotate signal as AbortSignal | undefined CI's `bun run typecheck` step was failing with TS2339 at test/minions.test.ts:2026 — `const signal = undefined` narrows to literal `undefined`, which has no `.aborted` property, so `signal?.aborted` doesn't compile. Fix uses `as AbortSignal | undefined` to preserve the union type. A plain type annotation gets narrowed back via control-flow analysis; the `as` cast doesn't. Runtime behavior is unchanged — the optional-chain still short-circuits as intended. Verified: bunx tsc --noEmit → exit 0; the 3 checkAborted cases still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(doctor): forward-progress override for stale minions partials The minions_migration check reads ~/.gbrain/migrations/completed.jsonl and flags any version that has a `partial` entry without a matching `complete`. Long-lived installs accumulate partial records from historical stopgap runs (notably v0.11.0). Without time decay or forward-progress detection, the FAIL flag fires forever once any partial lands, even on installs that have been running clean at v0.22+ for months. Concrete failure: test/e2e/mechanical.test.ts "gbrain doctor exits 0 on healthy DB" was flaking on dev machines whose ~/.gbrain/ carried v0.11.0 partials from earlier in the day. The fresh test DB had nothing wrong with it; doctor was just reading host filesystem state that bled in via $HOME. Fix: a partial vX.Y.Z is treated as stale (not stuck) if any vA.B.C where A.B.C >= X.Y.Z has a `complete` entry anywhere in the file. The reasoning: if a newer migration successfully landed, the install has clearly moved past the older partial. compareVersions() from src/commands/migrations/index.ts handles the semver compare. Cases preserved: - v0.10 complete + v0.11 partial → still FAILs (older complete doesn't supersede newer partial) - v0.16 partial alone → still FAILs (no override exists) - Fresh install (no completed.jsonl) → no warning - Real partial-then-complete-same-version → no warning Cases now fixed: - v0.16 complete + v0.11 partial → no FAIL (forward progress made; the v0.11 record is stale) Two regression tests in test/doctor-minions-check.test.ts cover both directions of the override (when it fires, when it doesn't). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(docs): regenerate llms-full.txt after CLAUDE.md updates CI's build-llms regen-drift guard caught that llms-full.txt was stale relative to CLAUDE.md after the wave's documentation commits (the "Version locations" section + 6 file-reference annotations for the wave's behavioral additions). CLAUDE.md notes that llms-full.txt is auto-derived — bumped via 'bun run build:llms' when CLAUDE.md's file-references change. This commit catches up. llms.txt is unchanged; the curated index doesn't pull from CLAUDE.md's file-reference body. Only llms-full.txt (the inlined single-fetch bundle) needed regeneration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: orendi84 <orendigergo@gmail.com> Co-authored-by: atrevino47 <atbuster47@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
8b3c24c891 |
v0.20.0 feat: extract BrainBench to sibling gbrain-evals repo (#195)
* fix(link-extraction): v0.10.5 drive works_at + advises accuracy on rich prose
Extends inferLinkType patterns to cover rich-prose phrasings that miss with
v0.10.4 regexes. Targets the residuals called out in TODOS.md: works_at at
58% type accuracy, advises at 41%.
WORKS_AT_RE additions:
- Rank-prefixed: "senior engineer at", "staff engineer at", "principal/lead"
- Discipline-prefixed: "backend/frontend/full-stack/ML/data/security engineer at"
- Possessive time: "his/her/their/my time at"
- Leadership beyond "leads engineering": "heads up X at", "manages engineering at",
"runs product at", "leads the [team] at"
- Role nouns: "role at", "position at", "tenure as", "stint as"
- Promotion patterns: "promoted to staff/senior/principal at"
ADVISES_RE additions:
- Advisory capacity: "in an advisory capacity", "advisory engagement/partnership/contract"
- "as an advisor": "joined as an advisor", "serves as technical advisor"
- Prefixed advisor nouns: "strategic/technical/security/product/industry advisor to|at"
- Consulting: "consults for", "consulting role at|with"
New EMPLOYEE_ROLE_RE page-level prior: fires when the page describes the subject
as an employee (senior/staff/principal engineer, director, VP, CTO/CEO/CFO) at
some company. Biases outbound company refs toward works_at when per-edge context
is possessive or narrative without an explicit work verb. Scoped to person -> company
links only. Precedence: investor > advisor > employee (investors often hold board
seats which would otherwise mis-classify as advise/works_at).
ADVISOR_ROLE_RE broadened from "full-time/professional/advises multiple" to catch
any page that self-identifies the subject as an advisor ("is an advisor",
"serves as advisor", possessive "her advisory work/role/engagement").
Tests: 65 pass (16 new v0.10.5 coverage tests + 4 regression guards against
v0.10.4 tightenings). Templated benchmark still 88.9% type_accuracy (10/10 on
works_at and advises). Rich-prose measurement requires the multi-axis report
upgrade (next commit) to validate retroactively.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(eval): type-accuracy runner on rich-prose corpus + wire into all.ts
New Category 2 in BrainBench: per-link-type accuracy measured directly on the
240-page rich-prose world-v1 corpus. Distinct from Cat 1's retrieval metrics,
this measures whether inferLinkType() correctly classifies extracted edges
when the prose varies (the 58% works_at and 41% advises residuals that v0.10.5
regexes targeted).
How it works:
1. Loads all pages from eval/data/world-v1/
2. Derives GOLD expected edges from each page's _facts metadata
(founders → founded, investors → invested_in, advisors → advises,
employees → works_at, attendees → attended, primary_affiliation +
role drives person-page outbound type)
3. Runs extractPageLinks() on each page → INFERRED edges
4. Per (from, to) pair, compares inferred type vs gold type
5. Emits per-link-type table: correct / mistyped / missed / spurious +
type accuracy + recall + precision + strict F1 (triple match)
6. Full confusion matrix rows=gold, cols=inferred
v0.10.5 validation on 240-page corpus (up from pre-v0.10.5 baselines):
- works_at: 58% → 100.0% (+42 pts) — 10/10 correct, 0 mistyped
- advises: 41% → 88.2% (+47 pts) — 15/17 correct
- attended: — → 100.0% 131/134 recall
- founded: 100% → 100.0% 40/40
- invested_in: 89% → 92.0% 69/75
- Overall: 88.5% → 95.7% type accuracy (conditional on edge found)
Strict F1 overall: 53.7%. Lower because the _facts-based gold set only
captures core relationships; rich prose extracts many peripheral mentions
(190 spurious "mentions" edges) that aren't bugs but are correctly-typed
prose references without a _facts counterpart. Spurious counts are signal
for future type-precision tuning, not failure.
Wired into eval/runner/all.ts as Cat 2 so every full benchmark run includes
the rich-prose type accuracy table alongside retrieval metrics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(eval): Phase 2 adapter interface + EXT-1 ripgrep+BM25 baseline
Phase 2 credibility unlock: BrainBench now compares gbrain to external
baselines on the same corpus and queries. Transforms the benchmark from
internal ablation ("gbrain-graph beats gbrain-grep") to category comparison
("gbrain-graph beats classic BM25 by 32 pts P@5"). This is the #1 fix
from the 4-review arc — addresses Codex's core critique that v1's
before/after was self-referential.
Added:
eval/runner/types.ts — Adapter interface (v1.1 spec)
eval/runner/adapters/ripgrep-bm25.ts — EXT-1 classic IR baseline
eval/runner/adapters/ripgrep-bm25.test.ts — 11 unit tests, all pass
eval/runner/multi-adapter.ts — side-by-side scorer
Adapter interface (eng pass 2 spec):
- Thin 3-method Strategy: init(rawPages, config), query(q, state), snapshot(state)
- BrainState is opaque to runner (never inspected)
- Raw pages passed in-memory; gold/ never crosses adapter boundary
(structural ingestion-boundary enforcement)
- PoisonDisposition enum reserved for future poison-resistance scoring
EXT-1 ripgrep+BM25:
- Classic Lucene-variant IDF + k1/b tuned at standard 1.5/0.75
- Title tokens double-weighted for entity-page slug-match bias
- Stopword filter, alphanumeric tokenization, stable lexicographic tie-break
- Pure in-memory inverted index — no external deps, ~100 LOC core
First side-by-side results on 240-page rich-prose corpus, 145 relational queries:
| Adapter | P@5 | R@5 | Correct top-5 |
|---------------|--------|--------|---------------|
| gbrain-after | 49.1% | 97.9% | 248/261 |
| ripgrep-bm25 | 17.1% | 62.4% | 124/261 |
| Delta | +32.0 | +35.5 | +124 |
gbrain-after is the hybrid graph+grep config from PR #188. Ripgrep+BM25 is
a genuinely strong classic-IR baseline (BM25 is what Lucene/Elasticsearch
ship). gbrain's ~+32-point lead on relational queries reflects real work
by the knowledge graph layer: typed links + traversePaths surface the
correct answers in top-K that BM25 only pulls in via partial-text overlap.
Next in Phase 2: EXT-2 vector-only RAG + EXT-3 hybrid-without-graph
adapters. Both plug into the same Adapter interface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(eval): Phase 2 EXT-2 vector-only RAG adapter
Second external baseline for BrainBench. Pure cosine-similarity ranking
using the SAME text-embedding-3-large model gbrain uses internally —
apples-to-apples on the embedding layer so any gbrain lead reflects the
graph + hybrid fusion, not a better embedder.
Files:
eval/runner/adapters/vector-only.ts ~130 LOC
eval/runner/adapters/vector-only.test.ts 6 unit tests (cosine math)
Design:
- One vector per page (title + compiled_truth + timeline, capped 8K chars).
- No chunking (intentional; chunked vector RAG would be EXT-2b later).
- No keyword fallback (that's EXT-3 hybrid-without-graph).
- Embeddings in batches of 50 via existing src/core/embedding.ts (retry+backoff).
- Cost on 240 pages: ~$0.02/run.
Three-adapter side-by-side on 240-page rich-prose corpus, 145 relational queries:
| Adapter | P@5 | R@5 | Correct top-5 |
|---------------|--------|--------|---------------|
| gbrain-after | 49.1% | 97.9% | 248/261 |
| ripgrep-bm25 | 17.1% | 62.4% | 124/261 |
| vector-only | 10.8% | 40.7% | 78/261 |
Interesting finding: vector-only scores WORSE than BM25 on relational queries
like "Who invested in X?" — exact entity match matters more than semantic
similarity for these templates. BM25 nails the entity-name term; vector-only
returns topically-similar-but-not-mentioning pages. This is the known failure
mode of pure-vector RAG on precise relational/identity queries. Real-world
vector RAG systems always add keyword fallback; EXT-3 (hybrid-without-graph)
will be that fairer comparator.
gbrain's lead widens in vector-only comparison: +38.4 pts P@5, +57.2 pts R@5.
The graph layer is doing the heavy lifting for relational traversal; pure
vector RAG can't express "traverse 'attended' edges from this meeting page."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(eval): Phase 2 EXT-3 hybrid-without-graph adapter — graph isolated
Third and closest-to-gbrain external baseline. Runs gbrain's full hybrid
search (vector + keyword + RRF fusion + dedup) WITHOUT the knowledge-graph
layer. Same engine, same embedder, same chunking, same hybrid fusion —
only traversePaths + typed-link extraction turned off.
This is the decisive comparator for "does the knowledge graph do useful
work?" Same everything-else, only graph differs. Any lead gbrain-after has
over EXT-3 is 100% attributable to the graph layer.
Files:
eval/runner/adapters/hybrid-nograph.ts — ~110 LOC
Implementation:
- New PGLiteEngine per run; auto_link set to 'false' (belt).
- importFromContent() used instead of bare putPage() so chunks +
embeddings get populated (hybridSearch needs them).
- NO runExtract() call — typed links/timeline stay empty (suspenders).
- hybridSearch(engine, q.text) answers every query. Aggregate chunks
to page-level by best chunk score.
FOUR-adapter side-by-side on 240-page rich-prose corpus, 145 relational queries:
| Adapter | P@5 | R@5 | Correct/Gold |
|-----------------|--------|--------|--------------|
| gbrain-after | 49.1% | 97.9% | 248/261 |
| hybrid-nograph | 17.8% | 65.1% | 129/261 |
| ripgrep-bm25 | 17.1% | 62.4% | 124/261 |
| vector-only | 10.8% | 40.7% | 78/261 |
The headline delta nobody can hand-wave away:
gbrain-after → hybrid-nograph = +31.4 P@5, +32.9 R@5
hybrid-nograph → ripgrep-bm25 = +0.7 P@5, +2.7 R@5
Hybrid search (vector+keyword+RRF) over pure BM25 gains ~1 point. The
knowledge graph layer over hybrid gains ~31 points. The graph is doing
the work; adding it to a retrieval stack is what actually moves the needle
on relational queries. The vector/keyword/BM25 debate is a footnote.
Timing: hybrid-nograph init is ~2 min (embeds 240 pages once); query loop
is fast. gbrain-after is ~1.5s total because traversePaths doesn't need
embeddings. Runs at ~$0.02 Opus-equivalent in embedding cost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(eval): Phase 2 query validator + Tier 5 Fuzzy + Tier 5.5 synthetic + N=5 tolerance bands
Closes multiple Phase 2 items in one commit since they form a cohesive
package: query schema enforcement + new query tiers + per-query-set
statistical rigor.
Added:
eval/runner/queries/validator.ts — hand-rolled Query schema validator
eval/runner/queries/validator.test.ts — 24 unit tests, all pass
eval/runner/queries/tier5-fuzzy.ts — 30 hand-authored Tier 5 Fuzzy/Vibe queries
eval/runner/queries/tier5_5-synthetic.ts — 50 SYNTHETIC-labeled outsider-style queries (author: "synthetic-outsider-v1")
eval/runner/queries/index.ts — aggregator + validateAll()
Modified:
eval/runner/multi-adapter.ts — N=5 runs per adapter (BRAINBENCH_N override), page-order shuffle, mean±stddev reporting
Query validator (hand-rolled, no zod dep to match gbrain codebase style):
- Temporal verb regex enforces as_of_date (per eng pass 2 spec):
/\\b(is|was|were|current|now|at the time|during|as of|when did)\\b/i
- Validates tier enum, expected_output_type enum, gold shape per type
- gold.relevant must be non-empty slug[] for cited-source-pages queries
- abstention requires gold.expected_abstention === true
- externally-authored tier requires author field
- batch validation catches duplicate IDs
Tier 5 Fuzzy/Vibe (30 queries, hand-authored):
- Vague recall: "Someone who was a senior engineer at a biotech company..."
- Trait-based: "The engineer who pushed back on microservices"
- Cultural/epithet: "Who is known as a 'systems builder' in security?"
- Abstention bait: "Which Layer 1 project did the crypto guy leave?" (prose
mentions but never names; good systems abstain)
- Addresses Codex's circularity critique — vague queries where graph-heavy
systems shouldn't inherently win.
Tier 5.5 Synthetic Outsider (50 queries, AI-authored placeholder):
- Clearly labeled author: "synthetic-outsider-v1"
- Phrasing variety not in the 4 template families:
* fragment style ("crypto founder Goldman Sachs background")
* polite/natural ("Can you pull up what we have on...")
* comparison ("What is the difference between X and Y?")
* follow-up ("And who else advises Orbit Labs?")
* typos/misspellings ("adam lopez bioinformatcis")
* similarity ("Find me someone like Alice Davis...")
* imperative ("Pull up Alice Davis")
- Real Tier 5.5 from outside researchers supersedes synthetic via
PRs to eval/external-authors/ (docs ship in follow-up commit).
N=5 tolerance bands:
- Default N=5, override via BRAINBENCH_N env var (e.g. BRAINBENCH_N=1 for dev loops)
- Per-run seeded Fisher-Yates shuffle of page ingest order (LCG seed = run_idx+1)
- Surfaces order-dependent adapter bugs (tie-break-by-first-seen etc.)
- Reports mean ± sample-stddev per metric
- "stddev = 0" is honest signal that the adapter is deterministic, not a bug.
LLM-judge metrics (future) will naturally produce non-zero stddev.
Validation: all 80 Tier 5 + 5.5 queries pass validateAll(). 24 validator
unit tests pass.
Next commit: world.html contributor explorer (Phase 3).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(eval): Phase 3 world.html explorer + eval:* CLI surface
Contributor DX magical moment. Static HTML explorer renders the full
canonical world (240 entities) as an explorable tree, opens in any browser,
zero install. Every string HTML-entity-encoded (XSS-safe — direct vuln
class per eng pass 2, confidence 9/10).
Added:
eval/generators/world-html.ts — renderer (~240 LOC; single-file
HTML with inline CSS + minimal JS)
eval/generators/world-html.test.ts — 16 tests (XSS + rendering correctness)
eval/cli/world-view.ts — render + open in default browser
eval/cli/query-validate.ts — CLI wrapper for queries/validator
eval/cli/query-new.ts — scaffold a query template
Modified:
package.json — 7 new eval:* scripts
.gitignore — ignore generated world.html
package.json scripts shipped:
bun run test:eval all eval unit tests (57 pass)
bun run eval:run full 4-adapter N=5 side-by-side
bun run eval:run:dev N=1 fast dev iteration
bun run eval:world:view render world.html + open in browser
bun run eval:world:render render only (CI-friendly, --no-open)
bun run eval:query:validate validate built-in T5+T5.5 (or a file path)
bun run eval:query:new scaffold a new Query JSON template
bun run eval:type-accuracy per-link-type accuracy report
XSS safety:
escapeHtml() encodes the 5 critical chars (& < > " '). Tested directly
with representative Opus-generated attacks:
<img src=x onerror=alert('xss')> → <img src=x onerror=alert('xss')>
<script>fetch('/steal')</script> → <script>fetch('/steal')</script>
Ledger metadata (generated_at, model) also escaped — covers the less
obvious attack surface where Opus could emit tag-like content into the
metadata file.
world.html structure:
- Left rail: entities grouped by type with counts (companies, people,
meetings, concepts), alphabetical within type
- Right pane: per-entity cards with title + slug + compiled_truth +
timeline + canonical _facts as collapsed JSON
- URL fragment deep-links (#people/alice-chen)
- Sticky rail on desktop; responsive stack on mobile
- Vanilla JS for active-link highlighting on scroll (no framework)
Generated file: ~1MB for 240 entities (full prose). Gitignored; rebuild
with `bun run eval:world:view`. Regeneration is ~50ms.
Contributor TTHW (Tier 5.5 query authoring):
1. bun run eval:world:view # see entities
2. bun run eval:query:new --tier externally-authored --author "@me"
3. edit template with real slug + query text
4. bun run eval:query:validate path/to/file.json
5. submit PR
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(eval): Phase 3 contributor docs + CI workflow for eval/ tests
Ships the contributor-onboarding surface promised in the plan. With this
commit, external researchers have a self-serve path from clone to PR in
under 5 minutes.
Added:
eval/README.md — 5-minute quickstart,
directory map, methodology
one-pager, adapter scorecard
eval/CONTRIBUTING.md — three contributor paths:
1. Write Tier 5.5 queries
2. Submit an external adapter
3. Reproduce a scorecard
eval/RUNBOOK.md — operational troubleshooting:
generation failures, runner
failures, query validation,
world.html rendering, CI
eval/CREDITS.md — contributor attribution
(synthetic-outsider-v1 labeled
as placeholder; real submissions
land here)
.github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template
for Tier 5.5 submissions
.github/workflows/eval-tests.yml — CI: validates queries,
runs all eval unit tests,
renders world.html on every PR
touching eval/** or
src/core/link-extraction.ts
CI scope (intentionally narrow):
- Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/**
- Runs: bun run eval:query:validate (80 queries), test:eval (57 tests),
eval:world:render (smoke-test the HTML renderer)
- Pinned actions by commit SHA (matches existing .github/workflows/test.yml)
- Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests
- Fast: ~30s total wall clock
Contributor TTHW (clone → first merged PR):
- Path 1 (Tier 5.5 queries): ~5 min
- Path 2 (external adapter): ~30 min for a simple adapter
- Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(eval): teardown PGLite engines so bun run eval:run exits 0
The multi-adapter runner left PGLite engines alive after each run.
GbrainAfterAdapter and HybridNoGraphAdapter both instantiate a
PGLiteEngine in init() but never disconnect it; Bun's shutdown path
exits with code 99 when embedded-Postgres workers outlive main().
Added optional `teardown?(state)` to the Adapter interface, implemented
it on both engine-backed adapters, and call it from scoreOneRun after
the N=5 loop. ripgrep-bm25 and vector-only hold no DB resources and
don't need a teardown.
Verified: gbrain-after, hybrid-nograph, ripgrep-bm25, vector-only all
exit 0 at N=1. Full test:eval passes (57 tests). No metric change.
* docs(bench): 2026-04-19 multi-adapter scorecard
Reproducibility run of the 4-adapter side-by-side at commit
|
||
|
|
55ca4984b2 |
feat: v0.17.0 — gbrain dream + runCycle primitive (one cycle, two CLIs) (#321)
* fix(sync): honor --dry-run in full-sync path + expose embedded count
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
The full-sync path (performFullSync) previously called runImport() even
when opts.dryRun was true, silently writing to the DB and advancing
sync.last_commit. `gbrain sync --dry-run` on a fresh brain (or with
--full) would mutate state without warning.
Fix:
- performFullSync now early-returns a `dry_run` SyncResult when
opts.dryRun is set. Walks the repo via collectMarkdownFiles +
isSyncable to count what WOULD be imported. No writes, no git
state advance.
- SyncResult gains an `embedded: number` field (required). Tracks
pages re-embedded during the sync's auto-embed step. Existing
return sites set 0; the synced + first_sync paths set real counts
(best-estimate until commit 2 sharpens runEmbedCore's return type).
- first_sync path now returns real added + chunksCreated counts
from runImport instead of hardcoded zeros.
- printSyncResult shows embedded count in human output.
Tests (test/sync.test.ts, new `performSync dry-run never writes`
block, PGLite + temp git repo, no DATABASE_URL required):
- first-sync --dry-run: no pages, no sync.last_commit
- incremental --dry-run after real sync: bookmark unchanged
- --full --dry-run: no reimport, bookmark unchanged
- SyncResult.embedded is a number
Codex outside-voice caught this. Would have shipped silent DB writes
on dry-run for anyone using `gbrain sync --dry-run --full`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(embed): add dry-run mode + return EmbedResult with counts
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
runEmbedCore previously returned Promise<void> and had no dry-run mode.
That made it impossible for runCycle to (a) report accurate embedded
counts or (b) honor --dry-run without also skipping the entire embed
phase (which would have required runCycle to know embed's internal
semantics — a layering violation).
Changes:
- EmbedOpts gains `dryRun?: boolean`. When set, embedPage and
embedAll enumerate stale chunks (or would-be-created chunks for
unchunked pages, via local chunkText without engine.upsertChunks)
but never call embedBatch and never write to the engine.
- runEmbedCore: Promise<void> -> Promise<EmbedResult>. Result shape:
{ embedded, skipped, would_embed, total_chunks, pages_processed,
dryRun }.
embedded = chunks newly embedded (0 in dryRun).
would_embed = chunks that WOULD be embedded (0 in non-dryRun).
skipped = chunks with pre-existing embeddings.
- runEmbed CLI wrapper honors --dry-run flag and returns the result
through. `gbrain embed --stale --dry-run` is now a safe preview.
- Callers ignoring the return value (sync auto-embed, autopilot
inline fallback, jobs.ts handlers, CLI) keep compiling — the new
return type is additive for `await` callers.
Tests (test/embed.test.ts, new `runEmbedCore --dry-run` block, uses
the existing mock.module embedBatch pattern, no API key required):
- dry-run --all: zero embedBatch calls, zero upsertChunks calls,
would_embed matches stale chunk total
- dry-run --stale correctly splits stale vs already-embedded counts
- dry-run --slugs on a single page tallies per-chunk counts
- non-dry-run regression guard: embedded count matches across
concurrent workers
Codex outside-voice flagged the Promise<void> return as a blocker for
accurate CycleReport.totals.pages_embedded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(orphans): engine-injected queries, drop db.getConnection() global
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
findOrphans + queryOrphanPages previously reached into the postgres-js
singleton via db.getConnection(), which (a) didn't compose with
runCycle's explicit-engine contract and (b) was wrong for PGLite test
fixtures and for any caller not using the default global connection.
Codex outside-voice flagged this as a blocker.
Changes:
- BrainEngine interface gains findOrphanPages() — returns pages with
no inbound links via the same NOT EXISTS anti-join. Implemented on
both postgres-engine (sql tag) and pglite-engine (db.query).
- findOrphans signature: findOrphans(engine, { includePseudo }).
Engine is required. Uses engine.findOrphanPages() and
engine.getStats().page_count instead of raw SQL + global counts.
- queryOrphanPages signature: queryOrphanPages(engine). Delegates to
engine.findOrphanPages().
- src/commands/orphans.ts drops the `import * as db` — no more
global-state coupling.
- Callers updated: src/core/operations.ts find_orphans handler now
passes ctx.engine through; runOrphans CLI entry uses its engine arg.
- No signature change needed in cli.ts (it was already passing engine
via CLI_ONLY dispatch).
Tests (test/orphans.test.ts, new `findOrphans (engine-injected)`
describe block, PGLite in-memory, no DATABASE_URL required):
- links correctly scope orphans (alice links to bob -> bob not
an orphan; alice is)
- includePseudo:true surfaces _atlas-style pages
- queryOrphanPages delegates to passed engine
- empty brain returns {orphans: [], total_pages: 0} without crashing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): add runCycle primitive in src/core/cycle.ts
The brain maintenance cycle as a single function. Six phases in
semantically-driven order (fix files → sync → extract → embed →
report orphans). Pure composition of existing library calls — no
execSync, no subprocess anti-patterns, no regex-parsed output.
┌───────────────────────────────────────────────────┐
│ runCycle(engine, opts) → CycleReport │
│ Phase 1: lint --fix (fs writes) │
│ Phase 2: backlinks --fix (fs writes) │
│ Phase 3: sync (DB picks up 1+2) │
│ Phase 4: extract (DB picks up links) │
│ Phase 5: embed --stale (DB writes) │
│ Phase 6: orphans (DB read, report) │
└───────────────────────────────────────────────────┘
Why the commit-4 primitive:
- CEO + Eng + Codex reviews all converged on "extract one cycle
function, wire both dream and autopilot through it." Two CLIs,
one definition of what the brain does overnight.
- Phase order was wrong in PR #309's original dream.ts (sync
before lint+backlinks lost the "fix files, then index them"
semantic).
- This commit is the bisectable foundation; commit 5 (dream)
and commit 6 (autopilot+jobs) just call into it.
Coordination — the codex-flagged blocker:
Session-scoped pg_try_advisory_lock does not survive PgBouncer
transaction pooling (the v0.15.4 fix made pooled connections the
default). Replaced with a DB lock table (gbrain_cycle_locks) that
works through every pooler:
- Acquire: INSERT ... ON CONFLICT DO UPDATE ... WHERE ttl < NOW()
- Refresh: UPDATE ttl_expires_at between phases via hook
- Release: DELETE in finally{}
- TTL: 30 min; crashed holders auto-release
PGLite / engine=null path uses a file lock at ~/.gbrain/cycle.lock
with PID liveness check. kill(pid, 0) with EPERM treated as alive
(so init/launchd-pid holders aren't mis-classified as stale).
Lock-skip: only phases that mutate state (lint, backlinks, sync,
extract, embed) trigger lock acquisition. orphans is read-only.
Single-phase --phase orphans runs never block on a held lock.
Engine-null mode preserved: filesystem phases run, DB phases skip
with {status:'skipped', reason:'no_database'}. Matches current
dream's capability that would have been lost if runCycle required
a connected engine.
Contract details:
- CycleReport has schema_version:"1" (stable, additive) so agents
consuming --json can rely on the shape
- status: 'ok' | 'clean' | 'partial' | 'skipped' | 'failed'.
'clean' = ran successfully with zero activity; agents trivially
detect a healthy brain.
- PhaseResult.error: { class, code, message, hint?, docs_url? }
(Stripe-API-tier structured failure info) when status='fail'
- yieldBetweenPhases hook: awaited between EVERY phase and before
return, runs even after phase failure, exceptions logged but
non-fatal. Required so the Minions autopilot-cycle handler can
renew its job lock between phases (prevents the v0.14 stall-death
regression codex flagged).
- git pull explicit: opts.pull defaults to false (cron-safe).
Autopilot daemon callers opt in if user configured it.
- extract phase doesn't have a dry-run mode in the underlying
library function, so runCycle honestly skips extract when
dryRun=true (status:'skipped', reason:'no_dry_run_support').
Schema migration v16: gbrain_cycle_locks table + idx_cycle_locks_ttl.
Also appended to src/schema.sql and src/core/pglite-schema.ts for
fresh installs. schema-embedded.ts regenerated via build:schema.
Tests (test/core/cycle.test.ts, PGLite in-memory + mocked library
functions, no DATABASE_URL required):
- dryRun × phases matrix: dryRun:true reaches lint/backlinks/sync/
embed; extract is honestly skipped
- Phase selection: default runs all 6 in order; --phase lint runs
only lint; --phase orphans runs only orphans
- Lock semantics: acquire + release on mutating phases, skip
entirely for read-only selections
- cycle_already_running: seeded live-holder lock → status:skipped,
zero phase runs; TTL-expired holder → auto-claimed
- Engine null: filesystem phases run, DB phases skip
- File lock (engine=null) blocks when PID 1 holds lock with fresh
mtime — exercises the PID liveness branch including EPERM
- Status derivation: 'ok' vs 'clean' vs 'partial' vs 'skipped'
- yieldBetweenPhases called N times, hook exceptions non-fatal
Next: commit 5 rewrites dream.ts as a thin CLI alias over runCycle,
commit 6 migrates autopilot daemon + jobs.ts handler to delegate to
runCycle too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dream): add gbrain dream CLI as a thin alias over runCycle
`gbrain dream` is the README brand-promise command: "the agent runs
while I sleep, the dream cycle ... I wake up and the brain is smarter."
Cron-friendly, JSON-reportable, phase-selectable. Same maintenance
cycle as `gbrain autopilot`, just scheduled differently — both
converge on runCycle (added in commit 4) so there's one source of
truth for what happens overnight.
Contract:
gbrain dream # full 6-phase cycle
gbrain dream --dry-run # preview, no writes
gbrain dream --json # CycleReport JSON (agent-readable)
gbrain dream --phase <name> # single-phase run
gbrain dream --pull # git pull before syncing
gbrain dream --dir /path/to/brain # explicit brain location
Cron: 0 2 * * * gbrain dream --json >> /var/log/gbrain-dream.log
Behavior details:
- Brain-dir resolution: requires explicit --dir OR sync.repo_path
in engine config. No more walk-up-cwd-for-.git footgun that
PR #309's original dream.ts had (would lint unrelated git repos).
- engine=null mode preserved via cli.ts's try/catch around
connectEngine — filesystem phases (lint, backlinks) still run
without a DB, DB phases report skipped/no_database in the output.
- status=clean prints "Brain is healthy. N phase(s) checked in Ns."
status=skipped prints the reason (cycle_already_running, etc.).
Partial/failed prints the phase-by-phase detail.
- Exit code 1 when status=failed (cron spots real problems).
'partial' is not a failure — warnings shouldn't page you.
- --help text cross-references `autopilot --install` for users
who want continuous maintenance as a daemon.
CLI registration (src/cli.ts):
- 'dream' added to CLI_ONLY
- handleCliOnly has a pre-engine branch mirroring doctor's pattern:
try connectEngine() → ok path; catch → runDream(null, args) so
filesystem phases still run when DB is down
- Help text updated with one-line dream entry and autopilot cross-ref
Tests (test/dream.test.ts, real PGLite + real library calls, no mocks
to avoid `mock.module` leakage across test files):
- brainDir resolution: explicit --dir wins, engine config fallback,
missing + nonexistent errors
- phase selection: --phase lint|orphans produces single-phase report
- phase validation: --phase garbage exits 1
- output: --json parses as CycleReport with schema_version:"1"
- human output mentions "Brain is healthy" on clean status
- dry-run: cycle runs but DB stays untouched
- exit code: clean/ok/partial do not call process.exit
Also (test/core/cycle.test.ts): refactored to use beforeAll/afterAll
with one shared PGLite engine per describe + truncateCycleLocks
between tests. Cuts test time from ~11s to ~4s; avoids the 15-migration
penalty per test that was causing parallel-suite timeout flakes.
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.17.0 — autopilot + jobs delegate to runCycle (unifies the cycle)
Autopilot daemon (`--inline` path) and Minions `autopilot-cycle`
handler both now delegate to `runCycle` (introduced in commit 4).
Three callers, one cycle definition:
1. `gbrain dream` — one-shot cron cycle
2. `gbrain autopilot` daemon inline path — scheduled cycles
3. `autopilot-cycle` Minions handler — durable queue with retry
All three share:
- Same 6 phases in same order (lint → backlinks → sync → extract →
embed → orphans)
- Same DB lock table coordination (`gbrain_cycle_locks`)
- Same yieldBetweenPhases discipline (prevents v0.14 stall-death)
- Same structured CycleReport output
Autopilot inline path gains lint + orphan sweep that the old path
skipped. Minions autopilot-cycle handler also gains lint + orphans.
Users who run `gbrain autopilot --install` see 6-phase reports in
`gbrain jobs get <id>` starting on next interval. No config change
required.
Changes:
- `src/commands/autopilot.ts`: inline fallback path (~20 lines)
replaces the ~22-line sync+extract+embed sequence with a single
runCycle call. Uses pull:true (matches pre-v0.17 autopilot
behavior). Uses setImmediate yield hook. Status/failure reporting
derives from CycleReport.status. `--help` cross-references `gbrain
dream` for one-shot use.
- `src/commands/jobs.ts:579` (`autopilot-cycle` handler): replaces
the 4-step try/catch sequence with a runCycle call. Returns
`{ partial, status, report }` so `gbrain jobs get <id>` shows the
full structured CycleReport. Preserves partial-failure semantic
(one phase failing does NOT throw; next cycle still runs).
yieldBetweenPhases yields the event loop between phases for the
worker's lock-renewal timer.
Release scaffolding:
- VERSION: 0.16.0 → 0.17.0
- CHANGELOG.md: v0.17.0 entry in GStack voice — headline, numbers
table, "what this means" paragraph, "To take advantage" block
per CLAUDE.md post-ship rules. Itemized changes below the fold.
Credit to @Wintermute for the original PR #309 thesis.
- skills/migrations/v0.17.0.md: documents what changed for
upgrading users. No mechanical action required — schema migration
v16 (cycle locks table) + handler delegation both apply
automatically. Includes opt-out paths for users who don't want
their daemon modifying files (use `dream --phase orphans` in cron
and skip autopilot-install, or other explicit configs).
- CLAUDE.md: new entries for `src/core/cycle.ts` and
`src/commands/dream.ts` with contract details.
Tests: no new test file needed for this commit — the cycle primitive
is extensively tested in test/core/cycle.test.ts (18 cases), dream
in test/dream.test.ts (11), and autopilot's delegation is mechanical
(calls runCycle with specific opts). The handler contract is covered
implicitly: if runCycle returns a CycleReport, the handler wraps it
in `{ partial, status, report }` — nothing else to assert.
Verified:
- `bun test test/autopilot-install.test.ts test/autopilot-resolve-cli.test.ts test/core/cycle.test.ts test/dream.test.ts` → 37 pass, 0 fail
Completes the v0.17.0 feature: 6 bisectable commits on one branch
(garrytan/v0.17-dream-cycle), ready to push as one PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): add runCycle + dream E2E coverage against real Postgres
Gap from the v0.17 commit series: PR #321 shipped unit-level tests
for runCycle (test/core/cycle.test.ts) and dream (test/dream.test.ts)
but no E2E coverage that exercises the real Postgres paths. Filling
that in before merge.
test/e2e/cycle.test.ts (6 cases):
- schema migration v16 created gbrain_cycle_locks + index
- dry-run full cycle: zero DB writes + lock table empty after
- live cycle: pages + chunks materialize, sync.last_commit set
- concurrent cycle blocked by lock → status:'skipped'
- TTL-expired lock auto-claimed (crashed-holder recovery)
- --phase orphans skips lock entirely (read-only optimization)
test/e2e/dream.test.ts (3 cases):
- dream --dry-run --json emits valid CycleReport + DB stays empty
- dream (no --dry-run) syncs pages into real DB
- dream --phase orphans doesn't touch the cycle-lock table
Both files mock embedBatch via mock.module so the embed phase never
calls OpenAI even when the full 6-phase cycle runs (zero API cost,
zero flakiness from network calls).
Verified locally:
- `docker run pgvector/pgvector:pg16` on port 5434
- `DATABASE_URL=... bun test test/e2e/cycle.test.ts test/e2e/dream.test.ts` → 9 pass, 0 fail
- Full E2E suite (`bun run test:e2e`): 16 files, 150 tests, 0 fail
- Container torn down after: `docker stop + rm gbrain-test-pg`
Per CLAUDE.md E2E test DB lifecycle. These tests skip gracefully when
DATABASE_URL isn't set (via hasDatabase() helper + describe.skip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wintermute <wintermute@garrytan.com>
|