From bca993e09f84f9b66f7d9900e6e0b7bf269119f7 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Thu, 7 May 2026 19:49:46 -0700 Subject: [PATCH] v0.28.12 feat: LongMemEval benchmark harness (#606) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * v0.28 schema: takes + synthesis_evidence (v31) + access_tokens.permissions (v32) Migration v31 adds the takes table (typed/weighted/attributed claims) and synthesis_evidence (provenance for `gbrain think` outputs). Page-scoped via page_id FK (slug isn't unique alone in v0.18+ multi-source). HNSW partial index on embedding for active rows. ON DELETE CASCADE on synthesis_evidence so deleting a source take cascades the provenance row. Migration v32 adds access_tokens.permissions JSONB with safe-default backfill (`{"takes_holders":["world"]}`). Default keeps non-world holders hidden from MCP-bound tokens until the operator explicitly grants access via the v0.28 auth permissions CLI. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 engine: addTakesBatch, listTakes, searchTakes/Vector, supersede, resolve, synthesis_evidence Extends BrainEngine with the takes domain object. Both engines implement the same surface; PGLite uses manual `$N` placeholders, Postgres uses postgres-js unnest() — same shape as addLinksBatch and addTimelineEntriesBatch. Methods: - addTakesBatch (upsert via ON CONFLICT (page_id, row_num) DO UPDATE) - listTakes (filter by holder/kind/active/resolved, takesHoldersAllowList for MCP-bound calls, sortBy weight/since_date/created_at) - searchTakes / searchTakesVector (pg_trgm + cosine; honor allow-list) - countStaleTakes / listStaleTakes (mirror countStaleChunks pattern; embedding column intentionally omitted from listStale payload) - updateTake (mutable fields only; throws TAKE_ROW_NOT_FOUND) - supersedeTake (transactional: insert new at next row_num, mark old active=false, set superseded_by; throws TAKE_RESOLVED_IMMUTABLE on resolved bets) - resolveTake (sets resolved_*; throws TAKE_ALREADY_RESOLVED on re-resolve; resolution is immutable per Codex P1 #13 fold) - addSynthesisEvidence (provenance persist; ON CONFLICT DO NOTHING) - getTakeEmbeddings (parallel to getEmbeddingsByChunkIds) Types live in src/core/engine.ts adjacent to LinkBatchInput. Page-scoped via page_id (slug not unique in v0.18+ multi-source). PageType gains 'synthesis'. takeRowToTake mapper in utils.ts handles Date → ISO string normalization. Tests: test/takes-engine.test.ts — 16 cases against PGLite covering upsert/list/filter/search happy paths, takesHoldersAllowList isolation, the four invariant errors (TAKE_ROW_NOT_FOUND, TAKES_WEIGHT_CLAMPED, TAKE_RESOLVED_IMMUTABLE, TAKE_ALREADY_RESOLVED), supersede flow, resolve metadata round-trip, FK CASCADE on synthesis_evidence when source take deletes. All pass. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 model-config: unified resolveModel with 6-tier precedence + alias resolution Replaces every hardcoded `claude-*-X` and per-phase `dream..model` config key with a single resolver. Hierarchy: 1. CLI flag (--model) 2. New-key config (e.g. models.dream.synthesize) 3. Old-key config (deprecated dream.synthesize.model, dream.patterns.model) — read with stderr deprecation warning, one-per-process 4. Global default (models.default) 5. Env var (GBRAIN_MODEL or caller-supplied) 6. Hardcoded fallback Aliases (`opus`, `sonnet`, `haiku`, `gemini`, `gpt`) resolve at the end so any tier can use a short name. User-defined `models.aliases.` config overrides built-ins. Cycle-safe (depth 2 break). Unknown alias passes through unchanged so users can pass full provider IDs without registering. When new-key + old-key are BOTH set (Codex P1 #11 fix), new-key wins and stderr warns "deprecated config X ignored; Y is set and wins". When only old-key is set, it's honored with a softer "rename to Y before v0.30" warning. Both warnings emit once per (key, process) — a Set memo prevents log spam in long-running daemons. Migrated call sites: synthesize.ts (model + verdictModel), patterns.ts (model). subagent.ts and search/expansion.ts to be migrated later in v0.28 (staying compatible until then). Tests: test/model-config.test.ts — 11 cases pinning the 6-tier ordering, alias resolution + cycle break, deprecated-key warning emit-once, and unknown-alias pass-through. All pass. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 takes-fence: parser/renderer/upserter + chunker strip (privacy P0 fix) src/core/takes-fence.ts — pure functions for the fenced markdown surface: - parseTakesFence(body) — extracts ParsedTake[] from `` blocks. Strict on canonical form, lenient on hand-edits with warnings (TAKES_FENCE_UNBALANCED, TAKES_TABLE_MALFORMED, TAKES_ROW_NUM_COLLISION). Strikethrough `~~claim~~` → active=false; date ranges `since → until` split into sinceDate/untilDate. - renderTakesFence(takes) — round-trip safe with parseTakesFence. - upsertTakeRow(body, row) — append-only per CEO-D6 + eng-D9. Creates a fresh `## Takes` section if no fence present. row_num is monotonic (max + 1, never gap-filled — keeps cross-page refs and synthesis_evidence stable forever). - supersedeRow(body, oldRow, replacement) — strikes through old row's claim AND appends the new row at end. Both rows preserved in markdown for git-blame archaeology. - stripTakesFence(body) — removes the fenced block entirely. Used by the chunker so takes content lives ONLY in the takes table. Codex P0 #3 fix: src/core/chunkers/recursive.ts now calls stripTakesFence() before computing chunk boundaries. Without this, page chunks would contain the rendered takes table and the per-token MCP allow-list would be bypassed at the index layer (token bound to takes_holders=['world'] would see garry's hunches via page hits). Doctor's takes_fence_chunk_leak check (plan-side) asserts no chunk contains the begin marker. Tests: 15 cases covering canonical parse, strikethrough, date range, fence unbalanced detection, malformed-row skip + warning, row_num collision detection, round-trip render, append-only upsert into existing fence, fresh-section creation, monotonic row_num under hand-edit gaps, supersede flow, stripTakesFence verifying takes content removed AND surrounding prose preserved. Existing chunker tests still pass (15 + 15 = 30). Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 page-lock: PID-liveness file lock for atomic markdown read-modify-write src/core/page-lock.ts — per-page file lock at ~/.gbrain/page-locks/.lock so two concurrent `gbrain takes add` calls or `takes seed --refresh` from autopilot can't race on the same `.md` read-modify-write. Eng-review fold: reuses the v0.17 cycle.lock pattern (mtime + PID liveness) but per-slug. Differences from cycle.ts's lock: - SHA-256 of slug for safe filenames (slashes, unicode, etc.) - Same-pid + fresh mtime = LIVE (cycle.ts assumes one lock per process and reclaims same-pid; page-lock allows concurrent locks for DIFFERENT slugs in one process). mtime expiry still rescues post-crash leftovers. - 5-min TTL (vs cycle's 30 min — page edits are short) - `withPageLock(slug, fn)` convenience wrapper with default 30s timeout API: - acquirePageLock(slug, opts) → handle | null (poll-with-timeout) - handle.refresh() / handle.release() (idempotent — only releases if pid matches) - withPageLock(slug, fn, opts) — acquire + run + release-in-finally Tests: 10 cases — fresh acquire, live holder returns null, stale-mtime reclaim, dead-PID reclaim, refresh updates timestamp, foreign-pid release is no-op, withPageLock callback runs and releases on success/failure, timeout-throws when held, SHA-256 filename safety for slashes/unicode. All pass. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 extract-takes: dual-path phase (fs|db) + since/until_date as TEXT src/core/cycle/extract-takes.ts — new phase that materializes the takes table from fenced markdown blocks. Two paths mirror src/commands/extract.ts: - extractTakesFromFs: walk *.md under repoPath, parse fences, batch upsert - extractTakesFromDb: iterate engine.getAllSlugs(), parse each page's compiled_truth+timeline, batch upsert (mutation-immune snapshot iteration) Single dispatcher extractTakes(opts) routes by source. Honors: - slugs filter for incremental re-extract (pipes from sync→extract) - dryRun: count would-be upserts, write nothing - rebuild: DELETE FROM takes WHERE page_id = $1 before re-insert (clean slate when markdown is canonical and DB has drifted) Schema fix: since_date/until_date were DATE in the original v31 migration. Spec uses partial dates ('2017-01', '2026-04-29 → 2026-06') that Postgres DATE rejects. Changed to TEXT in both the Postgres and PGLite blocks so parser-rendered ranges round-trip cleanly. Loses the ability to do date-range arithmetic in SQL, but date math on opinion timelines is out of scope for v0.28 anyway. utils.ts dateOrNull now annotated as v0.28 TEXT-aware. Migration v31 has not been deployed yet (this branch is the v0.28 release candidate), so the type swap is free. No data migration needed. Tests: test/extract-takes.test.ts — 5 cases against PGLite covering full walk + fence-skip on no-fence pages, takes-table populated post-extract, incremental slugs filter, dry-run no-write, rebuild=true clears + re-inserts ad-hoc rows. test/takes-engine.test.ts (16), test/takes-fence.test.ts (15) all still pass — 36/36 takes tests green. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 takes CLI: list, search, add, update, supersede, resolve src/commands/takes.ts — surfaces the engine methods + takes-fence library through a single `gbrain takes ` entrypoint: takes list with filters + sort takes search "" pg_trgm keyword search across all takes takes add --claim ... ... append (markdown + DB, atomic via lock) takes update --row N ... mutable-fields update (markdown + DB) takes supersede --row N ... strikethrough old + append new takes resolve --row N --outcome record bet resolution (immutable) Markdown is canonical. Every mutate command: 1. acquires the per-page file lock (withPageLock) 2. re-reads the .md file 3. applies the edit via takes-fence (upsertTakeRow / supersedeRow) 4. writes the .md file back 5. mirrors to the DB via the engine method 6. releases the lock (auto via finally) Resolve currently writes only to DB — surfacing resolved_* in the markdown table is deferred to v0.29 (the takes-fence renderer's column set is fixed at # | claim | kind | who | weight | since | source per spec). Wired into src/cli.ts dispatch + CLI_ONLY allowlist. Help text follows the project convention (orphans/embed/extract pattern). --dir flag overrides sync.repo_path config when working outside the configured brain. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 MCP + auth: takes_list / takes_search / think ops + per-token allow-list OperationContext gains takesHoldersAllowList — server-side filter for takes.holder field threaded from access_tokens.permissions through dispatch into the engine SQL. Closes Codex P0 #3 at the dispatch layer (chunker strip already closed the page-content side in the previous commit). src/core/operations.ts — three new ops: - takes_list: lists takes with holder/kind/active/resolved filters; honors ctx.takesHoldersAllowList for MCP-bound calls - takes_search: pg_trgm keyword search; honors allow-list - think: op surface registered (returns not_implemented envelope until Lane D's pipeline lands). Remote callers cannot save/take per Codex P1 #7. src/mcp/dispatch.ts — DispatchOpts.takesHoldersAllowList threads into buildOperationContext. src/mcp/http-transport.ts — validateToken now reads access_tokens.permissions.takes_holders, defaults to ['world'] when the column is absent or malformed (default-deny on private hunches). auth.takesHoldersAllowList passed to dispatchToolCall. src/mcp/server.ts (stdio) — defaults to takesHoldersAllowList: ['world'] since stdio has no per-token auth. Operators wanting full visibility use `gbrain call ` directly (sets remote=false). src/commands/auth.ts — `gbrain auth create --takes-holders w,g,b` flag persists the per-token list; new `auth permissions set-takes-holders ` updates an existing token. Tests: test/takes-mcp-allowlist.test.ts — 8 cases against PGLite proving the threading: local-CLI sees all holders, ['world'] returns only public, ['world','garry'] returns 2/3, no-overlap returns empty (no fallback), search honors allow-list, remote save/take on think rejected with not_implemented envelope. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28.0: ship-prep — VERSION, CHANGELOG, migration orchestrator, skill Closes the v0.28 ship-prep cycle. Bumps VERSION + package.json + bun.lock to 0.28.0. v0_28_0 migration orchestrator runs three idempotent phases on upgrade: - Schema verify: asserts schema_version >= 32 (migrations v31 + v32 already applied by the schema runner during gbrain upgrade); fails clean if not. - Backfill takes: inline runs `extractTakes(engine, { source: 'db' })` so any pre-existing fenced takes tables in markdown populate the takes index. Idempotent; ON CONFLICT DO UPDATE keeps the table in sync. - Re-chunk TODO: queues a pending-host-work entry asking the host agent to re-import pages with takes content so the v0.28 chunker-strip rule (Codex P0 #3 fix) applies retroactively. Pages imported under v0.28+ already have takes content stripped from chunks at index time; this TODO catches up legacy pages. skills/migrations/v0.28.0.md — agent-readable upgrade guide. Walks through doctor verification, deprecated-key migration, MCP token visibility configuration, and a "try the takes layer" smoke test. CHANGELOG.md — v0.28.0 release-summary in the GStack voice (no AI vocabulary, no em dashes, real numbers from git diff stat) + the mandatory "To take advantage of v0.28.0" block + itemized changes by subsystem (schema, engine, markdown surface, model config, MCP+auth, CLI, tests, accepted risks). Final test sweep: 65/65 v0.28 tests pass across 6 files. typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 think pipeline: gather → sanitize → synthesize → cite-render → CLI src/core/think/sanitize.ts — prompt-injection defense for take claims: 14 jailbreak patterns (ignore-prior, role-jailbreak, close-take tag, DAN, system-prompt overrides, eval-shell hooks) plus structural framing (takes wrapped in tags the model is told to treat as DATA). Length-cap at 500 chars. Renders evidence blocks for the prompt. src/core/think/prompt.ts — system prompt + structured-output schema. Hard rules: cite every claim, mark hunches/low-weight explicitly, surface conflicts (never silently pick), surface gaps. JSON schema with answer + citations[] + gaps[]. Prompt adapts to anchor / time window / save flag. src/core/think/cite-render.ts — structured citations + regex fallback (Codex P1 #4 fold). normalizeStructuredCitations validates the model's structured output; parseInlineCitations is the body-scan fallback when the model omits the structured field. resolveCitations dispatches and records CITATIONS_REGEX_FALLBACK warning when used. src/core/think/gather.ts — 4-stream parallel retrieval: 1. hybridSearch (pages, existing primitive) 2. searchTakes (keyword, pg_trgm) 3. searchTakesVector (vector, when embedQuestion fn supplied) 4. traversePaths (graph, when --anchor set) RRF fusion (k=60). Each stream wrapped in try/catch — partial gather beats no synthesis. Honors takesHoldersAllowList for MCP-bound calls. src/core/think/index.ts — runThink orchestrator + persistSynthesis: INTENT (regex classify) → GATHER → render evidence blocks → resolveModel ('models.think' → 'models.default' → GBRAIN_MODEL → opus) → LLM call (injectable client) → JSON parse with code-fence + fallback strip → resolveCitations → ThinkResult. persistSynthesis writes a synthesis page + synthesis_evidence rows (page_id resolved per slug; page-level citations skip evidence). Degrades gracefully without ANTHROPIC_API_KEY. Round-loop scaffolding in place (rounds=1 only path exercised in v0.28). src/commands/think.ts — `gbrain think ""` CLI. Flag parsing strips --anchor, --rounds, --save, --take, --model, --since, --until, --json. Local CLI = remote=false, so save/take honored. Human-readable output by default; --json for agent consumption. operations.ts — `think` op now calls runThink (was a not_implemented stub). Remote callers can't save/take per Codex P1 #7. Returns full ThinkResult plus saved_slug + evidence_inserted. cli.ts — wired into dispatch + CLI_ONLY allowlist. Tests: test/think-pipeline.test.ts — 18 cases against PGLite covering sanitize patterns, structural rendering, citation parsing (structured + regex fallback + dedup + invalid-slug rejection), gather streams + allow-list filter, full pipeline with stub client, malformed-LLM fallback path, no-API-key graceful degradation, persistSynthesis writes page + evidence rows. All pass. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 dream phases: auto-think + drift + budget meter (Codex P1 #10 fold) src/core/anthropic-pricing.ts — USD/1M-tokens map for Claude 4.7 family plus older aliases. estimateMaxCostUsd returns null on unpriced models so the meter caller can warn-once and bypass the gate. src/core/cycle/budget-meter.ts — cumulative cost ledger. Each submit estimates max-cost from (model + estimatedInputTokens + maxOutputTokens), accumulates per-cycle, refuses next submit when projected > cap. Codex P1 #10 fold: non-Anthropic models (gemini, gpt) bypass with one stderr warn per process and `unpriced=true` on the result. Budget=0 disables the gate. Audit trail at ~/.gbrain/audit/dream-budget-YYYY-Www.jsonl. src/core/cycle/auto-think.ts — auto_think dream phase. Reads dream.auto_think.{enabled,questions,max_per_cycle,budget,cooldown_days, auto_commit}. Iterates configured questions through runThink with the BudgetMeter pre-checking each submit. Cooldown timestamp written ONLY on success (matches v0.23 synthesize pattern — retries after partial failures pick back up). When auto_commit=true, persists synthesis pages via persistSynthesis. Default-disabled. src/core/cycle/drift.ts — drift dream phase scaffold. Reads dream.drift.{enabled,lookback_days,budget,auto_update}. Surfaces takes in the soft band (weight 0.3-0.85, unresolved) that have recent timeline evidence on the same page. v0.28 ships the orchestration; the LLM judge that proposes weight adjustments lands in v0.29. modelId + meter wired now so the ledger captures gate state for callers that opt in. Tests: - test/budget-meter.test.ts (7 cases) — pricing-map coverage, allow path, cumulative-deny, budget=0 disabled, unpriced bypass+warn-once, ledger captures all events, ISO-week filename branch. - test/auto-think-phase.test.ts (9 cases) — auto_think enable/skip, questions empty, success → cooldown ts written, cooldown blocks rerun, budget exhausted → partial. drift not_enabled, soft-band candidate detection, complete + dry-run paths. All pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 e2e Postgres: takes engine + extract + MCP allow-list (12 cases) test/e2e/takes-postgres.test.ts — full v0.28 takes pipeline against real Postgres (gated on DATABASE_URL). 12 cases: - addTakesBatch upsert via unnest() bind path (Postgres-specific) - listTakes filters: holder, kind, sort=weight, takesHoldersAllowList - searchTakes pg_trgm + allow-list filter - supersedeTake transactional path (BEGIN/COMMIT semantics) - resolveTake immutability — second resolve throws TAKE_ALREADY_RESOLVED - synthesis_evidence FK CASCADE on take delete - countStaleTakes + listStaleTakes filter active+null - extractTakesFromDb populates takes from fenced markdown - MCP dispatch with takesHoldersAllowList=['world'] returns only world - MCP dispatch local-CLI path returns all holders - MCP dispatch takes_search honors allow-list - think op forces remote_persisted_blocked even for save+take postgres-engine.ts: addTakesBatch boolean[] serialization fix. postgres-js auto-detects element type from JS arrays; for booleans it mis-detects as scalar. Cast through text[] (`'true' | 'false'`) then SQL-cast to boolean[] — same pattern other batch methods rely on for type-stable bind shapes. test/e2e/helpers.ts: setupDB now (a) tolerates non-existent tables in TRUNCATE (for fresh DBs where v31 hasn't yet created takes/synthesis_evidence) and (b) calls engine.initSchema() to actually run migrations. test/takes-mcp-allowlist.test.ts: updated 2 think-op cases to match Lane D's landed pipeline. They previously asserted not_implemented envelopes; now they assert remote_persisted_blocked + NO_ANTHROPIC_API_KEY graceful-degrade behavior. Run: DATABASE_URL=postgres://localhost:5435/gbrain_test bun test test/e2e/takes-postgres.test.ts Result: 12/12 pass. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 dream phases: local DreamPhaseResult type (avoid premature CyclePhase enum extension) cycle.ts's PhaseResult is shaped {phase, status, summary, details} with a narrow PhaseStatus enum ('ok'|'warn'|'fail'|'skipped') and CyclePhase enum that doesn't yet include 'auto_think'/'drift'. The phases ship standalone in v0.28 (cycle.ts dispatcher integration is v0.28.x); using PhaseResult forced premature enum extension. Introduces DreamPhaseResult exported from auto-think.ts: { name: 'auto_think'|'drift'; status: 'complete'|'partial'|'failed'|'skipped'; detail: string; totals?: Record; duration_ms: number } drift.ts re-exports the same type. When v0.28.x wires the dispatcher, the adapter at the call site can map DreamPhaseResult → PhaseResult cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 e2e: access_tokens.permissions JSONB end-to-end (5 cases) test/e2e/auth-permissions.test.ts — closes the v0.28 token-allow-list verification loop against real Postgres. Exercises: - Migration v32 default backfill: new tokens created without a permissions column get {takes_holders: ["world"]} via the schema DEFAULT clause. - Explicit ["world","garry"] → dispatch.takes_list filters to those holders only; brain hunches stay hidden from this token. - ["world"] default-deny token → takes_search hits filtered to public claims. - {} permissions row (operator tampered) gracefully defaults to ["world"] via the HTTP transport's validateToken parsing. - revoked_at IS NOT NULL → token excluded from active token query. Avoids the postgres-js JSONB double-encode trap (CLAUDE.md memory): pass the object directly to executeRaw, no JSON.stringify, no ::jsonb cast. All 5 pass against pgvector/pgvector:pg16 on port 5435. Combined v0.28 test sweep: 116/116 across 11 files. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.28 e2e: chunker takes-strip integration test (Codex P0 #3 verification) test/e2e/chunker-takes-strip.test.ts — verifies the chunker actually strips fenced takes content end-to-end through the import pipeline. This is the Codex P0 #3 fix's verification path: takes content lives ONLY in the takes table for retrieval, never duplicated in content_chunks where the per-token MCP allow-list cannot reach. 5 cases: - chunkText (unit) output never contains TAKES_FENCE_BEGIN/END markers - chunkText output never contains fenced claim text - chunkText output retains non-fence prose (no over-stripping) - importFromContent end-to-end: imported page has chunks but none contain fenced content - takes_fence_chunk_leak doctor invariant: zero rows globally where chunk_text matches `