diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fe07fe09..92a85a5c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,73 @@ All notable changes to GBrain will be documented in this file. +## [0.41.30.0] - 2026-05-30 + +**`gbrain lsd --save` (and `brainstorm --save`) now actually writes the +committable `wiki/ideas/.md` file it always promised, makes the saved +idea searchable, and tells you the truth when a save fails instead of always +printing "Saved".** + +The old behavior was a quiet lie. `--save` printed `Saved to ` every +time, but under the hood it only did a lightweight database write. The help +text promised a `.md` file on disk for you to commit to your brain repo, and +that file never appeared. Worse, when the database write itself failed (which +happens under PgBouncer transaction-mode), it still said "Saved" — a real LSD +run reported success while `gbrain get ` returned `page_not_found`. +Nothing landed anywhere, and the only way to recover the idea was to scrape it +back out of the terminal. + +Now the save runs through the same ingestion path the rest of gbrain uses. The +page is written canonically (chunked and tagged, so `gbrain search` can find +it), and when your `sync.repo_path` points at a real brain repo, the +`wiki/ideas/--.md` file is written to disk, +rendered from the saved row so the two can't drift apart. The success message +now names exactly which sinks landed. If nothing persisted, you get a loud +`NOT persisted` error and a nonzero exit code, so a scripted `--save` can no +longer be mistaken for success. + +Same-day ideas no longer clobber each other either: the slug gets a short +random suffix, so two quick `lsd` runs on similar questions keep both files. + +How to use it: + +``` +gbrain config set sync.repo_path /path/to/your/brain # if not already set +gbrain lsd "why are agent tools converging" --save +gbrain get wiki/ideas/2026-05-30-lsd-why-are-agent-tools-converging- +ls /path/to/your/brain/wiki/ideas/ # the .md file is really there +``` + +No `sync.repo_path`? You still get the database page (queryable immediately), +and the message tells you the file write was skipped. Nothing changes for +`--json` callers (that path stays database-only, as before). + +### For contributors + +The disk write-through that `put_page` shipped in v0.38 was extracted into a +shared `src/core/write-through.ts` helper that `put_page` and brainstorm save +now both call, and it was upgraded to write atomically (unique temp file + +rename) so a crash or a concurrently-running `gbrain sync` can never read a +half-written `.md`. That hardens `put_page`'s own write-through as a side +effect. + +## To take advantage of v0.41.30.0 + +`gbrain upgrade` is all you need. There is no schema migration and no data +backfill in this release. To confirm the fix: + +1. **Set a repo path if you haven't:** `gbrain config set sync.repo_path /path/to/your/brain` +2. **Save an idea and verify both sinks:** + ```bash + gbrain lsd "test idea" --save + gbrain get # database page exists + ls /path/to/your/brain/wiki/ideas/ # the .md file exists + ``` +3. **If a save ever reports `NOT persisted`,** that is the new honest failure + path doing its job — check the stderr line above it for the underlying DB + or file error, then please file an issue with `gbrain doctor` output: + https://github.com/garrytan/gbrain/issues + ## [0.41.29.0] - 2026-05-29 **Your meeting transcripts that look like `**Garry Tan:** ...` now actually diff --git a/CLAUDE.md b/CLAUDE.md index 4b26413c4..6a0cb6406 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,7 +40,7 @@ strict behavior when unset. ## Key files -- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. +- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). @@ -256,7 +256,8 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. +- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. **v0.41.30.0:** `--save` for both `brainstorm` and `lsd` rewritten to persist through the canonical ingestion path instead of the lightweight DB-only write that never produced the advertised `.md` file. New exported `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so `gbrain search` finds the idea; no embedding cost at save time) THEN renders the saved row to disk via the shared `writePageThrough` helper (see `src/core/write-through.ts`) — file rendered FROM the row so the two sinks can't diverge and a later `gbrain sync` doesn't churn it. New exported pure `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message: both-sinks success, DB-only (no `sync.repo_path` / repo-not-a-dir → file write skipped, exit 0), DB-saved-but-file-errored (exit 0, sync reconciles), and the total-failure case (neither sink → loud `save FAILED … NOT persisted` on stderr + nonzero exit). Closes the silent-false-success bug class where `--save` printed "Saved" unconditionally even when the PgBouncer-transaction-mode DB write failed and `gbrain get ` then returned `page_not_found`. `buildIdeaSlug(question, label, nonce?)` gains a random nonce suffix (injectable for deterministic tests) so two same-day runs whose questions share the first 60 slug chars no longer clobber each other's page + file. `--json` callers stay DB-only (unchanged). New exported `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (the string `buildBrainstormFrontmatter` is left untouched). Pinned by `test/brainstorm/save.test.ts` (covers `persistSavedIdea` both-sinks / DB-only / total-failure paths, `formatSaveOutcome` per-branch messages + exit codes, and `buildIdeaSlug` nonce collision-resistance). +- `src/core/write-through.ts` (v0.41.30.0, NEW) — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` reads `sync.repo_path`, re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown` + `resolvePageFilePath`, and writes the `.md` under the repo so the brain has a committable artifact that round-trips through `gbrain sync`. Rendering FROM the row means the file and the DB row cannot diverge. Extracted from the v0.38 `put_page` write-through in `src/core/operations.ts` (which the op now calls instead of hand-rolling its own copy) AND upgraded to ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync` into place, cleaning up the temp on any failure, so a crash or a concurrently-running `gbrain sync`/autopilot walking the live git tree can never read a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op (write-through behavior preserved, now atomic) and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts` (no-repo / repo-not-found / page-not-found-after-write skips, atomic-rename happy path + no stray `.tmp` on write failure, frontmatter-override stamping). - `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. - `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. diff --git a/TODOS.md b/TODOS.md index e8ce40ba6..c04699441 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,13 @@ # TODOS +## brainstorm/lsd --save source-awareness (v0.42+) + +Filed from the `--save` dual-sink hardening wave (route through the canonical +ingestion path: `importFromContent({noEmbed:true})` + the shared +`writePageThrough` helper extracted from `put_page`). + +- [ ] **v0.42+: make `gbrain brainstorm/lsd --save` source-aware.** Today the save path always writes to `source='default'` — `persistSavedIdea` (`src/commands/brainstorm.ts`) hardcodes `sourceId ?? 'default'`, and there is no `--save`-side `--source` flag. Both sinks stay consistent at default (no live bug), but on a multi-source brain a generated idea can't be filed to a non-default source. **What:** add a `--source ` option to brainstorm/lsd, resolve it via `resolveSourceWithTier`, and thread `sourceId` into `persistSavedIdea` → `importFromContent({sourceId})` + `writePageThrough({sourceId})`. **Why:** complete the multi-source story for generated ideas; the disk layout already handles it. **Context:** `writePageThrough` and `resolvePageFilePath` already take `sourceId` and emit `.sources//.md` for non-default sources, and `importFromContent` already accepts `sourceId` — so the only missing piece is the CLI flag + threading. `runBrainstorm` (orchestrator) already accepts `sourceId` for the close/far READ side. **Depends on:** nothing; purely additive. Priority: P3 (default-source is the common case). + ## v0.41.29.0 orphan source-scoping follow-ups (v0.42+) Filed from the v0.41.29.0 wave (bold-name-no-time pattern + orphan_ratio diff --git a/VERSION b/VERSION index 5625f3eba..13bcae3ae 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.29.0 +0.41.30.0 diff --git a/llms-full.txt b/llms-full.txt index 0752d9c1d..422ff20b2 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -182,7 +182,7 @@ strict behavior when unset. ## Key files -- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. +- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). @@ -398,7 +398,8 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. +- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. **v0.41.30.0:** `--save` for both `brainstorm` and `lsd` rewritten to persist through the canonical ingestion path instead of the lightweight DB-only write that never produced the advertised `.md` file. New exported `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so `gbrain search` finds the idea; no embedding cost at save time) THEN renders the saved row to disk via the shared `writePageThrough` helper (see `src/core/write-through.ts`) — file rendered FROM the row so the two sinks can't diverge and a later `gbrain sync` doesn't churn it. New exported pure `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message: both-sinks success, DB-only (no `sync.repo_path` / repo-not-a-dir → file write skipped, exit 0), DB-saved-but-file-errored (exit 0, sync reconciles), and the total-failure case (neither sink → loud `save FAILED … NOT persisted` on stderr + nonzero exit). Closes the silent-false-success bug class where `--save` printed "Saved" unconditionally even when the PgBouncer-transaction-mode DB write failed and `gbrain get ` then returned `page_not_found`. `buildIdeaSlug(question, label, nonce?)` gains a random nonce suffix (injectable for deterministic tests) so two same-day runs whose questions share the first 60 slug chars no longer clobber each other's page + file. `--json` callers stay DB-only (unchanged). New exported `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (the string `buildBrainstormFrontmatter` is left untouched). Pinned by `test/brainstorm/save.test.ts` (covers `persistSavedIdea` both-sinks / DB-only / total-failure paths, `formatSaveOutcome` per-branch messages + exit codes, and `buildIdeaSlug` nonce collision-resistance). +- `src/core/write-through.ts` (v0.41.30.0, NEW) — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` reads `sync.repo_path`, re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown` + `resolvePageFilePath`, and writes the `.md` under the repo so the brain has a committable artifact that round-trips through `gbrain sync`. Rendering FROM the row means the file and the DB row cannot diverge. Extracted from the v0.38 `put_page` write-through in `src/core/operations.ts` (which the op now calls instead of hand-rolling its own copy) AND upgraded to ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync` into place, cleaning up the temp on any failure, so a crash or a concurrently-running `gbrain sync`/autopilot walking the live git tree can never read a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op (write-through behavior preserved, now atomic) and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts` (no-repo / repo-not-found / page-not-found-after-write skips, atomic-rename happy path + no stray `.tmp` on write failure, frontmatter-override stamping). - `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. - `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. diff --git a/package.json b/package.json index 63e002a35..85a907ce1 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.29.0" + "version": "0.41.30.0" } diff --git a/src/commands/brainstorm.ts b/src/commands/brainstorm.ts index 14976ca78..253bd4ad9 100644 --- a/src/commands/brainstorm.ts +++ b/src/commands/brainstorm.ts @@ -16,13 +16,17 @@ import type { BrainEngine } from '../core/engine.ts'; import { runBrainstorm, formatBrainstormMarkdown, - buildBrainstormFrontmatter, + buildBrainstormFrontmatterObject, BRAINSTORM_PROFILE, LSD_PROFILE, type BrainstormProfile, } from '../core/brainstorm/orchestrator.ts'; import { loadConfig } from '../core/config.ts'; import { StructuredAgentError } from '../core/errors.ts'; +import { serializeMarkdown } from '../core/markdown.ts'; +import { importFromContent } from '../core/import-file.ts'; +import { writePageThrough, type WriteThroughResult } from '../core/write-through.ts'; +import { randomBytes } from 'crypto'; export interface BrainstormCliArgs { question?: string; @@ -305,37 +309,144 @@ async function runBrainstormCli( const shouldSave = parsed.save ?? profile.default_save; if (shouldSave) { const slug = buildIdeaSlug(parsed.question, profile.label); - const frontmatter = buildBrainstormFrontmatter(result, { slug }); - // Re-render content for save: include filtered ideas too so --retry-judge - // (when implemented) has the full set to re-score. + const title = `${profile.label === 'lsd' ? 'LSD' : 'Brainstorm'}: ${parsed.question.slice(0, 100)}`; + // Build ONE frontmatter object and render via the canonical serializer so + // the saved file round-trips through `gbrain sync` byte-for-byte. Include + // filtered ideas (onlyPassed:false) so a future --retry-judge has the full + // set to re-score. + const fmObj = buildBrainstormFrontmatterObject(result); const body = formatBrainstormMarkdown(result, { onlyPassed: false, includeMeta: true }); - const content = frontmatter + body; - try { - await engine.putPage(slug, { - title: `${profile.label === 'lsd' ? 'LSD' : 'Brainstorm'}: ${parsed.question.slice(0, 100)}`, - type: 'note', - compiled_truth: content, - frontmatter: { - mode: profile.frontmatter_mode, - generated_at: new Date().toISOString(), - question: parsed.question, - judge_failed: result.judge_failed, - unscored: result.judge_failed, - close_slugs: result.close_set.map((c) => c.slug), - far_slugs: result.far_set.map((f) => f.slug), - }, - timeline: '', - }); - console.log(`\n_Saved to \`${slug}\`._`); - } catch (err) { - const msg = err instanceof Error ? err.message : String(err); - console.error(`gbrain ${profile.label}: save failed: ${msg}`); - } + const content = serializeMarkdown(fmObj, body, '', { type: 'note', title, tags: [] }); + + const outcome = await persistSavedIdea(engine, { slug, content, provenanceVia: profile.label }); + const msg = formatSaveOutcome(outcome, { profileLabel: profile.label, slug }); + if (msg.stdout) console.log(msg.stdout); + for (const line of msg.stderr) console.error(line); + if (msg.exitCode) process.exitCode = msg.exitCode; } } -/** Slugify the question for the saved page path. Capped + collision-resistant via date prefix. */ -function buildIdeaSlug(question: string, label: 'brainstorm' | 'lsd'): string { +/** Outcome of persisting a saved idea to both sinks. */ +export interface SaveOutcome { + /** True when the canonical DB import (importFromContent) succeeded. */ + dbSaved: boolean; + /** Set when the DB import threw. */ + dbError?: string; + /** Disk write-through result (rendered from the saved row). */ + writeThrough: WriteThroughResult; +} + +export interface SaveMessage { + /** Human-readable success line for stdout (omitted when nothing persisted). */ + stdout?: string; + /** Error / warning lines for stderr. */ + stderr: string[]; + /** Nonzero ONLY when nothing was persisted (no DB row AND no file). */ + exitCode: number; +} + +/** + * Persist a saved idea through the CANONICAL ingestion path: importFromContent + * (chunks + tags + content_hash + source_path, but `noEmbed` so we don't pay + * embedding cost at save time) writes the DB row, then the shared + * `writePageThrough` helper renders that row to disk. Rendering from the row + * means the two sinks cannot diverge, and the row matches what `gbrain sync` + * would produce — so a later sync doesn't churn it. The file is only attempted + * when the DB write landed (it's rendered from the row). + */ +export async function persistSavedIdea( + engine: BrainEngine, + args: { slug: string; content: string; sourceId?: string; provenanceVia: string }, +): Promise { + const sourceId = args.sourceId ?? 'default'; + let dbSaved = false; + let dbError: string | undefined; + try { + await importFromContent(engine, args.slug, args.content, { + noEmbed: true, + sourceId, + sourcePath: `${args.slug}.md`, + }); + dbSaved = true; + } catch (err) { + dbError = err instanceof Error ? err.message : String(err); + } + const writeThrough: WriteThroughResult = dbSaved + ? await writePageThrough(engine, args.slug, { + sourceId, + frontmatterOverrides: { source_kind: args.provenanceVia }, + }) + : { written: false, skipped: 'page_not_found_after_write' }; + return { dbSaved, dbError, writeThrough }; +} + +/** + * Render an honest save message from the outcome. Every branch names the real + * state; the only nonzero exit is the total-failure case (nothing persisted), + * so scripts can't read a failed `--save` as success. A file-write failure when + * the DB row landed stays exit 0 — the row is durable and `gbrain sync` + * reconciles the disk file on the next run. + */ +export function formatSaveOutcome( + outcome: SaveOutcome, + ctx: { profileLabel: string; slug: string }, +): SaveMessage { + const { dbSaved, dbError, writeThrough } = outcome; + const stderr: string[] = []; + if (dbError) stderr.push(`gbrain ${ctx.profileLabel}: DB save failed: ${dbError}`); + if (writeThrough.error) { + stderr.push(`gbrain ${ctx.profileLabel}: file write failed: ${writeThrough.error}`); + } + + if (dbSaved && writeThrough.written) { + return { + stdout: `\n_Saved to DB page \`${ctx.slug}\` and file \`${writeThrough.path}\`._`, + stderr, + exitCode: 0, + }; + } + if (dbSaved && writeThrough.skipped === 'no_repo_configured') { + return { + stdout: `\n_Saved to DB page \`${ctx.slug}\` (no \`sync.repo_path\` set — skipped file write)._`, + stderr, + exitCode: 0, + }; + } + if (dbSaved && writeThrough.skipped === 'repo_not_found') { + return { + stdout: `\n_Saved to DB page \`${ctx.slug}\` (\`sync.repo_path\` is not a directory — skipped file write)._`, + stderr, + exitCode: 0, + }; + } + if (dbSaved) { + // File write attempted but errored (already on stderr). Row is durable. + return { + stdout: `\n_Saved to DB page \`${ctx.slug}\` (file NOT written — see error above; \`gbrain sync\` will reconcile)._`, + stderr, + exitCode: 0, + }; + } + // Nothing persisted — the silent-false-success bug class. Exit nonzero. + stderr.push( + `gbrain ${ctx.profileLabel}: save FAILED — neither DB page nor file was written. The idea is NOT persisted.`, + ); + return { stderr, exitCode: 1 }; +} + +/** + * Slugify the question for the saved page path. Collision-resistant via a date + * prefix AND a random nonce suffix — two same-day runs whose questions share + * the first 60 slug chars (or both slugify to empty → `untitled`) would + * otherwise produce the same slug, and both the DB upsert and the file write + * would silently clobber the earlier idea. The nonce is injectable so tests are + * deterministic; production uses crypto random. + */ +export function buildIdeaSlug( + question: string, + label: 'brainstorm' | 'lsd', + nonce?: string, +): string { const date = new Date().toISOString().slice(0, 10); const stem = question .toLowerCase() @@ -343,7 +454,8 @@ function buildIdeaSlug(question: string, label: 'brainstorm' | 'lsd'): string { .replace(/^-+|-+$/g, '') .slice(0, 60) .replace(/^-+|-+$/g, ''); - return `wiki/ideas/${date}-${label}-${stem || 'untitled'}`; + const suffix = nonce ?? randomBytes(3).toString('hex'); + return `wiki/ideas/${date}-${label}-${stem || 'untitled'}-${suffix}`; } /** CLI entry: `gbrain brainstorm`. */ diff --git a/src/core/brainstorm/orchestrator.ts b/src/core/brainstorm/orchestrator.ts index 2a1f8c6eb..7a4b2f5dc 100644 --- a/src/core/brainstorm/orchestrator.ts +++ b/src/core/brainstorm/orchestrator.ts @@ -1045,3 +1045,32 @@ cost_usd: ${result.cost.actual_usd.toFixed(4)} `; } + +/** + * Object form of {@link buildBrainstormFrontmatter} for callers that feed the + * canonical `serializeMarkdown` serializer (which owns YAML escaping). The + * string builder above is left untouched for back-compat — its output shape is + * relied on by existing callers, so this is a SEPARATE helper, not a wrapper. + * `title` is intentionally omitted: serializeMarkdown takes it via its `meta` + * argument so it isn't duplicated in the frontmatter map. + */ +export function buildBrainstormFrontmatterObject( + result: BrainstormResult, +): Record { + const obj: Record = { + mode: result.profile_label, + generated_at: new Date().toISOString(), + date: new Date().toISOString().slice(0, 10), + question: result.question.slice(0, 200), + close_slugs: result.close_set.map((c) => c.slug), + far_slugs: result.far_set.map((f) => f.slug), + short_of_target: result.short_of_target, + calibration_cold_start: result.active_bias_tags === null, + cost_usd: Number(result.cost.actual_usd.toFixed(4)), + }; + if (result.judge_failed) { + obj.judge_failed = true; + obj.unscored = true; + } + return obj; +} diff --git a/src/core/operations.ts b/src/core/operations.ts index d58fe1084..597af390a 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -10,9 +10,7 @@ import { clampSearchLimit } from './engine.ts'; import type { GBrainConfig } from './config.ts'; import type { PageType } from './types.ts'; import { importFromContent } from './import-file.ts'; -import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts'; -import { mkdirSync, writeFileSync, existsSync, statSync } from 'fs'; -import { dirname } from 'path'; +import { writePageThrough } from './write-through.ts'; import { hybridSearch, hybridSearchCached } from './search/hybrid.ts'; import { expandQuery } from './search/expansion.ts'; import { dedupResults } from './search/dedup.ts'; @@ -715,38 +713,20 @@ const put_page: Operation = { const isSandboxSubagent = ctx.viaSubagent === true && !(Array.isArray(ctx.allowedSlugPrefixes) && ctx.allowedSlugPrefixes.length > 0); if (!ctx.dryRun && result.status !== 'error' && !isSandboxSubagent) { - try { - const repoPath = await ctx.engine.getConfig('sync.repo_path'); - if (!repoPath) { - writeThrough = { written: false, skipped: 'no_repo_configured' }; - } else if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) { - writeThrough = { written: false, skipped: 'repo_not_found' }; - } else { - const sourceId = ctx.sourceId ?? 'default'; - const writtenPage = await ctx.engine.getPage(result.slug, { sourceId }); - if (writtenPage) { - const tags = await ctx.engine.getTags(result.slug, { sourceId }); - const provenanceVia = ctx.remote === false ? 'put_page' : 'mcp:put_page'; - const md = serializePageToMarkdown(writtenPage, tags, { - frontmatterOverrides: { - ingested_via: provenanceVia, - ingested_at: new Date().toISOString(), - source_kind: provenanceVia, - }, - }); - const filePath = resolvePageFilePath(repoPath as string, result.slug, sourceId); - mkdirSync(dirname(filePath), { recursive: true }); - writeFileSync(filePath, md, 'utf8'); - writeThrough = { written: true, path: filePath }; - } else { - writeThrough = { written: false, skipped: 'page_not_found_after_write' }; - } - } - } catch (e) { - const msg = e instanceof Error ? e.message : String(e); - ctx.logger.warn(`[put_page] write-through failed for ${result.slug}: ${msg}`); - writeThrough = { written: false, error: msg }; - } + const sourceId = ctx.sourceId ?? 'default'; + const provenanceVia = ctx.remote === false ? 'put_page' : 'mcp:put_page'; + // Shared canonical write-through (also used by `gbrain brainstorm/lsd + // --save`). Renders the file from the saved DB row and writes it + // atomically; never throws (failures land in skipped/error). + writeThrough = await writePageThrough(ctx.engine, result.slug, { + sourceId, + frontmatterOverrides: { + ingested_via: provenanceVia, + ingested_at: new Date().toISOString(), + source_kind: provenanceVia, + }, + logger: ctx.logger, + }); } else if (isSandboxSubagent) { writeThrough = { written: false, skipped: 'subagent_sandbox' }; } else if (ctx.dryRun) { diff --git a/src/core/write-through.ts b/src/core/write-through.ts new file mode 100644 index 000000000..b21d2cb24 --- /dev/null +++ b/src/core/write-through.ts @@ -0,0 +1,114 @@ +/** + * Shared disk write-through for the canonical ingestion path. + * + * After a page row lands in the DB (via importFromContent / putPage), this + * renders the row to markdown via `serializePageToMarkdown` and writes it to + * `sync.repo_path` so the brain repo has a committable `.md` artifact that + * round-trips cleanly through `gbrain sync`. The file is rendered FROM the DB + * row, so the two sinks cannot diverge. + * + * Extracted from the v0.38 `put_page` write-through (operations.ts) so the + * `put_page` op AND `gbrain brainstorm/lsd --save` share one implementation + * instead of hand-rolling parallel (and divergent) copies. The extraction also + * upgraded the write to be ATOMIC — the original used a bare `writeFileSync` + * into a live git tree that `gbrain sync` / autopilot actively walk, so a crash + * mid-write left a partial `.md` that sync would fail to parse. We now write to + * a unique temp sibling and `rename` into place (rename is atomic on the same + * filesystem), matching the `.tmp + rename` convention used by + * import-checkpoint.ts / op-checkpoint.ts. + * + * Trust gating (subagent sandbox, dry-run) stays at the CALLER — this helper + * only does "row exists + repo is a real dir → render + atomic write". + */ + +import { existsSync, statSync, mkdirSync, writeFileSync, renameSync, unlinkSync } from 'fs'; +import { dirname } from 'path'; +import { randomBytes } from 'crypto'; +import type { BrainEngine } from './engine.ts'; +import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts'; + +/** Minimal logger surface — structurally compatible with operations.ts `Logger`. */ +export interface WriteThroughLogger { + warn(msg: string): void; +} + +export interface WriteThroughResult { + written: boolean; + path?: string; + /** + * Non-error reasons the file was not written: + * - no_repo_configured: `sync.repo_path` is unset (DB-only by design). + * - repo_not_found: `sync.repo_path` set but missing / not a directory. + * - page_not_found_after_write: the DB row isn't readable back (the caller's + * DB write failed or targeted a different source). + */ + skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write'; + /** Set when the render/write/rename itself threw (EACCES, ENOTDIR, disk full). */ + error?: string; +} + +export interface WritePageThroughOpts { + sourceId?: string; + /** Merged over the page's own frontmatter at render time (e.g. provenance). */ + frontmatterOverrides?: Record; + logger?: WriteThroughLogger; +} + +/** + * Render the DB row for `slug` to markdown and atomically write it under + * `sync.repo_path`. Never throws — failures are reported via the result's + * `skipped` / `error` fields (the DB write is the durable sink; the file is + * best-effort and reconciled by the next `gbrain sync`). + */ +export async function writePageThrough( + engine: BrainEngine, + slug: string, + opts: WritePageThroughOpts = {}, +): Promise { + const sourceId = opts.sourceId ?? 'default'; + try { + const repoPath = await engine.getConfig('sync.repo_path'); + if (!repoPath) { + return { written: false, skipped: 'no_repo_configured' }; + } + if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) { + return { written: false, skipped: 'repo_not_found' }; + } + + const writtenPage = await engine.getPage(slug, { sourceId }); + if (!writtenPage) { + return { written: false, skipped: 'page_not_found_after_write' }; + } + + const tags = await engine.getTags(slug, { sourceId }); + const md = serializePageToMarkdown(writtenPage, tags, { + frontmatterOverrides: opts.frontmatterOverrides, + }); + + const filePath = resolvePageFilePath(repoPath, slug, sourceId); + mkdirSync(dirname(filePath), { recursive: true }); + + // Atomic write: unique temp sibling + rename. Unique name (pid + random) + // so two concurrent saves to the same target can't clobber each other's + // temp file. Clean up the temp on any failure so we never leak a stray + // `.tmp` next to the real file. + const tmpPath = `${filePath}.tmp.${process.pid}.${randomBytes(4).toString('hex')}`; + try { + writeFileSync(tmpPath, md, 'utf8'); + renameSync(tmpPath, filePath); + } catch (writeErr) { + try { + if (existsSync(tmpPath)) unlinkSync(tmpPath); + } catch { + // best-effort cleanup; surface the original write error below + } + throw writeErr; + } + + return { written: true, path: filePath }; + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + opts.logger?.warn(`[write-through] failed for ${slug}: ${msg}`); + return { written: false, error: msg }; + } +} diff --git a/test/brainstorm/save.test.ts b/test/brainstorm/save.test.ts new file mode 100644 index 000000000..224d77f2f --- /dev/null +++ b/test/brainstorm/save.test.ts @@ -0,0 +1,187 @@ +/** + * brainstorm/lsd --save persistence tests. + * + * Covers persistSavedIdea (canonical importFromContent + shared write-through), + * formatSaveOutcome (honest message + exit code), and buildIdeaSlug + * (collision-resistant nonce). Includes the two regression cases for the + * silent-false-success bug class: DB import throws → exit 1 "NOT persisted"; + * file write fails but DB row landed → exit 0 (durable, sync reconciles). + */ + +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { resetGateway } from '../../src/core/ai/gateway.ts'; +import type { BrainEngine } from '../../src/core/engine.ts'; +import { + persistSavedIdea, + formatSaveOutcome, + buildIdeaSlug, + type SaveOutcome, +} from '../../src/commands/brainstorm.ts'; +import { serializeMarkdown } from '../../src/core/markdown.ts'; + +let engine: PGLiteEngine; +let tmpRoot: string; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); + resetGateway(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + resetGateway(); + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-bs-save-')); + brainDir = path.join(tmpRoot, 'brain'); + fs.mkdirSync(brainDir, { recursive: true }); +}); + +afterEach(() => { + fs.rmSync(tmpRoot, { recursive: true, force: true }); +}); + +const sampleContent = () => + serializeMarkdown({ mode: 'lsd', question: 'why X' }, '# LSD: why X\n\nbody', '', { + type: 'note', + title: 'LSD: why X', + tags: [], + }); + +describe('persistSavedIdea', () => { + test('both sinks: canonical DB import (chunks written) + file rendered from row', async () => { + await engine.setConfig('sync.repo_path', brainDir); + const slug = buildIdeaSlug('why X', 'lsd', 'nonce01'); + const o = await persistSavedIdea(engine, { slug, content: sampleContent(), provenanceVia: 'lsd' }); + + expect(o.dbSaved).toBe(true); + expect(o.writeThrough.written).toBe(true); + expect(fs.existsSync(o.writeThrough.path!)).toBe(true); + + // Canonical proof: importFromContent wrote chunks. Raw engine.putPage (the + // pre-fix path) would write ZERO chunks, so the page would be unsearchable + // and the next sync would churn the row. + const rows = await engine.executeRaw<{ n: number }>( + 'SELECT COUNT(*)::int AS n FROM content_chunks ch JOIN pages p ON p.id = ch.page_id WHERE p.slug = $1', + [slug], + ); + expect(Number(rows[0].n)).toBeGreaterThan(0); + }); + + test('no repo configured → DB canonical, writeThrough skipped, exit 0', async () => { + await engine.setConfig('sync.repo_path', ''); + const slug = buildIdeaSlug('why Y', 'lsd', 'nonce02'); + const o = await persistSavedIdea(engine, { slug, content: sampleContent(), provenanceVia: 'lsd' }); + + expect(o.dbSaved).toBe(true); + expect(o.writeThrough.skipped).toBe('no_repo_configured'); + const m = formatSaveOutcome(o, { profileLabel: 'lsd', slug }); + expect(m.exitCode).toBe(0); + expect(m.stdout).toContain('no `sync.repo_path`'); + }); + + test('[REGRESSION] DB import throws → nothing persisted → exit 1 NOT persisted', async () => { + // A dead/garbage engine makes importFromContent throw immediately — the + // shape of the original PgBouncer "DB write failed but said saved" bug. + const deadEngine = {} as unknown as BrainEngine; + const slug = buildIdeaSlug('why Z', 'lsd', 'nonce03'); + const o = await persistSavedIdea(deadEngine, { slug, content: sampleContent(), provenanceVia: 'lsd' }); + + expect(o.dbSaved).toBe(false); + expect(typeof o.dbError).toBe('string'); + expect(o.writeThrough.written).toBe(false); + + const m = formatSaveOutcome(o, { profileLabel: 'lsd', slug }); + expect(m.exitCode).toBe(1); + expect(m.stdout).toBeUndefined(); + expect(m.stderr.some((l) => l.includes('NOT persisted'))).toBe(true); + }); + + test('[REGRESSION] repo set but file write fails (ENOTDIR) → DB saved, exit 0, warns', async () => { + await engine.setConfig('sync.repo_path', brainDir); + fs.writeFileSync(path.join(brainDir, 'wiki'), 'blocker'); // blocks wiki/ideas/ + const slug = buildIdeaSlug('why W', 'lsd', 'nonce04'); // wiki/ideas/...-nonce04 + const o = await persistSavedIdea(engine, { slug, content: sampleContent(), provenanceVia: 'lsd' }); + + expect(o.dbSaved).toBe(true); + expect(o.writeThrough.written).toBe(false); + expect(typeof o.writeThrough.error).toBe('string'); + + const m = formatSaveOutcome(o, { profileLabel: 'lsd', slug }); + expect(m.exitCode).toBe(0); // row is durable; sync reconciles the file + expect(m.stdout).toContain('file NOT written'); + expect(m.stderr.some((l) => l.includes('file write failed'))).toBe(true); + }); +}); + +describe('formatSaveOutcome (pure)', () => { + const ctx = { profileLabel: 'lsd', slug: 'wiki/ideas/x' }; + + test('both written → exit 0, names both sinks', () => { + const o: SaveOutcome = { dbSaved: true, writeThrough: { written: true, path: '/r/x.md' } }; + const m = formatSaveOutcome(o, ctx); + expect(m.exitCode).toBe(0); + expect(m.stdout).toContain('and file'); + expect(m.stderr).toEqual([]); + }); + + test('db only, repo_not_found → exit 0', () => { + const o: SaveOutcome = { dbSaved: true, writeThrough: { written: false, skipped: 'repo_not_found' } }; + const m = formatSaveOutcome(o, ctx); + expect(m.exitCode).toBe(0); + expect(m.stdout).toContain('not a directory'); + }); + + test('db saved but file errored → exit 0, warns on stderr', () => { + const o: SaveOutcome = { dbSaved: true, writeThrough: { written: false, error: 'EACCES' } }; + const m = formatSaveOutcome(o, ctx); + expect(m.exitCode).toBe(0); + expect(m.stdout).toContain('file NOT written'); + expect(m.stderr.some((l) => l.includes('EACCES'))).toBe(true); + }); + + test('nothing persisted → exit 1, both error lines on stderr', () => { + const o: SaveOutcome = { + dbSaved: false, + dbError: 'boom', + writeThrough: { written: false, skipped: 'page_not_found_after_write' }, + }; + const m = formatSaveOutcome(o, ctx); + expect(m.exitCode).toBe(1); + expect(m.stdout).toBeUndefined(); + expect(m.stderr.some((l) => l.includes('DB save failed: boom'))).toBe(true); + expect(m.stderr.some((l) => l.includes('NOT persisted'))).toBe(true); + }); +}); + +describe('buildIdeaSlug', () => { + test('distinct nonces → distinct slugs (no same-day clobber)', () => { + const a = buildIdeaSlug('same question', 'lsd', 'aaa'); + const b = buildIdeaSlug('same question', 'lsd', 'bbb'); + expect(a).not.toBe(b); + expect(a.startsWith('wiki/ideas/')).toBe(true); + }); + + test('empty question → untitled stem, still unique by nonce', () => { + const a = buildIdeaSlug('', 'lsd', 'aaa'); + const b = buildIdeaSlug('', 'lsd', 'bbb'); + expect(a).toContain('-untitled-'); + expect(a).not.toBe(b); + }); + + test('production nonce (no arg) is random → two calls differ', () => { + const a = buildIdeaSlug('q', 'brainstorm'); + const b = buildIdeaSlug('q', 'brainstorm'); + expect(a).not.toBe(b); + }); +}); diff --git a/test/write-through.test.ts b/test/write-through.test.ts new file mode 100644 index 000000000..67ed55117 --- /dev/null +++ b/test/write-through.test.ts @@ -0,0 +1,139 @@ +/** + * Shared write-through helper tests (src/core/write-through.ts). + * + * Covers the skip/error branches and the atomic-write guarantee. The helper is + * the canonical disk sink shared by `put_page` and `gbrain brainstorm/lsd + * --save`, extracted from the v0.38 put_page write-through and upgraded to write + * atomically (.tmp + rename). + */ + +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { resetGateway } from '../src/core/ai/gateway.ts'; +import { writePageThrough } from '../src/core/write-through.ts'; +import { importFromContent } from '../src/core/import-file.ts'; +import { serializePageToMarkdown, resolvePageFilePath } from '../src/core/markdown.ts'; + +let engine: PGLiteEngine; +let tmpRoot: string; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); + resetGateway(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + resetGateway(); + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-wt-helper-')); + brainDir = path.join(tmpRoot, 'brain'); + fs.mkdirSync(brainDir, { recursive: true }); +}); + +afterEach(() => { + fs.rmSync(tmpRoot, { recursive: true, force: true }); +}); + +async function seedPage(slug: string): Promise { + await importFromContent(engine, slug, `---\ntitle: T\ntype: note\n---\n\n# Body ${slug}\n`, { + noEmbed: true, + sourceId: 'default', + sourcePath: `${slug}.md`, + }); +} + +function walkFiles(dir: string): string[] { + const out: string[] = []; + const walk = (d: string) => { + for (const e of fs.readdirSync(d, { withFileTypes: true })) { + const p = path.join(d, e.name); + if (e.isDirectory()) walk(p); + else out.push(p); + } + }; + walk(dir); + return out; +} + +describe('writePageThrough', () => { + test('writes the file rendered from the saved row; no .tmp leftover', async () => { + await engine.setConfig('sync.repo_path', brainDir); + const slug = 'wiki/ideas/2026-01-01-lsd-foo-abc123'; + await seedPage(slug); + + const res = await writePageThrough(engine, slug, { + sourceId: 'default', + frontmatterOverrides: { source_kind: 'lsd' }, + }); + + expect(res.written).toBe(true); + const expectedPath = resolvePageFilePath(brainDir, slug, 'default'); + expect(res.path).toBe(expectedPath); + expect(fs.existsSync(expectedPath)).toBe(true); + + // Content is the canonical serialization of the saved row (the file is + // rendered FROM the row, so the sinks can't diverge). + const page = await engine.getPage(slug, { sourceId: 'default' }); + const tags = await engine.getTags(slug, { sourceId: 'default' }); + const expected = serializePageToMarkdown(page!, tags, { + frontmatterOverrides: { source_kind: 'lsd' }, + }); + expect(fs.readFileSync(expectedPath, 'utf8')).toBe(expected); + + // Atomic write left no temp sibling. + const dir = path.dirname(expectedPath); + expect(fs.readdirSync(dir).some((f) => f.includes('.tmp.'))).toBe(false); + }); + + test('no sync.repo_path → skipped no_repo_configured', async () => { + await engine.setConfig('sync.repo_path', ''); + const slug = 'wiki/ideas/x-1'; + await seedPage(slug); + const res = await writePageThrough(engine, slug); + expect(res).toEqual({ written: false, skipped: 'no_repo_configured' }); + }); + + test('sync.repo_path is a file, not a directory → skipped repo_not_found', async () => { + const fileAsRepo = path.join(tmpRoot, 'not-a-dir'); + fs.writeFileSync(fileAsRepo, 'x'); + await engine.setConfig('sync.repo_path', fileAsRepo); + const slug = 'wiki/ideas/x-2'; + await seedPage(slug); + const res = await writePageThrough(engine, slug); + expect(res).toEqual({ written: false, skipped: 'repo_not_found' }); + }); + + test('row missing → skipped page_not_found_after_write', async () => { + await engine.setConfig('sync.repo_path', brainDir); + const res = await writePageThrough(engine, 'wiki/ideas/does-not-exist'); + expect(res).toEqual({ written: false, skipped: 'page_not_found_after_write' }); + }); + + test('[REGRESSION] mkdir ENOTDIR (parent is a file) → error, no partial .md, no .tmp', async () => { + await engine.setConfig('sync.repo_path', brainDir); + // Block the `wiki/` directory by putting a FILE named "wiki" under the repo, + // so `mkdir -p /wiki/ideas` throws ENOTDIR deterministically. + fs.writeFileSync(path.join(brainDir, 'wiki'), 'blocker'); + const slug = 'wiki/ideas/blocked-1'; + await seedPage(slug); + + const res = await writePageThrough(engine, slug, { sourceId: 'default' }); + + expect(res.written).toBe(false); + expect(typeof res.error).toBe('string'); + const files = walkFiles(brainDir); + expect(files.some((f) => f.endsWith('.md'))).toBe(false); + expect(files.some((f) => f.includes('.tmp.'))).toBe(false); + }); +});