diff --git a/CHANGELOG.md b/CHANGELOG.md index dab1799ce..83d62dc89 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,42 @@ All notable changes to GBrain will be documented in this file. +## [0.42.60.0] - 2026-07-16 + +**Eleven verified community fixes: Windows brains no longer risk losing subdirectory pages on a full sync, agent tool loops on non-Anthropic providers survive interruption instead of dead-lettering, multi-source brains get two source-isolation gaps closed, and the search cache stops leaking results across exclude policies. Every fix was reproduced and reviewed against master before landing.** + +### Fixed +- **Windows: `sync --full` no longer deletes subdirectory pages.** A path-separator mismatch made every subdirectory page look stale during full-sync reconcile, so a routine full sync could delete them. Paths are now normalized before comparison, and a mass-delete safety valve blocks any reconcile that would remove most of a source's pages. (#2828, #2836, contributed by @1alessio) +- **Gateway tool loops on non-Anthropic providers are reliable across resume.** Tool-result turns are persisted as they happen, interrupted jobs reconcile dangling tool calls on resume instead of dead-lettering with unbalanced-transcript errors, `Date` values in tool outputs no longer crash serialization, DeepSeek reasoning-only replies are read correctly instead of as empty, and `openrouter_api_key` in config reaches the gateway. (#2820, consolidating community fixes #2062, #2065, #2257, #2274, #2336, #2487, #2491, #2572, #2614, #2617, #2806; contributed by @time-attack and the original PR authors) +- **Claude 5 models get output-token headroom.** Thinking-default models no longer have long answers silently truncated by the old 4096-token default output cap; Claude 5 chat calls now default to 32000 output tokens (16000 for `think`). Other providers keep their existing caps, so smaller-limit providers are unaffected. (#2820) +- **Bulk import survives huge fence-less files.** The markdown lexer is skipped when a page contains no code fences, removing an out-of-memory crash on large tables and notes during bulk import. (#2437, #2440, contributed by @irresi) +- **`file_list` no longer crashes on Postgres brains over MCP.** BIGINT file sizes are normalized before JSON serialization; the CLI files listing gets the same fix. (#472, contributed by @vinsew) +- **`gbrain config set auto_chronicle true` works as documented.** The Life Chronicle config keys (and `takes.bootstrap_enabled`) are registered, so the documented enable commands stop being rejected as unknown keys. (#2632, contributed by @p3ob7o) +- **Orphan reports skip generated corpus roots.** `raw/`, `atoms/`, and `skills/` no longer inflate the orphan ratio by default; `--include-pseudo` still shows everything. (#2068, contributed by @mgunnin) + +### Security +- **The search cache honors your hard-exclude policy.** Cached search results are now keyed on the effective hard-exclude/include slug-prefix policy, so a process with `GBRAIN_SEARCH_EXCLUDE` set can never be served cached rows written under a different policy — and vice versa. (#2825, #2885) +- **Take-writes are source-scoped.** When a source resolves (via `--source`, `GBRAIN_SOURCE`, or the dotfile chain), CLI take commands look pages up within that source instead of first-match-by-slug, closing a cross-source write path on brains where the same slug exists in multiple sources. Brains without a resolvable source keep the previous lookup. (#2684, #2698, contributed by @RerankerGuo) +- **Image pages land in the right source.** Imported images are stamped with the syncing source (and their auto-links stay within it) instead of always landing in `default`. (#2706, #2718, contributed by @RerankerGuo) +- **The admin bootstrap token no longer prints to a non-terminal stream.** The one-time token is withheld when its output stream is a pipe, log, or CI capture instead of an interactive terminal. (#2625, contributed by @irresi) + +### Internal +- Pinned embedding dimensions in a doctor test to eliminate a shard-order flake in CI. (#2801, contributed by @p3ob7o) + +### To take advantage of v0.42.60.0 + +`gbrain upgrade`. No new schema migrations. + +1. **Windows users with git-synced sources:** re-run `gbrain sync --full` once after upgrading — if a pre-upgrade sync deleted subdirectory pages, they re-import from the repo. +2. **Your first search after upgrading may be a cache miss** (the cache key now includes the exclude policy). Speeds return to normal as the cache refills within its TTL. +3. **If agent jobs previously dead-lettered** with unbalanced tool-call transcript errors on OpenAI-compatible providers, retry them with `gbrain jobs retry ` — resume now reconciles the transcript. +4. **Verify:** + ```bash + gbrain doctor + gbrain stats + ``` +5. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`. + ## [0.42.59.0] - 2026-07-13 **Five community-reported fixes, each reproduced and verified before/after on both engines (PGLite + real Postgres): an upgrade wedge that locked pre-v121 brains out of migrations, two data-integrity holes in engine migration, silent deletion of facts containing pipe characters, confidently-wrong entity attribution on ambiguous names, and tightened source-scope enforcement in `think`.** diff --git a/TODOS.md b/TODOS.md index a3dd7a9d1..238599e33 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,19 @@ # TODOS +## community fix-wave follow-ups (filed v0.42.60.0) + +- [ ] **P1 — take-writes source scoping fails open when source resolution errors (#2684 residual).** + `resolveTakesSourceId` (src/commands/takes.ts) swallows resolution errors and returns + `undefined`, which falls back to the unscoped slug-only page lookup — so an invalid + `GBRAIN_SOURCE` (or a broken dotfile chain) silently restores the pre-#2698 cross-source + write behavior on multi-source brains. Decide fail-closed semantics: error out when a + source was explicitly requested but doesn't resolve; keep the unscoped fallback only for + brains with no source configuration at all. Add a regression test for the invalid-source + path. Found by cross-model adversarial review during the v0.42.60.0 release ship. +- [ ] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded + most of #2112 but not its `checkSubagentCapability` fix (check explicit `models.subagent` + before `models.tier.subagent`). Refile or cherry-pick; the rest of that PR is covered. + ## v0.42.59.0 follow-ups (five-fix rollup #2735–#2739) Filed as follow-ups from v0.42.59.0 (bootstrap probe for @@ -219,10 +233,7 @@ job) and sync. See CLAUDE.md "Pace Mode". `sourceScopeOpts`) and `code_def` (`operations.ts:4155` — brain-wide raw SQL over `content_chunks`; confirm whether brain-wide is intentional before scoping). A remote federated client (grant set, dispatch-default `ctx.sourceId='default'`) reads these against - `default` or unscoped, not its grant. *(Partially done by v0.42.59.0: the engine methods - `searchTakes`/`searchTakesVector` now accept the source-scope predicates and the `think` - gather path threads them; the standalone `takes_search` op handler still doesn't route - through `sourceScopeOpts(ctx)`.)* + `default` or unscoped, not its grant. - **Why:** same cross-source correctness/isolation class #2200 targets; a federated client can't read chunks/raw-data/versions for an authorized non-default source, `resolve_slugs` can fuzzy-resolve across all sources, and `takes_search`/`code_def` query without the grant. @@ -964,7 +975,7 @@ default-off; these are the gates and extensions before any default flip. - [ ] **v0.42+: cross-surface ablation before flipping `search.adaptive_return` default.** The gate ships default-off. Before turning it on in any `MODE_BUNDLES` tier, run the recall ablation (adaptive off vs on, recall-preserving caps) across `gbrain eval longmemeval`, `gbrain eval whoknows`, `gbrain eval suspected-contradictions`, and the BrainBench-Real replay (sibling gbrain-evals repo). Confirm recall@k / answer quality does not regress; pick the safe caps; probably flip `tokenmax` first (broadest searchLimit, most noise). On-surface evidence (the PrecisionMemBench precision/recall frontier: off 0.076/0.99, e1/o2 0.40/0.91, e1/o1 0.58/0.82) is recorded in `gbrain-evals/docs/benchmarks/2026-05-29-precisionmembench.md`. Priority: P2. - [ ] **v0.42+: fold adaptive-return params into KNOBS_HASH so adaptive-on calls can cache.** v0.41.33.0 skips `hybridSearchCached` entirely when the gate is on (cache-safe but cache-cold). Fold `adaptive_return` enabled + caps + `minKeep` into `knobsHash()` (append-only, bump `KNOBS_HASH_VERSION`) so a gate-on write segregates from a gate-off row and adaptive calls cache correctly. Required before any default flip (else default-on means cache-cold everywhere). See `src/core/search/mode.ts` KNOBS_HASH parts + `return-policy.ts`. Priority: P2 (paired with the default-flip ablation above). -- [ ] **v0.42+: gentle adaptive gate on `think`'s gather stage (A3).** The plan's A3 decision was a gentler return-gate on `runThink`'s gather candidates (cleaner context, fewer tokens per reasoning call). Deferred because the benefit is unvalidated without a longmemeval answer-quality run, and trimming the answer path (even default-off) carries regression risk. gather fuses 4 streams (page / takes-keyword / takes-vector / graph); the gate must operate on the fused output with a higher min-keep than search, validated on `gbrain eval longmemeval` answer quality (not retrieval precision). *(The scope plumbing this was gated on landed in v0.42.59.0: `RunThinkOpts` carries `sourceId`/`allowedSources` and `runGather` threads the scope through every stream, so the gate work no longer blocks on it.)* Priority: P2. +- [ ] **v0.42+: gentle adaptive gate on `think`'s gather stage (A3).** The plan's A3 decision was a gentler return-gate on `runThink`'s gather candidates (cleaner context, fewer tokens per reasoning call). Deferred because the benefit is unvalidated without a longmemeval answer-quality run, and trimming the answer path (even default-off) carries regression risk. gather fuses 4 streams (page / takes-keyword / takes-vector / graph); the gate must operate on the fused output with a higher min-keep than search, validated on `gbrain eval longmemeval` answer quality (not retrieval precision). Also: `RunThinkOpts` has no `sourceId` today, so think's gather runs unscoped (codex finding) — scope-isolated think needs that plumbing first. Priority: P2. - [ ] **v0.42+: `--explain` human header for adaptive_return.** The decision is in `HybridSearchMeta.adaptive_return` and surfaces in `--json` today. The per-result `explain-formatter.ts` is result-scoped and can't render a per-query meta line; the human `gbrain search --explain` header needs the meta threaded through `cli.ts:formatResult` (it currently only receives `results`). Add a one-line gate-decision header (intent / cap / kept of total). Priority: P3. - [ ] **v0.42+: structured-alias / facts-mode fidelity for the PrecisionMemBench eval.** The gbrain-evals benchmark seeds beliefs as pages with aliases in the body (real FTS). A second fidelity that exercises gbrain's structured alias/entity-resolution layer (facts with `valid_until` + entity resolution) would measure gbrain's structured-belief path on the 23 alias cases. Lives in gbrain-evals (`eval/precisionmembench/seed.ts` throws on `fidelity:'structured'` today). Priority: P3. diff --git a/VERSION b/VERSION index 228faf484..daa98aa36 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.59.0 \ No newline at end of file +0.42.60.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index c11619774..5484e8c21 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -116,7 +116,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/search/embedding-column.ts` — single source of truth for "which `content_chunks.*` column does this query rank against?" Pure functions, no engine I/O: `loadRegistry(cfg)` walks the `embedding_columns` config (DB plane, JSON map keyed by column name with `{provider, dimensions, type}` entries), seeds the OpenAI `embedding` builtin when unset, validates everything before it lands (column-name regex, type ∈ `vector | halfvec`, dims in [1, 8192], provider format) using `Object.create(null)` + `Object.hasOwn` so a key like `constructor` rejects instead of resolving to `Object.prototype.constructor`. `resolveColumn(registry, override?, cfg)` is the boundary call: returns a frozen `ResolvedColumn` descriptor (`{name, provider, dimensions, type}`) honoring per-call override → `search_embedding_column` config → `'embedding'` default; throws `UnknownEmbeddingColumnError` with the list of registered names on miss. `isCacheSafe(resolved, cfg)` compares the full embedding SPACE (provider + dimensions + name) against cfg's default so a repointed `embedding` builtin doesn't serve OpenAI-shaped cache rows. `validateResolvedColumn(descriptor)` re-validates hand-rolled descriptors that bypass the registry (internal-SDK passthrough) so the SQL-injection escape hatch through the descriptor field is closed. Consumed by `hybridSearch`, `gateway.embedQuery(text, {embeddingModel, dimensions})`, `cosineReScore`, and the `query` MCP op (per-call `embedding_column` param). Pinned by `test/search/embedding-column.test.ts` (prototype-pollution, descriptor passthrough, env-only Postgres install, empty-brain coverage gate, cache-space comparison). - `src/core/search/rerank.ts` — the call-site abstraction. `applyReranker(query, results, opts)` slots between `dedupResults()` and `enforceTokenBudget()` in `src/core/search/hybrid.ts`. Slices `opts.topNIn` (default 30) by current RRF order, sends to `gateway.rerank()`, reorders by `relevanceScore` desc, appends the un-reranked tail unchanged (recall protection). Fail-open on every `RerankError.reason`: any error logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items so downstream telemetry sees the new ordering signal. `topNOut: null` is the explicit "don't truncate" signal — semantically distinct from `undefined` ("fall through to mode bundle"). Test seam: `opts.rerankerFn` stubs `gateway.rerank` without the network. - `src/core/search/return-policy.ts` (default OFF) — intent-aware adaptive return-sizing. Pure, dependency-light module that trims the final ranked candidate set to an intent-driven cap instead of returning the full top-K. `entity` intent gets a tight cap; `temporal`/`event`/`general` get a recall-preserving cap. A `minKeep` failsafe (≥1) guarantees a human never gets a silent blank when candidates exist. WHY a cap, not a score-cliff detector: PrecisionMemBench instrumentation (gbrain-evals) measured the rank1→rank2 RRF gap is ~identical whether rank-1 is correct (0.602) or wrong (0.569) — mechanical decay, not a separatrix; rank-1 is right in 94% of single-answer cases, so "return a tight set" is the whole win and cliff-cutting just adds noise. Exports `AdaptiveReturnConfig`, `DEFAULT_ADAPTIVE_RETURN` (frozen: `enabled=false`, `entityMax=2`, `otherMax=6`, `minKeep=1`), `AdaptiveReturnDecision` (`{applied, intent, cap, kept, total}`), `AdaptiveReturnInput` (`boolean | Partial | undefined`), `adaptiveReturnFromConfig(cfg)`, `resolveAdaptiveReturn(perCall, fromConfig)` (defaults → config → per-call merge), `adaptiveReturnEnabled(...)` (cache-skip gate check), `applyAdaptiveReturn(results, intent, cfg)` (the trim). Config knobs (DB or file plane): `search.adaptive_return` (master switch), `search.adaptive_return_entity_max`, `search.adaptive_return_other_max`, `search.adaptive_return_min_keep` (each clamped ≥1). Wired into `hybridSearch` AFTER `applyReranker`, BEFORE the `limit` slice, and ONLY on the first page (`offset===0`) — paginating a confidence-gated set is incoherent, so paginated calls fall through to the fixed limit. Stamps the decision onto `HybridSearchMeta.adaptive_return` for `gbrain search --explain`. `hybridSearchCached` SKIPS the cache when the gate is on (a trimmed set must not be served to a gate-off lookup and vice versa). `SearchOpts.adaptiveReturn` + `HybridSearchMeta.adaptive_return` declared in `src/core/types.ts`. Agent-facing: the `query` op (`src/core/operations.ts`) exposes an `adaptive_return` boolean param whose description instructs the agent WHEN to set it (single-answer → on; breadth/exploration → off; pass `limit:1` for a hard single-answer cap), threaded into `hybridSearchCached` — end users never touch the config knob; their agent decides per query (same pattern as `salience`/`recency`). Pinned by `test/search/return-policy.test.ts` (mechanism) + `test/search/query-op-adaptive-return.test.ts` (agent surface: param exists + description teaches both directions + the never-empty contract). -- `src/core/search/autocut.ts` (default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). WHY rerank_score and NOT RRF/cosine: gbrain measured (see `return-policy.ts`) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. Cache-key integration (clean path, not the adaptive-return cache-skip hack): enable+sensitivity flow through `ModeBundle` → `ResolvedSearchKnobs` → `knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump` (conservative false, balanced/tokenmax true@0.20) AND sets `reranker_top_n_in = searchLimit` for reranked modes (so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the load-bearing recall finding). `KNOBS_HASH_VERSION` is 8 (title_boost claimed 7; autocut appends 8 — one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column`. Preserves alias-hop exact matches: `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score`, `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`. Default-ON backed by an in-repo eval gate — `test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries. Env-overridable floors. Pinned by `test/search/autocut.test.ts` (pure-fn), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the v=8 knobsHash assertions in `test/search-mode.test.ts`. +- `src/core/search/autocut.ts` (default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). WHY rerank_score and NOT RRF/cosine: gbrain measured (see `return-policy.ts`) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. Cache-key integration (clean path, not the adaptive-return cache-skip hack): enable+sensitivity flow through `ModeBundle` → `ResolvedSearchKnobs` → `knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump` (conservative false, balanced/tokenmax true@0.20) AND sets `reranker_top_n_in = searchLimit` for reranked modes (so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the load-bearing recall finding). Autocut folds into `knobsHash` as its own parts entry (`mode.ts:KNOBS_HASH_VERSION` is the single source of truth for the current hash version; every bump is a one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column`. Preserves alias-hop exact matches: `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score`, `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`. Default-ON backed by an in-repo eval gate — `test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries. Env-overridable floors. Pinned by `test/search/autocut.test.ts` (pure-fn), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the knobsHash assertions in `test/search-mode.test.ts`. - `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. Declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio), avoiding the backfill loop where tiktoken-grounded budgeting undercounted Voyage's actual token usage. Declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. Recipe docstring at `:7-16` names the seven hosted flexible-dim models that accept `output_dimension` (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and notes `voyage-4-nano` is the open-weight variant fixed at 1024-dim that does NOT accept the parameter (negative regression assertion in `test/ai/gateway.test.ts`: `dimsProviderOptions` returns `undefined` for `voyage-4-nano`). `voyage-code-3` is the recommended embedding model for gstack per-worktree code brains (Topology 3 in `docs/architecture/topologies.md`); discoverability surfaces: decision-tree branch in `docs/integrations/embedding-providers.md`, Topology 3 "Recommended embedding model" subsection, runtime nudge from `gbrain reindex --code` against non-code-tuned models. Recipe-shape regression pinned by `test/ai/voyage-code-3-recipe.test.ts`. - `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). Canonical id is `claude-sonnet-4-6` (no date suffix); a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize`). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts`. - `src/core/model-pricing.ts` — single source of truth for paid-cloud CHAT/completion model pricing (USD per 1M tokens, input | output). `CANONICAL_PRICING` is a `provider:model`-keyed table (Anthropic Opus 4.8/4.7/4.6 `$5/$25`, Sonnet 4.6 `$3/$15`, Haiku 4.5 `$1/$5` both dateless + dated, plus OpenAI / Google / Together / DeepSeek panel models). `canonicalLookup(modelId)` resolves bare (`claude-opus-4-8`), colon (`anthropic:claude-opus-4-8`), and slash (`anthropic/...`) forms — bare ids default to the `anthropic:` provider; nested OpenRouter ids (`openrouter:anthropic/...`) intentionally MISS so OpenRouter markup isn't repriced as the inner vendor. Every other chat-pricing table is a DERIVED view of this one (NOT a hand-copied duplicate), so cross-table price drift is structurally impossible. Embeddings live separately in `embedding-pricing.ts` (different unit). Pinned by `test/model-pricing.test.ts` whose drift guard asserts each derived view equals canonical and that the cross-modal panel models are all present. @@ -265,7 +265,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/mcp/server.ts` — MCP stdio server (generated from operations). Tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. Stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'` — gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`) pipe the handshake then close their stdin half, which would otherwise kill the server before the first tool call; signal handlers (SIGTERM/SIGINT/SIGHUP) + the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`. - `src/mcp/dispatch.ts` — shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP transports. Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults `remote: true` (untrusted); local CLI callers pass `remote: false`. Also exports `summarizeMcpParams(opName, params)` — privacy-preserving redactor for `mcp_request_log` and the admin SSE feed, returns `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. Intersects submitted top-level keys against the operation's declared `params` allow-list (declared keys preserved sorted; unknown keys counted but never named, closing the attacker-controlled-key-name leak). Byte counts bucketed up to nearest 1KB so an attacker can't binary-search secret-content sizes by probing. Raw payload visibility is opt-in via `gbrain serve --http --log-full-params` (loud stderr warning). New logging paths route through this helper, not `JSON.stringify(params)`. - `src/mcp/rate-limit.ts` — Bounded-LRU token-bucket limiter. `buildDefaultLimiters()` returns the two-bucket pipeline: pre-auth IP (30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is capped) + post-auth token-id (60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap bounds memory under attacker-controlled key growth. -- `src/commands/serve-http.ts` — Express 5 HTTP MCP server with OAuth 2.1, admin dashboard, and SSE live activity feed. Started via `gbrain serve --http [--port N] [--token-ttl N] [--enable-dcr] [--public-url URL] [--bind HOST] [--log-full-params]`. Combines MCP SDK's `mcpAuthRouter` (authorize/token/register/revoke), a custom `client_credentials` handler running BEFORE the router (SDK's token endpoint throws `UnsupportedGrantTypeError` for CC; custom handler falls through for `auth_code` / `refresh_token`), `requireBearerAuth` middleware for `/mcp` with scope enforcement + `localOnly` rejection before op dispatch, and `express-rate-limit` at 50 req / 15 min on `/token`. Serves the built admin SPA from `admin/dist/` with SPA fallback. `/admin/events` SSE broadcasts every MCP request. `cookie-parser` wired (Express 5 has no built-in). Startup logging prints port, engine, issuer URL (honors `--public-url`), client count, DCR status, admin bootstrap token. The `/mcp` request handler's OperationContext literal sets `remote: true` explicitly (without it `submit_job`'s protected-name guard at `operations.ts:1391` saw a falsy undefined and a `read+write`-scoped OAuth token could submit `shell` jobs — RCE). `summarizeMcpParams` from `src/mcp/dispatch.ts` feeds both `mcp_request_log` writes and the SSE feed by default (raw via `--log-full-params`). Cookie `Secure` flag set behind HTTPS or a public-URL proxy; magic-link nonce store LRU-bounded; DCR disable routes through the `GBrainOAuthProvider` `dcrDisabled` constructor option (not a router monkey-patch); `transport.handleRequest` wrapped in try/catch to return a JSON-RPC 500 envelope; OperationError + unexpected exceptions unified through `buildError` / `serializeError` so `/mcp` always returns the same envelope. `/health` is liveness-only via `probeLiveness(sql, engineName, version, timeoutMs)` racing `sql\`SELECT 1\`` against the exported `HEALTH_TIMEOUT_MS = 3000` (returns the same `ProbeHealthResult` tagged-union as `probeHealth`, single timer-cleanup site, single 503 envelope); body shape `{status, version, engine}` only. Full stats moved to admin-only `/admin/api/full-stats` (gated by `requireAdmin`, calls `probeHealth(engine, ...)`) — keeps `getStats()`'s 6× count(*) off the public route so a saturated pool doesn't trigger orchestrator restart cascades. Every OAuth/admin/audit SQL call routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` so it works against PGLite; the four `mcp_request_log.params` INSERT sites (success / auth_failed / scope_denied / server-error) go through `executeRawJsonb(engine, ...)` so the column stores real objects (`params->>'op'` returns `search`, not the quoted string). `--bind HOST` defaults `127.0.0.1` (self-hosters pass `--bind 0.0.0.0`); a stderr WARN fires when `--public-url` is set without `--bind`; the banner prints a `Bind:` line. `AuthInfo.sourceId` + `AuthInfo.allowedSources` are the typed source of truth, populated by `oauth-provider.ts:verifyAccessToken` from the `oauth_clients` row. The HTTP MCP `tools/list` handler at `:837-849` uses `paramDefToSchema(v)` from `src/mcp/tool-defs.ts` so array params keep `items` (strict-mode OAuth clients otherwise reject the whole tool list). +- `src/commands/serve-http.ts` — Express 5 HTTP MCP server with OAuth 2.1, admin dashboard, and SSE live activity feed. Started via `gbrain serve --http [--port N] [--token-ttl N] [--enable-dcr] [--public-url URL] [--bind HOST] [--log-full-params]`. Combines MCP SDK's `mcpAuthRouter` (authorize/token/register/revoke), a custom `client_credentials` handler running BEFORE the router (SDK's token endpoint throws `UnsupportedGrantTypeError` for CC; custom handler falls through for `auth_code` / `refresh_token`), `requireBearerAuth` middleware for `/mcp` with scope enforcement + `localOnly` rejection before op dispatch, and `express-rate-limit` at 50 req / 15 min on `/token`. Serves the built admin SPA from `admin/dist/` with SPA fallback. `/admin/events` SSE broadcasts every MCP request. `cookie-parser` wired (Express 5 has no built-in). Startup logging prints port, engine, issuer URL (honors `--public-url`), client count, DCR status, and the admin bootstrap token line — but the generated token's raw value only prints when stderr is an interactive TTY (`shouldSuppressBootstrapPrint`): a non-TTY (containerized/piped) start hides it so the secret never lands in centralized log storage, env-sourced tokens (`$GBRAIN_ADMIN_BOOTSTRAP_TOKEN`) are always hidden, `--print-admin-token` forces the raw value on a trusted terminal, and `--suppress-bootstrap-token` hides everything. The `/mcp` request handler's OperationContext literal sets `remote: true` explicitly (without it `submit_job`'s protected-name guard at `operations.ts:1391` saw a falsy undefined and a `read+write`-scoped OAuth token could submit `shell` jobs — RCE). `summarizeMcpParams` from `src/mcp/dispatch.ts` feeds both `mcp_request_log` writes and the SSE feed by default (raw via `--log-full-params`). Cookie `Secure` flag set behind HTTPS or a public-URL proxy; magic-link nonce store LRU-bounded; DCR disable routes through the `GBrainOAuthProvider` `dcrDisabled` constructor option (not a router monkey-patch); `transport.handleRequest` wrapped in try/catch to return a JSON-RPC 500 envelope; OperationError + unexpected exceptions unified through `buildError` / `serializeError` so `/mcp` always returns the same envelope. `/health` is liveness-only via `probeLiveness(sql, engineName, version, timeoutMs)` racing `sql\`SELECT 1\`` against the exported `HEALTH_TIMEOUT_MS = 3000` (returns the same `ProbeHealthResult` tagged-union as `probeHealth`, single timer-cleanup site, single 503 envelope); body shape `{status, version, engine}` only. Full stats moved to admin-only `/admin/api/full-stats` (gated by `requireAdmin`, calls `probeHealth(engine, ...)`) — keeps `getStats()`'s 6× count(*) off the public route so a saturated pool doesn't trigger orchestrator restart cascades. Every OAuth/admin/audit SQL call routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` so it works against PGLite; the four `mcp_request_log.params` INSERT sites (success / auth_failed / scope_denied / server-error) go through `executeRawJsonb(engine, ...)` so the column stores real objects (`params->>'op'` returns `search`, not the quoted string). `--bind HOST` defaults `127.0.0.1` (self-hosters pass `--bind 0.0.0.0`); a stderr WARN fires when `--public-url` is set without `--bind`; the banner prints a `Bind:` line. `AuthInfo.sourceId` + `AuthInfo.allowedSources` are the typed source of truth, populated by `oauth-provider.ts:verifyAccessToken` from the `oauth_clients` row. The HTTP MCP `tools/list` handler at `:837-849` uses `paramDefToSchema(v)` from `src/mcp/tool-defs.ts` so array params keep `items` (strict-mode OAuth clients otherwise reject the whole tool list). - `src/core/sql-query.ts` — engine-aware tagged-template SQL adapter for OAuth/admin/auth infrastructure. `sqlQueryForEngine(engine)` returns a `SqlQuery` (`(strings, ...values) => Promise`) that walks the template, builds `$N` positional SQL, asserts every value is a `SqlValue` (string | number | bigint | boolean | Date | null), and routes through `engine.executeRaw(sql, params)` (Postgres via postgres.js `unsafe(sql, params)`, PGLite via `db.query(sql, params)`). Deliberately narrower than postgres.js's `sql` tag: no nested fragments, `sql.json()`, `sql.unsafe()`, `sql.begin()`, or array binding — the narrow scalar-only surface is the feature (keeps it from drifting into a partial postgres.js clone). JSONB writes go through `executeRawJsonb(engine, sql, scalarParams, jsonbParams)` which composes positional `$N::jsonb` casts and passes JS **objects** through; an object reaches the wire with the correct type oid, so executeRawJsonb is safe (verified by `test/sql-query.test.ts` on PGLite, `test/e2e/auth-permissions.test.ts:67` on Postgres). Positional binding is NOT universally immune, though: binding a `JSON.stringify(x)` **string** to a bare `$N::jsonb` via `unsafe()` double-encodes it into a jsonb string scalar on real Postgres (the #2339 class; PGLite hides it). Fixes: pass a raw object (executeRawJsonb / `sql.json`), or cast through `$N::text::jsonb`. `scripts/check-jsonb-pattern.sh` (template grep) doesn't fire on `executeRawJsonb(...)` because it passes objects; the positional `$N::jsonb` + `JSON.stringify` form is caught by `scripts/check-jsonb-params.mjs`. Consumed by `src/commands/auth.ts`, `src/commands/serve-http.ts`, `src/core/oauth-provider.ts`, `src/commands/files.ts`, `src/mcp/http-transport.ts` so all five work uniformly against PGLite and Postgres. - `src/commands/serve.ts` — `gbrain serve` stdio MCP entrypoint with idempotent shutdown across every parent-disconnect signal. Stdio EOF, SIGTERM, SIGINT, SIGHUP, and parent-process death (every reparent case — PID 1, launchd subreaper, systemd, tmux, or a parent shell with `PR_SET_CHILD_SUBREAPER`) all funnel into one `cleanup(reason)` that releases the engine and the PGLite write-lock dir within 5 seconds (otherwise the lock is held indefinitely after Claude Desktop / Cursor / launchd-managed gateways disconnect, forcing a 5-minute stale-lock wait on next start). Watchdog reparent check is `getParentPid() !== initialParentPid` (the `=== 1` check missed the subreaper case under launchd/systemd). Bun's `process.ppid` cache is stale across reparenting ([oven-sh/bun#30305](https://github.com/oven-sh/bun/issues/30305)) so `getParentPid()` runs `spawnSync('ps', ['-o', 'ppid=', '-p', PID])` per tick. Startup probe verifies `ps` is on PATH; if not (stripped containers, busybox), the watchdog skips installing AND emits a loud `[gbrain serve] watchdog disabled: ps unavailable ...` stderr line so operators see the degraded mode. Pinned by `test/serve-stdio-lifecycle.test.ts` (22 cases). Credit @Aragorn2046 + @seungsu-kr. - `src/core/oauth-provider.ts` — `GBrainOAuthProvider` implementing the MCP SDK's `OAuthServerProvider` + `OAuthRegisteredClientsStore`. Backed by raw SQL (works on both PGLite and Postgres — OAuth is infrastructure, not a BrainEngine concern). Full OAuth 2.1: `authorize` + `exchangeAuthorizationCode` with PKCE, `client_credentials`, `refresh_token` with rotation, `revokeToken`, `registerClient` (DCR validates redirect_uri is `https://` or loopback per RFC 6749 §3.1.2.1). All tokens + client secrets SHA-256 hashed before storage. Auth codes single-use with 10-minute TTL via atomic `DELETE...RETURNING` (closes RFC 6749 §10.5 TOCTOU); refresh rotation also `DELETE...RETURNING` (§10.4 stolen-token detection). `pgArray()` escapes commas/quotes/braces so a comma-bearing redirect_uri can't smuggle a second array element. Legacy `access_tokens` fallback in `verifyAccessToken` grandfathers pre-v0.26 bearer tokens as `read+write+admin`. `sweepExpiredTokens()` runs on startup in try/catch and returns the count via `RETURNING 1` + array length. RFC hardening: `client_id` folded atomically into the `DELETE WHERE` for both auth-code exchange and refresh rotation (wrong-client paths don't burn the row); refresh-scope-subset enforced against the original grant on the row (RFC 6749 §6, so revoking a scope shrinks existing refresh tokens); `client_id` bound on `revokeToken` (RFC 7009 §2.1); `/token` `redirect_uri` validated against the `/authorize` value (RFC 6749 §4.1.3, empty-string treated as missing not wildcard); bare `catch {}` in `verifyAccessToken`/`getClient` replaced by `isUndefinedColumnError` from `src/core/utils.ts` (only SQLSTATE 42703 falls through to legacy; lock timeouts/network blips throw); `dcrDisabled` constructor option lets `serve-http.ts` disable `/register` without monkey-patching the router. Module-private `coerceTimestamp()` normalizes postgres-driver-as-string BIGINT columns to JS numbers at 5 read sites (`getClient` for RFC 7591 §3.2.1 numeric timestamps, `exchangeRefreshToken` + `verifyAccessToken` for the SDK's `typeof === 'number'` check); throws on NaN/Infinity (fail loud at boundary), returns undefined for SQL NULL (callers treat NULL as expired). Not promoted to `utils.ts` — generic BIGINT precision-loss risk. `registerClient` honors `token_endpoint_auth_method: "none"` (RFC 7591 §3.2.1): public PKCE clients store `client_secret_hash = NULL` and the response omits `client_secret`; confidential clients (`client_secret_post` / `client_secret_basic`) keep their one-time-reveal shape; `getClient` normalizes NULL `client_secret_hash` to JS `undefined` so the SDK's clientAuth path accepts public clients. `verifyAccessToken` JOINs `oauth_clients.source_id` (write scope, scalar) + `oauth_clients.federated_read` (read scope, TEXT[]) onto the returned `AuthInfo`; legacy brains degrade via `isUndefinedColumnError` fallback. @@ -302,7 +302,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). - `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). diff --git a/docs/mcp/DEPLOY.md b/docs/mcp/DEPLOY.md index deec780c2..62bd84156 100644 --- a/docs/mcp/DEPLOY.md +++ b/docs/mcp/DEPLOY.md @@ -74,13 +74,20 @@ to the HTTP server, so no migration is required. gbrain serve --http --port 3131 ``` -On first start, the server prints an **admin bootstrap token** to stderr: +On first start in an interactive terminal, the server prints an **admin +bootstrap token** to stderr: ``` Admin bootstrap token: 3a1f9c... Open http://localhost:3131/admin and paste it to log in. ``` +On a non-TTY start (systemd, Docker, any piped or captured logs) the generated +token is hidden so it never lands in log storage. For headless deploys either +set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` to a value you control before starting, or +run `gbrain serve --http --print-admin-token` once on a trusted terminal to +force printing. + Save this token. Open `http://localhost:3131/admin` and paste it to access the dashboard. The dashboard shows live activity, registered clients, request logs, and per-client config export. diff --git a/docs/tutorials/company-brain.md b/docs/tutorials/company-brain.md index 6c4e8a641..b2dc98b00 100644 --- a/docs/tutorials/company-brain.md +++ b/docs/tutorials/company-brain.md @@ -158,7 +158,7 @@ gbrain serve --http --port 3131 --bind 0.0.0.0 The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface. -The server prints an admin bootstrap token to stderr on first start. Save it. You'll use it once for the admin dashboard. +The server prints an admin bootstrap token to stderr on first start when run in an interactive terminal. Save it. You'll use it once for the admin dashboard. On a non-TTY start (systemd, Docker, piped logs) the token is hidden from logs — set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` yourself or pass `--print-admin-token` on a trusted terminal instead. For development, tunnel the local server out via ngrok: diff --git a/llms-full.txt b/llms-full.txt index d3ad276d8..607d27af2 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -3643,13 +3643,20 @@ to the HTTP server, so no migration is required. gbrain serve --http --port 3131 ``` -On first start, the server prints an **admin bootstrap token** to stderr: +On first start in an interactive terminal, the server prints an **admin +bootstrap token** to stderr: ``` Admin bootstrap token: 3a1f9c... Open http://localhost:3131/admin and paste it to log in. ``` +On a non-TTY start (systemd, Docker, any piped or captured logs) the generated +token is hidden so it never lands in log storage. For headless deploys either +set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` to a value you control before starting, or +run `gbrain serve --http --print-admin-token` once on a trusted terminal to +force printing. + Save this token. Open `http://localhost:3131/admin` and paste it to access the dashboard. The dashboard shows live activity, registered clients, request logs, and per-client config export. diff --git a/package.json b/package.json index ec882602c..9d7da20c3 100644 --- a/package.json +++ b/package.json @@ -144,5 +144,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.59.0" + "version": "0.42.60.0" } diff --git a/src/commands/autopilot.ts b/src/commands/autopilot.ts index 927b571e5..417064cf1 100644 --- a/src/commands/autopilot.ts +++ b/src/commands/autopilot.ts @@ -130,6 +130,37 @@ export function shouldSpawnAutopilotWorker(args: string[]): boolean { return !args.includes('--no-worker'); } +export function isPidAlive(pid: number): boolean { + if (!Number.isFinite(pid) || pid <= 0) return false; + try { + process.kill(pid, 0); + return true; + } catch (error: unknown) { + return (error as NodeJS.ErrnoException).code === 'EPERM'; + } +} + +export function decideLockAcquisition( + lockPath: string, + currentPid: number, +): { action: 'acquire' } | { action: 'exit'; holderPid: number } | { action: 'takeover'; reason: string } { + if (!existsSync(lockPath)) return { action: 'acquire' }; + + let raw = ''; + try { + raw = readFileSync(lockPath, 'utf-8').trim(); + } catch { + // An unreadable lock cannot prove another process is alive. + } + + const holderPid = Number.parseInt(raw, 10); + const sameProcess = Number.isFinite(holderPid) && holderPid === currentPid; + const alive = !sameProcess && isPidAlive(holderPid); + + if (alive) return { action: 'exit', holderPid }; + return { action: 'takeover', reason: `dead pid ${raw || ''}` }; +} + // ── Self-upgrade silent channel (v0.42; opt-in, supervisor-relaunch) ───────── /** @@ -351,14 +382,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) { const lockPath = gbrainHomePath('autopilot.lock'); try { mkdirSync(gbrainHomePath(), { recursive: true }); - if (existsSync(lockPath)) { - const stat = require('fs').statSync(lockPath); - const ageMinutes = (Date.now() - stat.mtimeMs) / 60000; - if (ageMinutes < 10) { - console.error('Another autopilot instance is running (lock file is fresh). Exiting.'); - process.exit(0); - } - console.log('Stale lock file found (>10 min). Taking over.'); + const decision = decideLockAcquisition(lockPath, process.pid); + if (decision.action === 'exit') { + console.error(`Another autopilot instance is running (pid ${decision.holderPid}). Exiting.`); + process.exit(0); + } + if (decision.action === 'takeover') { + console.log(`Stale autopilot lock found (${decision.reason}). Taking over.`); } writeFileSync(lockPath, String(process.pid)); } catch { /* best-effort */ } diff --git a/src/commands/book-mirror.ts b/src/commands/book-mirror.ts index 58e59a249..365c7f829 100644 --- a/src/commands/book-mirror.ts +++ b/src/commands/book-mirror.ts @@ -257,7 +257,9 @@ function buildChapterPrompt( return `You are analyzing one chapter of "${bookTitle}"${authorLine} for the user. -Your output is a markdown two-column table where the LEFT column preserves the chapter's actual content (stories, frameworks, statistics, named examples) and the RIGHT column maps each idea to the user's actual life using their words, situations, and patterns from the brain. +Your output is a two-column HTML table where the LEFT column preserves the chapter's actual content (stories, frameworks, statistics, named examples) and the RIGHT column maps each idea to the user's actual life using their words, situations, and patterns from the brain. + +CRITICAL: Use an HTML with valign="top" on EVERY cell — NOT a markdown pipe table. Markdown pipe tables have no way to set vertical alignment, so every renderer except GitHub middle-aligns the rows, which is unreadable when the two columns have different lengths. The HTML
form top-aligns everywhere (GitHub, PDF, Obsidian). This is chapter ${chapter.index} of ${totalChapters}. @@ -276,11 +278,12 @@ Return ONLY a single markdown section in this exact shape: ### Key Ideas [2-4 sentence thesis of the chapter — what the author is actually arguing.] -| What the Author Says | How This Applies to You | -|---|---| -| [Detailed paragraph: a section/argument from the chapter, preserving stories, stats, frameworks, named examples. Use \`

\` for paragraph breaks within the cell.] | [Specific personal connection: name dates, people, exact quotes from the user, real situations. Same \`

\` for breaks.] | -| [Next section] | [Next mirror] | -| [4-10 rows depending on chapter density] | | +
+ + + + [4-10 rows depending on chapter density] +
What the Author SaysHow This Applies to You
[Detailed paragraph: a section/argument from the chapter, preserving stories, stats, frameworks, named examples. Use \`

\` for paragraph breaks within the cell.]		[Specific personal connection: name dates, people, exact quotes from the user, real situations. Same \`

\` for breaks.]
[Next section][Next mirror]
\`\`\` ## RULES @@ -290,6 +293,7 @@ Return ONLY a single markdown section in this exact shape: - 4-10 rows per chapter. If a section honestly doesn't apply, write \`*This section is less directly relevant because [specific reason].*\` Don't force connections. - Never generic ("This might apply if you've ever felt..."). Never sycophantic. Never preach. - Use \`

\` for paragraph breaks inside table cells, not literal newlines. +- EVERY MUST carry valign="top". Never emit a markdown pipe table (| ... | ... |) — always the HTML form above. You have ${DEFAULT_MAX_TURNS} turns and read-only tools (get_page, search). You CANNOT call put_page — your output is the markdown text in your final message. The CLI assembles all chapters and writes the brain page. @@ -314,7 +318,7 @@ title: "${opts.title} — Personalized" type: book-analysis${authorLine} date: ${today} context: "${contextSummary.replace(/"/g, '\\"')}" -tags: [book, personalized, two-column] +tags: [book, personalized, two-column-htmltable-valign-top] ---`; const intro = `# ${opts.title} — Personalized diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 01f5dbe43..6c9a92e6b 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -494,6 +494,38 @@ export function extractTimelineFromContent(content: string, slug: string): Extra entries.push({ slug, date: match[1], source: 'markdown', summary: match[2].trim(), detail: detail || undefined }); } + // Format 3: Inline citation — [Source: , YYYY-MM-DD] + // + // This is the citation convention gbrain's own quality rules require on + // every brain write (skills/conventions/quality.md), so dated evidence is + // pervasive in curated pages — but until now the extractor could not see + // it, and a page whose dates all live in citations scored zero timeline + // coverage. The entry's summary is the sentence the citation annotates + // (the surrounding line with citation markers stripped). + // + // Lines already captured by Format 1 are skipped: a timeline bullet often + // carries its own [Source: ...] citation, and re-extracting it would file + // a duplicate entry under a different (source, summary) shape that the + // DB-level uniqueness cannot collapse. + const citationPattern = /\[Source:\s*([^\]]+?),\s*(\d{4}-\d{2}-\d{2})\s*\]/g; + const bulletLinePattern = /^-\s+\*\*\d{4}-\d{2}-\d{2}\*\*\s*\|/; + for (const line of content.split(/\r?\n/)) { + if (bulletLinePattern.test(line)) continue; + const lineMatches = [...line.matchAll(citationPattern)]; + if (lineMatches.length === 0) continue; + // Strip every citation marker from the line to leave the annotated text. + const summary = line + .replace(/\[Source:[^\]]*\]/g, '') + .replace(/^[-*>#\s]+/, '') + .replace(/\s+/g, ' ') + .trim() + .slice(0, 300); + if (!summary) continue; // a bare citation with no surrounding text is not an event + for (const m of lineMatches) { + entries.push({ slug, date: m[2], source: m[1].trim().slice(0, 200), summary }); + } + } + return entries; } diff --git a/src/commands/files.ts b/src/commands/files.ts index a8990bfdd..d38bd5dcf 100644 --- a/src/commands/files.ts +++ b/src/commands/files.ts @@ -116,8 +116,7 @@ async function listFiles(engine: BrainEngine, slug?: string) { console.log(`${rows.length} file(s):`); for (const row of rows) { - const sizeBytes = row.size_bytes as number | null; - const size = sizeBytes ? `${Math.round(sizeBytes / 1024)}KB` : '?'; + const size = row.size_bytes ? `${Math.round(Number(row.size_bytes) / 1024)}KB` : '?'; console.log(` ${row.page_slug || '(unlinked)'} / ${row.filename} [${size}, ${row.mime_type || '?'}]`); } } diff --git a/src/commands/orphans.ts b/src/commands/orphans.ts index 04765d05f..a440c1017 100644 --- a/src/commands/orphans.ts +++ b/src/commands/orphans.ts @@ -53,7 +53,15 @@ const DENY_PREFIXES = [ ]; /** First slug segments where no inbound links is expected */ -const FIRST_SEGMENT_EXCLUSIONS = new Set(['scratch', 'thoughts', 'catalog', 'entities']); +const FIRST_SEGMENT_EXCLUSIONS = new Set([ + 'scratch', + 'thoughts', + 'catalog', + 'entities', + 'raw', + 'atoms', + 'skills', +]); // --- Filter logic --- diff --git a/src/commands/schema.ts b/src/commands/schema.ts index ead29a100..27c31875f 100644 --- a/src/commands/schema.ts +++ b/src/commands/schema.ts @@ -23,6 +23,7 @@ import { addAliasToType, addLinkTypeToPack, addPrefixToType, + BUNDLED_PACK_NAMES, addTypeToPack, invalidatePackCache, loadActivePack, @@ -179,7 +180,7 @@ async function runActive(_args: string[]): Promise { } function runList(_args: string[]): void { - const bundled = ['gbrain-base', 'gbrain-recommended']; + const bundled = [...BUNDLED_PACK_NAMES]; const installedDir = gbrainPath('schema-packs'); const installed: string[] = []; if (existsSync(installedDir)) { @@ -366,12 +367,12 @@ function runUse(args: string[]): void { } function packPathByName(name: string): string | null { - if (name === 'gbrain-base') { + if (BUNDLED_PACK_NAMES.has(name)) { // Resolve bundled YAML — try a few locations. const here = dirname(new URL(import.meta.url).pathname); const candidates = [ - join(here, '..', 'core', 'schema-pack', 'base', 'gbrain-base.yaml'), - join(here, '..', '..', 'src', 'core', 'schema-pack', 'base', 'gbrain-base.yaml'), + join(here, '..', 'core', 'schema-pack', 'base', `${name}.yaml`), + join(here, '..', '..', 'src', 'core', 'schema-pack', 'base', `${name}.yaml`), ]; for (const c of candidates) { if (existsSync(c)) return c; diff --git a/src/commands/serve-http.ts b/src/commands/serve-http.ts index 5a0539466..20a165a50 100644 --- a/src/commands/serve-http.ts +++ b/src/commands/serve-http.ts @@ -90,6 +90,26 @@ export function resolveBootstrapToken( return { kind: 'ok', token: trimmed, fromEnv: true }; } +/** + * #2624: decide whether the generated admin bootstrap token is hidden from + * the startup banner. Fail-safe default: a generated token is NOT printed + * unless stderr is an interactive TTY, so containerized (non-TTY) deploys + * never ship the secret to centralized log storage. Env-sourced tokens are + * always hidden (operator already holds them). Explicit --suppress hides + * everything; --print-admin-token forces the raw value even on a non-TTY. + */ +export function shouldSuppressBootstrapPrint(opts: { + suppress: boolean; + fromEnv: boolean; + forcePrint: boolean; + isTty: boolean; +}): boolean { + if (opts.suppress) return true; + if (opts.fromEnv) return true; + if (opts.forcePrint) return false; + return !opts.isTty; +} + export type ProbeHealthResult = | { ok: true; status: 200; body: { status: 'ok'; version: string; engine: string; [k: string]: unknown } } | { ok: false; status: 503; body: { error: 'service_unavailable'; error_description: string } }; @@ -304,6 +324,14 @@ interface ServeHttpOptions { * tracking the regenerated value through other means. */ suppressBootstrapToken?: boolean; + /** + * #2624: force-print the generated admin bootstrap token even on a + * non-TTY (containerized) start. By default the raw token is only printed + * when stderr is an interactive TTY, so it never lands in centralized log + * storage for headless deploys. Set this when you genuinely need the value + * captured to a non-interactive log and accept the leak. + */ + printAdminToken?: boolean; } /** @@ -530,7 +558,12 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption let bootstrapToken: string = resolved.token; let bootstrapFromEnv: boolean = resolved.fromEnv; const bootstrapHash = createHash('sha256').update(bootstrapToken).digest('hex'); - const suppressBootstrapPrint = options.suppressBootstrapToken === true; + const suppressBootstrapPrint = shouldSuppressBootstrapPrint({ + suppress: options.suppressBootstrapToken === true, + fromEnv: bootstrapFromEnv, + forcePrint: options.printAdminToken === true, + isTty: process.stderr.isTTY === true, + }); const adminSessions = new Map(); // sessionId → expiresAt // SSE clients for live activity feed @@ -2166,10 +2199,10 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption ║ MCP: http://localhost:${port}/mcp${' '.repeat(Math.max(0, 21 - String(port).length))}║ ║ Health: http://localhost:${port}/health${' '.repeat(Math.max(0, 18 - String(port).length))}║ ╠══════════════════════════════════════════════════════╣ -${suppressBootstrapPrint - ? '║ Admin Token: suppressed (--suppress-bootstrap-token) ║\n╚══════════════════════════════════════════════════════╝' - : bootstrapFromEnv - ? '║ Admin Token: from $GBRAIN_ADMIN_BOOTSTRAP_TOKEN ║\n╚══════════════════════════════════════════════════════╝' +${bootstrapFromEnv + ? '║ Admin Token: from $GBRAIN_ADMIN_BOOTSTRAP_TOKEN ║\n╚══════════════════════════════════════════════════════╝' + : suppressBootstrapPrint + ? '║ Admin Token: hidden (non-TTY log-leak guard) ║\n║ set $GBRAIN_ADMIN_BOOTSTRAP_TOKEN, or pass ║\n║ --print-admin-token on a trusted terminal. ║\n╚══════════════════════════════════════════════════════╝' : `║ Admin Token (paste into /admin login): ║\n║ ${bootstrapToken.substring(0, 50)} ║\n║ ${bootstrapToken.substring(50).padEnd(50)} ║\n╚══════════════════════════════════════════════════════╝`} `); }); diff --git a/src/commands/serve.ts b/src/commands/serve.ts index de7352dc4..adfc604f7 100644 --- a/src/commands/serve.ts +++ b/src/commands/serve.ts @@ -118,8 +118,13 @@ export async function runServe( // restart. const suppressBootstrapToken = args.includes('--suppress-bootstrap-token'); + // #2624: by default the generated token only prints on an interactive + // TTY (never into container log storage). --print-admin-token forces the + // raw value even on a non-TTY start. + const printAdminToken = args.includes('--print-admin-token'); + const { runServeHttp } = await import('./serve-http.ts'); - await runServeHttp(engine, { port, tokenTtl, enableDcr, enableDcrInsecure, publicUrl, logFullParams, bind, suppressBootstrapToken }); + await runServeHttp(engine, { port, tokenTtl, enableDcr, enableDcrInsecure, publicUrl, logFullParams, bind, suppressBootstrapToken, printAdminToken }); return; } diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 2f8d5e42a..ff245ef0a 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -3109,23 +3109,44 @@ async function performFullSync( // repo-relative (importFile uses `relative(dir, filePath)`), so relativize // to the same form before membership-testing — otherwise every page looks // stale and the reconcile would wrongly delete live pages. - const current = new Set( - collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' }) - .map(abs => relative(repoPath, abs)), - ); + // + // #2828: planReconcileDeletes ALSO normalizes path separators on both sides + // of the membership test. On a Windows checkout `path.relative` yields + // backslash paths while a stored source_path can hold git-derived forward + // slashes; without normalization every file-backed page mismatches, looks + // stale, and the reconcile wipes the whole source. + const currentFiles = collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' }) + .map(abs => relative(repoPath, abs)); const rows = await engine.executeRaw<{ slug: string; source_path: string | null }>( `SELECT slug, source_path FROM pages WHERE source_id = $1 AND source_path IS NOT NULL AND deleted_at IS NULL`, [sid], ); - const staleSlugs = rows - .filter(r => r.source_path != null - && isSyncable(r.source_path, reconcileSyncOpts) - && !current.has(r.source_path)) - .map(r => r.slug); - if (staleSlugs.length > 0) { + const plan = planReconcileDeletes( + rows, + currentFiles, + p => isSyncable(p, reconcileSyncOpts), + ); + if (plan.staleSlugs.length > 0 && plan.massDelete && !massReconcileAllowed()) { + // #2828 mass-delete safety valve: a reconcile that would sweep more than + // half of the pages this strategy manages, on a source with a non-trivial + // number of them, is almost always a path-comparison bug or the wrong repo + // path — NOT a genuine bulk deletion. Skip the delete and warn loudly + // instead of silently wiping the brain. + serr( + `\n WARNING: refusing to reconcile-delete ${plan.staleSlugs.length} of ` + + `${plan.reconcilableCount} file-backed page(s) for source '${sid}' ` + + `(> ${Math.round(MASS_RECONCILE_RATIO * 100)}% of them).\n` + + ` A full sync removes pages only when their backing file is gone. Deleting\n` + + ` this many at once almost always means the paths were compared wrong (e.g.\n` + + ` a path-separator mismatch) or the WRONG repo path was synced — not that\n` + + ` you actually deleted that many files. No pages were deleted.\n` + + ` If this bulk removal is genuinely intended, re-run with ` + + `GBRAIN_ALLOW_MASS_RECONCILE=1 to restore the old behavior.`, + ); + } else if (plan.staleSlugs.length > 0) { const deleteScopedOpts = { sourceId: sid }; - for (let i = 0; i < staleSlugs.length; i += DELETE_BATCH_SIZE) { - const batch = staleSlugs.slice(i, i + DELETE_BATCH_SIZE); + for (let i = 0; i < plan.staleSlugs.length; i += DELETE_BATCH_SIZE) { + const batch = plan.staleSlugs.slice(i, i + DELETE_BATCH_SIZE); try { const deleted = await engine.deletePages(batch, deleteScopedOpts); reconciledDeletes += deleted.length; @@ -3179,6 +3200,82 @@ async function performFullSync( }; } +/** + * #2828 full-sync reconcile safety-valve thresholds. A reconcile that would + * delete more than MASS_RECONCILE_RATIO of the file-backed pages a strategy + * manages, on a source that holds more than MASS_RECONCILE_MIN_PAGES of them, is + * treated as a suspected path-comparison bug rather than a real bulk deletion. + */ +export const MASS_RECONCILE_RATIO = 0.5; +export const MASS_RECONCILE_MIN_PAGES = 20; + +/** + * Normalize path separators so a page whose stored `source_path` was written + * with a different separator than the local OS's `path.relative` produces (e.g. + * git-derived forward-slash paths on a Windows checkout) still compares equal. + * Without this, on Windows every file-backed page looks stale and the reconcile + * wrongly deletes the whole source (#2828). + */ +function normalizeReconcilePath(p: string): string { + return p.replace(/\\/g, '/'); +} + +export interface ReconcilePlan { + /** Slugs whose backing file is genuinely gone; safe to reconcile-delete. */ + staleSlugs: string[]; + /** + * File-backed, in-strategy pages the reconcile can act on. This is the + * denominator for the mass-delete valve (the exact population at risk). + */ + reconcilableCount: number; + /** + * True when `staleSlugs` would sweep more than MASS_RECONCILE_RATIO of + * `reconcilableCount`, on a source with more than MASS_RECONCILE_MIN_PAGES of + * them — the mass-delete signal that trips the safety valve. + */ + massDelete: boolean; +} + +/** + * #2828: decide which file-backed pages a full-sync reconcile should delete, and + * whether that deletion is suspiciously large. Pure and exported so both the + * separator normalization and the mass-delete valve are unit-testable without a + * live engine or a Windows host. + * + * @param rows pages with a non-null `source_path` (deleted_at IS NULL). + * @param currentFiles repo-relative paths present in the working tree. + * @param isSyncablePath predicate excluding metafiles and the wrong strategy. + */ +export function planReconcileDeletes( + rows: ReadonlyArray<{ slug: string; source_path: string | null }>, + currentFiles: Iterable, + isSyncablePath: (p: string) => boolean, +): ReconcilePlan { + const current = new Set(); + for (const f of currentFiles) current.add(normalizeReconcilePath(f)); + const reconcilable = rows.filter( + r => r.source_path != null && isSyncablePath(r.source_path), + ); + const staleSlugs = reconcilable + .filter(r => !current.has(normalizeReconcilePath(r.source_path as string))) + .map(r => r.slug); + const massDelete = + reconcilable.length > MASS_RECONCILE_MIN_PAGES && + staleSlugs.length > reconcilable.length * MASS_RECONCILE_RATIO; + return { staleSlugs, reconcilableCount: reconcilable.length, massDelete }; +} + +/** + * #2828 escape hatch: `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the pre-valve + * behavior for the rare intentional bulk removal. Env-only (an incident-time + * override), mirroring `resolveStallAbortSeconds`' pure, env-parameterized shape. + */ +export function massReconcileAllowed( + env: Record = process.env, +): boolean { + return env.GBRAIN_ALLOW_MASS_RECONCILE === '1'; +} + /** * Grace window (seconds) between the watchdog's SIGTERM and SIGKILL. SIGTERM * gives a responsive loop a clean shutdown; SIGKILL is the starvation backstop. diff --git a/src/commands/takes.ts b/src/commands/takes.ts index a89513b4e..5dd24911e 100644 --- a/src/commands/takes.ts +++ b/src/commands/takes.ts @@ -28,6 +28,7 @@ import { type ParsedTake, } from '../core/takes-fence.ts'; import { withPageLock } from '../core/page-lock.ts'; +import { resolveSourceId } from '../core/source-resolver.ts'; // --- Helpers --- @@ -83,18 +84,31 @@ function ensureFloat(raw: string | undefined, fallback: number): number { return n; } -async function getPageId(engine: BrainEngine, slug: string): Promise { - const rows = await engine.executeRaw<{ id: number }>( - `SELECT id FROM pages WHERE slug = $1 LIMIT 1`, - [slug], - ); +async function getPageId(engine: BrainEngine, slug: string, sourceId?: string): Promise { + const rows = sourceId + ? await engine.executeRaw<{ id: number }>( + `SELECT id FROM pages WHERE slug = $1 AND source_id = $2 LIMIT 1`, + [slug, sourceId], + ) + : await engine.executeRaw<{ id: number }>( + `SELECT id FROM pages WHERE slug = $1 LIMIT 1`, + [slug], + ); if (!rows[0]) { - console.error(`Page not found in brain: ${slug}. Run \`gbrain sync\` first.`); + console.error(`Page not found in brain: ${slug}${sourceId ? ` (source=${sourceId})` : ''}. Run \`gbrain sync\` first.`); process.exit(1); } return rows[0].id; } +async function resolveTakesSourceId(engine: BrainEngine): Promise { + try { + return await resolveSourceId(engine, null); + } catch { + return undefined; + } +} + function readBodyOrEmpty(path: string): string { if (!existsSync(path)) return ''; return readFileSync(path, 'utf-8'); @@ -169,7 +183,7 @@ async function cmdSearch(engine: BrainEngine, args: string[]): Promise { } } -async function cmdAdd(engine: BrainEngine, args: string[]): Promise { +async function cmdAdd(engine: BrainEngine, args: string[], sourceId?: string): Promise { const slug = args[0]; if (!slug) { console.error('Usage: gbrain takes add --claim "..." --kind --who [--weight 0.5] [--source "..."] [--since YYYY-MM]'); @@ -195,7 +209,7 @@ async function cmdAdd(engine: BrainEngine, args: string[]): Promise { writeBody(path, nextBody); // Mirror to DB. Page may not be in DB yet if not synced — caller must run sync first. - const pageId = await getPageId(engine, slug); + const pageId = await getPageId(engine, slug, sourceId); await engine.addTakesBatch([{ page_id: pageId, row_num: rowNum, claim, kind, holder, weight, since_date: since, source, active: true, superseded_by: null, @@ -204,7 +218,7 @@ async function cmdAdd(engine: BrainEngine, args: string[]): Promise { }); } -async function cmdUpdate(engine: BrainEngine, args: string[]): Promise { +async function cmdUpdate(engine: BrainEngine, args: string[], sourceId?: string): Promise { const slug = args[0]; const rowNumStr = flagValue(args, '--row'); if (!slug || !rowNumStr) { @@ -223,7 +237,7 @@ async function cmdUpdate(engine: BrainEngine, args: string[]): Promise { const brainDir = await resolveBrainDir(engine, dirArg ?? null); await withPageLock(slug, async () => { - const pageId = await getPageId(engine, slug); + const pageId = await getPageId(engine, slug, sourceId); await engine.updateTake(pageId, rowNum, fields); // Sync the markdown table: read fence, find row, apply field updates, re-render. @@ -254,7 +268,7 @@ async function cmdUpdate(engine: BrainEngine, args: string[]): Promise { }); } -async function cmdSupersede(engine: BrainEngine, args: string[]): Promise { +async function cmdSupersede(engine: BrainEngine, args: string[], sourceId?: string): Promise { const slug = args[0]; const rowNumStr = flagValue(args, '--row'); if (!slug || !rowNumStr) { @@ -268,7 +282,7 @@ async function cmdSupersede(engine: BrainEngine, args: string[]): Promise const brainDir = await resolveBrainDir(engine, dirArg ?? null); await withPageLock(slug, async () => { - const pageId = await getPageId(engine, slug); + const pageId = await getPageId(engine, slug, sourceId); // Read existing row to inherit kind/holder unless overridden const existing = await engine.listTakes({ page_id: pageId, active: false, limit: 500 }); @@ -302,7 +316,7 @@ async function cmdSupersede(engine: BrainEngine, args: string[]): Promise }); } -async function cmdResolve(engine: BrainEngine, args: string[]): Promise { +async function cmdResolve(engine: BrainEngine, args: string[], sourceId?: string): Promise { const slug = args[0]; const rowNumStr = flagValue(args, '--row'); const qualityStr = flagValue(args, '--quality'); @@ -347,7 +361,7 @@ async function cmdResolve(engine: BrainEngine, args: string[]): Promise { const resolvedBy = flagValue(args, '--by') ?? 'garry'; const dirArg = flagValue(args, '--dir'); - const pageId = await getPageId(engine, slug); + const pageId = await getPageId(engine, slug, sourceId); await engine.resolveTake(pageId, rowNum, { quality, outcome, @@ -564,10 +578,10 @@ Common flags: switch (sub) { case 'search': return cmdSearch(engine, rest); - case 'add': return cmdAdd(engine, rest); - case 'update': return cmdUpdate(engine, rest); - case 'supersede': return cmdSupersede(engine, rest); - case 'resolve': return cmdResolve(engine, rest); + case 'add': return cmdAdd(engine, rest, await resolveTakesSourceId(engine)); + case 'update': return cmdUpdate(engine, rest, await resolveTakesSourceId(engine)); + case 'supersede': return cmdSupersede(engine, rest, await resolveTakesSourceId(engine)); + case 'resolve': return cmdResolve(engine, rest, await resolveTakesSourceId(engine)); case 'scorecard': return cmdScorecard(engine, rest); case 'calibration': return cmdCalibration(engine, rest); case 'revisit': return cmdRevisit(engine, rest); @@ -591,7 +605,8 @@ async function cmdExtract(engine: BrainEngine, rest: string[]): Promise { const sub = rest[0]; if (sub !== '--from-pages') { process.stderr.write( - 'Usage: gbrain takes extract --from-pages [--yes] [--dry-run] [--source-id ] [--max-pages N] [--holder ]\n', + 'Usage: gbrain takes extract --from-pages [--yes] [--dry-run] [--source-id ] [--max-pages N (clamped to 1000)] [--include-covered] [--holder ]\n' + + 'Runs progress: pages that already hold takes are skipped, so repeat runs sweep a large corpus in slices. --include-covered rescans everything (refresh).\n', ); process.exit(1); } @@ -604,6 +619,7 @@ async function cmdExtract(engine: BrainEngine, rest: string[]): Promise { const maxPages = maxPagesRaw ? Math.max(1, Math.min(1000, parseInt(maxPagesRaw, 10) || 50)) : 50; const holderIdx = rest.indexOf('--holder'); const holder = holderIdx >= 0 ? rest[holderIdx + 1] : 'system'; + const includeCovered = rest.includes('--include-covered'); // A12 consent gate. const bootstrapEnabledCfg = await engine.getConfig('takes.bootstrap_enabled'); @@ -628,6 +644,7 @@ async function cmdExtract(engine: BrainEngine, rest: string[]): Promise { dryRun, sourceIdFilter, maxPages, + includeCovered, holder, }); if (result.llm_unavailable) { diff --git a/src/core/ai/build-gateway-config.ts b/src/core/ai/build-gateway-config.ts index 1b5eb9613..9996d3ffa 100644 --- a/src/core/ai/build-gateway-config.ts +++ b/src/core/ai/build-gateway-config.ts @@ -34,6 +34,10 @@ export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig { // plane field now exists (GBrainConfig type) and gets mapped here, so // setting it via `~/.gbrain/config.json` propagates into the gateway. if (c.zeroentropy_api_key) envFromConfig.ZEROENTROPY_API_KEY = c.zeroentropy_api_key; + // Same seam for OpenRouter: `gbrain config set openrouter_api_key X` (or + // config.json) must reach the openrouter recipe's OPENROUTER_API_KEY. + // process.env still wins via the later spread. + if (c.openrouter_api_key) envFromConfig.OPENROUTER_API_KEY = c.openrouter_api_key; // v0.32 codex finding #4+#5 fix: thread local-server _BASE_URL env vars // into base_urls so the gateway hits the user's configured port. Without diff --git a/src/core/ai/gateway.ts b/src/core/ai/gateway.ts index fdcae8779..25c09223b 100644 --- a/src/core/ai/gateway.ts +++ b/src/core/ai/gateway.ts @@ -377,7 +377,8 @@ export function applyOpenAICompatConfig( cfg: AIGatewayConfig, ): { baseURL: string; fetch?: typeof fetch } { if (recipe.resolveOpenAICompatConfig) { - return recipe.resolveOpenAICompatConfig(cfg.env); + const resolved = recipe.resolveOpenAICompatConfig(cfg.env); + return { ...resolved, fetch: resolved.fetch ?? recipe.compat?.fetch }; } const baseURL = cfg.base_urls?.[recipe.id] ?? recipe.base_url_default; if (!baseURL) { @@ -386,7 +387,7 @@ export function applyOpenAICompatConfig( recipe.setup_hint, ); } - return { baseURL }; + return { baseURL, fetch: recipe.compat?.fetch }; } /** @@ -2383,6 +2384,58 @@ export interface ChatToolDef { * production subagent jobs) throws "messages do not match the ModelMessage[] * schema" the moment the model calls a tool. Surfaced by the SkillOpt eval. */ +/** + * Default per-call max output tokens. Thinking-by-default Claude 5 models + * (`anthropic:claude-*-5`) burn a large chunk of the budget on internal + * reasoning before emitting any text, so a 4096 default leaves them with empty + * final text on the subagent tool loop. Give those models headroom; providers + * bill actual tokens, not the cap, so it is free for the models that don't use + * it. Everything else keeps 4096 on purpose: raising the default blanket-wide + * would exceed some openai-compat providers' hard max-output caps (DeepSeek + * 8192, gpt-4o 16384) and 400 on them — a regression for exactly the + * non-Anthropic subagent users the gateway loop exists to serve. + */ +const DEFAULT_MAX_OUTPUT_TOKENS = 4096; +const THINKING_MODEL_MAX_OUTPUT_TOKENS = 32000; +const THINKING_BY_DEFAULT_MODEL_RE = /^anthropic[:/]claude-[a-z0-9]+-5(?:[.-]|$)/i; +function defaultMaxOutputTokens(modelStr: string | undefined): number { + return modelStr && THINKING_BY_DEFAULT_MODEL_RE.test(modelStr) + ? THINKING_MODEL_MAX_OUTPUT_TOKENS + : DEFAULT_MAX_OUTPUT_TOKENS; +} + +/** + * Deep-serialize a tool output into a plain JSON value for the AI SDK v6 + * ModelMessage schema. node-postgres returns `timestamptz` columns as JS + * `Date` instances, and AI SDK v6's `JSONValue` schema rejects a raw Date, + * throwing "Invalid prompt ... ModelMessage[] schema" the moment a + * timestamp-bearing tool result (e.g. `brain_get_page`, `brain_list_pages`) + * is fed back — dead-lettering the whole multi-tool loop. The JSON round-trip + * runs `Date.prototype.toJSON` (ISO string) recursively and drops `undefined`. + * This is a serialization fix at the SDK boundary, NOT a `::jsonb` DB cast — + * it never touches Postgres. (BigInt / circular outputs still throw in + * JSON.stringify; those aren't LLM-serializable and are out of scope.) + */ +function toJsonSafe(value: unknown): unknown { + try { + return JSON.parse(JSON.stringify(value ?? null)); + } catch { + // BigInt / circular output isn't LLM-serializable; degrade to a string + // rather than throwing and dead-lettering the whole tool loop. + return safeStringify(value); + } +} + +/** Stringify that never throws (bigint/circular fall back to String()). */ +function safeStringify(value: unknown): string { + if (typeof value === 'string') return value; + try { + return JSON.stringify(value ?? null); + } catch { + return String(value); + } +} + export function toModelMessages(messages: ChatMessage[]): unknown[] { return messages.map((m) => { if (typeof m.content === 'string') return { role: m.role, content: m.content }; @@ -2398,24 +2451,86 @@ export function toModelMessages(messages: ChatMessage[]): unknown[] { toolCallId: b.toolCallId, toolName: b.toolName, output: b.isError - ? { type: 'error-text' as const, value: typeof b.output === 'string' ? b.output : JSON.stringify(b.output) } + ? { type: 'error-text' as const, value: safeStringify(b.output) } : (typeof b.output === 'string' ? { type: 'text' as const, value: b.output } - : { type: 'json' as const, value: (b.output ?? null) as never }), + : { type: 'json' as const, value: toJsonSafe(b.output) as never }), })), }; } return { role: m.role, - content: blocks.map((b) => { - if (b.type === 'text') return { type: 'text' as const, text: b.text }; - if (b.type === 'tool-call') return { type: 'tool-call' as const, toolCallId: b.toolCallId, toolName: b.toolName, input: b.input }; - return b; - }), + // Drop text blocks whose `text` isn't a string: reasoning models + // (DeepSeek v4, etc.) surface `text: null/undefined` thinking parts that + // AI SDK v6's Zod schema rejects, poisoning the whole call. `''` is valid + // and kept. + content: blocks + .filter((b) => b.type !== 'text' || typeof b.text === 'string') + .map((b) => { + if (b.type === 'text') return { type: 'text' as const, text: b.text }; + if (b.type === 'tool-call') return { type: 'tool-call' as const, toolCallId: b.toolCallId, toolName: b.toolName, input: b.input }; + return b; + }), }; }); } +/** + * Last-resort normalization at the `chat()` boundary: back-fill error stubs for + * any assistant tool-call that isn't answered by the immediately-following + * tool-result turn. The subagent handler already balances its own transcript + * (see reconcileGatewayReplay), so this is a no-op there — it exists for the + * paths reconcile can't reach: a partially-answered turn, a provider that + * duplicates or drops tool-call IDs (local vLLM), or a `finishReason:'length'` + * truncation mid-batch. Without it those histories throw + * AI_MissingToolResultsError inside `generateText`. No-op on balanced input. + * + * @internal exported for tests. + */ +export function repairToolPairing(messages: ChatMessage[]): ChatMessage[] { + const out: ChatMessage[] = []; + for (let i = 0; i < messages.length; i++) { + const m = messages[i]; + out.push(m); + if (typeof m.content === 'string' || m.role !== 'assistant') continue; + + const calls = m.content.filter( + (b): b is Extract => b.type === 'tool-call', + ); + if (calls.length === 0) continue; + + // v6 only accepts results in the immediately-following message. + const next = messages[i + 1]; + const nextBlocks = next && typeof next.content !== 'string' ? next.content : []; + const resolved = new Set( + nextBlocks + .filter((b): b is Extract => b.type === 'tool-result') + .map((b) => b.toolCallId), + ); + + const missing = calls.filter((c) => !resolved.has(c.toolCallId)); + if (missing.length === 0) continue; + + const stubs: ChatBlock[] = missing.map((c) => ({ + type: 'tool-result', + toolCallId: c.toolCallId, + toolName: c.toolName, + output: 'tool result unavailable (recovered after interrupted run)', + isError: true, + })); + + if (resolved.size > 0) { + // A tool-result message follows but is incomplete — merge the stubs in. + out.push({ role: next!.role, content: [...(nextBlocks as ChatBlock[]), ...stubs] }); + i++; // the merged message replaces the original; don't emit it twice. + } else { + // No following tool-result message at all — synthesize one. + out.push({ role: 'user', content: stubs }); + } + } + return out; +} + export interface ChatResult { /** Final text content concatenated from text blocks. */ text: string; @@ -2676,6 +2791,22 @@ async function classifyGatewayGuardrail(input: { } } +export function toAISDKTools(tools: ChatToolDef[] | undefined): Record | undefined { + if (!tools || tools.length === 0) return undefined; + return tools.reduce((acc, t) => { + acc[t.name] = { + description: t.description, + // AI SDK v6 requires a Schema (carrying the schema symbol), not a plain + // `{jsonSchema}` object — the bare object makes asSchema() treat it as a + // thunk and call schema(), throwing "schema is not a function". Wrap the + // raw JSON Schema with the SDK's jsonSchema() helper so tool calls work + // through the real toolLoop (skillopt rollouts + subagent jobs). + inputSchema: jsonSchema(t.inputSchema as any), + }; + return acc; + }, {} as Record); +} + export async function chat(opts: ChatOpts): Promise { const tracker = __budgetStore.getStore() ?? null; const modelStrEarly = opts.model ?? getChatModel(); @@ -2697,7 +2828,7 @@ export async function chat(opts: ChatOpts): Promise { } } const estimatedInputTokens = estimateChatInputTokens(opts); - const maxOutputTokens = opts.maxTokens ?? 4096; + const maxOutputTokens = opts.maxTokens ?? defaultMaxOutputTokens(modelStrEarly); // TX5: reserve BEFORE the provider call. Throws BudgetExhausted on cost, // runtime, or no_pricing (when cap is set). Pre-resolution model id is @@ -2763,21 +2894,7 @@ export async function chat(opts: ChatOpts): Promise { const supportsCache = recipe.touchpoints.chat?.supports_prompt_cache === true; const useCache = !!opts.cacheSystem && supportsCache; - // Build messages. Anthropic prompt-cache markers ride on system + last tool - // via providerOptions; the AI SDK accepts the system as a string for - // generateText, so cache markers go through providerOptions.anthropic. - const tools = (opts.tools ?? []).reduce((acc, t) => { - acc[t.name] = { - description: t.description, - // AI SDK v6 requires a Schema (carrying the schema symbol), not a plain - // `{jsonSchema}` object — the bare object makes asSchema() treat it as a - // thunk and call schema(), throwing "schema is not a function". Wrap the - // raw JSON Schema with the SDK's jsonSchema() helper so tool calls work - // through the real toolLoop (skillopt rollouts + subagent jobs). - inputSchema: jsonSchema(t.inputSchema as any), - }; - return acc; - }, {} as Record); + const tools = toAISDKTools(opts.tools); const providerOptions: Record = {}; if (useCache) { @@ -2804,9 +2921,9 @@ export async function chat(opts: ChatOpts): Promise { const result = await generateText({ model, system: opts.system, - messages: toModelMessages(opts.messages) as any, + messages: toModelMessages(repairToolPairing(opts.messages)) as any, tools: opts.tools && opts.tools.length > 0 ? tools : undefined, - maxOutputTokens: opts.maxTokens ?? 4096, + maxOutputTokens: opts.maxTokens ?? defaultMaxOutputTokens(modelStr), // v0.42.20.0 — default a chat timeout (composes with the caller's signal, // shorter wins). Covers native-anthropic (the default provider + facts Haiku). abortSignal: withDefaultTimeout(opts.abortSignal, AI_CHAT_TIMEOUT_MS), @@ -2957,6 +3074,14 @@ export interface ToolLoopOpts { ) => Promise<{ gbrainToolUseId: string }>; onToolCallComplete?: (gbrainToolUseId: string, output: unknown) => Promise; onToolCallFailed?: (gbrainToolUseId: string, error: string) => Promise; + /** + * Persist the tool-result user turn that closes each tool round, BEFORE it is + * appended to the in-memory history. Without this the loop only kept the + * tool-result turn in memory, so a resumed job reloaded assistant tool-calls + * with no matching results and non-Anthropic providers rejected the + * unbalanced history (AI_MissingToolResultsError). Fires per completed round. + */ + onToolResultTurn?: (turnIdx: number, messageIdx: number, blocks: ChatBlock[]) => Promise; /** Optional per-call heartbeat for observability. */ onHeartbeat?: (event: string, data: Record) => void; @@ -2991,7 +3116,7 @@ export interface ToolLoopResult { */ export async function toolLoop(opts: ToolLoopOpts): Promise { const maxTurns = opts.maxTurns ?? 20; - const maxTokens = opts.maxTokens ?? 4096; + const maxTokens = opts.maxTokens ?? defaultMaxOutputTokens(opts.model ?? getChatModel()); const handlers = opts.toolHandlers; const totalUsage: ChatResult['usage'] = { input_tokens: 0, @@ -3180,9 +3305,11 @@ export async function toolLoop(opts: ToolLoopOpts): Promise { if (stopReason === 'aborted') break; - // Feed all tool results back as a single user message. + // Persist + feed all tool results back as a single user message. The + // persist-before-push mirrors onAssistantTurn's write-ordering: a crash + // after this leaves a balanced transcript for the next resume. const userMessageIdx = messageIdx++; - void userMessageIdx; + await opts.onToolResultTurn?.(turnIdx, userMessageIdx, toolResultBlocks); messages.push({ role: 'user', content: toolResultBlocks }); turnIdx++; diff --git a/src/core/ai/recipes/deepseek.ts b/src/core/ai/recipes/deepseek.ts index daa7cf2e9..d2dd4971d 100644 --- a/src/core/ai/recipes/deepseek.ts +++ b/src/core/ai/recipes/deepseek.ts @@ -1,5 +1,64 @@ import type { Recipe } from '../types.ts'; +/** + * `deepseek-reasoner` returns its answer in a separate `reasoning_content` + * field and leaves `content` empty/whitespace when the whole response was + * reasoning. The AI SDK's openai-compatible adapter reads only `content`, so + * the model appears to answer with nothing. This transport shim promotes + * `reasoning_content` into `content` when `content` is empty, before the + * adapter parses the body. Fail-open: any error returns the original response. + * Non-streaming JSON chat completions only. + * + * @internal exported for tests. + */ +// Cast through `unknown` because TS's `typeof fetch` includes a `preconnect` +// member the arrow function does not implement (matches azure-openai.ts). +export const deepseekReasoningContentCompatFetch = (async ( + input: RequestInfo | URL, + init?: RequestInit, +): Promise => { + const res = await fetch(input as any, init as any); + try { + if (!res.ok) return res; + const ctype = res.headers.get('content-type') ?? ''; + if (!ctype.includes('application/json')) return res; + const json = await res.clone().json(); + const choices = Array.isArray(json?.choices) ? json.choices : []; + let modified = false; + for (const choice of choices) { + const msg = choice?.message; + if (!msg) continue; + // A tool-call turn legitimately carries content:null — the answer is the + // tool call, not text. NEVER promote reasoning_content there: DeepSeek's + // chain-of-thought must not be fed back to the model (it would be + // persisted as assistant text and replayed every subsequent turn, + // contaminating context and inflating tokens). Only promote on a terminal + // text turn whose content is empty. + const hasToolCalls = Array.isArray(msg.tool_calls) && msg.tool_calls.length > 0; + const content = msg.content; + const reasoning = msg.reasoning_content; + const contentEmpty = content == null || (typeof content === 'string' && content.trim() === ''); + if (!hasToolCalls && contentEmpty && typeof reasoning === 'string' && reasoning.trim() !== '') { + msg.content = reasoning; + modified = true; + } + } + if (!modified) return res; + // Rebuild with a fresh header set: the body length changed, so the + // upstream content-length / content-encoding would now be wrong. + const headers = new Headers(res.headers); + headers.delete('content-length'); + headers.delete('content-encoding'); + return new Response(JSON.stringify(json), { + status: res.status, + statusText: res.statusText, + headers, + }); + } catch { + return res; + } +}) as unknown as typeof fetch; + /** * DeepSeek exposes an OpenAI-compatible /v1/chat/completions endpoint. * Useful as the second hop in a refusal-fallback chain and for cheap- @@ -29,4 +88,5 @@ export const deepseek: Recipe = { }, }, setup_hint: 'Get an API key at https://platform.deepseek.com/api_keys, then `export DEEPSEEK_API_KEY=...`', + compat: { fetch: deepseekReasoningContentCompatFetch }, }; diff --git a/src/core/ai/types.ts b/src/core/ai/types.ts index 4dced5c6d..8835c838f 100644 --- a/src/core/ai/types.ts +++ b/src/core/ai/types.ts @@ -326,6 +326,18 @@ export interface Recipe { baseURL: string; fetch?: typeof fetch; }; + /** + * Optional inbound-response rewriter for openai-compatible recipes whose wire + * shape needs normalizing before the AI SDK adapter parses it. `fetch` wraps + * the transport and MUST be fail-open (return the original response on any + * error). Used by DeepSeek to promote `reasoning_content` into `content` when + * the reasoner returns an empty `content` (the adapter reads only `content`). + * Applied by `applyOpenAICompatConfig`; a `resolveOpenAICompatConfig`-provided + * fetch takes precedence when both are present. + */ + compat?: { + fetch?: typeof fetch; + }; /** * v0.32 (D13=A): optional runtime readiness check for local-server * recipes (ollama, llama-server, future lmstudio-recipe). Returns diff --git a/src/core/config.ts b/src/core/config.ts index 8a79b0e06..388bee38c 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -41,6 +41,13 @@ export interface GBrainConfig { * merge → buildGatewayConfig env dict → recipe reads ZEROENTROPY_API_KEY. */ zeroentropy_api_key?: string; + /** + * OpenRouter API key. File-plane slot so `gbrain config set + * openrouter_api_key X` (or config.json) reaches the openrouter recipe: + * file plane → loadConfig env merge → buildGatewayConfig env dict → recipe + * reads OPENROUTER_API_KEY. + */ + openrouter_api_key?: string; /** AI gateway config (v0.14+). v0.36+ default: "zeroentropyai:zembed-1" / 1280 / "anthropic:claude-haiku-4-5-20251001". */ embedding_model?: string; embedding_dimensions?: number; @@ -526,6 +533,7 @@ export function loadConfig(): GBrainConfig | null { ...(process.env.OPENAI_API_KEY ? { openai_api_key: process.env.OPENAI_API_KEY } : {}), ...(process.env.ANTHROPIC_API_KEY ? { anthropic_api_key: process.env.ANTHROPIC_API_KEY } : {}), ...(process.env.ZEROENTROPY_API_KEY ? { zeroentropy_api_key: process.env.ZEROENTROPY_API_KEY } : {}), + ...(process.env.OPENROUTER_API_KEY ? { openrouter_api_key: process.env.OPENROUTER_API_KEY } : {}), ...(process.env.GBRAIN_EMBEDDING_MODEL ? { embedding_model: process.env.GBRAIN_EMBEDDING_MODEL } : {}), ...(process.env.GBRAIN_EMBEDDING_DIMENSIONS ? { embedding_dimensions: parseInt(process.env.GBRAIN_EMBEDDING_DIMENSIONS, 10) } : {}), ...(process.env.GBRAIN_EXPANSION_MODEL ? { expansion_model: process.env.GBRAIN_EXPANSION_MODEL } : {}), @@ -815,6 +823,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ 'database_path', 'openai_api_key', 'anthropic_api_key', + 'zeroentropy_api_key', + 'openrouter_api_key', 'embedding_model', 'embedding_dimensions', 'embedding_disabled', @@ -836,6 +846,11 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ 'sync', 'sync.repo_path', 'sync.last_commit', + // Gateway-native subagent loop toggle (routes subagent jobs through the + // provider-agnostic gateway.toolLoop for non-Anthropic providers). The + // subagent handler's error message tells users to `config set` this, so it + // must be a known key or `config set` rejects it without --force. + 'agent.use_gateway_loop', // DB-plane (v0.32.3 search modes + related) 'search.mode', 'search.cache.enabled', @@ -921,6 +936,16 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ // operator had to discover these by reading source. Registered so `config // set` accepts them directly. See docs/operations/spend-controls.md. 'spend.posture', + // Life Chronicle (v0.42.56.0, #2390). The release notes' enable command is + // `gbrain config set auto_chronicle true`, but the key was never registered + // — so the documented command failed with "Unknown config key" and the + // operator had to discover --force by reading source. Same class as the + // spend-controls registration above. + 'auto_chronicle', + // Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate + // consent reads this key, and enabling it is the documented path to + // `gbrain takes extract --from-pages` — same unregistered-key class. + 'takes.bootstrap_enabled', 'sync.cost_gate_min_usd', 'sync.federated_v2', 'embed.backfill_cooldown_min', @@ -943,6 +968,7 @@ export const KNOWN_CONFIG_KEY_PREFIXES: readonly string[] = [ 'content_sanity.', // v0.41 content-sanity tunables 'mcp.', // mcp.publish_skills, mcp.skills_dir (PR1 skill catalog) 'autopilot.', // autopilot.nightly_quality_probe.*, autopilot.auto_drain.* (#1685) + 'chronicle.', // chronicle.tz + future Life Chronicle knobs (#2390) 'self_upgrade.', // v0.42 self-upgrade (mode, quiet_hours, state) ]; diff --git a/src/core/cycle/extract-atoms.ts b/src/core/cycle/extract-atoms.ts index f1e94bb06..4c73ea400 100644 --- a/src/core/cycle/extract-atoms.ts +++ b/src/core/cycle/extract-atoms.ts @@ -9,24 +9,27 @@ // 4. Write each atom via engine.putPage(slug, page, {sourceId}) // with sourceId threaded so federated brains route correctly. // -// Idempotency (D1 from /plan-eng-review): -// Each atom carries frontmatter.source_hash (16-char sha256 prefix). -// Before processing a transcript/page, query "any atom with this -// source_hash exists in this source?". If yes, skip. Closes both: -// - PR #1414's primary concern (page-side re-extraction) -// - Pre-existing v0.41.2.0 transcript-side date-stamp duplicate bug -// (atom slugs are `atoms/YYYY-MM-DD/`, so re-discovered -// transcripts on day N+1 used to write second atoms; now skipped). +// Idempotency (per-atom, via deterministic slug): +// Each atom's slug is `atoms/<source-date>/<stem>-<title-hash>` — built from +// the SOURCE date (the transcript's own date / the page slug), NOT the run +// date, plus a 6-char hash of the title. Re-extracting the same atom resolves +// to the SAME slug, so engine.putPage upserts in place instead of minting a +// duplicate. This closes three bugs in one scheme: +// - PR #1414's page-side re-extraction. +// - The cross-day transcript duplicate: append-only transcripts grow daily, +// so a run-date prefix (`atoms/<today>/…`) used to re-mint the same atom +// under a new date every day. A source-date prefix is stable, so it now +// upserts. +// - The "trailing-dash twin": the stem routes through slugifySegment (the +// FS-import normalizer) and re-strips a trailing dash after the 60-char +// truncation, so the two write paths can no longer disagree on `…would` +// vs `…would-` and persist the same atom twice. // -// Known limitation (D9 #2 — documented, not blocking): -// If extraction writes atom 1 of 3 then atom 2 throws, source_hash -// filter sees atom 1 exists and skips on next discovery. Atoms 2+3 -// stay missing until content_hash changes. Acceptable for v0.41.2.1: -// - Haiku call failure is rare; network/budget failures rarer. -// - Content edits trigger natural re-extract via new content_hash. -// - The original incident (duplicate atoms) is fully closed. -// Per-atom idempotency via deterministic slug is v0.42+ TODO -// (see TODOS.md). +// The source_hash batch check (atomsExistingForHashes) is retained ONLY as a +// cost fast-path — it skips re-running Haiku on a transcript whose whole-file +// hash is unchanged. On append-only sources that hash changes daily so the +// fast-path won't skip, but the deterministic slug makes the re-run upsert +// rather than duplicate, so correctness no longer depends on it. // // Config: // Reads dream.synthesize.session_corpus_dir + meeting_transcripts_dir @@ -51,6 +54,8 @@ import type { ProgressReporter } from '../progress.ts'; import { chat as gatewayChat } from '../ai/gateway.ts'; import { writeReceipt } from '../extract/receipt-writer.ts'; import { upsertExtractRollup } from '../extract/rollup-writer.ts'; +import { createHash } from 'crypto'; +import { slugifySegment } from '../sync.ts'; const DEFAULT_BUDGET_USD = 0.3; @@ -61,16 +66,56 @@ const ATOM_TYPES = [ 'critique', 'collection', ] as const; -// v0.41.2.1 (D2): brain-page discovery constants. Hardcoded for now; -// future pack-aware refactor is a one-line change to pull from the -// active pack manifest (symmetric with the existing -// src/core/facts/eligibility.ts:49 TODO). -const EXTRACTABLE_PAGE_TYPES = [ +// v0.41.2.1 (D2): brain-page discovery constants. +// +// Legacy floor: the pre-pack hardcoded atom-extraction types. Retained as a +// back-compat union member so a gbrain-base brain never loses an extraction +// target when we begin honoring the pack manifest's `extractable` flags. +const LEGACY_EXTRACTABLE_TYPES = [ 'meeting', 'source', 'article', 'video', 'book', 'original', ] as const; + +// Synthesis outputs are never extraction inputs: extracting atoms from atoms or +// concepts would loop (concepts are synthesized FROM atoms). Mirrors +// facts/eligibility.ts, which likewise excludes `concept` despite its +// extractable:true flag being a documented forward-compat marker. +const SYNTHESIS_OUTPUT_TYPES = new Set<string>(['atom', 'concept']); + const PAGE_DISCOVERY_BUDGET = 50; const MIN_PAGE_CHARS_FOR_EXTRACTION = 500; +/** + * Pure allowlist policy: the legacy floor UNION the pack's `extractable: true` + * types, MINUS synthesis outputs. Exported for unit tests; keep I/O-free. + */ +export function unionExtractableTypes(packExtractable: Iterable<string>): string[] { + const types = new Set<string>(LEGACY_EXTRACTABLE_TYPES); + for (const t of packExtractable) types.add(t); + for (const t of SYNTHESIS_OUTPUT_TYPES) types.delete(t); + return [...types]; +} + +/** + * Resolve the atom-extraction type allowlist from the active schema pack. + * Closes the D2 TODO of honoring the pack manifest (so a type declared + * extractable — e.g. `note` — actually extracts) while preserving behavior for + * gbrain-base via the legacy-floor union. Fail-soft: any pack-load error falls + * back to the legacy floor. + */ +async function resolveExtractableTypes(): Promise<string[]> { + let packExtractable: Iterable<string> = []; + try { + const { loadConfig } = await import('../config.ts'); + const { loadActivePack } = await import('../schema-pack/load-active.ts'); + const { extractableTypesFromPack } = await import('../schema-pack/extractable.ts'); + const resolved = await loadActivePack({ cfg: loadConfig(), remote: false }); + packExtractable = extractableTypesFromPack(resolved.manifest); + } catch { + // Pack unavailable (test seams, bootstrap) — legacy floor only. + } + return unionExtractableTypes(packExtractable); +} + export interface ExtractAtomsOpts { brainDir?: string; sourceId?: string; @@ -195,7 +240,7 @@ export async function discoverExtractablePages( `; const params: unknown[] = [ sourceId, - EXTRACTABLE_PAGE_TYPES as unknown as string[], + await resolveExtractableTypes(), MIN_PAGE_CHARS_FOR_EXTRACTION, PAGE_DISCOVERY_BUDGET, ]; @@ -272,9 +317,10 @@ export async function countExtractAtomsBacklog( AND atom.frontmatter->>'source_hash' = substring(p.content_hash from 1 for 16) AND atom.deleted_at IS NULL )`; + const extractableTypes = await resolveExtractableTypes(); const params = scoped - ? [sourceId, EXTRACTABLE_PAGE_TYPES as unknown as string[], MIN_PAGE_CHARS_FOR_EXTRACTION] - : [EXTRACTABLE_PAGE_TYPES as unknown as string[], MIN_PAGE_CHARS_FOR_EXTRACTION]; + ? [sourceId, extractableTypes, MIN_PAGE_CHARS_FOR_EXTRACTION] + : [extractableTypes, MIN_PAGE_CHARS_FOR_EXTRACTION]; const rows = await engine.executeRaw<{ cnt: string | number }>(sql, params); return Number(rows[0]?.cnt ?? 0); } catch (err) { @@ -517,7 +563,8 @@ export async function runPhaseExtractAtoms( if (!opts.dryRun) { for (const atom of atoms) { - const slug = `atoms/${todayDate()}/${slugify(atom.title)}`; + const srcRef = item.kind === 'transcript' ? item.filePath : item.slug; + const slug = atomSlug(atom.title, srcRef); const originFrontmatter = item.kind === 'transcript' ? { source_path: item.filePath } @@ -691,12 +738,40 @@ function todayDate(): string { return new Date().toISOString().slice(0, 10); } -function slugify(s: string): string { - return s - .toLowerCase() - .replace(/[^a-z0-9\s-]/g, '') - .trim() - .replace(/\s+/g, '-') - .replace(/-+/g, '-') - .slice(0, 60); +/** + * Canonical slug stem for an atom title. Routes through slugifySegment (the + * same normalizer the FS-import path uses) and RE-STRIPS a trailing dash after + * the 60-char truncation — the cut can land on a hyphen and re-introduce one. + * Two writers disagreeing on that trailing dash (`…would` vs `…would-`) was the + * "trailing-dash twin" duplicate bug. + */ +function atomSlugStem(title: string): string { + return slugifySegment(title).slice(0, 60).replace(/-+$/g, '') || 'untitled'; +} + +/** + * Pull a YYYY-MM-DD date from a source reference — a transcript file path like + * `…/2026-06-11-telegram.md`, or a dated page slug. Checks the basename first + * to avoid matching a date in a parent directory. Falls back to the run date + * only when the source carries no date, so dated sources are fully deterministic. + */ +function sourceDate(ref: string): string { + const base = ref.split('/').pop() ?? ref; + const m = base.match(/(\d{4}-\d{2}-\d{2})/) ?? ref.match(/(\d{4}-\d{2}-\d{2})/); + return m ? m[1] : todayDate(); +} + +/** + * Deterministic per-atom slug: `atoms/<source-date>/<stem>-<title-hash>`. + * - Date comes from the SOURCE, not the run date, so re-extracting an + * append-only transcript on a later day yields the SAME slug → putPage + * upserts instead of minting a cross-day duplicate. + * - The 6-char title hash keeps two distinct atoms whose titles share the + * first 60 chars on separate slugs, so a deterministic slug never silently + * clobbers a *different* atom. Hash is over the title only (not body) so an + * LLM rewording the body on re-extraction still upserts rather than dupes. + */ +function atomSlug(title: string, srcRef: string): string { + const hash = createHash('sha256').update(title).digest('hex').slice(0, 6); + return `atoms/${sourceDate(srcRef)}/${atomSlugStem(title)}-${hash}`; } diff --git a/src/core/extract-takes-from-pages.ts b/src/core/extract-takes-from-pages.ts index 931eb0624..784f152ac 100644 --- a/src/core/extract-takes-from-pages.ts +++ b/src/core/extract-takes-from-pages.ts @@ -46,6 +46,13 @@ export interface ExtractTakesFromPagesOpts { sourceIdFilter?: string; /** Max pages to classify per run (caps cost). Default 50. */ maxPages?: number; + /** + * Also rescan pages that already hold takes (refresh semantics). + * Default false: bootstrap runs skip covered pages, so repeated runs + * PROGRESS through a corpus larger than one run's cap instead of + * rescanning the same most-recently-updated slice forever. + */ + includeCovered?: boolean; /** Owner identifier for the inserted takes. Default 'system'. */ holder?: string; /** Model override; defaults to facts.extraction_model. */ @@ -132,12 +139,21 @@ export async function extractTakesFromPages( // Fetch eligible pages. Order by updated_at DESC so recently-edited // pages get bootstrapped first. const typesList = ALLOWED_PAGE_TYPES.map((t) => `'${t}'`).join(', '); + // Bootstrap progression: skip pages that already hold takes (opt out via + // includeCovered). Without this, the updated_at-DESC + LIMIT selection made + // every re-run rescan the same most-recent slice — a corpus larger than one + // run's cap could never be fully bootstrapped (and each rescan re-spent LLM + // budget on covered pages for upsert-identical rows). + const coveredFilter = opts.includeCovered + ? '' + : `AND NOT EXISTS (SELECT 1 FROM takes t WHERE t.page_id = pages.id)`; const pages = await engine.executeRaw<PageRow>( `SELECT id, slug, source_id, type, compiled_truth, updated_at FROM pages WHERE type IN (${typesList}) AND deleted_at IS NULL AND length(COALESCE(compiled_truth, '')) > 200 + ${coveredFilter} ${sourceFilter} ORDER BY updated_at DESC LIMIT ${maxPages}`, diff --git a/src/core/import-file.ts b/src/core/import-file.ts index cbf2d03a0..4e7722abb 100644 --- a/src/core/import-file.ts +++ b/src/core/import-file.ts @@ -115,6 +115,17 @@ async function extractFencedChunks( startChunkIndex: number, ): Promise<ChunkInput[]> { const out: ChunkInput[] = []; + // Fast path: most pages (prose, tables, converted docs) contain no code + // fence at all, so there is nothing for this function to extract. marked's + // lexer still allocates transient memory proportional to page size on every + // call — a ~2MB table-heavy page spikes ~110MB of heap just to produce zero + // fenced chunks. During bulk import those per-page spikes stack on top of + // accumulated chunk/embedding memory and can OOM the worker, and the + // try/catch below cannot rescue an OOM (it is process death, not a throw). + // Skip the lexer entirely when no fence marker (``` or ~~~) is present. + // The `\r` in the line-start class mirrors marked's own `\r\n|\r → \n` + // normalization, so CR/CRLF-only documents don't lose a real fence. + if (!/(^|[\r\n])[ \t]{0,3}(```|~~~)/.test(markdown)) return out; let tokens: ReturnType<typeof marked.lexer>; try { tokens = marked.lexer(markdown); @@ -1304,6 +1315,8 @@ const NEEDS_DECODE = new Set(['.heic', '.heif', '.avif']); export interface ImportTransactionSpec { slug: string; hadExisting: boolean; + /** Source containing the page, chunks, file row, and type-specific writes. */ + sourceId?: string; page: PageInput; /** When undefined, no chunk write happens. When [], deletes any prior chunks. */ chunks?: ChunkInput[]; @@ -1317,23 +1330,26 @@ export async function withImportTransaction( engine: BrainEngine, spec: ImportTransactionSpec, ): Promise<void> { + const sourceId = spec.sourceId ?? 'default'; + const txOpts = spec.sourceId ? { sourceId: spec.sourceId } : undefined; await engine.transaction(async (tx) => { - if (spec.hadExisting) await tx.createVersion(spec.slug); - await tx.putPage(spec.slug, spec.page); + if (spec.hadExisting) await tx.createVersion(spec.slug, txOpts); + await tx.putPage(spec.slug, spec.page, txOpts); if (spec.file) { // page_id resolution after putPage so the new row's id is available. - const stored = await tx.getPage(spec.slug); + const stored = await tx.getPage(spec.slug, txOpts); await tx.upsertFile({ ...spec.file, + source_id: sourceId, page_slug: spec.slug, page_id: stored?.id ?? null, }); } if (spec.chunks !== undefined) { if (spec.chunks.length > 0) { - await tx.upsertChunks(spec.slug, spec.chunks); + await tx.upsertChunks(spec.slug, spec.chunks, txOpts); } else { - await tx.deleteChunks(spec.slug); + await tx.deleteChunks(spec.slug, txOpts); } } if (spec.after) await spec.after(tx); @@ -1563,10 +1579,14 @@ export async function importImageFile( // and slugifyPath would already preserve it). Recompute with the file // extension preserved so the page slug is stable + collision-free. const imageSlug = relativePath.replace(/[\\\/]/g, '/').toLowerCase(); + const sourceOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined; + const linkOpts = opts.sourceId + ? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId, originSourceId: opts.sourceId } + : undefined; const buf = readFileSync(filePath); const hash = createHash('sha256').update(buf).digest('hex'); - const existing = await engine.getPage(imageSlug); + const existing = await engine.getPage(imageSlug, sourceOpts); if (existing?.content_hash === hash) { return { slug: imageSlug, status: 'skipped', chunks: 0 }; } @@ -1642,6 +1662,7 @@ export async function importImageFile( await withImportTransaction(engine, { slug: imageSlug, hadExisting: !!existing, + sourceId: opts.sourceId, page: { type: 'image', page_kind: 'image', @@ -1659,13 +1680,14 @@ export async function importImageFile( // throws when the target doesn't exist; we silently skip for now and // let `gbrain reconcile-links` pick up later additions. for (const candidate of imageOfCandidates(imageSlug)) { - const sibling = await tx.getPage(candidate); + const sibling = await tx.getPage(candidate, sourceOpts); if (sibling) { try { await tx.addLink( imageSlug, candidate, filename, 'image_of', 'manual', imageSlug, 'frontmatter', + linkOpts, ); } catch { /* sibling vanished mid-tx; skip */ } break; // one canonical link per image diff --git a/src/core/link-extraction.ts b/src/core/link-extraction.ts index 8ebbac318..c0fc2644a 100644 --- a/src/core/link-extraction.ts +++ b/src/core/link-extraction.ts @@ -1156,6 +1156,31 @@ export function parseTimelineEntries(content: string): TimelineCandidate[] { result.push({ date, summary, detail: detailLines.join(' ').trim() }); i = j; } + + // Format 3: inline citation — [Source: <source>, YYYY-MM-DD]. The citation + // convention gbrain's own quality rules require on every brain write; + // until now this parser (the db-source extract + ingest path) could not + // see it, so a page whose dates all live in citations scored zero + // timeline coverage. Kept in sync with extractTimelineFromContent's + // Format 3 (the fs-source path). Lines already captured by the timeline + // bullet pass are skipped (a bullet often carries its own citation). + const citationRe = /\[Source:\s*([^\]]+?),\s*(\d{4}-\d{2}-\d{2})\s*\]/g; + for (const line of lines) { + if (TIMELINE_LINE_RE.test(line)) continue; + const matches = [...line.matchAll(citationRe)]; + if (matches.length === 0) continue; + const summary = line + .replace(/\[Source:[^\]]*\]/g, '') + .replace(/^[-*>#\s]+/, '') + .replace(/\s+/g, ' ') + .trim() + .slice(0, 300); + if (!summary) continue; + for (const m of matches) { + if (!isValidDate(m[2])) continue; + result.push({ date: m[2], summary, detail: `Source: ${m[1].trim().slice(0, 200)}` }); + } + } return result; } diff --git a/src/core/minions/handlers/subagent.ts b/src/core/minions/handlers/subagent.ts index 450dc8da0..f0b1ec251 100644 --- a/src/core/minions/handlers/subagent.ts +++ b/src/core/minions/handlers/subagent.ts @@ -777,22 +777,57 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu }); } - // Convert prior Anthropic-shape messages → ChatMessage with ChatBlock content. - // v1 rows store Anthropic content blocks ({type:'tool_use'|'tool_result'|...}); - // we adapt them to ChatBlock shape (type: 'tool-call' | 'tool-result' | 'text'). - const priorChatMessages: ChatMessage[] = priorMessages.map(m => ({ - role: m.role as 'user' | 'assistant', - content: adaptContentBlocksToChatBlocks(m.content_blocks), - })); + // Token rollup across the prior transcript (returned as-is on the terminal + // early-return path; the loop adds only NEW-turn usage otherwise). + const priorTokens = { in: 0, out: 0, cache_read: 0, cache_create: 0 }; + for (const m of priorMessages) { + if (m.tokens_in) priorTokens.in += m.tokens_in; + if (m.tokens_out) priorTokens.out += m.tokens_out; + if (m.tokens_cache_read) priorTokens.cache_read += m.tokens_cache_read; + if (m.tokens_cache_create) priorTokens.cache_create += m.tokens_cache_create; + } + + // Reconcile an unbalanced transcript from a prior crashed/resumed run. The + // gateway loop persists each assistant turn but (pre-fix) never persisted the + // following tool-result user turn, so a resumed job reloads assistant + // tool-calls with no matching results — which non-Anthropic (openai-compat) + // providers reject with AI_MissingToolResultsError, dead-lettering the job. + // reconcileGatewayReplay heals every such dangling turn from settled tool + // executions (mirroring the legacy Anthropic path) and reports the terminal + // case where the prior run already reached end_turn. + const priorToolsV1 = await loadPriorTools(engine, ctx.id); + const { chatMessages: priorChatMessages, nextMessageIdx: reconciledNextIdx, terminalText } = + await reconcileGatewayReplay({ + engine, + jobId: ctx.id, + priorMessages, + priorTools: priorToolsV1, + toolDefs, + signal: ctx.signal, + }); + + // Terminal early-return (#1151 parity): the prior run already reached + // end_turn. Non-Anthropic providers reject a trailing assistant "prefill", + // so surface the persisted text and skip the loop entirely. + if (terminalText !== null) { + return { + result: terminalText, + turns_count: priorChatMessages.filter(m => m.role === 'assistant').length, + stop_reason: 'end_turn', + tokens: priorTokens, + }; + } // Initial seed message if no prior state. const initialMessages: ChatMessage[] = priorChatMessages.length === 0 ? [{ role: 'user', content: data.prompt }] : []; - // Persist seed user message at idx 0 if fresh start. - let nextMessageIdx = priorChatMessages.length; - if (nextMessageIdx === 0) { + // Persist seed user message at idx 0 if fresh start. reconciledNextIdx is + // max(known message_idx) + 1 (0 when no prior rows), which keeps the loop's + // subsequent writes clear of any healed tool-result turn we just inserted. + let nextMessageIdx = reconciledNextIdx; + if (priorChatMessages.length === 0) { await persistMessage(engine, ctx.id, { message_idx: 0, role: 'user', @@ -904,6 +939,22 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu [errorMsg, gbrainToolUseId], ); }, + // Persist the tool-result user turn so a resume reloads a balanced + // transcript. JSON.stringify inside persistMessage ISO-izes any Date in + // tool output, and the value binds through $N::text::jsonb (never + // JSON.stringify into a bare ::jsonb). + onToolResultTurn: async (_turnIdx, messageIdx, blocks) => { + await persistMessage(engine, ctx.id, { + message_idx: messageIdx, + role: 'user', + content_blocks: blocks as unknown as ContentBlock[], + tokens_in: null, + tokens_out: null, + tokens_cache_read: null, + tokens_cache_create: null, + model: null, + }); + }, onHeartbeat: heartbeat, }); @@ -934,6 +985,158 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu }; } +interface ReconcileArgs { + engine: BrainEngine; + jobId: number; + priorMessages: PersistedMessage[]; + priorTools: PersistedToolExec[]; + toolDefs: ToolDef[]; + signal: AbortSignal; +} + +interface ReconcileResult { + chatMessages: ChatMessage[]; + /** max(known/healed message_idx) + 1 — 0 when there is no prior transcript. */ + nextMessageIdx: number; + /** Non-null when the transcript already ended on an assistant text turn. */ + terminalText: string | null; +} + +/** + * Heal an unbalanced gateway replay transcript before it reaches the provider. + * + * The gateway loop persists each assistant turn but historically never + * persisted the following tool-result user turn, so a resumed job reloaded + * assistant tool-calls with no matching results. Non-Anthropic (openai-compat) + * providers reject that with AI_MissingToolResultsError and the job + * dead-letters. This walks EVERY prior assistant turn that carries tool-call + * blocks (not just the tail — a pre-fix multi-turn job persisted several + * consecutive dangling assistant turns) and, if the turn isn't already answered + * by the following tool-result user turn, synthesizes that turn from the + * settled `subagent_tool_executions` rows with the same semantics as the legacy + * Anthropic replay path: + * - complete → stored output + * - failed → stored error (isError) + * - idempotent-pending / missing row → re-dispatch, persist, use output + * - non-idempotent-pending → throw (cannot safely re-run) + * - tool no longer registered → persist failed + error stub + * Each synthesized turn is persisted at `assistant.message_idx + 1` (the free + * slot the pre-fix loop skipped) via `ON CONFLICT DO NOTHING`, so the next + * resume stays balanced. + */ +async function reconcileGatewayReplay(args: ReconcileArgs): Promise<ReconcileResult> { + const { engine, jobId, priorMessages, priorTools, toolDefs, signal } = args; + + const work = priorMessages.map(m => { + const adapted = adaptContentBlocksToChatBlocks(m.content_blocks); + return { + message_idx: m.message_idx, + role: m.role, + blocks: typeof adapted === 'string' ? [{ type: 'text', text: adapted } as ChatBlock] : adapted, + }; + }); + + // Settled executions, looked up by (assistant message_idx, provider + // tool_use_id) with an ordinal-position fallback for legacy rows. + const execByKey = new Map<string, PersistedToolExec>(); + const execByMsg = new Map<number, PersistedToolExec[]>(); + for (const t of priorTools) { + execByKey.set(`${t.message_idx}:${t.tool_use_id}`, t); + const arr = execByMsg.get(t.message_idx) ?? []; + arr.push(t); + execByMsg.set(t.message_idx, arr); + } + + let maxIdx = work.reduce((mx, w) => Math.max(mx, w.message_idx), -1); + + for (let i = 0; i < work.length; i++) { + const msg = work[i]; + if (msg.role !== 'assistant') continue; + const toolCalls = msg.blocks.filter( + (b): b is Extract<ChatBlock, { type: 'tool-call' }> => b.type === 'tool-call', + ); + if (toolCalls.length === 0) continue; + + // Skip if a following tool-result user turn exists AT ALL. A fully-answered + // turn is already balanced; a PARTIALLY-answered turn (only reachable from + // externally-corrupted data — gbrain persists all of a turn's results in one + // message) is left for repairToolPairing() at the chat() boundary, which + // back-fills only the missing ids. Synthesizing a full turn here would + // duplicate the answered results and collide with the persisted row. + const next = work[i + 1]; + if (next && next.role === 'user' && next.blocks.some(b => b.type === 'tool-result')) continue; + + const results: ChatBlock[] = []; + for (let callIdx = 0; callIdx < toolCalls.length; callIdx++) { + const call = toolCalls[callIdx]; + // Prefer an exact (message_idx, provider tool_use_id) match. The + // positional fallback is used only when the row at that ordinal is for + // the SAME tool, so a missing row can't mis-attribute a sibling's output. + const fallback = execByMsg.get(msg.message_idx)?.[callIdx]; + const exec = execByKey.get(`${msg.message_idx}:${call.toolCallId}`) + ?? (fallback && fallback.tool_name === call.toolName ? fallback : undefined); + if (exec?.status === 'complete') { + results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: exec.output ?? null }); + continue; + } + if (exec?.status === 'failed') { + results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: exec.error ?? 'tool failed', isError: true }); + continue; + } + const toolDef = toolDefs.find(t => t.name === call.toolName); + if (!toolDef) { + await persistToolExecFailed(engine, jobId, msg.message_idx, call.toolCallId, call.toolName, call.input, `tool "${call.toolName}" is not in the registry for this subagent`); + results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: `tool "${call.toolName}" is not available`, isError: true }); + continue; + } + if (exec?.status === 'pending' && !toolDef.idempotent) { + throw new Error(`non-idempotent tool "${call.toolName}" pending on resume; cannot safely re-run`); + } + await persistToolExecPending(engine, jobId, msg.message_idx, call.toolCallId, call.toolName, call.input); + try { + const output = await toolDef.execute(call.input, { engine, jobId, remote: true, signal }); + await persistToolExecComplete(engine, jobId, call.toolCallId, output); + results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output }); + } catch (e) { + const errText = e instanceof Error ? (e.stack ?? e.message) : String(e); + await persistToolExecFailed(engine, jobId, msg.message_idx, call.toolCallId, call.toolName, call.input, errText); + results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: errText, isError: true }); + } + } + + const resultIdx = msg.message_idx + 1; + await persistMessage(engine, jobId, { + message_idx: resultIdx, + role: 'user', + content_blocks: results as unknown as ContentBlock[], + tokens_in: null, tokens_out: null, tokens_cache_read: null, tokens_cache_create: null, model: null, + }); + maxIdx = Math.max(maxIdx, resultIdx); + work.splice(i + 1, 0, { message_idx: resultIdx, role: 'user' as const, blocks: results }); + i++; // skip the turn we just inserted + } + + const chatMessages: ChatMessage[] = work.map(w => ({ role: w.role, content: w.blocks })); + + // Terminal case: the transcript already ends on an assistant turn that + // carried real text and made no tool calls (prior run reached end_turn). + // Surface its text; skip the loop. An assistant turn whose blocks are all + // empty (e.g. a reasoning-only/null-text turn that adaptation dropped) is NOT + // terminal — falling through lets the loop re-issue the call rather than + // returning an empty result. + const lastMsg = work[work.length - 1]; + let terminalText: string | null = null; + if (lastMsg && lastMsg.role === 'assistant' && !lastMsg.blocks.some(b => b.type === 'tool-call')) { + const text = lastMsg.blocks + .filter((b): b is Extract<ChatBlock, { type: 'text' }> => b.type === 'text') + .map(b => b.text) + .join('\n'); + if (text.trim() !== '') terminalText = text; + } + + return { chatMessages, nextMessageIdx: maxIdx + 1, terminalText }; +} + function recipeIdFromModel(modelString: string): string { const idx = modelString.indexOf(':'); return idx > 0 ? modelString.slice(0, idx) : 'anthropic'; @@ -1087,7 +1290,8 @@ async function loadPriorTools(engine: BrainEngine, jobId: number): Promise<Persi const rows = await engine.executeRaw<Record<string, unknown>>( `SELECT message_idx, tool_use_id, tool_name, input, status, output, error FROM subagent_tool_executions - WHERE job_id = $1`, + WHERE job_id = $1 + ORDER BY message_idx, COALESCE(ordinal, 0), id`, [jobId], ); return rows.map(r => ({ diff --git a/src/core/model-pricing.ts b/src/core/model-pricing.ts index db1e8854e..090310cef 100644 --- a/src/core/model-pricing.ts +++ b/src/core/model-pricing.ts @@ -52,11 +52,18 @@ export interface ModelPricing { */ export const CANONICAL_PRICING: Record<string, ModelPricing> = { // ── Anthropic ────────────────────────────────────────────────────────── + // Fable 5: Anthropic's top tier, above Opus. $10 in / $50 out. + 'anthropic:claude-fable-5': { input: 10.00, output: 50.00 }, // Opus 4.x: $5 in / $25 out. 4.8 (released 2026-05-28) shares 4.7's // per-token rate — closes gbrain#1819. 'anthropic:claude-opus-4-8': { input: 5.00, output: 25.00 }, 'anthropic:claude-opus-4-7': { input: 5.00, output: 25.00 }, 'anthropic:claude-opus-4-6': { input: 5.00, output: 25.00 }, + // Sonnet 5 (released 2026-06-29): same $3/$15 sticker as 4.6. The launch + // intro discount ($2/$10 through 2026-08-31) is deliberately NOT modeled — + // the table carries standard rates so estimates stay conservative and + // don't need a time-bombed edit when the promo lapses. + 'anthropic:claude-sonnet-5': { input: 3.00, output: 15.00 }, 'anthropic:claude-sonnet-4-6': { input: 3.00, output: 15.00 }, // Haiku 4.5 — both the dateless canonical id and the dated snapshot. 'anthropic:claude-haiku-4-5': { input: 1.00, output: 5.00 }, diff --git a/src/core/operations.ts b/src/core/operations.ts index 41f376b6f..98e02d3c0 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -2686,10 +2686,16 @@ const file_list: Operation = { handler: async (_ctx, p) => { const sql = db.getConnection(); const slug = p.slug as string | undefined; - if (slug) { - return sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files WHERE page_slug = ${slug} ORDER BY filename LIMIT ${FILE_LIST_LIMIT}`; - } - return sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files ORDER BY page_slug, filename LIMIT ${FILE_LIST_LIMIT}`; + const rows = slug + ? await sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files WHERE page_slug = ${slug} ORDER BY filename LIMIT ${FILE_LIST_LIMIT}` + : await sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files ORDER BY page_slug, filename LIMIT ${FILE_LIST_LIMIT}`; + // Postgres returns size_bytes (BIGINT) as native BigInt — JSON.stringify + // throws on those, breaking MCP callers. PGLite returns Number already. + // 9 PB ceiling (2^53 bytes) is far above any plausible file size. + return rows.map((r: Record<string, unknown>) => ({ + ...r, + size_bytes: r.size_bytes == null ? null : Number(r.size_bytes), + })); }, }; diff --git a/src/core/search/hybrid.ts b/src/core/search/hybrid.ts index 7b451ea63..d4669a1cc 100644 --- a/src/core/search/hybrid.ts +++ b/src/core/search/hybrid.ts @@ -15,6 +15,7 @@ import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts'; import { embed, embedQuery } from '../embedding.ts'; import { registerBackgroundWorkDrainer } from '../background-work.ts'; import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts'; +import { resolveHardExcludes } from './source-boost.ts'; import { resolveAdaptiveReturn, applyAdaptiveReturn, @@ -1631,6 +1632,12 @@ export async function hybridSearchCached( const cacheKnobsHash = knobsHash(resolvedForCache, { embeddingColumn: resolvedColCached.name, embeddingModel: resolvedColCached.embeddingModel, + // #2825 — fold the resolved hard-exclude prefix list (defaults ∪ + // GBRAIN_SEARCH_EXCLUDE ∪ per-call exclude_slug_prefixes, minus + // include_slug_prefixes — exactly what the engines' query-build path + // resolves) into the cache key so a row written under one exclude + // policy can't be served to a lookup under another. + hardExcludes: resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes), }); // Cache decision: opts.useCache (explicit) wins over global config; global diff --git a/src/core/search/mode.ts b/src/core/search/mode.ts index f851913c0..cb5a14782 100644 --- a/src/core/search/mode.ts +++ b/src/core/search/mode.ts @@ -747,7 +747,16 @@ export function attributeKnob<K extends keyof ModeBundle>( // to post-fix lookups. Same one-time global cold-miss pattern as the bumps // above (the hash is global, not per-provider); refills within // cache.ttl_seconds (3600s default). -export const KNOBS_HASH_VERSION = 11; +// +// bump 11→12 (2026-07-16, #2825): the resolved hard-exclude slug-prefix list +// (defaults ∪ GBRAIN_SEARCH_EXCLUDE ∪ exclude_slug_prefixes, minus +// include_slug_prefixes) folds into the key via ctx.hardExcludes. It only +// applied at DB-query build time (cache miss), so a process with +// GBRAIN_SEARCH_EXCLUDE set could be served cached rows containing excluded +// slugs written by a process without it, and vice versa. Same one-time +// global cold-miss pattern as the bumps above; refills within +// cache.ttl_seconds (3600s default). +export const KNOBS_HASH_VERSION = 12; /** * v0.36 (D8 / CDX-2) — second-arg context for the cache key. The @@ -776,6 +785,16 @@ export interface KnobsHashContext { */ schemaPack?: string; schemaPackVersion?: string; + /** + * v=12 (#2825): the RESOLVED effective hard-exclude prefix list — the same + * value resolveHardExcludes() produces at query-build time (defaults ∪ + * GBRAIN_SEARCH_EXCLUDE ∪ per-call exclude_slug_prefixes, minus + * include_slug_prefixes). Folded (sorted, so input order is irrelevant) + * into the hash so a cache row written under one exclude policy can never + * be served to a lookup under another. Undefined falls back to the literal + * 'none' for legacy callers that don't thread excludes. + */ + hardExcludes?: string[]; } export function knobsHash( @@ -863,6 +882,12 @@ export function knobsHash( // test/model-pricing.test.ts-style drift guards and the mode tests. `rel=${knobs.relationalRetrieval ? 1 : 0}`, `reld=${knobs.relational_retrieval_depth ?? 2}`, + // v=12 addition (#2825, append-only): resolved hard-exclude prefixes. + // Before this, resolveHardExcludes() only ran at DB-query build time + // (cache miss), so cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs + // across processes. Sorted copy so ['a/','b/'] and ['b/','a/'] hash + // identically; undefined falls back to 'none' for legacy callers. + `hx=${ctx?.hardExcludes ? [...ctx.hardExcludes].sort().join(',') : 'none'}`, ]; const h = createHash('sha256'); h.update(parts.join('|')); diff --git a/src/core/takes-quality-eval/pricing.ts b/src/core/takes-quality-eval/pricing.ts index b8305a417..6c2aed6be 100644 --- a/src/core/takes-quality-eval/pricing.ts +++ b/src/core/takes-quality-eval/pricing.ts @@ -37,6 +37,7 @@ const SUPPORTED_MODELS = [ 'openai:gpt-5.5', 'anthropic:claude-opus-4-8', 'anthropic:claude-opus-4-7', + 'anthropic:claude-sonnet-5', 'anthropic:claude-sonnet-4-6', 'anthropic:claude-haiku-4-5', 'google:gemini-1.5-pro', diff --git a/src/core/think/index.ts b/src/core/think/index.ts index 8f3ab94cc..9c31c81dc 100644 --- a/src/core/think/index.ts +++ b/src/core/think/index.ts @@ -152,6 +152,19 @@ export interface ThinkResult { const DEFAULT_MAX_OUTPUT_TOKENS = 4000; +// Thinking-by-default Claude 5 models (`anthropic:claude-*-5`) spend a large +// share of the output budget on internal reasoning before emitting any answer, +// so the 4000 default leaves `think` with empty or truncated text. Give those +// models headroom; providers bill actual tokens, not the cap. Everything else +// keeps 4000. +const THINKING_DEFAULT_MAX_OUTPUT_TOKENS = 16000; +const THINKING_BY_DEFAULT_MODEL_RE = /^anthropic[:/]claude-[a-z0-9]+-5(?:[.-]|$)/i; +export function maxOutputTokensFor(modelStr: string): number { + return THINKING_BY_DEFAULT_MODEL_RE.test(modelStr) + ? THINKING_DEFAULT_MAX_OUTPUT_TOKENS + : DEFAULT_MAX_OUTPUT_TOKENS; +} + function inferIntent(question: string, anchor?: string): string { if (anchor) return 'entity'; const q = question.toLowerCase(); @@ -465,7 +478,7 @@ export async function runThink( } const result = await client.create({ model: modelUsed, - max_tokens: DEFAULT_MAX_OUTPUT_TOKENS, + max_tokens: maxOutputTokensFor(normalizeModelId(modelUsed)), system: systemPrompt, messages: [{ role: 'user', content: userMessage }], }); diff --git a/test/ai/build-gateway-config.test.ts b/test/ai/build-gateway-config.test.ts index 5284c5bc8..b9ebb27d7 100644 --- a/test/ai/build-gateway-config.test.ts +++ b/test/ai/build-gateway-config.test.ts @@ -86,6 +86,26 @@ describe('buildGatewayConfig env-baseURL passthrough', () => { }); }); +describe('buildGatewayConfig config-plane API-key folding', () => { + test('openrouter_api_key folds into gateway env as OPENROUTER_API_KEY', async () => { + await withEnv({ OPENROUTER_API_KEY: undefined }, async () => { + const cfg = buildGatewayConfig({ + openrouter_api_key: 'sk-or-config-plane', + } as unknown as GBrainConfig); + expect(cfg.env.OPENROUTER_API_KEY).toBe('sk-or-config-plane'); + }); + }); + + test('a real OPENROUTER_API_KEY process.env value wins over the config-plane fallback', async () => { + await withEnv({ OPENROUTER_API_KEY: 'sk-or-env-plane' }, async () => { + const cfg = buildGatewayConfig({ + openrouter_api_key: 'sk-or-config-plane', + } as unknown as GBrainConfig); + expect(cfg.env.OPENROUTER_API_KEY).toBe('sk-or-env-plane'); + }); + }); +}); + describe('buildGatewayConfig env empty-string clobber guard (#1249)', () => { test('an empty-string process.env value does NOT clobber a valid config-plane key', async () => { // Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls. diff --git a/test/ai/deepseek-reasoning-content.test.ts b/test/ai/deepseek-reasoning-content.test.ts new file mode 100644 index 000000000..8c1d6481b --- /dev/null +++ b/test/ai/deepseek-reasoning-content.test.ts @@ -0,0 +1,124 @@ +/** + * Pins the DeepSeek reasoning_content transport shim. `deepseek-reasoner` + * returns its answer in a separate `reasoning_content` field and leaves + * `content` empty when the whole turn was reasoning; the AI SDK's + * openai-compatible adapter reads only `content`, so the model appears to + * answer with nothing. The shim promotes `reasoning_content` into `content` + * when `content` is empty, fail-open on anything unexpected. + */ +import { describe, test, expect, afterEach } from 'bun:test'; +import { deepseekReasoningContentCompatFetch, deepseek } from '../../src/core/ai/recipes/deepseek.ts'; +import { applyOpenAICompatConfig } from '../../src/core/ai/gateway.ts'; +import type { Recipe, AIGatewayConfig } from '../../src/core/ai/types.ts'; + +const realFetch = globalThis.fetch; +afterEach(() => { globalThis.fetch = realFetch; }); + +function stubFetch(body: unknown, init?: { status?: number; contentType?: string }) { + globalThis.fetch = (async () => + new Response(JSON.stringify(body), { + status: init?.status ?? 200, + headers: { 'content-type': init?.contentType ?? 'application/json' }, + })) as unknown as typeof fetch; +} + +describe('deepseekReasoningContentCompatFetch', () => { + test('promotes reasoning_content when content is empty', async () => { + stubFetch({ choices: [{ message: { role: 'assistant', content: '', reasoning_content: 'the answer' } }] }); + const res = await deepseekReasoningContentCompatFetch('https://api.deepseek.com/v1/chat/completions'); + const json = await res.json(); + expect(json.choices[0].message.content).toBe('the answer'); + }); + + test('promotes when content is null or whitespace-only', async () => { + stubFetch({ choices: [{ message: { content: null, reasoning_content: 'from null' } }, { message: { content: ' ', reasoning_content: 'from ws' } }] }); + const res = await deepseekReasoningContentCompatFetch('u'); + const json = await res.json(); + expect(json.choices[0].message.content).toBe('from null'); + expect(json.choices[1].message.content).toBe('from ws'); + }); + + test('leaves non-empty content untouched (no duplication)', async () => { + stubFetch({ choices: [{ message: { content: 'real content', reasoning_content: 'ignored' } }] }); + const res = await deepseekReasoningContentCompatFetch('u'); + const json = await res.json(); + expect(json.choices[0].message.content).toBe('real content'); + }); + + test('both empty: stays empty, no crash', async () => { + stubFetch({ choices: [{ message: { content: '', reasoning_content: '' } }] }); + const res = await deepseekReasoningContentCompatFetch('u'); + const json = await res.json(); + expect(json.choices[0].message.content).toBe(''); + }); + + test('tool-call turn (content:null + tool_calls) is NOT promoted — never feed CoT back', async () => { + // content:null is the standard OpenAI shape on a tool-call turn. Promoting + // reasoning_content here would inject the whole chain-of-thought as assistant + // text, which the loop persists + replays every turn. Must be left alone. + stubFetch({ choices: [{ finish_reason: 'tool_calls', message: { + content: null, + reasoning_content: 'INTERNAL CHAIN OF THOUGHT — must not leak', + tool_calls: [{ id: 'call_1', type: 'function', function: { name: 'brain_search', arguments: '{}' } }], + } }] }); + const res = await deepseekReasoningContentCompatFetch('u'); + const json = await res.json(); + expect(json.choices[0].message.content).toBeNull(); + expect(json.choices[0].message.tool_calls).toHaveLength(1); + }); + + test('rebuilt response drops stale content-length header', async () => { + globalThis.fetch = (async () => new Response(JSON.stringify({ choices: [{ message: { content: '', reasoning_content: 'x' } }] }), { + status: 200, headers: { 'content-type': 'application/json', 'content-length': '999999' }, + })) as unknown as typeof fetch; + const res = await deepseekReasoningContentCompatFetch('u'); + expect(res.headers.get('content-length')).toBeNull(); + expect((await res.json()).choices[0].message.content).toBe('x'); + }); +}); + +describe('applyOpenAICompatConfig — compat.fetch wiring (gateway seam)', () => { + const cfg = { env: {}, base_urls: {} } as unknown as AIGatewayConfig; + + test('threads recipe.compat.fetch onto the resolved config (deepseek)', () => { + // Guards the src/core/ai/gateway.ts wiring: without `?? recipe.compat?.fetch` + // the DeepSeek shim would never install in production. + const resolved = applyOpenAICompatConfig(deepseek, cfg); + expect(resolved.fetch).toBe(deepseekReasoningContentCompatFetch); + }); + + test('a resolveOpenAICompatConfig-provided fetch takes precedence over compat.fetch', () => { + const ownFetch = (async () => new Response('{}')) as unknown as typeof fetch; + const recipe = { + id: 'x', name: 'X', tier: 'openai-compat', implementation: 'openai-compatible', + touchpoints: {}, + compat: { fetch: deepseekReasoningContentCompatFetch }, + resolveOpenAICompatConfig: () => ({ baseURL: 'http://x', fetch: ownFetch }), + } as unknown as Recipe; + expect(applyOpenAICompatConfig(recipe, cfg).fetch).toBe(ownFetch); + }); + + test('falls back to compat.fetch when resolveOpenAICompatConfig omits a fetch', () => { + const recipe = { + id: 'y', name: 'Y', tier: 'openai-compat', implementation: 'openai-compatible', + touchpoints: {}, + compat: { fetch: deepseekReasoningContentCompatFetch }, + resolveOpenAICompatConfig: () => ({ baseURL: 'http://y' }), + } as unknown as Recipe; + expect(applyOpenAICompatConfig(recipe, cfg).fetch).toBe(deepseekReasoningContentCompatFetch); + }); + + test('fail-open on non-ok / non-json responses', async () => { + stubFetch({ error: 'nope' }, { status: 500 }); + const res = await deepseekReasoningContentCompatFetch('u'); + expect(res.status).toBe(500); + globalThis.fetch = (async () => + new Response('plain text', { status: 200, headers: { 'content-type': 'text/plain' } })) as unknown as typeof fetch; + const res2 = await deepseekReasoningContentCompatFetch('u'); + expect(await res2.text()).toBe('plain text'); + }); + + test('recipe wires the shim via compat.fetch', () => { + expect(deepseek.compat?.fetch).toBe(deepseekReasoningContentCompatFetch); + }); +}); diff --git a/test/ai/gateway-tool-loop.test.ts b/test/ai/gateway-tool-loop.test.ts index b9700c379..4e6fb60c0 100644 --- a/test/ai/gateway-tool-loop.test.ts +++ b/test/ai/gateway-tool-loop.test.ts @@ -150,6 +150,47 @@ describe('gateway.toolLoop (v0.38 D11 — provider-agnostic loop control)', () = expect(events[4]).toBe('onAssistantTurn(1)'); // final assistant turn }); + it('persists the tool-result user turn via onToolResultTurn before the next chat', async () => { + let turn = 0; + __setChatTransportForTests(async () => { + turn++; + if (turn === 1) { + return { + text: '', + blocks: [{ type: 'tool-call', toolCallId: 'tc1', toolName: 'search', input: { q: 'x' } }] as ChatBlock[], + stopReason: 'tool_calls', + usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'anthropic:claude-sonnet-4-6', + providerId: 'anthropic', + }; + } + return { + text: 'done', + blocks: [{ type: 'text', text: 'done' }] as ChatBlock[], + stopReason: 'end', + usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'anthropic:claude-sonnet-4-6', + providerId: 'anthropic', + }; + }); + + const resultTurns: Array<{ turnIdx: number; messageIdx: number; blocks: ChatBlock[] }> = []; + await toolLoop({ + initialMessages: [{ role: 'user', content: 'go' }], + tools: [{ name: 'search', description: 's', inputSchema: { type: 'object' } }], + toolHandlers: new Map([['search', { idempotent: true, async execute() { return { hits: 1 }; } }]]), + onToolResultTurn: async (turnIdx, messageIdx, blocks) => { + resultTurns.push({ turnIdx, messageIdx, blocks }); + }, + }); + + // Fired exactly once, for the single tool round, carrying the tool-result. + expect(resultTurns).toHaveLength(1); + expect(resultTurns[0].turnIdx).toBe(0); + expect(resultTurns[0].blocks[0].type).toBe('tool-result'); + expect((resultTurns[0].blocks[0] as Extract<ChatBlock, { type: 'tool-result' }>).toolCallId).toBe('tc1'); + }); + it('replay short-circuits a complete prior tool execution', async () => { let chatCalls = 0; __setChatTransportForTests(async () => { @@ -230,6 +271,30 @@ describe('gateway.toolLoop (v0.38 D11 — provider-agnostic loop control)', () = ).rejects.toThrow(/non-idempotent.*pending/i); }); + it('defaults max output tokens per model: 4096 for non-thinking, 32000 for Claude 5', async () => { + const seen: Array<number | undefined> = []; + __setChatTransportForTests(async (opts) => { + seen.push(opts.maxTokens); + return { + text: 'ok', + blocks: [{ type: 'text', text: 'ok' }] as ChatBlock[], + stopReason: 'end', + usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: opts.model ?? 'anthropic:claude-sonnet-4-6', + providerId: 'anthropic', + }; + }); + + await toolLoop({ model: 'openai:gpt-4o', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() }); + await toolLoop({ model: 'anthropic:claude-sonnet-4-6', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() }); + await toolLoop({ model: 'anthropic:claude-sonnet-5', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() }); + await toolLoop({ model: 'anthropic:claude-fable-5', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() }); + + // Non-thinking / non-Claude-5 stay 4096 (safe under openai-compat caps); + // thinking-by-default Claude 5 models get 32000 headroom. + expect(seen).toEqual([4096, 4096, 32000, 32000]); + }); + it('hits max_turns when the model keeps calling tools', async () => { __setChatTransportForTests(async () => ({ text: '', diff --git a/test/ai/gateway-toolcall-pairing.test.ts b/test/ai/gateway-toolcall-pairing.test.ts new file mode 100644 index 000000000..03681df60 --- /dev/null +++ b/test/ai/gateway-toolcall-pairing.test.ts @@ -0,0 +1,100 @@ +/** + * Pins `repairToolPairing` (the chat()-boundary safety net) and proves, against + * the REAL AI SDK v6 `generateText`, that the two failure modes this wave fixes + * are gone: + * - an unbalanced tool history (assistant tool-call with no tool-result) is + * back-filled so v6 no longer throws AI_MissingToolResultsError, and + * - a Date-bearing tool-result (Postgres timestamptz) passes v6's ModelMessage + * JSONValue schema after `toModelMessages` ISO-izes it, whereas a raw Date + * is still rejected (control). + * + * MockLanguageModelV3 = no network / no keys. + */ +import { describe, expect, it } from 'bun:test'; +import { generateText } from 'ai'; +import { MockLanguageModelV3 } from 'ai/test'; +import { repairToolPairing, toModelMessages, type ChatMessage } from '../../src/core/ai/gateway.ts'; + +function mockModel(): MockLanguageModelV3 { + return new MockLanguageModelV3({ + doGenerate: async () => ({ + content: [{ type: 'text', text: 'ok' }], + finishReason: 'stop', + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + warnings: [], + }), + } as any); +} + +describe('repairToolPairing', () => { + it('is a no-op on a balanced history', () => { + const msgs: ChatMessage[] = [ + { role: 'user', content: 'hi' }, + { role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }] }, + { role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { ok: 1 } }] }, + ]; + expect(repairToolPairing(msgs)).toEqual(msgs); + }); + + it('synthesizes a tool-result turn when the assistant tool-call is fully unanswered', () => { + const msgs: ChatMessage[] = [ + { role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }] }, + ]; + const out = repairToolPairing(msgs); + expect(out).toHaveLength(2); + expect(out[1].role).toBe('user'); + const block = (out[1].content as any[])[0]; + expect(block).toMatchObject({ type: 'tool-result', toolCallId: 'c1', isError: true }); + }); + + it('merges stubs into a PARTIALLY-answered turn without duplicating the answered id', () => { + const msgs: ChatMessage[] = [ + { role: 'assistant', content: [ + { type: 'tool-call', toolCallId: 'a', toolName: 'search', input: {} }, + { type: 'tool-call', toolCallId: 'b', toolName: 'search', input: {} }, + ] }, + { role: 'user', content: [{ type: 'tool-result', toolCallId: 'a', toolName: 'search', output: { ok: 1 } }] }, + ]; + const out = repairToolPairing(msgs); + expect(out).toHaveLength(2); // merged in place, not appended + const ids = (out[1].content as any[]).map((x) => x.toolCallId); + expect(ids).toEqual(['a', 'b']); // 'a' kept once, 'b' back-filled + expect((out[1].content as any[]).filter((x) => x.toolCallId === 'a')).toHaveLength(1); + }); +}); + +describe('real AI SDK v6 validation', () => { + it('an unbalanced history passes generateText after repairToolPairing', async () => { + const model = mockModel(); + const unbalanced: ChatMessage[] = [ + { role: 'user', content: 'go' }, + { role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }] }, + // no tool-result turn + ]; + const result = await generateText({ + model: model as any, + messages: toModelMessages(repairToolPairing(unbalanced)) as any, + }); + expect(result.text).toBe('ok'); + const prompt = model.doGenerateCalls[0]!.prompt as any[]; + expect(prompt.some((m) => m.role === 'tool')).toBe(true); // stub promoted to tool role + }); + + it('a Date-bearing tool-result passes after toModelMessages ISO-izes it; a raw Date is rejected', async () => { + const withDate: ChatMessage[] = [ + { role: 'user', content: 'go' }, + { role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'brain_get_page', input: {} }] }, + { role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'brain_get_page', output: { updated_at: new Date('2026-06-26T06:56:59.000Z') } }] }, + ]; + // Fixed path: converted history validates. + await expect(generateText({ model: mockModel() as any, messages: toModelMessages(withDate) as any })).resolves.toBeDefined(); + + // Control: a raw Date placed straight into a v6 ModelMessage json value is rejected. + const rawDateMessages = [ + { role: 'user', content: 'go' }, + { role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'brain_get_page', input: {} }] }, + { role: 'tool', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'brain_get_page', output: { type: 'json', value: { updated_at: new Date('2026-06-26T06:56:59.000Z') } } }] }, + ]; + await expect(generateText({ model: mockModel() as any, messages: rawDateMessages as any })).rejects.toThrow(); + }); +}); diff --git a/test/ai/gateway-tools-schema.test.ts b/test/ai/gateway-tools-schema.test.ts index 83d4b4a7b..f5c57790c 100644 --- a/test/ai/gateway-tools-schema.test.ts +++ b/test/ai/gateway-tools-schema.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from 'bun:test'; import { generateText, jsonSchema } from 'ai'; import { MockLanguageModelV3 } from 'ai/test'; -import { toModelMessages, type ChatMessage } from '../../src/core/ai/gateway.ts'; +import { toAISDKTools, toModelMessages, type ChatMessage } from '../../src/core/ai/gateway.ts'; // v0.42 AI SDK v6 fix — the regression guard that the original bug evaded. // Every gateway/toolLoop test stubs the chat transport, which short-circuits @@ -36,13 +36,13 @@ const internalMessages: ChatMessage[] = [ describe('gateway tool schema + message shape (real AI SDK v6)', () => { it('jsonSchema()-wrapped tools + adapted messages pass generateText without throwing', async () => { const model = mockModel(); - // Built exactly as gateway.chat() builds it (the primary fix). - const tools = { - search: { + const tools = toAISDKTools([ + { + name: 'search', description: 'search the brain', - inputSchema: jsonSchema({ type: 'object', properties: { q: { type: 'string' } } } as any), + inputSchema: { type: 'object', properties: { q: { type: 'string' } } }, }, - }; + ]); const result = await generateText({ model: model as any, diff --git a/test/autopilot-lock.test.ts b/test/autopilot-lock.test.ts new file mode 100644 index 000000000..f3795ec5a --- /dev/null +++ b/test/autopilot-lock.test.ts @@ -0,0 +1,58 @@ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { decideLockAcquisition, isPidAlive } from '../src/commands/autopilot.ts'; + +let tmp: string; +let lockPath: string; +beforeEach(() => { + tmp = mkdtempSync(join(tmpdir(), 'gbrain-autopilot-lock-')); + lockPath = join(tmp, 'autopilot.lock'); +}); + +afterEach(() => { + rmSync(tmp, { recursive: true, force: true }); +}); + +describe('isPidAlive', () => { + test('returns true for the current process', () => { + expect(isPidAlive(process.pid)).toBe(true); + }); + + test('returns false for invalid process ids', () => { + expect(isPidAlive(0)).toBe(false); + expect(isPidAlive(-1)).toBe(false); + expect(isPidAlive(Number.NaN)).toBe(false); + expect(isPidAlive(Number.POSITIVE_INFINITY)).toBe(false); + }); +}); + +describe('decideLockAcquisition', () => { + test('acquires when no lock exists', () => { + expect(decideLockAcquisition(lockPath, process.pid)).toEqual({ action: 'acquire' }); + }); + + test('takes over a lock whose holder is dead', () => { + writeFileSync(lockPath, '4194303'); + expect(decideLockAcquisition(lockPath, process.pid)).toEqual({ + action: 'takeover', + reason: 'dead pid 4194303', + }); + }); + + test('keeps a lock whose holder is alive regardless of age', () => { + writeFileSync(lockPath, String(process.pid)); + expect(decideLockAcquisition(lockPath, process.pid + 100_000)).toEqual({ + action: 'exit', + holderPid: process.pid, + }); + }); + + test('takes over malformed and empty locks', () => { + writeFileSync(lockPath, 'not-a-pid'); + expect(decideLockAcquisition(lockPath, process.pid).action).toBe('takeover'); + writeFileSync(lockPath, ''); + expect(decideLockAcquisition(lockPath, process.pid).action).toBe('takeover'); + }); +}); diff --git a/test/commands/schema-packpath.test.ts b/test/commands/schema-packpath.test.ts new file mode 100644 index 000000000..fffb2aeaa --- /dev/null +++ b/test/commands/schema-packpath.test.ts @@ -0,0 +1,50 @@ +import { afterEach, describe, expect, test } from 'bun:test'; +import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { _testHelpers } from '../../src/commands/schema.ts'; +import { withEnv } from '../helpers/with-env.ts'; + +const tempDirs: string[] = []; + +function tempDir(prefix: string): string { + const dir = mkdtempSync(join(tmpdir(), prefix)); + tempDirs.push(dir); + return dir; +} + +afterEach(() => { + for (const dir of tempDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +describe('schema packPathByName', () => { + test('resolves all bundled schema pack names to bundled YAML files', () => { + for (const name of ['gbrain-base', 'gbrain-recommended', 'gbrain-base-v2']) { + const path = _testHelpers.packPathByName(name); + expect(path).toBeTruthy(); + expect(path!.endsWith(`src/core/schema-pack/base/${name}.yaml`)).toBe(true); + expect(existsSync(path!)).toBe(true); + } + }); + + test('returns null for an unknown non-bundled pack name', async () => { + const home = tempDir('gbrain-packpath-home-'); + await withEnv({ GBRAIN_HOME: home }, async () => { + expect(_testHelpers.packPathByName('definitely-not-a-pack')).toBeNull(); + }); + }); + + test('resolves a user-installed pack by name', async () => { + const home = tempDir('gbrain-packpath-home-'); + await withEnv({ GBRAIN_HOME: home }, async () => { + const packDir = join(home, '.gbrain', 'schema-packs', 'custom-pack'); + mkdirSync(packDir, { recursive: true }); + const packPath = join(packDir, 'pack.yaml'); + writeFileSync(packPath, 'name: custom-pack\n', 'utf-8'); + + expect(_testHelpers.packPathByName('custom-pack')).toBe(packPath); + }); + }); +}); diff --git a/test/config-set.test.ts b/test/config-set.test.ts index 8d353c838..4c0c0b372 100644 --- a/test/config-set.test.ts +++ b/test/config-set.test.ts @@ -38,6 +38,15 @@ describe('KNOWN_CONFIG_KEYS', () => { expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd'); }); + test('includes the gateway-loop toggle and provider API keys the wave wires', () => { + // The subagent handler's error message tells users to run + // `gbrain config set agent.use_gateway_loop true`; it must be a known key + // or `config set` rejects the wave's own enable command without --force. + expect(KNOWN_CONFIG_KEYS).toContain('agent.use_gateway_loop'); + expect(KNOWN_CONFIG_KEYS).toContain('openrouter_api_key'); + expect(KNOWN_CONFIG_KEYS).toContain('zeroentropy_api_key'); + }); + test('no duplicate entries', () => { const set = new Set(KNOWN_CONFIG_KEYS); expect(set.size).toBe(KNOWN_CONFIG_KEYS.length); diff --git a/test/config.test.ts b/test/config.test.ts index 9ea97d858..cdfcbf116 100644 --- a/test/config.test.ts +++ b/test/config.test.ts @@ -253,3 +253,15 @@ describe('loadConfig — GBRAIN_MAX_MARKUP_RATIO env (v0.42 #1699)', () => { }); }); }); + +describe('KNOWN_CONFIG_KEYS — documented enable commands must be registered', () => { + test('Life Chronicle keys are registered (v0.42.56.0 release notes say `config set auto_chronicle true`)', async () => { + const { KNOWN_CONFIG_KEYS, KNOWN_CONFIG_KEY_PREFIXES } = await import('../src/core/config.ts'); + // The flag the chronicle backstop reads (isAutoChronicleEnabled). + expect(KNOWN_CONFIG_KEYS).toContain('auto_chronicle'); + // The takes bootstrap two-gate consent flag (v0.41.18.0 A12). + expect(KNOWN_CONFIG_KEYS).toContain('takes.bootstrap_enabled'); + // chronicle.tz (chronicleTz) + future chronicle.* knobs. + expect(KNOWN_CONFIG_KEY_PREFIXES.some(p => 'chronicle.tz'.startsWith(p))).toBe(true); + }); +}); diff --git a/test/cross-modal-phase1.test.ts b/test/cross-modal-phase1.test.ts index 4f5243d96..19689a324 100644 --- a/test/cross-modal-phase1.test.ts +++ b/test/cross-modal-phase1.test.ts @@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { return resolveSearchMode({ mode: 'balanced' }); } - test('KNOBS_HASH_VERSION is 11 (cross-modal still appended; 10→11 asymmetric input_type fix)', () => { + test('KNOBS_HASH_VERSION is 12 (cross-modal still appended; 11→12 hard-exclude fold #2825)', () => { // v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3 // with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) + // v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields. @@ -145,8 +145,8 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { // T2: 6→7 title_boost. v0.42.3.0: 7→8 autocut. issue #1777: 8→9 archive/ demote. // v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type // finally reaches asymmetric providers — pre-fix rows were keyed on - // document-side query vectors. - expect(KNOBS_HASH_VERSION).toBe(11); + // document-side query vectors. #2825: 11→12 hard-exclude fold (hx=). + expect(KNOBS_HASH_VERSION).toBe(12); }); test('flipping unified_multimodal changes the hash', () => { diff --git a/test/doctor-hidden-by-search-policy.test.ts b/test/doctor-hidden-by-search-policy.test.ts index 47b28fb0f..b2ed5d4b6 100644 --- a/test/doctor-hidden-by-search-policy.test.ts +++ b/test/doctor-hidden-by-search-policy.test.ts @@ -14,6 +14,7 @@ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts'; import { resetPgliteState } from './helpers/reset-pglite.ts'; import { withEnv } from './helpers/with-env.ts'; import { checkHiddenBySearchPolicy } from '../src/commands/doctor.ts'; @@ -58,6 +59,23 @@ async function seed( } beforeAll(async () => { + // Pin the embedding dim to 1536 BEFORE initSchema. basisEmbedding() + // hardcodes Float32Array(1536) vectors, but initSchema sizes vector + // columns from process-global gateway state (getEmbeddingDimensions(), + // default 1280 = zeroentropyai). Whether this file passes therefore + // depended on which test files happened to run before it in the shard: a + // predecessor that leaves the gateway configured without dims (or a bare + // CI env) yields vector(1280) and every upsertChunks here dies with + // "expected 1280 dimensions, not 1536". Adding test files to the repo + // reshuffles the weight-packed shards, so unrelated PRs trip it (seen on + // #2800 CI, test (1)). Same fix + rationale as + // engine-find-trajectory.test.ts and cosine-rescore-column.test.ts, which + // document this exact class. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test-hidden-by-search-policy' }, + }); engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); @@ -65,6 +83,7 @@ beforeAll(async () => { afterAll(async () => { await engine.disconnect(); + resetGateway(); }); beforeEach(async () => { diff --git a/test/e2e/mechanical.test.ts b/test/e2e/mechanical.test.ts index 5428e1d27..8d6c52568 100644 --- a/test/e2e/mechanical.test.ts +++ b/test/e2e/mechanical.test.ts @@ -582,6 +582,11 @@ describeE2E('E2E: Files', () => { const files = await callOp('file_list', {}) as any[]; expect(files.length).toBe(1); + // Regression: Postgres BIGINT(size_bytes) returned native BigInt before + // v0.22.5 so the MCP serializer threw and CLI listFiles div-by-1024 threw. + expect(typeof files[0].size_bytes).toBe('number'); + expect(() => JSON.stringify(files)).not.toThrow(); + // Verify file_url returns URI format const url = await callOp('file_url', { storage_path: result.storage_path }) as any; expect(url.url).toContain('gbrain:files/'); diff --git a/test/e2e/subagent-gateway-resume-reconciliation.test.ts b/test/e2e/subagent-gateway-resume-reconciliation.test.ts new file mode 100644 index 000000000..d7c479a3a --- /dev/null +++ b/test/e2e/subagent-gateway-resume-reconciliation.test.ts @@ -0,0 +1,280 @@ +/** + * E2E: gateway-loop resume reconciliation (fix-wave A). + * + * The gateway-native subagent loop persists each assistant turn but historically + * never persisted the following tool-result user turn. A resumed job therefore + * reloaded assistant tool-calls with no matching tool-result, and non-Anthropic + * (openai-compat) providers reject that unbalanced history with + * AI_MissingToolResultsError — dead-lettering the job. This wave: + * 1. forward-persists the tool-result user turn (onToolResultTurn), and + * 2. reconciles an already-corrupted transcript on resume by rebuilding the + * missing tool-result turns from settled subagent_tool_executions, + * re-dispatching idempotent-pending tools and throwing on non-idempotent. + * + * Hermetic: PGLite in-memory engine, gateway transport stubbed. Seeds use the + * sanctioned `$N::text::jsonb` positional bind (NEVER JSON.stringify into a + * bare `::jsonb`) so the seed is Postgres-safe too. + * + * Supersedes the resume/replay work in #1934 #2062 #2065 #2112 #2274 #2487 + * #2802 #2336 #2257 #2499. + */ + +import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { makeSubagentHandler } from '../../src/core/minions/handlers/subagent.ts'; +import type { MinionJobContext, ToolDef, ToolCtx } from '../../src/core/minions/types.ts'; +import { + __setChatTransportForTests, + configureGateway, + resetGateway, + toModelMessages, + type ChatBlock, + type ChatMessage, + type ChatResult, +} from '../../src/core/ai/gateway.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); +afterAll(async () => { + __setChatTransportForTests(null); + resetGateway(); + await engine.disconnect(); +}); +beforeEach(async () => { + await resetPgliteState(engine); + await engine.setConfig('version', '85'); + await engine.setConfig('agent.use_gateway_loop', 'true'); + configureGateway({ + chat_model: 'anthropic:claude-sonnet-4-6', + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + expansion_model: 'anthropic:claude-haiku-4-5', + env: { ANTHROPIC_API_KEY: 'stub', OPENAI_API_KEY: 'stub' }, + }); +}); + +async function makeJob(prompt: string, model: string): Promise<{ jobId: number; ctx: MinionJobContext }> { + const rows = await engine.executeRaw<{ id: number }>( + `INSERT INTO minion_jobs (name, status, data, queue, priority, created_at) + VALUES ('subagent', 'active', $1::text::jsonb, 'default', 0, now()) RETURNING id`, + [JSON.stringify({ prompt, model })], + ); + const jobId = rows[0].id; + const ctx: MinionJobContext = { + id: jobId, name: 'subagent', data: { prompt, model }, attempts_made: 1, + signal: new AbortController().signal, shutdownSignal: new AbortController().signal, + updateProgress: async () => {}, updateTokens: async () => {}, log: async () => {}, + isActive: async () => true, readInbox: async () => [], + }; + return { jobId, ctx }; +} + +function makeTools(executions: string[]): ToolDef[] { + return [ + { name: 'search', description: 's', input_schema: { type: 'object' }, idempotent: true, + async execute(input: unknown, _c: ToolCtx) { executions.push('search'); return { results: ['fresh'] }; } }, + { name: 'put_page', description: 'p', input_schema: { type: 'object' }, idempotent: false, + async execute(_input: unknown, _c: ToolCtx) { executions.push('put_page'); return { saved: true }; } }, + ]; +} + +function buildHandler(toolRegistry: ToolDef[]) { + return makeSubagentHandler({ + engine, config: {} as any, toolRegistry, + makeAnthropic: () => ({ messages: { create: async () => { throw new Error('legacy path unused'); } } }) as any, + }); +} + +async function seedMessage(jobId: number, idx: number, role: string, blocks: ChatBlock[]): Promise<void> { + await engine.executeRaw( + `INSERT INTO subagent_messages (job_id, message_idx, role, content_blocks, schema_version) + VALUES ($1, $2, $3, $4::text::jsonb, 2)`, + [jobId, idx, role, JSON.stringify(blocks)], + ); +} + +async function seedExec(jobId: number, msgIdx: number, toolUseId: string, name: string, status: string, output: unknown, ordinal: number, error?: string): Promise<void> { + await engine.executeRaw( + `INSERT INTO subagent_tool_executions + (job_id, message_idx, tool_use_id, tool_name, input, status, output, error, schema_version, ordinal) + VALUES ($1, $2, $3, $4, $5::text::jsonb, $6, $7::text::jsonb, $8, 2, $9)`, + [jobId, msgIdx, toolUseId, name, JSON.stringify({}), status, output == null ? null : JSON.stringify(output), error ?? null, ordinal], + ); +} + +/** Assert every assistant tool-call turn is answered by a following tool-result. */ +function assertBalanced(messages: ChatMessage[]): void { + for (let i = 0; i < messages.length; i++) { + const m = messages[i]; + if (m.role !== 'assistant' || typeof m.content === 'string') continue; + const calls = m.content.filter((b): b is Extract<ChatBlock, { type: 'tool-call' }> => b.type === 'tool-call'); + if (calls.length === 0) continue; + const next = messages[i + 1]; + expect(next, `assistant tool-call turn at ${i} must be followed by a tool-result turn`).toBeDefined(); + const answered = new Set( + (typeof next!.content === 'string' ? [] : next!.content) + .filter((b): b is Extract<ChatBlock, { type: 'tool-result' }> => b.type === 'tool-result') + .map(b => b.toolCallId), + ); + for (const c of calls) expect(answered.has(c.toolCallId), `tool-call ${c.toolCallId} unanswered`).toBe(true); + } + // The real provider path: this must not throw the ModelMessage schema error. + expect(() => toModelMessages(messages)).not.toThrow(); +} + +describe('gateway resume reconciliation', () => { + it('forward-persists the tool-result user turn (idx 2) in a 2-turn flow', async () => { + let turn = 0; + __setChatTransportForTests(async () => { + turn++; + if (turn === 1) return { + text: '', blocks: [{ type: 'tool-call', toolCallId: 'tc1', toolName: 'search', input: { q: 'x' } }] as ChatBlock[], + stopReason: 'tool_calls', usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'anthropic:claude-sonnet-4-6', providerId: 'anthropic', + } satisfies ChatResult; + return { + text: 'done', blocks: [{ type: 'text', text: 'done' }] as ChatBlock[], + stopReason: 'end', usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'anthropic:claude-sonnet-4-6', providerId: 'anthropic', + } satisfies ChatResult; + }); + const { jobId, ctx } = await makeJob('go', 'anthropic:claude-sonnet-4-6'); + await buildHandler(makeTools([]))(ctx); + + const msgs = await engine.executeRaw<{ message_idx: number; role: string; content_blocks: unknown }>( + `SELECT message_idx, role, content_blocks FROM subagent_messages WHERE job_id = $1 ORDER BY message_idx`, [jobId]); + expect(msgs.map(m => [m.message_idx, m.role])).toEqual([[0, 'user'], [1, 'assistant'], [2, 'user'], [3, 'assistant']]); + const toolResultTurn = typeof msgs[2].content_blocks === 'string' ? JSON.parse(msgs[2].content_blocks as string) : msgs[2].content_blocks; + expect((toolResultTurn as any[])[0].type).toBe('tool-result'); + expect((toolResultTurn as any[])[0].toolCallId).toBe('tc1'); + }); + + it('self-heals a pre-fix corrupted job from the stored output (no re-execute), transcript balanced', async () => { + const { jobId, ctx } = await makeJob('resume me', 'openai:gpt-4o'); // non-Anthropic: strict pairing + // Corrupted pre-fix state: seed user + assistant(tool-call), a complete + // exec row, but NO tool-result user turn at idx 2. + await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'resume me' }]); + await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'prov-tc-1', toolName: 'search', input: { q: 'x' } }]); + await seedExec(jobId, 1, 'prov-tc-1', 'search', 'complete', { results: ['from-prior-run'] }, 0); + + let captured: ChatMessage[] = []; + __setChatTransportForTests(async (opts) => { + captured = opts.messages; + return { + text: 'recovered and done', blocks: [{ type: 'text', text: 'recovered and done' }] as ChatBlock[], + stopReason: 'end', usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'openai:gpt-4o', providerId: 'openai', + } satisfies ChatResult; + }); + const executions: string[] = []; + const result = await buildHandler(makeTools(executions))(ctx); + + expect(result.result).toBe('recovered and done'); + expect(executions.length).toBe(0); // stored output reused, tool NOT re-run + assertBalanced(captured); + // The healed tool-result carries the real stored output. + const healed = (captured[2].content as ChatBlock[])[0] as Extract<ChatBlock, { type: 'tool-result' }>; + expect(healed.output).toEqual({ results: ['from-prior-run'] }); + + // Durably persisted so the next resume stays balanced. + const msgs = await engine.executeRaw<{ message_idx: number; role: string }>( + `SELECT message_idx, role FROM subagent_messages WHERE job_id = $1 ORDER BY message_idx`, [jobId]); + expect(msgs.map(m => [m.message_idx, m.role])).toEqual([[0, 'user'], [1, 'assistant'], [2, 'user'], [3, 'assistant']]); + }); + + it('heals MULTIPLE consecutive dangling assistant turns (pre-fix multi-turn corruption)', async () => { + const { jobId, ctx } = await makeJob('multi', 'openai:gpt-4o'); + // Pre-fix loop persisted assistants at 1 and 3 (gaps at 2 = skipped user idx). + await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'multi' }]); + await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-a', toolName: 'search', input: {} }]); + await seedExec(jobId, 1, 'tc-a', 'search', 'complete', { results: ['a'] }, 0); + await seedMessage(jobId, 3, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-b', toolName: 'search', input: {} }]); + await seedExec(jobId, 3, 'tc-b', 'search', 'complete', { results: ['b'] }, 0); + + let captured: ChatMessage[] = []; + __setChatTransportForTests(async (opts) => { + captured = opts.messages; + return { text: 'ok', blocks: [{ type: 'text', text: 'ok' }] as ChatBlock[], stopReason: 'end', + usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult; + }); + const executions: string[] = []; + await buildHandler(makeTools(executions))(ctx); + + expect(executions.length).toBe(0); + assertBalanced(captured); + // Both dangling turns healed and persisted (idx 2 and 4). + const msgs = await engine.executeRaw<{ message_idx: number; role: string }>( + `SELECT message_idx, role FROM subagent_messages WHERE job_id = $1 ORDER BY message_idx`, [jobId]); + expect(msgs.map(m => m.message_idx)).toContain(2); + expect(msgs.map(m => m.message_idx)).toContain(4); + }); + + it('re-dispatches an idempotent tool that was still pending on resume', async () => { + const { jobId, ctx } = await makeJob('redispatch', 'openai:gpt-4o'); + await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'redispatch' }]); + await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-pending', toolName: 'search', input: {} }]); + await seedExec(jobId, 1, 'tc-pending', 'search', 'pending', null, 0); + + __setChatTransportForTests(async () => ({ text: 'ok', blocks: [{ type: 'text', text: 'ok' }] as ChatBlock[], stopReason: 'end', + usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult)); + const executions: string[] = []; + await buildHandler(makeTools(executions))(ctx); + expect(executions).toEqual(['search']); // idempotent-pending re-executed once + }); + + it('throws on a non-idempotent tool still pending on resume', async () => { + const { jobId, ctx } = await makeJob('unsafe', 'openai:gpt-4o'); + await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'unsafe' }]); + await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-mut', toolName: 'put_page', input: {} }]); + await seedExec(jobId, 1, 'tc-mut', 'put_page', 'pending', null, 0); + + __setChatTransportForTests(async () => ({ text: '', blocks: [] as ChatBlock[], stopReason: 'end', + usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult)); + await expect(buildHandler(makeTools([]))(ctx)).rejects.toThrow(/non-idempotent tool "put_page" pending on resume/i); + }); + + it('error-stubs a dangling tool-call whose tool is no longer registered', async () => { + const { jobId, ctx } = await makeJob('gone tool', 'openai:gpt-4o'); + await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'gone tool' }]); + await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-gone', toolName: 'removed_tool', input: {} }]); + // No exec row and the tool isn't in the registry (only 'search'/'put_page'). + + let captured: ChatMessage[] = []; + __setChatTransportForTests(async (opts) => { + captured = opts.messages; + return { text: 'handled', blocks: [{ type: 'text', text: 'handled' }] as ChatBlock[], stopReason: 'end', + usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult; + }); + const result = await buildHandler(makeTools([]))(ctx); + expect(result.result).toBe('handled'); + assertBalanced(captured); + const stub = (captured[2].content as ChatBlock[])[0] as Extract<ChatBlock, { type: 'tool-result' }>; + expect(stub.isError).toBe(true); + expect(String(stub.output)).toContain('removed_tool'); + // Persisted as a failed exec so the next resume is stable. + const rows = await engine.executeRaw<{ status: string }>( + `SELECT status FROM subagent_tool_executions WHERE job_id = $1 AND tool_use_id = 'tc-gone'`, [jobId]); + expect(rows[0].status).toBe('failed'); + }); + + it('terminal resume: a completed transcript returns its text without calling the model', async () => { + const { jobId, ctx } = await makeJob('already done', 'openai:gpt-4o'); + await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'already done' }]); + await seedMessage(jobId, 1, 'assistant', [{ type: 'text', text: 'the final answer' }]); + + let chatCalls = 0; + __setChatTransportForTests(async () => { chatCalls++; return { text: 'SHOULD NOT RUN', blocks: [] as ChatBlock[], stopReason: 'end', + usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult; }); + const result = await buildHandler(makeTools([]))(ctx); + expect(chatCalls).toBe(0); + expect(result.result).toBe('the final answer'); + expect(result.stop_reason).toBe('end_turn'); + }); +}); diff --git a/test/extract-atoms-extractable-types.test.ts b/test/extract-atoms-extractable-types.test.ts new file mode 100644 index 000000000..4dacb2ccb --- /dev/null +++ b/test/extract-atoms-extractable-types.test.ts @@ -0,0 +1,39 @@ +/** + * Pack-driven extractable-type allowlist (unionExtractableTypes): honors the + * schema-pack manifest's `extractable: true` flags while preserving the legacy + * hardcoded floor and excluding synthesis outputs. Closes the D2 TODO in + * extract-atoms.ts (page discovery was ignoring the pack's extractable flag, so + * a type declared extractable — e.g. `note` — never actually extracted). + */ +import { describe, test, expect } from 'bun:test'; +import { unionExtractableTypes } from '../src/core/cycle/extract-atoms.ts'; + +const LEGACY = ['meeting', 'source', 'article', 'video', 'book', 'original']; + +describe('unionExtractableTypes', () => { + test('legacy floor is always present (back-compat)', () => { + const r = unionExtractableTypes([]); + for (const t of LEGACY) expect(r).toContain(t); + }); + + test('pack-declared extractable types are added (e.g. note)', () => { + const r = unionExtractableTypes(['note', 'writing']); + expect(r).toContain('note'); + expect(r).toContain('writing'); + for (const t of LEGACY) expect(r).toContain(t); + }); + + test('synthesis outputs are excluded even when the pack marks them extractable', () => { + // gbrain-base declares `concept` extractable:true, but extracting atoms FROM + // concepts would loop (concepts are synthesized from atoms). + const r = unionExtractableTypes(['note', 'concept', 'atom']); + expect(r).toContain('note'); + expect(r).not.toContain('concept'); + expect(r).not.toContain('atom'); + }); + + test('no duplicates when the pack repeats a legacy type', () => { + const r = unionExtractableTypes(['meeting', 'source']); + expect(r.filter((t) => t === 'meeting')).toHaveLength(1); + }); +}); diff --git a/test/extract-atoms-page-discovery.test.ts b/test/extract-atoms-page-discovery.test.ts index db459c64a..7289cc52e 100644 --- a/test/extract-atoms-page-discovery.test.ts +++ b/test/extract-atoms-page-discovery.test.ts @@ -50,7 +50,7 @@ function stubChat(text: string): (o: ChatOpts) => Promise<ChatResult> { /** * Stub that returns a unique-title atom on each call so atoms write to - * distinct slugs (`atoms/${date}/${slugify(title)}`) instead of upserting + * distinct slugs (`atoms/<source-date>/<stem>-<title-hash>`) instead of upserting * into one row. Needed for tests that count atoms after multiple work items. */ function stubChatUnique(): (o: ChatOpts) => Promise<ChatResult> { @@ -108,12 +108,15 @@ async function seedPage(opts: { } describe('v0.41.2.1: discoverExtractablePages SQL contract', () => { - test('filters by all 6 extractable types', async () => { - for (const type of ['meeting', 'source', 'article', 'video', 'book', 'original']) { + test('discovers legacy + pack-extractable types, excludes synthesis outputs', async () => { + // Legacy floor + `note` (declared extractable:true in gbrain-base, now + // honored via the pack manifest — the D2 fix). + for (const type of ['meeting', 'source', 'article', 'video', 'book', 'original', 'note']) { await seedPage({ slug: `${type}/x`, type }); } - // Add a non-extractable page that should NOT appear - await seedPage({ slug: 'notes/skip-me', type: 'note' }); + // `concept` is also extractable:true in gbrain-base, but extracting atoms + // FROM concepts would loop — synthesis outputs are always excluded. + await seedPage({ slug: 'wiki/concepts/skip-me', type: 'concept' }); const discovered = await discoverExtractablePages(engine, 'default'); const slugs = discovered.map((d) => d.slug).sort(); @@ -121,6 +124,7 @@ describe('v0.41.2.1: discoverExtractablePages SQL contract', () => { 'article/x', 'book/x', 'meeting/x', + 'note/x', 'original/x', 'source/x', 'video/x', @@ -339,6 +343,41 @@ describe('v0.41.2.1: runPhaseExtractAtoms — dual-source merge + idempotency', expect(after[0].count).toBe(before[0].count); }); + test('deterministic slug: source-dated + title-hashed, trailing dash stripped, re-extract upserts (no cross-day twin)', async () => { + // 16 three-letter words → slugifySegment output truncates ON a hyphen at the + // 60-char cut, exercising the trailing-dash re-strip (Bug A). + const title = 'aaa bbb ccc ddd eee fff ggg hhh iii jjj kkk lll mmm nnn ooo ppp'; + const chat = stubChat(`[{"title":"${title}","atom_type":"insight","body":"b"}]`); + // Transcript filename carries a date DIFFERENT from the run date, so a + // source-dated slug is observably distinct from the old run-date one. + const filePath = '/srv/transcripts/2026-06-12-telegram.md'; + await runPhaseExtractAtoms(engine, { + _transcripts: [{ filePath, content: 'first', contentHash: 'aaaa1111bbbb2222' }], + _pages: [], + _chat: chat, + }); + // Same file, GROWN content (append-only) → different contentHash, so the + // source-hash fast-path does NOT skip and the atom is re-extracted. Pre-fix + // this minted a second atom under a new run-date prefix (Bug B); the + // source-dated, title-hashed slug must upsert into the same row instead. + await runPhaseExtractAtoms(engine, { + _transcripts: [{ filePath, content: 'first plus appended', contentHash: 'cccc3333dddd4444' }], + _pages: [], + _chat: chat, + }); + const rows = await engine.executeRaw<{ slug: string }>( + `SELECT slug FROM pages WHERE type = 'atom'`, + ); + expect(rows.length).toBe(1); // upsert, not a cross-day duplicate + const slug = rows[0].slug; + expect(slug.startsWith('atoms/2026-06-12/')).toBe(true); // SOURCE date, not run date + expect(slug).toMatch(/-[0-9a-f]{6}$/); // 6-char title-hash suffix + expect(slug).not.toContain('--'); // trailing dash stripped before -<hash> + const stem = slug.slice('atoms/2026-06-12/'.length).replace(/-[0-9a-f]{6}$/, ''); + expect(stem.endsWith('-')).toBe(false); + expect(stem.length).toBeLessThanOrEqual(60); + }); + test('PhaseResult.details has additive page fields populated', async () => { const chat = stubChat(`[{"title":"x","atom_type":"insight","body":"b"}]`); const result = await runPhaseExtractAtoms(engine, { diff --git a/test/extract-takes-from-pages-progression.test.ts b/test/extract-takes-from-pages-progression.test.ts new file mode 100644 index 000000000..f8bc2dbc2 --- /dev/null +++ b/test/extract-takes-from-pages-progression.test.ts @@ -0,0 +1,87 @@ +/** + * Bootstrap progression regression test. + * + * extractTakesFromPages selected pages by updated_at DESC + LIMIT with no + * exclusion of pages that already hold takes — so on a corpus larger than + * one run's cap (the CLI clamps --max-pages to 1000), every re-run rescanned + * the same most-recent slice: the older tail could never be bootstrapped, + * and each rescan re-spent LLM budget producing upsert-identical rows. + * Seen live on a 2,311-eligible-page brain where the second run would have + * covered 0 new pages. + * + * Pins: covered pages are skipped by default (runs progress), and + * includeCovered restores the full rescan (refresh semantics). + */ +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + configureGateway, + resetGateway, + __setChatTransportForTests, +} from '../src/core/ai/gateway.ts'; +import { extractTakesFromPages } from '../src/core/extract-takes-from-pages.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + + configureGateway({ + chat_model: 'anthropic:claude-haiku-4-5-20251001', + env: { ANTHROPIC_API_KEY: 'sk-ant-test-takes-progression' }, + }); + __setChatTransportForTests(async () => ({ + text: '[{"claim":"a stubbed claim","kind":"take","weight":0.7}]', + blocks: [{ type: 'text' as const, text: '[{"claim":"a stubbed claim","kind":"take","weight":0.7}]' }], + stopReason: 'end' as const, + usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'anthropic:claude-haiku-4-5-20251001', + providerId: 'anthropic', + })); + + const body = 'An opinion-bearing body long enough to clear the 200-char eligibility floor. '.repeat(5); + await engine.putPage('concepts/progression-a', { + type: 'concept', title: 'A', compiled_truth: body, frontmatter: {}, + }); + await engine.putPage('concepts/progression-b', { + type: 'concept', title: 'B', compiled_truth: body, frontmatter: {}, + }); +}); + +afterAll(async () => { + __setChatTransportForTests(null); + resetGateway(); + await engine.disconnect(); +}); + +describe('extractTakesFromPages — bootstrap progression', () => { + test('first run covers the eligible pages', async () => { + const r1 = await extractTakesFromPages(engine, { bootstrapEnabled: true, maxPages: 50 }); + expect(r1.pages_scanned).toBe(2); + expect(r1.claims_extracted).toBe(2); + }); + + test('second run skips covered pages — repeat runs progress instead of rescanning', async () => { + const r2 = await extractTakesFromPages(engine, { bootstrapEnabled: true, maxPages: 50 }); + expect(r2.pages_scanned).toBe(0); + expect(r2.claims_extracted).toBe(0); + }); + + test('a page added after the first run is picked up (progression, not a frozen set)', async () => { + const body = 'Another opinion-bearing body long enough to clear the eligibility floor. '.repeat(5); + await engine.putPage('concepts/progression-c', { + type: 'concept', title: 'C', compiled_truth: body, frontmatter: {}, + }); + const r3 = await extractTakesFromPages(engine, { bootstrapEnabled: true, maxPages: 50 }); + expect(r3.pages_scanned).toBe(1); + }); + + test('includeCovered rescans everything (refresh semantics)', async () => { + const r4 = await extractTakesFromPages(engine, { + bootstrapEnabled: true, maxPages: 50, includeCovered: true, + }); + expect(r4.pages_scanned).toBe(3); + }); +}); diff --git a/test/extract.test.ts b/test/extract.test.ts index 1c9b15f14..5764de52b 100644 --- a/test/extract.test.ts +++ b/test/extract.test.ts @@ -135,6 +135,56 @@ describe('extractTimelineFromContent', () => { const entries = extractTimelineFromContent(content, 'test'); expect(entries).toHaveLength(1); }); + + it('extracts inline citation format entries', () => { + const content = `Closed the seed round with fund-a leading. [Source: board meeting notes, 2025-04-02]`; + const entries = extractTimelineFromContent(content, 'deals/acme-seed'); + expect(entries).toHaveLength(1); + expect(entries[0].date).toBe('2025-04-02'); + expect(entries[0].source).toBe('board meeting notes'); + expect(entries[0].summary).toBe('Closed the seed round with fund-a leading.'); + }); + + it('keeps commas inside the citation source', () => { + const content = `Alice joined as CTO. [Source: email from alice-example re: offer, signed, 2025-05-10]`; + const entries = extractTimelineFromContent(content, 'people/alice-example'); + expect(entries).toHaveLength(1); + expect(entries[0].date).toBe('2025-05-10'); + expect(entries[0].source).toBe('email from alice-example re: offer, signed'); + }); + + it('extracts one entry per citation when a line carries several', () => { + const content = `Both sides confirmed the partnership. [Source: call with widget-co, 2025-06-01] [Source: follow-up email, 2025-06-03]`; + const entries = extractTimelineFromContent(content, 'companies/widget-co'); + expect(entries).toHaveLength(2); + expect(entries[0].date).toBe('2025-06-01'); + expect(entries[1].date).toBe('2025-06-03'); + expect(entries[0].summary).toBe(entries[1].summary); + }); + + it('does not double-extract a timeline bullet that carries its own citation', () => { + const content = `- **2025-03-18** | Meeting — Discussed partnership [Source: meeting notes, 2025-03-18]`; + const entries = extractTimelineFromContent(content, 'test'); + expect(entries).toHaveLength(1); // Format 1 only + expect(entries[0].source).toBe('Meeting'); + }); + + it('skips a bare citation with no surrounding text', () => { + const content = `[Source: import batch, 2025-07-01]`; + expect(extractTimelineFromContent(content, 'test')).toHaveLength(0); + }); + + it('ignores citations without a date', () => { + const content = `Some claim here. [Source: undated memo]`; + expect(extractTimelineFromContent(content, 'test')).toHaveLength(0); + }); + + it('strips list markers from the citation summary', () => { + const content = `- Landed the enterprise pilot with acme-example. [Source: CRM update, 2025-08-15]`; + const entries = extractTimelineFromContent(content, 'companies/acme-example'); + expect(entries).toHaveLength(1); + expect(entries[0].summary).toBe('Landed the enterprise pilot with acme-example.'); + }); }); describe('walkMarkdownFiles', () => { diff --git a/test/fence-extraction.test.ts b/test/fence-extraction.test.ts index e65fe6ecd..366693ba2 100644 --- a/test/fence-extraction.test.ts +++ b/test/fence-extraction.test.ts @@ -136,4 +136,43 @@ echo hi const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code'); expect(fenceChunks.length).toBe(0); }); + + // #2437 — extractFencedChunks skips marked.lexer entirely when the body has + // no fence marker (the lexer transiently allocates ~60x the page size, which + // OOMs the import worker under memory pressure). These two guard that the + // fast-path neither breaks tilde fences nor lexes fence-less pages. + test('tilde-fenced (~~~) code is still extracted after the no-fence fast-path (#2437)', async () => { + const md = 'Docs.\n\n~~~ts\nexport const x = 1;\n~~~\n'; + await importFromContent(engine, 'guides/fence-tilde', md, { noEmbed: true }); + const chunks = await engine.getChunks('guides/fence-tilde'); + const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code'); + expect(fenceChunks.length).toBeGreaterThan(0); + expect(fenceChunks[0]!.language).toBe('typescript'); + }); + + test('CR-only (\\r) line endings still extract a fenced chunk (#2437)', async () => { + // marked normalizes \r → \n before lexing, so the fast-path's fence probe + // must too; otherwise a classic-Mac line-ended page loses its fenced code. + const md = 'intro\r```ts\rexport const x = 1;\r```\r'; + await importFromContent(engine, 'guides/fence-cr', md, { noEmbed: true }); + const chunks = await engine.getChunks('guides/fence-cr'); + const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code'); + expect(fenceChunks.length).toBeGreaterThan(0); + expect(fenceChunks[0]!.language).toBe('typescript'); + }); + + test('large fence-less table page imports with zero fenced chunks (no lexer pass) (#2437)', async () => { + const cols = 16; + const header = '| ' + Array.from({ length: cols }, (_, i) => 'col' + i).join(' | ') + ' |'; + const sep = '| ' + Array.from({ length: cols }, () => '---').join(' | ') + ' |'; + const body = Array.from({ length: 2000 }, (_, r) => + '| ' + Array.from({ length: cols }, (_, c) => 'v' + r + '_' + c).join(' | ') + ' |').join('\n'); + const md = '# Overview\n\n' + header + '\n' + sep + '\n' + body + '\n'; + // sanity: the page is genuinely fence-less, so the fast-path applies + expect(/(^|[\r\n])[ \t]{0,3}(```|~~~)/.test(md)).toBe(false); + await importFromContent(engine, 'guides/fence-less-table', md, { noEmbed: true }); + const chunks = await engine.getChunks('guides/fence-less-table'); + const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code'); + expect(fenceChunks.length).toBe(0); + }); }); diff --git a/test/files.test.ts b/test/files.test.ts index 596b45cba..8d8a7e58a 100644 --- a/test/files.test.ts +++ b/test/files.test.ts @@ -1,10 +1,12 @@ -import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { describe, test, expect, beforeAll, afterAll, spyOn } from 'bun:test'; import { writeFileSync, mkdirSync, rmSync, symlinkSync, mkdtempSync } from 'fs'; import { join, basename } from 'path'; import { createHash } from 'crypto'; import { extname } from 'path'; import { tmpdir } from 'os'; import { collectFiles } from '../src/commands/files.ts'; +import { operationsByName } from '../src/core/operations.ts'; +import * as db from '../src/core/db.ts'; const TMP = join(import.meta.dir, '.tmp-files-test'); @@ -183,6 +185,38 @@ describe('collectFiles (production import)', () => { } }); + test('file_list normalizes BigInt size_bytes for JSON serialization', async () => { + // Postgres BIGINT(size_bytes) returns native BigInt under postgres.js's + // {bigint: postgres.BigInt} type map. Both JSON.stringify (MCP) and the + // CLI's `size_bytes / 1024` divide trip on it. Regression for the bug + // openclaw's agent surfaced in v0.22.4. + const fakeRows = [ + { id: 1, page_slug: 'a', filename: 'f1', storage_path: 'a/f1', + mime_type: 'text/plain', size_bytes: 4096n, content_hash: 'h1', + created_at: '2026-04-27' }, + { id: 2, page_slug: 'a', filename: 'f2', storage_path: 'a/f2', + mime_type: null, size_bytes: null, content_hash: 'h2', + created_at: '2026-04-27' }, + ]; + const fakeSql: any = (..._: unknown[]) => Promise.resolve(fakeRows); + const spy = spyOn(db, 'getConnection').mockReturnValue(fakeSql); + + try { + const op = operationsByName['file_list']; + const ctx: any = { engine: null, config: {}, logger: { info() {}, warn() {}, error() {} }, dryRun: false, remote: true }; + const result = await op.handler(ctx, {}) as Array<Record<string, unknown>>; + + expect(result.length).toBe(2); + expect(typeof result[0].size_bytes).toBe('number'); + expect(result[0].size_bytes).toBe(4096); + expect(result[1].size_bytes).toBeNull(); + // The exact failure mode openclaw reported. + expect(() => JSON.stringify(result)).not.toThrow(); + } finally { + spy.mockRestore(); + } + }); + test('collectFiles skips node_modules', () => { const tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-nodemod-')); try { diff --git a/test/gateway-model-messages.test.ts b/test/gateway-model-messages.test.ts index 3fe3dde3d..8e56465c8 100644 --- a/test/gateway-model-messages.test.ts +++ b/test/gateway-model-messages.test.ts @@ -107,6 +107,63 @@ describe('toModelMessages — v6 ModelMessage shape', () => { ]); }); + test('Date in tool-result json output serializes to ISO string (Postgres timestamptz)', () => { + // node-postgres returns timestamptz columns as JS Date; AI SDK v6's + // JSONValue schema rejects a raw Date, dead-lettering the tool loop. + const msgs: ChatMessage[] = [ + { + role: 'user', + content: [{ + type: 'tool-result', + toolCallId: 'c1', + toolName: 'brain_get_page', + output: { rows: [{ updated_at: new Date('2026-06-26T06:56:59.000Z'), nested: { created_at: new Date('2026-01-02T03:04:05.000Z') } }] }, + }], + }, + ]; + const out = toModelMessages(msgs) as any[]; + const value = out[0].content[0].output.value; + expect(out[0].content[0].output.type).toBe('json'); + expect(value.rows[0].updated_at).toBe('2026-06-26T06:56:59.000Z'); + expect(value.rows[0].nested.created_at).toBe('2026-01-02T03:04:05.000Z'); + // No Date instance survives (would throw in AI SDK v6). + expect(value.rows[0].updated_at instanceof Date).toBe(false); + }); + + test('non-string text block is dropped (reasoning-model null-text guard)', () => { + // DeepSeek v4 / reasoning models can emit text:null/undefined thinking + // parts; AI SDK v6 rejects them. Dropped here; tool-call sibling kept. + const msgs: ChatMessage[] = [ + { + role: 'assistant', + content: [ + { type: 'text', text: null as unknown as string }, + { type: 'text', text: undefined as unknown as string }, + { type: 'text', text: 'kept' }, + { type: 'text', text: '' }, // empty string is valid — kept + { type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }, + ], + }, + ]; + const out = toModelMessages(msgs) as any[]; + expect(out[0].content).toEqual([ + { type: 'text', text: 'kept' }, + { type: 'text', text: '' }, + { type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }, + ]); + }); + + test('errored tool-result never throws on circular/bigint output (safeStringify)', () => { + const circular: any = {}; + circular.self = circular; + const msgs: ChatMessage[] = [ + { role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'x', output: circular, isError: true }] }, + ]; + const out = toModelMessages(msgs) as any[]; + expect(out[0].content[0].output.type).toBe('error-text'); + expect(typeof out[0].content[0].output.value).toBe('string'); + }); + test('full multi-turn conversation: user → assistant(tool-call) → tool(result)', () => { const msgs: ChatMessage[] = [ { role: 'user', content: 'find widget' }, diff --git a/test/import-image-file.test.ts b/test/import-image-file.test.ts index 8542e1d83..1fef8228b 100644 --- a/test/import-image-file.test.ts +++ b/test/import-image-file.test.ts @@ -120,6 +120,29 @@ describe('importImageFile happy path (noEmbed)', () => { expect(r2.status).toBe('skipped'); }); + test('routes page, chunks, and file metadata to the requested source (#2706)', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`, + ['image-source', 'Image Source'], + ); + const target = join(tmpDir, 'source-photo.png'); + writeFileSync(target, Buffer.from('fake-png-bytes-source-scoped')); + + const result = await importImageFile(engine, target, 'photos/source-photo.png', { + noEmbed: true, + sourceId: 'image-source', + }); + + expect(result.status).toBe('imported'); + expect(await engine.getPage('photos/source-photo.png', { sourceId: 'default' })).toBeNull(); + const page = await engine.getPage('photos/source-photo.png', { sourceId: 'image-source' }); + expect(page).not.toBeNull(); + expect(await engine.getChunks('photos/source-photo.png', { sourceId: 'default' })).toHaveLength(0); + expect(await engine.getChunks('photos/source-photo.png', { sourceId: 'image-source' })).toHaveLength(1); + expect(await engine.getFile('default', 'photos/source-photo.png')).toBeNull(); + expect(await engine.getFile('image-source', 'photos/source-photo.png')).not.toBeNull(); + }); + test('refuses oversized files (>20MB)', async () => { const target = join(tmpDir, 'huge.png'); // Write a 21MB file. Buffer.alloc is fast. diff --git a/test/link-extraction.test.ts b/test/link-extraction.test.ts index 0c180f232..e0a741220 100644 --- a/test/link-extraction.test.ts +++ b/test/link-extraction.test.ts @@ -1265,3 +1265,29 @@ describe("v0.18.0 migration v22 — links_resolution_type", () => { }); }); + +describe('parseTimelineEntries — Format 3: inline [Source: ..., YYYY-MM-DD] citations', () => { + test('extracts an entry from a dated citation', () => { + const entries = parseTimelineEntries('Closed the seed round. [Source: board notes, 2025-04-02]'); + expect(entries).toHaveLength(1); + expect(entries[0].date).toBe('2025-04-02'); + expect(entries[0].summary).toBe('Closed the seed round.'); + expect(entries[0].detail).toBe('Source: board notes'); + }); + + test('keeps commas inside the citation source', () => { + const entries = parseTimelineEntries('Alice joined. [Source: email re: offer, signed, 2025-05-10]'); + expect(entries).toHaveLength(1); + expect(entries[0].detail).toBe('Source: email re: offer, signed'); + }); + + test('does not double-extract a timeline bullet carrying its own citation', () => { + const entries = parseTimelineEntries('- **2025-03-18** | Meeting notes [Source: notes, 2025-03-18]'); + expect(entries).toHaveLength(1); // bullet pass only + }); + + test('skips invalid calendar dates and bare citations', () => { + expect(parseTimelineEntries('Claim. [Source: memo, 2026-13-45]')).toHaveLength(0); + expect(parseTimelineEntries('[Source: import batch, 2025-07-01]')).toHaveLength(0); + }); +}); diff --git a/test/model-pricing.test.ts b/test/model-pricing.test.ts index 40be89818..26da0100d 100644 --- a/test/model-pricing.test.ts +++ b/test/model-pricing.test.ts @@ -43,6 +43,14 @@ describe('CANONICAL_PRICING — table integrity', () => { expect(CANONICAL_PRICING['anthropic:claude-opus-4-7']).toEqual({ input: 5.0, output: 25.0 }); }); + test('Sonnet 5 present at $3/$15 (standard rate, intro discount not modeled)', () => { + expect(CANONICAL_PRICING['anthropic:claude-sonnet-5']).toEqual({ input: 3.0, output: 15.0 }); + }); + + test('Fable 5 present at $10/$50', () => { + expect(CANONICAL_PRICING['anthropic:claude-fable-5']).toEqual({ input: 10.0, output: 50.0 }); + }); + test('Gemini 2.0 Flash reconciled to $0.10/$0.40; legacy alias agrees', () => { expect(CANONICAL_PRICING['google:gemini-2.0-flash']).toEqual({ input: 0.1, output: 0.4 }); expect(CANONICAL_PRICING['google:gemini-2-flash']).toEqual( diff --git a/test/orphans-pure-fn.test.ts b/test/orphans-pure-fn.test.ts index 4906e9124..ada6a7d09 100644 --- a/test/orphans-pure-fn.test.ts +++ b/test/orphans-pure-fn.test.ts @@ -169,6 +169,7 @@ describe('shouldExclude — orphan filter regression (preserve curation)', () => test('raw segment is excluded', () => { expect(shouldExclude('media/x/raw/post')).toBe(true); + expect(shouldExclude('raw/chats/claude-code/session')).toBe(true); }); test('deny-prefixes are excluded', () => { @@ -183,6 +184,8 @@ describe('shouldExclude — orphan filter regression (preserve curation)', () => expect(shouldExclude('thoughts/today')).toBe(true); expect(shouldExclude('catalog/movies')).toBe(true); expect(shouldExclude('entities/anonymous')).toBe(true); + expect(shouldExclude('atoms/fact-123')).toBe(true); + expect(shouldExclude('skills/gbrain-operations')).toBe(true); }); test('regular slugs are NOT excluded', () => { diff --git a/test/query-cache-knobs-hash.serial.test.ts b/test/query-cache-knobs-hash.serial.test.ts index fdfc02745..d11d42ef3 100644 --- a/test/query-cache-knobs-hash.serial.test.ts +++ b/test/query-cache-knobs-hash.serial.test.ts @@ -19,6 +19,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { SemanticQueryCache, cacheRowId } from '../src/core/search/query-cache.ts'; import type { SearchResult } from '../src/core/types.ts'; import { knobsHash, resolveSearchMode } from '../src/core/search/mode.ts'; +import { resolveHardExcludes } from '../src/core/search/source-boost.ts'; import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts'; let engine: PGLiteEngine; @@ -230,3 +231,49 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => { expect(conservativeHit.hit).toBe(false); }); }); + +describe('hard-exclude cache isolation (#2825)', () => { + // Hashes computed the way hybridSearchCached does: same resolved mode, ctx + // carrying the resolved hard-exclude list. A row written by a process + // WITHOUT GBRAIN_SEARCH_EXCLUDE (defaults only) must not be served to a + // process WITH it, and vice versa. + const noEnvHash = knobsHash(resolveSearchMode({ mode: 'balanced' }), { + hardExcludes: resolveHardExcludes(undefined, undefined, undefined), + }); + const envExcludeHash = knobsHash(resolveSearchMode({ mode: 'balanced' }), { + hardExcludes: resolveHardExcludes(undefined, undefined, 'private/'), + }); + + test('row written without excludes is NOT served to a lookup with excludes', async () => { + const cache = new SemanticQueryCache(engine); + const emb = makeEmbedding(6); + + // Simulate a no-exclude process writing results that include a slug the + // excluding process must never see. + const leaky = makeResults('private', 5); + await cache.store('who is alice', emb, leaky, { + vector_enabled: true, detail_resolved: null, expansion_applied: false, + }, { knobsHash: noEnvHash }); + + // Excluding process → MISS (falls through to a fresh, filtered query). + const excluded = await cache.lookup(emb, { knobsHash: envExcludeHash }); + expect(excluded.hit).toBe(false); + + // Original no-exclude process still hits its own row. + const original = await cache.lookup(emb, { knobsHash: noEnvHash }); + expect(original.hit).toBe(true); + expect(original.results?.length).toBe(5); + }); + + test('row written WITH excludes is not served back once excludes are lifted', async () => { + const cache = new SemanticQueryCache(engine); + const emb = makeEmbedding(7); + + await cache.store('who is alice', emb, makeResults('filtered', 3), { + vector_enabled: true, detail_resolved: null, expansion_applied: false, + }, { knobsHash: envExcludeHash }); + + expect((await cache.lookup(emb, { knobsHash: noEnvHash })).hit).toBe(false); + expect((await cache.lookup(emb, { knobsHash: envExcludeHash })).hit).toBe(true); + }); +}); diff --git a/test/search-alias-resolved-boost.test.ts b/test/search-alias-resolved-boost.test.ts index 57fe7ace0..224d2d988 100644 --- a/test/search-alias-resolved-boost.test.ts +++ b/test/search-alias-resolved-boost.test.ts @@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => { }); describe('KNOBS_HASH_VERSION', () => { - it('is 11 (10→11 asymmetric input_type fix invalidates document-side query-vector rows, #1400)', () => { - expect(KNOBS_HASH_VERSION).toBe(11); + it('is 12 (11→12 hard-exclude fold invalidates rows written under a different exclude policy, #2825)', () => { + expect(KNOBS_HASH_VERSION).toBe(12); }); }); diff --git a/test/search-mode.test.ts b/test/search-mode.test.ts index a8e9ecf18..223d112d9 100644 --- a/test/search-mode.test.ts +++ b/test/search-mode.test.ts @@ -407,7 +407,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => { // now produces query-side vectors for asymmetric providers (zembed-1, // Voyage v3+), so rows keyed on pre-fix document-side query vectors // must not be served to post-fix lookups. - expect(KNOBS_HASH_VERSION).toBe(11); + // #2825: bumped 11→12 to fold the resolved hard-exclude prefix list + // (hx=) — cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across + // processes. + expect(KNOBS_HASH_VERSION).toBe(12); }); test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => { @@ -572,8 +575,8 @@ describe('v0.40.4 — graph_signals knob', () => { }); describe('v0.42.3.0 — autocut knobs', () => { - test('KNOBS_HASH_VERSION is 11 (10→11 asymmetric input_type fix, #1400)', () => { - expect(KNOBS_HASH_VERSION).toBe(11); + test('KNOBS_HASH_VERSION is 12 (11→12 hard-exclude fold, #2825)', () => { + expect(KNOBS_HASH_VERSION).toBe(12); }); test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => { diff --git a/test/search/knobs-hash-reranker.test.ts b/test/search/knobs-hash-reranker.test.ts index bd793d427..9f73ac394 100644 --- a/test/search/knobs-hash-reranker.test.ts +++ b/test/search/knobs-hash-reranker.test.ts @@ -27,6 +27,7 @@ import { MODE_BUNDLES, type ResolvedSearchKnobs, } from '../../src/core/search/mode.ts'; +import { resolveHardExcludes } from '../../src/core/search/source-boost.ts'; /** Build a baseline resolved knob set with all reranker fields filled. */ function baseKnobs(): ResolvedSearchKnobs { @@ -43,7 +44,7 @@ function baseKnobs(): ResolvedSearchKnobs { } describe('KNOBS_HASH_VERSION + version invariants', () => { - test('version is 11 (…; 8→9 archive-demote #1777; 9→10 relational recall; 10→11 asymmetric input_type #1400)', () => { + test('version is 12 (…; 9→10 relational recall; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825)', () => { // v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold // floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs // (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation). @@ -61,7 +62,9 @@ describe('KNOBS_HASH_VERSION + version invariants', () => { // #1400: 10→11 asymmetric input_type fix — embedQuery() now produces // query-side vectors for asymmetric providers, so rows keyed on // pre-fix document-side query vectors must not be served. - expect(KNOBS_HASH_VERSION).toBe(11); + // #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) — + // cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes. + expect(KNOBS_HASH_VERSION).toBe(12); }); test('hash is 16 hex chars regardless of reranker config', () => { @@ -208,3 +211,36 @@ describe('append-only convention (CDX2-F13)', () => { expect(bare).toBe(explicit); }); }); + +describe('v=12 hard-exclude participation (#2825)', () => { + test('different exclude lists → different hashes', () => { + const k = baseKnobs(); + const noEnv = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, undefined) }); + const withEnv = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, 'private/') }); + expect(noEnv).not.toBe(withEnv); + }); + + test('include (opt-back-in) changes the hash too', () => { + const k = baseKnobs(); + const a = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, undefined) }); + const b = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, ['test/'], undefined) }); + expect(a).not.toBe(b); + }); + + test('same prefixes in different input order → SAME hash (normalization)', () => { + const k = baseKnobs(); + const a = knobsHash(k, { hardExcludes: ['a/', 'b/', 'test/'] }); + const b = knobsHash(k, { hardExcludes: ['test/', 'b/', 'a/'] }); + expect(a).toBe(b); + }); + + test('undefined hardExcludes is stable (legacy-caller fallback)', () => { + const k = baseKnobs(); + expect(knobsHash(k)).toBe(knobsHash(k)); + // ...and distinct from an explicit resolved default list — a legacy + // caller can never collide with a policy-carrying cache row. + expect(knobsHash(k)).not.toBe( + knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, undefined) }), + ); + }); +}); diff --git a/test/serve-http-bootstrap-token.test.ts b/test/serve-http-bootstrap-token.test.ts index fcf6b6412..9b30fccd5 100644 --- a/test/serve-http-bootstrap-token.test.ts +++ b/test/serve-http-bootstrap-token.test.ts @@ -7,7 +7,7 @@ * the rule can't drift without the suite catching it. */ import { describe, test, expect } from 'bun:test'; -import { resolveBootstrapToken } from '../src/commands/serve-http.ts'; +import { resolveBootstrapToken, shouldSuppressBootstrapPrint } from '../src/commands/serve-http.ts'; describe('resolveBootstrapToken (v0.36.1.x #1024)', () => { test('unset env → generates a fresh token via the injected RNG', () => { @@ -72,3 +72,27 @@ describe('resolveBootstrapToken (v0.36.1.x #1024)', () => { expect(r.kind).toBe('error'); }); }); + +describe('shouldSuppressBootstrapPrint (#2624 log-leak default)', () => { + const base = { suppress: false, fromEnv: false, forcePrint: false, isTty: true }; + + test('generated token on non-TTY (container) → hidden by default', () => { + expect(shouldSuppressBootstrapPrint({ ...base, isTty: false })).toBe(true); + }); + + test('generated token on interactive TTY → printed', () => { + expect(shouldSuppressBootstrapPrint({ ...base, isTty: true })).toBe(false); + }); + + test('--print-admin-token forces raw value on non-TTY', () => { + expect(shouldSuppressBootstrapPrint({ ...base, isTty: false, forcePrint: true })).toBe(false); + }); + + test('env-sourced token is never printed', () => { + expect(shouldSuppressBootstrapPrint({ ...base, fromEnv: true, isTty: true })).toBe(true); + }); + + test('--suppress overrides even a forced print', () => { + expect(shouldSuppressBootstrapPrint({ ...base, suppress: true, forcePrint: true, isTty: true })).toBe(true); + }); +}); diff --git a/test/sync-reconcile-mass-delete.test.ts b/test/sync-reconcile-mass-delete.test.ts new file mode 100644 index 000000000..a0fff89bc --- /dev/null +++ b/test/sync-reconcile-mass-delete.test.ts @@ -0,0 +1,124 @@ +/** + * Test: full-sync reconcile separator normalization + mass-delete safety valve + * (#2828 — Windows reconcile mass-delete). + * + * Pure-helper surface: `planReconcileDeletes` and `massReconcileAllowed` take + * plain inputs and read no engine / no ambient env (the env reader is + * parameterized), so these run without PGLite and without touching the shared + * `process.env` — parallel-loop safe by construction. + */ + +import { describe, test, expect } from 'bun:test'; +import { + planReconcileDeletes, + massReconcileAllowed, + MASS_RECONCILE_RATIO, + MASS_RECONCILE_MIN_PAGES, +} from '../src/commands/sync.ts'; + +/** Build stored page rows from a list of source_paths (slug = `slug-<index>`). */ +function rows(paths: Array<string | null>): Array<{ slug: string; source_path: string | null }> { + return paths.map((p, i) => ({ slug: `slug-${i}`, source_path: p })); +} + +/** + * Reconcile scenario: `total` file-backed pages, of which the first `stale` + * are absent from the working tree (deleted) and the rest are present. + */ +function scenario(total: number, stale: number) { + const stored = rows(Array.from({ length: total }, (_, i) => `p/${i}.md`)); + const present = stored.slice(stale).map((r) => r.source_path as string); + return planReconcileDeletes(stored, present, () => true); +} + +describe('planReconcileDeletes — separator normalization (#2828)', () => { + test('backslash working-tree paths match forward-slash stored source_path', () => { + // Windows `path.relative` yields backslashes; source_path was stored with + // forward slashes (e.g. git-derived). Both must compare equal. + const stored = rows(['topics/foo.md', 'topics/bar.md', 'notes/baz.md']); + const workingTree = ['topics\\foo.md', 'topics\\bar.md', 'notes\\baz.md']; + const plan = planReconcileDeletes(stored, workingTree, () => true); + expect(plan.staleSlugs).toEqual([]); + expect(plan.reconcilableCount).toBe(3); + expect(plan.massDelete).toBe(false); + }); + + test('forward-slash working-tree paths match backslash stored source_path', () => { + const stored = rows(['topics\\foo.md', 'topics\\bar.md']); + const workingTree = ['topics/foo.md', 'topics/bar.md']; + const plan = planReconcileDeletes(stored, workingTree, () => true); + expect(plan.staleSlugs).toEqual([]); + }); + + test('a genuinely removed file is the only stale slug, regardless of separator', () => { + const stored = rows(['topics/foo.md', 'topics/bar.md', 'topics/gone.md']); + const workingTree = ['topics\\foo.md', 'topics\\bar.md']; // gone.md deleted + const plan = planReconcileDeletes(stored, workingTree, () => true); + expect(plan.staleSlugs).toEqual(['slug-2']); + }); + + test('null source_path rows are never reconcilable (manual / put_page pages)', () => { + const stored = rows([null, 'x.md']); + const plan = planReconcileDeletes(stored, [], () => true); + expect(plan.reconcilableCount).toBe(1); + expect(plan.staleSlugs).toEqual(['slug-1']); + }); + + test('the strategy predicate excludes wrong-strategy pages from both stale set and denominator', () => { + const stored = rows(['a.md', 'b.md', 'code.ts', 'gone.md']); + const onlyMarkdown = (p: string) => p.endsWith('.md'); + const plan = planReconcileDeletes(stored, [], onlyMarkdown); // nothing present + expect(plan.reconcilableCount).toBe(3); // code.ts excluded + expect(plan.staleSlugs).toEqual(['slug-0', 'slug-1', 'slug-3']); + }); +}); + +describe('planReconcileDeletes — mass-delete safety valve (#2828)', () => { + test('trips when > 50% of a > 20-page source would be deleted', () => { + const plan = scenario(21, 11); // 11/21 ≈ 52% > 50%, and 21 > 20 + expect(plan.reconcilableCount).toBe(21); + expect(plan.staleSlugs.length).toBe(11); + expect(plan.massDelete).toBe(true); + }); + + test('holds at exactly 50% (threshold is strictly greater)', () => { + const plan = scenario(40, 20); // 20/40 == 50%, not > 50% + expect(plan.massDelete).toBe(false); + }); + + test('ignores small sources (<= 20 pages) even at 100% stale', () => { + expect(scenario(MASS_RECONCILE_MIN_PAGES, MASS_RECONCILE_MIN_PAGES).massDelete).toBe(false); + expect(scenario(15, 15).massDelete).toBe(false); + }); + + test('trips just past the min-pages boundary with a majority stale', () => { + const plan = scenario(21, 20); + expect(plan.massDelete).toBe(true); + }); + + test('thresholds are the documented constants', () => { + expect(MASS_RECONCILE_RATIO).toBe(0.5); + expect(MASS_RECONCILE_MIN_PAGES).toBe(20); + }); +}); + +describe('massReconcileAllowed — GBRAIN_ALLOW_MASS_RECONCILE escape hatch (#2828)', () => { + test('=1 restores the old behavior', () => { + expect(massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: '1' })).toBe(true); + }); + + test('unset or any other value keeps the valve active', () => { + expect(massReconcileAllowed({})).toBe(false); + expect(massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: '0' })).toBe(false); + expect(massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: 'true' })).toBe(false); + }); + + test('effective gate: the valve blocks the delete unless the override is set', () => { + const plan = scenario(21, 11); + // This mirrors the guard in performFullSync. + const blocked = plan.massDelete && !massReconcileAllowed({}); + const overridden = plan.massDelete && !massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: '1' }); + expect(blocked).toBe(true); // skip delete + loud warning + expect(overridden).toBe(false); // old behavior restored + }); +}); diff --git a/test/takes-command-source-scope.test.ts b/test/takes-command-source-scope.test.ts new file mode 100644 index 000000000..9c88d54f4 --- /dev/null +++ b/test/takes-command-source-scope.test.ts @@ -0,0 +1,76 @@ +import { afterEach, describe, expect, test } from 'bun:test'; +import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { runTakes } from '../src/commands/takes.ts'; +import type { BrainEngine, TakeBatchInput } from '../src/core/engine.ts'; +import { withEnv } from './helpers/with-env.ts'; + +const tmpRoots: string[] = []; + +afterEach(() => { + for (const root of tmpRoots.splice(0)) { + rmSync(root, { recursive: true, force: true }); + } +}); + +function makeEngine() { + const added: TakeBatchInput[][] = []; + const pageLookups: unknown[][] = []; + const engine = { + getConfig: async () => null, + executeRaw: async (sql: string, params: unknown[] = []) => { + if (sql.includes('FROM sources WHERE id = $1')) { + return [{ id: params[0] as string }]; + } + if (sql.includes('FROM pages WHERE slug = $1 AND source_id = $2')) { + pageLookups.push(params); + if (params[0] === 'shared/page' && params[1] === 'dept') return [{ id: 22 }]; + if (params[0] === 'shared/page' && params[1] === 'default') return [{ id: 11 }]; + return []; + } + if (sql.includes('FROM pages WHERE slug = $1 LIMIT 1')) { + pageLookups.push(params); + return [{ id: 11 }]; + } + return []; + }, + addTakesBatch: async (rows: TakeBatchInput[]) => { + added.push(rows); + return rows.length; + }, + } as unknown as BrainEngine; + return { engine, added, pageLookups }; +} + +describe('gbrain takes CLI source scoping', () => { + test('add mirrors to the page in GBRAIN_SOURCE, not an arbitrary same-slug page (#2684)', async () => { + const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-takes-source-')); + const home = mkdtempSync(join(tmpdir(), 'gbrain-takes-home-')); + tmpRoots.push(brainDir, home); + const { engine, added, pageLookups } = makeEngine(); + + await withEnv({ GBRAIN_SOURCE: 'dept', GBRAIN_HOME: home }, async () => { + await runTakes(engine, [ + 'add', + 'shared/page', + '--claim', + 'Dept-scoped claim', + '--kind', + 'take', + '--who', + 'self', + '--dir', + brainDir, + ]); + }); + + expect(pageLookups).toEqual([['shared/page', 'dept']]); + expect(added).toHaveLength(1); + expect(added[0]![0]!.page_id).toBe(22); + + const written = join(brainDir, 'shared/page.md'); + expect(existsSync(written)).toBe(true); + expect(readFileSync(written, 'utf-8')).toContain('Dept-scoped claim'); + }); +}); diff --git a/test/think-max-output-tokens.test.ts b/test/think-max-output-tokens.test.ts new file mode 100644 index 000000000..879217ad6 --- /dev/null +++ b/test/think-max-output-tokens.test.ts @@ -0,0 +1,28 @@ +/** + * Pins `maxOutputTokensFor` — the per-model output-token budget `runThink` + * passes to `client.create`. Thinking-by-default Claude 5 models + * (`anthropic:claude-*-5`) spend a large share of the budget on internal + * reasoning before emitting an answer, so the 4000 default left `think` with + * empty/truncated text. They now get 16000; everything else stays 4000. + */ +import { describe, test, expect } from 'bun:test'; +import { maxOutputTokensFor } from '../src/core/think/index.ts'; + +describe('maxOutputTokensFor — thinking-default headroom', () => { + test('Claude 5 family gets 16000', () => { + expect(maxOutputTokensFor('anthropic:claude-sonnet-5')).toBe(16000); + expect(maxOutputTokensFor('anthropic:claude-opus-5')).toBe(16000); + expect(maxOutputTokensFor('anthropic:claude-fable-5')).toBe(16000); + expect(maxOutputTokensFor('anthropic:claude-haiku-5')).toBe(16000); + expect(maxOutputTokensFor('anthropic/claude-sonnet-5')).toBe(16000); // slash form + }); + + test('non-Claude-5 and non-Anthropic keep 4000', () => { + expect(maxOutputTokensFor('anthropic:claude-opus-4-8')).toBe(4000); + expect(maxOutputTokensFor('anthropic:claude-haiku-4-5')).toBe(4000); + expect(maxOutputTokensFor('anthropic:claude-sonnet-4-6')).toBe(4000); + expect(maxOutputTokensFor('anthropic:claude-3-haiku')).toBe(4000); + expect(maxOutputTokensFor('openai:gpt-4o')).toBe(4000); + expect(maxOutputTokensFor('deepseek:deepseek-reasoner')).toBe(4000); + }); +});